Python HTML解析器BeautifulSoup用法實(shí)例詳解【爬蟲(chóng)解析器】

更新時(shí)間：2019年04月05日 11:10:14 作者：薔薇Nina

這篇文章主要介紹了Python HTML解析器BeautifulSoup用法,結(jié)合實(shí)例形式詳細(xì)分析了第三方庫(kù)BeautifulSoup實(shí)現(xiàn)的爬蟲(chóng)解析器功能具體操作技巧,需要的朋友可以參考下

本文實(shí)例講述了Python HTML解析器BeautifulSoup用法。分享給大家供大家參考，具體如下：

BeautifulSoup簡(jiǎn)介

我們知道，Python擁有出色的內(nèi)置HTML解析器模塊——HTMLParser，然而還有一個(gè)功能更為強(qiáng)大的HTML或XML解析工具——BeautifulSoup（美味的湯），它是一個(gè)第三方庫(kù)。簡(jiǎn)單來(lái)說(shuō)，BeautifulSoup最主要的功能是從網(wǎng)頁(yè)抓取數(shù)據(jù)。本文我們來(lái)感受一下BeautifulSoup的優(yōu)雅而強(qiáng)大的功能吧！

BeautifulSoup安裝

BeautifulSoup3 目前已經(jīng)停止開(kāi)發(fā)，推薦在現(xiàn)在的項(xiàng)目中使用BeautifulSoup4，不過(guò)它已經(jīng)被移植到bs4了，也就是說(shuō)導(dǎo)入時(shí)我們需要 import bs4 ?？梢岳?pip 或者 easy_install 兩種方法來(lái)安裝。下面采用pip安裝。

pip install beautifulsoup4
pip install lxml

建議同時(shí)安裝"lxml"模塊，BeautifulSoup支持Python標(biāo)準(zhǔn)庫(kù)中的HTML解析器（HTMLParser），還支持一些第三方的解析器，如果我們不安裝它，則 Python 會(huì)使用 Python默認(rèn)的解析器，lxml 解析器更加強(qiáng)大，速度更快，推薦安裝。

創(chuàng)建對(duì)象

安裝后，創(chuàng)建對(duì)象：

soup = BeautifulSoup(markup='html文件', 'lxml')

格式化輸出：

soup.prettify()

BeautifulSoup四大對(duì)象類(lèi)型

BeautifulSoup將復(fù)雜HTML文檔轉(zhuǎn)換成一個(gè)復(fù)雜的樹(shù)形結(jié)構(gòu),每個(gè)節(jié)點(diǎn)都是Python對(duì)象,所有對(duì)象可以歸納為4種:

Tag（標(biāo)簽）
NavigableString（內(nèi)容）
BeautifulSoup（文檔）
Comment（注釋?zhuān)?/li>

1.Tag類(lèi)型

即HTML的整個(gè)標(biāo)簽，如獲取<title>標(biāo)簽：

print soup.title
#<title>The Dormouse's story</title>

Tag有兩個(gè)重要屬性：name，attrs。

name

即HTML的標(biāo)簽名稱(chēng)：

print soup.name
#[document]
print soup.head.name
#head

attrs

即HTML的標(biāo)簽屬性字典：

print soup.p.attrs
#{'class': ['title'], 'name': 'dromouse'}

如果想要單獨(dú)獲取某個(gè)屬性：

print soup.p['class']
#['title']

2.NavigableString類(lèi)型

既然我們已經(jīng)得到了整個(gè)標(biāo)簽，那么問(wèn)題來(lái)了，我們要想獲取標(biāo)簽內(nèi)部的文字內(nèi)容怎么辦呢？很簡(jiǎn)單，用 string 即可：

print soup.p.string
#The Dormouse's story

3.BeautifulSoup類(lèi)型

BeautifulSoup 對(duì)象表示的是一個(gè)文檔的全部?jī)?nèi)容.：

print soup.name
# [document]

4.Comment類(lèi)型

HTML的注釋內(nèi)容，注意的是，不包含注釋符號(hào)。我們首先判斷它的類(lèi)型，是否為 Comment 類(lèi)型，然后再進(jìn)行其他操作，如打印輸出：

if type(soup.a.string)==bs4.element.Comment:
  print soup.a.string
#<!-- Elsie -->

遍歷文檔樹(shù)

1.子節(jié)點(diǎn)

contents

獲取所有子節(jié)點(diǎn)，返回列表：

print soup.head.contents
#[<title>The Dormouse's story</title>]

children

獲取所有子節(jié)點(diǎn)，返回列表生成器：

print soup.head.children
#<listiterator object at 0x7f71457f5710>
## 需要遍歷
for child in soup.body.children:
  print child
## 結(jié)果
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1"><!-- Elsie --></a>,
<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a> and
<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>

2.節(jié)點(diǎn)內(nèi)容

string

返回單個(gè)文本內(nèi)容。如果一個(gè)標(biāo)簽里面沒(méi)有標(biāo)簽了，那么 string 就會(huì)返回標(biāo)簽里面的內(nèi)容。如果標(biāo)簽里面只有唯一的一個(gè)標(biāo)簽了，那么 string 也會(huì)返回最里面的內(nèi)容。如果tag包含了多個(gè)子節(jié)點(diǎn),tag就無(wú)法確定，string 方法應(yīng)該調(diào)用哪個(gè)子節(jié)點(diǎn)的內(nèi)容，string 的輸出結(jié)果是 None。例如：

print soup.head.string
print soup.title.string
#The Dormouse's story
#The Dormouse's story
print soup.html.string
# None

strings

返回多個(gè)文本內(nèi)容，且包含空行和空格。

stripped_strings

返回多個(gè)文本內(nèi)容，且不包含空行和空格：

for string in soup.stripped_strings:
  print(repr(string))
  # u"The Dormouse's story"
  # u"The Dormouse's story"
  # u'Once upon a time there were three little sisters; and their names were'
  # u'Elsie'
  # u','
  # u'Lacie'
  # u'and'
  # u'Tillie'
  # u';\nand they lived at the bottom of a well.'
  # u'...'

get_text()方法

返回當(dāng)前節(jié)點(diǎn)和子節(jié)點(diǎn)的文本內(nèi)容。

from bs4 import BeautifulSoup
html_doc = """
<html><head><title>The Dormouse's story</title></head>
<body>
  <p class="title"><b>The Dormouse's story</b></p>
  <p class="story">Once upon a time there were three little sisters; and their names were
    <a  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" class="sister1" id="link1">Elsie</a>,
    <a  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" class="sister2" id="link2">Lacie</a> and
    <a  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" class="sister3" id="link3">Tillie</a>;
    and they lived at the bottom of a well.
  </p>
  <p class="story">...</p>
</body>
</html>
"""
soup = BeautifulSoup(markup=html_doc,features='lxml')
node_p_text=soup.find('p',class_='story').get_text()　　　　# 注意class_帶下劃線
print(node_p_text)
# 結(jié)果
Once upon a time there were three little sisters; and their names were
    Elsie,
    Lacie and
    Tillie;
    and they lived at the bottom of a well.

3.父節(jié)點(diǎn)

parent

返回某節(jié)點(diǎn)的直接父節(jié)點(diǎn)：

p = soup.p
print p.parent.name
#body

parents

返回某節(jié)點(diǎn)的所有父輩及以上輩的節(jié)點(diǎn)：

content = soup.head.title.string
for parent in content.parents:
  print parent.name
## 結(jié)果
title
head
html
[document]

4.兄弟節(jié)點(diǎn)

next_sibling

next_sibling 屬性獲取該節(jié)點(diǎn)的下一個(gè)兄弟節(jié)點(diǎn)，結(jié)果通常是字符串或空白，因?yàn)榭瞻谆蛘邠Q行也可以被視作一個(gè)節(jié)點(diǎn)。

previous_sibling

previous_sibling 屬性獲取該節(jié)點(diǎn)的上一個(gè)兄弟節(jié)點(diǎn)。

print soup.p.next_sibling
#    實(shí)際該處為空白
print soup.p.prev_sibling
#None  沒(méi)有前一個(gè)兄弟節(jié)點(diǎn)，返回 None
print soup.p.next_sibling.next_sibling
#<p class="story">Once upon a time there were three little sisters; and their names were
#<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1"><!-- Elsie --></a>,
#<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a> and
#<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link3">Tillie</a>;
#and they lived at the bottom of a well.</p>
#下一個(gè)節(jié)點(diǎn)的下一個(gè)兄弟節(jié)點(diǎn)是我們可以看到的節(jié)點(diǎn)

next_siblings、previous_siblings

迭代獲取全部兄弟節(jié)點(diǎn)。

5.前后節(jié)點(diǎn)

next_element、previous_element

不是針對(duì)于兄弟節(jié)點(diǎn)，而是在于所有節(jié)點(diǎn)，不分層次的前一個(gè)和后一個(gè)節(jié)點(diǎn)。

next_elements、previous_elements

迭代獲取所有前和后節(jié)點(diǎn)。

搜索文檔樹(shù)

1.find_all(name=None, attrs={}, recursive=True, text=None, limit=None, **kwargs)

find_all()方法搜索當(dāng)前tag的所有tag子節(jié)點(diǎn)，并判斷是否符合過(guò)濾器的條件。

參數(shù)說(shuō)明

name參數(shù)

name參數(shù)很強(qiáng)大，可以傳多種方式的參數(shù)，查找所有名字為 name 的tag，字符串對(duì)象會(huì)被自動(dòng)忽略掉。

（a）傳標(biāo)簽名

最簡(jiǎn)單的過(guò)濾器是標(biāo)簽名。在搜索方法中傳入一個(gè)標(biāo)簽名參數(shù)，BeautifulSoup會(huì)查找與標(biāo)簽名完整匹配的內(nèi)容，下面的例子用于查找文檔中所有的<a>標(biāo)簽：

print soup.find_all('a')
#[<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1"><!-- Elsie --></a>, <a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>, <a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link3">Tillie</a>]

返回結(jié)果列表中的元素仍然是BeautifulSoup對(duì)象。

（b）傳正則表達(dá)式

如果傳入正則表達(dá)式作為參數(shù)，BeautifulSoup會(huì)通過(guò)正則表達(dá)式的 match() 來(lái)匹配內(nèi)容。下面例子中找出所有以b開(kāi)頭的標(biāo)簽,這表示<body>和<b>標(biāo)簽都應(yīng)該被找到：

import re
for tag in soup.find_all(re.compile("^b")):
  print(tag.name)
# body
# b

（c）傳列表

如果傳入列表參數(shù)，BeautifulSoup會(huì)將與列表中任一元素匹配的內(nèi)容返回。下面代碼找到文檔中所有<a>標(biāo)簽和<b>標(biāo)簽：

soup.find_all(["a", "b"])
# [<b>The Dormouse's story</b>,
# <a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1">Elsie</a>,
# <a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>,
# <a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link3">Tillie</a>]

（d）傳True

True 可以匹配任何值，下面代碼查找到所有的tag，但是不會(huì)返回字符串節(jié)點(diǎn)：

for tag in soup.find_all(True):
  print(tag.name)
# html
# head
# title
# body
# p
# b
# p
# a
# a

（e）傳函數(shù)

如果沒(méi)有合適過(guò)濾器，那么還可以定義一個(gè)方法，方法只接受一個(gè)元素參數(shù)。如果這個(gè)方法返回 True 表示當(dāng)前元素匹配并且被找到，如果不是則反回 False：

def has_class_but_no_id(tag):
  return tag.has_attr('class') and not tag.has_attr('id')
soup.find_all(has_class_but_no_id)
# [<p class="title"><b>The Dormouse's story</b></p>,
# <p class="story">Once upon a time there were...</p>,
# <p class="story">...</p>]

keyword參數(shù)

注意的是，如果一個(gè)指定名字的參數(shù)不是搜索內(nèi)置的參數(shù)名，搜索時(shí)會(huì)把該參數(shù)當(dāng)作指定名字tag的屬性來(lái)搜索，如果包含一個(gè)名字為 id 的參數(shù),BeautifulSoup會(huì)搜索每個(gè)tag的”id”屬性：

soup.find_all(id='link2')
# [<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>]

如果傳入 href 參數(shù)，Beautiful Soup會(huì)搜索每個(gè)tag的"href"屬性：

soup.find_all(href=re.compile("elsie"))
# [<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1">Elsie</a>]

使用多個(gè)指定名字的參數(shù)可以同時(shí)過(guò)濾tag的多個(gè)屬性:

soup.find_all(href=re.compile("elsie"), id='link1')
# [<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1">three</a>]

在這里我們想用 class 過(guò)濾，不過(guò) class 是 python 的關(guān)鍵詞，這怎么辦？加個(gè)下劃線就可以：

soup.find_all("a", class_="sister")
# [<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1">Elsie</a>,
# <a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>,
# <a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link3">Tillie</a>]

attrs參數(shù)

有些tag屬性在搜索不能使用,比如HTML5中的 " data-* " 自定義屬性：

data_soup = BeautifulSoup('<div data-foo="value">foo!</div>')
data_soup.find_all(data-foo="value")
# SyntaxError: keyword can't be an expression
## 但是可以通過(guò) find_all() 方法的 attrs 參數(shù)定義一個(gè)字典參數(shù)來(lái)搜索包含特殊屬性的tag
data_soup.find_all(attrs={"data-foo": "value"})
# [<div data-foo="value">foo!</div>]

text參數(shù)

通過(guò) text 參數(shù)可以搜搜文檔中的字符串內(nèi)容。與 name 參數(shù)的可選值一樣，text 參數(shù)接受字符串、正則表達(dá)式、列表、True。

soup.find_all(text="Elsie")
# [u'Elsie']
soup.find_all(text=["Tillie", "Elsie", "Lacie"])
# [u'Elsie', u'Lacie', u'Tillie']
soup.find_all(text=re.compile("Dormouse"))　　# 模糊查找
[u"The Dormouse's story", u"The Dormouse's story"]

limit參數(shù)

find_all() 方法返回全部的搜索結(jié)構(gòu)，如果文檔樹(shù)很大那么搜索會(huì)很慢。如果我們不需要全部結(jié)果，可以使用 limit 參數(shù)限制返回結(jié)果的數(shù)量。效果與SQL中的limit關(guān)鍵字類(lèi)似，當(dāng)搜索到的結(jié)果數(shù)量達(dá)到 limit 的限制時(shí)，就停止搜索返回結(jié)果。

soup.find_all("a", limit=2)
# [<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1">Elsie</a>,
# <a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>]

recursive參數(shù)

調(diào)用tag的 find_all() 方法時(shí)，BeautifulSoup會(huì)檢索當(dāng)前tag的所有子孫節(jié)點(diǎn)，如果只想搜索tag的直接子節(jié)點(diǎn)，可以使用參數(shù) recursive=False。

soup.html.find_all("title")
# [<title>The Dormouse's story</title>]
soup.html.find_all("title", recursive=False)
# []

2.find( name , attrs , recursive , text , **kwargs )

它與 find_all() 方法唯一的區(qū)別是 find_all() 方法的返回結(jié)果是值包含一個(gè)元素的列表，而 find() 方法直接返回結(jié)果。

3.find_parents() 和 find_parent()

find_all() 和 find() 只搜索當(dāng)前節(jié)點(diǎn)的所有子節(jié)點(diǎn)，孫子節(jié)點(diǎn)等。find_parents() 和 find_parent() 用來(lái)搜索當(dāng)前節(jié)點(diǎn)的父輩節(jié)點(diǎn)，搜索方法與普通tag的搜索方法相同，搜索文檔搜索文檔包含的內(nèi)容。

4.find_next_siblings() 和 find_next_sibling()　　

這2個(gè)方法通過(guò) .next_siblings 屬性對(duì)當(dāng) tag 的所有后面解析的兄弟 tag 節(jié)點(diǎn)進(jìn)行迭代， find_next_siblings() 方法返回所有符合條件的后面的兄弟節(jié)點(diǎn)，find_next_sibling() 只返回符合條件的后面的第一個(gè)tag節(jié)點(diǎn)。

5.find_previous_siblings() 和 find_previous_sibling()

這2個(gè)方法通過(guò) .previous_siblings 屬性對(duì)當(dāng)前 tag 的前面解析的兄弟 tag 節(jié)點(diǎn)進(jìn)行迭代， find_previous_siblings() 方法返回所有符合條件的前面的兄弟節(jié)點(diǎn)，find_previous_sibling() 方法返回第一個(gè)符合條件的前面的兄弟節(jié)點(diǎn)。

6.find_all_next() 和 find_next()

這2個(gè)方法通過(guò) .next_elements 屬性對(duì)當(dāng)前 tag 的之后的 tag 和字符串進(jìn)行迭代， find_all_next() 方法返回所有符合條件的節(jié)點(diǎn)， find_next() 方法返回第一個(gè)符合條件的節(jié)點(diǎn)。

7.find_all_previous() 和 find_previous()

這2個(gè)方法通過(guò) .previous_elements 屬性對(duì)當(dāng)前節(jié)點(diǎn)前面的 tag 和字符串進(jìn)行迭代，find_all_previous() 方法返回所有符合條件的節(jié)點(diǎn)， find_previous()方法返回第一個(gè)符合條件的節(jié)點(diǎn)。

CSS選擇器

我們?cè)趯?xiě) CSS 時(shí)，標(biāo)簽名不加任何修飾，類(lèi)名前加點(diǎn)，id名前加 #，在這里我們也可以利用類(lèi)似的方法來(lái)篩選元素，用到的方法是 soup.select()，返回類(lèi)型是 list。

通過(guò)標(biāo)簽名查找

print soup.select('title')
#[<title>The Dormouse's story</title>]
print soup.select('a')
#[<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1"><!-- Elsie --></a>, <a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>, <a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link3">Tillie</a>]
print soup.select('b')
#[<b>The Dormouse's story</b>]

通過(guò)類(lèi)名查找

print soup.select('.sister')
#[<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1"><!-- Elsie --></a>, <a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>, <a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link3">Tillie</a>]

通過(guò) id 名查找

print soup.select('#link1')
#[<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1"><!-- Elsie --></a>]

組合查找

組合查找即和寫(xiě) class 文件時(shí)，標(biāo)簽名與類(lèi)名、id名進(jìn)行的組合原理是一樣的，例如查找 p 標(biāo)簽中，id 等于 link1的內(nèi)容，二者需要用空格分開(kāi)。

print soup.select('p #link1')
#[<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1"><!-- Elsie --></a>]

直接子標(biāo)簽查找：

print soup.select("head > title")
#[<title>The Dormouse's story</title>]

屬性查找

查找時(shí)還可以加入屬性元素，屬性需要用中括號(hào)括起來(lái)，注意屬性和標(biāo)簽屬于同一節(jié)點(diǎn)，所以中間不能加空格，否則會(huì)無(wú)法匹配到。

print soup.select('a[class="sister"]')
#[<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1"><!-- Elsie --></a>, <a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>, <a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link3">Tillie</a>]
print soup.select('a[ rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" ]')
#[<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1"><!-- Elsie --></a>]

同樣，屬性仍然可以與上述查找方式組合，不在同一節(jié)點(diǎn)的空格隔開(kāi)，同一節(jié)點(diǎn)的不加空格：

print soup.select('p a[ rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" ]')
#[<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1"><!-- Elsie --></a>]

以上的 select 方法返回的結(jié)果都是列表形式，可以遍歷形式輸出，然后用 string或get_text() 方法來(lái)獲取它的內(nèi)容：

soup = BeautifulSoup(html, 'lxml')
print type(soup.select('title'))
print soup.select('title')[0].get_text()
for title in soup.select('title'):
  print title.get_text()

更多關(guān)于Python相關(guān)內(nèi)容可查看本站專(zhuān)題：《Python Socket編程技巧總結(jié)》、《Python正則表達(dá)式用法總結(jié)》、《Python數(shù)據(jù)結(jié)構(gòu)與算法教程》、《Python函數(shù)使用技巧總結(jié)》、《Python字符串操作技巧匯總》、《Python入門(mén)與進(jìn)階經(jīng)典教程》及《Python文件與目錄操作技巧匯總》

希望本文所述對(duì)大家Python程序設(shè)計(jì)有所幫助。

您可能感興趣的文章: