python中httpx庫(kù)的詳細(xì)使用方法及案例詳解
1. 安裝 httpx
首先,確保已經(jīng)安裝了 httpx。可以通過以下命令安裝:pip install httpx
如果需要支持 HTTP/2,可以安裝額外依賴:pip install httpx[http2]
2. 同步請(qǐng)求
發(fā)送 GET 請(qǐng)求
import httpx
# 發(fā)送 GET 請(qǐng)求
response = httpx.get('https://httpbin.org/get')
print(response.status_code) # 狀態(tài)碼
print(response.text) # 響應(yīng)內(nèi)容
發(fā)送 POST 請(qǐng)求
# 發(fā)送 POST 請(qǐng)求
data = {'key': 'value'}
response = httpx.post('https://httpbin.org/post', json=data)
print(response.json()) # 解析 JSON 響應(yīng)
設(shè)置請(qǐng)求頭
headers = {'User-Agent': 'my-app/1.0.0'}
response = httpx.get('https://httpbin.org/headers', headers=headers)
print(response.json())
設(shè)置查詢參數(shù)
params = {'key1': 'value1', 'key2': 'value2'}
response = httpx.get('https://httpbin.org/get', params=params)
print(response.json())
處理超時(shí)
try:
response = httpx.get('https://httpbin.org/delay/5', timeout=2.0)
except httpx.TimeoutException:
print("請(qǐng)求超時(shí)")
3. 異步請(qǐng)求
httpx 支持異步操作,適合高性能場(chǎng)景。
發(fā)送異步 GET 請(qǐng)求
import httpx
import asyncio
async def fetch(url):
async with httpx.AsyncClient() as client:
response = await client.get(url)
print(response.text)
asyncio.run(fetch('https://httpbin.org/get'))
發(fā)送異步 POST 請(qǐng)求
async def post_data(url, data):
async with httpx.AsyncClient() as client:
response = await client.post(url, json=data)
print(response.json())
asyncio.run(post_data('https://httpbin.org/post', {'key': 'value'}))
并發(fā)請(qǐng)求
async def fetch_multiple(urls):
async with httpx.AsyncClient() as client:
tasks = [client.get(url) for url in urls]
responses = await asyncio.gather(*tasks)
for response in responses:
print(response.text)
urls = ['https://httpbin.org/get', 'https://httpbin.org/ip']
asyncio.run(fetch_multiple(urls))
4. 高級(jí)功能
使用 HTTP/2
# 啟用 HTTP/2
client = httpx.Client(http2=True)
response = client.get('https://httpbin.org/get')
print(response.http_version) # 輸出協(xié)議版本
文件上傳
files = {'file': open('example.txt', 'rb')}
response = httpx.post('https://httpbin.org/post', files=files)
print(response.json())
流式請(qǐng)求
# 流式上傳
def generate_data():
yield b"part1"
yield b"part2"
response = httpx.post('https://httpbin.org/post', data=generate_data())
print(response.json())
流式響應(yīng)
# 流式下載
with httpx.stream('GET', 'https://httpbin.org/stream/10') as response:
for chunk in response.iter_bytes():
print(chunk)
5. 錯(cuò)誤處理
httpx 提供了多種異常類,方便處理錯(cuò)誤。
處理網(wǎng)絡(luò)錯(cuò)誤
try:
response = httpx.get('https://nonexistent-domain.com')
except httpx.NetworkError:
print("網(wǎng)絡(luò)錯(cuò)誤")
處理 HTTP 錯(cuò)誤狀態(tài)碼
response = httpx.get('https://httpbin.org/status/404')
if response.status_code == 404:
print("頁(yè)面未找到")
6. 配置客戶端
可以通過 httpx.Client 或 httpx.AsyncClient 配置全局設(shè)置。
設(shè)置超時(shí)
client = httpx.Client(timeout=10.0)
response = client.get('https://httpbin.org/get')
print(response.text)
設(shè)置代理
proxies = {
"http://": "http://proxy.example.com:8080",
"https://": "http://proxy.example.com:8080",
}
client = httpx.Client(proxies=proxies)
response = client.get('https://httpbin.org/get')
print(response.text)
設(shè)置基礎(chǔ) URL
client = httpx.Client(base_url='https://httpbin.org')
response = client.get('/get')
print(response.text)
7. 結(jié)合 Beautiful Soup 使用
httpx 可以與 Beautiful Soup 結(jié)合使用,抓取并解析網(wǎng)頁(yè)。
import httpx
from bs4 import BeautifulSoup
# 抓取網(wǎng)頁(yè)
response = httpx.get('https://example.com')
html = response.text
# 解析網(wǎng)頁(yè)
soup = BeautifulSoup(html, 'lxml')
title = soup.find('title').text
print("網(wǎng)頁(yè)標(biāo)題:", title)
8. 示例:抓取并解析網(wǎng)頁(yè)
以下是一個(gè)完整的示例,展示如何使用 httpx 抓取并解析網(wǎng)頁(yè)數(shù)據(jù):
import httpx
from bs4 import BeautifulSoup
# 抓取網(wǎng)頁(yè)
url = 'https://example.com'
response = httpx.get(url)
html = response.text
# 解析網(wǎng)頁(yè)
soup = BeautifulSoup(html, 'lxml')
# 提取標(biāo)題
title = soup.find('title').text
print("網(wǎng)頁(yè)標(biāo)題:", title)
# 提取所有鏈接
links = soup.find_all('a', href=True)
for link in links:
href = link['href']
text = link.text
print(f"鏈接文本: {text}, 鏈接地址: {href}")
9. 注意事項(xiàng)
性能:httpx 的異步模式適合高并發(fā)場(chǎng)景。
兼容性:httpx 的 API 與 requests 高度兼容,遷移成本低。
HTTP/2:如果需要使用 HTTP/2,確保安裝了 httpx[http2]。
通過以上方法,可以使用 httpx 高效地發(fā)送 HTTP 請(qǐng)求,并結(jié)合其他工具(如 Beautiful Soup)實(shí)現(xiàn)數(shù)據(jù)抓取和解析。
到此這篇關(guān)于python中httpx庫(kù)的詳細(xì)使用方法及案例詳解的文章就介紹到這了,更多相關(guān)python httpx庫(kù)使用及案例內(nèi)容請(qǐng)搜索腳本之家以前的文章或繼續(xù)瀏覽下面的相關(guān)文章希望大家以后多多支持腳本之家!
相關(guān)文章
flask/django 動(dòng)態(tài)查詢表結(jié)構(gòu)相同表名不同數(shù)據(jù)的Model實(shí)現(xiàn)方法
今天小編就為大家分享一篇flask/django 動(dòng)態(tài)查詢表結(jié)構(gòu)相同表名不同數(shù)據(jù)的Model實(shí)現(xiàn)方法,具有很好的參考價(jià)值,希望對(duì)大家有所幫助。一起跟隨小編過來看看吧2019-08-08
Python3 操作 MySQL 插入一條數(shù)據(jù)并返回主鍵 id的實(shí)例
這篇文章主要介紹了Python3 操作 MySQL 插入一條數(shù)據(jù)并返回主鍵 id的實(shí)例,具有很好的參考價(jià)值,希望對(duì)大家有所幫助。一起跟隨小編過來看看吧2020-03-03

