Scrapy爬蟲實(shí)例講解_?；ňW(wǎng)
更新時(shí)間：2017年10月23日 09:06:45 作者：Python學(xué)習(xí)者
下面小編就為大家?guī)硪黄猄crapy爬蟲實(shí)例講解_校花網(wǎng)。小編覺得挺不錯(cuò)的，現(xiàn)在就分享給大家，也給大家做個(gè)參考。一起跟隨小編過來看看吧
學(xué)習(xí)爬蟲有一段時(shí)間了，今天使用Scrapy框架將?；ňW(wǎng)的圖片爬取到本地。Scrapy爬蟲框架相對于使用requests庫進(jìn)行網(wǎng)頁的爬取，擁有更高的性能。
Scrapy官方定義：Scrapy是用于抓取網(wǎng)站并提取結(jié)構(gòu)化數(shù)據(jù)的應(yīng)用程序框架，可用于廣泛的有用應(yīng)用程序，如數(shù)據(jù)挖掘，信息處理或歷史存檔。
建立Scrapy爬蟲工程
在安裝好Scrapy框架后，直接使用命令行進(jìn)行項(xiàng)目的創(chuàng)建：
E:\ScrapyDemo>scrapy startproject xiaohuar New Scrapy project 'xiaohuar', using template directory 'c:\\users\\lei\\appdata\\local\\programs\\python\\python35\\lib \\site-packages\\scrapy\\templates\\project', created in: E:\ScrapyDemo\xiaohuar You can start your first spider with: cd xiaohuar scrapy genspider example example.com
創(chuàng)建一個(gè)Scrapy爬蟲
創(chuàng)建工程的時(shí)候，會自動創(chuàng)建一個(gè)與工程同名的目錄，進(jìn)入到目錄中執(zhí)行如下命令：
E:\ScrapyDemo\xiaohuar>scrapy genspider -t basic xiaohua xiaohuar.com Created spider 'xiaohua' using template 'basic' in module: xiaohuar.spiders.xiaohua命令中"xiaohua"
是生成Spider中*.py文件的文件名，"xiaohuar.com"是將要爬取網(wǎng)站的URL，可以在程序中更改。
編寫Spider代碼
編寫E:\ScrapyDemo\xiaohuar\xiaohuar\spiders中的xiaohua.py文件。主要是配置URL和對請求到的頁面的解析方式。
# -*- coding: utf-8 -*-
import scrapy
from scrapy.http import Request
import re
class XiaohuaSpider(scrapy.Spider):
name = 'xiaohua'
allowed_domains = ['xiaohuar.com']
start_urls = []
for i in range(43):
url = "http://www.xiaohuar.com/list-1-%s.html" %i
start_urls.append(url)
def parse(self, response):
if "www.xiaohuar.com/list-1" in response.url:
# 下載的html源代碼
html = response.text
# 網(wǎng)頁中圖片存儲地址：src="/d/file/20160126/905e563421921adf9b6fb4408ec4e72f.jpg"
# 通過正則匹配到所有的圖片
# 獲取的是圖片的相對路徑的列表
img_urls = re.findall(r'/d/file/\d+/\w+\.jpg',html)
# 使用循環(huán)對圖片頁進(jìn)行請求
for img_url in img_urls:
# 將圖片的URL補(bǔ)全
if "http://" not in img_url:
img_url = "http://www.xiaohuar.com%s" %img_url
# 回調(diào)，返回response
yield Request(img_url)
else:
# 下載圖片
url = response.url
# 保存的圖片文件名
title = re.findall(r'\w*.jpg',url)[0]
# 保存圖片
with open('E:\\xiaohua_img\\%s' % title, 'wb') as f:
f.write(response.body)
這里使用正則表達(dá)式對圖片的地址進(jìn)行匹配，其他網(wǎng)頁也都大同小異，需要根據(jù)具體的網(wǎng)頁源代碼進(jìn)行分析。
運(yùn)行爬蟲
E:\ScrapyDemo\xiaohuar>scrapy crawl xiaohua
2017-10-22 22:30:11 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: xiaohuar)
2017-10-22 22:30:11 [scrapy.utils.log] INFO: Overridden settings: {'BOT_NAME': 'xiaohuar', 'SPIDER_MODULES': ['xiaohuar.
spiders'], 'ROBOTSTXT_OBEY': True, 'NEWSPIDER_MODULE': 'xiaohuar.spiders'}
2017-10-22 22:30:11 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.logstats.LogStats']
2017-10-22 22:30:12 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-10-22 22:30:12 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-10-22 22:30:12 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2017-10-22 22:30:12 [scrapy.core.engine] INFO: Spider opened
2017-10-22 22:30:12 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min
)
2017-10-22 22:30:12 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-10-22 22:30:12 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/robots.txt> (referer: None)
2017-10-22 22:30:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/list-1-0.html> (referer: None
)
2017-10-22 22:30:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170721/cb96f1b106b3d
b4a6bfcf3d2e880dea0.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170824/dcc166b0eba6a
37e05424cfc29023121.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170916/7f78145b1ca16
2eb814fbc03ad24fbc1.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170919/2f728d0f110a2
1fea95ce13e0b010d06.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170819/9c3dfeef7e08c
c0303ce233e4ddafa7f.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170917/715515e7fe1f1
cb9fd388bbbb00467c2.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170628/f3d06ef49965a
edbe18286a2f221fd9f.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170513/6121e3e90ff3b
a4c9398121bda1dd582.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170516/6e295fe48c332
45be858c40d37fb5ee6.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170707/f7ca636f73937
e33836e765b7261f036.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170528/b352258c83776
b9a2462277dec375d0c.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170527/4a7a7f1e6b69f
126292b981c90110d0a.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170715/61110ba027f00
4fb503ff09cdee44d0c.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170520/dd21a21751e24
a8f161792b66011688c.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170529/8140c4ad797ca
01f5e99d09c82dd8a42.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170603/e55f77fb3aa3c
7f118a46eeef5c0fbbf.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170529/e5902d4d3e408
29f9a0d30f7488eab84.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170604/ec3794d0d42b5
38bf4461a84dac32509.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170603/c34b29f68e8f9
6d44c63fe29bf4a66b8.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170701/fb18711a6af87
f30942d6a19f6da6b3e.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170619/e0456729d4dcb
ea569a1acbc6a47ab69.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170626/0ab1d89f54c90
df477a90aa533ceea36.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:15 [scrapy.core.engine] INFO: Closing spider (finished)
2017-10-22 22:30:15 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 8785,
'downloader/request_count': 24,
'downloader/request_method_count/GET': 24,
'downloader/response_bytes': 2278896,
'downloader/response_count': 24,
'downloader/response_status_count/200': 24,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2017, 10, 22, 14, 30, 15, 892287),
'log_count/DEBUG': 25,
'log_count/INFO': 7,
'request_depth_max': 1,
'response_received_count': 24,
'scheduler/dequeued': 23,
'scheduler/dequeued/memory': 23,
'scheduler/enqueued': 23,
'scheduler/enqueued/memory': 23,
'start_time': datetime.datetime(2017, 10, 22, 14, 30, 12, 698874)}
2017-10-22 22:30:15 [scrapy.core.engine] INFO: Spider closed (finished)
scrapy crawl xiaohua
圖片保存
在圖片保存過程中"\"需要進(jìn)行轉(zhuǎn)義。
>>> r = requests.get("https://timgsa.baidu.com/timg?image&quality=80&size=b9999_10000&sec=1508693697147&di=23eb655d8e450f84cf39453bc1029bc0&imgtype=0&src=http%3A%2F%2Fb.hiphotos.baidu.com%2Fimage%2Fpic%2Fitem%2Fc9fcc3cec3fdfc038b027f7bde3f8794a5c226fe.jpg")
>>> open("E:\xiaohua_img\01.jpg",'wb').write(r.content)
File "<stdin>", line 1
SyntaxError: (unicode error) 'unicodeescape' codec can't decode by
>>> open("E:\\xiaohua_img\1.jpg",'wb').write(r.content)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OSError: [Errno 22] Invalid argument: 'E:\\xiaohua_img\x01.jpg'
>>> open("E:\\xiaohua_img\\1.jpg",'wb').write(r.content)
以上這篇Scrapy爬蟲實(shí)例講解_校花網(wǎng)就是小編分享給大家的全部內(nèi)容了，希望能給大家一個(gè)參考，也希望大家多多支持腳本之家。
相關(guān)文章
Python實(shí)現(xiàn)url長短鏈接的轉(zhuǎn)換
短鏈接，通俗來說，就是將長的URL網(wǎng)址，通過程序計(jì)算等方式，轉(zhuǎn)換為簡短的網(wǎng)址字符串。本文將用Python語言實(shí)現(xiàn)這一效果，需要的可以參考一下2022-11-11
在本篇文章里小編給大家整理了一篇關(guān)于python shutil操作文件實(shí)例講解內(nèi)容，有興趣的朋友們可以學(xué)習(xí)下。2021-03-03
python torch.utils.data.DataLoader使用方法
這篇文章主要介紹了python torch.utils.data.DataLoader使用方法，文中通過示例代碼介紹的非常詳細(xì)，對大家的學(xué)習(xí)或者工作具有一定的參考學(xué)習(xí)價(jià)值，需要的朋友們下面隨著小編來一起學(xué)習(xí)學(xué)習(xí)吧2020-04-04
python pyinstaller打包exe報(bào)錯(cuò)的解決方法
這篇文章主要給大家介紹了關(guān)于python pyinstaller打包exe報(bào)錯(cuò)的解決方法，文中通過示例代碼介紹的非常詳細(xì)，對大家的學(xué)習(xí)或者使用python具有一定的參考學(xué)習(xí)價(jià)值，需要的朋友們下面來一起學(xué)習(xí)學(xué)習(xí)吧2019-11-11
這篇文章主要介紹了Python使用Pandas庫常見操作,結(jié)合實(shí)例形式詳細(xì)分析了Python Pandas模塊的功能、原理、數(shù)據(jù)對象創(chuàng)建、查看、選擇等相關(guān)操作技巧與注意事項(xiàng),需要的朋友可以參考下2021-10-10
Scrapy爬蟲框架集成selenium及全面詳細(xì)講解
這篇文章主要為大家介紹了Scrapy集成selenium，以及scarpy爬蟲框架全面講解，有需要的朋友可以借鑒參考下，希望能夠有所幫助，祝大家多多進(jìn)步早日升職加薪2022-04-04
Python?3.12安裝庫報(bào)錯(cuò)解決方案
這篇文章主要介紹了Python?3.12安裝庫報(bào)錯(cuò)的解決方案,講解了Python?3.12移除pkgutil.ImpImporter支持導(dǎo)致的AttributeError錯(cuò)誤,并提供了兩種解決方案,需要的朋友可以參考下2025-03-03
使用Python自動化Microsoft Excel和Word的操作方法
這篇文章主要介紹了使用Python自動化Microsoft Excel和Word,本文給大家介紹的非常詳細(xì)，對大家的學(xué)習(xí)或工作具有一定的參考借鑒價(jià)值，需要的朋友可以參考下2021-04-04
Python淘寶或京東等秒殺搶購腳本實(shí)現(xiàn)(秒殺腳本)
本篇文章主要介紹了Python 通過selenium實(shí)現(xiàn)毫秒級自動搶購的示例代碼，通過掃碼登錄即可自動完成一系列操作，搶購時(shí)間精確至毫秒，可搶加購物車等待時(shí)間結(jié)算的，也可以搶聚劃算、火車票等的商品，感興趣的朋友跟隨小編一起看看吧2022-10-10
這篇文章主要介紹了Python之多進(jìn)程與多線程的使用，文中通過示例代碼介紹的非常詳細(xì)，對大家的學(xué)習(xí)或者工作具有一定的參考學(xué)習(xí)價(jià)值，需要的朋友們下面隨著小編來一起學(xué)習(xí)學(xué)習(xí)吧2021-02-02