詳解Python之Scrapy爬蟲(chóng)教程N(yùn)BA球員數(shù)據(jù)存放到Mysql數(shù)據(jù)庫(kù)
更新時(shí)間：2021年01月24日 17:08:13 作者：我不是禿頭哆唻咪
這篇文章主要介紹了詳解Python之Scrapy爬蟲(chóng)教程N(yùn)BA球員數(shù)據(jù)存放到Mysql數(shù)據(jù)庫(kù)，文中通過(guò)示例代碼介紹的非常詳細(xì)，對(duì)大家的學(xué)習(xí)或者工作具有一定的參考學(xué)習(xí)價(jià)值，需要的朋友們下面隨著小編來(lái)一起學(xué)習(xí)學(xué)習(xí)吧
獲取要爬取的URL
爬蟲(chóng)前期工作
用Pycharm打開(kāi)項(xiàng)目開(kāi)始寫(xiě)爬蟲(chóng)文件
字段文件items
# Define here the models for your scraped items # # See documentation in: # https://docs.scrapy.org/en/latest/topics/items.html import scrapy class NbaprojectItem(scrapy.Item): # define the fields for your item here like: # name = scrapy.Field() # pass # 創(chuàng)建字段的固定格式-->scrapy.Field() # 英文名 engName = scrapy.Field() # 中文名 chName = scrapy.Field() # 身高 height = scrapy.Field() # 體重 weight = scrapy.Field() # 國(guó)家英文名 contryEn = scrapy.Field() # 國(guó)家中文名 contryCh = scrapy.Field() # NBA球齡 experience = scrapy.Field() # 球衣號(hào)碼 jerseyNo = scrapy.Field() # 入選年 draftYear = scrapy.Field() # 隊(duì)伍英文名 engTeam = scrapy.Field() # 隊(duì)伍中文名 chTeam = scrapy.Field() # 位置 position = scrapy.Field() # 東南部 displayConference = scrapy.Field() # 分區(qū) division = scrapy.Field()
爬蟲(chóng)文件
import scrapy
import json
from nbaProject.items import NbaprojectItem
class NbaspiderSpider(scrapy.Spider):
name = 'nbaSpider'
allowed_domains = ['nba.com']
# 第一次爬取的網(wǎng)址,可以寫(xiě)多個(gè)網(wǎng)址
# start_urls = ['http://nba.com/']
start_urls = ['https://china.nba.com/static/data/league/playerlist.json']
# 處理網(wǎng)址的response
def parse(self, response):
# 因?yàn)樵L(fǎng)問(wèn)的網(wǎng)站返回的是json格式,首先用第三方包處理json數(shù)據(jù)
data = json.loads(response.text)['payload']['players']
# 以下列表用來(lái)存放不同的字段
# 英文名
engName = []
# 中文名
chName = []
# 身高
height = []
# 體重
weight = []
# 國(guó)家英文名
contryEn = []
# 國(guó)家中文名
contryCh = []
# NBA球齡
experience = []
# 球衣號(hào)碼
jerseyNo = []
# 入選年
draftYear = []
# 隊(duì)伍英文名
engTeam = []
# 隊(duì)伍中文名
chTeam = []
# 位置
position = []
# 東南部
displayConference = []
# 分區(qū)
division = []
# 計(jì)數(shù)
count = 1
for i in data:
# 英文名
engName.append(str(i['playerProfile']['firstNameEn'] + i['playerProfile']['lastNameEn']))
# 中文名
chName.append(str(i['playerProfile']['firstName'] + i['playerProfile']['lastName']))
# 國(guó)家英文名
contryEn.append(str(i['playerProfile']['countryEn']))
# 國(guó)家中文
contryCh.append(str(i['playerProfile']['country']))
# 身高
height.append(str(i['playerProfile']['height']))
# 體重
weight.append(str(i['playerProfile']['weight']))
# NBA球齡
experience.append(str(i['playerProfile']['experience']))
# 球衣號(hào)碼
jerseyNo.append(str(i['playerProfile']['jerseyNo']))
# 入選年
draftYear.append(str(i['playerProfile']['draftYear']))
# 隊(duì)伍英文名
engTeam.append(str(i['teamProfile']['code']))
# 隊(duì)伍中文名
chTeam.append(str(i['teamProfile']['displayAbbr']))
# 位置
position.append(str(i['playerProfile']['position']))
# 東南部
displayConference.append(str(i['teamProfile']['displayConference']))
# 分區(qū)
division.append(str(i['teamProfile']['division']))
# 創(chuàng)建item字段對(duì)象,用來(lái)存儲(chǔ)信息 這里的item就是對(duì)應(yīng)上面導(dǎo)的NbaprojectItem
item = NbaprojectItem()
item['engName'] = str(i['playerProfile']['firstNameEn'] + i['playerProfile']['lastNameEn'])
item['chName'] = str(i['playerProfile']['firstName'] + i['playerProfile']['lastName'])
item['contryEn'] = str(i['playerProfile']['countryEn'])
item['contryCh'] = str(i['playerProfile']['country'])
item['height'] = str(i['playerProfile']['height'])
item['weight'] = str(i['playerProfile']['weight'])
item['experience'] = str(i['playerProfile']['experience'])
item['jerseyNo'] = str(i['playerProfile']['jerseyNo'])
item['draftYear'] = str(i['playerProfile']['draftYear'])
item['engTeam'] = str(i['teamProfile']['code'])
item['chTeam'] = str(i['teamProfile']['displayAbbr'])
item['position'] = str(i['playerProfile']['position'])
item['displayConference'] = str(i['teamProfile']['displayConference'])
item['division'] = str(i['teamProfile']['division'])
# 打印爬取信息
print("傳輸了",count,"條字段")
count += 1
# 將字段交回給引擎 -> 管道文件
yield item
配置文件->開(kāi)啟管道文件
# Scrapy settings for nbaProject project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
# https://docs.scrapy.org/en/latest/topics/settings.html
# https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
# https://docs.scrapy.org/en/latest/topics/spider-middleware.html
# ----------不做修改部分---------
BOT_NAME = 'nbaProject'
SPIDER_MODULES = ['nbaProject.spiders']
NEWSPIDER_MODULE = 'nbaProject.spiders'
# ----------不做修改部分---------
# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'nbaProject (+http://www.yourdomain.com)'
# Obey robots.txt rules
# ----------修改部分(可以自行查這是啥東西)---------
# ROBOTSTXT_OBEY = True
# ----------修改部分---------
# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32
# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16
# Disable cookies (enabled by default)
#COOKIES_ENABLED = False
# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False
# Override the default request headers:
#DEFAULT_REQUEST_HEADERS = {
# 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
# 'Accept-Language': 'en',
#}
# Enable or disable spider middlewares
# See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
# 'nbaProject.middlewares.NbaprojectSpiderMiddleware': 543,
#}
# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
# 'nbaProject.middlewares.NbaprojectDownloaderMiddleware': 543,
#}
# Enable or disable extensions
# See https://docs.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
# 'scrapy.extensions.telnet.TelnetConsole': None,
#}
# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
# 開(kāi)啟管道文件
# ----------修改部分---------
ITEM_PIPELINES = {
'nbaProject.pipelines.NbaprojectPipeline': 300,
}
# ----------修改部分---------
# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False
# Enable and configure HTTP caching (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
管道文件 -> 將字段寫(xiě)進(jìn)mysql
# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html
# useful for handling different item types with a single interface
from itemadapter import ItemAdapter
import pymysql
class NbaprojectPipeline:
# 初始化函數(shù)
def __init__(self):
# 連接數(shù)據(jù)庫(kù) 注意修改數(shù)據(jù)庫(kù)信息
self.connect = pymysql.connect(host='域名', user='用戶(hù)名', passwd='密碼',
db='數(shù)據(jù)庫(kù)', port=端口號(hào))
# 獲取游標(biāo)
self.cursor = self.connect.cursor()
# 創(chuàng)建一個(gè)表用于存放item字段的數(shù)據(jù)
createTableSql = """
create table if not exists `nbaPlayer`(
playerId INT UNSIGNED AUTO_INCREMENT,
engName varchar(80),
chName varchar(20),
height varchar(20),
weight varchar(20),
contryEn varchar(50),
contryCh varchar(20),
experience int,
jerseyNo int,
draftYear int,
engTeam varchar(50),
chTeam varchar(50),
position varchar(50),
displayConference varchar(50),
division varchar(50),
primary key(playerId)
)charset=utf8;
"""
# 執(zhí)行sql語(yǔ)句
self.cursor.execute(createTableSql)
self.connect.commit()
print("完成了創(chuàng)建表的工作")
#每次yield回來(lái)的字段會(huì)在這里做處理
def process_item(self, item, spider):
# 打印item增加觀賞性
print(item)
# sql語(yǔ)句
insert_sql = """
insert into nbaPlayer(
playerId, engName,
chName,height,
weight,contryEn,
contryCh,experience,
jerseyNo,draftYear
,engTeam,chTeam,
position,displayConference,
division
) VALUES (null,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)
"""
# 執(zhí)行插入數(shù)據(jù)到數(shù)據(jù)庫(kù)操作
# 參數(shù)(sql語(yǔ)句,用item字段里的內(nèi)容替換sql語(yǔ)句的占位符)
self.cursor.execute(insert_sql, (item['engName'], item['chName'], item['height'], item['weight']
, item['contryEn'], item['contryCh'], item['experience'], item['jerseyNo'],
item['draftYear'], item['engTeam'], item['chTeam'], item['position'],
item['displayConference'], item['division']))
# 提交，不進(jìn)行提交無(wú)法保存到數(shù)據(jù)庫(kù)
self.connect.commit()
print("數(shù)據(jù)提交成功!")
啟動(dòng)爬蟲(chóng)
屏幕上滾動(dòng)的數(shù)據(jù)
去數(shù)據(jù)庫(kù)查看數(shù)據(jù)
簡(jiǎn)簡(jiǎn)單單就把球員數(shù)據(jù)爬回來(lái)啦~
到此這篇關(guān)于詳解Python之Scrapy爬蟲(chóng)教程N(yùn)BA球員數(shù)據(jù)存放到Mysql數(shù)據(jù)庫(kù)的文章就介紹到這了,更多相關(guān)Scrapy爬蟲(chóng)員數(shù)據(jù)存放到Mysql內(nèi)容請(qǐng)搜索腳本之家以前的文章或繼續(xù)瀏覽下面的相關(guān)文章希望大家以后多多支持腳本之家！
您可能感興趣的文章:
- Python基于httpx模塊實(shí)現(xiàn)發(fā)送請(qǐng)求
- Python爬蟲(chóng)爬取愛(ài)奇藝電影片庫(kù)首頁(yè)的實(shí)例代碼
- Python爬蟲(chóng)之必備chardet庫(kù)
- Python爬蟲(chóng)進(jìn)階之Beautiful Soup庫(kù)詳解
- Python爬蟲(chóng)之爬取某文庫(kù)文檔數(shù)據(jù)
- Python爬蟲(chóng)爬取全球疫情數(shù)據(jù)并存儲(chǔ)到mysql數(shù)據(jù)庫(kù)的步驟
- 小眾實(shí)用的Python 爬蟲(chóng)庫(kù)RoboBrowser
- python爬蟲(chóng)利器之requests庫(kù)的用法(超全面的爬取網(wǎng)頁(yè)案例)
- python爬蟲(chóng)開(kāi)發(fā)之使用Python爬蟲(chóng)庫(kù)requests多線(xiàn)程抓取貓眼電影TOP100實(shí)例
- python爬蟲(chóng)開(kāi)發(fā)之使用python爬蟲(chóng)庫(kù)requests，urllib與今日頭條搜索功能爬取搜索內(nèi)容實(shí)例
- python爬蟲(chóng)請(qǐng)求庫(kù)httpx和parsel解析庫(kù)的使用測(cè)評(píng)
相關(guān)文章
詳解Python list 與 NumPy.ndarry 切片之間的對(duì)比
這篇文章主要介紹了詳解Python list 與 NumPy.ndarry 切片之間的區(qū)別的相關(guān)資料,list 切片返回的是不原數(shù)據(jù)，對(duì)新數(shù)據(jù)的修改不會(huì)影響原數(shù)據(jù)而NumPy.ndarry 的切片返回的是原數(shù)據(jù)需要的朋友可以參考下2017-07-07
PyCharm中Matplotlib繪圖不能顯示UI效果的問(wèn)題解決
這篇文章主要介紹了PyCharm中Matplotlib繪圖不能顯示UI效果的問(wèn)題解決，文中通過(guò)示例代碼介紹的非常詳細(xì)，對(duì)大家的學(xué)習(xí)或者工作具有一定的參考學(xué)習(xí)價(jià)值，需要的朋友們下面隨著小編來(lái)一起學(xué)習(xí)學(xué)習(xí)吧2020-03-03
Python函數(shù)必須先定義,后調(diào)用說(shuō)明(函數(shù)調(diào)用函數(shù)例外)
這篇文章主要介紹了Python函數(shù)必須先定義,后調(diào)用說(shuō)明(函數(shù)調(diào)用函數(shù)例外)，具有很好的參考價(jià)值，希望對(duì)大家有所幫助。一起跟隨小編過(guò)來(lái)看看吧2020-06-06
OpenCV學(xué)習(xí)之圖像的分割與修復(fù)詳解
圖像分割本質(zhì)就是將前景目標(biāo)從背景中分離出來(lái)。在當(dāng)前的實(shí)際項(xiàng)目中，應(yīng)用傳統(tǒng)分割的并不多，大多是采用深度學(xué)習(xí)的方法以達(dá)到更好的效果。本文將詳細(xì)介紹一下OpenCV中的圖像分割與修復(fù)，需要的可以參考一下2022-01-01
這篇文章主要介紹了Python的deque雙端隊(duì)列詳解,deque(雙端隊(duì)列)是一種數(shù)據(jù)結(jié)構(gòu),允許使用O(1)時(shí)間復(fù)雜度從兩端添加和刪除元素, Python的deque類(lèi)實(shí)現(xiàn)了此數(shù)據(jù)結(jié)構(gòu),需要的朋友可以參考下2023-09-09
python實(shí)現(xiàn)FTP文件傳輸?shù)姆椒ǎǚ?wù)器端和客戶(hù)端）
FTP（File Transfer Protocol，文件傳輸協(xié)議） 是 TCP/IP 協(xié)議組中的協(xié)議之一。接下來(lái)通過(guò)本文給大家介紹關(guān)于python實(shí)現(xiàn)FTP文件傳輸?shù)南嚓P(guān)知識(shí)（服務(wù)器端和客戶(hù)端） ,需要的朋友可以參考下2020-03-03
Python實(shí)現(xiàn)定時(shí)監(jiān)測(cè)網(wǎng)站運(yùn)行狀態(tài)的示例代碼
這篇文章主要介紹了Python實(shí)現(xiàn)定時(shí)監(jiān)測(cè)網(wǎng)站狀態(tài)的示例代碼，幫助大家更好的管理自己的網(wǎng)站，感興趣的朋友可以了解下2020-09-09
python實(shí)現(xiàn)關(guān)鍵詞提取的示例講解
下面小編就為大家分享一篇python實(shí)現(xiàn)關(guān)鍵詞提取的示例講解，具有很好的參考價(jià)值，希望對(duì)大家有所幫助。一起跟隨小編過(guò)來(lái)看看吧2018-04-04
django修改models重建數(shù)據(jù)庫(kù)的操作
這篇文章主要介紹了django修改models重建數(shù)據(jù)庫(kù)的操作，具有很好的參考價(jià)值，希望對(duì)大家有所幫助。一起跟隨小編過(guò)來(lái)看看吧2020-03-03
這篇文章主要介紹了使用Pyqt5制作屏幕錄制界面,主要介紹如何使用ffmpeg將同時(shí)錄制的屏幕錄像和音頻合成為有聲音的屏幕錄像，需要的朋友可以參考下2022-04-04