在centos7中分布式部署pyspider
更新時(shí)間：2017年05月03日 08:39:48 作者：howie6879
PySpider：一個(gè)國人編寫的強(qiáng)大的網(wǎng)絡(luò)爬蟲系統(tǒng)并帶有強(qiáng)大的WebUI。采用Python語言編寫，分布式架構(gòu)，支持多種數(shù)據(jù)庫后端，強(qiáng)大的WebUI支持腳本編輯器，任務(wù)監(jiān)視器，項(xiàng)目管理器以及結(jié)果查看器。
1.搭建環(huán)境：
系統(tǒng)版本：Linux centos-linux.shared 3.10.0-123.el7.x86_64 #1 SMP Mon Jun 30 12:09:22 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
python版本：Python 3.5.1
1.1.搭建python3環(huán)境：
本人在嘗試過后選擇集成環(huán)境Anaconda
1.1.1.編譯
# 下載依賴 yum install -y ncurses-devel openssl openssl-devel zlib-devel gcc make glibc-devel libffi-devel glibc-static glibc-utils sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-deve # 下載python版本 wget https://www.python.org/ftp/python/3.5.1/Python-3.5.1.tgz # 或者使用國內(nèi)源 wget http://mirrors.sohu.com/python/3.5.1/Python-3.5.1.tgz mv Python-3.5.1.tgz /usr/local/src;cd /usr/local/src # 解壓 tar -zxf Python-3.5.1.tgz;cd Python-3.5.1 # 編譯安裝 ./configure --prefix=/usr/local/python3.5 --enable-shared make && make install # 建立軟鏈接 ln -s /usr/local/python3.5/bin/python3 /usr/bin/python3 echo "/usr/local/python3.5/lib" > /etc/ld.so.conf.d/python3.5.conf ldconfig # 驗(yàn)證python3 python3 # Python 3.5.1 (default, Oct 9 2016, 11:44:24) # [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux # Type "help", "copyright", "credits" or "license" for more information. # >>> # pip /usr/local/python3.5/bin/pip3 install --upgrade pip ln -s /usr/local/python3.5/bin/pip /usr/bin/pip # 本人在安裝時(shí)出現(xiàn)問題 將pip重裝 wget https://bootstrap.pypa.io/get-pip.py --no-check-certificate python get-pip.py
1.1.2.集成環(huán)境anaconda
# 集成環(huán)境anaconda(推薦) wget https://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh # 直接安裝即可 ./Anaconda3-4.2.0-Linux-x86_64.sh # 若出錯(cuò)，可能是解壓失敗 yum install bzip2
1.2.安裝mariaDB
# 安裝 yum -y install mariadb mariadb-server # 啟動(dòng) systemctl start mariadb # 設(shè)置為開機(jī)啟動(dòng) systemctl enable mariadb # 配置密碼 默認(rèn)為空 mysql_secure_installation # 登錄 mysql -u root -p # 創(chuàng)建一個(gè)用戶 自己設(shè)定賬戶密碼 CREATE USER 'user_name'@'localhost' IDENTIFIED BY 'user_pass'; GRANT ALL PRIVILEGES ON *.* TO 'user_name'@'localhost' WITH GRANT OPTION; CREATE USER 'user_name'@'%' IDENTIFIED BY 'user_pass'; GRANT ALL PRIVILEGES ON *.* TO 'user_name'@'%' WITH GRANT OPTION;
1.3.安裝pyspider
本人使用Anaconda
# 搭建虛擬環(huán)境sbird python版本3.* conda create -n sbird python=3* # 進(jìn)入環(huán)境 source activate sbird # 安裝pyspider pip install pyspider # 報(bào)錯(cuò) # it does not exist. The exported locale is "en_US.UTF-8" but it is not supported # 執(zhí)行 可寫入.bashrc export LC_ALL=en_US.utf-8 export LANG=en_US.utf-8 #ImportError: pycurl: libcurl link-time version (7.29.0) is older than compile-time version (7.49.0) conda install pycurl # 退出 source deactivate sbird # 若在虛擬機(jī)內(nèi) 出現(xiàn)無法訪問localhost:5000 可關(guān)閉防火墻 systemctl stop firewalld.service #########直接運(yùn)行源碼============== mkdir git;cd git # 下載 git clone https://github.com/binux/pyspider.git # 安裝 /root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py
其他方法
# 搭建虛擬環(huán)境 pip install virtualenv mkdir python;cd python # 創(chuàng)建虛擬環(huán)境pyenv3 virtualenv -p /usr/bin/python3 pyenv3 # 進(jìn)入虛擬環(huán)境 激活環(huán)境 cd pyenv3/ source ./bin/activate pip install pyspider # 若pycurl報(bào)錯(cuò) yum install libcurl-devel # 繼續(xù) pip install pyspider # 關(guān)閉 deactivate
本人推薦用anaconda方式安裝
若pyspider運(yùn)行過程中出現(xiàn)錯(cuò)誤，參考anaconda安裝部分，至此，訪問localhost:5000可看到頁面。
1.4.安裝Supervisor
# 安裝 yum install supervisor -y # 若無法檢索 則添加阿里的epel源 vim /etc/yum.repos.d/epel.repo # 添加以下內(nèi)容 [epel] name=Extra Packages for Enterprise Linux 7 - $basearch baseurl=http://mirrors.aliyun.com/epel/7/$basearch http://mirrors.aliyuncs.com/epel/7/$basearch failovermethod=priority enabled=1 gpgcheck=0 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 [epel-debuginfo] name=Extra Packages for Enterprise Linux 7 - $basearch - Debug baseurl=http://mirrors.aliyun.com/epel/7/$basearch/debug http://mirrors.aliyuncs.com/epel/7/$basearch/debug failovermethod=priority enabled=0 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 gpgcheck=0 [epel-source] name=Extra Packages for Enterprise Linux 7 - $basearch - Source baseurl=http://mirrors.aliyun.com/epel/7/SRPMS http://mirrors.aliyuncs.com/epel/7/SRPMS failovermethod=priority enabled=0 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 gpgcheck=0 # 安裝 yum install supervisor -y # 測試是否安裝成功 echo_supervisord_conf
1.4.1.Supervisor用法
supervisord #supervisor的服務(wù)器端部分 啟動(dòng) supervisorctl #啟動(dòng)supervisor的命令行窗口 # 假設(shè)創(chuàng)建進(jìn)程pyspider01 vim /etc/supervisord.d/pyspider01.ini # 寫入以下內(nèi)容 [program:pyspider01] command = /root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py directory = /root/git/pyspider user = root process_name = %(program_name)s autostart = true autorestart = true startsecs = 3 redirect_stderr = true stdout_logfile_maxbytes = 500MB stdout_logfile_backups = 10 stdout_logfile = /pyspider/supervisor/pyspider01.log # 重載 supervisorctl reload # 啟動(dòng) supervisorctl start pyspider01 # 也可這樣啟動(dòng) supervisord -c /etc/supervisord.conf # 查看狀態(tài) supervisorctl status # output pyspider01 RUNNING pid 4026, uptime 0:02:40 # 關(guān)閉 supervisorctl shutdown
1.5.安裝redis
# 消息隊(duì)列采用redis mkdir download;cd download wget http://download.redis.io/releases/redis-3.2.4.tar.gz tar xzf redis-3.2.4.tar.gz cd redis-3.2.4 make # 或者直接yum安裝 yum -y install redis # 啟動(dòng) systemctl start redis.service # 重啟 systemctl restart redis.service # 停止 systemctl stop redis.service # 查看狀態(tài) systemctl status redis.service # 更改文件/etc/redis.conf vim /etc/redis.conf # 更改內(nèi)容 daemonize no 改為 daemonize yes bind 127.0.0.1 改為 bind 10.211.55.22(當(dāng)前服務(wù)器ip) # 重啟redis systemctl restart redis.service
1.6.關(guān)于自啟動(dòng)
# Supervisor添加到自啟動(dòng)服務(wù) systemctl enable supervisord.service # redis添加到自啟動(dòng)服務(wù) systemctl enable redis.service # 關(guān)閉防火墻自啟動(dòng) systemctl disable firewalld.service
至此，pyspider單個(gè)服務(wù)器運(yùn)行環(huán)境搭建且部署完畢，啟動(dòng)localhost:5000進(jìn)入web界面。
也可編寫腳本運(yùn)行，在/pyspider/supervisor/pyspider01.log查看運(yùn)行狀態(tài)。
2.分布式部署
剛才配置的服務(wù)器，將其命名為centos01，按照這樣的配置，再分別部署兩臺centos02、centos03。
如下：
服務(wù)器名稱 ip 說明
centos01 10.211.55.22 redis,mariaDB, scheduler centos02 10.211.55.23 fetcher, processor, result_worker,phantomjs centos03 10.211.55.24 fetcher, processor,,result_worker,webui
2.1.centos01
進(jìn)入服務(wù)器centos01，經(jīng)過第一步，基本環(huán)境已經(jīng)搭好，首先編輯配置文件/pyspider/config.json
{
"taskdb": "mysql+taskdb://user_name:user_pass@10.211.55.22:3306/taskdb",
"projectdb": "mysql+projectdb://user_name:user_pass@10.211.55.22:3306/projectdb",
"resultdb": "mysql+resultdb://user_name:user_pass@10.211.55.22:3306/resultdb",
"message_queue": "redis://10.211.55.22:6379/db",
"logging-config": "/pyspider/logging.conf",
"phantomjs-proxy":"10.211.55.23:25555",
"webui": {
"username": "",
"password": "",
"need-auth": false,
"host":"10.211.55.24",
"port":"5000",
"scheduler-rpc":"http:// 10.211.55.22:5002",
"fetcher-rpc":"http://10.211.55.23:5001"
},
"fetcher": {
"xmlrpc":true,
"xmlrpc-host": "0.0.0.0",
"xmlrpc-port": "5001"
},
"scheduler": {
"xmlrpc":true,
"xmlrpc-host": "0.0.0.0",
"xmlrpc-port": "5002"
}
}
嘗試運(yùn)行下：
/root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json scheduler # 報(bào)錯(cuò) ImportError: No module named 'mysql' # 下載 mysql-connector-python cd ~/git/ git clone https://github.com/mysql/mysql-connector-python.git # 安裝 source activate sbird cd mysql-connector-python python setup.py install # 安裝redis pip install redis source deactivate # 運(yùn)行 /root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json scheduler # 輸出 ok [I 161010 15:57:25 scheduler:644] scheduler starting... [I 161010 15:57:25 scheduler:779] scheduler.xmlrpc listening on 0.0.0.0:5002 [I 161010 15:57:25 scheduler:583] in 5m: new:0,success:0,retry:0,failed:0
運(yùn)行成功后，可直接更改/etc/supervisord.d/pyspider01.ini如下：
[program:pyspider01] command = /root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json scheduler directory = /root/git/pyspider user = root process_name = %(program_name)s autostart = true autorestart = true startsecs = 3 redirect_stderr = true stdout_logfile_maxbytes = 500MB stdout_logfile_backups = 10 stdout_logfile = /pyspider/supervisor/pyspider01.log # 重載 supervisorctl reload # 查看狀態(tài) supervisorctl status
centos01部署完畢。
2.2.centos02
在centos02中，需要運(yùn)行result_worker、processor、phantomjs、fetcher
分別建立文件：
/etc/supervisord.d/result_worker.ini [program:result_worker] command = /root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json result_worker directory = /root/git/pyspider user = root process_name = %(program_name)s autostart = true autorestart = true startsecs = 3 redirect_stderr = true stdout_logfile_maxbytes = 500MB stdout_logfile_backups = 10 stdout_logfile = /pyspider/supervisor/result_worker.log /etc/supervisord.d/processor.ini [program:processor] command = /root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json processor directory = /root/git/pyspider user = root process_name = %(program_name)s autostart = true autorestart = true startsecs = 3 redirect_stderr = true stdout_logfile_maxbytes = 500MB stdout_logfile_backups = 10 stdout_logfile = /pyspider/supervisor/processor.log /etc/supervisord.d/phantomjs.ini [program:phantomjs] command = /pyspider/phantomjs --config=/pyspider/pjsconfig.json /pyspider/phantomjs_fetcher.js 25555 directory = /root/git/pyspider user = root process_name = %(program_name)s autostart = true autorestart = true startsecs = 3 redirect_stderr = true stdout_logfile_maxbytes = 500MB stdout_logfile_backups = 10 stdout_logfile = /pyspider/supervisor/phantomjs.log /etc/supervisord.d/fetcher.ini [program:fetcher] command = /root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json fetcher directory = /root/git/pyspider user = root process_name = %(program_name)s autostart = true autorestart = true startsecs = 3 redirect_stderr = true stdout_logfile_maxbytes = 500MB stdout_logfile_backups = 10 stdout_logfile = /pyspider/supervisor/fetcher.log
在pyspider目錄中建立pjsconfig.json
{
/*--ignore-ssl-errors=true */
"ignoreSslErrors": true,
/*--ssl-protocol=true */
"sslprotocol": "any",
/* Same as: --output-encoding=utf8 */
"outputEncoding": "utf8",
/* persistent Cookies. */
/*cookiesfile="e:/phontjscookies.txt",*/
cookiesfile="pyspider/phontjscookies.txt",
/* load image */
autoLoadImages = false
}
下載phantomjs至/pyspider/文件夾，將git/pyspider/pyspider/fetcher/phantomjs_fetcher.js復(fù)制到phantomjs_fetcher.js
# 重載 supervisorctl reload # 查看狀態(tài) supervisorctl status # output fetcher RUNNING pid 3446, uptime 0:00:07 phantomjs RUNNING pid 3448, uptime 0:00:07 processor RUNNING pid 3447, uptime 0:00:07 result_worker RUNNING pid 3445, uptime 0:00:07
centos02部署完畢。
2.3.centos03
部署這三個(gè)進(jìn)程fetcher, processor, result_worker和centos02 一樣，本服務(wù)器主要是在前面的基礎(chǔ)上加上webui
建立文件：
/etc/supervisord.d/webui.ini [program:webui] command = /root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json webui directory = /root/git/pyspider user = root process_name = %(program_name)s autostart = true autorestart = true startsecs = 3 redirect_stderr = true stdout_logfile_maxbytes = 500MB stdout_logfile_backups = 10 stdout_logfile = /pyspider/supervisor/webui.log # 重載 supervisorctl reload # 查看狀態(tài) supervisorctl status # output fetcher RUNNING pid 2724, uptime 0:00:07 processor RUNNING pid 2725, uptime 0:00:07 result_worker RUNNING pid 2723, uptime 0:00:07 webui RUNNING pid 2726, uptime 0:00:07
3.總結(jié)
訪問 http://10.211.55.24:5000 即可，盡情爬取吧。
相關(guān)文章
Python爬蟲beautifulsoup4常用的解析方法總結(jié)
今天小編就為大家分享一篇關(guān)于Python爬蟲beautifulsoup4常用的解析方法總結(jié)，小編覺得內(nèi)容挺不錯(cuò)的，現(xiàn)在分享給大家，具有很好的參考價(jià)值，需要的朋友一起跟隨小編來看看吧2019-02-02
tensorflow實(shí)現(xiàn)softma識別MNIST
這篇文章主要為大家詳細(xì)介紹了tensorflow實(shí)現(xiàn)softma識別MNIST，具有一定的參考價(jià)值，感興趣的小伙伴們可以參考一下2018-03-03
關(guān)于Python使用logging庫進(jìn)行有效日志管理的方法詳解
在開發(fā)大型軟件或處理復(fù)雜問題時(shí)，我們經(jīng)常需要一種方法來記錄和跟蹤程序的運(yùn)行狀態(tài)，Python 提供了一個(gè)名為 logging 的標(biāo)準(zhǔn)庫，可以幫助我們更好地完成這項(xiàng)任務(wù)，在這篇文章中，我們將介紹如何使用 Python 的 logging 庫進(jìn)行日志記錄2023-06-06
pandas:get_dummies()與pd.factorize()的用法及區(qū)別說明
這篇文章主要介紹了pandas:get_dummies()與pd.factorize()的用法及區(qū)別說明，具有很好的參考價(jià)值，希望對大家有所幫助。如有錯(cuò)誤或未考慮完全的地方，望不吝賜教2021-05-05
python默認(rèn)參數(shù)調(diào)用方法解析
這篇文章主要介紹了python默認(rèn)參數(shù)調(diào)用方法解析,文中通過示例代碼介紹的非常詳細(xì)，對大家的學(xué)習(xí)或者工作具有一定的參考學(xué)習(xí)價(jià)值,需要的朋友可以參考下2020-02-02
今天小編就為大家分享一篇關(guān)于Tesserocr庫的正確安裝方式，小編覺得內(nèi)容挺不錯(cuò)的，現(xiàn)在分享給大家，具有很好的參考價(jià)值，需要的朋友一起跟隨小編來看看吧2018-10-10
Python統(tǒng)計(jì)字符內(nèi)容的占比的實(shí)現(xiàn)
本文介紹了如何使用Python統(tǒng)計(jì)字符占比，包括字符串中字母、數(shù)字、空格等字符的占比，對大家的學(xué)習(xí)或者工作具有一定的參考學(xué)習(xí)價(jià)值，需要的朋友們下面隨著小編來一起學(xué)習(xí)學(xué)習(xí)吧2023-08-08
Python處理時(shí)間戳和時(shí)間計(jì)算等的腳本分享
這篇文章主要為大家整理總結(jié)了5個(gè)實(shí)用的Python小，可以實(shí)現(xiàn)時(shí)間戳處理和時(shí)間計(jì)算。文中的示例代碼講解詳細(xì)，感興趣的小伙伴可以學(xué)習(xí)一下2022-07-07
今天小編就為大家分享一篇Python 腳本的三種執(zhí)行方式小結(jié)，具有很好的參考價(jià)值，希望對大家有所幫助。一起跟隨小編過來看看吧2019-12-12
一個(gè)基于flask的web應(yīng)用誕生 bootstrap框架美化（3）
一個(gè)基于flask的web應(yīng)用誕生第三篇，這篇文章主要介紹了前端框架bootstrap與flask框架進(jìn)行整合，具有一定的參考價(jià)值，感興趣的小伙伴們可以參考一下2017-04-04