Scrapy框架實現(xiàn)的登錄網(wǎng)站操作示例

更新時間：2020年02月06日 09:00:26 作者：hankleo

這篇文章主要介紹了Scrapy框架實現(xiàn)的登錄網(wǎng)站操作,結(jié)合實例形式分析了Scrapy登錄網(wǎng)站cookies方式、post請求方式登錄網(wǎng)站相關(guān)實現(xiàn)技巧,需要的朋友可以參考下

本文實例講述了Scrapy框架實現(xiàn)的登錄網(wǎng)站操作。分享給大家供大家參考，具體如下：

一、使用cookies登錄網(wǎng)站

import scrapy
class LoginSpider(scrapy.Spider):
  name = 'login'
  allowed_domains = ['xxx.com']
  start_urls = ['https://www.xxx.com/xx/']
  cookies = ""
  def start_requests(self):
    for url in self.start_urls:
      yield scrapy.Request(url, cookies=self.cookies, callback=self.parse)
  def parse(self, response):
    with open("01login.html", "wb") as f:
      f.write(response.body)

二、發(fā)送post請求登錄, 要手動解析網(wǎng)頁獲取登錄參數(shù)

import scrapy
class LoginSpider(scrapy.Spider):
  name='login_code'
  allowed_domains = ['xxx.com']
  #1. 登錄頁面
  start_urls = ['https://www.xxx.com/login/']
  def parse(self, response):
    #2. 代碼登錄
    login_url='https://www.xxx.com/login'
    formdata={
      "username":"xxx",
      "pwd":"xxx",
      "formhash":response.xpath("http://input[@id='formhash']/@value").extract_first(),
      "backurl":response.xpath("http://input[@id='backurl']/@value").extract_first()
    }
    #3. 發(fā)送登錄請求post
    yield scrapy.FormRequest(login_url, formdata=formdata, callback=self.parse_login)
  def parse_login(self, response):
    #4.訪問目標(biāo)頁面
    member_url="https://www.xxx.com/member"
    yield scrapy.Request(member_url, callback=self.parse_member)
  def parse_member(self, response):
    with open("02login.html",'wb') as f:
      f.write(response.body)

三、發(fā)送post請求登錄, 自動解析網(wǎng)頁獲取登錄參數(shù)

import scrapy
class LoginSpider(scrapy.Spider):
  name='login_code2'
  allowed_domains = ['xxx.com']
  #1. 登錄頁面
  start_urls = ['https://www.xxx.com/login/']
  def parse(self, response):
    #2. 代碼登錄
    login_url='https://www.xxx.com/login'
    formdata={
      "username":"xxx",
      "pwd":"xxx"
    }
    #3. 發(fā)送登錄請求post
    yield scrapy.FormRequest.from_response(
      response,
      formxpath="http://*[@id='login_pc']",
      formdata=formdata,
      method="POST", #覆蓋之前的get請求
      callback=self.parse_login
    )
  def parse_login(self, response):
    #4.訪問目標(biāo)頁面
    member_url="https://www.xxx.com/member"
    yield scrapy.Request(member_url, callback=self.parse_member)
  def parse_member(self, response):
    with open("03login.html",'wb') as f:
      f.write(response.body)

希望本文所述對大家基于Scrapy框架的Python程序設(shè)計有所幫助。

您可能感興趣的文章: