通過scrapy登錄該站點

我在網站上進行官方視頻授課。如果用戶名和密碼不正確，那麼過渡到回調方法是成功的，如果登錄名和密碼是正確的，那麼轉換到方法是不可行的。我的代碼：進口scrapy通過scrapy登錄該站點

class QuotesSpider(scrapy.Spider): 
    name = "quotes" 
    start_urls = ["https://www.darkorbit.com"] 

    def parse(self, response): 
     login_url = response.css('form[name="bgcdw_login_form"]::attr(action)').extract_first() 
     data = { 
      'username': 'testscrapy', 
      'password': 'testtest', 
     } 
     yield scrapy.FormRequest(url=login_url, formdata=data, callback=self.after_login) 

    def after_login(self, response): 
     print('----------------------------------------')

有了正確的輸入數據，可以得到一個日誌（長片段切）：

2017-06-03 22:04:40 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.darkorbit.com/robots.txt> (referer: None) 
2017-06-03 22:04:41 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.darkorbit.com> (referer: None) 
2017-06-03 22:04:42 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://auth3.bpsecure.com/robots.txt> (referer: None) 
2017-06-03 22:04:42 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://www.darkorbit.com/ProjectAp........> 
2017-06-03 22:04:42 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://ru4.darkorbit.com/Pro..........> 
2017-06-03 22:04:43 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://ru4.darkorbit.com/robots.txt> (referer: None) 
2017-06-03 22:04:43 [scrapy.downloadermiddlewares.robotstxt] DEBUG: Forbidden by robots.txt: <GET https://ru4.darkorbit.com/Pro......>

來源

2017-06-03 asurovenko

ROBOTSTXT_OBEY要設置的變量= FALSE，它幫助我！謝謝！ – asurovenko

從這一行你的日誌：

2017-06-03 22:04:43 [scrapy.downloadermiddlewares.robotstxt] DEBUG: Forbidden by robots.txt: <GET https://ru4.darkorbit.com/Pro......>

我可以告訴你，你需要改變你的設置settings.py文件。

ROBOTSTXT_OBEY需要在False

ROBOTSTXT_OBEY=False

來源

2017-06-03 18:12:17

通過scrapy登錄該站點

回答

相關問題