Scrapy：連接被拒絕

Question

當試圖測試scrapy安裝我收到一個錯誤：

$ scrapy shell http://www.google.es 
j2011-02-16 10:54:46+0100 [scrapy] INFO: Scrapy 0.12.0.2536 started (bot: scrapybot) 
2011-02-16 10:54:46+0100 [scrapy] DEBUG: Enabled extensions: TelnetConsole, SpiderContext, WebService, CoreStats, MemoryUsage, CloseSpider 
2011-02-16 10:54:46+0100 [scrapy] DEBUG: Enabled scheduler middlewares: DuplicatesFilterMiddleware 
2011-02-16 10:54:46+0100 [scrapy] DEBUG: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, RedirectMiddleware, CookiesMiddleware, HttpProxyMiddleware, HttpCompressionMiddleware, DownloaderStats 
2011-02-16 10:54:46+0100 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware 
2011-02-16 10:54:46+0100 [scrapy] DEBUG: Enabled item pipelines: 
2011-02-16 10:54:46+0100 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6023 
2011-02-16 10:54:46+0100 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080 
2011-02-16 10:54:46+0100 [default] INFO: Spider opened 
2011-02-16 10:54:47+0100 [default] DEBUG: Retrying <GET http://www.google.es> (failed 1 times): Connection was refused by other side: 111: Connection refused. 
2011-02-16 10:54:47+0100 [default] DEBUG: Retrying <GET http://www.google.es> (failed 2 times): Connection was refused by other side: 111: Connection refused. 
2011-02-16 10:54:47+0100 [default] DEBUG: Discarding <GET http://www.google.es> (failed 3 times): Connection was refused by other side: 111: Connection refused. 
2011-02-16 10:54:47+0100 [default] ERROR: Error downloading <http://www.google.es>: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionRefusedError'>: Connection was refused by other side: 111: Connection refused. 
    ] 
2011-02-16 10:54:47+0100 [scrapy] ERROR: Shell error 
    Traceback (most recent call last): 
    Failure: scrapy.exceptions.IgnoreRequest: Connection was refused by other side: 111: Connection refused. 

2011-02-16 10:54:47+0100 [default] INFO: Closing spider (shutdown) 
2011-02-16 10:54:47+0100 [default] INFO: Spider closed (shutdown) 

版本：

• Scrapy 0.12.0.2536
• 的Python 2.6.6
• OS ：Ubuntu 10.10

編輯：我可以達到它用我的瀏覽器，wget，telnet google.es 80，它發生在所有的網站上。

Answer 1

適用於與我相同的Ubuntu，相同的Python，相同的scrapy。你有沒有多次打google.es？你可以從你的瀏覽器，並捲曲？也許它只是想從蜘蛛的休息？

Answer 2

任務1： Scrapy將發送帶有'bot'的usergent。網站也可能基於用戶代理進行阻止。

嘗試通過騎在USER_AGENT settings.py

例如：USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64; rv:7.0.1) Gecko/20100101 Firefox/7.7'

團2： 先給請求之間的延遲，欺騙是人正在發送該請求。

DOWNLOAD_DELAY = 0.25 

任務3： 如果實在不行，安裝Wireshark的，看看在請求頭（或）職位數據的差異 而scrapy發送，當你的瀏覽器發送。

Answer 3

可能是您的網絡連接存在問題。

首先，檢查您的互聯網連接。

如果你是通過代理服務器接入網絡，您應該一段代碼到您的scrapy項目（http://doc.scrapy.org/en/latest/topics/downloader-middleware.html#scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware）

無論如何，嘗試升級scrapy版本。