Scrapy不拿起我的網址

我有一個scrapy代碼，應該是能夠在網絡中採取電話和地址從表：Scrapy不拿起我的網址

import scrapy 

class PeopleSpider(scrapy.Spider): 
name="People" 
start_urls=[ 
    'http://canada411.yellowpages.ca/search/si/1/519-896-7080/', 
] 
def parse(self,response): 
    for people in response.css("div.person-search__table--row"): 
    yield { 
    'Name': people.css('div.person-search__table--name::text').extract_first(), 
    'Phone Number': people.css('div.person-search__table--phoneNumber::text').extract_first(), 
    'Street': people.css('div.person-search__table--name::street').extract_first(), 
    'City': people.css('div.person-search__table--city::text').extract_first(), 
    'Province': people.css('div.person-search__table--province::text').extract_first(), 
    'Postal Code': people.css('div.person-search__table--postalCode::text').extract_first(), 
    }

但我不斷收到0抓取網頁...

scrapy runspider get.py -o people.json 
2017-02-15 20:14:26 [scrapy.utils.log] INFO: Scrapy 1.3.2 started (bot: scrapybot) 
2017-02-15 20:14:26 [scrapy.utils.log] INFO: Overridden settings: {'FEED_FORMAT': 'json', 'FEED_URI': 'people.json'} 
2017-02-15 20:14:26 [scrapy.middleware] INFO: Enabled extensions: 
['scrapy.extensions.feedexport.FeedExporter', 
'scrapy.extensions.logstats.LogStats', 
'scrapy.extensions.telnet.TelnetConsole', 
'scrapy.extensions.corestats.CoreStats'] 
2017-02-15 20:14:26 [scrapy.middleware] INFO: Enabled downloader middlewares: 
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 
'scrapy.downloadermiddlewares.retry.RetryMiddleware', 
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 
'scrapy.downloadermiddlewares.stats.DownloaderStats'] 
2017-02-15 20:14:26 [scrapy.middleware] INFO: Enabled spider middlewares: 
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 
'scrapy.spidermiddlewares.referer.RefererMiddleware', 
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 
'scrapy.spidermiddlewares.depth.DepthMiddleware'] 
2017-02-15 20:14:26 [scrapy.middleware] INFO: Enabled item pipelines: 
[] 
2017-02-15 20:14:26 [scrapy.core.engine] INFO: Spider opened 
2017-02-15 20:14:26 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 
2017-02-15 20:14:26 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023 
2017-02-15 20:14:27 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET http://99.227.194.212/> from <GET http://canada411.yellowpages.ca/search/si/1/519-896-7080/> 
2017-02-15 20:14:27 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET http://99.227.194.212/login.html> from <GET http://99.227.194.212/> 
2017-02-15 20:14:27 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://99.227.194.212/login.html> (referer: None) 
2017-02-15 20:14:27 [scrapy.core.engine] INFO: Closing spider (finished) 
2017-02-15 20:14:27 [scrapy.statscollectors] INFO: Dumping Scrapy stats: 
{'downloader/request_bytes': 681, 
'downloader/request_count': 3, 
'downloader/request_method_count/GET': 3, 
'downloader/response_bytes': 7931, 
'downloader/response_count': 3, 
'downloader/response_status_count/200': 1, 
'downloader/response_status_count/302': 2, 
'finish_reason': 'finished', 
'finish_time': datetime.datetime(2017, 2, 16, 1, 14, 27, 273208), 
'log_count/DEBUG': 4, 
'log_count/INFO': 7, 
'response_received_count': 1, 
'scheduler/dequeued': 3, 
'scheduler/dequeued/memory': 3, 
'scheduler/enqueued': 3, 
'scheduler/enqueued/memory': 3, 
'start_time': datetime.datetime(2017, 2, 16, 1, 14, 26, 788223)} 
2017-02-15 20:14:27 [scrapy.core.engine] INFO: Spider closed (finished)

我的代碼有問題嗎？或者是網址無法解析？

來源

2017-02-16 Maciek Semik

當您嘗試抓取網址時，您會收到Redirecting (302)消息，這意味着您將被從您嘗試抓取的網頁發送出去。您需要添加一個元值來停止重定向;有關潛在解決方案，請參閱the answer here。

來源

2017-02-16 02:09:02 FiscalTwinkie

你正在刮Yellopages網站，它不容易刮他們。

我確定他們正在重定向到驗證碼頁面。我過去曾經刮過那個網站。

您可以嘗試使用這段代碼來查看重定向到的頁面。

from scrapy.utils.response import open_in_browser 

def parse_details(self, response): 
     open_in_browser(response)

它將在瀏覽器中打開刮URL（如果你在Ubuntu上運行，Windows或Mac此scrapy項目）

來源

2017-02-16 10:06:03 Umair

Scrapy不拿起我的網址

回答

相關問題