Scrapy RSS刮板

Question

我試圖刮掉從雅虎的RSS Feed（其開放的公司RSS訂閱| https://developer.yahoo.com/finance/company.html）

我想湊以下網址：https://feeds.finance.yahoo.com/rss/2.0/headline?s=BPMX

出於某種原因，我的蜘蛛ISN」我認爲它可能與生成的XPath有關，如果不是，定義parse_item可能會有一些問題。

import scrapy 
from scrapy.spiders import CrawlSpider 
from YahooScrape.items import YahooScrapeItem 

class Spider(CrawlSpider): 
    name= "YahooScrape" 
    allowed_domains = ["yahoo.com"] 
    start_urls = ('https://feeds.finance.yahoo.com/rss/2.0/headline?s=BPMX',) 

    def parse_item(self, response): 
     self.logger.info('Hi, this is an item page! %s', response.url) 
     item = EmperyscraperItem() 
     item['title'] = response.xpath('//*[@id="collapsible"]/div[1]/div[2]/span',).extract()    #define XPath for title 
     item['link'] = response.xpath('//*[@id="collapsible"]/div[1]/div[2]/span',).extract()     #define XPath for link 
     item['description'] = response.xpath('//*[@id="collapsible"]/div[1]/div[2]/span',).extract()   #define XPath for description 
     return item 

代碼有什麼問題？如果不是，那麼正確的XPath方向是提取標題，desc和鏈接。我是Scrapy的新手，只需要一些幫助就可以搞定！

編輯：我已經更新了我的蜘蛛並把它轉換成一個XMLFeedSpider如下圖所示：

import scrapy 

from scrapy.spiders import XMLFeedSpider 
from YahooScrape.items import YahooScrapeItem 

class Spider(XMLFeedSpider): 
    name = "YahooScrape" 
    allowed_domains = ["yahoo.com"] 
    start_urls = ('https://feeds.finance.yahoo.com/rss/2.0/headline?s=BPMX') #Crawl BPMX 
    itertag = 'item' 

    def parse_node(self, response, node): 
     self.logger.info('Hi, this is a <%s> node!: %s', self.itertag, ''.join(node.extract())) 

     item = YahooScrapeItem() 
     item['title'] = node.xpath('item/title/text()',).extract()    #define XPath for title 
     item['link'] = node.xpath('item/link/text()').extract() 
     item['pubDate'] = node.xpath('item/link/pubDate/text()').extract() 
     item['description'] = node.xpath('item/category/text()').extract()    #define XPath for description 
     return item 

#Yahoo RSS feeds http://finance.yahoo.com/rss/headline?s=BPMX,APPL 

現在我發現了以下錯誤：

2017-06-13 11:25:57 [scrapy.core.engine] ERROR: Error while obtaining start requests 

知道爲什麼錯誤發生了？我的HTML路徑看起來正確。

Answer 1

從我所看到的，CrawlSpider only works for HTML responses。所以我建議你建立一個更簡單的scrapy.Spider，或更專業的XMLFeedSpider。

然後，您在parse_items中使用的XPath似乎是從您的瀏覽器以HTML/RSS提供的HTML形式構建的。 Feed中沒有*[@id="collapsible"]或<div> s。

看view-source:https://feeds.finance.yahoo.com/rss/2.0/headline?s=BPMX代替：

<?xml version="1.0" encoding="UTF-8" standalone="yes"?> 
<rss version="2.0"> 
    <channel> 
     <copyright>Copyright (c) 2017 Yahoo! Inc. All rights reserved.</copyright> 
     <description>Latest Financial News for BPMX</description> 
     <image> 
      <height>45</height> 
      <link>http://finance.yahoo.com/q/h?s=BPMX</link> 
      <title>Yahoo! Finance: BPMX News</title> 
      <url>http://l.yimg.com/a/i/brand/purplelogo/uh/us/fin.gif</url> 
      <width>144</width> 
     </image> 
     <item> 
      <description>MENLO PARK, Calif., June 7, 2017 /PRNewswire/ -- BioPharmX Corporation (NYSE MKT: BPMX), a specialty pharmaceutical company focusing on dermatology, today announced that it will release its financial results ...</description> 
      <guid isPermaLink="false">f56d5bf8-f278-37fd-9aa5-fe04b2e1fa53</guid> 
      <link>https://finance.yahoo.com/news/biopharmx-report-first-quarter-financial-101500259.html?.tsrc=rss</link> 
      <pubDate>Wed, 07 Jun 2017 10:15:00 +0000</pubDate> 
      <title>BioPharmX to Report First Quarter Financial Results</title> 
     </item> 

---

工作蜘蛛例如：

import scrapy 

from scrapy.spiders import XMLFeedSpider 
#from YahooScrape.items import YahooScrapeItem 

class Spider(XMLFeedSpider): 
    name = "YahooScrape" 
    allowed_domains = ["yahoo.com"] 
    start_urls = ('https://feeds.finance.yahoo.com/rss/2.0/headline?s=BPMX',) #Crawl BPMX 
    itertag = 'item' 

    def parse_node(self, response, node): 
     self.logger.info('Hi, this is a <%s> node!: %s', self.itertag, ''.join(node.extract())) 

     item = {} 
     item['title'] = node.xpath('title/text()',).extract_first()    #define XPath for title 
     item['link'] = node.xpath('link/text()').extract_first() 
     item['pubDate'] = node.xpath('link/pubDate/text()').extract_first() 
     item['description'] = node.xpath('description/text()').extract_first()    #define XPath for description 
     return item