2
我正在嘗試Scrapy for第一次。經過公平的研究後，我得到了基本知識。現在我正在嘗試獲取表格的數據。它不工作。 沒有刮掉任何數據。檢查下面的源代碼。用Scrapy刮掉表格但沒有刮傷項目的數據
settings.py
BOT_NAME = 'car'
SPIDER_MODULES = ['car.spiders']
NEWSPIDER_MODULE ='car.spiders'
DEFAULT_ITEM_CLASS = 'car.items.Car58Item'
ITEM_PIPELINES = {'car.pipelines.JsonLinesItemExporter': 300}
items.py
from scrapy.item import Item,Field
class Car58Item(Item):
# define the fields for your item here like:
# name = scrapy.Field()
url = Field()
tip = Field()
name = Field()
size = Field()
region = Field()
amt = Field()
car_spider.py
# -*- coding=utf-8 -*-
from __future__ import absolute_import
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider,Rule,Spider
from car.items import Car58Item
class CarSpider (CrawlSpider):
name ='car'
allowed_domains = ['58.com']
start_urls = ['http://quanguo.58.com/ershouche']
rules = [Rule(LinkExtractor(allow=('/pn\d+')),'parse_item')] #//頁面讀取策略
def parse_item(self,response):
trs = response.xpath("//div[@id='infolist']/table[@class='tbimg']/tr")[1:-2]
items = []
#for tr in sel.xpath("id('infolist')/table/tr"):
for tr in trs:
item = Car58Item()
item['url'] = tr.xpath("td[@class='img']/a/@href").extract()
item['tip'] = tr.xpath("td[@class='t']/a/font/text()").extract()
item['name'] = tr.xpath("td[@class='t']/a[1]/text()").extract()
item['size'] = tr.xpath("td[@class='t']/p").extract()
item['region'] = tr.xpath("td[@class='tc']/a/text()").extract()
item['amt'] = tr.xpath("td[@class='tc']/b/text()").extract()
items.append(item)
return items
pipelines.py
# -*- coding: utf-8 -*-
import json
import codecs
class JsonLinesItemExporter(object):
def __init__(self):
self.file = codecs.open('car.json','w',encoding='utf-8')
def process_item(self, items, spider):
line = json.dumps(dict(items),ensure_ascii=False) + "\n"
self.file.write(line)
return items
def spider_closed(self,spider):
self.file.close()
我跑scrapy殼
[[email protected]~/PycharmProjects/car] $**scrapy crawl car**
2016-05-18 10:35:36 [scrapy] INFO: Scrapy 1.0.6 started (bot: car)
2016-05-18 10:35:36 [scrapy] INFO: Optional features available: ssl, http11
2016-05-18 10:35:36 [scrapy] INFO: Overridden settings: {'DEFAULT_ITEM_CLASS': 'car.items.Car58Item', 'NEWSPIDER_MODULE': 'car.spiders', 'SPIDER_MODULES': ['car.spiders'], 'BOT_NAME': 'car'}
2016-05-18 10:35:36 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState
2016-05-18 10:35:36 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2016-05-18 10:35:36 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2016-05-18 10:35:36 [scrapy] INFO: Enabled item pipelines: JsonLinesItemExporter
2016-05-18 10:35:36 [scrapy] INFO: Spider opened
2016-05-18 10:35:36 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-05-18 10:35:36 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-05-18 10:35:37 [scrapy] DEBUG: Redirecting (301) to <GET http://quanguo.58.com/ershouche/> from <GET http://quanguo.58.com/ershouche>
2016-05-18 10:35:39 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/> (referer: None)
2016-05-18 10:35:40 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn2/> (referer: http://quanguo.58.com/ershouche/)
2016-05-18 10:35:42 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn7/> (referer: http://quanguo.58.com/ershouche/)
2016-05-18 10:35:42 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn6/> (referer: http://quanguo.58.com/ershouche/)
2016-05-18 10:35:42 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn12/> (referer: http://quanguo.58.com/ershouche/)
2016-05-18 10:35:43 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn11/> (referer: http://quanguo.58.com/ershouche/)
2016-05-18 10:35:44 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn9/> (referer: http://quanguo.58.com/ershouche/)
2016-05-18 10:35:45 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn8/> (referer: http://quanguo.58.com/ershouche/)
2016-05-18 10:35:45 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn5/> (referer: http://quanguo.58.com/ershouche/)
2016-05-18 10:35:46 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn10/> (referer: http://quanguo.58.com/ershouche/)
2016-05-18 10:35:46 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn4/> (referer: http://quanguo.58.com/ershouche/)
2016-05-18 10:35:46 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn3/> (referer: http://quanguo.58.com/ershouche/)
2016-05-18 10:35:47 [scrapy] INFO: Closing spider (finished)
2016-05-18 10:35:47 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 5550,
'downloader/request_count': 13,
'downloader/request_method_count/GET': 13,
'downloader/response_bytes': 339809,
'downloader/response_count': 13,
'downloader/response_status_count/200': 12,
'downloader/response_status_count/301': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2016, 5, 18, 2, 35, 47, 45187),
'log_count/DEBUG': 14,
'log_count/INFO': 7,
'request_depth_max': 1,
'response_received_count': 12,
'scheduler/dequeued': 13,
'scheduler/dequeued/memory': 13,
'scheduler/enqueued': 13,
'scheduler/enqueued/memory': 13,
'start_time': datetime.datetime(2016, 5, 18, 2, 35, 36, 733155)}
2016-05-18 10:35:47 [scrapy] INFO: Spider closed (finished)
但不scrapy任何數據......在car.json
[[email protected]~/PycharmProjects/car] $more car.json
零項目輸出。
謝謝
「parse_item」是否與代碼中一樣縮進？ – alecxe
謝謝，parse_item沒有縮進。現在縮進後就OK了。 – robinma
我已經在scrapy shell中試過了你的xpaths，並且它們返回了合理的輸出結果，所以我認爲你的蜘蛛的身體是可以的。您可以嘗試在for循環中打印項目['url']以確認正在提取數據。我懷疑這與你的物品出口商有關。您是否嘗試過使用JSON導出器構建的scrapy？ – Steve