scrapy在上傳到db時拋出一個錯誤

我得到的錯誤消息是：exceptions.TypeError：'NoneType'對象沒有屬性'getitem'。這裏是我的代碼：scrapy在上傳到db時拋出一個錯誤

# -*- coding: utf-8 -*- 
import scrapy,MySQLdb,codecs 
from scrapy import Request 

class MyItems(scrapy.Item): 
    topLinks = scrapy.Field() #links at top of home page to grab artist page 
    artists = scrapy.Field() # list of all the artists 
    artists_urls = scrapy.Field() #url of the page for the artist + songs 
    song_name = scrapy.Field() #name of each song 
    song_duration = scrapy.Field() #duration of the song, duh 
    song_dl = scrapy.Field() #dl link for the proxy site for songs 
    rd = scrapy.Field() 


class UpdaterSpider(scrapy.Spider): 
    name = "updater" 
    allowed_domains = [ 
     'myfreemp3.cc', 'unref.eu', 'safeurl.eu' 
    ] 
    start_urls = (
     'http://www.example.com/', 
    ) 

    def __init__(self, *a, **kw): 
     super(UpdaterSpider, self).__init__(*a, **kw) 

     self.item = MyItems() 
     self.db = MySQLdb.connect(#setup my SQL database info 
      "127.0.0.1", #host 
      "root", #user 
      "uconn3", #password 
      "Music" #database 
     ) 

     self.cursor = self.db.cursor() #create a cursor to execute SQL statements 

     self.item = MyItems() 
     self.y = 0 

    def parse(self, response): 
     sql = "SELECT * FROM Songs" 

     self.cursor.execute(sql) 

     result = self.cursor.fetchone() 

     while result is not None: 
      yield Request(url=result[3], callback= lambda r: self.parse_dl(r, result[1], result[2])) 

      result = self.cursor.fetchone() 

    def parse_dl(self, response, songname, duration): 
     self.item['rd'] = response.xpath("//head/script/text()").extract() 

     for x in range(len(self.item['rd'])): 
      self.item['rd'][x] = self.item['rd'][x].split('"')[1] 


      sql = "INSERT INTO Songs (" \ 
       "Artist, songName, Duration, URL, Genre) " \ 
       "VALUES ('Placeholder', '%s', '%s', '%s', 'Placeholder')" % \ 
       (songname.decode('utf-8'), duration.decode('utf-8'), self.item['rd'][x]) 

      self.cursor.execute(sql) 

      self.db.commit()

當我刪除的sql語句行的錯誤是固定的，但我想不通這是爲什麼事情搞亂了。

EDIT1：回溯

Traceback (most recent call last): 
     File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1201, in mainLoop 
     self.runUntilCurrent() 
     File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 824, in runUntilCurrent 
     call.func(*call.args, **call.kw) 
     File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 382, in callback 
     self._startRunCallbacks(result) 
     File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 490, in _startRunCallbacks 
     self._runCallbacks() 
    --- <exception caught here> --- 
     File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in _runCallbacks 
     current.result = callback(current.result, *args, **kw) 
     File "/home/user/PycharmProjects/untitled/mp3_scraper/mp3_scraper/spiders/updater.py", line 48, in <lambda> 
     yield Request(url=result[3], callback= lambda r: self.parse_dl(r, result[1], result[2])) 
    exceptions.TypeError: 'NoneType' object has no attribute '__getitem__'

來源

2014-10-03 johnc31

當你從一個循環返回lambda時，你並沒有捕獲到你認爲你所做的事情，請參閱http://stackoverflow.com/questions/7514093/lambda-function-dont-closure-the-p arameter-in-python或http://stackoverflow.com/questions/2295290/what-do-lambda-function-closures-capture-in-python。 – immerrr 2014-10-03 05:02:14

ahhh yep，得使用scrapy提供的詞典元。 – johnc31 2014-10-03 05:22:48

它看起來像

self.item = MyItems()

沒有它由括號[] 我想調用的方法的GetItem - 問題是不是與數據庫，請提供全部回溯

來源

2014-10-03 04:37:12

k上傳了traceback – johnc31 2014-10-03 04:40:17

對不起 - 我沒有機會在env中運行此代碼。我可以說的唯一一件事 - 問題不在數據庫中，它也在大括號中調用也是 - 在異步程序中使用阻塞數據庫調用是不好的做法 - 使用tx-mongo + mongo可以節省更多時間 – 2014-10-03 04:55:03

scrapy在上傳到db時拋出一個錯誤

回答

相關問題