的Python /扭曲 - 如何打印更詳細的錯誤信息

我運行此代碼：的Python /扭曲 - 如何打印更詳細的錯誤信息

import argparse 

from tqdm import tqdm 
from sys import argv 
from pprint import pformat 

from twisted.internet.task import react 
from twisted.web.client import Agent, readBody 
from twisted.web.http_headers import Headers 
from twisted.internet.task import cooperate 
from twisted.internet.defer import gatherResults 

import sys 
from twisted.python import log 

log.startLogging(sys.stdout) 

import lxml.html 

from geoip import geolite2 
import pycountry 

from tld import get_tld 
import json 
import socket 

poweredby = "" 
server = "" 
ip = "" 

f = open("errors.txt", "w") 


def error(response, url): 
    f.write("Error: "+url+"\n") 


def cbRequest(response, url): 
    global poweredby, server, ip 
    # print 'Response version:', response.version 
    # print 'Response code:', response.code 
    # print 'Response phrase:', response.phrase 
    # print 'Response headers:' 
    # print pformat(list(response.headers.getAllRawHeaders())) 
    poweredby = response.headers.getRawHeaders("X-Powered-By")[0] 
    server = response.headers.getRawHeaders("Server")[0] 

    #print poweredby 
    #print server 

    d = readBody(response) 
    d.addCallback(cbBody, url) 
    return d 


def cbBody(body, ourl): 
    global poweredby, server,ip 

    #print body 
    html_element = lxml.html.fromstring(body) 
    generator = html_element.xpath("//meta[@name='generator']/@content") 

    ip = socket.gethostbyname(ourl) 

    try: 
     match = geolite2.lookup(ip) 
     if match is not None: 
      country = match.country 
      try: 

       c = pycountry.countries.lookup(country) 
       country = c.name 
      except: 
       country = "" 

    except: 
     country = "" 
    try: 
     res = get_tld("http://www" + ourl, as_object=True) 
     tld = res.suffix 
    except: 
     tld = "" 

    try: 
     match = re.search(r'[\w\.-][email protected][\w\.-]+', body) 
     email = match.group(0) 
    except: 
     email = "" 

    permalink=ourl.rstrip().replace(".","-") 

    try: 
     item = generator[0] 
     val = "{ \"Domain\":" + json.dumps(
      "http://" + ourl.rstrip()) + ",\"IP\":\"" + ip + "\",\"Server\":" + json.dumps(
      str(server)) + ",\"PoweredBy\":" + json.dumps(
       str(poweredby)) + ",\"MetaGenerator\":" + json.dumps(item) + ",\"Email\":" + json.dumps(
        email) + ",\"Suffix\":\"" + tld + "\",\"CountryHosted\":\"" + country+"\",\"permalink\":\""+permalink+"\" }" 
    except: 
     val = "{ \"Domain\":" + json.dumps(
      "http://" + ourl.rstrip()) + ",\"IP\":\"" + ip + "\"," + "\"Server\":" + json.dumps(
      str(server)) + ",\"PoweredBy\":" + json.dumps(
       str(poweredby)) + ",\"MetaGenerator\":\"\",\"Email\":" + json.dumps(
        email) + ",\"Suffix\":\"" + tld + "\",\"CountryHosted\":\"" + country+"\",\"permalink\":\""+permalink+"\" }" 


    print val 


def main(reactor, url_path): 
    urls = open(url_path) 
    return mainjob(reactor, (url.strip() for url in urls)) 

def mainjob(reactor, urls=argv[2:]): 
    #for url in urls: 
    # print url 
    agent = Agent(reactor) 
    work = (process(agent, url) for url in tqdm(urls)) 
    tasks = list(cooperate(work) for i in range(100)) 
    return gatherResults(list(task.whenDone() for task in tasks)) 



def process(agent, url): 
    d = agent.request(
     'GET', "http://" + url, 
     Headers({'User-Agent': ['crawler']}), 
     None) 
    d.debug=1 
    d.addCallback(cbRequest, url) 
    d.addErrback(error, url) 
    return d 

react(main, ["./test.txt"]) 

f.close()

我啓用了調試並以某種方式對那些2項addErrback總是（4開出4次）調用，因此錯誤條目：

[email protected]:~/crawler$ python scanner.py 
2017-02-25 20:35:36+0100 [-] Log opened. 
0it [00:00, ?it/s]2017-02-25 20:35:36+0100 [-] Starting factory <twisted.web.client._HTTP11ClientFactory instance at 0x7fffefac1248> 
2017-02-25 20:35:36+0100 [-] Starting factory <twisted.web.client._HTTP11ClientFactory instance at 0x7fffefac16c8> 
2it [00:00, 660.31it/s] 
2017-02-25 20:35:37+0100 [-] Stopping factory <twisted.web.client._HTTP11ClientFactory instance at 0x7fffefac1248> 
2017-02-25 20:35:37+0100 [-] Stopping factory <twisted.web.client._HTTP11ClientFactory instance at 0x7fffefac16c8> 
2017-02-25 20:35:37+0100 [-] Main loop terminated. 
[email protected]:~/crawler$ cat errors.txt 
Error: google.al 
Error: fau.edu.al

我很困惑，因爲在捕獲時，我看到這些請求被髮送出去並收到響應。

如何打印在addErrback準確的錯誤原因（），以瞭解行爲的更多？

這涉及到這樣一個問題：

Twisted/Python - processing a large file line by line

我感謝所有幫助。新的Python和扭曲。

感謝，

更新1：

我修改了錯誤的功能是：

def error(failure, url): 
    f.write("Error: "+url+"\n") 
    print type(failure.value), failure # catch error here 
    print failure.value.reasons[0].printTraceback()

這裏是輸出，似乎是在每次運行相同的：

[email protected]:~/crawler$ python scanner.py 
2017-02-25 21:24:48+0100 [-] Log opened. 
0it [00:00, ?it/s]2017-02-25 21:24:48+0100 [-] Starting factory <twisted.web.client._HTTP11ClientFactory instance at 0x7fffefac1200> 
2017-02-25 21:24:48+0100 [-] Starting factory <twisted.web.client._HTTP11ClientFactory instance at 0x7fffefac1680> 
2it [00:00, 788.33it/s] 
2017-02-25 21:24:48+0100 [-] <type 'exceptions.TypeError'> [Failure instance: Traceback: <type 'exceptions.TypeError'>: 'NoneType' object has no attribute '__getitem__' 
2017-02-25 21:24:48+0100 [-] /usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py:565:_startRunCallbacks 
2017-02-25 21:24:48+0100 [-] /usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py:651:_runCallbacks 
2017-02-25 21:24:48+0100 [-] /usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py:457:callback 
2017-02-25 21:24:48+0100 [-] /usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py:565:_startRunCallbacks 
2017-02-25 21:24:48+0100 [-] --- <exception caught here> --- 
2017-02-25 21:24:48+0100 [-] /usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py:651:_runCallbacks 
2017-02-25 21:24:48+0100 [-] scanner.py:47:cbRequest 
2017-02-25 21:24:48+0100 [-] ] 
2017-02-25 21:24:48+0100 [HTTP11ClientProtocol,client] main function encountered error 
    Traceback (most recent call last): 
    Failure: twisted.internet.defer.FirstError: FirstError[#0, [Failure instance: Traceback: <type 'exceptions.AttributeError'>: 'exceptions.TypeError' object has no attribute 'reasons' 
    /usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py:565:_startRunCallbacks 
    /usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py:651:_runCallbacks 
    /usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py:457:callback 
    /usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py:565:_startRunCallbacks 
    --- <exception caught here> --- 
    /usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py:651:_runCallbacks 
    scanner.py:37:error 
    ]] 

2017-02-25 21:24:48+0100 [-] <class 'twisted.internet.error.ConnectError'> [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectError'>: An error occurred while connecting: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion: Connection lost. 
2017-02-25 21:24:48+0100 [-] ]. 
2017-02-25 21:24:48+0100 [-] ] 
2017-02-25 21:24:48+0100 [-] Stopping factory <twisted.web.client._HTTP11ClientFactory instance at 0x7fffefac1680> 
2017-02-25 21:24:48+0100 [-] Stopping factory <twisted.web.client._HTTP11ClientFactory instance at 0x7fffefac1200> 
2017-02-25 21:24:48+0100 [-] Main loop terminated. 
2017-02-25 21:24:48+0100 [-] Unhandled error in Deferred: 
2017-02-25 21:24:48+0100 [-] Unhandled Error 
    Traceback (most recent call last): 
     File "/usr/local/lib/python2.7/dist-packages/twisted/internet/base.py", line 1084, in connectionFailed 
     self.factory.clientConnectionFailed(self, reason) 
     File "/usr/local/lib/python2.7/dist-packages/twisted/internet/endpoints.py", line 246, in clientConnectionFailed 
     self._onConnection.errback(reason) 
     File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 498, in errback 
     self._startRunCallbacks(fail) 
     File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 565, in _startRunCallbacks 
     self._runCallbacks() 
    --- <exception caught here> --- 
     File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 651, in _runCallbacks 
     current.result = callback(current.result, *args, **kw) 
     File "scanner.py", line 37, in error 
     print failure.value.reasons[0].printTraceback() 
    exceptions.AttributeError: 'ConnectError' object has no attribute 'reasons' 

2017-02-25 21:24:48+0100 [-] Unhandled error in Deferred: 
2017-02-25 21:24:48+0100 [-] Unhandled Error 
    Traceback (most recent call last): 
     File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 565, in _startRunCallbacks 
     self._runCallbacks() 
     File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 651, in _runCallbacks 
     current.result = callback(current.result, *args, **kw) 
     File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 457, in callback 
     self._startRunCallbacks(result) 
     File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 565, in _startRunCallbacks 
     self._runCallbacks() 
    --- <exception caught here> --- 
     File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 651, in _runCallbacks 
     current.result = callback(current.result, *args, **kw) 
     File "scanner.py", line 37, in error 
     print failure.value.reasons[0].printTraceback() 
    exceptions.AttributeError: 'exceptions.TypeError' object has no attribute 'reasons'

更新2：

2017-02-27 17:46:57+0100 [-] Log opened. 0it [00:00, ?it/s]2017-02-27 17:46:57+0100 [-] Starting factory <twisted.web.client._HTTP11ClientFactory instance at 0x7fffefac0320> 2017-02-27 17:46:57+0100 [-] Starting factory <twisted.web.client._HTTP11ClientFactory instance at 0x7fffefac07a0> 2it [00:00, 763.36it/s] 2017-02-27 17:46:57+0100 [-] Traceback (most recent call last): 2017-02-27 17:46:57+0100 [-] File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 565, in _startRunCallbacks 2017-02-27 17:46:57+0100 [-] self._runCallbacks() 2017-02-27 17:46:57+0100 [-] File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 651, in _runCallbacks 2017-02-27 17:46:57+0100 [-] current.result = callback(current.result, *args, **kw) 2017-02-27 17:46:57+0100 [-] File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 457, in callback 2017-02-27 17:46:57+0100 [-] self._startRunCallbacks(result) 2017-02-27 17:46:57+0100 [-] File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 565, in _startRunCallbacks 2017-02-27 17:46:57+0100 [-] self._runCallbacks() 2017-02-27 17:46:57+0100 [-] --- <exception caught here> --- 2017-02-27 17:46:57+0100 [-] File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 651, in _runCallbacks 2017-02-27 17:46:57+0100 [-] current.result = callback(current.result, *args, **kw) 2017-02-27 17:46:57+0100 [-] File "scan.py", line 59, in cbRequest 2017-02-27 17:46:57+0100 [-] poweredby = response.headers.getRawHeaders("X-Powered-By")[0] 2017-02-27 17:46:57+0100 [-] exceptions.TypeError: 'NoneType' object has no attribute '__getitem__' 2017-02-27 17:46:57+0100 [-] Stopping factory <twisted.web.client._HTTP11ClientFactory instance at 0x7fffefac0320> 2017-02-27 17:46:59+0100 [-] Traceback (most recent call last): 2017-02-27 17:46:59+0100 [-] File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 565, in _startRunCallbacks 2017-02-27 17:46:59+0100 [-] self._runCallbacks() 2017-02-27 17:46:59+0100 [-] File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 651, in _runCallbacks 2017-02-27 17:46:59+0100 [-] current.result = callback(current.result, *args, **kw) 2017-02-27 17:46:59+0100 [-] File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 457, in callback 2017-02-27 17:46:59+0100 [-] self._startRunCallbacks(result) 2017-02-27 17:46:59+0100 [-] File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 565, in _startRunCallbacks 2017-02-27 17:46:59+0100 [-] self._runCallbacks() 2017-02-27 17:46:59+0100 [-] --- <exception caught here> --- 2017-02-27 17:46:59+0100 [-] File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 651, in _runCallbacks 2017-02-27 17:46:59+0100 [-] current.result = callback(current.result, *args, **kw) 2017-02-27 17:46:59+0100 [-] File "scan.py", line 59, in cbRequest 2017-02-27 17:46:59+0100 [-] poweredby = response.headers.getRawHeaders("X-Powered-By")[0] 2017-02-27 17:46:59+0100 [-] exceptions.TypeError: 'NoneType' object has no attribute '__getitem__' 2017-02-27 17:46:59+0100 [-] Stopping factory <twisted.web.client._HTTP11ClientFactory instance at 0x7fffefac07a0> 2017-02-27 17:46:59+0100 [-] Main loop terminated.

正如你所看到的，投訴響應是NoneType：

修改由@讓 - 保羅·Calderone的

我得到這個輸出給出的誤差函數示例後對象，所以沒有真正的原因是有提及。

然而，當我運行老版本的誤差函數，它打印什麼，你可以在更新1看：

<class 'twisted.internet.error.ConnectError'> [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectError'>: An error occurred while connecting: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion: Connection lost.

爲什麼不能在第一個（答案通過@讓 - 保羅建議打印Calderone的）版本？任何爲什麼得到這個錯誤有什麼看法？

我驗證了它的工作原理，當我連接到這些領域，也可通過爬蟲（也用Python）的線程版本的作品，你可以看到它也對線（Wireshark的截圖）。看起來Twisted 不是看到它（認爲它是丟失的連接？）

來源

2017-02-25 android_dev

您的error函數中存在一個錯誤。回溯告訴你：

File "scanner.py", line 37, in error 
    print failure.value.reasons[0].printTraceback() 
exceptions.AttributeError: 'exceptions.TypeError' object has no attribute 'reasons'

掃描儀。py，第37行，在函數error中，行print failure.value.reasons[0].printTraceback()激發AttributeError，因爲TypeError實例沒有reasons屬性。

我認爲您正在查找的reasons屬性屬於RequestGenerationFailed，RequestTransmissionFailed或ResponseFailed。

因此，定義error這樣，而不是：

from twisted.web._newclient import (
    RequestGenerationFailed, 
    RequestTransmissionFailed, 
    ResponseFailed, 
) 

def error(failure, url): 
    f.write("Error: "+url+"\n") 
    if failure.check(
     RequestGenerationFailed, 
     RequestTransmissionFailed, 
     ResponseFailed, 
    ): 
     failure.value.reasons[0].printTraceback() 
    else: 
     failure.printTraceback()

如果這被證明是有用的，文件對雙絞線一票，使這些異常類型公共（因爲沒有保證_newclient進口將繼續在未來工作）。

來源

2017-02-27 14:19:53

謝謝！我根據你的回答結果更新了問題（更新2 :)。 –

的Python /扭曲 - 如何打印更詳細的錯誤信息

回答

相關問題