蟒蛇的urllib2不能得到

我在與獲得此URL的結果頁面與Python的urllib2的一個非常艱難的時間，谷歌網址：蟒蛇的urllib2不能得到

http://www.google.com/search?tbs=sbi:AMhZZitAaz7goe6AsfVSmFw1sbwsmX0uIjeVnzKHjEXMck70H3j32Q-6FApxrhxdSyMo0OedyWkxk3-qYbyf0q1OqNspjLu8DlyNnWVbNjiKGo87QUjQHf2_1idZ1q_1vvm5gzOCMpChYiKsKYdMywOLjJzqmzYoJNOU2UsTs_1zZGWjU-LsjdFXt_1D5bDkuyRK0YbsaLVcx4eEk_1KMkcJpWlfFEfPMutxTLGf1zxD-9DFZDzNOODs0oj2j_1KG8FRCaMFnTzAfTdl7JfgaDf_1t5Vti8FnbeG9i7qt9wF6P-QK9mdvC15hZ5UR29eQdYbcD1e4woaOQCmg8Q1VLVPf4-kf8dAI7p3jM_1MkBBwaxdt_1TsM4FLwh0oHAYKOS5qBRI28Vs0aw5_1C5-WR4dC902Eqm5eAkLiQyAM9J2bioR66g3tMWe-j9Hyh1ID40R1NyXEJDHcGxp7xOn_16XxfW_1Cq5ArdSNzxFvABb1UcXCn5s4_1LpXZxhZbauwaO8cg3CKGLUvl_1wySDB7QIkMIF2ZInEPS4K-eyErVKqOdY9caYUD8X7oOf6sDKFjT7pNHwlkXiuYbKBRYjlvRHPlcPN1WHWCJWdSNyXdZhwDI3VRaKwmi4YNvkryeNMMbhGytfvlNaaelKcOzWbvzCtSNaP2lJziN1x3btcIAplPcoZxEpb0cDlQwId3A5FDhczxpVbdRnOB-Xeq_1AiUTt_1iI6bSgUAinWXQFYWveTOttdSNCgK-VTxV4OCtlrCrZerk27RBLAzT0ol9NOfYmYhiabzhUczWk4NuiVhKN-M4eo76cAsi74PY4V_1lWjvOpI35V_1YLJQrm0fxVcD34wxFYCIllT2gYW09fj3cuBDMNbsaJqPVQ04OOGlwmcmJeAnK96xd_1aMUd6FsVLOSDS7RfS5MNUSyd1jnXvRU_1MF_1Dj8oC8sm7PfVdjm3firiMcaKM28j9kGWbY0heIGLtO_1m6ad-iKfxYEzSux2b5w62LQlP57yS7vX8RFoyKzHA0RrFIEbPBQdNMA3Vpw0G_1LvEjCAPSCV1HH1pDp0l4EnNCvUIAppVXzNMyWT_1gKITj1NLqAn-Z1tH323JwZSc77OftDSreyHJ-BPxn3n7JMkNZFcQx6S7tfBxeqJ1NuDlpax11pw0_1Oi_1nF3vyEP0NbGKSVgNvBv_1tv8ahxvrHn9UnP78FleiOpzUBfdfRPZiT20VEq5-oXtV_1XwIzrd-5_15-cf2yoL7ohyPuv3WKGUGr4YCsYje7_1D8VslqMPsvbwMg9haj3TrBKH7go70ZfPjUv3h1K7lplnnCdV0hrYVQkSLUY1eEor3L--Vu5PlewS60ZH5YEn4qTnDxniV95h8q0Y3RWXJ6gIXitR5y6CofVg

我用下面的標題，而這應該是簡單的我會想：

headers = {'Host':'www.google.com','User-Agent':user_agent,'Accept-Language':'en-us,en;q=0.5','Accept-Encoding':'gzip, deflate','Accept-Charset':'ISO-8859-1,utf-8;q=0.7,*;q=0.7','Connection':'keep-alive','Referer':'http://www.google.co.in/imghp?hl=en&tab=ii','Cookie':'PREF=ID=1d7bc4ff2a5d8bc6:U=1d37ba5a518b9be1:FF=4:LD=en:TM=1300950025:LM=1302071720:S=rkk0IbbhxUIgpTyA; NID=51=uNq6mZ385WlV1UTfXsiWkSgnsa6PdjH4l9ph-vSQRszBHRcKW3VRJclZLd2XUEdZtxiCtl5hpbJiS3SpEV7670w_x738h75akcO6Viw47MUlpCZfy4KZ2vLT4tcleeiW; SID=DQAAAMEAAACoYm-3B2aiLKf0cRU8spJuiNjiXEQRyxsUZqKf8UXZXS55movrnTmfEcM6FYn-gALmyMPNRIwLDBojINzkv8doX69rUQ9-'}

當我這樣做，我得到一個不包含任何東西普通Web瀏覽器返回結果：

request=urllib2.Request(url,,None,headers) 
    response=urllib2.urlopen(request) 
    html=response.read()

同樣，這段代碼返回一堆垃圾十六進制的我看不懂：

request=urllib2.Request(url,headers=headers) 
    response=urllib2.urlopen(request) 
    html=response.read()

請幫幫忙，我敢肯定，這是很簡單的，我必須只是失去了一些東西。我能得到類似的方式這個環節，而且還上傳圖片使用下面的代碼將images.google.com到：

import httplib, mimetypes, android, sys, urllib2, urllib, simplejson 

    def post_multipart(host, selector, fields, files): 
     """ 
     Post fields and files to an http host as multipart/form-data. 
     fields is a sequence of (name, value) elements for regular form fields. 
     files is a sequence of (name, filename, value) elements for data to be uploaded as files 
     Return the server's response page. 
     """ 
     content_type, body = encode_multipart_formdata(fields, files) 
     h = httplib.HTTP(host) 
     h.putrequest('POST', selector) 
     h.putheader('content-type', content_type) 
     h.putheader('content-length', str(len(body))) 
     h.endheaders() 
     h.send(body) 
     errcode, errmsg, headers = h.getreply() 
     return h.file.read() 

    def encode_multipart_formdata(fields, files): 
     """ 
     fields is a sequence of (name, value) elements for regular form fields. 
     files is a sequence of (name, filename, value) elements for data to be uploaded as files 
     Return (content_type, body) ready for httplib.HTTP instance 
     """ 
     BOUNDARY = '----------ThIs_Is_tHe_bouNdaRY_$' 
     CRLF = '\r\n' 
     L = [] 
     for (key, value) in fields: 
      L.append('--' + BOUNDARY) 
      L.append('Content-Disposition: form-data; name="%s"' % key) 
      L.append('') 
      L.append(value) 
     for (key, filename, value) in files: 
      L.append('--' + BOUNDARY) 
      L.append('Content-Disposition: form-data; name="%s"; filename="%s"' % (key, filename)) 
      L.append('Content-Type: %s' % get_content_type(filename)) 
      L.append('') 
      L.append(value) 
     L.append('--' + BOUNDARY + '--') 
     L.append('') 
     body = CRLF.join(L) 
     content_type = 'multipart/form-data; boundary=%s' % BOUNDARY 
     return content_type, body 

    def get_content_type(filename): 
     return mimetypes.guess_type(filename)[0] or 'application/octet-stream' 


    host = 'www.google.co.in' 
    selector = '/searchbyimage/upload' 
    fields = [('user-agent','Mozilla/5.0 (Windows NT 5.1; rv:6.0.2) Gecko/20100101 Firefox/6.0.2'),('connection','keep-alive'),('referer','')] 

    with open('jpeg.jpg', 'rb') as jpeg: 
     files = [('encoded_image', 'jpeg.jpg', jpeg.read())] 
    response = post_multipart(host, selector, fields, files) #added: response = 
    responseLen=(len(response)-1) 
    x=22 
    if response[(x-21):(x+1)]!='EF=\"http://www.google': 
     x+=1 
    x+=145 
    link='' 
    while response[(x+1):(x+7)]!='amp;us': #>here< 
     link=link+response[x] 
     x+=1 
    print(link)

上面的代碼返回而不是頁面的瀏覽器將返回，而是HTML帶有「已移動的鏈接」，這是我在此郵件中首先發布的「網址」。如果我可以上傳圖片並返回結果頁面，爲什麼我無法獲得生成的鏈接html頁面？這是令人沮喪的嚴重:(

請幫幫忙，我已經燒燬了我的大腦了一個多月這個問題。是的，我是一個newbee，但我認爲這將是直截了當:(

請幫我回到這一個小小的鏈接的搜索結果頁：

http://www.google.com/search?tbs=sbi:AMhZZitAaz7goe6AsfVSmFw1sbwsmX0uIjeVnzKHjEXMck70H3j32Q-6FApxrhxdSyMo0OedyWkxk3-qYbyf0q1OqNspjLu8DlyNnWVbNjiKGo87QUjQHf2_1idZ1q_1vvm5gzOCMpChYiKsKYdMywOLjJzqmzYoJNOU2UsTs_1zZGWjU-LsjdFXt_1D5bDkuyRK0YbsaLVcx4eEk_1KMkcJpWlfFEfPMutxTLGf1zxD-9DFZDzNOODs0oj2j_1KG8FRCaMFnTzAfTdl7JfgaDf_1t5Vti8FnbeG9i7qt9wF6P-QK9mdvC15hZ5UR29eQdYbcD1e4woaOQCmg8Q1VLVPf4-kf8dAI7p3jM_1MkBBwaxdt_1TsM4FLwh0oHAYKOS5qBRI28Vs0aw5_1C5-WR4dC902Eqm5eAkLiQyAM9J2bioR66g3tMWe-j9Hyh1ID40R1NyXEJDHcGxp7xOn_16XxfW_1Cq5ArdSNzxFvABb1UcXCn5s4_1LpXZxhZbauwaO8cg3CKGLUvl_1wySDB7QIkMIF2ZInEPS4K-eyErVKqOdY9caYUD8X7oOf6sDKFjT7pNHwlkXiuYbKBRYjlvRHPlcPN1WHWCJWdSNyXdZhwDI3VRaKwmi4YNvkryeNMMbhGytfvlNaaelKcOzWbvzCtSNaP2lJziN1x3btcIAplPcoZxEpb0cDlQwId3A5FDhczxpVbdRnOB-Xeq_1AiUTt_1iI6bSgUAinWXQFYWveTOttdSNCgK-VTxV4OCtlrCrZerk27RBLAzT0ol9NOfYmYhiabzhUczWk4NuiVhKN-M4eo76cAsi74PY4V_1lWjvOpI35V_1YLJQrm0fxVcD34wxFYCIllT2gYW09fj3cuBDMNbsaJqPVQ04OOGlwmcmJeAnK96xd_1aMUd6FsVLOSDS7RfS5MNUSyd1jnXvRU_1MF_1Dj8oC8sm7PfVdjm3firiMcaKM28j9kGWbY0heIGLtO_1m6ad-iKfxYEzSux2b5w62LQlP57yS7vX8RFoyKzHA0RrFIEbPBQdNMA3Vpw0G_1LvEjCAPSCV1HH1pDp0l4EnNCvUIAppVXzNMyWT_1gKITj1NLqAn-Z1tH323JwZSc77OftDSreyHJ-BPxn3n7JMkNZFcQx6S7tfBxeqJ1NuDlpax11pw0_1Oi_1nF3vyEP0NbGKSVgNvBv_1tv8ahxvrHn9UnP78FleiOpzUBfdfRPZiT20VEq5-oXtV_1XwIzrd-5_15-cf2yoL7ohyPuv3WKGUGr4YCsYje7_1D8VslqMPsvbwMg9haj3TrBKH7go70ZfPjUv3h1K7lplnnCdV0hrYVQkSLUY1eEor3L--Vu5PlewS60ZH5YEn4qTnDxniV95h8q0Y3RWXJ6gIXitR5y6CofVg

戴夫

來源

2012-07-12 user1488252

爲什麼不使用[谷歌自定義搜索API（https：//開頭開發商。 google.com/custom-search/v1/overview）他們提供的特別？ – katrielalex 2012-07-12 11:29:07

@katrielalex：該API只讓你搜索一個特定的網站，而不是一般的索引。爲此，目前只有JavaScript的一個棄用的API。 OP可能試圖搜索總體索引。 – 2012-07-12 11:34:40

http://www.google.com/robots.txt告訴你什麼，爲什麼你認爲你能勝過世界上最優秀的工程師，並誘使他們忽略它？ – geoffspear 2012-07-12 12:35:00

谷歌有幾個抗刮到位的技術，因爲他們不希望用戶去的結果，而這些API或實瀏覽器。

如果您認真對待這類網頁，我建議您看看：Selenium或Spynner。

另一個優點是都執行javascript。

來源

2012-07-12 12:34:51 marbdq

我會看看Selenium和Spynner。然而，我的意圖是使用Python標準庫。我將實現腳本，以便它可以跨平臺並且可以與android sl4a一起使用。我遵循谷歌條款和條件，因爲腳本只能在用戶的單個實例上運行。 – user1488252 2012-07-12 23:03:33

您的用戶代理未定義！

拿一個：

#!/usr/bin/python 

import urllib2 

url = "http://www.google.com/search?q=mysearch"; 
opener = urllib2.build_opener() 
opener.addheaders = [('User-agent', 'Mozilla/5.0')] 
print opener.open(url).read() 
raw_input()

如果你想找到一個其他用戶代理，你可以在Firefox中寫about:config。和搜索「用戶代理」：

的Mozilla/5.0（X11; U; Linux的i686的;的en-US; rv中：1.7.8）壁虎/ 20050511

的Googlebot/2.1（+ HTTP： //www.google.com/bot.html）

歌劇院/ 7.23（Windows 98中; U）[英文]

來源

2012-10-09 09:55:00 Ownelone

蟒蛇的urllib2不能得到

回答

相關問題