無法使用python刮取特定網站，urllib2

Question

我使用urllib來提取網頁，它似乎不適用於zomato網站。它可以與我嘗試過的其他幾個網站一起使用。我試圖禁用我的防火牆，防病毒軟件，並在嘗試捕獲中包裹urlopen行，因爲我發現它here。我曾嘗試使用互聯網上的代碼似乎工作正常，但我收到一條錯誤消息，

v = self._sslobj.read(len or 1024) socket.error: [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

可能的原因是什麼？

這是它的簡單代碼。

import urllib2 
from bs4 import BeautifulSoup 

def extract_link(url):    
    page = urllib2.urlopen(url).read()      
    return BeautifulSoup(page) 

def main(): 
    link = 'https://www.zomato.com/kolkata' 
    soup=extract_link(link) 
    print soup.prettify() 

if __name__== '__main__': 
    main() 

Answer 1

由於zomato.com是一個動態的網站，並在很大程度上依賴於JavaScript或AJAX來填補它的內容，你需要爲了得到這些內容的瀏覽器。最重要的是，zomato會過濾掉它認爲是由機器人制作的請求。所以，由於這些原因，我建議您使用Selenium，因爲它不會模擬瀏覽器會話，而是瀏覽器會話。編寫硒基本上是寫一套操作並將它們提供給瀏覽器（通常是Firefox，但它可以與其他工作）。你也可以使用PhantomJs作爲替代，因爲你正在使用python。

這裏有一點幫助建立硒和複製你的上面的代碼。從https://chromedriver.storage.googleapis.com/index.html?path=2.25/

1. 下載chromedriver.exe然後提取該.exe文件到 「鉻驅動程序」 文件夾中。
2. 然後將包含該文件的文件夾剪切並粘貼到C驅動器中。
3. 現在，做pip install selenium.

然後運行該代碼

import time 
from selenium import webdriver 
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.common.by import By 
from selenium.webdriver.support import expected_conditions as EC 

driver = webdriver.Chrome("C:\Chrome Driver\chromedriver") 
driver.get("https://www.zomato.com/kolkata") 
time.sleep(3) 

wait = WebDriverWait(driver, 30) 

body = wait.until(EC.presence_of_element_located((By.XPATH,"/html/body"))) 
print(body.text) 

這是輸出你會得到：

We're Hiring! 
Order Food Online! 
Log in 
Find the best restaurants, cafés, and bars in Kolkata 
Kolkata 
Search 
Collections 
Explore curated lists of top restaurants, cafes, pubs, and bars in and around Kolkata, based on trends 
Trending this week 
The most popular restaurants in town this week 
New restaurants you probably haven't tried yet 
The best new places in town 
Happy hours 
Great deals on booze. Happy hours indeed 
Hookah bars 
Great places to enjoy flavored Hookah. 
All collections in Kolkata 
Order Food Online 
From the best restaurants delivering to your doorstep 
Enter your delivery location 
Detect 
Order Food Online! 
Get 15% off (up to Rs. 100) on your first order with the code EATIN when you pay online. 
Quick Searches 
Discover restaurants by type of meal 
Delivery 
Breakfast 
Lunch 
Dinner 
Drinks & Nightlife 
Cafés 
Pocket-Friendly Delivery 
Desserts & Bakes 
Popular localities in and around Kolkata 
Explore restaurants, bars, and cafés by locality 
Park Street Area (111 places) 
Sector 5, Salt Lake (102 places) 
Ballygunge (154 places) 
Sector 1, Salt Lake (167 places) 
Rajarhat New Town (212 places) 
Southern Avenue (86 places) 
Elgin (81 places) 
Prince Anwar Shah Road (108 places) 
Kankurgachi (116 places) 
Kasba (118 places) 
Camac Street Area (61 places) 
Gariahat (62 places) 
Park Circus Area (80 places) 
Desapriya Park (61 places) 
New Market Area (105 places) 
Behala (136 places) 
Hindustan Park (30 places) 
Hatibagan (70 places) 
Sector 3, Salt Lake (83 places) 
Esplanade (64 places) 
Jadavpur (91 places) 
Golpark (37 places) 
Bhawanipur (100 places) 
Science City Area (19 places) 
Theatre Road (47 places) 
Shyam Bazar (54 places) 
Garia (108 places) 
Tangra (32 places) 
Nagerbazar (60 places) 
Tollygunge (110 places) 
Top Reviewers in Kolkata 
Pamela Nandi 
446 Reviews , 245 Followers 
Follow 
Dipyaman Basu 
883 Reviews , 5854 Followers 
Follow 
Anusreea Paul 
613 Reviews , 2114 Followers 
Follow 
Krishanu Das 
832 Reviews , 1599 Followers 
Follow 
Priyabrataa Ganguly 
536 Reviews , 683 Followers 
Follow 
Reviewers leaderboard » 
Top Photographers in Kolkata 
Avijit Biswas 
698 Reviews , 1381 Followers 
Follow 
Krishanu Das 
832 Reviews , 1599 Followers 
Follow 
Mani 
153 Reviews , 4473 Followers 
Follow 
Anusreea Paul 
613 Reviews , 2114 Followers 
Follow 
Anamitraa Chakraborty 
610 Reviews , 1255 Followers 
Follow 
Photographers leaderboard » 
Top Bloggers in Kolkata 
Arghya Deep 
137 Reviews , 3498 Followers 
Follow 
Subham Ghosh 
208 Reviews , 6497 Followers 
Follow 
Snehasis 
230 Reviews , 929 Followers 
Follow 
Koninika De 
356 Reviews , 4695 Followers 
Follow 
Dipyaman Basu 
883 Reviews , 5854 Followers 
Follow 
Bloggers leaderboard» 
Looking for the Food Feed? Get the app! 
Follow foodies to see their reviews and photos in your Feed, and discover great new restaurants! 
We'll send you a link, open it on your phone to download the app 
Text App Link 
OR 
Email App Link 
23 
COUNTRIES 
1.2M 
RESTAURANTS 
80M 
FOODIES EVERY MONTH 
30M 
PHOTOS 
10M 
REVIEWS 
18M 
BOOKMARKS 
About Careers Culture Mobile Apps Businesses Developers Blog Community Contact 
English 
Businesses 
Add a Restaurant 
Claim your Listing 
Business App 
Restaurant Widgets 
Guidelines 
Business Blog 
Advertise 
Book 
Order 
Base 
Whitelabel 
Countries 
Australia 
Brasil 
Canada 
Chile 
Czech Republic 
India 
Indonesia 
Ireland 
Italy 
Lebanon 
Malaysia 
New Zealand 
Philippines 
Poland 
Portugal 
Qatar 
Slovakia 
South Africa 
Sri Lanka 
Turkey 
UAE 
United Kingdom 
United States 
Privacy Terms Code of Conduct API Policy CSR Security Sitemap 
By continuing past this page, you agree to our Terms of Service, Cookie Policy, Privacy Policy and Content Policies. All trademarks are properties of their respective owners. © 2008-2016 - Zomato™ Media Pvt Ltd. All rights reserved.