獲取包含谷歌反饋框中所有信息的源代碼？

Question

我想使用BeautifulSoup解析Google反饋卡片中的所有文字。

對於查詢「定義蘋果」谷歌顯示信息卡與 drop down menu。我想解析there中的所有文本。

類似的問題是here 但該解決方案不解析全框（下拉後），因爲源代碼獲取requests.get(url)不包含該信息。

有沒有一種方法，我可以得到整個源代碼沒有硒。

Answer 1

你得到所有的源回用請求，你可能有唯一的問題是一些輸出的是圖像：

headers = { 
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36", 
} 

soup = BeautifulSoup(requests.get("https://www.google.ie/search?q=define+apple", headers=headers).content, "lxml") 


div = soup.select("div.lr_dct_ent span") 
desc = "\n".join(sp.text.strip("; ") for sp in div) 
img = soup.select_one("div.lr_dct_ent").find_next("img") 
print(desc) 
print(img["src"]) 
print "\n".join(d.text for d in soup.select("div.vk_sh.vk_gy")) 

它給你：

apple 
ˈap(ə)l/ 
ˈap(ə)l 

noun 
noun: apple 
plural noun: apples 
noun: apple tree 
plural noun: apple trees 
the round fruit of a tree of the rose family, which typically has thin green or red skin and crisp flesh. 
used in names of unrelated fruits or other plant growths that resemble apples in some way, e.g. custard apple, oak apple. 
the tree bearing apples, with hard pale timber that is used in carpentry and to smoke food. 
Old English æppel, of Germanic origin; related to Dutch appel and German Apfel . 
data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw== 
Origin 
Use over time for: apple