lxml找不到Chrome提供的xpath？

Question

我正在嘗試從鏈接的搜索頁面獲取公司的行業信息。我使用Chrome的開發工具獲取xpath表單，但它會返回空括號。這裏似乎是什麼問題？

from lxml import html 
import requests 

page = requests.get('https://www.linkedin.com/search/results/companies/?keywords=cisco.com') 
tree = html.fromstring(page.content) 

industry = tree.xpath('//*[@id="ember3734"]/div/div[1]/p[1]') 

print(industry) 

Answer 1

我用selenium和phantomjs製作了腳本，因爲網站使用了很多javascript。

#!/usr/bin/env python 
# -*- coding: utf-8 -*- 

import lxml.html 
import re 
from selenium import webdriver 
from time import sleep 
from selenium.webdriver import DesiredCapabilities 
from pprint import pprint 

desired_capabilities = DesiredCapabilities.PHANTOMJS.copy() 
desired_capabilities['phantomjs.page.customHeaders.User-Agent'] = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) ' \ 
                    'AppleWebKit/537.36 (KHTML, like Gecko) ' \ 
                    'Chrome/39.0.2171.95 Safari/537.36' 
driver = webdriver.PhantomJS(desired_capabilities=desired_capabilities) 

username = 'email@email.com' 
password = 'password' 


# driver = webdriver.PhantomJS() 
driver.set_window_size(1120, 550) 
driver.get("https://www.linkedin.com") 

driver.find_element_by_id('login-password').send_keys(password) 
driver.find_element_by_id('login-email').send_keys(username) 
driver.find_element_by_id("login-submit").click() 
driver.get("https://www.linkedin.com/search/results/companies/?keywords=cisco.com") 
sleep(3) 
html = driver.page_source 
root = lxml.html.fromstring(html) 

reg = re.compile('ember-view\">\s+<h3\s+class=\"search\-result__title\s+Sans\-17px\-black\-85\%\-semibold-dense\">(.*?)<\/h3>') 
names = reg.findall(html) 

pprint(names) 

driver.quit() 

Answer 2

我覺得這個頁面是由JavaScript生成的。由於請求在不執行JavaScript的情況下下載頁面，因此只會獲得主頁面/模板，而不是您期望的數據。

嘗試在Chrome瀏覽器中查看源頁面進行確認。