實際的問題 - 價格會加載獨立的異步請求到不同的端點。您需要模擬在你的代碼:
from bs4 import BeautifulSoup
import requests
url = "https://www.predictit.org/Contract/4393/Will-Obama-pardon-Hillary-Clinton#openoffers"
price_url = "https://www.predictit.org/PrivateData/GetPriceListAjax?contractId=4393"
with requests.Session() as session:
session.headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36'}
session.get(url) # visit main page
# request prices
response = session.get(price_url)
soup = BeautifulSoup(response.content, "html.parser")
tables = soup.select("table.table-info")
for row in tables[0].select("tr")[2:]:
values = [td.find(text=True, recursive=False) for td in row('td') if td.text]
print(values)
打印第一「是」表的內容(用於演示目的):
[u'13', u'1555', u'12', u'240']
[u'14', u'707', u'11', u'2419']
[u'15', u'2109', u'10', u'3911']
[u'16', u'1079', u'9', u'2634']
[u'17', u'760', u'8', u'2596']
[u'18', u'510', u'7', u'970']
[u'19', u'973', u'6', u'1543']
[u'20', u'483', u'5', u'2151']
[u'21', u'884', u'4', u'1195']
[u'22', u'701', u'3', u'950']
注意通過,我們維持網絡的刮會議requests.Session()
這裏。
另請注意,price_url
包含contractId
GET參數 - 如果您要請求具有價格的其他頁面,請務必使用相應的contractId
。
此功能完美。謝謝!! – gtownrower