爲什麼我從BeautifulSoup獲得的HTML與我在檢查元素時看到的HTML相同？

我正在創建一個用戶名刮板，我真的不明白爲什麼當我解析它時，HTML會'消失'。讓我們這個網站，例如： http://www.lolking.net/leaderboards#/eune/1 爲什麼我從BeautifulSoup獲得的HTML與我在檢查元素時看到的HTML相同？

瞭解如何有一個TBODY和一堆在它的表？那麼當我把它和輸出解析它的外殼TBODY是空

<div style="background: #333; box-shadow: 0 0 2px #000; padding: 10px;"> 
    <table class="lktable" id="leaderboard_table" width="100%"> 
    <thead> 
     <tr> 
     <th style="width: 80px;"> 
     Rank 
     </th> 
     <th style="width: 80px;"> 
     Change 
     </th> 
     <th style="width: 100px;"> 
     Tier 
     </th> 
     <th> 
     Summoner 
     </th> 
     <th style="width: 150px;"> 
     Top Champions 
     </th> 
     </tr> 
    </thead> 
    <tbody> 
    </tbody> 
    </table> 
    </div> 
    </div>

這究竟是爲什麼以及如何解決呢？

來源

2016-08-23 edsheeran

看起來好像表格內容是使用JavaScript生成的。 BeautifulSoup不執行JavaScript，所以表是空的。請看Selenium。 –

看看[這裏]（http://stackoverflow.com/questions/8047666/how-to-combine-scrapy-and-htmlunit-to-crawl-urls-with-javascript）和[here]（http：/ /stackoverflow.com/questions/8550114/can-scrapy-be-used-to-scrape-dynamic-content-from-websites-that-are-using-ajax）。你可能會發現一些有用的信息。 –

你不需要Selenium，只是模仿ajax請求，你可以得到所有json格式的數據 –

你可以得到所有的數據JSON格式，-111，你需要做的是分析原始頁面的源代碼內從腳本代碼的值，並把它傳遞給「http://www.lolking.net/leaderboards/some_value_here/eune/1.json」：

from bs4 import BeautifulSoup 
import requests 
import re 

patt = re.compile("\$\.get\('/leaderboards/(\w+)/") 
js = "http://www.lolking.net/leaderboards/{}/eune/1.json" 
soup = BeautifulSoup(requests.get("http://www.lolking.net/leaderboards#/eune/1").content) 
script = soup.find("script", text=re.compile("\$\.get\('/leaderboards/")) 

val = patt.search(script.text).group(1) 
data = requests.get(js.format(val)).json()

數據給您包含所有播放器信息的json，如：

{'data': [{'division': '1', 
      'global_ranking': '12', 
      'league_points': '1217', 
      'lks': '2961', 
      'losses': '31', 
      'most_played_champions': [{'assists': '238', 
             'champion_id': '236', 
             'creep_score': '7227', 
             'deaths': '131', 
             'kills': '288', 
             'losses': '5', 
             'played': '39', 
             'wins': '34'}, 
            {'assists': '209', 
             'champion_id': '429', 
             'creep_score': '5454', 
             'deaths': '111', 
             'kills': '204', 
             'losses': '3', 
             'played': '27', 
             'wins': '24'}, 
            {'assists': '155', 
             'champion_id': '81', 
             'creep_score': '4800', 
             'deaths': '103', 
             'kills': '168', 
             'losses': '8', 
             'played': '26', 
             'wins': '18'}], 
      'name': 'Sadastyczny', 
      'previous_ranking': '2', 
      'profile_icon_id': 7, 
      'ranking': '1', 
      'region': 'eune', 
      'summoner_id': '42893043', 
      'tier': '6', 
      'tier_name': 'CHALLENGER', 
      'wins': '128'}, 
      {'division': '1', 
      'global_ranking': '30', 
      'league_points': '1128', 
      'lks': '2956', 
      'losses': '180', 
      'most_played_champions': [{'assists': '928', 
             'champion_id': '24', 
             'creep_score': '37601', 
             'deaths': '1426', 
             'kills': '1874', 
             'losses': '64', 
             'played': '210', 
             'wins': '146'}, 
            {'assists': '501', 
             'champion_id': '67', 
             'creep_score': '16836', 
             'deaths': '584', 
             'kills': '662', 
             'losses': '37', 
             'played': '90', 
             'wins': '53'}, 
            {'assists': '124', 
             'champion_id': '157', 
             'creep_score': '5058', 
             'deaths': '205', 
             'kills': '141', 
             'losses': '14', 
             'played': '28', 
             'wins': '14'}], 
      'name': 'Richor', 
      'previous_ranking': '1', 
      'profile_icon_id': 577, 
      'ranking': '2', 
      'region': 'eune', 
      'summoner_id': '40385818', 
      'tier': '6', 
      'tier_name': 'CHALLENGER', 
      'wins': '254'}, 
      {'division': '1', 
      'global_ranking': '49', 
      'league_points': '1051', 
      'lks': '2953', 
      'losses': '47', 
      'most_played_champions': [{'assists': '638', 
             'champion_id': '117', 
             'creep_score': '11927', 
             'deaths': '99', 
             'kills': '199', 
             'losses': '7', 
             'played': '66', 
             'wins': '59'}, 
            {'assists': '345', 
             'champion_id': '48', 
             'creep_score': '8061', 
             'deaths': '99', 
             'kills': '192', 
             'losses': '11', 
             'played': '43', 
             'wins': '32'}, 
            {'assists': '161', 
             'champion_id': '114', 
             'creep_score': '5584', 
             'deaths': '64', 
             'kills': '165', 
             'losses': '11', 
             'played': '31', 
             'wins': '20'}],

來源

2016-08-23 13:11:10