如何通過腳本標籤使用python解析和beautifulsoup

Question

我試圖提取幀標籤，它是內部document.write功能的頁面上，如下所示的屬性：

<script language="javascript"> 
. 
. 
. 
document.write('<frame name="nav" src="/nav/index_nav.html" marginwidth="0" marginheight="0" scrolling="no" frameborder="0" border = "no" noresize>'); 
if (anchor != "") { 
    document.write('<frame name="body" src="http://content.members.fidelity.com/mfl/summary/0,,' + cusip + ',00.html?' + anchor + '" marginwidth="0" marginheight="0" scrolling="auto" frameborder="0" noresize>'); 
} else { 
    document.write('<frame name="body" src="http://content.members.fidelity.com/mfl/summary/0,,' + cusip + ',00.html" marginwidth="0" marginheight="0" scrolling="auto" frameborder="0" noresize>'); 
} 
document.write('</frameset>'); 


// end hiding --> 
</script> 

findAll('frame')方法沒有幫助。有沒有辦法讀取幀標籤的內容？

我使用python 2.5和BeautifulSoup 3.0.8。

只要我能夠得到結果，我也可以使用Python 3.1與BeautifulSoup 3.1 。

謝謝

Answer 1

你無法單獨使用BeautifulSoup。 BeautifulSoup解析HTML到達瀏覽器（在任何重寫或DOM操作之前），並且它不解析（更不用說執行）Javascript。

您可能想在這種特殊情況下使用簡單的正則表達式。

Answer 2

Pyparsing可能會幫助您彌合JS和HTML的混合。此解析器查找包含帶引號的字符串或多個帶引號的字符串和標識符的字符串表達式的document.write語句，準評估字符串表達式，解析嵌入的<frame>標記，並將幀屬性作爲pyparsing ParseResults對象返回給您訪問命名屬性，就好像它們是對象屬性或字典鍵（您的首選項）。

jssrc = """ 
<script language="javascript"> 
. 
. 
. 
document.write('<frame name="nav" src="/nav/index_nav.html" marginwidth="0" marginheight="0" scrolling="no" frameborder="0" border = "no" noresize>'); 
if (anchor != "") 
{ document.write('<frame name="body" src="http://content.members.fidelity.com/mfl/summary/0,,' + cusip + ',00.html?' + anchor + '" marginwidth="0" marginheight="0" scrolling="auto" frameborder="0" noresize>'); } 
else 
{ document.write('<frame name="body" src="http://content.members.fidelity.com/mfl/summary/0,,' + cusip + ',00.html" marginwidth="0" marginheight="0" scrolling="auto" frameborder="0" noresize>'); } 
document.write('</frameset>'); 
    // end hiding --> 
    </script>""" 

from pyparsing import * 

# define some basic punctuation, and quoted string 
LPAR,RPAR,PLUS = map(Suppress,"()+") 
qs = QuotedString("'") 

# use pyparsing helper to define an expression for opening <frame> 
# tags, which includes support for attributes also 
frameTag = makeHTMLTags("frame")[0] 

# some of our document.write statements contain not a sting literal, 
# but an expression of strings and vars added together; define 
# an identifier expression, and add a parse action that converts 
# a var name to a likely value 
ident = Word(alphas).setParseAction(lambda toks: evalvars[toks[0]]) 
evalvars = { 'cusip' : "CUSIP", 'anchor' : "ANCHOR" } 

# now define the string expression itself, as a quoted string, 
# optionally followed by identifiers and quoted strings added 
# together; identifiers will get translated to their defined values 
# as they are parsed; the first parse action on stringExpr concatenates 
# all the tokens; then the second parse action actually parses the 
# body of the string as a <frame> tag and returns the results of parsing 
# the tag and its attributes; if the parse fails (that is, if the 
# string contains something that is not a <frame> tag), the second 
# parse action will throw an exception, which will cause the stringExpr 
# expression to fail 
stringExpr = qs + ZeroOrMore(PLUS + (ident | qs)) 
stringExpr.setParseAction(lambda toks : ''.join(toks)) 
stringExpr.addParseAction(lambda toks: 
    frameTag.parseString(toks[0],parseAll=True)) 

# finally, define the overall document.write(...) expression 
docWrite = "document.write" + LPAR + stringExpr + RPAR 

# scan through the source looking for document.write commands containing 
# <frame> tags using scanString; print the original source fragment, 
# then access some of the attributes extracted from the <frame> tag 
# in the quoted string, using either object-attribute notation or 
# dict index notation 
for dw,locstart,locend in docWrite.scanString(jssrc): 
    print jssrc[locstart:locend] 
    print dw.name 
    print dw["src"] 
    print 

打印：

document.write('<frame name="nav" src="/nav/index_nav.html" marginwidth="0" marginheight="0" scrolling="no" frameborder="0" border = "no" noresize>') 
nav 
/nav/index_nav.html 

document.write('<frame name="body" src="http://content.members.fidelity.com/mfl/summary/0,,' + cusip + ',00.html?' + anchor + '" marginwidth="0" marginheight="0" scrolling="auto" frameborder="0" noresize>') 
body 
http://content.members.fidelity.com/mfl/summary/0,,CUSIP,00.html?ANCHOR 

document.write('<frame name="body" src="http://content.members.fidelity.com/mfl/summary/0,,' + cusip + ',00.html" marginwidth="0" marginheight="0" scrolling="auto" frameborder="0" noresize>') 
body 
http://content.members.fidelity.com/mfl/summary/0,,CUSIP,00.html