爲什麼這個泡菜沒有遞歸到達最大遞歸深度？

Question

這是我的代碼，它不包含遞歸，但它擊中第一泡菜最大遞歸深度...

代碼：

#!/usr/bin/env python 
from bs4 import BeautifulSoup 
from urllib2 import urlopen 
import pickle 

# open page and return soup list 
def get_page_startups(page_url): 
    html = urlopen(page_url).read() 
    soup = BeautifulSoup(html, "lxml") 
    return soup.find_all("div","startup item") 

# 
# Get certain text from startup soup 
# 
def get_name(startup): 
    return startup.find("a", "profile").string 
def get_website(startup): 
    return startup.find("a", "visit")["href"] 
def get_status(startup): 
    return startup.find("p","status").strong.string[8:] 
def get_twitter(startup): 
    return startup.find("a", "comment").string 
def get_high_concept_pitch(startup): 
    return startup.find("div","headline").find_all("em")[1].string 
def get_elevator_pitch(startup): 
    startup_soup = BeautifulSoup(urlopen("http://startupli.st" + startup.find("a","profile")["href"]).read(),"lxml") 
    return startup_soup.find("p", "desc").string.rstrip().lstrip() 
def get_tags(startup): 
    return startup.find("p","tags").string 
def get_blog(startup): 
    try: 
    return startup.find("a","visit blog")["href"] 
    except TypeError: 
    return None 
def get_facebook(startup): 
    try: 
    return startup.find("a","visit facebook")["href"] 
    except TypeError: 
    return None 
def get_angellist(startup): 
    try: 
    return startup.find("a","visit angellist")["href"] 
    except TypeError: 
    return None 
def get_linkedin(startup): 
    try: 
    return startup.find("a","visit linkedin")["href"] 
    except TypeError: 
    return None 
def get_crunchbase(startup): 
    try: 
    return startup.find("a","visit crunchbase")["href"] 
    except TypeError: 
    return None 


# site to scrape 
BASE_URL = "http://startupli.st/startups/latest/" 

# scrape all pages 
for page_no in xrange(1,142): 
    startups = get_page_startups(BASE_URL + str(page_no)) 

    # search soup and pickle data 
    for i, startup in enumerate(startups): 
    s = {} 
    s['name'] = get_name(startup) 
    s['website'] = get_website(startup) 
    s['status'] = get_status(startup) 
    s['high_concept_pitch'] = get_high_concept_pitch(startup) 
    s['elevator_pitch'] = get_elevator_pitch(startup) 
    s['tags'] = get_tags(startup) 
    s['twitter'] = get_twitter(startup) 
    s['facebook'] = get_facebook(startup) 
    s['blog'] = get_blog(startup) 
    s['angellist'] = get_angellist(startup) 
    s['linkedin'] = get_linkedin(startup) 
    s['crunchbase'] = get_crunchbase(startup) 

    f = open(str(i)+".pkl", "wb") 
    pickle.dump(s,f) 
    f.close() 

    print "Done " + str(page_no) 

這是0.pkl內容的異常引發之後：

http://pastebin.com/DVS1GKzz千行長！

有從泡菜中BASE_URL一些HTML ...但我沒有鹹菜任何HTML串...

Answer 1

BeautifulSoup .string屬性是不實際的字符串：

>>> from bs4 import BeautifulSoup 
>>> soup = BeautifulSoup('<div>Foo</div>') 
>>> soup.find('div').string 
    u'Foo' 
>>> type(soup.find('div').string) 
    bs4.element.NavigableString 

嘗試使用str(soup.find('div').string)代替，看看是否有幫助。此外，我不認爲醃菜真的是這裏最好的格式。在這種情況下JSON更容易。

Answer 2

最有可能的鹹菜做遞歸內部，而您試圖解析該文件是大。你可以嘗試增加允許遞歸次數的限制。

import sys 
sys.setrecursionlimit(10000) 

但是，這是不建議任何類型的生產就緒的應用程序，因爲它可能會掩蓋實際的問題，但在調試過程中可以幫助高亮的問題（一個或多個）。

Answer 3

醃汁不能處理BeautifulSoup節點。類似的問題有一些解決方法：

• RuntimeError: maximum recursion depth exceeded with Python 3.2 pickle.dump
• pickle.dump meet RuntimeError: maximum recursion depth exceeded in cmp