<class'google.appengine.runtime.DeadlineExceededError'>：如何解決？

Question

好吧，我有很多問題讓我的工作開發服務器工作到生產服務器:)。我有一個任務會通過並請求網址並收集和更新數據。運行需要30分鐘。

我上傳到生產服務器，並在30秒後轉到與其對應的.py腳本appname.appspot.com/tasks/rrs的網址我得到類google.appengine.runtime.DeadlineExceededError'有沒有辦法解決這個問題？這是一頁30秒的截止日期嗎？此腳本在開發服務器中正常工作，我轉到url並關聯.py腳本運行直到完成。

import time 
import random 
import string 
import cPickle 
from StringIO import StringIO 
try: 
    import json 
except ImportError: 
    import simplejson as json 
import urllib 
import pprint 
import datetime 
import sys 
sys.path.append("C:\Program Files (x86)\Google\google_appengine") 
sys.path.append("C:\Program Files (x86)\Google\google_appengine\lib\yaml\lib") 
sys.path.append("C:\Program Files (x86)\Google\google_appengine\lib\webob") 
from google.appengine.api import users 
from google.appengine.ext import webapp 
from google.appengine.ext.webapp.util import run_wsgi_app 
from google.appengine.ext import db 
class SR(db.Model): 
    name = db.StringProperty() 
    title = db.StringProperty() 
    url = db.StringProperty() 

##request url and returns JSON_data 
def overview(page): 
    u = urllib.urlopen(page) 
    bytes = StringIO(u.read()) 
    ##print bytes 
    u.close() 
    try: 
     JSON_data = json.load(bytes) 
     return JSON_data 
    except ValueError,e: 
     print e," Couldn't get .json for %s" % page 
     return None 

##specific code to parse particular JSON data and append new SR objects to the given url list 
def parse_json(JSON_data,lists): 
    sr = SR() 
    sr.name = ##data gathered 
    sr.title = ##data gathered 
    sr.url = ##data gathered 
    lists.append(sr) 
    return lists 

## I want to be able to request lets say 500 pages without timeing out 
page = 'someurlpage.com'##starting url 
url_list = [] 
for z in range(0,500): 
    page = 'someurlpage.com/%s'%z 
    JSON_data = overview(page)##get json data for a given url page 
    url_list = parse_json(JSON_data,url_list)##parse the json data and append class objects to a given list 
db.put(url_list)##finally add object to gae database 

Answer 1

是的，App Engine規定了30秒的最後期限。其中一種方法可能是try/except DeadlineExceededError，其餘部分放在taskqueue。

但是，您無法讓您的請求長時間運行。

您也可以嘗試Bulkupdate

例子：

class Todo(db.Model): 
    page = db.StringProperty() 

class BulkPageParser(bulkupdate.BulkUpdater): 
    def get_query(self): 
     return Todo.all() 

    def handle_entity(self, entity): 
     JSON_data = overview(entity.page) 
     db.put(parse_json(JSON_data, []) 
     entity.delete() 

# Put this in your view code: 
for i in range(500): 
    Todo(page='someurlpage.com/%s' % i).put() 

job = BulkPageParser() 
job.start() 

Answer 2

好了，所以如果我動態添加鏈接爲我解析的頁面，像這樣我相信我會添加到待辦事項隊列。

def handle_entity(self, entity): 
    JSON_data = overview(entity.page) 
    data_gathered,new_links = parse_json(JSON_data, [])##like earlier returns the a list of sr objects, and now a list of new links/pages to go to 
    db.put(data_gathered) 
    for link in new_links: 
     Todo(page=link).put() 
    entity.delete()