我寫一個小的多線程HTTP文件下載,並希望能夠縮小可用線程的代碼遇到錯誤Python的 - 收縮的線程池動態/停止線程
的錯誤都會被具體到http服務器不允許任何連接的錯誤返回
例如,如果我設置了5個線程池,每個線程都試圖打開它自己的連接並下載一個文件塊。該服務器可能只允許2個連接,我相信會返回503錯誤,我想檢測到這一點並關閉一個線程,最終限制池的大小,大概只有服務器將允許的2個
我可以做一個線程本身?
是自我。 線程 _stop()是否足夠?
我還需要加入()嗎?
這裏是我的工人的類,它的下載,從隊列過程抓起,一旦下載它轉儲結果爲要保存resultQ由主線程文件
它在這兒,我想檢測一個HTTP 503和停止/終止/從可用池中取出一個線程 - 當然後面再添加失敗塊的隊列,以便剩餘的線程將處理它
class Downloader(threading.Thread):
def __init__(self, queue, resultQ, file_name):
threading.Thread.__init__(self)
self.workQ = queue
self.resultQ = resultQ
self.file_name = file_name
def run(self):
while True:
block_num, url, start, length = self.workQ.get()
print 'Starting Queue #: %s' % block_num
print start
print length
#Download the file
self.download_file(url, start, length)
#Tell queue that this task is done
print 'Queue #: %s finished' % block_num
self.workQ.task_done()
def download_file(self, url, start, length):
request = urllib2.Request(url, None, headers)
if length == 0:
return None
request.add_header('Range', 'bytes=%d-%d' % (start, start + length))
while 1:
try:
data = urllib2.urlopen(request)
except urllib2.URLError, u:
print "Connection did not start with", u
else:
break
chunk = ''
block_size = 1024
remaining_blocks = length
while remaining_blocks > 0:
if remaining_blocks >= block_size:
fetch_size = block_size
else:
fetch_size = int(remaining_blocks)
try:
data_block = data.read(fetch_size)
if len(data_block) == 0:
print "Connection: [TESTING]: 0 sized block" + \
" fetched."
if len(data_block) != fetch_size:
print "Connection: len(data_block) != length" + \
", but continuing anyway."
self.run()
return
except socket.timeout, s:
print "Connection timed out with", s
self.run()
return
remaining_blocks -= fetch_size
chunk += data_block
resultQ.put([start, chunk])
下面是我的init線程游泳池,進一步下來我把項目排隊
# create a thread pool and give them a queue
for i in range(num_threads):
t = Downloader(workQ, resultQ, file_name)
t.setDaemon(True)
t.start()
謝謝,我需要仔細閱讀。 我終於和你說的一樣得出了同樣的結論,只要退出線程run(),它就會停止嘗試從隊列中拔出。 我喜歡你所提出的建議,謝謝! – MikeM 2013-05-05 21:37:52