每秒發送更多than1000請求在python

我試圖同時發送請求到服務器，然後使用此代碼記錄的平均等待時間：每秒發送更多than1000請求在python

import Queue 
import time 
import threading 
import urllib2 

data = "{"image_1":"abc/xyz.jpg"}" 
headers = {.....} 
def get_url(q, url): 
    num = 1 
    sum = 0 
    while num <= 200: 
     start = time.time() 
     req = urllib2.Request(url, data, headers) 
     response = urllib2.urlopen(req) 
     end = time.time() 
     print end - start 
     num = num + 1 
     q.put(response.read()) 
     sum = sum + (end - start) 
    print sum 


theurls = ["http://example.com/example"] 
q = Queue.Queue() 

for u in theurls: 
    t = threading.Thread(target = get_url, args = (q, u)) 
    t.daemon = True 
    t.start() 

while True: 
    s = q.get() 
    print s

該代碼工作得很好，但現在我打算每秒發送超過1000個請求。我遇到了this answer，但我不知道如何使用grequests作爲我的情況。一些見解將會非常有幫助。

謝謝

來源

2016-08-13 the_unknown_spirit

這裏任何見解傢伙.. –

可能是你可以看看到異步架構像[aiohttp]（http://aiohttp.readthedocs.io/en/stable/）在python內置asyncio的頂部？在這裏或網上有一些很好的帖子（如https://compiletoi.net/fast-scraping-in-python-with-asyncio/或http://aosabook.org/en/500L/a-web-crawler -with-asyncio-coroutines.html或https://magic.io/blog/uvloop-blazing-fast-python-networking/）關於使用aiohttp進行幾個（幾乎是併發的）請求。 – mgc

查看存儲庫中的一些示例。 https://github.com/kennethreitz/grequests – 0xcaff

該文件不是很好，但來源是。閱讀來源！ Check out the first few lines of grequests.py on github：

""" 
grequests 
~~~~~~~~~ 
This module contains an asynchronous replica of ``requests.api``, powered 
by gevent. All API methods return a ``Request`` instance (as opposed to 
``Response``). A list of requests can be sent with ``map()``. 
"""

The package exports the following：

__all__ = (
    'map', 'imap', 
    'get', 'options', 'head', 'post', 'put', 'patch', 'delete', 'request' 
)

Those symbols are defined further down the file：

# Shortcuts for creating AsyncRequest with appropriate HTTP method 
get = partial(AsyncRequest, 'GET') 
options = partial(AsyncRequest, 'OPTIONS') 
head = partial(AsyncRequest, 'HEAD') 
post = partial(AsyncRequest, 'POST') 
put = partial(AsyncRequest, 'PUT') 
patch = partial(AsyncRequest, 'PATCH') 
delete = partial(AsyncRequest, 'DELETE')

partial was imported from functools at the top of the file.

from functools import partial

The documentation for functool.partial says:

返回一個新的部分對象，該對象在調用時將表現得像使用位置參數args和關鍵字參數關鍵字調用的func。如果更多的參數被提供給調用，它們被附加到參數。如果提供了其他關鍵字參數，它們將擴展並覆蓋關鍵字。

基本上打電話grequests.get來電AsyncRequest("GET", **other args go here**)。 AsyncRequest is a function which creates a new AsyncRequest.其文檔說：

""" Asynchronous request. 
Accept same parameters as ``Session.request`` and some additional: 
:param session: Session which will do request 
:param callback: Callback called on response. 
       Same as passing ``hooks={'response': callback}`` 
"""

會議是早先定義：

from requests import Session

Here's a guide on using requests sessions.

來源

2016-11-28 22:37:07 0xcaff

每秒發送更多than1000請求在python

回答

相關問題