ThreadPoolExecutor vs threading.Thread

Question

我有一個關於ThreadPoolExecutor vs Thread類的性能問題，在我看來，我缺乏一些基本的理解。

我有兩個功能的網絡刮板。首先來分析一個網站主頁和第二的每個圖像的鏈接，加載圖像關閉解析鏈接：

import threading 
import urllib.request 
from bs4 import BeautifulSoup as bs 
import os 
from concurrent.futures import ThreadPoolExecutor 

path = r'C:\Users\MyDocuments\Pythom\Networking\bbc_images_scraper_test' 
url = 'https://www.bbc.co.uk' 

# Function to parse link anchors for images 
def img_links_parser(url, links_list): 
    res = urllib.request.urlopen(url) 
    soup = bs(res,'lxml') 
    content = soup.findAll('div',{'class':'top-story__image'}) 

    for i in content: 
     try: 
      link = i.attrs['style'] 
      # Pulling the anchor from parentheses 
      link = link[link.find('(')+1 : link.find(')')] 
      # Putting the anchor in the list of links 
      links_list.append(link) 
     except: 
      # links might be under 'data-lazy' attribute w/o paranthesis 
      links_list.append(i.attrs['data-lazy']) 

# Function to load images from links 
def img_loader(base_url, links_list, path_location): 
    for link in links_list: 
     try: 
      # Pulling last element off the link which is name.jpg 
      file_name = link.split('/')[-1] 
      # Following the link and saving content in a given direcotory 
      urllib.request.urlretrieve(urllib.parse.urljoin(base_url, link), 
      os.path.join(path_location, file_name)) 
     except: 
      print('Error on {}'.format(urllib.parse.urljoin(base_url, link))) 

下面的代碼是在兩種情況分裂：

案例1：我使用多線程：

threads = [] 
t1 = threading.Thread(target = img_loader, args = (url, links[:10], path)) 
t2 = threading.Thread(target = img_loader, args = (url, links[10:20], path)) 
t3 = threading.Thread(target = img_loader, args = (url, links[20:30], path)) 
t4 = threading.Thread(target = img_loader, args = (url, links[30:40], path)) 
t5 = threading.Thread(target = img_loader, args = (url, links[40:50], path)) 
t6 = threading.Thread(target = img_loader, args = (url, links[50:], path)) 

threads.extend([t1,t2,t3,t4,t5,t6]) 
for t in threads: 
    t.start() 
for t in threads: 
    t.join() 

上述代碼在我的機器上執行了10秒鐘的工作。

情況2：我使用ThreadPoolExecutor

with ThreadPoolExecutor(50) as exec: 
    results = exec.submit(img_loader, url, links, path) 

上面的代碼結果18秒。

我的理解是，ThreadPoolExecutor爲每個工人創建一個線程。所以，假設我將max_workers設置爲50會導致50個線程，因此應該更快地完成作業。

有人可以請解釋我在這裏錯過了什麼？我承認我在這裏犯了一個愚蠢的錯誤，但我不明白。

非常感謝！

Answer 1

在案例2中，您將所有鏈接發送給一名工作人員。取而代之的

exec.submit(img_loader, url, links, path) 

你需要：

for link in links: 
    exec.submit(img_loader, url, [link], path) 

我不嘗試一下我自己，從reading the documentation of ThreadPoolExecutor