Python樹建設與多處理池 - 沒有加速

Question

我想實現使用python中的多處理池的二進制樹的多線程建設。主要思想是有N個硬件線程可用，在某種程度上，我想在單獨的線程中創建樹的每個分支。沒有數據依賴關係，並且每個遞歸調用都對自己的數據起作用，所以不存在與數據競爭相關的問題。我所熟悉的GIL的限制，所以我已經決定使用多池：

​​

的問題是我不能夠得到任何的加速。創建樹的功能如下所示：

def createTreeMT(points, pool, level = 2): 
    # If there are no more points to process 
    if len(points) < 1: 
     return 

    # Divide points into two groups: 
    left_points = .... 
    right_points = .... 

    tree = Tree() 

    if(level == 0): 
     if len(left_points) > 0: 
      leftPointsResult = pool.apply_async(createTree, (left_points)) 
     if len(right_points) > 0: 
      rightPointsResult = pool.apply_async(createTree, (right_points)) 

     if(leftPointsResult): 
      tree.left = leftPointsResult.get() 
     if(rightPointsResult): 
      tree.right = rightPointsResult.get() 
    else: 
     if len(left_points) > 0: 
      tree.left = createTreeMT(left_points, pool, level-1) 

     if len(right_points) > 0: 
      tree.right = createTreeMT(right_points, pool, level-1) 

    return tree 


def createTree(points): 
    # If there are no more points to process 
    if len(points) < 1: 
     return 

    # Divide points into two groups: 
    left_points = .... 
    right_points = .... 

    tree = Tree() 

    if len(left_points) > 0: 
     tree.left = createTree(left_points) 

    if len(right_points) > 0: 
     tree.right = createTree(right_points) 

    return tree 

我做錯了什麼？有沒有更好的方法在標準的Python 2.7中執行這樣的任務？

Answer 1

apply_async返回一個AsyncResult對象，當主進程調用它的「get」函數時，它將阻塞，直到計算出結果。所以，你的主進程總是阻塞它正在創建的每個子任務。因此，沒有加速！

一個選項是在apply_async中使用「回調」選項。所以你更換你的下面的代碼：與

if(level == 0): 
    if len(left_points) > 0: 
     leftPointsResult = pool.apply_async(
      createTree, (left_points), 
      callback=lambda x: tree.left=x) 
    if len(right_points) > 0: 
     rightPointsResult = pool.apply_async(
      createTree, (right_points), 
      callback=lambda x: tree.right=x) 

if(level == 0): 
    if len(left_points) > 0: 
     leftPointsResult = pool.apply_async(createTree, (left_points)) 
    if len(right_points) > 0: 
     rightPointsResult = pool.apply_async(createTree, (right_points)) 

    if(leftPointsResult): 
     tree.left = leftPointsResult.get() 
    if(rightPointsResult): 
     tree.right = rightPointsResult.get() 

回調lambda函數被調用一次的結果是準備好了。正如你在代碼中看到的那樣，你的樹會被並行地正確填充，因爲結果是由池進程計算的。