Python/Django比較和更新模型對象

Question

Iv只是剛剛啓動了python，但在過去幾個月中學到了很多東西，現在我已經碰到了關於更新模型中對象的牆壁，並且速度很快。

我有一個名爲產品的模型，這是從一個csv文件填充，每天這個文件得到更新與成本和數量的變化，我可以比較文件的每一行與產品模型，但有120K線這需要3-4小時。

我可以採取什麼過程來使此過程更快。我只想在成本和數量發生變化的情況下修改對象

任何建議我如何解決這個問題？

Ver3我所嘗試過的。

from django.core.management import BaseCommand 
from multiprocessing import Pool 
from django.contrib.auth.models import User 
from pprint import pprint 
from CentralControl.models import Product, Supplier 
from CentralControl.management.helpers.map_ingram import * 
from CentralControl.management.helpers.helper_generic import * 
from tqdm import tqdm 

from CentralControl.management.helpers.config import get_ingram 
import os, sys, csv, zipfile, CentralControl 

# Run Script as 'SYSTEM' 
user = User.objects.get(id=1) 

# Get Connection config. 
SUPPLIER_CODE, FILE_LOCATION, FILE_NAME = get_ingram() 


class Command(BaseCommand): 
def handle(self, *args, **options): 
    list_in = get_file() 
    list_current = get_current_list() 

    pool = Pool(6) 

    pool.map(compare_lists(list_in, list_current)) 

    pool.close() 



def compare_lists(list_in, list_current): 
    for row_current in tqdm(list_current): 
     for row_in in list_in: 
      if row_in['order_code'] == row_current['order_code']: 

       #do more stuff here. 

       pass 

def get_current_list(): 
    try: 
     supplier = Supplier.objects.get(code='440040') 
     current_list = Product.objects.filter(supplier=supplier).values() 
     return current_list 
    except: 
     print('Error no products with supplier') 
     exit() 


def get_file(): 
    with zipfile.ZipFile(FILE_LOCATION + 'incoming/' + FILE_NAME, 'r') as zip: 
    with zip.open('228688 .csv') as csvfile: 
     reader = csv.DictReader(csvfile) 
     list_in = (list(reader)) 

     for row in tqdm(list_in): 
      row['order_code'] = row.pop('Ingram Part Number') 
      row['order_code'] = (row['order_code']).lstrip("0") 
      row['name'] = row.pop('Ingram Part Description') 
      row['description'] = row.pop('Material Long Description') 
      row['mpn'] = row.pop('Vendor Part Number') 
      row['gtin'] = row.pop('EANUPC Code') 
      row['nett_cost'] = row.pop('Customer Price') 
      row['retail_price'] = row.pop('Retail Price') 
      row['qty_at_supplier'] = row.pop('Available Quantity') 
      row['backorder_date'] = row.pop('Backlog ETA') 
      row['backorder_date'] = (row['backorder_date']) 
      row['backorder_qty'] = row.pop('Backlog Information') 

     zip.close() 
     #commented out for dev precess. 
     #os.rename(FILE_LOCATION + 'incoming/' + FILE_NAME, FILE_LOCATION + 'processed/' + FILE_NAME) 
     return list_in 

Answer 1

我曾經面對數據的加載速度緩慢的問題，我可以告訴你，我沒有，也許它可以幫助你不知何故，我通過執行調試模式，並試圖找出至極colomn導致了慢加載，每次我看到一個colomn導致的問題，我添加一個索引（在SGBD - >在我的情況下postgreSQL），它的工作。我希望你面臨同樣的問題，所以我的回答可以幫助你。

Answer 2

這是粗略的想法： 1，閱讀CSV時，用大熊貓作爲建議由@BearBrow到array_csv 2，從Django中obj的數據轉換成numpy的Arrary array_obj 3，不一一對它們進行比較，使用numpy的減法

compare_index = (array_csv[['cost',['quantity']]] - array[['cost',['quantity']]] == 0) 

4，發現更新列 obj_need_updated = array_obj [np.logic_any（compare_index [ '成本']，比較[ '數量']）]

然後使用Django批量更新https://github.com/aykut/django-bulk-update批量更新

希望這會給你提示加快你的代碼