熊貓：將字段值設置爲由字典值設置的限制

Question

如何刪除iterrows()？這可以用numpy或熊貓更快地完成嗎？

import pandas as pd 
import numpy as np 
df = pd.DataFrame({'A': 'foo bar foo bar foo bar foo foo'.split(), 
        'B': 'one one two three two two one three'.split(), 
        'C': np.arange(8)*0 }) 
print(df) 
#  A  B C 
# 0 foo one 0 
# 1 bar one 0 
# 2 foo two 0 
# 3 bar three 0 
# 4 foo two 0 
# 5 bar two 0 
# 6 foo one 0 
# 7 foo three 0 

selDict = {"foo":2, "bar":3} 

這工作：

for i, r in df.iterrows(): 
    if selDict[r["A"]] > 0: 
     selDict[r["A"]] -=1   
     df.set_value(i, 'C', 1) 

    print df 
#  A  B C 
# 0 foo one 1 
# 1 bar one 1 
# 2 foo two 1 
# 3 bar three 1 
# 4 foo two 0 
# 5 bar two 1 
# 6 foo one 0 
# 7 foo three 0 

Answer 1

這裏有一個方法 -

1）輔助功能：

def argsort_unique(idx): 
    # Original idea : http://stackoverflow.com/a/41242285/3293881 by @Andras 
    n = idx.size 
    sidx = np.empty(n,dtype=int) 
    sidx[idx] = np.arange(n) 
    return sidx 

def get_bin_arr(grplens, stop1_idx): 
    count_stops_corr = np.minimum(stop1_idx, grplens) 

    limsc = np.maximum(grplens, count_stops_corr) 
    L = limsc.sum() 

    starts = np.r_[0,limsc[:-1].cumsum()] 

    shift_arr = np.zeros(L,dtype=int) 
    stops = starts + count_stops_corr 
    stops = stops[stops<L] 

    shift_arr[starts] += 1 
    shift_arr[stops] -= 1 
    bin_arr = shift_arr.cumsum() 
    return bin_arr 

可能更快的替代方案有糊塗的切片爲主的輔助功能：

def get_bin_arr(grplens, stop1_idx): 
    stop1_idx_corr = np.minimum(stop1_idx, grplens)  
    clens = grplens.cumsum() 
    out = np.zeros(clens[-1],dtype=int)  
    out[:stop1_idx_corr[0]] = 1 
    for i,j in zip(clens[:-1], clens[:-1] + stop1_idx_corr[1:]): 
     out[i:j] = 1 
    return out 

2）主要功能：

def out_C(A, selDict): 
    k = np.array(selDict.keys()) 
    v = np.array(selDict.values()) 
    unq, C = np.unique(A, return_counts=1) 
    sidx3 = np.searchsorted(unq, k) 
    lims = np.zeros(len(unq),dtype=int) 
    lims[sidx3] = v 
    bin_arr = get_bin_arr(C, lims) 
    sidx2 = A.argsort() 
    out = bin_arr[argsort_unique(sidx2)]  
    return out 

樣品試驗 -

原始的方法：

def org_app(df, selDict): 
    df['C'] = 0 
    d = selDict.copy()  
    for i, r in df.iterrows(): 
     if d[r["A"]] > 0: 
      d[r["A"]] -=1   
      df.set_value(i, 'C', 1) 
    return df 

案例＃1：

>>> df = pd.DataFrame({'A': 'foo bar foo bar res foo bar res foo foo res'.split()}) 
>>> selDict = {"foo":2, "bar":3, "res":1} 
>>> org_app(df, selDict) 
     A C 
0 foo 1 
1 bar 1 
2 foo 1 
3 bar 1 
4 res 1 
5 foo 0 
6 bar 1 
7 res 0 
8 foo 0 
9 foo 0 
10 res 0 
>>> out_C(df.A.values, selDict) 
array([1, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0]) 

案例＃2：

>>> selDict = {"foo":20, "bar":30, "res":10} 
>>> org_app(df, selDict) 
     A C 
0 foo 1 
1 bar 1 
2 foo 1 
3 bar 1 
4 res 1 
5 foo 1 
6 bar 1 
7 res 1 
8 foo 1 
9 foo 1 
10 res 1 
>>> out_C(df.A.values, selDict) 
array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) 

Answer 2

如果我理解正確的話，你可以使用cumcount：

df['C'] = (df.groupby('A').cumcount() < df['A'].map(selDict)).astype('int') 

df 
Out: 
    A  B C 
0 foo one 1 
1 bar one 1 
2 foo two 1 
3 bar three 1 
4 foo two 0 
5 bar two 1 
6 foo one 0 
7 foo three 0 

Answer 3

scipy.stats.rankdata可以幫助在這裏。爲了得到它的桶中的每個元素的排名，我們採取了「分」和「有序」的方法之間的區別：

>>> from scipy.stats import rankdata as rd 
>>> rd(df.A, 'ordinal') - rd(df.A, 'min') 
array([0, 0, 1, 1, 2, 2, 3, 4]) 

然後，我們只是比較df.A.map(selDict)：

df.C = (rd(df.A, 'ordinal') - rd(df.A, 'min') < df.A.map(selDict)).astype(int) 

這可能會有點低效（調用rankdata兩次），但在scipy中使用優化的例程應該可以彌補這一點。

如果您不能使用SciPy的，你可以使用unique和bincount爲「最小」的方法使用重複argsort()爲「序」的方法和我的解決方案：

>>> _, v = np.unique(df.A, return_inverse=True) 
>>> df.A.argsort().argsort() - (np.cumsum(np.concatenate(([0], np.bincount(v)))))[v] 
0 0 
1 0 
2 1 
3 1 
4 2 
5 2 
6 3 
7 4 
Name: A, dtype: int64 

然後比較df.A.map(selDict)如上。