快速的方法來計算所有值出現在熊貓數據幀

Question

假設我有以下數據：

import pandas as pd 
import numpy as np 
import random 
from string import ascii_uppercase 

random.seed(100) 

n = 1000000 

# Create a bunch of factor data... throw some NaNs in there for good measure 
data = {letter: [random.choice(list(ascii_uppercase) + [np.nan]) for _ in range(n)] for letter in ascii_uppercase} 

df = pd.DataFrame(data) 

我想快速算在集合中的所有值的每個值的全球發生數據幀。

這工作：

from collections import Counter 
c = Counter([v for c in df for v in df[c].fillna(-999)]) 

但速度很慢：

%timeit Counter([v for c in df for v in df[c].fillna(-999)]) 
1 loop, best of 3: 4.12 s per loop 

我想這個功能可以通過使用一些大熊貓的馬力加快速度：

def quick_global_count(df, na_value=-999): 
    df = df.fillna(na_value) 
    # Get counts of each element for each column in the passed dataframe 
    group_bys = {c: df.groupby(c).size() for c in df} 
    # Stack each of the Series objects in `group_bys`... This is faster than reducing a bunch of dictionaries by keys 
    stacked = pd.concat([v for k, v in group_bys.items()]) 
    # Call `reset_index()` to access the index column, which indicates the factor level for each column in dataframe 
    # Then groupby and sum on that index to get global counts 
    global_counts = stacked.reset_index().groupby('index').sum() 
    return global_counts 

這絕對更快（以前方法的75％），但必須有更快的速度......

個

%timeit quick_global_count(df) 
10 loops, best of 3: 3.01 s per loop 

上述兩種方法的結果是相同的（與結果稍加修改通過quick_global_count返回）：

dict(c) == quick_global_count(df).to_dict()[0] 
True 

中有什麼數據框計算值的全球發生率的更快的方法？

Answer 1

方法＃1

嘛NumPy的伎倆將轉換爲數字（這就是NumPy的閃耀），只是讓bincount執行計數 -

a = df.fillna('[').values.astype(str).view(np.uint8) 
count = np.bincount(a.ravel())[65:-1] 

此作品爲單個字符。 np.bincount(a.ravel())包含所有字符的計數。

方法＃1S（超級充電）

上一頁方法有在字符串轉換的瓶頸：astype(str)。另外，fillna()是另一個表演停止。通過避開這些瓶頸，需要更多的欺騙手段來超額收費。現在，可以預先使用astype('S1')以強制所有內容爲單個字符。所以，單個字符保持放置，而NaN減少到只有一個字符'n'。這讓我們跳過fillna，因爲'n'的計數可以在稍後通過索引簡單地跳過。

因此，實現起來 - 在pandas-0.20.3

def app1S(df): 
    ar = df.values.astype('S1') 
    a = ar.view(np.uint8) 
    count = np.bincount(a.ravel())[65:65+26] 
    return count 

時序和numpy-1.13.3 -

In [3]: # Setup input 
    ...: random.seed(100) 
    ...: n = 1000000 
    ...: data = {letter: [random.choice(list(ascii_uppercase) + 
    ...:   [np.nan]) for _ in range(n)] for letter in ascii_uppercase} 
    ...: df = pd.DataFrame(data) 
    ...: 

# @Wen's soln 
In [4]: %timeit df.melt().value.value_counts() 
1 loop, best of 3: 2.5 s per loop 

# @andrew_reece's soln 
In [5]: %timeit df.apply(pd.value_counts).sum(axis=1) 
1 loop, best of 3: 2.14 s per loop 

# Super-charged one 
In [6]: %timeit app1S(df) 
1 loop, best of 3: 501 ms per loop 

---

一般情況下

我們也np.unique可以以覆蓋用於一般情況下（具有多於單個字符數據） -

unq, count = np.unique(df.fillna(-999), return_counts=1) 

Answer 2

df.apply(pd.value_counts).sum(axis=1) 

標杆：

# example data 
N = 10000000 
rownum = int(N/1000.) 
colnum = int(N/10000.) 

str_vals = ['A','B','C','D'] 
str_data = np.random.choice(str_vals, size=N).reshape(rownum, colnum) 
str_df = pd.DataFrame(str_data) 

num_vals = [1,2,3,4] 
num_data = np.random.choice(num_vals, size=N).reshape(rownum, colnum) 
num_df = pd.DataFrame(num_data) 

num_df.shape 
# (10000, 1000) 

%%timeit 
num_df.apply(pd.value_counts).sum(axis=1) 
# 1 loop, best of 3: 883 ms per loop 

%%timeit 
str_df.apply(pd.value_counts).sum(axis=1) 
# 1 loop, best of 3: 2.76 s per loop 

Answer 3

melt然後value_counts（PS，仍然不能與numpy溶液相比）

df.melt().value.value_counts() 

---

時間

%timeit df.melt().value.value_counts() 
100 loops, best of 3: 1.43 ms per loop 
%timeit c = Counter([v for c in df for v in df[c].fillna(-999)]) 
100 loops, best of 3: 5.23 ms per loop 
%timeit df.apply(pd.value_counts).sum() 
100 loops, best of 3: 18.5 ms per loop