python中的鏈式記憶器

Question

我已經有一個記錄器，工作得很好。它使用pickle轉儲序列化輸入並創建MD5散列作爲鍵。函數結果非常大，並且存儲爲文件名爲MD5哈希的pickle文件。當我連續調用兩個memoized函數時，memoizer將加載第一個函數的輸出並將其傳遞給第二個函數。第二個函數將序列化它，創建MD5然後加載輸出。這裏是一個非常簡單的代碼：

@memoize 
def f(x): 
    ... 
    return y 

@memoize 
def g(x): 
    ... 
    return y 

y1 = f(x1) 
y2 = g(y1) 

y1從磁盤加載時f進行評估，然後當g評估它的序列化。是否有可能繞過這一步並將y1（即MD5哈希）的密鑰傳遞給g？如果g已經有此密鑰，它將從磁盤加載y2。如果不是，則「請求」完整y1以評估g。

編輯：

import cPickle as pickle 
import inspect 
import hashlib 

class memoize(object): 
    def __init__(self, func): 
     self.func = func 

    def __call__(self, *args, **kwargs): 
     arg = inspect.getargspec(self.func).args 
     file_name = self._get_key(*args, **kwargs) 
     try: 
      f = open(file_name, "r") 
      out = pickle.load(f) 
      f.close() 
     except: 
      out = self.func(*args, **kwargs) 
      f = open(file_name, "wb") 
      pickle.dump(out, f, 2) 
      f.close() 

     return out 

    def _arg_hash(self, *args, **kwargs): 
     _str = pickle.dumps(args, 2) + pickle.dumps(kwargs, 2) 
     return hashlib.md5(_str).hexdigest() 

    def _src_hash(self): 
     _src = inspect.getsource(self.func) 
     return hashlib.md5(_src).hexdigest() 

    def _get_key(self, *args, **kwargs): 
     arg = self._arg_hash(*args, **kwargs) 
     src = self._src_hash() 
     return src + '_' + arg + '.pkl' 

Answer 1

我認爲你可以做到這一點以自動的方式，但我一般認爲這是更好更明確一些「懶」的評價。因此，我將介紹一種爲您的記憶功能添加額外參數的方法：lazy。但是，而不是文件，鹹菜和MD5我會簡化傭工位：

# I use a dictionary as storage instead of files 
storage = {} 

# No md5, just hash 
def calculate_md5(obj): 
    print('calculating md5 of', obj) 
    return hash(obj) 

# create dictionary entry instead of pickling the data to a file 
def create_file(md5, data): 
    print('creating file for md5', md5) 
    storage[md5] = data 

# Load dictionary entry instead of unpickling a file 
def load_file(md5): 
    print('loading file with md5 of', md5) 
    return storage[md5] 

我使用自定義的類作爲中間目標：

class MemoizedObject(object): 
    def __init__(self, md5): 
     self.md5 = result_md5 

    def get_real_data(self): 
     print('load...') 
     return load_file(self.md5) 

    def __repr__(self): 
     return '{self.__class__.__name__}(md5={self.md5})'.format(self=self) 

最後我展示改變Memoize假設你的功能如果你打電話的功能，並在正確的位置指定懶你能避免負載（取儲存）文件

class Memoize(object): 
    def __init__(self, func): 
     self.func = func 
     # The md5 to md5 storage is needed to find the result file 
     # or result md5 for lazy evaluation. 
     self.md5_to_md5_storage = {} 

    def __call__(self, x, lazy=False): 
     # If the argument is a memoized object no need to 
     # calculcate the hash, we can just look it up. 
     if isinstance(x, MemoizedObject): 
      key = x.md5 
     else: 
      key = calculate_md5(x) 

     if lazy and key in self.md5_to_md5_storage: 
      # Check if the key is present in the md5 to md5 storage, otherwise 
      # we can't be lazy 
      return MemoizedObject(self.md5_to_md5_storage[key]) 
     elif not lazy and key in self.md5_to_md5_storage: 
      # Not lazy but we know the result 
      result = load_file(self.md5_to_md5_storage[key]) 
     else: 
      # Unknown argument 
      result = self.func(x) 
      result_md5 = calculate_md5(result) 
      create_file(result_md5, result) 
      self.md5_to_md5_storage[key] = result_md5 
     return result 

現在：只取一個參數

@Memoize 
def f(x): 
    return x+1 

@Memoize 
def g(x): 
    return x+2 

正常（第一）運行：

>>> x1 = 10 
>>> y1 = f(x1) 
calculating md5 of 10 
calculating md5 of 11 
creating file for md5 11 
>>> y2 = g(y1) 
calculating md5 of 11 
calculating md5 of 13 
creating file for md5 13 

沒有lazy：

>>> x1 = 10 
>>> y1 = f(x1) 
calculating md5 of 10 
loading file with md5 of 11 
>>> y2 = g(y1) 
calculating md5 of 11 
loading file with md5 of 13 

隨着lazy=True

>>> x1 = 10 
>>> y1 = f(x1, lazy=True) 
calculating md5 of 10 
>>> y2 = g(y1) 
loading file with md5 of 13 

的最後一個選項只計算的 「MD5」第一參數並加載最終結果的文件。這應該正是你想要的。