當改變輸入數據時，Tensorflow模型沒有訓練

Question

我正在嘗試使用張量流來訓練使用交通標誌圖像的神經網絡（LeNet）。我想檢查預處理技術對nn性能的影響。因此，我預處理了圖像，並將結果（訓練圖像，驗證圖像，testimages，最終testimages）作爲一個字典存儲。

然後我試圖遍歷該字典，然後使用tensorflow的訓練和驗證操作如下

import tensorflow as tf 
from sklearn.utils import shuffle 


output_data = [] 
EPOCHS = 5 
BATCH_SIZE = 128 
rate = 0.0005 

for key in finalInputdata.keys(): 
    for procTypes in range(0,(len(finalInputdata[key]))): 
     if np.shape(finalInputdata[key][procTypes][0]) !=(): 
      X_train = finalInputdata[key][procTypes][0] 
      X_valid = finalInputdata[key][procTypes][1] 
      X_test = finalInputdata[key][procTypes][2] 
      X_finaltest = finalInputdata[key][procTypes][3] 


      x = tf.placeholder(tf.float32, (None, 32, 32,np.shape(X_train)[-1])) 
      y = tf.placeholder(tf.int32, (None)) 
      one_hot_y = tf.one_hot(y,43) 

      # Tensor Operations 
      logits = LeNet(x,np.shape(X_train)[-1]) 

      cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits,one_hot_y) 
      softmax_probability = tf.nn.softmax(logits) 

      loss_operation = tf.reduce_mean(cross_entropy) 
      optimizer = tf.train.AdamOptimizer(learning_rate=rate) 
      training_operation = optimizer.minimize(loss_operation) 
      correct_prediction = tf.equal(tf.argmax(logits,1), tf.argmax(one_hot_y,1)) 
      accuracy_operation = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)) 


       # Pipeline for training and evaluation 

      sess = tf.InteractiveSession() 

      sess.run(tf.global_variables_initializer()) 
      num_examples = len(X_train) 

      print("Training on %s images processed as %s" %(key,dict_fornames['proctypes'][procTypes])) 
      print() 
      for i in range(EPOCHS): 
       X_train, y_train = shuffle(X_train, y_train) 
       for offset in range(0, num_examples, BATCH_SIZE): 
        end = offset + BATCH_SIZE 
        batch_x, batch_y = X_train[offset:end], y_train[offset:end] 
        sess.run(training_operation, feed_dict = {x: batch_x, y: batch_y}) 

       training_accuracy = evaluate(X_train,y_train) 

       validation_accuracy = evaluate(X_valid, y_valid) 

       testing_accuracy = evaluate(X_test, y_test) 

       final_accuracy = evaluate(X_finaltest, y_finalTest) 

       print("EPOCH {} ...".format(i+1)) 
       print("Training Accuracy = {:.3f}".format(training_accuracy)) 
       print("Validation Accuracy = {:.3f}".format(validation_accuracy)) 
       print() 
       output_data.append({'EPOCHS':EPOCHS, 'LearningRate':rate, 'ImageType': 'RGB',\ 
            'PreprocType': dict_fornames['proctypes'][0],\ 
            'TrainingAccuracy':training_accuracy, 'ValidationAccuracy':validation_accuracy, \ 
            'TestingAccuracy': testing_accuracy}) 


      sess.close() 

的評價功能如下

def evaluate(X_data, y_data): 
    num_examples = len(X_data) 
    total_accuracy = 0 
    sess = tf.get_default_session() 
    for offset in range(0,num_examples, BATCH_SIZE): 
     batch_x, batch_y = X_data[offset:offset+BATCH_SIZE], y_data[offset:offset+BATCH_SIZE] 
     accuracy = sess.run(accuracy_operation, feed_dict = {x:batch_x, y:batch_y}) 
     total_accuracy += (accuracy * len(batch_x)) 
    return total_accuracy/num_examples 

一旦我執行程序，它對數據集的第一次迭代有效，但從第二次迭代開始，網絡不會訓練並繼續爲所有其他迭代執行此操作。

Training on RGB images processed as Original 

EPOCH 1 ... 
Training Accuracy = 0.525 
Validation Accuracy = 0.474 

EPOCH 2 ... 
Training Accuracy = 0.763 
Validation Accuracy = 0.682 

EPOCH 3 ... 
Training Accuracy = 0.844 
Validation Accuracy = 0.723 

EPOCH 4 ... 
Training Accuracy = 0.888 
Validation Accuracy = 0.779 

EPOCH 5 ... 
Training Accuracy = 0.913 
Validation Accuracy = 0.795 

Training on RGB images processed as Mean Subtracted Data 

EPOCH 1 ... 
Training Accuracy = 0.056 
Validation Accuracy = 0.057 

EPOCH 2 ... 
Training Accuracy = 0.057 
Validation Accuracy = 0.057 

EPOCH 3 ... 
Training Accuracy = 0.057 
Validation Accuracy = 0.056 

EPOCH 4 ... 
Training Accuracy = 0.058 
Validation Accuracy = 0.056 

EPOCH 5 ... 
Training Accuracy = 0.058 
Validation Accuracy = 0.058 

Training on RGB images processed as Normalized Data 

EPOCH 1 ... 
Training Accuracy = 0.058 
Validation Accuracy = 0.054 

EPOCH 2 ... 
Training Accuracy = 0.058 
Validation Accuracy = 0.054 

EPOCH 3 ... 
Training Accuracy = 0.058 
Validation Accuracy = 0.054 

EPOCH 4 ... 
Training Accuracy = 0.058 
Validation Accuracy = 0.054 

EPOCH 5 ... 
Training Accuracy = 0.058 
Validation Accuracy = 0.054 

但是，如果我重新啓動內核並使用任何數據類型（任何迭代），它都可以工作。我發現我必須清除圖表或者爲多個數據類型運行多個會話，但是我不清楚如何實現這一點。我嘗試使用tf.reset_default_graph()，但似乎沒有任何效果。有人能指出我正確的方向嗎？

感謝

Answer 1

你可能想使之與被歸爲零均值，方差其送入網絡之前的數據，例如嘗試通過縮放圖像到-1..1範圍;那說，0..1範圍大多也聽起來理智。取決於網絡中使用的激活，數值範圍可能會造成所有差異：例如，對於低於零的輸入，ReLUs將消失，當值低於-4或高於+4並且tanh激活錯過半數時，S形開始飽和它們的值範圍如果沒有任何值低於0 - 如果值範圍太大，漸變也可能爆炸，從而無法完全訓練。從this paper，作者似乎減去（批）圖像的平均值，而不是數值範圍的平均值。

你可以嘗試使用小學習速率以及（雖然就個人而言，我通常開始試驗周圍0.0001亞當）。

至於你的多個會話部分的問題：它是目前在你的代碼中實現的方式，你基本上是混亂的默認圖形。通過調用

for key in finalInputdata.keys(): 
    for procTypes in range(0,(len(finalInputdata[key]))): 
     if np.shape(finalInputdata[key][procTypes][0]) !=(): 

    # ... 

    x = tf.placeholder(tf.float32, (None, 32, 32,np.shape(X_train)[-1])) 
    y = tf.placeholder(tf.int32, (None)) 
    one_hot_y = tf.one_hot(y,43) 

    # Tensor Operations 
    logits = LeNet(x,np.shape(X_train)[-1]) 

    # ... etc ... 

要創建len(finalInputdata) * NLeNet的不同情況下，都在默認的圖形。當變量在網絡中被內部重用時，這可能是一個問題。

如果你想重置默認圖，以嘗試不同的超參數，儘量

for key in finalInputdata.keys(): 
    for procTypes in range(0,(len(finalInputdata[key]))): 

     tf.reset_default_graph() 
     # define the graph 

     sess = tf.InteractiveSession() 
     # train 

，但它可能是更好的顯式地創建圖形和會話像這樣：

for key in finalInputdata.keys(): 
    for procTypes in range(0,(len(finalInputdata[key]))): 

     with tf.Graph().as_default() as graph: 
      # define the graph 

     with tf.Session(graph=graph) as sess: 
      # train 

您可以直接使用sess參考，而不是調用sess = tf.get_default_session()。

我還發現，在網絡上迭代時，Jupyter內核和GPU啓用的TensorFlow不能很好地一起玩，有時會遇到內存不足錯誤或徹底崩潰瀏覽器選項卡。