如何避免給定Convnet中的過度配合

Question

我正在嘗試實現CNN網絡進行句子分類;我正在嘗試遵循paper中提出的架構。我正在使用Keras（張量流）。以下是我的型號總結：

____________________________________________________________________________________________________ 
Layer (type)      Output Shape   Param #  Connected to      
==================================================================================================== 
input_4 (InputLayer)    (None, 56)   0            
____________________________________________________________________________________________________ 
embedding (Embedding)   (None, 56, 300)  6510000          
____________________________________________________________________________________________________ 
dropout_7 (Dropout)    (None, 56, 300)  0            
____________________________________________________________________________________________________ 
conv1d_10 (Conv1D)    (None, 54, 100)  90100           
____________________________________________________________________________________________________ 
conv1d_11 (Conv1D)    (None, 53, 100)  120100          
____________________________________________________________________________________________________ 
conv1d_12 (Conv1D)    (None, 52, 100)  150100          
____________________________________________________________________________________________________ 
max_pooling1d_10 (MaxPooling1D) (None, 27, 100)  0            
____________________________________________________________________________________________________ 
max_pooling1d_11 (MaxPooling1D) (None, 26, 100)  0            
____________________________________________________________________________________________________ 
max_pooling1d_12 (MaxPooling1D) (None, 26, 100)  0            
____________________________________________________________________________________________________ 
flatten_10 (Flatten)    (None, 2700)   0            
____________________________________________________________________________________________________ 
flatten_11 (Flatten)    (None, 2600)   0            
____________________________________________________________________________________________________ 
flatten_12 (Flatten)    (None, 2600)   0            
____________________________________________________________________________________________________ 
concatenate_4 (Concatenate)  (None, 7900)   0            
____________________________________________________________________________________________________ 
dropout_8 (Dropout)    (None, 7900)   0            
____________________________________________________________________________________________________ 
dense_7 (Dense)     (None, 50)   395050          
____________________________________________________________________________________________________ 
dense_8 (Dense)     (None, 5)    255           
==================================================================================================== 
Total params: 7,265,605.0 
Trainable params: 7,265,605.0 
Non-trainable params: 0.0 

由於給定的架構，我遇到嚴重的過度配合。以下是我的結果：

我無法理解過度配合的原因是什麼，請建議我對架構進行一些更改以避免這種情況。如果您需要更多信息，請讓我知道。

的源代碼：

if model_type in ['CNN-non-static', 'CNN-static']: 
    embedding_wts = train_word2vec(np.vstack((x_train, x_test, x_valid)), 
            ind_to_wrd, num_features = embedding_dim) 
    if model_type == 'CNN-static': 
     x_train = embedding_wts[0][x_train] 
     x_test = embedding_wts[0][x_test] 
     x_valid = embedding_wts[0][x_valid] 

elif model_type == 'CNN-rand': 
    embedding_wts = None 

else: 
    raise ValueError("Unknown model type") 

batch_size = 50 
filter_sizes = [3,4,5] 
num_filters = 75 
dropout_prob = (0.5, 0.8) 
hidden_dims = 50 

l2_reg = 0.3 

# Deciding dimension of input based on the model 
input_shape = (max_sent_len, embedding_dim) if model_type == "CNN-static" else (max_sent_len,) 
model_input = Input(shape = input_shape) 

# Static model do not have embedding layer 
if model_type == "CNN-static": 
    z = Dropout(dropout_prob[0])(model_input) 
else: 
    z = Embedding(vocab_size, embedding_dim, input_length = max_sent_len, name="embedding")(model_input) 
    z = Dropout(dropout_prob[0])(z) 

# Convolution layers 
z1 = Conv1D( filters=num_filters, kernel_size=3, 
       padding="valid", activation="relu", 
       strides=1)(z) 
z1 = MaxPooling1D(pool_size=2)(z1) 
z1 = Flatten()(z1) 

z2 = Conv1D( filters=num_filters, kernel_size=4, 
       padding="valid", activation="relu", 
       strides=1)(z) 
z2 = MaxPooling1D(pool_size=2)(z2) 
z2 = Flatten()(z2) 

z3 = Conv1D( filters=num_filters, kernel_size=5, 
       padding="valid", activation="relu", 
       strides=1)(z) 
z3 = MaxPooling1D(pool_size=2)(z3) 
z3 = Flatten()(z3) 

# Concatenate the output of all convolution layers 
z = Concatenate()([z1, z2, z3]) 
z = Dropout(dropout_prob[1])(z) 

# Dense(64, input_dim=64, kernel_regularizer=regularizers.l2(0.01), activity_regularizer=regularizers.l1(0.01)) 

z = Dense(hidden_dims, activation="relu", kernel_regularizer=regularizers.l2(0.01))(z) 
model_output = Dense(N_category, activation="sigmoid")(z) 

model = Model(model_input, model_output) 
model.compile(loss="categorical_crossentropy", optimizer=optimizers.Adadelta(lr=1, decay=0.005), metrics=["accuracy"]) 
model.summary() 

Answer 1

從模型摘要看，您正在處理具有5個輸出類的多類問題。對於多級設置softmax很適合，而不是sigmoid。我想這是表現非常差的主要原因。關於多類問題，如果使用sigmoid和categorical_crossentropy設置網絡，那麼在訓練過程中，通過簡單預測所有類爲1可以達到100％的訓練準確性，但是這種假設在測試數據上失敗了。

如果你正在處理二元分類問題，那麼binary_crossentropy是丟失函數的好選擇。

其他一些有用的建議可能會有幫助，但這一切都取決於您的建模假設。

• 我想你是遵循example中解釋的類似方法。但是如果你的訓練數據很小，最好的做法是使用預訓練詞嵌入並允許微調。您可以找到預訓練嵌入初始化示例here。但通過在嵌入層中設置trainable=True可以進行微調。

• 在提到的論文中，他從每個濾鏡pool_size = max_sent_len-filter_size+1中提取單個特徵。這將只從整句中提取單個重要特徵。此設置將允許您降低模型複雜性。很好的解釋在參考：here

• 在同樣的方法，他們只使用單一的密集層，你可以刪除你的一個隱藏的密集層。輸出特徵的數量從拉動層退出很小（將等於濾波器的數量，每個濾波器將僅輸出一個特徵）

Answer 2

不看在深度模型我會說，你應該儘量不訓練的嵌入和重用下載的矩陣之一。即使你削減了這一點，你仍然有幾乎10倍的參數，因爲你有數據點，所以模型必然是過度擬合。

它應該是相反的。對於800k參數，你應該有8M數據點。

如果你看看圖，驗證鬆散在第一個（少數）時代下降，然後上升，這是另一個沒有足夠數據的跡象。