我正在嘗試使用TFrecord文件來訓練張量流中的網絡。問題是它開始運行良好,但過了一段時間,它變得非常慢。即使GPU利用率在一段時間內也會達到0%。 我已經測量了迭代之間的時間,並且它明顯增加。 我已經在某處讀過這可能是因爲在訓練循環中向圖中添加操作,並且可以使用graph.finalize()來解決。使用TFrecords訓練變得越來越慢
我的代碼是這樣的:
self.inputMR_,self.CT_GT_ = read_and_decode_single_example("data.tfrecords")
self.inputMR, self.CT_GT = tf.train.shuffle_batch([self.inputMR_, self.CT_GT_], batch_size=self.batch_size, num_threads=2,
capacity=500*self.batch_size,min_after_dequeue=2000)
batch_size_tf = tf.shape(self.inputMR)[0] #variable batchsize so we can test here
self.train_phase = tf.placeholder(tf.bool, name='phase_train')
self.G = self.Network(self.inputMR,batch_size_tf)# create the network
self.g_loss=lp_loss(self.G, self.CT_GT, self.l_num, batch_size_tf)
print 'learning rate ',self.learning_rate
self.g_optim = tf.train.GradientDescentOptimizer(self.learning_rate).minimize(self.g_loss)
self.saver = tf.train.Saver()
然後,我有一個看起來像這樣的訓練階段:
def train(self, config):
init=tf.initialize_all_variables()
with tf.Session() as sess:
sess.run(init)
coord = tf.train.Coordinator()
threads=tf.train.start_queue_runners(sess=sess, coord=coord)
sess.graph.finalize()# **WHERE SHOULD I PUT THIS?**
try:
while not coord.should_stop():
_,loss_eval = sess.run([self.g_optim, self.g_loss],feed_dict={self.train_phase: True})
.....
except:
e = sys.exc_info()[0]
print "Exception !!!", e
finally:
coord.request_stop()
coord.join(threads)
sess.close()
當我加入grapgh.finalize,有一個exeption,上面寫着:類型'exceptions.RuntimeError' 任何人都可以向我解釋什麼是在訓練過程中使用TFrecord文件的正確方法,以及如何在QueueRunner執行過程中不使用干涉使用graph.finalize()?
完整的錯誤是:
File "main.py", line 37, in <module>
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "main.py", line 35, in main
gen_model.train(FLAGS)
File "/home/dongnie/Desktop/gan/TF_record_MR_CT/model.py", line 143, in train
self.global_step.assign(it).eval() # set and update(eval) global_step with index, i
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py", line 505, in assign
return state_ops.assign(self._variable, value, use_locking=use_locking)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_state_ops.py", line 45, in assign
use_locking=use_locking, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 490, in apply_op
preferred_dtype=default_dtype)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 657, in convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/constant_op.py", line 180, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/constant_op.py", line 167, in constant
attrs={"value": tensor_value, "dtype": dtype_value}, name=name).outputs[0]
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2337, in create_op
self._check_not_finalized()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2078, in _check_not_finalized
raise RuntimeError("Graph is finalized and cannot be modified.")
RuntimeError: Graph is finalized and cannot be modified.
通常情況下,您構建圖形,然後確定它,然後做你的第一個session.run調用。看到RuntimeError的完整堆棧跟蹤會很有幫助 –
感謝Yaroslav,我這樣做,使用QueueRunner時出現問題,然後graph.finalize導致錯誤。 如何打印完整的堆棧跟蹤? –
即複製粘貼所有打印的內容,而不僅僅是'exceptions.RuntimeError''部分 –