training speed problem about tffrcnn

[xinghedyc]

I see this
if iter >= cfg.TRAIN.STEPSIZE:
sess.run(tf.assign(lr, cfg.TRAIN.LEARNING_RATE * cfg.TRAIN.GAMMA))
else:
sess.run(tf.assign(lr, cfg.TRAIN.LEARNING_RATE))
this op execute every iter, I try to move it outside the loop, it seems speed up the training.

[CharlesShang]

I have the same problem with you, my GPU is GTX 980, runs pretty fast at the beginning, but gets worse after saveral epoches.
I tried to profile the process to check how much the loading data module costed, (see ./experiments/profiling/profile.png) but still got no idea.

[CharlesShang]

BTW, I encountered the same problems when running other tasks with tf.
This is really weird.

[xinghedyc]

@CharlesShang Hi , I move this lines outside the loop: at line 176 in lib/fast_rcnn/train.py
if iter >= cfg.TRAIN.STEPSIZE:
sess.run(tf.assign(lr, cfg.TRAIN.LEARNING_RATE * cfg.TRAIN.GAMMA))
else:
sess.run(tf.assign(lr, cfg.TRAIN.LEARNING_RATE))
And it get faster than before and the problem that the training speed goes down seems to disappear.
You can have a try?

[CharlesShang]

definitely
Thank you for your suggestions

[xinghedyc]

You're welcome!
Hope to help you!

[CharlesShang]

I'm closing this issue.
If we find out the speed problems still there unsolved, we'll reopen it.