Maybe the code's core rnn is not the most appropriate optimize code style?
In my machine (GPU: Titan X), modified 100x100 translated image,sensorBandwidth = 8, the code runs rather slow.
1.How long did you get 2% error? training 2 days?
2. can you bother to write a detailed explanation : how did you seperate 2 parts ,where is it?

this is the loss function:
J = tf.concat(values=[tf.log(p_y + SMALL_NUM) * (onehot_labels_placeholder), tf.log(p_loc + SMALL_NUM) * (R - no_grad_b)],axis=1) # axis = 1 acctually concat columns
p_loc is made by :
p_loc = gaussian_pdf(mean_locs, sampled_locs) # ?? mean_locs is not stop_gradient, but sampled_locs is stop_gradient
your code seems just stop_gradient sampled_locs and not stop_gradient in mean_locs?
your seperate 2 parts is not seen from the your code?

1. location network, baseline network : learn with gradients of reinforcement learning only.

2. glimpse network, core network : learn with gradients of supervised learning only.