shakedzy / notebooks

Some notebooks

notebooks's Issues

issues with: Learning to Play Acrobot with Actor-Critic

Hi,
I have found one issue with 'Learning to Play Acrobot with Actor-Critic' notebook. Please check it out and let me know if I'm right:
You call learning method (actor_cost, critic_cost = ac.learn(sess)) with every step of game play - it means that update is not done in batches, although your networks (Actor and Critc) are build properly to update with batches of data (placeholders have batch axis).

Better Q Table Update

In your 'Q Learning and Deep Q Network' notebook you note that the algorithm, while right, didn't always have good values for some actions. This can be fixed by adding in alpha to the Q Table update:

    `q_table.loc[action,state_to_str(state)] = (1-alpha) * q_table.loc[action,state_to_str(state)] + alpha * (reward + gamma * next_state_max_q_value)`

With alpha = 0.1 and gamma = 0.95 you'll find the Q Table looks much better.

board: [0 0 0 0] predicted Q values: [4, 4, 4, 4] best action: 0 correct action? True
board: [0 0 0 1] predicted Q values: [3, 1, 3, 2] best action: 0 correct action? True
board: [0 0 1 0] predicted Q values: [3, 0, 2, 3] best action: 0 correct action? True
board: [0 0 1 1] predicted Q values: [2, 0, 0, 0] best action: 0 correct action? True
board: [0 1 0 0] predicted Q values: [3, 0, 3, 0] best action: 0 correct action? True
board: [0 1 0 1] predicted Q values: [2, 0, 0, 0] best action: 0 correct action? True
board: [0 1 1 0] predicted Q values: [2, 0, -1, 0] best action: 0 correct action? True
board: [0 1 1 1] predicted Q values: [0, 0, 0, 0] best action: 0 correct action? True
board: [1 0 0 0] predicted Q values: [2, 3, 3, 3] best action: 1 correct action? True
board: [1 0 0 1] predicted Q values: [1, 2, 2, 0] best action: 1 correct action? True
board: [1 0 1 0] predicted Q values: [1, 2, 1, 2] best action: 1 correct action? True
board: [1 0 1 1] predicted Q values: [-1, 1, 0, 0] best action: 1 correct action? True
board: [1 1 0 0] predicted Q values: [1, 1, 2, 2] best action: 2 correct action? True
board: [1 1 0 1] predicted Q values: [0, 0, 1, 0] best action: 2 correct action? True
board: [1 1 1 0] predicted Q values: [0, 0, 0, 1] best action: 3 correct action? True

With your current code, each update forgets what was learned before.

Or you could use the Temporal Difference equation in Josh Stern's code that you point to.