Lecture 6: Value Function Approximation by David Silver
Website: http://www0.cs.ucl.ac.uk/staff/D.Silver/web/Teaching.html
Videos: https://www.bilibili.com/video/av32149008/?p=6
-
So far we have represented value function by a lookup table
- Every state s has an entry V(s)
- Or every state-action pair s, a has an entry Q(s, a)
-
Problem with large MDPs:
- There are too many states and/or actions to store in memory
- It is too slow to learn the value of each state individually
-
Solution for large MDPs:
- Estimate value function with function approximation
- vˆ(s, w) ≈ vπ(s)
- or ˆq(s, a, w) ≈ qπ(s, a)
- Generalise from seen states to unseen states
- Update parameter w using MC or TD learning
- Estimate value function with function approximation
-
There are many function approximators, We consider differentiable function approximators
- Linear combinations of features
- Neural network
- Decision tree
- Nearest neighbour
- Fourier / wavelet bases
-
Furthermore, we require a training method that is suitable for non-stationary, non-iid data
- Gradient Descent
- Let J(w) be a differentiable function of parameter vector w
- Define the gradient of J(w)
- To find a local minimum of J(w)
- Adjust w in direction of -ve gradient