enjeeneer / sutton_and_barto

Notes and solutions to exercises in Sutton and Barto's Reinforcement Learning textbook

Reinforcement Learning: An Introduction (2018) Notes

Notes and solutions to the exercises from Sutton and Barto's Reinforcement Learning: An Introduction.

sutton_and_barto's Issues

Wrong answer 2.1

If we have 2 actions and $\varepsilon=0.5$, the probability of selecting the greedy action is $0.5 + 0.25=0.75$. We add $0.25$ because we may select a random action with $\varepsilon=0.5$ which can be the greedy action itself. Since there are 2 actions, the "random" probability will be distributed equally $\frac{0.5}{2}=0.25$.

Equation 28

In the calculation of expected returns, if a =0, the equation becomes 0. I think it should be modified to a = 1

Exercise 8.4: Dyna-Q-new method cannot reach terminal state

When I run the script main.py in the Exercise 8.4, I noticed that the Dyna-Q-new method never terminates.

Upon inspection, I found that the agent gets stuck either next to the blocks or next to a wall.

sutton_and_barto/code/chapter8/ex_8.4/main.py

Line 55 in 8e99b48

# run dyna_new

To showcase this, I have pasted the $(A_{t}, S_{t+1})$ pairs for the first 500 steps below:

	action	next-state
1	➡	(5, 1)
2	➡	(6, 1)
3	➡	(7, 1)
4	⬇	(7, 1)
5	⬆	(7, 2)
6	⬅	(6, 2)
7	➡	(7, 2)
8	⬇	(7, 1)
9	⬆	(7, 2)
10	⬆	(7, 2)
11	⬅	(6, 2)
12	⬅	(5, 2)
13	⬇	(5, 1)
14	⬅	(4, 1)
15	➡	(5, 1)
16	⬅	(4, 1)
17	⬇	(4, 1)
18	⬅	(3, 1)
19	⬅	(2, 1)
20	⬆	(2, 2)
21	⬅	(1, 2)
22	⬆	(1, 3)
23	➡	(1, 3)
24	⬅	(1, 3)
25	⬆	(1, 4)
26	⬅	(1, 4)
27	⬅	(1, 4)
28	⬇	(1, 3)
29	⬆	(1, 4)
30	⬆	(1, 5)
31	⬇	(1, 4)
32	⬅	(1, 4)
33	⬆	(1, 5)
34	⬇	(1, 4)
35	⬆	(1, 5)
36	⬅	(1, 5)
37	⬆	(1, 6)
38	⬆	(1, 6)
39	➡	(2, 6)
40	⬇	(2, 5)
41	⬇	(2, 4)
42	➡	(3, 4)
43	⬅	(2, 4)
44	⬅	(1, 4)
45	⬅	(1, 4)
46	⬇	(1, 3)
47	⬆	(1, 4)
48	➡	(2, 4)
49	⬇	(2, 4)
50	⬇	(2, 4)
51	⬆	(2, 5)
52	⬅	(1, 5)
53	⬅	(1, 5)
54	⬇	(1, 4)
55	⬆	(1, 5)
56	➡	(2, 5)
57	⬅	(1, 5)
58	⬆	(1, 6)
59	➡	(2, 6)
60	⬇	(2, 5)
61	⬅	(1, 5)
62	⬆	(1, 6)
63	⬇	(1, 5)
64	⬅	(1, 5)
65	⬅	(1, 5)
66	➡	(2, 5)
67	⬇	(2, 4)
68	➡	(3, 4)
69	⬇	(3, 4)
70	⬅	(2, 4)
71	⬅	(1, 4)
72	➡	(2, 4)
73	⬇	(2, 4)
74	⬇	(2, 4)
75	➡	(3, 4)
76	➡	(4, 4)
77	➡	(5, 4)
78	⬆	(5, 5)
79	➡	(6, 5)
80	⬅	(5, 5)
81	➡	(6, 5)
82	⬅	(5, 5)
83	➡	(6, 5)
84	⬅	(5, 5)
85	➡	(6, 5)
86	⬅	(5, 5)
87	➡	(6, 5)
88	⬅	(5, 5)
89	➡	(6, 5)
90	⬅	(5, 5)
91	➡	(6, 5)
92	⬅	(5, 5)
93	➡	(6, 5)
94	⬅	(5, 5)
95	➡	(6, 5)
96	⬅	(5, 5)
97	➡	(6, 5)
98	⬅	(5, 5)
99	➡	(6, 5)
100	⬅	(5, 5)
101	➡	(6, 5)
102	⬅	(5, 5)
103	➡	(6, 5)
104	⬅	(5, 5)
105	➡	(6, 5)
106	⬅	(5, 5)
107	➡	(6, 5)
108	⬅	(5, 5)
109	➡	(6, 5)
110	⬅	(5, 5)
111	➡	(6, 5)
112	⬅	(5, 5)
113	➡	(6, 5)
114	⬅	(5, 5)
115	➡	(6, 5)
116	⬅	(5, 5)
117	➡	(6, 5)
118	⬅	(5, 5)
119	➡	(6, 5)
120	⬅	(5, 5)
121	➡	(6, 5)
122	⬅	(5, 5)
123	➡	(6, 5)
124	⬅	(5, 5)
125	➡	(6, 5)
126	⬅	(5, 5)
127	➡	(6, 5)
128	⬅	(5, 5)
129	➡	(6, 5)
130	⬅	(5, 5)
131	➡	(6, 5)
132	⬅	(5, 5)
133	➡	(6, 5)
134	⬅	(5, 5)
135	➡	(6, 5)
136	⬅	(5, 5)
137	➡	(6, 5)
138	⬅	(5, 5)
139	➡	(6, 5)
140	⬅	(5, 5)
141	➡	(6, 5)
142	⬅	(5, 5)
143	➡	(6, 5)
144	⬅	(5, 5)
145	➡	(6, 5)
146	⬅	(5, 5)
147	➡	(6, 5)
148	⬅	(5, 5)
149	➡	(6, 5)
150	⬅	(5, 5)
151	➡	(6, 5)
152	⬅	(5, 5)
153	➡	(6, 5)
154	⬅	(5, 5)
155	➡	(6, 5)
156	⬅	(5, 5)
157	➡	(6, 5)
158	⬅	(5, 5)
159	➡	(6, 5)
160	⬅	(5, 5)
161	➡	(6, 5)
162	⬅	(5, 5)
163	➡	(6, 5)
164	⬅	(5, 5)
165	➡	(6, 5)
166	⬅	(5, 5)
167	➡	(6, 5)
168	⬅	(5, 5)
169	➡	(6, 5)
170	⬅	(5, 5)
171	➡	(6, 5)
172	⬅	(5, 5)
173	➡	(6, 5)
174	⬅	(5, 5)
175	➡	(6, 5)
176	⬅	(5, 5)
177	➡	(6, 5)
178	⬅	(5, 5)
179	➡	(6, 5)
180	⬅	(5, 5)
181	➡	(6, 5)
182	⬅	(5, 5)
183	➡	(6, 5)
184	⬅	(5, 5)
185	➡	(6, 5)
186	⬅	(5, 5)
187	➡	(6, 5)
188	⬅	(5, 5)
189	➡	(6, 5)
190	⬅	(5, 5)
191	➡	(6, 5)
192	⬅	(5, 5)
193	➡	(6, 5)
194	⬅	(5, 5)
195	➡	(6, 5)
196	⬅	(5, 5)
197	➡	(6, 5)
198	⬅	(5, 5)
199	➡	(6, 5)
200	⬅	(5, 5)
201	➡	(6, 5)
202	⬅	(5, 5)
203	➡	(6, 5)
204	⬅	(5, 5)
205	➡	(6, 5)
206	⬅	(5, 5)
207	➡	(6, 5)
208	⬅	(5, 5)
209	➡	(6, 5)
210	⬅	(5, 5)
211	➡	(6, 5)
212	⬅	(5, 5)
213	➡	(6, 5)
214	⬅	(5, 5)
215	➡	(6, 5)
216	⬅	(5, 5)
217	➡	(6, 5)
218	⬅	(5, 5)
219	➡	(6, 5)
220	⬅	(5, 5)
221	➡	(6, 5)
222	⬅	(5, 5)
223	➡	(6, 5)
224	⬅	(5, 5)
225	➡	(6, 5)
226	⬅	(5, 5)
227	➡	(6, 5)
228	⬅	(5, 5)
229	➡	(6, 5)
230	⬅	(5, 5)
231	➡	(6, 5)
232	⬅	(5, 5)
233	➡	(6, 5)
234	⬅	(5, 5)
235	➡	(6, 5)
236	⬅	(5, 5)
237	➡	(6, 5)
238	⬅	(5, 5)
239	➡	(6, 5)
240	⬅	(5, 5)
241	➡	(6, 5)
242	⬅	(5, 5)
243	➡	(6, 5)
244	⬅	(5, 5)
245	➡	(6, 5)
246	⬅	(5, 5)
247	➡	(6, 5)
248	⬅	(5, 5)
249	➡	(6, 5)
250	⬅	(5, 5)
251	➡	(6, 5)
252	⬅	(5, 5)
253	➡	(6, 5)
254	⬅	(5, 5)
255	➡	(6, 5)
256	⬅	(5, 5)
257	➡	(6, 5)
258	⬅	(5, 5)
259	➡	(6, 5)
260	⬅	(5, 5)
261	➡	(6, 5)
262	⬅	(5, 5)
263	➡	(6, 5)
264	⬅	(5, 5)
265	➡	(6, 5)
266	⬅	(5, 5)
267	➡	(6, 5)
268	⬅	(5, 5)
269	➡	(6, 5)
270	⬅	(5, 5)
271	➡	(6, 5)
272	⬅	(5, 5)
273	➡	(6, 5)
274	⬅	(5, 5)
275	➡	(6, 5)
276	⬅	(5, 5)
277	➡	(6, 5)
278	⬅	(5, 5)
279	➡	(6, 5)
280	⬅	(5, 5)
281	➡	(6, 5)
282	⬅	(5, 5)
283	➡	(6, 5)
284	⬅	(5, 5)
285	➡	(6, 5)
286	⬅	(5, 5)
287	➡	(6, 5)
288	⬅	(5, 5)
289	➡	(6, 5)
290	⬅	(5, 5)
291	➡	(6, 5)
292	⬅	(5, 5)
293	➡	(6, 5)
294	⬅	(5, 5)
295	➡	(6, 5)
296	⬅	(5, 5)
297	➡	(6, 5)
298	⬅	(5, 5)
299	➡	(6, 5)
300	⬅	(5, 5)
301	➡	(6, 5)
302	⬅	(5, 5)
303	➡	(6, 5)
304	⬅	(5, 5)
305	➡	(6, 5)
306	⬅	(5, 5)
307	➡	(6, 5)
308	⬅	(5, 5)
309	➡	(6, 5)
310	⬅	(5, 5)
311	➡	(6, 5)
312	⬅	(5, 5)
313	➡	(6, 5)
314	⬅	(5, 5)
315	➡	(6, 5)
316	⬅	(5, 5)
317	➡	(6, 5)
318	⬅	(5, 5)
319	➡	(6, 5)
320	⬅	(5, 5)
321	➡	(6, 5)
322	⬅	(5, 5)
323	➡	(6, 5)
324	⬅	(5, 5)
325	➡	(6, 5)
326	⬅	(5, 5)
327	➡	(6, 5)
328	⬅	(5, 5)
329	➡	(6, 5)
330	⬅	(5, 5)
331	➡	(6, 5)
332	⬅	(5, 5)
333	➡	(6, 5)
334	⬅	(5, 5)
335	➡	(6, 5)
336	⬅	(5, 5)
337	➡	(6, 5)
338	⬅	(5, 5)
339	➡	(6, 5)
340	⬅	(5, 5)
341	➡	(6, 5)
342	⬅	(5, 5)
343	➡	(6, 5)
344	⬅	(5, 5)
345	➡	(6, 5)
346	⬅	(5, 5)
347	➡	(6, 5)
348	⬅	(5, 5)
349	➡	(6, 5)
350	⬅	(5, 5)
351	➡	(6, 5)
352	⬅	(5, 5)
353	➡	(6, 5)
354	⬅	(5, 5)
355	➡	(6, 5)
356	⬅	(5, 5)
357	➡	(6, 5)
358	⬅	(5, 5)
359	➡	(6, 5)
360	⬅	(5, 5)
361	➡	(6, 5)
362	⬅	(5, 5)
363	➡	(6, 5)
364	⬅	(5, 5)
365	➡	(6, 5)
366	⬅	(5, 5)
367	➡	(6, 5)
368	⬅	(5, 5)
369	➡	(6, 5)
370	⬅	(5, 5)
371	➡	(6, 5)
372	⬅	(5, 5)
373	➡	(6, 5)
374	⬅	(5, 5)
375	➡	(6, 5)
376	⬅	(5, 5)
377	➡	(6, 5)
378	⬅	(5, 5)
379	➡	(6, 5)
380	⬅	(5, 5)
381	➡	(6, 5)
382	⬅	(5, 5)
383	➡	(6, 5)
384	⬅	(5, 5)
385	➡	(6, 5)
386	⬅	(5, 5)
387	➡	(6, 5)
388	⬅	(5, 5)
389	➡	(6, 5)
390	⬅	(5, 5)
391	➡	(6, 5)
392	⬅	(5, 5)
393	➡	(6, 5)
394	⬅	(5, 5)
395	➡	(6, 5)
396	⬅	(5, 5)
397	➡	(6, 5)
398	⬅	(5, 5)
399	➡	(6, 5)
400	⬅	(5, 5)
401	➡	(6, 5)
402	⬅	(5, 5)
403	➡	(6, 5)
404	⬅	(5, 5)
405	➡	(6, 5)
406	⬅	(5, 5)
407	➡	(6, 5)
408	⬅	(5, 5)
409	➡	(6, 5)
410	⬅	(5, 5)
411	➡	(6, 5)
412	⬅	(5, 5)
413	➡	(6, 5)
414	⬅	(5, 5)
415	➡	(6, 5)
416	⬅	(5, 5)
417	➡	(6, 5)
418	⬅	(5, 5)
419	➡	(6, 5)
420	⬅	(5, 5)
421	➡	(6, 5)
422	⬅	(5, 5)
423	➡	(6, 5)
424	⬅	(5, 5)
425	➡	(6, 5)
426	⬅	(5, 5)
427	➡	(6, 5)
428	⬅	(5, 5)
429	➡	(6, 5)
430	⬅	(5, 5)
431	➡	(6, 5)
432	⬅	(5, 5)
433	➡	(6, 5)
434	⬅	(5, 5)
435	➡	(6, 5)
436	⬅	(5, 5)
437	➡	(6, 5)
438	⬅	(5, 5)
439	➡	(6, 5)
440	⬅	(5, 5)
441	➡	(6, 5)
442	⬅	(5, 5)
443	➡	(6, 5)
444	⬅	(5, 5)
445	➡	(6, 5)
446	⬅	(5, 5)
447	➡	(6, 5)
448	⬅	(5, 5)
449	➡	(6, 5)
450	⬅	(5, 5)
451	➡	(6, 5)
452	⬅	(5, 5)
453	➡	(6, 5)
454	⬅	(5, 5)
455	➡	(6, 5)
456	⬅	(5, 5)
457	➡	(6, 5)
458	⬅	(5, 5)
459	➡	(6, 5)
460	⬅	(5, 5)
461	➡	(6, 5)
462	⬅	(5, 5)
463	➡	(6, 5)
464	⬅	(5, 5)
465	➡	(6, 5)
466	⬅	(5, 5)
467	➡	(6, 5)
468	⬅	(5, 5)
469	➡	(6, 5)
470	⬅	(5, 5)
471	➡	(6, 5)
472	⬅	(5, 5)
473	➡	(6, 5)
474	⬅	(5, 5)
475	➡	(6, 5)
476	⬅	(5, 5)
477	➡	(6, 5)
478	⬅	(5, 5)
479	➡	(6, 5)
480	⬅	(5, 5)
481	➡	(6, 5)
482	⬅	(5, 5)
483	➡	(6, 5)
484	⬅	(5, 5)
485	➡	(6, 5)
486	⬅	(5, 5)
487	➡	(6, 5)
488	⬅	(5, 5)
489	➡	(6, 5)
490	⬅	(5, 5)
491	➡	(6, 5)
492	⬅	(5, 5)
493	➡	(6, 5)
494	⬅	(5, 5)
495	➡	(6, 5)
496	⬅	(5, 5)
497	➡	(6, 5)
498	⬅	(5, 5)
499	➡	(6, 5)
500	⬅	(5, 5)