Noob question about backprop implementation about micrograd

[Richard004]

Hi, this is my guess. It would definitely work.
But at a certain moment, the depth of the graph can be so large that one risks stack overflow.
It feels like a tradeoff between simplicity and robustness of the example. But I am not sure if that was the real concern.

My concern was that we are creating the order with each call of the back propagation, while it typically remains constant over the whole optimization process. So the order could easily be cached somewhere. But that is just performance problem, I undestand that this was supposed to be explanation of the principle.

And it is a great one! ❤

[deermichel]

Great question, I was wondering the same 👍🏽

[t0lya]

Hi, this is my guess. It would definitely work. But at a certain moment, the depth of the graph can be so large that one risks stack overflow. It feels like a tradeoff between simplicity and robustness of the example. But I am not sure if that was the real concern.

My concern was that we are creating the order with each call of the back propagation, while it typically remains constant over the whole optimization process. So the order could easily be cached somewhere. But that is just performance problem, I undestand that this was supposed to be explanation of the principle.

And it is a great one! ❤

Topological sort in micrograd is implemented using recursion, so the stack overflow concern is same as when calling backward() recursively

[t0lya]

The graph is not necessarily a tree, it can have diamonds, see tinygrad/tinygrad#165 (comment)

Checking visited nodes during topological sort prevents repeat backward calls on same node