Investigate Changing Input Current Delay and Integration about norse

[LennardBo]

To some degree I understand @Jegp 's viewpoint that this behaviour is intended to simulate the neuron's dynamics. That's why I introduced this issue as a discussion rather than an issue.

After reading the merge request I understand this might ahve been mostly motivated by the recurrent layers and their consistency. However, for straight feed-forward networks this change is really important. Many SNN papers such as this and this mainly concern themselves with a short sequence length to reduce computational overhead. With the current implementation, the input lag actually makes implementing a comparably fast network impossible.

I'm very thankful for @cpehle to take my topic to heart and writing this update. I'm looking forward to future versions of Norse containg these changes.

[Jegp]

I'm not sure I follow the logic here. I completely understand that the delay is "annoying" from a practical point of view. But it's also an inherent property of SNNs that they simply do not propagate signals immediately.

I'm wondering whether this is A) something Norse should consider and, if so, B) something we can include in the synapse dynamics. I'm saying that because we recently had talks about separating synapse and cell dynamics. That could be one way of getting about this issue, because we could have models without any form of delay.

[cpehle]

The current behavior was mostly motivated to be consistent with the current recurrent implementation, where a delay of 1 timestep seems sensible. For the forward integration that is less relevant and can lead to undesirably long delays. I think doing this change would probably be less work than doing a wholesale migration to a synapse neuron dynamics split.

[cpehle]

There is one additional concern that I haven't mentioned yet and that is compatibility with the "adjoint code", I admittedly have not fully investigated this, as there the order also affects the correct order in which the "backward dynamics" needs to be implemented.

[Jegp]

Thanks for your understanding @LennardBo. I'm grateful you brought it up, and I strongly believe we should act on the shorter term.

As @cpehle mentioned, we've been thinking about ways to progress Norse without loss of generality. Your issue is (unfortunately) touching on a longer discussion that I would like to think carefully about, hence the skepticism. One of the reasons Norse is attractive/viable, I believe, is because we're serious about stability and correctness. But @cpehle correctly acted on this and I'm sure we can wrap up the short-term matters quickly.

So, thanks again for bringing this to our attention! :-) I think we should try to move the discussion about integration out in the open and see how we can proceed there. Inputs are deeply appreciated.