yhl04 / s4

Reimplementation of Structured State Space sequence model (S4)

Tinkering with Structured State Space sequence model (S4)

S4 achieved state of the art results in many long sequence benchmarks

Generating Sequence MNIST Results:

(Treating mnist as a sequence and finish sequence after 400 pixels using s4)

0 Epoch / 100 Epoch / 200 Epoch

Following blog post from The Annotated S4:

The Annotated S4: https://srush.github.io/annotated-s4/

The Annotated S4D: https://srush.github.io/annotated-s4/s4d.html

Original Github:

https://github.com/HazyResearch/state-spaces

Series of papers:

HiPPO: https://arxiv.org/pdf/2008.07669.pdf

Linear State Space Layer: https://arxiv.org/pdf/2110.13985.pdf

Structured State Spaces: https://arxiv.org/pdf/2111.00396.pdf

Diagonal Structured State Space: https://arxiv.org/pdf/2203.14343.pdf

Additional Resources:

HiPPO blog: https://hazyresearch.stanford.edu/blog/2020-12-05-hippo

Presentation from author on s4: https://www.youtube.com/watch?v=luCBXCErkCs