whlzy / fit

[ICML 2024] FiT: Flexible Vision Transformer for Diffusion Model

Hi 👋, I'm Lu Zeyu

• 🔭 I’m currently working on AIGC, AI4SCI and ROBOTs Model using Deep Learning

• 📫 How to reach me: [email protected] or [email protected]

• ⚡ I like ACG(animation, comic and game). I want to help artists, painters and designers with deep learning. If you have any interesting ideas, please contact me.

📊 Last week I spent my time on

Other      29 hrs 55 mins  ████████████████████████▓   99.30 %
Python     10 mins         ░░░░░░░░░░░░░░░░░░░░░░░░░   00.60 %
YAML       1 min           ░░░░░░░░░░░░░░░░░░░░░░░░░   00.06 %
Markdown   0 secs          ░░░░░░░░░░░░░░░░░░░░░░░░░   00.04 %

fit's Issues

waiting code

Position embedding is added inside each block?

DiT is to add positional embedding once before the transformer block. And according to Figure 3 (c) of FiT's paper, it seems that the position embedding is added inside each block. Is this an unfair comparison?

PI for DiT

Hi @whlzy, thanks for your work. As far as I am concerned, DiT also uses absolute position encoding.
How do you apply PI to DiT? Even though it is implemented on absolute PE, will the effect be just like EI (embedding interpolation)?

In contrast, the original PI publication is based on Rotary position encoding. Please correct it if I miss anything.

License

Hi,
Thanks for releasing this amazing project!
Since DIT is NC licensed, is there any chance parts of FIT could be reimplemented so as not to use DIT?
Thanks!