register_buffer for masks and position encodings breaks DDP about coca-pytorch

[lucidrains]

yea you are right, that is acceptable

only advantage to using buffers is that it would manage its device with the rest of the network parameters, if you were to move it

[lucidrains]

@gshaikov-paige oh yes, thanks! i'll also go make the change for a contending method here

[lucidrains]

@gshaikov-paige so one note is that since i open sourced this, there has been research suggesting PaLM to have some sort of inherent instability. i think i should start removing PaLM architecture from all my repositories soon, and replace it with the one from llama (which is close, just serial)

[gshaikov-paige]

Thanks for the heads up! I can help with transition to LLAMA in this repo, if you're open to PRs?

[gshaikov-paige]

I also added type hints on my branch, I can commit them back if you'd like.

[gshaikov-paige]

@lucidrains thanks - yes, I am doing research to modify CoCa or related methods for my use case, so needed to be able to train it from scratch. I do plan to experiment with a pretrained model though!

Re Llama port: I am planning to make the change from my personal account (@gshaikov) but I can do it only next week since I am away this week, so if this work is urgent please don't wait for me :)

[lucidrains]

@gshaikov-paige sg! i think finetuning it for the medical domain, if that's what you are still doing, should work well

yea sg, i won't wait haha

[lucidrains]

@gshaikov-paige let me know if 0.0.8 takes care of it

[gshaikov-paige]

Thanks! Almost - you need to change it in __init__ as well

[lucidrains]

happy training!

[lucidrains]

@gshaikov-paige yea absolutely! 🙏

[lucidrains]

@gshaikov-paige hmm, i think this project is small enough type hints are not that crucial

[lucidrains]

@gshaikov-paige are you aware that Laion has already trained a CoCa? https://laion.ai/blog/coca/

[gshaikov-paige]

Ok fair enough :)