Save interval not working about composer

[mvpatel2000]

How many epochs does your run go for? Also, what is the save_filename? Is it unique per timestamp (to verify it isn't being overwritten each time)

[naimavahab]

Only 1 checkpoint is getting saved..in the format of 'ep3-ba1458-rank0.pt'. If I specify save interval 1ep only 1st epoch gets saved even if I have 30 epochs. And if save interval is 3ep, the 3rd epoch gets saved, rest are ignored

[mvpatel2000]

Hm... do you get any errors or traces? Mind sharing a minimal repro please?

[naimavahab]

There are no errors and everything working fine including plotting the loss etc.
I am using this git branch for mosaicbert pretraining. https://github.com/Skylion007/mosaicml-examples/tree/skylion007/add-fa2-to-bert/examples

[eracah]

Mind sharing what version of composer you are using?

[naimavahab]

0.17.2

[eracah]

That's a very old (>6 months old) version of composer. is there a reason you need to use that old of a version?

[naimavahab]

I have been using this particular docker image https://github.com/Skylion007/mosaicml-examples/tree/skylion007/add-fa2-to-bert/examples to use flashattention, triton etc. These set up require sepcific versions, if I update to latest composer, triton fails.
But now I managed by updating to a slightly higher version like 0.19. And it works fine. But I wonder why composer failed only at checkpoint part for the previous 0.17.2 version

[mvpatel2000]

Going to close as it seems to work.

I'm not super sure what the bug was, but it's certainly possible there was an issue in older versions. Definitely recommend upgrading to latest :)