mkturkcan / gptune Goto Github PK

Thanks for this very cool variation of the nshepperd original repo.

I've used --truncate_training 64 to fine-tune a 10 meg corpus with 774M adam/all on a 8 gig headless 1070. The sampled output is better than 345M (imho).

Given that result, does that mean the script is passing 64 parameters of a possible 436 training parameters* in a batch, completing processing, then moving on to the next 64 parameters until all 436 are completed OR are only 64 of a possible 436 loaded into memory for the full fine-tuning session resulting in "774M-lite" ?

For reference, I've run the same test on PaperSpace P6000 (24 gig) without the truncate value and it required only 16.6 gigs of the 23+ gigs available. It required 6x less total iterations compared to the 8 gig test and both checkpoints are the same 3.1 gig size.

Unless I'm mistaken it appears that it's possible to fine-tune a corpus using current 774M model and adam/all layers using memory saving gradients on a home gpu which is a major accomplishment to say the least. If that's true (and it's not 774M lite) then the only advantage to using a 24 gig card is 6x savings on processing time.

Would appreciate your confirmation on how it works.

Cheers, and thanks again for the great repo.

• From the Adadelta optimizer fork.

https://twitter.com/skyli0n/status/1164595634734845952?s=21