[MusicBERT]: Need help understanding loop in preprocess.F method about muzic

[mlzeng]

Hello @aspil,

1. Some octuple token sequences from the LMD dataset can be very long (more than 1024 octuple tokens), and the Transformer model cannot handle long sequences due to GPU memory size bound. So we use the sliding window style random sampling method to crop very long sequences into multiple shorter segments (which may overlap) for pre-training.
2. We randomly select multiple segments to avoid overfitting and wasting train data. Randomly cropping long sequences on-the-fly during training could be better, but it requires some code.
3. The performance of the model won't significantly degrade if only one segment is used for every sequence (n_time = 1).

Thanks for using MusicBERT!

[aspil]

Thanks for the reply!