Weights of MockingjayModel not initialized from pretrained model about s3prl

[andi611]

I see that you are mixing the MOCKINGJAY class with your own downstream solver modified from either mockingjay/solver.py or downstream/solver.py.
I believed most of the errors are from the above modification.

The problem with model before the actual attributes is probably caused by the model initialization method.
I see that you initialized like this:

self.mockingjay = MOCKINGJAY(options=options, inp_dim=160).to(self.device) 

If you examine the code of MOCKINGJAY, the actual model is assigned to the attribute self.model
So you should save like this, as you have discovered:

'Mockingjay': self.mockingjay.model.state_dict()

I've realized our example_extract_finetune.py missed the attribute while saving (which probably mislead you, sorry about that),
I will fix it in the next update, thanks for pointing it out!

As to your parameters mismatch problem:
Warnings are only thrown when model architecture does not match the weight.
For example, loading a 3-layer model weight into a 12-layer model.
By the look of it,
the MOCKINGJAY class builds a 12-layer model (85425408 parameters), and you are loading a 3-layer model weight (21388800 parameters) with your solver, which causes the error.

When you save with this:

all_states = {
                'Classifier': self.classifier.state_dict(),
                'Mockingjay': self.mockingjay.model.state_dict(),
                'Optimizer': self.optimizer.state_dict(),
                'Global_step': self.global_step,
                'Settings': {
                    'Config': self.config,
                    'Paras': self.paras,
                },
            }

The 'Config': self.config probably specifies a 12-layer model,
while the 'Mockingjay': self.mockingjay.model.state_dict() saves the weight of a 3-layer model.
Then when you later loads with the MOCKINGJAY class,
it uses 'Config': self.config to construct a 12-layer model, and loads the 3-layer weight,
which causes the warning.

Please verify if this is the case,
Let me know if you have further problems.
Thanks.