The output is nearly the same about pytorch-transfomer

[LiamMaclean216]

Could you explain further? Where are you using a random tensor? It could be your data is too difficult for the network, so it just gives the average value every time in order to minimize loss as much as it can.

[LiuShaw]

Thanks for your reply! The examples you gave is to learn a sigmoid function. The input is [batch, 6, 1] and the output is [batch, 1]. I used my dataset, the input size is like [batch, 12, 10] and the target tensor size is [batch, 1, 10]. I have changed the parameters and train a model.
But when I test it, I found for all the different input, I got the similar output, even the input is a tensor with all the data is 1. So I tried small epochs to test. Then I found after the second epoch, the model I trained start output the similar result, as the picture shows(the target output(test dataset) is different). I have tried to fix it for more than 20days >_<.

So, I have changed the in and out tensor size. In addition to hyperparameters, where else should I adjust to make the model work?
Thank you！

[LiamMaclean216]

It sounds like either the problem is too difficult for the network, or there is no correlation between inputs and outputs. Either one would cause the network to produce the average output (expected value) in an attempt to minimize loss, and ignore the inputs.

Adding more layers or widening layers may help. Also try training for more time, at least 100 epochs.

Could also be an error in the code somewhere, maybe the inputs got disconnected from the outputs somehow.

[LiuShaw]

Thanks a lot. I have tried the official implementation by Pytorch, the problem is still remains. And the larger the epochs, the more likely it is to do so. I'll double check the code to see if there's a problem with it. My task is a multi-dim time series predict, maybe it could not be solved by Transformer fundamentally. Thank you again.