not working with pytorch 0.4 about annotated-transformer

[srush]

Yeah, I will try to do next week.

[ictnlp-wshugen]

Does anyone have a fix that works? I've tried a few of the suggestions above but I still get this error:

RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.IntTensor instead (while checking arguments for embedding)

when trying to evalute this line:

 run_epoch(data_gen(V, 30, 20), model, 
           SimpleLossCompute(model.generator, criterion, model_opt))

I'm using pytorch version 1.0.0

Try this: https://github.com/ictnlp-wshugen/annotated-transformer_codes

[QianhuiWu]

@davidalbertonogueira you can refer to my repository.
https://github.com/DaoD/annotated-transformer/tree/master/src
Hope this can help you.

I forked you repo. But get the result below:
Process finished with exit code 136 (interrupted by signal 8: SIGFPE)

Do you know why?

It seems that there is something wrong with the float point.
I made a little change in the run_epoch(), then the problem is solved:

total_tokens += batch.ntokens.float()
tokens += batch.ntokens.float()

[ictnlp-wshugen]

I'm still getting the same error when using the code from ictnlp-wshugen above. Here's the full error (note: I'm using python 3.7.1 on Windows, pytorch 1.0.0, CUDA 9.0.)

Error: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.IntTensor instead (while checking arguments for embedding)

python first.py
C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\_reduction.py:49: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead.
  warnings.warn(warning.format(ret))
Traceback (most recent call last):
  File "first.py", line 58, in <module>
    run_epoch(data_gen(V, 30, 20), model, SimpleLossCompute(model.generator, criterion, model_opt))
  File "C:\Users\ernichol\OneDrive - Microsoft\code\pytorch\transformer\refactored-annotated-transformer_codes-master\transformer\flow.py", line 54, in run_epoch
    out = model.forward(batch.src, batch.trg, batch.src_mask, batch.trg_mask)
  File "C:\Users\ernichol\OneDrive - Microsoft\code\pytorch\transformer\refactored-annotated-transformer_codes-master\transformer\encoder_decoder.py", line 24, in forward
    return self.decode(self.encode(src, src_mask), src_mask, tgt, tgt_mask)
  File "C:\Users\ernichol\OneDrive - Microsoft\code\pytorch\transformer\refactored-annotated-transformer_codes-master\transformer\encoder_decoder.py", line 27, in encode
    return self.encoder(self.src_embed(src), src_mask)
  File "C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\container.py", line 92, in forward
    input = module(input)
  File "C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\ernichol\OneDrive - Microsoft\code\pytorch\transformer\refactored-annotated-transformer_codes-master\transformer\embeddings.py", line 16, in forward
    return self.lut(x) * math.sqrt(self.d_model)
  File "C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\sparse.py", line 118, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\functional.py", line 1454, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.IntTensor instead (while checking arguments for embedding)

OK, the problem occurs in Windows Platform, now I make it runnable in Windows.
you can update the code or see the change �ictnlp-wshugen/annotated-transformer_codes@ffe3bcc

[shehel]

I gave it a try and found the loss not decreasing beyond 1.5 for the first copying example.. It was the same for pytorch 0.2 aswell.

[vthorsteinsson]

I have gotten it to work with pytorch 0.4 up until and including the copying example. It required inserting a few .item() and .float() calls. Would you like a PR?

[shehel]

Yep, its as simple as @vthorsteinsson mentioned until first example atleast. Jumping between pytorch versions, I missed the dim parameter in softmax.

[teucer]

@vthorsteinsson would it be possible to have your "corrections"?

[richielo]

@srush Any updates on how to fix things for PyTorch 0.4? Have been stuck on this for quite some time, thank you!

[richielo]

Many may have already solved it. But for those who face the same issue with the toy problem, I fixed the error by changing SimpleLossCompute and runEpoch, mostly due to the tensor norm, which has to be changed to FloatTensor from LongTensor:

class SimpleLossCompute:
    "A simple loss compute and train function."
    def __init__(self, generator, criterion, opt=None):
        self.generator = generator
        self.criterion = criterion
        self.opt = opt
    
    def __call__(self, x, y, norm):
        x = self.generator(x)
        loss = self.criterion(x.contiguous().view(-1, x.size(-1)), y.contiguous().view(-1)) / norm.float()
        loss.backward()
        if self.opt is not None:
            self.opt.step()
            self.opt.optimizer.zero_grad()
        return loss.item() * norm.float().item()

The run_epoch function can be fixed accordingly by casting things like batch.ntokens to FloatTensor

[DaoD]

@richielo hi, I find that after moving to Pytorch 0.4.
The memory cached keeps increasing in training even I have freed it after every batch trianing.

[richielo]

@DaoD Hi, I think this may be a different issue. After fixing it with the way I mentioned above, it works fine except for multi-gpu training. are you sure you are freeing memory correctly or declaring variable correctly?

[DaoD]

@richielo Thx. I find that I do not use norm.item(), this may cause creating the parameter multiple times.
And I also replace loss.data[0] with loss.item(), I guess this is more stable without any warnings.
Anyway, thx for your warm help.

[richielo]

@DaoD Oh yes, I missed that too! Glad that you figured it out 👍

[davidalbertonogueira]

Any updates on this? I've fixed some fixes, but many others are still required...

[DaoD]

@davidalbertonogueira you can refer to my repository.
https://github.com/DaoD/annotated-transformer/tree/master/src
Hope this can help you.

[QianhuiWu]

@davidalbertonogueira you can refer to my repository.
https://github.com/DaoD/annotated-transformer/tree/master/src
Hope this can help you.

I forked you repo. But get the result below:
Process finished with exit code 136 (interrupted by signal 8: SIGFPE)

Do you know why?

[DaoD]

@QianhuiWu hi, it looks like there is a "divide by zero" error.
I guess it would happen when 'total_tokens = 0'. And I think there should be a check in the function run_epoch().
But, the parameter 'total_tokens' and 'tokens' are the number of tokens in the epoch, thus should be integers. I'm confused why this can be fixed by setting them as float numbers.

[bigempire]

SimpleLossCompute

Maybe it is the norm that should be blamed, I solved this by fixing the Batch class,
self.ntokens = (self.trg_y != pad).data.sum()
-> self.ntokens = (self.trg_y != pad).data.sum().item()
for (self.trg_y != pad).data.sum() is a tensor.

[eraoul]

Does anyone have a fix that works? I've tried a few of the suggestions above but I still get this error:

RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.IntTensor instead (while checking arguments for embedding)

when trying to evalute this line:

 run_epoch(data_gen(V, 30, 20), model, 
           SimpleLossCompute(model.generator, criterion, model_opt))

I'm using pytorch version 1.0.0

[eraoul]

Many thanks! I'll try this tomorrow.

…

On Mon, Jan 7, 2019 at 7:17 PM Wang Shugen ***@***.***> wrote: Does anyone have a fix that works? I've tried a few of the suggestions above but I still get this error: RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.IntTensor instead (while checking arguments for embedding) when trying to evalute this line: run_epoch(data_gen(V, 30, 20), model, SimpleLossCompute(model.generator, criterion, model_opt)) I'm using pytorch version 1.0.0 Try this: https://github.com/ictnlp-wshugen/annotated-transformer_codes — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#11 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABBIPlmhqp_5WG2kBnPcJTEcwKvihusYks5vBA2-gaJpZM4UCZC5> .

[eraoul]

I'm still getting the same error when using the code from ictnlp-wshugen above. Here's the full error (note: I'm using python 3.7.1 on Windows, pytorch 1.0.0, CUDA 9.0.)

Error: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.IntTensor instead (while checking arguments for embedding)

python first.py
C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\_reduction.py:49: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead.
  warnings.warn(warning.format(ret))
Traceback (most recent call last):
  File "first.py", line 58, in <module>
    run_epoch(data_gen(V, 30, 20), model, SimpleLossCompute(model.generator, criterion, model_opt))
  File "C:\Users\ernichol\OneDrive - Microsoft\code\pytorch\transformer\refactored-annotated-transformer_codes-master\transformer\flow.py", line 54, in run_epoch
    out = model.forward(batch.src, batch.trg, batch.src_mask, batch.trg_mask)
  File "C:\Users\ernichol\OneDrive - Microsoft\code\pytorch\transformer\refactored-annotated-transformer_codes-master\transformer\encoder_decoder.py", line 24, in forward
    return self.decode(self.encode(src, src_mask), src_mask, tgt, tgt_mask)
  File "C:\Users\ernichol\OneDrive - Microsoft\code\pytorch\transformer\refactored-annotated-transformer_codes-master\transformer\encoder_decoder.py", line 27, in encode
    return self.encoder(self.src_embed(src), src_mask)
  File "C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\container.py", line 92, in forward
    input = module(input)
  File "C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\ernichol\OneDrive - Microsoft\code\pytorch\transformer\refactored-annotated-transformer_codes-master\transformer\embeddings.py", line 16, in forward
    return self.lut(x) * math.sqrt(self.d_model)
  File "C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\sparse.py", line 118, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\functional.py", line 1454, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.IntTensor instead (while checking arguments for embedding)

[eraoul]

Excellent -- thanks! This works great. I'm surprised that "torch.from_numpy" works differently in Windows vs other platforms. I wonder if this should be fixed in the pytorch source?

…

On Tue, Jan 8, 2019 at 7:11 PM Wang Shugen ***@***.***> wrote: I'm still getting the same error when using the code from ictnlp-wshugen above. Here's the full error (note: I'm using python 3.7.1 on Windows, pytorch 1.0.0, CUDA 9.0.) Error: *Expected tensor for argument #1 <#1> 'indices' to have scalar type Long; but got torch.IntTensor instead (while checking arguments for embedding)* python first.py C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\_reduction.py:49: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead. warnings.warn(warning.format(ret)) Traceback (most recent call last): File "first.py", line 58, in <module> run_epoch(data_gen(V, 30, 20), model, SimpleLossCompute(model.generator, criterion, model_opt)) File "C:\Users\ernichol\OneDrive - Microsoft\code\pytorch\transformer\refactored-annotated-transformer_codes-master\transformer\flow.py", line 54, in run_epoch out = model.forward(batch.src, batch.trg, batch.src_mask, batch.trg_mask) File "C:\Users\ernichol\OneDrive - Microsoft\code\pytorch\transformer\refactored-annotated-transformer_codes-master\transformer\encoder_decoder.py", line 24, in forward return self.decode(self.encode(src, src_mask), src_mask, tgt, tgt_mask) File "C:\Users\ernichol\OneDrive - Microsoft\code\pytorch\transformer\refactored-annotated-transformer_codes-master\transformer\encoder_decoder.py", line 27, in encode return self.encoder(self.src_embed(src), src_mask) File "C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 489, in __call__ result = self.forward(*input, **kwargs) File "C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\container.py", line 92, in forward input = module(input) File "C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 489, in __call__ result = self.forward(*input, **kwargs) File "C:\Users\ernichol\OneDrive - Microsoft\code\pytorch\transformer\refactored-annotated-transformer_codes-master\transformer\embeddings.py", line 16, in forward return self.lut(x) * math.sqrt(self.d_model) File "C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 489, in __call__ result = self.forward(*input, **kwargs) File "C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\sparse.py", line 118, in forward self.norm_type, self.scale_grad_by_freq, self.sparse) File "C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\functional.py", line 1454, in embedding return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse) RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.IntTensor instead (while checking arguments for embedding) OK, the problem occurs in Windows Platform, now I make it runnable in Windows. you can update the code or see the change � ***@***.*** <ictnlp-wshugen/annotated-transformer_codes@ffe3bcc> [image: image] <https://user-images.githubusercontent.com/45655027/50874191-1a936900-13fe-11e9-9be2-d2f99f706294.png> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#11 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABBIPuwXVxAr_nOSWuq1gASn11PDattDks5vBV3egaJpZM4UCZC5> .

[AmoghM]

@vthorsteinsson @srush @QianhuiWu @shehel @teucer @DaoD @eraoul @ictnlp-wshugen
Please try and look into this issue:
#28
Need help in running it on pytorch 1.0.1.post2

[benbijituo]

I'm still getting the same error when using the code from ictnlp-wshugen above. Here's the full error (note: I'm using python 3.7.1 on Windows, pytorch 1.0.0, CUDA 9.0.)
Error: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.IntTensor instead (while checking arguments for embedding)

python first.py
C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\_reduction.py:49: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead.
  warnings.warn(warning.format(ret))
Traceback (most recent call last):
  File "first.py", line 58, in <module>
    run_epoch(data_gen(V, 30, 20), model, SimpleLossCompute(model.generator, criterion, model_opt))
  File "C:\Users\ernichol\OneDrive - Microsoft\code\pytorch\transformer\refactored-annotated-transformer_codes-master\transformer\flow.py", line 54, in run_epoch
    out = model.forward(batch.src, batch.trg, batch.src_mask, batch.trg_mask)
  File "C:\Users\ernichol\OneDrive - Microsoft\code\pytorch\transformer\refactored-annotated-transformer_codes-master\transformer\encoder_decoder.py", line 24, in forward
    return self.decode(self.encode(src, src_mask), src_mask, tgt, tgt_mask)
  File "C:\Users\ernichol\OneDrive - Microsoft\code\pytorch\transformer\refactored-annotated-transformer_codes-master\transformer\encoder_decoder.py", line 27, in encode
    return self.encoder(self.src_embed(src), src_mask)
  File "C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\container.py", line 92, in forward
    input = module(input)
  File "C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\ernichol\OneDrive - Microsoft\code\pytorch\transformer\refactored-annotated-transformer_codes-master\transformer\embeddings.py", line 16, in forward
    return self.lut(x) * math.sqrt(self.d_model)
  File "C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\sparse.py", line 118, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\functional.py", line 1454, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.IntTensor instead (while checking arguments for embedding)

OK, the problem occurs in Windows Platform, now I make it runnable in Windows.
you can update the code or see the change �ictnlp-wshugen/annotated-transformer_codes@ffe3bcc

Well, I've tried your work. Everything is going well running on single GPU.
But when I trained it on multi-GPU, condition would be like this #16. Do you have any solution about it?

[qzxyxiaobao]

@davidalbertonogueira you can refer to my repository.
https://github.com/DaoD/annotated-transformer/tree/master/src
Hope this can help you.

I forked you repo. But get the result below:
Process finished with exit code 136 (interrupted by signal 8: SIGFPE)
Do you know why?

It seems that there is something wrong with the float point.
I made a little change in the run_epoch(), then the problem is solved:

total_tokens += batch.ntokens.float()
tokens += batch.ntokens.float()

thank you for your sharing code!

[yuvaraj91]

I'm still getting the same error when using the code from ictnlp-wshugen above. Here's the full error (note: I'm using python 3.7.1 on Windows, pytorch 1.0.0, CUDA 9.0.)
Error: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.IntTensor instead (while checking arguments for embedding)

python first.py
C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\_reduction.py:49: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead.
  warnings.warn(warning.format(ret))
Traceback (most recent call last):
  File "first.py", line 58, in <module>
    run_epoch(data_gen(V, 30, 20), model, SimpleLossCompute(model.generator, criterion, model_opt))
  File "C:\Users\ernichol\OneDrive - Microsoft\code\pytorch\transformer\refactored-annotated-transformer_codes-master\transformer\flow.py", line 54, in run_epoch
    out = model.forward(batch.src, batch.trg, batch.src_mask, batch.trg_mask)
  File "C:\Users\ernichol\OneDrive - Microsoft\code\pytorch\transformer\refactored-annotated-transformer_codes-master\transformer\encoder_decoder.py", line 24, in forward
    return self.decode(self.encode(src, src_mask), src_mask, tgt, tgt_mask)
  File "C:\Users\ernichol\OneDrive - Microsoft\code\pytorch\transformer\refactored-annotated-transformer_codes-master\transformer\encoder_decoder.py", line 27, in encode
    return self.encoder(self.src_embed(src), src_mask)
  File "C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\container.py", line 92, in forward
    input = module(input)
  File "C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\ernichol\OneDrive - Microsoft\code\pytorch\transformer\refactored-annotated-transformer_codes-master\transformer\embeddings.py", line 16, in forward
    return self.lut(x) * math.sqrt(self.d_model)
  File "C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\sparse.py", line 118, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "C:\Users\ernichol\AppData\Local\Continuum\anaconda3\envs\pytorch\lib\site-packages\torch\nn\functional.py", line 1454, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.IntTensor instead (while checking arguments for embedding)

OK, the problem occurs in Windows Platform, now I make it runnable in Windows.
you can update the code or see the change �ictnlp-wshugen/annotated-transformer_codes@ffe3bcc

Well, I've tried your work. Everything is going well running on single GPU. But when I trained it on multi-GPU, condition would be like this #16. Do you have any solution about it?

Hi, could you mind sharing how to run it on a single gpu? I keep getting a CUDA out of memory error (using Google Colab)