vilab-ucsd / geonet Goto Github PK

RUNDIR: runs/plain/plain
2023-10-06 19:24:13,056 INFO Start logging
2023-10-06 19:24:13,056 INFO Namespace(config='configs/plain.yml', source='usa', target='asia', lr_rate=0.003, num_class=205, data_root='/remote-home/GeoNet/', json_dir='/remote-home/GeoNet/geoPlaces_metadata.json', trainer='plain', num_iter=100000, batch_size=32, resume=None, exp_name='plain')
2023-10-06 19:24:19,882 INFO train: 178110
2023-10-06 19:24:23,843 INFO test: 17234
2023-10-06 19:24:28,758 INFO train: 187426
2023-10-06 19:24:32,661 INFO test: 26923
2023-10-06 19:24:34,710 INFO Using sgd optimizer
2023-10-06 19:24:34,711 INFO Start training from iteration 0
2023-10-06 19:24:34,711 INFO Using multiple GPUs
Iteration:  0
Iteration:  1
Iteration:  2
Iteration:  3
Iteration:  4
Iteration:  5
/root/anaconda3/envs/openclip/lib/python3.10/site-packages/torch/autograd/__init__.py:200: UserWarning: Error detected in LogSoftmaxBackward0. Traceback of forward call that caused the error:
  File "/root/GeoNet/train.py", line 271, in <module>
    main()
  File "/root/GeoNet/train.py", line 153, in main
    trainer(batch_iterator, model_fe, model_cls, *d_list, opt, it, *criterion_list,
  File "/root/GeoNet/UDA_trainer/plain.py", line 16, in train_plain
    loss = torch.mean(criterion_cls(output_src, lbl_src).squeeze())
  File "/root/anaconda3/envs/openclip/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/GeoNet/losses/loss.py", line 27, in forward
    return F.cross_entropy(input, target, weight=weight, reduction=self.reduction)
  File "/root/anaconda3/envs/openclip/lib/python3.10/site-packages/torch/nn/functional.py", line 3029, in cross_entropy
    return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
 (Triggered internally at ../torch/csrc/autograd/python_anomaly_mode.cpp:114.)
  Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
Traceback (most recent call last):
  File "/root/GeoNet/train.py", line 271, in <module>
    main()
  File "/root/GeoNet/train.py", line 153, in main
    trainer(batch_iterator, model_fe, model_cls, *d_list, opt, it, *criterion_list,
  File "/root/GeoNet/UDA_trainer/plain.py", line 19, in train_plain
    loss.backward()
  File "/root/anaconda3/envs/openclip/lib/python3.10/site-packages/torch/_tensor.py", line 487, in backward
    torch.autograd.backward(
  File "/root/anaconda3/envs/openclip/lib/python3.10/site-packages/torch/autograd/__init__.py", line 200, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: Function 'LogSoftmaxBackward0' returned nan values in its 0th output.