Error after first epoch about featdepth

[sconlyshootery]

Could you please paste more details error information, I seem to meet similar issues before, but I fail to remember where to fix it, more information may help me remember, thanks.

[MohsenMoradiArt]

Could you please paste more details error information, I seem to meet similar issues before, but I fail to remember where to fix it, more information may help me remember, thanks.

`evaluation..............................................
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 697/697, 14.1 task/s, elapsed: 49s, ETA: 0s

FPS: 96.1853807579687

results len is 697
2020-11-19 19:26:26,335 - INFO - Epoch [1][39810/39810] lr: 0.00010, abs_rel: 0.1431, sq_rel: 1.1167, rmse: 5.7732, rmse_log: 0.2421, a1: 0.7692, a2: 0.9220, a3: 0.9697, scale mean: 39.6816, scale std: 3.6023221015930176
Traceback (most recent call last):
File "train.py", line 103, in
main()
File "train.py", line 99, in main
logger=logger)
File "/media/armin/DATA/FeatDepth-master/mono/apis/trainer.py", line 68, in train_mono
_dist_train(model, dataset_train, dataset_val, cfg, validate=validate)
File "/media/armin/DATA/FeatDepth-master/mono/apis/trainer.py", line 178, in _dist_train
runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
File "/media/armin/DATA/anaconda3/envs/test/lib/python3.7/site-packages/mmcv/runner/runner.py", line 380, in run
epoch_runner(data_loaders[i], **kwargs)
File "/media/armin/DATA/anaconda3/envs/test/lib/python3.7/site-packages/mmcv/runner/runner.py", line 288, in train
self.call_hook('after_train_epoch')
File "/media/armin/DATA/anaconda3/envs/test/lib/python3.7/site-packages/mmcv/runner/runner.py", line 241, in call_hook
getattr(hook, fn_name)(self)
File "/media/armin/DATA/anaconda3/envs/test/lib/python3.7/site-packages/mmcv/runner/hooks/logger/base.py", line 51, in after_train_epoch
self.log(runner)
File "/media/armin/DATA/anaconda3/envs/test/lib/python3.7/site-packages/mmcv/runner/hooks/logger/text.py", line 115, in log
self._dump_log(log_dict, runner)
File "/media/armin/DATA/anaconda3/envs/test/lib/python3.7/site-packages/mmcv/runner/hooks/logger/text.py", line 82, in _dump_log
mmcv.dump(json_log, f, file_format='json')
File "/media/armin/DATA/anaconda3/envs/test/lib/python3.7/site-packages/mmcv/fileio/io.py", line 82, in dump
handler.dump_to_fileobj(obj, file, **kwargs)
File "/media/armin/DATA/anaconda3/envs/test/lib/python3.7/site-packages/mmcv/fileio/handlers/json_handler.py", line 13, in dump_to_fileobj
json.dump(obj, file, **kwargs)
File "/media/armin/DATA/anaconda3/envs/test/lib/python3.7/json/init.py", line 179, in dump
for chunk in iterable:
File "/media/armin/DATA/anaconda3/envs/test/lib/python3.7/json/encoder.py", line 431, in _iterencode
yield from _iterencode_dict(o, _current_indent_level)
File "/media/armin/DATA/anaconda3/envs/test/lib/python3.7/json/encoder.py", line 405, in _iterencode_dict
yield from chunks
File "/media/armin/DATA/anaconda3/envs/test/lib/python3.7/json/encoder.py", line 438, in _iterencode
o = _default(o)
File "/media/armin/DATA/anaconda3/envs/test/lib/python3.7/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.class.name} '
TypeError: Object of type float32 is not JSON serializable
Traceback (most recent call last):
File "/media/armin/DATA/anaconda3/envs/test/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/media/armin/DATA/anaconda3/envs/test/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/media/armin/DATA/anaconda3/envs/test/lib/python3.7/site-packages/torch/distributed/launch.py", line 260, in
main()
File "/media/armin/DATA/anaconda3/envs/test/lib/python3.7/site-packages/torch/distributed/launch.py", line 256, in main
cmd=cmd)
subprocess.CalledProcessError: Command '['/media/armin/DATA/anaconda3/envs/test/bin/python', '-u', 'train.py', '--local_rank=0', '--config', '/media/armin/DATA/FeatDepth-master/config/cfg_kitti_fm.py', '--work_dir', 'results']' returned non-zero exit status 1.`

[sconlyshootery]

It is an error of mmcv, we should revise the codes in mmcv (which is /media/armin/DATA/anaconda3/envs/test/lib/python3.7/site-packages/mmcv/runner/hooks/logger/text.py in your situation), change the dump_log function to follows:

This error happens original dump_log fails to change float32 to a proper data type for JSON serialization.

[MohsenMoradiArt]

Thank you, now it is working