hadusam / blueoil-sagemaker Goto Github PK

Exception during training: Return Code: 1, CMD: ['/usr/local/pyenv/versions/python3.6/bin/python', '/home/blueoil/blueoil/cmd/main.py', 'train', '--experiment_id', 'cifar10_sample', '--config', 'cifar10_sample.yml'], Err: b'/usr/local/pyenv/versions/python3.6/lib/python3.6/site-packages/pandas/compat/__init__.py:117: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.\n  warnings.warn(msg)\nWARNING:tensorflow:From /usr/local/pyenv/versions/python3.6/lib/python3.6/site-packages/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n\nWARNING:tensorflow:From /usr/local/pyenv/versions/python3.6/lib/python3.6/site-packages/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n\nTraceback (most recent call last):\n  File "/home/blueoil/blueoil/cmd/main.py", line 169, in <module>\n    main()\n  File "/usr/local/pyenv/versions/python3.6/lib/python3.6/site-packages/click/core.py", line 764, in __call__\n    return self.main(*args, **kwargs)\n  File "/usr/local/pyenv/versions/python3.6/lib/python3.6/site-packages/click/core.py", line 717, in main\n    rv = self.invoke(ctx)\n  File "/usr/local/pyenv/versions/python3.6/lib/python3.6/site-packages/click/core.py", line 1137, in invoke\n    return _process_result(sub_ctx.command.invoke(sub_ctx))\n  File "/usr/local/pyenv/versions/python3.6/lib/python3.6/site-packages/click/core.py", line 956, in invoke\n    return ctx.invoke(self.callback, **ctx.params)\n  File "/usr/local/pyenv/versions/python3.6/lib/python3.6/site-packages/click/core.py", line 555, in invoke\n    return callback(*args, **kwargs)\n  File "/home/blueoil/blueoil/cmd/main.py", line 79, in train\n    experiment_id, checkpoint_name = run_train(config, experiment_id, recreate)\n  File "/home/blueoil/blueoil/cmd/train.py", line 302, in train\n    run(None, None, config_file, experiment_id, recreate)\n  File "/home/blueoil/blueoil/cmd/train.py", line 273, in run\n    config = config_util.load(config_file)\n  File "/home/blueoil/blueoil/utils/config.py", line 124, in load\n    check_config(config)\n  File "/home/blueoil/blueoil/utils/config.py", line 106, in check_config\n    raise KeyError("config file should include {} parameter".format(key))\nKeyError: \'config file should include IS_DEBUG parameter\'\n'
Traceback (most recent call last):
  File "/opt/ml/code/train", line 119, in <module>
    _run(train_cmd)
  File "/opt/ml/code/train", line 60, in _run
    raise Exception(error_msg)
Exception: Return Code: 1, CMD: ['/usr/local/pyenv/versions/python3.6/bin/python', '/home/blueoil/blueoil/cmd/main.py', 'train', '--experiment_id', 'cifar10_sample', '--config', 'cifar10_sample.yml'], Err: b'/usr/local/pyenv/versions/python3.6/lib/python3.6/site-packages/pandas/compat/__init__.py:117: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.\n  warnings.warn(msg)\nWARNING:tensorflow:From /usr/local/pyenv/versions/python3.6/lib/python3.6/site-packages/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n\nWARNING:tensorflow:From /usr/local/pyenv/versions/python3.6/lib/python3.6/site-packages/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n\nTraceback (most recent call last):\n  File "/home/blueoil/blueoil/cmd/main.py", line 169, in <module>\n    main()\n  File "/usr/local/pyenv/versions/python3.6/lib/python3.6/site-packages/click/core.py", line 764, in __call__\n    return self.main(*args, **kwargs)\n  File "/usr/local/pyenv/versions/python3.6/lib/python3.6/site-packages/click/core.py", line 717, in main\n    rv = self.invoke(ctx)\n  File "/usr/local/pyenv/versions/python3.6/lib/python3.6/site-packages/click/core.py", line 1137, in invoke\n    return _process_result(sub_ctx.command.invoke(sub_ctx))\n  File "/usr/local/pyenv/versions/python3.6/lib/python3.6/site-packages/click/core.py", line 956, in invoke\n    return ctx.invoke(self.callback, **ctx.params)\n  File "/usr/local/pyenv/versions/python3.6/lib/python3.6/site-packages/click/core.py", line 555, in invoke\n    return callback(*args, **kwargs)\n  File "/home/blueoil/blueoil/cmd/main.py", line 79, in train\n    experiment_id, checkpoint_name = run_train(config, experiment_id, recreate)\n  File "/home/blueoil/blueoil/cmd/train.py", line 302, in train\n    run(None, None, config_file, experiment_id, recreate)\n  File "/home/blueoil/blueoil/cmd/train.py", line 273, in run\n    config = config_util.load(config_file)\n  File "/home/blueoil/blueoil/utils/config.py", line 124, in load\n    check_config(config)\n  File "/home/blueoil/blueoil/utils/config.py", line 106, in check_config\n    raise KeyError("config file should include {} parameter".format(key))\nKeyError: \'config file should include IS_DEBUG parameter\'\n'