juliensimon / dlnotebooks Goto Github PK
View Code? Open in Web Editor NEWINACTIVE - please go to https://gitlab.com/juliensimon/dlnotebooks
INACTIVE - please go to https://gitlab.com/juliensimon/dlnotebooks
Error while running : estimator.fit(inputs=data_channels)
Below is the issue:
2019-12-11 12:51:36 Starting - Starting the training job...
2019-12-11 12:51:39 Starting - Launching requested ML instances......
2019-12-11 12:52:37 Starting - Preparing the instances for training......
2019-12-11 12:53:52 Downloading - Downloading input data...
2019-12-11 12:54:19 Training - Downloading the training image..Arguments: train
[12/11/2019 12:54:41 INFO 140441326479168] Reading default configuration from /opt/amazon/lib/python2.7/site-packages/algorithm/resources/default-input.json: {u'num_dynamic_feat': u'auto', u'dropout_rate': u'0.10', u'mini_batch_size': u'128', u'test_quantiles': u'[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]', u'_tuning_objective_metric': u'', u'_num_gpus': u'auto', u'num_eval_samples': u'100', u'learning_rate': u'0.001', u'num_cells': u'40', u'num_layers': u'2', u'embedding_dimension': u'10', u'_kvstore': u'auto', u'_num_kv_servers': u'auto', u'cardinality': u'auto', u'likelihood': u'student-t', u'early_stopping_patience': u''}
[12/11/2019 12:54:41 INFO 140441326479168] Reading provided configuration from /opt/ml/input/config/hyperparameters.json: {u'dropout_rate': u'0.05', u'learning_rate': u'0.00001', u'num_cells': u'40', u'prediction_length': u'30', u'epochs': u'250', u'time_freq': u'D', u'context_length': u'30', u'num_layers': u'2', u'mini_batch_size': u'32', u'likelihood': u'gaussian', u'early_stopping_patience': u'10'}
[12/11/2019 12:54:41 INFO 140441326479168] Final configuration: {u'dropout_rate': u'0.05', u'test_quantiles': u'[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]', u'_tuning_objective_metric': u'', u'num_eval_samples': u'100', u'learning_rate': u'0.00001', u'num_layers': u'2', u'epochs': u'250', u'embedding_dimension': u'10', u'num_cells': u'40', u'_num_kv_servers': u'auto', u'mini_batch_size': u'32', u'likelihood': u'gaussian', u'num_dynamic_feat': u'auto', u'cardinality': u'auto', u'_num_gpus': u'auto', u'prediction_length': u'30', u'time_freq': u'D', u'context_length': u'30', u'_kvstore': u'auto', u'early_stopping_patience': u'10'}
Process 1 is a worker.
[12/11/2019 12:54:41 INFO 140441326479168] Detected entry point for worker worker
[12/11/2019 12:54:42 INFO 140441326479168] Using early stopping with patience 10
[12/11/2019 12:54:42 INFO 140441326479168] [cardinality=auto] cat
field was NOT found in the file /opt/ml/input/data/train/deepar_training.json
and will NOT be used for training.
[12/11/2019 12:54:42 INFO 140441326479168] [num_dynamic_feat=auto] dynamic_feat
field was NOT found in the file /opt/ml/input/data/train/deepar_training.json
and will NOT be used for training.
[12/11/2019 12:54:42 ERROR 140441326479168] Customer Error: Error when parsing json (source: /opt/ml/input/data/train/deepar_training.json, row: 139)
UnexpectedStatusException Traceback (most recent call last)
in ()
----> 1 estimator.fit(inputs=data_channels)
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/estimator.py in fit(self, inputs, wait, logs, job_name, experiment_config)
462 self.jobs.append(self.latest_training_job)
463 if wait:
--> 464 self.latest_training_job.wait(logs=logs)
465
466 def _compilation_job_name(self):
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/estimator.py in wait(self, logs)
1059 # If logs are requested, call logs_for_jobs.
1060 if logs != "None":
-> 1061 self.sagemaker_session.logs_for_job(self.job_name, wait=True, log_type=logs)
1062 else:
1063 self.sagemaker_session.wait_for_job(self.job_name)
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/session.py in logs_for_job(self, job_name, wait, poll, log_type)
2972
2973 if wait:
-> 2974 self._check_job_status(job_name, description, "TrainingJobStatus")
2975 if dot:
2976 print()
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/session.py in _check_job_status(self, job, desc, status_key_name)
2566 ),
2567 allowed_statuses=["Completed", "Stopped"],
-> 2568 actual_status=status,
2569 )
2570
UnexpectedStatusException: Error for Training job daily-temperature-2019-12-11-12-51-36-487: Failed. Reason: ClientError: Error when parsing json (source: /opt/ml/input/data/train/deepar_training.json, row: 139)
Hi ,
This a great example. Can you please post how your json looks like?
def init(modelname, gpu=False):
model = loadModel(modelname,gpu=False)
categories = loadCategories()
return model, categories
MXNetError Traceback (most recent call last)
< ipython-input-23-d3cf351fe555 > in < module > ()
4 return model, categories
5
----> 6 vgg16,categories = init("vgg16")
7 resnet152,categories = init("resnet-152")
8 inceptionv3,categories = init("Inception-BN")
in init(modelname, gpu)
1 def init(modelname, gpu=False):
----> 2 model = loadModel(modelname,gpu=False)
3 categories = loadCategories()
4 return model, categories
5
in loadModel(modelname, gpu)
1 def loadModel(modelname, gpu=False):
----> 2 sym, arg_params, aux_params = mx.model.load_checkpoint(modelname, 0)
3 arg_params['prob_label'] = mx.nd.array([0])
4 arg_params['softmax_label'] = mx.nd.array([0])
5 if gpu:
~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/model.py in load_checkpoint(prefix, epoch)
448 - Parameters will be loaded from prefix-epoch.params
.
449 """
--> 450 symbol = sym.load('%s-symbol.json' % prefix)
451 save_dict = nd.load('%s-%04d.params' % (prefix, epoch))
452 arg_params = {}
~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/symbol/symbol.py in load(fname)
2726 raise TypeError('fname need to be string')
2727 handle = SymbolHandle()
-> 2728 check_call(_LIB.MXSymbolCreateFromFile(c_str(fname), ctypes.byref(handle)))
2729 return Symbol(handle)
2730
~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/base.py in check_call(ret)
251 """
252 if ret != 0:
--> 253 raise MXNetError(py_str(_LIB.MXGetLastError()))
254
255
MXNetError: [16:29:18] /home/travis/build/dmlc/mxnet-distro/mxnet-build/3rdparty/dmlc-core/include/dmlc/././json.h:731: Check failed: ch == '{' (60 vs. {) : Error at Line 0, around ^!DOCTYPE html>
, Expect '{' but get '<'
Stack trace:
[bt] (0) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x25b2ab) [0x7f22a479b2ab]
[bt] (1) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x31f8b23) [0x7f22a7738b23]
[bt] (2) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x320a4b6) [0x7f22a774a4b6]
[bt] (3) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x32033c8) [0x7f22a77433c8]
[bt] (4) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x3203f7a) [0x7f22a7743f7a]
[bt] (5) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x26578b1) [0x7f22a6b978b1]
[bt] (6) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x31ce25d) [0x7f22a770e25d]
[bt] (7) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2664fbf) [0x7f22a6ba4fbf]
[bt] (8) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x26578b1) [0x7f22a6b978b1]
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.