Training error about eesen

[yajiemiao]

Looks like no training has happened at all.
Can you look into the more detailed log exp/train_phn_l2_c200/log/tr.iter1.log?

[madhavsund]

I couldn't get any clue...
I have attached the log file...

tr.iter1.txt

[madhavsund]

I modified the transcripts and now it is working but the accuracy keeps constant
NO updates in network..

LOG (copy-feats:main():copy-feats.cc:100) Copied 184 feature matrices.
Initializing model as exp/train_phn_l5_c200/nnet/nnet.iter0
TRAINING STARTS [2016-Feb-12 14:08:58]
[NOTE] TOKEN_ACCURACY refers to token accuracy, i.e., (1.0 - token_error_rate).
EPOCH 1 RUNNING ... ENDS [2016-Feb-12 14:17:41]: lrate 4e-05, TRAIN ACCURACY 3.7802%, VALID ACCURACY 4.5504%
EPOCH 2 RUNNING ... ENDS [2016-Feb-12 14:26:34]: lrate 4e-05, TRAIN ACCURACY 3.7802%, VALID ACCURACY 4.5504%
EPOCH 3 RUNNING ...
\

[yajiemiao]

Training simply blows up, as shown by -1e+30 objective.
This is likely to happen if a label (e.g., 100) from label sequences exceeds the range of softmax layer (e.g., 0 - 90). You may check if the number of labels truly corresponds to the size of the softmax layer.

[madhavsund]

I checked the transcript every thing is ok.I tried with tedlium corpus ( with 200 waves) , I get the same error...
I have attached log of tedlium corpus....kindly help me to resolve..

tr.iter1.txt

[yajiemiao]

are you using CPU rather than GPU for training? Eesen does not support CPU training.

[madhavsund]

I have configured for GPU during installation and I use NIVIDEA NVS 300 card.
How to check whether it runs on cpu or gpu?

[yajiemiao]

If you had GPU running, you should have seen something like the following at the beginning of the log file.

LOG (train-ctc-parallel:SelectGpuIdAuto():cuda-device.cc:262) Selecting from 1 GPUs
LOG (train-ctc-parallel:SelectGpuIdAuto():cuda-device.cc:277) cudaSetDevice(0): Tesla K80 free:12163M, used:124M, total:12287M, free/total:0.989858
LOG (train-ctc-parallel:SelectGpuIdAuto():cuda-device.cc:310) Selected device: 0 (automatically)
LOG (train-ctc-parallel:FinalizeActiveGpu():cuda-device.cc:194) The active GPU is [0]: Tesla K80 free:12149M, used:138M, total:12287M, free/total:0.988719 version 3.7
LOG (train-ctc-parallel:PrintMemoryUsage():cuda-device.cc:334) Memory used: 0 bytes.

[madhavsund]

yes, the problem was with GPU . now GPU is configured correctly and it is working ..
Thanks a lot...