yn-sun / ea-cnn Goto Github PK
View Code? Open in Web Editor NEWThis is the code for the paper of "Completely automated CNN architecture design based on blocks"
This is the code for the paper of "Completely automated CNN architecture design based on blocks"
2022-04-13 19:53:09,031 INFO : *************************
2022-04-13 19:53:09,031 INFO : Initialize from existing population data
2022-04-13 19:53:09,031 INFO : Initialize from 0-th generation
2022-04-13 19:53:09,044 INFO : EVOLVE[0-gen]-Begin to evaluate the fitness
2022-04-13 19:53:09,044 INFO : Begin to generate python files
2022-04-13 19:53:09,055 INFO : Finish the generation of python files
2022-04-13 19:53:09,055 INFO : Query fitness from cache
2022-04-13 19:53:09,055 INFO : Total hit 0 individuals for fitness
2022-04-13 19:53:09,055 INFO : Begin to train indi0000
2022-04-13 19:53:09,597 INFO : Begin to train indi0001
2022-04-13 19:53:09,604 INFO : Begin to train indi0002
2022-04-13 19:53:09,612 INFO : Begin to train indi0003
2022-04-13 19:53:09,619 INFO : Begin to train indi0004
2022-04-13 19:53:09,626 INFO : Begin to train indi0005
2022-04-13 19:53:09,634 INFO : Begin to train indi0006
2022-04-13 19:53:09,641 INFO : Begin to train indi0007
2022-04-13 19:53:09,649 INFO : Begin to train indi0008
2022-04-13 19:53:09,662 INFO : Begin to train indi0009
2022-04-13 19:53:09,675 INFO : Begin to train indi0010
2022-04-13 19:53:09,688 INFO : Begin to train indi0011
2022-04-13 19:53:09,701 INFO : Begin to train indi0012
2022-04-13 19:53:09,715 INFO : Begin to train indi0013
2022-04-13 19:53:09,728 INFO : Begin to train indi0014
2022-04-13 19:53:09,742 INFO : Begin to train indi0015
2022-04-13 19:53:09,756 INFO : Begin to train indi0016
2022-04-13 19:53:09,770 INFO : Begin to train indi0017
2022-04-13 19:53:09,792 INFO : Begin to train indi0018
2022-04-13 19:53:09,822 INFO : Begin to train indi0019
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Exception occurs, file:indi0019, pid:30939...CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Process Process-20:
Exception occurs, file:indi0003, pid:30923...CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Process Process-4:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0003.py", line 227, in do_work
m = TrainModel()
File "/work/sunbiao/ea-cnn-master/scripts/indi0003.py", line 145, in init
net = net.cuda()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in cuda
return self._apply(lambda t: t.cuda(device))
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
[Previous line repeated 1 more time]
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 552, in _apply
param_applied = fn(param)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in
return self._apply(lambda t: t.cuda(device))
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0003.py", line 234, in do_work
m.log_record('Exception occur:%s'%(str(e)))
UnboundLocalError: local variable 'm' referenced before assignment
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/work/sunbiao/ea-cnn-master/scripts/indi0003.py", line 236, in do_work
m.log_record('Finished-Acc:%.3f'%best_acc)
UnboundLocalError: local variable 'm' referenced before assignment
Exception occurs, file:indi0009, pid:30929...CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Process Process-10:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0009.py", line 225, in do_work
m = TrainModel()
File "/work/sunbiao/ea-cnn-master/scripts/indi0009.py", line 143, in init
net = net.cuda()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in cuda
return self._apply(lambda t: t.cuda(device))
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
[Previous line repeated 1 more time]
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 552, in _apply
param_applied = fn(param)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in
return self._apply(lambda t: t.cuda(device))
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0009.py", line 232, in do_work
m.log_record('Exception occur:%s'%(str(e)))
UnboundLocalError: local variable 'm' referenced before assignment
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/work/sunbiao/ea-cnn-master/scripts/indi0009.py", line 234, in do_work
m.log_record('Finished-Acc:%.3f'%best_acc)
UnboundLocalError: local variable 'm' referenced before assignment
Exception occurs, file:indi0014, pid:30934...CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Process Process-15:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0014.py", line 223, in do_work
m = TrainModel()
File "/work/sunbiao/ea-cnn-master/scripts/indi0014.py", line 141, in init
net = net.cuda()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in cuda
return self._apply(lambda t: t.cuda(device))
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
[Previous line repeated 1 more time]
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 552, in _apply
param_applied = fn(param)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in
return self._apply(lambda t: t.cuda(device))
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0014.py", line 230, in do_work
m.log_record('Exception occur:%s'%(str(e)))
UnboundLocalError: local variable 'm' referenced before assignment
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/work/sunbiao/ea-cnn-master/scripts/indi0014.py", line 232, in do_work
m.log_record('Finished-Acc:%.3f'%best_acc)
UnboundLocalError: local variable 'm' referenced before assignment
Exception occurs, file:indi0017, pid:30937...CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Process Process-18:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0017.py", line 225, in do_work
m = TrainModel()
File "/work/sunbiao/ea-cnn-master/scripts/indi0017.py", line 143, in init
net = net.cuda()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in cuda
return self._apply(lambda t: t.cuda(device))
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
[Previous line repeated 1 more time]
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 552, in _apply
param_applied = fn(param)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in
return self._apply(lambda t: t.cuda(device))
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0017.py", line 232, in do_work
m.log_record('Exception occur:%s'%(str(e)))
UnboundLocalError: local variable 'm' referenced before assignment
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/work/sunbiao/ea-cnn-master/scripts/indi0017.py", line 234, in do_work
m.log_record('Finished-Acc:%.3f'%best_acc)
UnboundLocalError: local variable 'm' referenced before assignment
Exception occurs, file:indi0012, pid:30932...CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Process Process-13:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0012.py", line 225, in do_work
m = TrainModel()
File "/work/sunbiao/ea-cnn-master/scripts/indi0012.py", line 143, in init
net = net.cuda()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in cuda
return self._apply(lambda t: t.cuda(device))
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
[Previous line repeated 1 more time]
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 552, in _apply
param_applied = fn(param)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in
return self._apply(lambda t: t.cuda(device))
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0012.py", line 232, in do_work
m.log_record('Exception occur:%s'%(str(e)))
UnboundLocalError: local variable 'm' referenced before assignment
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/work/sunbiao/ea-cnn-master/scripts/indi0012.py", line 234, in do_work
m.log_record('Finished-Acc:%.3f'%best_acc)
UnboundLocalError: local variable 'm' referenced before assignment
Exception occurs, file:indi0007, pid:30927...CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Process Process-8:
Exception occurs, file:indi0016, pid:30936...CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Process Process-17:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0007.py", line 227, in do_work
m = TrainModel()
File "/work/sunbiao/ea-cnn-master/scripts/indi0007.py", line 145, in init
net = net.cuda()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in cuda
return self._apply(lambda t: t.cuda(device))
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
[Previous line repeated 1 more time]
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 552, in _apply
param_applied = fn(param)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in
return self._apply(lambda t: t.cuda(device))
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0007.py", line 234, in do_work
m.log_record('Exception occur:%s'%(str(e)))
UnboundLocalError: local variable 'm' referenced before assignment
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
Traceback (most recent call last):
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/work/sunbiao/ea-cnn-master/scripts/indi0007.py", line 236, in do_work
m.log_record('Finished-Acc:%.3f'%best_acc)
UnboundLocalError: local variable 'm' referenced before assignment
File "/work/sunbiao/ea-cnn-master/scripts/indi0016.py", line 225, in do_work
m = TrainModel()
File "/work/sunbiao/ea-cnn-master/scripts/indi0016.py", line 143, in init
net = net.cuda()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in cuda
return self._apply(lambda t: t.cuda(device))
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
[Previous line repeated 1 more time]
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 552, in _apply
param_applied = fn(param)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in
return self._apply(lambda t: t.cuda(device))
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0016.py", line 232, in do_work
m.log_record('Exception occur:%s'%(str(e)))
UnboundLocalError: local variable 'm' referenced before assignment
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/work/sunbiao/ea-cnn-master/scripts/indi0016.py", line 234, in do_work
m.log_record('Finished-Acc:%.3f'%best_acc)
UnboundLocalError: local variable 'm' referenced before assignment
Exception occurs, file:indi0015, pid:30935...CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Process Process-16:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0015.py", line 225, in do_work
m = TrainModel()
File "/work/sunbiao/ea-cnn-master/scripts/indi0015.py", line 143, in init
net = net.cuda()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in cuda
return self._apply(lambda t: t.cuda(device))
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
[Previous line repeated 1 more time]
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 552, in _apply
param_applied = fn(param)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in
return self._apply(lambda t: t.cuda(device))
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0015.py", line 232, in do_work
m.log_record('Exception occur:%s'%(str(e)))
UnboundLocalError: local variable 'm' referenced before assignment
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/work/sunbiao/ea-cnn-master/scripts/indi0015.py", line 234, in do_work
m.log_record('Finished-Acc:%.3f'%best_acc)
UnboundLocalError: local variable 'm' referenced before assignment
Exception occurs, file:indi0013, pid:30933...CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Process Process-14:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0013.py", line 227, in do_work
m = TrainModel()
File "/work/sunbiao/ea-cnn-master/scripts/indi0013.py", line 145, in init
net = net.cuda()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in cuda
return self._apply(lambda t: t.cuda(device))
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
[Previous line repeated 1 more time]
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 552, in _apply
param_applied = fn(param)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in
return self._apply(lambda t: t.cuda(device))
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0013.py", line 234, in do_work
m.log_record('Exception occur:%s'%(str(e)))
UnboundLocalError: local variable 'm' referenced before assignment
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/work/sunbiao/ea-cnn-master/scripts/indi0013.py", line 236, in do_work
m.log_record('Finished-Acc:%.3f'%best_acc)
UnboundLocalError: local variable 'm' referenced before assignment
Exception occurs, file:indi0018, pid:30938...CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Exception occurs, file:indi0006, pid:30926...CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.Process Process-19:
Process Process-7:
Traceback (most recent call last):
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0018.py", line 223, in do_work
m = TrainModel()
File "/work/sunbiao/ea-cnn-master/scripts/indi0006.py", line 225, in do_work
m = TrainModel()
File "/work/sunbiao/ea-cnn-master/scripts/indi0018.py", line 141, in init
net = net.cuda()
File "/work/sunbiao/ea-cnn-master/scripts/indi0006.py", line 143, in init
net = net.cuda()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in cuda
return self._apply(lambda t: t.cuda(device))
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in cuda
return self._apply(lambda t: t.cuda(device))
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
[Previous line repeated 1 more time]
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 552, in _apply
param_applied = fn(param)
[Previous line repeated 1 more time]
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in
return self._apply(lambda t: t.cuda(device))
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 552, in _apply
param_applied = fn(param)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in
return self._apply(lambda t: t.cuda(device))
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
During handling of the above exception, another exception occurred:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0018.py", line 230, in do_work
m.log_record('Exception occur:%s'%(str(e)))
File "/work/sunbiao/ea-cnn-master/scripts/indi0006.py", line 232, in do_work
m.log_record('Exception occur:%s'%(str(e)))
UnboundLocalError: local variable 'm' referenced before assignment
UnboundLocalError: local variable 'm' referenced before assignment
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
During handling of the above exception, another exception occurred:
Exception occurs, file:indi0008, pid:30928...CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.Traceback (most recent call last):
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/work/sunbiao/ea-cnn-master/scripts/indi0018.py", line 232, in do_work
m.log_record('Finished-Acc:%.3f'%best_acc)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/work/sunbiao/ea-cnn-master/scripts/indi0006.py", line 234, in do_work
m.log_record('Finished-Acc:%.3f'%best_acc)
UnboundLocalError: local variable 'm' referenced before assignment
UnboundLocalError: local variable 'm' referenced before assignment
Process Process-9:
Exception occurs, file:indi0011, pid:30931...CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Process Process-12:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0008.py", line 225, in do_work
m = TrainModel()
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0008.py", line 143, in init
net = net.cuda()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in cuda
return self._apply(lambda t: t.cuda(device))
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
[Previous line repeated 1 more time]
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 552, in _apply
param_applied = fn(param)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in
return self._apply(lambda t: t.cuda(device))
File "/work/sunbiao/ea-cnn-master/scripts/indi0011.py", line 223, in do_work
m = TrainModel()
File "/work/sunbiao/ea-cnn-master/scripts/indi0011.py", line 141, in init
net = net.cuda()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in cuda
return self._apply(lambda t: t.cuda(device))
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
During handling of the above exception, another exception occurred:
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
Traceback (most recent call last):
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/work/sunbiao/ea-cnn-master/scripts/indi0008.py", line 232, in do_work
m.log_record('Exception occur:%s'%(str(e)))
[Previous line repeated 1 more time]
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 552, in _apply
param_applied = fn(param)
UnboundLocalError: local variable 'm' referenced before assignment
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in
return self._apply(lambda t: t.cuda(device))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
During handling of the above exception, another exception occurred:
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0008.py", line 234, in do_work
m.log_record('Finished-Acc:%.3f'%best_acc)
File "/work/sunbiao/ea-cnn-master/scripts/indi0011.py", line 230, in do_work
m.log_record('Exception occur:%s'%(str(e)))
UnboundLocalError: local variable 'm' referenced before assignment
UnboundLocalError: local variable 'm' referenced before assignment
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/work/sunbiao/ea-cnn-master/scripts/indi0011.py", line 232, in do_work
m.log_record('Finished-Acc:%.3f'%best_acc)
UnboundLocalError: local variable 'm' referenced before assignment
Exception occurs, file:indi0010, pid:30930...CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Process Process-11:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0010.py", line 219, in do_work
m = TrainModel()
File "/work/sunbiao/ea-cnn-master/scripts/indi0010.py", line 137, in init
net = net.cuda()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in cuda
return self._apply(lambda t: t.cuda(device))
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
[Previous line repeated 1 more time]
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 552, in _apply
param_applied = fn(param)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in
return self._apply(lambda t: t.cuda(device))
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0010.py", line 226, in do_work
m.log_record('Exception occur:%s'%(str(e)))
UnboundLocalError: local variable 'm' referenced before assignment
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/work/sunbiao/ea-cnn-master/scripts/indi0010.py", line 228, in do_work
m.log_record('Finished-Acc:%.3f'%best_acc)
UnboundLocalError: local variable 'm' referenced before assignment
Exception occurs, file:indi0005, pid:30925...CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Process Process-6:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0005.py", line 229, in do_work
m = TrainModel()
File "/work/sunbiao/ea-cnn-master/scripts/indi0005.py", line 147, in init
net = net.cuda()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in cuda
return self._apply(lambda t: t.cuda(device))
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
[Previous line repeated 1 more time]
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 552, in _apply
param_applied = fn(param)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in
return self._apply(lambda t: t.cuda(device))
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0005.py", line 236, in do_work
m.log_record('Exception occur:%s'%(str(e)))
UnboundLocalError: local variable 'm' referenced before assignment
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/work/sunbiao/ea-cnn-master/scripts/indi0005.py", line 238, in do_work
m.log_record('Finished-Acc:%.3f'%best_acc)
UnboundLocalError: local variable 'm' referenced before assignment
Exception occurs, file:indi0002, pid:30922...CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Process Process-3:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0002.py", line 223, in do_work
m = TrainModel()
File "/work/sunbiao/ea-cnn-master/scripts/indi0002.py", line 141, in init
net = net.cuda()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in cuda
return self._apply(lambda t: t.cuda(device))
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
[Previous line repeated 1 more time]
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 552, in _apply
param_applied = fn(param)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in
return self._apply(lambda t: t.cuda(device))
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0002.py", line 230, in do_work
m.log_record('Exception occur:%s'%(str(e)))
UnboundLocalError: local variable 'm' referenced before assignment
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/work/sunbiao/ea-cnn-master/scripts/indi0002.py", line 232, in do_work
m.log_record('Finished-Acc:%.3f'%best_acc)
UnboundLocalError: local variable 'm' referenced before assignment
Exception occurs, file:indi0004, pid:30924...CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Process Process-5:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0004.py", line 225, in do_work
m = TrainModel()
File "/work/sunbiao/ea-cnn-master/scripts/indi0004.py", line 143, in init
net = net.cuda()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in cuda
return self._apply(lambda t: t.cuda(device))
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
[Previous line repeated 1 more time]
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 552, in _apply
param_applied = fn(param)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in
return self._apply(lambda t: t.cuda(device))
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0004.py", line 232, in do_work
m.log_record('Exception occur:%s'%(str(e)))
UnboundLocalError: local variable 'm' referenced before assignment
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/work/sunbiao/ea-cnn-master/scripts/indi0004.py", line 234, in do_work
m.log_record('Finished-Acc:%.3f'%best_acc)
UnboundLocalError: local variable 'm' referenced before assignment
Exception occurs, file:indi0001, pid:30921...CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Process Process-2:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0001.py", line 227, in do_work
m = TrainModel()
File "/work/sunbiao/ea-cnn-master/scripts/indi0001.py", line 145, in init
net = net.cuda()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in cuda
return self._apply(lambda t: t.cuda(device))
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
[Previous line repeated 1 more time]
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 552, in _apply
param_applied = fn(param)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in
return self._apply(lambda t: t.cuda(device))
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0001.py", line 234, in do_work
m.log_record('Exception occur:%s'%(str(e)))
UnboundLocalError: local variable 'm' referenced before assignment
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/work/sunbiao/ea-cnn-master/scripts/indi0001.py", line 236, in do_work
m.log_record('Finished-Acc:%.3f'%best_acc)
UnboundLocalError: local variable 'm' referenced before assignment
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0019.py", line 225, in do_work
m = TrainModel()
File "/work/sunbiao/ea-cnn-master/scripts/indi0019.py", line 143, in init
net = net.cuda()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in cuda
return self._apply(lambda t: t.cuda(device))
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
[Previous line repeated 1 more time]
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 552, in _apply
param_applied = fn(param)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in
return self._apply(lambda t: t.cuda(device))
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0019.py", line 232, in do_work
m.log_record('Exception occur:%s'%(str(e)))
UnboundLocalError: local variable 'm' referenced before assignment
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/work/sunbiao/ea-cnn-master/scripts/indi0019.py", line 234, in do_work
m.log_record('Finished-Acc:%.3f'%best_acc)
UnboundLocalError: local variable 'm' referenced before assignment
Exception occurs, file:indi0000, pid:30920...CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Process Process-1:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0000.py", line 227, in do_work
m = TrainModel()
File "/work/sunbiao/ea-cnn-master/scripts/indi0000.py", line 145, in init
net = net.cuda()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in cuda
return self._apply(lambda t: t.cuda(device))
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
[Previous line repeated 1 more time]
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 552, in _apply
param_applied = fn(param)
File "/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in
return self._apply(lambda t: t.cuda(device))
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/work/sunbiao/ea-cnn-master/scripts/indi0000.py", line 234, in do_work
m.log_record('Exception occur:%s'%(str(e)))
UnboundLocalError: local variable 'm' referenced before assignment
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/home/nature/anaconda3/envs/nas/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/work/sunbiao/ea-cnn-master/scripts/indi0000.py", line 236, in do_work
m.log_record('Finished-Acc:%.3f'%best_acc)
UnboundLocalError: local variable 'm' referenced before assignment
Traceback (most recent call last):
File "evolve.py", line 1, in
from utils import StatusUpdateTool, Utils, Log
File "/home/guohy/eacnn-sum/ea-cnn/utils.py", line 5, in
from genetic.population import Population, Individual, DenseUnit, ResUnit, PoolUnit
ModuleNotFoundError: No module named 'genetic.population'
Can you tell me how to solve the ModuleNotFound Problem? Thank you very much.
Hi,yanan.Thanks for your excellent jobs . Could you please upload a copy of training code since i am not familiar with the training procedure of the evolutionary algorithms.
Hi, it seems that there is a missing file called No such file or directory: './populations/
.
直接跑生成的script报错
indi-gen00-no00.py
Files already downloaded and verified
Files already downloaded and verified
/home/nature/anaconda3/envs/nas/lib/python3.8/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at /pytorch/c10/core/TensorImpl.h:1156.)
return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
Exception occurs, file:indi-gen00-no00, pid:18314...invalid index of a 0-dim tensor. Use tensor.item()
in Python or tensor.item<T>()
in C++ to convert a 0-dim tensor to a number
以下是indi-gen00-no00.py文件内部内容:
"""
2022-04-13 19:46:58
"""
from future import print_function
import torch
from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F
import torch.backends.cudnn as cudnn
import torch.optim as optim
import data_loader
import os
from datetime import datetime
import multiprocessing
from utils import StatusUpdateTool
class ResNetBottleneck(nn.Module):
expansion = 1
def __init__(self, in_planes, planes, stride=1):
super(ResNetBottleneck, self).__init__()
self.conv1 = nn.Conv2d(in_planes, planes, kernel_size=1, bias=False)
self.bn1 = nn.BatchNorm2d(planes)
self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=stride, padding=1, bias=False)
self.bn2 = nn.BatchNorm2d(planes)
self.conv3 = nn.Conv2d(planes, self.expansion*planes, kernel_size=1, bias=False)
self.bn3 = nn.BatchNorm2d(self.expansion*planes)
self.shortcut = nn.Sequential()
if stride != 1 or in_planes != self.expansion*planes:
self.shortcut = nn.Sequential(
nn.Conv2d(in_planes, self.expansion*planes, kernel_size=1, stride=stride, bias=False),
nn.BatchNorm2d(self.expansion*planes)
)
def forward(self, x):
out = F.relu(self.bn1(self.conv1(x)))
out = F.relu(self.bn2(self.conv2(out)))
out = self.bn3(self.conv3(out))
out += self.shortcut(x)
out = F.relu(out)
return out
class ResNetUnit(nn.Module):
def init(self, amount, in_channel, out_channel):
super(ResNetUnit, self).init()
self.in_planes = in_channel
self.layer = self._make_layer(ResNetBottleneck, out_channel, amount, stride=1)
def _make_layer(self, block, planes, num_blocks, stride):
strides = [stride] + [1]*(num_blocks-1)
layers = []
for stride in strides:
layers.append(block(self.in_planes, planes, stride))
self.in_planes = planes * block.expansion
return nn.Sequential(*layers)
def forward(self, x):
out = self.layer(x)
return out
class DenseNetBottleneck(nn.Module):
def init(self, nChannels, growthRate):
super(DenseNetBottleneck, self).init()
interChannels = 4*growthRate
self.bn1 = nn.BatchNorm2d(nChannels)
self.conv1 = nn.Conv2d(nChannels, interChannels, kernel_size=1,
bias=False)
self.bn2 = nn.BatchNorm2d(interChannels)
self.conv2 = nn.Conv2d(interChannels, growthRate, kernel_size=3,
padding=1, bias=False)
def forward(self, x):
out = self.conv1(F.relu(self.bn1(x)))
out = self.conv2(F.relu(self.bn2(out)))
out = torch.cat((x, out), 1)
return out
class DenseNetUnit(nn.Module):
def init(self, k, amount, in_channel, out_channel, max_input_channel):
super(DenseNetUnit, self).init()
self.out_channel = out_channel
if in_channel > max_input_channel:
self.need_conv = True
self.bn = nn.BatchNorm2d(in_channel)
self.conv = nn.Conv2d(in_channel, max_input_channel, kernel_size=1, bias=False)
in_channel = max_input_channel
self.layer = self._make_dense(in_channel, k, amount)
def _make_dense(self, nChannels, growthRate, nDenseBlocks):
layers = []
for _ in range(int(nDenseBlocks)):
layers.append(DenseNetBottleneck(nChannels, growthRate))
nChannels += growthRate
return nn.Sequential(*layers)
def forward(self, x):
out = x
if hasattr(self, 'need_conv'):
out = self.conv(F.relu(self.bn(out)))
out = self.layer(out)
assert(out.size()[1] == self.out_channel)
return out
class EvoCNNModel(nn.Module):
def init(self):
super(EvoCNNModel, self).init()
#resnet and densenet unit
self.op0 = ResNetUnit(amount=7, in_channel=3, out_channel=64)
self.op1 = ResNetUnit(amount=7, in_channel=64, out_channel=256)
self.op6 = DenseNetUnit(k=12, amount=10, in_channel=256, out_channel=248, max_input_channel=128)
self.op7 = ResNetUnit(amount=7, in_channel=248, out_channel=128)
self.op8 = ResNetUnit(amount=6, in_channel=128, out_channel=256)
#linear unit
self.linear = nn.Linear(1024, 10)
def forward(self, x):
out_0 = self.op0(x)
out_1 = self.op1(out_0)
out_2 = F.avg_pool2d(out_1, 2)
out_3 = F.avg_pool2d(out_2, 2)
out_4 = F.max_pool2d(out_3, 2)
out_5 = F.avg_pool2d(out_4, 2)
out_6 = self.op6(out_5)
out_7 = self.op7(out_6)
out_8 = self.op8(out_7)
out = out_8
out = out.view(out.size(0), -1)
out = self.linear(out)
return out
class TrainModel(object):
def init(self):
trainloader, validate_loader = data_loader.get_train_valid_loader('../data_cifar10', batch_size=128, augment=True, valid_size=0.1, shuffle=True, random_seed=2312390, show_sample=False, num_workers=1, pin_memory=True)
#testloader = data_loader.get_test_loader('../data_cifar10', batch_size=128, shuffle=False, num_workers=1, pin_memory=True)
net = EvoCNNModel()
cudnn.benchmark = True
net = net.cuda()
criterion = nn.CrossEntropyLoss()
best_acc = 0.0
self.net = net
self.criterion = criterion
self.best_acc = best_acc
self.trainloader = trainloader
self.validate_loader = validate_loader
self.file_id = os.path.basename(file).split('.')[0]
#self.testloader = testloader
#self.log_record(net, first_time=True)
#self.log_record('+'*50, first_time=False)
def log_record(self, _str, first_time=None):
dt = datetime.now()
dt.strftime( '%Y-%m-%d %H:%M:%S' )
if first_time:
file_mode = 'w'
else:
file_mode = 'a+'
f = open('./log/%s.txt'%(self.file_id), file_mode)
f.write('[%s]-%s\n'%(dt, _str))
f.flush()
f.close()
def train(self, epoch):
self.net.train()
if epoch ==0: lr = 0.01
if epoch > 0: lr = 0.1;
if epoch > 148: lr = 0.01
if epoch > 248: lr = 0.001
optimizer = optim.SGD(self.net.parameters(), lr=lr, momentum = 0.9, weight_decay=5e-4)
running_loss = 0.0
total = 0
correct = 0
for _, data in enumerate(self.trainloader, 0):
inputs, labels = data
inputs, labels = Variable(inputs.cuda()), Variable(labels.cuda())
optimizer.zero_grad()
outputs = self.net(inputs)
loss = self.criterion(outputs, labels)
loss.backward()
optimizer.step()
running_loss += loss.data[0]*labels.size(0)
_, predicted = torch.max(outputs.data, 1)
total += labels.size(0)
correct += (predicted == labels.data).sum()
self.log_record('Train-Epoch:%3d, Loss: %.3f, Acc:%.3f'% (epoch+1, running_loss/total, (correct/total)))
def test(self, epoch):
self.net.eval()
test_loss = 0.0
total = 0
correct = 0
for _, data in enumerate(self.validate_loader, 0):
inputs, labels = data
inputs, labels = Variable(inputs.cuda()), Variable(labels.cuda())
outputs = self.net(inputs)
loss = self.criterion(outputs, labels)
test_loss += loss.data[0]*labels.size(0)
_, predicted = torch.max(outputs.data, 1)
total += labels.size(0)
correct += (predicted == labels.data).sum()
if correct/total > self.best_acc:
self.best_acc = correct/total
#print('*'*100, self.best_acc)
self.log_record('Validate-Loss:%.3f, Acc:%.3f'%(test_loss/total, correct/total))
def process(self):
total_epoch = StatusUpdateTool.get_epoch_size()
for p in range(total_epoch):
self.train(p)
self.test(total_epoch)
return self.best_acc
class RunModel(object):
@classmethod
def do_work(self, gpu_id, file_id):
os.environ['CUDA_VISIBLE_DEVICES'] = gpu_id
best_acc = 0.0
try:
m = TrainModel()
m.log_record('Used GPU#%s, worker name:%s[%d]'%(gpu_id, multiprocessing.current_process().name, os.getpid()), first_time=True)
best_acc = m.process()
#import random
#best_acc = random.random()
except BaseException as e:
print('Exception occurs, file:%s, pid:%d...%s'%(file_id, os.getpid(), str(e)))
m.log_record('Exception occur:%s'%(str(e)))
finally:
m.log_record('Finished-Acc:%.3f'%best_acc)
f = open('./populations/after_%s.txt'%(file_id[4:6]), 'a+')
f.write('%s=%.5f\n'%(file_id, best_acc))
f.flush()
f.close()
if name == 'main':
RunModel.do_work(gpu_id='0', file_id='indi-gen00-no00')
do I need to write the main function in cifar10.py to implement this model
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.