leeyeehoo / gennas Goto Github PK

Thanks for the very interesting work. I'm having a hard time finding the imagenet accuracies or ranks of networks used to calculate the spearman correlations for the NDS imagenet part of table 2 in the paper. When I download the NDS data from the link, I get a JSON per NAS space, but I can only find accuracies for CIFAR models there (as far as I understand). Can you please reference the right location?

Hello，I really enjoyed your paper!
But I have a question about the file nasbench1_search_20samples.json, how do you sample the architectures in the file on the NASBench101.

I get this error when I run the code with Torch 1.11 with CUDA 11.3:

$ python do_search.py --search_space=nasbench101 --json_path=data/nasbench101/nasbench1_search_20samples.json --job_description=nasbench1_search_20samples
[23 03:05:49] args = Namespace(batch_size=16, dataset='cifar10', device='cuda:0', eval_interval=10, eval_weights=[0.25, 0.5, 1.0], evolve_size=400, gpu=0, init_b_type='none', init_channels=16, init_w_type='none', input_size=32, job_description='nasbench1_search_20samples', json_path='data/nasbench101/nasbench1_search_20samples.json', last_channels=64, learning_rate=0.1, length=8, momentum=0.9, mutate_ratio=0.8, nds_path='../../GenNAS/data/nds_data/', nlp_path='data/nasbenchnlp', output_size=8, pad=False, population_size=50, samples=20, search_space='nasbench101', seed=1, total_iters=100, tournament_size=10, train_weights=[0.25, 0.5, 1.0], weight_decay=4e-05, workers=2)
Files already downloaded and verified
Files already downloaded and verified
[23 03:05:55] begin generating the polulation
Files already downloaded and verified
Files already downloaded and verified
[23 03:06:08] 0 [603.6029052734375, 18.78582191467285, 1.7875449657440186, 1.084944248199463, 0.9210254549980164, 0.8724834322929382, 0.8407966494560242, 0.8279027342796326, 0.8090664148330688, 0.8027111291885376]
[23 03:06:11] 1 [7.916046142578125, 6.412542819976807, 2.0949835777282715, 1.2406392097473145, 0.9746987223625183, 0.912070631980896, 0.8887152671813965, 0.8689042925834656, 0.8529442548751831, 0.8387445211410522]
[23 03:06:14] 2 [7963.259765625, 12.292606353759766, 1.4073045253753662, 1.0705939531326294, 1.0319645404815674, 0.9279698729515076, 0.904671847820282, 0.8742901086807251, 0.8570302724838257, 0.8436279296875]
[23 03:06:20] 3 [1016.9471435546875, 940.298095703125, 77.07316589355469, 4.334371089935303, 1.261746883392334, 1.0088449716567993, 0.9369798898696899, 0.8920031785964966, 0.8651416301727295, 0.8406433463096619]
[23 03:06:22] 4 [101.91466522216797, 41.327049255371094, 2.255126953125, 1.3973069190979004, 1.1995346546173096, 1.13328218460083, 1.0910677909851074, 1.069493293762207, 1.0351336002349854, 1.0279858112335205]
[23 03:06:26] 5 [68.38812255859375, 3.78884220123291, 1.437398910522461, 1.0700480937957764, 1.0040539503097534, 0.9591802358627319, 0.9330179691314697, 0.8953936696052551, 0.8718138933181763, 0.8582808375358582]
[23 03:06:30] 6 [35244.65625, 18515.8515625, 413.9317626953125, 11.979930877685547, 1.5659211874008179, 1.0613737106323242, 0.9780424237251282, 0.993895411491394, 0.9229803085327148, 0.8759559392929077]
[23 03:06:34] 7 [3677107.5, 211679.890625, 760.4144287109375, 5.534331798553467, 2.4527201652526855, 1.7783145904541016, 1.1818974018096924, 1.0777795314788818, 1.0919148921966553, 1.0736695528030396]
[23 03:06:39] 8 [1529877.125, 236449.78125, 5800.85595703125, 129.42852783203125, 4.628438949584961, 1.0668970346450806, 0.9529215097427368, 0.9630799293518066, 0.8951679468154907, 0.8593810796737671]
[23 03:06:42] 9 [35.36070251464844, 2.827066421508789, 1.2877510786056519, 1.0069392919540405, 0.9152586460113525, 0.8782352209091187, 0.8488408327102661, 0.8283950686454773, 0.8130360841751099, 0.8044896125793457]
[23 03:06:46] 10 [5013602.5, 16187.2802734375, 139.59803771972656, 2.849884510040283, 1.1417686939239502, 1.0377901792526245, 0.9177370667457581, 0.8702065944671631, 0.8509625792503357, 0.8414421081542969]
[23 03:06:50] 11 [73.81085968017578, 50.359012603759766, 3.6614279747009277, 1.3720015287399292, 1.006035327911377, 0.8994525671005249, 0.8598047494888306, 0.8355705738067627, 0.8226103186607361, 0.8156954050064087]
[23 03:06:55] 12 [36629.6875, 152.09881591796875, 6.923776626586914, 1.547558069229126, 1.1278163194656372, 1.0172944068908691, 0.9815342426300049, 0.9631819725036621, 0.95339035987854, 0.9517905712127686]
Traceback (most recent call last):
File "do_search.py", line 147, in
tau,spearmanr,preds = trainval(archs_accs, model_builder, task, evaluator)
File "do_search.py", line 61, in trainval
losses = evaluator.evaluate(task,model_builder,arch)
File "C:\Users\spook\Dropbox\Desktop\GenNAS\builder_evaluator.py", line 30, in evaluate
return self.evaluate_cv(task,model_builder,arch)
File "C:\Users\spook\Dropbox\Desktop\GenNAS\builder_evaluator.py", line 54, in evaluate_cv
loss.backward()
File "C:\Users\spook\Dropbox\Desktop\GenNAS\gennas\lib\site-packages\torch_tensor.py", line 307, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "C:\Users\spook\Dropbox\Desktop\GenNAS\gennas\lib\site-packages\torch\autograd_init_.py", line 156, in backward
allow_unreachable=True, accumulate_grad=True) # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [16, 64, 8, 8]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

Getting this issue while running: python do_explore.py --search_space=nasbench101 --config=CONF_NB101 (Just replicated the code). Here are the traceback lines for debug:

[W python_anomaly_mode.cpp:104] Warning: Error detected in ReluBackward0. Traceback of forward call that caused the error:
File "/export/hdd/scratch/tshah74/GenNAS/do_explore.py", line 96, in
losses = evaluator.evaluate(task,model_builder,arch_info)
File "/export/hdd/scratch/tshah74/GenNAS/builder_evaluator.py", line 34, in evaluate
output = model_builder.learn(model,data)
File "/export/hdd/scratch/tshah74/GenNAS/builder_model.py", line 154, in learn
return model(x)
File "/usr/scratch/tshah74/miniconda/envs/genNAS-rep/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/export/hdd/scratch/tshah74/GenNAS/pynbs/model_nb101_tri_bn.py", line 66, in forward
x = layer(x)
File "/usr/scratch/tshah74/miniconda/envs/genNAS-rep/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/export/hdd/scratch/tshah74/GenNAS/pynbs/model_nb101_tri_bn.py", line 131, in forward vertex_output = self.vertex_opt
File "/usr/scratch/tshah74/miniconda/envs/genNAS-rep/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/export/hdd/scratch/tshah74/GenNAS/pynbs/base_ops.py", line 43, in forward
x = self.conv1x1(x)
File "/usr/scratch/tshah74/miniconda/envs/genNAS-rep/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/export/hdd/scratch/tshah74/GenNAS/pynbs/base_ops.py", line 22, in forward
return self.conv_bn_relu(x)
File "/usr/scratch/tshah74/miniconda/envs/genNAS-rep/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/scratch/tshah74/miniconda/envs/genNAS-rep/lib/python3.9/site-packages/torch/nn/modules/container.py", line 141, in forward
input = module(input)
File "/usr/scratch/tshah74/miniconda/envs/genNAS-rep/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/scratch/tshah74/miniconda/envs/genNAS-rep/lib/python3.9/site-packages/torch/nn/modules/activation.py", line 98, in forward
return F.relu(input, inplace=self.inplace)
File "/usr/scratch/tshah74/miniconda/envs/genNAS-rep/lib/python3.9/site-packages/torch/nn/functional.py", line 1299, in relu
result = torch.relu(input)
(function _print_stack)

Traceback (most recent call last): File "/export/hdd/scratch/tshah74/GenNAS/do_explore.py", line 96, in losses = evaluator.evaluate(task,model_builder,arch_info)
File "/export/hdd/scratch/tshah74/GenNAS/builder_evaluator.py", line 37, in evaluate loss.backward()
File "/usr/scratch/tshah74/miniconda/envs/genNAS-rep/lib/python3.9/site-packages/torch/_tensor.py", line 307, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "/usr/scratch/tshah74/miniconda/envs/genNAS-rep/lib/python3.9/site-packages/torch/autograd/init.py", line 154, in backward Variable._execution_engine.run_backward( RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [16, 64, 8, 8]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

Your instructions refer to a requirement.txt file but there isn't one in your repo.

Hello Yuhong,

I ran the do_explore.py and it gave me two files: record.json and train-21111636441957
Could you please guide me on how can I extract the best model(s) from these?