How to get the best model? about deephyper

[Deathn0t]

Hello @anuragverma77,

In fact, after the run is finished you have two options. Looking at the results.csv file or parsing the log file. If you look at the CSV file:

df = pd.read_csv("results.csv")
i_best = df.objective.argmax()
row_best = df.iloc[i_best].tolist()
objective = row_best[-2]
arch_seq = row_best[:-2]

print("Best Objective: ", objective)
print("Best Arch: ", arch_seq)

The "arch_seq" is an embedding of the best architecture corresponding to the search space used in the Problem definition. Then, to instantiate a Keras model you can do:

model = Problem.get_keras_model(arch_seq)

I hope it helps!

[Deathn0t]

So when you followed the tutorial you executed the following command:

deephyper new-problem nas polynome2

This command will create a folder called polynome which contains different files (like a default project template). One of these file is called problem.py when you execute later:

deephyper nas regevo --evaluator ray --problem nas_problems.polynome2.problem.Problem --max-evals 100

the --problem nas_problems.polynome2.problem.Problem argument refer to the Problem object inside this problem.py file.
To use the Problem object you can therefore do the following from Python:

from nas_problems.polynome2.problem import Problem

...
some code
...

model = Problem.get_keras_model(arch_seq)

[Deathn0t]

Yes that's correct, the result is just a neural network architecture, but the trained parameters are not available by default. We could add this feature in the future. For now, users re-train the best architecture with more data and tend to use less data for the NAS to reduce the cost of the evaluations. The goal of NAS evaluations is mainly to have an estimate of the performance of each neural network.

Also, saving all trained parameters for each architecture can become very costly or take a lot of time if you have many evaluations in parallel.

A possible solution is to use the Keras ModelCheckpoint from Keras and add it to the problem definition.

[anuragverma77]

I found the fix.
arch_seq = [21.0, 0.0, 15.0, 0.0, 1.0, 8.0, 0.0, 0.0, 0.0]
should be changed to all integers
arch_seq = [int(x) for x in arch_seq]

so new arch_seq = [21, 0, 15, 0, 1, 8, 0, 0, 0]

[anuragverma77]

Hi @Deathn0t ,

Thanks a lot for giving your time.

I am not sure where/how to run this command:
model = Problem.get_keras_model(arch_seq)
After running in linux terminal:
deephyper nas regevo --evaluator ray --problem nas_problems.polynome2.problem.Problem --max-evals 100
the Problem object is lost in space

[anuragverma77]

Thanks @Deathn0t ,

I did exactly mentioned by you:

from nas_problems.polynome2.problem import Problem
import pandas as pd

df = pd.read_csv("results.csv")
i_best = df.objective.argmax()
row_best = df.iloc[i_best].tolist()
objective = row_best[-2]
arch_seq = row_best[:-2]

model = Problem.get_keras_model(arch_seq)
model.save('best_model.h5')

from keras.models import load_model
model = load_model("best_model.h5")

The output I am getting is: WARNING:tensorflow:No training configuration found in the save file, so the model was *not* compiled. Compile it manually.

I can get the architecture using model.summary() but it's not a trained model. model.optimizer gives no output.

[anuragverma77]

Thanks a lot @Deathn0t . I was stuck in this since a week.

[anuragverma77]

Hi @Deathn0t ,

I hope you are doing well.

I'm facing an issue while using Problem.get_keras_model(arch_seq) for a regevo algorithm . There is no issue for a random algorithm.

On using deephyper nas random --evaluator ray --problem nas_problems_train_final.polynome2.problem.Problem --max-evals 100 for a categorical problem having 2 output class, I am able to get the best model

Best Objective:  0.7957990169525146
Best Arch:  [0.4778779951921, 0.07639606523385922, 0.6510517383493892, 0.8246633960523665, 0.9247976565950988, 0.6426766935951087, 0.0016192013406978178, 0.1284215170414026, 0.032100961222626734]

On using deephyper nas regevo--evaluator ray --problem nas_problems_train_final.polynome2.problem.Problem --max-evals 100 for a categorical problem having 2 output class, I am not able to get the best model

Best Objective:  0.7994987368583679
Best Arch:  [21.0, 0.0, 15.0, 0.0, 1.0, 8.0, 0.0, 0.0, 0.0]

When I run Problem.get_keras_model(arch_seq) I am getting the following error.

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-27-18287f3280ac> in <module>
----> 1 model = Problem.get_keras_model(arch_seq)

/usr/local/lib/python3.6/dist-packages/deephyper/problem/neuralarchitecture.py in get_keras_model(self, arch_seq)
    376         """
    377         search_space = self.build_search_space()
--> 378         search_space.set_ops(arch_seq)
    379         keras_model = search_space.create_model()
    380         return keras_model

/usr/local/lib/python3.6/dist-packages/deephyper/nas/space/keras_search_space.py in set_ops(self, indexes)
    110 
    111         for op_i, node in zip(indexes, self.variable_nodes):
--> 112             node.set_op(op_i)
    113 
    114         for node in self.mime_nodes:

/usr/local/lib/python3.6/dist-packages/deephyper/nas/space/node.py in set_op(self, index)
    109 
    110     def set_op(self, index):
--> 111         self.get_op(index).init(self)
    112 
    113     def get_op(self, index):

/usr/local/lib/python3.6/dist-packages/deephyper/nas/space/node.py in get_op(self, index)
    116         ), f"found type is : {type(index)}"
    117         if "float" in str(type(index)):
--> 118             self._index = self.denormalize(index)
    119         else:
    120             assert 0 <= index and index < len(

/usr/local/lib/python3.6/dist-packages/deephyper/nas/space/node.py in denormalize(self, index)
    136             return index
    137         else:
--> 138             assert 0.0 <= index and index <= 1.0
    139             res = int(index * len(self._ops))
    140             if index == 1.0:

AssertionError: 

Could you please help me out?

[Deathn0t]

Hello @anuragverma77, yes exactly you got the workaround yourself! This will be directly handled in the next release! I am preparing some more documentation before releasing the new version.

[anuragverma77]

Hi @Deathn0t

For categorical problem I used class_weight and accuracy as metric and objective but I want to use auc as metrics. When I used it's value remains constant at = -3.4028235e+38 and it does not gives any best model because all the objective values are same in results.csv file.

So I tried with custom function
auctf = tf.keras.metrics.AUC(num_thresholds=200, curve='ROC', summation_method='interpolation', name='auctf', dtype=None, thresholds=None, multi_label=False, label_weights=None)
this function worked in normal keras problem but is not working in deephyper

I used Problem.metrics(['auctf']) and Problem.objective('val_auctf__last') but still same problem auc value remains constant at = -3.4028235e+38
Then another auc function

def aucnew(y_true, y_pred):
    try:
        auc = tf.py_function(metrics.roc_auc_score, (y_true[:,1], y_pred[:, 1]), tf.float32)
    except:
        auc = 0
    return auc

I used Problem.metrics(['aucnew']) and Problem.objective('val_aucnew__last') but still same problem auc value remains constant at = -3.4028235e+38

Do you know if there is any way to use auc as metric and objective?

[Deathn0t]

Hi @anuragverma77 ,

In fact, yes since Tensorflow 2 they changed the metric interface and this creates some issues. I have a few questions:

1. did you use the pip install deephyper installation or did you pull the repo?
2. did you manage to have some printing of errors about why it is failing? a way to force this is to set --evaluator subprocess

I recently added these kind of metrics in the develop version under the names if you want I can help you get this develop versions to directly have them.

[Deathn0t]

Also you can pass a "callable" function as objective, see the documentation here. This way you can use scikit-learn code to compute the AUC.

[anuragverma77]

Hi @Deathn0t ,

I used pip install deephyper

used a custom auc function:

from sklearn import metrics

def aucnew(y_true, y_pred):
    try:
        auc = tf.py_function(metrics.roc_auc_score, (y_true[:,1], y_pred[:, 1]), tf.float32)
    except:
        auc = 0
    return auc

Problem.metrics([aucnew])

Problem.objective('val_aucnew__last') # or 'val_r2__last' ?

The output/error after running deephyper nas regevo --evaluator subprocess --problem nas_problems_train_final.polynome2.problem.Problem --max-evals 1 is

---

train_X shape: (33517, 36)
train_y shape: (33517, 2)
valid_X shape: (8379, 36)
valid_y shape: (8379, 2)
This search doesn't have an exiting procedure...
2021-04-14 09:53:10.712648: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/gurobi/linux64/lib/:/opt/gurobi/linux64/lib
2021-04-14 09:53:10.712684: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2021-04-14 09:53:13.171812: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-04-14 09:53:13.172017: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/gurobi/linux64/lib/:/opt/gurobi/linux64/lib
2021-04-14 09:53:13.172044: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)
2021-04-14 09:53:13.172060: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (projects2azure): /proc/driver/nvidia/version does not exist
2021-04-14 09:53:13.172263: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-04-14 09:53:13.174212: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-04-14 09:53:13.731824: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2021-04-14 09:53:13.734052: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2693670000 Hz
2021-04-14 09:53:13.828148: W tensorflow/core/framework/op_kernel.cc:1751] Invalid argument: ValueError: continuous format is not supported
Traceback (most recent call last):

  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 247, in __call__
    return func(device, token, args)

  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 135, in __call__
    ret = self._func(*args)

  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/impl/api.py", line 620, in wrapper
    return func(*args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/sklearn/utils/validation.py", line 63, in inner_f
    return f(*args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_ranking.py", line 550, in roc_auc_score
    sample_weight=sample_weight)

  File "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_base.py", line 74, in _average_binary_score
    raise ValueError("{0} format is not supported".format(y_type))

ValueError: continuous format is not supported


train_X shape: (33517, 36)
train_y shape: (33517, 2)
valid_X shape: (8379, 36)
valid_y shape: (8379, 2)
Epoch 1/5
Uncaught exception <class 'tensorflow.python.framework.errors_impl.InvalidArgumentError'>:  ValueError: continuous format is not supported
Traceback (most recent call last):

  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 247, in __call__
    return func(device, token, args)

  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 135, in __call__
    ret = self._func(*args)

  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/impl/api.py", line 620, in wrapper
    return func(*args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/sklearn/utils/validation.py", line 63, in inner_f
    return f(*args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_ranking.py", line 550, in roc_auc_score
    sample_weight=sample_weight)

  File "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_base.py", line 74, in _average_binary_score
    raise ValueError("{0} format is not supported".format(y_type))

ValueError: continuous format is not supported


         [[node EagerPyFunc (defined at home/a.verma/nas_problems_train_final/nas_problems_train_final/polynome2/problem.py:54) ]] [Op:__inference_train_function_1038]

Function call stack:
train_function
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/deephyper/evaluator/runner.py", line 36, in <module>
    retval = func(d)
  File "/usr/local/lib/python3.6/dist-packages/deephyper/nas/run/alpha.py", line 107, in run
    history = trainer.train(with_pred=with_pred, last_only=last_only)
  File "/usr/local/lib/python3.6/dist-packages/deephyper/nas/trainer/train_valid.py", line 436, in train
    validation_steps=self.valid_steps_per_epoch,
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py", line 1100, in fit
    tmp_logs = self.train_function(iterator)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 828, in __call__
    result = self._call(*args, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 888, in _call
    return self._stateless_fn(*args, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 2943, in __call__
    filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1919, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 560, in call
    ctx=ctx)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError:  ValueError: continuous format is not supported
Traceback (most recent call last):

  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 247, in __call__
    return func(device, token, args)

  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 135, in __call__
    ret = self._func(*args)

  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/impl/api.py", line 620, in wrapper
    return func(*args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/sklearn/utils/validation.py", line 63, in inner_f
    return f(*args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_ranking.py", line 550, in roc_auc_score
    sample_weight=sample_weight)

  File "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_base.py", line 74, in _average_binary_score
    raise ValueError("{0} format is not supported".format(y_type))

ValueError: continuous format is not supported


         [[node EagerPyFunc (defined at home/a.verma/nas_problems_train_final/nas_problems_train_final/polynome2/problem.py:54) ]] [Op:__inference_train_function_1038]

Function call stack:
train_function

[anuragverma77]

Thanks, I would be glad to have develop version.

[Deathn0t]

The second topic of this conversation about AUC metric was moved to #64