How to use AUC metric in DeepHyper about deephyper

[Deathn0t]

I just released a new version of DeepHyper accessible on PyPI so if you just do pip install deephyper --upgrade you should be able to use these latest metrics.

[Deathn0t]

Could you please share with me the problem.py and the search space used

[anuragverma77]

Thank you so much. I will try auroc with the upgraded version.

[anuragverma77]

Hi @Deathn0t

After upgrading to deephyper 0.2.4
I have used Problem.metrics(['auroc']) and Problem.objective('val_auroc__last') in problem.py The model is training, with output like this after every eval :

2021-04-15 10:03:52.310779: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/gurobi/linux64/lib/:/opt/gurobi/linux64/lib
2021-04-15 10:03:52.310811: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2021-04-15 10:03:54.757293: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-04-15 10:03:54.757502: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/gurobi/linux64/lib/:/opt/gurobi/linux64/lib
2021-04-15 10:03:54.757522: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)
2021-04-15 10:03:54.757544: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (projects2azure): /proc/driver/nvidia/version does not exist
2021-04-15 10:03:54.757763: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-04-15 10:03:54.758862: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-04-15 10:03:55.472191: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2021-04-15 10:03:55.476643: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2693670000 Hz
train_X shape: (29327, 34)
train_y shape: (29327, 2)
valid_X shape: (12569, 34)
valid_y shape: (12569, 2)
DH-OUTPUT: 0.0

After this, I looked into results.csv, the objective column has all values = 0.0

and on using Problem.metrics(['aucpr']) and Problem.objective('val_aucpr__last') in problem.py The model is training, with output like this after every eval :

2021-04-15 10:16:36.767987: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/gurobi/linux64/lib/:/opt/gurobi/linux64/lib
2021-04-15 10:16:36.768026: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2021-04-15 10:16:39.146836: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-04-15 10:16:39.147028: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/gurobi/linux64/lib/:/opt/gurobi/linux64/lib
2021-04-15 10:16:39.147045: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)
2021-04-15 10:16:39.147061: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (projects2azure): /proc/driver/nvidia/version does not exist
2021-04-15 10:16:39.147261: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-04-15 10:16:39.148480: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-04-15 10:16:39.988206: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2021-04-15 10:16:39.993598: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2693670000 Hz
train_X shape: (29327, 34)
train_y shape: (29327, 2)
valid_X shape: (12569, 34)
valid_y shape: (12569, 2)
DH-OUTPUT: 1.0

After this, I looked into results.csv, the objective column has all values = 1.0

[Deathn0t]

Hello @anuragverma77 , do you have information about the evolution of the loss?
To see the loss of trained models you can use add verbose to the hyper parameters such as:

Problem.hyperparameters(
    ...
    verbose=1,
    ...
)

Also here is an example of a problem using AUROC and AUCPR: https://github.com/deephyper/NASBigData/blob/master/nas_big_data/attn/problem_ae.py

[anuragverma77]

Hi @Deathn0t

After upgrading to deephyper 0.2.4
I have used Problem.metrics(['auroc']) and Problem.objective('val_auroc__last') in problem.py The model is training, with output like this after every eval :

2021-04-16 13:55:42.527138: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/gurobi/linux64/lib/:/opt/gurobi/linux64/lib
2021-04-16 13:55:42.527178: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2021-04-16 13:55:45.587380: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-04-16 13:55:45.587711: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/gurobi/linux64/lib/:/opt/gurobi/linux64/lib
2021-04-16 13:55:45.587744: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)
2021-04-16 13:55:45.587781: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (projects2azure): /proc/driver/nvidia/version does not exist
2021-04-16 13:55:45.588091: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-04-16 13:55:45.589824: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-04-16 13:55:46.601087: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2021-04-16 13:55:46.606083: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2693670000 Hz
train_X shape: (29327, 34)
train_y shape: (29327, 2)
valid_X shape: (12569, 34)
valid_y shape: (12569, 2)

  1/230 [..............................] - ETA: 4:01 - loss: -0.0831 - auroc: 0.0000e+00
 22/230 [=>............................] - ETA: 0s - loss: -0.2750 - auroc: 0.0000e+00  
 40/230 [====>.........................] - ETA: 0s - loss: -0.3898 - auroc: 0.0000e+00
 61/230 [======>.......................] - ETA: 0s - loss: -0.4624 - auroc: 0.0000e+00
 81/230 [=========>....................] - ETA: 0s - loss: -0.5050 - auroc: 0.0000e+00
104/230 [============>.................] - ETA: 0s - loss: -0.5452 - auroc: 0.0000e+00
127/230 [===============>..............] - ETA: 0s - loss: -0.5818 - auroc: 0.0000e+00
151/230 [==================>...........] - ETA: 0s - loss: -0.6169 - auroc: 0.0000e+00
176/230 [=====================>........] - ETA: 0s - loss: -0.6059 - auroc: 0.0000e+00
199/230 [========================>.....] - ETA: 0s - loss: -0.5188 - auroc: 0.0000e+00
222/230 [===========================>..] - ETA: 0s - loss: -0.5564 - auroc: 0.0000e+00
230/230 [==============================] - 2s 5ms/step - loss: -0.5977 - auroc: 0.0000e+00 - val_loss: 1.2176 - val_auroc: 0.0000e+00
DH-OUTPUT: 0.0

After this, I looked into results.csv, the objective column has all values = 0.0

and on using Problem.metrics(['aucpr']) and Problem.objective('val_aucpr__last') in problem.py The model is training, with output like this after every eval :

2021-04-16 13:34:55.667678: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/gurobi/linux64/lib/:/opt/gurobi/linux64/lib
2021-04-16 13:34:55.667720: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2021-04-16 13:34:58.564415: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-04-16 13:34:58.564644: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/gurobi/linux64/lib/:/opt/gurobi/linux64/lib
2021-04-16 13:34:58.564666: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)
2021-04-16 13:34:58.564682: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (projects2azure): /proc/driver/nvidia/version does not exist
2021-04-16 13:34:58.564912: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-04-16 13:34:58.566016: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-04-16 13:34:59.563249: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2021-04-16 13:34:59.569014: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2693670000 Hz
train_X shape: (29327, 34)
train_y shape: (29327, 2)
valid_X shape: (12569, 34)
valid_y shape: (12569, 2)

  1/230 [..............................] - ETA: 3:53 - loss: -0.0831 - aucpr: 1.0000
 21/230 [=>............................] - ETA: 0s - loss: -0.2674 - aucpr: 1.0000  
 44/230 [====>.........................] - ETA: 0s - loss: -0.4081 - aucpr: 1.0000
 65/230 [=======>......................] - ETA: 0s - loss: -0.4726 - aucpr: 1.0000
 85/230 [==========>...................] - ETA: 0s - loss: -0.5125 - aucpr: 1.0000
105/230 [============>.................] - ETA: 0s - loss: -0.5468 - aucpr: 1.0000
129/230 [===============>..............] - ETA: 0s - loss: -0.5850 - aucpr: 1.0000
152/230 [==================>...........] - ETA: 0s - loss: -0.6179 - aucpr: 1.0000
177/230 [======================>.......] - ETA: 0s - loss: -0.6000 - aucpr: 1.0000
202/230 [=========================>....] - ETA: 0s - loss: -0.5147 - aucpr: 1.0000
225/230 [============================>.] - ETA: 0s - loss: -0.5674 - aucpr: 1.0000
230/230 [==============================] - 2s 5ms/step - loss: -0.5977 - aucpr: 1.0000 - val_loss: 1.2176 - val_aucpr: 1.0000
DH-OUTPUT: 1.0

After this, I looked into results.csv, the objective column has all values = 1.0

[anuragverma77]

Hi @Deathn0t,

Sorry for the late reply.
Below is the search_space.py

import collections

import tensorflow as tf

from deephyper.nas.space import AutoKSearchSpace
from deephyper.nas.space.node import ConstantNode, VariableNode
from deephyper.nas.space.op.basic import Tensor
from deephyper.nas.space.op.connect import Connect
from deephyper.nas.space.op.merge import AddByProjecting
from deephyper.nas.space.op.op1d import Dense, Identity


def add_dense_to_(node):
    node.add_op(Identity()) # we do not want to create a layer in this case

    activations = [None, tf.nn.relu, tf.nn.tanh, tf.nn.sigmoid]
    for units in range(16, 97, 16):
        for activation in activations:
            node.add_op(Dense(units=units, activation=activation))


def create_search_space(input_shape=(34,),
                        output_shape=(2,),
                        num_layers=10,
                        *args, **kwargs):
    
    # print("input_shape:", input_shape, ", output_shape:", output_shape,", num_layers:", num_layers)
    arch = AutoKSearchSpace(input_shape, output_shape, regression=False)
    source = prev_input = arch.input_nodes[0]

    # look over skip connections within a range of the 3 previous nodes
    anchor_points = collections.deque([source], maxlen=3)

    for _ in range(num_layers):
        vnode = VariableNode()
        add_dense_to_(vnode)

        arch.connect(prev_input, vnode)

        # * Cell output
        cell_output = vnode

        cmerge = ConstantNode()
        cmerge.set_op(AddByProjecting(arch, [cell_output], activation='relu'))

        for anchor in anchor_points:
            skipco = VariableNode()
            skipco.add_op(Tensor([]))
            skipco.add_op(Connect(arch, anchor))
            arch.connect(skipco, cmerge)

        # ! for next iter
        prev_input = cmerge
        anchor_points.append(prev_input)


    return arch


def test_create_search_space():
    """Generate a random neural network from the search_space definition.
    """
    from random import random
    from tensorflow.keras.utils import plot_model
    import tensorflow as tf

    search_space = create_search_space(num_layers=10)
    ops = [random() for _ in range(search_space.num_nodes)]

    print(f'This search_space needs {len(ops)} choices to generate a neural network.')

    search_space.set_ops(ops)

    model = search_space.create_model()
    model.summary()

    plot_model(model, to_file='sampled_neural_network.png', show_shapes=True)
    print("The sampled_neural_network.png file has been generated.")


if __name__ == '__main__':
    test_create_search_space()

Below is the problem.py

from deephyper.problem import NaProblem
from nas_problems_train_final.polynome2.load_data import load_data
from nas_problems_train_final.polynome2.search_space import create_search_space
from deephyper.nas.preprocessing import minmaxstdscaler,stdscaler
import numpy as np
import tensorflow as tf
tf.config.run_functions_eagerly(True)

# from deephyper.nas.train_utils import *
from sklearn import metrics

Problem = NaProblem(seed=2019)

Problem.load_data(load_data)

Problem.preprocessing(stdscaler)

Problem.search_space(create_search_space, num_layers=6)

cw = {0: 1.0, 1: 4.0}
Problem.hyperparameters(
    batch_size=128,
    learning_rate=0.001,
    optimizer='adam',
    verbose=1,
#     class_weight = cw,
    num_epochs=1,
    callbacks=dict(
        EarlyStopping=dict(
            monitor='val_loss', # or 'val_r2' or 'val_acc' ?
            # mode='max',
            verbose=0,
            patience=5,
            restore_best_weights=True
        )
    )
)

Problem.loss('categorical_crossentropy') # or 'mse' ?

def AUC(y_true,y_pred):
    y_true = tf.argmax(y_true, axis = 1)
    m = tf.keras.metrics.AUC(num_thresholds=3)
    m.update_state(y_true, y_pred[:,1])
    return m.result()

Problem.metrics(['aucpr']) # or 'acc' ?

Problem.objective('val_aucpr__last') # or 'val_r2__last' ?

# Just to print your problem, to test its definition and imports in the current python environment.
if __name__ == '__main__':
    print(Problem)

[Deathn0t]

I see that you want to use class_weights therefore you should do it this way:

Problem.loss(
    "categorical_crossentropy",
    class_weights={0: 1.0, 1: 4.0}
)

Also, do not hesitate to have multiple metrics, it can always help analyse the performance of trained model. For example I would do:

Problem.metrics(["acc", "auroc", "aucpr"])

In case there is an issue with the __last functionality of the objective can you try:

Problem.objective("val_aucpr")

Did you try with/without the preprocessing?
Do you have the performance of a baseline neural network on the same data?