Trouble Registering a Tensorflow Model about piml-toolbox

[ZebinYang]

Hi @faradawn

PiML only supports sklearn style models.
You can wrap your tensorflow models by following codes.

from tensorflow.keras.wrappers.scikit_learn import KerasClassifier

model = KerasClassifier(build_fn = lambda : dnn_model, verbose=False)
model.fit(train_x, train_y)
model._estimator_type = "classifier"

pipeline_1 = exp.make_pipeline(model=model, train_x=train_x, train_y=train_y.ravel(),
                             test_x=test_x, test_y=test_y.ravel(),
                             feature_names=feature_names, target_name="latency")
exp.register(pipeline_1, "my-dnn")

After that, you will be able to do model_explain and model_diagnose. However, this model cannot be interpreted by exp.model_interpret, as it is not a built-in model of PiML.

[faradawn]

Hi Zebin,

Thank you for suggesting a scikit_learn wrapper!

normalizer = layers.Normalization()
# normalizer.adapt(np.array(train_x)) # this causes model_to_estimator to fail
dnn_model = keras.Sequential([
        normalizer,
        layers.Dense(512, activation='relu', input_dim=train_x.shape[1]),
        layers.Dense(512, activation='relu'),
        layers.Dense(1, activation='sigmoid')
])

dnn_model.compile(loss='binary_crossentropy', optimizer=keras.optimizers.Adam(learning_rate=0.0001), metrics=['accuracy'])

from tensorflow.keras.wrappers.scikit_learn import KerasClassifier
model = KerasClassifier(build_fn = lambda : dnn_model, verbose=False)
model.fit(train_x, train_y)
model._estimator_type = "classifier"

from piml import Experiment
exp = Experiment()

pipeline_1 = exp.make_pipeline(model=model, train_x=train_x, train_y=train_y.ravel(),
                             test_x=test_x, test_y=test_y.ravel(),
                             feature_names=feature_names, target_name="latency")
exp.register(pipeline_1, "my-dnn")

I tried but received a normalization layer problem:

ValueError                                Traceback (most recent call last)
[<ipython-input-8-99a663406776>](https://localhost:8080/#) in <cell line: 23>()
     21                              test_x=test_x, test_y=test_y.ravel(),
     22                              feature_names=feature_names, target_name="latency")
---> 23 exp.register(pipeline_1, "my-dnn")
     24 
     25 

4 frames
[/usr/local/lib/python3.9/dist-packages/keras/engine/training.py](https://localhost:8080/#) in tf__predict_function(iterator)
     13                 try:
     14                     do_return = True
---> 15                     retval_ = ag__.converted_call(ag__.ld(step_function), (ag__.ld(self), ag__.ld(iterator)), None, fscope)
     16                 except:
     17                     do_return = False

ValueError: in user code:

    File "/usr/local/lib/python3.9/dist-packages/keras/engine/training.py", line 2169, in predict_function  *
        return step_function(self, iterator)
    File "/usr/local/lib/python3.9/dist-packages/keras/engine/training.py", line 2155, in step_function  **
        outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/usr/local/lib/python3.9/dist-packages/keras/engine/training.py", line 2143, in run_step  **
        outputs = model.predict_step(data)
    File "/usr/local/lib/python3.9/dist-packages/keras/engine/training.py", line 2111, in predict_step
        return self(x, training=False)
    File "/usr/local/lib/python3.9/dist-packages/keras/utils/traceback_utils.py", line 70, in error_handler
        raise e.with_traceback(filtered_tb) from None

    ValueError: Exception encountered when calling layer 'normalization_4' (type Normalization).
    
    Dimensions must be equal, but are 8 and 9 for '{{node sequential_3/normalization_4/sub}} = Sub[T=DT_FLOAT](IteratorGetNext, sequential_3/normalization_4/sub/y)' with input shapes: [32,8], [1,9].
    
    Call arguments received by layer 'normalization_4' (type Normalization):
      • inputs=tf.Tensor(shape=(32, 8), dtype=float32)

I can try to debug more. But the method you suggested was making progress!

Thank you very much, and will let you know if the issue is solved!

[faradawn]

Hi,
The tensorflow.keras.wrappers.scikit_learn wrapper allowed me to register a TensorFlow model! After removing the normalization layer, the problem above was resolved.

dnn_model = keras.Sequential([
        layers.Dense(512, activation='relu', input_dim=train_x.shape[1]),
        layers.Dense(512, activation='relu'),
        layers.Dense(1, activation='sigmoid')
])

dnn_model.compile(loss='binary_crossentropy', optimizer=keras.optimizers.Adam(learning_rate=0.0001), metrics=['accuracy'])

from tensorflow.keras.wrappers.scikit_learn import KerasClassifier
model = KerasClassifier(build_fn = lambda : dnn_model, verbose=False)
model.fit(train_x, train_y)
model._estimator_type = "classifier"

from piml import Experiment
exp = Experiment()

pipeline_1 = exp.make_pipeline(model=model, train_x=train_x, train_y=train_y.ravel(),
                             test_x=test_x, test_y=test_y.ravel(),
                             feature_names=feature_names, target_name="latency")
exp.register(pipeline_1, "my-dnn")

And the output is

  model = KerasClassifier(build_fn = lambda : dnn_model, verbose=False)
156/156 [==============================] - 0s 2ms/step
156/156 [==============================] - 0s 3ms/step
Register my-dnn Done

May I ask two more questions:

1. Why global-explainability is not showing?

2. Is there a way to silence the training output during exp.model_explain()?

I read that we can set model.fit(verbose = 0) during training. But how to specify this parameter when PIML calls this model?

Thanks!

[ZebinYang]

Hi @faradawn,

It seems that the verbose parameter in KerasClassifier is not passed to its predict function.
To turn off the messages, a workaround is to override the predict functions manually, see below.

def wrap_predict(x):
    proba = model.model.predict(x, verbose=False)
    if proba.shape[-1] > 1:
        classes = proba.argmax(axis=-1)
    else:
        classes = (proba > 0.5).astype("int32")
    return model.classes_[classes]

def wrap_predict_proba(x):
    probs = model.model.predict(x, verbose=False)
    # check if binary classification
    if probs.shape[1] == 1:
        # first column is probability of class 0 and second is of class 1
        probs = np.hstack([1 - probs, probs])
    return probs

model.predict = wrap_predict
model.predict_proba = wrap_predict_proba

The explainability output can be empty if the model output is constant. I tried on my side the built-in 'CoCircles' dataset and the output looks good. Can you double-check the data and model you used? Or just put a reproducible Colab link here.

from piml import Experiment

exp = Experiment()
exp.data_loader("CoCircles", silent=True)
exp.data_summary(silent=True)
exp.data_prepare(silent=True)
feature_names = exp.get_feature_names()
train_x, train_y, train_sample_weight = exp.get_data(train=True)
test_x, test_y, test_sample_weight = exp.get_data(test=True)

[faradawn]

Hi Zebin,

The wrap_predict worked perfectly in that it suppressed the training output!

Regarding the empty permutation feature importance, I created a Colab Notebook that produces the empty graph.

I re-checked the dataset and the model which uses 8 features to make a 0/1 binary prediction.

You already helped so much. Truly appreciate any guidance!

Thanks!

[ZebinYang]

Hi @faradawn

It seems your response variable only has one class

Also, for neural network models, we usually standardize x before model training.

[faradawn]

Hi Zebin,

Got that it was due to y having a single class! Thanks for finding that out!

The issue is resolved. Appreciate your numerous help, such as teaching how to wrap a tensor flow model into skilearn!

Can close issue anytime!