testingautomated-usi / uncertainty-wizard Goto Github PK

tensorflow_probability can be used seamlessly using StochasticModels.

This should be explained in the docs and a runnable example should be provided.

Executing stochastic_from_keras in different processes at the same time, using default parameters, can lead to failures, as both processes use the same temporary weight path.

A workaround exists: Just pass custom, differing (e.g. random) temp_weights_path when calling stochastic_from_keras.

Ensemble models should expose a possibility to run passed tasks only on specifically selected atomic models,
for example by specifying an iterable of model ids.

Hello,

Nice project !

I am trying to make it work for a multi-input keras model, represented by a list of numpy arrays.

Unfortunately, I am getting this following error:

ValueError: At the moment, uwiz stochastic models support only (unbatched)numpy arrays and tf.data.Datasets as inputs. Please transform your input in one of these forms

Is there any workaround for making this project work for multi-input?

If the main thread initializes TensorFlow in its default settings, all GPU space will be used up by it, disallowing LazyEnsemble multiprocessing to run on that GPU.

Thus, the user can typically do one of the following:

• Make GPU invisible
• Configure TF not to use GPU or to allow gpu allocation growth

Both these changes are not particularly user friendly. However, uncertainty-wizard offers a utility function which can be used for this: DynamicGpuGrowthContextManager.enable_dynamic_gpu_growth().

Simply calling this utility method will configure tf to use dynamic gpu memory growth. This is probably easier and nicer for the end-user.
It will also allow increasing end-user code version compatibility with future versions of tf: TensorFlows GPU config is still experimental and future versions may require different imports (which will be dynamically resolved in future uncertainty-wizard versions based on the used tf version).

This should be described in the docs and also added to the examples.

Such class already exists (AbstractStochastic), but is not yet exposed to user.

There appear to be problems with the multi-gpu configuration with newer tensorflow versions (identified with tf 2.10).

• The gpu api is not experimental anymore, so this can be refactored
• For some reason, the virtual devices are now already created when we want to create them, leading to an error. But some new APIs should help around that. Need to investigate.

Such an interface would allow running tasks without loading or persisting any model.

Advantage: Pre- and post-processing tasks that don't rely on a specific keras model (but are still model dependent, i.e., they should run for every model id) will run faster (no model serialization or deserialization) and can potentially be more heavily parallelized (as memory is not filled with keras models).

The methods predict_quantified on stochastic models and predict_quantified and quantify_predictions on ensembles should accept an optional flag (default False) allowing the following behavior:

• The results are returned as Dict[str, Dict[str, np.ndarray]], where the outer dict keys are a particular quantifiers alias and the inner dict keys are 'prediction' and 'quantification' (see the example below)
• If a quantifier was passed as an alias (str), the same alias should be used as dict key. If the quantifier was passed as an object, the first alias of the class should be used.

E.g., the return value of predict_quantified(quantifiers=['var_ratio', PredictiveEntropy()], as_dict=True) should be

{
     "var_ratio": {"prediction": <pred_array>, "quantification": <quantification_array>},
     "predictive_entropy": {"prediction": <pred_array>, "quantification": <quantification_array>}
}

The goal of this issue is to allow to generate outputs that are self-explanatory and do not need to remember the order of the passed quantifiers.

the VariationRatio quantifier misses the corresponding class-name alias, which should be there, according to the docs:

https://uncertainty-wizard.readthedocs.io/en/latest/user_guide_quantifiers.html