ibm / ai-privacy-toolkit

A toolkit for tools and techniques related to the privacy and compliance of AI models.

ai-privacy-toolkit

A toolkit for tools and techniques related to the privacy and compliance of AI models.

The anonymization module contains methods for anonymizing ML model training data, so that when a model is retrained on the anonymized data, the model itself will also be considered anonymous. This may help exempt the model from different obligations and restrictions set out in data protection regulations such as GDPR, CCPA, etc.

The minimization module contains methods to help adhere to the data minimization principle in GDPR for ML models. It enables to reduce the amount of personal data needed to perform predictions with a machine learning model, while still enabling the model to make accurate predictions. This is done by by removing or generalizing some of the input features.

The dataset assessment module implements a tool for privacy assessment of synthetic datasets that are to be used in AI model training.

Official ai-privacy-toolkit documentation: https://ai-privacy-toolkit.readthedocs.io/en/latest/

Installation: pip install ai-privacy-toolkit

For more information or help using or improving the toolkit, please contact Abigail Goldsteen at [email protected], or join our Slack channel: https://aip360.mybluemix.net/community.

We welcome new contributors! If you're interested, take a look at our contribution guidelines.

Related toolkits:

ai-minimization-toolkit - has been migrated into this toolkit.

differential-privacy-library: A general-purpose library for experimenting with, investigating and developing applications in, differential privacy.

adversarial-robustness-toolbox: A Python library for Machine Learning Security. Includes an attack module called inference that contains privacy attacks on ML models (membership inference, attribute inference, model inversion and database reconstruction) as well as a privacy metrics module that contains membership leakage metrics for ML models.

Citation

Abigail Goldsteen, Ola Saadi, Ron Shmelkin, Shlomit Shachor, Natalia Razinkov, "AI privacy toolkit", SoftwareX, Volume 22, 2023, 101352, ISSN 2352-7110, https://doi.org/10.1016/j.softx.2023.101352.

ai-privacy-toolkit's Issues

Minimization - Improve performance of transform method

Currently loops over all samples and all features, which takes a long time for large datasets. Try to find a more efficient way to do this.

Minimization - Inconsistent NCP calculation

NCP score is calculated on overall generalizations (combined from all cells), this includes when looking for a feature to remove from generalization (also taken from overall generalizations).
Accuracy is computed on the representative values taken from cells. There can be a mismatch (i.e., generalization applied at cell level but not at overall level) which can lead to not achieving the target accuracy. Need separate NCP calculation for this purpose.

minimization - fix inconsistent one hot encoding

Different places in the code use either scikitlearn's OneHotEncoder (before fitting decision tree) or pandas get_dummies (in _modify_categorical_features). This could lead to inconsistent results. Need to use the same mechanism.

Update/new minimization notebooks to showcase new features

Categorical features + minimizing only some features

Wrappers - improve implementation of Model.get_nb_classes

To cover all cases of y format (one-hot encoded or not).

Fix attribute_inference_anonymization_nursery notebook

The social feature indexes are not computed correctly, leading to a 100% accuracy of the attack (the feature gets one hot encoded and then one column can be used to definitively infer the second column). This should be fixed to use a slice (21,22) as the attacked feature.

Wrappers: Add information to blackbox classifiers

Add option to receive (non-trained) model skeleton object OR specific model type name (e.g., sklearn decision tree) to enable building similar dummy model.

anonymization - support regression models

The current implementation only supports classifiers.

Look at possibilities for code reuse between anonymization and minimization modules

minimization - enable integer feature indexes

In numpy typically columns are numbered using integers. Currently our implementation converts these into strings (i.e., '0', '1', etc.) in the returned generalizations. This is due to the bahavior of scikitlearn's OneHotEncoder which differs between integers and strings. But it would be better to leave these as integer keys to be consistent with numpy.
One workaround is to store in the beginning of the code a boolean stating whether the column indexes were received as integers, and at the end cast the strings back to integers in the return value.

minimization - support scaled features

At the moment - minimizer receives scaled features and returns generalizations in the scaled domain. Need to think if there is a way to support returning generalizations in the original domain (perhaps by sending the minimizer the original data + list of columns to scale + scaling params so it can be performed internally).

minimization - Support additional frameworks

E.g., pytorch, Keras...

Enhancements to BlackboxClassifier

1. get_predictions + wrapping of the predict_fn + method that would return the wrapped method (get_predict_fn)
2. create self._art_model
3. put not None values in self._nb_classes and self._input_shape - use init params to fill
4. add an @AbstractMethod decorator to prevent instantiation of the base class
5. add into BlackboxClassifier loss and optimizer properties

Deal with not-needed imports and requirements

At the moment due to the project structure all requirements need to be present even to use just some of the toolkit's functionality. This is especially annoying for the underlying ML frameworks, which all need to be installed just to use one of them. Need to rearrange imports such that unused frameworks are not required.

minimization - add option to apply generalizations to training data and retrain model

This can enable a new flow where the model is also trained on the generalized feature domains. Thus the model is trained on the same type of data it will see during inference, which may improve overall accuracy.

Support scikit-learn 1.2

Minimization transform method requires X as positional argument as of version 1.2. This does not conform to our current usage with ArraDataset instead. Need to think of a solution to enable both modes but still conform to scikit-learn requirements.

Investigate warnings

tests/test_minimizer.py: 16473 tests with warnings
/site-packages/sklearn/base.py:450: UserWarning: X does not have valid feature names, but DecisionTreeClassifier was fitted with feature names

tests/test_minimizer.py: 1068 tests with warnings
/site-packages/sklearn/base.py:450: UserWarning: X does not have valid feature names, but DecisionTreeRegressor was fitted with feature names

tests/test_minimizer.py::test_keras_model
tests/test_model.py::test_keras_classifier
tests/test_model.py::test_keras_regressor
/site-packages/keras/engine/training_v1.py:2079: UserWarning: Model.state_updates will be removed in a future version. This property should not be used in TensorFlow 2.0, as updates are applied automatically.
updates=self.state_updates,

Add balanced accuracy scoring method to model wrappers

Wrapper for xgboost

minimization - Support 1-hot encoded features

Current implementation assumes numeric features only.

minimization - Support for generalizing only some of the input features

Some input features may not be personal data and therefore should not be generalized. Similar to QI definition in anonymization.

anonymization - support for additional methods for selecting representatives

For example: per feature mean, per feature median, overall mean, overall median, real closest to mean, real closest to median

anonymization - support 1-hot encoded features

At the moment the anonymization code refers to each column separately. This may cause inconsistent results for 1-hot encoded features (i.e., more than one 1 column or none). Need to address this so that the columns belonging to the same feature are anonymized in a consistent manner.

Support multi-label classification

Both anonymization and minimization only support a single label. Check whether we can add support for multi-label models as well.

Stop pruning one step before passing accuracy threshold

After calculating the initial accuracy achieved by the generalizations derived from the decision tree leaves, there are two directions in which the algorithm can proceed: improving accuracy (to get above the threshold) or improving generalizations (while reducing accuracy). The second direction will at some point will cause the accuracy to go below the desired threshold (the last iteration of the loop that is run). But when this happens, the generalizations need to be rolled back to the previous state, the one where the target accuracy was not passed. This can be done either by storing locally the generalizations and generalized data from the previous loop iteration and replacing them when the threshold is passed, or by only "speculatively" applying the generalizations in each iteration, and applying them finally only if the threshold was not passed.

minimization - support regression models

The current implementation only supports classifiers.

anonymization: Improve error thrown by Anonymizer

anonymizer line 75 - does not give useful indication of the underlying error. Be more specific.
For pandas - should be column names and not indexes.

Wrappers: deal with incompatible model outputs

In blackbox classifiers, if the ModelOutputType is passed as CLASSIFIER_PROBABILITIES or CLASSIFIER_LOGITS but the model actually outputs a 1D array of categorical scalars, throw an exception. Throw as soon as possible (in init if data is available, otherwise in predict)

minimization - add notebooks with examples