thoughtworks-datakind / anonymizer Goto Github PK

• Build wheel package
• Create docker-compose for spark cluster
• Update SparkSession builder to use new configurations
• Update Readme with working commands to spark-submit to cluster
• Set up circleci
• Run e2e test

Design a contract for supporting different inputs and formats
Decide on a viable output format for the next step (ex pandas df)

As an anonymizer user,
When I run the application,
Anonymized output needs to be created in the output directory with the name <input_file_name>_anonymized_.csv.
input_file_name here is obtained from the config json that is passed as an input to the application.
output_directory needs to be obtained from the config json file as a parameter in the 'anonymize' section.

Considerations:

1. An empty output file needs to be created if the input file is empty
2. If no PII detected in the fields, the same input file needs to be available as the anonymized version.
3. In the course of errors during anonymization, no file should be created and appropriate error message should be available to the user.

Capability to identify following PII data related to Singaporeans

• NRIC / FIN
• Work Permit Id
• Passport

As far as I can see, if the directory in the output path specified in the config file does not exist then there is an error as like:

FileNotFoundError: [Errno 2] No such file or directory: '<parent_directory>/anonymizer/output/comma_delimited_file_anonymized_.csv'

Adding a control rule that is like create if not exist would be useful to pass this error.

This story assumes that anonymisation would happen right after detection and not stop at the report generation step

Compare support for :

1. basic PII like name, email, phone number, NRIC, etc
2. custom PII identification
3. localization (esp. Asia) - support for PDPA policies
4. free-text PII identification
5. Python

KPIs :

1. True positives vs false positives
2. Ease of use and configuration
3. Easy to extend
4. Subsets that are identified accurately

Tools to explore :

1. PII analyzer
2. CommonRegex
3. Spacy
4. FuzzyWuzzy
5. de-identify from Google
6. presidio from Microsoft

Requirements and findings consolidated in this document

Add a test that takes in csv data with PII in it and a json config, parses them and generates output data with no PII information

Features to compare based on -

1. Support for basic PII like name, email, phone number, NRIC, etc
2. Support for custom PII identification
3. Support for localization (esp. Asia) - support for PDPA policies
4. Support for free text PII identification
5. Python preferably

KPIs

1. True positives vs false positives
2. Ease of use and configuration
3. Easy to extend
4. Subsets that are identified accurately

Tools to explore -

1. PII analyzer
2. CommonRegex
3. Spacy
4. FuzzyWuzzy
5. deidentify

Updating context and having sufficient on How to use and Setup this tool.
Also why certain things / design decisions were made.

• acquire
• analyze
• anonymize
  report
• write

Acceptance criteria: output from spark is the same as from pandas previously

Add a section in the schema that can hold the information on whether a given column has to be taken into account for PII detection or not.

Take parser output,
Split it if required in an efficient way
Run it against all the available regex matchers
Record the result whether a particular cell is PII or not

Allows specifying output file format to be csv or parquet

As part of this story, we would build an anonymizer action that hashes out PII elements in the column values. This might include a spike for choosing the default Hashing algorithm

Run the main.py file in the src_spark folder against datasets of increasing size from 500mb to 10GB and compare it against runs in the src folder when using pandas.

Done, with pandas used for formatting output.

Running analyze on null values in the dataframe will cause an exception to occur, and the program to force terminate.
Currently, the input file is checked for null values during parsing stage and the program terminates if null values are found.
Ideally, the PiiDetectors will be able to handle the null values.

When a glob is specified as an input, it should ingest all files that matches into the anonymizer.

As part of this story, we would build an anonymizer action that deals with dropping of the PII elements in the column values.

Given the findings from the regex matchers, generate a report that is user friendly that displays :

1. Columns with PII data
2. Low level granularity - showing which cells have PII data, along with the values

parse a delimited file
output as columns / pandas dataframe

Build a pool for identifying personal contact information

• Phone number
• Email