johnsnowlabs / langtest

Deliver safe & effective language models

John Snow Labs: State-of-the-art NLP in Python

The John Snow Labs library provides a simple & unified Python API for delivering enterprise-grade natural language processing solutions:

1. 15,000+ free NLP models in 250+ languages in one line of code. Production-grade, Scalable, trainable, and 100% open-source.
2. Open-source libraries for Responsible AI (NLP Test), Explainable AI (NLP Display), and No-Code AI (NLP Lab).
3. 1,000+ healthcare NLP models and 1,000+ legal & finance NLP models with a John Snow Labs license subscription.

Homepage: https://www.johnsnowlabs.com/

Docs & Demos: https://nlp.johnsnowlabs.com/

Features

Powered by John Snow Labs Enterprise-Grade Ecosystem:

• 🚀 Spark-NLP : State of the art NLP at scale!
• 🤖 NLU : 1 line of code to conquer NLP!
• 🕶 Visual NLP : Empower your NLP with a set of eyes!
• 💊 Healthcare NLP : Heal the world with NLP!
• ⚖ Legal NLP : Bring justice with NLP!
• 💲 Finance NLP : Understand Financial Markets with NLP!
• 🎨 NLP-Display Visualize and Explain NLP!
• 📊 NLP-Test : Deliver Reliable, Safe and Effective Models!
• 🔬 NLP-Lab : No-Code Tool to Annotate & Train new Models!

Installation

! pip install johnsnowlabs

from johnsnowlabs import nlp
nlp.load('emotion').predict('Wow that was easy!')

See the documentation for more details.

Usage

These are examples of getting things done with one line of code. See the General Concepts Documentation for building custom pipelines.

# Example of Named Entity Recognition
nlp.load('ner').predict("Dr. John Snow is an British physician born in 1813")

Returns :

entities	entities_class	entities_confidence
John Snow	PERSON	0.9746
British	NORP	0.9928
1813	DATE	0.5841

# Example of Question Answering 
nlp.load('answer_question').predict("What is the capital of Paris")

Returns :

text	answer
What is the capital of France	Paris

# Example of Sentiment classification
nlp.load('sentiment').predict("Well this was easy!")

Returns :

text	sentiment_class	sentiment_confidence
Well this was easy!	pos	0.999901

nlp.load('ner').viz('Bill goes to New York')

Returns:
For a full overview see the 1-liners Reference and the Workshop.

Use Licensed Products

To use John Snow Labs' paid products like Healthcare NLP, [Visual NLP], [Legal NLP], or [Finance NLP], get a license key and then call nlp.install() to use it:

! pip install johnsnowlabs
# Install paid libraries via a browser login to connect to your account
from johnsnowlabs import nlp
nlp.install()
# Start a licensed session
nlp.start()
nlp.load('en.med_ner.oncology_wip').predict("Woman is on  chemotherapy, carboplatin 300 mg/m2.")

Usage

These are examples of getting things done with one line of code. See the General Concepts Documentation for building custom pipelines.

# visualize entity resolution ICD-10-CM codes 
nlp.load('en.resolve.icd10cm.augmented')
    .viz('Patient with history of prior tobacco use, nausea, nose bleeding and chronic renal insufficiency.')

returns:

# Temporal Relationship Extraction&Visualization
nlp.load('relation.temporal_events')\
    .viz('The patient developed cancer after a mercury poisoning in 1999 ')

returns:

Helpful Resources

Take a look at the official Johnsnowlabs page page: https://nlp.johnsnowlabs.com for user documentation and examples

Resource	Description
General Concepts	General concepts in the Johnsnowlabs library
Overview of 1-liners	Most common used models and their results
Overview of 1-liners for healthcare	Most common used healthcare models and their results
Overview of all 1-liner Notebooks	100+ tutorials on how to use the 1 liners on text datasets for various problems and from various sources like Twitter, Chinese News, Crypto News Headlines, Airline Traffic communication, Product review classifier training,
Connect with us on Slack	Problems, questions or suggestions? We have a very active and helpful community of over 2000+ AI enthusiasts putting Johnsnowlabs products to good use
Discussion Forum	More indepth discussion with the community? Post a thread in our discussion Forum
Github Issues	Report a bug
Custom Installation	Custom installations, Air-Gap mode and other alternatives
The nlp.load(<Model>) function	Load any model or pipeline in one line of code
The nlp.load(<Model>).predict(data) function	Predict on Strings, List of Strings, Numpy Arrays, Pandas, Modin and Spark Dataframes
The nlp.load(<train.Model>).fit(data) function	Train a text classifier for 2-Class, N-Classes Multi-N-Classes, Named-Entitiy-Recognition or Parts of Speech Tagging
The nlp.load(<Model>).viz(data) function	Visualize the results of Word Embedding Similarity Matrix, Named Entity Recognizers, Dependency Trees & Parts of Speech, Entity Resolution,Entity Linking or Entity Status Assertion
The nlp.load(<Model>).viz_streamlit(data) function	Display an interactive GUI which lets you explore and test every model and feature in Johnsowlabs 1-liner repertoire in 1 click.

License

This library is licensed under the Apache 2.0 license. John Snow Labs' paid products are subject to this End User License Agreement.
By calling nlp.install() to add them to your environment, you agree to its terms and conditions.

langtest's Issues

Create a tutorial notebook for bias testing

Implement GH automated build for repo

Make invalid metrics_output_format impossible to pass to robustness testing

Implement token mismatch issue solution

Add unittest for noisy labels testing

Convert all docstrings from reStructuredText to Google style docstrings

Use this package to automate the conversion: https://github.com/cbillingham/docconvert

Add minimum pass rate config to each test in yaml

In yaml:

- uppercase:
    - min_pass_rate: 75%

or maybe better to have 1 config for all tests - @ArshaanNazir you can decide when implementing

NERModelHandler Roadmap

Description

Create a NERModelHandler class that establishes a common way for inference and training on NER models from different libraries. This includes:

• Wrapping NER inference pipelines for Spark NLP, transformers and spaCy
• Standardizing output formats for all pipeline predictions
• Wrapping training process for Spark NLP, transformers and spaCy models

This issue will be used to track the sub-tasks required to launch and maintain this class.

Tasks

• Ensure class supports robustness testing with Spark NLP
• Ensure class supports bias testing with Spark NLP
• Ensure class supports noisy label testing/fixing with Spark NLP
• Ensure class supports robustness testing with transformers
• Ensure class supports bias testing with transformers
• Ensure class supports noisy label testing/fixing with transformers
• Ensure class supports robustness testing with spaCy
• Ensure class supports bias testing with spaCy
• Ensure class supports noisy label testing/fixing with spaCy

Add Google style docstrings to all code

Make invalid test names impossible to pass to robustness testing

Add unittest for bias testing

Create and implement a NERModelHandler class that establishes a common method to process NER pipelines in Spark NLP

The new NERModelHandler class will include methods to:

• Wrap NER inference pipelines in Spark NLP
• Standardize output formats for all pipeline predictions

It will go under nlptest/model_handlers/

The first version of this class will support robustness testing/fixing in Spark NLP, but adaptability to other tasks and libraries should be kept in mind when developing it.

Features Backlog

Parked Ideas 🚗

• Testing social stereotypes for MLMs (paper)
• Adding support for cloud provider models to be tested
• Support an installation for airgapped environments
• Need to generate a spec of what data the model expects and can safely run with (for example, if a model has only been validated on females aged 18 and up, then the model should not be used on people outside that demographic group) - see https://ianwhitestone.work/hello-great-expectations/
• Toxicity tests (swear words, offensive answers)
• Data leakage tests (PHI)
• Adversarial attacks tests
• Freshness tests (replace _2023_name)
• Runtime tests
• Question answering
• Text generation
• Summarization
• Paraphrasing
• Translation

Add support for Spark NLP text classification models

Refactor token filtering in robustness_testing

Token filtering was created to delete extra added tokens to match token lengths for comparing predictions from NER models. There are other ways we can do this like implementing something into metrics to ignore token length differences.

See slides for more details on possible approaches.

Add support for HF transformers NER models

Tasks for release-0.0.1:

• Implement a minimum working version for 3 popular NER models

Task led by @JulesBelveze

Maintenance Backlog

Parked Ideas 🚗

NERDataHandler Roadmap

Description

Create a NERDataHandler class that establishes a common CoNLL data structure for all libraries to process labeled NER data. This includes:

• Write and read methods
• Storing docs indexes
• Easily filtering
• Converting inputs to match external library requirements (including direct dataset download from HF datasets)

This issue will be used to track the sub-tasks required to launch and maintain this class.

Tasks

• Ensure class supports robustness testing/fixing with Spark NLP
• Ensure class supports bias testing with Spark NLP
• Ensure class supports noisy label testing/fixing with Spark NLP
• Ensure class supports robustness testing/fixing with transformers
• Ensure class supports bias testing with transformers
• Ensure class supports noisy label testing/fixing with transformers
• Ensure class supports robustness testing/fixing with spaCy
• Ensure class supports bias testing with spaCy
• Ensure class supports noisy label testing/fixing with spaCy

Rename Harness config_path param to config

Add support for spaCy NER models

Add installation instructions in README

Fill out design sheet

Sheet can be found in Development channel - nlptest features.xlsx\

Please fill out the Design tab

Noisy Labels Testing Roadmap

Adaptations

• Adapt handler classes for noisy labels testing in Spark NLP
• Adapt handler classes for noisy labels testing in transformers - notebook
• Adapt handler classes for noisy labels testing in spaCy

Improvements

• #20
• #31
• Improve scoring using sentence label quality score
• Add type checking to main function
• Supporting classification, assertion, relation extraction tasks

Bug Fixes

🎉

Add support for johnsnowlabs library

Should support both classic Spark NLP pipelines & new JSL library

Reformat report method output

.report() should print this:

test factory	test type	pass count	fail count	pass rate	minimum pass rate	pass
Perturbation	uppercase	34	16	68%	75%	False

Noisy Labels Fixing Roadmap

Adaptations

• Adapt handler classes for noisy labels testing in Spark NLP
• Adapt handler classes for noisy labels testing in transformers
• Adapt handler classes for noisy labels testing in spaCy

Improvements

• #32
• Prettify UI dropdown, groupby sentences like in ALAB UI
• Add type checking to main function
• Supporting classification, assertion, relation extraction tasks

Bug Fixes

• Fix UI jupyter lab compatibility

Create a tutorial notebook for noisy labels testing/fixing

Robustness Testing Roadmap

Adaptations

• #10
• #11
• #12
• #13
• Adapt handler classes for robustness testing in transformers
• Adapt handler classes for robustness testing in spaCy

Colab notebook used to experiment: https://colab.research.google.com/drive/1VkBn3xn0yS1MzxR_w30ONRgZNNDjVByt?usp=sharing

Improvements

• #16
• #28
• Apply entity swapping to all entities in sentence (instead of 1)
• Add type checking to main function
• Supporting classification, assertion, relation extraction tasks

Bug Fixes

🚀

Convert perturbations to subclasses instead of methods of PerturbationFactory

Regroup & standardize perturbations for robustness_testing and robustness_fixing into 1 module: perturbations.py

Add support for HF datasets for Text Classification task

Fill out test type sheet

Sheet can be found in Development channel - nlptest features.xlsx

Robustness fixing should accept a simple dictionary where keys are perturbation names and values are proportions to apply to all entities for that perturbation

Currently we are passing perturbations as single params:

augment_robustness(conll_path = 'data.conll',
                   uppercase = {'PROBLEM':0.05, 'TEST':0.05, 'TREATMENT':0.05},
                   lowercase = {'PROBLEM':0.05, 'TEST':0.05, 'TREATMENT':0.05})

We should change this to a new parameter that accepts a perturbation map that looks like this:

detailed_proportions = {
   "uppercase": {'PROBLEM':0.05, 'TEST':0.05, 'TREATMENT':0.05},
   "lowercase": {'PROBLEM':0.05, 'TEST':0.05, 'TREATMENT':0.05},
   "title": {'PROBLEM':0.05, 'TEST':0.05, 'TREATMENT':0.05},
   "add_punctuation": {'PROBLEM':0.05, 'TEST':0.05, 'TREATMENT':0.05},
}

augment_robustness(conll_path = 'data.conll',
                   entity_perturbation_map = detailed_proportions)

we should also accept a more simple version of this in another parameter:

proportions= {
   "uppercase": 0.05,
   "lowercase":  0.05}

augment_robustness(conll_path = 'data.conll',
                   perturbation_map = proportions)

Add unittest for robustness fixing

Create a tutorial notebook for robustness testing/fixing

Adding different test types

Robustness Fixing Roadmap

Adaptations

• #10
• #12
• Adapt NERDataHandler class for robustness fixing in transformers
• Adapt NERDataHandler class for robustness fixing in spaCy

Improvements

• #19
• #29
• Add optimization algo for all augmentations so that we get 100% augmentation coverage for all tests as much as possible (use similar method to new noise_proportion method in robustness testing -> sample after augmenting all)
• Track if change occurred in every sentence for every augmentation
• Apply entity swapping to all entities in sentence (instead of 1)
• Add type checking to main function
• Supporting classification, assertion, relation extraction tasks
• Become compatible with many types of DOCSTART

Bug Fixes

🎉

Bias Testing Roadmap

Adaptations

• Pick and implement final gender classification method
• Replace NerDLMetrics with sklearn classification_report
• Adapt handler classes for bias testing in Spark NLP
• Adapt handler classes for bias testing in transformers
• Adapt handler classes for bias testing in spaCy

Improvements

• #30
• Add type checking to main function
• Supporting classification, assertion, relation extraction tasks

Bug Fixes

🎉

Rename k to k_fold in noisy labels testing

Add usage instructions in README

Create docs website in line with Spark NLP theme

Tasks for release-0.0.1:

• Clone spark nlp docs
• Make inventory of content needed, create GH issues for each item

Tasks for release-0.0.2:

• @agsfer deploy locally
• @luca-martial edit content and structure

Harness.generate().run().report() should be able to be chained

Fixes Backlog

Parked Ideas 🚗

• Removing transformers dependency if possible

Implement unit tests to automate testing of the nlptest library

Improve sampling method for `noise_prob` param by replacing with new `noise_proportion` param in robustness testing

noise_proportion = 0.5

# step 1: apply perturbation to all samples
1000 sentences -> apply contraction

# step 2: sample as many successfully augmented sentences as possible to reach noise_proportion
# we don't mind if some are already augmented
50 samples successfully contracted (augmented) + 100 already contracted

50 augmented + 50 (random.sample(n=50) from 1000 - 50) -> we don't mind if sampled ones are already augmented


noise_proportion = 0.5

contraction -> 5 samples to augment + 5 original samples -> f1 score: 0.60

uppercase -> 500 samples to augment + 500 original samples -> f1 score: 0.75

samples to augment == 0 -> "No samples to apply {test_name}, skipping this test."

samples to augment < 50  -> "Low number of samples ({n_samples}) to apply {test_name} to."
                            "F1-Score may not be representative of true perturbation effect."

total sentences —> 1000
noise_prob —> 0.5
for low augmentation coverage —> add, strip punc, accent_conversion, entity_swapping, add contraction
For this ones, we can apply some samples —> not all
add_punction —> sentence already have punctuation —> skip
Not all sentences can be contracted —> is not —> isn’t
1000 sentences —> noise prob 0.5 —> we can try to apply augmentation to only around 500, bcs of the noise prob
1000 —> 500 (added with noise prob) — 500 (will be searched for contraction)
Among 500 samples —> 25 contraction augmentation
While we are testing our perturbation —> perturbation set contains 500 original sentence and 25 augmented samples
Problem 1 -> we know 500 of them (original sentences) will be correct.
total noise samples will be 500 + 25 but we are comparing only 25 of them
this cause high f1 —> it seems like model don’t have problem in this perturbation test

Documentation Roadmap

Description

This is the roadmap for all things related to knowledge translation and documentation. This includes:

• tutorial notebooks
• readme instructions
• blogposts
• docs

Tasks

• #23
• #24
• #25
• #26
• #27
• #34

Privacy Attack Testing Roadmap

Description

We need to build mechanisms to test data for different categories of privacy attacks:

• Membership inference attack: An adversary predicts whether a known subject was a present in the training data used for training the synthetic data model.

• Re-identification attack: The adversary explores the probability of some features being re-identified using synthetic data and matching to the training data.

• Attribute inference attack: The adversary predicts the value of sensitive features using synthetic data.

Main article discussing mechanisms.

Tasks

🕐

Add unittest for noisy labels fixing

Create and implement a NERDataHandler class that establishes a common method to process labeled NER data in Spark NLP

The new NERDataHandler class will include:

• Write and read methods
• Storing docs indexes
• Easily filtering
• Converting inputs to match library requirements

It will go under nlptest/data_handlers/

The first version of this class will support robustness testing/fixing in Spark NLP, but adaptability to other tasks and libraries should be kept in mind when developing it.

Add unittest for robustness testing