FairLabs Data Analytics Challenge

Welcome to the Third Annual ADLM Data Analytics Competition!

This year's competition is cohosted by the ADLM Data Analytics Steering Committee, the ADLM Health Equity and Access Division, the ADLM Informatics Division, and Washington University Section of Pathology Informatics. A representative of the winning team will be invited to speak at the 2024 ADLM Annual Scientific Meeting with travel support and free meeting registration included.

The Challenge

Develop a Tool for Analyzing Fairness in Laboratory Testing

This year's competition aims to develop novel tools for monitoring fairness in laboratory testing. Competitors will be provided with a real de-identified clinical dataset documenting peripartum urine drug screening in the labor and delivery department of a single institution. The challenge is to create a dashboard that analyzes and visualizes fairness using these data. Solutions will be judged based upon functionality and adherence to best practices (see 'scoring' section). Extra points will be awarded to teams that go beyond the dataset provided by engaging local stakeholders and applying their tool to analyze fairness in their own laboratory practice.

Context

Fairness in Laboratory Testing

Laboratory testing carries benefits and unintended harms that are distributed unequally among patients. Patients belonging to vulnerable subgroups often bear a disproportionate burden of harms while enjoying fewer benefits. Inequities in Laboratory Medicine may be underappreciated due to limited real time monitoring in practice.

Peripartum Urine Drug Screening

Peripartum Urine Drug Screening (UDS) is an example of laboratory testing with benefits and unintended harms.

Benefits

Detection of specific compounds can be highly informative and guide immediate management decisions, as in the case of opiate positivity leading to treatment of neonatal abstinence syndrome. Additional benefits include the opportunity to provide counseling and support cessation efforts. In other cases the clinical benefit is unclear, as in the detection of isolated cannabis positivity.

Harms

Peripartum UDS testing can undermine the mother's trust in the clinical team and lead to reporting to Child Protective Services (CPS), which can be traumatic and harmful to the new family.

A recent retrospective study performed at Washington University School of Medicine identified history of isolated marijuana use (IMU) as the most frequently cited indication for peripartum UDS¹. They demonstrated that 99% of the positive UDS ordered for this indication were positive for cannabis only. The vast majority (89.8%) of mothers with a positive UDS ordered for IMU were reported to CPS. Of those mothers reported to CPS, 20.9% were White and 79.1% were Black despite only about 50% of mothers identifying as Black and positivity for a nonprescribed substance other than marijuana being 3 times more likely among White mothers.

To try to address these inequities in UDS harms, a quality improvement initiative was developed. This initiative included revising the labor and delivery policies to prohibit ordering UDS for an indication of IMU. To promote accountability to these policy changes, an electronic question was added to the peripartum UDS order requiring the clinician select an appropriate testing indication.

Your task is to create a dashboard that can evaluate the impact of this quality improvement initiative by analyzing and visualizing the fairness of peripartum UDS testing over time.

Scoring

Competitors will be evaluated based on the functionality of their tool, their adherence to best practices for application development, and their ability to deploy their tool to address fairness in their local laboratory practice. Scoring will be performed by expert reviewers using a standardized scoring rubric and blinded to the competitor's identity.

Functionality (45 points)

There are many ways to conceptualize and quantify fairness². Points will be awarded for each concept that is correctly implemented. Additional functionality points will be awarded for quality of visualization and user experience.

Implementation of fairness concepts (20 points): Demographic parity (4 points)
Equalized odds (4 points)
Predictive parity (4 points)
Equal outcomes (4 points)
Other (4 points)
Quality of visualizations (10 points): Impactful visuals (2 points)
Clean and appealing visualizations (2 points)
Axes and plots labeled (2 points)
Appropriate use of colors (2 points)
Appropriate sized fonts (2 points)
Statistics: Confidence intervals provided for point estimates (2 points)
Formal hypothesis testing (3 points)
User experience (10 points): Intuitive navigation (5 points)
Interactivity (5 points)

Local activation (20 points)

Competitors are highly encouraged to identify local stakeholders that would utilize a fairness monitor in practice. Points will be awarded for running the analytical tool on a local dataset. Please provide a screenshot of your local analysis (without PHI) and a short letter from the end user documenting their engagement. Please summarize key findings from local activation.

Local data analysis (10 points): Local testing data analyzed for fairness (10 points)
End-user engagement (5 points): End-user engages with fairness dashboard running on local data (5 points)
Local insights (5 points): Describe insight provided by local analysis in 500 words or less (5 points)

Best practices (12 points)

Best practices in software development help to optimize quality, maintainability, and reusability. Note that source code and commit histories are required to be eligible for points in this section.

Readability (4 points): Code is clearly and cleanly commented (1 point)
Code is simple and not bloated (1 point)
Code utilizes a clear and consistent naming convention (1 point)
Code is organized into hierarchy of modular functional units (1 point)
Reusability (2 points): Code utilizes functionalized or object oriented programming (1 point)
Code can be configured to new dataset without modification of source code (1 point)
Version control (2 points): Version control system used to track development (1 point)
Commits are modular, logical, and appropriately scoped (1 point)
Documentation and deployment (4 points): Usage notes provided (1 point)
Dependencies are defined (1 point)
Virtualized or containerized environment used (1 point)
Tool accessible through web hosting (1 point)

Significance

The tools developed in this competition could be used more broadly to help laboratories to identify unfairness in laboratory testing and monitor quality improvement projects aimed to promote equity in laboratory practice.

Timeline

January 15^th, 2024	Competition Begins
May 15^th, 2024	Competition Ends
June 15^th, 2024	Annoucement of Winning Team
July 29^th, 2024 (Anticipated)	Presentation from Winning Team

How to Participate

Sign up for a GitHub.com account

Navigate to https://github.com
Click 'Sign up'
Follow the prompts to create your personal account

Fork the competition repository

Navigate to https://github.com/WUSM-LGM-Informatics-Section/adlm-fairlabs-competition
Click 'Fork'
Select 'Create a new fork'
Set your GitHub account as the owner (defaut)
Click 'Create fork'

Build your solution

Clone the forked repository
git clone https://github.com/myGitHubUsername/adlm-fairlabs-competition.git
Note: Replace myGitHubUsername with your GitHub handle
Make a folder in the cloned repository with your team name
Build your solution within your team folder

Submit your solution via a pull request

Navigate to https://github.com/myGitHubUsername/adlm-fairlabs-competition
Note: Replace myGitHubUsername with your GitHub handle
Click 'Contribute'
Select 'Open pull request'
Click 'Create pull request'

Important: You must submit your solution by May 15th, 2024 to be eligible to win the competition

Need Help?

If you are unfamiliar with GitHub, need help getting starting, or have other questions, please email [email protected] for assistance.

Reference

Rubin A, Zhong L, Nacke L, Woolfolk C, Raghuraman N, Carter E, Kelly J. Urine Drug Screening for Isolated Marijuana Use in Labor and Delivery Units. Obstet Gynecol. 2022 Oct 1;140(4):607-609. doi: 10.1097/AOG.0000000000004930. Epub 2022 Sep 9. PMID: 36083598.
Azimi V, Zaydman MA. Optimizing Equity: Working towards Fair Machine Learning Algorithms in Laboratory Medicine. J Appl Lab Med. 2023 Jan 4;8(1):113-128. doi: 10.1093/jalm/jfac085. PMID: 36610413.

nspies13 / adlm-fairlabs-competition Goto Github PK

adlm-fairlabs-competition's Introduction