raiffeisen-dgtl / checkita-data-quality Goto Github PK

Implement following project functionality by means of GitHub actions:

• Build and test project in pool requests
• Verify that pool request title follows angular commits style
• Build project after push to main branch (release will be added later)
• Add semantic release CI. It should be started manually to prevent making releases too often
• Add CI to publish documentation (only during semantic release).

Move to open source the major updates on Checkita Data Quality. Framework have been significantly refactored. Configuration API is changed as well.

It would be good to have functionality that also allows customisation of emails' subjects.

Since this project was published, there are some features had been added internally. This issue addresses their implementation into open-source version of the framework.

One of the most commonly used check for data sources is to search for duplicate values. It could be either a values from single column, or a tuple of values from several columns, depending on what is primary key in the source.

Currently, in order to do that it is required to calculate two metrics: find the number of distinct values and number of rows. Then, it is possible to create check that will indicate the presence of duplicates in the data set by comparing these two metrics.

It would be good to have a separate metric for this case, that will also allows for error collection when duplicate values are found.