darenasc / aeda Goto Github PK

Is your feature request related to a problem? Please describe.
In the profiling of columns, an optional analysis of anomalies could be included.

Describe the solution you'd like
The search of outliers in numerical columns and search of biased classes in categorical columns.

Describe alternatives you've considered
None

Add filter data types such as TEXT, NTEXT, JSONB, and others when processing data_values.

Using confidence and support. A → B is P(A & B) / P(A)

Log all the operations and processing times in a log file

• columns
• tables
• uniques
• data values
• dates
• stats

Is your feature request related to a problem? Please describe.
Currently, aeda allows to profile one database at a time. Ideally, it can accept a list of connections defined in databases.ini and process them in parallel or sequence, using only one API call.

Describe the solution you'd like
A clear and concise description of what you want to happen.
The explore call can receive a list of database connections separated by space and generate a queue to process them, the last database connection should represent the target metadata database.

For small tables in the source database, process them from a dataframe and insert them into the metadata database. Tables with less than 10k rows or according to the number of rows suggested by benchmark #26 between in-memory processing or querying the source database.

Can be used for filtering.

Is your feature request related to a problem? Please describe.
It would be ideal to provide an interface to transform data at column level. For example adding representation codes so that when a code appears in the table it can be visualized with its interpretation.

Describe the solution you'd like
Include a dict where to store a dictionary alike for code transformations at the code level, so that reports can be visualized with these clear data.

Describe alternatives you've considered
Adding a user-input in the streamlit app or by passing a dictionary as a config file.

the structure of the metadata database can be taken from the config.py file

Verify length of the data value before sending it to the metadata database

A table similar to stats but for datetime related data.

• max date
• min date
• range (total number of days)
• data received per day
• median

• columns
• tables
• uniques
• data values
• dates
• stats