rumen-cholakov / grao-tables-processing

A repository containing scripts for processing tables with information for the population of settlements in Bulgaria published by GRAO

GRAO-tables-processing

A repository containing scripts for processing tables with information for the population of settements in Bulgaria published by GRAO

The initial version of the script was automatically generated by Colaboratory.

Original file is located at GRAO-tables-processing and was developed in colaboration with @trailov and @SimeonGerginov

grao-tables-processing's Issues

Add already processed data to the repo

Add already processed data to the repo

Description:

Add all currently existing processed data.

Acceptance criteria:

• Single year tables are added
• Combined table is added
• Serialized "EKATTE to triple" data is added

Add documentation to all existing functions

Add documentation to all existing functions

Description

There is next to no documentation at the moment. This may be an issue in the future

Acceptance criteria

• All classes and functions have adequate docstrings

Split code base into separate modules

Split code base into separate modules

Description

The current code base is too big for a single script. The different logical units should be split up in separate module that are imported by the main script.

Acceptance criteria

• New modules exposing the main functions of each section are created

Update README

Update README

Description

Update README to provide more information for the different parts of the project. This should be done after #12

Acceptance criteria

• Update README

Add logging to the project

Add logging to the project

Description

There is a lot of information that can be logged to allow for easier understanding of the scripts inner workings

Acceptance criteria

• Add a unified logging system that can be used in all sub-modules

• Transition all print statements to calls the new logging system

• Evaluate all exception messages and determine if any of them can be replaced with logging

• Add new logs where it will be appropriate

Update the "Update bot" part of the code

Update the "Update bot" part of the code

Description

Allow the code used for updating WikiData to be more configurable and add functionality to update the population values in the matched data csv file

Acceptance criteria

• Reference URL can be passed at call time of update_item()
• Path to credentials is added to the configuration
• Paths to matched file and credentials file are read from the configuration
• Add functionality to update matched data file with new data when available

Add configuration to the script

Add configuration to the script

Description:

Add functionality to the script that allows it to be configured. For example:

• Where to store the output
• Where to check for already processed data
• from where to read current execution task data

Acceptance criteria

• The main function reads from the command line all paths need for the configuration of the script
• The description of the tables is moved to a json file that is read by the script

Update the code in the diagram creation section

Update the code in the diagram creation section

Description

The code needs to be refactored to remove duplication of code blocks and to have a cleaner flow

Acceptance criteria

• The current functionality is preserved
• Repeated code blocks are extracted into separate functions
• Paths to the needed files are read from the configuration

Wrap all code in appropriate functions and add a main function

Wrap all code in appropriate functions and add a main function

Description:

Refactor all existing code in grao_tables_parsing.py in appropriate functions and add a main function that calls the new functions in the proper order

Acceptance criteria

• All free floating code is wrapped in appropriate functions
• A new main function is added
• There are no deterioration of the functionality

Add unit tests

Add unit tests

Description

There are no tests for the code at the moment. After #12 is complete all newly created modules should receive unit test

Acceptance criteria

• All modules have unit test

• Scripts for running the tests related to a given module are created

• All issues found by unit tests are fixed