lab-work / i24-transform-module

I24's Transform module for post-post processing

i24-transform-module

I24's Transform module for downstream traffic visualization and analysis (post-post processing).

The module is collection-agnostic, meaning it can transform both RAW and RECONCILED collections without explicit configuration.

High level overview

The Transform module:

1. Subscribes to a post-processed collection via mongoDB's change stream (change_stream_reader.py)
2. Transforms trajectories into a timestamp-based document (transformation.py)
3. Writes and updates the documents into a transformed mongoDB collection (batch_update.py)
4. Parallelizes step 1-3 with multiprocessing (main.py)

Initialize config

1. Clone the repository

2. Create a config.json file using config.template.json.

3. Enter the credentials for connecting to MongoDB and the collection info

   read_database_name & read_collection_name: The name of the collection and its database to transform (must already exist in db)

   write_database_name & write_collection_name: The name of the transformed collection and its database (to be generated by main.py)

   {
   	"host": "x.x.x.x",
   	"port": 27017,
   	"username": "username",
   	"password": "password",
   	"read_database_name": "db_name",
   	"read_collection_name": "collection_name"
   	"write_database_name": "database_name"
   	"write_collection_name": "collection_name"
   }
   

How to transform an existing (static) collection using static_collection_transformer.py

1. Initialize config
2. Run run_static_transformer.py

How to transform a dynamic collection using run_dynamic_transformer.py

1. Initialize config
2. Run run_dynamic_transformer.py