rtarway / ecommerce-streamlytics-clickstream2 Goto Github PK

    //TODO : remove hardcoded fixed duration of 5 sec, also this duration needs to be lower for ecommerce env. We need to test with different values and figure out minimal time system can support with varying number of worker nodes and capacities

                            //TODO : enable timestamp as partition, currently the schema defines it as long but it is not encoded as logical type of timestamp, rather string.

        //TODO:Commented bigtable sink to save on costs during poc

              // TODO : enable time partitioning on this table 

      //TODO explore how schema conversions can be overridden for logical types such as timestamps

//TODO : Modify transaction schema to add type of transaction from event hub: 

//TODO : remove all Experimental annotation once verified that its working fine and no-longer experiement

                            //TODO: Need to make sure that sessionlized data goes to time partitioned table in bigquery

      //TODO: this  needs to be replaced by real store coming from transaction data. We can not have stores created artificically but we should design to create stores as it comes in feed.

      //TODO WTF? The store is hardcoded to a single store? This definitely needs to come from a master data table which contains all store ids and corresponding location

           //TODO : remove the hardcoding of 10 seconds and paramterize it

//TODO  convert to a service which requires a few hundred ms to respond.

//TODO: We need to research how does this table get populated with data on first run? From code, it seems the table should be preloaded in bigquery? If it should be pre-loaded, then we need to create programs to do same.

// TODO: Add an attribute by projecting future availability by date

//TODO convert to a service which requires a few hundred ms to respond.

                    //TODO : need to partition the bigquery table on timestamp or start time

  //TODO: remove hardcoded seconds     

   //TODO: remove hardcoded seconds in counting inventory

// TODO: Add an attribute by calculating availability as "total supply - total demand"    

//TODO : Filter transactions to calculate availability, rest should be written to bigquery only for analytical purposes

//todo check the data type for this field, should be timestamp

  //TODO: remove hardcoded seconds