jpatrickpark / bigdataproject Goto Github PK

tc.query("colsWithinRange 0,100")
should be parsed as tc.colsWithinRange(0, 100)

For example, something like

class TableCollections:
    ...
    def query(self, queryStr):
        tokens = query.split(", ")
        if tokens[0] == 'colsWithinRange':
            self.colsWithinRange(int(tokens[1]), int(tokens[2]))
tc = TableCollections()
tc.register(...)
tc.query("colsWithinRange 0,100")

We can use library such as 'sqlparse' instead of manual coding

https://stackoverflow.com/questions/31093179/how-to-use-custom-classes-with-apache-spark-pyspark

tc = TableCollections(decisionCriteria=100) and only int columns which have less than self.decisionCriteria distinct values will have hash table at the register time

Alternatively, it can be tc = TableCollection() and then tc.register(df, 'parking', decisionCriteria=100

What should directory structure be???

datacollections/
      __init__.py
      collections.py
            class DataCollections

??