Some variables (e.g., dates in the standard Twitter format YYYY-MM-DDTHH:MM:SS.000Z
) are likely to take a different value for each text, making the final results for the metrics sparse, uninformative, or even useless (e.g., a PMI rank for each exact datetime). It would make sense to run the computation on a given granularity instead (e.g., "year", "year-month", "year-month-day").
I strongly support to leave to the user any preprocessing stuff (we cannot handle any data variant the user is thinking about!), but the case of datetimes is quite standard (we can just support 2-3 formats and document it), and temporal aggregation seems a good feature that could be appreciated!
Extra: the same principle can be applied on spatial data with coordinates, for which a given granularity would instead be an integer denoting kilometers. Using the Haversine formula and creating a set of bounding boxes based on the granularity, the results will make much more sense.
Note: this issue is complementary to the var_subsets
feature, here we are still working on the variable values before determining which ones are of interest to the user