rwalk / rsk Goto Github PK

Implement the smoothing technique from Jo's paper

Jo's paper provides a maximum likelihood estimator that allows us to leave more model parameters unspecified. This is important because the current implementation requires quite a bit of information to be provided via parameters and it may be difficult to have this info up front.

Currently, the interface expects that the same number of individuals appear in each period which is not realistic.

To resolve this:

• Move aggregate_method out of the RSK class and into a util class.
• Refactor data input structure to list of matrices. The list dimension will hold time slices and the matrices will be n_individuals x n_vars. at time t.
• It may be helpful for addressing #5 to add class to support panels. This would allow us to compute the structure matrix J_t more easily which is an ingredient to our computation of the RSK filter with multiple groups.

Currently, we support only a total mean over all individuals. Jo's paper actually allows for means computed on a per group basis. It should be straightforward to support grouped means in the current code, however we'll have to think about a good API for specifying the grouping.

To actually fit this on survey data, we'll want a nice way to specify the cross-sectional identifiers, time series, and observed variables. Seems like an interface with Pandas might make the most sense for this in the Python world.

Currently, we've implemented a custom panel data type interface. It's not obvious (to me at least) how to interface properly with Pandas but maybe someone with experience can figure it out.

This project might ultimately be a better fit for the R language where there is much richer support for statistical analysis, factor labels, and survey data.

A pure R implementation of the RSK algo will probably be sufficiently fast as R will just offload the heavy matrix computations to core linear algebra libraries.