Other types of grouping about dplyr

[hadley]

Ideally these should combine elegantly, so that you can (e.g.) bootstrapping by group. Probably not possible to implement all grouping types for all data sources (e.g. random sampling in sql might be hard)

[piccolbo]

plyrmr also has a recursive grouping, whereby the operations on the grouped data can be applied on each leaf, then to the results of that and so on up to the root, useful with associative and commutative operations and when the number of groups is small, which makes flat grouping hit memory limits (in the obvious implementation where each group is in memory at once, alternatives based on iterators are also possible, albeit less convenient).

[hadley]

@piccolbo Can you provide an example? Do you mean (e.g.) that you could split the data groupwise, then bootstrap each group multiple times and fit a linear model?

[piccolbo]

No, it's the same thing as a hadoop combiner but with a name that makes it possible to explain it less in terms of its implementation. I am not sure I nailed it. The model is sum() of a big collection. You can group the summands any way you see fit, sum and then apply recursively. In hadoop it is such a performance boost that it makes certain programs possible altogether, not just faster. In principle, recursive grouping should not change the semantics of the program wrt flat grouping. Bootstrapping is a separate idea and very interesting. One of http://github.com/RevolutionAnalytics/rmr2 users has implemented it on top of map reduce in R. It may be of interest http://blog.cloudera.com/blog/2013/02/how-to-resample-from-a-large-data-set-in-parallel-with-r-on-hadoop/

[hadley]

Moved to individual issues.