jeblundell / multiplyr Goto Github PK

View Code? Open in Web Editor NEW

3.0 3.0 2.0 354 KB

multiplyr R package

R 100.00%

multiplyr's People

Contributors

Stargazers

Watchers

Forkers

privefl

multiplyr's Issues

Add new functions from 0.1.1 to vignette

Minor dplyr functions

add_rownames
between
cumall
cumany
cummeandesc
first
lag
last
lead
n
nth
n_distinct
n_groups

Can't use group_by on nycflights13

library (nycflights13)

dat <- Multiplyr (flights)
dat %>% group_by (origin, dest)

Error in checkForRemoteErrors(lapply(cl, recvResult)) :
2 nodes produced errors; first error: Fatal error in attach: big.matrix could not be attached.

Parallelise calc_group_sizes (or make it redundant elsewhere)

Support for data tables in $initialize

$sort with descending as a vector

nodes not funding functions

I am attempting to summarise on a multiplyr table but the nodes can not find functions loaded in the R packages I am using.

I have also tried loading the packages using cluster_library, etc with the same result. In the Multiplyr documentation I can not seem to find anything referencing the need to load libraries on the nodes.

Error in checkForRemoteErrors(lapply(cl, recvResult)) :
7 nodes produced errors; first error: could not find function "hdi"

exists('hdi', mode='function')
[1] TRUE

auto_alloc

alloc_rows parameter to Multiplyr

SQL support

indices_cache

Like group_cache, but for first/last. Also needs each cluster node to have an ID

Support for multiple frames on same cluster

Distribute group assignment via group_cache

Give nodes IDs, assign ID to a new column in group_cache

Parallelise $compact() group size calculation

summarise doesn't correctly handle empty groups

dat <- Multiplyr (x=1:100, G=rep(1:4, each=25), cl=cl2, alloc=1, auto_compact=FALSE)
dat %>% group_by (G)
dat %>% filter (x <= 50)
dat %>% summarise (N=n())

Expected: dat["G"] == 1:4 && dat["N"] == c(25, 25, 0, 0)
Observed: dat["G"] == 1:2 && dat["N"] == c(25, 25)

Error "Insufficient free columns available" when adding new column to existing Multiplyr.

To reproduce:

install.packages('multiplyr')
library(multiplyr)
df <- Multiplyr(1:100)
df %>% mutate(b=1:100)
#Error in .self$alloc_col(.resnames, update = TRUE) : 
#  Insufficient free columns available

> sessionInfo()
R version 3.4.0 (2017-04-21)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.2 LTS

Matrix products: default
BLAS: /usr/lib/openblas-base/libblas.so.3
LAPACK: /usr/lib/libopenblasp-r0.2.18.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=pl_PL.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=pl_PL.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=pl_PL.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=pl_PL.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] multiplyr_0.1.1 magrittr_1.5   

loaded via a namespace (and not attached):
[1] bigmemory.sri_0.1.3 bigmemory_4.5.19    compiler_3.4.0     
[4] parallel_3.4.0      tools_3.4.0         Rcpp_0.12.10