The bigvis package provides tools for exploratory data analysis of large datasets (10-100 million obs). The aim is to have most operations take less than 1 second on commodity hardware, even for 100,000,000 data points.
The bigvis package is structured around the following workflow:
- summarise data with
summary1d()
- optionally smooth or standardise with
smooth_1d()
andstandardise()
- visualising the results
Bigvis also provides a number of standard statistics efficiently implemented on weighted/binned data: weighted.median
, weighted.IQR
, weighted.var
, weighted.sd
, weighted.ecdf
and weighted.quantile
.
The package can be installed using the devtools
package. Once devtools
has been installed on your system, you can run the following commands in R:
library(devtools)
install_github("bigvis", "hadley")
Note that you will also need the Boost C++
headers and libraries installed on your system in order to compile the package.
This package wouldn't be possible without:
-
the fantastic Rcpp package, which makes it amazingly easy to integrate R and C++
-
JJ Allaire, who has indefatigably answered my many C++ questions (see above)
-
the generous support of Revolution Analytics who supported the early development.
-
Yue Hu, who implemented a proof of concepts that showed that it might be possible to work with this much data in R.