talegari / bigdist

Store Distance Matrices on Disk:

Welcome! I am Srikanth Komala Sheshachala

• linkedin profile
• Short profile for recruiters
• Download CV / Resume

• Interests:

  • Causal inference
  • Interpretable machine learning
  • Setting up, running and infering from AB tests and multi-armed bandits
  • Recommender Systems and Personalization
  • Optimization (Operations Research)
  • Geospatial analysis / services
  • Graph theoretic approaches to problems in data science

• Job: Senior manager (data scientist) at Walmart Gobal Tech, India.
  Previously, Olacabs, DISH corporation, Diet Code, Cognizant, Infosys, ...

• Languages: Python, R, sprinking of C++

• Contributions: Author and maintainer of these open source projects:

  • tidypandas -- A dplyr like API for pandas
  • tidypyspark -- make pyspark sing dplyr
  • tidier -- Apache Spark style window aggregations for R dataframes
  • tidyrules -- Rules as tidy dataframes from rule based models like C50, Cubist, rpart
  • solitude -- An implementation of isolation forest
  • safer -- A consistent interface to encrypt and decrypt strings, R objects, files
  • disto -- Unified Interface for Distances Matrices in R
  • Notes in Statistical Learning, Machine learning and Data Science
  • ...

Drop me a message: gmail sri.teach, linkedin srikanthks01, stackoverflow

bigdist's Issues

Package not in CRAN currently

After release of R 4.4.0, it seems this package was archived, just because a note (https://cran-archive.r-project.org/web/checks/2024/2024-04-20_check_results_bigdist.html)

[pondering] Consider `lvec` as a backend instead of `fbm`

Advantages:

1. Store only symmetric part. Hence size on disk will be small.

Disadvantages:

1. Retrieval might not be as fast as fbm, needs benchmarking.

Error: Not enough disk space to create 'E:\test\temp_ex1_738842_float.bk'.

Is there any workaround for this error (for example, saving just a triangular matrix)? I do not know how many space I need for such data (738842 rows, 6 columns), but I have 308 GB free...

cc @privefl as this error comes from bigstatsst::FBM

Thanks!

Does parallelization works for Windows?

I see that you are using {pbmcapply}. This works for Unix systems only, right?

Maybe you could use {pbapply} instead?

Is it posisble to use a bigdist FM as an input for fastcluster::hclust?

Dear Prof Shrikanth,

I am faced with a problem that requires the production of a very large distance matrix (6.9 gB) and I wish to create a hierarchical clustering using the Ward method.

So far I have been able to successfully utilize your R library (BigDist) to create a FMB of the distance matrix and store it on a local drive. However, I have been searching for a hierarchical clustering solution for such a large distance matrix, and have yet to find a solution. I started with the obvious choice of fastcluster::hclust() as follows:

d3bigdist <- bigdist(mat = d3fordist, file = file.path("Output/distYFTbig")) ## note d3fordist is a large matrix with 257370 elements, size = 2Mb

can't get bigDist FBM to work with hclust

##Connect to that big dist object on file
temp2 <- bigdist(file = file.path("Output/distYFTbig_42895_float"))

hcYFT <- fastcluster::hclust(temp2$fbm, method = "ward.D2")
print(Sys.time())

Error in fastcluster::hclust(temp2$fbm, method = "ward.D2") :
'N' must be a single integer.

Do you know of an approach that allows a function, like hclust, to access the data within the FBM and piece, by piece, build a hierarchical tree? Or must I appropriately sample the FBM, build a tree, and then append remaining data from the FBM?

I have been working on a Windows machine with 8 gB of RAM - would working on a linux platform make any difference?

post note: Why am I trying to do this? I have been handed some legacy code, written in SAS, that has successfully used the Ward method to hierarchically cluster 42895 observations of 6 variables.
I have not been able to find/construct a solution that mirrors this process in R. I have had success using partition-based clustering (from the Kmeans, CLARA packages) – however, it would be great if I could also compare these approaches with the hierarchical approach used in the original SAS code.

Warm regards

Jim Dell

bigdist_subset fails for last index

If you have a bigdist-object created as follows:

D = as_bigdist(dist(c(1:12)), file='somefile')

and you try to create a subset including the last row...

D_sub = bigdist_subset(D, 10:12, file='subset_file')

you will get an error:

max(index) not less than sz

I assume that one would just have to replace a '<' by a '<=' somewhere in the code in order to fix this