Memory management about kknn

[KlausVigo]

Hello Timo @grssnbchr,

kknn constructs internally 2 distance matrices which have dimension q * k (* 4 / 8 byte) which will be the bottle neck.
You should be able to subsets of dialects_test to kknn and just combine the results in the end, you can even compute these on different machines.
Also kernel="rectangular" may gives very similar results for a smaller choice of k, as the contribution from far away (distance wise) speakers to the estimate will be low.
If you can send me some sample data I may can find some more parts to improve.

Have a nice weekend
Regards,
Klaus

[grssnbchr]

Thanks a lot for your fast reply. I quickly thought about splitting the raster but then I thought the results wouldn't match and rejected the idea. Buut - of course - if always the same training data set is taken, the tiles fit nicely together (i.e. ==> not split the training data set, too). Doing it that way, the memory problems vanish. Also, somehow, the whole computation if faster by about 10-20%. And what's even better: I can now use the foreach package to do parallel processing and gain another 30-40% of computation time. So thanks a lot. Once I have this all together, I will write a blog post and gladly point out your package and your help!
Timo

[grssnbchr]

And here's that post - finally: https://timogrossenbacher.ch/2018/03/categorical-spatial-interpolation-with-r/
Thanks again!

[KlausVigo]

@grssnbchr Looks amazing! Glad I could help