Why drop caching Tensors? about tensoroperations.jl

[lkdvos]

I think one of the main reasons we removed this feature from the default TensorOperations installation is that we were unhappy about the interplay with multi-threaded environments. While for small tests this is hard to spot, in real-world larger applications (MPSKit.jl) we noticed that often the cache just filled up and took up most of the available memory and processes got killed because of this. This was mainly because the key for the cache had to include the task-id, and Julia seems to spawn quite a lot of these.

Nevertheless, caching can be a good strategy to reduce the number of allocations, and we have kept this in mind with this rewrite. The updated documentation should lay out the allocation procedure that we now choose, which has explicit calls to tensoralloc and tensorfree!, thus allowing a Backend system to implement a cache, but there are many different strategies there and we have not yet experimented to find the best one.

So, TL;DR, while removed as a default, users can now more easily implement there own caching strategy, either via LRU, Recycler, or any other custom backend.

[Jutho]

In addition to the response of Lukas, I'd like to add that we came to the conclusion that, in typical use cases (at least typical to us, i.e. tensor network algorithms) there are various other places where temporaries are created. As such, a solution to cache or otherwise reduce the allocation cost of a these temporaries should be more general, probably to be developed in a standalone package. The cache in the older TensorOperations versions also did not offer much control as to when to release the cached objects, other than setting the maximum size and relying on the LRU mechanism.

Also, it seems other communities are facing similar issues, so hopefully a more general global solution will be developed, either in a package (e.g. Recyclers.jl ) or in the Julia language itself.

I think an ideal solution would be that a user/developer is able to mark a particular section of code, e.g. a particular hot loop, such that all temporaries (possibly created with a special allocate call, and also manually released with a free cal) within that section of code are cached or otherwise recycled using some buffer, which is then released completely when that piece of code is finished.

We hope that the current interface of TensorOperations v4, and the way the macro is parsed, is sufficiently general to allow an easy adoption of such future developments.

[sents]

Thank you a lot for your answers. I ran into the same problem of needing to cache other things but TensorOperation calls but I just reused the TensorOperations cache for that. It makes sense to decouple caching and I'm satisfied understanding that tensoralloc and tensorfree! are meant as interfaces to implement an allocation strategy that fits the need of the problem.