Identify and Eliminate Unscalable Operations with Lazy GC about js-encryptedfs

[CMCDragonkai]

#63 (comment)

Currently ftruncate is unscalable because it reads in all the file blocks before deleting.

With our transactional iteration, we should be able to identify the block index we need to truncate to, and then do a more precise deletion of the blocks above, and the overwrite of that specific block that needs to be done. Will address this after all bugs are solved.

[CMCDragonkai]

I want to note that our write buffer is placed in the DB (disk) itself and is therefore unbounded by memory. However our write operations are buffered in-memory and not in the DB. You might wonder why not push all write operations also into the DB transaction data path? Well because at the end you still end up reading them all in-memory to perform the write batch, so there's no advantage with putting the operations onto disk. Therefore no matter what our transaction sizes are still bounded by memory.

Basically remember that our DB will always have transactions bounded by memory. So if there's an atomic operation that is larger than main memory, it will cause a problem.

Any solution has to avoid buffering too many operations into memory. Which is why I'm mentioning GC in this issue. If you get GC involved, you can "stream" mutate the DB, and thus side-effects occurs lazily (or in batches).

The best solution is that the underlying DB eventually has a disk-based write batch. But even in rocksdb, write batches are limited by main memory: facebook/rocksdb#5938.

So yea, eventually the design has to be capable of doing things lazily or in small batches while preserving consistency.