ImportFile performance drop fast after import hundreds of files about filefs

[seclerp]

Thanks for your interest in the project!

First of all, there are some specifics of how OS filesystem works. Bigger files are divided into chunks and that's why the system needs some time to figure out what chunk (correct naming here is "block") basically should be used for insertion or retrieval.

That could also be optimized if using caches and different lighter files for storing data in FileFS, but it was not the scope of this project. But if you want to help implement such a feature - please go ahead! This project is open for any contributions.

From the .NET point of view problem is in .Seek method call for FileStream on large files.

You could also find out more information here.

TL;DR:

FileStream.Seek is actually quite fast - though it will be slower on larger files than smaller ones. It's not a perfectly linear jump, since the file may/will be fragmented on disk (especially if it's a large file), so multiple blocks need to be read, and more need to be read in larger files.