seclerp / filefs

Simple single file-based filesystem

FileFS

Simple single file-based filesystem.

Overview

FileFS stores file files information inside single file, called storage.

Features:

• Basic file operations: create, read, update, delete, rename, exists, list all files
• Tree-based files and directiories structure
• Import and export of external files (with buffering)
• Defragmentation of storage space (manually or automatically when there is not enough space)
• Afto-extending of storage when there is no space to allocate new file
• Exclusive access support across multiple client instances
• Debugging using verbose logging
• Thread-safe

You could find latest library and CLI release in Releases section.

Documentation

• Client library getting started
• Client library API
• CLI usage guide
• Architecture overview

Limitations

• Maximum supported size of file is 2 147 483 647 bytes. This is due to Int32 and some additional challenges with streaming and in-memory copying (see #15)).
• Exclusive access works only inside 1 machine, so FileFS storage file stored at remote network will lose such feature.
• Filename are stored in fixed-sized memory blocks, so they need to have fixed maximum size.
• Defragmentation of storage with very large files may be very slow

Build

FileFS.Cli project is a .NET Core 3.1 application, and FileFS.Client project is .NET Standard 2.1 compatible library.

For building both of them you should have .NET Core SDK 3.1 installed.

To build whole solution use:

> dotnet build

From root repository folder.

Main dependencies:

• CommandLineParser (CLI)
• Serilog (DataAccess, Client, CLI)
• xUnit (Tests)
• Moq (Tests)
• StyleCop.Analysers (All projects)
• Microsoft.Extensions.DependencyInjection (Tests and Extensions.DependencyInjection)

Tests

There are 3 test projects:

• FileFS.DataAccess.Tests: contains tests for low level part of a library
• FileFS.Client.Tests: contains tests for library's API itself
• FileFS.Extensions.DependencyInjection.Tests: tests for dependency injection project

To run all tests from solution use:

> dotnet test

Fields for improvement

• Fix Int32 size limitation for large files support (see #15
• Improve memory management, especially byte arrays by using Span (see #1)

filefs's Issues

Improve known size byte array allocations using Span<T>

2.0.0 Documentation

Add auto-resize of the storage if there is not enough space

Implement inter process locking to allow usage of same FileFS file across multiple CLI instances

ImportFile performance drop fast after import hundreds of files

Implement folders

Use long instead of int in offset and size

To prevent overflow for large file sizes

Add overloads for .Write and .Read methods

In some cases, when large data is provided to such methods, these operations could fail because of RAM size limitations.

Should prove such overloads:

void Write(string fileName, Stream sourceStream);
void Read(string fileName, Stream destinationStream);

And also such overloads should use buffering instead of writing all data at once.

Implement FileFS memory usage optimization (defragmentation)

Cover code with unit tests

DataAccess:

• StorageStreamProvider
• StorageConnection
• StorageInitializer
• FileRepository
• FileDescriptorRepository
• FilesystemDescriptorAccessor
• FileDescriptorSerializer
• FilesystemDescriptorSerializer
• FileAllocator
• StorageOptimizer

Client:

• ExternalFileManager
• FileFsClient

Extensions.DependencyInjection:

• ServiceCollectionExtensions.AddFileFsClient

Implement CLI

Create documentation pages

Populate README with such sections:

• Quick description
• Features
• Documentation (links)
• Limitations
• Build (and dependencies)
• Fields for improvement

Create such documentation pages:

• Client library getting started
• Client library API
• CLI usage guide
• Architecture overview

Setup CI through GitHub Actions to review code quaility and to run unit tests

Add CreatedOn and UpdatedOn information for file descriptors

Implement validation

Implement logging and --debug option to the CLI

Make library thread-safe

Implement block-based allocator

This will help to highly reduce optimizations loads during Delete and Update actions

Cover code with clear XML documentation