ttu / json-flatfile-datastore

Simple JSON flat file data store with support for typed and dynamic data.

JSON Flat File Data Store

Build server	Platform	Build status
GH Actions	Linux
GH Actions	Windows

Simple data store that saves the data in JSON format to a single file.

• Small API with basic functionality that is needed for handling data
• Works with dynamic and typed data
• Synchronous and asynchronous methods
• Data is stored in a JSON file
  • Easy to initialize
  • Easy to edit
  • Perfect for small apps and prototyping
  • Optional encryption for file content
• .NET implementation & version support: .NET Standard 2.0
  • e.g. .NET 6, .NET Core 2.0, .NET Framework 4.6.1

Docs website

https://ttu.github.io/json-flatfile-datastore/

---

Installation

You can install the latest version via NuGet.

# .NET Core CLI
$ dotnet add package JsonFlatFileDataStore

# Package Manager Console
PM> Install-Package JsonFlatFileDataStore

Example

Typed data

public class Employee
{
    public int Id { get; set; }
    public string Name { get; set; }
    public int Age { get; set; }
}

// Open database (create new if file doesn't exist)
var store = new DataStore("data.json");

// Get employee collection
var collection = store.GetCollection<Employee>();

// Create new employee instance
var employee = new Employee { Id = 1, Name = "John", Age = 46 };

// Insert new employee
// Id is updated automatically to correct next value
await collection.InsertOneAsync(employee);

// Update employee
employee.Name = "John Doe";

await collection.UpdateOneAsync(employee.Id, employee);

// Use LINQ to query items
var results = collection.AsQueryable().Where(e => e.Age > 30);

// Save instance as a single item
await store.InsertItemAsync("selected_employee", employee);

// Single items can be of any type
await store.InsertItemAsync("counter", 1);
var counter = await store.GetItem<int>("counter");

Dynamically typed data

Dynamic data can be Anonymous type, ExpandoObject, JSON objects (JToken, JObject, JArray) or Dictionary<string, object>. Internally dynamic data is serialized to ExpandoObject.

// Open database (create new if file doesn't exist)
var store = new DataStore(pathToJson);

// Get employee collection
var collection = store.GetCollection("employee");

// Create new employee
var employee = new { id = 1, name = "John", age = 46 };

// Create new employee from JSON
var employeeJson = JToken.Parse("{ 'id': 2, 'name': 'Raymond', 'age': 32 }");

// Create new employee from dictionary
var employeeDict = new Dictionary<string, object>
{
    ["id"] = 3,
    ["name"] = "Andy",
    ["age"] = 32
};

// Insert new employee
// Id is updated automatically if object is updatable
await collection.InsertOneAsync(employee);
await collection.InsertOneAsync(employeeJson);
await collection.InsertOneAsync(employeeDict);

// Update data from anonymous type
var updateData = new { name = "John Doe" };

// Update data from JSON
var updateJson = JToken.Parse("{ 'name': 'Raymond Doe' }");

// Update data from dictionary
var updateDict = new Dictionary<string, object> { ["name"] = "Andy Doe" };

await collection.UpdateOneAsync(e => e.id == 1, updateData);
await collection.UpdateOneAsync(e => e.id == 2, updateJson);
await collection.UpdateOneAsync(e => e.id == 3, updateDict);

// Use LINQ to query items
var results = collection.AsQueryable().Where(x => x.age > 30);

Example project

Fake JSON Server is an ASP.NET Core Web App which uses JSON Flat File Data Store with dynamic data.

Functionality

Collections

Example user collection in JSON

{
  "user": [
    { "id": 1, "name": "Phil", "age": 40, "city": "NY" },
    { "id": 2, "name": "Larry", "age": 37, "city": "London" }
  ]
}

Query

Collection can be queried with LINQ by getting queryable from the collection with AsQueryable method.

NOTE: AsQueryable returns IEnumerable, instead of IQueryable, because IQueryable doesn't support Dynamic types in LINQ queries. With this data store it won't matter as all data is already loaded into memory.

AsQueryable LINQ query with dynamic data:

var store = new DataStore(pathToJson);

var collection = store.GetCollection("user");

// Find item with name
var userDynamic = collection
                    .AsQueryable()
                    .FirstOrDefault(p => p.name == "Phil");

AsQueryable LINQ query with typed data:

var store = new DataStore(pathToJson);

var collection = store.GetCollection<User>();

// Find item with name
var userTyped = collection
                    .AsQueryable()
                    .FirstOrDefault(p => p.Name == "Phil");

Full-text search

Full-text search can be performed with Find method. Full-text search does deep search on all child objects. By default the search is not case sensitive.

var store = new DataStore(pathToJson);

var collection = store.GetCollection("user");

// Find all users that contain text Alabama in any of property values
var matches = collection.Find("Alabama");

// Perform case sensitive search
var caseSensitiveMatches = collection.Find("Alabama", true);

Insert

InsertOne and InsertOneAsync will insert a new item to the collection. Method returns true if insert was successful.

// Asynchronous method and dynamic data
// Before update : { }
// After update  : { "id": 3, "name": "Raymond", "age": 32, "city" = "NY" }
await collection.InsertOneAsync(new { id = 3, name = "Raymond", age = 32, city = "NY" });

// Dynamic item can also be JSON object
var user = JToken.Parse("{ 'id': 3, 'name': 'Raymond', 'age': 32, 'city': 'NY' }");
await collection.InsertOneAsync(user);

// Synchronous method and typed data
// Before update : { }
// After update  : { "id": 3, "name": "Raymond", "age": 32, "city" = "NY" }
collection.InsertOne(new User { Id = 3, Name = "Raymond", Age = 32, City = "NY" });

InsertMany and InsertManyAsync will insert a list of items to the collection.

var newItems = new[]
{
    new User { Id = 3, Name = "Raymond", Age = 32, City = "NY" },
    new User { Id = 4, Name = "Ted", Age = 43, City = "NY" }
};

collection.InsertMany(newItems);

Insert-methods will update the inserted object's Id-field, if it has a field with that name and the field is writable. If the Id-field is missing from the dynamic object, a field is added with the correct value. If an Anonymous type is used for insert, id will be added to the persisted object if the id-field is missing. If the id is present, then that value will be used.

var newItems = new[]
{
    new { id = 14, name = "Raymond", age = 32, city = "NY" },
    new { id = 68, name = "Ted", age = 43, city = "NY" },
    new { name = "Bud", age = 43, city = "NY" }
};

// Last user will have id 69
collection.InsertMany(newItems);
// Item in newItems collection won't have id property as anonymous types are read only

If the id-field 's type is a number, value is incremented by one. If the type is a string, incremented value number is added to the end of the initial text.

// Latest id in the collection is hello5
var user = JToken.Parse("{ 'id': 'wrongValue', 'name': 'Raymond', 'age': 32, 'city': 'NY' }");
await collection.InsertOneAsync(user);
// After addition: user["id"] == "hello6"

// User data doesn't have an id field
var userNoId = JToken.Parse("{ 'name': 'Raymond', 'age': 32, 'city': 'NY' }");
await collection.InsertOneAsync(userNoId);
// After addition: userNoId["id"] == "hello7"

If collection is empty and the type of the id-field is number, then first id will be 0. If type is string then first id will be "0".

Replace

ReplaceOne and ReplaceOneAsync will replace the first item that matches the filter or provided id-value matches the defined id-field. Method will return true if item(s) found with the filter.

// Sync and dynamic
// Before update : { "id": 3, "name": "Raymond", "age": 32, "city": "NY" }
// After update  : { "id": 3, "name": "Barry", "age": 42 }
collection.ReplaceOne(3, new { id = 3, name = "Barry", age = 33 });
// or with predicate
collection.ReplaceOne(e => e.id == 3, new { id = 3, name = "Barry", age = 33 });

// Async and typed
// Before update : { "id": 3, "name": "Raymond", "age": 32, "city": "NY" }
// After update  : { "id": 3, "name": "Barry", "age": 42 }
await collection.ReplaceOneAsync(3, new User { Id = 3, Name = "Barry", Age = 33 });

ReplaceMany and ReplaceManyAsync will replace all items that match the filter.

collection.ReplaceMany(e => e.City == "NY", new { City = "New York" });

ReplaceOne and ReplaceOneAsync have an upsert option. If the item to replace doesn't exists in the data store, new item will be inserted. Upsert won't update id, so new item will be inserted with the id that it has.

// New item will be inserted with id 11
collection.ReplaceOne(11, new { id = 11, name = "Theodor" }, true);

Update

UpdateOne and UpdateOneAsync will update the first item that matches the filter or provided id-value matches the defined id-field. Properties to update are defined with dynamic object. Dynamic object can be an Anonymous type or an ExpandoObject. Method will return true if item(s) found with the filter.

// Dynamic
// Before update : { "id": 1, "name": "Barry", "age": 33 }
// After update  : { "id": 1, "name": "Barry", "age": 42 }
dynamic source = new ExpandoObject();
source.age = 42;
await collection.UpdateOneAsync(1, source as object);
// or with predicate
await collection.UpdateOneAsync(e => e.id == 1, source as object);

// Typed
// Before update : { "id": 1, "name": "Phil", "age": 40, "city": "NY" }
// After update  : { "id": 1, "name": "Phil", "age": 42, "city": "NY" }
await collection.UpdateOneAsync(e => e.Name == "Phil", new { age = 42 });

UpdateMany and UpdateManyAsync will update all items that match the filter.

await collection.UpdateManyAsync(e => e.Age == 30, new { age = 31 });

Update can also update items from the collection and add new items to the collection. null items in the passed update data are skipped, so with null items data in the correct index can be updated.

var family = new Family
{
    Id = 12,
    FamilyName = "Andersen",
    Parents = new List<Parent>
    {
        new Parent {  FirstName = "Jim", Age = 52 }
    },
    Address = new Address { City = "Helsinki" }
};

await collection.InsertOneAsync(family);

// Adds a second parent to the list
await collection.UpdateOneAsync(e => e.Id == 12, new { Parents = new[] { null, new { FirstName = "Sally", age = 41 } } });

// Updates the first parent's age to 42
await collection.UpdateOneAsync(e => e.Id == 12, new { Parents = new[] { new { age = 42 } } });

Easy way to create a patch ExpandoObject on runtime is to create a Dictionary and then to serialize it to a JSON and deserialize to an ExpandoObject.

var user = new User
{
    Id = 12,
    Name = "Timmy",
    Age = 30,
    Work = new WorkPlace { Name = "EMACS" }
};

// JSON: { "Age": 41, "Name": "James", "Work": { "Name": "ACME" } }
// Anonymous type: new { Age = 41, Name = "James", Work = new { Name = "ACME" } };
var patchData = new Dictionary<string, object>();
patchData.Add("Age", 41);
patchData.Add("Name", "James");
patchData.Add("Work", new Dictionary<string, object> { { "Name", "ACME" } });

var jobject = JObject.FromObject(patchData);
dynamic patchExpando = JsonConvert.DeserializeObject<ExpandoObject>(jobject.ToString());

await collection.UpdateOneAsync(e => e.Id == 12, patchExpando);

Limitations

Dictionaries won't work when serializing JSON or data to ExpandoObjects. This is becauses dictionaries and objects are similar when serialized to JSON, so serialization creates an ExpandoObject from Dictionary. Update's are mainly meant to be used with HTTP PATCH, so normally Replace is easier and better way to update data.

If the update ExpandoObject is created manually, then the Dictionaries content can be updated. Unlike List, Dictionary's whole content is replaced with the update data's content.

var player = new Player
{
    Id = 423,
    Scores = new Dictionary<string, int>
    {
        { "Blue Max", 1256 },
        { "Pacman", 3221 }
    },
};

var patchData = new ExpandoObject();
var items = patchData as IDictionary<string, object>;
items.Add("Scores", new Dictionary<string, string> { { "Blue Max", 1345 }, { "Outrun", 1234 }, { "Pacman", 3221 }, });

await collection.UpdateOneAsync(e => e.Id == 423, patchData);

Delete

DeleteOne and DeleteOneAsync will remove the first object that matches the filter or provided id-value matches the defined id-field. Method returns true if item(s) found with the filter or with the id.

// Dynamic
await collection.DeleteOneAsync(3);
await collection.DeleteOneAsync(e => e.id == 3);

// Typed
await collection.DeleteOneAsync(3);
await collection.DeleteOneAsync(e => e.Id == 3);

DeleteMany and DeleteManyAsync will delete all items that match the filter. Method returns true if item(s) found with the filter.

// Dynamic
await collection.DeleteManyAsync(e => e.city == "NY");

// Typed
await collection.DeleteManyAsync(e => e.City == "NY");

Get next Id-field value

If incrementing Id-field values is used, GetNextIdValue returns next Id-field value. If Id-property is integer, last item's value is incremented by one. If field is not an integer, it is converted to a string and number is parsed from the end of the string and incremented by one.

var store = new DataStore(newFilePath, keyProperty: "myId");

// myId is an integer
collection.InsertOne(new { myId = 2 });
// nextId = 3
var nextId = collection.GetNextIdValue();

// myId is a string
collection.InsertOne(new { myId = "hello" });
// nextId = "hello0"
var nextId = collection.GetNextIdValue();

collection.InsertOne(new { myId = "hello3" });
// nextId = "hello4"
var nextId = collection.GetNextIdValue();

Single item

{
  "selected_user": { "id": 1, "name": "Phil", "age": 40, "city": "NY" },
  "temperature": 23.45,
  "temperatues": [ 12.4, 12.42, 12.38 ],
  "note": "this is a test"
}

Data store supports single items. Items can be value and reference types. Single item supports dynamic and typed data.

Arrays are concidered as single items if they contain value types. If Array is empty it is listed as a collection.

Single item's support same methods as Collections (Get, Insert, Replace, Update, Delete).

Get

var store = new DataStore(pathToJson);
// Typed data
var counter = store.GetItem<int>("counter");
// Dynamic data
var user = store.GetItem("myUser");

Typed data will throw KeyNotFoundException if key is not found. Dynamic data and nullable types will return null.

// throw KeyNotFoundException
var counter = store.GetItem<int>("counter_NotFound");
var user = store.GetItem<User>("user_NotFound");
// return null
var counter = store.GetItem<int?>("counter_NotFound");
var counter = store.GetItem("counter_NotFound");

Insert

InsertItem and InsertItemAsync will insert a new item to the JSON. Method returns true if insert was successful.

// Value type
var result = await store.InsertItemAsync("counter", 2);
// Reference type
var user = new User { Id = 12, Name = "Teddy" }
var result = await store.InsertItemAsync<User>("myUser", user);

Replace

ReplaceItem and ReplaceItemAsync will replace the item with the key. Method will return true if item is found with the key.

// Value type
var result = await store.ReplaceItemAsync("counter", 4);
// Reference type
var result = await store.ReplaceItemAsync("myUser", new User { Id = 2, Name = "James" });

ReplaceSingleItem and ReplaceSingleItem have an upsert option. If the item to replace doesn't exists in the data store, new item will be inserted.

// Value type
var result = await store.ReplaceItemAsync("counter", 4, true);
// Reference type
var result = await store.ReplaceItemAsync("myUser", new User { Id = 2, Name = "James" }, true);

Update

UpdateItem and UpdateItemAsync will update the first item that matches the filter with passed properties from dynamic object. Dynamic object can be an Anonymous type or an ExpandoObject. Method will return true if item is found with the key.

// Value type
var result = await store.UpdateItemAsync("counter", 2);
// Reference type
var result = await store.UpdateItemAsync("myUser", new { name = "Harold" });

Delete

DeleteItem and DeleteItemAsync will remove the item that matches the key. Method returns true if item is found and deleted with the key.

// Sync
var result = store.DeleteItem("counter");
// Async
var result = await store.DeleteItemAsync("counter");

Encrypt JSON-file content

It is possible to encrypt the written JSON-data. When encryptionKey-parameter is passed to constructor, data will be encypted with Aes256.

var secretKey = "Key used for encryption";
var store = new DataStore(newFilePath, encryptionKey: secretKey);

Data Store and collection lifecycle

When the data store is created, it reads the JSON file to the memory. Data store starts a new background thread that handles the file access.

When the collection is created it has a lazy reference to the data and it will deserialize the JSON to objects when it is accessed for the first time.

All write operations in collections are executed immediately internally in the collection and then the same operation is queued on DataStore's BlockingCollection. Operations from the BlockingCollection are executed on background thread to DataStore's internal collection and saved to file.

// Data is loaded from the file
var store = new DataStore(newFilePath);

// Lazy reference to the data is created
var collection1st = store.GetCollection("hello");
var collection2nd = store.GetCollection("hello");

// Data is loaded from the store to the collection and new item is inserted
collection1st.InsertOne(new { id = "hello" });

// Data is loaded from the store to the collection and new item is inserted
// This collection will also have item with id: hello as data is serialized when it is used for the first time
collection2nd.InsertOne(new { id = "hello2" });

// collection1st won't have item with id hello2

If multiple DataStores are initialized and used simultaneously, each DataStore will have its own internal state. They might become out of sync with the state in the JSON file, as data is only loaded from the file when DataStore is initialized and after each commit.

It is also possible to reload JSON data manually, by using DataStore's Reload method or set reloadBeforeGetCollection constructor parameter to true.

// Data is loaded from the file
var store = new DataStore(newFilePath);
var store2 = new DataStore(newFilePath, reloadBeforeGetCollection: true);

var collection1_1 = store.GetCollection("hello");
collection1_1.InsertOne(new { id = "hello" });

// Because of reload collection2_1 will also have item with id: hello
var collection2_1 = store2.GetCollection("hello");

collection2_1.InsertOne(new { id = "hello2" });

store.Reload()

// Because of reload collection1_2 will also have item with id: hello2
var collection1_2 = store.GetCollection("hello");

// collection1_1 will not have item with id: hello2 even after reload, because it was initialized before reload

If JSON Flat File Data Store is used with e.g. ASP.NET, add the DataStore to the DI container as a singleton. This way DataStore's internal state is correct and application does not have to rely on the state on the file as read operation is pretty slow. Reload can be triggered if needed.

Disposing Data Store

Data store should be disposed after it is not needed anymore. Dispose will wait that all writes to the file are completed and after that it will stop the background thread. Then Garabge Collector can collect the data store that is not used anymore.

// Call dispose method
var store = new DataStore();
// ...
store.Dispose();

// Call dispose automatically with using
using(var store = new DataStore())
{
    // ...
}

Collection naming

Collection name must be always defined when dynamic collections are used. Collection name is converted to selected case.

If collection name is not defined with a typed collection, class-name is converted to selected case. E.g. with lower camel case User is user, UserFamily is userFamily etc.

var store = new DataStore(newFilePath);
// JSON { "movie": [] };
var collection = store.GetCollection("movie");
// JSON { "movie": [] };
var collection = store.GetCollection("Movie");
// JSON { "movie": [] };
var collection = store.GetCollection<Movie>();
// JSON { "movies": [] };
var collection = store.GetCollection<Movie>("movies");

Writing data to a file

By default JSON is written in lower camel case. This can be changed with useLowerCamelCase parameter in DataStore's constructor.

// This will write JSON in lower camel case
// e.g. { "myMovies" : [ { "longName": "xxxxx" } ] }
var store = new DataStore(newFilePath);

// This will write JSON in upper camel case
// e.g. { "MyMovies" : [ { "LongName": "xxxxx" } ] }
var store = new DataStore(newFilePath, false);

Additionaly the output of the file can be minfied. The default is an intended output.

var store = new DataStore(newFilePath, minifyJson: true);

Dynamic types and error CS1977

When Dynamic type is used with lambdas, compiler will give you error CS1977:

CS1977: Cannot use a lambda expression as an argument to a dynamically dispatched operation without first casting it to a delegate or expression tree type

A lambda needs to know the data type of the parameter at compile time. Cast dynamic to an object and compiler will happily accept it, as it believes you know what you are doing and leaves validation to Dynamic Language Runtime.

dynamic dynamicUser = new { id = 11, name = "Theodor" };

// This will give CS1977 error
collection2.ReplaceOne(e => e.id == 11, dynamicUser);

// Compiler will accept this
collection2.ReplaceOne(e => e.id == 11, dynamicUser as object);

// Compiler will also accept this
collection2.ReplaceOne((Predicate<dynamic>)(e => e.id == 11), dynamicUser);

Unit tests & benchmarks

JsonFlatFileDataStore.Test and JsonFlatFileDataStore.Benchmark are .NET 6 projects.

Unit Tests are executed automatically with CI builds.

Benchmarks are not part of CI builds. Benchmarks can be used as a reference when making changes to the existing functionality by comparing the execution times before and after the changes.

Run benchmarks from command line:

$ dotnet run --configuration Release --project JsonFlatFileDataStore.Benchmark\JsonFlatFileDataStore.Benchmark.csproj

API

API is heavily influenced by MongoDB's C# API, so switching to the MongoDB or DocumentDB might be easy.

• MongoDB-C#-linq
• MongoDB-C#-crud

Changelog

Changelog

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

License

Licensed under the MIT License.

json-flatfile-datastore's Issues

Updating class properties are PascalCase but the updated data has camelCase properties

Updating class properties are PascalCase but the updated data has camelCase properties, then nothing is updated.

class MyClass {
   public string Id [get; set;}
   public string Value [get; set;}
}
...

collection.UpdateOneAsync(x => x.id == "1", JToken.Parse("{ value: \"new value\"} "));

... nothing is updated. I think it should update the "value" property.

When I rename the MyClass.Value to MyClass.value then it works.

Update function not working

Given a "data.json" file with:

"worlds": [
    {
        "id": 0,
        "position": [
            3.6640577,
            5.5219193,
            3.4788187
          ],
         "cameraRotationX": -0.66699946,
         "cameraRotationY": -0.003000007
    }
]

Running the following update code:

var collection = gameStore.GetCollection("worlds");
await collection.UpdateOneAsync(e => e.id == 0, new { position = new float[] { 10, 10, 10} } );

The database does not save the update and will remain the same as the above mentioned example. This code was pulled directly from the docs. Is there something I missed? I tried out any other update function I could find in the docs as well with the same results.

Current solution is to delete and re-insert the entry.

GetItem_DynamicAndTyped_SimpleType failed on german test system

On a german system (windows client OS) the
test GetItem_DynamicAndTyped_SimpleType failed,

• the internal _jsonData is correct build
• but the _jsonData["myValue"] gives 2,1 not 2.1
• so the JsonConvert.DeserializeObject runs into a new exception type
• after shown changes the test fails not anymore

JsonFlatFileDataStore.DataStore:
``private dynamic SingleDynamicItemReadConverter(JToken e)
{
try
{
// As we don't want to return JObject when using dynamic, JObject will be converted to ExpandoObject
return JsonConvert.DeserializeObject(e.ToString(), _converter) as dynamic;
}
catch (Exception ex) when (ex is InvalidCastException || ex is JsonReaderException)
{
return e.ToObject();
}
}

Support filesystem abstraction

I had a need to test this with a mocked filesystem. Used System.io.Abstraction. for this purpose.

Here are the changes in case it is helpful.

ykarpeev@83d4bc8

Using DI with a console app should I inject as singelton or transient?

I am wondering how I should treat the lifetime of my DataStore. If it matters, im using a console app that runs and quits. Its very linear.

• If I do transient, I may have two different instances of it at once. Would that cause issues?
• If I do singleton, I suppose its possible to ask the same instance of DataStore to write to my file at once too.

Just trying to think through this, thanks!

.AddSingleton<IDataStore>(serviceProvider =>
{
    var connectionString = serviceProvider.GetRequiredService<ConnectionStringsOptions>();
    return new DataStore(connectionString.DefaultDatabase, false);
})

VS

.AddTransient<IDataStore>(serviceProvider =>
{
    var connectionString = serviceProvider.GetRequiredService<ConnectionStringsOptions>();
    return new DataStore(connectionString.DefaultDatabase, false);
})

DataStore to implement IDisposable

Make sure all writes are done when disposing.

Feature branch: https://github.com/ttu/json-flatfile-datastore/tree/use-dispose

Add Support for using a directory as the store, with each document within as a child

InsertItem gets stuck in .NET Core 3 WPF project

Hello!

Steps to reproduce:
Create a new WPF project with .NET Core 3 platform. Add JasonFlatFileDataStore (2.20) nuget package to it.

Add your example Employee class and add DataStore code to MainWindow:

public MainWindow()
{
    InitializeComponent();

    var store = new DataStore("data.json");
   
    var collection = store.GetCollection<Employee>();
   
    var employee = new Employee { Id = 1, Name = "John", Age = 46 };
   
    collection.InsertOne(employee); // Works

    store.InsertItem("counter", 1); // Stuck forever

    collection.InsertOne(new Employee { Name = "John", Age = 46 }); // Never reached
}

Run the WPF-project in debug mode.

Additional info:
data.json gets both the first employee and the counter inserted correctly, but the InsertItem code line gets stuck. ReplaceItem has the same problem. I tried above code (without InitializeComponent) in a .NET Core 3 console project and there were no problems.

No Timeout/Error on file inaccessibility

It looks like the package has no problem creating a new file in a directory on creation of the datastore object.

private static DataStore store = new DataStore($"{logLocation}\\{fileName}.json");

But if the modify permission is not given to that file on creation, the IDocumentCollection<Object>.InsertOne(Object) does not seem to timeout or error.

Add exception-free way of retrieving nullable items

As of 2.0.0 if there's no item with a provided key, _dataStore.GetItem<Guid?>(myKey) will throw an exception. Please provide a way to retrieve a nullable value without getting an exception.

Replace Func delegates from DocumentCollection methods with local functions

Local functions are more readable. Example change for UpdateOneAsync.

Current:

public async Task<bool> UpdateOneAsync(Predicate<T> filter, dynamic item)
{
	var updateAction = new Func<List<T>, bool>(data =>
	{
		var matches = data.Where(e => filter(e));

		if (!matches.Any())
			return false;

		var toUpdate = matches.First();
		ObjectExtensions.CopyProperties(item, toUpdate);

		return true;
	});

	if (!ExecuteLocked(updateAction, _data.Value))
		return false;

	return await _commit(_path, updateAction, true).ConfigureAwait(false);
}

Changed:

public async Task<bool> UpdateOneAsync(Predicate<T> filter, dynamic item)
{
	bool UpdateAction(List<T> data)
	{
		var matches = data.Where(e => filter(e));

		if (!matches.Any())
			return false;

		var toUpdate = matches.First();
		ObjectExtensions.CopyProperties(item, toUpdate);

		return true;
	}

	if (!ExecuteLocked(UpdateAction, _data.Value))
		return false;

	return await _commit(_path, UpdateAction, true).ConfigureAwait(false);
}

Weird NotFoundException on Debian 12

I tested this on my main Windows system and it works perfectly. However, when I move my project to my Debian 12 VPS, I get this exception.

An exception occurred while exeting the command callback. System.IO.FileNotFoundException: Could not load file or assembly 'JsonFlatFileDataStore, Version=2.4.2.0, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified

Proofread README.md

Fix all grammar, styling, spelling etc. errors from README.md

Simple encryption support

Hello,
I think it would be beneficial to add encryption support to the library.

Please see #60 for a possible implementation.

Update1: I will probably add more tests to it if there is enough interest for this functionality as well as doc updates.
Update2: #60 is now merged. :)

Best Regards,
/Gabor

GetNextIdValue should return next id based on the largest id in the collection, not based on the last item's id

GetNextIdValue checks only the last item from the collection and gives a new id based on that item's id. Method should find the item with the largest id value from the collection and give a new id based on that item's id.

Function is in JsonFlatFileDataStore/DocumentCollection.cs
Tests are in JsonFlatFileDataStore.Test/CollectionTests.cs

Providing My Own Primary Keys

This might be more of a question...

I was planning on using GUIDs and generate my own primary keys. I have keyId on the DataStore constructor set to 'primaryKey':

_dataStore = new DataStore(FileName, true, nameof(PrimaryKeyBase.PrimaryKey));

It seems to take the first value an then just start incrementing on that value. It seems like I should just let the library set it own primary keys. It doesn't seem to like me setting the primary key itself. Some I'm looking more for confirmation of functionality. Below is an example of what I'm seeing

"questionSet": [
{
"name": "Question Set 1",
"notes": null,
"deleted": null,
"primaryKey": "54ccb8f2-5f2c-4cfa-b525-f0871eca787c"
},
{
"name": "Question Set 2",
"notes": null,
"deleted": null,
"primaryKey": "54ccb8f2-5f2c-4cfa-b525-f0871eca787c0"
},
{
"name": "Question Set 3",
"notes": null,
"deleted": null,
"primaryKey": "54ccb8f2-5f2c-4cfa-b525-f0871eca787c1"
}
],

How to get ToList() ?

Is there a way to do a GetCollection().AsQueryable().ToList(), other than manually creating a List and individually adding queried items to the list?
NVM...missing System.Linq. Sorry for the trouble.

Add netstandard 2.0 to TargetFrameworks

https://twitter.com/davidfowl/status/1008453902058917888?s=19
https://twitter.com/davidfowl/status/1008456597205499904?s=19

Id is not added to the persisted object if the new object is added with anonymous type that doesn't have id-field

Add new object with anonymous type

collection.InsertOne(new { name = "Teddy" });

Id-field is missing from the JSON

users: [
 { "name": "teddy" }
]

Add CancellationToken support for async operations

Add CancellationToken support for async operations.

Minified or compact version of JSON possible?

Currently when JSON objects are stored in the file it looks like this:

{
  "priceData": [
    {
      "dateTime": "2022-10-06T12:05:00.834994+02:00",
      "exchangeName": "NYSE",
      "coinPair": "BLA/USD",
      "price": 1.1,
      "bid": 1.0,
      "ask": 1.2
    },
    {
      "dateTime": "2022-10-06T12:05:00.834994+02:00",
      "exchangeName": "NYSE",
      "coinPair": "BLA/USD",
      "price": 1.1,
      "bid": 1.0,
      "ask": 1.2
    }
  ]
}

This way the file gets very bloated. Is there a way to store it minified or compact like this:

{"priceData":[{"dateTime":"2022-10-06T12:02:34.4853061+02:00","exchangeName":"NYSE","coinPair":"BLA/USD","price":1.1,"bid":1.0,"ask":1.2},{"dateTime":"2022-10-06T12:05:00.834994+02:00","exchangeName":"NYSE","coinPair":"BLA/USD","price":1.1,"bid":1.0,"ask":1.2}]}

Synchronizing Async Methods

Synchronizing an async method (via a 'Result' or 'Wait()') seems to cause the entire app to lockup. This is in a .NET 6 WPF application.

I know this seems odd, but I'm slowly integrating async/await into my application. For the moment I have to synchronize the async calls in my ICommands from the UI.

            using (var ds = new DataStore(_fileName, true, nameof(QuestionSet.PrimaryKey)))
            {
                var coll = ds.GetCollection<QuestionSet>();
                var success = coll.InsertOneAsync(value).Result;   **Application locks up here**
                if (success == false)
                {
                    throw new OperationFailedException("Add failed");
                }
            }

Query Async

First of all, thanks for this library.

Quick question: is there support for async querying?

trying to update List doesnt update or throws " No parameterless constructor defined for this object." when adding more items

my patch example:
[{"op": "Replace", "path":"/", "value": {
"id": "8b174be2-c623-42f1-8aae-73ce83413214",
"type": "yyyyy",
"fragmentIds": [
"91d1643c-015b-4977-b961-ef4177875711", "1acd4eb0-41ec-4865-ba65-26ef23548688",
]
}
}
]

I am trying to update list in my root of object using this:
Predicate p = x => x.id == id;
await this.collection.UpdateOneAsync(p, operation.value);
when I have same count of fragmentIds it doesnt update values
when I add more fragments I am getting:
System.MissingMethodException: No parameterless constructor defined for this object.
at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean wrapExceptions, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor)
at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean wrapExceptions, Boolean skipCheckThis, Boolean fillCache)..........

can you please let me know if this can be fixed? I know you wrote some limitation on List

Add benchmarks for IDocumentCollection methods

Add benchmarks for all IDocumentCollection methods.

Now has benchmarks only for InsertOneAsync and InsertOne.

Update is not updating the json File

void Main()
{
var phone1 = new Phone { Id=1, Name = "CELL", Number = "123" };
var phone2 = new Phone { Id=2, Name = "LAND", Number = "456" };

var customerOne = new Customer
{
	Id=1,
	Name = "Jay",
	Phones = new List<Phone> { phone1, phone2 },
};

var customerTwo = new Customer
{
	Id = 2,
	Name = "Jay",
	Phones = new List<Phone> { phone1, phone2 },
};

var store = new DataStore("c:\\dev\\data.json");
var collection = store.GetCollection<Customer>();
var done = collection.DeleteMany(x=>x.Name != "");

collection.InsertOneAsync(customerOne);
collection.InsertOneAsync(customerTwo);

var results = collection.AsQueryable();
results.Dump();

foreach(var cus in results) {
	cus.Phones.RemoveAll(x=>x is not null);

// Does not update the json file
var updateTask = collection.UpdateOneAsync(cus.Id, cus);
updateTask.Wait();
}

results = collection.AsQueryable();
results.Dump();

}

On the above code, the UpdateOneAzync updates the in-memory data but not the actual file.

Also what is the best way to drop the whole table? I am doing

var done = collection.DeleteMany(x=>x.Name != "");

and it works, but that's ugly

Thanks

Jay

Typed data : Using a property named "Id" in Two classes !!!

I've created 2 classes Person and PersonFiled . When I try to insertOne in collection of the second class . It throws an exception : Unhandled Exception: System.ArgumentException: Object of type 'System.Int32' cannot be converted to type 'System.String'.
And I'm Obliged to change the name of the property "id" in PersonFiled class , to anything else than "id"

public class Person
    {
        public string Id;

        public string Name; 
    }

    public class PersonField
    {
       // I have to change Id to FieldId or anything  else
        public string Id { get; set; }

        public string Index { get; set; }

        public string Name { get; set; }

        public string Type { get; set; }
   
    }

看到这个我醉了

——File.WriteAllText(path, content);
磁盘定位都没有，如何用datastore词语，只适合config 或 setting。简单的事情搞复杂了。

Dynamic update of child items

Please could you help me and tell me if what I am trying to achieve is possible; finding child elements and updating them.

{
  "Parents": [
    {
      "Name": "Parent1",
      "Children": [
        {
          "Name": "Parent1Child1",
          "Description": "P1-C1"
        },
        {
          "Name": "Parent1Child2",
          "Description": "P1-C2"
        },
      ]
    },
    {
      "Name": "Parent2",
      "Children": [
        {
          "Name": "Parent2Child1",
          "Description": "P2-C1"
        }
      ]
    }
  ]
}

var store = new DataStore(@"C:\test.json");
var parents = store.GetCollection("Parents");
var parent1 = parents
			.AsQueryable()
			.FirstOrDefault(p => p.Name == "Parent1");
var children = parent1.Children;
var child2 = ((IEnumerable<dynamic>)children).FirstOrDefault(t => t.Name == "Parent1Child2");
	
((dynamic)child2).Description = "UPDATED";
	
parents.UpdateOneAsync("Parent1", child2).GetAwaiter().GetResult();

This updates the correct item (parent1 child 2), but it's not saved back to the test.json file

How to get whole document?

Is there a way to retrieve a whole document, not just some of it's properties or collections?

For a json file from a docs' sample:

{
  "selected_user": { "id": 1, "name": "Phil", "age": 40, "city": "NY" },
  "temperature": 23.45,
  "note": "this is a test"
}

How could one retrieve it a as a whole into an instance of a class?

public class Settings
{
    public User SelectedUser { get; set; }
    public double Temperature { get; set; }
    public string Note { get; set; }
}

Error handling Guid in ObjectExtensions.AddDataToField

idProperty.SetValue(item, data) throws an error when data is Guid
System.ArgumentException {"Object of type 'System.String' cannot be converted to type 'System.Guid'."}

My solution proposal :

No Iterator over Collections?

This is a very neat little library, thanks.

Now I've implemented my code, tested it, and realized to my dismay, that there's no iterator over the collections. Or is there a trick that I'm missing?

I'm just looking for something like db.GetCollections(). Would it be possible to provide this? I imagine it can't be too difficult to implement.

Single Item last field gets duplicated on Replace (When upsert: true and field names in Datastore are in camel case)

Bug:

• When replacing a field value on single item mode, the last field that gets updated duplicates itself in the DataStore file.

Replication:

public static class Program
{
    public static void Main(string[] args)
    {
        var entity = new Entity();
        
        for (int i = 1; i <= 2; i++)
        {
            // First to Update
            entity.DoesNotDuplicate = $"Updated {i} Times, Doesn't Duplicate";
                    
            // Last to Update
            entity.Duplicates = $"Updated {i} Times, Duplicates";
            Thread.Sleep(500);
        }
    }
    internal class Entity
    {
        private static readonly DataStore _dataStore = new DataStore("DuplicateBug.json");

        internal string? DoesNotDuplicate
        {
            get => _dataStore.GetItem(nameof(DoesNotDuplicate));
            set => _dataStore.ReplaceItem(nameof(DoesNotDuplicate), value, true);
        }

        internal string? Duplicates
        {
            get => _dataStore.GetItem(nameof(Duplicates));
            set => _dataStore.ReplaceItem(nameof(Duplicates), value, true);
        }
    }
}

Output:

{
  "doesNotDuplicate": "Updated 2 Times, Doesn't Duplicate",
  "duplicates": "Updated 1 Times, Duplicates",
  "duplicates": "Updated 2 Times, Duplicates"
}

Observation:

• the doesNotDuplicate field is updated correctly, does not duplicate itself since it is the first field updated.
• the duplicates field is not updated correctly, and duplicates itself instead.

Add support for single values

Data store only supports now collections. Add support for single values, kind of same way as Redis has a support for string data type (https://redis.io/topics/data-types-intro).

{
  "my_key": 2.4,
  "some_other_key": {
    "id": 22,
    "value": "this is the one"
  },
  "already_supported": [
    { "key": 2 },
    { "key": 3 }
  ]
}

var store = new DataStore("ds.json");

var itemDynamic = store.GetItem("my_key");
var itemTyped = store.GetItem<double>("my_key");

Feature branch: https://github.com/ttu/json-flatfile-datastore/tree/support-single-value

Guid persistence as Id

Hi!
I have entity with Id (keyProperty: nameof(Entity.Id)); as Guid but i receive this error

Can you help me?
Thanks!

Clarification, not issue

Can I query data through sql e.g. "SELECT c.name from c..."?

Add benchmarks

Branch: https://github.com/ttu/json-flatfile-datastore/tree/add-benchmarks

Collection-methods with id-field value parameter

Instead of Predicate-func pass the value that will be compared to value of the id-field.

/// <summary>
/// Update the item matching the id field value
/// </summary>
/// <param name="idValue">Item with id will be replaced</param>
/// <param name="item">New content</param>
/// <returns>true if item found for update</returns>
Task<bool> UpdateOneAsync(dynamic idValue, dynamic item);

// Update employee
employee.Name = "John Doe";

await collection.UpdateOneAsync(employee.Id, employee);

// Same as
await collection.UpdateOneAsync(e => e.Id == employee.Id, employee);

This way using collections in more dynamic way is easier as there is no need to create predicates dynamically.

How to Update child item

Is it possible to update child item level?
or I need update whole alert list of EID/AlertList level? Problem is it could be 100 to thousand.
eg: below, to update Collection/ID1/AlertList/ID1/AlertMsg (from James1 to XXX)

{
  "AlertCollection": [
    {
      "id": 1,
      "EID": "E-16",
      "AlertList": [
        {
          "id": 1,
          "alertMsg": "James1"
        },
        {
          "id": 2,
          "alertMsg": "James2"
        }
      ]
    },
    {
      "id": 2,
      "EID": "E-17",
      "AlertList": [
        {
          "id": 1,
          "alertMsg": "James"
        },
        {
          "id": 2,
          "alertMsg": "James"
        }
      ]
    }
  ]
}

Add optional _last_modified metadata to JSON

Option to add last modified timestamp for each object.

{
  "user": [
    {
      "id": 1,
      "name": "James",
      "age": 40,
      "location": "NY",
      "work": {
        "name": "ACME",
        "location": "NY"
      },
      "_last_modified": "2017-10-14T09:45:10"
    },
    {
      "id": 2,
      "name": "Phil",
      "age": 25,
      "location": "London",
      "work": {
        "name": "Box Company",
        "location": "London"
      },
      "_last_modified":  "2017-10-15T12:12:12"
    }

Typed models need to have property defined

public DateTime _last_modified { get; set; }

Feature branch: https://github.com/ttu/json-flatfile-datastore/tree/add-last-modified

Make DocumentCollection thread safe

Use lock when accessing DocumentCollection's internal list

[Fact]
public async Task InsertOne_1000ParallelAsync()
{
	var newFilePath = UTHelpers.Up();

	var store = new DataStore(newFilePath);

	var collection = store.GetCollection<User>("user");
	Assert.Equal(3, collection.Count);

	var tasks = Enumerable.Range(0, 1000)
		.AsParallel()
		.Select(i => collection.InsertOneAsync(new User { Id = i, Name = "Teddy" }))
		.ToList();

	await Task.WhenAll(tasks);

	var store2 = new DataStore(newFilePath);
	var collection2 = store2.GetCollection<User>("user");
	Assert.Equal(1003, collection2.Count);
	var distinct = collection2.AsQueryable().GroupBy(e => e.Id).Select(g => g.First());
	Assert.Equal(1003, distinct.Count());

	UTHelpers.Down(newFilePath);
}

Result StackTrace:	
at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource)
   at System.Collections.Generic.List`1.Enumerator.MoveNextRare()
   at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source)
   at JsonFlatFileDataStore.DocumentCollection`1.GetNextIdValue(List`1 data) in C:\src\GitHub\json-flatfile-datastore\JsonFlatFileDataStore\DocumentCollection.cs:line 364
   at JsonFlatFileDataStore.DocumentCollection`1.<>c__DisplayClass14_0.<InsertOneAsync>b__0(List`1 data) in C:\src\GitHub\json-flatfile-datastore\JsonFlatFileDataStore\DocumentCollection.cs:line 58
   at JsonFlatFileDataStore.DocumentCollection`1.<InsertOneAsync>d__14.MoveNext() in C:\src\GitHub\json-flatfile-datastore\JsonFlatFileDataStore\DocumentCollection.cs:line 63
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at JsonFlatFileDataStore.Test.CollectionTests.<InsertOne_100Async>d__26.MoveNext() in C:\src\GitHub\json-flatfile-datastore\JsonFlatFileDataStore.Test\CollectionTests.cs:line 771
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
Result Message:	System.InvalidOperationException : Collection was modified; enumeration operation may not execute.

System.OperationCanceledException When Disposing DataStore

I'm using your library to implement a set of local data stores. I love this library and the implementation. I am seeing a behavior I am trying to resolve as I am implementing unhandled exception handling now in the my .NET 6 WPF application. So this keeps tripping it...

Here is an example of one of my methods on one of the repositories:

        public IEnumerable<QuestionSet> GetAll()
        {
            using (var ds = new DataStore(_fileName, true, nameof(QuestionSet.PrimaryKey)))
            {
                var coll = ds.GetCollection<QuestionSet>();
                return coll.AsQueryable().ToList();
            }
        }

After the 'using' exits I always get a: System.OperationCanceledException

If I don't use 'using' or do not Dispose, I don't see the exception.

Should I not be wrapping the DataStore object in a using? Or am I using the library in the wrong way?

Index on property

Hi

Are there any plans to add indexes on properties?

It takes a long time to add items to a collection with half a million objects.

Thanks

Provided inserting object `Id` type is `string` -> `Id` is still autogenerated

When the inserting object Id type is string and value is for example valid GUID, the engine still auto-generates the Id and ignore the provided value.

System.Text.Json

Any plans to move the library to native system.text.Json from Newtonsoft?
It is an unnecessary dependency and going forward likely to be deprecated

NuGet Error NU1108

I am getting the A circular dependency was detected error if I use Net Core 2.2, if I use Net 4.5.X all is good.