devrexlabs / memstate Goto Github PK

OrigoDB models derived from MarshalByRefObject and used the built in proxy. Memstate proxy feature is based on System.ReflectionDispatchProxy which can only proxy interfaces. The tests need to be updated to use interface based models.

Also, make subscriptions push based using domain events. See http://origodb.com/docs/core-0.19/models/messaging/

Port origodb docs, modify accordingly. Use GH flavored markdown in the docs sub folder.

When the engine starts, the entire journal is replayed. We should support uses cases where we load a portion of the journal by providing arguments such as LoadFromRecord, LoadToRecord, LoadFromPointInTime, LoadToPointInTime.

In the case that we do not read to the end, the engine should be in readonly mode, or perhaps start writing a new stream.

possible deadlock/locking issue. Investigate SystemTests.Smoke()

draft suggestion for recording Requirements

Refactor the project structure in order to capture requirements as executable acceptance tests, so that anyone volunteering to work on memstate can know the status of what features have been built, and are currently available to use and work (based on the current release).

• requirements should be written at a high enough level describing the required benefits that the feature should deliver, and not describe the how, so that if we change the specific implementation the test should as far as possible not require much changing.

• requirements for any feature that a dev is going to work on should be written before any code is written. (so that we can make sure the requirements are clear and fully understood.)

• Suggestion : requirements can be attached to the issue or project board and recorded in Gherkin syntax, see example below

• be written as BDD style acceptance tests so that we can run the tests as part of CICD.

• acceptance tests should contain sufficient examples that anyone new can easily see the intention of the feature.

• the final acceptance test that is written should show the simplest code possible that proves that the system behaves as the documentation describes. (see example at the bottom) This is really to prove to a user that the behaviour works. The real proof for us, will be done through a complete set of unit, integration, performance, load and security tests. These tests will cover a wide set of edge cases that are not necessary to include in the acceptance tests

• the acceptance tests should use as few Mocks as possible to simulate as far as possible the actual code that a user could put together in a simple console application. (so that the tests can serve as valuable and real how-to code examples.)

• there will be some minor duplication between work on the requirements and the documentation. The difference between the 'docs' and 'requirements' is that the docs and getting started focuses on a slice through memstate and covers only the basics. Requirements are written 'per feature', every time someone extends memstate with new functionality. If a new piece of functionality is something that a user needs to be aware of from the very beginning of using memstate then we'd update the documentation at the same time.

sample feature expressed in Gherkin

# sample `feature` text notes for a feature, saved in the  github project or issue
# this get's translated into a C# test when a developer starts to work on the issue or feature.
# these are written and reviewed *before* development starts. 

Epic : Account/ClosingAccount
Feature : user cannot close account with a negative balance

Given an account with <balance>
when the account is closed()
then the result is <result>

Examples

    balance | result
    --------| ------
    [NULL]  | closed
    0       | closed
    -10     | exception
    10      | closed

acceptance test translated to C#

• in this example, someone pulled a card from the backlog called, "user cannot close account with negative balance". This sample shows that features can have more features (e.g. epics) and can be modified over time, and still keep all the tests and requirements together in one place.
• rough sample below showing suggested folder structure and test naming for epics
• class ClosingAccount would reside in a folder in file /Memstate.Requirements/Account/ClosingAccount.cs
• this convention allows us to have any amount of nesting as appropriate on a case by case basis.

  namespace Memstate.Requirements.Account
{

  // using nested classes for epics to group all the requirements in one place
   public class ClosingAccount
  {
     [TestCase(null)]
     [TestCase(0M)]
     [TestCase(100M)]
      [Test]
       public void user_closing_account_with_positive_zero_or_null_balance_should_close_account(decimal balance)
      {
          var account = new Account(balance);
          var result = account.Close();
          Assert.AreEqual(result, Result.Closed);
      }
 
     [Test]
       public void user_closing_account_with_negative_balance_should_throw_exception()
      {
          var account = new Account(-10M);
         Assert.Throws<Exception>( a=> account.Close());
      }
 
 }

}

Suggestion above based on a gitter discussion where I proposed updating the project to record requirements using the same syntax I use for requirements in my Konsole project here : https://github.com/goblinfactory/konsole/blob/master/Konsole.Tests/WindowTests/SplitTests.cs

Implement JSON/XML over HTTP for commands and queries. This will allow access from any client that can speak http.

Using the pgsql notfications feature we can notify replicas when new commands are written.

When using JsonSerializerAdapter and default settings, the resulting generated journal file cannot be read.

The test below fails with;

System.NotSupportedException : Specified method is not supported.
   at Memstate.JsonNet.SurrogateConverter.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer) in C:\src\git-alan-public\memstate-fork-alan\Memstate.JsonNet\SurrogateConverter.cs:line 43

Steps to reproduce the error, run this test

        [Test]
        public async Task Json_file_created_should_be_able_to_be_deserialized()
        {
            Print("Given a journal file");
            var settings = new MemstateSettings { StreamName = "json-example2" };
            settings.Serializers.Register("a-unique-key", _ => new JsonSerializerAdapter(settings));
            settings.Serializer = "a-unique-key";
            var model1 = await new EngineBuilder(settings).BuildAsync<LedgerDB>();

            await model1.ExecuteAsync(new InitAccount(accountNumber: 1, openingBalance: 100.11M, currency: '£'));
            await model1.ExecuteAsync(new InitAccount(accountNumber: 2, openingBalance: 200.22M, currency: '£'));
            await model1.ExecuteAsync(new Transfer(fromAccount: 1, toAccount: 2, amount: 0.01M, currency: '£'));
            var accounts1 = await model1.ExecuteAsync(new GetAccounts());
            await model1.DisposeAsync();
            Assert.True(File.Exists("json-example2.journal"));

            Print("When I create an new LedgerDB that will read the recently created ledger file");
            var model2 = await new EngineBuilder(settings).BuildAsync<LedgerDB>();

            Print("then the journal entries should have been read in");
            var accounts2 = await model2.ExecuteAsync(new GetAccounts());
            Assert.AreEqual(2, accounts2.Count);

            Print("And the in memory object database should be back to the latest state");
            CollectionAssert.AreEqual(accounts1, accounts2);

        }

You can use any Engine<T> with any commands. The important point is the serializer settings.

Can you clarify the difference between using a subscription:

var sub = config.GetStorageProvider().CreateJournalSubscriptionSource();
sub.Subscribe(nextRecord, record =>
{
}

and using the CommandExecute event:

Engine.CommandExecuted += EngineOnCommandExecuted;
void EngineOnCommandExecuted(JournalRecord journalrecord, bool islocal, IEnumerable<Event> events)
{
}

Also whether there's a difference at startup / playback?

I've found an issue where a null value in an 'exotic' type from a third party library doesn't get deserialized correctly due to the settings for Json.NET.

I've posted a spike to my fork:

https://github.com/Simcon/memstate/commit/71870e91aad95a5a3d23bf6803b5b47b56c87501

Check out the addition of NullValueHandling = NullValueHandling.Ignore to the JsonSerializerAdapter constructor. This fixes the issue in the corresponding test.

The simple fix is to add the fix to mainline. Would this be possible, @rofr ?

The 'right' fix would be to allow specifying serialization settings but this non-trivial as far as I can see.

The file journal is a single file that grows indefinitely. We want to have it in manageably sized chunks and a naming scheme that reflects the ordering and recordnumbers, eg my-journal.00000042.000023414.journal

https://msdn.microsoft.com/en-us/library/system.net.security.sslstream.aspx

1. Ensure all files only contain a single type (Inner types allowed).
2. Rename MemstateServer and MemstateClient to follow a convention.
3. Remove unused usings.
4. Remove unused types, methods, properties and fields.

Sustainable and clean way to handle configuration settings across the library including modules.

Replace all the Xunit tests with nunit.

There's no need to have both Xunit and nunit. Nunit appears to have better support for automating .net standard tests in Cake.

copy origodb.com site and tweak from there. origodb.com is based on jekyll, host with firebase because gh-pages does not (yet) support https for custom domains.

Specifically : Sample commands and DTOs are no longer valid.
Should be immutable.
Customer.cs sample code is missing.
Remove anon constructor from sample Command.
Update commentary.

Build and push nuget packages on demand or based on release branch. appveyor? circleci? cake build?

Settings are populated by the configuration subsystem. This means users don't create the settings objects, instead they obtain a reference to them by calling Config.Current.GetSettings<TSettings>().

This is ugly but can be hidden from the user by accepting an Action to various methods, for example var engine = Engine.For<MyModel>((EngineSettings s) => s.StreamName="myname") ; or Config.Current.Configure<EngineSettings>(s => s.StreamName)

A proxy looks like a model of type T but wraps an Engine and translates method calls to commands and queries. See http://origodb.com/docs/core-0.19/modeling/proxy/

Tests that measure throughput and tail latency for various workloads

engine.Execute(string query, params object[] args) where query is compiled and cached on the node. See
https://github.com/DevrexLabs/OrigoDB/tree/dev/src/OrigoDB.Core/Linq

Implement built-in metrics using the AppMetrics library.

• https://github.com/AppMetrics/AppMetrics
• https://github.com/AppMetrics/AppMetrics

Start with the following metrics:

• Execution time of queries (Histogram)
• Execution time of commands executed by kernel (Histogram)
• Execution time of commands executed by engine (Histogram)
• Commands pending (Gauge)
• Commands executed (Counter)
• Queries executed (Counter)
• Failed commands (Counter)
• Failed queries (Counter)

https://github.com/ufcpp/ContextFreeTask

Update getting started Commands to not hide base ID property so that command can be immutable.

The demo commands in src/Memstate.Docs.GettingStarted/QuickStart/Commands/EarnPoints.cs are mutable, because they can't be serialized due to a problem caused by hiding the base ID property.

If the Command DTO's in the quickstart getting started guide are changed to not hide the ID property then they can be made into proper immutable classes, and that would be fabulous, the way it's supposed to be!

RecordNumber should start from 0 or 1 but must be consistent across all storage providers. Currently, eventstore starts with 0, postgres starts with 1.
Consolidate storage providers to use 0 and fix engine/builder/storage interfaces accordingly.

This would be a .NET Framework console or windows application that references OrigoDB (with modules) and memstate (with all modules). It will need to dynamically reference the assembly containing types serialized to the journal.

Failed CanWriteManyAndReadFromMany(Provider:Memstate.EventStore.EventStoreProvider, Memstate.EventStore, Version=0.1.0.0, Culture=neutral, PublicKeyToken=null, Serializer: Memstate.JsonNet.JsonSerializerAdapter, Memstate.JsonNet, Version=0.1.0.0, Culture=neutral, PublicKeyToken=null, Name:memstatebae4e34664)
Error Message:
Expected: 300
But was: 0

Stack Trace:
at System.Test.ClusterTests.d__4.MoveNext() in /Users/rf/memstate/src/System.Test/ClusterTests.cs:line 136

Standard Output Messages:
C: Provider:Memstate.EventStore.EventStoreProvider, Memstate.EventStore, Version=0.1.0.0, Culture=neutral, PublicKeyToken=null, Serializer: Memstate.JsonNet.JsonSerializerAdapter, Memstate.JsonNet, Version=0.1.0.0, Culture=neutral, PublicKeyToken=null, Name:memstateb34b2a1454
[23,11:55:07.856,DEBUG] Deserialization to EventStore.ClientAPI.Messages.ClientMessage+StreamEventAppeared failed : System.MissingMethodException: No parameterless constructor defined for this object.
at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor)
at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark)
at System.Activator.CreateInstance(Type type, Boolean nonPublic)
at System.Activator.CreateInstance(Type type)
at ProtoBuf.Serializers.TypeSerializer.CreateInstance(ProtoReader source, Boolean includeLocalCallback) in C:\code\protobuf-net\src\protobuf-net\Serializers\TypeSerializer.cs:line 314
at ProtoBuf.Serializers.TypeSerializer.Read(Object value, ProtoReader source) in C:\code\protobuf-net\src\protobuf-net\Serializers\TypeSerializer.cs:line 209
at ProtoBuf.ProtoReader.ReadTypedObject(Object value, Int32 key, ProtoReader reader, Type type) in C:\code\protobuf-net\src\protobuf-net\ProtoReader.cs:line 607
at proto_16(Object , ProtoReader )
at ProtoBuf.Meta.TypeModel.DeserializeCore(ProtoReader reader, Type type, Object value, Boolean noAutoCreate) in C:\code\protobuf-net\src\protobuf-net\Meta\TypeModel.cs:line 748
at ProtoBuf.Meta.TypeModel.Deserialize(Stream source, Object value, Type type, SerializationContext context) in C:\code\protobuf-net\src\protobuf-net\Meta\TypeModel.cs:line 606
at ProtoBuf.Serializer.Deserialize[T](Stream source) in C:\code\protobuf-net\src\protobuf-net\Serializer.cs:line 84
at EventStore.ClientAPI.Transport.Tcp.ProtobufExtensions.Deserialize[T](ArraySegment`1 data)

Design and implement nuget packaging and naming strategy. We want a core package with 0 dependencies besides NET Standard. Memstate.Core is not a good name, it can be confused with dotnet core. How about Memstate.Minimal?

Modules could have Module in the name. For example:

• Memstate.Modules.EventStore
• Memstate.Modules.Postgres

And an easy to get started, single package with everything bundled. This package would also contain versions verified to work together. What would be a good name?

What about 3rd party contributions? Memstate.Contrib.*?

In the EventStore project, implement a storage provider that uses an embedded event store node, see https://eventstore.org/docs/dotnet-api/4.0.2/embedded-client/

Provide project templates for both Visual Studio and dotnet new along with documentation on how to obtain, install and use.
How To Create A dotnet new Project Template In .NET Core | Pioneer Code
https://pioneercode.com/post/how-to-create-a-dot-net-new-project-template-in-dot-net-core

Storage Providers

• EventStore
• PostgreSql

Note: InMemoryStorage and FileStorage do not support multiple nodes.

Test cases

• Start multiple engines from an empty store, execute commands on a single engine, verify replication to other engines
• Execute commands on multiple engines, verify that all nodes have identical models
• Start single engine, execute commands, start additional engines, ensure they are in sync

json-example.journal.txt

failing test

(Update : these tests are now out of date.)

Given a memstate db configured to write journal extries as json, 1 line of json per journal entry
When I run a few commands
then the journal file should exist
And the journal file should have saved the entries as text, 1 line per entry

The following code below produces the attached file (see atop) If you open the text file and see multiple lines, turn off word wrap :D

        [Test]
        public async Task Simple_end_to_end_with_human_readable_json_journal_file()
        {
            Print("Given a memstate db configured to write journal extries as json, 1 line of json per journal entry");
            var settings = new MemstateSettings { StreamName = "json-example" };
            settings.Serializers.Register("a-unique-key", _ => new JsonSerializerAdapter(settings));
            settings.Serializer = "a-unique-key";
            var model1 = await new EngineBuilder(settings).BuildAsync<LedgerDB>();

            Print("When I run a few commands");
            await model1.ExecuteAsync(new InitAccount(accountNumber: 1, openingBalance: 100.11M, currency: '£'));
            await model1.ExecuteAsync(new InitAccount(accountNumber: 2, openingBalance: 200.22M, currency: '£'));
            await model1.ExecuteAsync(new Transfer(fromAccount: 1, toAccount: 2, amount: 0.01M, currency:'£'));
            var accounts = await model1.ExecuteAsync(new GetAccounts());
            await model1.DisposeAsync();
            accounts.ForEach(Console.WriteLine);

            Print("then the journal file should exist");
            Assert.True(File.Exists("json-example.journal"));

            Print("And the journal file should have saved the entries as text, 1 line per entry");
            var lines = File.ReadAllLines("json-example.journal");
            Assert.AreEqual(3, lines.Length);
            StringAssert.Contains("M100.11", lines[0]);
            StringAssert.Contains("M200.22", lines[1]);
            StringAssert.Contains("M0.01", lines[2]);

            // to view the journal file in an editor comment out the File.Delete in the test Setup/teardown.
            // The journal file will be located in 
            // Memstate.Docs.GettingStarted\bin\Debug\netcoreapp2.0\json-example.journal
        }

Queries and Commands may carry mutable objects reachable from the outside into the model or return mutable references to objects within the model breaking isolation.

One extreme is to serialize all incoming commands and outgoing results, the other extreme is to document the phenomenon well and let users deal with it. Somewhere in between, intelligent or configurable seems like a reasonable approach. OrigoDB takes this approach but the design could be improved. Here's a quote from the docs:

There are two types of isolation to consider when using OrigoDB: * Isolation between transactions * Isolation of the model from user code when running OrigoDB in-process

Commands are executed sequentially, thus fully isolated. The default ISynchronizer uses a ReaderWriterLockSlim to allow either a single writer or multiple readers at any given time. This guarantees that reads always see the most recent state (and that the state is not modified) for the entire duration of the transaction.

By default, commands, command results and query results are cloned to prevent leaking references to mutable objects within the model. Cloning uses serialization/deserialization and can have a significant impact on performance. By designing for isolation all or some of the cloning can be disabled. See Configuration/Isolation for details on how to configure what gets cloned and not.

And here's the documentation page on isolation: http://origodb.com/docs/core-0.19/configuration/isolation/

When running memstate server, as opposed to in-process, all objects in and out are serialized over the wire. In this case isolation is ALMOST completely solved. The exception is if a mutable reference is returned from an operation, there is a small time window where corruption can take place: after the kernel releases the lock and before the result object is serialized.

Bt default, the kernel uses a global ReaderWriterLockSlim to allow either a single writer or multiple readers at any given time. This is a simple but powerful locking model that guarantees serializable isolation. But a major drawback is that command/query execution time will add to latency and any long running operation will block throughput.

A simple scheme to address this issue would be to used named locks. A given command/query is either associated with a named lock, or will default to the global lock. This will allow more granular locking, bypassing locks and other relaxed locking options.

And if the user has a thread safe model, we should be able to disable locking altogether. So an ILockingStrategy interface with NullLockingStrategy and ReaderWriterLockSlimStrategy implementations looks like a good approach.

OrigoDB has an immutability model where the entire model is immutable, and commands return a new model. This might at some point be ported to memstate. However, using locking options as proposed above would allow portions of the model to be immutable. Example:

public class MyModel
{
   ImmutableList<Customer> Customers {get;set;}
}

public class AddCustomer : Command<MyModel>
{
   public Customer Customer{get;set;}

   public void Execute(MyModel model)
   {
      model.Customers = model.Customers.Add(Customer);
   }
}

The huge benefit here is that a query can grab a reference to the Customers list and will get a point in time snapshot without blocking concurrent AddCustomer commands.

Ability to emit domain events when a command executes. Events should propagate to subscribed clients connected to any node. See http://origodb.com/docs/core-0.19/modeling/domain-events/ for context

A thin wrapper taking a config file and/or command line args and hosting a server node

Add an http endpoint to memstate server exposing metrics, dashboard, etc. Future version could have the ability to execute commands/queries, browse data.

Hangs for both postgres and eventstore providers.
Only writes a single event to eventstore and then hangs.
Postgres benchmark produces output then hangs at the end of the benchmark run.

Unit tests in Memstate.Test take an unnecessary long time to run.

• It looks like this might be down to GeopointTests. maybe due to the custom test case provider which appears to hang the test runner, not the tests itself. The result is that running the unit tests takes around a minute to run wheras all the tests should be running within milliseconds each.

• the result is that I'm not able to sort the tests by duration since the problem appears to be the interaction with the runner, not the test itself. According to the measured tests the tests themselves run very quickly.

• have not tested this with Ncrunch, but having something unstable means I don't want to crunch (i.e. have these tests running all the time).

• sometimes running the tests leaves the test runner in an unstable state. (needs more info). I'll update this with more information when this card gets pulled into play.

• This tests also causes 46 warnings, unit test element ... reported as active although it's already finished (Success)
  

Separate journal writer process that accepts journal entries/batches over tcp, commits to pluggable backing store and notifies connected memstate engines. This process will be responsible for global orderings of commands and assigning record numbers. Can only run as a single process so needs to be lightweight and load quickly to ensure availability.

powershell ./build.ps1 -Target Test

========================================
Test
========================================
Executing task: Test
xUnit.net Console Runner (64-bit .NET 4.0.30319.42000)
System.InvalidOperationException: Unknown test framework: could not find xunit.dll (v1) or xunit.execution.*.dll (v2) in
 C:\onedrive\git\memstate\EventStore.Tests\bin\Debug\netcoreapp1.1
System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=4.1.0.0, Culture=neutral, Publ
icKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
System.InvalidOperationException: Unknown test framework: could not find xunit.dll (v1) or xunit.execution.*.dll (v2) in
 C:\onedrive\git\memstate\EventStore.Tests\obj\Debug\netcoreapp1.1
System.InvalidOperationException: Unknown test framework: could not find xunit.dll (v1) or xunit.execution.*.dll (v2) in
 C:\onedrive\git\memstate\Memstate.Tests\bin\Debug\netcoreapp1.1
System.IO.FileNotFoundException: Could not load file or assembly 'Memstate.Tests, Version=0.0.0.0' or one of its depende
ncies. The system cannot find the file specified.
System.InvalidOperationException: Unknown test framework: could not find xunit.dll (v1) or xunit.execution.*.dll (v2) in
 C:\onedrive\git\memstate\Memstate.Tests\obj\Debug\netcoreapp1.1
System.InvalidOperationException: Unknown test framework: could not find xunit.dll (v1) or xunit.execution.*.dll (v2) in
 C:\onedrive\git\memstate\Memstate.Tests\obj\Debug\netcoreapp1.1
An error occurred when executing task 'Test'.
Error: xUnit.net (v2): Process returned an error (exit code 1).

accept and process commands / queries over a TCP connection. Domain event subscriptions for pushing domain events from the server to connected clients

see http://origodb.com/docs/core-0.19/models/geo/