gs1993 / so

Application created to test the optimization of EF on a large data set

DDD and CQRS on a large-scale dataset

The technology demonstration app uses Entity Framework Core to manage 400GB of data from the StackOverflow Database in a clean and consistent manner

Table of Contents

• Architecture
• Features
  • CQRS
  • Tests
  • Domain Driven Design
  • APIs
  • Machine Learning
  • Feature flags
• Setup

Architecture

Features

CQRS

Command-query separation:

Read-write separation on application level:

1. Separate connection strings for read and write databases
2. Independent Read and write db contexts
3. More efficient scaling on micro and macro levels

EF Compiled Queries:

Employing Compiled Queries serves as an effective strategy to significantly boost the performance on the data retrieval side of the application.

Tests

1. Unit tests - in-depth tests for complex domain logic
2. Integration tests - api-level test with a separate database created dynamically using Docker
3. Benchmark test - comparison of REST and GraphQL performance
4. Mutation test - using stryker-net

Domain Driven Design

Bounded Contexts:

Rich domain model:

Value Objects:

APIs

REST - with Swagger documentation

GraphQL - using ChilliCream GraphQL Platform

gRPC - using Google.Protobuf

Machine Learning

Spam detection is accomplished through a machine learning model trained on a large dataset of previous posts from a database. The project harnesses the power of the ML.NET library to efficiently analyze and identify spam, ensuring a high level of accuracy and reliability.

Training results:

Feature flags

Feature flags provide a easy way to turn features on or off dynamically. FeatureFlagController.cs - demo edpoints FeatureFlags.cs - logic for the flags appsettings.json - set by "FeatureFlags" section in appsettings

---

Setup

Api

1. Download and install StackOverflow Database - version 2018-06 or newer
2. Go to Api/appsettings.json and set the connection strings to database
3. Run database migrations

dotnet ef database update

4. Run project and go to app url: http://localhost:5000/swagger/index.html

Machine learning models

1. Go to ModelBuilder/appsettings.json. Set ConnectionString to database and SaveModelPath - (path to generate ML model)
2. Run ModelBuilder in release mode and wait for ML model to generate .zip file in specified location
3. Go to PredictionEngineApi/appsettings.json. Set PredictionModelPath (path and name of generated ML model)
4. Run project and go to app url: http://localhost:5001/swagger/index.html
5. aa docker-compose -f elasticsearch-kibana-docker-compose.yaml up -d