Microflow

Welcome to Microflow. This is a fun project that I am working on in my free time. The idea came from the business need to execute workflows in the best scalable way. Durable Functions is a great choice for this, the only drawback is that Durable Function workflows are normally hard coded by developers. This leads to a situation where workflow changes must go through the normal development life cycle (dev, test, production). What if the users designing the workflows are not developers? What if the business needs to make workflow changes quickly without the need for code changes? These are the problems that Microflow addresses, keeping the greatness of serverless Durable Functions and making the workflows dynamic for easy changeability.

Overview

Microflow is a dynamic serverless micro-service workflow orchestration engine built with C#, .NET Core 3.1, and Azure Durable Functions. Microflow separates the workflow design from the function code so that workflows don`t have to be hard coded as with normal Durable Functions. The workflow can be designed outside of Microflow and then passed in as JSON for execution, so no code changes or deployments are needed to modify any aspect of a workflow. Microflow can be deployed to the Azure Functions Consumption or Premium plans, to an Azure App Service, or to Kubernetes.

Microflow functionality:

dynamic json workflows separate and changeable outside of Microflow, workflows are dynamic and changeable without Microflow knowing about it
the micro-service implementations are cleanly separated from Microflow
Microflow workflow projects can run as single instances (like a risk model that should always run as 1 instance), or can run as multiple parallel overlapping instances (like ecommerce orders)
for custom logic like response interpretations, this can be included in Microflow, but best practice is to separate these response proxies as functions outside of Microflow, and then these will call back to Microflow
parent-child-sibling dependencies, parallel optimized execution, parent steps execute in parallel
auto scale out to 200 small virtual machines in the Consumption Plan, and to 100, 4 core CPUs, 14GB memory virtual machines in the Premium Plan
easily manage step configs with merge fields
do batch processing by looping the workflow execution with Microflow`s "Loop" setting, set up variation sets, 1 variation set per loop/batch
set outgoing http calls to be inline (wait for response at the point of out call), or wait asynchronously by setting CallbackAction (wait for external action/callback)
timeouts can be set per step for inline and callback
retry policies can be set for each step and there can also be a default retry policy for the entire workflow
StopOnActionFailed can be set per step to indicate for when there is a failure (not a success callback), which will make Microflow stop the workflow execution or to log the failure and continue with the workflow
stateful and durable! Microflow leverages Durable Functions, so even when the vm crashes, Microflow will continue from where it left off, before the crash, when a new vm becomes available (Azure will do this in the background)
leverage the Azure serverless plans: Serverless Consumption and Premium plans
Microflow can run anywhere on Kubernetes when the Azure serverless environment is not available
Microflow is lightweight and will auto scale when there is a usage spike

Microflow use cases:

any business workflow that needs to leverage serverless autoscaling durable stateful workflows that can run anywhere
avoid hard coded workflows
use Microflow as a state-of-the-art back end for your workflow designer, the workflow is just a simple json definition
use the Microflow SDK to build code-based workflows outside of Microflow, and then apply changes to the workflow without the need for any deployments
Microflow is open-source so feel free to modify Microflow to your needs

Future enhancements:

workflow stop, pause, and continue
Azure AD authentication
gRPC communication
proper logging implementation

Test Examples

The code for these can be found in the console app\Tests.cs. There is also a SimpleSteps test that consists of 1 parent with 2 children and the 2 children have a common child step (See JSON Workflow Example below). This is the diagram for the 2 more complex test workflows:

JSON Single Step with All Config:

{
   "StepId":1,
   "CalloutUrl":"http://localhost:7071/api/SleepTestOrchestrator_HttpStart",
   "CallbackAction":"approve",
   "StopOnActionFailed":true,
   "IsHttpGet":true,
   "ActionTimeoutSeconds":30,
   "SubSteps":[2,3],
   "RetryOptions":{
      "DelaySeconds":5,
      "MaxDelaySeconds":60,
      "MaxRetries":5,
      "BackoffCoefficient":1,
      "TimeOutSeconds":30
   }
}

JSON Workflow Example:

This simple workflow contains 1 parent step (StepId 1) with 2 sub steps (StepId 2 and StepId 3), and each sub step has 1 common sub step (StepId 4). This is the same structure as the included test Tests.CreateTestWorkflow_SimpleSteps(). StepId 1 has a callback action set, and StepId 3 has a retry set. There is 1 merge field set and is used as a default callout URL.

{
  "ProjectName": "MicroflowDemo",
  "DefaultRetryOptions": null,
  "Loop": 1,
  "MergeFields": {
    "default_post_url": "https://reqbin.com/echo/post/json?workflowid=<workflowId>stepid=<stepId>"
  },
  "Steps": [
    {
      "StepId": 1,
      "CalloutUrl": "{default_post_url}",
      "CallbackAction": "approve",
      "StopOnActionFailed": true,
      "ActionTimeoutSeconds": 30,
      "SubSteps": [2,3],
      "RetryOptions": null
    },
    {
      "StepId": 2,
      "CalloutUrl": "{default_post_url}",
      "CallbackAction": null,
      "StopOnActionFailed": true,
      "ActionTimeoutSeconds": 1000,
      "SubSteps": [4],
      "RetryOptions": null
    },
    {
      "StepId": 3,
      "CalloutUrl": "{default_post_url}",
      "CallbackAction": null,
      "StopOnActionFailed": true,
      "ActionTimeoutSeconds": 1000,
      "SubSteps": [4],
      "RetryOptions": {
        "DelaySeconds": 5,
        "MaxDelaySeconds": 10,
        "MaxRetries": 2,
        "BackoffCoefficient": 5,
        "TimeOutSeconds": 30
      }
    },
    {
      "StepId": 4,
      "CalloutUrl": "{default_post_url}",
      "CallbackAction": null,
      "StopOnActionFailed": true,
      "ActionTimeoutSeconds": 1000,
      "SubSteps": [],
      "RetryOptions": null
    }
  ]
}

Solution Description

MicroflowFunctionApp

This is the core of the workflow engine.

MicroflowStart

This contains 3 functions responsible starting project execution.

"Microflow_InsertOrUpdateProject" : Must be called first after project creation or modification before "Microflow_HttpStart" can be called. This will persist the needed project meta data that is needed for the project to run. When running multiple concurrent instances, it is not needed to call this every time when a new instance starts to run.
"Microflow_HttpStart" : This will start the run of a project by calling "Start".
"Start" : This starts the project run by getting a list of top level steps and then calls "ExecuteStep" for each top level step.

FlowControl

This contains 3 classes responsible for workflow execution.

CanStepExecuteNow.cs : Locks and checks the parent completed count to determine if a child step can execute, all parents must be completed for a child step to start execution. Parent steps execute in parallel.
Microflow.cs : This contains the recursive function "ExecuteStep". It calls the action URL and then calls CanExecuteNow for child steps of the current step.
MicroflowStart.cs : This is where the workflow JSON payload is received via http post and then prepares the workflow and calls start.

API

This currently contains 2 folders each with 1 class, 1 for internal and 1 for external api calls.

MicroflowConsoleApp

This console app is used to create test workflow projects and post to Microflow. After created or modifying a project, always call "Microflow_InsertOrUpdateProject" before calling the run http call "Microflow_HttpStart". After a call to "Microflow_InsertOrUpdateProject" is made, then "Microflow_HttpStart" can be called multiple times as long as the project definition stays the same.

MicroflowModels

This library holds model classes that is useful outside of the Microflow function app, but is also used by the Microflow function app.

MicroflowSDK

This library is used to help with project workflow and steps creates.

Setup Guide

Clone the repo locally. It is advised to separate the MicroflowConsoleApp from the MicroflowFunctionApp in Visual Studio, this is to be able to run MicroflowFunctionApp separately, and then run the MicroflowConsoleApp to post workflows to it:

Microflow Solution:

MicroflowConsoleApp Solution:

Microflow Solution Nugets:

MicroflowConsoleApp Solution Nugets:

Run the MicroflowApp (Function App)
Run the MicroflowConsole
- Choose which of the test workflows to post (in the file Tests.cs: CreateTestWorkflow_SimpleSteps(), CreateTestWorkflow_Complex1(), or CreateTestWorkflow_10StepsParallel())
- look at Program.cs: by default 1 instance with id 39806875-9c81-4736-81c0-9be562dae71e will run, but there is also a commented out loop for multiple concurrent instances
The run will 1st log to console in red: "Started Run ID 2d779289-01a5-50c5-b4f4-e6fa22a9fc96..."
Then each step will log success in orange" "Step 1 done at 10:38:57 - Run ID: 2d779289-01a5-50c5-b4f4-e6fa22a9fc96"
Then the run end will log in red: "Run ID 2d779289-01a5-50c5-b4f4-e6fa22a9fc96 completed successfully..."
Then the final last log in red: "Project run MicroflowDemo completed successfully..." and "<!!! A GREAT SUCCESS !!!>"
The step completions are also logged to storage table "LogSteps"
Main orchestration completions are logged to storage table "LogOrchestration"

Logging

Console Logging:

Red at start: "Started Run ID db7d8fd4-e7e3-5e57-9017-e7ec3ad6bb01..."
Orange for each completed step: "Step 1 done at 10:55:02 - Run ID: db7d8fd4-e7e3-5e57-9017-e7ec3ad6bb01"
Red at run end: "Run ID db7d8fd4-e7e3-5e57-9017-e7ec3ad6bb01 completed successfully..."
Red at project end: "Project run MicroflowDemo completed successfully..."
Red at project end: "<-----> !!! A GREAT SUCCESS !!! <----->"

Table Storage Logging

Logs top level orchestration processing info to storage table "LogOrchestration", logs step processing info to storage table "LogSteps". For both there is a start and end date in the log entry. When an orchestration or step starts, an entry with the start date is saved, and when this item completes, the same entry with the start date will get an end date. Errors are logged to the table "MicroflowErrors". The Microflow storage account is configured in the local.settings.json file.

gorillapower / microflow Goto Github PK

microflow's Introduction