thomhurst / modularpipelines

Write your pipelines in C# !

ModularPipelines

Define your pipeline in .NET! Strong types, intellisense, parallelisation, and the entire .NET ecosystem at your fingertips.

Documentation

https://thomhurst.github.io/ModularPipelines

Features

• Parallel execution
• Dependency management
• Familiar C# code
• Ability to debug pipelines
• Ability to run pipelines locally, even creating versions for setting up local development
• Strong typing, where different modules/steps can pass data to one another
• Dependency collision detection - Don't worry about accidentally making two modules dependent on each other
• Numerous helpers to do things like: Search files, check checksums, (un)zip folders, download files, install files, execute CLI commands, hash data, and more
• Easy to Skip or Ignore Failures for each individual module by passing in custom logic
• Hooks that can run before and/or after modules
• Pipeline requirements - Validate your requirements are met before executing your pipeline, such as a Linux operating system
• Easy to use File and Folder classes, that can search, read, update, delete and more
• Source controlled pipelines
• Build agent agnostic - Can easily move to a different build system without completely recreating your pipeline
• No need to learn new syntaxes such as YAML defined pipelines
• Strongly typed wrappers around command line tools
• Utilise existing .NET libraries
• Secret obfuscation
• Grouped logging, and the ability to extend sources by adding to the familiar ILogger
• Run based on categories
• Easy to read exceptions
• Dynamic console progress reporting (if the console supports interactive mode)
• Pretty results table

Available Modules

Package	Description	Version
ModularPipelines	Write your pipelines in C#!
ModularPipelines.AmazonWebServices	Helpers for interacting with Amazon Web Services.
ModularPipelines.Azure	Helpers for interacting with Azure.
ModularPipelines.Azure.Pipelines	Helpers for interacting with Azure Pipeline agents.
ModularPipelines.Chocolatey	Helpers for interacting with the Chocolatey CLI.
ModularPipelines.Cmd	Helpers for interacting with the Windows cmd process.
ModularPipelines.Docker	Helpers for interacting with the Docker CLI.
ModularPipelines.DotNet	Helpers for interacting with dotnet CLI.
ModularPipelines.Email	Helpers for sending emails.
ModularPipelines.Ftp	Helpers for downloading and uploading via FTP.
ModularPipelines.Git	Helpers for interacting with git.
ModularPipelines.GitHub	Helpers for interacting with GitHub Actions build agents.
ModularPipelines.Google	Helpers for interacting with the Google gcloud CLI.
ModularPipelines.Helm	Helpers for interacting with Helm CLI.
ModularPipelines.Kubernetes	Helpers for interacting with kubectl CLI.
ModularPipelines.MicrosoftTeams	Helpers for sending Microsoft Teams cards.
ModularPipelines.Node	Helpers for interacting with node / npm CLI.
ModularPipelines.Slack	Helpers for sending Slack cards.
ModularPipelines.TeamCity	Helpers for interacting with TeamCity build agents.
ModularPipelines.Terraform	Helpers for interacting with Terraform CLI.
ModularPipelines.WinGet	Helpers for interacting with the Windows Package Manager.
ModularPipelines.Yarn	Helpers for interacting with Yarn CLI.

Getting Started

If you want to see how to get started, or want to know more about ModularPipelines, read the Documentation here

Console Progress

Results

How does this compare to Cake / Nuke

• Strong types! You have complete control over what data, and what shape of data to pass around from and to different modules
• No external tooling is required. Pipelines are run with a simple dotnet run
• Full dependency injection support for your services
• Similar and familiar setup to frameworks like ASP.NET Core
• Real C# - Whereas frameworks like cake are a scripted form of C#
• Parallelism - Work will run concurrently unless it is dependent on something else
• The style of writing pipelines is very different - Work is organised into separate module classes, keeping code organised and more closely following SRP than having all your work in one main class. This also helps multiple contributors avoid things like merge conflicts

Code Examples

Program.cs - Main method

await PipelineHostBuilder.Create()
    .ConfigureAppConfiguration((context, builder) =>
    {
        builder.AddJsonFile("appsettings.json")
            .AddUserSecrets<Program>()
            .AddEnvironmentVariables();
    })
    .ConfigureServices((context, collection) =>
    {
        collection.Configure<NuGetSettings>(context.Configuration.GetSection("NuGet"));
        collection.Configure<PublishSettings>(context.Configuration.GetSection("Publish"));
        collection.AddSingleton<ISomeService1, SomeService1>();
        collection.AddTransient<ISomeService2, SomeService2>();
    })
    .AddModule<FindNugetPackagesModule>()
    .AddModule<UploadNugetPackagesModule>()
    .ExecutePipelineAsync();

Custom Modules

public class FindNugetPackagesModule : Module<FileInfo>
{
    protected override Task<List<File>?> ExecuteAsync(IPipelineContext context, CancellationToken cancellationToken)
    {
        return context.Git()
            .RootDirectory
            .GetFiles(path => path.Extension is ".nupkg")
            .ToList()
            .AsTask();
    }
}

[DependsOn<FindNugetPackagesModule>]
public class UploadNugetPackagesModule : Module<FileInfo>
{
    private readonly IOptions<NuGetSettings> _nugetSettings;

    public UploadNugetPackagesModule(IOptions<NuGetSettings> nugetSettings)
    {
        _nugetSettings = nugetSettings;
    }

    protected override async Task<CommandResult?> ExecuteAsync(IPipelineContext context, CancellationToken cancellationToken)
    {
        var nugetFiles = await GetModule<FindNugetPackagesModule>();

        return await nugetFiles.Value!
            .SelectAsync(async nugetFile => await context.DotNet().Nuget.Push(new DotNetNugetPushOptions
            {
                Path = nugetFile,
                Source = "https://api.nuget.org/v3/index.json",
                ApiKey = _nugetSettings.Value.ApiKey!,
            }, cancellationToken), cancellationToken: cancellationToken)
            .ProcessOneAtATime();
    }
}

Breaking changes

While I will try to limit breaking changes, there may be some changes within minor versions. These will be noted on release notes.

modularpipelines's Issues

Submodules are run twice per submodule when using SubModule<T>

When you try to run a submodule where the action returns type Task<T>, it runs the action twice simultaneously. The reason is because SubModule<T> inherits from SubModule and both constructors call Task.Run on the action.

You should be able to reproduce the error by running a simple program using the context.Command.ExecuteCommandLineTool action as a submodule. The command will be run twice.

Add yarn package

Log time taken for commands

Mark a module as non parallel

Some modules may not be dependent on others, but also can't run in parallel.
Think installers. Windows for example doesn't like multiple applications being installed at the same time.

So a way to mark modules to not run in parallel with other modules would help this.

Sub Module Times in Results Table

Add GCP Provisioning helpers

These should work similarly to the Azure provisioning helpers. Like a PUT request.
If the resource doesn't exist, create it. If it does, then update it with the latest data.

When running categories, try and build results from history

This would provide for the ability to run sections of pipelines at different stages. And using the history from a different stage later on.

Log time taken for http responses

🐛 Interactive Console Progress - Module appears to finish before actually finished

Document framework fully in wiki section

Log http status codes

Feedback/Ideas for Improvement

Hi,

I've been trying out your project so I can do local test builds that are similar to a pipeline I have in TeamCity. In my opinion, I think the most opportunity for improvement might be with progress logging. With that mind, here are the ideas I have to improve it:

• Submodule progress
  I've been using the GetFiles to retrieve multiple solutions and process them in the same module within a foreach loop. (I have around 20 solutions to build) It'd be nice to see progress/time elapsed for each solution as a "submodule".

• Estimated time remaining
  Currently, the percent goes up 0.1% every 100ms and stops at 70%, but I think it'd more useful to keep track of the time that a module or submodule took previously and use that to show an estimated time remaining and synchronize it with the progress percentage updates. A concept I had in mind was to serialize the estimated times to a json and update it for each run. SQLite might be another option. However, from the perspective of the library, maybe just a convenient way to tap into the Progress Printer would be fine.

Interactive Progress in Console Not Working

Hi,

I'm using Modular Pipelines on windows (Windows Terminal with Powershell 7), and inside Github Runners (Ubuntu Latest).

I cannot get the interactive console progress to work - the console just seems to hang while it does work in the background, and then displays all of the results once complete. Am I missing something?

Figure: Hangs at this point

I noticed all of the logs which are displayed once it unhangs, all have the same timestamp.

Analyser for use of Context within a module's constructor

The context object in a module is initialised after a module object is instantiated.

This means that a user cannot use the Context object within a module's constructor.

Currently, this will throw a runtime exception. But it would be even better to have an analyser raise a compile time error for a faster and clearer feedback loop.

✨ Add Bicep/Azure Resource Support

It would be awesome if I could deploy my azure resources with Bicep using this tool :)

Analyzer and code fix for making modules async

Returning tasks directly can cause stacktraces to become confusing.

We want to clearly know which module has failed and to have clear stacktraces.

We should have an analyzer to detect non-async modules that return a task directly. We should then have a code fix that adds in the async modifier keyword and adds an await to any return statement.

If a method is purely synchronous and doesn't return asynchronous calls, but simply a Task.FromResult, then we probably don't need to flag this as this method would throw exceptions synchronously and normally.

Add AWS Provisioning helpers

These should work similarly to the Azure provisioning helpers. Like a PUT request.
If the resource doesn't exist, create it. If it does, then update it with the latest data.

webapi project support？

aspnet.core project, if you call through the api, web interface display, such behaviour is supported?

Refactor StartAsync

The Module StartAsync method is doing a lot of things.
It might be sensible to split logic out into separate classes.

E.g.
ModuleCancellationHandler
ModuleSkipHandler
ModuleHistoryHandler
ModuleErrorHandler
ModuleSuccessHandler
ModuleOnEndHandler

Retries

Transient errors can be caused by things such as network issues. This can be frustrating when it then fails a pipeline. To avoid this, we could allow the user to plugin some retry rules.

• provide an overrideable property within a module that allows the user to plugin a retry rule (use Polly?)
• expose a property on PipelineOption where a user can set a default retry count for modules
• set a default basic retry policy that will retry for the amount of times defined in PipelineOption, waiting between each try?

Introduce Skip Reason

Try to automatically produce pipeline yaml if doesn't exist

Add xmldocs for all public interfaces + classes

Node full Options objects

Node implementation is currently very basic.

Let's have more options with more properties, similar to docker etc