microsoft / abstrakt

Business oriented, config driven DevOps

Abstrakt

Table of contents

• Quick Start
  • Dependencies
  • Running
  • Samples
    • Running the local Helm Microservice Sample
    • Running the sample
    • Remove the install
    • Debugging Templates
• Contributing

Quick Start

If you want to contribute to the project, we've made things easy for you to get everything up and running using devcontainers.

Dependencies

To run the devcontainer, there's a couple things you need to install first on your machine.

• Visual Studio Code
• Remote Development
• Docker

A full devcontainer setup guide can be found here.

Running

With everything installed and running, you can continue.

1. Open this project in a new instance of Visual Studio Code.
2. You may be prompted to Reopen in Container. Click on it.
   • If you do not see the prompt, bring up your command palette (View -> Command Palette...) and type Remote Containers: Open Folder in Container...
3. Wait for Visual Studio Code to reopen and build your container.
4. Once your container has finished building, you can now start developing!
5. Open a new terminal in Visual Studio Code (Terminal -> New Terminal).
6. Type go run main.go.

Samples

You can find sample constellation files in the following location:

/examples/constellation/

Using these files you can test the app binary.

Running the local Helm Microservice Sample

This sample uses the Http Microservice test constellation file located in sample/constellation/http_constellation.yaml.

The sample will install three services with relationships between them:

• A sender
• Another sender
• A receiver

The concept is that you send a message down the chain.

• The first sender takes a message via http, adds value to the message and forwards it to the second sender.
• The second sender repeats this, also adding a unique value before sending on to the receiver.
• The receiver echos back the message
• Each of the sender returns the message back down the chain before the final message pops out the original http request.

Running the sample

Run:

make run-http-demo

This will deploy the demo to the cluster. The templates are output to output/http_sample/Output.

You can test it by:

kubectl wait pod -n default --for condition=ready --timeout=120s --all && kubectl exec -it sender-sender bash
curl sender:8080/api/CallReceiver?message=yourmessage

You should see the response "yourmessage--Chain-sender--Chain-sender1" which proves the message traversed the chain!

Remove the install

You can remove the installed pods and services by calling make http-demo-delete

Debugging Templates

You can debug the templates by outputting them by running make http-demo-template-all

Contributing

Please read CONTRIBUTING.md for details on our process for submitting pull requests, and CODE_OF_CONDUCT.md for details on our code of conduct.

abstrakt's Issues

Wormhole constellations most likely do not need to use GUIDS, use name instead as the ID

Name: WormholeSender_1
    Id: 6209fed9-901f-4ad8-93eb-4e6ef11ad0a4
    Type: WormholeSender

This make for hard to read relationships etc.

Name: Generated Link
    Id: 73c51a4d-ff85-4e5a-aa68-3cd5380db499
    Description: This relationship was generated by a tool
    From: b7a9c21c-eb93-4a93-8433-b7198b300be4
    To: 402cc1c8-b715-427c-af69-c1dd5a8651b0

This is messy and hard to undertand - the following would be better

Name: Generated Link
    Id: 73c51a4d-ff85-4e5a-aa68-3cd5380db499
    Description: This relationship was generated by a tool
    From: WormholeSender_1
    To: WormholeSender_2

Abstrakt needs work to be able to download charts

Currently Abstrakt is only configured to pull charts from a directory. We need to learn how to download charts over https and implement it.

Bug: ineffectual test handling of error in defer

Describe the bug
These tests will not work as they do not modify the t object, they just return.

https://github.com/microsoft/abstrakt/blob/master/internal/chartservice/chartservice_test.go#L29

Security Vulnerability

Looks like semantic-release has a dependency on a package with a known security vulnerability. We need to update semantic-release.

Bug: Duplicate Names and Ids Are Not Caught

Describe the bug
If you have two services with either the same Id or Name, abstrakt does not pick up on this and does not throw an error. This is especially troublesome when it comes to relationships and mapping between different services.

To Reproduce
Steps to reproduce the behavior:

1. Create a new constellation.yaml and make sure to have two services with the same name. Use sample/constellation/http_constellation.yaml to get you started.
2. Run go run main.go visualise -f sample/constellation/http_constellation.yaml | dot -Tpng > result.png and you see all your services mapped out.

Expected behavior
Abstrakt should be smart and check for unique names and IDs before doing anything.

Screenshots
N/A.

Additional context
N/A.

Bug: <Brief description of bug>

Describe the bug
Golang linting fails for compose command.

To Reproduce
run make lint-all

Expected behavior
run make lint-all should complete successfully.

Bug: Need a more reliable way to test contents of tgz files

Describe the bug
Currently equivalent helm tgz files can have different contents due to reordered entries, removal of yaml comments etc. The current solution is to check the size which will be changed in the manipulation of these files.

Bug: Failed Tests Do Not Cause Pipeline to Fail

Describe the bug
If one of the unit tests fail, the pipeline does not fail and reports back everything is ok. Now you can view the unit test results and you will see that a test has failed. However, it appears by looking at the pipeline, everything succeeded.

To Reproduce
Steps to reproduce the behavior:

• Add a breaking change that will case a unit test to fail.
• Notice the pipeline does not fail

Expected behavior
The pipeline should fail so that PRs do not get merged into master.

Screenshots
If you look at the build pipeline here you'll see everything is ok. However, if you look at the test results tab you will see a test has failed.

Bug: Abstrakt exit 0s on errors

Describe the bug
Abstrakt exit 0s on error

To Reproduce
Steps to reproduce the behavior:

➜  /workspace git:(feature/testdata_golden_dataset) ✗ ./abstrakt compose -f sample/constellation/sample_constellation.yaml -m sample/constellation/sample_constellation_maps.yaml -o /tmp
event_hub_sample_event_generatorevent_hub_sample_event_hubevent_hub_sample_event_loggerevent_hub_sample_event_logger1Chart was saved to: /tmpThere was an error saving the chart: directory /tmp/deps/event_hub_sample_event_generator not found#                                                                               
➜  /workspace git:(feature/testdata_golden_dataset) ✗ echo $?
0
➜  /workspace git:(feature/testdata_golden_dataset) ✗ 

Expected behavior
expect an non 0 exit code

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

Enhancement: Artifact of Abstrakt Build Published to GitHub Releases

Description

Once the build pipeline has publish a new build artifact we would like to trigger a release pipeline which takes the build artifact and creates a new release in GitHub.

Work has started on this on branch 160-git-releases and comments on this can be found here: AB#160

Acceptance Criteria

• Release pipeline is setup.
• Release pipeline is triggered when a new publish artifact is available.
• Release pipeline publishes build artifact to GitHub as a new release.
• Git messages are present in release notes.
• Versioning stratergy for GitHub releases
• Cool badges
• Releases use semver to ensure not all CI builds on master/dev go up (fill this out a bit with recommendations - branching for versions etc)

Bug: make lint-all is failing because of kind

Describe the bug
For some reason the build pipeline is failing when running make lint-all because for some reason the pipeline is trying to access kind and it's returning an error.

Script contents:
make lint-all
========================== Starting Command Output ===========================
/bin/bash --noprofile --norc /home/vsts/work/_temp/00f7308d-6ee8-4265-99a2-732e18f5788d.sh
No kind clusters found.
ERROR: unknown flag: --name
Installing golangci-lint
curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b /home/vsts/work/1/s/../gopath/bin v1.23.1 > /dev/null 2>&1
Linting
golangci-lint run ./...
Vetting
go vet ./...
##[error]Bash wrote one or more lines to the standard error stream.
##[error]No kind clusters found.

##[error]ERROR: unknown flag: --name

To Reproduce
Steps to reproduce the behavior:

1. Run build pipeline and see error.

Expected behavior
make lint-all shouldn't access kind at all.

Screenshots
N/A

Additional context
N/A

Enhancement: Abstrakt Version in Version.go Should Reflect Current Semantic Version

Description

Currently abstrakt's version is hardcoded in version.go. If we're going to implement auto versioning referenced in #54 with semantic release then the version number returned by version.go should reflect the current version in master.

Acceptance Criteria

• abstrakt version should return the current version generated by semantic release.
• The version returned matches the current release of abstrakt.
• The version returned should match the corresponding release version (e.g. the version should match the GitHub release version).
• The version should not be hardcoded in version.go and be easily updatable.

Create sample helm charts

Sample charts that can be used to demonstrate real composition of a constellation

A/C

• sample Charts deploy using helm3
• abstrakt tests show creation of composed constellation based chart
• composed chart can be deployed to local kind cluster

Create 'hello world' helm chart based on DAG

Create a basic first Helm Chart output. Use the maps and the DAG to produce the output chart.

A/C

• Uses the samlple DAG and maps files from here: https://github.com/microsoft/wormhole/tree/master/samples/constellation/basic_azure_event_hubs
• Outputs a simple basic working helm chart.
• Test with Helm template to ensure the output is as expected.

Enhancement: Refactor Project and Clean-up Repo

Many contributors have committed code to the project and it's about time we cleaned things up.

1. Put anything docs related into a docs folder in the root of project.
   • There's a usage.md sitting under cmd.
2. Restructuring packages under internal.
   • buildmapservice, chartservice, composeservice, and dagconfigservice can be renamed to something a little nicer.
   • Functions can be renamed to something simpler e.g. LoadMapFromFile can be renamed to LoadMapFile.
3. Functions that return something can be simpler and utilise naked returns.
4. General refactoring.
5. rbac.yaml in the root of the project should be moved to samples.

Helpful video on how we can refactor the project: https://www.youtube.com/watch?v=cmkKxNN7cs4

Bug: Compose Does Not Handle Multiple Relationships to the Same Service

Describe the bug
If a service referenced in multiple relationships' To and/or From, compose will not pick up on them and instead take the first relationship it encounters.

If you look here, Compose loops over the services in the constellation but when it comes to the relationships, it only looks if the service is referenced and returns the first relationship instance it appears in.

To Reproduce
Steps to reproduce the behavior:

1. Edit sample/constellation/http_constellation.yaml to look like the following:

   Name: "HTTP Microservice Sample"
   Id: "d6e4a5e9-696a-4626-ba7a-534d6ff450a5"
   Services:
   - Name: "Sender 1"
     Id: "9e1bcb3d-ff58-41d4-8779-f71e7b8800f8"
     Type: "WormholeSender"
     Properties:
       CHAIN_ARG: "1"
   - Name: "Sender 2"
     Id: "5d43a954-7214-4872-9d8f-758bb4238aee"
     Type: "WormholeSender"
     Properties:
       CHAIN_ARG: "2"
   - Name: "Receiver"
     Id: "aa382e25-8662-4d11-bca9-d9a230c22487"
     Type: "WormholeReceiver"
     Properties: {}
   Relationships:
   - Name: "Sender to Sender Link"
     Id: "211a55bd-5d92-446c-8be8-190f8f0e623e"
     Description: "Link between the first and second senders"
     From: "9e1bcb3d-ff58-41d4-8779-f71e7b8800f8"
     To: "5d43a954-7214-4872-9d8f-758bb4238aee"
     Properties: {}
   - Name: "Sender to Receiver Link"
     Id: "ff5444b5-35b7-4688-98ef-56c2587b314d"
     Description: "Link between the second sender and receiver"
     From: "5d43a954-7214-4872-9d8f-758bb4238aee"
     To: "aa382e25-8662-4d11-bca9-d9a230c22487"
     Properties: {}
   - Name: "Sender to Receiver Link"
     Id: "ff5444b5-35b7-4688-98ef-56c2587b314e"
     Description: "Link between the second sender and receiver"
     From: "9e1bcb3d-ff58-41d4-8779-f71e7b8800f8"
     To: "aa382e25-8662-4d11-bca9-d9a230c22487"
     Properties: {}

2. Put a breakpoint on this line.

3. Put this in your .vscode/launch.json:

   {
           "name": "Launch",
           "type": "go",
           "request": "launch",
           "mode": "auto",
           "program":"${workspaceFolder}/main.go",
           "env": {},
           "dlvLoadConfig": {
               "followPointers": true,
               "maxVariableRecurse": 1,
               "maxStringLen": 5000,
               "maxArrayValues": 64,
               "maxStructFields": -1
           },
           "args": ["compose", "-f", "./sample/constellation/http_constellation.yaml", "-m", "./sample/constellation/http_constellation_maps.yaml", "-o", "./output/http_sample", "-v"]
       }

4. Run abstrakt in debug mode and notice how relationship ff5444b5-35b7-4688-98ef-56c2587b314e is ignored.

Expected behavior
If a service if referenced in multiple relationships, then compose should be able to handle it.

Screenshots
N/A.

Additional context
N/A.

Bug: make lint is downloading the gocli tool every time it's run

Describe the bug
This makes linting locally slower

To Reproduce
Steps to reproduce the behavior:
run make lint-all more than once and observe the delay

Expected behavior
The tool will only be downloaded if it doesn't already exist locally

Enhancement: Constellation maps should allow specifying namespaces

Currently helm charts are generated without a namespace. It would be a useful feature to be able to deploy services to separate namespaces

Add kind dev cluster to .devcontainer

Create a real set of sample charts that can be used by the sample constellation to deploy to a Kind based Kube cluster in a .devcontainer.

A/C

• Main abstract .devcontainer installs kind

Add additional examples

Spike: Research GitHub Action Minutes for Projects Under Microsoft Org

Find out how many minutes available to us.

Update Abstrakt to Go 1.14 and Update Dependencies

Description

Go 1.14 is out and it's time to upgrade!

Acceptance Criteria

• .devcontainer is updated to use Go 1.14.0.
• go.mod is updated to use Go 1.14.0.
• Dependencies are updated to work with Go 1.14.0.
• All tests succeed without fails.

Enhancement: Use `logger.go` Instead of `fmt`

There's many places where fmt is used to print to console. We should use logger.go instead.

Add caching for .devcontainer to speed things up

The devcontainer is slow if you have to rebuild it as it re-downloads all the go and other objects each time. This fix will cache these assets where possible to mounted volumes.

As per this comment on databricks operator Azure/azure-databricks-operator#100 (review)

Add code coverage badge to README.md

Code coverage should be reported in the default readme file along side the build status badge.

Combine `command` and `cmd`

It doesn't make much sense to have both command and cmd folders since they both mean the same thing.

Either move cmd to command or command to cmd.

Visualize tests doesn't test behaviour of command

The visualize test doesn't execute the command. It only tests the internal methods. Unit tests should test the execution of the command.

Bug: Abstrakt produces a chart named "Output"

Describe the bug
Run abstrakt and examine the Chart.yaml. the Name is Output.

The way commands are instantiated causes flaky tests

While doing the PR #42 for issue #26, I ran into a flaky test. This seems to be because of the way we're instantiating commands. We essentially create a command and assign it to a variable, then use that variable in the tests. This seems to make tests prone to side-effects. The following test passes when run on it's own, but fails when it's run as part of the package tests:

func TestComposeCmdVerifyRequiredFlags(t *testing.T) {
	expected := "required flag(s) \"constellationFilePath\", \"mapsFilePath\", \"outputPath\" not set"

	output, err := executeCommand(composeCmd, "")
	if err != nil {
		checkStringContains(t, err.Error(), expected)
	} else {
		t.Errorf("Expecting error: \n %v\nGot:\n %v\n", expected, output)
	}
}

We could attempt to fix this by changing the way we create our commands to use a function that returns a new command, as shown here: https://github.com/gohugoio/hugo/blob/0efb00c2a86ec3f52000a643f26f54bb2a9dfbd6/commands/version.go#L28

Version tests doesn't test the behaviour of the command.

The tests only executes the command, it doesn't test that the version was actually printed.