lyraproj / lyra Goto Github PK

View Code? Open in Web Editor NEW

213.0 16.0 37.0 4.47 MB

Open Source Workflow Engine for Cloud Native Infrastructure

Home Page: https://lyraproj.github.io

License: Apache License 2.0

Makefile 6.55% Go 81.53% Puppet 10.04% LLVM 1.08% Dockerfile 0.79%

infrastructure provisioning aws gcp azure

lyra's Introduction

lyra's People

Contributors

Stargazers

Watchers

lyra's Issues

Preview resources

Is your feature request related to a problem? Please describe.
Before applying a resource, I would like the ability to preview what will be created.

so that I don't spend £ I wasn't intending to on resources
so that I see how my instantiation data affect the execution of the workflow
so that I gain confidence in how Lyra works
so that I can run the output of the preview through a testing framework to ensure it's been configured correctly
so that I can get sign-off from other team members prior to provisioning

Describe the solution you'd like
Some sort of lyra apply pre-step or a separate lyra preview function that shows me resources that will be created as part of the Lyra apply.

Additional context
Ideally, this should both be machine readable and human readable. It would be great to leverage an rspec-like testing framework to do assertion testing on the "catalog" or resulting resources that would be created.

Leverage Terraform resource providers

Is your feature request related to a problem? Please describe.
I would really like a LOT more content for Lyra. Doing incremental content releases is good, but to prove out what Lyra can do, it needs a lot more content to be useful.

Describe the solution you'd like
It would be really great if Lyra could leverage Terraform's resource providers. They already have a ton of content that would be awesome if we could use without re-inventing the wheel.

Additional context
Prior art in this space:

Externalise user-facing strings (where possible)

Is your feature request related to a problem? Please describe.
Developing contextual guidance text (help, errors, etc) is difficult when the bits of text are spread across many source files. Hard-coded messages makes i18n and l10n impossible.

Describe the solution you'd like
Extract user-facing strings into a central location.

Logging and status updates

As a Lyra user, I want more verbose logging and status updates so that I can see what is happening when I apply my Workflow both from the CLI as well as from Kubernetes.

Kubernetes CRD facade

Is your feature request related to a problem? Please describe.
I want to execute a Lyra Workflow as a Kubernetes CRD so that I can leverage all my native Kubernetes tooling (i.e. kubectl, helm) to also deploy infrastructure and workflows.

Describe the solution you'd like
I can package up my workflow -> Submit the package to Lyra which registers it as a Kubernetes CRD -> Deploy/update/delete custom resource with kubectl/helm etc.

The Lyra controller should leverage the same library as the CLI and listen for Workflow CRDs in Kubernetes. It should provide the following operations:

Deploy
Upgrade
Delete
Get
List

Describe alternatives you've considered
helm plugin

Additional context
User experience should have the following steps:

Author Workflow
Package Workflow
Register Workflow with Kubernetes
Deploy Workflow with Kubernetes tooling (i.e. kubectl)

Produce consumable Lyra artefact

Produce consumable Lyra artefact.

USER STORIES:

I want to easily install and get started with Lyra so that I can quickly realize the value of Lyra.
I want to execute Lyra Workflows with Lyra CLI so that I can see value and do basic operations without faffing around with K8s.

TypeScript Support

Is your feature request related to a problem? Please describe.
When I am authoring a Workflow manifest, I want to use TypeScript to describe the resources I'm creating so that I can leverage my existing knowledge and skillset.

Describe the solution you'd like
I'd like to be able to describe a set of declarative resources or imperative actions with TypeScript.

Describe alternatives you've considered
None

external_id is not picked up

Describe the bug
Thomas, Scott and I had previously discussed that a resource definition could contain a property external_id which would result in the identity store being bypassed.

For example

  resource keypair {
    input  => ($tags),
    output => ($key_fingerprint),
    external_id => 'lyra-test-kp',
    something_else => 'gets_ignored'
  }{
....
  }

To Reproduce
Steps to reproduce the behavior:

Write a workflow like the keypair definition above
Run command lyra apply <workflow> --debug
Lyra attempts to re-create the resource, rather than respecting the external_id

Expected behavior
Lyra should identity that the external_id has been set and not re-create the resource.

Logs
I added some diagnostics to the workflow engine and uses go mod's replace to pick up my local copy.

The change to wfe

➜  wfe git:(master) ✗ git --no-pager diff -- wfe/resource.go
diff --git a/wfe/resource.go b/wfe/resource.go
index 7388eca..05e7441 100644
--- a/wfe/resource.go
+++ b/wfe/resource.go
@@ -1,6 +1,8 @@
 package wfe

 import (
+	"bytes"
+	"github.com/hashicorp/go-hclog"
 	"github.com/lyraproj/puppet-evaluator/eval"
 	"github.com/lyraproj/puppet-evaluator/types"
 	"github.com/lyraproj/servicesdk/serviceapi"
@@ -25,6 +27,7 @@ func (r *resource) HandlerId() eval.TypedName {
 }

 func (r *resource) ExtId() eval.Value {
+	hclog.Default().Debug("Returning external_id", "external_id", r.extId)
 	return r.extId
 }

@@ -36,7 +39,12 @@ func Resource(def serviceapi.Definition) api.Activity {

 func (r *resource) Init(d serviceapi.Definition) {
 	r.Activity.Init(d)
+	b := bytes.NewBufferString(`"`)
+	d.ToString(b, eval.PRETTY_EXPANDED, nil)
+
+	hclog.Default().Debug("initting", "d", b.String(), "\nd.Properties()", d.Properties())
 	if eid, ok := service.GetOptionalProperty(d, `external_id`, types.DefaultStringType()); ok {
+		hclog.Default().Debug("setting external_id", "external_id", eid)
 		r.extId = eid
 	}
 	r.typ = service.GetProperty(d, `resource_type`, types.NewTypeType(types.DefaultObjectType())).(eval.ObjectType)

The output

2019-01-23T18:36:02.895Z [DEBUG] lyra: applying: activityID=attach
2019-01-23T18:36:02.897Z [DEBUG] lyra: initting: d=""Service::Definition(
  'identifier' => TypedName(
    'namespace' => 'definition',
    'name' => 'attach::keypair'
  ),
  'serviceId' => TypedName(
    'namespace' => 'service',
    'name' => 'Plugins::AttachPp'
  ),
  'properties' => {
    'input' => [
      Parameter(
        'name' => 'tags',
        'type' => Any
      )],
    'output' => [
      Parameter(
        'name' => 'key_fingerprint',
        'type' => Any
      )],
    'resource_type' => Object[{
      name => 'Aws::KeyPair',
      attributes => {
        'public_key_material' => String,
        'key_name' => String,
        'key_fingerprint' => {
          'type' => String,
          'value' => ''
        }
      }
    }],
    'style' => 'resource'
  }
)"
d.Properties()="{'input' => [Parameter('name' => 'tags', 'type' => Any)], 'output' => [Parameter('name' => 'key_fingerprint', 'type' => Any)], 'resource_type' => Aws::KeyPair, 'style' => 'resource'}"
2019-01-23T18:36:02.897Z [DEBUG] lyra: Returning external_id: external_id=<nil>

Note that d.Properties() does not contain external_id and the external_id is nil. We could expect to see something like

...... [DEBUG] lyra: Returning external_id: external_id=lyra-test-kp

Environment (please complete the following information):

OS: [e.g. macOS 10.14.1]

➜  lyra git:(master) ✗ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.14.2
BuildVersion:	18C54

Version/Build number [e.g. 0.0.1, f61edcc]
https://github.com/lyraproj/lyra/tree/65575a9c8393002c74fde40eb571b308865db2ee
https://github.com/lyraproj/wfe/tree/32d74d38984cea616a6e7cc0586af39643cf83dc
Cloud target [e.g. AWS, GCP]
AWS

Additional context
Relates to #52

Create a Helm plugin for Lyra

As a Helm user, I want Lyra to be available to me so that I can create Workflows and then use them as part of my Helm deployments. Helm plugins make it easy to install and run Lyra either locally or in Kubernetes.

Workflow versioning

When I am updating a Lyra workflow (or set of Lyra workflows), I want a way to version these artifacts so that I can:

Ensure that the right version is being applied to my environment
Upgrade (and test) one version to another version

Ideally, this would happen on the "package" artifact (#41).

Produce Lyra builds (via Travis)

Enable and configure Travis to build (and optionally store) builds of Lyra for Linux and macOS

Build takes 20 minutes on travis

Example link https://travis-ci.org/lyraproj/lyra/builds/488962500?utm_source=github_status&utm_medium=notification

make takes 1140.27s (i.e. just shy of 20 minutes)

Probably the time is spent downloading packages.

Explore caching in travis? https://docs.travis-ci.com/user/caching/

YAML support

Is your feature request related to a problem? Please describe.
When I am authoring a Workflow manifest, I want to use YAML to describe the resources I'm creating so that I can leverage my existing knowledge and skillset.

Describe the solution you'd like
I'd like to be able to describe a set of declarative resources with YAML.

Describe alternatives you've considered
None.

Additional context
YAML is the defacto standard in the Kubernetes community. There are lots of users who are already familiar with YAML. It does not require someone to learn Puppet.

Of course, there are a lot of things that YAML is not good at (e.g. imperative actions). I propose that we exclude any procedural or imperative actions from YAML support.

Here's an example:

lyra::aws::vpc:  
    type: resource  
    input:    
        region:
            type: String
            lookup: aws.region    
        tags:
            type: Hash[String,String]
            lookup: aws.tags  
    output: vpc_id  
    body:    
            ensure: present    
            region: !input region    
            cidr_block: 192.168.0.0/16    
            tags: !input tags    
            enable_dns_hostnames: true    
            enable_dns_support  : true

Update the workflow-semantics.md doc.

The docs/workflow-semantics.md document is out of date. The following topics must be updated:

CRUD. We have introduced a better separation of concern by moving CRUD actions to a specific layer that deals with specifics regarding resource activities. The workflow is no longer aware of CRUD.
Deletion. Text is no longer relevant due to the introduction of a garbage collector.
Guards. We no longer have special guard activities. A guard is a boolean filter on input variables.

Turn /docs/ into GH Pages website

Implement the "attach" workflow using terraform content

Prove the concept that Lyra can integrate with Terraform content by creating a version of the "attach" workflow and a corresponding provider that use the Terraform AWS provider to make the changes required in the real world.

Apply fails on duplicate key pair

Describe the bug
When running apply, if keypair has previously been created (either through Lyra or not), apply fails with the following:

2019-01-18T10:01:46.732-0800 [DEBUG] lyra.lyra: 2019-01-18T10:01:46.732-0800 [DEBUG] lyra: Creating KeyPair: desired="&{ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQCX363gh/q6DGSL963/LlYcILkYKtEjrq5Ze4gr1BJdY0pqLMIKFt/VMJ5UTyx85N4Chjb/jEQhZzlWGC1SMsXOQ+EnY72fYrpOV0wZ4VraxZAz3WASikEglHJYALTQtsL8RGPxlBhIv0HpgevBkDlHvR+QGFaEQCaUhXCWDtLWYw== nyx-test-keypair-nopassword lyra-test-kp }"
2019-01-18T10:01:47.056-0800 [DEBUG] lyra.lyra: 2019-01-18T10:01:47.056-0800 [DEBUG] lyra: Error creating KeyPair: error="InvalidKeyPair.Duplicate: The keypair 'lyra-test-kp' already exists.
2019-01-18T10:01:47.056-0800 [DEBUG] lyra.lyra: 	status code: 400, request id: 701e0653-efbf-4468-b36b-0ff0d7e9a862"
panic: invocation of Aws::KeyPairHandler create failed: EVAL_GO_FUNCTION_ERROR Go function Create returned error 'InvalidKeyPair.Duplicate: The keypair 'lyra-test-kp' already exists.
	status code: 400, request id: 701e0653-efbf-4468-b36b-0ff0d7e9a862'

To Reproduce
Steps to reproduce the behavior:

Write a workflow like

resource keypair {
    input  => ($tags),
    output => ($key_fingerprint)
  }{
    public_key_material => "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQCX363gh/q6DGSL963/LlYcILkYKtEjrq5Ze4gr1BJdY0pqLMIKFt/VMJ5UTyx85N4Chjb/jEQhZzlWGC1SMsXOQ+EnY72fYrpOV0wZ4VraxZAz3WASikEglHJYALTQtsL8RGPxlBhIv0HpgevBkDlHvR+QGFaEQCaUhXCWDtLWYw== nyx-test-keypair-nopassword",
    key_name => "lyra-test-kp"
  }

Run command lyra apply attach --debug
Delete identity.db
Re-run command lyra apply attach --debug
See error

Expected behavior
I expect Lyra to record keypair into identity.db and continue forward with execution given the resource already exists as specified.

Logs
If applicable, add the full terminal output (try adding the --debug flag for more verbosity) to help explain your problem.

Environment (please complete the following information):

OS: [e.g. macOS 10.14.1]
Version/Build number [e.g. 0.0.1, f61edcc]
Cloud target [e.g. AWS, GCP]

Additional context
Add any other context about the problem here.

Missing attributes in TF aws_security_group_rule resource

Describe the bug
There are a couple missing attributes in the aws_security_group_rule resource generated from TF. Specifically, the from_port, to_port, and cidr_blocks attributes are not present in the type. From our cmd/goplugin-tf-aws/generated/types.go:

type Aws_security_group_rule struct {
     Aws_security_group_rule_id *string
     Resource_type string
     Description *string
     Security_group_id string
     Source_security_group_id *string
     Self *bool
     Protocol string
}

To Reproduce
Steps to reproduce the behavior:

Write a workflow like:

resource aws_security_group_rule {
    input => ($aws_security_group_id),
  }{
      resource_type => "ingress",
      from_port => 0,
      to_port => 65535,
      protocol => "tcp",
      security_group_id => $aws_security_group_id,
  }

Run command lyra apply sg --debug
See error

2019-01-29T14:54:29.401-0800 [DEBUG] lyra.lyra: panic: AwsTerraform::Aws_security_group_rule: expects one of:
2019-01-29T14:54:29.401-0800 [DEBUG] lyra.lyra:   (String 1, String 2, String 3, Optional[String] 4, Optional[String] 5, Optional[String] 6, Optional[Boolean] 7)
2019-01-29T14:54:29.401-0800 [DEBUG] lyra.lyra:     rejected:expects between 3 and 7 arguments, got 1
2019-01-29T14:54:29.401-0800 [DEBUG] lyra.lyra:   (Struct[{'aws_security_group_rule_id' => Optional[String], 'resource_type' => String, 'description' => Optional[String], 'security_group_id' => String, 'source_security_group_id' => Optional[String], 'self' => Optional[Boolean], 'protocol' => String}] 1)
2019-01-29T14:54:29.401-0800 [DEBUG] lyra.lyra:     rejected: parameter 1 unrecognized key 'from_port'
2019-01-29T14:54:29.401-0800 [DEBUG] lyra.lyra:     rejected: parameter 1 unrecognized key 'to_port' [recovered]

Expected behavior
Ability to specify from_port, to_port, and cidr_blocks attributes in the resource declaration.

Need a LICENSE file

...for each of the repos in the lyraproj organization

Provider configuration

We need a better way to specify or override provider configuration. Right now, we pull the ~/.aws/config for AWS configuration, but that should be the default, not the main way.

time.Time should map to Puppet Timestamp, but does not.

Describe the bug
The golang time.Time{} type should map to Puppet Timestamp but does not. A Time type is added to the current TypeSet.

To Reproduce
Steps to reproduce the behavior:
(refer to the test mentioned below)

Add a field to a managed type of type time.Time (native go Time)
Register it
Invoke the Metadata endpoint
Output the typeset with eval.PRETTY_EXPANDED

Expected behavior
I expected the time.Time field to be mapped to Timestamp in the outputted typeset.

Logs
I added 2 commits to https://github.com/markfuller/lyra.

The First commit (markfuller@c70913b) adds a working godocs example which verifies the TypeSet for the existing example resources.

The second commit (markfuller@a4f2d2c) adds a time.Time field to the Person object, and the expected typeset change to the ExampleServer test.

The test passes in the first commit but fails in the second commit (output mentioned below).

Environment (please complete the following information):

OS: [e.g. macOS 10.14.1]

 lyra git:(Time) ✗ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.14.2
BuildVersion:	18C54

Version/Build number [e.g. 0.0.1, f61edcc]
65575a9 (master as at Jan-23-2019)
Cloud target [e.g. AWS, GCP]
N/A (example)

Additional context
Thomas indicated that he expected time.Time to be mapped to Timestamp (in Puppet).

Output of failing test (https://gist.github.com/markfuller/8c10bda9df34fce200b3a5e8afa253f4) shows that an type called Time is outputted as part of the typeset (https://gist.github.com/markfuller/8c10bda9df34fce200b3a5e8afa253f4#file-gistfile1-txt-L101)

The type where I first came across this is a 3rd party (AWS SDK) type, so I cannot directly annotate it with struct tags.

Implement Garbage Collection of external resources

A garbage collector is needed that is capable of deleting external resources that are no longer in use. Lyra uses the Identity service to keep track of such services. This ticket entails the following:

Extend the data stored by the Identity service to also contain a timestamp for when the mapping was made and a GC era.
Add a search method to the Identity service that can return a set of mappings based on a partial hierarchical internal identifier. This to enable deletion of all identifiers given a certain root such as a Workflow or a resource that acts as a container for contained resources.
Implement a simple mark/sweep type of garbage collection such that an apply of a workflow does a 'mark' and a sweep, called explicitly, deletes all resources that were not touched by the 'mark'.
Implement deletion of everything related to a partial internal identifier (i.e. a workflow or a resource container).

Share the Identity store in the cloud

The Identity service is currently embedded in Lyra and accessed as a go-plugin. As such, its storage is confined to the local disk on the machine where lyra executes. This means that an infrastructure can only be reliably managed from the machine where it was first created. A way to run the Identity service in the cloud must be provided to overcome this limitation.

Remove "Puppet DSL" support from Lyra

Is your feature request related to a problem? Please describe.
Having a chat with @scottyw this morning led to the idea that it's time to have a conversation about dropping Puppet DSL support in Lyra. Here are some reasons, in no particular order:

What we're calling "Puppet DSL" bears only passing resemblance to what current users of Puppet actually interact with; it's increasingly disingenuous to conflate the two; they share some syntax conventions, but conceptually and semantically, only passing resemblance.
Now that we're able to author workflows in YAML, a major use case for Puppet (declarative workflow actions) is solved elsewhere.
New users coming from other cloud native tooling will not expect or be familiar with Puppet, so choosing something more widespread (TypeScript, Go) will ease the learning curve.

Describe the solution you'd like
Remove Puppet as a supported language for authoring workflows and invest in TypeScript as the preferred method for authoring the imperative bits of workflows and their actions.

Implement CLI for deploying controller

Supports #37

Lyra needs a way to deploy the controller from the CLI. This could be lyra init similar to Helm or could be something else.

Need a CONTRIBUTING.md

Need a contribution model document.

CRD delete (and other associated lifecycle events) should trigger Lyra delete

Try integrating additional Terraform providers

The Terraform bridge prototype uses only the AWS provider.

Other providers can be found here: https://github.com/terraform-providers/

Initially we'll look at:

Typo in GitHub project description (Infrastructure is misspelled)

Open Source Workflow Engine for Cloud Native Infastructure

should be

Open Source Workflow Engine for Cloud Native Infrastructure

Improve workflow activity terminology.

The current activity terminology is confusing. What's the difference between an "action" and a "stateless"? The current meaning of the terms are:

"action" an activity that implements an interface with several functions.
"stateless" an activity that acts as a function.

For "action", the only interface that is currently possible to implement is the "CRUD" interface used when operating on a resource state.

The term "action" is bad because it doesn't really describe what it currently means.
The term "stateless", added to be the opposite of "stateful" (which is a resource) isn't very clear either.

A couple of different suggestions have been made. Listed here in no particular order.

Let "action" be CRUD, since that's what it really is. An implementation of a CRUD interface.
Rename "resource" to "stateful" or perhaps simply "state" so that it matches "stateless".
Rename "action" to "resource_provider" and then "stateless" to "action" (a.k.a. back to where we started).
Rename "action" to "state_handler" and then "stateless" to "action".

We should probably avoid terms that are non-descriptive, have Puppet baggage, or are heavily overloaded.

Provider generation for aws and google is switched

Describe the bug
Provider generation for GCP and Azure are reversed.

https://github.com/lyraproj/lyra/blob/master/cmd/tf-gen/main.go#L20:L24

AWS IAM resources

Is your feature request related to a problem? Please describe.
When I create new Kubernetes applications that require some AWS permissions (e.g. for logging for instance), I need to create or apply a new IAM role. I want to do that with a Lyra Workflow.

Describe the solution you'd like

I can author a Workflow with declarative IAM role resources
I can package that workflow up and reference it from my K8s Workflow CRD

Describe alternatives you've considered
None

Additional context
Something like this:

workflow "aws_iam_role" {
  typespace => 'aws',
} {
  resource IAM_role {
    assume_role_policy = <<EOF
     {
       "Version": "2012-10-17",
       "Statement": [
         {
           "Action": "sts:AssumeRole",
           "Principal": {
             "Service": "ec2.amazonaws.com"
           },
        "Effect": "Allow",
        "Sid": ""
         }
       ]
     }
     EOF
  }

  tags = {
      tag-key = "tag-value"
  }
}

Determine contribution strategy

Determine contribution strategy and plan and add those details to the repo.

Remove logo from most help messages

Including the Lyra logo figlet in every help message is a bit much (takes up many lines, doesn't add information etc). Let's limit it to the root level lyra --help and lyra --version only.

Reword `lyra apply` help text

Describe the bug
The usage messages don't match reality and talk about specifying a manifest file, where one actually specifies a workflow to apply. The text needs to be clarified based on actual behaviour.

To Reproduce
Steps to reproduce the behavior:
2. Run command lyra apply manifest.pp
3. Observe the error and ʕノ•ᴥ•ʔノ︵ ┻━┻

Logs

$ lyra apply -h

Usage:
  lyra apply [puppet manifest] [flags]

[ok, got it, wants a file name]

$ lyra apply plugins/example.pp
[snip]
panic: Name 'plugins/example.pp contains invalid characters. 
Must start with letter and only contain letters, digits, and underscore'

[nope, doesn't want a file name, argh!]

Environment (please complete the following information):

Applies to commit 69a56d1

A slice of pointers in go is incorrectly mapped when generating a TypeSet

Describe the bug
Arrays of pointers

To Reproduce
Steps to reproduce the behavior:

Run this test https://github.com/markfuller/lyra/blob/ec30ae60245be637e4ce83c3370fe6fc1828ce1f/cmd/goplugin-example/example/typeset_test.go

Expected behavior
I expected a []*Baz to be mapped to Array[Optional[Baz]].

Expected:
https://gist.github.com/markfuller/e1078b57a1079d6cf3bc5ee755641545#file-gistfile1-txt-L52-L53

Actual:
https://gist.github.com/markfuller/e1078b57a1079d6cf3bc5ee755641545#file-gistfile1-txt-L25-L26

Note that it appears the output is different depending on whether the type is used elsewhere (in the case of Baz)

Logs
Test output https://gist.github.com/markfuller/e1078b57a1079d6cf3bc5ee755641545

Environment (please complete the following information):

OS: [e.g. macOS 10.14.1]

ProductName:	Mac OS X
ProductVersion:	10.14.2
BuildVersion:	18C54

Version/Build number [e.g. 0.0.1, f61edcc]
65575a9 (master as at Jan-23-2019)
Cloud target [e.g. AWS, GCP]
N/A (example)

Additional context
Found with 3rd party (AWS SDK) library, so cannot add inline annotations.

Implement controller k8s deployment as Helm chart

Supports #37.

The Lyra controller should be deployable as a Kubernetes deployment. This means figuring out what deployments, statefulsets, pods, secrets etc are required to deploy the Lyra controller and how to package it together. The initial proposal here is to package it as a Helm chart.

Destroy

Is your feature request related to a problem? Please describe.
I want the ability to destroy infrastructure that I have created.

Describe the solution you'd like
I want the ability to destroy infrastructure that I have created:

CLI - lyra destroy or some likeness that destroys previously created resources
Kubernetes - When the CRD object associated with the workflow is deleted, it should trigger the same destroy workflow

Describe alternatives you've considered
None.

Additional context
Couple other behavioral suggestions:

Destroy should happen in a sensible order that respects implicit dependencies (e.g. VPC and subnet)
Resources that have been previously created, but subsequently removed from the manifest should be destroyed via garbage collection or similar mechanism

Support update in the Terraform bridge

The Terraform bridge prototype does not yet support Update, only Create/Read/Delete.

We need to check out what the options are there and preferably implement support in the bridge.

Prototype UX for dual interfaces: CLI & K8s Object Spec

We will support multiple interfaces for using Lyra, starting with a CLI version and a K8s object spec.

We should do some investigation on how these interfaces will work with each other. Additionally, we may want the CLI options for deploying the controller to Kubernetes.

Supports #37.

Loadpath should support subdirs

Today, loadpath only looks in plugins/ for Workflows. Lyra should support subdirectories so we can organize the plugin directory in some way.

2019-01-25T13:29:47.584-0800 [DEBUG] lyra.loader: reading plugins from filesystem: dir=./plugins
...
[error] Unable to find definition for activity attachterraformyaml

Alongside our package format (#41), Lyra should load Workflows from subdirectories or well-known locations.

Remove code generation from the Terraform bridge

The Terraform bridge uses code generation to create a Go struct and handler for every supported AWS type, hundreds in total. At runtime our provider interface deals with meta-level descriptions of types and functions which are mapped to those concrete types which are in turn passed to the provider CRUD functions. To integrate with Terraform content, our provider uses one of many almost-identical bridging provider implementations. These convert the concrete types back into a meta-level descriptions (i.e. *terraform.ResourceConfig) before calling the TF provider.

In principle we can instead generate all Lyra type information directly from the TF schema at runtime, without the need to use a code generator to produce intermediate concrete types. Similarly a single provider implementation could handle all possible TF types.

This issue captures design and implementation of this approach.

Does not seem possible to add AWS tags via lookup data when run as a controller

Describe the bug
When running Lyra as a controller it does not seem possible to add AWS tags via lookup data:

did this:
kind: Workflow
metadata:
  name: dbt-workflow
spec:
  workflowName: "dbt"
  refreshTime: 10
  data:
    aws:
      tags:
        created_by: lyra
        department: engineering
        project: incubator
        lifetime: 1h
        Name: dimitri-lyra-test
Now the output is:
E0125 12:14:07.962996 1865 reflector.go:134] pkg/mod/k8s.io/[email protected]+incompatible/tools/cache/reflector.go:95: Failed to list *v1alpha1.Workflow: v1alpha1.WorkflowList.Items: []v1alpha1.Workflow: v1alpha1.Workflow.Spec: v1alpha1.WorkflowSpec.Data: ReadString: expects " or n, but found {, error found in #10 byte of ...|":{"aws":{"tags":{"N|..., bigger context ...|a-11e9-89fa-42010a840124"},"spec":{"data":{"aws":{"tags":{"Name":"dimitri-lyra-test","created_by":"l|...
and this repeats indefinitely

Lyra works as expected when run directly with CLI if the data is hard-wired into the manifest.

To Reproduce
Run Lyra in controller mode and create the above Workflow resource.

Expected behavior
Lyra runs the workflow the same in both controller and CLI mode.

Additional context
Found by Dimitri

VPC IsDefault not working

Describe the bug
Creating a VPC with isDefault = true creates a VPC with isDefault = false.

Workflow

workflow attach2 {
  typespace => 'aws',
  input => (
    Hash[String,String] $tags = lookup('aws.tags'),
  ),
  output => (
    String $vpc_id,
  )
} {
  resource vpc {
    input  => ($tags),
    output => ($vpc_id)
  }{
    amazon_provided_ipv6_cidr_block => false,
    cidr_block => '192.168.0.0/16',
    enable_dns_hostnames => false,
    enable_dns_support => false,
    is_default => true,
    state => 'available',
    tags => $tags,
    instance_tenancy => 'default',
  }
}

To Reproduce
Steps to reproduce the behavior:

Write a workflow like above
Run command lyra apply attach --debug
See error

2019-01-18T10:35:22.714-0800 [DEBUG] lyra.lyra: 2019-01-18T10:35:22.714-0800 [DEBUG] lyra: Creating VPC: desired="(*resource.Vpc)(0xc0002f2c00)({
2019-01-18T10:35:22.714-0800 [DEBUG] lyra.lyra:  AmazonProvidedIpv6CidrBlock: (bool) false,
2019-01-18T10:35:22.714-0800 [DEBUG] lyra.lyra:  CidrBlock: (string) (len=14) "192.168.0.0/16",
2019-01-18T10:35:22.714-0800 [DEBUG] lyra.lyra:  InstanceTenancy: (*string)(0xc000211380)((len=7) "default"),
2019-01-18T10:35:22.714-0800 [DEBUG] lyra.lyra:  EnableDnsHostnames: (bool) false,
2019-01-18T10:35:22.714-0800 [DEBUG] lyra.lyra:  EnableDnsSupport: (bool) false,
2019-01-18T10:35:22.714-0800 [DEBUG] lyra.lyra:  Tags: (map[string]string) (len=4) {
2019-01-18T10:35:22.714-0800 [DEBUG] lyra.lyra:   (string) (len=8) "lifetime": (string) (len=2) "1h",
2019-01-18T10:35:22.714-0800 [DEBUG] lyra.lyra:   (string) (len=10) "created_by": (string) (len=4) "lyra",
2019-01-18T10:35:22.714-0800 [DEBUG] lyra.lyra:   (string) (len=10) "department": (string) (len=11) "engineering",
2019-01-18T10:35:22.714-0800 [DEBUG] lyra.lyra:   (string) (len=7) "project": (string) (len=9) "incubator"
2019-01-18T10:35:22.714-0800 [DEBUG] lyra.lyra:  },
2019-01-18T10:35:22.714-0800 [DEBUG] lyra.lyra:  VpcId: (*string)(<nil>),
2019-01-18T10:35:22.714-0800 [DEBUG] lyra.lyra:  **IsDefault: (bool) true,**
2019-01-18T10:35:22.714-0800 [DEBUG] lyra.lyra:  State: (string) (len=9) "available",
2019-01-18T10:35:22.714-0800 [DEBUG] lyra.lyra:  DhcpOptionsId: (*string)(<nil>)
2019-01-18T10:35:22.714-0800 [DEBUG] lyra.lyra: })
2019-01-18T10:35:22.714-0800 [DEBUG] lyra.lyra: "
2019-01-18T10:35:23.437-0800 [DEBUG] lyra.lyra: 2019-01-18T10:35:23.437-0800 [DEBUG] lyra: Created VPC: actual="(*resource.Vpc)(0xc000cdfe00)({
2019-01-18T10:35:23.437-0800 [DEBUG] lyra.lyra:  AmazonProvidedIpv6CidrBlock: (bool) false,
2019-01-18T10:35:23.437-0800 [DEBUG] lyra.lyra:  CidrBlock: (string) (len=14) "192.168.0.0/16",
2019-01-18T10:35:23.437-0800 [DEBUG] lyra.lyra:  InstanceTenancy: (*string)(0xc000273b28)((len=7) "default"),
2019-01-18T10:35:23.437-0800 [DEBUG] lyra.lyra:  EnableDnsHostnames: (bool) false,
2019-01-18T10:35:23.437-0800 [DEBUG] lyra.lyra:  EnableDnsSupport: (bool) false,
2019-01-18T10:35:23.437-0800 [DEBUG] lyra.lyra:  Tags: (map[string]string) {
2019-01-18T10:35:23.437-0800 [DEBUG] lyra.lyra:  },
2019-01-18T10:35:23.437-0800 [DEBUG] lyra.lyra:  VpcId: (*string)(0xc000272e08)((len=21) "vpc-0ebf75e8ad02d5995"),
2019-01-18T10:35:23.437-0800 [DEBUG] lyra.lyra:  **IsDefault: (bool) false,**
2019-01-18T10:35:23.437-0800 [DEBUG] lyra.lyra:  State: (string) (len=7) "pending",
2019-01-18T10:35:23.437-0800 [DEBUG] lyra.lyra:  DhcpOptionsId: (*string)(0xc0002737c8)((len=22) "dopt-01f58b09b62dbb9cb")
2019-01-18T10:35:23.437-0800 [DEBUG] lyra.lyra: })
2019-01-18T10:35:23.437-0800 [DEBUG] lyra.lyra: " externalID=vpc-0ebf75e8ad02d5995
2019-01-18T10:35:23.440-0800 [DEBUG] lyra: Apply done: result="{'vpc_id' => 'vpc-0ebf75e8ad02d5995'}"

Expected behavior
VPC expected to have isDefault = true.

Logs
If applicable, add the full terminal output (try adding the --debug flag for more verbosity) to help explain your problem.

Attached above.

Environment (please complete the following information):

OS: [e.g. macOS 10.14.1]
Version/Build number [e.g. 0.0.1, f61edcc]
Cloud target [e.g. AWS, GCP]

Additional context
Add any other context about the problem here.

Support TypeLists in the tf-gen to enable integration of the Kubernetes provider

Is your feature request related to a problem? Please describe.
The TF Kubernetes provider uses a nested schema with TypeLists, which are not currently supported in tf-gen. While other providers remain partly functional without this support, the Kube provider does not.

Describe the solution you'd like
Support for nested schemas using TypeList.

Reference Workflow package by URL

As a Lyra workflow producer, I want to be able to store my Workflows in a GitHub repo and reference them from Lyra so I can have a single source of content.

Add a docs/workflow-yaml.md document describing the YAML Workflow Syntax

At present, we have a workflow-semantics.md document that describes the generic semantics of a workflow and a workflow-puppet-dsl.md that describes how to write a workflow manifest using the Puppet DSL but there's no document describing how to write a YAML manifest.

Package Format for Workflows

Is your feature request related to a problem? Please describe.

I want to package one or more Workflows up and submit it to Lyra to be consumed through Kubernetes so that I can produce consumable and reusable Workflows. Inspired by Helm for UX.

Describe the solution you'd like

Proposed package architecture

metadata.yaml which includes name of the package, description, version information, dependencies(?)
data.yaml for external hiera data
workflows directory containing all Workflows to be packaged up together

Outputs an artifact in some archive format.

Consuming packages

Workflow packages can be referenced for consumption in several ways:

Loadpath or well-known location
URL to the package
Direct

Operations

Lyra should support the following operations:

Apply - Deploy the package
Upgrade - Upgrade the version of the package
Delete - Delete the package
Package - Create the package
Generate - Create a sample package structure

Additional context

Deferred

I would posit that first class repo support should be delayed to a later implementation.

Validation logic for admission for k8s objects

Supports #37

The Lyra k8s controller needs some validation logic on the K8s object spec to ensure that the object being passed to it conforms to the required input spec.

Add Lyra intermediate states to Kubernetes

As a Lyra workflow consumer, I want to see how my Lyra workflow is progressing before trying to interact with it. I would really like to be able to see in kubernetes kubectl get workflow attach-workflow some more detailed status on whether the apply is completed or not.

lyraproj / lyra Goto Github PK

lyra's Introduction

What is Lyra?

Getting Started

Build

Deploying Workflows with the Lyra CLI

Deploying Workflows with Kubernetes

Project Status

Roadmap

User Experience

Language Support

Content Ecosystem

Releases

[X] Proof of Concept

[ ] Release 0.3.0 - IN PROGRESS

[X] Release 0.2.0

[X] Release 0.1.0

Contributing

lyra's People

Contributors

Stargazers

Watchers

Forkers

lyra's Issues

Is your feature request related to a problem? Please describe.

Describe the solution you'd like

Proposed package architecture

Consuming packages

Operations

Additional context

Deferred

Recommend Projects

Recommend Topics

Recommend Org