amazon-archives / aws-service-operator

AWS Service Operator allows you to create AWS resources using kubectl.

See aws/aws-service-operator-k8s.

aws-service-operator's Issues

Adding SQS resource

Creates SQS Queues Using CRDS like this:

apiVersion: operator.aws/v1alpha1
kind: SQS
metadata:
  name: chrishein-test-sqs-1
spec:
  contentBasedDeduplication: false
  delaySeconds: 1
  usedeadletterQueue: false

Adding SNS Topic Creation

Allow CFT S3 Bucket Storage to be Dogfooded

As of right now, you must supply an S3 bucket for the operator to use for storing the cfts, it should be able to be added using a CRD meaning you deploy and have a reference to an s3bucket that gets deployed when the operator comes alive

Adding License file

Adding Base Development Guidelines

Operator Controller and CFT packages

Right now the cft.go and the controller.go are packed into the same package they should individually be their own templates and sub-packages. Allowing independent testing. Also, rename the controller.go to operator.go

Base Operator without resources

Without model files just the core code base.

Add a License file

Always important to know what will happen to the project

Write Usage Documentation as linked in the readme

This should talk about how to deploy each type of resource and have information about the keys that are available including best practices etc. This issue should be broken down and linked to other issues for each type of resource. Maybe these are added into the code generation libraries.

Create Cloudformation Resources

Should upload the raw template to the S3 bucket setup when creating the
operator and be formatted like so

apiVersion: operator.aws/v1alpha1
kind: CloudFormationTemplate
metadata:
  name: s3bucket
  data:
    key: s3bucket.yaml
    template: |
      # ... Raw JSON or YAML template

Adding Go Releaser Codebase

Figure out least perms RBAC rule for operator

Right now just to make this work it has a very open policy and should be paired
down before we have production users.

Adding Base Operator without Resources

This doesn't technically do anything yet, but since the resources are code generated this will make the history really simple.

Adding S3Bucket Resource

This should create s3buckets and be formatted like so:

apiVersion: operator.aws/v1alpha1
kind: S3Bucket
metadata:
  name: test-bucket
spec:
  bucketName: test-bucket-name
  versioning: false
  logging:
    enabled: false
    prefix: "archive"

Use a Service Role and an Execution Role for CFTS

After talking with @jaymccon about how the Service Broker works separating these roles will help with long-running CFTs.

Add Support for Internal/External Password Managers

Start with building the support for AWS Secrets Manager and Vault then extend this to support other password managers.

The idea is to not require the use of Kubernetes Secrets to store sensitive information for the workloads. This should allow for a seamless configuration potentially referencing a secret manager at boot which will automatically be used OR each manifest could reference the secret manager of choice.

SyncAdditionalResources Should be after CFT Complete

Right now the resources are created after the resource is created this makes it difficult to have a proper lifecycle if the configmaps and services come up after the fact I can halt pods and deployments from being alive prior.

Create DynamoDB Resource

Should manage DynamoDB resources and be formatted like:

apiVersion: operator.aws/v1alpha1
kind: DynamoDB
metadata:
  name: chrishein-dynamodb-table-2
spec:
  tableName: chrishein-dynamodb-table-2
  hashAttribute:
    name: user_id
    type: S
  rangeAttribute:
    name: created_at
    type: S
  readCapacityUnits: 5
  writeCapacityUnits: 5

Create ElastiCache Resource

This resource will allow a developer to create an ElastiCache instance using
the CRD model, it should be generic enough to support both memcached and redis.

apiVersion: operator.aws/v1alpha1
kind: ElastiCache
metadata:
  name: elasticache
spec:
  autoMinorVersionUpgrade: true
  azMode: # (single-az|cross-az) only memcached
  nodeType: cache.m3.medium
  engine: # (memcached|redis)
  version: 4.0.10
  cacheNodes: 3
  tags:
  - name: Usage
    value: caching

Add Goreleaser

Making it so the project can be released with ease.

Create S3 Bucket CRD Resource

This should create s3buckets and be formatted like so:

apiVersion: operator.aws/v1alpha1
kind: S3Bucket
metadata:
  name: test-bucket
spec:
  bucketName: test-bucket-name
  versioning: false
  logging:
    enabled: false
    prefix: "archive"

Add SNS Subscription Resource

This should take an SNSTopic resource and bind it using a Subscription, this
should be able to both reference local SQS resource types as well as non-SQS
types.

apiVersion: operator.aws/v1alpha1
kind: SNSSubscription
metadata:
  name: sns-sqs-subscription
spec:
  snsTopicName: sns-topic
  subject:
    protocol: sqs
    name: sqs-queue
    # endpoint: optional

Add Development or Contributing Guidelines

This should show how to setup the operator locally and build using the codegen
libraries to build and deploy new resources.

Create CloudFormation Resources

This allows you to create CFTs and when they are deployed they are pushed to S3

Implement Flag for only allowing specific resources

The --resources flag is meant to turn on and off operators for the subsequent resources that are selected but it's currently implemented.

Add demo screen capture of the operator working

Idea is to give more clarity into what this project solves upfront since it can
be fairly complex to understand all that it can do.

Adding cmd directory and edit .gitignore

Add UUID Custom Resources type

This should be considered a string but be a generated type, when this type is
used it shouldn't be apart of the actual object other than stored in the
status of the object.

Use cases:

1. An S3 bucket's name can't change so having bucket name in the CRD Spec feels
   wrong.
2. An SQS Queue name can't change so having the queue name in the CRD is again
   wrong.

Add the Ability to Provision an ECR registry

This makes it possible to get an ECR registry for a project while using the operator.

apiVersion: operator.aws/v1alpha1
kind: ECR
metadata:
  name: test-app
spec:
  lifecyclePolicy: |
    # ...
  repositoryPolicy: |
    # ...

Create KMS Resource Type

This resource type will be pulled into other resources just like the CloudFormationTemplates, should be gotten by a reference for services like rds, s3, etc.

Create SQS Queue Resource

Should create SQS Queues and be formatted:

apiVersion: operator.aws/v1alpha1
kind: SQS
metadata:
  name: chrishein-test-sqs-1
spec:
  contentBasedDeduplication: false
  delaySeconds: 1
  usedeadletterQueue: false

Cut Docker Container and Update Manifest

This issue is to create an update deployment manifest for the operator and push
a built image (or multiple) to some registry for usage.

The end result is a deployable manifest that someone could deploy into their clusters.

Image shouldn't be named aws-service-operator until this is OSS'ed

Delete AdditionalResources when Deleting a Resource

This should remove any services or comfigmaps when you clean up the cluster resources for example when you make an s3 bucket and it creates a configmap and service those should be cleaned up after the fact.

Integrate S3 and Additional Resources

Use the Additional resources for a Service

Rip out and Replace Operator Kit

Operator Kit, although was great to get started with and allowed the project to move really fast, has created complications around not having the ability to create a shared context across both queue and the informer library. Using a builtin package will help remove this issue. Use https://github.com/christopherhein/operator-kit as a reference for what needs to be done.

Updates that should make this better would be using a set timeout on the informer so that the cache is resynced periodically, this will cause issues with API limits but that will be easy to deal with. Also making it so that there is an interface that each operator can implement instead of what we do
now will help.

I think this will be successful if the server.go package loads up one single package from the root of pkg/operator/base which then loops through all the loaded operators and initializing.

Auto generate list of operators to load

This feature will be mostly implemented in the aws-operator-codegen package but this would setup a separate package with some function that the server calls which is auto-generated based on the model files. Removing the last manual step for adding new code.

Write readme for project

Write out the thoughts behind the project.

Adding S3 Bucket Service Using ExternalName

BUG: ./cmd/aws-operator removed via gitignore

My .gitignore was too open and removed my cmd directory

Add OWNERS file for the project

Need some more contributors!

Enable Scale Endpoint to Increment Cache Nodes

With Kubernetes 1.11 CRDs support subcommands like scale use this to enable
the elasticache instances to scale when you call that command.

kubectl scale elasticache my-elasticache --cache-nodes 5

Create example IAM Role for the Operator

This should live in configs/aws-operator.json where the readme documents it
and should have a sample role that allows CFTs to create any resources we have
a type for.

Create RDS Resource Type

This should create any engine of the AWS::RDS:DBInstance class meaning it should support creating mysql, postgres, mariadb, etc all via the same spec.

apiVersion: operator.aws/v1alpha1
kind: RDS
metadata:
  name: my-rds
spec:
  engine: postgres
  backupRetentionPeriod: 30
  instanceClass: db.m1.small
  version: 9.6.3
  storage:
    capacity: 100GB
    type: io1
    encrypted: true
    iops: 1000
  versioning:
    allowMajorUpgrade: true
    allowMinorUpgrade: true
  user: # optional
    username: chris # optional
    password: foobar # optional

Create SNS Topic Resource

Should create an SNSTopic, this is meant to be used with a SNSTopicSubscription. TopicName for the CFT should be a UUID resource type after issue #34 is finished

apiVersion: operator.aws/v1alpha1
kind: SNSTopic
metadata:
  name: sns-topic
spec:
  displayName: sns-topic

Update Definitions to Remove Non Changeable fields

After issue #34 is complete, update the S3Bucket, SQSQueue, and
DynamoDBTable resources to remove the resource names which aren't changeable.

Allow Support for Creating Secrets

Just like support for #43 but this one will create Kubernetes Secrets.

Convert Queuing into Single SQS Queue with Fan Out

Right now the queuing internally uses an SQS queue and SNS topic per resource meaning that when you add additional resources we have to create new watchers for each component, this turns into a lot of resources created for the full operator to run.

Instead what would be better is to use a single SQS queue and many SNS topics than in the singular queue have it fan out to the necessary resources for processing.

This should go along with the refactoring out of OperatorKit #23

Add Support For Creating ConfigMaps

Just like the ExternalName Service integration allow for ConfigMaps to be
created and templatized with the proper parameters from the CFTs

Add a CI Pipeline to test with

Using some cluster in AWS have test published back to the repo so we can test PRs with some better reliability as well as using this to release new packages.

Add DynamoDB resource

Allows you to create Dynamodb resources using an operator

apiVersion: operator.aws/v1alpha1
kind: DynamoDB
metadata:
  name: chrishein-dynamodb-table-2
spec:
  tableName: chrishein-dynamodb-table-2
  hashAttribute:
    name: user_id
    type: S
  rangeAttribute:
    name: created_at
    type: S
  readCapacityUnits: 5
  writeCapacityUnits: 5