run-x / opta

The next generation of Infrastructure-as-Code. Work with high-level constructs instead of getting lost in low-level cloud configuration.

Automated, secure, scalable cloud infrastructure

Documentation | Slack Community | Email: [email protected]

What is Opta?

Opta is a new kind of Infrastructure-as-Code (IaC) framework that lets engineers work with high-level constructs instead of getting lost in low-level cloud configuration. Opta has a vast library of modules (like EKS, RDS, DynamoDB, GKE, Cloud SQL, and even third-party services like Datadog) that engineers can compose together to build their ideal infrastructure stack. It's built on top of Terraform, and designed so engineers aren’t locked in – anyone can write custom Terraform or even take the Opta-generated Terraform and work independently.

Opta gives any engineering team, whether it’s a team of 2 or 200, the same infrastructure advantages that companies like Google or Facebook have, without having to invest in infrastructure or DevOps engineers.

Why use Opta?

Infrastructure-as-Code (IaC) solutions are now widely accepted as the standard for provisioning and managing cloud infrastructure, and Terraform is widely considered to be the best IaC platform on the market – and it is – but Terraform is also quite complex and requires deep Cloud/infrastructure expertise. We developed Opta to help eliminate this complexity. Opta is a simpler IaC framework with best practices built-in. It lets users set up automated, scalable and secure infrastructure on any cloud, without having to be an infrastructure expert, or getting lost in the minutiae of cloud configuration.

We are confident it can drastically reduce the complexity and headaches that come with DevOps and infrastructure at most fast moving organizations. Opta is currently being used by dozens of companies of all sizes.

To read more about the vision behind Opta, check out this blog post.

If you'd like to try it out or have any questions - feel free to join our Slack or explore the Getting Started Guide!

Deploying a Ruby on Rails application to AWS

Features

Cloud services

Opta supports the 3 major clouds - AWS, GCP and Azure. It has modules for the most commonly used services in these clouds like:

• Microservices (powered by Kubernetes)
• Databases - Postgres, MySQL, Redis
• Serverless workloads
• Networking - VPCs, Subnets, Load balancers
• CDN (Content Delivery Network)
• Object storage (S3, GCS)

Additionally, Opta has cloud and security best practices built in, including:

• Observability (Datadog, LogDNA)
• SOC2 compliance
• Continuous Deployment
• Hardened network and security configurations (AWS, GCP, Azure)
• Auto-scaling and high availability (HA)

Coexistence with existing infrastructure

Opta aims to be compatible with your existing infrastructure setup. You can:

• Import existing Terraform infrastructure into Opta
• Write custom Terraform modules (for services that Opta doesn't support yet)
• Run Opta in existing VPCs (WIP)
• Export the generated Terraform

Try out Opta

Check out the Getting Started Guide.

You can also explore some examples to get a better idea of what you can do with Opta.

Development

We love user contributions! Check out our Contributing and Dev guide to get started.

Important Resources

• The Opta Team
• Check Out The Blog
• How Opta delivers upgrades
• Bugfix / Feature request policy
• Comparison with other tools (WIP)
• Our Public roadmap
• Case studies - Flyte, Fastbreak Labs, Canvas app

opta's Issues

Do not require input variables in non apply/deploy commands on Opta CLI

What happened:

I've noticed that all opta cli commands require you to pass input variables when they shouldn't be required.
For example, I have the following one in my service file:
input_variables:

• name: github_sha
  which ends up populating en env var.

What you expected to happen:

But if I want to shell into my running container with opta shell, see logs, etc, I shouldn't need to specify these, but the cli requires me to.

One option is to mark them as optional (put a sane default) but I want them to be required when applying / deploying the YAML but not for everything else.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Opta version (use opta version): 0.32.2
• Cloud provider (or hardware configuration for local):
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

Peer file linking

Currently Opta allows to have parent-child relationship between any 2 Opta files (like env, and service). There are use cases where you want to import/link resources from multiple other files, like db creds from another service.

UI to visualize Opta services and environments

Implement RBAC for opta services and environments

Process for module migrations

Given a breaking module upgrade, implement a way to seamlessly migrate users from the older version to the newer version.

Standardize linking API between modules

Standardize how one module can export it's variables which are discoverable by other modules, without having explicit dependencies between the 2 modules

Advanced pipelines for CD - Argo support

Support remote terraform modules in custom-terraform

Ability to refer to external terraform modules like the modules here: https://registry.terraform.io/browse/modules

This would enable the users to not rely on Opta modules for every infrastructure piece out there

Refactor module API

is it really necessary to apply the "aws_s3_bucket_public_access_block" restrictions to s3 buckets?

Places where these permissions are needed:

• opta/modules/aws_base/tf_module/log_bucket.tf

  Line 45 in 634d9b8

  resource "aws_s3_bucket_public_access_block" "block" {

• opta/modules/aws_s3/tf_module/replication.tf

  Line 107 in aa2cbd4

  resource "aws_s3_bucket_public_access_block" "block_for_replica" {

• opta/modules/aws_s3/tf_module/bucket.tf

  Line 91 in fc4b1ac

  resource "aws_s3_bucket_public_access_block" "block" {

• opta/opta/core/terraform.py

  Line 757 in 634d9b8

  s3.put_public_access_block(

My problem is that I can't get those permissions.

Is it possible to get rid of this code and just create private buckets? For example, via acl = "private" https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket#private-bucket-w-tags
It is not very clear why the bucket should not be able to be public at all...

Support for the depends_on Terraform feature

What would you like to be added:

Support for Terraform's depends_on feature.

Why is this needed:

Forcing terraform to create certain modules in the same manifest file before others, particularly for jobs and other applications that need to be up before the latest version of the application code can be put up.

An example of this would be for database migrations. Migrations to add the new column should be run before the application code can query the database. While you can create a separate manifest to manage it, as described in the docs, that means you have to have reference to the database outside where the database is created and managed.

Extra info (e.g. existing slack convo link):

https://opta-group.slack.com/archives/C0240NBKUMT/p1653326679695919?thread_ts=1653099767.603579&cid=C0240NBKUMT

Opta rollback to resolve terraform state conflicts

If a terraform apply is canceled midway, it can lead to scenarios where the resource is created but terraform state isn't updated. This can cause any future runs to fail. Implement an opta rollback command which can import such resources in the terraform state when this happens.

Add support for ETL workloads

Support installation via Brew

Support GCP spot machines

What would you like to be added:

Google Cloud recently rolled out spot VMs. Unlike the existing preemptible machine type, a spot VM has a notionally unlimited lifetime. Pricing for spot VMs is the same as for preemptible VMs, typically about 30% of the cost of a dedicated VM.

Why is this needed:

Imagine running a cluster using relatively expensive machines, like Tau 2D instances. Your pods are part of a big data serving platform with automatic shard management - great! That means you can use ephemeral VMs because the cluster will automatically reflow data away from non-operational nodes, and redistribute when new nodes come up.

Of course, this takes time, and loading the indices to memory also takes more time. The tradeoff is that we can run at a fraction of the cost!!

Overall, this use case is well-served by spot instances. It is poorly served by preemptible instances. When using preemtible VMs, the data shuffle and index build/load requires ~15% of life just for reshuffle data. Spot is more like 4% in practice. Overall impact on query latency exceeds that level of improvement.

Extra info (e.g. existing slack convo link):

The SPOT provisioning_model is supported in Terraform 4.23 as a beta feature.

(Optional, Beta) Describe the type of preemptible VM. This field accepts the value STANDARD or SPOT. If the value is STANDARD, there will be no discount. If this is set to SPOT, preemptible should be true and auto_restart should be false.

https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_instance#provisioning_model

Slack link

Continuous release + automatic upgrades for Opta CLI

Delete information about the account id, in the process of generation `opta_crash_report.zip`

What would you like to be added:

Why is this needed:

This is sensitive information, I'd rather not share it.

Extra info (e.g. existing slack convo link):

Support for waits_for_job

What would you like to be added:

Helm Release has a waits_for_job argument that will hold the terraform from completing until the jobs complete.

I would like to wait for jobs to complete before I deploy my k8s-service module.

Why is this needed:

When running migrations, you want the migrations to be complete before deploying your app code, otherwise your app could start querying a column that is not there yet, causing 500 errors.

Extra info (e.g. existing slack convo link):

https://opta-group.slack.com/archives/C0240NBKUMT/p1653335558506789?thread_ts=1653099767.603579&cid=C0240NBKUMT

Related:

#893

k8s-manifest module broken

What happened:

When trying to use k8s-manifest, I get an error upon generate-terraform. It expects a kubeconfig file to be present at a specific path, instead of using the currently active kubeconfig context in terraform.

Error:

ERROR: [Errno 2] No such file or directory: '/home/nvarz/.opta/kubeconfigs/kubeconfig-integration-aws.yaml'
Traceback (most recent call last):
  File "opta/cli.py", line 77, in <module>
  File "click/core.py", line 1128, in __call__
  File "click/core.py", line 1053, in main
  File "click/core.py", line 1659, in invoke
  File "click/core.py", line 1395, in invoke
  File "click/core.py", line 754, in invoke
  File "click/decorators.py", line 26, in new_func
  File "opta/commands/generate_terraform.py", line 104, in generate_terraform
  File "opta/layer.py", line 421, in validate_required_path_dependencies
  File "opta/layer.py", line 415, in get_required_path_dependencies
  File "opta/layer.py", line 507, in processor_for
  File "modules/k8s_manifest/k8s_manifest.py", line 23, in __init__
    ) = self.get_k8s_config_context(layer)
  File "modules/k8s_manifest/k8s_manifest.py", line 38, in get_k8s_config_context
    with open(config_file_name, "r") as stream:
FileNotFoundError: [Errno 2] No such file or directory: '/home/nvarz/.opta/kubeconfigs/kubeconfig-integration-aws.yaml'
Unhandled error encountered -- a crash report zipfile has been created for you. If you need more help please reach out (passing the crash report) to the contributors in our slack channel at: https://slack.opta.dev/

What you expected to happen:

Generate valid terraform from my k8s-manifest.

How to reproduce it (as minimally and precisely as possible):

opta.yaml:

org_name: ddmit
name: integration
providers:
  aws:
    region: eu-central-1
    account_id: "123456789"
modules:
  - type: base
    vpc_log_retention: 3
  - type: k8s-cluster
    min_nodes: 1
    eks_log_retention: 3
    k8s_version: "1.22"
    node_launch_template:
      user_data: ""
  - type: k8s-base
    admin_arns:
      - arn:aws:iam::123456789:user/*
  - type: k8s-manifest
    file_path: custom/service-account-gitlab-runner.yaml

k8s-manifest:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: gitlab-runner
  namespace: gitlab-runner

Anything else we need to know?:

Environment:

{
  "opta_version": "0.30.2",
  "platform": "Linux",
  "os_name": "posix",
  "os_version": "#1 SMP PREEMPT Wed Apr 13 22:09:20 UTC 2022"
}

• Cloud provider (or hardware configuration for local):
  AWS/Azure/GCP

opta upgrade broken on fedora

What happened:

opta notified me of an available upade, upon running opta upgrade, it fails.

$ opta upgrade
Checking for version upgrades...
WARNING: New version available.
You have 0.30.2 installed. Latest version is 0.31.0.
Welcome to the opta installer.
Checking Prerequisites...
Opta would require (docker) to run properly. Please install these.
Determining latest version
curl: /home/nvarz/.opta/libcrypto.so.1.1: version `OPENSSL_1_1_1b' not found (required by /lib64/libssh.so.4)
curl: /home/nvarz/.opta/libcrypto.so.1.1: version `OPENSSL_1_1_1b' not found (required by /lib64/libk5crypto.so.3)
./opta_installation.sh: line 20: MAC_ADDRESS: unbound variable
Going to install opta v
Downloading installation package...
curl: /home/nvarz/.opta/libcrypto.so.1.1: version `OPENSSL_1_1_1b' not found (required by /lib64/libssh.so.4)
curl: /home/nvarz/.opta/libcrypto.so.1.1: version `OPENSSL_1_1_1b' not found (required by /lib64/libk5crypto.so.3)
curl: /home/nvarz/.opta/libcrypto.so.1.1: version `OPENSSL_1_1_1b' not found (required by /lib64/libssh.so.4)
curl: /home/nvarz/.opta/libcrypto.so.1.1: version `OPENSSL_1_1_1b' not found (required by /lib64/libk5crypto.so.3)
Version  not found.
Please check the available versions at https://github.com/run-x/opta/releases.

Using the way described in the docs works (more or less, there is an error from mv but apparently it's not critical):

$ VERSION=0.31.0 /bin/bash -c "$(curl -fsSL https://docs.opta.dev/install.sh)"
Welcome to the opta installer.
Checking Prerequisites...
Opta would require (docker) to run properly. Please install these.
Going to install opta v0.31.0
Downloading installation package...
Downloaded
Opta already installed. Overwrite? y
mv: cannot stat '/home/nvarz/.opta/local': No such file or directory

Installing...
mv: cannot stat '/tmp/opta_local/local': No such file or directory
Please symlink the opta binary to one of your path directories; for example using 'sudo ln -fs /home/nvarz/.opta/opta /usr/local/bin/opta'
Alternatively, you could add the .opta installation directory to your path like so
export PATH=$PATH:/home/nvarz/.opta
to your terminal profile.

What you expected to happen:

opta should upgrade itself to the latest version

How to reproduce it (as minimally and precisely as possible):

run opta upgrade from version 0.30.2

Anything else we need to know?:

Environment:

{
  "opta_version": "0.30.2",
  "platform": "Linux",
  "os_name": "posix",
  "os_version": "#1 SMP PREEMPT Wed Apr 13 22:09:20 UTC 2022"
}

Error when using a region with less than 3 availability zones

What happened:

I ran into an error during opta apply:

ERROR:  Opta requires a region with at least *3* availability zones.
You configured us-west-1, which only has the availability zones: ['us-west-1a', 'us-west-1c'].
Please choose a different region.

What you expected to happen:

I think it should be mentioned in Getting Started that you need a region with at least 3 availability zones.

Environment:

• Opta version (use opta version): v0.24.3
• Cloud provider (or hardware configuration for local): AWS
• OS (e.g: cat /etc/os-release): Ubuntu 20.04
• Kernel (e.g. uname -a):
• Install tools:
• Others:

Add VictoriaMetrics to integration list

Hi there!

I see that you have support Prometheus integration on https://www.runx.dev/integrations
My question is, what about to add VictoriaMetrics to integration list because it support PromQL but have many other features in MetricsQL and improvements?

Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "Ingress" in version "networking.k8s.io/v1"

What happened:

opta apply -c flyte.yaml --auto-approve returned this error:

╷
│ Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "Ingress" in version "networking.k8s.io/v1"
│
│   with module.helmchart.helm_release.local_chart[0],
│   on ../../../.opta/modules/helm_chart/tf_module/main.tf line 18, in resource "helm_release" "local_chart":
│   18: resource "helm_release" "local_chart" {
│
╵

How to reproduce it (as minimally and precisely as possible):

I followed this instruction with two changes:

1. I deleted these lines from `env.yaml':

  - type: dns
    domain: <domain>
    delegated: false # set to true once ready https://docs.opta.dev/miscellaneous/ingress/

2. I changed files .opta/modules/aws_s3/aws-s3.yaml, .opta/modules/aws_s3/tf_module/replication.tf, .opta/modules/aws_base/tf_module/log_bucket.tf to remove public access blocks for s3 buckets.

Environment:

• Opta version (use opta version): v0.24.3
• Cloud provider (or hardware configuration for local): AWS
• OS (e.g: cat /etc/os-release): Ubuntu 20.04
• Kernel (e.g. uname -a):
• Install tools:
• Others:

Please add the ability to add other headers here

opta/modules/cloudfront_distribution/tf_module/main.tf

Line 86 in 249887d

headers = ["Origin", "Access-Control-Request-Headers", "Access-Control-Request-Method", "Host"]

Auto import existing resources into Opta

Error While Installing CLI on Linux

Welcome to the opta installer.
Determining latest version
Going to install opta v0.14.7
Downloading installation package...
Downloaded
Installing...
/bin/bash: line 60: unzip: command not found
chmod: cannot access '/root/.opta/opta': No such file or directory
Successfully installed! Now you can run it via invoking /root/.opta/opta

`opta version` command failed

What happened:

$ .opta/opta version
[3335] Error loading Python lib '/home/ec2-user/.opta/libpython3.8.so.1.0': dlopen: /lib64/libc.so.6: version `GLIBC_2.27' not found (required by /home/ec2-user/.opta/libpython3.8.so.1.0)

How to reproduce it (as minimally and precisely as possible):

/bin/bash -c "$(curl -fsSL https://docs.opta.dev/install.sh)"

Environment:

• Opta version (use opta version): failed
• Cloud provider (or hardware configuration for local): AWS
• OS (e.g: cat /etc/os-release):

NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

• Kernel (e.g. uname -a): Linux ip-172-31-89-70.ec2.internal 5.10.93-87.444.amzn2.x86_64 #1 SMP Thu Jan 20 22:50:50 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
• Install tools:
• Others:

k8s-cron module to support cron job

A k8s-cron module, similar to the k8s-service module but for cron jobs.

Azure GA support

Support policy engine like OPA

Define API for other cloud providers

AWS, GCP and Azure providers are assumed at various places in the codebase. We want to abstract away the cloud provider details so that it's possible to easily extend Opta for other cloud providers

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool

What happened:

Run command: opta apply -c flyte/opta/aws/env.yaml failed, see error below.

Error

ERROR: HTTPSConnectionPool(host='04803a22277127da2dff766d486c4519.gr7.us-east-1.eks.amazonaws.com', port=443): Max retries exceeded with url: /api/v1/services (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fa6a3f10580>: Failed to establish a new connection: [Errno -2] Name or service not known'))
Traceback (most recent call last):
  File "urllib3/connection.py", line 174, in _new_conn
  File "urllib3/util/connection.py", line 72, in create_connection
  File "socket.py", line 918, in getaddrinfo
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "urllib3/connectionpool.py", line 703, in urlopen
  File "urllib3/connectionpool.py", line 386, in _make_request
  File "urllib3/connectionpool.py", line 1040, in _validate_conn
  File "urllib3/connection.py", line 358, in connect
  File "urllib3/connection.py", line 186, in _new_conn
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7fa6a3f10580>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "opta/cli.py", line 66, in <module>
  File "click/core.py", line 1128, in __call__
  File "click/core.py", line 1053, in main
  File "click/core.py", line 1659, in invoke
  File "click/core.py", line 1395, in invoke
  File "click/core.py", line 754, in invoke
  File "opta/commands/apply.py", line 105, in apply
  File "opta/commands/apply.py", line 323, in _apply
  File "opta/commands/apply.py", line 210, in _apply
  File "opta/core/generator.py", line 84, in gen
  File "opta/layer.py", line 383, in gen_tf
  File "modules/aws_k8s_base/aws_k8s_base.py", line 45, in process
    self._process_nginx_extra_ports(self.module.data)
  File "modules/base.py", line 396, in _process_nginx_extra_ports
    service_port_mapping = reconcile_nginx_extra_ports(update_config_map=False)
  File "modules/base.py", line 651, in reconcile_nginx_extra_ports
    services = kubernetes.list_services()
  File "opta/core/kubernetes.py", line 566, in list_services
  File "kubernetes/client/api/core_v1_api.py", line 17764, in list_service_for_all_namespaces
  File "kubernetes/client/api/core_v1_api.py", line 17867, in list_service_for_all_namespaces_with_http_info
  File "kubernetes/client/api_client.py", line 348, in call_api
  File "kubernetes/client/api_client.py", line 180, in __call_api
  File "kubernetes/client/api_client.py", line 373, in request
  File "kubernetes/client/rest.py", line 239, in GET
  File "kubernetes/client/rest.py", line 212, in request
  File "urllib3/request.py", line 74, in request
  File "urllib3/request.py", line 96, in request_encode_url
  File "urllib3/poolmanager.py", line 375, in urlopen
  File "urllib3/connectionpool.py", line 813, in urlopen
  File "urllib3/connectionpool.py", line 813, in urlopen
  File "urllib3/connectionpool.py", line 813, in urlopen
  File "urllib3/connectionpool.py", line 785, in urlopen
  File "urllib3/util/retry.py", line 592, in increment
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='04803a22277127da2dff766d486c4519.gr7.us-east-1.eks.amazonaws.com', port=443): Max retries exceeded with url: /api/v1/services (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fa6a3f10580>: Failed to establish a new connection: [Errno -2] Name or service not known'))

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

opta_crash_report (2).zip

Environment:

• Opta version (use opta version): v0.24.3
• Cloud provider (or hardware configuration for local): AWS
• OS (e.g: cat /etc/os-release): Ubuntu 20.04
• Kernel (e.g. uname -a):
• Install tools:
• Others:

Digital Ocean GA support

`opta destroy -c flyte/opta/aws/env.yaml` doesn't destroy CloudWatch Log Group

What happened:

First I ran the command: opta apply -c flyte/opta/aws/env.yaml that failed, see error below:

Error

module.k8sbase.helm_release.ingress-nginx: Still creating... [5m0s elapsed]
╵
╷
│ Error: release ingress-nginx failed, and has been uninstalled due to atomic being set: timed out waiting for the condition
│
│   with module.k8sbase.helm_release.ingress-nginx,
│   on .opta/modules/aws_k8s_base/tf_module/ingress_nginx.tf line 2, in resource "helm_release" "ingress-nginx":
│    2: resource "helm_release" "ingress-nginx" {
│
╵
ERROR: Command '['exec', 'terraform', 'apply', '-compact-warnings', 'tf.plan']' returned non-zero exit status 1.

Then I ran the command: opta destroy -c flyte/opta/aws/env.yaml which completed successfully.

After that I ran the command opta apply -c flyte/opta/aws/env.yaml again that failed, see error below:

Error

╷
│ Error: Creating CloudWatch Log Group failed: ResourceAlreadyExistsException: The specified log group already exists:  The CloudWatch Log Group '/aws/eks/opta-amyachev-env/cluster' already exists.
│
│   with module.k8scluster.aws_cloudwatch_log_group.cluster_logs,
│   on .opta/modules/aws_eks/tf_module/eks.tf line 74, in resource "aws_cloudwatch_log_group" "cluster_logs":
│   74: resource "aws_cloudwatch_log_group" "cluster_logs" {
│
╵
ERROR: Command '['exec', 'terraform', 'apply', '-compact-warnings', 'tf.plan']' returned non-zero exit status 1.
Traceback (most recent call last):
  File "opta/cli.py", line 66, in <module>
  File "click/core.py", line 1128, in __call__
  File "click/core.py", line 1053, in main
  File "click/core.py", line 1659, in invoke
  File "click/core.py", line 1395, in invoke
  File "click/core.py", line 754, in invoke
  File "opta/commands/apply.py", line 105, in apply
  File "opta/commands/apply.py", line 323, in _apply
  File "opta/commands/apply.py", line 315, in _apply
  File "opta/commands/apply.py", line 310, in _apply
  File "opta/core/terraform.py", line 145, in apply
  File "opta/nice_subprocess.py", line 190, in nice_run
  File "opta/nice_subprocess.py", line 160, in nice_run
  File "opta/utils/runtee.py", line 157, in run
subprocess.CalledProcessError: Command '['exec', 'terraform', 'apply', '-compact-warnings', 'tf.plan']' returned non-zero exit status 1.
Unhandled error encountered -- a crash report zipfile has been createded for you. If you need more help please reach out (passing the crash report) to the contributors in our slack channel at: https://slack.opta.dev

What you expected to happen:

I expect that after the destroy command, there are no resources left

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Opta version (use opta version): x0.24.3
• Cloud provider (or hardware configuration for local): AWS
• OS (e.g: cat /etc/os-release): Ubuntu 20.04
• Kernel (e.g. uname -a):
• Install tools:
• Others:

Generate terraform files given Opta file

Allow user to exit out of Opta by generating terraform hcl files given their Opta files

Automatic selection of a domain to create a DNS resource

What would you like to be added:

Now I have to manually fill <domain> field in the file flyte/opta/aws/env.yaml. However, for example, it is possible to create ec2 instances with default DNS (example: ec2-3-237-182-243.compute-1.amazonaws.com). Can this approach be reused for opta?

Why is this needed:

Simplifying the user experience in situations where it is not needed.

Extra info (e.g. existing slack convo link):

Support private Opta modules

Define an API on how a user can implement an Opta module outside core Opta code and use it in their Opta installation.

Generate a terraform security report given an Opta file

Predict cloud spend given Opta config

Build a tool to predict cloud spend for an Opta yaml file.

Module versions

Implement versions for each module so that if a breaking change needs to be made for a module, existing users have the option to stick with the old version to avoid downtime

Error: release ingress-nginx failed, and has been uninstalled due to atomic being set: timed out waiting for the condition

What happened:

opta apply -c flyte.yaml --auto-approve returned this error:

module.k8sbase.helm_release.ingress-nginx: Still creating... [5m0s elapsed]


╵
╷
│ Error: release ingress-nginx failed, and has been uninstalled due to atomic being set: timed out waiting for the condition
│
│   with module.k8sbase.helm_release.ingress-nginx,
│   on .opta/modules/aws_k8s_base/tf_module/ingress_nginx.tf line 2, in resource "helm_release" "ingress-nginx":
│    2: resource "helm_release" "ingress-nginx" {
│
╵
ERROR: Command '['exec', 'terraform', 'apply', '-compact-warnings', 'tf.plan']' returned non-zero exit status 1.

How to reproduce it (as minimally and precisely as possible):

I followed this instruction with two changes:

1. I deleted these lines from `env.yaml':

  - type: dns
    domain: <domain>
    delegated: false # set to true once ready https://docs.opta.dev/miscellaneous/ingress/

2. Made some changes suggested here: #647 (comment)

Anything else we need to know?:

Environment:

• Opta version (use opta version): v0.24.3
• Cloud provider (or hardware configuration for local): AWS
• OS (e.g: cat /etc/os-release): Ubuntu 20.04
• Kernel (e.g. uname -a):
• Install tools:
• Others:

Multi-region support for AWS

Enable multi-region support in Opta for AWS which includes:

• A k8s-cluster in each of the specified region
• Each database has the write replica in one of the chosen regions and read replicas in the rest
• A smart geolocation based traffic routing to the nearest AWS region

Enable deploying a K8s cluster to an existing VPC

A lot of companies have their VPC already setup when they want to try out Opta. Currently it's not possible to use the k8s-cluster module without using the base module (which creates the VPC). We want to remove this strong dependency

OPTA doesn't delete the S3 bucket that created

What happened:

An error occurred while calling the command: opta apply -c flyte/opta/aws/env.yaml.

Error

ERROR: An error occurred (AccessDenied) when calling the PutPublicAccessBlock operation: Access Denied
Traceback (most recent call last):
  File "opta/core/terraform.py", line 808, in _create_aws_state_storage
  File "botocore/client.py", line 391, in _api_call
  File "botocore/client.py", line 719, in _make_api_call
botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the GetBucketEncryption operation: The specified bucket does not exist

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "opta/cli.py", line 66, in <module>
  File "click/core.py", line 1128, in __call__
  File "click/core.py", line 1053, in main
  File "click/core.py", line 1659, in invoke
  File "click/core.py", line 1395, in invoke
  File "click/core.py", line 754, in invoke
  File "opta/commands/apply.py", line 105, in apply
  File "opta/commands/apply.py", line 182, in _apply
  File "opta/core/terraform.py", line 935, in create_state_storage
  File "opta/core/terraform.py", line 851, in _create_aws_state_storage
  File "botocore/client.py", line 391, in _api_call
  File "botocore/client.py", line 719, in _make_api_call
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutPublicAccessBlock operation: Access Denied
Unhandled error encountered -- a crash report zipfile has been createded for you. If you need more help please reach out (passing the crash report) to the contributors in our slack channel at: https://slack.opta.dev

After that I use the command to destroy the created resources: opta destroy -c flyte/opta/aws/env.yaml
Output: The opta state could not be found. This may happen if destroy ran successfully before.

However, using the AWS web interface, I see that the bucket created by the opta remains. It's kind of a resource leaking.

What you expected to happen:

I expect that regardless of the success of the previous command, it is possible to delete all created resources.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Opta version (use opta version): v0.24.3
• Cloud provider (or hardware configuration for local): AWS
• OS (e.g: cat /etc/os-release): Ubuntu 20.04.3 LTS
• Kernel (e.g. uname -a):
• Install tools:
• Others:

Expose the --image-tag command line parameter as a variable or output of the k8s-service module.

What would you like to be added:

I would like to be able to reference the --image-tag command line parameter as a variable in my Opta YAML.

I see that you can refer to the entire image name: https://docs.opta.dev/reference/aws/modules/aws-k8s-service/#outputs, but that's the entire image name, including the repo, where all I need is the tag.

Why is this needed:

Without this, you have to create a duplicitous input_variable.

Implement VPN module

Add features to k8s-service module

What would you like to be added:

1. Would like dashes allowed in k8-service module, current error is:

ERROR: Invalid module name, can only contain letters and numbers!
If you need more help please reach out to the contributors in our slack channel at: https://slack.opta.dev

2. Would like to have custom health checks in same k8s-service module like in standard k8s yaml

  - name: my-service
    type: k8s-service
...
    livenessProbe:
      exec:
        command:
        - cat
        - /tmp/healthy
...

Why is this needed:

1. This is a request from my team to stay consistent with current environments (pre opta migration)
2. A request from my team to allow this while we create actual health-check endpoints AND for 3rd part containers where we do not control the code to expose endpoints

Extra info (e.g. existing slack convo link):

https://opta-group.slack.com/archives/C0240NBKUMT/p1652223456899139

Cannot use variables for CPU and Memory Resource Requests / Limits

What happened:

When using variables for CPU and memory, we get this error:

Opta file validation failed with errors:
  : modules.0: resource_request.cpu: '{vars.cpu}' is not a int.
  : modules.0: resource_request.cpu: '{vars.cpu}' is not a regex match.
  : modules.0: resource_request.memory: '{vars.memory}' is not a int.
  : modules.0: resource_request.memory: '{vars.memory}' is not a regex match.

What you expected to happen:

The variable replacement for CPU and memory to work as documented similarly to min_containers and max_containers.

How to reproduce it (as minimally and precisely as possible):

name: app
environments:
  - name: staging
    path: "staging.yml"
    vars:
      min_containers: 1
      max_containers: 1
      cpu: 1000
      memory: 2048
      environment: staging
modules:
  - type: k8s-service
    name: api
    image: AUTO
    resource_request:
      cpu: "{vars.cpu}"
      memory: "{vars.memory}"
    min_containers: "{vars.min_containers}"
    max_containers: "{vars.max_containers}"

Anything else we need to know?:

Environment:

• Opta version (use opta version): 0.31.0
• Cloud provider (or hardware configuration for local): AWS
• OS (e.g: cat /etc/os-release): MacOS
• Kernel (e.g. uname -a): Darwin 19.6.0 Darwin Kernel Version 19.6.0: Sun Jul 5 00:43:10 PDT 2020; root:xnu-6153.141.1~9/RELEASE_X86_64 x86_64

Support for preview environments

Circle CI plugin for Opta