Coder Social home page Coder Social logo

camptocamp / terraboard Goto Github PK

View Code? Open in Web Editor NEW
1.9K 41.0 159.0 10.98 MB

:earth_africa: :clipboard: A web dashboard to inspect Terraform States

Home Page: https://terraboard.io

License: Apache License 2.0

Go 96.27% HTML 0.07% JavaScript 0.02% Makefile 0.11% Dockerfile 0.02% Vue 3.45% TypeScript 0.07%
terraform terraform-states web-dashboard aws go devops devops-tools versioning diff infrastructure-as-code

terraboard's Introduction

Terraboard

Terraboard logo

๐ŸŒ ๐Ÿ“‹ A web dashboard to inspect Terraform States

Docker Pulls Go Report Card Gitter Build Status Coverage Status By Camptocamp Documentation

Website: https://terraboard.io


Caution: Terraboard's Docker registry was migrated from Dockerhub to GHCR! All new tags will be now pushed here. You can still access to old tags on the legacy Dockerhub repository.

Table of content

What is it?

Terraboard is a web dashboard to visualize and query Terraform states. It currently features:

  • an overview page listing the most recently updated state files with their activity
  • a state page with state file details, including versions and resource attributes
  • a search interface to query resources by type, name or attributes
  • a diff interface to compare state between versions

It currently supports several remote state backend providers:

Terraboard is now able to handle multiple buckets/providers configuration! ๐Ÿฅณ Check configuration section for more details.

Overview

The overview presents all the state files in the S3 bucket, by most recent modification date.

Screenshot Overview

Search

The search view allows to find resources by various criteria.

Screenshot Search

State

The state view presents details of a Terraform state at a given version.

Screenshot State

Compare

From the state view, you can compare the current state version with another version.

Screenshot Compare

Requirements

Independently of the location of your statefiles, Terraboard needs to store an internal version of its dataset. For this purpose it requires a PostgreSQL database. Data resiliency is not paramount though as this dataset can be rebuilt upon your statefiles at anytime.

AWS S3 (state) + DynamoDB (lock)

  • A versioned S3 bucket name with one or more Terraform states, named with a .tfstate suffix
  • AWS credentials with the following IAM permissions over the bucket:
    • s3:GetObject
    • s3:ListBucket
    • s3:ListBucketVersions
    • s3:GetObjectVersion
  • If you want to retrieve lock states from a dynamoDB table, you need to make sure the provided AWS credentials have dynamodb:Scan access to that table.

Terraform Cloud

  • Account on Terraform Cloud
  • Existing organization
  • Token assigned to an organization

Configuration

Terraboard currently supports configuration in three different ways:

  1. Environment variables (only usable for single provider configuration)
  2. CLI parameters (only usable for single provider configuration)
  3. Configuration file (YAML). A configuration file example can be found in the root directory of this repository and in the test/ subdirectory.

Important: all flags/environment variables related to the providers settings aren't compatible with multi-provider configuration! Instead, you must use the YAML config file to be able to configure multiples buckets/providers. YAML config is able to load values from environments variables.

The precedence of configurations is as described below.

Multiple buckets/providers

In order for Terraboard to import states from multiples buckets or even providers, you must use the YAML configuration method:

  • Set the CONFIG_FILE environment variable or the -c/--config-file flag to point to a valid YAML config file.
  • In the YAML file, specify your desired providers configuration. For example with two MinIO buckets (using the AWS provider with compatible mode):
# Needed since MinIO doesn't support versioning or locking
provider:
  no-locks: true
  no-versioning: true

aws:
  - endpoint: http://minio:9000/
    region: ${AWS_DEFAULT_REGION}
    s3:
      - bucket: test-bucket
        force-path-style: true
        file-extension:
          - .tfstate

  - endpoint: http://minio:9000/
    region: eu-west-1
    s3:
      - bucket: test-bucket2
        force-path-style: true
        file-extension:
          - .tfstate

In the case of AWS, don't forget to set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.

That's it! Terraboard will now fetch these two buckets on DB refresh. You can also mix providers like AWS and Gitlab or anything else. You can find a ready-to-use Docker example with two MinIO buckets in the test/multiple-minio-buckets/ sub-folder.

Available parameters

Application Options

  • -V, --version Display version.
  • -c, --config-file <default: $CONFIG_FILE> Config File path
    • Env: CONFIG_FILE

General Provider Options

  • --no-versioning <default: $TERRABOARD_NO_VERSIONING> Disable versioning support from Terraboard (useful for S3 compatible providers like MinIO)
    • Env: TERRABOARD_NO_VERSIONING
    • Yaml: provider.no-versioning
  • --no-locks <default: $TERRABOARD_NO_LOCKS> Disable locks support from Terraboard (useful for S3 compatible providers like MinIO)
    • Env: TERRABOARD_NO_LOCKS
    • Yaml: provider.no-locks

Logging Options

  • -l, --log-level <default: "info"> Set log level ('debug', 'info', 'warn', 'error', 'fatal', 'panic').
    • Env: TERRABOARD_LOG_LEVEL
    • Yaml: log.level
  • --log-format <default: "plain"> Set log format ('plain', 'json').
    • Env: TERRABOARD_LOG_FORMAT
    • Yaml: log.format

Database Options

  • --db-host <default: "db"> Database host.
    • Env: DB_HOST
    • Yaml: database.host
  • --db-port <default: "5432"> Database port.
    • Env: DB_PORT
    • Yaml: database.port
  • --db-user <default: "gorm"> Database user.
    • Env: DB_USER
    • Yaml: database.user
  • --db-password <default: $DB_PASSWORD> Database password.
    • Env: DB_PASSWORD
    • Yaml: database.password
  • --db-name <default: "gorm"> Database name.
    • Env: DB_NAME
    • Yaml: database.name
  • --db-sslmode <default: "require"> Database SSL mode.
    • Env: DB_SSLMODE
    • Yaml: database.sslmode
  • --no-sync Do not sync database.
    • Yaml: database.no-sync
  • --sync-interval <default: "1"> DB sync interval (in minutes)
    • Yaml: database.sync-interval

AWS (and S3 compatible providers) Options

  • --aws-access-key <default: $AWS_ACCESS_KEY_ID> AWS account access key.
    • Env: AWS_ACCESS_KEY_ID
    • Yaml: aws.access-key
  • --aws-secret-access-key <default: $AWS_SECRET_ACCESS_KEY> AWS secret account access key.
    • Env: AWS_SECRET_ACCESS_KEY
    • Yaml: aws.secret-access-key
  • --aws-session-token <default: $AWS_SESSION_TOKEN> AWS session token.
    • Env: AWS_SESSION_TOKEN
    • Yaml: aws.session-token
  • --dynamodb-table <default: $AWS_DYNAMODB_TABLE> AWS DynamoDB table for locks.
    • Env: AWS_DYNAMODB_TABLE
    • Yaml: aws.dynamodb-table
  • --aws-endpoint <default: $AWS_ENDPOINT> AWS endpoint.
    • Env: AWS_ENDPOINT
    • Yaml: aws.endpoint
  • --aws-region <default: $AWS_REGION> AWS region.
    • Env: AWS_REGION
    • Yaml: aws.region
  • --aws-role-arn <default: $APP_ROLE_ARN> Role ARN to Assume.
    • Env: APP_ROLE_ARN
    • Yaml: aws.app-role-arn
  • --aws-external-id <default: $AWS_EXTERNAL_ID> External ID to use when assuming role.
    • Env: AWS_EXTERNAL_ID
    • Yaml: aws.external-id

S3 Options

  • --s3-bucket <default: $AWS_BUCKET> AWS S3 bucket.
    • Env: AWS_BUCKET
    • Yaml: aws.s3.bucket
  • --key-prefix <default: $AWS_KEY_PREFIX> AWS Key Prefix.
    • Env: AWS_KEY_PREFIX
    • Yaml: aws.s3.key-prefix
  • --file-extension <default: ".tfstate"> File extension(s) of state files.
    • Env: AWS_FILE_EXTENSION
    • Yaml: aws.s3.file-extension
  • --force-path-style <default: $AWS_FORCE_PATH_STYLE> Force path style S3 bucket calls.
    • Env: AWS_FORCE_PATH_STYLE
    • Yaml: aws.s3.force-path-style

Terraform Enterprise Options

  • --tfe-address <default: $TFE_ADDRESS> Terraform Enterprise address for states access
    • Env: TFE_ADDRESS
    • Yaml: tfe.address
  • --tfe-token <default: $TFE_TOKEN> Terraform Enterprise Token for states access
    • Env: TFE_TOKEN
    • Yaml: tfe.token
  • --tfe-organization <default: $TFE_ORGANIZATION> Terraform Enterprise organization for states access
    • Env: TFE_ORGANIZATION
    • Yaml: tfe.organization

Google Cloud Platform Options

  • --gcs-bucket Google Cloud bucket to search
    • Yaml: gcp.gcs-bucket
  • --gcp-sa-key-path <default: $GCP_SA_KEY_PATH> The path to the service account to use to connect to Google Cloud Platform
    • Env: GCP_SA_KEY_PATH
    • Yaml: gcp.gcp-sa-key-path

GitLab Options

  • --gitlab-address <default: *"https://gitlab.com"*> GitLab address (root)
    • Env: GITLAB_ADDRESS
    • Yaml: gitlab.address
  • --gitlab-token <default: $GITLAB_TOKEN> Token to authenticate upon GitLab
    • Env: GITLAB_TOKEN
    • Yaml: gitlab.token

Web

  • -p, --port <default: "8080"> Port to listen on.
    • Env: TERRABOARD_PORT
    • Yaml: web.port
  • --base-url <default: "/"> Base URL.
    • Env: TERRABOARD_BASE_URL
    • Yaml: web.base-url
  • --logout-url <default: $TERRABOARD_LOGOUT_URL> Logout URL.
    • Env: TERRABOARD_LOGOUT_URL
    • Yaml: web.logout-url

Help Options

  • -h, --help Show this help message

Push plans to Terraboard

In order to send Terraform plans to Terraboard, you must wrap it in this JSON format:

{
    "lineage": "<Plan's lineage>",
    "terraform_version": "<Terraform version>",
    "git_remote": "<The URL of the remote that generated this plan>",
    "git_commit": "<Commit hash>",
    "ci_url": "<The URL of the CI that sent this plan>",
    "source": "<Free field for the triggering event>",
    "plan_json": "<Terraform plan JSON export>"
}

And send it to /api/plans using POST method

Use with Docker

Docker-compose

Configuration file can be provided to the container using a volume or a configuration.

# Set AWS credentials as environment variables:
export AWS_ACCESS_KEY_ID=<access_key>
export AWS_SECRET_ACCESS_KEY=<access_secret>

# Set AWS configuration as environment variables:
export AWS_DEFAULT_REGION=<AWS default region>
export AWS_BUCKET=<S3 Bucket name>
export AWS_DYNAMODB_TABLE=<Aws DynamoDB Table>

docker-compose up

Then point your browser to http://localhost:8080.

Docker command line

# Set AWS credentials as environment variables:
export AWS_ACCESS_KEY_ID=<access_key>
export AWS_SECRET_ACCESS_KEY=<access_secret>

# Set AWS configuration as environment variables:
export AWS_DEFAULT_REGION=<AWS default region>
export AWS_BUCKET=<S3 Bucket name>
export AWS_DYNAMODB_TABLE=<AWS_DYNAMODB_TABLE>

# Spin up the two containers and a network for them to communciate on:
docker network create terraboard
docker run --name db \
  -e POSTGRES_USER=gorm \
  -e POSTGRES_DB=gorm \
  -e POSTGRES_PASSWORD="<mypassword>" \
  -e GODEBUG="netdns=go" \
  --net terraboard \
  --detach \
  --restart=always \
  postgres:9.5

docker run -p 8080:8080 \
  -e AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}" \
  -e AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}" \
  -e AWS_REGION="${AWS_DEFAULT_REGION}" \
  -e AWS_BUCKET="${AWS_BUCKET}" \
  -e AWS_DYNAMODB_TABLE="${AWS_DYNAMODB_TABLE}" \
  -e DB_PASSWORD="<mypassword>" \
  -e DB_SSLMODE="disable" \
  --net terraboard \
  ghcr.io/camptocamp/terraboard:latest

Then point your browser to http://localhost:8080.

Use with Kubernetes

A Helm chart is available on Camptocamp's repository.

In order to install it:

$ helm repo add c2c https://camptocamp.github.io/charts
$ helm install -v values.yaml terraboard c2c/terraboard

Use with Rancher

Camptocamp's Rancher Catalog contains a Terraboard template to automate its installation in Cattle.

Authentication and base URL

Terraboard does not implement authentication. Instead, it is recommended to use an authentication proxy such as oauth2_proxy.

If you need to set a route path for Terraboard, you can set a base URL by passing it as the BASE_URL environment variable.

When using an authentication proxy, Terraboard will retrieve the logged in user and email from the headers passed by the proxy. Terraboard expects you to setup the HTTP Headers X-Forwarded-User and X-Forwarded-Email when passing the logged in user and email. A Nginx example can be found below:

location / {
  ....
  auth_request_set $user   $upstream_http_x_auth_request_user;
  auth_request_set $email  $upstream_http_x_auth_request_email;
  proxy_set_header X-Forwarded-User  $user;
  proxy_set_header X-Forwarded-Email $email;
  ...
  proxy_pass http://terraboard/;
}

You can also pass a TERRABOARD_LOGOUT_URL parameter to allow users to sign out of the proxy.

Install from source

$ go get github.com/camptocamp/terraboard

Compatibility Matrix

Terraboard Max Terraform version
0.15.0 0.12.7
0.16.0 0.12.7
0.17.0 0.12.18
0.18.0 0.12.18
0.19.0 0.12.20
0.20.0 0.12.26
0.21.0 0.12.28
0.22.0 0.13.0
1.0.0 0.14.5
1.1.0 0.14.10

Development

Architecture

Terraboard is made of two components:

A server process

The server is written in go and runs a web server which serves:

  • the API on known access points, taking the data from the PostgreSQL database
  • the index page (from static/index.html) on all other URLs

The server also has a routine which regularly (every 1 minute) feeds the PostgreSQL database from the S3 bucket.

A web UI

The UI is an AngularJS application served from index.html. All the UI code can be found in the static/ directory.

Testing

$ docker-compose build && docker-compose up -d
# Point your browser to http://localhost

Contributing

See CONTRIBUTING.md

terraboard's People

Contributors

alemuro avatar andrescidoncha avatar binlab avatar cryptobioz avatar fafalafafa avatar filiptepper avatar giaco9 avatar github-actions[bot] avatar gliptak avatar hbollon avatar headincl0ud avatar hoshsadiq avatar kwerey avatar leonidcliqz avatar lorodoes avatar mcanevet avatar mhaley-miovision avatar mihaiplesa avatar moglum avatar mvisonneau avatar ouranos avatar raphink avatar renato-farias avatar rickhlx avatar slitsevych avatar tedder avatar tristanvanthielen avatar uritau avatar vampouille avatar wiston999 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

terraboard's Issues

Disable versioning

It's a cool tool and I'd like to know if it is possible to disable S3 versioning scan?
Because our tfstate file has many many version on S3 and I only like to see the latest version on the chart.

Panic at start with 0.22.0

Hello!

I just attempted an update to 0.22.0 and received a panic on startup:

time="2020-08-14T19:51:13Z" level=info msg="Terraboard v0.22.0 (built for Terraform v0.13.0) is starting..."
panic: terraform.io/builtin/terraform is not a legacy addrs.Provider

Has 0.22.0 dropped support for state file versions older than 0.13.0? This issue describes a similar error which may have been fixed in newer versions of the module? hashicorp/terraform#25803

replace S3 and DynamoDB with opensource counterparts?

Hi there!
I was going through the documentation, and I had the feeling that we're getting back to those days, when Microsoft was dictating his rules.
S3, DynamoDB are proprietary and closed source technologies. AWS has never defined a RfC, or tried to create standards or try to reveal their source code and to me it's clear, that if we deal with opensource, we must promote opensource.
Furthermore, AWS is not standard de-facto: I can be on premises, I can have google cloud, and even if, in my case I have AWS (and I really have it) I still don't understand the reasons behind this decision.
Why not using Consul for the backend (which support the session locking)? Why not try something else for the versioning?

simple install issues

I am new to Docker and have tried using the docker compose and using the manual commands in GitHub. I tried local and on AWS.

[root@ip-10-30-8-91 terraboard]# docker-compose build && docker-compose up -d
WARNING: The AWS_ACCESS_KEY_ID variable is not set. Defaulting to a blank string.
WARNING: The AWS_SECRET_ACCESS_KEY variable is not set. Defaulting to a blank string.
WARNING: The AWS_DEFAULT_REGION variable is not set. Defaulting to a blank string.
WARNING: The AWS_BUCKET variable is not set. Defaulting to a blank string.
WARNING: The AWS_FILE_EXTENSION variable is not set. Defaulting to a blank string.
WARNING: The AWS_DYNAMODB_TABLE variable is not set. Defaulting to a blank string.
WARNING: The AWS_KEY_PREFIX variable is not set. Defaulting to a blank string.
WARNING: The TERRABOARD_LOG_LEVEL variable is not set. Defaulting to a blank string.
WARNING: The OAUTH_CLIENT_ID variable is not set. Defaulting to a blank string.
WARNING: The OAUTH_CLIENT_SECRET variable is not set. Defaulting to a blank string.
WARNING: The OAUTH_COOKIE_SECRET variable is not set. Defaulting to a blank string.
proxy uses an image, skipping
db uses an image, skipping
Building terraboard
Step 1/12 : FROM golang:1.11 as builder
---> 43a154fee764
Step 2/12 : WORKDIR /go/src/github.com/camptocamp/terraboard
---> Using cache
---> 50f00fdf843d
Step 3/12 : COPY . .
---> Using cache
---> ada28d56f340
Step 4/12 : RUN go get -u github.com/golang/dep/cmd/dep && dep ensure
---> Using cache
---> 06ca804aa2b6
Step 5/12 : RUN make terraboard
---> Using cache
---> 89c25b69d717
Step 6/12 : FROM scratch
--->
Step 7/12 : WORKDIR /
---> Using cache
---> def89a600e59
Step 8/12 : COPY static /static
---> Using cache
---> 207da17b168c
Step 9/12 : COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
---> Using cache
---> 61ce997c2945
Step 10/12 : COPY --from=builder /go/src/github.com/camptocamp/terraboard/terraboard /
---> Using cache
---> da20a6fdeaeb
Step 11/12 : ENTRYPOINT ["/terraboard"]
---> Using cache
---> 89d59becd60b
Step 12/12 : CMD [""]
---> Using cache
---> 078397b52bba
Successfully built 078397b52bba
Successfully tagged camptocamp/terraboard:devel
WARNING: The AWS_ACCESS_KEY_ID variable is not set. Defaulting to a blank string.
WARNING: The AWS_SECRET_ACCESS_KEY variable is not set. Defaulting to a blank string.
WARNING: The AWS_DEFAULT_REGION variable is not set. Defaulting to a blank string.
WARNING: The AWS_BUCKET variable is not set. Defaulting to a blank string.
WARNING: The AWS_FILE_EXTENSION variable is not set. Defaulting to a blank string.
WARNING: The AWS_DYNAMODB_TABLE variable is not set. Defaulting to a blank string.
WARNING: The AWS_KEY_PREFIX variable is not set. Defaulting to a blank string.
WARNING: The TERRABOARD_LOG_LEVEL variable is not set. Defaulting to a blank string.
WARNING: The OAUTH_CLIENT_ID variable is not set. Defaulting to a blank string.
WARNING: The OAUTH_CLIENT_SECRET variable is not set. Defaulting to a blank string.
WARNING: The OAUTH_COOKIE_SECRET variable is not set. Defaulting to a blank string.
Starting terraboard_proxy_1 ...
Starting terraboard_proxy_1 ... done
Starting terraboard_terraboard_1 ... done

Terraboard container issue

while running Terraboard container It showing "level=fatal msg="pq: SSL is not enabled on the server""
Please do the needful.

Error when doing a go get

Fri Sep 15:~/terraboard:~go get -u -v github.com/camptocamp/terraboard

../src/src/github.com/hashicorp/terraform/config/testing.go:9: t.Helper undefined (type *testing.T has no field or method Helper)

I went inside the source and did a 'go build' and could see similar issues

src/github.com/camptocamp/terraboard:~go build .
# github.com/hashicorp/terraform/config
../../../src/github.com/hashicorp/terraform/config/testing.go:9: t.Helper undefined (type *testing.T has no field or method Helper)

Can you please help me with this.

panic: runtime error: invalid memory address or nil pointer dereference

Running Terraboard in containers, and when I get to a specific state file, the terraboard container will exit this error.

I've tried relaunching the terraboard container and it will fail as it tries to import the same state file. I've also tried removing the db container and starting fresh, and it will fail again on the same state file.

time="2020-04-30T19:52:15Z" level=info msg="Retrieving state from S3" path="path/terraform.tfstate" version_id=id
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xc8db68]

goroutine 34 [running]:
github.com/camptocamp/terraboard/db.marshalAttributeValues(0x0, 0xc0003cf140, 0xc0004e7610, 0x0)
        /go/src/github.com/camptocamp/terraboard/db/db.go:82 +0x78
github.com/camptocamp/terraboard/db.(*Database).stateS3toDB(0xc000368180, 0xc0003cea80, 0xc000027f80, 0x3b, 0xc000358900, 0x20, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/camptocamp/terraboard/db/db.go:70 +0x23a
github.com/camptocamp/terraboard/db.(*Database).InsertState(0xc000368180, 0xc000027f80, 0x3b, 0xc000358900, 0x20, 0xc0003cea80, 0x0, 0x0)
        /go/src/github.com/camptocamp/terraboard/db/db.go:114 +0xa1
main.refreshDB(0x1, 0xc000368180, 0x11f0220, 0xc0001d70e0)
        /go/src/github.com/camptocamp/terraboard/main.go:105 +0xb6b
created by main.main
        /go/src/github.com/camptocamp/terraboard/main.go:164 +0x1c12

It would be nice if an import error occurs, skip the state file and keep importing.

Feat: Do not show sensitive values in UI

There are many cases where the output of a resource (or an input for that matter) will be marked as sensitive. It would be nice if terraboard was able to recognize this and not display those sensitive values in the UI.

I have opened up hashicorp/terraform#25646 to address the issue of this information not being in the state file.

Terraform v0.12.26

Are there plans to create a release that is compatible with Terraform v0.12.26?

View for outputs

It would be very good if terraform outputs also has they own view. I see that the contributor is no longer develop this beautiful tool but maybe some one else can do this.
Thanks a lot.

Use S3 compatible bucket

Hi,

First of all thank for developing terraboard, it's a really nice tool!

We are using our own S3 compatible system and currently I think can't change the address with AWS_BUCKET environment variable.

I would like to be able to use AWS_BUCKET=s3://s3.offline.net/bucket_xxx. Maybe I missed something but I think AWS addresses are deeply linked to aws-sdk-go.

Index creation issue on blocking parsing of some statefiles

On a specific statefile that my bucket contains, I can see the following errors being thrown by postgres:

STATEMENT:  INSERT INTO "attributes" ("resource_id","key","value") VALUES ($1,$2,$3) RETURNING "attributes"."id"
ERROR:  index row size 5048 exceeds maximum 2712 for index "idx_attributes_value"
HINT:  Values larger than 1/3 of a buffer page cannot be indexed.
	Consider a function index of an MD5 hash of the value, or use full text indexing.
STATEMENT:  INSERT INTO "attributes" ("resource_id","key","value") VALUES ($1,$2,$3) RETURNING "attributes"."id"
ERROR:  index row size 4984 exceeds maximum 2712 for index "idx_attributes_value"
HINT:  Values larger than 1/3 of a buffer page cannot be indexed.
	Consider a function index of an MD5 hash of the value, or use full text indexing.
STATEMENT:  INSERT INTO "attributes" ("resource_id","key","value") VALUES ($1,$2,$3) RETURNING "attributes"."id"
ERROR:  index row size 5048 exceeds maximum 2712 for index "idx_attributes_value"

Had a look into the config and the code but couldn't figure out a fix yet.

Feat: Display map type attributes in an easier to read format

As it stands in the UI today map values appear as a single line in the table of values.
image

This is not an easy to read format for large maps of values. We should try to "pretty print" these values, or parse the maps and present them in the table as separate values. For example, using the screenshot above as a reference, we could have

Attribute Value
id "8591566720176406857"
triggers.index 4
triggers.tf_stack_name "terraboard_develop"
triggers.value1 "val 1"
triggers.value2 "val 2"
triggers.value3 "val 3"

Make sparklines clickable

Fix display of values in sparklines, displaying date & resource count, and allow to click points, linking to the state page on given version.

panic: runtime error: invalid memory address or nil pointer dereference

running Terraboard on my macOS using docker 2.3.0.4
After a while scarping data from terraform.state files I receive the following error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xec9778]

goroutine 28 [running]:
github.com/camptocamp/terraboard/db.marshalAttributeValues(0x0, 0xc0005bc9f0, 0xc00052d5f0, 0x2)
/go/src/github.com/camptocamp/terraboard/db/db.go:83 +0x78
github.com/camptocamp/terraboard/db.(*Database).stateS3toDB(0xc0000c4190, 0xc0005bc600, 0xc0000a2960, 0x51, 0xc00064d1c0, 0x20, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/camptocamp/terraboard/db/db.go:71 +0x25d
github.com/camptocamp/terraboard/db.(*Database).InsertState(0xc0000c4190, 0xc0000a2960, 0x51, 0xc00064d1c0, 0x20, 0xc0005bc600, 0x0, 0x0)
/go/src/github.com/camptocamp/terraboard/db/db.go:115 +0xa1
main.refreshDB(0x1, 0xc0000c4190, 0x156e460, 0xc0001e7d60)
/go/src/github.com/camptocamp/terraboard/main.go:105 +0xb6b
created by main.main
/go/src/github.com/camptocamp/terraboard/main.go:164 +0x1c2f

Display outputs for state file

I understand that the dashboard can show attributes of a tfstate file, but is it possible to show outputs as well?
Sometimes we have customized outputs and it will be great if I can see output from the dashboard

Feat: Show difference between resource vs data references

There are data references and resource blocks that take up most of the space in state_files. A resource will have "mode": "manged" and a data reference will have "mode": "data" in the state file. This information should be exposed through the UI to give users more context on what they are looking at. For example, right now it is impossible to tell if the aws_vpc that is defined in our one state file is just referencing (via the data object) a pre-made VPC or if the terraform scripts are managing that vpc through a resource declaration. Both appear the same in terraboard.

Can't get terraboard running locally (without docker)

So I run go get https://github.com/camptocamp/terraboard
and the go install . compiled and installed it.

With following env variables set

AWS_SECRET_ACCESS_KEY=1xxxxxx
AWS_ACCESS_KEY_ID=AKIAJMCDDREEFEEDE
AWS_REGION=ap-southeast-2
AWS_BUCKET=blah-tfstates
ฮป terraboard                                                             
INFO[0000] Automigrate
INFO[0000] New db is &{<nil> <nil> 0 0xc42024c3c0 false 2 {0xc42022c910} <nil> map[] 0xc42013f830 0x4c25160 0xc42025a560 false}
INFO[0000] Refreshing DB from S3

But running terraboard only create a sqlite db then quit.

What have I done wrong?

Allow filtering and smart sorting in overview

Add a filter on path and allow sorting by column on all results.

This means:

  • adding a path query parameter to the api/states/stats point, which will filter on queries on '%%'
  • adding orderby and order query parameters to the api/states/stats point, which will order by the field (default to last_modified and desc respectively)
  • add a filter field on top of the results to enter a path
  • have the UI refresh the results when the filter field or sort column are changed

Notes: this should fix #4

panic: runtime error: db.marshalAttributeValues

This is the same error as #90, but .. I can provide information and not close the ticket. The only thing I masked in the error was our s3 path. Note we're using terraform 12.24.

2020-06-29T12:17:48.896000 {"level":"info","msg":"Retrieving state from S3","path":"our/file","time":"2020-06-29T19:17:48Z","version_id":"0z7Dy7Qpm4SILXmTf5nxO4TJLeuL7RUt"}
2020-06-29T12:17:48.910000 panic: runtime error: invalid memory address or nil pointer dereference
2020-06-29T12:17:48.910000 [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xc8db68]
2020-06-29T12:17:48.910000 goroutine 25 [running]:
2020-06-29T12:17:48.910000 github.com/camptocamp/terraboard/db.marshalAttributeValues(0x0, 0xc00098ab70, 0xc000111610, 0x0)
2020-06-29T12:17:48.910000  /go/src/github.com/camptocamp/terraboard/db/db.go:82 +0x78
2020-06-29T12:17:48.910000 github.com/camptocamp/terraboard/db.(*Database).stateS3toDB(0xc00009f5c8, 0xc00098aa50, 0xc0002acc60, 0x14, 0xc000b2e840, 0x20, 0x0, 0x0, 0x0, 0x0, ...)
2020-06-29T12:17:48.910000  /go/src/github.com/camptocamp/terraboard/db/db.go:70 +0x23a
2020-06-29T12:17:48.910000 github.com/camptocamp/terraboard/db.(*Database).InsertState(0xc00009f5c8, 0xc0002acc60, 0x14, 0xc000b2e840, 0x20, 0xc00098aa50, 0x0, 0x0)
2020-06-29T12:17:48.910000  /go/src/github.com/camptocamp/terraboard/db/db.go:114 +0xa1
2020-06-29T12:17:48.910000 main.refreshDB(0x1, 0xc00009f5c8, 0x11f0220, 0xc0001c1130)
2020-06-29T12:17:48.910000  /go/src/github.com/camptocamp/terraboard/main.go:105 +0xb6b
2020-06-29T12:17:48.910000 created by main.main
2020-06-29T12:17:48.910000  /go/src/github.com/camptocamp/terraboard/main.go:164 +0x1c12

Pulling up that last-logged state file, it is linked to version v0.12.24. Here's the resources and types found in the 1100+ line file:

   1             "type": "AWS",
   1       "type": "aws_api_gateway_account",
   1       "type": "aws_api_gateway_deployment",
   1       "type": "aws_api_gateway_integration",
   1       "type": "aws_api_gateway_integration_response",
   1       "type": "aws_api_gateway_method",
   1       "type": "aws_api_gateway_method_response",
   1       "type": "aws_api_gateway_method_settings",
   1       "type": "aws_api_gateway_resource",
   1       "type": "aws_api_gateway_rest_api",
   1       "type": "aws_caller_identity",
   2       "type": "aws_cloudwatch_log_group",
   1       "type": "aws_iam_account_alias",
   2       "type": "aws_iam_group",
   2       "type": "aws_iam_group_membership",
   2       "type": "aws_iam_group_policy_attachment",
   6       "type": "aws_iam_policy",
   4       "type": "aws_iam_policy_document",
   2       "type": "aws_iam_role",
   5       "type": "aws_iam_role_policy_attachment",
   1       "type": "aws_iam_user",
   1       "type": "aws_kms_alias",
   1       "type": "aws_kms_key",
   1       "type": "aws_lambda_function",
   1       "type": "aws_lambda_permission",
   1       "type": "string"

It's been barfing for a few weeks; we were previously using a dockerhub build sha that supported Terraform 12.20 (and it was failing), but today to fix the panics we updated to terraboard v0.20. It's still failing.

We deleted that one specific state file and things successfully starting up. I'm hoping these clues help to find the cause of the panic. I can provide specific snippets from the state file if you have suspicions about what caused it, but I don't want to have to scrub the entire state file.

Thanks.

Terraform version in changelog

I have noticed that Terraboard can't read some of my tfstate files in my bucket. Logs are reporting that xxx.tfstate file has been created with a newer Terraform version that Terraboard can't support yet.
It would be nice just to have some indications when putting a new image on DockerHub, saying what is the latest supported Terraform version. Thanks

Using terraform with workspaces, not seeing the states.

When running a terraform workspaces, "env:/workspace" folders gets created in the s3, however terraboard doesn't pick that up, even when empty AWS_KEY_PREFIX and AWS_FILE_EXTENSION.

The workaround that seems to do it for me was to comment out the if in the GetStates() of s3.go

for _, obj := range result.Contents {
//		if strings.HasSuffix(*obj.Key, ".tfstate") {
			keys = append(keys, *obj.Key)
//		}
	}

and rebuild, then all the possible folders and states files (regardless of extension) show up in the gui - which is the behaviour I needed. Not sure why or how - my go knowledge and time to debug is limited. However since terraboard is an awesome tool (big thanks), thought I shared my finding.

Add a compare view

Add a compare view to compare two versions of a given path.

This means adding an api/compare/<path>?from=<v1>&to=<v2> API point returning a JSON of the processed comparison, and displaying the result in a new compare/<path>?from=<v1>&to=<v2> page.

Use docker networks instead of deprecated --link.

Hi folks,

This is an awesome project!! Thanks for putting it out there.

Some quick feedback on getting the docker image running locally: I had issues getting a link to a postgres container working on Docker for Mac (17.12.0-ce-mac49). It might just be that my VPN setup wasn't playing nice with it, but Docker strongly recommends against link nowadays, and using a shared network is simple and clean to set up.

Instead, I did something along these lines and it worked a treat:

docker network create terranet
docker run --name db \
  -e POSTGRES_USER=gorm \
  -e POSTGRES_DB=gorm \
  -e POSTGRES_PASSWORD="<mypassword>" \
   --net terranet \
  --restart=always postgres -d
docker run -p 8080:8080 \
 -e AWS_REGION="us-east-1" \
 -e AWS_ACCESS_KEY_ID="${ACCESS_KEY_ID}" \
 -e AWS_SECRET_ACCESS_KEY="${ACCESS_KEY}" \
 -e AWS_BUCKET="<bucket>" \
 -e AWS_DYNAMODB_TABLE="<table>" \
 -e DB_PASSWORD="<mypassword>" \
 --net terranet \
 camptocamp/terraboard:latest

Compatibility Issue with Terraform v0.12.0

Once I upgrade Terraform version to v0.12.0(provider.aws v2.10.0) and "terraform apply" to update state files, then state files disappear in the main dashboard page. Terraborad still can detect that those state files are v0.12.0 but won't display resources within the files.
Could someone suggest, please?

Local state files?

Hello, this project looks great. Are there any plans to support local state files?

pq sslmode

I've just tried to use latest version of Docker image (0.19.0) but I just can't connect to postgres:

level=fatal msg="pq: unsupported sslmode "prefer"; only "require" (default), "verify-full", "verify-ca", and "disable" supported"

Same behaviour with 0.18.0.

I've found a "sslmode" version on DockerHub and it's working fine.

Is it a regression? Or, can you point out how to manage sslmode on newer image versions?

Terraboard unable to DNS look up DB on ECS

What I'm trying to do

Using ECS Fargate, I'm trying to run Terraboard and the Postgres DB in a single ECS task (two side by side containers). Terraboard fails to start since it cannot find the postgres db.

What is failing

Terraboard attempts to dns lookup 'db' and fails since the postgres db isn't registered in the DNS, its not able to be looked up.

level=fatal msg="dial tcp: lookup db on 172.16.0.2:53: no such host"

Currently with ECS Fargate

Expected behavior

Terraboard should try looking up localhost for the postgres db before fatally ending.

OR

There should be an ability to use hosted postgres (RDS) or a separate container/task.

v0.21.0 can't connect to RDS db when running on AWS ECS

background:
Running postgres on an RDS instance
Trying to run terraboard on ECS (EC2 instance)

When I try to run terraboard's docker image on AWS ECS I get an error on startup that the RDS DB instance is busy

level=info msg="Terraboard v0.21.0 (built for Terraform v0.12.28) is starting..."
level=info msg="Using AWS (S3+DynamoDB) as the state/locks provider"
level=fatal msg="dial tcp: lookup terraboard.abcd1234.us-east-1.rds.amazonaws.com: device or resource busy"

I confirmed all my configs/permissions are correct by running the container locally with no issues, also confirmed that the ECS instance the container runs on is able to connect to the RDS instance directly so that end is setup correctly as well.

Does not work with AWS cross-account assumed role

Hi.

I have my cli configured with its token id and secret key. It has permission to assume a role in another AWS account, where the s3 bucket and dynamo table are located.

How do I configure terraboard for this setup? Simply passing AWS_SECRET_ACCESS_KEY and
AWS_ACCESS_KEY_ID of the linked account to the terraboard image does not work of course.

The goal here is to have one terraboard showcasing multiplestate files from multiple aws accounts. That would require setting multiple dynamodb tables. Is this possible?

Thanks in advance.

Get lock status

Get lock status for statefiles using DynamoDB when available.

Don't know how to install & run

Hi,

I'm sorry to say this - I tried all day long and couldn't figure out how to install the software and make it work.

Tried with go and docker - still, no luck.

Thanks.

Using terraboard on local environment

Can I use terraboard on local server without AWS infrastructure?

I trying to create local environment like:
Minio (instead S3)
MongoDB (instead DocumentDB)
PostgreSQL (instead RDS PSQL)

API of those services are pretty close to original ones.

  1. How can I override S3 endpoint URL and port (for example http://127.0.0.1:1234/bucket_name)
  2. How can I set MongoDB URL and port (127.0.0.1:27017)
  3. Looks like Terraform Cloud is not necessary requirement if I plan to use only local resources?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.