HashiCorp Application Workflow over Technology

This repository is used to demonstrate each of HashiCorp's key technologies and where they are frequently used throughout a typical application workflow.

About the app

A simple web-page-counter application with the backend developed in Golang and the frontend developed in Angular. No programmers were harmed in the writting of this application, I hacked it together myself (be gentle).

Backend APIs are available on port 9090 and the frontend Angular application is served from port 9091.

Angular frontend code

In brief, Vagrant is used for initial development and testing of the application. This is integrated with Travis-CI to ensure the scripts are valid, not all scripts are idempotent yet, sorry, will get there eventually. There's an example Packer template file that's used to package everything needed for both Virtualbox images and Azure images. An monolithic terraform template file is provided to demonstrate how the application and infrastructure can be provisioned into Azure's public cloud. This will be broken down into modules and follow best practices in future iterations of this repository. The application is deployed using Nomad's exec driver and queries Consul to discover the details of it's Redis database and then Vault provides access to the secret material. Another simple bootstraping application is leveraged to show how applications, virtual machines or containers could securely receive secret material. Consul's service mesh capability is also leveraged to secure access to the Redis database.

Detailed Overview

Vagrant Setup Development Environment Locally
Packer Package the Artefacts for Virtualbox and Azure
Terraform Setup Development Environment on Azure
Travis-CI Continuous Integration Testing and Releases
Consul
Vault
- Secret Management
- Consul-Template
Nomad

DataDog

When all is good it should look like this

When we envoke a failure the self healing kicks in

TODO

Tidy up Terraform Cloud deployment - move to certificate authentication based bastion and local hosts
Review possibility of integrating Consul Connect Layer 7 demos
Start building out documentation for workshops
add SSH CA public key to image and remove insecure build ssh user
possibly enable pki engine on vault and secure frontend of nginx service
test out L7 envoy features
create packer images for Azure, GCP and AWS
build terraform monolith for Azure, GCP & AWS
transform terraform monolith to Good IAC examples using HC best practices
enable Nomad and Consul Connect Integration
Secure NGINX to Application also using Connect

New Features

Write up note on new web front end (just as a reminder for myself - lots of CORS challenges)
Write up details on terraform code and improve documentation in general

Refactor

Done

Build own box using packer with above scripts
Upload to vagrantcloud
Update Vagrant file to consume the new box
Add a Consul(1) server
Remove scripts from Packer and link back to repo master scripts, use SCRIPTS feature
Add 'manual' test scripts to TravisCI
- source env file
- over write variables for local redis
- use curl -s to verify app returns http 200 & exit 0
- use lynx --dump to capture counter updates between refreshes
- Add TravisCI for Go APP
scripted added to read in var.env and upload to consul
main.go modified to use consul for variables when it's present
Add a Vault (1) server
if golang app can't reach redis or any other error return a zero
if consul is not up - fallback to var.env values
update goapp to have a /health to provide the status of it's services (text only)
- GOOD
- NOTGOOD
Register Redis on Consul
What: Register the redis service on consul
Why: So the GOApp can connect dynamically (discover) the service
How: Check ping, write a dummy k/v, read the dummy k/v
Register Go App in Consul
What: Register the goapp service on consul
Why: So the webtier (NGINX) can connect dynamically (discover) the service
How: Check the website is up & the /health url returns GOOD
* update goapp consul healthcheck to expect GOOD/NOTGOOD
Make readme pretty (again)
Make goAPP use consul to find redis port and ip
Make nginx use consul to populate the conf file (consul-template)
3 IPs per godev machine main.go using ip1:8080, ip2:8080, ip3:8080
WRONG: Modify existing goapp to listen and respond on 3 ports -8081, 8082, 8083 (3 x gorountines) 3 separate main.go instances on the one server
If running on travis only create a single listener on port 8080
Update Consul service healthcheck to accomodate new changes
Update NGINX Consul-template to receive updates
Update Travis to new requirements
Repeat all of the above with a 2nd instance of the goapp server
Sign up for Datadog trial account using a gmail alias e.g. [email protected] becomes [email protected]
Install DDog Agent on the Macbook and test using Guages & Counters
Update application to submit datadog pagecount counter once redis has been successfully updated
Add datadog guage that reports on the number of available goapp services
Remove redis02 and vault servers - not required yet!
Create another Datadog Test Account for the 5 servers
Install datadog agent on all servers
Update consul-template nginx routine to include goapp service guage routine
Rollback to a single network ip on the vagrantfile
Don't execute the goapp upon installation - nomad will be used to do this later
Write a nomad job using the raw_exec driver (NOT DOCKER) to launch the application ONCE on any node
Add tag that includes port details to ddog metric
Add service checks/tests to nomad job
Move all logs to /vagrant/logs/
Add UI ability to terminate current go-app service - facilitate quicker nomad demo - either add button to main page or create new endpoint feature
Add github templates
Add Travis Deployment on Tags
Update App deployment to use releases
Backout the unauthorised architectural switch from Vault v1 Secrets to v2 Secrets (versioned)
UNDO :Move Redis Password into VAULT KV - demonstrates best practice of using centralised secret management vault - time 4 hrs
modify vault installation to make binary accessable on all nodes
install envconsul on all nodes
auto-generate redis password on leader node and store in vault using vault client (binary)
update redis installation to utilise consul-template to get redis password from vault
revert main.go application to consume vault v1 secrets (unversioned)
Add Vault App-Role to Application__ for Vault Access Method - demonstrates best practice of using centralised secret management vault - time est: 4hrs
AddDataDog event when crash occurs e.g. “app 8081 crashed” - provides enhance reporting metrics - time est:2hrs
Moved to it's own repo as requirement does not belong here Metrics: Consul KV versus Vault KV. Test with 100-1000 entries. Simple bash script timed - see if there's a significant overhead when using vault as a KV over Consul KV. - time est: 4hrs
Build a new service (Secret-ID Factory) that generates a wrapped secret-id upon receipt of an app-role - (api only)
Build this in a separate repository using a similar CI/CD pipeline mentality
Deploy the new Secret-ID Factory as a service within this repo once complete
Create Bash Functions to Replace the curl -s -s statements in configure_app_role.sh to make it nicer to read
Modify the application to request a wrapped secret-id token from the new Secret-ID Factory outlined above inorder to obtain its vault token.
Change colour from Red to Blue in hand drawn architecture diagram for statement in Redis boc "Password Stored in Vault"
Remove all comments from redis.conf.ctpl
Moved Redis service registration from HCL file to API - fixed tests too
Move all the application service checks creation process into the application deployment itself rather than relying on external bash scripts for both redis and webpage counter
Refactor application to leverage consul service discovery for VAULT details .
Added new, prettier, frontend that facilitates smoother demonstrations by separating the frontend and backend code, written in Angular 6 and served from the NGINX proxy server on port 9091
Added Consul Connect for Redis Service
Refactored app to handle consul connect services
Upgraded to latest binaries for Nomad, Vault, Consul, Golang
Upgraded packer base image from Ubuntu 16.04 to Ubuntu 18.04
Updated readme for VaultFactoryID Service - the application bootstrapping service used
Added Consul Connect Service Mesh for application bootstrapping service
Configure a Consul Connect intention to permit the applications to communicate with the new Secret-ID factory
Added TLS to Consul services
Enabled Consul ACLs
Reconfigured all services to work with TLS and ACLs now required on consul
Configured consul connect service mesh for the webpagecounter go application
Configured consul intentions for REDIS and APPROLE services
Updated Documentation
Added Chef's Inspec test framework to test packer builds before release to production
Added TLS to Vault Services and verified on VMware and Virtualbox
Refactor EVERYTHING - General tidy up! Scripts getting very untidy and not battle hardened due to timimg constraints
- moved users and services to packer build
- added TLS certificates to packer build
add UDP encryption to Consul (never bothered up until now, should be trivial)
Add TLS to Nomad
Verify Vagrant file deployment
migrate from internal consul connect proxy to envoy (did this ages ago)
Moved from using local Terrafrom statefile to Terraform Cloud Account with a remote backend and VCS integration.