pavedroad-io / pavedroad

A currated set of tools for building a complete development environment for Go/GoLang, Vim, Docker, Kuberentes and supporting CI/CD tools with a single command

PavedRoad

Overview

PavedRoad.io is an OSS project for modeling the Software Development and Operations (SDO) life cycle. While Infrastructure as Code (IaC) gave us the ability to model our service, networks, storage, and compute resources. PavedRoad.io introduces Stacks as Code (SaC) which encompass the entire tool network including libraries, development tools, CI/CD, operations, and advance analytics using ML/AI.

What is a 'Paved Road'?

The term "Paved Road" was coined by the Netflix tools teams which created several fully integrated end-to-end tool networks for writing, testing, deploying, and operating their streaming video service. For each support execution framework such as Java, Python, or Go, an integrated working CI/CD tool network was created. This method of pre-integrated and tested tool networks converts a bumpy and difficult road into a delightfully smooth road which dramatically increases the velocity of development teams

Development Kit for GO

The Development Kit provides a foundation for delivering microservices, serverless functions, and integrating existing traditional and cloud applications running on bare-metal and virtual machines. It comes with a complete integrated tool network for developing, deploying, and operating those services.

For information on installing the Development Kit see Dev Kit README and for information on document generation see Doc Gen README.

Stacks as Code (SaC)

Stacks as Code (SaC) is the process of managing and provisioning computer data centers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools. See IaC on Wikipedia.

In SaC, we first automate the entire tool chain by using Kubernetes Custom Resource Definitions (CRD) as an abstraction layers/data model between each step. Kubernetes metadata provides an abstraction layer for passing data between each step. Customer controllers manage the flow and create a data presentation layer for tools making up the chain. This enables the tool network to be formed using standard k8s labels and selectors.

Project Status

The project is an early preview. We realize that it's going to take a village to arrive at the vision of a multi-cloud control plane, and we wanted to open this up early to get your help and feedback. Please see the Roadmap for details on what we are planning for future releases.

Official Releases

Official releases of PavedRoad can be found here: Official Releases. Please note that it is strongly recommended that you use the official releases of PavedRoad, as unreleased versions from the master branch are subject to changes and incompatibilities that will not be supported in the official releases. Builds from the master branch can have functionality changed and even removed at any time without compatibility support and without prior notice.

Links to More Information

Getting Help

For contact information or to report a bug see Support.

How to Contribute

For guidelines on contributions see Contributing.

Code of Conduct

This project follows this Code of Conduct.

License

This project is licensed under the following License.

pavedroad's Issues

Add jq to DevKit

jq

JQ is used to pretty print JSON in the scripts we generate. It is also used to process responses and extract the UUID.

Without this, the scripts break when doing demos

Improper permissions for installed files

After a new install, several directories are left being owned by root in the users home directory.

In particular, .kube and .skaffold will cause failures

ls -la ~jscharber/
drwx------ 3 root root 4096 Sep 19 13:06 .dbus
drwxr-x--- 4 root root 4096 Sep 19 13:17 .kube
-rw------- 1 root root 36 Sep 19 14:40 .lesshst
drwxr-xr-x 3 root root 4096 Sep 19 13:19 NONE
drwxr-xr-x 4 root root 4096 Sep 19 13:01 pr-root
drwxr--r-- 2 root root 4096 Sep 19 13:22 .skaffold

Improve descriptions in check-state.sh

Make clear the difference between -r -C and -X

Install sonar scanner

The sonar-scanner linter is used by SonarSloud. Now that that integration work is ready, we need it to be installed.

SonarScanner

The sonar-scanner shell scripts calls a Java jar file, see below. So it is probably best to install in the users $HOME/bin directory and add it to the PATH

script_path=$(real_path "$0")
sonar_scanner_home=$(dirname "$script_path")/..

# make it fully qualified
sonar_scanner_home=$(cd "$sonar_scanner_home" && pwd -P)

jar_file=$sonar_scanner_home/lib/sonar-scanner-cli-4.2.0.1873.jar

# check that sonar_scanner_home has been correctly set
if [ ! -f "$jar_file" ] ; then
  echo "File does not exist: $jar_file"
  echo "'$sonar_scanner_home' does not point to a valid installation directory: $sonar_scanner_home"
  exit 1

Add tools we commonly use

[ ] Kafkacat
[ ] Cockroach

Add upgrade capabilty to golang package

Perform regular devkit maintenance

Remove unsupported OS versions and add latest versions
Completely remove all traces of python2

Remove dep

Update blueprints and devkit to remove deprecated tools like dep and go-lint

user not added to microk8s group

When installing microk8s, the user is not added to the microk8s group. This results in the following message:

microk8s.config
Insufficient permissions to access MicroK8s.
You can either try again with sudo or add the user jscharber to the 'microk8s' group:

sudo usermod -a -G microk8s jscharber

The new group will be available on the user's next login.

After the user is added and they re-login, the system works as expected.

New release of OpenSuSE fails with invalid repo

Add $HOME/go/bin to users .bashrc path

On a clean install, compilation failed because we do not build in the default $HOME/src/package directory.

As we use "go get package" to install go utilities, once we reset GOPATH they can't be found.

Adding $HOME/go/bin to a users path fixes the problem

Add new dev script to create ./kube/config from microk8s.config

Each time microk8s is started, it's IP address can change.

We need to write this configuration data to .kube/confit for kubectl to function correctly with mikrok8s.

Since skaffold uses kubectl, the make process will fail without a proper configuration.

Adding a script will be a convince for developers.

We provide directions on how

add roadctl to bootstrap script

Not that we have it, we need to install it.

We need to talk about how we want to tag releases

Fix nocompatible setting in pr_vimrc_* files

Secret linter support

https://github.com/secretlint/secretlint

Check to make sure secrets are not getting committed in the source code.

Moving files from /tmp to home directory failing in unix bootstrap script

Include base state cleanup in this fix

Bring devkit up to date

Enable download tracking by converting bootstrap code to exec

Today, we support downloading a devkit by issuing a curl command and piping it's output to a sh. This does not allow downloads to be tracked.

For this task, we want to create a download executable "devKitInstaller", the executes the correct command based on the operating system.

The existing shell scripts can be bundled into the binary and then executed.
See for bundling, https://golangbot.com/read-files/

Add kakfa logger to core functions

[] create core directory structure
[] create log subdirectory
[] create client / consumer directories
[] need high-level design document
[] client should insert standard elements like timestamps
[] client should set our defaults

Basically follow normal go conventions, use http.client as example
klog := NewLogger(nil) // use default client

kclient := LoggerClient{Broker: "", .......} // use custom client
klog := NewLogger(kclient)

klog.Println("message")

Same thoughts around the consumer

Fix and clean up some devkit versions

Add ability to get latest version of direnv and kustomize
Update docker-compose completion file urls that have changed even though old ones still work
Update latest version settings in minion file for kubebuilder, sonar-scanner and stern
Replace hard coded versions with latest in sonar-scanner and stern