cloudfoundry-incubator / fissile

Turn a BOSH release into container images / files

Project Tracker: https://www.pivotaltracker.com/n/projects/2192232

Fissile converts existing BOSH final or dev releases into docker images.

It does this using just the releases, without a BOSH deployment, CPIs, or a BOSH agent.

Getting fissile

Prerequisites

Building fissile needs Go 1.12 or higher and Docker.

Build procedure

Fissile requires generated code using additional tools, and therefore isn't go get-able.

$ mkdir -p src                                  # make the directory src in your workspace
$ export GOPATH=$PWD                            # set GOPATH to current working directory
$ go get -d code.cloudfoundry.org/fissile       # Download sources
$ cd $GOPATH/src/code.cloudfoundry.org/fissile
$ make tools                              # install required tools; only needed first time
$ make docker-deps                        # pull docker images required to build
$ make all

Depending on your architecture you can use the fissile binary files from those directories: fissile/build/darwin-amd64 or fissile/build/linux-amd64.

Using Fissile

Please refer to the following additional documentation:

• Walkthrough on configuring and using fissile to build a docker image and corresponding Kubernetes resource definition
• Additional Kubernetes usage instructions and resource definition details
• Information on stemcells and how to build them
• Auto-generated usage reference

Releases using Fissile

• https://github.com/HeavyWombat/fissile-dev
• https://github.com/SUSE/SCF
• https://github.com/SUSE/nats-containerized-release
• https://github.com/SUSE/scf-helper-release
• https://github.com/SUSE/uaa-fissile-release

For testing and developing fissile itself we use the NATS containerized release.

Developing Fissile

In general, use of the default make target is preferred before making a pull request. This will run the unit tests, as well as various linters. To manually build fissile only, run make bindata build. This will run the necessary code generation before building the binary.

Testing

Run tests with make test (or use go test directly if you want to filter for specific tests, etc.) There are environment variables that can be set to adjust how tests are run:

Name	Value
FISSILE_TEST_DOCKER_IMAGE	the name of the default docker image for testing(e.g. splatform/fissile-opensuse-stemcell:42.2)

Vendoring

Fissile uses dep for vendoring required source code. To update the vendored source tree, please run dep ensure and double-check that it has not done anything silly.

Using the Example NATS Release

Requirements:

• bosh-cli
• Docker

First compile your version of fissile and put it into the shells $PATH:

$ cd $GOPATH/src/code.cloudfoundry.org/fissile
$ make tools build
$ cp build/linux-amd64/fissile /usr/local/bin

Compiling the BOSH Releases

Clone the containerized NATS release repository: git clone https://github.com/SUSE/nats-container-deployment.

Inside the repository is a containerize.sh script. It will clone the git repositories of all involved BOSH releases and use the bosh CLI to create the releases. Finally the BOSH releases are build inside a Docker container.

Source the .envrc if you're running the commands from the script manually, it contains all the necessary configuration for Fissile.

Building the Docker Images

Requirements:

• minikube

A full Kubernetes installation is not necessary to run the containerized release. Install minikube and start it with minikube start.

Since the docker images need to be accessible from the Kubernetes cluster we are going to build them in minikube's Docker daemon. The deploy.sh script does that, by setting the variables from minikube docker-env.

With the environment from .envrc loaded, fissile build images --force will create two docker images:

$ fissile show image
docker.io/bosh/fissile-nats:5546d958767e1c5a6b9f1b057ec0ffee51c556df
docker.io/bosh/fissile-secret-generation:aff03b8b7fade23c98514a52fb020ad4c1738d71

It's also possible to build the images locally, tag them with a public repositories address and push them there.

Generating Helm Charts and Deploying to Kubernetes

Requirements:

• kubectl
• Helm

With the Docker images built we can now generate Helm charts:

$ fissile build helm --auth-type rbac --defaults-file defaults.txt

The generated chart is missing a basic Chart.yaml:

cat > "$FISSILE_OUTPUT_DIR/Chart.yaml" << EOF
apiVersion: 1
description: A Helm chart for NATS
name: my-nats
version: 1
EOF

After installing the Helm CLI and running helm init we can deploy the NATS release:

$ helm install nats-chart --name my-nats --namespace my-nats --values vars.yml

Removing the Example Release

Most work is done in the output folder. The Helm chart is generated in nats-chart/. These folders can be safely removed to start from scratch.

Use docker rmi to remove images from docker.

To remove the Helm deployment and release run helm delete --purge my-nats. To be sure all k8s resources are removed run kubectl delete namespace my-nats.

Acceptance Tests with Example Release

Several manual test cases around secret generation exist, which we run against the example release. We're currently evaluating more test cases and automated CI.

fissile's Issues

Race condition for active/passive pods when no leader is available

For HA pods like the diego-api with active/passive roles, there is a race condition that will not allow the skiff-role-active label to be set to its correct state(true/false).

For the specific use case(diego-api pod), this can happen when the readines-probe.sh its executed when the bbs job in the leader pod stops working(not marked as running), which will block the readiness-probe.sh on reaching the script part where the label can be set to false.

An example of such situation is when a connection to the db is lost, which will leave one diego-api pod marked as skiff-role-active = true, when that value is not true.

This is not critical, but I considered important to highlight.

Support for BOSH deployment manifests in configgin

• Change configgin so it accepts input from a BOSH deployment manifest
• Change fissile so it generates pods that mount a deployment manifest secret
• Add --deployment-manifest-template to fissile build helm

Full spec for this feature lives here.

What is the difference between '"(())"', '(())' and (())?

In role-manifest.yml, the "templates" part contains some properties, there are several formats:
templates:
one: '"((ONE))"'
two: '((TWO))'
three: ((THREE))

What is the difference between '"(())"', '(())' and (())?

Create a repo for the NATS "example release"

• Create a github repository for it
• Talk to the guys at SAP to get the latest work
• Test to see everything works
• Update the fissile README to point to the new repo

`fissile diff` only works with already unpacked release directories and not with URLs

I expected to be able to run

$ fissile diff --release https://bosh.io/d/github.com/cloudfoundry/nats-release?v=23,https://bosh.io/d/github.com/cloudfoundry/nats-release?v=24

But that doesn't work:

dev config diff: error loading release information: Error loading dev release information: Path /Users/jan/go/src/code.cloudfoundry.org/fissile/https:/bosh.io/d/github.com/cloudfoundry/nats-release?v=23 (release directory) does not exist

Where is the dockerfile?

I use fissile build images successfully. In the work dir there is a dockerfiles directory, but no dockerfile in it. How to find the docker files?

Services generated by fissile make Istio malfunction

When Istio is in use on a Kubenrnetes cluster, port names of all Services have to follow the naming convention required by Istio. This is also true when the components generated by fissile are not using Istio themselves. While this can be considered a design flaw and might be fixed in future we have to deal with it somehow in the meanwhile.

Here are a couple of things that could be done to mitigate the issue:

• Add networking.istio.io/exportTo= . annotation to the Services generated by fissile which are not supposed to be accessed through Istio - helps to work around istio/istio#9784 and istio/istio#14520
• Rename ports to comply with the naming conventions required by Istio https://istio.io/docs/setup/kubernetes/additional-setup/requirements keeping in mind that a certain port number must have the same name prefix over the whole cluster unless the service is scoped to a single namespace by networking.istio.io/exportTo= ., e.g. for port 443 the name must always start from https - helps with istio/istio#14520

cc: @iandyh @viovanov

Update:

• I looks like I'm mixing two problems here (istio/istio#9784 and istio/istio#14520) though it's fine as they are similar in a way they affect the rest of the cluster and the approaches towards solving the issues are likely to be connected

Colocated role port collision check fails for same port with TCP/UDP

With PR #340, a collision check for colocated roles has been introduced that has an issue when you specify the same port number with different protocols, because the collision check only checks the port number.

- name: consul
  colocated_containers:
  - logs-forwarder
  run:
    exposed-ports:
    - name: serf-tcp
      extenral: 8301
      internal: 8301
      protocol: TCP
    - name: serf-udp
      extenral: 8301
      internal: 8301
      protocol: UDP

The configuration is totally fine, but the collision checks reports that as an issue: Error loading roles manifest: role[consul]: Invalid value: 8301: port collision, the same port is used by: consul, consul

To resolve this, the combination of protocol and port has to be checked against one another, for example: TCP-8301 and UDP-8301. I will start to work on a fix for this as soon as possible.

Build images successfully, but build kube and heml failed.

I can build images successfully, but when I tried to build kube or build helm, the error below showed:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x7bafd4]

goroutine 1 [running]:
github.com/SUSE/fissile/helm.(*Encoder).writeNode(0xc4202d5880, 0x0, 0x0, 0xc4201136d0, 0x0, 0x0, 0x1)
/root/workspace/goworkspace/src/github.com/SUSE/fissile/helm/config.go:561 +0x154
github.com/SUSE/fissile/helm.(*Encoder).Encode(0xc4202d5880, 0x0, 0x0, 0x0, 0x0)
/root/workspace/goworkspace/src/github.com/SUSE/fissile/helm/config.go:438 +0xe0
github.com/SUSE/fissile/app.(*Fissile).generateKubeRoles(0xc42005c840, 0xc420017ac0, 0x1, 0xc420017a10, 0x7, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/root/workspace/goworkspace/src/github.com/SUSE/fissile/app/fissile.go:1078 +0x77a
github.com/SUSE/fissile/app.(*Fissile).GenerateKube(0xc42005c840, 0xc42006a5a0, 0x5e, 0x0, 0x0, 0x0, 0xc420017ac0, 0x1, 0xc420017a10, 0x7, ...)
/root/workspace/goworkspace/src/github.com/SUSE/fissile/app/fissile.go:924 +0x391
github.com/SUSE/fissile/cmd.glob..func2(0xc8a220, 0xcb3698, 0x0, 0x0, 0x0, 0x0)
/root/workspace/goworkspace/src/github.com/SUSE/fissile/cmd/build-helm.go:65 +0x45d
github.com/SUSE/fissile/vendor/github.com/spf13/cobra.(*Command).execute(0xc8a220, 0xcb3698, 0x0, 0x0, 0xc8a220, 0xcb3698)
/root/workspace/goworkspace/src/github.com/SUSE/fissile/vendor/github.com/spf13/cobra/command.go:646 +0x3e8
github.com/SUSE/fissile/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc8aee0, 0x20, 0xc42005c800, 0xc420113e80)
/root/workspace/goworkspace/src/github.com/SUSE/fissile/vendor/github.com/spf13/cobra/command.go:737 +0x2fe
github.com/SUSE/fissile/vendor/github.com/spf13/cobra.(*Command).Execute(0xc8aee0, 0x1, 0xc42005c840)
/root/workspace/goworkspace/src/github.com/SUSE/fissile/vendor/github.com/spf13/cobra/command.go:695 +0x2b
github.com/SUSE/fissile/cmd.Execute(0xc42005c840, 0xa0dce0, 0x19, 0x4, 0x4)
/root/workspace/goworkspace/src/github.com/SUSE/fissile/cmd/root.go:75 +0x69
main.main()
/root/workspace/goworkspace/src/github.com/SUSE/fissile/main.go:45 +0x186

Properties with job namespaces in fissile

• Remove support for the configuration.templates key in fissile role manifests (both global and role based)
• Remove support for opinions.yaml as well as dark-opinions.yaml

Both templates and opinions will be part of jobs, just like a deployment manifest.
The role manifest should look like:

---
roles:
- name: nats
  jobs:
  - name: nats
    properties:
      nats:
        password: '"((NATS_PASSWORD))"'
        user: nats

Add support for multi-containers aka sidecar per Pod

Hi,

We currently have a strong use case for supporting multiple containers per Pod(aka sidecars). By default in Kube, a pod configuration file allows to have multiple entries under spec.containers.

However, the current setup of fissile(e.g. running a fissile build helm), does not support this by default. We can always patch the YAML templates generated by fissile, but ideally we want to support this sidecar feature as part of the fissile natural behavior(e.g. throught fissile build helm).

We would like to open a discussion on how to achieve this, and then PRing this to you. The main concern is that we will need to modify/introduce some of the YAML syntax for the role-manifest .

Our current idea would be to introduce two new keys in the role-manifest.yml, as follows:

roles:
- name: nginx
  jobs:
  - name: nginx
    release_name: nginx
  tags:
  - clustered
  sidecars:                 # new list to reference sidecar roles
  - log-forwarder
  run:
    memory: 256
    virtual-cpus: 4
    exposed-ports: [...]
    shared-volumes:
      - path: /mnt/share
        tag: common-share
        empty-dir: true    # new reference for a "volumes" entry in the template YAML
- name: log-forwarder
  type: sidecar            # new type called sidecar
  jobs:
  - name: elk
    release_name: elk
  tags:
  - clustered
  run:
    memory: 256
    virtual-cpus: 4
    exposed-ports: [...]
    shared-volumes:
      - path: /mnt/share
        tag: common-share
        empty-dir: true    # new reference for a "volumes" entry in the template YAML

As illustrated above, the roles.nginx.sidecars will ideally be the trigger to colocate in the nginx pod template, a second item under spec.containers with the properties of the log-forwarder sidecar.

Inside the log-forwarder role, the roles.component.type will indicate that this Pod YAML template should not be generated, whilst we expect the docker image to be generated.

With this sidecar, we would also require a simple volume sharing solution. For example, as defined in the Creating a Pod that runs two Containers scenario, we just want to share one directory between the containers. For that, an additional role-manifest.yml modification would be required to enable support for emptyDir: {}.

What do you think?

Thanks in advance for your feedback.
Regards,
@HeavyWombat @qu1queee

Better error message for missing stemcell

We've had multiple instances (from people on our team) where the stemcell wasn't downloaded, and we just report a vague and misleading "missing image" instead of being more specific about stemcell failures:

Compiling packages for dev releases:
         uaa (29+dev.1)
         cf-mysql (34+dev.10)
         hcf (0+dev.1)
compile: cf-mysql/cf-mysql-common
compile: cf-mysql/boost
done:    cf-mysql/cf-mysql-common
result   > failure: cf-mysql/cf-mysql-common - Error compiling package cf-mysql-common: no such image

We should have a better error message here so we can stop wasting time tripping over this over an over 😄

The image can not docker run

I made a role manifest to build the docker image successfully, but when I tried to use docker run $imageName:tag, I got the error as below:
/usr/local/rvm/gems/ruby-2.3.1/gems/bosh-template-1.3262.24.0/lib/bosh/template/evaluation_context.rb:93:in p': Can't find property '["fissile.monit.port"]' (Bosh::Template::UnknownProperty) from /opt/scf/monitrc.erb:9:in get_binding'
from /usr/local/rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/erb.rb:864:in eval' from /usr/local/rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/erb.rb:864:in result'
from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.12.0.pre/lib/generate.rb:30:in generate' from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.12.0.pre/bin/configgin:34:in block (2 levels) in <top (required)>'
from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.12.0.pre/bin/configgin:23:in each' from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.12.0.pre/bin/configgin:23:in block in <top (required)>'
from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.12.0.pre/bin/configgin:20:in each' from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.12.0.pre/bin/configgin:20:in <top (required)>'
from /usr/local/rvm/gems/ruby-2.3.1/bin/configgin:23:in load' from /usr/local/rvm/gems/ruby-2.3.1/bin/configgin:23:in

'
from /usr/local/rvm/gems/ruby-2.3.1/bin/ruby_executable_hooks:15:in eval' from /usr/local/rvm/gems/ruby-2.3.1/bin/ruby_executable_hooks:15:in '

[WIP] Dockerfile for Fissile build

@JulzDiverse, @viovanov: As discussed in https://cloudfoundry.slack.com/archives/C8RU3BZ26/p1536217742000100, here is my very-much-work-in-progress version: https://gist.github.com/bkrannich/883e0bea4fe1dd2a060dafbf598418c2

This is an updated version based on earlier work by @zhangtbj in SUSE/scf#1174

Cut releases with release notes?

Its a year since last github release and 352 commits since then. I've discovered the magic S3 bucket with builds from CI; but will we return to github releases with release notes? Especially for notifications of updates to the fissile build release-images command :)

For anyone else wondering how to get latest fissile, here's an example command to download/install latest darwin fissile into $GOPATH/bin:

aws s3 cp "s3://cf-opensusefs2/fissile/develop/$(aws s3 ls s3://cf-opensusefs2/fissile/develop/ | sort | grep darwin | tail -n1 | awk '{print $4}')" - | tar -C $GOPATH/bin -xvz fissile

Error in step "make all"

Hi, I follow the steps :

$ go get -d github.com/SUSE/fissile       # Download sources
$ cd $GOPATH/src/github.com/SUSE/fissile
$ make tools                              # install required tools; only needed first time
$ make docker-deps                        # pull docker images required to build
$ make all

Then in the last step "make all", the following error shows. Any idea?

/Users/qiyang/dev/goworkspace/fissile/make/clean
/Users/qiyang/dev/goworkspace/fissile/make/format
==> Formatting
diff -u /Users/qiyang/dev/goworkspace/fissile/app/fissile.go.orig /Users/qiyang/dev/goworkspace/fissile/app/fissile.go
--- /Users/qiyang/dev/goworkspace/fissile/app/fissile.go.orig	2017-07-20 10:44:02.000000000 +0800
+++ /Users/qiyang/dev/goworkspace/fissile/app/fissile.go	2017-07-20 10:44:02.000000000 +0800
@@ -22,7 +22,6 @@

 	"github.com/fatih/color"
 	"github.com/joho/godotenv"
-	"gopkg.in/yaml.v2"
 )

 // OutputFormat is one of the known output formats for commands showing loaded
diff -u /Users/qiyang/dev/goworkspace/fissile/builder/role_image.go.orig /Users/qiyang/dev/goworkspace/fissile/builder/role_image.go
--- /Users/qiyang/dev/goworkspace/fissile/builder/role_image.go.orig	2017-07-20 10:44:02.000000000 +0800
+++ /Users/qiyang/dev/goworkspace/fissile/builder/role_image.go	2017-07-20 10:44:02.000000000 +0800
@@ -20,7 +20,6 @@

 	"github.com/fatih/color"
 	workerLib "github.com/jimmysawczuk/worker"
-	"gopkg.in/yaml.v2"
 )

 const (
diff -u /Users/qiyang/dev/goworkspace/fissile/compilator/compilator.go.orig /Users/qiyang/dev/goworkspace/fissile/compilator/compilator.go
--- /Users/qiyang/dev/goworkspace/fissile/compilator/compilator.go.orig	2017-07-20 10:44:02.000000000 +0800
+++ /Users/qiyang/dev/goworkspace/fissile/compilator/compilator.go	2017-07-20 10:44:02.000000000 +0800
@@ -22,7 +22,6 @@
 	"github.com/fatih/color"
 	workerLib "github.com/jimmysawczuk/worker"
 	"github.com/pborman/uuid"
-	"github.com/termie/go-shutil"
 )

 const (
diff -u /Users/qiyang/dev/goworkspace/fissile/model/dev_release.go.orig /Users/qiyang/dev/goworkspace/fissile/model/dev_release.go
--- /Users/qiyang/dev/goworkspace/fissile/model/dev_release.go.orig	2017-07-20 10:44:02.000000000 +0800
+++ /Users/qiyang/dev/goworkspace/fissile/model/dev_release.go	2017-07-20 10:44:02.000000000 +0800
@@ -8,7 +8,6 @@
 	"github.com/SUSE/fissile/util"

 	"github.com/cppforlife/go-semi-semantic/version"
-	"gopkg.in/yaml.v2"
 )

 // NewDevRelease will create an instance of a BOSH development release
diff -u /Users/qiyang/dev/goworkspace/fissile/model/job.go.orig /Users/qiyang/dev/goworkspace/fissile/model/job.go
--- /Users/qiyang/dev/goworkspace/fissile/model/job.go.orig	2017-07-20 10:44:02.000000000 +0800
+++ /Users/qiyang/dev/goworkspace/fissile/model/job.go	2017-07-20 10:44:02.000000000 +0800
@@ -14,7 +14,6 @@
 	"github.com/SUSE/fissile/util"

 	"github.com/pivotal-golang/archiver/extractor"
-	"gopkg.in/yaml.v2"
 )

 // Job represents a BOSH job
diff -u /Users/qiyang/dev/goworkspace/fissile/model/opinions.go.orig /Users/qiyang/dev/goworkspace/fissile/model/opinions.go
--- /Users/qiyang/dev/goworkspace/fissile/model/opinions.go.orig	2017-07-20 10:44:02.000000000 +0800
+++ /Users/qiyang/dev/goworkspace/fissile/model/opinions.go	2017-07-20 10:44:02.000000000 +0800
@@ -4,8 +4,6 @@
 	"fmt"
 	"io/ioutil"
 	"reflect"
-
-	"gopkg.in/yaml.v2"
 )

 // Opinions holds the light and dark opinions given to fissile
diff -u /Users/qiyang/dev/goworkspace/fissile/model/release.go.orig /Users/qiyang/dev/goworkspace/fissile/model/release.go
--- /Users/qiyang/dev/goworkspace/fissile/model/release.go.orig	2017-07-20 10:44:02.000000000 +0800
+++ /Users/qiyang/dev/goworkspace/fissile/model/release.go	2017-07-20 10:44:02.000000000 +0800
@@ -8,8 +8,6 @@
 	"regexp"

 	"github.com/SUSE/fissile/util"
-
-	"gopkg.in/yaml.v2"
 )

 // Release represents a BOSH release
diff -u /Users/qiyang/dev/goworkspace/fissile/model/roles.go.orig /Users/qiyang/dev/goworkspace/fissile/model/roles.go
--- /Users/qiyang/dev/goworkspace/fissile/model/roles.go.orig	2017-07-20 10:44:02.000000000 +0800
+++ /Users/qiyang/dev/goworkspace/fissile/model/roles.go	2017-07-20 10:44:02.000000000 +0800
@@ -12,8 +12,6 @@
 	"strings"

 	"github.com/SUSE/fissile/validation"
-
-	"gopkg.in/yaml.v2"
 )

 // RoleType is the type of the role; see the constants below
diff -u /Users/qiyang/dev/goworkspace/fissile/src/github.com/axw/gocov/gocovutil/packages.go.orig /Users/qiyang/dev/goworkspace/fissile/src/github.com/axw/gocov/gocovutil/packages.go
--- /Users/qiyang/dev/goworkspace/fissile/src/github.com/axw/gocov/gocovutil/packages.go.orig	2017-07-20 10:44:02.000000000 +0800
+++ /Users/qiyang/dev/goworkspace/fissile/src/github.com/axw/gocov/gocovutil/packages.go	2017-07-20 10:44:02.000000000 +0800
@@ -2,10 +2,11 @@

 import (
 	"encoding/json"
-	"github.com/axw/gocov"
 	"io/ioutil"
 	"os"
 	"sort"
+
+	"github.com/axw/gocov"
 )

 // Packages represents a set of gocov.Package structures.
diff -u /Users/qiyang/dev/goworkspace/fissile/util/serialization_helper_test.go.orig /Users/qiyang/dev/goworkspace/fissile/util/serialization_helper_test.go
--- /Users/qiyang/dev/goworkspace/fissile/util/serialization_helper_test.go.orig	2017-07-20 10:44:05.000000000 +0800
+++ /Users/qiyang/dev/goworkspace/fissile/util/serialization_helper_test.go	2017-07-20 10:44:05.000000000 +0800
@@ -8,7 +8,6 @@
 	"github.com/SUSE/fissile/testhelpers"

 	"github.com/stretchr/testify/assert"
-	"gopkg.in/yaml.v2"
 )

 type jsonHelpInputTestData struct {
make: *** [format] Error 1

Implement the no-monit tag

• Spike to find what information we need to be able to remove monit?
• Spike to see how we can use BPM to get the information needed at 1)
• Use knowledge from 1) and 2) generate helm charts with an alternative entrypoint
• Add no-monit for metron in SCF

Make metron agent a colocated container in SCF

• Create a colocated container in SCF for metron
• Replace all metron jobs in SCF with the colocated container
• Make sure all SCF/UAA tests pass

New compilation cache code creates broken archives

@viovanov @Tibeos:

The problem is that relative symlinks are turned into absolute symlinks:

# ls -l /var/vcap/packages/postgres-9.6.6/lib/libpq.so.5
lrwxrwxrwx 1 root root 140 Sep 28 19:42 /var/vcap/packages/postgres-9.6.6/lib/libpq.so.5 -> /home/vagrant/.fissile/compilation/4c162e7204bd9cd84142a77c6de81eaf154bf16a/64abaa2d40d8a9482fa1a4ee8b9e8301b6898dc8/compiled/lib/libpq.so.5

This should have been a simple relative symlink:

# ls -l /var/vcap/packages/postgres-9.6.6/lib/libpq.so.5
lrwxrwxrwx 1 root root 12 Sep 11 01:35 /var/vcap/packages/postgres-9.6.6/lib/libpq.so.5 -> libpq.so.5.9

The problem arises because the absolute symlink points into the fissile compilation cache directory (which doesn't exist inside the container) and not to the actual install location.

This seems to be a known limitation of the archiver package: mholt/archiver#74

Update dependencies once mholt/archiver#92 has landed

Waiting for mholt/archiver#92

We have a local change to the vendored dependency in #414, which should be replaced by the proper fix as soon as available.

Create one Kubernetes Service per job

• Change helm charts so they generate one service per job
• Remove the headless tag from fissile
• Exposed ports have to be moved to the job section
• Update fissile doc
• Update SCF/UAA

Use the following rule to determine the service name (which in turn determines the hostname):

<role-name>-<job-name>

Ports must be exposed like this:

roles:
- name: nats
  jobs:
  - name: nats
    properties:
      bosh_containerization:
        ports:
        - name: nats
          protocol: TCP
          internal: 4222
        - name: nats-routes
          protocol: TCP
          internal: 4223

fissile does not build

~/go/src/github.com/cloudfoundry-incubator/fissile% make build
==> Binary Data 
==> Building 
# github.com/cloudfoundry-incubator/fissile
./main.go:43:32: cannot use ui (type *"github.com/cloudfoundry-incubator/fissile/vendor/github.com/SUSE/termui".UI) as type *"github.com/SUSE/fissile/vendor/github.com/SUSE/termui".UI in argument to app.NewFissileApplication
make: *** [Makefile:29: build] Error 2

Since we moved the repos, fissile won't build from a new checkout. I think we need to update all the import statements?

% git grep github.com/SUSE/fissile/ |wc -l
125

Fix:
perl -pi -e '[email protected]/SUSE/fissile/@github.com/cloudfoundry-incubator/fissile/@' **/*.go

Walkthrough manifests are unusable

I tried the walkthrough manifests at https://github.com/cloudfoundry-incubator/fissile/blob/develop/docs/configuration.md and failed multiple times.

Some of the bugs I found:

• yaml structure no longer complies with what fissile cli expects. variables node must be at top level.
• bosh_containerization property is required on each job
• tags: indexed no longer works
• must use dark-opinions.yml file even though I did not need them for my bosh release
• can't have empty dark-opinions.yml (or just a properties yml node)
• variable node structure different. e.g. description must be under options node instead

I finally succeeded by looking at the code and using the test assets as template for my own manifests.

I suggest to add the walkthrough example manifests to daily builds to make sure they work.

Make the role manifest compatible with BOSH deployment manifests

• Rename roles to instance_groups
• For jobs definitions, use release instead of release_name
• Change secret generator so it supports consuming information from a variables section
• Convert variables and secret information to a variables section in SCF/UAA
• Move containerization-specific configuration to <job>.properties.bosh_containerization
• cf-deployment + an ops file == role manifest

Why not create docker images from rev releases?

I'm trying to debug a bosh release -> fissile -> cf-operator [1] and want to create a dev release -> fissile build release-images but get an error:

$ version=15.3.3.$(date +%s)
$ bosh create-release --tarball tmp/redis-$version.tgz
$ fissile build release-images --stemcell splatform/fissile-stemcell-opensuse:42.3-38.g82067a9-30.95 --name redis --url tmp/redis-$version.tgz --sha1 $(sha1sum tmp/redis-$version.tgz | awk '{print $1}') --version $version -w $PWD/tmp/fissile
Error loading release information: 2 errors occurred:
	* Dev release redis is not supported as manifest references
	* open /Users/drnic/.final-releases/redis-15.3.3.1569804399-66bae730f6a83ef317b9c18cb1f9f278d3d54298/release.MF: no such file or directory

Why can't I create a fissile OCI from a dev release?

[1] cloudfoundry-incubator/quarks-operator#596

Build docker image from bosh stemcells

Hi,

I want to build a docker image from bosh stemcells following guide. Since I can't access the pipeline, I'm not sure what the bosh stemcell stages are included to build the image. Would you please help provide the detailed stage list?

Thanks a lot!

Role manifest shared volume validation bug

We ran into an issue where the role manifest validation failed for our setup, where one instance group has multiple colocated container instance groups. Not all of the colocated instance groups have an emptyDir volume defined.

However, the validation we initially wrote for fissile did not take that scenario into account and simply checked that the number of volumes match. This does not work for the aforementioned setup. The number of colocated containers is two, but the number of volumes is one. Therefore, the validation fails.

instance_groups:
- name: main-instance-group
  jobs:
  - name: tor
    release: tor
    properties:
      bosh_containerization:
        colocated_containers:
        - to-be-colocated-1
        - to-be-colocated-2
        run:
          memory: 1
          volumes:
          - path: /shared/data
            type: emptyDir
            tag: shared-data

- name: to-be-colocated-1
  type: colocated-container
  jobs:
  - name: tor
    release: tor
    properties:
      bosh_containerization:
        run:
          memory: 1
          volumes:
          - path: /shared/data
            type: emptyDir
            tag: shared-data

- name: to-be-colocated-2
  type: colocated-container
  jobs:
  - name: ntpd
    release: ntp
    properties:
      bosh_containerization:
        # no volume defined here ...
        run:
          memory: 1

Roles are mandatory but not generated

When trying to Fissilify a simple BOSH release, I followed the walkthrough and upon deployment, I received the error pods "spacebearsdbnode-0" is forbidden: User "system:serviceaccount:simple:default" cannot get pods in the namespace "simple": Unknown user "system:serviceaccount:simple:default" (KubeException). My Helm chart didn't include any service accounts.

@viovanov kindly pointed out that I'd need to add auth roles for configgin and secrets.

This isn't mentioned anywhere in the walkthrough, and if it's mandatory, Fissile should probably make these for the user.

a typo issue in run.sh

"chmod 1730 /var/spool/cron/tabs/"
The directory "/var/spool/cron/tabs/" is not accessible in ubuntu os, it is specific for openSuSe

Spike: figure out how to not create services for every job in an instance group

Some jobs don't require a service. Nothing connects to them, or they advertise themselves through an internal mechanism.
BOSH/fissile needs a way to describe this.

This will improve #367

Move to Go Modules

The build instructions seem complicated -- would they be greatly simplified if we move to Go Modules?

Don't break backwards compatibility

There have been a few recent changes that assumes the project uses the new https://github.com/SUSE/scf-secret-generator mechanism to manage secrets that break the automatic generation of password when the generator pod isn't used.

These changes should be gated by a feature flag and the backwards compatibility to the old functionality should be restored.

Fix /docs dir

[question] do you have a plan to support the Cloud Foundry releases for this?

so we can deploy one Cloud Foundry clusters on kubernetes quickly

Pod runtime information is in a non-sensical spot

Currently, in the role manifest:

• the manifest has multiple instance groups
• each instance group has a bunch of jobs
• each job has properties
• inside the properties for each job, there is a bosh_containerization section
• inside of that, there is the run information, such as limits, affinities, and scaling information

This makes no sense: you can't scale each job apart from the pods ( = instance groups) they are in. They should be moved to under instance groups (though we may want to keep limits per-job in case we get to container-per-job). Otherwise it's unclear what happens if I have two jobs in an instance group, one saying I can't HA it, and one saying I can…

Unauthorized

testing a simple bosh release, and the container is not running.

10.202.128.91	proxy-0.proxy-set.default.svc.cluster.local proxy-0
/usr/local/rvm/gems/ruby-2.3.1/gems/kubeclient-2.5.2/lib/kubeclient/common.rb:117:in `rescue in handle_exception': Unauthorized (KubeException)
	from /usr/local/rvm/gems/ruby-2.3.1/gems/kubeclient-2.5.2/lib/kubeclient/common.rb:109:in `handle_exception'
	from /usr/local/rvm/gems/ruby-2.3.1/gems/kubeclient-2.5.2/lib/kubeclient/common.rb:474:in `fetch_entities'
	from /usr/local/rvm/gems/ruby-2.3.1/gems/kubeclient-2.5.2/lib/kubeclient/common.rb:464:in `load_entities'
	from /usr/local/rvm/gems/ruby-2.3.1/gems/kubeclient-2.5.2/lib/kubeclient/common.rb:121:in `discover'
	from /usr/local/rvm/gems/ruby-2.3.1/gems/kubeclient-2.5.2/lib/kubeclient/common.rb:88:in `method_missing'
	from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.17.0/lib/job.rb:40:in `self_pod'
	from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.17.0/lib/job.rb:44:in `self_role'
	from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.17.0/lib/job.rb:14:in `initialize'
	from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.17.0/bin/configgin:65:in `new'
	from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.17.0/bin/configgin:65:in `block in <top (required)>'
	from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.17.0/bin/configgin:52:in `each'
	from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.17.0/bin/configgin:52:in `<top (required)>'
	from /usr/local/rvm/gems/ruby-2.3.1/bin/configgin:23:in `load'
	from /usr/local/rvm/gems/ruby-2.3.1/bin/configgin:23:in `<main>'
	from /usr/local/rvm/gems/ruby-2.3.1/bin/ruby_executable_hooks:24:in `eval'
	from /usr/local/rvm/gems/ruby-2.3.1/bin/ruby_executable_hooks:24:in `<main>'

The cluster has a self signed CA and all api clients use a client cert, I assume I am missing some configuration in my role manifest, but I am unsure where to put it.

fissile build error when commit_hash is a scientific format string

When dev.yml looks like below. The commit_hash is a scientific format string, fissile build will fail:

name: scf
version: 0+dev.1
commit_hash: 9018e89
uncommitted_changes: true
jobs:

Error:

Please allow a long time for mariadb to compile
Error loading dev release information: Error trying to load release scf metadata from YAML manifest /home/vagrant/scf/src/scf-release/dev_releases/scf/scf-0+dev.1.yml: interface conversion: interface {} is float64, not string
Makefile:258: recipe for target 'compile' failed
make: *** [compile] Error 1

1-1 roles with cf_deployment instance groups

• Spike to see if hairpin is required for talking to another container in the same pod
• Create colocated roles for the jobs that need to talk back to a colocated job
• Test that everything works in an HA deployment

First part of this effort is here.
Second part requires that jobs can talk back to other colocated jobs.
One service per job is also a requirement.

docker run example for nats-release fails: cannot access '/usr/local/bin/create-release.sh': No such file or directory

https://github.com/cloudfoundry-incubator/fissile/blob/develop/docs/configuration.md#building-the-nats-image contains a docker run command that fails. What is the correct command for the image?

$ docker run --rm \
    --volume "${HOME}/.bosh/cache:/bosh-cache" \
    --volume "${PWD}/nats-release:${PWD}/nats-release" \
    --env "RUBY_VERSION=2.2.3" \
    splatform/bosh-cli \
    /usr/local/bin/create-release.sh \
        "$(id -u)" "$(id -g)" /bosh-cache --dir "${PWD}/nats-release" --force --name "nats"
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"/usr/local/bin/create-release.sh\": stat /usr/local/bin/create-release.sh: no such file or directory": unknown.

I found a bosh.sh and quickly tried /usr/local/bin/bosh.sh create-release but it failed a different way:

docker run --rm \
    --volume "${HOME}/.bosh/cache:/bosh-cache" \
    --volume "${PWD}/nats-release:${PWD}/nats-release" \
    --env "RUBY_VERSION=2.2.3" \
    splatform/bosh-cli \
    /usr/local/bin/bosh.sh create-release \
        "$(id -u)" "$(id -g)" /bosh-cache --dir "${PWD}/nats-release" --force --name "nats"
/usr/local/bin/bosh.sh: line 27: addgroup: command not found

Remove dark opinions

I talked to @viovanov and if I understood correctly dark opinions are no longer necessary and could be removed from the source code and documentation.

Example doc for build in configuration.md is wrong (create-release.sh not found)

I tried to go through the example in https://github.com/cloudfoundry-incubator/fissile/blob/develop/docs/configuration.md, but i got stuck at the first building step: https://github.com/cloudfoundry-incubator/fissile/blob/develop/docs/configuration.md#building-the-nats-image

docker: Error response from daemon: OCI runtime create failed: container_linux.go:344: starting container process caused "exec: \"/usr/local/bin/create-release.sh\": stat /usr/local/bin/create-release.sh: no such file or directory": unknown.

Apparently, the referenced docker images has no sign of the create-release.sh script in it.

Deployment fails on clusters with containerd when credentials are empty

This line does not work when we don't need credentials for Docker hub

fissile/kube/registry_credentials.go

Line 23 in c617cf6

value = `{{ printf "{%q:{%q:%q,%q:%q,%q:%q}}" ` +

Deployment fails. See more in slack thread https://cloudfoundry.slack.com/archives/C8RU3BZ26/p1551369471021700

add pre-built binaries to github releases?

Hello happy ppl. Could your CI publish linux/darwin binaries to github releases?

CF in kubernetes

Hello
We as a Fujitsu are very interested to run CF in Kubernetes. I am not a CF expert. I have experience in k8s.
I've seen a presentation how to do it but I can not find and documentation or examples which could help me. Can someone drive me ho to do it ? Any help is appreciated.
I assume I have to convert CF components to Docker images and create pods. So If someone have some scripts or k8s deployment file it would be very helpful.
Please excuse my lack of knowledge because I've already started picking up knowledge about CF and fissile

Generated K8s resources will not be supported in K8s 1.16

As announced here, deprecated APIs are being removed from K8s 1.16, which is scheduled to be released tomorrow, 18 Sep 2019 as specified here.

The APIs that affect fissiled helm charts are statefulsets and deployments. Fissile should generate them in apps/v1 API and not apps/v1beta1 or extensions/v1beta1

Container start error

Every container failed with the same error msg as below: What is the problem?

/usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.12.0.pre/lib/generate.rb:24:in rescue in generate': failed to read template file /opt/fissile/monitrc.erb (RuntimeError) from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.12.0.pre/lib/generate.rb:19:in generate'
from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.12.0.pre/bin/configgin:34:in block (2 levels) in <top (required)>' from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.12.0.pre/bin/configgin:23:in each'
from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.12.0.pre/bin/configgin:23:in block in <top (required)>' from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.12.0.pre/bin/configgin:20:in each'
from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.12.0.pre/bin/configgin:20:in <top (required)>' from /usr/local/rvm/gems/ruby-2.3.1/bin/configgin:23:in load'
from /usr/local/rvm/gems/ruby-2.3.1/bin/configgin:23:in <main>' from /usr/local/rvm/gems/ruby-2.3.1/bin/ruby_executable_hooks:15:in eval'
from /usr/local/rvm/gems/ruby-2.3.1/bin/ruby_executable_hooks:15:in `

'

Make all fails on go 1.8.1

Fissile fails make all on openSUSE Leap 42.2 using go 1.8.1 (from the official tarball) with

docker/docker_test.go:17:2: import "github.com/hpcloud/fissile/vendor/github.com/golang/mock/mockgen" is a program, not an importable package

The go environment was empty, the build fails when trying to reproduce the steps from the readme (adding the missing make docker-deps and make tools)

Complete Log:

/home/user/GOPATH/src/github.com/hpcloud/fissile/make/clean
/home/user/GOPATH/src/github.com/hpcloud/fissile/make/format==> Formatting                                                                                                                                                                                                                                                                    /home/user/GOPATH/src/github.com/hpcloud/fissile/make/lint
==> Linting
/home/user/GOPATH/src/github.com/hpcloud/fissile/make/vet
/home/user/GOPATH/src/github.com/hpcloud/fissile/make/configgin
make[1]: Entering directory '/home/user/GOPATH/src/github.com/hpcloud/fissile/scripts/configgin'
/home/user/GOPATH/src/github.com/hpcloud/fissile/scripts/configgin/make/package
Fetching gem metadata from https://rubygems.org/...........
Installing ast 2.2.0
Installing semi_semantic 1.1.0
Installing bosh-template 1.3072.0
Installing deep_merge 1.0.1
Installing diff-lcs 1.2.5
Installing mustache 1.0.2
Installing parser 2.3.0.4
Installing powerpack 0.1.1
Installing rainbow 2.1.0
Installing rspec-support 3.4.1
Installing rspec-core 3.4.2
Installing rspec-expectations 3.4.0
Installing rspec-mocks 3.4.1
Installing rspec 3.4.0
Installing ruby-progressbar 1.7.5
Installing unicode-display_width 0.3.1
Installing rubocop 0.37.2
Using bundler 1.7.3
Your bundle is complete!
It was installed into
/home/user/GOPATH/src/github.com/hpcloud/fissile/scripts/configgin/output/configgin-1.1.8+0.gdb16e86.linux-amd64/lib/vendor
make[1]: Leaving directory '/home/user/GOPATH/src/github.com/hpcloud/fissile/scripts/configgin'
/home/user/GOPATH/src/github.com/hpcloud/fissile/make/bindata
/home/user/GOPATH/src/github.com/hpcloud/fissile/make/build
/home/user/GOPATH/src/github.com/hpcloud/fissile/make/test 
# github.com/hpcloud/fissile/docker
docker/docker_test.go:17:2: import "github.com/hpcloud/fissile/vendor/github.com/golang/mock/mockgen" is a program, not an importable package                                                                                                                                     FAIL    github.com/hpcloud/fissile/docker [setup failed]
?       github.com/hpcloud/fissile      [no test files]
ok      github.com/hpcloud/fissile/app  1.742s  coverage: 32.8% of statements
ok      github.com/hpcloud/fissile/builder      9.939s  coverage: 73.6% of statements
?       github.com/hpcloud/fissile/cmd  [no test files]
ok      github.com/hpcloud/fissile/compilator   16.001s coverage: 75.1% of statements
ok      github.com/hpcloud/fissile/kube 1.843s  coverage: 81.5% of statements
ok      github.com/hpcloud/fissile/model        2.085s  coverage: 81.1% of statements
ok      github.com/hpcloud/fissile/mustache     1.030s  coverage: 77.6% of statements
ok      github.com/hpcloud/fissile/util 1.045s  coverage: 75.8% of statements
ok      github.com/hpcloud/fissile/validation   1.020s  coverage: 76.7% of statements
Makefile:38: recipe for target 'test' failed
make: *** [test] Error 1