cloudfoundry-incubator / fissile Goto Github PK

I can build images successfully, but when I tried to build kube or build helm, the error below showed:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x7bafd4]

goroutine 1 [running]:
github.com/SUSE/fissile/helm.(*Encoder).writeNode(0xc4202d5880, 0x0, 0x0, 0xc4201136d0, 0x0, 0x0, 0x1)
/root/workspace/goworkspace/src/github.com/SUSE/fissile/helm/config.go:561 +0x154
github.com/SUSE/fissile/helm.(*Encoder).Encode(0xc4202d5880, 0x0, 0x0, 0x0, 0x0)
/root/workspace/goworkspace/src/github.com/SUSE/fissile/helm/config.go:438 +0xe0
github.com/SUSE/fissile/app.(*Fissile).generateKubeRoles(0xc42005c840, 0xc420017ac0, 0x1, 0xc420017a10, 0x7, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/root/workspace/goworkspace/src/github.com/SUSE/fissile/app/fissile.go:1078 +0x77a
github.com/SUSE/fissile/app.(*Fissile).GenerateKube(0xc42005c840, 0xc42006a5a0, 0x5e, 0x0, 0x0, 0x0, 0xc420017ac0, 0x1, 0xc420017a10, 0x7, ...)
/root/workspace/goworkspace/src/github.com/SUSE/fissile/app/fissile.go:924 +0x391
github.com/SUSE/fissile/cmd.glob..func2(0xc8a220, 0xcb3698, 0x0, 0x0, 0x0, 0x0)
/root/workspace/goworkspace/src/github.com/SUSE/fissile/cmd/build-helm.go:65 +0x45d
github.com/SUSE/fissile/vendor/github.com/spf13/cobra.(*Command).execute(0xc8a220, 0xcb3698, 0x0, 0x0, 0xc8a220, 0xcb3698)
/root/workspace/goworkspace/src/github.com/SUSE/fissile/vendor/github.com/spf13/cobra/command.go:646 +0x3e8
github.com/SUSE/fissile/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc8aee0, 0x20, 0xc42005c800, 0xc420113e80)
/root/workspace/goworkspace/src/github.com/SUSE/fissile/vendor/github.com/spf13/cobra/command.go:737 +0x2fe
github.com/SUSE/fissile/vendor/github.com/spf13/cobra.(*Command).Execute(0xc8aee0, 0x1, 0xc42005c840)
/root/workspace/goworkspace/src/github.com/SUSE/fissile/vendor/github.com/spf13/cobra/command.go:695 +0x2b
github.com/SUSE/fissile/cmd.Execute(0xc42005c840, 0xa0dce0, 0x19, 0x4, 0x4)
/root/workspace/goworkspace/src/github.com/SUSE/fissile/cmd/root.go:75 +0x69
main.main()
/root/workspace/goworkspace/src/github.com/SUSE/fissile/main.go:45 +0x186

• Remove support for the configuration.templates key in fissile role manifests (both global and role based)
• Remove support for opinions.yaml as well as dark-opinions.yaml

Both templates and opinions will be part of jobs, just like a deployment manifest.
The role manifest should look like:

---
roles:
- name: nats
  jobs:
  - name: nats
    properties:
      nats:
        password: '"((NATS_PASSWORD))"'
        user: nats

@JulzDiverse, @viovanov: As discussed in https://cloudfoundry.slack.com/archives/C8RU3BZ26/p1536217742000100, here is my very-much-work-in-progress version: https://gist.github.com/bkrannich/883e0bea4fe1dd2a060dafbf598418c2

This is an updated version based on earlier work by @zhangtbj in SUSE/scf#1174

Fissile fails make all on openSUSE Leap 42.2 using go 1.8.1 (from the official tarball) with

docker/docker_test.go:17:2: import "github.com/hpcloud/fissile/vendor/github.com/golang/mock/mockgen" is a program, not an importable package

The go environment was empty, the build fails when trying to reproduce the steps from the readme (adding the missing make docker-deps and make tools)

Complete Log:

/home/user/GOPATH/src/github.com/hpcloud/fissile/make/clean
/home/user/GOPATH/src/github.com/hpcloud/fissile/make/format==> Formatting                                                                                                                                                                                                                                                                    /home/user/GOPATH/src/github.com/hpcloud/fissile/make/lint
==> Linting
/home/user/GOPATH/src/github.com/hpcloud/fissile/make/vet
/home/user/GOPATH/src/github.com/hpcloud/fissile/make/configgin
make[1]: Entering directory '/home/user/GOPATH/src/github.com/hpcloud/fissile/scripts/configgin'
/home/user/GOPATH/src/github.com/hpcloud/fissile/scripts/configgin/make/package
Fetching gem metadata from https://rubygems.org/...........
Installing ast 2.2.0
Installing semi_semantic 1.1.0
Installing bosh-template 1.3072.0
Installing deep_merge 1.0.1
Installing diff-lcs 1.2.5
Installing mustache 1.0.2
Installing parser 2.3.0.4
Installing powerpack 0.1.1
Installing rainbow 2.1.0
Installing rspec-support 3.4.1
Installing rspec-core 3.4.2
Installing rspec-expectations 3.4.0
Installing rspec-mocks 3.4.1
Installing rspec 3.4.0
Installing ruby-progressbar 1.7.5
Installing unicode-display_width 0.3.1
Installing rubocop 0.37.2
Using bundler 1.7.3
Your bundle is complete!
It was installed into
/home/user/GOPATH/src/github.com/hpcloud/fissile/scripts/configgin/output/configgin-1.1.8+0.gdb16e86.linux-amd64/lib/vendor
make[1]: Leaving directory '/home/user/GOPATH/src/github.com/hpcloud/fissile/scripts/configgin'
/home/user/GOPATH/src/github.com/hpcloud/fissile/make/bindata
/home/user/GOPATH/src/github.com/hpcloud/fissile/make/build
/home/user/GOPATH/src/github.com/hpcloud/fissile/make/test 
# github.com/hpcloud/fissile/docker
docker/docker_test.go:17:2: import "github.com/hpcloud/fissile/vendor/github.com/golang/mock/mockgen" is a program, not an importable package                                                                                                                                     FAIL    github.com/hpcloud/fissile/docker [setup failed]
?       github.com/hpcloud/fissile      [no test files]
ok      github.com/hpcloud/fissile/app  1.742s  coverage: 32.8% of statements
ok      github.com/hpcloud/fissile/builder      9.939s  coverage: 73.6% of statements
?       github.com/hpcloud/fissile/cmd  [no test files]
ok      github.com/hpcloud/fissile/compilator   16.001s coverage: 75.1% of statements
ok      github.com/hpcloud/fissile/kube 1.843s  coverage: 81.5% of statements
ok      github.com/hpcloud/fissile/model        2.085s  coverage: 81.1% of statements
ok      github.com/hpcloud/fissile/mustache     1.030s  coverage: 77.6% of statements
ok      github.com/hpcloud/fissile/util 1.045s  coverage: 75.8% of statements
ok      github.com/hpcloud/fissile/validation   1.020s  coverage: 76.7% of statements
Makefile:38: recipe for target 'test' failed
make: *** [test] Error 1

Currently, in the role manifest:

• the manifest has multiple instance groups
• each instance group has a bunch of jobs
• each job has properties
• inside the properties for each job, there is a bosh_containerization section
• inside of that, there is the run information, such as limits, affinities, and scaling information

This makes no sense: you can't scale each job apart from the pods ( = instance groups) they are in. They should be moved to under instance groups (though we may want to keep limits per-job in case we get to container-per-job). Otherwise it's unclear what happens if I have two jobs in an instance group, one saying I can't HA it, and one saying I can…

I tried the walkthrough manifests at https://github.com/cloudfoundry-incubator/fissile/blob/develop/docs/configuration.md and failed multiple times.

Some of the bugs I found:

• yaml structure no longer complies with what fissile cli expects. variables node must be at top level.
• bosh_containerization property is required on each job
• tags: indexed no longer works
• must use dark-opinions.yml file even though I did not need them for my bosh release
• can't have empty dark-opinions.yml (or just a properties yml node)
• variable node structure different. e.g. description must be under options node instead

I finally succeeded by looking at the code and using the test assets as template for my own manifests.

I suggest to add the walkthrough example manifests to daily builds to make sure they work.

• Rename roles to instance_groups
• For jobs definitions, use release instead of release_name
• Change secret generator so it supports consuming information from a variables section
• Convert variables and secret information to a variables section in SCF/UAA
• Move containerization-specific configuration to <job>.properties.bosh_containerization
• cf-deployment + an ops file == role manifest

For HA pods like the diego-api with active/passive roles, there is a race condition that will not allow the skiff-role-active label to be set to its correct state(true/false).

For the specific use case(diego-api pod), this can happen when the readines-probe.sh its executed when the bbs job in the leader pod stops working(not marked as running), which will block the readiness-probe.sh on reaching the script part where the label can be set to false.

An example of such situation is when a connection to the db is lost, which will leave one diego-api pod marked as skiff-role-active = true, when that value is not true.

This is not critical, but I considered important to highlight.

testing a simple bosh release, and the container is not running.

10.202.128.91	proxy-0.proxy-set.default.svc.cluster.local proxy-0
/usr/local/rvm/gems/ruby-2.3.1/gems/kubeclient-2.5.2/lib/kubeclient/common.rb:117:in `rescue in handle_exception': Unauthorized (KubeException)
	from /usr/local/rvm/gems/ruby-2.3.1/gems/kubeclient-2.5.2/lib/kubeclient/common.rb:109:in `handle_exception'
	from /usr/local/rvm/gems/ruby-2.3.1/gems/kubeclient-2.5.2/lib/kubeclient/common.rb:474:in `fetch_entities'
	from /usr/local/rvm/gems/ruby-2.3.1/gems/kubeclient-2.5.2/lib/kubeclient/common.rb:464:in `load_entities'
	from /usr/local/rvm/gems/ruby-2.3.1/gems/kubeclient-2.5.2/lib/kubeclient/common.rb:121:in `discover'
	from /usr/local/rvm/gems/ruby-2.3.1/gems/kubeclient-2.5.2/lib/kubeclient/common.rb:88:in `method_missing'
	from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.17.0/lib/job.rb:40:in `self_pod'
	from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.17.0/lib/job.rb:44:in `self_role'
	from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.17.0/lib/job.rb:14:in `initialize'
	from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.17.0/bin/configgin:65:in `new'
	from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.17.0/bin/configgin:65:in `block in <top (required)>'
	from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.17.0/bin/configgin:52:in `each'
	from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.17.0/bin/configgin:52:in `<top (required)>'
	from /usr/local/rvm/gems/ruby-2.3.1/bin/configgin:23:in `load'
	from /usr/local/rvm/gems/ruby-2.3.1/bin/configgin:23:in `<main>'
	from /usr/local/rvm/gems/ruby-2.3.1/bin/ruby_executable_hooks:24:in `eval'
	from /usr/local/rvm/gems/ruby-2.3.1/bin/ruby_executable_hooks:24:in `<main>'

The cluster has a self signed CA and all api clients use a client cert, I assume I am missing some configuration in my role manifest, but I am unsure where to put it.

Every container failed with the same error msg as below: What is the problem?

/usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.12.0.pre/lib/generate.rb:24:in rescue in generate': failed to read template file /opt/fissile/monitrc.erb (RuntimeError) from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.12.0.pre/lib/generate.rb:19:in generate'
from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.12.0.pre/bin/configgin:34:in block (2 levels) in <top (required)>' from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.12.0.pre/bin/configgin:23:in each'
from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.12.0.pre/bin/configgin:23:in block in <top (required)>' from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.12.0.pre/bin/configgin:20:in each'
from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.12.0.pre/bin/configgin:20:in <top (required)>' from /usr/local/rvm/gems/ruby-2.3.1/bin/configgin:23:in load'
from /usr/local/rvm/gems/ruby-2.3.1/bin/configgin:23:in <main>' from /usr/local/rvm/gems/ruby-2.3.1/bin/ruby_executable_hooks:15:in eval'
from /usr/local/rvm/gems/ruby-2.3.1/bin/ruby_executable_hooks:15:in `

I'm trying to debug a bosh release -> fissile -> cf-operator [1] and want to create a dev release -> fissile build release-images but get an error:

$ version=15.3.3.$(date +%s)
$ bosh create-release --tarball tmp/redis-$version.tgz
$ fissile build release-images --stemcell splatform/fissile-stemcell-opensuse:42.3-38.g82067a9-30.95 --name redis --url tmp/redis-$version.tgz --sha1 $(sha1sum tmp/redis-$version.tgz | awk '{print $1}') --version $version -w $PWD/tmp/fissile
Error loading release information: 2 errors occurred:
	* Dev release redis is not supported as manifest references
	* open /Users/drnic/.final-releases/redis-15.3.3.1569804399-66bae730f6a83ef317b9c18cb1f9f278d3d54298/release.MF: no such file or directory

Why can't I create a fissile OCI from a dev release?

[1] cloudfoundry-incubator/quarks-operator#596

Hi,

We currently have a strong use case for supporting multiple containers per Pod(aka sidecars). By default in Kube, a pod configuration file allows to have multiple entries under spec.containers.

However, the current setup of fissile(e.g. running a fissile build helm), does not support this by default. We can always patch the YAML templates generated by fissile, but ideally we want to support this sidecar feature as part of the fissile natural behavior(e.g. throught fissile build helm).

We would like to open a discussion on how to achieve this, and then PRing this to you. The main concern is that we will need to modify/introduce some of the YAML syntax for the role-manifest .

Our current idea would be to introduce two new keys in the role-manifest.yml, as follows:

roles:
- name: nginx
  jobs:
  - name: nginx
    release_name: nginx
  tags:
  - clustered
  sidecars:                 # new list to reference sidecar roles
  - log-forwarder
  run:
    memory: 256
    virtual-cpus: 4
    exposed-ports: [...]
    shared-volumes:
      - path: /mnt/share
        tag: common-share
        empty-dir: true    # new reference for a "volumes" entry in the template YAML
- name: log-forwarder
  type: sidecar            # new type called sidecar
  jobs:
  - name: elk
    release_name: elk
  tags:
  - clustered
  run:
    memory: 256
    virtual-cpus: 4
    exposed-ports: [...]
    shared-volumes:
      - path: /mnt/share
        tag: common-share
        empty-dir: true    # new reference for a "volumes" entry in the template YAML

As illustrated above, the roles.nginx.sidecars will ideally be the trigger to colocate in the nginx pod template, a second item under spec.containers with the properties of the log-forwarder sidecar.

Inside the log-forwarder role, the roles.component.type will indicate that this Pod YAML template should not be generated, whilst we expect the docker image to be generated.

With this sidecar, we would also require a simple volume sharing solution. For example, as defined in the Creating a Pod that runs two Containers scenario, we just want to share one directory between the containers. For that, an additional role-manifest.yml modification would be required to enable support for emptyDir: {}.

What do you think?

Thanks in advance for your feedback.
Regards,
@HeavyWombat @qu1queee

As announced here, deprecated APIs are being removed from K8s 1.16, which is scheduled to be released tomorrow, 18 Sep 2019 as specified here.

The APIs that affect fissiled helm charts are statefulsets and deployments. Fissile should generate them in apps/v1 API and not apps/v1beta1 or extensions/v1beta1

Hi,

I want to build a docker image from bosh stemcells following guide. Since I can't access the pipeline, I'm not sure what the bosh stemcell stages are included to build the image. Would you please help provide the detailed stage list?

Thanks a lot!

There have been a few recent changes that assumes the project uses the new https://github.com/SUSE/scf-secret-generator mechanism to manage secrets that break the automatic generation of password when the generator pod isn't used.

These changes should be gated by a feature flag and the backwards compatibility to the old functionality should be restored.

With PR #340, a collision check for colocated roles has been introduced that has an issue when you specify the same port number with different protocols, because the collision check only checks the port number.

- name: consul
  colocated_containers:
  - logs-forwarder
  run:
    exposed-ports:
    - name: serf-tcp
      extenral: 8301
      internal: 8301
      protocol: TCP
    - name: serf-udp
      extenral: 8301
      internal: 8301
      protocol: UDP

The configuration is totally fine, but the collision checks reports that as an issue: Error loading roles manifest: role[consul]: Invalid value: 8301: port collision, the same port is used by: consul, consul

To resolve this, the combination of protocol and port has to be checked against one another, for example: TCP-8301 and UDP-8301. I will start to work on a fix for this as soon as possible.

When trying to Fissilify a simple BOSH release, I followed the walkthrough and upon deployment, I received the error pods "spacebearsdbnode-0" is forbidden: User "system:serviceaccount:simple:default" cannot get pods in the namespace "simple": Unknown user "system:serviceaccount:simple:default" (KubeException). My Helm chart didn't include any service accounts.

@viovanov kindly pointed out that I'd need to add auth roles for configgin and secrets.

This isn't mentioned anywhere in the walkthrough, and if it's mandatory, Fissile should probably make these for the user.

We ran into an issue where the role manifest validation failed for our setup, where one instance group has multiple colocated container instance groups. Not all of the colocated instance groups have an emptyDir volume defined.

However, the validation we initially wrote for fissile did not take that scenario into account and simply checked that the number of volumes match. This does not work for the aforementioned setup. The number of colocated containers is two, but the number of volumes is one. Therefore, the validation fails.

instance_groups:
- name: main-instance-group
  jobs:
  - name: tor
    release: tor
    properties:
      bosh_containerization:
        colocated_containers:
        - to-be-colocated-1
        - to-be-colocated-2
        run:
          memory: 1
          volumes:
          - path: /shared/data
            type: emptyDir
            tag: shared-data

- name: to-be-colocated-1
  type: colocated-container
  jobs:
  - name: tor
    release: tor
    properties:
      bosh_containerization:
        run:
          memory: 1
          volumes:
          - path: /shared/data
            type: emptyDir
            tag: shared-data

- name: to-be-colocated-2
  type: colocated-container
  jobs:
  - name: ntpd
    release: ntp
    properties:
      bosh_containerization:
        # no volume defined here ...
        run:
          memory: 1

• Spike to find what information we need to be able to remove monit?
• Spike to see how we can use BPM to get the information needed at 1)
• Use knowledge from 1) and 2) generate helm charts with an alternative entrypoint
• Add no-monit for metron in SCF

• Create a colocated container in SCF for metron
• Replace all metron jobs in SCF with the colocated container
• Make sure all SCF/UAA tests pass

In role-manifest.yml, the "templates" part contains some properties, there are several formats:
templates:
one: '"((ONE))"'
two: '((TWO))'
three: ((THREE))

What is the difference between '"(())"', '(())' and (())?

• Change helm charts so they generate one service per job
• Remove the headless tag from fissile
• Exposed ports have to be moved to the job section
• Update fissile doc
• Update SCF/UAA

Use the following rule to determine the service name (which in turn determines the hostname):

<role-name>-<job-name>

Ports must be exposed like this:

roles:
- name: nats
  jobs:
  - name: nats
    properties:
      bosh_containerization:
        ports:
        - name: nats
          protocol: TCP
          internal: 4222
        - name: nats-routes
          protocol: TCP
          internal: 4223

Waiting for mholt/archiver#92

We have a local change to the vendored dependency in #414, which should be replaced by the proper fix as soon as available.

• Create a github repository for it
• Talk to the guys at SAP to get the latest work
• Test to see everything works
• Update the fissile README to point to the new repo

I use fissile build images successfully. In the work dir there is a dockerfiles directory, but no dockerfile in it. How to find the docker files?

I made a role manifest to build the docker image successfully, but when I tried to use docker run $imageName:tag, I got the error as below:
/usr/local/rvm/gems/ruby-2.3.1/gems/bosh-template-1.3262.24.0/lib/bosh/template/evaluation_context.rb:93:in p': Can't find property '["fissile.monit.port"]' (Bosh::Template::UnknownProperty) from /opt/scf/monitrc.erb:9:in get_binding'
from /usr/local/rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/erb.rb:864:in eval' from /usr/local/rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/erb.rb:864:in result'
from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.12.0.pre/lib/generate.rb:30:in generate' from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.12.0.pre/bin/configgin:34:in block (2 levels) in <top (required)>'
from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.12.0.pre/bin/configgin:23:in each' from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.12.0.pre/bin/configgin:23:in block in <top (required)>'
from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.12.0.pre/bin/configgin:20:in each' from /usr/local/rvm/gems/ruby-2.3.1/gems/configgin-0.12.0.pre/bin/configgin:20:in <top (required)>'
from /usr/local/rvm/gems/ruby-2.3.1/bin/configgin:23:in load' from /usr/local/rvm/gems/ruby-2.3.1/bin/configgin:23:in

so we can deploy one Cloud Foundry clusters on kubernetes quickly

~/go/src/github.com/cloudfoundry-incubator/fissile% make build
==> Binary Data 
==> Building 
# github.com/cloudfoundry-incubator/fissile
./main.go:43:32: cannot use ui (type *"github.com/cloudfoundry-incubator/fissile/vendor/github.com/SUSE/termui".UI) as type *"github.com/SUSE/fissile/vendor/github.com/SUSE/termui".UI in argument to app.NewFissileApplication
make: *** [Makefile:29: build] Error 2

Since we moved the repos, fissile won't build from a new checkout. I think we need to update all the import statements?

% git grep github.com/SUSE/fissile/ |wc -l
125

Fix:
perl -pi -e '[email protected]/SUSE/fissile/@github.com/cloudfoundry-incubator/fissile/@' **/*.go

• Spike to see if hairpin is required for talking to another container in the same pod
• Create colocated roles for the jobs that need to talk back to a colocated job
• Test that everything works in an HA deployment

First part of this effort is here.
Second part requires that jobs can talk back to other colocated jobs.
One service per job is also a requirement.

Its a year since last github release and 352 commits since then. I've discovered the magic S3 bucket with builds from CI; but will we return to github releases with release notes? Especially for notifications of updates to the fissile build release-images command :)

For anyone else wondering how to get latest fissile, here's an example command to download/install latest darwin fissile into $GOPATH/bin:

aws s3 cp "s3://cf-opensusefs2/fissile/develop/$(aws s3 ls s3://cf-opensusefs2/fissile/develop/ | sort | grep darwin | tail -n1 | awk '{print $4}')" - | tar -C $GOPATH/bin -xvz fissile

This line does not work when we don't need credentials for Docker hub

"chmod 1730 /var/spool/cron/tabs/"
The directory "/var/spool/cron/tabs/" is not accessible in ubuntu os, it is specific for openSuSe

The build instructions seem complicated -- would they be greatly simplified if we move to Go Modules?

I talked to @viovanov and if I understood correctly dark opinions are no longer necessary and could be removed from the source code and documentation.

When Istio is in use on a Kubenrnetes cluster, port names of all Services have to follow the naming convention required by Istio. This is also true when the components generated by fissile are not using Istio themselves. While this can be considered a design flaw and might be fixed in future we have to deal with it somehow in the meanwhile.

Here are a couple of things that could be done to mitigate the issue:

• Add networking.istio.io/exportTo= . annotation to the Services generated by fissile which are not supposed to be accessed through Istio - helps to work around istio/istio#9784 and istio/istio#14520
• Rename ports to comply with the naming conventions required by Istio https://istio.io/docs/setup/kubernetes/additional-setup/requirements keeping in mind that a certain port number must have the same name prefix over the whole cluster unless the service is scoped to a single namespace by networking.istio.io/exportTo= ., e.g. for port 443 the name must always start from https - helps with istio/istio#14520

cc: @iandyh @viovanov

Update:

• I looks like I'm mixing two problems here (istio/istio#9784 and istio/istio#14520) though it's fine as they are similar in a way they affect the rest of the cluster and the approaches towards solving the issues are likely to be connected

• Change configgin so it accepts input from a BOSH deployment manifest
• Change fissile so it generates pods that mount a deployment manifest secret
• Add --deployment-manifest-template to fissile build helm

Full spec for this feature lives here.

I tried to go through the example in https://github.com/cloudfoundry-incubator/fissile/blob/develop/docs/configuration.md, but i got stuck at the first building step: https://github.com/cloudfoundry-incubator/fissile/blob/develop/docs/configuration.md#building-the-nats-image

docker: Error response from daemon: OCI runtime create failed: container_linux.go:344: starting container process caused "exec: \"/usr/local/bin/create-release.sh\": stat /usr/local/bin/create-release.sh: no such file or directory": unknown.

Apparently, the referenced docker images has no sign of the create-release.sh script in it.

https://github.com/cloudfoundry-incubator/fissile/blob/develop/docs/configuration.md#building-the-nats-image contains a docker run command that fails. What is the correct command for the image?

$ docker run --rm \
    --volume "${HOME}/.bosh/cache:/bosh-cache" \
    --volume "${PWD}/nats-release:${PWD}/nats-release" \
    --env "RUBY_VERSION=2.2.3" \
    splatform/bosh-cli \
    /usr/local/bin/create-release.sh \
        "$(id -u)" "$(id -g)" /bosh-cache --dir "${PWD}/nats-release" --force --name "nats"
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"/usr/local/bin/create-release.sh\": stat /usr/local/bin/create-release.sh: no such file or directory": unknown.

I found a bosh.sh and quickly tried /usr/local/bin/bosh.sh create-release but it failed a different way:

docker run --rm \
    --volume "${HOME}/.bosh/cache:/bosh-cache" \
    --volume "${PWD}/nats-release:${PWD}/nats-release" \
    --env "RUBY_VERSION=2.2.3" \
    splatform/bosh-cli \
    /usr/local/bin/bosh.sh create-release \
        "$(id -u)" "$(id -g)" /bosh-cache --dir "${PWD}/nats-release" --force --name "nats"
/usr/local/bin/bosh.sh: line 27: addgroup: command not found

We've had multiple instances (from people on our team) where the stemcell wasn't downloaded, and we just report a vague and misleading "missing image" instead of being more specific about stemcell failures:

Compiling packages for dev releases:
         uaa (29+dev.1)
         cf-mysql (34+dev.10)
         hcf (0+dev.1)
compile: cf-mysql/cf-mysql-common
compile: cf-mysql/boost
done:    cf-mysql/cf-mysql-common
result   > failure: cf-mysql/cf-mysql-common - Error compiling package cf-mysql-common: no such image

We should have a better error message here so we can stop wasting time tripping over this over an over 😄

I expected to be able to run

$ fissile diff --release https://bosh.io/d/github.com/cloudfoundry/nats-release?v=23,https://bosh.io/d/github.com/cloudfoundry/nats-release?v=24

But that doesn't work:

dev config diff: error loading release information: Error loading dev release information: Path /Users/jan/go/src/code.cloudfoundry.org/fissile/https:/bosh.io/d/github.com/cloudfoundry/nats-release?v=23 (release directory) does not exist

Hi, I follow the steps :

$ go get -d github.com/SUSE/fissile       # Download sources
$ cd $GOPATH/src/github.com/SUSE/fissile
$ make tools                              # install required tools; only needed first time
$ make docker-deps                        # pull docker images required to build
$ make all

Then in the last step "make all", the following error shows. Any idea?

/Users/qiyang/dev/goworkspace/fissile/make/clean
/Users/qiyang/dev/goworkspace/fissile/make/format
==> Formatting
diff -u /Users/qiyang/dev/goworkspace/fissile/app/fissile.go.orig /Users/qiyang/dev/goworkspace/fissile/app/fissile.go
--- /Users/qiyang/dev/goworkspace/fissile/app/fissile.go.orig	2017-07-20 10:44:02.000000000 +0800
+++ /Users/qiyang/dev/goworkspace/fissile/app/fissile.go	2017-07-20 10:44:02.000000000 +0800
@@ -22,7 +22,6 @@

 	"github.com/fatih/color"
 	"github.com/joho/godotenv"
-	"gopkg.in/yaml.v2"
 )

 // OutputFormat is one of the known output formats for commands showing loaded
diff -u /Users/qiyang/dev/goworkspace/fissile/builder/role_image.go.orig /Users/qiyang/dev/goworkspace/fissile/builder/role_image.go
--- /Users/qiyang/dev/goworkspace/fissile/builder/role_image.go.orig	2017-07-20 10:44:02.000000000 +0800
+++ /Users/qiyang/dev/goworkspace/fissile/builder/role_image.go	2017-07-20 10:44:02.000000000 +0800
@@ -20,7 +20,6 @@

 	"github.com/fatih/color"
 	workerLib "github.com/jimmysawczuk/worker"
-	"gopkg.in/yaml.v2"
 )

 const (
diff -u /Users/qiyang/dev/goworkspace/fissile/compilator/compilator.go.orig /Users/qiyang/dev/goworkspace/fissile/compilator/compilator.go
--- /Users/qiyang/dev/goworkspace/fissile/compilator/compilator.go.orig	2017-07-20 10:44:02.000000000 +0800
+++ /Users/qiyang/dev/goworkspace/fissile/compilator/compilator.go	2017-07-20 10:44:02.000000000 +0800
@@ -22,7 +22,6 @@
 	"github.com/fatih/color"
 	workerLib "github.com/jimmysawczuk/worker"
 	"github.com/pborman/uuid"
-	"github.com/termie/go-shutil"
 )

 const (
diff -u /Users/qiyang/dev/goworkspace/fissile/model/dev_release.go.orig /Users/qiyang/dev/goworkspace/fissile/model/dev_release.go
--- /Users/qiyang/dev/goworkspace/fissile/model/dev_release.go.orig	2017-07-20 10:44:02.000000000 +0800
+++ /Users/qiyang/dev/goworkspace/fissile/model/dev_release.go	2017-07-20 10:44:02.000000000 +0800
@@ -8,7 +8,6 @@
 	"github.com/SUSE/fissile/util"

 	"github.com/cppforlife/go-semi-semantic/version"
-	"gopkg.in/yaml.v2"
 )

 // NewDevRelease will create an instance of a BOSH development release
diff -u /Users/qiyang/dev/goworkspace/fissile/model/job.go.orig /Users/qiyang/dev/goworkspace/fissile/model/job.go
--- /Users/qiyang/dev/goworkspace/fissile/model/job.go.orig	2017-07-20 10:44:02.000000000 +0800
+++ /Users/qiyang/dev/goworkspace/fissile/model/job.go	2017-07-20 10:44:02.000000000 +0800
@@ -14,7 +14,6 @@
 	"github.com/SUSE/fissile/util"

 	"github.com/pivotal-golang/archiver/extractor"
-	"gopkg.in/yaml.v2"
 )

 // Job represents a BOSH job
diff -u /Users/qiyang/dev/goworkspace/fissile/model/opinions.go.orig /Users/qiyang/dev/goworkspace/fissile/model/opinions.go
--- /Users/qiyang/dev/goworkspace/fissile/model/opinions.go.orig	2017-07-20 10:44:02.000000000 +0800
+++ /Users/qiyang/dev/goworkspace/fissile/model/opinions.go	2017-07-20 10:44:02.000000000 +0800
@@ -4,8 +4,6 @@
 	"fmt"
 	"io/ioutil"
 	"reflect"
-
-	"gopkg.in/yaml.v2"
 )

 // Opinions holds the light and dark opinions given to fissile
diff -u /Users/qiyang/dev/goworkspace/fissile/model/release.go.orig /Users/qiyang/dev/goworkspace/fissile/model/release.go
--- /Users/qiyang/dev/goworkspace/fissile/model/release.go.orig	2017-07-20 10:44:02.000000000 +0800
+++ /Users/qiyang/dev/goworkspace/fissile/model/release.go	2017-07-20 10:44:02.000000000 +0800
@@ -8,8 +8,6 @@
 	"regexp"

 	"github.com/SUSE/fissile/util"
-
-	"gopkg.in/yaml.v2"
 )

 // Release represents a BOSH release
diff -u /Users/qiyang/dev/goworkspace/fissile/model/roles.go.orig /Users/qiyang/dev/goworkspace/fissile/model/roles.go
--- /Users/qiyang/dev/goworkspace/fissile/model/roles.go.orig	2017-07-20 10:44:02.000000000 +0800
+++ /Users/qiyang/dev/goworkspace/fissile/model/roles.go	2017-07-20 10:44:02.000000000 +0800
@@ -12,8 +12,6 @@
 	"strings"

 	"github.com/SUSE/fissile/validation"
-
-	"gopkg.in/yaml.v2"
 )

 // RoleType is the type of the role; see the constants below
diff -u /Users/qiyang/dev/goworkspace/fissile/src/github.com/axw/gocov/gocovutil/packages.go.orig /Users/qiyang/dev/goworkspace/fissile/src/github.com/axw/gocov/gocovutil/packages.go
--- /Users/qiyang/dev/goworkspace/fissile/src/github.com/axw/gocov/gocovutil/packages.go.orig	2017-07-20 10:44:02.000000000 +0800
+++ /Users/qiyang/dev/goworkspace/fissile/src/github.com/axw/gocov/gocovutil/packages.go	2017-07-20 10:44:02.000000000 +0800
@@ -2,10 +2,11 @@

 import (
 	"encoding/json"
-	"github.com/axw/gocov"
 	"io/ioutil"
 	"os"
 	"sort"
+
+	"github.com/axw/gocov"
 )

 // Packages represents a set of gocov.Package structures.
diff -u /Users/qiyang/dev/goworkspace/fissile/util/serialization_helper_test.go.orig /Users/qiyang/dev/goworkspace/fissile/util/serialization_helper_test.go
--- /Users/qiyang/dev/goworkspace/fissile/util/serialization_helper_test.go.orig	2017-07-20 10:44:05.000000000 +0800
+++ /Users/qiyang/dev/goworkspace/fissile/util/serialization_helper_test.go	2017-07-20 10:44:05.000000000 +0800
@@ -8,7 +8,6 @@
 	"github.com/SUSE/fissile/testhelpers"

 	"github.com/stretchr/testify/assert"
-	"gopkg.in/yaml.v2"
 )

 type jsonHelpInputTestData struct {
make: *** [format] Error 1

Some jobs don't require a service. Nothing connects to them, or they advertise themselves through an internal mechanism.
BOSH/fissile needs a way to describe this.

This will improve #367

@viovanov @Tibeos:

The problem is that relative symlinks are turned into absolute symlinks:

# ls -l /var/vcap/packages/postgres-9.6.6/lib/libpq.so.5
lrwxrwxrwx 1 root root 140 Sep 28 19:42 /var/vcap/packages/postgres-9.6.6/lib/libpq.so.5 -> /home/vagrant/.fissile/compilation/4c162e7204bd9cd84142a77c6de81eaf154bf16a/64abaa2d40d8a9482fa1a4ee8b9e8301b6898dc8/compiled/lib/libpq.so.5

This should have been a simple relative symlink:

# ls -l /var/vcap/packages/postgres-9.6.6/lib/libpq.so.5
lrwxrwxrwx 1 root root 12 Sep 11 01:35 /var/vcap/packages/postgres-9.6.6/lib/libpq.so.5 -> libpq.so.5.9

The problem arises because the absolute symlink points into the fissile compilation cache directory (which doesn't exist inside the container) and not to the actual install location.

This seems to be a known limitation of the archiver package: mholt/archiver#74

Hello happy ppl. Could your CI publish linux/darwin binaries to github releases?

When dev.yml looks like below. The commit_hash is a scientific format string, fissile build will fail:

name: scf
version: 0+dev.1
commit_hash: 9018e89
uncommitted_changes: true
jobs:

Error:

Please allow a long time for mariadb to compile
Error loading dev release information: Error trying to load release scf metadata from YAML manifest /home/vagrant/scf/src/scf-release/dev_releases/scf/scf-0+dev.1.yml: interface conversion: interface {} is float64, not string
Makefile:258: recipe for target 'compile' failed
make: *** [compile] Error 1

Hello
We as a Fujitsu are very interested to run CF in Kubernetes. I am not a CF expert. I have experience in k8s.
I've seen a presentation how to do it but I can not find and documentation or examples which could help me. Can someone drive me ho to do it ? Any help is appreciated.
I assume I have to convert CF components to Docker images and create pods. So If someone have some scripts or k8s deployment file it would be very helpful.
Please excuse my lack of knowledge because I've already started picking up knowledge about CF and fissile