cloudfoundry / eirini-release Goto Github PK

Helm release for Project Eirini

License: Apache License 2.0

Shell 100.00%

eirini release cloudfoundry kubernetes helm-chart helm

eirini-release's Introduction

Eirini Release

This is a helm release for Project Eirini. In a nutshell Eirini is a Kubernetes backend for Cloud Foundry, made in the effort to decouple Cloud Foundry from Diego, the only current option of a scheduler. It deploys CF apps to a kube backend, using OCI images and Kube deployments.

Installation

The following CFAR (Cloud Foundry Application Runtime) distributions deploy CF on top of Kubernetes and bundle Eirini with it:

Building the yaml release

To build the pure yaml files included in our release please run:

./scripts/render-templates.sh <system-namespace> <output-directory>

This will produce the yamls for all eirini components in separate directories. The components needed for cf-for-k8s are core, events and workloads.

Security

Security Overview

Of an overview of how secure Eirini is compared to other popular container runtimes please look at this table

Securing the Eirini Deployment

To learn about how you can use Kubernetes security primitives to make your deployment more secure, please take a look at our Security Guidelines.

Scalability

As of v1.5.0 a single instance of the eirini deployment can take a sustained load of 90 parallel desire LRP operations. A desire operation takes about 300ms on average when under load.

In order to better understand this result we have to state some condtitions that we assumed when performing the tests:

Performing well under load means that the eirini server will respond in less than 30s.
The tests were performed directly against the Eirini API (bypassing the cloud controller) and agains a sufficiently large cluster in order to make sure the eirini is the only bottleneck. So these results apply to eirini in isolation. The whole cf system will be as scalable as it's weakest subsystem.
There results describe the throughput of eirini itself. Our measurements apply from the moment a desire request is placed to the moment a stateful set is created on Kubernetes. These are not scalability results for Kubernetes.

For details about high availability see this doc.

Differences with Diego

We are working hard towards feature parity with Diego, but still there are some differences in behaviour

Environment variables

It is not possible to set environment variables containing : to your apps containers because of Kubernetes restrictions.

Docker images running with the root user

By default Eirini does not allow docker images running with the root user. Diego allows this because the application runs in a separate user namespace, which is not supported in Kubernetes as of now. However, you can configure Eirini to allow such docker images - see Security Guidelines for more information.

Task retries and parallelism

Tasks in Diego are run at most once and once completed you can determine whether they failed or not. In Eirini we run tasks as Jobs in Kubernetes with both completions and parallelism set to 1. However, as per the Kubernetes documentation, there is no guarantee that the task won't be ran more than once.

Troubleshooting

Disk full on blobstore

If all the CF apps are running, it is safe to delete all files in /var/vcap/store/shared/cc-droplets/sh/a2/ directory on the blobstore-0 pod. To do so, you can run this command:

kubectl exec -n <cf-system-namespace> blobstore-0 -c blobstore -- \
  /bin/sh -c 'rm -rf /var/vcap/store/shared/cc-droplets/sh/a2/sha256:*'

Resources

Eirini Continuous Integration Pipeline

eirini-release's People

Contributors

Stargazers

Watchers

eirini-release's Issues

Uneven distribution of load on Dopplers

Description

In our eirini-setup we have 4 dopplers deployed. But the load is going to one doppler alone.
Is doppler intended only for HA ? Because general Cloud Foundry setup we can see load being served by all the doppler instances

Steps to reproduce

Deploy 4 replicas of dopplers and generate load

What was expected to happen

Workload should be distributed across the replicas

What actually happened

Complete workload flows to only one replica

Additional information (optional)

Eirini Release : 0.16.0

Unable to stage due to registry connection issues

We have an scf+eirini (on GCP and CFCR) that looks like it can stage, but fails at the Waiting for app to start step. We see k8s attempt to pull the image for the app pod from the registry but gets a connection refused error.

Looking closer. Possibly related we (also) appear to be hitting an error in the post-deployment-setup job at the Configuring Eirini staging secrets step. It looks like kubectl isn’t able to connect to the kubernetes deployment to transfer the secrets from the scf namespace to the opi namespace. We think that the problem is our EIRINI_KUBE_CONFIG in our config-values.yml file, possibly combined with the kube.external_ips also being wrong. Can you explain these settings a bit more, plus suggest any other debugging tips here. We assume that this post-deployment-setup job needs to complete successfully before we will be able to push apps to the platform. But it is just that, an assumption.

Many thanks

CF CLI command restart-app-instance fails with Eirini

Description

A cf restart-app-instance [...] CLI call fails with Eirini backend.

Steps to reproduce

cf push my-app -i 2
cf restart-app-instance my-app 1

What was expected to happen

The respective instance of the app is restarted.

What actually happened

cf restart-app-instance my-app 1
Restarting instance 1 of application my-app as [email protected]
FAILED
Server error, status code: 500, error code: 10001, message: An unknown error occurred.

The cloud controller logs give the following more detailed clue: undefined method 'stop_index' for #<OPI::Client:0x000055727e125188>

{
  "timestamp": 1553596199.2098746,
  "message": "Request failed: 500: {
    \"error_code\" : \"UnknownError\", \"description\" : \"An unknown error occurred.\", \"code\" : 10001, \"test_mode_info\" : {
      \"description\" : \"undefined method `stop_index' for #<OPI::Client:0x000055727e125188>\", \"error_code\" : \"CF-stop_index\", \"backtrace\" : [\"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/cloud_controller_ng/lib/cloud_controller/diego/messenger.rb:35:in `send_stop_index_request'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/cloud_controller_ng/lib/cloud_controller/diego/runner.rb:37:in `block in stop_index'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/cloud_controller_ng/lib/cloud_controller/diego/runner.rb:41:in `with_logging'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/cloud_controller_ng/lib/cloud_controller/diego/runner.rb:37:in `stop_index'\", \"/var/vcap/packages-src/726cbb2214e138b576700db6a30698edb2b994e2/lib/ruby/2.4.0/delegate.rb:83:in `method_missing'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/cloud_controller_ng/lib/cloud_controller/index_stopper.rb:8:in `stop_index'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/cloud_controller_ng/app/controllers/runtime/instances_controller.rb:43:in `kill_instance'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/cloud_controller_ng/app/controllers/base/base_controller.rb:86:in `dispatch'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/cloud_controller_ng/lib/cloud_controller/rest_controller/routes.rb:16:in `block in define_route'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1634:in `call'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1634:in `block in compile!'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:992:in `block (3 levels) in route!'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1011:in `route_eval'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:992:in `block (2 levels) in route!'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1040:in `block in process_route'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1038:in `catch'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1038:in `process_route'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:990:in `block in route!'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:989:in `each'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:989:in `route!'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1097:in `block in dispatch!'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1076:in `block in invoke'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1076:in `catch'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1076:in `invoke'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1094:in `dispatch!'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:924:in `block in call!'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1076:in `block in invoke'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1076:in `catch'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1076:in `invoke'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:924:in `call!'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:913:in `call'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/rack-protection-2.0.5/lib/rack/protection/xss_header.rb:18:in `call'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/rack-protection-2.0.5/lib/rack/protection/path_traversal.rb:16:in `call'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/rack-protection-2.0.5/lib/rack/protection/json_csrf.rb:26:in `call'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/rack-protection-2.0.5/lib/rack/protection/base.rb:50:in `call'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/rack-protection-2.0.5/lib/rack/protection/base.rb:50:in `call'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/rack-protection-2.0.5/lib/rack/protection/frame_options.rb:31:in `call'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/rack-2.0.6/lib/rack/null_logger.rb:9:in `call'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/rack-2.0.6/lib/rack/head.rb:12:in `call'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:194:in `call'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1957:in `call'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/rack-2.0.6/lib/rack/urlmap.rb:68:in `block in call'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `each'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `call'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/cloud_controller_ng/middleware/request_logs.rb:22:in `call'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/cloud_controller_ng/middleware/security_context_setter.rb:19:in `call'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/cloud_controller_ng/middleware/vcap_request_id.rb:15:in `call'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/cloud_controller_ng/middleware/cors.rb:49:in `call_app'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/cloud_controller_ng/middleware/cors.rb:14:in `call'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/cloud_controller_ng/middleware/request_metrics.rb:12:in `call'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/rack-2.0.6/lib/rack/builder.rb:153:in `call'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/thin-1.7.2/lib/thin/connection.rb:86:in `block in pre_process'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/thin-1.7.2/lib/thin/connection.rb:84:in `catch'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/thin-1.7.2/lib/thin/connection.rb:84:in `pre_process'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/thin-1.7.2/lib/thin/connection.rb:50:in `block in process'\", \"/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/gem_home/ruby/2.4.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:1067:in `block in spawn_threadpool'\"]
    }
  }",
  "log_level": "error",
  "source": "cc.api",
  "data": {
    "request_guid": "8ea55d52-c7ee-4cdc-85d0-8330b52f37fb"
  },
  "thread_id": 46975124543120,
  "fiber_id": 46975122236300,
  "process_id": 1816,
  "file": "/var/vcap/packages-src/fce0f1c91d13120d23750fa007bae6824bbe419c/cloud_controller_ng/lib/sinatra/vcap.rb",
  "lineno": 46,
  "method": "block in registered"
}

Suggested fix (optional)

Implement method stop_index in OPI::Client.

Additional information (optional)

We think this use case is not covered in the non-windows multiple instances lifecycle CATs tests.

Deploy eirini and/or apps to specific nodepools

Description

It is often useful to deploy CF and its apps to specific nodepools.

Suggested fix (optional)

This can be configured by adding a nodeSelector block to the helm charts.

This is a common pattern used in helm/charts
https://github.com/helm/charts/search?q=nodeSelector&unscoped_q=nodeSelector

This might best be solved by updating SCF alongside the eirini helm chart, instead of tacking functionality on top of the generated helm charts

Eirini-cf failing with error "diego_ssh must have at least 1 instances"

Description

I am deploying eirini-release on GKE cluster. When I deploy eirini-cf I am getting the following error
Error: render error in "cf/templates/diego-ssh.yaml": template: cf/templates/diego-ssh.yaml:181:17: executing "cf/templates/diego-ssh.yaml" at <fail "diego_ssh must have at least 1 instances">: error calling fail: diego_ssh must have at least 1 instances

From the templates I can see default was set to 1 and min_replicas 181

Steps to reproduce

Deploy eirini-release latest release on GKE.

Is HA broken on master?

Description

Configured multiple instances of NATs with HA turned on, the second instance fails to start.

Steps to reproduce

configure valus.yaml

config:
  HA: true
sizing:
  nats:
    count: 2

What was expected to happen

2 NATs instances would start and all would be good.

What actually happened

Second nats instance failed to start, logs are quite helpful.

++++ curl --silent --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H 'Authorization: bearer <token>' https://172.20.0.1:443/apis/apps/v1/statefulsets/nats/namespaces/scf/
Traceback (most recent call last):
  File "<string>", line 1, in <module>
KeyError: 'spec'
+++ replicas=
+++ '[' nats '!=' '' -a '' '!=' '' ']'
+++ '[' nats == '' ']'
+++ '[' '' == '' ']'
+++ echo 'Cannot get replicas from kubernetes API, retrying'
Cannot get replicas from kubernetes API, retrying
+++ sleep 1
+++ (( i ++ ))
+++ (( i < 5  ))
+++ '[' nats == '' ']'
+++ '[' '' == '' ']'
+++ echo 'Cannot get replicas from kubernetes API, exit'
Cannot get replicas from kubernetes API, exit
+++ exit 1
++ KUBE_NATS_CLUSTER_IPS=

Additional information (optional)

Having looked into this on a working container (the first nats instance).
It looks like the error is down too: /opt/fissile/startup/scripts/configure-HA-hosts.sh:51

replicas=$(k8s_api apis/apps/v1"/statefulsets/${statefulset_name}" | json_get [\'spec\'][\'replicas\'])

This should be

replicas=$(k8s_api apis/apps/v1 "/statefulsets/${statefulset_name}" | json_get [\'spec\'][\'replicas\'])

Note the space between v1 and statefulsets, the issue here is the building of the URL. Leading too apis/apps/v1/statefulsets/nats/namespaces/scf/ where it should be apis/apps/v1/namespaces/scf/statefulsets/nats/

This is on commit ref a5f2a697c04ad6b401e3dc114a26a25352ea4ff1

I am struggling to find where this code is coming from to provide a fix as it does not match https://github.com/SUSE/scf/blob/master/container-host-files/etc/scf/config/scripts/configure-HA-hosts.sh

rootfs-patcher fails when not in `scf` namespace

Description

rootfs-patcher job is hard coded to look at scf namespace.

Steps to reproduce

helm install --namespace cf

What was expected to happen

it should have worked

What actually happened

it didn't.

Suggested fix (optional)

remove hard coded scf namespace in job.

Additional information (optional)

[Fill in]

CF CLI command restart-app-instance didn't start the instance with Eirini

Description

[After setup eirini 1.0 release on IKS cluster, cf restart-app-instance [...] CLI call returned 204, but it only deleted the instance pod, and didn't start the instance.]

Steps to reproduce

# cf push my-app
# cf restart-app-instance my-app 0
Restarting instance 0 of application my-app as admin
OK
# cf apps
Getting apps in org system / space dev as admin...
OK
name                               requested state   instances   memory   disk   urls
my-app   started           0/1         128M     128M   my-app.teststream.eu-de.containers.appdomain.cloud]

What was expected to happen

The respective instance of the app is restarted.

What actually happened

As shown in "Steps to reproduce", application instance was not running after the command.

Suggested fix (optional)

Additional information (optional)

eirinifs.tar file not found in blobstore causing bits container ending in CrashLoopBackOff.

Description

bits container not starting

log:

{"level":"info","ts":1552911988.4398773,"caller":"bitsgo/factory.go:334","msg":"Creating local blobstore","path-prefix":"/"}
{"level":"debug","ts":1552911988.4399006,"caller":"local/local_blobstore.go:38","msg":"GetNoRedirect","local-path":"/assets/eirinifs.tar"}

panic: Could not find assets/eirinifs.tar in root FS blobstore. Please make sure that copy it to the root FS blobstore as part of your deployment.

goroutine 1 [running]:
github.com/cloudfoundry-incubator/bits-service/oci_registry.NewBitsImageManager(0xec2bc0, 0xc420439ca0, 0xec2aa0, 0xc420438b70, 0xec2aa0, 0xc420438b70, 0x0)
	/tmp/build/37652675/bits-service-release/src/github.com/cloudfoundry-incubator/bits-service/oci_registry/registry.go:103 +0x249
main.main()
	/tmp/build/37652675/bits-service-release/src/github.com/cloudfoundry-incubator/bits-service/cmd/bitsgo/main.go:63 +0x1a5f

Steps to reproduce

Normal installation via helm

What was expected to happen

bits container should start

What actually happened

assets/eirinifs.tar file was not created / copied into blobstore

Suggested fix (optional)

Probably caused by commit:
82fd766dfb1811069d613017db7ea79c332d4a18

//Edit:
Probably fixed by:
017c734818fe60e10ca89f0df74bd48799f4898b

Additional information (optional)

Bulild and copy initialized by:
https://github.com/cloudfoundry-incubator/eirini-release/blob/master/scripts/buildfs.sh

Locket fails to start due to the container being non-privileged

When deploying Eirini 0.13.0 on a fresh environment (microk8s) locket fails to start with the following error: ulimit: open files: cannot modify limit: Operation not permitted.

The following part of configuration seems to be related https://github.com/cloudfoundry-incubator/eirini-release/blob/v0.13.0/helm/cf/templates/locket.yaml#L376-L378.

Can we implement Eirini on Kubernetes 1.15

We are having Kubernetes cluster 1.15 deployed and planning to implement Eirini on that.
Can we go head for that.

NetworkPolicy prevents calling apps

Description

I tried to deploy cf-for-k8s and run its smoke test. I manage to cf push the docker image, but the subsequent HTTP-GET always failed.

I found out that a misconfigured NetworkPolicy from eirini called deny-app-ingress is responsible for prohibiting all traffic to apps that does not come from the cf-system namespace.
Since the GET in the smoke test is routed via the istio-ingress in the istio-system namespace, the connection is dropped.

Steps to reproduce

Deploy cf-for-k8s using the scripts from github.com/cloudfoundry/cf-for-k8s.

What was expected to happen

I can successfully execute the smoke-tests which pushes a docker image and curls the app.
There is no networkpolicy prohibitting traffic via the istio-ingress.

What actually happened

The HTTP-GET in the smoke-test fails with error code 503, since the network policy deny-app-ingress prohibits traffic from the istio-system namespace which is where the istio-ingress is deployed.

Suggested fix (optional)

Add istio-system namespace to list of allowed namespaces.

Doesn't work with nginx:alpine based containers

Description

Doesn't seem to work with my website.

It's a straightforward nginx:alpine container with some config and static files copied over.

Dockerfile here

Steps to reproduce

cf push astandke -o xanderstrike/web:astandke

What was expected to happen

Site is available at astandke.CF_DOMAIN and looks right

What actually happened

App never reachable. cf apps says started with 0/1 instances. Kubectl get pods eventually says CrashLoopBackOff.

Logs include:

2020-02-19T10:35:28.66-0800 [unknown/769922a8-5346-11ea-8d1d-42010a8a016a] OUT 2020/02/19 18:35:28 [emerg] 1#1: mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied)
2020-02-19T10:35:28.66-0800 [unknown/769922a8-5346-11ea-8d1d-42010a8a016a] OUT nginx: [emerg] mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied)

Suggested fix (optional)

Allow processes in the container to run as root maybehaps

Additional information (optional)

Related to cloudfoundry/cf-for-k8s#31

Eirini regression on app environment variable handling

Description

Applications deployed with eirini with a special environment variable with a ":" in the name will never end up in a pod.

Example:

env:
  "TE:ST": "bla"

CF will wait until the health check timeout but no pod gets created and we were not able to find a detailed error other than the timeout.

Steps to reproduce

You can use this application as example
cf push eirini-regression-app

What was expected to happen

The app should be deployed without errors like it happens on CF using diego and the variable should be present

 cf env eirini-regression-app
Getting env variables for app eirini-regression-app in org MYORG / space MYSPACE as MYUSER...
OK

System-Provided:
[...]

User-Provided:
TE:ST: bla

What actually happened

0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
FAILED
Error restarting application: Start app timeout

Additional information

One use case of this unusual variable name would be the following:

"npm_config_@sap:registry":"https://npm.sap.com"

This is to setup npm_config options as described here where the configuration is @sap:registry https://npm.sap.com as described here

Failed to push apps in large concurrency with eirini runtime on AKS

Description

On Microsoft AKS, failed to push apps in large concurrency with eirini runtime. Error message from cf cli:

Staging app and tracing logs...
Error staging application: Staging error: unable to communicate to compatible cells
FAILED

after the failure, cloud foundry won't handle any other app push requests.

the "large" is not certain, sometimes concurrency 10 can lead to failure, sometimes it is 20 or more.

Steps to reproduce

Write a script or use tool to concurrently push apps. (cf push)

What was expected to happen

Push successfully and all apps are running correctly.

What actually happened

Push apps failed.

Suggested fix (optional)

n/a

Additional information (optional)

n/a

CAT test "the sleuth error response has no error " fail by using bosh release 0.0.15

Description

We deploy by using new Eirini bosh release 0.0.15:
https://s3.amazonaws.com/suse-final-releases/eirini-release-0.0.15.tgz
And used SCF 2.16.4 and new cc release 1.82.0

When we run CATs test which based on the config:
https://github.com/cloudfoundry-incubator/eirini-ci/blob/master/pipelines/modules/run-core-cats.yml

The builpack version:

staticfile_buildpack   1      true     false    staticfile-buildpack-cflinuxfs3-v1.4.35.zip
java_buildpack         2      true     false    java-buildpack-cflinuxfs3-v4.16.1.zip
ruby_buildpack         3      true     false    ruby-buildpack-cflinuxfs3-v1.7.42.zip
nodejs_buildpack       4      true     false    nodejs-buildpack-cflinuxfs3-v1.6.34.zip
go_buildpack           5      true     false    go-buildpack-cflinuxfs3-v1.8.29.zip
python_buildpack       6      true     false    python-buildpack-cflinuxfs3-v1.6.23.zip
php_buildpack          7      true     false    php-buildpack-cflinuxfs3-v4.3.64.zip
binary_buildpack       8      true     false    binary-buildpack-cflinuxfs3-v1.0.27.zip

We found there are still 3 failures.
one is
[Fail] [routing] Zipkin Tracing when zipkin tracing is enabled when zipkin headers are in the request [It] the sleuth error response has no error
/gopath/src/github.com/cloudfoundry/cf-acceptance-tests/routing/zipkin_tracing.go:82

and

[Fail] [routing] Zipkin Tracing when zipkin tracing is enabled when zipkin headers are not in the request [It] the sleuth error response has no error
/gopath/src/github.com/cloudfoundry/cf-acceptance-tests/routing/zipkin_tracing.go:58

Steps to reproduce

cf push CATS-2-APP-2fe173608135bcea -b java_buildpack -m 1024M -p assets/spring-sleuth/spring-sleuth.jar -d bcfdeirinidevelop.us-south.containers.appdomain.cloud
But fail to find the x_b3_traceid message in cf logs --recent CATS-2-APP-2fe173608135bcea

What was expected to happen

        Waiting for:
            x_b3_traceid:"fee1f7ba6aeec41c"

in cf logs

What actually happened

But cannot find in the real log:

            Retrieving logs for app CATS-2-APP-2fe173608135bcea in org CATS-2-ORG-90e5bb6f83d7b66a / space CATS-2-SPACE-10192ce0005c18fd as CATS-2-USER-364bfc491d9364ae...
            
               2019-08-20T04:30:20.57+0000 [API/1] OUT Created app with guid b2ebb58f-2b21-4d57-a6f0-bdd4ec478fba
               2019-08-20T04:30:41.02+0000 [API/1] OUT Uploading bits for app with guid b2ebb58f-2b21-4d57-a6f0-bdd4ec478fba
               2019-08-20T04:31:21.39+0000 [API/0] OUT Creating build for app with guid b2ebb58f-2b21-4d57-a6f0-bdd4ec478fba
               2019-08-20T04:31:22.04+0000 [API/0] OUT Updated app with guid b2ebb58f-2b21-4d57-a6f0-bdd4ec478fba ({"state"=>"STARTED"})
               2019-08-20T04:31:27.00+0000 [STG/0] OUT 2019/08/20 04:31:23 downloader-started
               2019-08-20T04:31:27.00+0000 [STG/0] OUT 2019/08/20 04:31:26 executor-started
               2019-08-20T04:31:27.00+0000 [STG/0] OUT 2019/08/20 04:31:25 downloader-done
               2019-08-20T04:31:29.00+0000 [STG/0] OUT -----> Java Buildpack v4.16.1 | https://github.com/cloudfoundry/java-buildpack.git#41b8ff8
               2019-08-20T04:31:30.00+0000 [STG/0] OUT -----> Downloading Jvmkill Agent 1.16.0_RELEASE from https://java-buildpack.cloudfoundry.org/jvmkill/bionic/x86_64/jvmkill-1.16.0-RELEASE.so (0.5s)
               2019-08-20T04:31:32.00+0000 [STG/0] OUT -----> Downloading Open Jdk JRE 1.8.0_222 from https://java-buildpack.cloudfoundry.org/openjdk/bionic/x86_64/openjdk-jre-1.8.0_222-bionic.tar.gz (2.3s)
               2019-08-20T04:31:33.00+0000 [STG/0] OUT -----> Downloading Open JDK Like Memory Calculator 3.13.0_RELEASE from https://java-buildpack.cloudfoundry.org/memory-calculator/bionic/x86_64/memory-calculator-3.13.0-RELEASE.tar.gz (0.3s)
               2019-08-20T04:31:33.00+0000 [STG/0] OUT Expanding Open Jdk JRE to .java-buildpack/open_jdk_jre (1.1s)
               2019-08-20T04:31:34.00+0000 [STG/0] OUT Loaded Classes: 16870, Threads: 250
               2019-08-20T04:31:35.00+0000 [STG/0] OUT -----> Downloading Spring Auto Reconfiguration 2.8.0_RELEASE from https://java-buildpack.cloudfoundry.org/auto-reconfiguration/auto-reconfiguration-2.8.0-RELEASE.jar (0.3s)
               2019-08-20T04:31:35.00+0000 [STG/0] OUT -----> Downloading Container Security Provider 1.16.0_RELEASE from https://java-buildpack.cloudfoundry.org/container-security-provider/container-security-provider-1.16.0-RELEASE.jar (0.2s)
               2019-08-20T04:31:35.00+0000 [STG/0] OUT -----> Downloading Client Certificate Mapper 1.9.0_RELEASE from https://java-buildpack.cloudfoundry.org/client-certificate-mapper/client-certificate-mapper-1.9.0-RELEASE.jar (0.1s)
               2019-08-20T04:31:47.00+0000 [STG/0] OUT 2019/08/20 04:31:47 executor-done
               2019-08-20T04:31:50.00+0000 [STG/0] OUT 2019/08/20 04:31:49 uploader-started
               2019-08-20T04:31:50.00+0000 [STG/0] OUT 2019/08/20 04:31:49 droplet info: https://cc-uploader-cc-uploader.cf.svc.cluster.local:9091/v1/droplet/024baf61-4f40-4480-af50-841e6ba75b18?cc-droplet-upload-uri=https://api.cf.svc.cluster.local:9023/internal/v4/droplets/024baf61-4f40-4480-af50-841e6ba75b18/upload /out/droplet.tgz
               2019-08-20T04:31:50.77+0000 [API/0] OUT Creating droplet for app with guid b2ebb58f-2b21-4d57-a6f0-bdd4ec478fba
               2019-08-20T04:31:55.00+0000 [STG/0] OUT 2019/08/20 04:31:55 uploader-done
               2019-08-20T04:32:28.00+0000 [APP/PROC/WEB/0] OUT ARGS: [/lifecycle/launch]
               2019-08-20T04:32:28.00+0000 [APP/PROC/WEB/0] OUT JVM Memory Configuration: -Xmx419988K -Xss1M -XX:ReservedCodeCacheSize=240M -XX:MaxDirectMemorySize=10M -XX:MaxMetaspaceSize=116587K
               2019-08-20T04:32:30.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:30.648  INFO [bootstrap,,,] 7 --- [           main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@6d9c638: startup date [Tue Aug 20 04:32:30 UTC 2019]; root of context hierarchy
               2019-08-20T04:32:30.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:30.526  INFO [bootstrap,,,] 7 --- [           main] nfigurationApplicationContextInitializer : Reconfiguration enabled
               2019-08-20T04:32:30.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:30.499  INFO [bootstrap,,,] 7 --- [           main] pertySourceApplicationContextInitializer : 'cloud' property source added
               2019-08-20T04:32:31.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:31.407  INFO [bootstrap,,,] 7 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'configurationPropertiesRebinderAutoConfiguration' of type [class org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$2e685478] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
               2019-08-20T04:32:31.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:31.351  INFO [bootstrap,,,] 7 --- [           main] f.a.AutowiredAnnotationBeanPostProcessor : JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
               2019-08-20T04:32:32.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:32.250  WARN [,,,] 7 --- [           main] c.c.c.ConfigServicePropertySourceLocator : Could not locate PropertySource: I/O error on GET request for "http://localhost:8888/application/cloud": Connection refused (Connection refused); nested exception is java.net.ConnectException: Connection refused (Connection refused)
               2019-08-20T04:32:32.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:32.275  INFO [,,,] 7 --- [           main] ationConfigEmbeddedWebApplicationContext : Refreshing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@327471b5: startup date [Tue Aug 20 04:32:32 UTC 2019]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@6d9c638
               2019-08-20T04:32:32.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:32.255  INFO [,,,] 7 --- [           main] hello.Application                        : The following profiles are active: cloud
               2019-08-20T04:32:32.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:32.254  INFO [,,,] 7 --- [           main] nfigurationApplicationContextInitializer : Reconfiguration enabled
               2019-08-20T04:32:32.00+0000 [APP/PROC/WEB/0] OUT '  |____| .__|_| |_|_| |_\__, | / / / /
               2019-08-20T04:32:32.00+0000 [APP/PROC/WEB/0] OUT \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
               2019-08-20T04:32:32.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:32.253  INFO [,,,] 7 --- [           main] pertySourceApplicationContextInitializer : 'cloud' property source added
               2019-08-20T04:32:32.00+0000 [APP/PROC/WEB/0] OUT =========|_|==============|___/=/_/_/_/
               2019-08-20T04:32:32.00+0000 [APP/PROC/WEB/0] OUT :: Spring Boot ::        (v1.4.0.RELEASE)
               2019-08-20T04:32:32.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:32.123  INFO [,,,] 7 --- [           main] c.c.c.ConfigServicePropertySourceLocator : Fetching config from server at: http://localhost:8888
               2019-08-20T04:32:32.00+0000 [APP/PROC/WEB/0] OUT ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
               2019-08-20T04:32:32.00+0000 [APP/PROC/WEB/0] OUT .   ____          _            __ _ _
               2019-08-20T04:32:32.00+0000 [APP/PROC/WEB/0] OUT /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
               2019-08-20T04:32:34.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:34.607  INFO [,,,] 7 --- [           main] f.a.AutowiredAnnotationBeanPostProcessor : JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
               2019-08-20T04:32:34.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:34.769  INFO [,,,] 7 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.sleuth.instrument.async.AsyncDefaultAutoConfiguration$DefaultAsyncConfigurerSupport' of type [class org.springframework.cloud.sleuth.instrument.async.AsyncDefaultAutoConfiguration$DefaultAsyncConfigurerSupport$$EnhancerBySpringCGLIB$$4f8fef3e] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
               2019-08-20T04:32:34.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:34.574  INFO [,,,] 7 --- [           main] o.s.cloud.context.scope.GenericScope     : BeanFactory id=8a5f9aa8-7bcf-3eae-a795-32417afe47a6
               2019-08-20T04:32:34.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:34.355  WARN [,,,] 7 --- [           main] o.s.c.a.ConfigurationClassPostProcessor  : Cannot enhance @Configuration bean definition 'refreshScope' since its singleton instance has been created too early. The typical cause is a non-static @Bean method with a BeanDefinitionRegistryPostProcessor return type: Consider declaring such methods as 'static'.
               2019-08-20T04:32:35.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:35.231  INFO [,,,] 7 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration' of type [class org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$2e685478] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
               2019-08-20T04:32:36.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:36.437  INFO [,,,] 7 --- [ost-startStop-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
               2019-08-20T04:32:36.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:36.276  INFO [,,,] 7 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet Engine: Apache Tomcat/8.5.4
               2019-08-20T04:32:36.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:36.274  INFO [,,,] 7 --- [           main] o.apache.catalina.core.StandardService   : Starting service Tomcat
               2019-08-20T04:32:36.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:36.438  INFO [,,,] 7 --- [ost-startStop-1] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 4163 ms
               2019-08-20T04:32:36.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:36.249  INFO [,,,] 7 --- [           main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat initialized with port(s): 8080 (http)
               2019-08-20T04:32:38.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:38.394  INFO [,,,] 7 --- [ost-startStop-1] o.s.b.w.servlet.ServletRegistrationBean  : Mapping servlet: 'dispatcherServlet' to [/]
               2019-08-20T04:32:38.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:38.391  INFO [,,,] 7 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'webRequestLoggingFilter' to: [/*]
               2019-08-20T04:32:38.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:38.393  INFO [,,,] 7 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'applicationContextIdFilter' to: [/*]
               2019-08-20T04:32:38.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:38.390  INFO [,,,] 7 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'requestContextFilter' to: [/*]
               2019-08-20T04:32:38.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:38.389  INFO [,,,] 7 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'httpPutFormContentFilter' to: [/*]
               2019-08-20T04:32:38.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:38.388  INFO [,,,] 7 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'hiddenHttpMethodFilter' to: [/*]
               2019-08-20T04:32:38.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:38.386  INFO [,,,] 7 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'metricFilter' to: [/*]
               2019-08-20T04:32:38.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:38.387  INFO [,,,] 7 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'traceFilter' to: [/*]
               2019-08-20T04:32:38.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:38.386  INFO [,,,] 7 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'characterEncodingFilter' to: [/*]
               2019-08-20T04:32:38.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:38.384  INFO [,,,] 7 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'clientCertificateMapper' to: [/*]
               2019-08-20T04:32:40.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:40.920  INFO [,,,] 7 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error]}" onto public org.springframework.http.ResponseEntity<java.util.Map<java.lang.String, java.lang.Object>> org.springframework.boot.autoconfigure.web.BasicErrorController.error(javax.servlet.http.HttpServletRequest)
               2019-08-20T04:32:40.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:40.919  INFO [,,,] 7 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error],produces=[text/html]}" onto public org.springframework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)
               2019-08-20T04:32:40.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:40.915  INFO [,,,] 7 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/]}" onto public java.lang.String hello.HelloController.index() throws java.lang.Exception
               2019-08-20T04:32:40.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:40.555  INFO [,,,] 7 --- [           main] s.w.s.m.m.a.RequestMappingHandlerAdapter : Looking for @ControllerAdvice: org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@327471b5: startup date [Tue Aug 20 04:32:32 UTC 2019]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@6d9c638
               2019-08-20T04:32:41.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:41.178  INFO [,,,] 7 --- [           main] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
               2019-08-20T04:32:41.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:41.327  INFO [,,,] 7 --- [           main] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped URL path [/**/favicon.ico] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
               2019-08-20T04:32:41.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:41.178  INFO [,,,] 7 --- [           main] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped URL path [/webjars/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
               2019-08-20T04:32:43.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:43.992  INFO [,,,] 7 --- [           main] c.n.c.sources.URLConfigurationSource     : To enable URLs as dynamic configuration sources, define System property archaius.configurationSource.additionalUrls or make config.properties available on classpath.
               2019-08-20T04:32:43.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:43.992  WARN [,,,] 7 --- [           main] c.n.c.sources.URLConfigurationSource     : No URLs will be polled as dynamic configuration sources.
               2019-08-20T04:32:43.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:43.970  WARN [,,,] 7 --- [           main] o.s.c.n.a.ArchaiusAutoConfiguration      : No spring.application.name found, defaulting to 'application'
               2019-08-20T04:32:43.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:43.982  WARN [,,,] 7 --- [           main] c.n.c.sources.URLConfigurationSource     : No URLs will be polled as dynamic configuration sources.
               2019-08-20T04:32:43.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:43.982  INFO [,,,] 7 --- [           main] c.n.c.sources.URLConfigurationSource     : To enable URLs as dynamic configuration sources, define System property archaius.configurationSource.additionalUrls or make config.properties available on classpath.
               2019-08-20T04:32:43.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:43.451  INFO [,,,] 7 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/env/reset],methods=[POST]}" onto public java.util.Map<java.lang.String, java.lang.Object> org.springframework.cloud.context.environment.EnvironmentManagerMvcEndpoint.reset()
               2019-08-20T04:32:43.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:43.442  INFO [,,,] 7 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/features || /features.json],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
               2019-08-20T04:32:43.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:43.424  INFO [,,,] 7 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/env || /env.json],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
               2019-08-20T04:32:43.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:43.424  INFO [,,,] 7 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/env/{name:.*}],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EnvironmentMvcEndpoint.value(java.lang.String)
               2019-08-20T04:32:43.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:43.412  INFO [,,,] 7 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/resume || /resume.json],methods=[POST]}" onto public java.lang.Object org.springframework.cloud.endpoint.GenericPostableMvcEndpoint.invoke()
               2019-08-20T04:32:43.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:43.451  INFO [,,,] 7 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/env],methods=[POST]}" onto public java.lang.Object org.springframework.cloud.context.environment.EnvironmentManagerMvcEndpoint.value(java.util.Map<java.lang.String, java.lang.String>)
               2019-08-20T04:32:43.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:43.448  INFO [,,,] 7 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/archaius || /archaius.json],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
               2019-08-20T04:32:43.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:43.438  INFO [,,,] 7 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/trace || /trace.json],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
               2019-08-20T04:32:43.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:43.437  INFO [,,,] 7 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/dump || /dump.json],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
               2019-08-20T04:32:43.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:43.429  INFO [,,,] 7 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/pause || /pause.json],methods=[POST]}" onto public java.lang.Object org.springframework.cloud.endpoint.GenericPostableMvcEndpoint.invoke()
               2019-08-20T04:32:43.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:43.427  INFO [,,,] 7 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/health || /health.json],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.invoke(java.security.Principal)
               2019-08-20T04:32:43.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:43.426  INFO [,,,] 7 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/beans || /beans.json],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
               2019-08-20T04:32:43.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:43.420  INFO [,,,] 7 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/heapdump || /heapdump.json],methods=[GET],produces=[application/octet-stream]}" onto public void org.springframework.boot.actuate.endpoint.mvc.HeapdumpMvcEndpoint.invoke(boolean,javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException,javax.servlet.ServletException
               2019-08-20T04:32:43.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:43.432  INFO [,,,] 7 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/configprops || /configprops.json],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
               2019-08-20T04:32:43.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:43.428  INFO [,,,] 7 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/info || /info.json],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
               2019-08-20T04:32:43.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:43.410  INFO [,,,] 7 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/metrics || /metrics.json],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
               2019-08-20T04:32:43.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:43.410  INFO [,,,] 7 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/metrics/{name:.*}],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.MetricsMvcEndpoint.value(java.lang.String)
               2019-08-20T04:32:43.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:43.416  INFO [,,,] 7 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/mappings || /mappings.json],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
               2019-08-20T04:32:43.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:43.446  INFO [,,,] 7 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/refresh || /refresh.json],methods=[POST]}" onto public java.lang.Object org.springframework.cloud.endpoint.GenericPostableMvcEndpoint.invoke()
               2019-08-20T04:32:43.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:43.414  INFO [,,,] 7 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/restart || /restart.json],methods=[POST]}" onto public java.lang.Object org.springframework.cloud.context.restart.RestartMvcEndpoint.invoke()
               2019-08-20T04:32:43.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:43.440  INFO [,,,] 7 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/autoconfig || /autoconfig.json],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
               2019-08-20T04:32:44.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:44.760  INFO [,,,] 7 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Located managed bean 'configurationPropertiesRebinder': registering with JMX server as MBean [org.springframework.cloud.context.properties:name=configurationPropertiesRebinder,context=327471b5,type=ConfigurationPropertiesRebinder]
               2019-08-20T04:32:44.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:44.738  INFO [,,,] 7 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Located managed bean 'refreshScope': registering with JMX server as MBean [org.springframework.cloud.context.scope.refresh:name=refreshScope,type=RefreshScope]
               2019-08-20T04:32:44.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:44.676  INFO [,,,] 7 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Bean with name 'restartEndpoint' has been autodetected for JMX exposure
               2019-08-20T04:32:44.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:44.673  INFO [,,,] 7 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Bean with name 'refreshEndpoint' has been autodetected for JMX exposure
               2019-08-20T04:32:44.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:44.772  INFO [,,,] 7 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Located managed bean 'refreshEndpoint': registering with JMX server as MBean [org.springframework.cloud.endpoint:name=refreshEndpoint,type=RefreshEndpoint]
               2019-08-20T04:32:44.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:44.703  INFO [,,,] 7 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Located managed bean 'restartEndpoint': registering with JMX server as MBean [org.springframework.cloud.context.restart:name=restartEndpoint,type=RestartEndpoint]
               2019-08-20T04:32:44.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:44.681  INFO [,,,] 7 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Located managed bean 'environmentManager': registering with JMX server as MBean [org.springframework.cloud.context.environment:name=environmentManager,type=EnvironmentManager]
               2019-08-20T04:32:44.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:44.671  INFO [,,,] 7 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Bean with name 'configurationPropertiesRebinder' has been autodetected for JMX exposure
               2019-08-20T04:32:44.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:44.668  INFO [,,,] 7 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Bean with name 'environmentManager' has been autodetected for JMX exposure
               2019-08-20T04:32:44.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:44.666  INFO [,,,] 7 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Bean with name 'refreshScope' has been autodetected for JMX exposure
               2019-08-20T04:32:44.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:44.647  INFO [,,,] 7 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Registering beans for JMX exposure on startup
               2019-08-20T04:32:44.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:44.385  INFO [,,,] 7 --- [           main] o.s.s.c.ThreadPoolTaskScheduler          : Initializing ExecutorService
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT zipkinSpanListener
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT websocketContainerCustomizer
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT viewControllerHandlerMapping
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT webRequestLoggingFilter
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT zipkinEndpointLocator
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT traceFilter
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT standardJacksonObjectMapperBuilderCustomizer
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT spring.resources-org.springframework.boot.autoconfigure.web.ResourceProperties
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT spring.mvc-org.springframework.boot.autoconfigure.web.WebMvcProperties
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT spring.metrics.export-org.springframework.boot.actuate.metrics.export.MetricExportProperties
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT simpleControllerHandlerAdapter
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT servoMetricNaming
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT scopedTarget.eurekaClient
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT ribbonFeature
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT resumeEndpoint
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT refreshScopeHealthIndicator
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT randomForSpanIds
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.context.event.internalEventListenerFactory
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.context.annotation.internalCommonAnnotationProcessor
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.context.annotation.ConfigurationClassPostProcessor.importAwareProcessor
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.context.annotation.ConfigurationClassPostProcessor.enhancedConfigurationProcessor
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.sleuth.log.SleuthLogAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.sleuth.instrument.web.TraceWebAutoConfiguration$SkipPatternProviderConfig
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.sleuth.instrument.web.TraceWebAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.netflix.metrics.servo.ServoMetricsAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.netflix.metrics.MetricsInterceptorConfiguration$MetricsWebResourceConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.netflix.metrics.MetricsInterceptorConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.netflix.archaius.ArchaiusAutoConfiguration$ArchaiusEndpointConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.config.client.ConfigClientAutoConfiguration$ConfigServerHealthIndicatorConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.autoconfigure.RefreshAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.context.properties.ConfigurationPropertiesBindingPostProcessor
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.websocket.WebSocketAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.web.WebMvcAutoConfiguration$WebMvcAutoConfigurationAdapter$FaviconConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.web.WebMvcAutoConfiguration$WebMvcAutoConfigurationAdapter
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.web.WebMvcAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.web.MultipartAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.web.JacksonHttpMessageConvertersConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.web.HttpEncodingAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.web.ErrorMvcAutoConfiguration$WhitelabelErrorViewConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$JacksonObjectMapperConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.PropertyPlaceholderAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.actuate.autoconfigure.MetricFilterAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.actuate.autoconfigure.MetricExportAutoConfiguration$MetricExportPropertiesConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.actuate.autoconfigure.MetricExportAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.actuate.autoconfigure.HealthIndicatorAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.aop.config.internalAutoProxyCreator
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT mvcPathMatcher
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT mvcContentNegotiationManager
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT multipartConfigElement
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT metricsEndpoint
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT managementContextResolver
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT management.health.status-org.springframework.boot.actuate.autoconfigure.HealthIndicatorProperties
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT localeCharsetMappingsCustomizer
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT inetUtilsProperties
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT inetUtils
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT healthMvcEndpoint
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT healthAggregator
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT faviconRequestHandler
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT environmentEndpoint
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT endpoints.metrics.filter-org.springframework.boot.actuate.autoconfigure.MetricFilterProperties
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT endpointMBeanExporter
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT embeddedServletContainerCustomizerBeanPostProcessor
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT dispatcherServlet
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT diskSpaceHealthIndicator
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT discoveryClientHealthIndicator
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT defaultTraceSampler
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT defaultServletHandlerMapping
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT default.org.springframework.cloud.netflix.ribbon.eureka.RibbonEurekaAutoConfiguration.RibbonClientSpecification
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT default.org.springframework.cloud.netflix.ribbon.RibbonAutoConfiguration.RibbonClientSpecification
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT configurableEnvironmentConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT beanNameHandlerMapping
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT traceSchedulingAspect
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT traceAsyncAspect
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT tomcatEmbeddedServletContainerFactory
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT spring.zipkin-org.springframework.cloud.sleuth.zipkin.ZipkinProperties
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT spring.sleuth.sampler-org.springframework.cloud.sleuth.sampler.SamplerProperties
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT spring.jackson-org.springframework.boot.autoconfigure.jackson.JacksonProperties
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT spanReporterCounterService
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT slf4jSpanLogger
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT sleuthMetricProperties
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT sleuthHystrixConcurrencyStrategy
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT singleReturnValueHandler
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT servoMetricServices
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT scopedTarget.eurekaApplicationInfoManager
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT resumeMvcEndpoint
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT restTemplateCustomizer
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT requestMappingHandlerMapping
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT refreshScope
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT refreshEventListener
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT refreshEndpoint
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT propertySourcesPlaceholderConfigurer
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT preserveErrorControllerTargetClassPostProcessor
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT pauseMvcEndpoint
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.context.annotation.internalScheduledAnnotationProcessor
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.context.annotation.internalConfigurationAnnotationProcessor
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.context.annotation.internalAutowiredAnnotationProcessor
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.context.annotation.internalAsyncAnnotationProcessor
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.sleuth.log.SleuthLogAutoConfiguration$Slf4jConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.sleuth.instrument.web.client.TraceWebClientAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.sleuth.instrument.scheduling.TraceSchedulingAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.netflix.rx.RxJavaAutoConfiguration$RxJavaReturnValueHandlerConfig
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.netflix.rx.RxJavaAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.netflix.ribbon.RibbonAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.netflix.eureka.EurekaClientAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.netflix.archaius.ArchaiusAutoConfiguration$PropagateEventsConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.config.client.ConfigClientAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.client.loadbalancer.LoadBalancerAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.autoconfigure.LifecycleMvcEndpointAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.context.properties.ConfigurationPropertiesBindingPostProcessor.store
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.web.ServerPropertiesAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.web.HttpMessageConvertersAutoConfiguration$StringHttpMessageConverterConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.web.HttpMessageConvertersAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.web.ErrorMvcAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.web.DispatcherServletAutoConfiguration$DispatcherServletConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$Jackson2ObjectMapperBuilderCustomizerConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.actuate.autoconfigure.TraceRepositoryAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.actuate.autoconfigure.ManagementServerPropertiesAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextFilterConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT objectNamingStrategy
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT mvcValidator
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT mvcUrlPathHelper
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT mvcResourceUrlProvider
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT multipartResolver
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT monitorRegistry
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT metricWritersMetricExporter
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT managementServerProperties
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT loadBalancerClient
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT httpRequestHandlerAdapter
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT httpPutFormContentFilter
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT heapdumpMvcEndpoint
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT healthEndpoint
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT handlerExceptionResolver
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT eurekaInstanceConfigBean
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT eurekaFeature
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT errorPageCustomizer
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT environmentManagerEndpoint
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT endpoints-org.springframework.boot.actuate.endpoint.EndpointProperties
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT endpointHandlerMapping
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT defaultViewResolver
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT contextRefresher
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT configurationPropertiesRebinder
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT configurationPropertiesBeans
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT configClientProperties
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT commonsFeatures
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT beanNameViewResolver
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT basicErrorController
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT viewResolver
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT traceRestTemplateInterceptor
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT traceKeys
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT traceAsyncRestTemplate
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT traceAsyncClientHttpRequestFactory
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT tomcatPublicMetrics
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT systemPublicMetrics
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT spring.info-org.springframework.boot.autoconfigure.info.ProjectInfoProperties
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT spring.http.multipart-org.springframework.boot.autoconfigure.web.MultipartProperties
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT spanNamer
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT sleuthTracer
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT skipPatternForManagementServerProperties
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT servoPublicMetrics
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT servoMetricReader
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT serverProperties
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT rxFeature
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT ribbonInterceptor
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT restartMvcEndpoint
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT restTemplateBuilder
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT requestMappingEndpoint
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT refreshMvcEndpoint
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT pauseEndpoint
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.scheduling.annotation.SchedulingConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.scheduling.annotation.ProxyAsyncConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.context.event.internalEventListenerProcessor
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.context.annotation.internalRequiredAnnotationProcessor
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.sleuth.zipkin.ZipkinAutoConfiguration$DiscoveryClientEndpointLocatorConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.sleuth.zipkin.ZipkinAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.sleuth.metric.TraceMetricsAutoConfiguration$CounterServiceSpanReporterConfig
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.sleuth.metric.TraceMetricsAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.sleuth.instrument.rxjava.RxJavaAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.sleuth.instrument.hystrix.SleuthHystrixAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.netflix.ribbon.eureka.RibbonEurekaAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.autoconfigure.RefreshEndpointAutoConfiguration$RefreshEndpointConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.websocket.WebSocketAutoConfiguration$TomcatWebSocketConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.web.WebMvcAutoConfiguration$EnableWebMvcConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.web.WebClientAutoConfiguration$RestTemplateConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.web.EmbeddedServletContainerAutoConfiguration$EmbeddedTomcat
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.web.DispatcherServletAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.jmx.JmxAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.context.ConfigurationPropertiesAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.aop.AopAutoConfiguration$CglibAutoProxyConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.aop.AopAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.actuate.autoconfigure.InfoContributorAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.actuate.autoconfigure.EndpointWebMvcManagementContextConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$EndpointWebMvcConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.actuate.autoconfigure.EndpointMBeanExportAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.actuate.autoconfigure.EndpointAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.actuate.autoconfigure.AuditAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT observableMVCConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT mvcUriComponentsContributor
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT mvcEndpoints
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT metricsMvcEndpoint
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT metricReaderPublicMetrics
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT metricFilter
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT messageConverters
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT mappingJackson2HttpMessageConverter
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT managementServletContext
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT management.trace-org.springframework.boot.actuate.trace.TraceProperties
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT management.info-org.springframework.boot.actuate.autoconfigure.InfoContributorProperties
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT jsonComponentModule
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT httpServletResponseSpanInjector
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT faviconHandlerMapping
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT eurekaClientConfigBean
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT eurekaClient
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT errorPageRegistrarBeanPostProcessor
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT error
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT environmentManager
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT endpoints.jmx-org.springframework.boot.actuate.autoconfigure.EndpointMBeanExportProperties
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT endpoints.cors-org.springframework.boot.actuate.autoconfigure.EndpointCorsProperties
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT dispatcherServletRegistration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT discoveryCompositeHealthIndicator
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT discoveryClientOptionalArgs
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT discoveryClient
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT defaultMetricsTagProvider
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT conventionErrorViewResolver
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT clientCertificateMapperFilterRegistrationBean
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT clientCertificateMapper
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT characterEncodingFilter
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT auditListener
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT Let's inspect the beans provided by Spring Boot:
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT traceWebAspect
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT traceRepository
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT traceEndpoint
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT stringHttpMessageConverter
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT springClientFactory
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT spring.sleuth.keys-org.springframework.cloud.sleuth.TraceKeys
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT spring.http.encoding-org.springframework.boot.autoconfigure.web.HttpEncodingProperties
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT sleuthRxJavaSchedulersHook
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT shutdownEndpoint
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT servoMonitoringWebResourceInterceptor
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT servoMetricsConfig
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT restartEndpoint
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT resourceHandlerMapping
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT requestMappingHandlerAdapter
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT requestContextFilter
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT reporter
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.sleuth.instrument.web.client.TraceWebClientAutoConfiguration$TraceInterceptorConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.sleuth.instrument.web.client.TraceWebAsyncClientAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.sleuth.instrument.async.AsyncDefaultAutoConfiguration$DefaultAsyncConfigurerSupport
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.sleuth.instrument.async.AsyncDefaultAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.sleuth.autoconfig.TraceAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.netflix.hystrix.HystrixAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.netflix.eureka.EurekaClientAutoConfiguration$RefreshableEurekaClientConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.netflix.archaius.ArchaiusAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.commons.util.UtilAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.client.CommonsClientAutoConfiguration$ActuatorConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.client.CommonsClientAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.autoconfigure.RefreshEndpointAutoConfiguration$RestartEndpointWithoutIntegration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.cloud.autoconfigure.RefreshEndpointAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.web.WebClientAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.web.JacksonHttpMessageConvertersConfiguration$MappingJackson2HttpMessageConverterConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.web.EmbeddedServletContainerAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.web.DispatcherServletAutoConfiguration$DispatcherServletRegistrationConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$JacksonObjectMapperBuilderConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.internalCachingMetadataReaderFactory
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.info.ProjectInfoAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.condition.BeanTypeRegistry
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.autoconfigure.AutoConfigurationPackages
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.actuate.autoconfigure.TraceWebFilterAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.actuate.autoconfigure.PublicMetricsAutoConfiguration$TomcatMetricsConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.actuate.autoconfigure.PublicMetricsAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.actuate.autoconfigure.MetricRepositoryAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.actuate.autoconfigure.HealthIndicatorAutoConfiguration$DiskSpaceHealthIndicatorConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.actuate.autoconfigure.EndpointAutoConfiguration$RequestMappingEndpointConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.springframework.boot.actuate.autoconfigure.AuditAutoConfiguration$AuditEventRepositoryConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT org.cloudfoundry.router.ClientCertificateMapperAutoConfiguration
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT mvcViewResolver
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT mvcConversionService
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT monitorCache
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT mbeanServer
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT mbeanExporter
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT loggingRebinder
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT loadBalancedRestTemplateInitializer
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT jacksonObjectMapperBuilder
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT jacksonObjectMapper
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT infoEndpoint
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT hystrixHealthIndicator
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT httpServletRequestSpanExtractor
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT httpRequestSpanInjector
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT hiddenHttpMethodFilter
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT helloController
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT featuresEndpoint
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT eurekaApplicationInfoManager
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT errorAttributes
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT environmentMvcEndpoint
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT envInfoContributor
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT endpoints.health-org.springframework.boot.actuate.autoconfigure.HealthMvcEndpointProperties
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT duplicateServerPropertiesDetector
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT dumpEndpoint
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT diskSpaceHealthIndicatorProperties
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT configurationPropertiesReportEndpoint
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT configServerHealthIndicator
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT beansEndpoint
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT autoConfigurationReportEndpoint
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT auditEventRepository
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT applicationContextIdFilter
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT archaiusEndpoint
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:45.166  INFO [,,,] 7 --- [           main] hello.Application                        : Started Application in 15.902 seconds (JVM running for 17.288)
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:45.021  INFO [,,,] 7 --- [           main] o.s.c.support.DefaultLifecycleProcessor  : Starting beans in phase 0
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:45.159  INFO [,,,] 7 --- [           main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 8080 (http)
               2019-08-20T04:32:45.00+0000 [APP/PROC/WEB/0] OUT application
               2019-08-20T04:32:56.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:56.942 DEBUG [,fee1f7ba6aeec41c,7007f35c690d1c1c,true] 7 --- [nio-8080-exec-3] o.s.web.servlet.DispatcherServlet        : Successfully completed request
               2019-08-20T04:32:56.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:56.871  INFO [,fee1f7ba6aeec41c,7007f35c690d1c1c,true] 7 --- [nio-8080-exec-3] hello.HelloController                    : handling request
               2019-08-20T04:32:56.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:56.856 DEBUG [,fee1f7ba6aeec41c,7007f35c690d1c1c,true] 7 --- [nio-8080-exec-3] o.s.web.servlet.DispatcherServlet        : Last-Modified value for [/] is: -1
               2019-08-20T04:32:56.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:56.841 DEBUG [,fee1f7ba6aeec41c,7007f35c690d1c1c,true] 7 --- [nio-8080-exec-3] o.s.web.servlet.DispatcherServlet        : DispatcherServlet with name 'dispatcherServlet' processing GET request for [/]
               2019-08-20T04:32:56.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:56.809 DEBUG [,,,] 7 --- [nio-8080-exec-3] o.s.web.servlet.DispatcherServlet        : Servlet 'dispatcherServlet' configured successfully
               2019-08-20T04:32:56.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:56.808  INFO [,,,] 7 --- [nio-8080-exec-3] o.s.web.servlet.DispatcherServlet        : FrameworkServlet 'dispatcherServlet': initialization completed in 94 ms
               2019-08-20T04:32:56.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:56.808 DEBUG [,,,] 7 --- [nio-8080-exec-3] o.s.web.servlet.DispatcherServlet        : Unable to locate FlashMapManager with name 'flashMapManager': using default [org.springframework.web.servlet.support.SessionFlashMapManager@d81d431]
               2019-08-20T04:32:56.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:56.779 DEBUG [,,,] 7 --- [nio-8080-exec-3] o.s.web.servlet.DispatcherServlet        : Unable to locate RequestToViewNameTranslator with name 'viewNameTranslator': using default [org.springframework.web.servlet.view.DefaultRequestToViewNameTranslator@393327eb]
               2019-08-20T04:32:56.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:56.900 DEBUG [,fee1f7ba6aeec41c,7007f35c690d1c1c,true] 7 --- [nio-8080-exec-3] o.s.web.servlet.DispatcherServlet        : Null ModelAndView returned to DispatcherServlet with name 'dispatcherServlet': assuming HandlerAdapter completed request handling
               2019-08-20T04:32:56.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:56.808 DEBUG [,,,] 7 --- [nio-8080-exec-3] o.s.web.servlet.DispatcherServlet        : Published WebApplicationContext of servlet 'dispatcherServlet' as ServletContext attribute with name [org.springframework.web.servlet.FrameworkServlet.CONTEXT.dispatcherServlet]
               2019-08-20T04:32:56.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:56.756 DEBUG [,,,] 7 --- [nio-8080-exec-3] o.s.web.servlet.DispatcherServlet        : Unable to locate ThemeResolver with name 'themeResolver': using default [org.springframework.web.servlet.theme.FixedThemeResolver@6c249f0]
               2019-08-20T04:32:56.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:56.729 DEBUG [,,,] 7 --- [nio-8080-exec-3] o.s.web.servlet.DispatcherServlet        : Unable to locate LocaleResolver with name 'localeResolver': using default [org.springframework.web.servlet.i18n.AcceptHeaderLocaleResolver@7602b445]
               2019-08-20T04:32:56.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:56.713  INFO [,,,] 7 --- [nio-8080-exec-3] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring FrameworkServlet 'dispatcherServlet'
               2019-08-20T04:32:56.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:56.711 DEBUG [,,,] 7 --- [nio-8080-exec-3] o.s.web.servlet.DispatcherServlet        : Initializing servlet 'dispatcherServlet'
               2019-08-20T04:32:56.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:56.715 DEBUG [,,,] 7 --- [nio-8080-exec-3] o.s.web.servlet.DispatcherServlet        : Using MultipartResolver [org.springframework.web.multipart.support.StandardServletMultipartResolver@15dcfae7]
               2019-08-20T04:32:56.00+0000 [APP/PROC/WEB/0] OUT 2019-08-20 04:32:56.713  INFO [,,,] 7 --- [nio-8080-exec-3] o.s.web.servlet.DispatcherServlet        : FrameworkServlet 'dispatcherServlet': initialization started

So in these three failures, Applications all push and work as design, just cannot find the required message in cf logs. I am not sure if it is because of the Eirini release difference or the buildpack difference.

Please let me know if you need more information.

Thanks a lot!

Released UAA chart 2.18.0 has broken apiVersion

bd83ac8

This resolved the apiVersion in the codebase but the released version of 2.18.0 still has the broken apiVersion

helm fetch eirini/uaa --version 2.18.0

# Chart.yml
apiVersion: 2.18.0+cf9.5.0.0.ga2697c75
appVersion: 1.5.0
description: A Helm chart for SUSE UAA
name: uaa
version: 2.18.0

which means helm can't install it

  Error: apiVersion '2.18.0+cf9.5.0.0.ga2697c75' is not valid. The value must be "v1"

Can the UAA chart be bumped to 2.18.1 and cut with a release with the fixed apiVersion?

I'll just make a local copy of the chart in the mean time, it's not a big deal.

Thanks for your time 👍

Eirini docker image tag 'latest' missing

Description

The current version of the eirini helm chart uses latest tag for its images.

Unfortunately rootfs-patcher images don't have a latest tag:

eirini/bits-waiter
- template
- dockerhub image tags
eirini/rootfs-patcher
- template
- dockerhub image tags

Steps to reproduce

Try to deploy the helm chart. Kubernetes will not be able to pull these images.

Suggested fix

As workaround, we changed the mentioned template

 image: eirini/bits-waiter:{{ .Values.opi.image_tag }}
 image: eirini/rootfs-patcher:{{ .Values.opi.image_tag }}

with

 image: eirini/bits-waiter:ci-57.0.0
 image: eirini/rootfs-patcher:ci-57.0.0

It was not possible to use ci-57.0.0 as global value in values.yaml because helm would then try to change the image of already finished jobs and fail with Field is immutable error.

Improve startup logs for `opi`

When starting opi, this is what the user sees:

{"timestamp":"...","level":"info","source":"handler","message":"handler.opi-connected","data":{}}

This is not great, as it's not clear what handler and opi-connected mean. It would be great to see something like:

{"timestamp":"...","level":"info","source":"api","message":"api.started","data":{}}

Improve logs for `GET /apps`

This is how logs look like when LRPs are listed:

{"timestamp":"...","level":"debug","source":"handler","message":"handler.list-apps.requested","data":{"session":"218"}}

It's not very clear what handler means, and we don't see the start and end of the request processing, which could be useful especially once we introduce extra debug logs or just to see request duration. Also, this should probably be an info log. So something like this:

{"timestamp":"...","level":"info","source":"api","message":"list-apps.started","data":{"session":"218"}}
{"timestamp":"...","level":"info","source":"api","message":"list-apps.listing-stateful-sets","data":{"session":"218", "label-selector": "cloufoundry.org/source-type=APP"}}
{"timestamp":"...","level":"info","source":"api","message":"list-apps.ended","data":{"session":"218", "response-code": 200}}

Let's also make sure that any errors are logged correctly and only once. Refer to #177562455 for details about how we want to log errors.

Dev Notes

Ensure logs are correlated using lager session IDs. This requires using the logger.Session method and passing the logger through method calls. It will not work if the logger is stored in a struct.

Logger can be passed as an explicit method argument, or might also be passed as part of the context object that is already passed around (see the https://github.com/cloudfoundry/lager/tree/master/lagerctx package, and note in this story). With lagerctx, there is the risk of accidentally and unknowingly using the fallback discard logger if the original context is somehow lost while passing through methods. So explicitly passing both context and logger as separate method parameters throughout might be the safest option, giving compile-time guarantees of having logging correctly propagated.

Minor Q: Any reason for duplication of registry URL logic?

I found that there's a duplication of logic between the job-secret-smuggler job (https://github.com/cloudfoundry-incubator/eirini-release/blob/3a1bc774dac7eb07b38a81f50edc112d3c8e3185/helm/eirini/templates/job-secret-smuggler.yaml#L25-L32) and eirini configmap (https://github.com/cloudfoundry-incubator/eirini-release/blob/3a1bc774dac7eb07b38a81f50edc112d3c8e3185/helm/eirini/templates/configmap.yaml#L14-L20)?

I was wondering if the job couldn't just use the configmap that exists anyhow. Thanks.

helm repo specifies v0.11 but only v0.10 available on github releases

Steps:

$ helm install eirini/uaa --namespace uaa --name uaa --values tmp/values.out.yaml --debug
[debug] Created tunnel using local port: '58003'

[debug] SERVER: "127.0.0.1:58003"

[debug] Original chart version: ""
Error: Failed to fetch https://github.com/cloudfoundry-incubator/eirini-release/releases/download/v0.11.0/eirini-uaa.tgz : 404 Not Found

But https://github.com/cloudfoundry-incubator/eirini-release/releases only includes v0.10 release.

$ cat ~/.helm/repository/cache/eirini-index.yaml
apiVersion: v1
entries:
  cf:
  - apiVersion: 2.14.5+cf2.7.0.3.gbc13ddc
    created: 2019-03-22T12:34:42.761163816Z
    description: A Helm chart for SUSE Cloud Foundry
    digest: bcb839ea7dbb7fb8a20f3f9efd1c129a0ff6a32fe33e2bdf4769677509cd5f31
    name: cf
    urls:
    - https://github.com/cloudfoundry-incubator/eirini-release/releases/download/v0.11.0/eirini-cf.tgz
    version: 2.14.5
  eirini:
  - apiVersion: v1
    appVersion: 0.3.0
    created: 2019-03-22T12:34:42.795133365Z
    description: A Helm chart for CloudFoundry/Eirini
    digest: 64164852f0e5c8ce1a37bfd50a6528e71df9cb1eeeeb8219963e6327013f4455
    name: eirini
    urls:
    - https://github.com/cloudfoundry-incubator/eirini-release/releases/download/v0.11.0/eirini.tgz
    version: 0.11.0
  - apiVersion: v1
    appVersion: 0.3.0
    created: 2019-02-26T10:45:50.742059369Z
    description: A Helm chart for CloudFoundry/Eirini
    digest: f88f54b113921ead9a37b8b46c75c919667c22eba03c2ab891730e954733bb06
    name: eirini
    urls:
    - https://github.com/cloudfoundry-incubator/eirini-release/releases/download/v0.10.0/eirini.tgz
    version: 0.10.0
  - apiVersion: v1
    appVersion: 0.3.0
    created: 2019-02-14T14:12:38.504124861Z
    description: A Helm chart for CloudFoundry/Eirini
    digest: f33c492ed6937c4d208de363ba8f0c3ac2b7e79aee6c482e4edf2a1c3b583e8b
    name: eirini
    urls:
    - https://github.com/cloudfoundry-incubator/eirini-release/releases/download/v0.9.0/eirini.tgz
    version: 0.9.0
  uaa:
  - apiVersion: 2.14.5+cf2.7.0.3.gbc13ddc
    created: 2019-03-22T12:34:42.794555378Z
    description: A Helm chart for SUSE UAA
    digest: 466beb091fe5cbe3d1f202661c86ea689c798baec2926369e8bd2b278ed1fcb1
    name: uaa
    urls:
    - https://github.com/cloudfoundry-incubator/eirini-release/releases/download/v0.11.0/eirini-uaa.tgz
    version: 2.14.5
generated: 2019-03-22T12:34:42.755425241Z

Cannot push due to missing registry credentials

Description

I'm unable to push an app to the Eirini deployed from the master branch.

Steps to reproduce

Deploy the Eirini with helm from the master branch.

What was expected to happen

cf push works

What actually happened

An application image cannot be pulled from the registry due to authentication failure.

104s        Warning   Failed             pod/hello-myspace-bc646f02de-0                   Failed to pull image "registry.192.168.51.101.nip.io:6666/cloudfoundry/5094703d-48c1-4600-8f21-f1df72a6e049:32a07ba5175cd9ce32f48d3f9d32df9da8379e5f": rpc error: code = Unknown desc = failed to resolve image "registry.192.168.51.101.nip.io:6666/cloudfoundry/5094703d-48c1-4600-8f21-f1df72a6e049:32a07ba5175cd9ce32f48d3f9d32df9da8379e5f": no available registry endpoint: unexpected status code https://registry.192.168.51.101.nip.io:6666/v2/cloudfoundry/5094703d-48c1-4600-8f21-f1df72a6e049/manifests/32a07ba5175cd9ce32f48d3f9d32df9da8379e5f: 401 Unauthorized

Suggested fix (optional)

It looks like the tag latest for secret-smuggler which is used in Eirini release does not include the following change: cloudfoundry/eirini@4e8c8fe so it has to be updated.

Additional information (optional)

Should the master branch have the fixed versions of images? Any other ways to ensure the master branch working?

loggregator-fluentd doesn't read logs from docker containers

cf logs shows only API logs if docker is used on k8s cluster.
Here is what I found to be the cause:

$ kubectl -n scf log loggregator-fluentd-ggwpj loggregator-fluentd --tail=100 | grep scf_router | tail -n1
2019-07-14 14:55:46 +0000 [warn]: #0 [in_tail_container_logs] /var/log/containers/router-0_scf_router-215f3442d0d4ad6cf77e79619899434cc0a6fc7250e34018ca3a294a9637b559.log unreadable. It is excluded and would be examined next time.

root@loggregator-fluentd-ggwpj:~# stat $(readlink -m /var/log/containers/router-0_scf_router-215f3442d0d4ad6cf77e79619899434cc0a6fc7250e34018ca3a294a9637b559.log)
stat: cannot stat '/var/lib/docker/containers/215f3442d0d4ad6cf77e79619899434cc0a6fc7250e34018ca3a294a9637b559/215f3442d0d4ad6cf77e79619899434cc0a6fc7250e34018ca3a294a9637b559-json.log': No such file or directory

The following change mostly (with an exception for RTR logs) made it work for me (found here):

$ git diff helm/eirini/templates/daemonset-fluentd.yaml
diff --git a/helm/eirini/templates/daemonset-fluentd.yaml b/helm/eirini/templates/daemonset-fluentd.yaml
index f63477a..e899789 100644
--- a/helm/eirini/templates/daemonset-fluentd.yaml
+++ b/helm/eirini/templates/daemonset-fluentd.yaml
@@ -43,6 +43,9 @@ spec:
           mountPath: /var/data
         - name: varlog
           mountPath: /var/log
+        - name: varlibdockercontainers
+          mountPath: /var/lib/docker/containers
+          readOnly: true
         - name: config-volume
           mountPath: /fluentd/etc/
           readOnly: false
@@ -80,6 +83,9 @@ spec:
       - name: varlog
         hostPath:
           path: /var/log
+      - name: varlibdockercontainers
+        hostPath:
+          path: /var/lib/docker/containers
       - name: fluentd-conf
         configMap:
           name: fluentd-conf

There is still one concern though. I've noticed that loggregator-fluentd is trying to read logs from all containers on a particular node no matter if the containers have been deployed with Eirini or not. Is this a bug or Eirini should be using dedicated nodes?

Staging tasks keeps deleted immediately.

Description

I deployed eirini-release of Kubernetes cluster version 1.13.7
When I push applications, out off 4 pushes, 2 applications getting staged and started running and for remaining staging jobs gets deleted immediately after creation.
I could see from error message blobstore was not reachable
Get https://blobstore-blobstore.scf.svc.cluster.local:4443/read/cc-buildpacks/23/bd/23bd33d2-f7f2-480c-a8e0-dee12ad1dc8b_fb7c2c7778f2294c01bc90aa10e744db29dd496d58c2db88607f491fb013fa3b?md5=7QhPSEK_nOOVOm2MVy8ikQ&expires=1562247858: dial tcp: lookup blobstore-blobstore.scf.svc.cluster.local: no such host

But I am able to curl the blobstore service inside the cluster.

Steps to reproduce

Deploy eirini-release

What was expected to happen

For all the applications, staging job has to complete successfully and app should be running.

What actually happened

For most of the time staging job get immediately deleted.

Additional information (optional)

Versions:
cf-2.15.2
uaa-2.14.5
Kubernetes version: 1.13.7
CoreDNS version: 1.4.0

App logs are missing sub-second precision

Description

We recently noticed that after enabling app logs via: https://github.com/cloudfoundry-incubator/eirini-release/blob/master/helm/eirini/templates/daemonset-fluentd.yaml that we do successfully receive all logs from our apps; however, they are missing precision.

When tailing/following logs this is not an issue in terms of log message order, but when looking back at recent logs via cf logs --recent we see that order is no longer preserved which we suspect to be the result of this lack of precision for the log messages.

Steps to reproduce

Deploy CF+Eirini
Apply the aforementioned daemonset (and other needed things) to Eirini's target cluster
Push an app
Run cf logs my-app --recent

What was expected to happen

2019-05-08T10:09:29.20-0700 [APP/PROC/WEB/0] OUT 2019-05-08T17:09:29.203203397+00:00 line 0
2019-05-08T10:09:29.20-0700 [APP/PROC/WEB/0] OUT 2019-05-08T17:09:29.203256377+00:00 line 1

NOTE: The second set of timestamps on each of the above lines (which have correct precision) are logged by the app itself from its STDOUT

What actually happened

2019-05-08T10:09:29.00-0700 [APP/PROC/WEB/0] OUT 2019-05-08T17:09:29.203203397+00:00 line 0
2019-05-08T10:09:29.00-0700 [APP/PROC/WEB/0] OUT 2019-05-08T17:09:29.203256377+00:00 line 1

NOTE: The second set of timestamps on each of the above lines (which have correct precision) are logged by the app itself from its STDOUT

Additional information (optional)

We are deploying CF using cf-deployment and eirini-bosh-release with BOSH.

We then manually apply the fluentd daemonset to our k8s cluster

Error with Blobstore: Execution expired

Description

Beyond 2500 applications, we started getting the following error from cloud controller and getting Staging error

{"timestamp":1566464473.7276309,"message":"staging package: #<VCAP::CloudController::PackageModel @values={:id=>2811, :guid=>\"40bf40b7-fcff-4e51-b7a3-9711d506b16c\", :created_at=>2019-08-22 09:00:58 UTC, :updated_at=>2019-08-22 09:01:06 UTC, :type=>\"bits\", :package_hash=>\"7633ef37e0541bb784347d3762a17541a17a1e7e\", :state=>\"READY\", :error=>nil, :app_guid=>\"0ac6cf92-5d89-4bdd-af2b-dbb09b17af91\", :docker_image=>nil, :sha256_checksum=>\"13aba6f71116666c87d5e6bd3535c0bba279cad3780a36ca80c8f3c57826c001\", :docker_username=>nil, :docker_password_salt=>nil, :encrypted_docker_password=>nil, :encryption_key_label=>nil, :encryption_iterations=>2048}> for build e078729e-552a-45d9-87a4-01bc3ce3d4e6","log_level":"info","source":"cc.action.build_create","data":{"request_guid":"a17e1497-b5c8-4dd1-6f77-5710280d8931::159d5fc2-6be2-4773-aeae-fec2e19e737d"},"thread_id":47032801735740,"fiber_id":47032804847560,"process_id":1,"file":"/var/vcap/packages/.src/82912d96bb9af0933de6dc262d6a3b8f4d6f4a8f0a753b5f3d38236d1edcff54/cloud_controller_ng/app/actions/build_create.rb","lineno":76,"method":"create_and_stage"}

{"timestamp":1566466010.312788,"message":"Error with blobstore: execution expired","log_level":"error","source":"cc.blobstore.dav_client","data":{"request_guid":"2ca3e034-c7d4-43eb-6499-8f9d0bae9348::9448d989-0d14-4b0b-8d3f-659959a6aab9"},"thread_id":47432094746100,"fiber_id":47432144148020,"process_id":1,"file":"/var/vcap/packages/.src/82912d96bb9af0933de6dc262d6a3b8f4d6f4a8f0a753b5f3d38236d1edcff54/cloud_controller_ng/lib/cloud_controller/blobstore/webdav/dav_client.rb","lineno":203,"method":"raise_blobstore_error"}

We are using webdav and disk gave as 500 GB. Out of which only 26% is consumed.

Filesystem      Size  Used Avail Use% Mounted on
overlay          47G   17G   28G  38% /
tmpfs            64M     0   64M   0% /dev
tmpfs            15G     0   15G   0% /sys/fs/cgroup
/dev/sda9        47G   17G   28G  38% /etc/hosts
shm              64M     0   64M   0% /dev/shm
tmpfs            15G  4.0K   15G   1% /opt/fissile/config
/dev/sdb        458G  115G  344G  26% /var/vcap/store
tmpfs            15G   12K   15G   1% /run/secrets/kubernetes.io/serviceaccount

Steps to reproduce

Deploy CF applications beyond 2500 applications.

What was expected to happen

No staging error and should be able to push the applications.

What actually happened

Failed in staging error as pasted above

Additional information (optional)

Eirini Release : 0.16.0
Blobstore : webdav

/tmp/lifecycle/shell missing from app pods

After deploying SCF/Eirini, I kubectl exec -ti -n scf-eirini into a pod, and tried to access node -v after setting up the environment as per [1]:

$ /tmp/lifecycle/shell
bash: /tmp/lifecycle/shell: No such file or directory

[1] https://docs.cloudfoundry.org/devguide/deploy-apps/ssh-apps.html#-ssh-session-environment

UAA pod is not in ready state

Description

Referring https://github.com/cloudfoundry-incubator/eirini-release/blob/master/docs/installation.md for deployment after performing step 5 UAA pod is not in ready state giving,
waiting 30s trying: detect mysql-proxy-set.KUBERNETES_NAMESPACE.svc.KUBERNETES_CLUSTER_DOMAIN 3306 failed
Giving up
uaa-pre-start - starting
trying to run update-ca-certificates
Starting monit daemon with http interface at

Readiness probe failed: Waiting for monit to be ready when checked with describe

Any inputs for this

Failed to pull image/filesystem layer verification failed for digest

With my current deployment of https://github.com/cloudfoundry-incubator/eirini-release/tree/master/scf on Gardener (setup as described in #13), I am able to cf push my app once. Pushing it subsequently leads to:

Failed to pull image "<value_of:kube/external_ips>:5800/cloudfoundry/app-name:<guid>": rpc error: code = Unknown desc = filesystem layer verification failed for digest sha256:<other_guid>

Here is what I have:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.0", GitCommit:"0ed33881dc4355495f623c6f22e7dd0b7632b7c0", GitTreeState:"clean", BuildDate:"2018-09-28T15:20:58Z", GoVersion:"go1.11", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.8", GitCommit:"7eab6a49736cc7b01869a15f9f05dc5b49efb9fc", GitTreeState:"clean", BuildDate:"2018-09-14T15:54:20Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

$ kubectl describe node
System Info:
<...>
 Container Runtime Version:  docker://18.6.1
 Kubelet Version:            v1.10.8
 Kube-Proxy Version:         v1.10.8

# on the Kubernetes node
$ docker --version
Docker version 18.06.1-ce, build e68fc7a

$ docker images
# doesn't show any images from <value_of:kube/external_ips>/cloudfoundry/app-name

eirini deployment should fail when ingress_endpoint is not set but use_registry_ingress is set to true

Description

Deployment should fail when the values.yaml is missing the ingress_endpoint but use_registry_ingress is set to true.

Steps to reproduce

set use_registry_ingress= true in the values.yaml without ingress_endpoint being set. Notice that the deployment succeeds.

What was expected to happen

The deployment should have failed in the absence of ingress_endpoint.

What actually happened

The deployment proceeded but then pushing an app failed with the following error.

  Normal   BackOff    25s                kubelet, 10.184.120.241  Back-off pulling image "registry.no-ingress-set:443/cloudfoundry/f234fd7d-06c6-4ca3-82c1-5e57ddd1dda4:0d047db5d98923d9508c72d8183b314531efe282"
  Warning  Failed     25s                kubelet, 10.184.120.241  Error: ImagePullBackOff
  Normal   Pulling    11s (x2 over 26s)  kubelet, 10.184.120.241  pulling image "registry.no-ingress-set:443/cloudfoundry/f234fd7d-06c6-4ca3-82c1-5e57ddd1dda4:0d047db5d98923d9508c72d8183b314531efe282"
  Warning  Failed     11s (x2 over 26s)  kubelet, 10.184.120.241  Failed to pull image "registry.no-ingress-set:443/cloudfoundry/f234fd7d-06c6-4ca3-82c1-5e57ddd1dda4:0d047db5d98923d9508c72d8183b314531efe282": rpc error: code = Unknown desc = failed to resolve image "registry.no-ingress-set:443/cloudfoundry/f234fd7d-06c6-4ca3-82c1-5e57ddd1dda4:0d047db5d98923d9508c72d8183b314531efe282": failed to do request: Head https://registry.no-ingress-set:443/v2/cloudfoundry/f234fd7d-06c6-4ca3-82c1-5e57ddd1dda4/manifests/0d047db5d98923d9508c72d8183b314531efe282: dial tcp: lookup registry.no-ingress-set: no such host

Particularly notice

Unknown desc = failed to resolve image "registry.no-ingress-set:443/cloudfoundry/f234fd7d-06c6-4ca3-82c1-5e57ddd1dda4:0d047db5d98923d9508c72d8183b314531efe282

where no-ingress-set is part of the url for the registry endpoint.

UAA Pod doesn't get ready with a valid certificate and domain configured

Description

I think there is an issue in this line:
https://github.com/cloudfoundry-incubator/eirini-release/blob/master/helm/uaa/templates/uaa.yaml#L315

The problem is that I'm using certificate that is signed for the uaa domain. I'm using the default uaa. for uaa. The problem is that the uaa pod never gets ready. This happens because the curl in the line above fails since my certificate doesn't have the domain: "uaa" signed in it.

Steps to reproduce

Deploy with a signed certificate for uaa. or any uaa domain you choose.

What was expected to happen

UAA pods won't get ready

What actually happened

The readiness probe fails.

Suggested fix (optional)

The probe should check if there is a valid certificate and a domain configured, then it should use the configured domain in the check.

Additional information (optional)

[Fill in]

CAT test "has its staging log streamed during a push" fail by using bosh release 0.0.15

CAT test "applies default environment variables while staging apps" fail by using bosh release 0.0.15

Description

We deploy by using new Eirini bosh release 0.0.15:
https://s3.amazonaws.com/suse-final-releases/eirini-release-0.0.15.tgz
And used SCF 2.16.4 and new cc release 1.82.0

When we run CATs test which based on the config:
https://github.com/cloudfoundry-incubator/eirini-ci/blob/master/pipelines/modules/run-core-cats.yml

the buildpack version:

staticfile_buildpack   1      true     false    staticfile-buildpack-cflinuxfs3-v1.4.35.zip
java_buildpack         2      true     false    java-buildpack-cflinuxfs3-v4.16.1.zip
ruby_buildpack         3      true     false    ruby-buildpack-cflinuxfs3-v1.7.42.zip
nodejs_buildpack       4      true     false    nodejs-buildpack-cflinuxfs3-v1.6.34.zip
go_buildpack           5      true     false    go-buildpack-cflinuxfs3-v1.8.29.zip
python_buildpack       6      true     false    python-buildpack-cflinuxfs3-v1.6.23.zip
php_buildpack          7      true     false    php-buildpack-cflinuxfs3-v4.3.64.zip
binary_buildpack       8      true     false    binary-buildpack-cflinuxfs3-v1.0.27.zip

We found there are still 3 failures.
one is
[Fail] [apps] [Fail] [apps] An application being staged [It] has its staging log streamed during a push
/gopath/src/github.com/cloudfoundry/cf-acceptance-tests/apps/staging_log.go:50

Steps to reproduce

[Binary buildpack] cf push CATS-2-APP-2ddd93e66d36c272 -b binary_buildpack -m 256M -p assets/catnip/bin -c ./catnip -d bcfdeirinidevelop.us-south.containers.appdomain.cloud
But fail to find any keywords in [Installing dependencies, Uploading droplet, App started] in cf logs --recent CATS-2-APP-2ddd93e66d36c272

What was expected to happen

find any keywords in [Installing dependencies, Uploading droplet, App started] in cf logs --recent CATS-2-APP-2ddd93e66d36c272

What actually happened

Cannot find any keyword in the log below:

[2019-08-20 04:18:15.23 (UTC)]> cf logs --recent CATS-2-APP-2ddd93e66d36c272 
Retrieving logs for app CATS-2-APP-2ddd93e66d36c272 in org CATS-2-ORG-90e5bb6f83d7b66a / space CATS-2-SPACE-10192ce0005c18fd as CATS-2-USER-364bfc491d9364ae...

   2019-08-20T04:17:46.11+0000 [API/0] OUT Created app with guid c2440c5d-8d77-484b-99d7-401e408711f5
   2019-08-20T04:17:47.09+0000 [API/0] OUT Uploading bits for app with guid c2440c5d-8d77-484b-99d7-401e408711f5
   2019-08-20T04:17:51.02+0000 [API/1] OUT Creating build for app with guid c2440c5d-8d77-484b-99d7-401e408711f5
   2019-08-20T04:17:51.66+0000 [API/1] OUT Updated app with guid c2440c5d-8d77-484b-99d7-401e408711f5 ({"state"=>"STARTED"})
   2019-08-20T04:17:56.00+0000 [STG/0] OUT 2019/08/20 04:17:54 downloader-done
   2019-08-20T04:17:56.00+0000 [STG/0] OUT -----> Binary Buildpack version 1.0.27
   2019-08-20T04:17:56.00+0000 [STG/0] OUT 2019/08/20 04:17:53 downloader-started
   2019-08-20T04:17:56.00+0000 [STG/0] OUT 2019/08/20 04:17:55 executor-started
   2019-08-20T04:17:56.00+0000 [STG/0] OUT 2019/08/20 04:17:56 executor-done
   2019-08-20T04:17:57.96+0000 [API/0] OUT Creating droplet for app with guid c2440c5d-8d77-484b-99d7-401e408711f5
   2019-08-20T04:17:58.00+0000 [STG/0] OUT 2019/08/20 04:17:57 droplet info: https://cc-uploader-cc-uploader.cf.svc.cluster.local:9091/v1/droplet/662c5e1f-9201-4f68-ab74-7cd4570c6ed9?cc-droplet-upload-uri=https://api.cf.svc.cluster.local:9023/internal/v4/droplets/662c5e1f-9201-4f68-ab74-7cd4570c6ed9/upload /out/droplet.tgz
   2019-08-20T04:17:58.00+0000 [STG/0] OUT 2019/08/20 04:17:57 uploader-started
   2019-08-20T04:18:03.00+0000 [STG/0] OUT 2019/08/20 04:18:03 uploader-done
   2019-08-20T04:18:10.00+0000 [APP/PROC/WEB/0] OUT listening on port 8080...
   2019-08-20T04:18:10.00+0000 [APP/PROC/WEB/0] OUT ARGS: [/lifecycle/launch]

Please let me know if you need more information.

Thanks a lot!

Deploying an app with multiple buildpacks is not supported by eirini-staging

Description

Deploying an app with multiple buildpacks as described in How Buildpacks Work in CFAR is not really supported by eirini-staging.

Steps to reproduce

Deploy an app with multiple buildpacks specified in its manifest. In my case, I have an app with one standard buildpack (java_buildpack) and one custom buldpack:

- buildpacks:
  - uas_dataflow_server_buildpack
  - java_buildpack

What was expected to happen

The sequence of interactions as specified in No Buildpack Detection. In this case , the supply script of the first buildpack should be invoked, and then the application should start with the second buildpack.

What actually happened

Depending on whether or not the first buildpack has a detect script that returns 0, one of the following outcomes:

If the first buildpack has a `detect`` script that returns 0, this buildpack is "detected" and the second buildpack is ignored. The app fails to start.
If the first buildpack doesn't have a detect script or it returns 1, this buildpack is simply ignored. The app starts with the second buildpack, but since the supply script of the first buildpack was never called, the app doesn't work as expected.

Suggested fix (optional)

The bug is in https://github.com/cloudfoundry-incubator/eirini-staging/blob/ae1327186284f350673df0d5e3b4f21045fd71fa/builder/config.go#L48-L55 . The skipDetect flag is never set if len(buildpacks) > 0. To fix this issue, this flag should be set if len(buildpacks) > 0 and the skipDetect flags of all buildpacks are true.

Additional information (optional)

Discussed with @georgethebeatle . I intend to open a PR for the fix.

Eirini docker app support only supports port 8080

Deploying a docker app with Eirini only works if it uses port 8080.

The documentation claims that CF will parse the Exposed Port from the container metadata. See "Port Configuration" on https://docs.cloudfoundry.org/devguide/deploy-apps/push-docker.html:

When configuring a Docker image for CFAR, you can control the exposed port and the corresponding value of PORT by specifying the EXPOSE directive in the image Dockerfile. If you specify the EXPOSE directive, then the corresponding app pushed to CFAR listens on that exposed port. For example, if you set EXPOSE to 7070, then the app listens for connections on port 7070.

I've verified this with the nginxdemos/hello image, which uses port 80:

$ docker inspect --format "{{.Config.ExposedPorts}}" nginxdemos/hello
map[80/tcp:{}]

This app works fine when deployed with Diego, but fails to start with Eirini. Inspecting the pod shows that Eirini has not detected the correct port settings:

Containers:
  opi:
    Port:           8080/TCP
    [...]
    Environment:
      VCAP_APP_HOST:            0.0.0.0
      VCAP_APP_PORT:            8080
      PORT:                     8080
      CF_INSTANCE_PORTS:        [{"external":8080,"internal":8080}]
      CF_INSTANCE_ADDR:         0.0.0.0:8080
      CF_INSTANCE_PORT:         8080

Eirini Events High memory consumption

Description

We did a scalability tests on cf-for-k8s version v-0.6.0. We deployed 1000 apps out of which 50 apps are meant to crash frequently. We could see eirini-events pod was consuming high memory > 10Gi and automatically restart happens and memory reduces. We suspect some memory leak.

Steps to reproduce

Deploy crashing apps - Example and observe the eirini-events pod memory usage.

What actually happened

Memory consumption is very high

CreateContainerConfigError status on newly deployed pods

Description

The newly deployed pods when installing CF after following the installation instructions (step 8) are marked with CreateContainerConfigError status.

Steps to reproduce

Installation instructions

What was expected to happen

A successful deployment.

What actually happened

All the newly started pods were marked with CreateContainerConfigError status.

Additional information (optional)

The logs for all the pods had a similar error -

Warning  Failed     4m11s (x6 over 4m53s)  kubelet, <redacted-address>  Error: secrets "secrets-2.16.4-1" not found

Not sure if relevant, but the UAA secret version is -

$ SECRET=$(kubectl get pods --namespace uaa -o jsonpath='{.items[?(.metadata.name=="uaa-0")].spec.containers[?(.name=="uaa")].env[?(.name=="INTERNAL_CA_CERT")].valueFrom.secretKeyRef.name}')
$ echo $SECRET
secrets-2.17.1-1

Update (2018-08-13)

The following error corresponding to the eirini pod is more descriptive -

  Warning  FailedMount  2m57s (x8 over 4m1s)  kubelet, <redacted-address>  MountVolume.SetUp failed for volume "cf-secrets" : secrets "secrets-2.16.4-1" not found
  Warning  FailedMount  118s                  kubelet, <redacted-address> Unable to mount volumes for pod "eirini-6fc8768c54-7vhkd_scf(678e105e-bdb2-11e9-9382-baf1f4ff5aef)": timeout expired waiting for volumes to attach or mount for pod "scf"/"eirini-6fc8768c54-7vhkd". list of unmounted volumes=[cf-secrets]. list of unattached volumes=[config-map-volume config-volume cf-secrets opi-token-rmh9w]
  Normal   Pulling      105s                  kubelet, <redacted-address>  pulling image "eirini/opi-init@sha256:a17cfbef0a67cdf5ec6af5d96440627d2f5cf893fab70b36300ab05c6f29d682"
  Normal   Pulled       98s                   kubelet, <redacted-address> Successfully pulled image "eirini/opi-init@sha256:a17cfbef0a67cdf5ec6af5d96440627d2f5cf893fab70b36300ab05c6f29d682"
  Normal   Created      98s                   kubelet, <redacted-address>  Created container
  Normal   Started      98s                   kubelet, <redacted-address>  Started container
  Normal   Started      76s (x2 over 80s)     kubelet, <redacted-address> Started container
  Warning  BackOff      73s                   kubelet, <redacted-address>  Back-off restarting failed container

cf logs doesn't work on GKE

Hi,
cf logs doesn't work on GKE due to missing mounted volumes.

We had a conversation on slack about the issue:
https://cloudfoundry.slack.com/archives/C8RU3BZ26/p1556095403095400

Mounting the volumes solves the problem and allows us to get the logs.

Thanks,
Axel

Converting values.yaml to a template

Converting values.yaml to a proper template would help in getting a development environment in an automated fashion without a need for unreliable replacements like this https://github.com/giner/eirini-on-microk8s/blob/3a788e3/Vagrantfile#L193-L201.

Misleading `INCLUDE_DEPLOYMENTS: true` for CATS

Description

Deployments CATS are enabled, but since they depend on V3 tests they are skipped in CI

Steps to reproduce

Execute CATS in CI

What was expected to happen

Do not include deployments (https://github.com/cloudfoundry-incubator/eirini-ci/blob/7fd82ea15666ac2e92767cba12d76faf3ac47691/pipelines/modules/run-core-cats.yml#L35) until V3 are enabled

What actually happened

Misleading INCLUDE_DEPLOYMENTS: true param

Suggested fix (optional)

Remove INCLUDE_DEPLOYMENTS: true from CI

Additional information (optional)

Proposal: add Service to each application

Towards allowing cf push applications to become more native to Kubernetes, I'd like to propose that each application also includes a Service.

An example manually created Service example below:

$ curl sample-app-nodejs-dev-d50e44dc8d.scf-eirini.svc.cluster.local:8080
Hello World!

$ kubectl get all -n scf-eirini
NAME                                     READY   STATUS    RESTARTS   AGE
pod/kwt-net                              1/1     Running   0          8m32s
pod/sample-app-nodejs-dev-bcb9c9b5a7-0   1/1     Running   0          52s
pod/sample-app-nodejs-dev-bcb9c9b5a7-1   1/1     Running   0          52s
pod/sample-app-nodejs-dev-bcb9c9b5a7-2   0/1     Running   0          52s

NAME                                       TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
service/sample-app-nodejs-dev-d50e44dc8d   ClusterIP   10.12.2.168   <none>        8080/TCP   10m

NAME                                                READY   AGE
statefulset.apps/sample-app-nodejs-dev-bcb9c9b5a7   2/3     54s

The definition of the service would bind to the guid label on the application, such as:

# Idea for an Eirini service mapping to a CF app's pods
# `cf app sample-app-nodejs --guid`
# => 51c11bc3-f2ce-407a-836d-b5bd66847f74
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    guid: 51c11bc3-f2ce-407a-836d-b5bd66847f74
  name: sample-app-nodejs-dev-d50e44dc8d
  namespace: scf-eirini
spec:
  ports:
  - name: 8080-8080
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    guid: 51c11bc3-f2ce-407a-836d-b5bd66847f74
  type: ClusterIP
status:
  loadBalancer: {}

Would there be risks/downsides to including a Service with each application?

Maybe this is only half a feature - what is the right & useful name for the service etc; what annotations to attach; what are the range of use cases?

I could investigate an implementation (ooh via eirinix maybe) if you're ok with this proposal.

Contributing guidelines test script is missing

./scripts/check-everything.sh is referenced in https://github.com/cloudfoundry-incubator/eirini-release/blob/develop/.github/contributing.md#unit-tests--static-checks but it does not exist.

It should either exist or the documentation should be updated to remove references to it.

CAT test "applies default environment variables while staging apps" fail by using bosh release 0.0.15

Description

We deploy by using new Eirini bosh release 0.0.15:
https://s3.amazonaws.com/suse-final-releases/eirini-release-0.0.15.tgz
And used SCF 2.16.4 and new cc release 1.82.0

When we run CATs test which based on the config:
https://github.com/cloudfoundry-incubator/eirini-ci/blob/master/pipelines/modules/run-core-cats.yml

the buildpack version:

staticfile_buildpack   1      true     false    staticfile-buildpack-cflinuxfs3-v1.4.35.zip
java_buildpack         2      true     false    java-buildpack-cflinuxfs3-v4.16.1.zip
ruby_buildpack         3      true     false    ruby-buildpack-cflinuxfs3-v1.7.42.zip
nodejs_buildpack       4      true     false    nodejs-buildpack-cflinuxfs3-v1.6.34.zip
go_buildpack           5      true     false    go-buildpack-cflinuxfs3-v1.8.29.zip
python_buildpack       6      true     false    python-buildpack-cflinuxfs3-v1.6.23.zip
php_buildpack          7      true     false    php-buildpack-cflinuxfs3-v4.3.64.zip
binary_buildpack       8      true     false    binary-buildpack-cflinuxfs3-v1.0.27.zip

We found there are still 3 failures.
one is
[Fail] [apps] Default Environment Variables Default staging environment variables [It] applies default environment variables while staging apps
/gopath/src/github.com/cloudfoundry/cf-acceptance-tests/apps/default_environment_variables.go:104

Steps to reproduce

[Custom buildpack] cf push CATS-1-APP-a5f0ea4e151f1404 -p assets/hello-world -b CATS-1-BPK-58f902089bc0e29f -m 256M -d bcfdeirinidevelop.us-south.containers.appdomain.cloud
But fail to find the CF_INSTANCE_INTERNAL_IP=.* message in cf logs --recent CATS-1-APP-a5f0ea4e151f1404

What was expected to happen

Get message like:

      to match regular expression
          <string>: CF_INSTANCE_INTERNAL_IP=.*

What actually happened

But get message like and no message above:

          <string>: Retrieving logs for app CATS-1-APP-a5f0ea4e151f1404 in org CATS-1-ORG-8d1fbf15573dcf30 / space CATS-1-SPACE-2f934991d05d263e as CATS-1-USER-f343f7d8ebca8147...
          
             2019-08-20T03:55:45.25+0000 [API/0] OUT Created app with guid def8bd4b-7d3c-4ad5-bfc6-f67fcd3708c8
             2019-08-20T03:55:46.07+0000 [API/0] OUT Uploading bits for app with guid def8bd4b-7d3c-4ad5-bfc6-f67fcd3708c8
             2019-08-20T03:55:50.07+0000 [API/1] OUT Creating build for app with guid def8bd4b-7d3c-4ad5-bfc6-f67fcd3708c8
             2019-08-20T03:55:50.83+0000 [API/1] OUT Updated app with guid def8bd4b-7d3c-4ad5-bfc6-f67fcd3708c8 ({"state"=>"STARTED"})
             2019-08-20T03:55:54.00+0000 [STG/0] OUT KUBERNETES_SERVICE_HOST=172.21.0.1
             2019-08-20T03:55:54.00+0000 [STG/0] OUT DOWNLOAD_URL=http://bcfdeirinidevelopussouthcontai.20190819.154357-cc-packages.s3.us.cloud-object-storage.appdomain.cloud/33/67/336728c7-7d67-4131-ae61-55424ebad052?X-Amz-Expires=3600&X-Amz-Date=20190820T035550Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=gGVWvxt67tmin1LQvlaE/20190820/us-east-1/s3/aws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=52a7b87e34277767ea66d37f090d260928f10fea28d9202b8e4f77500cf31f94
             2019-08-20T03:55:54.00+0000 [STG/0] OUT VCAP_SERVICES={}
             2019-08-20T03:55:54.00+0000 [STG/0] OUT KUBERNETES_SERVICE_PORT=443
             2019-08-20T03:55:54.00+0000 [STG/0] OUT KUBERNETES_PORT_443_TCP=tcp://172.21.0.1:443
             2019-08-20T03:55:54.00+0000 [STG/0] OUT HOME=/root
             2019-08-20T03:55:54.00+0000 [STG/0] OUT STAGING_GUID=b8e168c7-81ab-44ef-9979-c1932375f3b3
             2019-08-20T03:55:54.00+0000 [STG/0] OUT EIRINI_ADDRESS=http://eirini-opi.cf.svc.cluster.local:8085
             2019-08-20T03:55:54.00+0000 [STG/0] OUT KUBERNETES_PORT_443_TCP_ADDR=172.21.0.1
             2019-08-20T03:55:54.00+0000 [STG/0] OUT KUBERNETES_PORT_443_TCP_PROTO=tcp
             2019-08-20T03:55:54.00+0000 [STG/0] OUT APP_ID=def8bd4b-7d3c-4ad5-bfc6-f67fcd3708c8
             2019-08-20T03:55:54.00+0000 [STG/0] OUT SHLVL=1
             2019-08-20T03:55:54.00+0000 [STG/0] OUT KUBERNETES_PORT_443_TCP_PORT=443
             2019-08-20T03:55:54.00+0000 [STG/0] OUT HOSTNAME=b8e168c7-81ab-44ef-9979-c1932375f3b3-5prb2
             2019-08-20T03:55:54.00+0000 [STG/0] OUT LANG=en_US.UTF-8
             2019-08-20T03:55:54.00+0000 [STG/0] OUT COMPLETION_CALLBACK=https://internal_user:KPV8yr19U5ZXLVdsNSbfAGUKQUEWZFLuQ6zH6jdGnA1dJ7x4GSTj92Cw9IX4WW9X@api.cf.svc.cluster.local:9023/internal/v3/staging/b8e168c7-81ab-44ef-9979-c1932375f3b3/build_completed?start=true
             2019-08-20T03:55:54.00+0000 [STG/0] OUT KUBERNETES_SERVICE_PORT_HTTPS=443
             2019-08-20T03:55:54.00+0000 [STG/0] OUT BUILDPACKS=[{"name":"CATS-1-BPK-58f902089bc0e29f","key":"3b464b63-1729-4b97-84df-8b44f2d7194f_536663766532e9aeb86886ab3e576efdbaaa20283c056c0651e323ebb3f6403d","url":"http://bcfdeirinidevelopussouthcontai.20190819.154357-cc-buildpacks.s3.us.cloud-object-storage.appdomain.cloud/3b/46/3b464b63-1729-4b97-84df-8b44f2d7194f_536663766532e9aeb86886ab3e576efdbaaa20283c056c0651e323ebb3f6403d?X-Amz-Expires=3600\u0026X-Amz-Date=20190820T035550Z\u0026X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=gGVWvxt67tmin1LQvlaE/20190820/us-east-1/s3/aws4_request\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Signature=8c6e708d4c087fb2f9c45edd3d5f30e89765c5b19ec5e1c982bc55bcfc3e1641","skip_detect":true}]
             2019-08-20T03:55:54.00+0000 [STG/0] OUT PWD=/workspace
             2019-08-20T03:55:54.00+0000 [STG/0] OUT KUBERNETES_PORT=tcp://172.21.0.1:443
             2019-08-20T03:55:54.00+0000 [STG/0] OUT MEMORY_LIMIT=1024m
             2019-08-20T03:55:54.00+0000 [STG/0] OUT VCAP_APPLICATION={"cf_api":"https://api.bcfdeirinidevelop.us-south.containers.appdomain.cloud","limits":{"fds":16384,"mem":1024,"disk":4096},"application_name":"CATS-1-APP-a5f0ea4e151f1404","application_uris":["cats-1-app-a5f0ea4e151f1404.bcfdeirinidevelop.us-south.containers.appdomain.cloud"],"name":"CATS-1-APP-a5f0ea4e151f1404","space_name":"CATS-1-SPACE-2f934991d05d263e","space_id":"fa0a5d3f-4ce2-40b3-8dff-0eedcb5230e6","uris":["cats-1-app-a5f0ea4e151f1404.bcfdeirinidevelop.us-south.containers.appdomain.cloud"],"users":null,"application_id":"def8bd4b-7d3c-4ad5-bfc6-f67fcd3708c8","version":"2134ef06-e331-426b-839b-dbbbda9aa787","application_version":"2134ef06-e331-426b-839b-dbbbda9aa787"}
             2019-08-20T03:55:54.00+0000 [STG/0] OUT 2019/08/20 03:55:52 downloader-done
             2019-08-20T03:55:54.00+0000 [STG/0] OUT 2019/08/20 03:55:53 executor-started
             2019-08-20T03:55:54.00+0000 [STG/0] OUT _=/usr/bin/env
             2019-08-20T03:55:54.00+0000 [STG/0] OUT PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
             2019-08-20T03:55:54.00+0000 [STG/0] OUT DROPLET_UPLOAD_URL=https://cc-uploader-cc-uploader.cf.svc.cluster.local:9091/v1/droplet/b8e168c7-81ab-44ef-9979-c1932375f3b3?cc-droplet-upload-uri=https://api.cf.svc.cluster.local:9023/internal/v4/droplets/b8e168c7-81ab-44ef-9979-c1932375f3b3/upload
             2019-08-20T03:55:54.00+0000 [STG/0] OUT CF_STACK=cflinuxfs3
             2019-08-20T03:55:54.00+0000 [STG/0] OUT 2019/08/20 03:55:52 downloader-started
             2019-08-20T03:55:58.00+0000 [STG/0] OUT exit status 1
             2019-08-20T03:56:02.00+0000 [STG/0] OUT COMPLETION_CALLBACK=https://internal_user:KPV8yr19U5ZXLVdsNSbfAGUKQUEWZFLuQ6zH6jdGnA1dJ7x4GSTj92Cw9IX4WW9X@api.cf.svc.cluster.local:9023/internal/v3/staging/b8e168c7-81ab-44ef-9979-c1932375f3b3/build_completed?start=true
             2019-08-20T03:56:02.00+0000 [STG/0] OUT KUBERNETES_PORT_443_TCP=tcp://172.21.0.1:443
             2019-08-20T03:56:02.00+0000 [STG/0] OUT DROPLET_UPLOAD_URL=https://cc-uploader-cc-uploader.cf.svc.cluster.local:9091/v1/droplet/b8e168c7-81ab-44ef-9979-c1932375f3b3?cc-droplet-upload-uri=https://api.cf.svc.cluster.local:9023/internal/v4/droplets/b8e168c7-81ab-44ef-9979-c1932375f3b3/upload
             2019-08-20T03:56:02.00+0000 [STG/0] OUT BUILDPACKS=[{"name":"CATS-1-BPK-58f902089bc0e29f","key":"3b464b63-1729-4b97-84df-8b44f2d7194f_536663766532e9aeb86886ab3e576efdbaaa20283c056c0651e323ebb3f6403d","url":"http://bcfdeirinidevelopussouthcontai.20190819.154357-cc-buildpacks.s3.us.cloud-object-storage.appdomain.cloud/3b/46/3b464b63-1729-4b97-84df-8b44f2d7194f_536663766532e9aeb86886ab3e576efdbaaa20283c056c0651e323ebb3f6403d?X-Amz-Expires=3600\u0026X-Amz-Date=20190820T035550Z\u0026X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=gGVWvxt67tmin1LQvlaE/20190820/us-east-1/s3/aws4_request\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Signature=8c6e708d4c087fb2f9c45edd3d5f30e89765c5b19ec5e1c982bc55bcfc3e1641","skip_detect":true}]
             2019-08-20T03:56:02.00+0000 [STG/0] OUT HOME=/root
             2019-08-20T03:56:02.00+0000 [STG/0] OUT PWD=/workspace
             2019-08-20T03:56:02.00+0000 [STG/0] OUT 2019/08/20 03:56:00 downloader-started
             2019-08-20T03:56:02.00+0000 [STG/0] OUT PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
             2019-08-20T03:56:02.00+0000 [STG/0] OUT KUBERNETES_SERVICE_PORT=443
             2019-08-20T03:56:02.00+0000 [STG/0] OUT MEMORY_LIMIT=1024m
             2019-08-20T03:56:02.00+0000 [STG/0] OUT KUBERNETES_PORT_443_TCP_PROTO=tcp
             2019-08-20T03:56:02.00+0000 [STG/0] OUT APP_ID=def8bd4b-7d3c-4ad5-bfc6-f67fcd3708c8
             2019-08-20T03:56:02.00+0000 [STG/0] OUT HOSTNAME=b8e168c7-81ab-44ef-9979-c1932375f3b3-wlv8n
             2019-08-20T03:56:02.00+0000 [STG/0] OUT _=/usr/bin/env
             2019-08-20T03:56:02.00+0000 [STG/0] OUT VCAP_SERVICES={}
             2019-08-20T03:56:02.00+0000 [STG/0] OUT SHLVL=1
             2019-08-20T03:56:02.00+0000 [STG/0] OUT STAGING_GUID=b8e168c7-81ab-44ef-9979-c1932375f3b3
             2019-08-20T03:56:02.00+0000 [STG/0] OUT LANG=en_US.UTF-8
             2019-08-20T03:56:02.00+0000 [STG/0] OUT 2019/08/20 03:56:00 downloader-done
             2019-08-20T03:56:02.00+0000 [STG/0] OUT KUBERNETES_SERVICE_HOST=172.21.0.1
             2019-08-20T03:56:02.00+0000 [STG/0] OUT DOWNLOAD_URL=http://bcfdeirinidevelopussouthcontai.20190819.154357-cc-packages.s3.us.cloud-object-storage.appdomain.cloud/33/67/336728c7-7d67-4131-ae61-55424ebad052?X-Amz-Expires=3600&X-Amz-Date=20190820T035550Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=gGVWvxt67tmin1LQvlaE/20190820/us-east-1/s3/aws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=52a7b87e34277767ea66d37f090d260928f10fea28d9202b8e4f77500cf31f94
             2019-08-20T03:56:02.00+0000 [STG/0] OUT KUBERNETES_PORT_443_TCP_PORT=443
             2019-08-20T03:56:02.00+0000 [STG/0] OUT KUBERNETES_SERVICE_PORT_HTTPS=443
             2019-08-20T03:56:02.00+0000 [STG/0] OUT KUBERNETES_PORT=tcp://172.21.0.1:443
             2019-08-20T03:56:02.00+0000 [STG/0] OUT 2019/08/20 03:56:01 executor-started
             2019-08-20T03:56:02.00+0000 [STG/0] OUT CF_STACK=cflinuxfs3
             2019-08-20T03:56:02.00+0000 [STG/0] OUT KUBERNETES_PORT_443_TCP_ADDR=172.21.0.1
             2019-08-20T03:56:02.00+0000 [STG/0] OUT EIRINI_ADDRESS=http://eirini-opi.cf.svc.cluster.local:8085
             2019-08-20T03:56:02.00+0000 [STG/0] OUT VCAP_APPLICATION={"cf_api":"https://api.bcfdeirinidevelop.us-south.containers.appdomain.cloud","limits":{"fds":16384,"mem":1024,"disk":4096},"application_name":"CATS-1-APP-a5f0ea4e151f1404","application_uris":["cats-1-app-a5f0ea4e151f1404.bcfdeirinidevelop.us-south.containers.appdomain.cloud"],"name":"CATS-1-APP-a5f0ea4e151f1404","space_name":"CATS-1-SPACE-2f934991d05d263e","space_id":"fa0a5d3f-4ce2-40b3-8dff-0eedcb5230e6","uris":["cats-1-app-a5f0ea4e151f1404.bcfdeirinidevelop.us-south.containers.appdomain.cloud"],"users":null,"application_id":"def8bd4b-7d3c-4ad5-bfc6-f67fcd3708c8","version":"2134ef06-e331-426b-839b-dbbbda9aa787","application_version":"2134ef06-e331-426b-839b-dbbbda9aa787"}

Please let me know if you need more information.

Thanks a lot!

Garden cannot start on diego-cell pod due to grootfs error

Description

Full error from /var/vcap/sys/log/monit/garden.err.log in the diego-cell container.

/var/vcap/packages/greenskeeper/bin/system-preparation: line 50: /sys/kernel/mm/transparent_hugepage/enabled: Read-only file system
Could not disable automatic transparent hugepage allocation. This is normal in bosh lite.
{"timestamp":"1558527832.484694481","source":"grootfs","message":"grootfs.init-store.store-manager-init-store.overlayxfs-init-filesystem.mounting-filesystem-failed","log_level":2,"data":{"error":"exit status 32: mount: wrong fs type, bad option, bad superblock on /dev/loop0,\n       missing codepage or helper program, or other error\n\n       In some cases useful info is found in syslog - try\n       dmesg | tail or so.\n","filesystemPath":"/var/vcap/data/grootfs/store/unprivileged.backing-store","session":"1.1.2","spec":{"UIDMappings":[{"HostID":4294967294,"NamespaceID":0,"Size":1},{"HostID":1,"NamespaceID":1,"Size":4294967293}],"GIDMappings":[{"HostID":4294967294,"NamespaceID":0,"Size":1},{"HostID":1,"NamespaceID":1,"Size":4294967293}],"StoreSizeBytes":49536401408},"storePath":"/var/vcap/data/grootfs/store/unprivileged"}}
{"timestamp":"1558527832.484861851","source":"grootfs","message":"grootfs.init-store.store-manager-init-store.initializing-filesystem-failed","log_level":2,"data":{"backingstoreFile":"/var/vcap/data/grootfs/store/unprivileged.backing-store","error":"Mounting filesystem: exit status 32: mount: wrong fs type, bad option, bad superblock on /dev/loop0,\n       missing codepage or helper program, or other error\n\n       In some cases useful info is found in syslog - try\n       dmesg | tail or so.\n","session":"1.1","spec":{"UIDMappings":[{"HostID":4294967294,"NamespaceID":0,"Size":1},{"HostID":1,"NamespaceID":1,"Size":4294967293}],"GIDMappings":[{"HostID":4294967294,"NamespaceID":0,"Size":1},{"HostID":1,"NamespaceID":1,"Size":4294967293}],"StoreSizeBytes":49536401408},"storePath":"/var/vcap/data/grootfs/store/unprivileged"}}
{"timestamp":"1558527832.484996080","source":"grootfs","message":"grootfs.init-store.init-store-failed","log_level":2,"data":{"error":"initializing filesyztem: Mounting filesystem: exit status 32: mount: wrong fs type, bad option, bad superblock on /dev/loop0,\n       missing codepage or helper program, or other error\n\n       In some cases useful info is found in syslog - try\n       dmesg | tail or so.\n","session":"1"}}

We are using a Kubernetes cluster on AWS provisioned with Gardener.

Steps to reproduce

I imagine it is difficult for your if your don't use Gardener as Kubernetes cluster provider, but we are available to provide more information about the environment.

Suggested fix

We fixed the issue by patching the current docker image used by diego-cell with the latest grootfs binary (compiled from master of grootfs repo).

The suggestion would then be to use a newer version of grootfs in the diego-cell.

Deployment of Eirini-cf on GKE ~charts showing error~ bits.yaml:42: unexpected "=" in operand

Description

I tried to deploy Eirini on GKE cluster. Eirini-uaa deployed successfully. Eirini-cf showing error in charts.
Error: parse error in "cf/charts/eirini/templates/bits.yaml": template: cf/charts/eirini/templates/bits.yaml:42: unexpected "=" in operand

If I check the charts, line 42 , it checks for blobstore_config. If I am not wrong that section is introduced to support configurable blobstore based on PR. But I want to use the default blobstore.

Steps to reproduce

Deploy Eirini-release on GKE.
GKE version: 1.12.7-gke.10

TCP Routing tests not executed as part of CATS

Description

While comparing Diego & Eirini features I decided to take a look at the CATS that are used to confirm feature parity. Turns out TCP Routing tests are not included.

Steps to reproduce

Execute CATS

What was expected to happen

TCP Routing should be clearly marked as supported (or unsupported?)

What actually happened

It's not clear if Eirini installation can pass the TCP Routing tests or not.

Suggested fix (optional)

Explicitly specify either INCLUDE_TCP_ROUTING: true or INCLUDE_TCP_ROUTING: false in config

Additional information (optional)

Supporting Ingress rather than LoadBalancer services

Description

In order to leverage existing Ingress Controllers for a cluster and reduce waste of outward-facing IPs/loadbalanacers && overall reduce deployment complexity (I... I think?), it would be beneficial to have the option to use an Ingress rather than Service: LoadBalancer for services.

Additional information (optional)

E.g. something similar to the suse/scf chart part for Ingress...?

Thanks so much for your time & consideration.

Please configure GITBOT

Pivotal provides the Gitbot service to synchronize issues and pull requests made against public GitHub repos with Pivotal Tracker projects.

If you do not want to use Pivotal Tracker to manage this GitHub repo, you do not need to take any action.

If you are a Pivotal employee, you can configure Gitbot to sync your GitHub repo to your Pivotal Tracker project with a pull request.

Steps:

Fork this repo: cfgitbot-config (an ask+cf@ ticket is the fastest way to get read access if you get a 404)
Add the Toolsmiths-Bots team to have admin access to your repo
Add the cf-gitbot ([email protected]) user to have owner access to your Pivotal Tracker project
Add your new repo and or project to config-production.yml file
Submit a PR, which will get auto-merged if you've done it right

If you are not a pivotal employee, you can request that [email protected] set up the integration for you.

You might also be interested in configuring GitHub's Service Hook for Tracker on your repo so you can link your commits to Tracker stories. You can do this yourself by following the directions at:

https://www.pivotaltracker.com/blog/guide-githubs-service-hook-tracker/

If there are any questions, please reach out to [email protected].

deny-app-ingress doesn't work for me

Description

If found that in my deployment of SCF/Eirini, the deny-app-ingress K8s network policy prevents access to my apps from the gorouter, resulting in 502 Bad Gateway: Registered endpoint failed to handle the request. errors.

I think, the reason is that the network policy tries to restrict access to only allow access from the scf namespace (https://github.com/cloudfoundry-incubator/eirini-release/blob/master/helm/eirini/templates/networkpolicy.yml#L14 and https://github.com/cloudfoundry-incubator/eirini-release/blob/master/helm/eirini/templates/networkpolicy.yml#L20), but as far as I understood (see also kubernetes/kubernetes#47797) namespaceSelector/matchLabels indeed only filters for labels. At least in my setup (maybe I'm doing something wrong?), the scf namespace does not have a name=scf label.

Steps to reproduce

Deploy SCF and Eirini. Call an arbitrary CF app. Result for me is 502 Bad Gateway: Registered endpoint failed to handle the request..

What was expected to happen

Being able to call my CF app.

What actually happened

502 Bad Gateway: Registered endpoint failed to handle the request.

Suggested fix (optional)

Change the network policy to determine the source namespace based on some other criteria.

Additional information (optional)

As soon as I do

kubectl label namespace scf name=scf

, the access works properly.

GKE deployment instructions for SCF/Eirini please

I was hoping that the incompatible SCF requirements for GKE [1] would be lifted for Erini. I was hoping they were Diego requirements.

Could we please have dedicated GKE/SCF/Eirini instructions?

[1] SUSE/scf#1641 (comment)

cloudfoundry / eirini-release Goto Github PK

eirini-release's Introduction

Eirini Release

Installation

Building the yaml release

Security

Security Overview

Securing the Eirini Deployment

Scalability

Differences with Diego

Environment variables

Docker images running with the root user

Task retries and parallelism

Troubleshooting

Disk full on blobstore

Resources

eirini-release's People

Contributors

Stargazers

Watchers

Forkers

eirini-release's Issues

Description

Steps to reproduce

What was expected to happen

What actually happened

Additional information (optional)

Description

Steps to reproduce

What was expected to happen

What actually happened

Suggested fix (optional)

Additional information (optional)

Description

Suggested fix (optional)

Description

Steps to reproduce

Description

Steps to reproduce

What was expected to happen

What actually happened

Additional information (optional)

Description

Steps to reproduce

What was expected to happen

What actually happened

Suggested fix (optional)

Additional information (optional)

Description

Steps to reproduce

What was expected to happen

What actually happened

Suggested fix (optional)

Additional information (optional)

Description

Steps to reproduce

What was expected to happen

What actually happened

Suggested fix (optional)

Additional information (optional)

Description

Steps to reproduce

What was expected to happen

What actually happened

Suggested fix (optional)

Description

Steps to reproduce

What was expected to happen

What actually happened

Suggested fix (optional)

Additional information (optional)

Description

Steps to reproduce

What was expected to happen

What actually happened

Additional information

Description

Steps to reproduce

What was expected to happen

What actually happened