ibm-cloud-architecture / refarch-cloudnative-spring

Base repository for Spring Reference Architecture

Run the Spring Stack on a Kubernetes Cluster

• Introduction
• Spring Stack Overview
• Project Repositories
• Pre-Requisites
  • Download required CLIs
  • Create a Kubernetes Cluster
• Deploy the Spring Stack
  • 1. Deploy RabbitMQ
  • 2. Deploy Spring Stack Chart
  • 3. Validate the Spring Stack
• Delete the Spring Stack and RabbitMQ
• Optional Deployments
  • Deploy Spring Stack to IBM Cloud Private

Introduction

Run existing Spring Cloud applications in Kubernetes cluster

Spring Stack Overview

Project Repositories

This project organized itself like a microservice project, as such each component in the architecture has its own Git Repository and tutorial listed below.

• refarch-cloudnative-spring - The root repository (Current repository)
• refarch-cloudnative-netflix-eureka - This repository contains a basic Netflix Eureka application.
• refarch-cloudnative-netflix-zuul - This repository contains a basic Netflix Zuul proxy application.
• refarch-cloudnative-spring-config - This repository contains a packaged Spring Config config server for use in a Netflix OSS-based microservices architecture.
• refarch-cloudnative-netflix-hystrix - This repository contains a basic Netflix Hystrix Dashboard application, configured to use messaging for inter-component communication.
• refarch-cloudnative-netflix-turbine - This repository contains a basic Netflix Turbine Server application, configured to use messaging for inter-component communication.
• refarch-cloudnative-zipkin - This repository contains Zipkin, a distributed tracing system.

Pre-Requisites

To run the Spring Stack you will need to configure your environment for the Kubernetes and Microservices runtimes.

Download required CLIs

To deploy the Sprint Stack, you require the following tools:

• kubectl (Kubernetes CLI) - Follow the instructions here to install it on your platform.
• helm (Kubernetes package manager) - Follow the instructions here to install it on your platform.

Create a Kubernetes Cluster

The following clusters have been tested with this sample application:

• minikube - Create a single node virtual cluster on your workstation
• IBM Cloud Container Service - Create a Kubernetes cluster in IBM Cloud. The application runs in the Lite cluster, which is free of charge. Follow the instructions here.
• IBM Cloud private - Create a Kubernetes cluster in an on-premise datacenter. The community edition (IBM Cloud private-ce) is free of charge. Follow the instructions here to install IBM Cloud private-ce.

Deploy the Spring Stack

We have packaged all the application components as Kubernetes Charts. To deploy the Spring Stack, follow the instructions to configure kubectl for access to the Kubernetes cluster.

1. Deploy RabbitMQ

RabbitMQ is used as a message bus between the Spring Stack components and needs to be installed first. Use the following to install RabbitMQ.

a. Clone the Repository

$ git clone https://github.com/ibm-cloud-architecture/refarch-cloudnative-spring.git
$ cd refarch-cloudnative-spring

b. Go to rabbitmq folder.

$ cd rabbitmq

c. Deploy RabbitMQ PVC.

The PVC (Persistent Volume Claim) is needed by RabbitMQ to retain its data. Use the following to create the PVC.

$ kubectl create -f rabbitmq-pvc.yaml

Note that the PVC usually takes a few minutes to be ready. To check the PVC provisioning status, use the following.

$ kubectl get persistentvolumeclaims rabbitmq-data
NAME            STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
rabbitmq-data   Bound     pvc-bb7f0615-b35d-11e7-ad06-aee2d2458315   20Gi       RWO            default        22h

Use the command above until you see a STATUS of Bound, which means that the PVC is bound to the cluster and ready to use.

d. Create RabbitMQ Deployment

The deployment creates and deploys the RabbitMQ container. Create the deployment as follows.

$ kubectl create -f rabbitmq-deployment.yaml

Note that the deployment creates the following RabbitMQ credentials:

• Username: guest
• Password: guest

The above credentials will be used when deploying the Spring Stack chart.

e. Create RabbitMQ Service

In order to expose RabbitMQ to other applications/services, a service needs to be created. Create the service as follows.

$ kubectl create -f rabbitmq-service.yaml

You have successfully deployed RabbitMQ in your cluster!

2. Deploy Spring Stack Chart

a. Initialize helm in your cluster.

$ helm init

This initializes the helm client as well as the server side component called tiller.

b. Add the helm package repository containing the reference application:

$ helm repo add ibmcase-spring https://raw.githubusercontent.com/ibm-cloud-architecture/refarch-cloudnative-spring/master/docs/charts/

c. Install the reference application:

$ helm install --name spring-stack ibmcase-spring/spring-stack \
--set global.rabbitmq.host=rabbitmq \
--set global.rabbitmq.username=guest \
--set global.rabbitmq.password=guest \
--set spring-config-server.spring.cloud.config.server.git.uri=https://github.com/ibm-cloud-architecture/fortune-teller \
--set spring-config-server.spring.cloud.config.server.git.searchPaths=configuration \
--set spring-eureka-server.service.type=NodePort

After around 5 minutes minutes the containers will be deployed to the cluster and ready to use. The command above does the following:

• Installs all of the individual components' charts:
  • Eureka
  • Hystrix
  • Turbine
  • Zuul
  • Zipkin
  • Config Server
• Sets the installed RabbitMQ service as its message bus
• Sets the YAMLs at https://github.com/ibm-cloud-architecture/fortune-teller/tree/master/configuration as the Spring Config Server source of truth
• Exposes both the Eureka and Hystrix dashboards to the internet via a cluster NodePort.

You have succesfully deployed the Spring Stack to your cluster! In the following section, we will validate the Spring Stack deployment

3. Validate the Spring Stack

In order to validate a successful Spring Stack installation, we need to open the Eureka dashboard on a web browser and see if all of the Spring components show up in the dashboard. Showing on the dashboard means that all of the services booted up successfully and were able to retrieve Eureka dashboard URL from the Spring Config server and, therefore, register themselves against Eureka.

Since we exposed the Eureka service through NodePort, we are going to need to retrieve the following:

• Public IP of any of the cluster's worker nodes
• The port number, orNodePort, that was randomly assigned to Eureka service upon deployment.

a. Get Node IPs

The approach to get the public IP of any of your worker nodes will vary based on you cluster setup, but the most common way of doing so is as follows:

$ kubectl get nodes
NAME            STATUS    ROLES     AGE       VERSION
167.51.31.123   Ready     <none>    132d      v1.5.6-4+abe34653415733
167.51.31.124   Ready     <none>    132d      v1.5.6-4+abe34653415733
167.51.31.125   Ready     <none>    132d      v1.5.6-4+abe34653415733

The NAME column on the left lists the public IPs of all of your worker nodes. Any of the public IP's will work, so pick one.

Minikube

If you've installed on minikube you can find the IP with the following.

$ minikube ip

IBM Cloud Container Service

If you've installed on a Kubernetes cluster from IBM Cloud Container Service, you can find the IP with the following

$ bx cs workers ${cluster_name}

b. Get NodePort

To get the Eureka service NodePort, use the following.

$ kubectl get services spring-stack-spring-eureka-server
NAME                                TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
spring-stack-spring-eureka-server   NodePort   10.10.10.252   <none>        8761:31476/TCP   29m

The NodePort is listed under the PORT(S) column between 8761: and /TCP. In this case, the NodePort is 31476. Just keep in mind that this port number gets assigned randomly and can change if Eureka service was deleted and recreated.

c. Open Eureka Dashboard

Now that you have the Node IP and the NodePort, open a new browser window and enter the following:

http://${node_ip}:${node_port}

Where,

• ${node_ip} is the Node IP.
• ${node_port} is the NodePort.

You should see Dashboard similar to the following:

If you can see EUREKA, HYSTRIX-DASHBOARD, ZIPKIN-TRACING, and ZUUL-PROXY listed in the Instances currently registered with Eureka table, that means all of the components are booted up and ready to go!

Delete the Spring Stack and RabbitMQ

To delete the Spring Stack and RabbitMQ from your cluster, run the following:

$ helm delete spring-stack --purge
$ kubectl delete services rabbitmq
$ kubectl delete deployments rabbitmq
$ kubectl delete persistentvolumeclaims rabbitmq-data

Optional Deployments

Deploy Spring Stack to IBM Cloud Private

IBM Cloud Private (ICP) contains integration with Helm that allows you to install the application without the need to go to a command line.

1. Install RabbitMQ Chart

1. Click on the three bars in the top left corner, and go to Catalog.
2. Search for RabbitMQ, then click on the first result.
3. Click Configure button at bottom right.
4. Enter rabbitmq for Release name and leave Target Namespace as default.
5. Click Install.
6. Get the RabbitMQ password as follows:

$ kubectl get secrets rabbitmq-rabbitmq -o=jsonpath='{.data.rabbitmq-password}' | base64 --decode; echo

You will need this password when installing spring-stack chart.

2. Add ibmcase-spring Helm Repository

1. Click on the three bars in the top left corner, and go to Admin.

2. Click on the Repositories tab

3. Click on Add repository. Use the following values:

   • Repository Name: ibmcase-spring
   • URL: https://raw.githubusercontent.com/ibm-cloud-architecture/refarch-cloudnative-spring/master/docs/charts/

   Click Add to add the repository.

3. Deploy Spring Stack Chart

1. Click on the three bars in the top left corner, and go to Catalog.
2. Search for spring-stack, then click on the first result.
3. Click Configure button at bottom right.
4. Enter spring-stack for Release name and leave Target Namespace as default.
5. Enter the following values for the following fields:
   • global.rabbitmq.host: rabbitmq-rabbitmq
   • global.rabbitmq.user: user
   • global.rabbitmq.password: Password you retrieved after installing RabbitMQ.
   • spring-config-server.spring.cloud.config.server.git.uri: https://github.com/ibm-cloud-architecture/fortune-teller
   • spring-config-server.spring.cloud.config.server.git.searchPaths: configuration
   • spring-eureka-server.service.type: NodePort
6. Click Install.

refarch-cloudnative-spring's Issues

Refine selector strings for Services exposing Deployments

The current selectors are too coarse-grained for some of the Services which expose Deployments.

For example, Spring Config Server:

selector:
    component: config-server

will match any deployment that specifies a component of config-server.

We should integrate {{ .Release.Name }} at a minimum (and preferably {{ .Chart.Name }} as well) across all the selectors in the Chart components.

Doc issues for deploy Spring Stack to IBM Cloud Private

1. Install RabbitMQ Chart
   The latest RabbitMQ Chart has been renamed to ibm-rabbitmq-dev, need to update doc about RabbitMQ secret accordingly to get the RabbitMQ password.

2. Deploy Spring Stack Chart
   Need to update the following parameters as changes of the latest RabbitMQ Chartwhen install the chart:

global.rabbitmq.host: rabbitmq-ibm-rabbitmq-dev
global.rabbitmq.port: 5671
global.rabbitmq.username: admin
global.rabbitmq.password: admin

spring-stack chart release on IBM Cloud Private keep CrashLoopBackOff...

env:

ubuntu: 16.04
IBM Cloud Private version: 2.1.0.3
kubenetes version: v1.10.0

Scenario
Have followed the installation steps of spring-stack on: https://github.com/ibm-cloud-architecture/refarch-cloudnative-spring#deploy-spring-stack-to-ibm-cloud-private
after that chart is installed on my IBM Cloud Private2.1.0.3 env, found that some pods keep CrashLoopBackOff

root@master:~/SpringCloud/refarch-cloudnative-spring# kubectl get pods
NAME                                                    READY     STATUS             RESTARTS   AGE
rabbitmq-ibm-rabbitmq-dev-6555d4889b-vvjzb              1/1       Running            0          1h
spring-stack-spring-config-server-c68959d66-fpw7s       1/1       Running            0          10m
spring-stack-spring-eureka-server-ddb7bbfb7-kksqf       1/1       Running            0          14m
spring-stack-spring-hystrix-dashboard-7748594bc-mkmqv   0/1       CrashLoopBackOff   6          14m
spring-stack-spring-turbine-server-5959f587b9-sxrmk     0/1       CrashLoopBackOff   6          14m
spring-stack-spring-zuul-proxy-6b5bc7fcdb-gn9d4         1/1       Running            0          14m
spring-stack-zipkin-server-7545b9d5df-qwzdv             0/1       CrashLoopBackOff   6          9m

Take zipkin as an example to get container logs:

root@master:~/SpringCloud/refarch-cloudnative-spring# kubectl logs -f spring-stack-zipkin-server-7545b9d5df-qwzdv
2018-05-25 07:20:16.724  INFO [zipkin-tracing,,,] 1 --- [           main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@762efe5d: startup date [Fri May 25 07:20:16 GMT 2018]; root of context hierarchy
2018-05-25 07:20:17.461  INFO [zipkin-tracing,,,] 1 --- [           main] f.a.AutowiredAnnotationBeanPostProcessor : JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
2018-05-25 07:20:17.834  INFO [zipkin-tracing,,,] 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'configurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$1bb54277] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
                                    ********
                                  **        **
                                 *            *
                                **            **
                                **            **
                                 **          **
                                  **        **
                                    ********
                                      ****
                                      ****
        ****                          ****
     ******                           ****                                 ***
  ****************************************************************************
    *******                           ****                                 ***
        ****                          ****
                                       **
                                       **


             *****      **     *****     ** **       **     **   **
               **       **     **  *     ***         **     **** **
              **        **     *****     ****        **     **  ***
             ******     **     **        **  **      **     **   **

:: Powered by Spring Boot ::         (v1.5.7.RELEASE)

2018-05-25 07:20:19.614  INFO [zipkin-tracing,,,] 1 --- [           main] c.c.c.ConfigServicePropertySourceLocator : Fetching config from server at: http://spring-stack-spring-config-server:8888
2018-05-25 07:20:24.839 ERROR [zipkin-tracing,,,] 1 --- [           main] o.s.boot.SpringApplication               : Application startup failed

java.lang.IllegalStateException: Could not locate PropertySource and the fail fast property is set, failing
	at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator.locate(ConfigServicePropertySourceLocator.java:132) ~[spring-cloud-config-client-1.3.2.RELEASE.jar!/:1.3.2.RELEASE]
	at org.springframework.cloud.bootstrap.config.PropertySourceBootstrapConfiguration.initialize(PropertySourceBootstrapConfiguration.java:93) ~[spring-cloud-context-1.2.3.RELEASE.jar!/:1.2.3.RELEASE]
	at org.springframework.boot.SpringApplication.applyInitializers(SpringApplication.java:567) [spring-boot-1.5.7.RELEASE.jar!/:1.5.7.RELEASE]
	at org.springframework.boot.SpringApplication.prepareContext(SpringApplication.java:338) [spring-boot-1.5.7.RELEASE.jar!/:1.5.7.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:301) [spring-boot-1.5.7.RELEASE.jar!/:1.5.7.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1118) [spring-boot-1.5.7.RELEASE.jar!/:1.5.7.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1107) [spring-boot-1.5.7.RELEASE.jar!/:1.5.7.RELEASE]
	at com.ibm.microservices.refapp.eureka.ZipkinApplication.main(ZipkinApplication.java:14) [classes!/:na]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_121]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_121]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_121]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_121]
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [app.jar:na]
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [app.jar:na]
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) [app.jar:na]
	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51) [app.jar:na]
Caused by: org.springframework.web.client.HttpServerErrorException: 500 null
	at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:66) ~[spring-web-4.3.11.RELEASE.jar!/:4.3.11.RELEASE]
	at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:700) ~[spring-web-4.3.11.RELEASE.jar!/:4.3.11.RELEASE]
	at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:653) ~[spring-web-4.3.11.RELEASE.jar!/:4.3.11.RELEASE]
	at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:613) ~[spring-web-4.3.11.RELEASE.jar!/:4.3.11.RELEASE]
	at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:531) ~[spring-web-4.3.11.RELEASE.jar!/:4.3.11.RELEASE]
	at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator.getRemoteEnvironment(ConfigServicePropertySourceLocator.java:172) ~[spring-cloud-config-client-1.3.2.RELEASE.jar!/:1.3.2.RELEASE]
	at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator.locate(ConfigServicePropertySourceLocator.java:93) ~[spring-cloud-config-client-1.3.2.RELEASE.jar!/:1.3.2.RELEASE]
	... 15 common frames omitted

Open Eureka Dashboard and found that only EUREKA and ZUUL-PROXY components shown in the "Instances currently registered with Eureka" table.

Expected
All pods should be running and after all the components are ready for use when the "Instances currently registered with Eureka" table shows these values: EUREKA, HYSTRIX-DASHBOARD, and ZIPKIN-TRACING.

Is there any workaround for this?

/cc @gyliu513