snowdrop / istio-distributed-tracing-example

A booster demonstrating Istio Distributed Tracing functionality

Istio Distributed Tracing Mission

Table of Contents

• Purpose
• Prerequisites
• Environment preparation
• Build and deploy the application
  • With Dekorate:
• Use Cases
  • Access the application via the Istio ingress-gateway
  • View application traces
  • Undeploy the application
• Integration tests
  • Remove Jaeger UI security
  • Execute the Integration Tests

Purpose

Showcase Istio’s Distributed Tracing via a (minimally) instrumented set of Spring Boot applications

Prerequisites

1. JDK 11+ installed with JAVA_HOME configured appropriately

2. Openshift 4.12 cluster

3. Istio installed on the aforementioned cluster using the Red Hat OpenShift Service Mesh Operator.

4. Login to the cluster with the admin user

Environment preparation

Create a new project/namespace on the cluster. This is where your application will be deployed.

oc new-project <whatever valid project name you want>

Add the <whatever valid project name you want> project to the Istio member roll. More information in this link.

Build and deploy the application

With Dekorate:

Execute the following command to build the project and deploy it to OpenShift:

mvn clean verify -Popenshift -Ddekorate.deploy=true

This configuration is used to define service names and deployments that control how pods are labeled/versioned on the OpenShift cluster. Labels and versions are key concepts for creating load-balanced or multi-versioned pods in a service.

Use Cases

Access the application via the Istio ingress-gateway

1. Create a RouteRule to forward traffic from istio-ingress to the demo application

   oc create -f rules/greeting-gateway.yml

2. Access the application

   Run the following command to determine the appropriate URL to access our demo. Make sure you access the url with the HTTP scheme. HTTPS is NOT enabled by default:

   echo http://$(oc get route istio-ingressgateway -o jsonpath='{.spec.host}{"\n"}' -n istio-system)/greeting/

   The result of the above command is the istio-system istio-ingress URL, appended with the RouteRule path. Open this URL in your a web browser.

3. Follow the instructions in the application UI

View application traces

1. Access the Jaeger tracing dashboard

   The traces from the invocation of the two endpoints should look like the following:

   

   Note that it could take a few seconds for all the spans to be collected and presented in a trace that matches the picture above

Undeploy the application

Remove the namespace

This will delete the project from the OpenShift cluster

oc delete project <your project name>

Integration tests

To run integration tests, first follow the steps on the Environment preparation section to create and prepare a new namespace.

Remove Jaeger UI security

Caution

For the integration tests to work it is required that the Jaeger UI isn’t secured by authentication, as it is in the default configuration.

Warning

Disabling security isn’t the suggested configuration but it is required for the integration tests to work.

By default, when installing the Red Hat OpenShift distributed tracing platform operator, it will install a Jaeger instance with the security enabled:

Secured Jaeger UI configuration

  ingress:
    enabled: true
    openshift:
      htpasswdFile: /etc/proxy/htpasswd/auth
      sar: '{"namespace": "istio-system", "resource": "pods", "verb": "get"}'
    options: {}
    resources: {}
    security: oauth-proxy

Disabling security is accomplished by setting the spec.ingress.security on the Jaeger operator to none.

This can be done either by using the OpenShift console (Installed operators > Red Hat OpenShift distributed tracing platform > Jaguer > Edit Jaeger) or using the CLI.

Note	More information on the Jaeger configuration at the Jaeger configuration reference page.

To disable security execute the following CLI commands.

Set the istio-system as the default project

oc project istio-system

Patch jaeger to disable console security

oc patch jaeger jaeger --patch '{"spec":{"ingress":{"security": "none" }}}' --type=merge

The result should be:

jaeger.jaegertracing.io/jaeger patched

Which will result in the following configuration.

Unsecured Jaeger UI configuration

  ingress:
    enabled: true
    options: {}
    resources: {}
    security: none

Execute the Integration Tests

After everything is set up, move back to the project where this example will be installed.

oc project <whatever valid project name you want>

Then run the following commands.

Commands to execute the integration tests

mvn clean verify -pl spring-boot-istio-distributed-tracing-cute-name-service -Popenshift -Ddekorate.deploy=true
mvn clean verify -pl spring-boot-istio-distributed-tracing-greeting-service -Popenshift -Ddekorate.deploy=true
oc create -f rules/greeting-gateway.yml
mvn clean verify -pl tests -Popenshift-it

istio-distributed-tracing-example's Issues

Why do we continue to use our deprecated GAV

Why do we continue to use our deprecated GAV instead of the opentracing spring web starter ?

This GAV - https://goo.gl/sGpwj1

<groupId>me.snowdrop</groupId>
<artifactId>opentracing-tracer-jaeger-spring-web-starter</artifactId>
<version>${opentracing-tracer.version}</version>

should be

https://github.com/opentracing-contrib/java-spring-web/blob/release-0.3.2/opentracing-spring-web/pom.xml#L5-L10

<groupId>io.opentracing.contrib</groupId>
<artifactId>opentracing-spring-web</artifactId>
<version>0.3.2-SNAPSHOT</version>

REMARK : The opentracing spring web starter should be part of our next Spring Boot BOM !

REAMDE link pointing to non exists file

istio-distributed-tracing-example/spring-boot-istio-distributed-tracing-greeting-service/src/main/resources/templates/index.html

Line 45 in 3832a44

Please reference the mission documentation or <a href="https://github.com/snowdrop/istio-distributed-tracing-example/blob/master/README.md">README</a> for next steps and instructions.

Add integration tests to cover this example

At the moment, this example is lacking any kind of test. We should cover the functionality and ensure the correct configuration with Istio.
The tests should be executed in pull requests.

Enable istio in your namespace first

You need to enable istio in your namespace after you create namespace (please add to the README after oc new-project ... 2 places):
create this istio-smm.yaml file in example repository:
apiVersion: maistra.io/v1
kind: ServiceMeshMember
metadata:
name: default
spec:
controlPlaneRef:
namespace: istio-system
name: basic

add to the README:
oc apply -f istio-smm.yaml

istio-distributed-tracing-example/README.adoc

Line 19 in 9853fa7

OpenshiftIT.tracingTest:70 HTTP status 403

[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 16.559 s <<< FAILURE! - in me.snowdrop.istio.dt.OpenshiftIT
[ERROR] tracingTest Time elapsed: 16.189 s <<< FAILURE!
org.opentest4j.AssertionFailedError:

expected: 200
but was: 403
at me.snowdrop.istio.dt.OpenshiftIT.tracingTest(OpenshiftIT.java:70)

[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] OpenshiftIT.tracingTest:70
expected: 200
but was: 403

istio-distributed-tracing-example/tests/src/test/java/me/snowdrop/istio/dt/OpenshiftIT.java

Line 70 in a0b5ddb

assertThat(response.statusCode()).isEqualTo(200);

Migrate Opentracing to Open Telemetry

Plus, update the documentation accordingly.

Update Instructions to use OpenShift 4.x and installation of Istio

Jaeger query

Openshift integration tests are failing with error:

[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.05 s <<< FAILURE! - in me.snowdrop.istio.dt.OpenshiftIT
[ERROR] tracingTest Time elapsed: 0.762 s <<< ERROR!
java.lang.NullPointerException
at me.snowdrop.istio.dt.OpenshiftIT.getBaseUrlByRouteName(OpenshiftIT.java:135)
at me.snowdrop.istio.dt.OpenshiftIT.setup(OpenshiftIT.java:40)

because of bad route name:

istio-distributed-tracing-example/tests/src/test/java/me/snowdrop/istio/dt/OpenshiftIT.java

Line 29 in 8573a9f

private static final String JAEGER_QUERY_NAME = "jaeger-query";

The correct istio jaeger route name is jaeger.