corda / corda-kubernetes-deployment

Corda Kubernetes Deployment

CORDA KUBERNETES DEPLOYMENT

This repository (https://github.com/corda/corda-kubernetes-deployment) contains the means with which you can stand up a Corda Enterprise Node.

This is meant to be a customizable version of the Node deployment that you can take as-is if it fits your needs or then customize it to your liking.

DISCLAIMER:

This Kubernetes deployment for a Corda Enterprise Node is considered a reference implementation and should not be used in a production environment until sufficient testing has been done.

Licensed under Apache License, version 2.0.

---

IMPORTANT

Kubernetes is a complex system and setting up a successful deployment for the first time can be challenging as well.

Please make sure you step through the SETUP CHECKLIST section carefully the first time you deploy, to avoid problems down the road.

It is strongly recommended you review the documentation before setting this up for the first time to familiarize yourself with the topic at hand.

---

SETUP CHECKLIST

Since there are a number of prerequisites that need to be met and then a certain order of running everything, a checklist has been collated that you may find useful.

Please see CHECKLIST.md for more information.

Note! It is strongly recommended you follow the CHECKLIST, to not skip an important step, especially the first time you set up this deployment,

---

PREREQUISITES

• A cloud environment with Kubernetes Cluster Services that has access to a Docker Container Registry, see CLOUD_SETUP.md
• Building the images requires local Docker installation
• kubectl is used to manage Kubernetes cluster
• Helm version 2.x
• Access to Corda Enterprise binary files or access to R3 Artifactory for Enterprise (licensed users)

---

BINARIES

This deployment is targeting an Enterprise deployment, which should include a Corda Node, but also the Corda Firewall, which is an Enterprise only feature.

In order to execute the following scripts correctly, you will have to have access to the Corda Enterprise binaries.

The files should be downloaded first and placed in the following folder: docker-images/bin

You can use the helper script download_binaries.sh to download binaries for you, as long as you have the necessary login details available for the R3 Artifactory.

If you have R3 Artifactory access, the download will be automatic as part of the one-time-setup.sh script, which is the recommended way of performing the first time setup.

Please see docker-images/README.md for more information.

---

CONFIGURATION

You must completely fill out the helm/values.yaml file according to your configuration. For more details on how it should be filled out, follow the CHECKLIST.md document.

---

SHORT USAGE GUIDE (see SETUP CHECKLIST for a full guide)

This is a brief view of the steps you will take, for the full set of steps, please review CHECKLIST.md.

1. Customize the Helm values.yaml file according to your deployment (this step is used by initial-registration and Helm compile, very important to fill in correctly and completely)
2. Execute one-time-setup.sh which will do the following (you can also step through the steps on your own, just follow what the one-time-setup.sh would have done):
   1. Build the docker images and push them to the Container Registry
   2. Generate the Corda Firewall PKI certificates
   3. Execute initial registration step (which should copy certificates to the correct locations under helm/files)
3. Build Helm templates and install them onto the Kubernetes Cluster (by way of executing either deploy.sh or helm/helm_compile.sh)
4. Ensure that the deployment has been successful (log in to the pods and check that they are working correctly, please see below link for information on how to do that)

---

DOCUMENTATION

For more details and instructions it is strongly recommended to visit the documentation:

Documentation

---

Contributing

The Corda Kubernetes Deployment is an open-source project and contributions are welcome as seen here: Contributing

The contributors can be found here: Contributors

---

Feedback

Any suggestions / issues are welcome in the issues section: https://github.com/corda/corda-kubernetes-deployment/issues/new

Fin.

corda-kubernetes-deployment's Issues

Repeated readlink errors

When I run ./one-time-setup.sh I get the following errors:

• /Users/simon/corda-kubernetes-deployment/docker-images/build_docker_images.sh
• DIR=.
• GetPathToCurrentlyExecutingScript
  ++ readlink -f /Users/simon/corda-kubernetes-deployment/docker-images/build_docker_images.sh
  readlink: illegal option -- f
  usage: readlink [-n] [file ...]

+++ readlink -f /Users/simon/corda-kubernetes-deployment/docker-images/docker_config.sh
readlink: illegal option -- f
usage: readlink [-n] [file ...]

Database Migration

such error occurs during initial registration. Although the error logs says "database.initialiseSchema=true" would fix the issue, it does not. might need to consider use db migration tool.

Database connection url is : jdbc:sqlserver://ps-db-k8s-test.database.windows.net:1433;database=corda;user=corda@ps-db-k8s-test;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;;sendStringParametersAsUnicode=false [ERROR] 20:33:45+0800 [main] internal.NodeStartupLogging.invoke - Exception during node registration: Incompatible database schema version detected, please run the node with configuration option database.initialiseSchema=true. Reason: There are 92 outstanding database changes that need to be run. Please use the advanced migration tool. See: https://docs.corda.r3.com/database-management.html [errorCode=1nkwe50, moreInformationAt=https://errors.corda.net/ENT/4.2/1nkwe50] [ERROR] 20:33:45+0800 [main] internal.NodeStartupLogging.invoke - Exception during node startup: net.corda.core.utilities.Try$Failure cannot be cast to net.corda.core.utilities.Try$Success [errorCode=rmnkpl, moreInformationAt=https://errors.corda.net/ENT/4.2/rmnkpl]

Network Parameters Not Downloading for one-time-setup.sh

@henrikr3 I'm noticing a consistent error downloading network parameters in one-time-setup.sh

This is the output:

Corda Node network-parameters step
Next we will launch the Corda Node just in order to download network-parameters file. Please wait...
Checking for network-parameters file...
Found network-parameters file!
waitTillNetworkParametersIsAvailable finished.
Copying network-parameters to helm/files/network (for Corda Firewall use).
cp: illegal option -- u
usage: cp [-R [-H | -L | -P]] [-fi | -n] [-apvXc] source_file target_file
cp [-R [-H | -L | -P]] [-fi | -n] [-apvXc] source_file ... target_directory

---

• All rights reserved. *
• This software is proprietary to and embodies the confidential technology of R3 LLC ("R3"). *
• Possession, use, duplication or dissemination of the software is authorized only pursuant to a valid written license from R3. *
• IF YOU DO NOT HAVE A VALID WRITTEN LICENSE WITH R3, DO NOT USE THIS SOFTWARE. *

---

---

/ / / / _ | | \ | | | | _ | _ | _ | / || |
/ / __ / / __ / __ `/ | | | | | | | | | | |) | |) | |) || |_ | |
/ / // / / / // / // / | || |\ | | | | || _ <| /| _ < | | ) | |
_/ // _,/_,/ |_|| _| || |_____|| __| || __|____/|___|

--- Corda Enterprise Edition 4.4.1 (c0ce776) ------------------------------------------------

💡 Use the "flow watch" shell command to see what flows your node is running

Logs can be found in : /Users/simon/corda-kubernetes-deployment/helm/initial_registration/output/corda/templates/workspace/logs
⚠️ ATTENTION: If you make use of confidential identities, there is now a more secure way of storing the associated keys, but you have to explicitly enable it with the appropriate configuration. Review the documentation to see how this can be enabled via the freshIdentitiesConfiguration entry. Alternatively, you can disable this warning by setting disableFreshIdentitiesWarning to true in the node's configuration.
I> Using policy access restrictor classpath:/jolokia-access.xml
I> Cannot join multicast group on NIF utun1: Can't assign requested address
I> Cannot join multicast group on NIF utun0: Can't assign requested address
I> Cannot join multicast group on NIF llw0: Can't assign requested address
I> Cannot join multicast group on NIF awdl0: Can't assign requested address
I> Cannot join multicast group on NIF utun1: Can't assign requested address
I> Cannot join multicast group on NIF utun0: Can't assign requested address
I> Cannot join multicast group on NIF llw0: Can't assign requested address
I> Cannot join multicast group on NIF awdl0: Can't assign requested address
network-parameters download has completed, time to ensure Corda Node process terminates as well
Killed Corda Node process.
network-parameters download has completed, time to ensure Corda Node process terminates as well
Unfortunately the network-parameters file was not downloaded/copied to the correct folder, MANUAL intervention is required.

Quick fix for "readlink -f illegal option" on Mac

For anyone who encounters this issue on Mac
brew install coreutils
export PATH="/usr/local/opt/coreutils/libexec/gnubin:$PATH"
alias readlink=greadlink

Provide CorDapps to containers deployment options

Currently CorDapps are copied to the Corda Node container once it is running to the persistent storage assigned to that pod. The CorDapps should ideally be segregated.

a) Init containers
We could deploy the CorDapps to the Corda Node pod by use of init containers to install the CorDapps before the pod starts up:
https://kubernetes.io/docs/concepts/workloads/pods/init-containers/
This would essentially work as such:

• Corda Node is deployed to Kubernetes cluster
• Corda Node init container is automatically launched
• Corda Node init container starts up and downloads CorDapps assigned to be loaded to the Corda Node from a dedicated distribution site on for example Azure Storage or Amazon S3.
• Once init container has completed successfully (verifiably installed the CorDapps), the Corda Node will start up with the CorDapps already available

b) Other options could be considered as well:

• CorDapp Distribution Service (CDS) - https://github.com/corda/corda-solutions/tree/master/bn-apps/cordapp-updates-distribution
• other possibilites...

A few options could be defined in Helm charts as options that the user can select from for deploying CorDapps.

Add link to remove folder when starting new node.

Add comment to remove /Users/simon/corda-kubernetes-deployment/helm/initial_registration/output/corda/templates/workspace when starting new node.

Ubuntu Deployment of Scripts in Aure

Running initial registration, there is a source command in /corda-kubernetes-deployment/helm/initial_registration/output/corda/templates/initial_registration.sh that is produced by initial_registration.sh.yml references the docker_config.sh variables. This fails on execution so variables were added directly to the initial_registration.sh.yml file.

Lack of JDBC drivers handling

there is no handling regarding JDBC driver. i.e. there is no "drivers" folder in workspace and no JDBC driver copied there

Feedback

Thanks for preparing this, Henrik. I haven't yet had a chance to try it out, and apols for providing comments a week late, but here are early comments from just reading the doc:

• I was looking for "what problem is this solving" information. Maybe there is a predecessor document that provides all that context. Is there? If not, then this doc needs to explain what value does Kubernetes bring to a Corda Deployment, and what specific use cases are applicable (or not applicable).
• Is it the intent to explain the usage for nn number of cloud environments or is it sufficient to use Azure as a reference environment, an example, to explain how to use Kubernetes with Corda?
• Iike what I'm seeing in the configuration options, e.g., "Enable/disable Corda Firewall use" and look forward to giving it a try.

Output from Terraform

@henrikr3 It would be useful to pull this information into values.yml for your deployment?

Outputs:

acr_admin_password =
acr_admin_username =
acr_host =
aks_host =
aks_password =
aks_username =
storage_account_primary_access_key =

Ubuntu Deployment of Scripts in Azure

Ubuntu 18.04 x86_64 GNU/Linux

1. Made the following changes to build_docker_images.sh and push_docker_images.sh

Added the variables from docker_config.sh directly to build_docker_images.sh and push_docker_images.sh because these lines error out i.e. the variables are not sourced:

#source $DIR/docker_config.sh
#source ./docker_config.sh

Scripts completed successfully following changes.

2. On Ubuntu generate_pki.sh I copied the proper key tool binary to the required folder and it generated this error:

/home/cordaadmin/corda-kubernetes-deployment/corda-pki-generator/pki-firewall/bin/keytool: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory

Added the following line to generate_pki.sh to get past this error.

#KEYTOOL_EXE=$DIR/bin/keytool
KEYTOOL_EXE=/usr/bin/keytool

Scripts completed successfully following changes.

@irvnet @henrikr3

Docker Login Prompt logging into ACE

In values.yaml I specified the following which are correct (I double checked)

serverAddress:
username:
password:

When I run one-time-setup.sh for a new node it has prompted me on 2 occasions now for a Username/Password to connect to the ACR. Please investigate the script and see where I may be going wrong.

no revocation config for testnet from v4.4

This parameter should be added to float.conf and bridge.conf in 4.4 in order for notarisation to complete
revocationConfig: { mode: "OFF"}

cp: illegal option -- u still an issue in one-time-setup.sh

Found network-parameters file!
waitTillNetworkParametersIsAvailable finished.
Copying network-parameters to helm/files/network (for Corda Firewall use).
cp: illegal option -- u
usage: cp [-R [-H | -L | -P]] [-fi | -n] [-apvXc] source_file target_file
cp [-R [-H | -L | -P]] [-fi | -n] [-apvXc] source_file ... target_directory

[BUG] - Missing info around where to get the network root truststore

The checklist, under network-setup says to "Download network root truststore to ./helm/files/network" but it doesn't say where to download it from or how to generate it. Can you please provide some info around where to download this from or how to generate it?

Automation option for updating values.yml.

Looking at it this deployment from the point of view of the end user say being experienced in DevOps so Docker and k8s but still learning Corda.

1. The CENM k8s deployment has values that need to go into values.yml
2. The Terraform deployment has values that need to go into values.yml

This means if the values.yml file is edited manually, the DevOps person really needs to know Corda right now.

I'd like to see a PPT document that comes with the deployment that explains CENM k8s, Terraform tech footprint and the variables from both that need to go in values.yml. Also I'd like the user to have an option of automated deployment i.e. script that updates values.yml.

In addition the PPT should have a section that shows end user how to enter the k8s shell and manually run Corda Health Survey to ping Notary.

MacOS Issues during Deployment

Corda Node, Bridge, Float in k8s deployment now up and running and connected to CENM k8s deployment AND have done end to end transaction. Issues raised:

1. Document that Public IP's SKU's need to configured as Standard and not Basic when setting up the Public IP Addresses or else this error occurs:

Normal EnsuringLoadBalancer 5s (x5 over 82s) service-controller Ensuring load balancer
Warning CreateOrUpdateLoadBalancer 4s (x5 over 81s) azure-cloud-provider Code="PublicIPAndLBSkuDoNotMatch" Message="Standard sku load balancer /subscriptions/af4f0732-dd8c-4330-a357-b9593255c3f0/resourceGroups/mc_sjw-k8s-bridge_sjwaksclusterbridge_eastus2/providers/Microsoft.Network/loadBalancers/kubernetes cannot reference Basic sku publicIP /subscriptions/af4f0732-dd8c-4330-a357-b9593255c3f0/resourceGroups/MC_sjw-k8s-bridge_sjwAKSClusterBridge_eastus2/providers/Microsoft.Network/publicIPAddresses/floatLBIP." Details=[]
Warning SyncLoadBalancerFailed 4s (x5 over 81s) service-controller Error syncing load balancer: failed to ensure load balancer: timed out waiting for the condition

2. Fix issue with Artifactory downloads not working, right now Corda Software Binaries need to be manually copied from Artifactory to : corda-kubernetes-deployment/docker-images/bin

3. Don't remove the network-root-truststore.jks file when user runs reset_environment.sh, this file should remain there to make users life easier doing a reset.

4. Please advised on any fixes that were made to the shell scripts to address the "readlink" issue which caused a lot of scripts to break from Windows to Mac.

Hardware Security Module (HSM) support

HSMs (Hardware Security Module) allow for storing private key material in secure manner. In order to generate the key material we have to perform the Corda Node initial registration with the HSM option enabled. This is done by use of HA utilities:
[https://docs.corda.r3.com/ha-utilities.html](HA utilities)

This item is for modifying the initial registration to call HA utilities to allow for HSMs to be used.