datakaveri / iudx-deployment

Installation and setup scripts for single and multi node (clustered) IUDX services.

iudx-deployment

This repository gives installation and setup scripts to deploy the IUDX platform. We provide and support two types of IUDX platform deployment :

1. Docker Swarm based deployment
2. K8s based deployment

The IUDX platform consists of various IUDX-built services and open-source components. The overview IUDX platform with components is shown in the below figure.

IUDX is a data exchange platform facilitating the seamless discovery, and exchange of authorized data. Following is a short explanation of how various components interact in the IUDX platform:

• Through the IUDX Catalogue server users discovers different datasets available on the platform.

• A user can register with one or more roles (consumer/provider, data ingester/delegate) in IUDX AAA and keycloak. The keycloak is used to manage the identities of users.

• The user can get set/request policies at the AAA server, and get a token. IUDX AAA platform manages the policies through credentials/Policy Datastore(Postgres).

• Through this token, the user can publish(input)/consume (output) data from any of the IUDX resource access services (resource server, rs-proxy, GIS server, Data ingestion server, File server)

• IUDX platform supports the following input data flows

  • A data ingester ( delegate ) can pull the data from the downstream source (ICCC) and push it to the databroker (Rabbitmq). Which then is consumed by Logstash, the latest ingestion pipeline, and is pushed to the Meta Data/Data Store (Elasticsearch) and Latest Datastore (Redis).
  • Also a data ingester can directly push data through HTTPS APIs exposed by Data Ingestion Server.

• IUDX platform supports the following output data flows

  • Get data through standardized Resource access service APIs - spatial, temporal, complex, file, gis, and async queries.
  • Get live streaming data through Rabbitmq using a resource server Subscription
  • Get data from non-IUDX resource server through resource-server proxy (rs-proxy). This is done through IUDX RS API query translation to non-IUDX RS-specific queries by a set of adapters that reside close to non-IUDX RS. The query and response are communicated to adapters and rs-proxy through databroker(Rabbitmq).

• IUDX platform is monitored through the micrometer, Prometheus for metrics and promtail, Loki for logs, and Grafana for Visualisation

• The alerting through SMTP server for emails or Telegram bot for telegram messages.

• All HTTPS API requests are processed through the API gateway.

• The Rabbitmq specific communication i.e. streaming of data through AMQPS and HTTPS management interface is through the streaming gateway

• Hazlecast with Zookeeper is used as the cluster manager for all Vert.x based API servers.

• Successful API calls are audited in tamper proof database - immudb and in postgres through an auditing server

To know more about IUDX, refer following resources:

1. What is IUDX? To get an overview of the IUDX platform and its main motivation
2. IUDX Architecture Overview
3. IUDX Developer Section

Features

• Service Mesh Architecture based Vert.x API servers.
• Each microservice is a well-defined module that can be containerized and discovered using service discovery.
• Docker Swarm deployment enables easy, cost-effective deployment of the IUDX platform suitable for prototyping and PoC.
• Kubernetes-based deployment of the IUDX platform gives a scalable, highly available system through the clustered deployment of each component. It's suitable for production-grade deployment.
• Both docker and K8s-based deployment is cloud agnostic* and can be deployed on any cloud or on-prem. It has been tested currently on AWS and Azure.

*Note: K8s deployment depends on certain cloud services - Load Balancer, Storage, Object Storage, K8s cluster autoscaling but since this is offered by major clouds. It can be integrated into these cloud providers.

Contributing

We follow Git Merge based workflow

1. Fork this repo
2. Create a new feature branch in your fork. Multiple features must have a hyphen-separated name, or refer to a milestone name as mentioned in Github -> Projects
3. Commit to your fork and raise a Pull Request upstream.
   Detailed instructions are present here.

iudx-deployment's Issues

1.4 Redis Deployment in Docker swarm

• Orchestrate redis server in docker swarm,
• Automate user creation.
• Use custom redis-rejson images based on bitnami image.
• Add some test cases to see end-to-end working

Need Installation step to Setup single node IUDX

Hi Team,

I am trying to setup single node IUDX (Catalogue, Resource and Auth ) server on my local.

I exlopered Repo https://github.com/datakaveri/iudx-deployment and found one stable release v2.5.0.
But Installation steps are missing there.

so can you guys please tell me installation steps or any other workaround so that i can setup IUDX on my local?

Thanks
Deepak Kumar

Logstash [v8.3.3] adding "event" field to payload.

Logstash pipeline is adding an event field to the payload containing the original payload from RabbitMQ. This is causing Elasticsearch to throw an error as it goes against the strict mapping set for the indices.

Tested using the RMQ test script here

Error from Logstash:

{
	"level": "WARN",
	"loggerName": "logstash.outputs.elasticsearch",
	"timeMillis": 1664274188010,
	"thread": "[resource-group]>worker0",
	"logEvent": {
		"message": "Could not index event to Elasticsearch.",
		"status": 400,
		"action": ["index", {
			"_index": "test-itms"
		}, {
			"id": "test-itms",
			"last_stop_arrival_time": "15:09:58",
			"trip_delay": 948,
			"actual_trip_start_time": "2020-11-03T14:22:30+05:30",
			"license_plate": "10",
			"last_stop_id": "2028",
			"event": {
				"original": "{\"trip_direction\": \"NT\", \"trip_id\": \"24374871\", \"route_id\": \"17AD\", \"trip_delay\": 948, \"last_stop_arrival_time\": \"15:09:58\", \"actual_trip_start_time\": \"2020-11-03T14:22:30+05:30\", \"vehicle_label\": \"A09\", \"observationDateTime\": \"2022-09-27T10:23:07+00:00\", \"speed\": 25.0, \"license_plate\": \"10\", \"last_stop_id\": \"2028\", \"location\": {\"coordinates\": [72.870511, 21.218943], \"type\": \"Point\"}, \"id\": \"test-itms\"}"
			},
			"trip_id": "24374871",
			"vehicle_label": "A09",
			"route_id": "17AD",
			"observationDateTime": "2022-09-27T10:23:07+00:00",
			"location": {
				"coordinates": [72.870511, 21.218943],
				"type": "Point"
			},
			"speed": 25.0,
			"trip_direction": "NT"
		}],
		"response": {
			"index": {
				"_index": "test-itms",
				"_id": "BMN4foMBfBEPN2SVxrea",
				"status": 400,
				"error": {
					"type": "strict_dynamic_mapping_exception",
					"reason": "mapping set to strict, dynamic introduction of [event] within [_doc] is not allowed"
				}
			}
		}
	}
}

Current solution:

This was temporarily fixed in commit- 8ef8f5a by manually removing the event field in the pipeline filter. Need to debug what's causing this and make a permanent fix.

1.3 Auth server deployment in Docker Swarm

• Orchestrate authorization server in docker swarm
• Add nginx proxy in front - centralised nginx model.
• Add some test cases to see end-to-end working

Improve Docker images

• 1. Explore practices among different industries in containersation and draw up set of best practices
• 2. Explore flaws in current docker files
• 3. improve and adhere to best practices drawn

Have a standard dockerfiles following best practices, inspiration being bitnami. Example

Improve capacity planning and scaling strategies

1. elasticsearch
2. postgres
3. redis

6.1 Catalogue helm chart creation

• Create helm charts for catalogue servers following best practices. Refer #issue45
• Add some test cases to see end-to-end working.
• Helm chart should bundle ingress, deployment, environment vars, HPA autoscaling, sealed secrets too!

Update IUDX 3.0.0 components in docker swarm deployment

Include orchestration of IUDX 3.0.0 components in docker:

• 1. keycloak behind proxy
• 2. auth server
• 3. latest ingestion pipeline
• 4. Redis
• 5. File server
• 6. immudb
• 7. update resource server and catalogue server appropriately
• 8. possibly use bitnami images for backend components similar to what used in K8s -> redis, postgres, keycloak
• 9. have a centralised nginx similar to ingress controller in k8s
• 10. video server - maybe done later part of 3.5

Upgradation of components

• Strategy of updates process for K8s, backend components
  • Track upstream component updates
  • test the updated version with iudx requirements
  • report and resolve new breaking changes

3. Docker registry and related tools

Purpose

Store and distribute public iudx docker images - cat, rs, auth, lip, file server application and custom backend docker images like from keycloak, elasticsearch, redis. May need some private repos in future.

Requirements

Explore docker registries and related tools - managed vs self managed with following requirements:

• Teams, RBAC fine-grain access for team members , eg: push access on a image repo for a team member .
• allow high (how much?) docker image pulls,
• high availability of registry
• private repos functionality
• lifecycle policy for images
• cost
• easier integration with CI and CD pipelines
• secure system
• Docker images security scanning
• Docker images signing
• Easier to maintain and manage

Resources

1. Comparison between registries : https://cloudonaut.io/amazon-ecr-vs-docker-hub-vs-github-container-registry/

1.7. Update resource server and Catalogue server in Docker swarm Deployment

• Update resource server and catalogue server appropriately to 3.0 version in docker swarm.
• Restructure nginx to be centralised nginx - similar to k8s ingress
• Add some test cases to see end-to-end working

POST requests on old URLs failing during redirection from old urls to new 5.0.0 urls

1. Different User Agents interpret this status code differently:

• cURL changes the HTTP method from POST to GET after the redirect;
  this causes the request body to be dropped. it later makes a POST to
  the redirected URL, but since the request body is dropped, we get a
  400 Bad Request due to a null body from the server
• Postman converts the POST to GET outright, and we get a 404 Not
  Found error since there is no GET /token API.

2. Another issue occurred at AAA server specifically. The 'Authorization' header is stripped of during the
   redirect by clients (curl, postman) for security purpose (ref: https://stackoverflow.com/a/28671822).
   This header is used only at AAA server and hence effects AAA server specifically.

have security headers at vertx servers instead of at nginx

why?

• non-uniformity of security headers that needs to be added at nginx for different api-servers. better implementation and lesser confusion on what needs to be added at nginx
• similar eg: keycloak adds all security headers at its jboss server.

Issues from Mattermost

[datakaveri/iudx-deployment] New comment by @abhi on #389 updated rs docker image tag to 5.0.0-alpha-7d8504a:

LGTM

Issue created from a Mattermost message by @github.

Add logging tags in each stack file

Add logging tags to each stack file to get container name and id as labels in logs

 logging:
      driver: "json-file"
      options:
        max-file: "5"
        max-size: "10m"
        tag: "{\"name\":\"{{.Name}}\",\"id\":\"{{.ID}}\"}"

1.9 Improve Documentation for docker deployment

Maintain uniform structure, git version documentation just like a code, mainly because docs will change with code changes and releases!

Structure the documentation , according to needs

• How to install guide
• docker. docker swarm starter
• Also includes overview configuration details for backend components
• Documentation of each components - config, settings, and how to deploy
• Design decisions, reasoning

HA setup of Rancher

K8s Deployment Azure

• Autoscaling in Azure
  • Cloud auto scaling group (ASG) support
  • K8s cluster autoscaler refer here
• Creation of cluster in Azure and auto joining of nodes to cluster when a new node is added by cluster autoscaler using rancher. Have a look how is done for aws. The auto joining of nodes is through cloud init script.
• Storage in Azure - use of csi driver instead of in-tree as for as possible
• Choose and integrate with loadbalancer in Azure
• Backup strategy
  • mainly storage similar to s3 for storing backups
  • integrate velero with azure backend object storage

Migrate vertx cluster manager from Hazelcast to Zookeeper

Reasons for migration

Testing requirements

Deployment Architecture Diagram - Enhance

• - Mention HTTPS for API access
• - Include Consent Validator
• - Include Central Catalogue, Keycloak
• - Showcase Cos, UAC model of deployment
• - Rename maybe as DX deployment

Redis keys cleaning

Problem
In resource server, there are some resource items whose unique attribute changes almost at every publication. It can be avoided in some but not in all. This result in amount of data holding in a single redis key to higher value as number of days goes on.
Possible Solution
Delete redis key and recreate it again (why recreate? because latest ingestion pipeline and resource server thinks the key always exists once the key is created on arrival of first packet). This can be done as cron job and with some threshold on no. of entries.
Drawback: Might lose data.

Improve testing, documentation

1.1 Keycloak deployment in Docker swarm

• Bringing up keycloak through use of keycloak bitnami image , version 14.0.0
• Use bitnami postgres image and connect the keycloak with the postgres. Can be tested , by seeing if the keycloak related tables are created.
• Migrate current auth-cred-db stack file to postgres stack file with image and other config related changes.
• Bring a nginx proxy in-front of keycloak and test if its working. Set up reverse proxy, ssl, rate limits , refer keycloak official docs, blog
  • Verifying the configuration https://www.keycloak.org/docs/latest/server_installation/#verifying-the-configuration
• Automate realm creation using imports. Create a realm , configure with general settings and export it and then import that in deployment and need to configure minimal things only like client secret, smtp/optional things.

8. Automation of letsencrypt certificate renewal

• Try out and test cert manager for letsencrypt certificate renewal in K8s cluster

Immudb script changes - 3.5.0

1. iudx-deployment/K8s-deployment/Charts/immudb/docker/immudb-config-generator.py

   Line 45 in 977d040

   client.databaseUse("iudxrs")

   , change it 'iudxrsorg'

2. iudx-deployment/K8s-deployment/Charts/immudb/docker/immudb-config-generator.py

   Line 46 in 977d040

   client.sqlExec("CREATE TABLE auditing(id VARCHAR NOT NULL, userid VARCHAR NOT NULL,api VARCHAR NOT NULL,resourceid VARCHAR NOT NULL,time INTEGER NOT NULL,PRIMARY KEY id);")

   , to be changed to CREATE TABLE IF NOT EXISTS rsauditingtable (id VARCHAR NOT NULL, api VARCHAR, userid VARCHAR,epochtime INTEGER,resourceid VARCHAR, isotime VARCHAR, providerid VARCHAR, PRIMARY KEY (id));

3. iudx-deployment/K8s-deployment/Charts/immudb/docker/immudb-config-generator.py

   Line 40 in 977d040

   client.sqlExec("CREATE TABLE auditingtable(id VARCHAR NOT NULL, userRole VARCHAR NOT NULL,userID VARCHAR NOT NULL,iid VARCHAR NOT NULL,api VARCHAR NOT NULL,method VARCHAR NOT NULL,time INTEGER NOT NULL,iudxID VARCHAR NOT NULL,PRIMARY KEY id);")

   to CREATE TABLE auditingtable1 (id VARCHAR NOT NULL, userRole VARCHAR NOT NULL, userId VARCHAR NOT NULL, iid VARCHAR NOT NULL, api VARCHAR NOT NULL, method VARCHAR NOT NULL, time INTEGER, iudxID VARCHAR NOT NULL,PRIMARY KEY id)

1.6 File server deployment in Docker swarm

• Orchestrate file server in docker swarm
• Add nginx proxy in front - centralised nginx similar to k8s ingress
• Add some test cases to see end-to-end working

Required exchanges, queues, bindings and vHost for RMQ

Currently, imported through definitions file for 3.0.0 release. It has some drawbacks - uncertainty while upgrading existing instances, users change.

Solution
Write a python script to create required exchanges, queues, bindings and policies

Unable to Find Rabbit MQ Docker Image

Hi Team,

I am trying to setup databrober instance by following https://github.com/datakaveri/iudx-deployment/tree/master/Docker-Swarm-deployment/single-node/databroker

As mentioned in databroker-stack.yml RMQ Image rabbitmq:3.10.5-management is not avaibale on docker repo.
and also tried with command docker build -f backup/Dockerfile -t ghcr.io/datakaveri/rabbitmq-backup:1.0 backup/ but it is stuck as per below logs -

root@horizon:~/iudx-deployment/Docker-Swarm-deployment/single-node/databroker# docker build -f backup/Dockerfile -t ghcr.io/datakaveri/rabbitmq-backup:1.0 backup/
Sending build context to Docker daemon 9.216kB
Step 1/5 : FROM python:3-alpine
---> ce4168535f30
Step 2/5 : WORKDIR /usr/share/app
---> Using cache
---> d0c1aa8e54ae
Step 3/5 : RUN apk update && apk add curl && apk add openssh-client && pip install --upgrade pip && pip install watchdog
---> Running in 8b4692cf0ce6
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/main/x86_64/APKINDEX.tar.gz

Can you guys please suggest me correct image path so that i can proceed further.

Thanks
Deepak Kumar

Grafana notifiers issue

Not able to create notifiers automatically using script files. Need to create notifier manually in grafana

Helm charts creation

create helm charts for :

• catalogue
• resource server
• file server
• auth server
• latest ingestion pipeline?
• rabbitmq ? or use from bitnami?
• Improve immudb
• umbrella chart and all other components as sub-charts
• GIS server
• DI server

[Known Issue] Rancher certificate expired-webhook, k3s-serving certificates after one year

• For Rancher versions that have rancher-webhook installed, these certificates will expire after one year. It will be necessary for to rotate your webhook certificate when this occurs.
• Ref: https://rancher.com/docs/rancher/v2.6/en/troubleshooting/expired-webhook-certificates/

5. Improve Monitoring Stack

• Add and improve existing dashboards according to give more insight on the happening in the K8s cluster, each of the components
• Add more alerts for detection of application/system failure

Sealed secrets

Extend sealed secrets to all modules

• ELK stack
• rs
• cat
• file
• Latest ingestion pipeline
• auth
• immudb
• Rabbitmq

1.5 Latest ingestion pipeline deployment in Docker Swarm

• Orchestrate latest ingestion pipeline vertx server in docker swarm,
• Add some test cases to see end-to-end working

Improve resilency of iudx systems in K8s

Motivation

What happens when one/more of the dependent systems say 'y' are down/unavailable of a particular system x and how would the system 'x' behave and improve the system 'x' resiliency accordingly. This is also needs to be coupled with lessons learnt/feedback from monitoring/incidents in k8s testing and production environment

This maybe a far fetched, but can look at the chaos mesh, use that principles for testing and may be slowly adopt the chaos mesh for testing the resiliency of systems in K8s

Clear-Site-Data header is removed, need to add with changes to accomodate UI

• currently Clear-Site-Data header is removed due to UI not functioning properly with the header
• need to add it with proper settings and testing with UI

Swarm: Migrate to bitnami ELK images for 8.x

Since we now use bitnami elk charts, its better to use bitnami elk docker images in swarm too.

Add guidelines/docs to use git based working

• Like how to contribute
• remotes, PRs, linking of issue.
• milestones, projects

1.8 Restructuring Docker swarm deployment

• Restructure docker swarm deployment folder structure,
• have a centralised nginx similar to ingress controller in K8s.
• Improvise Ansible automation
• Explore the need for any more automation using installation scripts.

1.2 Immudb Deployment in Docker Swarm

• Bring up immudb container through official immudb docker image - v1.0.5, https://docs.immudb.io/master/ , https://hub.docker.com/r/codenotary/immudb
• Make it pgsql compatible
• Explore config options of immudb
• Add immudb auditor, if needed
• Automate creation of users,dbs, schemas
• Add simple tests

Simplify ignoring of files through .gitignore

• Right now I feel there is lot of redundant .gitignore files.
• Refer - https://git-scm.com/docs/gitignore

Alter Rabbitmq Backup in docker swarm for bitnami rabbitmq docker image

1.10 Video server deployment in Docker Swarm

• Orchestrate the video server in docker swarm.
• Add some test cases to see end-to-end working

7. helm chart repository

Purpose

Store and distribute public helm charts for the K8s deployment of IUDX system.

Requirements

Need to explore the requirements of good helm repo, some might be:

• highly available
• secure system
• some scalability/handle load

Resources

1. use of github pages for hosting helm repos

K8s Azure

• Autoscaling in Azure
  • Cloud support ASG
  • K8s cluster autoscaler
• Creation of cluster in Azure using rancher
• Storage in Azure - use of csi driver instead of in-tree

2 Improve Docker images

i). Explore dockerfile best practices. Some sources are:

1. https://engineering.bitnami.com/articles/best-practices-writing-a-dockerfile.html
2. https://www.youtube.com/watch?v=plpgm12i7Hs
   And update the internal doc with the above, and why we need to do it

ii) Explore flaws in current docker files and improve and adhere to best practices drawn. Can be done first on calculator :-
https://github.com/abhi4578/redesigned-enigma or directly cat server.