digi-serve / ab_runtime

The Server Side runtime for our AppBuilder

The server side runtime for our AppBuilder project.

Installation

Use ab-cli tool.

Starting

Run ./UP.sh to bring up the Docker Stack. Options:

• -d : dev run develop mode (uses docker-compose.dev.yml)
• -t : test run in test mode (allows test reset route)
• -q : quiet run without starting logs

The default account credentials are:

• email: [email protected]
• password: admin

Stop AppBuilder

$ ./Down.sh

View console

$ node logs
// or
$ ./logs.sh [option]

Updating

Rungit pull to update the version file (version.json).
Then run node dockerImageUpdate to update the running services.

Data

Directories used for persistent storage

• ./mysql/init

  The SQL files contained here will be used to populate the database for the first time.

Much of the config and data are actually saved onto Docker volumes, and are not directly binded to directories anymore. The data backup procedure is left as an exercise for the reader.

Manual Installation

The ab-cli will help set the config and write .env, docker-compose.yml, and docker-compose.override.yml. Examples can be found in /examples/, for a manual install copy these files to the root directory and adjust them as needed.

Config

The most important setting to change is MYSQL_PASSWORD. This will be your DB root password. Choose something secure.

Important note: When the MariaDB container starts up for the first time, it will set the database root password to the value you have specified. After that first time, changing this value will not affect the already established password.

Make sure to add your own settings before running the setup steps below.

Preparation

Follow the Config section on configuring your DB credentials and other settings.

The following steps need to be done with the same docker stack name you plan to run AppBuilder with (referred to here as mystack). This should match the STACKNAME setting in your .env file.

1. Turn on Docker Swarm if needed

   $ docker swarm init

2. Copy dbinit-compose.yml from /examples to the project root.

   $ cp ./examples/dbinit-compose.yml ./dbinit-compose.yml

3. Run dbinit-compose.yml

   $ docker stack deploy -c dbinit-compose.yml mystack

   • Open the logs for mystack_db

     $ docker service logs -f mystack_db

   • Bring down the stack when you see mysqld: ready for connections

     $ docker stack rm mystack

4. Remove dbinit-compose.yml

   $ rm ./dbinit-compose.yml

5. Run db migrations

   $ ./migrate.sh

Podman

The runtime can be alternatively used with podman. Most of the scripts already support podman. Make sure the environment variable PLATFORM is set to podman.

ab_runtime's Issues

Add/Edit Roles and Scopes

Terraform/cloud-init config for our standard server set up

Auto Version all services

Requirements

1. Add For:

• relay digi-serve/ab_service_relay#20
• notification_email digi-serve/ab_service_notification_email#22
• log_manager digi-serve/ab_service_log_manager#32
• db digi-serve/ab_service_db#2
• api_sails digi-serve/ab_service_api_sails#103
• custom_reports digi-serve/ab_service_custom_reports#63
• appbuilder digi-serve/ab_service_appbuilder#59
• web digi-serve/ab_service_web#12, digi-serve/ab_service_web#13
• definition_manager digi-serve/ab_service_definition_manager#53
• file_processor digi-serve/ab_service_file_processor#38
• process_manager digi-serve/ab_service_process_manager#67
• user_manager digi-serve/ab_service_user_manager#47
• platform_service (& propagate to dependent service) digi-serve/appbuilder_platform_service#81
• platform_web (&propagate to web) digi-serve/ab_platform_web#374
• core (& propagate to platform_service, platform_web) digi-serve/appbuilder_class_core#186
• ABDesigner digi-serve/plugin_ABDesigner#212
• Runtime label check and release notes

Reference

• branch from master
• merge in develop
• add / edit workflows
• update dependabot.yml
• add labels (major, minor, patch, no_release) https://github.com/digi-serve/ab_runtime/labels
• pr: label major (so it bumps to 1.0.0)

Simple User Admin Page

“As a Tenant's Admin, I want a simple user admin page, so that I can easily add new users.”

Requirements

1. I can see a list of users.
2. I can edit a users roles (not scope)
3. I can add a new user.
   • New users get applied 'default roles'
   • New users get sent an email

Tasks

• Write Documentation

Reference

Used dev digiserve site to make change to the admin app
Afterward definitions should be updated here?

Roles will need a new checkbox field Default.
A process can assign new users the Default roles.
Send email that links to create password (similair to digi-serve/ab_platform_web#47)

Set up Monitoring

“We want to know the memory / cpu usage when a service fails"

Requirements

1. Add promotheus (and grafana?)

Tasks

• Write Documentation

Reference

Research: #364
https://docs.google.com/document/d/1utvJsxi-smHk1HlYw6kBeLUYqgw56vBTgodip68e-64/edit?usp=drive_link

Promotheus can be set up to monitor docker swarm https://prometheus.io/docs/guides/dockerswarm/
But will it work with podman?

• #387
• #386
• #385

Deploy Grafana and Grafana Mimir (terraform)

Service Versioning

Requirements

1. Runtime contains a versioning file, that holds the most recent version of each service
2. Script updates running service based on the version file.
3. Service has a Workflow that updates the version file.
   • e2e tests must pass

Tasks

• Version file in runtime (updated with git pull)
• Script to update .env and running services
• Action to auto-version
• Write Documentation

Add a podman script to update images without downtime

“As a [user], I [want to], [so that].”

Requirements

1. Updates podman containers to new images for the current stack.
2. Uses versions from the version.json file.

Tasks

• Write Documentation

Reference

Like dockerImageUpdate script, but for podman.

Add cadvisor and promethus to default stack

Research: Service Stats on Crashes

“When the site crashes, I want to see report of the current/avg cpu and memory usage of the various services

Requirements

1. Research and suggest 1-2 possible approaches

Tasks

• Write Documentation

Reference

Service Update Workflow handle concurrent calls

Is this related to a problem?
The update-service-version workflow cannot handle receiving many calls at the same time. If we trigger changes in core, it will update multiple services within a few minutes. Currently we create a branch for each update. It fails due to merge conflicts or creating duplicate tag/release versions.

Desired Solution

1. Use a common branch to stage changes.
2. Update that branch
3. Test
4. If Pass Release

Steps 3/4 should use a concurrency group and cancel-in-progress to ensure only one process is trying to create a release at a time.