nordic-institute / x-road-security-server-toolkit

X-Road Security Server Toolkit is a tool for automating Security Server configuration and maintenance tasks.

X-Road Security Server Toolkit

About the repository

This repository contains information about the X-Road Security Server Toolkit, source code, its development, installation and documentation.

X-Road Security Server Toolkit source code

Source code of X-Road Security Server Toolkit is open for all and it is licenced under MIT licence.

Introduction to X-Road Security Server Toolkit

X-Road Security Server Toolkit User Guide

Installing the latest development version from GitHub

Prerequisites to Installation

• Ubuntu 18.04 LTS or 20.04 LTS
• Python version 3.6+
• apt-get update needs to be run before installing
• PIP 21.0+
  • apt install -y python3-pip
  • python3 -m pip install --upgrade pip
  • pip3 install cement
• Installed X-Road security server packages on target machine(s)

Installation is performed with pip (use pip or pip3, whichever is used)

$ git clone https://github.com/nordic-institute/X-Road-Security-Server-toolkit.git

$ cd X-Road-Security-Server-toolkit

$ pip3 install -r requirements.txt

$ python3 setup.py install

Development

This project includes a number of helpers in the Makefile to streamline common development tasks.

Testing

In order to run unit tests with code coverage the following target in the Makefile should be run:

$ make test

In order to run unit tests with integration tests the following target in the Makefile should be run:

$ make test-all

End to end tests are run using the following script:

$ run_end_to_end_tests.sh

More details about the required input parameters are provided in the script file

Environment Setup

The following demonstrates setting up and working with a development environment:

### create a virtualenv for development

$ pip3 install virtualenv

$ make virtualenv

$ source env/bin/activate

Project versioning

Pre-release:

$ 0.x.x-alpha.0

First release:

$ 1.0.0-final.0

Post first release:

$ 1.x.x-beta.0

Updating project version

• Update patch/minor/major/release/build (major.minor.patch-release.build)

$ bump2version patch/minor/major/release/build (e.g. bump2version minor)

In case of minor updates, use bump2version patch In case of major updates, use bump2version minor When releasing, use bump2version major

In case of releasing, also bump2version release should be performed to update the release part of the version number, which can contain values:

• beta
• final

Releasing

Use the included helper function via the Makefile:

$ make dist

x-road-security-server-toolkit's Issues

Dockerfile issue

When I try to build a docker image as specified in the readme, I get this:

Building wheel for jq (PEP 517): finished with status 'error'

Multi-server configuration issues

The sequential multi-security configurations are basically supported for all single commands and autoconfiguration, as there can be any number of security servers in configuration file:

security_server:
- api_key: aaaa7381-80c1-4910-a259-aaaaaaaaaaaa
  api_key_url: https://localhost:4000/api/v1/api-keys
  name: ss1
  software_token_id: 0
  url: https://ss1:4000/api/v1
- api_key: bbbb7381-80c1-4910-a259-cccccccccccc
  api_key_url: https://localhost:4000/api/v1/api-keys
  name: ss2
  software_token_id: 0
  url: https://ss2:4000/api/v1

Problems

Only the last server defined is accessible (recently broken)

Despite correct configuration, xrdsst status in 1.0 fails to show information about other servers that the LAST defined in configuration:

╒══════════════════╤══════════════════════╤═══════════════════════╤════════════════════╤══════════╤═════════════╤══════════╤══════════╤═════════╕
│ GLOBAL           │ SERVER               │ ROLES                 │ INIT               │ TSAS     │ TOKEN       │ KEYS     │ CSRS     │ CERTS   │
╞══════════════════╪══════════════════════╪═══════════════════════╪════════════════════╪══════════╪═════════════╪══════════╪══════════╪═════════╡
│                  │ ss5                  │ NO ACCESS             │                    │          │             │          │          │         │
├──────────────────┼──────────────────────┼───────────────────────┼────────────────────┼──────────┼─────────────┼──────────┼──────────┼─────────┤
│ OK (SUCCESS)     │ ss1                  │ System Administrator  │ ANCHOR INITIALIZED │ Test TSA │ ID 0        │ SIGN (4) │ SIGN (3) │ SIGN*   │
│ LAST 102530 0412 │ VER 6.25.0           │ Service Administrator │ CODE INITIALIZED   │          │ softToken-0 │ AUTH (9) │ AUTH (8) │ AUTH*   │
│ NEXT 102630 0412 │ DEV:GOV:9876:UNS-SS1 │ Registration Officer  │ OWNER INITIALIZED  │          │ STATUS OK   │ 15 KEYS  │ 11 CSRS  │         │
│                  │                      │ Security Officer      │ TOKEN INITIALIZED  │          │ LOGIN NO    │          │          │         │
╘══════════════════╧══════════════════════╧═══════════════════════╧════════════════════╧══════════╧═════════════╧══════════╧══════════╧═════════╛

As discussed with @bertofl90 on Friday, access to other servers except the /last/ defined in configuration was borken in PR #44, in commit eb80387 that changed the logic so that servers to which access is not acquired do not cause exception to be thrown (which would break servers' configuration loop). The controllers' code snippets doing that are:

# acquire list of pairs (security server, api_config for the security server)
ss_api_conf_tuple = list(zip(config["security_server"], map(lambda ss: self.create_api_config(ss, config), config["security_server"])))

# proceed only with security servers with API config present
for security_server, ss_api_config in [t for t in ss_api_conf_tuple if t[1]]:
    # do the operation on security server, with given ss_api_config

# log the other, keyless servers that were not configured
BaseController.log_keyless_servers(ss_api_conf_tuple)

Breakage root comes from Swagger autogenerated Configuration(six.with_metaclass(TypeWithDefault, object)) that contains the API config for security server, which, though not a full singleton, derives from a superclass that performs shallow copy on non-primitive Configuration field objects and the dictionary object containing the API key. Which in effect is then will be the same for all created API configurations at the end of list construction, breaking the in multi-configuration logic.

Patching generated Configuration is not preferable, so one fix is to patchBaseController#create_api_config to create separate instance of only used non-primitive api_key field as follows:

def create_api_config(self, security_server, config=None):
  api_key = self.get_api_key(config, security_server)
  if not api_key:
    return None

  api_config = Configuration()
  api_config.api_key['Authorization'] = BaseController.authorization_header(api_key)
  api_config.host = security_server["url"]
  api_config.verify_ssl = False

+ # Prevent overwrite by Configuration superclass' TypeWithDefault shallow copy.
+ api_config.api_key = copy.deepcopy(api_config.api_key)  # ^^ GitHub issue #1

  return api_config

Some operations prone to break configuration loop in multi-server configurations

The existing (non-)mutating commands do perform their actions on all the defined security servers and functionality for sequential configuration should return to work when aforementioned problem is solved.

Caveat is that they can only be reliable on those controller operations that handle rising exceptions in non-fatal way -- log the human-readable error instead of throwing it and proceed with applying operation to next security server (not breaking configuration loop). This should be case for mutating operations that are part the operation graph.

Mutating administrative user creation command throws exceptions immediately when e.g. configuration keys like admin_credentials are missing, breaking the configuration loop.

Non-mutating commands (like e.g. token list) outside operation graph are orphaned in that regard, do not sometimes use API error resolving logging, but print statements.

Addressing the problems

1. Work around Configuration class behaviour to ensure correct API key use, either in described or some other fitting way (if multithreading considerations so dictate).
2. Go over the controller command implementations, especially with regard to non-mutating commands and administrative user creation command and fix the behaviours that can cause untimely breaks in the multi-server configuration loops.
3. Create minimal test for multi-server configuration that is able to catch the case where server query is made with parameters that do no match the server configuration.

pip install fails

Current installation instructions don't work. Problem with module "cement".

% python3 --version
Python 3.11.5

% pip3 --version
pip 23.2.1 (python 3.11)

% pip3 install --extra-index-url https://artifactory.niis.org/artifactory/xroad-extensions-release-pypi/ xrdsst --trusted-host artifactory.niis.org

Looking in indexes: https://pypi.org/simple, https://artifactory.niis.org/artifactory/xroad-extensions-release-pypi/
Collecting xrdsst
Using cached https://artifactory.niis.org/artifactory/xroad-extensions-release-pypi/xrdsst/xrdsst-4.0.0.tar.gz (127 kB)
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'error'
error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [10 lines of output]
Traceback (most recent call last):
File "", line 2, in
File "", line 34, in
File "/private/var/folders/l4/g_505tmn65b6rjvsrq82g9jh0000gp/T/pip-install-f_ptw4nx/xrdsst_92dbdb5d65a5407a9e0ffc3cc113773c/setup.py", line 3, in
from xrdsst.core.version import get_version
File "/private/var/folders/l4/g_505tmn65b6rjvsrq82g9jh0000gp/T/pip-install-f_ptw4nx/xrdsst_92dbdb5d65a5407a9e0ffc3cc113773c/xrdsst/init.py", line 1, in
from .core.version import get_version
File "/private/var/folders/l4/g_505tmn65b6rjvsrq82g9jh0000gp/T/pip-install-f_ptw4nx/xrdsst_92dbdb5d65a5407a9e0ffc3cc113773c/xrdsst/core/version.py", line 2, in
from cement.utils.version import get_version as cement_get_version
ModuleNotFoundError: No module named 'cement'
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.