foundation-model-stack / train-conductor

Training job management tool for foundation model service

Train Conductor

⚠️ EXPERIMENTAL ⚠️

Train Conductor is a lightweight Python-based tool for the deployment and management of Kubernetes batch jobs. It is comprised of three components:

• API Server (gRPC) - This serves training-related requests to the end user. Any changes requested by the user are recorded into the database.

• Watcher/Controller - This component reads desired state of training requests (i.e. new requests, modification of existing requests) and accordingly sends requests to the Kubernetes API server in order to launch new or cancel exiting training requests. It also watches the kubernetes API server and records state changes of the running jobs, and records them in the database.

• Database - Currently implemented in Redis. Train Conductor assumes the existence of the database, the user will provide the connection information as part of configuration.

Architecture Diagram

Below is an example flow of how a new training request is managed.

train-conductor's Issues

Enable mTLS on the gRPC server

Fix Known Python Vulnerabilities in Miniconda

There's a Low and High vulnerability in Miniconda Which is fixed from the latest version from:
https://repo.anaconda.com/miniconda/

Currently that's: Miniconda3-py311_23.11.0-2-Linux-x86_64.sh

Before the latest fix:

<style> </style>

severity	severityCHML	cvss	riskFactors	cve	link	hasFix	status	packageType
low	L	1	Has fix,Recent vulnerability	GHSA-v8gr-m533-ghj9	GHSA-v8gr-m533-ghj9	Y	fixed in 41.0.4	python
high	H	7.5	Attack vector: network,DoS - High,Has fix,High severity,Recent vulnerability	CVE-2023-49083	https://nvd.nist.gov/vuln/detail/CVE-2023-49083	Y	fixed in 41.0.6	python

After the latest fix, no vulnerabilities.

Switch Image to run as non-root

Noticed that image is running as root. Twistlock scan marks that as a high security issue.

description

It is a good practice to run the container as a non-root user, if possible. Though user namespace mapping is now available, if a user is already defined in the container image, the container is run as that user by default and specific user namespace remapping is not required

---

Proposing to have it as user 1000 and doing a OS update to catch any latest OS fix that might be available at build time.

Decide on and implement on a global logging solution

List parameters to expose on integration with SFTTrainer

Description

As part of integration with SFTtrainer codebase, we need to enlist all parameters to expose to users . This issue captures the information.

Acceptance criteria
we have agreed upon parameters to expose to users

FYI @tharapalanivel @kellyaa

Ensure sufficient simulation for unit tests

Our solution requires integration with both a database and kubernetes. Ensure we can perform sufficient unit tests that does not need a full blow ecosystem installation.

Decide on mechanism for watching and leader election

Our solution will require:

• Watch for new job requests, or job cancellation requests in the database
• Watch for job events in k8s (started, failed, completed, etc)
• Compare the delta and:
  • Update the database based upon changed state of k8s job
  • Deploy or cancel k8s jobs based upon changed desired state of jobs in DB
  • Optionally: Delete k8s jobs that shouldn't exist, according to the DB

We will need to either use an existing or write new python code to perform this watching, and also employ leader election

Validate records before writing to the database

Setup Automated Tests for Train Conductor

Kelly mentioned that a good repo to emulate would be: caikit: https://github.com/caikit/caikit

And... As far as testing, our biggest barrier was the fact that we have external dependencies on k8s and redis.. And trying to emulate that for proper unit tests..

Better open source Hygiene/repo Setup for Train Conductor

A good repo to emuulate would be - Caikit: https://github.com/caikit/caikit