linus-berg / apc

Artifact Processor Complex - Sync packages and artifacts for offline / airgapped consumption

Artifact Processor Complex

⚠️ Warning: This project is still under heavy development, to be considered a beta, at best.

The APC is used to collect and store a varied and large amounts of artifacts, for example npm, nuget...

Primary use-case is transferring to airgapped / high security networks for offline consumption via an artifact repository.

The complex keeps track of already collected artifacts, dependencies, and versions.

The APC's collection of artifacts is greedy, it will by default collect every version and every dependency version, recursively, until every item is collected, artifact filters can be applied to reduce the amount collected.

It builds daily deltas to reduce the day-to-day load on the transfer device.

The design is microservice oriented and intended to minimize the amount of work needed to build a new "Processor", for example maven.

Processors can safely rely on the core infrastructure of the APC, the core infrastructure handles the memorization of metadata, and the collection via standard protocols, for example http, git, or docker.

Recommended minimum requirements (single node)

Resource	Req
RAM	16GB
CPU	8 cores
Storage	2TB * APM (depending on usage)
Network	>= 1 Gbps

The APC is designed to be horizontally scalable and suitable for kubernetes, however, kubernetes deployments have not been tested thoroughly.

The following services are required to run the complete suite of APC capabilities. A batteries included starter pack is included under /examples and /Compose, however, these are not production ready configurations, and are only intended as an development/test environment.

Service	Requirement
Keycloak	Required (only for API and GUI)
RabbitMq	Required
Minio	Required
MongoDb	Required
Redis	Required
Container Registry	Required (only for ACM.Container)
OpenTelemetry Collector	Recommended but required for Telemetry
Grafana	Recommended
Prometheus	Recommended
Tempo	Recommended

Modules

Tag	Name	Description
APC	Artifact Processing Complex	The name of the entire suite.
ATM	Artifact Tracking Module	The ATM handles the tracking of artifacts to request an update by the APC.
APM	Artifact Processing Module	The APM handles the processing of a defined artifact-type.
ACM	Artifact Collector Module	The ACM handles the collection of artifact-types based on standard protocols.

Environment Variables

Name	Default	Modules
APC_OTEL_HOST	-	Ingest, API, APM, ACM
APC_RABBIT_MQ_HOST	localhost	Ingest, API, APM, ACM
APC_RABBIT_MQ_USER	guest	Ingest, API, APM, ACM
APC_RABBIT_MQ_PASS	guest	Ingest, API, APM, ACM
APC_REDIS_HOST	localhost	Ingest, API
APC_REDIS_USER	-	Ingest, API
APC_REDIS_PASS	-	Ingest, API
APC_MONGO_STR	-	Ingest, API
APC_API_HOST	localhost	GUI, ATM, API
APC_API_PORT	4000	GUI, ATM, API
APC_ACM_DIR	/data/	ACM.Git
ACM_HTTP_DELTA	true	ACM.Http
ACM_HTTP_MODE	lake	ACM.Http
ACM_S3_ACCESS_KEY	-	ACM
ACM_S3_SECRET_KEY	-	ACM
ACM_S3_REGION	-	ACM
ACM_S3_ENDPOINT	-	ACM
ACM_S3_BUCKET	-	ACM
ACM_CONTAINER_REGISTRY	-	ACM.Container

apc's Issues

[ACM] Separate directory per collector

Git collector not consistent during high loads

Git collector is supposed to store 3 files, index, timestamp, and the bundle file, however sometimes they do not all get uploaded during high loads, needs to be investigated to ensure this happens otherwise Git repository will break over time.

Ability to filter artifacts using semver ranges

Currently, artifacts can be filtered using a regex:

APC/ACM.Router/Router.cs

Line 18 in bb615d8

bool collect = regex == null || regex.IsMatch(kv.Key);

For many artifact types, the version is a semver string, and it would be great to be able to instead (or additionally) filter using semver ranges.

I see a few options:

• Add a Artifact.filterType (enum FilterType { Regex, SemverRange }) indicating whether the filter (currently only opaque string) should be used as a regex, or as a semver range:
• Add an additional Artifact.semverRangeFilter that will filter using semver ranges in addition to the current regex filter.

Use full-uri path for storing data during ACM collection.

Currently only the path after the domain is used to save the files.

Updating ACM.Http to utilize full paths, HTTP collection could be a "data lake" where each individual module can symlink to already collected files.