melezhik / sparky

Sparky is a flexible and minimalist continuous integration server and distribute tasks runner written in Raku.

SYNOPSIS

Sparky is a flexible and minimalist continuous integration server and distribute tasks runner written in Raku.

Sparky features:

• Defining jobs scheduling times in crontab style
• Triggering jobs using external APIs and custom logic
• Jobs scenarios are pure Raku code with additional support of Sparrow6 automation framework
• Use of plugins on different programming languages
• Everything is kept in SCM repository - easy to port, maintain and track changes
• Jobs get run in one of 3 flavors - 1) on localhost 2) on remote machines via ssh 3) on docker instances
• Nice web UI to run jobs and read reports
• Could be runs in a peer-to-peer network fashion with distributed tasks support

Build status

Sparky workflow in 4 lines:

$ nohup sparkyd & # run Sparky daemon to trigger jobs
$ nohup cro run & # run Sparky CI UI to see job statuses and reports
$ nano ~/.sparky/projects/my-project/sparrowfile  # write a job scenario
$ firefox 127.0.0.1:4000 # run jobs and get reports

Installation

$ sudo apt-get install sqlite3
$ git clone https://github.com/melezhik/sparky.git
$ cd sparky && zef install .

Database initialization

Sparky requires a database to operate.

Run database initialization script to populate database schema:

$ raku db-init.raku

Sparky components

Sparky comprises of several components:

• Jobs scheduler

• Jobs Definitions

• Jobs workers (including remote jobs)

• Jobs UI

• CLI

Job scheduler

To run Sparky jobs scheduler (aka daemon) runs in console:

$ sparkyd

Scheduler logic:

• Sparky daemon traverses sub directories found at the project root directory.

• For every directory found initiate job run process invoking sparky worker ( sparky-runner.raku ).

• Sparky root directory default location is ~/.sparky/projects.

• Once all the sub directories are passed, sparky daemon sleeps for $timeout seconds.

• A timeout option allows to balance a load on your system.

• You can change a timeout by applying --timeout parameter when running sparky daemon:

$ sparkyd --timeout=600 # sleep 10 minutes

• You can also set a timeout by using SPARKY_TIMEOUT environment variable:

$ SPARKY_TIMEOUT=30 sparkyd ...

Running job scheduler in demonized mode:

$ nohup sparkyd &

To install sparkyd as a systemd unit:

$ nano utils/install-sparky-web-systemd.raku # change working directory and user
$ sparrowdo --sparrowfile=utils/install-sparkyd-systemd.raku --no_sudo --localhost

Sparky Jobs UI

Sparky has a simple web UI to allow trigger jobs and get reports.

To run Sparky UI web application:

$ cro run

To install Sparky CI web app as a systemd unit:

$ nano utils/install-sparky-web-systemd.raku # change working directory, user and root directory
$ sparrowdo --sparrowfile=utils/install-sparky-web-systemd.raku --no_sudo --localhost

By default Sparky UI application listens on host 0.0.0.0, port 4000, to override these settings set SPARKY_HOST, SPARKY_TCP_PORT in ~/sparky.yaml configuration file:

SPARKY_HOST: 127.0.0.1
SPARKY_TCP_PORT: 5000 

Sparky jobs definitions

Sparky job needs a directory located at the sparky root directory:

$ mkdir ~/.sparky/projects/teddy-bear-app

To create a job scenario, create file named sparrowfile located in job directory.

Sparky uses pure Raku for job syntax, for example:

$ nano ~/.sparky/projects/hello-world/sparrowfile

#!raku
say "hello Sparky!";

To allow job to be executed by scheduler one need to create sparky.yaml - yaml based job definition, minimal form would be:

$ nano ~/.sparky/projects/hello-world/sparky.yaml

allow_manual_run: true

Extending scenarios with Sparrow automation framework

To extend core functions, Sparky is fully integrated with Sparrow automation framework.

Here in example of job that uses Sparrow plugins, to build typical Raku project:

$ nano ~/.sparky/projects/raku-build/sparrowfile

directory "project";

git-scm 'https://github.com/melezhik/rakudist-teddy-bear.git', %(
  to => "project",
);

zef "{%*ENV<PWD>}/project", %( 
  depsonly => True 
);

zef 'TAP::Harness App::Prove6';

bash 'prove6 -l', %(
  debug => True,
  cwd => "{%*ENV<PWD>}/project/"
);

Repository of Sparrow plugins is available at https://sparrowhub.io

Sparky workers

Sparky uses Sparrowdo to launch jobs in three fashions:

• on localhost ( the same machine where Sparky is installed, default)
• on remote host with ssh
• docker container on localhost / remote machine

/--------------------\                                             [ localhost ]
| Sparky on localhost| --> sparrowdo client --> job (sparrow) -->  [ container ]
\--------------------/                                             [ ssh host  ]

By default job scenarios get executed on the same machine you run Sparky at, to run jobs on remote host set sparrowdo section in sparky.yaml file:

$ nano ~/.sparky/projects/teddy-bear-app/sparky.yaml

sparrowdo:
  host: '192.168.0.1'
  ssh_private_key: /path/to/ssh_private/key.pem
  ssh_user: sparky
  no_index_update: true
  sync: /tmp/repo

Follow sparrowdo cli documentation for sparrowdo configuration section explanation.

Skip bootstrap

Sparrowdo client bootstrap might take some time.

To disable bootstrap use bootstrap: false option.

Useful if sparrowdo client is already installed on target host.

sparrowdo:
  bootstrap: false

Purging old builds

To remove old job builds set keep_builds parameter in sparky.yaml:

$ nano ~/.sparky/projects/teddy-bear-app/sparky.yaml

Put number of builds to keep:

keep_builds: 10

That makes Sparky remove old builds and only keep last keep_builds builds.

Run jobs by cron

To run Sparky jobs periodically, set crontab entry in sparky.yaml file.

For example, to run a job every hour at 30,50 or 55 minutes:

$ nano ~/.sparky/projects/teddy-bear-app/sparky.yaml

crontab: "30,50,55 * * * *"

Follow Time::Crontab documentation on crontab entries format.

Manual run

To trigger job manually from web UI, use allow_manual_run:

$ nano ~/.sparky/projects/teddy-bear-app/sparky.yaml

allow_manual_run: true

Trigger job by SCM changes

To trigger Sparky jobs on SCM changes, define scm section in sparky.yaml file:

scm:
  url: $SCM_URL
  branch: $SCM_BRANCH

Where:

• url - git URL
• branch - git branch, optional, default value is master

For example:

scm:
  url: https://github.com/melezhik/rakudist-teddy-bear.git
  branch: master

Once a job is triggered respected SCM data is available via tags()<SCM_*> function:

directory "scm";

say "current commit is: {tags()<SCM_SHA>}";

git-scm tags()<SCM_URL>, %(
  to => "scm",
  branch => tags<SCM_BRANCH>
);

bash "ls -l {%*ENV<PWD>}/scm";

To set default values for SCM_URL and SCM_BRANCH, use sparrowdo tags:

sparky.yaml:

  sparrowdo:
    tags: SCM_URL=https://github.com/melezhik/rakudist-teddy-bear.git,SCM_BRANCH=master

These is useful when trigger job manually.

Flappers protection mechanism

Flapper protection mechanism kicks out SCM urls that are timeouted (certain amount of time) during git connection, from scheduling, this mechanism protects sparkyd worker from stalling.

To disable flappers protection mechanism, set SPARKY_FLAPPERS_OFF environment variable or adjust ~/sparky.yaml configuration file:

worker:
  flappers_off: true

Disable jobs

To prevent Sparky job from execution use disable option:

$ nano ~/.sparky/projects/teddy-bear-app/sparky.yaml

disabled: true

Advanced topics

Following are advanced topics covering some cool Sparky features.

Job UIs

Sparky UI DSL allows to grammatically describe UI for Sparky jobs and pass user input into a scenario as variables.

Read more at docs/ui.md

Downstream jobs

Downstream jobs get run after some main job has finished.

Read more at docs/downstream.md

Sparky triggering protocol (STP)

Sparky triggering protocol allows to trigger jobs automatically by creating files in special format.

Read more at docs/stp.md

Job API

Job API allows to orchestrate multiple Sparky jobs.

Read more at docs/job_api.md

Sparky plugins

Sparky plugins is way to extend Sparky jobs by writing reusable plugins as Raku modules.

Read more at docs/plugins.md

HTTP API

Sparky HTTP API allows execute Sparky jobs remotely over HTTP.

Read more at docs/api.md

Security

Authentication

Sparky web server comes with two authentication protocols, choose proper one depending on your requirements.

Read more at docs/auth.md

ACL

Sparky ACL allows to create access control lists to manage role based access to Sparky resources.

Read more at docs/acl.md

Databases support

Sparky keeps it's data in database, by default it uses sqlite, following databases are supported:

• SQLite
• MySQL/MariaDB
• PostgreSQL

Read more at docs/database.md

TLS Support

Sparky web server may run on TLS. To enable this add a couple of parameters to ~/sparky.yaml

configuration file:

SPARKY_USE_TLS: true
tls:
 private-key-file: '/home/user/.sparky/certs/www.example.com.key'
 certificate-file: '/home/user/.sparky/certs/www.example.com.cert'

SPARKY_USE_TLS enables SSL mode and tls section has paths to ssl certificate ( key and certificate parts ).

Additional topics

Sparky cli

Sparky cli allows to trigger jobs in terminal.

Read more at docs/cli.md

Sparky Environment variables

Use environment variables to tune Sparky configuration.

Read more at docs/env.md

Glossary

Some useful glossary.

Read more at docs/glossary.md

CSS

Sparky uses Bulma as CSS framework for web UI.

Sparky job examples

Examples of various Sparky jobs could be found at examples/ folder.

See also

• Cro - Raku Web Framework

• Sparky-docker - Run Sparky as Docker container

Author

Alexey Melezhik

sparky's Issues

Badges ?

It would be more than cool to have badges for easy visual reporting in external services.

Wall of status

A new page to list all status for a bitbucket project, github org, module namespace etc...

I do this kind of thing, it's a bit hacky and dirty but it is working for a user point of view.
Here is my homemade thing :

buildbot has a page that looks like this, but he mixing workers with status (matrix commit/worker).

Replace JSON::Tiny by JSON::Fast

It's meant to be faster and much better maintained, it has a much more integral place within the ecosystem.

The only gotcha I can imagine with this is if there is a test that relies on the formatting of some JSON... other than that, I think it should be a trivial change. I can't do it right now but probably could do it some other time, so making this note.

Change shebang to raku ?

Document a deliver/publish

Is there any plugin to deliver to s3, artifactory, ftp, whatever ?

Could we document it ?

Can we assign task to a group of workers ?

More on sparrow side, but can we assign a project to a group of workers, let's say "big nodes"... And some workers defined elsewhere as being "big nodes" can handle the build ?

Build along a "reference build"

For instance when rebuilding a failed build, I would like to rebuild the "reference OK" build : means the last one that was green.
We won't be able to do this completely, because inputs are maybe not available (commit or deps) but we could at least try to emulate some parts : the sparrowfile and sparky.yaml and therefore, as it is inside these files, the docker image and/or host target (but not its state of course).

It requires to store this "last OK config" somehow (versioning ? db ?).

It's not 100% clear in my mind and it's a nice to have, but I really think it has great added value for end user.

On end user side, I would like to have an option to start 2 builds at the same time, one behing the reference and the other the current.

Should play well with #32 😀

Diff between build

I would like to be able to compare builds, especially when it goes from OK to KO.

It means a view where we can see these kind of thing, to be implemented as best effort :D

• the config/sparrowfile diffs if any
• the input diffs (commits if coming from scm, md5sum or similar if tarball) if any
• log diffs
• the diffs between docker images version (only the versions, not deeper)
• the diff between runner

Replay with edit ?

When we do a rebuild, it would be nice to edit "on the fly" the configuration/sparrowfile.

Licence ?

What is the licence of sparky ?

Turn off/refine "flapper protection"

Hello,

in my use case, sparkyd runs forever and ever, as a pm2 task. From what I understand in the code, timeouts are never reset, and once they reach 5 occasions, the project is permanently marked as a "flapper" which defeats the purpose of having the CI running 24/7 continuously.

I can imagine that there are timeouts for whatever reason but I generally don't even want that to make any difference, as long as there is any chance of recovering, whether that's 5 minutes later, 2 hours later or a couple of days later. I think there should be either a way to turn this feature off completely, or at least the timeouts should wear off. If every second attempt times out, that's still okay. If 5 attempts time out in a row, even that might be okay, depending on the timing circumstances. In any case, only massive bursts of timeouts should be considered.

Hamburger menu does not expand on safari mobile

Reboot sparky and sparky web from UI

Ability to ask for a soft restart from UI.

This feature depends on #17

Fixed size for status buttons

Very UI related, but I guess having the "running/failed/success" button with the same width would render better.

Can we connect to a running job ?

I know we can, at least with docker (something like docker -H tcp://thenodewhereitisexecuted exec -ti thelayerhash /bin/bash) but do we have the info somewhere (in the logs or better in the running build ("report") page ?

Extra bonus : It could go with a "pause" feature for short living jobs (I'm greedy sorry)

Add notifications to nav bar

I would like to have a visual feedback to running jobs, something like this :

I do not checked too much the sparky internals but it probably needs to technically implement something like websocket or similar ?

FTP for File Triggerring is confusing

While I perfectly understand and love the feature, the acronym collision makes me uncomfortable.

Can we find something more around "file/dir watcher" or something else like "file creation" or "file event".

Spin up a test Sparky instance for dog fooding

I'd be good to have one with public IP. http://rakudist.raku.org/sparky/ should be considered booked for Raku community purposes and is not good candidate if users want just to play with web hooks or other test requests ...

I have free tier aws account which expires pretty soon, any ideas where we could have (free?) resources? We need something 2-4GB RAM and 2-4 CPU core setup.

cc @thibaultduponchelle

Print sparky and sparky web logs in web UI

Can be put behind admin auth after #17 is implemented

Cascading between projects

This is another one that comes from my mind.

Could we chain easily some projects "harmoniously".
At the end of a running build, trigger one or more other projects, potentially passing one or more parameters (sha1 or tags).

It is often needed for docker image creation (one repo contains low level image, one contains technical and is built on top of low level one, one contains functional etc...).
It is also useful for version propagation.

It would be very cool to do not have to implement cascading mechanism by ourself through http form calls or auto git commit.

When using scm or delivering, show link in report page

When building from scm, or delivering things during build, put links in report page.

View to see overall sparky stats

An overall view of what sparky have done from "all time".

Something like :

To put somewhere either on main page or on a "statistic" or "status" page

Animated "running" button/badge

Can we make the "running" button blink ?

Update file extensions

I saw the modules still have pm6 extension which is not "trendy" anymore and eventually will get deprecated anyway. It's probably a quick non-breaking change to make it rakumod.

Hightlight code/config snippets

Like scenario and config.

It will improve a lot user experience and beauty 😄

sparkyd refuses to start if a sparky.yaml is missing

Hello,

When a projects does not contains a sparky.yaml or contains an empty one, sparkyd refuses to boot.

Adding even something dumb like

a: "b"

is fixing the boot.

It is maybe "by design", but the documentation does not clearly mention this (to my understanding it seems to say implicitly the contrary).

Thank you 😁

Thibault

Tag a new release

It's been a while.

I'm considering self-hosting this, but I'd like to check out/build from a tag.

What is downstream projects ?

bin/sparky-web.raku - rewrite from Bailador to Cro

Reason: Bailador seems is not being in actively developing phase, last commit was more then 1 year ago. It also has some annoying glitches making web application crashed on and off. On other hand Cro seems growing, well maintained and at the end of the day looks more convenient and easier to use ...

Workers status page ?

The same way we have a list of builds, we could have a list of workers with an info about if the machine is up or down.

Same project config for multiple items

The idea is :

• Define one project dir with a sparrowfile and a sparky.yaml
• Use it for multiple items (repos or modules) through a list (implementation to be defined) (*)

(*) An extra feature would be github/bitbucket/gitlab org/project/whatever scanning

SCM triggering support

I'd be good to have sparky builds get triggered by changes in SCM:

sparky.yaml:

  scm:
   url: https://github.com/melezhik/rakudist-teddy-bear.git
   branch: main

Outdates build status

Principle

An old build should not have the same value than a recent build.

Imagine a package that have a slow releasing life-cycle.
It has an old green status, but it should not be flashy green but dull gray of with a small banner.

Demo

Here is a classic (github action) badge :

An outdated could be :

Or with banner :

On what criterias a build is outdated ?

At least a docker image version and a different runner (different host or same host but upgraded).
These infos could be found in the sparky.yaml and sparrowfile.

Does it makes sense ?

Can we think about "easy install"

I think the install should be dead simple, which is not the case today.
(Ok there is the docker image but even)

I don't know what to do, but can we imagine at least a "fatpacked" version of sparky (sparky + modules).
It would be even better to have a raku + zef + modules + sparky since raku is not always available on distributions and for sure not with a recent version (and the correct interpreter name).

Maybe a curl .../install.sh | bash would do the trick even if some people does not like this 😋

According to me, it is a very important issue to start with for better adoption.

Show last builds on project page

Show last builds (at least) of a project on his project page.

Problem loading configs hash

Hello,

raku bin/sparky-runner.raku --dir=~/.sparky/projects/hello/ is not working because the ~ is not translated to absolute path at this line.

You end up with this empty configuration because we never enter the if block :
merged sparrowdo configuration: { }

CI looks broken, project stuck in queue

Project git-pheix-core-perl6.git stuck in queue: https://sparky.sparrowhub.io/queue. Looks like no runners are available or smth like this.

Also I see last successful build about a week ago: https://sparky.sparrowhub.io/builds

Add user management

Some informations and actions should be only accessible to an administrator.
Some informations and actions should be only accessible to its owner(s) and administrator(s).
The administrator should be the one and the only one to be able to manage this user management (create user, delete user, attach/detach to project etc...).

Technically, I think it means adding some tables in DB + handle sessions etc.. in web UI.

Fixed size for status buttons

Very UI related, but I guess having the "running/failed/success" button with the same width would render better.

Sparky needs a logo

This is my favorite issue 😎

HTML produced by sparky is not valid

The HTML produced by spark web is weird. No html tag, script and css before head, head almost empty etc...
Can we fix it ?

Could we enable discussion tab ?

This tab :

To be enabled in general settings from repo :

Sparky CI Web restarting too frequently

Hello,

I feel like Sparky CI Web is restarting too frequently, like if it was watching logs and restarting on change:

Is it normal?

Use columns instead table ?

Filter queue per worker

Worker workload

Somewhat related to #27 and #28 but I would like a view with a graph to monitor the health of workers. We can play with kubernetes also for this (but it won't always be deployed in k8s...).