blendle / automaat Goto Github PK

After using and talking about Automaat for a while now, I think the naming scheme is confusing – especially for non-technical people – and thus should be changed.

Specifically, right now a pipeline is a collection of steps that you can run, and when you run a pipeline, it creates a task based on the configuration of that pipeline.

Technically, the name pipeline makes sense, since you are chaining together a set of steps which each run a single processor with some configuration attached. But, when talking about it within the UI, in the abstract, without (having to) knowing how a pipeline works, this analogy breaks down.

Instead, talking about a task makes things much more clear, as the dictionary defines a task as "a piece of work to be done or undertaken.", which fits perfectly in this context.

Similarly, whereas tasks currently mean "a pipeline that is actively running", it makes more sense to call this a job: "the object or material on which work is being done ".

So, in conclusion: the term pipeline will disappear, the term task will take the role of a pipeline, and the term job will take the role of a task.

This work was already done for the web client in 2d9bc81, so this issue is to track the change of the terms in the rest of the code base (mostly the API server).

The CLI could be the second client next to the web client.

The idea would be that the CLI provides limited, but useful functionality without having to visit the website.

Basically, you would be able to do things like:

automaat task search PATTERN
automaat task run NAME --var1=value1 --var2=value2
...

Those two would be the most basic commands needed to make it useful, but there are others as well.

Depending on how far we want to go, the second one could be interactive if no variables are provided, asking you to provide the correct values for each required (or optional) variable.

Currently, all pipeline variables are required to be set before creating a task.

I've come across pipelines that could benefit from optional variables. For example, if your pipeline returns a large set of objects, and you want to allow the person running a pipeline to be able to filter those objects based on some extra variable.

You could add a {My Attribute Filter} variable, but now you are always required to provide a filter. You could then use (eventually) step templating (#23) combined with default variable values (#3) to (f.e.) ignore the filter if it's set to the * value, but that would make steps more complicated to implement and it makes it less clear when using a pipeline what the * is supposed to mean, requiring you to be more explicit about it in the variable description.

So instead, if we allow setting optional: true on pipeline variables, they are allowed to be empty. If you use these variables in the steps, a variable that isn't provided simply ends up as an empty string when using the variable (which might still require some templating to use, but I'm not sure there's a way around this).

By providing first-class support, we can also automatically convey the fact that this variable is optional in the UI, for example:

The "(optional)" part is added automatically.

Another way to achieve this is to use the default variable value feature (#3), and set the default to an empty string (""), which basically results in the same outcome. However, that feels like misusing the system, and it also prevents us from making it more explicit in the UI what is happening, so I'm leaning against that "hack".

Currently the default error object is returned. By implementing a custom error object, the returned error can be more descriptive.

see: https://graphql-rust.github.io/juniper/master/types/objects/error_handling.html#structured-errors

Task URLs can be shared using:

• https://automaat.test/#/task/11
• https://automaat.test/#/task/12

This works, but if you are sharing with others, and potentially share multiple tasks, it makes it easier to understand what is what by having the task title slug be part of the url:

• https://automaat.test/#/task/11-customer-capabilities
• https://automaat.test/#/task/12-find-customer-uuid

Thinking a bit more on #22, I think if we are going to add more complexity to the current self-rolled templating support, we might as well switch to an existing/well-known templating language, so that we can trim down on the added complexity of our own system, and expand the templating capabilities beyond what was proposed in #22.

It seems to make most sense to switch to Handlebars for templating:

• it's well-known
• easy to learn
• good Rust support
• extensible

Given that last point, we could solve the issues in #22 by adding our own helpers, such that you could access variables like so (to be bikeshedded):

• {{ var "my pipeline variable" }}
• {{ global "my global variable" }}
• {{ local "my temporary variable" }}

I'm still on the fence on that last one, as we might as well combine 1 and 3 into a single var helper. In #22 I mentioned that temporary step output variables could shadow pipeline variables, which is true, but given that the pipeline author is in control of both what step output and pipeline variables exist in a pipeline run, it seems less of an issue that shadowing is allowed, and is easily prevented by not naming your step output variables and temporary variables the same.

I also think we should rename the $input variable to last step output, so you can use the implicitly created {{ var "last step output" }}. We'll just have to make that a reserved variable name, so that pipelines can't use that name as a pipeline variable. Then, the issue raised at the end of #22 is also moot, as we always implicitly assign the last step output value to this variable, even if you also explicitly assigned the output to a different variable.

An excellent point by @jurre (who has plenty of experience in this field).

The current authentication/authorization setup (#19, #21, #39) is not meant for public security (as mentioned in those issues, and as will be mentioned in the documentation).

However, RTFM is a thing, and insecure tools (such as Wordpress, MongoDB, etc) are exploited on a daily basis.

So, at some point, we probably want another layer of security that re-introduces the concepts built (and later removed/changed) in 411e8e2 and 5552179.

This would be an on-by-default "zero access without authentication" feature that can be disabled using a CLI flag for those running the tool within their organisation's internal network or behind a VPN (such as we are at Blendle), to make its usage more organisation friendly.

It should still be clearly documented that even with this feature, you'd be wise to not expose this tool to the public internet, but at least the default setup is a little bit more resistant against people not reading manuals or not understanding the power a tool like this holds.

Each view in the application should be shareable via a link (to a certain extend, excluding the things that are "unique" to your own session, such as favourites (#9), one-clicks (#10), etc).

We can use the History API for this.

This would be the list of interesting deeplinks:

• /pipelines/<pipeline_id> #/task/<task_id> – direct modal window
• #/task/<task_id>?field1=<value>&... – modal window with pre-filled fields
• ?search=<query> – direct search result
• ?filter=<regular|favourites|one-click> – filtered lists (see #11)
• ?search=<query>&filter=... – combination of multiple queries

Deeplinking to a pipeline should never automatically run the pipeline, even if all fields are pre-filled.

The URL should instantly update as you type in form fields (e.g. search or pipeline variable fields), so that you can instantly copy/paste the links without any extra work.

When someone searches for a pipeline, but comes up empty, there should be a message, telling the person that no pipeline with the given name could be found, but that they could ask "us" to create one (us being the owner of the hosted Automaat instance).

To make this seamless, clicking that "let us know" link should open a pipeline that allows you to fill in your request details, and that request being sent somewhere (the exact thing that is asked, and what happens when submitting the pipeline is dependent on what steps the Automaat owner configured for this pipeline, e.g. sending an API request to your server, or posting a chat message in your chat service of choice).

This would probably need #16 to allow people to describe what they are looking for with more ease.

To support this, there needs to be some mechanism to designate a pipeline as "designated for the "ask for a pipeline" feature". I'm not sure yet how that would look, but there are multiple solutions to this.

One solution could be to start with the possibility of hiding a pipeline, meaning it won't ever show up in search results, but can still be linked to directly using a deeplink (#14). Then all that is left to do, is to be able to assign a pipeline ID as the designated "ask for pipeline" pipeline, which would then insert a link in the web client linking to that pipeline.

Another solution, that might be less intuitive in usage, but requires less "special care" would be for pipelines to be marked as "show always, regardless if it matches the search query". Coupled with a "weighting" system to allow pipelines to always be the last (or first) search result, this would mean a regular "Ask for a pipeline" pipeline would show up in the search results, regardless of the search query.

I'm not too big a fan of this last solution though, as it's less intuitive than a dedicated visual queue that no results were returned, but that there is a way to ask for the thing you need, as shown in the above screenshot.

Right now, when a task (née pipeline, #28) runs to completion (either successfully, or as a failure), the output of the last step that ran is considered to be the final output of the task.

This output is considered to be regular text which can be formatted using markdown.

The GraphQL API currently returns this text in its raw form, at which point the web client parses it as markdown, and converts it into HTML to show it on the screen.

This works, but specifically for the web client, this results in a binary that is 50% larger than it could be, because the markdown parser library used loads in the entire Unicode character table, which adds 160 kB of extra data (see #27 (comment)).

To solve this, I propose we move the markdown parsing to the server, and let the clients dictate which format they want the output to be returned in.

The API currently looks like this:

I propose the following change:

type TaskStep { 
   id: ID! 
   name: String! 
   description: String 
   processor: Processor 
   position: Int! 
   startedAt: DateTimeUtc 
   finishedAt: DateTimeUtc 
   status: TaskStepStatus! 
-  output: String
+  output: StepOutput
   task: Task 
}

+ type StepOutput {
+   text: String!
+   html: String!
+ }

This allows for a couple of improvements:

1. Markdown parsing happens server-side, improving client performance and reducing the Wasm binary size by 50% for the web-client.
2. Clients can ask multiple formats from the server, which makes it easy to support #17.
3. We can add new output formats in the future if needed, without introducing breaking changes to the API.

_{side note: the regular output is called text instead of markdown, since any step output can be formatted as markdown but doesn't have to be.}

There's currently no pagination implemented, which won't work well for very large sets of data.

One option would be to support the Relay spec, which has support for this as well, but I have a feeling it might be overkill to solve this particular problem, and it's also currently not supported out of the box by the GraphQL library we use.

Once the list of tasks grows, you get a lot of copy/pasting of commonly-used steps, such as this one:

steps:
  - name: Validate Customer UUID
    description: |-
      Validate the customer UUID format as UUIDv4.
    processor:
      stringRegex:
        input: '{{ var["Customer UUID"] }}'
        regex: >-
          \A[a-f0-9]{8}-[a-f0-9]{4}-4[a-f0-9]{3}-[89ab][a-f0-9]{3}-[a-f0-9]{12}\z
        mismatchError: The provided customer UUID has an invalid format.

  - name: Other Step
    ...

It would be nice if there was some kind of (optional) feature that allows you to define global steps (somewhat similar to global variables #24) that can be shared across tasks.

For starters, such a step could be used as:

steps:
  - template: Validate Customer UUID
  - name: Other Step
    ...

The concept would probably be based on inheritance. You can define a template key to inherit any properties set in that template, and then override what you need changed in your own task.

There are still plenty of design decisions to make, and gotchas to consider, but the general concept of this is very useful for larger Automaat instances.

Right now, the API returns database record IDs as strings in the GraphQL API.

As per the spec, these IDs are "not intended to be human‐readable". We should probably add some encoding logic to obfuscate the final ID string that's sent to the clients.

see: https://graphql.github.io/graphql-spec/draft/#sec-ID

There are quite a few features (#9, #10, #11) that require some kind of state. In those feature descriptions, its mentioned to use client-side storage for the time being. This feature request is to track storing that information server-side and adding authentication support.

The goal right now is to keep it as simple as possible, while also providing enough flexibility for the current and near-term needs.

If you have any thoughts on this, or a better proposal, please speak up!

Here's the gist of it:

Automaat Server

• Add a users table to the database, containing the following columns:
  • api_token – a personal token used to authenticate
  • preferences – a JSON-based column to store preferences
• Add a CLI argument (e.g. --enable-authentication) to require API requests to contain a header with a user token.
• Add a new GraphQL query to get the preferences of a user.
• Add a new GraphQL mutation to allow resetting the API token, and also changing the stored preferences.

Automaat Web Client

• When receiving a 401 response, show a full-screen dialogue asking the visitor to provide a valid user token (which they would get by asking a server admin to provide them one).
• Store this token in a secure cookie for future use.
• Add this token to all requests.
• On startup, fetch user preferences from server using the server API

Notably missing from this proposal is creating user accounts. This will have to be done manually by server admins by adding a record to the database using a raw SQL insert statement. This keeps things simple, and allows us in the future to add an optional --allow-user-registration server flag to start exposing this functionality (if there is ever a need).

This implementation is not the meant for public access security. Given that Automaat is a powerful tool if exposed in your organisations infrastructure, I think we should explicitly mention that this tool is meant to run inside your internal network (or exposed over a VPN), the authentication added here is merely meant to:

• Allow a specific set of people within your organisation to access its functionality.
• Persist user preferences across clients/devices.
• Provide future-proofing for when we add role-based access models to further segregate who gets to do what.

When starting a long-running task, and then closing the pipeline details view in the web client, the task will continue to run on the server, and the client will continue polling for results in the background.

While some kind of "background tasks with notifications" system might be nice to have in the future, the web client currently isn't build to support this use-case, and the above will result in weird errors.

For now, three things need to happen:

1. when a task is started, and the pipeline details view is closed, the web client should stop polling for the results of this task #18 (comment).
2. Additionally, when the pipeline details view is closed, a query should be sent to the server, letting it know that the client no longer cares about the results.
3. On the server, all running tasks should include a cancellation context, allowing tasks to be cancelled mid-execution.

Steps 2 and 3 are more involved, and less of a priority for immutable tasks, but when running a task that performs some destructive action, you might want to be able to quickly cancel a task when you accidentally started one with the wrong variables.

For 2 and 3, the most straightforward solution would be to:

• When a cancellation request comes in, set the Task to status Cancelling.
• Before running each task step, check if the task status was changed to cancelling.
• If it was, stop execution of any steps still waiting to run, and update both the pending task steps and the task itself to Cancelled (tasks and task steps have their own statuses in the database).
• By fetching task details, the client can precisely show which steps did run, and which got cancelled, so the user knows if their accidental run caused any harm.

For supporting the "cancel accidental run" case, some additional changes need to happen:

1. When starting a task, the Run button needs to change to Cancel.
2. When clicking that button, a cancellation request needs to happen.
3. The response should be a Task object, after it was cancelled, so the client can show which task steps ran, and which got cancelled.

In the case of step 3, we might actually want to make the cancellation asynchronous, since a long-running step still in progress might block the response for a significant time. So instead, we should immediately return an acknowledgement that the task will be cancelled, and let the client poll until cancellation happened.

After implementing #8, #9 and #10, the list of tasks can become quite cluttered, especially if you have a lot of one-click tasks created from the same regular task, and thus sharing a similar search keyword among them.

One solution is to be able to filter the search results based on the "task types":

We should probably keep this a client-side implementation, and simply show/hide pipelines based on the active tab, and the types of tasks shown. (see: #19)

Bonus points if we persist the selected filter across browser sessions.

A pipeline variable is currently defined as:

struct Variable {
    key: String,
    description: Option<String>,
}

I propose changing it to:

struct Variable {
    key: String,
    description: Option<String>,
    values: Option<Vec<String>>,
}

By adding a values (open to bikeshedding, such as acceptable_values) field, we can optionally limit the accepted variable values to a pre-determined set of values when constructing a pipeline.

This can be useful, for example when you want a pipeline variable that consists of a boolean yes/no value, or a set of strings such as development, staging and production.

In essence, the value then becomes an untyped enum, checked at runtime.

This information can also be used by the UI to show a select field, instead of an input field, when the set of acceptable values are finite (and perhaps even dynamically make it a list of checkboxes, if there are three or fewer options to choose from, as that makes the UI more easy to use).

Not sure yet if we want to do anything special with a one-value enum, such as using a hidden field for the variable value, since only one value is acceptable, or just not accepting it as input when creating the pipeline variables, as I can't see any reason why you'd want to do that. But it's not that important to tackle right now.

In an ideal situation, variable values would be more strongly typed, so that more advanced variables become possible (arrays, objects, true enums, integers, etc), but that's still a long way off (and perhaps even undesired, as it adds more complexity to the system), and this will already be an improvement that enables more types of pipelines.

Another quality-of-life improvement for pipeline-heavy workflows, is having "one-click pipelines" that (as the name implies) don't require any configuration, and run instantly when you click them.

This can again be made client-side only for now, but could be migrated to a server-side implementation once user sessions are a thing.

The idea would be as follows:

• You can create a "one-click pipeline" based on an existing pipeline by opening that pipeline, filling in all required details, and instead of running the pipeline, selecting "save", and giving the pipeline a name.
• These pipelines are visually distinct from others.
• They also have their description removed, as it's assumed you know what it does, so that they take less room in the list of pipelines.
• They are always listed as the top results, above favoured (#9) and regular pipelines.
• If you click them, they open, and then directly run the pipeline with the pre-filled variables.
• Other than that, they behave the same as other pipelines, so they can be filtered using the search bar.

As for the implementation itself, there are a few things to do:

• Keep a local store of "favourites", which is basically a hashmap of name ➜ { pipelineId, variables }.
• When a list of pipelines is rendered, check if one of the favourite titles matches the search pattern (if any), and inject those at the top.
• When a favourite pipeline is clicked, open a regular pipeline view, pre-fill the variables, and submit the form instantly.

Not sure yet what the best way is to one-click pipelines, but I guess we could start by adding a small x to the left of the pipeline on hover.

The alignment of task names without a description is off:

While working with some tasks, I noticed a pattern that there is a need to embed secrets into tasks that shouldn't be provided by a task variable, but you also don't want to have exposed in the step definitions in your own Automaat repository.

For example, say I have a SKU Sales Numbers task that does a query to one of your internal APIs, based on a {SKU} variable. This API requires authentication by providing an Authorization header. You don't want the people using your task having to input this token (both for convenience and security), so you want to bake this into the step definitions of your task.

However, you also want to be able to store your step definitions in source control, so that they are versioned, and can be tweaked by others, so hard-coding the token in your tasks/sku_sales_numbers.json file is out of the question.

You could resort to external templating systems that pre-process your task definition files to embed secrets when needed, but that adds extra overhead.

Instead, I think it makes sense to add a new concept to the server component: global variables.

Basically, the following added functionality is needed:

• store global variables as key/value pairs in the database (both are strings)
• creating/updating global variables using GraphQL (no need to query, for now)
• support referencing global variables in step definitions (see #23)

I think it also makes sense to store these variables in encrypted form in the database. Not all of them are necessarily secrets, but I see no reason to distinguish between the two types of globals, and do see a need to store some of these values encrypted, and decrypt them on the fly, based on an encryption key provided on server start-up.

The application should be usable (and feel natural) when only using the keyboard as an input device.

Right now, a few shortcuts are implemented

• F for search field focus
• ENTER in pipeline view to run pipeline
• ESC in pipeline view to return to home screen

These should be expanded to make the entire UI easily navigable. Also, ? should bring up a keyboard shortcuts cheat sheet view.

Pipelines are currently sorted by their database ID (and thus in essence by a "created at" timestamp).

Ideally, the sorting would be a bit more dynamic, and useful.

There are a couple of data points we could use to base our sorting on:

• last used
• most viewed (or searched for)
• most weighted

The first one is already possible in a roundabout way, by looking at the run-time of the last task created from a given pipeline, but ideally we'd add a cache column to just add that data to the pipeline record itself.

The second and third one are not possible yet, but all we'd need is to store some more data in the database.

All of the above is based on global state, meaning, since there is no concept of user sessions (yet), there's no way to keep track of your last usage of a pipeline. One solution is to use local client storage to keep track of this data, with the two downsides being that it's not shared across devices, and it's lost when local storage is emptied.

Personally, I'm leaning towards starting with the "last used" solution, and storing it locally for now, until there's such a thing as server-side user sessions. It's annoying that it's not shared across devices, but for now it's fine.

Currently, the task description is used as a short-form explanation of the task when showing a list of tasks:

When opening a task, that same description is shown at the top:

In this last case however, you sometimes want to provide additional minutiae details that aren't important enough to clutter the task listing (or are too long to list), but are important enough to show somewhere.

This is what a long-form description could provide.

• If one is provided, it is shown in the task details window
• If none is provided, the regular short-form description is shown

I'm not sure on the naming yet.

While working on #23 I realised that we currently have a split situation in how variable values are handled:

When a job (#28) is created, the variable values that are sent along in the request are directly embedded into the serialized processor JSON object:

Then, when a job step starts running, it compiles in the {$input} variable value at that time (think of it as a JIT):

This last part is needed, because we have to wait for the output of the processor of step 1, to be able to inject its output as the input of the processor of step 2.

However, the injecting of the variables on job creation is not needed, we can delay this until a step actually runs.

There are several advantages to this:

1. code simplification, because there's only one point in time, and at one place in the code where variables are replaced
2. it provides more flexibility in terms of data access, as we can fetch the variable values of a job at a later point in time, if we need to
3. most importantly, by separating the storage of the processor config, and the variable values that will be injected at runtime, we can encrypt the variables in the database, and only decrypt them once they are needed to run the processor, and the actual values only live in memory for the duration of that processor run.

This ties in nicely with #24 as that feature will also require encrypted data storage.

So in order for this to work, we need:

• A new database table to store job variable values.
• A way to provide an encryption key to the server, and use that key to encrypt and decrypt stored data.
• Consolidation of the templating code into a single location, ran right before a processor starts running.

Right now, a pipeline variable automatically becomes an input form field in the web-client pipeline view:

But there are pipelines that become more usable if they'd support some kind of "batch input" process.

One example would be to allow people to copy rows from a spreadsheet, and paste them into a text field. The processors handling this could then do their thing based on newlines in the input, and handle each line as needed.

There is no need for special treatment of this data in the processors themselves, it's simply something you take care of when configuring the processor. For example the shell-command processor could deal with newlines in whatever way is needed by using the proper Unix utilities.

So, I propose we allow some kind of flag to be set for variables to let the clients know to special-case these variables to be shown like this (in case of the web-client, using a textarea form field):

One way to do this, would be to make this change:

struct Variable {
    key: String,
    description: Option<String>,
+   multi_line: bool,
}

In the server's GraphQL interface, we'd make this field optional, and default it to false.

Another solution would be to actually start using variable types for this, not too different from how you'd use varchar and text in SQL:

+ enum Type {
+     String,
+     Text, // to be bikesheded
+ }

struct Variable {
    key: String,
    description: Option<String>,
+   type: Type,
}

This approach seems more flexible, and more future-proof, as it would eventually allow us to natively support boolean variables (which the web-client could them interpret as a checkbox field), and others, which is on the long-term roadmap, but perhaps there are downsides to this that I haven't considered yet.

Similar to the other solution, in the GraphQL interface I would make the type input field optional, and default to String, as that's the most common case.

When #19 lands, it will be something you can opt into (eventually, it might be mandatory at first).

By allowing the clients to request the enabled server capabilities, it can show or hide specific features.

For example, if there is no authentication and server-side user-state enabled, clients can store the user state locally (in a cookie, for the web client), to still provide the functionality described in #9 and #10, without supporting cross-device session state.

It'd be nice if you could use a template function add links to other tasks in your task output.

For example, say a task shows a list of customer capabilities, it could then show it as this:

Capabilities:

• is_awesome _revoke
• has_pizzazz _revoke
• can_💃🏽 _revoke

With the revoke links linking to another task that allows you to remove capabilities from a customer.

The link would have to support the following:

• Link to an existing task.
• Optionally pre-fill variable values.

I don't know what the best way to do this is, but perhaps links could look something like [revoke][automaat:task id:12 variable:Var Name=value here variable:Variable 2=value]

• [revoke] will be the link text.
• automaat:task is a special prefix to let the parser know it should link to an existing Automaat task (we could add more automaat:... links later, if required, for example linking to search results).
• id:12 is the ID of the task to link to. We could instead use task names here, but then renaming a task would break this.
• variable:name=value is an optional (repeatable) pre-filled variable value.

(see next comment)

Knowing how often a pipeline has ran, is useful information to keep the list of pipelines to a minimum, so that search results don't get cluttered with useless pipelines that someone used once, but never since.

The server already persists each task (which is basically a copy of a pipeline, stored forever as the exact state of the pipeline during that run), and those tasks have a weak reference to the pipeline they were created from.

Getting that information is cumbersome though.

I think it'd be good to add a counter cache to the pipelines table, to keep track of this data. We can then add a runsCount field to the Pipeline GraphQL object.

There's no need to make this visible in the UI, as it's only relevant for someone doing "maintenance" on the system, and can be easily queried by using the GraphQL UI included with the server.

As for the counter itself, let's just add a trigger to the database, see:

• https://www.postgresql.org/docs/current/sql-createtrigger.html
• https://stackoverflow.com/a/24876687

Add the possibility to "favourite" one or more pipelines, for more convenient access.

There are two ways to go with this:

1. We could simply make favoured pipelines be distinct in their representation, and always show them as the top search results

2. Or, introduce a second column that is shown next to the default results, so that more pipelines are visible on screen.

For now, I think the first solution is the simplest to implement.

Similar to #8, the biggest hurdle right now is that the server is session-agnostic, so it cannot know which pipelines you've marked as favourites.

I think for now, storing this locally is fine, it just means that we'll have to sort the list of pipelines client-side, but we're going to have to do that for #8 anyway, so that's not a big deal.

Often times when a pipeline ran, you either want to use that result in a new pipeline, share the results with someone else, or if an error was returned and you don't know how to proceed, ask someone to help you out.

To make these situations more convenient, it makes sense to add a "copy to clipboard" button to the output results, so that you can share them.

When copying the contents to the clipboard, we should convert the raw HTML into Markdown, as it makes more sense to share that format in chat messages, etc (and the original pipeline output is in markdown, so we can be sure that the reverse conversion will result in acceptable markdown data).

Right now, the schema is generated by (manually) copying the output of the GraphQL schema API based on the code.

It would be nice to reverse this, using some codegen to build the proper objects based on an existing schema definition.

See: https://github.com/davidpdrsn/juniper-from-schema

New implementations have been tagged on for some time now, and while doing so, new insights have been gained. It makes sense to do some clean-up in the near future before cutting a 1.0 release.

Some thoughts:

• Split off models into their own model directory (similar to what was done for the web-client in 2d9bc81).
• Enable same linters as in web-client, and fix any issues (such as missing documentation).
• Split off processor deserialisation and template parsing into its own module, to make it easier to test.
• Speaking of tests, extend the test coverage (severely lacking right now).
• Improve worker logic, look into using an existing worker library, and support parallel running of jobs. solved in 6de4c96
• Improve Diesel ORM integration to properly compose SQL queries, reducing the need to pass around a connection object, and reducing N+1 queries.
• Remove the Content-Length header work-around, now that actix/actix-web#930 is fixed.

Instead of this:

struct Variable {
    key: String,
    description: Option<String>,
}

Do this:

struct Variable {
    key: String,
    description: Option<String>,
    default_value: Option<String>,
}

This can then be used in the UI to pre-fill a value. Combined with #1, this would also allow you to pre-select a specific value from a set of acceptable values (for example, always selecting production by default, from a list of environments).

Right now, when you don't provide a value that's required, we return an error, I think we should keep doing that, and don't swap it for a default value, if one is available for a variable, as that only confuses things.

So, this is purely a cosmetic aid to the UI, not a property that is magically used in lieu of an actual provided value.

Building on top of #19.

Once people can/have to authenticate themselves, it becomes possible to segregate people into groups with different access rights.

Similar to the linked issue, this is not about keeping outside people out, it's about having a simple way to give people within your organisation access to the exact pipelines they need, or keep some of the more destructive pipelines limited to a small part of the organisation.

The goal is to make it easy to restrict people from having access to one or more pipelines (denied list), or the reverse, to give them access to a subset of pipelines (allowed list).

For this to work, there needs to be a common type to base these access rights on.

One option would be to allow adding tags to pipelines, which can then be used in these allow/deny lists for access control.

I don't know what the right solution is, but I do think it's important to keep it as simple as possible, but also give us some flexibility for the future (that's why tags could be a solution, as it allows more useful patterns in the future based on these primitives).

We currently have a Shell Command processor that is easy to use for any simple shell command you want to run.

Its simplicity comes with limits in its capabilities though.

For example, you can only run a single command with zero or more arguments:

{
  "shellCommand": {
    "command": "echo",
    "arguments": ["hello", "world"]
  }
}

But you cannot pipe data from one command to the next, without using two processors:

{
  "shellCommand": {
    "command": "echo",
    // can't pipe output of `echo` to `grep`
    "arguments": ["hello", "world", "|", "grep", "world"]
  }
}

The simplicity of this processor make it a valuable tool for building simplistic tasks, but its limit prevent – or at least make it harder than needed – the construction of more powerful tasks that require shell access.

On top of that, even though having access to #23 is great for building tasks, you are limited in templating support in this processor, because each template's scope is limited to a single processor configuration value string (so echo, hello and world are each their own template scope in the first example, see also 858d44d).

By introducing a new Shell Script processor, we can keep the simplicity of the shell command processor, but also add a more flexible "free form" processor that also supports more extensive use of the templating support in Automaat.

The processor would be defined like this (mostly a copy/paste of the shell command processor, with a few changes):

pub struct ShellScript {
    /// The contents of the shell script to execute.
    pub script: String,

    /// The _current working directory_ in which the script is executed.
    ///
    /// This allows you to move to a child path within the [`Context`]
    /// workspace.
    ///
    /// If set to `None`, the root of the workspace is used as the default.
    ///
    /// [`Context`]: automaat_core::Context
    pub cwd: Option<String>,
}

You can then define your processor like this:

{
  "shellScript": {
    "script": "#!/bin/sh \n echo 'hello world'",
    "cwd": "path/to/data",
  }
}

The downside to this is that multi-line strings in JSON templates aren't great to work with.

I don't think it makes sense to help with this in the processor itself, but there are ways around this, for example by defining your processor configuration in YAML, and then converting them to JSON before submitting the processor configuration to Automaat using its API:

---
shellScript:
  cwd: path/to/data
  script: |
    #!/bin/sh
    echo "hello world"

$ yq read hello-world.yml --tojson | jq

{
  "shellScript": {
    "cwd": "path/to/data",
    "script": "#!/bin/sh\necho \"hello world\"\n"
  }
}

Some open questions:

• Do we need the cwd configurable? We have it for the shell command processor, but in that case, you are using existing commands, that might not work unless you run them in a specific location. In this case, you are building your own script, and so your script can just cd into the appropriate folder. If it does turn out to be useful, we can always add it later as a non-breaking change.

• Do we need to validate that a shebang is provided? Do we maybe want a separate configuration that has an explicit type to set the shebang by setting the script type to something like sh, bash, or ruby? Feels more restrictive, but also requires less work to validate that the script works.

This will build upon #19.

One of the things I dislike about the current authentication model is that it's not well suited for the purpose it's supposed to fill.

Specifically, the authentication implementation is not meant for outside access restriction, but is purely meant to add some more control and auditing to internal Automaat access within your company (and to eventually segregate who can run what task, with #21).

With the current authentication model, if you visit your company's Automaat instance, you're greeted with this giant display of "NO ACCESS", and you have to jump through hoops to even be allowed to peek behind the curtain to see what this tool could offer you in terms of tasks built by your colleagues useful to your daily work:

I'd much rather see a more fine-grained implementation that allows you to browse the available tasks (and potentially use tasks that can be run without authentication), and then when the time comes that you want to run a task that requires certain access rights, you are asked to provide the correct credentials.

This requires some fine-tuning of the authentication implementation, and also requires a more central role of authentication and access rights throughout the web client code-base.

Here are some thoughts:

• The server API no longer implements a blanket "error unless authenticated" approach, but instead:

  • allows unauthenticated queries to fetch task details
  • allows unauthenticated job scheduling by default
  • supports setting job scheduling access rights for specific tasks (which requires both authentication, and optionally a set of authorization configurations for your account)

• The client allows viewing/opening tasks

• If a task requires authentication to run, and you are not authenticated, the usual "Run Task" button changes colour, and let's you know you need to authenticate (clicking the button allows you to log in), something like this:

  

• If you are authenticated, but lack the proper privileges, this is also explained via the button (in this case, clicking the button doesn't work, as you need to ask someone to give you the correct privileges to make things work):

  

• Other than this, the client will also need to take the availability of session state into account when implementing #9, #10 and the likes. Meaning, when authenticated, these functionalities should be available, but when not authenticated, they are either hidden or provide some reduced functionality (tbd.).

Right now, the http server also handles the execution of scheduled jobs.

This works for development purposes, but isn't usable in production scenarios.

Some problems with this are:

• Right now only one job can run at a time. If a job takes a minute to run, other jobs will have to wait for that one job to finish.
• If a job is running, and the API server is restarted, the job is terminated while running, and it won't run again.

Ideally, in a production environment, you would run the worker and server as separate processes, so that you can isolate them from each other, and for example choose to run the worker in a more durable environment with less chance of service interruptions.

In the end, it would be nice to have one binary automaat which you can start using automaat server or automaat worker, and you'd either start the API server, or the job worker.

But for starters, I'm going to create to separate binaries (automaat-server and automaat-worker), both living in the automaat-server crate.

As for the worker itself.

• It will have a connection to the Automaat database to fetch and update job details.
• I'm thinking of using Postgres' LISTEN/NOTIFY pub/sub system to know when to start a new job (see also tokio-postgres' poll_message).
• We'll add a DB migration that adds a trigger to the jobs table, triggering a notification when the table changes.
• The worker will check for any pending jobs on startup and run them, and will from then on listen for a notification of new jobs waiting to run.
• To start, each worker handles a single job at a time, to handle more jobs concurrently, you start multiple workers.
• A single job is fetched from the database, and marked as running in a transaction, to allow other workers to fetch other jobs.
• Once a job finishes, its status is updated (failed/succeeded).
• After finishing a job, the worker checks for another job, if none is found, it sleeps, waiting for a database notification.
• If a worker is shut down while running a job, it tries to update the job status to Cancelled.
• A job can be cancelled in-between task steps, but not while running a specific step (for now, to support this we might have to change the Automaat processor interface to take a cancellation context, so processors themselves can implement cancellation within a step).
• If a worker is interrupted without cleanly exiting, a job might stay stuck in the running state, in such a situation, we'd probably want some kind of GC behaviour that updates the status of a task to Timeout if its status stays unchanged for longer than x minutes.

Currently, once you start running a task, you cannot close that active task until the job running that ask is completed. There are several places in the code that uphold this invariant, such as here:

This provides a less-than-ideal user experience, and also makes the code more complicated, as these guards have to be upheld for things not to break.

With some of the improvements being made in #20, I think we're close to lifting this restriction.

The goal of this feature is as follows:

• You are allowed to close tasks with running jobs.
• Jobs will continue to run in the background.
• If you re-open a task with an active job, the in-progress indicator is shown in the task details.
• It is still not possible to run more than one concurrent job per task, this is mostly a UI constraint (it would require a new UI element to list all running task jobs and their results), but also to keep things simple, until we can identify a sufficiently relevant use-case that warrants making this possible as well.

The biggest change that will have to be made, is to switch from a RefCell to a Mutex, so that concurrent jobs can mutate the application state whenever they receive a result from the server.

Other than that change, I think we're already very close to making this possible.

This would improve the story around variables, similar to #1 and #3.

Let's extend variables:

struct Variable {
    key: String,
    description: Option<String>,
+   example_value: Option<String>,
}

Which allows us to turn the current situation:

Into one where we have an extra (optional) tool in our toolbox to make the purpose of the variable as clear as possible to the user:

In this case, the example_value is set to hello world, and is shown as e.g. "hello world". If no example value is set, the field is kept empty.

If a default value is set (see #3), it takes precedence, and no example value is shown (unless the default value is removed, and an empty field is left, in which case the UI would put back the example value).

Note that the above example uses the placeholder attribute, so it's not actually a value that can be used as the input value.

Processors are pre-defined sets of code combined with a set of configuration options. The name of a processor, combined with its options is serialised and stored into the database when a pipeline is created, then upon running that pipeline, the stored data is deserialised back into an actual processor that can be executed.

The downside to this design is that deserialising into the original processor might fail, if the type signature of that processor changed after it was serialised to the database.

To mitigate this, a processor should be versioned, and that version should be part of the serialised information stored in the database.

Any change to a processor that breaks deserialisation (meaning, a change to the type signature) would require a version bump. Technically, runtime changes (so, what a processor actually does when it runs) is not a breaking change for deserialising, but that too has to be handled with care, in order to not break existing pipelines.

There's still some thinking to do on how this impacts the API, as this would also have to be conveyed somehow via GraphQL.

Until this is solved, whenever a processor is changed in a backward breaking way, a query has to be executed to update all records that serialised this processor, to modify the data to make it deserialisable again.

So, until this is solved, let's keep backward breaking changes to a minimum 😅

This is related to #8, but different.

We should implement the Levenshtein distance algorithm to support a more natural "fuzzy matching" pattern in the search bar.

For example, right now, having a task called find user uuid will match when searching for find user, but not for find uuid, it should.

Postgres has native support for this in the fuzzystrmatch module.

Here's an example on StackOverflow on how to make this into a query:

CREATE EXTENSION fuzzystrmatch;

SELECT * 
FROM pipelines
WHERE levenshtein(name, 'find uuid') <= 3 OR levenshtein(description, 'find uuid') <= 3
ORDER BY levenshtein(name, 'find uuid'), levenshtein(description, 'find uuid')
LIMIT 50

We'll have to play with the edit distance for both fields to see what gives the most natural results.

At first I wanted to focus on providing a UI for using Automaat, while using the GraphQL API for managing its state (during development, and in production).

But, given that the user-facing UI is in a pretty good state right now, and as development progresses, testing is slowed down by having to manually use the GraphQL API to set up certain states, I think it makes sense to introduce a very simple admin UI.

The current thinking:

• Add some kind of way to enable "editing mode" in the UI
• When on the home screen, in editing mode, there's a + sign somewhere to create a new task.
• Similarly, on the home screen, any tasks shown have a red X sign to delete tasks (with confirmation dialogue).
• When visiting a task, there's a way to switch to "editing mode" there as well.
• In this mode, you can change the title, description, and edit variables.
• This is the same view/mode you get into when you use the + sign to create a new task.

Most of this is already supported in the GraphQL API. Probably the most work will go into 1) implementing the UI elements, and 2) dynamically creating processor configurations based on the API types from GraphQL.

All of this is done without any support for authentication/authorization, so anyone can do this.

Eventually, this will be merged with #19 to limit who can activate this editing mode.

see #20 (comment) for the most up to date design decisions

Solve the general problem of "this pipeline requires me to provide a variable value, that I don't have yet" by allowing pipelines to "expose" their capability to provide output that matches the required input of a different pipeline.

Example

Let's take this example:

Say I want to use the "List Feature Flags" pipeline (second one), and it requires me to provide a customer UUID. However, I currently only have a customer email at my disposal.

Let's imagine there to also be a "Find Customer UUID" pipeline that accepts either an email address or a customer name, and returns the matching UUID.

I can now:

1. open the Find Customer UUID pipeline (this is assuming I know this pipeline exists),
2. enter the customer's email address,
3. run the pipeline,
4. copy (#17) the returned UUID output,
5. close this pipeline,
6. open the List Feature Flags pipeline,
7. paste the copied UUID,
8. and run the pipeline.

Proposed Solution

It would be great if we can programmatically proof that there is a relationship between these two pipelines, by adding some extra metadata to the output of pipelines, which can be linked to the input needed by other pipelines.

One possible solution I'm thinking of:

• When creating a pipeline, you can optionally name the output of each step in that pipeline.
• The output of a pipeline equals the output of the last step in a pipeline, and so the output name of a pipeline equals the one used in the last step
• When opening a different pipeline, and it asks for a variable named Customer UUID, the system will search for one (or more?) pipelines for which the output name is set to that same name.
• It will then suggest to you this other pipeline
• What happens then in terms of UI, moving back and forth between pipelines, I'm not sure about.

Here's an example of how such a help message could be displayed. In this case, the field is for the Customer UUID variable in the "List Feature Flags" pipeline, and there is another pipeline called "Find Customer UUID" which has its output name set to Customer UUID:

side note on the second point in the proposed solution:

We can optionally simplify this part by defining the output on the pipeline itself, instead of the steps and then picking the last step in a pipeline. However, this is error prone, because step ordering might change, or new steps might be added, at which point the output might no longer match.

I think it's safer to set the output names on the steps themselves. This also provides more composability in the future, when we work on a UI for creating new pipelines, using step templates, for example.

Design Goals

The above proposal is one possible solution, but there are more. In general, I'd like to try and find a solution that matches these goals:

• composable – using existing features (or features already proposed elsewhere), little to no new concepts/data structures required
• simple – fits well in the UI, is not confusing to use, should be self-explaining
• generic – should solve the overarching problem of "I need to provide a value that I don't know yet"
• safe – no automated pipeline runs without explicitly triggering them

Right now the API only supports creating tasks. In a continuous deployment setup of Automaat, you'd want to be able to (for example) have your tasks defined as YAML in Git, and then deployed to your Automaat instance using your CI pipeline once a task change is pushed to your base branch.

I think it makes sense to use the OnConflict object that already exists for creating/updating global variables:

For tasks, there are some considerations, that will probably require an extra OnConflict directive.

Here are the thoughts I currently have:

• The same conflict directives (ABORT, UPDATE) apply to tasks.
• The default if unset is the same as global variables (and the same as the current behaviour of tasks), which is to abort.
• Updating a task means that the following should happen:
  • The task is matched using the task name (which is unique).
  • Any mutable task properties are updated (everything except its ID and name*)
  • All steps matching the task step names are updated.
  • All steps with new task step names are added.
  • All task steps in the database that aren't part of this mutation are removed.
  • Same thing happens for variables.
• In addition to the existing two directives, I think it makes sense to add an ARCHIVE directive, that works specifically for tasks (for now).
  • It's purpose is to keep a record of previous tasks and their configurations.
  • When this directive is given, an existing task with the same name has its archived flag set.
  • This task is no longer returned in API responses.
  • It is also no longer invalid to have two or more tasks with the same name, as long as only one is non-archived.
  • Other than that, this directive creates as new task, as if it didn't exist before.
  • The upside of this is that we can keep the job -> task connections for archived tasks.

Another solution to the above point would be to simply remove the relationship between jobs and tasks. However, that would mean we can no longer show a history of jobs for any given task. The reference is already weak, so tasks are allowed to be removed without removing the jobs they created, this just means you get a null value back for the reference from a job back to a deleted task.

One more point to cover when we allow updating tasks is that it can no longer be assumed that the configuration of a job matches that of a task (meaning, a job could have been triggered when a task had different variables or steps configured). This is fine, but I think it makes sense to at least make sure there are created_at/updated_at fields available, so that we can easily detect if a task was changed after a job was created, which could be used in the future to signal that the representation of a job might not match that of an existing task.

All of this could also be avoided by making tasks immutable, and simply creating new ones and archiving old ones for every change you make. But I think that's a bit too extreme of a solution in this case (something like event sourcing would help here, but that's way out of scope).

_{* because we are re-using the createTask mutation, we can't ask for a task ID because none might exist yet, so we also can't rename the task. We might also supply an updateTaskName mutation that takes and ID, so that you can change the name as well, but I'll leave that out of the scope for now. You can always rename a task using the database itself, until we find a nice way to use this API.}

Cloning a repository in a pipeline can take quite some time for bigger projects. Currently, there has been only one reason to clone a repository: to use one of the files in that repository. Using historical data hasn't come up yet, so we can significantly improve the clone speed by doing a shallow clone instead.

In the future, we might want to make this an option, but I don't think that's warranted right now.

see: https://git-scm.com/docs/git-clone#Documentation/git-clone.txt---depthltdepthgt

Compilation times have become significantly worse due to the fact that the worker loads all processors, which by their very nature all depend on different crates, ballooning the total number of dependencies to 400+ crates.

The server doesn't do anything with these processors, other than needing to know their type definitions so that it knows how to expose the processor configurations via GraphQL.

I think it makes sense to start splitting all of this up, but I haven't yet come to a good design.

The worker and server depend on the processors for different reasons:

• server depends on all processors to know their GraphQL input types
• worker depends on all processors to run them

Aside from that, both server and worker need typed information about the database, as the server uses it to push new jobs, and fetch job status based on GraphQL requests, and the worker fetches pending jobs and pushes job updates to the database.

So if we were to split all that up into separate crates, so that they don't have to be re-compiled all the time, you'd have to end up with something like:

• server – GraphQL API
• storage – types related to storing and fetching data
• worker – run jobs
• processor-types – a set of processor type signatures
• processor-shell-command/.../... – all the different processor implementations

In such as situation, the server would depend on storage and processor-types, but not on all the actual processor implementations, and also not on the worker.

The worker would depend on everything except the server itself.

This is still not great, as now creating a new processor involves not just writing one, but also adding it to the list of processor types in another crate, making it less easy to create a processor and add it to your set-up (which isn't possible right now either, since processors are compiled into the server/worker binaries).

Another thought I had was to actually change processors to become binaries, and have the server and worker communicate over RPC, which would solve most of these issues (and would allow processors to not be built in Rust), but would add an extra layer of complexity to the cross-binary communication, it would also reduce type safety.

One way to do that would be to have something like this:

• when starting the server, you pass in a set of strings, representing the processor binaries you want to "enable"
• on start-up, the server runs these binaries with some kind of signature argument to ask each processor for their type signature
• the server then uses this type signature to configure GraphQL (however, the current GraphQL library we use doesn't support dynamic schemas), and to serialise the data before storing it in the database
• When the worker needs a processor to do some work, it runs its binary with the correct data passed in.
• this would already work reasonably easily, since we're only passing in JSON-serialised data to the processors as input, and get a string (or error) as output back, which translates well to passing in some JSON-formatted string to the binary, getting back an exit code + string output.

Still, it's quite some work, and there are still some gaps (such as dynamic schemas).

With the addition of #20, and others like it that will follow, there will come a time when we want to programmatically run tasks, without manually clicking the “run task” button.

Supporting this would make usage of Automaat more streamlined and it allows to get results faster.

The biggest blocker for this feature is that you cannot be sure that a task won’t mutate state or have other side-effects that are unexpected.

One way to tackle this would be to add a special flag on tasks (or steps, tbd), that’s signals wether a task is a query or a mutation.

If a task is a query (and should thus be side-effect free), it can be programmatically run.

There are still lots of design considerations to make on this, but creating this tracking issue is at least a step towards solving this.

I haven't thought this through yet, but I've come across some real-world cases that would've benefited from conditional steps.

It would add more complexity to the system, so it's still debatable if the trade-offs are worth it.

As for if we were to do this, the cases I've seen require some way to determine "if the output of step A matches value B, then run this step".

So something like:

steps:
  - name: Step A
    description: ...
    processor:
      printOutput
        output: "hello"

  - name: Step B
    conditional: '{{ output["Step A"] == "hello" }}true{{ endif }}'
    description: ...
    processor:
      printOutput
        output: "world"

Basically, if the conditional field is anything other than a blank string, the step will run. This allows for using the existing templating support (#23) to decide if a step should run.

We'd probably also want to add a Skipped state to the existing step states enum, to make it clear if a step ran or not.

But again, this is very preliminary, we'll see if this really makes things easier, instead of too complex to work with.

To give one example that I came across (with some fictional details):

1. A task takes a variable X, that variable can be a URL to either a post, or a user page
2. if it's a user page, fetch the url to the latest post, then go to step 3, otherwise skip this step
3. fetch the post details

Right now, each step output is assigned to a special variable {$input}, which can be used in the first step after the current step.

While this makes things easy to understand (data flowing out of a step can be used to flow into the next step), it also limits the possibilities of more complex pipelines.

Say for example you have a step that requires input from two other steps, this is currently not possible.

I propose we make the current behaviour the default behaviour, but add more flexibility to the system by allowing steps to define an output attribute, which determines to which temporary variable the output is assigned.

By default, this attribute is set to {$input}, but it can be assigned to any other variable name, and that variable and its attached value live for the duration of the pipeline unless overwritten by the same variable name in a future pipeline step (which conceptually matches what happens with {$input} right now, it is overwritten by each following step).

Using this technique, you could solve the above problem like so:

1. run step 1, assign output to the {step 1 output} variable
2. run step 2, output is implicitly assigned to {$input}
3. run step 3, use {step 1 output} and {$input} wherever they are needed

This will still need some more technical thinking on how it actually works in the server, but conceptually this fits the current model of pipeline variables and the special {$input} variable well, and it unlocks a much wider array of capabilities for complex pipelines.

Note that there's one rough edge that I think we should just accept, shown in the following example:

1. run step 1, no special output assignment (implicitly assigned to {$input})
2. run step 2, assign output to {step 2 output}
3. run step 3, use {$input} and {step 2 output}

In this case, you could reason that, because we explicitly assigned the step 2 output to {step 2 output}, the special {$input} variable wasn't overwritten, and would still contain the step 1 output.

However, I think because {$input} is a special variable (and is even marked this way using $), we should just always assign the last step output to that variable, and so in the above example, in step 3, both {$input} and {step 2 output} would contain the same value, the output of step 2.

We might also want to rename {$input} to make its meaning more clear, but I haven't thought of a better, concise name yet. Perhaps just renaming it to {step output} makes it more clear, and also makes the above wrinkly moot, as that variable would then indeed contain the step 1 output when used in step 3 in the above example.

Also, perhaps these temporary variables should use a different notation than regular variables, to avoid name clashes. One possibility would be ${step output} (implicit) and ${my step output}.

The current (uncompressed) file size of the wasm binary is ~2MB (about 800K compressed):

stat -f '%N %z' src/web-client/static/app/automaat_bg.wasm
src/web-client/static/app/automaat_bg.wasm 2077976

This is with release optimizations as defined in Makefile.toml:

And Cargo.toml:

Most of this is "infrastructure" code, meaning it is there even if we remove a lot of functionality, so there's no risk in getting to a 10MB binary, but there is still a lot to be gained here by trimming on the required infrastructure code (by preventing panics, and reducing the "string format" infrastructure).

I think the following steps should make this situation better:

• split off the web-client crate from the workspace and make it a stand-alone crate
• this allows us to use panic=abort to remove some of the embedded panic infrastructure
• look into combining no_std with the alloc crate to force removal of code that bloats the binary (enforcing no_std might be hard, given some of our dependencies)
• change unwrap and expect to unwrap_throw
• follow this guide to further reduce size
• there's also this guide specifically for wasm
• we should also look into pre-compressing binaries, so they don't have to be compressed on the fly on the server

In general, web-client should favour size over speed (since the final code will still be more than fast enough), and use unstable features to do so, if required.