gchq / maestro

A framework for configurable operation executors

Maestro

maestro's Issues

updateCopyWriteHeaders.sh year is incorrect

Branch: adding-travis. If it stays or is merged.
The year is incorrect within the shell script.

Add CacheService

Can Cache service be configured in the executor instead of properties file?

Add class uk.gov.gchq.maestro.cache.impl.JcsCacheService.
Add to example config:

"properties": {
    "maestro.cache.service": {
      "class": "uk.gov.gchq.maestro.cache.impl.JcsCacheService"
    }
  }

Modify CacheServiceLoader.initialise to get the cache instance object from executor instead of creating and invoking an new instance.

Move Executor hooks from Gaffer

On the branch develop-2 there are hooks to move to Maestro that are Executor and not Graph specific.

This will also mean anything else related, such as NamedOperations will need moving.

Create hooks to enable operation execution monitoring

It would be great if hooks were available that send out application monitoring messages that can be collected in a central monitoring system. Then information about operation chains that execute across a set of federated services can be gathered and visualised in one place.

So consider an operation chain ([operationA, executor1] -> [[operationB1 -> operationB2], executor2] -> [operationC, executor3]). The federated operation chain handler is going to run through these operations one by one, delegating to the relevant executor for each operation in the chain.

If each of the executors report back information about the operations they are carrying out and the context in which they are doing it (an executor id, the job id of the operation chain, the timestamp of beginning and end of operation, for example), then by aggregating this information we can see the state of the distributed workflow in one place.

In future, this information could be mined to inform users about their operations. It would be possible to estimate the runtime of an operation chain based on previous execution statistics, for example.

The logs could either be sent to an external logging system, or collected as part of the existing job tracker. We could pass the configured monitoring endpoint between systems using the operation context.

Executor exception does not get returned in JUnit tests.

See Branch: develop-GCHQDev404 commit: 0312911 test: AddExecutorHandlerTest
Note: Handler test passes, but the execute test fails.
The execute test does not throw a unsupported operation exception it lost/hidden, when executor in the test is incorrectly missing the needed handler.

Log4j xml files copyright have wrong dates

Exceptions not thrown within handleOperation method

If there is an exception thrown within the handler.doOperation it is not thrown within the handleOperation method in the Executor.

Create InitialiseOp

We need an initialise operation that will be force run within the Executor creation to run Executor specific code

Create ExecutorPropertiesUtil class to simplify ExecutorProperties

An ExecutorProperties Util class should be created for helper getter/setters for properties. This should be remove the need for logic within ExecutorProperties, and the util class can be used for specifics.

Add simple rest api to Maestro

Add a simple rest api for Maestro to use operations

Dependency management common-util

Place version dependency in Dependency Management.

Create initial python operations

Subtask of #3.

Initial implementation will use the cache for access control and storing of python scripts.
Handlers will just run the python scripts locally.

OperationAuthoriser does not allow a list of roles to be assigned to an operation

I wish the OperationAuthoriser to restrict permission to run an operation to users who are either members of ROLE1 or ROLE2, but not ROLE3. However, if I specify an auth e.g.:
"uk.gov.qchq.gaffer.this.that.theother.myop": ["ROLE1", "ROLE2"] then the Authoriser only allows execution to users who are members of ROLE1 AND ROLE2

Copy of gchq/Gaffer#2130.

Create Python operations to work with executor

Create operations and handlers which can execute python that get run by an executor.

It must execute python using a users credentials, ideally in a containerised or lambda environment. There should be a way for certain users to install python libraries for all users to use in their scripts.

There should be a way of users storing, sharing and executing scripts.

It would be nice if there was an easy way of executing other operations on the same executor.

Executor 1 handler to many operations

Change Executor structure to allow 1 handler to support many operations with different field declarations.

This should enable 1 handler respond to an alias operation names.

This should lead to structures such as a parent FederatedHandler which supports AddExecutor, RemoveExecutor and FederateOp.

Clean up JSONSerialise Config properties

Is JSONSerialise needed required in Config Properties?

"maestro.executor.properties.class" : "uk.gov.gchq.maestro.ExecutorProperties"

Alias operations

Update checkstyle

Checkstyle currently isn't run over any test files. This should be changed so some/all of the checkstyle modules are run over the tests.

Copied from gchq/Gaffer#2144 but needs to be done in both Maestro and Gaffer.

Scheduled Jobs will not reschedule

If the REST service is restarted currently then any Scheduled Jobs will not be rescheduled.

For more info NamedOperations and NamedViews have caches that will remember these if the REST service is restarted, something similar should be implemented for Scheduled Jobs.

This is a copy of gchq/Gaffer#2126.

StreamUtil has references to Schema