Exec command: Order and repetition of jobs about bootique-job

[andrus]

Another recent idea is to define job dependencies in JobMetadata and do the automated ordering via topological sorting of the dependency graph.. Worth a discussion...

[atomashpolskiy]

I'd say the most simple and straight-forward solution would be to add an extra option for --exec command, which will trigger sequential execution. @andrus , you OK with this?

[andrus]

I don't think it even needs to be an option. It can be the default and only behavior.

[atomashpolskiy]

Seems like it's kind of orthogonal to your own design (see ExecCommand and your comment). Why use a Spring scheduler then? Second thought - why not just run several execs one by one?

…

On Thu, 8 Dec 2016 at 13:33, Andrus Adamchik ***@***.***> wrote: I don't think it even needs to be an option. It can be the default and only behavior. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#15 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADk0hszOvWrP-NU-AOh4SUeo-n2KPj4kks5rF9zdgaJpZM4LGiGr> .

[andrus]

you are right ... parallel execution is the default. Forgot about that (I only focused on the order of command line flags). So yeah, a "--serial" option is a good idea.

[atomashpolskiy]

As discussed, we're going to add a notion of "job groups". A job group is a collection of jobs, which may (optionally) depend on each other and should be executed as a batch. The particular order of execution is going to be determined by Bootique based on the job group definition. Job group definition is basically a directed acyclic graph and may be represented in YAML in the following way:

jobs:
  j1:
     param1: x
     param2: y
  g1:
    type: group
    jobs:
      j1:
        dependsOn:
          - j2
          - j3
      j2:
        dependsOn:
          - j3

Running a job group is similar to running a single job:
--exec -job=g1
Or --g1, when we switch to using jobs as commands.

[atomashpolskiy]

After taking a closer look at Jackson's deserialization possibilities, it looks like it's not possible to deserialize this kind of config natively. The reason being that Jackson can't "unflatten" j1's params. This would not be a problem if we used an explicit params property, like:

jobs:
  j1:
    params:
      param1: x
      param2: y

As a bonus, this would also allow us to:

1. Specify params for elements of a job group

jobs:
  g1:
    type: group
    jobs:
      j1:
        params:
          p1: x
          p2: y
        dependsOn:
          - j2
          - j3
      j2:
        dependsOn:
          - j3

2. Specify "default" dependencies for stand-alone jobs:

jobs:
  j1:
    params:
      param1: x
      param2: y
    dependsOn:
      - j2
      - j3

3. Combine (1) and (2) to provide "default" job configurations to make job group definitions more compact

@andrus , let me know your thoughts

[andrus]

let's do it. Please update the central UPGRADE.md so that the users are aware of this change

[atomashpolskiy]

@andrus , our idea turns out to be really AWESOME! Can't stop generating ideas, my todo list is growing :) But even the basics that I've managed to implement today already solve Tony's problems with 14 yaml configs. See config example and ExecutionIT test in my recent commit.

[andrus]

@atomashpolskiy : I just added an upgrade note to https://github.com/bootique/bootique/blob/master/UPGRADE.md for this task with a note on 'jobPropertiesPrefix' removed property. If there's anything else we need to mention about upgrading existing job configs or code, let's add it as well.

[atomashpolskiy]

Yep, one thing that immediately comes to mind is that params should be declared in params section now. I'll add this info as soon as I have time