Feedback/Ideas for Improvement about modularpipelines

[thomhurst]

Thanks for the feedback @joshslark !

For estimated time remaining, absolutely. In the next version I release there'll be an out-of-the-box provider that will persist module times to the file system, where it can retrieve it on subsequent runs. There'll also be the option to configure your own implementation, where you can persist elsewhere, such as an external database, or a distributed cache, so different build agents can read it if you need.

As for the submodule progress, this isn't really do-able from how I'm thinking currently. I've no way to know how many iterations you're going to be doing, or that each "solution build" is a "submodule" as you say. They're just bits of code being executed. Thoughts?

[joshslark]

I guess what I'm thinking is that there should be some kind of API for the progress printer so it can be customized. I'd like to be able to communicate that I have a collection of tasks (submodule) to work on, that a task has started, has finished, etc.

And then for the estimation part to work with submodules, there would need to either be a way to communicate details needed for the implementation you have in mind or I could just communicate an estimated time it will take for a task.

[thomhurst]

The estimated time will just work inheritantly once the estimated time provider is implemented though. I think reporting times yourself is overkill, as we can just solely use the history?

[joshslark]

That makes sense, I suppose as long as there's some way to specify submodules, you could store a history of time taken to complete and draw from that. I was also thinking that if you do incorporate submodules, you could sum up the estimated times for the submodules so that you can get more accurate timing if the module executes a subset of a previous run's submodules.

e.g An outcome something like this

First Run

Build 0:00/8:00
-Solution1 0:00/5:00
-Solution2 0:00/3:00

Second Run

Build 0:00/3:00
-Solution2 0:00/3:00

[thomhurst]

I'm not sure I'm keen on trying to maintain executable bits of code as sub modules. Feels like it'd complicate things quite a lot.

Could you not build each solution as it's own module? If you don't want them to run in parallel then you can make one module depend on another. You could then have a module at the end that collects all of their results (if you need that).

[joshslark]

Yes, understandable, I don't want to insist on anything that will make it hard to maintain the project. However, I would just say to keep this scenario in mind, because having to define every module as a class is constraining. For example, I can build each solution as it's own module, but the issue is that you can't do it dynamically. You have concretely specify every solution as opposed to searching a directory for solutions and making it more flexible.

I've been busy lately, but if I get freed up, I'll see what I can come up with and pass it over for review.

[thomhurst]

So I've just had a little go and it presents another problem.

The module progress logger is set up at startup, and because the Module types already exist in the Type system on start up, we can locate and register them no problem.

However since a SubModule wouldn't be a Type, but a delegate that is only constructed during the runtime after execution has begun, this information isn't available to plug in to the progress logger on startup.

I could possibly register them later on, but then it's all going to be out of order and isn't going to present nicely.

If you can think of a way around that then I'm all ears.

[thomhurst]

So the real limitation for this currently is the library I'm using for the console progress output. I've actually gone and raised a PR that would unblock this, so watch this space.

[thomhurst]

@joshslark I've had a PR open for Spectre.Console for 3 months now and I don't know when it's likely to get looked at.
In the mean time, I've had to do a bit of a hacky workaround with reflection, but I've got submodules printing underneath their parent modules in the progress printer.

You can find this in the latest version.

I've also made tonnes of other changes, breakages and upgrades since this thread was started.

If you've got some time, I'd be really keen to hear what you think of it now, and again, any ideas you might have.

Thanks :)

[joshslark]

Hi @thomhurst, sorry I haven't had much time to try out the new features, but I read through the documentation and it looks quite promising! My only recommendation until I get a chance to try it is to put the link to the documentation at the top of the Readme.md so it's easier to find. It's really well written and one of the problems I had trying out the project the first time was the lack of documentation. I'm looking forward to using it when I get a chance!

[thomhurst]

Thanks @joshslark , and good shout. Have added it to the top of the readme