Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
The hamburger menu item has an extremely unintuitive purpose and I'm not sure why it exists. It might make a little sense if it completely hid the side bar, but it's confusing and unnecessary as-is.
This is a bug introduced when I fixed long task lists being inaccessible. It should completely hide the sidebar. I will fix.
The workers tab did not survive the transition very well. The tasks are fairly hard to read compared to the nice layout of everything else in the new vis. It's a bit jarring.
I haven't worked on the worker tab at all. I would rather see this addressed as a future update unless it is completely broken.
Some information that was displayed in the original visualiser is no longer displayed in the new theme. In particular users have requested the re-instation of resources and priority.
It's hard to tell when something in the side bar is selected. I got confused a few times as to why my searches were giving such limited results until I realized I had left something selected. This would be more obvious if the highlight color stood out more than black on gray.
I agree, some more visual feedback on what the search constraints are would be useful. I can do that.
Same issue with disabled and upstream disabled, it's not obvious that they're selected because the highlight color blends in with the background.
I can change the colours but they look clear to me.
First, I have a couple thousand pending tasks but only 367 show up as pending on the main page. The worker page shows more than that. Did the internal representation change in a way that requires clearing the pickle?
Are any of these pending tasks upstream failures or disabled? I'm counting these independently. Maybe this is also related to the server limit (see below)?
The task page seems to be ignoring tasks that were disabled by the scheduler. They're missing when I search for them and don't show up in the disabled task count. I need to be able to see them and re-enable them with an action button.
I haven't tested with disabled tasks because my workflows typically don't use them. Providing they are easy to simulate I should be able to get this fixed.
Using D3 as the default graph is way too slow and makes the graph view unusable for me.
We know the D3 visualisation doesn't scale to thousands of tasks but one of the visualisations has to be default. It's awkward to make it configurable without changing how tornado serves the content.
Would it suffice for the visualiser to be specified in the URL fragment (http:/.../index.html#visType=...)?
My done task count is beyond the limit where the server sends them all, so I get a question mark rather than a count on the tasks page.
I've lost the ability to search within done tasks when there are too many to display. This used to trigger a call to the server to fetch tasks that matched the query but no longer does.
I wasn't aware that the server has a limit to how many tasks are returned. I'll have to take a look at how this could be overcome. Although DataTables can work with server-side queries it works best when all data is retrieved and sorting/filtering happens in the client. An extra box to "filter server side" might be needed. Is there a way to tell from the API that not all results have been returned?
The sidebar has two titles but neither one is informative and the link in the top one duplicates the Task List link. They can probably be safely removed.
The intention is that the "Luigi Task Status" title is part of the top bar, not the sidebar. I'm agnostic on whether we need the "TASKS" title or what it should say. Is "Task Family" well understood? The code uses "Task Name" interchangeably.