jeffersonlab / btm Goto Github PK

Currently the BTM vs DTM cross check simply ensures that BTM has some program recorded to match DTM Event Downtime. This specifically avoids the common scenario where BTM records OFF (SAD), yet DTM has Downtime. This check could be expanded to ensure that DTM Event Downtime roughly matches BTM Physics + Internal Down. This improved distinction is possible since Internal Down is tracked explicitly, unlike Physics Down, which is included in the Physics metric.

Currently the cross-checks compare shift totals side-by-side, but it would be useful to have an hourly side-by-side comparison of crew chief vs experimenter time accounting numbers.

One hour before an experimenter's shift ends they are required to save all but the last hour of their timesheet such that the Crew Chief can do a comparison with their own shift timesheet, which starts and ends an hour before experimenters. The only indicator/feedback that the experimenter has completed this obligation is by examining the table of hours and looking at the Source column to ensure all but last hour read "DB". We could add some extra indicator to make this clearer. In the previous BTA app opening the "Info for MCC" summary provided the assurance. Perhaps we could add a new Crew Chief Hours item to the Timesheet status window:

Note: Crew Chief timesheets have their own separate indicator for each hall in the Comparison Section of whether the experimenter has completed this obligation (saved all but last hour).

The Run table needs to be populated and maintained in an automated fashion.

The PAC Schedule database can be examined to determine the bounds of runs, but it is complex so runs are now determined in advance and cached in the btm_owner.run table. Ideally this cache of runs is kept up-to-date in an automated way when schedules are modified. Also, the initial population of runs from the existing database would be useful. Currently, just the most recent two runs are in the run table, and they were manually inserted, and new runs can be missed if not manually added.

Historically the run determination was automated and calculated on the fly each time the /rest/runs endpoint was requested. This was costly to calculate, plus assumed all months had data (explicit OFF) for SAD times. If SAD times were not explicit the automated routine found zero results.

If the Crew Chief has set a hall as OFF when a hall device indicates ER we should override the ER (subtract OFF amount). See attached screenshot:

The machine is in INTERNAL mode doing RESTORE so halls are off. However, in this case Hall A is set to ER so the total for the hour is 120 minutes instead of 60 minutes. Plus, since the total doesn't match 60 within the threshold the rounding mechanism that cleans up the IOC counting punts (doesn't run) and we see the value is not a crisp whole number, but the raw fraction.

Hall B shift takers are required to fill out the BTM timesheet every hour at the top of the hour, and they typically don't touch the timesheet any other time; since the user session duration is less than an hour, the typical shift taker must log in eight times during a typical 8 hour shift, unless they routinely interact with the BTM webpage (e.g., refreshing it).

It would be ideal if the shift taker only has to log in once per shift; logging in can be distracting from more critical shift responsibilities, especially if the user's password is difficult to type.

Related: JeffersonLab/smoothness#15

At the end of each hour a new measured value becomes available. We need to add a Reload button to allow users to manually request the update without possibly clobbering their work if they happen to be in the middle of editing form data. It may also be possible to dynamically update the data on the form without a full page reload if the form is currently not showing / not in edit mode.

Currently there isn't an easy way to search/filter the audit report by a particular shift. This makes it difficult to find the records associated with a given shift.

The CC Cross Check reports the experimenter timesheet status inconsistently with how the experimenter timesheet status is reported when viewed on the experimenter timesheet itself. This is because the CC status line ignores signatures and shift info. It actually doesn't really matter so much to the CC if the experimenter signed their timesheet and filled out shift info - they really just need the numbers to add to their shift report. However, to be consistent and avoid confusion we should probably make the CC Cross Check consider experimenter timesheets without signatures or shift info Incomplete.

Currently the hourly update occurs 1 minute after the hour; but that means all instances of the app open around site (in each counting house) will issue a DB request at roughly the exact same time. If multiple instances are open per counting house we can run into a concurrent DB request limit (currently configured such that requests are queued for 5 seconds before timeout - and 3 are allowed concurrently). We could add a random number between 0 and 10 seconds for example to the refresh to spread the hourly refresh spike out. The current limits are fine for probably 10 concurrent users (due to queuing), but it appears users sometimes leave many more instances than that open in virtual desktops.

The charge report is not working since moving BTM to ace.jlab.org because the myquery app is unreachable. When the app was on accweb a proxy was created at /myquery to allow the Charge report to query for data (sidestepping same origin restrictions and taking advantage of on-site no-auth). This is problematic on ace.jlab.org because myquery requires auth from offsite.

A working example can be viewed internally on the soon to be decommissioned legacy wildfly2 server.

Solutions:

A. Proxy

One solution is to include an in-app restricted proxy (just the exact myquery requests needed for charge report). A few variants of this exist:

1. Maintain client-side requests, but update to use proxied endpoint
2. Perform requests entirely server-side (with long waits) and then write results inside response HTML
3. An elaborate caching setup in which a daily query to myquery for previous day runs in a scheduled task and caches results in Oracle database

The elaborate in-app caching setup has the nice feature of page load by users is lightning quick, whereas right now the first person to request a given date range must wait for a myquery request that does an integration too (1 year span takes roughly 15 seconds). Also, we're currently relying on a precarious Apache HTTPD caching configuration to keep subsequent requests to myquery snappy. On the con side the precariousness is likely just transferred from the httpd config to the app. For example the automated nightly scheduled task may fail. On each page load the count of number of records (days) would need to be made to check if the span includes all data. If not, the daily updater routine would need to be invoked to crawl the given range to fix any gaps.

B. OAUTH

If we update the httpd forms and httpd sessions auth on epicsweb.jlab.org to use OAuth we could then authenticate to the service using OAuth.

See:

• https://stackoverflow.blog/2022/12/22/the-complete-guide-to-protecting-your-apis-with-oauth2/
• https://stackoverflow.blog/2022/04/14/the-authorization-code-grant-in-excruciating-detail/

In #15 we made the cross check for BTM vs DTM downtime cross check stricter to catch more problems. The Review Calendar delegates to the same cross check, however, the highlight in-red DTM discrepancy logic is still using program time when making comparisons to determine if the day should be "red". We need to update the highlight logic to replace program time with possible downtime (Physics + Internal Down) to reflect the stricter check.

There appears to be a regression from moving from accweb.acc.jlab.org to ace.jlab.org in which the experimenter timesheet log entries no longer include the hall qualifier and no longer appear in the hall specific logbooks.

Example Hall C entry: https://logbooks.jlab.org/entry/4265704

Initial triage suggests the experimenter sign timesheet and send elog function is calling the crew chief timesheet variant erroneously:

Reported by Paul Vasilauskis

There are a few improvements to leniency that could be implemented for the Excel file parser including:

• Program names sometimes have extraneous spaces - add trim() function before compare
• Program names often are capitalized differently - add upper() function before compare
• Program names often use aliases (INSTALL vs INSTALLATION) - create import_alias_csv column?

Further, the "official" program names are often just whatever Javier originally included in the first Excel upload, but they're capitalized inconsistently - perhaps make all program names ALL CAPS or All Titlecase or all lowercase.

Official program names displayed on the schedule also include more than may be helpful / useful given audience of accelerator ops. Many activities such as Install and Commissioning map to OFF as far as time accounting goes. It may just be creating confusion as ops/ PD is required to do this mental program conversion themselves. We could map these programs to OFF automatically. Perhaps add an extra OFF_ACTIVITY column or something with the details for those who are interested.

There are other common restrictions we may be able to make more lenient such:

• Months must start at day 1 - we could make the parser automatically fill in OFF days leading up to given day.
• Spurious comments in columns that are supposed to be empty due to program being "OFF" and that normally contain specially formatted data if not OFF - this one is a little dangerous, but we could default to ignore instead of throw error as is done now.

"Pass change" is tricky with time accounting as it's treated as an ACCELERATOR activity by Crew Chiefs and as a PHYSICS activity by Experimenters - more specifically Crew Chiefs set halls to OFF during pass changes while Experimenters have a special ACC metric they are supposed to use. Probably just stays as is on the schedule.

"Reconfigure" may need clarification else leave as is. It could possibly fall under "Hall Procedure", however those generally aren't scheduled.