stormsurgelive / asgs-mon Goto Github PK

Integration with asgs_main.sh

• asgs_main.sh PID find:

ps aux | grep [a]sgs_main.sh | grep "$ASGS_CONFIG" | awk '{ print $2 }'

• implement support for the following

# snippet of my proposed $ASGS_CONF, with some context variables to demonstrate what I am copying
OPENDAPPOST=opendap_post2.sh
POSTPROCESS=( includeWind10m.sh createOPeNDAPFileList.sh $OPENDAPPOST )
ASGSMON=( 000-asgs_main-pid-check 001-instance-status-check 002-hook-status-check 003-syslog-progress 004-instance-status-progress 005-hook-status-progress 006-rundir-du 007-failed-dir )

Triage the following issues as they related,

• StormSurgeLive/asgs#1291
• StormSurgeLive/asgs#1214
• StormSurgeLive/asgs#1162
• StormSurgeLive/asgs#1103
• StormSurgeLive/asgs#897
• StormSurgeLive/asgs#55

Additional plugins to create:

• "is this thing still running" for long running modes (daily heartbeat email)
• SU budget check (will be machine specific most likely)
• quota based checked
• ASGS directory "age" check (for systems with a purge)
• basic check on jobs stuck in queue

Future check ideas:

• deep inspection of SYSLOG
• deep inspection instance JSON
• deep inspection of hooks JSON
• availability of TDS servers, verify delivery of output on remote servers
• integration with a heartbeat dashboard
• check status of meteorological data and just report when new data or NHC forecast is ready/update
• integrate notifications with Slack, Discord, ntfy.sh

Additional asgs_main.sh integration could be:

1. start asgs-mon with run command, pass directly --pid $$
2. fork and detach from parent process so that asgs-mon doesn't go away
3. output a log that can be tail'd for output
4. ASGSH command that can tail -f this output log so it can be observed at will