pandawms / autopyfactory

PanDA WMS

This is the open source repository for the PanDA Workload Management System.

The PanDA (Production and Distributed Analysis) system was originally developed for the ATLAS experiment at CERN's Large Hadron Collider (LHC) by teams at Brookhaven National Laboratory and the University of Texas at Arlington.

While PanDA continues to be developed for ATLAS, where it is used for the massively scaled distributed data-intensive production and analysis processing of the experiment at over 100 sites globally (150k concurrent jobs around the clock, a million jobs a day, analyzing a data set currently 150 petabytes in size), it is also being generalized, extended and packaged for use by other scientific communities through the BigPanDA project supported by the US Department of Energy.

For more information see:

PanDA twiki at CERN: https://twiki.cern.ch/twiki/bin/view/PanDA/PanDA

PanDA WMS project site: http://pandawms.org

ATLAS experiment: http://atlas.ch/

autopyfactory's Issues

Continuous Integration builds

Suggest implementing a CI environment for APF builds. Use case is to deploy patches as quickly as possible.

Agis, apf-queue label name

We didn't discuss this. APF queue [section] names need to be unique so check if ce_name is ok otherwise use ce_id. ce_name can be rather verbose.

Currently constructed using ce_name but previsouly we use [nickname+ce_id].

Agis, separate default files

Add config file agis.conf with sections for separate queries. This is to allow for example different defaults for analy/production (or other categories).

Agis, logging

Excluded queues should be logging at INFO level.

Review schedplugin messages

APF-2.4 message example:

Ready:in=0;activated=18,offset=0,pending=16;ret=2;Scale:in=2;factor=0.5;ret=1;MaxPerCycle:in=1;maxpercycle=50;ret=1;MinPerCycle:in=1;minpercycle=0;ret=1;StatusTest:(not test);input=1; Return=1;StatusOffline:(not offline);in=1;ret=0;MaxPending:in=1;pending=16,maxpending=100;ret=1

What structure do these have? The monitor needs to tokenize this so I suggest these changes:

1. semi-colon to separate scheds
2. change ; to , within a single sched
3. StatusTest Return change to ret (to be consistent)
4. make consistent the comment part eg. (not offline)

Or can this be improved further?

Common wrapper pilot STATUSCODE

This issue belongs in the wrapper repo when it gets onto github.

Reminder to check if wrapper signals the pilot statuscode to apfmon.

Empty batchinfo

Occassionally the sched plugins report batchinfo = None. An example in APF-2.3.1 when using Condor:
ReadySchedPlugin.py:39 calcSubmitNum(): Missing info. wmsinfo is WMSQueueInfo: notready=590, ready=52, running=4658, done=0, failed=0, unknown=0 batchinfo is None

This is a showstopper for the scheduling and no pilots get sheduled. I just restarted the factory and pilots are flowing again but only after the queue config was refreshed. I think my question for this issue is how can we clarify whats going on? Under what circumstances is batchinfo=None?

Proxy manager warnings

Check exit code of voms-proxy-init command and log (loudly) if non-zero.

Throttle admin email notifications

(Requested by Peter Love. )
Especially for proxy problems, but useful for everything. Limit the number of admin email notifications by specifying a minimum delay between identical emails.

Agis, cli process single pandaqueue

Produce config file for short list of PQ on cli for test/debug purposes.
-P PQ1,PQ2

ThrottleAborted sched plugin

This issue to track progress on the "feature request" email on Feb 11. This is where pilot submission is throttled based on the number of Removed jobs in condor_history.

An exmaple is where the CE rejects jobs, such as this:
CREAM error: BLAH error: submission command failed (exit code = 1) (stdout:) (stderr:qsub: Maximum number of jobs already in queue for user MSG=total number of current user's jobs exceeds the queue limit: user [email protected], queue hmem_sl6-) N/A (j

Avoid factory restart when reloading queue config

There may be advantages to have a 'reload' option in order to re-read the queue config from disk. Currently we issue a 'service factory restart' to do this.

Only include *.conf files

Only include *.conf files in the config directory /etc/autopyfactory/queues.d/