cmgtools batch priority is failing? about cmg-cmssw

[adavidzh]

These are all the groups cmgtools is part of (after resolving the hierarchy):

$ bugroup -r -l -w | grep -B1 cmgtools | cut -b 1-50
GROUP_NAME:  u_zh
USERS:       stasheva mwalker mtrovato wmueller da
SHARES:      [stasheva, 1] [mwalker, 1] [mtrovato,
--
GROUP_NAME:  COMM_GLOBAL
USERS:       mzanetti spadhi relval gowdy bbozsogi
SHARES:      [mzanetti, 1] [gouskos, 1] [relval, 1
--
GROUP_NAME:  production
USERS:       cmgtools
SHARES:      [default, 1] [cmgtools, 1]
--
GROUP_NAME:  u_CMSREPACK
USERS:       rovere beaucero gowdy relval spadhi e
SHARES:      [rovere, 1] [beaucero, 1] [relval, 1]
--
GROUP_NAME:  u_CMSCOMM
USERS:       vutova konec kbunkow jgomezca azagozd
--
GROUP_NAME:  u_CMST3
USERS:       cmgtools querten gmasetti janot jad b
--
GROUP_NAME:  u_CMS
USERS:       cassel magini yuyi wildish phedex gif

and

$ bugroup -l u_CMST3
GROUP_NAME:  u_CMST3
USERS:       production/ CMS_CERN01_YODA/
SHARES:      [production, 5000] [CMS_CERN01_YODA, 5000]
$ bugroup -l production
GROUP_NAME:  production
USERS:       cmgtools
SHARES:      [default, 1] [cmgtools, 1]

Can you try to run bsub -q test -G production ls as cmgtools (bsub -G only knows of groups you are directly a member of, not the full hierarchy)?

And you may also find it useful to look at bhpart -r SHARE (credit to @cbernet) or:

$  bhpart -r | egrep "HOST|cmgtools|SHARE" | grep cmgtools -B2
SHARE_INFO_FOR: CMSCAF/u_CMSCOMM/COMM_GLOBAL/
USER/GROUP   SHARES  PRIORITY  STARTED  RESERVED  CPU_TIME  RUN_TIME   ADJUST
cmgtools        1       0.333     0        0         0.0        0       0.000
--
SHARE_INFO_FOR: SHARE/u_CMST3/production/
USER/GROUP   SHARES  PRIORITY  STARTED  RESERVED  CPU_TIME  RUN_TIME   ADJUST
cmgtools        1       0.001    69        0   3840489.0  8482526       0.000
--
SHARE_INFO_FOR: SHARE/u_CMS/u_CMSREPACK/
USER/GROUP   SHARES  PRIORITY  STARTED  RESERVED  CPU_TIME  RUN_TIME   ADJUST
cmgtools        1       0.002     4        0   2140151.0  2999014       0.000
--
SHARE_INFO_FOR: SHARE/u_CMS/u_zh/
USER/GROUP   SHARES  PRIORITY  STARTED  RESERVED  CPU_TIME  RUN_TIME   ADJUST
cmgtools        1       0.016     0        0    314079.7   321427       0.000

So priority looks high and shares looks low.

[cbernet]

Hi Giovanni,

I think it's not only a question of priority.
With the command
bqueues

we see the following:

8nh 24 Open:Active - - - - 18900 18000 900 0
1nd 23 Open:Active - - - - 17413 14177 3236 0
2nd 22 Open:Active - - - - 15913 13880 2033 0
1nw 21 Open:Active 5000 - - - 6850 4563 2287 0
2nw 21 Open:Active 4500 - - - 10730 6219 4511 0

So only 900 jobs running on 8nh…

I think it's due to the fact that thousands of jobs are running on long queues at the moment.
They keep the nodes for a long time, so even with a high priority, difficult to get a node.

Cheers,

Colin

On Mar 21, 2014, at 12:26 PM, Giovanni Petrucciani [email protected] wrote:

It appears that the cmgtools user is getting only very few batch slots: at the moment we're getting less than ~200 running jobs, which seems low even for an ordinary user.

Does anybody have a way to check if we still have the proper priorities set in the batch system?

Giovanni
—
Reply to this email directly or view it on GitHub.

[gpetruc]

According to @ssekmen it could be that we're just out of CPU nodes since when submitting from lxplus5 one gets only SL5 nodes, and so we should submit from SL6.
I'm now trying to see if this works,

Giovanni

[gpetruc]

Ok, it appears that Sezen's suggestion was right: to get enough slots, one should run the production from a lxplus6 machine and not from a lxplus5 one.
I'll add this information on the production page, and close the issue.