automatic "*.root" appending to end of .ds about factorytools

[rsmith54]

This is quite odd. Could you send me the exact command you run?

Thanks for reporting.

On Thursday, May 12, 2016, JOliver-GH [email protected] wrote:

When running on the grid any .ds file i provide gets a ".root" appended to
the end. I've provided an the output below. I looked to where the error was
but couldn't pin it down

INFO:root:creating new sample handler
INFO:root:creating new job
INFO:root:creating algorithms
BasicEventSelection::B... INFO Calling constructor
INFO:root:adding basicEventSelection to algs
BasicEventSelection::s... INFO Calling setupJob
INFO:root:adding mcEventVeto to algs
INFO:root:adding calibrateST to algs
INFO:root:adding preselectDileptonicWW to algs
INFO:root:adding selectDileptonicWW to algs
INFO:root:adding postselectDileptonicWW to algs
INFO:root:adding calculateRJigsawVariables to algs
INFO:root:adding calculateRegionVars to algs
INFO:root:adding writeOutputNtupleSR to algs
INFO:root:adding writeOutputNtupleCR1L to algs
INFO:root:adding writeOutputNtupleCR0L to algs
INFO:root:submit_dir already exists.
INFO:root:Overwriting previous submitDir
INFO:root:creating driver
INFO:root:grid driver
gridname: user.jaoliver.%in:name[2]%.%in:name[3]%.120516_lvlv
INFO:root:submit job
BasicEventSelection::B... INFO Calling constructor
TFile::TFile ERROR file /afs/cern.ch/work/j/jaoliver/private/RJigsawTools_110516/submit_dir/input/RJigsawTools/data/mc15_13TeV_SM_SUSY2_p2470_failed.ds.root
http://cern.ch/work/j/jaoliver/private/RJigsawTools_110516/submit_dir/input/RJigsawTools/data/mc15_13TeV_SM_SUSY2_p2470_failed.ds.root
can not be opened (No such file or directory)
TROOT::WriteTObject ERROR The current directory (PyROOT) is not associated
with a file. The object (sample) has not been written.
TFile::TFile ERROR file /afs/
cern.ch/work/j/jaoliver/private/RJigsawTools_110516/submit_dir/hist/RJigsawTools/data/mc15_13TeV_SM_SUSY2_p2470_failed.ds.root
can not be opened (No such file or directory)
TROOT::WriteTObject ERROR The current directory (PyROOT) is not associated
with a file. The object (sample) has not been written.
^Z

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#106

[lawrenceleejr]

Hi Jason,

Are you sure

/afs/cern.ch/work/j/jaoliver/private/RJigsawTools_110516/submit_dir/input/RJigsawTools/data/mc15_13TeV_SM_SUSY2_p2470_failed.ds

exists? This might be a problem with the automatic filling of the sample handler. Do you definitely mean to be looking in submit_dir/input?

-L

[joliver92]

So i checked this, and i definitely didn’t write that, it seems the “default” is to set RJigsawTools/data/… to be from /submit_dir/input.

I then ran it from ~/work/j/jaoliver/private/RJigsaw/RJigsaw/data/mc… and it runs locally but not on the grid

at this point I’m unsure what the issue is as my grid submissions will stall at submission

On 12 May 2016, at 3:39 PM, Lawrence Lee [email protected] wrote:

Hi Jason,

Are you sure

/afs/cern.ch/work/j/jaoliver/private/RJigsawTools_110516/submit_dir/input/RJigsawTools/data/mc15_13TeV_SM_SUSY2_p2470_failed.ds
exists? This might be a problem with the automatic filling of the sample handler. Do you definitely mean to be looking in submit_dir/input?

-L

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub #106 (comment)

[lawrenceleejr]

Ok then this is another issue. Send the command you run to try and submit
to the grid and send the terminal output you see please.

-L

On Fri, May 13, 2016 at 4:40 AM JOliver-GH [email protected] wrote:

So i checked this, and i definitely didn’t write that, it seems the
“default” is to set RJigsawTools/data/… to be from /submit_dir/input.

I then ran it from ~/work/j/jaoliver/private/RJigsaw/RJigsaw/data/mc… and
it runs locally but not on the grid

at this point I’m unsure what the issue is as my grid submissions will
stall at submission

On 12 May 2016, at 3:39 PM, Lawrence Lee [email protected]
wrote:

Hi Jason,

Are you sure

/afs/
cern.ch/work/j/jaoliver/private/RJigsawTools_110516/submit_dir/input/RJigsawTools/data/mc15_13TeV_SM_SUSY2_p2470_failed.ds
exists? This might be a problem with the automatic filling of the sample
handler. Do you definitely mean to be looking in submit_dir/input?

-L

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub <
#106 (comment)

—
You are receiving this because you commented.

Reply to this email directly or view it on GitHub
#106 (comment)

-L

Sent from my iPhone

[joliver92]

Sorry for the late reply,

This is what i run, with the output below :

python RJigsawTools/util/run_lvlv.py --doOverwrite --driver grid --inputDS ~/work/private/RJigsawTools/RJigsawTools/data/mc15_13TeV_HIGG3D1_TEST.ds

RestFrames v1.0.0 -- Developed by Christopher Rogan ([email protected])
Copyright (c) 2014-2015, Christopher Rogan
http://RestFrames.com

xAOD::Init INFO Environment initialised for data access
INFO:root:creating new sample handler
INFO:root:creating new job
INFO:root:creating algorithms
BasicEventSelection::B... INFO Calling constructor
INFO:root:adding basicEventSelection to algs
BasicEventSelection::s... INFO Calling setupJob
INFO:root:adding mcEventVeto to algs
INFO:root:adding calibrateST to algs
INFO:root:adding preselectDileptonicWW to algs
INFO:root:adding selectDileptonicWW to algs
INFO:root:adding postselectDileptonicWW to algs
INFO:root:adding calculateRJigsawVariables to algs
INFO:root:adding calculateRegionVars to algs
INFO:root:adding writeOutputNtupleSR to algs
INFO:root:adding writeOutputNtupleCR1L to algs
INFO:root:adding writeOutputNtupleCR0L to algs
INFO:root:submit_dir already exists.
INFO:root:Overwriting previous submitDir
INFO:root:creating driver
INFO:root:grid driver
INFO:root:submit job
BasicEventSelection::B... INFO Calling constructor
BasicEventSelection::B... INFO Calling constructor
Submitting mc15_13TeV.341006.PowhegPythia8EvtGen_CT10_AZNLOCTEQ6L1_ggH200NWA_WWlvlv_15_5.merge.DAOD_HIGG3D1.e3944_a766_a777_r6282_p2495...

Its at this point I get what seems to be a stall, i’ve left it for 30-40 minutes and nothing changes, its also much longer than any other submission i’ve ever had, this also stalls for other data-sets.

I’ve done the usual lsetup panda and proxy setup but with no luck.

upon “ctrl- x ctrl-c “‘ing it I get the following:

INFO:root:submit_dir already exists.
INFO:root:Overwriting previous submitDir
INFO:root:creating driver
INFO:root:grid driver
INFO:root:submit job
BasicEventSelection::B... INFO Calling constructor
BasicEventSelection::B... INFO Calling constructor
Submitting mc15_13TeV.341006.PowhegPythia8EvtGen_CT10_AZNLOCTEQ6L1_ggH200NWA_WWlvlv_15_5.merge.DAOD_HIGG3D1.e3944_a766_a777_r6282_p2495...
^X^C^X^CTraceback (most recent call last):
File "", line 1, in
File "/afs/cern.ch/work/j/jaoliver/private/RJigsawTools/RootCoreBin/python/EventLoopGrid/ELG_prun.py", line 97, in ELG_prun
out = subprocess.check_output(dummycmd, stderr=subprocess.STDOUT)
File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/2.7.9p1-x86_64-slc6-gcc49/sw/lcg/releases/LCG_81b/Python/2.7.9.p1/x86_64-slc6-gcc49-opt/lib/python2.7/subprocess.py", line 567, in check_output
output, unused_err = process.communicate()
File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/2.7.9p1-x86_64-slc6-gcc49/sw/lcg/releases/LCG_81b/Python/2.7.9.p1/x86_64-slc6-gcc49-opt/lib/python2.7/subprocess.py", line 791, in communicate
stdout = _eintr_retry_call(self.stdout.read)
File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/2.7.9p1-x86_64-slc6-gcc49/sw/lcg/releases/LCG_81b/Python/2.7.9.p1/x86_64-slc6-gcc49-opt/lib/python2.7/subprocess.py", line 476, in _eintr_retry_call
return func(*args)
KeyboardInterrupt
TypeError: an integer is required

but i don’t have the know-how to know whats happening

Let me know if we need anything else to diagnose the problem,

cheers,

Jason

On 13 May 2016, at 2:48 PM, Lawrence Lee [email protected] wrote:

Ok then this is another issue. Send the command you run to try and submit
to the grid and send the terminal output you see please.

-L

On Fri, May 13, 2016 at 4:40 AM JOliver-GH [email protected] wrote:

So i checked this, and i definitely didn’t write that, it seems the
“default” is to set RJigsawTools/data/… to be from /submit_dir/input.

I then ran it from ~/work/j/jaoliver/private/RJigsaw/RJigsaw/data/mc… and
it runs locally but not on the grid

at this point I’m unsure what the issue is as my grid submissions will
stall at submission

On 12 May 2016, at 3:39 PM, Lawrence Lee [email protected]
wrote:

Hi Jason,

Are you sure

/afs/
cern.ch/work/j/jaoliver/private/RJigsawTools_110516/submit_dir/input/RJigsawTools/data/mc15_13TeV_SM_SUSY2_p2470_failed.ds
exists? This might be a problem with the automatic filling of the sample
handler. Do you definitely mean to be looking in submit_dir/input?

-L

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub <
#106 (comment)

—
You are receiving this because you commented.

Reply to this email directly or view it on GitHub
#106 (comment)

-L

Sent from my iPhone
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub #106 (comment)

[lawrenceleejr]

Hi Jason,

What happens if you try to test it over FAX instead of submitting to the grid. Just run

python RJigsawTools/util/run_lvlv.py --doOverwrite --inputDS ~/work/private/RJigsawTools/RJigsawTools/data/mc15_13TeV_HIGG3D1_TEST.ds

if that starts running then your grid submission should work. Does your directory

~/work/private/RJigsawTools/

have anything weird in it? Any huge files or anything? Can you just send the list of the contents of that folder?

Have you tried starting from absolute scratch to see if it works like when you first set it all up?

-L

[joliver92]

when i run what you provided it still stalls,

this is whats in it:

drwxr-xr-x. 17 jaoliver zp 2048 Apr 8 03:11 070416_lvlv
drwxr-xr-x. 6 jaoliver zp 2048 Mar 15 13:23 CommonTools
drwxr-xr-x. 5 jaoliver zp 2048 Mar 15 13:20 Ext_RestFrames
drwxr-xr-x. 8 jaoliver zp 2048 Mar 15 13:24 JetJvtEfficiency
lrwxr-xr-x. 1 jaoliver zp 80 May 15 04:29 rcSetup.csh -> /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rcSetup/00-04-16/rcSetup.csh
lrwxr-xr-x. 1 jaoliver zp 79 May 15 04:29 rcSetup.sh -> /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rcSetup/00-04-16/rcSetup.sh
drwxr-xr-x. 10 jaoliver zp 2048 Apr 7 02:46 RJigsawTools
-rw-r--r--. 1 jaoliver zp 109 May 4 05:42 RJTSetup.sh
-rw-r--r--. 1 jaoliver zp 139 May 4 05:41 RJTSetup.sh~
drwxr-xr-x. 11 jaoliver zp 2048 May 15 04:30 RootCoreBin
drwxr-xr-x. 7 jaoliver zp 2048 May 15 04:31 submit_dir
drwxr-xr-x. 14 jaoliver zp 2048 Mar 15 13:23 SUSYTools
drwxr-xr-x. 9 jaoliver zp 2048 Mar 15 13:23 xAODAnaHelpers

I also tried it from scratch in a “RJigsawTools_110516” folder and it still stalls

RestFrames v1.0.0 -- Developed by Christopher Rogan ([email protected])
Copyright (c) 2014-2015, Christopher Rogan
http://RestFrames.com

xAOD::Init INFO Environment initialised for data access
INFO:root:creating new sample handler
INFO:root:creating new job
INFO:root:creating algorithms
BasicEventSelection::B... INFO Calling constructor
INFO:root:adding basicEventSelection to algs
BasicEventSelection::s... INFO Calling setupJob
INFO:root:adding mcEventVeto to algs
INFO:root:adding calibrateST to algs
INFO:root:adding preselectDileptonicWW to algs
INFO:root:adding selectDileptonicWW to algs
INFO:root:adding postselectDileptonicWW to algs
INFO:root:adding calculateRJigsawVariables to algs
INFO:root:adding calculateRegionVars to algs
INFO:root:adding writeOutputNtupleSR to algs
INFO:root:adding writeOutputNtupleCR1L to algs
INFO:root:adding writeOutputNtupleCR0L to algs
INFO:root:submit_dir already exists.
INFO:root:Overwriting previous submitDir
INFO:root:creating driver
INFO:root:direct driver
INFO:root:submit job
BasicEventSelection::B... INFO Calling constructor
BasicEventSelection::B... INFO Calling constructor
Running sample: mc15_13TeV.341006.PowhegPythia8EvtGen_CT10_AZNLOCTEQ6L1_ggH200NWA_WWlvlv_15_5.merge.DAOD_HIGG3D1.e3944_a766_a777_r6282_p2495

I then uninstalled it all opened a fresh terminal and follow the instructions and when i get to the following i get the below output:

python RJigsawTools/util/run_tls.py --doOverwrite --nevents 10 --verbosity debug --inputDS /afs/cern.ch/work/r/rsmith/public/p2613/

[jaoliver@lxplus081 RJigsawTools_160516]$ python RJigsawTools/util/run_tls.py --doOverwrite --nevents 10 --verbosity debug --inputDS /afs/cern.ch/work/r/rsmith/public/p2613/

RestFrames v1.0.0 -- Developed by Christopher Rogan ([email protected])
Copyright (c) 2014-2015, Christopher Rogan
http://RestFrames.com

xAOD::Init INFO Environment initialised for data access
INFO:root:creating new sample handler
WARNING:root:failed to find any samples in fillSampleHandler
INFO:root:creating new job
INFO:root:creating algorithms
BasicEventSelection::B... INFO Calling constructor
INFO:root:adding basicEventSelection to algs
BasicEventSelection::s... INFO Calling setupJob
INFO:root:adding mcEventVeto to algs
INFO:root:adding calibrateST to algs
INFO:root:adding preselectTwoLepton to algs
INFO:root:adding selectTwoLepton to algs
INFO:root:adding postselectTwoLepton to algs
INFO:root:adding calculateRJigsawVariables to algs
INFO:root:adding calculateRegionVars to algs
INFO:root:adding writeOutputNtuple_SR to algs
INFO:root:Running 10 events
INFO:root:creating driver
INFO:root:direct driver
INFO:root:submit job
BasicEventSelection::B... INFO Calling constructor
[jaoliver@lxplus081 RJigsawTools_160516]$

Now i’m unsure if it is still doing the analysis like it was in the previous versions or if it is stopping at this point?

thanks for the help

Jason

On 15 May 2016, at 11:47 AM, Lawrence Lee [email protected] wrote:

Hi Jason,

What happens if you try to test it over FAX instead of submitting to the grid. Just run

python RJigsawTools/util/run_lvlv.py --doOverwrite --inputDS ~/work/private/RJigsawTools/RJigsawTools/data/mc15_13TeV_HIGG3D1_TEST.ds
if that starts running then your grid submission should work. Does your directory

~/work/private/RJigsawTools/
have anything weird in it? Any huge files or anything? Can you just send the list of the contents of that folder?

Have you tried starting from absolute scratch to see if it works like when you first set it all up?

-L

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub #106 (comment)

[lawrenceleejr]

Hi Jason,

I suspect when you're trying to submit to the grid, your problem was the directory 070416_lvlv.

Frankly the first output you sent looks fine to me. I'm not sure what you're expecting to see. Why do you think it's broken when it says:

Running sample: mc15_13TeV.341006.PowhegPythia8EvtGen_CT10_AZNLOCTEQ6L1_ggH200NWA_WWlvlv_15_5.merge.DAOD_HIGG3D1.e3944_a766_a777_r6282_p2495

I would say - clear out the whole thing. Start from scratch. Start with a new directory and a new terminal session. Follow the instructions on installing RJigsawTools and then just run

python RJigsawTools/util/run_lvlv.py --doOverwrite --verbosity debug --inputDS [point it again to your mc15_13TeV_HIGG3D1_TEST.ds]

And you should see your output in submit_dir/data-trees/.

-L

[dylanpeukert]

Hi Jason,
Just thought I should add that the reason that your most recent test using --inputDS /afs/cern.ch/work/r/rsmith/public/p2613/ failed is that this test dataset no longer exists (I got the same output when I tried to run it). If you run over an existing dataset you will get the standard output.
Dylan

[rsmith54]

Hi @dylanpeukert and @JOliver-GH ,

Yes, sorry, I deleted that dataset. Since I have been making some 0L group datasets, I have been running out of room sometimes. I'll probably move a bunch of things to eos at some point and make sure that's setup out of the box here.

[rsmith54]

This is essentially the same as issue #101 now I think. This is just SampleHandler not finding any datasets.