danielwinterbottom / ichiggstautau Goto Github PK
View Code? Open in Web Editor NEWCMSSW package for Imperial group analysis code
Home Page: http://danielwinterbottom.github.io/ICHiggsTauTau
CMSSW package for Imperial group analysis code
Home Page: http://danielwinterbottom.github.io/ICHiggsTauTau
It is currently calculated and provided (as a ValueMap) by this external package: https://github.com/ajgilbert/ICAnalysis-ElectronConversionCalculator
At the time this was due to compiler problems in CMSSW_4_X_Y and early CMSSW_5_X_Y releases.
Someone should check:
#if CMSSW_MAJOR_VERSION >= X
preprocessor checks to work around thisInspired by #26 - should go through FnPredicates and FnPairs and:
This can be considered a low priority task :-)
If running ./bin/HTT on a local file with EventChecker enabled it will run to the end and write the output tree, then crash complaining about a corrupted double-linked list. This never happens when running off files on dcache, or when running on a local file without using EventChecker.
So there is some memory problem, though atm I fail to understand why this is only a problem under these very specific circumstances.
Anyway, low priority, but one to fix should we run out of stuff to do.
Appears the new cut-based electron ID has a new variable we don't save at the moment:
dEtaInSeed => appears to be calculated as:
also same definition here:
https://twiki.cern.ch/twiki/bin/view/CMS/HEEPElectronIdentificationRun2#Selection_Cuts_HEEP_V5_1
If we want to add this should check it works ok on miniaod (should do), i.e. that:
ele->superCluster().isNonnull() && ele->superCluster()->seed().isNonnull()
are both non-null.
Currently CheckEvents reads the events to be checked from within HTTSequence, which means that you have to recompile every time you add/remove events numbers. Should be modified so that it can read from an external file to
I'm not sure if anybody else is affected by this, but as I can't figure out how to solve this problem...
This is what happens:
R__unzipLZMA: error 5 in lzma_code
Error in <TBasket::ReadBasketBuffers>: fNbytes = 4312, fKeylen = 91, fObjlen = 30012, noutot = 0, nout=0, nin=4221, nbuf=30012
Error in <TBranchElement::GetBasket>: File: root://xrootd.grid.hep.ph.ic.ac.uk//store/user/adewit/July08_MC_74X/DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/crab_DYJetsToLL-2/150710_084817/0000/EventTree_964.root at byte:187660, branch:genParticles.pdgid_, entry:2, badread=1, nerrors=1, basketnumber=0
before throwing a bad_alloc
==10381== 7,024 (80 direct, 6,944 indirect) bytes in 2 blocks are definitely lost in loss record 269,416 of 269,644
==10381== at 0x4806FB5: operator new(unsigned long) (in /cvmfs/cms.cern.ch/slc6_amd64_gcc481/external/valgrind/3.10.0/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10381== by 0x4935D60: std::vector<ic::Tau*, std::allocator<ic::Tau*> >& ic::TreeEvent::GetPtrVec<ic::Tau>(std::string const&, std::string) (in /vols/cms04/amd12/CMSSW_7_2_0/src/UserCode/ICHiggsTauTau/Analysis/HiggsTauTau/lib/libICHiggsTauTau.so)
==10381== by 0x49DB4F7: ic::SimpleFilter<ic::Tau>::Execute(ic::TreeEvent*) (in /vols/cms04/amd12/CMSSW_7_2_0/src/UserCode/ICHiggsTauTau/Analysis/HiggsTauTau/lib/libICHiggsTauTau.so)
==10381== by 0x4B13F1F: ic::AnalysisBase::RunAnalysis() (in /vols/cms04/amd12/CMSSW_7_2_0/src/UserCode/ICHiggsTauTau/Analysis/Core/lib/libICCore.so)
==10381== by 0x40DF11: main (in /vols/cms04/amd12/CMSSW_7_2_0/src/UserCode/ICHiggsTauTau/Analysis/HiggsTauTau/bin/HTT)
This suggests GetPtrVec is causing a memory leak. I don't see what the issue with it is though. Has anybody else seen this before/any ideas where else to look for the problem?
This package didn't compile in an old CMSSW version. It should now and should be reintegrated into ICPhotonProducer
... and I mean really broken:
this doesn't compile anymore:
edm::RefToBaseProd<T> reftobase = edm::RefToBaseProd<T>((handle->refAt(0)));
we are now forced to use the edm::consumes
mechanism. Info here and here. This isn't available in 5_3_X so will need some sort of workaround - ideally without a huge amount of #ifdef'ing.
Need to check if CMSSW producers work ok in multi-threaded mode. Is this enabled by default?
For @ajgilbert and @adewit, it is clear that the number of events in the output trees is now a lot larger than it was in Run 1, due to moving cuts like isolation to post output tree. This poses a problem for our SVfit workflow, which currently requires more than 10 000 jobs for the full set of MC samples (if I use 7000 events per job, which I think was what we had in Run 1 and still makes the jobs several hours long), which is only going to get larger as we add the exclusive samples. The question is whether we have to live with this and we simply risk having to wait longer for SVFit jobs (given that so many of our cuts are now at ntuple level, in principle we could have to run this less often, although this has not necessarily reflected reality so far with the fairly frequent changes of triggers and other pre-output ntuple level choices), or whether there could be value to altering our workflow to apply some preselection and only run the calculation for a subset of the events in the tree (and when creating the tree, fill the SVFit mass branch with -999 for the events for which there is no calculation). Thoughts? Experiences of trying to get >10000 jobs of that length through any kind of batch system?
We currently store various root files and inputs that are needed for the analyses here (JEC files, mva trainings etc). Unfortunately it turns out that crab scans recursively for any directory named data under $CMSSW_BASE/src and ships this off with each job. This wastes storage space and time packing and sending these files to the crab server. I suggest we rename the folder from "data" to "input" to avoid this
Have to deal with two separate collections:
https://twiki.cern.ch/twiki/bin/view/CMSPublic/WorkBookMiniAOD#MC_Truth
The prunedGenParticles are a normal reco::GenParticle collection and should contain everything we need in the analysis (matrix element, full tau chain decay, final state electrons/muons). We can just save this as normal with ICGenParticleProducer.
The packedGenParticles are the new pat::PackedGenParticle type and contain all status 1 up to some high rapidity value, and are mainly for clustering gen jets.
After using the CLASS_MEMBER compiler macro the class is left in the public state.
The jet flavour calculator for reclustered miniAOD jets fails because of course the prunedGenParticles don't contain all status 2 and 3 particles. I don't think there's much we can do about this as these particles are simply dropped from the event. If we did need the jet flavour for reclustered jets we could match them to the slimmedJets collection and use the stored flavour (if the reclustered jets are ak4 CHS too).
Two issues here:
I found these slides useful in explaining the details:
https://indico.cern.ch/event/388914/contribution/0/material/slides/0.pdf
I will try and look into this next week, but if someone else wants to get started in the meantime please go ahead.
@adewit @rcl11 While working with the new status flags I found that our ICGenParticlePruner doesn't keep the status flag information. The ICGenParticlePruner seems to be a copy of an older version of the cmssw GenParticlePruner, the new version of which does keep the status flags. I've switched from ICGenParticlePruner to GenParticlePruner in my config and not seen any issues.
Hi Patrick,
These folders need a file like .gitignore to be tracked by git. Currently this prevents a "make all" from completing successfully.
cheers,
Andrew
In the latest of IC batch improvements we can now only use < 1/3 of the short queue at the same time, so need to do something slightly cleverer with the hundreds of jobs that we have at the moment... Or face half day-long waits to rerun everything.
I haven't got any ideas (or time to mess about with it), but definitely something to sort out in the next few months.
Not very urgent but something I noticed when testing the code in CMSSW 7_6:
The L1FastjetCorrectionESProducer (which we currently use to apply JECs to reclustered jets) doesn't work anymore. Haven't found a straightforward workaround/alternative yet, though I'm almost certain there must be one. If not I think we can fix the module ourselves as the only reason it doesn't work is that getByLabel is used without a consumes call, which shouldn't be too hard to add in.
Both Adinda and I have noticed this behaviour in CMSSW_7_4_5, it only seems to affect the jet collections. Accessing them with TTree::Draw works fine though.
Exact cuts depend on the training used, so shouldn't be maintained in PFJet.hh anymore. Remove this class method once usage is replaced by, e.g. PileupJetID function in FnPredicates.
Main thing to check is updating dictionaries and checking existing ntuples can still be read.
Non-exhaustive list of code updates that will be needed in the next 3-4 months, towards being able to analyse the full 2016 dataset and future analyses
CMSSW-facing part of the code/CMSSW config:
Analysis code:
_NOT AN ISSUE ON OUR SIDE_ but still relevant to anyone trying to use the code to produce ntuples with mvamet included (so mainly @pjdunne and @amagnan) - in CMSSW_7_4_12 (and _15), puJetIdForPFMVAMEt crashes if you try to run on more than one file in the same job.
The solution (for now) is to use FileBased splitting and run one file per job - this behaviour has been flagged up to the puJetID people, will report back when I know more.
Just noticed that the trigger filter module is currently always run after HTTPairSelector which isn't in line with the method used for previous analyses. Should remember to add the module to be run t before the pair selector for mssmspring16 and fall15 strategies.
Compilation:
/afs/cern.ch/work/a/agilbert/CMSSW_TEST/CMSSW_5_3_7/src/UserCode/ICHiggsTauTau/plugins/ICMetProducer.cc:16:62: fatal error: DataFormats/METReco/interface/PFMEtSignCovMatrix.h: No such file or directory
--> PFMEtSignCovMatrix no longer needed, as we won't be importing an external MET covariance matrix anymore. Will remove this option from the code. Possible knock-on effect to any cfg files where the option "InputSig" is defined.
/afs/cern.ch/work/a/agilbert/CMSSW_TEST/CMSSW_5_3_7/src/UserCode/ICHiggsTauTau/plugins/ICPhotonProducer.hh:10:73: fatal error: EgammaAnalysis/ElectronTools/interface/PFIsolationEstimator.h: No such file or directory
--> Pending
The new pythia 8 status codes don't provide a direct replacement for status 3. The new status 21-29 is similar, but differs in that the lepton from a W->lnu decay for instance isn't always status 21-29.
I've emailed Josh Bendavid to ask if there is a recommendation, and it also appears from slides from Gen group meetings in early May that they are preparing a new set of "status flags" to try and solve this issue.
In run I we implemented the muon ID by storing in the ntuples all the variables used and then having a function in FnPredicates which implemented the ID. This now needs to be updated to the run II ID.
The IC disk server doesn't really like too many files being open at the same time, which means running systematic shifts as well as the central values from the same job makes vols super slow (and our colleagues who then can't do any work super annoyed). Can work around this by running syst shifts separately, but at some point should investigate better options, for example writing a separate tree for each shift into one file, then writing them to separate files after the jobs finish. Not sure this would actually be better than current workaround/wouldn't overload the disk server in some other way, but we should check.
The JECs calculated by ICPFJetProducer for jets built from (packed) PF candidates are different from the ones calculated by PAT, need to understand why if we want to produce jets without running the pat jet sequence.
This concerns @adewit and @ajgilbert - so far we have kept HTTSequence capable of running the paper2013 strategy to produce flat trees in sync with those we made for the run1 analyses. However, it is potentially more complicated to keep the flat tree reading code (i.e. HTTAnalysisTools linking up with HiggsTauTauPlot4) compatible with both old and new strategies. Background methods and aliases are coded very specifically for the run 1 selections (using the "method") quantity, and HTTAnalysisTools is already 1700 lines long - do we want to try to keep both strategies available in this part of the code? Of course if we don't then we cannot remake 8 TeV datacards without using an old branch. If we are to keep both then I expect defining a new set of "method"s would probably be the simplest. Thoughts?
In CMSSW_7_2_X there is no way to rerun the PU jet ID MVA on jet collections reclustered by the user. This is being fixed in CMSSW_7_4_X and we should remember to fix this.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.