Seg Fault seen in DTTSTheta::setConfig in production about cmssw

[makortel]

assign L1Trigger/DTTriggerServerTheta

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @Dr15Jones.

@makortel, @smuzaffar, @Dr15Jones, @rappoccio, @antoniovilela, @sextonkennedy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[cmsbuild]

New categories assigned: l1

@epalencia,@aloeliger you have been requested to review this Pull request/Issue and eventually sign? Thanks

[Dr15Jones]

The full traceback is

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Mon May 13 21:11:02 UTC 2024
Thread 14 (Thread 0x1456c0800700 (LWP 9105)):
[xrootd]

Thread 13 (Thread 0x1456c0c00700 (LWP 9104)):
[xrootd]

Thread 12 (Thread 0x1456c1000700 (LWP 9103)):
[xrootd]

Thread 11 (Thread 0x1456c1400700 (LWP 9102)):
[xrootd]

Thread 10 (Thread 0x1456c1800700 (LWP 9101)):
[xrootd]

Thread 9 (Thread 0x1456cc600700 (LWP 9100)):
#4  0x00001456f77ed3b6 in ?? () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_17_patch1/external/slc7_amd64_gcc700/lib/libz.so.1
#5  0x00001456f77eee37 in inflate () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_17_patch1/external/slc7_amd64_gcc700/lib/libz.so.1
#6  0x00001456f7ad73c5 in R__unzip () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_17_patch1/external/slc7_amd64_gcc700/lib/libCore.so
#7  0x00001456f86a8336 in TBasket::ReadBasketBuffers(long long, int, TFile*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_17_patch1/external/slc7_amd64_gcc700/lib/libTree.so
#8  0x00001456f86adcaa in TBranch::GetBasket(int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_17_patch1/external/slc7_amd64_gcc700/lib/libTree.so
#9  0x00001456f86ae480 in TBranch::GetEntry(long long, int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_17_patch1/external/slc7_amd64_gcc700/lib/libTree.so
#10 0x00001456f86c42e9 in TBranchElement::GetEntry(long long, int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_17_patch1/external/slc7_amd64_gcc700/lib/libTree.so
#11 0x00001456f86c41b6 in TBranchElement::GetEntry(long long, int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_17_patch1/external/slc7_amd64_gcc700/lib/libTree.so
#12 0x00001456d52b842f in edm::RootTree::getEntry(TBranch*, long long) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginIOPoolInput.so
#13 0x00001456d52c2c29 in edm::RootDelayedReader::getProduct_(edm::BranchID const&, edm::EDProductGetter const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginIOPoolInput.so
#14 0x00001456f895d4c7 in edm::DelayedReader::getProduct(edm::BranchID const&, edm::EDProductGetter const*, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#15 0x00001456f8942a46 in edm::InputProductResolver::resolveProduct_(edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#16 0x00001456f8a0232a in edm::Principal::findProductByLabel(edm::KindOfType, edm::TypeID const&, edm::InputTag const&, edm::EDConsumerBase const*, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#17 0x00001456f8a027c1 in edm::Principal::getByLabel(edm::KindOfType, edm::TypeID const&, edm::InputTag const&, edm::EDConsumerBase const*, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#18 0x00001456c842b62f in PreMixingSiStripWorker::addPileups(PileUpEventPrincipal const&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimTrackerSiStripDigitizerPlugins.so
#19 0x00001456c83814e5 in edm::PreMixingModule::pileWorker(edm::EventPrincipal const&, int, int, edm::EventSetup const&, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimGeneralPreMixingModulePlugins.so
#20 0x00001456c838701b in unsigned long edm::VectorInputSource::loopOverEvents<edm::RecordEventID<std::_Bind<bool (edm::PreMixingModule::*(std::reference_wrapper<edm::PreMixingModule>, std::_Placeholder<1>, int, std::_Placeholder<2>, std::reference_wrapper<edm::EventSetup const>, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*))(edm::EventPrincipal const&, int, int, edm::EventSetup const&, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*)> > >(edm::EventPrincipal&, unsigned long&, unsigned long, edm::RecordEventID<std::_Bind<bool (edm::PreMixingModule::*(std::reference_wrapper<edm::PreMixingModule>, std::_Placeholder<1>, int, std::_Placeholder<2>, std::reference_wrapper<edm::EventSetup const>, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*))(edm::EventPrincipal const&, int, int, edm::EventSetup const&, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*)> >, CLHEP::HepRandomEngine*, edm::EventID const*, bool) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimGeneralPreMixingModulePlugins.so
#21 0x00001456c8387175 in void edm::PileUp::readPileUp<std::_Bind<bool (edm::PreMixingModule::*(std::reference_wrapper<edm::PreMixingModule>, std::_Placeholder<1>, int, std::_Placeholder<2>, std::reference_wrapper<edm::EventSetup const>, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*))(edm::EventPrincipal const&, int, int, edm::EventSetup const&, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*)> >(edm::EventID const&, std::vector<edm::SecondaryEventIDAndFileInfo, std::allocator<edm::SecondaryEventIDAndFileInfo> >&, std::_Bind<bool (edm::PreMixingModule::*(std::reference_wrapper<edm::PreMixingModule>, std::_Placeholder<1>, int, std::_Placeholder<2>, std::reference_wrapper<edm::EventSetup const>, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*))(edm::EventPrincipal const&, int, int, edm::EventSetup const&, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*)>, int, edm::StreamID const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimGeneralPreMixingModulePlugins.so
#22 0x00001456c838316a in edm::PreMixingModule::doPileUp(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimGeneralPreMixingModulePlugins.so
#23 0x00001456c85dbb5d in edm::BMixingModule::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libMixingBase.so
...

Thread 8 (Thread 0x1456cde00700 (LWP 9099)):
#3  0x00001456e8d5c468 in sig_dostack_then_abort () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00001456c140497e in DTTSTheta::setConfig(DTConfigManager const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libL1TriggerDTTriggerServerTheta.so
#6  0x00001456c180cfe1 in DTSCTrigUnit::setConfig(DTConfigManager const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libL1TriggerDTSectorCollector.so
#7  0x00001456c144e338 in DTTrig::updateES(edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginL1TriggerDTTrigger.so
#8  0x00001456c144f7c4 in DTTrig::triggerReco(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginL1TriggerDTTrigger.so
#9  0x00001456c1448f48 in DTTrigProd::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginL1TriggerDTTrigger.so

Thread 7 (Thread 0x1456cee00700 (LWP 9098)):
#3  <signal handler called>
#4  0x00001456c8618380 in HcalSiPMShape::operator()(double) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libSimCalorimetryHcalSimAlgos.so
#5  0x00001456c8619e44 in HcalSiPMHitResponse::makeSiPMSignal(DetId const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, CLHEP::HepRandomEngine*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libSimCalorimetryHcalSimAlgos.so
#6  0x00001456c8619782 in HcalSiPMHitResponse::finalizeHits(CLHEP::HepRandomEngine*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libSimCalorimetryHcalSimAlgos.so
#7  0x00001456c865f9d1 in HcalDigitizer::finalizeEvent(edm::Event&, edm::EventSetup const&, CLHEP::HepRandomEngine*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libSimCalorimetryHcalSimProducers.so
#8  0x00001456c864bea4 in HcalDigiProducer::finalizeEvent(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libSimCalorimetryHcalSimProducers.so
#9  0x00001456c837f33d in edm::PreMixingModule::put(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimGeneralPreMixingModulePlugins.so
...

Thread 6 (Thread 0x1456cfc00700 (LWP 9097)):
#3  <signal handler called>
#4  0x00001456f50f8e29 in syscall () from /lib64/libc.so.6
#5  0x00001456f7336b38 in std::__atomic_futex_unsigned_base::_M_futex_wait_until (this=<optimized out>, __addr=0x1456004e2ef0, __val=2147483648, __has_timeout=<optimized out>, __s=..., __ns=...) at ../../../../../libstdc++-v3/src/c++11/futex.cc:55
#6  0x00001456f2a3f200 in XrdFile::read(void*, unsigned long, long) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libUtilitiesXrdAdaptor.so
#7  0x00001456f1ec657a in StorageAccountProxy::read(void*, unsigned long, long) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libUtilitiesStorageFactory.so
#8  0x00001456f1ecf01a in LocalCacheFile::cache(long, long) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libUtilitiesStorageFactory.so
#9  0x00001456f1ecf5ce in LocalCacheFile::prefetch(IOPosBuffer const*, unsigned long) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libUtilitiesStorageFactory.so
#10 0x00001456f1ec6914 in StorageAccountProxy::prefetch(IOPosBuffer const*, unsigned long) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libUtilitiesStorageFactory.so
#11 0x00001456f53ebd9f in TStorageFactoryFile::ReadBuffers(char*, long long*, int*, int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libIOPoolTFileAdaptor.so
#12 0x00001456f7f2d87d in TFileCacheRead::ReadBufferExtNormal(char*, long long, int, int&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_17_patch1/external/slc7_amd64_gcc700/lib/libRIO.so
#13 0x00001456f7f2d25a in TFileCacheRead::ReadBuffer(char*, long long, int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_17_patch1/external/slc7_amd64_gcc700/lib/libRIO.so
#14 0x00001456f87287fa in TTreeCache::ReadBufferNormal(char*, long long, int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_17_patch1/external/slc7_amd64_gcc700/lib/libTree.so
#15 0x00001456f86a7ceb in TBasket::ReadBasketBuffers(long long, int, TFile*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_17_patch1/external/slc7_amd64_gcc700/lib/libTree.so
#16 0x00001456f86adcaa in TBranch::GetBasket(int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_17_patch1/external/slc7_amd64_gcc700/lib/libTree.so
#17 0x00001456f86ae480 in TBranch::GetEntry(long long, int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_17_patch1/external/slc7_amd64_gcc700/lib/libTree.so
#18 0x00001456f86c42e9 in TBranchElement::GetEntry(long long, int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_17_patch1/external/slc7_amd64_gcc700/lib/libTree.so
#19 0x00001456d52b842f in edm::RootTree::getEntry(TBranch*, long long) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginIOPoolInput.so
#20 0x00001456d52e72cc in edm::RootFileEventFinder::getEventNumberOfEntry(long long) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginIOPoolInput.so
#21 0x00001456f8ce85c4 in edm::IndexIntoFile::fillUnsortedEventNumbers() const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libDataFormatsProvenance.so
#22 0x00001456f8ceb5ba in edm::IndexIntoFile::fillEventNumbersOrEntries(bool, bool) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libDataFormatsProvenance.so
#23 0x00001456d52e1f59 in edm::RootFile::validateFile(edm::InputType, bool) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginIOPoolInput.so
#24 0x00001456d52e4244 in edm::RootFile::RootFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::ProcessConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<edm::InputFile>, std::shared_ptr<edm::EventSkipperByID>, bool, int, int, unsigned int, unsigned int, int, edm::InputSource::ProcessingMode, edm::RunHelperBase*, bool, edm::ProductSelectorRules const&, edm::InputType, std::shared_ptr<edm::BranchIDListHelper>, std::shared_ptr<edm::ThinnedAssociationsHelper>, std::vector<edm::BranchID, std::allocator<edm::BranchID> > const*, std::shared_ptr<edm::DuplicateChecker>, bool, edm::ProcessHistoryRegistry&, std::vector<std::shared_ptr<edm::IndexIntoFile>, std::allocator<std::shared_ptr<edm::IndexIntoFile> > > const&, unsigned long, std::vector<edm::Hash<2>, std::allocator<edm::Hash<2> > >&, bool, bool, bool, bool, bool) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginIOPoolInput.so
#25 0x00001456d52ee7f7 in edm::RootEmbeddedFileSequence::makeRootFile(std::shared_ptr<edm::InputFile>) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginIOPoolInput.so
#26 0x00001456d52a78c0 in edm::RootInputFileSequence::initTheFile(bool, bool, edm::InputSource*, char const*, edm::InputType) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginIOPoolInput.so
#27 0x00001456d52eddac in edm::RootEmbeddedFileSequence::readOneRandom(edm::EventPrincipal&, unsigned long&, CLHEP::HepRandomEngine*, edm::EventID const*, bool) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginIOPoolInput.so
#28 0x00001456c8386fcf in unsigned long edm::VectorInputSource::loopOverEvents<edm::RecordEventID<std::_Bind<bool (edm::PreMixingModule::*(std::reference_wrapper<edm::PreMixingModule>, std::_Placeholder<1>, int, std::_Placeholder<2>, std::reference_wrapper<edm::EventSetup const>, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*))(edm::EventPrincipal const&, int, int, edm::EventSetup const&, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*)> > >(edm::EventPrincipal&, unsigned long&, unsigned long, edm::RecordEventID<std::_Bind<bool (edm::PreMixingModule::*(std::reference_wrapper<edm::PreMixingModule>, std::_Placeholder<1>, int, std::_Placeholder<2>, std::reference_wrapper<edm::EventSetup const>, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*))(edm::EventPrincipal const&, int, int, edm::EventSetup const&, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*)> >, CLHEP::HepRandomEngine*, edm::EventID const*, bool) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimGeneralPreMixingModulePlugins.so
#29 0x00001456c8387175 in void edm::PileUp::readPileUp<std::_Bind<bool (edm::PreMixingModule::*(std::reference_wrapper<edm::PreMixingModule>, std::_Placeholder<1>, int, std::_Placeholder<2>, std::reference_wrapper<edm::EventSetup const>, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*))(edm::EventPrincipal const&, int, int, edm::EventSetup const&, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*)> >(edm::EventID const&, std::vector<edm::SecondaryEventIDAndFileInfo, std::allocator<edm::SecondaryEventIDAndFileInfo> >&, std::_Bind<bool (edm::PreMixingModule::*(std::reference_wrapper<edm::PreMixingModule>, std::_Placeholder<1>, int, std::_Placeholder<2>, std::reference_wrapper<edm::EventSetup const>, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*))(edm::EventPrincipal const&, int, int, edm::EventSetup const&, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*)>, int, edm::StreamID const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimGeneralPreMixingModulePlugins.so
#30 0x00001456c838316a in edm::PreMixingModule::doPileUp(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimGeneralPreMixingModulePlugins.so
#31 0x00001456c85dbb5d in edm::BMixingModule::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libMixingBase.so
...

Thread 5 (Thread 0x1456d0800700 (LWP 9096)):
#3  <signal handler called>
#4  je_tcache_bin_flush_small (tsd=tsd@entry=0x1456d07f9af0, tcache=tcache@entry=0x1456d07f9cb0, tbin=0x1456d07f9d50, binind=binind@entry=6, rem=100) at src/tcache.c:116
#5  0x00001456f74078a6 in tcache_dalloc_small (slow_path=false, binind=6, ptr=0x145677221d20, tcache=<optimized out>, tsd=<optimized out>) at include/jemalloc/internal/tcache_inlines.h:178
#6  arena_dalloc (slow_path=false, alloc_ctx=<synthetic pointer>, tcache=<optimized out>, ptr=0x145677221d20, tsdn=<optimized out>) at include/jemalloc/internal/arena_inlines_b.h:224
#7  idalloctm (slow_path=false, is_internal=false, alloc_ctx=<synthetic pointer>, tcache=<optimized out>, ptr=0x145677221d20, tsdn=<optimized out>) at include/jemalloc/internal/jemalloc_internal_inlines_c.h:118
#8  ifree (slow_path=false, tcache=<optimized out>, ptr=0x145677221d20, tsd=0x1456d07f9af0) at src/jemalloc.c:2222
#9  free (ptr=0x145677221d20) at src/jemalloc.c:2393
#10 0x00001456c8491871 in std::_Rb_tree<int, std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude>, std::_Select1st<std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude> >, std::less<int>, std::allocator<std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude> > >::_M_erase(std::_Rb_tree_node<std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude> >*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimTrackerSiPixelDigitizerPlugins.so
#11 0x00001456c84917f3 in std::_Rb_tree<int, std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude>, std::_Select1st<std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude> >, std::less<int>, std::allocator<std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude> > >::_M_erase(std::_Rb_tree_node<std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude> >*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimTrackerSiPixelDigitizerPlugins.so
#12 0x00001456c84917f3 in std::_Rb_tree<int, std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude>, std::_Select1st<std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude> >, std::less<int>, std::allocator<std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude> > >::_M_erase(std::_Rb_tree_node<std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude> >*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimTrackerSiPixelDigitizerPlugins.so
#13 0x00001456c84917f3 in std::_Rb_tree<int, std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude>, std::_Select1st<std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude> >, std::less<int>, std::allocator<std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude> > >::_M_erase(std::_Rb_tree_node<std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude> >*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimTrackerSiPixelDigitizerPlugins.so
#14 0x00001456c84a2400 in SiPixelDigitizerAlgorithm::hitSignalReweight(PSimHit const&, std::map<int, float, std::less<int>, std::allocator<std::pair<int const, float> > >&, unsigned long, unsigned int, PixelTopology const*, unsigned int, std::map<int, SiPixelDigitizerAlgorithm::Amplitude, std::less<int>, std::allocator<std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude> > >&, unsigned short) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimTrackerSiPixelDigitizerPlugins.so
#15 0x00001456c84a757e in SiPixelDigitizerAlgorithm::induce_signal(__gnu_cxx::__normal_iterator<PSimHit const*, std::vector<PSimHit, std::allocator<PSimHit> > >, __gnu_cxx::__normal_iterator<PSimHit const*, std::vector<PSimHit, std::allocator<PSimHit> > >, PSimHit const&, unsigned long, unsigned int, PixelGeomDetUnit const*, std::vector<SiPixelDigitizerAlgorithm::SignalPoint, std::allocator<SiPixelDigitizerAlgorithm::SignalPoint> > const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimTrackerSiPixelDigitizerPlugins.so
#16 0x00001456c84aa1a9 in SiPixelDigitizerAlgorithm::accumulateSimHits(__gnu_cxx::__normal_iterator<PSimHit const*, std::vector<PSimHit, std::allocator<PSimHit> > >, __gnu_cxx::__normal_iterator<PSimHit const*, std::vector<PSimHit, std::allocator<PSimHit> > >, unsigned long, unsigned int, PixelGeomDetUnit const*, Vector3DBase<float, GlobalTag> const&, TrackerTopology const*, CLHEP::HepRandomEngine*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimTrackerSiPixelDigitizerPlugins.so
#17 0x00001456c848ae97 in cms::SiPixelDigitizer::accumulatePixelHits(edm::Handle<std::vector<PSimHit, std::allocator<PSimHit> > >, unsigned long, unsigned int, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimTrackerSiPixelDigitizerPlugins.so
#18 0x00001456c848b4e6 in cms::SiPixelDigitizer::accumulate(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimTrackerSiPixelDigitizerPlugins.so
#19 0x00001456c86d4e1b in edm::MixingModule::accumulateEvent(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimGeneralMixingModulePlugins.so
#20 0x00001456c86d4e3f in edm::MixingModule::addSignals(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimGeneralMixingModulePlugins.so
#21 0x00001456c85dbbae in edm::BMixingModule::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libMixingBase.so
...

Thread 4 (Thread 0x1456d1a00700 (LWP 9095)):
#3  <signal handler called>
#4  0x00001456c83ba013 in SiStripFedZeroSuppression::suppress(std::vector<SiStripDigi, std::allocator<SiStripDigi> > const&, std::vector<SiStripDigi, std::allocator<SiStripDigi> >&, unsigned int, edm::ESHandle<SiStripNoises>&, edm::ESHandle<SiStripThreshold>&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libRecoLocalTrackerSiStripZeroSuppression.so
#5  0x00001456c842e70b in PreMixingSiStripWorker::put(edm::Event&, edm::EventSetup const&, std::vector<PileupSummaryInfo, std::allocator<PileupSummaryInfo> > const&, int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimTrackerSiStripDigitizerPlugins.so
#6  0x00001456c837f33d in edm::PreMixingModule::put(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimGeneralPreMixingModulePlugins.so
#7  0x00001456f8a82c27 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libFWCoreFramework.so

Thread 3 (Thread 0x1456d2600700 (LWP 9094)):
#3  <signal handler called>
#4  0x00001456c8420abf in SiStripDigitizerAlgorithm::accumulateSimHits(__gnu_cxx::__normal_iterator<PSimHit const*, std::vector<PSimHit, std::allocator<PSimHit> > >, __gnu_cxx::__normal_iterator<PSimHit const*, std::vector<PSimHit, std::allocator<PSimHit> > >, unsigned long, unsigned int, StripGeomDetUnit const*, Vector3DBase<float, GlobalTag> const&, TrackerTopology const*, CLHEP::HepRandomEngine*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimTrackerSiStripDigitizerPlugins.so
#5  0x00001456c843fd0f in SiStripDigitizer::accumulateStripHits(edm::Handle<std::vector<PSimHit, std::allocator<PSimHit> > >, TrackerTopology const*, unsigned long, unsigned int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimTrackerSiStripDigitizerPlugins.so
#6  0x00001456c844029a in SiStripDigitizer::accumulate(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimTrackerSiStripDigitizerPlugins.so
#7  0x00001456c86d4e1b in edm::MixingModule::accumulateEvent(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimGeneralMixingModulePlugins.so
#8  0x00001456c86d4e3f in edm::MixingModule::addSignals(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimGeneralMixingModulePlugins.so
#9  0x00001456c85dbbae in edm::BMixingModule::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libMixingBase.so

Thread 2 (Thread 0x1456db600700 (LWP 9088)):
[cmsRun helper thread]

Thread 1 (Thread 0x1456f7735480 (LWP 8970)):
#3  <signal handler called>
#4  0x00001456c842ee24 in PreMixingSiStripWorker::put(edm::Event&, edm::EventSetup const&, std::vector<PileupSummaryInfo, std::allocator<PileupSummaryInfo> > const&, int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimTrackerSiStripDigitizerPlugins.so
#5  0x00001456c837f33d in edm::PreMixingModule::put(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimGeneralPreMixingModulePlugins.so
#6  0x00001456f8a82c27 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libFWCoreFramework.so
...
#13 0x00001456f89efa90 in edm::EventProcessor::processLumis(std::shared_ptr<void> const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#14 0x00001456f89f94e1 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#15 0x000000000040fe78 in main::{lambda()#1}::operator()() const ()
#16 0x000000000040e142 in main ()

Current Modules:

Module: DTTrigProd:simDtTriggerPrimitiveDigis (crashed)
Module: PreMixingModule:mixData
Module: MixingModule:mix
Module: MixingModule:mix
Module: PreMixingModule:mixData
Module: PreMixingModule:mixData
Module: PreMixingModule:mixData
Module: PreMixingModule:mixData

[Dr15Jones]

The full log info can be found https://cernbox.cern.ch/files/link/public/CQvjzkONaprof65?tiles-size=1&items-per-page=100&view-mode=resource-table with file vocms0253.cern.ch-283595-0-log.tar.gz

[Dr15Jones]

I see at least one (probably unrelated) bug in the code

cmssw/L1Trigger/DTTrigger/src/DTTrig.cc

Lines 227 to 231 in d376eb3

	if (iSetup.get<MuonGeometryRecord>().cacheIdentifier() != _configid) {
	if (_debug)
	std::cout << "DTTrig::updateES updating muon geometry" << std::endl;
	_geomid = iSetup.get<MuonGeometryRecord>().cacheIdentifier();

The use of _configid should be changed to _geomid at line 227.

[Dr15Jones]

So the code crashes in

cmssw/L1Trigger/DTTriggerServerTheta/src/DTTSTheta.cc

Lines 67 to 70 in d376eb3

	void DTTSTheta::setConfig(const DTConfigManager* conf) {
	DTChamberId sid = ChamberId();
	_config = conf->getDTConfigTSTheta(sid);
	}

where the only two possible bad pointers are conf (which was obtained from the EventSetup system in such a way that if it were a nullptr, the framework would have thrown an exception) or this. The this pointer is from an object in a std::map<DTChamberId, DTSCTrigUnit, std::less<DTChamberId> > and the call comes from looping over that container which should me the this is OK.

[Dr15Jones]

So I used the configurations and ran them at FNAL on a local machine interactively and the jobs processed just fine.