sbnsoftware / sbndcode Goto Github PK

Prodsingle scripts should always inherit T0, since this value is subject to change with future TPC readout and cosmic sim adjustments. Multi-particle guns currently list out the array explicitly (T0: [blah, blah..]). Is it possible to slot in a fcl parameter via @local or @sequence or @table?

The last version that I found to be working is v08_55_00.

This commit a834604 by @absolution1 changed the simulations services to address an issue with the noise service. Whilst it did manage to get back the event display, it didn't fix the Ortho3D and Display3D views.

Unfortunately the current version of sbndcode v09_08_00 introduce new changes that prevent files produced with it to be displayed with a previous version of the event display.

The single PE waveform vector from the test bench measurement is stored in sbnd_data in OpDetSim/digi_pmt_sbnd.root with name SinglePEVec.

In https://github.com/SBNSoftware/sbndcode/blob/develop/sbndcode/OpDetSim/DigiPMTSBNDAlg.cc#L48 is retrieved with name SinglePEWave, which causes a segmentation fault.

There are two distinct electronics used in SBND to readout XARAPUCAS: DAPHNE electronics and APSAIA+CAEN v1730.

Their behaviour is different, for instance their single PE response and digitisation clock.

I suggest to add a field on the json PDS map:
https://github.com/SBNSoftware/sbndcode/blob/develop/sbndcode/OpDetSim/sbnd_pds_mapping.json

Something like "electronics": "apsaia"/"daphne" It's enough to add the field on the xarapuca channels

{
  "channel": 0,
  "pd_type": "xarapuca_vis",
  "pds_box": 0,
  "sensible_to_vis": true,
  "sensible_to_vuv": false,
  "tpc": 0,
  "electronics": "daphne"
},

This issue is related to which would also need to be resolved: #78

When migrating to the refactored LArG4 in #134, and with PR #127, we switched to the Correlated light and charge model, but we didn't enable the LArQL model. The idea is to enable this for the Oct 2021 production.

The CI has picked up fails in the c7 builds coming from sbndPDMapAlg. Looks like it's a simple fix to the function. Error is:

2107: In file included from /scratch/workspace/sbnd_ci/label_exp/SLF7/label_exp2/swarm/SBND/srcs/sbndcode/sbndcode/OpDetSim/opDetDigitizerWorker.cc:4: 2108: In file included from /scratch/workspace/sbnd_ci/label_exp/SLF7/label_exp2/swarm/SBND/srcs/sbndcode/sbndcode/OpDetSim/opDetDigitizerWorker.hh:17: 2109: /scratch/workspace/sbnd_ci/label_exp/SLF7/label_exp2/swarm/SBND/srcs/sbndcode/sbndcode/OpDetSim/sbndPDMapAlg.hh:56:10: error: 'isPDType' overrides a member function but is not marked 'override' [-Werror,-Winconsistent-missing-override] 2110: bool isPDType(size_t ch, std::string pdname) const; 2111: ^ 2112: /scratch/workspace/sbnd_ci/label_exp/SLF7/label_exp2/swarm/SBND/srcs/sbncode/sbncode/OpDet/PDMapAlg.h:33:18: note: overridden virtual function is here 2113: virtual bool isPDType(size_t ch, std::string pdname) const 2114: ^

I got the following errors when running g4_no_opticalsim.fcl
ServicesManager unable to find the service of type 'phot::PhotonVisibilityService'.

The recent PR #134 which updated sbnd to the new larg4 has changed the module label for the production of sim::SimChannels for the TPC to be simdrift not largeant. This needs reflecting in the anatree configuration and module in order to run successfully. This was raised by the CI build when attempting to update the reference files as a result of the g4 update.

This issue is mainly for book-keeping, I have prepared a bugfix branch that solves this particular issue.

I have also noted that the following message is now raised once per event in the reco1 and reco2 stages.

Rebuild failed to get the SimChannels. This is expected when running on a generation or simulation step.

I haven't yet had a chance to look at exactly where this message originates from.

The standard single interaction intrnue generation fcl:

prodgenie_intrnue_singleinteraction_tpc_gsimple-configf-v1.fcl

Uses a rotated bucket configuration (inherited from prodgenie_nu_spill_tpc_gsimple-configf-v1.fcl). The equivalent bnb file does not and neither do the equivalent overlay fcls for bnb or nue.

The convention used for the optical detector channel numbers is effectively opposite to the numbering scheme of TPCs.

Effectively the convention used for channels numbers, and other identifiers is:

EVEN numbers are given to channels on the APA with x>0, and
ODD numbers given to channels on the APA with x<0.

For clarity this convention is shown in the picture below:

For the benefit of a user to select collections of optical detector, I created other indices in the map:

• pds_box: to identify all the opdets that are fixated to the same optical module
• tpc: to identify all the opdets that belong to one TPC

This last identifier is opposite to the convention sbndcode currently uses for the TPCs:

• tpc0: has x<0
• tpc1: has x>0

These contradictory conventions don't hinder the simulation and as long as everyone who uses them is aware of this asymmetry there shouldn't be any issue.

I believe the PDS map channels, nor any other identifier there, has to change to fix this issue. It would be enough to update the spreadsheet to swap convention.

I'm not knowledgable enough about how or where in the code the channel numbers are assigned to fix this myself.

reco2_sc2.fcl and reco2_3drift_windows_sce.fcl do not run in version v09_09_00 and are needed for production. When trying to run I get the following error:

cet::exception caught in art
---- Configuration BEGIN
  The following error was encountered while processing a path configuration:
  Entry flashmatch in path reco2_sce does not have a module configuration.
---- Configuration END
%MSG```

This wasn't thought an urgent change, but the flash matching was only added to reco1/reco2 and issues in that have been missed. The CI fcls and cfg need updating to run the new workflow.

As mentioned in PR #134, the prodcorsika_proton_intime_filter.fcl and g4_simphotontime_filter.fcl, will have to be reviewed to make sure the in-time and out-time paths are both running correctly. Same is true for all intime fcls.

Beam axis should be at x=-74 cm in the larsoft SBND detector coordinate system based on the latest survey measurements.
Currently it is at x=+50cm

Cross check of this number is ongoing with the data taken with the CRT panels in the pit.

From conversation with @fjnicolas

I was yesterday testing the deconvolution with the last sbndcode version and found out a bug with the PDS digitization that may affect the last production. The OpDetSim module assumes there are only two collections of SimPhotons (direct and reflected). However in the new LArG4, if the hybrid model is enabled 4 SimPhotons collections are generated: pdfastsim (direct and reflected) and pdfastsimout (direct and reflected). For instance for the uncoated PMTs, it simulates two waveforms (for the two reflected SimPhotons collections, in and out) but assigns the waveform created to the last collection read. Here an screenshot of sbndcode/OpDetSim/opDetDigitizerWorker.cc. Same logic applies to the XARAPUCAS and coated PMTs.
I’ve been discussing with Diego and have thought about these possible solutions:
(1) Straightforward fix: disable the light simulation outside the volume should fix it (in “standard_g4_sbnd.fcl” comment “ionandscintout” and “pdfastsimout” in the ‘simulate’ vector), so the digitization module only receives two photon collections. This doesn’t really solve the bug, but a least we have proper PDS signals for the light produced in the active volume as we’ve been doing thus far. This temporary fix will leave the light-simulation case identical to what we have been doing so far with the Legacy-LArG4"
(2) We’ve though about adding a producer that merges the 4 different SimPhotons collections before digitizing. In this way we can keep the logic followed by the current digitizer code and no huge changes in the module will be required. A function inside the digitization module could also do the job. I could work on this from next week (I will be on vacation the next few days)
(3) Nonetheless I think there is large room for optimization in the digitization module, but this implies changing big part of the digitization module. But maybe this a long-term task and the patch stressed in point 2 does the job by now.

In sbndcode/Commissioning/HitDumper_module.cc, line 517:

    _crt_time.push_back(ctime);
    _crt_adc.push_back(adc1 + adc2 - 127.2); // -127.2/131.9 correct for gain and 2*ped to get pe
    _crt_pos.push_back(center.Y());
    if (tagger.second == kCRTVertical) {
      _crt_pos.push_back(center.X());
    };

When the if statement "if (tagger.second == kCRTVertical)" is satisfied, the strip center position is saved twice (it first appends Y value, then X value) and the _crt_* vectors saved in the TTree have a different size.

Also in line 686 the number of CRT hits is not properly assigned (it currently loads the number of TPC hits):

if (evt.getByLabel(fCRTHitModuleLabel, crtHitListHandle))  {
  art::fill_ptr_vector(chitlist, crtHitListHandle);
  _nchits = hitlist.size();
}

Pull request #141 swapped induction planes 0 and 1 in TPC 0. This was meant to be a temporary fix to the fact that the View was not being parser correctly and was creating problems down the road. We need to fix how the View is assigned in LArSoft, so we can undo the action taken in the PR #141. Moreover, we will need to check the code added with PR #128.

I have this from a message/quick look from Joseph. Don't know if it's accurate.

1. standard_g4_sbnd.fcl --> Electronlifetime: 3000
2. standard_detsim_sbnd.fcl --> Electronlifetime: 3000
3. standard_reco2_sbnd.fcl --> Electronlifetime: 3000
4. Need to add SCE effects to SBND in-time generation
5. standard_reco2_sbnd.fcl --> "CalAreaConstants: [ 0.02354, 0.02130, 0.02354 ]" Doesn't match DocDB 18991

I'd assign this to Mike Mooney, but don't see him as a member of SBNSoftware. So, sorry @jzennamo.

When I run cafmakerjob_sbnd_sce.fcl after reco2 I get the below error. The input and log for the local test can be found here:
/sbnd/app/users/jaz8600/AugustPrep/runtest

%MSG-s ArtException:  PostEndJob 18-Aug-2021 11:50:41 CDT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- ProductNotFound BEGIN
      Found zero products matching all selection criteria
        C++ type: std::vector<recob::Track>
        Module label: 'pandoraSCETrack'
        Product instance name: ''
        Process name: (empty)The above exception was thrown while processing module MCSFitAllPID/pandoraTrackMCS run: 1 subRun: 0 event: 1
    ---- ProductNotFound END
    Exception going through path runprod
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- DictionaryNotFound BEGIN
  No dictionary found for the following classes:

       art::Assns<sbn::Stub,recob::PFParticle,void>
       art::Wrapper<art::Assns<sbn::Stub,recob::PFParticle,void> >

  Most likely they were never generated, but it may be that they were generated in the wrong package.

  Please add (or move) the specification

       <class name="MyClassName"/>

  to the appropriate classes_def.xml file.

  Also, if this class has any transient members,
  you need to specify them in classes_def.xml.
---- DictionaryNotFound END

The SBND LArSoft event display currently reads the simulation_services fcl table. The lar instance loading the event display is unable to load the noise service correctly, resulting in a crash.
I don't think the event display needs any simulation-level info, so the services block should be updated to load sbnd_services rather than sbnd_simulation_services

The view fix for the new geometry has fixed many broken features in the reconstruction but it has had the unintended effect of negatively impacting the shower dE/dx reconstruction.

The working hypothesis is that the relevant module/algs in the shower reconstruction need a similar consistency fix as introduced in PR #128.

The 2021 production freeze should not go ahead until we have understood where the issue lies in the workflow. If the problem occurs in the high-level reconstruction then this could be 'trivially' re-run post-production. If the problem occurs earlier in the workflow then this should be fixed before the production gears start spinning.

Relevant plots can be found in a recent CI run.

Make sure cosmic generation surface is high enough in Y and does not intersect with the building.

We've had a long running issue in the CI system where the nucosmics_gen test occasionally (ball park 1 in 20 runs) produces 2 neutrino interactions (2 MCTruth objects) rather than one. An example of the CI output can be found here at the end of this log.

The fcl used to run this test specifies "perEvent" and a specific seed so should theoretically always produce the same output. A few people have had brief looks at locating the issue but nothing detailed.

This issue has become more pressing now as two attempts to update reference events have failed due to the new reference files being produced with 2 MCTruth's and therefore failing the immediate ci_test checks and therefore not being replaced.

Am assigning anyone I've already discussed this with at any point for book keeping.

While testing the production for MCP 2020A, we encountered an issue where the simple flash matching (Iker's) was not producing output. We traced this to an issue where all of the OpHit's from PMT's are produced with "0" filled in every variable in each hit as seen by Iker's module.

From my view, it seems like we should hold off on production until we can confirm the integrity of the OpHits.

We have added an association between hits, MCParticles, and their IDEs. This is all the information needed for truth matching and much lighter than SimChannels. There are truth matching utilities that need to be modified to accommodate this new association

The refactored branch feature/repositoryRestructuring is branched from v08_57_00.
If that is the branch for the final refactoring, it needs to be updated to the target version for refactoring (if not, the correct branch needs the same update).
There might be other sbndcode branches that need to be integrated/merged together. I am not aware of any other.

sbndcode's top-level CMakeLists.txt does not setup the required larreco dependency i.e. find_ups_product( larreco ) is missing from CMakeLlists.txt

Bizarrely, sbndcode has always compiled just fine in isolation despite this omission. This missing dependency only seems to become apparent when simultaneously compiling sbndcode with another package that does have the larreco dependency set (e.g. larpandora).

sbndcode should explicitly state its dependency on larreco.

Reported by Varuna and Ed. OpHit times are always negative when running with a 3-drift configuration.

Following the updates to the XArapuca simulation (#133) the flash matching no longer works for the XArapucas. It has been temporarily disabled (#138) until it can be fixed.

Issue found by Francisco Javier Nicolás, thank you!
The OpHit PE calculation is done by taking the OpHit area under the waveform and then scaling by a factor Area1pePMT. When calculating the area, the sampling frequency currently used is in MHz, while it should be in GHz so as to have an area in ADC * ns, which will then be compatible with the settable Area1pePMT.

In some events, there are some OpCh with SimPhotonsLite saved but no waveforms built.

The problem is in OpDetSim/opDetDigitizerWorker.cc and how the waveforms are created. In the PMTs case, the module only creates a waveform (if conditional in lines 164-165) if there are REFLECTED photons arriving the OpChannels (direct photons are added later to the waveform, but only if there is at least one reflected SimPhoton in the channel).
There may be some events (generally those near the PDS/further from the TPB foils) in which there are plenty of direct photons arriving the PMTs, but no reflected ones. Hence no waveform is built, though there are a lot of direct photons reaching the OpCh.

While testing the production for MCP 2020A, we encountered an issue where the simple flash matching (Iker's) was not producing output. We traced this to an issue where all of the OpHit's from PMT's are produced with "0" filled in every variable in each hit as seen by Iker's module.

From my view, it seems like we should hold off on production until we can confirm the integrity of the OpHits.

in SBNDCode v09_28_01 it looks like OpT0Finder has a few issues which it either seg faults or gives the following error:
Error: Invalid optical detector type. 0 = rectangular, 1 = dome, 2 = disk. Or corrections for chosen optical detector type missing. before crashing

There are a large number of the following message when running the g4 stage since the change to the new geometry and refactored larg4:

%MSG-w SimDriftElectrons:  SimDriftElectrons:simdrift@BeginModule  06-Sep-2021 02:29:44 CDT run: 1 subRun: 0 event: 1
unable to drift electrons from point (117.305,203.314,1.21832) with exception ---- Geometry BEGIN
Can't find nearest wire for position (201.75,203.385,1.27117) in plane C:0 T:1 P:1 approx wire number # 0 (capped from -17)
---- Geometry END
%MSG

If this isn't actually an issue, the messages need to be suppressed as they are swamping everything else --- e.g. there is ~700,000 lines of this in a recent CI test log: https://dbweb8.fnal.gov:8443/LarCI/app/ns:sbnd/storage/docs/2021/09/06/stdout%23zPg0ktQ.log

Looking into issue #56, I was getting the following exception:

%MSG-s ArtException:  Early 28-Oct-2020 10:42:08 CDT JobSetup
cet::exception caught in art
---- OtherArt BEGIN
  ServiceCreation
  ---- ServiceNotFound BEGIN
    Unable to create ServiceHandle.
    Perhaps the FHiCL configuration does not specify the necessary service?
    The class of the service is noted below...
    ---- ServiceNotFound BEGIN
      ServicesManager unable to find the service of type 'art::TFileService'.
    ---- ServiceNotFound END
  ---- ServiceNotFound END
  cet::exception caught during construction of service type SBNDuBooNEDataDrivenNoiseService:
---- OtherArt END
%MSG

@ascarff suggests there's some redundancy about histograms no longer needed. Some code cleanup might be needed.

Get bottom panel of CRT into SBND geometry

Currently we have ScintillationByParticleType: false set in our G4 stage, I think this means we aren't correctly modeling our prompt to late-light ratios for the different particle types

With LArSoft v9, changes to sbndcode have been made to comply with the new Services structure. The DetectorClocksData, used in the optical simulation code to get the optical clock, is set in the constructor of the LArSoft plugin opDetDigitizerSBND, and then passed to the digitizer worker via a reference wrapper std::cref. Such reference gets broken as the DetectorClocksData only exists in the constructor.

The result is that the current optical simulation code is broken and doesn't produce any waveform, as the optical clock returns inf when asked for the clock frequency.

Per #134, At minimum, new LArG4/geometry needs new reference files for standard fhicls to pass in CI:

• single_detsim_quick_test_sbndcode
• nucosmics_detsim_quick_test_sbndcode

Likely we want to do a more complete update of reference files.

Assigning to @chilge for now, who can I hope ask for updates elsewhere...

As discussed offline, I have discovered a seg fault issue when running the g4 stage in sbndcode v09_28_01 onwards. This corresponds to the migration to the new larg4 via PR #134. The seg fault occurs sporadically in genie samples (~1/600 events) but far more often in an overlay sample (~1/50 events). This wasn't picked up by the CI due to its low-ish frequency.

The back trace leads to the following line in a larsim file used by mcreco. The segfault occurs when trying to read a position from an MCParticle.

Adding print statements and looking at a single example I found the MCParticle in question to be a muon which could yield the following information:
Track ID: 158 PDG: 13 Mother: 0 Process:
before segfaulting.

Reading further up the output, this muon with the same track ID exists earlier, i.e. it has already been looped over. However, previously it does contain a process (primary) and doesn't seg fault when asked for its position.

These seg faults are occurring using either the standard_g4_sbnd.fcl or g4_sce.fcl.

PR #141 disabled the WireIntersection test from the geometry tests. This test will need to be re-enabled once LArSoft issue 26127 is solved. We'll also need to check if we need or not all the other disabled tests:

With the new geometry v02_00, the number of CRT auxiliary detectors returned by the GeometryService is 0. This likely has to do with the fact that the physvol name doesn't start with volAuxDet.

To quickly verify this behavior, with sbndcode set up:

python
import SBNDservices as serv
geom = serv.ServiceManager('Geometry')
geom.NAuxDets()

This returns:

• 142 with sbndcode v09_24_00 (geometry v01_05).
• 0 with sbndcode v09_28_00 (geometry v02_00).

Pull request #134 marks the switch to the refactored LArG4. The CRT is excluded by the detsim stage because it needs an extra module to parse the new larg4 output.

The pedestal info in detsim was updated within the last 1/2 year (collection ped==650) but that change did not propagate to the pedestal service (collection ped==400).
The SBND LArSoft event display, by default, reads pedestal information from the database which means you get a very angry raw event display with a baseline of 650-400==250 ADU.

Other areas of sbndcode, which require pedestal use, take their information directly from the raw::RawDigit whose pedestal info is calculated in the detsim stage which uses the firstly mentioned pedestal value. So, internally, a consistent pedestal value is subtracted in the places that really matter.
However, I think we should only maintain one set of pedestal values.
We can set the pedestal values in the pedestal service and point the detsim fcl param set to those values.

All genie fcl files should set GENIEGen parameter MaxFluxFileMB to limit the number of flux files that are opened to a reasonable number.

I observed this issue a while back, when working on the light simulation. Actually I opened an issue in LArSoft redmine: https://cdcvs.fnal.gov/redmine/issues/24430

The X position of the cathode is wrong. It reports to be at +- 2.6 cm. It should be something closer to 0 cm.

TPC: 0
MinX():    -200
MaxX():    -2.6

TPC: 1
MinX():    2.6
MaxX():    200

I don't know if this is an issue with the geometry functions upstream, the way the SBND geometry is parsed or the actual geometry file.

There's at least two directories containing filters in SBND:

• Filters
• SimulationFilters

The filters contained in these directories (and other directories) should be combined in a single directory and we should similarly define all of the FCL parameter sets in filters_sbnd.fcl

Not a high priority, but something to keep in mind.

Currently the digitiser clock of PMTs and (X)ARAPUCAS is set to 500 MHz. That's the clock of the CAEN v1730. However the DAPHNE electronics works at 80MHz.

There's a set of XARAPUCAS on the middle of the detector that are connected to APSAIA electronics; preamplifiers and power supplies served by a microcontroller, these are also digitised by the CAEN v1730 boards an so have the same clock as PMTs at 500 MHz.

This is set in this fcl file:

We access and use this value with an incantation that looks like this:

auto const clockData = art::ServiceHandle<detinfo::DetectorClocksService const>()->DataForJob();
double fSampling = clockData.OpticalClock().Frequency(); // MHz

It comes from: https://github.com/LArSoft/lardataalg/blob/10b7a5bba3e236475f5eff83097f25686be24564/lardataalg/DetectorInfo/DetectorClocksData.h#L403

This clock is necessary to simulate the digitised waveforms:

params.frequency = 0.16 * clockData.OpticalClock().Frequency();

Another instance where is relevant is:
https://github.com/SBNSoftware/sbndcode/blob/develop/sbndcode/OpDetSim/opDetSBNDTriggerAlg.cc
Where similar hacks could be made.

The other place where the clock is relevant is for OpHitFinding:
https://github.com/SBNSoftware/sbndcode/blob/develop/sbndcode/OpDetReco/OpHit/SBNDOpHitFinder_module.cc
There's no easy workaround or hack that I can think of to sort that.

I'm not advocating for hacks or quick work arounds, rather I'm highlighting the complexity of solving this issue.

I don't have time myself to fix this, hopefully someone will find time to tackle this one.

Currently it looks as if prodcorsika_proton_intime_filter.fcl is trying to call fcls that no longer exist after the migration to the new LArG4