Comments (21)
metson: Milestone T0 2_0_0 deleted
from t0.
hufnagel: Include the job splitting parameters in the WMSpec, including the jobNamePrefix, which should be "Repack-Run-Stream".
from t0.
hufnagel: To get the ball rolling on this, there are open questions about a missing Config.DP method that gives you a repack config and also in general about how to configure the multiple outputs from repacking.
For the first version of this, you could just use a merge configuration for both the repack and the repackmerge part. The 'repack' job would get a merge configuration from Config.DP with a single output (single dataset), but otherwise it would be configured like a processing job, with normal unmerged output and support for direct-to-merge. We can also use this for developing the support for the error datasets (two output modules in the spec, one for normal, one for error dataset and selecting one of them at runtime based on information passed from the job splitter).
from t0.
hufnagel: Another thing we could also already cover with this 'fake' repack spec is #3124.
from t0.
evansde: I have commit rights in Conf/DP, lets just write a repacking method in there and go with that.
def repack(self, whatArgsGoeHere, *streamers):
process = cms.Process("Repackappottamus")
...
return process
from t0.
sfoulkes: Working on this. There are a couple things I guess about that we need to firm up:
- Format of this "selectEvents" data structure. I assumed it was a dict where each key is the primary dataset name. The value is a list of select events statements (I think?). The whole dict is passed to the repack Conf/DP method at runtime.
- The names of the Repack and RepackMerge job splitting algorithms. I have this hard coded as "T0.Repack" and "T0.RepackMerge". Dirk added code to the JobCreator to allow the use of non-wmcore job splitting algorithms.
- The names of the output modules that the Conf/DP method generates. I have it as primaryDataSetName + "output"
I'll have to do 2 patches, one to bring the Repack spec in the T0 repo up to date and one to remove all the assumptions we have in WMBase that all workflows only produce a single primary dataset.
from t0.
sfoulkes: We should also fix 2949 and verify that the dataset naming in StdBase is OK.
from t0.
sfoulkes: I'm also unsure of how to error datasets work. I set the merge task up to have two output modules: the regular one and a "MergedError" output module. The primary dataset in the MergeError module has "Error" appened to it. One of these will be turned off at runtime depending on the size of the input data.
from t0.
sfoulkes: The WMSpec stuff doesn't like complex types as values, so my SelectEvents datastructure isn't going to fly. We'll have to make the repack method in Conf/DP something like:
def repack(self, globalTag, **selectEvents)
from t0.
hufnagel: There is already support in ConfigBuilder to pass in dictionaries to select output modules. I haven't gotten around to testing it yet though. Would look something like this
outputs = [ { 'dataTier' : 'RECO',
'selectEvents' : 'HLT:path1, HLT:path2' },
{ 'dataTier' : 'RECO',
'selectEvents' : 'HLT:path3, HLT:path4' } ])
However we package this, the content needs to be passed to Config.DP. That is , a list of output modules and for each output module the tier and the selectEvents string.
from t0.
hufnagel: Btw, the determination when to use the errorDataset output isn't on size primarily.
Output only goes to the error dataset if we have to breakup a lumi and the decision whether to breakup a lumi is done at job scheduling time. When that decision is made, the information somehow needs to be stored with the job and passed to the runtime environment.
from t0.
sfoulkes: Replying to [comment:19 hufnagel]:
Btw, the determination when to use the errorDataset output isn't on size primarily.
Output only goes to the error dataset if we have to breakup a lumi and the decision whether to breakup a lumi is done at job scheduling time. When that decision is made, the information somehow needs to be stored with the job and passed to the runtime environment.
This decision happens in the RepackMerge splitting algo, right? The spec creates the RepackMerge jobs with normal output modules and error output modules. Then one of the modules is turned off at runtime. We don't have to do anything with error datasets in the Repack tasks themselves, right?
from t0.
hufnagel: Yes, the decision is made in the repackmerge splitting algo, because only it has the relevant Tie0configuration related parameters (whether to split at all and at what thresholds) and the information about which files belong to which lumi and the sizes of each.
The actual repackmerge job itself and it's CMSSW configuration has one output module, the only thing we need to change at runtime is whether the resulting file is accounted for the normal dataset or the error dataset.
Not sure what the best way to do this is. I though (from our previous discussions) that we could overload the single CMSSW output with two output definitions in the spec, one for the normal and one for the error dataset and then remove the one we do not need at runtime (or at the very least only use the one we need).
from t0.
sfoulkes: Initial Repack Spec is attached. There's not a lot going on, almost everything is already handled by the setupProcessingTask() and setupMergeTask() methods in StdBase. Questions for Dirk:
- Verify that the inputs to the spec are as they should be. It's pretty much a ScramArch, CMSSW version, Acquisition Era, parameters to be passed to the job splitting and the SelectEvents data structure.
- We create a single output module for each primary dataset in the SelectEvents data structure. That whole thing is passed to the Config.DP method which we'll have to better define.
- This doesn't support ConfigCache configs, should probably be added.
- Each merge task has two output modules, a regular and an error. We'll have to decide on the best way to handle this.
from t0.
sfoulkes: Second patch contains changes to WMCore. It's about 50% cleanup, 25% better support for more involved Config.DP configs and 25% support for the more elaborate T0 merging. Dave, could you review this?
from t0.
sfoulkes: (In 15201) Modify StdBase so that it doesn't assume that all workflows have only run
over a single primary dataset. Modify the addMergeTask() method to support
error datasets. Minor cleanup in the other specs. Fixes #1796.
From: Steve Foulkes [email protected]
from t0.
hufnagel: Still need to look at the changes in the T0 code.
from t0.
hufnagel: De-scope this a bit to get something working more quickly. The version attached is a fully featured repacking, but instead of the repack merge with the active split lumi protections and with error dataset support I am using a standard merge for now.
from t0.
hufnagel: Please review (both #1796 and #3578) by checking that the RunConfig and Tier0Feeder unit tests work
from t0.
mnorman: Tested in conjunction with #3578
from t0.
hufnagel: (In eae1a75) Create Repack WMSpec, fixes #1796
Signed-off-by: Dirk Hufnagel [email protected]
from t0.
Related Issues (20)
- Cleanup script needs to check if run is active or not HOT 11
- change the wmbs_location_senames table HOT 2
- Report Repacked Files For Unclosed Runs HOT 1
- fix Express AlcaSkim task to use single core HOT 1
- Run tests in CI
- Where is the real Tier0 config? HOT 6
- Create the change for Tier0 on PNN, PSN mapping. HOT 2
- Consider throwing in case of duplicate data set definitions in Tier0 config
- Overriding SoftTimeout and GracePeriod ? HOT 25
- Fileset closing is not waiting for the data in the run_stream_fileset to be complete HOT 6
- Versioning T0 replay runs when the same run is used more than once HOT 9
- Condition_t is not working properly HOT 1
- Tier0 and Rucio Subscriptions HOT 3
- SQL queries return unicode strings while CMSSW only accepts byte strings HOT 7
- Open files/urls with context manager HOT 3
- Update the T0 deployment script with the Rucio project metadata value HOT 1
- Replay test with 12_1_0_pre3 HOT 10
- Post description of Replay PR to HN
- Clean up Heavy Ion Config files HOT 2
- version #100 is not clear HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from t0.