Input and Output folder structure for new preprocessor/backend about esmvaltool

[jvegreg]

I see now that my suggestion was not correctly written in my comment, for some reason.

Anyway, I still think that we should split and reverse the namelist and date part from the path:

<output_dir>/<namelist>/YYYYMMDD_HHMMSS/preproc/

This will make a lot easier to search for outputs if you run multiple namelists multiple times

If we don't want to add another nesting level, I think we should at least switch the order.

<output_dir>/<namelist>_YYYYMMDD_HHMMSS/preproc/

[mattiarighi]

Ready to merge (PR167)

The output of the perfmetrics namelist now looks like this:

<output_dir>/tmp
<output_dir>/tmp/ANNUAL_CYCLE_ta
<output_dir>/preproc
<output_dir>/preproc/CMIP5
<output_dir>/preproc/OBS
<output_dir>/work
<output_dir>/work/perfmetrics_main
<output_dir>/plots
<output_dir>/plots/perfmetrics_main

Don't forget to update your config-user.yml file!

[nielsdrost]

New plan:

Have two folder, one for really "temporary files", and one for output.

temporary files default to /tmp

the output folder will have further subfolders automatically generated for plots, netdf output, preprocessed files, etc.

[bouweandela]

To create a temporary dir, it is possibly best to use the standard library function tempfile.mkdtemp

[bouweandela]

At the moment, if the output directory already exists, it is moved to some other name and a new directory is created. This can cause nasty surprises for users, who suddenly find their files moved. Of course this can be remedied by generating a new config(-user).yml file for every run, but that is not very convenient.

A better setup would be the following:
Inside the run_dir defined in config.yml, create a directory with the name of the namelist and current datetime, e.g. for a run starting on 2017-09-25 15:48:03 UTC, create /path/to/run_dir/20170925_154803_namelist_MyVar and put all output that goes to run_dir in there.

For the other paths specified in config.yml:

• preproc_dir
• work_dir
• plot_dir
  If they are relative paths, put them inside the directory mentioned above, e.g.
  /path/to/run_dir/20170925_154803_namelist_MyVar/preproc_dir
  /path/to/run_dir/20170925_154803_namelist_MyVar/work_dir
  /path/to/run_dir/20170925_154803_namelist_MyVar/plots_dir
  if they are absolute paths, create a namelist + current datetime subfolder inside them too, like so:
  /path/to/preproc_dir/20170925_154803_namelist_MyVar
  /path/to/work_dir/20170925_154803_namelist_MyVar
  /path/to/plot_dir/20170925_154803_namelist_MyVar

This is convenient and minimizes the risk of overwriting files.

Edge case: it seems very unlikely that two namelists are started at the exact same second, but if this happens, we can add _1, _2, etc to the name, or raise an exception, TBD.

[valeriupredoi]

/run_dir/DATE_TIME_namelist_userTag should be easy to do - I'll fix this tomorrow; I'll added a userTag (string) to be defined in config-user because we may well have different variations of the same namelist in the future, or the user will alter the namelist (again, in the future). Cheers, V

…

On Mon, Sep 25, 2017 at 3:03 PM, Bouwe Andela ***@***.***> wrote: At the moment, if the output directory already exists, it is moved to some other name and a new directory is created. This can cause nasty surprises for users, who suddenly find their files moved. Of course this can be remedied by generating a new config(-user).yml file for every run, but that is not very convenient. A better setup would be the following: Inside the run_dir defined in config.yml, create a directory with the name of the namelist and current datetime, e.g. for a run starting on 2017-09-25 15:48:03 UTC, create /path/to/run_dir/20170925_154803_namelist_MyVar and put all output that goes to run_dir in there. For the other paths specified in config.yml: - preproc_dir - work_dir - plot_dir If they are relative paths, put them inside the directory mentioned above, e.g. /path/to/run_dir/20170925_154803_namelist_MyVar/preproc_dir /path/to/run_dir/20170925_154803_namelist_MyVar/work_dir /path/to/run_dir/20170925_154803_namelist_MyVar/plots_dir if they are absolute paths, create a namelist + current datetime subfolder inside them too, like so: /path/to/preproc_dir/20170925_154803_namelist_MyVar /path/to/work_dir/20170925_154803_namelist_MyVar /path/to/plot_dir/20170925_154803_namelist_MyVar This is convenient and minimizes the risk of overwriting files. Edge case: it seems very unlikely that two namelists are started at the exact same second, but if this happens, we can add _1, _2, etc to the name, or raise an exception, TBD. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#15 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AbpCo3qSdZzK9TItdjw5SgMQizRdDpxIks5sl7KsgaJpZM4L6uuK> .

-- Dr. Valeriu Predoi Computational Scientist for UKESM Core Team Department of Meteorology, University of Reading Earley Gate, Office 1U08 READING, RG6 6BB United Kingdom Mobile number: 07847416092 "If one day you be questioning your ability to come up with professional results, think of this: Noah's ark was built by farmers whereas the Titanic was crafted by skilled engineers"

[mattiarighi]

I find the structure of the output directory a bit confusing. An unexperienced user may have problems finding the output.

I think we should have only 2 output paths in config (as originally suggested): one for the output (output_dir), containing the subdirs preproc, work and plots, and one for the temporary files (run_dir or tmp_dir), containing a subdir with the diagnostic id (I don't think we need to have another subdir interface_data here).

The directory structure should look like

<output_dir>/YYYYMMDD_HHMMSS_<namelist>/preproc/
<output_dir>/YYYYMMDD_HHMMSS_<namelist>/work/
<output_dir>/YYYYMMDD_HHMMSS_<namelist>/plots/

<tmp_dir>/YYYYMMDD_HHMMSS_<namelist>/<diag_id>/

There is no risk of overwriting files in work and plots, as every diagnostic in the namelist produces different ones. The advantage would be that the files in preproc could be recycled, for example when 2 variables in the same namelist are processed using the same preproc_id.

One problem I could foreseen is if we are going to allow for parallel execution of the diagnostics from the same namelist. But in that case we would need anyway to sort out in advance the diagnostics using the same preproc_id, to allow for recycling the preprocessed file, and also to manage the dependencies (i.e., diagnostics which needs other diagnostics to be run first).

[jvegreg]

We also have the issue that we can not use previously preprocessed files with any of this structures...
... but anyway, I also think that the preproc dir should be in the output folder.

One small thing: I think we should also change the folder to the following structure
<output_dir>//YYYYMMDD_HHMMSS /

My output folder is getting too many folders and also becomes difficult to find which ones correspond to a given namelist.

[mattiarighi]

We also have the issue that we can not use previously preprocessed files with any of this structures...

Right, only within the same namelist. But I think this is OK for the moment: recycling of preproc files across different namelists shouldn't be a very common case (@axel-lauer ?).

One small thing: I think we should also change the folder to the following structure
<output_dir>//YYYYMMDD_HHMMSS /

Fine with me.

[valeriupredoi]

Hey guys, this has been changed and changed again from my initial implementation back in August (me, Bouwe, then me again, Mattia...) :P Can we decide on a viable structure and keep it? I got no preferences apart from the ones that should not overwrite already produced files, so we keep track of runs and their output. Cheers, V

…

On Fri, Dec 1, 2017 at 3:34 PM, Mattia Righi ***@***.***> wrote: We also have the issue that we can not use previously preprocessed files with any of this structures... ... but anyway, I also think that the preproc dir should be in the output folder. Right, only within the same namelist. But I think this is OK for the moment: recycling of preproc files across different namelists shouldn't be a very common case ***@***.*** <https://github.com/axel-lauer> ?). One small thing: I think we should also change the folder to the following structure <output_dir>//YYYYMMDD_HHMMSS / Fine with me. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#15 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AbpCowPucKKkZ4y5FQTczME4lW7ekxG0ks5s8Bx6gaJpZM4L6uuK> .

-- Dr. Valeriu Predoi Computational Scientist for UKESM Core Team Department of Meteorology, University of Reading Earley Gate, Office 1U08 READING, RG6 6BB United Kingdom Mobile number: 07847416092 "If one day you be questioning your ability to come up with professional results, think of this: Noah's ark was built by farmers whereas the Titanic was crafted by skilled engineers"

[mattiarighi]

As discussed in the telecon, we now go for 1 user-specified output path (output_dir), containing the following:

<output_dir>/YYYYMMDD_HHMMSS_<namelist>/preproc/
<output_dir>/YYYYMMDD_HHMMSS_<namelist>/work/
<output_dir>/YYYYMMDD_HHMMSS_<namelist>/plots/
<output_dir>/YYYYMMDD_HHMMSS_<namelist>/tmp/<diag_id>/

The path is mandatory.

[valeriupredoi]

Yuup, agreed with this one, Javier! Cheers, V

…

On Tue, Dec 5, 2017 at 3:16 PM, Javier Vegas-Regidor < ***@***.***> wrote: I see now that my suggestion was not correctly written in my comment, for some reason. Anyway, I still think that we should split and reverse the namelist and date part from the path: <output_dir>/<namelist>/YYYYMMDD_HHMMSS/preproc/ This will make a lot easier to search for outputs if you run multiple namelists multiple times If we don't want to add another nesting level, I think we should at least switch the order. <output_dir>/<namelist>_YYYYMMDD_HHMMSS/preproc/ — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#15 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AbpCo5mhp5eUGWvMtk2qn0S81HPBugYHks5s9V44gaJpZM4L6uuK> .

-- Dr. Valeriu Predoi Computational Scientist for UKESM Core Team Department of Meteorology, University of Reading Earley Gate, Office 1U08 READING, RG6 6BB United Kingdom Mobile number: 07847416092 "If one day you be questioning your ability to come up with professional results, think of this: Noah's ark was built by farmers whereas the Titanic was crafted by skilled engineers"

[nielsdrost]

Sound good. I have a slight preference for the "combined" format, but not by much.

<output_dir>/<namelist>_YYYYMMDD_HHMMSS/

[mattiarighi]

I'd also like to avoid too much nesting.

[valeriupredoi]

Oh can we put a 'cowsay' suffix there as well? :)) Cheers, V

…

On Tue, Dec 5, 2017 at 4:12 PM, Niels Drost ***@***.***> wrote: Sound good. I have a slight preference for the "combined" format, but not by much. <output_dir>/<namelist>_YYYYMMDD_HHMMSS/ — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#15 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AbpCo3oNSMKNaft21mZs--r2E5Dr4vWkks5s9Wt5gaJpZM4L6uuK> .

-- Dr. Valeriu Predoi Computational Scientist for UKESM Core Team Department of Meteorology, University of Reading Earley Gate, Office 1U08 READING, RG6 6BB United Kingdom Mobile number: 07847416092 "If one day you be questioning your ability to come up with professional results, think of this: Noah's ark was built by farmers whereas the Titanic was crafted by skilled engineers"