Rename all cdscan errored/warning files to "*latestX-WARN.xml" about cmip5

[durack1]

This info from Mark Z is useful to consider whilst implementing changes:

Ok so it appears that data exists, but it throws a warning as the directory contains duplicated data:
[durack1@oceanonly ~]$ ls /cmip5_css01/scratch/cmip5/output1/IPSL/IPSL-CM5A-LR/abrupt4xCO2/mon/atmos/Amon/r1i1p1/v20110921/hus
hus_Amon_IPSL-CM5A-LR_abrupt4xCO2_r1i1p1_185001-191912.nc
hus_Amon_IPSL-CM5A-LR_abrupt4xCO2_r1i1p1_185001-194912.nc
hus_Amon_IPSL-CM5A-LR_abrupt4xCO2_r1i1p1_195001-204912.nc
hus_Amon_IPSL-CM5A-LR_abrupt4xCO2_r1i1p1_205001-210912.nc

These weird quirks are the reason I suggested changing the behavior of the script to keep the “warning” files (my blurb below..) You’ve just highlighted another case that would appear to be fine.. Just need someone to go and delete deprecated/duplicated files.. Unless of course you can think of some intelligent way of coding around such problems..?

I’ll add this example to my “wish list” of bug fixes/enhancements.. I’d hope first to switch the default delete behavior to “WARN” as noted below, and if/when I have time, then go through these cases and figure out if they can be automagically resolved..

Cheers for the heads up..

P

-----Original Message-----
From: , Mark [email protected]
Date: Thursday, September 4, 2014 at 5:38 PM
To: "Paul J. Durack" [email protected]
Subject: Re: August xml update

Hi Paul,

Thanks for your hard work on this. Everything looks like it is there to
me. One thing that confuses me is that, for IPSL-CM5A-LR abrupt4xCO2 hus
& ta, there are data from both the cfMon and Amon tables, though in some
cases (r1i1p1) only the cfMon data are available.

Mark

[durack1]

Should be implemented in d636eab - Will await testing and validation of the case documented above before closing

Additionally the logfile warn code (1-5) is also added, so the user (if they have an idea of the cdscan warning list) should be able to ascertain what the issue was quickly

[durack1]

Only error code 2 will yield a file, possibly 1 may yield a file, however this will point to 0-sized data.. Potentially re-labelling all WARN2 files into their associated problems, time axis, lat/lon axis etc etc.. Might have to look at cdscan to see what the warning list includes

[durack1]

The following warnings are thrown:

[durack1@oceanonly ~]$ more /work/cmip5/_logs/140919_120004_make_cmip5_xml-oceanonly-threads40-PID3284.log | grep 'PROBLEM 2' > Desktop/140926_cdscanWarn2.txt
PROBLEM 2 (cdscan error - 'Warning: Axis values for axis time are not monotonic') indexing /cmip5_css02/scratch/cmip5/output1/NCC/NorESM1-M/historical/mon/ocean/Omon/r3i1p1/v20110901/thetao
PROBLEM 2 (cdscan error - 'Warning: resetting latitude values') indexing /cmip5_css01/scratch/cmip5/output1/MRI/MRI-ESM1/historical/mon/ocean/Omon/r1i1p1/v20130307/wfo
PROBLEM 2 (cdscan error - 'Warning, file zg_Amon_IPSL-CM5A-LR_historicalNat_r3i1p1_195001-200512.nc, dimension time overlaps file zg_Amon_IPSL-CM5A-LR_historicalNat_r3i1p1_195001-201212.nc'
PROBLEM 2 (cdscan error - 'Warning, file zg_Amon_HadGEM2-ES_piControl_r1i1p1_217312-219811.nc, dimension time contains values in file zg_Amon_HadGEM2-ES_piControl_r1i1p1_185912-188411.nc

[durack1]

There also appears to be unreported issues with incomplete time dimensions which are not throwing warnings: crunchy (as reported by Kate Marvel)
https://gist.github.com/durack1/c1b71bc1e06895b59d2e#file-badfiles-txt

[durack1]

@pochedls another reincarnation in cmiplib candidate? If yes, I'll copy it across

[pochedls]

I've implemented a new warning system - we can discuss and open a separate ticket if need be.

[durack1]

@pochedls great, we can chat about this face-to-face then