Comments (8)
This info from Mark Z is useful to consider whilst implementing changes:
Ok so it appears that data exists, but it throws a warning as the directory contains duplicated data:
[durack1@oceanonly ~]$ ls /cmip5_css01/scratch/cmip5/output1/IPSL/IPSL-CM5A-LR/abrupt4xCO2/mon/atmos/Amon/r1i1p1/v20110921/hus
hus_Amon_IPSL-CM5A-LR_abrupt4xCO2_r1i1p1_185001-191912.nc
hus_Amon_IPSL-CM5A-LR_abrupt4xCO2_r1i1p1_185001-194912.nc
hus_Amon_IPSL-CM5A-LR_abrupt4xCO2_r1i1p1_195001-204912.nc
hus_Amon_IPSL-CM5A-LR_abrupt4xCO2_r1i1p1_205001-210912.nc
These weird quirks are the reason I suggested changing the behavior of the script to keep the “warning” files (my blurb below..) You’ve just highlighted another case that would appear to be fine.. Just need someone to go and delete deprecated/duplicated files.. Unless of course you can think of some intelligent way of coding around such problems..?
I’ll add this example to my “wish list” of bug fixes/enhancements.. I’d hope first to switch the default delete behavior to “WARN” as noted below, and if/when I have time, then go through these cases and figure out if they can be automagically resolved..
Cheers for the heads up..
P
-----Original Message-----
From: , Mark [email protected]
Date: Thursday, September 4, 2014 at 5:38 PM
To: "Paul J. Durack" [email protected]
Subject: Re: August xml update
Hi Paul,
Thanks for your hard work on this. Everything looks like it is there to
me. One thing that confuses me is that, for IPSL-CM5A-LR abrupt4xCO2 hus
& ta, there are data from both the cfMon and Amon tables, though in some
cases (r1i1p1) only the cfMon data are available.
Mark
from cmip5.
Should be implemented in d636eab - Will await testing and validation of the case documented above before closing
Additionally the logfile warn code (1-5) is also added, so the user (if they have an idea of the cdscan warning list) should be able to ascertain what the issue was quickly
from cmip5.
Only error code 2 will yield a file, possibly 1 may yield a file, however this will point to 0-sized data.. Potentially re-labelling all WARN2
files into their associated problems, time axis, lat/lon axis etc etc.. Might have to look at cdscan to see what the warning list includes
from cmip5.
The following warnings are thrown:
[durack1@oceanonly ~]$ more /work/cmip5/_logs/140919_120004_make_cmip5_xml-oceanonly-threads40-PID3284.log | grep 'PROBLEM 2' > Desktop/140926_cdscanWarn2.txt
PROBLEM 2 (cdscan error - 'Warning: Axis values for axis time are not monotonic') indexing /cmip5_css02/scratch/cmip5/output1/NCC/NorESM1-M/historical/mon/ocean/Omon/r3i1p1/v20110901/thetao
PROBLEM 2 (cdscan error - 'Warning: resetting latitude values') indexing /cmip5_css01/scratch/cmip5/output1/MRI/MRI-ESM1/historical/mon/ocean/Omon/r1i1p1/v20130307/wfo
PROBLEM 2 (cdscan error - 'Warning, file zg_Amon_IPSL-CM5A-LR_historicalNat_r3i1p1_195001-200512.nc, dimension time overlaps file zg_Amon_IPSL-CM5A-LR_historicalNat_r3i1p1_195001-201212.nc'
PROBLEM 2 (cdscan error - 'Warning, file zg_Amon_HadGEM2-ES_piControl_r1i1p1_217312-219811.nc, dimension time contains values in file zg_Amon_HadGEM2-ES_piControl_r1i1p1_185912-188411.nc
from cmip5.
There also appears to be unreported issues with incomplete time dimensions which are not throwing warnings: crunchy (as reported by Kate Marvel)
https://gist.github.com/durack1/c1b71bc1e06895b59d2e#file-badfiles-txt
from cmip5.
@pochedls another reincarnation in cmiplib candidate? If yes, I'll copy it across
from cmip5.
I've implemented a new warning system - we can discuss and open a separate ticket if need be.
from cmip5.
@pochedls great, we can chat about this face-to-face then
from cmip5.
Related Issues (20)
- Compress logs/pkl files for xml runs
- make_cmip5_xml.py: Correct archived file extension
- Update make_cmip5_xml.py to deal with non-warning errors
- Update make_cmip5_xml.py to index energy budget terms
- logFiles index error
- make_cmip5_xml.py - modularize, extract functions and rewrite driver file HOT 2
- Fix hyphenated spliced directory purge attempt HOT 2
- Add mrso and equivalent land variables to cmip5 xml list
- make_cmip5_xml.py: Move mrsofc & sftgif from land_vars to fx_var list HOT 1
- make_cmip5_xml.py - Add $EXP/land/fx_new cleanup
- make_cmip5_xml.py - Add new experiments/variables
- Faulty DRS-related failure: make_cmip5_xmls.py HOT 3
- make_cmip5_xml.py - intelligent sort duplicates to prefer /data files HOT 3
- make_cmip5_xml.py - Add new variables
- Deal with splice directories (historical-rcp85) better HOT 3
- Migrate scans to CSS03 hardware HOT 1
- Update make_cmip5_xmls preparing for CMIP6 HOT 2
- Update dictionary format - make_cmip5_xml.py
- Update spawninfo to work with conda envs
- Shrink repos
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cmip5.