ncar / neon-visualization

Repository to include all neon-related visualization scripts.

NEON-visualization

Repository to include all neon-related visualization scripts.

---

To install all required libraries:

conda env create --name neon -f environment.yml
conda activate neon

---

bokeh_script folder:
Includes the stand-alone script for running the diurnal cycle outside the notebook: to show case stand-alone application.

First, load your environment. Next, run the following:

python -m bokeh serve  --show ./02-Neon_Evaluation_diel_cycle_dev.py  --port 5012

neon-visualization's Issues

Binder functionality

Are we going to need the binder functionality here? I assume not as it was never included in our teachings or our paper. If not, I prefer removing it altogether.

Visualization priorities

I'm just linking this repo with a similar issue that's up for CTSM. Maybe it's easier to migrate the CTSM issue to this repo for clarity, @negin513 ?

Cleaning up this reposiotry

Hello Everyone,
We need to update and maintain this repository appropriately. Currently, we have identified a few items that should not be in the repository, including at least the following:

• The .Rproj directory
• NEON-visualization.Rproj
• Conflicting requirement.txt and yml files.
• The sample_data, which is no longer needed.

I am uncertain about the purpose of some directories and files in the repository, such as the .Rproj directory and files.

However, from my understanding the .Rproj file is typically used as a project file by the R programming language in the RStudio integrated development environment (IDE).
If it is not necessary for our repository, I plan to remove it to avoid any potential confusion or issues.

Parameter Sensitivity Workflow

Since the NEON run is so efficient (less than 10 minutes on my laptop); it will make sense to add parameter sensitivity workflow building on the CLM5 PPE experiment.

soil profile plot function errors

I encountered two error messages for us to consider:

1. An error message in the plot_soil_profile_timeseries function in the NEON_Simulation_Tutorial:

'PCT_SAND' is not present in all datasets.

I encountered this error for the WOOD site in 2018 for both soil temperature and soil moisture, although there are no plotting errors for this site in 2019 and 2020.

Is 'PCT_SAND' required for this function to plot properly? Does this suggest that 'PCT_SAND' is not included on the surface dataset?

---

2. An error message when plotting incomplete years:

index 24 is out of bounds for axis 0 with size 1

Should we consider enabling plotting of partial years of data?

NEON plots as html don't render properly on github.

maybe this isn't a big issue, as they an be viewed once you download the repo locally, but I wonder if we should post plots that look nice on github?

Updates to run_neon tutorial

A few things that we can improve in the tutorial:

1. running the 'run_neon' code block ends with a 'data not found' line and it isn't evident when it is finished running. Adding a print statement at the end that says 'simulation submitted to queue' or similar would be helpful.
2. when the jupyter cells complete, people think that the model completes and then they get an error in the 'ls' step. Update the text to make it more obvious that the model needs to complete before they will be able to see the files.

webpage improvements

• additional participants
• add university partners (involved in beta testing)?
• tab with information on getting started, getting help, know and new issues, and feedback
• adding NEON and CICompass logos to the banner
• include information about how NEON data were generated and gap filled

mask out bad data from NEON eval files

To mask out absurd measurements from NEON data @ddurden recommended using these min and max thresholds that are used in Ameriflux data processing.

@negin513 , it's not urgent but can you bring these thresholds into scripts that plot up NEON observations?

Flags used for Ameriflux data
Rng$Min <- data.frame(
"FC" = -100, #[umol m-2 s-1]
"SC" = -100, #[umol m-2 s-1]
"NEE" = -100, #[umol m-2 s-1
"LE" = -500, #[W m-2]
"H" = -500, #[W m-2]
"USTAR" = 0, #[m s-1]
"CO2" = 200, #[umol mol-1]
"H2O" = 0, #[mmol mol-1]
"WS_1_1_1" = 0, #[m s-1]
"WS_MAX_1_1_1" = 0, #[m s-1]
"WD_1_1_1" = -0.1, #[deg]
"T_SONIC" = -55.0, #[C]
)

Rng$Max <- data.frame(
"FC" = 100, #[umol m-2 s-1]
"SC" = 100, #[umol m-2 s-1]
"NEE" = 100, #[umol m-2 s-1]
"LE" = 1000, #[W m-2]
"H" = 1000, #[W m-2]
"USTAR" = 5, #[m s-1]
"CO2" = 800, #[umol mol-1]
"H2O" = 100, #[mmol mol-1]
"WS_1_1_1" = 50, #[m s-1]
"WS_MAX_1_1_1" = 50, #[m s-1]
"WD_1_1_1" = 360, #[deg]
"T_SONIC" = 45.0, #[C]
)

CLM-NEON Tutorial run time error

Hello Will:

I am trying to run CLM-NEON Tool using the tutorial referred in the manuscript. But it shows the following error and runs is not complete, further log file is not present in the run directory.

Any suggestion how to resolve this issue!

---

Exception from case_run: ERROR: RUN FAIL: Command ' /home/user/CLM-NEON/TALL/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed
See log file for details: /home/user/CLM-NEON/TALL.transient/run/cesm.log.230125-011043
Submit job case.st_archive
Cannot find a TALL.transient.cpl*.r.*.nc file in directory /home/user/CLM-NEON/TALL.transient/run

Below is fill error message after running the run_neon command in jupyter lab
run_neon --neon-sites $site --output-root ~/CLM-NEON --overwrite

/opt/ncar/cesm/tools/site_and_regional/run_neon.py:535: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if run_type is not "transient":
---- building a base case -------
Removing the existing case at: /home/user/CLM-NEON/TALL
---- creating a base case -------
---- base case created ------
---- base case setup ------
---- base case build ------
WARNING: No .input_data_list files found in dir 'Buildconf'
Time required to building the base case: 267.76820278167725 s.
using this version: latest
---- removing the existing case -------
---- cloning the base case in /home/user/CLM-NEON/TALL.transient
varname datafiles is already in stream mod dictionary
varname year_first is already in stream mod dictionary
varname year_last is already in stream mod dictionary
varname year_align is already in stream mod dictionary
varname dtlimit is already in stream mod dictionary
varname datafiles is already in stream mod dictionary
varname year_first is already in stream mod dictionary
varname year_last is already in stream mod dictionary
varname year_align is already in stream mod dictionary
varname dtlimit is already in stream mod dictionary
varname year_first is already in stream mod dictionary
varname year_last is already in stream mod dictionary
varname year_align is already in stream mod dictionary
varname dtlimit is already in stream mod dictionary
Model ctsm missing file finidat = '/home/user/CLM-NEON/TALL.transient/run/inputdata/lnd/ctsm/initdata/TALL.2022-11-11.clm2.r.0318-01-01-00000.nc'
Model datm missing file file1 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2018-01.nc'
Model datm missing file file2 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2018-02.nc'
Model datm missing file file3 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2018-03.nc'
Model datm missing file file4 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2018-04.nc'
Model datm missing file file5 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2018-05.nc'
Model datm missing file file6 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2018-06.nc'
Model datm missing file file7 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2018-07.nc'
Model datm missing file file8 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2018-08.nc'
Model datm missing file file9 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2018-09.nc'
Model datm missing file file10 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2018-10.nc'
Model datm missing file file11 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2018-11.nc'
Model datm missing file file12 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2018-12.nc'
Model datm missing file file13 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2019-01.nc'
Model datm missing file file14 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2019-02.nc'
Model datm missing file file15 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2019-03.nc'
Model datm missing file file16 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2019-04.nc'
Model datm missing file file17 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2019-05.nc'
Model datm missing file file18 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2019-06.nc'
Model datm missing file file19 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2019-07.nc'
Model datm missing file file20 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2019-08.nc'
Model datm missing file file21 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2019-09.nc'
Model datm missing file file22 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2019-10.nc'
Model datm missing file file23 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2019-11.nc'
Model datm missing file file24 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2019-12.nc'
Model datm missing file file25 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2020-01.nc'
Model datm missing file file26 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2020-02.nc'
Model datm missing file file27 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2020-03.nc'
Model datm missing file file28 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2020-04.nc'
Model datm missing file file29 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2020-05.nc'
Model datm missing file file30 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2020-06.nc'
Model datm missing file file31 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2020-07.nc'
Model datm missing file file32 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2020-08.nc'
Model datm missing file file33 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2020-09.nc'
Model datm missing file file34 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2020-10.nc'
Model datm missing file file35 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2020-11.nc'
Model datm missing file file36 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2020-12.nc'
Model datm missing file file37 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2021-01.nc'
Model datm missing file file38 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2021-02.nc'
Model datm missing file file39 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2021-03.nc'
Model datm missing file file40 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2021-04.nc'
Model datm missing file file41 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2021-05.nc'
Model datm missing file file42 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2021-06.nc'
Model datm missing file file43 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2021-07.nc'
Model datm missing file file44 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2021-08.nc'
Model datm missing file file45 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2021-09.nc'
Model datm missing file file46 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2021-10.nc'
Model datm missing file file47 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2021-11.nc'
Model datm missing file file48 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2021-12.nc'
Model datm missing file file49 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2022-01.nc'
Model datm missing file file50 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2022-02.nc'
Model datm missing file file51 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2022-03.nc'
Model datm missing file file52 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2022-04.nc'
varname datafiles is already in stream mod dictionary
varname year_first is already in stream mod dictionary
varname year_last is already in stream mod dictionary
varname year_align is already in stream mod dictionary
varname dtlimit is already in stream mod dictionary
varname datafiles is already in stream mod dictionary
varname year_first is already in stream mod dictionary
varname year_last is already in stream mod dictionary
varname year_align is already in stream mod dictionary
varname dtlimit is already in stream mod dictionary
varname year_first is already in stream mod dictionary
varname year_last is already in stream mod dictionary
varname year_align is already in stream mod dictionary
varname dtlimit is already in stream mod dictionary
submit_jobs case.run
Submit job case.run
varname datafiles is already in stream mod dictionary
varname year_first is already in stream mod dictionary
varname year_last is already in stream mod dictionary
varname year_align is already in stream mod dictionary
varname dtlimit is already in stream mod dictionary
varname datafiles is already in stream mod dictionary
varname year_first is already in stream mod dictionary
varname year_last is already in stream mod dictionary
varname year_align is already in stream mod dictionary
varname dtlimit is already in stream mod dictionary
varname year_first is already in stream mod dictionary
varname year_last is already in stream mod dictionary
varname year_align is already in stream mod dictionary
varname dtlimit is already in stream mod dictionary
Exception from case_run: ERROR: RUN FAIL: Command ' /home/user/CLM-NEON/TALL/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed
See log file for details: /home/user/CLM-NEON/TALL.transient/run/cesm.log.230125-011043
Submit job case.st_archive
Cannot find a TALL.transient.cpl*.r.*.nc file in directory /home/user/CLM-NEON/TALL.transient/run

Thank You
Sanjiv Kumar