m2lines / pyqg_parameterization_benchmarks Goto Github PK

Public-facing git repository for JAMES submission, "Benchmarking of machine learning ocean parameterizations in an idealized model"

License: MIT License

Jupyter Notebook 98.46% Python 1.54%

pyqg_parameterization_benchmarks's People

Stargazers

Watchers

Forkers

pbluc cambridge-iccs

pyqg_parameterization_benchmarks's Issues

Dataset files on Globus seem to be unaccessible

Great work and paper. But I got this error when I tried to fetch the datafile on globus. Could you help me fix it?

Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/fsspec/mapping.py", line 143, in __getitem__
    result = self.fs.cat(k)
  File "/opt/conda/lib/python3.8/site-packages/fsspec/asyn.py", line 115, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/fsspec/asyn.py", line 100, in sync
    raise return_result
  File "/opt/conda/lib/python3.8/site-packages/fsspec/asyn.py", line 55, in _runner
    result[0] = await coro
  File "/opt/conda/lib/python3.8/site-packages/fsspec/asyn.py", line 414, in _cat
    raise ex
  File "/opt/conda/lib/python3.8/asyncio/tasks.py", line 455, in wait_for
    return await fut
  File "/opt/conda/lib/python3.8/site-packages/fsspec/implementations/http.py", line 231, in _cat_file
    self._raise_not_found_for_status(r, url)
  File "/opt/conda/lib/python3.8/site-packages/fsspec/implementations/http.py", line 213, in _raise_not_found_for_status
    raise FileNotFoundError(url)
FileNotFoundError: https://g-402b74.00888.8540.data.globus.org/jet/forcing2.zarr/.zmetadata

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 21, in <module>
    jet_high_res = get_dataset('jet/forcing2').isel(run=0)
  File "test.py", line 17, in get_dataset
    return xr.open_zarr(mapper, consolidated=True)
  File "/opt/conda/lib/python3.8/site-packages/xarray/backends/zarr.py", line 688, in open_zarr
    ds = open_dataset(
  File "/opt/conda/lib/python3.8/site-packages/xarray/backends/api.py", line 501, in open_dataset
    backend_ds = backend.open_dataset(
  File "/opt/conda/lib/python3.8/site-packages/xarray/backends/zarr.py", line 736, in open_dataset
    store = ZarrStore.open_group(
  File "/opt/conda/lib/python3.8/site-packages/xarray/backends/zarr.py", line 336, in open_group
    zarr_group = zarr.open_consolidated(store, **open_kwargs)
  File "/opt/conda/lib/python3.8/site-packages/zarr/convenience.py", line 1300, in open_consolidated
    meta_store = ConsolidatedStoreClass(store, metadata_key=metadata_key)
  File "/opt/conda/lib/python3.8/site-packages/zarr/storage.py", line 2861, in __init__
    meta = json_loads(self.store[metadata_key])
  File "/opt/conda/lib/python3.8/site-packages/zarr/storage.py", line 724, in __getitem__
    return self._mutable_mapping[key]
  File "/opt/conda/lib/python3.8/site-packages/fsspec/mapping.py", line 147, in __getitem__
    raise KeyError(key)
KeyError: '.zmetadata'

Access denied when attempting to retrieve datasets

Hello,

When attempting to run the code within the dataset_description.ipynb Jupyter notebook to retrieve Ross' datasets, I'm met with the following error message FileNotFoundError: https://g-402b74.00888.8540.data.globus.org/eddy/forcing1.zarr/.zmetadata . Upon further inspection, through following the hyperlink in the error message, it appears that this exception is showing due to user permissions/privileges.

However, it is only of late that I'm meeting this error and have been able to access these datasets in the recent past. Has the data been moved around or changed in any way not long ago?

Any insight into this would be greatly appreciated!

Unable to retrieve single dataset file on Globus

Early on last week, I ran into an issue with accessing the datasets hosted on Globus. It appears just one dataset no longer exists anymore which is jet/forcing3.zarr. When I attempt to retrieve it I get the following error message:

FileNotFoundError: https://g-402b74.00888.8540.data.globus.org/jet/forcing3.zarr/.zmetadata

Upon following the hyperlink in the above error message, I was met with the following:

GlobusError: v=1 c=INTERNAL_ERROR
GridFTP-Error: globus_xio_register_open
globus_xio: Unable to open file /jet/forcing3.zarr/.zmetadata
globus_xio: System error in open: No such file or directory
globus_xio: A system call failed: No such file or directory

I've been able to retrieve all other datasets just fine except this one. This isn't blocking my work or in need to be fixed urgently, or at all, but I just wanted to make it known that this was occurring.

Training a new model with already existing directory - Neural_network.py

Good afternoon,

While I was reading the code of Neural_network.py, I found myself intrigued by the following line of code inside the class method train_on of FCNNParameterization. As a matter of fact, when one gives a directory to save a new model, this condition is triggered:

if os.path.exists(model_dir):
   models2.append(FullyCNN.load(model_dir))

However, the list models2 has not been initialized before. My first thought is that it's a minor typo, indeed if models2 was replaced by models, it would imply that a former model inside the given directory would be trained in addition to a new one.

However, if it is not the case, I am not sure that I understand the meaning of this if.

I wish you a pleasant day !

All the best,

Victor

m2lines / pyqg_parameterization_benchmarks Goto Github PK

pyqg_parameterization_benchmarks's People

Stargazers

Watchers

Forkers

pyqg_parameterization_benchmarks's Issues

Dataset files on Globus seem to be unaccessible

Access denied when attempting to retrieve datasets

Unable to retrieve single dataset file on Globus

Training a new model with already existing directory - Neural_network.py

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent