sciqlop / speasy Goto Github PK

View Code? Open in Web Editor NEW

24.0 6.0 7.0 25.22 MB

Space Physics made EASY! A simple Python package to deal with main Space Physics WebServices (CDA,SSC,AMDA,..)

License: Other

Makefile 0.58% Python 94.45% Jupyter Notebook 4.97%

space-physics webservices python3 python-3 plasma-physics nasa-api esa cnes sciqlop space-plasma

speasy's Introduction

Latest release

What Is SciQLop?

SciQLop (SCIentific Qt application for Learning from Observations of Plasmas) is a powerful and user-friendly tool designed for the visualization and analysis of in-situ space plasma data.

Using SciQLop will let you:

have a super easy access to tens of thousands of products from the top main data archives in the world,
explore multivariate time series effortlessly, with lightning-fast and transparent downloads as you scroll, zoom in, and zoom out,should
visualize custom products with simple python code executed on-the-fly,
easily label time intervals and make or edit catalog of events graphically and rapidely,
analyze your data in jupyter notebooks,

Heliophysicists now benefit from decades of space exploration through many spacecraft missions. Exploring this massive amount of data to find events of interest, build catalogs, and conduct statistical multi-mission data analysis can be a daunting task if not having the right tool.

SciQLop aims at being this tool! A simple lightweight yet powerful graphical interface coupled to the limitless options brought by the Jupyter Notebook integration, that focuses on providing users with the easiest possible way to explore, label and analyze huge amounts of data. SciQLop is also the right tool for teaching space physics and in situ spacecraft data handling to students effortlessly.

Main Features

Interactive and responsive: SciQLop can handle millions of data points without compromising on interactivity. Users can scroll, zoom, move, and export plots with ease.
User-friendly: Accessing data in SciQLop is as simple as a drag and drop from the tens of thousands of products readily available. Custom user products defined in Python behave exactly the same way and bring infinite possibilities.
Jupyter notebook integration: SciQLop can be used as a backend for Jupyter notebooks, allowing users to create and manipulate plots from within their notebooks, define new products and much more.
Catalogs: SciQLop provides a catalog system that allows users to easily label events in their data or visualize existing catalogs of events.
Evolving and growing list of examples: SciQLop comes with a growing list of examples that demonstrate how to perform common tasks such as loading data, creating plots, and using the catalog system.

Upcoming features

community-driven plugins repository: SciQLop will soon have a plugin system that will allow users to extend the software's capabilities by installing community-driven plugins.
catalogs coediting: SciQLop will allow users to coedit catalogs, making it easier to collaborate on event labeling and visualization, thereby also improving reproducibility of space physics studies.

How to install SciQLop

Mac Users

Since SciQLop 0.7.1 we produce a Mac App Bundle that you can download from the latest release page just pick the right architecture for your Mac (ARM64 for Apple M1/2/3 chips and x86_64 for intel ones).

Linux Users

If you are using a Linux distribution, you may not need to install anything, you can just download the AppImage from the latest release and run it (after making it executable).

From sources

Since SciQLop depends on specific versions of PySide6 you might prefer to use a dedicated virtualenv for SciQLop to avoid any conflict with any other Python package already installed in your system.

Using releases from PyPi

python -m pip install sciqlop

Using the latest code from GitHub

python -m pip install git+https://github.com/SciQLop/SciQLop

Once installed the sciqlop launcher should be in your PATH and you should be able to start SciQLop from your terminal.

sciqlop

python -m SciQLop.app

Python user API Examples:

SciQLop has a public API that allows users to create custom products and plots. Here are some examples:

Creating plot panels:

from SciQLop.user_api import TimeRange
from SciQLop.user_api.plot import create_plot_panel
from datetime import datetime

# all plots are stacked
p = create_plot_panel()
p.time_range = TimeRange(datetime(2015, 10, 22, 6, 4, 30), datetime(2015, 10, 22, 6, 6, 0))
p.plot("speasy/cda/MMS/MMS1/FGM/MMS1_FGM_BRST_L2/mms1_fgm_b_gsm_brst_l2")
p.plot("speasy/cda/MMS/MMS1/DIS/MMS1_FPI_BRST_L2_DIS_MOMS/mms1_dis_bulkv_gse_brst")
p.plot("speasy/cda/MMS/MMS1/DIS/MMS1_FPI_BRST_L2_DIS_MOMS/mms1_dis_energyspectr_omni_brst")

# tha_peif_sc_pot and tha_peif_en_eflux will share the same plot 
p2 = create_plot_panel()
p2.plot("speasy/cda/THEMIS/THA/L2/THA_L2_ESA/tha_peif_en_eflux")
p2.plots[0].plot("speasy/cda/THEMIS/THA/L2/THA_L2_ESA/tha_peif_sc_pot")
p2.plot("speasy/cda/THEMIS/THA/L2/THA_L2_ESA/tha_peif_velocity_dsl")

NOTE: An easy way to get product paths, is to drag a product from Products Tree to any text zone or even your Python terminal.

Custom products AKA virtual products:

import numpy as np
from SciQLop.user_api.virtual_products import create_virtual_product, VirtualProductType
from SciQLop.user_api.plot import create_plot_panel
import speasy as spz


# define a custom product that calculates the magnitude of the magnetic field measured by ACE

def ace_b_magnitude(start: float, stop: float) -> spz.SpeasyVariable:
  b_gse = spz.get_data(spz.inventories.tree.amda.Parameters.ACE.MFI.ace_imf_all.imf, start, stop)
  return np.sqrt(b_gse["bx"] ** 2 + b_gse["by"] ** 2 + b_gse["bz"] ** 2)


ace_b_magnitude_virtual_prod = create_virtual_product(path='my_virtual_products/ace_b_magnitude',
                                                      product_type=VirtualProductType.Scalar, callback=ace_b_magnitude,
                                                      labels=["|b|"])

# plot the virtual product
p = create_plot_panel()
p.plot(ace_b_magnitude_virtual_prod)

More examples can be found in the examples folder, they are also available from the welcome screen.

How to contribute

Just fork the repository, make your changes and submit a pull request. We will be happy to review and merge your changes. Reports of bugs and feature requests are also welcome. Do not forget to star the project if you like it!

Credits

The development of SciQLop is supported by the CDPP.
We acknowledge support from the federation Plas@Par

Thanks

We would like to thank the developers of the following libraries that SciQLop depends on:

PySide6 for the GUI framework and Qt bindings.
QCustomPlot for providing the plotting library.
DiskCache for providing a simple and efficient caching system used in Speasy.
The Jupyter project for providing the Jupyter notebook integration.
Numpy for providing fast Python array library.

speasy's People

Contributors

Stargazers

Watchers

Forkers

jeandet brenard-irap dolgalad cmoissar lgtm-migrator nicolasaunai jgieseler

speasy's Issues

Can't get data from proxy server http://sciqlop.lpp.polytechnique.fr/cache

Space Physics WebServices Client version:
Python version: Python 3.10.6
Operating System: Ubuntu 22.04.1 LTS

Description

import speasy

What I Did

Input:
import speasy

Output:
Can't get data from proxy server http://sciqlop.lpp.polytechnique.fr/cache

This is not a problem I had two weeks ago.

Best wishes,
Clement

Add netCDF4 support

netCDF4 seems to be the best common format across web services for now.
Before we enable it, we have to ensure that:

it can be installed with just pip install netCDF4 on OSX, Windows 10/11, Linux (at least Ubuntu 18+, Fedora 33+).
we get the same metadata and data than csv
it doesn't suffer the same multi process or thread issues than CDF lib.

Then do we make it a strong dep of speasy or an opt in feature.

AttributeError: module 'pycdfpp' has no attribute 'CDF_CHAR

Space Physics WebServices Client version: 1.1.2
Python version: 3.10.12
Operating System: macOS

Description

Trying to download STEREO B Field data from cda raised errors.
AttributeError: module 'pycdfpp' has no attribute 'CDF_CHAR'

What I Did

sat_fgm_product = 'cda/STA_L1_MAG_RTN/BFIELD'
products = [sat_fgm_product]
dataset = spz.get_data(products, test_trange, disable_proxy=True)

{
	"name": "AttributeError",
	"message": "module 'pycdfpp' has no attribute 'CDF_CHAR'",
	"stack": "---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/Users/zijin/projects/planet/nbs/03_stereo.ipynb Cell 15 line <cell line: 4>()
      <a href='vscode-notebook-cell:/Users/zijin/projects/planet/nbs/03_stereo.ipynb#X15sZmlsZQ%3D%3D?line=1'>2</a> sat_fgm_product = 'cda/STA_L1_MAG_RTN/BFIELD'
      <a href='vscode-notebook-cell:/Users/zijin/projects/planet/nbs/03_stereo.ipynb#X15sZmlsZQ%3D%3D?line=2'>3</a> products = [sat_fgm_product]
----> <a href='vscode-notebook-cell:/Users/zijin/projects/planet/nbs/03_stereo.ipynb#X15sZmlsZQ%3D%3D?line=3'>4</a> dataset = spz.get_data(products, test_trange, disable_proxy=True)

File ~/mambaforge/envs/cool_planet/lib/python3.10/site-packages/speasy/core/requests_scheduling/request_dispatch.py:309, in get_data(*args, **kwargs)
    307 product = args[0]
    308 if is_collection(product) and not isinstance(product, SpeasyIndex):
--> 309     return list(map(lambda p: get_data(p, *args[1:], **kwargs), progress_bar(leave=True, **kwargs)(product)))
    311 if len(args) == 1:
    312     return _get_catalog_or_timetable(*args, **kwargs)

File ~/mambaforge/envs/cool_planet/lib/python3.10/site-packages/speasy/core/requests_scheduling/request_dispatch.py:309, in get_data.<locals>.<lambda>(p)
    307 product = args[0]
    308 if is_collection(product) and not isinstance(product, SpeasyIndex):
--> 309     return list(map(lambda p: get_data(p, *args[1:], **kwargs), progress_bar(leave=True, **kwargs)(product)))
    311 if len(args) == 1:
    312     return _get_catalog_or_timetable(*args, **kwargs)

File ~/mambaforge/envs/cool_planet/lib/python3.10/site-packages/speasy/core/requests_scheduling/request_dispatch.py:316, in get_data(*args, **kwargs)
    314 t_range = args[1]
    315 if _is_dtrange(t_range):
--> 316     return _get_timeserie1(*args, **kwargs)
    317 if is_collection(t_range):
    318     return list(
    319         map(lambda r: get_data(product, r, *args[2:], **kwargs),
    320             progress_bar(leave=False, **kwargs)(t_range)))

File ~/mambaforge/envs/cool_planet/lib/python3.10/site-packages/speasy/core/requests_scheduling/request_dispatch.py:159, in _get_timeserie1(index, dtrange, **kwargs)
    158 def _get_timeserie1(index, dtrange, **kwargs):
--> 159     return _scalar_get_data(index, dtrange[0], dtrange[1], **kwargs)

File ~/mambaforge/envs/cool_planet/lib/python3.10/site-packages/speasy/core/requests_scheduling/request_dispatch.py:150, in _scalar_get_data(index, *args, **kwargs)
    148 provider_uid, product_uid = provider_and_product(index)
    149 if provider_uid in PROVIDERS:
--> 150     return PROVIDERS[provider_uid].get_data(product_uid, *args, **kwargs)
    151 raise ValueError(f\"Can't find a provider for {index}\")

File ~/mambaforge/envs/cool_planet/lib/python3.10/site-packages/speasy/core/__init__.py:221, in AllowedKwargs.__call__.<locals>.wrapped(*args, **kwargs)
    218 unexpected_args = list(
    219     filter(lambda arg_name: arg_name not in self.allowed_list, kwargs.keys()))
    220 if not unexpected_args:
--> 221     return func(*args, **kwargs)
    222 raise TypeError(
    223     f\"Unexpected keyword argument {unexpected_args}, allowed keyword arguments are {self.allowed_list}\")

File ~/mambaforge/envs/cool_planet/lib/python3.10/site-packages/speasy/core/dataprovider.py:30, in ParameterRangeCheck.__call__.<locals>.wrapped(wrapped_self, product, start_time, stop_time, **kwargs)
     28     log.warning(f\"You are requesting {product} outside of its definition range {p_range}\")
     29     return None
---> 30 return get_data(wrapped_self, product=product, start_time=start_time, stop_time=stop_time, **kwargs)

File ~/mambaforge/envs/cool_planet/lib/python3.10/site-packages/speasy/core/cache/_providers_caches.py:247, in UnversionedProviderCache.__call__.<locals>.wrapped(wrapped_self, product, start_time, stop_time, **kwargs)
    243 fragment_duration = timedelta(hours=fragment_hours)
    244 data_chunks, maybe_outdated_fragments, missing_fragments = self.split_fragments(fragments, product,
    245                                                                                 fragment_duration, **kwargs)
    246 data_chunks += \\
--> 247     list(filter(lambda d: d is not None, [
    248         self._cache.add_to_cache(
    249             get_data(
    250                 wrapped_self, product=product, start_time=fragment_group[0],
    251                 stop_time=fragment_group[-1] + fragment_duration, **kwargs),
    252             fragments=fragment_group, product=product, fragment_duration_hours=fragment_hours,
    253             version=datetime.utcnow(), **kwargs)
    254         for fragment_group
    255         in progress_bar(leave=False, desc=\"Downloading missing fragments from cache\", **kwargs)(
    256             missing_fragments)]))
    258 for group in progress_bar(leave=False, desc=\"Checking if cache fragments are outdated\", **kwargs)(
    259     maybe_outdated_fragments):
    260     oldest = max(group, key=lambda item: item[1].version)[1].version

File ~/mambaforge/envs/cool_planet/lib/python3.10/site-packages/speasy/core/cache/_providers_caches.py:249, in <listcomp>(.0)
    243 fragment_duration = timedelta(hours=fragment_hours)
    244 data_chunks, maybe_outdated_fragments, missing_fragments = self.split_fragments(fragments, product,
    245                                                                                 fragment_duration, **kwargs)
    246 data_chunks += \\
    247     list(filter(lambda d: d is not None, [
    248         self._cache.add_to_cache(
--> 249             get_data(
    250                 wrapped_self, product=product, start_time=fragment_group[0],
    251                 stop_time=fragment_group[-1] + fragment_duration, **kwargs),
    252             fragments=fragment_group, product=product, fragment_duration_hours=fragment_hours,
    253             version=datetime.utcnow(), **kwargs)
    254         for fragment_group
    255         in progress_bar(leave=False, desc=\"Downloading missing fragments from cache\", **kwargs)(
    256             missing_fragments)]))
    258 for group in progress_bar(leave=False, desc=\"Checking if cache fragments are outdated\", **kwargs)(
    259     maybe_outdated_fragments):
    260     oldest = max(group, key=lambda item: item[1].version)[1].version

File ~/mambaforge/envs/cool_planet/lib/python3.10/site-packages/speasy/core/requests_scheduling/split_large_requests.py:21, in SplitLargeRequests.__call__.<locals>.wrapped(wrapped_self, product, start_time, stop_time, **kwargs)
     19 max_range_per_request = self.threshold(product)
     20 if duration <= max_range_per_request:
---> 21     return get_data(wrapped_self, product=product, start_time=start_time, stop_time=stop_time, **kwargs)
     22 else:
     23     fragments = range.split(max_range_per_request)

File ~/mambaforge/envs/cool_planet/lib/python3.10/site-packages/speasy/core/proxy/__init__.py:130, in Proxyfiable.__call__.<locals>.wrapped(*args, **kwargs)
    128     except:  # lgtm [py/catch-base-exception]
    129         log.error(f\"Can't get data from proxy server {proxy_cfg.url()}\")
--> 130 return func(*args, **kwargs)

File ~/mambaforge/envs/cool_planet/lib/python3.10/site-packages/speasy/webservices/cda/__init__.py:178, in CDA_Webservice.get_data(self, product, start_time, stop_time, if_newer_than, extra_http_headers)
    169 @AllowedKwargs(
    170     PROXY_ALLOWED_KWARGS + CACHE_ALLOWED_KWARGS + GET_DATA_ALLOWED_KWARGS + ['if_newer_than'])
    171 @ParameterRangeCheck()
   (...)
    175 def get_data(self, product, start_time: datetime, stop_time: datetime, if_newer_than: datetime or None = None,
    176              extra_http_headers: Dict or None = None):
    177     dataset, variable = self._to_dataset_and_variable(product)
--> 178     return self._dl_variable(start_time=start_time, stop_time=stop_time, dataset=dataset,
    179                              variable=variable, if_newer_than=if_newer_than, extra_http_headers=extra_http_headers)

File ~/mambaforge/envs/cool_planet/lib/python3.10/site-packages/speasy/webservices/cda/__init__.py:160, in CDA_Webservice._dl_variable(self, dataset, variable, start_time, stop_time, if_newer_than, extra_http_headers)
    158 log.debug(resp.url)
    159 if resp.status_code == 200 and 'FileDescription' in resp.json():
--> 160     return _read_cdf(resp.json()['FileDescription'][0]['Name'], variable)
    161 elif not resp.ok:
    162     if resp.status_code == 404 and \"No data available\" in resp.json().get('Message', [\"\"])[0]:

File ~/mambaforge/envs/cool_planet/lib/python3.10/site-packages/speasy/webservices/cda/__init__.py:59, in _read_cdf(url, variable)
     57 def _read_cdf(url: str, variable: str) -> SpeasyVariable:
     58     with urlopen_with_retry(url) as remote_cdf:
---> 59         return load_variable(buffer=remote_cdf.read(), variable=variable)

File ~/mambaforge/envs/cool_planet/lib/python3.10/site-packages/speasy/core/cdf/__init__.py:41, in load_variable(variable, file, buffer)
     40 def load_variable(variable=\"\", file=None, buffer=None) -> SpeasyVariable or None:
---> 41     istp = pyistp.load(file=file, buffer=buffer)
     42     if istp:
     43         if variable in istp.data_variables():

File ~/.local/lib/python3.10/site-packages/pyistp/__init__.py:11, in load(file, buffer)
     10 def load(file=None, buffer=None) -> _ISTPLoader:
---> 11     return _ISTPLoader(file=file, buffer=buffer)

File ~/.local/lib/python3.10/site-packages/pyistp/loader.py:9, in ISTPLoader.__init__(self, file, buffer)
      7 def __init__(self, file=None, buffer=None):
      8     from ._impl import ISTPLoaderImpl
----> 9     self._impl = ISTPLoaderImpl(file=file, buffer=buffer)

File ~/.local/lib/python3.10/site-packages/pyistp/_impl.py:92, in ISTPLoaderImpl.__init__(self, file, buffer)
     90 self.cdf = current_driver(file or buffer)
     91 self.data_variables = []
---> 92 self._update_data_vars_lis()

File ~/.local/lib/python3.10/site-packages/pyistp/_impl.py:108, in ISTPLoaderImpl._update_data_vars_lis(self)
    105 var_type = self.cdf.variable_attribute_value(var, 'VAR_TYPE')
    106 param_type = (self.cdf.variable_attribute_value(var,
    107                                                 'PARAMETER_TYPE') or \"\").lower()  # another cluster CSA crap
--> 108 if (var_type == 'data' or param_type == 'data') and not self.cdf.is_char(var):
    109     self.data_variables.append(var)

File ~/.local/lib/python3.10/site-packages/pyistp/drivers/pycdfpp.py:27, in Driver.is_char(self, var)
     26 def is_char(self, var):
---> 27     return self.cdf[var].type == pycdfpp.CDF_CHAR

AttributeError: module 'pycdfpp' has no attribute 'CDF_CHAR'"
}```

Issue with SpeasyVariable.plot()

Space Physics WebServices Client version: Not sure what you mean
Python version: 3.10.6
Operating System: Ubuntu

Description

Reproduce AMDA First Steps.

What I Did

Copy and paste the first steps from the link above.

Output:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/tmp/ipykernel_17102/2607499050.py in <module>
      1 plt.figure()
      2 ace_mag = spz.get_data(amda_tree.Parameters.ACE.MFI.ace_imf_all.imf, datetime(2016,6,2), datetime(2016,6,5))
----> 3 ace_mag.plot()
      4 plt.tight_layout()
      5 plt.show()

~/.local/lib/python3.10/site-packages/speasy/plotting/__init__.py in __call__(self, backend, *args, **kwargs)
     69         if self._infer_plot_type() == PlotType.SPECTRO:
     70             return self.colormap(backend=backend, *args, **kwargs)
---> 71         return self.line(backend=backend, *args, **kwargs)
     72 
     73     def __getitem__(self, item):

~/.local/lib/python3.10/site-packages/speasy/plotting/__init__.py in line(self, backend, *args, **kwargs)
     46         units = self.values.unit
     47         yaxis_label = self.values.name
---> 48         return self._get_backend(backend).line(x=self.axes[0].values, y=self.values, labels=self.columns_names,
     49                                                units=units,
     50                                                xaxis_label=self.axes[0].name,

~/.local/lib/python3.10/site-packages/speasy/plotting/mpl_backend/__init__.py in line(self, x, y, ax, labels, units, xaxis_label, yaxis_label, *args, **kwargs)
     20         ax = self._get_ax(ax)
     21         ax.tick_params(axis='x', labelrotation = 45)
---> 22         ax.plot(x, y)
     23         if labels is not None:
     24             ax.legend(labels)

/usr/lib/python3/dist-packages/matplotlib/axes/_axes.py in plot(self, scalex, scaley, data, *args, **kwargs)
   1630         """
   1631         kwargs = cbook.normalize_kwargs(kwargs, mlines.Line2D)
-> 1632         lines = [*self._get_lines(*args, data=data, **kwargs)]
   1633         for line in lines:
   1634             self.add_line(line)

/usr/lib/python3/dist-packages/matplotlib/axes/_base.py in __call__(self, data, *args, **kwargs)
    310                 this += args[0],
    311                 args = args[1:]
--> 312             yield from self._plot_args(this, kwargs)
    313 
    314     def get_next_color(self):

/usr/lib/python3/dist-packages/matplotlib/axes/_base.py in _plot_args(self, tup, kwargs, return_kwargs)
    486         if len(xy) == 2:
    487             x = _check_1d(xy[0])
--> 488             y = _check_1d(xy[1])
    489         else:
    490             x, y = index_of(xy[-1])

/usr/lib/python3/dist-packages/matplotlib/cbook/__init__.py in _check_1d(x)
   1325                     message='Support for multi-dimensional indexing')
   1326 
-> 1327                 ndim = x[:, None].ndim
   1328                 # we have definitely hit a pandas index or series object
   1329                 # cast to a numpy array.

AttributeError: 'DataContainer' object has no attribute 'ndim'

Summary

Everything works great and I really wish I had access to such a great tool 5 years ago at the start of my PhD.
But the .plot() function a speasy.products.variable.SpeasyVariable doesn't seem to work on my laptop. Is it only me?

Best wishes,
Clement

Be more verbose about the current config and what it implies for the user

As pointed out by @brenard-irap, the proxy usage implications on Speasy are not that obvious. One option could be to print a message with a summary of the current config and what it means to Speasy user. The user should be able to disable this message.

It will be helpful to have the possibility to get a catalog as a pandas dataframe, for example: amda.get_catalog(...).to_dataframe()

Space Physics WebServices Client version:
Python version:
Operating System:

Description

Describe what you were trying to get done.
Tell us what happened, what went wrong, and what you expected to happen.

What I Did

Paste the command(s) you ran and the output.
If there was a crash, please include the traceback here.

Setting config username/password subsequently causes crash

Space Physics WebServices Client version: 1.0.3
Python version: 3.10.6
Operating System: ubuntu-22.04

Description

Code crashes after setting username/password config

What I Did

I want to log in amda, so i

>>> from speasy import config 
>>> config.amda.username.set('myusername')
>>> config.amda.password.set('mypassword')

Then I ran code and it crashed after

import speasy as spz



Traceback (most recent call last):
  File "~/MyProject/trySpeasy.py", line 4, in <module>
    import speasy as spz
  File "~/MyProject/venv/lib/python3.10/site-packages/speasy/__init__.py", line 18, in <module>
    from .core.requests_scheduling.request_dispatch import get_data, list_providers, amda, cda, csa, ssc
  File "~/MyProject/venv/lib/python3.10/site-packages/speasy/core/requests_scheduling/__init__.py", line 2, in <module>
    from .request_dispatch import get_data
  File "~/MyProject/venv/lib/python3.10/site-packages/speasy/core/requests_scheduling/request_dispatch.py", line 21, in <module>
    amda = AMDA_Webservice()
  File "~/MyProject/venv/lib/python3.10/site-packages/speasy/webservices/amda/ws.py", line 84, in __init__
    DataProvider.__init__(self, provider_name='amda')
  File "~/MyProject/venv/lib/python3.10/site-packages/speasy/core/dataprovider.py", line 47, in __init__
    self.update_inventory()
  File "~/MyProject/venv/lib/python3.10/site-packages/speasy/core/dataprovider.py", line 67, in update_inventory
    self._update_private_inventory(tree.__dict__[self.provider_name])
  File "~/MyProject/venv/lib/python3.10/site-packages/speasy/core/dataprovider.py", line 57, in _update_private_inventory
    return self.build_private_inventory(root)
  File "~/MyProject/venv/lib/python3.10/site-packages/speasy/webservices/amda/ws.py", line 93, in build_private_inventory
    return self._impl.build_private_inventory(root)
  File "~/MyProject/venv/lib/python3.10/site-packages/speasy/webservices/amda/_impl.py", line 91, in build_private_inventory
    self._update_private_lists(TimeTables=root.TimeTables, Catalogs=root.Catalogs, root=root)
  File "~/MyProject/venv/lib/python3.10/site-packages/speasy/webservices/amda/_impl.py", line 68, in _update_private_lists
    meta=user_param.ws.paramList.__dict__)
AttributeError: 'SpeasyIndex' object has no attribute 'ws'

Improve Caches notebook

This notebook basically measure and plots raw results without any interpretation and explanation.
A multi-process parallel request scaling measurement might also be interesting.

Add migration tool to convert cache and SpwcVariables to SPEASY

amda.list_user_parameters() is not up to date

Space Physics WebServices Client version: 0.10.1
Python version: 3.10
Operating System: Windows 10

Description

I'm using amda.list_user_parameters() to get the list of my parameters, it works very well. The problem is that when I create a new parameter directly with "http://amda.cdpp.eu/" and then if I run the same method the returned list of parameters is not up to date.

What I Did

config.amda_username.set('...')
config.amda_password.set('....')
print(amda.list_user_parameters())

I can't find how to set amda username and password with Speasy 1.0.0

Space Physics WebServices Client version:
Python version:
Operating System:

Description

I can't find how to set amda username and password

What I Did

spz.config.amda_username.set('testKernel')
spz.config.amda_password.set('amda')

CDAWeb documentation

CDAWeb module should be documented like AMDA module.

PSP IS⊙IS instruments cause "invalid character"

speasy version: 0.1.0 (?)
Python version: 3.9.7
Operating System: Ubuntu 20.04.3 LTS

Description

When trying to address data of Parker Solar Probe's (PSP) particle instruments "IS⊙ISEPI_Hi" or "IS⊙ISEPI_Lo" via dynamic inventory, the special sign "⊙" causes a SyntaxError: invalid character.

What I Did

First create a clean Python 3.9 environment with conda, install latest speasy version from git via pip, and start python:

$ conda create --name speasy python=3.9
$ conda activate speasy
$ pip install git+https://github.com/SciQLop/speasy
Collecting git+https://github.com/SciQLop/speasy
  Cloning https://github.com/SciQLop/speasy to /tmp/pip-req-build-lvjo1tfp
  Running command git clone -q https://github.com/SciQLop/speasy /tmp/pip-req-build-lvjo1tfp
  Resolved https://github.com/SciQLop/speasy to commit 27b9e6c32e890c3bba5d9681459b6a3ffc7e5664
[..]
Successfully installed Click-8.0.3 PyYAML-6.0 appdirs-1.4.4 astropy-5.0.1 attrs-21.4.0 cached-property-1.5.2 charset-normalizer-2.0.11 diskcache-5.4.0 idna-3.3 isodate-0.6.1 lxml-4.7.1 numpy-1.22.1 packaging-21.3 pandas-1.4.0 platformdirs-2.4.1 pyerfa-2.0.0.1 pyparsing-3.0.7 python-dateutil-2.8.2 pytz-2021.3 requests-2.27.1 requests-file-1.5.1 requests-toolbelt-0.9.1 six-1.16.0 speasy-0.9.0 spwc-0.9.0 urllib3-1.26.8 xmltodict-0.12.0 zeep-4.1.0
$ python

Then in python:

Python 3.9.7 (default, Sep 16 2021, 13:09:58) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import speasy as spz
>>> amda_tree = spz.inventory.data_tree.amda

>>> amda_tree.Parameters.PSP.FIELDSMAG
AMDAPathIndex(name='FIELDS MAG', desc='Fluxgate magnetometer (MAG) of the FIELDS suite<br/>Magnetometer', xmlid='PSP_mag', att='auto/PSP_mag', is_public=True, psp_mag_1min=<DatasetIndex: 1min>, psp_mag_4cyc=<DatasetIndex: 4/cyc>, Venusflybys=AMDAPathIndex(xmlid='psp-venus-flybys', name='Venus flybys', is_public=True, psp_mag_vflyby1=<DatasetIndex: Venus flyby #1>, psp_mag_vflyby2=<DatasetIndex: Venus flyby #2>, psp_mag_vflyby3=<DatasetIndex: Venus flyby #3>, psp_mag_vflyby4=<DatasetIndex: Venus flyby #4>), psp_fld_qfl=<DatasetIndex: quality flag>)

>>> amda_tree.Parameters.PSP.IS⊙ISEPI_Hi
  File "<stdin>", line 1
    amda_tree.Parameters.PSP.IS⊙ISEPI_Hi
                               ^
SyntaxError: invalid character '⊙' (U+2299)

>>> amda_tree.Parameters.PSP.IS⊙ISEPI_Lo
  File "<stdin>", line 1
    amda_tree.Parameters.PSP.IS⊙ISEPI_Lo
                               ^
SyntaxError: invalid character '⊙' (U+2299)

>>> amda_tree.Parameters.PSP.IS0ISEPI_Lo
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'AMDAPathIndex' object has no attribute 'IS0ISEPI_Lo'

>>> amda_tree.Parameters.PSP.ISOISEPI_Lo
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'AMDAPathIndex' object has no attribute 'ISOISEPI_Lo'

>>> amda_tree.Parameters.PSP.ISoISEPI_Lo
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'AMDAPathIndex' object has no attribute 'ISoISEPI_Lo'

Running amda_tree.Parameters.PSP.FIELDSMAG just to verify that it works in general. Trying to replace "⊙" with "0/O/o" doesn't solve the problem.

Enable SciQLop cache by default

Speasy comes with default settings where it doesn't use SciQLop proxy.
Since SciQLop proxy seems to work quite well for a long time now, we might consider enabling it by default.

html repr for notebeooks

Few resources:
https://matthewrocklin.com/blog/2019/07/04/html-repr
https://github.com/pydata/xarray/pull/3425/files

Add more examples notebooks

In order to illustrate speasy benefits we should add more notebooks with real use cases.

Notify user about unused extra kwargs in get_data

get_data method takes kwargs that could be specific to each webservice or speasy internals (cache/proxy) but doesn't necessary tell the user about wrong kwargs. At least with sscweb it's possible to pass wrong coordinate system or misspell coordinate_system and speasy will silently ignore it :/.

Document more speasy config

Speasy doc misses common setting entries such as cache and tips like where config file is written.

Allow to filter providers

Adding a setting to filter which provider is loaded or discarded could speedup Speasy loading times for users who only care about a single provider.

astropy units and tables

Space Physics WebServices Client version: 0.10.1
Python version: 3.8
Operating System: linux

Description

Convert SpeasyVariable object to astropy.table.Table with correct units (when conversion to astropy.unit.Unit is possible).
Only works if :

param.meta contains PARAMETER_COMPONENTS
param.meta["PARAMETER_UNITS"] is astropy compatible

Note : dataframe columns are renamed to param.meta["PARAMETER_COMPONENTS"] when possible

What I Did

def to_astropy_table(parameter):
    # some units may not be convertable
    try:
        units = astropy.units.Unit(parameter.meta["PARAMETER_UNITS"])
    except:
        units = None
        
    df = parameter.to_dataframe(datetime_index=True)
    
    unit_map = {c:units for c in df.columns}
    return astropy.table.Table.from_pandas(df, units=unit_map, index=True)

Issue with import speasy

Speasy version: 1.0.3
Python version: 3.10.6
Operating System: Ubuntu 22.04.2 LTS

Description

I do not manage to import speasy

What I Did

INPUT:

import speasy as spz

OUTPUT:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-4-96cca6dfb4c8> in <cell line: 1>()
----> 1 import speasy as spz

~/.local/lib/python3.10/site-packages/speasy/__init__.py in <module>
     16 from .products import SpeasyVariable, Catalog, Event, Dataset, TimeTable, MaybeAnyProduct
     17 from typing import List
---> 18 from .core.requests_scheduling.request_dispatch import get_data, list_providers, amda, cda, csa, ssc
     19 
     20 

~/.local/lib/python3.10/site-packages/speasy/core/requests_scheduling/__init__.py in <module>
      1 from .split_large_requests import SplitLargeRequests
----> 2 from .request_dispatch import get_data

~/.local/lib/python3.10/site-packages/speasy/core/requests_scheduling/request_dispatch.py in <module>
     19 TimeRangeCollectionT = Union[TimetableIndex, CatalogIndex, Iterable[Iterable[Union[TimeT]]]]
     20 
---> 21 amda = AMDA_Webservice()
     22 cda = CDA_Webservice()
     23 ssc = SSC_Webservice()

~/.local/lib/python3.10/site-packages/speasy/webservices/amda/ws.py in __init__(self, server_url)
     82         from ._impl import AmdaImpl
     83         self._impl = AmdaImpl(server_url=server_url)
---> 84         DataProvider.__init__(self, provider_name='amda')
     85 
     86     def __del__(self):

~/.local/lib/python3.10/site-packages/speasy/core/dataprovider.py in __init__(self, provider_name, provider_alt_names)
     45         for alt_name in self.provider_alt_names:
     46             flat_inventories.__dict__[alt_name] = self.flat_inventory
---> 47         self.update_inventory()
     48         PROVIDERS[provider_name] = self
     49 

~/.local/lib/python3.10/site-packages/speasy/core/dataprovider.py in update_inventory(self)
     61         with lock:
     62             new_inventory = self._inventory(provider_name=self.provider_name)
---> 63             if inventory_has_changed(tree.__dict__.get(self.provider_name, SpeasyIndex("", "", "")), new_inventory):
     64                 if self.provider_name in tree.__dict__:
     65                     tree.__dict__[self.provider_name].clear()

~/.local/lib/python3.10/site-packages/speasy/core/inventory/indexes.py in inventory_has_changed(orig, new)
    135 
    136 def inventory_has_changed(orig, new):
--> 137     if orig.__dict__.keys() != new.__dict__.keys():
    138         return True
    139     for orig_key, orig_value in orig.__dict__.items():

AttributeError: 'NoneType' object has no attribute '__dict__'

Thank you for your help,
Clement

`coordinate_system`-args of `get_orbit()` requires all lower-case string

The transition of spwc 0.6 -> 0.7 starts using the coordinate_system-args. If the value is given in all upper-case, it won't find anything.

Also, maybe, bailing out if a not-supported coordinate-system is given would be nice.

Prevent user from setting sampling parameter with AMDA

If a sneaky user tries to add a sampling freq parameter with an AMDA parameter request, this could lead to a broken cache since speasy doesn't handle this.
The simplest thing for now is to prevent the user from doing this.

Variable: add overload for view() to slice with a t_start-t_stop-interval

Exposing some core Speasy methods to users

I'm thinking about exposing some Speasy core methods as user API since they might be handy for users.
We could expose and document as public API the following methods:

any_loc_open
list_files
make_utc_datetime
epoch_to_datetime64
load_variable
Maybe some cache decorators to allow users to cache custom functions

The main drawback to this is that we lose the ability to change the behavior or remove those functions.

Add support for 3DVIEW Webservice

here is an example of client with a list of coordinate systems.

Basic and minimalist schema to reproduce plots across tools

This is an idea that I got few days ago while working on plots with speasy (having in mind that SciQLop session or quick setup thing).
It doesn't seems that impossible to describe a plot panel in an interoperable format allowing users to reproduce plots across several tools (Speasy, AMDA, SciQLop,...).

Something like this:

panel0:
  graphs:
  - index: 0
    logy: true
    product: some_spase_id
  - index: 1
    logy: false
    product: some_spase_id
  time_range:
  - '2018-01-01T12:30:55'
  - '2018-01-01T18:00:27'

The idea is to share the minimum info we need to reproduce the plot.
@brenard-irap do you think we could create and export plots from AMDA using something like this?

Orbit module/sscweb: handle minimum request duration

sscweb REST API reject small requests, spwc module should ask more and return only requested range.

Missing description for AMDA catalogs

from XML file (VOTable):
from python with speasy:

In [1]: import speasy as spz                                                                                                                                                                                                                                  

In [2]: c=spz.get_data(spz.inventory.data_tree.amda.Catalogs.SharedCatalogs.MARS.choc_MPB_catalogue_MEX)                                                                                                                                                      

In [3]: c.meta                                                                                                                                                                                                                                                
Out[3]: {}

ImportError: cannot import name 'LegacyVersion' from 'packaging.version'

As you can see at https://packaging.pypa.io/en/stable/changelog.html, "LegacyVersion" has been removed from the last version (22.0) of packaging.

Unfortunately, Speasy uses it in speasy/core/cache/version.py

So, we have an exception during the import:

ImportError: cannot import name 'LegacyVersion' from 'packaging.version' (C:\Users\benjamin.renard\Documents\AMDA\cu2022_3_10_venv\lib\site-packages\packaging\version.py)

I have downgrade packaging to version 21.3 to fix the issue

Add mechanism to avoid sending multiple identical requests at the same time

If you ask twice for the same data in parallel while not in cache, spwc will issue two identical requests to the server (AMDA, CDA, proxy...).
This shouldn't happen too often and isn't so trivial to solve.

Wrong file format from proxy with AMDA/CDF_ISTP

Here we wrongly assume ASCII while we should check speasy config.

AttributeError: module 'speasy' has no attribute 'inventory'

Space Physics WebServices Client version: -
Python version: 3.9.7 or 3.8.12
Operating System: Ubuntu 20.04.3 LTS

Description

The example using the dynamic inventory does not work because module 'speasy' has no attribute 'inventory'.

What I Did

First create a clean Python 3.9 environment with conda, install speasy via pip, and start python:

$ conda create --name speasy_test python=3.9
$ conda activate speasy_test
$ pip install --no-cache-dir speasy
Collecting speasy
  Downloading speasy-0.9.1-py2.py3-none-any.whl (21 kB)
[..]
Successfully installed Click-8.0.3 PyYAML-6.0 appdirs-1.4.4 astropy-5.0 attrs-21.4.0 cached-property-1.5.2 charset-normalizer-2.0.10 diskcache-5.4.0 idna-3.3 isodate-0.6.1 lxml-4.7.1 numpy-1.22.1 packaging-21.3 pandas-1.4.0 platformdirs-2.4.1 pyerfa-2.0.0.1 pyparsing-3.0.7 python-dateutil-2.8.2 pytz-2021.3 requests-2.27.1 requests-file-1.5.1 requests-toolbelt-0.9.1 six-1.16.0 speasy-0.9.1 spwc-0.9.0 urllib3-1.26.8 xmltodict-0.12.0 zeep-4.1.0
$ python

Then in python:

Python 3.9.7 (default, Sep 16 2021, 13:09:58) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import speasy
>>> speasy.__version__
'0.9.1'
>>> import speasy as spz
>>> ace_mag = spz.get_data('amda/imf', "2016-6-2", "2016-6-5")
>>> ace_mag
<speasy.common.variable.SpeasyVariable object at 0x7fb0de43e310>
>>> amda_tree = spz.inventory.data_tree.amda
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'speasy' has no attribute 'inventory'

I get the same result when using Python 3.8.12 (starting with conda create --name speasy_test python=3.8).

Using Python 3.6 or 3.7 completely fails to import speasy at all (ValueError: unsupported pickle protocol: 5), while importing with Python 3.5 causes an SyntaxError: invalid syntax. So the Python requirements according on pypi might need an update, too.

Cache "loses" astropy.units of stored orbit-trajectories

Space Physics WebServices Client version: "main-branch" of 01/04/2021
Python version: 3.8
Operating System: Linux Debian testing (but virtualenv)

Description

Orbits acquired with get_orbit() have data-values annotated with astropy-units, but only when the orbit is loaded from the web-services. It seems that when orbits are fetched from the (local) diskcache the unit-annotation has been lost.

Issue with speasy cache: ValueError: could not convert string to float: 'x_component'

It looks like speasy has two issues here:

not all cdf files are correctly converted to SpeasyVariable which is not necessary the biggest issue
if a cache entry can't be read back then speasy raises an error and fails

The second issue must be addressed, speasy should gracefully continue and get the data from either the proxy or the remote WS. Then Speasy can still log a warning so we can resolve the first issue product per product.

Find a better name!

@nicolasaunai

CDA inventory problems with underscores

Space Physics WebServices Client version: 1.1.2
Python version: 3.9.13
Operating System: Ubuntu 22.04

Description

It seems to me that the CDA inventory has problems obtaining data if there are _ in the name. At some point, they are transformed into -, and then they are not recognized any more.

What I Did

>>> import speasy as spz
>>> cda_tree = spz.inventories.data_tree.cda
>>> solo_swa = spz.get_data(cda_tree.Solar_Orbiter.SOLO.SWA_PAS.SOLO_L2_SWA_PAS_GRND_MOM, "2021-11-3", "2021-11-4")
ValueError: Unknown parameter: SOLO_L2_SWA-PAS-GRND-MOM
>>> solo_ept = spz.get_data(cda_tree.Solar_Orbiter.SOLO.EPD_EPT_NORTH_RATES.SOLO_L2_EPD_EPT_NORTH_RATES, "2021-11-3", "2021-11-4")
ValueError: Unknown parameter: SOLO_L2_EPD-EPT-NORTH-RATES

Cache entries name only depend on product name + time range

This is a limitation for web-services where data can change depending on another parameter value.
For example sscweb accepts a coordinate system as parameter which is just ignored for cache entry naming :/.

AMDA : downloading multidimensional data fails

Space Physics WebServices Client version: 0.9.0
Python version: 3.8.10
Operating System: Linux Ubuntu 20.04.2

Description

When downloading multidimensional data pandas fails to load CSV data because of duplicates in the column names. This is due to the fact that column names are obtained by splitting the character sequence by "," when comma characters appear in individual column names.
For multidimensional data columns are named "x[i,j],x[i,j+1],...,x[i+1,0],....etc"

Simple fix : split using ", " separator instead (file speasy/webservices/amda/utils.py, line 36)

What I Did

import speasy as spz
p = spz.get_data("amda/psp_spe_EvsEvspa", "2020-07-30", "2020-07-31")

  File "/home/aschulz/Documents/python/speasy_module/speasy_official/speasy/speasy/webservices/amda/utils.py", line 38, in load_csv
    data = pds.read_csv(f, comment='#', delim_whitespace=True, header=None, names=columns).values.transpose()
  File "/home/aschulz/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 676, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/home/aschulz/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 445, in _read
    _validate_names(kwds.get("names", None))
  File "/home/aschulz/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 413, in _validate_names
    raise ValueError("Duplicate names are not allowed.")

How to speedup 'import speasy as spz' ?

Space Physics WebServices Client version: 1.0.2
Python version: 3.8.10
Operating System: Ubuntu 20.04.5 LTS

Description

When I use speasy, of course I 'import speasy as spz'
But for tests purpose I run my module very often.
Unfortunately the import line takes much time.

I know it is possible to cache the amda inventory.
Is there a way to allow caching for the whole speasy import ?

What I Did

Read the doc.

Add Ipython completion

For get_data or get_xxxx parameters. This might be handy for users to find products.

The type of is_public should be boolean and not str

Space Physics WebServices Client version:
Python version:
Operating System:

Description

speasy/webservices/amda/ws.py line 44:
collection[xmlid].is_public

The type of is_public should be boolean and not str. In my case I have user_parameter and _is_user_prod returns always False because of this condition return not collection[xmlid].is_public then I checked the type of is_public and I found that it is a str and not boolean.

What I Did

spz.amda.get_data(amda_list_user_parameters[0],
                                           start_time, end_time)

Loading speasy is slow

Space Physics WebServices Client version: 0.10.1
Python version: 3.8
Operating System: ubuntu

Description

Loading speasy takes a couple seconds. Could we not store the inventory contents in the cache folder, accelerating the module initialization on subsequent imports. Perhaps we could start a detached process (in another thread) who's job would be to check all providers inventory and update the contents of the cache in a non-blocking way
On average importing speasy takes 10 seconds on my machine.

What I Did

import time
t0 = time.time()
import speasy as spz
t0 = time.time() - t0
print(f"speasy loaded in {t0} seconds")

Output

speasy loaded in 10.583851099014282 seconds

AMDA get_data never returns on long requests

Space Physics WebServices Client version: 0.10.1
Python version: 3.8.10
Operating System: Ubuntu

Description

AMDA creates background jobs to deal with requests that take too long to answer (timeout exceeded). Speasy is not notified of this fact, thus when trying to retrieve a large dataset the get_data method may never return, it will wait indefinitely.

What I Did

import speasy as spz
param_id = "amda/solo_b_rtn_hr"
start = "2020/01/01T00:00:00"
stop = "2021/01/01T00:00:00"
p = spz.get_data(param_id, start, stop)

Solution

Modified the dl_parameter function in speasy.webservices.amda._impl module :

import numpy as np
from datetime import timedelta
....
    def parameter_concat(self, param1, param2):
        """Concatenate parameters
        """
        if param1 is None and param2 is None:
            return None
        if param1 is None:
            return param2
        if param2 is None:
            return param1
        param1.time = np.hstack((param1.time, param2.time))
        param1.data = np.hstack((param1.data, param2.data))
        return param1

    def dl_parameter(self, start_time: datetime, stop_time: datetime, parameter_id: str, **kwargs) -> Optional[
        SpeasyVariable]:
        if isinstance(start_time, datetime):
            start_time = start_time.timestamp()
        if isinstance(stop_time, datetime):
            stop_time = stop_time.timestamp()
        dt = timedelta(days=1).total_seconds()
        if stop_time - start_time > dt:
            var = None
            curr_t = start_time
            while curr_t < stop_time:
                #print(f"Getting block {datetime.utcfromtimestamp(curr_t)} -> {datetime.utcfromtimestamp(curr_t + dt)}")
                if curr_t + timedelta(days=1).total_seconds() < stop_time:
                    var = self.parameter_concat(var , self.dl_parameter(curr_t, curr_t + dt, parameter_id, **kwargs))
                else:
                    var = self.parameter_concat(var, self.dl_parameter(curr_t, stop_time, parameter_id, **kwargs))
                curr_t += dt
            return var

        url = rest_client.get_parameter(
            startTime=start_time, stopTime=stop_time, parameterID=parameter_id, timeFormat='UNIXTIME',
            server_url=self.server_url, **kwargs)
        if url is not None:
            var = load_csv(url)
            if len(var):
                log.debug(
                    f'Loaded var: data shape = {var.values.shape}, data start time = {datetime.utcfromtimestamp(var.time[0])}, data stop time = {datetime.utcfromtimestamp(var.time[-1])}')
            else:
                log.debug('Loaded var: Empty var')
            return var
        return None

Error getting data from CDAWeb

Space Physics WebServices Client version: 0.10.1
Python version: 3.8
Operating System: Ubuntu

Description

Retrieving data from CDAWeb encountered the following error :

File ~/Documents/pyhc/venv/lib/python3.8/site-packages/speasy/webservices/cda/__init__.py:36, in _read_csv(url, *args, **kwargs)
     34     time = np.array([t.timestamp() for t in df.index])
     35     return SpeasyVariable(time=time, data=df.values, columns=[c for c in df.columns])
---> 36 except pds.io.common.EmptyDataError:
     37     return SpeasyVariable()

AttributeError: module 'pandas.io.common' has no attribute 'EmptyDataError'

What I Did

param_id = "cdaweb/WI_ELSP_3DP/FLUX"
start = "2022/01/15T00:00:00"
stop = "2022/01/16T00:00:00"
p = spz.get_data(param_id, start, stop)

Better handling of catalog download error when xmlid doesn't exist

raw call stack:

In [2]: spz.amda.get_catalog("sharedcatalog_1")
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
/usr/lib64/python3.10/urllib/request.py in open_local_file(self, req)
   1504         try:
-> 1505             stats = os.stat(localfile)
   1506             size = stats.st_size

FileNotFoundError: [Errno 2] No such file or directory: '/home/jeandet/Documents/prog/speasy/{"error":"No such object sharedcatalog_1 for user impex"}'

During handling of the above exception, another exception occurred:

URLError                                  Traceback (most recent call last)
<ipython-input-2-41164a0829ab> in <module>
----> 1 spz.amda.get_catalog("sharedcatalog_1")

~/Documents/prog/speasy/speasy/core/cache/__init__.py in wrapped(disable_cache, force_refresh, *args, **kwargs)
    209                 return self.add_to_cache(cache_entry, function(*args, **kwargs))
    210             else:
--> 211                 return self.get_from_cache(cache_entry) or self.add_to_cache(cache_entry, function(*args, **kwargs))
    212 
    213         return wrapped

~/Documents/prog/speasy/speasy/webservices/amda/ws.py in get_catalog(self, catalog_id, **kwargs)
    391 
    392         """
--> 393         return self._impl.dl_catalog(to_xmlid(catalog_id), **kwargs)
    394 
    395     def parameter_range(self, parameter_id: str or AMDAParameterIndex or AMDADatasetIndex) -> Optional[DateTimeRange]:

~/Documents/prog/speasy/speasy/core/cache/__init__.py in wrapped(disable_cache, force_refresh, *args, **kwargs)
    209                 return self.add_to_cache(cache_entry, function(*args, **kwargs))
    210             else:
--> 211                 return self.get_from_cache(cache_entry) or self.add_to_cache(cache_entry, function(*args, **kwargs))
    212 
    213         return wrapped

~/Documents/prog/speasy/speasy/webservices/amda/_impl.py in dl_catalog(self, catalog_id, **kwargs)
    131         url = rest_client.get_catalog(catID=catalog_id, server_url=self.server_url, **kwargs)
    132         if url is not None:
--> 133             catalog = load_catalog(url)
    134             if catalog:
    135                 log.debug(f'Loaded catalog: id = {catalog_id}')  # lgtm[py/clear-text-logging-sensitive-data]

~/Documents/prog/speasy/speasy/webservices/amda/utils.py in load_catalog(filename)
    111     if '://' not in filename:
    112         filename = f"file://{os.path.abspath(filename)}"
--> 113     with urlopen(filename) as votable:
    114         # save the timetable as a dataframe, speasy.common.SpeasyVariable
    115         # get header data first

/usr/lib64/python3.10/urllib/request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context)
    214     else:
    215         opener = _opener
--> 216     return opener.open(url, data, timeout)
    217 
    218 def install_opener(opener):

/usr/lib64/python3.10/urllib/request.py in open(self, fullurl, data, timeout)
    517 
    518         sys.audit('urllib.Request', req.full_url, req.data, req.headers, req.get_method())
--> 519         response = self._open(req, data)
    520 
    521         # post-process response

/usr/lib64/python3.10/urllib/request.py in _open(self, req, data)
    534 
    535         protocol = req.type
--> 536         result = self._call_chain(self.handle_open, protocol, protocol +
    537                                   '_open', req)
    538         if result:

/usr/lib64/python3.10/urllib/request.py in _call_chain(self, chain, kind, meth_name, *args)
    494         for handler in handlers:
    495             func = getattr(handler, meth_name)
--> 496             result = func(*args)
    497             if result is not None:
    498                 return result

/usr/lib64/python3.10/urllib/request.py in file_open(self, req)
   1481                 raise URLError("file:// scheme is supported only on localhost")
   1482         else:
-> 1483             return self.open_local_file(req)
   1484 
   1485     # names for the localhost

/usr/lib64/python3.10/urllib/request.py in open_local_file(self, req)
   1520                 return addinfourl(open(localfile, 'rb'), headers, origurl)
   1521         except OSError as exp:
-> 1522             raise URLError(exp)
   1523         raise URLError('file not on local host')
   1524 

URLError: <urlopen error [Errno 2] No such file or directory: '/home/jeandet/Documents/prog/speasy/{"error":"No such object sharedcatalog_1 for user impex"}'>

Download job manager

While working on Catalog to DataFrame conv, I was thinking that maybe we could add a new module to easy scheduling data download.
For example when an user want to retrieve a list of products data for each event in a Catalog, he has to do something like:

data = { product:[spz.get_data(product,*event) for event in catalog] for product in product_list}

Which is OK for the simplicity but lacks some control over parallelism.

We could provide something like:

data = speasy.jobs.get_data( product_list, catalogs, max_concurrent_jobs)

sciqlop / speasy Goto Github PK

speasy's Introduction

What Is SciQLop?

Main Features

Upcoming features

How to install SciQLop

Mac Users

Linux Users

From sources

Python user API Examples:

How to contribute

Credits

Thanks

speasy's People

Contributors

Stargazers

Watchers

Forkers

speasy's Issues

Description

What I Did

Description

What I Did

Description

What I Did

Summary

Description

What I Did

Description

What I Did

Description

What I Did

Description

What I Did

Description

What I Did

Description

What I Did

Description

What I Did

Description

What I Did

Description

Description

What I Did

Description

What I Did

Description

What I Did

Description

What I Did

Description

What I Did

Description

What I Did

Solution

Description

What I Did

Recommend Projects

Recommend Topics

Recommend Org