pysat / pysatseasons

Seasonal Analysis routines for pysat

pysat: Python Satellite Data Analysis Toolkit

The Python Satellite Data Analysis Toolkit (pysat) provides a simple and flexible interface for robust data analysis from beginning to end - including downloading, loading, cleaning, managing, processing, and analyzing data. Pysat's plug-in design allows analysis support for any data, including user provided data sets. The pysat team provides a variety of plug-ins to support public scientific data sets in packages such as pysatNASA, pysatMadrigal, and more, available as part of the general pysat ecosystem.

Full Documentation

JGR-Space Physics Publication

Pysat Ecosystem Publication

Citation Info

Come join us on Slack! An invitation to the pysat workspace is available in the 'About' section of the pysat GitHub Repository. Development meetings are generally held fortnightly.

Main Features

• Instrument independent analysis routines.
• Instrument object providing an interface for downloading and analyzing a wide variety of science data sets.
  • Uses pandas or xarray for the underlying data structure; capable of handling the many forms scientific measurements take in a consistent manner.
  • Standard scientific data handling tasks (e.g., identifying, downloading, and loading files and cleaning and modifying data) are built into the Instrument object.
  • Supports metadata consistent with the netCDF CF-1.6 standard. Each variable has a name, long name, and units. Note units are informational only.
• Simplifies data management
  • Iterator support for loading data by day/file/orbit, independent of data storage details.
  • Orbits are calculated on the fly from loaded data and span day breaks.
  • Iterate over custom seasons
• Supports rigorous time-series calculations that require spin up/down time across day, orbit, and file breaks.
• Includes helper functions to reduce the barrier in adding new science instruments to pysat

Installation

The following instructions provide a guide for installing pysat and give some examples on how to use the routines.

Prerequisites

pysat uses common Python modules, as well as modules developed by and for the Space Physics community. This module officially supports Python 3.X+.

Common modules	Community modules
dask	netCDF4
numpy >= 1.12
pandas
portalocker
pytest
scipy
toolz
xarray

PyPi Installation

pip install pysat

GitHub Installation

git clone https://github.com/pysat/pysat.git

Change directories into the repository folder and run the pyproject.toml or setup.py file. For a local install use the "--user" flag after "install".

cd pysat/
python -m build .
pip install .

Using pysat

• The first time pysat is run, you will need to specify a directory to store the data. In Python, run:

pysat.params['data_dirs'] = 'path/to/directory/that/may/or/may/not/exist'

• Nominal organization of data is top_dir/platform/name/tag/inst_id/files

Detailed examples and tutorials for using pysat are available in the documentation.

pysatseasons's Issues

Check standard docs for pysatSeasons

There is now a central repository for templates at https://github.com/pysat/pysatEcosystem_Template

• Make sure current issues / actions / etc are up to standard
• Make sure any extra bits here are captured in the templates

MAINT: update to use pyproject.toml

Is your feature request related to a problem? Please describe.
setup.py is deprecated by pip.

Describe the solution you'd like
Update using pyproject.toml standards

Describe alternatives you've considered
😴

Additional context
Followup from pysat/pysat#1051

ENH: Streamlining seasonal averaging

Migrating from pysat/pysat#165. From @aburrell.

The seasonal binning and averaging is not very flexible. Currently, there are 1D and 2D binning functions that only allow medians to be calculated. I think the best way to improve this is to write an ND median/ave/other stat function. The statistical function could be provided as an input, allowing people to do clever things like find the moments from a Gaussian fit the the data in the desired bin. Then all the stat functions could go into utils.stats, and the seasonal binning part would live in ssnl. And if we do ND, then we won't have to keep re-writting things when we eventually want 3D or 4D binning.

BUG: Panel deprecation

Migrating from pysat/pysat#159. See full history there.

/home/travis/build/rstoneback/pysat/pysat/ssnl/avg.py:167: FutureWarning: 
Panel is deprecated and will be removed in a future version.
The recommended way to represent these types of 3-dimensional data are with a MultiIndex 
on a DataFrame, via the Panel.to_frame() method
Alternatively, you can use the xarray package http://xarray.pydata.org/en/stable/.
Pandas provides a `.to_xarray()` method to help automate this conversion.

MAINT: remove deprecated instruments from tests

Is your feature request related to a problem? Please describe.
pysat 3.2.0 will drop 3 instruments that are used in the tests here. They either represent redundancy or features that will no longer be supported.

Describe the solution you'd like

• Drop tests for pysat_testing2d (no longer supported)
• Replace pysat_testing_xarray with pysat_ndtesting
• Replace pysat_testing_2dxarray with pysat_ndtesting

Describe alternatives you've considered
😴

Additional context
There may need to be some updates at pysat to make pysat_ndtesing a suitable replacement for pysat_testing_xarray needed to update tests so that profile size is not hard wired

Impacts from Occurrence Probability

Describe the bug
Running Occurrence probability analysis leaves Instrument object on last day. If a user tries to run it again, it won't work. Already at end of bounds.

To Reproduce
Steps to reproduce the behavior:

#  Orbit breakdown info
orbit_info = {'kind': 'lt', 'index':'Magnetic_Local_Time'}

# We will employ a centered time-based calculation. Engage pysat's data padding.
pad = pds.DateOffset(seconds=30)

# Instantiate IVM instrument data and 
ivm = pysat.Instrument('icon', 'ivm', inst_id='a', orbit_info=orbit_info,
                       pad=pad)

# Set the range of dates for the analysis
ivm.bounds = (dt.datetime(2020, 1, 1), dt.datetime(2020, 1, 15))

answer = pysatSeasons.occur_prob.by_orbit2D(ivm, [0, 360, 24], 'Longitude',
                                            [-20, 20, 10], 'Magnetic_Latitude',
                                            ['Ion_Density'], [7.E3], returnBins=True)

answer = pysatSeasons.occur_prob.by_orbit2D(ivm, [0, 360, 24], 'Longitude',
                                            [-20, 20, 10], 'Magnetic_Latitude',
                                            ['Ion_Density'], [7.E3], returnBins=True)

Expected behavior
Running it twice should yield the same result, twice.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: MacOS
• Version Python 3.9
• Other details about your setup that could be relevant

Additional context
Add any other context about the problem here.

Performance binning 1D data from xarray

Is your feature request related to a problem? Please describe.
unit tests show that binning 1D data from xarray is quite a bit slower than for pandas.

Describe the solution you'd like
Perform a performance analysis and sort out why.

Describe alternatives you've considered
Leave as is.

Additional context
Add any other context or screenshots about the feature request here.

ENH: Smoothed bins in seasonal averaging

Migrating from pysat/pysat#170. From @aburrell.

The current bins in seasonal averaging don't have any overlap. Add an option for this, to permit smoothing.

ENH: Support for constellations

In pysat._constellation, support for the .add and .difference is being removed for pysat 3.0.0. pysatSeasons already supports constellation objects. Make sure that the intent of those functions can be replicated here.

ENH: Support for xarray

Linked from pysat/pysat#47.

Current seasonal routines assume pandas data. This needs to be generalized to accept xarray data.

TST: update testing standards

Is your feature request related to a problem? Please describe.
After the version caps for pandas are removed, CI tests and standards need to be updated.

Describe the solution you'd like

• Test against python 3.8 and 3.9
• Add style and docstring testing
• Update style throughout

Describe alternatives you've considered
🦀

Additional context
Needs approval and merge of #27.

ENH: Consider alternative binning method that may scale better

Is your feature request related to a problem? Please describe.
Binning algorithms of order (N) scales better to large bin sets. Current algorithm is worse than that.

Describe the solution you'd like
For sorted arrays we can use a variant of this solution, which has already be tweaked as part of this pull.

np.digitize now works on multidimensional arrays, that's good. This can then be parsed using np.unique save having to do a bunch of np.where's, which is also good. We may have to sort multiple times... less than good. Unique doesn't work how I'd like over multidimensional arrays which prevents the best solution.

Describe alternatives you've considered
Leaving support as is.

Additional context
Add any other context or screenshots about the feature request here.

DOC: update demo codes

Describe the bug

• Demo codes currently use old custom.attach rather than custom_attach.
• Add docstrings to functions in demo codes to assist flow for users
• Scrub for modern style

To Reproduce
Try to run a demo with pysat 3.0+

Expected behavior
Code should match standards for first time users