pydap / pydap Goto Github PK

View Code? Open in Web Editor NEW

131.0 131.0 88.0 8.84 MB

A Python library implementing the Data Access Protocol (DAP, aka OPeNDAP or DODS).

Home Page: https://pydap.github.io/pydap/

License: MIT License

CSS 0.17% Python 97.61% HTML 2.21%

dap data dods opendap science

pydap's Introduction

pydap

pydap is an implementation of the Opendap/DODS protocol, written from scratch in pure python. You can use pydap to access scientific data on the internet without having to download it; instead, you work with special array and iterable objects that download data on-the-fly as necessary, saving bandwidth and time. The module also comes with a robust-but-lightweight Opendap server, implemented as a WSGI application.

Quickstart

pydap is a lighweight python package that you can use in either of the two modalities: a client and as a server. You can install the latest version using pip. After installing pip you can install pydap with this command:

    $ pip install pydap

This will install pydap together with all the required dependencies. pydap is also available through Anaconda. Below we install pydap and its required dependencies, along with common additional packages in a fresh conda environment named pydap:

$ mamba create -n pydap -c conda-forge python=3.10 pydap numpy jupyterlab ipython netCDF4 scipy matplotlib

Now you simply activate the pydap environment:

mamba activate pydap

You can now use pydap as a client and open any remotely served dataset, and pydap will download the accessed data on-the-fly as needed:

    >>> from pydap.client import open_url
    >>> dataset = open_url('http://test.opendap.org/dap/data/nc/coads_climatology.nc')
    >>> var = dataset['SST']
    >>> var.shape
    (12, 90, 180)
    >>> var.dtype
    dtype('>f4')
    >>> data = var[0,10:14,10:14]  # this will download data from the server
    >>> data
    <GridType with array 'SST' and maps 'TIME', 'COADSY', 'COADSX'>
    >>> print(data.data)
    [array([[[ -1.26285708e+00,  -9.99999979e+33,  -9.99999979e+33,
              -9.99999979e+33],
            [ -7.69166648e-01,  -7.79999971e-01,  -6.75454497e-01,
              -5.95714271e-01],
            [  1.28333330e-01,  -5.00000156e-02,  -6.36363626e-02,
              -1.41666666e-01],
            [  6.38000011e-01,   8.95384610e-01,   7.21666634e-01,
               8.10000002e-01]]], dtype=float32), array([ 366.]), array([-69., -67., -65., -63.]), array([ 41.,  43.,  45.,  47.])]

For more information, please check the documentation on using pydap as a client.

pydap also comes with a simple server, implemented as a WSGI application. To use it, you first need to install the server and optionally a data handler:

    $ pip install "pydap[server,handlers.netcdf]"

This will install support for netCDF files; more handlers for different formats are available, if necessary. Now create a directory for your server data.

To run the server just issue the command:

    $ pydap --data ./myserver/data/ --port 8001 --workers 4 --threads 4

This will start a standalone server running on the default http://localhost:8001/, serving netCDF files from ./myserver/data/, similar to the test server at http://test.pydap.org/. Since the server uses the WSGI standard, pydap uses by default 1 worker and 1 thread, but these can be defined by the user like in the case above (4 workers and 4 threads). Pydap can also easily be run behind Apache. The server documentation has more information on how to better deploy pydap.

Documentation

For more information, see the pydap documentation.

Help

If you need any help with pydap, please feel free to send an email to the mailing list

pydap's People

Contributors

Stargazers

Watchers

Forkers

tomkralidis igormilovanovic bolkhovsky seandunford efiring pacificclimate cedadev openearth-attic pymonger tuw-geo opendap hdfeos d70-t freemansw1 ryanfeather jfburkhart bwallin iuriiblokhin siggyf castelao lewismc laliberte taeold ocefpaf mbarimike niklasphabian omkar20895 ghisvail jenfly phrrngtn adamcandy jiaxinxu1994 hugoroaster danbgray mihow jblarsen neishm pauleckhardt nublia flackdl bobfrat etringer monodeldiablo rabernat francescalted gunjandu kxt123 gitter-badger rsignell-usgs betodealmeida ysliaw01 mandyellow crossi202 wholmgren joelrahman wallinb astrojuanlu epifanio nathanlcarlson olimcc shreddd tirkarthi qpanscience ternaustralia ikhomyakov mhaberler jondoesntgit cskarby johnmlarkin anthonybaxter raybellwaves mbexhrs3 pmav99 captainkirk99 jgliss mschulzie headtr1ck rbeucher hyoklee ndp-opendap bgiebl raphaeljolivet trail-of-forks mikejmnez

pydap's Issues

Reading MERRA-2 data

I'm having the same issue as noted in #63, trying to read MERRA-2 data. I have updated my Pydap to the latest master version from Github, using pip install -e git+https://github.com/pydap/pydap.git#egg=pydap (I think this is the correct way to do it, but please let me know if I've gone astray - I haven't installed packages other than the official release versions before, so I searched around somewhat blindly until finding something that seemed applicable.) I added a print() statement to the open_url() source code to confirm that I was importing the new version in my ipython shell, and it looks like I'm running the new version. However, I'm getting an error for 3-D, as noted in #63, and also now the same error for 2-D data (whereas I was able to read 2-D data with the release version of pydap).

from pydap.client import open_url
from pydap.cas.urs import setup_session

url2 = ('https://goldsmr4.gesdisc.eosdis.nasa.gov/opendap/MERRA2/'
        'M2I1NXASM.5.12.4/2015/08/MERRA2_400.inst1_2d_asm_Nx.20150802.nc4')
url3 = ('https://goldsmr5.gesdisc.eosdis.nasa.gov/opendap/hyrax/MERRA2/'
       'M2I3NVASM.5.12.4/2015/08/MERRA2_400.inst3_3d_asm_Nv.20150802.nc4')

session = setup_session(username, password)
ds2 = open_url(url2, session=session)
ds3 = open_url(url3, session=session)

print(ds2['TS'].shape)
print(ds2['TS'][0, 0, 0])
print(ds3['T'].shape)
print(ds3['T'][0, 0, 0, 0])

I can print the shapes of the arrays and other metadata, but when I try to access the array values in either ds2 or ds3 I get this error:

ValueError                                Traceback (most recent call last)
<ipython-input-23-80aa8f6dccdc> in <module>()
----> 1 ds3['T'][:, 0, 0, 0]

/home/jwalker/dynamics/python/src/pydap/src/pydap/model.pyc in __getitem__(self, index)
    251             return np.vectorize(decode_np_strings)(self.data[index])
    252         else:
--> 253             return self.data[index]
    254
    255     def __len__(self):

/home/jwalker/dynamics/python/src/pydap/src/pydap/handlers/dap.pyc in __getitem__(self, index)
    137         r = GET(url, self.application, self.session)
    138         raise_for_status(r)
--> 139         dds, data = r.body.split(b'\nData:\n', 1)
    140         dds = dds.decode(r.content_encoding or 'ascii')
    141

ValueError: need more than 1 value to unpack

I'm not sure what is going on. Can you please try reading the files in my example and look into it? Any help with this issue would be greatly appreciated. And thank you so so so much for developing these updates to handle the new authentication procedures for NASA data. Such a lifesaver!!

FutureWarning regarding use of structured arrays

While testing pydap-3.2.0 on Debian Stretch (Numpy 1.12.x), some tests throw a FutureWarning which you guys might want to check out. Here is the relevant part of the nose test log:

[...]
Test item retrieval. ... /usr/lib/python3/dist-packages/numpy/core/records.py:507: FutureWarning: Numpy has detected that you may be viewing or writing to an array returned by selecting multiple fields in a structured array.
[...]
Test using density as a selection. ... /usr/lib/python3/dist-packages/numpy/core/records.py:507: FutureWarning: Numpy has detected that you may be viewing or writing to an array returned by selecting multiple fields in a structured array.

Tests cannot be run standalone

Despite the fact that pydap.tests is provided in the PyPI tarball, the test suite cannot be run for the following reasons:

Missing data: *.das, *.dds, *.dods. These should be added to the package_data parameter of the setup object in setup.py.
Reliance on Pydap.egg-info. This prevents the possibility of running the tests in a standalone way by doing:

python setup.py build
cd build/lib
python -m nose -a '!auth'

This is particularly desirable for packaging purposes (Debian for instance), where the package is built for all supported Python versions concurrently. Each Python version has its own build directory in which the test suite is run. Right now, this is not possible for these 2 reasons.

AttributeError: module 'string' has no attribute 'strip' when installing with Python3

Hi folks, as noted some time ago Pydap cannot be installed using Python3.
This issue should fix the issue below.

Collecting pydap>=3.1.1 (from -r ocw-pip-dependencies.txt (line 3))
  Using cached Pydap-3.1.1.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-bpljh_u8/pydap/setup.py", line 10, in <module>
        paver.tasks.main()
      File "paver-minilib.zip/paver/tasks.py", line 883, in main
      File "paver-minilib.zip/paver/tasks.py", line 851, in _launch_pavement
      File "pavement.py", line 31, in <module>
        classifiers=filter(None, map(string.strip, '''
    AttributeError: module 'string' has no attribute 'strip'

Namespace testcase fails to pass

In a fresh checkout of pydap in a clean venv, the namespace testcase does not pass:

======================================================================
FAIL: Test the namespace.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gvaillan/.virtualenvs/pydap/src/pydap/src/pydap/tests/test_pydap.py", line 20, in test_namespace
    pydap.__doc__, "Declare the namespace ``pydap`` here.")
AssertionError: None != 'Declare the namespace ``pydap`` here.'

----------------------------------------------------------------------
Ran 311 tests in 0.582s

Indeed, if I manually run:

>>> import pydap
>>> print(pydap.__doc__)
None

I am wondering whether this test case is actually useful, beyond increasing the code coverage?

Drop support for 3.3

I recommend we drop support for python 3.3. Most major projects are doing the same and security patches will stop being issued by September 2017.

Merge URS authentication branch

The Merge request for the URS authentication seems like it's languishing. See Pull request #11 (Added support for NASA/URS SSO service)

Thanks,
James

model.GridType differences from 3.1.1

GridType in Version 3.1.1 has len() method and other attributes:

# These attributes all come from the array object.
def __len__(self): return len(self.array)
@property
def type(self): return self.array.type
def _get_shape(self): return self.array.shape
def _set_shape(self, shape): self.array.shape = shape
shape = property(_get_shape, _set_shape)

Should they be added to maintain compatibility?

pydap.responses.das attribute order is non-deterministic

Description

I have been using tox to test the develop branch across Python versions 2.6, 2.7 and 3.3. I have noticed that when running the tests with tox, I get sporadic failures (ranging between 0 and 4 failures per run) in a number of tests, most commonly in test_responses_das.py.

When running the tests with the standard nosetests command, all of the tests pass.

I think that I've narrowed it down to the fact that tox randomizes the Python hash seed on every run, so (correctly) the order of dictionary items is not guaranteed unless you take measures to do the ordering yourself.

Example

Here is an example of running the test suite with different hash seeds:

james@basalt ~/code/git/pydap $ tox -e py26 --hashseed 4016568519 
py26 develop-inst-nodeps: /home/james/code/git/pydap
py26 runtests: PYTHONHASHSEED='4016568519'
py26 runtests: commands[0] | pip install -e .[testing,functions,tests]
Obtaining file:///home/james/code/git/pydap
  Running setup.py (path:/home/james/code/git/pydap/setup.py) egg_info for package from file:///home/james/code/git/pydap
  Installing extra requirements: 'testing,functions,tests'
Requirement already satisfied (use --upgrade to upgrade): Numpy in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): requests in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): singledispatch in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): Webob in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): Jinja2 in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): docopt in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): gunicorn in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): six>=1.4.0 in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): ordereddict in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): gsw in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): coards in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): scipy in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): WebTest in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): beautifulsoup4 in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): nose in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): coverage in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): markupsafe in ./.tox/py26/lib/python2.6/site-packages (from Jinja2->Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): waitress>=0.8.5 in ./.tox/py26/lib/python2.6/site-packages (from WebTest->Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): setuptools in ./.tox/py26/lib/python2.6/site-packages (from waitress>=0.8.5->WebTest->Pydap==3.2)
Installing collected packages: Pydap
  Running setup.py develop for Pydap
    Creating /home/james/code/git/pydap/.tox/py26/lib/python2.6/site-packages/Pydap.egg-link (link to src)
    Pydap 3.2 is already the active version in easy-install.pth
    Installing dods script to /home/james/code/git/pydap/.tox/py26/bin
    Installing pydap script to /home/james/code/git/pydap/.tox/py26/bin

    Installed /home/james/code/git/pydap/src
Successfully installed Pydap
Cleaning up...
py26 runtests: commands[1] | python setup.py nosetests --pdb
running nosetests
running egg_info
writing requirements to src/Pydap.egg-info/requires.txt
writing namespace_packages to src/Pydap.egg-info/namespace_packages.txt
writing src/Pydap.egg-info/PKG-INFO
writing top-level names to src/Pydap.egg-info/top_level.txt
writing dependency_links to src/Pydap.egg-info/dependency_links.txt
writing entry points to src/Pydap.egg-info/entry_points.txt
reading manifest file 'src/Pydap.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'src/Pydap.egg-info/SOURCES.txt'
.....SS......................./home/james/code/git/pydap/src/pydap/handlers/dap.py:145: DeprecationWarning: converting an array with ndim > 0 to an index will result in an error in the future
  out.append(data[:n])
/home/james/code/git/pydap/src/pydap/handlers/dap.py:146: DeprecationWarning: converting an array with ndim > 0 to an index will result in an error in the future
  data = data[n + (-n % 4):]
...
......................................................................................................................................................................................Ran 311 tests in 3.132s

OK (SKIP=2)
_____________________________________________________________________________________________________ summary _____________________________________________________________________________________________________
  py26: commands succeeded
  congratulations :)

Compare this to a separate run with a different hash seed:

james@basalt ~/code/git/pydap $ tox -e py26 --hashseed 3775216848
py26 develop-inst-nodeps: /home/james/code/git/pydap
py26 runtests: PYTHONHASHSEED='3775216848'
py26 runtests: commands[0] | pip install -e .[testing,functions,tests]
Obtaining file:///home/james/code/git/pydap
  Running setup.py (path:/home/james/code/git/pydap/setup.py) egg_info for package from file:///home/james/code/git/pydap
  Installing extra requirements: 'testing,functions,tests'
Requirement already satisfied (use --upgrade to upgrade): Numpy in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): requests in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): singledispatch in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): Webob in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): Jinja2 in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): docopt in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): gunicorn in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): six>=1.4.0 in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): ordereddict in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): gsw in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): coards in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): scipy in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): WebTest in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): beautifulsoup4 in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): nose in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): coverage in ./.tox/py26/lib/python2.6/site-packages (from Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): markupsafe in ./.tox/py26/lib/python2.6/site-packages (from Jinja2->Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): waitress>=0.8.5 in ./.tox/py26/lib/python2.6/site-packages (from WebTest->Pydap==3.2)
Requirement already satisfied (use --upgrade to upgrade): setuptools in ./.tox/py26/lib/python2.6/site-packages (from waitress>=0.8.5->WebTest->Pydap==3.2)
Installing collected packages: Pydap
  Running setup.py develop for Pydap
    Creating /home/james/code/git/pydap/.tox/py26/lib/python2.6/site-packages/Pydap.egg-link (link to src)
    Pydap 3.2 is already the active version in easy-install.pth
    Installing pydap script to /home/james/code/git/pydap/.tox/py26/bin
    Installing dods script to /home/james/code/git/pydap/.tox/py26/bin

    Installed /home/james/code/git/pydap/src
Successfully installed Pydap
Cleaning up...
py26 runtests: commands[1] | python setup.py nosetests
running nosetests
running egg_info
writing dependency_links to src/Pydap.egg-info/dependency_links.txt
writing requirements to src/Pydap.egg-info/requires.txt
writing namespace_packages to src/Pydap.egg-info/namespace_packages.txt
writing top-level names to src/Pydap.egg-info/top_level.txt
writing src/Pydap.egg-info/PKG-INFO
writing entry points to src/Pydap.egg-info/entry_points.txt
reading manifest file 'src/Pydap.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'src/Pydap.egg-info/SOURCES.txt'
.....SS......................./home/james/code/git/pydap/src/pydap/handlers/dap.py:145: DeprecationWarning: converting an array with ndim > 0 to an index will result in an error in the future
  out.append(data[:n])
/home/james/code/git/pydap/src/pydap/handlers/dap.py:146: DeprecationWarning: converting an array with ndim > 0 to an index will result in an error in the future
  data = data[n + (-n % 4):]
......................................................................................................................................................................................FF.....F.................................F.........................................................
======================================================================
FAIL: Test the generated DAS response.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/james/code/git/pydap/src/pydap/tests/test_responses_das.py", line 113, in test_body
    """)
AssertionError: u'Attributes {\n    String description "A simple grid for testing.";\n    SimpleGrid {\n    }\n    x {\n        String axis "X";\n        String units "degrees_east";\n    }\n    y {\n        String axis "Y";\n        String units "degrees_north";\n    }\n}\n' != 'Attributes {\n    String description "A simple grid for testing.";\n    SimpleGrid {\n    }\n    x {\n        String units "degrees_east";\n        String axis "X";\n    }\n    y {\n        String units "degrees_north";\n        String axis "Y";\n    }\n}\n'

======================================================================
FAIL: Test the generated DAS response.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/james/code/git/pydap/src/pydap/tests/test_responses_das.py", line 88, in test_body
    """)
AssertionError: u'Attributes {\n    nested {\n        Int32 value 42;\n    }\n    String description "A simple sequence for testing.";\n    cast {\n        id {\n        }\n        lon {\n            String axis "X";\n        }\n        lat {\n            String axis "Y";\n        }\n        depth {\n            String axis "Z";\n        }\n        time {\n            String axis "T";\n            String units "days since 1970-01-01";\n        }\n        temperature {\n        }\n        salinity {\n        }\n        pressure {\n        }\n    }\n}\n' != 'Attributes {\n    String description "A simple sequence for testing.";\n    nested {\n        Int32 value 42;\n    }\n    cast {\n        id {\n        }\n        lon {\n            String axis "X";\n        }\n        lat {\n            String axis "Y";\n        }\n        depth {\n            String axis "Z";\n        }\n        time {\n            String units "days since 1970-01-01";\n            String axis "T";\n        }\n        temperature {\n        }\n        salinity {\n        }\n        pressure {\n        }\n    }\n}\n'

======================================================================
FAIL: Test the generated DAS response.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/james/code/git/pydap/src/pydap/tests/test_responses_das.py", line 154, in test_body
    """)
AssertionError: u'Attributes {\n    types {\n        nested {\n            Float64 float 1000;\n            Int32 array 1;\n            String string "bar";\n            Int32 list 42, 43;\n        }\n        String key "value";\n        b {\n        }\n        i32 {\n        }\n        ui32 {\n        }\n        i16 {\n        }\n        ui16 {\n        }\n        f32 {\n        }\n        f64 {\n        }\n        s {\n        }\n        u {\n        }\n    }\n}\n' != 'Attributes {\n    types {\n        String key "value";\n        nested {\n            Int32 array 1;\n            Float64 float 1000;\n            Int32 list 42, 43;\n            String string "bar";\n        }\n        b {\n        }\n        i32 {\n        }\n        ui32 {\n        }\n        i16 {\n        }\n        ui16 {\n        }\n        f32 {\n        }\n        f64 {\n        }\n        s {\n        }\n        u {\n        }\n    }\n}\n'

======================================================================
FAIL: Test that the data redirect works with a subset request.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/james/code/git/pydap/src/pydap/tests/test_responses_html.py", line 88, in test_post_multiple_selection
    res.location, "http://localhost/.ascii?sequence.int,sequence.byte")
AssertionError: 'http://localhost/.ascii?sequence.byte,sequence.int' != 'http://localhost/.ascii?sequence.int,sequence.byte'

Ran 311 tests in 3.127s

FAILED (SKIP=2, failures=4)
ERROR: InvocationError: '/home/james/code/git/pydap/.tox/py26/bin/python setup.py nosetests'
_____________________________________________________________________________________________________ summary _____________________________________________________________________________________________________
ERROR:   py26: commands failed

If you look very closely there, you'll notice that the attribute "String description ..." is transposed with the attribute "nested {...}". The behaviour is similar for Python 2.7.

How to fix

This obvious fix seems to be to make stronger guarantees about the ordering of attributes. I don't necessarily know what those guarantees should be, but there are a couple options:

Use and an OrderedDict for the DapType.attributes dictionary. In this case attributes returned in the DAS response will be ordered according to insertion order.
Sort the dict keys before iterating over them in pydap.responses.das:das(). There appear to be three places where this would need to be changed. In this case attributes returned in the DAS response will be ordered alphabetically.

I can fix it in either way, if someone would like to provide an opinion on the better option (or comment otherwise).

pip3 install pydap.responses.netcdf gives error

The error is as follows :

Collecting pydap.responses.netcdf
Downloading pydap.responses.netcdf-0.1.4.1.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-build-90qbdmot/pydap.responses.netcdf/setup.py", line 10, in
paver.tasks.main()
File "/usr/local/lib/python3.5/dist-packages/paver/tasks.py", line 878, in main
_launch_pavement(args)
File "/usr/local/lib/python3.5/dist-packages/paver/tasks.py", line 846, in _launch_pavement
exec(compile(source, environment.pavement_file, 'exec'), mod.dict)
File "pavement.py", line 23, in
classifiers=filter(None, map(string.strip, '''
AttributeError: module 'string' has no attribute 'strip'

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-90qbdmot/pydap.responses.netcdf/

Check byte use

I noticed in some places of the code byte is signed, while in others it's unsigned. I should verify how the DAP spec defines it (unsigned, I think) and ensure the proper type in the code.

Document simple test server

#92 adds a lightweight server class to be used for development purposes Add a section in the sever docs to describe its existence and how to use it.

Confuse about the parameters of the function `setup_session()` when login ESGF

The function setup_session() has three parameters: uri, username and password, but in the document it shows >>> session = setup_session(openid, password), and when I get the data in this way the open_url() throws 401. Dose it mean that the uri is openid or username is openid?

Add a release checklist to developer docs

Unable to parse token: <!DOCTYPE with HDF5 files

I'm having troubles when trying to open a hdf5 file:

from pydap.client import open_url

In [2]: url_name='http://gpm1.gesdisc.eosdis.nasa.gov/opendap/GPM_L3/GPM_3IMERGH
   ...: HE.03/2016/105/3B-HHR-E.MS.MRG.3IMERG.20160414-S000000-E002959.0000.V03E
   ...: .HDF5'

In [3]: dataset = open_url(url_name)
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-3-854224b09f3a> in <module>()
----> 1 dataset = open_url(url_name)

/Users/raulvalenzuela/miniconda2/envs/grib_api/lib/python2.7/site-packages/pydap/client.pyc in open_url(url, application, session, output_grid)
     62     never retrieve coordinate axes.
     63     """
---> 64     dataset = DAPHandler(url, application, session, output_grid).dataset
     65 
     66     # attach server-side functions

/Users/raulvalenzuela/miniconda2/envs/grib_api/lib/python2.7/site-packages/pydap/handlers/dap.pyc in __init__(self, url, application, session, output_grid)
     62 
     63         # build the dataset from the DDS and add attributes from the DAS
---> 64         self.dataset = build_dataset(dds)
     65         add_attributes(self.dataset, parse_das(das))
     66 

/Users/raulvalenzuela/miniconda2/envs/grib_api/lib/python2.7/site-packages/pydap/parsers/dds.pyc in build_dataset(dds)
    159 def build_dataset(dds):
    160     """Return a dataset object from a DDS representation."""
--> 161     return DDSParser(dds).parse()
    162 
    163 

/Users/raulvalenzuela/miniconda2/envs/grib_api/lib/python2.7/site-packages/pydap/parsers/dds.pyc in parse(self)
     47         dataset = DatasetType('nameless')
     48 
---> 49         self.consume('dataset')
     50         self.consume('{')
     51         while not self.peek('}'):

/Users/raulvalenzuela/miniconda2/envs/grib_api/lib/python2.7/site-packages/pydap/parsers/dds.pyc in consume(self, regexp)
     39     def consume(self, regexp):
     40         """Consume and return a token."""
---> 41         token = super(DDSParser, self).consume(regexp)
     42         self.buffer = self.buffer.lstrip()
     43         return token

/Users/raulvalenzuela/miniconda2/envs/grib_api/lib/python2.7/site-packages/pydap/parsers/__init__.pyc in consume(self, regexp)
    180             self.buffer = self.buffer[len(token):]
    181         else:
--> 182             raise Exception("Unable to parse token: %s" % self.buffer[:10])
    183         return token

Exception: Unable to parse token: <!DOCTYPE

I'm not sure how to fix it. Besides the pydap module I installed pydap.handlers.hdf5

new release

HI do you think you could have a new release? Then perhaps the latest fixes could be built on conda-forge?

Merge handlers and responses into pydap proper

I see that @laliberte has already started working on this, at least for the CSV and NetCDF handlers.

I think it could make a lot of sense to bundle these projects into the main pydap repo and pypi distribution. This would ensure that the handlers distributed with pydap remain up to date.

Doing this sort of thing with optional dependencies is a pretty standard practice for Python project packaging. You can specify requirements for subpackages by using extras_require in setup.py (e.g., as used in dask).

gsw is deprecated for python < 3.5

Should any use of gsw simply be removed from pydap?

Python 3 support

Python 3 support using the six module is in progress. Finish the port for release 3.2.1.

Unexpected 'flattening' of the array when slicing on BaseProxy

Hello,

I notice the following (unexpected) behavior when using pydap to accessing n-dimensional variable:

In [1]: import pydap.client

In [2]: ds = pydap.client.open_url('http://test.opendap.org/opendap/hyrax/data/nc/123.nc')

In [3]: ds
Out[3]: <DatasetType with children 'bears', 'order', 'shot', 'aloan', 'cross', 'i', 'j', 'l'>

In [4]: ds.bears
Out[4]: <GridType with array 'bears' and maps 'i', 'j'>

In [5]: ds.bears.array
Out[5]: <BaseType with data BaseProxy('http://test.opendap.org/opendap/hyrax/data/nc/123.nc', 'bears.bears', dtype('S128'), (2, 3), (slice(None, None, None), slice(None, None, None)))>

In [6]: ds.bears.array.shape
Out[6]: (2, 3)

## slicing the array flattens the output array
In [7]: ds.bears.array[:]
Out[7]:
array([u'ind', u'ist', u'ing', u'uis', u'hab', u'le'],
      dtype='<U3')

In [8]: ds.bears.array[:].shape
Out[8]: (6,)

## And this won't work?
In [9]: np.asarray(ds.bears.array)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-8-5da01e73285e> in <module>()
----> 1 np.asarray(ds.bears.array)

/Users/daniel.lee/anaconda/envs/pydap/lib/python2.7/site-packages/numpy/core/numeric.pyc in asarray(a, dtype, order)
    480
    481     """
--> 482     return array(a, dtype, copy=False, order=order)
    483
    484 def asanyarray(a, dtype=None, order=None):

ValueError: cannot copy sequence with size 2 to array axis with dimension 6

From my best attempt to read over the pydap code, I find this line suspicious:

return np.array([ text_type(x.decode('ascii')) for x in out ], 'S')

Namely, I think it's missing a .reshape(shape) call at the end.

return np.array([ text_type(x.decode('ascii')) for x in out ], 'S').reshape(shape)

Is what I'm suggesting here reasonable?

Discovered while investigating pydata/xarray#1174

can't access 3d data from MERRA2 openDAP server

Hi All,

I have tried to work with MERRA2 openDap team with this problem but they say the issue is not on their end. Basically I can access 2d data but not 3d data. Please see the code below.

from pydap.client import open_url
from pydap.cas.urs import setup_session
session = setup_session(earthLoginUser, earthLoginPass)

fullUrl4 = "https://goldsmr4.gesdisc.eosdis.nasa.gov/opendap/MERRA2/M2I1NXASM.5.12.4/2015/08/MERRA2_400.inst1_2d_asm_Nx.20150802.nc4"
d = open_url(fullUrl4, session=session)
var = d['lat']
var[0:10]

I get the data
array([-90. , -89.5, -89. , -88.5, -88. , -87.5, -87. , -86.5, -86. , -85.5])

however if I try to access the 3d data it fails.

fullUrl5 = "https://goldsmr5.gesdisc.eosdis.nasa.gov/opendap/hyrax/MERRA2/M2I3NVASM.5.12.4/2015/08/MERRA2_400.inst3_3d_asm_Nv.20150802.nc4"
d = open_url(fullUrl5, session=session)
var = d['lat']
var[0:10]

I get the following error:

Traceback (most recent call last):

File "", line 1, in
var[0:10]

File "/Applications/anaconda/lib/python2.7/site-packages/pydap/model.py", line 258, in getitem
return self.data[index]

File "/Applications/anaconda/lib/python2.7/site-packages/pydap/handlers/dap.py", line 135, in getitem
dds, data = r.body.split(b'\nData:\n', 1)

ValueError: need more than 1 value to unpack

Any help would be greatly appreciated!

URS authentication tests are currently not working

The current implementation of the URS authentication scheme fails in the provided tests. It appears that redirects are not followed properly.

In PR #57, a fix is proposed.

Release Pydap 3.2.0

Hi @jameshiebert are you open to releasing Pydap 3.2.0? I see no blockers and it would be ideal to make a Python 2.7 + 3.X-compatible artifact available.

Test Erddap responses

Some Erddap servers return invalid DDS responses when a deep object is returned. In that case, we should parse the DDS from the DODS response and extract the data using it.

Overhaul namespace packages (e.g. `pydap.tests`)

Is there a particular reason to declare pydap.tests a namespace package? This prevents running the tests as installed from the package (via python -m pytest --pyargs pydap.tests), which is a desirable feature from a CI perspective for Linux distributions.

model.BaseType required name argument

I just upgraded from Pydap 3.1.1 to Pydap 3.2 from PyPI and my unit tests failed with:

 File "/home/travis/build/MBARIMike/stoqs/stoqs/loaders/DAPloaders.py", line 1362, in Dorado_Loader
    chl = pydap.model.BaseType()
TypeError: __init__() takes at least 2 arguments (1 given)

Version 3.1.1 provides a default value for name:

    def __init__(self, name='nameless', data=None, shape=None,
            dimensions=None, type=Int32, attributes=None):
        DapType.__init__(self, name, attributes)

Is there any harm in providing this same default value version 3.2 (this line) so that we maintain backward compatibility?

Cannot remove scipy from testing yet

gsw has not been updated on PYPI in a long time This means that we have to keep scipy on travis for the time being.

Use of deprecated gunicorn command

On running the pydap application, the following message is logged:

!!!
!!! WARNING: This command is deprecated.
!!! 
!!!     You should now use the `--paste` option. Ex.:
!!! 
!!!         gunicorn --paste development.ini
!!!     
!!!

ValueError: dictionary update sequence element #0 has length 1; 2 is required

Hi,

I'm having problems with Pydap 3.2.0 installed on a 64 bit Linux system (however, have also seen the problem on Windows 10). The URL I try to open works fine under Pydap 3.1.1, but under 3.2.0 I get an error message. I can reproduce the error as shown below:

>>> from pydap.client import open_url
>>> dataset = open_url('http://thredds.met.no/thredds/dodsC/fou-hi/norkyst800m-1h/NorKyst-800m_ZDEPTHS_his.fc.2017030700.nc')          Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/malv/.local/lib/python3.5/site-packages/pydap/client.py", line 59, in open_url
    dataset = DAPHandler(url, application, session).dataset
  File "/home/malv/.local/lib/python3.5/site-packages/pydap/handlers/dap.py", line 61, in __init__
    add_attributes(self.dataset, parse_das(das))
  File "/home/malv/.local/lib/python3.5/site-packages/pydap/parsers/das.py", line 127, in add_attributes
    var.attributes.update(nested.pop(k))
ValueError: dictionary update sequence element #0 has length 1; 2 is required
>>>

Host Pydap's documentation on Read the Docs

As discussed in #14, we need to transition Pydap's documentation hosting to something that can be maintained by the community. Read the Docs was favored by most, and I support that as well, but there have been some challenges building the documentation.

@shoyer To fix the import error, I think that you just have to enable virtualenv in the "Advanced settings" tab. Alternatively (or in addition to), you can add me as a collaborator, and I can work on fixing it.

Client.rst refers to pydap.util

Looks like pydap.util is no longer with us. The documentation for using proxies refers to pydap.util, and could use an update. Text from client.rst below.

Configuring a proxy


It's possible to configure Pydap to access the network through a proxy server. Here's an example for an HTTP proxy running on ``localhost`` listening on port 8000:

.. code-block:: python

    >>> import httplib2
    >>> from pydap.util import socks

Documentation does not provide information how to display raw data

When I copy link (sample_query) contained within my gist:
https://gist.github.com/kedziorm/619141108839f154a8ded009d9cb30e8
and paste it to the web-browser, I see raw data.

However, I have no idea how to get exactly the same data in ASCII format using pydap.

List of example DAP data sources

It would be great to start a list of known data sources that can be used by the pydap client.

Incomplete list of install dependencies

pydap/wsgi/cache.py requires h5py, requests and webob, which are not listed as install requirements. I am surprised travis has not picked that up, assuming it is operational for pydap.

Title level inconsistencies in docs

As reported by Sphinx:

/tmp/autopkgtest.5BNvOg/build.VX1/python-pydap-3.2.2+ds1/docs/client.rst:306: SEVERE: Title level inconsistent:

Basic & Digest
~~~~~~~~~~~~~~
/tmp/autopkgtest.5BNvOg/build.VX1/python-pydap-3.2.2+ds1/docs/client.rst:316: SEVERE: Title level inconsistent:

CAS
~~~
/tmp/autopkgtest.5BNvOg/build.VX1/python-pydap-3.2.2+ds1/docs/client.rst:370: SEVERE: Title level inconsistent:

Calling server-side functions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/tmp/autopkgtest.5BNvOg/build.VX1/python-pydap-3.2.2+ds1/docs/client.rst:389: SEVERE: Title level inconsistent:

Opening a specific URL
~~~~~~~~~~~~~~~~~~~~~~
/tmp/autopkgtest.5BNvOg/build.VX1/python-pydap-3.2.2+ds1/docs/client.rst:410: SEVERE: Title level inconsistent:

Accessing raw data
~~~~~~~~~~~~~~~~~~
/tmp/autopkgtest.5BNvOg/build.VX1/python-pydap-3.2.2+ds1/docs/client.rst:432: SEVERE: Title level inconsistent:

Using a cache
~~~~~~~~~~~~~
/tmp/autopkgtest.5BNvOg/build.VX1/python-pydap-3.2.2+ds1/docs/client.rst:442: SEVERE: Title level inconsistent:

Timeout
~~~~~~~
/tmp/autopkgtest.5BNvOg/build.VX1/python-pydap-3.2.2+ds1/docs/client.rst:453: SEVERE: Title level inconsistent:

Configuring a proxy
~~~~~~~~~~~~~~~~~~~

How can I read array of Structures?

It seems that PyDAP doesn't provide a way to access data from array of structures.
Dataset { Structure { Float64 Time; Float32 Latitude; Float32 Longitude; } Point[13356]; }

Here's a real test case:

https://eosdap.hdfgroup.org:8989/opendap/hyrax/data/NASAFILES/hdf5/AMSR_U2_L2_Land_B01_201207232336_D.he5.html

Shape for structure returns an error. I tried open_dods() but it did not work, either. Is there a working solution for accessing array of structure data?

Move `_original_keys` to `pydap.model.SequenceType`.

Right now IterData tracks the original key order by setting an attribute called _original_keys in the sequence. This behavior should be moved to the original class, to keep the code more simple.

Missing manpages for command-line tools

It is a recommended practice in the Linux community to provide corresponding manpages to publicly installed command-line executables. Pydap installs the dods and pydap commands, but those lack respective manpages currently.

Since sphinx is already setup to build the package documentation, it would be quite straightforward to extend to also generate the manpages from ReST files.

Switch tests to pytest?

Should we tweak the tests to make them compatible with pytest and use py.test instead of nosetests on Travis?

Deprecation warnings regarding future array conversion behaviour

After checking out pydap on a fresh virtualenv, running the testsuite produces a bunch of VisibleDeprecationWarning:

 VisibleDeprecationWarning: converting an array with ndim > 0 to an index will result in an error in the future
  out.append(data[:n])
<VENV>/src/pydap/src/pydap/handlers/dap.py:154: VisibleDeprecationWarning: converting an array with ndim > 0 to an index will result in an error in the future
  data = data[n + (-n % 4):]
...................<VENV>/src/pydap/src/pydap/handlers/dap.py:153: VisibleDeprecationWarning: converting an array with ndim > 0 to an index will result in an error in the future
  out.append(data[:n])
<VENV>/src/pydap/src/pydap/handlers/dap.py:154: VisibleDeprecationWarning: converting an array with ndim > 0 to an index will result in an error in the future
  data = data[n + (-n % 4):]

CEDA authentification fails

CEDA openid authentication fails. This has been raised on the pydap mailing list.

Missing 'proxy' module

While attempting to run HDFCompass with the last PyPI release (3.2.0) of pydap, I noticed that the proxy module is gone together with ProxyArray class. Can you confirm it? If yes, is BaseType class the 'natural' substitute of ProxyArray?

The ProxyArray was imported here: https://github.com/HDFGroup/hdf-compass/blob/develop/hdf_compass/opendap_model/model.py#L19

Launch a local pydap server from Python

For testing pydap read support in xarray, it would be nice to able to launch a pydap server locally as part of the test suite, avoiding the need to contact an external test server. For example:

expected_dataset = ...
with local_pydap_test_server() as port:
    actual_dataset = xarray.open_dataset('http://localhost:%s' % port, engine='pydap')
    assert actual_dataset.equals(expected_datset)

Is there a straightforward way to do this with the current version of pydap?

NASA openDAP

Anything on the LP DAAC pool and openDap servers
http://opendap.cr.usgs.gov/opendap/hyrax/
now requires an OAuth 2 login redirected through https://urs.earthdata.nasa.gov. I'm still quite new to pydap but I find it so useful and I hope there is a solution through pydap.

Any help would be fantastic!

"Running a server" documentation outdated

Thanks for the hard work on this project!

This problem may be due to my lack of experience with Python Paste (and the fact the documentation on http://pythonpaste.org/ has been down for two months now), but we are trying to setup a pydap docker instance and have trouble getting the instructions to work for us.

(env) root@cf53f30390c0:/var/www/pydap# paster create -t pydap server
Traceback (most recent call last):
  File "/var/www/pydap/env/bin/paster", line 11, in <module>
    sys.exit(run())
  File "/var/www/pydap/env/lib/python2.7/site-packages/paste/script/command.py", line 102, in run
    invoke(command, command_name, options, args[1:])
  File "/var/www/pydap/env/lib/python2.7/site-packages/paste/script/command.py", line 141, in invoke
    exit_code = runner.run(args)
  File "/var/www/pydap/env/lib/python2.7/site-packages/paste/script/command.py", line 236, in run
    result = self.command()
  File "/var/www/pydap/env/lib/python2.7/site-packages/paste/script/create_distro.py", line 72, in command
    self.extend_templates(templates, tmpl_name)
  File "/var/www/pydap/env/lib/python2.7/site-packages/paste/script/create_distro.py", line 264, in extend_templates
    'Template by name %r not found' % tmpl_name)
LookupError: Template by name 'pydap' not found

Is there a way to get some guidance on what is needed to get paste to work?

Even better might be documentation or a code update to get a uwsgi server setup without having to pass via Python Paste? There is no conventional "application" method in the wsgi/app.py and the "main" method relies on gunicorn.app.pasterapp.paste_server as well.

Paste seems to add significant complexity for a limited benefit

xarray handler

xarray is basically an in-memory representation of netCDF-like data structures. It would be amazing to have a pydap handler for xarray. That way we could launch an opendap server from within python (possibly related to #33) and feed it an xarray dataset. Such services could be chained together to form streaming data pipelines without ever touching a hard disk.

pydap.util module is not installed by default

It looks like there's no (??) way to install util module using Linux packages nor pip install tool.
I described this issue at Stack Overflow: http://stackoverflow.com/questions/41520007/how-can-i-install-python-pydap-utils-module-pydap-util

Missing extra requires on PasteDeploy

Even with the all list of dependencies installed, the pydap command cannot be run without doing a pip install PasteDeploy. Perhaps it would be worth adding it to the list of extra requires for the cli.

Switch pydap.handlers.netcdf to use scipy instead of pupynere

Scipy's netcdf3 file handling is a maintained fork of pupynere, so (for example) it also runs on Python 3.

Any concerns?

New maintainer for pydap?

@robertodealmeida -- Many of us still find pydap to be quite useful, but it's clear that you no longer have the time or inclination to support it. That's OK -- we all need to move on eventually -- but would you be willing to add someone else as an administrator who can merge pull requests and issue releases on pypi? In particular, it would be extremely valuable if someone is able to merge in Python 3 support.

If you're not willing to add another maintainer, then I will probably fork the project. But I would rather avoid such a split.

@jameshiebert @SiggyF @jgallagher59701 @jblarsen @jfburkhart -- All of you have outstanding pull requests. Are any of you interested in being a maintainer for this project? If need be, I will step up, but I have way too many open source obligations already...