pydap.util module is not installed by default about pydap

[ghisvail]

sudo pip install Pydap

...is a really, really, really bad idea. Do not mess up with your system Python installation. Use a virtual environment or use pip in "user mode", but never sudo pip install anything unless your really know what you are doing.

sudo apt-get install python-dap

...will install the old pydap v2.x which used the dep namespace. Again, probably not what you want.

However as I can see there's no util path:
/usr/local/lib/python2.7/dist-packages/pydap/util

Why should it be under /usr/local? You said yourself you used the system pip so if it installed something somewhere, it should be under /usr. Again bad idea, use a venv. It is as easy as:

sudo apt-get install virtualenv
virtualenv pydap
source pydap/bin/acitvate
pip install pydap

Then the pydap command will be available to you within the pydap venv. Run deactivate when you are done to leave the venv.

You might need to also have to run pip install PasteDeploy (see #53) or the pydap command will fail to run.

[kedziorm]

@ghisvail Thank you for your help, but I don't understand your comments.
I don't want to use multiple versions of Python packages, I would like to use the most typical, stable releases of Python packages in whole system, distributed with Ubuntu. Why should I create virtual env in such situation?

Regardless of that, as I can see I have Pydap in version 3.2 and the same version installs when I create virtual env.
Please let me prove it - when I type: pip show pydap
I get:

Name: Pydap
Version: 3.2.0
Summary: An implementation of the Data Access Protocol.
Home-page: http://pydap.org/
Author: James Hiebert
Author-email: [email protected]
License: MIT
Location: /usr/local/lib/python2.7/dist-packages
Requires: numpy, singledispatch, Webob, Jinja2, docopt, gunicorn, six, mechanicalsoup

My problem is that I'm receiving an error (because of lack of module) when trying to execute the following line of code:
from pydap.util.urs import install_basic_client

[ghisvail]

I don't want to use multiple versions of Python packages, I would like to use the most typical, stable releases of Python packages in whole system, distributed with Ubuntu.

Because running sudo pip install foo may bring more dependencies than foo, and then silently overwrite the system installed ones because you gave pip sudo powers. Your system Python packages are here to serve other packaged applications, not to be used for testing or development purposes.

Disclaimer: I am myself a Debian maintainer for some Python packages.

Why should I create virtual env in such situation?

To avoid inadvertent breakage of your system Python modules. You want to test something out that is not packaged, do it in a venv. It costs nothing, and can save you from potential misery.

Requires: numpy, singledispatch, Webob, Jinja2, docopt, gunicorn, six, mechanicalsoup

I believe you need additional dependencies to run the cli command. See the extras_require field in setup.py.

[kedziorm]

I believe you need additional dependencies to run the cli command. See the extras_require field in setup.py.

I guess you're referring to comment in setup.py. But which setup.py? Setup.py from this repository?

[ghisvail]

Yes.

[jameshiebert]

I answered your question on SE already, but the short answer is that you basically don't need pydap.util anymore (also it doesn't exist), because NASA/URS authentication is now (as of version 3.2) a supported feature and its usage is described in the documentation.

Hope that helps.

PS I agree with @ghisvail in that you really want to use venv and not sudo pip install if you can avoid it.

[kedziorm]

@jameshiebert After installation of additional packages ('numpy singledispatch Webob Jinja2 docopt gunicorn six mechanicalsoup') I'm able to import 'open_url' and 'setup_session', but it seems that I received and error when executing line with setup_session (probably, because I'm behind the proxy (ConnectionError: HTTPSConnectionPool(host='urs.earthdata.nasa.gov', port=443)).

[jameshiebert]

Hi @kedziorm, if you're behind an HTTP(S) proxy, I don't think that I (or Pydap) can help you with that. You'll have to figure that out on your own or with your network admin. The Pydap client necessarily assumes that it has a functional network connection to work with :)

[kedziorm]

Hi @jameshiebert,

okay, it's no problem. Fortunately I have other computer without proxy...but I have different issue.
When I execute (first lines just to read my credentials from ~/.netrc, rest are from your example:

import netrc

authData = netrc.netrc().hosts['urs.earthdata.nasa.gov']
myLogin = authData[0]
myPassword = authData[2]

from pydap.client import open_url
from pydap.cas.urs import setup_session
session = setup_session(myLogin, myPassword)
dataset = open_url('http://hydro1.sci.gsfc.nasa.gov/dods/_expr_%7BGLDAS_NOAH025SUBP_3H%7D%7Bave(rainf,time=00Z29Sep2016,time=00Z30Sep2016)%7D%7B17.00:25.25,48.75:54.50,1:1,00Z29Sep2016:00Z29Sep2016%7D.ascii?result', session=session)

I'm receiving:

Exception Traceback (most recent call last)
in ()
----> 1 dataset = open_url('http://hydro1.sci.gsfc.nasa.gov/dods/_expr_%7BGLDAS_NOAH025SUBP_3H%7D%7Bave(rainf,time=00Z29Sep2016,time=00Z30Sep2016)%7D%7B17.00:25.25,48.75:54.50,1:1,00Z29Sep2016:00Z29Sep2016%7D.ascii?result', session=session)

/usr/local/lib/python2.7/dist-packages/pydap/client.pyc in open_url(url, application, session)
57 def open_url(url, application=None, session=None):
58 """Open a remote URL, returning a dataset."""
---> 59 dataset = DAPHandler(url, application, session).dataset
60
61 # attach server-side functions

/usr/local/lib/python2.7/dist-packages/pydap/handlers/dap.pyc in init(self, url, application, session)
58
59 # build the dataset from the DDS and add attributes from the DAS
---> 60 self.dataset = build_dataset(dds)
61 add_attributes(self.dataset, parse_das(das))
62

/usr/local/lib/python2.7/dist-packages/pydap/parsers/dds.pyc in build_dataset(dds)
157 def build_dataset(dds):
158 """Return a dataset object from a DDS representation."""
--> 159 return DDSParser(dds).parse()
160
161

/usr/local/lib/python2.7/dist-packages/pydap/parsers/dds.pyc in parse(self)
45 dataset = DatasetType('nameless')
46
---> 47 self.consume('dataset')
48 self.consume('{')
49 while not self.peek('}'):

/usr/local/lib/python2.7/dist-packages/pydap/parsers/dds.pyc in consume(self, regexp)
37 def consume(self, regexp):
38 """Consume and return a token."""
---> 39 token = super(DDSParser, self).consume(regexp)
40 self.buffer = self.buffer.lstrip()
41 return token

/usr/local/lib/python2.7/dist-packages/pydap/parsers/init.pyc in consume(self, regexp)
180 self.buffer = self.buffer[len(token):]
181 else:
--> 182 raise Exception("Unable to parse token: %s" % self.buffer[:10])
183 return token

Exception: Unable to parse token: <!DOCTYPE

I have no problem with retrieving those data by copy and paste link to browser and provide exactly the same credentials. Could you suggest anything, please?

[jameshiebert]

From the "<!DOCTYPE" prefix, it looks like the server response is some document other than an OpenDAP DDS response. Any chance that you can drop into a pdb debugger there and print out self.buffer? That would tell us what response you're getting and might give us some clues...

[kedziorm]

@jameshiebert
Yes, you were correct. It was a HTML page that informs about 301

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://hydro1.gesdisc.eosdis.nasa.gov/dods/_expr_{GLDAS_NOAH025SUBP_3H}{ave(rainf,time=00Z29Sep2016,time=00Z30Sep2016)}{17.00:25.25,48.75:54.50,1:1,00Z29Sep2016:00Z29Sep2016}.ascii.dds?result">here</a>.</p>
</body></html>

As a result of changing query, now I'm receiving different error:
ValueError: need more than 1 value to unpack