Launch a local pydap server from Python about pydap

[jameshiebert]

I think that most of our tests work one level down at the WSGI level and skip HTTP altogether. This is quite a bit faster and then avoids any problems with the network stack that you don't really want your tests to rely on..

We do this by passing a WSGI app object into WebOb.request.blank (IIRC). Would something like that work for you?

[shoyer]

We do this by passing a WSGI app object into WebOb.request.blank (IIRC). Would something like that work for you?

Sure, though I would prefer to be able to use a utility function from pydap rather than constructing such objects myself. WSGI and WebOb feel like implementation details that I would rather not leak into the xarray test suite.

[laliberte]

@shoyer, @jameshiebert
I have actually been working on this at the end of last week. It's in merging/handlers.csv.

To get this branch together, I wanted to clean-up the CSV handler so that I could have a real sever test. In that branch, the test server is in the file test_devel_server.py.

I've also been working on a netcdf4 handler merging/handlers.netcdf to test more complex use cases.

I was thinking that I could also code an xarray-based handler. It would then make it possible to create a suite of unittests for xarray-pydap integration. Would that be something you would be interested in @shoyer ?

[shoyer]

I was thinking that I could also code an xarray-based handler. It would then make it possible to create a suite of unittests for xarray-pydap integration. Would that be something you would be interested in @shoyer ?

Indeed, an xarray based handler seems like a natural addition. I can see xarray (especially with dask) being a nice way to setup a pydap server, e.g., as a more flexible substitute for pydap.handlers.nca. This isn't something I would use personally (I would be more likely to just launch a remote dask-distributed server), but you should build it if you would find it useful.

For a comprehensive integration test (like the tests we run in xarray/test/test_backends.py), I think we would indeed want to use lower level integration to avoid relying on HTTP stack -- several seconds to launch a server per test case and the inability to drop into a debugger could be painfully slow to debug. For what it's worth, I'm pretty sure that if you did set this up you would expose bugs in pydap (or at least the xarray integration) for some edge cases -- this is true for basically every new backend I've run the full backends suite against.

[jameshiebert]

WSGI and WebOb feel like implementation details that I would rather not leak into the xarray test suite.

Ah yes, that's a good point. I can see your motivation there.

I think we would indeed want to use lower level integration to avoid relying on HTTP stack -- several seconds to launch a server per test case and the inability to drop into a debugger could be painfully slow to debug.

Indeed. I believe that there are some tests in Pydap that do use a local server and, you're right, if anything goes wrong in the test from the client side, you're basically unable to debug it from the server side.

For what it's worth, I'm pretty sure that if you did set this up you would expose bugs in pydap (or at least the xarray integration) for some edge cases -- this is true for basically every new backend I've run the full backends suite against.

Hrm, can you explain that a bit more? What kind of bugs in Pydap have you been seeing? Do you mind posting issues about them?

[laliberte]

I can only guess but the way I understood this question was that @shoyer meant that if one was to implement an xarray handler and essentially tests xarray onto that pydap-xarray handler then one would fairly easily uncover edge cases bugs, in both the OPeNDAP integration within xarray and within the DAPHandler of pydap. It would be some work but it seems like it could greatly improve the stability of pydap...

[shoyer]

@jameshiebert Yes, I didn't mean that I know about bugs in pydap -- just that in my experience there are lots of weird edge cases (indexing operations, different dtypes, scalar or size 0 arrays, bytes vs unicode, fill values, Python 2 vs 3, ...) that are easy to miss. Maybe pydap is better tested than projects like netcdf4-python but certainly our test suite turned up plenty of issues there.

[jameshiebert]

Ahh, sure that makes sense. Yes, if Rob had not written such a high coverage test suite for Pydap, I would have had a *really* rough time doing the 2 to 3 conversion.

…

On Dec 13, 2016, at 4:14 PM, Stephan Hoyer ***@***.***> wrote: @jameshiebert Yes, I didn't mean that I know about bugs in pydap -- just that in my experience there are lots of weird edge cases (indexing operations, different dtypes, scalar or size 0 arrays, bytes vs unicode, fill values, Python 2 vs 3, ...) that are easy to miss. Maybe pydap is better tested than projects like netcdf4-python but certainly our test suite turned up plenty of issues there. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[laliberte]

@shoyer
I've revisited #34 and it turns out that one can drop any delay after the sever has been set-up, resulting in just a small overhead to testing through the simple_server. So while it should probably not be used for pydap tests, it could certainly be used for a subset of xarray-pydap tests.

[jameshiebert]

With #92 merged, I think that we can safely close this issue. Feel free to reopen if I missed anything.