HTTPS issues with Python 2. about nbt

[macfreek]

The patches did not work (I expect because I was modifying ssl parameters, but those changes were local to my script, not to urllib2).

Anyhow, I got requests to work. Indeed great package, much cleaner API.

The bad news, the underlying library is still ssl, which refuses to talk to TLS (any version) for Python 2.6. According to the docs, use PROTOCOL_SSLv23 | OP_NO_SSLv2 | OP_NO_SSLv3, thus 'SSL protocol v2 or v3, but not v2 nor v3'. Right. Apparently, "PROTOCOL_SSLv23" does not mean "protocol v2 or v3", but "the highest protocol that both client and server speak". At least, that's what it means in Python3, and now also in Python 2.7.9. But in Python 2.6-2.7.8, it still means "protocol v2 or v3, but not TLSv1.0, even though TLSv1.0 is supported". So, I can select ssl.PROTOCOL_TLSv1, right? Yes, I can, but only if I use ssl directly. Of course, that only means I have to rewrite both httplib and urllib2.

It starts to daunt on me that there authors of libraries like GnuTLS and NaCl. I hated OpenSSL for it GNU-incompatible licensing (I don't really like GNU for the same reason by the way, but at least they reasoning is correct). I now also start to hate it because it's terrible, terrible interface, and wished Python would have used a sane library.

Maybe I should just drop Python 2.6-2.7.8 and do something fun.

Or just get some sleep.

[macfreek]

If you read this, please reply with the version of Python you use (e.g. my default is 3.3.2 nowadays, though I still use 2.7.9 in some rare cases.).

[macfreek]

OK, the issue was much easier: The ssl module in Python 2.6-2.7.8 does not support HTTPS redirects.

Downloading https://cloud.github.com/downloads/twoolie/NBT/Sample_World.tar.gz instead of https://github.com/downloads/twoolie/NBT/Sample_World.tar.gz works perfectly fine.

Apparently, this is also valid for requests.

I'll think a bit what to do -- I'll probably use urllib2 after all, even though I like requests better.

Edit: I was not thinking straight, I was using 2.7.9 instead of 2.7.8. Still doesn't work for 2.7.8 and earlier.

[macfreek]

I wrote a wrapper to call wget to download the file.

I picked wget over curl, because on my laptop, wget was compiled with GnuTLS instead of the dreaded OpenSSL library.

Travis job 89.1 tells the tale:

INFO     Downloading https://github.com/downloads/twoolie/NBT/Sample_World.tar.gz
ERROR    Download https://github.com/downloads/twoolie/NBT/Sample_World.tar.gz failed: <class 'urllib2.URLError'> <urlopen error [Errno 1] _ssl.c:493: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure>
ERROR    Download https://github.com/downloads/twoolie/NBT/Sample_World.tar.gz failed: _ssl.c:493: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
INFO     Downloading https://github.com/downloads/twoolie/NBT/Sample_World.tar.gz (with wget)
--2015-01-03 15:33:18--  https://github.com/downloads/twoolie/NBT/Sample_World.tar.gz
Resolving github.com (github.com)... 192.30.252.129
Connecting to github.com (github.com)|192.30.252.129|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://cloud.github.com/downloads/twoolie/NBT/Sample_World.tar.gz [following]
--2015-01-03 15:33:19--  https://cloud.github.com/downloads/twoolie/NBT/Sample_World.tar.gz
Resolving cloud.github.com (cloud.github.com)... 54.230.68.7, 54.230.71.206, 54.230.68.42, ...
Connecting to cloud.github.com (cloud.github.com)|54.230.68.7|:443... connected.
OpenSSL: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
Unable to establish SSL connection.
('exitcode =', 4)

So urllib2 fails with a sslv3 alert handshake failure. Then, wget -O Sample_World.tar.gz https://cloud.github.com/downloads/twoolie/NBT/Sample_World.tar.gz is used. This also fails with a sslv3 alert handshake failure.

I was surprised.

Just to make it clear: wget is not using any Python at all. 2.6, or anything else. The problem lies in some OpenSSL libraries, and both Python's ssl modules, as well as the wget on the Travis VM (wget 1.3.4 with OpenSSL 1.0.1) don't behave well.

I suspect the following is the root of the problem:

According to the wget 1.12 man page (at the text on --secure-protocol):

[T]he SSL library is given the liberty of choosing the appropriate protocol automatically, which is achieved by sending an SSLv2 greeting and announcing support for SSLv3 and TLSv1.

According to the wget 1.16 man page:

[T]he SSL library is given the liberty of choosing the appropriate protocol automatically, which is achieved by sending a TLSv1 greeting.

Since wget 1.12 fails and wget 1.16 succeeds (although on different machines), this may be the root cause. Or it may not. I no longer care.

I switched from wget to curl for backup-download mechanism and that works on Travis.

Case closed.