cedadev / ndg_httpsclient

HTTPS client implementation for httplib and urllib2 based on PyOpenSSL

A HTTPS client implementation for

• httplib (Python 2), http.client (Python 3) and
• urllib2 (Python 2) and urllib (Python 3)

... based on PyOpenSSL. PyOpenSSL provides a more fully featured SSL implementation over the default provided with Python and importantly enables full verification of the SSL peer using pyasn1.

Releases

0.5.1

• Clean up handling for description file - pull in content from this file into setup()
• Allows the nightly build to fail
• Add Trove version classifiers to make it explicit what is supported
• Add python_requires to help pip
• Drop support for EOL Python 2.6 and 3.3

Thanks to @hugovk for contributions

0.5.0

• Fix to Subject Alternative Name handling to allow for certificates with more than 64 names (max now 1024). Thanks to Matt Pegler
• Fix to subjectAltName string to use byte type for correct matching
• Updated SSL Context objects to default to TLS 1.2

0.4.4

• Updated test certificates

0.4.3

• Fix to ndg namespace package warning issue (#3).
  __init__.py file now included in ndg directory so that there are no longer warnings with imports when using Python 2.x. Thanks to Max Mauntner for fix.
• Minor fix for installation: set minimum release for pyasn1 to avoid conflicts with Ubuntu install - see #5 and #10. pyasn1 also becomes mandatory rather than optional package for install. - It required by cryptography anyway which is a dependency for pyOpenSSL from version 0.14.

0.4.2

• Fix to bug in ndg.httpsclient.utils.open_url - duplicate open call.
  Nb. This bug and the fix DO NOT affect the httpliband urllib2 interfaces that this package provides.

0.4.1

• Added explicit ref to Python 3 in classifier text for Python 3 checking tools.
• Moved LICENSE file into package

0.4.0

• Made dual compatible with Python 2 / 3.

0.3.3

• Fix to add in AnotherName for subjectAltNames field - added for support for CACert issued certs (thanks to Gu1).
• Fix to HTTP Basic Auth option for ndg.httpsclient.utils.main
• Fix to ServerSSLCertVerification so that it can pass a function-based callback instead of using __call__. In newer versions of OpenSSL (>= 0.14) the latter failed because of a request for __name__ attribute.

0.3.2

• Fix to SubjectAltNames support check - should only be enabled if pyasn1 is installed.
• Fix to open_url: HTTP Request object was being created inside if headers is None block - now corrected to create regardless.
• Added http basic auth support to script. (Thanks to Willem van Engen)

0.3.1

• extended utils functions to support keyword for passing additional urllib2 handlers.

0.3.0

• Added ndg.httpsclient.utils.fetch_stream_from_url function and added parameter for data to post in open_url and fetch_* methods.
• fix to ndg.httpsclient.utils module _should_use_proxy and open_url functions

0.2.0

• added support for SSL verification with subjectAltNames using pyasn1
• fixed minor bug - SSL cert DN prefix matching

0.1.0

Initial release

Prerequisites

This has been developed and tested for Python 2.7 with pyOpenSSL 0.13 and 0.14. Version 0.4.0 tested with pyOpenSSL 0.15.1 and Python 2.7 and 3.4. pyasn1 is required for correct SSL verification with subjectAltNames.

Installation

Installation can be performed using easy_install or pip.

Running ndg_httpclient

A simple script for fetching data using HTTP or HTTPS GET from a specified URL.

Parameter:

url The URL of the resource to be fetched

Options:

-h, --help Show help message and exit.

-c FILE, --certificate=FILE Certificate file - defaults to $HOME/credentials.pem

-k FILE, --private-key=FILE Private key file - defaults to the certificate file

-t DIR, --ca-certificate-dir=DIR Trusted CA certificate file directory.

-d, --debug Print debug information - this may be useful in solving problems with HTTP or HTTPS access to a server.

-p FILE, --post-data-file=FILE POST data file

-f FILE, --fetch=FILE Output file

-n, --no-verify-peer Skip verification of peer certificate.

ndg_httpsclient's Issues

pyasn1 is a dependency that doesn't get upgraded

On an ubuntu 12.04 precise machine, upgrading ndg-httpsclient when python-asn1 is installed will break with

  File "/usr/local/lib/python2.7/dist-packages/ndg/httpsclient/subj_alt_name.py", line 105, in AnotherName
    namedtype.NamedType('value', univ.Any().subtype(
AttributeError: 'module' object has no attribute 'Any'

Upgrading pyasn1 before upgrading ndg-httpsclient results in a working system. I think pyasn1 should be a dep here with a minimum version number (just not sure what it is)

maximum number of alt names

There is a hardcoded limit of 64 alt names that is causing some problems for us. Specifically, https://files.pythonhosted.org points to Fastly, which uses a shared cert with more than 64 alt names. This is causing pip to be unable to install anything, which is an interesting catch-22 if the limit for this package does get increased. I'm curious why this limit exists because I don't believe there is a hard limit when creating certificates.

https://github.com/cedadev/ndg_httpsclient/blob/master/ndg/httpsclient/subj_alt_name.py#L28
https://github.com/cedadev/ndg_httpsclient/blob/master/ndg/httpsclient/subj_alt_name.py#L150

LICENSE missing from pypi package

The tarball from pypi misses the LICENSE file. Currently type of license can only be determined by visiting the GitHub project repo.

BSD License is specified yet there should at least be mention of the type of BSD license (2-clause/simplified, 3-clause/New/Revised, 4-clause/Original/Old).

test suite fails with openssl 1.1.1e

The test suite fails after upgrading openssl to 1.1.1e on Fedora rawhide ("Fedora 33"):

======================================================================
ERROR: test02_fetch_from_url (__main__.TestUtilsModule)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test_utils.py", line 32, in test02_fetch_from_url
    res = fetch_from_url(Constants.TEST_URI, config)
  File "/builddir/build/BUILD/ndg_httpsclient-0.5.1/ndg/httpsclient/utils.py", line 100, in fetch_from_url
    raise URLFetchError(return_message)
ndg.httpsclient.utils.URLFetchError: Error: [('SSL routines', 'ssl3_read_n', 'unexpected eof while reading')]
======================================================================
FAIL: test03_open_url (__main__.TestUtilsModule)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test_utils.py", line 38, in test03_open_url
    self.assertEqual(res[0], 200,
AssertionError: 0 != 200 : open_url for 'https://localhost:4443' failed
----------------------------------------------------------------------

This might be related/similar to another bug report for urllib3: unexected eof with OpenSSL 1.1.1e #1825.

Downstream Fedora issue will complete build logs: https://bugzilla.redhat.com/show_bug.cgi?id=1815903

version 0.3.3 does not work with python3

This also causes pip to fail!

 ndg_httpsclient-0.3.3  python3 -c "from ndg.httpsclient.ssl_peer_verification import SUBJ_ALT_NAME_SUPPORT"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.4/dist-packages/ndg/httpsclient/ssl_peer_verification.py", line 17
    except ImportError, e:
                      ^
SyntaxError: invalid syntax

my pip is now completely broken:

 ndg_httpsclient-0.3.3  pip3 --version
Traceback (most recent call last):
  File "/usr/bin/pip3", line 9, in <module>
    load_entry_point('pip==1.5.4', 'console_scripts', 'pip3')()
  File "/usr/lib/python3/dist-packages/pkg_resources.py", line 351, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3/dist-packages/pkg_resources.py", line 2363, in load_entry_point
    return ep.load()
  File "/usr/lib/python3/dist-packages/pkg_resources.py", line 2088, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
  File "/usr/lib/python3/dist-packages/pip/__init__.py", line 11, in <module>
    from pip.vcs import git, mercurial, subversion, bazaar  # noqa
  File "/usr/lib/python3/dist-packages/pip/vcs/mercurial.py", line 9, in <module>
    from pip.download import path_to_url
  File "/usr/lib/python3/dist-packages/pip/download.py", line 22, in <module>
    import requests, six
  File "/usr/lib/python3/dist-packages/requests/__init__.py", line 53, in <module>
    from urllib3.contrib import pyopenssl
  File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 23, in <module>
    from ndg.httpsclient.ssl_peer_verification import SUBJ_ALT_NAME_SUPPORT
  File "/usr/local/lib/python3.4/dist-packages/ndg_httpsclient-0.3.3-py3.4.egg/ndg/httpsclient/ssl_peer_verification.py", line 17
    except ImportError, e:
                      ^
SyntaxError: invalid syntax

Setup.py not copying __init__.py into package.

Hi there,

This is an old issue but shows when you run -Wall. Perhaps you can fix it - It looks like setup.py is not copying __init__.py into the package when run pip install ndg_httpsclient

/usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pkgutil.py:186: ImportWarning: Not importing directory '/Users/sklass/.virtualenvs/axis/lib/python2.7/site-packages/ndg': missing __init__.py file, filename, etc = imp.find_module(subname, path)

v0.4.1 was not pushed to PyPI

See: ubershmekel/python3wos#22

Python 3 Wall of Superpowers http://python3wos.appspot.com/

Wheel support

http://pythonwheels.com/

Right now, only the tar.gz file is being distributed on PyPI for ndg-httpsclient, which isn't that much of an issue, but it does add some time to installing the package.

This package should be compatible with the Wheel format, considering it doesn't appear to have any C dependencies and it is compatible with both Python 2 and 3. As a result, you should only need to generate a universal wheel and then everyone (on all systems) will get the ability to install ndg-httpsclient with just the wheel, without having to do any extra work.

OpenSSL.SSL.WantReadError

Hi,

I'm trying to use pyopenssl with ndg and the following error occurs in con.getresponse():

Error
Raise WantReadError ()
OpenSSL.SSL.WantReadError

Code:

from OpenSSL import crypto , SSL
from ndg.httpsclient.https import HTTPSConnection

pkcs12 = crypto.load_pkcs12(open('certi.pfx', 'rb').read(), 'pass')
certi = crypto.dump_certificate( crypto.FILETYPE_PEM , pkcs12.get_certificate() )
key = crypto.dump_privatekey( crypto.FILETYPE_PEM , pkcs12.get_privatekey() )

cert_openssl = crypto.load_certificate(crypto.FILETYPE_PEM, certi)
key_openssl = crypto.load_privatekey(crypto.FILETYPE_PEM, key)
ctx = SSL.Context(SSL.SSLv23_METHOD)
ctx.use_certificate(cert_openssl)
ctx.use_privatekey(key_openssl)

con = HTTPSConnection('server', port=443, ssl_context=ctx)
con.connect()
con.request( b'POST', b'/' + cURL.encode('utf-8'), xml_soap.encode('utf-8'), header)

resp = con.getresponse()

Test certificates expired (again)

$ ./scripts/openssl_https_server.sh
Using default temp DH parameters
ACCEPT
139899582240576:error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca:ssl/record/rec_layer_s3.c:1543:SSL alert number 48
139899582240576:error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired:ssl/record/rec_layer_s3.c:1543:SSL alert number 45
139899582240576:error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired:ssl/record/rec_layer_s3.c:1543:SSL alert number 45
139899582240576:error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired:ssl/record/rec_layer_s3.c:1543:SSL alert number 45
139899582240576:error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca:ssl/record/rec_layer_s3.c:1543:SSL alert number 48
139899582240576:error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired:ssl/record/rec_layer_s3.c:1543:SSL alert number 45
 $ openssl x509 -in localhost.crt -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 0 (0x0)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = NDG Test CA, O = NDG, OU = Security
        Validity
            Not Before: Dec 16 01:07:32 2016 GMT
            Not After : Dec 16 01:07:32 2019 GMT
...

test certs have expired (20 Jan 2018). Could those be refreshed?