softlayer / softlayer-object-storage-python

SoftLayer Object Storage Python Client

SoftLayer Object Storage Python Client

Python bindings for SoftLayer Object Storage

Installation

Install via pip:

pip install softlayer-object-storage

Or install from source:

git clone git://github.com/softlayer/softlayer-object-storage-python.git
cd softlayer-object-storage-python
python setup.py install

To build the documentation (requires sphinx):

python setup.py build_sphinx

Basic Usage

import object_storage

sl_storage = object_storage.get_client('YOUR_USERNAME', 'YOUR_API_KEY', datacenter='dal05')

sl_storage.containers()
# []

sl_storage['foo'].create()
# Container(foo)

sl_storage.containers()
# [Container(foo)]

sl_storage['foo'].properties
# {'count': 0, 'object_count': 0, 'name': 'foo', 'size': 0.0}

sl_storage['foo']['bar.txt'].create()
# StorageObject(foo, sample_object.txt)

sl_storage['foo']['bar.txt'].send('Plain-Text Content')
# True

sl_storage['foo']['bar.txt'].read()
# 'Plain-Text Content'

sl_storage['foo'].objects()
# [StorageObject(foo, bar.txt)]

sl_storage['foo']['bar.txt'].delete()
# True

sl_storage['foo'].delete()
# True

Search Usage

sl_storage.search('foo')
# {'count': 2, 'total': 2, 'results': [Container(foo), StorageObject(bar, foo)]}

sl_storage['foo'].search('bar.txt')
# {'count': 1, 'total': 1, 'results': [StorageObject(foo, bar.txt)]}

sl_storage.search('foo', type='container')
# {'count': 1, 'total': 1, 'results': [Container(foo)]}

sl_storage.search('foo*baz')
# {'count': 1, 'total': 1, 'results': [Container(foobarbaz)]}

Development

Here's how to get started:

• Fork repo on github

• Clone your new repo to your local machine:

git clone [email protected]:username/softlayer-object-storage-python.git

* Configure remote for upstream:

* ```
cd softlayer-object-storage-python
git remote add upstream git://github.com/softlayer/softlayer-object-storage-python.git

softlayer-object-storage-python's Issues

Softlayer Object Storage Containers: Unable to find the server

Link to StackOverflow Question here.
Problem:

I'm trying to debug some of my code that uses Softlayer Object Storage however I kept getting some errors from SL itself. Since I was getting the error from SL I went ahead and tried writing some code that reproduces the error. Which can be seen below followed by the stack trace I get.

Question:

Does anyone know why I'm getting the below error besides the possible security prevention from spamming sl so many times?

Source Code:

    #!/usr/local/bin/python2.7
    import argparse
    import object_storage
    
    def main():
        parser = argparse.ArgumentParser(description='Spam multiple sl storage containers.')
    
        parser.add_argument("--username", type=str, required=True, help="softlayer username")
        parser.add_argument("--apikey", type=str, required=True, help="softlayer api key")
        parser.add_argument("--datacenter", type=str, required=True, help="softlayer datacenter")
        parser.add_argument("--count", type=int, required=True, help="Amount of times to iterate")
    
        args = parser.parse_args()
        username = args.username
        api_key = args.apikey
        datacenter = args.datacenter
        count = args.count
    
        for i in range(0, count):
            print "Trying to create sl_storage.containers() #{0}".format(i)
            sl_storage = object_storage.get_client(username, api_key, datacenter=datacenter)
            containers = sl_storage.containers()
            del containers
            del sl_storage
    
    if __name__ == "__main__":
        main()

Stack Trace:

    Traceback (most recent call last):
      File "/root/sl_test.py", line 32, in <module>
        main()
      File "/root/sl_test.py", line 27, in main
        containers = sl_storage.containers()
      File "/usr/local/lib/python2.7/site-packages/object_storage/client.py", line 293, in containers
        formatter=_formatter)
      File "/usr/local/lib/python2.7/site-packages/object_storage/client.py", line 354, in make_request
        result = self.conn.make_request(method, url, *args, **kwargs)
      File "/usr/local/lib/python2.7/site-packages/object_storage/transport/httplib2conn.py", line 55, in make_request
        response = _make_request(headers)
      File "/usr/local/lib/python2.7/site-packages/object_storage/transport/httplib2conn.py", line 48, in _make_request
        body=data)
      File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1659, in request
        (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
      File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1399, in _request
        (response, content) = self._conn_request(conn, request_uri, method, body, headers)
      File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1325, in _conn_request
        raise ServerNotFoundError("Unable to find the server at %s" % conn.host)
    httplib2.ServerNotFoundError: Unable to find the server at dal05.objectstorage.softlayer.net

409 Client Error

Deleting a container often gives me a "409 Client Error". I don't understand what this error is being caused by and it seems very random. The files are still deleted, but the exception causes my code to bomb out and the error message "client error" gives me no insight into what is causing the error. It would be nice to fix it so that this error does not happen or so that the error message is more helpful.

Adding headers to uploads

What's the proper way to add headers to an upload? For example, when I'm creating a manifest it request a couple of headers like X-Object-Manifest (or something to that extent), and there doesn't seem to be any native support in the library for uploading such data type.

SoftLayer servers fail TLS 1.1 negotiation

Using this client library results in the following error on Ubuntu 12.04:
SSLError: [Errno 8] _ssl.c:504: EOF occurred in violation of protocol

The bug needs to be fixed on SoftLayer's servers. See https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/965371

How do i set Expires, and cache control Headers

How do i set Expires, and cache control Headers?
This doesn't set the expires and cache control headers.

    sl_storage = object_storage.get_client('XXX', 'YYY',datacenter='sjc01')
    f=sl_storage['video']['aa.txt'].create()
    f.load_from_filename('aa.txt')
    f.update(headers={'Expires':'Thu, 31 Dec 2037 23:55:55 GMT', 'Cache-Control':'public'})

Headers are not set.

    curl -I https://sjc01.objectstorage.softlayer.net/v1/AUTH_e3f9614e-c713-4151-a633-45d4a4d1381b/video/aa.txt
    HTTP/1.1 200 OK
    Content-Length: 8
    X-Backend-Timestamp: 1433976739.38300
    Accept-Ranges: bytes
    Last-Modified: Wed, 10 Jun 2015 22:52:20 GMT
    Etag: 93845a628143e88c97bae71e4cd4cbae
    X-Timestamp: 1433976739.38300
    Content-Type: text/plain
    X-Trans-Id: txe3e01937473d4386b540a-005578bfa9
    Date: Wed, 10 Jun 2015 22:52:25 GMT

How Upload Files

How upload files with Object Storage, how using a method Write or chunk_upload, help me please

Reading file in in chunks?

When doing:

object_storage[container][object].read()

Does it return data in chunks? I.e.:

for data_block in object_storage[container][object].read():
    print "got data block"

is_dir() AttributeError: StorageObjectModel instance has no attribute 'content_type'

Looks like the line:

        return self.model.content_type in ['text/directory',
                                           'application/directory']

should be:

        return self.model['content_type'] in ['text/directory',
                                              'application/directory']

AttributeError on auth token

I'm trying to authenticate using Requests for the transport, and am receiving this issue:

Traceback (most recent call last):
  File "test.py", line 12, in <module>
    sl = sl.SoftLayer()
  File "/usr/lib/python2.7/site-packages/inject/injections.py", line 118, in injection_wrapper
    return func(*args, **kwargs)
  File "/path/sl.py", line 27, in __init__
    self.os = object_storage.get_requests_client(sl_un, sl_pw, datacenter=def_dc)
  File "/usr/lib/python2.7/site-packages/object_storage/__init__.py", line 47, in get_requests_client
    conn = AuthenticatedConnection(auth)
  File "/usr/lib/python2.7/site-packages/object_storage/transport/requestsconn.py", line 24, in __init__
    self.auth.authenticate()
  File "/usr/lib/python2.7/site-packages/object_storage/transport/requestsconn.py", line 104, in authenticate
    self.auth_headers = {'X-Auth-Token': self.auth_token}
AttributeError: can't set attribute

From what I can tell the sl_un, sl_pw and def_dc are populated correctly and being passed (I can verify they're populated). I'm using Requests library 0.11.2 and SL library 0.4.4, with Python 2.7.3. I don't get why this error is happening, sadly.

storage_object.send() doesn't check for None in name attribute

Often we use a SpooledTemporaryFile to write data to SL object storage. Previously we were using Python 2.7.2 and this worked fine. After updating to Python 2.7.9 we were running into an issue with this scenario where we would get an exception:

TypeError: expected string or buffer
File "python2.7/site-packages/object_storage/storage_object.py", line 349, in send
File "python2.7/mimetypes.py", line 291, in guess_type
File "python2.7/mimetypes.py", line 114, in guess_type
File "python2.7/urllib.py", line 1080, in splittype

It turns out in Python 2.7.4 a name attribute was added to SpooledTemporaryFile which has a value of None. (See http://bugs.python.org/issue10355) That breaks this code in send():

    if not content_type:
        _type = None
        if hasattr(data, 'name'):
            _type = mimetypes.guess_type(data.name)[0]
        content_type = (_type or
                        mimetypes.guess_type(self.name)[0] or
                        'application/octet-stream')

The issue can be fixed by changing the hasattr line:

    if not content_type:
        _type = None
        if hasattr(data, 'name') and data.name:
            _type = mimetypes.guess_type(data.name)[0]
        content_type = (_type or
                        mimetypes.guess_type(self.name)[0] or
                        'application/octet-stream')

WindowsError - [Error 5] Access is denied - Python27\lib\ssl.py in _load_windows_store_certs, line 365

Hey,

I am getting following error, i have tried setting up on different server, on local server code is working fine.

Request Method: GET
Request URL: http://*********/
Django Version: 1.9.5
Exception Type: WindowsError
Exception Value:
[Error 5] Access is denied
Exception Location: c:\Python27\lib\ssl.py in _load_windows_store_certs, line 365
Python Executable: c:\Python27\python.exe
Python Version: 2.7.9
Python Path:
['.',
'c:\pyLGUC',
'C:\Windows\SYSTEM32\python27.zip',
'c:\Python27\DLLs',
'c:\Python27\lib',
'c:\Python27\lib\plat-win',
'c:\Python27\lib\lib-tk',
'c:\Python27',
'c:\Python27\lib\site-packages',
'C:\pyLGUC']

Getting error on :-

c:\Python27\lib\ssl.py in _load_windows_store_certs
for cert, encoding, trust in enum_certificates(storename):

Local Vars

purpose
_ASN1Object(nid=129, shortname='serverAuth', longname='TLS Web Server Authentication', oid='1.3.6.1.5.5.7.3.1')

storename
'CA'

It has been a week couldn't find anything on web.

Python 3.x Support

Hey guys,

Has anyone tested this on python 3.x? Is it supported?

Thanks,
Wissam

PEP-8 Fixes and tox test

Codebase needs all PEP-8 violations fixed, and an additional tox test to run flake8.

save_to_filename is broken

Hi, I'm having a bit of a bad day :-( I'm having lots of trouble restoring my backup because save_to_filename is broken. When I restore the files they end up being smaller than the originals.

E.g. I uploaded a file name connectifier.11 which is 2047 MB. When I've restored it, it is much smaller:
-rw-r--r-- 1 root root 1687725706 Nov 8 12:08 connectifier.11

I've filed a support ticket on this as well (https://manage.softlayer.com/Support/viewTicket/5955452), but the issue definitely seems to be that function given all signs thus far.

Can't work in a private only network

I have a virtual server on softlayer with closed public network, and when I try to run the following, it fails (Exception below):
import object_storage
sl_storage = object_storage.get_client(X,Y, datacenter='dal05',auth_url='http://dal05.objectstorage.softlayer.net/auth/v1.0') # WORKS
sl_storage['barda'].exists() # FAILED

In order to debug the problem, we changed the "/usr/lib/python2.7/site-packages/object_storage/transport/httplib2conn.py" file to print the request url and we found that it sends a request to the public url "http://dal05.objectstorage.softlayer.net/auth/v1.0" instead of the private url in the input auth_url "http://dal05.objectstorage.service.networklayer.com/auth/v1.0"

Exception:
File "/usr/lib/python2.7/site-packages/object_storage/client.py", line 249, in create_container
return self.container(name).create()
File "/usr/lib/python2.7/site-packages/object_storage/container.py", line 167, in create
headers={'Content-Length': '0'})
File "/usr/lib/python2.7/site-packages/object_storage/container.py", line 317, in make_request
return self.client.make_request(method, path, _args, *_kwargs)
File "/usr/lib/python2.7/site-packages/object_storage/client.py", line 354, in make_request
result = self.conn.make_request(method, url, _args, *_kwargs)
File "/usr/lib/python2.7/site-packages/object_storage/transport/httplib2conn.py", line 51, in make_request
response = _make_request(headers)
File "/usr/lib/python2.7/site-packages/object_storage/transport/httplib2conn.py", line 44, in _make_request
res, content = self.http.request(url, method, headers=headers, body=data)
File "/usr/lib/python2.7/site-packages/httplib2/init.py", line 1593, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/usr/lib/python2.7/site-packages/httplib2/init.py", line 1335, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/usr/lib/python2.7/site-packages/httplib2/init.py", line 1291, in _conn_request
response = conn.getresponse()
File "/usr/lib64/python2.7/httplib.py", line 1033, in getresponse
raise ResponseNotReady()
httplib.ResponseNotReady

Remove Python 2.5 support

In line with our other projects, 2.6 should become the minimum supported.

Error to create the client in Windows 7 + python 3.4

When I tried to create the client (using windows 7 + python 3.4 ) I got the following error:

Traceback (most recent call last):
File "C:\scripst\getdetails\customerObjectStorageIssue.py", line 3, in
sl_storage = object_storage.get_client('XXXXX', 'XXXX', datacenter='dal05')
File "c:\python34\lib\site-packages\object_storage__init__.py", line 21, in get_client
return get_httplib2_client(args, *kwargs)
File "c:\python34\lib\site-packages\object_storage__init.py", line 41, in get_httplib2_client
conn = AuthenticatedConnection(auth)
File "c:\python34\lib\site-packages\object_storage\transport\httplib2conn.py", line 31, in init
self.auth.authenticate()
File "c:\python34\lib\site-packages\object_storage\transport\httplib2conn.py", line 89, in authenticate
res, content = http.request(self.auth_url, 'GET', headers=headers)
File "c:\python34\lib\site-packages\httplib2__init__.py", line 1174, in request
self.disable_ssl_certificate_validation)
File "c:\python34\lib\site-packages\httplib2__init__.py", line 829, in init
check_hostname=True)
File "c:\python34\lib\http\client.py", line 1211, in init
raise ValueError("check_hostname needs a SSL context with "
ValueError: check_hostname needs a SSL context with either CERT_OPTIONAL or CERT_REQUIRED

In Ubuntu the same code is working fine.

AuthenticationError at Bluemix ObjectStorage

Failing to connect using sl_storage = object_storage.get_client('YOUR_USERNAME', 'YOUR_API_KEY', datacenter='dal05')

Don't know which credentials to use. Here is what my Bluemix OS credentials looks like

{
  "auth_url": "https://identity.open.softlayer.com",
  "project": "object_storage_072d5151_ee1a_4432_afa8_e822cbaa146c",
  "projectId": "1af89xxxxx1853",
  "region": "dallas",
  "userId": "fdcd47xxxxx9a60b3ac5",
  "username": "member_aa460xxxxxxx0d5b70",
  "password": "*******",
  "domainId": "c4exxxxxxx1338",
  "domainName": "10xxxx61",
  "role": "member"
}

I tried the userIdand username as YOUR_USERNAME and password as YOUR_API_KEY. And always got AuthenticationError: Invalid Credentials

Anything I am doing wrong here or any code must be updated???

Add Travis-CI configuration

We have a test suite... we should be doing automated tests for pull requests via Travis CI. Need a configuration and if I recall correctly need to flip a switch on for Travis.

Provide some way to get files back out of Softlayer!

I'm so frustrated right now. My backup is totally locked up in the cloud storage container with absolutely no way for me to get it out. Using the Python client I get back a corrupted file (#11). Meanwhile cloudfuse just totally hangs. Thus I'm left with absolutely no way to get my files back out of this black hole.

consolidate logging in object_storage package

There are (at least) three logging instances being created in the object_storage package, and that makes it a bit messy for callers to redirect logging to another logging instance owned by the caller. It would be nicer if object_storage.__init__.py created a singleton that the rest of the library used. Then replacing the logging instance would be as simple as reassigning the top level logger with the callers logger.

Custom ports cause getaddrinfo-related exceptions with chunked uploads

ChunkedUploadConnection doesn't seem to be ensuring that any port it parses out of the connection string is actually an int() before passing it onto an HTTPConnection/HTTPSConnection object. From there, it is expected to be an int, and getaddrinfo will also want this.

Currently this barfs with an extremely misleading "Disconnected" exception with a 0 status code.

Is this library/module thread-safe?

I'm looking to use this for a project I'm working on using Tornado and asynchronous calls, and for it to be viable I need this to be thread-safe. So, yeah, heh...basically, is it safe to use in threaded scenarios?

Large object support

Uploading a >5 GiB object will fail. Need to handle these as SLOs, or add a dependency for the official Swift client module and call it for uploads. This would allow us to still retain backwards compatibility with our module's API, as well as the service APIs for search and CDN.

Update to comply with OpenStack Hacking rules in flake8

Would make sense and improve this codebase a bit if we included the hacking checks in our flake8/pep8 checks.

How to get cdn_url - Not getting cdn_url None

what is the right way to get cdn_url from object_storage?
after doing prime_cd and get_info, cdn_url is None

  >>> sl_storage = object_storage.get_client('###', '###',datacenter='sjc01')
  >>> f=c['YUk1-27-Apr-15-10-33-04.xml']
  >>> f.prime_cdn()
  f<object_storage.transport.Response object at 0x7f65480ac5d0>
  >>> f.get_info()
  {'cdn_url': None, 'hash': '74be24b6ddb41357275dd3ad2a2f4cc9', 'name': 'YUk1-27-Apr-15-10-33-04.xml', 'url': u'https://sjc01.objectstorage.softlayer.net/v1/AUTH_e3f9614e-c713-4151-a633-45d4a4d1381b/foo/YUk1-27-Apr-15-10-33-04.xml', 'cdn_ssl_url': None, 'manifest': None, 'last_modified': 'Wed, 10 Jun 2015 17:47:26 GMT', 'meta': {}, 'content_encoding': None, 'content_type': 'text/xml', 'path': 'foo/YUk1-27-Apr-15-10-33-04.xml', 'container': 'foo', 'cache_control': None, 'size': 2536}
  >>> f=c['YUk1-27-Apr-15-10-33-04.xml']
  >>> f.get_info()
  {'cdn_url': None, 'hash': '74be24b6ddb41357275dd3ad2a2f4cc9', 'name': 'YUk1-27-Apr-15-10-33-04.xml', 'url': u'https://sjc01.objectstorage.softlayer.net/v1/AUTH_e3f9614e-c713-4151-a633-45d4a4d1381b/foo/YUk1-27-Apr-15-10-33-04.xml', 'cdn_ssl_url': None, 'manifest': None, 'last_modified': 'Wed, 10 Jun 2015 17:47:26 GMT', 'meta': {}, 'content_encoding': None, 'content_type': 'text/xml', 'path': 'foo/YUk1-27-Apr-15-10-33-04.xml', 'container': 'foo', 'cache_control': None, 'size': 2536}
  >>>