open-io / oio-swift

Openio flavor of the Openstack Swift Gateway and middlewares, allowing access to OpenIO SDS via an Openstack Swift gateway.

oioswift

OpenIO SDS Swift Gateway.

Features

Included:

• Operations on objects, containers and accounts
• Authentication support
• Metadata support
• Swift Middleware support

Installation

If you want to work with the current development version you can:

You can install directly from trunk on GitHub:

pip install git+git://github.com/open-io/oio-swift.git

Download and install from source by running:

python setup.py install

Note that at least swift 2.7.0 is required.

Updates

From GitHub:

pip install --upgrade git+git://github.com/open-io/oio-swift.git

Configuration

The SDS Swift Gateway uses the OpenStack Swift Proxy.

There is a sample configuration in conf/default.cfg.

Configuration items:

• sds_namespace - the SDS Namespace to serve. Example: OPENIO
• sds_proxy_url - the URL of the SDS proxy. Example: http://127.0.0.1:6000

Run

Use the proxy-server launch script from Swift.

Alternatively you can simply use:

# for development only
python runserver.py

Links

Resources:

• OpenIO SDS
• OpenIO SDS Python API
• OpenStack Swift API

oio-swift's Issues

Dependencies unaligned

oio-sds and oio-swift don't have the same requirements on eventlet.

• oio-sds: eventlet!=0.18.3,!=0.20.1,<0.21.0,>=0.18.2
• oio-swift/third_party/swift: eventlet>=0.17.4 # MIT

If one follows the .travis.yml script, we end up with an eventlet in conflict with an explicit statement for oio-sds.

aws cli with keystone

My proxy-server.conf:

OpenIO managed

[DEFAULT]
use_stderr = False
bind_ip = 192.168.2.154
bind_port = 6007
workers = 3
user = openio
log_facility = /dev/log
log_level = INFO
eventlet_debug = false

sds_namespace = OPENIO
sds_proxy_url = http://192.168.2.154:6006
sds_default_account = default

sds_connection_timeout = 5
sds_read_timeout = 35
sds_write_timeout = 35

sds_pool_connections = 500
sds_pool_maxsize = 500
sds_max_retries = 0

[pipeline:main]
#pipeline = catch_errors gatekeeper healthcheck proxy-logging cache bulk tempurl proxy-logging swift3 tempauth proxy-logging copy container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server
pipeline = catch_errors gatekeeper healthcheck proxy-logging cache tempurl ratelimit authtoken swift3 s3token copy container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server

[filter:catch_errors]
use = egg:swift#catch_errors

[filter:gatekeeper]
use = egg:swift#gatekeeper

[filter:healthcheck]
use = egg:swift#healthcheck

[filter:proxy-logging]
access_log_headers = False
use = egg:swift#proxy_logging
access_log_headers_only =

[filter:cache]
use = egg:swift#memcache
memcache_max_connections = 500
memcache_servers = 192.168.2.25:6019,192.168.2.26:6019,192.168.2.154:6019

[filter:bulk]
use = egg:swift#bulk

[filter:tempurl]
use = egg:swift#tempurl

[filter:authtoken]
paste.filter_factory = keystonemiddleware.auth_token:filter_factory
auth_url = http://192.168.2.154:35357
www_authenticate_uri = http://192.168.2.154:5000
auth_type = password
project_domain_id = default
user_domain_id = default
project_name = service
username = swift
password = SWIFT_PASS
delay_auth_decision = True
include_service_catalog = False
#memcached_servers = 127.0.0.1:11211

[filter:swift3]
use = egg:swift3#swift3
check_bucket_owner = True
force_swift_request_proxy_log = True
location = us-east-1
s3_acl = True

#[filter:tempauth]
#use = egg:oioswift#tempauth
#user_demo_demo = DEMO_PASS .admin

[filter:s3token]
paste.filter_factory = keystonemiddleware.s3_token:filter_factory
auth_port = 35357
auth_host = 192.168.2.154
auth_protocol = http

[filter:copy]
object_post_as_copy = False
use = egg:swift#copy

[filter:container-quotas]
use = egg:swift#container_quotas

[filter:account-quotas]
use = egg:swift#account_quotas

[filter:slo]
use = egg:swift#slo

[filter:dlo]
use = egg:swift#dlo

[filter:versioned_writes]
use = egg:oioswift#versioned_writes
allow_versioned_writes = True
[filter:tempurl]
use = egg:swift#tempurl

[filter:catch_errors]
use = egg:swift#catch_errors

[filter:ratelimit]
use = egg:swift#ratelimit

[filter:healthcheck]
use = egg:swift#healthcheck

[filter:copy]
use = egg:swift#copy
object_post_as_copy = false

[app:proxy-server]
object_post_as_copy = False
use = egg:oioswift#main
log_name = OIO,OPENIO,oioswift,0
account_autocreate = True
sds_chunk_checksum_algo = md5
allow_account_management = True

I also create ec2 credential in my keystone and 'aws configure list' but still signature missmatch? How can i debug it? And does region of project in keystone related with this in proxy-server.conf?
[filter:swift3]
use = egg:swift3#swift3
check_bucket_owner = True
force_swift_request_proxy_log = True
location = us-east-1 **********************
s3_acl = True

Thank you!

Document the storage policy settings

The storage policy settings are not obvious, and are currently not documented. We would like some documentation. Maybe we should also change the format of the settings.

"+" (plus) in filenames causes 400 error

Attempting to upload files with S3-illegal characters in the filename does not typically cause any problem. For example, I can upload objects with "&" in the name. However, files containing "+" in the name fail with a 400 error which reads "Header 'x-amz-content-sha256' set to the hex-encoded SHA256 hash of the request payload is required for AWS version 4 request signing, please set this on: PUT".

Of course, if this is an upstream problem, then I can move the report to Launchpad.

support the x-trans-id

use the swift x-trans-id to trace requests in oio-sds

max file size?

it seems the proxy enforces a max file size of 5GB by default, please provide info on how to change this.

Add tests for storage policies depending on the size of objects

Support ACLs

Upon a PUT, ensure the expected number of bytes has been received

Bad bucket creation date for S3

We need to handle the CreationDate of buckets for S3 compatibility.
The creation date is stuck to 2009-02-03T16:45:09.000Z.

As described here, it is a creation date, not a modification time.

aws --endpoint-url http://192.168.1.49:6007 --no-verify-ssl s3api list-buckets
{
    "Buckets": [
        {
            "Name": "bucket",
            "CreationDate": "2009-02-03T16:45:09.000Z"
        },
        {
            "Name": "bucket1",
            "CreationDate": "2009-02-03T16:45:09.000Z"
        },
        {
            "Name": "bucket2",
            "CreationDate": "2009-02-03T16:45:09.000Z"
        },
        {
            "Name": "bucket3",
            "CreationDate": "2009-02-03T16:45:09.000Z"
        }
    ],
    "Owner": {
        "DisplayName": "demo:demo",
        "ID": "demo:demo"
    }
}

Code retour 204 during put (then put fail) when file size is greater than multipart size

oio-version: 4.0.0-1
oio-swift: 1.0.0-0.b4

packages:

python2-swiftclient-3.3.0-1.el7.noarch
openio-sds-swift-1.0.0-0.b4.el7.oio.noarch
openio-sds-swift-plugin-swift3-1.12.0.b0-0.el7.oio.noarch
python-swift-2.13.0-1.el7.noarch
openstack-swift-proxy-2.13.0-1.el7.noarch
openstack-swift-container-2.13.0-1.el7.noarch

client: s3cmd-1.6.1-1.el7.noarch

Each time we try to put a file with a greater size than the default mutlipart size (15M) we encountered this error with s3cmd:

DEBUG: s3cmd version 1.6.1
DEBUG: ConfigParser: Reading file 'config_test'
DEBUG: ConfigParser: use_https->True
DEBUG: ConfigParser: signature_v2->True
DEBUG: ConfigParser: access_key->pr...28_chars...y
DEBUG: ConfigParser: secret_key->xq...13_chars...f
DEBUG: ConfigParser: host_base->xxxxxxxx
DEBUG: ConfigParser: host_bucket->xxxxxxxxxx
DEBUG: ConfigParser: socket_timeout->100
DEBUG: Updating Config.Config cache_file ->
DEBUG: Updating Config.Config follow_symlinks -> False
DEBUG: Updating Config.Config verbosity -> 10
DEBUG: Unicodising 'put' using UTF-8
DEBUG: Unicodising 'vingtMBfile' using UTF-8
DEBUG: Unicodising 's3://openiotest/vingtMBfile' using UTF-8
DEBUG: Command: put
DEBUG: DeUnicodising u'vingtMBfile' using UTF-8
INFO: Compiling list of local files...
DEBUG: DeUnicodising u'vingtMBfile' using UTF-8
DEBUG: DeUnicodising u'vingtMBfile' using UTF-8
DEBUG: Unicodising '' using UTF-8
DEBUG: DeUnicodising u'vingtMBfile' using UTF-8
DEBUG: DeUnicodising u'vingtMBfile' using UTF-8
DEBUG: Applying --exclude/--include
DEBUG: CHECK: vingtMBfile
DEBUG: PASS: u'vingtMBfile'
INFO: Running stat() and reading/calculating MD5 values on 1 files, this may take some time...
DEBUG: DeUnicodising u'vingtMBfile' using UTF-8
DEBUG: doing file I/O to read md5 of vingtMBfile
DEBUG: DeUnicodising u'vingtMBfile' using UTF-8
INFO: Summary: 1 local files to upload
DEBUG: attr_header: {'x-amz-meta-s3cmd-attrs': 'uid:0/gname:root/uname:root/gid:0/mode:33188/mtime:1501663079/atime:1501663132/md5:8f4e33f3dc3e414ff94e5fb6905cba8c/ctime:1501663079'}
DEBUG: DeUnicodising u'vingtMBfile' using UTF-8
DEBUG: DeUnicodising u'vingtMBfile' using UTF-8
DEBUG: DeUnicodising u'vingtMBfile' using UTF-8
DEBUG: String 'vingtMBfile' encoded to 'vingtMBfile'
DEBUG: CreateRequest: resource[uri]=/vingtMBfile?uploads
DEBUG: Using signature v2
DEBUG: SignHeaders: 'POST\n\napplication/octet-stream\n\nx-amz-date:Wed, 02 Aug 2017 08:53:03 +0000\nx-amz-meta-s3cmd-attrs:uid:0/gname:root/uname:root/gid:0/mode:33188/mtime:1501663079/atime:1501663132/md5:8f4e33f3dc3e414ff94e5fb6905cba8c/ctime:1501663079\nx-amz-storage-class:STANDARD\n/openiotest/vingtMBfile?uploads'
DEBUG: Processing request, please wait...
DEBUG: get_hostname(openiotest): xxxxxxxxxxxxx
DEBUG: ConnMan.get(): creating new connection: https://xxxxxxxxxxxxxx
DEBUG: Using ca_certs_file None
DEBUG: httplib.HTTPSConnection() has both context and check_hostname
DEBUG: non-proxied HTTPSConnection(xxxxxxxxxxxxxxx)
DEBUG: format_uri(): /openiotest/vingtMBfile?uploads
DEBUG: Sending request method_string='POST', uri='/openiotest/vingtMBfile?uploads', headers={'x-amz-meta-s3cmd-attrs': 'uid:0/gname:root/uname:root/gid:0/mode:33188/mtime:1501663079/atime:1501663132/md5:8f4e33f3dc3e414ff94e5fb6905cba8c/ctime:1501663079', 'content-type': 'application/octet-stream', 'Authorization': 'AWS XXXX:XXXXX:XXXXX=', 'x-amz-date': 'Wed, 02 Aug 2017 08:53:03 +0000', 'x-amz-storage-class': 'STANDARD'}, body=(0 bytes)
DEBUG: Response: {'status': 500, 'headers': {'transfer-encoding': 'chunked', 'server': 'nginx/1.13.3', 'x-amz-id-2': 'tx016597d721b04644ba766-00598192e8', 'connection': 'keep-alive', 'x-amz-request-id': 'tx016597d721b04644ba766-00598192e8', 'x-trans-id': 'tx016597d721b04644ba766-00598192e8', 'date': 'Wed, 02 Aug 2017 08:53:03 GMT', 'content-type': 'application/xml', 'x-openstack-request-id': 'tx016597d721b04644ba766-00598192e8'}, 'reason': 'Internal Server Error', 'data': "<?xml version='1.0' encoding='UTF-8'?>\n<Error><Code>InternalError</Code><Message>unexpected status code 204</Message><RequestId>tx016597d721b04644ba766-00598192e8</RequestId></Error>"}
DEBUG: ConnMan.put(): connection put back to pool (https://xxxxxxxxxxxx)
DEBUG: S3Error: 500 (Internal Server Error)
DEBUG: HttpHeader: transfer-encoding: chunked
DEBUG: HttpHeader: server: nginx/1.13.3
DEBUG: HttpHeader: x-amz-id-2: tx016597d721b04644ba766-00598192e8
DEBUG: HttpHeader: connection: keep-alive
DEBUG: HttpHeader: x-amz-request-id: tx016597d721b04644ba766-00598192e8
DEBUG: HttpHeader: x-trans-id: tx016597d721b04644ba766-00598192e8
DEBUG: HttpHeader: date: Wed, 02 Aug 2017 08:53:03 GMT
DEBUG: HttpHeader: content-type: application/xml
DEBUG: HttpHeader: x-openstack-request-id: tx016597d721b04644ba766-00598192e8
DEBUG: ErrorXML: Code: 'InternalError'
DEBUG: ErrorXML: Message: 'unexpected status code 204'
DEBUG: ErrorXML: RequestId: 'tx016597d721b04644ba766-00598192e8'
WARNING: Retrying failed request: /vingtMBfile?uploads
WARNING: 500 (InternalError): unexpected status code 204
WARNING: Waiting 3 sec...

Related swift error:

2017-08-02T12:11:03.632649+03:00 oio1 OIO,OPENIO,oioswift,1: err  500 Internal Server Error: #012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/swift3/middleware.py", line 119, in __call__#012    resp = self.handle_request(req)#012  File "/usr/lib/python2.7/site-packages/swift3/middleware.py", line 150, in handle_request#012    res = handler(req)#012  File "/usr/lib/python2.7/site-packages/swift3/controllers/base.py", line 59, in wrapped#012    return func(self, req)#012  File "/usr/lib/python2.7/site-packages/swift3/controllers/base.py", line 71, in check_container#012    return func(self, req)#012  File "/usr/lib/python2.7/site-packages/swift3/controllers/multi_upload.py", line 340, in POST#012    req.get_response(self.app, 'PUT', container, '')#012  File "/usr/lib/python2.7/site-packages/swift3/request.py", line 1353, in get_response#012    headers, body, query)#012  File "/usr/lib/python2.7/site-packages/swift3/request.py", line 1338, in _get_response#012    raise InternalError('unexpected status code %d' % status)#012InternalError: 500 Internal Server Error (txn: txcee6b435347f48d687c4f-0059819727) (client_ip: 149.202.213.199)
2017-08-02T12:11:03.634296+03:00 oio1 OIO,OPENIO,oioswift,1: info  149.202.213.199 149.202.213.199 02/Aug/2017/09/11/03 POST /openiotest/vingtMBfile%3Fuploads HTTP/1.0 500 - - - - 182 - txcee6b435347f48d687c4f-0059819727 - 0.0364 - - 1501665063.596266985 1501665063.632709026 -

The workaround is to force the multipart size greater than the filesize. (option: --multipart-chunk-size=xx)

More information for the "RAWX write failure" problem

This error need to give more information about the failure: rawx that caused the problem, something useful.

/var/log/oio/sds/NS/oioswift-0/oioswift-0.log:2017-03-14T23:15:50.261350+01:00 gw-swift1 OIO,NS,oioswift,0: err  ERROR Exception transferring data {'path': '/v1.0/AUTH_4ce1bbdaad214898b120a1952a971732/ar-backup-ac3006-test/4xXgKw9z1qXvSqZs2S6-xSQPy7A-n51EMVF3laCVAWg-segment-1489529353-13'}: #012Traceback (most recent call last):#012  File "/usr/lib/python2.7/dist-packages/oioswift/proxy/controllers/obj.py", line 340, in _store_object#012    etag=req.headers.get('etag', '').strip('"'), metadata=metadata)#012  File "/usr/lib/python2.7/dist-packages/oio/api/object_storage.py", line 70, in _wrapped#012    return fnc(self, account, container, *args, **kwargs)#012  File "/usr/lib/python2.7/dist-packages/oio/api/object_storage.py", line 404, in object_create#012    policy=policy, headers=headers, key_file=key_file)#012  File "/usr/lib/python2.7/dist-packages/oio/api/object_storage.py", line 662, in _object_create#012    final_chunks, bytes_transferred, content_checksum = handler.stream()#012  File "/usr/lib/python2.7/dist-packages/oio/api/replication.py", line 255, in stream#012    size)#012  File "/usr/lib/python2.7/dist-packages/oio/api/replication.py", line 163, in stream#012    raise exc.OioException("RAWX write failure")#012OioException: RAWX write failure (txn: tx460e2a371e8c4c5faf140-0058c86a4c) (client_ip: 95.128.42.70)

Fix stats count for containers

easy to fix
https://github.com/open-io/oio-swift/blob/master/oioswift/proxy/controllers/container.py#L224

Exception occurs when username doesn't contain colon

Traceback (most recent call last):
  File "/usr/bin/oioswift-proxy-server", line 80, in <module>
    **options)
  File "/usr/lib/python2.7/dist-packages/swift/common/wsgi.py", line 914, in run_wsgi
    loadapp(conf_path, global_conf=global_conf)
  File "/usr/lib/python2.7/dist-packages/swift/common/wsgi.py", line 397, in loadapp
    return ctx.create()
  File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
    return self.object_type.invoke(self)
  File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 207, in invoke
    app = filter(app)
  File "/usr/lib/python2.7/dist-packages/oioswift/common/middleware/tempauth.py", line 88, in tempauth_filter
    return OioTempAuth(app, conf)
  File "/usr/lib/python2.7/dist-packages/oioswift/common/middleware/tempauth.py", line 32, in __init__
    super(OioTempAuth, self).__init__(app, conf)
  File "/usr/lib/python2.7/dist-packages/swift/common/middleware/tempauth.py", line 233, in __init__
    account, username = conf_key.split('_', 1)[1].split('_')
ValueError: need more than 1 value to unpack

The above exception occurs, and crashes the process, when a user "name" does not conform to the pattern [^:]+:[^:]+.

Instead, it would be better for it to return an explicit error.

No Last-Modified header in Swift PUT response

ISSUE TYPE

• Bug Report

COMPONENT NAME

oio-swift

SDS VERSION

$ openio --version
openio 4.1.21

CONFIGURATION

$ cat /etc/oio/sds.conf.d/OPENIO 
[OPENIO]
conscience=10.0.2.161:6000
zookeeper=10.0.2.161:6005,10.0.2.162:6005,10.0.2.163:6005
proxy=10.0.2.163:6006
event-agent=beanstalk://10.0.2.163:6014
meta1_digits=3
udp_allowed=yes
ns.storage_policy=THREECOPIES
ns.chunk_size=10485760
ns.service_update_policy=meta2=KEEP|3|1|;rdir=KEEP|1|1|user_is_a_service=rawx;sqlx=KEEP|3|1|;

OS / ENVIRONMENT

$ cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

SUMMARY

PUT requests don't return Last-Modified response header.

STEPS TO REPRODUCE

With oio-swift installed, upload an object:

swift upload my_container my_object --debug
[...]
DEBUG:swiftclient:RESP STATUS: 201 Created
DEBUG:swiftclient:RESP HEADERS: {u'Content-Length': u'0', u'Etag': u'"1289584a84b60bc282dc9d444a2920f6"', u'X-Trans-Id': u'tx4f37a112397046d09499b-005ad5ae11', u'Date': u'Tue, 17 Apr 2018 08:19:29 GMT', u'Content-Type': u'text/html; charset=UTF-8', u'X-Openstack-Request-Id': u'tx4f37a112397046d09499b-005ad5ae11'}
openrc

EXPECTED RESULTS

Last-Modified header to be present in response headers, according to this: https://developer.openstack.org/api-ref/object-store/#create-or-replace-object.

ACTUAL RESULTS

Final headers don't contain Last-Modified

DEBUG:swiftclient:RESP HEADERS: {u'Content-Length': u'0', u'Etag': u'"1289584a84b60bc282dc9d444a2920f6"', u'X-Trans-Id': u'tx4f37a112397046d09499b-005ad5ae11', u'Date': u'Tue, 17 Apr 2018 08:19:29 GMT', u'Content-Type': u'text/html; charset=UTF-8', u'X-Openstack-Request-Id': u'tx4f37a112397046d09499b-005ad5ae11'}

Support Range Headers

__version__ number not updated ?

In the following file : https://github.com/open-io/oio-swift/blob/master/oioswift/__init__.py the version number for oio-swift is 0.6.1 which have not been updated since October 2016.

However the latest oio-swift version number in the release tab and the 17.04 distribution is 1.1.0.

Is there a mismatch somewhere ?

Review Response Headers

useful for #16

empty metadata POST container request fails

Results in

No properties found in JSON object (HTTP 400) (STATUS 400)