Swift3 also failed in creating object name with encoded character "/" (%2F), we tried this:

./s3curl.pl --id personal --put="/Users/minc/Downloads/README" -- http://10.102.192.184:8080/mybucket/s3curl%2Freadme

Got the following error:

AccessDenied
Access denied

Swift3 would crash with a bogus content_md5 :

swift Error: Incorrect padding:
Traceback (most recent call last):
File "/opt/stack/swift/swift/common/middleware/catch_errors.py", line 47, in call
return self.app(env, my_start_response)
File "/opt/stack/swift/swift/common/middleware/healthcheck.py", line 38, in call
return self.app(env, start_response)
File "/opt/stack/swift/swift/common/middleware/memcache.py", line 47, in call
return self.app(env, start_response)
File "/opt/stack/swift/swift/common/middleware/swift3.py", line 465, in call
res = getattr(controller, req.method)(env, start_response)
File "/opt/stack/swift/swift/common/middleware/swift3.py", line 377, in PUT
env['HTTP_ETAG'] = value.decode('base64').encode('hex')
File "/usr/lib/python2.7/encodings/base64_codec.py", line 42, in base64_decode
output = base64.decodestring(input)
File "/usr/lib/python2.7/base64.py", line 321, in decodestring
return binascii.a2b_base64(s)
Error: Incorrect padding (txn: tx7d530974e1664167ad6dbf1ab24b9f79) (client_ip: 127.0.0.1)

gerrit reviews (was abandoned):

https://review.openstack.org/6245

I try set bucket acl using boto, but get a 409 Conflict error.
After debug, I found the PUT request in swift3 would return 202 HTTP_ACCEPTE，202 would go to the case which return get_err_response('BucketAlreadyExists').
I think in old swift3 version, it would adapt the acl set requests to POST method.
The openstack document is vague whether it should be using POST or PUT.
The document said POST http://docs.openstack.org/api/openstack-object-storage/1.0/content/Update_Container_Metadata-d1e1900.html.
But the curl example section use PUT http://docs.openstack.org/api/openstack-object-storage/1.0/content/special-metadata-acls.html
Anybody can clarify this?
I think PUT is all right, but some code need to be fixed.

('PUT', '/bsdf18/?acl', '/bsdf18/?acl', {}, {'x-amz-acl': 'public-read'}, '', '192.168.2.148:38889')
send: 'PUT /bsdf18/?acl HTTP/1.1\r\nHost: 192.168.2.148:38889\r\nAccept-Encoding: identity\r\nDate: Mon, 13 May 2013 06:52:04 GMT\r\nContent-Length: 0\r\nx-amz-acl: public-read\r\nAuthorization: AWS 8710718e7e454b03b6367085e49d8b41:nA5qn/+8Fyz04jWAZ9wElQv8MK8=\r\nUser-Agent: Boto/2.8.0-dev (darwin)\r\n\r\n'
reply: 'HTTP/1.1 409 Conflict\r\n'
header: Content-Length: 160
header: Content-Type: text/xml
header: Date: Mon, 13 May 2013 06:52:04 GMT
Traceback (most recent call last):
File "./mss.py", line 91, in
main()
File "./mss.py", line 68, in main
rs.set_acl("public-read")
File "/Users/chenyin/Code/boto/boto/s3/bucket.py", line 771, in set_acl
headers, version_id)
File "/Users/chenyin/Code/boto/boto/s3/bucket.py", line 737, in set_canned_acl
response.status, response.reason, body)

From vitoordaz in #15

After I have an error when create buckets via Cyberduck, it's because you translate PUT request with X-AMZ-ACL to POST request, but Cyberduck usually add X-AMZ-ACL when create bucket.

PUT /lol/ HTTP/1.1
Date: Tue, 16 Oct 2012 22:56:57 GMT
x-amz-acl: public-read
Content-Type: application/octet-stream
Authorization: AWS admin:admin:qibESADS8jtU8aZTJa2K0KJqo1g=
Content-Length: 0
Host: 192.168.1.20:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.2.1 (Mac OS X/10.8.2) (i386)

HTTP/1.1 400 Bad Request
Content-Length: 142
Content-Type: text/xml
Date: Tue, 16 Oct 2012 22:56:53 GMT
Connection: keep-alive

account-server 127.0.0.1 - - [16/Oct/2012:22:56:53 +0000] "HEAD /sdb4/127477/AUTH_admin" 204 - "tx0530e3d57e004575ad43eddb90db4832" "-" "-" 0.0054 ""
container-server 127.0.0.1 - - [16/Oct/2012:22:56:53 +0000] "POST /sdb3/231295/AUTH_admin/lol" 404 - "tx0530e3d57e004575ad43eddb90db4832" "-" "-" 0.0012
container-server 127.0.0.1 - - [16/Oct/2012:22:56:53 +0000] "POST /sdb2/231295/AUTH_admin/lol" 404 - "tx0530e3d57e004575ad43eddb90db4832" "-" "-" 0.0013
container-server 127.0.0.1 - - [16/Oct/2012:22:56:53 +0000] "POST /sdb4/231295/AUTH_admin/lol" 404 - "tx0530e3d57e004575ad43eddb90db4832" "-" "-" 0.0010
proxy-server 192.168.1.50 192.168.1.50 16/Oct/2012/22/56/53 POST /v1/AUTH_admin/lol HTTP/1.0 499 - Cyberduck/4.2.1%20%28Mac%20OS%20X/10.8.2%29%20%28i386%29 admin%2Cs3 - 70 - tx0530e3d57e004575ad43eddb90db4832 - 0.0226 -

S3 clients hang when waiting for a response from an ACL query on an object. This is because the swift3 ObjectController GETorHEAD operation changes the request from a GET to a HEAD, but WSGI will not return an http body if the request method is "HEAD".

To fix this, you must change the req.method back to "GET" after the swift HEAD returns, just before calling get_acl.

Using s3cmd or boto Swift3 returns what appear as static (hard-coded) timestamps of containers. The hard-coded value seems to be in middleware.py on lines 400-407. (Timestamps returned using the python-swiftclient provide correct timestamps.)

We tried upload the file with name contained '-' and ‘_’, but they turn out to be encoded. eg. '-' -> '%2D'. Is this a bug or not?

I'm unable to authenticate using s3curl or any other tools.

Configuration:
OS : Centos 6
GlusterFS version 3.4 (volume name POC-slave)
Openstack swift version: 1.13 icehouse
Swiftonfile version: 1.13.1-0
Swift3 version: 1.7
Auth type: Tempauth

Everything is working fine via the Swift API but when I try to call the S3 API, I got 403 Forbidden.

Command:
[root@filer1a s3curl]# ./s3curl.pl --debug --id "admin:admin" --key 'admin' --get -- -v -s http://localhost:8080/

Results in /var/log/messages:

"Jun 11 17:10:11 XXXX proxy-server: 127.0.0.1 127.0.0.1 11/Jun/2014/15/10/11 GET /v1/AUTH_admin%3Fformat%3Djson HTTP/1.0 403 - curl/7.19.7%20%28x86_64-redhat-linux-gnu%29%20libcurl/7.19.7%20NSS/3.13.1.0%20zlib/1.2.3%20libidn/1.18%20libssh2/1.2.2 R0VUCgoKV2VkLCAxMSBKdW4gMjAxNCAxNToxMDoxMSArMDAwMAov - 124 - tx8e19cc01365b4d03b7630-0053987153 - 0.0018 - - 1402499411.762089014 1402499411.763856888"

I guess that my URI should be something like /v1/AUTH_POC-slave/ .

Hi,

do you support the Amazon multi-part upload feature?
I understand that the way amazon handles large files is slightly different from OpenStack in that you have to announce a large file upload first, and then use an ID to identify the different parts, while you don't have to do that with swift.

Hi,

How is swift3 versioned against various release of Swift?

For example, I submitted a fix yesterday: https://review.openstack.org/#/c/14133/ that introduces a new feature that could simplify some of the swift3 implementation for bucket.set_acl('private')

Should I change the code to use this new feature, or should we aim to remain compatible with Folsom?

any idea why I only ever get access denied?

from proxy-server.conf
pipeline = catch_errors healthcheck cache swift3 authtoken keystone proxy-logging proxy-server

I've got absolutely no idea why this isn't working, swift client works, I can access the account using the swift api with curl

GET / HTTP/1.1
User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5
Host: xxxxxx
Accept: /
Date: Tue, 22 Jan 2013 03:35:29 +0000
Authorization: AWS xxxxxxxx

< HTTP/1.1 401 Unauthorized
< Www-Authenticate: Keystone uri='https://xxxxxxxx:35357'
< Content-Length: 381
< Content-Type: text/html; charset=UTF-8
< X-Trans-Id: tx4fa42f32f6914168b453bdb5a6fa12a1
< Date: Tue, 22 Jan 2013 03:32:12 GMT
<

<title>401 Unauthorized</title>

401 Unauthorized

This server could not verify that you are authorized to access the document you requested. Either you supplied the wrong credentials (e.g., bad password), or your browser does not understand how to supply the credentials required.

Authentication required

From Keystone.log

2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] ******************** REQUEST ENVIRON ********************
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] SCRIPT_NAME = /v2.0
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] webob.adhoc_attrs = {'response': <Response at 0x3c508d0 200 OK>}
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] REQUEST_METHOD = GET
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] PATH_INFO = /tokens/revoked
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] SERVER_PROTOCOL = HTTP/1.0
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] HTTP_X_AUTH_TOKEN = ff631edfdc5aee16a1be
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] eventlet.posthooks = []
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] RAW_PATH_INFO = /v2.0/tokens/revoked
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] REMOTE_ADDR = 127.0.0.1
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] eventlet.input = <eventlet.wsgi.Input object at 0x3aefa50>
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] wsgi.url_scheme = https
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] SERVER_PORT = 35357
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] wsgi.input = <eventlet.wsgi.Input object at 0x3aefa50>
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] openstack.context = {'token_id': 'ff631edfdc5aee16a1be', 'is_admin': True}
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] HTTP_HOST = 127.0.0.1:35357
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] HTTPS = on
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] wsgi.multithread = True
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] HTTP_ACCEPT = application/json
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] wsgi.version = (1, 0)
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] SERVER_NAME = 127.0.0.1
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] GATEWAY_INTERFACE = CGI/1.1
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] wsgi.run_once = False
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] wsgi.errors = <open file '', mode 'w' at 0x7f78dc7c21e0>
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] wsgi.multiprocess = False
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] CONTENT_TYPE = application/json
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] HTTP_ACCEPT_ENCODING = identity
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi]
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] ******************** REQUEST BODY ********************
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi]
2013-01-22 15:43:21 DEBUG [routes.middleware] Matched GET /tokens/revoked
2013-01-22 15:43:21 DEBUG [routes.middleware] Route path: '{path_info:.}', defaults: {'controller': <keystone.contrib.ec2.core.Ec2Extension object at 0x3621c10>}
2013-01-22 15:43:21 DEBUG [routes.middleware] Match dict: {'controller': <keystone.contrib.ec2.core.Ec2Extension object at 0x3621c10>, 'path_info': '/tokens/revoked'}
2013-01-22 15:43:21 DEBUG [routes.middleware] Matched GET /tokens/revoked
2013-01-22 15:43:21 DEBUG [routes.middleware] Route path: '{path_info:.}', defaults: {'controller': <keystone.contrib.s3.core.S3Extension object at 0x3616950>}
2013-01-22 15:43:21 DEBUG [routes.middleware] Match dict: {'controller': <keystone.contrib.s3.core.S3Extension object at 0x3616950>, 'path_info': '/tokens/revoked'}
2013-01-22 15:43:21 DEBUG [routes.middleware] Matched GET /tokens/revoked
2013-01-22 15:43:21 DEBUG [routes.middleware] Route path: '{path_info:.}', defaults: {'controller': <keystone.contrib.admin_crud.core.CrudExtension object at 0x36121d0>}
2013-01-22 15:43:21 DEBUG [routes.middleware] Match dict: {'controller': <keystone.contrib.admin_crud.core.CrudExtension object at 0x36121d0>, 'path_info': '/tokens/revoked'}
2013-01-22 15:43:21 DEBUG [routes.middleware] Matched GET /tokens/revoked
2013-01-22 15:43:21 DEBUG [routes.middleware] Route path: '{path_info:.}', defaults: {'controller': <keystone.service.AdminRouter object at 0x2910790>}
2013-01-22 15:43:21 DEBUG [routes.middleware] Match dict: {'controller': <keystone.service.AdminRouter object at 0x2910790>, 'path_info': '/tokens/revoked'}
2013-01-22 15:43:21 DEBUG [routes.middleware] Matched GET /tokens/revoked
2013-01-22 15:43:21 DEBUG [routes.middleware] Route path: '/tokens/revoked', defaults: {'action': u'revocation_list', 'controller': <keystone.service.TokenController object at 0x34efd90>}
2013-01-22 15:43:21 DEBUG [routes.middleware] Match dict: {'action': u'revocation_list', 'controller': <keystone.service.TokenController object at 0x34efd90>}
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] arg_dict: {}
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] ******************** RESPONSE HEADERS ********************
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] Vary = X-Auth-Token
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] Content-Type = application/json
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] Content-Length = 522
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi]
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] ******************** RESPONSE BODY ********************
2013-01-22 15:43:21 DEBUG [keystone.common.wsgi] {"signed": "-----BEGIN CMS-----

I have a problem regarding the s3 middleware in swift. I have everything up and running so far but have a problem with subfolder creations. I get the following error: Request Error [ AccessDenied Access denied ]. Forbidden.

Meaning the creation of an inital folder works fine and also storing files in that folder. But creating a subfolder in this returns a 403 error.

HEAD requests to S3 respond with the exact same header as a GET, swift3 responds with an incorrect Content-Type and Content-Length. A quick hack to fix this issue is below, but should probably be replaced with something involving DiskFile and such from Swift's API, because of innecessary overhead involved in reading in the file.

--- middleware_old.py   2012-07-10 11:18:26.258779790 -0700
+++ middleware.py   2012-07-10 11:13:52.924778983 -0700
@@ -331,7 +331,17 @@
                                              object_name)

     def GETorHEAD(self, env, start_response):
+        if env['REQUEST_METHOD'] == 'HEAD':
+            head = True
+            env['REQUEST_METHOD'] = 'GET'
+        else:
+            head = False
+
         app_iter = self._app_call(env)
+
+        if head:
+            app_iter = None
+
         status = self._get_status_int()
         headers = dict(self._response_headers)

I'm having a very strange issue when using Django+Django-storages via S3Storage backend, which uses boto when connecting to swift 1.7.6+latest pull of your swift3 module. It works fine with swift 1.4.8. Here are the two cases. Can you comment on what is broken with either how swift is processing the responses from middleware?

Problem: In the broken case, boto does a HEAD /object to see if the key exists and gets a HTTP 404, which is correct. It then PUTS the object. Here's where the fun starts. In the working case (swift 1.4.8) it gets a HTTP 200 after the PUT has succeeded and that's all. In the broken case, it receives the error response body for NoSuchKey in the swift3 code and then gets the HTTP 200. This seems to confuse it and it just hangs.

--Straces--
WORKING CASE:
Using swift 1.4.8 (CentOS 6 RPM) which comes with the old swift3 module
recvfrom(6, "H", 1, 0, NULL, NULL) = 1
recvfrom(6, "T", 1, 0, NULL, NULL) = 1
recvfrom(6, "T", 1, 0, NULL, NULL) = 1
recvfrom(6, "P", 1, 0, NULL, NULL) = 1
recvfrom(6, "/", 1, 0, NULL, NULL) = 1
recvfrom(6, "1", 1, 0, NULL, NULL) = 1
recvfrom(6, ".", 1, 0, NULL, NULL) = 1
recvfrom(6, "1", 1, 0, NULL, NULL) = 1
recvfrom(6, " ", 1, 0, NULL, NULL) = 1
recvfrom(6, "2", 1, 0, NULL, NULL) = 1
recvfrom(6, "0", 1, 0, NULL, NULL) = 1
recvfrom(6, "0", 1, 0, NULL, NULL) = 1
recvfrom(6, " ", 1, 0, NULL, NULL) = 1
recvfrom(6, "O", 1, 0, NULL, NULL) = 1
recvfrom(6, "K", 1, 0, NULL, NULL) = 1
recvfrom(6, "\r", 1, 0, NULL, NULL) = 1
recvfrom(6, "\n", 1, 0, NULL, NULL) = 1
recvfrom(6, "C", 1, 0, NULL, NULL) = 1
recvfrom(6, "o", 1, 0, NULL, NULL) = 1
recvfrom(6, "n", 1, 0, NULL, NULL) = 1
recvfrom(6, "t", 1, 0, NULL, NULL) = 1
recvfrom(6, "e", 1, 0, NULL, NULL) = 1
recvfrom(6, "n", 1, 0, NULL, NULL) = 1
recvfrom(6, "t", 1, 0, NULL, NULL) = 1
recvfrom(6, "-", 1, 0, NULL, NULL) = 1
recvfrom(6, "T", 1, 0, NULL, NULL) = 1
recvfrom(6, "y", 1, 0, NULL, NULL) = 1
recvfrom(6, "p", 1, 0, NULL, NULL) = 1
recvfrom(6, "e", 1, 0, NULL, NULL) = 1
recvfrom(6, ":", 1, 0, NULL, NULL) = 1
recvfrom(6, " ", 1, 0, NULL, NULL) = 1
recvfrom(6, "t", 1, 0, NULL, NULL) = 1
recvfrom(6, "e", 1, 0, NULL, NULL) = 1
recvfrom(6, "x", 1, 0, NULL, NULL) = 1
recvfrom(6, "t", 1, 0, NULL, NULL) = 1
recvfrom(6, "/", 1, 0, NULL, NULL) = 1
recvfrom(6, "h", 1, 0, NULL, NULL) = 1
recvfrom(6, "t", 1, 0, NULL, NULL) = 1
recvfrom(6, "m", 1, 0, NULL, NULL) = 1
recvfrom(6, "l", 1, 0, NULL, NULL) = 1
recvfrom(6, ";", 1, 0, NULL, NULL) = 1
recvfrom(6, " ", 1, 0, NULL, NULL) = 1
recvfrom(6, "c", 1, 0, NULL, NULL) = 1
recvfrom(6, "h", 1, 0, NULL, NULL) = 1
recvfrom(6, "a", 1, 0, NULL, NULL) = 1
recvfrom(6, "r", 1, 0, NULL, NULL) = 1
recvfrom(6, "s", 1, 0, NULL, NULL) = 1
recvfrom(6, "e", 1, 0, NULL, NULL) = 1
recvfrom(6, "t", 1, 0, NULL, NULL) = 1
recvfrom(6, "=", 1, 0, NULL, NULL) = 1
recvfrom(6, "U", 1, 0, NULL, NULL) = 1
recvfrom(6, "T", 1, 0, NULL, NULL) = 1
recvfrom(6, "F", 1, 0, NULL, NULL) = 1
recvfrom(6, "-", 1, 0, NULL, NULL) = 1
recvfrom(6, "8", 1, 0, NULL, NULL) = 1
recvfrom(6, "\r", 1, 0, NULL, NULL) = 1
recvfrom(6, "\n", 1, 0, NULL, NULL) = 1
recvfrom(6, "C", 1, 0, NULL, NULL) = 1
recvfrom(6, "o", 1, 0, NULL, NULL) = 1
recvfrom(6, "n", 1, 0, NULL, NULL) = 1
recvfrom(6, "t", 1, 0, NULL, NULL) = 1
recvfrom(6, "e", 1, 0, NULL, NULL) = 1
recvfrom(6, "n", 1, 0, NULL, NULL) = 1
recvfrom(6, "t", 1, 0, NULL, NULL) = 1
recvfrom(6, "-", 1, 0, NULL, NULL) = 1
recvfrom(6, "L", 1, 0, NULL, NULL) = 1
recvfrom(6, "e", 1, 0, NULL, NULL) = 1
recvfrom(6, "n", 1, 0, NULL, NULL) = 1
recvfrom(6, "g", 1, 0, NULL, NULL) = 1
recvfrom(6, "t", 1, 0, NULL, NULL) = 1
recvfrom(6, "h", 1, 0, NULL, NULL) = 1
recvfrom(6, ":", 1, 0, NULL, NULL) = 1
recvfrom(6, " ", 1, 0, NULL, NULL) = 1
recvfrom(6, "0", 1, 0, NULL, NULL) = 1
recvfrom(6, "\r", 1, 0, NULL, NULL) = 1
recvfrom(6, "\n", 1, 0, NULL, NULL) = 1
recvfrom(6, "E", 1, 0, NULL, NULL) = 1
recvfrom(6, "t", 1, 0, NULL, NULL) = 1
recvfrom(6, "a", 1, 0, NULL, NULL) = 1
recvfrom(6, "g", 1, 0, NULL, NULL) = 1
recvfrom(6, ":", 1, 0, NULL, NULL) = 1
recvfrom(6, " ", 1, 0, NULL, NULL) = 1
recvfrom(6, """, 1, 0, NULL, NULL) = 1
recvfrom(6, "f", 1, 0, NULL, NULL) = 1
recvfrom(6, "7", 1, 0, NULL, NULL) = 1
recvfrom(6, "9", 1, 0, NULL, NULL) = 1
recvfrom(6, "9", 1, 0, NULL, NULL) = 1
recvfrom(6, "7", 1, 0, NULL, NULL) = 1
recvfrom(6, "9", 1, 0, NULL, NULL) = 1
recvfrom(6, "0", 1, 0, NULL, NULL) = 1
recvfrom(6, "4", 1, 0, NULL, NULL) = 1
recvfrom(6, "0", 1, 0, NULL, NULL) = 1
recvfrom(6, "6", 1, 0, NULL, NULL) = 1
recvfrom(6, "c", 1, 0, NULL, NULL) = 1
recvfrom(6, "2", 1, 0, NULL, NULL) = 1
recvfrom(6, "a", 1, 0, NULL, NULL) = 1
recvfrom(6, "3", 1, 0, NULL, NULL) = 1
recvfrom(6, "e", 1, 0, NULL, NULL) = 1
recvfrom(6, "e", 1, 0, NULL, NULL) = 1
recvfrom(6, "2", 1, 0, NULL, NULL) = 1
recvfrom(6, "9", 1, 0, NULL, NULL) = 1
recvfrom(6, "c", 1, 0, NULL, NULL) = 1
recvfrom(6, "6", 1, 0, NULL, NULL) = 1
recvfrom(6, "4", 1, 0, NULL, NULL) = 1
recvfrom(6, "9", 1, 0, NULL, NULL) = 1
recvfrom(6, "f", 1, 0, NULL, NULL) = 1
recvfrom(6, "7", 1, 0, NULL, NULL) = 1
recvfrom(6, "0", 1, 0, NULL, NULL) = 1
recvfrom(6, "f", 1, 0, NULL, NULL) = 1
recvfrom(6, "d", 1, 0, NULL, NULL) = 1
recvfrom(6, "1", 1, 0, NULL, NULL) = 1
recvfrom(6, "8", 1, 0, NULL, NULL) = 1
recvfrom(6, "3", 1, 0, NULL, NULL) = 1
recvfrom(6, "4", 1, 0, NULL, NULL) = 1
recvfrom(6, "9", 1, 0, NULL, NULL) = 1
recvfrom(6, """, 1, 0, NULL, NULL) = 1
recvfrom(6, "\r", 1, 0, NULL, NULL) = 1
recvfrom(6, "\n", 1, 0, NULL, NULL) = 1
recvfrom(6, "X", 1, 0, NULL, NULL) = 1
recvfrom(6, "-", 1, 0, NULL, NULL) = 1
recvfrom(6, "T", 1, 0, NULL, NULL) = 1
recvfrom(6, "r", 1, 0, NULL, NULL) = 1
recvfrom(6, "a", 1, 0, NULL, NULL) = 1
recvfrom(6, "n", 1, 0, NULL, NULL) = 1
recvfrom(6, "s", 1, 0, NULL, NULL) = 1
recvfrom(6, "-", 1, 0, NULL, NULL) = 1
recvfrom(6, "I", 1, 0, NULL, NULL) = 1
recvfrom(6, "d", 1, 0, NULL, NULL) = 1
recvfrom(6, ":", 1, 0, NULL, NULL) = 1
recvfrom(6, " ", 1, 0, NULL, NULL) = 1
recvfrom(6, "t", 1, 0, NULL, NULL) = 1
recvfrom(6, "x", 1, 0, NULL, NULL) = 1
recvfrom(6, "a", 1, 0, NULL, NULL) = 1
recvfrom(6, "9", 1, 0, NULL, NULL) = 1
recvfrom(6, "8", 1, 0, NULL, NULL) = 1
recvfrom(6, "1", 1, 0, NULL, NULL) = 1
recvfrom(6, "a", 1, 0, NULL, NULL) = 1
recvfrom(6, "6", 1, 0, NULL, NULL) = 1
recvfrom(6, "b", 1, 0, NULL, NULL) = 1
recvfrom(6, "c", 1, 0, NULL, NULL) = 1
recvfrom(6, "6", 1, 0, NULL, NULL) = 1
recvfrom(6, "1", 1, 0, NULL, NULL) = 1
recvfrom(6, "1", 1, 0, NULL, NULL) = 1
recvfrom(6, "d", 1, 0, NULL, NULL) = 1
recvfrom(6, "4", 1, 0, NULL, NULL) = 1
recvfrom(6, "6", 1, 0, NULL, NULL) = 1
recvfrom(6, "c", 1, 0, NULL, NULL) = 1
recvfrom(6, "4", 1, 0, NULL, NULL) = 1
recvfrom(6, "b", 1, 0, NULL, NULL) = 1
recvfrom(6, "4", 1, 0, NULL, NULL) = 1
recvfrom(6, "b", 1, 0, NULL, NULL) = 1
recvfrom(6, "3", 1, 0, NULL, NULL) = 1
recvfrom(6, "e", 1, 0, NULL, NULL) = 1
recvfrom(6, "b", 1, 0, NULL, NULL) = 1
recvfrom(6, "4", 1, 0, NULL, NULL) = 1
recvfrom(6, "8", 1, 0, NULL, NULL) = 1
recvfrom(6, "8", 1, 0, NULL, NULL) = 1
recvfrom(6, "b", 1, 0, NULL, NULL) = 1
recvfrom(6, "6", 1, 0, NULL, NULL) = 1
recvfrom(6, "8", 1, 0, NULL, NULL) = 1
recvfrom(6, "e", 1, 0, NULL, NULL) = 1
recvfrom(6, "b", 1, 0, NULL, NULL) = 1
recvfrom(6, "f", 1, 0, NULL, NULL) = 1
recvfrom(6, "6", 1, 0, NULL, NULL) = 1
recvfrom(6, "\r", 1, 0, NULL, NULL) = 1
recvfrom(6, "\n", 1, 0, NULL, NULL) = 1
recvfrom(6, "D", 1, 0, NULL, NULL) = 1
recvfrom(6, "a", 1, 0, NULL, NULL) = 1
recvfrom(6, "t", 1, 0, NULL, NULL) = 1
recvfrom(6, "e", 1, 0, NULL, NULL) = 1
recvfrom(6, ":", 1, 0, NULL, NULL) = 1
recvfrom(6, " ", 1, 0, NULL, NULL) = 1
recvfrom(6, "F", 1, 0, NULL, NULL) = 1
recvfrom(6, "r", 1, 0, NULL, NULL) = 1
recvfrom(6, "i", 1, 0, NULL, NULL) = 1
recvfrom(6, ",", 1, 0, NULL, NULL) = 1
recvfrom(6, " ", 1, 0, NULL, NULL) = 1
recvfrom(6, "0", 1, 0, NULL, NULL) = 1
recvfrom(6, "9", 1, 0, NULL, NULL) = 1
recvfrom(6, " ", 1, 0, NULL, NULL) = 1
recvfrom(6, "N", 1, 0, NULL, NULL) = 1
recvfrom(6, "o", 1, 0, NULL, NULL) = 1
recvfrom(6, "v", 1, 0, NULL, NULL) = 1
recvfrom(6, " ", 1, 0, NULL, NULL) = 1
recvfrom(6, "2", 1, 0, NULL, NULL) = 1
recvfrom(6, "0", 1, 0, NULL, NULL) = 1
recvfrom(6, "1", 1, 0, NULL, NULL) = 1
recvfrom(6, "2", 1, 0, NULL, NULL) = 1
recvfrom(6, " ", 1, 0, NULL, NULL) = 1
recvfrom(6, "2", 1, 0, NULL, NULL) = 1
recvfrom(6, "1", 1, 0, NULL, NULL) = 1
recvfrom(6, ":", 1, 0, NULL, NULL) = 1
recvfrom(6, "0", 1, 0, NULL, NULL) = 1
recvfrom(6, "1", 1, 0, NULL, NULL) = 1
recvfrom(6, ":", 1, 0, NULL, NULL) = 1
recvfrom(6, "4", 1, 0, NULL, NULL) = 1
recvfrom(6, "5", 1, 0, NULL, NULL) = 1
recvfrom(6, " ", 1, 0, NULL, NULL) = 1
recvfrom(6, "G", 1, 0, NULL, NULL) = 1
recvfrom(6, "M", 1, 0, NULL, NULL) = 1
recvfrom(6, "T", 1, 0, NULL, NULL) = 1
recvfrom(6, "\r", 1, 0, NULL, NULL) = 1
recvfrom(6, "\n", 1, 0, NULL, NULL) = 1
recvfrom(6, "\r", 1, 0, NULL, NULL) = 1
recvfrom(6, "\n", 1, 0, NULL, NULL) = 1

BROKEN CASE:
Using swift 1.7.6 (latest git pull) + swift3 (latest git pull)

Strace:
recvfrom(6, "<", 1, 0, NULL, NULL) = 1
recvfrom(6, "?", 1, 0, NULL, NULL) = 1
recvfrom(6, "x", 1, 0, NULL, NULL) = 1
recvfrom(6, "m", 1, 0, NULL, NULL) = 1
recvfrom(6, "l", 1, 0, NULL, NULL) = 1
recvfrom(6, " ", 1, 0, NULL, NULL) = 1
recvfrom(6, "v", 1, 0, NULL, NULL) = 1
recvfrom(6, "e", 1, 0, NULL, NULL) = 1
recvfrom(6, "r", 1, 0, NULL, NULL) = 1
recvfrom(6, "s", 1, 0, NULL, NULL) = 1
recvfrom(6, "i", 1, 0, NULL, NULL) = 1
recvfrom(6, "o", 1, 0, NULL, NULL) = 1
recvfrom(6, "n", 1, 0, NULL, NULL) = 1
recvfrom(6, "=", 1, 0, NULL, NULL) = 1
recvfrom(6, """, 1, 0, NULL, NULL) = 1
recvfrom(6, "1", 1, 0, NULL, NULL) = 1
recvfrom(6, ".", 1, 0, NULL, NULL) = 1
recvfrom(6, "0", 1, 0, NULL, NULL) = 1
recvfrom(6, """, 1, 0, NULL, NULL) = 1
recvfrom(6, " ", 1, 0, NULL, NULL) = 1
recvfrom(6, "e", 1, 0, NULL, NULL) = 1
recvfrom(6, "n", 1, 0, NULL, NULL) = 1
recvfrom(6, "c", 1, 0, NULL, NULL) = 1
recvfrom(6, "o", 1, 0, NULL, NULL) = 1
recvfrom(6, "d", 1, 0, NULL, NULL) = 1
recvfrom(6, "i", 1, 0, NULL, NULL) = 1
recvfrom(6, "n", 1, 0, NULL, NULL) = 1
recvfrom(6, "g", 1, 0, NULL, NULL) = 1
recvfrom(6, "=", 1, 0, NULL, NULL) = 1
recvfrom(6, """, 1, 0, NULL, NULL) = 1
recvfrom(6, "U", 1, 0, NULL, NULL) = 1
recvfrom(6, "T", 1, 0, NULL, NULL) = 1
recvfrom(6, "F", 1, 0, NULL, NULL) = 1
recvfrom(6, "-", 1, 0, NULL, NULL) = 1
recvfrom(6, "8", 1, 0, NULL, NULL) = 1
recvfrom(6, """, 1, 0, NULL, NULL) = 1
recvfrom(6, "?", 1, 0, NULL, NULL) = 1
recvfrom(6, ">", 1, 0, NULL, NULL) = 1
recvfrom(6, "\r", 1, 0, NULL, NULL) = 1
recvfrom(6, "\n", 1, 0, NULL, NULL) = 1
recvfrom(6, "\r\n NoSuchKey\r\n The resource you requested does not exist\r\n\r\n", 8192, 0, NULL, NULL) = 109
recvfrom(6, "HTTP/1.1 200 OK\r\nEtag: "a5737b361dab736fcc8f1c0bb66abe17"\r\nContent-Type: text/html; charset=UTF-8\r\nX-Trans-Id: tx58e5d25c2ab94a32bb701e97073d1c75\r\nContent-Length: 0\r\nDate: Fri, 09 Nov 2012 20:38:23 GMT\r\n\r\n", 8192, 0, NULL, NULL) = 205
recvfrom(6,

Thanks for any help!

In the GETorHEAD() method I found that most HEAD requests do not have a body when we execute this code:

  if env['REQUEST_METHOD'] == 'HEAD':
    app_iter = None

  # then later:
  if is_success(status):
    return Response(status=status, headers=new_hdrs, app_iter=app_iter)

But when there are errors (such as a key-not-found 404), we return the result of a call to get_err_response. This method always returns a body:

    resp = Response(content_type='text/xml')
    resp.status = error_table[code][0]
    resp.body = """%s
        <Error><Code>%s</Code><Message>%s</Message></Error>
    """ % (XML_HEADER, code, error_table[code][1])
    return resp

HEAD requests must not return a body or we're breaking the HTTP spec. This manifested for me when I did a HEAD followed by a PUT request over the same HTTP connection. The 2nd request fails horribly because there's an unexpected body still on the wire. I think it's easy to fix, just make sure that the get_err_response method does not add a body if the original request is a HEAD.

The presigned URL expiration time check is incorrect. It currently only allows for URLs which are within 5 minutes of the expiration date. It should allow any request that hits before the expiration time + 5 minutes. I will submit a fix shortly.

The readme should be updated to reflect the need for s3token in the pipeline if using keystone; otherwise, new users are going to be very confused, as was I, about why they get nothing but access denied when attempting to use swift3.

Swift3 does not take care of HTTP 403 error responses when performing subsequent
requests to swift and return in that case the default InvalidURI error to client.
In some cases this can confuse the user, for instance when ACL is configured on a container the keystoneauth middleware return a 403 error but swift3 return InvalidURI to user. I join a pull request with this issue.

Regards

Hi everyone,
When I create an object with a name containing an "URL special character", I won't be able to list that object correctly.
ex:

• I create two object named "my name" and "my%20name", those are the finale names I want, which means one containing a space and the other having "%20" in his name.
  For "my%20name" I will escape the "%" character using "%25".

like so:

curl -i http://10.0.2.15:8080/v1/AUTH_427a792847c64eb89be0d5921e038ea9/mycontainer/my%20name -X PUT -H "X-Auth-Token: $T" --data-binary "golgoth"

curl -i http://10.0.2.15:8080/v1/AUTH_427a792847c64eb89be0d5921e038ea9/mycontainer/my%2520name -X PUT -H "X-Auth-Token: $T" --data-binary "another data"

That works just fine, and I can list my objects correctly using swift API

stack@devstack:/opt/stack/swift3$ curl -i http://10.0.2.15:8080/v1/AUTH_427a792847c64eb89be0d5921e038ea9/mycontainer/ -X GET -H "X-Auth-Token: $T" 
HTTP/1.1 200 OK
Content-Length: 18
X-Container-Object-Count: 2
Accept-Ranges: bytes
X-Timestamp: 1392055737.21633
X-Container-Bytes-Used: 14
Content-Type: text/plain; charset=utf-8
X-Trans-Id: txf383f7b290c04cfe871ad-0052f91899
Date: Mon, 10 Feb 2014 18:21:13 GMT

my name
my%20name << this is the finale name I want, so it's OK.

But when I list the container using S3 (boto lib) the name returned will be "my name"
for both object.

ex:

... 
bucket = conn.get_bucket('mycontainer')
for b in bucket.list():
    print b.name

will return

[nacim@grindizer py (master)]$ python /tmp/test.py 
my name
my name

This is not a printing problem nor a boto error, I 've checked and it seems related to this commit 9587a76 where objects name are "unquote" in listing result before returned.

The fact that this problem comes from a precious fix, is a little bite confusing to me, I don"t think this should be the normal behavior but may be I'm missing something ? (this is why I wanted to fill this bug report before proposing any fix by the way !)

Nacim.

Moving https://bugs.launchpad.net/swift/+bug/973516 to here:

I ran the s3 test suites :

https://github.com/ceph/s3-tests

against swift + s3 (with keystone auth but this should be exactly the same for the other auth).

This is the results :

Ran 250 tests in 129.963s
FAILED (errors=61, failures=62)

and this is the full output :

http://sprunge.us/cECP

there is quite a bit to fix but at least it's a start to document the incompatibilities.

We are running Ubuntu CA Folsom with Keystone, Swift and swift3. Many of our users connect via python-swift but quite a few also use S3 API. We periodically see one of our S3 users (not always the same one) suddenly start getting 403s for all requests. After much troubleshooting we have isolated it to the fact that something in memcached (the one for Keystone) is getting messed up.

If we restart memcached the issue goes away. Has anyone else experienced this?

Thanks,
Graham

In invoking AWS S3 java API against Swift, we failed with the following NPE
java.lang.NullPointerException
at com.amazonaws.util.BinaryUtils.fromHex(BinaryUtils.java:69)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1140)
at com.cloud.utils.S3Utils.putObject(S3Utils.java:178)
After investigation, we realized that Amazon S3 expects that returned metadata should have "ETag" as the metadata key, but the metadata returned from Swift3 has "Etag" as the key, thus causing this NPE. Here is the returned metadata from Swift3:

{Content-Length=0, Etag="5e714348185ffe355a76b754f79176d6", X-Trans-Id=tx1b475494fc1e41049a8be-00530fc274, Content-Type=text/html; charset=UTF-8, Connection=keep-alive}

This project has been moved to openstack/swift3 please put deprecation notice with the link to the new upstream repository in the description.

Hello,

Swift3 is the only external repository that is not inside openstack infra deployed by devstack. It would be nice if we could consider moving this repository to stackforge (i'll gladly helping you doing that).

Cheers,
Chmouel.

I will report two bug about swift3 middleware.

I created container TEST1 and test1

date

Wed May 14 16:05:56 JST 2014

swift post TEST

swift post test

swift list -l

 0 0 2014-05-14 07:06:33 TEST
 0 0 2014-05-14 07:06:37 test

and then, using s3cmd listed container.

s3cmd ls

2009-02-03 16:45 s3://TEST
2009-02-03 16:45 s3://test

Container creating date is shown 2009-02-03.
This is one bug.

In next, using s3cmd file upload to TEST container

s3cmd put file1 s3://TEST

file1 -> s3://TEST/file1 [1 of 1]
801 of 801 100% in 6s 123.01 B/s done

this is no problem
but using s3cmd file update to test container, this failed.

s3cmd put file1 s3://test

file1 -> s3://test/file1 [1 of 1]
801 of 801 100% in 0s 49.17 kB/s done
ERROR: S3 error: 403 (AccessDenied): Access denied

In case of lower case container, when i put using s3cmd, access deined error occured.

Seeing a few of these in the proxy.err.log:

swift 'generator' object is not callable: #012Traceback (most recent call last):#012  File "/usr/local/lib/python2.7/dist-packages/swift3-1.7.0-py2.7.egg/swift3/middleware.py", line 874, in __call__#012    return self.handle_request(env, start_response)#012  File "/usr/local/lib/python2.7/dist-packages/swift3-1.7.0-py2.7.egg/swift3/middleware.py", line 954, in handle_request#012    return res(env, start_response)#012TypeError: 'generator' object is not callable (txn: tx0f4151105d054da8a164d-0053d6d050hostname04.proc.dal.moz.com) (client_ip: 10.XX.XX.XX)

Does this middleware allows to set the container quota?

I recently did a fresh in stall of Swift 1.4.8 on 12.04 LTS. The normal swift cli works fine, but as soon as I change the proxy-server.conf to include swift3 in the pipeline I get the error from the issue title. Here is the full stack trace:

root@cloudblock1:/etc/swift# swift-init proxy-server start
Starting proxy-server...(/etc/swift/proxy-server.conf)
Traceback (most recent call last):
File "/usr/bin/swift-proxy-server", line 22, in
run_wsgi(conf_file, 'proxy-server', default_port=8080, *_options)
File "/usr/lib/python2.7/dist-packages/swift/common/wsgi.py", line 122, in run_wsgi
loadapp('config:%s' % conf_file, global_conf={'log_name': log_name})
File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
return loadobj(APP, uri, name=name, *_kw)
File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 271, in loadobj
global_conf=global_conf)
File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext
global_conf=global_conf)
File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 320, in _loadconfig
return loader.get_context(object_type, name, global_conf)
File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 450, in get_context
global_additions=global_additions)
File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 562, in _pipeline_app_context
for name in pipeline[:-1]]
File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 454, in get_context
section)
File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 476, in _context_from_use
object_type, name=use, global_conf=global_conf)
File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 406, in get_context
global_conf=global_conf)
File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext
global_conf=global_conf)
File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 328, in _loadegg
return loader.get_context(object_type, name, global_conf)
File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 620, in get_context
object_type, name=name)
File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 646, in find_egg_entry_point
possible.append((entry.load(), protocol, entry.name))
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1989, in load
entry = import(self.module_name, globals(),globals(), ['name'])
File "/usr/local/lib/python2.7/dist-packages/swift3-1.0.0-py2.7.egg/swift3/middleware.py", line 64, in
from swift.common.wsgi import WSGIContext
ImportError: cannot import name WSGIContext

And here is my proxy-server.conf:

[DEFAULT]
bind_port = 8080
user = swift

[pipeline:main]
pipeline = catch_errors healthcheck cache swift3 s3token authtoken keystone proxy-server

[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true
account_autocreate = true

[filter:keystone]
paste.filter_factory = keystone.middleware.swift_auth:filter_factory
operator_roles = admin, swiftoperator

[filter:authtoken]
paste.filter_factory = keystone.middleware.auth_token:filter_factory
delay_auth_decision = 1
auth_port = 35357
auth_host = 192.168.50.249
auth_protocol = http
admin_tenant_name = service
admin_user = swift
admin_password = swift

[filter:cache]
use = egg:swift#memcache
set log_name = cache
memcache_servers = 10.10.10.7:11211,10.10.10.8:11211;10.10.10.4:11211

[filter:catch_errors]
use = egg:swift#catch_errors

[filter:healthcheck]
use = egg:swift#healthcheck

[filter:s3token]
paste.filter_factory = keystone.middleware.s3_token:filter_factory
auth_port = 35357
auth_host = 192.168.50.249
auth_protocol = http
admin_tenant_name = service
admin_user = swift
admin_password = swift

[filter:swift3]
use = egg:swift3#swift3

Any help would be great.

The canonicalize_string function is broken, it does not properly preserve parameters and does not have the correct keywords list.

What mechanism is required for the deletion of a Large Object using a direct API?
(i.e. http://docs.openstack.org/developer/swift/overview_large_objects.html)
Thanks
David