Coder Social home page Coder Social logo

undocumented-s3-apis's Introduction

Undocumented S3 APIs

Undocumented Amazon S3 APIs and third-party extensions. This documents features outside the official Amazon S3 REST APIs. Note that these are not consistently supported across implementations.

Amazon APIs

GET object by multipart number

Object metadata exposes the number of multipart upload parts via the x-amz-mp-parts-count header. Subsequent GET object requests can retrieve individual parts via the partNumber parameter. This allows retrieving individual multipart upload parts, including the original ETag.

HEAD bucket exposes region

HEAD bucket exposes the region via x-amz-bucket-region even if the request lacks authentication. This allows clients to interrogate a bucket so that they can route subsequent requests to the correct endpoint.

Multipart Upload ETag

Single part upload ETag are the MD5 hash of the object content. Multipart upload ETag concatenate the MD5 of each part, MD5 this data, and append a hyphen and the number of parts.

Third-party APIs

GET bucket unordered

allow-unordered=true parameter overrides key sorting which allows higher performance with large buckets. Cannot be used with delimiter.

HEAD bucket extended

Returns bucket statistics via x-rgw-bytes-used and x-rgw-object-count headers.

PUT object append

Append to an existing object at a given offset via the append and position parameters. Responses include x-oss-next-append-position and x-rgw-next-append-position which indicate the next valid append offset.

PUT object conditional

If-Match and If-None-Match (Ceph) and x-goog-if-generation-match and x-goog-if-metageneration-match (Google) headers allow atomic replacement of objects.

Resumable uploads

Initiate a resumable upload with POST object with the x-goog-resumable header. Use the response Location to issue a PUT object with given Content-Length and Content-Range headers. Continuing issuing these requests to build larger objects.

Small file archives

Allows interaction with ZIP file contents as if they were objects. HeadObject, GetObject, and ListObjectsV2 support these operations when the header x-minio-extract has a true value.

References

undocumented-s3-apis's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

undocumented-s3-apis's Issues

Also capture S3 specification inconsistency?

Hi!

I work heavily on Ceph/RGW's S3 interface for DigitalOcean. The last two years at the Ceph conference (Cephalocon) I've given examples of how the AWS S3 specification doesn't match various things, including the AWS S3 Examples, the actual behavior of AWS S3, and the behavior of non-AWS S3 implementations.

Do you think more documentation of such would have a good home here?

As examples:

  • AWS S3 signature calculation is done twice, this happens in both AWS S3 implementation, and Ceph, once with a quirk for a Boto bug, and once without the quirk: ceph/ceph#17009
  • The Bucket Lifecycle XML schema has changed over time.
  • The "List Multipart Uploads" XML response behavior & documentation are inconsistent regarding the <Prefix> vs <CommonPrefixes.Prefix>. https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadListMPUpload.html

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.