humancellatlas / cloud-blobstore Goto Github PK
View Code? Open in Web Editor NEWSimple API that abstracts out differences between the blobstores across different cloud providers.
License: MIT License
Simple API that abstracts out differences between the blobstores across different cloud providers.
License: MIT License
Unfortunately GCP does not provide straightforward timeout/retry configurations analogous to boto.
However, google.cloud.storage
requests are forwarded all the way to the stdlib method requests.Session().request
, which accepts a timeout
keyword argument. Usage would look like:
import google.auth
import google.auth.transport.requests
connect_timeout = 1.0
read_timeout = 1.0
class Session(google.auth.transport.requests.AuthorizedSession):
def request(self, *args, **kwargs):
kwargs['timeout'] = (connect_timeout, read_timeout)
return super().request(*args, **kwargs)
credentials, _ = google.auth.default()
gcp_client = Client(
_http=Session(credentials)
)
The problem with this approach is Google may pull the rug out from under us by changing the _http
argument:
:param _http: (Optional) HTTP object to make requests. Can be any object
that definesrequest()
with the same interface as
:meth:requests.Session.request
. If not passed, an
_http
object is created that is bound to the
credentials
for the current object.
This parameter should be considered private, and could
change in the future.
Things are a bit more promising for retry behavior, which appears configurable through a global:
import google.resumable_media.common
google.resumable_media.common.MAX_CUMULATIVE_RETRY = 2
@ttung thoughts?
I would like to able to add meta-data associated with the file handle when upload to a blobstore. For example the ExtraArgs
keyword in s3_client.upload_fileobj
can take dictionary and will add to the files metadata.
The following functions were added to support S3 checkout service.
There were initially implemented for S3 only.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.