humancellatlas / cloud-blobstore Goto Github PK

Unfortunately GCP does not provide straightforward timeout/retry configurations analogous to boto.

However, google.cloud.storage requests are forwarded all the way to the stdlib method requests.Session().request, which accepts a timeout keyword argument. Usage would look like:

import google.auth
import google.auth.transport.requests

connect_timeout = 1.0
read_timeout = 1.0
class Session(google.auth.transport.requests.AuthorizedSession):
    def request(self, *args, **kwargs):
        kwargs['timeout'] = (connect_timeout, read_timeout)
        return super().request(*args, **kwargs)

credentials, _ = google.auth.default()
gcp_client = Client(
    _http=Session(credentials)
)

The problem with this approach is Google may pull the rug out from under us by changing the _http argument:

:param _http: (Optional) HTTP object to make requests. Can be any object
that defines request() with the same interface as
:meth:requests.Session.request. If not passed, an
_http object is created that is bound to the
credentials for the current object.
This parameter should be considered private, and could
change in the future.

Things are a bit more promising for retry behavior, which appears configurable through a global:

import google.resumable_media.common
google.resumable_media.common.MAX_CUMULATIVE_RETRY = 2

@ttung thoughts?

I would like to able to add meta-data associated with the file handle when upload to a blobstore. For example the ExtraArgs keyword in s3_client.upload_fileobj can take dictionary and will add to the files metadata.

The following functions were added to support S3 checkout service.
There were initially implemented for S3 only.

• def check_bucket_exists(self, bucket: str) -> bool:
• def get_bucket_region(self, bucket) -> str: