Coder Social home page Coder Social logo

omarryhan / aiogoogle Goto Github PK

View Code? Open in Web Editor NEW
177.0 4.0 38.0 5.12 MB

Async Google API Client + Async Google Auth

Home Page: https://aiogoogle.readthedocs.io/en/latest/

License: MIT License

Python 99.89% Shell 0.11%
async google-auth google-authentication discovery-service google-api google-cloud python-asyncio

aiogoogle's People

Contributors

ajax-kovalchuk-d avatar alanbriolat avatar astreatss avatar astroanax avatar bebopchrome avatar bobronium avatar dependabot[bot] avatar erangalon avatar fredo838 avatar gitznik avatar jgayfer avatar jignesh-crest avatar jupiterbjy avatar konstantanxiety avatar lucasknode avatar luciism avatar lukasthaler avatar nabilmostafa avatar neel004 avatar oliver-ni avatar omarryhan avatar phil305 avatar pluieelectrique avatar pyryjook avatar rocha avatar ryanlewist avatar sean-purcell avatar shmookoff avatar thanegill avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

aiogoogle's Issues

google sheets api batchGet only accepts a single range

_sheetServiceAsync = await aiogoogle.discover("sheets", "v4")
resp = await aiogoogle.as_service_account(_sheetServiceAsync .spreadsheets.values.batchGet(spreadsheetId=id, ranges=ranges))

value of ranges should be a list ie ["'DARK'!A4:I83", "'OHW'!A4:I83", "'JESOS'!A4:I83"]
ranges only accepts a single string range making this no different than get.

feature request: support for in memory upload/download media

Thanks a lot for this great library!

I would like to use it in a tool that performs automated backups of autodesk bim360 files (but it could be any other storage service) to a google drive folder.

As of now, the only way to use the drive method files.create() is to specify the upload_file parameter as the file path, but I would like to use a file object (the one downloaded from the other service) without writing it to memory first.

I see that the fire_request coroutine in AiohttpSession.send does the reading of the file via aiofiles.open, and then passes the file object (or MultipartWriter object) to self.request.

Do you think it could be a good addition to support my use case?

If so, I could try to make a PR for that, but I would like to know your thoughts on this (and tips on how to accomplish this in a clean way).

Chat API {name=spaces/*} and unquoting issue

How to reproduce:

creds = ServiceAccountCreds(scopes=["https://www.googleapis.com/auth/chat.bot"])
...
bot = await _client.discover("chat", "v1")
spaces = await _client.as_service_account(bot.spaces.list())

{'spaces': [{'name': 'spaces/_kWXNwAAAAE', 'type': 'DM', 'singleUserBotDm': True, 'threaded': False, 'displayName': ''}], 'nextPageToken': ''}

await _client.as_service_account(bot.spaces.get("spaces/_kWXNwAAAAE"))

aiogoogle.excs.HTTPError:

Not Found

Request URL:
https://chat.googleapis.com/v1/spaces%2F_kWXNwAAAAE

The URL doesn't exist because '/' is getting encoded as '%2F' though 'spaces/*' is a valid url param (reference).

Potential fix?

I suspect the issue starts at resource.py#L599:

sorted_required_path_params[k] = quote_plus(str(v))

dict_values(['spaces%2F_kWXNwAAAAE'])

Perhaps urllib.quote or quote_plus(string, safe='/') would fix it? For reference:

sorted_required_path_params[k] = quote(str(v))

dict_values(['spaces/_kWXNwAAAAE'])

... bot.spaces.get("spaces/_kWXNwAAAAE") ...

{'name': 'spaces/_kWXNwAAAAE', 'type': 'DM', 'singleUserBotDm': True, 'threaded': False, 'displayName': ''}

I didn't test all the use cases though (e.g. #47)

System Info

Thanks for the great lib!

Gmail Message with upload_file doesn't work

Sending an gmail using /upload endpoint fails with either of there two error messages in response:

  1. Reciepient should be provided
  2. application/json can't be used as media type
    I couldn't find any restrictions in google docs but from trial and error it seems like application/json mime should be included always and should be before message/rfc822.
# Fails as `Reciepient` should be defined, happens in aiogoogle if json is None
Mime:
     message/rfc822

# Fails as `application/json` can't be used as media type, happens in aiogoogle if json is not None
Mime:
     message/rfc822
     application/json

# Works always
Mime
    application/json  # can be empty
    message/rfc822

Rough Code:

# works
req = gmail.users.messages.send(userId='me', json={"raw": base_64(mime.as_bytes()).decode()})

# fails Error- 1
file.write(mime.as_bytes())
req = gmail.users.messages.send(userId='me',upload_file=file.path, json={})
req.upload_file_content_type = "message/rfc822"


# fails Error- 2
file.write(mime.as_bytes())
req = gmail.users.messages.send(userId='me',upload_file=file.path, json={"threadId": "some thread id"})
req.upload_file_content_type = "message/rfc822"

For me these changes fixes the issue: nkitsaini@6f3445b. I can make a pull request if the changes seems okay.

ValidationError false positives

Hi,
If I try to run the download_drive_file.py example (modified to use the ServiceAccountCreds) I got a ValidationError:

Invalid instance: "alt" media isn't valid. Expected a value that meets the following criteria: Must be one of the following: "json"

Indeed the drive v3 api spec doesn't list "media" in the "alt" parameter:

    "parameters": {
        "alt": {
            "type": "string",
            "description": "Data format for the response.",
            "default": "json",
            "enum": [
                "json"
            ],
            "enumDescriptions": [
                "Responses with Content-Type of application/json"
            ],
            "location": "query"
        },

I need to set validate parameter to False in order to make it work.
Should we add a special validation case just for this enum?
The thing is the "media" value is only listed in method/parameters descriptions, so it's hard to retrieve this info automatically.

[feature] add auth caching

the original google oauth2 client had token caching to avoid having to re-auth if the token was still valid. Would be awesome if this had it as well

HTTPError quotaExceeded No documentation for handling pagination without generator

Fetching comments to videos from YouTube data API sometimes I can get over million of them, which will always consume all of my daily quota. 
In a situation like this I'm getting HTTPError 403 quotaExceeded error, and after caching raised exception, generator loses its internal state (It's python's generator thing). 
I'm looking for some way to continue fetching comments from place that thrown the exception after quota reset.

Handling repeated values in query parameters

How are repeated query parameters meant to be handled? I am referring to parameters that are marked as "repeated": true in the discovery document, e.g the ranges parameter in spreadsheets.values.batchGet (see below)

"ranges": {
  "description": "The A1 notation of the values to retrieve.",
  "repeated": true,
  "location": "query",
  "type": "string"
},

I couldn't get a query containing multiple of these values to work: passing them as a list of strings triggered a validation error, disabling validation then yielded a bad request from the API. Joining them to a string (via '&ranges='.join(my_ranges) to emulate what would be passed to the URL didn't work either due to URL-encoding. Digging into the library soure code only got me one mention of the "repeated" parameter in the data.py-file. Is there currently no support for repeated query parameters in aiogoogle or am I missing something?

Token expiry check broken in Python 3.6 and lower

I am not entirely sure if this is a bug or if there simply is no intention to support Python 3.6, but given I was able to pip install it into my Python 3.6.6 environment I suspect the former, so here we go:

Line 25 of auth.utils will not work for Python 3.6 and lower, as .fromisoformat was introduced to the datetime module in version 3.7. You'd need to use something like .strptime(expires_at, '%Y-%m-%dT%H:%M:%S.%f') instead for lower versions

I can submit a PR for this if it is deemed a bug indeed

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate

On my macOS I getting this error while using ClientSession with default ssl context. It's known issue of aiohttp

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/sslproto.py", line 526, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/sslproto.py", line 189, in feed_ssldata
    self._sslobj.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 763, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)
SSL handshake failed on verifying the certificate
protocol: <asyncio.sslproto.SSLProtocol object at 0x10c1a56a0>
transport: <_SelectorSocketTransport fd=10 read=polling write=<idle, bufsize=0>>

See reasons here:

aio-libs/aiohttp#2822

Solved here (library level):

dephell/dephell#185

Also for quick solution on user level (without changing library itself) see (macOS only):

https://stackoverflow.com/questions/50236117/scraping-ssl-certificate-verify-failed-error-for-http-en-wikipedia-org

Add support of service account

This library looks really awesome but the one thing that stopped me before using it, it lacks of support service account. Any changes to implement it?

Bug in `authorization_url` `include_granted_scopes` parameter

When include_granted_scopes is not passed in as a parameter, the following error occurs:
Screenshot (444)

This is because when you do json.dumps(include_granted_scopes) it bypasses the if statement.

for param_name, param in {
"client_id": client_creds["client_id"],
"response_type": response_type,
"state": state,
"access_type": access_type,
"include_granted_scopes": json.dumps(include_granted_scopes),
"login_hint": login_hint,
"prompt": prompt,
}.items():
if param is not None:
uri += "&"
uri += parse.urlencode({param_name: param})

Maybe something like this instead:

for param_name, param in {
    "client_id": client_creds["client_id"],
    "response_type": response_type,
    "state": state,
    "access_type": access_type,
    "include_granted_scopes": include_granted_scopes,
    "login_hint": login_hint,
    "prompt": prompt,
}.items():
    if param is not None:
        uri += f"&{parse.urlencode({param_name: json.dumps(param) if isinstance(param, bool) else param})}"

TypeError: 'NoneType' object is not callable while using async for

Executing first example from readme starts output files, but right after first iteration gives me this:

my_file_1.txt
my_other_file.jpeg
...
Traceback (most recent call last):
    ...
  File "/Users/rocky/dev/py/gdrive.py", line 12, in list_files
    async for page in full_res:
  File ".../aiogoogle/models.py", line 290, in _next_page_generator
    async with session_factory() as sess:
TypeError: 'NoneType' object is not callable

Additional info:
Due to some SSL issues, I used AiohttpSession with custom connector through session_factory
Just was able to run it without specifying any custom sessions, get same result

Possible reason:

session_factory = self.__class__ if session_factory is None else None

Proposed solution:

Change line above to:

        session_factory = self.__class__ if session_factory is None else session_factory

Missing URL escape listing calendar events

I'm using aiogoogle 1.1.4.

I have code that lists calendars, then events for each calendar, roughly like this:

calendars = await aiogoogle.as_user(service.calendarList.list())
for calendar in calendars["items"]:
    calendar_id = calendar["id"]
    events = await aiogoogle.as_user(service.events.list(calendarId=calendar_id))

Some calendar IDs have non-URL-safe characters in them. For example, the "Holidays in United States" calendar has the ID "en.usa#[email protected]". If I pass the ID as-is to service.events.list, I get a 404 error. If I first escape the ID like this, then it works as expected:

calendar_id = urllib.parse.quote_plus(calendar["id"])

I expected the library to do the escaping, since it is the one building the URL, rather than requiring it to be done by the caller.

Support Compute Engine default service account

This repo is looking awesome! I know the documentation mentions that
Compute Engine default credentials are not yet supported. I was just
wondering if this will be supported any time soon? I would help, but async is
simply above me. I'll remove the issue as soon as you give an indication (not in
the foreseeable future / I don't know is also an indication). Thanks!

Getting an access token

So I've been looking through the documentation for over an hour now and I still have no idea how you get an access token from an OAuth client. Can someone explain how to do it?

Using this to access Sheets API

Hey there! I found your module and it seems to be exactly what I need. But I can't find how to use it with Sheets API..
Could you explain me how to use it? I also looked at docs and couldn't understand

Is it possible to use your library in same manner?

https://github.com/eshmu/gphotos-upload/blob/master/client_id.json
Here is example of client auth parameters.
So I am reading this config to CLIENT_CREDS dict and creating instance like this:

async with Aiogoogle(client_creds=CLIENT_CREDS) as aiogoogle:
    photos = await aiogoogle.discover('photoslibrary', 'v1')

In doc examples you are using as_user everywhere..

        json_res = await aiogoogle.as_user(
            drive_v3.files.list(),
        )

What should I use in this case.. since I have some kind of client auth. I need to use API in that manner https://github.com/eshmu/gphotos-upload/blob/master/upload.py#L156

Google Sign-in tokens are issued under ISS="accounts.google.com"

It would be nice to use aiogoogle's OpenID connect manager to validate and decode JWT id_tokens issued by Google Sign-in. According to the documentation linked below, tokens can be issued under one of two ISSs: htttps://accounts.google.com or accounts.google.com. Currently aiogoogle only checks for the former.

https://developers.google.com/identity/sign-in/web/backend-auth
https://developers.google.com/identity/protocols/oauth2/openid-connect

double validator fails for int

When trying to use Classroom API I get the following error: 1 isn't valid. Expected a value that meets the following criteria: Double type. My code is

works = await aiogoogle.as_user(
            classroom.courses.courseWork.list(courseId=str(courseId)),
            full_res=True)

Oauth2Manager.refresh does not return refresh_token, causing further refresh calls to fail

Example

async with Aiogoogle(
        client_creds = ClientCreds(client_id = GOOGLE_CLIENT_ID, client_secret = GOOGLE_CLIENT_SECRET),
        user_creds = UserCreds(refresh_token = GOOGLE_REFRESH_TOKEN)
    ) as google:
        youtube = await google.discover('youtube', 'v3')

        while True:
            data = await google.as_user(
                youtube.subscriptions.list(mine=True, part='snippet', maxResults=50),
            )

            await asyncio.sleep(1000 * 10)

The initial call to youtube.subscriptions.list will use the refresh_token to get an access_token, after which google.user_creds will look like:

{'access_token': '<REDACTED>', 'refresh_token': None, 'expires_in': 3599, 'expires_at': '2021-02-27T23:37:29.343504', 'scopes': ['https://www.googleapis.com/auth/youtube.readonly'], 'id_token': None, 'id_token_jwt': None, 'token_type': 'Bearer', 'token_uri': 'https://oauth2.googleapis.com/token', 'token_info_uri': 'https://www.googleapis.com/oauth2/v4/tokeninfo', 'revoke_uri': 'https://oauth2.googleapis.com/revoke'}

The refresh_token has been set to none, so after ~30 minutes when the access_token expires, and Oauth2Manager.refresh tries to refresh the token, it throws the exception:

Traceback (most recent call last):
  File "/PROJECT/app.py", line 147, in youtube
    async for video in self.get_videos(channel):
  File "/PROJECT/app.py", line 126, in get_videos
    data = await self.google.as_user(
  File "/PROJECT-VENV/lib/python3.8/site-packages/aiogoogle/client.py", line 227, in as_user
    user_creds = await self.oauth2.refresh(
  File "/PROJECT-VENV/lib/python3.8/site-packages/aiogoogle/auth/managers.py", line 625, in refresh
    json_res = await self._send_request(request)
  File "/PROJECT-VENV/lib/python3.8/site-packages/aiogoogle/auth/managers.py", line 180, in _send_request
    res = await sess.send(req)
  File "/PROJECT-VENV/lib/python3.8/site-packages/aiogoogle/sessions/aiohttp_session.py", line 185, in send
    results = await schedule_tasks()
  File "/PROJECT-VENV/lib/python3.8/site-packages/aiogoogle/sessions/aiohttp_session.py", line 177, in schedule_tasks
    return await asyncio.gather(*tasks, return_exceptions=False)
  File "/PROJECT-VENV/lib/python3.8/site-packages/aiogoogle/sessions/aiohttp_session.py", line 163, in get_content
    response = await get_response(request)
  File "/PROJECT-VENV/lib/python3.8/site-packages/aiogoogle/sessions/aiohttp_session.py", line 159, in get_response
    response.raise_for_status()
  File "/PROJECT-VENV/lib/python3.8/site-packages/aiogoogle/models.py", line 406, in raise_for_status
    raise HTTPError(msg=self.reason, req=self.req, res=self)
aiogoogle.excs.HTTPError: 



Bad Request

Content:
'invalid_grant'

Request URL:
https://oauth2.googleapis.com/token

My workaround has been to set

google.user_creds.refresh_token = GOOGLE_REFRESH_TOKEN

But I don't see why this couldn't be handled internally.

Thanks!

TypeError when requesting youtube.channels.list(part='statistics')

here is my test code

async def create_api():
    async with Aiogoogle(api_key=my_api_key) as google:
        youtube = await google.discover('youtube', 'v3')
        result = await google.as_api_key(
            youtube.channels.list(
                part='snippet',
                id='UCwRKt_raV3N5KZgxcFyC1vw'
            )
        )
        print(result)
        result = await google.as_api_key(
            youtube.channels.list(
                part='statistics',
                id='UCwRKt_raV3N5KZgxcFyC1vw'
            )
        )
        print(result)

For the first request, I can get the result succesfully
While for the second one, I get the following information:

Traceback (most recent call last):
File "testt.py", line 24, in
create_api()
File "C:\Users\xybri\AppData\Local\Programs\Python\Python37\lib\asyncio\runners.py", line 43, in run
return loop.run_until_complete(main)
File "C:\Users\xybri\AppData\Local\Programs\Python\Python37\lib\asyncio\base_events.py", line 584, in run_until_complete
return future.result()
File "testt.py", line 17, in create_api
id='UCwRKt_raV3N5KZgxcFyC1vw'
File "C:\Users\xybri\AppData\Local\Programs\Python\Python37\lib\site-packages\aiogoogle\client.py", line 255, in as_api_key
return await self.active_session.send(*authorized_requests, timeout=timeout, full_res=full_res, session_factory=self.session_factory)
File "C:\Users\xybri\AppData\Local\Programs\Python\Python37\lib\site-packages\aiogoogle\sessions\aiohttp_session.py", line 153, in send
results = await schedule_tasks()
File "C:\Users\xybri\AppData\Local\Programs\Python\Python37\lib\site-packages\aiogoogle\sessions\aiohttp_session.py", line 144, in schedule_tasks
return await asyncio.gather(tasks, return_exceptions=False)
File "C:\Users\xybri\AppData\Local\Programs\Python\Python37\lib\site-packages\aiogoogle\sessions\aiohttp_session.py", line 135, in get_content
response = await get_response(request)
File "C:\Users\xybri\AppData\Local\Programs\Python\Python37\lib\site-packages\aiogoogle\sessions\aiohttp_session.py", line 132, in get_response
response = _call_callback(request, response)
File "C:\Users\xybri\AppData\Local\Programs\Python\Python37\lib\site-packages\aiogoogle\sessions\common.py", line 5, in _call_callback
response.json = request.callback(response.content)
File "C:\Users\xybri\AppData\Local\Programs\Python\Python37\lib\site-packages\aiogoogle\resource.py", line 401, in
callback=lambda res: self._validate_response(res, validate)
File "C:\Users\xybri\AppData\Local\Programs\Python\Python37\lib\site-packages\aiogoogle\resource.py", line 505, in _validate_response
self.validate(res, response_schema, schema_name)
File "C:\Users\xybri\AppData\Local\Programs\Python\Python37\lib\site-packages\aiogoogle\resource.py", line 242, in validate
return validate
(instance, schema, self.schemas, schema_name)
File "C:\Users\xybri\AppData\Local\Programs\Python\Python37\lib\site-packages\aiogoogle\validate.py", line 337, in validate
validate_object()
File "C:\Users\xybri\AppData\Local\Programs\Python\Python37\lib\site-packages\aiogoogle\validate.py", line 317, in validate_object
validate(instance[k], v, schemas, schema_name=k)
File "C:\Users\xybri\AppData\Local\Programs\Python\Python37\lib\site-packages\aiogoogle\validate.py", line 341, in validate
validate_array()
File "C:\Users\xybri\AppData\Local\Programs\Python\Python37\lib\site-packages\aiogoogle\validate.py", line 324, in validate_array
validate(item, schema
, schemas, schema_name=schema_name)
File "C:\Users\xybri\AppData\Local\Programs\Python\Python37\lib\site-packages\aiogoogle\validate.py", line 337, in validate
validate_object()
File "C:\Users\xybri\AppData\Local\Programs\Python\Python37\lib\site-packages\aiogoogle\validate.py", line 317, in validate_object
validate(instance[k], v, schemas, schema_name=k)
File "C:\Users\xybri\AppData\Local\Programs\Python\Python37\lib\site-packages\aiogoogle\validate.py", line 337, in validate
validate_object()
File "C:\Users\xybri\AppData\Local\Programs\Python\Python37\lib\site-packages\aiogoogle\validate.py", line 317, in validate_object
validate(instance[k], v, schemas, schema_name=k)
File "C:\Users\xybri\AppData\Local\Programs\Python\Python37\lib\site-packages\aiogoogle\validate.py", line 344, in validate
validate_all(instance, schema, schema_name)
File "C:\Users\xybri\AppData\Local\Programs\Python\Python37\lib\site-packages\aiogoogle\validate.py", line 229, in validate_all
validate_format(instance, schema, schema_name)
File "C:\Users\xybri\AppData\Local\Programs\Python\Python37\lib\site-packages\aiogoogle\validate.py", line 210, in validate_format
format_validator(instance, schema_name)
File "C:\Users\xybri\AppData\Local\Programs\Python\Python37\lib\site-packages\aiogoogle\validate.py", line 130, in uint64_validator
if (value > 9223372036854775807
2) or (value < 0):
TypeError: '>' not supported between instances of 'str' and 'int'

I did not find out why this error happened.

Pip install on windows UnicodeDecodeError

Hi,

This package generates the following error if it's being installed on Windows with pip. No errors on Linux.

 ERROR: Complete output from command python setup.py egg_info:
    ERROR: ['MANIFEST.in', 'pip-delete-this-directory.txt', 'PKG-INFO', 'README.md', 'requirements.txt', 'setup.cfg', 'setup.py', 'test_requirements.txt', 'tox.ini']
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\user\AppData\Local\Temp\pip-install-_nfr2x8z\aiogoogle\setup.py", line 18, in <module>
        long_description = fh.read()
      File "C:\ProgramData\Anaconda3\envs\project\lib\encodings\cp1252.py", line 23, in decode
        return codecs.charmap_decode(input,self.errors,decoding_table)[0]
    UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in position 2972: character maps to <undefined>
    ----------------------------------------
ERROR: Command "python setup.py egg_info" failed with error code 1 in C:\Users\user\AppData\Local\Temp\pip-install-_nfr2x8z\aiogoogle\

If open statements in setup.py are used with encoding="utf-8" the package can be installed without errors.

Thanks,

Uploading a file with multipart not working properly

I have following code:

async def async_gcs(file_list: List[Tuple[str, str, str]], bucket: str) -> List[Dict]:
    """Upload files in parallel to GCS

    Args:
        file_list (List[Tuple[str, str, str]]): list of files to upload
        bucket (str): bucket to upload to

    Returns:
        List[Dict]: list http responses payloads
    """
    async with aiogoogle_factory:
        storage = await aiogoogle_factory.discover("storage", "v1")
        requests = []
        for file_name, file_path, mime_type in file_list:
            req = storage.objects.insert(bucket=bucket, name=file_name, upload_file=file_path)
            req.headers = {'Content-Type': mime_type}
           # req.media_upload.multipart = False
            requests.append(req)
        results = await aiogoogle_factory.as_service_account(*requests, full_res=True)
    return [results.json] if len(file_list) == 1 else [result.json for result in results]

file_list = [('<destination_name>','<path_to_csv_file>', '<file_mime_type>')] # in this case file_mime_type = "text/csv"
asyncio.run(async_gcs(file_list, bucket))

Most files get uploaded fine, but when I try to upload a text/csv file, my file content changes to something like

--1d2d26ead9e648f9a3732162b709808a
Content-Type: application/octet-stream

,text,conf,page_num,xmin,xmax,ymin,ymax,font_size,line_id,box_id,origin
0,SPORTING,0.99375,1,6.751644020536765,160.24835550262605,35.00000000000006,74.00001525878909,39,0,0,image/png
1,A,0.92,1,55.999996185302734,117.0,71.0,149.0,78,1,1,image/png
2,TICKET,0.995,1,216.00001525878906,321.0,193.0,227.0,34,2,3,image/png
3,JAARABONNEMENTEN,0.84,1,340.0,662.0,193.0,227.0,34,2,3,image/png
4,&,0.94,1,673.0,691.0,193.0,227.0,34,2,3,image/png
5,MEERBEL,0.9342857142857143,1,703.0,838.0,193.0,227.0,34,2,3,image/png
6,STEDELIJKE,0.9940000000000001,1,253.0,523.0,246.0,290.99999618530273,45,3,4,image/png
7,ZWEMBADEN,0.9855555555555555,1,533.0,825.0,246.0,290.99999618530273,45,3,4,image/png

--1d2d26ead9e648f9a3732162b709808a
Content-Type: application/json
Content-Length: 4

null
--1d2d26ead9e648f9a3732162b709808a--

The original content is

,text,conf,page_num,xmin,xmax,ymin,ymax,font_size,line_id,box_id,origin
0,SPORTING,0.99375,1,6.751644020536765,160.24835550262605,35.00000000000006,74.00001525878909,39,0,0,image/png
1,A,0.92,1,55.999996185302734,117.0,71.0,149.0,78,1,1,image/png
2,TICKET,0.995,1,216.00001525878906,321.0,193.0,227.0,34,2,3,image/png
3,JAARABONNEMENTEN,0.84,1,340.0,662.0,193.0,227.0,34,2,3,image/png
4,&,0.94,1,673.0,691.0,193.0,227.0,34,2,3,image/png
5,MEERBEL,0.9342857142857143,1,703.0,838.0,193.0,227.0,34,2,3,image/png
6,STEDELIJKE,0.9940000000000001,1,253.0,523.0,246.0,290.99999618530273,45,3,4,image/png
7,ZWEMBADEN,0.9855555555555555,1,533.0,825.0,246.0,290.99999618530273,45,3,4,image/png

The issue is fixed when I uncomment the line:

req.media_upload.multipart = False

in the first code block above. Seems like something is going on with multipart uploads that is not working correctly. Any idea what I could be doing wrong?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.