singer-io / tap-google-sheets Goto Github PK

I think this is safe to remove because it's only used in this one initialization, but it's not really used because that variable is overwritten moments later

I'm getting this error from google:

{  "error": "invalid_grant",  "error_description": "Token has been expired or revoked."}

this causes this error, as error.code property is missing:

...
tap_google_sheets/client.py", line 124, in raise_for_error
    error_code = response.get('error', {}).get('code')
AttributeError: 'str' object has no attribute 'get'

obviously property error cannot be a string an an object at the same time

Same sheet was getting pulled earlier; now it is giving this error; Tried to create a new sheet and pulled it but the same error.

INFO HTTP request to "file_metadata" endpoint took 0.203s, returned status code 404
2020-03-03 08:45:46,070Z tap - CRITICAL {'code': 404, 'message': 'File not found: xxxxxxxxxxxx.', 'errors': [{'location': 'fileId', 'message': 'File not found: xxxxxxxxxxx.', 'locationType': 'parameter', 'reason': 'notFound', 'domain': 'global'}]}: Unknown Error

After setting up the integration with Stitch, I have found that every time the data is loaded into the warehouse, every single row is re-imported during the process. This is going to quickly increase the monthly row count for the integration. Is there a way to add only the new rows during each import?

The current exponential backoff factor is 3.

Context

• I have an airflow pipeline that extracts data from 12 spreadsheets with 20k rows each.
• The steps ran sequentially.
• In one of the steps, intermittently, the pipeline failed multiple times because the number of retries of the exponential backoff is exceeded. As you can see in the image below:
  

Proposal

By changing the exponential factor from 3 to 4 in a fork of this repository, I was able to fix this issue on my pipeline.

    @backoff.on_exception(backoff.expo,
                          (Server5xxError, ConnectionError, Server429Error),
                          max_tries=7,
                          factor=3,
                          jitter=None)
    @utils.ratelimit(100, 100)

Can we implement this change or are there any problems that I'm not aware of?

I really would like to use the official version of this tap.

Best Regards,

Currently, names of the columns for a single google-sheet data stream is inferred from the first row of the sheet. Due to bad practice from sheet owners, column names often end up with non-ASCII characters, or problematic characters such as newlines. Such cases can be problematic depending on the singer target.

Would it be hard to implement an option to use python-slugify, or another way to standardize strings, to standardize column names?

Running

tap-google-sheets --config config.json --discover > catalog.json

Gives the catalog.json file, as expected (with schema, stream name, metadata, etc).

But now, I want to actually get data from one sheet (say, "Sheet 1"). How am I supposed to do that? Because running:

tap-google-sheets --config config.json --catalog catalog.json

gives me nothing.

So I tried to understand what was happening, and found this: https://github.com/singer-io/singer-python/blob/6c6c773d8b6dc6223551e598574eb0df41f0c415/singer/catalog.py#L47, which basically verifies if a stream is selected. But it turns out this is not automatically generated in the catalog file; so I needed to go to the specific stream ("Sheet 1") and add "selected": true inside the schema property.

Am I missing something here? I think it should be a way to automatically select which stream ("Sheet 1", "file_metadata", etc) you want to get data from.

Machine - Ubuntu 20.04 LTS
python version - 3.8

ran the following commands.
pip install tap-google-sheets
tap-google-sheets --config config.json --discover > catalog.json

the config.json contains the following:

{
"refresh_token":"---------",
"client_id":"----------",
"client_secret":"-------------",
"user_agent": "tap-google-sheets (via singer.io)",
"start_date": "2019-01-01T00:00:00Z",
"spreadsheet_id": "--------------",
"range": "A2:E2"
}

INFO Authorized, token expires = 2020-08-20 14:57:22.412650
INFO Starting discover
CRITICAL module 'singer.metadata' has no attribute 'get_standard_metadata'
Traceback (most recent call last):
  File "/home/owais/anaconda3/envs/singer5/bin/tap-google-sheets", line 8, in <module>
    sys.exit(main())
  File "/home/owais/anaconda3/envs/singer5/lib/python3.8/site-packages/singer/utils.py", line 225, in wrapped
    return fnc(*args, **kwargs)
  File "/home/owais/anaconda3/envs/singer5/lib/python3.8/site-packages/tap_google_sheets/__init__.py", line 49, in main
    do_discover(client, spreadsheet_id)
  File "/home/owais/anaconda3/envs/singer5/lib/python3.8/site-packages/tap_google_sheets/__init__.py", line 26, in do_discover
    catalog = discover(client, spreadsheet_id)
  File "/home/owais/anaconda3/envs/singer5/lib/python3.8/site-packages/tap_google_sheets/discover.py", line 6, in discover
    schemas, field_metadata = get_schemas(client, spreadsheet_id)
  File "/home/owais/anaconda3/envs/singer5/lib/python3.8/site-packages/tap_google_sheets/schema.py", line 269, in get_schemas
    mdata = metadata.get_standard_metadata(
AttributeError: module 'singer.metadata' has no attribute 'get_standard_metadata'