bug: AIRTABLE Plugin: sync failed with error message from the log about cloudquery

[yevgenypats]

I've opened this issue that should improve experience https://github.com/cloudquery/cloudquery-issues/issues/1479 . Our airtable plugin was written in python mostly serving as an example but all our official plugins are in Go to avoid having docker as a pre-requirement. we will update here once plugin is re-written in Go.

[shotleybuilder]

@erezrokah okay, thank-you. I'm going to try and run as a container - I've just generated my API key.

[erezrokah]

Hi @shotleybuilder, can you confirm:

1. You have Docker installed and running via docker --version
2. The CloudQuery CLI logged in via cloudquery login

[shotleybuilder]

@erezrokah
Docker -> Docker version 26.0.0, build 2ae903e
Cloudquery login -> Not sure what behaviour to expect. I did run cloudquery login which opened the web page, I logged in using my github creds and then closed the page.

[erezrokah]

Not sure what behaviour to expect

That's the correct behavior 💯

Let me try and reproduce the issue and I'll respond here in a bit

[erezrokah]

What is the environment you're running cloudquery sync in? Running docker plugins inside Docker won't work out of the box, as the CLI uses docker internally. There are some solutions to running docker inside docker, for example https://stackoverflow.com/questions/27879713/is-it-ok-to-run-docker-from-inside-docker

[shotleybuilder]

What is the environment you're running cloudquery sync in?
Just the terminal on my laptop running Ubuntu 22.04.4 LTS Jammy

[shotleybuilder]

Running this cmd:
sudo ./cloudquery sync cloudquery-config/airtable.yml cloudquery-config/postgresql.yml

[erezrokah]

Thanks for sharing the information @shotleybuilder, I'm unable to reproduce (running on MacOS).
Any reason you're trying with sudo? Maybe drop it?
Can you also try a sanity test and run docker run ghcr.io/cloudquery/cloudquery:5.13.1?
That should invoke our CLI via our Docker image

[shotleybuilder]

Okay, so there was a problem with my Docker install -> see here. This has fixed 'sudo'.
srw-rw---- 1 root docker 0 Apr 3 16:41 /var/run/docker.sock
-rw-rw-r-- 1 root docker 295 Mar 20 15:16 /lib/systemd/system/docker.socket

The result of running:

$ docker run ghcr.io/cloudquery/cloudquery:5.13.1
Unable to find image 'ghcr.io/cloudquery/cloudquery:5.13.1' locally
5.13.1: Pulling from cloudquery/cloudquery
4abcf2066143: Already exists 
d9eacfa5a0f3: Pull complete 
b1ca05ac361b: Pull complete 
aa39d93e82aa: Pull complete 
Digest: sha256:dc6b2bb1741903e03611fbb6e0c12c156f1af410863e9ffb36f36e14c47f5d20
Status: Downloaded newer image for ghcr.io/cloudquery/cloudquery:5.13.1
CloudQuery CLI

But, I'm still getting the problem Error

$ ./cloudquery sync cloudquery-config/airtable.yml cloudquery-config/postgresql.yml
Loading spec(s) from cloudquery-config/airtable.yml, cloudquery-config/postgresql.yml
Error: failed to pull Docker image: Error response from daemon: Head "https://docker.cloudquery.io/v2/cloudquery/source-airtable/manifests/v2.1.2": unknown: The server could not parse your request body

Have restarted docker sudo systemctl restart docker
Will now try a reboot :)

[shotleybuilder]

Okay, so the reboot hasn't worked

$ ./cloudquery login
Opened browser at https://accounts.cloudquery.io?returnTo=http://localhost:36651/callback. Waiting for authentication to complete.
Gtk-Message: 13:14:38.906: Not loading module "atk-bridge": The functionality is provided by GTK natively. Please try to not load it.
Authenticating...
Your current team is set to spongl.
CLI successfully authenticated.
$ ./cloudquery sync cloudquery-config/airtable.yml cloudquery-config/postgresql.yml
Loading spec(s) from cloudquery-config/airtable.yml, cloudquery-config/postgresql.yml
Error: failed to pull Docker image: Error response from daemon: Head "https://docker.cloudquery.io/v2/cloudquery/source-airtable/manifests/v2.1.2": unknown: The server could not parse your request body
$ 

Will revert to running as described here

[shotleybuilder]

Wow, this is frustrating, I've pretty much come to the end of my experiment and did not achieve a working sync. There are errors in the documentation too, which for a beginner is really tough. Might drop back in when this is matured or someone has blogged about a successful Airtable sync.
Here's about as far as I was able to get:

$ sudo systemctl restart docker.socket
$ curl --unix-socket /var/run/docker.sock  http://localhost/_ping; echo
OK
$ docker run --pull=always -v /home/jason/Desktop/cloudquery/cloudquery-config:/config.yml -e CLOUDQUERY_API_KEY==api-key ghcr.io/cloudquery/cloudquery:latest sync /config.yml
latest: Pulling from cloudquery/cloudquery
Digest: sha256:bcecd4af17ee0ff87fe167ac2c8ef7b852e2df1204bcf43660839e6463c47abc
Status: Image is up to date for ghcr.io/cloudquery/cloudquery:latest
Loading spec(s) from /config.yml
Error: failed to list Docker images: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

I'm going to reinstall Docker but I have to say there were no problems running Airbyte locally yesterday.

[erezrokah]

Hi @shotleybuilder sorry you're experiencing this issue. Can I recommend using the CLI directly and not inside a docker container? Especially if you're running from Ubuntu

[bbernays]

@shotleybuilder We noticed a typo in our docs we have -e CLOUDQUERY_API_KEY==api-key instead of -e CLOUDQUERY_API_KEY=api-key . Can you remove the second = and retry?

We have also opened a PR to fix this in our docs: #17537

[shotleybuilder]

@bbernays
there is another typo docker run --pull always should be docker run --pull=always
I should've caught the == :)
But, neither are causing my error, which I suspect is buried in Docker somewhere. I've since decided I don't need to sync from Airtable. I'm going to write to the db directly.

[erezrokah]

Our airtable plugin was written in python

Small correction it's written in JavaScript

[erezrokah]

I've also opened #17547

[erezrokah]

Re-opened as #17546 only fix the docs error mentioned in #17531 (comment)

[erezrokah]

Hi 👋 We published a guide on how to use CloudQuery docker plugins in a containerized environment https://docs.cloudquery.io/docs/advanced-topics/using-cloud-query-docker-registry-plugins-inside-a-containerized-environment
We're also looking to porting the Airtable plugin to a Go binary to simplify deployment, but that might take a bit of time