Comments (7)
Hi @vikas-velora apologies for the slow turnaround on your question - our team is verifying if this is the case.
As a general rule, we advocate deployment from source (in this spirit: https://cloud.google.com/run/docs/deploying-source-code), and will provide the easiest recipe that works in this direction. The intended experience is something like this one with Render: https://pathway.com/developers/user-guide/deployment/render-deploy/.
from pathway.
from pathway.
from pathway.
Hi Vikas,
To give a small technical heads-up, there is a way to dockerize the airbyte connector code for the local tests. Precisely, you would need to install Docker in your Dockerfile as follows:
RUN apt update && apt install docker.io -y
And then run with mounting two volumes, as follows:
docker run -v /var/run/docker.sock:/var/run/docker.sock -v /tmp:/tmp <your_image_name>
The first volume is required to enable DinD, while the second one is needed because the /tmp
is currently used to store the temporary artifacts of the airbyte connector. Note that this wouldn't be so easy to deploy (and I suppose, it's impossible to deploy it at Google Cloud) because of giving access to the Docker socket.
While I've also tried to use the docker:dind
image as a base, I've also figured out that it's unusable for our case because of using Alpine Linux as the base for docker:dind
which is not supported by Pathway yet. Thus, I think we need to do something different and implement running the airbyte connector without depending on Docker, in GCP. It would need to be done for the Pathway framework.
So, to wrap it up, the way to go will be to run the airbyte connector in the GCP - a feature that must be added to Pathway. I am currently checking this possibility and will be back to you today or in a few days.
from pathway.
Thanks so much @zxqfd555-pw . We tried multiple ways, and were unable to deploy - at least this confirms that it was not something to do with our knowledge 😊. Will wait for your update.
from pathway.
Hi Vikas!
A quick heads-up: we can eliminate the need for the DinD technique for airbyte connectors by introducing a mode where they run as GCP jobs. I am in the process of implementing it, and we can release the corresponding update next week.
from pathway.
from pathway.
Related Issues (20)
- [Bug]: Json type does not behave like a transparent wrapper for the value HOT 1
- Allow sending columns with raw bytes in the python connector
- Add `first_value`/`last_value` reducers. HOT 3
- Static table import (table_from_...) should not be under `pw.debug`
- [Bug]: "AttributeError: 'DataFrame' object has no attribute 'map'" when running Live Data Jupyter notebook. HOT 4
- [Bug]: Reproduction of Twitter's Custom Python Connector code encours Auth issues HOT 1
- Adjust examples using Twitter API HOT 1
- support datetime + formatting in connectors
- How to Integrate Replicate API using LiteLLMEmbedder and LiteLLMChat HOT 2
- Parser giving unicode characters for Arabic language HOT 10
- getting error in "Streaming ETL pipelines" showcase HOT 5
- Cannot import name 'default_vector_document_index' from 'pathway.stdlib.indexing' (/usr/local/lib/python3.10/dist-packages/pathway/stdlib/indexing/__init__.py) HOT 4
- ImportError: cannot import name 'DataIndex' from 'pathway.stdlib.indexing' HOT 3
- Private Rag Example not working HOT 5
- [Bug]: *Table desugaring + *.ix(pw.this) desugaring fails
- [Bug]: `pw.run()` never terminates when used with `multiprocessing`
- [Bug]: Error messages are not printed to screen when error happens on PATHWAY_PROCESS_ID > 0
- How to Embed each dict in jsonline format HOT 6
- Download files in parallel in gdrive connector
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pathway.