prabodh1194 / ray_snowflake

Ray Data Connector for Snowflake

ray_snowflake

This repo does a test drive of using snowflake as a source in ray. This has been done after I struggled to make -- anyscale snowflake connector work. I have written a test driver as well which shows how to do a basic transform on this dataset.

Setup

You just need a valid snowflake account to test this out. I tested this code on a trial snowflake account and hence could get this to work with ACCOUNTADMIN creds.

1. make all: create python 3.10 venv and install the deps.
2. cp .env.example .env: setup the credentials environment.
3. Open the .env file and setup the 3 creds.

Run

Once you have filled up the creds correctly, you can run the script and see things happen. The test script performs a write to a DB as well. You can tinker the script to put in the correct db name.

ray_snowflake's Issues

write arrow to snowflake directly

1. snowflake supports default write by pandas.
2. pandas write actually happens by parquet files.
3. since arrow tables can be converted to parquet as well, we can redo that code.

same as -- snowflakedb/snowflake-connector-python#1358

Support for Ray Snowflake datasource writer

I appreciate the author's [@prabodh1194] efforts in creating the Ray Snowfalke datasource reader.
It would be an added benefit if this repository supports the writer too.

Thanks!