jaysnm / dremio-arrow

Dremio Arrow Flight Client

Dremio SQL Lakehouse Arrow Flight Client

Arrow Flight is a high-speed, distributed protocol designed to handle big data, providing increase in throughput between client applications and Dremio. This Dremio Arrow Flight Client is based on python official examples.

Disclaimer: This project is not affliated to dremio in any way. It is a tool that I developed while at CIFOR-ICRAF and now we have decided to open source it for wider community use. While I may not have enough time to actively maintain it, the tool is stable enough to sustain future use cases. Besides, community contribution is warmly welcome in form of PRs and forks.

• Documentation: https://jaysnm.github.io/dremio-arrow/
• GitHub: https://github.com/jaysnm/dremio-arrow
• PyPI: https://pypi.org/project/dremio-arrow/
• Free software: Apache-2.0

Flight Basics

The Arrow Flight libraries provide a development framework for implementing a service that can send and receive data streams. A Flight server supports several basic kinds of requests:

• Handshake: a simple request to determine whether the client is authorized and, in some cases, to establish an implementation-defined session token to use for future requests
• ListFlights: return a list of available data streams
• GetSchema: return the schema for a data stream
• GetFlightInfo: return an “access plan” for a dataset of interest, possibly requiring consuming multiple data streams. This request can accept custom serialized commands containing, for example, your specific application parameters.
• DoGet: send a data stream to a client
• DoPut: receive a data stream from a client
• DoAction: perform an implementation-specific action and return any results, i.e. a generalized function call
• ListActions: return a list of available action types

More details can be found here

Installation

Please see installation notes here

dremio-arrow's Issues

Any plans to support conda packages

• dremio-arrow version: 1.0.1
• Python version: 3.7
• Operating System: Linux

Description

Do you intend to support a conda packaged version on conda-forge

Back to active development and maintenance

Dear user,

I hope you found this project useful. While it is expected that the package is compatible with most (if not all) dremio engines, a lot has changed in the dremio world. For this reason, I am back to actively maintain this package to keep it useful and reliable.

Regards,
Jason

Inconsitency between Python version in docs and pyproject.toml

The pyproject.toml says one thing

dremio-arrow/pyproject.toml

Line 24 in cebb6f3

python = ">=3.10,<3.12"

Docs says 3.7 to 3.10:
https://jaysnm.github.io/dremio-arrow/installation/

Upgrade to Support PyArrow 9.0.0

dremio-arrow version: 1.0.1
Python version: 3.7
Operating System: Linux

Description

Latest Dremio version ships with Pyarrow 9.0.0, so far everything in arrow and flight works well with arrow 9.0.0

Any reason to not change the pins from <8.0 to <10..0 allowing installtion without downgrading existing environments from pyarrow 9.0 to 8.0