Comments (8)
Thanks for the question! Quilt supports reading packages in pySpark. Structured datasets are read into Spark DataFrames. In Spark, most often, you'll want to access a large partitioned table as a Spark DataFrame by accessing the GroupNode and calling _data() (or just calling the object()). There's no support in Quilt yet for distributed package creation (build) in pySpark. Is that something you would find useful?
Quilt doesn't yet support running a separate registry on HDFS, but that's definitely on the roadmap.
from quilt.
from quilt.
@chanansh, build
is a Quilt command that creates a new snapshot of your data (as a new package version). By call
I simply meant the Python __call__
method, which is callable by just adding () to a node in the Quilt package.
My guess is that Quilt will need a couple of small extensions (which are all on the roadmap) before it will do what you want. Basically, we'll need to add support for using HDFS in a PackageStore. That could be as easy as swapping the normal Python os.path and pathlib calls for hdfs3 when using HDFS and pyspark.
What type of data are you storing in HDFS? Parquet files?
from quilt.
from quilt.
Great! And, is it a set of partitioned tables? We just added support for Parquet input data, but the code doesn't handle partitioned tables/dataframes yet. I can probably get a prototype of that working this afternoon if you'd like to give it a try.
from quilt.
from quilt.
I'm also interested on loading and versioning data from HDFS through Quilt, so it can be integrated on a Spark pipeline of distributed data processing. Can someone give me details of this integration Quilt + HDFS? is it feasible? Thanks!
from quilt.
Current support is for S3 and S3 connectors can be used to provide access via Hadoop. General trend is away from HDFS towards S3. Feel free to reopen if still an issue.
from quilt.
Related Issues (20)
- package ignoring locally installed versions HOT 9
- Catalog failing to render thumbnails on package view but succeeding on file view HOT 2
- Catalog does not render large Vega JSON files (created by altair) HOT 2
- Accessing package causes ascii decoding error HOT 3
- [question] Quilt and min.io? HOT 15
- Add package created and last updated timestamps part of the API response HOT 2
- Can't resolve 'jsoneditor/dist/jsoneditor.css' HOT 4
- Dependency Dashboard HOT 2
- Add a way to pre-fill metadata on push HOT 4
- Cannot retrieve the old version data HOT 1
- quilt3 verify enhancement
- Remove $ref restriction HOT 1
- Remove schema version restriction HOT 3
- add kms docs
- Quilt 2.9.15 within AWS Lambda execution HOT 3
- Support for running Quilt in an EKS pod with OpenID authentication to AWS disappeared/missing HOT 2
- Workflows should enforce push-level tag completeness
- quilt3 push: --dir is a required argument
- Config resolution at import
- Object level metadata from JSON schema
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from quilt.