Comments (6)
deprio this because we're demoing via tabular datastores which go to a pandas dataframe
from aml.
reprio becuase we want to support files and tables now
from aml.
According to Andrei, the download path is not handled by rslex currently. This might be resolved automatically by a future update.
from aml.
from aml.
This is caused by the inability to detect folder structure from python (because it does a list today and tries to find a common prefix)
There needs to be an improvement in the SDK to make the logic smarter
As of right now it does the safe option of creating a fully qualified path so even if your dataset has multiple paths pointing to different pachyderm clusters files dowloaded would not collide.
from aml.
I wonder if we control the path that gets passed back to whatever code handles the downloads though
…
On Tue, 6 Jul 2021, 19:22 Albert, @.***> wrote: According to Andrei, the download path is not handled by rslex currently. This might be resolved automatically by a future update. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#21 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACATUSILUSYQQAYDIBYDALTWNCYFANCNFSM4637W7WQ .
As a matter of fact, we do! We were adding the http://localhost:30600...
prefix to the StreamInfos returned by the Searcher. We didn't need to do this because the rslex-http-stream library calls the ReadRequest
API which is implemented by the RequestBuilder
. The RequestBuilder
contains the HTTP schema, host, and port info already.
from aml.
Related Issues (20)
- rewrite test_pachyderm.py as a rust test HOT 1
- Probably remove `blob_dto.rs` - it's all about parsing XML ourselves
- test with large number of files (pagination)
- remove ApplyCredential, replace with String
- Support deploying syncer for existing Pachyderm instance
- Update README with instruction on how to activate private preview in your account (needs collaboration with Microsoft!) HOT 1
- Update terraform code to deploy the marketplace VM
- ADLS Gen 2 spout HOT 3
- [Syncer] Fix `UnboundLocalError: local variable 'ds_new' referenced before assignment`
- [rslex] Improve how we pass `uri` into `request_builder` HOT 1
- Flatten out the `vec![ListBucketResult]` returned by `bucket.list()`
- Implement `list_directory()` and `get_entry()` in `stream_handler.rs` so that we can support mounting.
- Searcher does not recurse on subdirectories given a glob pattern
- Hide kubectl port-forward outputs
- Support migrating data from ADLS Gen2 to Pachyderm
- Programmatically define multiple AML workspaces.
- Update to Pachyderm 2.0
- Deployment uses standard storage class HOT 1
- Fix docs on configuring "Advanced: using pachctl locally" HOT 1
- pachyderm-aml-syncer.service is not enabled on startup on syncer VM HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aml.