Comments (3)
Currently, mars.tensor.array
and mars.tensor.asarray
which are the counterparts of numpy.array
and numpy.asarray
are used to create mars tensor with array like objet including numpy ndarray.
Sadly, IO interface is truly absent in Mars because we don't come up with IO interfaces to support various file format which are compatible with numpy ones.
The dask.array.from_array
and dask.array.store
look great, and may be a good solution, we will do some study first and try to find out how to add IO interface into Mars.
Discussion is welcomed in this thread.
from mars.
I want to bring more backgrounds that for the inner projects which use Mars, we write the interface like mars.tensor.read_files(distributed_storage_directory)
, this requires users to write a MapReduce-like program to process the data into different directories in the distributed storage directory, each directory contains files with parquet or some columnar storage format, stored with i, j, k, value
schema.
We were stuck in how to provide a commonly used interface which need to be as compatible as possible.
from mars.
Indeed, file formats for multi-dimensional arrays are challenging. I think the main virtue of dask's approach here is that it's quite flexible, and allow reusing existing code for working with files. You would not want to write specialized support for reading/writing only a single file-format (users tend to have strong feelings about file formats).
The downside is that now distributed workers need to know how to run arbitrary Python code from users (but this can also be quite useful, as a form of foreign function interface).
from mars.
Related Issues (20)
- [BUG] scipy 1.10 raises ValueError: `popmean.shape[axis]` must equal 1.
- [BUG] Support for higher versions of python HOT 3
- [BUG] md.read_ray_dataset is not compatible with latest Ray
- [BUG] mars.tensor.int is not compatible with numpy since v1.24
- [BUG] Creating an ndarray from ragged nested sequences is deprecated since numpy 1.24
- [BUG] anaconda3/lib/python3.8/uuid.py", line 180, raise ValueError('bytes is not a 16-char string') HOT 12
- [BUG] throw `AttributeError` when using mars tensor with `gpu=True` on colab HOT 2
- [BUG] throw a `No NVIDIA GPU detected` warning or a similar error when deploying Mars on Ray
- [BUG] Apply Function get HTTP Error HOT 6
- [CI] Pandas 2.0.0 will break down Mars CI
- Want some examples of custom apply function
- [GRAPH][ENHANCEMENT] Optimize `SubtakGraph` generation
- [BUG] `test_ray_client` run failed
- [BUG]How to set the ip address in the cluster mode HOT 3
- [BUG] lightgbm TypeError: a bytes-like object is required, not 'list'
- [BUG][Ray] LightGBMError: Machine list file doesn't contain the local machine
- A more universal function to replace subgraph in OptimizationRule
- [BUG] Series doesn't support attribute access
- [BUG] Series with multi index accessing data by index error
- [PERF] Performance issue in pandas 1.3.4's read_csv Function with Large Column CSV Files
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mars.