Loading and saving external data about mars

[qinxuye]

Currently, mars.tensor.array and mars.tensor.asarray which are the counterparts of numpy.array and numpy.asarray are used to create mars tensor with array like objet including numpy ndarray.

Sadly, IO interface is truly absent in Mars because we don't come up with IO interfaces to support various file format which are compatible with numpy ones.

The dask.array.from_array and dask.array.store look great, and may be a good solution, we will do some study first and try to find out how to add IO interface into Mars.

Discussion is welcomed in this thread.

[qinxuye]

I want to bring more backgrounds that for the inner projects which use Mars, we write the interface like mars.tensor.read_files(distributed_storage_directory), this requires users to write a MapReduce-like program to process the data into different directories in the distributed storage directory, each directory contains files with parquet or some columnar storage format, stored with i, j, k, value schema.

We were stuck in how to provide a commonly used interface which need to be as compatible as possible.

[shoyer]

Indeed, file formats for multi-dimensional arrays are challenging. I think the main virtue of dask's approach here is that it's quite flexible, and allow reusing existing code for working with files. You would not want to write specialized support for reading/writing only a single file-format (users tend to have strong feelings about file formats).

The downside is that now distributed workers need to know how to run arbitrary Python code from users (but this can also be quite useful, as a form of foreign function interface).