cortex-lab / npytools

[WIP] Command-line utilities to deal with NPY array files

Command-line utilities to deal with NPY array files

The NPY format

The NPY file format is used to store NumPy arrays. It is becoming a de facto standard for storing arbitrarily large multidimensional arrays. For instance, the International Brain Laboratory uses it for a large part of its data.

• Documentation of NPY
• Motivation behind the NPY format

Although first implemented in a Python library, there are now libraries in many languages, including the following (some of the libraries below may be experimental):

• Python
• MATLAB
• Julia
• R
• C/C++
• C#
• Java
• Rust
• Javascript
• Go
• Haskell
• Swift
• Lisp

npytools

Even if the NPY format is quite simple (a header with metadata about the array, such as its dtype and shape, followed by the raw binary data), and NPY files can be easily open in Python, there is also a need for command-line tools to manage and quickly inspect NPY files.

npytools is a minimal Python library that provides such tools.

Installation

Dependencies : Python 3, NumPy, click (which can be installed with pip install click).

To install npytools: pip install git+https://github.com/cortex-lab/npytools.git

npyshow

npypshow is a simple command-line tool that will display metadata and possibly basic statistics about an array stored in a NPY file. The array is memmapped and therefore not entirely loaded in memory, unless the --show-stats option is used.

$ npyshow myarray.npy --show-stats
+----------|----------------+
| shape    | (524, 82, 374) |
| dtype    | float32        |
| filesize | 64.3M          |
| size     | 16070032       |
| min      | -0.59231997    |
| mean     | -3.6333304e-06 |
| median   | 0.0            |
| max      | 0.58112603     |
| zero     | 15049624 (93%) |
| nan      | 0              |
| inf      | 0              |
+----------|----------------+
[[[ 0.      0.     ...  0.      0.    ]
  [ 0.      0.     ...  0.      0.    ]
  ...
  [ 0.0007  0.0099 ...  0.      0.    ]
  [ 0.0003  0.004  ...  0.      0.    ]]

 ...

 [[ 0.      0.     ...  0.      0.    ]
  [ 0.      0.     ...  0.      0.    ]
  ...
  [ 0.      0.     ...  0.      0.    ]
  [ 0.      0.     ...  0.      0.    ]]]

Running npyshow on multiple files (Unix)

To quickly call npyshow on all _ibl_*.npy file, call:

ls -1 _ibl_*.npy | xargs -l npyshow