mslusarz / csv-nix-tools

List system information as CSV, manipulate it, pretty print, or export.

csv-nix-tools

csv-nix-tools is a collection of tools for gathering and processing system information using CSV as an intermediate format.

Although CSV is in the name of the project and this format is used for information storage, it's not that critical aspect of this project.

What is more important is that it allows safe processing of structured data using *NIX-like philosophy by employing pipes and tools with familiar names (such as grep, sort, etc, see examples). "Safe" processing means that the handling of special characters, like new lines, is effort-less - it just works, without bothering users with hacks like IFS.

This project also solves the problem of combining data from multiple sources into a single stream (see the tables tutorial) and processing it in any way user wants to, including powerful SQL syntax, without leaving the shell.

Status

This project is in beta stage. Everything listed below is functional and works as documented. Currently, it builds and runs on Linux with glibc only, but portability patches are welcomed. Although the project has NIX in the name and many source tools are Unix or even Linux-specific, sane portability patches, even for non-Unix systems, will be accepted.

If you discover any issue in the project or a missing feature, don't hesitate to file an issue. Any feedback is welcomed.

Dependencies

• cmake >= 3.3
• glibc-devel
• pkg-config
• pandoc (optional, required to generate man pages)
• bison (optional, required by csv-sql, csv-grep-sql and csv-add-sql)
• flex (optional, required by csv-sql, csv-grep-sql and csv-add-sql)
• libmnl-devel (optional, required by csv-netstat)
• libncursesw-devel (optional, used by csv-show)
• libprocps-devel (optional, required by csv-ps)
• libsqlite3-devel (optional, required by csv-sqlite)
• tmux (optional, needed by some csv-show tests)

On Debian-based distros you can install all dependencies by:

apt install build-essential cmake pkg-config flex bison libsqlite3-dev \
            libprocps-dev libncursesw5-dev libmnl-dev pandoc tmux

On Fedora-based distros you can install all dependencies by:

yum install gcc make glibc-devel cmake pkg-config flex bison sqlite-devel \
            procps-ng-devel ncurses-devel libmnl-devel pandoc tmux

Installation

mkdir build
cd build
cmake .. -DCMAKE_INSTALL_PREFIX=/usr
make
sudo make install

Available tools

Source:

• csv-env - lists environment variables
• csv-groups - lists system groups
• csv-group-members - lists system groups and users that belong to them
• csv-ls - lists files
• csv-lstree - lists files in hierachical manner (wrapper around csv-ls and csv-tree)
• csv-netstat - lists network connections
• csv-ps - lists processes
• csv-pstree - lists processes in hierachical manner (wrapper around csv-ps and csv-tree)
• csv-users - lists system users

Filtering/processing:

• csv-add-concat - adds a new column by concatenation of columns and fixed strings
• csv-add-exec - pipes data to standard input of an external command and creates a new column from its standard output
• csv-add-replace - adds a new column by performing a string substitution on another column (similar to sed s/$str/$str/)
• csv-add-rev - adds a new column by reversing another column characterwise
• csv-add-rpn - adds a new column from RPN expression
• csv-add-split - adds two new columns by splitting another one using a delimiter
• csv-add-sql - adds a new column from SQL expression
• csv-add-substring - adds a new column by extracting a substring of another column
• csv-avg - takes an average of numerical column(s)
• csv-cat - concatenates multiple csv files
• csv-count - counts the number of columns and/or rows
• csv-cut - removes columns and reorders them
• csv-diff - compares 2 or more files
• csv-grep - filters rows matching a pattern
• csv-grep-rpn - filters rows using RPN expression
• csv-grep-sql - filters rows using SQL expression
• csv-head - outputs the first N rows
• csv-header - processes data header
• csv-max - takes a maximum value of numerical or string column(s)
• csv-merge - merges multiple input streams
• csv-min - takes a minimum value of numerical or string column(s)
• csv-peek - peek at the CSV stream and pass it unmodified
• csv-sort - sorts input by column(s)
• csv-sql - processes input data using simplified (but very fast) SQL-based syntax
• csv-sqlite - processes input data using SQLite (requires loading the whole input before processing)
• csv-sum - takes a sum of numerical or string column(s)
• csv-tac - concatenates files in reverse
• csv-tail - outputs the last N rows
• csv-tree - processes hierarchical data
• csv-uniq - merges adjacent duplicate rows

Sink:

• csv-exec - executes an external command for each row
• csv-plot - generates 2D or 3D graph using gnuplot
• csv-printf - formats data using printf(3)
• csv-show - formats data in human-readable form (also available as a "-s/-S" option in all source and processing tools)
• csv-to-html - converts data to HTML
• csv-to-json - converts data to JSON
• csv-to-xml - converts data to XML

Documentation

See the man pages.

Examples

See EXAMPLES

TODO

See TODO

csv-nix-tools's Issues

Error when column has colon in the column (:) name

I am trying to csv-sqlite to perform some cleanup on a csv file, and I'm having trouble when loading a csv with a colon in the name:

#no issue here echo "ID,Title,test" > a.csv cat a.csv | csv-sqlite "select * from input" -s #error here echo "ID,Title,test:" > a.csv cat a.csv | csv-sqlite "select * from input" -s #error here echo "ID,Title,\"test:\"" > a.csv cat a.csv | csv-sqlite "select * from input" -s

It prints the message:

unsupported type '"'
EOF while reading header

And exits with exit code 2, I tried with current trunk (a39e4e6)

Vague message when csv-show is not in PATH

After compiling and before installing, when running with -s we get:

$ ./csv-ps -s
execvp: No such file or directory

The message doesn't mention csv-show. It would be nicer if the error stated something like "csv-show not found in PATH".