akavel / up

Ultimate Plumber is a tool for writing Linux pipes with instant live preview

up - the Ultimate Plumber

up is the Ultimate Plumber, a tool for writing Linux pipes in a terminal-based UI interactively, with instant live preview of command results.

The main goal of the Ultimate Plumber is to help interactively and incrementally explore textual data in Linux, by making it easier to quickly build complex pipelines, thanks to a fast feedback loop. This is achieved by boosting any typical Linux text-processing utils such as grep, sort, cut, paste, awk, wc, perl, etc., etc., by providing a quick, interactive, scrollable preview of their results.

Usage

Download up for Linux   |   ArchLinux: aur/up   |   FreeBSD: pkg install up   |   macOS: brew install up   |   Other OSes

To start using up, redirect any text-emitting command (or pipeline) into it — for example:

$ lshw |& ./up

then:

• use PgUp/PgDn and Ctrl-[←]/Ctrl-[→] for basic browsing through the command output;
• in the input box at the top of the screen, start writing any bash pipeline; then press Enter to execute the command you typed, and the Ultimate Plumber will immediately show you the output of the pipeline in the scrollable window below (replacing any earlier contents)

  • For example, you can try writing: grep network -A2 | grep : | cut -d: -f2- | paste - - — on my computer, after pressing Enter, the screen then shows the pipeline and a scrollable preview of its output like below:

       | grep network -A2 | grep : | cut -d: -f2- | paste - -
       Wireless interface      Centrino Advanced-N 6235
       Ethernet interface      RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
    

  • WARNING: Please be careful when using it! It could be dangerous. In particular, writing "rm" or "dd" into it could be like running around with a chainsaw. But you'd be careful writing "rm" anywhere in Linux anyway, no?

• when you are satisfied with the result, you can press Ctrl-X to exit the Ultimate Plumber, and the command you built will be written into up1.sh file in the current working directory (or, if it already existed, up2.sh, etc., until 1000, based on Shlemiel the Painter's algorithm). Alternatively, you can press Ctrl-C to quit without saving.
• If the command you piped into up is long-running (in such case you will see a tilde ~ indicator character in the top-left corner of the screen, meaning that up is still waiting for more input), you may need to press Ctrl-S to temporarily freeze up's input buffer (a freeze will be indicated by a # character in top-left corner), which will inject a fake EOF into the pipeline; otherwise, some commands in the pipeline may not print anything, waiting for full input (especially commands like wc or sort, but grep, perl, etc. may also show incomplete results). To unfreeze back, press Ctrl-Q.

Additional Notes

• The pipeline is passed verbatim to a bash -c command, so any bash-isms should work.
• The input buffer of the Ultimate Plumber is currently fixed at 40 MB. If you reach this limit, a + character should get displayed in the top-left corner of the screen. (This is intended to be changed to a dynamically/manually growable buffer in a future version of up.)
• MacOSX support: I don't have a Mac, thus I have no idea if it works on one. You are welcome to try, and also to send PRs. If you're interested in me providing some kind of official-like support for MacOSX, please consider trying to find a way to send me some usable-enough Mac computer. Please note I'm not trying to "take advantage" of you by this, as I'm actually not at all interested in achieving a Mac otherwise. (Also, trying to commit to this kind of support will be an extra burden and obligation on me. Knowing someone out there cares enough to do a fancy physical gesture would really help alleviate this.) If you're serious enough to consider this option, please contact me by email (mailto:[email protected]) or keybase (https://keybase.io/akavel), so that we could try to research possible ways to achieve this. Thanks for understanding!
• Prior art: I was surprised no one seemed to write a similar tool before, that I could find. It should have been possible to write this since the dawn of Unix already, or earlier! And indeed, after I announced up, I got enough publicity that my attention was directed to one such earlier project already: Pipecut. Looks interesting! You may like to check it too! (Thanks @TronDD.)
• Other influences: I don't remember the fact too well already, but I'm rather sure that this must have been inspired in big part by The Bret Victor's Talk(s).

Future Ideas

• I have quite a lot of ideas for further experimentation of development of up, including but not limited to:
  • RIIR (once I learn enough of Rust... at some point in future... maybe...) — esp. to hopefully make up be a smaller binary (and also to maybe finally learn some Rust); though I'm somewhat afraid if it might ossify the codebase and make harder to develop further..? ...but maybe actually converse?...
  • Maybe it could be made into an UI-less, RPC/REST/socket/text-driven service, like gocode or Language Servers, for integration with editors/IDEs (emacs? vim? VSCode?...) I'd be especially interested in eventually merging it into Luna Studio; RIIR may help in this. (Before this, as a simpler approach, multi-line editing may be needed, or at least left&right scrolling of the command editor input box. Also, some kind of jumping between words in the command line; readline's Alt-b & Alt-f?)
  • Make it possible to capture output of already running processes! (But maybe that could be better made as a separate, composable tool! In Rust?)
  • Adding tests... (ahem; see also #1) ...also write --help...
  • Making it work on Windows, somehow? Also, obviously, would be nice to have some CI infrastructure enabling porting it to MacOSX, BSDs, etc., etc...
  • Integration with fzf and other TUI tools? I only have some vague thoughts and ideas about it as of now, not even sure how this could look like.
  • Adding more previews, for each | in the pipeline; also forking of pipelines, merging, feedback loops, and other mixing and matching (though I'd strongly prefer if Luna was to do it eventually).
• If you are interested in financing my R&D work, contact me by email at: [email protected], or on keybase.io as akavel. I suppose I will probably be developing the Ultimate Plumber further anyway, but at this time it's purely a hobby project, with all the fun and risks this entails.

— Mateusz Czapliński
October 2018

PS. The UP logo was conceived and generously sponsored by Thoai Nguyen and GPU Exchange, with a helping hand from Many Pixels.

up's Issues

add --version option with full credits

With short licensing info for all dependency libraries.

Add delay to unsafe-full-throttle mode

The new press-enter-to-run mode makes up a lot safer, but it would still be nice to have something like the old mode. Maybe a new mode where the command is automatically run, but only after a configurable amount of time without any input?

Probably still shouldn't be the default, but it would be a nice in-between option.

(Would happily dive in and make a PR to implement this, if you're interested!)

allow providing command as parameter — e.g.: up grep

please allow giving command to up as parameter (also to avoid the rm nightmare)

github: upload a darwin binary (tests succeeded)

Environment:

• macOS High Sierra (v10.13.6) - Darwin 17.7.0
• Go 1.11.1

Download & Compile

go get -u github.com/akavel/up

Test

man man |& up

Output

Add history

Every time return is hit, a copy of the pipeline is pushed to the history, allowing the user to go back to older version.

This lets one work on different parts, in different ways before getting to the final result

TCell $TERM not recognised

TCell seems to need to recognise the value of $TERM. There's an issue over at the TCell repo, but it's pretty old. This means that up won't work for many people using less-popular terminals (Kitty for me).

A workaround at the moment is to alias:

$ alias up="TERM=xterm up "

Is there some way to bypass this from the up side?

Allow configurable "no execute" regexes

I feel like a lot of us could be more comfortable using this if we could set a regex or some other way of preventing the program from previewing commands that might be destructive (rm, dd, shutdown, etc).

Run the pipeline up to the cursor only

given: very_large | filter_1 | filter_2 | filter_3 | filter_4

if the cursor is in the filter_2, only run: very_large | filter_1 | filter_2

why? when writing a series of filter it happens that after writing filter_4 the results contain data that one thought would be filtered/transformed by filter_1. It is of course possible possible to add another filter but maybe not logical. when changing filter_1, a view of the output would be very helpful but that's not possible without deleting the pipeline.

terminfo and kitty term

        TERM=xterm-kitty
was not found in the database provided by tcell library. Please try checking if
a supplemental database is found for your terminal at one of the following URLs:
        https://github.com/gdamore/tcell/raw/master/terminfo/database/98/98d2a8a0
        https://github.com/gdamore/tcell/raw/master/terminfo/database/98/98d2a8a0.gz
If yes, download it and save in the following directory:
        $HOME/.tcelldb/98/
then try running "up" again. If that does not work for you, please first consult:
        https://github.com/akavel/up/issues/15
and if you don't see your terminal code mentioned there, please try asking on:
        https://github.com/gdamore/tcell/issues
Or, you might try changing TERM temporarily to some other value, for example by
running "up" with:
        TERM=xterm up
Good luck!```

How to get it to work with `xterm-kitty` ? http://www.9bis.net/kitty/

Logo time

Hi there,

Our team at GPU Exchange uses this tool a lot and we want to give back to the project.

We sketched together some logo ideas for this (see below). Let us know which one you like most and I'll send over the source file for your future branding use.

Thanks again,

Input line not scrolling or wrapping when longer than the screen width

Hello.

Just discovered the tool and overjoyed about it.
However, being one prone to kilometre-long pipelines (which clutter my history with all the trial and errors, hence why I love the tool even more) the input line seems awfully short, in general, given how input lines longer than the screen width just go beyond the visible area, with no horizontal scrolling or line-wrapping allowing to see what one is writing.

Could you do something about that?

Thanks a bunch!

Logo proposal

Hello, I'm a designer and I want to know if I can create a logo for your project

add --help option describing usage

Write/copy it based on readme + update with features added in subsequent releases.

Make Ctrl-z work

Didn't work for me when I tried up on MacOS. Need to try if it works on Linux or not, then need to make it work.

snap package

Hi! I made a quick snap package for up, so that I could easily install/update the program on ubuntu.
It is classic (not confined) so it has access to the system as a normal program.
Are you interested in supporting this? If not can I publish it on the snap store (with a note saying it is unofficial)?

Add to homebrew for Mac

Given that it works on a mac #6, perhaps it can be submitted to homebrew so installation is super easy on a mac with:

brew install up

Just wanted to give you a thumbs up!

Love what you've done!

I wrote something similar (but not the same) a while back. https://github.com/sebglazebrook/infinite-pipe

I think stuff like this is super useful and really changed the way you approach comma d line usage.

So cool!

Feature Request: Shell Widget

FZF and Navi have the concept of shell widgets, they can take over your prompt and then when you're done leave you with the text you wanted filled-in to your prompt.

I think Navi's shell widget is a good example of what I'm talking about (see navi widget zsh, for instance). You can press ctrl-g, fill in a template, and when you're done your prompt is pre-populated with the command you selected and filled in. If you already have stuff filled in when you press ctrl-g then Navi takes that into account and starts a search for you.

It would be cool to, for instance, press ctrl-u with some command already filled in and have up take over at that point. Once done, exiting up would leave the prompt pre-filled with the final pipeline.

Mention AUR in installation methods

Someone already created a PKGBUILD for up so it's just a matter of yaourt -Syy up to install Ultimate Plumber on Arch/Manjaro systems. Thanks a bunch!

Unable to paste text longer than 11 characters

I tried to do some pipelining for comparison.
But when I tried to paste the content of my clipboard to this tools command prompt, only the first 11 characters got copied to it.

Here's the screencast in case needed.

Executing incomplete commands is too dangerous

This looks like a very useful tool, but I won't use it on my system as long as it automatically executes incomplete commands as they're being typed.

It shouldn't execute a command until it's explicitly told to do so. Hitting the TAB key might be one reasonable way to do that.

Allow to change colors

It would be great to allow to change colors. My terminal color scheme displays up command line in low contrast.

[idea] diff mode

A mode to compare the difference between the output of two files.

Like:

$ diff <(cat file1.txt |& ./up) <(cat file2.txt |& ./up)

It will be very useful to compare two different log of a process. For instance, looking for a bug comparing the correct and the incorrect log output.

Make Warning more prominent

At the risk of ignoring tongue in cheekiness; this warning seems too buried. Also there is a world of difference between "being careful using rm or dd" vs having a simple typo immediately be run as a command vs having to type something fully before hitting enter. I understand the impulse of closing this loop of iterate and see results; doing it this way though seems beyond dangerous.

WARNING: Please be careful when using it! It could be dangerous. In particular, writing "rm" or "dd" 
into it could be like running around with a chainsaw. But you'd be careful writing "rm" anywhere in 
Linux anyway, no? Also, why would you want to pipe something into "rm"? Other than that, I don't 
really have good ideas how to protect against cases like this. And in the other, non-dangerous 
cases, I find the tool immensely useful. If you have some ideas how to try to protect, please share! 
That said, a tool wouldn't be really Unixy if you couldn't hurt yourself with it, right? ;P

Add tab completion functionality

Including the ability to tab complete would make this tool truly perfect.

Great job either way, love this idea!

Adding checksum to release artifacts

Would it be possible to add a checksum.txt for release artifacts going forward? Its nice to be able to verify the builds for extra piece of mind. An example would be something like fabio: https://github.com/fabiolb/fabio/releases

Allow running up with no input

I like to run up to just run grep searches interactively (using ripgrep on the current directory).

Make Ctrl-W work to erase the word before the cursor.

It the terminal (with readline for example, Ctrl-W erase the command until finding a space, which is very practical and is missing in up.

[idea] show output from last successful command

While typing a command, when I'm in the middle, of course it's going to fail while I'm still writing it because it's not complete. In addition to showing the error from the command in progress, it would be great to also show the result of the previous command so I can remember what the output looks like. Cool tool though :)

Allow use on filename(s) without using stdin

I feel like it would make sense for up filename to be a valid invocation, with it having the same effect as cat filename | up.

Does it support tailing a file?

I'm not sure if this is the right place to ask. But tailing & using the grep filter on the fly would be super usefull.

Feel free to close the issue if out of place.

Run pipeline as nobody

A lot of posters are rightly worried about the safety risks. Why not just run the pipeline as a nobody user? Pipelines should be textual manipulations by default, so they don’t need any filesystem permissions.

For extra security, you could unshare networking to avoid accidental network access, although that seems less likely to occur entirely by accident.

list of cool usages

where is?

Readline movement support

It would be great if up would support my Readline movement commands.

https://www.gnu.org/software/bash/manual/html_node/Bindable-Readline-Commands.html

Thanks.

[idea] support for ANSI "color" escape sequences

Great tool! I have a non-critical feature request: It would be nice if up supported color codes such as those output by grep --color=always.
Thanks!

Chinese support got an issue

Hi, I am new to up and encountered an issue while using it.
When the file is in Chinese, the characters are not displayed normally in the program

$ cat 1.txt
吃饭
喝茶
睡觉

Inside up, it showed only one character

cat 1.txt | up 
 |
吃
喝
睡

And grep cannot work well either.

Use a persistent shell instance, preferably an interactive one

There are two missing pieces in up:

• Persistent shell: Currently up forks a new shell for each command. This is very inefficient (especially if you have a lot of shell functions and heavy dotfiles like I do), and also doesn't persist variables. Solving this one alone is not very hard; I have a library for Python for using a persistent shell, and a similar thing should be easily implementable here. We just need a socket (I currently use stdin/stdout for brish, but I plan to change this to proper sockets) to communicate between the interpreter and the host, and a helper server written in the interpreted shell that reads the commands the host sends and sends their results back.

• Autocompletion: Getting this to work is harder. I know of this zsh plugin that does this for zsh, but my experience of it has been slow, and buggy and it doesn't return all the completions. However, I think that if we can get the shell to run in an emulated terminal thingy, we should be able to use the shell's builtin autocomplete and also get it to persist at the same time. Now, that linked plugin is kinda doing this as well, though I haven't looked at it too deeply and anyways I don't know these gritty low-level details at all. The idea I have is using tmux. Solving the first problem with tmux should be easy (we can send the needed keystrokes to tmux. Its shell API is tmux send-keys ....), however, reading the results and also capturing completions triggered by pressing tab is harder. But I think this is the most promising avenue, and it works with any interpreter.

Add tests using tcell.SimulationScreen

https://godoc.org/github.com/gdamore/tcell#SimulationScreen

don't use hardcoded /bin/bash, allow other shells/engines

I use NixOS which does not have a /bin/bash, it only has /bin/sh. Would that work instead?

Safety: ptrace, seccomp, whitelists/blacklists

You're on Hacker News this morning: `https://news.ycombinator.com/item?id=26644110

Regarding safety issues users on HN have suggested using ptrace to intercept syscalls to simulate commands like "rm" and "dd". This seems like it might lead to some frustrating edge cases, though. Especially if you're trying to support MacOS and Linux.

https://news.ycombinator.com/item?id=26644508

Another user suggested using the seccomp syscall in Linux, pledge in BSD to disallow writes from the UP process.

https://news.ycombinator.com/item?id=26644319

Finally another user suggested using blacklists/whitelists to disallow certain destructive executables.

https://news.ycombinator.com/item?id=26644442

I think all three of these are good ideas with varying levels of complexity. It would protect users who don't realize that this tool can be destructive (or those of us who are absentminded) and perhaps make it possible to use a live search mode again.

ZSH integration

This is not an issue per se, but I wrote a small zsh function to integrate up:

zle-upify() {
    buf="$(echo "$BUFFER" | sed 's/[ |]*$//')"
    tmp="$(mktemp)"
    eval "$buf |& up --unsafe-full-throttle -o '$tmp' 2>/dev/null"
    cmd="$(tail -n +2 "$tmp")"
    rm -f "$tmp"
    BUFFER="$BUFFER | $cmd"
    zle end-of-line
}
zle -N zle-upify

bindkey '^U' zle-upify

If you add this to your zshrc, Ctrl+U will run up passing the current commands output, and when you exit up with Ctrl+X, the result will replace current prompt. It's slightly easier than dealing with upN.sh files.

It was quite useful for me, I'm just sharing this so maybe you'd like to put it to README.md or somewhere people can find out.