ywangd / stash

StaSh - Shell for Pythonista

StaSh - Shell Like an Expert in Pythonista

Inspired by shellista and its variants, StaSh is a serious attempt to implement a Bash-like shell for Pythonista.

Since its initial release, valuable contributions and advices have been received constantly from the Pythonista community. The two most popular utilities are pip (authored by @briarfox) and git (authored by @jsbain). Remarkable contributions are also made by @dgelessus, @pudquick, @oefe, @cclauss, @georg.viehoever, @BBOOXX, @bennr01, @glider-gun, @steljas, @zrzka, @seanld, @zed, @sdushantha and @ywang-bom.

StaSh stands for Pythonista Shell. While Sta may not be the best abbreviation for Pythonista, it forms a concise and meaningful word with the following Sh part. So the name StaSh was chosen to indicate it is a confined environment and great treasures may be found within.

Installation

StaSh can be easily installed via one line of python command (courtesy of @whitone).

import requests as r; exec(r.get('https://bit.ly/get-stash').content)

Simply copy the above line, paste into Pythonista interactive prompt and execute. It installs StaSh as a Python module under the site-packages folder (~/Documents/site-packages/stash) and copies a launching script, ~/Documents/launch_stash.py for easy access.

StaSh works with both Pythonista 2 and 3, though not all commands support python3.

If you have previous versions of StaSh installed (e.g. v0.4.x), You may need to restart Pythonista BEFORE the installation.

If you have a GitHub tool available in Pythonista, such as gitview or gitrepo, you can choose to directly clone or download the repository.

StaSh requires Pythonista v2.0 or Pythonista3 as the new ObjC feature is heavily used. For older Pythonista 1.5 compatible version, please refer to the v0.4 branch.

Starting with 0.7.4, StaSh supports being run on a PC using the tkinter module. This is intended for development purposes and may not offer you the best user experience. To install StaSh on your PC, either use the line above or clone this repository and run setup.py.

Upgrade

Once StaSh is installed, it can be easily updated by running the selfupdate command from within the shell.

• selfupdate defaults to the master branch. To update from a different branch, e.g. dev, use selfupdate dev.
• By default, selfupdate compares local and remote versions and only performs update if newer version is found. You can however force the update without version checking via selfupdate -f.
• To check for newer version without actually install it, use selfupdate -n.
• selfupdate manages StaSh installation folder and may delete files in the process. It is therefore recommended to not place your own scripts under $STASH_ROOT/bin. Instead, save your own scripts in~/Documents/bin or customise the locations with the BIN_PATH environment variable.
• You may need to restart Pythonista after the update for changes to take full effects.

selfupdate cannot be used for version 0.4.x and under. A fresh installation is needed. Version 0.7.0 requires a forced update. Please run selfupdate -f.

Notable Features

StaSh has a pile of features that are expected from a real shell. These features are what really set the difference from shellista.

• Panel UI program that is completely event driven

  • No blocking thread, builtin interactive prompt is accessible at all time
  • Consistent look and feel as a proper PC terminal
  • Almost all scripts can be called from within StaSh, including programs using UI and Scene packages.
  • Attributed text (color and style) support
  • Multiple sessions are possible by opening additional Panel tabs
  • Being a pure UI program, it is possible to launch and forget. The program stays active indefinitely. Non-UI scripts can only run for 10 minutes in background. But StaSh can stay up forever (till memory runs out due to other Apps). You can just launch StaSh to run a few commands and leave it. It will still be there for you when you return later.

• Comprehensive command line parsing and handling using pyparsing

  • Environmental variables, e.g echo $HOME, NAME=value
  • Aliases, e.g. alias l1='ls -1'
  • Single and double quotes behave like Bash, e.g. "*" means literal *, "$HOME" expands while '$HOME' does not.
  • Backslash escaping, e.g. ls My\ Script.py
  • Glob, e.g. ls ~/*.py
  • Backtick quotes for subprocess, e.g. touch `ls *.py`
  • Pipes to chain commands, e.g. find . -name "*.txt" | grep interesting
  • IO redirect (actually just Output redirect), e.g. ls *.py > py_files.txt. Input redirect can be achieved by using pipes.
    • It is possible to redirect to the Pythonista builtin console, e.g. ls > &3
  • Bang(!) to search command history, e.g. ls -1, !l. Bang commands like !! and !-1 also works.

• Smart auto-completion just as expected

  • One UI button, Tab, is provided to enable command line auto-completion.
  • It is smart to complete either commands or files based on the cursor position
  • It also completes environment variables and aliases.
  • It also features a sub-command auto-completion system. For an example, type git sta and press Tab. It will auto-completes to git status . You can easily add your own sub-commands completion via JSON files.

• Thread management allows multiple commands running in parallel

  • One foreground jobs and unlimited number of background jobs can run simultaneously.
  • A foreground job can be stopped by pressing the CC button or Ctrl-C on an external keyboard.
  • A background job is issued by appending an ampersand character (&) at the end of a normal command, e.g. httpserver &. It can be terminated by the kill command using its job ID.
  • A few utilities are provided for thread management.
    • jobs to list current running background jobs.
    • kill to kill a running job.
    • fg to bring background jobs to foreground
    • CZ button (Ctrl-Z) to send a foreground job to background

• Command line history management. Three UI buttons are provided to navigate through the history.

• On-screen virtual keys - an extra row of keys on top of the on-screen keyboard to provide control functions and easier access to symbols

  • Virtual keys for control functions including:
    • Tab - command line auto-completion
    • CC (Ctrl-C) - terminate the running job
    • CD (Ctrl-D) - end of Input
    • CU (Ctrl-U) - kill line
    • CZ (Ctrl-Z) - Send current running foreground job to background
    • KB - show/hide keyboard
    • H - display a popup window to show command history
    • Up - recall the previous command in history
    • Dn - recall the next command in history
  • Customisable virtual keys for commonly used symbols, e.g. ~/.-*|>.
    • The Symbols can be customized via the VK_SYMBOLS option in stash config file (default is .stash_config).

• Swipe on the virtual key row to position cursor (similar to what Pythonista builtin editor offers)

• External keyboard support

  • Tab key for auto-completion
  • Up (↑) / Down (↓) for navigating through command history
  • Ctrl-A and Ctrl-E to jump to the beginning and end of the input line, respectively
  • Ctrl-U to erase the input line
  • Ctrl-W to erase one word before cursor
  • Ctrl-L to clear the screen

• You can run (almost) any regular python scripts from within StaSh

  • There is no need to customize them for the shell. If it can be executed by a python interpreter via python your_script.py, you can just call it from within StaSh by just typing your_script
  • The shell object is made available to scripts being called. This enables a range of complex interactions between the shell and called scripts. For an example, the running script can use the shell object to execute more commands, e.g. _stash('pwd').

• You can give it a resource file, similar to .bashrc, to customize its behaviour. Like the Bash resource file, aliases, environment variables can be set here. The default resource file is .stashrc under StaSh installation root (i.e. ~/Documents/site-packages/stash).

  • The prompt is customizable with the PROMPT environment variable.
    • \w - current working directory with HOME folder abbreviated as ~
    • \W - last path component of current working directory
    • All other strings are displayed literally
    • The default setting is PROMPT='[\W]$ '

• Easy self update to keep update with the development by running a single selfupdate command from within the shell.

• The UI can be configured via configuration file to customize its font size and color. The default config file is .stash_config or stash.cfg under StaSh installation root.

Usage

The usage of StaSh is in principle similar to Bash. A few things to note are:

• The search paths for executable scripts is set via an environment variable called BIN_PATH as PATH is used by the system. The default BIN_PATH is ~/Documents/bin:~/Documents/stash_extensions/bin:$STASH_ROOT/bin.

• The executable files are either Python scripts or StaSh scripts. The type of script is determined by looking at the file extensions ".py" and ".sh". A file without extension is considered as a shell script.

  • When invoking a script, you can omit the extension, StaSh will try find the file with one of the extensions. For an example, StaSh interprets the command selfupdate and find the file selfupdate.py to execute.
  • Files without extension won't show up as an auto-completion possibility.

• Commands can only be written in a single line. No line continuation is possible. However, multiple commands can be written in a single line by separating them with semicolons, e.g. ls -1 > file_list; cat file_list.

• There are many Python scripts provided along with StaSh. These scripts range from performing regular shell tasks to advanced utilities like ssh and git. Note the scripts are by no means complete when compared to a real Linux shell. The collection will be gradually expanded should the need arise. It is also expected and appreciated that the community would come up with more scripts. The pip command may also install new commands.

  • alias.py - Define or print aliases
  • cat.py - Print contents of file
  • cd.py - Change current directory
  • clear.py - Clear console
  • cowsay.py - configurable speaking cow
  • cp.py - Copy file
  • crypt.py - File encryption using AES in CBC mode
  • curl.py - Transfer from an URL
  • cut.py - Cut out selection portions of each line of a file
  • dropbox_setup.py - Configure dropbox accounts for other commands
  • du.py - Summarize disk usage of the set of FILEs, recursively for directories
  • easy_config.py - UI for configuring stash
  • echo.py - Output text to console
  • edit.py - Open any text type files in Pythonista editor
  • find.py - Powerful file searching tool
  • fg.py - Bring a background job to foreground
  • ftpserver.py - A simple FTP server
  • gci.py - Interface to Python's built-in garbage collector
  • gh.py - GitHub interface
  • git.py - Git client ported from shellista
  • grep.py - search contents of file(s)
  • head.py - Display first lines of a file
  • httpserver.py - A simple HTTP server with upload function (ripped from https://gist.github.com/UniIsland/3346170)
  • jobs.py - List all jobs that are currently running
  • kill.py - Terminate a running job
  • latte.py - package manager
  • ls.py - List files
  • mail.py - Send emails with optional file attachment
  • man.py - Show help message (docstring) of a given command
  • mc.py - Easily work with multiple filesystems (e.g. local and FTP) synchronously.
  • md5sum.py - Print or check MD5 checksums
  • mkdir.py - Create directory
  • monkeylord.py - Manage monkey patches with the goal to make Pythonista more viable
  • more.py - Display output one screen page at a time
  • mount.py - Mount filesystems
  • mv.py - Move file
  • openin.py - Show the open in dialog to open a file in external apps.
  • pbcopy.py - Copy to iOS clipboard
  • pbpaste.py - Paste from iOS clipboard
  • ping.py - Ping remote hosts
  • pip.py - Search, download, install, update and uninstall pure Python packages from PyPI.
  • printenv.py - List environment variables
  • printhex.py - Print hexadecimal dump of the given file
  • pwd.py - Print current directory
  • python.py - Run python scripts or modules
  • python3.py - Run python3 scripts or modules
  • quicklook.py - iOS quick look for files of known types
  • rm.py - delete (remove) file
  • rmdir.py - delete (remove) directories
  • scp.py - Copy files from/to remote servers.
  • selfupdate.py - Update StaSh from its GitHub repo
  • sha1sum.py - Print of check SHA1 checksums
  • sha256sum.py - Print of check SHA256 checksums
  • sort.py - Sort a list, also see unique
  • source.py - Evaluate a script in the current environment
  • ssh.py - SSH client to either execute a command or spawn an interactive session on remote servers. pyte is used for terminal emulation and gives the command the feel of a full-fledged SSH client.
  • ssh-keygen.py - Generate RSA/DSA SSH Keys.
  • stashconf.py - Change StaSh configuration on the fly
  • tail.py - Print last lines of a FILE.
  • tar.py - Manipulate archive files
  • touch.py - Update timestamp of the given file or create it if not exist
  • totd.py - Print a random tip
  • umount.py - Unmount filesystems
  • uniq.py - Remove duplicates from list, also see sort
  • unzip.py - Unzip file, also see zip
  • version.py - Show StaSh installation and version information
  • wc.py - Line, word, character counting
  • webviewer.py - Open a url in the webbrowser
  • wget.py - get data from the net
  • whatis.py - Show a description for some of the commands
  • which.py - Find the exact path to a command script
  • wol.py- Wake on LAN using MAC address for launching a sleeping system
  • xargs.py - Command constructing and executing utility
  • zip.py - Zip file, also see unzip

Acknowledgements

• Pythonista is a wonderful piece of software.
• StaSh is inspired by shellista and its variants, including ShellistaExt and ShellistaUI.
• The UI part of StaSh has its root from ShellistaUI.
• Many of the command scripts, e.g. ls.py, cp.py, mv.py, are taken from ShellistaExt with some modifications.

Known Issues

• Pickled objects are not restored correctly and generate AttributeError as if the class definition cannot be found.
• Some commands may still not fully support python3.

Contributing

• Check any open issues or open a new issue to start discussions about your ideas of features and/or bugs
• Fork the repository, make changes, and send pull requests
  • Please send pull requests to the dev branch instead of master

stash's Issues

pythonista and edit module similarity

• pythonista module opens a file for editing in pythonista
• edit module opens a temp file in the editor and requires confimation on saving and allows a rename.

Should these two modules be merged into one?

Background job (&) to be handled by ui.in_background

The good thing about using ui.in_background for background jobs is that they are executed in the main interpreter's thread. This means they can be stopped by tapping the X button on the interactive prompt page. The X button fires a KeyboardInterupt exception. This allows house-keeping functions to kick in.

A downside of this approach is that the shell has absolutely no control of the background jobs. It is then up to users how they wanna use it.

Question: what does the Sta in StaSh stand for?

Startup arguments

Stash could use startup arguments to control some behaviors without modifying the source code, e.g. the DEBUG settings, flag to not-loading-external-rcfile etc.

Also need a debug print function so the DEBUG setting is only tested in one place.

Export a simple stash run function to external script

So external commands can do something like

_('cp afile bfile')

to ask stash to run the command.

$? should not be listed by env command

The special variable $? should not be listed by the env command.

Undefined variable should expand to empty string

$A should expand to '' if it is not defined. It is currently expand to itself.

find command

A find command similar to the Linux command of the same name. It is useful for various tasks. Pythonista's builtin find UI is very limited as it only search for current directory excluding sub-directories.

Parser to handle comments and empty lines

Allow external scripts to register callbacks for UI actions

For an example, if a button is tapped, fire the callback function to notify the external script.

Deletion at the start of input line should not move cursor to the end

Command completion after leading assignment does not work

e.g. A=1 l, press tab

More comprehensive history management

The current history (!) function is very minimal, only !search_string, !!, !-1.
A more comprehensive history sub-command system could be useful?

pip module

The pip module is almost done. Does anyone know if pypi modules are only tar.gz or are .zip used as well? I have not been able to find a .zip example.

Currently pip is using a .sh script to install the pypi package since setup.py will not work in pythonista. The .sh uses common stash modules. Is this exceptable or should pip handle ungziping, mkdir and rm by itself? Currently the .sh uses: tar, mkdir, mv and rm stash modules.

Runtime stack simplification

The first member in the stack should not be too special. It should be the same as all other members except the environment persisting part.

Milestone Changes - Unified Input/Output areas with a single TextView

@briarfox @dgelessus @jsbain
I am very excited to announce a major change to the UI design. The user input and output areas are now unified by a single TextView. This is more like a real terminal (no more split input and iutput areas). The advantages are significant:

• No more screen flashing when scrolling. Smooth scroll all the time.
• Tab completion support for external keyboard (still working on arrow keys)
• Cursor position manipulation (allow symbols from virtual keys to be inserted in the middle of a line)
• A long input line is now easy to manage as it automatically wraps
• Show/Hide keyboard easily with a button tap
• Device rotation no longer breaks the UI
• Possible to emulate text editors directly in the terminal
• Consistent look and feel to computer terminals

The changes are pushed to a new branch single-tv. Please give it a go when convenient (just type SELFUPDATE_BRANCH=single-tv selfupdate and restart StaSh afterwards).

Before merge this branch to dev, I'd like to wrap up current version and tag version 0.3.0 (which will be the last version to have the split UI design). The unified TextView change will be released as version 0.4.0.

Suggestions/comments are welcome.

@briarfox I also made a few changes to ssh.py to suit the new UI.

multi-sessions

This would make stash more like a real deal. It will be a long-term goal. It does seems possible from the first look at it.

Auto-completion at cursor position

The completion should be done for the word fragment at the cursor position.

Add PYTHONPATH support

Allow stash to add new path to sys.path by setting environment variable PYTHONPATH.

Add more virtual key buttons for commonly used symbols

Symbols like ~ / . * $ - _ < >

Use cStringIO and type comparison

cStringIO is available and should be faster than StringIO. But it does not allow comparison like isinstance(io, StringIO) as StringIO from cStringIO is a function instead of a class.

Necessity of bh.py

As I understand it bh.py is meant to work like /dev/null in that it eats any input it receives. However a command like echo "foobar" > /dev/null works just fine - Apple does not seem to block access to the real /dev/null. Do we still need bh.py for anything then?

argparse in python.py

Submitted a pull request for this. Please close.

Remove complete_command out of the loop

So that commands after the erroneous one do not run, e.g. echo Success should not run if previous wget ... fails. EDIT: Bash in fact does NOT abort on error!! So it is not necessary at all?!

complete_command is no longer necessary after the generator changes.

Allow comments in shell scripts

enough said ...

Strange problem in git

I'm stumped here. Git add doesn't work within stash. It works from the console, or long pressing play.
I put a pdb.set_trace, and traced within stash! and everything seemed to go through fine, until it tried to write to the index.
Is there any reason writing files would work within the stash environment? I recall some discussion about not using os.getcwd?

Interleave shell script with python code

So that the shell script could be more useful. Maybe create a template language, similar to Django's templating system.

git command

I personally didn't have much success in using the git command in shellista. I can clone but not able to pull after updates to original repo. It would be great to have a git command that perform at least following tasks:

• clone
• pull
• commit and push
• status

Since the git command requires external libraries (dulwich and gittle as I understand), I didn't want to work on it before there is way to manage libraries. Now that @briarfox is working on the pip module, maybe it is finally possible to work on git once pip is ready.

Stash function for installing pypi packages.

I like how stash updates and installs the /bin modules by executing a .sh file. I used this to import pyte inside ssh. It would be nice if stash exposed a function to replicate pulling packages off of pypi. I am currently working on a pip module for stash which could be used for this, but it may be better to have an accessable function in stash for it. Here is an example of the function I'm using to impor pyte.

def get_pyte():
    import tempfile
    commands = '''
    echo StaSh ssh installing pyte...
    wget https://codeload.github.com/selectel/pyte/zip/master -o ~/Documents/site-packages/pyte.zip
    mkdir ~/Documents/site-packages/pyte_folder
    unzip ~/Documents/site-packages/pyte.zip -d ~/Documents/site-packages/pyte_folder
    rm -r ~/Documents/site-packages/pyte.zip
    mv ~/Documents/site-packages/pyte_folder/pyte ~/Documents/site-packages/
    rm -r ~/Documents/site-packages/pyte_folder
    echo done
    '''
    temp = tempfile.NamedTemporaryFile()
    try:
        temp.write(commands)
        temp.seek(0)
        globals()['_stash'].runtime.exec_sh_file(temp.name)
    finally:
    # Automatically cleans up the file
        temp.close()

Readme update for v0.2.0

To reflect the latest changes.

Allow external scripts to take control of input text field, history and completion

Make it plug and play?

xargs command

A xargs command will be useful to enable some basic batch processing, e.g. deleting all temp files through the entire directory tree.

Add the `which` command

It also helps debugging.

Optimize the parser and make it more faithful to Bash

Currently the parser's workflow is still quite convoluted. I need to refer to Bash again and set it straight.

One Command to rule them all, One Command to find them ...

Forked from discussions on #34, it seems we could use a command that manages other command scripts.

Here are some ideas on how this could be achieved:

• A command, lets say, scsm (stash command script manager) pcsm (Pythonista Command Script Manager) or maybe lotc (the Lord of the Commands) :)
• An index hosted on GitHub which is a registry for available command scripts. It can simply be a json file, e.g.:

{
    'awesome_command_name': {
        'url': 'http://.....'  // the url to retrieve the script
        }
    },
    'another_awesome_command' : {
        ...
    }
}

• Some example usage of scm could be as follows:
  • pcsm list
  • pcsm install awesome_command_name
  • pcsm remove awesome_command_name
  • `pcsm info awesome_command_name

One obvious benefit is that script authors can host their own repos. They do have to register on the index. But the index url itself can be customized to point a different location. So it is very flexible.

One missing feature from the above design is command versioning. I am not sure what is the best way to deal with this unless we assume all commands are hosted on GitHub.

Runtime should be fully stack aware.

To emulate sub-shell environment.

bin/*.py files need docstrings

bin/man.py needs docstrings from the other files in /bin to populate thier manual pages.

Utility folder for store commonly used helper functions

For an example, un-expanduser is a common need by many other programs like pwd, ls, etc.

openin command hangs

The openin command freezes the entire Pythonisata app when running inside StaSh.

Tried to run console.open_in() with ui.in_background. However this only works for the first time openin is called. A second call to openin still freezes the app.

May need to try manually put it into another thread.

Cat barfs on binary files

Binary files make cat unhappy, complaining about UTF8 not being able to decode. For instance, cat .git/index in a git repo.

Proposed fix, just replace no printable with blanks. I'm honestly not sure what Unix normally does...or maybe we need a strings command, etc?
jsbain@9528102#diff-0

Replace input TextField with one-line TextView

TextView has more functionalities compared to a TextField. Textfield does not even respond to the "Tab" key on an external keyboard ... With a TextView, it is more likely to support Tab completion etc. on an external keyboard?

Convert all unicode to str before calling python command scripts?

Because Pythonista (or iOS) UIs by default use unicode, all the inputs read from the TextField is of unicode type. In the meanwhile, Python 2.x is not fully unicode compatible. Therefore a rather elusive bug is generated when a Python script wants strictly a str but an unicode is provided. Although values of the two types are essentially the same, e.g. 'SimpleHTTPServer' v.s. u'SimpleHTTPServer', the script sees the type difference and errors out.

This issue has been encountered twice by @briarfox with the python command and @jsbain with the git command (#57).

As @jsbain suggested, this bug breaks the paradigm that generic python scripts could run in stash without changes. So this time I am really thinking about converting all the unicode inputs to str type before passing them to external command scripts. @dgelessus

Your comments and suggestions are highly appreciated.

Cut release for v0.3.0

I plan to tag v0.3.0 release based on the current dev branch in a day or two. README and CHANGES are both updated. Please let me know if there are any other things you'd like to be in this release.

Work towards the next 0.4.0 version will still be done on the dev branch after the release cut.

Multiple statements in one line not working

A=42;  echo $A

The above code generates empty output instead of 42. This is because the entire line is parsed and expanded in one go before any of the statements are executed. So the $A is expanded before the A=42 is executed. This leads to the empty expansion.

In principle, multiple statements in one line should work the same as if they are entered in separate lines, i.e.

A=42
echo $A

The similar bug also affects other statements like:
alias l1='ls -1'; l1
The bug causes an error for the usage of l1.
EDIT: Bash in fact does the same thing. So it is not an issue.

Send pull request to dev branch instead of master

@briarfox @dgelessus
I think it is better to make the master branch relatively stable. Development works are better done in the separate dev branch. When we make a release cut, code in dev branch is merged to master. The code in master branch is mostly static except bug fixes (but no new features). I think this setup is more friendly and less confusing to end-users.

Since default selfupdate branch is master, as developers, you'll have to config the branch to dev via the environment variable SELFUPDATE_BRANCH, i.e. put SELFUPDATE_BRANCH=dev to your .stashrc file.

I'd like to make this happen for the next v0.3.0 release. So please send future pull requests to dev branch. Please let me know your thoughts. Thanks!

Auto-completion for sub-commands

For an example, git has many sub-commands (status, status, add etc.), it would be helpful to make them auto-completable.

gunzip, gzip modules missing.

Need modules to handle .tar.gz compressed files.

Rotating device makes input TextField barely visible

How to handle device rotation?

A systematic way to pass exit value

So Stash can know what to do.

Print exception type for py file exceptions, and perhaps traceback

E.g. Something like

    except Exception as e:
        err_msg = '%s: %s (%s)\n' % (str(e.type), str(e), sys.exc_value)

In exec_py_file()

Possibly, it might be good to also print the traceback, at least one or two levels.