amoffat / sh Goto Github PK

pbs.py in ()
40
41 VERSION = "0.4"
---> 42 IS_PY3 = sys.version_info.major == 3
43
44 if IS_PY3: raw_input = input

AttributeError: 'tuple' object has no attribute 'major'

in python 2.6 it seems there is no ".major"

In [2]: sys.version_info
Out[2]: (2, 6, 5, 'final', 0)

Is there any way to pipe in a Python string (or readable file object) to a command? Looked through the source and I noticed this doesn't seem to be possible.

dusty:pbs $ python
Python 3.2.3 (default, Apr 23 2012, 23:14:44) 
[GCC 4.7.0 20120414 (prerelease)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pbs
>>> pbs.ifconfig
/sbin/ifconfig
>>> pbs.__version__
'0.108'
>>> pbs.ifconfig()
<RunningCommand '/sbin/ifconfig', pid:22849, special_args:{'bg': False, 'ok_code': [0], 'err': None, 'err_to_out': None, 'in': None, 'env': environ({'XDG_DATA_HOME': '/home/dusty/.local/share', 'DE': 'xfce', 'LOGNAME': 'dusty', 'USER': 'dusty', 'INPUTRC': '/home/dusty/.inputrc', 'HOME': '/home/dusty', 'AWT_TOOLKIT': 'MToolkit', 'PATH': '/home/dusty/bin:/home/dusty/bin:/usr/loca /bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/android-sdk/platform-tools:/opt/android-sdk/tools:/usr/bin/core_perl:/opt/jython /usr/local/bin:/opt/jython:/usr/local/bin', 'PS1': '$(hg_ps1) \\[\\e[1;34m\\]\\u:\\W \\$\\[\\e[0m\\] ', 'HG': '/usr/bin/hg', 'LANG': 'en_CA.UTF-8', 'LESS_TERMCAP_se': '\x1b[00;00m', 'TERM': 'xterm', 'SHELL': '/bin/bash', 'DJANGO_SETTINGS_MODULE': 'settings', 'XDG_SESSION_COOKIE': 'a66397ab914fe147f8101978000008d9-1342909192.908614-430473562', 'SHLVL': '1', 'LESS_TERMCAP_me': '\x1b[00;00m', 'LESS_TERMCAP_md': '\x1b[01;34m', 'G_BROKEN_FILENAMES': '1', 'LESS_TERMCAP_mb': '\x1b[00;34m', 'DISPLAY': ':0.0', 'WINDOWID': '27305580', 'EDITOR': 'subl -w', 'DESKTOP_STARTUP_ID': '', 'MPD_HOST': 'mpdrocks@localhost', 'JAVA_HOME': '/usr/lib/jvm/java-6-openjdk', 'HISTFILESIZE': '5000', 'XDG_CONFIG_HOME': '/home/dusty/.config', 'PYTHONPATH': '.', 'XDG_CACHE_HOME': '/home/dusty/.cache', 'J2SDKDIR': '/usr/lib/jvm/java-6-openjdk', 'ANDROID_HOME': '/opt/android-sdk', '_': '/usr/bin/python', 'XAUTHORITY': '/home/dusty/.Xauthority', 'HISTIGNORE': '\\&:fg:bg:ls:pwd:cd ..:jobs:ls -l:ls -a:%1:%2:popd:pushd:', 'LESS_TERMCAP_ue': '\x1b[00;00m', 'J2REDIR': '/usr/lib/jvm/java-6-openjdk/jre', 'XDG_CONFIG_DIRS': '/etc/xdg', 'MOZ_PLUGIN_PATH': '/usr/lib/mozilla/plugins', 'OLDPWD': '/home/dusty', 'ANDROID_SWT': '/usr/share/java', 'HISTCONTROL': 'ignoredups', 'XDG_DATA_DIRS': '/usr/local/share/:/usr/share/', 'PWD': '/home/dusty/code/pbs', 'LESS_TERMCAP_us': '\x1b[01;32m', 'COLORTERM': 'Terminal', 'MAIL': '/var/mail/dusty', 'LESS_TERMCAP_so': '\x1b[01;31m', 'LS_COLORS': 'no=00:fi=00:di=36:ln=35:pi=30;44:so=35;44:do=35;44:bd=33;44:cd=37;44:or=05;37;41:mi=05;37;41:ex=01;31:*.cmd=01;31:*.exe=01;31:*.com=01;31:*.bat=01;31:*.reg=01;31:*.app=01;31:*.txt=32:*.org=32:*.md=32:*.mkd=32:*.h=32:*.c=32:*.C=32:*.cc=32:*.cxx=32:*.objc=32:*.sh=32:*.csh=32:*.zsh=32:*.el=32:*.vim=32:*.java=32:*.pl=32:*.pm=32:*.py=32:*.rb=32:*.hs=32:*.php=32:*.htm=32:*.html=32:*.shtml=32:*.xml=32:*.rdf=32:*.css=32:*.js=32:*.man=32:*.0=32:*.1=32:*.2=32:*.3=32:*.4=32:*.5=32:*.6=32:*.7=32:*.8=32:*.9=32:*.l=32:*.n=32:*.p=32:*.pod=32:*.tex=32:*.bmp=33:*.cgm=33:*.dl=33:*.dvi=33:*.emf=33:*.eps=33:*.gif=33:*.jpeg=33:*.jpg=33:*.JPG=33:*.mng=33:*.pbm=33:*.pcx=33:*.pdf=33:*.pgm=33:*.png=33:*.ppm=33:*.pps=33:*.ppsx=33:*.ps=33:*.svg=33:*.svgz=33:*.tga=33:*.tif=33:*.tiff=33:*.xbm=33:*.xcf=33:*.xpm=33:*.xwd=33:*.xwd=33:*.yuv=33:*.aac=33:*.au=33:*.flac=33:*.mid=33:*.midi=33:*.mka=33:*.mp3=33:*.mpa=33:*.mpeg=33:*.mpg=33:*.ogg=33:*.ra=33:*.wav=33:*.anx=33:*.asf=33:*.avi=33:*.axv=33:*.flc=33:*.fli=33:*.flv=33:*.gl=33:*.m2v=33:*.m4v=33:*.mkv=33:*.mov=33:*.mp4=33:*.mp4v=33:*.mpeg=33:*.mpg=33:*.nuv=33:*.ogm=33:*.ogv=33:*.ogx=33:*.qt=33:*.rm=33:*.rmvb=33:*.swf=33:*.vob=33:*.wmv=33:*.doc=31:*.docx=31:*.rtf=31:*.dot=31:*.dotx=31:*.xls=31:*.xlsx=31:*.ppt=31:*.pptx=31:*.fla=31:*.psd=31:*.7z=1;35:*.apk=1;35:*.arj=1;35:*.bin=1;35:*.bz=1;35:*.bz2=1;35:*.cab=1;35:*.deb=1;35:*.dmg=1;35:*.gem=1;35:*.gz=1;35:*.iso=1;35:*.jar=1;35:*.msi=1;35:*.rar=1;35:*.rpm=1;35:*.tar=1;35:*.tbz=1;35:*.tbz2=1;35:*.tgz=1;35:*.tx=1;35:*.war=1;35:*.xpi=1;35:*.xz=1;35:*.z=1;35:*.Z=1;35:*.zip=1;35:*.ANSI-30-black=30:*.ANSI-01;30-brblack=01;30:*.ANSI-31-red=31:*.ANSI-01;31-brred=01;31:*.ANSI-32-green=32:*.ANSI-01;32-brgreen=01;32:*.ANSI-33-yellow=33:*.ANSI-01;33-bryellow=01;33:*.ANSI-34-blue=34:*.ANSI-01;34-brblue=01;34:*.ANSI-35-magenta=35:*.ANSI-01;35-brmagenta=01;35:*.ANSI-36-cyan=36:*.ANSI-01;36-brcyan=01;36:*.ANSI-37-white=37:*.ANSI-01;37-brwhite=01;37:*.log=01;32:*~=01;32:*#=01;32:*.bak=01;33:*.BAK=01;33:*.old=01;33:*.OLD=01;33:*.org_archive=01;33:*.off=01;33:*.OFF=01;33:*.dist=01;33:*.DIST=01;33:*.orig=01;33:*.ORIG=01;33:*.swp=01;33:*.swo=01 33:*,v=01;33:*.gpg=34:*.gpg=34:*.pgp=34:*.asc=34:*.3des=34:*.aes=34:*.enc=34:'}), 'fg': False, 'with': False, 'cwd': None, 'out': None}

Note that it is outputting some kind of env information, and that it is also not performing the correct command or providing it's output. This happens no matter what pbs command I try to run. This happens in both python2 and python 3, under Arch Linux on a Lenovo Thinkpad T510. It's been happening for some time; since it hasn't been fixed yet, I'm assuming the problem is Arch specific (I've experienced it under both 32 bit and 64 bit Arch).

Using git bisect, I've traced this bug to this commit:

https://github.com/amoffat/pbs/commit/8d61b31d5acd6baba4584d427d0721a5c731a5d5

All commits after and including this commit exhibit this behavior. Earlier commits do not. It seems like an innocuous commit, but something about it is making pbs unusable on my system.

Redirect stderr or stdout to a StringIO or a file must be almost the same process, but is not working with StringIO:

In [1]: from pbs import Command

In [2]: from StringIO import StringIO

In [3]: s = StringIO()

In [4]: ls = Command('ls')

In [5]: ls(_out=s)
Out[5]: 

In [6]: s.bu
s.buf      s.buflist  

In [6]: s.buf
Out[6]: ''

In [7]: s.bu
s.buf      s.buflist  

In [7]: s.buflist
Out[7]: []

In [8]: ls()
Out[8]: 
build
dist
ext

after adding the windows support I I notice this issue:

pbs.ipconfig("/?")
Traceback (most recent call last):
File "", line 1, in
File "c:\Python27\lib\site-packages\pbs.py", line 310, in call
if rc != 0: raise get_rc_exc(rc)(self._command_ran, self.stdout, self.stderr)
pbs.ErrorReturnCode_1:

Ran: 'C:\WINDOWS\system32\ipconfig.exe /D'

didn't under stood why my /? turned into /D
after debugging it a bit a noticed that this is cause by the call to glob

I think that should be a why to disable globbing
similar to bash "set -o noglob"

I don't know if it should be module wide, or per command configuration (maybe both options)

I just tried to do this:

import sys
from pbs import ls
ls(_err=sys.stderr, _out=sys.stdout)

and get the error: RuntimeError: maximum recursion depth exceeded

A simple A | B pipeline works:

from pbs import ls, du, sort, head, HOME
y = sort(du(HOME), "-nr")
print y

But a still-simple A | B | C pipeline does not:

from pbs import ls, du, sort, head, HOME
y = head(sort(du(HOME), "-nr"))
print y

Dies with a broken pipe OS error. (using Python 2.7.1 on Mac OS X Lion 10.7.3 (11D50b))

Traceback (most recent call last):
  File "pbstest.py", line 8, in <module>
    y = head(sort(du(HOME), "-nr"))
  File "/Library/Python/2.7/site-packages/pbs.py", line 424, in __call__
    return RunningCommand(command_ran, process, call_args, actual_stdin)
  File "/Library/Python/2.7/site-packages/pbs.py", line 134, in __init__
    self._stdout, self._stderr = self.process.communicate(stdin)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 740, in communicate
    return self._communicate(input)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1259, in _communicate
    stdout, stderr = self._communicate_with_select(input)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1361, in _communicate_with_select
    bytes_written = os.write(self.stdin.fileno(), chunk)
OSError: [Errno 32] Broken pipe

The PBS keyword command manipulation is a beautiful feature, the translation of python keywords to "--" options makes this feel even more pythonic.

However, python will not allow non-keyword arguments after keyword arguments. This is unfortunately contrary to many commands in linux. For example GPG,

Perhaps could a end_args or tail_args or other better named keyword argument be set up so the python version can naturally order these the way the command would? Worth pondering.

All tests pass on ubuntu 12.4 but I've got 4 failures on the last debian stable (python 2.6.6):

======================================================================
FAIL: test_multiple_bakes (__main__.Basic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/sh/test.py", line 544, in test_multiple_bakes
    self.assertTrue(getpass.getuser() == out.strip())
AssertionError

======================================================================
FAIL: test_subcommand_and_bake (__main__.Basic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/sh/test.py", line 530, in test_subcommand_and_bake
    self.assertTrue(getpass.getuser() in out)
AssertionError

======================================================================
FAIL: test_with_context (__main__.Basic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/sh/test.py", line 429, in test_with_context
    self.assertTrue(getpass.getuser() in out)
AssertionError

======================================================================
FAIL: test_with_context_args (__main__.Basic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/sh/test.py", line 451, in test_with_context_args
    self.assertTrue(getpass.getuser() == out.strip())
AssertionError

----------------------------------------------------------------------
Ran 63 tests in 12.351s

FAILED (failures=4)

I got a SyntaxError trying to import pbs from python3:

$ python3 -m pbs
Traceback (most recent call last):
File "/usr/lib/python3.2/runpy.py", line 140, in _run_module_as_main
mod_name, loader, code, fname = _get_module_details(mod_name)
File "/usr/lib/python3.2/runpy.py", line 114, in _get_module_details
code = loader.get_code(mod_name)
File "/usr/lib/python3.2/pkgutil.py", line 281, in get_code
self.code = compile(source, self.filename, 'exec')
File "/home/dktrkranz/pack/py/pbs-0.7/pbs.py", line 183
else: return u""
^
SyntaxError: invalid syntax
$

>>> import pbs
>>>
>>> pbs.ifconfig("eth0", "_bh")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/pn/dvcs/pbs/pbs.py", line 520, in __call__
    if rc != 0: raise get_rc_exc(rc)(self._command_ran, self.stdout, self.stderr)
pbs.ErrorReturnCode_1: 

Ran: '/sbin/ifconfig en0 _bh'

STDOUT:



STDERR:

  ifconfig: _bh: bad value

So this results in an error from ifconfig, but not from pbs. I think it'd be worth it to make it possible to add a class "pbs.opt" to pass in options:

>>> import pbs
>>> from pbs import opt
>>> 
>>> pbs.ifconfig("eth0", opt('bh'))

and at this point, an exception can be raised. pbs.opt would have a strict list of options that could be passed in, and the right error is returned.

It doesn't need to obviate the current simple mechanism, but provides some extra safety and feedback to the user.

sh is really cool, good work.

I was messing around and had a problem with sh.man("bash").

Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sh
>>> sh.__version__
'1.0'
>>> man_bash = sh.man("bash")

Wait a few seconds

^C
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.6/dist-packages/sh.py", line 581, in __call__
    return RunningCommand(cmd, call_args, stdin, stdout, stderr)
  File "/usr/local/lib/python2.6/dist-packages/sh.py", line 237, in __init__
    self.wait()
  File "/usr/local/lib/python2.6/dist-packages/sh.py", line 241, in wait
    self._handle_exit_code(self.process.wait())
  File "/usr/local/lib/python2.6/dist-packages/sh.py", line 920, in wait
    pid, exit_code = os.waitpid(self.pid, 0)
KeyboardInterrupt

Same results with sh.man("bash", _tty_in)

This works:
b = subprocess.Popen(["man", "bash"], stdout=subprocess.PIPE).communicate()[0]

I have a situation where I want to call one program which can only accept a certain kind of input via a file which must be named in the arguments, but I want to feed it content generated from another program.

In other words, I have a situation that would be expressed in bash with a process substitution like this:

 tail -f <(echo "generated")

(Relevant: https://en.wikipedia.org/wiki/Process_substitution )

In python, I can solve this with a tempfile fairly easily.

A step better: I can also solve it with a named pipe with a mkfifo call fairly easily, which gives me the joys of in-memory rather than actually hitting the filesystem needlessly.

However, that still leaves something to be desired; I have to pick a name for my fifo, and I have to remove it again when I'm done. If I get SIGKILL, I leave a dangling fifo hanging around on my filesystem. What would really be excellent is if I could tap into the magic stuff in the /proc/$pid/fd and /dev/fd/$fd areas common in a linux world... that would give me a system where the kernel itself is functioning as my cleanup.

That example of process substitution in bash up above does something clever like that. If you run that example and then look at what actually happened with ps, you'll see something like this:

tail -f /dev/fd/63

Bash created a fifo somewhere where I don't have to worry about it (I think it's somewhere under /proc/ so it just goes away when the processes die?); stdout of the echo writes into the fifo and the reading end of the fifo is made into file descriptor 63 for tail. And then the "/dev/fd/63" part is magic that happens to be a name for the fifo that is fd 63 to the current process.

What would really be excellent is if I could tap into the same level of magic up in the python world.

In the course of writing this, I ended up realizing that I can use "/dev/fd/0" as an argument to get a program to read its own standard in as a file, and since I don't happen to be using stdin already in my current case, this solves my immediate problem. A more general solution would still be excellent, though, and for that we would need the ability to pass arbitrarily numbered file descriptors into child processes, instead of being limited to stdin/stdout/stderr aka 0/1/2.

Also, I'm not sure how portable the "/dev/fd/$fd" stuff is; I feel a little uncomfortable hardcoding that in, and bash takes care of it for me, but I have no idea how I'd go about finding out in a cross platform way what the location is for the magic filenames-to-selfprocess-file-descriptors.

Hi,

It didn't say if pbs was for python2 or python (3), so I tried both:

Python 3:

python pbs.py
File "pbs.py", line 251
print out
^
SyntaxError: invalid syntax

Python 2:

python2 pbs.py
Traceback (most recent call last):
File "pbs.py", line 261, in
frame, script, line, module, code, index = inspect.stack()[1]
IndexError: list index out of range

How should it be run?

I'm on 64-bit Arch Linux.

Thanks.

I ran into a really dastardly gotcha today. In bash, I would call the command like this:

control_cluster --connection 10.10.10.214:8000 --add-component checker checker01 10.10.10.105:5000

Mapping this to pbs seemed easy enough. First I created a baked "control_cluster" object, then tried to perform the call:

import pbs

cc = pbs.control_cluster.bake(connection="10.10.10.214:8000")
cc(add_component="checker checker01 10.10.10.105:5000")

Unfortunately this did not work because shlex was creating "--add-component=checker" followed by "checker01" and "10.10.10.105:5000", when in reality, add_component takes one argument, a string of 3 components. So I tried to escape some quotes:

cc(add_component="\"checker checker01 10.10.10.105:5000\"")

Didn't work. Some other things didn't work as well. It turns out, this was happening because of how pbs interprets a keyword argument. It plays it "safe" and assumes that a keyword argument resolves to the form "--name=value" What was needed however, was to resolve to the following form: "--name value". Once I figured that out, the correct call became obvious:

cc("--add-component", "checker checker01 10.10.10.105:5000")

So the gotcha is:

Only use a keyword argument to pass arguments to the program if you are positive resolving to the form "--name=value" will work.

Hopefully this helps someone else in the meantime while they wait on more documentation.

First, thank you for the code: it's really handy.

I'm experimenting with it but I found a problem with (short) arguments that have a value. For instance, to run

ls -l | cut -f 1 - d " "

with pbs I did

cut(ls('-l'),'-f 1', '-d " " ')

that yield an exception complaining about cut -d parameter that should be a single character. I guess the problem is that in your Popen you pass an array containing '-d " "' instead of two values '-d' and ' ' (as you should). I think that parsing arguments can be done with shlex, but I'm too lazy to investigate it further.

My fix is to replace the first two parameter of the Popen call with

' '.join(cmd), shell=True

and let the Popen do the magic. I'm not forking and asking you to pull such "fix" because I'm not sure it is in line with the treatment of args (long and short) you have in mind.

Hope this helps,
Massimo

P.S.
Your example shows also a bug, if I run

curl("http://duckduckgo.com/", "-o page.html", silent=True)

a file named ' page.html' gets created in the wd (notice the space in front of the name) that I guess is not the intended outcome.

The issue stems from using repr() and %r when parsing the commandline arguments. The problem is, repr escapes escapes, and unicode characters are escaped. So we can't use repr to easily handle escaping. Example of problem:

from pbs import echo
print echo(u"漢")
print echo("漢")

Prints

u\u6f22
\xe6\xbc\xa2

When it should be printing the actual unicode character. This is because the subprocess "echo" is getting something like u\u6f22 and \xe6\xbc\xa2 (double escaped). So that's bad. Need some more thought on this..

http://pypi.python.org/pypi/pbs has only 0.105

I am currently trying to work out how PBS can be used with a mocking set up to test tools built with it.
I regularly use unittest and the voidspace mock library with my code - and a pattern for mocking both the presence/behaviour of a tool, and mocking the absence of a tool (with the rejection exception).

For example (in a unittest.TestCase) a pattern for a not found program is:

@mock.patch("myapp.pbs.resolve_program")
def test_myapp(self, rp_mock):
   rp_mock.side_effect = pbs.CommandNotFound()
   ... test stuff ...

Is it possible to add new Environment Variables? or we have to use "export" command.

PBS generates dynamic class members, which is a great thing but can cause pydev, pylint and other checkers to fail.
Could attributes be added to the pbs module (in docs, comment and other) to prevent the most common python validators from complaining about these.

The readme is starting to get pretty unwieldy. It should get trimmed down to basic examples and put the comprehensive docs on http://readthedocs.org/

It might be cool to overload "|" for piping ops. I tried this at my branch. It would be cool to grow that into a proper feature.

this code:

from pbs import find
print find('-name', '*.mp3')

gives this warning:

yossi@ubuntu3:~$ python test.py 
/usr/local/lib/python2.6/dist-packages/pbs.py:207: DeprecationWarning: object.__init__() takes no parameters
partial.__init__(self, cmd)
<snip - paths to all the mp3s under my home directory>

using pbs 0.93 as obtained by pip install pbs

Python (in) Bash Scripts abstracts away the horrible syntax of bash and allows me to use Python awesomeness like this:

try:
    print ls("/some/non-existant/folder")
except ErrorReturnCode_2:
    print "folder doesn't exist!"
    create_the_folder()
except ErrorReturnCode:
    print "unknown error"
    exit(1)

However, this abstraction can leak in some cases, one of which I highlight below.

Considering the behaviour expected above, I assumed the following code will exhibit similar behaviour:

try:
    print ls("*.bin")
except ErrorReturnCode_2:
    print ".bin doesn't exist!"
    create_the_file()
except ErrorReturnCode:
    print "unknown error"
    exit(1)

Of course, the documentation is careful in telling us that globs should be handled by the user, and hence the code needs to change to:

try:
    print ls(pbs.glob("*.bin"))
except ErrorReturnCode_2:
    print ".bin doesn't exist!"
    create_the_file()
except ErrorReturnCode:
    print "unknown error"
    exit(1)

Unfortunately, the "expected behavior" (as can be assumed from the code) is not what PBS translates our code to: leading to disparity.

Assume that the current directory does not have any files that match the glob pattern: in that case, instead of returning an empty list/raising exception, ALL files are returned

Otherwise, it works "as expected" (only those files that match the glob are returned)

Consider:

$ python
Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41) 
[GCC 4.4.3] on linux2
>>> import pbs
>>> a=pbs.glob("*.bin")
>>> from pbs import ls
>>> ls(a)
bootstrap.py
bootstrap.py~
correctChefIssues.py
pbs.py
pbs.pyc
>>> b=pbs.glob("*.py")
>>> ls(b)
bootstrap.py
correctChefIssues.py
pbs.py

In the above scenario, no *.bin files existed but a few *.py did.

I hesitate to call this a bug: the demonstrated behaviour is consistent with the way PBS translates all commands, and it's trivial to work around this (by checking if len(pbs.glob("*.bin")) == 0)

PBS by itself is doing no wrong: the glob() returns [] and PBS faithfully executes ls([]) which is essentially ls on the command line and that lists all files (as does the "un-expected" output we see above)

However, this might cause some headaches for people, like me (!), naively assuming ls(pbs.glob('*.bin')) would translate to ls *.bin

This could perhaps be either explained in Create FAQ of common gotchas, have URLs to the most common ones listed as part of the Exception message,
or,
final_args could be checked for length 0 and an exception thrown, alerting the user to pass None as a parameter instead, to signal none vs empty command line arguments

When I say final_args, I am referring to:

https://github.com/amoffat/pbs/blob/master/pbs.py#L268

def _compile_args(self, args, kwargs):
        processed_args = []
...
        split_args = self._partial_baked_args + processed_args
...
        final_args = split_args
...
        FIXME: Is final_args empty? Hmm..
...
        cmd.extend(final_args)

This would make code like this work:

try:
    print ifconfig(interfaces_I_need_to_check())
except ErrorReturnCode_1:
    print "error fetching interface information: Device not found"
    handle_the_error()
except ErrorReturnCode:
    print "There are no interfaces to check"
    exit(1)

instead of requiring code like:

das_interfaces = interfaces_I_need_to_check()

if len(das_interfaces): # If we don't check this, then the output would be same as if we had selected all interfaces for checking (both of which are separate use cases)
    try:
        print ifconfig(das_interfaces)
    except ErrorReturnCode_1:
        print "error fetching interface information: Device not found"
        handle_the_error()
    except ErrorReturnCode:
        print "unknown error"
        exit(1)
else:
        print "There are no interfaces to check"

I do not want to distract you from the rewrite_popen branch, but your (and others') thoughts on this would be interesting.

... while 'time' is not a UNIX command; it is provided by the shell (bash does, but sh does not).

But it should probably be implemented in pbs in order to provide a full shell.

I think it would be cool (but maybe impossible) to be able to use "from pbs import *" from a module even if pbs is not main.

It would allow us to make scripts more easily, that's to say without importing/declaring all commands.

This is a problem, example:

import pbs as sh

with sh.sudo:
    sh.sed(sh.echo("omg"), "s/omg/lol/", _out="/root/wat")

This is because PBS internally handles open()ing the redirection files, so that it can pass the file descriptors to subprocess. Unfortunately, PBS doesn't have sudo access for the open(), only the commands PBS is running. One possible solution would be to give PBS sudo access for its operations, but this means making PBS handle all the authentication, and all the fun that goes along with that.

import pbs

pbs.vim(file_path)

If I run this, I can in my terminal that the script is doing something, but vim nevers opens te file.
It is as it is waiting or doing it in the background.

All other tests (with ls command or so) do work.

Is this kind of behaviour intended? As it worked before when just using the subprocess module. (subprocess.call('vim ' + file_path))

here is the command:

import pbs
pbs.convert("-crop +0+67", "some.jpg", "crop.jpg")

raises an error:

Traceback (most recent call last):
  File "test.py", line 5, in <module>
    pbs.convert("-crop +0+67", "some.jpg", "crop.jpg")
  File "/usr/lib/python2.7/site-packages/pbs.py", line 436, in __call__
    return RunningCommand(command_ran, process, call_args, actual_stdin)
  File "/usr/lib/python2.7/site-packages/pbs.py", line 142, in __init__
    self._handle_exit_code(self.process.wait())
  File "/usr/lib/python2.7/site-packages/pbs.py", line 207, in _handle_exit_code
    raise get_rc_exc(rc)(self.command_ran, self._stdout, self._stderr)
pbs.ErrorReturnCode_1: 

Ran: '/usr/bin/convert -crop +0+67 some.jpg crop.jpg'

STDOUT:



STDERR:

  convert: unrecognized option `-crop +0+67' @ error/convert.c/ConvertImageCommand/10

I found that subprocess.Popen(["/usr/bin/convert", "-crop", "+0+67", "some.jpg", "crop.jpg"]) is OK,
but subprocess.Popen(["/usr/bin/convert", "-crop +0+67", "some.jpg", "crop.jpg"]) is NOT.

Maybe this is not a bug, maybe it's subprocess's bug.
However, can psb take this situation into account and work around?

Thanks.

Best regards,
Du Yue

Because pbs assumes that the module that first imports comes from a readable .py file, it won't work when imported from modules where there's only a .pyc. For example:

test_a.py:

import test_b
test_b.main()

test_b.py:

from pbs import echo
def main():
    echo('Hello world!')

Run python test_a.py once, then delete python test_b.py (leaving only python test_b.pyc). Now, running test_a.py will crash:

Traceback (most recent call last):
File "test_a.py", line 2, in <module>
    import test_b
File "/home/me/pbs/test_b.py", line 2, in <module>
File "/home/me/pbs/pbs.py", line 419, in <module>
    with open(script, "r") as h: source = h.readlines()
IOError: [Errno 2] No such file or directory: '/home/me/pbs/test_b.py'

This could come up when a program is packaged somehow (e.g. frozen in a zip file).

All in all, the magic here is pretty fragile and will undoubtedly fail in other mysterious ways.
Maybe disallowing import * and allowing the above instead (and also import from the REPL) would be the lesser evil?

The PBS acronym is already taken by the Portable Batch System and its many forks. So, for instance, googling for "python pbs" yields an entire page of python code related to interfacing with said batch
system, before this PBS package.

The name clash makes things a bit awkward for those (like me) who work with Python and PBS (the batch system), since much code out there has already some pbs.py module.

May I suggest that the name is changed to something that avoids the obvious name clash? (e.g., "PyBS") That would also make googling for the package easier.

Quotes are not handled properly, or at least in a predictable fashion:

import pbs
pbs.echo('"foo"')
foo

pbs.echo('"foo"')
foo

$ echo "foo"
foo

$ echo "foo"
"foo"

This prevents one from using commands with quoted parameters, such as --filename "foo bar.jpg".

I've got a version working using context managers (so you can do the following):

with sudo:
  whoami()

Unfortunately, there's no way to provide arguments, since the with statement uses the constructor to handle arguments for the context manager. Doing something like:

with sudo("-p'>'"):
  whoami()

just causes an error because sudo is executed immediately. I'm still poking around at the code, so I can't presently think of a way to resolve this issue.

who can to write a PEP for putting pbs into the Standard Library ?
I know it might be a bit early, and pbs need to be a bit more mature for that...

But I was banging my head with subprocess, so many times until I got it right.
I need to head to documentation, every time I want to use subprocess.

with pbs it's much much more simple...

Andrew,

How do you recommend capturing stdout and stderr from programs that might return non-zero return codes?

The naive way echoes truncated stdout and stderr to the terminal.

I see two ways to capture them in the calling program:

1. Catch the exception and get stdout and stderr from the exception object:

   In [1]: import pbs

   In [2]: pbs.ls('.', 'non-existent-file')

   ErrorReturnCode_1 Traceback (most recent call last)
   /Users/mlm/w/pbs/ in ()
   ----> 1 pbs.ls('.', 'non-existent-file')

   /Users/mlm/w/pbs/pbs.py in call(self, _args, *_kwargs)
   444 stdin=stdin, stdout=stdout, stderr=stderr)
   445
   --> 446 return RunningCommand(command_ran, process, call_args, actual_stdin)
   447
   448

   /Users/mlm/w/pbs/pbs.py in init(self, command_ran, process, call_args, stdin)
   141 rc = self.process.wait()
   142
   --> 143 if rc != 0: raise get_rc_exc(rc)(self.command_ran, self._stdout, self._stderr)
   144
   145 def enter(self):

   ErrorReturnCode_1:

   Ran: '/bin/ls . non-existent-file'

   STDOUT:

   .:
   AUTHORS.md
   LICENSE.txt
   MANIFEST.in
   README.md
   pbs.py
   pbs.pyc
   setup.py
   test.py

   STDERR:

   ls: non-existent-file: No such file or directory

   In [3]: try: pbs.ls('.', 'non-existent-file')
   except pbs.ErrorReturnCode as e:
   print e.stdout
   ...:
   .:
   AUTHORS.md
   LICENSE.txt
   MANIFEST.in
   README.md
   pbs.py
   pbs.pyc
   setup.py
   test.py

2. Run the command in the background so that you can assign the command to a variable before it raises an exception. Catch the exception that wait() raises. Examine stdout and stderr properties of the command object.

   In [6]: c = pbs.ls('.', 'non-existent-file', _bg=True)

   In [7]: try: c.wait()
   except pbs.ErrorReturnCode:
   pass
   ...:

   In [8]: print c.stdout
   .:
   AUTHORS.md
   LICENSE.txt
   MANIFEST.in
   README.md
   pbs.py
   pbs.pyc
   setup.py
   test.py

   In [9]: print c.stderr
   ls: non-existent-file: No such file or directory

Neither of these methods is documented. Would you recommend either of the above, or something different?

Hi, Fedora packager here.

While github's feature to download tarballs (or zips) based on tags is nice, it doesn't quite get there. It includes the repo's owner name and partial sha1 in the name of the tarball, as well as the leading path in the tarball. Little things but they make like harder for packagers.

Could you make actual tarballs available via Github Downloads? In addition to fixing the above problems, this would also be an indicator that a given tag of sh is "stable". I'm in the same boat for some of my projects, so I'm here's my little script, if it's useful:

#!/usr/bin/env python

import os
import sys

repo_path = "/home/agrover/git"
outpath = "/home/agrover/tars"

def s(cmd):
    print "++", cmd
    os.system(cmd)

if not len(sys.argv) == 3:
    print "usage %s <package> <version>" % sys.argv[0]
    sys.exit(-1)

pkg = sys.argv[1]
ver = sys.argv[2]
pkg_ver = pkg + "-" + ver

os.chdir(repo_path+"/"+pkg)

s("git archive %s --prefix %s/ |gzip -n > %s/%s.tar.gz"
  % (ver, pkg_ver, outpath, pkg_ver))

The python docs at http://docs.python.org/reference/datamodel.html#object.__repr__ say:

If at all possible, this should look like a valid Python expression that could be used to recreate an object with the same value (given an appropriate environment). If this is not possible, a string of the form <...some useful description...> should be returned.

I see the appeal of making command output be as prominent as possible, but I think it would be less confusing if we stuck with the default repr, or maybe something like <RunningCommand "/bin/ls": "Desktop\nDocuments\nproje...">.

Might also be worth trying to avoid repr having side effects like blocking.

In [1]: from pbs import scp

In [2]: scp('-P 2222 testfile [email protected]:~/')

ErrorReturnCode_1 Traceback (most recent call last)
/home/ubuntu/dev/ssh-keymgmt/ in ()
----> 1 scp('-P 2222 testfile [email protected]:~/')

/usr/local/lib/python2.6/dist-packages/pbs.pyc in call(self, _args, *_kwargs)
430 stdin=stdin, stdout=stdout, stderr=stderr)
431
--> 432 return RunningCommand(command_ran, process, call_args, actual_stdin)
433
434

/usr/local/lib/python2.6/dist-packages/pbs.pyc in init(self, command_ran, process, call_args, stdin)
143 if stdin: stdin = stdin.encode("utf8")
144 self._stdout, self._stderr = self.process.communicate(stdin)
--> 145 self._handle_exit_code(self.process.wait())
146
147 def enter(self):

/usr/local/lib/python2.6/dist-packages/pbs.pyc in _handle_exit_code(self, rc)
210 def _handle_exit_code(self, rc):
211 if rc not in self.call_args["ok_code"]:
--> 212 raise get_rc_exc(rc)(self.command_ran, self._stdout, self._stderr)
213
214 def len(self):

ErrorReturnCode_1:

Ran: '/usr/bin/scp -P 2222 testfile [email protected]:~/'

STDOUT:

STDERR:

usage: scp [-1246BCpqrv] [-c cipher] [-F ssh_config] [-i identity_file]
[-l limit] [-o ssh_option] [-P port] [-S program]
[[user@]host1:]file1 ... [[user@]host2:]file2

Running the command returned as Ran works fine.

I love the idea of this library! Can you make it legit by adding a license, a setup.py file and submitting it to pypi?

The problem exists when you try to reload pbs:

import pbs
reload(pbs)

You get

TypeError: reload() argument must be module

This is because the SelfWrapper, which does the dynamic lookups, isn't a subclass of the module type. @nemec has implemented the dynamic lookup with import hooks here 071a98b, which is probably the more pythonic way of doing this.

It looks like pbs doesn't handle files in non UTF-8 codec. Given the following text file named numbers.txt which uses iso 8859-1, for example:

um
três
açúcar
água

Next, I tried this code:

from pbs import cat
print cat('numbers.txt')

which results in the following error:

$ python cat.py
Traceback (most recent call last):
  File "cat.py", line 3, in <module>
    print cat('numeros.txt')
  File "/usr/local/lib/python2.7/dist-packages/pbs.py", line 155, in __str__
    else: return unicode(self).encode("utf-8")
  File "/usr/local/lib/python2.7/dist-packages/pbs.py", line 159, in __unicode__
    if self.stdout: return self.stdout.decode("utf-8") # byte string
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xea in position 12: invalid continuation byte

It would be better to have a result like the one produced by the following code:

from subprocess import check_output
cat = check_output(['/bin/cat', 'numbers.txt'])
print cat

which produces:

um
dois
tr�s
cinq�enta
�gua
a��car

I can do

>>> pbs.git.remote
/usr/bin/git remote

but if I try to go one level deeper, I get

>>> pbs.git.remote.add
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'BakedCommand' object has no attribute 'add'

OTOH, I can do

>>> pbs.git.bake('remote').add
/usr/bin/git remote add

but this breaks:

>>> pbs.git.bake('remote').bake('add')

and so does this:

>>> pbs.git.remote.bake('add')

I'd think those should all have the same effect.

How about to implement commands piping by using Pipe package?
For example:

# sort this directory by biggest file
print(sort(du(glob("*"), "-sb"), "-rn"))

could be

print(glob("*") | du("-sb") | sort("-rn"))

which is more readable and intuitive. Thoughts?

I've implemented a test at https://github.com/pcn/pbs in the generator branch. I think it's clumsy, but I'm still feeling my way around.

Use case:

Do something useful with vmstat, tcpdump, etc. These commands are most useful when producing output constantly, and pbs should be able to consume these line by line.

The implementation avoids calling subprocess.communicate and selects directly from the pbs_object.process objects stdout.

I'm looking for thoughts on how to do this "right" (e.g. should it be a subclass or a separate module that inherits from pbs?). Currently it changes how it's used, but it is convenient because if only the last command in a pipeline has _generator=True set, then everything should work as normal (once I hook stdin back up :) except the last command will yield lines that can be worked with.

I saw this example in document:

from sh import ls

ls = ls.bake("-la")
print(ls) # "/usr/bin/ls -la"

# resolves to "ls / -la"
print(ls("/"))

This should work fine for GNU-style ls. For POSIX-style ls, this is wrong because non-option argument should appear after all options.

My test on mac:

$ /bin/ls / -la >/dev/null
ls: -la: No such file or directory

POSIX style: http://pubs.opengroup.org/onlinepubs/009604499/basedefs/xbd_chap12.html
GNU style: http://www.gnu.org/prep/standards/html_node/Command_002dLine-Interfaces.html

If I'm nitpicking too much, just say so.

But I think, for writing quick command chains, pbs_bg=True is a teeny bit verbose to type and read in-line. What if it were something shorter, such as bg_, or _bg? It's highly unlikely to see a command-line flag beginning or ending with an underscore, and using the underscore prefix make it obvious that it isn't a program flag to be passed along.

The example in the README would then just be:

p = sleep(3, _bg=True)

when doing like this:

from pbs import ls
ls("nonexistent", _err="error.txt")
ls("nonexistent")

ls is no dynamic created, and the second ls command will also redirect errors to "error.txt"
this is quite unexpected.

I suggest moving the self.call_args creation from __init__ to __call__

I have a recurrent problem: in a web2py app create an interface to console commands. I would like to be able to add a button to run a command and have the command run as a daemon. At the same time, when the user visit the associated web page, the user should be able to interact with the daemon: read the output, provide input if the daemon is waiting for input. The user should be able to navigate away from the page and come back.

I think sh.py could be used for this purpose with some improvements. First of all it could have an option to fork twice before executing a command and setting its one group id. Then, again before executing the command, it should open an input named pipe and an output named pipe (or simply a file), so that external processes can connect to them. It should also store the pid in a file so that one can monitor whether the process is running or not and eventually re-start it. there should be a serializable object that can be passed between processed with enough information to connect to the input/output of the running process.

If you are interested adding this features please contact me personally. I have already written some of the relevant code but I have not tried to integrate it in sh.py.