I'm trying to run this command through KennethReitz's Envoy package:

$ sqlite3 foo.db 'select * from sqlite_master' 

I've tried this:

r = envoy.run("sqlite3 foo.db 'select * from sqlite_master'")
sqlite3: Error: too many options: "*"

and this:

r = envoy.run(['sqlite3', 'foo.db', 'select * from sqlite_master'])
AttributeError: 'NoneType' object has no attribute 'returncode'

additional quoting & escaping doesn't seem to help. Any suggestions?

FYI: This is what I had to do for now:

cmd = "sqlite3 %(database)s 'select * from sqlite_master'" % locals()
os.system(cmd)

Note that this is a contrived example, and that most of the unix shell commands that I'd like to issue aren't just a simple select that could be easily done via SQLAlchemy.

From http://stackoverflow.com/questions/9106350/how-to-use-wildcards-with-envoy

envoy.run('yes | head') gets stuck

That would be great :-)

A thread can be alive when the process doesn't exist yet. It it's terminated, and the command runs array. Very bad.

~/envoy$ tar cfz test.tgz envoy/
~/envoy$ cat test.tgz | tar -tz
envoy/
envoy/init.pyc
envoy/init.py
envoy/core.py
envoy/core.pyc
~/envoy$ python
Python 2.6.6 (r266:84292, Dec 26 2010, 22:31:48)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import envoy
r = envoy.run("cat test.tgz | tar -tz")
r.status_code
2
r.std_err
'\ngzip: stdin: invalid compressed data--format violated\ntar: Child
returned status 1\ntar: Error is not recoverable: exiting now\n'

I feel the pain when using subprocess. It is nice but quite low level and is certainly not nice when writing shell-equivalent script.

For that I wrote: https://github.com/aht/extproc

Please check it out, I think we have very similar goals and we can contribute code either way.

Sample code: https://gist.github.com/43b962a5e0d1f7cd8c89

Originally, the problem was grabbing data from an RSS feed with Russian text in it from Google Reader via the requests module. I reproduced the issue in ipython.

Python version 2.7.1

I think envoy.run is something of a massive foot-gun right now if the canonical way to manage arguments is with %s or new-style format strings. Although you have set shell=False and eschewed the system-dependent shell language for the cross-platform shlex, the following is just as vulnerable as bash or the system function in C (implemented in Perl and Ruby via backtick characters) to quoting problems and injection attacks:

envoy.run('ls %s' % directory)

Where directory could have spaces, then passed to shlex and then take the form of additional arguments.

If you feel you must use an embedded languages for .run (which is a lot more terse than using vanilla lists or tuples in Python, such as ['ls', directory]) please consider doing something akin to what database drivers do, which is supporting parameters that are interpreted by that driver rather than using the string substitution mechanisms.

For example:

Python 2.6.5 (r265:79063, Oct  1 2012, 22:04:36) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import envoy
>>> r = envoy.run(u"ls")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.6/dist-packages/envoy/core.py", line 209, in run
    out, err = cmd.run(data, timeout, kill_timeout, env, cwd)
  File "/usr/local/lib/python2.6/dist-packages/envoy/core.py", line 89, in run
    raise self.exc
OSError: [Errno 2] No such file or directory
>>> 

Is this the expected behaviour? Should I "de-unicode" commands before passing them to envoy?

I run:

command = """samtools mpileup foobar.bam -r '1|FR0001.1:100000-100500' """"
r = envoy.run(command)

got:
File "", line 1, in
File "build/bdist.linux-x86_64/egg/envoy/core.py", line 162, in run
File "build/bdist.linux-x86_64/egg/envoy/core.py", line 151, in expand_args
File "/users/rg/dkedra/soft/lib/python2.7/shlex.py", line 279, in split
return list(lex)
File "/users/rg/dkedra/soft/lib/python2.7/shlex.py", line 269, in next
token = self.get_token()
File "/users/rg/dkedra/soft/lib/python2.7/shlex.py", line 96, in get_token
raw = self.read_token()
File "/users/rg/dkedra/soft/lib/python2.7/shlex.py", line 172, in read_token
raise ValueError, "No closing quotation"

Command by itself works from a shell command line.

Thanks for your help

Darek Kedra

I tried the following in a script and got an error

>>> r = envoy.run("mail --subject='MPAdmin Notice: Server out of RAM' '[email protected]'", data="\nTest Message\n")
>>> r.std_err 
"Cannot parse address `Notice:' (while expanding `Notice:'): Format of RFC822 object is bad\n"
>>> r.command 
['mail',
 '--subject=MPAdmin',
 'Notice:',
 'Server',
 'out',
 'of',
 'RAM',
 '[email protected]']

Had to double escape the subject line like so:

envoy.run("mail --subject='MPAdmin\ Notice:\ Server\ out\ of\ RAM' '[email protected]'", data="Test Message\n")

As an additional little tidbit, it seems the data kwarg requires a trailing newline to work properly, that's not such a big deal though. It might be specific to mail and not related to envoy.

Thanks for the great lib :)

Reference example is here:
http://ipython.org/ipython-doc/rel-0.12/interactive/shell.html#string-lists

The String list utiltiy class allows for more programmatic opportunities of meshing Python logic and command line data. If I did the work to add support for this, do you think this would be something that would be incorporated into Envoy?

https://pypi.python.org/pypi/subprocess32/3.2.6

In [24]: n = envoy.run("cat requirements.txt >> py.txt")

In [25]: n.std_err
Out[25]: 'cat: >>: No such file or directory\ncat: py.txt: No such file or directory\n'

I have a command that sends 600kB of output to stdout.

cat /path/to/file.fastq | seqtk sample - 3656 | seqtk seq -a -

Called with os.system(), it does exactly that, however, when run through Envoy the output is truncated to 6kB. I've tried using the inbuilt .std_out, redirection, and teeing to an output file, but the output is always truncated. Is this a problem related to pipes? I have many calls to envoy.run() in the same script – does a buffer need flushing? I'm unfamiliar with the finer points of subprocess – is this a problem inherited from subprocess, or is it envoy, or something else entirely?

how can I get the right output

Here is what I got

Traceback (most recent call last):

[...], in _run
    r = envoy.run(cmd)
File "/home/conquete/.local/lib/python2.6/site-packages/envoy/core.py", line 167, in run
    out, err = cmd.run(data, timeout)
File "/home/conquete/.local/lib/python2.6/site-packages/envoy/core.py", line 52, in run
    self.returncode = self.process.returncode
AttributeError: 'NoneType' object has no attribute 'returncode'

My command was

ps xf | egrep "[0-9]:[0-9]{2} (.+/)?uwsgi --ini .+/uwsgi.conf" | sed -r "s/^\s+//"

when i read requests code, according test_requests.py , i using envoy in ipython, i get a error.

In [2]: envoy.connect('guncorn httpbin:app --bind=0.0.0.0:8080')
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-2-3bd699ccabe1> in <module>()
----> 1 envoy.connect('guncorn httpbin:app --bind=0.0.0.0:8080')

/Users/guo/Code/repository/02.Learn/00.python_learn/chai_requests/venv/lib/python2.7/site-packages/envoy/core.pyc in connect(command, data, env, cwd)
    249         stderr=subprocess.PIPE,
    250         bufsize=0,
--> 251         cwd=cwd,
    252     )
    253

/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.pyc in __init__(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags)
    707                                 p2cread, p2cwrite,
    708                                 c2pread, c2pwrite,
--> 709                                 errread, errwrite)
    710         except Exception:
    711             # Preserve original exception in case os.close raises.

/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.pyc in _execute_child(self, args, executable, preexec_fn, close_fds, cwd, env, universal_newlines, startupinfo, creationflags, shell, to_close, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite)
   1324                         raise
   1325                 child_exception = pickle.loads(data)
-> 1326                 raise child_exception
   1327
   1328

OSError: [Errno 2] No such file or directory

do you have any idea about this?

Looks like a problem lexing unicode commands. If I cast the command to a str, it works as expected.

This is with current latest envoy as pulled from pip.

Python 2.7.10 (default, Jul 14 2015, 19:46:27)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import envoy
>>> r = envoy.run(u'uptime')
>>> r.std_out
''
>>> r.std_err
u'No such file or directory\nTraceback (most recent call last):\n  File "/Users/mwhooker/dev/wercker/devbox/formation/venv/lib/python2.7/site-packages/envoy/core.py", line 214, in run\n    out, err = cmd.run(data, timeout, kill_timeout, env, cwd)\n  File "/Users/mwhooker/dev/wercker/devbox/formation/venv/lib/python2.7/site-packages/envoy/core.py", line 93, in run\n    raise self.exc\nOSError: [Errno 2] No such file or directory\n'
>>> r.__dict__
{'std_err': u'No such file or directory\nTraceback (most recent call last):\n  File "/Users/mwhooker/dev/wercker/devbox/formation/venv/lib/python2.7/site-packages/envoy/core.py", line 214, in run\n    out, err = cmd.run(data, timeout, kill_timeout, env, cwd)\n  File "/Users/mwhooker/dev/wercker/devbox/formation/venv/lib/python2.7/site-packages/envoy/core.py", line 93, in run\n    raise self.exc\nOSError: [Errno 2] No such file or directory\n', 'std_out': '', 'status_code': 127, '_process': <envoy.core.Command object at 0x107fe4390>, 'command': u'e', 'history': [<Response [u]>, <Response [p]>, <Response [t]>, <Response [i]>, <Response [m]>]}

r = envoy.run("lsof -F n -c vlc | egrep -i 'mp3' | sed 's/^n//g'")
print(r.std_out)

returns as null...

Currently the behavior is :

>>> envoy.run('fdqfsdfqsdfds')

/home/sam/.virtualenvs/swaaper/local/lib/python2.7/site-packages/envoy/core.pyc in run(self, data, timeout)
     50             self.process.terminate()
     51             thread.join()
---> 52         self.returncode = self.process.returncode
     53         return self.out, self.err
     54 

AttributeError: 'NoneType' object has no attribute 'returncode'

Something like "SubprocessError: command not found" would be easier to undestand and to try/catch.

I'm trying envoy on windows7 and have a problem with it.
If I run a command like "dir" I get an error on the line self.returncode = self.process.returncode ("AttributeError: 'NoneType' object has no attribute 'returncode'). This is caused by an WindowsError in the subprocess.Popen call. (WindowsError: [Error 2] The system cannot find the file specified)

After some googling I noticed that the option shell=False causes this problem. When I call subprocess.Popen manually with shell=True, I don't get this error.

More info can be found here: http://bugs.python.org/issue8224
In the bugreport you can see that the conclusion is that this is a windows bug, so they closed the bug as invalid.
But to use envoy on windows, I propose that you check for this, or that you document it somewhere.

I've used the last version of envoy on pypi (0.2), but checked your github code and I suppose that the error is still there.

http://code.activestate.com/recipes/577853-timeout-decorator-with-multiprocessing/

envoy or subprocess doesn't expand alias. I am planning to add the support.

Idea

r = run('ll', aliases=['ll'])

inside run func, aliases default value will be None.

parse .bash_alias, .bashrc, .bash_profile and collect all aliases

        for alias in alias:
           #Replace the alias

I am trying to handle a situation where i can call multiple 3rd party command line functions and need to tell if they worked or not (the apps may not be installed).

The current version is silent on errors. An status_code is set on the result object, however there is no documentation on what this status code means or what expected values could be.

In my case, the error code was 127 and is being set manually by package. Thankfully the source was brief, and I quickly saw this https://github.com/kennethreitz/envoy/blob/master/envoy/core.py#L216-L218

It would be great if a future version simply noted what good/bad values are for status codes -- or at least documented what known errors would be. I dug through several stdlib packages and could not find a corresponding 127 error.

Allow setting working directory when running a command.

I was getting a bug with envoy, and I think that expand_args might be the problem:

>>> expand_args('git --work-tree="C:\\Documents and Settings\\User\\whatever"')
[['git', '--work-tree=C:Documents', 'and', 'SettingsUserwhatever']]

This output is bad, isn't it? Is this a bug in expand_args?

F='resources/ch06-mailboxes/data/enron.mbox.json.bz2'
----> 9 r = envoy.run("bunzip2 %s" % (F,))
10 print r.std_out
11 print r.std_err

There is no LICENSE file for this module.

Envoy needs to provide a way to pawn and interface with long-running processes.

Use case: sshout

Functionality

• Non-blocking
• Stream
• Readline-style reactors?

c = envoy.connect('command')

# Expect.
c.expect('Password: ')
c.sendline('mypassword')

# Pid
c.pid

# Kill.
c.kill()

# Block.
c.block()

The first thing that falls over is seeking _status_code.
As far as I can tell _status_code isn't actually assigned any value in def status_code(self).

ERROR: test_status_code_success (main.ConnectedCommandTests)

Traceback (most recent call last):
File "test_envoy.py", line 38, in test_status_code_success
self.assertEqual(c.status_code, 0)
File "/mnt/gen2/TmpDir/portage/dev-python/envoy-0.0.2-r1/work/envoy-0.0.2/envoy/core.py", line 80, in status_code
if self._status_code is not None:
AttributeError: 'ConnectedCommand' object has no attribute '_status_code'

I took a punt at copying
self._status_code = self._process.wait()
from further on and it seemed to make a couple of tests pass.

    self.assertEqual(r.std_out, test_string.upper())
    self.assertEqual(r.status_code, 0)

both fail in def test_input(self), line 53, the second one.

envoy-0.0.2 $ PYTHONPATH=. python2.7 test_envoy.py

yields

.......................................................

FAIL: test_input (main.ConnectedCommandTests)

Traceback (most recent call last):
File "test_envoy.py", line 48, in test_input
self.assertEqual(r.std_out, test_string.upper())
AssertionError: None != 'ASDFQWER'

FAIL: test_quoted_args (main.SimpleTest)

Traceback (most recent call last):
File "test_envoy.py", line 26, in test_quoted_args
self.assertEqual(r.std_out.rstrip(), sentinel)
AssertionError: '' != 'quoted_argsquoted_argsquoted_args'

Ran 8 tests in 3.347s

FAILED (failures=2, errors=3)

eeek

Hello,

When I use the envoy.connect function, the process systematically terminates as a zombie process.

I tried to modify the code by adding "close_fds=True" to Popen, but I still have the problem.

I'm on solaris 10 (x86).

If you need any information about my environnment, please let me know!

I really like envoy, and I like how the project develops, but I am getting tired of always checking out the git version.

The last release was on 2011-10-15, what needs to be fixed until you release a new version?

Thanks,
Daniel

import envoy

c = envoy.connect('echo 111 && sleep 1 && echo 222')
c.block()

print c.status_code,c.std_err,c.std_out

output:
0 None None

expect:
0 111 222

Requesting a 'shell=True' option for envoy.run().

I pushed some code to PBS dev earlier today that allowed to do some really cool stuff like this:

from pbs import tail

# runs forever
for line in tail("-f", "/var/log/some_log_file.log", _for=True):
    print line

Or use callbacks:

from pbs import tail

def process_output(line):
    print line

p = tail("-f", "/var/log/some_log_file.log", _out=process_output)
p.wait()

As well as working with stdin dynamically for interactive programs.

I was wondering if there were any plans to support something similar in envoy? If so, I could probably brief you on some of the gotchas. Any plans for incremental output, ken?