tomerfiliba / plumbum Goto Github PK
View Code? Open in Web Editor NEWPlumbum: Shell Combinators
Home Page: https://plumbum.readthedocs.io
License: MIT License
Plumbum: Shell Combinators
Home Page: https://plumbum.readthedocs.io
License: MIT License
The output of commands is treated like text strings... need a better way to differentiate between byte-level commands and text-level commands. Maybe local.text["ls"]
vs. local.bytes["ls"]
?
Hi,
shouldn't _allow_root be initialized outside the method allow_as_root in that example
http://plumbum.readthedocs.org/en/latest/cli.html#switch-functions ?
just a cosmetically nice to have feature,
if help could accept a list and correctly indent the lines
log_path = SwitchAttr(["-l", "--log-path"], argtype = str,
help=["path to the log directory", "default to current directory"] )
...
Switches:
-l, --log-path VALUE:str path to the log directory
default to current directory
...
This might be a bug??
from plumbum import local
machine = local
home = machine.path("~")
print home
Output:
/homes/harlowja/~
Maybe a special case is needed for "~"??
I would have expected /homes/harlowja/
and not /homes/harlowja/~
ParamikoMachine
always tries to connect with default user, even if user
arg is given to ParamikoMachine.__init__
. This leads to login as the wrong user when e.g. SSH keys are available for multiple users to the same host.
suggested fix:
if user is not None:
kwargs["username"] = user
>>> from plumbum import SshMachine
>>> from plumbum.cmd import plink, pscp
>>> rem = SshMachine('app5',ssh_command=plink,scp_command=pscp,ssh_opts=['-ssh'])
#
# wait then type ctrl+C
#
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "c:\python\lib\site-packages\plumbum-1.0.0-py2.7-win32.egg\plumbum\remote_machine.py", line 357, in __init__
BaseRemoteMachine.__init__(self)
File "c:\python\lib\site-packages\plumbum-1.0.0-py2.7-win32.egg\plumbum\remote_machine.py", line 151, in __init__
self._session = self.session()
File "c:\python\lib\site-packages\plumbum-1.0.0-py2.7-win32.egg\plumbum\remote_machine.py", line 381, in session
return ShellSession(self.popen((), ["-tt"] if isatty else []), self.encoding, isatty)
File "c:\python\lib\site-packages\plumbum-1.0.0-py2.7-win32.egg\plumbum\session.py", line 132, in __init__
self.run("")
File "c:\python\lib\site-packages\plumbum-1.0.0-py2.7-win32.egg\plumbum\session.py", line 213, in run
return run_proc(self.popen(cmd), retcode)
File "c:\python\lib\site-packages\plumbum-1.0.0-py2.7-win32.egg\plumbum\commands.py", line 152, in run_proc
stdout, stderr = proc.communicate()
File "c:\python\lib\site-packages\plumbum-1.0.0-py2.7-win32.egg\plumbum\session.py", line 91, in communicate
line = pipe.readline()
File "c:\python\lib\site-packages\plumbum-1.0.0-py2.7-win32.egg\plumbum\session.py", line 39, in readline
line = self.pipe.readline()
KeyboardInterrupt
>>>
In python 2.6, running a local command failed by raising an AttributeError when attempting to access subprocess._subprocess.
Looks like this is a regression caused by the FIX of issue #30.
The fix is to simply check if hasattr( subprocess, "_subprocess" ) and act accordingly.
Using from __future__ import unicode_literals
gives tons of errors due to explicit type checking of the form
if not isinstance(newdir, (str, LocalPath)):
raise TypeError("newdir must be a string or a LocalPath, not %r" % (newdir,))
Could you either use basestring
instead of str
or, preferably, remove the explicit typechecks altogether? I'd be glad to send in a patch if you are willing to accept this change.
The removal of the checks would also allow the passing of objects that define a reasonable __str__
method according to "we are all consenting adults here".
like the standard one http://docs.python.org/2/library/os.html#os.symlink
I suggest changing run to something like that:
@classmethod
def run(cls, argv = sys.argv, exit=True):
"""Runs the application, taking the arguments from ``sys.argv``, and exiting with the
appropriate exit code; when ``exit`` is True this function does not return, else it return the return code """
_, retcode = cls._run(argv)
if exit:
sys.exit(retcode)
else:
return retcode
it can help when unittesting or when invoking from other applications like that:
def test_01_running_from_cmd_line(self):
self.assertEqual(
PostProcessor.run(['app', '-l', self.test_data, '-s', self.test_scripts], exit=False),
0
)
BTW, really nice package (got me to ditch pbs
)
At least I think so :) I can make changes / pull request. Wanted to bring up for discussion first. My usecase is:
foo = plumbum.local.path('/some/path/to/file.txt')
foo.dirname.mkdir()
when one switch function requires another, invoke them in a topological order. if there's a cycle, invoke them in the order they are given on the command line
make local.cwd, local.env thread-safe?
does it make sense to emulate a separate cwd/env each thread?
When trying to import any command in python 3.3 following exception occurs
>>> from plumbum.cmd import wc
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<frozen importlib._bootstrap>", line 1597, in _handle_fromlist
File "/usr/lib/python3.3/site-packages/plumbum/local_machine.py", line 537, in __getitem__
return LocalCommand(self.which(cmd))
File "/usr/lib/python3.3/site-packages/plumbum/local_machine.py", line 508, in which
raise CommandNotFound(progname, list(cls.env.path))
plumbum.commands.CommandNotFound: ('__path__', [<LocalPath /usr/local/bin>, <LocalPath /usr/bin>, <LocalPath /bin>, <LocalPath /usr/local/sbin>, <LocalPath /usr/sbin>, <LocalPath /sbin>, <LocalPath ~/bin>, <LocalPath /usr/bin/vendor_perl>, <LocalPath /usr/bin/core_perl>, <LocalPath /usr/lib/qt4/bin>, <LocalPath /home/blin/bin>, <LocalPath /home/blin/bin>])
This seems to be due to http://docs.python.org/3/whatsnew/3.3.html#importlib
What happens is that
__import__('plumbum.cmd', fromlist=('wc')
eventually calls
hasattr(plumbum.cmd, '__path__')
which calls
plumbum.local_machine.local.__getitem__('__path__').
which fails.
Proposed solution is to replace
__getattr__ = local.__getitem__
in cmd with
def __getattr__(self, name):
if name != '__path__':
return local.__getitem__(name)
else:
pass
remote: piping doesn't work as expected (it happens on the local host).
need a way to convert foo | bar
to rem[foo | bar]
@switch("--foo", [str, int])
def foo(self, a, b):
pass
Trying to generate ssh keys:
keygen = local['ssh-keygen']
keygen('-t', 'rsa', '-N', '', '-f', 't_key.pub')
results in:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/site-packages/plumbum/commands.py", line 219, in __call__
return self.run(args, **kwargs)[1]
File "/usr/lib/python2.7/site-packages/plumbum/commands.py", line 275, in run
return run_proc(p, retcode, timeout)
File "/usr/lib/python2.7/site-packages/plumbum/commands.py", line 173, in run_proc
stdout, stderr)
plumbum.commands.ProcessExecutionError: Command line: ['/usr/bin/ssh-keygen', '-t', 'rsa', '-N', '-f', 't_key.pub']
Exit code: 1
Stdout: | Too many arguments.
i.e. the empty single quoted passphrase for -N is ignored.
Whereas the subprocess equivalent works as expected.
>>> com = ['ssh-keygen', '-t', 'rsa', '-N', '', '-f', 't_key.pub']
>>> r = subprocess.Popen(com, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> r.returncode
>>> (output, error) = r.communicate()
>>> r.returncode
0
>>> output
"Generating public/private rsa key pair.\nYour identification has been saved in t_key.pub.\nYour public key has been saved in t_key.pub.pub.\nThe key fingerprint is:\n16:28:94:a5:73:e0:56:1f:73:88:b1:86:e4:32:8a:28 [email protected]\nThe key's randomart image is:\n+--[ RSA 2048]----+\n| +o+oo.. |\n| =.=.+.+ |\n| o O = o |\n|o. + = . |\n|E S |\n|. . |\n| |\n| |\n| |\n+-----------------+\n"
It might be interesting to see what the effects of if a pure-python ssh library is used instead of shelling out to ssh/sftp?
Not sure if it will be better (I haven't used it, but might be a neat idea).
I've been using plumbum paths in a script i'm writing lately. My script uses local.cwd to find out the name of the current directory, then uses that to navigate to a path on a remote PC.
Unfortunately the script doesn't work at all when the local machine is windows, the remote is linux and the current directory name is not all lowercase. This seems to be because plumbum throws away case information from paths on windows.
For example:
>>> import os
>>> from plumbum import local
>>> os.getcwd()
'C:\\src\\NVR'
>>> local.cwd.basename
'nvr'
If I then try to use local.cwd.basename to navigate to the equivalent folder on my remote PC it will fail as the folder is actually named NVR
rather than nvr
.
When attempting to use plumbum with python 2.7 (32 bit) on windows 7 I receive the following error:
Traceback (most recent call last):
File "D:\src\scripts\updatecg.py", line 17, in <module>
net('stop', 'camgat')
File "C:\Python27\lib\site-packages\plumbum\commands.py", line 151, in __call__
return self.run(args, **kwargs)[1]
File "C:\Python27\lib\site-packages\plumbum\commands.py", line 199, in run
p = self.popen(args, **kwargs)
File "C:\Python27\lib\site-packages\plumbum\local_machine.py", line 376, in popen
kwargs["startupinfo"].dwFlags |= subprocess.STARTF_USESHOWWINDOW #@UndefinedVariable
AttributeError: 'module' object has no attribute 'STARTF_USESHOWWINDOW'
Similarly, subprocess.SW_HIDE
also doesn't exist in this version of python.
I have been able to fix this by using subprocess._subpprocess.STARTF_USESHOWWINDOW
instead, though I'm not sure how cross platform this fix is.
It seems odd to me that you can't set the value of a SwitchAttr from within a CLI application.
Here's a simple test case:
from plumbum import cli
class MyApp(cli.Application):
param = cli.SwitchAttr(['-p', '--param'], str, default='default')
def main(self):
if self.param == 'default':
# then do some stuff here, like maybe get the value for param
# from a config file
config_file_param = 'fromconfig'
self.param = config_file_param
if __name__ == '__main__':
MyApp.run()
which results in:
Traceback (most recent call last):
File "temp.py", line 16, in <module>
MyApp.run()
File "C:\Documents and Settings\funsize\Desktop\plumbum_test\venv\lib\site-packages\plumbum\cli.py", line 541, in run
_, retcode = cls._run(argv)
File "C:\Documents and Settings\funsize\Desktop\plumbum_test\venv\lib\site-packages\plumbum\cli.py", line 532, in _run
retcode = inst.main(*tailargs)
File "temp.py", line 12, in main
self.param = config_file_param
TypeError: __set__() takes exactly 4 arguments (3 given)
Looking at the code for SwitchAttr, it doesn't look like a regular descriptor to me (at least it doesn't seem to follow the descriptor protocol, but perhaps I'm missing something), so I confess I don't really understand how it's supposed to work, but it seems pretty strange to me.
At any rate, it would be pretty convenient to be able to set SwitchAttrs inside my App.
Don't know who to blame but plumbum 1.1.0
fails to run in trivial combination with Flask 0.9 while plumbum 1.0.1
works well. Here is code to reproduce issue.
And here is output:
>python index.py
* Running on http://127.0.0.1:5000/
* Restarting with reloader
Traceback (most recent call last):
File "index.py", line 12, in <module>
app.run()
File "/home/user/pyenv/fplu/local/lib/python2.7/site-packages/flask/app.py", line 739, in run
run_simple(host, port, self, **options)
File "/home/user/pyenv/fplu/local/lib/python2.7/site-packages/werkzeug/serving.py", line 615, in run_simple
run_with_reloader(inner, extra_files, reloader_interval)
File "/home/user/pyenv/fplu/local/lib/python2.7/site-packages/werkzeug/serving.py", line 533, in run_with_reloader
reloader_loop(extra_files, interval)
File "/home/user/pyenv/fplu/local/lib/python2.7/site-packages/werkzeug/serving.py", line 439, in _reloader_stat_loop
for filename in chain(_iter_module_files(), extra_files or ()):
File "/home/user/pyenv/fplu/local/lib/python2.7/site-packages/werkzeug/serving.py", line 413, in _iter_module_files
filename = getattr(module, '__file__', None)
File "/home/user/pyenv/fplu/local/lib/python2.7/site-packages/plumbum/local_machine.py", line 537, in __getitem__
return LocalCommand(self.which(cmd))
File "/home/user/pyenv/fplu/local/lib/python2.7/site-packages/plumbum/local_machine.py", line 508, in which
raise CommandNotFound(progname, list(cls.env.path))
plumbum.commands.CommandNotFound: ('__file__', [<LocalPath /home/user/pyenv/fplu/bin>, <LocalPath /home/user/.pythonbrew/bin>, <LocalPath /home/user/.pythonbrew/bin>, <LocalPath /usr/local/bin>, <LocalPath /usr/bin>, <LocalPath /bin>, <LocalPath /usr/games>])
code:
lpath = local.path('/var/ftp/deploy/images/current/') / target_vm // '*'
rpath = remote.path(datastore.summary.url) / target_vm
for path in lpath:
print "lpath = '%s'" % path
print "rpath = '%s'" % rpath
remote.upload(path, rpath / path.basename)
output:
lpath = '/var/ftp/deploy/images/current/MgmtVM/MgmtVM.ovf'
rpath = '/vmfs/volumes/4f41cb66-5f5228fd-e818-001cc06d4ba7/Test Cloud 1 - Management'
Traceback (most recent call last):
File "create-vm.py", line 181, in <module>
remote.upload(path, rpath / path.basename)
File "/usr/lib/python2.6/site-packages/plumbum/remote_machine.py", line 562, in upload
self._scp_command(src, "%s:%s" % (self._fqhost, dst))
File "/usr/lib/python2.6/site-packages/plumbum/commands.py", line 151, in __call__
return self.run(args, **kwargs)[1]
File "/usr/lib/python2.6/site-packages/plumbum/commands.py", line 201, in run
return run_proc(p, retcode)
File "/usr/lib/python2.6/site-packages/plumbum/commands.py", line 105, in run_proc
proc.returncode, stdout, stderr)
plumbum.commands.ProcessExecutionError: Command line: ['/usr/bin/scp', '-r', '/var/ftp/deploy/images/current/MgmtVM/MgmtVM.ovf', '[email protected]:/vmfs/volumes/4f41cb66-5f5228fd-e818-001cc06d4ba7/Test Cloud 1 - Management/MgmtVM.ovf']
Exit code: 1
Stderr: | scp: ambiguous target
and use proper slashes/backslashes in paths
It might be useful to say do the following:
ls = local['ls'] (or use SshMachine here...)
with Root(ls) as root_ls:
root_ls.run()
This way people can run as other users in a pythonic way (root show as an example here). Similarly the same would be neat with ssh/remote shells. Sometimes u just need to run a single command as another user (ie apt-get install
) and u don't want the rest of the commands ran as that user... Thoughts?
@switch(argtype = str)
def log_to_file(self, filename):
pass
==>
@switch("--log-to-file", argtype = str)
def log_to_file(self, filename):
pass
Plumbum should provide an API for "subcommand" style interfaces, in which the cli app has a number of subcommands.
e.g. svn, hg, bzr, git, etc.
One way to do this might be to provide a cli.subcommand
decorator which you would use to decorate functions in a cli.Application
. Then the parsing logic would look for any of these and use whichever one was specified on the command line instead of the current main
method.
I have a parameter the get a comma separated list:
app --skip "A,B,C , D, F"
in my code I'm doing this:
parsers_skip = []
@switch("--skip")
def skip(self, input):
self.parsers_skip = [ i.strip() for i in input.split(',') ]
I'm looking for a shorter decorator for this, maybe making the autoswitch
to accept a default
something like that :
@autoswitch("--skip", default=[])
def parsers_skip(self, input):
return [ i.strip() for i in input.split(',') ]
I've looked into the code, but didn't thought of a way to implement it.
but that example is really DRY counter example (repeating the name of the switch three times)
Hi! I really love the idea of this package (no shell anymore, what a bliss :) ) and will use it very soon. It installed without problems on python 2.7, but with python 3.3 gave me this error:
Installing collected packages: plumbum
Running setup.py install for plumbum
File "/usr/local/lib/python3.3/dist-packages/plumbum/paramiko_machine.py", line 64
print "!!", repr(line)
^
SyntaxError: invalid syntax
I installed it with:
sudo pip install -U plumbum
on a Ubuntu 12.10 machine
I've most of the locale class use cases and everything work smoothly, but have no remote machine to test so I'm not sure if the paramiko class really works or not.
Hi,
when using some sudo commands I found out that they were failing where they shouldn't have. E.g.:
from plumbum.cmd import sudo, yum
sudo(yum['install']['rubygem(rails)'])
This fails, because rubygem(rails) gets overquoted by formulate (which calls shquote). This ends up formulated like this:
['/usr/bin/sudo', '/usr/bin/yum', 'install', "'rubygem(rails)'"]
So yum in fact gets "'rubygem(rails)'" on commandline (because Popen puts the outer quotes there.
I think that the problem can be solved by adding '(' and ')' to _safechars, which seems ok to me, but I'm not sure of other consequences of this. Any other options how to solve this?
Thanks.
In path.walk, the filter function is not passed recursively:
for p in self.list():
if filter(p):
yield p
if p.isdir():
for p2 in p.walk(FILTERNEEDSTOGOHERE):
yield p2
$ ./myapp.py --foo x --bar y
should result in foo
being run first, then bar
.
local['ls'] seems strange to me. I think local('ls') makes more sense - it looks like a constructor call.
Instead of having the Popen code in LocalCommand
, move it into local._popen
or something. That way we could apply context managers on local commands, e.g.,
with local.as_user("foo"):
ls()
(see also #9)
Tried 0.9.0, and also latest git. Same result. Seems that grep isn't understanding unix line endings?
In [4]: chain = ls["-1"] | grep[r"\.py"]
In [5]: chain()
Out[5]:
u'ave_freq_est.py\nave_freq_est.py~\nave_freq_est.pyc\nconst16apsk.py\nconst16apsk.pyc\nconstant_gen.py\nconstant_gen.pyc\nread_hist.py\nread_hist.py~\nsync_time.py\nsync_time.py~\ntest_ave_freq_est.py\ntest_ave_freq_est.py~\ntest_btr2.py\ntest_btr2.py~\ntest_conv.py\ntest_freq_est.py\ntest_uw_freq.py\ntest_uw_freq.py~\ntest_uw.py\ntest_uw.py~\n'
I'm interested in using plumbum to start long-running simulations (replacing my own code that I use for this purpose). One feature I use is to start programs as daemons, so that they will not be killed by HUP. I usually build this into my client code:
if opt.daemon:
from daemon import basic_daemonize
basic_daemonize()
But it seems a nice, simple addition to plumbum 'run'. Along with, my own code will handle logging of stdout, stderr. Again, plumbum could handle this.
The only examples I see are like:
ls = local['ls']
res = ls('some string')
But I need:
res = ls([some, list, of, args])
A command set to run in the background (or with cmd.popen) will block after filling its stdout buffer until .wait() is called.
I'm not sure there is a way to fix this. Just adding a warning in the documentation might be the best that can be done.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.