saleyn / erlexec Goto Github PK

We need a timeout that is more than 5 seconds. It'd be nice to make this configurable - possibly on a per-process level.

It'd be nice to be able to do exec:stop(Pid) without crashing whatever has linked to the Pid. My use case: I have a gen_server that starts and stops programs at various times, and it'd be nice to differentiate between "oh no, the external program has crashed" and "I wanted to stop it".

I think it might be sufficient to do something like

unlink_and_stop(Pid) ->
    erlang:unlink(Pid),
    stop(Pid).

Here's what I see consistently with latest version from master branch:

%% 0 exit code is correct
exec:run("ls", [sync, stdout]).
{ok,[{stdout,[<<"AUTHORS\nLICENSE\nMakefile\nREADME\nTODO\nc_src\nebin\ninclude\npriv\nrebar.config\nrebar.config.scrip"...>>]}]}

%% Any non-0 exit code is broken
exec:run("blah", [sync, stdout]).
{error,[{exit_status,32512}]}

%% Real exit code
exec:run("blah; echo \$?", [sync, stdout]).
{ok,[{stdout,[<<"127\n">>]}]}

Happens both on x86_64-unknown-linux-gnu and x86_64-apple-darwin13.1.0.

Versions are:

Linux precise64 3.8.0-35-generic #52~precise1-Ubuntu SMP Thu Jan 30 17:24:40 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Mac is: OS X 10.10 preview 2

I'm not sure entirely what yet, but there is something wrong with the stop_child code.

I've been doing some debugging, and get very strange result for the time difference. I think it has something to do with this:

    uint32_t sec()      const   { return m_tv.tv_sec;  }

By returning it unsigned, it's causing problems, because in reality the difference is a negative number.

I'm trying the following code:

exec:run(["/usr/bin/docker", "commit", TempName, FinalName], [sync, stdout, stderr])

And sometimes I get the usual response with exit codes and stdout/stderr strings, but sometimes I get this:

{error, [{reason,
            {'DOWN', #Ref<0.0.65.102600>, process, <0.14791.3>, normal}}]}

I checked and the command actually was executed just fine and exited properly, but for some reason the return is not the usual exit status.

I've enabled verbose/debug mode and enabled tracer for exec module – https://gist.github.com/dsabanin/80a8a1edb705377bfe34. Any ideas are greatly appreciated. Thanks!

Assuming a program called foo which takes a file as an argument, how do I do something like this e.g. in bash I could redirect an argument's contents into a temp file with something like foo <(echo "hello world"). Is there a way to do something similar in pure Erlang without having to shell out to bash?

When PTY mode is enabled, it looks like there some mixup happening between STDOUT and STDERR buffers. Here's how it looks like:

Cmd = "for i in `seq 1 400`; do echo $i; done",
{ok, [{stdout, Stdout}, {stderr, Stderr}]} = exec:run(Cmd, [sync,pty,stdout,stderr]).

Results in:

{ok,[{stdout,[<<"1\r\n2\r\n3\r\n">>,<<"6\r\n">>,<<"7\r\n">>,
              <<"8\r\n">>,<<"9\r\n10\r\n">>,<<"11\r\n">>,<<"12\r\n">>,
              <<"13\r\n">>,<<"14\r\n">>,<<"15\r\n">>,<<"16\r\n">>,
              <<"17\r\n">>,<<"18\r\n">>,<<"19\r\n">>,<<"21\r\n22\r\n">>,
              <<"23\r\n">>,<<"24\r\n">>,<<"25\r\n">>,<<"27\r\n">>,
              <<"30\r\n">>,<<"31\r\n">>,<<"32\r\n33\r\n">>,<<"35\r\n">>,
              <<...>>|...]},
     {stderr,[<<"4\r\n5\r\n">>,<<"20\r\n">>,<<"26\r\n">>,
              <<"28\r\n29\r\n">>,<<"34\r\n">>,<<"41\r\n">>,<<"45\r\n">>,
              <<"49\r\n">>,<<"56\r\n">>,<<"63\r\n">>,<<"68\r\n">>,
              <<"70\r\n">>,<<"77\r\n">>,<<"85\r\n">>,<<"88\r\n">>,
              <<"91\r\n">>,<<"97\r\n">>,<<"101\r\n">>,<<"111\r\n">>,
              <<"116\r\n">>,<<"120\r\n">>,<<"123\r"...>>,<<...>>|...]}]}

Please note that the command produces no STDERR at all. As far as I understand it, in pseudo TTY mode there can be no STDERR/STDOUT separation, I'm not sure why it puts some of the read from STDOUT into STDERR.

OS: Linux, Mac.
erlexec version at commit 9178aec

Any help or advice on how to debug this is highly appreciated.

Under a high number of child proceses erlexec starts leaving defunct processes which eventually fill up the process table space in the kernel if the user has no process limits. Rather dangerous.
To replicate, run this in the shell:

[ exec:run("echo ho",[stdout]) || T <- lists:seq(1,10000) ].

You may need more than 10000 processes if you have a faster machine. I tried it on a quad core AMD 6, it always leaves defunct processes. I think that the problem is that the port program does not call wait() in some extreme cases when the child process is barely able to start (and dies immediately because of lack of resources?) . But I am not sure. You should be able to replicate it, use more than 10,000 if you have a bigger machine.

Hello,

Steps to replicate the issue:

06:38 PM  ~
tim@gil$ erl
Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false]

Eshell V6.2  (abort with ^G)
(grout@gil)1> exec:start([debug]).
{ok,<0.50.0>}
(grout@gil)2> exec:run("ls -l", [ sync, {cd, "/non/existent/directory"} ]).
  Redirecting [stdin -> null]
  Redirecting [stdout -> null]
  Redirecting [stderr -> null]
Starting child: 'ls -l'
  child  = (stdin=null(fd:5), stdout=null(fd:5), stderr=null(fd:5))
  parent = (stdin=none, stdout=none, stderr=none)
  Args[0]: /bin/bash
  Args[1]: -c
  Args[2]: ls -l

and the shell sits there, waiting, and a core loops at 100%.

Then I type Ctrl-G q to exit:

^G
User switch command
 --> q
Broken Erlang command pipe (0): Success
Setting alarm to 12 seconds
Got signal: 15 (oktojump=0)
Called kill(pid=0, sig=15) -> 0
Called kill(pid=11058, sig=15) -> 0
Sent SIGTERM to pid 11058 (timeout=5s)
Got signal: 15 (oktojump=1)

06:38 PM  ~
tim@gil$ 

... some time passes, and then ...

Called kill(pid=11058, sig=9) -> 0
Child process 11058 exited
* Process 11058 (ret=-1, status=0, sig=17, oktojump=0, exited_count=0)
Got signal: 13 (oktojump=0)
Exiting (0)

On another run, before exiting the Erlang VM, I attached gdb to the exec-port process while it was spinning and gdb stopped it inside of the check_pending() function:

tim@gil$ gdb ./exec-port 9916
GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/tim/erl-libs/erlexec/priv/x86_64-unknown-linux-gnu/exec-port...done.
Attaching to program: /home/tim/erl-libs/erlexec/priv/x86_64-unknown-linux-gnu/exec-port, process 9916
Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libgcc_s.so.1
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x00007f7605ec04bc in sigpending () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007f7605ec04bc in sigpending () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x0000000000405bb2 in check_pending () at c_src/exec.cpp:498
#2  0x0000000000406533 in main (argc=2, argv=0x7fff8bb3e9d8) at c_src/exec.cpp:629
(gdb) 

If this is not easily replicated on another system I can dig further.

First of all, thanks for the awesome library! You really covered a lot of use cases and I'm really enjoying how it works. I'm having a minor issue, but the overall experience is great. Thanks for the docs too!

I'm trying to enable PTY mode, but for some reason it doesn't work with the following error:

Cannot disable pty echo: Inappropriate ioctl for device

What could be the reason for this? Am I doing something wrong? Thanks!

Code excerpt:

OutputHandler = fun (Stream, OsPid, Data) -> ... end,
exec:run(SetupCmd, [sync, pty, {kill_timeout, ?SETUP_TIMEOUT},
                               {stdout, OutputHandler},
                               {stderr, OutputHandler}]).

Just in case it matters, Erlang VM (R17) is running in VirtualBox VM under Mac OS X. The command I executes starts a docker container.

This bash script:

#!/bin/bash

rm -f /tmp/lpr

trap 'echo vampire_process_wont_die >> /tmp/lrp' SIGINT SIGTERM

while [ true ] ;
 do date >> /tmp/lrp
    sleep 1
done

Doesn't get killed by exec in a clean way.

exec:start(),
{ok, Pid, _} = exec:run("./lrp.sh", []),
timer:sleep(1000),
exec:stop(Pid).

Under some circumstances, it is necessary to use the root user. For example, I'm developing a remote manager for docker instances, and I want to do things like using iptables. For sure there are other ways to do this, but they are much more complex. I don't see any problem in running as root a program I have the source code and it is not setuid.

I have commented out the root-checking lines from exec.cpp and it seems to work perfectly. Would it be interesting to add an option to exec.cpp and exec.erl to allow root access? Do you see any problem in that?

If it is ok I will contribute it.
Thanks

When using {stdout, stderr} redirection option, output is printed to screen instead of being delivered to the calling process:

6> f(I), {ok, _, I} = exec:run("echo TEST", [stderr, {stdout, stderr}]).         
{ok,<0.1396.0>,11509}
7> TEST

7> flush().
ok

Workaround - add 1>&2 redirect on command line:

9> f(I), {ok, _, I} = exec:run("echo TEST 1>&2", [stderr, {stdout, stderr}]).
{ok,<0.1399.0>,11510}
10> flush().                                                                  
Shell got {stderr,11510,<<"TEST\n">>}
ok

Hi @saleyn!

Thanks for the great library. Building via hex dep on OS X is failing currently, because 2 files are present in the c_src dir in the hex package: ei++.d and exec.d. After deleting these files, the build works perfectly. Could you bump the version and publish to hex without those two files?

Thanks!

It'd be convenient, in some circumstances, to accept iolist() commands, rather than strings.

I need some kind of call so that I can ensure that when it returns, or sends me a message or something, the underlying process is really and truly dead.

Since I'm calling this from terminate() of a gen_server, I think something that just plain blocks is probably what I'm after.

Hello,

I just added erlexec to my application dependencies and then in the sys.config I added:

    {exec,[
        {root, true}
    ]},

Once the application is starting is not finding any environment variable. I suppose because the name of the app is erlexec and not exec.

So changing the code to case application:get_env(erlexec, Option) of into the following function

add_option(Option, Acc) ->
    case application:get_env(exec, Option) of
    {ok, Value} -> [{Option, Value} | Acc];
    undefined   -> Acc
    end.

And also the app name into erlexec into sys.config fixed my issue.

What do you think ?

I've been trying to build a program with erlexec on mac os x, but noticed some fairly strange behaviours in my tests of the erlexec part. I've got some processes that run exec:run_link. If I run a few of these processes, everything works fine. But if I run a lot at once, most of them start timing out in a genserver call inside exec:run_link. From that point on, no exec:run_link call will work.

I used observer & some tracing to see what's going on when this happens: the exec process trans queue has a ton of queued transactions, and it stops receiving any handle_info calls from the exec-port application.

So, I attached a debugger to exec-port, and it seems like it's getting stuck in an infinite loop inside the check_pending function. It gets as far as this while loop, with errno set to EINTR. Which seems to be causing an infinite loop - as far as I can see sigtimedwait is stubbed on on mac, so nothing ever clears errno and the program loops forever.

Any ideas how to go about fixing this?

Symptom:

In my code I have a call to

exec:stop_and_wait(I, 5000),

This crashes with an error {badmatch, []} when called, because of this match in exec:

Expected output:

I would probably have expected the code to return an error term, something like {error, not_found} in this case. The mon referenced by I is gone when the call is made, but I'm doing it to clean up in the case it is not dead yet.

Question:

Is this intended behavior or an error? Am I using the API incorrectly?

Running erlexec with monitor as an option, I receive {'DOWN', #Ref<0.0.0.3303>, process, <0.207.0>, {exit_status, 256}}, however running the command in the shell returns the error code 1.

The command is started with:

exec:run(["/path/to/command", "--id", "1", "--control", ":3333"], [monitor, {stdout, print}, {stderr, print}, {kill_timeout, 1}])

The output and message received is:

Got stderr from 66903: <<"time=\"2015-07-16T16:18:53+02:00\" level=fatal msg=\"Could not create controller: dial tcp 127.0.0.1:23000: connection refused\" \n">>
{'DOWN', #Ref<0.0.0.3303>, process, <0.207.0>, {exit_status, 256}}

However, running the command in the shell shows the error code is actually 1:

$ /path/to/command --id 1 --control :3333
FATA[0000] Could not create controller: dial tcp :3333: connection refused
$ echo $?
1

In libuv you can pass any file descripto to the process after stdin,stdout,stderr, having an option to do it in erlexec would help:

http://nikhilm.github.io/uvbook/processes.html#child-process-i-o

I'm currently using this module to provide bash shell, and it works great.
The only problem that I've encountered so far is that the tty is sized at 80X24, which is moderately inconvenient for actually using the shell.

At the time when I'm actually exec'ing the subprocess, I know what terminal dimensions I would need, (as well as some other tty options, but those are gracefully handled by reset) so a mechanism to set tty options or at least dimensions would be very helpful.

It would be nice if we could issues a SIGINT, then a SIGTERM and then a SIGKILL so our python processes can clean up their resources nicely. Or alternatively just choose the signal we send before sending the SIGKILL.

{kill_timeout, Sec::integer()}
Number of seconds to wait after issueing a SIGTERM or executing the custom kill command (if specified) before killing the process with the SIGKILL signal

This may be because Lion is not only 64 bit but also uses LLVM instead of the gcc tool chain. I will install gcc et al via MacPorts and try that way as well, but it's probably worth being aware that this fails on a default Lion install. Here's the debug output from rebar:

$ rebar compile -v 4
DEBUG: Evaluating config script "/usr/local/src/erlang/erlexec/rebar.config.script"
DEBUG: Consult config file "/usr/local/src/erlang/erlexec/rebar.config"
DEBUG: Rebar location: "/Users/t4/bin/rebar"
DEBUG: Available deps: []
DEBUG: Missing deps  : []
DEBUG: Predirs: []
==> erlexec (compile)
DEBUG: Matched required ERTS version: 5.9.1 -> .*
DEBUG: Matched required OTP release: R15B01 -> .*
DEBUG: erl_opts [debug_info,warnings_as_errors,warn_export_all]
DEBUG: Starting 3 compile worker(s)
DEBUG: Worker exited cleanly
INFO:  Skipped src/exec_app.erl
DEBUG: Worker exited cleanly
INFO:  Skipped src/exec.erl
DEBUG: Worker exited cleanly
INFO:  Skipping c_src/ei++.cpp
INFO:  sh info:
    cwd: "/usr/local/src/erlang/erlexec"
    cmd: g++ -c $CXXFLAGS -g -Wall -fPIC  -I/Users/t4/Library/Erlang/Versions/R15B01-l64/lib/erlang/lib/erl_interface-3.7.7/include -I/Users/t4/Library/Erlang/Versions/R15B01-l64/lib/erlang/erts-5.9.1/include   c_src/exec.cpp -o c_src/exec.o
DEBUG:  opts: [{env,[{"ABBOT_HOME","/usr/local/src/web/abbot"},
                     {"Apple_PubSub_Socket_Render",
                      "/tmp/launch-mSCKGl/Render"},
                     {"Apple_Ubiquity_Message",
                      "/tmp/launch-JFoaqK/Apple_Ubiquity_Message"},
                     {"BINDIR",
                      "/Users/t4/Library/Erlang/Versions/R15B01-l64/lib/erlang/erts-5.9.1/bin"},
                     {"CABAL_BIN","/Users/t4/.cabal/bin"},
                     {"CATALINA_HOME","/usr/local/tomcat"},
                     {"CATALINA_OPTS",
                      "-Xms512m -Xmx2046m -XX:MaxPermSize=512m"},
                     {"CC","g++"},
                     {"CLICOLOR","1"},
                     {"COLOR_BLACK","\\e[0;30m"},
                     {"COLOR_BLUE","\\e[0;34m"},
                     {"COLOR_BROWN","\\e[0;33m"},
                     {"COLOR_CYAN","\\e[0;36m"},
                     {"COLOR_GRAY","\\e[0;30m"},
                     {"COLOR_GREEN","\\e[0;32m"},
                     {"COLOR_LIGHT_BLUE","\\e[1;34m"},
                     {"COLOR_LIGHT_CYAN","\\e[1;36m"},
                     {"COLOR_LIGHT_GRAY","\\e[0;37m"},
                     {"COLOR_LIGHT_GREEN","\\e[1;32m"},
                     {"COLOR_LIGHT_PURPLE","\\e[1;35m"},
                     {"COLOR_LIGHT_RED","\\e[1;31m"},
                     {"COLOR_NC","\\e[0m"},
                     {"COLOR_PURPLE","\\e[0;35m"},
                     {"COLOR_RED","\\e[0;31m"},
                     {"COLOR_WHITE","\\e[1;37m"},
                     {"COLOR_YELLOW","\\e[1;33m"},
                     {"COMMAND_MODE","unix2003"},
                     {"CXX","g++"},
                     {"DISPLAY","/tmp/launch-ohLkAD/org.x:0"},
                     {"DRV_CC_TEMPLATE",
                      "g++ -c $CFLAGS -g -Wall -fPIC  -I/Users/t4/Library/Erlang/Versions/R15B01-l64/lib/erlang/lib/erl_interface-3.7.7/include -I/Users/t4/Library/Erlang/Versions/R15B01-l64/lib/erlang/erts-5.9.1/include   $PORT_IN_FILES -o $PORT_OUT_FILE"},
                     {"DRV_CFLAGS",
                      "-g -Wall -fPIC  -I/Users/t4/Library/Erlang/Versions/R15B01-l64/lib/erlang/lib/erl_interface-3.7.7/include -I/Users/t4/Library/Erlang/Versions/R15B01-l64/lib/erlang/erts-5.9.1/include  "},
                     {"DRV_CXX_TEMPLATE",
                      "g++ -c $CXXFLAGS -g -Wall -fPIC  -I/Users/t4/Library/Erlang/Versions/R15B01-l64/lib/erlang/lib/erl_interface-3.7.7/include -I/Users/t4/Library/Erlang/Versions/R15B01-l64/lib/erlang/erts-5.9.1/include   $PORT_IN_FILES -o $PORT_OUT_FILE"},
                     {"DRV_LDFLAGS",
                      "-bundle -flat_namespace -undefined suppress  -L/Users/t4/Library/Erlang/Versions/R15B01-l64/lib/erlang/lib/erl_interface-3.7.7/lib -lerl_interface -lei"},
                     {"DRV_LINK_TEMPLATE",
                      "g++ $PORT_IN_FILES $LDFLAGS -bundle -flat_namespace -undefined suppress  -L/Users/t4/Library/Erlang/Versions/R15B01-l64/lib/erlang/lib/erl_interface-3.7.7/lib -lerl_interface -lei -o $PORT_OUT_FILE"},
                     {"DYLD_LIBRARY_PATH",
                      "/lib:/usr/lib:/usr/local/lib::/opt/local/lib:"},
                     {"EMU","beam"},
                     {"ERLANG_ARCH","64"},
                     {"ERLANG_TARGET",
                      "R15B01-i386-apple-darwin11.3.0-64-unix"},
                     {"ERL_CFLAGS",
                      " -I/Users/t4/Library/Erlang/Versions/R15B01-l64/lib/erlang/lib/erl_interface-3.7.7/include -I/Users/t4/Library/Erlang/Versions/R15B01-l64/lib/erlang/erts-5.9.1/include  "},
                     {"ERL_EI_LIBDIR",
                      "/Users/t4/Library/Erlang/Versions/R15B01-l64/lib/erlang/lib/erl_interface-3.7.7/lib"},
                     {"ERL_LDFLAGS",
                      " -L/Users/t4/Library/Erlang/Versions/R15B01-l64/lib/erlang/lib/erl_interface-3.7.7/lib -lerl_interface -lei"},
                     {"ERL_LIBS","/Users/t4/Library/Erlang/Site"},
                     {"ERL_ROOT","/Users/t4/Library/Erlang/Current"},
                     {"ERL_TOP","/Users/t4/Library/Erlang"},
                     {"EXE_CC_TEMPLATE",
                      "g++ -c $CFLAGS -g -Wall -fPIC  -I/Users/t4/Library/Erlang/Versions/R15B01-l64/lib/erlang/lib/erl_interface-3.7.7/include -I/Users/t4/Library/Erlang/Versions/R15B01-l64/lib/erlang/erts-5.9.1/include   $PORT_IN_FILES -o $PORT_OUT_FILE"},
                     {"EXE_CFLAGS",
                      "-g -Wall -fPIC  -I/Users/t4/Library/Erlang/Versions/R15B01-l64/lib/erlang/lib/erl_interface-3.7.7/include -I/Users/t4/Library/Erlang/Versions/R15B01-l64/lib/erlang/erts-5.9.1/include  "},
                     {"EXE_CXX_TEMPLATE",
                      "g++ -c $CXXFLAGS -g -Wall -fPIC  -I/Users/t4/Library/Erlang/Versions/R15B01-l64/lib/erlang/lib/erl_interface-3.7.7/include -I/Users/t4/Library/Erlang/Versions/R15B01-l64/lib/erlang/erts-5.9.1/include   $PORT_IN_FILES -o $PORT_OUT_FILE"},
                     {"EXE_LDFLAGS",
                      " -L/Users/t4/Library/Erlang/Versions/R15B01-l64/lib/erlang/lib/erl_interface-3.7.7/lib -lerl_interface -lei"},
                     {"EXE_LINK_TEMPLATE",
                      "g++ $PORT_IN_FILES $LDFLAGS  -L/Users/t4/Library/Erlang/Versions/R15B01-l64/lib/erlang/lib/erl_interface-3.7.7/lib -lerl_interface -lei -o $PORT_OUT_FILE"},
                     {"FEDORA_HOME","/usr/local/fedora"},
                     {"FLASH_HOME",
                      "/Applications/Flash Player.app/Contents/MacOS"},
                     {"GODI_TOP","/Users/t4/Library/OCaml"},
                     {"GREP_COLOR","1;32"},
                     {"GREP_OPTIONS","--color=auto"},
                     {"HAMCREST_PYPATH",
                      "/usr/local/src/python/hamcrest/hamcrest-read-only/hamcrest-python"},
                     {"HISTCONTROL","ignoredups"},
                     {"HOME","/Users/t4"},
                     {"JAVA_HOME",
                      "/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home"},
                     {"JDK_HOME",
                      "/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home"},
                     {"JRUBY_HOME","/usr/local/src/java/jruby/jruby1.0-svn/"},
                     {"JYTHON_HOME","/usr/local/jython"},
                     {"LANG","en_GB.UTF-8"},
                     {"LD_LIBRARY_PATH","/lib:/usr/lib:/usr/local/lib:"},
                     {"LIBPATH","/usr/local/xerces/lib:"},
                     {"LOGNAME","t4"},
                     {"M2_HOME","/usr/local/src/java/maven"},
                     {"MANPATH","/Users/t4/Library/Erlang/Current/man:"},
                     {"MAVEN_OPTS","-Xms768m -Xmx2046m -XX:MaxPermSize=1024m"},
                     {"ORACLE_HOME",[]},
                     {"PARSE_TRANS_ROOT",
                      "/Users/t4/Library/Erlang/Site/parse_trans"},
                     {"PATH",
                      "/Users/t4/Library/Erlang/Versions/R15B01-l64/lib/erlang/erts-5.9.1/bin:/Users/t4/Library/Erlang/Versions/R15B01-l64/lib/erlang/bin:/usr/local/pgsql/bin:/Library/PostgreSQL/8.4/bin::/usr/local/fedora/server/bin:/usr/local/fedora/client/bin:/usr/local/src/java/maven/bin:/usr/local/tomcat/bin:/bin:/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home:/usr/local/src/java/jruby/jruby1.0-svn//bin:/usr/local/jython:/usr/local/jython/bin:/Applications/Flash Player.app/Contents/MacOS:/usr/local/src/web/abbot/bin:/Users/t4/Library/Erlang/Current/bin:/Users/t4/Library/OCaml/bin:/Users/t4/.cabal/bin:/System/Library/Frameworks/Ruby.framework/Versions/Current/usr/bin:/Library/Frameworks/Python.framework/Versions/Current/bin:/Users/t4/.bash/utilities:/Users/t4/bin:/Users/t4/bin/cccs:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin:/usr/local/git/bin:/usr/local/MacGPG2/bin"},
                     {"PGDATA","/usr/local/pgsql/data"},
                     {"PGSQLHOME","/Library/PostgreSQL/8.4"},
                     {"POSTGRES_BIN","/usr/local/pgsql/bin"},
                     {"PROGNAME","erl"},
                     {"PWD","/usr/local/src/erlang/erlexec"},
                     {"PYTHONPATH",
                      "/usr/local/src/python/hamcrest/hamcrest-read-only/hamcrest-python:/usr/local/lib:/usr/local/src/python/matplotlib"},
                     {"PYTHON_BASE",
                      "/Library/Frameworks/Python.framework/Versions/Current"},
                     {"ROOTDIR",
                      "/Users/t4/Library/Erlang/Versions/R15B01-l64/lib/erlang"},
                     {"RUBY_HOME",
                      "/System/Library/Frameworks/Ruby.framework/Versions/Current/usr"},
                     {"SHELL","/bin/bash"},
                     {"SHLIB_PATH","/usr/local/xerces/lib:"},
                     {"SHLVL","1"},
                     {"SSH_AUTH_SOCK","/tmp/launch-Hg60op/Listeners"},
                     {"TERM","xterm-color"},
                     {"TERM_PROGRAM","Apple_Terminal"},
                     {"TERM_PROGRAM_VERSION","303.2"},
                     {"TERM_SESSION_ID",
                      "BF8E37CD-05C0-4E33-8FD9-0B954DFE4533"},
                     {"TMPDIR",
                      "/var/folders/lt/jg6yqg751pgfpmxn92l1sz1r0000gn/T/"},
                     {"TNS_ADMIN","/admin"},
                     {"USER","t4"},
                     {"XALANCROOT","/usr/local/src/native/xml-xalan2/c"},
                     {"XERCESCROOT","/usr/local/xerces"},
                     {"__CF_USER_TEXT_ENCODING","0x1F5:0:0"}]},
               return_on_error,
               {use_stdout,false}]
Compiling /usr/local/src/erlang/erlexec/c_src/exec.cpp
/usr/local/src/erlang/erlexec/c_src/exec.cpp: In function ‘void check_pending()’:
/usr/local/src/erlang/erlexec/c_src/exec.cpp:249: error: ‘sigtimedwait’ was not declared in this scope
/usr/local/src/erlang/erlexec/c_src/exec.cpp: In function ‘int main(int, char**)’:
/usr/local/src/erlang/erlexec/c_src/exec.cpp:349: error: ‘setresuid’ was not declared in this scope
/usr/local/src/erlang/erlexec/c_src/exec.cpp: In function ‘pid_t start_child(const char*, const char*, char* const*, int, int)’:
/usr/local/src/erlang/erlexec/c_src/exec.cpp:561: error: ‘setresuid’ was not declared in this scope
ERROR: compile failed while processing /usr/local/src/erlang/erlexec: rebar_abort

When using {stdout, stderr} redirection option, output is printed to screen instead of being delivered to the calling process:

6> f(I), {ok, _, I} = exec:run("echo TEST", [stderr, {stdout, stderr}]).         
{ok,<0.1396.0>,11509}
7> TEST

7> flush().
ok

Workaround: add 1>&2 redirect on command line:

9> f(I), {ok, _, I} = exec:run("echo TEST 1>&2", [stderr]).
{ok,<0.1399.0>,11510}
10> flush().                                                                  
Shell got {stderr,11510,<<"TEST\n">>}
ok

This issue doesn't affect the {stderr, stdout} redirection, which works correctly.

I have enabled verbose debugging and found that exec-port sometimes exits as a result of this line: https://github.com/saleyn/erlexec/blob/master/c_src/exec.cpp#L1620

Could you explain the reasoning behind return -1 there? Unfortunately, I am still working on a reproducible case however it was my understanding that if my application depends on exec then the exec application (and therefore exec-port) should only exit when my application does?

The lines after

check_cmd_options([{Std, I}|T], State) when Std=:=stderr, I=/=Std; Std=:=stdout, I=/=Std ->

in exec.erl include

element(1,I)=:="append"

When the docs describe it as an atom. In the C code, it appears to be treated as an atom as well. Changing that to 'append' generates this error, though:

{error, "Atom, string or {'append', Name} tuple required for option stdout"}

Unfortunately I'm out of time for today.

I faced that problem last week and I can't figure out how it happens. OS processes, that have been ran from erlang process, have parent id = 1. Maybe there are some problems with SIGHUP or SIGPIPE handling in child processes?

This is surprising to me:

30> exec:run_link("sleep 1; exit 1", []).                   
{ok,<0.118.0>,20435}
** exception error: {exit_status,256}
31> exec:run_link("sleep 1; exit 2", []).
{ok,<0.121.0>,20437}
** exception error: {exit_status,512}

Why is the reported exit status shifted 8 bits?

Sync run always fails with non empty process mailbox box due to this piece of code.

Example

1> exec:run("ls", [sync]).
{ok,[]}
2> self() ! test.         
test
3> exec:run("ls", [sync]).
{error,[{reason,test}]}

Hello Sergei,

I have an issue with passing fake OS pid to manage function:

FakeOsPid = 12345.
exec:manage(FakeOsPid, []).

I expect 'error', but always got 'ok'.

c_src/exec.cpp: In function ‘pid_t start_child(const char*, const char*, char* const*, int, int)’:
c_src/exec.cpp:581:48: error: ‘PRIO_PROCESS’ was not declared in this scope
c_src/exec.cpp:581:71: error: ‘setpriority’ was not declared in this scope

Adding #include <sys/resource.h> fixes it.

Lots of apps wait for stdin to be closed as an indication it's time to do something meaningful (like exit). Without the ability to close stdin, one needs to create an intermediary wrapper that knows uses some inbound signal to close stdin (e.g. two \n, EOF, etc.)

The test example conveniently reads one line and exits.

A trivial example is to use write to cat's stdin.

Trying to include {erlexec, "1.2.1"} as a dependency via rebar3 fails with an uncaught error in rebar_core.

From the crash dump:

Error: {badmatch,{error,{35,file,
                         {error,{badmatch,{error,enoent}},
                                [{erl_eval,expr,3,[]}]}}}}

The default behavior should be for child processes to inherit the environment - that's what people expect when exec'ing.

A workaround is to pass in the environment every time:

[{env, os:getenv()}]

But that's a pain in the neck.

Hi,

I'm running into some issues with compiling v1.1.1 of erlexec from hex.pm for x86_64-alpine-linux-musl. Specifically, it appears that the test.cpp file is included in the compilation process even though it should not be (defines a second main, has #include <ei++.hpp> at the top which causes compilation to fail, etc.). Should this file be removed, or excluded from compilation somehow? I'm on 1.1.0 for now which is fine, but would like to upgrade to v1.1.1.

Thanks!

Hi,

My reading of the source code indicates you can't capture stderr and stdout programmatically, by, say, having them sent to Erlang as messages, like open_port does.

It'd be nice to have some options along the lines of

{stdout, Pid}, {stderr, Pid}

So that you could have erlexec send the output back to Erlang for processing.

There is an interface lag when large applications I run through erlexec are run while in console mode.

First the commands, set up an environment in an empty directory with:

[ -d erlexec ] || git clone https://github.com/saleyn/erlexec.git
cd erlexec
rebar compile
[ -f 'minecraft_server.1.6.2.jar' ] || wget 'https://s3.amazonaws.com/Minecraft.Download/versions/1.6.2/minecraft_server.1.6.2.jar'
erl -pa $PWD/ebin

The erlang console will now be running, I have the same result on both R15B01 and R16B01 on two different systems, both Debian based of differing versions, the main shell used is bash but also tried dash and zsh, both through ssh to remote servers and local with no change.

In the erlang shell now run:

exec:start([debug]).
{_, P, _} = exec:run_link("java -Xms1G -Xmx1G -jar minecraft_server.1.6.2.jar nogui", [{stdout, self()}, {stderr, self()}]).

Now try typing, a large amount of the key presses vanish at this point, press and hold, say, the q key and you will, say, none for 3 seconds, then 1, then none for a second, then like 10, then none for another second, then 1, then none for 5 seconds and so forth.

This does not happen with erlang ports.

It stops when the above launched server process dies, whether with exec:kill(P, 9) or on another shell with kill pidof java 9 or whatever, at which point the erlang shell becomes perfectly responsive again.

It also happens if it is linked to another spawned process and not the shell process, though I have not tested launching over a remote erlang shell.

Something else noticed, the above server java program prints, say '2013-08-11 04:05:59 [INFO] Unknown command. Try /help for a list of commands\n' for any unknown text command sent it it, and if I 'flush().' after running the above command to start the server it will also print out a few (random?) lines of the above, of which if I run 'flush().' again I get more, and if I run 'flush().' again then I get more of the 'Unknown command', thus meaning it is getting an input, and I have not attempted to start using stdin yet.

And yet I have just noticed that when I copied in "flush().\n" to my clipboard and paste it into the console terminal, the number of times that I tap Ctrl+Alt+V and it does nothing equals the number of times that the 'Unknown command' message is received, thus indicating that a large number of randomish input that is typed into the erlang console window is actually passed to the started process. Further testing seems to confirm this. Just tested again, this does not happen with erlang ports.

It did do all of the above as of a week ago as well, prior to the recently made changes and added stdin support.

When a process that has called exec:run_link dies, the exec process just exits with owner_died. I was under the assumption that this should cause exec to run the kill_cmd / kill_group or at least kill the linked OS processes.

it would be great to support pty for the spawned children. for example,

exec:run("ed", [stdin, stdout, pty]).

then it will be easy to interact with some line-buffer program correctly.

None of the examples cover how to close stdin after providing custom generated input. For example calling tac.

1> application:start(erlexec).
ok
2> Watcher = spawn(fun F() -> receive Msg -> io:format("Got: ~p~n", [Msg]), F() end end).
<0.103.0>
3> {ok, Pid, OsPid} = exec:run("tac", [stdin, {stdout, Watcher}, {stderr, Watcher}]).
{ok,<0.105.0>,15922}
4> exec:send(Pid, <<"foo\n">>).                                                      
ok
5> exec:send(Pid, <<"bar\n">>).                                                      
ok
6> exec:send(Pid, <<"baz\n">>).                                                      
ok

And what I am supposed to do now? exec:manage/2 doesn't do the trick:

7> exec:manage(OsPid, [{stdin, close}]).
{ok,<0.110.0>,15922}

I am missing command for closing stdin of the existing and opened stdin of the existing command.

Compare with exec:run("echo -e 'foo\\nbar\\nbaz' | tac", [{stdout, Watcher}, {stderr, Watcher}]).

I got port crash when command line length is equal 256 bytes (255 or 257 working fine). Look like some allocation problem. Cmd params was just [sync].

Error: glibc detected *** /app/deps/erlexec/priv/x86_64-unknown-linux-gnu/exec-port: double free or corruption (out): 0x00007fffe535ef00

First, using erlangs ports instead of erlexec does not have the issues described below, it is fast, responsive, get each line in a message and so forth

The program below when run will start printing information to stderr immediately (it does not use stdout, no do not know why, but that part is not related, add 2>&1 to the call and still the same issue), however the erlang process does not receive it until I send the kill command, then instead of receiving them one message per line like I do with erlang ports I get it all at once along with the message that it was killed, though I do get those messages as soon as it was killed.

In the same program another issue is demonstratable, remove the kill call and everything after it and try using the erl shell. On all 3 computers I tested a lot of my typed characters are seemingly silently dropped while the called process is running, though when killed then everything is responsive again.

Occasionally and often when the erlang program is quit (with say q().) then the erlexec program and associated process it started does not die either and they continue to live as zombies until I kill them from bash.

All of the above issues that is shown with the below program are identically shown on 3 very different computer running 3 very different debian version with two different Erlang versions (15B and 16B).

Demonstration script: https://gist.github.com/OvermindDL1/1d8ca244db81d9000050

Result:

timeout {1375,893135,404052}
timeout {1375,893136,405023}
timeout {1375,893137,406024}
timeout {1375,893138,407660}
timeout {1375,893139,408163}
timeout {1375,893141,410052}
timeout {1375,893142,411054}
timeout {1375,893143,412156}
timeout {1375,893144,413055}
timeout {1375,893145,414090}
timeout {1375,893156,420051}
timeout {1375,893157,421057}
timeout {1375,893158,422124}
timeout {1375,893159,423055}
timeout {1375,893160,424056}
Message {1375,893160,451161}: {stderr,24122,
                                      <<"2013-08-07 10:32:19 [INFO] Starting minecraft server version 1.6.2\n2013-08-07 10:32:19 [INFO] Loading properties\n2013-08-07 10:32:19 [INFO] Default game type: SURVIVAL\n2013-08-07 10:32:19 [INFO] Generating keypair\n2013-08-07 10:32:20 [INFO] Starting Minecraft server on *:25565\n2013-08-07 10:32:21 [INFO] Preparing level \"world\"\n2013-08-07 10:32:21 [INFO] Preparing start region for level 0\n2013-08-07 10:32:22 [INFO] Preparing spawn area: 13%\n2013-08-07 10:32:23 [INFO] Preparing spawn area: 33%\n2013-08-07 10:32:24 [INFO] Preparing spawn area: 56%\n2013-08-07 10:32:25 [INFO] Preparing spawn area: 78%\n2013-08-07 10:32:26 [INFO] Preparing spawn area: 99%\n2013-08-07 10:32:26 [INFO] Done (5.284s)! For help, type \"help\" or \"?\"\n">>}
Message {1375,893160,452038}: {'EXIT',<0.47.0>,{exit_status,9}}
timeout {1375,893161,453052}
timeout {1375,893162,454024}
timeout {1375,893163,455053}
timeout {1375,893165,457055}
timeout {1375,893166,458387}
timeout {1375,893167,459703}
timeout {1375,893168,460936}
timeout {1375,893169,462051}

(kerl:r17) ~/p/erlexec (master|✔)                                                                                                                                       
> erl -pa ebin
Erlang/OTP 17 [erts-6.4] [source] [64-bit] [smp:8:8] [async-threads:10] [kernel-poll:false]

Eshell V6.4  (abort with ^G)
1> application:start(exec).
ok
2> application:which_applications().
[{exec,"OS Process Manager","v1.0-156-g05edfd2"},
 {stdlib,"ERTS  CXC 138 10","2.4"},
 {kernel,"ERTS  CXC 138 10","3.2"}]
3> spawn(fun () -> io:format("~p~n", [exec:run("sleep 3 && echo done", [sync, stdout])]) end) ! test.
test
{error,[{reason,test}]}

As I can see, there is a clause in exec:wait_for_ospid_exit which treats any incoming message as an error message. Are you sure this behaviour is correct?

Not sure what's behind this. It appears the the -m64 option to g++ is what's breaking it. The errors I get are:

ld: warning: ld: warning: ignoring file c_src/exec.o, file was built for unsupported file format ( 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 ) which is not the architecture being linked (x86_64): c_src/exec.oignoring file c_src/ei++.o, file was built for unsupported file format ( 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 ) which is not the architecture being linked (x86_64): c_src/ei++.o

Undefined symbols for architecture x86_64:
  "_main", referenced from:
     implicit entry/start for main executable
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Removing the -m64 from the build seems to have fixed it. I'm not sure if this is the right way to do it, but you can see a working build at jvantuyl/erlexec@ffa8bc2.

Would be interresting to be able to pass file descriptor number there instead of a proper file, so we could eventually pass any pipe or already opened socket/file

Currently it's only possible to start programs like this:

exec:run("cat " ++ File, []).

This means all parameters have to be carefully escaped. If you could start programs like this:

exec:run(["cat", File], []).

You wouldn't have to do that. It should also be possible to start them without a shell to prevent security issues. See the excellent subprocess module of python:
http://docs.python.org/2/library/subprocess.html#subprocess.Popen

v1.0.0 is from 2012

Can we please tag more recent build that is known to be stable? Thanks!