nickstenning / honcho Goto Github PK

View Code? Open in Web Editor NEW

1.6K 1.6K 144.0 393 KB

Honcho: a python clone of Foreman. For managing Procfile-based applications.

Home Page: http://pypi.python.org/pypi/honcho

License: MIT License

Makefile 1.32% Python 98.47% Shell 0.21%

devtools operations python

honcho's People

Contributors

Stargazers

Watchers

Forkers

jamescasbon msabramo redpie jsullivanlive flashingpumpkin axiak pombredanne nielsmh sergiopica blaze33 lemuelf zen4ever gausam acjay hghazal yoloseem mattions jessepollak ionrock skiold dicato miguelgrinberg jeanphix micurley mardiros web5design baitcode casio slafs kimballfrank xymz pepijndevos sunliwen krallin gratipay philjc shayneoneill eliagbenu tonyseek shannonzylstra hipchat hongphi rmuslimov shipci mgrandi joshfriend michiganlabs jlirochon ziadsawalha hongxinsun azov ratazzi migurski tomas-edwardsson maroux janusnic myyk agmathew xarisd lhrkkk mackjoner mblayman davinirjr barkinet averroes mjumbewu boldbigflank mattrobenolt nicholasserra marckrull heart2009 cloudxtreme adamchainz open-craft russelldavies georgexsh awesome-python mostowski conradlink unterwegi guohongliu01 kydragon x-ops rbramwell bobquest33 rowhit ltsampros jusic jkleckner 1stvamp rmoorman diek wkentaro ronaldosaheki mattllvw hartym gaoshengyue trendingtechnology lins05 achillesrasquinha

honcho's Issues

Allow log format to be configurable

Allow the log format to be overridable.

Default port behavior differs from Foreman

I was trying to use Honcho and Python 3 because Foreman was buggy with file handling, and I noticed some differences.

I set up this gist to test and demonstrate the differences:

Foreman's port precedence is -p option, .env file, PORT variable, default of 5000
Honcho's port precedence is .env file, PORT variable, -p option, default of 5000

Sending SIGINT to processes for graceful shutdown?

Is it possible to get honcho to send SIGINT to child processes so they perform a graceful shutdown instead of SIGTERM?

In particular, Celery (http://www.celeryproject.org/) will respond to SIGINT (CTRL+C) by no longer accepting new tasks from the queue, letting the workers finish, and then shutting down gracefully. When SIGTERM is sent, the processes immediately die.

Would it be possible to have honcho do something similar--where you hit CTRL+C once and it sends SIGINT to the child processes, and if you hit CTRL+C a second time it switches to sending SIGTERM?

Windows-only bug

I did hit this bug today: http://bugs.python.org/issue14252
which is also documented here I think:
http://stackoverflow.com/questions/5573257/windowserror-error-5-access-is-denied-when-trying-to-kill-a-subprocess-pytho
I will come with a patch to support the same approach patched in the main Python 2.7 regardless of the Python version

pdb experience is sub-par

If I set PYTHONUNBUFFERED I can kind of use pdb, but it's less than ideal because the (Pdb) prompt isn't displayed until after I enter a command, and all output is prefixed with the timestamp/process indicator.

"honcho start" gives no output

     $ honcho start
     (...stalled here...)
    ^C
    22:00:48 system | web.1 started (pid=20863)
    22:01:24 system | SIGINT received
    22:01:24 system | sending SIGTERM to web.1 (pid 20863)
    22:01:24 system | web.1 stopped (rc=-15)

But, if I user PYTHONUNBUFFERED it works

    $ PYTHONUNBUFFERED=1 honcho start
    22:02:10 system | web.1 started (pid=20877)
    22:02:10 web.1  | serving on http://0.0.0.0:8080

Sys info :

    $ uname -a
    Darwin Juliens-MacBook-Pro.local 14.1.0 Darwin Kernel Version 14.1.0: Mon Dec 22 23:10:38 PST 2014; root:xnu-2782.10.72~2/RELEASE_X86_64 x86_64
    $ python --version
    Python 3.4.2

Undocumented PORT behavior is surprising

I have one webserver that looks for PORT in its environment, and a few other processes that do bookkeeping with the database and so don't have a listening port. I converted a previous script that manually set PORT= in the webserver's environment over to Honcho, and I couldn't understand why the webserver wasn't listening on the port I specified.

It turns out that I was hitting the undocumented PORT management feature, and I understand the usefulness of that, and how it's there to match Foreman's behavior (which is also mostly undocumented). I just put the webserver first, and it works now. But having PORT change based on Procfile ordering was surprising.

different behaviour of PORT environmental variable when running vs exporting to supervisord

.env:

PORT=8000

Procfile:

web: echo $PORT

Then, when starting:

honcho start
14:11:19 system | web.1 started (pid=73996)
14:11:19 web.1  | 8000
4:11:19 system | web.1 stopped (rc=0)

I'd expect that when exporting to supervisord, the port number defined in .env would also get used, but it doesnt. It falls back to the default port 5000:

honcho export -l . . supervisord
cat myproject.conf
[program:myproject-web]
command=/bin/sh -c 'echo $PORT'
autostart=true
autorestart=true
stopsignal=QUIT
stdout_logfile=./web-0.log
stderr_logfile=./web-0.error.log
user=dries
directory=/Users/dries/Dropbox (Urga)/src/myproject
environment=PORT="5000"

[group:myproject]
programs=myproject-web

Why does export require a log directory?

Why does export require a log directory? It doesn't seem to do anything with it?

$ honcho export . supervisord
[Errno 13] Permission denied: '/var/log/export_test'
2014-11-26 10:11:06 [5971] [ERROR] Can not create /var/log/export_test

$ honcho export . supervisord -l log.d

$ ls -al log.d
total 0
drwxr-xr-x+ 2 marca  staff   68 Nov 26 10:11 ./
drwxr-xr-x+ 5 marca  staff  170 Nov 26 10:11 ../

So why does it need that log directory, if it doesn't write anything to it?

`honcho run` stderr

It looks like honcho run redirects stderr to stdout. This is problematic when stderr is used for logging and stdout writes some data you want to capture to a file. By contrast, foreman keeps the streams separate.

`honcho run` captures arguments that begin with `-`

For example, the command honcho run ./manage.py --version fails because honcho tries to parse the --version argument instead of passing it on to manage.py.

Scriptfile tor one-off scripts

Procfile lists long-running processes that make up the application. But what about one-off scripts like database migrations and cron jobs? I propose we define a new file, Scriptfile, with similar syntax as Procfile, to hold these:

syncdb: ./manage.py syncdb
migrate: ./manage.py schemamigrations myapp

And you would call the scripts by name:

honcho run migrate

Catch KeyboardInterrupt

Thank you for the great project.

If I run honcho run python then Ctl+C to exit python I get the following exception. The exception is just that KeyboardInterrupt is just not caught. But after that no output is sent to the terminal?

Traceback (most recent call last):
  File "/home/simon/.virtualenvs/default/bin/honcho", line 9, in <module>
    load_entry_point('honcho==0.5.0', 'console_scripts', 'honcho')()
  File "/home/simon/.virtualenvs/default/lib/python2.7/site-packages/honcho/command.py", line 315, in main
     app.parse()
  File "/home/simon/.virtualenvs/default/lib/python2.7/site-packages/honcho/command.py", line 129, in parse
    options.func(self, options)
  File "/home/simon/.virtualenvs/default/lib/python2.7/site-packages/honcho/command.py", line 161, in run
    p.wait()
  File "/usr/lib64/python2.7/subprocess.py", line 1376, in wait
    pid, sts = _eintr_retry_call(os.waitpid, self.pid, 0)
  File "/usr/lib64/python2.7/subprocess.py", line 476, in _eintr_retry_call
    return func(*args)
KeyboardInterrupt

support `run {target}`

With foreman I can do foreman run web and it will dereference the command pointed to by web in my Procfile. It seems that honcho tries to interpret web as a shell command in its own right.

Running multiple instances of one command

Is there a way to run multiple instances of a command?

WIth foreman I could run a command like foreman web=2 or have a .foreman file containing concurrency: web=2,bravo=1 - is there an equivalent with honcho?

Add -t/--template option to honcho export

It would be nice to be able to specify a custom Jinja2 template for honcho export.

Foreman docs mention a -t/--template option for specifying an erb template. Honcho could support the same option but use Jinja2 in place of erb.

http://ddollar.github.io/foreman/#EXPORTING

Fix race condition in honcho output

It seems like adding coverage instrumentation to our tests has exposed a race condition in our code: https://travis-ci.org/nickstenning/honcho/builds/20405233

Nor .env or environment variable PORT is used as port

While environment variable PORT is set to 8000 or a .env file containing PORT=8000 exists, honcho still use 5000 as the port value.

Pre-exec support? (related to virtualenvs)

Hello,

I am interested in using honcho, but my project uses a virtualenv. I searched around, and couldn't find anything in the documentation, but i did find that you posted this in December , saying that you can just add the "activate" script to your Procfile and it should work

Well, I tried, that, but it seems the problem is that honcho sees that the source command exits (running source activate.sh in bash), and therefore kills everything else. (This is also assuming that the jobs are started in order, which seems to be this case but i feel like it shouldn't be relied on.)

my proc file: (mac os x 10.10 / python 3.4 if that matters)

venv: source <SOURCE_PATH>/venv_dir/bin/activate
ldb_server: python3 server.py config.yaml

and that produces:

Corvidae:daemon_src markgrandi$ honcho start
04:10:44 venv.1    | started with pid 10991
04:10:44 ldb_server.1 | started with pid 10992
04:10:44 venv.1    | process terminated
04:10:44 system    | sending SIGTERM to all processes
04:10:44 system    | sending SIGTERM to pid 10992
04:10:44 ldb_server.1 | process terminated

So since you said in the above linked bug report that you want to keep this language agnostic and therefore not really add 'specific support for virtualenvs', I propose adding something in the Procfile, or maybe a command switch that allows you to specify what should be run before and after honcho starts everything / kills everything.

It would execute it, and when it exits, honcho knows it can start running everything in the Procfile (as it does now) normally, and when those all exit/are terminated, honcho runs whatever is specified for 'post exec' (like running the 'deactivate' command for venvs). And this would also solve a potential race condition where its not specified anywhere that the honcho jobs are started in order, so to not have the possibility of the other jobs starting before they should. I do not know how this would effect the 'official' use case for Procfiles (heroku) and the exporting functionality

I'm guessing theoretically my specific problem (activating the virtualenv) could be solved by manually putting the the ENV variables that virtualenv creates when you run the activate script, but that is cumbersome, and i feel a pre/post exec feature would be useful in other cases too.

Exported supervisord conf doesn't handle environmental variables well

System:
http://aws.amazon.com/amazon-linux-ami/

Steps to reproduce the bug:
honcho -f Procfile -e .env export supervisord /etc/supervisord.d/app.conf

Use case (in Procfile):

web: $VENV_HOME/bin/newrelic-admin run-program $VENV_HOME/bin/gunicorn -c $SRC_HOME/gunicorn.py wsgi:application

The export command doesn't handle environmental variables found in the system $ env or the .env file loaded by honcho.

command

The command that will be run when this program is started. The command can be either absolute (e.g. /path/to/programname) or relative (e.g. programname). If it is relative, the supervisord's environment $PATH will be searched for the executable. Programs can accept arguments, e.g. /path/to/program foo bar. The command line can use double quotes to group arguments with spaces in them to pass to the program, e.g. /path/to/program/name -p "foo bar". Note that the value of command may include Python string expressions, e.g. /path/to/programname --port=80%(process_num)02d might expand to /path/to/programname --port=8000 at runtime. String expressions are evaluated against a dictionary containing the keys group_name, host_node_name, process_num, program_name, here (the directory of the supervisord config file), and all supervisord's environment variables prefixed with ENV_.
--http://supervisord.org/configuration.html#program-x-section-values

Note my emphasis.

honcho should either expand environmental variables or prefix them with ENV_ as per supervisord's documentation.

Remove coverage monkeypatch for measuring coverage across process boundaries

This came up as an issue in dealing with #106.

honcho/honcho/test/unit/test_manager.py

Line 10 in 9550461

    
           multiprocessing.Process = monkeypatch_process_for_coverage(multiprocessing.Process)

@nedbat suspects that we can probably replace with something better after looking at http://nedbatchelder.com/code/coverage/subprocess.html

Upstart exports the same port for every service

What can I say... my instances don't start because they al try to use the same port.

It just takes the base port given in the config and passes it to all templates.

honcho export takes arguments in reverse order of foreman export

foreman takes FORMAT and then LOCATION.
honcho takes LOCATION and then FORMAT.

Accidental or intentional?

[marca@marca-mac2 export_test]$ foreman help export | head -n 2
Usage:
  foreman export FORMAT LOCATION

[marca@marca-mac2 export_test]$ honcho help export | head -n 4
usage: honcho export [-h] [-e ENV] [-d APP_ROOT] [-f PROCFILE] [-v] [-a APP]
                     [-l DIR] [-p N] [-c process=num,process=num] [-u USER]
                     [-s SHELL]
                     LOCATION FORMAT

Handle multiline strings in .env file

see ddollar/foreman#250

Very useful for working with keys/certs loaded from environment variables.

Processes not always killed on exit

Ocassionally, I am left with all the child processes still running when honcho exits.

Can you be a bit more careful to term the processes?

Issues with Python 2.7.9

14 tests fail on Python 2.7.9, all are integration tests.

======================================================================
FAIL: test_start_with_arg (honcho.test.integration.test_start.TestStart)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/paul/experiments/gratipay/honcho/honcho/test/integration/test_start.py", line 24, in test_start_with_arg
    self.assertEqual(ret, 0)
AssertionError: 1 != 0

======================================================================
FAIL: test_start_with_arg_returncode (honcho.test.integration.test_start.TestStart)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/paul/experiments/gratipay/honcho/honcho/test/integration/test_start.py", line 59, in test_start_with_arg_returncode
    self.assertEqual(ret, 42)
AssertionError: 1 != 42

======================================================================
FAIL: test_start_with_multiple_args (honcho.test.integration.test_start.TestStart)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/paul/experiments/gratipay/honcho/honcho/test/integration/test_start.py", line 35, in test_start_with_multiple_args
    self.assertEqual(ret, 0)
AssertionError: 1 != 0

======================================================================
FAIL: test_unicode_decode_error (honcho.test.integration.test_unicode_decode.TestUnicodeDecode)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/paul/experiments/gratipay/honcho/honcho/test/integration/test_unicode_decode.py", line 10, in test_unicode_decode_error
    self.assertEqual(ret, 0)
AssertionError: 1 != 0

----------------------------------------------------------------------
Ran 86 tests in 6.461s

FAILED (failures=14)

Honcho doesn't seem to be picking up the environment variables - we're having issues over at gratipay/gratipay.com#3025 (comment)

Stop pinning jinja2 versions

I don't get why jinja2==2.7 is pinned for honcho. Jinja2 isn't essential to typical noninteractive honcho operation (honcho start), afaik it's only used to generate the start scripts? So we're protecting against the scenario where a hapless sysadmin (like me) runs into a problem where Armin steathily released a new, broken jinja2 version that is not backwards compatible? When's that going to happen?

It may seem nice to have that extra perceived stability, but if this keeps up, we won't be able to use anything anymore, because libA wants foo==1.0.1 and libB wants foo==1.0.0.

In my case, pinning the version achieved the exact opposite of what was intended: Honcho just broke my heroku app, because it insisted on Jinja2==2.7, while 2.7.1 was installed (no magic, simply adding 'honcho' to the requirements). For some reason, Pip/setup.py did not resolve this properly, but I am highly annoyed at having to spend half an hour to track this down and fixing it by adding jinja2==2.7 to my own setup.py just to make honcho happy.

Note that honcho wasn't throwing errors, it was setuptools/distribute complaining of the requirements mismatch before even starting it.

I haven't even seen a Jinja3 announced yet (and my guess is that the package might be named Jinja3 instead of Jinja2). So can we please just put this to Jinja2 >= 2.7, or just Jinja2?

get_honcho_output in test.helpers seems to assume honcho is installed first

... which is kind of weird when running tests...

Could we instead run the local python pointing to the honcho entry point rather than assuming that "honcho" as a command is available?

Can't stop exported upstart service

I have a simple Procfile:

web: gunicorn myapp.wsgi --bind 0.0.0.0:$PORT --log-file -

Exported to upstart with honcho export -u deploy upstart /etc/init. All good! service myapp start works fine, and service myapp stop appears to work, but upon inspecting ps I can see that the gunicorn proc. is still running.

Foreman incompatibility: nested Honcho not propagating stderr

I'm following the recipe here (http://www.radekdostal.com/content/heroku-running-multiple-python-processes-single-dyno-using-honcho) to run multiple processes in my worker dyno by calling Honcho in my main Procfile. I'm testing locally running Foreman on my main Procfile, which then invokes Honcho on a separate procfile. I've noticed that output from the inner procfile processes Honcho runs is not propagated to the main Foreman process. When using foreman instead for the inner procfile, I can see the nested output from the inner Foreman instance.

So the issue seems to be with the inner honcho process. In particular, this seems to be happening when the inner procfile is running a Python script that writes to stderr, like using the logging module for output. Here's a minimal test case:

test.py

import time
import logging
import sys

logging.basicConfig()
logger = logging.getLogger(__name__)

while True:
    print 'Stdout message'
    print >>sys.stderr,'Stderr message'
    logger.warning('Logged message!')
    time.sleep(2)

TestProcfile

worker: honcho start -f TestSubProcfile

TestSubProcfile

subworker: python test.py

When running using either honcho -f TestProcfile start or foreman -f TestProcfile start, this is what I see:

11:40:07 worker.1 | started with pid 27447
^CSIGINT received
11:40:14 system   | sending SIGTERM to all processes
11:40:14 system   | sending SIGTERM to pid 27447
11:40:14 worker.1 | SIGINT received
11:40:14 worker.1 | process terminated

But if I change TestProcfile to run the inner process with Foreman instead with the line worker: foreman start -f TestSubProcfile, both commands correctly propagate output (with Python's stdout appearing to buffer in the nested process):

12:00:57 worker.1 | started with pid 27944
12:00:57 worker.1 | 12:00:57 subworker.1 | started with pid 27945
12:00:57 worker.1 | 12:00:57 subworker.1 | Stderr message
12:00:57 worker.1 | 12:00:57 subworker.1 | WARNING:__main__:Logged message!
12:00:59 worker.1 | 12:00:59 subworker.1 | Stderr message
12:00:59 worker.1 | 12:00:59 subworker.1 | WARNING:__main__:Logged message!
12:01:01 worker.1 | 12:01:01 subworker.1 | Stderr message
12:01:01 worker.1 | 12:01:01 subworker.1 | WARNING:__main__:Logged message!
^CSIGINT received
12:01:03 system   | sending SIGTERM to all processes
SIGTERM received
12:01:03 worker.1 | SIGINT received
12:01:03 worker.1 | 12:01:03 system      | sending SIGTERM to all processes
12:01:03 worker.1 | SIGTERM received
12:01:03 worker.1 | 12:01:03 subworker.1 | Stdout message
12:01:03 worker.1 | 12:01:03 subworker.1 | terminated by SIGTERM
12:01:03 worker.1 | 12:01:03 subworker.1 | Stdout message
12:01:03 worker.1 | 12:01:03             | Stdout message
12:01:03 worker.1 | 12:01:03             | Traceback (most recent call last):
12:01:03 worker.1 | 12:01:03             |   File "test.py", line 12, in <module>
12:01:03 worker.1 | 12:01:03             |     time.sleep(2)
12:01:03 worker.1 | 12:01:03             | KeyboardInterrupt
12:01:03 worker.1 | exited with code 0

Replacing the command TestSubProcfile with something that simply prints to the standard output, like yes, both a Foreman and Honcho inner process propagate output as expected, without buffering.

Docs for export options don't match `honcho help export`

https://honcho.readthedocs.org/en/latest/export.html has:

-s SHELL, --shell SHELL
    Specify the shell that should run the application.
    Only used for upstart.

-u USER, --user USER
    Specify the user the application should be run as.

-a APP, --app APP
    Specify the name of the application you are going to export.

-c process=num,process=num, --concurrency process=num,process=num
    Specify the number of each process type to run.

-p N, --port N
    Specify which port to use as the base for this application.
    Should be a multiple of 1000.

-l DIR, --log DIR
    Specify the directory to place process logs in.

OTOH, the built-in help says:

$ honcho help export
...
  -a APP, --app APP     alternative app name (default: honcho)
  -l DIR, --log DIR     directory to place process logs in (default:
                        /var/log/APP)
  -p N, --port N
  -c process=num,process=num, --concurrency process=num,process=num
                        number of each process type to run. (default: None)
  -u USER, --user USER  user the application should run as (default: None)
  -s SHELL, --shell SHELL
                        the shell that should run the application (default:
                        /bin/sh)

They are close, but there are some subtle differences.

readthedocs says of --shell: "Only used for upstart" (which isn't true). The help doesn't (more accurate).
The help doesn't have a description for the -p/--port option. readthedocs is better here (unless the "Should be a multiple of 1000" bit isn't true, which I didn't verify).
readthedocs version is more verbose. All option descriptions start with "Specify..."
listed in different order (this isn't so important, except that it makes it a little hard to check whether they match each other

port 12 new pull requests from foreman

I'm not even sure what all they are, but yesterday there were 21 open pull requests on foreman, and today there are 9. See also:

https://twitter.com/whit537/status/290646577352171520

Windows Support

Heya - A few of my co-workers are on Windows, having them being able to use Honcho too would be great!

The only blocker I can see is the use of SIGALARM in process.py.

Differences between `foreman export supervisord` and `honcho export supervisord`

Maybe some of the differences with Foreman are intentional; some are not?

Just going to catalog the differences for now.

[marca@marca-mac2 export_test]$ foreman export supervisord . -a myapp && mv myapp.conf foreman_myapp.conf
[foreman export] writing: myapp.conf
[marca@marca-mac2 export_test]$ honcho export supervisord . -a myapp && mv myapp.conf honcho_myapp.conf
[marca@marca-mac2 export_test]$ diff -u foreman_myapp.conf honcho_myapp.conf

diff --git a/foreman_myapp.conf b/honcho_myapp.conf
index cec5788..1733ae5 100644
--- a/foreman_myapp.conf
+++ b/honcho_myapp.conf
@@ -1,14 +1,13 @@
-
-[program:myapp-ansvc-1]
-command=bin/ansvc start
+[program:myapp-ansvc]
+command=/bin/sh -c 'bin/ansvc start'
 autostart=true
 autorestart=true
 stopsignal=QUIT
-stdout_logfile=/var/log/myapp/ansvc-1.log
-stderr_logfile=/var/log/myapp/ansvc-1.error.log
-user=myapp
+stdout_logfile=/var/log/myapp/ansvc-0.log
+stderr_logfile=/var/log/myapp/ansvc-0.error.log
+user=marca
 directory=/Users/marca/dev/git-repos/honcho/export_test
 environment=PORT="5000"

 [group:myapp]
-programs=myapp-ansvc-1
+programs=myapp-ansvc

How to launch the python smptd with honcho

Hello,

I've tried to use honcho to start the dummy python smptd server, however I'm not able to get it working.

the Procfile looks like:

smtp: python -m smtpd -n -c DebuggingServer localhost:1025

The system seems to start with no complain, but no mail gets printed on the console when sent. If I write the same line n the console, everything works properly.

Any idea why?

After honcho 0.6 port should be a multiple of 1000

I used honcho for a while and liked it ability to use custom base ports, so its no need to start base ports from multiple of 1000 as in foreman.

But after 0.6 release this changed and now I got something like:

PORT=8110 . venv/bin/activate && honcho start web -f ec2Procfile
. venv/bin/activate && honcho start worker
Traceback (most recent call last):
  File "/home/playpauseandstop/Projects/.../.../venv/bin/honcho", line 11, in <module>
    sys.exit(main())
  File "/home/playpauseandstop/Projects/.../.../venv/lib/python2.7/site-packages/honcho/command.py", line 269, in main
    COMMANDS[args.command](args)
  File "/home/playpauseandstop/Projects/.../.../venv/lib/python2.7/site-packages/honcho/command.py", line 212, in command_start
    port=port):
  File "/home/playpauseandstop/Projects/.../.../venv/lib/python2.7/site-packages/honcho/environ.py", line 114, in expand_processes
    assert port % 1000 == 0, "port must be multiple of 1000"
AssertionError: port must be multiple of 1000

when try to start honcho on 8110 port.

Do we really need mimicrify foreman here? Or can continue live with custom base ports?

Proper Handling of SIGINT/SIGQUIT

In researching #70, I ran across Proper Handling of SIGINT/SIGQUIT, which recommends that if a child execed by honcho run exits due to SIGINT, Honcho should kill(SIGINT) itself in order to properly terminate shell scripts that may have invoked Honcho.

It's a fascinating, if long-winded article, but the premise is that the child should determine how to handle SIGINT, and parents (while ignoring SIGINT themselves), should honor the child's decision if it decides to exit by propagating the SIGINT exit status back up the ancestor chain to the shell.

Unicode error in printer.py

This issue seems different from #54. I stumbled upon it investigating a downtime on my production site on Heroku; unfortunately have no idea what Honcho or waitress was trying to print. I'm using honcho 0.5.0.

2014-05-02 00:43:25.449719+00:00 heroku router - - at=error code=H13 desc="Connection closed without response" method=GET path=/de/catalogue/hinterrad-kotfluegel-carbon-r-1200-gs-lc_476/?gclid=CNrU942BjL4CFdQgtAod41AALg host=www.bikepartsmarket.com request_id=6a08f961-87db-4270-b1cf-3d2a31109ad4 fwd="141.6.11.20/NX, 108.162.254.212/NX" dyno=web.1 connect=1ms service=211ms status=503 bytes=0
2014-05-02 00:43:25.408825+00:00 app web.1     - - Traceback (most recent call last):
  File "/app/.heroku/python/bin/honcho", line 9, in <module>
    load_entry_point('honcho==0.5.0', 'console_scripts', 'honcho')()
  File "/app/.heroku/python/lib/python2.7/site-packages/honcho/command.py", line 315, in main
    app.parse()
  File "/app/.heroku/python/lib/python2.7/site-packages/honcho/command.py", line 129, in parse
    options.func(self, options)
  File "/app/.heroku/python/lib/python2.7/site-packages/honcho/command.py", line 197, in start
    sys.exit(process_manager.loop())
  File "/app/.heroku/python/lib/python2.7/site-packages/honcho/process.py", line 118, in loop
    print(line, end='', file=proc.printer)
  File "/app/.heroku/python/lib/python2.7/site-packages/honcho/printer.py", line 22, in write
    self.output.write(*new_args, **kwargs)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in position 95: ordinal not in range(128)

The output of honcho looks weird

For instance the weird control chars in this:

←[36m15:27:19 es.1   | ←[0mstarted with pid 2952
←[33m15:27:19 task.1 | ←[0mstarted with pid 5012
←[32m15:27:19 web.1  | ←[0mstarted with pid 4640

Is this to color things in the shell? if so that would not port well to non-posix...
Is this out of compatibility with Foreman?
Is this something that could be optionally disabled with a command flag?

Intermittent test failures on OS X

I don't know yet if it has anything to do with OS X or my local system or if it's a bug in honcho, but I'm periodically getting test failures on the master branch:

❯ tox -e py27
GLOB sdist-make: /Users/marca/dev/git-repos/honcho/setup.py
py27 inst-nodeps: /Users/marca/dev/git-repos/honcho/.tox/dist/honcho-0.5.0.zip
py27 runtests: PYTHONHASHSEED='3869475230'
py27 runtests: commands[0] | pip install -q -e .[export]
py27 runtests: commands[1] | nosetests
......FFF.............................................................................
======================================================================
FAIL: test_get_port_from_dot_env (honcho.test.integration.test_ports.TestPorts)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/marca/dev/git-repos/honcho/honcho/test/integration/test_ports.py", line 40, in test_get_port_from_dot_env
    self.assertRegexpMatches(out, r'web\.1    \| (....)?PORT=8000')
AssertionError: Regexp didn't match: 'web\\.1    \\| (....)?PORT=8000' not found in '14:53:43 system   | es.1 started (pid=87007)\n14:53:43 system   | worker.1 started (pid=87009)\n14:53:43 system   | redis.1 started (pid=87010)\n14:53:43 system   | web.1 started (pid=87011)\n\x1b[35m14:53:43 es.1     | \x1b[0mPORT=8300\n\x1b[33m14:53:43 worker.1 | \x1b[0mPORT=8100\n14:53:43 system   | es.1 stopped (rc=0)\n14:53:43 system   | sending SIGTERM to worker.1 (pid 87009)\n14:53:43 system   | sending SIGTERM to redis.1 (pid 87010)\n14:53:43 system   | sending SIGTERM to web.1 (pid 87011)\n14:53:43 system   | worker.1 stopped (rc=-15)\n14:53:43 system   | web.1 stopped (rc=-15)\n14:53:43 system   | redis.1 stopped (rc=-15)\n'

======================================================================
FAIL: test_get_port_from_env (honcho.test.integration.test_ports.TestPorts)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/marca/dev/git-repos/honcho/honcho/test/integration/test_ports.py", line 50, in test_get_port_from_env
    self.assertRegexpMatches(out, r'web\.1    \| (....)?PORT=3000')
AssertionError: Regexp didn't match: 'web\\.1    \\| (....)?PORT=3000' not found in '14:53:44 system   | es.1 started (pid=87017)\n14:53:44 system   | redis.1 started (pid=87020)\n14:53:44 system   | worker.1 started (pid=87019)\n\x1b[35m14:53:44 es.1     | \x1b[0mPORT=3300\n14:53:44 system   | web.1 started (pid=87021)\n14:53:44 system   | es.1 stopped (rc=0)\n14:53:44 system   | sending SIGTERM to worker.1 (pid 87019)\n14:53:44 system   | sending SIGTERM to redis.1 (pid 87020)\n14:53:44 system   | sending SIGTERM to web.1 (pid 87021)\n14:53:44 system   | web.1 stopped (rc=-15)\n14:53:44 system   | worker.1 stopped (rc=-15)\n14:53:44 system   | redis.1 stopped (rc=-15)\n'

======================================================================
FAIL: test_proctype_increment (honcho.test.integration.test_ports.TestPorts)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/marca/dev/git-repos/honcho/honcho/test/integration/test_ports.py", line 15, in test_proctype_increment
    self.assertRegexpMatches(out, r'web\.1    \| (....)?PORT=5000')
AssertionError: Regexp didn't match: 'web\\.1    \\| (....)?PORT=5000' not found in '14:53:44 system   | es.1 started (pid=87027)\n14:53:44 system   | worker.1 started (pid=87028)\n14:53:44 system   | redis.1 started (pid=87029)\n14:53:44 system   | web.1 started (pid=87030)\n\x1b[35m14:53:44 es.1     | \x1b[0mPORT=5300\n\x1b[33m14:53:44 worker.1 | \x1b[0mPORT=5100\n14:53:44 system   | es.1 stopped (rc=0)\n14:53:44 system   | sending SIGTERM to worker.1 (pid 87028)\n14:53:44 system   | sending SIGTERM to redis.1 (pid 87029)\n14:53:44 system   | sending SIGTERM to web.1 (pid 87030)\n14:53:44 system   | worker.1 stopped (rc=-15)\n14:53:44 system   | redis.1 stopped (rc=-15)\n14:53:44 system   | web.1 stopped (rc=-15)\n'

----------------------------------------------------------------------
Ran 86 tests in 6.038s

FAILED (failures=3)
ERROR: InvocationError: '/Users/marca/dev/git-repos/honcho/.tox/py27/bin/nosetests'
___________________________________________________________________________________ summary ____________________________________________________________________________________
ERROR:   py27: commands failed

It's intermittent though. Sometimes all of the tests pass and sometimes 1-3 of them fail.

Export upstart: generates no newline at end of files

I noticed while writing export integration tests (coming soon 😄) that the exported upstart files might be missing a newline at the end. I fixed the same issue for supervisord files in #96

Support Windows

@nickstenning : Hi Nick! I am trying to revive this: #6

You wrote then "If you can fix this (possibly by using signal and ignoring interrupts in the timer thread) I'll happily pull this in."
==> How would you do this?

Also alternatively and more simply on Windows, could we just kill the processes abruptly with no timer/no alarm?
That could be good enough to get only part of the code that cannot run on Windows to work i.e. here

honcho/honcho/process.py

Line 167 in 7f2cfa4

signal.signal(signal.SIGALRM, kill)

instead of this code:

        signal.signal(signal.SIGALRM, kill)
        signal.alarm(5)

we could use something along these lines:

        if ON_POSIX:
            signal.signal(signal.SIGALRM, kill)
            signal.alarm(5)
        else:
            # SIGALRM is not supported on Windows: just kill instead
            kill(None, None)

What do you think?
let me try and test it in the meantime... as there are likely several other areas in the code and tests where some posix assumptions are made

Exported supervisord conf fails with certain characters in Procfile

System:
http://aws.amazon.com/amazon-linux-ami/

Steps to reproduce the bug:
honcho export supervisord /etc/supervisord.d/app.conf

Use case (in Procfile):

scheduler: /home/user/project/virtual_env/bin/celery worker --beat --events --maxtasksperchild=1000 --hostname=scheduler.%h --app=webapp

The export command doesn't add an extra % character to escape existing % characters per the supervisor documentation:

environment

A list of key/value pairs in the form KEY="val",KEY2="val2" that will be placed in the supervisord process’ environment (and as a result in all of its child process’ environments). This option can include the value %(here)s, which expands to the directory in which the supervisord configuration file was found. Values containing non-alphanumeric characters should be quoted (e.g. KEY="val:123",KEY2="val,456"). Otherwise, quoting the values is optional but recommended. To escape percent characters, simply use two. (e.g. URI="/first%%20name") Note that subprocesses will inherit the environment variables of the shell used to start supervisord except for the ones overridden here and within the program’s environment option.
-- http://supervisord.org/configuration.html#supervisord-section-values

Please upload last code version to pip

I'm often get this kind of error in my local development. I also know that this already has been fixed in latest code on github. So, please upload this version to PIP.

File "/Users/rmuslimov/envs/fokker/lib/python2.7/site-packages/honcho/printer.py", line 22, in write
self.output.write(_new_args, *_kwargs)
UnicodeEncodeError: 'ascii' codec can't encode character u'\x87' in position 83: ordinal not in range(128)

Thanks!

Honcho fails on Heroku because, USER variable is not set

Tested version: 0.4.0

Steps to reproduce: run Honcho on Heroku, for example by putting "web: honcho -f ProcfileHoncho start" into your Procfile and "honcho==0.4.0" into your requirements.txt.

Actual behaviour: following error message is printed into heroku log:

2013-04-01T17:31:29+00:00 heroku[web.1]: State changed from crashed to starting
2013-04-01T17:31:48+00:00 heroku[web.1]: Starting process with command `honcho -f ProcfileHoncho start`
2013-04-01T17:31:49+00:00 app[web.1]:     load_entry_point('honcho==0.4.0', 'console_scripts', 'honcho')()
2013-04-01T17:31:49+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/pkg_resources.py", line 337, in load_entry_point
2013-04-01T17:31:49+00:00 app[web.1]:     return get_distribution(dist).load_entry_point(group, name)
2013-04-01T17:31:49+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/pkg_resources.py", line 2279, in load_entry_point
2013-04-01T17:31:49+00:00 app[web.1]:     return ep.load()
2013-04-01T17:31:49+00:00 app[web.1]: Traceback (most recent call last):
2013-04-01T17:31:49+00:00 app[web.1]:   File "/app/.heroku/python/bin/honcho", line 9, in <module>
2013-04-01T17:31:49+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/pkg_resources.py", line 1989, in load
2013-04-01T17:31:49+00:00 app[web.1]:     entry = __import__(self.module_name, globals(),globals(), ['__name__'])
2013-04-01T17:31:49+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/honcho/command.py", line 66, in <module>
2013-04-01T17:31:49+00:00 app[web.1]:     class Honcho(object):
2013-04-01T17:31:49+00:00 app[web.1]:     default=os.environ['USERNAME' if compat.ON_WINDOWS else 'USER'],
2013-04-01T17:31:49+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/honcho/command.py", line 205, in Honcho
2013-04-01T17:31:49+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/UserDict.py", line 23, in __getitem__
2013-04-01T17:31:49+00:00 app[web.1]:     raise KeyError(key)
2013-04-01T17:31:49+00:00 app[web.1]: KeyError: 'USER'
2013-04-01T17:31:50+00:00 heroku[web.1]: Process exited with status 1
2013-04-01T17:31:50+00:00 heroku[web.1]: State changed from starting to crashed

Expected behaviour: tasks defined in ProcfileHoncho are started correctly

Temporary workaround: define USER variable on heroku:

heroku config:set USER=heroku

Crash when printing to console with special characters.

I'm experiencing crashes when Honcho tries to print to console with special characters:

Traceback (most recent call last):
File "/app/.heroku/python/bin/honcho", line 9, in <module>
    load_entry_point('honcho==0.4.2', 'console_scripts', 'honcho')()
File "/app/.heroku/python/lib/python2.7/site-packages/honcho/command.py", line 292, in main
    app.parse()
File "/app/.heroku/python/lib/python2.7/site-packages/honcho/command.py", line 129, in parse
    options.func(self, options)
File "/app/.heroku/python/lib/python2.7/site-packages/honcho/command.py", line 190, in start
    sys.exit(process_manager.loop())
File "/app/.heroku/python/lib/python2.7/site-packages/honcho/process.py", line 114, in loop
File "/app/.heroku/python/lib/python2.7/site-packages/honcho/printer.py", line 22, in write

Here is a Stackoverflow question with more details:

http://stackoverflow.com/questions/19100116/looks-like-logs-are-crashing-my-django-app

Problems with certain process names

I used "less-watcher" as a process name and that process didn't start.

After I changed the process name to "lesswatcher" everything worked well. So this happens when there's a dash on process name.

Port number allocation differs from Foreman

Foreman ports:

web.1    | 5000
worker.1 | 5100
redis.1  | 5200
es.1     | 5300

Honcho ports:

web.1    | 5000
redis.1  | 6001
worker.1 | 7002
es.1     | 8003

So there are three bugs to fix:

Each process type should get only 100 ports, not 1000.
The counter for allocating next port should be process-type-specific instead of global.
Processes should be allocated port ranges in the order they appear in the Procfile.

This bug matters because, without stable port numbers, it's hard to configure process types to talk to each other.

Supervisor export should expand command with environ variables

Exporting to upstart or doing run will work if you have a $PORT in your procfile expression but supervisord does not expand the command for you and thus will leave expressions like $PORT alone.

honcho is not listed as python3 compatible

https://caniusepython3.com/project/honcho says honcho is not python3 compatible