thesharp / daemonize Goto Github PK

Its will be very useful when program use SocketServer. For example within this action we can make server shutdown and unlink socket file(if server uses AF_UNIX sockets)

If I have two modules - first db.py like this

import threading
import time

class Database(threading.Thread):

    def __init__(self):
        super(Database, self).__init__()
        self.daemon = True


    def run(self):
        while True:
            time.sleep(2)


db = Database()
db.start()

And second main.py

#!/usr/bin/env python
from time import sleep
import logging

from daemonize import Daemonize

import db


pid = "/tmp/test.pid"
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
logger.propagate = False
fh = logging.FileHandler("/tmp/test.log", "w")
fh.setLevel(logging.DEBUG)
logger.addHandler(fh)
keep_fds = [fh.stream.fileno()]

def main():
    while True:
         sleep(0.5)

if __name__ == '__main__':
    daemon = Daemonize(app="test_app", pid=pid, action=main, keep_fds=keep_fds)
    daemon.start()
    logger.debug(db.db) # <Database(Thread-1, stopped daemon 140401702676224)>

If I run main.py so I have a stopped daemon that I've started previously in db.py. Do you have any idea why it doesn't work with daemonize? If I don't use daemonize it works fine.

First, thanks for writing this library, is very simple to use.

I think that would be useful to have a callback like the start one to be executed when SIGTERM signal is received.

daemon = Daemonize(app='my_daemon',
                           user='user', group='group',
                           pid='/tmp/daemon.pid', foreground=Fase,
                           action=start, shutdown_cb=stop, auto_close_fds=False)

from src import main
from functools import partial

for i in range(6):
	pid = f"/tmp/app_{i}.pid"
	Daemonize(app=f"app_{i}", pid=pid, action=partial(main, some_arg=True), auto_close_fds=False).start()

Could I make something like this work somehow? As of now It will exit after the first iteration.

I have this code:

daemon = Daemonize(
    app="Encoder",
    pid=pid_file,
    action=self.__daemonize,
    foreground=False if self.env == 'production' else True,
    logger=logging
)
daemon.start()

And in the exit handler for the process (when someone either hits ctrl-c or run the stop process):

with open(pid_file, 'r') as content_file:
    pid = content_file.read()
os.kill(int(pid), signal.SIGTERM)

That works ok, but when it exists (although it executes everything ok) I see this error:

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/daemonize.py", line 71, in exit
    os.remove(self.pid)
FileNotFoundError: [Errno 2] No such file or directory: '/var/run/encoder.pid'

Reading the code, I see this:

        signal.signal(signal.SIGTERM, self.sigterm)
        atexit.register(self.exit)

And that has this:

    def sigterm(self, signum, frame):
        """
        These actions will be done after SIGTERM.
        """
        self.logger.warn("Caught signal %s. Stopping daemon." % signum)
        os.remove(self.pid)
        sys.exit(0)

    def exit(self):
        """
        Cleanup pid file at exit.
        """
        self.logger.warn("Stopping daemon.")
        os.remove(self.pid)
        sys.exit(0)

If I understand the code correctly, that will try to delete the pid file twice, is that right? Am I doing something wrong?

It appears everything works under it, so...

The following code fails on my Mac OS X in daemon but not in foreground mode:

import requests

def main():
    while True:
        requests.get('https://httpbin.org/get')
        sleep(10)

daemon = Daemonize(app="demo_app", pid="/tmp/demo_app.pid", action=main)
daemon.start()

I'm using Python 3.8.5, daemonize 2.5.0 and requests 2.24.0

Hi,

the pypi page looks strange. Maybe the markup has changed?

https://pypi.python.org/pypi/daemonize

When I use daemonize, but no logger was initialized the following erorr is retuned:

  File "/home/lcubo/.pyenv/versions/project/local/lib/python2.7/site-packages/daemonize.py", line 70, in exit
    self.logger.warn("Stopping daemon.")
AttributeError: 'NoneType' object has no attribute 'warn'

The code to reproduce the issue:

daemon = Daemonize('faraday-server', PID, main)       
daemon.exit()

When a script tries to daemonize to perform a long running action, it should have the chance to handle errors via exceptions prior to the forking of the child.
Currently if there is already a daemon running so the parent can't get a lock on the pid file Daemonize.start() just exits. As an option at least it should just raise an exception which can be more gracefully handled in the main script.

start-stop-daemon --stop --pidfile /tmp/test.pid would result in getting the daemon to stop. Was wondering if it's possible to trigger a function i.e. handle the signal when stop is called ?. Thus one can gracefully exit by closing connections to other services etc.

If a script is launched as root and it calls Daemonize(... user='notroot'... ) to execute the daemon with non-root permissions, then .pid file will be created by as owned by root since that happens early in setup.

But when the daemon exits, an exception occurs because the nonroot user attempts to delete the root-owned file.

I believe correct behavior would be to change ownership of the .pid file to the daemon user prior to the daemon giving up its root privileges.

Stack trace

KeyboardInterrupt
Stopping daemon.
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/local/lib/python2.7/site-packages/daemonize.py", line 71, in exit
    os.remove(self.pid)
OSError: [Errno 13] Permission denied: '/var/run/test.pid'

Readed lots of posts, all mention we should fork twice for daemonize a py process, why this lib dont?

@penpen

Tests fail.

Further investigation shows that a simple application like tests/daemon_sigterm.py will just silently fail.

Let's confirm that our thingie works on NetBSD.

Currently, at daemonize.py#L137:

for fd in range(3, resource.getrlimit(resource.RLIMIT_NOFILE)[0]):

It's not very wise to use RLIMIT_NOFILE for this purpose. In real-world production servers this value may be as high as few tens of millions. In which case, daemonization will spam the system with useless syscalls and be delayed up to tens of seconds because of that.

It's not reasonable to attempt to close more that few thousands of descriptors. After all, daemonization, almost exclusively takes place immediately after the start of application. There's no way that a normal app is going to have this many open descriptors during that time.

Traceback (most recent call last):
File "crm.py", line 218, in
def main():
File "/home/heweiwei/.pyenv/versions/3.8.3/lib/python3.8/site-packages/click/decorators.py", line 130, in decorator
cmd = _make_command(f, name, attrs, cls)
File "/home/heweiwei/.pyenv/versions/3.8.3/lib/python3.8/site-packages/click/decorators.py", line 98, in _make_command
return cls(
File "/home/heweiwei/.pyenv/versions/3.8.3/lib/python3.8/site-packages/daemonocle/cli.py", line 32, in init
context_settings = {'obj': self.daemon_class(**self.daemon_params)}
File "/home/heweiwei/.pyenv/versions/3.8.3/lib/python3.8/site-packages/daemonocle/core.py", line 37, in init
self.detach = detach & self._is_detach_necessary()
File "/home/heweiwei/.pyenv/versions/3.8.3/lib/python3.8/site-packages/daemonocle/core.py", line 259, in _is_detach_necessary
if cls._is_socket(sys.stdin):
File "/home/heweiwei/.pyenv/versions/3.8.3/lib/python3.8/site-packages/daemonocle/core.py", line 224, in _is_socket
sock = socket.fromfd(fd, socket.AF_INET, socket.SOCK_RAW)
File "/home/heweiwei/.pyenv/versions/3.8.3/lib/python3.8/socket.py", line 544, in fromfd
return socket(family, type, proto, nfd)
File "/home/heweiwei/.pyenv/versions/3.8.3/lib/python3.8/socket.py", line 231, in init
_socket.socket.init(self, family, type, proto, fileno)
OSError: [Errno 88] Socket operation on non-socket

See attached patch that uses python.executable instead of hardcoding interpreter calls thus making the testsuite always run the interpreter the tests are executed with instead of always /usr/bin/python.

python-exec.txt

In glorificarion of 2.5 milestone.

I'm asking before I go down this road...it looks like there are open issues for a long time and no recent commits. If this project is indeed abandoned, please recommend an alternative.

Which will explain all the features.

This is as of commit 9018be

By default, daemonize inits a logger to syslog, catches any exceptions and writes them to that logger. That's useful in daemon mode but it also does so when in foreground mode - with the odd behaviour that anything the application prints normally ends up on stdout/stderr but any exception is only printed to syslog. This makes debugging harder and is quite confusing to those that expect an exception.

IMO the behaviour for the logger should either be if self.logger is None and not self.foreground or the propagation should be to stdout/stderr as well in foreground mode.

It would be great to be able to pass arguments to Daemonize() - actually this prevents me from using it in a project.

def main(*args, **kwargs):
    while 1:
        doSomething(*args, **kwargs)

daemon = Daemonize(
    action='main', args=('foo','bar',), kwargs={'foo':'bar'}
)
daemon.start()

I wonder how we can pass parameter to the function we want to daemonize.

This is my test code for daemonize lib
http://pastebin.com/JwrbwkSP
The code start the daemon successfully but fails to stop the deamon.
The logger seems to be not initialized ?

I am getting error while trying to stop the daemon:
Traceback (most recent call last):
File "dtest.py", line 27, in
daemon.exit()
File "/usr/local/lib/python2.7/dist-packages/daemonize.py", line 70, in exit
self.logger.warn("Stopping daemon.")
AttributeError: 'NoneType' object has no attribute 'warn'

We should use '/' (as we do now), if it won't be provided.

It would be nice to change documentation to sphinx format, including write usage documentation in module header and describe forgotten arguments.

I know that the app keyword is for syslog, but couldn't it also set the process? otherwise the demon has the same name as the (dead) parent which is confusing.

P.S: Pretty clean package, good work

atexit registers a callback that can't be exectuted. the atexit call refers to your sigterm handler which requires 2 parameters. atexit callbacks require 0.

So Daemonize uses sys.exit in the parent process after the fork. The intention is likely that this will cause the parent process to immediately end.

However, this is not actually the case:

• The parent python process raises SystemExit instead, which returns control flow outside of the Daemonize call if an exception handler for SystemExit is present. This means that code can continue to execute in the parent process after the Fork. Strange, but something a programmer can avoid with careful coding.
• However, if the parent process has passed file descriptors of sockets/pipes/etc. to keep_fds that it intends for the client to use, when the parent process exits, the parent process will garbage collect many of these client objects and these garbage collection routines will then attempt to do clean-ups. These clean-ups may involve sending shutdown notices over the sockets/pipes. This corrupts the child process. We noticed this when we changed a client library to properly implement garbage collection.

I suggest that, instead of callingsys.exit(0), that Daemonize reset the signal handlers to the system default handlers (away from the python default handlers that raise exceptions) and then call os.kill on itself with SIGTERM (or SIGKILL). This would prevent Python garbage collection routines in the parent from having the ability to interfere with the child process.

Currently daemonize will gladly accept a relative path to a pid file, then chdir to "/" and then throw an exception when it can't remove the now invalid pid file path.

A simple patch to make this work is to save the absolute path of the pid file on instantiation.

--- "..\\envs\\mn\\Lib\\site-packages\\daemonize.py"
+++ "plusdashfilter\\daemonize.py"
@@ -44,6 +44,7 @@
                  user=None, group=None, verbose=False, logger=None,
                  foreground=False, chdir="/"):
         self.app = app
+        self._pid = None
         self.pid = pid
         self.action = action
         self.keep_fds = keep_fds or []
@@ -56,6 +57,14 @@
         self.foreground = foreground
         self.chdir = chdir

+    @property
+    def pid(self):
+        return self._pid
+
+    @pid.setter
+    def pid(self, pid):
+        self._pid = os.path.abspath(pid)
+
     def sigterm(self, signum, frame):
         """
         These actions will be done after SIGTERM.

Shouldn't you be able to stop/restart the process?

Hi, when killing an application using daemonize with e.g. pkill this following warning(?) appears since Python 3.10 I believe

Exception ignored in atexit callback: <bound method Daemonize.exit of <daemonize.Daemonize object at 0x7fa00ab1bb80>>
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/daemonize.py", line 72, in exit
    sys.exit(0)
SystemExit: 0

Is there something I can do in the application to remove this warning or something?

daemon = Daemonize(app="test_app", pid=pid, action=main, keep_fds=keep_fds)
daemon.start()
daemon.exit()  # this line is never reached because the parent process does a sys.exit(0) after the fork

daemon = Daemonize(app="test_app", pid=pid, action=main, keep_fds=keep_fds)
daemon.exit()  # raises AttributeError when self.logger is None

Since Daemon.exit() also does a sys.exit(0) any subsequent lines of code are never reached either. Shouldn't the sys.exit() calls for the original parent process be left to the user?

Shouldn't logging be initialized in init()?

I'm running into a really specific and weird bug on OSX.

Reproduction instructions

Use OSX. I am using 10.15.4 right now; I don't know if the specific version matters.

Install daemonize 2.5.0.

Run this script:

#!/usr/bin/env python3
import socket
from daemonize import Daemonize

def run():
    print(socket.gethostbyname('google.com'))


if __name__ == '__main__':
    print(socket.gethostbyname('google.com'))
    daemon = Daemonize(app="monitor_script", pid="./test.pid", action=run, foreground=False)
    daemon.start()

Symptoms of bug

Before forking, the Python program can successfully do a DNS lookup.

The following message appears in Console.app under Log Reports > system.log:

Jul 31 14:22:24 Nicks-MacBook-Pro monitor_script: Starting daemon.
Jul 31 14:22:24 Nicks-MacBook-Pro monitor_script: Traceback (most recent call last):
Jul 31 14:22:24 Nicks-MacBook-Pro monitor_script:   File "/Users/nick/.pyenv/versions/3.6.10/lib/python3.6/site-packages/daemonize.py", line 248, in start
Jul 31 14:22:24 Nicks-MacBook-Pro monitor_script:     self.action(*privileged_action_result)
Jul 31 14:22:24 Nicks-MacBook-Pro monitor_script:   File "test.py", line 7, in run
Jul 31 14:22:24 Nicks-MacBook-Pro monitor_script:     print(socket.gethostbyname('google.com'))
Jul 31 14:22:24 Nicks-MacBook-Pro monitor_script: socket.gaierror: [Errno 8] nodename nor servname provided, or not known
Jul 31 14:22:24 Nicks-MacBook-Pro monitor_script: Stopping daemon.

Variations tried

I have run the reproduction script on Linux, and it can successfully resolve DNS before and after forking. (This is why I marked this as an OSX-specific bug.)

I have run the reproduction script without the first DNS lookup. This works.

I have run the reproduction script with foreground=True. This works.

I've tried putting a sleep between the two DNS lookups. This doesn't help.

Workaround

Don't do DNS lookups before forking.

If attempting to open the lockfile fails (eg for permissions), the start method bails without reporting anything. (I suspect there are several possible failure points where this is the case. More system logging would be helpful.

Hi,
It may be the lack of documentation but use of privileged_actions is unclear to me. I tried using daemonize and as of today, there is no support to change current working directory. It defaults to '/'
However, I do have a work directory I need to use so I tried to use the privileged_actions and it works fine but then, you expect the privileged_actions to return something and put it as an argument to the action itself.

My action does not expect any arguments but yet and it fails to kick it off. If I remove the *privileged_actions in my arguments, then it works perfectly.

The method start() does not initialize the logging object properly due to its use of the os.path.isfile() method which returns false on sockets. Linux systems use a socket for /dev/log so this prevents daemonize from correctly choosing handlers.SysLogHandler for the logging object. Offending line below.

So, I've tried a few times to get this to work to no avail. Does this work?? Here's the script I'm running ... pretty much based off the example provided. (contents of spade/__main__.py)

System Details

• MacOS (10.14.6 (18G95))
• Python (3.7.4)
• Using pipenv and venv (shouldn't matter)

import logging
from sys import argv
from daemonize import Daemonize

PID = "/tmp/spade.pid"
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
logger.propagate = False
fh = logging.FileHandler("/tmp/test.log", "w")
fh.setLevel(logging.DEBUG)
logger.addHandler(fh)
keep_fds = [fh.stream.fileno()]


def main():
    logger.debug("daemonized")


if __name__ == "__main__":
    daemon = Daemonize(app="spade", pid=PID, action=main, keep_fds=keep_fds)
    if argv[1] == "start":
        daemon.start()
    if argv[1] == "stop":
        daemon.exit()

I run this like so: python -u spade/ start (to stop run the same but replace start with stop)

The script runs without error but does nothing. Can't find it in process list. No pid file is created. The log file is created but contains nothing. Start & stop seem to do nothing as well. I've scanned my filesystem for results ... nothing. Perhaps I'm missing something but it seems like daemonizing Python scripts is a lost art because all the libraries I've tried just don't seem to work.

The log.warn() function is deprecated formally in 3.3 (and was tacitly deprecated prior to that by simply not being documented - see http://bugs.python.org/issue13235). It would be great if Daemonize used log.warning() instead of log.warn() - I only saw calls to self.log.warn() in 3 places - sigterm(), exit(), and almost at the end of start().

Thanks to daemonize, I could daemonize my process simply.

However, I found a tiny problem with it.

Even If the pid file cannot be locked, the process looks like starting normally.
(But child process exits with an error. and the logger logs the error. it mentioned at #6(closed))

I think pid file locking must be done before fork().
Because, the original process should end with status non zero.
If the original process exits with non zero state, the rc-script cannot report anything.

I wrote a sample below.
I tested it under centos 6.3 with /etc/init.d/function. (It should return Fail when the pid file was already locked.)

# . /etc/init.d/functions; daemon foo.py -i /tmp/pidfile

# diff /usr/lib/python2.6/site-packages/daemonize-2.2.3-py2.6.egg/daemonize.py /usr/lib/python2.6/site-packages/daemonize-2.2.3-py2.6.egg/daemonize.py.bk 
62,79d61
<         try:
<             # Create a lockfile so that only one instance of this daemon is running at any time.
<             lockfile = open(self.pid, "w")
<         except IOError:
<             print "Unable to create a pidfile."
<             sys.exit(1)
<         try:
<             # Try to get an exclusive lock on the file. This will fail if another process has the file
<             # locked.
<             fcntl.flock(lockfile, fcntl.LOCK_EX | fcntl.LOCK_NB)
<             # Record the process id to the lockfile. This is standard practice for daemons.
<             lockfile.write("%s" % (os.getpid()))
<             lockfile.flush()
<         except IOError:
<             print "Unable to lock on the pidfile."
<             # os.remove(self.pid)
<             sys.exit(1)
< 
106,108d87
<         self.keep_fds.append(lockfile.fileno())
<         print self.keep_fds
<         print resource.getrlimit(resource.RLIMIT_NOFILE)[0]
175a155,172
>         try:
>             # Create a lockfile so that only one instance of this daemon is running at any time.
>             lockfile = open(self.pid, "w")
>         except IOError:
>             self.logger.error("Unable to create a pidfile.")
>             sys.exit(1)
>         try:
>             # Try to get an exclusive lock on the file. This will fail if another process has the file
>             # locked.
>             fcntl.lockf(lockfile, fcntl.LOCK_EX | fcntl.LOCK_NB)
>             # Record the process id to the lockfile. This is standard practice for daemons.
>             lockfile.write("%s" % (os.getpid()))
>             lockfile.flush()
>         except IOError:
>             self.logger.error("Unable to lock on the pidfile.")
>             os.remove(self.pid)
>             sys.exit(1)
> 

Common usage example: open socket at port 80, drop privileges to www-data user and group and continue serving.