derpston / python-multitail2

Python module to follow new lines from multiple files at once, taking account of new files, deleted files, and rotated files.

python-multitail2

Python module to follow new lines from multiple files at once, taking account of new files, deleted files, and rotated files.

Suitable for following sets of logs where you don't necessarily know how many files there are, what their names are, and don't want to have to keep track of changes yourself.

Doesn't (yet) support inotify for detecting file change events because of portability concerns and the lack of an inotify module in the Python standard library.

Another one?

Yeah, another one. Other multitail implementations I found didn't support automatically following new files.

Features

• Accepts globbing syntax, like /var/log/*.log
• Opens new files
• Closes deleted files
• Reopens rotated files
• Non-blocking
• Supports iteration

Examples

Emits ((path, offset), line) tuples representing the path to the file that each line comes from, along with a byte offset where the line begins.

Generator

>>> import multitail2
>>> mt = multitail2.MultiTail("/home/user/test/*")
>>> for line in mt:
...  print line
... 
(('/home/user/test/foo', 0), 'bar')
(('/home/user/test/foo', 4), 'bar')

Non-blocking polling

>>> import multitail2
>>> mt = multitail2.MultiTail("/home/user/test/*")
>>> list(mt.poll())
[(('/home/user/test/foo', 0), 'bar')]
>>> list(mt.poll())
[]

TODO

• Use inotify, where available. Should not be a hard dependency.

Bugs

• Does not handle truncated files - new content will be ignored until the file size exceeds what it was previously.

Contributing

Contributions welcome!

python-multitail2's Issues

windows support for non-blocking tailing

Thanks for the library, it's great. Just a small one for windows users getting the following, this can be fixed by removing the non-block flag in the os.open call. Not sure if there is a better way but this appears to fix the problem for me, and works for tailing multiple files

Patch:

import sys

is_windows = sys.platform == 'win32'
fh = os.fdopen(os.open(path, os.O_RDONLY | (0 if is_windows else os.O_NONBLOCK)))

Error:

Traceback (most recent call last):
File "userfile.py", line 6, in
mt = multitail2.MultiTail(glob_str)
File "C:\Python310\lib\site-packages\multitail2.py", line 152, in init
self._rescan(skip_to_end = skip_to_end)
File "C:\Python310\lib\site-packages\multitail2.py", line 177, in _rescan
self._tailedfiles[path] = TailedFile(path, skip_to_end = skip_to_end, offset = self._offsets.get(path, None))
File "C:\Python310\lib\site-packages\multitail2.py", line 16, in init
self._open(path, skip_to_end, offset)
File "C:\Python310\lib\site-packages\multitail2.py", line 30, in _open
fh = os.fdopen(os.open(path, os.O_RDONLY | os.O_NONBLOCK))
AttributeError: module 'os' has no attribute 'O_NONBLOCK'

skipping long lines

There appears to be a bug in handling skipping long lines. I think there should be an else condition before returning in ValueError. That is, long lines are checking in "if not at_eof" and then there should be an "else: return" if we are at_eof. Currently any long line will stop all further evaluation (e.g. if using poll()).

Separately, it would be great to be able to handle long lines in general. For example, this comes up in FIX logs when receiving security definitions. Perhaps there could be an option that emulates fh.readline()?

License?

Could you add a license for this project (or declare it freeware somewhere if you don't intend on having a traditional license)?