reagentx / logria-py Goto Github PK

A powerful CLI tool that puts log analytics at your fingertips. Python implementation.

License: GNU General Public License v3.0

Python 99.48% Shell 0.52%

cli logging analytics tail python multiprocessing parsing regex grep

logria-py's Introduction

Logria Python Proof of Concept

A powerful CLI tool that puts log analytics at your fingertips. This is the Python Proof of Concept repository; the main version is written in Rust.

tl;dr

Live filtering of data from other processes, replacing grep
Use shell commands or files as input, save sessions and come back later
Replace regex/filter without killing the process or losing the stream's history
Parse logs using user-defined rules, apply analytics methods on top
Pure Python, fully type hinted, zero dependencies

Installation

There are several options to install this app.

Normal Usage

pip install logria

Development

See Advanced Installation.

Usage

There are a few main ways to invoke Logria:

Directly:
- logria
- Opens to the setup screen
With args:
- logria -e 'tail -f log.txt'
- Opens a pipe to tail -f log.txt and skips setup
- logria -h will show the help page with all possible options

For more details, see Sample Usage Session.

Key Commands

Key	Command
`:`	command mode
`/`	regex search
`h`	if regex active, toggle highlighting of matches
`i`	toggle insert mode (default off)
`s`	swap reading `stdin` and `stdout`
`p`	activate parser
`a`	toggle analytics mode when parser is active
`z`	deactivate parser
↑	scroll buffer up one line
↓	scroll buffer down one line
→	skip and stick to end of buffer
←	skip and stick to beginning of buffer

Features

Here are some of the ways you can leverage Logria:

Live stream of log data

Interactive, live, editable grep

Live log message parsing

Live analytics/statistics tracking

User-defined saved sessions

See session docs.

User-defined saved log parsing methods

See patterns docs.

logria-py's People

Contributors

Stargazers

Watchers

Forkers

skymoore sumonst21

logria-py's Issues

Write history tape to file to preserve history across runs

Crash when given regex is not valid

https://github.com/ReagentX/Logria/blob/2d77aa871e4f9e20e4814dd2e6e45e0a01344b5e/logria/utilities/regex_generator.py#L11

Add command to send user back to start screen

Cannot quit app after creating a config

invoke with logria
:config command
create any new parser
:q does not quit the app

`[39 not in colors` crash when calling `addstr`

Some log streams use the escape code [39 instead of [0 to reset the background color to default.

Explore running the app in an asyncio event loop

This will allow us to theoretically keep rendering output while the user is entering commands instead of context switching between the two.

100% CPU when waiting for textbox output

Since textbox results are poll-based, we need to limit the amount of time we wait when we poll for new data the same way we read from the queues now: https://github.com/ReagentX/Logria/blob/develop/logria/communication/shell_output.py#L741

View/search tape history and allow scroll back to any arbitrary position

Improve unit test coverage

Name                                    Stmts   Miss  Cover   Missing
---------------------------------------------------------------------
logria/__init__.py                          2      0   100%
logria/commands/__init__.py                 0      0   100%
logria/commands/command.py                 37     31    16%   18-24, 32-63
logria/commands/config.py                 176    142    19%   51-82, 89-114, 122-148, 155-224, 231-244
logria/commands/edit.py                     5      4    20%   13-17
logria/commands/parser.py                 116    106     9%   20-42, 50-146, 153-155, 162-171, 178
logria/commands/regex.py                   52     44    15%   19-31, 38-60, 68-83, 90-96
logria/commands/scroll.py                  33      2    94%   31, 46
logria/commands/window.py                  11      7    36%   15-22, 29
logria/communication/__init__.py            0      0   100%
logria/communication/input_handler.py      60     35    42%   36, 42, 69-99, 106-110, 123-134
logria/communication/input_history.py      79      4    95%   33, 39, 48-49
logria/communication/render.py             35      0   100%
logria/communication/setup.py              74     63    15%   26-33, 41-118
logria/communication/shell_output.py      192    117    39%   104-122, 134-171, 177-181, 187-189, 197-201, 207-212, 218-236, 242-243, 249-252, 267, 271, 278-356
logria/interface/__init__.py                0      0   100%
logria/interface/color_handler.py          89     54    39%   68-75, 79-87, 101-124, 128-139, 152-166
logria/interface/textbox.py               159    144     9%   18-25, 54-62, 68-70, 77-87, 93-115, 121-198, 204-220, 226, 235-246
logria/logger/__init__.py                   0      0   100%
logria/logger/default_logger.py            10      0   100%
logria/logger/parser.py                   154      2    99%   42, 104
logria/logger/processor.py                 20      0   100%
logria/utilities/__init__.py                0      0   100%
logria/utilities/command_parser.py         51      0   100%
logria/utilities/constants.py              30      0   100%
logria/utilities/fs.py                      7      0   100%
logria/utilities/keystrokes.py             10      0   100%
logria/utilities/regex_generator.py        11      0   100%
logria/utilities/session.py                65      2    97%   53, 57
tests/__init__.py                           0      0   100%
tests/test_app_launch.py                   23      0   100%
tests/test_color_handler.py                12      0   100%
tests/test_command_parser.py               56      0   100%
tests/test_constants.py                    11      0   100%
tests/test_fs.py                           17      0   100%
tests/test_history_tape.py                134      0   100%
tests/test_input_handler.py                20      0   100%
tests/test_keystrokes.py                    8      0   100%
tests/test_parser.py                      194      0   100%
tests/test_processors.py                  194      0   100%
tests/test_regex_generator.py              19      0   100%
tests/test_render_position.py             371      0   100%
tests/test_resolve_delete.py               33      0   100%
tests/test_session.py                      82      0   100%
---------------------------------------------------------------------
TOTAL                                    2652    757    71%

Smart poll rate

We can automatically reduce the poll rate while we are just listening and increase it when asking for input to reduce CPU impact.

Quitting app does not end streaming processes

Support pgup/pgdn to scroll faster

Crash when a PATH folder has a ~ in its env var

#If a folder on your PATH is defined with a ~ Logria will crash because it interprets the tilde as a folder name and not the user’s home directory. This can be rectified by calling resolve_home_dir as we iterate.

App shows nothing if stderr is empty, must manually switch to stdout

We should check which queue starts with data and make that the first one we look at instead of always defaulting to stderr.

Input duplicated when using textbox when running Django's `runserver`

Logria installs `sessions` and `patterns` as separate packages, not as separate folders

We should instead write these folders to the users' home directory on launch

Convert to Rust

We need to go deeper!

Ignore .DS_Store in MacOS directory listings

Session creation from single file does not work, missing array level

['path', 'to', 'file'] instead of [['path', 'to', 'file']]

Startup screen list cannot scroll because textbox is active

We should make it scroll by default and accept keystrokes like normal; if a single digit gets typed we should immediately load it, but this only works with single digit items.

Command :o # to open any arbitrary #.

Changing the poll rate resets command line status

Store history "tape" of searches

Crash with -n cli arg

Traceback (most recent call last):
  File "/Users/chris/.pyenv/versions/3.8.4/bin/logria", line 8, in <module>
    sys.exit(main())
  File "/Users/chris/.pyenv/versions/3.8.4/lib/python3.8/site-packages/logria/__main__.py", line 46, in main
    app = Logria(stream, history_tape_cache=args.no_cache, )
AttributeError: 'list' object has no attribute 'no_cache'

Not compatible with Powershell/Windows CMD

PS C:\Users\Development\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\Scripts> .\logria.exe                                                                                       Namespace(e=None)
Traceback (most recent call last):
  File "C:\Users\Development\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\Scripts\logria-script.py", line 11, in <module>
    load_entry_point('logria', 'console_scripts', 'logria')()
  File "c:\users\development\documents\code\logria\logria\__main__.py", line 31, in main
    app.start()
  File "c:\users\development\documents\code\logria\logria\communication\shell_output.py", line 246, in start
    curses.wrapper(self.main)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.496.0_x64__qbz5n2kfra8p0\lib\curses\__init__.py", line 105, in wrapper
    return func(stdscr, *args, **kwds)
  File "c:\users\development\documents\code\logria\logria\communication\shell_output.py", line 267, in main
    self.command_line = curses.newwin(1, width, height - 2, 1)
_curses.error: curses function returned NULL

Escape charachter \ printed when we highlight matches

We need to sanitize the pattern or do something different here:

https://github.com/ReagentX/Logria/blob/54c3165556c5e663fe1b9c53ff2c0d9f660bb20f/logria/communication/shell_output.py#L159

Add tests for curses app somehow

Curses requires a terminal to attach to, but running tests does not attach a terminal, so it crashes when launched that way.

Allow user to override files with environment variables

Constants like folders etc are all handled with constants.py, but users may want to override these defaults.

Give shell_output module some love

A lot of this module can be refactored so that we dont need to keep track of so much state in a single class.

:config crash

  File "/Users/h14384/.pyenv/versions/3.8.6/lib/python3.8/site-packages/logria/commands/config.py", line 233, in config_mode
    logria.refresh()
AttributeError: 'Logria' object has no attribute 'refresh'

Exiting Logria does not kill piped processes

It only kills the connection to them. We need to kill them on exit.

Rendering Issue when doing screen split

To reproduce --

Open logria in a new terminal window
(In my case, I use iterm2) Split screen horizontally

Have fun 👃

Traceback (most recent call last):
  File "/Users/chris/.pyenv/versions/3.8.3/bin/logria", line 8, in <module>
    sys.exit(main())
  File "/Users/chris/.pyenv/versions/3.8.3/lib/python3.8/site-packages/logria/__main__.py", line 47, in main
    app.start()
  File "/Users/chris/.pyenv/versions/3.8.3/lib/python3.8/site-packages/logria/communication/shell_output.py", line 248, in start
    curses.wrapper(self.main)
  File "/Users/chris/.pyenv/versions/3.8.3/lib/python3.8/curses/__init__.py", line 105, in wrapper
    return func(stdscr, *args, **kwds)
  File "/Users/chris/.pyenv/versions/3.8.3/lib/python3.8/site-packages/logria/communication/shell_output.py", line 344, in main
    self.render_text_in_output()
  File "/Users/chris/.pyenv/versions/3.8.3/lib/python3.8/site-packages/logria/communication/shell_output.py", line 165, in render_text_in_output
    color_handler.addstr(self.outwin, current_row, 0, item.rstrip())
  File "/Users/chris/.pyenv/versions/3.8.3/lib/python3.8/site-packages/logria/interface/color_handler.py", line 152, in addstr
    return _inner_addstr(window, string, y_coord, x_coord)
  File "/Users/chris/.pyenv/versions/3.8.3/lib/python3.8/site-packages/logria/interface/color_handler.py", line 123, in _inner_addstr
    _add_line(y_coord, x_coord, window, line)
  File "/Users/chris/.pyenv/versions/3.8.3/lib/python3.8/site-packages/logria/interface/color_handler.py", line 94, in _add_line
    window.addstr(y_coord, x_coord, color_split[0], curses.color_pair(default_color_pair))
ValueError: embedded null character