agusmakmun / django-log-viewer Goto Github PK

I have been trying to make this work with various logs but could really do with some help

['^)*.(-)S(-)}3{d,}2{d:}2{d:}2{d\s}2{d-}2{d-}4{d($','^).(-)*S(-)}3{d,}2{d:}2{d:}2{d\s}2{d-}2{d-}4{d($',']OFNI[', ']GUBED[', ']GNINRAW[', ']RORRE[', ']LACITIRC[']

log looks like below, but is displaying with no line endings as one complete string.

2021-01-27 16:21:12,520-INFO-/opt/project/settings/base.py changed, reloading.
2021-01-27 16:21:19,359-INFO-Watching for file changes with StatReloader
2021-01-27 16:21:41,568-INFO-"GET /admin/ HTTP/1.0" 200 1456
2021-01-27 16:21:41,859-INFO-"GET /log_viewer/ HTTP/1.0" 200 1920
2021-01-27 16:21:42,113-INFO-"GET /log_viewer/json/ HTTP/1.0" 200 292
2021-01-27 16:21:44,427-INFO-"GET /log_viewer/json/django_2Fdjango.log HTTP/1.0" 200 485
2021-01-27 16:21:48,531-INFO-"GET /log_viewer/ HTTP/1.0" 200 1920
2021-01-27 16:21:48,864-INFO-"GET /log_viewer/json/ HTTP/1.0" 200 292
2021-01-27 16:21:50,844-INFO-"GET /log_viewer/json/django_2Fdjango.log HTTP/1.0" 200 517
2021-01-27 16:24:05,794-INFO-/opt/project/settings/base.py changed, reloading.
2021-01-27 16:24:13,582-INFO-Watching for file changes with StatReloader
2021-01-27 16:24:40,182-INFO-"GET /log_viewer/ HTTP/1.0" 200 1920
2021-01-27 16:24:40,326-INFO-"GET /log_viewer/json/ HTTP/1.0" 200 292
2021-01-27 16:24:56,614-INFO-"GET /log_viewer/json/django_2Fdjango.log HTTP/1.0" 200 562
2021-01-27 16:25:21,954-INFO-"GET /log_viewer/json/nginx_2Faccess.log HTTP/1.0" 200 3989
2021-01-27 16:25:43,075-INFO-/opt/project/settings/base.py changed, reloading.
2021-01-27 16:25:49,910-INFO-Watching for file changes with StatReloader
2021-01-27 16:26:03,346-INFO-/opt/project/settings/base.py changed, reloading.
2021-01-27 16:26:11,133-INFO-Watching for file changes with StatReloader
2021-01-27 16:26:16,512-INFO-"GET /log_viewer/ HTTP/1.0" 200 1920
2021-01-27 16:26:16,783-INFO-"GET /log_viewer/json/ HTTP/1.0" 200 292
2021-01-27 16:26:18,879-INFO-"GET /log_viewer/json/django_2Fdjango.log HTTP/1.0" 200 632
2021-01-27 16:27:13,888-INFO-/opt/project/settings/base.py changed, reloading.
2021-01-27 16:27:21,914-INFO-Watching for file changes with StatReloader
2021-01-27 16:27:28,847-INFO-"GET /log_viewer/ HTTP/1.0" 200 1920
2021-01-27 16:27:29,004-INFO-"GET /log_viewer/json/ HTTP/1.0" 200 292
2021-01-27 16:27:31,070-INFO-"GET /log_viewer/json/django_2Fdjango.log HTTP/1.0" 200 678
2021-01-27 16:27:44,974-INFO-/opt/project/settings/base.py changed, reloading.
2021-01-27 16:27:52,028-INFO-Watching for file changes with StatReloader
2021-01-27 16:27:58,853-INFO-"GET /log_viewer/ HTTP/1.0" 200 1920
2021-01-27 16:27:59,153-INFO-"GET /log_viewer/json/ HTTP/1.0" 200 292
2021-01-27 16:28:00,652-INFO-"GET /log_viewer/json/django_2Fdjango.log HTTP/1.0" 200 719
2021-01-27 16:29:50,592-INFO-/opt/project/settings/base.py changed, reloading.
2021-01-27 16:29:57,104-INFO-Watching for file changes with StatReloader

Hi, I want a new feature.

For example, I usually write logs each day, but I need to store 100 days.

And there are 2~3 Loggers. That means, after 100 days, there are 300 log files in LOG FILES area.

So, I think LOG FILES need paging and search form. Thank you!

if log text contains uncomplete HTML tag such as:

"[DEBUG] 2020:10:29 11:03:01 <hello world:hello"

then, it is shown like this: "[DEBUG] 2020:10:29 11:03:01 "

Maybe, It needs to add HTML escape.

My error logs usually comes with a stacktrace text, the logfile already has a formatted text with spaces, new lines and indentation, but when I see it in the log viewer, the format is gone and all the text is in the same line.

Here is an example of the formatted text in logfile

Here is how it is shown in the log viewer

When I use this with swagger, I get the following error:

url pattern does not end in $ ('admin/log-viewer/^json/(?P<file_name>[\.\w-]*)/(?P<page>[0-9]+)') - unexpected things might happen
url pattern does not end in $ ('admin/log-viewer/^json/(?P<file_name>[\.\w-]*)') - unexpected things might happen

Can you fix this?

Some problem with version 1.1.3 prevents the the package from being properly installed. There is no folder "log_viewer" in site-packages after installing with pip install (tested on different platforms). There are no error messages and pip list indicates that the package is installed. But Django fails to start with the following error message:
ModuleNotFoundError: No module named 'log_viewer'
The problem does not occur with version 1.1.2

My LOG_VIEWER_PATTERNS setting

LOG_VIEWER_PATTERNS = ['\d\d\d\d\d\d\d\dT\d\d\d\d\d\dZ#USR=\d#AU=\d#IU=\d#DVC=\d#AD=\d#PD=\d']

How my log looks like on django-log-viewer with this setting (pattern not working as I would hope):

20220113T142500Z#USR=1#AU=1#IU=0#DVC=0#AD=0#PD=020220113T213000Z#USR=1#AU=1#IU=0#DVC=0#AD=0#PD=1 

How the log is formatted:

'fticks':{
            'format':'{asctime}{message}',
            'datefmt':'%Y%m%dT%H%M%SZ',
            'style':'{',
        },

log_message='#USR={}#AU={}#IU={}#DVC={}#AD={}#PD={}'.format(ldap_users,active_users,inactive_users,devices,approved_devices,pending_devices)

An external test of the same regex pattern I've used with django-log-viewer successfully parses this:

# regex-test.py
import re

stringLog='20220113T142500Z#USR=1#AU=1#IU=0#DVC=0#AD=0#PD=020220113T213000Z#USR=1#AU=1#IU=0#DVC=0#AD=0#PD=1'

pattern=re.compile(r'\d\d\d\d\d\d\d\dT\d\d\d\d\d\dZ#USR=\d#AU=\d#IU=\d#DVC=\d#AD=\d#PD=\d')
matches=pattern.finditer(stringLog)

for match in matches:
  print(match)

$ python regex-test.py
<re.Match object; span=(0, 48), match='20220113T142500Z#USR=1#AU=1#IU=0#DVC=0#AD=0#PD=0'>
<re.Match object; span=(48, 96), match='20220113T213000Z#USR=1#AU=1#IU=0#DVC=0#AD=0#PD=1'>

I know I have done something wrong here and would appreciate someone pointing out how I can get this log of mine to split into newlines instead of getting tangled into a single entry in the log viewer. Thanks!

I think it can be better to have an ability to filter logs entities by LOGLEVEL: INFO, WARNING, ERROR, etc.

Hello! Thanks for this package!

I'm having an issue seeing the log files in production when serving staticfiles and an upstream daphne server with nginx. Everything is fine when accessing the view with just daphne or with python manage.py runserver, so I'm wondering what aspect of the response isn't working with nginx since log files aren't being served statically, but as json data. I'm also not seeing any errors or exceptions in the daphne logs coming from this package, so it's hard to tell what's going on

When viewing a log file that spans multiple pages, the Next Page button is enabled, but when pressed, it does not display the next page. The individual page number buttons work as expected, as well as the Previous Page button.

Using version 1.07 on the package.

LOG_VIEWER_PAGE_LENGTH setting does nothing

Steps to reproduce

1. Set LOG_VIEWER_PAGE_LENGTH in settings other than 25
2. Open admin and load any file with 25+ entries, just 25 will be loaded. Changing from admin doesn't work at all

I've found the bug, will create a pull request in a moment

This one took me way too long to realize, only after inspecting the Network console in Firefox I found that static files were missing from the deployment and returning a 404 for each request.

Add step python manage.py collectstatic to the quickstart instructions

Details

• OS (Operating System) version:
• Browser and browser version:
• Django version:
• Log viewer version:

Steps to reproduce

1. no special configuration
2. view logs

Below looping logic is overwriting result["logs"] which is a bug. It ends up showing last sub-folder only.

def get_log_files(directory, max_items_per_page, current_page):
    result = {}
    for root, _, files in os.walk(directory):
        all_files = list(filter(lambda x: x.find("~") == -1, files))

        all_log_files = []
        all_log_files.extend(
            list(filter(lambda x: x in settings.LOG_VIEWER_FILES, all_files))
        )
        all_log_files.extend(
            [x for x in all_files if fnmatch(x, settings.LOG_VIEWER_FILES_PATTERN)]
        )
        log_dir = os.path.relpath(root, directory)
        if log_dir == ".":
            log_dir = ""

        result["logs"] = {log_dir: list(set(all_log_files))}
        result["next_page_files"] = current_page + 1
        result["last_files"] = (
            all_log_files.__len__() <= current_page * max_items_per_page
        )

    return result

Hi, Thank you for this nice module. I have been looking for something like this. But I have a question. In the log location settings, can we indicate a boto3 object (S3, GCP Bucket, DigitalOcean Space,...) so that the logs files are read from a remote location? Or even a remote server? I am looking for something like a nagios/prometeus configuration where it reads remote log files generated by Django and add the lines with logs info to the local db.

Thanks.

#LOG_VIEWER_FILES = ['info.log','debug.log','warning.log','error.log']
#LOG_VIEWER_FILES_PATTERN = '*.log*'
#LOG_VIEWER_FILES_DIR = '/my_app/logs/'

I have tried many combinations of this, even commenting all of these settings altogether, but somehow I will always end up with all five logs I have in the '/my_app/logs' directory (which includes stats.log) - despite wanting only four of the logs specified in LOG_VIEWER_FILES.

I'm confused as to how this is possible even when I give no information whatsoever (by commenting all of them) on where my log files are located or what pattern to use in finding these files. One thing to note however, if I set the wrong location to my logs to LOG_VIEWER_FILES_DIR, instead of commenting the variable entirely, django-log-viewer will indeed not find my log files.

I'm not sure what I am doing wrong here, and would appreciate any help I can get.

There's no newline when viewing the logs. Everything is one big line.

After upgrading to Django 3.2. (from 2.2), the base url /admin/log_viewer cannot be resolved. No view is displayed. This may be due to the fact that I overrode the default admin site.

However, including the log_viewer.urls in my own app works:
path('myownapp/logs/', include('log_viewer.urls')),

I edited the original comment and removed my question regarding the permissions since a quick code review showed that the log views are only accessible to superusers. This should be the optimal solution in most scenarios.

Details

• OS (Operating System) version: OS
• Browser and browser version: Chrome
• Django version: 4.1.6
• Log viewer version: latest

Steps to reproduce

1. Go to /logs
2. refresh page

ERROR 2023-04-21 22:10:28,947 django.request.log_response:241- Internal Server Error: /logs/json/
Traceback (most recent call last):
  File "/Users/magedhelmy/Library/Caches/pypoetry/virtualenvs/compute-local-ILDnfiEN-py3.10/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/Users/magedhelmy/Library/Caches/pypoetry/virtualenvs/compute-local-ILDnfiEN-py3.10/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/Users/magedhelmy/Library/Caches/pypoetry/virtualenvs/compute-local-ILDnfiEN-py3.10/lib/python3.10/site-packages/sentry_sdk/integrations/django/views.py", line 85, in sentry_wrapped_callback
    return callback(request, *args, **kwargs)
  File "/Users/magedhelmy/Library/Caches/pypoetry/virtualenvs/compute-local-ILDnfiEN-py3.10/lib/python3.10/site-packages/django/views/generic/base.py", line 104, in view
    return self.dispatch(request, *args, **kwargs)
  File "/Users/magedhelmy/Library/Caches/pypoetry/virtualenvs/compute-local-ILDnfiEN-py3.10/lib/python3.10/site-packages/django/utils/decorators.py", line 46, in _wrapper
    return bound_method(*args, **kwargs)
  File "/Users/magedhelmy/Library/Caches/pypoetry/virtualenvs/compute-local-ILDnfiEN-py3.10/lib/python3.10/site-packages/django/contrib/auth/decorators.py", line 23, in _wrapper_view
    return view_func(request, *args, **kwargs)
  File "/Users/magedhelmy/Library/Caches/pypoetry/virtualenvs/compute-local-ILDnfiEN-py3.10/lib/python3.10/site-packages/django/utils/decorators.py", line 46, in _wrapper
    return bound_method(*args, **kwargs)
  File "/Users/magedhelmy/Library/Caches/pypoetry/virtualenvs/compute-local-ILDnfiEN-py3.10/lib/python3.10/site-packages/django/contrib/auth/decorators.py", line 23, in _wrapper_view
    return view_func(request, *args, **kwargs)
  File "/Users/magedhelmy/Library/Caches/pypoetry/virtualenvs/compute-local-ILDnfiEN-py3.10/lib/python3.10/site-packages/log_viewer/views.py", line 23, in dispatch
    return super(TemplateView, self).dispatch(*args, **kwargs)
  File "/Users/magedhelmy/Library/Caches/pypoetry/virtualenvs/compute-local-ILDnfiEN-py3.10/lib/python3.10/site-packages/django/views/generic/base.py", line 143, in dispatch
    return handler(request, *args, **kwargs)
  File "/Users/magedhelmy/Library/Caches/pypoetry/virtualenvs/compute-local-ILDnfiEN-py3.10/lib/python3.10/site-packages/django/views/generic/base.py", line 227, in get
    return self.render_to_response(context)
  File "/Users/magedhelmy/Library/Caches/pypoetry/virtualenvs/compute-local-ILDnfiEN-py3.10/lib/python3.10/site-packages/log_viewer/views.py", line 106, in render_to_response
    log_json = self.get_log_json(context)
  File "/Users/magedhelmy/Library/Caches/pypoetry/virtualenvs/compute-local-ILDnfiEN-py3.10/lib/python3.10/site-packages/log_viewer/views.py", line 47, in get_log_json
    context["next_page_files"] = log_file_data["next_page_files"]
KeyError: 'next_page_files'

Hi, I use TimedRotatingFileHandler now.

It generates timed rotating log files and it is renamed log file such as:

daily.log.2020-08-25
daily.log.2020-08-26
daily.log.2020-08-27

But in this case, I can't show all logs files in the log viewer.

Any method that I see all of logs in the directory?

Hi,

I set my firefox to dark mode, which automatically sets django admin to dark mode.

What is not affected is the page for this project. It looks as follows:

I have never done open source, but this seems like an "easy" to fix issue, I'll get on to fixing it.

Creating issue to make it known :)

I'm testing with log_viewer_demo and running into an issue with Django 3.1. After setting the demo up and logging into the admin site, it does not list the log files on the web page: /admin/log_viewer. I'm also seeing the following message in the console from the development server: [05/Aug/2020 19:15:38] "GET /admin/log_viewer/json/ HTTP/1.1" 500 110088. When I run the server with Python -Wd, I also see the following message: RemovedInDjango40Warning: TemplateView passing URL kwargs to the context is deprecated. Reference file_name in your template through view.kwargs instead.

When I install Django 3.0.6, the log_viewer_demo works correctly and get this message: [05/Aug/2020 19:28:23] "GET /admin/log_viewer/json/ HTTP/1.1" 200 448

I'm running this on Python 3.6.8.

Hi,

It would be nice to have a download button next to the .log files to be able to download the entire files for further investigation.
Great work by the way, keep going!

I assume the following is not supported right now:

LOG_VIEWER_FILES_DIR = ['path/to/dir1', 'path/to/dir2']

It would be to have such feature, sometimes different parts of the system write to different target dirs.

The (single) log file is read, but the file content (in this simple example just two lines) is shown as a single entry, i.e. ignoring line breaks (see screenshot). The log file is UTF-8 encoded, uses the common \n end-of-line character and is displayed correctly in text editors on Linux and Windows.

This problem was found when "Log files" was clicked, and go to another page,
and log entries loaded all. But after the dropdown "Show entries (page length)" the pagination working fine.

The current version at PyPi is 1.1.2 but at this repository there is already a release for 1.1.3.
See: https://pypi.org/project/django-log-viewer/#history

Is it possible to upload the current version to PyPi?

When the log file contains non-ascii characters, it seems they are parsed incorrectly.

Say I have a log file with content:
[ERROR] [567719dee80742868e17b9414bed686e] 解析脚本异常

Will be parsed as:

[ERROR] [567719dee80742868e17b9414bed686e] 解析析析脚脚脚本本本异异异常常常