invitebox / django-live-profiler

A database access profiler for Django-based applications that can be ran in production

django-live-profiler

Django-live-profiler is a low-overhead data access and code profiler for Django-based applications. For more information, check out http://invitebox.github.com/django-live-profiler/

Installation

1. Run pip install django-live-profiler
2. Add 'profiler' app to INSTALLED_APPS
3. Add 'profiler.middleware.ProfilerMiddleware' to MIDDLEWARE_CLASSES
4. Optionally add 'profiler.middleware.StatProfMiddleware' to MIDDLEWARE_CLASSES to enable Python code statistical profiling (using statprof). WARNING: this is an experimental feature, beware of possible incorrect output.
5. Add url(r'^profiler/', include('profiler.urls')) to your urlconf

Usage

In order to start gathering data you need to start the aggregation server:

$ aggregated --host 127.0.0.1 --port 5556

Visit http://yoursite.com/profiler/ for results.

django-live-profiler's Issues

Snapshots

Make aggregation service capable of saving snapshots.

Signal only works in main thread

I am getting the following exception when trying to run my app.

ValueError at /
signal only works in main thread

/srv/www/envs/cwu/lib/python2.7/site-packages/statprof.py in stop
       260.  signal.signal(signal.SIGPROF, signal.SIG_IGN) 

If I remove 'profiler.middleware.StatProfMiddleware', from MIDDLEWARE_CLASSES, the problem disappears but I have installed django-live-profiler specifically for python profiling.

I run the test server (./manage.py runserver). Any help, please?

ImportError: cannot import name empty_iter

  File "[...]/venv/lib/python2.7/site-packages/profiler/models.py", line 5, in <module>
    from profiler.instrument import *
  File "[...]/venv/lib/python2.7/site-packages/profiler/instrument.py", line 3, in <module>
    from django.db.models.sql.compiler import SQLCompiler, empty_iter
ImportError: cannot import name empty_iter

Drill by view

ImportError while running django server: cannot import name 'empty_iter' from 'django.db.models.sql.compiler'

Got an exception while running libe profiler(0.0.9) with Django==2.2.17

(venv) savva.genchevskiy@NB072 src % python manage.py runserver --noreload --nothreading
Traceback (most recent call last):
  File "manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/savva.genchevskiy/PycharmProjects/django-profiling-poc/venv/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/Users/savva.genchevskiy/PycharmProjects/django-profiling-poc/venv/lib/python3.7/site-packages/django/core/management/__init__.py", line 357, in execute
    django.setup()
  File "/Users/savva.genchevskiy/PycharmProjects/django-profiling-poc/venv/lib/python3.7/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/Users/savva.genchevskiy/PycharmProjects/django-profiling-poc/venv/lib/python3.7/site-packages/django/apps/registry.py", line 114, in populate
    app_config.import_models()
  File "/Users/savva.genchevskiy/PycharmProjects/django-profiling-poc/venv/lib/python3.7/site-packages/django/apps/config.py", line 211, in import_models
    self.models_module = import_module(models_module_name)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/savva.genchevskiy/PycharmProjects/django-profiling-poc/venv/lib/python3.7/site-packages/profiler/models.py", line 5, in <module>
    from profiler.instrument import *
  File "/Users/savva.genchevskiy/PycharmProjects/django-profiling-poc/venv/lib/python3.7/site-packages/profiler/instrument.py", line 3, in <module>
    from django.db.models.sql.compiler import SQLCompiler, empty_iter
ImportError: cannot import name 'empty_iter' from 'django.db.models.sql.compiler' (/Users/savva.genchevskiy/PycharmProjects/django-profiling-poc/venv/lib/python3.7/site-packages/django/db/models/sql/compiler.py)

Measure avg time per HTTP request for SQL queries

TypeError: unicode not allowed, use setsockopt_string with python 3.2 on Debian Wheezy

Hi
I am launching server with aggregated --host 127.0.0.1 --port 5556 and it throws me an exception:
Traceback (most recent call last):
File "/home/webapp/www/env/bin/aggregated", line 9, in
load_entry_point('django-live-profiler==0.0.9', 'console_scripts', 'aggregated')()
File "/home/webapp/www/env/lib/python3.2/site-packages/aggregate/server.py", line 73, in main
socket.setsockopt(zmq.SUBSCRIBE,'')
File "socket.pyx", line 310, in zmq.backend.cython.socket.Socket.set (zmq/backend/cython/socket.c:3184)
TypeError: unicode not allowed, use setsockopt_string

I use python 3.2 on Debian Wheezy and virtualenv.

TypeError: setsockopt() argument 3 must be string or read-only buffer, not None

saboor@ubuntu:~/staticdhcpd/staticDHCPd$ sudo python staticDHCPd
2019-03-29 12:23:40,990 : INFO : Console-based logging online
2019-03-29 12:23:40,991 : WARNING : ----------------------------------------
2019-03-29 12:23:40,991 : WARNING : ----------------------------------------
2019-03-29 12:23:40,991 : WARNING : ----------------------------------------
2019-03-29 12:23:40,991 : WARNING : System startup in progress; PID=29870
2019-03-29 12:23:40,991 : WARNING : staticDHCPd version 2.0.0-rc1 : http://uguu.ca/
2019-03-29 12:23:40,991 : WARNING : libpydhcpserver version 2.0.0-rc1 : http://uguu.ca/
2019-03-29 12:23:40,992 : WARNING : Continuing with subsystem initialisation
2019-03-29 12:23:40,992 : WARNING : ----------------------------------------
2019-03-29 12:23:40,992 : INFO : Webservice module enabled; configuring...
2019-03-29 12:23:41,049 : INFO : Prepared to bind to 0.0.0.0:30880
2019-03-29 12:23:41,051 : INFO : Configured Webservice engine
2019-03-29 12:23:41,053 : INFO : Webservice engine beginning normal operation
2019-03-29 12:23:41,058 : INFO : Webservice logging module enabled; configuring...
2019-03-29 12:23:41,059 : INFO : Configuring web-accessible logging...
2019-03-29 12:23:41,059 : INFO : Web-accessible logging online; buffer-size=200
2019-03-29 12:23:41,060 : INFO : Initialising custom code...
2019-03-29 12:23:41,109 : INFO : Prepared to bind to 192.168.1.1; ports: server: 67, client: 68, proxy: None; raw-response-interface: -
2019-03-29 12:23:41,159 : CRITICAL : System shutdown triggered by unhandled exception:
Traceback (most recent call last):
File "staticDHCPd", line 294, in
_initialiseDHCP()
File "staticDHCPd", line 224, in _initialiseDHCP
dhcp = staticdhcpdlib.dhcp.DHCPService(database)
File "/home/saboor/staticdhcpd/staticDHCPd/staticdhcpdlib/dhcp.py", line 865, in init
database
File "/home/saboor/staticdhcpd/staticDHCPd/staticdhcpdlib/dhcp.py", line 410, in init
response_interface_qtags=response_interface_qtags,
File "/home/saboor/staticdhcpd/staticDHCPd/libpydhcpserver/dhcp.py", line 115, in init
self._network_link = _NetworkLink(str(server_address), server_port, client_port, proxy_port, response_interface, response_interface_qtags=response_interface_qtags)
File "/home/saboor/staticdhcpd/staticDHCPd/libpydhcpserver/dhcp.py", line 285, in init
(dhcp_socket, proxy_socket) = self._setupListeningSockets(server_port, proxy_port, server_address)
File "/home/saboor/staticdhcpd/staticDHCPd/libpydhcpserver/dhcp.py", line 358, in _setupListeningSockets
dhcp_socket.setsockopt(socket.SOL_SOCKET, _SO_BINDTODEVICE, listen_interface)
File "/usr/lib/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)

Collapse long queries

Socket EOFError

I'm running the command $ aggregated --host 127.0.0.1 --port 5557
And when I try to load the url http://127.0.0.1:8000/profiler/ it raises the Socket EOFError.

Here is the Traceback

Traceback (most recent call last):
  File "/Users/peteralaoui/Sites/Powergun/rocky-mountain-6109/venv/bin/aggregated", line 9, in <module>
    load_entry_point('django-live-profiler==0.0.8', 'console_scripts', 'aggregated')()
  File "/Users/peteralaoui/Sites/Powergun/rocky-mountain-6109/venv/src/django-live-profiler/aggregate/server.py", line 80, in main
    q = socket.recv_pyobj()
  File "/Users/peteralaoui/Sites/Powergun/rocky-mountain-6109/venv/lib/python2.7/site-packages/zmq/core/pysocket.py", line 294, in recv_pyobj
    return pickle.loads(s)
EOFError

I will investigate on my side why is it crashing.
Do you think it's because I'm running inside a virtualenv ?

Allow sorting views

Reset button

Mongo profiling

redirecting URL to http://localhost:8000/accounts/login/?next=/profiler/

Whenever I type http://localhost:8000/profiler/

It redirects me to

http://localhost:8000/accounts/login/?next=/profiler/

I do not have accounts/login in my application.

Significant overhead on code profiling

Seems to be mostly caused by transmitting the data to aggregated - serialization?

ImportError - No module named defaults

Django : 1.6.6
not able to get the profiler to work.

from django.conf.urls.defaults import * ...
urlpatterns = patterns(
'profiler.views',
url(r'^$', 'global_stats', name='profiler_global_stats'),
url(r'^by_view/$', 'stats_by_view', name='profiler_stats_by_view'),
url(r'^code/$', 'python_stats', name='profiler_python_stats'),

Outgoing HTTP calls profiling