django-statsd / django-statsd

Django Statsd

Documentation is on Read the Docs.

License

BSD and MPL

Portions of this are from commonware:

https://github.com/jsocol/commonware/blob/master/LICENSE

django-statsd's Issues

Cannot import name patches

$ ./manage.py shell
/home/deploy/Builds/PyEnv/django/local/lib/python2.7/site-packages/nose/plugins/manager.py:395: RuntimeWarning: Unable to load plugin django_statsd = django_statsd:NoseStatsd: [Errno -2] Name or service not known
  RuntimeWarning)

>>> from django_statsd.clients import statsd
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/deploy/Builds/PyEnv/django/local/lib/python2.7/site-packages/django_statsd/__init__.py", line 1, in <module>
    from django_statsd import patches
ImportError: cannot import name patches

stop raising 500's

We should try and raise as few 500's as possible in the recording view, for example when the client is wrong.

Django 1.9 'WSGIRequest' object has no attribute 'REQUEST'

ERROR 2015-12-13 11:42:36,721 base 8857 140332125685504 Internal Server Error: /services/timing/record
Traceback (most recent call last):
  File "/home/aaron/.virtualenvs/dna/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 149, in get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/aaron/.virtualenvs/dna/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 147, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/aaron/.virtualenvs/dna/local/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/home/aaron/.virtualenvs/dna/local/lib/python2.7/site-packages/django_statsd/views.py", line 149, in record
    if 'client' not in request.REQUEST:
AttributeError: 'WSGIRequest' object has no attribute 'REQUEST'
[13/Dec/2015 11:42:37] "POST /services/timing/record HTTP/1.1" 500 8581

new logger not included in distribution

Hi,
I just installed 0.3.1 from pypi to get the new logger, but it does not seem to be in the distro.
I think the loggers (and managent) directory need to be added to setup.py for them to show up.

Cheers,
Gijsbert

readthedocs links are dead

The documentation links to https://django-statsd.readthedocs.io/en/latest/index.html
It gives a 404 error. Documentation is not viewable anywhere now.

Syntax error installing django-statsd under Python 3

When I install django-statsd under Python 3.4.0, it seems to install correctly, but also reports a syntax error:

$ pip install django-statsd-mozilla
Downloading/unpacking django-statsd-mozilla
  Downloading django-statsd-mozilla-0.3.14.tar.gz
  Running setup.py (path:/home/mcollum/.virtualenvs/testing/build/django-statsd-mozilla/setup.py) egg_info for package django-statsd-mozilla

Downloading/unpacking statsd>=2.0.0 (from django-statsd-mozilla)
  Downloading statsd-3.1-py2.py3-none-any.whl
Installing collected packages: django-statsd-mozilla, statsd
  Running setup.py install for django-statsd-mozilla

      File "/home/mcollum/.virtualenvs/testing/lib/python3.4/site-packages/django_statsd/parse.py", line 1683
        print '<tr>'
                   ^
    SyntaxError: invalid syntax

Successfully installed django-statsd-mozilla statsd
Cleaning up...

I would have submitted a pull request, but parse.py doesn't actually seem to exist in this repository. Perhaps something went wrong in the release process?

the presence of the statsd panel on django_toolbar seems to break the templates panel

• My Templates panel on the django debug toolbar broke.
• I've done a git bisect and found that the breaking point is when I added the statsd panel.
• I've confirmed that removing the statsd panel fixes the problem. Just commenting out the line in DEBUG_TOOLBAR_PANELS without changing any other settings

I have latest versions of django, django-debug-toolbar and django-statsd-mozilla. I haven't created a minimal example yet to confirm this. Is this a known issue? are there any fixes?

Travis CI no longer running on new commits

Seems like it's because the repo was changed from andymckay/django-statsd to django-statsd/django-statsd.

It would be nice to know if the new tests enabled by bce92c6 are actually passing :)

Can you update the pypi for support Django 1.9 ?

#93 #90

django-statsd-mozilla now still stay at 2015-10-01

Thanks for your helpful statsd tool !

Newest release doesn't include datadog

statsd 3.1 breaks metrics collecting

New version of statsd broke metrics collecting. Please lock version in requirements. It happened because etsy statsd have ipv6 disabled by default.

Document How To Run Tests

I was going to try to create a pull request for #45, but I wasn't sure how to run the tests to verify that what I was doing was going to work 😃

Django 1.5 seems to be incompatible, but 1.4.3 still works for me

$ nosetests -sv .
Failure: ImproperlyConfigured (Requested setting STATSD_PATCHES, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.) ... ERROR

======================================================================
ERROR: Failure: ImproperlyConfigured (Requested setting STATSD_PATCHES, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/victorng/.virtualenvs/django-statsd/local/lib/python2.7/site-packages/nose/loader.py", line 390, in loadTestsFromName
    addr.filename, addr.module)
  File "/home/victorng/.virtualenvs/django-statsd/local/lib/python2.7/site-packages/nose/importer.py", line 39, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/home/victorng/.virtualenvs/django-statsd/local/lib/python2.7/site-packages/nose/importer.py", line 86, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/home/victorng/dev/django-statsd/django_statsd/__init__.py", line 1, in <module>
    from django_statsd import patches
  File "/home/victorng/dev/django-statsd/django_statsd/patches/__init__.py", line 6, in <module>
    patches = getattr(settings, 'STATSD_PATCHES', [])
  File "/home/victorng/.virtualenvs/django-statsd/local/lib/python2.7/site-packages/django/conf/__init__.py", line 52, in __getattr__
    self._setup(name)
  File "/home/victorng/.virtualenvs/django-statsd/local/lib/python2.7/site-packages/django/conf/__init__.py", line 45, in _setup
    % (desc, ENVIRONMENT_VARIABLE))
ImproperlyConfigured: Requested setting STATSD_PATCHES, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

----------------------------------------------------------------------
Ran 1 test in 0.004s

FAILED (errors=1)

Use csrf_exempt decorator for record() view

The record() view, used to receive navigation timing data currently fails if Django's CSRF functionality is globally turned on.

The view should use csrf_exempt decorator to accept POST data without a CSRF token.

add metlog backend as an alternate statsd client

Add suppport for metlog-py as a statsd client

Merge in the statsd log handler

I think the error handler we use zamboni is actually quite small and useful, we should merge that in (minus the whitelisting code).

https://github.com/mozilla/zamboni/blob/master/lib/misc/admin_log.py#L75

SQL patch fails to load with debug_toolbar

The db patch expects debug_toolbar.panels.sql to have a DatabaseStatTracker class but at least with 0.9.4 this is not the case and I get an AttributeError when django_statsd attempts to patch the sql panel:

https://github.com/andymckay/django-statsd/blob/master/django_statsd/patches/db.py#L52

util.CursorDebugWrapper = wrap_class(sql.DatabaseStatTracker)

toolbar panel doesn't work if 'django_statsd' is not in INSTALLED_APPS

I think this step should be added to docs.

Got "maximum recursion depth exceeded" with django_statsd.patches.db

in

File ".../env/local/lib/python2.7/site-packages/django/db/backends/util.py" in execute
  69.             return super(CursorDebugWrapper, self).execute(sql, params)

Django 1.6.1
django-statsd-mozilla 0.3.10

debug toolbar panel content not showing, django 1.6 + debug-toolbar 1.2 + django-statsd 0.3.12

With the latest version of both debug-toolbar and django-statsd the debug-toolbar does not show the panel contents and instead outputs:

<bound method statsdpanel.content of <django_statsd.panel.statsdpanel object at 0x1157c9a90>>

Don't use .timing for statsd_ping

The statsd_ping management command currently uses StatsClient.timing to send non-timing data.

I admit that this isn't well documented, but the second argument to StatsClient.timing is an integer number of milliseconds, designed to time how long something took. (It's used internally by StatsClient.timer, for an example.) Currently we're passing a totally nonsensical value for this.

We should really be using incr to send a ping here. For our use case, the drawAsInfinite modifier should still work exactly the same way.

update to use pipeline

jsocol added a pipeline, https://github.com/jsocol/pystatsd/pull/20/commits we should use that

0.3.13 Changes

0.3.13 has been pushed to PyPi, but the Changes.rst file hasn't been updated

When there is an exception the middleware records response time twice

In the GraphiteRequestTimingMiddleware because the self._record_time() method is called in both the process_response and process_exception methods the timing details are recorded twice. I understand that it is recording from the point up to the exception and then recording the actual 500 response time but since they are both marked as the same stat it could skew the data and make it look like the view was responding really fast but it was actually exceptions.

You can see from the screenshot, the 371ms is (I think) the time up to the exception, then the 70ms is the time for the 500 response. I just do not think these should be marked with the same name. If you agree I would be happy to submit a patch.

Python 3 won't install unittest2 in requirements.txt

https://travis-ci.org/andymckay/django-statsd/builds/37311591

Tests pass in Python 3, but it doesn't like unittest2 in the requirements. Maybe we need to separate into two requirements files?

Lock in statsd version (semver)

>= causes issues generally (as you can see with older versions of this project)

Switch to a monotonic timer in middleware and celery integration

Django 3.0 has already switched to time.monotonic() and this package should as well.
Need to check on pystatd itself as well of course.

Version Inconsistencies

Hi,

Not sure if there is any other place to put up such questions. Here are a bunch of questions on version inconsistency:

$ pip install django-statsd-mozilla
Downloading/unpacking django-statsd-mozilla
  Downloading django-statsd-mozilla-0.3.8.7.tar.gz

pip downloads version 0.3.8.7 while the only release on github is 0.3.8.5

Further, it seems to downgrade statsd from version 2.0.1 to 1.0.0:

Downloading/unpacking statsd==1.0.0 (from django-statsd-mozilla)
  Downloading statsd-1.0.0.tar.gz
  Running setup.py egg_info for package statsd

Installing collected packages: django-statsd-mozilla, statsd
  Running setup.py install for django-statsd-mozilla

  Found existing installation: statsd 2.0.1
    Uninstalling statsd:
      Successfully uninstalled statsd
  Running setup.py install for statsd

Is there any hardcoded dependency that version 1.0.0 is required? If not we would like to use latest and greatest version of statsd.

requirements file should support more recent djangos

Django 1.6 support is apparently in but the requirements file appears be a bit behind.

moz_metlog client does not honor the STATSD_PREFIX setting in Django

The STATSD_PREFIX setting is not used at all by the moz_metlog client which will result in statsd messages using an incorrect namespace

Heroku no longer resolving localhost

It looks like Heroku's no longer resolving localhost so is throwing an error on the statsd middleware if STATSD_HOSTNAME is not set. Looks like this would also be an issue if STATSD_HOSTNAME was an invalid hostname that DNS could not resolve.

We weren't really using statsd so I've just pulled from our app, but the trace is below for reference.

Traceback (most recent call last):
File "/app/.heroku/python/lib/python2.7/site-packages/newrelic-1.10.2.38/newrelic/api/web_transaction.py", line 780, in call result = application(environ, _start_response)
File "/app/.heroku/python/lib/python2.7/site-packages/newrelic-1.10.2.38/newrelic/api/object_wrapper.py", line 220, in call self._nr_instance, args, kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/newrelic-1.10.2.38/newrelic/api/function_trace.py", line 93, in literal_wrapper return wrapped(_args, *_kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/newrelic-1.10.2.38/newrelic/api/web_transaction.py", line 669, in call return self._nr_next_object(environ, start_response)
File "/app/.heroku/python/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 219, in call self.load_middleware()
File "/app/.heroku/python/lib/python2.7/site-packages/newrelic-1.10.2.38/newrelic/api/object_wrapper.py", line 220, in call self._nr_instance, args, kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/newrelic-1.10.2.38/newrelic/api/post_function.py", line 9, in wrapper result = wrapped(_args, *_kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/django/core/handlers/base.py", line 45, in load_middleware mod = import_module(mw_module)
File "/app/.heroku/python/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module import(name)
File "/app/.heroku/python/lib/python2.7/site-packages/django_statsd/init.py", line 1, in from django_statsd import patches
File "/app/.heroku/python/lib/python2.7/site-packages/django_statsd/patches/init.py", line 11, in import_module(patch).patch()
File "/app/.heroku/python/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module import(name)
File "/app/.heroku/python/lib/python2.7/site-packages/django_statsd/patches/db.py", line 3, in from django_statsd.patches.utils import wrap
File "/app/.heroku/python/lib/python2.7/site-packages/django_statsd/patches/utils.py", line 1, in from django_statsd.clients import statsd
File "/app/.heroku/python/lib/python2.7/site-packages/django_statsd/clients/init.py", line 28, in _statsd = get_client()
File "/app/.heroku/python/lib/python2.7/site-packages/django_statsd/clients/init.py", line 25, in get_client return import_module(client).StatsClient(host, port,\ prefix)
File "/app/.heroku/python/lib/python2.7/site-packages/statsd/client.py", line\ 39, in init self._addr = (socket.gethostbyname(host), port)
File "/app/.heroku/python/lib/python2.7/site-packages/gevent/socket.py", line\ 683, in gethostbyname return _socket.gethostbyname(hostname)
gaierror: [Errno -2] Name or service not known

Stop using `is_managed`, which is deprecated (and removed) in Django 1.6

Error I'm getting when using django-statd 0.3.10 with Django 1.6:

  File "myproject/.env/site-packages/django_statsd/patches/db.py", line 16, in __getattr__
    if self.db.is_managed():
AttributeError: 'DatabaseWrapper' object has no attribute 'is_managed'

Here's the usage in the project: patches/db.py

Here's the section in the Django deprecation docs and a relevant ticket

Not connecting to Celery signals on Python 2 with v0.4

Since upgrading from v0.3.16 to v0.4.0, handlers are not connecting to Celery's signals - no stats are getting sent from Celery.

With v0.3.16

Relevant packages:

celery==4.1.0
Django==1.8.19
django-statsd-mozilla==0.3.16

Testing via iPython:

In [1]: from celery import signals

In [2]: signals.task_sent.has_listeners()
Out[2]: True

In [3]: signals.task_prerun.has_listeners()
Out[3]: True

In [4]: signals.task_postrun.has_listeners()
Out[4]: True

In [5]: signals.task_failure.has_listeners()
Out[5]: True

In [6]: signals.task_sent.receivers
Out[6]: 
[((139822539469840, 9392928),
  <weakref at 0x7f2af8d29cb0; to 'function' at 0x7f2af8cbf410 (on_task_sent)>)]

... which is correctly connected to celery.on_task_sent:

django-statsd/django_statsd/celery.py

Lines 8 to 13 in cddc6dc

	def on_task_sent(sender=None, task_id=None, task=None, **kwds):
	"""
	Handle Celery ``task_sent`` signals.
	"""
	# Increase statsd counter.
	statsd.incr('celery.%s.sent' % task)

With v0.4.0

Relevant packages:

celery==4.1.0
Django==1.8.19
django-statsd-mozilla==0.4.0

iPython:

In [1]: from celery import signals

In [2]: signals.task_sent.has_listeners()
Out[2]: False

In [3]: signals.task_prerun.has_listeners()
Out[3]: False

In [4]: signals.task_postrun.has_listeners()
Out[4]: False

In [5]: signals.task_failure.has_listeners()
Out[5]: False

Release new package with recent changes

Would you release new package version containing latest master? With the latest depreciation fixes.

Thanks in advance

docs: there is no django_statsd.clients.statsd

I think docs are incorrect: there is no 'django_statsd.clients.statsd', it should be 'statsd.client' or 'django_statsd.clients.normal'.

Incompatibilty with eventlet

Hi,

when django-statsd is used along with eventlet (0.15.1) package there is a Runtime Exception:

RuntimeError("Second simultaneous write on fileno 5 detected. Unless you really know what you're doing, make sure that only one greenthread can write any particular socket. Consider using a pools.Pool. If you do know what you're doing and want to disable this error, call eventlet.debug.hub_prevent_multiple_readers(False) - MY THREAD=<built-in method switch of greenlet.greenlet object at 0x2e62910>; THAT THREAD=FdListener('write', 5, <built-in method switch of greenlet.greenlet object at 0x30de7d0>, <built-in method throw of greenlet.greenlet object at 0x30de7d0>)",)

File "statsd/client.py", line 88, in incr
self._send_stat(stat, '%s|c' % count, rate)
File "statsd/client.py", line 112, in _send_stat
self._after(self._prepare(stat, value, rate))
File "statsd/client.py", line 127, in _after
self._send(data)
File "statsd/client.py", line 132, in _send
self._sock.sendto(data.encode('ascii'), self._addr)
File "eventlet/greenio.py", line 370, in sendto
self._trampoline(self.fd, write=True)
File "eventlet/greenio.py", line 185, in _trampoline
mark_as_closed=self._mark_as_closed)
File "eventlet/hubs/init.py", line 157, in trampoline
listener = hub.add(hub.WRITE, fileno, current.switch, current.throw, mark_as_closed)
File "eventlet/hubs/epolls.py", line 48, in add
listener = BaseHub.add(self, evtype, fileno, cb, tb, mac)
File "eventlet/hubs/hub.py", line 169, in add
evtype, fileno, evtype, cb, bucket[fileno]))

Incorrect gauge signature, confusion with required statsd version

django-statsd current statsd gauge signature
(in log and toolbar clients)

def gauge(self, stat, value, rate=1):

like in statsd version 1.0.0 (this version also in requirements.txt)

but django-statsd-mozilla pypi package requires statsd>=2.0.0
there gauge signature was changed to

def gauge(self, stat, value, rate=1, delta=False):

Update to statsd ~2.0

statsd 1.0 is nearly a year old and now I'm getting runtime warnings for using django_statsd and the new statsd.