elastic / apm-agent-python

Official Python agent for Elastic APM

elastic-apm -- Elastic APM agent for Python

This is the official Python module for Elastic APM.

It provides full out-of-the-box support for many of the popular frameworks, including Django, and Flask. Elastic APM is also easy to adapt for most WSGI-compatible web applications via custom integrations.

Your application doesn't live on the web? No problem! Elastic APM is easy to use in any Python application.

Read the documentation, including instructions on running the tests locally.

If you're interested in contributing, start here!

License

BSD-3-Clause

Made with ♥️ and ☕️ by Elastic and our community.

apm-agent-python's Issues

Add support for Metrics / Stats API

Work is underway to add support for Metrics to the API/server: elastic/apm-server#964
Support needs to be added to this agent too.
The default minimum set of metrics should be supported. TBD which additional metrics will be supported for this agent.

Consider using black code formatter

Currently, we try to adhere to pep8 and ensure adherence to pep8 (and some other code style rules) by running flake8 on each PR (via Hound). This has the benefit of enforcing PEP8 via CI, but it can be annoying for the person who pushes code: they push the code, and immediately get a whole bunch of drily formulated messages that point out style errors.

In a relatively recent development, a code formatter ala gofmt is gaining a lot of traction in the Python world: black. Similar to gofmt, it completely ignores the style of the input. Instead, it parses the Python file and rebuilds it from scratch, in a stable way.

The benefit is that the user can completely forget about style. Just write it however you want, and let black reformat it. The drawback (some might call it a benefit) is that black has complete control over the style.

Needed tasks:

• add pre-commit hooks to run black before committing
• add documentation on used code style to CONTRIBUTING.md
• add test runner that ensures that running black on a commit results in an empty diff
• make sure flake8 doesn't check any style violations (for the rare case where black and flake8 might disagree)

Add `process.ppid`

In elastic/apm-server/pull/557, apm-server added the process.ppid field, which should contain the parent process ID, found by os.getppid().

Add instrumentation for pylibmc (memcache)

Currently, we instrument python-memcached. There's a second relatively popular memcache client library, pylibmc. Looking at pypi download numbers since Jan 1, python-memcached has 612k downloads, while pylibmc has 170k.

'NoneType' object has no attribute 'pop'

Traceback (most recent call last):
  File ".../sth.py", line 328, in listCategory
    return es.search(index=index_name, doc_type=my_doc_type, body=query, size=20, from_=skip, analyzer='nfkc_cf_normalized', lenient=True).get('hits', {}).get('hits', [])
  File "/usr/local/lib/python2.7/dist-packages/elasticapm/instrumentation/packages/base.py", line 110, in call_if_sampling
    return self.call(module, method, wrapped, instance, args, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/elasticapm/instrumentation/packages/elasticsearch.py", line 147, in call
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py", line 76, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py", line 655, in search
    doc_type, '_search'), params=params, body=body)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 311, in perform_request
    connection = self.get_connection()
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 171, in get_connection
    self.sniff_hosts()
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 235, in sniff_hosts
    node_info = self._get_sniff_data(initial)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 198, in _get_sniff_data
    timeout=self.sniff_timeout if not initial else None)
  File "/usr/local/lib/python2.7/dist-packages/elasticapm/instrumentation/packages/base.py", line 110, in call_if_sampling
    return self.call(module, method, wrapped, instance, args, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/elasticapm/instrumentation/packages/elasticsearch.py", line 35, in call
    body = params.pop(BODY_REF_NAME, None)
AttributeError: 'NoneType' object has no attribute 'pop'

and this fixes it:

37:            body = params.pop(BODY_REF_NAME, None) if bool(params) else None
38:
39:            api_method = params.pop(API_METHOD_KEY_NAME, None) if bool(params) else None
...
46:            if bool(params) and 'q' in params:

Transaction.Response.StatusCode is expected to be Number not String

I experienced an error being sent to the transactions endpoint. It clearly happened in a development scenario, when I applied a change in a running Django app. Feel free to disregard this 'bug' report if it cannot happen in a 'real-world-scenario'.

Django console logging:

HTTP 400:
Failed to submit message: '<no message value>'
Traceback (most recent call last):
  File "/Users/simitt/.pyenv/versions/django-test-3.6.2/lib/python3.6/site-packages/elasticapm/transport/http_urllib3.py", line 75, in send_sync
    url = Urllib3Transport.send(self, data, headers)
  File "/Users/simitt/.pyenv/versions/django-test-3.6.2/lib/python3.6/site-packages/elasticapm/transport/http_urllib3.py", line 61, in send
    raise TransportException(message, data, print_trace=print_trace)
elasticapm.transport.base.TransportException: HTTP 400:

APM Server logging:

  I[#/transactions/0/context/response/status_code] S[#/properties/transactions/items/properties/context/properties/response/properties/status_code/type] expected number or null, but got string, code=400

Document tagging API

There's an example in

apm-agent-python/elasticapm/traces.py

Line 244 in 54082d9

def tag(**tags):

but it would be great to have this in the official docs.

Also, the example in there should say elasticapm or similar instead of opbeat, probably.

Support for web2py framework

web2py is still an active and widely used framework. It would be nice to have nativesupport for it along with Django and Flask.

Problem validating JSON document against schema

The apm server keeps dropping certain python agent requests with the following error:

2018-01-17T13:41:51.252+0100	ERROR	[request]	beater/handlers.go:384	error handling request	
{"request_id": "104c6f2b-8e7a-4480-aca7-cc292b890b4e", "error": "Problem validating JSON document against schema: I[#] S[#] doesn't validate with \"transaction#\"\n  I[#] S[#/required] missing properties: \"service\""}

Since it's a schema requirement error, and the schema definition here specifies that the service field should be present , i thought this was the best place to post the issue

Problem validating JSON - I[#/service/name] S[#/properties/service/properties/name/type] expected string, but got null

Hi there,

Right after update apm-server from 6.1.2 to 6.2.0 I've started to get these:

HTTP 400: Problem validating JSON document against schema: I[#] S[#] doesn't validate with "transaction#"
I[#/service/name] S[#/properties/service/properties/name/type] expected string, but got null
Failed to submit message: ''
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/elasticapm/transport/http_base.py", line 45, in send_sync
url = self.sync_transport.send(self, data, headers)
File "/usr/lib/python2.7/site-packages/elasticapm/transport/http.py", line 77, in send
raise TransportException(message, data, print_trace=print_trace)
TransportException: HTTP 400: Problem validating JSON document against schema: I[#] S[#] doesn't validate with "transaction#"
I[#/service/name] S[#/properties/service/properties/name/type] expected string, but got null

apm-server: 6.2.0 (docker)
apm-agent: 2.0.0 (django with uwsgi)
elasticsearch: 6.2.1 (elastic cloud)

Disable automatic Celery instrumentation

Is there any way to not instrument Celery in a Django project?

I run thousands of celery tasks every minute using gevent. Elastic APM will make the tasks halt and lock up the database.

Remove Python 3.3 from test matrix?

Python 3.3 has reached EOL in September 2017. While currently it doesn't require a huge amount of work to stay compatible with Python 3.3, a lot of libraries that we instrument already stopped supporting it, making our matrix exclusion list ever more unwieldy (about a third of all exclusions are for Python 3.3).

Opinions?

HTTP 400: Problem validating JSON document against schema: unexpected EOF

I'm having troubles with submitting transactions that have more than 900KB:

INFO 2018-02-05 10:45:01,326 /var/virtualenv/lib/python3.5/site-packages/elasticapm/transport/http.py 54 12002 140542526174976 Sent request, url=XXX/v1/transactions size=957.61kb status=400
ERROR 2018-02-05 10:45:04,590 /var/virtualenv/lib/python3.5/site-packages/elasticapm/base.py 256 11718 140542526174976 HTTP 400: Problem validating JSON document against schema$
ERROR 2018-02-05 10:45:05,075 /var/virtualenv/lib/python3.5/site-packages/elasticapm/base.py 263 11718 140542526174976 Failed to submit message: '<no message value>'
Traceback (most recent call last):
  File "/var/virtualenv/lib/python3.5/site-packages/elasticapm/transport/http_base.py", line 45, in send_sync
    url = self.sync_transport.send(self, data, headers)
  File "/var/virtualenv/lib/python3.5/site-packages/elasticapm/transport/http.py", line 77, in send
    raise TransportException(message, data, print_trace=print_trace)
elasticapm.transport.base.TransportException: HTTP 400: Problem validating JSON document against schema: unexpected EOF

Agent config:

{'APP_NAME': 'XXX',
 'MAX_EVENT_QUEUE_LENGTH': 150,
 'SERVER_URL': 'XXX',
 'TIMEOUT': 10,
 'TRACES_SEND_FREQ': 15}

apm-server config:

apm-server:
  host: '0.0.0.0:8200'
  max_unzipped_size: 52428800
  read_timeout: '15s'
  write_timeout: '15s'
  shutdown_timeout: '10s'
  concurrent_requests: 20
queue:
  mem:
    events: 16384
    flush.min_events: 512

I've found similar problem here:
elastic/apm-agent-nodejs#141

AFAIK my settings are ok, but issue is still showing up...

PS: Thanks for sharing great piece of software!

Agent produces empty field names

Version 6.0 rc 1

I see fields like this being indexed

"cookies": {
    "Max-Age": "86400",
    "Path": "/",
    "sessionid": "mdoieddd6stjvfm3obprohg8ebam8xpc",
    "expires": "Tue, 07-Nov-2017 12:01:27 GMT",
    "": "HttpOnly",
    "csrftoken": "mvy4wo38fXRwGmPwQn7huElMvDhRn9Hk"
    }

I'm alittle unsure how these are successfully indexed as this object should be rejected. Attempting to index these objects containing this structure results in an error.

IndexError: pop from empty list

elasticapm/traces.py

def end_span(self, skip_frames):
        span = self.span_stack.pop()

span_stack may be empty

IndexError: pop from empty list

[APM-CI][Python] Improve Jenkins builds to support `python nightly`

There is no nightly docker image available at python docker images. Investigate if there are other docker repositories offering python nightly builds, and add to Jenkins test handling if available.

Spans in StreamingHttpResponse._iterator are not captured

Django supports returning a StreamingHttpResponse with an iterator. This iterator will be consumed by the WSGI server (e.g. Gunicorn or uwsgi), crucially after Django finished its handling of the response, including passing through TracingMiddleware. Because starting/stopping of the transaction is done in the TracingMiddleware, spans caused by consuming the iterator in the response will not be captured, e.g.:

def streaming_view(request):
    def my_generator():
        for i in range(5):
            with capture_span('iter', 'code'):
                time.sleep(1)
                yield str(i)
    return StreamingHttpResponse(my_generator())

This will currently result in a transaction with 0 spans and a duration of close to 0, while the real request duration is more than 5 seconds.

Instead of starting/stopping the transaction in the middleware, we could use the request_started / request_finished to start and stop the transaction. request_finished is called when the response is closed, which happens in the WSGI app server.

Unfortunately, these signals provide very little contextual information (e.g. we don't get the request or response object), so we'd still need the middleware to collect this information. Also, the API would need to be changed slightly to allow ending a transaction without providing a name and result (these would also be set in the middleware).

Investigate options to support async (Twisted)

We currently don't support any asynchronous Python frameworks like Twisted, Tornado & Co. Supporting these frameworks is inherently difficult due to the fact that we made a lot of design decisions on the assumption that a thread only handles a single request at a time. This is not true for async frameworks.

There are some ways we could achieve at least some support for Twisted by storing data of the current transaction on the request handler instead of a thread local. Similar workarounds may or may not exist for other async frameworks.

In the future, we may be able to use execution contexts as proposed by PEP 550, but this will take some time until it is available.

Error while capturing exception that contains a extension module stack frame

The following line

apm-agent-python/elasticapm/utils/stacks.py

Line 80 in 6328e3d

pre_context = [line.strip('\r\n') for line in source[lower_bound:lineno]]

raises

TypeError: 'NoneType' object is not subscriptable

When loader.get_source(...) returns None, which in my case it does because the loader is a ExtensionFileLoader that always returns None.

I am running elastic-apm 2.1.1 under Python 3.6

Instrumentation for PyMSSQL

We use SQLAlchemy heavily and would like to instrument it. Might be able to find time to make an implementation myself if no one beats me to it.

https://pypi.org/project/SQLAlchemy/
http://www.sqlalchemy.org/

When running a development server in debug, events are not sent until server is shut down

I don't know if this is intended or not, but when running locally using the builtin runserver in Django, the events seem to not be shipped to the APM server until the runserver process is killed, making for some annoying debugging sessions.

Upon shutting down the development server, it sends everything it has collected up until that point:

PID 13776: ElasticAPM is attempting to send 1 pending messages
Waiting up to 10 seconds, press Ctrl-C to quit.
(webshop.env)
# henrik at Henriks-MacBook-Pro.local in ~/Development/webshop on git:SHOP-6355 ● [12:01:47]
→ PID 13776: done, took 0.97 seconds to complete.

What I would have expected was for events to be shipped after a certain threshold, given any events was collected.

Add support for distributed tracing

Work is underway to add support for distributed tracing to the API/server: elastic/apm-server#913 / elastic/apm-server#968.
Support needs to be added to this agent too.

• intake protocol v2 (#281)
• traceparent header extract/inject functionality
• update opbeans-python to include a distributed tracing call
• transaction.span_count.total (elastic/apm-server#1241)
• auto-instrument http module(s) to use extact/inject functionality

Problem validating JSON document against schema - length must be <= 1024, but got 1523

Continuously seeing these errors in the log -

2018-02-13T05:06:49.104Z ERROR [request] beater/handlers.go:384 error handling request {"request_id": "79f54c15-f99e-4d1d-ab14-07bf622983c8", "error": "Problem validating JSON document against schema: I[#] S[#] doesn't validate with "transaction#"\n I[#/transactions/29/context/request/url/full] S[#/properties/transactions/items/properties/context/properties/request/properties/url/properties/full/maxLength] length must be <= 1024, but got 1523"}

apm-server version is 6.2.1
elastic-apm agent version 2.0.0 with Flask.

Is there any config property that I need to change to let the apm server accept urls of a longer length?

Create OpenTracing API compatibility bridge

Implement lightweight OpenTracing API compatibility bridge.

Consider using new URL syntax in Django 2.0 as transaction name

Since the Opbeat days, we use the view name as the transaction name for Django, while in Flask we use the parametrized URL. The main reason for this was that Django used unwieldy regex in its URLs, which would make the transaction names somewhat ugly.

In Django 2.0, a new way for defining URLs was added and made the default, URL paths. These quite closely mirror Flask.

As such, it might be interesting to use these paths as transaction name for Django to align both Flask and Django.

Benefits of using view name

• as a user, it's easier to find the actual code if you don't have to look up the view in your URLs first (and Django URL patterns tend to be deeply nested, making it even more difficult to look up the view name). This could be alleviated somewhat by putting the view name in the context or using it as a tag
• less "weird" characters in the transaction name

Benefits of using path name

• aligned with Flask and AFAIK, Node
• multiple paths can point to the same view (or view class). Using the path would keep them separate. This is especially true for e.g. Django Rest Framework or tastypie. If a) we used paths and b) those frameworks would use paths instead of regexes, we wouldn't need to do any magic like https://gist.github.com/beniwohli/b494bd4b368ea4095dad3b629333ae58

No message sending to APM server

• I'm trying simple Elastic APM example with python flask agent on local. I noticed that there're several logs saying that APM threads are sending messages to server after I hit Ctrl + C. However, I checked kibana and elastic cluster, there're no documents.

PID 8745: ElasticAPM is attempting to send 1 pending messages Waiting up to 10 seconds, press Ctrl-C to quit. PID 8745: done, took 9.90 seconds to complete.

My settings:
{'MAX_EVENT_QUEUE_LENGTH': 500, 'TIMEOUT': 5, 'DISABLE_SEND': False, 'SECRET_TOKEN': '0123456789abcdefghijklmnopqrstxyz', 'DEBUG': False, 'SERVER_URL': 'http://localhost:8200/', 'APP_NAME': ''}
Flask debug set to False.

Code to send message:

apm.capture_message('hello elastic apm')

<lambda>() missing 1 required positional argument: 'environ' on elastic-apm==2.2.0

Just updated elasticapm to version 2.2.0 and ran into this issue:

2018-06-13 17:50:28,893 - ERROR - server - Exception inside application: <lambda>() missing 1 required positional argument: 'environ'
  File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
    result = coro.throw(exc)
  File "/var/www/brownpapersession/dev/env/lib/python3.5/site-packages/channels/http.py", line 190, in __call__
    await self.handle(body)
  File "/var/www/brownpapersession/dev/env/lib/python3.5/site-packages/asgiref/sync.py", line 108, in __call__
    return await asyncio.wait_for(future, timeout=None)
  File "/usr/lib/python3.5/asyncio/tasks.py", line 381, in wait_for
    return (yield from fut)
  File "/usr/lib/python3.5/asyncio/futures.py", line 380, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 304, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result
    raise self._exception
  File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/var/www/brownpapersession/dev/env/lib/python3.5/site-packages/asgiref/sync.py", line 123, in thread_handler
    return self.func(*args, **kwargs)
  File "/var/www/brownpapersession/dev/env/lib/python3.5/site-packages/channels/http.py", line 200, in handle
    signals.request_started.send(sender=self.__class__, scope=self.scope)
  File "/var/www/brownpapersession/dev/env/lib/python3.5/site-packages/django/dispatch/dispatcher.py", line 178, in send
    for receiver in self._live_receivers(sender)
  File "/var/www/brownpapersession/dev/env/lib/python3.5/site-packages/django/dispatch/dispatcher.py", line 178, in <listcomp>
    for receiver in self._live_receivers(sender)
  <lambda>() missing 1 required positional argument: 'environ'

Some of the python packages I'm using on Python 3.5.3:

Django==2.0.6
celery==4.2.0
certifi==2018.4.16
cffi==1.11.5
channels==2.1.1
channels-redis==2.2.1
chardet==3.0.4
constantly==15.1.0
cryptography==2.2.2
csscompressor==0.9.5
daphne==2.1.2
redis==2.10.6
requests==2.19.0

The issue is not here when I revert to elastic-apm==2.1.1. Will try to do some digging

Add frame context line settings paragraph to Tuning & Overhead doc

Collecting frame context lines is quite expensive, especially for spans, we should mention the settings to configure the amount of collected lines in the Tuning & Overhead docs

`check` cmd output differs from documentation.

Running python manage.py elasticapm check returns an error if no Secret_Token is configured. Documentation states that configuration is optional.

Possible memory-leak

We are running 2 flask apps serving APIs for two separate projects.
Last two days we started getting alerts from monitoring systems on high memory usage on nodes running these apps.

We spent some time reviewing shared components and recent changes for potential cause until the only one left was elastic-apm module. API for one of the apps is normally running on 3 nodes and the traffic is load balanced via Amazon's Application Load Balancer, so every node receives a fair share. We disabled APM via the config on one of nodes and restarted gunicorns on all three. After 48 hours running we see the following:

• app-01 node without elastic-apm: 3.4% to 5.4% node memory usage (across multiple gunicorn processes)
• app-02 node with elastic-apm: 12.9% ... 16.6% memory usage
• app-03 node with elastic-apm: 11.6% ... 18.9% memory usage

In absolute numbers, it is ~200Mb on app-01 vs 600-700Mb on 02 and 03.
All 3 nodes are brought up with orchestration tools, so identical in hardware and software and serve identical workload.

Due to the nature of the API, on weekends we do have quite low usage, which probably explains why after 48 hours we don't yet have alerts firing.

Versions installed:

Description:	Ubuntu 16.04.3 LTS

Flask==0.10.1
elastic-apm==2.1.1
gunicorn==19.4.1

Initialization:

    app = Flask(__name__, ...)
    ...
    # elastic apm
    app.apm = None
    if app.config.get('elastic', {}).get('service_name'):
        from elasticapm.contrib.flask import ElasticAPM
        app.apm = ElasticAPM(app, **app.config['elastic'])

Config:

[elastic]
service_name = "Production Application API"
server_url = "{{ .Data.server_url }}"
secret_token = "{{ .Data.secret_token }}"
environment = "production"

Please suggest how we can help investigating this issue. Thanks!

Implement order of precedence for configuration

As per elastic/apm-agent-nodejs#287

TypeError: write() argument must be str, not bytes

Here message pass to to_string and then to sys.stderr.write()
to_string – returns bytes
sys.stderr.write receives str

TypeError: write() argument must be str, not bytes:

Configuring restricted APP_NAME pattern leads to 500 during runtime.

I configured a restricted pattern for the APP_NAME containing an * leading to a Server Error with following exception. This prevents a Django app from working properly when the APP_NAME is misconfigured.

Unable to process log entry: 'DjangoClient' object has no attribute 'filter_exception_types_dict'
Traceback (most recent call last):
  File "/Users/simitt/.pyenv/versions/django-test-3.6.2/lib/python3.6/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/Users/simitt/.pyenv/versions/django-test-3.6.2/lib/python3.6/site-packages/django/utils/deprecation.py", line 138, in __call__
    response = self.process_request(request)
  File "/Users/simitt/.pyenv/versions/django-test-3.6.2/lib/python3.6/site-packages/elasticapm/contrib/django/middleware/__init__.py", line 157, in process_request
    self.client.begin_transaction("request")
  File "/Users/simitt/.pyenv/versions/django-test-3.6.2/lib/python3.6/site-packages/elasticapm/base.py", line 442, in begin_transaction
    self.instrumentation_store.begin_transaction(transaction_type)
AttributeError: 'DjangoClient' object has no attribute 'instrumentation_store'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/simitt/.pyenv/versions/django-test-3.6.2/lib/python3.6/site-packages/elasticapm/contrib/django/handlers.py", line 59, in actually_do_stuff
    client.capture('Exception', exc_info=exc_info, request=request)
  File "/Users/simitt/.pyenv/versions/django-test-3.6.2/lib/python3.6/site-packages/elasticapm/contrib/django/client.py", line 200, in capture
    result = super(DjangoClient, self).capture(event_type, **kwargs)
  File "/Users/simitt/.pyenv/versions/django-test-3.6.2/lib/python3.6/site-packages/elasticapm/base.py", line 279, in capture
    data = self.build_msg_for_logging(event_type, data, date, extra, stack, **kwargs)
  File "/Users/simitt/.pyenv/versions/django-test-3.6.2/lib/python3.6/site-packages/elasticapm/base.py", line 178, in build_msg_for_logging
    if self._filter_exception_type(result):
  File "/Users/simitt/.pyenv/versions/django-test-3.6.2/lib/python3.6/site-packages/elasticapm/base.py", line 332, in _filter_exception_type
    if exc_type in self.filter_exception_types_dict:
AttributeError: 'DjangoClient' object has no attribute 'filter_exception_types_dict'

I am testing with Django-1.11.5 and python 3.6.2.

Add additional OS for test runs on Jenkins

Enhance Jenkins builds to also support tests running on Windows and Mac.

Document which modules get auto-instrumented

We should update the documentation to help users find out if the will agent auto-instrument the modules they are using.

Create compatibility documentation to outline supported versions of Python/Django/Flask/apm-server etc

Currently, compatibility information is sprinkled all over the place, often outdated, and sometimes missing. We should have a compatibility page in our docs similar to the Node agent that outlines supported versions of

• Python
• Django
• Flask
• Elastic Stack

Possibly, we could also add compatibility information on specific instrumentations (e.g. we don't support elasticsearch < 2.0 in our instrumentation).

Consider using patterns for exception filter

We have two settings to filter events: filter_exception_types and transactions_ignore_patterns. The former uses exact string matching on the class/module name of the exception, the latter uses a regex match against the transaction name.

This is somewhat confusing. Ideally, we use regex for both filters.

Change the default port to 8200 to improve the getting started experience

apm-agent-python/elasticapm/conf/defaults.py

Line 24 in e2c4374

SERVERS = ['http://localhost:8080']

APM server not getting events from Django

I setup the apm agent in my django app, verified that the configs are correct but I'm not getting any transactions saved in elastic.

I only get transactions if there's an error (404, 500). The python manage.py elasticapm test sends the event to apm server too.

My Django version is 1.9.7

Thanks!

Bug when sending message contains formatter

This code on events.py (line 144) will introduce error not enough arguments for format string if I send normal messages that contain string formatter:

@staticmethod def capture(client, param_message=None, message=None, level=None, logger_name=None, **kwargs): if message: param_message = {'message': message} params = param_message.get('params', ()) message = param_message['message'] % params data = kwargs.get('data', {})

for example: apm.capture_message('abc%def') which usually happens when I capture the http request. My suggestion is not to force normal message to become param message when param message is None.

Implement pyodbc instrumentation

https://pypi.org/project/pyodbc/
https://github.com/mkleehammer/pyodbc

apm client is unable to connect to the apm server when the user attempts to use basic authentication in the server_url

I tried to connect to the apm-server using server_url with basic authentication, but that fails. I read that's because I'm using a secret token, which is passed along via the HTTP Authorization header. It would be nice if that case would be caught by the elasticapm check management command.

I'm using elastic apm on Django 2.0.6 and it appears to work properly.

PR coming up

IT is open APM Elastic

hi ,
i want install ELastic APM on python application . it is open source ?

Not getting any events on the apm server

The elasticapm check command fails with following error.

$ python manage.py elasticapm check
Service name and secret token are set, good job!

DEBUG mode is disabled! Looking good!

Traceback (most recent call last):
  File "manage.py", line 12, in <module>
    execute_from_command_line(sys.argv)
  File "/home/ubuntu/.virtualenvs/backend/lib/python3.5/site-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
    utility.execute()
  File "/home/ubuntu/.virtualenvs/backend/lib/python3.5/site-packages/django/core/management/__init__.py", line 355, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/ubuntu/.virtualenvs/backend/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/ubuntu/.virtualenvs/backend/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/home/ubuntu/.virtualenvs/backend/lib/python3.5/site-packages/elasticapm/contrib/django/management/commands/elasticapm.py", line 104, in handle
    )(self, subcommand, **options)
  File "/home/ubuntu/.virtualenvs/backend/lib/python3.5/site-packages/elasticapm/contrib/django/management/commands/elasticapm.py", line 207, in handle_check
    middleware = list(getattr(settings, 'MIDDLEWARE', getattr(settings, 'MIDDLEWARE_CLASSES', [])))
TypeError: 'NoneType' object is not iterable
Sentry is attempting to send 1 pending error messages
Waiting up to 10 seconds
Press Ctrl-C to quit

OPTIONS Request in Flask uses content-type of None

OPTIONS requests (CORS Preflight) are checked for content-type on the request object in the flask contrib module (elasticapm/contrib/flask/utils.py):

    if request.method not in ('GET', 'HEAD'):
        body = None
        if request.content_type == 'application/x-www-form-urlencoded':
            body = compat.multidict_to_dict(request.form)
        elif request.content_type.startswith('multipart/form-data'):
            body = compat.multidict_to_dict(request.form)

However an OPTIONS requests does not have a content_type by default which causes the startswith() method to be called on a NoneType resulting in a crash as this exception is unhandled.

Problem validating JSON document against schema: I[#] S[#] doesn't validate with "transaction#" I[#/service/name] S[#/properties/service/properties/name/type] expected string, but got null

I'm trying to get started with elastic-apm[flask] (version 2.1.1, on Python 3.6.5) and apm-server (6.2.4) running in docker.

My virtualenv after running, pip install Flask elastic-apm[flask] looks like:

blinker==1.4
certifi==2018.4.16
click==6.7
elastic-apm==2.1.1
Flask==1.0.2
itsdangerous==0.24
Jinja2==2.10
MarkupSafe==1.0
pyspark==2.2.1
urllib3==1.23
Werkzeug==0.14.1

My docker-compose.yml:

version: '3.6'
services:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.2.4

  kibana:
    image: docker.elastic.co/kibana/kibana:6.2.4
    depends_on:
    - elasticsearch
    ports:
    - '5601:5601'

  apm-server:
    image: docker.elastic.co/apm/apm-server:6.2.4
    depends_on:
    - elasticsearch
    ports:
    - '8200:8200'

Kibana apm dashboard shows status of apm-server is green.

from elasticapm.contrib.flask import ElasticAPM
from flask import Flask

app = Flask(__name__)
apm = ElasticAPM(app)

@app.route("/")
def hello():
    return "Hello World!

I use the following command to start hello.py

env FLASK_APP=hello.py flask run

When accessing doing curl localhost:5000, I see "Hello, world!" printed, but the following error occurs:

127.0.0.1 - - [08/Jun/2018 16:27:46] "GET / HTTP/1.1" 200 -
HTTP 400: Problem validating JSON document against schema: I[#] S[#] doesn't validate with "transaction#"
  I[#/service/name] S[#/properties/service/properties/name/type] expected string, but got null
Failed to submit message: '<no message value>'
Traceback (most recent call last):
  File "/Users/ksmets/.virtualenvs/elastic-apm-flask/lib/python3.6/site-packages/elasticapm/transport/http_base.py", line 45, in send_sync
    url = self.sync_transport.send(self, data, headers)
  File "/Users/ksmets/.virtualenvs/elastic-apm-flask/lib/python3.6/site-packages/elasticapm/transport/http.py", line 77, in send
    raise TransportException(message, data, print_trace=print_trace)
elasticapm.transport.base.TransportException: HTTP 400: Problem validating JSON document against schema: I[#] S[#] doesn't validate with "transaction#"
  I[#/service/name] S[#/properties/service/properties/name/type] expected string, but got null

apm-server_1     | 2018-06-08T14:43:46.346Z	ERROR	[request]	beater/handlers.go:384	error handling request	{"request_id": "b448367c-c79b-46cf-8e16-9a9fb08726c1", "error": "Problem validating JSON document against schema: I[#] S[#] doesn't validate with \"transaction#\"\n  I[#/service/name] S[#/properties/service/properties/name/type] expected string, but got null"}
apm-server_1     | 2018-06-08T14:43:46.346Z	INFO	[request]	beater/handlers.go:150	handled request	{"request_id": "b448367c-c79b-46cf-8e16-9a9fb08726c1", "response_code": 400, "method": "POST", "URL": "/v1/transactions", "content_length": 676, "remote_address": "172.20.0.1", "user_agent": "elasticapm-python/2.1.1"

Kibana apm dashboard shows no apm data is flowing in.

Any ideas?

threading exception under flask

Per elastic/apm-integration-testing/#73, the agent is apparently trying to join a thread that it shouldn't under flask.

env:

# pip freeze
blinker==1.4
certifi==2018.4.16
click==6.7
elastic-apm==2.1.1
Flask==1.0.2
gunicorn==19.8.1
itsdangerous==0.24
Jinja2==2.10
MarkupSafe==1.0
urllib3==1.23
Werkzeug==0.14.1

flask says:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2309, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2295, in wsgi_app
    response = self.handle_exception(e)
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1741, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1816, in full_dispatch_request
    return self.finalize_request(rv)
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1834, in finalize_request
    request_finished.send(self, response=response)
  File "/usr/local/lib/python3.6/site-packages/blinker/base.py", line 267, in send
    for receiver in self.receivers_for(sender)]
  File "/usr/local/lib/python3.6/site-packages/blinker/base.py", line 267, in <listcomp>
    for receiver in self.receivers_for(sender)]
  File "/usr/local/lib/python3.6/site-packages/elasticapm/contrib/flask/__init__.py", line 151, in request_finished
    self.client.end_transaction(rule, result)
  File "/usr/local/lib/python3.6/site-packages/elasticapm/base.py", line 257, in end_transaction
    self._collect_transactions()
  File "/usr/local/lib/python3.6/site-packages/elasticapm/base.py", line 297, in _collect_transactions
    self._stop_send_timer()
  File "/usr/local/lib/python3.6/site-packages/elasticapm/base.py", line 324, in _stop_send_timer
    self._send_timer.join()
  File "/usr/local/lib/python3.6/threading.py", line 1051, in join
    raise RuntimeError("cannot join thread before it is started")
RuntimeError: cannot join thread before it is started

Reproduceable using elastic/apm-integration-testing#73 (comment)

apm-agent attempts to unpack localhost URLs with service/region via botocore

When running locally (localhost:13579), it for some reason attempts to utilize botocore and try to unpack the URL into
service, region, _ = parsed_url.hostname.split('.', 2)

Naturally this fails since the hostname is localhost. I'm not sure why it's using botocore here, or if it should just be defensive against accepting localhost as a valid param

Refactor serialization mechanism to be extendable

At the moment, our serialization code is just a really long if-elif-elif-... block:

apm-agent-python/elasticapm/utils/encoding.py

Lines 85 to 138 in ee45d21

	def transform(value, stack=None, context=None):
	# TODO: make this extendable
	if context is None:
	context = {}
	if stack is None:
	stack = []
	objid = id(value)
	if objid in context:
	return '<...>'
	context[objid] = 1
	transform_rec = lambda o: transform(o, stack + [value], context)
	if any(value is s for s in stack):
	ret = 'cycle'
	elif isinstance(value, (tuple, list, set, frozenset)):
	try:
	ret = type(value)(transform_rec(o) for o in value)
	except Exception:
	# We may be dealing with a namedtuple
	class value_type(list):
	__name__ = type(value).__name__
	ret = value_type(transform_rec(o) for o in value)
	elif isinstance(value, uuid.UUID):
	ret = repr(value)
	elif isinstance(value, dict):
	ret = dict((to_unicode(k), transform_rec(v)) for k, v in compat.iteritems(value))
	elif isinstance(value, compat.text_type):
	ret = to_unicode(value)
	elif isinstance(value, compat.binary_type):
	ret = to_string(value)
	elif not isinstance(value, compat.class_types) and \
	_has_elasticapm_metadata(value):
	ret = transform_rec(value.__elasticapm__())
	elif isinstance(value, bool):
	ret = bool(value)
	elif isinstance(value, float):
	ret = float(value)
	elif isinstance(value, int):
	ret = int(value)
	elif compat.PY2 and isinstance(value, long): # noqa F821
	ret = long(value) # noqa F821
	elif value is not None:
	try:
	ret = transform(repr(value))
	except Exception:
	# It's common case that a model's __unicode__ definition may try to query the database
	# which if it was not cleaned up correctly, would hit a transaction aborted exception
	ret = u'<BadRepr: %s>' % type(value)
	else:
	ret = None
	del context[objid]
	return ret

.

This has several drawbacks:

• it's not extendable (we e.g. might want to serialize Django QuerySet objects in a sane way that doesn't evaluate the queryset unnecessarily, but only if Django is installed)
• it's brittle
• it's hard to read
• ideally, serialization would also include shortening of values, which is hard to do in its current state

One option could be to have a registry, to which transformers can register themselves, together with a list of supported types for that transformer.

Meta Issue to collect Improvement Ideas for Jenkins CI

This is a Meta Issue to collect ideas and improvement suggestions for further Improvements regarding Jenkins setup.

• Test on different OS #21
• Test with Python nightly build #20
• Slack Notifications on every build
• Email Notifications on failure