Coder Social home page Coder Social logo

mozilla-services / python-dockerflow Goto Github PK

View Code? Open in Web Editor NEW
38.0 12.0 21.0 344 KB

A Python package to implement tools and helpers for Mozilla Dockerflow

Home Page: https://python-dockerflow.readthedocs.io

License: Mozilla Public License 2.0

Python 99.53% Mako 0.47%

python-dockerflow's Introduction

Python Dockerflow

This package implements a few helpers and tools for Mozilla's Dockerflow pattern. The documentation can be found on python-dockerflow.readthedocs.io

GitHub Actions Codecov Documentation Status CalVer - Timely Software Versioning

Installation

Please install the package using your favorite package installer:

pip install dockerflow

Issues & questions

See the issue tracker on GitHub to open tickets if you have issues or questions about python-dockerflow.

python-dockerflow's People

Contributors

archaeopteryx avatar diox avatar djmitche avatar g-k avatar grahamalama avatar groovecoder avatar jezdez avatar jwhitlock avatar kernicpanel avatar leplatrem avatar mozilla-github-standards avatar relud avatar robotblake avatar tomprince avatar willkg avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

python-dockerflow's Issues

ModuleNotFoundError: No module named 'aioredis.client' when running sanic tests

  __________________ ERROR collecting tests/sanic/test_sanic.py __________________
  ImportError while importing test module '/home/runner/work/python-dockerflow/python-dockerflow/tests/sanic/test_sanic.py'.
  Hint: make sure your test modules/packages have valid Python names.
  Traceback:
  /opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/importlib/__init__.py:127: in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
  tests/sanic/test_sanic.py:11: in <module>
      import sanic_redis.core
  .tox/py39-s20/lib/python3.9/site-packages/sanic_redis/__init__.py:1: in <module>
      from .core import SanicRedis
  .tox/py39-s20/lib/python3.9/site-packages/sanic_redis/core.py:1: in <module>
      import aioredis.client
  E   ModuleNotFoundError: No module named 'aioredis.client'

Seen in https://github.com/mozilla-services/python-dockerflow/runs/4717358686?check_suite_focus=true for instance, which contains completely unrelated changes.

This is because sanic-redis 0.3.0 was released, depending on aioredis 2.x, which we have restricted to 1.x. Medium to long term we should probably change this to support aioredis 2.x, but in the short term, to unblock the situation, restrict sanis-redis to 0.2.x in tests.

Flask 0.10 tests fail

   ______________________ test_preserves_existing_request_id ______________________
  Traceback (most recent call last):
    File "/home/runner/work/python-dockerflow/python-dockerflow/tests/flask/test_flask.py", line 246, in test_preserves_existing_request_id
      assert getattr(g, "_request_id") is not None
    File "/home/runner/work/python-dockerflow/python-dockerflow/.tox/py39-fl10/lib/python3.9/site-packages/werkzeug/local.py", line 410, in __get__
      obj = instance._get_current_object()
    File "/home/runner/work/python-dockerflow/python-dockerflow/.tox/py39-fl10/lib/python3.9/site-packages/werkzeug/local.py", line 530, in _get_current_object
      return self.__local()  # type: ignore
    File "/home/runner/work/python-dockerflow/python-dockerflow/.tox/py39-fl10/lib/python3.9/site-packages/flask/globals.py", line 45, in _lookup_app_object
      raise RuntimeError(_app_ctx_err_msg)
  RuntimeError: Working outside of application context.
  
  This typically means that you attempted to use functionality that needed
  to interface with the current application object in some way. To solve
  this, set up an application context with app.app_context().  See the
  documentation for more information.

I'm guessing this is a newer version of werkzeug, as that is unconstrained. I'm also guessing that maybe we could drop this support? According to https://github.com/pallets/flask/releases?after=0.12.4 0.10 was last released in 2013.

dockerflow.logging.JsonLogFormatter is weird about %s style logging

I have this config:

    # Logging
    LOGGING_USE_JSON = values.BooleanValue(True)
    LOGGING_DEFAULT_LEVEL = values.Value("INFO")

    @property
    def LOGGING(self):
        return {
            "version": 1,
            "disable_existing_loggers": False,
            "formatters": {
                "json": {
                    "()": "dockerflow.logging.JsonLogFormatter",
                    "logger_name": "buildhub",
                },
                "verbose": {"format": "%(levelname)s %(asctime)s %(name)s %(message)s"},
            },
            "handlers": {
                "console": {
                    "level": self.LOGGING_DEFAULT_LEVEL,
                    "class": "logging.StreamHandler",
                    "formatter": ("json" if self.LOGGING_USE_JSON else "verbose"),
                },
                "sentry": {
                    "level": "ERROR",
                    "class": (
                        "raven.contrib.django.raven_compat.handlers" ".SentryHandler"
                    ),
                },
                "null": {"class": "logging.NullHandler"},
            },
            "root": {"level": "INFO", "handlers": ["sentry", "console"]},

...

When I do:

In [1]: import logging

In [2]: logger = logging.getLogger("myproject")

In [3]: logger.info("Hi!")
{"Timestamp": 1541787363833421056, "Type": "myproject", "Logger": "buildhub", "Hostname": "MacBook-Pro-131.local", "EnvVersion": "2.0", "Severity": 6, "Pid": 68488, "Fields": {"msg": "Hi!"}}

In [4]: logger.info("Hi %s", "peter")
{"Timestamp": 1541787379742094080, "Type": "myproject", "Logger": "buildhub", "Hostname": "MacBook-Pro-131.local", "EnvVersion": "2.0", "Severity": 6, "Pid": 68488, "Fields": {"msg": "Hi peter"}}

In [5]: logger.info("Hi %s", {'name': 'Peter'})
{"Timestamp": 1541787416647823104, "Type": "myproject", "Logger": "buildhub", "Hostname": "MacBook-Pro-131.local", "EnvVersion": "2.0", "Severity": 6, "Pid": 68488, "Fields": {}}

Notice that when the param to logger.debug(template, THIS) isn't a string the final "msg" becomes empty ("Fields": {}).

Support Sanic 21 and later

Sanic 21.12 will require changes in tests, and possibly in code.

Sanic 21.3 moved testing support from sanic.testing to a new module sanic-testing. This will require a different requirements file than Sanic 20.x to install sanic-testing.

There are other changes needed to support sanic-testing, such as changes to app registration and the test client. The changes are large enough that a new test file may be useful, rather than dynamically changing code based on the version.

There may be other changes in the framework itself. For example, this test:

dockerflow = Dockerflow()
assert "dockerflow.heartbeat" not in app.router.routes_names

fails with AttributeError: 'Router' object has no attribute 'routes_names'.

Currently, tests pass for Sanic 20.12.3, which is supported until December 2022. After this date, if we don't support Sanic 21.12, then we may want to drop Sanic support entirely.

Documentation build broken (api/flask with Sphinx v3.0.2)

The documentation build (tox -e py38-docs) fails when processing api/flask:

py38-docs run-test: commands[0] | sphinx-build -b html -d /Users/john/src/python-dockerflow/.tox/py38-docs/tmp/doctrees docs /Users/john/src/python-dockerflow/.tox/py38-docs/tmp/html
Running Sphinx v3.0.2
making output directory... done
loading intersphinx inventory from http://docs.djangoproject.com/en/stable/_objects/...
loading intersphinx inventory from http://flask.pocoo.org/docs/objects.inv...
loading intersphinx inventory from https://redis-py.readthedocs.io/en/latest/objects.inv...
loading intersphinx inventory from https://flask-sqlalchemy.readthedocs.io/en/2.x/objects.inv...
loading intersphinx inventory from https://docs.python.org/objects.inv...
loading intersphinx inventory from https://whitenoise.readthedocs.io/en/stable/objects.inv...
loading intersphinx inventory from https://sanic.readthedocs.io/en/stable/objects.inv...
intersphinx inventory has moved: http://docs.djangoproject.com/en/stable/_objects/ -> http://docs.djangoproject.com/en/3.0/_objects/
intersphinx inventory has moved: https://docs.python.org/objects.inv -> https://docs.python.org/3/objects.inv
intersphinx inventory has moved: http://flask.pocoo.org/docs/objects.inv -> https://flask.palletsprojects.com/en/1.1.x/objects.inv
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 13 source files that are out of date
updating environment: [new config] 13 added, 0 changed, 0 removed
reading sources... [  7%] api/django
reading sources... [ 15%] api/flask

Exception occurred:
  File "/Users/john/src/python-dockerflow/.tox/py38-docs/lib/python3.8/site-packages/flask/globals.py", line 52, in _find_app
    raise RuntimeError(_app_ctx_err_msg)
RuntimeError: Working outside of application context.

This typically means that you attempted to use functionality that needed
to interface with the current application object in some way. To solve
this, set up an application context with app.app_context().  See the
documentation for more information.
The full traceback has been saved in /var/folders/61/s6_xxhqd3nl27_vgq9fzjmkr0000gq/T/sphinx-err-yu9dmo2o.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
ERROR: InvocationError for command /Users/john/src/python-dockerflow/.tox/py38-docs/bin/sphinx-build -b html -d /Users/john/src/python-dockerflow/.tox/py38-docs/tmp/doctrees docs /Users/john/src/python-dockerflow/.tox/py38-docs/tmp/html (exited with code 2)

This appears to be an issue with automodule that can be worked around by using autoclass directly.

Django 2.0 Support

From my attempt to update to Django 2.0:

File "/usr/local/lib/python3.6/site-packages/dockerflow/django/middleware.py", line 52, in _build_extra_meta
out['uid'] = (request.user.is_authenticated() and
TypeError: 'bool' object is not callable

Use time.time() to make a timestamp

The middleware adds a timestamp to the request which tells how long the request took.
It's currently doing this by creating two datetime.datetime.utcnow() instances.
It's much cheaper to use time.time() to make timestamps.

Deprecation warning on `request.user.is_authenticated()`

As of Django 1.10, user.is_authenticated is a property. Not a method. https://docs.djangoproject.com/en/1.11/ref/contrib/auth/#django.contrib.auth.models.User.is_authenticated

So you get these warnings in Django 1.11:

  /usr/local/lib/python3.5/site-packages/dockerflow/django/middleware.py:52: RemovedInDjango20Warning: Using user.is_authenticated() and user.is_anonymous() as a method is deprecated. Remove the parentheses to use it as an attribute.
    out['uid'] = (request.user.is_authenticated() and

In django-session-csrf we did this hack: mozilla/django-session-csrf@88d1b14

Perhaps there are better ways to deal with it.

dockerflow.logging.JsonLogFormatter should handle dates better

#8 fixes the exception that can happen when you log objects. What'd be even nicer, as @jezdez pointed out, would be to handle datetimes/dates : currently they are serialized as "datetime.datetime(2017, 4, 6, 9, 27, 7, 479427)" because that's the repr() for a datetime object, but we could do better by using DjangoJSONEncoder, which knows how to handle those objects and more.

Something like this should work, to test:

class SafeJSONEncoder(json.JSONEncoder):
    def default(self, o):
        return repr(o)

class SafeDjangoJSONEncoder(DjangoJSONEncoder, SafeJSONEncoder):
    pass

Then use cls=SafeDjangoJSONEncoder when doing the json.dumps.

Deal with exc_info not being a tuple while not being None either in JsonLogFormatter

JsonLogFormatter does:

# If there is an error, format it for nice output.
if record.exc_info is not None:
fields["error"] = repr(record.exc_info[1])
fields["traceback"] = safer_format_traceback(*record.exc_info)

However, python logging docs say:

If exc_info does not evaluate as false, it causes exception information to be added to the logging message. If an exception tuple (in the format returned by sys.exc_info()) or an exception instance is provided, it is used; otherwise, sys.exc_info() is called to get the exception information.

The difference is notable, because there is code out there that calls log.something(..., exc_info=False). In AMO, which has a logger inheriting from JsonLogFormatter, that causes the following traceback:

  File "/usr/local/lib/python3.6/logging/__init__.py", line 994, in emit
    msg = self.format(record)
  File "/usr/local/lib/python3.6/logging/__init__.py", line 840, in format
    return fmt.format(record)
  File "/deps/lib/python3.6/site-packages/dockerflow/logging.py", line 138, in format
    out = self.convert_record(record)
  File "/code/src/olympia/core/logger.py", line 58, in convert_record
    out = super().convert_record(record)
  File "/deps/lib/python3.6/site-packages/dockerflow/logging.py", line 127, in convert_record
    fields["error"] = repr(record.exc_info[1])
TypeError: 'bool' object is not subscriptable

Document how to add app-specific Fields

The mozlog format allows for a Fields hash of application-specific fields. This is a useful feature for structured logging, and it should be documented how to access it.

As discovered in mozilla.fx-private-relay, this can be populated through the extra parameter:

logger.info(
    "Subsystem foo running at foo.example.com:4021",
    extra: {"phase": "started", "host": "foo.example.com", "port": 4021})

aioredis has changed and no longer works in CI

[2947] /home/runner/work/python-dockerflow/python-dockerflow$ /home/runner/work/python-dockerflow/python-dockerflow/.tox/py37-s20/bin/python -m pip freeze >.tox/py37-s20/log/py37-s20-3.log
py37-s20 installed: aiofiles==0.6.0,aiohttp==4.0.0a1,aioredis==2.0.0a1,async-timeout==3.0.1,attrs==20.3.0,certifi==2020.12.5,chardet==3.0.4,coverage==5.6b1,-e git+https://github.com/mozilla-services/python-dockerflow@0878ba3264b10636c068eb6b5d189780d6ff3adf#egg=dockerflow,fakeredis==1.5.0,h11==0.9.0,httpcore==0.11.1,httptools==0.1.1,httpx==0.15.4,idna==3.1,importlib-metadata==4.0.1,iniconfig==1.1.1,jsonschema==3.2.0,mock==4.0.3,multidict==4.7.6,packaging==20.9,pluggy==1.0.0.dev0,py==1.10.0,pyparsing==3.0.0b2,pyrsistent==0.17.3,pytest==6.2.3,pytest-cov==2.11.1,pytest-cover==3.0.0,pytest-coverage==0.0,pytest-mock==3.5.1,pytest-pythonpath==0.7.3,redis==3.1.0,rfc3986==1.4.0,sanic==20.9.0,sanic-redis==0.2.0,six==1.15.0,sniffio==1.2.0,sortedcontainers==2.3.0,toml==0.10.2,typing-extensions==3.7.4.3,ujson==4.0.2,uvloop==0.15.2,websockets==8.1,yarl==1.6.3,zipp==3.4.1
  E   ImportError: cannot import name 'create_redis_pool' from 'aioredis' (/home/runner/work/python-dockerflow/python-dockerflow/.tox/py37-s20/lib/python3.7/site-packages/aioredis/__init__.py)

JsonLogFormatter unexpectedly squelches message(s) from Django?

See https://github.com/mozilla/fx-private-relay/pull/82/files#r395097199 ... not sure why this code:

if message and not self.is_value_jsonlike(message):
fields["msg"] = message

is skipping log records where messages are already json-like? This means code like the following:

     logger.info({
         'event_type': 'email_relay',
         'relay_address_id': relay_address.id,
         'relay_address': sha256(local_portion.encode('utf-8')).hexdigest(),
         'real_address': sha256(
             relay_address.user.email.encode('utf-8')
         ).hexdigest(),
     })

Omits the dictionary from the output. So this outputs:

{"Timestamp": 1584647678788551936, "Type": "events", "Logger": "fx-private-relay", "Hostname": "groovetop", "EnvVersion": "2.0", "Severity": 6, "Pid": 4031, "Fields": {}}

?

Which makes no sense to me. Is there a different way I should be logging JSON-like objects via dockerflow.

Document developer environment setup

This was my first time using tox, so it took me a while to get up to speed to start making changes. It would be nice to have some documentation for "here's how you get a development environment set up, make a change, and run the tests". Even better if it was in a Makefile or bash script.

dockerflow.logging.JsonLogFormatter doesn't handle extra arguments that are not json-serializable

Enabling dockerflow.logging.JsonLogFormatter in your project can cause TypeError exceptions when extra, non json-serializable arguments are passed to the logging call.

>>> from logging import getLogger
>>> getLogger('foo').warning('warning with extra info', extra={'something': object()})
Traceback (most recent call last):
(...)
TypeError: <object object at 0x7f5e88a54630> is not JSON serializable

It's a perfect valid logging call, you might have other formatters configured taking advantage of the extra data. Django does that in some of its core, for instance passing request objects when logging 404 or 500 request errors.

I think the best approach here would be to have a custom serializer class that would just use repr() on data it can not serialize.

access request id in Flask view?

I want to include the request ID to add to error messages for bug reports from a flask app.

In a Flask view, is g._request_id the best way to access the request ID or is there a public property or method to access it?

Current usability for AWS lambda?

Similar to #1, but with a non-app-framework in mind. In particular the logging functionality, but possibly others such as config.

I'm starting a new project intended to be deployed as an AWS Lambda function. Would your recommendation be to start with python-dockerflow, or stick with mozlog for now?

Should dockerflow heartbeat fail on warnings?

While using dockerflow, the __heartbeat__ view fails if any checks throw a message of checks.messages.WARNING or higher. Should it fail on WARNINGs, or only fail on ERROR or CRITICAL messages?

CODE_OF_CONDUCT.md file missing

As of January 1 2019, Mozilla requires that all GitHub projects include this CODE_OF_CONDUCT.md file in the project root. The file has two parts:

  1. Required Text - All text under the headings Community Participation Guidelines and How to Report, are required, and should not be altered.
  2. Optional Text - The Project Specific Etiquette heading provides a space to speak more specifically about ways people can work effectively and inclusively together. Some examples of those can be found on the Firefox Debugger project, and Common Voice. (The optional part is commented out in the raw template file, and will not be visible until you modify and uncomment that part.)

If you have any questions about this file, or Code of Conduct policies and procedures, please reach out to [email protected].

(Message COC001)

Default checks in Django 3.2 break in dockerflow-python

Experimenter just updated to Django 3.2 and discovered this:

https://code.djangoproject.com/ticket/32665#ticket

In the heartbeat endpoint, we run the check and then optionally silence its output if it's in the silenced outputs:

https://github.com/mozilla-services/python-dockerflow/blob/main/src/dockerflow/django/views.py#L72-L73

but because the check throws altogether, this causes the heartbeat to 500. Perhaps we should do that in the opposite order?

Update supported versions of Python, Django, Flask and Sanic

The supported and tested versions should be updated to be limited to currently supported releases.

Python

The current Python list:

python =
3.6: py36
3.7: py37
3.8: py38
3.9: py39

Supported versions are at: https://www.python.org/downloads/

We should:

  • Drop Python 3.6 - Reached end-of-life on 2021-12-23 PEP 494
  • Add Python 3.10 - Released 2021-10-04, end of support in 2026-10 PEP 619

Django:

The current Django list:

dj22: -ctests/constraints/django-2.2.txt
dj30: -ctests/constraints/django-3.0.txt
dj31: -ctests/constraints/django-3.1.txt
dj40: -ctests/constraints/django-4.0.txt

Supported versions are at: https://www.djangoproject.com/download/

We should:

  • Drop Django 2.2 - Extended support ended April 2022
  • Drop Django 3.0 - Extended support ended April 2021
  • Drop Django 3.1 - Extended support ended December 2021
  • Add Django 3.2 LTS - Supported until April 2024

Flask

The current Flask list:

fl011: -ctests/constraints/flask-0.11.txt
fl012: -ctests/constraints/flask-0.12.txt
fl10: -ctests/constraints/flask-1.0.txt

Flask doesn't appear to have a supported release policy, but a v0.12.5 release was made in 2020-02-10 to fix an issue, almost 2 years after 1.0 was released, and a long-running v0.12 branch still exists. At the same time, this repo didn't keep up with version 1.1.0. Flask developers should speak up!

Possible next steps:

  • Drop Flask 0.11 support, not tracked upstream
  • Fix Flask 0.12 support
  • Fix Flask 1.0 support
  • Add Flask 1.1 support
  • Add Flask 2.0 support
  • Add Flask 2.1 support

Sanic

The current Sanic list:

s19: -ctests/constraints/sanic-19.txt
s20: -ctests/constraints/sanic-20.txt

Sanic versioning is calendar based, so this represents 2019 and 2020. The final release of the year (like 21.12) is the LTS release, supported for 24 months: https://sanic.dev/en/org/policies.html#versioning

Next steps:

  • Drop Sanic 19, no longer supported
  • Add Sanic 21, 21.12 supported until December 2023
  • Add Sanic 22, currently 22.6

Django specific?

The code I see here seems very Django specific. I'm ok with this, but it is in contrast with the name of the package, python-dockerflow. Is the goal of this package to be generic across any Python app? Or should we rename the repo to django-dockerflow?

Port mozilla-cloud-services-logger here

There is already a package called https://github.com/mozilla/mozilla-cloud-services-logger by @lmorchard that implements two things:

  1. A Python logging formatter consistent with mozlog

  2. A Django middleware that emits up request.summary following the Dockerflow specs

Both are essential to follow Docker and should live in a generic purpose library like python-dockerflow to allow a "one stop shop" experience for Dockerflow users.

@lmorchard Are you okay with us porting over your code into this repo?

Log query strings

Related discussion

It is possible to override the summary_extra method to log query strings, but it feels a bit hackish and would look much cleaner as a feature of the library hidden behind a config option.

Document CircleCI examples

Building the docker images for example contain building a version.json file

We should document how to do that.

Document how to serve static content

The Dockerflow requires the app to serve static content on its own.

For Django the best option currently is to use Whitenoise as it does this:

WhiteNoise takes care of best-practices for you, for instance:

Serving compressed content (gzip and Brotli formats, handling Accept-Encoding and Vary headers correctly)
Setting far-future cache headers on content which won’t change

Additionally it supports non-Django pure-WSGI use as well, so should work for Flask and Pyramid, too.

Travis CI free usage ends Dec 3; mozilla repos should switch to other CI platforms

We're opening this issue because your project has used Travis CI within the last 6 months. If you have already migrated off it, you can close and ignore this issue.

Travis CI is ending free builds on public repositories. travis-ci.com stopped providingthem in early November, and travis-ci.org will stop after December 31, 2020. To avoid disruptions to your workflows, you must migrate to another CI service.

For production use cases, we recommend switching to CircleCI. This service is already widely used within Mozilla. There is a guide to migrating from Travis CI to CircleCI available here.

For non production use cases, we recommend either CircleCI or Github Actions. There is a guide to migrating from Travis CI to Github Actions available here. Github Actions usage within Mozilla is new, and you will have to work with our github administrators to enable specific actions following this process.

If you have any questions, reach out in #github-admin:mozilla.org on matrix.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.