nhairs / python-json-logger

JSON Formatter for Python Logging

Python JSON Logger

Python JSON Logger enables you produce JSON logs when using Python's logging package.

JSON logs are machine readable allowing for much easier parsing and ingestion into log aggregation tools.

🚨 Important 🚨

This repository is a maintained fork of madzak/python-json-logger pending a PEP 541 request for the PyPI package. The future direction of the project is being discussed here.

Documentation

• Documentation
• Quickstart Guide
• Change Log
• Contributing

License

This project is licensed under the BSD 2 Clause License - see LICENSE

Authors and Maintainers

This project was originally authored by Zakaria Zajac and our wonderful contributors

It is currently maintained by:

• Nicholas Hairs - nicholashairs.com

python-json-logger's Issues

Add benchmarking

Given that we are adding different encoders, primarily for their performance it may be worth actually documenting their performance.

This looks like a good candidate: https://github.com/ionelmc/pytest-benchmark/

asctime handling

Currently using the asctime field will produce a string that is formatted separately from the JSON encoders.

By default it is produced by logging.Formatter.formatTime which uses converter=time.localtime and datefmt.

It feels like instead it should be a datetime encoded by the JSON encoder. However doing this would raise other questions like:

• What do we do with the datefmt argument?
• What do we do with the converter attribute?
• Do we remove the timestamp argument?
  • This could be converted to an alias that adds asctime to required_fields and "asctime": "timestamp" to rename_fields. Though then what do we do if asctime and timestamp are used?
  • I'll note that there doesn't appear to be any reasoning for why timestamp was added included in the commit that added it. Searching old issues doesn't surface anything either.
• Do we support time zones that aren't UTC?
• What about users still using asctime, datefmt, and converter?
  • Searching old issues - it looks like there people who are using these to control the output of asctime.

As such any decision to move to using the JSON encoder or changing / removing timestamp is likely a breaking change which we probably want to avoid for as long as possible.

Another approach would be to set "saner" defaults for formatTime or (datefmt and converter). This could be considered a non-breaking change (anyone using defaults would experience the change, but anyone using custom shouldn't see breakages).

Another approach would be to use some kind of "opt-in" keyword argument that toggles new behaviour and otherwise uses old behaviour.

Perhaps in the short term all we do is update the docs to suggest that using timestamp over asctime.

Opinions welcome

Using rename fields changes the key order of the log output

Original Issue: madzak/python-json-logger#169

When I rename some of the default field names using rename_fields={key:key} they are logged at the end of the json output.

current behaviour:

Log output without rename fields: {"asctime": "2023-02-21T22:02:12+0100", "levelname": "INFO", "name": "werkzeug", "funcName": "_log", "message": "127.0.0.1"}

Log output with rename fields: {"name": "werkzeug", "funcName": "_log", "message": "127.0.0.1", "time": "2023-02-21T22:02:12+0100", "level": "INFO"}

expected behaviour:

Log output without rename fields: {"asctime": "2023-02-21T22:02:12+0100", "levelname": "INFO", "name": "werkzeug", "funcName": "_log", "message": "127.0.0.1"}

Log output with rename fields: {"time": "2023-02-21T22:02:12+0100", "level": "INFO", "name": "werkzeug", "funcName": "_log", "message": "127.0.0.1"}

--

used version: v2.0.7 used format: "%(asctime)s %(levelname)s %(name)s %(funcName)s %(message)s"

And as a side comment I'm pretty sure this was working as expected a few versions before so some of the newer commits must have changed the behaviour. v2.0.4 works correctly v2.0.5 works unexpectedly

Tests failing with Python 3.12 (taskName)

Original Issue: madzak/python-json-logger#185

With Python 3.12 we are seeing the following test cases fail. They don't fail with Python 3.11.

python-json-logger> _______________ TestJsonLogger.test_custom_object_serialization ________________
python-json-logger> 
python-json-logger> self = <tests.test_jsonlogger.TestJsonLogger testMethod=test_custom_object_serialization>
python-json-logger> 
python-json-logger>     def test_custom_object_serialization(self):
python-json-logger>         def encode_complex(z):
python-json-logger>             if isinstance(z, complex):
python-json-logger>                 return (z.real, z.imag)
python-json-logger>             else:
python-json-logger>                 type_name = z.__class__.__name__
python-json-logger>                 raise TypeError("Object of type '{}' is no JSON serializable".format(type_name))
python-json-logger>     
python-json-logger>         formatter = jsonlogger.JsonFormatter(json_default=encode_complex,
python-json-logger>                                              json_encoder=json.JSONEncoder)
python-json-logger>         self.log_handler.setFormatter(formatter)
python-json-logger>     
python-json-logger>         value = {
python-json-logger>             "special": complex(3, 8),
python-json-logger>         }
python-json-logger>     
python-json-logger>         self.log.info(" message", extra=value)
python-json-logger>         msg = self.buffer.getvalue()
python-json-logger> >       self.assertEqual(msg, "{\"message\": \" message\", \"special\": [3.0, 8.0]}\n")
python-json-logger> E       AssertionError: '{"message": " message", "taskName": null, "special": [3.0, 8.0]}\n' != '{"message": " message", "special": [3.0, 8.0]}\n'
python-json-logger> E       - {"message": " message", "taskName": null, "special": [3.0, 8.0]}
python-json-logger> E       ?                        ------------------
python-json-logger> E       + {"message": " message", "special": [3.0, 8.0]}
python-json-logger> 
python-json-logger> tests/test_jsonlogger.py:277: AssertionError
python-json-logger> ------------------------------ Captured log call -------------------------------
python-json-logger> INFO     logging-test-99:test_jsonlogger.py:275  message
python-json-logger> ____________________ TestJsonLogger.test_percentage_format _____________________
python-json-logger> 
python-json-logger> self = <tests.test_jsonlogger.TestJsonLogger testMethod=test_percentage_format>
python-json-logger> 
python-json-logger>     def test_percentage_format(self):
python-json-logger>         fr = jsonlogger.JsonFormatter(
python-json-logger>             # All kind of different styles to check the regex
python-json-logger>             '[%(levelname)8s] %(message)s %(filename)s:%(lineno)d %(asctime)'
python-json-logger>         )
python-json-logger>         self.log_handler.setFormatter(fr)
python-json-logger>     
python-json-logger>         msg = "testing logging format"
python-json-logger>         self.log.info(msg)
python-json-logger>         log_json = json.loads(self.buffer.getvalue())
python-json-logger>     
python-json-logger>         self.assertEqual(log_json["message"], msg)
python-json-logger> >       self.assertEqual(log_json.keys(), {'levelname', 'message', 'filename', 'lineno', 'asctime'})
python-json-logger> E       AssertionError: dict_keys(['levelname', 'message', 'filename', 'lineno', 'asctime', 'taskName']) != {'asctime', 'levelname', 'message', 'filename', 'lineno'}
python-json-logger> 
python-json-logger> tests/test_jsonlogger.py:53: AssertionError
python-json-logger> ------------------------------ Captured log call -------------------------------
python-json-logger> INFO     logging-test-51:test_jsonlogger.py:49 testing logging format
python-json-logger> __________________ TestJsonLogger.test_rename_reserved_attrs ___________________
python-json-logger> 
python-json-logger> self = <tests.test_jsonlogger.TestJsonLogger testMethod=test_rename_reserved_attrs>
python-json-logger> 
python-json-logger>     def test_rename_reserved_attrs(self):
python-json-logger>         log_format = lambda x: ['%({0:s})s'.format(i) for i in x]
python-json-logger>         reserved_attrs_map = {
python-json-logger>             'exc_info': 'error.type',
python-json-logger>             'exc_text': 'error.message',
python-json-logger>             'funcName': 'log.origin.function',
python-json-logger>             'levelname': 'log.level',
python-json-logger>             'module': 'log.origin.file.name',
python-json-logger>             'processName': 'process.name',
python-json-logger>             'threadName': 'process.thread.name',
python-json-logger>             'msg': 'log.message'
python-json-logger>         }
python-json-logger>     
python-json-logger>         custom_format = ' '.join(log_format(reserved_attrs_map.keys()))
python-json-logger>         reserved_attrs = [_ for _ in jsonlogger.RESERVED_ATTRS if _ not in list(reserved_attrs_map.keys())]
python-json-logger>         formatter = jsonlogger.JsonFormatter(custom_format, reserved_attrs=reserved_attrs, rename_fields=reserved_attrs_map)
python-json-logger>         self.log_handler.setFormatter(formatter)
python-json-logger>         self.log.info("message")
python-json-logger>     
python-json-logger>         msg = self.buffer.getvalue()
python-json-logger> >       self.assertEqual(msg, '{"error.type": null, "error.message": null, "log.origin.function": "test_rename_reserved_attrs", "log.level": "INFO", "log.origin.file.name": "test_jsonlogger", "process.name": "MainProcess", "process.thread.name": "MainThread", "log.message": "message"}\n')
python-json-logger> E       AssertionError: '{"taskName": null, "error.type": null, "err[227 chars]"}\n' != '{"error.type": null, "error.message": null,[209 chars]"}\n'
python-json-logger> E       - {"taskName": null, "error.type": null, "error.message": null, "log.origin.function": "test_rename_reserved_attrs", "log.level": "INFO", "log.origin.file.name": "test_jsonlogger", "process.name": "MainProcess", "process.thread.name": "MainThread", "log.message": "message"}
python-json-logger> E       ?  ------------------
python-json-logger> E       + {"error.type": null, "error.message": null, "log.origin.function": "test_rename_reserved_attrs", "log.level": "INFO", "log.origin.file.name": "test_jsonlogger", "process.name": "MainProcess", "process.thread.name": "MainThread", "log.message": "message"}
python-json-logger> 
python-json-logger> tests/test_jsonlogger.py:299: AssertionError
python-json-logger> ------------------------------ Captured log call -------------------------------
python-json-logger> INFO     logging-test-68:test_jsonlogger.py:296 message

Per this comment

gh-91513: Added taskName attribute to logging module for use with asyncio tasks. python/cpython#91513

Defaults parameter is ignored

The defaults parameter that Logging.Formatter takes is ignored. We can make it work 😄

I can submit a patch :-)

orjson module docstring refers to msgspec instead of orjson

https://github.com/nhairs/python-json-logger/blob/main/src/pythonjsonlogger/orjson.py#L1

JSON Formatter using [msgspec](https://github.com/ijl/orjson)
                       ^ this should say orjson not msgspec

Move to importlib.util.find_spec for checking if packages are installed

Per title, move to importlib.util.find_spec over try: import name; catch ImportError.

This allows us to check for installed modules (i.e. orjson, msgspec) without actually importing them.

Ref: https://stackoverflow.com/a/14050282/12281814

Cannot rename fields unless they're present in every log record

Original Issue: madzak/python-json-logger#171

I tried the following configuration to get field names matching the OpenTelemetry semantic conventions:

json:
  (): pythonjsonlogger.jsonlogger.JsonFormatter
  rename_fields:
    otelTraceID: trace_id
    otelSpanID: span_id
    otelServiceName: service.name
     levelname: severity
     message: body
     threadName: thread.name
     exc_type: exception.type
     exc_val: exception.message
     traceback: exception.stacktrace
     pathname: code.filepath
     lineno: code.lineno
     funcName: code.function
  format: '%(levelname)s %(name)s %(threadName)s %(message)s %(pathname)s %(lineno)s %(funcName)s'

It would not work because some of these fields were not present in every log record, but the field rename function requires cannot handle missing fields. In particular, renaming the exception related fields is impossible.

Addendum: explicitly adding these fields to format removes the error, but then they have null values where a log record does not contain an exception. Would it be possible to get fields renamed without making them mandatory?

madzak/python-json-logger#189 appears to be a duplicate

Project Direction

Hi all,

Pending the PEP 541 Request, I wanted to gather community feedback on the direction of this project.

My current thoughts are:

Version 3.0.0

Goal: bring project into maintained state

• Move to pyproject.toml
• Ensure local development works (local linting, testing, building, etc)
• Ensure CI works (test all versions, run on PRs etc)
• Update supported python versions
  • drop 3.6 support
  • CPython 3.7-3.12 (ubuntu, win, mac)
  • Pypy 3.7-3.10 (ubuntu, win, mac)
• Bug fixes
  • madzak/python-json-logger#185
    #2
  • madzak/python-json-logger#173
    #4

Version 3.x

Goal: modernise package and include feature requests

• Existing feature requests from old repository
• ORJSON (and other?) encoders
  #9

Support Policy

Security support: any python version that makes up at least 5% of total downloads and within 2 years of EOL (currently 3.7+)

Bug fixes and new features: only currently maintained versions of python (currently 3.8+)

Maintainers

I'd like to look for 1-3 persons who are interested in helping maintain the project. These persons would also serve as a "backup" to prevent this repository from becoming unmaintained. If you'd like to be involved let me know

Support comma format

The "comma" format (as mentioned in the cookbook), is probably worth adding as a directly supported format.

Pros:

• very simple to create from a list (or any iterable) using ",".join(x)
• simpler format to write so less prone to errors in the format
  • Other styles may have missing brackets, tokens, etc.
• very readable

Cons:

• This format is incompatible with other formatters.
  • i.e. once you move to this config it is harder to move to other formatters

Improve fileConfig / dictConfig documentation

The current documentation is copied from the original project.

It can probably be improved especially to document around using the () syntax (as mentioned in this comment) to gain access to non-standard __init__ arguments / which ones support turning strings into objects etc.

How to lazy-log extras / lazy-evaluate interpolations

With classic logging you'd:

logger.debug("some var: %s", the_var)

so that a perhaps costly conversion or interpolation isn't payed if the log level isn't debug or trace.
How would we do the same with structured logging?

logger.debug("some var: %s", the_var, extras={})

could work for the message but what if I need it inside another field?

logger.debug({"key1": "%s", "key2": "%s"}, var1, var2)

This looks nasty.

Or is there already something in place I've missed?

Set JsonFormatter.__init__ return type to None

Original Issue: madzak/python-json-logger#173

Hello, according to PEP 484, __init__ methods need to be marked as if they return None.

(Note that the return type of init ought to be annotated with -> None. The reason for this is subtle. If init assumed a return annotation of -> None, would that mean that an argument-less, un-annotated init method should still be type-checked? Rather than leaving this ambiguous or introducing an exception to the exception, we simply say that init ought to have a return annotation; the default behavior is thus the same as for other methods.)

In my code I have a custom formatter that derives JsonFormatter and calls super().__init__, but the lack of types triggers mypy. I can "solve" this on my end by adding a # type: ignore, but it's such a low hanging fruit that it's probably worth adding it here.

I'm willing to make a PR for this

StringTemplateStyle ($) does not support unbraced names

Original Issue: madzak/python-json-logger#194

Fix regex for string template style

valid String template: '$asctime $level $message'

regex for string template in JsonFormatter.parse:

....
        elif isinstance(self._style, logging.StrFormatStyle):
            formatter_style_pattern = re.compile(r"\{(.+?)\}", re.IGNORECASE)
...

would require the string template to be '${asctime} ${level} ${message}'

Support list / tuple of strings for format

Many other JSON formatting libraries allow or directly use a list of strings for the format. This suggests that it might be worth us adding in support for it.

Pros:

• No parsing means no testing for a correct format - we can directly use the list.
• Allows for clearer config when using dictConfig with the () user-defined object indicator, or instantiating within Python itself.

Cons:

• Must always use the () custom class indicator when using dictConfig.
• Not compatible with fileConfig.
• Not compatible with other formatters

Add the package to pypi

Add the package to pypi