yelp / yelp_clog

yelp_clog is an archived project and is no longer maintained by Yelp.

yelp_clog - Python package for sending things to scribe

Documentation

Read the documentation at Read the Docs.

License

yelp_clog is licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0

Contributing

Everyone is encouraged to contribute to yelp_clog by forking the Github repository and making a pull request or opening an issue.

yelp_clog's Issues

log_line() python 3.6 incompatibility with file logging enabled

With python3.6, yelp-clog 2.6.0, and clog_enable_file_logging: true a clog.log_line() call generates the following traceback:

  File "/code/virtualenv_run/lib/python3.6/site-packages/clog/global_state.py", line 79, in log_line
    logger.log_line(stream, line)
  File "/code/virtualenv_run/lib/python3.6/site-packages/clog/loggers.py", line 252, in log_line
    self.stream_files[stream] = self._create_file(stream)
  File "/code/virtualenv_run/lib/python3.6/site-packages/clog/loggers.py", line 272, in _create_file
    return open(os.path.join(config.log_dir, stream + '.log'), 'ab', 0)
  File "/code/virtualenv_run/lib/python3.6/posixpath.py", line 78, in join
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not ValueProxy

yelp_clog silently fails for all future logs if called once before being configured

If log_line is called before logging is configured, check_create_default_loggers will not add any loggers to the global list, but will change it's value from None. Then, every subsequent time log_line is called, we'll skip adding loggers because the global variable is no longer None, but won't log anything because there are no loggers. This produces no errors and is very difficult to identify and track down when it occurs.

eliot@ proposed a rough solution seen here: https://i.fluffy.cc/zqpqGxDBv8Rd5Mb99b2p9xZg83fCSt8J.html. Something along these lines would make an early call to log_line fail loudly and prevent us from getting into a silently failing logging situation in the future.

Uwsgi Plugin serialization format

RE: My recent PR we know that marshal is dangerous but fast. However doing some rudimentary testing I think using struct + simple byte concat is faster on pre-Python3.5 (apparently someone decided to overhaul the marshal serialization I suppose). Here's a rudimentary test script I used to run some very basic timings. Using this approach means we have to deal with unicode vs. bytes ourselves, but since clog converts lines into UTF-8 anyways I don't think that will be much of an issue. How about stream names? It doesn't look like the ScribeLogger converts that to UTF-8, so we might need to reconstitute the unicode format by storing the encoding somehow.

Anyways, struct + raw bytes would be safer than marshal and would work in this case since we can make some strong assumptions about the arguments. I'll look into changing it but I think we should get the current PR in first just to fix the marshal.loads problem.

Read the Docs Builds Fail

Currently the docs fail to build on Read the Docs with the follow error on this step:

python /home/docs/checkouts/readthedocs.org/user_builds/yelp-clog/envs/latest/bin/sphinx-build -T -E -b readthedocs -d _build/doctrees-readthedocs -D language=en . _build/html

Running Sphinx v1.3.1
making output directory...

Traceback (most recent call last):
File "/home/docs/checkouts/readthedocs.org/user_builds/yelp-clog/envs/latest/local/lib/python2.7/site-packages/sphinx/cmdline.py", line 244, in main
opts.warningiserror, opts.tags, opts.verbosity, opts.jobs)
File "/home/docs/checkouts/readthedocs.org/user_builds/yelp-clog/envs/latest/local/lib/python2.7/site-packages/sphinx/application.py", line 126, in init
confoverrides or {}, self.tags)
File "/home/docs/checkouts/readthedocs.org/user_builds/yelp-clog/envs/latest/local/lib/python2.7/site-packages/sphinx/config.py", line 277, in init
execfile_(filename, config)
File "/home/docs/checkouts/readthedocs.org/user_builds/yelp-clog/envs/latest/local/lib/python2.7/site-packages/sphinx/util/pycompat.py", line 128, in execfile_
exec_(code, globals)
File "/home/docs/checkouts/readthedocs.org/user_builds/yelp-clog/envs/latest/local/lib/python2.7/site-packages/six.py", line 699, in exec
exec("""exec code in globs, locs""")
File "", line 1, in
File "conf.py", line 43, in
from clog import version as version
File "/home/docs/checkouts/readthedocs.org/user_builds/yelp-clog/checkouts/latest/clog/init.py", line 34, in
from clog.loggers import ScribeLogger, ScribeIsNotForkSafeError
File "/home/docs/checkouts/readthedocs.org/user_builds/yelp-clog/checkouts/latest/clog/loggers.py", line 62, in
scribe_thrift = _load_thrift()
File "/home/docs/checkouts/readthedocs.org/user_builds/yelp-clog/checkouts/latest/clog/loggers.py", line 59, in _load_thrift
fn, module_name='scribe_thrift', include_dirs=[include_dir])
File "/home/docs/checkouts/readthedocs.org/user_builds/yelp-clog/envs/latest/local/lib/python2.7/site-packages/thriftpy-0.3.2-py2.7-linux-x86_64.egg/thriftpy/parser/init.py", line 30, in load
include_dir=include_dir)
File "/home/docs/checkouts/readthedocs.org/user_builds/yelp-clog/envs/latest/local/lib/python2.7/site-packages/thriftpy-0.3.2-py2.7-linux-x86_64.egg/thriftpy/parser/parser.py", line 484, in parse
with open(path) as fh:
IOError: [Errno 2] No such file or directory: 'scribe.thrift'

Exception occurred:
File "/home/docs/checkouts/readthedocs.org/user_builds/yelp-clog/envs/latest/local/lib/python2.7/site-packages/thriftpy-0.3.2-py2.7-linux-x86_64.egg/thriftpy/parser/parser.py", line 484, in parse
with open(path) as fh:
IOError: [Errno 2] No such file or directory: 'scribe.thrift'
The full traceback has been saved in /tmp/sphinx-err-LNhokb.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at https://github.com/sphinx-doc/sphinx/issues. Thanks!

yelp_clog should acquire the logging lock with a timeout

If we have been asked to log lines with a timeout, we probably shouldn't block forever waiting to acquire the RLock in the _log_line_no_size_limit method. Instead we probably want to just fail to log the line.
Basically we need to change this code to not block forever on the lock.

Zipkin instrumentation breaks logging

Logging breaks when enabling zipkin instrumentation in python3.5

Put the latest release on pypi?

Currently have to install from git to use anything modern here :(

Remove hard-dep of thriftpy==0.1.15

Currently it forces all other packages depping on thriftpy to also hard-pin 0.1.15, otherwise any service which uses both breaks with version conflict. This should be removed.

Tests Require testifycompat Module

Our tests currently rely on the testifycompat module. However, we have only open sourced testify. We should either fallback to testify or fully migrate to pytest (preferred).

Update __version__ string

Despite being version 2.6.5, clog.__version__ shows '2.6.1'.

https://github.com/Yelp/yelp_clog/blob/master/clog/__init__.py#L56