Store job logs about grimoirelab-kingarthur

[sduenas]

Yeah, probably that's what we need although I'm not sure about rq.worker. Probably it will be better to store arthur package too.

[dlumbrer]

I'm trying to get the worker log (that contains the logs of the jobs) in order to filter it by a Handler and save them in the corresponding meta log (just for testing, this will not be the code):

class MetaJobLogHandler(StreamHandler):
    def __init__(self, job):
        StreamHandler.__init__(self)
        self.job = job
        self.job.meta['log'] = ""
        self.current_job = None

    def emit(self, record):
        msg = self.format(record)
        if self.job.id in msg or not self.current_job:
            self.current_job = self.job
        self.current_job.meta['log'] += "{} \n".format(msg)

The problem that I faced is that the logger (logger = logging.getLogger('rq.worker')) does not contain the logs from the Perceval job, I have to find in what logger are the Perceval logs writing.

Any idea @sduenas ?

[dlumbrer]

This is the logger tree level:

<--""
   Level INFO
   Handler <ColorizingStreamHandler <stderr> (NOTSET)>
     Formatter fmt='[%(asctime)s] - %(message)s' datefmt=None
   |
   o<--"arthur"
   |   Level NOTSET so inherits level INFO
   |   Handler <NullHandler (NOTSET)>
   |   |
   |   o<--"arthur.events"
   |   |   Level NOTSET so inherits level INFO
   |   |
   |   o<--"arthur.jobs"
   |   |   Level NOTSET so inherits level INFO
   |   |
   |   o<--"arthur.worker"
   |       Level NOTSET so inherits level INFO
   |
   o<--"asyncio"
   |   Level NOTSET so inherits level INFO
   |
   o<--[concurrent]
   |   |
   |   o<--"concurrent.futures"
   |       Level NOTSET so inherits level INFO
   |
   o<--[django]
   |   |
   |   o<--"django.template"
   |       Level NOTSET so inherits level INFO
   |
   o<--"grimoirelab_toolkit"
   |   Level NOTSET so inherits level INFO
   |   Handler <NullHandler (NOTSET)>
   |   |
   |   o<--"grimoirelab_toolkit.datetime"
   |       Level NOTSET so inherits level INFO
   |
   o<--[perceval]
   |   |
   |   o<--"perceval.archive"
   |   |   Level NOTSET so inherits level INFO
   |   |
   |   o<--"perceval.backend"
   |   |   Level NOTSET so inherits level INFO
   |   |
   |   o<--[perceval.backends]
   |   |   |
   |   |   o<--[perceval.backends.core]
   |   |       |
   |   |       o<--"perceval.backends.core.askbot"
   |   |       |   Level NOTSET so inherits level INFO
   |   |       |
   |   |       o<--"perceval.backends.core.bugzilla"
   |   |       |   Level NOTSET so inherits level INFO
   |   |       |
   |   |       o<--"perceval.backends.core.bugzillarest"
   |   |       |   Level NOTSET so inherits level INFO
   |   |       |
   |   |       o<--"perceval.backends.core.confluence"
   |   |       |   Level NOTSET so inherits level INFO
   |   |       |
   |   |       o<--"perceval.backends.core.discourse"
   |   |       |   Level NOTSET so inherits level INFO
   |   |       |
   |   |       o<--"perceval.backends.core.dockerhub"
   |   |       |   Level NOTSET so inherits level INFO
   |   |       |
   |   |       o<--"perceval.backends.core.gerrit"
   |   |       |   Level NOTSET so inherits level INFO
   |   |       |
   |   |       o<--"perceval.backends.core.git"
   |   |       |   Level NOTSET so inherits level INFO
   |   |       |
   |   |       o<--"perceval.backends.core.github"
   |   |       |   Level NOTSET so inherits level INFO
   |   |       |
   |   |       o<--"perceval.backends.core.gitlab"
   |   |       |   Level NOTSET so inherits level INFO
   |   |       |
   |   |       o<--"perceval.backends.core.googlehits"
   |   |       |   Level NOTSET so inherits level INFO
   |   |       |
   |   |       o<--"perceval.backends.core.groupsio"
   |   |       |   Level NOTSET so inherits level INFO
   |   |       |
   |   |       o<--"perceval.backends.core.hyperkitty"
   |   |       |   Level NOTSET so inherits level INFO
   |   |       |
   |   |       o<--"perceval.backends.core.jenkins"
   |   |       |   Level NOTSET so inherits level INFO
   |   |       |
   |   |       o<--"perceval.backends.core.jira"
   |   |       |   Level NOTSET so inherits level INFO
   |   |       |
   |   |       o<--"perceval.backends.core.launchpad"
   |   |       |   Level NOTSET so inherits level INFO
   |   |       |
   |   |       o<--"perceval.backends.core.mattermost"
   |   |       |   Level NOTSET so inherits level INFO
   |   |       |
   |   |       o<--"perceval.backends.core.mbox"
   |   |       |   Level NOTSET so inherits level INFO
   |   |       |
   |   |       o<--"perceval.backends.core.mediawiki"
   |   |       |   Level NOTSET so inherits level INFO
   |   |       |
   |   |       o<--"perceval.backends.core.meetup"
   |   |       |   Level NOTSET so inherits level INFO
   |   |       |
   |   |       o<--"perceval.backends.core.nntp"
   |   |       |   Level NOTSET so inherits level INFO
   |   |       |
   |   |       o<--"perceval.backends.core.phabricator"
   |   |       |   Level NOTSET so inherits level INFO
   |   |       |
   |   |       o<--"perceval.backends.core.pipermail"
   |   |       |   Level NOTSET so inherits level INFO
   |   |       |
   |   |       o<--"perceval.backends.core.redmine"
   |   |       |   Level NOTSET so inherits level INFO
   |   |       |
   |   |       o<--"perceval.backends.core.rss"
   |   |       |   Level NOTSET so inherits level INFO
   |   |       |
   |   |       o<--"perceval.backends.core.slack"
   |   |       |   Level NOTSET so inherits level INFO
   |   |       |
   |   |       o<--"perceval.backends.core.stackexchange"
   |   |       |   Level NOTSET so inherits level INFO
   |   |       |
   |   |       o<--"perceval.backends.core.supybot"
   |   |       |   Level NOTSET so inherits level INFO
   |   |       |
   |   |       o<--"perceval.backends.core.telegram"
   |   |       |   Level NOTSET so inherits level INFO
   |   |       |
   |   |       o<--"perceval.backends.core.twitter"
   |   |           Level NOTSET so inherits level INFO
   |   |
   |   o<--"perceval.client"
   |   |   Level NOTSET so inherits level INFO
   |   |
   |   o<--"perceval.utils"
   |       Level NOTSET so inherits level INFO
   |
   o<--"requests"
   |   Level WARNING
   |   Handler <NullHandler (NOTSET)>
   |
   o<--"rq"
   |   Level NOTSET so inherits level INFO
   |   Handler <MetaJobLogHandler <stderr> (INFO)>
   |     Level INFO
   |   |
   |   o<--"rq.worker"
   |       Level INFO
   |
   o<--"urllib3"
   |   Level NOTSET so inherits level INFO
   |   Handler <NullHandler (NOTSET)>
   |   |
   |   o<--"urllib3.connection"
   |   |   Level NOTSET so inherits level INFO
   |   |
   |   o<--"urllib3.connectionpool"
   |   |   Level NOTSET so inherits level INFO
   |   |
   |   o<--"urllib3.poolmanager"
   |   |   Level NOTSET so inherits level INFO
   |   |
   |   o<--"urllib3.response"
   |   |   Level NOTSET so inherits level INFO
   |   |
   |   o<--[urllib3.util]
   |       |
   |       o<--"urllib3.util.retry"
   |           Level NOTSET so inherits level INFO
   |
   o<--"urrlib3"
       Level WARNING

[dlumbrer]

I was thinking on adding the handler to the perceval logger and rq.worker logger, it works for now.

WDYT @sduenas ?

[dlumbrer]

@sduenas I updated the PR adding more information in the log record (date, level, module, and msg for now).

Please, check it when you have time

[dlumbrer]

PR updated with some tests.

Please @sduenas , check it when you have time