Coder Social home page Coder Social logo

eliottree's Introduction

eliottree

build_ coverage_

Render Eliot logs as an ASCII tree.

This output:

image

(or as text)

$ eliot-tree eliot.log
f3a32bb3-ea6b-457c-aa99-08a3d0491ab4
└── app:soap:client:request/1 ⇒ started 2015-03-03 04:28:56 ⧖ 1.238s
    ├── dump: /home/user/dump_files/20150303/1425356936.28_Client_req.xml
    ├── soapAction: a_soap_action
    ├── uri: http://example.org/soap
    ├── app:soap:client:success/2/1 ⇒ started 2015-03-03 04:28:57 ⧖ 0.000s
    │   └── app:soap:client:success/2/2 ⇒ succeeded 2015-03-03 04:28:57
    │       └── dump: /home/user/dump_files/20150303/1425356937.52_Client_res.xml
    └── app:soap:client:request/3 ⇒ succeeded 2015-03-03 04:28:57
        └── status: 200

 89a56df5-d808-4a7c-8526-e603aae2e2f2
 └── app:soap:service:request/1 ⇒ started 2015-03-03 04:31:08 ⧖ 3.482s
     ├── dump: /home/user/dump_files/20150303/1425357068.03_Service_req.xml
     ├── soapAction: method
     ├── uri: /endpoints/soap/method
     ├── app:soap:service:success/2/1 ⇒ started 2015-03-03 04:31:11 ⧖ 0.001s
     │   └── app:soap:service:success/2/2 ⇒ succeeded 2015-03-03 04:31:11
     │       └── dump: /home/user/dump_files/20150303/1425357071.51_Service_res.xml
     └── app:soap:service:request/3 ⇒ succeeded 2015-03-03 04:31:11
         └── status: 200

was generated from:

{"dump": "/home/user/dump_files/20150303/1425356936.28_Client_req.xml", "timestamp": 1425356936.278875, "uri": "http://example.org/soap", "action_status": "started", "task_uuid": "f3a32bb3-ea6b-457c-aa99-08a3d0491ab4", "action_type": "app:soap:client:request", "soapAction": "a_soap_action", "task_level": [1]}
{"timestamp": 1425356937.516579, "task_uuid": "f3a32bb3-ea6b-457c-aa99-08a3d0491ab4", "action_type": "app:soap:client:success", "action_status": "started", "task_level": [2, 1]}
{"task_uuid": "f3a32bb3-ea6b-457c-aa99-08a3d0491ab4", "action_type": "app:soap:client:success", "dump": "/home/user/dump_files/20150303/1425356937.52_Client_res.xml", "timestamp": 1425356937.517077, "action_status": "succeeded", "task_level": [2, 2]}
{"status": 200, "task_uuid": "f3a32bb3-ea6b-457c-aa99-08a3d0491ab4", "task_level": [3], "action_type": "app:soap:client:request", "timestamp": 1425356937.517161, "action_status": "succeeded"}
{"dump": "/home/user/dump_files/20150303/1425357068.03_Service_req.xml", "timestamp": 1425357068.032091, "uri": "/endpoints/soap/method", "action_status": "started", "task_uuid": "89a56df5-d808-4a7c-8526-e603aae2e2f2", "action_type": "app:soap:service:request", "soapAction": "method", "task_level": [1]}
{"timestamp": 1425357071.51233, "task_uuid": "89a56df5-d808-4a7c-8526-e603aae2e2f2", "action_type": "app:soap:service:success", "action_status": "started", "task_level": [2, 1]}
{"task_uuid": "89a56df5-d808-4a7c-8526-e603aae2e2f2", "action_type": "app:soap:service:success", "dump": "/home/user/dump_files/20150303/1425357071.51_Service_res.xml", "timestamp": 1425357071.513453, "action_status": "succeeded", "task_level": [2, 2]}
{"status": 200, "task_uuid": "89a56df5-d808-4a7c-8526-e603aae2e2f2", "task_level": [3], "action_type": "app:soap:service:request", "timestamp": 1425357071.513992, "action_status": "succeeded"}

Command-line options

Consult the output of eliot-tree --help to see a complete list of command-line options.

Streaming

It's possible to pipe data into eliot-tree, from a tailed log for example, and have it rendered incrementally. There is a caveat though: Trees are only rendered once an end message—a success or failure status—for the tree's root action appears in the data.

Selecting / filtering tasks

By task UUID

Entire task trees can be selected by UUID with the --task-uuid (-u) command-line option.

By start / end date

Individual tasks can be selected based on their timestamp, use --start to select tasks after an ISO8601 date-time, and --end to select tasks before an ISO8601 date-time.

By custom query

Custom task selection can be done with the --select command-line option, the syntax of which is JMESPath, and is applied to the original Eliot JSON structures. Any data that appears within an Eliot task structure can be queried. Only the matching tasks (and all of their children) will be displayed, the parents of the task structure (by task_uuid) will be elided.

An important thing to note is that the query should be used as a predicate (it should describe a boolean condition), not to narrow a JSON data structure, as many of the examples on the JMESPath site illustrate. The reason for this is that Eliot tasks are not stored as one large nested JSON structure, they are instead many small structures that are linked together by metadata (task_uuid), which is not a structure than JMESPath is ideally equipped to query.

The --select argument can be supplied multiple times to mimic logical AND, that is all --select predicates must pass in order for a task or node to be selected.

Examples

Select all tasks that contain a uri key, regardless of its value:

--select 'uri'

Select all Eliot action tasks of type http_client:request:

--select 'action_type == `"http_client:request"`'

Backquotes are used to represent raw JSON values in JMESPath, `500 is the number 500, "500"` is the string "500".

Select all tasks that have an http_status of 401 or 500:

--select 'contains(`[401, 500]`, status)'

Select all tasks that have an http_status of 401 that were also made to a uri containing the text /criticalEndpoint:

--select 'http_status == `401`' \
--select 'uri && contains(uri, `"/criticalEndpoint"`)'

Here --select is passed twice to mimic a logical AND condition, it is also possible to use the JMESPath && operator. There is also a test for the existence of the uri key to guard against calling the contains() function with a null subject.

See the JMESPath specification for more information.

Programmatic usage

import json, sys
from eliottree import tasks_from_iterable, render_tasks
# Or `codecs.getwriter('utf-8')(sys.stdout).write` on Python 2.
render_tasks(sys.stdout.write, tasks, colorize=True)

See help(render_tasks) and help(tasks_from_iterable) from a Python REPL for more information.

Configuration

Command-line options may have custom defaults specified by way of a config file. The config file can be passed with the --config argument, or will be read from ~/.config/eliot-tree/config.json. See config.example.json for an example.

Use the --show-default-config command-line option to display the default configuration, suitable for redirecting to a file. Use the --show-current-config command-line option to display the current effective configuration.

Theme overrides

Theme colors can be overridden via the theme_overrides key in the config file. The value of this key is itself a JSON object, each key is the name of a theme color and each value is a JSON list. This list should contain three values:

  1. Foreground color, terminal color name or code; or null for the default color.
  2. Background color, terminal color name or code; or null for the default color.
  3. An optional array of color attribute names or codes; or null for the default attribute.

For example, to override the root theme color to be bold magenta, and the prop theme color to be red:

{
  "theme_overrides": {
    "root": ["magenta", null, ["bold"]],
    "prop_key": ["red"]
  }
}

See _theme.py for theme color names and the colored Python package for available color and attribute constants.

Contribute

See <https://github.com/jonathanj/eliottree> for details.

eliottree's People

Contributors

itamarst avatar jml avatar jonathanj avatar mithrandi avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

eliottree's Issues

Command prompt & python usage

Hi,

I am using jupyter lab notebook to execute the following:

from eliot import log_call, to_file

eliot_file =r"c:\Users\user\Downloads\eliot.log"
to_file(open(eliot_file, "w"))

@log_call
def add(a, b):
    return a + b

@log_call
def multiply(a, b):
    return a * b

multiply(add(1,2), 4)

Result is 12, and content of eliot_file is:

{"a": 1, "b": 2, "action_status": "started", "timestamp": 1578133156.6194398, "task_uuid": "ecd3e4f3-3f20-428f-8392-2c719e60e674", "action_type": "__main__.add", "task_level": [1]}
{"result": 3, "action_status": "succeeded", "timestamp": 1578133156.6204777, "task_uuid": "ecd3e4f3-3f20-428f-8392-2c719e60e674", "action_type": "__main__.add", "task_level": [2]}
{"a": 3, "b": 4, "action_status": "started", "timestamp": 1578133156.6204777, "task_uuid": "fcd2c4d6-83c0-4133-b5c6-d9049ed960fb", "action_type": "__main__.multiply", "task_level": [1]}
{"result": 12, "action_status": "succeeded", "timestamp": 1578133156.6214757, "task_uuid": "fcd2c4d6-83c0-4133-b5c6-d9049ed960fb", "action_type": "__main__.multiply", "task_level": [2]}

From windows 10, command line "eliot-tree eliot.log", situation is as following:

image

The picture above is with command prompt code page 437 (OEM - United States).
I tried to change code page to UTF-8 ("CHCP 65001"), but I still got special characters instead of symbols.

Question 1: How to set up windows 10 in order to get output of eliot-tree readable?

I tried to get the tree directly from python3, but I am not sure how to create tasks in:

import json, sys
from eliottree import tasks_from_iterable, render_tasks
render_tasks(sys.stdout.write, tasks, colorize=True)

I tried:

with open(eliot_file, "r") as f:
    x = f.readlines()

and then

render_tasks(sys.stdout.write, x, colorize=True)

or

y = tasks_from_iterable(x)
# or y = tasks_from_iterable(eliot_file)
render_tasks(sys.stdout.write, y, colorize=True)

but with no success ("").

Question 2: How to generate tasks from log file?

Regards.

Following long vertical lines can be visually challenging

Sometimes it's difficult to follow a particular vertical line from parent to child, particularly when these lines are long or span multiple pages.

Alternating through several high-contrast colors for adjacent vertical lines might help with this.

Strange format

Dear Jonathan,

I would like to ask for your kind help.

This is the first time that I try to use eliot whit eliot-tree.

The problem that the result formatted by eliot-tree looks very both in cmd, powershell and PyCharm's terminal.

I use Windows 10.

render

Thanks for your help in advance.

Factor out core functionality into a library

Currently, eliottree is a single script that does a bunch of stuff. If the core functionality were factored out into a library that the script used, it would be more maintainable (and testable!) and allow other tools to more easily build on top of eliottree.

Filtering by end fails on version 18.1.1

I get an error when I write the below query

λ eliot-tree logs\2020-06-01_0006.log --select "log_level=='DEBUG'" --end "2020-06-01T14:40:48"
Traceback (most recent call last):
  File "%userprofile%\.venv\Scripts\eliot-tree-script.py", line 9, in <module>
    sys.exit(main())
  File "%userprofile%\.venv\lib\site-packages\eliottree\_cli.py", line 178, in main
    human_readable=args.human_readable)
  File "%userprofile%\.venv\lib\site-packages\eliottree\_cli.py", line 92, in display_tasks
    colorize=colorize)
  File "%userprofile%\.venv\lib\site-packages\eliottree\_render.py", line 264, in render_tasks
    for task in tasks:
  File "%userprofile%\.venv\lib\site-packages\eliottree\_parse.py", line 19, in tasks_from_iterable
    for message_dict in iterable:
  File "%userprofile%\.venv\lib\site-packages\toolz\functoolz.py", line 488, in __call__
    ret = f(ret)
  File "%userprofile%\.venv\lib\site-packages\eliottree\filter.py", line 47, in _filter
    return _parse_timestamp(task[u'timestamp']) < end_date
TypeError: 'bool' object is not subscriptable

I conda installed it from conda-forge (thank you for putting it there btw).
the eliot-tree version is '18.1.1'

Tasks reported out of order if some events are deleted / missing

Jonathan,

Fantastic tool! I have been making great use of it!

I have an application which logs debug events when a debug option is enabled. I've been experimenting with the idea of always logging the debug information, and filtering the debug events out of the log when I don't want/need to see them. I found that if I delete an event it can cause eliot-tree to render the associated task out of (chronological) order (presumably because the parser is waiting for the missing event to show up in the log data, and then eventually gives up?)

Here's an example:

from eliot import start_action, to_file, Message
import subprocess

to_file(open("normal.log", "w"))
with start_action(action_type="first_action"):
    Message.log(message_type="some_message", level='debug')
    Message.log(message_type="some_message", level='info')

with start_action(action_type="second_action"):
    pass

# Create a version of the log file in which the 'debug' event has been deleted
with open('normal.log') as file:
    with open('filtered.log', 'w') as filtered_file:
        for line in file:
            if 'debug' not in line:
                filtered_file.write(line)

subprocess.call('eliot-tree --color never normal.log > normal.tree', shell=True)
subprocess.call('eliot-tree --color never filtered.log > filtered.tree', shell=True)

Which results in:

normal.tree

ceed1fcb-51fc-42a6-8219-e3157e98b402
└── first_action/1 ⇒ started
    ├── timestamp: 2017-11-04 02:31:24.041994
    ├── some_message/2
    │   ├── level: debug
    │   └── timestamp: 2017-11-04 02:31:24.042184
    ├── some_message/3
    │   ├── level: info
    │   └── timestamp: 2017-11-04 02:31:24.042294
    └── first_action/4 ⇒ succeeded
        └── timestamp: 2017-11-04 02:31:24.042397

47d6d5a3-4f7c-4276-8bae-014a9a2e2875
└── second_action/1 ⇒ started
    ├── timestamp: 2017-11-04 02:31:24.042547
    └── second_action/2 ⇒ succeeded
        └── timestamp: 2017-11-04 02:31:24.042654

filtered.tree

47d6d5a3-4f7c-4276-8bae-014a9a2e2875
└── second_action/1 ⇒ started
    ├── timestamp: 2017-11-04 02:31:24.042547
    └── second_action/2 ⇒ succeeded
        └── timestamp: 2017-11-04 02:31:24.042654

ceed1fcb-51fc-42a6-8219-e3157e98b402
└── first_action/1 ⇒ started
    ├── timestamp: 2017-11-04 02:31:24.041994
    ├── some_message/3
    │   ├── level: info
    │   └── timestamp: 2017-11-04 02:31:24.042294
    └── first_action/4 ⇒ succeeded
        └── timestamp: 2017-11-04 02:31:24.042397

Perhaps I am crazy to be trying to filter events, but it seems so close to working that I thought I'd ask what you thought.

Cheers!

Eliot-tree fails to parse log that does not have task_level of 1

In long running (> 1 hour) Flocker compute nodes, Eliot logs rotate, resulting in latest logs not having task_level of 1: https://clusterhq.atlassian.net/browse/FLOC-2943

vagrant@node1:/var/log/flocker$ eliot-tree flocker-dataset-agent.log
Traceback (most recent call last):
File "/usr/local/bin/eliot-tree", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/eliottree/_cli.py", line 94, in main
display_task_tree(args)
File "/usr/local/lib/python2.7/dist-packages/eliottree/_cli.py", line 47, in display_task_tree
nodes=tree.nodes(tree.merge_tasks(tasks, filter_funcs())),
File "/usr/local/lib/python2.7/dist-packages/eliottree/tree.py", line 164, in merge_tasks
raise RuntimeError('Some tasks have no start parent', pending)

Would be useful to have eliot-tree help debug Flocker issues using production logs. Thanks a lot for any help fixing this issue!

Render timestamp in a more consistent position

Currently the timestamp is rendered as a message field, depending on what other fields are present the visual position of this field will move around. An improvement would be to render the timestamp after the status (for an action) or level (for a message).

Regular messages aren't supported

E.g. try piping in:

{"task_uuid": "e96b14e5-21a6-421c-884e-af82e0018a85", "error": false, "timestamp": 1427725835.188358, "message": "Log opened.", "message_type": "twisted:log", "task_level": [1]}

and you'll get an error. Here's a fix:

diff --git a/eliottree.py b/eliottree.py
index 13af0b1..a9c1427 100644
--- a/eliottree.py
+++ b/eliottree.py
@@ -28,7 +28,7 @@ def _taskName(task):
     """
     level = u','.join(map(unicode, task[u'task_level']))
     messageType = task.get('message_type', None)
-    if messageType == u'eliot:traceback':
+    if messageType is not None:
         status = None
     elif messageType is None:
         messageType = task['action_type']

pip install fails on python 3

Requirement already satisfied: jmespath>=0.7.1 in c:\anaconda3\lib\site-packages (from eliot-tree)
Collecting iso8601>=0.1.10 (from eliot-tree)
  Downloading iso8601-0.1.12-py3-none-any.whl
Collecting tree-format>=0.1.1 (from eliot-tree)
  Downloading tree-format-0.1.1.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\Gal\AppData\Local\Temp\pip-build-0inmxyd1\tree-format\setup.py", line 23, in <module>
        long_description = readme.read()
      File "C:\Anaconda3\lib\encodings\cp1252.py", line 23, in decode
        return codecs.charmap_decode(input,self.errors,decoding_table)[0]
    UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in position 223: character maps to <undefined>

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in C:\Users\Gal\AppData\Local\Temp\pip-build-0inmxyd1\tree-format\

Messages without a message_type appear as unnamed tasks

Messages without a message_type appear as <UNNAMED TASK>s.

So the following code:

import eliot
import sys

eliot.to_file(sys.stdout)

with eliot.start_action(action_type=u"task"):
    eliot.Message.log(key=u"value")

Produces the following output:

7d074742-0116-4a57-a062-afa05bf13d5d
+-- task@1/started
    `-- timestamp: 2017-02-15 20:14:00.742261
    +-- <UNNAMED TASK>
        |-- key: value
        `-- timestamp: 2017-02-15 20:14:00.742498
    +-- task@3/succeeded
        `-- timestamp: 2017-02-15 20:14:00.742647

The message appears as an <UNNAMED TASK>. Providing a message_type field results in the message appearing as a message. This should be fixed or at least documented.

All tasks are displayed if jmespath query matches nothing

The expected behaviour is, of course, that no tasks are displayed.

I believe the cause of this issue is that merge_tasks returns None if nothing matched; this value is then passed to nodes which interprets None as "all tasks". I think the fix is just to return [] instead of None.

Be able to chunk the display of the tree

I have some logs with really long tasks, that do not allow me to scroll all the way through the history.
I would like to be able to iterate the the log lines some chunk size at a time.
like

(pipeline_opt) λ eliot-tree logs/2020-08-19_0004.log -l 0 --chunk-size 10
2450eced-dde4-477d-96fc-64409a0b371f
└── pipeline_optimize.initialization.initialize_tanks.new_tank/1 ⇒ started 2020-08-19 23:33:03Z ⧖ 13.326s
    ├── fac: InletFacility(InletFacTag(site='SANEAST2', service='UIS', facility_name='THIS_FAC', property_num='121312'), [PumpFacility(FacilityTag(site='SANEAST2', service='UIS', facilityid='THIS_FAC_LACT'))]
    ├── initialization_days: 7
    ├── log_level: 30
    ├── pipeline_optimize.services.cygnet_services.cygnet.get_current_values_for_facility/2/1 ⇒ started 2020-08-19 23:33:03Z ⧖ 0.409s
    │   ├── fac: PumpFacility(FacilityTag(site='SANEAST2', service='UIS', facilityid='THIS_FAC_LACT'))
    │   ├── log_level: 30
    │   ├── pipeline_optimize.services.cygnet_services.cygnet.load_current_value/2/2/1 ⇒ started 2020-08-19 23:33:03Z ⧖ 0.017s
    │   │   ├── log_level: 30
Press any key to continue or 'X' to exit or 'T' for next task:

encoding error / windows traceback

I'm not entirely sure what's going on here, but on a GitHub Actions windows runner, I'm seeing this from eliot-tree:

Traceback (most recent call last):
  File "c:\hostedtoolcache\windows\python\3.8.10\x64\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\hostedtoolcache\windows\python\3.8.10\x64\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\hostedtoolcache\windows\Python\3.8.10\x64\Scripts\eliot-tree.exe\__main__.py", line 7, in <module>
  File "c:\hostedtoolcache\windows\python\3.8.10\x64\lib\site-packages\eliottree\_cli.py", line 308, in main
    display_tasks(
  File "c:\hostedtoolcache\windows\python\3.8.10\x64\lib\site-packages\eliottree\_cli.py", line 116, in display_tasks
    render_tasks(
  File "c:\hostedtoolcache\windows\python\3.8.10\x64\lib\site-packages\eliottree\_render.py", line 289, in render_tasks
    write(format_tree(task, _format_node, _get_children, options))
  File "c:\hostedtoolcache\windows\python\3.8.10\x64\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 38-40: character maps to <undefined>

I've attached the eliot.log from the run (but when using eliot-tree on it on a Linux machine, I do not see the above error).

Tree structure isn't always correct

Consider the following example:

4350e65c-bf31-4ecd-8d2d-b48a601dc6f6
    +-- check_links@1/started
        `-- url: http://localhost:8080
    +-- download@2,1/started
        `-- url: http://localhost:8080
        +-- download@2,2/succeeded
    +-- download@3,1/started
        `-- url: http://nosuchdomain
        +-- download@3,2/failed
            |-- exception: requests.exceptions.ConnectionError
            |-- reason: HTTPConnectionPool(host='nosuchdomain', port=80): Max retries exceeded with url: / (Caused by <class [...]
    +-- download@4,1/started
        `-- url: http://google.com
        +-- download@4,2/succeeded
    +-- check_links@5/succeeded

Notice that download@2,1 and friends are child actions of check_links@1, but that is not reflected in the structure.

Protect me against terminal hacking

Sometimes when I use eliottree to dump a tree to stdout, something like this happens:

    +-- network-client:create@3/succeeded
        |-- response_object:
            |-- apiVersion: v1
            |-- data:
                |-- .3: N\u2d1d\u42a3\uacfe\u7e8e\U000cb9dd
                |-- .3.g.h.a.2g.d-b.lk6tr.7.le0cajs.6dfq: \xa6\x8c\x96
                |-- .ai.dvvh6.1.zzxuy.7.ve: \xba
                |-- .sd.u3g0zc1.l.v: 
                |-- 473: 
                    [\U000e6273 \u03d7\u98cao$\u8c93\u9f53\u013b
                |-- 8ka7kk89vv.9tlhgrivtk: '\U000d2fff)
                |-- 8q.8.d: \u9974\U00021389
                    \U000b4e5a
                    9(\u80a1\┤b3ec\┤b7°3\┤4711
                ≠↑↑ ┴°≥d⎽┼11⎺┬│d: \┤7▮b7
                    \┤▮167H
                ◆↑↑ ≥⎼2↓⎺┼↑7d↓☃⎽: \┤17bb\U▮▮▮21588\U▮▮▮44cdc│
            ≠↑↑ ┐☃┼d: C⎺┼°☃±M▒⎻
            ◆↑↑ └e├▒d▒├▒:
                ≠↑↑ ▒┼┼⎺├▒├☃⎺┼⎽:
                ≠↑↑ c┌┤⎽├e⎼N▒└e: N⎺┼e
                ≠↑↑ de┌e├☃⎺┼G⎼▒cePe⎼☃⎺dSec⎺┼d⎽: N⎺┼e
                ≠↑↑ °☃┼▒┌☃≥e⎼⎽: []
                ≠↑↑ ±e┼e⎼▒├eN▒└e: N⎺┼e
                ≠↑↑ ±e┼e⎼▒├☃⎺┼: N⎺┼e
                ≠↑↑ ┌▒be┌⎽:
                    ≠↑↑ b: ┐
                    ≠↑↑ ─↓b┴┤▮├e26↓d▮49/⎽┐5⎺☃: 67▒e1c
                    ≠↑↑ ⎼±┌°└⎽┌9│: ├c
...
FAILED (e⎼⎼⎺⎼⎽=5← ⎽┤cce⎽⎽e⎽=84)
(├│┐┤be) e│▒⎼┐┤┼@b▒⎼≤⎺┼:·/W⎺⎼┐/Le▒⎽├A┤├▒⎺⎼☃├≤/├│┐┤be$

Now my terminal is unusable and the latter portion of the tree report has all been corrupted and is garbage.

eliottree should take care in formatting output so as not to provoke this behavior from terminals.

Add an option to print full failure messages

In this example, the interesting error is

exceptions.IOError: [Errno 2] No such file or directory: '/etc/flocker/agent.yml'\n"

But Eliot tree doesn't show it.

(1988)(flocker-ca-output-FLOC-1988 ? =)[~/.../HybridLogic/flocker]$ flocker-dataset-agent  > log
(1988)(flocker-ca-output-FLOC-1988 ? =)[~/.../HybridLogic/flocker]$ cat log
{"task_uuid": "cdeb220d-7605-4d5f-8341-1a170222e308", "error": false, "timestamp": 1432731519.112539, "message": "Log opened.", "message_type": "twisted:log", "task_level": [1]}
{"status": 1, "zfs_command": "zfs set readonly=on flocker", "output": "[Errno 2] No such file or directory", "timestamp": 1432731519.117719, "message_type": "filesystem:zfs:error", "task_uuid": "cdeb220d-7605-4d5f-8341-1a170222e308", "task_level": [2]}
{"status": 1, "zfs_command": "zfs set canmount=off flocker", "output": "[Errno 2] No such file or directory", "timestamp": 1432731519.122177, "message_type": "filesystem:zfs:error", "task_uuid": "cdeb220d-7605-4d5f-8341-1a170222e308", "task_level": [3]}
{"task_uuid": "cdeb220d-7605-4d5f-8341-1a170222e308", "error": true, "timestamp": 1432731519.123826, "message": "Unhandled Error\nTraceback (most recent call last):\n  File \"/home/richard/projects/HybridLogic/flocker/flocker/node/script.py\", line 404, in flocker_dataset_agent_main\n    options=options,\n  File \"/home/richard/projects/HybridLogic/flocker/flocker/common/script.py\", line 187, in main\n    self._react(run_and_log, [], _reactor=self._reactor)\n  File \"/home/richard/.virtualenvs/1988/lib/python2.7/site-packages/twisted/internet/task.py\", line 875, in react\n    finished = main(_reactor, *argv)\n  File \"/home/richard/projects/HybridLogic/flocker/flocker/common/script.py\", line 178, in run_and_log\n    d = maybeDeferred(self.script.main, reactor, options)\n--- <exception caught here> ---\n  File \"/home/richard/.virtualenvs/1988/lib/python2.7/site-packages/twisted/internet/defer.py\", line 140, in maybeDeferred\n    result = f(*args, **kw)\n  File \"/home/richard/projects/HybridLogic/flocker/flocker/volume/service.py\", line 452, in main\n    return self._volume_script.main(reactor, options, service)\n  File \"/home/richard/projects/HybridLogic/flocker/flocker/node/script.py\", line 366, in main\n    configuration = yaml.safe_load(agent_config.getContent())\n  File \"/home/richard/.virtualenvs/1988/lib/python2.7/site-packages/twisted/python/filepath.py\", line 290, in getContent\n    fp = self.open()\n  File \"/home/richard/.virtualenvs/1988/lib/python2.7/site-packages/twisted/python/filepath.py\", line 845, in open\n    return open(self.path, mode + 'b')\nexceptions.IOError: [Errno 2] No such file or directory: '/etc/flocker/agent.yml'\n", "message_type": "twisted:log", "task_level": [4]}
{"task_uuid": "cdeb220d-7605-4d5f-8341-1a170222e308", "error": true, "timestamp": 1432731519.124293, "message": "main function encountered error\nTraceback (most recent call last):\n  File \"/home/richard/projects/HybridLogic/flocker/flocker/node/script.py\", line 404, in flocker_dataset_agent_main\n    options=options,\n  File \"/home/richard/projects/HybridLogic/flocker/flocker/common/script.py\", line 187, in main\n    self._react(run_and_log, [], _reactor=self._reactor)\n  File \"/home/richard/.virtualenvs/1988/lib/python2.7/site-packages/twisted/internet/task.py\", line 875, in react\n    finished = main(_reactor, *argv)\n  File \"/home/richard/projects/HybridLogic/flocker/flocker/common/script.py\", line 178, in run_and_log\n    d = maybeDeferred(self.script.main, reactor, options)\n--- <exception caught here> ---\n  File \"/home/richard/.virtualenvs/1988/lib/python2.7/site-packages/twisted/internet/defer.py\", line 140, in maybeDeferred\n    result = f(*args, **kw)\n  File \"/home/richard/projects/HybridLogic/flocker/flocker/volume/service.py\", line 452, in main\n    return self._volume_script.main(reactor, options, service)\n  File \"/home/richard/projects/HybridLogic/flocker/flocker/node/script.py\", line 366, in main\n    configuration = yaml.safe_load(agent_config.getContent())\n  File \"/home/richard/.virtualenvs/1988/lib/python2.7/site-packages/twisted/python/filepath.py\", line 290, in getContent\n    fp = self.open()\n  File \"/home/richard/.virtualenvs/1988/lib/python2.7/site-packages/twisted/python/filepath.py\", line 845, in open\n    return open(self.path, mode + 'b')\nexceptions.IOError: [Errno 2] No such file or directory: '/etc/flocker/agent.yml'\n", "message_type": "twisted:log", "task_level": [5]}
{"task_uuid": "cdeb220d-7605-4d5f-8341-1a170222e308", "error": false, "timestamp": 1432731519.12443, "message": "Main loop terminated.", "message_type": "twisted:log", "task_level": [6]}
(1988)(flocker-ca-output-FLOC-1988 ? =)[~/.../HybridLogic/flocker]$ eliot-tree log 
cdeb220d-7605-4d5f-8341-1a170222e308
    +-- twisted:log@1/None
        |-- error: False
        |-- message: Log opened.
        |-- message_type: twisted:log
        `-- timestamp: 2015-05-27 13:58:39.112539
    +-- filesystem:zfs:error@2/None
        |-- message_type: filesystem:zfs:error
        |-- output: [Errno 2] No such file or directory
        |-- status: 1
        |-- timestamp: 2015-05-27 13:58:39.117719
        `-- zfs_command: zfs set readonly=on flocker
    +-- filesystem:zfs:error@3/None
        |-- message_type: filesystem:zfs:error
        |-- output: [Errno 2] No such file or directory
        |-- status: 1
        |-- timestamp: 2015-05-27 13:58:39.122177
        `-- zfs_command: zfs set canmount=off flocker
    +-- twisted:log@4/None
        |-- error: True
        |-- message: Unhandled Error...
        |-- message_type: twisted:log
        `-- timestamp: 2015-05-27 13:58:39.123826
    +-- twisted:log@5/None
        |-- error: True
        |-- message: main function encountered error...
        |-- message_type: twisted:log
        `-- timestamp: 2015-05-27 13:58:39.124293
    +-- twisted:log@6/None
        |-- error: False
        |-- message: Main loop terminated.
        |-- message_type: twisted:log
        `-- timestamp: 2015-05-27 13:58:39.124430

It'd be nice if it had an option to show full un-elipsized (is that the term) message field contents.
Something like journalctl s --full option, maybe?

Or have a way to show twisted:log@5/None in full?

Failure trying to parse a task tree with missing messages

This often happens because of log rotation; an operation that straddles midnight will have the start action go into the previous day's log, but the end action into the next day's log.

The traceback looks like this:

Traceback (most recent call last):
  File "/appenv/bin/eliot-tree", line 11, in <module>
    sys.exit(main())
  File "/appenv/local/lib/python2.7/site-packages/eliottree/_cli.py", line 108, in main
    display_task_tree(args)
  File "/appenv/local/lib/python2.7/site-packages/eliottree/_cli.py", line 58, in display_task_tree
    human_readable=args.human_readable)
  File "/appenv/local/lib/python2.7/site-packages/eliottree/_cli.py", line 46, in build_task_nodes
    return tree.nodes(tree.merge_tasks(tasks, filter_funcs()))
  File "/appenv/local/lib/python2.7/site-packages/eliottree/tree.py", line 176, in merge_tasks
    pending = _merge(tasks)
  File "/appenv/local/lib/python2.7/site-packages/eliottree/tree.py", line 170, in _merge
    node.add_child(_TaskNode(task))
  File "/appenv/local/lib/python2.7/site-packages/eliottree/tree.py", line 80, in add_child
    _add_child(self, node.task['task_level'])
  File "/appenv/local/lib/python2.7/site-packages/eliottree/tree.py", line 77, in _add_child
    _add_child(children[level], levels)
  File "/appenv/local/lib/python2.7/site-packages/eliottree/tree.py", line 74, in _add_child
    level = levels.pop(0)
IndexError: pop from empty list

I didn't construct a minimal reproduction yet, but @jonathanj has access to the data that I used; if someone else wants to look into this, just yell at me, and I'll come up with an example.

Handle exceptions during formatting more gracefully

Sometimes timestamps are missing / None for example:

  File "/home/mithrandi/code/eliottree/eliottree/format.py", line 47, in _format_timestamp_value
    result = datetime.utcfromtimestamp(float(value)).isoformat(' ')
TypeError: float() argument must be a string or a number

--raw will work around this but then you get no formatting for anything.

`iso8601.iso8601.Utc` is not reliably available

Version 0.1.12 doesn't expose this for Python 3.2.0 and newer, it looks like this is not really a public export anyway and we ought to be using UTC which is an instance of a UTC tzinfo implementation.

Give me a way to select a time range of log messages

I have a big chunk of eliot logs. I have another log file with timestamps in it like "2015-10-29 12:45:16+0000" that identify interesting time ranges. I'd like to use eliot-tree (among other tools, perhaps) to examine the eliot logs specifically from those time ranges.

Something like:

eliot-tree --start "2015-10-29 12:45:16+0000" --end "..."

would be pretty great.

Something using the --select feature may well do the job too (though it's probably guaranteed to involve more typing - but shell aliases or something I guess 😉).

I tried --select "timestamp > ..." but even with a ton of extra support code to convert date formats and such, eliot-tree (or jmespath, dunno) eventually gets angry at comparison between a datetime and an int.

Parse errors lack sufficient context to track down the problem log/data

When eliot-tree encounters a parse error of some kind (perhaps the json is malformed, perhaps the structure of the encoded object violates some assumption, etc) eliot-tree mostly just tosses up an unhandled exception to the user. The exception may give some general idea about what went wrong (eg KeyError for timestamp - not bad - or some more obscure error from the json parser) but it doesn't seem to give much useful information about where it went wrong.

A line number along with such error reports would go a long way towards helping fix these problems. Better exception formatting for common cases (like "I wanted this key and it was missing") would be pretty nice too.

Prettier tree output

Over the weekend, I hacked together jml/tree-format, which renders Python trees in the format of the Unix tree command, e.g.

foo
├── bar
│   ├── a
│   └── b
├── baz
└── qux
    └── c

Would you be interested in a patch that adds this mode of output to eliottree?

Render without graphics characters or display properly in less(1)

Maybe his is a help request for less(1), and not a complaint about eliot-tree, but I think it might be handy if either:

  • eliot-tree could be coaxed into rendering trees without VT100 graphics characters

or

  • mention could be made in the README of how to get less(1) to properly render those characters.

Error occur when using Eliot-tree from docker container

Hi, I'm using Eliot-tree and it works very well.
The error occurs when I'm logging into a running docker container, and trying to produce the structured logs with Eliot-tree.
I'm getting the following error:

Screen Shot 2020-10-01 at 15 33 12

I'm using ubuntu 18.04, Docker version 19.03.12.

Any idea why do I get this error when using Eliot-tree inside a container?
Thanks!

Ability to customize default behaviour and colors

Sometimes the default colors don't work out well on certain themes (#78), sometimes a user might want a certain flag to be the default (--ascii), etc.

A comprehensive config file is probably the right solution for this.

Unicode error when piping to file

Hi, I'm trying to pipe the output of eliot-tree to a file on Windows 10, but I'm getting the following traceback:

Traceback (most recent call last):
  File "c:\users\zach\appdata\local\programs\python\python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\zach\appdata\local\programs\python\python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\Zach\AppData\Local\pypoetry\Cache\virtualenvs\ns-trcd-5J_fInKt-py3.7\Scripts\eliot-tree.exe\__main__.py", line 9, in <module>
  File "c:\users\zach\appdata\local\pypoetry\cache\virtualenvs\ns-trcd-5j_finkt-py3.7\lib\site-packages\eliottree\_cli.py", line 318, in main
    theme_overrides=config.get('theme_overrides'))
  File "c:\users\zach\appdata\local\pypoetry\cache\virtualenvs\ns-trcd-5j_finkt-py3.7\lib\site-packages\eliottree\_cli.py", line 126, in display_tasks
    theme=theme)
  File "c:\users\zach\appdata\local\pypoetry\cache\virtualenvs\ns-trcd-5j_finkt-py3.7\lib\site-packages\eliottree\_render.py", line 288, in render_tasks
    write(format_tree(task, _format_node, _get_children, make_options()))
  File "c:\users\zach\appdata\local\programs\python\python37\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u21d2' in position 71: character maps to <undefined>

The file I'm using looks like this:

{"action_status": "started", "timestamp": 1582840009.482032, "task_uuid": "6ffa3c9b-3e24-4ee7-bb57-2e40e01171b6", "action_type": "_set_initial_widget_states", "task_level": [1]}
{"new_max": 5, "action_status": "started", "timestamp": 1582840009.482032, "task_uuid": "6ffa3c9b-3e24-4ee7-bb57-2e40e01171b6", "action_type": "update_max_measurements", "task_level": [2, 1]}
{"action_status": "succeeded", "timestamp": 1582840009.482032, "task_uuid": "6ffa3c9b-3e24-4ee7-bb57-2e40e01171b6", "action_type": "update_max_measurements", "task_level": [2, 2]}
{"action_status": "succeeded", "timestamp": 1582840009.482032, "task_uuid": "6ffa3c9b-3e24-4ee7-bb57-2e40e01171b6", "action_type": "_set_initial_widget_states", "task_level": [3]}
{"new_max": 50, "action_status": "started", "timestamp": 1582840013.952636, "task_uuid": "ab19b06e-fb4d-406f-be82-109b5530df3b", "action_type": "update_max_measurements", "task_level": [1]}
{"action_status": "succeeded", "timestamp": 1582840013.952636, "task_uuid": "ab19b06e-fb4d-406f-be82-109b5530df3b", "action_type": "update_max_measurements", "task_level": [2]}
{"action_status": "started", "timestamp": 1582840018.5424337, "task_uuid": "9a2da24e-bf98-442e-b158-7b84845f7a0b", "action_type": "set_time_axis", "task_level": [1]}
{"action_status": "succeeded", "timestamp": 1582840018.5424337, "task_uuid": "9a2da24e-bf98-442e-b158-7b84845f7a0b", "action_type": "set_time_axis", "task_level": [2]}
{"action_status": "started", "timestamp": 1582840023.4722626, "task_uuid": "6c6ecdac-b624-4e88-849a-91ffee883af5", "action_type": "close", "task_level": [1]}
{"action_status": "succeeded", "timestamp": 1582840023.4722626, "task_uuid": "6c6ecdac-b624-4e88-849a-91ffee883af5", "action_type": "close", "task_level": [2]}

I've tried making the encoding process as simple as possible by using the --ascii and --color never flags, but the issue persists. Am I missing something simple, or is this a bug?

Rendering problems with updated tree code

The new code apparently always assumes message 1 is the parent... but sometimes it isn't. In particular in the root of the action tree it might be a standalone message.

 $ python reproducer.py | eliot-tree 
b201ac9a-f6bb-4660-a0b4-c0ad2646659b
+-- sibling1@1
    `-- timestamp: 2015-08-08 14:52:33.515778
    +-- sibling2@2
        `-- timestamp: 2015-08-08 14:52:33.516109

$ cat reproducer.py 
from eliot import Message, to_file
from sys import stdout
to_file(stdout)

Message.log(message_type="sibling1")
Message.log(message_type="sibling2")

Error running eliot-tree on Python 3

Versions:

eliot (0.8.0)
eliot-tree (15.3.0)

Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 23 2015, 02:52:03)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin

%> uname -a
Darwin dmt 14.5.0 Darwin Kernel Version 14.5.0: Wed Jul 29 02:26:53 PDT 2015; root:xnu-2782.40.9~1/RELEASE_X86_64 x86_64
%> eliot-tree
Traceback (most recent call last):
  File "/Users/vg/Devel/DetailsAuto/new/backend/.venv/bin/eliot-tree", line 9, in <module>
    load_entry_point('eliot-tree==15.3.0', 'console_scripts', 'eliot-tree')()
  File "/Users/vg/Devel/DetailsAuto/new/backend/.venv/lib/python3.4/site-packages/pkg_resources/__init__.py", line 474, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/Users/vg/Devel/DetailsAuto/new/backend/.venv/lib/python3.4/site-packages/pkg_resources/__init__.py", line 2582, in load_entry_point
    return ep.load()
  File "/Users/vg/Devel/DetailsAuto/new/backend/.venv/lib/python3.4/site-packages/pkg_resources/__init__.py", line 2265, in load
    return self._load()
  File "/Users/vg/Devel/DetailsAuto/new/backend/.venv/lib/python3.4/site-packages/pkg_resources/__init__.py", line 2268, in _load
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/Users/vg/Devel/DetailsAuto/new/backend/.venv/lib/python3.4/site-packages/eliottree/__init__.py", line 3, in <module>
    from eliottree.tree import Tree
  File "/Users/vg/Devel/DetailsAuto/new/backend/.venv/lib/python3.4/site-packages/eliottree/tree.py", line 129
    return sorted(nodes, key=lambda (_, n): n.task[u'timestamp'])
                                    ^
SyntaxError: invalid syntax

Argument Error when use eliot tree

Test code:

from eliot import log_call, to_file

to_file(open('log.log', 'w'))

@log_call(action_type='CALC', include_args=['x'], include_result=True)
def calculate(x, y):
    return x * y

x = 2
y = 10
result = calculate(x, y)
print(result)

Then use eliot-tree log.log it returns an Argument Error.

Exceptions (1) occurred during processing:
Traceback (most recent call last):
  File "C:\Users\paule\Anaconda3\lib\site-packages\toolz\functoolz.py", line 781, in __call__
    return self.func(*args, **kwargs)
  File "C:\Users\paule\Anaconda3\lib\site-packages\eliottree\_render.py", line 117, in format_node
    format.escape_control_characters(node.root().task_uuid)))
  File "C:\Users\paule\Anaconda3\lib\site-packages\eliottree\_color.py", line 43, in __color
    return colored(text, fg, bg, attrs=attrs)
  File "C:\Users\paule\Anaconda3\lib\site-packages\eliottree\_color.py", line 30, in colored
    _colored.fg(fg) if fg is not None else u'',
  File "C:\Users\paule\Anaconda3\lib\site-packages\colored\colored.py", line 402, in fg
    return colored(color).foreground()
  File "C:\Users\paule\Anaconda3\lib\site-packages\colored\colored.py", line 22, in __init__
    self.enable_windows_terminal_mode()
  File "C:\Users\paule\Anaconda3\lib\site-packages\colored\colored.py", line 373, in enable_windows_terminal_mode
    hStdout = windll.kernel32.GetStdHandle(STD_OUTPUT_HANDLE)
ctypes.ArgumentError: argument 1: <class 'TypeError'>: wrong type

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.