Coder Social home page Coder Social logo

opencyphal / yakut Goto Github PK

View Code? Open in Web Editor NEW
44.0 44.0 10.0 14.01 MB

Simple CLI tool for diagnostics and debugging of Cyphal networks

Home Page: https://opencyphal.org

License: MIT License

Python 100.00%
aerospace cli command-line-tool cyphal data-distribution diagnostics distributed-computing drone dsdl embedded ethernet hacktoberfest networking opencyphal pubsub real-time robotics rpc uavcan vehicular-networks

yakut's People

Contributors

clyde-johnston avatar coderkalyan avatar maksimdrachov avatar pavel-kirienko avatar silverv avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

yakut's Issues

Plotter command

Implement a new command that plots a specified set of fields on a graph in real-time. The syntax should be similar to yakut subscribe with the addition that the user should be able to specify the visualization data mapping. In the simplest form, the usage should be like:

yakut plot SUBJECT [SUBJECT]...

Where SUBJECT follows [SUBJECT_ID:]TYPE_NAME.MAJOR.MINOR like in the existing commands yakut subscribe et al. This should result in all fields of the message being plotted on a 2D graph where X is the local time and Y are the values.

Example -- 5 values on a simple 2D time-Y plot:

yakut plot 2345:reg.drone.physics.kinematics.rotation.Planar.0.1 2346:reg.drone.physics.electricity.Power.0.1

Simple time-Y plots are useful but not sufficient for non-trivial use cases. It should be possible to map values to axes explicitly when necessary like:

yakut plot                                                          \
    4444:reg.drone.physics.kinematics.cartesian.PointStateVarTs.0.1 \
    5555:reg.drone.physics.electricity.PowerTs.0.1                  \
    --line=a.value.position.value.meter                             \
    --line=b.timestamp,b.value

Where a refers to the first subscription, b second, and so on; the values of --line are arbitrary Python expressions fed into eval() whose results are then flattened to obtain a tuple of lists/scalars which are then forwarded to the plotting engine. This example should result in two plots being shown: one 3D visualizing the coordinates from subject 4444, and one 2D time-Y similar to the above example. Other types of plots can be added later as necessary (e.g., --scatter).

The preferred plotting backend is Plotly (supports real-time streaming with uirevision).

For giggles it would be interesting to render the output in the terminal using the sixel format like GNUplot does it (via Kaleido) although this is expected to be much less useful than the conventional browser-based UX offered by Plotly out of the box.

image

Would anybody like to work on this?

Yakut is very slow on windows

yakut monitor is very slow to refresh on windows.

It takes about 10 seconds to begin rendering and another 10 seconds for all the characters to apear in the terminal. Subsequent frames are also slow.

Tested on both cmd.exe and git bash.

Machine: Windows 10 on i7-7700

Yakut pub/sub/client should allow specifying the subject/service by name/ID instead of separate port-ID and type

Instead of this:

yakut sub 33:uavcan.si.unit.angle.Scalar.1.0

I want to be able to say this:

yakut sub angle

Yakut should go ask every online node if there are registers named uavcan.pub.angle.id and uavcan.pub.angle.type. If there are multiple nodes with that register and the port-IDs don't match, report an ambiguity error and ask the user to clarify which specific node is of interest:

yakut sub angle@123

If all nodes share the same port configuration, pick the first one whose data type is known to the local Yakut instance and subscribe to that.

This resembles the experience with traditional pub/sub frameworks with decentralization.

The same applies to yakut publish. The case of yakut call is simpler because we already have the node-ID to work with, hence we won't have to scan the network to find the service-ID.

Further, I want to be able to specify the port-ID only and omit the type, letting Yakut infer the type automatically:

yakut sub 33

In this case, it will have to subscribe to this topic using uavcan.primitive.Empty.1.0 (relying on structural subtyping) and observe which nodes publish on this topic. Then scan the registers and determine the type. Then re-subscribe using the new type. Eventually, we should extend the standard node API that allows querying the type from ID without the need to fetch all registers.

The same applies to yakut publish.

Support joystick/knobs in yakut publish

When debugging certain kinds of hardware it is often desirable to be able to manipulate the published values in real-time and observe how the system responds. yakut publish should be able to source inputs from a specified input device and use that to populate the fields in the published messages.

image

The field values are to be specified using a custom YAML tag with an arithmetic expression that is fed into eval(). The name of the tag is the index/name of the input device to use.

For example, this is how one would currently publish a constant value:

yakut pub 33:uavcan.si.unit.angle.Scalar.1.0 'radian: 2.31'

With the input device one would instead write something like:

yakut pub 33:uavcan.si.unit.angle.Scalar.1.0 'radian: !0 (a+1)*3.14'

Where !0 is a custom YAML tag that specifies which device to use (first device in this case), and a is the value of the first channel read from the input device normalized into [-1, +1]. This is handled by Ruamel YAML (the YAML library used by Yakut) as follows:

>>> ruamel.yaml.YAML().load('radian: !0 (a+1)*3.14')
ordereddict([('radian', <ruamel.yaml.comments.TaggedScalar object at 0x7f24fb7bea60>)])
>>> ruamel.yaml.YAML().load('radian: !0 (a+1)*3.14')['radian'].__dict__
{'value': '(a+1)*3.14', 'style': None, '_yaml_tag': Tag('!0')}

Relevant:

Yakut uavcan.file.GetInfo.0.1 Path.1.0 stringsserialization

Path1.0 is using the uint8 date type but passing strings to it using Yakut doesn't work.

hovergames@hovergames:~/uavcan$ yakut call 42 uavcan.file.GetInfo.0.1 '{path: "/cool/file/here"}'
Traceback (most recent call last):
  File "/home/hovergames/.local/bin/yakut", line 11, in <module>
    sys.exit(main())
  File "/home/hovergames/.local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/hovergames/.local/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/hovergames/.local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/hovergames/.local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/hovergames/.local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/hovergames/.local/lib/python3.7/site-packages/click/decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/home/hovergames/.local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/hovergames/.local/lib/python3.7/site-packages/yakut/cmd/call.py", line 122, in call
    request = pyuavcan.dsdl.update_from_builtin(dtype.Request(), request_fields)
  File "/home/hovergames/.local/lib/python3.7/site-packages/pyuavcan/dsdl/_builtin_form.py", line 150, in update_from_builtin
    update_from_builtin(field_obj, value)
  File "/home/hovergames/.local/lib/python3.7/site-packages/pyuavcan/dsdl/_builtin_form.py", line 126, in update_from_builtin
    f"Invalid format: cannot update an instance of type {type(destination).__name__!r} "
TypeError: Invalid format: cannot update an instance of type 'Path_1_0' from value of type 'str'
hovergames@hovergames:~/uavcan$ 

Capture/tracing

EDIT 2021: see https://forum.uavcan.org/t/uavcan-v1-wireshark-plugin/1058/2?u=pavel.kirienko

We need the ability to log network frames into dump files and read them back for later analysis. This feature is vital for various R&D and flight recording purposes.

image

The feature has been discussed first in this thread: https://forum.uavcan.org/t/gui-tool-next-generation/229/2?u=pavel.kirienko. At the application level, its purpose is close to that of ROS bags: http://wiki.ros.org/Bags.

First, it was proposed to define a dedicated format, where log files are flat binaries (no global structure) consisting of an ordered set of frames: https://docs.google.com/spreadsheets/d/1yP9zXChKTaIm92Bd60jgrOSwGIeXp-CO5tGyYcaBopg/edit. As explained on the forum in the linked above thread, lack of global structure (e.g., no file header) is desirable because it allows trivial manipulations over log files (which can be huge) to be performed by naive concatenation/truncation/splitting.

Later it was observed that the log file frame format and a then-hypothetical serial link frame format would be largely identical because flat log files and serial links are structurally similar: https://forum.uavcan.org/t/yukon-design-megathread/390/115?u=pavel.kirienko. So we can use the existing (albeit still extremely experimental) serial transport for log file storage. At the architectural level, the only missing piece is the set of DSDL definitions describing transport frame data formats for various transports. Once such definitions are available, the task of log recording can be defined as listening to the network traffic in promiscuous mode, encoding each received frame into a DSDL object, and then serializing that as a message transfer over serial transport. The resulting byte sequence is then appended to the log file. Likewise, it is possible to log arbitrary metadata by using appropriate DSDL message types.

Shall this approach prove successful, we could expand this to become a de-facto standard for data logging not only by software tools but also by hardware data recorders.

Remove need for libjack and libasound2

There's no good reason why I need to install support for multi-media libraries to run yakut. This makes it difficult to install in thin platforms that don't come with such support. My best guess is that this is coming in through python-rtmidi. I'd suggest either removing that dependency or finding a way to use with without audio support.

Yakut Ubuntu 18.04 compatibility issues

Ubuntu 18.04 running Python 3.6.9 gives SyntaxError: future feature annotations is not defined error I don't see any mention which minimum version of python is required so might not be a bug.

Furthermore I've got Python 3.7.5 running which is working but compiling DSDL gives the following error message

hovergames@hovergames:~/uavcan$ yakut compile  ./public_regulated_data_types/uavcan  ./public_regulated_data_types/reg
Traceback (most recent call last):
  File "/home/hovergames/.local/bin/yakut", line 11, in <module>
    sys.exit(main())
  File "/home/hovergames/.local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/hovergames/.local/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/hovergames/.local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/hovergames/.local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/hovergames/.local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/hovergames/.local/lib/python3.7/site-packages/yakut/cmd/compile.py", line 114, in compile_
    allow_unregulated_fixed_port_id=allow_unregulated_fixed_port_id,
  File "/home/hovergames/.local/lib/python3.7/site-packages/yakut/cmd/compile.py", line 184, in _generate_dsdl_packages
    allow_unregulated_fixed_port_id=allow_unregulated_fixed_port_id,
  File "/home/hovergames/.local/lib/python3.7/site-packages/pyuavcan/dsdl/_compiler.py", line 199, in generate_package
    allow_unregulated_fixed_port_id=allow_unregulated_fixed_port_id,
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/_namespace.py", line 191, in read_namespace
    allow_unregulated_fixed_port_id)
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/_namespace.py", line 239, in _read_namespace_definitions
    raise ex
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/_namespace.py", line 236, in _read_namespace_definitions
    allow_unregulated_fixed_port_id)
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/_dsdl_definition.py", line 149, in read
    raise ex
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/_dsdl_definition.py", line 137, in read
    _parser.parse(f.read(), builder)
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/_parser.py", line 36, in parse
    raise ex
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/_parser.py", line 30, in parse
    pr.parse(text)  # type: ignore
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 254, in parse
    return self._parse_or_match(text, pos, 'parse')
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 289, in _parse_or_match
    return self.visit(getattr(self.grammar, method_name)(text, pos=pos))
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in visit
    return method(node, [self.visit(n) for n in node])
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in <listcomp>
    return method(node, [self.visit(n) for n in node])
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in visit
    return method(node, [self.visit(n) for n in node])
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in <listcomp>
    return method(node, [self.visit(n) for n in node])
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in visit
    return method(node, [self.visit(n) for n in node])
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in <listcomp>
    return method(node, [self.visit(n) for n in node])
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in visit
    return method(node, [self.visit(n) for n in node])
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in <listcomp>
    return method(node, [self.visit(n) for n in node])
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in visit
    return method(node, [self.visit(n) for n in node])
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in <listcomp>
    return method(node, [self.visit(n) for n in node])
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in visit
    return method(node, [self.visit(n) for n in node])
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in <listcomp>
    return method(node, [self.visit(n) for n in node])
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in visit
    return method(node, [self.visit(n) for n in node])
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in <listcomp>
    return method(node, [self.visit(n) for n in node])
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in visit
    return method(node, [self.visit(n) for n in node])
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in <listcomp>
    return method(node, [self.visit(n) for n in node])
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in visit
    return method(node, [self.visit(n) for n in node])
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in <listcomp>
    return method(node, [self.visit(n) for n in node])
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in visit
    return method(node, [self.visit(n) for n in node])
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in <listcomp>
    return method(node, [self.visit(n) for n in node])
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in visit
    return method(node, [self.visit(n) for n in node])
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/_parser.py", line 235, in visit_type_versioned
    return self._statement_stream_processor.resolve_versioned_data_type(name, version)
  File "/home/hovergames/.local/lib/python3.7/site-packages/pydsdl/_data_type_builder.py", line 195, in resolve_versioned_data_type
    raise _error.InternalError('Conflicting definitions: %r' % found)
pydsdl._error.InternalError: /home/hovergames/uavcan/public_regulated_data_types/uavcan/file/405.GetInfo.0.2.uavcan:3: Conflicting definitions: [DSDLDefinition(full_name='uavcan.file.Path', version=Version(major=2, minor=0), fixed_port_id=None, file_path='/home/hovergames/uavcan/public_regulated_data_types/uavcan/file/Path.2.0.uavcan'), DSDLDefinition(full_name='uavcan.file.Path', version=Version(major=2, minor=0), fixed_port_id=None, file_path='/home/hovergames/uavcan/public_regulated_data_types/uavcan/file/Path.2.0.uavcan')]
hovergames@hovergames:~/uavcan$ 

e2e tests are flaky

End-to-end tests are time-sensitive and so they expect a certain degree of responsiveness from the OS and the Python runtime. The CI environment is not always responsive enough and so tests tend to fail randomly. Obviously, each separate test may seem robust enough, but we have 102 of them run separately per Python version, so at the moment CI jobs are rarely green ever.

We could go the hard way and improve the tests, which I don't want to do unless I must, or we could set up a self-hosted system that will hopefully perform better than the cloud runner.

Extend the installation docs with a recommendation on how to initialize the environment variables

Example for SLCAN:

# Common UAVCAN register configuration for testing & debugging.
# Source this file into your sh/bash/zsh session before using Yakut and other UAVCAN tools.
# Other helpful commands:
#   canbusload -tcbr slcan0@1000000 slcan1@1000000
#   candump -decaxta any

export UAVCAN__CAN__IFACE='socketcan:slcan0 socketcan:slcan1'
export UAVCAN__CAN__MTU=8
export UAVCAN__NODE__ID=$(yakut accommodate)

echo "Auto-selected node-ID for this session: $UAVCAN__NODE__ID"

function init_slcan()
{
    # https://gist.github.com/pavel-kirienko/32e395683e8b7f49e71413aebf5e1a89
    sudo setup_slcan -r /dev/serial/by-id/usb-Zubax*Babel*
}

e2e test issues on Windows: stderr buffer too small

On Windows, when the integration test suite runs a process with -v or -vv, it may get blocked at an arbitrary location waiting for the parent to read its stderr. This results in non-obvious timeout errors and it may cost one hours of debugging on Windows while it works on other platforms (don't ask how I know).

The pipes are currently configured to be unbuffered because otherwise, Windows discards the output when the child process is interrupted instead of flushing the streams properly.

Perhaps the child runner should start background tasks/threads to read from the pipes so that the child process is never blocked. The runner is defined in tests/subprocess.py:

class Subprocess:

yakut won't find transcompiled dsdl files

Dear all,

I've installed yakut via pip3 on ubuntu focal. Despite all environmental variables are set correctly in my .bashrc and yakut compile runs without error, I am not able to monitor the UAVCAN network. It seems like yakut won't find the transcompiled files for some reason. (I've tried on 3 different machines with the same results.)

This is what I get: Error: Run `yakut compile <path>/uavcan` to compile DSDL namespace 'uavcan'

Name: yakut
Version: 0.4.1
Summary: Simple CLI tool for diagnostics and debugging of UAVCAN networks.
Home-page: https://uavcan.org
Author: UAVCAN Consortium
Author-email: [email protected]
License: MIT
Location: /home/nils/.local/lib/python3.8/site-packages
Requires: coloredlogs, psutil, pyuavcan, simplejson, click, ruamel.yaml, scipy, requests

Thank you for your help in advance.

yakut reg produces wrong format on output for natural8 with option -d

yakut reg produces wrong format on output for natural8 with option -d

bernhard@notebookR60P-22:~$ y -i 'CAN(can.media.socketcan.SocketCANMedia("slcan0",8),59)' r 42 uavcan.node.id
11                     
bernhard@notebookR60P-22:~$ y -i 'CAN(can.media.socketcan.SocketCANMedia("slcan0",8),59)' r -d 42 uavcan.node.id
natural8: {value: "\v"}
bernhard@notebookR60P-22:~$ y --version
yakut, version 0.11.1

Usability concerns

I don't know if it's just me, but I find the yakut tool a bit inconvenient to use. It requires a lot of initial configuration compared to the old uavcan_gui tool. And this is especially hard for less tech-savvy field operators.

Initial confusion is that the first command given in the README is yakut --path=/the/path compile path/to/my_namespace --output=destination/directory. However, I could not understand why it needs 3 paths for compilation. As far as I understand only the arguments and the --output paths are used, while --path is ignored?

Then README explains to setup environment variables so that default compilation output goes to user data directory. Wouldn't it be better to have this as default instead of current working directory? AFAIK, this is the most common use case.

Would it be possible to ship public regulated data types together with yakut? Or maybe add a command that would download and compile them for you?

Another convienient option would be to load custom DSDLs from user directory and compile them automatically, as was the case with uavcan_gui tool.

I can contribute some of these changes, but firstly wanted to hear some opinions about this.

compile: If the zip archive contains directories that are not root namespaces, compilation may fail

The following command:

y compile https://github.com/Zubax/zubax_dsdl/archive/refs/heads/master.zip https://github.com/OpenCyphal/public_regulated_data_types/archive/refs/heads/master.zip

Fails with:

RootNamespaceNameCollisionError: /tmp/yakut-dsdl-2hyfdl7u/zubax_dsdl-master/.github: The name of this namespace conflicts with /tmp/yakut-dsdl-4z20ljx5/public_regulated_data_types-master/.github

Alert user when updates are available

Yakut should check for new releases via https://pypi.org/pypi/yakut/json and warn the user when a new version is available with an explicit suggestion to run pip install --upgrade yakut.

The check should not be performed more often than once per day and it should be possible to disable it via parameter/environment variable.

`yakut compile` deletes directory initially

It seems that it will delete the output directory first, so if the DSDL namespace you're working on is directly in the working directory, it will delete the directory, and have no files to work on, quitting. It will output a warning, but only after deleting the namespace.

IMO the warning should be changed to an error, with a suggestion to nest the namespaces down a level, if the current behaviour of clearing the output directory is to be kept.

Simple reproduction

git clone https://github.com/UAVCAN/public_regulated_data_types
yakut compile public_regulated_data_types

"yakut monitor" is throwing errors with SocketCAN

Any ideas? I just upgraded pyuavcan, yakut, nunavut, and a number of other dependent packages tonight.

Using a PCAN-USB CAN adapter (shows up as a socketcan interface; works fine with can-utils and yakut {pub|sub}).

Python 3.9.1
Yakut 0.5.2
pyuavcan 1.2.4

NodID Mode Health VSSC Uptime         VProtcl VHardwr VSoftware(major.minor.vcs.crc)            Unique-ID                        Name                    
    1 oper nomina    0     0d00:00:49 ?       ?       ?                                         ?                                ?                       
    3 ?    ?      ?    offline          1.0     0.0     0.5                                     8d5361a31e63e8b36942aea7855b9d8d org.uavcan.yakut.monitor
   20 oper nomina    0     0d00:01:16 ?       ?       ?                                         ?                                ?                       
APPLICATION LAYER CONNECTIVITY MATRIX [t/s=transfer/second] Colors: pub/clnโ”‚sub/srvโ”‚(pub+sub)/(cln+srv)โ”‚activityโ”‚uavcan.node.port.List is published/notโ”‚
MESSG    1   20 โˆ‘t/s โˆ‘B/s     
   22   4         4  242    22
   30   2         2   36    30
   31   2         2   33    31
 7509  30    1   31  220  7509
โˆ‘t/s   38    1   39      โ†– t/s
โˆ‘B/s  525    6       532      
RQ+RS    1   20 โˆ‘t/s โˆ‘B/s     
โˆ‘t/s    0    0    0      โ†– t/s
โˆ‘B/s    0    0         0      
TOTAL    1   20 โˆ‘t/s โˆ‘B/s     
โˆ‘t/s   38    1   39           
โˆ‘B/s  525    6       532      
Total transport layer errors:        0         Values averaged over 10.0 sec
2021-04-21 20:15:55 0023949 ERR pyuavcan.application.heartbeat_publisher: HeartbeatPublisher(heartbeat=uavcan.node.Heartbeat.1.0(uptime=8, health=uavcan.node.Health.1.0(value=0), mode=uavcan.node.Mode.1.0(value=0), vendor_specific_status_code=49), priority=NOMINAL, period=1.0) publisher task exception: [Errno 22] Invalid argument
Traceback (most recent call last):
  File "/home/jacob/Workspace/uavcan-v1/venv/lib/python3.9/site-packages/pyuavcan/application/heartbeat_publisher.py", line 203, in _task_function
    if not await pub.publish(self.make_message()):
  File "/home/jacob/Workspace/uavcan-v1/venv/lib/python3.9/site-packages/pyuavcan/presentation/_port/_publisher.py", line 113, in publish
    return await self._maybe_impl.publish(message, self._priority, self._loop.time() + self._send_timeout)
  File "/home/jacob/Workspace/uavcan-v1/venv/lib/python3.9/site-packages/pyuavcan/presentation/_port/_publisher.py", line 190, in publish
    return await self.transport_session.send(transfer, monotonic_deadline)
  File "/home/jacob/Workspace/uavcan-v1/venv/lib/python3.9/site-packages/pyuavcan/transport/can/_session/_output.py", line 206, in send
    return await self._do_send(can_id, transfer, monotonic_deadline)
  File "/home/jacob/Workspace/uavcan-v1/venv/lib/python3.9/site-packages/pyuavcan/transport/can/_session/_output.py", line 163, in _do_send
    if await self._send_handler(transaction):
  File "/home/jacob/Workspace/uavcan-v1/venv/lib/python3.9/site-packages/pyuavcan/transport/can/_can.py", line 327, in _do_send
    num_sent = await self._maybe_media.send(
  File "/home/jacob/Workspace/uavcan-v1/venv/lib/python3.9/site-packages/pyuavcan/transport/can/media/socketcan/_socketcan.py", line 128, in send
    await asyncio.wait_for(
  File "/home/jacob/local/lib/python3.9/asyncio/tasks.py", line 478, in wait_for
    return fut.result()
  File "/home/jacob/local/lib/python3.9/asyncio/selector_events.py", line 446, in sock_sendall
    n = sock.send(data)
OSError: [Errno 22] Invalid argument
^C
Aborted!
(venv) jacob@jacob-desktop:~/Workspace/uavcan-v1$ python --version
Python 3.9.1
(venv) jacob@jacob-desktop:~/Workspace/uavcan-v1$ yakut --version
yakut, version 0.5.2
(venv) jacob@jacob-desktop:~/Workspace/uavcan-v1$ pip freeze | grep pyuavcan
pyuavcan==1.2.4

Structure metadata should contain the name of the type

$ yakut sub 33:uavcan.si.unit.angle.Scalar.1.0
---
33:
  _metadata_:
    timestamp: {system: 1608987583.298886, monotonic: 788272.540747}
    priority: nominal
    transfer_id: 0
    source_node_id: 42
  radian: 2.309999942779541

We need the type name under _metadata_.

Is there a way to filter out certain CAN IDs in yakut monitor?

Hello might be a dump question, but I have a CAN bus that has UAVCAN and non UAVCAn devices. I know this is not optimal and can cause PnP issues, but I have no other way around it.

I would like to know if it is possible to filter out the other nodes in the yakut monitor without filtering them form socket can, as I use the other nodes for other hardware interfaces.

Any pointer in how to deal with this are very welcome

asyncio.run() terminates the event loop incorrectly

The close() methods should be called properly at the end of each command to avoid resource lifetime issues. The following log is produced at the very end of a successful (sic!) command execution.

2022-01-06 18:47:08 0248354 ERR pyuavcan.transport.redundant._session._output: RedundantOutputSession(OutputSessionSpecifier(data_specifier=ServiceDataSpecifier(service_id=430, role=<Role.RESPONSE: 2>), remote_node_id=21), PayloadMetadata(extent_bytes=448)): Inferior UnicastCANOutputSession(OutputSessionSpecifier(data_specifier=ServiceDataSpecifier(service_id=430, role=<Role.RESPONSE: 2>), remote_node_id=21), PayloadMetadata(extent_bytes=448)) failed: OSError: [Errno 9] Bad file descriptor
2022-01-06 18:47:08 0248354 ERR pyuavcan.transport.redundant._session._output: RedundantOutputSession(OutputSessionSpecifier(data_specifier=ServiceDataSpecifier(service_id=430, role=<Role.RESPONSE: 2>), remote_node_id=21), PayloadMetadata(extent_bytes=448)): Inferior UnicastCANOutputSession(OutputSessionSpecifier(data_specifier=ServiceDataSpecifier(service_id=430, role=<Role.RESPONSE: 2>), remote_node_id=21), PayloadMetadata(extent_bytes=448)) failed: OSError: [Errno 9] Bad file descriptor
2022-01-06 18:47:08 0248354 ERR asyncio: Future exception was never retrieved
future: <Future finished exception=OSError(9, 'Bad file descriptor')>
Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/tasks.py", line 432, in wait_for
    await waiter
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pavel/.local/lib/python3.10/site-packages/pyuavcan/transport/redundant/_session/_output.py", line 327, in _inferior_worker_task
    result = await ses.send(wrk.transfer, wrk.monotonic_deadline)
  File "/home/pavel/.local/lib/python3.10/site-packages/pyuavcan/transport/can/_session/_output.py", line 252, in send
    return await self._do_send(can_id, transfer, monotonic_deadline)
  File "/home/pavel/.local/lib/python3.10/site-packages/pyuavcan/transport/can/_session/_output.py", line 163, in _do_send
    if await self._send_handler(transaction):
  File "/home/pavel/.local/lib/python3.10/site-packages/pyuavcan/transport/can/_can.py", line 319, in _do_send
    num_sent = await self._maybe_media.send(
  File "/home/pavel/.local/lib/python3.10/site-packages/pyuavcan/transport/can/media/socketcan/_socketcan.py", line 137, in send
    raise err
  File "/home/pavel/.local/lib/python3.10/site-packages/pyuavcan/transport/can/media/socketcan/_socketcan.py", line 126, in send
    await asyncio.wait_for(
  File "/usr/lib/python3.10/asyncio/tasks.py", line 435, in wait_for
    return fut.result()
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 446, in sock_sendall
    n = sock.send(data)
OSError: [Errno 9] Bad file descriptor
2022-01-06 18:47:08 0248354 ERR asyncio: Future exception was never retrieved
future: <Future finished exception=OSError(9, 'Bad file descriptor')>
Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/tasks.py", line 432, in wait_for
    await waiter
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pavel/.local/lib/python3.10/site-packages/pyuavcan/transport/redundant/_session/_output.py", line 327, in _inferior_worker_task
    result = await ses.send(wrk.transfer, wrk.monotonic_deadline)
  File "/home/pavel/.local/lib/python3.10/site-packages/pyuavcan/transport/can/_session/_output.py", line 252, in send
    return await self._do_send(can_id, transfer, monotonic_deadline)
  File "/home/pavel/.local/lib/python3.10/site-packages/pyuavcan/transport/can/_session/_output.py", line 163, in _do_send
    if await self._send_handler(transaction):
  File "/home/pavel/.local/lib/python3.10/site-packages/pyuavcan/transport/can/_can.py", line 319, in _do_send
    num_sent = await self._maybe_media.send(
  File "/home/pavel/.local/lib/python3.10/site-packages/pyuavcan/transport/can/media/socketcan/_socketcan.py", line 137, in send
    raise err
  File "/home/pavel/.local/lib/python3.10/site-packages/pyuavcan/transport/can/media/socketcan/_socketcan.py", line 126, in send
    await asyncio.wait_for(
  File "/usr/lib/python3.10/asyncio/tasks.py", line 435, in wait_for
    return fut.result()
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 446, in sock_sendall
    n = sock.send(data)
OSError: [Errno 9] Bad file descriptor

Implement the register read/write command

Using the standard register API.

yakut register 42 -- read all registers from node 42 and print them into stdout.

yakut register 42 foo.bar -- read register foo.bar and print its value.

yakut register 42 foo.bar 123.456 -- write 123.456 into foo.bar. The command may need to send two service requests; the first one is needed to determine the type of the register so that the value can be converted into a representation expected by the server.

This issue is migrated from OpenCyphal/pycyphal#56

file-server --force-update 1234

Relates to #27

The file-server command should provide an option to force a BEGIN_SOFTWARE_UPDATE request to a specific node bypassing all checks. Perhaps this could be done via an additional option, like --force-update=1234.

Flaky test

The test suite randomly fails on Windows as:

Traceback (most recent call last):
  File "c:\projects\yakut\.nox\test-3-9\lib\site-packages\serial\urlhandler\protocol_socket.py", line 63, in open
    self._socket = socket.create_connection(self.from_url(self.portstr), timeout=POLL_TIMEOUT)
  File "c:\python39-x64\lib\socket.py", line 843, in create_connection
    raise err
  File "c:\python39-x64\lib\socket.py", line 831, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "C:\projects\yakut\tests\cmd\monitor.py", line 207, in _run_nodes
    instantiate(
  File "C:\projects\yakut\tests\cmd\monitor.py", line 199, in instantiate
    node = make_node(info, reg)
  File "c:\projects\yakut\.nox\test-3-9\lib\site-packages\pyuavcan\application\_node_factory.py", line 199, in make_node
    node = SimpleNode(pyuavcan.presentation.Presentation(init_transport()), info, registry)
  File "c:\projects\yakut\.nox\test-3-9\lib\site-packages\pyuavcan\application\_node_factory.py", line 172, in init_transport
    out = make_transport(registry, reconfigurable=reconfigurable_transport)
  File "c:\projects\yakut\.nox\test-3-9\lib\site-packages\pyuavcan\application\_transport_factory.py", line 215, in make_transport
    transports = list(itertools.chain(*(f(registers, node_id) for f in _SPECIALIZATIONS)))
  File "c:\projects\yakut\.nox\test-3-9\lib\site-packages\pyuavcan\application\_transport_factory.py", line 263, in _make_serial
    yield SerialTransport(str(port), node_id, service_transfer_multiplier=srv_mult, baudrate=baudrate)
  File "c:\projects\yakut\.nox\test-3-9\lib\site-packages\pyuavcan\transport\serial\_serial.py", line 147, in __init__
    serial_port = serial.serial_for_url(serial_port)
  File "c:\projects\yakut\.nox\test-3-9\lib\site-packages\serial\__init__.py", line 90, in serial_for_url
    instance.open()
  File "c:\projects\yakut\.nox\test-3-9\lib\site-packages\serial\urlhandler\protocol_socket.py", line 66, in open
    raise SerialException("Could not open port {}: {}".format(self.portstr, msg))
serial.serialutil.SerialException: Could not open port socket://127.0.0.1:50905: [WinError 10061] No connection could be made because the target machine actively refused it

The possible culprit is here:

https://github.com/UAVCAN/yakut/blob/e48c6adf67becedc76defc0274ed0bced55ce282/tests/transport.py#L74-L107

Windows is usually slow to spawn a new process. It could be that the TCP broker is still starting when the test that depends on it is already launched, which leads to the connection refused error. Consider adding a small delay after starting the broker.

yakut call: Spurious redundant transport errors on remote end

yakut call will shut down the local node immediately after the response is received. If the transport is redundant and one of the inferiors delivered the response faster than the other, the transport connections may be closed by the client while the remote end (the server) is still trying to send the response, causing the slower inferiors to fail. This obviously does not affect the connectivity since the primary purpose of the redundant transport is to be resilient against these types of errors, but it does result in rather useless error reports on the server side like this:

ERROR
  pycyphal.transport.redundant._session._output:_output.py:335
    RedundantOutputSession(OutputSessionSpecifier(data_specifier=ServiceDataSpecifier(service_id=222, role=<Role.RESPONSE: 2>), remote_node_id=88), PayloadMetadata(extent_bytes=64)):
      Inferior UDPOutputSession(OutputSessionSpecifier(data_specifier=ServiceDataSpecifier(service_id=222, role=<Role.RESPONSE: 2>), remote_node_id=88), PayloadMetadata(extent_bytes=64)) failed:
        ConnectionRefusedError: [Errno 111] Connection refused

This particular error is occasionally generated by the test suite when testing the heterogeneous UDP+serial configuration (slightly related OpenCyphal/pycyphal#222).

Perhaps the "call" command should wait briefly before shutting down the connections. This is a low-priority problem though.

Subscribe to multiple topics / verbose bus monitor method?

Hey there,

In my "uavcan as housebus" project I am thinking if I should migrate to uavcan v1. In v0, I have the gui tool which works great for configuring and monitoring nodes.

I am now developing my first v1 node and miss anything like the bus monitor.

yakut allows me only to subscribe to one port at a time, the yakut monitor command only shows traffic statistics but not content. Am I missing something?

yakut replay

A command is needed that accepts message objects expressed in YAML with metadata and publishes them on the specified subjects. The subject-ID, dtype, and publication period are to be sourced from the metadata as produced by yakut subscribe (the period is the delta of ts_monotonic):

$ yakut sub 33:uavcan.si.unit.angle.scalar --with-metadata | jq
{
  "33": {
    "_meta_": {
      "ts_system": 1651525686.008718,
      "ts_monotonic": 1827560.239782,
      "source_node_id": 112,
      "transfer_id": 0,
      "priority": "nominal",
      "dtype": "uavcan.si.unit.angle.Scalar.1.0"
    },
    "radian": 2.309999942779541
  }
}
{
  "33": {
    "_meta_": {
      "ts_system": 1651525687.015653,
      "ts_monotonic": 1827561.247286,
      "source_node_id": 112,
      "transfer_id": 1,
      "priority": "nominal",
      "dtype": "uavcan.si.unit.angle.Scalar.1.0"
    },
    "radian": 2.309999942779541
  }
}

The intended usages are two:

  1. Replaying data collected earlier with yakut subscribe
  2. Publishing data produced by other software, e.g.:
printf("{33: {_meta_: {ts_monotonic: 1827561.247286, dtype: uavcan.si.unit.angle.Scalar.1.0}, radian: %f}}\n", get_value());

If transfer-ID and source node-ID are specified, spoofing should be used.

Inaccurate and confusing error message

Even if the uavcan namespace is available, you may get this error which is incorrect and badly misleading:

$ y sub 24:uavcan.nonexistent.nested
NotFoundError: Run `yakut compile <path>/uavcan` to compile DSDL namespace 'uavcan'

The problem is not that uavcan is not available (it is!) but rather that it doesn't contain uavcan.nonexistent.

If the last part .nested is removed, we get the correct error message:

NotFoundError: Could not locate nonexistent.*.* in module 'uavcan'

SIGINT may occasionally result in the stack trace being printed

"Exception ignored in" reports should be suppressed entirely. Here's an example where a publishing command was interrupted normally, and the Python runtime printed the unwanted stack trace:

$ y pub -T 0.01 22:reg.udral.service.actuator.common.sp.vector6 '!$ "[0,0,0,300,0,0]"'
Exception ignored in: <coroutine object publish at 0x7f65cd3d86d0>
Traceback (most recent call last):
  File "/home/groom/.local/lib/python3.10/site-packages/yakut/cmd/publish/_cmd.py", line 311, in publish
    pycyphal.util.broadcast(finalizers[::-1])()
  File "/home/groom/.local/lib/python3.10/site-packages/pycyphal/util/_broadcast.py", line 46, in delegate
    r: typing.Union[R, Exception] = fn(*args, **kwargs)
  File "/home/groom/.local/lib/python3.10/site-packages/yakut/cmd/publish/_executor.py", line 75, in close
    pycyphal.util.broadcast(p.close for p in self._publications)()
  File "/home/groom/.local/lib/python3.10/site-packages/pycyphal/util/_broadcast.py", line 46, in delegate
    r: typing.Union[R, Exception] = fn(*args, **kwargs)
  File "/home/groom/.local/lib/python3.10/site-packages/yakut/cmd/publish/_executor.py", line 135, in close
    self._publisher.close()
  File "/home/groom/.local/lib/python3.10/site-packages/pycyphal/presentation/_port/_publisher.py", line 142, in close
    impl.remove_proxy()
  File "/home/groom/.local/lib/python3.10/site-packages/pycyphal/presentation/_port/_publisher.py", line 212, in remove_proxy
    self.close()  # RAII auto-close
  File "/home/groom/.local/lib/python3.10/site-packages/pycyphal/presentation/_port/_publisher.py", line 223, in close
    self._maybe_finalizer([self.transport_session])
  File "/home/groom/.local/lib/python3.10/site-packages/pycyphal/presentation/_presentation.py", line 394, in finalizer
    ts.close()
  File "/home/groom/.local/lib/python3.10/site-packages/pycyphal/transport/redundant/_session/_output.py", line 316, in close
    s.close()
  File "/home/groom/.local/lib/python3.10/site-packages/pycyphal/transport/redundant/_session/_output.py", line 86, in close
    self.worker.result()
  File "/home/groom/.local/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/groom/.local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/groom/.local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/groom/.local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/groom/.local/lib/python3.10/site-packages/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/home/groom/.local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/groom/.local/lib/python3.10/site-packages/yakut/main.py", line 332, in proxy
    for e in loop.run_until_complete(asyncio.gather(*orphans, return_exceptions=True)):
  File "/usr/lib/python3.10/asyncio/base_events.py", line 633, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 600, in run_forever
    self._run_once()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 1896, in _run_once
    handle._run()
  File "/usr/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/home/groom/.local/lib/python3.10/site-packages/pycyphal/transport/redundant/_session/_output.py", line 331, in _inferior_worker_task
    result = await ses.send(wrk.transfer, wrk.monotonic_deadline)
  File "/home/groom/.local/lib/python3.10/site-packages/pycyphal/transport/can/_session/_output.py", line 206, in send
    return await self._do_send(can_id, transfer, monotonic_deadline)
  File "/home/groom/.local/lib/python3.10/site-packages/pycyphal/transport/can/_session/_output.py", line 163, in _do_send
    if await self._send_handler(transaction):
  File "/home/groom/.local/lib/python3.10/site-packages/pycyphal/transport/can/_can.py", line 329, in _do_send
    num_sent = await self._maybe_media.send(
  File "/home/groom/.local/lib/python3.10/site-packages/pycyphal/transport/can/media/socketcan/_socketcan.py", line 140, in send
    await asyncio.wait_for(
  File "/usr/lib/python3.10/asyncio/tasks.py", line 435, in wait_for
    return fut.result()
  File "/home/groom/.local/lib/python3.10/site-packages/yakut/main.py", line 314, in proxy
    return loop.run_until_complete(f(*args, **kwargs))
  File "/usr/lib/python3.10/asyncio/base_events.py", line 633, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 600, in run_forever
    self._run_once()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 1896, in _run_once
    handle._run()
  File "/usr/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 441, in sock_sendall
    n = sock.send(data)
KeyboardInterrupt: 

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.