amorenoz / ovs-dbg

Scripts to help debug OVS and OVN

ovs-dbg

Scripts to help debug OVS and OVN

Full documentation here: https://ovs-dbg.readthedocs.io/en/latest

Install

Latest released version:

pip install ovs-dbg

From the git checkout

./prepare.sh && pip install .

Enable bash autocomplete

eval "$(ovs-dbg-complete)"

Available tools

ovs-ofparse

ofparse parses the output of commands such as ovs-ofproto dump-flows and prints the files in different outputs including json and html. It suports formatting and filtering.

ovs-ofparse --help

ovs-lgrep

ovs-lgrep helps you grep though many OVS log files to find interleaving the results to help you find what happened on a OVS/OVN cluster

ovs-lgrep --help

ovs-offline

ovs-offline is a script that locally recreates a running OVS so you can debug offline (including running ovs-appctl ofproto/trace)

ovs-offline

Contribute

PRs are welcome!

ovs-dbg's Issues

ovs-offline: rename deacrivate to ovs-offline-deactivate

To avoid collision with python venv's deactivate

ovs-offline: Support running standard processes instead of containers

offline-dbg: add OVN support

Collect OVN databases if possible, named them appropriately and spin up OVSDB containers to serve them

ofp: Bundle not being parsed correctly

decode_bundle fields are not being decoded into their appropriate fields (IPMask, EthMask, etc)

ovs offline: output to kernel tunnels omitted from offline traces

ovs offline ofproto/trace behaves differently from an online ofproto/trace when the action would be to send a packet to a tunnel.

i.e.

online ofproto-trace
move:NXM_NX_REG14[0..14]->NXM_NX_TUN_METADATA0[16..30]
             -> NXM_NX_TUN_METADATA0[16..30] is now 0x1
            output:10
             -> output to kernel tunnel

[...]

Datapath actions: ct(commit,zone=47,label=0/0x1),ct_clear,ct_clear,set(tunnel(tun_id=0xe,dst=10.0.92.172,ttl=64,tp_dst=6081,geneve({class=0x102,type=0x80,len=4,0x1000b}),flags(df|csum|key))),set(eth(src=0a:58:0a:81:02:01,dst=0a:58:0a:81:02:0d)),set(ipv4(ttl=63)),4

offline ofproto-trace
move:NXM_NX_REG14[0..14]->NXM_NX_TUN_METADATA0[16..30]
             -> NXM_NX_TUN_METADATA0[16..30] is now 0x1
            output:10

[...]


Datapath actions: ct(commit,zone=47,label=0/0x1),ct_clear,ct_clear,set(eth(src=0a:58:0a:81:02:01,dst=0a:58:0a:81:02:0d)),set(ipv4(ttl=63)),9

Could be due to the "dummy" ports representing tunnels in the offline environment.

actions: publish to Test pypi is failing

We need to change the version at everyuild

Rename offline-dbg to ovs-offline-dbg

Or even ovs-offline. something that, when added to ones PATH is easy to find

ofparse: Failed to parse load()

Flow:

cookie=0xf9c067da66857ee2, duration=5346306.443s, table=10, n_packets=77859441166, n_bytes=80449118008430, idle_age=0, hard_age=65534, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=0xf9c067da66857ee2,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:OXM_OF_IN_PORT[]),output:1

Result:

$ ofparse -i of-br-tun openflow logic

Traceback (most recent call last):
  File "/home/ktraynor/.local/lib/python3.9/site-packages/ovs_dbg/kv.py", line 233, in parse
    key, val = self._decoders.decode(keyword, value_str)
  File "/home/ktraynor/.local/lib/python3.9/site-packages/ovs_dbg/kv.py", line 114, in decode
    result = decoder(value_str)
  File "/home/ktraynor/.local/lib/python3.9/site-packages/ovs_dbg/ofp_act.py", line 99, in decode_load_field
    return {"value": int(parts[0], 0), "dst": decode_field(parts[1])}
ValueError: invalid literal for int() with base 0: 'NXM_NX_TUN_ID[]'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ktraynor/.local/lib/python3.9/site-packages/ovs_dbg/kv.py", line 233, in parse
    key, val = self._decoders.decode(keyword, value_str)
  File "/home/ktraynor/.local/lib/python3.9/site-packages/ovs_dbg/kv.py", line 114, in decode
    result = decoder(value_str)
  File "/home/ktraynor/.local/lib/python3.9/site-packages/ovs_dbg/ofp_act.py", line 173, in decode_exec
    exec_parser.parse(value)
  File "/home/ktraynor/.local/lib/python3.9/site-packages/ovs_dbg/kv.py", line 235, in parse
    raise ParseError(
ovs_dbg.kv.ParseError: Error parsing key-value (load, NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[])

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ktraynor/.local/bin/ofparse", line 6, in <module>
    main.main()
  File "/home/ktraynor/.local/lib/python3.9/site-packages/ovs_dbg/ofparse/main.py", line 153, in main
    maincli()
  File "/home/ktraynor/.local/lib/python3.9/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/home/ktraynor/.local/lib/python3.9/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/home/ktraynor/.local/lib/python3.9/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ktraynor/.local/lib/python3.9/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ktraynor/.local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ktraynor/.local/lib/python3.9/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/home/ktraynor/.local/lib/python3.9/site-packages/click/decorators.py", line 38, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/home/ktraynor/.local/lib/python3.9/site-packages/ovs_dbg/ofparse/ofp.py", line 110, in logic
    process_flows(
  File "/home/ktraynor/.local/lib/python3.9/site-packages/ovs_dbg/ofparse/process.py", line 24, in process_flows
    flow = flow_factory(line, idx)
  File "/home/ktraynor/.local/lib/python3.9/site-packages/ovs_dbg/ofparse/ofp.py", line 213, in create_ofp_flow
    return OFPFlow.from_string(string, idx)
  File "/home/ktraynor/.local/lib/python3.9/site-packages/ovs_dbg/ofp.py", line 95, in from_string
    aparser.parse(actions)
  File "/home/ktraynor/.local/lib/python3.9/site-packages/ovs_dbg/kv.py", line 235, in parse
    raise ParseError(
ovs_dbg.kv.ParseError: Error parsing key-value (learn, table=20,hard_timeout=300,priority=1,cookie=0xf9c067da66857ee2,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:OXM_OF_IN_PORT[])

ofparse does not work in python3.6

(dbg) [amorenoz@supportshell 02988202]$ ofparse -i 0130-dump_flow_ovs-appctl.txt datapath pretty
/usr/lib64/python3.6/re.py:212: FutureWarning: split() requires a non-empty pattern match.
  return _compile(pattern, flags).split(string, maxsplit)
Traceback (most recent call last):
  File "/cases/02988202/dbg/bin/ofparse", line 6, in <module>
    main.main()
  File "/cases/02988202/dbg/lib64/python3.6/site-packages/ovs_dbg/ofparse/main.py", line 153, in main
    maincli()
  File "/cases/02988202/dbg/lib64/python3.6/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/cases/02988202/dbg/lib64/python3.6/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/cases/02988202/dbg/lib64/python3.6/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/cases/02988202/dbg/lib64/python3.6/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/cases/02988202/dbg/lib64/python3.6/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/cases/02988202/dbg/lib64/python3.6/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/cases/02988202/dbg/lib64/python3.6/site-packages/click/decorators.py", line 38, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/cases/02988202/dbg/lib64/python3.6/site-packages/ovs_dbg/ofparse/dp.py", line 81, in pretty
    with print_context(console.console, opts):
  File "/cases/02988202/dbg/lib64/python3.6/site-packages/ovs_dbg/ofparse/console.py", line 189, in print_context
    return contextlib.nullcontext()
AttributeError: module 'contextlib' has no attribute 'nullcontext'

offline-dbg: use 'local' for local variables

Most of the code recently moved from global scope to functions, so most variables are now local

Optimization: create flow factory

Currently each Flow is created using a constructor that parses the string.
This is inefficient because we call the same @classmethods over and over for each flow.
Since flows are typically parsed in batch, it makes sense to create a class that has the parsing logic and just creates flows so the decoder information can be reused

DP tree: support multiple recirc actions

An action such as this:
actions:br-link1,pop_vlan,ct,recirc(0x612),ct(zone=8),recirc(0x613), dp-extra-info:miniflow_bits(6,2)

is valid, we should show both recirculated flows

Datapath graph not working with latest graphviz

Something broke in 0.19 release.

The Context generator created by subgraph is no longer reentrant.

k8s-ofparse: Support SSL

To make k8s-ofparse production ready, we need SSL configuration

ovs-offline stop: permission denied

Currently sockets are created by containers with uid 0. We need sudo to use them (e.g. use ovs-vsct) or delete them (ovs-offline stop).

We should handle users properly on container instantiation

ofparse datapath logic filtering could be smarter

Right now when you apply a filter on "datapath logic" view the filtered flows are removed from all the subtrees. This means that you could end up with incomplete flow trees or loose flows that would not be filtered but that "hang" from a filtered flow.

I think the right behavior should be: If "any" of the flows in a subtree matches the filter, keep the entire subtree

[RFE] Add support for podman

It would be great if you could add support for podman

I just replaced docker with podman in /bin/ovs-offline and everything is working well :)

ModuleNotFoundError: No module named 'ovs.flow' when running "ofparse openflow html > /myflows.html"

This may be just a doc enhancement or issue fix:

I followed the instruction in installation section of https://github.com/amorenoz/ovs-dbg to install, specifically this command pip install ovs-dbg, after the installation:

ofparse openflow html > /myflows.html failed with error:

/usr/local/bin/ovs-ofparse openflow html > /myflows.html
Traceback (most recent call last):
  File "/usr/local/bin/ovs-ofparse", line 3, in <module>
    from ovs_dbg.ofparse import main
  File "/usr/local/lib/python3.6/site-packages/ovs_dbg/ofparse/__init__.py", line 1, in <module>
    import ovs_dbg.ofparse.ofp  # noqa: F401
  File "/usr/local/lib/python3.6/site-packages/ovs_dbg/ofparse/ofp.py", line 4, in <module>
    from ovs.flow.ofp import OFPFlow
ModuleNotFoundError: No module named 'ovs.flow'

python -V
Python 3.6.8

OS -rhel86

cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.6 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.6"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.6"

ovs-offline: collect-sos-ovn tar: stdout: write error

ovs-offline collect-sos-ovn ~/devel/sosreports/offline/sosreport-hareshcontrollersriov-0-2021-09-28-kakcvko.tar.xz 
tar: stdout: write error
Extracting OVN data from sos report...

It seems to work but we need to see where that error comes from and fix it.
Note: also happens on collect-sos-ovs

Error parsing (set_field, 4501->vlan_vid)

echo "cookie=0xb8c7cc70, duration=94969.048s, table=65, n_packets=29904, n_bytes=2485836, idle_age=0, priority=100,reg15=0x1,metadata=0x2 actions=push_vlan:0x8100,set_field:4501->vlan_vid,output:3,pop_vlan" | ./bin/ofparse openflow pretty
Traceback (most recent call last):
  File "/home/amorenoz/code/ovs-dbg/ovs_dbg/kv.py", line 233, in parse
    key, val = self._decoders.decode(keyword, value_str)
  File "/home/amorenoz/code/ovs-dbg/ovs_dbg/kv.py", line 114, in decode
    result = decoder(value_str)
  File "/home/amorenoz/code/ovs-dbg/ovs_dbg/ofp_act.py", line 116, in decode_set_field
    val_result = field_decoders.decode(dst, val)
  File "/home/amorenoz/code/ovs-dbg/ovs_dbg/kv.py", line 114, in decode
    result = decoder(value_str)
  File "/home/amorenoz/code/ovs-dbg/ovs_dbg/decoders.py", line 92, in __init__
    raise ValueError(
ValueError: Integer value 4501 is bigger than size 12

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/amorenoz/code/ovs-dbg/./bin/ofparse", line 6, in <module>
    main.main()
  File "/home/amorenoz/code/ovs-dbg/ovs_dbg/ofparse/main.py", line 148, in main
    maincli()
  File "/usr/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/home/amorenoz/code/ovs-dbg/ovs_dbg/ofparse/ofp.py", line 40, in pretty
    return pprint(flow_factory=create_ofp_flow, opts=opts)
  File "/home/amorenoz/code/ovs-dbg/ovs_dbg/ofparse/process.py", line 84, in pprint
    process_flows(flow_factory, callback, opts.get("filename"), opts.get("filter"))
  File "/home/amorenoz/code/ovs-dbg/ovs_dbg/ofparse/process.py", line 34, in process_flows
    flow = flow_factory(line, idx)
  File "/home/amorenoz/code/ovs-dbg/ovs_dbg/ofparse/ofp.py", line 211, in create_ofp_flow
    return OFPFlow.from_string(string, idx)
  File "/home/amorenoz/code/ovs-dbg/ovs_dbg/ofp.py", line 95, in from_string
    aparser.parse(actions)
  File "/home/amorenoz/code/ovs-dbg/ovs_dbg/kv.py", line 235, in parse
    raise ParseError(
ovs_dbg.kv.ParseError: Error parsing key-value (set_field, 4501->vlan_vid)

ovs-offline: vhost ports not working

Vhost ports are shown as errored:

        Port vhua1332400-06                                                                                                                                                                                          
            Interface vhua1332400-06                                                                                                                                                                                 
                type: dpdkvhostuserclient                                                                                                                                                                            
                options: {vhost-server-path="/var/lib/vhost_sockets/vhua1332400-06"}                                                                                                                                 
                error: "could not open network device vhua1332400-06 (Address family not supported by protocol)"   

And they re not part of the ofproto bridge:

 ovs-appctl ofproto/trace br-int 'in_port="vhubba4c399-f7",dl_src=fa:16:3e:e1:84:3e,dl_dst=fa:16:3e:87:22:3c,ip_src=192.168.2.177,ip_dst=192.168.2.186,icmp'                                                
Bad openflow flow syntax: in_port="vhubba4c399-f7",dl_src=fa:16:3e:e1:84:3e,dl_dst=fa:16:3e:87:22:3c,ip_src=192.168.2.177,ip_dst=192.168.2.186,icmp: bad value for in_port ("vhubba4c399-f7": invalid or unknown port
 for in_port)                                                                                                                                                                                                   

k8s-ofparse: Add datapath flows as well

failed to parse sample dp flow

Traceback (most recent call last):                                                                                                                                                                                                                                                
  File "/home/amorenoz/.local/lib/python3.9/site-packages/ovs_dbg/kv.py", line 233, in parse                                                                                                                                                                                      
    key, val = self._decoders.decode(keyword, value_str)                                                                                                                                                                                                                          
  File "/home/amorenoz/.local/lib/python3.9/site-packages/ovs_dbg/kv.py", line 114, in decode                                                                                                                                                                                     
    result = decoder(value_str)                                                                                                                                                                                                                                                   
  File "/home/amorenoz/.local/lib/python3.9/site-packages/ovs_dbg/odp.py", line 245, in <lambda>                                                                                                                                                                                  
    "sample": (lambda x: decode_int(x.strip("%"))),  File "/home/amorenoz/.local/lib/python3.9/site-packages/ovs_dbg/decoders.py", line 48, in decode_int
    return int(value, 0)ValueError: invalid literal for int() with base 0: '0.2'
The above exception was the direct cause of the following exception: 

Traceback (most recent call last):
  File "/home/amorenoz/.local/lib/python3.9/site-packages/ovs_dbg/kv.py", line 233, in parse
    key, val = self._decoders.decode(keyword, value_str)
  File "/home/amorenoz/.local/lib/python3.9/site-packages/ovs_dbg/kv.py", line 114, in decode
    result = decoder(value_str)
  File "/home/amorenoz/.local/lib/python3.9/site-packages/ovs_dbg/kv.py", line 266, in decode_nested_kv    parser.parse(value)
  File "/home/amorenoz/.local/lib/python3.9/site-packages/ovs_dbg/kv.py", line 235, in parse    raise ParseError(
ovs_dbg.kv.ParseError: Error parsing key-value (sample, 0.2%)
The above exception was the direct cause of the following exception: 

Traceback (most recent call last):
  File "/home/amorenoz/.local/bin/ofparse", line 6, in <module>    main.main()
  File "/home/amorenoz/.local/lib/python3.9/site-packages/ovs_dbg/ofparse/main.py", line 148, in main
    maincli()
  File "/usr/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)  File "/usr/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)  File "/usr/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.9/site-packages/click/core.py", line 1259, in invoke    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.9/site-packages/click/core.py", line 1066, in invoke    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 610, in invoke    return callback(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/click/decorators.py", line 33, in new_func    return f(get_current_context().obj, *args, **kwargs)
  File "/home/amorenoz/.local/lib/python3.9/site-packages/ovs_dbg/ofparse/dp.py", line 45, in pretty
    return pprint(flow_factory=ODPFlow.from_string, opts=opts)
  File "/home/amorenoz/.local/lib/python3.9/site-packages/ovs_dbg/ofparse/process.py", line 90, in pprint    process_flows(flow_factory, callback, opts.get("filename"), opts.get("filter"))  File "/home/amorenoz/.local/lib/python3.9/site-packages/ovs_dbg/ofparse/process.py", line 35, in process_flows
    flow = flow_factory(line, idx)  File "/home/amorenoz/.local/lib/python3.9/site-packages/ovs_dbg/odp.py", line 102, in from_string
    aparser.parse(actions)
  File "/home/amorenoz/.local/lib/python3.9/site-packages/ovs_dbg/kv.py", line 235, in parse
    raise ParseError(ovs_dbg.kv.ParseError: Error parsing key-value (sample, sample=0.2%,actions(userspace(pid=3500947952,ipfix(output_port=2),tunnel_out_port=2)))

ovs-offline: Seems to not support python virtualenv

Install steps:

git clone https://github.com/amorenoz/ovs-dbg.git
cd ovs-dbg
virtualenv .venv && source .venv/bin/activate
pip install --upgrade pip
pip install .

Error:

❯ ovs-offline build

Error: context must be a directory: "/home/ldenny/ovs-dbg/.venv/containers/ovs-dbg"

Showing current CONTAINER_PATH var

echo $CONTAINER_PATH
/home/ldenny/ovs-dbg/.venv/bin/../containers/ovs-dbg

and actual location:

/home/ldenny/ovs-dbg/containers/ovs-dbg

Container won't start without required directories on the host

ovs-offline start fails to run due to missing directories on the host

The error:

ovs-offline start
Error: statfs /tmp/ovs-offline/var-run/ovn_sb: no such file or directory
Error: statfs /tmp/ovs-offline/var-run/ovn_nb: no such file or directory

Resolution:

mkdir -p /tmp/ovs-offline/var-run/ovn_nb
mkdir -p /tmp/ovs-offline/var-run/ovn_sb

ofparse: datapath logic view does not detect nested recirc actions

e.g:

Datapath Flows (logical)                                                                                                                                                                                             
├── ufid:7c2d7c8d-b933-4bf8-8434-87052c99798f, recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(eth0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=00:00:00:00:00:00/00:00:00:00:00
│   :00,dst=02:42:ac:12:00:03),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=6,tos=0/0,ttl=0/0,frag=no),tcp(src=0/0,dst=0/0),tcp_flags(0/0), packets:2744565, bytes:1191318984, used:0.065s,   
│   flags:SFPR., dp:ovs, actions:ct(zone=64000),recirc(0x8)                                                                                                                                                          
│   ├── ufid:52fe15ca-b2c8-449a-8edb-8dd4495f9c7a, recirc_id(0x8),dp_hash(0/0),skb_priority(0/0),in_port(eth0),skb_mark(0/0),ct_state(0x22/0x23),ct_zone(0/0),ct_mark(0x1),ct_label(0/0x2),eth(src=02:42:ac:12:00:02,
│   │   dst=02:42:ac:12:00:03),eth_type(0x0800),ipv4(src=172.18.0.2,dst=172.18.0.3,proto=6,tos=0/0,ttl=64,frag=no),tcp(src=0/0,dst=0/0),tcp_flags(0/0), packets:2153640, bytes:774440916, used:0.638s, flags:SP.,    
│   │   dp:ovs, actions:check_pkt_len(size=1414,gt(breth0),le(ct_clear,ct(nat),recirc(0x10)))                                                                                                                        
│   ├── ufid:20e15fb6-8631-4935-b18b-f9c4c2610a0c, recirc_id(0x8),dp_hash(0/0),skb_priority(0/0),in_port(eth0),skb_mark(0/0),ct_state(0x22/0x22),ct_zone(0/0),ct_mark(0x2),ct_label(0/0),eth(src=00:00:00:00:00:00/00
│   │   :00:00:00:00:00,dst=00:00:00:00:00:00/00:00:00:00:00:00),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:49, bytes:5727, used:0.064s, flags:FP.,   
│   │   dp:ovs, actions:breth0                                                                                                                                                                                       
│   └── ufid:d4702c7d-d762-45b1-9810-0b393b52fed9, recirc_id(0x8),dp_hash(0/0),skb_priority(0/0),in_port(eth0),skb_mark(0/0),ct_state(0x20/0x26),ct_zone(0/0),ct_mark(0),ct_label(0/0),eth(src=00:00:00:00:00:00/00:0
│       0:00:00:00:00,dst=02:42:ac:12:00:03),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=6,tos=0/0,ttl=0/0,frag=no),tcp(src=0/0,dst=0/0),tcp_flags(0/0), packets:5, bytes:600, used:0.082s,  
│       flags:SP., dp:ovs, actions:breth0

actions:check_pkt_len(size=1414,gt(breth0),le(ct_clear,ct(nat),recirc(0x10)))
the inner recirc is not being detected

ovs-offline collect-sos-XXX should support extracted sosreports

It is quite common that analysis systems automatically extract each sosreport to a folder, so it would be good if both collect-sos-ovs and collect-sos-ovn could work on the root of a sosreport folder.

docker file seems to be missing from the pip version

pip venv:

❯ find | grep container
./.venv/lib/python3.9/site-packages/rich/containers.py
./.venv/lib/python3.9/site-packages/rich/__pycache__/containers.cpython-39.pyc

Using the git repo method:

❯ find | grep container
./containers
./containers/ovs-dbg
./containers/ovs-dbg/Dockerfile
./containers/ovs-dbg/start.sh
./.venv/lib/python3.9/site-packages/rich/containers.py
./.venv/lib/python3.9/site-packages/rich/__pycache__/containers.cpython-39.pyc

[RFE] Install ovn package inside containers so it's not required on the host for ovs-offline

Currently once the two ovsdb containers are running the instructions from the tool are to use the ovn* commands from the host and target the socket mounted from the container to the host.

❯ ovn-nbctl --db unix:/tmp/ovs-offline/var-run/ovn_nb/db.sock show
zsh: ovn-nbctl: command not found...
Install package 'ovn' to provide command 'ovn-nbctl'? [N/y] ^C

If we installed the OVN tools inside the container image we could check if the host has the ovn package installed and if not create two aliases like the following:

alias ovn-nbctl='podman exec ovsdb-server-ovn_nb ovn-nbctl --db=unix:/usr/local/var/run/openvswitch/db.sock'
alias ovn-sbctl='podman exec ovsdb-server-ovn_sb ovn-sbctl --db=unix:/usr/local/var/run/openvswitch/db.sock'

Results

❯ ovn-sbctl show
Chassis "4fdd83d2-63b4-4a98-a24f-b1cdb7a57136"
    hostname: overcloud-controller-0.localdomain
    Encap geneve
        ip: "172.16.0.69"
        options: {csum="true"}
Chassis "107da918-4d12-4661-b363-88bc3da6f367"
    hostname: overcloud-novacompute-0.localdomain
    Encap geneve
        ip: "172.16.0.242"
        options: {csum="true"}
[...]

❯ ovn-nbctl show
switch e6a8d460-c2e3-4d8b-bbfb-a58a0316f16b (neutron-f2d744b1-dc6e-469f-8023-074f0d3aa325) (aka homelab-network)
    port c51b6cd7-244f-4d2f-8281-ba7010ced1cf
        addresses: ["fa:16:3e:be:cf:c2 10.10.1.185"]
    port 72d927a5-7406-4ba5-b894-6d70c802a045
        addresses: ["fa:16:3e:6f:76:41 10.10.1.198"]
    port edd406e1-8035-4418-8f10-c0b89d7c4df2
        type: localport
        addresses: ["fa:16:3e:10:da:80 10.10.1.100"]
    port 99490ad8-65f1-4c0e-81ce-99e63506ba15
        type: router
        router-port: lrp-99490ad8-65f1-4c0e-81ce-99e63506ba15
[...]

Add a Tutorial on how to perform offline debugging with ovs-offline

It should include the process of obtaining the needed information from

• k8s/oc
• osp w/ sos report latest changes
• generic system, w/o latest changes in sos (i.e: based on ovs-save)

And performing:

• basic ovs/ovn introspection
• appctl ofproto/trace
• ovn-detrace

ovs-offline: better allow user to choose OVS and OVN version inside the image

Compiled-in C-idl makes ovn-nbctl et.al to point to a specific ovs/ovn schema. So choosing the right OVS/OVN version inside the image can be crucial to proper debugging.

First, we should add ovn version selection to the normal build command.
But we can do better than that. An idea by @palonsoro: use an OCP release version to pin a specific OVS/OVN version tuple.

Implement proper PMD support for netdev datapath

Datapath flows generated by ovs-appctl dump-flows contain lines that indicate what pmd cpu they come from. We should properly parse it and add it to the flow metadata

ovs-offline: Add a bash environment script

Many OVS and OVN tools use environment variables to determine the default values of database connection strings.

It would be great to add a script that the user can just source to set those variables pointing to the right places

k8s-ofparse: Test in Openshift

ovs-offline: lack routing information for native tunneling

If the OVS set up being replicated uses native tunneling, the ovs routing information is needed to recreate the live behavior.

For example, ofproto/trace traces that require tunneling will throw an error "native tunnel routing failed".

Related issue: #66