opennetworkinglab / fabric-p4test

PTF-based data plane tests for ONOS fabric.p4

PTF tests for ONOS fabric.p4

This project contains PTF-based tests for fabric.p4, a P4 program distributed as part of ONOS, designed to work with Trellis, a set of SDN applications running on top of ONOS to provide the control plane for an IP fabric based on MPLS segment-routing.

PTF is a framework for data plane testing: https://github.com/p4lang/PTF

This project maintains the test case definition (written in Python), as well as the scripts to run them on different targets. Test cases are defined inside the directory tests/ptf/fabric.ptf Run scripts can be found in run/.

Currently, we provide scripts to test the following targets:

• stratum_bmv2 (run/bmv2/)
• tofino-model with stratum_bf (run/tm/)

To learn more about Stratum: https://github.com/stratum/stratum

Requirements

All scripts are based on a containerized version of the required tools and can be executed by installing the following dependencies:

• Docker (tested with v19.03, but it should work with older versions as well)
• make
• Bash-like Unix shell

Steps to run tests on stratum_bmv2

stratum_bmv2 is a version of BMv2's simple_switch (the reference P4 software switch) built with Stratum support to expose a P4Runtime interface that is used by the test cases to populate tables and other P4 objects.

1. Obtain the fabric.p4 pre-compiled artifacts for BMv2 (bmv2.json and p4info.txt). These files are distributed with ONOS:

   git clone https://github.com/opennetworkinglab/onos
   

2. Set the ONOS_ROOT environment variable to the location where you just cloned the ONOS repo:

   export ONOS_ROOT=<path-to-onos>
   

3. Run PTF tests:

   ./run/bmv2/run <profile> [test-case]
   

   To learn more about fabric.p4 "profiles" check these instructions.

   For example, to run all test cases for all profiles:

   ./run/bmv2/run all
   

   To run all test cases for the fabric-spgw profile:

   ./run/bmv2/run fabric-spgw
   

   To run a specific test case against a specific fabric profile (or all), for example test.FabricBridgingTest for the basic fabric profile:

   ./run/bmv2/run fabric TEST=test.FabricBridgingTest
   

4. If tests fail, check logs in:

   • run/bmv2/log for stratum_bmv2 logs
   • tests/ptf/ for PTF-related logs and PCAP traces (ptf.log and ptf.pcap)

Steps to run tests on tofino-model with stratum_bf

Requirements

Steps are similar to the previous case with a few differences:

• You will need to compile fabric.p4 for Tofino;
• You will need to create or obtain a containerized version of tofino-model;
• stratum_bf is used to provide a P4Runtime server to control tofino-model (a Docker image for it will be downloaded automatically.)

The run scripts assume that you have access to a containerized version of the Intel/Barefoot SDE that includes tofino-model. We do not provide such Docker image, but one can be easily generated by executing the SDE install instructions inside a Dockerfile.

The run script will use docker run to invoke the tofino-model command inside the container. For this reason, the script expects a Docker image that has the whole Barefoot SDE installed in it or just the tofino-model package. In both cases, the tofino-model executable should be on PATH.

IMPORTANT: make sure to reserve at least 8GB of RAM for your Docker host system (or VM if running Docker Desktop for Mac), otherwise tofino-model might fail to start or affect test results negatively.

Steps

1. Clone fabric-tofino repo and follow instructions to compile fabric.p4 for Tofino:

   https://github.com/opencord/fabric-tofino.git
   cd fabric-tofino
   <open README and follow instructions>
   

2. Set the FABRIC_TOFINO environment variable to the location where you cloned fabric-tofino:

   export FABRIC_TOFINO=<path-to-fabric-tofino>
   

3. Set the SDE_DOCKER_IMG environment variable to the location of a Docker image that can be downloaded via docker pull and that contains a Barefoot SDE installation, including tofino-model:

   export SDE_DOCKER_IMG=my-docker-repo/bf-sde:9.0.0
   

4. Run PTF tests using the run/tm/run script:

   ./run/tm/run <profile> [test-case]
   

   NOTE: Testing all profiles is not supported on this target. You must execute the run command for each profile.

   To run all test cases for the basic fabric profile:

   ./run/tm/run fabric
   

   To run a specific test case against a specific fabric profile, for example test.FabricBridgingTest for the fabric-spgw profile:

   ./run/tm/run fabric-spgw TEST=test.FabricBridgingTest
   

5. If tests fail, check logs in:

   • run/tm/log for tofino-model and stratum_bf logs
   • tests/ptf/ for PTF-related logs and PCAP traces (ptf.log and ptf.pcap)

Run tests on other targets

To run tests on targets other than BMv2 (e.g. a real Tofino-based switch), check the instructions available inside the tests/ptf directory.

Status

All test cases are executed daily against the ONOS 2.5 LTS branch (onos-2.5) using Travis CI. The current status is:

Migrating to Stratum Test Vectors

We are currently in the process of migrating the test runner framework from PTF to Stratum's testvectors-runner.

For this reason, some PTF-based test case definitions are currently instrumented to generate protobuf-based TestVectors (TVs) (under tests/ptf/testvectors). The long-term goal is to remove all PTF references from the codebase, but continue using Python as a convenient way to generate TVs. For now, we use a Python library (tvutils) to wrap PTF and P4Runtime calls in methods that generate TV's actions, stimuli and expectations instead of calling the corresponding PTF or P4Runtime gRPC methods.

Steps to generate TestVectors

TestVectors can be generated for bmv2 and tofino targets. The instructions are similar to running ptf tests on bmv2 and tofino model.

1. Obtain the fabric.p4 pre-compiled artifacts for BMv2 (bmv2.json and p4info.txt). These files are distributed with ONOS:

   git clone https://github.com/opennetworkinglab/onos
   

2. Set the ONOS_ROOT environment variable to the location where you just cloned the ONOS repo:

   export ONOS_ROOT=<path-to-onos>
   

3. Clone fabric-tofino repo and follow instructions to compile fabric.p4 for Tofino:

   https://github.com/opencord/fabric-tofino.git
   cd fabric-tofino
   <open README and follow instructions>
   

4. Set the FABRIC_TOFINO environment variable to the location where you cloned fabric-tofino:

   export FABRIC_TOFINO=<path-to-fabric-tofino>
   

5. Set the optional SDE_VERSION environment variable. Default value is 9.0.0

   export SDE_VERSION=9.0.0
   

6. Generate TestVectors using the run/tv/run script:

   ./run/tv/run <profile> [device] [portmap] [grpcaddr] [cpuport] [test-case]
   

   Default values for optional arguments are:

   1. device: tofino
   2. portmap: portmap.veth.json
   3. grpcaddr: 127.0.0.1:28000
   4. cpuport: 320 for tofino and 255 for bmv2

   Example command with all the optional arguments set:

   ./run/tv/run fabric DEVICE=tofino PORTMAP=port_map.hw.json GRPCADDR=10.128.13.111:28000 CPUPORT=320 TEST=test.FabricBridgingTest
   

   NOTE: Testing all profiles is not supported on this target. You must execute the run command for each profile.

   To generate TestVectors for the basic fabric profile:

   ./run/tv/run fabric
   

   To generate a specific test case for a specific fabric profile, for example test.FabricBridgingTest for the fabric-spgw profile:

   ./run/tv/run fabric-spgw TEST=test.FabricBridgingTest
   

Support

For help running the tests please write to the P4 Brigade mailing list:

https://groups.google.com/a/onosproject.org/forum/#!forum/brigade-p4

fabric-p4test's Issues

Rename parameter --external-hercules-urls for --external-stratum-urls for Stratum usage.

Parameter --external-hercules-urls was recently renamed for --external-stratum-urls in the Stratum code.

Function get_stratum_cmd should be updated accordingly.

Running PTF tests on Tofino

The current readme instructions to run PTF tests on Tofino say:

sudo ./ptf_runner.py --device tofino \ 
    --p4info p4info.txt \
    --tofino-bin tofino.bin \
    --tofino-ctx-json context.json \
    --ptf-dir fabric.ptf --port-map port_map.veth.json \
    all ^spgw

However, I don't think this can realistically work. Updating the P4 program on Tofino will remove the dataplane ports. Maybe we need a --skip-tests flag so that the runner can be invoked twice:

./ptf_runner.py --skip-tests ....`
// add ports
./ptf_runner.py --skip-config ...`

or some other solution...

Maybe I'm missing something @ccascone? How do you run the tests?

Remove use of p4config_pb2 in ptf_runner.py

Pull Request Deprecate temporary P4DeviceConfig message in PI removed the requirement on message P4DeviceConfig.

This change was brought to Stratum through PR #194. Removing this dependency facilitates running PTF tests with Stratum.

Reference to p4config_pb2 should be removed from ptf_runner.py since it is no longer needed with recent versions of PI.

Tests are broken because stratum_bmv2 crashes when processing packets via action profiles

The issue can be reproduced by running ./docker_run.sh bug from commit d4b8f1f in the fix-travis branch.

stratum_bmv2.log
write-reqs.log

From stratum_bmv2.log:

[21:07:10.163] [bmv2] [D] [thread 83] [52.0] [cxt 0] Looking up key:
* next_id             : 00000064

[21:07:10.163] [bmv2] [D] [thread 83] [52.0] [cxt 0] Choosing member 1 from group 0
stratum_bmv2: action_profile.cpp:182: const bm::ActionEntry& bm::ActionProfile::lookup(const bm::Packet&, const bm::ActionProfile::IndirectIndex&) const: Assertion `is_valid_mbr(mbr)' failed.
*** Aborted at 1590095230 (unix time) try "date -d @1590095230" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGABRT (@0x4f) received by PID 79 (TID 0x7f3ce7f10700) from PID 79; stack trace: ***
    @     0x7f3ceb0c20e0 (unknown)
    @     0x7f3ceab2dfff gsignal
    @     0x7f3ceab2f42a abort
    @     0x7f3ceab26e67 (unknown)
    @     0x7f3ceab26f12 __assert_fail
    @     0x7f3cebb9ff4f bm::ActionProfile::lookup()
    @     0x7f3cebc397c4 bm::MatchTableIndirect::lookup()
    @     0x7f3cebc3ac93 bm::MatchTableIndirectWS::lookup()
    @     0x7f3cebc36b5c bm::MatchTableAbstract::apply_action()
    @     0x7f3cebd0ed7e bm::MatchActionTable::operator()()
    @     0x7f3cebce0b77 bm::Pipeline::apply()
    @     0x7f3cebb047f4 SimpleSwitch::ingress_thread()
    @     0x7f3cebb213ae std::__invoke_impl<>()
    @     0x7f3cebb212b3 std::__invoke<>()
    @     0x7f3cebb211f0 _ZNKSt12_Mem_fn_baseIM12SimpleSwitchFvvELb1EEclIJPS0_EEEDTcl8__invokedtdefpT6_M_pmfspcl7forwardIT_Efp_EEEDpOS6_
    @     0x7f3cebb2116f _ZNSt12_Bind_simpleIFSt7_Mem_fnIM12SimpleSwitchFvvEEPS1_EE9_M_invokeIJLm0EEEEvSt12_Index_tupleIJXspT_EEE
    @     0x7f3cebb20ef3 std::_Bind_simple<>::operator()()
    @     0x7f3cebb20dc0 std::thread::_State_impl<>::_M_run()
    @     0x7f3cec494e6f (unknown)
    @     0x7f3ceb0b84a4 start_thread
    @     0x7f3ceabe3d0f clone
    @                0x0 (unknown)

Consider using p4runtime-shell for P4Runtime operations in PTF tests

We currently use a custom library to read/write P4Runtime entities such as table entries. We might consider dropping that in favor of p4runtime-shell, as explained here:
https://github.com/p4lang/p4runtime-shell#using-p4runtime-shell-in-scripts

FabricIntSourceTest fails on onos-1.14

FAIL: test.FabricIntSourceTest
----------------------------------------------------------------------
Traceback (most recent call last):
 File “fabric.ptf/test.py”, line 469, in runTest
   tagged1=tagged[0], tagged2=tagged[1])
 File “/home/sdn/fabric-p4test/tests/ptf/base_test.py”, line 665, in handle
   return f(*args, **kwargs)
 File “fabric.ptf/test.py”, line 452, in doRunTest
   self.runIntSourceTest(**kwargs)
 File “/home/sdn/fabric-p4test/tests/ptf/fabric_test.py”, line 765, in runIntSourceTest
   tagged1=tagged1, tagged2=tagged2)
 File “/home/sdn/fabric-p4test/tests/ptf/fabric_test.py”, line 483, in runIPv4UnicastTest
   testutils.verify_each_packet_on_each_port(self, exp_pkts, exp_ports)
 File “/usr/local/lib/python2.7/dist-packages/ptf-0.9.1-py2.7.egg/ptf/testutils.py”, line 2455, in verify_each_packet_on_each_port
   % (port, device_number, result.format()))
AssertionError: Did not receive expected packets on port 2 for device 0.

Perform PTF tests for BMv2 on latest ONOS master via Travis CI

This might require using Docker support in Travis since Ubuntu 16.04 is not offered as an environment target. At a high level we need a script that checks out ONOS master, installs all dependencies (using ONOS p4vm scripts https://github.com/opennetworkinglab/onos/tree/master/tools/dev/p4vm), runs PTF tests on BMv2. This Travis build should be executed for each PR, as well periodically (e.g. every day) to make sure changes to fabric.p4 in ONOS are passing tests.