omec-project / bess Goto Github PK
View Code? Open in Web Editor NEWThis project forked from netsys/bess
BESS: Berkeley Extensible Software Switch, Used to develop 5G NF UPF (user plane function)
License: Other
This project forked from netsys/bess
BESS: Berkeley Extensible Software Switch, Used to develop 5G NF UPF (user plane function)
License: Other
Hi Team,
We have deployed omec UPF (bess) pod as a part of 4G core deployment. We are using ipvlan (on access) and host-device (on Gi) plugins.
We see that traffic from enb goes to the UPF access interface enters Bess pipeline and goes out to the internet from the core interface. Downlink traffic from the Internet comes to the Gi interface and enters the bess pipeline CoreFast, and goes to Enb via the access interface.
We do not see any issue for a few days ( like 15-20 days approx), and then UPF stops processing packets. Packets are reaching access and core interfaces of UPF but are not entering the Bess pipeline.
There is no container restart/crash.
The issue gets resolved after the UPF pod is restarted.
Has anyone faced this issue before?
How can I see all the rules present in the module? When I show the details of the pdrLookup module using the command
show module pdrLookup
I see the following response:
pdrLookup::WildcardMatch(8 fields, 16 rules)
Per-packet metadata fields:
src_iface: read 1 bytes at offset 20
tunnel_ipv4_dst: read 4 bytes at offset 28
teid: read 4 bytes at offset 32
src_ip: read 4 bytes at offset 44
dst_ip: read 4 bytes at offset 40
src_port: read 2 bytes at offset 36
dst_port: read 2 bytes at offset 22
ip_proto: read 1 bytes at offset 21
pdr_id: write 4 bytes (no downstream reader)
fseid: write 8 bytes at offset 0
ctr_id: write 4 bytes (no downstream reader)
qer_id: write 4 bytes at offset 24
far_id: write 4 bytes at offset 16
Input gates:
0: batches 0 packets 0 pktParse:1 ->
Output gates:
0: batches 0 packets 0 -> 0:appQERLookup Track::track0
1: batches 0 packets 0 -> 0:gtpuDecap Track::track0
2: batches 88669 packets 88962 -> 0:pdrLookupFail Track::track0
Deadends: 0
I am trying to see all the 16 rules available in the module. Kindly guide ma how I can do that.
Packet loss occurs, when "measure_upf" flag is true. "measure_upf" flag enables Telemetry data collection functionality (storing and collection of data from UPF )
I'm running the latest version of the UPF on a VM with Mellanox VFs attached.
To be able to use this i patched the Dockerfile
of the UPF to build DPDK with support for Mellanox:
diff --git a/Dockerfile b/Dockerfile
index 6913931..90839f0 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -14,6 +14,7 @@ RUN apt-get update && apt-get install -y \
--no-install-recommends \
git \
ca-certificates \
+ wget \
libbpf0 \
libelf-dev && \
apt-get clean && \
@@ -21,10 +22,24 @@ RUN apt-get update && apt-get install -y \
# BESS pre-reqs
WORKDIR /bess
+COPY mellanox_fix.patch /tmp/mellanox_fix.patch
RUN git clone https://github.com/omec-project/bess.git . && \
git checkout ${BESS_COMMIT} && \
+ mv /tmp/mellanox_fix.patch . && git apply mellanox_fix.patch && \
cp -a protobuf /protobuf
+# Install Mellanox user-space libraries for support in DPDK
+ARG MLNX_EN_VERSION=23.10-2.1.3.1
+RUN \
+ cd /tmp && \
+ wget "https://www.mellanox.com/downloads/ofed/MLNX_EN-${MLNX_EN_VERSION}/mlnx-en-${MLNX_EN_VERSION}-ubuntu20.04-x86_64.tgz" && \
+ tar zxf mlnx-en-${MLNX_EN_VERSION}-ubuntu20.04-x86_64.tgz && \
+ rm mlnx-en-${MLNX_EN_VERSION}-ubuntu20.04-x86_64.tgz && \
+ cd mlnx-en-${MLNX_EN_VERSION}-ubuntu20.04-x86_64 && \
+ echo y | ./install --dpdk --user-space-only --without-fw-update && \
+ cd .. && \
+ rm -rf mlnx-en-${MLNX_EN_VERSION}-ubuntu20.04-x86_64
+
# Build DPDK
RUN ./build.py dpdk
Since DPDK with Mellanox doesn't require to unbind the card from the normal driver i moved the network interfaces to the container like this:
sudo ip link set enp6s16np0 netns pause
sudo ip link set enp6s17np0 netns pause
Using the 59c79ca commit of PR #39 everything works but with commit af34576 it stopped, the problem is this check.
I'm not sure why but since that rte_eth_dev_socket_id
check the NUMA zone of the interface i think it may be related to the fact that's running in a Docker container and bess report -1 as NUMA zone:
EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:101e) device: 0000:06:10.0 (socket -1)
EAL: Probe PCI driver: mlx5_pci (15b3:101e) device: 0000:06:11.0 (socket -1)
EAL: Probe PCI driver: net_virtio (1af4:1000) device: 0000:06:12.0 (socket -1)
eth_virtio_pci_init(): Failed to init PCI device
EAL: Requested device 0000:06:12.0 cannot be used
TELEMETRY: No legacy callbacks, legacy socket not created
Segment 0-0: IOVA:0x380000000, len:1073741824, virt:0x140000000, socket_id:0, hugepage_sz:1073741824, nchannel:0, nrank:0 fd:10
Segment 0-2: IOVA:0x400000000, len:1073741824, virt:0x1c0000000, socket_id:0, hugepage_sz:1073741824, nchannel:0, nrank:0 fd:11
I0311 11:30:57.800395 1 packet_pool.cc:53] Creating DpdkPacketPool for 262144 packets on node 0
I0311 11:30:57.800410 1 packet_pool.cc:74] PacketPool0 requests for 262144 packets
I0311 11:30:57.812450 1 packet_pool.cc:161] PacketPool0 has been created with 262144 packets
I0311 11:30:57.812634 1 pmd.cc:74] 2 DPDK PMD ports have been recognized:
I0311 11:30:57.812654 1 pmd.cc:98] DPDK port_id 0 (mlx5_pci) RXQ 1024 TXQ 1024 94:6d:ae:00:00:00 00000000:06:10.00 15b3:101e numa_node -1
I0311 11:30:57.812659 1 pmd.cc:98] DPDK port_id 1 (mlx5_pci) RXQ 1024 TXQ 1024 94:6d:ae:00:00:01 00000000:06:11.00 15b3:101e numa_node -1
The crash happens when the UPF start script run the command docker exec bess ./bessctl run up4
and the output is the following (I edited the python script to also print the exception string):
*** Error: Unhandled exception in the configuration script (most recent call last)
File "/opt/bess/bessctl/conf/up4.bess", line 141, in <module>
p.init_port(idx, parser.mode)
File "/opt/bess/bessctl/conf/ports.py", line 244, in init_port
sys.exit()
SystemExit
Command failed: run up4
max_ip_defrag_flows value not set. Not installing IP4Defrag module.
ip_frag_with_eth_mtu value not set. Not installing IP4Frag module.
Can't parse unix socket paths for notify! Setting it to default values (/tmp/notifycp)
Can't parse unix socket paths for end marker! Setting it to default values (/tmp/pfcpport)
Setting up port access on worker ids [0, 1]
#####################
errno=19 (ENODEV: No such device), Port id 0 is not attached
#####################
Registered dpdk ports do not exist.
Am I doing something wrong or is this a bug with Mellanox NICs ?
When there are changes such as DPDK version, env/Dockerfile
pulls/clones the BESS repo from GitHub instead of using the local copy/changes. So, when trying to build the image, it uses what is in the master branch and not what is in the PR.
For example, the changes below are to bump up the DPDK version, and as can be seen in the log below, the image building process fails because BESS downloads DPDK Downloading https://fast.dpdk.org/rel/dpdk-20.11.3.tar.gz
. So the /build/bess/deps/dpdk-22.11/build/app/dpdk-testpmd
does not exist.
There are two possible solution I can think of:
BESS
) instead of pulling it from GitHub; orubuntu-machine:~/bess/env$ ./rebuild_images.py focal64
[+] Building 415.3s (36/37) docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 3.72kB 0.0s
=> [internal] load metadata for docker.io/library/ubuntu:focal 1.1s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load build context 0.1s
=> => transferring context: 3.61kB 0.0s
=> [stage-1 1/19] FROM docker.io/library/ubuntu:focal@sha256:f2034e7195f61334e6caff6ecf2e965f92d11e888309065da8 0.1s
=> => resolve docker.io/library/ubuntu:focal@sha256:f2034e7195f61334e6caff6ecf2e965f92d11e888309065da85ff50c6177 0.0s
=> => sha256:f2034e7195f61334e6caff6ecf2e965f92d11e888309065da85ff50c617732b8 1.13kB / 1.13kB 0.0s
=> => sha256:bac6081102aae54ba4bcc714695b8f637e42768c7f376f374c428bab043ddc0f 424B / 424B 0.0s
=> => sha256:f78909c2b360d866b3220655c0b079838258b8891a12ac25fc670f0cbb54229f 2.30kB / 2.30kB 0.0s
=> [stage-1 2/19] RUN apt-get update && apt-get install -y build-essential ethtool libbsd-dev 62.5s
=> [cndp-build 2/16] RUN apt-get update && apt-get install -y build-essential golang libelf-dev 135.5s
=> [cndp-build 3/16] RUN git clone https://github.com/xdp-project/xdp-tools.git 1.0s
=> [cndp-build 4/16] WORKDIR /xdp-tools/ 0.1s
=> [cndp-build 5/16] RUN git checkout v1.2.2 0.4s
=> [cndp-build 6/16] RUN ./configure 2.8s
=> [cndp-build 7/16] WORKDIR /xdp-tools/ 0.1s
=> [cndp-build 8/16] RUN make -j; PREFIX=/usr make -j install 6.8s
=> [cndp-build 9/16] RUN git clone https://github.com/CloudNativeDataPlane/cndp.git 2.3s
=> [cndp-build 10/16] WORKDIR /cndp 0.1s
=> [cndp-build 11/16] RUN git checkout d5ce4b9edc2e7ddb46a61b395deffafaf11a0500 0.6s
=> [cndp-build 12/16] RUN make && make install 8.8s
=> [cndp-build 13/16] RUN make static_build=1 rebuild install 7.4s
=> [cndp-build 14/16] WORKDIR /cndp/lang/go/stats/prometheus/ 0.1s
=> [cndp-build 15/16] RUN go build prometheus.go 8.5s
=> [stage-1 3/19] COPY --from=cndp-build /cndp/usr/local/bin/cndpfwd /usr/bin/ 0.1s
=> [stage-1 4/19] COPY --from=cndp-build /cndp/usr/local/lib/x86_64-linux-gnu/*.so /usr/lib/ 0.3s
=> [stage-1 5/19] COPY --from=cndp-build /cndp/usr/local/lib/x86_64-linux-gnu/*.a /usr/lib/ 0.2s
=> [stage-1 6/19] COPY --from=cndp-build /cndp/usr/local/include/cndp/ /usr/local/include/cndp/ 0.1s
=> [stage-1 7/19] COPY --from=cndp-build /cndp/usr/local/lib/pkgconfig/libcndp.pc /usr/lib/pkgconfig/ 0.1s
=> [stage-1 8/19] COPY --from=cndp-build /cndp/lang/go/stats/prometheus/prometheus /usr/bin/ 0.1s
=> [stage-1 9/19] COPY --from=cndp-build /usr/lib/libxdp* /usr/lib/ 0.2s
=> [stage-1 10/19] COPY --from=cndp-build /usr/include/xdp/ /usr/include/xdp/ 0.1s
=> [stage-1 11/19] RUN echo "APT::Install-Recommends false;" >> /etc/apt/apt.conf.d/00recommends && echo "APT:: 0.3s
=> [stage-1 12/19] COPY build-dep.yml /tmp/ 0.1s
=> [stage-1 13/19] COPY kmod.yml /tmp/ 0.1s
=> [stage-1 14/19] COPY ci.yml /tmp/ 0.1s
=> [stage-1 15/19] RUN apt-get -q update && apt-get install -y ansible curl && ansible-playbook /tmp/ 120.4s
=> [stage-1 16/19] RUN update-alternatives --install /usr/local/bin/python python /usr/bin/python3 3 0.4s
=> [stage-1 17/19] RUN mkdir -p /build/bess 0.5s
=> ERROR [stage-1 18/19] RUN cd /build/bess && curl -s -L https://github.com/omec-project/bess/archive/master 115.8s
------
> [stage-1 18/19] RUN cd /build/bess && curl -s -L https://github.com/omec-project/bess/archive/master.tar.gz | tar zx --strip-components=1 && ./build.py dpdk && cp /build/bess/deps/dpdk-22.11/build/app/dpdk-testpmd /usr/local/bin/ && rm -rf /build/bess:
73.26 Downloading https://fast.dpdk.org/rel/dpdk-20.11.3.tar.gz ...
73.26 Configuring DPDK...
73.26 Applying patch /build/bess/deps/bpf_validate.patch
73.26 Applying patch /build/bess/deps/0001-af_packet-Avoid-set-ioctl-if-there-is-no-flag-diff.patch
73.26 Applying patch /build/bess/deps/ethdev_include.patch
73.26 Building DPDK...
73.27 cp: cannot stat '/build/bess/deps/dpdk-22.11/build/app/dpdk-testpmd': No such file or directory
------
Dockerfile:112
--------------------
111 | ARG BESS_DPDK_BRANCH=master
112 | >>> RUN cd /build/bess && \
113 | >>> curl -s -L https://github.com/omec-project/bess/archive/${BESS_DPDK_BRANCH}.tar.gz | tar zx --strip-components=1 && \
114 | >>> ./build.py dpdk && \
115 | >>> cp /build/bess/deps/dpdk-22.11/build/app/dpdk-testpmd /usr/local/bin/ && \
116 | >>> rm -rf /build/bess
117 |
--------------------
ERROR: failed to solve: process "/bin/sh -c cd /build/bess && \tcurl -s -L https://github.com/omec-project/bess/archive/${BESS_DPDK_BRANCH}.tar.gz | tar zx --strip-components=1 && \t./build.py dpdk && \tcp /build/bess/deps/dpdk-22.11/build/app/dpdk-testpmd /usr/local/bin/ && \trm -rf /build/bess" did not complete successfully: exit code: 1
Traceback (most recent call last):
File "/home/garrobo/upf-ipm/bess/env/./rebuild_images.py", line 103, in <module>
sys.exit(main(sys.argv))
File "/home/garrobo/upf-ipm/bess/env/./rebuild_images.py", line 90, in main
version, tag_suffix = build(argv[1])
File "/home/garrobo/upf-ipm/bess/env/./rebuild_images.py", line 66, in build
run_cmd('docker build '
File "/home/garrobo/upf-ipm/bess/env/./rebuild_images.py", line 57, in run_cmd
subprocess.check_call(shlex.split(cmd))
File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['docker', 'build', '--build-arg', 'BASE_IMAGE=ubuntu:focal', '--build-arg', 'BESS_DPDK_BRANCH=master', '-t', 'ghcr.io/omec-project/upf-epc/bess_build:latest', '-t', 'ghcr.io/omec-project/upf-epc/bess_build:240131', '.']' returned non-zero exit status 1.
In PR #10, to enable several GitHub Actions, a few tests were temporarily disabled. That is four files were changed of extension to txt. The tests that were disabled are:
This needs to be addressed by properly updating the tests based on the changes made to the ExactMatch class
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.