License: Apache License 2.0
This repo has been archived. Network Service Mesh continues to be very actively developed in multiple repos.
Network Service Mesh (NSM) is the Hybrid/Multi-cloud IP Service Mesh.
For more information, have a look at our website
Get Started with our latest release
Documentation can be found on our website
Information on getting involved: meetings, slack, etc can be found on our commmunity page
We need to update VPP to the latest possible version. It must match the version used by Calico-vpp.
(look at projectcalico/vpp-dataplane@d8288e1)
At start up of the forwarder a veth pair is created in vpp to AF_PACKET interface. This interface is selected by it's IP - must be equal with TUNNEL_IP or if TUNNEL_IP is not set then one of the IP from default route.
If later an other veth pair is needed to an oter AF_PACKET this code should be reused. The sdk-vpp repository should be a good place.
Currently AF_XDP management interface works only on kind cluster.
When trying to run it on public clusters, we encountered the following issues:
Doesn't start. Logs from forwarder:
Apr 3 05:38:16.954 [INFO] [cmd:vpp] libbpf: Kernel error message: virtio_net: XDP expects header/data in single page, any_header_sg required
Apr 3 05:38:16.954 [INFO] [cmd:vpp] vpp[10244]: af_xdp: af_xdp_load_program: bpf_set_link_xdp_fd(eth0) failed: Invalid argument
Apr 3 05:38:18.228 [ERRO] [cmd:/bin/forwarder] [duration:12.809608ms] [hostIfName:eth0] [vppapi:AfXdpCreate] VPPApiError: System call error #6 (-16)
panic: error: VPPApiError: System call error #6 (-16)
Doesn't start. Logs from forwarder:
Apr 3 13:24:25.406 [INFO] [cmd:vpp] libbpf: Kernel error message: veth: Peer MTU is too large to set XDP
Apr 3 13:24:25.406 [INFO] [cmd:vpp] vpp[10508]: af_xdp: af_xdp_load_program: bpf_set_link_xdp_fd(eth0) failed: Numerical result out of range
Apr 3 13:24:26.563 [ERRO] [cmd:/bin/forwarder] [duration:18.015838ms] [hostIfName:eth0] [vppapi:AfXdpCreate] VPPApiError: System call error #6 (-16)
panic: error: VPPApiError: System call error #6 (-16)
Started, but ping doesn't work. This is most likely due to the fact that af_xdp vpp plugin doesn't process bonded interfaces (they are used by Packet servers)
Ping works only without hostNetwork: true flag. But poor performance (compared to AF_PACKET about 2 times slower)
During a test we set the host interface administrative down then up again within a second. The traffic immediately stops on the affected node and can only be recovered by restarting the forwarder on that node.
I attached the forwarder-vpp.log and the output of the vppctl show interface command. It seems the interfaces are deleted then re-created but cannot come up properly in an endless loop.
I managed to reproduce it in kind environment too in a basic setup, but instead of changing the administrative state of the host interface I just disconnected a kind-worker for a moment then connect it again. For example:
docker network disconnect kind kind-worker2
docker network connect kind kind-worker2
The same behavior was observed on NSM v1.11.0 and on previous versions before the release.
I tested several forwarder restarts sequentially and sometimes it thrown panic on a segmentation fault:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xf23672]
goroutine 276 [running]:
github.com/networkservicemesh/sdk-vpp/pkg/networkservice/pinhole.tunnelIPSwIfIndex({0x147caf0, 0xc000986b10}, {0x7f7d585f7c18?, 0xc0000bf120}, {0xc00028bb00, 0x10, 0x10})
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/pinhole/common.go:210 +0x3d2
github.com/networkservicemesh/sdk-vpp/pkg/networkservice/pinhole.create({0x147caf0, 0xc000986b10}, {0x7f7d585f7c18, 0xc0000bf120}, {0xc00028bb00, 0x10, 0x10}, 0x12b5, {0xc00002a8a0, 0x15})
See the logs attached: forwarder-restart-segfault.log
I Hope I post the issue on the correct section.
I'm working on a project involving the usage of network service mesh.
In the project I want to use a kernel-based forwarder instead of the VPP-forwarder.
In the below link I found an example for VPP-forwarder:
https://github.com/networkservicemesh/cmd-forwarder-vpp/blob/main/main.go
I searched for a kernel-based forwarder but found an outdated version in the below link:
https://github.com/networkservicemesh/networkservicemesh/tree/v0.2.0/forwarder/kernel-forwarder
Is there any example or implementation of a kernel-based forwarder that I can use?
If not, do you have recommendations on getting/implementing one?
Thanks,
Ben Agai Meherabanan
An issue was discovered in Mbed TLS before 2.28.2 and 3.x before 3.3.0. There is a potential heap-based buffer overflow and heap-based buffer over-read in DTLS if MBEDTLS_SSL_DTLS_CONNECTION_ID is enabled and MBEDTLS_SSL_CID_IN_LEN_MAX > 2 * MBEDTLS_SSL_CID_OUT_LEN_MAX.
We observed the following runtime error during our tests with NSM v1.11.0. One time it came at upgrade but next time during normal operation without any maintenance activities on the system. See the whole log attached: forwarder-vpp-2d2xp.log
Oct 11 13:06:34.182 [TRAC] [id:1fc7c187-6f2b-4acb-80f9-921d721d2b3f] [type:networkService] (55.1) request-response-diff={"path":{"path_segments":{"4":{"metrics":{"+client_drops":"0","+client_rx_bytes":"0","+client_rx_packets":"0","+client_tx_bytes":"0","+client_tx_packets":"0"}}}}}
time="2023-10-11T13:06:34Z" level=info msg="No subscription found for the notification message." msg_id=714 msg_size=19
Oct 11 13:06:34.182 [DEBU] [id:a331c9ce-8e08-4c68-90a9-39d241df71f7] [DstAddress:172.18.0.11] [SrcAddress:172.18.0.9] [Vni:3448853] [duration:306.803µs] [isAdd:true] [swIfIndex:9] [type:networkService] [vppapi:VxlanAddDelTunnel] (33.2) completed
panic: runtime error: index out of range [9] with length 9
goroutine 11380 [running]:
go.fd.io/govpp/adapter/statsclient.(*statSegmentV2).UpdateEntryData(0x430?, 0x7f62c4c3aa60, 0xc000cfdff8)
/root/go/pkg/mod/go.fd.io/[email protected]/adapter/statsclient/statseg_v2.go:354 +0x77c
go.fd.io/govpp/adapter/statsclient.(*StatsClient).updateStatOnIndex(0xc000200690, 0xc000cfdfc8, 0xc000fde3e8?)
/root/go/pkg/mod/go.fd.io/[email protected]/adapter/statsclient/statsclient.go:633 +0x171
go.fd.io/govpp/adapter/statsclient.(*StatsClient).UpdateDir(0xc000200690, 0xc0008ecba0)
/root/go/pkg/mod/go.fd.io/[email protected]/adapter/statsclient/statsclient.go:294 +0x1e7
go.fd.io/govpp/core.(*StatsConnection).updateStats.func1()
/root/go/pkg/mod/go.fd.io/[email protected]/core/stats.go:220 +0x5b
When we use an OVS interface on the host, the IP and MAC address can change at worker node restart.
In such cases it will not stored in VPP neighbor table which causes issues at restart (address resolution does not work).
It occurs with NSM version 1.6.2 and 1.8.0 too.
https://github.com/networkservicemesh/cmd-forwarder-vpp/blob/main/go.mod#L18
cmd-forwarder-vpp was not designed to use sdk-k8s.
we should refactor code and get rid of using sdk-k8s dep in cmd-forwarder-vpp.
We need to know the performance of NSM cmd-forwarder-vpp.
N is the number from step1. mechanisms: memif2memif, kerne2kernel, memif2vxlan2memif, kernel2vxlan2kernel, kernel2wireguard2kernel, memif2wireguard2memif.N is the number from step1. memif2kernel, kernel2memif, memif2vxlan2kernel, kernel2vxlan2memif, memif2wireguard2kernel, kernel2wireguard2memif.8d
TestForwarderTestSuite/TestCombinations/Ip/Kernel/Wireguard is unstable
https://github.com/networkservicemesh/cmd-forwarder-vpp/runs/4342269191?check_suite_focus=true
When running in AF_PACKET mode there seems to be a memory increase in the NSM supplied VPP process on NSM 1.8.
For me it seems the root cause might be the amount of memory assigned to an af_packet socket.
Previously it was ~20MB per socket while with 1.8 it appears to be 135MB per af_packet socket.
In case multiple interfaces are provided in the device-selector-file for vpp-forwarder, vpp seems to create an af_packet socket for each, which can significantly increase the memory footprint (RssFile) and risk OOM.
Is the increase due to the vpp version update? Are there any known NSM issues that are meant to be fixed by the increased af_packet memory?
Might be similar/related to: #332
NSM 1.8.0
# smaps
7f6f5d22f000-7f6f6562f000 rw-s 00000000 00:07 20735 socket:[20735]
Size: 135168 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Rss: 135168 kB
Pss: 135168 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 129516 kB
Private_Dirty: 5652 kB
Referenced: 135168 kB
Anonymous: 0 kB
LazyFree: 0 kB
AnonHugePages: 0 kB
ShmemPmdMapped: 0 kB
FilePmdMapped: 0 kB
Shared_Hugetlb: 0 kB
Private_Hugetlb: 0 kB
Swap: 0 kB
SwapPss: 0 kB
Locked: 0 kB
THPeligible: 0
VmFlags: rd wr sh mr mw me ms mm
# lsof
vpp_main 6800 root mem REG 0,7 20735 socket:[20735]
vpp_main 6800 root 13u pack 20735 0t0 ALL type=SOCK_RAW
Before:
7f0655b5f000-7f0656f5f000 rw-s 00000000 00:08 134146836 socket:[134146836]
Size: 20480 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Rss: 20480 kB
Pss: 20480 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 16384 kB
Private_Dirty: 4096 kB
Referenced: 20480 kB
Anonymous: 0 kB
LazyFree: 0 kB
AnonHugePages: 0 kB
ShmemPmdMapped: 0 kB
FilePmdMapped: 0 kB
Shared_Hugetlb: 0 kB
Private_Hugetlb: 0 kB
Swap: 0 kB
SwapPss: 0 kB
Locked: 0 kB
THPeligible: 0
ProtectionKey: 0
VmFlags: rd wr sh mr mw me ms sd mm
Description
Ran part of the tests locally.
TestForwarderTestSuite/TestCombinations/Ethernet/Kernel/* tests are unstable.
Output:
--- FAIL: TestForwarderTestSuite (16.14s)
--- FAIL: TestForwarderTestSuite/TestCombinations (14.70s)
--- FAIL: TestForwarderTestSuite/TestCombinations/Ethernet (12.32s)
--- FAIL: TestForwarderTestSuite/TestCombinations/Ethernet/Kernel (12.32s)
--- PASS: TestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Kernel (0.63s)
--- FAIL: TestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Memif (10.86s)
--- PASS: TestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan (0.82s)
--- PASS: TestForwarderTestSuite/TestCombinations/Ethernet#01 (2.39s)
--- PASS: TestForwarderTestSuite/TestCombinations/Ethernet#01/Kernel (2.39s)
--- PASS: TestForwarderTestSuite/TestCombinations/Ethernet#01/Kernel/Kernel (0.69s)
--- PASS: TestForwarderTestSuite/TestCombinations/Ethernet#01/Kernel/Memif (0.84s)
--- PASS: TestForwarderTestSuite/TestCombinations/Ethernet#01/Kernel/Vxlan (0.85s)
--- PASS: TestForwarderTestSuite/TestHealthCheck (0.00s)
FAIL
Logs
Note: this log contains tests that were run twice
https://jira.fd.io/browse/VPP-2081
This issues is a continuation of this discussion - networkservicemesh/govpp#9 (comment)
In short, there are two main suspects:
During the integration, we noticed several performance issues
iperf3 TCPAF_PACKET:
Connecting to host 172.16.1.100, port 5201
[ 5] local 172.16.1.101 port 43488 connected to 172.16.1.100 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 46.6 MBytes 391 Mbits/sec 174 969 KBytes
[ 5] 1.00-2.00 sec 48.8 MBytes 409 Mbits/sec 0 1.02 MBytes
[ 5] 2.00-3.00 sec 58.8 MBytes 493 Mbits/sec 0 1.07 MBytes
[ 5] 3.00-4.00 sec 53.8 MBytes 451 Mbits/sec 0 1.10 MBytes
[ 5] 4.00-5.00 sec 46.2 MBytes 388 Mbits/sec 0 1.12 MBytes
[ 5] 5.00-6.00 sec 62.5 MBytes 524 Mbits/sec 0 1.13 MBytes
[ 5] 6.00-7.00 sec 45.0 MBytes 377 Mbits/sec 0 1.14 MBytes
[ 5] 7.00-8.00 sec 65.0 MBytes 545 Mbits/sec 0 1.18 MBytes
[ 5] 8.00-9.00 sec 56.2 MBytes 472 Mbits/sec 0 1.22 MBytes
[ 5] 9.00-10.00 sec 45.0 MBytes 377 Mbits/sec 0 1.24 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 528 MBytes 443 Mbits/sec 174 sender
[ 5] 0.00-10.08 sec 526 MBytes 438 Mbits/sec receiver
AF_XDP:
Connecting to host 172.16.1.100, port 5201
[ 5] local 172.16.1.101 port 36586 connected to 172.16.1.100 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 46.9 MBytes 393 Mbits/sec 1326 113 KBytes
[ 5] 1.00-2.00 sec 41.3 MBytes 346 Mbits/sec 1114 42.2 KBytes
[ 5] 2.00-3.00 sec 36.2 MBytes 304 Mbits/sec 1058 34.0 KBytes
[ 5] 3.00-4.00 sec 54.2 MBytes 455 Mbits/sec 1560 20.4 KBytes
[ 5] 4.00-5.00 sec 36.3 MBytes 304 Mbits/sec 1149 44.9 KBytes
[ 5] 5.00-6.00 sec 27.9 MBytes 234 Mbits/sec 953 20.4 KBytes
[ 5] 6.00-7.00 sec 37.9 MBytes 318 Mbits/sec 1106 25.9 KBytes
[ 5] 7.00-8.00 sec 33.1 MBytes 278 Mbits/sec 964 25.9 KBytes
[ 5] 8.00-9.00 sec 39.2 MBytes 329 Mbits/sec 1448 32.7 KBytes
[ 5] 9.00-10.00 sec 51.1 MBytes 429 Mbits/sec 1445 23.1 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 404 MBytes 339 Mbits/sec 12123 sender
[ 5] 0.00-10.00 sec 403 MBytes 338 Mbits/sec receiver
Note: many Retrs
AF_PACKET:
Connecting to host 172.16.1.100, port 5201
[ 5] local 172.16.1.101 port 49978 connected to 172.16.1.100 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 88.3 MBytes 740 Mbits/sec 2 487 KBytes
[ 5] 1.00-2.00 sec 87.4 MBytes 733 Mbits/sec 0 606 KBytes
[ 5] 2.00-3.00 sec 76.5 MBytes 642 Mbits/sec 6 495 KBytes
[ 5] 3.00-4.00 sec 74.6 MBytes 626 Mbits/sec 0 596 KBytes
[ 5] 4.00-5.00 sec 42.3 MBytes 355 Mbits/sec 0 649 KBytes
[ 5] 5.00-6.00 sec 21.7 MBytes 182 Mbits/sec 8 473 KBytes
[ 5] 6.00-7.00 sec 36.9 MBytes 310 Mbits/sec 0 545 KBytes
[ 5] 7.00-8.00 sec 88.9 MBytes 746 Mbits/sec 0 636 KBytes
[ 5] 8.00-9.00 sec 82.4 MBytes 691 Mbits/sec 8 539 KBytes
[ 5] 9.00-10.00 sec 92.0 MBytes 772 Mbits/sec 0 664 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 691 MBytes 580 Mbits/sec 24 sender
[ 5] 0.00-10.03 sec 690 MBytes 577 Mbits/sec receiver
AF_XDP:
Connecting to host 172.16.1.100, port 5201
[ 5] local 172.16.1.101 port 46608 connected to 172.16.1.100 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 104 MBytes 873 Mbits/sec 47 645 KBytes
[ 5] 1.00-2.00 sec 98.7 MBytes 828 Mbits/sec 39 538 KBytes
[ 5] 2.00-3.00 sec 90.9 MBytes 763 Mbits/sec 0 655 KBytes
[ 5] 3.00-4.00 sec 65.2 MBytes 547 Mbits/sec 14 533 KBytes
[ 5] 4.00-5.00 sec 53.3 MBytes 447 Mbits/sec 7 603 KBytes
[ 5] 5.00-6.00 sec 52.4 MBytes 440 Mbits/sec 0 660 KBytes
[ 5] 6.00-7.00 sec 39.1 MBytes 328 Mbits/sec 8 526 KBytes
[ 5] 7.00-8.00 sec 38.7 MBytes 325 Mbits/sec 0 587 KBytes
[ 5] 8.00-9.00 sec 94.8 MBytes 796 Mbits/sec 0 675 KBytes
[ 5] 9.00-10.00 sec 96.0 MBytes 805 Mbits/sec 7 618 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 733 MBytes 615 Mbits/sec 122 sender
[ 5] 0.00-10.05 sec 732 MBytes 611 Mbits/sec receiver
Average of 10 runs
Ethernet:
AF_PACKET is faster than AF_XDP by ~13% (460.3 Mbits/sec vs 407.2 Mbits/sec)
IP:
AF_XDP is equal to AF_PACKET (372,1 Mbits/sec vs 370,2 Mbits/sec)
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 46.9 MBytes 393 Mbits/sec 1326 113 KBytes
[ 5] 1.00-2.00 sec 41.3 MBytes 346 Mbits/sec 1114 42.2 KBytes
...
(we don't see them with AF_PACKET)
2. We was able to reproduce something similar on bare vpp instances:
https://lists.fd.io/g/vpp-dev/topic/af_xdp_performance/98105671
3. If we look at the vpp gerrit, we can see several open af_xdp patches:
https://gerrit.fd.io/r/c/vpp/+/37653
https://gerrit.fd.io/r/c/vpp/+/38135
https://github.com/networkservicemesh/cmd-forwarder-vpp/actions/runs/6850487114/job/18624747368
=== RUN TestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan
Nov 13 13:18:04.935 [INFO] Launching TestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan test server (time since start: 172.309µs)
Nov 13 13:18:04.936 [INFO] [test:TestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan] Launching TestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan test server (took : 266.413µs)
Nov 13 13:18:04.936 [INFO] [type:registry] nse-server-register={"name":"nse-TestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan","network_service_names":["nsTestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan"],"url":"tcp://127.0.0.1:42513"}
Nov 13 13:18:04.936 [INFO] [expireNSEServer:Register] [type:registry] selected expiration time 2023-11-13 14:18:04.936097591 +0000 UTC for nse-TestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan
Nov 13 13:18:04.936 [INFO] [type:registry] nse-server-register-response={"name":"nse-TestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan","network_service_names":["nsTestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan"],"url":"tcp://127.0.0.1:42513","expiration_time":{"seconds":1699885084,"nanos":936097591}}
Nov 13 13:18:04.936 [INFO] [test:TestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan] Registered ns and nse for server TestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan (took : 472.223µs)
Nov 13 13:18:04.936 [INFO] Sending Request to forwarder (time since start: 689.634µs)
Nov 13 13:18:04.936 [INFO] [id:] [type:networkService] client-request={"connection":{"network_service":"nsTestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan","payload":"ETHERNET"}}
Nov 13 13:18:04.937 [INFO] [id:b67a055f-b7cb-41ea-b3a3-c4da15115760] [type:networkService] server-request={"connection":{"id":"b67a055f-b7cb-41ea-b3a3-c4da15115760","network_service":"nsTestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan","path":{"path_segments":[{"name":"kernelVerifiableClient","id":"b67a055f-b7cb-41ea-b3a3-c4da15115760"}]},"payload":"ETHERNET"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"inodeURL":"inode://4/4026532465","name":"nsTestForw-b67a"}}]}
Nov 13 13:18:04.939 [INFO] [type:registry] ns-client-find={"network_service":{"name":"nsTestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan","payload":"ETHERNET"}}
Nov 13 13:18:04.942 [INFO] [type:registry] ns-client-find-response={"ClientStream":{}}
Nov 13 13:18:04.942 [INFO] [type:registry] ns-client-recv-response={"network_service":{"name":"nsTestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan","payload":"ETHERNET"}}
Nov 13 13:18:04.942 [INFO] [type:registry] ns-client-recv={"network_service":{"name":"nsTestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan","payload":"ETHERNET"}}
Nov 13 13:18:04.942 [INFO] [type:registry] nse-client-find={"network_service_endpoint":{"network_service_names":["nsTestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan"]}}
Nov 13 13:18:04.945 [INFO] [type:registry] nse-client-find-response={"ClientStream":{}}
Nov 13 13:18:04.945 [INFO] [type:registry] nse-server-find={"network_service_endpoint":{"network_service_names":["nsTestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan"]}}
Nov 13 13:18:04.945 [INFO] [type:registry] nse-server-send={"network_service_endpoint":{"name":"nse-TestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan","network_service_names":["nsTestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan"],"url":"tcp://127.0.0.1:42513","expiration_time":{"seconds":1699885084,"nanos":936097591}}}
Nov 13 13:18:04.945 [INFO] [type:registry] nse-server-send-response={"network_service_endpoint":{"name":"nse-TestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan","network_service_names":["nsTestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan"],"url":"tcp://127.0.0.1:42513","expiration_time":{"seconds":1699885084,"nanos":936097591}}}
Nov 13 13:18:04.945 [INFO] [type:registry] nse-server-find-response={"network_service_endpoint":{"network_service_names":["nsTestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan"]}}
Nov 13 13:18:04.946 [INFO] [type:registry] nse-client-recv-response={"network_service_endpoint":{"name":"nse-TestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan","network_service_names":["nsTestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan"],"url":"tcp://127.0.0.1:42513","expiration_time":{"seconds":1699885084,"nanos":936097591}}}
Nov 13 13:18:04.946 [INFO] [type:registry] nse-client-recv={"network_service_endpoint":{"name":"nse-TestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan","network_service_names":["nsTestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan"],"url":"tcp://127.0.0.1:42513","expiration_time":{"seconds":1699885084,"nanos":936097591}}}
Nov 13 13:18:04.946 [INFO] [id:b67a055f-b7cb-41ea-b3a3-c4da15115760] [type:networkService] client-request={"connection":{"id":"53ccb0c1-7e29-484a-b109-a810c5404650","network_service":"nsTestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan","mechanism":{"cls":"LOCAL","type":"KERNEL","parameters":{"inodeURL":"file:///proc/3886/fd/29","name":"nsTestForw-b67a"}},"context":{"MTU":9000},"path":{"index":1,"path_segments":[{"name":"kernelVerifiableClient","id":"b67a055f-b7cb-41ea-b3a3-c4da15115760","token":"***","expires":{"seconds":1699882084,"nanos":937151543}},{"name":"forwarder","id":"53ccb0c1-7e29-484a-b109-a810c5404650","token":"***","expires":{"seconds":1699882084,"nanos":937677069}}]},"network_service_endpoint_name":"nse-TestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan","payload":"ETHERNET"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"inodeURL":"file:///proc/3886/fd/29","name":"nsTestForw-b67a"}}]}
Nov 13 13:18:04.949 [INFO] [id:b67a055f-b7cb-41ea-b3a3-c4da15115760] [type:networkService] server-request={"connection":{"id":"53ccb0c1-7e29-484a-b109-a810c5404650","network_service":"nsTestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan","context":{"MTU":9000},"path":{"index":1,"path_segments":[{"name":"kernelVerifiableClient","id":"b67a055f-b7cb-41ea-b3a3-c4da15115760","token":"***","expires":{"seconds":1699882084,"nanos":937151543}},{"name":"forwarder","id":"53ccb0c1-7e29-484a-b109-a810c5404650","token":"***","expires":{"seconds":1699882084,"nanos":937677069}}]},"network_service_endpoint_name":"nse-TestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan","payload":"ETHERNET"},"mechanism_preferences":[{"cls":"REMOTE","type":"VXLAN","parameters":{"MTU":"1446","src_ip":"10.0.2.1","src_port":"4789"}},{"cls":"REMOTE","type":"VLAN"}]}
Nov 13 13:18:04.960 [DEBU] [id:b67a055f-b7cb-41ea-b3a3-c4da15115760] [VNIserver:request] [mechanism.DstIP:10.0.2.3] [mechanism.DstPort:4789] [type:networkService] set mechanism dst
Nov 13 13:18:04.960 [DEBU] [id:b67a055f-b7cb-41ea-b3a3-c4da15115760] [VNIserver:request] [type:networkService] [vni:12029528] vni generated and stored in metadata
Nov 13 13:18:04.960 [INFO] [id:b67a055f-b7cb-41ea-b3a3-c4da15115760] [type:networkService] server-request-response={"id":"a027cec7-cfb6-4b93-b[434](https://github.com/networkservicemesh/cmd-forwarder-vpp/actions/runs/6850487114/job/18624747368#step:5:435)-8924b6a98234","network_service":"nsTestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan","mechanism":{"cls":"REMOTE","type":"VXLAN","parameters":{"MTU":"1446","dst_ip":"10.0.2.3","dst_port":"4789","src_ip":"10.0.2.1","src_port":"4789","vni":"12029528"}},"context":{"ip_context":{"src_ip_addrs":["10.0.0.1/32","fc00::1/128"],"dst_ip_addrs":["10.0.0.0/32","fc00::/128"],"src_routes":[{"prefix":"10.0.0.0/32"},{"prefix":"fc00::/128"}],"dst_routes":[{"prefix":"10.0.0.1/32"},{"prefix":"fc00::1/128"}]},"MTU":1446},"path":{"index":2,"path_segments":[{"name":"kernelVerifiableClient","id":"b67a055f-b7cb-41ea-b3a3-c4da15115760","token":"***","expires":{"seconds":1699882084,"nanos":937151543}},{"name":"forwarder","id":"53ccb0c1-7e29-484a-b109-a810c5404650","token":"***","expires":{"seconds":1699882084,"nanos":949247536}},{"name":"vxlanVerifiableEndpoint","id":"a027cec7-cfb6-4b93-b434-8924b6a98234","token":"***","expires":{"seconds":1699882084,"nanos":949737260}}]},"network_service_endpoint_name":"nse-TestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan","payload":"ETHERNET"}
Nov 13 13:18:04.961 [DEBU] [id:b67a055f-b7cb-41ea-b3a3-c4da15115760] [NextIndex:3] [NextName:ethernet-input] [NodeName:vxlan4-input] [duration:141.407µs] [isAdd:true] [type:networkService] [vppapi:AddNodeNext] completed
time="2023-11-13T13:18:04Z" level=info msg="No subscription found for the notification message." msg_id=714 msg_size=19
Nov 13 13:18:04.961 [DEBU] [id:b67a055f-b7cb-41ea-b3a3-c4da15115760] [DstAddress:10.0.2.1] [SrcAddress:10.0.2.3] [Vni:12029528] [duration:357.818µs] [isAdd:true] [swIfIndex:2] [type:networkService] [vppapi:VxlanAddDelTunnel] completed
Nov 13 13:18:04.961 [DEBU] [id:b67a055f-b7cb-41ea-b3a3-c4da15115760] [type:networkService] [vxlan:addDel] not vxlan mechanism
Nov 13 13:18:04.961 [DEBU] [id:b67a055f-b7cb-41ea-b3a3-c4da15115760] [duration:254.212µs] [swIfIndex:1] [type:networkService] [vppapi:SwInterfaceDump] found interface with ip 10.0.2.3
Nov 13 13:18:04.961 [DEBU] [id:b67a055f-b7cb-41ea-b3a3-c4da15115760] [duration:126.706µs] [swIfIndex:1] [type:networkService] [vppapi:ACLInterfaceListDump] initiated
Nov 13 13:18:04.963 [DEBU] [id:b67a055f-b7cb-41ea-b3a3-c4da15115760] [aclIndex:0] [duration:1.045151ms] [type:networkService] [vppapi:ACLDump] completed
Nov 13 13:18:04.963 [DEBU] [id:b67a055f-b7cb-41ea-b3a3-c4da15115760] [aclIndex:1] [duration:101.405µs] [type:networkService] [vppapi:ACLDump] completed
Nov 13 13:18:04.963 [DEBU] [id:b67a055f-b7cb-41ea-b3a3-c4da15115760] [aclIndex:2] [duration:453.923µs] [type:networkService] [vppapi:ACLAddReplace] completed
Nov 13 13:18:04.963 [DEBU] [id:b67a055f-b7cb-41ea-b3a3-c4da15115760] [aclIndex:0] [duration:55.803µs] [type:networkService] [vppapi:ACLAddReplace] completed
time="2023-11-13T13:18:04Z" level=error msg="Channel ID not known, ignoring the message." channel=52 msg_id=71
Nov 13 13:18:04.964 [DEBU] [id:b67a055f-b7cb-41ea-b3a3-c4da15115760] [NInput:2] [acls:[2 0 0 1]] [duration:189.609µs] [swIfIndex:1] [type:networkService] [vppapi:ACLInterfaceSetACLList] completed
Nov 13 13:18:04.964 [DEBU] [id:b67a055f-b7cb-41ea-b3a3-c4da15115760] [duration:183.709µs] [isAdd:true] [prefix:[10.0.0.0/32 fc00::/128]] [swIfIndex:2] [type:networkService] [vppapi:SwInterfaceAddDelAddress] completed
Nov 13 13:18:04.964 [DEBU] [id:b67a055f-b7cb-41ea-b3a3-c4da15115760] [duration:279.614µs] [isAdd:true] [prefix:[10.0.0.0/32 fc00::/128]] [swIfIndex:2] [type:networkService] [vppapi:SwInterfaceAddDelAddress] completed
Nov 13 13:18:04.965 [DEBU] [id:b67a055f-b7cb-41ea-b3a3-c4da15115760] [duration:731.636µs] [isAdd:true] [isIpV6:false] [prefix:10.0.0.1/32] [swIfIndex:2] [tableID:0] [type:networkService] [vppapi:IPRouteAddDel] completed
Nov 13 13:18:04.965 [DEBU] [id:b67a055f-b7cb-41ea-b3a3-c4da15115760] [duration:129.206µs] [isAdd:true] [isIpV6:true] [prefix:fc00::1/128] [swIfIndex:2] [tableID:0] [type:networkService] [vppapi:IPRouteAddDel] completed
Nov 13 13:18:04.965 [DEBU] [id:b67a055f-b7cb-41ea-b3a3-c4da15115760] [duration:129.707µs] [isAdd:true] [isIpV6:false] [prefix:10.0.0.1/32] [swIfIndex:2] [tableID:0] [type:networkService] [vppapi:IPRouteAddDel] completed
Nov 13 13:18:04.966 [DEBU] [id:b67a055f-b7cb-41ea-b3a3-c4da15115760] [duration:319.916µs] [isAdd:true] [isIpV6:true] [prefix:fc00::1/128] [swIfIndex:2] [tableID:0] [type:networkService] [vppapi:IPRouteAddDel] completed
Nov 13 13:18:04.966 [DEBU] [id:b67a055f-b7cb-41ea-b3a3-c4da15115760] [MTU:[1446 1446 1446 1446]] [SwIfIndex:2] [duration:79.404µs] [type:networkService] [vppapi:SwInterfaceSetMtu] completed
Nov 13 13:18:04.966 [INFO] [id:b67a055f-b7cb-41ea-b3a3-c4da15115760] [type:networkService] client-request-response={"id":"53ccb0c1-7e29-484a-b109-a810c5404650","network_service":"nsTestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan","mechanism":{"cls":"REMOTE","type":"VXLAN","parameters":{"MTU":"1446","dst_ip":"10.0.2.3","dst_port":"4789","src_ip":"10.0.2.1","src_port":"4789","vni":"12029528"}},"context":{"ip_context":{"src_ip_addrs":["10.0.0.1/32","fc00::1/128"],"dst_ip_addrs":["10.0.0.0/32","fc00::/128"],"src_routes":[{"prefix":"10.0.0.0/32"},{"prefix":"fc00::/128"}],"dst_routes":[{"prefix":"10.0.0.1/32"},{"prefix":"fc00::1/128"}]},"MTU":1446},"path":{"index":1,"path_segments":[{"name":"kernelVerifiableClient","id":"b67a055f-b7cb-41ea-b3a3-c4da15115760","token":"***","expires":{"seconds":1699882084,"nanos":937151543}},{"name":"forwarder","id":"53ccb0c1-7e29-484a-b109-a810c5404650","token":"***","expires":{"seconds":1699882084,"nanos":949247536}},{"name":"vxlanVerifiableEndpoint","id":"a027cec7-cfb6-4b93-b434-8924b6a98234","token":"***","expires":{"seconds":1699882084,"nanos":949737260}}]},"network_service_endpoint_name":"nse-TestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan","payload":"ETHERNET"}
time="2023-11-13T13:18:04Z" level=info msg="No subscription found for the notification message." msg_id=714 msg_size=19
Nov 13 13:18:04.970 [INFO] [id:b67a055f-b7cb-41ea-b3a3-c4da15115760] [type:networkService] server-request-response={"id":"53ccb0c1-7e29-484a-b109-a810c5404650","network_service":"nsTestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan","mechanism":{"cls":"LOCAL","type":"KERNEL","parameters":{"inodeURL":"file:///proc/3886/fd/29","name":"nsTestForw-b67a"}},"context":{"ip_context":{"src_ip_addrs":["10.0.0.1/32","fc00::1/128"],"dst_ip_addrs":["10.0.0.0/32","fc00::/128"],"src_routes":[{"prefix":"10.0.0.0/32"},{"prefix":"fc00::/128"}],"dst_routes":[{"prefix":"10.0.0.1/32"},{"prefix":"fc00::1/128"}]},"MTU":1446},"path":{"index":1,"path_segments":[{"name":"kernelVerifiableClient","id":"b67a055f-b7cb-41ea-b3a3-c4da15115760","token":"***","expires":{"seconds":1699882084,"nanos":937151543}},{"name":"forwarder","id":"53ccb0c1-7e29-484a-b109-a810c5404650","token":"***","expires":{"seconds":1699882084,"nanos":949247536},"metrics":{"client_drops":"0","client_rx_bytes":"0","client_rx_packets":"0","client_tx_bytes":"0","client_tx_packets":"0"}},{"name":"vxlanVerifiableEndpoint","id":"a027cec7-cfb6-4b93-b434-8924b6a98234","token":"***","expires":{"seconds":1699882084,"nanos":949737260}}]},"network_service_endpoint_name":"nse-TestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan","payload":"ETHERNET"}
time="2023-11-13T13:18:05Z" level=info msg="No subscription found for the notification message." msg_id=714 msg_size=19
time="2023-11-13T13:18:06Z" level=info msg="No subscription found for the notification message." msg_id=714 msg_size=19
Nov 13 13:18:06.024 [INFO] [id:] [type:networkService] client-request-response={"id":"b67a055f-b7cb-41ea-b3a3-c4da15115760","network_service":"nsTestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan","mechanism":{"cls":"LOCAL","type":"KERNEL","parameters":{"inodeURL":"inode://4/4026532465","name":"nsTestForw-b67a"}},"context":{"ip_context":{"src_ip_addrs":["10.0.0.1/32","fc00::1/128"],"dst_ip_addrs":["10.0.0.0/32","fc00::/128"],"src_routes":[{"prefix":"10.0.0.0/32"},{"prefix":"fc00::/128"}],"dst_routes":[{"prefix":"10.0.0.1/32"},{"prefix":"fc00::1/128"}]},"MTU":1446},"path":{"path_segments":[{"name":"kernelVerifiableClient","id":"b67a055f-b7cb-41ea-b3a3-c4da15115760","token":"***","expires":{"seconds":1699882084,"nanos":937151543}},{"name":"forwarder","id":"53ccb0c1-7e29-484a-b109-a810c5404650","token":"***","expires":{"seconds":1699882084,"nanos":949247536},"metrics":{"client_drops":"0","client_rx_bytes":"0","client_rx_packets":"0","client_tx_bytes":"0","client_tx_packets":"0","server_drops":"0","server_rx_bytes":"0","server_rx_packets":"0","server_tx_bytes":"0","server_tx_packets":"0"}},{"name":"vxlanVerifiableEndpoint","id":"a027cec7-cfb6-4b93-b434-8924b6a98234","token":"***","expires":{"seconds":1699882084,"nanos":949737260}}]},"network_service_endpoint_name":"nse-TestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan","payload":"ETHERNET"}
Nov 13 13:18:06.024 [INFO] [id:] [type:networkService] expiration after 9m58.912243895s at 2023-11-13 13:28:04.937151543 +0000 UTC
Nov 13 13:18:06.025 [INFO] [test:TestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan] Sending Request to forwarder (took : 1.088756255s)
Nov 13 13:18:06.025 [INFO] Verifying Connection (time since start: 1.089495191s)
PING 10.0.0.0 (10.0.0.0) 3000(3028) bytes of data.
--- 10.0.0.0 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
suite_combinatronics_test.go:225:
Error Trace: /build/internal/tests/suite_combinatronics_test.go:225
Error: Received unexpected error:
exit status 1
failed to ping with command "ping -c 1 -s 3000 10.0.0.0"
github.com/networkservicemesh/cmd-forwarder-vpp/internal/tests.pingKernel
/build/internal/tests/suite_kernel_test.go:271
github.com/networkservicemesh/cmd-forwarder-vpp/internal/tests.(*kernelVerifiableClient).VerifyConnection
/build/internal/tests/suite_kernel_test.go:194
github.com/networkservicemesh/cmd-forwarder-vpp/internal/tests.(*ForwarderTestSuite).TestCombinations.func11.1.1
/build/internal/tests/suite_combinatronics_test.go:225
testing.tRunner
/go/src/testing/testing.go:1576
runtime.goexit
/go/src/runtime/asm_amd64.s:1598
Test: TestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan
Nov 13 13:18:16.127 [DEBU] [Cmd:&{ping 10.0.0.1 interval 0.1 repeat 1 verbose}] [Reply:Source address: 10.0.0.0
Statistics: 1 sent, 0 received, 100% packet loss
] [duration:100.400642ms] [test:TestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan] [vppapi:CliInband] completed;
Nov 13 13:18:16.228 [DEBU] [Cmd:&{ping 10.0.0.1 interval 0.1 repeat 1 verbose}] [Reply:Source address: 10.0.0.0
Statistics: 1 sent, 0 received, 100% packet loss
] [duration:100.35744ms] [test:TestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan] [vppapi:CliInband] completed;
suite_combinatronics_test.go:226:
Error Trace: /build/internal/tests/suite_combinatronics_test.go:226
Error: Received unexpected error:
Ping failed
github.com/networkservicemesh/cmd-forwarder-vpp/internal/tests.pingVpp
/build/internal/tests/suite_memif_test.go:182
github.com/networkservicemesh/cmd-forwarder-vpp/internal/tests.(*vxlanVerifiableEndpoint).VerifyConnection
/build/internal/tests/suite_vxlan_test.go:78
github.com/networkservicemesh/cmd-forwarder-vpp/internal/tests.(*ForwarderTestSuite).TestCombinations.func11.1.1
/build/internal/tests/suite_combinatronics_test.go:226
testing.tRunner
/go/src/testing/testing.go:1576
runtime.goexit
/go/src/runtime/asm_amd64.s:1598
Test: TestForwarderTestSuite/TestCombinations/Ethernet/Kernel/Vxlan
When running local-nse-death example we reconfigure NSE for different CIDR prefix. Example suggests pinging new IP address 172.16.1.102. In the same time, the old address 172.16.1.100 still available and ping passes.
If we shell into NSE pod
NSE=$(kubectl get pods -l app=nse-kernel -n ${NAMESPACE} --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')
kubectl exec -it ${NSE} -n ${NAMESPACE} -- shwe can see that its interface has two addresses assigned
ip addr
# ...
4: icmp-respo-9b77: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UNKNOWN qlen 1000
link/ether 02:fe:8c:fd:17:34 brd ff:ff:ff:ff:ff:ff
inet 172.16.1.100/32 scope global icmp-respo-9b77
valid_lft forever preferred_lft forever
inet 172.16.1.102/32 scope global icmp-respo-9b77
valid_lft forever preferred_lft foreverThe old address 172.16.1.100/32 still available even though it doesn't satisfy newly configured CIDR prefix 172.16.1.102/31
When running tests I sometimes see that forwarder is restarted, and the previous container stopped working after vpp binary got killed via SIGILL signal.
I got it several times in TestRunHealSuite/TestVl3_nscs_death test.
It seems like there is something wrong with IPSEC mechanism.
Jun 21 08:57:11.860�[37m [DEBU] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(11.1) passed clientURL: unix:///proc/1372636/fd/71
Jun 21 08:57:11.860�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(12) ⎆ sdk/pkg/networkservice/common/roundrobin/selectEndpointServer.Request()
Jun 21 08:57:11.860�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(13) ⎆ sdk-vpp/pkg/networkservice/stats/statsServer.Request()
Jun 21 08:57:11.860�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(14) ⎆ sdk-vpp/pkg/networkservice/up/upServer.Request()
Jun 21 08:57:11.861�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(15) ⎆ sdk/pkg/networkservice/core/next/nextServer.Request()
Jun 21 08:57:11.861�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(16) ⎆ sdk-vpp/pkg/networkservice/xconnect/l2xconnect/l2XconnectServer.Request()
Jun 21 08:57:11.861�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(17) ⎆ sdk-vpp/pkg/networkservice/xconnect/l3xconnect/l3XconnectServer.Request()
Jun 21 08:57:11.861�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(18) ⎆ sdk-vpp/pkg/networkservice/xconnect/l2bridgedomain/l2BridgeDomainServer.Request()
Jun 21 08:57:11.861�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(19) ⎆ sdk/pkg/networkservice/core/next/nextServer.Request()
Jun 21 08:57:11.861�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(20) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/mtu/mtuServer.Request()
Jun 21 08:57:11.862�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(21) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipneighbors/ipNeighborsServer.Request()
Jun 21 08:57:11.862�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(22) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/iprule/ipruleServer.Request()
Jun 21 08:57:11.862�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(23) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/routes/routesServer.Request()
Jun 21 08:57:11.862�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(24) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipaddress/ipaddressServer.Request()
Jun 21 08:57:11.862�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(25) ⎆ sdk-kernel/pkg/kernel/networkservice/ethernetcontext/vfEthernetContextServer.Request()
Jun 21 08:57:11.862�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(26) ⎆ sdk-kernel/pkg/kernel/networkservice/pinggrouprange/pinggrouprangeServer.Request()
Jun 21 08:57:11.863�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(27) ⎆ sdk-vpp/pkg/networkservice/tag/tagServer.Request()
Jun 21 08:57:11.863�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(28) ⎆ sdk-vpp/pkg/networkservice/connectioncontext/mtu/mtuServer.Request()
Jun 21 08:57:11.863�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(29) ⎆ sdk/pkg/networkservice/common/mechanisms/mechanismsServer.Request()
Jun 21 08:57:11.863�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(30) ⎆ sdk/pkg/networkservice/core/next/nextServer.Request()
Jun 21 08:57:11.863�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(31) ⎆ sdk-vpp/pkg/networkservice/mechanisms/wireguard/peer/wireguardPeerServer.Request()
Jun 21 08:57:11.863�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(32) ⎆ sdk-vpp/pkg/networkservice/mechanisms/wireguard/mtu/mtuServer.Request()
Jun 21 08:57:11.863�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(33) ⎆ sdk-vpp/pkg/networkservice/mechanisms/wireguard/wireguardServer.Request()
Jun 21 08:57:11.863�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(34) ⎆ sdk-vpp/pkg/networkservice/afxdppinhole/afxdppinholeServer.Request()
Jun 21 08:57:11.863�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(35) ⎆ sdk-vpp/pkg/networkservice/pinhole/pinholeServer.Request()
Jun 21 08:57:11.864�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(36) ⎆ sdk/pkg/networkservice/common/connect/connectServer.Request()
Jun 21 08:57:11.864�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(37) ⎆ sdk/pkg/networkservice/common/updatepath/updatePathClient.Request()
Jun 21 08:57:11.864�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(38) ⎆ sdk/pkg/networkservice/common/begin/beginClient.Request()
Jun 21 08:57:11.864�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(39) ⎆ sdk/pkg/networkservice/utils/metadata/metaDataClient.Request()
Jun 21 08:57:11.864�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(40) ⎆ sdk/pkg/networkservice/common/null/nullClient.Request()
Jun 21 08:57:11.864�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(41) ⎆ sdk/pkg/networkservice/common/clienturl/clientURLClient.Request()
Jun 21 08:57:11.864�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(42) ⎆ sdk/pkg/networkservice/common/clientconn/clientConnClient.Request()
Jun 21 08:57:11.864�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(43) ⎆ sdk/pkg/networkservice/common/null/nullClient.Request()
Jun 21 08:57:11.864�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(44) ⎆ sdk/pkg/networkservice/common/dial/dialClient.Request()
Jun 21 08:57:11.866�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(45) ⎆ sdk/pkg/networkservice/common/cleanup/cleanupClient.Request()
Jun 21 08:57:11.866�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(46) ⎆ sdk/pkg/networkservice/common/mechanismtranslation/mechanismTranslationClient.Request()
Jun 21 08:57:11.866�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(46.1) request-diff={"connection":{"mechanism":{"cls":"LOCAL","parameters":{"+inodeURL":"file:///proc/1372636/fd/91","+socketfile":"@/tmp/memif/54c18148-d1c4-4e78-bba7-09ff1c03588e/memif.socket","-MTU":"1440","-dst_ip":"172.18.0.2","-dst_port":"51820","-dst_public_key":"wrZT1YoKRJ8VyfYuw0h620f/s/69h95kQbq8gDsWxDQ=","-src_ip":"172.18.0.4","-src_port":"51820","-src_public_key":"q0JtDnTCVDvJLoTUpXvQzxU5UY1GZgaUt9IGb0HODTo="},"type":"MEMIF"}},"mechanism_preferences":{"-0":{"cls":"REMOTE","type":"WIREGUARD","parameters":{"MTU":"1440","src_ip":"172.18.0.4","src_port":"51820","src_public_key":"q0JtDnTCVDvJLoTUpXvQzxU5UY1GZgaUt9IGb0HODTo="}},"-1":{"cls":"REMOTE","type":"IPSEC","parameters":{"MTU":"1419","src_ip":"172.18.0.4","src_port":"4500","src_public_key":"MIICrzCCAZegAwIBAgIRAcuEpImNTUgYRQEjIoS9t5QwDQYJKoZIhvcNAQELBQAwETEPMA0GA1UEAxMGY2xpZW50MB4XDTIzMDYyMTA4NTcxMVoXDTIzMDYyMTA5MDIxMVowETEPMA0GA1UEAxMGY2xpZW50MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzqa5HLZ/Af/rZWKDlYjft3LWgN1ENLvmFpCTfjLmllXRU7YAge5Q8UAMAj3G/MCg33EVbigYYajT5Bh95apnpWPP5KiABR9RKofcxuX1w3KP/6vETCl2OXTVvai39Ec/FB6wLeq9/Wz0u953LjnaE2cRnOdBIEX42GfJ8xw502g5uCyXlAfRQ7shWC4YuxrM5Uf9tEu1JCPgCulLXeCD04tsNN7zifnexPp3LNBHtIVASqv/BSmZfJflr/uiZRYjTOrwu+H8EZIvXdANqieHouo2SGQtbsMpshosanCVwdc9ACJodqodCs+H3hHWl6PSrAJ/XobtssY6sZ52FA3KmwIDAQABowIwADANBgkqhkiG9w0BAQsFAAOCAQEAsCQo6aPQiGjBFRkb8PwnQHviO5I0eRrs49xjMUYAeCI+7NYVsLzQUFSFX076FOTLKEWJsx58PyJdXT7ZHTbj4W7UBIv8ps782nt1DQLI0h6MttO8T+m984yuAtq1+zCCM5SM1D3MgAMLyiIMGZ1N/i03uWIx8b1ZS8XL6Nccq5cH5Z752mPydEo0gWP5lFsC5h8Gr7UtemziRwRuEDHIF0RT2HHMpn8Qpdv7kAQAZiqc5ksSVvWCGlfZ8RxYS0a27lcEVZCnJ3Su/ljiNZk8/oMaTHNf36bfzpQfD9bBxwTndo+GzN3nc2rdozoLByh7S8TQ2hr6pSOxmv5ppOCK+Q=="}},"-2":{"cls":"REMOTE","type":"VLAN"}}}
Jun 21 08:57:11.866�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(47) ⎆ sdk/pkg/networkservice/core/next/nextClient.Request()
Jun 21 08:57:11.866�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(48) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/mtu/mtuClient.Request()
Jun 21 08:57:11.866�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(49) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipneighbors/ipNeighborsClient.Request()
Jun 21 08:57:11.867�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(50) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/routes/routesClient.Request()
Jun 21 08:57:11.867�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(51) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipaddress/ipaddressClient.Request()
Jun 21 08:57:11.867�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(52) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/routelocalnet/routeLocalNetClient.Request()
Jun 21 08:57:11.867�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(53) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/iptables4nattemplate/iptablesClient.Request()
Jun 21 08:57:11.867�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(54) ⎆ sdk-kernel/pkg/kernel/networkservice/pinggrouprange/pinggrouprangeClient.Request()
Jun 21 08:57:11.867�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(55) ⎆ sdk-vpp/pkg/networkservice/stats/statsClient.Request()
Jun 21 08:57:11.867�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(56) ⎆ sdk/pkg/networkservice/core/next/nextClient.Request()
Jun 21 08:57:11.868�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(57) ⎆ sdk-vpp/pkg/networkservice/up/peerup/peerupClient.Request()
Jun 21 08:57:11.868�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(58) ⎆ sdk-vpp/pkg/networkservice/up/upClient.Request()
Jun 21 08:57:11.868�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(59) ⎆ sdk-vpp/pkg/networkservice/up/ipsecup/ipsecUpClient.Request()
Jun 21 08:57:11.868�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(60) ⎆ sdk-vpp/pkg/networkservice/connectioncontext/mtu/mtuClient.Request()
Jun 21 08:57:11.868�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(61) ⎆ sdk-vpp/pkg/networkservice/tag/tagClient.Request()
Jun 21 08:57:11.868�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(62) ⎆ sdk/pkg/networkservice/core/next/nextClient.Request()
Jun 21 08:57:11.868�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(63) ⎆ sdk-vpp/pkg/networkservice/mechanisms/memif/memifrxmode/memifrxmodeClient.Request()
Jun 21 08:57:11.868�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(64) ⎆ sdk-vpp/pkg/networkservice/mechanisms/memif/memifClient.Request()
Jun 21 08:57:11.869�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(64.1) request-diff={"mechanism_preferences":{"+0":{"cls":"LOCAL","type":"MEMIF","parameters":{"inodeURL":""}}}}
Jun 21 08:57:11.869�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(65) ⎆ sdk-vpp/pkg/networkservice/mechanisms/kernel/kerneltap/kernelTapClient.Request()
Jun 21 08:57:11.869�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(65.1) request-diff={"mechanism_preferences":{"+1":{"cls":"LOCAL","type":"KERNEL"}}}
Jun 21 08:57:11.869�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(66) ⎆ sdk/pkg/networkservice/core/next/nextClient.Request()
Jun 21 08:57:11.869�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(67) ⎆ sdk-vpp/pkg/networkservice/mechanisms/vxlan/vxlanClient.Request()
Jun 21 08:57:11.869�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(68) ⎆ sdk-vpp/pkg/networkservice/mechanisms/vxlan/mtu/mtuClient.Request()
Jun 21 08:57:11.869�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(69) ⎆ sdk/pkg/networkservice/common/mechanisms/vxlan/vni/vniClient.Request()
Jun 21 08:57:11.869�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(70) ⎆ sdk/pkg/networkservice/core/next/nextClient.Request()
Jun 21 08:57:11.869�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(71) ⎆ sdk-vpp/pkg/networkservice/mechanisms/wireguard/peer/wireguardPeerClient.Request()
Jun 21 08:57:11.869�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(72) ⎆ sdk-vpp/pkg/networkservice/mechanisms/wireguard/wireguardClient.Request()
Jun 21 08:57:11.870�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(72.1) request-diff={"mechanism_preferences":{"+2":{"cls":"REMOTE","type":"WIREGUARD","parameters":{"src_ip":"172.18.0.2","src_port":"51820","src_public_key":"q2hp0Ds53jbV9Pl/rBPd5lJqxgiPlL/PeJrTlf+8s2w="}}}}
Jun 21 08:57:11.870�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(73) ⎆ sdk-vpp/pkg/networkservice/mechanisms/wireguard/mtu/mtuClient.Request()
Jun 21 08:57:11.870�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(73.1) request-diff={"mechanism_preferences":{"2":{"parameters":{"+MTU":"1440"}}}}
Jun 21 08:57:11.870�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(74) ⎆ sdk/pkg/networkservice/core/next/nextClient.Request()
Jun 21 08:57:11.870�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(75) ⎆ sdk-vpp/pkg/networkservice/mechanisms/ipsec/ipsecClient.Request()
Jun 21 08:57:12.068�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(75.1) request-diff={"mechanism_preferences":{"+3":{"cls":"REMOTE","type":"IPSEC","parameters":{"src_ip":"172.18.0.2","src_port":"4500","src_public_key":"MIICrzCCAZegAwIBAgIRAhVCi7+tDlRG9Be00cdg5R8wDQYJKoZIhvcNAQELBQAwETEPMA0GA1UEAxMGY2xpZW50MB4XDTIzMDYyMTA4NTcxMloXDTIzMDYyMTA5MDIxMlowETEPMA0GA1UEAxMGY2xpZW50MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxepQyn7j3z+bgu30WpEunznuINEAzNTfLr5TQ4yEEy9kCV10czklrR3VT0uRemR1QL1JH6myJF2bWZ7jy/bpAEy2U6QR37UYbjHn+ubxj0PDv+x8ZK1NPRqJIT4HQDDJ7s7vgkbs3CmtMp3yT1z4q0ya3urUE0Pz6+i7Dl6IHgugu8sMxe9Kk4ifigwmNz3oM1+ZsMAAoHtgMZdpEiPxayxUBDcGdgE52fw7SRfRc3aoEh71C5+wMXOg8bC+LncV32dg0jmaPb7l9i9D52YMyUyUdYtuW/6kbLSGWeheurKFxfQbvFzhfYI9dJJepPuS5Fnq5h8ZA3u0ljQdiwy6JQIDAQABowIwADANBgkqhkiG9w0BAQsFAAOCAQEAQXk5t1Ibfn7hzse1iQat16l/A1V4vKeSz/0OUVYRrDjk6Zkfx2rudtwn2HsFBj+PVRglSAkjhQ2Y/GMg/1/C9dAukJT5diq/gXdLM6xYI1qjQ0hanRf/JUGY/oXBNYv9aBbOcFptYxkcqvEzJJMCWATpwdayxhZJMqFYFDRH+HsiVzFxlhlNLKFPT4g/Pbqg4O296NKMGRLJI4Pc47Lzb3ABNlbdc6WoOVhmC9MXAkfSnvVckuOmDe7Oi9uK4vsTcCpChapRLVHptH7Xk8gPCr3e+HFozlyhA0uhELT5zul0KsNfUhzxCtBnDkB0KmjcM41rccOAQhS8Fo4j6veGDg=="}}}}
Jun 21 08:57:12.068�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(76) ⎆ sdk-vpp/pkg/networkservice/mechanisms/ipsec/mtu/mtuClient.Request()
Jun 21 08:57:12.068�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(76.1) request-diff={"mechanism_preferences":{"3":{"parameters":{"+MTU":"1419"}}}}
Jun 21 08:57:12.069�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(77) ⎆ sdk/pkg/networkservice/core/next/nextClient.Request()
Jun 21 08:57:12.069�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(78) ⎆ sdk-vpp/pkg/networkservice/mechanisms/vlan/mtu/mtuClient.Request()
Jun 21 08:57:12.069�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(79) ⎆ sdk-vpp/pkg/networkservice/mechanisms/vlan/l2vtr/l2vtrClient.Request()
Jun 21 08:57:12.069�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(80) ⎆ sdk-vpp/pkg/networkservice/mechanisms/vlan/vlanClient.Request()
Jun 21 08:57:12.069�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(80.1) request-diff={"mechanism_preferences":{"+4":{"cls":"REMOTE","type":"VLAN"}}}
Jun 21 08:57:12.069�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(81) ⎆ sdk/pkg/networkservice/common/filtermechanisms/filterMechanismsClient.Request()
Jun 21 08:57:12.070�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(81.1) request-diff={"mechanism_preferences":{"-2":{"cls":"REMOTE","type":"WIREGUARD","parameters":{"MTU":"1440","src_ip":"172.18.0.2","src_port":"51820","src_public_key":"q2hp0Ds53jbV9Pl/rBPd5lJqxgiPlL/PeJrTlf+8s2w="}},"-3":{"cls":"REMOTE","type":"IPSEC","parameters":{"MTU":"1419","src_ip":"172.18.0.2","src_port":"4500","src_public_key":"MIICrzCCAZegAwIBAgIRAhVCi7+tDlRG9Be00cdg5R8wDQYJKoZIhvcNAQELBQAwETEPMA0GA1UEAxMGY2xpZW50MB4XDTIzMDYyMTA4NTcxMloXDTIzMDYyMTA5MDIxMlowETEPMA0GA1UEAxMGY2xpZW50MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxepQyn7j3z+bgu30WpEunznuINEAzNTfLr5TQ4yEEy9kCV10czklrR3VT0uRemR1QL1JH6myJF2bWZ7jy/bpAEy2U6QR37UYbjHn+ubxj0PDv+x8ZK1NPRqJIT4HQDDJ7s7vgkbs3CmtMp3yT1z4q0ya3urUE0Pz6+i7Dl6IHgugu8sMxe9Kk4ifigwmNz3oM1+ZsMAAoHtgMZdpEiPxayxUBDcGdgE52fw7SRfRc3aoEh71C5+wMXOg8bC+LncV32dg0jmaPb7l9i9D52YMyUyUdYtuW/6kbLSGWeheurKFxfQbvFzhfYI9dJJepPuS5Fnq5h8ZA3u0ljQdiwy6JQIDAQABowIwADANBgkqhkiG9w0BAQsFAAOCAQEAQXk5t1Ibfn7hzse1iQat16l/A1V4vKeSz/0OUVYRrDjk6Zkfx2rudtwn2HsFBj+PVRglSAkjhQ2Y/GMg/1/C9dAukJT5diq/gXdLM6xYI1qjQ0hanRf/JUGY/oXBNYv9aBbOcFptYxkcqvEzJJMCWATpwdayxhZJMqFYFDRH+HsiVzFxlhlNLKFPT4g/Pbqg4O296NKMGRLJI4Pc47Lzb3ABNlbdc6WoOVhmC9MXAkfSnvVckuOmDe7Oi9uK4vsTcCpChapRLVHptH7Xk8gPCr3e+HFozlyhA0uhELT5zul0KsNfUhzxCtBnDkB0KmjcM41rccOAQhS8Fo4j6veGDg=="}},"-4":{"cls":"REMOTE","type":"VLAN"}}}
Jun 21 08:57:12.070�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(82) ⎆ sdk/pkg/networkservice/common/mechanismpriority/mechanismPriorityClient.Request()
Jun 21 08:57:12.070�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(83) ⎆ sdk-vpp/pkg/networkservice/afxdppinhole/afxdppinholeClient.Request()
Jun 21 08:57:12.070�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(84) ⎆ sdk-vpp/pkg/networkservice/pinhole/pinholeClient.Request()
Jun 21 08:57:12.070�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(85) ⎆ sdk/pkg/networkservice/common/mechanisms/recvfd/recvFDClient.Request()
Jun 21 08:57:12.070�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(86) ⎆ sdk-vpp/pkg/networkservice/nsmonitor/netNSMonitorClient.Request()
Jun 21 08:57:12.070�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(87) ⎆ sdk/pkg/networkservice/common/mechanisms/sendfd/sendFDClient.Request()
Jun 21 08:57:12.149�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(87.1) request-diff={"connection":{"mechanism":{"parameters":{"inodeURL":"inode://4/4026536825"}}}}
Jun 21 08:57:12.149�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(88) ⎆ sdk/pkg/networkservice/common/null/nullClient.Request()
Jun 21 08:57:12.149�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(89) ⎆ sdk/pkg/networkservice/common/trimpath/trimpathClient.Request()
Jun 21 08:57:12.150�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(90) ⎆ sdk/pkg/networkservice/common/connect/connectClient.Request()
Jun 21 08:57:12.162�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(90.1) request-response={"id":"6f2a74b1-8374-4a46-9fc4-6a80a699a195","network_service":"vl3-nscs-death","mechanism":{"cls":"LOCAL","type":"MEMIF","parameters":{"inodeURL":"inode://4/4026536825","socketfile":"@/tmp/memif/54c18148-d1c4-4e78-bba7-09ff1c03588e/memif.socket"}},"context":{"ip_context":{"src_ip_addrs":["172.16.1.0/32"],"dst_ip_addrs":["172.16.0.0/32"],"src_routes":[{"prefix":"172.16.0.0/32","nextHop":"172.16.0.0"},{"prefix":"172.16.0.0/24","nextHop":"172.16.0.0"},{"prefix":"172.16.0.0/16","nextHop":"172.16.0.0"}],"dst_routes":[{"prefix":"172.16.1.0/32","nextHop":"172.16.1.0"},{"prefix":"172.16.1.0/24","nextHop":"172.16.1.0"}],"excluded_prefixes":["10.96.0.0/16","10.244.0.0/16"]},"dns_context":{"configs":[{"dns_server_ips":["172.16.1.1"]},{"dns_server_ips":["172.16.0.1"]}]},"MTU":1419},"labels":{"nodeName":"kind-1-worker"},"path":{"index":4,"path_segments":[{"name":"nse-vl3-vpp-6fcbdd789-vnzxf","id":"70535973-443f-4804-8675-d7ebc4c1bd45","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9rOHMubnNtL25zL25zLXZsMy1uc2NzLWRlYXRoL3BvZC9uc2UtdmwzLXZwcC02ZmNiZGQ3ODktdm56eGYiLCJhdWQiOlsic3BpZmZlOi8vazhzLm5zbS9ucy9uc20tc3lzdGVtL3BvZC9uc21nci1scGxwciJdLCJleHAiOjE2ODczMzg0MzF9.-sQjTAA9nFTIMWb6zPm_cwkZMC5fRbZjOWmEQ_BrIiJLdYY-v_8g5DfeaEkTq9KaKEO7mwQi4FeoQqW58EupWw","expires":{"seconds":1687338431,"nanos":612578270}},{"name":"nsmgr-lplpr","id":"c598d08e-60ac-4964-a572-ab1d498d735a","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9rOHMubnNtL25zL25zbS1zeXN0ZW0vcG9kL25zbWdyLWxwbHByIiwiYXVkIjpbInNwaWZmZTovL2s4cy5uc20vbnMvbnNtLXN5c3RlbS9wb2QvZm9yd2FyZGVyLXZwcC14dGJjaCJdLCJleHAiOjE2ODczMzg0MzF9.-wXAXIZCIYPyXR2JhKfn63K6KtxW65OARAHuNjxnyEStVzhPORI8H2Hhl8wGpxhOjwtObOkXLibFWgMFfmIjVw","expires":{"seconds":1687338431,"nanos":635961004}},{"name":"forwarder-vpp-xtbch","id":"bd1f6393-690c-45b6-a735-b92c76eed59d","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9rOHMubnNtL25zL25zbS1zeXN0ZW0vcG9kL2ZvcndhcmRlci12cHAteHRiY2giLCJhdWQiOlsic3BpZmZlOi8vazhzLm5zbS9ucy9uc20tc3lzdGVtL3BvZC9uc21nci13ZjJkYyJdLCJleHAiOjE2ODczMzg0MzF9.L9xQ1QACSEV2E-4Aqxp5fqyZGcjGA52HTfR3jShGeotwlH54oRcO7qg6llL5cuuAVJUataX86_rKFwhgs_1reA","expires":{"seconds":1687338431,"nanos":794206618},"metrics":{"client_drops":"0","client_rx_bytes":"0","client_rx_packets":"1","client_tx_bytes":"0","client_tx_packets":"0","server_drops":"0","server_rx_bytes":"0","server_rx_packets":"0","server_tx_bytes":"0","server_tx_packets":"0"}},{"name":"nsmgr-wf2dc","id":"a800c425-88dd-46a6-ab76-649116e565d6","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9rOHMubnNtL25zL25zbS1zeXN0ZW0vcG9kL25zbWdyLXdmMmRjIiwiYXVkIjpbInNwaWZmZTovL2s4cy5uc20vbnMvbnNtLXN5c3RlbS9wb2QvZm9yd2FyZGVyLXZwcC12eHFodyJdLCJleHAiOjE2ODczMzg0MzF9.wfC2J9AyzuWq1iWT8YyPoE2mchNoeTDLArp9zrDfuYCQPnKli1jXRzgRAyudEkp9k7pzywUkRwdol7cuZF769Q","expires":{"seconds":1687338431,"nanos":813842128}},{"name":"forwarder-vpp-vxqhw","id":"6f2a74b1-8374-4a46-9fc4-6a80a699a195","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9rOHMubnNtL25zL25zbS1zeXN0ZW0vcG9kL2ZvcndhcmRlci12cHAtdnhxaHciLCJhdWQiOlsic3BpZmZlOi8vazhzLm5zbS9ucy9ucy12bDMtbnNjcy1kZWF0aC9wb2QvbnNlLXZsMy12cHAtNmZjYmRkNzg5LWd4ZnJ6Il0sImV4cCI6MTY4NzMzODQzMn0.vMQ2pLvYbQYBxlhnNoUmzgZoIobMK7m316nwUaT0gTKg2SxQMqQwA2YjDZTNexlEcUjHQYIolZ5jgJr5T4DiKw","expires":{"seconds":1687338432,"nanos":150173859},"metrics":{"client_drops":"0","client_rx_bytes":"0","client_rx_packets":"0","client_tx_bytes":"0","client_tx_packets":"0","server_drops":"0","server_rx_bytes":"0","server_rx_packets":"0","server_tx_bytes":"0","server_tx_packets":"0"}},{"name":"nse-vl3-vpp-6fcbdd789-gxfrz","id":"54c18148-d1c4-4e78-bba7-09ff1c03588e","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9rOHMubnNtL25zL25zLXZsMy1uc2NzLWRlYXRoL3BvZC9uc2UtdmwzLXZwcC02ZmNiZGQ3ODktZ3hmcnoiLCJhdWQiOlsic3BpZmZlOi8vazhzLm5zbS9ucy9uc20tc3lzdGVtL3BvZC9mb3J3YXJkZXItdnBwLXZ4cWh3Il0sImV4cCI6MTY4NzMzODQzMn0.-HBARSKu8GDZpEwECr_dFyXpIEnS2Wy_iPpHDDL4VJjE6zkMUPC2Kwxn6Wj3QsI3X5twsfSNbZWCZzbXseqszg","expires":{"seconds":1687338432,"nanos":151602764}}]},"network_service_endpoint_name":"nse-vl3-vpp-6fcbdd789-gxfrz","payload":"IP"}
Jun 21 08:57:12.162�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(87.2) request-response-diff={"mechanism":{"parameters":{"inodeURL":"file:///proc/1372636/fd/91"}}}
Jun 21 08:57:12.165�[37m [DEBU] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [duration:127.129µs] [swIfIndex:6] [tag:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] [vppapi:SwInterfaceTagAddDel] �[0m(61.1) completed
Jun 21 08:57:12.165�[37m [DEBU] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [MTU:[1419 1419 1419 1419]] [SwIfIndex:6] [duration:88.887µs] [type:networkService] [vppapi:SwInterfaceSetMtu] �[0m(60.1) completed
Jun 21 08:57:12.165�[37m [DEBU] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [duration:62.527µs] [swIfIndex:6] [type:networkService] [vppapi:SwInterfaceSetFlags] �[0m(58.1) completed
Jun 21 08:57:12.165�[37m [DEBU] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [details.Flags:IF_STATUS_API_FLAG_ADMIN_UP|IF_STATUS_API_FLAG_LINK_UP] [duration:100.619µs] [swIfIndex:6] [type:networkService] [vppapi:SwInterfaceDump] �[0m(58.2) completed
Jun 21 08:57:12.167�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(46.2) request-response-diff={"mechanism":{"cls":"REMOTE","parameters":{"+MTU":"1440","+dst_ip":"172.18.0.2","+dst_port":"51820","+dst_public_key":"wrZT1YoKRJ8VyfYuw0h620f/s/69h95kQbq8gDsWxDQ=","+src_ip":"172.18.0.4","+src_port":"51820","+src_public_key":"q0JtDnTCVDvJLoTUpXvQzxU5UY1GZgaUt9IGb0HODTo=","-inodeURL":"file:///proc/1372636/fd/91","-socketfile":"@/tmp/memif/54c18148-d1c4-4e78-bba7-09ff1c03588e/memif.socket"},"type":"WIREGUARD"}}
Jun 21 08:57:12.168�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(36.1) request-diff={"connection":{"context":{"MTU":1419,"dns_context":{"configs":{"+1":{"dns_server_ips":["172.16.0.1"]}}}},"mechanism":{"cls":"REMOTE","parameters":{"+MTU":"1440","+dst_ip":"172.18.0.2","+dst_port":"51820","+dst_public_key":"wrZT1YoKRJ8VyfYuw0h620f/s/69h95kQbq8gDsWxDQ=","+src_ip":"172.18.0.4","+src_port":"51820","+src_public_key":"q0JtDnTCVDvJLoTUpXvQzxU5UY1GZgaUt9IGb0HODTo=","-inodeURL":"inode://4/4026536825","-socketfile":"@/tmp/memif/54c18148-d1c4-4e78-bba7-09ff1c03588e/memif.socket"},"type":"WIREGUARD"},"path":{"path_segments":{"4":{"expires":{"nanos":150173859,"seconds":1687338432},"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9rOHMubnNtL25zL25zbS1zeXN0ZW0vcG9kL2ZvcndhcmRlci12cHAtdnhxaHciLCJhdWQiOlsic3BpZmZlOi8vazhzLm5zbS9ucy9ucy12bDMtbnNjcy1kZWF0aC9wb2QvbnNlLXZsMy12cHAtNmZjYmRkNzg5LWd4ZnJ6Il0sImV4cCI6MTY4NzMzODQzMn0.vMQ2pLvYbQYBxlhnNoUmzgZoIobMK7m316nwUaT0gTKg2SxQMqQwA2YjDZTNexlEcUjHQYIolZ5jgJr5T4DiKw"},"5":{"expires":{"nanos":151602764,"seconds":1687338432},"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9rOHMubnNtL25zL25zLXZsMy1uc2NzLWRlYXRoL3BvZC9uc2UtdmwzLXZwcC02ZmNiZGQ3ODktZ3hmcnoiLCJhdWQiOlsic3BpZmZlOi8vazhzLm5zbS9ucy9uc20tc3lzdGVtL3BvZC9mb3J3YXJkZXItdnBwLXZ4cWh3Il0sImV4cCI6MTY4NzMzODQzMn0.-HBARSKu8GDZpEwECr_dFyXpIEnS2Wy_iPpHDDL4VJjE6zkMUPC2Kwxn6Wj3QsI3X5twsfSNbZWCZzbXseqszg"}}}},"mechanism_preferences":{"+2":{"cls":"REMOTE","type":"VLAN"},"0":{"cls":"REMOTE","parameters":{"+MTU":"1440","+src_ip":"172.18.0.4","+src_port":"51820","+src_public_key":"q0JtDnTCVDvJLoTUpXvQzxU5UY1GZgaUt9IGb0HODTo=","-inodeURL":""},"type":"WIREGUARD"},"1":{"cls":"REMOTE","parameters":{"+MTU":"1419","+src_ip":"172.18.0.4","+src_port":"4500","+src_public_key":"MIICrzCCAZegAwIBAgIRAcuEpImNTUgYRQEjIoS9t5QwDQYJKoZIhvcNAQELBQAwETEPMA0GA1UEAxMGY2xpZW50MB4XDTIzMDYyMTA4NTcxMVoXDTIzMDYyMTA5MDIxMVowETEPMA0GA1UEAxMGY2xpZW50MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzqa5HLZ/Af/rZWKDlYjft3LWgN1ENLvmFpCTfjLmllXRU7YAge5Q8UAMAj3G/MCg33EVbigYYajT5Bh95apnpWPP5KiABR9RKofcxuX1w3KP/6vETCl2OXTVvai39Ec/FB6wLeq9/Wz0u953LjnaE2cRnOdBIEX42GfJ8xw502g5uCyXlAfRQ7shWC4YuxrM5Uf9tEu1JCPgCulLXeCD04tsNN7zifnexPp3LNBHtIVASqv/BSmZfJflr/uiZRYjTOrwu+H8EZIvXdANqieHouo2SGQtbsMpshosanCVwdc9ACJodqodCs+H3hHWl6PSrAJ/XobtssY6sZ52FA3KmwIDAQABowIwADANBgkqhkiG9w0BAQsFAAOCAQEAsCQo6aPQiGjBFRkb8PwnQHviO5I0eRrs49xjMUYAeCI+7NYVsLzQUFSFX076FOTLKEWJsx58PyJdXT7ZHTbj4W7UBIv8ps782nt1DQLI0h6MttO8T+m984yuAtq1+zCCM5SM1D3MgAMLyiIMGZ1N/i03uWIx8b1ZS8XL6Nccq5cH5Z752mPydEo0gWP5lFsC5h8Gr7UtemziRwRuEDHIF0RT2HHMpn8Qpdv7kAQAZiqc5ksSVvWCGlfZ8RxYS0a27lcEVZCnJ3Su/ljiNZk8/oMaTHNf36bfzpQfD9bBxwTndo+GzN3nc2rdozoLByh7S8TQ2hr6pSOxmv5ppOCK+Q=="},"type":"IPSEC"}}}
Jun 21 08:57:12.169�[37m [DEBU] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [MTU:[1419 1419 1419 1419]] [SwIfIndex:7] [duration:94.207µs] [type:networkService] [vppapi:SwInterfaceSetMtu] �[0m(28.1) completed
Jun 21 08:57:12.169�[37m [DEBU] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [duration:60.143µs] [swIfIndex:7] [tag:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] [vppapi:SwInterfaceTagAddDel] �[0m(27.1) completed
Jun 21 08:57:12.170�[37m [DEBU] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [IsIP6:false] [Paths[0].SwIfIndex:7] [SwIfIndex:6] [duration:123.788µs] [type:networkService] [vppapi:L3xcUpdate] �[0m(17.1) completed
Jun 21 08:57:12.171�[37m [DEBU] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [IsIP6:true] [Paths[0].SwIfIndex:7] [SwIfIndex:6] [duration:108.083µs] [type:networkService] [vppapi:L3xcUpdate] �[0m(17.2) completed
Jun 21 08:57:12.171�[37m [DEBU] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [IsIP6:false] [Paths[0].SwIfIndex:6] [SwIfIndex:7] [duration:106.415µs] [type:networkService] [vppapi:L3xcUpdate] �[0m(17.3) completed
Jun 21 08:57:12.171�[37m [DEBU] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [IsIP6:true] [Paths[0].SwIfIndex:6] [SwIfIndex:7] [duration:78.172µs] [type:networkService] [vppapi:L3xcUpdate] �[0m(17.4) completed
Jun 21 08:57:12.171�[37m [DEBU] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [duration:76.779µs] [swIfIndex:6] [type:networkService] [vppapi:SwInterfaceSetFlags] �[0m(14.1) completed
Jun 21 08:57:12.171�[37m [DEBU] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [details.Flags:IF_STATUS_API_FLAG_ADMIN_UP|IF_STATUS_API_FLAG_LINK_UP] [duration:89.889µs] [swIfIndex:6] [type:networkService] [vppapi:SwInterfaceDump] �[0m(14.2) completed
Jun 21 08:57:12.171�[37m [DEBU] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [duration:82.374µs] [swIfIndex:7] [type:networkService] [vppapi:SwInterfaceSetFlags] �[0m(14.3) completed
Jun 21 08:57:12.172�[37m [TRAC] [id:6f2a74b1-8374-4a46-9fc4-6a80a699a195] [type:networkService] �[0m(13.1) request-response-diff={"path":{"path_segments":{"4":{"metrics":{"server_rx_packets":"1"}}}}}
Jun 21 08:57:12.173�[37m [TRAC] [id:a800c425-88dd-46a6-ab76-649116e565d6] [type:networkService] �[0m(1.3) request-response-diff={"id":"a800c425-88dd-46a6-ab76-649116e565d6","path":{"index":3}}
time="2023-06-21T08:57:12Z" level=info msg="No subscription found for the notification message." msg_id=701 msg_size=19
Jun 21 08:57:12.265�[37m [DEBU] [id:0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [duration:343.415µs] [swIfIndex:11] [type:networkService] [vppapi:IpsecItfCreate] �[0m(32.2) completed
Jun 21 08:57:12.265�[37m [DEBU] [id:0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [IsAdd:true] [Name:server-0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [duration:303.715µs] [type:networkService] [vppapi:Ikev2ProfileAddDel] �[0m(32.3) completed
Jun 21 08:57:12.265�[37m [DEBU] [id:0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [Name:server-0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [duration:81.523µs] [type:networkService] [vppapi:Ikev2ProfileSetUDPEncap] �[0m(32.4) completed
Jun 21 08:57:12.267�[37m [DEBU] [id:0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [operation:dumpCertBase64ToFile] [type:networkService] �[0m(32.5) completed
Jun 21 08:57:12.268�[37m [DEBU] [id:0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [operation:dumpPrivateKeyToFile] [type:networkService] �[0m(32.6) completed
Jun 21 08:57:12.268�[37m [DEBU] [id:0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [Name:server-0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [duration:199.826µs] [type:networkService] [vppapi:Ikev2ProfileSetAuth] �[0m(32.7) completed
Jun 21 08:57:12.268�[37m [DEBU] [id:0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [duration:115.366µs] [type:networkService] [vppapi:Ikev2SetLocalKey] �[0m(32.8) completed
Jun 21 08:57:12.268�[37m [DEBU] [id:0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [IsLocal:true] [Name:server-0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [duration:64.571µs] [type:networkService] [vppapi:Ikev2ProfileSetID] �[0m(32.9) completed
Jun 21 08:57:12.268�[37m [DEBU] [id:0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [IsLocal:false] [Name:server-0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [duration:142.788µs] [type:networkService] [vppapi:Ikev2ProfileSetID] �[0m(32.10) completed
Jun 21 08:57:12.268�[37m [DEBU] [id:0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [Address:172.16.0.2/32] [IsLocal:false] [Name:server-0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [duration:66.265µs] [type:networkService] [vppapi:Ikev2ProfileSetTs] �[0m(32.11) completed
Jun 21 08:57:12.268�[37m [DEBU] [id:0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [Address:172.16.0.0/32] [IsLocal:true] [Name:server-0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [duration:192.813µs] [type:networkService] [vppapi:Ikev2ProfileSetTs] �[0m(32.12) completed
Jun 21 08:57:12.269�[37m [DEBU] [id:0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [Name:server-0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [SwIfIndex:11] [duration:76.264µs] [type:networkService] [vppapi:Ikev2SetTunnelInterface] �[0m(32.13) completed
Jun 21 08:57:12.269�[37m [TRAC] [id:0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [type:networkService] �[0m(32.14) request-response-diff={"mechanism":{"parameters":{"+dst_public_key":"MIICrzCCAZegAwIBAgIRAeKKTaoXXjtH5VdvVIIEa1swDQYJKoZIhvcNAQELBQAwETEPMA0GA1UEAxMGc2VydmVyMB4XDTIzMDYyMTA4NTcxMloXDTIzMDYyMTA5MDIxMlowETEPMA0GA1UEAxMGc2VydmVyMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAul/GxRPs7g/5kK0uLz9xNr1FEM/8q6eQyDKHzypYr+DftUku0OGOPZmmgGU348LxYHT4+N0QgCqoGx8bz0r4iO50zwf3UluxaInD7aOfpLJGyyd+QtCIykY3s07uCJs5cxF5reXheN9V5YwixsrP9I5G+yEDLLCyrPOffafRgS674XJAS25eupj8G8vwPWTB6JOhBfV8atKfIg6k3zU1Oni1Exk1ItrfuY402HmPamkj3gWh8K0OCuUXNORZMMTvS0Io8DpHyyrxF82DLf1wvWjrRj8N68d6sAVI0ZziDLY3yzq5TAUX6/z/bBahIDtnt1bO0Zv755nrvxNfYPB0yQIDAQABowIwADANBgkqhkiG9w0BAQsFAAOCAQEADVGkNXog0QwIUYRb9yhFMIcjsdtn81WuMoteeguXblBz7OCbveBbOmMk+1aS3UBiwl/gB07WJZ/MjgYKnfhIMwHJR9sVAEoGQ+kWk+ajI0PxX0YLUjz4TvPR+57/9aKj3XpBVdeGB0dYSGr4WQ54f+Chvba4Qwjq+TkMDgAaXdhvjjzS9/j6LERfP9vyCwWsuPmUPTqpq51fIqaH8f8UNQ+gbz8m6Vi6Qt5Iu9YQkWPY3TxRMJTnSMNjdbhwS2NhOdPkHytyz1wJIaGrO90C0sQoAV1NL2D8W5rLpuOz2Ly7zwg3IxhHnoHecM4Vw56cn5o7vYdUEFPTIzlf9emVjw=="}}}
Jun 21 08:57:12.269�[37m [DEBU] [id:0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [MTU:[1419 1419 1419 1419]] [SwIfIndex:11] [duration:77.631µs] [type:networkService] [vppapi:SwInterfaceSetMtu] �[0m(28.1) completed
Jun 21 08:57:12.269�[37m [DEBU] [id:0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [duration:58.976µs] [swIfIndex:11] [tag:0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [type:networkService] [vppapi:SwInterfaceTagAddDel] �[0m(27.1) completed
Jun 21 08:57:12.271�[37m [DEBU] [id:0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [IsIP6:false] [Paths[0].SwIfIndex:11] [SwIfIndex:10] [duration:160.371µs] [type:networkService] [vppapi:L3xcUpdate] �[0m(17.1) completed
Jun 21 08:57:12.271�[37m [DEBU] [id:0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [IsIP6:true] [Paths[0].SwIfIndex:11] [SwIfIndex:10] [duration:211.206µs] [type:networkService] [vppapi:L3xcUpdate] �[0m(17.2) completed
Jun 21 08:57:12.271�[37m [DEBU] [id:0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [IsIP6:false] [Paths[0].SwIfIndex:10] [SwIfIndex:11] [duration:162.916µs] [type:networkService] [vppapi:L3xcUpdate] �[0m(17.3) completed
Jun 21 08:57:12.271�[37m [DEBU] [id:0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [IsIP6:true] [Paths[0].SwIfIndex:10] [SwIfIndex:11] [duration:201.86µs] [type:networkService] [vppapi:L3xcUpdate] �[0m(17.4) completed
Jun 21 08:57:12.272�[37m [DEBU] [id:0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [duration:88.576µs] [swIfIndex:10] [type:networkService] [vppapi:SwInterfaceSetFlags] �[0m(14.1) completed
Jun 21 08:57:12.272�[37m [DEBU] [id:0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [details.Flags:IF_STATUS_API_FLAG_ADMIN_UP|IF_STATUS_API_FLAG_LINK_UP] [duration:104.852µs] [swIfIndex:10] [type:networkService] [vppapi:SwInterfaceDump] �[0m(14.2) completed
time="2023-06-21T08:57:12Z" level=info msg="No subscription found for the notification message." msg_id=701 msg_size=19
Jun 21 08:57:12.272�[37m [DEBU] [id:0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [duration:185.453µs] [swIfIndex:11] [type:networkService] [vppapi:SwInterfaceSetFlags] �[0m(14.3) completed
Jun 21 08:57:12.273�[37m [TRAC] [id:0f824f1c-7bda-44c0-9a96-3fb6a9d5c7a2] [type:networkService] �[0m(13.1) request-response-diff={"path":{"path_segments":{"4":{"metrics":{"+server_drops":"0","+server_rx_bytes":"0","+server_rx_packets":"0","+server_tx_bytes":"0","+server_tx_packets":"0"}}}}}
Jun 21 08:57:12.276�[37m [TRAC] [id:2f12ea96-0946-416e-a262-afa8c445afe0] [type:networkService] �[0m(1.3) request-response-diff={"id":"2f12ea96-0946-416e-a262-afa8c445afe0","path":{"index":3}}
Jun 21 08:57:12.298�[36m [INFO] [cmd:vpp] �[0mvpp[1372682]: received signal SIGILL, PC 0x7f3c779b9aa3
Jun 21 08:57:12.300�[36m [INFO] [cmd:vpp] �[0mvpp[1372682]: #0 0x00007f3cb9238a5b 0x7f3cb9238a5b
Jun 21 08:57:12.300�[36m [INFO] [cmd:vpp] �[0mvpp[1372682]: #1 0x00007f3cb914f420 0x7f3cb914f420
Jun 21 08:57:12.300�[36m [INFO] [cmd:vpp] �[0mvpp[1372682]: #2 0x00007f3c779b9aa3 0x7f3c779b9aa3
Jun 21 08:57:12.300�[36m [INFO] [cmd:vpp] �[0mvpp[1372682]: #3 0x00007f3c779c8e26 0x7f3c779c8e26
Jun 21 08:57:12.301�[36m [INFO] [cmd:vpp] �[0mvpp[1372682]: #4 0x00007f3cb97f938b vnet_crypto_key_add + 0x8ab
Jun 21 08:57:12.301�[36m [INFO] [cmd:vpp] �[0mvpp[1372682]: #5 0x00007f3cb9813353 ipsec_sa_add_and_lock + 0x733
Jun 21 08:57:12.301�[36m [INFO] [cmd:vpp] �[0mvpp[1372682]: #6 0x00007f3c773f9239 0x7f3c773f9239
Jun 21 08:57:12.301�[36m [INFO] [cmd:vpp] �[0mvpp[1372682]: #7 0x00007f3cbad1c96a vl_api_rpc_call_main_thread + 0xaa
Jun 21 08:57:12.301�[36m [INFO] [cmd:vpp] �[0mvpp[1372682]: #8 0x00007f3c773f5f17 0x7f3c773f5f17
Jun 21 08:57:12.301�[36m [INFO] [cmd:vpp] �[0mvpp[1372682]: #9 0x00007f3c773ec003 0x7f3c773ec003
Jun 21 08:57:12.301�[36m [INFO] [cmd:vpp] �[0mvpp[1372682]: #10 0x00007f3cb91cc1d0 0x7f3cb91cc1d0
Jun 21 08:57:12.301�[36m [INFO] [cmd:vpp] �[0mvpp[1372682]: #11 0x00007f3cb91ca755 0x7f3cb91ca755
Jun 21 08:57:12.301�[36m [INFO] [cmd:vpp] �[0mvpp[1372682]: #12 0x00007f3cb923833b 0x7f3cb923833b
Jun 21 08:57:12.301�[36m [INFO] [cmd:vpp] �[0mvpp[1372682]: #13 0x00007f3cb90d36c8 0x7f3cb90d36c8
Jun 21 08:57:13.051�[31m [ERRO] [cmd:/bin/forwarder] �[0msignal: aborted
MTU of the node interface hosting NSM_TUNNEL_IP raised from initial 1500 bytes when using vpp-forwarder v1.3.0-rc.1 (AF_PACKET mode).
Thus causing traffic disturbances if the "base" interface (e.g. tap) MTU is 1500 bytes on the host.
Mostly the new value ends up being 1544 bytes (but in one occasion it turned out to be 1588 bytes).
Also, in case there are multiple network interfaces on the VM, their MTU might get changed as well even though TUNNEL_IP is not hosted by them. (I guess it's because I have a DeviceSelectorFile configured as well. Although occasionally not all the devices in the file are affected by the MTU change).
Interface MTUs are left intact when running with vpp-forwarder v1.2.0.
Also, when rebuilt with the VPP version used in v1.2.0, the problem does not appear.
To reproduce:
IPv4 traffic impact can be verified simply with netcat (assuming the new MTU is 1544 bytes):
k8s impact:
If kube-apiserver is running on a VM that also hosts a vpp-forwarder, then kubernetes API server requests from other nodes might randomly fail resulting in the following error:
vm-002 ~ # kubectl get pods -n nsm
E0406 17:32:54.015653 15189 request.go:1085] Unexpected error when reading response body: http2: client connection lost
unexpected error when reading response body. Please retry. Original error: http2: client connection lost
Forwarder unexpectedly restarted with a segfault when then endpoint got removed.
The issue is rarely reproducible..
Feb 10 20:09:19.978�[36m [INFO] [cmd:vpp] �[0mvpp[14]: received signal SIGSEGV, PC 0x7f274a25b520, faulting address 0x0
Feb 10 20:09:19.978�[36m [INFO] [cmd:vpp] �[0mvpp[14]: #0 0x00007f2749cef640 unix_signal_handler + 0x100
Feb 10 20:09:19.978�[36m [INFO] [cmd:vpp] �[0mvpp[14]: #1 0x00007f2749602050 0x7f2749602050
Feb 10 20:09:19.978�[36m [INFO] [cmd:vpp] �[0mvpp[14]: #2 0x00007f274a25b520 vnet_hw_if_update_runtime_data + 0x1a30
Feb 10 20:09:19.978�[36m [INFO] [cmd:vpp] �[0mvpp[14]: #3 0x00007f274a226347 vnet_delete_hw_interface + 0x87
Feb 10 20:09:19.978�[36m [INFO] [cmd:vpp] �[0mvpp[14]: #4 0x00007f274a26d238 ethernet_delete_interface + 0x548
Feb 10 20:09:19.978�[36m [INFO] [cmd:vpp] �[0mvpp[14]: #5 0x00007f274a7be8ed tap_delete_if + 0x13d
Feb 10 20:09:19.978�[36m [INFO] [cmd:vpp] �[0mvpp[14]: #6 0x00007f274a7c3a30 vl_api_tap_delete_v2_t_handler + 0x70
Feb 10 20:09:19.978�[36m [INFO] [cmd:vpp] �[0mvpp[14]: #7 0x00007f274af611a1 vl_msg_api_handler_no_free + 0xd1
Feb 10 20:09:19.978�[36m [INFO] [cmd:vpp] �[0mvpp[14]: #8 0x00007f274af399a5 vl_socket_process_api_msg + 0x15
Feb 10 20:09:19.978�[36m [INFO] [cmd:vpp] �[0mvpp[14]: #9 0x00007f274af437cb vl_api_clnt_process + 0xdeb
Feb 10 20:09:19.978�[36m [INFO] [cmd:vpp] �[0mvpp[14]: #10 0x00007f2749c96966 vlib_process_bootstrap + 0x16
Feb 10 20:09:19.978�[36m [INFO] [cmd:vpp] �[0mvpp[14]: #11 0x00007f27491d51c8 0x7f27491d51c8
Feb 10 20:17:13.298�[31m [ERRO] [cmd:/bin/forwarder] �[0msignal: aborted
Feb 10 20:17:13.299�[31m [ERRO] [cmd:/bin/forwarder] �[0m%!s(<nil>)
NSM component versions are v1.2.0-rc.1
The vlan mechanism should contain vlan tag and interface name. The NSE plays a controller role only set the mechanism parameters and provides IPAM. There is no data plane for the Network Service. The forwarder should contain client processing the remote vlan parameters and "push pop of vlan encap on the wire".
Example sketched by @edwarnicke:
Say we have the NetworkService 'finance-bridge-domain' being provided by the physical network.
An NSE can be run on a Pod, registering itself directly (rather than via the NSMGR) for that Network Service.
When a Client requests the 'finance-bridge-domain', its local NSMGR finds a URL for the Pod of the NSE in the Registry
The NSMGR sends the Request to the NSE
The NSE maps the Network Service Name 'finance-bridge-domain' to a vlan tag (say 12) and returns the remote mechanism of type vlan (class remote) with tag 12 as its parameter.
The NSMgr plumbs in whatever the local mechanism is for the client in such a way it will arrive at the physical network on vlan 12.
Related Issue: cmd-forwarder-vpp#355 Move some generic functions from internal/vppinit to sdk-vpp
Related PRs:
Sample NSE for reference: https://github.com/Nordix/nsm-nse-generic/tree/vlansup-dev
Please implement some functionality that can be used as livenessProbe and readinessProbe in a helm chart, like the HTTP probes for the old-gen NSM version.
cmd-forwarder-vpp shouldn't reach memory limit during tests.
cmd-forwarder-vpp reaches current memory limit (400Mi) during basic, heal tests on packet after VPP upgrade.
Run basic, heal tests on packet.
Tested on deployments commits:
For security reasons I want to start the VPP forwarder container as non root. Can you provide some info if it can be done, what capabilities are needed, etc?
Thanks for the help
In startup phase 2 (run vpp and get a connection to it) forwarder-vpp does not wait for the VPP conn to get ready before moving on to subsequent phases. Thus vppinit.InitLinks might fail thus basically rendering the forwarder non functional.
VPP is not running on the host when starting the forwarder.
The problem seems to occur sporadically.
A DeviceSelectorFile is used containing around 7-8 interface entries.
In vpphelper.StartAndDialContext() func DialContext() returns with the connection while the connect might be still in progress.
However, the ready channel is not checked by the forwarder (the Connection interface doesn't seem to provide any means for that). Also, error returned by vppinit.InitLinks() is not considered fatal.
NSM version: v1.4.0-rc.1
k8s: v1.22.2
The CONNECT_TO variable is used for the NSM URL. It is not informative and should be replaced with something better, e.g "NSM_URL". This goes for all NSM modules where "CONNECT_TO" is used.
In this particular module, as a consequence of the poor naming, CONNECT_TO is incorrectly used to connect to the registry.
Lines 175 to 185 in 1e88454
I use this fix for the problem;
--- a/main.go
+++ b/main.go
@@ -56,6 +56,7 @@ type Config struct {
NSName string `default:"xconnectns" desc:"Name of Network Service to Register with Registry"`
TunnelIP net.IP `desc:"IP to use for tunnels" split_words:"true"`
ConnectTo url.URL `default:"unix:///connect.to.socket" desc:"url to connect to" split_words:"true"`
+ RegistryURL url.URL `default:"tcp://nsm-registry-svc:5002" desc:"url to registry" split_words:"true"`
MaxTokenLifetime time.Duration `default:"24h" desc:"maximum lifetime of tokens" split_words:"true"`
}
@@ -178,8 +179,9 @@ func main() {
now = time.Now()
registryCreds := credentials.NewTLS(tlsconfig.MTLSClientConfig(source, source, tlsconfig.AuthorizeAny()))
registryCreds = grpcfd.TransportCredentials(registryCreds)
+ logrus.Infof("NSM: Connecting to NSE registry %v", config.RegistryURL.String())
registryCC, err := grpc.DialContext(ctx,
- config.ConnectTo.String(),
+ grpcutils.URLToTarget(&config.RegistryURL),
grpc.WithTransportCredentials(registryCreds),
grpc.WithBlock(),
)Jaeger traces are not being sent from the cmd-forwarder-vpp instance--testing with all-in-one Jaeger deployment in k8s and JAEGER_AGENT_HOST set to point to the Jaeger agent.
Image: ghcr.io/networkservicemesh/cmd-forwarder-vpp:v1.0.0
The other cmd-forwarder implementations invoke InitJaeger() in main.go but this is missing from cmd-forwarder-vpp.
This is a follow up of the issue #358 .
A new repository should be added to demonstrate the remote vlan mechanis support - functionality added to vpp forwarder. The name of the new repository should be: 'cmd-nse-remote-vlan' if that correspond to used naming conventions (open for any other name)
Sample NSE for reference: https://github.com/Nordix/nsm-nse-generic/tree/vlansup-dev
Currently NSM uses v19.06 of VPP. The current version is v21.06. We should upgrade.
To do this I would suggest first upgrading a for of https://github.com/edwarnicke/govpp to VPP v21.06. Many (but not all) of the patches/ will have already been incorporated into that version of VPP.
Then I would suggest upgrading sdk-vpp to use the updated binapi, and testing with cmd-forwarder-vpp with the base Docker images updated.
Occasionally (not frequently) we are seeing issues like https://github.com/networkservicemesh/cmd-forwarder-vpp/pull/184/checks?check_run_id=2556708327
These are characterized by:
May 11 14:34:23.449 [WARN] [name:forwarder] (1.2) an error during getting token from the context: metadata is missed in ctx
github.com/networkservicemesh/sdk/pkg/tools/token.FromContext
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/tools/token/context.go:38
And appear to happen around VXLAN cases.
May be related to: #64
Foud on aws long ci testing
"ready": true,
"restartCount": 2,
"image": "ghcr.io/networkservicemesh/ci/cmd-forwarder-vpp:836920c",
"imageID": "docker-pullable://ghcr.io/networkservicemesh/ci/cmd-forwarder-vpp@sha256:4a8d6cf9f9565a1897786c4737bd418524bd27d5e1f4e184f3be20d86ba3500a",
"containerID": "docker://f2bc9cf5045bc0b07dee25d58bf798980afc85f6b495111c84e8aa9cbf76f40a",
"started": true
actual: OOM
expected: everything is good
Run forwarder-vpp with mem profiler
This issue is related to the following PR: networkservicemesh/sdk-vpp#634
When an NSM connection is set up to a service provided by Remote VLAN NSE, the interface injected into the NSC POD is not usable for minutes. According to vppcli the bridge wasn't available (present) and showed up with the same delay.
What seemingly fixes the "connection"/bridge is the first refresh of the NSM connection.
Also, I rebuilt forwarder-vpp on the version where above changed got included, and run that forwarder on NSM v1.6 with the same results.
Note: When testing preliminary drop of the feature such delay was not visible, unfortunately I have no information of its code base (image = registry.nordix.org/cloud-native/nsm/cmd-forwarder-vpp:with-bd-wout-mac).
Reproduction (on Kind):
cat <<EOF | kubectl apply -f -
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nse-vlan
labels:
app: nse-vlan
spec:
selector:
matchLabels:
app: nse-vlan
template:
metadata:
labels:
app: nse-vlan
"spiffe.io/spiffe-id": "true"
spec:
containers:
- name: nse
image: registry.nordix.org/cloud-native/nsm/cmd-nse-remote-vlan:v1.6.0
imagePullPolicy: Always
env:
- name: SPIFFE_ENDPOINT_SOCKET
value: unix:///run/spire/sockets/agent.sock
- name: NSM_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NSM_LOG_LEVEL
value: TRACE
- name: NSM_CONNECT_TO
value: "nsm-registry-svc.nsm:5002"
- name: NSM_CIDR_PREFIX
value: "169.254.100.0/24,100:100::/64"
- name: NSM_SERVICES
value: "vlan-svc { vlan: 100; via: eth0 }"
- name: NSM_LISTEN_ON
value: "tcp://:5003"
- name: NSM_POINT2POINT
value: "False"
- name: NSM_REGISTER_SERVICE
value: "True"
volumeMounts:
- name: spire-agent-socket
mountPath: /run/spire/sockets
readOnly: true
resources:
requests:
cpu: 100m
memory: 40Mi
limits:
memory: 80Mi
cpu: 200m
nodeSelector:
kubernetes.io/hostname: kind-worker2
volumes:
- name: spire-agent-socket
hostPath:
path: /run/spire/sockets
type: Directory
EOF
cat <<EOF | kubectl apply -f -
---
apiVersion: v1
kind: Pod
metadata:
name: nsc
labels:
app: nsc
"spiffe.io/spiffe-id": "true"
spec:
containers:
- name: nsc
image: registry.nordix.org/cloud-native/nsm/cmd-nsc:v1.6.0
imagePullPolicy: Always
env:
- name: NSM_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: SPIFFE_ENDPOINT_SOCKET
value: unix:///run/spire/sockets/agent.sock
- name: NSM_LOG_LEVEL
value: TRACE
- name: NSM_NETWORK_SERVICES
value: kernel://vlan-svc/ext-1
- name: NSM_LIVENESSCHECKENABLED
value: "false"
- name: NSM_LOCALDNSSERVERENABLED
value: "false"
- name: NSM_DIAL_TIMEOUT
value: "30s"
- name: NSM_REQUEST_TIMEOUT
value: "300s"
volumeMounts:
- name: spire-agent-socket
mountPath: /run/spire/sockets
readOnly: true
- name: nsm-socket
mountPath: /var/lib/networkservicemesh
readOnly: true
resources:
requests:
cpu: 100m
memory: 40Mi
limits:
memory: 80Mi
cpu: 200m
volumes:
- name: spire-agent-socket
hostPath:
path: /run/spire/sockets
type: Directory
- name: nsm-socket
hostPath:
path: /var/lib/networkservicemesh
type: DirectoryOrCreate
nodeSelector:
kubernetes.io/hostname: kind-worker
EOF
root@kind-worker:/# vppctl show bridge-domain
no bridge-domains in use
Logs:
forwarder-vpp-ddqqp.5ed4cbb45dd25ee.log collocated with the NSC
First Request for service vlan-svc at Dec 15 14:51:22.210.
Refreshed at Dec 15 14:53:22.291 after which the bridge appeared in vpp.
root@kind-worker:/# date; vppctl show bridge-domain
Thu Dec 15 14:53:28 UTC 2022
BD-ID Index BSN Age(min) Learning U-Forwrd UU-Flood Flooding ARP-Term arp-ufwd Learn-co Learn-li BVI-Intf
7271263 1 0 off on on flood on off off 0 16777216 N/A
root@kind-worker:/# date; vppctl show bridge-domain 7271263 details
Thu Dec 15 14:53:37 UTC 2022
BD-ID Index BSN Age(min) Learning U-Forwrd UU-Flood Flooding ARP-Term arp-ufwd Learn-co Learn-li BVI-Intf
7271263 1 0 off on on flood on off off 0 16777216 N/A
SPAN (span-l2-input)
INPUT_CLASSIFY (l2-input-classify)
INPUT_FEAT_ARC (l2-input-feat-arc)
POLICER_CLAS (l2-policer-classify)
ACL (l2-input-acl)
VPATH (vpath-input-l2)
L2_IP_QOS_RECORD (l2-ip-qos-record)
VTR (l2-input-vtr)
LEARN (l2-learn)
RW (l2-rw)
FWD (l2-fwd)
UU_FLOOD (l2-flood)
FLOOD (l2-flood)
XCONNECT (l2-output)
Interface If-idx ISN SHG BVI TxFlood VLAN-Tag-Rewrite
tap0 3 1 1 - * none
host-eth0.100 2 1 0 - * pop-1
Client or Endpoint could change connection proto-state on refresh requests. For example, change Mac addresses or IP addresses or change routes, and so on.
The question is: Should forwarder-vpp apply the change?
Actual: client still have old IPs
Expected: ?
Forwarder mounts /var/run/vpp/vpp-api.sock by default to work with an external vpp (Calico case).
https://github.com/networkservicemesh/cmd-forwarder-vpp/blob/main/internal/config/config.go#L49
https://github.com/networkservicemesh/deployments-k8s/blob/main/apps/forwarder-vpp/forwarder.yaml#L56-L57
But if we use several forwarders without Calico, they can affect each other, because they create local vpp sockets in the same folder.
Chain of nsm applications should support refresh scenario when tokens are update whole the chain.
We need to be sure that this scenario is working correctly with cmd-forwarder-vpp.
Any issues with this should be fixed.
Reproduce via refresh
Reproduce via double request
Currently cmd-forwarder-vpp uses AF_PACKET to bind to an existing Node interface using LinkToAfPacket
AF_XDP is faster than than AF_PACKET, but AF_XDP is only useable for our purposes from kernel version 5.4 onward. The good news is that lots of places have kernel versions that new (including the more recent version of Docker Desktop).
Because AF_XDP is only supported for newer kernels, a check will need to be made and then the correct method (AF_XDP if available, otherwise AF_PACKET) before choosing the method to use.
In case of an IPv6 cluster dynamic neighbor address resolution does not work, because ingress ACL rules drop ICMPv6 Neighbor Advertisement messages.
This for example might lead to non-working remote VxLAN xconnects. (In case a VPP instance started by the forwarder was not informed about a certain neighbor at start, or a new node is added to the cluster.)
Trace in vppctl:
Packet 41
06:31:58:815638: af-packet-input
af_packet: hw_if_index 2 next-index 4
tpacket2_hdr:
status 0x20000005 len 86 snaplen 86 mac 66 net 80
sec 0x645130d0 nsec 0x12f3d524 vlan 0 vlan_tpid 0
06:31:58:815641: ethernet-input
IP6: fa:16:3e:a9:a5:9f -> fa:16:3e:d7:89:d1
06:31:58:815642: ip6-input
ICMP6: fd08::d -> fd08::21
tos 0x00, flow label 0x0, hop limit 255, payload length 32
06:31:58:815643: acl-plugin-in-ip6-fa
acl-plugin: lc_index: 1, sw_if_index 2, next index 0, action: 0, match: acl 0 rule 3 trace_bits 00000000
pkt info 00000000000008fd 0d00000000000000 00000000000008fd 2100000000000000 0002033a00000088 0a00ffff00000001
lc_index 1 l3 ip6 fd08::d -> fd08::21 l4 lsb_of_sw_if_index 2 proto 58 l4_is_input 1 l4_slow_path 1 l4_flags 0x03
port 136 -> 0 tcp flags (invalid) 00 rsvd 0
06:31:58:815646: error-drop
rx:host-eth0
06:31:58:815647: drop
acl-plugin-in-ip6-fa: ACL deny packets
vpp# show acl-plugin acl
acl-index 0 count 4 tag {nsm-vppinit-denyall-ingress}
0: ipv6 permit src ::/0 dst ::/0 proto 58 sport 0 dport 134
1: ipv6 permit src ::/0 dst ::/0 proto 58 sport 0 dport 136
2: ipv4 deny src 0.0.0.0/0 dst 0.0.0.0/0 proto 0 sport 0 dport 0
3: ipv6 deny src ::/0 dst ::/0 proto 0 sport 0 dport 0
applied inbound on sw_if_index: 2
applied outbound on sw_if_index:
used in lookup context index: 1
acl-index 1 count 4 tag {nsm-vppinit-denyall-egress}
0: ipv4 permit src 0.0.0.0/0 dst 0.0.0.0/0 proto 58 sport 133 dport 0
1: ipv6 permit src ::/0 dst ::/0 proto 58 sport 135 dport 0
2: ipv4 deny src 0.0.0.0/0 dst 0.0.0.0/0 proto 0 sport 0 dport 0
3: ipv4 deny src 0.0.0.0/0 dst 0.0.0.0/0 proto 0 sport 0 dport 0
applied inbound on sw_if_index:
applied outbound on sw_if_index: 2
used in lookup context index: 0
acl-index 2 count 1 tag {nsm-pinhole port 4789}
0: ipv6 permit src ::/0 dst fd08::21/128 proto 17 sport 0-65535 dport 4789
applied inbound on sw_if_index: 2
applied outbound on sw_if_index:
used in lookup context index: 1
acl-index 3 count 1 tag {nsm-pinhole port 4789}
0: ipv6 permit src fd08::21/128 dst ::/0 proto 17 sport 0-65535 dport 4789
applied outbound on sw_if_index: 2
used in lookup context index: 0
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: forwarder-vpp
labels:
app: forwarder-vpp
spec:
selector:
matchLabels:
app: forwarder-vpp
template:
metadata:
labels:
app: forwarder-vpp
spec:
containers:
- image: networkservicemeshci/cmd-forwarder-vpp:latest
imagePullPolicy: IfNotPresent
name: forwarder-vpp
securityContext:
privileged: true
env:
- name: SPIFFE_ENDPOINT_SOCKET
value: unix:///run/spire/sockets/agent.sock
- name: NSM_TUNNEL_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: NSM_CONNECT_TO
value: unix:///var/lib/networkservicemesh/nsm.io.sock
- name: NSM_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
volumeMounts:
- name: spire-agent-socket
mountPath: /run/spire/sockets
readOnly: true
- name: nsm-socket
mountPath: /var/lib/networkservicemesh
volumes:
- name: spire-agent-socket
hostPath:
path: /run/spire/sockets
type: Directory
- name: nsm-socket
hostPath:
path: /var/lib/networkservicemesh
type: DirectoryOrCreate
ifconfig in nseActual:
eth0 Link encap:Ethernet HWaddr BA:22:C2:E9:DA:8F
inet addr:10.244.2.10 Bcast:10.244.2.255 Mask:255.255.255.0
inet6 addr: fe80::b822:c2ff:fee9:da8f/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:10 errors:0 dropped:0 overruns:0 frame:0
TX packets:22 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1374 (1.3 KiB) TX bytes:1662 (1.6 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:6 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
Expected: nsm interface is created
I wrote a simple script for run the Docker tests repeatedly and discovered test failing after several iterations (around 3-5 more often). In rather cases tests fail on the first run.
The case was reproduced on two different platforms, includes Ubuntu 20.04.
I collected logs from a few failed runs and attached it to the issue.
Script:
#!/bin/bash
DOCKER_IMAGE=$(docker build -q --target test .)
LOG_FILE='/tmp/test.log'
i=1
while true; do
echo -n "Test ${i}..."
docker run --privileged --rm ${DOCKER_IMAGE} > "${LOG_FILE}"
result=$?
if [ $result -ne 0 ]; then
echo 'FAIL'
exit $result
fi
echo 'DONE'
i=$((i+1))
doneRepeat steps from https://drive.google.com/file/d/1Pwd7T6iYPItXEqjx_Yd5NfPE-T0zgUy5
Packet automation cluster.
VPP forwarder restarts with panic:
Apr 6 04:08:40.826 [INFO] [name:forwarder-vpp-bpjs6] (32.1) request-diff={"mechanism_preferences":{"2":{"parameters":{"+src_ip":"192.168.0.4"}}}}
Apr 6 04:08:40.826 [INFO] [name:forwarder-vpp-bpjs6] (33) ⎆ sdk/pkg/networkservice/common/mechanisms/recvfd/recvFDClient.Request()
Apr 6 04:08:40.826 [INFO] [name:forwarder-vpp-bpjs6] (34) ⎆ sdk/pkg/networkservice/common/mechanisms/sendfd/sendFDClient.Request()
Apr 6 04:08:40.826 [INFO] [name:forwarder-vpp-bpjs6] (35) ⎆ sdk/pkg/networkservice/common/null/nullClient.Request()
Apr 6 04:08:40.827 [INFO] [name:forwarder-vpp-bpjs6] (36) ⎆ api/pkg/api/networkservice/networkServiceClient.Request()
Apr 6 04:08:40.897 [ERRO] [name:forwarder-vpp-bpjs6] (36.1) rpc error: code = Canceled desc = context canceled
Error returned from api/pkg/api/networkservice/networkServiceClient.Request
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:62
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:99
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/common/null.(*nullClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/null/client.go:43
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:58
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:99
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/common/mechanisms/sendfd.(*sendFDClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/mechanisms/sendfd/client.go:54
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:58
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:99
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/common/mechanisms/recvfd.(*recvFDClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/mechanisms/recvfd/client.go:55
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:58
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:99
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:99
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk-vpp/pkg/networkservice/mechanisms/vxlan/vni.(*vniClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/mechanisms/vxlan/vni/client.go:48
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:58
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:99
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk-vpp/pkg/networkservice/mechanisms/vxlan.(*vxlanClient).Request
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/mechanisms/vxlan/client.go:65
Apr 6 04:08:40.897 [INFO] [name:forwarder-vpp-bpjs6] (35.1) response=null
Apr 6 04:08:40.901 [ERRO] [name:forwarder-vpp-bpjs6] (35.2) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Canceled desc = context canceled
Apr 6 04:08:40.904 [ERRO] [name:forwarder-vpp-bpjs6] (34.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Canceled desc = context canceled
Apr 6 04:08:40.907 [ERRO] [name:forwarder-vpp-bpjs6] (33.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Canceled desc = context canceled
Apr 6 04:08:40.910 [ERRO] [name:forwarder-vpp-bpjs6] (32.2) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Canceled desc = context canceled
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xad36b0]
goroutine 177 [running]:
os.(*File).Name(...)
/go/src/os/file.go:54
github.com/networkservicemesh/sdk/pkg/networkservice/common/mechanisms/recvfd.recvFDAndSwapInodeToFile.func1()
/root/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/mechanisms/recvfd/common.go:76 +0xf0
github.com/edwarnicke/serialize.(*Executor).process(0xc000511a00, 0xc000722980)
/root/go/pkg/mod/github.com/edwarnicke/[email protected]/serialize.go:68 +0x3f
created by github.com/edwarnicke/serialize.(*Executor).AsyncExec
/root/go/pkg/mod/github.com/edwarnicke/[email protected]/serialize.go:56 +0x1f8
No restart, no panic.
Actual: TestKernel2Vxlan2Memif and TestMemif2Vxlan2Kernel delete NSC and NSE too long
On deleting NAMESPACE we are doing this:
https://github.com/networkservicemesh/cmd-nsc-vpp/blob/master/main.go#L208-L211
It looks like RPC Close in these tests work incorrectly.
We should investigate the problem and fix it.
A device selector file might contain devices that do not exist on the host when the forwarder is starting.
Devices are stored in a map, thus it's non-deterministic when InitLinks() will try to process the missing interface, but when it does it will return without considering the remaining devices. (The outcome might be different among the running vpp forwarders.)
This might result in issues when for example trying to use Remote VLAN NSE with an interface that got ignored, hence the forwarder/vpp will not know about it.
IMHO an error during InitLinks should either be considered fatal, or the code should filter out devices that does not exist on the host without skipping others.
So, basically the question is if a device selector file with unknown interfaces is considered an error or not.
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: forwarder-vpp
labels:
app: forwarder-vpp
spec:
selector:
matchLabels:
app: forwarder-vpp
template:
metadata:
labels:
app: forwarder-vpp
spec:
containers:
- image: networkservicemesh/cmd-vpp-forwarder:latest
imagePullPolicy: IfNotPresent
name: forwarder-vpp
securityContext:
privileged: true
env:
- name: SPIFFE_ENDPOINT_SOCKET
value: unix:///run/spire/sockets/agent.sock
- name: NSM_TUNNEL_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: NSM_CONNECT_TO
value: unix:///var/lib/networkservicemesh/nsm.io.sock
- name: NSM_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
volumeMounts:
- name: spire-agent-socket
mountPath: /run/spire/sockets
readOnly: true
- name: nsm-socket
mountPath: /var/lib/networkservicemesh
volumes:
- name: spire-agent-socket
hostPath:
path: /run/spire/sockets
type: Directory
- name: nsm-socket
hostPath:
path: /var/lib/networkservicemesh
type: DirectoryOrCreate
Use deployment that described above
Follow these steps: https://github.com/networkservicemesh/deployments-k8s/tree/main/examples/LocalConnection
Hi,
I'm not sure if this is the best location for the issue as it is relevant to any NSE using connect.NewServer(..) chain, not only the VPP forwarder.
Let's say we have the following topology:
NSC (Node A) <--> Forwarder X (Node A) <--> Forwarder Y (Node B) <--> NSE (Node B)
The NSC initiates a request to the NSE which is sent to Forwarder X.
A timeout is set for the request's response - let's call it NSC-timeout.
Forwarder X receives the request and then initiates new request to Forwarder Y.
A timeout is set for the request's response - let's call it Forwarder-X-timeout.
Forwarder Y receives the request and then initiates new request to NSE.
A timeout is set for the request's response - let's call it Forwarder-Y-timeout.
Before NSE returns a successful response to Forwarder Y, the timeout expires on a previous chain,
for example, Forwarder-X-timeout is expired.
Forwarder-X had an error - propagating it to the NSC
Forwarder-Y received successful response from the NSE and is unaware that Forwarder-X-timeout has expired.
Now we have a scenario where we need to tell Forwarder-Y to release all relevant resources because the request flow failed.
From the Forwarder-X's perspective the call to the "connect.NewServer" chain failed, so
it won't initiate a close flow.
What is the best practice in such scenario? should NSE's using the connect.NewServer chain always initiate a close flow if a call to the next chain failed?
If i'm missing something please let me know.
Thanks!
At first the issue popped up during an upgrade test in which we try to upgrade NSM from a pre v1.1.1 version to NSM v1.1.1
It is an intermittent issue (timing is key), but can be reporoduced with simple pod restarts by "kubectl delete pod XY --force".
NSM elements gets restarted in the following order:
I was able to reproduce the issue with this timing:
forwarder restart -> sleep 2.5 -> registry restart -> sleep 2 -> NSMgr restart
By issueing different timing for the restarts, heal function looks to be working great, keeping everything up and running, waiting for connections, etc..
I attached the logs of the forwarder. Unfortunately logs from the "old" registry and NSMgr are not available due to them being killed.
The timing to reporoduce the issue might differ for you according to the available resources for the k8s cluster, I tried it on kind.
forwarder.log
In case of some deployments, the default vxlan port 4789 (currently used by NSM) may already be used by the underlying infrastructure, thus making the routers/switches to drop packets sent by NSM over the very same port due to security policies.
After a short test of building vxlan tunnels between k8s workers, we found out that if the vxlan tunnel is built on a different port (i.e. 4790), then traffic could flow without any issues.
How vxlan tunnels were built for testing:
worker1:
ip link add vxlan1 type vxlan id 1 remote 10.228.218.11 dstport 4790 dev eth1
ip link set vxlan1 up
ip addr add 176.16.1.99/24 dev vxlan1
worker2:
ip link add vxlan1 type vxlan id 1 remote 10.228.218.8 dstport 4790 dev eth1
ip link set vxlan1 up
ip addr add 176.16.1.98/24 dev vxlan1
For cases like this, It would be good if the vxlan port could be configurable in NSM. I think the easiest (from user perspective) would be if it could be set in the forwarder ENV just like NSM_LISTEN_ON and NSM_CONNECT_TO sockets, etc...
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.