everoute / everoute

Everoute provide cloud-native networking and security solution

everoute

Overview

Everoute is a cloud-native networking and security solution. It works with OSI Network Model's Layer3/4. It leverages Open vSwitch as the networking data plane.

Everoute provides network and security services for the following platforms:

• Kubernetes cloud-native platform
• Private Cloud
• Public Cloud
• Hybrid Cloud
• Legacy virtualized data center

It can be integrated within these platforms easily.

Architecture

Everoute takes the SDN (Software Defined Network) methodology as foundation. It decouples the control plane and data plane, and utilizes software programming to control the software network and security services.

All the Everoute related services are deployed as Kubernetes Pod, which makes these services easy to be deployed and managed.

Everoute contains four main parts:

• Everoute Central Controller: The Everoute central controller consists of control service, API Server and etcd, it leverages the cloud-native architecture, so all the services are deployed as containers. The Everoute Central Controller provides the controller cluster management and data persistence based on the Kube API Server and etcd. Everoute supports controllers' high availability and scalability through cluster. The main task of control service is to manage network and security policies. It generates network transimit policy rules according to resource types, security policies and discovered IP addresses.

• Everoute Distributed Agent: The Everoute Agent is deployed in each K8s worker node or hypervisor host, it focuses on the local node policy management. The Everoute Agent is mainly responsible for discovering endpoint IP address and watching policy rules from Central controller.

• Everoute Datapath: Everoute utilizes the Open vSwitch as it's network data plane, Everoute leverages it to implement virtual network forwarding, Pod networking and security features. Everoute uses Open vSwitch openflow mode to control the network forwarding and security rules.

• 3rd party plugins: Everoute provides a plugin framework to integrate within 3rd party platforms, such as SmartX - SMTX OS virtualization platform or other cloud platform.

Main functions

In the current phase, Everoute support native Kubernetes platform and SmartX virtualization platform SMTX OS.

• Kubernetes Platform: For the Kubernetes platform, Everoute provides the native K8s CNI network plug-in. The Everoute CNI supports Pod connection management, Network Policies, cluster service and NodePort etc. Details please refer to Everoute CNI

• Virtualization Platform: SMTX OS is SmartX native virtualization platform. Everoute can be integrated with SMTX OS through the CloudTower plugin to provide the Micro-Segmentation service.

Roadmap

The following features are considered for the near future:

• Network Visibility: to support the network visibility, service map, traffic monitor etc.
• Overlay support: to support the VXLAN tunnel.
• L3 routing: distributed virtual routing.
• Kubernetes networking enhancement: endPort, ingress LoadBalancer, cluster service enhancement etc.
• Some function enhancement and performance improvement of the control plane and data plane.
• Service Function Chain: to support integrated with 3rd party services such as AV, IPS, IDS, traffic monitor etc.

License

Everoute is licensed under the Apache License, version 2.0

everoute's Issues

Change label value in policyRule

label value has 63 length limit

Support global rule

e2e: can't found valid ip addr

https://github.com/smartxworks/lynx/pull/104/checks?check_run_id=2715120434#step:6:421

SecurityPolicy should allow multiple port range separated by commas

e.g.

apiVersion: security.lynx.smartx.com/v1alpha1
kind: SecurityPolicy
metadata:
 name: security-policy
spec:
...
 egressRules:
   - name: egress01
     ports:
       - portRange: 10-22,80,8080-8082
...

UT for TestAgentMonitorRestart failed

logs_686.zip

kube-controller-manager will log "list error" when crd is removed

After everoute was removed from k8s, it will occur lots of errors in kube-controller-manager
[reflector.go:138] k8s.io/client-go/metadata/metadatainformer/informer.go:90: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource

Details in kubernetes/kubernetes#79610

unit test: failed to delete ovs port

https://github.com/smartxworks/lynx/runs/2533102802?check_suite_focus=true
https://github.com/smartxworks/lynx/runs/2532937684?check_suite_focus=true

E2e on tower: can't clean endpoints

[91mUnexpected error:
      <*errors.errorString | 0xc0004b01f0>: {
          s: "clean endpoints: mutation from tower, reply: &{{\"deleteVm\":null} [message: Cannot unwrap Ok value of Result.Err.\r\ncode: ORM_ERROR\r\nmessage: You provided an invalid argument for the where selector on Cluster. Please provide exactly one unique field and value., errcode: ORM_ERROR]}, err: %!s(<nil>)",
      }
      clean endpoints: mutation from tower, reply: &{{"deleteVm":null} [message: Cannot unwrap Ok value of Result.Err.
      code: ORM_ERROR
      message: You provided an invalid argument for the where selector on Cluster. Please provide exactly one unique field and value., errcode: ORM_ERROR]}, err: %!s(<nil>)
  occurred[0m

Need migration codecov from v1 to v2

On February 1, 2022, this version will be fully sunset and no longer function

https://github.com/marketplace/actions/codecov#%EF%B8%8F--deprecration-of-v1

E2e should update to adapt new dataplane

wrong expression to judge empty SecurityPeer

https://github.com/smartxworks/lynx/blob/d25b1a59f933401fa6dcc993637c2cf2a042fd14/pkg/controller/policy/policy_controller.go#L432
https://github.com/smartxworks/lynx/blob/d25b1a59f933401fa6dcc993637c2cf2a042fd14/pkg/controller/policy/policy_controller.go#L466

should be len(rule.From.IPBlocks)+len(rule.From.EndpointGroups) + len(rule.From.Endpoints) == 0

Remove unused CRD Tier

We define CRD security.everoute.io/v1alph1 tier, but now it's not a configurable resource.
We have three constants tiers: tier0, tier1, and tier2.

For now, we must create default tiers first, and check whether the tier in SecurityPolicy.Spec.Tier exists in the webhook.
So remove CRD Tier may be a good choice for use.

cc @kylezh @KeyongSun @wu-cl @echkenluo

agentMonitor monitor ovsdb interface table and init local endpoint flow

GQL client data race

==================
WARNING: DATA RACE
Write at 0x00c000438c70 by goroutine 108:
  runtime.closechan()
      /opt/hostedtoolcache/go/1.15.15/x64/src/runtime/chan.go:357 +0x0
  github.com/smartxworks/lynx/plugin/tower/pkg/client.lookReadMessage()
      /home/runner/work/lynx/lynx/plugin/tower/pkg/client/client.go:284 +0x10e

Previous read at 0x00c000438c70 by goroutine 110:
  runtime.chansend()
      /opt/hostedtoolcache/go/1.15.15/x64/src/runtime/chan.go:158 +0x0
  github.com/smartxworks/lynx/plugin/tower/pkg/client.lookReadMessage.func1()
      /home/runner/work/lynx/lynx/plugin/tower/pkg/client/client.go:273 +0xf4

Goroutine 108 (running) created at:
  github.com/smartxworks/lynx/plugin/tower/pkg/client.(*Client).Subscription()
      /home/runner/work/lynx/lynx/plugin/tower/pkg/client/client.go:82 +0x4c6
  github.com/smartxworks/lynx/plugin/tower/pkg/informer.(*reflector).listAndWatch()
      /home/runner/work/lynx/lynx/plugin/tower/pkg/informer/reflector.go:99 +0xb5
  github.com/smartxworks/lynx/plugin/tower/pkg/informer.(*reflector).reflectWorker.func1()
      /home/runner/work/lynx/lynx/plugin/tower/pkg/informer/reflector.go:93 +0x53
  k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1()
      /root/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x75
  k8s.io/apimachinery/pkg/util/wait.BackoffUntil()
      /root/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xba
  github.com/smartxworks/lynx/plugin/tower/pkg/informer.(*reflector).Run()
      /home/runner/work/lynx/lynx/plugin/tower/pkg/informer/reflector.go:83 +0x312
  github.com/smartxworks/lynx/plugin/tower/third_party/forked/client-go/informer.Reflector.Run-fm()
      /home/runner/work/lynx/lynx/plugin/tower/third_party/forked/client-go/informer/reflector.go:30 +0x50
  k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
      /root/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:56 +0x45
  k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
      /root/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73 +0x6d

Goroutine 110 (running) created at:
  github.com/smartxworks/lynx/plugin/tower/pkg/client.lookReadMessage()
      /home/runner/work/lynx/lynx/plugin/tower/pkg/client/client.go:265 +0xf3

datapath flowtable should consistent while failover or restart

E2E: should wait for lynx-controller ready before create resources

github action failed: https://github.com/smartxworks/lynx/runs/3520691521#step:5:146

Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "validator.lynx.smartx.com": Post https://127.0.0.1:9443/validate/crds?timeout=30s: dial tcp 127.0.0.1:9443: connect: connection refused
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "validator.lynx.smartx.com": Post https://127.0.0.1:9443/validate/crds?timeout=30s: dial tcp 127.0.0.1:9443: connect: connection refused
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "validator.lynx.smartx.com": Post https://127.0.0.1:9443/validate/crds?timeout=30s: dial tcp 127.0.0.1:9443: connect: connection refused

Before creating resources, we should wait for the kube-apiserver and lynx-controller ready.

Failed to add endpoint flow with vlan tag

Add local endpoint will trigger add local endpoint flow logic, it never returns any errors. But we found that the OpenFlow library will return an error (OpenFlow error message ). Endpoint flow failed to add to datapath.

Add unit tests for plugin-tower

We need to add unit tests for plugin-tower.

test agent monitor restart failed

https://github.com/echkenluo/lynx/runs/2690664505?check_suite_focus=true

Should generate api document

CNI Should setup OVS in container

Support Kubernetes NetworkPolicy

To support Kubernetes NetworkPolicy, we should do this adjustment on API:

• Resource Endpoint should be namespaced.
• Support namespace and namespace selector on EndpointGroup.

And more others ...

endpoint should include vlan id

see also #31

update IP for endpoint in CNI ADD

In some cases, pod will not send pkg after creation, so that ovs could not learn ip address, and policy rule will not be generated.

We should monitor uplink status

For endpoint Ip address learning, we should add some arp Redirect flow according to the current uplink port. Simultaneously，we should monitor uplink status mutation and update related flow immediately.

go.mod should update due to repository gobgp transferred

Asymmetric securityPolicy definition may cause unexpected connectivity among the group

https://docs.google.com/document/d/15w9_ITuOcXZv-Z4YGVnGqqXOxidm64ezfOMLsad7FR8 #1

[docs] Add comment for export API Types

flow table entry reduction plan

If user sets a port range like 1-65535, it will generate 65535 entries.
Here is an approach below
https://blog.csdn.net/ambzheng/article/details/104829359
It uses port mask to reduce table entires.

Automatically generate webhook certificate

agent learned unexpected IP and panic

see log:

time="2021-04-07T05:43:20Z" level=info msg="Flush Learned localOfPort endpointInfo 6 : 10.0.0.4." 
time="2021-04-07T05:43:20Z" level=info msg="Deleted flow: {\"Priority\":100,\"InputPort\":0,\"MacDa\":\"lhKZi2aa\",\"MacDaMask\":null,\"MacSa\":null,\"MacSaMask\":null,\"Ethertype\":2048,\"VlanId\":0,\"ArpOper\":0,\"IpSa\":null,\"IpSaMask\":null,\"IpDa\":null,\"IpDaMask\":null,\"Ipv6Sa\":null,\"Ipv6SaMask\":null,\"Ipv6Da\":null,\"Ipv6DaMask\":null,\"IpProto\":0,\"IpDscp\":0,\"TcpSrcPort\":0,\"TcpDstPort\":0,\"UdpSrcPort\":0,\"UdpDstPort\":0,\"Metadata\":null,\"MetadataMask\":null,\"TunnelId\":0,\"TcpFlags\":null,\"TcpFlagsMask\":null,\"CtStates\":null}" 
time="2021-04-07T05:43:20Z" level=info msg="local ofport 2 related endpoint ipaddress update from 10.0.0.126 to 10.0.0.88" 
time="2021-04-07T05:43:20Z" level=info msg="Learning localOfPort endpointInfo 2 : 10.0.0.88." 
time="2021-04-07T05:43:20Z" level=info msg="local ofport 10 related ovsport was't learned or ofPort update" 
time="2021-04-07T05:43:20Z" level=info msg="Flush Learned localOfPort endpointInfo 3 : 10.0.0.212." 
time="2021-04-07T05:43:20Z" level=info msg="Deleted flow: {\"Priority\":100,\"InputPort\":0,\"MacDa\":\"UlQAYtvT\",\"MacDaMask\":null,\"MacSa\":null,\"MacSaMask\":null,\"Ethertype\":2048,\"VlanId\":0,\"ArpOper\":0,\"IpSa\":null,\"IpSaMask\":null,\"IpDa\":null,\"IpDaMask\":null,\"Ipv6Sa\":null,\"Ipv6SaMask\":null,\"Ipv6Da\":null,\"Ipv6DaMask\":null,\"IpProto\":0,\"IpDscp\":0,\"TcpSrcPort\":0,\"TcpDstPort\":0,\"UdpSrcPort\":0,\"UdpDstPort\":0,\"Metadata\":null,\"MetadataMask\":null,\"TunnelId\":0,\"TcpFlags\":null,\"TcpFlagsMask\":null,\"CtStates\":null}" 
time="2021-04-07T05:43:20Z" level=info msg="local ofport 10 related ovsport was't learned or ofPort update" 
time="2021-04-07T05:43:20Z" level=info msg="local ofport 10 related ovsport was't learned or ofPort update" 
time="2021-04-07T05:43:20Z" level=info msg="local ofport 10 related ovsport was't learned or ofPort update" 
time="2021-04-07T05:43:20Z" level=info msg="local ofport 10 related ovsport was't learned or ofPort update" 
time="2021-04-07T05:43:20Z" level=info msg="local ofport 10 related ovsport was't learned or ofPort update" 
time="2021-04-07T05:43:20Z" level=info msg="local ofport 10 related ovsport was't learned or ofPort update" 
time="2021-04-07T05:43:20Z" level=info msg="local ofport 10 related ovsport was't learned or ofPort update" 
time="2021-04-07T05:43:20Z" level=info msg="local ofport 1 related endpoint ipaddress update from 10.0.0.168 to 192.168.29.62" 
time="2021-04-07T05:43:20Z" level=info msg="Learning localOfPort endpointInfo 1 : 192.168.29.62." 
time="2021-04-07T05:43:20Z" level=info msg="local ofport 10 related ovsport was't learned or ofPort update" 
time="2021-04-07T05:43:20Z" level=info msg="local ofport 10 related ovsport was't learned or ofPort update" 
time="2021-04-07T05:43:20Z" level=info msg="local ofport 10 related ovsport was't learned or ofPort update" 
time="2021-04-07T05:43:20Z" level=info msg="local ofport 10 related ovsport was't learned or ofPort update" 
time="2021-04-07T05:43:20Z" level=info msg="local ofport 10 related ovsport was't learned or ofPort update" 
time="2021-04-07T05:43:20Z" level=info msg="local ofport 10 related ovsport was't learned or ofPort update" 
time="2021-04-07T05:43:20Z" level=info msg="local ofport 10 related ovsport was't learned or ofPort update" 
time="2021-04-07T05:43:20Z" level=info msg="local ofport 10 related ovsport was't learned or ofPort update" 
time="2021-04-07T05:43:20Z" level=info msg="Flush Learned localOfPort endpointInfo 4 : 10.0.0.19." 
time="2021-04-07T05:43:20Z" level=info msg="Deleted flow: {\"Priority\":100,\"InputPort\":0,\"MacDa\":\"mjPu89Da\",\"MacDaMask\":null,\"MacSa\":null,\"MacSaMask\":null,\"Ethertype\":2048,\"VlanId\":0,\"ArpOper\":0,\"IpSa\":null,\"IpSaMask\":null,\"IpDa\":null,\"IpDaMask\":null,\"Ipv6Sa\":null,\"Ipv6SaMask\":null,\"Ipv6Da\":null,\"Ipv6DaMask\":null,\"IpProto\":0,\"IpDscp\":0,\"TcpSrcPort\":0,\"TcpDstPort\":0,\"UdpSrcPort\":0,\"UdpDstPort\":0,\"Metadata\":null,\"MetadataMask\":null,\"TunnelId\":0,\"TcpFlags\":null,\"TcpFlagsMask\":null,\"CtStates\":null}" 
time="2021-04-07T05:43:20Z" level=info msg="local ofport 10 related ovsport was't learned or ofPort update" 
time="2021-04-07T05:43:20Z" level=info msg="Flush Learned localOfPort endpointInfo 5 : 10.0.0.72." 
time="2021-04-07T05:43:20Z" level=info msg="Deleted flow: {\"Priority\":100,\"InputPort\":0,\"MacDa\":\"Bl1yhYCg\",\"MacDaMask\":null,\"MacSa\":null,\"MacSaMask\":null,\"Ethertype\":2048,\"VlanId\":0,\"ArpOper\":0,\"IpSa\":null,\"IpSaMask\":null,\"IpDa\":null,\"IpDaMask\":null,\"Ipv6Sa\":null,\"Ipv6SaMask\":null,\"Ipv6Da\":null,\"Ipv6DaMask\":null,\"IpProto\":0,\"IpDscp\":0,\"TcpSrcPort\":0,\"TcpDstPort\":0,\"UdpSrcPort\":0,\"UdpDstPort\":0,\"Metadata\":null,\"MetadataMask\":null,\"TunnelId\":0,\"TcpFlags\":null,\"TcpFlagsMask\":null,\"CtStates\":null}" 
time="2021-04-07T05:43:20Z" level=info msg="local ofport 10 related ovsport was't learned or ofPort update" 
time="2021-04-07T05:43:20Z" level=info msg="Flush Learned localOfPort endpointInfo 2 : 10.0.0.88." 
time="2021-04-07T05:43:20Z" level=info msg="Deleted flow: {\"Priority\":100,\"InputPort\":0,\"MacDa\":\"phdoe3wB\",\"MacDaMask\":null,\"MacSa\":null,\"MacSaMask\":null,\"Ethertype\":2048,\"VlanId\":0,\"ArpOper\":0,\"IpSa\":null,\"IpSaMask\":null,\"IpDa\":null,\"IpDaMask\":null,\"Ipv6Sa\":null,\"Ipv6SaMask\":null,\"Ipv6Da\":null,\"Ipv6DaMask\":null,\"IpProto\":0,\"IpDscp\":0,\"TcpSrcPort\":0,\"TcpDstPort\":0,\"UdpSrcPort\":0,\"UdpDstPort\":0,\"Metadata\":null,\"MetadataMask\":null,\"TunnelId\":0,\"TcpFlags\":null,\"TcpFlagsMask\":null,\"CtStates\":null}" 
time="2021-04-07T05:43:20Z" level=info msg="local ofport 10 related ovsport was't learned or ofPort update" 
time="2021-04-07T05:43:20Z" level=info msg="local ofport 10 related ovsport was't learned or ofPort update" 
time="2021-04-07T05:43:20Z" level=info msg="local ofport 10 related ovsport was't learned or ofPort update" 
time="2021-04-07T05:43:20Z" level=info msg="local ofport 1 related endpoint ipaddress update from 192.168.29.62 to 192.168.24.66" 
time="2021-04-07T05:43:20Z" level=info msg="Learning localOfPort endpointInfo 1 : 192.168.24.66." 
time="2021-04-07T05:43:20Z" level=info msg="local ofport 10 related ovsport was't learned or ofPort update" 
time="2021-04-07T05:43:20Z" level=info msg="local ofport 10 related ovsport was't learned or ofPort update" 
time="2021-04-07T05:43:20Z" level=info msg="local ofport 10 related ovsport was't learned or ofPort update" 
time="2021-04-07T05:43:20Z" level=info msg="local ofport 1 related endpoint ipaddress update from 192.168.24.66 to 10.0.0.168" 
time="2021-04-07T05:43:20Z" level=info msg="Learning localOfPort endpointInfo 1 : 10.0.0.168." 
time="2021-04-07T05:43:20Z" level=error msg="Flow {\"Priority\":100,\"InputPort\":0,\"MacDa\":\"mprLiPAf\",\"MacDaMask\":null,\"MacSa\":null,\"MacSaMask\":null,\"Ethertype\":2048,\"VlanId\":0,\"ArpOper\":0,\"IpSa\":null,\"IpSaMask\":null,\"IpDa\":null,\"IpDaMask\":null,\"Ipv6Sa\":null,\"Ipv6SaMask\":null,\"Ipv6Da\":null,\"Ipv6DaMask\":null,\"IpProto\":0,\"IpDscp\":0,\"TcpSrcPort\":0,\"TcpDstPort\":0,\"UdpSrcPort\":0,\"UdpDstPort\":0,\"Metadata\":null,\"MetadataMask\":null,\"TunnelId\":0,\"TcpFlags\":null,\"TcpFlagsMask\":null,\"CtStates\":null} already exists" 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5c041b]

goroutine 211 [running]:
github.com/contiv/ofnet/ofctrl.(*Flow).Next(0x0, 0x1cab638, 0xc0006b6820, 0x0, 0x0)
	/opt/golang/pkg/mod/github.com/smartxworks/[email protected]/ofctrl/fgraphFlow.go:551 +0x3b
github.com/contiv/ofnet.(*VlanArpLearnerBridge).installIngressSelectFlow(0xc0000ffb00, 0xc00066b040, 0x6, 0x7c0, 0x2, 0xc00066b046)
	/opt/golang/pkg/mod/github.com/smartxworks/[email protected]/vlanArpLearnBridge.go:426 +0x1ca
github.com/contiv/ofnet.(*VlanArpLearnerBridge).learnFromArp(0xc0000ffb00, 0x1040608000001, 0xc00066b040, 0x6, 0x7c0, 0xc00066b046, 0x4, 0x7ba, 0xc00066b04a, 0x6, ...)
	/opt/golang/pkg/mod/github.com/smartxworks/[email protected]/vlanArpLearnBridge.go:313 +0xefc
github.com/contiv/ofnet.(*VlanArpLearnerBridge).processArp(0xc0000ffb00, 0x0, 0xc0007f8494, 0x6, 0x6, 0xc0007f849a, 0x6, 0x6, 0x806000000000000, 0x1cca240, ...)
	/opt/golang/pkg/mod/github.com/smartxworks/[email protected]/vlanArpLearnBridge.go:297 +0x225
github.com/contiv/ofnet.(*VlanArpLearnerBridge).PacketRcvd(0xc0000ffb00, 0xc0005bea20, 0xc0007fb4a0)
	/opt/golang/pkg/mod/github.com/smartxworks/[email protected]/vlanArpLearnBridge.go:263 +0x105
github.com/contiv/ofnet.(*OfnetAgent).PacketRcvd(0xc0002c6fc0, 0xc0005bea20, 0xc0007fb4a0)
	/opt/golang/pkg/mod/github.com/smartxworks/[email protected]/ofnetAgent.go:466 +0xe7
github.com/contiv/ofnet/ofctrl.(*OFSwitch).handleMessages(0xc0005bea20, 0xc0006457b0, 0x8, 0x8, 0x1cc9df0, 0xc0007fb4a0)
	/opt/golang/pkg/mod/github.com/smartxworks/[email protected]/ofctrl/ofSwitch.go:203 +0x633
github.com/contiv/ofnet/ofctrl.(*OFSwitch).receive(0xc0005bea20)
	/opt/golang/pkg/mod/github.com/smartxworks/[email protected]/ofctrl/ofSwitch.go:135 +0x71
created by github.com/contiv/ofnet/ofctrl.NewSwitch
	/opt/golang/pkg/mod/github.com/smartxworks/[email protected]/ofctrl/ofSwitch.go:69 +0x3b6

Please note that：

"local ofport 1 related endpoint ipaddress update from 10.0.0.168 to 192.168.29.62" 
"Learning localOfPort endpointInfo 1 : 192.168.29.62." 
"local ofport 1 related endpoint ipaddress update from 192.168.29.62 to 192.168.24.66"
"Learning localOfPort endpointInfo 1 : 192.168.24.66." 
"local ofport 1 related endpoint ipaddress update from 192.168.24.66 to 10.0.0.168" 
"Learning localOfPort endpointInfo 1 : 10.0.0.168." 

Unexpected IP 192.168.29.62 and 192.168.29.66 found.

agent monitor unexpected stop sync agentinfo

agent monitor unexpected stop sync agentinfo

endpoint controller too slow !!!

When there have more than 2000 endpoints, it would take 10 seconds or more to sync the endpoint's IP from AgentInfo.

upgrade k8s api to v.1.27

end point in networkPolicy was bring up in api v1.21+.
controller runtime & client-go also need to upgrade.

uplink configuration init

1. lynx-agent should initialize datapath ofSwitch and table first，we init vlanarplearner datapath which sleep for 2 seconds before add uplink, but 2 seconds maybe not enough. if datapath init is not complete, add uplink maybe failed.
2. while lynx-agent complete datapath setup, but uplink was't added, datapath would learn all of ofport as local port. we should defer datapath ip learning procedure until uplink was configured

Intragroup traffic should always be allowed

should allow security policy with rule match all ports and protocols

If ports length 0 should matches all ports.
https://github.com/smartxworks/lynx/blob/f09fce94241a9788ba5d7ab131fe1f6d0f817410/pkg/apis/security/v1alpha1/types.go#L84-L86

Failed to register ovsdbEventHandler

When start agent, get error: Failed to register ovsdbEventHandler.

https://github.com/smartxworks/lynx/blob/0c81406531f51e73ae477f70b03f13333b3ea959/pkg/monitor/agent.go#L74-L78

PolicyRule.Name should include vid and tenant id

Otherwise, policy rules from different vlan and tenant would be conflicted.

Unittest for PR https://github.com/smartxworks/lynx/pull/149

Add a task for track the unittest PR #149

Plugin tower should ignore invalid labels

Support ARM

different policyrule with same flow key

stateless policy enforcement model with bidirectional flow processing

[test] Receive ginkgo warning message after unit-test

You're using deprecated Ginkgo functionality:

Ginkgo 2.0 is under active development and will introduce (a small number of) breaking changes.
To learn more, view the migration guide at https://github.com/onsi/ginkgo/blob/v2/docs/MIGRATING_TO_V2.md
To comment, chime in at onsi/ginkgo#711

You are using a custom reporter. Support for custom reporters will likely be removed in V2. Most users were using them to generate junit or teamcity reports and this functionality will be merged into the core reporter. In addition, Ginkgo 2.0 will support emitting a
JSON-formatted report that users can then manipulate to generate custom reports.

If this change will be impactful to you please leave a comment on onsi/ginkgo#711
Learn more at: https://github.com/onsi/ginkgo/blob/v2/docs/MIGRATING_TO_V2.md#removed-custom-reporters

To silence deprecations that can be silenced set the following environment variable:
ACK_GINKGO_DEPRECATIONS=1.16.4

k8s CNI E2E Test

CNI ipam will not release deleted IP resource

pod creation will block after ip pool is used up.

E2E: Isolated endpoint failed

Github action https://github.com/smartxworks/lynx/runs/3522735971#step:6 failed.

Flows https://github.com/smartxworks/lynx/runs/3522735971#step:6:393 :

table=0, priority=103,arp,in_port=10 actions=goto_table:46 
table=0, priority=100,ip actions=goto_table:5 
table=0, priority=102,arp actions=CONTROLLER:65535 
table=0, priority=1 actions=goto_table:46 
table=5, priority=100,ip actions=ct(table=6,zone=65535) 
table=6, priority=102,ct_state=-new+trk actions=goto_table:7 
table=6, priority=101,ct_state=+inv+trk actions=drop 
table=6, priority=1 actions=goto_table:7 
table=7, priority=102,ct_state=-new+est,ip actions=goto_table:45 
table=7, priority=101,ip,in_port=10 actions=goto_table:30 
table=7, priority=101,ipv6,in_port=10 actions=goto_table:30 
table=7, priority=1 actions=goto_table:10 
table=10, priority=10,ip,nw_src=10.0.0.156 actions=drop 
table=10, priority=1 actions=goto_table:11 
table=11, priority=1 actions=goto_table:12 
table=12, priority=1 actions=goto_table:20 
table=20, priority=102,ct_state=-new+est actions=goto_table:45 
table=20, priority=100,ip,dl_dst=42:4d:1d:b6:84:29 actions=goto_table:30 
table=20, priority=100,ip,dl_dst=9e:cd:26:11:ab:07 actions=goto_table:30 
table=20, priority=100,ip,dl_dst=36:65:d3:5a:0c:b1 actions=goto_table:30 
table=20, priority=100,ip,dl_dst=8a:6f:69:92:40:f9 actions=goto_table:30 
table=20, priority=1 actions=goto_table:45 
table=30, priority=110,tcp,nw_src=10.0.0.208,nw_dst=10.0.0.156,tp_dst=443 actions=goto_table:45 
table=30, priority=10,ip,nw_dst=10.0.0.156 actions=drop
table=30, priority=1 actions=goto_table:31 
table=31, priority=1 actions=goto_table:32 
table=32, priority=1 actions=goto_table:45 
table=45, priority=100,ct_state=+new+trk,ip actions=ct(commit,table=46,zone=65535) 
table=45, priority=1 actions=goto_table:46 
table=46, priority=1 actions=NORMAL

Endpoint IPs:

ep01 10.0.0.149
ep02 10.0.0.245
ep03 10.0.0.105
ep04 10.0.0.100

Connect TruthTable:

   -	ep01	ep02	ep03	ep04
ep01	.	x	x	.
ep02	.	.	.	.
ep03	.	.	.	.
ep04	.	.	.	.

failed to remove policyrules when securitypolicy deleted

https://github.com/smartxworks/lynx/runs/2139765974?check_suite_focus=true#step:8:371

Stateful policy enforce model corner case