deepflowio / deepflow

:rocket: eBPF-powered observability & zero-code distributed tracing :sparkles:

Instant Observability for Cloud-Native Applications

Zero Code, Full Stack, eBPF & Wasm

---

English | 简体中文

What is DeepFlow

The DeepFlow open-source project aims to provide deep observability for complex cloud infrastructures and cloud-native applications. DeepFlow implemented Zero Code data collection with eBPF for metrics, distributed tracing, request logs and function profiling, and is further integrated with SmartEncoding to achieve Full Stack correlation and efficient access to all observability data. With DeepFlow, cloud-native applications automatically gain deep observability, removing the heavy burden of developers continually instrumenting code and providing monitoring and diagnostic capabilities covering everything from code to infrastructure for DevOps/SRE teams.

Key Features

• Universal Map for Any Service: DeepFlow provides a universal map with Zero Code by eBPF for production environments, including your services in any language, third-party services without code and all cloud-native infrastructure services. In addition to analyzing common protocols, Wasm plugins are supported for your private protocols. Full Stack golden signals of applications and infrastructures are calculated, pinpointing performance bottlenecks at ease.
• Distributed Tracing for Any Request: Zero Code distributed tracing powered by eBPF supports applications in any language and infrastructures including gateways, service meshes, databases, message queues, DNS and NICs, leaving no blind spots. Full Stack network performance metrics and file I/O events are automatically collected for each Span. Distributed tracing enters a new era: Zero Instrumentation.
• Continuous Profiling for Any Function: DeepFlow collects profiling data at a cost of below 1% with Zero Code, plots OnCPU/OffCPU function call stack flame graphs, locates Full Stack performance bottleneck in application, library and kernel functions, and automatically relates them to distributed tracing data. DeepFlow can even analyze code performance through network profiling under old version kernels (2.6+).
• Seamless Integration with Popular Stack: DeepFlow can serve as storage backed for Prometheus, OpenTelemetry, SkyWalking and Pyroscope. It also provides SQL, PromQL and OLTP APIs to work as data source in popular observability stacks. It injects meta tags for all observability signals including cloud resource, K8s container, K8s labels, K8s annotations, CMDB business attributes, etc., eliminating data silos.
• Performance 10x ClickHouse: SmartEncoding injects standardized and pre-encoded meta tags into all observability data, reducing storage overhead by 10x compared to ClickHouse String or LowCard method. Custom tags and observability data are stored separately, making tags available for almost unlimited dimensions and cardinalities with uncompromised query experience like BigTable.

Documentation

For more information, please visit the documentation website.

Quick start

There are three editions of DeepFlow:

• DeepFlow Community: for developers
• DeepFlow Enterprise: for organizations, solving team collaboration problems
• DeepFlow Cloud: SaaS service, currently in beta

The DeepFlow Community Edition consists of the core components of the Enterprise Edition.

DeepFlow Community

Please refer to the deployment documentation.

At the same time, we have also built a complete DeepFlow Community Demo, welcome to experience it. Login account/password: deepflow/deepflow.

DeepFlow Enterprise

You can visit the DeepFlow Enterprise Demo, currently available in Chinese only.

Compile DeepFlow from Source

• compile deepflow-agent

Software Architecture

DeepFlow Community Edition consists of two components, Agent and Server. An Agent runs in each K8s node, legacy host and cloud host, and is responsible for AutoMetrics and AutoTracing data collection of all application processes on the host. Server runs in a K8s cluster and provides Agent management, tag injection, data ingest and query services.

Milestones

Here is our future feature plan. Issues and Pull Requests are welcome.

Contact Us

• Discord：Click here to join our discussion.
• Twitter：DeepFlow
• WeChat Group：

Acknowledgments

• Thanks eBPF, a revolutionary Linux kernel technology.
• Thanks OpenTelemetry, provides vendor-neutral APIs to collect application telemetry data.

Honors

• The paper Network-Centric Distributed Tracing with DeepFlow: Troubleshooting Your Microservices in Zero Code has been accepted by ACM SIGCOMM 2023.
• DeepFlow enriches the CNCF CLOUD NATIVE Landscape.
• DeepFlow enriches the eBPF Project Landscape.

deepflow's Issues

eBPF: Add periodic checks on system procfs to detect process changes

metaflow-agent: add MQTT parsing support

eBPF: get golang function parameters from stack

eBPF: Add goroutines_map delete logic and remove warning log '\n'

eBPF: add Go TLS HTTP support

deepflow-server access the evolution of all ClickHouse databases

deepflow-server accesses all clickhouse databases using DNS SRV instead of using all data nodes IP in the database :NodePort list

For follow-up DNS SRV query, use clusterIP=None

The current scenario that uses all data nodes in the database :NodePort list uses externalTrafficPolicy=local

Example:

# dig srv _querier._tcp.querier.metaflow.svc.cluster.local. +short
0 33 30416 192-168-1-1.querier.metaflow.svc.cluster.local.
0 33 30416 192-168-1-2.querier.metaflow.svc.cluster.local.
0 33 30416 192-168-1-3.querier.metaflow.svc.cluster.local.

metaflow 还没正式开源吗？

metaflow 还没正式开源吗？

[agent] hostNetWork mode dnsPolicy shoud use "ClusterFirstWithHostNet"

Modify flow log generation logic

[eBPF] TLS HTTP2 support

检查程序里所有的Exit，凡是Exit之前都要等1s，等着日志刷出去（避免被cache住丢日志）。

检查程序里所有的Exit，凡是Exit之前都要等1s，等着日志刷出去（避免被cache住丢日志）。

@SongZhen0704 检查的工作可以单独pr解决，并形成统一的规范。但这个pr中需要保证新增的都这样。

Originally posted by @sharang in #214 (comment)

[eBPF]: Add eBPF bytecodes to the executable file metaflow-agent

WHY no data on dashboard network-k8s-pod application-k8s-pod

server log:
root@ubuntu:~# kubectl logs po/deepflow-server-0 -n deepflow -c deepflow-server |grep -i error
2022-07-28 18:56:16.241 [WARN] [grpc] grpc_session.go:96 Sync from server 127.0.0.1 failed, reason: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:20035: connect: connection refused"
2022-07-28 18:56:16.242 [WARN] [grpc] grpc_session.go:96 Sync from server 127.0.0.1 failed, reason: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:20035: connect: connection refused"
2022-07-28 18:56:16.244 [WARN] [grpc] grpc_session.go:96 Sync from server 127.0.0.1 failed, reason: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:20035: connect: connection refused"
2022-07-28 18:56:16.245 [WARN] [grpc] grpc_session.go:96 Sync from server 127.0.0.1 failed, reason: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:20035: connect: connection refused"
root@ubuntu:~# kubectl logs po/deepflow-server-0 -n deepflow -c deepflow-server |grep -i warn
2022-07-28 18:56:15.212 [WARN] [config] config.go:102 controller-ips is empty
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
[GIN-debug] [WARNING] You trusted all proxies, this is NOT safe. We recommend you to set a value.
2022-07-28 18:56:15.273 [WARN] [droplet.statsd] statsd.go:82 stats can not find addr telegraf:20040, err: lookup telegraf on 10.4.0.10:53: no such host
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
[GIN-debug] [WARNING] You trusted all proxies, this is NOT safe. We recommend you to set a value.
2022-07-28 18:56:16.241 [WARN] [grpc] grpc_session.go:96 Sync from server 127.0.0.1 failed, reason: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:20035: connect: connection refused"
2022-07-28 18:56:16.242 [WARN] [grpc] grpc_session.go:96 Sync from server 127.0.0.1 failed, reason: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:20035: connect: connection refused"
2022-07-28 18:56:16.242 [WARN] [grpc] grpc_platformdata.go:285 No reachable server
2022-07-28 18:56:16.242 [WARN] [grpc] grpc_platformdata.go:285 No reachable server
2022-07-28 18:56:16.244 [WARN] [grpc] grpc_session.go:96 Sync from server 127.0.0.1 failed, reason: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:20035: connect: connection refused"
2022-07-28 18:56:16.244 [WARN] [grpc] grpc_platformdata.go:285 No reachable server
2022-07-28 18:56:16.245 [WARN] [grpc] grpc_session.go:96 Sync from server 127.0.0.1 failed, reason: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:20035: connect: connection refused"
2022-07-28 18:56:16.246 [WARN] [grpc] grpc_platformdata.go:285 No reachable server
2022-07-28 18:59:15.518 [WARN] [cloud.kubernetes_gather] kubernetes_gather.go:224 no vtap report cluster id:d-VrW6AUkZrv
agent log :
2022-07-28 18:58:58  [eBPF] WARNING: resolve_bin_file:375:Go process pid 525720 [path: /proc/525720/root/usr/bin/containerd-shim-runc-v2] (version: go1.13). Not find any symbols!
2022-07-28 18:58:58  [eBPF] WARNING: resolve_bin_file:375:Go process pid 525860 [path: /proc/525860/root/usr/bin/containerd-shim-runc-v2] (version: go1.13). Not find any symbols!
2022-07-28 18:58:58  [eBPF] WARNING: resolve_bin_file:375:Go process pid 526598 [path: /proc/526598/root/usr/bin/containerd-shim-runc-v2] (version: go1.13). Not find any symbols!
2022-07-28 18:58:58  [eBPF] WARNING: resolve_bin_file:375:Go process pid 526868 [path: /proc/526868/root/usr/bin/containerd-shim-runc-v2] (version: go1.13). Not find any symbols!
2022-07-28 18:58:58  [eBPF] WARNING: resolve_bin_file:375:Go process pid 526984 [path: /proc/526984/root/usr/bin/containerd-shim-runc-v2] (version: go1.13). Not find any symbols!
2022-07-28 18:58:58  [eBPF] WARNING: resolve_bin_file:375:Go process pid 527285 [path: /proc/527285/root/usr/bin/containerd-shim-runc-v2] (version: go1.13). Not find any symbols!
2022-07-28 18:58:58  [eBPF] WARNING: resolve_bin_file:375:Go process pid 528849 [path: /proc/528849/root/usr/bin/containerd-shim-runc-v2] (version: go1.13). Not find any symbols!
2022-07-28 18:58:58  [eBPF] WARNING: resolve_bin_file:375:Go process pid 529427 [path: /proc/529427/root/usr/bin/containerd-shim-runc-v2] (version: go1.13). Not find any symbols!
2022-07-28 18:59:2  [eBPF] WARNING: resolve_bin_file:375:Go process pid 531275 [path: /proc/531275/root/usr/bin/containerd-shim-runc-v2] (version: go1.13). Not find any symbols!
2022-07-28 18:59:2  [eBPF] WARNING: resolve_bin_file:375:Go process pid 532236 [path: /proc/532236/root/usr/bin/containerd-shim-runc-v2] (version: go1.13). Not find any symbols!
2022-07-28 18:59:5  [eBPF] WARNING: resolve_bin_file:375:Go process pid 532764 [path: /proc/532764/root/var/lib/grafana/plugins/grafana-clickhouse-datasource/gpx_clickhouse_linux_amd64] (version: go1.18). Not find any symbols!
root@ubuntu:~# kubectl logs  po/deepflow-agent-f9j8c -n deepflow |grep -i error
I0728 19:10:01.059890  604997 request.go:668] Waited for 1.125846604s due to client-side throttling, not priority and fairness, request: GET:https://192.168.39.244:6443/apis/cert-manager.io/v1alpha3?timeout=32s
[2022-07-28 18:55:59.831046 +08:00] ERROR [src/rpc/session.rs:142] dial server(10.4.191.146) failed transport error
[2022-07-28 18:59:33.264680 +08:00] ERROR [src/ebpf_collector/ebpf_collector.rs:901] The tracer_start() error. Kernel offset adapt failed.

[eBPF] Remove the no-trace logic between same sockets

eBPF: Add periodic checks on system procfs to detect process changes

eBPF: For go binary file without symbols output a log

eBPF: Fix uprobe attach faild on linux 5.4+

eBPF: Snoop process execute/exit events for uprobes

Stop sending me emails!

My GitHub user name is zhuofeng. Perhaps you mentioned my username in this code. Please modify it and Stop sending me emails!

do not submit generated code

do not submit generated code

Originally posted by @sharang in #30 (comment)

eBPF: Add documentation for eBPF uprobes

[eBPF] Fix MySQL packet loss problem

eBPF: Modify the Linux kernel adaptive method

eBPF: Add go offsets and version information to eBPF Map

eBPF: Add uprobe load, attach and detach interface support

eBPF: Set a name for each eBPF threads

ignore eBPF bytecodes file

[eBPF] eBPF data distribution framework add multi-type event handling mechanism

add otel trace id

No doc, How to use?

The product presentation was great, but no "Getting started", how to use it?

Ingester compilation error caused by unknown dependency revision

Executing make in server/ leads to the following error:

go generate ./...
go mod tidy && go mod vendor
go: downloading github.com/ionos-cloud/sdk-go/v6 v6.0.5851
github.com/metaflowys/metaflow/server/ingester/ext_metrics/decoder imports
        github.com/prometheus/prometheus/storage/remote imports
        github.com/prometheus/prometheus/config tested by
        github.com/prometheus/prometheus/config.test imports
        github.com/prometheus/prometheus/discovery/ionos imports
        github.com/ionos-cloud/sdk-go/v6: reading github.com/ionos-cloud/sdk-go/go.mod at revision v6.0.5851: unknown revision v6.0.5851
make: *** [vendor] 错误 1

fix flow state testcase

[eBPF] Remove unnecessary logs

不过现在不涉及到升级，可以留个TODO吧 @SongZhen0704

不过现在不涉及到升级，可以留个TODO吧 @SongZhen0704

Originally posted by @sharang in #214 (comment)

Documentation Supplement

Looking forward to this project.
Can you provide more complete best practice documents? Including: installation setup, configuration instructions, production environment recommendations etc.

[eBPF] Revert 'Remove the no-trace logic between same sockets'

how to use it?

how can we use it?

config doc by manual? or other docs?

remove clickhouse-go WriteNullable patch

remote clickhouse-go WriteNullable patch

Originally posted by @sharang in #30 (comment)

Projects to merge

• agent
  • @rvql sha1: 5c82546
  • @rvql lock trident-rs@gitlab
• server/querier, server/controller, cli
  • @SongZhen0704 sha1:
  • @SongZhen0704 lock metaflow@gitlab
• server/ingester
  • @lzf575 sha1:
  • @lzf575 lock metaflow@gitlab
• server/libs
  • @yuanchaoa sha1: ea78e40
  • @yuanchaoa lock droplet-libs@gitlab
• message
  • @yuanchaoa sha1: 81e0013
  • @yuanchaoa lock message@gitlab

[eBPF] Fun ’resolve_bin_file()‘ optimize

fix ebpf packet direction

DeepFlow Querier no data returned

按文档https://deepflow.yunshan.net/docs/zh/install/single-k8s/部署，
grafana的三个数据源，其中DeepFlow Querier查不到数据，其余两个数据源有数据，请问是哪里出问题了呢？

eBPF: Delete kernel header file vmlinux.h

The file vmlinux.h is too big, delete the file and use the libbpf library instead.

test

eBPF: Adjust directly use the external library libbpf via submodule

[eBPF]: Solve the problem of tracking breakage caused by non-tracking between the same sockets

eBPF: Add periodic checks on system procfs to detect process changes

add bpf filter

[eBPF] fix trace id

deepflow lost sql statement when using gorm.io/gorm v1.22.4

Info

NAME             STATUS   ROLES                       AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
10-102-119-109   Ready    worker                      8h    v1.20.7   10.102.119.109   <none>        Ubuntu 20.04.2 LTS   5.4.0-72-generic   docker://20.10.8

tcp dump

app-profile-7c9457bb6c-trn4x:~# tcpdump -i eth0 dst  port 3306 -vvvvv
tcpdump: listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
12:02:38.235223 IP (tos 0x0, ttl 64, id 50553, offset 0, flags [DF], proto TCP (6), length 114)
    app-profile-7c9457bb6c-trn4x.60966 > 172.16.58.14.3306: Flags [P.], cksum 0xf59c (incorrect -> 0xf281), seq 3807471271:3807471345, ack 94817439, win 501, length 74
12:02:38.235897 IP (tos 0x0, ttl 64, id 50554, offset 0, flags [DF], proto TCP (6), length 40)
    app-profile-7c9457bb6c-trn4x.60966 > 172.16.58.14.3306: Flags [.], cksum 0xf552 (incorrect -> 0x9cb3), seq 74, ack 1211, win 501, length 0
12:02:38.235997 IP (tos 0x0, ttl 64, id 50555, offset 0, flags [DF], proto TCP (6), length 82)
    app-profile-7c9457bb6c-trn4x.60966 > 172.16.58.14.3306: Flags [P.], cksum 0xf57c (incorrect -> 0xfb83), seq 74:116, ack 1211, win 501, length 42
12:02:38.236664 IP (tos 0x0, ttl 64, id 50556, offset 0, flags [DF], proto TCP (6), length 40)
    app-profile-7c9457bb6c-trn4x.60966 > 172.16.58.14.3306: Flags [.], cksum 0xf552 (incorrect -> 0x9810), seq 116, ack 2356, win 501, length 0
12:02:38.236757 IP (tos 0x0, ttl 64, id 50557, offset 0, flags [DF], proto TCP (6), length 49)
    app-profile-7c9457bb6c-trn4x.60966 > 172.16.58.14.3306: Flags [P.], cksum 0xf55b (incorrect -> 0x79fc), seq 116:125, ack 2356, win 501, length 9
12:02:38.236911 IP (tos 0x0, ttl 64, id 50558, offset 0, flags [DF], proto TCP (6), length 114)
    app-profile-7c9457bb6c-trn4x.60966 > 172.16.58.14.3306: Flags [P.], cksum 0xf59c (incorrect -> 0xe8d1), seq 125:199, ack 2356, win 501, length 74
12:02:38.237526 IP (tos 0x0, ttl 64, id 50559, offset 0, flags [DF], proto TCP (6), length 40)
    app-profile-7c9457bb6c-trn4x.60966 > 172.16.58.14.3306: Flags [.], cksum 0xf552 (incorrect -> 0x9303), seq 199, ack 3566, win 501, length 0
12:02:38.237660 IP (tos 0x0, ttl 64, id 50560, offset 0, flags [DF], proto TCP (6), length 82)
    app-profile-7c9457bb6c-trn4x.60966 > 172.16.58.14.3306: Flags [P.], cksum 0xf57c (incorrect -> 0xf2d2), seq 199:241, ack 3566, win 501, length 42
12:02:38.238349 IP (tos 0x0, ttl 64, id 50561, offset 0, flags [DF], proto TCP (6), length 40)
    app-profile-7c9457bb6c-trn4x.60966 > 172.16.58.14.3306: Flags [.], cksum 0xf552 (incorrect -> 0x8e60), seq 241, ack 4711, win 501, length 0
12:02:38.238479 IP (tos 0x0, ttl 64, id 50562, offset 0, flags [DF], proto TCP (6), length 49)
    app-profile-7c9457bb6c-trn4x.60966 > 172.16.58.14.3306: Flags [P.], cksum 0xf55b (incorrect -> 0x704b), seq 241:250, ack 4711, win 501, length 9
12:02:43.239446 IP (tos 0x0, ttl 64, id 50563, offset 0, flags [DF], proto TCP (6), length 45)
    app-profile-7c9457bb6c-trn4x.60966 > 172.16.58.14.3306: Flags [P.], cksum 0xf557 (incorrect -> 0x8c4a), seq 250:255, ack 4711, win 501, length 5
12:02:43.239635 IP (tos 0x0, ttl 64, id 50564, offset 0, flags [DF], proto TCP (6), length 40)
    app-profile-7c9457bb6c-trn4x.60966 > 172.16.58.14.3306: Flags [F.], cksum 0xf552 (incorrect -> 0x8e51), seq 255, ack 4711, win 501, length 0
12:02:43.239790 IP (tos 0x0, ttl 64, id 32723, offset 0, flags [DF], proto TCP (6), length 60)
    app-profile-7c9457bb6c-trn4x.32782 > 172.16.58.14.3306: Flags [S], cksum 0xf566 (incorrect -> 0xe23d), seq 1047319603, win 64800, options [mss 1440,sackOK,TS val 2114737624 ecr 0,nop,wscale 7], length 0
12:02:43.240205 IP (tos 0x0, ttl 64, id 50565, offset 0, flags [DF], proto TCP (6), length 40)
    app-profile-7c9457bb6c-trn4x.60966 > 172.16.58.14.3306: Flags [.], cksum 0xf552 (incorrect -> 0x8e50), seq 256, ack 4712, win 501, length 0
12:02:43.240601 IP (tos 0x0, ttl 64, id 32724, offset 0, flags [DF], proto TCP (6), length 40)
    app-profile-7c9457bb6c-trn4x.32782 > 172.16.58.14.3306: Flags [.], cksum 0xf552 (incorrect -> 0xe6d8), seq 1047319604, ack 2683540027, win 507, length 0
12:02:43.241463 IP (tos 0x0, ttl 64, id 32725, offset 0, flags [DF], proto TCP (6), length 40)
    app-profile-7c9457bb6c-trn4x.32782 > 172.16.58.14.3306: Flags [.], cksum 0xf552 (incorrect -> 0xe681), seq 0, ack 88, win 507, length 0
12:02:43.241548 IP (tos 0x0, ttl 64, id 32726, offset 0, flags [DF], proto TCP (6), length 127)
    app-profile-7c9457bb6c-trn4x.32782 > 172.16.58.14.3306: Flags [P.], cksum 0xf5a9 (incorrect -> 0x61ab), seq 0:87, ack 88, win 507, length 87
12:02:43.242333 IP (tos 0x0, ttl 64, id 32727, offset 0, flags [DF], proto TCP (6), length 40)
    app-profile-7c9457bb6c-trn4x.32782 > 172.16.58.14.3306: Flags [.], cksum 0xf552 (incorrect -> 0xe61f), seq 87, ack 99, win 507, length 0
12:02:43.242419 IP (tos 0x0, ttl 64, id 32728, offset 0, flags [DF], proto TCP (6), length 62)
    app-profile-7c9457bb6c-trn4x.32782 > 172.16.58.14.3306: Flags [P.], cksum 0xf568 (incorrect -> 0xb4ee), seq 87:109, ack 99, win 507, length 22
12:02:43.243173 IP (tos 0x0, ttl 64, id 32729, offset 0, flags [DF], proto TCP (6), length 40)
    app-profile-7c9457bb6c-trn4x.32782 > 172.16.58.14.3306: Flags [.], cksum 0xf552 (incorrect -> 0xe5fe), seq 109, ack 110, win 507, length 0
12:02:43.243273 IP (tos 0x0, ttl 64, id 32730, offset 0, flags [DF], proto TCP (6), length 114)
    app-profile-7c9457bb6c-trn4x.32782 > 172.16.58.14.3306: Flags [P.], cksum 0xf59c (incorrect -> 0x36c9), seq 109:183, ack 110, win 507, length 74
12:02:43.244089 IP (tos 0x0, ttl 64, id 32731, offset 0, flags [DF], proto TCP (6), length 40)
    app-profile-7c9457bb6c-trn4x.32782 > 172.16.58.14.3306: Flags [.], cksum 0xf552 (incorrect -> 0xe100), seq 183, ack 1320, win 501, length 0
12:02:43.244179 IP (tos 0x0, ttl 64, id 32732, offset 0, flags [DF], proto TCP (6), length 82)
    app-profile-7c9457bb6c-trn4x.32782 > 172.16.58.14.3306: Flags [P.], cksum 0xf57c (incorrect -> 0x3fd3), seq 183:225, ack 1320, win 501, length 42
12:02:43.245063 IP (tos 0x0, ttl 64, id 32733, offset 0, flags [DF], proto TCP (6), length 40)
    app-profile-7c9457bb6c-trn4x.32782 > 172.16.58.14.3306: Flags [.], cksum 0xf552 (incorrect -> 0xdc5d), seq 225, ack 2465, win 501, length 0
12:02:43.245176 IP (tos 0x0, ttl 64, id 32734, offset 0, flags [DF], proto TCP (6), length 49)
    app-profile-7c9457bb6c-trn4x.32782 > 172.16.58.14.3306: Flags [P.], cksum 0xf55b (incorrect -> 0xbe4b), seq 225:234, ack 2465, win 501, length 9
12:02:43.245362 IP (tos 0x0, ttl 64, id 32735, offset 0, flags [DF], proto TCP (6), length 114)
    app-profile-7c9457bb6c-trn4x.32782 > 172.16.58.14.3306: Flags [P.], cksum 0xf59c (incorrect -> 0x2d1f), seq 234:308, ack 2465, win 501, length 74
12:02:43.246138 IP (tos 0x0, ttl 64, id 32736, offset 0, flags [DF], proto TCP (6), length 40)
    app-profile-7c9457bb6c-trn4x.32782 > 172.16.58.14.3306: Flags [.], cksum 0xf552 (incorrect -> 0xd750), seq 308, ack 3675, win 501, length 0
12:02:43.246242 IP (tos 0x0, ttl 64, id 32737, offset 0, flags [DF], proto TCP (6), length 82)
    app-profile-7c9457bb6c-trn4x.32782 > 172.16.58.14.3306: Flags [P.], cksum 0xf57c (incorrect -> 0x3722), seq 308:350, ack 3675, win 501, length 42
12:02:43.247109 IP (tos 0x0, ttl 64, id 32738, offset 0, flags [DF], proto TCP (6), length 40)
    app-profile-7c9457bb6c-trn4x.32782 > 172.16.58.14.3306: Flags [.], cksum 0xf552 (incorrect -> 0xd2ad), seq 350, ack 4820, win 501, length 0
12:02:43.247198 IP (tos 0x0, ttl 64, id 32739, offset 0, flags [DF], proto TCP (6), length 49)
    app-profile-7c9457bb6c-trn4x.32782 > 172.16.58.14.3306: Flags [P.], cksum 0xf55b (incorrect -> 0xb49a), seq 350:359, ack 4820, win 501, length 9
12:02:48.248354 IP (tos 0x0, ttl 64, id 32740, offset 0, flags [DF], proto TCP (6), length 114)
    app-profile-7c9457bb6c-trn4x.32782 > 172.16.58.14.3306: Flags [P.], cksum 0xf59c (incorrect -> 0x236f), seq 359:433, ack 4820, win 501, length 74
12:02:48.249196 IP (tos 0x0, ttl 64, id 32741, offset 0, flags [DF], proto TCP (6), length 40)
    app-profile-7c9457bb6c-trn4x.32782 > 172.16.58.14.3306: Flags [.], cksum 0xf552 (incorrect -> 0xcda0), seq 433, ack 6030, win 501, length 0
12:02:48.249331 IP (tos 0x0, ttl 64, id 32742, offset 0, flags [DF], proto TCP (6), length 82)
    app-profile-7c9457bb6c-trn4x.32782 > 172.16.58.14.3306: Flags [P.], cksum 0xf57c (incorrect -> 0x2c71), seq 433:475, ack 6030, win 501, length 42
12:02:48.250175 IP (tos 0x0, ttl 64, id 32743, offset 0, flags [DF], proto TCP (6), length 40)
    app-profile-7c9457bb6c-trn4x.32782 > 172.16.58.14.3306: Flags [.], cksum 0xf552 (incorrect -> 0xc8fd), seq 475, ack 7175, win 501, length 0
12:02:48.250304 IP (tos 0x0, ttl 64, id 32744, offset 0, flags [DF], proto TCP (6), length 49)
    app-profile-7c9457bb6c-trn4x.32782 > 172.16.58.14.3306: Flags [P.], cksum 0xf55b (incorrect -> 0xaae9), seq 475:484, ack 7175, win 501, length 9
12:02:48.250504 IP (tos 0x0, ttl 64, id 32745, offset 0, flags [DF], proto TCP (6), length 114)
    app-profile-7c9457bb6c-trn4x.32782 > 172.16.58.14.3306: Flags [P.], cksum 0xf59c (incorrect -> 0x19bf), seq 484:558, ack 7175, win 501, length 74
12:02:48.251283 IP (tos 0x0, ttl 64, id 32746, offset 0, flags [DF], proto TCP (6), length 40)
    app-profile-7c9457bb6c-trn4x.32782 > 172.16.58.14.3306: Flags [.], cksum 0xf552 (incorrect -> 0xc3f0), seq 558, ack 8385, win 501, length 0
12:02:48.251408 IP (tos 0x0, ttl 64, id 32747, offset 0, flags [DF], proto TCP (6), length 82)
    app-profile-7c9457bb6c-trn4x.32782 > 172.16.58.14.3306: Flags [P.], cksum 0xf57c (incorrect -> 0x23c0), seq 558:600, ack 8385, win 501, length 42
12:02:48.252244 IP (tos 0x0, ttl 64, id 32748, offset 0, flags [DF], proto TCP (6), length 40)
    app-profile-7c9457bb6c-trn4x.32782 > 172.16.58.14.3306: Flags [.], cksum 0xf552 (incorrect -> 0xbf4d), seq 600, ack 9530, win 501, length 0
12:02:48.252352 IP (tos 0x0, ttl 64, id 32749, offset 0, flags [DF], proto TCP (6), length 49)
    app-profile-7c9457bb6c-trn4x.32782 > 172.16.58.14.3306: Flags [P.], cksum 0xf55b (incorrect -> 0xa138), seq 600:609, ack 9530, win 501, length 9
x^C
40 packets captured
40 packets received by filter
0 packets dropped by kerne