README

Analysis and diagnoistcs tools for linux system performance.

STAP

stap: analysis and diagnoistcs tools for linux system based on SystemTap.

stap README

TODO

https://github.com/iovisor/bcc

https://github.com/brendangregg/bpf-perf-tools-book

https://elixir.bootlin.com/linux/v3.10/source

文档助手

常用的辅助文档

Linux manual pages:directory by project

SNMP

proc

proc

Brendan Gregg's Homepage

Arthas

动态追踪技术

一般优先使用传统工具进行观测

动态追踪技术漫谈

Systemtap: 动态追踪框架

SystemTap Beginners Guide

SystemTap Tutorial

SystemTap Examples

SystemTap Tapset Reference Manual

SystemTap Language Reference

eBPF

Linux Extended BPF (eBPF) Tracing Tools

需要较高内核的版本支持

Tools

linux perf

Linux crisis tool packages

Package	Provides
procps	ps(1), vmstat(8), uptime(1), top(1)
util-linux	dmesg(1), lsblk(1), lscpu(1)
sysstat	iostat(1), mpstat(1), pidstat(1), sar(1)
iproute2	ip(8), ss(8), nstat(8), tc(8)
numactl	numastat(8)
linux-tools-common linux-tools-$(uname -r)	perf(1), turbostat(8)
bcc-tools (aka bpfcc-tools)	opensnoop(8), execsnoop(8), runqlat(8), runqlen(8), softirqs(8), hardirqs(8), ext4slower(8), ext4dist(8), biotop(8), biosnoop(8), biolatency(8), tcptop(8), tcplife(8), trace(8), argdist(8), funccount(8), stackcount(8), profile(8), and many more
bpftrace	bpftrace, basic versions of opensnoop(8),execsnoop(8), runqlat(8), runqlen(8), biosnoop(8), biolatency(8), and more
perf-tools-unstable	Ftrace versions of opensnoop(8), execsnoop(8),iolatency(8), iosnoop(8), bitesize(8), funccount(8), kprobe(8)
trace-cmd	trace-cmd(1)
nicstat	nicstat(1)
ethtool	ethtool(8)
tiptop	tiptop(1)
msr-tools	rdmsr(8), wrmsr(8)
msr-cloud-tools	showboost(8), cpuhot(8), cputemp(8)
pmc-cloud-tools	pmcarch(8), cpucache(8), icache(8), tlbstat(8), resstalls(8)

Linux Observability Sources

Type	Source
Per-process counters	/proc
System-wide counters	/proc, /sys
Device configuration and counters	/sys
Cgroup statistics	/sys/fs/cgroup
Per-process tracing	ptrace
Hardware counters (PMCs)	perf_event
Network statistics	netlink
Network packet capture	libpcap
Per-thread latency metrics	Delay accounting
System-wide tracing	Function profiling (Ftrace), tracepoints, software events,kprobes, uprobes, perf_event

Linux tracing tools use the events interfaces (tracepoints, kprobes, uprobes, USDT) for advanced performance analysis.

Tool	Desc
perf(1)	The official Linux profiler. it is excellent for CPU profiling (sampling of stack traces) and PMC analysis, and can instrument other events, typically recording to an output file for post-processing
Ftrace	The official Linux tracer, it is a multi-tool composed of different tracing utilities. It is suited for kernel code path analysis and resource-constrained systems, as it can be used without dependencies
BPF (BCC, bpftrace)	Extended BPF, it powers advanced tracing tools, the main ones being BCC and bpftrace. BCC provides powerful tools, and bpftrace provides a high-level language for custom one-liners and short programs
SystemTap	A high-level language and tracer with many tapsets (libraries) for tracing different targets. It has recently been developing a BPF backend, which I recommend (see the stapbpf(8) man page)
LTTng	A tracer optimized for black-box recording: optimally recording many events for later analysis