pingcap / sysutil

sysutil is a library which implementats the gRPC service Diagnostics and shares the diagnostics functions between TiDB and PD.

sysutil

sysutil is a library which implementats the gRPC service Diagnostics and shares the diagnostics functions between TiDB and PD.

Search log

The semantics of the log search service is: search for local log files, and filter using predicates, and then return the matched results.

The following are the predicates that the log interface needs to process:

• start_time: start time of the log retrieval (Unix timestamp, in milliseconds). If there is no such predicate, the default is 0.
• end_time:: end time of the log retrieval (Unix timestamp, in milliseconds). If there is no such predicate, the default is int64::MAX.
• pattern: filter pattern determined by the keyword. For example, SELECT * FROM tidb_cluster_log WHERE "%gc%" %gc% is the filtered keyword.
• level: log level; can be selected as DEBUG/INFO/WARN/WARNING/TRACE/CRITICAL/ERROR
• limit: the maximum of logs items to return, preventing the log from being too large and occupying a large bandwidth of the network.. If not specified, the default limit is 64k.

System information collect

Hardware

• CPU
• NIC
• Disk
• Memory

System

• sysctl
• process list

Load

• CPU
• Memory
• NIC
• Disk IO

sysutil's Issues

Support to handle illegal format lines

Related issues:

• pingcap/tidb-dashboard#665
• pingcap/tiup#570

Search log panic

panic: runtime error: index out of range [512] with length 512
goroutine 759257 [running]:
github.com/pingcap/sysutil.readLastLines(0xc1dc657278, 0x12bffcce, 0xc289794340, 0x0, 0x0, 0xc21d88c000)
/home/jenkins/agent/workspace/idb_multi_branch_v4.0.8-20210726/go/pkg/mod/github.com/pingcap/sysutil@v0.0.0-20200715082929-4c47bcac246a/search_log.go:210 +0x366
github.com/pingcap/sysutil.readLastValidLog(0xc1dc657278, 0xa, 0xc1f442b3c0, 0x0, 0x0)
/home/jenkins/agent/workspace/idb_multi_branch_v4.0.8-20210726/go/pkg/mod/github.com/pingcap/sysutil@v0.0.0-20200715082929-4c47bcac246a/search_log.go:144 +0xe5
github.com/pingcap/sysutil.resolveFiles.func1(0xc1c517b400, 0x36, 0x37949c0, 0xc289794270, 0x0, 0x0, 0x121e929, 0xc289794270)
/home/jenkins/agent/workspace/idb_multi_branch_v4.0.8-20210726/go/pkg/mod/github.com/pingcap/sysutil@v0.0.0-20200715082929-4c47bcac246a/search_log.go:83 +0x37f
path/filepath.walk(0xc1c517b400, 0x36, 0x37949c0, 0xc289794270, 0xc4d2b5f9c0, 0x0, 0x0)
/usr/local/go/src/path/filepath/path.go:358 +0x425
path/filepath.walk(0x7ffc87348f0c, 0x15, 0x37949c0, 0xc1de149450, 0xc4d2b5f9c0, 0x0, 0x7ffc87348f0c)
/usr/local/go/src/path/filepath/path.go:382 +0x2ff
path/filepath.Walk(0x7ffc87348f0c, 0x15, 0xc10c5dc9c0, 0x4, 0x15ff97b)
/usr/local/go/src/path/filepath/path.go:404 +0xff
github.com/pingcap/sysutil.resolveFiles(0x7ffc87348f0c, 0x1e, 0x17af5211580, 0x17af52ed120, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/jenkins/agent/workspace/idb_multi_branch_v4.0.8-20210726/go/pkg/mod/github.com/pingcap/sysutil@v0.0.0-20200715082929-4c47bcac246a/search_log.go:54 +0x1be
github.com/pingcap/sysutil.(*DiagnosticsServer).SearchLog(0xc01464fe60, 0xc150009490, 0x3798a40, 0xc1de123df0, 0x0, 0x0)
/home/jenkins/agent/workspace/idb_multi_branch_v4.0.8-20210726/go/pkg/mod/github.com/pingcap/sysutil@v0.0.0-20200715082929-4c47bcac246a/service.go:45 +0x94
github.com/pingcap/kvproto/pkg/diagnosticspb._Diagnostics_SearchLog_Handler(0x31d8760, 0xc0123396b0, 0x37946c0, 0xc1de2b1e00, 0

invalid time formath when search log

The time format of TiDB log file not always same, sometime it was invalid, such as：

2020/05/18 22:12:11.823 trackerRecorder.go:27: [info] Mem Profile Tracker started
[2020/05/18 22:12:11.823 +08:00] [INFO] [printer.go:42] ["Welcome to TiDB."] ["Release Version"=v4.0.0-beta.2-476-gac30f5322] [Edition=Community] ["Git Commit Hash"=ac30f5322e253125002663ad7c789807acfc9305] ["Git Branch"=master] ["UTC Build Time"="2020-05-18 02:11:04"] [GoVersion=go1.13] ["Race Enabled"=false] ["Check Table Before Drop"=false] ["TiKV Min Version"=v3.0.0-60965b006877ca7234adaced7890d7b029ed1306]

The resolveFiles function will skip this log file, Should read more lines to check the file start time.