slach / clickhouse-flamegraph Goto Github PK
View Code? Open in Web Editor NEWCLI utility for build flamegraph based on system.trace_log
License: MIT License
CLI utility for build flamegraph based on system.trace_log
License: MIT License
I am trying to visualize the system.trace_log via clickhouse-flamegraph. But readme doesn't help much as a first time user to use it.
Steps I followed till now:
installed clickhouse-flamegraph.
cloned Flamegraph repo by brendenGregg.
Desired configuration changed for profiling in config.xml and users.xml
CH server is running as a service
execute query with clickhouse-client.
I have a query id to profile.
I executed this query to see if trace_log exists for this Query . yes it exists.
'SELECT
arrayStringConcat(arrayReverse(arrayMap(x -> concat(addressToLine(x), '#', demangle(addressToSymbol(x))), trace)), ';') AS stack,
count() AS samples
FROM system.trace_log
WHERE (query_id = '2c3238dd-9de0-4f4b-82bb-511cfb69c810') AND (event_date = today())
GROUP BY trace
ORDER BY samples DESC
LIMIT 10
FORMAT TabSeparated'
What next , how I can use this query id to visualize system.trace_log in flamegraph.
Hi, is it possible to provide some use cases for the clickhouse-flamegraph tool. I still don't know how to use even with the usage in readme. ๐ As a user, what am I suppose to put for command and arguments, what do they even refer to? Where and when is the flamegraph svg generated, will a flamegraph be automatically generated whenever a client query? Or do I need to use the clickhouse-flamegraph tool to perform the query? It would be great if there could be a use case to follow through that helps the user to generate one flamegraph. Thank you! ๐
Error when output dir is .
clickhouse-flamegraph --clickhouse-dsn http://xxx?allow_introspection_functions=1 --query-id 7d6b01fa-89f0-4eaf-9d4b-e273a6e9ee84 --flamegraph-script /data1/home/wujianchao/cmd/FlameGraph --output-dir . --trace-type CPU --format svg --debug --from '2024-04-25 00:00:00' --to '2024-04-26 00:00:00'
{"level":"info","dsn":"http://xxx?allow_introspection_functions=1","time":"2024-04-25T15:38:21+08:00","message":"connected to ClickHouse"}
{"level":"debug","sql":"SELECT version() AS version","sqlArgs":"[]","time":"2024-04-25T15:38:21+08:00","message":"query OK"}
{"level":"debug","sql":"\nSELECT DISTINCT hostName() AS host_name, q.query, q.query_id \nFROM system.query_log AS q\nWHERE event_time >= ? AND event_time <= ? AND query_id IN ('7d6b01fa-89f0-4eaf-9d4b-e273a6e9ee84') \n","sqlArgs":"[2024-04-25 00:00:00 +0000 UTC 2024-04-26 00:00:00 +0000 UTC]","time":"2024-04-25T15:38:21+08:00","message":"query OK"}
{"level":"info","sqlFiles":1,"time":"2024-04-25T15:38:21+08:00","message":"write .sql files"}
{"level":"debug","sql":"\nSELECT \n\thostName() AS host_name,\n replaceAll(t.query_id,':','_') AS query_id,\n\ttrace_type,\n\tsum(abs(size)) AS total_size,\n\tcount() AS samples, \n\tconcat(\n\t\tmultiIf( \n\t\t\tposition( toString(trace_type), 'Memory') > 0 AND sum(size) >= 0, 'allocate;',\n\t\t\tposition( toString(trace_type), 'Memory') > 0 AND sum(size) < 0, 'free;',\n\t\t\tconcat( toString(trace_type), ';')\n\t\t),\n\t\tarrayStringConcat(arrayReverse(arrayMap(x -> concat( demangle(addressToSymbol(x)), '#', addressToLine(x) ), trace)), ';')\n\t) AS stack\nFROM system.trace_log AS t ANY LEFT JOIN system.query_log AS q ON q.query_id=t.query_id\nWHERE trace_type IN ('CPU') AND event_time >= ? AND event_time <= ? AND query_id IN ('7d6b01fa-89f0-4eaf-9d4b-e273a6e9ee84') \nGROUP BY host_name, query_id, trace_type, trace\nSETTINGS allow_introspection_functions=1\n","sqlArgs":"[2024-04-25 00:00:00 +0000 UTC 2024-04-26 00:00:00 +0000 UTC]","time":"2024-04-25T15:38:21+08:00","message":"query OK"}
panic: runtime error: index out of range [1] with length 1
goroutine 1 [running]:
main.generate(0xc0001ce680)
/data1/home/wujianchao/project/github/clickhouse-flamegraph/main.go:349 +0xb45
main.run(0xc000256160?)
/data1/home/wujianchao/project/github/clickhouse-flamegraph/main.go:199 +0x1e6
github.com/urfave/cli/v2.(*Command).Run(0xc000256160, 0xc0001ce680, {0xc0001aa000, 0x12, 0x12})
/data1/home/wujianchao/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:279 +0x9dd
github.com/urfave/cli/v2.(*App).RunContext(0xc00024c000, {0x8e9860?, 0xbb1b20}, {0xc0001aa000, 0x12, 0x12})
/data1/home/wujianchao/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:337 +0x5db
github.com/urfave/cli/v2.(*App).Run(...)
/data1/home/wujianchao/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:311
main.main()
/data1/home/wujianchao/project/github/clickhouse-flamegraph/main.go:152 +0x1725
The formatted sql in log :
SELECT
hostName() AS host_name,
replaceAll(t.query_id, ':', '_') AS query_id,
trace_type,
sum(abs(size)) AS total_size,
count() AS samples,
concat(
multiIf(
position(toString(trace_type), 'Memory') > 0 AND sum(size) >= 0, 'allocate;',
position(toString(trace_type), 'Memory') > 0 AND sum(size) < 0, 'free;',
concat(toString(trace_type), ';')
),
arrayStringConcat(arrayReverse(arrayMap(x -> concat(demangle(addressToSymbol(x)), '#', addressToLine(x)), trace)), ';')
) AS stack
FROM
system.trace_log AS t
ANY LEFT JOIN system.query_log AS q ON q.query_id = t.query_id
WHERE
trace_type IN ('CPU')
AND event_time >= '2024-04-25 00:00:00'
AND event_time <= '2024-04-26 00:00:00'
AND query_id IN ('7d6b01fa-89f0-4eaf-9d4b-e273a6e9ee84')
GROUP BY
host_name,
query_id,
trace_type,
trace
SETTINGS
allow_introspection_functions = 1
;
And got 79 rows in my ClickHouse instance.
Originally posted by @JackyWoo in #121 (comment)
Build with latest master branch
go version
go version go1.21.6 linux/amd64
And run as
clickhouse-flamegraph --clickhouse-dsn xx@xx/default --query-id 7d6b01fa-89f0-4eaf-9d4b-e273a6e9ee84 --flamegraph-script /data1/home/wujianchao/cmd/FlameGraph --output-dir . --trace-type CPU --format svg --debug
But got index out of range
{"level":"info","dsn":"xxxx","time":"2024-04-25T15:10:54+08:00","message":"connected to ClickHouse"}
panic: runtime error: index out of range [0] with length 0
goroutine 1 [running]:
github.com/mailru/go-clickhouse/v2.ensureHavePort({0x0, 0x0})
/data1/home/wujianchao/go/pkg/mod/github.com/mailru/go-clickhouse/[email protected]/config.go:187 +0x13f
github.com/mailru/go-clickhouse/v2.(*Config).url(0xc000174580, 0xc00023ced8?, 0x0)
/data1/home/wujianchao/go/pkg/mod/github.com/mailru/go-clickhouse/[email protected]/config.go:85 +0x47
github.com/mailru/go-clickhouse/v2.newConn(0xc000174580)
/data1/home/wujianchao/go/pkg/mod/github.com/mailru/go-clickhouse/[email protected]/conn.go:68 +0x105
github.com/mailru/go-clickhouse/v2.(*chDriver).Open(0x0?, {0x7ffcb1e6c472?, 0xc00023d150?})
/data1/home/wujianchao/go/pkg/mod/github.com/mailru/go-clickhouse/[email protected]/clickhouse.go:22 +0x3a
database/sql.dsnConnector.Connect(...)
/usr/local/go/src/database/sql/sql.go:758
database/sql.(*DB).conn(0xc000163110, {0x8e9860, 0xbb1b20}, 0x1)
/usr/local/go/src/database/sql/sql.go:1387 +0x717
database/sql.(*DB).query(0x0?, {0x8e9860, 0xbb1b20}, {0x84cd4a, 0x1b}, {0x0, 0x0, 0x0}, 0x98?)
/usr/local/go/src/database/sql/sql.go:1721 +0x57
database/sql.(*DB).QueryContext.func1(0x0?)
/usr/local/go/src/database/sql/sql.go:1704 +0x4f
database/sql.(*DB).retry(0xc00023d3b8?, 0xc00023d380)
/usr/local/go/src/database/sql/sql.go:1538 +0x42
database/sql.(*DB).QueryContext(0x17?, {0x8e9860?, 0xbb1b20?}, {0x84cd4a?, 0x83a9cc?}, {0x0?, 0x0?, 0x1?})
/usr/local/go/src/database/sql/sql.go:1703 +0xc5
database/sql.(*DB).Query(...)
/usr/local/go/src/database/sql/sql.go:1717
main.fetchQuery(0x7ffcb1e6c472?, {0x84cd4a, 0x1b}, {0x0, 0x0, 0x0}, 0xc00023d690)
/data1/home/wujianchao/project/github/clickhouse-flamegraph/main.go:586 +0x73
main.checkClickHouseVersion(...)
/data1/home/wujianchao/project/github/clickhouse-flamegraph/main.go:398
main.generate(0xc0001486c0)
/data1/home/wujianchao/project/github/clickhouse-flamegraph/main.go:263 +0x1ba
main.run(0xc00022c160?)
/data1/home/wujianchao/project/github/clickhouse-flamegraph/main.go:199 +0x1e6
github.com/urfave/cli/v2.(*Command).Run(0xc00022c160, 0xc0001486c0, {0xc0000240e0, 0xe, 0xe})
/data1/home/wujianchao/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:279 +0x9dd
github.com/urfave/cli/v2.(*App).RunContext(0xc000226000, {0x8e9860?, 0xbb1b20}, {0xc0000240e0, 0xe, 0xe})
/data1/home/wujianchao/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:337 +0x5db
github.com/urfave/cli/v2.(*App).Run(...)
/data1/home/wujianchao/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:311
main.main()
/data1/home/wujianchao/project/github/clickhouse-flamegraph/main.go:152 +0x1725
milovidov@server:~$ clickhouse-flamegraph --verbose --from=2019-11-30T00:00:00+03:00 --to=2019-11-30T01:00:00+03:00
{"level":"info","dsn":"tcp://localhost:9000?database=default","time":"2019-11-30T00:45:47+03:00","message":"conected to clickhouse"}
{"level":"debug","sql":"\nSELECT \n\tquery, query_id \nFROM system.query_log\nWHERE type = 1 AND event_time >= ? AND event_time <= ?\n","sqlArgs":"[2019-11-30 00:00:00 +0300 MSK 2019-11-30 01:00:00 +0300 MSK]","time":"2019-11-30T00:45:47+03:00","message":"query OK"}
{"level":"debug","sql":"\nSELECT \n\tquery_id,\n\tcount() AS samples, \n\tarrayStringConcat(arrayReverse(arrayMap(x -> concat( demangle(addressToSymbol(x)), '#', addressToLine(x) ), trace)), ';') AS stack\nFROM system.trace_log\nWHERE event_time >= ? AND event_time <= ?\nGROUP BY query_id, trace\n","sqlArgs":"[2019-11-30 00:00:00 +0300 MSK 2019-11-30 01:00:00 +0300 MSK]","time":"2019-11-30T00:45:47+03:00","message":"query OK"}
{"level":"fatal","error":"open clickouse-flamegraphs/1205caf2-5262-4825-85ea-521ffe9a68e2.txt: no such file or directory","stackFile":"clickouse-flamegraphs/1205caf2-5262-4825-85ea-521ffe9a68e2.txt","time":"2019-11-30T00:45:51+03:00"}
milovidov@server:~$ mkdir clickouse-flamegraphs
milovidov@server:~$ clickhouse-flamegraph --verbose --from=2019-11-30T00:00:00+03:00 --to=2019-11-30T01:00:00+03:00
{"level":"info","dsn":"tcp://localhost:9000?database=default","time":"2019-11-30T00:46:57+03:00","message":"conected to clickhouse"}
{"level":"debug","sql":"\nSELECT \n\tquery, query_id \nFROM system.query_log\nWHERE type = 1 AND event_time >= ? AND event_time <= ?\n","sqlArgs":"[2019-11-30 00:00:00 +0300 MSK 2019-11-30 01:00:00 +0300 MSK]","time":"2019-11-30T00:46:57+03:00","message":"query OK"}
{"level":"debug","sql":"\nSELECT \n\tquery_id,\n\tcount() AS samples, \n\tarrayStringConcat(arrayReverse(arrayMap(x -> concat( demangle(addressToSymbol(x)), '#', addressToLine(x) ), trace)), ';') AS stack\nFROM system.trace_log\nWHERE event_time >= ? AND event_time <= ?\nGROUP BY query_id, trace\n","sqlArgs":"[2019-11-30 00:00:00 +0300 MSK 2019-11-30 01:00:00 +0300 MSK]","time":"2019-11-30T00:46:57+03:00","message":"query OK"}
{"level":"info","processedFiles":4,"time":"2019-11-30T00:47:03+03:00","message":"done processing"}
clickouse-flamegraphs
directory.need refactoring main.go logic
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
This repository currently has no open or pending branches.
docker-compose.yml
docker/clickhouse-flamegraph/Dockerfile
.github/workflows/codeql-analysis.yml
actions/checkout v4
github/codeql-action v3
github/codeql-action v3
github/codeql-action v3
go.mod
go 1.22
github.com/araddon/dateparse v0.0.0-20210429162001-6b43995a97de@6b43995a97de
github.com/mailru/go-clickhouse/v2 v2.3.0
github.com/pkg/errors v0.9.1
github.com/rs/zerolog v1.33.0
github.com/urfave/cli/v2 v2.27.4
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.