honggyukim / heaptrace Goto Github PK

View Code? Open in Web Editor NEW

10.0 10.0 3.0 126 KB

a tool that collects and reports heap allocated memory

License: GNU General Public License v2.0

Makefile 6.33% C 6.03% C++ 86.07% CMake 1.57%

heaptrace's People

Contributors

Stargazers

Watchers

Forkers

bojun-seo qpakzk pthread0

heaptrace's Issues

Add SLR(simple linear regression) analysis on report

I suggest to add SLR analysis on report. SLR could help users to understand the trend of the memory allocation for each location(stack id). The trend denotes slope and r square. slope means the amount of usage(count or size) increasing as time goes, and r square means the similarity(1: very similar, 0: not similar). See details for more information.
https://online.stat.psu.edu/stat462/node/79/

I add pseudo SLR analysis result in the report below right after the age. The original contents come from README.md file.

$ heaptrace --top 3 --sort count /usr/bin/node --expose-gc -e 'gc()'
[heaptrace] initialized for /proc/5879/maps
[heaptrace] finalized for /proc/5879/maps
=================================================================
[heaptrace] dump allocation status for /proc/5879/maps (node)
=== backtrace #1 === [count/peak: 43/60] [size/peak: 22.704 KB/31.680 KB] [age: 6.236 ms] [slope: 0.11 KB/s] [r_square: 0.84]
 0 [0x7fe72e46476c] malloc +0x1b
 1 [0x7fe72dd61e78] operator new(unsigned long) +0x6
 2 [      0xf27811] /usr/bin/node (+0xf27811)
 3 [      0xf2e57d] v8::internal::MarkCompactCollector::RootMarkingVisitor::VisitRootPointer(v8::internal::Root, char const*, v8::internal::Objec... +0x1b
 4 [     0x121c32d] v8::internal::SerializerDeserializer::Iterate(v8::internal::Isolate*, v8::internal::RootVisitor*) +0x47
 5 [      0xf07e45] v8::internal::Heap::IterateStrongRoots(v8::internal::RootVisitor*, v8::internal::VisitMode) +0x91
 6 [      0xf40b6b] v8::internal::MarkCompactCollector::MarkLiveObjects() +0x72
 7 [      0xf42071] v8::internal::MarkCompactCollector::CollectGarbage() +0x4

=== backtrace #2 === [count/peak: 12/13] [size/peak: 6.336 KB/6.864 KB] [age: 6.400 ms] [slope: 0.53 KB/s] [r_square: 0.24]
 0 [0x7fe72e46476c] malloc +0x1b
 1 [0x7fe72dd61e78] operator new(unsigned long) +0x6
 2 [      0xf27811] /usr/bin/node (+0xf27811)
 3 [      0xf29dd4] v8::internal::MarkCompactCollector::RootMarkingVisitor::VisitRootPointers(v8::internal::Root, char const*, v8::internal::Obje... +0x29
 4 [      0xb27068] v8::internal::HandleScopeImplementer::IterateThis(v8::internal::RootVisitor*) +0x1e
 5 [      0xf07d33] v8::internal::Heap::IterateStrongRoots(v8::internal::RootVisitor*, v8::internal::VisitMode) +0x4c
 6 [      0xf40b6b] v8::internal::MarkCompactCollector::MarkLiveObjects() +0x72
 7 [      0xf42071] v8::internal::MarkCompactCollector::CollectGarbage() +0x4

=== backtrace #3 === [count/peak: 10/10] [size/peak: 5.280 KB/5.280 KB] [age: 6.322 ms] [slope: 0.04 KB/s] [r_square: 0.99]
 0 [0x7fe72e46476c] malloc +0x1b
 1 [0x7fe72dd61e78] operator new(unsigned long) +0x6
 2 [      0xf27811] /usr/bin/node (+0xf27811)
 3 [      0xf2e57d] v8::internal::MarkCompactCollector::RootMarkingVisitor::VisitRootPointer(v8::internal::Root, char const*, v8::internal::Objec... +0x1b
 4 [      0xf07f08] v8::internal::Heap::IterateStrongRoots(v8::internal::RootVisitor*, v8::internal::VisitMode) +0xc2
 5 [      0xf40b6b] v8::internal::MarkCompactCollector::MarkLiveObjects() +0x72
 6 [      0xf42071] v8::internal::MarkCompactCollector::CollectGarbage() +0x4
 7 [      0xf15621] v8::internal::Heap::MarkCompact() +0x28

[heaptrace] heap traced num of backtrace : 64
[heaptrace] heap traced allocation size  : 59.215 KB
[heaptrace] allocator info (virtual)     : 2.121 MB
[heaptrace] allocator info (resident)    : 414.288 KB
[heaptrace] statm info (VSS/RSS/shared)  : 134.201 MB / 40.960 KB / 0 bytes

Report not printing backtrace issue

In some cases,(64-bits for kernel, and 32-bits for user environment)

[heaptrace] dump allocation status for /proc/1234/maps (caeaq)
=== backtrace #1 === [count/peak: 118838/118842] [size/peak: 163.510 MB/168.129 MB] [age: 28 mins 31 secs]

Report doesn't print backtraces.

I checked the reason and found that backtrace function,
https://man7.org/linux/man-pages/man3/backtrace.3.html
which is called inside record_backtrace function, returns zero.

Self allocation reporting issue

Way to reproduce
Compile the following code
just call g++ a.cpp will generate a.out

#include <iostream>
#include <unistd.h>
#include <malloc.h>

int period = 5;

void printErrorAndExit() {
  std::cerr << "Failed to allocate memory\n";
  exit(-1);
}

int* test_malloc() {
  int* tmp = (int*)malloc(sizeof(int));
  if (nullptr == tmp) {
    printErrorAndExit();
  }
  *tmp = 41;
  return tmp;
}

int main(int argc, char* argv[]) {
  int* val = nullptr;
  while (true) {
    val = test_malloc();
    *val = 33;
    sleep(period);
  }
  return 0;
}

$ ./heaptrace a.out
[heaptrace] initialized for /proc/2220/maps (a.out)
=================================================================
[heaptrace] dump allocation sorted by 'size' for /proc/2220/maps (a.out)
=== backtrace #1 === [count/peak: 1/1] [size/peak: 32 bytes/32 bytes] [age: 111.305 us]
 0 [0x7f51b508d47f] operator new(unsigned long) +0x1f
 1 [0x7f51b5093bc3] void std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11... +0x70
 2 [0x7f51b50938fe] ./libheaptrace.so (+0x7f51b50938fe)
 3 [0x7f51b5090487] ./libheaptrace.so (+0x7f51b5090487)
 4 [0x7f51b4c6f520] /lib/x86_64-linux-gnu/libc.so.6 (+0x7f51b4c6f520)
 5 [0x7f51b4d127fa] clock_nanosleep +0x16
 6 [0x7f51b4d176e7] __nanosleep +0x5
 7 [0x7f51b4d1761e] sleep +0xf

=== backtrace #2 === [count/peak: 7/7] [size/peak: 28 bytes/28 bytes] [age: 31.304 secs]
 0 [0x7f51b508d5ef] malloc +0x1f
 1 [0x559e9021722a] samples/a.out (+0x559e9021722a)
 2 [0x559e9021726a] samples/a.out (+0x559e9021726a)
 3 [0x7f51b4c56d90] /lib/x86_64-linux-gnu/libc.so.6 (+0x7f51b4c56d90)
 4 [0x7f51b4c56e40] __libc_start_main +0x20
 5 [0x559e90217125] samples/a.out (+0x559e90217125)

[heaptrace] heap traced num of backtrace : 2
[heaptrace] heap traced allocation size  : 60 bytes
[heaptrace] allocator info (virtual)     : 135.168 KB
[heaptrace] allocator info (resident)    : 80.496 KB
[heaptrace] statm info (VSS/RSS/shared)  : 6.246 MB / 1.966 MB / 1.777 MB
=================================================================

The report shows a heaptrace self allocated memory usage.

Reporting self consuming memory usage

I wish there is a feature to report heaptrace own memory usage(memory overhead).
So, it could be possible to measure heaptrace memory overhead.

Add configuration file for heaptrace options

The design of option delivering via environmental variable is good if the number of options to deliver is small. But now, it would be better to use configuration file on delivering options, as heaptrace will add ignore option #28 and signal option #33, and something unknown options in the future.
It could be possible to include the contents of ignore.txt in this configuration file for ignore option implementation.

So I suggest configure file format, default name and location. like followings:

File format: JSON
Default location: current location
Default name: .heaptrace.config

Setting configuration file option should be given or delivered via environmental variable. The usage will be:

$ heaptrace --config="/home/name/my_config" a.out

and

$ LD_PRELOAD=/usr/lib/libheaptrace.so HEAPTRACE_CONFIG=/home/name/my_config a.out

Configuration file example:

$ cat .heaptrace.config
{
  "num_top_backtrace": 20,
  "sort_keys": ["size", "count"],
  "out_file": "/tmp/my.log",
  "flame_graph": false,
  "ignore": ["libfoo", "bar", "fac", "baz"],
  "signals": {
    "size": 40,
    "count": 41,
    "slope": 42,
    "r_square": 43
  }
}

I think this change could help users to set options more conveniently. User only need to set configuration file and locate it on the default location or give the location with --config option.

Add alarm filtering (suppressions) feature

Sometimes user wants to remove some reports which are not necessary to be trace.
So I suggest to add alarm filtering feature.
Like suppressions feature on llvm-project Sanitizers
https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer#suppressions

Add option to change signals

heaptrace use SIGUSR1 and SIGUSR2 to dump allocation status (in terms of size and count respectively) during process running.
But the signal handler will not work properly, if the target process uses one or both of the signals.

So I suggest to add option to change signals to dump allocation status.
For example, (only to show the intention not an actual option format)

# register signal 33 to dump the allocation status sorted in terms of size
./heaptrace --signal size:33 ./a.out

# register signal 34 to dump the allocation status sorted in terms of count
./heaptrace --signal count:34 ./a.out

# register signal 33 and 34 to dump the allocation status sorted in terms of size and count
./heaptrace --signal size:33 --signal count:34 ./a.out

`LD_PRELOAD` execution causes segfault

heaptrace can be run without heaptrace binary file.
Following two commands are interchangeable.

$ ./heaptrace samples/sample.out
$ LD_PRELOAD=./libheaptrace.so samples/sample.out

But the latter generates segfault.

$ LD_PRELOAD=./libheaptrace.so samples/sample.out
Segmentation fault (core dumped)

`GLIBC` version unmatching issue for some cross compile environment

heaptrace failed to run because of GLIBC version unmatching for following cross compile environment.
I'm curious about the reason why heaptrace require specific GLIBC version and the location of GLIBC version checking logic in heaptrace.

Target information

Arch: arm64 (raspberry pi)
OS: Debian GNU/Linux 11 (Raspbian 64 bits version)

Unmatching case

Host arch: x86_64
Host OS: Ubuntu 22.04.1 LTS
Host compiler: aarch64-linux-gnu-g++ (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Compile command: CROSS_COMPILE=aarch64-linux-gnu- make

I copied heaptrace file and libheaptrace.so to the same directory on the target.
Run heaptrace and got error like below.

$ ./heaptrace /usr/bin/ls
./heaptrace: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by ./heaptrace)

But, there's no problem on following environment.

Host arch: x86_64
Host OS: Ubuntu 18.04.6 LTS
Host compiler: aarch64-linux-gnu-g++ (Ubuntu/Linaro 7.5.0-3ubuntu1~18.04) 7.5.0
Compile command: CROSS_COMPILE=aarch64-linux-gnu- make

Change the report signal numbers

I suggest to change the report signal numbers from SIGUSR1 and SIGUSR2 to some other numbers greater than 32, which is not usually reserved for system. There are two reasons.
First, SIGUSR1 and SIGUSR2 are too common so some other library or program could easily overwrite heaptrace registered signal handler.
Second, to support another sort key. The report could be sorted in slope or r square order, if we accept to add SLR(simple linear regression) analysis on report(See #34 for details about adding SLR). We need to add another signal to report in slope and r square order but there is no SIGUSR3 and SIGUSR4.

Add --chrome option

Do you think it is necessary to add --chrome option for viewing heap trace info on the Chrome tracing similar to uftrace dump --chrome?

support multiple sort orders in --flame-graph

As mentioned at #9 (comment), we might need to support printing flamegraph output for multiple sort orders.

We have to care that the output cannot be directly passed to flamegraph.pl if heaptrace prints output for multiple sort orders so it should be separately printed in another way.

No report is printed for `sample.c`

Expected result:
Report printed for sample.c
Since sample.c allocate memory with calloc but not deallocate it.

Actual result:
No report is printed for sample.c

$ ./heaptrace samples/sample.out
[heaptrace] initialized for /proc/5016/maps (sample.out)
[heaptrace]   finalized for /proc/5016/maps (sample.out)

consider replacing `mallinfo` to `malloc_info`

We need to consider replacing mallinfo to malloc_info by following the warning message.

/home/honggyu/work/heaptrace/stacktrace.cc: In function ‘void dump_stackmap(alloc_sort_order, bool)’:
/home/honggyu/work/heaptrace/stacktrace.cc:336:40: warning: ‘mallinfo mallinfo()’ is deprecated [-Wdeprecated-declarations]
  336 |         struct mallinfo info = mallinfo();
      |                                ~~~~~~~~^~
In file included from /home/honggyu/work/heaptrace/stacktrace.cc:6:
/usr/include/malloc.h:114:24: note: declared here
  114 | extern struct mallinfo mallinfo (void) __THROW __MALLOC_DEPRECATED;
      |                        ^~~~~~~~

The man page of mallinfo has a BUG section as follows.

BUGS
       Information is returned for only the main memory allocation area.  Allocations in other are‐
       nas  are excluded.  See malloc_stats(3) and malloc_info(3) for alternatives that include in‐
       formation about other arenas.

       The fields of the mallinfo structure are typed as int.  However, because some internal book‐
       keeping values may be of type long, the reported values may wrap around zero and thus be in‐
       accurate.

Report in `size` order as well as `count` order at the same time

Current heaptrace need to decide the report order. size or count.
I want to check the report order in size and count in some cases.
So it would be nice if it is possible to report in size order as well as count order at the same time.