Coder Social home page Coder Social logo

hdrhistogram_c's Introduction

HdrHistogram_c: 'C' port of High Dynamic Range (HDR) Histogram

HdrHistogram

Gitter chat

This port contains a subset of the functionality supported by the Java implementation. The current supported features are:

  • Standard histogram with 64 bit counts (32/16 bit counts not supported)
  • All iterator types (all values, recorded, percentiles, linear, logarithmic)
  • Histogram serialisation (encoding version 1.2, decoding 1.0-1.2)
  • Reader/writer phaser and interval recorder

Features not supported, but planned

  • Auto-resizing of histograms

Features unlikely to be implemented

  • Double histograms
  • Atomic/Concurrent histograms
  • 16/32 bit histograms

Simple Tutorial

Recording values

#include <hdr_histogram.h>

struct hdr_histogram* histogram;

// Initialise the histogram
hdr_init(
    1,  // Minimum value
    INT64_C(3600000000),  // Maximum value
    3,  // Number of significant figures
    &histogram)  // Pointer to initialise

// Record value
hdr_record_value(
    histogram,  // Histogram to record to
    value)  // Value to record

// Record value n times
hdr_record_values(
    histogram,  // Histogram to record to
    value,  // Value to record
    10)  // Record value 10 times

// Record value with correction for co-ordinated omission.
hdr_record_corrected_value(
    histogram,  // Histogram to record to
    value,  // Value to record
    1000)  // Record with expected interval of 1000.

// Print out the values of the histogram
hdr_percentiles_print(
    histogram,
    stdout,  // File to write to
    5,  // Granularity of printed values
    1.0,  // Multiplier for results
    CLASSIC);  // Format CLASSIC/CSV supported.

More examples

For more detailed examples of recording and logging results look at the hdr_decoder and hiccup examples. You can run hiccup and decoder and pipe the results of one into the other.

$ ./examples/hiccup | ./examples/hdr_decoder

hdrhistogram_c's People

Contributors

ahwayakchih avatar baruch avatar beberlei avatar cesaref avatar chrisejones avatar chrisvest avatar chronoxor avatar cluon avatar davidkorczynski avatar dotnwat avatar filipecosta90 avatar giltene avatar htuch avatar jerrinot avatar jsgf avatar mattwarren avatar mikeb01 avatar mikedld avatar natoscott avatar orpiske avatar oskurat avatar penberg avatar peterfaiman avatar peterm0x avatar remicollet avatar rgacogne avatar stig avatar sv avatar tristan957 avatar yoav-steinberg avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

hdrhistogram_c's Issues

Road to 1.0.

The C implementation of HDR Histogram is fairly stable now, but there are a number of pending changes that should really be in place before making it a 1.0 version. The 3 outstanding features I think that should be required are:

  • packed arrays.
  • resizable counts.
  • double support.

However to complete this I think I would need to break the API to some degree. Mostly because the main struct is defined in the header which makes changing it potentially a breaking a change. To make future changes more flexible, I would like to make hdr_histogram an opaque point and hide the internal structure.

Feedback on how this may impact users would be appreciated. E.g. are there any users depending on stack allocation of the hdr_histogram?

Please reply to this issue if you have any issues or suggestions regarding the implementation for the next major revision of the HDR Histgoram.

Build issues

I'm facing some issues while I'm building cmake file in test dir. Below are some errors on Ubuntu 18.04.4 LTS

ashwini@ashwini-HP-Notebook:~/cisco_code/HdrHistogram_c-master/test$ cmake .
CMake Warning (dev) in CMakeLists.txt:
  No cmake_minimum_required command is present.  A line of code such as

    cmake_minimum_required(VERSION 3.10)

  should be added at the top of the file.  The version specified may be lower
  if you wish to support older CMake versions for this project.  For more
  information run "cmake --help-policy CMP0000".
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Configuring done
-- Generating done
-- Build files have been written to: /home/ashwini/cisco_code/HdrHistogram_c-master/test
ashwini@ashwini-HP-Notebook:~/cisco_code/HdrHistogram_c-master/test$ make
[  5%] Building C object CMakeFiles/hdr_histogram_atomic_test.dir/hdr_histogram_atomic_test.o
/home/ashwini/cisco_code/HdrHistogram_c-master/test/hdr_histogram_atomic_test.c:13:10: fatal error: hdr_histogram.h: No such file or directory
 #include <hdr_histogram.h>
          ^~~~~~~~~~~~~~~~~
compilation terminated.
CMakeFiles/hdr_histogram_atomic_test.dir/build.make:62: recipe for target 'CMakeFiles/hdr_histogram_atomic_test.dir/hdr_histogram_atomic_test.o' failed
make[2]: *** [CMakeFiles/hdr_histogram_atomic_test.dir/hdr_histogram_atomic_test.o] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/hdr_histogram_atomic_test.dir/all' failed
make[1]: *** [CMakeFiles/hdr_histogram_atomic_test.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2

Add atomic value recording APIs

Given the prevalence of the recorder pattern, I often find it useful to have atomic recording into a histogram (thus supporting concurrent recording activity from multiple threads) that does not require other histogram operations to be concurrent-safe.

By "Atomic" I mean that the incrementing of counts will be atomic, and that the updating of min and max will be atomic, but NOT that the overall combined operation will appear atomic (no need for that). We DO want the combined operation, when performed in a recorder, to happen in one phaser-mediated critical section (so that the combination of counts update and min/max update is properly synchronized against recorder samples).

The use pattern is simple: value recording is concurrent (usually into a recorder), but observing the histogram is single-threaded and performed only on histograms obtained via a recorder sample. A specific example use case I am looking at this for is histogram'ing allocated object sizes and reporting on those histograms periodically.

I think that this can be added quite easily to the current C implementation by adding the following APIs:
hdr_record_value_atomic
hdr_record_values_atomic
hdr_record_corrected_value_atomic
hdr_record_corrected_values_atomic
hdr_interval_recorder_record_value_atomic
hdr_interval_recorder_record_values_atomic
hdr_interval_recorder_record_corrected_value_atomic
hdr_interval_recorder_record_corrected_values_atomic

The implementation details are probably obvious from the above.

INTEGER_OVERFLOW in `src/hdr_histogram.c`

The array elements are summed in function void hdr_reset_internal_counters(struct hdr_histogram* h). However, both the array elements and the summation variable have the same int64_t data type:

int64_t observed_total_count = 0;

int64_t count_at_index;

And then summation:
observed_total_count += count_at_index;

Also, no overflow checks were found in the function itself.

Found by Linux Verification Center (portal.linuxtesting.ru) with SVACE.

Author A. Voronin.

Documentation of the hdr_init

Hello,
I would like to understand a bit more the parameters of the hdr_init, what do these parameters means:
lowest_discernible_value, highest_trackable_value, significant_figures ?
Is there a documentation available somewhere?

Many thanks

Incorrect start time in the header

I noticed that the header generated by the C library contains an invalid start time entry. The issue can be reproduced by simply running the hiccup example:

$ cat ~/tmp/reproducer.hdr 
#[foobar]
#[Histogram log format version 1.2]
#[StartTime: 4199232.004 (seconds since epoch), Wed Feb 14:27:12 GMT 1970]
"StartTimestamp","EndTimestamp","Interval_Max","Interval_Compressed_Histogram"
292663.168,292664.169,227.0,HISTFAAAAJp4nC2NsQ3CQBAE/5fjfDofp8c2BiMjOSagABISKkAUQEgBlEBOQEJntEEJPDIrjVaaDba/P+sQwiOMmfw7ZprN+xIOn1EUoEgGImICXJSYmSQRRJgaca88rUwssTJYq0Yt72oqXokzg5RIxYjd1Pnnkopq3+52w/UVb6f9+njeDq2ZIl9MM0vMMQM6FKjRcQSV2Za0wBe9gQxY

In the heather above, the Start Time is incorrectly displayed as "Wed Feb 14:27:12 GMT 1970".

This invalid header seems to prevent the Java code from HdrHistogram to parse the file:

$ ~/code/java/HdrHistogram/HistogramLogProcessor -i ~/tmp/reproducer.hdr
       Value     Percentile TotalCount 1/(1-Percentile)

#[Mean    =         0.00, StdDeviation   =         0.00]
#[Max     =         0.00, Total count    =            0]
#[Buckets =           13, SubBuckets     =          256]

The issue seems to be caused because the timestamp value is initialized. Additionally the hdr_gettime function is not suitable for initializing the timestamp because it relies on CLOCK_MONOTONIC.

Add a hdr_interval_recorder example to README

It would be useful to have a simple example of using hdr_interval_recorder (recording, sampling, producing log output from sample) in the README, since it's [presumably] a common use pattern.

document hdr_interval_recorder_sample() is "dangerous".

Currently, there are two potential "dangerous" things in hdr_interval_recorder_sample()'s behavior that would not be obvious to the caller without clear documentation:

  • hdr_interval_recorder_sample() is not thread safe when multiple threads may call it (is is still thread-safe for one reader against multiple recorders). E.g. if two readers called the method concurrently, they could both end up receiving the same (2nd) interval sample, and the 1st would be lost.

  • In addition, since the next sample call will recycle the r->inactive histogram returned by this call, a histogram returned by a previous-sample will become active without this being obvious to the caller (who may choose to retain multiple histograms).

In contrast, hdr_interval_recorder_sample_and_recycle is both multi-reader-thread safe, and will only "stomp" the histogram passe to it for recycling (can probably change parameter name there form "inactive_histogram" to "histogram_to_recycle" to make it abundantly clear).

While hdr_interval_recorder_sample does fit the dominant and common use case (single reader, which does not retain a histogram across sample calls), the current API probably needs documentation that would warn-off misuse and spell out the limitations or the implementation should change or dropped.

I believe that the common idiom can be changed from

inactive = hdr_interval_recorder_sample(&recorder);

to the already available:

histogram = hdr_interval_recorder_sample_and_recycle(&recorder, histogram);

This change is quite simple for the caller and does not require any additional code (histogram can be initialized to null at the start), and makes it very clear (in the calling code) that the histogram is being recycled, so that the "wrongness" of using it (the one-handed in for recycling, not the one returned from the sample) after the call is obvious.

Suggest fuzzer for hdr_log_read()

I suggest this fuzzer for continuous vulnerability checks.

/*
 * This fuzzer is generated by UTopia with some manual modifications.
 * (UTopia Project: https://github.com/Samsung/UTopia)
 */

#include "FuzzedDataProvider.h"
#include <fstream>
#include <hdr/hdr_histogram_log.h>
#include <math.h>

bool save_file(std::string Path, std::string Content) {
  std::ofstream OFS(Path);
  if (!OFS.is_open())
    return false;

  OFS << Content;
  return true;
}

void fuzz_hdr_log_read(FuzzedDataProvider &provider) {
  auto input = provider.ConsumeRemainingBytesAsString();
  const char *file_name = "input.log";
  save_file(file_name, input);

  struct hdr_histogram *h = NULL;
  hdr_timespec timestamp, interval;

  FILE *f = fopen(file_name, "r");
  if (NULL == f) {
    fprintf(stderr, "Open file: [%d] %s", errno, strerror(errno));
    return;
  }

  hdr_log_read(NULL, f, &h, &timestamp, &interval);
  fclose(f);
  remove(file_name);
}

extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, uint32_t size) {
  FuzzedDataProvider provider(data, size);
  fuzz_hdr_log_read(provider);
  
  return 0;
}

Segmentation fault

Hi @mikeb01 , this is in reference with issue( #40 )

i am still get segfault , please find the stack trace below

GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-94.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/wrk/wrk2/wrk...(no debugging symbols found)...done.
[New LWP 14233]
[New LWP 14216]
[New LWP 14232]
[New LWP 14217]
[New LWP 14226]
[New LWP 14212]
[New LWP 14215]
[New LWP 14230]
[New LWP 14221]
[New LWP 14219]
[New LWP 14225]
[New LWP 14229]
[New LWP 14223]
[New LWP 14228]
[New LWP 14231]
[New LWP 14224]
[New LWP 14227]
[New LWP 14222]
[New LWP 14218]
[New LWP 14213]
[New LWP 14214]
[New LWP 14220]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/opt/wrk/wrk2/wrk -t 20 -c 1000 -d 300 -p 10 -s wrk_script.lua -R 20000 https:/'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f12158c0292 in counts_get_direct () from /usr/local/lib/libhdr_histogram.so.1
Missing separate debuginfos, use: debuginfo-install glibc-2.17-157.el7_3.1.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.14.1-27.el7_3.x86_64 libcom_err-1.42.9-9.el7.x86_64 libgcc-4.8.5-11.el7.x86_64 libselinux-2.5-6.el7.x86_64 openssl-libs-1.0.1e-60.el7_3.1.x86_64 pcre-8.32-15.el7_2.1.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) bt
#0  0x00007f12158c0292 in counts_get_direct () from /usr/local/lib/libhdr_histogram.so.1
#1  0x00007f12158c02c8 in counts_get_normalised () from /usr/local/lib/libhdr_histogram.so.1
#2  0x00007f12158c138e in move_next () from /usr/local/lib/libhdr_histogram.so.1
#3  0x00007f12158c14da in _basic_iter_next () from /usr/local/lib/libhdr_histogram.so.1
#4  0x00007f12158c1962 in _recorded_iter_next () from /usr/local/lib/libhdr_histogram.so.1
#5  0x00007f12158c15f2 in hdr_iter_next () from /usr/local/lib/libhdr_histogram.so.1
#6  0x00007f12158c0e61 in hdr_add () from /usr/local/lib/libhdr_histogram.so.1
#7  0x0000000000408ae5 in gen_stats ()
#8  0x0000000000409106 in period_report_func ()
#9  0x000000000040bde2 in aeProcessEvents ()
#10 0x000000000040c06b in aeMain ()
#11 0x00000000004079fa in period_report ()
#12 0x00007f1216428dc5 in start_thread () from /lib64/libpthread.so.0
#13 0x00007f12151d973d in clone () from /lib64/libc.so.6

Build failed

There is a missed #endif at the end of the file for #if defined(_MSC_VER)

hdr_atomic.h(104): fatal error C1070: mismatched #if/#endif pair

segmentation fault

@mikeb01 Hi Mike, I am running wrk with histogram reporting and when i am running @ high TPS(>6000) getting segmentation fault and stack trace states that it crashes in the hdr histogram library.

i have tried with latest histogram, below are the WRK and Histogram repo's i am using

Wrk Repo: https://github.com/yicwang/wrk2.git
HdrHistogram: https://github.com/HdrHistogram/HdrHistogram_c.git

Below is the stack trace and @ahothan pointed me to raise a issue here and i need your help in resolving the issue.

GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-94.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/wrk/wrk2/wrk...(no debugging symbols found)...done.
[New LWP 14233]
[New LWP 14216]
[New LWP 14232]
[New LWP 14217]
[New LWP 14226]
[New LWP 14212]
[New LWP 14215]
[New LWP 14230]
[New LWP 14221]
[New LWP 14219]
[New LWP 14225]
[New LWP 14229]
[New LWP 14223]
[New LWP 14228]
[New LWP 14231]
[New LWP 14224]
[New LWP 14227]
[New LWP 14222]
[New LWP 14218]
[New LWP 14213]
[New LWP 14214]
[New LWP 14220]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/opt/wrk/wrk2/wrk -t 20 -c 1000 -d 300 -p 10 -s wrk_script.lua -R 20000 https:/'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f12158c0292 in counts_get_direct () from /usr/local/lib/libhdr_histogram.so.1
Missing separate debuginfos, use: debuginfo-install glibc-2.17-157.el7_3.1.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.14.1-27.el7_3.x86_64 libcom_err-1.42.9-9.el7.x86_64 libgcc-4.8.5-11.el7.x86_64 libselinux-2.5-6.el7.x86_64 openssl-libs-1.0.1e-60.el7_3.1.x86_64 pcre-8.32-15.el7_2.1.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) bt
#0  0x00007f12158c0292 in counts_get_direct () from /usr/local/lib/libhdr_histogram.so.1
#1  0x00007f12158c02c8 in counts_get_normalised () from /usr/local/lib/libhdr_histogram.so.1
#2  0x00007f12158c138e in move_next () from /usr/local/lib/libhdr_histogram.so.1
#3  0x00007f12158c14da in _basic_iter_next () from /usr/local/lib/libhdr_histogram.so.1
#4  0x00007f12158c1962 in _recorded_iter_next () from /usr/local/lib/libhdr_histogram.so.1
#5  0x00007f12158c15f2 in hdr_iter_next () from /usr/local/lib/libhdr_histogram.so.1
#6  0x00007f12158c0e61 in hdr_add () from /usr/local/lib/libhdr_histogram.so.1
#7  0x0000000000408ae5 in gen_stats ()
#8  0x0000000000409106 in period_report_func ()
#9  0x000000000040bde2 in aeProcessEvents ()
#10 0x000000000040c06b in aeMain ()
#11 0x00000000004079fa in period_report ()
#12 0x00007f1216428dc5 in start_thread () from /lib64/libpthread.so.0
#13 0x00007f12151d973d in clone () from /lib64/libc.so.6

Version info needed in the repository

Currently, there are no file/variable to identify the current version of the histogram package. Need to get the version from any of the file.

It could be something #define CURRENT_VERSION = 0.11.6.

It will be more useful to maintain the version of dependancies where the histogram is being used.

test failure on debian jessie, x86 and ARM7

commit a61ade5 , built with cmake -DCMAKE_BUILD_TYPE=Debug .

mah@nwheezy:~/src/HdrHistogram_c/test$ uname -a
Linux nwheezy 3.16.0-4-586 #1 Debian 3.16.7-ckt11-1 (2015-05-24) i686 GNU/Linux

mah@nwheezy:~/src/HdrHistogram_c/test$ make test
Running tests...
Test project /home/mah/src/HdrHistogram_c/test
    Start 1: Histogram
1/4 Test #1: Histogram ........................***Failed    0.00 sec
    Start 2: HistogramLogging
2/4 Test #2: HistogramLogging .................***Exception: SegFault  0.00 sec
    Start 3: DoubleHistogram
3/4 Test #3: DoubleHistogram ..................***Failed    0.00 sec
    Start 4: DoubleHistogramDataAccess
4/4 Test #4: DoubleHistogramDataAccess ........***Exception: SegFault  0.00 sec

0% tests passed, 4 tests failed out of 4

Total Test time (real) =   0.01 sec

The following tests FAILED:
      1 - Histogram (Failed)
      2 - HistogramLogging (SEGFAULT)
      3 - DoubleHistogram (Failed)
      4 - DoubleHistogramDataAccess (SEGFAULT)
Errors while running CTest
Makefile:137: recipe for target 'test' failed
make: *** [test] Error 8

Running under gdb:

mah@nwheezy:~/src/HdrHistogram_c/test$ gdb hdr_dbl_histogram_data_access_test
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i586-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from hdr_dbl_histogram_data_access_test...done.
(gdb) r
Starting program: /home/mah/src/HdrHistogram_c/test/hdr_dbl_histogram_data_access_test 

Program received signal SIGSEGV, Segmentation fault.
0xb7fd22b8 in hdr_dbl_record_values (h=0x0, value=1000, count=1) at /home/mah/src/HdrHistogram_c/src/hdr_dbl_histogram.c:254
254     if (value < h->current_lowest_value || h->current_highest_value <= value)
(gdb) backtrace 
#0  0xb7fd22b8 in hdr_dbl_record_values (h=0x0, value=1000, count=1) at /home/mah/src/HdrHistogram_c/src/hdr_dbl_histogram.c:254
#1  0xb7fd23d9 in hdr_dbl_record_corrected_values (h=0x0, value=1000, count=1, expected_interval=10000) at /home/mah/src/HdrHistogram_c/src/hdr_dbl_histogram.c:275
#2  0xb7fd2384 in hdr_dbl_record_corrected_value (h=0x0, value=1000, expected_interval=10000) at /home/mah/src/HdrHistogram_c/src/hdr_dbl_histogram.c:270
#3  0x08048f82 in load_histograms () at /home/mah/src/HdrHistogram_c/test/hdr_dbl_histogram_data_access_test.c:47
#4  0x080490e2 in test_scaling_equivalence () at /home/mah/src/HdrHistogram_c/test/hdr_dbl_histogram_data_access_test.c:64
#5  0x08049ca8 in all_tests () at /home/mah/src/HdrHistogram_c/test/hdr_dbl_histogram_data_access_test.c:325
#6  0x08049ee5 in hdr_dbl_histogram_data_access_run_tests () at /home/mah/src/HdrHistogram_c/test/hdr_dbl_histogram_data_access_test.c:339
#7  0x08049f4a in main (argc=1, argv=0xbffff1f4) at /home/mah/src/HdrHistogram_c/test/hdr_dbl_histogram_data_access_test.c:355
(gdb) 


mah@nwheezy:~/src/HdrHistogram_c/test$ ldd hdr_dbl_histogram_data_access_test 
    linux-gate.so.1 (0xb7747000)
    libhdr_histogram.so => /home/mah/src/HdrHistogram_c/src/libhdr_histogram.so (0xb7739000)
    libm.so.6 => /lib/i386-linux-gnu/i686/cmov/libm.so.6 (0xb76d1000)
    libz.so.1 => /lib/i386-linux-gnu/libz.so.1 (0xb76b3000)
    libc.so.6 => /lib/i386-linux-gnu/i686/cmov/libc.so.6 (0xb7509000)
    /lib/ld-linux.so.2 (0xb774a000)

also:

mah@nwheezy:~/src/HdrHistogram_c/test$ gdb hdr_histogram_log_test 
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i586-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from hdr_histogram_log_test...done.
(gdb) r
Starting program: /home/mah/src/HdrHistogram_c/test/hdr_histogram_log_test 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0xb7fd470d in get_bucket_index (h=0x0, value=1000) at /home/mah/src/HdrHistogram_c/src/hdr_histogram.c:110
110     int32_t pow2ceiling = 64 - __builtin_clzll(value | h->sub_bucket_mask); // smallest power of 2 containing value
(gdb) backtrace 
#0  0xb7fd470d in get_bucket_index (h=0x0, value=1000) at /home/mah/src/HdrHistogram_c/src/hdr_histogram.c:110
#1  0xb7fd480d in counts_index_for (h=0x0, value=1000) at /home/mah/src/HdrHistogram_c/src/hdr_histogram.c:137
#2  0xb7fd5603 in hdr_record_values (h=0x0, value=1000, count=1) at /home/mah/src/HdrHistogram_c/src/hdr_histogram.c:512
#3  0xb7fd55b3 in hdr_record_value (h=0x0, value=1000) at /home/mah/src/HdrHistogram_c/src/hdr_histogram.c:502
#4  0x0804960a in load_histograms () at /home/mah/src/HdrHistogram_c/test/hdr_histogram_log_test.c:160
#5  0x080496b1 in test_encode_and_decode_compressed () at /home/mah/src/HdrHistogram_c/test/hdr_histogram_log_test.c:196
#6  0x0804a728 in all_tests () at /home/mah/src/HdrHistogram_c/test/hdr_histogram_log_test.c:643
#7  0x0804abec in hdr_histogram_log_run_tests () at /home/mah/src/HdrHistogram_c/test/hdr_histogram_log_test.c:673
#8  0x0804ac51 in main (argc=1, argv=0xbffff204) at /home/mah/src/HdrHistogram_c/test/hdr_histogram_log_test.c:691
(gdb) 

NB: perftest works on all patforms.

make tests DOES pass on an amd64 platform:


mah@amd64:~/HdrHistogram_c$ uname -a
Linux amd64 4.1.3-rt3mah+ #2 SMP PREEMPT RT Mon Jul 27 06:46:00 CEST 2015 x86_64 GNU/Linux

mah@amd64:~/HdrHistogram_c$ make test
Running tests...
Test project /home/mah/HdrHistogram_c
    Start 1: Histogram
1/4 Test #1: Histogram ........................   Passed    0.17 sec
    Start 2: HistogramLogging
2/4 Test #2: HistogramLogging .................   Passed    0.18 sec
    Start 3: DoubleHistogram
3/4 Test #3: DoubleHistogram ..................   Passed    0.03 sec
    Start 4: DoubleHistogramDataAccess
4/4 Test #4: DoubleHistogramDataAccess ........   Passed    0.22 sec

100% tests passed, 0 tests failed out of 4

armv7l:

mah@cubox-i:/next/home/mah/src/HdrHistogram_c$ uname -a
Linux cubox-i 3.14.14-cubox-i #6 SMP Fri Jun 5 20:43:50 CEST 2015 armv7l GNU/Linux
ah@cubox-i:/next/home/mah/src/HdrHistogram_c$ clang --version
Debian clang version 3.5.0-10 (tags/RELEASE_350/final) (based on LLVM 3.5.0)
Target: arm-unknown-linux-gnueabihf
Thread model: posix

mah@cubox-i:/next/home/mah/src/HdrHistogram_c$ make test
Running tests...
Test project /next/home/mah/src/HdrHistogram_c
    Start 1: Histogram
1/4 Test #1: Histogram ........................***Exception: SegFault  0.03 sec
    Start 2: HistogramLogging
2/4 Test #2: HistogramLogging .................***Exception: SegFault  0.01 sec
    Start 3: DoubleHistogram
3/4 Test #3: DoubleHistogram ..................***Failed    0.01 sec
    Start 4: DoubleHistogramDataAccess
4/4 Test #4: DoubleHistogramDataAccess ........***Exception: SegFault  0.01 sec

0% tests passed, 4 tests failed out of 4

Total Test time (real) =   0.11 sec

The following tests FAILED:
      1 - Histogram (SEGFAULT)
      2 - HistogramLogging (SEGFAULT)
      3 - DoubleHistogram (Failed)
      4 - DoubleHistogramDataAccess (SEGFAULT)
Errors while running CTest
Makefile:136: recipe for target 'test' failed
make: *** [test] Error 8

mah@cubox-i:/next/home/mah/src/HdrHistogram_c/test$ gdb hdr_dbl_histogram_data_access_test
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from hdr_dbl_histogram_data_access_test...done.
(gdb) r
Starting program: /next/home/mah/src/HdrHistogram_c/test/hdr_dbl_histogram_data_access_test

Program received signal SIGSEGV, Segmentation fault.
0x76fbdcb8 in hdr_dbl_record_values (h=0x0, value=1000, count=1) at /next/home/mah/src/HdrHistogram_c/src/hdr_dbl_histogram.c:254
254     if (value < h->current_lowest_value || h->current_highest_value <= value)
(gdb) backtrace
#0  0x76fbdcb8 in hdr_dbl_record_values (h=0x0, value=1000, count=1) at /next/home/mah/src/HdrHistogram_c/src/hdr_dbl_histogram.c:254
#1  0x76fbdfe0 in hdr_dbl_record_corrected_values (h=0x0, value=1000, count=1, expected_interval=10000) at /next/home/mah/src/HdrHistogram_c/src/hdr_dbl_histogram.c:275
#2  0x76fbdfa0 in hdr_dbl_record_corrected_value (h=0x0, value=1000, expected_interval=10000) at /next/home/mah/src/HdrHistogram_c/src/hdr_dbl_histogram.c:270
#3  0x0001114c in load_histograms () at /next/home/mah/src/HdrHistogram_c/test/hdr_dbl_histogram_data_access_test.c:47
#4  0x00011340 in test_scaling_equivalence () at /next/home/mah/src/HdrHistogram_c/test/hdr_dbl_histogram_data_access_test.c:64
#5  0x000124d8 in all_tests () at /next/home/mah/src/HdrHistogram_c/test/hdr_dbl_histogram_data_access_test.c:325
#6  0x0001242c in hdr_dbl_histogram_data_access_run_tests () at /next/home/mah/src/HdrHistogram_c/test/hdr_dbl_histogram_data_access_test.c:339
#7  0x000124b8 in main (argc=1, argv=0x7efff484) at /next/home/mah/src/HdrHistogram_c/test/hdr_dbl_histogram_data_access_test.c:355
(gdb)

mah@cubox-i:/next/home/mah/src/HdrHistogram_c/test$ ldd  hdr_dbl_histogram_data_access_test
    libhdr_histogram.so => /next/home/mah/src/HdrHistogram_c/src/libhdr_histogram.so (0x76fa8000)
    libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0x76f28000)
    libz.so.1 => /lib/arm-linux-gnueabihf/libz.so.1 (0x76f06000)
    libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0x76e18000)
    /lib/ld-linux-armhf.so.3 (0x76fc8000)

Double histograms

I'm curious why double histograms are 'not likely to be implemented'. No criticism just wonder if it's a technical issue, simply no personal need for them, a strong distaste for doubles (which I can totally sympathise with)? Would a PR to that end be accepted (can't guarantee I'll make one as my c is not strong, but I might try)?

SOVERSION is not to be related to library version

Please revert 5810143

SOVERSION is related to API version, not to library version.

Exemple how is manage in another library (libsodium)

On linux the "soname" (18 in this example) will be used for dependency

Expose (de-)compressing/(de|en)ncoding HDRs to strings

Hi

would it be possible to change hdr_histogram_log.h in such a way that we have a new API just for encoding/decoding into char strings?

char* hdr_log_encode(struct hdr_histogram *hdr);
struct hdr_histogram* hdr_log_decode(char *data);

This would allow saving the histograms to other locations than a FILE.

hdr_value_at_percentile top on-cpu hotspots by function/LOC

Given the following benchmark numbers, we see that hdr_value_at_percentile/* is one of the heaviest functions of the library.

./hdr_histogram_benchmark 
2021-08-28 22:52:24
Running ./hdr_histogram_benchmark
Run on (20 X 1482.92 MHz CPU s)
CPU Caches:
  L1 Data 32K (x20)
  L1 Instruction 32K (x20)
  L2 Unified 1024K (x20)
  L3 Unified 28160K (x1)
Load Average: 0.13, 0.04, 0.01
***WARNING*** Library was built as DEBUG. Timings may be affected.
------------------------------------------------------------------------------------------------
Benchmark                                                      Time             CPU   Iterations
------------------------------------------------------------------------------------------------
BM_hdr_init/3/86400000                                     17583 ns        17583 ns       100000
BM_hdr_init/3/86400000000                                  36707 ns        36706 ns        19055
BM_hdr_init/4/86400000                                     36847 ns        36847 ns        19021
BM_hdr_init/4/86400000000                                  36947 ns        36947 ns        18887
BM_hdr_record_values/3/86400000                             24.3 ns         24.3 ns     28515087
BM_hdr_record_values/3/86400000000                          24.4 ns         24.4 ns     28735715
BM_hdr_record_values/4/86400000                             24.4 ns         24.4 ns     28830019
BM_hdr_record_values/4/86400000000                          24.4 ns         24.4 ns     28822254
BM_hdr_value_at_percentile/3/86400000                     286814 ns       286828 ns         2421
BM_hdr_value_at_percentile/3/86400000000                  288858 ns       288864 ns         2416

If we profile specifically hdr_value_at_percentile as follows:

perf record -F 999 -g ./hdr_histogram_benchmark --benchmark_filter=BM_hdr_value_at_percentile/3/86400000 --benchmark_min_time=30

We see the following on CPU hotspots/top-consumers ( that we can latter use this info to follow up on performance improvement work ) :

Top cpu consumers by function/LOC above 1% self-cpu:

pprof -lines -text perf.data 
Converting perf.data to a profile.proto... (May take a few minutes)
[INFO:src/quipper/perf_reader.cc:1030] Number of events stored: 172859
[INFO:src/quipper/perf_parser.cc:267] Parser processed: 131 MMAP/MMAP2 events, 2 COMM events, 0 FORK events, 1 EXIT events, 172590 SAMPLE events, 4 of these were mapped, 0 SAMPLE events with a data address, 0 of these were mapped
[ERROR:src/quipper/perf_parser.cc:289] Only 0% of samples had all locations mapped to a module, expected at least 95%
[INFO:src/perf_data_handler.cc:87] Using the build id found for the file name: [kernel.kallsyms], build id: 879ddde2aadd26976803676a5135ab42d7688164.
[INFO:src/perf_data_handler.cc:590] Guessing main mapping for PID=10046 /root/HdrHistogram_c/build/test/hdr_histogram_benchmark
[WARNING:src/perf_data_handler.cc:463] stat: missing_callchain_mmap 345367/3344026
File: perf
Build ID: 879ddde2aadd26976803676a5135ab42d7688164
perf-version:4.15.18
perf-command:/usr/lib/linux-tools-4.15.0-147/perf record -F 999 -g ./hdr_histogram_benchmark --benchmark_filter=BM_hdr_value_at_percentile/3/86400000 --benchmark_min_time=30
Type: cycles:ppp_event
Showing nodes accounting for 576861504835, 87.04% of 662738682035 total
Dropped 258 nodes (cum <= 3313693410)
      flat  flat%   sum%        cum   cum%
51390453949  7.75%  7.75% 51405945119  7.76%  has_buckets /root/HdrHistogram_c/src/hdr_histogram.c:792
37168294596  5.61% 13.36% 37179852393  5.61%  move_next /root/HdrHistogram_c/src/hdr_histogram.c:802
26216181352  3.96% 17.32% 26220014646  3.96%  value_from_index /root/HdrHistogram_c/src/hdr_histogram.c:192
23060712812  3.48% 20.80% 43893377265  6.62%  get_bucket_index /root/HdrHistogram_c/src/hdr_histogram.c:170
19509836855  2.94% 23.74% 19513719484  2.94%  hdr_size_of_equivalent_value_range_given_bucket_indices /root/HdrHistogram_c/src/hdr_histogram.c:227
18064378004  2.73% 26.47% 18068264884  2.73%  get_sub_bucket_index /root/HdrHistogram_c/src/hdr_histogram.c:176
17531896849  2.65% 29.11% 39026359019  5.89%  move_next /root/HdrHistogram_c/src/hdr_histogram.c:813
17361982372  2.62% 31.73% 17361982372  2.62%  count_leading_zeros_64 /root/HdrHistogram_c/src/hdr_histogram.c:164
12895656233  1.95% 33.68% 12899514921  1.95%  hdr_value_at_percentile /root/HdrHistogram_c/src/hdr_histogram.c:678
12603946424  1.90% 35.58% 12603946424  1.90%  hdr_size_of_equivalent_value_range_given_bucket_indices /root/HdrHistogram_c/src/hdr_histogram.c:228
12485866655  1.88% 37.46% 12485866655  1.88%  move_next /root/HdrHistogram_c/src/hdr_histogram.c:810
12362452038  1.87% 39.33% 12366263551  1.87%  get_bucket_index /root/HdrHistogram_c/src/hdr_histogram.c:171
11540349962  1.74% 41.07% 45865828766  6.92%  move_next /root/HdrHistogram_c/src/hdr_histogram.c:814
11171573929  1.69% 42.76% 11175397732  1.69%  hdr_value_at_index /root/HdrHistogram_c/src/hdr_histogram.c:205
11153837161  1.68% 44.44% 66183406396  9.99%  move_next /root/HdrHistogram_c/src/hdr_histogram.c:804
11102618031  1.68% 46.11% 11102618031  1.68%  move_next /root/HdrHistogram_c/src/hdr_histogram.c:818
10503358837  1.58% 47.70% 30137564862  4.55%  counts_get_normalised /root/HdrHistogram_c/src/hdr_histogram.c:64
9773549590  1.47% 49.17% 61594200400  9.29%  move_next /root/HdrHistogram_c/src/hdr_histogram.c:811
9731510926  1.47% 50.64% 71707133450 10.82%  move_next /root/HdrHistogram_c/src/hdr_histogram.c:812
9453944093  1.43% 52.07% 45956113673  6.93%  move_next /root/HdrHistogram_c/src/hdr_histogram.c:815
8875060842  1.34% 53.41% 8875060842  1.34%  counts_get_direct /root/HdrHistogram_c/src/hdr_histogram.c:59
8657705093  1.31% 54.71% 27463877621  4.14%  lowest_equivalent_value_given_bucket_indices /root/HdrHistogram_c/src/hdr_histogram.c:240
8465572682  1.28% 55.99% 8465572682  1.28%  value_from_index /root/HdrHistogram_c/src/hdr_histogram.c:191
7999879534  1.21% 57.20% 51394769433  7.75%  move_next /root/HdrHistogram_c/src/hdr_histogram.c:809
7993782954  1.21% 58.41% 7993782954  1.21%  update_iterated_values /root/HdrHistogram_c/src/hdr_histogram.c:854
7871105272  1.19% 59.59% 19493789693  2.94%  std::generate_canonical /usr/include/c++/10/bits/random.tcc:3300
7483967008  1.13% 60.72% 22556555121  3.40%  hdr_value_at_index /root/HdrHistogram_c/src/hdr_histogram.c:214
7347073114  1.11% 61.83% 7347073114  1.11%  hdr_value_at_percentile /root/HdrHistogram_c/src/hdr_histogram.c:676
7302211125  1.10% 62.93% 7302211125  1.10%  counts_get_direct /root/HdrHistogram_c/src/hdr_histogram.c:58
7150209289  1.08% 64.01% 7150209289  1.08%  hdr_value_at_index /root/HdrHistogram_c/src/hdr_histogram.c:206
7085787068  1.07% 65.08% 7093473042  1.07%  update_iterated_values /root/HdrHistogram_c/src/hdr_histogram.c:855
7006355443  1.06% 66.14% 542953851888 81.93%  hdr_value_at_percentile /root/HdrHistogram_c/src/hdr_histogram.c:674
6800407849  1.03% 67.16% 6804248058  1.03%  lowest_equivalent_value_given_bucket_indices /root/HdrHistogram_c/src/hdr_histogram.c:239
6533015887  0.99% 68.15% 6533015887  0.99%  move_next /root/HdrHistogram_c/src/hdr_histogram.c:819

Top cpu consumers by function above 1% self-cpu:

pprof -text perf.data 
Converting perf.data to a profile.proto... (May take a few minutes)
[INFO:src/quipper/perf_reader.cc:1030] Number of events stored: 172859
[INFO:src/quipper/perf_parser.cc:267] Parser processed: 131 MMAP/MMAP2 events, 2 COMM events, 0 FORK events, 1 EXIT events, 172590 SAMPLE events, 4 of these were mapped, 0 SAMPLE events with a data address, 0 of these were mapped
[ERROR:src/quipper/perf_parser.cc:289] Only 0% of samples had all locations mapped to a module, expected at least 95%
[INFO:src/perf_data_handler.cc:87] Using the build id found for the file name: [kernel.kallsyms], build id: 879ddde2aadd26976803676a5135ab42d7688164.
[INFO:src/perf_data_handler.cc:590] Guessing main mapping for PID=10046 /root/HdrHistogram_c/build/test/hdr_histogram_benchmark
[WARNING:src/perf_data_handler.cc:463] stat: missing_callchain_mmap 345367/3344026
File: perf
Build ID: 879ddde2aadd26976803676a5135ab42d7688164
perf-version:4.15.18
perf-command:/usr/lib/linux-tools-4.15.0-147/perf record -F 999 -g ./hdr_histogram_benchmark --benchmark_filter=BM_hdr_value_at_percentile/3/86400000 --benchmark_min_time=30
Type: cycles:ppp_event
Showing nodes accounting for 652421958877, 98.44% of 662738682035 total
Dropped 70 nodes (cum <= 3313693410)
      flat  flat%   sum%        cum   cum%
157834710634 23.82% 23.82% 462392895397 69.77%  move_next
55539714244  8.38% 32.20% 55562962510  8.38%  has_buckets
45592184859  6.88% 39.08% 66440135692 10.03%  get_bucket_index
38889020202  5.87% 44.94% 53965432118  8.14%  hdr_value_at_index
37366306506  5.64% 50.58% 37370189135  5.64%  hdr_size_of_equivalent_value_range_given_bucket_indices
35538327487  5.36% 55.94% 35542160781  5.36%  value_from_index
27341468200  4.13% 60.07% 563361816834 85.01%  hdr_value_at_percentile
25088609287  3.79% 63.85% 38552816194  5.82%  std::generate_canonical
23944322762  3.61% 67.47% 23944322762  3.61%  count_leading_zeros_64
23654707932  3.57% 71.04% 520623309857 78.56%  all_values_iter_next
23528037951  3.55% 74.59% 23539557803  3.55%  update_iterated_values
22355795639  3.37% 77.96% 22359682519  3.37%  get_sub_bucket_index
18986930685  2.86% 80.83% 37796943422  5.70%  lowest_equivalent_value_given_bucket_indices
18740792483  2.83% 83.65% 18740792483  2.83%  counts_get_direct
17403900107  2.63% 86.28% 37038106132  5.59%  counts_get_normalised
13017285737  1.96% 88.24% 534323811997 80.62%  hdr_iter_next
12326335441  1.86% 90.10% 47311821700  7.14%  std::normal_distribution::operator()
11405931136  1.72% 91.82% 11405931136  1.72%  normalize_index
11184263084  1.69% 93.51% 79088813428 11.93%  std::gamma_distribution::operator()
10018604011  1.51% 95.02% 10022476509  1.51%  __ieee754_log_fma

Build failed on latest

diff --git a/CMakeLists.txt b/CMakeLists.txt
index e4327e5..b49707a 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -9,7 +9,7 @@ project("hdr_histogram")
 ENABLE_TESTING()

 if(UNIX)
-    set(CMAKE_C_FLAGS "-Wall -Wno-unknown-pragmas -Wextra -Wshadow -Winit-self -Wmissing-prototypes -D_GNU_SOURCE")
+    set(CMAKE_C_FLAGS "-std=gnu99 -Wall -Wno-unknown-pragmas -Wextra -Wshadow -Winit-self -Wmissing-prototypes -D_GNU_SOURCE")
     set(CMAKE_C_FLAGS_DEBUG "-O0 -g")
     set(CMAKE_C_FLAGS_RELEASE "-O3 -g")
 endif()

FreeBSD 13.2 compilation error.

Short description is that hdr_endian.h doesn't expand correctly on FreeBSD resulting in these kinds of errors:

In file included from /src/_build/default/lib/libhdr_histogram/src/hdr_histogram_log.c:33:
/src/_build/default/lib/libhdr_histogram/src/hdr_endian.h:51:10: warning: 'be16toh' macro redefined [-Wmacro-redefined]
#       define be16toh(x) betoh16(x)
               ^
/usr/include/sys/_endian.h:111:9: note: previous definition is here
#define be16toh(x)      __bswap16((x))
        ^
In file included from /src/_build/default/lib/libhdr_histogram/src/hdr_histogram_log.c:33:
/src/_build/default/lib/libhdr_histogram/src/hdr_endian.h:52:10: warning: 'le16toh' macro redefined [-Wmacro-redefined]
#       define le16toh(x) letoh16(x)
               ^
/usr/include/sys/_endian.h:114:9: note: previous definition is here
#define le16toh(x)      ((uint16_t)(x))
        ^
In file included from /src/_build/default/lib/libhdr_histogram/src/hdr_histogram_log.c:33:

Full compile error ocaml-multicore/hdr_histogram_ocaml#6 with fixes available on ocaml-multicore/hdr_histogram_ocaml#7

Make hdr_reset_internal_counters public

I am using this function to implement a (de-)serialization for histograms, but its only defined in hdr_test.h, with the comment that its private. This API is however used inside hdr_histogram.c, so its not actually used only in tests.

Could you make it public in hdr_histogram.h? See beberlei/hdrhistogram-php#8 for reference.

Add hdr_reset() on inactive_histogram when non-null

Currently, the use of hdr_interval_recorder_sample_and_recycle() seems to require the caller to take care of resetting the recycled histogram before handing it in. Since there is probably no correct way for the interval recorder to work unless about-to-become-active histograms are reset in every sample, it would be better to move the reset into the sample call, and relieve the caller from the responsibility of doing so.

This would simplify the common idiomatic use of hdr_interval_recorder_sample_and_recycle to simply:

struct hdr_histogram* histogram = NULL;
...
some_loop {
...
histogram = hdr_interval_recorder_sample_and_recycle(&recorder, histogram);
...
}

Th reset line in the hiccup.c example currently at https://github.com/HdrHistogram/HdrHistogram_c/blob/master/examples/hiccup.c#L183 could then be removed.

Cannot compile against hdr_histogram_log.h

The hdr_time.h is not exported, but its required by hdr_histogram_log.h.

getting:

/usr/local/include/hdr/hdr_histogram_log.h:28:22: fatal error: hdr_time.h: No such file or directory
 #include "hdr_time.h"

This error was recently introduced in dfac123

hdr_histogram_log_test.decode_v3_log fails on Alpine Linux (musl libc)

    Start 1: Histogram
1/5 Test #1: Histogram ........................   Passed    0.20 sec
    Start 2: HistogramAtomic
2/5 Test #2: HistogramAtomic ..................   Passed    0.30 sec
    Start 3: HistogramLogging
3/5 Test #3: HistogramLogging .................***Failed    0.02 sec
    Start 4: Atomic
4/5 Test #4: Atomic ...........................   Passed    0.00 sec
    Start 5: HistogramAtomicConcurrency
5/5 Test #5: HistogramAtomicConcurrency .......   Passed    0.84 sec

80% tests passed, 1 tests failed out of 5
$ ./hdr_histogram_log_test
I/O error
hdr_histogram_log_test.decode_v3_log(): Failed to read histogram
Tests run: 20
$ strace ./hdr_histogram_log_test
...
open("jHiccup-2.0.7S.logV3.hlog", O_RDONLY) = 3
mmap(NULL, 274432, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f876a30a000
read(3, "#[Logged with jHiccup version 2."..., 1024) = 1024
read(3, "MwMAgxwABTBDKT4GBgdnNYMcCBvsPEBE"..., 1024) = 1024
...
read(3, "ExAQA/uwsC\n55.134,0.997,0.426,HI"..., 1024) = 876
brk(0x55e4a5b2f000)                     = 0x55e4a5b2f000
brk(0x55e4a5b35000)                     = 0x55e4a5b35000
brk(0x55e4a5b3b000)                     = 0x55e4a5b3b000
brk(0x55e4a5b41000)                     = 0x55e4a5b41000
brk(0x55e4a5b47000)                     = 0x55e4a5b47000
brk(0x55e4a5b4d000)                     = 0x55e4a5b4d000
brk(0x55e4a5b53000)                     = 0x55e4a5b53000
read(3, "", 1024)                       = 0
ioctl(1, TIOCGWINSZ, {ws_row=57, ws_col=211, ws_xpixel=0, ws_ypixel=0}) = 0
writev(1, [{iov_base="I/O error", iov_len=9}, {iov_base="\n", iov_len=1}], 2I/O error
) = 10
writev(1, [{iov_base="hdr_histogram_log_test.decode_v3"..., iov_len=64}, {iov_base="\n", iov_len=1}], 2hdr_histogram_log_test.decode_v3_log(): Failed to read histogram
) = 65

  • HdrHistogram_c 0.9.13
  • Alpine Linux Edge / x86_64
  • gcc 9.3.0
  • musl 1.1.24

Memory corruption in hdr_encode_compressed()

diff that fixes an under allocation of memory, causing memory corruption when the encoded varint string ends up being longer than the allocated length - the size of the header (which is the case when encoding an empty histogram or a histogram with very little content).

diff --git a/src/hdr_histogram_log.c b/src/hdr_histogram_log.c
index a795e85..c52268e 100644
--- a/src/hdr_histogram_log.c
+++ b/src/hdr_histogram_log.c
@@ -232,7 +232,7 @@ int hdr_encode_compressed(
     int32_t len_to_max = counts_index_for(h, h->max_value) + 1;
     int32_t counts_limit = len_to_max < h->counts_len ? len_to_max : h->counts_len;

-    if ((encoded = (_encoding_flyweight_v1*) calloc(MAX_BYTES_LEB128 * (size_t) counts_limit, sizeof(uint8_t))) == NULL)
+    if ((encoded = (_encoding_flyweight_v1*) calloc(sizeof(_encoding_flyweight_v1) + MAX_BYTES_LEB128 * (size_t) counts_limit, sizeof(uint8_t))) == NULL)
     {
         FAIL_AND_CLEANUP(cleanup, result, ENOMEM);
     }
diff --git a/test/hdr_histogram_log_test.c b/test/hdr_histogram_log_test.c
index ba1fa4b..d665754 100644
--- a/test/hdr_histogram_log_test.c
+++ b/test/hdr_histogram_log_test.c
@@ -617,6 +617,23 @@ static char* log_reader_fails_with_incorrect_version()
     return 0;
 }

+static char* test_encode_decode_empty()
+{
+    struct hdr_histogram *histogram, *hdr_new = NULL;
+    hdr_alloc(INT64_C(3600) * 1000 * 1000, 3, &histogram);
+
+    char *data;
+
+    mu_assert("Failed to encode histogram data", hdr_log_encode(histogram, &data) == 0);
+    mu_assert("Failed to decode histogram data", hdr_log_decode(&hdr_new, data, strlen(data)) == 0);
+    mu_assert("Histograms should be the same", compare_histogram(histogram, hdr_new));
+    // mu_assert("Mean different after encode/decode", compare_double(hdr_mean(histogram), hdr_mean(hdr_new), 0.001));
+    free(histogram);
+    free(hdr_new);
+    free(data);
+    return 0;
+}
+
 static char* test_string_encode_decode()
 {
     struct hdr_histogram *histogram, *hdr_new = NULL;
@@ -774,6 +791,7 @@ static struct mu_result all_tests()
 {
     tests_run = 0;

+    mu_run_test(test_encode_decode_empty);
     mu_run_test(test_encode_and_decode_compressed);
     mu_run_test(test_encode_and_decode_compressed2);
     mu_run_test(test_encode_and_decode_compressed_large);

Note that the assert on the mean value had to be commented out because it fails (to be investigated).
There are still many memory leaks while running the test code. I only fixed the ones related to the test function I added.

If you are interested, here is the list for hdr_histogram_log_test:

$ valgrind  --leak-check=full ./hdr_histogram_log_test
==83407== Memcheck, a memory error detector
==83407== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==83407== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==83407== Command: ./hdr_histogram_log_test
==83407==
--83407-- run: /usr/bin/dsymutil "./hdr_histogram_log_test"
--83407-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option
--83407-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 2 times)
--83407-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 4 times)
ALL TESTS PASSED
Tests run: 21
==83407==
==83407== HEAP SUMMARY:
==83407==     in use at exit: 1,674,419 bytes in 469 blocks
==83407==   total heap usage: 2,360 allocs, 1,891 frees, 33,272,520 bytes allocated
==83407==
==83407== 18 bytes in 1 blocks are definitely lost in loss record 13 of 124
==83407==    at 0x10000F877: calloc (vg_replace_malloc.c:715)
==83407==    by 0x100003991: assert_base64_decode (hdr_histogram_log_test.c:407)
==83407==    by 0x10000390A: base64_decode_decodes_strings_without_padding (hdr_histogram_log_test.c:416)
==83407==    by 0x1000025C6: all_tests (hdr_histogram_log_test.c:803)
==83407==    by 0x1000021DC: hdr_histogram_log_run_tests (hdr_histogram_log_test.c:829)
==83407==    by 0x10000227A: main (in ./hdr_histogram_log_test)
==83407==
==83407== 21 bytes in 1 blocks are definitely lost in loss record 14 of 124
==83407==    at 0x10000F877: calloc (vg_replace_malloc.c:715)
==83407==    by 0x100003991: assert_base64_decode (hdr_histogram_log_test.c:407)
==83407==    by 0x100003A0A: base64_decode_decodes_strings_with_padding (hdr_histogram_log_test.c:427)
==83407==    by 0x100002636: all_tests (hdr_histogram_log_test.c:804)
==83407==    by 0x1000021DC: hdr_histogram_log_run_tests (hdr_histogram_log_test.c:829)
==83407==    by 0x10000227A: main (in ./hdr_histogram_log_test)
==83407==
==83407== 21 bytes in 1 blocks are definitely lost in loss record 15 of 124
==83407==    at 0x10000F877: calloc (vg_replace_malloc.c:715)
==83407==    by 0x100003991: assert_base64_decode (hdr_histogram_log_test.c:407)
==83407==    by 0x100003A3F: base64_decode_decodes_strings_with_padding (hdr_histogram_log_test.c:433)
==83407==    by 0x100002636: all_tests (hdr_histogram_log_test.c:804)
==83407==    by 0x1000021DC: hdr_histogram_log_run_tests (hdr_histogram_log_test.c:829)
==83407==    by 0x10000227A: main (in ./hdr_histogram_log_test)
==83407==
==83407== 70 bytes in 1 blocks are definitely lost in loss record 62 of 124
==83407==    at 0x10000EEA1: malloc (vg_replace_malloc.c:303)
==83407==    by 0x10001D7C9: hdr_encode_compressed (hdr_histogram_log.c:279)
==83407==    by 0x100002F97: test_encode_and_decode_compressed (hdr_histogram_log_test.c:205)
==83407==    by 0x100002304: all_tests (hdr_histogram_log_test.c:795)
==83407==    by 0x1000021DC: hdr_histogram_log_run_tests (hdr_histogram_log_test.c:829)
==83407==    by 0x10000227A: main (in ./hdr_histogram_log_test)
==83407==
==83407== 264 bytes in 1 blocks are definitely lost in loss record 80 of 124
==83407==    at 0x10000F877: calloc (vg_replace_malloc.c:715)
==83407==    by 0x100003536: test_encode_and_decode_base64 (hdr_histogram_log_test.c:285)
==83407==    by 0x100002415: all_tests (hdr_histogram_log_test.c:798)
==83407==    by 0x1000021DC: hdr_histogram_log_run_tests (hdr_histogram_log_test.c:829)
==83407==    by 0x10000227A: main (in ./hdr_histogram_log_test)
==83407==
==83407== 273 bytes in 1 blocks are definitely lost in loss record 81 of 124
==83407==    at 0x10000F877: calloc (vg_replace_malloc.c:715)
==83407==    by 0x10001EE1F: hdr_log_encode (hdr_histogram_log.c:1067)
==83407==    by 0x10000481E: test_string_encode_decode (hdr_histogram_log_test.c:649)
==83407==    by 0x1000029B6: all_tests (hdr_histogram_log_test.c:815)
==83407==    by 0x1000021DC: hdr_histogram_log_run_tests (hdr_histogram_log_test.c:829)
==83407==    by 0x10000227A: main (in ./hdr_histogram_log_test)
==83407==
==83407== 353 bytes in 1 blocks are definitely lost in loss record 83 of 124
==83407==    at 0x10000F877: calloc (vg_replace_malloc.c:715)
==83407==    by 0x100003522: test_encode_and_decode_base64 (hdr_histogram_log_test.c:284)
==83407==    by 0x100002415: all_tests (hdr_histogram_log_test.c:798)
==83407==    by 0x1000021DC: hdr_histogram_log_run_tests (hdr_histogram_log_test.c:829)
==83407==    by 0x10000227A: main (in ./hdr_histogram_log_test)
==83407==
==83407== 4,096 bytes in 1 blocks are definitely lost in loss record 109 of 124
==83407==    at 0x10000EEA1: malloc (vg_replace_malloc.c:303)
==83407==    by 0x1001FE696: __smakebuf (in /usr/lib/system/libsystem_c.dylib)
==83407==    by 0x100201CF9: __srefill0 (in /usr/lib/system/libsystem_c.dylib)
==83407==    by 0x100201DF4: __srefill (in /usr/lib/system/libsystem_c.dylib)
==83407==    by 0x100201EC7: __srget (in /usr/lib/system/libsystem_c.dylib)
==83407==    by 0x1001FB3B3: fgetc (in /usr/lib/system/libsystem_c.dylib)
==83407==    by 0x10001E889: hdr_log_read_header (hdr_histogram_log.c:915)
==83407==    by 0x1000049BF: decode_v2_log (hdr_histogram_log_test.c:719)
==83407==    by 0x100002A26: all_tests (hdr_histogram_log_test.c:817)
==83407==    by 0x1000021DC: hdr_histogram_log_run_tests (hdr_histogram_log_test.c:829)
==83407==    by 0x10000227A: main (in ./hdr_histogram_log_test)
==83407==
==83407== 4,096 bytes in 1 blocks are definitely lost in loss record 110 of 124
==83407==    at 0x10000EEA1: malloc (vg_replace_malloc.c:303)
==83407==    by 0x1001FE696: __smakebuf (in /usr/lib/system/libsystem_c.dylib)
==83407==    by 0x100201CF9: __srefill0 (in /usr/lib/system/libsystem_c.dylib)
==83407==    by 0x100201DF4: __srefill (in /usr/lib/system/libsystem_c.dylib)
==83407==    by 0x100201EC7: __srget (in /usr/lib/system/libsystem_c.dylib)
==83407==    by 0x1001FB3B3: fgetc (in /usr/lib/system/libsystem_c.dylib)
==83407==    by 0x10001E889: hdr_log_read_header (hdr_histogram_log.c:915)
==83407==    by 0x100004C9F: decode_v1_log (hdr_histogram_log_test.c:674)
==83407==    by 0x100002A96: all_tests (hdr_histogram_log_test.c:818)
==83407==    by 0x1000021DC: hdr_histogram_log_run_tests (hdr_histogram_log_test.c:829)
==83407==    by 0x10000227A: main (in ./hdr_histogram_log_test)
==83407==
==83407== 4,096 bytes in 1 blocks are definitely lost in loss record 111 of 124
==83407==    at 0x10000EEA1: malloc (vg_replace_malloc.c:303)
==83407==    by 0x1001FE696: __smakebuf (in /usr/lib/system/libsystem_c.dylib)
==83407==    by 0x100201CF9: __srefill0 (in /usr/lib/system/libsystem_c.dylib)
==83407==    by 0x100201DF4: __srefill (in /usr/lib/system/libsystem_c.dylib)
==83407==    by 0x100201EC7: __srget (in /usr/lib/system/libsystem_c.dylib)
==83407==    by 0x1001FB3B3: fgetc (in /usr/lib/system/libsystem_c.dylib)
==83407==    by 0x10001E889: hdr_log_read_header (hdr_histogram_log.c:915)
==83407==    by 0x100004F7F: decode_v0_log (hdr_histogram_log_test.c:763)
==83407==    by 0x100002B06: all_tests (hdr_histogram_log_test.c:819)
==83407==    by 0x1000021DC: hdr_histogram_log_run_tests (hdr_histogram_log_test.c:829)
==83407==    by 0x10000227A: main (in ./hdr_histogram_log_test)
==83407==
==83407== 5,344 bytes in 1 blocks are definitely lost in loss record 112 of 124
==83407==    at 0x10000EEA1: malloc (vg_replace_malloc.c:303)
==83407==    by 0x10001D7C9: hdr_encode_compressed (hdr_histogram_log.c:279)
==83407==    by 0x1000031F7: test_encode_and_decode_compressed2 (hdr_histogram_log_test.c:232)
==83407==    by 0x10000235F: all_tests (hdr_histogram_log_test.c:796)
==83407==    by 0x1000021DC: hdr_histogram_log_run_tests (hdr_histogram_log_test.c:829)
==83407==    by 0x10000227A: main (in ./hdr_histogram_log_test)
==83407==
==83407== 5,344 bytes in 1 blocks are definitely lost in loss record 113 of 124
==83407==    at 0x10000EEA1: malloc (vg_replace_malloc.c:303)
==83407==    by 0x10001D7C9: hdr_encode_compressed (hdr_histogram_log.c:279)
==83407==    by 0x1000034C7: test_encode_and_decode_base64 (hdr_histogram_log_test.c:279)
==83407==    by 0x100002415: all_tests (hdr_histogram_log_test.c:798)
==83407==    by 0x1000021DC: hdr_histogram_log_run_tests (hdr_histogram_log_test.c:829)
==83407==    by 0x10000227A: main (in ./hdr_histogram_log_test)
==83407==
==83407== 5,344 bytes in 1 blocks are definitely lost in loss record 114 of 124
==83407==    at 0x10000EEA1: malloc (vg_replace_malloc.c:303)
==83407==    by 0x10001D7C9: hdr_encode_compressed (hdr_histogram_log.c:279)
==83407==    by 0x1000035F7: test_bounds_check_on_decode (hdr_histogram_log_test.c:259)
==83407==    by 0x100002476: all_tests (hdr_histogram_log_test.c:799)
==83407==    by 0x1000021DC: hdr_histogram_log_run_tests (hdr_histogram_log_test.c:829)
==83407==    by 0x10000227A: main (in ./hdr_histogram_log_test)
==83407==
==83407== 14,356 bytes in 1 blocks are definitely lost in loss record 117 of 124
==83407==    at 0x10000EEA1: malloc (vg_replace_malloc.c:303)
==83407==    by 0x10001D7C9: hdr_encode_compressed (hdr_histogram_log.c:279)
==83407==    by 0x100003395: test_encode_and_decode_compressed_large (hdr_histogram_log_test.c:312)
==83407==    by 0x1000023BA: all_tests (hdr_histogram_log_test.c:797)
==83407==    by 0x1000021DC: hdr_histogram_log_run_tests (hdr_histogram_log_test.c:829)
==83407==    by 0x10000227A: main (in ./hdr_histogram_log_test)
==83407==
==83407== 188,512 bytes in 1 blocks are definitely lost in loss record 118 of 124
==83407==    at 0x10000EEA1: malloc (vg_replace_malloc.c:303)
==83407==    by 0x10001B9D6: hdr_init (hdr_histogram.c:319)
==83407==    by 0x10001E20D: hdr_decode_compressed_v2 (hdr_histogram_log.c:642)
==83407==    by 0x10001D9A7: hdr_decode_compressed (hdr_histogram_log.c:719)
==83407==    by 0x10001ECE8: hdr_log_read (hdr_histogram_log.c:1034)
==83407==    by 0x100003F76: writes_and_reads_log (hdr_histogram_log_test.c:499)
==83407==    by 0x100002866: all_tests (hdr_histogram_log_test.c:811)
==83407==    by 0x1000021DC: hdr_histogram_log_run_tests (hdr_histogram_log_test.c:829)
==83407==    by 0x10000227A: main (in ./hdr_histogram_log_test)
==83407==
==83407== 188,512 bytes in 1 blocks are definitely lost in loss record 119 of 124
==83407==    at 0x10000EEA1: malloc (vg_replace_malloc.c:303)
==83407==    by 0x10001B9D6: hdr_init (hdr_histogram.c:319)
==83407==    by 0x10001E20D: hdr_decode_compressed_v2 (hdr_histogram_log.c:642)
==83407==    by 0x10001D9A7: hdr_decode_compressed (hdr_histogram_log.c:719)
==83407==    by 0x10001ECE8: hdr_log_read (hdr_histogram_log.c:1034)
==83407==    by 0x100004022: writes_and_reads_log (hdr_histogram_log_test.c:508)
==83407==    by 0x100002866: all_tests (hdr_histogram_log_test.c:811)
==83407==    by 0x1000021DC: hdr_histogram_log_run_tests (hdr_histogram_log_test.c:829)
==83407==    by 0x10000227A: main (in ./hdr_histogram_log_test)
==83407==
==83407== 188,512 bytes in 1 blocks are definitely lost in loss record 120 of 124
==83407==    at 0x10000EEA1: malloc (vg_replace_malloc.c:303)
==83407==    by 0x10001B9D6: hdr_init (hdr_histogram.c:319)
==83407==    by 0x10001BA74: hdr_alloc (in /openstack/HdrHistogram_c/src/libhdr_histogram.dylib)
==83407==    by 0x1000047D4: test_string_encode_decode (hdr_histogram_log_test.c:640)
==83407==    by 0x1000029B6: all_tests (hdr_histogram_log_test.c:815)
==83407==    by 0x1000021DC: hdr_histogram_log_run_tests (hdr_histogram_log_test.c:829)
==83407==    by 0x10000227A: main (in ./hdr_histogram_log_test)
==83407==
==83407== 188,512 bytes in 1 blocks are definitely lost in loss record 121 of 124
==83407==    at 0x10000EEA1: malloc (vg_replace_malloc.c:303)
==83407==    by 0x10001B9D6: hdr_init (hdr_histogram.c:319)
==83407==    by 0x10001E20D: hdr_decode_compressed_v2 (hdr_histogram_log.c:642)
==83407==    by 0x10001D9A7: hdr_decode_compressed (hdr_histogram_log.c:719)
==83407==    by 0x10001EF23: hdr_log_decode (hdr_histogram_log.c:1102)
==83407==    by 0x10000486F: test_string_encode_decode (hdr_histogram_log_test.c:650)
==83407==    by 0x1000029B6: all_tests (hdr_histogram_log_test.c:815)
==83407==    by 0x1000021DC: hdr_histogram_log_run_tests (hdr_histogram_log_test.c:829)
==83407==    by 0x10000227A: main (in ./hdr_histogram_log_test)
==83407==
==83407== 270,432 bytes in 1 blocks are definitely lost in loss record 122 of 124
==83407==    at 0x10000EEA1: malloc (vg_replace_malloc.c:303)
==83407==    by 0x10001B9D6: hdr_init (hdr_histogram.c:319)
==83407==    by 0x10000499B: decode_v2_log (hdr_histogram_log_test.c:710)
==83407==    by 0x100002A26: all_tests (hdr_histogram_log_test.c:817)
==83407==    by 0x1000021DC: hdr_histogram_log_run_tests (hdr_histogram_log_test.c:829)
==83407==    by 0x10000227A: main (in ./hdr_histogram_log_test)
==83407==
==83407== 270,432 bytes in 1 blocks are definitely lost in loss record 123 of 124
==83407==    at 0x10000EEA1: malloc (vg_replace_malloc.c:303)
==83407==    by 0x10001B9D6: hdr_init (hdr_histogram.c:319)
==83407==    by 0x100004C7B: decode_v1_log (hdr_histogram_log_test.c:665)
==83407==    by 0x100002A96: all_tests (hdr_histogram_log_test.c:818)
==83407==    by 0x1000021DC: hdr_histogram_log_run_tests (hdr_histogram_log_test.c:829)
==83407==    by 0x10000227A: main (in ./hdr_histogram_log_test)
==83407==
==83407== 270,432 bytes in 1 blocks are definitely lost in loss record 124 of 124
==83407==    at 0x10000EEA1: malloc (vg_replace_malloc.c:303)
==83407==    by 0x10001B9D6: hdr_init (hdr_histogram.c:319)
==83407==    by 0x100004F5B: decode_v0_log (hdr_histogram_log_test.c:754)
==83407==    by 0x100002B06: all_tests (hdr_histogram_log_test.c:819)
==83407==    by 0x1000021DC: hdr_histogram_log_run_tests (hdr_histogram_log_test.c:829)
==83407==    by 0x10000227A: main (in ./hdr_histogram_log_test)
==83407==
==83407== LEAK SUMMARY:
==83407==    definitely lost: 1,609,040 bytes in 21 blocks
==83407==    indirectly lost: 0 bytes in 0 blocks
==83407==      possibly lost: 0 bytes in 0 blocks
==83407==    still reachable: 26,876 bytes in 29 blocks
==83407==         suppressed: 38,503 bytes in 419 blocks
==83407== Reachable blocks (those to which a pointer was found) are not shown.
==83407== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==83407==
==83407== For counts of detected and suppressed errors, rerun with: -v
==83407== ERROR SUMMARY: 21 errors from 21 contexts (suppressed: 19 from 19)

Please explain tagging strategy

Hello,

I am working on Fedora RPM package and I would like to confirm the tagging strategy. I see two sets of tags, one without prefix and the other with HdrHistogram_c prefix. Which one out of these two is the right one?

I will be downloading releases directly from the github via the tag then.

Thanks!

hdr_add( ) causes memory corruption?

Hi developers, I have been using hdr_histogram c version for a while. It's very convenient and powerful. Thank you for the great work.

Meanwhile I'm concerned with thread safety. I have several threads adding values to hdr_histogram at the same time, and they appear to work fine without locking. But I want to confirm with the developers.

I also call hdr_add( ) to merge several hdr_histograms into one. But this method seems to cause memory overrun and my program crashes. Below is the backtrace produced by address sanitizer. You can see that the hdr_add( ) method call leads to crash.

ASAN:SIGSEGV
=================================================================
==68521==ERROR: AddressSanitizer: SEGV on unknown address 0x626c0157a960 (pc 0x00000169d041 bp 0x7fbb9b281e20 sp 0x7fbb9b281e10 T78)
    #0 0x169d040 in counts_get_direct /home/hcd/shawn/code/hstore/daelaam/common/lib/hdr_histogram/src/hdr_histogram.c:50
    #1 0x169d0ac in counts_get_normalised /home/hcd/shawn/code/hstore/daelaam/common/lib/hdr_histogram/src/hdr_histogram.c:55
    #2 0x16a027d in move_next /home/hcd/shawn/code/hstore/daelaam/common/lib/hdr_histogram/src/hdr_histogram.c:617
    #3 0x16a0783 in _basic_iter_next /home/hcd/shawn/code/hstore/daelaam/common/lib/hdr_histogram/src/hdr_histogram.c:651
    #4 0x16a13f9 in _recorded_iter_next /home/hcd/shawn/code/hstore/daelaam/common/lib/hdr_histogram/src/hdr_histogram.c:798
    #5 0x16a0b11 in hdr_iter_next /home/hcd/shawn/code/hstore/daelaam/common/lib/hdr_histogram/src/hdr_histogram.c:691
    #6 0x169f203 in hdr_add /home/hcd/shawn/code/hstore/daelaam/common/lib/hdr_histogram/src/hdr_histogram.c:436
    #7 0x1683335 in hcd::HcdHistogram::Merge(hcd::HcdHistogram*) /home/hcd/shawn/code/hstore/daelaam/common/lib/hcdcommon/src/hcd_histogram.cpp:152
    #8 0x1334750 in hcd::LatencyBreakdown::Merge(hcd::LatencyBreakdown*) /home/hcd/shawn/code/hstore/daelaam/hcdstruct/src/latency_recorder.cpp:96
    #9 0x1335ea8 in hcd::LatencyRecorder::Merge(hcd::LatencyBreakdown*, hcd::HTaskType) /home/hcd/shawn/code/hstore/daelaam/hcdstruct/src/latency_recorder.cpp:179
    #10 0x133650c in hcd::LatencyRecorder::Merge(hcd::LatencyRecorder*) /home/hcd/shawn/code/hstore/daelaam/hcdstruct/src/latency_recorder.cpp:194
    #11 0xc28592 in hcd::StatsPrinter::PrinterLoop() /home/hcd/shawn/code/hstore/daelaam/threadpool/src/stats_printer.cpp:55
    #12 0xc327f3 in void std::_Mem_fn_base<void (hcd::StatsPrinter::*)(), true>::operator()<, void>(hcd::StatsPrinter*) const /usr/include/c++/5/functional:600
    #13 0xc32682 in void std::_Bind_simple<std::_Mem_fn<void (hcd::StatsPrinter::*)()> (hcd::StatsPrinter*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /usr/include/c++/5/functional:1531
    #14 0xc323af in std::_Bind_simple<std::_Mem_fn<void (hcd::StatsPrinter::*)()> (hcd::StatsPrinter*)>::operator()() /usr/include/c++/5/functional:1520
    #15 0xc322cd in std::thread::_Impl<std::_Bind_simple<std::_Mem_fn<void (hcd::StatsPrinter::*)()> (hcd::StatsPrinter*)> >::_M_run() /usr/include/c++/5/thread:115
    #16 0x7fbbc93bfc7f  (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xb8c7f)
    #17 0x7fbbc98926b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
    #18 0x7fbbc8b2541c in clone (/lib/x86_64-linux-gnu/libc.so.6+0x10741c)

segfault on decode

When performing a hdr_log_decode() on a moderately sized histogram output with hdr_log_encode I hit a segfault later in a malloc/free call.

Tracked it to an out of bounds overwrite in the counts array because the limit wasn't being set properly.

The attached patch fixes it for me (note that I've only tested the v2 encoding but the v1 code looks to be similarly broken).

Broken histogram after decoding when setting values outside range

I am having a very weird problem, when I use "hdr_record_value(s)" with a value outside the range (see example using i+1). The iterator for the decoded version of this HDR will print nothing and the program will exit.

When I remove the "+1" inside the loop everything works fine.

I came accross this bug in my PHP extension when setting a value outside range would make "hdrhistogram_c" hang in what seems an infinite loop on iteration of the decoded hdr (High CPU).

Sorry that I could not find a better reproduce case for this yet:

#include <stdbool.h>
#include <string.h>
#include <inttypes.h>
#include <hdr_histogram.h>
#include <hdr_histogram_log.h>

int main(int argc, char **argv)
{
    struct hdr_histogram *histogram, *new = NULL;
    struct hdr_iter iterator;
    hdr_init(1, 1000 * 60, 3, &histogram);

    for (int i = 1; i < (1000 * 60); i++) {
        hdr_record_values(histogram, i+1, ((1000*60)-i) / 10);
    }

    char *data;

    if (hdr_log_encode(histogram, &data) != 0) {
        printf("error");
        return 1;
    }

    if (hdr_log_decode(&new, data, strlen(data)) != 0) {
        printf("error2");
        return 2;
    }

    hdr_iter_init(&iterator, new);

    while (hdr_iter_next(&iterator) == true) {
        printf("%"PRIu64": %"PRIu64"\n", iterator.value_from_index, iterator.count_at_index);
    }
}

%d.%d printf and sscanf formatting of timestamps is wrong

The printf formatting of timestamps (meant to be in units of seconds with three decimal points of resolution, so down to msec) will result in msec parts between 0 and 99 showing up as wrong numbers.

See:
https://github.com/HdrHistogram/HdrHistogram_c/blob/master/src/hdr_histogram_log.c#L774
https://github.com/HdrHistogram/HdrHistogram_c/blob/master/src/hdr_histogram_log.c#L847

Similarly, sscanf formatting is susceptible to a similar problem (although correctly formatted log file timestamps are likely to actually include all three decimal places, log files produced by the current version will not. E.g. 1.083 will be printed as 1.83, but scanned back as 1.083, making tests miss the issue).
See:
https://github.com/HdrHistogram/HdrHistogram_c/blob/master/src/hdr_histogram_log.c#L883
https://github.com/HdrHistogram/HdrHistogram_c/blob/master/src/hdr_histogram_log.c#L889
https://github.com/HdrHistogram/HdrHistogram_c/blob/master/src/hdr_histogram_log.c#L1048
https://github.com/HdrHistogram/HdrHistogram_c/blob/master/src/hdr_histogram_log.c#L1049

Should probably just use %.3f formatting for both printf and scanf, and convert sec.msec internal representations to/from a float.

HDR count between 2 values

Hello, thanks a lot for this lib, very great work. I would like to know if there is a better way to get the number of counts between 2 values other than iterating on them one by one and adding the result of
hdr_count_at_value(histogram,i)
Is there a method hdr_count_interval(histogram,i,j)

hdr_thread.h not installed

In hdr_writer_reader_phaser.h which is installed

 #include "hdr_thread.h"

So build of an application using this library fails because of missing header

AIX Support

Hi All,

Will it support AIX 6/7 in the future?

Sincerely,
ARIEN

potential memory leak

the potential leak is that the following sequence will not free successfully allocated memory if only one of the allocations fails.

    counts = calloc((size_t) cfg.counts_len, sizeof(int64_t));
    histogram = calloc(1, sizeof(struct hdr_histogram));

    if (!counts || !histogram)
    {
        return ENOMEM;
    }

it's not likely to be serious in most cases because they will usually both succeed.

Broken percentile computation

Built a simple test case to reproduce.

#include <stdlib.h>
#include <stdio.h>
#include <assert.h>
#include "hdr_histogram.h"

int main() {
    int measurements[] = {
        90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100,
        90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100
    };
    struct {
        int lowest;
        int highest;
    } trackable_values[] = {
        { 80, 110 },
        { 1, 1000 },
        { 1, 100000 }
    };

    for(size_t tv = 0; tv < sizeof(trackable_values)/sizeof(trackable_values[0]); tv++) {
        struct hdr_histogram *h = 0;
        printf("=== Test %d: hdr_init(%d, %d)\n", (int)tv+1,
                trackable_values[tv].lowest, trackable_values[tv].highest);

        /* Initialize histogram library */
        int ret = hdr_init(trackable_values[tv].lowest,
                           trackable_values[tv].highest, 5, &h);
        assert(ret == 0);
        assert(h);

        /* Fill with values. */
        printf("Values:");        for(size_t i=0; i < sizeof(measurements)/sizeof(measurements[0]); i++) {
            printf(" %d", measurements[i]);
            hdr_record_value(h, measurements[i]);
        }

        printf("\nPercentiles = %d/%d (50%%/90%%)\n",
            (int)hdr_value_at_percentile(h, 0.50),
            (int)hdr_value_at_percentile(h, 0.90)
        );
        hdr_percentiles_print(h, stdout, 2, 1, CLASSIC);
    }

    return 0;
}

Expected hdr_value_at_percentile) to give some values inside the 90..100 range. Meanwhile, the function returns weird output (like hdr_value_at_percentile(, 0.5) == 127), see the attached log:

=== Test 1: hdr_init(80, 110)
Values: 90 91 92 93 94 95 96 97 98 99 100 90 91 92 93 94 95 96 97 98 99 100
Percentiles = 127/127 (50%/90%)
       Value   Percentile   TotalCount 1/(1-Percentile)

   127.00000     0.000000           22         1.00
   127.00000     1.000000           22          inf
#[Mean    =       96.000, StdDeviation   =        0.000]
#[Max     =      127.000, Total count    =           22]
#[Buckets =            1, SubBuckets     =       262144]
=== Test 2: hdr_init(1, 1000)
Values: 90 91 92 93 94 95 96 97 98 99 100 90 91 92 93 94 95 96 97 98 99 100
Percentiles = 90/90 (50%/90%)
       Value   Percentile   TotalCount 1/(1-Percentile)

    90.00000     0.000000            2         1.00
    92.00000     0.250000            6         1.33
    95.00000     0.500000           12         2.00
    96.00000     0.625000           14         2.67
    98.00000     0.750000           18         4.00
    98.00000     0.812500           18         5.33
    99.00000     0.875000           20         8.00
    99.00000     0.906250           20        10.67
   100.00000     0.937500           22        16.00
   100.00000     1.000000           22          inf
#[Mean    =       95.000, StdDeviation   =        3.162]
#[Max     =      100.000, Total count    =           22]
#[Buckets =            1, SubBuckets     =       262144]
=== Test 3: hdr_init(1, 100000)
Values: 90 91 92 93 94 95 96 97 98 99 100 90 91 92 93 94 95 96 97 98 99 100
Percentiles = 90/90 (50%/90%)
       Value   Percentile   TotalCount 1/(1-Percentile)

    90.00000     0.000000            2         1.00
    92.00000     0.250000            6         1.33
    95.00000     0.500000           12         2.00
    96.00000     0.625000           14         2.67
    98.00000     0.750000           18         4.00
    98.00000     0.812500           18         5.33
    99.00000     0.875000           20         8.00
    99.00000     0.906250           20        10.67
   100.00000     0.937500           22        16.00
   100.00000     1.000000           22          inf
#[Mean    =       95.000, StdDeviation   =        3.162]
#[Max     =      100.000, Total count    =           22]
#[Buckets =            1, SubBuckets     =       262144]

hdr_record_value does not actually check value > highest_trackable_value

I noticed this when using a hdr_interval_recorder. I tried to add a value larger than highest_trackable_value and hdr_interval_recorder_record_value returned true.

I dug deeper and realized it was a problem with the check the underlying hdr_record_values does.

Instead of directly checking highest_trackable_value, it checks whether the value falls into a valid bucket via

int32_t counts_index = counts_index_for(h, value);
...
...
if (counts_index < 0 || h->counts_len <= counts_index)

h->counts_len reflects the total number of buckets the histogram has:

    cfg->counts_len = (cfg->bucket_count + 1) * (cfg->sub_bucket_count / 2);

However, the sub_bucket_count is derived from the significant_figures input:

int64_t largest_value_with_single_unit_resolution = 2 * power(10, significant_figures);
    int32_t sub_bucket_count_magnitude = (int32_t) ceil(log((double)largest_value_with_single_unit_resolution) / log(2));
    cfg->sub_bucket_half_count_magnitude = ((sub_bucket_count_magnitude > 1) ? sub_bucket_count_magnitude : 1) - 1;

    cfg->unit_magnitude = (int32_t) floor(log((double)lowest_trackable_value) / log(2));

    cfg->sub_bucket_count      = (int32_t) pow(2, (cfg->sub_bucket_half_count_magnitude + 1));

and bucket_count is only the minimum multiple of sub_bucket_counts needed to include highest_trackable_value.

Given this, it is quite possible to add values that are larger than highest_trackable_value to a histogram.

Easy to reproduce in the following way:

struct hdr_histogram * hist;
hdr_init(1, 50, 5, &hist);
bool result = hdr_record_value(hist, 500);

If you instrument hdr_record_value with some debug statements, you'll see the following:

Bucket index: 0
sub bucket index: 500
counts_index: 500
counts_len: 262144

It's possible this is only a factor when significant_figures is larger than the number of digits highest_trackable_value has. But as far as I understood the documentation there is no requirement that significant_figures be dependent on the range.

Am I reading the documentation incorrectly? Should I be modulating significant_figures based on what I set for highest_trackable_value? Or is there any reason to not directly check against highest_trackable_value? Especially since that is what the header file seems to imply is being checked?

linear iteration goes into infinite loop on recording new values

During the course of iteration if a new value gets inserted in a bucket that the iterator has already passed then the iteration never ends (hdr_iter_next continues to return true).

I made a quick change to one of the test cases to reproduce the issue.
I am not sure if this is proper usage of the api, but an infinite loop seems to be a very bad outcome.

static char* test_linear_iter_buckets_correctly()
{
    struct hdr_histogram *h;
    hdr_init(1, 255, 2, &h);

    hdr_record_value(h, 193);
    hdr_record_value(h, 255);
    hdr_record_value(h, 0);
    hdr_record_value(h, 1);
    hdr_record_value(h, 64);
    hdr_record_value(h, 128);

    struct hdr_iter iter;
    hdr_iter_linear_init(&iter, h, 64);

    int step_count = 0;
    int64_t total_count = 0;
    while (hdr_iter_next(&iter))
    {
        total_count += iter.specifics.linear.count_added_in_this_iteration_step;
        // start - changes to reproduce issue
        if (step_count == 0)
        {
            hdr_record_value(h, 2);
        }
        // end - changes to reproduce issue
        step_count++;
    }

    mu_assert("Number of steps", compare_int64(4, step_count));
    mu_assert("Total count", compare_int64(6, total_count));

    return 0;
}

In our application the issue was caused due to unsynchronized access from multiple threads. One thread was computing the hdr_value_at_percentile which another threaded called hdr_record_value (guessing this). The hdr_value_at_percentile call went into an infinite loop.

0x0081bf8e: has_next + 0x1e (7fff27779da0, 7f113380f040, 7fff27779de8, 1007fff27779ef0)
0x0081bf35: _basic_iter_next + 0x15 (81bf20, 7fff27779de8)
0x0081ba05: hdr_iter_next + 0x25 (7fff27779df0, 3635503b6, 4058c00000000000, 19, 26c17d008, 3635503b6) + 80
0x0081bb84: hdr_value_at_percentile + 0x114 (3fe, 4bb, 5a0, 709, c57, 196bff) + ffffff12161d1650

Build errors

Linux with gcc 10.1.0:

**/home/travis/build/chronoxor/CppBenchmark/modules/HdrHistogram/src/hdr_histogram_log.c:35: error: ignoring ‘#pragma clang diagnostic’ [-Werror=unknown-pragmas]
   35 | #pragma clang diagnostic push
      | 
/home/travis/build/chronoxor/CppBenchmark/modules/HdrHistogram/src/hdr_histogram_log.c:36: error: ignoring ‘#pragma ide diagnostic’ [-Werror=unknown-pragmas]
   36 | #pragma ide diagnostic ignored "readability-redundant-declaration"
      | 
/home/travis/build/chronoxor/CppBenchmark/modules/HdrHistogram/src/hdr_histogram_log.c:39: error: ignoring ‘#pragma clang diagnostic’ [-Werror=unknown-pragmas]
   39 | #pragma clang diagnostic pop
      | 
cc1: all warnings being treated as errors**

Linux with clang 10.0.1:

/home/travis/build/chronoxor/CppBenchmark/modules/HdrHistogram/src/hdr_histogram_log.c:36:9: error: unknown pragma ignored [-Werror,-Wunknown-pragmas]
#pragma ide diagnostic ignored "readability-redundant-declaration"
        ^
1 error generated.
modules/CMakeFiles/HdrHistogram.dir/build.make:88: recipe for target 'modules/CMakeFiles/HdrHistogram.dir/HdrHistogram/src/hdr_histogram_log.c.o' failed

OSX with AppleClang 11.0.3.11030032:

/Users/travis/build/chronoxor/CppBenchmark/modules/HdrHistogram/src/hdr_histogram_log.c:36:9: error: unknown pragma ignored [-Werror,-Wunknown-pragmas]
#pragma ide diagnostic ignored "readability-redundant-declaration"
        ^
1 error generated.

ARM (Raspberry PI) build

I'm trying to build this project for ARM but I'm having an issue. Please bear with me as I'm very rusty when it comes to C/C++ builds.

During the cmake --build . process I get the following error:

[ 50%] Linking C executable hdr_histogram_test
/usr/bin/ld: ../src/libhdr_histogram_static.a(hdr_histogram.c.o): in function `counts_inc_normalised_atomic':
hdr_histogram.c:(.text+0x204): undefined reference to `__atomic_fetch_add_8'
/usr/bin/ld: hdr_histogram.c:(.text+0x220): undefined reference to `__atomic_fetch_add_8'
/usr/bin/ld: ../src/libhdr_histogram_static.a(hdr_histogram.c.o): in function `update_min_max_atomic':
hdr_histogram.c:(.text+0x2dc): undefined reference to `__atomic_load_8'
/usr/bin/ld: hdr_histogram.c:(.text+0x32c): undefined reference to `__atomic_compare_exchange_8'
/usr/bin/ld: hdr_histogram.c:(.text+0x354): undefined reference to `__atomic_load_8'
/usr/bin/ld: hdr_histogram.c:(.text+0x398): undefined reference to `__atomic_compare_exchange_8'
/usr/bin/ld: ../src/libhdr_histogram_static.a(hdr_writer_reader_phaser.c.o): in function `_hdr_phaser_get_epoch':
hdr_writer_reader_phaser.c:(.text+0x18): undefined reference to `__atomic_load_8'
/usr/bin/ld: ../src/libhdr_histogram_static.a(hdr_writer_reader_phaser.c.o): in function `_hdr_phaser_set_epoch':
hdr_writer_reader_phaser.c:(.text+0x58): undefined reference to `__atomic_store_8'
/usr/bin/ld: ../src/libhdr_histogram_static.a(hdr_writer_reader_phaser.c.o): in function `_hdr_phaser_reset_epoch':
hdr_writer_reader_phaser.c:(.text+0x8c): undefined reference to `__atomic_exchange_8'
/usr/bin/ld: ../src/libhdr_histogram_static.a(hdr_writer_reader_phaser.c.o): in function `hdr_phaser_writer_enter':
hdr_writer_reader_phaser.c:(.text+0x1c0): undefined reference to `__atomic_fetch_add_8'
/usr/bin/ld: ../src/libhdr_histogram_static.a(hdr_writer_reader_phaser.c.o): in function `hdr_phaser_writer_exit':
hdr_writer_reader_phaser.c:(.text+0x238): undefined reference to `__atomic_fetch_add_8'
collect2: error: ld returned 1 exit status
gmake[2]: *** [test/CMakeFiles/hdr_histogram_test.dir/build.make:120: test/hdr_histogram_test] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:1007: test/CMakeFiles/hdr_histogram_test.dir/all] Error 2
gmake: *** [Makefile:182: all] Error 2

Now I went ahead, deleted my "build" folder and re-did it all like so:

LDFLAGS="-latomic" cmake ../HdrHistogram_c/
LDFLAGS="-latomic" cmake --build . -v

Got same error, but this time I can see this (thanks to "-v" argument):

/usr/bin/cc -latomic CMakeFiles/hdr_histogram_test.dir/hdr_histogram_test.c.o CMakeFiles/hdr_histogram_test.dir/minunit.c.o -o hdr_histogram_test  ../src/libhdr_histogram_static.a /usr/lib/arm-linux-gnueabihf/libz.so -lpthread -lm -lrt 

Now that won't work because -latomic is like first argument here.
But this works just fine, with -latomic at the end:

/usr/bin/cc CMakeFiles/hdr_histogram_test.dir/hdr_histogram_test.c.o CMakeFiles/hdr_histogram_test.dir/minunit.c.o -o hdr_histogram_test  ../src/libhdr_histogram_static.a /usr/lib/arm-linux-gnueabihf/libz.so -lpthread -lm -lrt  -latomic

What am I doing wrong? Anyone managed to build this project for ARM (Raspberry PI)?

Thanks!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.