plasma-umass / coz Goto Github PK

Coz: Causal Profiling

License: Other

C 96.72% C++ 2.28% Python 0.06% HTML 0.05% CSS 0.03% TypeScript 0.27% JavaScript 0.37% Rust 0.07% CMake 0.12% Shell 0.01% Makefile 0.04%

profiler optimization causal-inference performance-analysis performance-engineering

coz's People

Contributors

Stargazers

Watchers

Forkers

emaxerrno skbly7 bpowers jonathan-beard flandr fedorova pranavcode emeryberger yodamaster mpacer mitsutaka-takeda edwarddavidbaker sayounara jvilk petterreinholdtsen roger2000hk llvilanova mchang6137 mdmdj pradeepambati byte-lab utsasrg vitaut quadewarren 4144 behpn behnampn wcohen evilmcjerkface wyk9787 jam7 beyonddream-productions longjohncoder emperorearth arthmis igordzreyev cartazio alexcrichton sangohan misterzeus calinou danielcavalli sattishv coderesearchref nystrom hunger xd009642 arnidagur mohabouje camchenry martinvl tetzank blockspacer karllolee wenchaochen yibit slumduck yishengcheng jingmouren fetchadd zuiwanting tristan-potter winstic vshalts realkotob amedeedaboville contosworks sahwar antoyo curiousleo stefanfabian ksitko lsabc davidtran641 zeta1999 acolita drahnr naiah dvirtz gsumalo kramermt zicklag katharostech renecannao animesh code-in-cpp jordanvrtanoski nazpyro firedtoad ykg manuelmeraz-forks justanotherdot klimkin karatesnowmachine icodein blt bnelson619 stefanbuettner krummelur kurtkilpela

coz's Issues

Add support for larger code fragments

Source lines may be a bit too fine-grained to generate useful profiles on large applications. Basic blocks (or something like them) would be best. Functions are always possible for exported symbols.

Set up build testing

With merge requests #63 and #67, it’d be great if build testing could be set up to make sure that the code builds fine with GCC and Clang, and the tests pass (there is just one at the moment).

Travis CI has a GitHub integration so could be used for that.

Symbol resolution error when mixing coz/clang++ and libelfin/g++

I'm using upstream with clang++ 1:3.6-33 (debian's 1:3.6-33) and g++ 6.1.1 (debian's 4:6.1.1-1).

After compiling libelfin and coz, here's what I get:

$ ./coz run --- benchmarks/matrix_multiply/matrix_multiply 100 20
[libcoz.cpp:116] Including MAIN, which is /home/lluis/Projects/coz/benchmarks/matrix_multiply/matrix_multiply
[inspect.cpp:325] /lib/x86_64-linux-gnu/ld-2.23.so is not in scope
[inspect.cpp:325] /home/lluis/Projects/coz/libcoz/libcoz.so is not in scope
[inspect.cpp:325] /lib/x86_64-linux-gnu/libdl-2.23.so is not in scope
benchmarks/matrix_multiply/matrix_multiply: symbol lookup error: /home/lluis/Projects/coz/libcoz/libcoz.so: undefined symbol: _ZNK5dwarf5value9as_stringEv

If I look at the symbol name, there's two versions of the as_string() method without arguments, one of them undefined:

$ readelf -sW libcoz/libcoz.so | grep as_string
     3: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND _ZNK5dwarf5value9as_stringEv
   247: 0000000000026400    60 FUNC    GLOBAL DEFAULT   13 _ZNK5dwarf5value9as_stringERNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
   733: 0000000000026440    56 FUNC    GLOBAL DEFAULT   13 _ZNK5dwarf5value9as_stringB5cxx11Ev
   298: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND _ZNK5dwarf5value9as_stringEv
   581: 0000000000026400    60 FUNC    GLOBAL DEFAULT   13 _ZNK5dwarf5value9as_stringERNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
   930: 0000000000026440    56 FUNC    GLOBAL DEFAULT   13 _ZNK5dwarf5value9as_stringB5cxx11Ev
$ nm -C libcoz/libcoz.so  | grep as_string
0000000000026440 T dwarf::value::as_string[abi:cxx11]() const
0000000000026400 T dwarf::value::as_string(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) const
                 U dwarf::value::as_string() const

If I compile coz with g++, the mangled name for the "old-style" method symbol is not referenced anymore (there's no reference without the abi:cxx11 when we look at demangled names):

$ readelf -sW libcoz/libcoz.so | grep as_string
   250: 0000000000024960    60 FUNC    GLOBAL DEFAULT   12 _ZNK5dwarf5value9as_stringERNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
   731: 00000000000249a0    56 FUNC    GLOBAL DEFAULT   12 _ZNK5dwarf5value9as_stringB5cxx11Ev
   460: 0000000000024960    60 FUNC    GLOBAL DEFAULT   12 _ZNK5dwarf5value9as_stringERNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
   807: 00000000000249a0    56 FUNC    GLOBAL DEFAULT   12 _ZNK5dwarf5value9as_stringB5cxx11Ev
$ nm -C libcoz/libcoz.so | grep as_string
00000000000249a0 T dwarf::value::as_string[abi:cxx11]() const
0000000000024960 T dwarf::value::as_string(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) const

References to the undefined symbol appear only in memory_map::process_inlines:

$ objdump -D /home/lluis/Projects/coz/libcoz/libcoz.so >a
# search in ./a which function references the missing symbol
$ c++filt _ZN10memory_map15process_inlinesERKN5dwarf3dieERKNS0_10line_tableERKSt13unordered_setINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt4hashISD_ESt8equal_toISD_ESaISD_EEm
memory_map::process_inlines(dwarf::die const&, dwarf::line_table const&, std::unordered_set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, unsigned long)

This corresponds to

coz/libcoz/inspect.cpp

Line 377 in 5cd4021

name = name_val.as_string();

. The file is always compiled with "-std=c++11", so it should always use the same name mangling algorithm to reference the symbol.

This might be a bug in CLang, but I don't have a proof of it.

Add support for multi-process applications.

Is it possible to profile background processes?

I use distributed file system.
After I know this tool, I wonder if I'm able to profile background process of distributed file system, using this tool.
I want to profile the background process which handles I/O, when some I/O requests are issued.

Could this tool be used for the above use case?

Thank you

Look for libcoz.so in standard library locations

The coz Python wrapper sets LD_PRELOAD to be the directory name of where it is located. In system installs, this will not be the case. The Python executable will be in /usr/bin, whereas libcoz.so will be in /usr/lib. It would be better if LD_PRELOAD was left empty if it is passed in as empty.

Add options to control minimum plot size

It can be hard to see the trend in the plots in the coz plotter as the y axis is quite squashed. It would be better if each of the plots were given slightly more y-pixels to increase this resolution.

support for other (interpreted) languages not mentioned in README

Sorry if this is a stupid question, but it didn't seem to be addressed by the README.md or video talk - can coz be used with programs written in other languages, especially interpreted ones such Ruby or Python? I would expect this to be a FAQ, so it's probably worth documenting somewhere obvious. Thanks!

Kernel Profiling

Add support for profiling in the kernel. This requires loading the kernel image and locating symbols. Sampling will already pause in the kernel, so this should be relatively easy.

Errors in `make bench` for kmeans benchmark

Running make bench for the kmeans benchmark in a loop, I occasionally get errors from libcoz:

Aborted!
  0: /usr/share/coz/libcoz.so(_ZN8profiler8on_errorEiP9siginfo_tPv+0x7e) [0x7f16bc49d73e]
  1: /usr/lib/libpthread.so.0(+0x10d60) [0x7f16bc252d60]
  2: /usr/lib/libc.so.6(gsignal+0x38) [0x7f16bb4375f8]
  3: /usr/lib/libc.so.6(abort+0x16a) [0x7f16bb438a7a]
  4: /usr/share/coz/libcoz.so(+0x46ec9) [0x7f16bc4a5ec9]
  5: /usr/share/coz/libcoz.so(+0x403a5) [0x7f16bc49f3a5]
  6: /usr/share/coz/libcoz.so(+0x433e4) [0x7f16bc4a23e4]
  7: /usr/share/coz/libcoz.so(_ZN8profiler12start_threadEPv+0x15c) [0x7f16bc49eeac]
  8: /usr/lib/libpthread.so.0(+0x74a4) [0x7f16bc2494a4]
  9: /usr/lib/libc.so.6(clone+0x6d) [0x7f16bb4ed13d]
make: *** [bench] Error 2

Segmentation fault at (nil)
  0: /usr/share/coz/libcoz.so(_ZN8profiler8on_errorEiP9siginfo_tPv+0x7e) [0x7f39ad18573e]
  1: /usr/lib/libpthread.so.0(+0x10d60) [0x7f39acf3ad60]
  2: /usr/lib/libpthread.so.0(+0x127e0) [0x7f39acf3c7e0]
  3: /usr/share/coz/libcoz.so(pthread_mutex_unlock+0x2c) [0x7f39ad18a5bc]
  4: /usr/lib/libgcc_s.so.1(_Unwind_Find_FDE+0x164) [0x7f39ac4a2084]
  5: /usr/lib/libgcc_s.so.1(+0xea86) [0x7f39ac49ea86]
  6: /usr/lib/libgcc_s.so.1(+0xffeb) [0x7f39ac49ffeb]
  7: /usr/lib/libgcc_s.so.1(_Unwind_ForcedUnwind+0x64) [0x7f39ac4a0324]
  8: /usr/lib/libpthread.so.0(__pthread_unwind+0x40) [0x7f39acf39b00]
  9: /usr/lib/libpthread.so.0(+0x85e5) [0x7f39acf325e5]
  10: /usr/share/coz/libcoz.so(+0x403a5) [0x7f39ad1873a5]
  11: /usr/share/coz/libcoz.so(+0x433e4) [0x7f39ad18a3e4]
  12: /usr/share/coz/libcoz.so(_ZN8profiler12start_threadEPv+0x15c) [0x7f39ad186eac]
  13: /usr/lib/libpthread.so.0(+0x74a4) [0x7f39acf314a4]
  14: /usr/lib/libc.so.6(clone+0x6d) [0x7f39ac1d513d]
make: *** [bench] Error 2

Coz does not support OSX

This project has amazing potential as an additional tool for Xcode, which uses clang for compiling iOS and Mac apps. However, Xcode can only be run on Apple operating systems. Is there any plan (or guide) to getting this code to work on Windows/Mac? So far my attempts have not been successful.

"Thread state not found"

Hi!

I'm trying to run coz on my program, and it's failing after a little while with the following error. Maybe it's related to the process started by coz run forking a child process and then exiting (the coz error is coming from the child process)?

./profiler.h:111 Thread state not found
Aborted!
0: /usr/local/bin/libcoz/libcoz.so(_ZN8profiler8on_errorEiP7siginfoPv+0x7e) [0x7ff2655ae57e]
1: /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7ff264b00cb0]
2: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7ff262b480d5]
3: /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7ff262b4b83b]
4: /usr/local/bin/libcoz/libcoz.so(_ZN8profiler21handle_pthread_createEPmPK14pthread_attr_tPFPvS4_ES4_+0x14f) [0x7ff2655aafcf]
5: /usr/local/bin/libcoz/libcoz.so(pthread_create+0x72) [0x7ff2655a8ec2]

$ uname -a
Linux 3.13.0-32-generic #57~precise1-Ubuntu SMP Tue Jul 15 03:51:20 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Show warning when no debug symbols are found

I just discovered that there is no warning or message of any kind from coz when trying to instrument a binary without debug symbols included. I tested coz on a debian package (ffmpeg2theora) and could not get any information despite the fact that I had added COZ_PROGRESS to the source. It suddenly occured to me that this might be caused by the fact that the build was done without debug information, and a quick rebuild later I managed to get a profile out of coz.

Perhaps coz should report a warning or abort execution if it is unable to find debug symbols?

Write an abstract

Compiler complain about check of non-null sig in sigwait()

When building the source I see this compiler working:

libcoz.cpp:439:10: warning: nonnull parameter 'sig' will evaluate to 'true' on first
      encounter [-Wpointer-bool-conversion]
      if(sig) *sig = result;
      ~~ ^~~

It is the only warning in the code. The reason for the warning is this definition in /usr/include/signal.h:

/* Select any of pending signals from SET or wait for any to arrive.

   This function is a cancellation point and therefore not marked with
   __THROW.  */
extern int sigwait (const sigset_t *__restrict __set, int *__restrict __sig)
     __nonnull ((1, 2));

Why is the sig argument checked for NULL in the replacement implementation of sigwait()?
As far as I can see from the POSIX / Open Group definition sig is not supposed to be NULL, see http://pubs.opengroup.org/onlinepubs/009695399/functions/sigwait.html .

I suspect the code should look like this instead:


diff --git a/libcoz/libcoz.cpp b/libcoz/libcoz.cpp
index 83f4c51..db72cfb 100644
--- a/libcoz/libcoz.cpp
+++ b/libcoz/libcoz.cpp
@@ -436,7 +436,7 @@ extern "C" {
       return errno;
     } else {
       // If the sig pointer is not null, pass the received signal to the caller
-      if(sig) *sig = result;
+      *sig = result;
       return 0;
     }
   }

Experience report

Do some performance tuning on a real application (TBD) using Causal.

Run profiled application in a child process

Sample from a parent process instead of embedded in the profiled application.

Add copyright / license statements to the source files.

Would you be willing to add copyright / license statements to the source files? It would make it easier for packages reviewing the license status of the package and make it easier to make the package available in Debian.

It could look something like the comment below:

/*
 * Copyright (c) 2015 Charlie Curtsinger and Emery Berger
 * University of Massachusetts Amherst
 * All rights reserved.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions are met:
 *
 * 1. Redistributions of source code must retain the above copyright notice, this
 * list of conditions and the following disclaimer.
 *
 * 2. Redistributions in binary form must reproduce the above copyright notice,
 * this list of conditions and the following disclaimer in the documentation
 * and/or other materials provided with the distribution.
 *
 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"  AND
 * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
 * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
 * DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
 * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
 * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
 * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
 * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
 * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 */

Choose real applications

Test and find progress points for some real applications. Candidates include:

Firefox (or v8, or Chrome)
MySQL/Postgres (latency and throughput)
A game (frame rate as progress point)
Blender/POVRay
Handbrake/Lame

What else?

Profiling output contains no data

Some of the provided examples work correctly, but my own attempts using both COZ_PROGRESS and COZ_BEGIN/COZ_END produce output like the following.

startup time=1444935306391924102
shutdown        time=1444935317172626273        samples=522
runtime time=10787882229

This is a small number of samples, but I've tried longer runs with ~4k samples but the same result. Any suggestions on how I might resolve this?

"[./profiler.h:111] Thread state not found" when trying to profile openalpr

To test coz-profiler, I decided to try to profile the openalpr command line tool, available from http://www.openalpr.com/ . It is a image processing system written in C++, used for extracting car license plates from images and video.

I modified the code like this to try benchmark how quickly it could process images.

diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index 95d4e18..a970c39 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -110,7 +110,7 @@ IF (WIN32)
 ELSE()

        SET (Extra_LIBS
-               pthread
+               pthread dl
        )

 ENDIF()
diff --git a/src/main.cpp b/src/main.cpp
index 190072f..f32d923 100644
--- a/src/main.cpp
+++ b/src/main.cpp
@@ -22,6 +22,7 @@
 #include <iostream>
 #include <iterator>
 #include <algorithm>
+#include <coz.h>

 #include "opencv2/highgui/highgui.hpp"
 #include "opencv2/imgproc/imgproc.hpp"
@@ -360,7 +361,7 @@ bool detectandshow( Alpr* alpr, cv::Mat frame, std::string region, bool writeJso
     }
   }

-
+  COZ_PROGRESS;

   return results.plates.size() > 0;
 }

After building, I ran it with a random MP4 vidoe file I had lying around after generating a simple openalpr.conf file:

coz run --- ./alpr --config ../openalpr.conf ~/tmp/input.mp4

Instead of getting a useful profiling run done, I got the error message "[./profiler.h:111] Thread state not found" instead. Am I using coz the wrong way?

Looking at the code, I suspect the problem is that openalpr is bootstraping its threads the wrong way, but I am unable to pinpoint exactly what is wrong. Also, I suspect coz should handle any code, and thus decided to report the problem as an issue.

Update implementation

coz_plot: Better-looking graphs

I wanted coz_plot to produce graphs like Figure 2 in the paper, but it's ugly by default. :(

Please fix.

Please make profile viewer work while offline

I was offline for the weekend and spent some time working on coz. But when I tried to use the profile viewer implemented in javascript from the gh-pages branch, it did not work.

I tracked this down to the fact that the HTML code is not self contained, but depend on code fragments downloaded from the Internet.

These seem to be the problematic dependencies:

% grep https:/ index.htm 
    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.3.5/css/bootstrap.min.css">
    <link rel="stylesheet" type="text/css" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.3.0/css/font-awesome.min.css">
    <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/2.1.4/jquery.min.js"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.3.5/js/bootstrap.min.js"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/d3-tip/0.6.3/d3-tip.min.js"></script>
%

Please rewrite the web page to use local copies of these javascript and css files, to ensure the profiler work also without Internet access.

Exclude STL template-generated code from scope by default

When profiling openalpr, the profile.coz file (named profile.txt in the attachment to keep github editor happy) show that a lot of time is spent in locations in /usr/include/c++/4.9/bits/stl_vector.h and /usr/include/opencv2/core/mat.hpp (matrix operations). This is not really helpful without knowing from where the STL vector or the matrix operation is used. Perhaps Coz should be adjusted to collect a short backtrace to locate from where the generic library functions are called?

profile.txt

why cannot see result plot?

I run coz run --- "some command" in your benchmarks , and get profile.coz file, but when I upload profile.coz to http://plasma-umass.github.io/coz/ , I can not see result plot, your Sample Profiles is ok, but upload files is not.

And is there any way to view a result plot local?

Thank you.

Script benchmark execution

Port over the evaluation code from the comparison branch. Should be extensible to support Parsec, SPEC, etc.

CAUSAL_BEGIN and CAUSAL_END not documented in README.md

No simple way to run benchmarks with system coz.

I've built a package for coz for Arch Linux, so I can now run coz simply by calling coz on the command line. However, the benchmark scripts all run $(ROOT)/coz, meaning I cannot easily test my system install. If $(ROOT) had included the forward slash, I might be able to do make ROOT= bench, but this will fail because of the include of common.mk. Might you consider including ../../common.mk directly in the benchmark Makefiles, and then using $(ROOT) only to locate the executable?

Fix integer overflow in plotting interface

The new profiler interface at http://plasma-umass.github.io/coz cannot plot the memcached results because the sum over progress point visits overflows JavaScript integers.

Builds issues on linux 3.16.0

Hi folks, interesting project. I wanted to try coz out on a profiling task, but I haven't been able to get it to build. I'm using the latest clang and libc++ pulled from trunk, and the headers from the ubuntu package linux-headers-3.16.0-33{,-generic}. Unfortunately, the hw_breakpoint.h and perf_event.h header files are aggressively C++-hostile:

$ /opt/llvm/bin/clang++ --std=c++11 -g -O2 -fPIC -I../include -I../deps -I../deps -I/usr/src/linux-headers-3.16.0-33/arch/x86/include -I/usr/src/linux-headers-3.16.0-33/include -g -O2 -stdlib=libc++ -MMD -MP -o obj/perf.o -c perf.cpp
In file included from perf.cpp:1:
In file included from ./perf.h:4:
In file included from /usr/src/linux-headers-3.16.0-33/include/linux/hw_breakpoint.h:4:
In file included from /usr/src/linux-headers-3.16.0-33/include/linux/perf_event.h:17:
In file included from /usr/src/linux-headers-3.16.0-33/include/uapi/linux/perf_event.h:17:
In file included from /usr/src/linux-headers-3.16.0-33/include/linux/types.h:5:
In file included from /usr/src/linux-headers-3.16.0-33/include/uapi/linux/types.h:13:
In file included from /usr/include/linux/posix_types.h:4:
/usr/src/linux-headers-3.16.0-33/include/linux/stddef.h:11:2: error: expected identifier
        false   = 0,
        ^
/usr/src/linux-headers-3.16.0-33/include/linux/stddef.h:12:2: error: expected identifier
        true    = 1
        ^
In file included from perf.cpp:1:
In file included from ./perf.h:4:
In file included from /usr/src/linux-headers-3.16.0-33/include/linux/hw_breakpoint.h:4:
In file included from /usr/src/linux-headers-3.16.0-33/include/linux/perf_event.h:17:
In file included from /usr/src/linux-headers-3.16.0-33/include/uapi/linux/perf_event.h:17:
/usr/src/linux-headers-3.16.0-33/include/linux/types.h:29:9: error: unknown type name '_Bool'
typedef _Bool                   bool;
        ^
/usr/src/linux-headers-3.16.0-33/include/linux/types.h:29:17: error: expected unqualified-id
typedef _Bool                   bool;
                                ^
In file included from perf.cpp:1:
In file included from ./perf.h:4:
In file included from /usr/src/linux-headers-3.16.0-33/include/linux/hw_breakpoint.h:4:
In file included from /usr/src/linux-headers-3.16.0-33/include/linux/perf_event.h:38:
/usr/src/linux-headers-3.16.0-33/include/linux/list.h:37:49: error: expected ')'
static inline void __list_add(struct list_head *new,
                                                ^
/usr/src/linux-headers-3.16.0-33/include/linux/list.h:37:30: note: to match this '('
static inline void __list_add(struct list_head *new,
                             ^
/usr/src/linux-headers-3.16.0-33/include/linux/list.h:41:2: error: use of undeclared identifier 'next'
        next->prev = new;
        ^
/usr/src/linux-headers-3.16.0-33/include/linux/list.h:41:18: error: expected a type
        next->prev = new;
                        ^
/usr/src/linux-headers-3.16.0-33/include/linux/list.h:42:5: error: expected a type
        new->next = next;
           ^
/usr/src/linux-headers-3.16.0-33/include/linux/list.h:43:5: error: expected a type
        new->prev = prev;
           ^
/usr/src/linux-headers-3.16.0-33/include/linux/list.h:44:2: error: use of undeclared identifier 'prev'
        prev->next = new;
        ^
/usr/src/linux-headers-3.16.0-33/include/linux/list.h:44:18: error: expected a type
        prev->next = new;
                        ^
/usr/src/linux-headers-3.16.0-33/include/linux/list.h:60:47: error: expected ')'
static inline void list_add(struct list_head *new, struct list_head *head)
                                              ^
/usr/src/linux-headers-3.16.0-33/include/linux/list.h:60:28: note: to match this '('
static inline void list_add(struct list_head *new, struct list_head *head)
                           ^
/usr/src/linux-headers-3.16.0-33/include/linux/list.h:62:16: error: expected a type
        __list_add(new, head, head->next);
                      ^
/usr/src/linux-headers-3.16.0-33/include/linux/list.h:74:52: error: expected ')'
static inline void list_add_tail(struct list_head *new, struct list_head *head)
                                                   ^
/usr/src/linux-headers-3.16.0-33/include/linux/list.h:74:33: note: to match this '('
static inline void list_add_tail(struct list_head *new, struct list_head *head)
                                ^
/usr/src/linux-headers-3.16.0-33/include/linux/list.h:76:16: error: expected a type
        __list_add(new, head->prev, head);
                      ^
/usr/src/linux-headers-3.16.0-33/include/linux/list.h:107:16: error: arithmetic on a pointer to void
        entry->next = LIST_POISON1;
                      ^~~~~~~~~~~~
/usr/src/linux-headers-3.16.0-33/include/linux/poison.h:22:44: note: expanded from macro 'LIST_POISON1'
#define LIST_POISON1  ((void *) 0x00100100 + POISON_POINTER_DELTA)
                       ~~~~~~~~~~~~~~~~~~~ ^
In file included from perf.cpp:1:
In file included from ./perf.h:4:
In file included from /usr/src/linux-headers-3.16.0-33/include/linux/hw_breakpoint.h:4:
In file included from /usr/src/linux-headers-3.16.0-33/include/linux/perf_event.h:38:
/usr/src/linux-headers-3.16.0-33/include/linux/list.h:108:16: error: arithmetic on a pointer to void
        entry->prev = LIST_POISON2;
                      ^~~~~~~~~~~~
/usr/src/linux-headers-3.16.0-33/include/linux/poison.h:23:44: note: expanded from macro 'LIST_POISON2'
#define LIST_POISON2  ((void *) 0x00200200 + POISON_POINTER_DELTA)
                       ~~~~~~~~~~~~~~~~~~~ ^
In file included from perf.cpp:1:
In file included from ./perf.h:4:
In file included from /usr/src/linux-headers-3.16.0-33/include/linux/hw_breakpoint.h:4:
In file included from /usr/src/linux-headers-3.16.0-33/include/linux/perf_event.h:38:
/usr/src/linux-headers-3.16.0-33/include/linux/list.h:123:23: error: expected ')'
                                struct list_head *new)
                                                  ^
/usr/src/linux-headers-3.16.0-33/include/linux/list.h:122:32: note: to match this '('
static inline void list_replace(struct list_head *old,
                               ^
/usr/src/linux-headers-3.16.0-33/include/linux/list.h:125:5: error: expected a type
        new->next = old->next;
           ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.

I took a stab at working around this by #defining away the keywords such as true/false/new/bool, but there are still issues with pointer incompatibilty, literal suffixes, etc. In short it seems the linux headers really aren't meant to be compiled by a C++ compiler.

Could you share the version of the linux kernel headers you have been using?

Add `cleandeps` target to rebuild dependencies

I ran into a problem with the build today. I tried to rebuild the coz package using the new GCC compiler in Debian, but could not get the binary to work because it was missing a symbol in elf::elf. I tracked down the problem to the fact that libelfin had not been rebuilt with the new compiler, causing the required symbol to be missing due to different mangling of the function.

Please change the build system to clean the libelfin directory when doing 'make clean'.

Add adaptive progress points

Progress points specified on the command line currently only use sampling, which does not work well for all cases (short-running lines or infrequently-visited progress points). Command line-specified points would be more reliable if they switched between breakpoint and sampling mechanisms adaptively.

Reduce compiler requirement to C++0x?

The current code no longer seem to require --std=c++11, and build just fine with --std=c++0x. Perhaps the requirement in the README should be reduced a little bit? It work with GCC 4.9 in Debian Stable.

If you agree, I suggest something like this should be applied to the source.

index f0e3955..960f5b0 100644
--- a/README.md
+++ b/README.md
@@ -22,7 +22,7 @@ that Counts with Causal Profiling
 Coz, our prototype causal profiler, runs with unmodified Linux executables. Coz requires:

 - [Python](http://www.python.org)
-- [Clang 3.1 or newer](http://clang.llvm.org) or another compiler with C++11 support
+- [Clang 3.1 or newer](http://clang.llvm.org) or another compiler with C++0x support
 - [Linux](http://kernel.org) version 2.6.32 or newer (must support the `perf_event_open` system call)

 ## Building

Drop references to slowdown experiments

These don't seem to be necessary, and it complicates the story somewhat.

How do I get the .coz files for the benchmarks?

After I make bench for one of the benchmarks in benchmarks/, where is the profiling data stored?

Push libelfin changes upstream

Submit a pull request to get libelfin to build with -fPIC or to build a separate shared library target.

Allow users to mark blocking/unblocking points around direct system calls

My application uses futex to implement a spinlock. This is fairly common in high performance server code (eg gperftools uses futex for spinlocks, as do libraries like facebook's folly).

For better accuracy, libcoz should intercept the futex syscall and treat it similar to how condition variables are treated.

Show useful errors/warnings for small profiles

I am trying the benchmark tests and uploading the profile.coz file from toy/ it's not working (nothing happens), the file have the following content:

startup time=1459720345913594695
runtime time=3898923478
samples location=/home/edsiper/coding/coz/benchmarks/toy/toy.cpp:9  count=3854
samples location=/home/edsiper/coding/coz/benchmarks/toy/toy.cpp:13 count=3592
startup time=1459723393567704308
runtime time=3673730614
samples location=/home/edsiper/coding/coz/benchmarks/toy/toy.cpp:9  count=3668
samples location=/home/edsiper/coding/coz/benchmarks/toy/toy.cpp:13 count=3497

any advice is appreciated.

Mismatch of macro names in the documentation and in the include file

Guys, your documentation tells to use COZ_* macros (e.g., COZ_PROGRESS, COZ_BEGIN, etc.), but in the coz.h file they are named as CAUSAL_*.

Change license

Causal profile viewer fails on small profiles

Going to http://plasma-umass.github.io/coz/ and trying to upload the profileshown below, gives the following JavaScript error in the console (both Firefox and Chromium). Nothing happens in the UI after pressing "Open".

Uncaught TypeError: Cannot read property 'tagName' of null
  d @ d3-tip.min.js:1
  t @ d3-tip.min.js:1
  _a.call @ d3.min.js:3
  Profile.drawPlots @ profile.js:339
  update @ ui.js:18
  open_button.classed.on.reader.onload  @   ui.js:66

$ cat profile.coz
startup time=1444175051035166083
experiment  selected=/home/jon/dev/pkgs/coz-git/src/coz-git/benchmarks/kmeans/kmeans-pthread.c:200  speedup=0.00    duration=500061137  selected-samples=33
progress-point  name=kmeans-pthread.c:206   type=source delta=1603740
experiment  selected=/home/jon/dev/pkgs/coz-git/src/coz-git/benchmarks/kmeans/kmeans-pthread.c:154  speedup=0.50    duration=442057369  selected-samples=254
progress-point  name=kmeans-pthread.c:206   type=source delta=1428191
experiment  selected=/home/jon/dev/pkgs/coz-git/src/coz-git/benchmarks/kmeans/kmeans-pthread.c:154  speedup=0.95    duration=406957290  selected-samples=245
progress-point  name=kmeans-pthread.c:206   type=source delta=1482547
experiment  selected=/home/jon/dev/pkgs/coz-git/src/coz-git/benchmarks/kmeans/kmeans-pthread.c:193  speedup=1.00    duration=463013850  selected-samples=68
progress-point  name=kmeans-pthread.c:206   type=source delta=1480336
experiment  selected=/home/jon/dev/pkgs/coz-git/src/coz-git/benchmarks/kmeans/kmeans-pthread.c:193  speedup=1.00    duration=461058169  selected-samples=67
progress-point  name=kmeans-pthread.c:206   type=source delta=1491955
experiment  selected=/home/jon/dev/pkgs/coz-git/src/coz-git/benchmarks/kmeans/kmeans-pthread.c:186  speedup=0.65    duration=483157015  selected-samples=45
progress-point  name=kmeans-pthread.c:206   type=source delta=1573409
experiment  selected=/home/jon/dev/pkgs/coz-git/src/coz-git/benchmarks/kmeans/kmeans-pthread.c:154  speedup=0.50    duration=481681505  selected-samples=99
progress-point  name=kmeans-pthread.c:206   type=source delta=500000
shutdown    time=1444175054308446044    samples=9616
runtime time=3591296609
samples location=/home/jon/dev/pkgs/coz-git/src/coz-git/benchmarks/kmeans/kmeans-pthread.c:154  count=1789
samples location=/home/jon/dev/pkgs/coz-git/src/coz-git/benchmarks/kmeans/kmeans-pthread.c:156  count=3932
samples location=/home/jon/dev/pkgs/coz-git/src/coz-git/benchmarks/kmeans/kmeans-pthread.c:168  count=64
samples location=/home/jon/dev/pkgs/coz-git/src/coz-git/benchmarks/kmeans/kmeans-pthread.c:170  count=318
samples location=/home/jon/dev/pkgs/coz-git/src/coz-git/benchmarks/kmeans/kmeans-pthread.c:186  count=235
samples location=/home/jon/dev/pkgs/coz-git/src/coz-git/benchmarks/kmeans/kmeans-pthread.c:188  count=13
samples location=/home/jon/dev/pkgs/coz-git/src/coz-git/benchmarks/kmeans/kmeans-pthread.c:190  count=265
samples location=/home/jon/dev/pkgs/coz-git/src/coz-git/benchmarks/kmeans/kmeans-pthread.c:192  count=1277
samples location=/home/jon/dev/pkgs/coz-git/src/coz-git/benchmarks/kmeans/kmeans-pthread.c:193  count=449
samples location=/home/jon/dev/pkgs/coz-git/src/coz-git/benchmarks/kmeans/kmeans-pthread.c:200  count=217
samples location=/home/jon/dev/pkgs/coz-git/src/coz-git/benchmarks/kmeans/kmeans-pthread.c:202  count=3
samples location=/home/jon/dev/pkgs/coz-git/src/coz-git/benchmarks/kmeans/kmeans-pthread.c:203  count=1
samples location=/home/jon/dev/pkgs/coz-git/src/coz-git/benchmarks/kmeans/kmeans-pthread.c:206  count=38
samples location=/home/jon/dev/pkgs/coz-git/src/coz-git/benchmarks/kmeans/kmeans-pthread.c:230  count=486
samples location=/home/jon/dev/pkgs/coz-git/src/coz-git/benchmarks/kmeans/kmeans-pthread.c:232  count=405
samples location=/home/jon/dev/pkgs/coz-git/src/coz-git/benchmarks/kmeans/kmeans-pthread.c:234  count=106
samples location=/home/jon/dev/pkgs/coz-git/src/coz-git/benchmarks/kmeans/kmeans-pthread.c:235  count=1
samples location=/home/jon/dev/pkgs/coz-git/src/coz-git/benchmarks/kmeans/kmeans-pthread.c:244  count=1

Make package easier to package for Debian?

Hi. I suggested to add coz-profiler to Debian, and wonder if you might be interested in making this easier?

First and foremost, the package must be buildable without Internet access. The build process today clones a git repo and download some code using wget. Neither is going to work for Debian. Could you perhaps rewrite the build process to avoid such downloading?

Second, it would be easier to figure out which version number to use for the package if you versioned releases, either by tagging the git repo or by making tarball releases with version numbers. Would you be interested in doing this?

The request to add the package in Debian can be found using http://bugs.debian.org/830708 .

Subtasks:

Move gh-pages contents into master as-is, and add build logic to build script.
Include non-minimized versions of all javascript libraries next to the minimised versions.

Long term:

Check on status of TypeScript 2.0.
- If we cannot use it, we can just include the JS version of the profiler source. It's non-ideal, but still human-readable, commented, and modifiable.
Get d3-tip into Debian
Get science.js into Debian

Breakpoint progress points

Allow progress points to be set with watchpoints instead of source changes. Required for real applications. Source-level progress points should still work.

Only support three of these for now. As a future enhancement, progress points could be sampled.

End-to-end mode is broken

Experiments should never end before program shutdown when running in end-to-end mode.

Port to non-Linux platforms?

At the moment coz fail to build on non-Linux platforms like kfreebsd and hurd in Debian, because it uses the API provided in <linux/perf_event.h>. Are there any plans to port coz to non-linux platforms?

Select a larger variety of code fragments for virtual speedup

Programs with one or two hot lines of code don't get very good coverage because of how code fragments are selected for virtual speedup. Instead of picking from the distribution of execution time, choose randomly from the last N unique lines sampled.

Reduce number of visible symbols in libcoz.so?

I noticed several functions in libcoz.cpp without the 'static' keyword, and it occured to me that the shared library perhaps should hide more of the symbols to avoid make clashes with the profiled program. Here is the list of visible symbols without a C++ namespace indicator:

% nm libcoz/libcoz.so |grep ' T '|c++filt |grep -v ::
0000000000047f30 T Exit
0000000000044df0 T wrapped_main(int, char**, char**)
0000000000046bb0 T is_coz_signal(int)
000000000004a1b0 T perf_event_open(perf_event_attr, int, int, int, unsigned long)
0000000000038cc0 T get_loaded_files()
0000000000046b50 T remove_coz_signals(_sigset_t)
0000000000044880 T _coz_get_counter
0000000000047eb0 T _exit
000000000004cc94 T _fini
0000000000016638 T _init
0000000000046af0 T coz_libc_start_main
0000000000047e20 T exit
00000000000481d0 T kill
00000000000476b0 T pthread_barrier_wait
00000000000475e0 T pthread_cond_broadcast
0000000000047510 T pthread_cond_signal
00000000000473d0 T pthread_cond_timedwait
00000000000472a0 T pthread_cond_wait
0000000000046bd0 T pthread_create
0000000000046c70 T pthread_exit
0000000000046cf0 T pthread_join
00000000000482a0 T pthread_kill
00000000000470a0 T pthread_mutex_lock
00000000000471d0 T pthread_mutex_unlock
0000000000047870 T pthread_rwlock_rdlock
00000000000479a0 T pthread_rwlock_timedrdlock
0000000000047c10 T pthread_rwlock_timedwrlock
0000000000047d50 T pthread_rwlock_unlock
0000000000047ae0 T pthread_rwlock_wrlock
0000000000048120 T pthread_sigmask
0000000000048380 T pthread_sigqueue
0000000000046f60 T pthread_timedjoin_np
0000000000046e20 T pthread_tryjoin_np
0000000000047ff0 T sigaction
0000000000047fc0 T signal
0000000000048070 T sigprocmask
0000000000048a80 T sigsuspend
0000000000048860 T sigtimedwait
0000000000048460 T sigwait
0000000000048640 T sigwaitinfo
%

Perhaps some of these, like the get_loaded_files() function, should be marked static to make sure it do not mask functions in the profiled program?

Include profile analysis website source in main repository

Hi. Where can I find the source of the http://plasma-umass.github.io/coz/ web site? It seem unable to do anything with the profile.coz I created from running scrub, and I would like to understand why. Also, I would like to make sure it is possible to analyze the profile results without depending on external Internet access.