Coder Social home page Coder Social logo

cea-hpc / selfie Goto Github PK

View Code? Open in Web Editor NEW
16.0 14.0 4.0 188 KB

Self and Light proFIling Engine - a very light profiling tools for Linux commands and HPC codes

License: Other

Makefile 2.80% Shell 5.70% M4 6.09% C 46.11% Python 18.54% C++ 20.75%
selfie mpi papi profile light-profiling-engine profiling linux

selfie's Introduction

selFIe

Build Status

selFIe (SElf and Light proFIling Engine) is a tool to lightly profile Linux commands without compiling.

selFIe is a dynamic library which can be given to the LD_PRELOAD environment variable before the execution of the command.

$ LD_PRELOAD=selfie.so hostname
myhost

It doesn't affect the behaviour of the command and users don't see any changes at execution. At the end of the execution, it puts a line in system logs:

selfie[26058]: { "utime": 0.00, "stime": 0.01, "maxmem": 0.00, "posixio_time": 0.00, "posixio_count": 7569, "USER": "user", "wtime": 0.01, "command": "/bin/hostname" }

The log line is in JSON format for post-processing (like ELK for example).

selFIe can be compiled to profile other features:

  • MPI calls (Only C API)
  • PAPI hardware counters

Download

Official release can be downloaded in the release page.

To use development version:

$ git clone https://github.com/cea-hpc/selFIe.git
$ cd selFIe
$ ./autogen.sh

Installation

Prerequisites

  • C and C++ compiler
  • libYAML
  • Python

selFIe has been tested on GNU/Linux only

Optional:

  • PAPI library

Compilation

$ ./configure --prefix=<path-to-install> --with-libyaml=<path-to-libyaml> --with-posixio 
$ make                                  
$ make install                          

To compile with MPI and PAPI profiling:

$ ./configure --prefix=<path-to-install> --with-libyaml=<path-to-libyaml> --with-posixio --with-mpi --with-mpiio --with-papi=<path-to-papi>
$ make                                  
$ make install                          

Profiling

To profile your code with selFIe, you only have to add selfie.so to the LD_PRELOAD environment variable.

$ export LD_PRELOAD=<path-to-install>/lib/selfie.so
$ hostname
myhost
$ mpirun -n 1 ./hello_world
Hello world!
$ grep selfie /var/log/syslog
selfie[26135]: { "utime": 0.01, "stime": 0.00, "maxmem": 0.00, "posixio_time": 0.00, "posixio_count": 7569, "papi_ipc": 0.50, "papi_mem_bw": 13.62, "mpi_time": 0.00, "mpi_count": 0, "mpiio_time": 0.00, "mpiio_count": 0, "USER": "user", "wtime": 0.01, "command": "/bin/hostname" }
selfie[26040]: { "utime": 0.02, "stime": 0.04, "maxmem": 0.01, "posixio_time": 0.04, "posixio_count": 8383, "papi_ipc": 0.93, "papi_mem_bw": 143.62, "mpi_time": 0.10, "mpi_count": 2, "mpi_version": 3.00, "mpi_libversion": "open mpi", "mpiio_time": 0.00, "mpiio_count": 0, "mpiio_version": 3.00, "mpiio_libversion": "open mpi", "USER": "user", "wtime": 0.10, "command": "./a.out" }

Configuration

The behaviour of selFIe can be configured in the file /etc/selfie.conf. The default configuration has the following settings:

env_vars:
- USER
- OMP_NUM_THREADS

time_limit: 5.0

exclude_commands:
- ^/bin/
- ^/sbin/
- ^/usr/libexec/
- ^/usr/bin/

This file is divided in three parts:

  • env_vars: this section lists the environment variables you want to log along the profiling.
  • time_limit: to avoid flooding system logs, selFIe can be configured to log only commands that lasts at least a time. By default, this time is 5 seconds.
  • exclude_commands: to avoid flooding system logs, selFIe can be configured to not log some commands. The section lists the commands that will not generate logs in log facility. You can specify a regular expression. By default, selFIe doesn't not log commands located in directories /bin, /sbin, /usr/libexec and /usr/bin.

By setting the environment variable SELFIE_CONFIGFILE, you can specify another configuration file for selFIe.

$ export SELFIE_CONFIGFILE=${HOME}/myselfie.conf

For profiling MPI codes, you should add the following environment variable to track the MPI rank of MPI processus depending which MPI library you use:

  • SLURM_PROCID when using SLURM for launching MPI jobs
  • OMPI_COMM_WORLD_RANK for OpenMPI
  • PMI_RANK for IntelMPI

If you want to put log in other place than syslog, you can set SELFIE_OUTPUTFILE with a filename:

$ export SELFIE_OUTPUTFILE=/tmp/selfie.out
$ LD_PRELOAD=selfie.so hostname
$ cat $SELFIE_OUTPUTFILE
{ "utime": 0.00, "stime": 0.01, "maxmem": 0.00, "posixio_time": 0.00, "posixio_count": 7569, "USER": "user", "timestamp": 1500000, "wtime": 0.01, "command": "/bin/hostname" }

Using selFIe in a MPI job may result bad written output file. Then you can can use a directory for storing selFIe output. selFIe will choose a file name for each processes:

$ export SELFIE_OUTPUTDIR=${HOME}/selfie_outdir
$ LD_PRELOAD=selfie.so hostname
$ cat $SELFIE_OUTPUTDIR/*
{ "utime": 0.00, "stime": 0.01, "maxmem": 0.00, "posixio_time": 0.00, "posixio_count": 7569, "USER": "user", "timestamp": 1500000, "wtime": 0.01, "command": "/bin/hostname" }
...

Contributing

Authors

See the list of AUTHORS who participated in this project.

Contact

Laurent Nguyen - [email protected]

Website

CEA-HPC

Architecture

alt tag

selFIe is based on two features:

  • selFIe can execute some code before and after main program
  • selFIe uses the LD_PRELOAD environment variable to hook POSIXIO or MPI calls

License

Copyright 2015-2019 CEA/DAM/DIF

selFIe is distributed under the CeCILL-C. See the included files
Licence_CeCILL-C_V1-en.txt (English version) and
Licence_CeCILL-C_V1-fr.txt (French version) or visit
http://www.cecill.info for details.

Acknowledgments

selfie's People

Contributors

laurent-nguyen avatar martinetd avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

selfie's Issues

memory bandwidth counter

Hello,

I would like some information about the performance counter used to measure memory bandwidth:

OFFCORE_RESPONSE_0:ANY_DATA:ANY_RESPONSE

Is there any documentation about it? I was looking for it and I found some different counters to measure memory bandwidth like:

OFFCORE_RESPONSE_0:ANY_DATA_RD:OTHER:ANY_LLC_MISS
OFFCORE_RESPONSE_0:ANY_DATA:OTHER:ANY_RESPONSE

I have done some experiments and I found that for mono process applications it seems to work but I have found some incoherence in result with MPI applications, for example the benchmark hpl where I got more than the maximum theoretical bandwidth.

I would appreciate any help thank you in advance

RFE: catch sigterm/sigint/sigquit

If a job is cancelled, SelFIe will not log anything when the process is killed.

It is possible to intercept the stop signals (int term quit) and print stats (& remember it's been printed to avoid duplicates if handled properly?) then propagate (get existing handler, handle, call next one; if no next one unregister oneself and send again maybe? would need to check that part) so one could get logs on jobs being cancelled, unless they're stuck in IO or something until sigkill finally comes.

Test selfie_hellovar.sh fails if the current timestamp changes while test is executed

While running benchmarks on selFIe, I found that the test selfie_hellovar.sh is flaky, i.e. it fails sometimes.

The test selfie_hellovar.sh saves the current timestamp into the SELFIE_TEST variable. After the code under test finishes, it searches for the timestamp inside the error output of the code under test.

However, and I believe this causes the flaky behaivior, if the timestamp changes between the test itself and the code under test reading the current timestamp (e.g. by the test taking longer than a second or the test being started close to the "end of a second"), the grep call fails, causing the test to fail.

Writing profiling output to STDOUT

Hello,

Is it possible to print out the result of the profiling on STDOUT ? I tried using SELFIE_PRINT_=1 but it shows just a summary:

=== SELFIE DETAILS ===

  • Execution time : 74.247798

  • Command : /home/cr3db69n/benchmarks/nas/NPB3.3.1/NPB3.3-MPI/bin/lu.C.8

  • Environment variables -

              USER : cristian
    

I'm a normal user in the cluster and I cannot read the syslog file.

Thank you in advance.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.