gburd / libumem Goto Github PK

I'm trying to use LD_PRELOAD with libumem to replace the libc malloc(). I'm seeing it get loaded but with perf top I'm not seeing malloc from libc being used when I performance profile.

I'm benchmarking using Haywire

git clone https://github.com/gburd/libumem.git lib/libumem
cd lib/libumem
./autogen.sh
./configure
make
cd ..

LD_PRELOAD=./lib//libumem/.libs/libumem.so /usr/bin/ldd ./build/hello_world
        linux-vdso.so.1 =>  (0x00007fffdfdfb000)
	../libumem/.libs/libumem.so (0x00007f4a45ec0000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f4a45c9c000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f4a45a93000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f4a456c9000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f4a454c5000)
	/lib64/ld-linux-x86-64.so.2 (0x0000555dc31f9000)

perf top

   9.70%  hello_world         [.] http_parser_execute
   8.16%  libc-2.21.so        [.] malloc
   4.62%  libc-2.21.so        [.] free
   3.68%  libc-2.21.so        [.] __libc_calloc
   3.18%  hello_world         [.] http_request_buffer_reassign_pin
   3.14%  hello_world         [.] http_request_buffer_pin
   2.95%  libc-2.21.so        [.] 0x00000000001452a0
   2.43%  hello_world         [.] http_request_buffer_locate
   2.04%  libc-2.21.so        [.] 0x000000000014d6b0
   1.89%  [kernel]            [k] native_queued_spin_lock_slowpath
   1.67%  libc-2.21.so        [.] 0x000000000014d86c
   1.36%  [kernel]            [k] tcp_sendmsg
   1.34%  libc-2.21.so        [.] 0x00000000000806dd
   1.29%  libc-2.21.so        [.] 0x0000000000081890

I thought perhaps I needed to load libumem_malloc.so but if I try the following calloc() is running over and over even before I run any kind of benchmark against the service and the service never starts correctly.

LD_PRELOAD="./lib/libumem/.libs/libumem.so ./lib/libumem/.libs/libumem_malloc.so" ./build/hello_world

perf top
  96.40%  libumem_malloc.so.0.0.0  [.] calloc
   0.60%  libumem_malloc.so.0.0.0  [.] calloc@plt
   0.21%  [kernel]                 [k] native_write_msr_safe
   0.20%  [kernel]                 [k] rcu_check_callbacks
   0.15%  [kernel]                 [k] ktime_get
   0.12%  [kernel]                 [k] apic_timer_interrupt
   0.11%  [kernel]                 [k] cpu_needs_another_gp
   0.08%  [kernel]                 [k] update_sd_lb_stats
   0.08%  [kernel]                 [k] ktime_get_update_offsets_now
   0.07%  [kernel]                 [k] menu_select
   0.07%  [kernel]                 [k] native_queued_spin_lock_slowpath
   0.07%  [kernel]                 [k] cpuidle_enter_state
   0.06%  [kernel]                 [k] _raw_spin_lock_irqsave
   0.06%  [kernel]                 [k] note_gp_changes

Integrate the changes mentioned here (https://twitter.com/bcantrill/status/224762480755355650 TritonDataCenter/illumos-joyent@84994e8 http://dtrace.org/blogs/rm/2012/07/16/per-thread-caching-in-libumem/) and deal with cross-platform portability (because thread libraries are notoriously not portable) issues. This is the "trampoline" feature found here https://github.com/joyent/illumos-joyent/blob/master/usr/src/lib/libumem_trampoline/common/trampoline.c#L26

umem has many very useful features which are exposed via its integration with MDB (the "modular debugger" of Solaris/Illumos) which are missing on the target systems for this port, find a way to expose those features in a command line tool.

Evaluate and potentially use the TSX [1] instructions available on Intel Haswell (and later) chips.

[1] http://software.intel.com/en-us/blogs/2012/02/07/transactional-synchronization-in-haswell

umem has many very useful features which are exposed via its integration with MDB (the "modular debugger" of Solaris/Illumos) which are missing on the target systems for this port, find a way to expose those features in either or both of GDB and LLDB.

Just checking for maintenance status because last commit was in 2012 and last release was in 2010.
Is this still being used by someone?

See: [1]

1. http://people.cs.umass.edu/~emery/pubs/ccs03-novark.pdf

Long ago I remember a malloc-variant which recorded the sequence and size of allocations (and deallocations) from program startup until some point later in time. The goal was to record into a file that allocation pattern so that it could be included with the program in its final form. The result was that applications could startup faster and with much less memory fragmentation because the allocator knew what was going to happen and it pre-allocated and laid out space to match that pattern.

That would be interesting in and of itself. Taken a bit further one could keep track of allocation patterns and predict to some extent groups of future allocations based on current patterns. (maybe)

Zone-based memory allocator. A zone is a collection of fixed size data blocks for which quick allocation/deallocation is possible. [1] Include APIs/features inspired by "NSZone" allocation from Obj-C for a) fast-free of zones, b) different allocators per-zone, c) NUMA/core/thread-specific allocation. An example of a good use for (b) is if you had the ability to allocate transient memory from RAM as normal along side of persistent memory backed by some flash device (SSD, PCIe, Violin Memory, etc.). Add z-prefixed versions of all the standard malloc API calls and add a "set allocator for Zone" API. In addition, add a checkpoint call (for persistent stores).

[1] http://www.opensource.apple.com/source/xnu/xnu-2050.18.24/osfmk/kern/zalloc.c?txt
[2] http://www.webweavertech.com/libFoundation/doc/libFoundation_20.html

Evaluate integrating the features of TinySTM [1] with umem to enable.

[1] http://www.tmware.org/tinystm