gburd / libumem Goto Github PK
View Code? Open in Web Editor NEWThis is the user space slab memory allocator, umem, first available in Solaris 9.
Home Page: https://git.burd.me/greg/libumem
License: Other
This is the user space slab memory allocator, umem, first available in Solaris 9.
Home Page: https://git.burd.me/greg/libumem
License: Other
Portable libumem. ================ This is a port of Solaris libumem to non-Solaris systems. The port was made while integrating libumem with our Ecelerity MTA product, so your initial experience will not be 100% out-of-the-box, because there is no standalone configure script for the library at this time. (patches welcome!) In addition, since our deployment is threaded, we force the library into threaded mode. While the library is itself stable (it's the memory allocator used by the Solaris OS), the port may have a few rough edges. We're shipping umem with Linux and Windows versions of our product as we have found it to be stable. We will continue to update this project as and when we make improvements, and welcome third-party patches that improve the usability for everyone. Wez Furlong, Message Systems, Inc. wez (at) messagesystems (dot) com
I'm trying to use LD_PRELOAD
with libumem
to replace the libc malloc()
. I'm seeing it get loaded but with perf top I'm not seeing malloc from libc being used when I performance profile.
I'm benchmarking using Haywire
git clone https://github.com/gburd/libumem.git lib/libumem
cd lib/libumem
./autogen.sh
./configure
make
cd ..
LD_PRELOAD=./lib//libumem/.libs/libumem.so /usr/bin/ldd ./build/hello_world
linux-vdso.so.1 => (0x00007fffdfdfb000)
../libumem/.libs/libumem.so (0x00007f4a45ec0000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f4a45c9c000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f4a45a93000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f4a456c9000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f4a454c5000)
/lib64/ld-linux-x86-64.so.2 (0x0000555dc31f9000)
perf top
9.70% hello_world [.] http_parser_execute
8.16% libc-2.21.so [.] malloc
4.62% libc-2.21.so [.] free
3.68% libc-2.21.so [.] __libc_calloc
3.18% hello_world [.] http_request_buffer_reassign_pin
3.14% hello_world [.] http_request_buffer_pin
2.95% libc-2.21.so [.] 0x00000000001452a0
2.43% hello_world [.] http_request_buffer_locate
2.04% libc-2.21.so [.] 0x000000000014d6b0
1.89% [kernel] [k] native_queued_spin_lock_slowpath
1.67% libc-2.21.so [.] 0x000000000014d86c
1.36% [kernel] [k] tcp_sendmsg
1.34% libc-2.21.so [.] 0x00000000000806dd
1.29% libc-2.21.so [.] 0x0000000000081890
I thought perhaps I needed to load libumem_malloc.so
but if I try the following calloc()
is running over and over even before I run any kind of benchmark against the service and the service never starts correctly.
LD_PRELOAD="./lib/libumem/.libs/libumem.so ./lib/libumem/.libs/libumem_malloc.so" ./build/hello_world
perf top
96.40% libumem_malloc.so.0.0.0 [.] calloc
0.60% libumem_malloc.so.0.0.0 [.] calloc@plt
0.21% [kernel] [k] native_write_msr_safe
0.20% [kernel] [k] rcu_check_callbacks
0.15% [kernel] [k] ktime_get
0.12% [kernel] [k] apic_timer_interrupt
0.11% [kernel] [k] cpu_needs_another_gp
0.08% [kernel] [k] update_sd_lb_stats
0.08% [kernel] [k] ktime_get_update_offsets_now
0.07% [kernel] [k] menu_select
0.07% [kernel] [k] native_queued_spin_lock_slowpath
0.07% [kernel] [k] cpuidle_enter_state
0.06% [kernel] [k] _raw_spin_lock_irqsave
0.06% [kernel] [k] note_gp_changes
Integrate the changes mentioned here (https://twitter.com/bcantrill/status/224762480755355650 TritonDataCenter/illumos-joyent@84994e8 http://dtrace.org/blogs/rm/2012/07/16/per-thread-caching-in-libumem/) and deal with cross-platform portability (because thread libraries are notoriously not portable) issues. This is the "trampoline" feature found here https://github.com/joyent/illumos-joyent/blob/master/usr/src/lib/libumem_trampoline/common/trampoline.c#L26
umem has many very useful features which are exposed via its integration with MDB (the "modular debugger" of Solaris/Illumos) which are missing on the target systems for this port, find a way to expose those features in a command line tool.
Evaluate and potentially use the TSX [1] instructions available on Intel Haswell (and later) chips.
[1] http://software.intel.com/en-us/blogs/2012/02/07/transactional-synchronization-in-haswell
umem has many very useful features which are exposed via its integration with MDB (the "modular debugger" of Solaris/Illumos) which are missing on the target systems for this port, find a way to expose those features in either or both of GDB and LLDB.
Just checking for maintenance status because last commit was in 2012 and last release was in 2010.
Is this still being used by someone?
Long ago I remember a malloc-variant which recorded the sequence and size of allocations (and deallocations) from program startup until some point later in time. The goal was to record into a file that allocation pattern so that it could be included with the program in its final form. The result was that applications could startup faster and with much less memory fragmentation because the allocator knew what was going to happen and it pre-allocated and laid out space to match that pattern.
That would be interesting in and of itself. Taken a bit further one could keep track of allocation patterns and predict to some extent groups of future allocations based on current patterns. (maybe)
Zone-based memory allocator. A zone is a collection of fixed size data blocks for which quick allocation/deallocation is possible. [1] Include APIs/features inspired by "NSZone" allocation from Obj-C for a) fast-free of zones, b) different allocators per-zone, c) NUMA/core/thread-specific allocation. An example of a good use for (b) is if you had the ability to allocate transient memory from RAM as normal along side of persistent memory backed by some flash device (SSD, PCIe, Violin Memory, etc.). Add z-prefixed versions of all the standard malloc API calls and add a "set allocator for Zone" API. In addition, add a checkpoint call (for persistent stores).
[1] http://www.opensource.apple.com/source/xnu/xnu-2050.18.24/osfmk/kern/zalloc.c?txt
[2] http://www.webweavertech.com/libFoundation/doc/libFoundation_20.html
Evaluate integrating the features of TinySTM [1] with umem to enable.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.