memory usage with large numbers of objects about dbus-rs

[tasleson]

@diwic Tried out your changes and the memory improvement is impressive, thank you!

[diwic]

I clone strings a lot, (look at all the ArcMaps in the code), so that shouldn't be it.

[tasleson]

My theory was because each and every dbus object has multiple hash maps, but that's just a guess.

[diwic]

I wonder how much of this is jemalloc's fault. Want to give it a try with alloc_system and see if it helps?

What you add a lot of here are properties, and a property looks like this:

pub struct Property<M> {
    name: Arc<String>,
    value: Mutex<MessageItem>,
    emits: EmitsChangedSignal,
    rw: Access,
    set_cb: Option<M>,
    owner: Mutex<Option<(Arc<Path<'static>>, Arc<IfaceName<'static>>)>>,
    anns: BTreeMap<String, String>,
}

Which is 176 bytes directly, not counting strings etc.

[tasleson]

Running rust, rustc 1.12.0-nightly (7ad125c4e 2016-07-11):

307840 KB  alloc_system
357172 KB  default

An improvement, but still quite a bit of memory.

Note: memory is Resident Memory Size (KiB) as displayed from top.

[diwic]

300 MB for ~10000 properties? Yikes.

[diwic]

I can't reproduce here. Here's what I get:

 VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
53756  21912   2728 S   0,0  0,6   0:00.01 dbus_test 

I can't say 22 MB is tiny, but it's an order of magnitude better than the 300 MB you're getting.

This is running under Ubuntu 16.04 (x86_64) and rustc 1.10.0-nightly (179539f6e 2016-05-20)

[tasleson]

@diwic Wow! I did my test in Fedora 24, this result is encouraging.

[diwic]

Just tried upgrading to rustc 1.12.0-nightly (7ad125c4e 2016-07-11) - no significant in memory usage compared to 1.10.

[tasleson]

I spun up a xubuntu 16.04 VM and I'm see the same large memory consumption with rustc 1.12.0-nightly (7ad125c4e 2016-07-11) . I'm doing cargo run --release not sure what I'm doing different.

[diwic]

@tasleson Wild guess, but you haven't made any changes to your repo that you forgot to push? Last commit is from May.

[tasleson]

@diwic You are 100% correct, I had uncommitted changes in my directory. I committed and pushed. Your test had 604 dbus objects instead of the 10K I had.

[tasleson]

@diwic With the correct number of objects (10K) you should see similar memory consumption. Our main concern is that it takes 300+ MB memory to serve up ~5 MB of data to the client.

./dbus_size.py session com.blah.storage /com/blah/storage/block
{   'booleans': 49995,
    'byte_count': 0,
    'dbus_object_num': 10000,
    'dictionary_key_num': 0,
    'dictionary_key_size_total': 0,
    'double_count': 0,
    'int32_count': 0,
    'int64_count': 0,
    'interface_num': 30000,
    'interface_size_total': 939999,
    'object_path_num': 10000,
    'object_path_size_total': 278885,
    'property_name_num': 199980,
    'property_name_size_total': 1429857,
    'string_num': 89991,
    'string_size_total': 1648725,
    'uint32_count': 9999,
    'uint64_count': 49995}
We retrieved 10000 objects
Size fixed len data = 489951
Size variable len data (strings) = 4297466
Total bytes 4787417
Average bytes per object 478

[tasleson]

Tried using the https://github.com/gperftools/gperftools tools, generated this output (rust_mem.pdf) which doesn't appear super useful. The MB amounts are zero, but the percentages look interesting. I think it has potential if it worked better. I'm likely doing something wrong. I used http://stackoverflow.com/questions/38254937/how-do-i-debug-a-memory-issue-in-rust for suggestions.

[diwic]

Ok. I did some profiling based on your test and came up with one low-hanging fruit of not allocating unnecessary BTreeMaps when there are no annotations (apparently BTreeMap itself does not optimise for the empty case, like Vec does). It's now pushed to master.

For me, this cuts the memory usage in half, approximately.

Other than that, I'm not sure - 150 MB means 750 bytes per property and a lot of that is pointers/usizes (e g, an Arc<String> is 40 bytes, four usizes and a pointer), so moving to 32 bit code (if that's an option) would save a significant amount.

[diwic]

JFTR, the profiling was done with valgrind/massif after switching dbus_test over to alloc_system.

[diwic]

I've been thinking about this today and I think it would be possible to rewrite the Tree stuff a bit so that it would be possible to use the same Interface struct for many ObjectPaths. For the test case that would work fine and likely save a lot of memory. Calls to on_set (and on_get would be added), would then have to check what object path they're on in order to get/set the right object. Would that make sense in a real-world scenario too?

[tasleson]

@diwic I'm a little confused by the wording here. Do you mean to say same interface struct for objects that implement the same interface instead of for many object paths? The example is kind of arbitrary. In real world we would have dbus objects which have one or more interfaces that are dynamically added to and removed based on change in object.

[diwic]

@tasleson

I'm a little confused by the wording here.

Sorry. English is not my native language so it's hard to find the right words sometimes.

Do you mean to say same interface struct for objects that implement the same interface instead of for many object paths?

Yes. In your example, you would create the com.blah.storage.BlockDevice interface only once, then add a clone (which just increases the Arc refcount) of that interface to the 10K object paths.

In real world we would have dbus objects which have one or more interfaces that are dynamically added to and removed based on change in object.

That's fine (although we don't have any InterfacesAdded support in GetObjectManager yet). As long as it's not 10K different interfaces.

[tasleson]

@diwic The total number of different interfaces would certainly not be 10K. However, it could be in the 40-50 range with no specific object having more than 4-5. We are talking about a fairly large dbus service in terms of functionality and in terms of number of objects served.

Thanks for digging into all of this, it's encouraging and makes it more possible that rust is a viable language choice.

[tasleson]

@diwic Thanks for the tip on getting memory information better. I compiled with the system allocator and I'm seeing better results with gperf, however interpreting the results is not blatantly obvious to me at the moment.
dbus_test_mem.pdf

[diwic]

Thanks for digging into all of this, it's encouraging and makes it more possible that rust is a viable language choice.

Well, makes me more motivated to work on things if I know they'll be used. :-) I started writing some code today to decouple the property from the value it currently contains (and try to solve #39 in the process, if possible) but it needs some rewriting and rethinking...

[diwic]

Hi, my new tree module is shaping up to the point that I can make dbus_test work with it. The modified code is here. It takes 12 MB to serve the data that took 300 MB before.

Give it a try and see if you can replicate my results and if the new code is working the way you expect. Thanks!

[tasleson]

@diwic This is excellent progress, thanks! I will take a peek at all of this as soon as I can!

[diwic]

Apparently a month has gone and nothing happened on your side, so I'm releasing v0.4.0 anyhow. I hope it fixes your issue. Also have a look at the new adv_server.rs example which includes some ideas on how to make use of the new tree module. Enjoy!

[tasleson]

@diwic Sorry, will spend some time looking at this in the next week!