Loading filelist.xml.gz is extremely slow about libsolv

[mlschroe]

(back from parental leave)

Seems like your system is paging like hell. How much mem does your system have? Is there some limitation of the process RSS size in place?

I can write some specialized parser for the file list that further reduces the amount of needed memory, but the current code doesn't need that much mem already. So I wonder why it's so slow for you.

(Btw mremap doesn't really copy memory, it just changes the mapping for the pages)

[marmarek]

There is dynamic memory management on that virtual machine, but there is always few hundreds megs free. No additional limitations.

[user@testvm ~]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1212
max locked memory       (kbytes, -l) 131072
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1212
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Also when I assign that machine 2GB statically (from which 1.5GB is free), the problem is still there. I don't think it's about amount of memory (100MB isn't that much), but about the way it is assigned (a lot of small mremaps).

[v6ak]

Maybe mremaps are slower under virtualization. In case of marmarek, he probably uses Xen PV.

[mlschroe]

Hmm, I don't see how virtualization can slow down mremap, but I don't habe much experience with Xen.
Does the dnf process have a big enough RSS size? I.e. can you please run dnf makecache and monitor the RSS size with a tool like 'top'?

[mlschroe]

If you want to try less mremap calls, please change the EXTDATA_BLOCK define in repodata.c from 1023 to 65535.

[marmarek]

RSS starts from ~100M (after downloading repo md), then increases slowly to about 250M.
I'll try that modification.

[marmarek]

With that modification, it's much better - about 1 minute "sys" time (down from over 10min).
Maybe it is possible to start with some approximation of required memory and allocate that? If not, is it an option to make EXTDATA_BLOCK default to 64k?

Fragment of strace:

21:38:14.192737 mremap(0x7f5f0e903000, 111546368, 111611904, MREMAP_MAYMOVE) = 0x7f5f062fe000 <0.031068>
21:38:14.224068 mremap(0x7f5f062fe000, 111611904, 111677440, MREMAP_MAYMOVE) = 0x7f5f0e8e3000 <0.024803>
21:38:14.249684 mremap(0x7f5f0e8e3000, 111677440, 111742976, MREMAP_MAYMOVE) = 0x7f5f062de000 <0.025371>
21:38:14.275512 mremap(0x7f5f062de000, 111742976, 111808512, MREMAP_MAYMOVE) = 0x7f5f0e8c3000 <0.035265>
21:38:14.311151 mremap(0x7f5f0e8c3000, 111808512, 111874048, MREMAP_MAYMOVE) = 0x7f5f062be000 <0.038956>
21:38:14.350499 mremap(0x7f5f062be000, 111874048, 111939584, MREMAP_MAYMOVE) = 0x7f5f0e8a3000 <0.026496>

[mlschroe]

Ok, try commit a8220fe. It reduces the mremap calls from 96163 to 675 when converting Fedora22's Everything repo (I guess that's the repo you're talking about). It doesn't do much difference on my system, though...

Please change the EXTDATA_BLOCK value back to the original value, the new commit works in a different way.

I get the following stats for reading primary/filelist and then writing the filelist as solv file:
10.168u 0.272s 0:10.45 99.8% 0+0k 0+63304io 0pf+0w

10 seconds seems ok to me, 1 minute would IMHO be too slow.

[marmarek]

Thanks! This makes the whole dnf check-update needing only 4s of sys time.