Data miscompares about umap

[mcfadden8]

I added some additional instrumentation which gets us a little closer to when the pages are being updated. The umap library has been modified to check the consistency of the entire buffer of pages each time the thread returns from the userfaultfd poll() request. The consistency check looks at each page descriptor in the buffer and if the page is not marked as dirty a SHA1 comparison is performed against the data that was originally read from the backing store.

The logfile below contains the results from running umapsort in the following way:

$ time umapsort -p 100000 -b 1000 -f /mnt/intel/sortfile -t 128 -u 1 >& /tmp/logfile

This causes 128 application threads to perform a quicksort on the data that is in /mnt/intel/sortfile. One userfaultfd thread is present and the buffer is limited to just 1,000 pages.

logfile.txt

As can be seen in the logfile, there were 7 pages in the buffer that got into an inconsistent state:

223973 0x7f7721c9e000 WARNING: Dirty page found that was not previously marked dirty!
 223974 0x7f7721d01000 WARNING: Dirty page found that was not previously marked dirty!
 223975 0x7f7721d03000 WARNING: Dirty page found that was not previously marked dirty!
 223976 0x7f7721c91000 WARNING: Dirty page found that was not previously marked dirty!
 223977 0x7f7721c8f000 WARNING: Dirty page found that was not previously marked dirty!
 223978 0x7f7721c8d000 WARNING: Dirty page found that was not previously marked dirty!
 223979 0x7f7721d16000 WARNING: Dirty page found that was not previously marked dirty!

The umap() library does not do anything about the inconsistency until one of the pages is actually EVICTED in order to make room for a new page becoming present as shown at the following lines of the end of the logfile:

 226763 0x7f7721d16000 EVICT
 226764 0x7f7721d16000 WARNING: Dirty page found that was not previously marked dirty!
 226765 0x7f7721d16000 Dirty page found that was not previously marked dirty! Exiting

Looking at page 0x7f7721d16000, you will see that it was brought in to the buffer at line number:

 223623 0x7f7721d16000 NOT Present { 0 READ }

I believe that I am logging each message that I receive back from the poll() call to userfaultfd. Yet, I am not seeing any other messages for that particular page. It almost seems like the UFFCIO_COPY(UFFDIO_COPY_MODE_WP) is not sticking and the page seems to be writeable?