The dsi-progress-logger-rs from vigna

dsi-progress-logger-rs's People

Contributors

Stargazers

Watchers

dsi-progress-logger-rs's Issues

Logging non-shared resident memory

ProgressLogger prints the current resident memory (as "used").

However, because of how often we use it while working with large memory-mapped files, this is not very useful, as it includes the shared memory (ie. the parts of the mmaped files that are cached in memory).

Using the ProgressLogger with Rayon

Hi,

A pattern I find myself reusing when working with Rayon is the following:

            let mut pl = ProgressLogger::default().display_memory();
            pl.item_name = "arc";
            pl.local_speed = true;
            pl.expected_updates = Some(estimate_updates(&dataset_dir));
            pl.start("Sorting arcs");
            let pl = Mutex::new(pl);

            let counters = thread_local::ThreadLocal::new();
            let sorted_arc_lists: Vec<SortPairs<()>> = iter_arcs(&dataset_dir)
                .inspect(|_| {
                    // This is safe because only this thread accesses this and only from
                    // here.
                    let counter = counters.get_or(|| UnsafeCell::new(0));
                    let counter: &mut usize = unsafe { &mut *counter.get() };
                    *counter += 1;
                    if *counter % 32768 == 0 {
                        // Update but avoid lock contention at the expense
                        // of precision (counts at most 32768 too many at the
                        // beginning of each iterator)
                        pl.lock().unwrap().update_with_count(32768);
                        *counter = 0
                    }
                })
                .map(|arc| {
                    // ...
                })
                .collect();