nmcclatchey / threadpool

Portable (C++11) low-overhead concurrent task scheduling for fine-grained concurrency.

ThreadPool

Provides low-overhead concurrent scheduling in C++11 through thread pools and work stealing. The thread pool approach allows fine-grained parallelism by minimizing the overhead involved in scheduling a task. The work stealing approach allows efficient balancing of scheduled tasks across available threads.

Why use this library?

• It fulfills common scheduling needs:
  • Performs multiple tasks concurrently.
  • Tasks can be scheduled for an arbitrary later time-point. This provides an efficient replacement for timed waits.
  • A task can spawn subtasks, with a hint that the pool ought to complete them as soon as possible.
• It fulfills some uncommon scheduling needs:
  • The pool can be paused, and later resumed.
• It is designed for efficiency and scalability:
  • Load balancing ensures that as long as there is work to do, it is being done by as many threads as possible.
  • Lock-free data structures use weak atomic orderings; only when those cores that require new tasks are synchronized.
  • No busy-waiting. Idle threads use condition variables to wait without using the CPU.
• It is explicitly documented:
  • Full generated documentation via Doxygen.
  • Memory synchronization between task scheduling and execution is explicitly stated in terms of C++11's memory model.
  • Provides usage recommendations for maximizing performance.

Getting Started

This library consists of a single header and source file. One may compile the source file either as part of one's own project, or as a static library.

Prerequisites

You will require a C++11 compiler. If you are using MinGW-w64, you may require MinGW STD Threads to supply std::thread and similar.

Installing

Either compile threadpool.cpp as part of your project, or compile it as a static library.

Using the library

The library is designed to enable a simple use pattern:

1. Create a ThreadPool object.
2. Give tasks to the pool by calling the pool's schedule(), schedule_subtask(), or schedule_after() methods.
3. Wait for tasks to complete.

Full documentation for this library may be generated using Doxygen.

A simple example of how to the library follows:

#include "threadpool.hpp"

//  Create a new thread pool, letting the implementation determine the number of worker threads to use.
ThreadPool pool;

//  Put a task into the pool. Because this isn't called from within a worker thread, it takes the scheduler's slow path.
pool.schedule([](void)
{
//  Put a task into the pool. This is called from within a worker thread, so it takes the scheduler's fast path.
  pool.schedule([](void) {
    do_something();
  });
 
//  Put a task into the pool, treated as if it were part of the currently running task. This is called from within a worker thread, so it takes the scheduler's fast path.
  pool.schedule_subtask([](void) {
    do_something();
  });

//  Put a task into the pool, to be executed 2 seconds after it is scheduled.
  using namespace std::chrono;
  pool.schedule_after(seconds(2),
  [](void) {
    do_something();
  });
});

//  When the thread pool is destroyed, remaining unexecuted tasks are forgotten.

Authors

• Nathaniel J. McClatchey, PhD - Initial work

License

To encourage people to use this library freely and without concern, this project is licensed under the MIT License.

threadpool's Issues

Question about the new definition std :: jthread

How different is the std::thread from std::jthread?
Does it make sense to change definitions for c++20?
Is std::jthread only under c++20?
I was able to compile std::jthread under c++17. Is it compatible with mingw.thread?
https://github.com/josuttis/jthread

Task Re-entrancy

I want a custom way to pop and execute a task from the ThreadPool while "inside" Worker::execute(). My current attempt seems to corrupt the queue.

I think my call to worker->pop below (this is our thread's own worker) is corrupting the local queue. I'm getting later exceptions trying to call null function objects if I use that code.

Here's the context including my current attempt (which is added to your source file):

---

I'd like a way for a currently executing task to re-enter the ThreadPool, pick up an additional task and execute that task right now. This routine will be called either

1. From the main thread, while waiting for tasks to complete
2. From a worker thread, inside a task - waiting for spawned tasks to complete.

For case (2) that means we ARE inside Worker::execute and likely also inside ThreadPoolImpl::steal_work or whatever function name is used here.

For some nice diagrams, see https://software.intel.com/content/www/us/en/develop/documentation/tbb-documentation/top/intel-threading-building-blocks-developer-reference/task-scheduler.html - I'm hoping to implement a simple "Blocking Style"

When we last discussed this, you suggested continuations (with an example, thanks!) and coroutines. Both of these solutions require the user code to "understand" our threading model, where I want to hide it somewhat. I'm porting existing naive code to use a threadpool under C++11 so those solutions are hard.

I've created a simple first-pass implimentation:

bool ThreadPoolImpl::steal_work()
{
    Worker* worker = current_worker;
    // If a thread is attempting to schedule in its own pool...
    if ((worker != nullptr) && worker->belongs_to(this))
    {
        // Run one of the tasks due to be run in this (our) worker thread
        task_type task;
        if (worker->pop(task)) // <<< I cannot figure out how to do this without corruption
        {
            task();
            return true;
        }
    }
    else if (mutex_.try_lock())
    {
        // Run a task from the general thread-pool
        if (queue_.empty())
        {
            mutex_.unlock();
        }
        else
        {
            task_type task = extract_task();
            mutex_.unlock();
            task();
            return true;
        }
    }

    return false;
}

Here is how my threadpool wrapper uses this functionality m_pool.steal_work() :

// Do useful work until the given future is ready. 
    template< class T >
    void work_until(std::future<T>& wait)
    {
        // If the task is done quickly, just exit.
        if (wait.wait_for(
            std::chrono::milliseconds(1)) == std::future_status::ready)
        {
            return;
        }

        // While there is still waiting to do, do some useful work.
        while(wait.wait_for(std::chrono::seconds::zero()) != std::future_status::ready)
        {            
            if (!m_pool.steal_work())
            {
                if (wait.wait_for(
                    std::chrono::milliseconds(1)) == std::future_status::ready)
                {
                    return;
                }
            }
        }
    }

Feedback - no way to wait for subtasks to complete

Thanks for creating this pool and publishing it for the world! I really appreciate it.

I was considering using it as a replacement for our previous ThreadPool (which does not do work stealing and is very naive). Our current pool returns a std::future<return_type> when a task is enqueued for later execution.

We tend to collect the futures into a list and then wait on them from the spawning thread. I was hoping that you would provide a pool which allowed the following

1. Outer thread creates a number of tasks, some of which might run immediately in the outer thread if the pool is super busy
2. Outer thread eventually reaches and enters some sort of "wait on tasks" routine which waits on one or more subtasks to complete
3. Both threadpool and outer thread stay busy working on subtasks until the wait is done (and the outer thread has also completed its current task)

I think that I'm describing something like the "blocking style" from TBB

I couldn't see how the outer thread is supposed to wait on tasks in the pool to complete (note I would prefer you could wait on a subset of the pool, not the whole pool).

Perhaps I am mis-understanding work stealing, for instance does a call to pool.schedule() not "return" until the given task is done?

Thanks