Parallelization of emcee is working not as fast as expected about emcee

[dfm]

This could have something to do with the communication overhead on your specific system or the load on your CPU from other process. Another option is that this has something to do with how Jupyter handles parallelization. One experiment to try would be running this from a script instead of from Jupyter and monitoring the CPU usage during the run. Hope this helps!

[porpose]

This could have something to do with the communication overhead on your specific system or the load on your CPU from other process. Another option is that this has something to do with how Jupyter handles parallelization. One experiment to try would be running this from a script instead of from Jupyter and monitoring the CPU usage during the run. Hope this helps!

Thanks so much for the reply! I wrapped the code into a .py script file and ran it on my Windows laptop. The output makes me confused:

(Note that the final run was much faster, at 26 iterations/s)

So instead of seeing just the output from one serial run and one parallel run, I got ten different MCMC run outputs. And yes my CPU cores were at full speed during the parallel run:

[dfm]

Good discovery! I think that if you now wrap your script as follows:

if __name__ == "__main__":
    # The rest of the script goes here...

Then it'll only run everything once! This is probably because multiprocessing is forking the process and running all the setup on each worker.

[porpose]

Good discovery! I think that if you now wrap your script as follows:

if __name__ == "__main__":
    # The rest of the script goes here...

Then it'll only run everything once! This is probably because multiprocessing is forking the process and running all the setup on each worker.

Thanks! The result I got in my previous reply was actually exactly the effect of adding if __name__ == "__main__": . (I found this solution on StackOverflow just some moments earlier...) I added this line before the part on multiprocessing.

If I don't add anything, I got an error pop out:

RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.