Deploy Dask using MPI4Py
License: BSD 3-Clause "New" or "Revised" License
Easily deploy Dask Distributed in an existing MPI environment, such as one
created with the mpirun or mpiexec MPI launch commands. See documentation
for more details.
BSD 3-Clause (See License File)
![pre-commit-ci[bot] avatar](https://avatars.githubusercontent.com/in/68672?v=4 "pre-commit-ci[bot]")
Hi All,
As part of an upcoming release I would like to release dask-mpi. What should the version number be? Some options:
1.1 - a moderate bump from the current version2.2 - let's just match Dask version numbers going forwardWhat should we use?
Spotted on dask/dask-cloudprovider#244 (comment). Read The Docs was still building latest from the master branch and builds were failing. It was configured to build from the default branch, but had cached the value master somewhere. Manually pointing it to main in the RtD control panel fixed things.
I'm sitting with @zonca and he's asking how to start a Dask application from within an mpi4py application. I'll give a brief explanation, and then some code snippets.
You can start a scheduler, client, and workers from within your Python script after you do other work, maybe you call a barrier, then on rank 0 start a scheduler, rank 1 start a client (and a bunch of other dask array/dataframe/delayed code) then on all the other ranks you start workers.
There are docs on how to start dask schedulers and workers from Python here: https://docs.dask.org/en/latest/setup/python-advanced.html .
Then @zonca asks
Well how do I get my data from the existing process?
There are many ways to do that, but a simple (if perhaps inelegant way) would be to attach the data on each process to some worker, and then run a task that collects that data on that worker, something like the following:
# on worker
w = Worker('tcp://127.0.0.1:8786', name='rank-' + rank)
w.my_special_data = data # <--- we add this line to the docs mentioned above
w.start()
# on client
from dask.distributed import get_worker
def get_local_data():
worker = get_worker() # get the worker object from which this task is run
return worker.my_special_data
# run that function on every rank that holds a worker
futures = [client.submit(get_local_data, pure=False, workers=['rank-' + str(rank)]) for rank in range(2, comm_size)]Then you can do with those futures as you like
Then @zonca asks
Great, well then how do I cleanly shut down my Dask workers and continue on with my MPI execution?
I thought that calling client.retire_workers() would do this, but apparently it didn't. It looks like the Worker shuts down but the Torando IOLoop continues, which blocks the process. We can probably add a keyword argument to some method to improve this, or you can probably get around it by calling something like:
client.run(lambda : tornado.ioloop.IOLoop.current().stop())Though this is somewhat rude to the workers :)
I am currently experimenting with dask-mpi to replace point-to-point communication between scheduler<-->workers and workers<-->workers to use MPI. The goal is to exploit high-performance interconnects . I am aware of the ucx communication protocol already implemented but the MPI approach will be an alternative to that. Also, I have seen discussions on dask starting MPI jobs (#25) but obviously this post is for doing the opposite.
While starting dask jobs with dask-mpi, the COMM_WORLD has already been initialized in core.py. After that process 0 runs the scheduler code, process 1 runs that client code, and process >= 2 run worker processes. I can also see that the communication code is nicely abstracted out in dask/distributed/comm/ folder (inside dask.distributed). There are three implementations tcp.py, ucx.py, and inproc.py already available. All three of them implement the abstract class Comm. So should I be looking at producing mpi.py as the first step? Somehow pass the COMM_WORLD object to mpi.py and use this communicator for communication. My initial goal is to leave connection establishment as is and only replace data communication with MPI's point-to-point communication. Also there must be some code that multiplexes between these communication protocols based on user's parameters. I would appreciate if I can be pointed to that code.
Also once this is implemented, do we need to do anything special for communication between GPU devices. As per my understanding, GPU devices will be able to communicate as long as the underlying MPI library is CUDA-aware meaning that it supports communication between GPU devices directly. In other words, I am asking if dask-cuda has its own communication mechanism or does it also rely on distributed for its communication.
And also one last clarification please. Is there explicit communication between dask workers? Typically in a loosely coupled systems, workers are only aware of schedulers but going through the code I got this impression that dask workers also communicate with one another. The architecture diagram does not have connections between workers and hence this confusion. And if there is communication between workers, then does mpi.py that we envisaged above enables workers<-->workers communication or scheduler<-->workers communication or both.
Since a "major makeover" was mentioned for dask-mpi in dask/distributed#7192, I thought I would ask if support for non-MPI backends was a possibility. That is, use something like Slurm's srun to launch the ranks even if MPI is not available.
The reason I think this may be possible is because as far as I can tell, MPI is only being used to broadcast the address of the scheduler to the workers. Afterwards, it seems to me that any communication is done via direct TCP. If this is correct, could the startup be achieved instead by, e.g., writing the address to a file in a shared location?
The benefit of this is the elimination of a fairly heavyweight MPI dependency. When users start mixing, e.g. a conda install of MPI with a Lmod install of MPI, their jobs often get confused and fail. It seems like dask-mpi has a chance to sidestep these issues completely.
What happened:
When Dask-MPI is used in batch mode (i.e., using initialize()) on Linux with Python >3.8, it does not properly shut down the scheduler and worker processes when the client script completes. It hangs during shutdown. This means that the Python 3.9 and Python 3.10 tests of dask_mpi/tests/test_core.py and dask_mpi/teststest_no_exit.py hangs and never finish on CI.
Note that this only occurs on Linux. MacOS executes without hanging.
What you expected to happen:
When the client script completes, the scheduler and worker processes should be shut down without error or hanging.
Minimal Complete Verifiable Example:
Manually executing the dask_mpi/tests/core_basic.py script, with Python 3.9+ on Linux, like so:
mpirun -l -np 4 python dask_mpi/tests/core_basic.pyresults in:
[0] 2022-04-20 19:45:00,550 - distributed.scheduler - INFO - State start
[0] 2022-04-20 19:45:00,556 - distributed.scheduler - INFO - Clear task state
[0] 2022-04-20 19:45:00,557 - distributed.scheduler - INFO - Scheduler at: tcp://172.17.0.2:36407
[0] 2022-04-20 19:45:00,557 - distributed.scheduler - INFO - dashboard at: :8787
[2] 2022-04-20 19:45:00,573 - distributed.worker - INFO - Start worker at: tcp://172.17.0.2:45639
[2] 2022-04-20 19:45:00,573 - distributed.worker - INFO - Listening to: tcp://172.17.0.2:45639
[2] 2022-04-20 19:45:00,574 - distributed.worker - INFO - dashboard at: 172.17.0.2:37653
[2] 2022-04-20 19:45:00,574 - distributed.worker - INFO - Waiting to connect to: tcp://172.17.0.2:36407
[2] 2022-04-20 19:45:00,574 - distributed.worker - INFO - -------------------------------------------------
[2] 2022-04-20 19:45:00,574 - distributed.worker - INFO - Threads: 1
[2] 2022-04-20 19:45:00,575 - distributed.worker - INFO - Memory: 0.96 GiB
[2] 2022-04-20 19:45:00,576 - distributed.worker - INFO - Local Directory: /root/dask-mpi/dask_mpi/tests/dask-worker-space/worker-sev4vqjo
[3] 2022-04-20 19:45:00,579 - distributed.worker - INFO - Start worker at: tcp://172.17.0.2:38821
[3] 2022-04-20 19:45:00,580 - distributed.worker - INFO - Listening to: tcp://172.17.0.2:38821
[3] 2022-04-20 19:45:00,580 - distributed.worker - INFO - dashboard at: 172.17.0.2:45157
[3] 2022-04-20 19:45:00,580 - distributed.worker - INFO - Waiting to connect to: tcp://172.17.0.2:36407
[3] 2022-04-20 19:45:00,581 - distributed.worker - INFO - -------------------------------------------------
[3] 2022-04-20 19:45:00,581 - distributed.worker - INFO - Threads: 1
[3] 2022-04-20 19:45:00,581 - distributed.worker - INFO - Memory: 0.96 GiB
[3] 2022-04-20 19:45:00,581 - distributed.worker - INFO - Local Directory: /root/dask-mpi/dask_mpi/tests/dask-worker-space/worker-08kqqntu
[3] 2022-04-20 19:45:00,582 - distributed.worker - INFO - -------------------------------------------------
[2] 2022-04-20 19:45:00,585 - distributed.worker - INFO - -------------------------------------------------
[0] 2022-04-20 19:45:00,998 - distributed.scheduler - INFO - Receive client connection: Client-5d823051-c0e2-11ec-8020-0242ac110002
[0] 2022-04-20 19:45:01,009 - distributed.core - INFO - Starting established connection
[0] 2022-04-20 19:45:01,053 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.17.0.2:45639', name: 2, status: undefined, memory: 0, processing: 0>
[0] 2022-04-20 19:45:01,054 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.17.0.2:45639
[0] 2022-04-20 19:45:01,054 - distributed.core - INFO - Starting established connection
[2] 2022-04-20 19:45:01,055 - distributed.worker - INFO - Registered to: tcp://172.17.0.2:36407
[2] 2022-04-20 19:45:01,056 - distributed.worker - INFO - -------------------------------------------------
[0] 2022-04-20 19:45:01,057 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.17.0.2:38821', name: 3, status: undefined, memory: 0, processing: 0>
[2] 2022-04-20 19:45:01,059 - distributed.core - INFO - Starting established connection
[0] 2022-04-20 19:45:01,060 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.17.0.2:38821
[0] 2022-04-20 19:45:01,060 - distributed.core - INFO - Starting established connection
[3] 2022-04-20 19:45:01,060 - distributed.worker - INFO - Registered to: tcp://172.17.0.2:36407
[3] 2022-04-20 19:45:01,061 - distributed.worker - INFO - -------------------------------------------------
[3] 2022-04-20 19:45:01,063 - distributed.core - INFO - Starting established connection
[0] 2022-04-20 19:45:01,325 - distributed.scheduler - INFO - Remove client Client-5d823051-c0e2-11ec-8020-0242ac110002
[0] 2022-04-20 19:45:01,325 - distributed.scheduler - INFO - Remove client Client-5d823051-c0e2-11ec-8020-0242ac110002
[0] 2022-04-20 19:45:01,326 - distributed.scheduler - INFO - Close client connection: Client-5d823051-c0e2-11ec-8020-0242ac110002
[1] Error in atexit._run_exitfuncs:
[1] Traceback (most recent call last):
[1] File "/root/miniconda/envs/py-3.9/lib/python3.9/site-packages/distributed/utils.py", line 349, in f
[1] result = yield future
[1] File "/root/miniconda/envs/py-3.9/lib/python3.9/site-packages/tornado/gen.py", line 762, in run
[1] value = future.result()
[1] File "/root/miniconda/envs/py-3.9/lib/python3.9/site-packages/distributed/client.py", line 1193, in _start
[1] await self._ensure_connected(timeout=timeout)
[1] File "/root/miniconda/envs/py-3.9/lib/python3.9/site-packages/distributed/client.py", line 1256, in _ensure_connected
[1] comm = await connect(
[1] File "/root/miniconda/envs/py-3.9/lib/python3.9/site-packages/distributed/comm/core.py", line 289, in connect
[1] comm = await asyncio.wait_for(
[1] File "/root/miniconda/envs/py-3.9/lib/python3.9/asyncio/tasks.py", line 479, in wait_for
[1] return fut.result()
[1] File "/root/miniconda/envs/py-3.9/lib/python3.9/site-packages/distributed/comm/tcp.py", line 439, in connect
[1] stream = await self.client.connect(
[1] File "/root/miniconda/envs/py-3.9/lib/python3.9/site-packages/tornado/tcpclient.py", line 265, in connect
[1] addrinfo = await self.resolver.resolve(host, port, af)
[1] File "/root/miniconda/envs/py-3.9/lib/python3.9/site-packages/distributed/comm/tcp.py", line 424, in resolve
[1] for fam, _, _, _, address in await asyncio.get_running_loop().getaddrinfo(
[1] File "/root/miniconda/envs/py-3.9/lib/python3.9/asyncio/base_events.py", line 861, in getaddrinfo
[1] return await self.run_in_executor(
[1] File "/root/miniconda/envs/py-3.9/lib/python3.9/asyncio/base_events.py", line 819, in run_in_executor
[1] executor.submit(func, *args), loop=self)
[1] File "/root/miniconda/envs/py-3.9/lib/python3.9/concurrent/futures/thread.py", line 169, in submit
[1] raise RuntimeError('cannot schedule new futures after '
[1] RuntimeError: cannot schedule new futures after interpreter shutdown
HANGS HERE!!! Requires CTRL-C to exit.
Anything else we need to know?:
I believe this is due to changes in asyncio that occurred with the release of Python 3.9+. In particular, it seems that the asyncio.wait_for function blocks when cancelling a task due to timeout until the task has finished cancellation. (See the Python 3.9 release notes) This appears to be due to the dask_mpi.initialize() shutdown procedure depending upon an asyncio call taking place in an atexit handler. It seems that at the time the atexit handler is called, the asyncio loop has been closed, resulting in the RuntimeError: cannot schedule new futures after interpreter shutdown and the subsequent hanging.
Environment:
2022.4.13.9.12Using pip install dask-mpi
$ pip install dask-mpi
$ mpirun -np 2 dask-mpi --name=test-worker --nthreads=1 --memory-limit=0 --scheduler-file=test.json
distributed.http.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy
distributed.scheduler - INFO - Clear task state
distributed.scheduler - INFO - Scheduler at: tcp://172.16.66.109:36539
distributed.scheduler - INFO - dashboard at: :8787
distributed.nanny - INFO - Start Nanny at: 'tcp://172.16.66.109:36297'
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems. This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):
getting local rank failed
--> Returned value No permission (-17) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems. This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):
orte_ess_init failed
--> Returned value No permission (-17) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems. This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):
ompi_mpi_init: ompi_rte_init failed
--> Returned "No permission" (-17) instead of "Success" (0)
--------------------------------------------------------------------------
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
Using python setup.py install
$ python setup.py install
$ mpirun -np 2 dask-mpi --name=test-worker --nthreads=1 --memory-limit=0 --scheduler-file=test.json
distributed.http.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy
distributed.scheduler - INFO - Clear task state
distributed.scheduler - INFO - Scheduler at: tcp://172.16.66.109:44933
distributed.scheduler - INFO - dashboard at: :8787
distributed.nanny - INFO - Start Nanny at: 'tcp://172.16.66.109:42437'
distributed.diskutils - INFO - Found stale lock file and directory '/home/mpaipuri/downloads/dask-mpi/dask-worker-space/worker-6h2hf4i6', purging
distributed.worker - INFO - Start worker at: tcp://172.16.66.109:37893
distributed.worker - INFO - Listening to: tcp://172.16.66.109:37893
distributed.worker - INFO - dashboard at: 172.16.66.109:45119
distributed.worker - INFO - Waiting to connect to: tcp://172.16.66.109:44933
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Local Directory: /home/mpaipuri/downloads/dask-mpi/dask-worker-space/worker-t48hj0dc
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.16.66.109:37893', name: rascil-worker-1, status: undefined, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, tcp://172.16.66.109:37893
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://172.16.66.109:44933
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
What happened: Installing dask-mpi with wheel packaging fails but it works normally with egg packaging. Tested it on 2 different systems and same behaviour is observed
What you expected to happen: To work with both packaging methods
Anything else we need to know?: The only difference between two approaches is generated dask-mpi command line executable.
I have done everything that I can do without administrative privileges on this repository to remove the CircleCI checks. All testing has been moved to GitHub Actions (which appear to be working) and I have removed the .circleci directory from the repository. I have unfollowed the project on CircleCI...but the CircleCI checks still show up in new PRs and pushes to the master branch. Is this because of a branch protection rule?
If anyone else can fix this, I would greatly appreciate it.
Thanks!
This is in reference to issues discovered when attempting PR #16.
Currently, initialize is defined in the dask_mpi.core module, only. Therefore, importing initialize must be done with the slightly ugly:
from dask_mpi.core import initializeHowever, as @mrocklin noticed in PR #16, it is not really possible to move initialize up into dask_mpi easily.
Currently, I believe the main issues stem from multiple IOLoops being defined in both dask_mpi.core and dask_mpi.cli. ...But I know there is more to this, and it might be harder than just moving the IOLoop definitions into the funtions.
In the documentation, you clearly mention that --nanny flag works only for some MPI implementations because of process forking. As far as I know, OpenMPI implementations with verbs support complains about process forking. But, OpenMPI with UCX support do not have this problem, it is quite happy to allow process forking. We have few MPI codes that do this and we did not find any problems with OpenMPI+UCX. Unfortunately, dask-mpi with --nanny flags still fails with this OpenMPI+UCX implementation. Are there any MPI implementations that you have tested will work for this setting? Thanks
Originally posted by @mahendrapaipuri in #72 (comment)
What happened:
Code is run with mpirun across two nodes each with 24 cores. Dask launches scheduler on one core and 46 workers as expected. Code executes as expected and dask begins to shut down workers. However, one worker is not shut down and continues trying to reconnect with the scheduler until the job is canceled by the machine job scheduler.
What you expected to happen:
All workers close and the program ends.
Minimal Complete Verifiable Example:
from dask_mpi import initialize
initialize(interface='ib0', nanny=False)
from dask.distributed import Client
client = Client()
# The code running is too much to share, but it is essentially as follows
def main():
inputs = generate_inputs()
# worker is an expensive function (2 seconds to 3 minutes depending on inputs)
futures = [client.submit(worker, i) for i in inputs]
results = [f.result() for f in as_completed(futures)]
with open('output.pkl', 'wb') as f:
pickle.dump(results, f)
if __name__ == '__main__':
main()Anything else we need to know?:
The worker that does not stop apparently is never sent a stop command
distributed.worker - INFO - Start worker at: tcp://10.225.6.104:40013
distributed.worker - INFO - Listening to: tcp://10.225.6.104:40013
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.104:40013', name: 17, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.104:40013
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.104:40013', name: 17, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.104:40013
For example, a worker that does stop:
distributed.worker - INFO - Start worker at: tcp://10.225.6.159:45938
distributed.worker - INFO - Listening to: tcp://10.225.6.159:45938
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.159:45938', name: 24, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.159:45938
distributed.worker - INFO - Stopping worker at tcp://10.225.6.159:45938
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.159:45938', name: 24, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.159:45938
The script is launched with: mpirun -n 48 python real_script.py
I also sometimes get messages like: distributed.core - INFO - Event loop was unresponsive in Worker for 37.01s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. Not sure if this is causing it, but it's the only thing that looks out of place in the logs. My workers do also write work to disk.
I also tried using the nanny option, but it does not work on the machine I'm using.
Environment:
Full log:
distributed.http.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy
distributed.scheduler - INFO - Clear task state
distributed.scheduler - INFO - Scheduler at: tcp://10.225.6.104:44912
distributed.scheduler - INFO - dashboard at: :8787
distributed.scheduler - INFO - Receive client connection: Client-d9107e91-2b7c-11eb-97eb-7cd30ab15d36
distributed.core - INFO - Starting established connection
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.diskutils - INFO - Found stale lock file and directory '/gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-ba4tdd5_', purging
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.104:43113
distributed.worker - INFO - Listening to: tcp://10.225.6.104:43113
distributed.worker - INFO - dashboard at: 10.225.6.104:35418
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-z0grodjc
distributed.worker - INFO - -------------------------------------------------
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.104:39284
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.104:38768
distributed.worker - INFO - Listening to: tcp://10.225.6.104:39284
distributed.worker - INFO - Listening to: tcp://10.225.6.104:38768
distributed.worker - INFO - dashboard at: 10.225.6.104:37192
distributed.worker - INFO - dashboard at: 10.225.6.104:39707
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-enblknqx
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-fzhf05_d
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - -------------------------------------------------
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.104:34559
distributed.worker - INFO - Listening to: tcp://10.225.6.104:34559
distributed.worker - INFO - dashboard at: 10.225.6.104:34188
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-xywktaxn
distributed.worker - INFO - -------------------------------------------------
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.104:45376
distributed.worker - INFO - Listening to: tcp://10.225.6.104:45376
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.104:44944
distributed.worker - INFO - dashboard at: 10.225.6.104:35331
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Listening to: tcp://10.225.6.104:44944
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - dashboard at: 10.225.6.104:40318
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-zz36b0gf
distributed.worker - INFO - Threads: 1
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.104:40013
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-n1k201ri
distributed.worker - INFO - Listening to: tcp://10.225.6.104:40013
distributed.worker - INFO - dashboard at: 10.225.6.104:45114
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.104:33488
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Listening to: tcp://10.225.6.104:33488
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - dashboard at: 10.225.6.104:44399
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-9yuzlgug
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - -------------------------------------------------
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.104:43447
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Listening to: tcp://10.225.6.104:43447
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-fjyf9nr6
distributed.worker - INFO - dashboard at: 10.225.6.104:45311
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-m68db4nv
distributed.worker - INFO - -------------------------------------------------
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.104:35031
distributed.worker - INFO - Listening to: tcp://10.225.6.104:35031
distributed.worker - INFO - dashboard at: 10.225.6.104:39546
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-p7trnsmk
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.104:43432
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Listening to: tcp://10.225.6.104:43432
distributed.worker - INFO - dashboard at: 10.225.6.104:40536
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-3tlpc2xa
distributed.worker - INFO - -------------------------------------------------
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.104:42101
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.104:33746
distributed.worker - INFO - Listening to: tcp://10.225.6.104:42101
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.104:33415
distributed.worker - INFO - Listening to: tcp://10.225.6.104:33746
distributed.worker - INFO - dashboard at: 10.225.6.104:42755
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - dashboard at: 10.225.6.104:35304
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Listening to: tcp://10.225.6.104:33415
distributed.worker - INFO - dashboard at: 10.225.6.104:39469
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-56s8wlwo
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-s79044xj
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-wjle8lcv
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - -------------------------------------------------
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.104:38000
distributed.worker - INFO - Listening to: tcp://10.225.6.104:38000
distributed.worker - INFO - dashboard at: 10.225.6.104:46190
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-mhm6h_v8
distributed.worker - INFO - -------------------------------------------------
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.104:42900
distributed.worker - INFO - Listening to: tcp://10.225.6.104:42900
distributed.worker - INFO - dashboard at: 10.225.6.104:35810
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-tv24j2wo
distributed.worker - INFO - -------------------------------------------------
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.104:36236
distributed.worker - INFO - Listening to: tcp://10.225.6.104:36236
distributed.worker - INFO - dashboard at: 10.225.6.104:46029
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-6zl_co85
distributed.worker - INFO - -------------------------------------------------
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.104:36233
distributed.worker - INFO - Listening to: tcp://10.225.6.104:36233
distributed.worker - INFO - dashboard at: 10.225.6.104:40601
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-qc4ncig0
distributed.worker - INFO - -------------------------------------------------
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.104:44390
distributed.worker - INFO - Listening to: tcp://10.225.6.104:44390
distributed.worker - INFO - dashboard at: 10.225.6.104:42854
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-yappalm4
distributed.worker - INFO - -------------------------------------------------
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.104:34029
distributed.worker - INFO - Listening to: tcp://10.225.6.104:34029
distributed.worker - INFO - dashboard at: 10.225.6.104:46262
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-79y2ehjz
distributed.worker - INFO - -------------------------------------------------
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.104:40996
distributed.worker - INFO - Listening to: tcp://10.225.6.104:40996
distributed.worker - INFO - dashboard at: 10.225.6.104:38444
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-wzpgfxr3
distributed.worker - INFO - -------------------------------------------------
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.104:40832
distributed.worker - INFO - Listening to: tcp://10.225.6.104:40832
distributed.worker - INFO - dashboard at: 10.225.6.104:38454
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-8u_k7kul
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.104:43113', name: 10, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.104:43113
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.104:38768', name: 8, memory: 0, processing: 0>
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.104:38768
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.104:39284', name: 11, memory: 0, processing: 0>
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.104:39284
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.104:34559', name: 9, memory: 0, processing: 0>
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.104:34559
distributed.core - INFO - Starting established connection
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.104:45376', name: 6, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.104:45376
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.104:44944', name: 16, memory: 0, processing: 0>
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.104:44944
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.104:40013', name: 17, memory: 0, processing: 0>
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.104:40013
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.104:33488', name: 21, memory: 0, processing: 0>
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.104:33488
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.104:43447', name: 5, memory: 0, processing: 0>
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.104:43447
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.104:35031', name: 7, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.104:35031
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.104:43432', name: 20, memory: 0, processing: 0>
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.104:43432
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.104:42101', name: 22, memory: 0, processing: 0>
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.104:42101
distributed.core - INFO - Starting established connection
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.104:33415', name: 13, memory: 0, processing: 0>
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.104:33415
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.104:33746', name: 3, memory: 0, processing: 0>
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.104:33746
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.104:38000', name: 19, memory: 0, processing: 0>
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.104:38000
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.104:42900', name: 23, memory: 0, processing: 0>
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.104:42900
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.104:36236', name: 4, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.104:36236
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.104:36233', name: 2, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.104:36233
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.104:44390', name: 15, memory: 0, processing: 0>
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.104:44390
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.104:34029', name: 14, memory: 0, processing: 0>
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.104:34029
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.104:40996', name: 18, memory: 0, processing: 0>
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.104:40996
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.104:40832', name: 12, memory: 0, processing: 0>
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.104:40832
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.159:44103
distributed.worker - INFO - Listening to: tcp://10.225.6.159:44103
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.159:44927
distributed.worker - INFO - Listening to: tcp://10.225.6.159:44927
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.159:43228
distributed.worker - INFO - Listening to: tcp://10.225.6.159:43228
distributed.worker - INFO - dashboard at: 10.225.6.159:35445
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.159:40590
distributed.worker - INFO - Listening to: tcp://10.225.6.159:40590
distributed.worker - INFO - dashboard at: 10.225.6.159:38631
distributed.worker - INFO - dashboard at: 10.225.6.159:44185
distributed.worker - INFO - dashboard at: 10.225.6.159:37303
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Threads: 1
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.159:44333
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Listening to: tcp://10.225.6.159:44333
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-uezp5ofx
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-fo3lt7ma
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-4k1yy3bq
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-8i3gm77d
distributed.worker - INFO - dashboard at: 10.225.6.159:38596
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-n8g1mmro
distributed.worker - INFO - -------------------------------------------------
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.159:44440
distributed.worker - INFO - Listening to: tcp://10.225.6.159:44440
distributed.worker - INFO - dashboard at: 10.225.6.159:43946
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-wz3j2411
distributed.worker - INFO - -------------------------------------------------
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.159:44953
distributed.worker - INFO - Listening to: tcp://10.225.6.159:44953
distributed.worker - INFO - dashboard at: 10.225.6.159:45055
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-krh9x27h
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.159:41592
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.159:37663
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Listening to: tcp://10.225.6.159:41592
distributed.worker - INFO - Listening to: tcp://10.225.6.159:37663
distributed.worker - INFO - dashboard at: 10.225.6.159:44769
distributed.worker - INFO - dashboard at: 10.225.6.159:33547
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-7xr_3j0v
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-xekb4lc1
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.159:36325
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Listening to: tcp://10.225.6.159:36325
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - dashboard at: 10.225.6.159:44743
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-qhud_jb2
distributed.worker - INFO - -------------------------------------------------
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.159:34208
distributed.worker - INFO - Listening to: tcp://10.225.6.159:34208
distributed.worker - INFO - dashboard at: 10.225.6.159:34413
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-nkrvs6b0
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.159:40590', name: 34, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.159:40590
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.159:43228', name: 33, memory: 0, processing: 0>
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.159:43228
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.159:44103', name: 26, memory: 0, processing: 0>
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.159:44103
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.159:44927', name: 32, memory: 0, processing: 0>
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.159:44927
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.159:44333', name: 39, memory: 0, processing: 0>
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.159:44333
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.159:42154
distributed.worker - INFO - Listening to: tcp://10.225.6.159:42154
distributed.core - INFO - Starting established connection
distributed.worker - INFO - dashboard at: 10.225.6.159:34141
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.159:34808
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Listening to: tcp://10.225.6.159:34808
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - dashboard at: 10.225.6.159:36731
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-14gfprk1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-6y2g43t0
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.159:44440', name: 27, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.159:44440
distributed.core - INFO - Starting established connection
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.159:34466
distributed.worker - INFO - Listening to: tcp://10.225.6.159:34466
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.159:44953', name: 38, memory: 0, processing: 0>
distributed.worker - INFO - dashboard at: 10.225.6.159:41691
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-mo90cfq0
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.159:44953
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.159:37663', name: 29, memory: 0, processing: 0>
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.159:37663
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.159:41592', name: 28, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.159:41592
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.159:36325', name: 30, memory: 0, processing: 0>
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.159:36325
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.159:34208', name: 44, memory: 0, processing: 0>
distributed.core - INFO - Starting established connection
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.159:42255
distributed.worker - INFO - Listening to: tcp://10.225.6.159:42255
distributed.worker - INFO - dashboard at: 10.225.6.159:46453
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.159:34208
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-dy6qlnby
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.159:42154', name: 42, memory: 0, processing: 0>
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.159:45994
distributed.worker - INFO - Listening to: tcp://10.225.6.159:45994
distributed.worker - INFO - dashboard at: 10.225.6.159:44846
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-hd6zjz98
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.159:42154
distributed.core - INFO - Starting established connection
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.159:34466', name: 45, memory: 0, processing: 0>
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.159:34466
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.159:34808', name: 36, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.159:34808
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.159:45994', name: 46, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.159:45994
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.159:42255', name: 40, memory: 0, processing: 0>
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.159:42255
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.159:37798
distributed.worker - INFO - Listening to: tcp://10.225.6.159:37798
distributed.worker - INFO - dashboard at: 10.225.6.159:38649
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-8m1t189a
distributed.worker - INFO - -------------------------------------------------
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.159:35120
distributed.worker - INFO - Listening to: tcp://10.225.6.159:35120
distributed.worker - INFO - dashboard at: 10.225.6.159:39303
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-hommlr17
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Start worker at: tcp://10.225.6.159:46204
distributed.worker - INFO - Listening to: tcp://10.225.6.159:46204
distributed.worker - INFO - dashboard at: 10.225.6.159:43005
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.159:42744
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Listening to: tcp://10.225.6.159:42744
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - dashboard at: 10.225.6.159:39842
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-_j7iqol0
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-d63_uex6
distributed.worker - INFO - -------------------------------------------------
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.159:33401
distributed.worker - INFO - Listening to: tcp://10.225.6.159:33401
distributed.worker - INFO - dashboard at: 10.225.6.159:46755
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.159:37798', name: 35, memory: 0, processing: 0>
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-vr5hv790
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.159:37798
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.159:46204', name: 43, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.159:46204
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.159:35120', name: 37, memory: 0, processing: 0>
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.159:35120
distributed.core - INFO - Starting established connection
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.159:35201
distributed.worker - INFO - Listening to: tcp://10.225.6.159:35201
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.159:42744', name: 25, memory: 0, processing: 0>
distributed.worker - INFO - dashboard at: 10.225.6.159:39119
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-ypr2xh75
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.159:42744
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.159:33401', name: 47, memory: 0, processing: 0>
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.159:33401
distributed.core - INFO - Starting established connection
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.159:40599
distributed.worker - INFO - Listening to: tcp://10.225.6.159:40599
distributed.worker - INFO - dashboard at: 10.225.6.159:45595
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-8rm12_ao
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.159:35201', name: 41, memory: 0, processing: 0>
/projects/kygo7210/.conda/envs/simo_automator/lib/python3.8/site-packages/distributed/worker.py:482: UserWarning: The local_dir keyword has moved to local_directory
warnings.warn("The local_dir keyword has moved to local_directory")
distributed.worker - INFO - Start worker at: tcp://10.225.6.159:45938
distributed.worker - INFO - Listening to: tcp://10.225.6.159:45938
distributed.worker - INFO - dashboard at: 10.225.6.159:42668
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.159:35201
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 122.00 GB
distributed.worker - INFO - Local Directory: /gpfs/summit/scratch/kygo7210/SIMO-Automator/dask-worker-space/worker-po5qe660
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.159:40599', name: 31, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.159:40599
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.225.6.159:45938', name: 24, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.225.6.159:45938
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://10.225.6.104:44912
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.core - INFO - Event loop was unresponsive in Worker for 37.01s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 37.02s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 37.02s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 37.03s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 37.03s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 37.02s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 36.95s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 37.02s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 36.95s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 36.94s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 37.01s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 36.96s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 37.02s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 36.95s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 37.02s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 36.95s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 37.05s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 37.02s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 36.96s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 37.03s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 37.02s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 37.02s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
[11/20/2020 03:09:14 PM] Run 1/420
[11/20/2020 03:09:14 PM] Run 2/420
[11/20/2020 03:09:14 PM] Run 3/420
[11/20/2020 03:09:14 PM] Run 4/420
[11/20/2020 03:09:14 PM] Run 5/420
[11/20/2020 03:09:14 PM] Run 6/420
[11/20/2020 03:09:16 PM] Run 7/420
[11/20/2020 03:09:16 PM] Run 8/420
[11/20/2020 03:09:16 PM] Run 9/420
[11/20/2020 03:09:16 PM] Run 10/420
[11/20/2020 03:09:16 PM] Run 11/420
[11/20/2020 03:09:16 PM] Run 12/420
[11/20/2020 03:09:16 PM] Run 13/420
[11/20/2020 03:09:17 PM] Run 14/420
[11/20/2020 03:09:17 PM] Run 15/420
[11/20/2020 03:09:18 PM] Run 16/420
[11/20/2020 03:09:18 PM] Run 17/420
[11/20/2020 03:09:18 PM] Run 18/420
[11/20/2020 03:09:18 PM] Run 19/420
[11/20/2020 03:09:18 PM] Run 20/420
[11/20/2020 03:09:20 PM] Run 21/420
[11/20/2020 03:09:20 PM] Run 22/420
[11/20/2020 03:09:21 PM] Run 23/420
[11/20/2020 03:09:22 PM] Run 24/420
[11/20/2020 03:09:22 PM] Run 25/420
[11/20/2020 03:09:22 PM] Run 26/420
[11/20/2020 03:09:23 PM] Run 27/420
[11/20/2020 03:09:23 PM] Run 28/420
[11/20/2020 03:09:23 PM] Run 29/420
[11/20/2020 03:09:23 PM] Run 30/420
distributed.core - INFO - Event loop was unresponsive in Worker for 48.80s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 48.73s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 48.73s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 48.73s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 48.80s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 48.74s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 48.73s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 48.74s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 48.81s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 48.74s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 48.81s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 48.81s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 48.74s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 48.73s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 48.79s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 48.80s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 48.80s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 48.79s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 48.81s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 48.74s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 48.73s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 48.73s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 48.80s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.core - INFO - Event loop was unresponsive in Worker for 48.81s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
[11/20/2020 03:09:23 PM] Run 31/420
[11/20/2020 03:09:23 PM] Run 32/420
[11/20/2020 03:09:23 PM] Run 33/420
[11/20/2020 03:09:24 PM] Run 34/420
[11/20/2020 03:09:25 PM] Run 35/420
[11/20/2020 03:09:25 PM] Run 36/420
[11/20/2020 03:09:25 PM] Run 37/420
[11/20/2020 03:09:25 PM] Run 38/420
[11/20/2020 03:09:25 PM] Run 39/420
[11/20/2020 03:09:25 PM] Run 40/420
[11/20/2020 03:09:25 PM] Run 41/420
[11/20/2020 03:09:25 PM] Run 42/420
[11/20/2020 03:09:25 PM] Run 43/420
[11/20/2020 03:09:25 PM] Run 44/420
[11/20/2020 03:09:25 PM] Run 45/420
[11/20/2020 03:09:25 PM] Run 46/420
[11/20/2020 03:09:25 PM] Run 47/420
[11/20/2020 03:09:25 PM] Run 48/420
[11/20/2020 03:09:25 PM] Run 49/420
[11/20/2020 03:09:26 PM] Run 50/420
[11/20/2020 03:09:26 PM] Run 51/420
[11/20/2020 03:09:26 PM] Run 52/420
[11/20/2020 03:09:26 PM] Run 53/420
[11/20/2020 03:09:26 PM] Run 54/420
[11/20/2020 03:09:26 PM] Run 55/420
[11/20/2020 03:09:26 PM] Run 56/420
[11/20/2020 03:09:27 PM] Run 57/420
[11/20/2020 03:09:27 PM] Run 58/420
[11/20/2020 03:09:27 PM] Run 59/420
[11/20/2020 03:09:27 PM] Run 60/420
[11/20/2020 03:09:28 PM] Run 61/420
[11/20/2020 03:09:28 PM] Run 62/420
[11/20/2020 03:09:28 PM] Run 63/420
[11/20/2020 03:09:28 PM] Run 64/420
[11/20/2020 03:09:28 PM] Run 65/420
[11/20/2020 03:09:28 PM] Run 66/420
[11/20/2020 03:09:28 PM] Run 67/420
[11/20/2020 03:09:28 PM] Run 68/420
[11/20/2020 03:09:28 PM] Run 69/420
[11/20/2020 03:09:29 PM] Run 70/420
[11/20/2020 03:09:29 PM] Run 71/420
[11/20/2020 03:09:29 PM] Run 72/420
[11/20/2020 03:09:29 PM] Run 73/420
[11/20/2020 03:09:29 PM] Run 74/420
[11/20/2020 03:09:29 PM] Run 75/420
[11/20/2020 03:09:29 PM] Run 76/420
[11/20/2020 03:09:30 PM] Run 77/420
[11/20/2020 03:09:30 PM] Run 78/420
[11/20/2020 03:09:30 PM] Run 79/420
[11/20/2020 03:09:30 PM] Run 80/420
[11/20/2020 03:09:30 PM] Run 81/420
[11/20/2020 03:09:30 PM] Run 82/420
[11/20/2020 03:09:30 PM] Run 83/420
[11/20/2020 03:09:30 PM] Run 84/420
[11/20/2020 03:09:30 PM] Run 85/420
[11/20/2020 03:09:30 PM] Run 86/420
[11/20/2020 03:09:30 PM] Run 87/420
[11/20/2020 03:09:30 PM] Run 88/420
[11/20/2020 03:09:30 PM] Run 89/420
[11/20/2020 03:09:30 PM] Run 90/420
[11/20/2020 03:09:30 PM] Run 91/420
[11/20/2020 03:09:31 PM] Run 92/420
[11/20/2020 03:09:31 PM] Run 93/420
[11/20/2020 03:09:31 PM] Run 94/420
[11/20/2020 03:09:31 PM] Run 95/420
[11/20/2020 03:09:31 PM] Run 96/420
[11/20/2020 03:09:31 PM] Run 97/420
[11/20/2020 03:09:31 PM] Run 98/420
[11/20/2020 03:09:31 PM] Run 99/420
[11/20/2020 03:09:31 PM] Run 100/420
[11/20/2020 03:09:31 PM] Run 101/420
[11/20/2020 03:09:32 PM] Run 102/420
[11/20/2020 03:09:32 PM] Run 103/420
[11/20/2020 03:09:32 PM] Run 104/420
[11/20/2020 03:09:32 PM] Run 105/420
[11/20/2020 03:09:32 PM] Run 106/420
[11/20/2020 03:09:33 PM] Run 107/420
[11/20/2020 03:09:33 PM] Run 108/420
[11/20/2020 03:09:33 PM] Run 109/420
[11/20/2020 03:09:33 PM] Run 110/420
[11/20/2020 03:09:33 PM] Run 111/420
[11/20/2020 03:09:33 PM] Run 112/420
[11/20/2020 03:09:33 PM] Run 113/420
[11/20/2020 03:09:33 PM] Run 114/420
[11/20/2020 03:09:33 PM] Run 115/420
[11/20/2020 03:09:34 PM] Run 116/420
[11/20/2020 03:09:34 PM] Run 117/420
[11/20/2020 03:09:34 PM] Run 118/420
[11/20/2020 03:09:34 PM] Run 119/420
[11/20/2020 03:09:34 PM] Run 120/420
[11/20/2020 03:09:35 PM] Run 121/420
[11/20/2020 03:09:35 PM] Run 122/420
[11/20/2020 03:09:35 PM] Run 123/420
[11/20/2020 03:09:35 PM] Run 124/420
[11/20/2020 03:09:35 PM] Run 125/420
[11/20/2020 03:09:35 PM] Run 126/420
[11/20/2020 03:09:35 PM] Run 127/420
[11/20/2020 03:09:35 PM] Run 128/420
[11/20/2020 03:09:35 PM] Run 129/420
[11/20/2020 03:09:35 PM] Run 130/420
[11/20/2020 03:09:35 PM] Run 131/420
[11/20/2020 03:09:36 PM] Run 132/420
[11/20/2020 03:09:36 PM] Run 133/420
[11/20/2020 03:09:36 PM] Run 134/420
[11/20/2020 03:09:36 PM] Run 135/420
[11/20/2020 03:09:36 PM] Run 136/420
[11/20/2020 03:09:36 PM] Run 137/420
[11/20/2020 03:09:37 PM] Run 138/420
[11/20/2020 03:09:37 PM] Run 139/420
[11/20/2020 03:09:37 PM] Run 140/420
[11/20/2020 03:09:37 PM] Run 141/420
[11/20/2020 03:09:37 PM] Run 142/420
[11/20/2020 03:09:37 PM] Run 143/420
[11/20/2020 03:09:37 PM] Run 144/420
[11/20/2020 03:09:37 PM] Run 145/420
[11/20/2020 03:09:38 PM] Run 146/420
[11/20/2020 03:09:38 PM] Run 147/420
[11/20/2020 03:09:38 PM] Run 148/420
[11/20/2020 03:09:38 PM] Run 149/420
[11/20/2020 03:09:38 PM] Run 150/420
[11/20/2020 03:09:38 PM] Run 151/420
[11/20/2020 03:09:38 PM] Run 152/420
[11/20/2020 03:09:38 PM] Run 153/420
[11/20/2020 03:09:38 PM] Run 154/420
[11/20/2020 03:09:38 PM] Run 155/420
[11/20/2020 03:09:38 PM] Run 156/420
[11/20/2020 03:09:38 PM] Run 157/420
[11/20/2020 03:09:38 PM] Run 158/420
[11/20/2020 03:09:38 PM] Run 159/420
[11/20/2020 03:09:38 PM] Run 160/420
[11/20/2020 03:09:38 PM] Run 161/420
[11/20/2020 03:09:39 PM] Run 162/420
[11/20/2020 03:09:39 PM] Run 163/420
[11/20/2020 03:09:39 PM] Run 164/420
[11/20/2020 03:09:39 PM] Run 165/420
[11/20/2020 03:09:39 PM] Run 166/420
[11/20/2020 03:09:39 PM] Run 167/420
[11/20/2020 03:09:39 PM] Run 168/420
[11/20/2020 03:09:39 PM] Run 169/420
[11/20/2020 03:09:39 PM] Run 170/420
[11/20/2020 03:09:39 PM] Run 171/420
[11/20/2020 03:09:39 PM] Run 172/420
[11/20/2020 03:09:40 PM] Run 173/420
[11/20/2020 03:09:40 PM] Run 174/420
[11/20/2020 03:09:40 PM] Run 175/420
[11/20/2020 03:09:40 PM] Run 176/420
[11/20/2020 03:09:40 PM] Run 177/420
[11/20/2020 03:09:40 PM] Run 178/420
[11/20/2020 03:09:40 PM] Run 179/420
[11/20/2020 03:09:40 PM] Run 180/420
[11/20/2020 03:09:40 PM] Run 181/420
[11/20/2020 03:09:40 PM] Run 182/420
[11/20/2020 03:09:40 PM] Run 183/420
[11/20/2020 03:09:40 PM] Run 184/420
[11/20/2020 03:09:41 PM] Run 185/420
[11/20/2020 03:09:41 PM] Run 186/420
[11/20/2020 03:09:41 PM] Run 187/420
[11/20/2020 03:09:41 PM] Run 188/420
[11/20/2020 03:09:41 PM] Run 189/420
[11/20/2020 03:09:41 PM] Run 190/420
[11/20/2020 03:09:41 PM] Run 191/420
[11/20/2020 03:09:41 PM] Run 192/420
[11/20/2020 03:09:41 PM] Run 193/420
[11/20/2020 03:09:41 PM] Run 194/420
[11/20/2020 03:09:42 PM] Run 195/420
[11/20/2020 03:09:42 PM] Run 196/420
[11/20/2020 03:09:42 PM] Run 197/420
[11/20/2020 03:09:42 PM] Run 198/420
[11/20/2020 03:09:42 PM] Run 199/420
[11/20/2020 03:09:42 PM] Run 200/420
[11/20/2020 03:09:42 PM] Run 201/420
[11/20/2020 03:09:42 PM] Run 202/420
[11/20/2020 03:09:42 PM] Run 203/420
[11/20/2020 03:09:43 PM] Run 204/420
[11/20/2020 03:09:43 PM] Run 205/420
[11/20/2020 03:09:43 PM] Run 206/420
[11/20/2020 03:09:43 PM] Run 207/420
[11/20/2020 03:09:43 PM] Run 208/420
[11/20/2020 03:09:43 PM] Run 209/420
[11/20/2020 03:09:43 PM] Run 210/420
[11/20/2020 03:09:43 PM] Run 211/420
[11/20/2020 03:09:43 PM] Run 212/420
[11/20/2020 03:09:43 PM] Run 213/420
[11/20/2020 03:09:43 PM] Run 214/420
[11/20/2020 03:09:43 PM] Run 215/420
[11/20/2020 03:09:43 PM] Run 216/420
[11/20/2020 03:09:43 PM] Run 217/420
[11/20/2020 03:09:44 PM] Run 218/420
[11/20/2020 03:09:44 PM] Run 219/420
[11/20/2020 03:09:44 PM] Run 220/420
[11/20/2020 03:09:44 PM] Run 221/420
[11/20/2020 03:09:44 PM] Run 222/420
[11/20/2020 03:09:44 PM] Run 223/420
[11/20/2020 03:09:44 PM] Run 224/420
[11/20/2020 03:09:45 PM] Run 225/420
[11/20/2020 03:09:45 PM] Run 226/420
[11/20/2020 03:09:45 PM] Run 227/420
[11/20/2020 03:09:45 PM] Run 228/420
[11/20/2020 03:09:45 PM] Run 229/420
[11/20/2020 03:09:45 PM] Run 230/420
[11/20/2020 03:09:45 PM] Run 231/420
[11/20/2020 03:09:45 PM] Run 232/420
[11/20/2020 03:09:45 PM] Run 233/420
[11/20/2020 03:09:45 PM] Run 234/420
[11/20/2020 03:09:45 PM] Run 235/420
[11/20/2020 03:09:45 PM] Run 236/420
[11/20/2020 03:09:46 PM] Run 237/420
[11/20/2020 03:09:46 PM] Run 238/420
[11/20/2020 03:09:46 PM] Run 239/420
[11/20/2020 03:09:46 PM] Run 240/420
[11/20/2020 03:09:46 PM] Run 241/420
[11/20/2020 03:09:46 PM] Run 242/420
[11/20/2020 03:09:46 PM] Run 243/420
[11/20/2020 03:09:46 PM] Run 244/420
[11/20/2020 03:09:46 PM] Run 245/420
[11/20/2020 03:09:46 PM] Run 246/420
[11/20/2020 03:09:46 PM] Run 247/420
[11/20/2020 03:09:46 PM] Run 248/420
[11/20/2020 03:09:47 PM] Run 249/420
[11/20/2020 03:09:47 PM] Run 250/420
[11/20/2020 03:09:47 PM] Run 251/420
[11/20/2020 03:09:47 PM] Run 252/420
[11/20/2020 03:09:47 PM] Run 253/420
[11/20/2020 03:09:47 PM] Run 254/420
[11/20/2020 03:09:47 PM] Run 255/420
[11/20/2020 03:09:47 PM] Run 256/420
[11/20/2020 03:09:47 PM] Run 257/420
[11/20/2020 03:09:47 PM] Run 258/420
[11/20/2020 03:09:48 PM] Run 259/420
[11/20/2020 03:09:48 PM] Run 260/420
[11/20/2020 03:09:48 PM] Run 261/420
[11/20/2020 03:09:48 PM] Run 262/420
[11/20/2020 03:09:48 PM] Run 263/420
[11/20/2020 03:09:48 PM] Run 264/420
[11/20/2020 03:09:48 PM] Run 265/420
[11/20/2020 03:09:48 PM] Run 266/420
[11/20/2020 03:09:49 PM] Run 267/420
[11/20/2020 03:09:49 PM] Run 268/420
[11/20/2020 03:09:49 PM] Run 269/420
[11/20/2020 03:09:49 PM] Run 270/420
[11/20/2020 03:09:49 PM] Run 271/420
[11/20/2020 03:09:49 PM] Run 272/420
[11/20/2020 03:09:50 PM] Run 273/420
[11/20/2020 03:09:50 PM] Run 274/420
[11/20/2020 03:09:50 PM] Run 275/420
[11/20/2020 03:09:50 PM] Run 276/420
[11/20/2020 03:09:50 PM] Run 277/420
[11/20/2020 03:09:50 PM] Run 278/420
[11/20/2020 03:09:50 PM] Run 279/420
[11/20/2020 03:09:50 PM] Run 280/420
[11/20/2020 03:09:50 PM] Run 281/420
[11/20/2020 03:09:50 PM] Run 282/420
[11/20/2020 03:09:50 PM] Run 283/420
[11/20/2020 03:09:50 PM] Run 284/420
[11/20/2020 03:09:50 PM] Run 285/420
[11/20/2020 03:09:51 PM] Run 286/420
[11/20/2020 03:09:51 PM] Run 287/420
[11/20/2020 03:09:51 PM] Run 288/420
[11/20/2020 03:09:51 PM] Run 289/420
[11/20/2020 03:09:51 PM] Run 290/420
[11/20/2020 03:09:51 PM] Run 291/420
[11/20/2020 03:09:51 PM] Run 292/420
[11/20/2020 03:09:51 PM] Run 293/420
[11/20/2020 03:09:51 PM] Run 294/420
[11/20/2020 03:09:51 PM] Run 295/420
[11/20/2020 03:09:52 PM] Run 296/420
[11/20/2020 03:09:52 PM] Run 297/420
[11/20/2020 03:09:52 PM] Run 298/420
[11/20/2020 03:09:52 PM] Run 299/420
[11/20/2020 03:09:52 PM] Run 300/420
[11/20/2020 03:09:52 PM] Run 301/420
[11/20/2020 03:09:52 PM] Run 302/420
[11/20/2020 03:09:52 PM] Run 303/420
[11/20/2020 03:09:52 PM] Run 304/420
[11/20/2020 03:09:53 PM] Run 305/420
[11/20/2020 03:09:53 PM] Run 306/420
[11/20/2020 03:09:53 PM] Run 307/420
[11/20/2020 03:09:53 PM] Run 308/420
[11/20/2020 03:09:53 PM] Run 309/420
[11/20/2020 03:09:53 PM] Run 310/420
[11/20/2020 03:09:53 PM] Run 311/420
[11/20/2020 03:09:53 PM] Run 312/420
[11/20/2020 03:09:53 PM] Run 313/420
[11/20/2020 03:09:53 PM] Run 314/420
[11/20/2020 03:09:53 PM] Run 315/420
[11/20/2020 03:09:53 PM] Run 316/420
[11/20/2020 03:09:53 PM] Run 317/420
[11/20/2020 03:09:53 PM] Run 318/420
[11/20/2020 03:09:54 PM] Run 319/420
[11/20/2020 03:09:54 PM] Run 320/420
[11/20/2020 03:09:54 PM] Run 321/420
[11/20/2020 03:09:54 PM] Run 322/420
[11/20/2020 03:09:54 PM] Run 323/420
[11/20/2020 03:09:54 PM] Run 324/420
[11/20/2020 03:09:54 PM] Run 325/420
[11/20/2020 03:09:55 PM] Run 326/420
[11/20/2020 03:09:55 PM] Run 327/420
[11/20/2020 03:09:55 PM] Run 328/420
[11/20/2020 03:09:55 PM] Run 329/420
[11/20/2020 03:09:55 PM] Run 330/420
[11/20/2020 03:09:55 PM] Run 331/420
[11/20/2020 03:09:55 PM] Run 332/420
[11/20/2020 03:09:55 PM] Run 333/420
[11/20/2020 03:09:55 PM] Run 334/420
[11/20/2020 03:09:55 PM] Run 335/420
[11/20/2020 03:09:55 PM] Run 336/420
[11/20/2020 03:09:55 PM] Run 337/420
[11/20/2020 03:09:55 PM] Run 338/420
[11/20/2020 03:09:55 PM] Run 339/420
[11/20/2020 03:09:56 PM] Run 340/420
[11/20/2020 03:09:56 PM] Run 341/420
[11/20/2020 03:09:56 PM] Run 342/420
[11/20/2020 03:09:56 PM] Run 343/420
[11/20/2020 03:09:56 PM] Run 344/420
[11/20/2020 03:09:56 PM] Run 345/420
[11/20/2020 03:09:56 PM] Run 346/420
[11/20/2020 03:09:56 PM] Run 347/420
[11/20/2020 03:09:56 PM] Run 348/420
[11/20/2020 03:09:56 PM] Run 349/420
[11/20/2020 03:09:56 PM] Run 350/420
[11/20/2020 03:09:57 PM] Run 351/420
[11/20/2020 03:09:57 PM] Run 352/420
[11/20/2020 03:09:57 PM] Run 353/420
[11/20/2020 03:09:57 PM] Run 354/420
[11/20/2020 03:09:57 PM] Run 355/420
[11/20/2020 03:09:57 PM] Run 356/420
[11/20/2020 03:09:57 PM] Run 357/420
[11/20/2020 03:09:57 PM] Run 358/420
[11/20/2020 03:09:57 PM] Run 359/420
[11/20/2020 03:09:57 PM] Run 360/420
[11/20/2020 03:09:57 PM] Run 361/420
[11/20/2020 03:09:57 PM] Run 362/420
[11/20/2020 03:09:58 PM] Run 363/420
[11/20/2020 03:09:58 PM] Run 364/420
[11/20/2020 03:09:58 PM] Run 365/420
[11/20/2020 03:09:58 PM] Run 366/420
[11/20/2020 03:09:58 PM] Run 367/420
[11/20/2020 03:09:58 PM] Run 368/420
[11/20/2020 03:09:59 PM] Run 369/420
[11/20/2020 03:09:59 PM] Run 370/420
[11/20/2020 03:09:59 PM] Run 371/420
[11/20/2020 03:09:59 PM] Run 372/420
[11/20/2020 03:09:59 PM] Run 373/420
[11/20/2020 03:09:59 PM] Run 374/420
[11/20/2020 03:09:59 PM] Run 375/420
[11/20/2020 03:09:59 PM] Run 376/420
[11/20/2020 03:09:59 PM] Run 377/420
[11/20/2020 03:09:59 PM] Run 378/420
[11/20/2020 03:09:59 PM] Run 379/420
[11/20/2020 03:09:59 PM] Run 380/420
[11/20/2020 03:09:59 PM] Run 381/420
[11/20/2020 03:10:00 PM] Run 382/420
[11/20/2020 03:10:00 PM] Run 383/420
[11/20/2020 03:10:00 PM] Run 384/420
[11/20/2020 03:10:00 PM] Run 385/420
[11/20/2020 03:10:00 PM] Run 386/420
[11/20/2020 03:10:00 PM] Run 387/420
[11/20/2020 03:10:00 PM] Run 388/420
[11/20/2020 03:10:00 PM] Run 389/420
[11/20/2020 03:10:01 PM] Run 390/420
[11/20/2020 03:10:01 PM] Run 391/420
[11/20/2020 03:10:01 PM] Run 392/420
[11/20/2020 03:10:01 PM] Run 393/420
[11/20/2020 03:10:01 PM] Run 394/420
[11/20/2020 03:10:01 PM] Run 395/420
[11/20/2020 03:10:02 PM] Run 396/420
[11/20/2020 03:10:02 PM] Run 397/420
[11/20/2020 03:10:02 PM] Run 398/420
[11/20/2020 03:10:02 PM] Run 399/420
[11/20/2020 03:10:02 PM] Run 400/420
[11/20/2020 03:10:02 PM] Run 401/420
[11/20/2020 03:10:02 PM] Run 402/420
[11/20/2020 03:10:03 PM] Run 403/420
[11/20/2020 03:10:03 PM] Run 404/420
[11/20/2020 03:10:03 PM] Run 405/420
[11/20/2020 03:10:04 PM] Run 406/420
[11/20/2020 03:10:04 PM] Run 407/420
[11/20/2020 03:10:04 PM] Run 408/420
[11/20/2020 03:10:05 PM] Run 409/420
[11/20/2020 03:10:05 PM] Run 410/420
[11/20/2020 03:10:06 PM] Run 411/420
[11/20/2020 03:10:08 PM] Run 412/420
[11/20/2020 03:10:08 PM] Run 413/420
[11/20/2020 03:10:08 PM] Run 414/420
[11/20/2020 03:10:10 PM] Run 415/420
[11/20/2020 03:10:51 PM] Run 416/420
[11/20/2020 03:10:54 PM] Run 417/420
[11/20/2020 03:10:55 PM] Run 418/420
[11/20/2020 03:10:56 PM] Run 419/420
[11/20/2020 03:11:05 PM] Run 420/420
[11/20/2020 03:11:05 PM] Processed Run 1/420
[11/20/2020 03:11:06 PM] Processed Run 2/420
[11/20/2020 03:11:07 PM] Processed Run 3/420
[11/20/2020 03:11:08 PM] Processed Run 4/420
[11/20/2020 03:11:09 PM] Processed Run 5/420
[11/20/2020 03:11:11 PM] Processed Run 6/420
[11/20/2020 03:11:11 PM] Processed Run 7/420
[11/20/2020 03:11:11 PM] Processed Run 8/420
[11/20/2020 03:11:11 PM] Processed Run 9/420
[11/20/2020 03:11:11 PM] Processed Run 10/420
[11/20/2020 03:11:12 PM] Processed Run 11/420
[11/20/2020 03:11:12 PM] Processed Run 12/420
[11/20/2020 03:11:12 PM] Processed Run 13/420
[11/20/2020 03:11:12 PM] Processed Run 14/420
[11/20/2020 03:11:12 PM] Processed Run 15/420
[11/20/2020 03:11:12 PM] Processed Run 16/420
[11/20/2020 03:11:12 PM] Processed Run 17/420
[11/20/2020 03:11:12 PM] Processed Run 18/420
[11/20/2020 03:11:12 PM] Processed Run 19/420
[11/20/2020 03:11:12 PM] Processed Run 20/420
[11/20/2020 03:11:12 PM] Processed Run 21/420
[11/20/2020 03:11:12 PM] Processed Run 22/420
[11/20/2020 03:11:13 PM] Processed Run 23/420
[11/20/2020 03:11:13 PM] Processed Run 24/420
[11/20/2020 03:11:13 PM] Processed Run 25/420
[11/20/2020 03:11:13 PM] Processed Run 26/420
[11/20/2020 03:11:13 PM] Processed Run 27/420
[11/20/2020 03:11:13 PM] Processed Run 28/420
[11/20/2020 03:11:13 PM] Processed Run 29/420
[11/20/2020 03:11:13 PM] Processed Run 30/420
[11/20/2020 03:11:13 PM] Processed Run 31/420
[11/20/2020 03:11:21 PM] Processed Run 32/420
[11/20/2020 03:11:33 PM] Processed Run 33/420
[11/20/2020 03:11:41 PM] Processed Run 34/420
[11/20/2020 03:11:50 PM] Processed Run 35/420
[11/20/2020 03:12:01 PM] Processed Run 36/420
[11/20/2020 03:12:02 PM] Processed Run 37/420
[11/20/2020 03:12:02 PM] Processed Run 38/420
[11/20/2020 03:12:02 PM] Processed Run 39/420
[11/20/2020 03:12:10 PM] Processed Run 40/420
[11/20/2020 03:12:10 PM] Processed Run 41/420
[11/20/2020 03:12:10 PM] Processed Run 42/420
[11/20/2020 03:12:10 PM] Processed Run 43/420
[11/20/2020 03:12:10 PM] Processed Run 44/420
[11/20/2020 03:12:10 PM] Processed Run 45/420
[11/20/2020 03:12:10 PM] Processed Run 46/420
[11/20/2020 03:12:10 PM] Processed Run 47/420
[11/20/2020 03:12:10 PM] Processed Run 48/420
[11/20/2020 03:12:10 PM] Processed Run 49/420
[11/20/2020 03:12:11 PM] Processed Run 50/420
[11/20/2020 03:12:11 PM] Processed Run 51/420
[11/20/2020 03:12:11 PM] Processed Run 52/420
[11/20/2020 03:12:11 PM] Processed Run 53/420
[11/20/2020 03:12:11 PM] Processed Run 54/420
[11/20/2020 03:12:11 PM] Processed Run 55/420
[11/20/2020 03:12:11 PM] Processed Run 56/420
[11/20/2020 03:12:11 PM] Processed Run 57/420
[11/20/2020 03:12:11 PM] Processed Run 58/420
[11/20/2020 03:12:11 PM] Processed Run 59/420
[11/20/2020 03:12:11 PM] Processed Run 60/420
[11/20/2020 03:12:11 PM] Processed Run 61/420
[11/20/2020 03:12:11 PM] Processed Run 62/420
[11/20/2020 03:12:12 PM] Processed Run 63/420
[11/20/2020 03:12:12 PM] Processed Run 64/420
[11/20/2020 03:12:13 PM] Processed Run 65/420
[11/20/2020 03:12:14 PM] Processed Run 66/420
[11/20/2020 03:12:14 PM] Processed Run 67/420
[11/20/2020 03:12:14 PM] Processed Run 68/420
[11/20/2020 03:12:14 PM] Processed Run 69/420
[11/20/2020 03:12:14 PM] Processed Run 70/420
[11/20/2020 03:12:14 PM] Processed Run 71/420
[11/20/2020 03:12:14 PM] Processed Run 72/420
[11/20/2020 03:12:14 PM] Processed Run 73/420
[11/20/2020 03:12:14 PM] Processed Run 74/420
[11/20/2020 03:12:14 PM] Processed Run 75/420
[11/20/2020 03:12:14 PM] Processed Run 76/420
[11/20/2020 03:12:14 PM] Processed Run 77/420
[11/20/2020 03:12:14 PM] Processed Run 78/420
[11/20/2020 03:12:14 PM] Processed Run 79/420
[11/20/2020 03:12:14 PM] Processed Run 80/420
[11/20/2020 03:12:14 PM] Processed Run 81/420
[11/20/2020 03:12:14 PM] Processed Run 82/420
[11/20/2020 03:12:14 PM] Processed Run 83/420
[11/20/2020 03:12:14 PM] Processed Run 84/420
[11/20/2020 03:12:14 PM] Processed Run 85/420
[11/20/2020 03:12:14 PM] Processed Run 86/420
[11/20/2020 03:12:14 PM] Processed Run 87/420
[11/20/2020 03:12:14 PM] Processed Run 88/420
[11/20/2020 03:12:14 PM] Processed Run 89/420
[11/20/2020 03:12:14 PM] Processed Run 90/420
[11/20/2020 03:12:14 PM] Processed Run 91/420
[11/20/2020 03:12:25 PM] Processed Run 92/420
[11/20/2020 03:12:25 PM] Processed Run 93/420
[11/20/2020 03:12:26 PM] Processed Run 94/420
[11/20/2020 03:12:26 PM] Processed Run 95/420
[11/20/2020 03:12:27 PM] Processed Run 96/420
[11/20/2020 03:12:27 PM] Processed Run 97/420
[11/20/2020 03:12:27 PM] Processed Run 98/420
[11/20/2020 03:12:27 PM] Processed Run 99/420
[11/20/2020 03:12:27 PM] Processed Run 100/420
[11/20/2020 03:12:27 PM] Processed Run 101/420
[11/20/2020 03:12:27 PM] Processed Run 102/420
[11/20/2020 03:12:27 PM] Processed Run 103/420
[11/20/2020 03:12:27 PM] Processed Run 104/420
[11/20/2020 03:12:27 PM] Processed Run 105/420
[11/20/2020 03:12:27 PM] Processed Run 106/420
[11/20/2020 03:12:27 PM] Processed Run 107/420
[11/20/2020 03:12:28 PM] Processed Run 108/420
[11/20/2020 03:12:28 PM] Processed Run 109/420
[11/20/2020 03:12:28 PM] Processed Run 110/420
[11/20/2020 03:12:28 PM] Processed Run 111/420
[11/20/2020 03:12:28 PM] Processed Run 112/420
[11/20/2020 03:12:28 PM] Processed Run 113/420
[11/20/2020 03:12:28 PM] Processed Run 114/420
[11/20/2020 03:12:28 PM] Processed Run 115/420
[11/20/2020 03:12:28 PM] Processed Run 116/420
[11/20/2020 03:12:28 PM] Processed Run 117/420
[11/20/2020 03:12:28 PM] Processed Run 118/420
[11/20/2020 03:12:28 PM] Processed Run 119/420
[11/20/2020 03:12:28 PM] Processed Run 120/420
[11/20/2020 03:12:28 PM] Processed Run 121/420
[11/20/2020 03:12:28 PM] Processed Run 122/420
[11/20/2020 03:12:29 PM] Processed Run 123/420
[11/20/2020 03:12:29 PM] Processed Run 124/420
[11/20/2020 03:12:30 PM] Processed Run 125/420
[11/20/2020 03:12:30 PM] Processed Run 126/420
[11/20/2020 03:12:30 PM] Processed Run 127/420
[11/20/2020 03:12:30 PM] Processed Run 128/420
[11/20/2020 03:12:30 PM] Processed Run 129/420
[11/20/2020 03:12:31 PM] Processed Run 130/420
[11/20/2020 03:12:31 PM] Processed Run 131/420
[11/20/2020 03:12:31 PM] Processed Run 132/420
[11/20/2020 03:12:31 PM] Processed Run 133/420
[11/20/2020 03:12:31 PM] Processed Run 134/420
[11/20/2020 03:12:31 PM] Processed Run 135/420
[11/20/2020 03:12:31 PM] Processed Run 136/420
[11/20/2020 03:12:31 PM] Processed Run 137/420
[11/20/2020 03:12:31 PM] Processed Run 138/420
[11/20/2020 03:12:31 PM] Processed Run 139/420
[11/20/2020 03:12:31 PM] Processed Run 140/420
[11/20/2020 03:12:41 PM] Processed Run 141/420
[11/20/2020 03:12:41 PM] Processed Run 142/420
[11/20/2020 03:12:41 PM] Processed Run 143/420
[11/20/2020 03:12:41 PM] Processed Run 144/420
[11/20/2020 03:12:41 PM] Processed Run 145/420
[11/20/2020 03:12:42 PM] Processed Run 146/420
[11/20/2020 03:12:42 PM] Processed Run 147/420
[11/20/2020 03:12:42 PM] Processed Run 148/420
[11/20/2020 03:12:42 PM] Processed Run 149/420
[11/20/2020 03:12:42 PM] Processed Run 150/420
[11/20/2020 03:12:42 PM] Processed Run 151/420
[11/20/2020 03:12:42 PM] Processed Run 152/420
[11/20/2020 03:12:42 PM] Processed Run 153/420
[11/20/2020 03:12:43 PM] Processed Run 154/420
[11/20/2020 03:12:43 PM] Processed Run 155/420
[11/20/2020 03:12:44 PM] Processed Run 156/420
[11/20/2020 03:12:44 PM] Processed Run 157/420
[11/20/2020 03:12:44 PM] Processed Run 158/420
[11/20/2020 03:12:44 PM] Processed Run 159/420
[11/20/2020 03:12:44 PM] Processed Run 160/420
[11/20/2020 03:12:44 PM] Processed Run 161/420
[11/20/2020 03:12:44 PM] Processed Run 162/420
[11/20/2020 03:12:44 PM] Processed Run 163/420
[11/20/2020 03:12:44 PM] Processed Run 164/420
[11/20/2020 03:12:44 PM] Processed Run 165/420
[11/20/2020 03:12:44 PM] Processed Run 166/420
[11/20/2020 03:12:44 PM] Processed Run 167/420
[11/20/2020 03:12:44 PM] Processed Run 168/420
[11/20/2020 03:12:44 PM] Processed Run 169/420
[11/20/2020 03:12:44 PM] Processed Run 170/420
[11/20/2020 03:12:44 PM] Processed Run 171/420
[11/20/2020 03:12:45 PM] Processed Run 172/420
[11/20/2020 03:12:45 PM] Processed Run 173/420
[11/20/2020 03:12:45 PM] Processed Run 174/420
[11/20/2020 03:12:45 PM] Processed Run 175/420
[11/20/2020 03:12:45 PM] Processed Run 176/420
[11/20/2020 03:12:45 PM] Processed Run 177/420
[11/20/2020 03:12:45 PM] Processed Run 178/420
[11/20/2020 03:12:45 PM] Processed Run 179/420
[11/20/2020 03:12:45 PM] Processed Run 180/420
[11/20/2020 03:12:45 PM] Processed Run 181/420
[11/20/2020 03:12:45 PM] Processed Run 182/420
[11/20/2020 03:12:46 PM] Processed Run 183/420
[11/20/2020 03:12:46 PM] Processed Run 184/420
[11/20/2020 03:12:47 PM] Processed Run 185/420
[11/20/2020 03:12:47 PM] Processed Run 186/420
[11/20/2020 03:12:47 PM] Processed Run 187/420
[11/20/2020 03:12:47 PM] Processed Run 188/420
[11/20/2020 03:12:47 PM] Processed Run 189/420
[11/20/2020 03:12:47 PM] Processed Run 190/420
[11/20/2020 03:12:47 PM] Processed Run 191/420
[11/20/2020 03:12:48 PM] Processed Run 192/420
[11/20/2020 03:12:48 PM] Processed Run 193/420
[11/20/2020 03:12:48 PM] Processed Run 194/420
[11/20/2020 03:12:48 PM] Processed Run 195/420
[11/20/2020 03:12:48 PM] Processed Run 196/420
[11/20/2020 03:12:48 PM] Processed Run 197/420
[11/20/2020 03:12:48 PM] Processed Run 198/420
[11/20/2020 03:12:48 PM] Processed Run 199/420
[11/20/2020 03:12:48 PM] Processed Run 200/420
[11/20/2020 03:12:48 PM] Processed Run 201/420
[11/20/2020 03:12:48 PM] Processed Run 202/420
[11/20/2020 03:12:48 PM] Processed Run 203/420
[11/20/2020 03:12:48 PM] Processed Run 204/420
[11/20/2020 03:12:48 PM] Processed Run 205/420
[11/20/2020 03:12:48 PM] Processed Run 206/420
[11/20/2020 03:12:48 PM] Processed Run 207/420
[11/20/2020 03:12:48 PM] Processed Run 208/420
[11/20/2020 03:12:48 PM] Processed Run 209/420
[11/20/2020 03:12:48 PM] Processed Run 210/420
[11/20/2020 03:12:48 PM] Processed Run 211/420
[11/20/2020 03:12:48 PM] Processed Run 212/420
[11/20/2020 03:13:02 PM] Processed Run 213/420
[11/20/2020 03:13:02 PM] Processed Run 214/420
[11/20/2020 03:13:03 PM] Processed Run 215/420
[11/20/2020 03:13:03 PM] Processed Run 216/420
[11/20/2020 03:13:03 PM] Processed Run 217/420
[11/20/2020 03:13:03 PM] Processed Run 218/420
[11/20/2020 03:13:03 PM] Processed Run 219/420
[11/20/2020 03:13:03 PM] Processed Run 220/420
[11/20/2020 03:13:04 PM] Processed Run 221/420
[11/20/2020 03:13:04 PM] Processed Run 222/420
[11/20/2020 03:13:04 PM] Processed Run 223/420
[11/20/2020 03:13:04 PM] Processed Run 224/420
[11/20/2020 03:13:04 PM] Processed Run 225/420
[11/20/2020 03:13:04 PM] Processed Run 226/420
[11/20/2020 03:13:04 PM] Processed Run 227/420
[11/20/2020 03:13:04 PM] Processed Run 228/420
[11/20/2020 03:13:04 PM] Processed Run 229/420
[11/20/2020 03:13:04 PM] Processed Run 230/420
[11/20/2020 03:13:04 PM] Processed Run 231/420
[11/20/2020 03:13:04 PM] Processed Run 232/420
[11/20/2020 03:13:04 PM] Processed Run 233/420
[11/20/2020 03:13:04 PM] Processed Run 234/420
[11/20/2020 03:13:04 PM] Processed Run 235/420
[11/20/2020 03:13:04 PM] Processed Run 236/420
[11/20/2020 03:13:04 PM] Processed Run 237/420
[11/20/2020 03:13:04 PM] Processed Run 238/420
[11/20/2020 03:13:04 PM] Processed Run 239/420
[11/20/2020 03:13:04 PM] Processed Run 240/420
[11/20/2020 03:13:04 PM] Processed Run 241/420
[11/20/2020 03:13:05 PM] Processed Run 242/420
[11/20/2020 03:13:05 PM] Processed Run 243/420
[11/20/2020 03:13:06 PM] Processed Run 244/420
[11/20/2020 03:13:06 PM] Processed Run 245/420
[11/20/2020 03:13:07 PM] Processed Run 246/420
[11/20/2020 03:13:07 PM] Processed Run 247/420
[11/20/2020 03:13:07 PM] Processed Run 248/420
[11/20/2020 03:13:07 PM] Processed Run 249/420
[11/20/2020 03:13:07 PM] Processed Run 250/420
[11/20/2020 03:13:07 PM] Processed Run 251/420
[11/20/2020 03:13:07 PM] Processed Run 252/420
[11/20/2020 03:13:07 PM] Processed Run 253/420
[11/20/2020 03:13:07 PM] Processed Run 254/420
[11/20/2020 03:13:07 PM] Processed Run 255/420
[11/20/2020 03:13:07 PM] Processed Run 256/420
[11/20/2020 03:13:07 PM] Processed Run 257/420
[11/20/2020 03:13:07 PM] Processed Run 258/420
[11/20/2020 03:13:07 PM] Processed Run 259/420
[11/20/2020 03:13:07 PM] Processed Run 260/420
[11/20/2020 03:13:07 PM] Processed Run 261/420
[11/20/2020 03:13:07 PM] Processed Run 262/420
[11/20/2020 03:13:08 PM] Processed Run 263/420
[11/20/2020 03:13:08 PM] Processed Run 264/420
[11/20/2020 03:13:08 PM] Processed Run 265/420
[11/20/2020 03:13:08 PM] Processed Run 266/420
[11/20/2020 03:13:08 PM] Processed Run 267/420
[11/20/2020 03:13:08 PM] Processed Run 268/420
[11/20/2020 03:13:08 PM] Processed Run 269/420
[11/20/2020 03:13:08 PM] Processed Run 270/420
[11/20/2020 03:13:08 PM] Processed Run 271/420
[11/20/2020 03:13:08 PM] Processed Run 272/420
[11/20/2020 03:13:09 PM] Processed Run 273/420
[11/20/2020 03:13:09 PM] Processed Run 274/420
[11/20/2020 03:13:10 PM] Processed Run 275/420
[11/20/2020 03:13:26 PM] Processed Run 276/420
[11/20/2020 03:13:26 PM] Processed Run 277/420
[11/20/2020 03:13:26 PM] Processed Run 278/420
[11/20/2020 03:13:26 PM] Processed Run 279/420
[11/20/2020 03:13:26 PM] Processed Run 280/420
[11/20/2020 03:13:27 PM] Processed Run 281/420
[11/20/2020 03:13:27 PM] Processed Run 282/420
[11/20/2020 03:13:27 PM] Processed Run 283/420
[11/20/2020 03:13:27 PM] Processed Run 284/420
[11/20/2020 03:13:27 PM] Processed Run 285/420
[11/20/2020 03:13:27 PM] Processed Run 286/420
[11/20/2020 03:13:27 PM] Processed Run 287/420
[11/20/2020 03:13:27 PM] Processed Run 288/420
[11/20/2020 03:13:27 PM] Processed Run 289/420
[11/20/2020 03:13:27 PM] Processed Run 290/420
[11/20/2020 03:13:27 PM] Processed Run 291/420
[11/20/2020 03:13:27 PM] Processed Run 292/420
[11/20/2020 03:13:27 PM] Processed Run 293/420
[11/20/2020 03:13:27 PM] Processed Run 294/420
[11/20/2020 03:13:27 PM] Processed Run 295/420
[11/20/2020 03:13:27 PM] Processed Run 296/420
[11/20/2020 03:13:27 PM] Processed Run 297/420
[11/20/2020 03:13:27 PM] Processed Run 298/420
[11/20/2020 03:13:27 PM] Processed Run 299/420
[11/20/2020 03:13:27 PM] Processed Run 300/420
[11/20/2020 03:13:27 PM] Processed Run 301/420
[11/20/2020 03:13:27 PM] Processed Run 302/420
[11/20/2020 03:13:28 PM] Processed Run 303/420
[11/20/2020 03:13:28 PM] Processed Run 304/420
[11/20/2020 03:13:29 PM] Processed Run 305/420
[11/20/2020 03:13:29 PM] Processed Run 306/420
[11/20/2020 03:13:29 PM] Processed Run 307/420
[11/20/2020 03:13:30 PM] Processed Run 308/420
[11/20/2020 03:13:30 PM] Processed Run 309/420
[11/20/2020 03:13:30 PM] Processed Run 310/420
[11/20/2020 03:13:30 PM] Processed Run 311/420
[11/20/2020 03:13:30 PM] Processed Run 312/420
[11/20/2020 03:13:30 PM] Processed Run 313/420
[11/20/2020 03:13:30 PM] Processed Run 314/420
[11/20/2020 03:13:30 PM] Processed Run 315/420
[11/20/2020 03:13:30 PM] Processed Run 316/420
[11/20/2020 03:13:30 PM] Processed Run 317/420
[11/20/2020 03:13:30 PM] Processed Run 318/420
[11/20/2020 03:13:30 PM] Processed Run 319/420
[11/20/2020 03:13:30 PM] Processed Run 320/420
[11/20/2020 03:13:30 PM] Processed Run 321/420
[11/20/2020 03:13:30 PM] Processed Run 322/420
[11/20/2020 03:13:30 PM] Processed Run 323/420
[11/20/2020 03:13:30 PM] Processed Run 324/420
[11/20/2020 03:13:30 PM] Processed Run 325/420
[11/20/2020 03:13:30 PM] Processed Run 326/420
[11/20/2020 03:13:30 PM] Processed Run 327/420
[11/20/2020 03:13:30 PM] Processed Run 328/420
[11/20/2020 03:13:30 PM] Processed Run 329/420
[11/20/2020 03:13:30 PM] Processed Run 330/420
[11/20/2020 03:13:30 PM] Processed Run 331/420
[11/20/2020 03:13:31 PM] Processed Run 332/420
[11/20/2020 03:13:31 PM] Processed Run 333/420
[11/20/2020 03:13:32 PM] Processed Run 334/420
[11/20/2020 03:13:32 PM] Processed Run 335/420
[11/20/2020 03:13:33 PM] Processed Run 336/420
[11/20/2020 03:13:33 PM] Processed Run 337/420
[11/20/2020 03:13:33 PM] Processed Run 338/420
[11/20/2020 03:13:33 PM] Processed Run 339/420
[11/20/2020 03:13:33 PM] Processed Run 340/420
[11/20/2020 03:13:33 PM] Processed Run 341/420
[11/20/2020 03:13:33 PM] Processed Run 342/420
[11/20/2020 03:13:33 PM] Processed Run 343/420
[11/20/2020 03:13:33 PM] Processed Run 344/420
[11/20/2020 03:13:33 PM] Processed Run 345/420
[11/20/2020 03:13:33 PM] Processed Run 346/420
[11/20/2020 03:13:33 PM] Processed Run 347/420
[11/20/2020 03:13:33 PM] Processed Run 348/420
[11/20/2020 03:13:33 PM] Processed Run 349/420
[11/20/2020 03:13:33 PM] Processed Run 350/420
[11/20/2020 03:13:33 PM] Processed Run 351/420
[11/20/2020 03:13:33 PM] Processed Run 352/420
[11/20/2020 03:13:33 PM] Processed Run 353/420
[11/20/2020 03:13:33 PM] Processed Run 354/420
[11/20/2020 03:13:33 PM] Processed Run 355/420
[11/20/2020 03:13:33 PM] Processed Run 356/420
[11/20/2020 03:13:33 PM] Processed Run 357/420
[11/20/2020 03:13:33 PM] Processed Run 358/420
[11/20/2020 03:13:33 PM] Processed Run 359/420
[11/20/2020 03:13:33 PM] Processed Run 360/420
[11/20/2020 03:13:33 PM] Processed Run 361/420
[11/20/2020 03:13:34 PM] Processed Run 362/420
[11/20/2020 03:13:35 PM] Processed Run 363/420
[11/20/2020 03:13:35 PM] Processed Run 364/420
[11/20/2020 03:13:36 PM] Processed Run 365/420
[11/20/2020 03:13:36 PM] Processed Run 366/420
[11/20/2020 03:13:36 PM] Processed Run 367/420
[11/20/2020 03:13:36 PM] Processed Run 368/420
[11/20/2020 03:13:36 PM] Processed Run 369/420
[11/20/2020 03:13:37 PM] Processed Run 370/420
[11/20/2020 03:13:37 PM] Processed Run 371/420
[11/20/2020 03:13:37 PM] Processed Run 372/420
[11/20/2020 03:13:37 PM] Processed Run 373/420
[11/20/2020 03:13:37 PM] Processed Run 374/420
[11/20/2020 03:13:37 PM] Processed Run 375/420
[11/20/2020 03:13:37 PM] Processed Run 376/420
[11/20/2020 03:13:37 PM] Processed Run 377/420
[11/20/2020 03:13:37 PM] Processed Run 378/420
[11/20/2020 03:13:37 PM] Processed Run 379/420
[11/20/2020 03:13:37 PM] Processed Run 380/420
[11/20/2020 03:13:37 PM] Processed Run 381/420
[11/20/2020 03:13:37 PM] Processed Run 382/420
[11/20/2020 03:13:37 PM] Processed Run 383/420
[11/20/2020 03:13:37 PM] Processed Run 384/420
[11/20/2020 03:13:37 PM] Processed Run 385/420
[11/20/2020 03:13:37 PM] Processed Run 386/420
[11/20/2020 03:13:37 PM] Processed Run 387/420
[11/20/2020 03:13:37 PM] Processed Run 388/420
[11/20/2020 03:13:37 PM] Processed Run 389/420
[11/20/2020 03:13:37 PM] Processed Run 390/420
[11/20/2020 03:13:37 PM] Processed Run 391/420
[11/20/2020 03:13:56 PM] Processed Run 392/420
[11/20/2020 03:13:56 PM] Processed Run 393/420
[11/20/2020 03:13:57 PM] Processed Run 394/420
[11/20/2020 03:13:57 PM] Processed Run 395/420
[11/20/2020 03:13:58 PM] Processed Run 396/420
[11/20/2020 03:13:58 PM] Processed Run 397/420
[11/20/2020 03:13:58 PM] Processed Run 398/420
[11/20/2020 03:13:58 PM] Processed Run 399/420
[11/20/2020 03:13:58 PM] Processed Run 400/420
[11/20/2020 03:13:58 PM] Processed Run 401/420
[11/20/2020 03:13:58 PM] Processed Run 402/420
[11/20/2020 03:13:58 PM] Processed Run 403/420
[11/20/2020 03:13:58 PM] Processed Run 404/420
[11/20/2020 03:13:58 PM] Processed Run 405/420
[11/20/2020 03:13:58 PM] Processed Run 406/420
[11/20/2020 03:13:58 PM] Processed Run 407/420
[11/20/2020 03:13:58 PM] Processed Run 408/420
[11/20/2020 03:13:58 PM] Processed Run 409/420
[11/20/2020 03:13:58 PM] Processed Run 410/420
[11/20/2020 03:13:58 PM] Processed Run 411/420
[11/20/2020 03:13:58 PM] Processed Run 412/420
[11/20/2020 03:13:58 PM] Processed Run 413/420
[11/20/2020 03:13:58 PM] Processed Run 414/420
[11/20/2020 03:13:59 PM] Processed Run 415/420
[11/20/2020 03:13:59 PM] Processed Run 416/420
[11/20/2020 03:13:59 PM] Processed Run 417/420
[11/20/2020 03:13:59 PM] Processed Run 418/420
[11/20/2020 03:13:59 PM] Processed Run 419/420
[11/20/2020 03:13:59 PM] Processed Run 420/420
[11/20/2020 03:14:43 PM] Run 1/420
[11/20/2020 03:14:44 PM] Run 2/420
[11/20/2020 03:14:44 PM] Run 3/420
[11/20/2020 03:14:45 PM] Run 4/420
[11/20/2020 03:14:45 PM] Run 5/420
[11/20/2020 03:14:45 PM] Run 6/420
[11/20/2020 03:14:45 PM] Run 7/420
[11/20/2020 03:14:45 PM] Run 8/420
[11/20/2020 03:14:45 PM] Run 9/420
[11/20/2020 03:14:45 PM] Run 10/420
[11/20/2020 03:14:46 PM] Run 11/420
[11/20/2020 03:14:46 PM] Run 12/420
[11/20/2020 03:14:46 PM] Run 13/420
[11/20/2020 03:14:46 PM] Run 14/420
[11/20/2020 03:14:46 PM] Run 15/420
[11/20/2020 03:14:46 PM] Run 16/420
[11/20/2020 03:14:47 PM] Run 17/420
[11/20/2020 03:14:47 PM] Run 18/420
[11/20/2020 03:14:47 PM] Run 19/420
[11/20/2020 03:14:47 PM] Run 20/420
[11/20/2020 03:14:47 PM] Run 21/420
[11/20/2020 03:14:47 PM] Run 22/420
[11/20/2020 03:14:48 PM] Run 23/420
[11/20/2020 03:14:48 PM] Run 24/420
[11/20/2020 03:14:48 PM] Run 25/420
[11/20/2020 03:14:48 PM] Run 26/420
[11/20/2020 03:14:48 PM] Run 27/420
[11/20/2020 03:14:48 PM] Run 28/420
[11/20/2020 03:14:48 PM] Run 29/420
[11/20/2020 03:14:49 PM] Run 30/420
[11/20/2020 03:14:49 PM] Run 31/420
[11/20/2020 03:14:49 PM] Run 32/420
[11/20/2020 03:14:49 PM] Run 33/420
[11/20/2020 03:14:49 PM] Run 34/420
[11/20/2020 03:14:49 PM] Run 35/420
[11/20/2020 03:14:49 PM] Run 36/420
[11/20/2020 03:14:49 PM] Run 37/420
[11/20/2020 03:14:49 PM] Run 38/420
[11/20/2020 03:14:49 PM] Run 39/420
[11/20/2020 03:14:50 PM] Run 40/420
[11/20/2020 03:14:50 PM] Run 41/420
[11/20/2020 03:14:50 PM] Run 42/420
[11/20/2020 03:14:50 PM] Run 43/420
[11/20/2020 03:14:50 PM] Run 44/420
[11/20/2020 03:14:50 PM] Run 45/420
[11/20/2020 03:14:51 PM] Run 46/420
[11/20/2020 03:14:51 PM] Run 47/420
[11/20/2020 03:14:51 PM] Run 48/420
[11/20/2020 03:14:51 PM] Run 49/420
[11/20/2020 03:14:52 PM] Run 50/420
[11/20/2020 03:14:52 PM] Run 51/420
[11/20/2020 03:14:52 PM] Run 52/420
[11/20/2020 03:14:52 PM] Run 53/420
[11/20/2020 03:14:52 PM] Run 54/420
[11/20/2020 03:14:52 PM] Run 55/420
[11/20/2020 03:14:52 PM] Run 56/420
[11/20/2020 03:14:52 PM] Run 57/420
[11/20/2020 03:14:52 PM] Run 58/420
[11/20/2020 03:14:52 PM] Run 59/420
[11/20/2020 03:14:53 PM] Run 60/420
[11/20/2020 03:14:53 PM] Run 61/420
[11/20/2020 03:14:53 PM] Run 62/420
[11/20/2020 03:14:53 PM] Run 63/420
[11/20/2020 03:14:54 PM] Run 64/420
[11/20/2020 03:14:54 PM] Run 65/420
[11/20/2020 03:14:54 PM] Run 66/420
[11/20/2020 03:14:54 PM] Run 67/420
[11/20/2020 03:14:55 PM] Run 68/420
[11/20/2020 03:14:55 PM] Run 69/420
[11/20/2020 03:14:55 PM] Run 70/420
[11/20/2020 03:14:55 PM] Run 71/420
[11/20/2020 03:14:55 PM] Run 72/420
[11/20/2020 03:14:55 PM] Run 73/420
[11/20/2020 03:14:55 PM] Run 74/420
[11/20/2020 03:14:55 PM] Run 75/420
[11/20/2020 03:14:55 PM] Run 76/420
[11/20/2020 03:14:56 PM] Run 77/420
[11/20/2020 03:14:56 PM] Run 78/420
[11/20/2020 03:14:56 PM] Run 79/420
[11/20/2020 03:14:56 PM] Run 80/420
[11/20/2020 03:14:56 PM] Run 81/420
[11/20/2020 03:14:57 PM] Run 82/420
[11/20/2020 03:14:57 PM] Run 83/420
[11/20/2020 03:14:57 PM] Run 84/420
[11/20/2020 03:14:57 PM] Run 85/420
[11/20/2020 03:14:57 PM] Run 86/420
[11/20/2020 03:14:57 PM] Run 87/420
[11/20/2020 03:14:57 PM] Run 88/420
[11/20/2020 03:14:57 PM] Run 89/420
[11/20/2020 03:14:57 PM] Run 90/420
[11/20/2020 03:14:58 PM] Run 91/420
[11/20/2020 03:14:58 PM] Run 92/420
[11/20/2020 03:14:58 PM] Run 93/420
[11/20/2020 03:14:58 PM] Run 94/420
[11/20/2020 03:14:58 PM] Run 95/420
[11/20/2020 03:14:58 PM] Run 96/420
[11/20/2020 03:14:58 PM] Run 97/420
[11/20/2020 03:14:58 PM] Run 98/420
[11/20/2020 03:14:58 PM] Run 99/420
[11/20/2020 03:14:58 PM] Run 100/420
[11/20/2020 03:14:59 PM] Run 101/420
[11/20/2020 03:14:59 PM] Run 102/420
[11/20/2020 03:14:59 PM] Run 103/420
[11/20/2020 03:14:59 PM] Run 104/420
[11/20/2020 03:14:59 PM] Run 105/420
[11/20/2020 03:14:59 PM] Run 106/420
[11/20/2020 03:15:00 PM] Run 107/420
[11/20/2020 03:15:00 PM] Run 108/420
[11/20/2020 03:15:00 PM] Run 109/420
[11/20/2020 03:15:00 PM] Run 110/420
[11/20/2020 03:15:00 PM] Run 111/420
[11/20/2020 03:15:00 PM] Run 112/420
[11/20/2020 03:15:01 PM] Run 113/420
[11/20/2020 03:15:01 PM] Run 114/420
[11/20/2020 03:15:01 PM] Run 115/420
[11/20/2020 03:15:01 PM] Run 116/420
[11/20/2020 03:15:01 PM] Run 117/420
[11/20/2020 03:15:01 PM] Run 118/420
[11/20/2020 03:15:01 PM] Run 119/420
[11/20/2020 03:15:01 PM] Run 120/420
[11/20/2020 03:15:01 PM] Run 121/420
[11/20/2020 03:15:02 PM] Run 122/420
[11/20/2020 03:15:02 PM] Run 123/420
[11/20/2020 03:15:02 PM] Run 124/420
[11/20/2020 03:15:02 PM] Run 125/420
[11/20/2020 03:15:02 PM] Run 126/420
[11/20/2020 03:15:02 PM] Run 127/420
[11/20/2020 03:15:02 PM] Run 128/420
[11/20/2020 03:15:02 PM] Run 129/420
[11/20/2020 03:15:03 PM] Run 130/420
[11/20/2020 03:15:03 PM] Run 131/420
[11/20/2020 03:15:03 PM] Run 132/420
[11/20/2020 03:15:03 PM] Run 133/420
[11/20/2020 03:15:03 PM] Run 134/420
[11/20/2020 03:15:03 PM] Run 135/420
[11/20/2020 03:15:03 PM] Run 136/420
[11/20/2020 03:15:03 PM] Run 137/420
[11/20/2020 03:15:03 PM] Run 138/420
[11/20/2020 03:15:03 PM] Run 139/420
[11/20/2020 03:15:03 PM] Run 140/420
[11/20/2020 03:15:04 PM] Run 141/420
[11/20/2020 03:15:04 PM] Run 142/420
[11/20/2020 03:15:04 PM] Run 143/420
[11/20/2020 03:15:04 PM] Run 144/420
[11/20/2020 03:15:04 PM] Run 145/420
[11/20/2020 03:15:05 PM] Run 146/420
[11/20/2020 03:15:05 PM] Run 147/420
[11/20/2020 03:15:05 PM] Run 148/420
[11/20/2020 03:15:05 PM] Run 149/420
[11/20/2020 03:15:05 PM] Run 150/420
[11/20/2020 03:15:05 PM] Run 151/420
[11/20/2020 03:15:06 PM] Run 152/420
[11/20/2020 03:15:06 PM] Run 153/420
[11/20/2020 03:15:06 PM] Run 154/420
[11/20/2020 03:15:06 PM] Run 155/420
[11/20/2020 03:15:06 PM] Run 156/420
[11/20/2020 03:15:06 PM] Run 157/420
[11/20/2020 03:15:06 PM] Run 158/420
[11/20/2020 03:15:06 PM] Run 159/420
[11/20/2020 03:15:06 PM] Run 160/420
[11/20/2020 03:15:06 PM] Run 161/420
[11/20/2020 03:15:06 PM] Run 162/420
[11/20/2020 03:15:06 PM] Run 163/420
[11/20/2020 03:15:07 PM] Run 164/420
[11/20/2020 03:15:07 PM] Run 165/420
[11/20/2020 03:15:07 PM] Run 166/420
[11/20/2020 03:15:07 PM] Run 167/420
[11/20/2020 03:15:07 PM] Run 168/420
[11/20/2020 03:15:07 PM] Run 169/420
[11/20/2020 03:15:08 PM] Run 170/420
[11/20/2020 03:15:08 PM] Run 171/420
[11/20/2020 03:15:08 PM] Run 172/420
[11/20/2020 03:15:08 PM] Run 173/420
[11/20/2020 03:15:08 PM] Run 174/420
[11/20/2020 03:15:08 PM] Run 175/420
[11/20/2020 03:15:08 PM] Run 176/420
[11/20/2020 03:15:09 PM] Run 177/420
[11/20/2020 03:15:09 PM] Run 178/420
[11/20/2020 03:15:09 PM] Run 179/420
[11/20/2020 03:15:09 PM] Run 180/420
[11/20/2020 03:15:09 PM] Run 181/420
[11/20/2020 03:15:09 PM] Run 182/420
[11/20/2020 03:15:09 PM] Run 183/420
[11/20/2020 03:15:10 PM] Run 184/420
[11/20/2020 03:15:10 PM] Run 185/420
[11/20/2020 03:15:10 PM] Run 186/420
[11/20/2020 03:15:10 PM] Run 187/420
[11/20/2020 03:15:10 PM] Run 188/420
[11/20/2020 03:15:10 PM] Run 189/420
[11/20/2020 03:15:10 PM] Run 190/420
[11/20/2020 03:15:10 PM] Run 191/420
[11/20/2020 03:15:10 PM] Run 192/420
[11/20/2020 03:15:10 PM] Run 193/420
[11/20/2020 03:15:10 PM] Run 194/420
[11/20/2020 03:15:11 PM] Run 195/420
[11/20/2020 03:15:11 PM] Run 196/420
[11/20/2020 03:15:12 PM] Run 197/420
[11/20/2020 03:15:12 PM] Run 198/420
[11/20/2020 03:15:12 PM] Run 199/420
[11/20/2020 03:15:12 PM] Run 200/420
[11/20/2020 03:15:13 PM] Run 201/420
[11/20/2020 03:15:13 PM] Run 202/420
[11/20/2020 03:15:13 PM] Run 203/420
[11/20/2020 03:15:13 PM] Run 204/420
[11/20/2020 03:15:13 PM] Run 205/420
[11/20/2020 03:15:13 PM] Run 206/420
[11/20/2020 03:15:14 PM] Run 207/420
[11/20/2020 03:15:14 PM] Run 208/420
[11/20/2020 03:15:14 PM] Run 209/420
[11/20/2020 03:15:14 PM] Run 210/420
[11/20/2020 03:15:14 PM] Run 211/420
[11/20/2020 03:15:14 PM] Run 212/420
[11/20/2020 03:15:14 PM] Run 213/420
[11/20/2020 03:15:14 PM] Run 214/420
[11/20/2020 03:15:15 PM] Run 215/420
[11/20/2020 03:15:15 PM] Run 216/420
[11/20/2020 03:15:15 PM] Run 217/420
[11/20/2020 03:15:15 PM] Run 218/420
[11/20/2020 03:15:15 PM] Run 219/420
[11/20/2020 03:15:15 PM] Run 220/420
[11/20/2020 03:15:15 PM] Run 221/420
[11/20/2020 03:15:16 PM] Run 222/420
[11/20/2020 03:15:16 PM] Run 223/420
[11/20/2020 03:15:16 PM] Run 224/420
[11/20/2020 03:15:16 PM] Run 225/420
[11/20/2020 03:15:16 PM] Run 226/420
[11/20/2020 03:15:16 PM] Run 227/420
[11/20/2020 03:15:16 PM] Run 228/420
[11/20/2020 03:15:16 PM] Run 229/420
[11/20/2020 03:15:17 PM] Run 230/420
[11/20/2020 03:15:17 PM] Run 231/420
[11/20/2020 03:15:17 PM] Run 232/420
[11/20/2020 03:15:17 PM] Run 233/420
[11/20/2020 03:15:17 PM] Run 234/420
[11/20/2020 03:15:17 PM] Run 235/420
[11/20/2020 03:15:17 PM] Run 236/420
[11/20/2020 03:15:18 PM] Run 237/420
[11/20/2020 03:15:18 PM] Run 238/420
[11/20/2020 03:15:18 PM] Run 239/420
[11/20/2020 03:15:18 PM] Run 240/420
[11/20/2020 03:15:18 PM] Run 241/420
[11/20/2020 03:15:18 PM] Run 242/420
[11/20/2020 03:15:19 PM] Run 243/420
[11/20/2020 03:15:19 PM] Run 244/420
[11/20/2020 03:15:19 PM] Run 245/420
[11/20/2020 03:15:19 PM] Run 246/420
[11/20/2020 03:15:19 PM] Run 247/420
[11/20/2020 03:15:19 PM] Run 248/420
[11/20/2020 03:15:19 PM] Run 249/420
[11/20/2020 03:15:19 PM] Run 250/420
[11/20/2020 03:15:19 PM] Run 251/420
[11/20/2020 03:15:19 PM] Run 252/420
[11/20/2020 03:15:20 PM] Run 253/420
[11/20/2020 03:15:20 PM] Run 254/420
[11/20/2020 03:15:20 PM] Run 255/420
[11/20/2020 03:15:20 PM] Run 256/420
[11/20/2020 03:15:20 PM] Run 257/420
[11/20/2020 03:15:20 PM] Run 258/420
[11/20/2020 03:15:20 PM] Run 259/420
[11/20/2020 03:15:20 PM] Run 260/420
[11/20/2020 03:15:21 PM] Run 261/420
[11/20/2020 03:15:21 PM] Run 262/420
[11/20/2020 03:15:21 PM] Run 263/420
[11/20/2020 03:15:21 PM] Run 264/420
[11/20/2020 03:15:22 PM] Run 265/420
[11/20/2020 03:15:22 PM] Run 266/420
[11/20/2020 03:15:22 PM] Run 267/420
[11/20/2020 03:15:22 PM] Run 268/420
[11/20/2020 03:15:22 PM] Run 269/420
[11/20/2020 03:15:22 PM] Run 270/420
[11/20/2020 03:15:22 PM] Run 271/420
[11/20/2020 03:15:23 PM] Run 272/420
[11/20/2020 03:15:23 PM] Run 273/420
[11/20/2020 03:15:23 PM] Run 274/420
[11/20/2020 03:15:23 PM] Run 275/420
[11/20/2020 03:15:23 PM] Run 276/420
[11/20/2020 03:15:23 PM] Run 277/420
[11/20/2020 03:15:23 PM] Run 278/420
[11/20/2020 03:15:23 PM] Run 279/420
[11/20/2020 03:15:24 PM] Run 280/420
[11/20/2020 03:15:24 PM] Run 281/420
[11/20/2020 03:15:24 PM] Run 282/420
[11/20/2020 03:15:24 PM] Run 283/420
[11/20/2020 03:15:24 PM] Run 284/420
[11/20/2020 03:15:24 PM] Run 285/420
[11/20/2020 03:15:24 PM] Run 286/420
[11/20/2020 03:15:24 PM] Run 287/420
[11/20/2020 03:15:25 PM] Run 288/420
[11/20/2020 03:15:25 PM] Run 289/420
[11/20/2020 03:15:25 PM] Run 290/420
[11/20/2020 03:15:25 PM] Run 291/420
[11/20/2020 03:15:25 PM] Run 292/420
[11/20/2020 03:15:26 PM] Run 293/420
[11/20/2020 03:15:26 PM] Run 294/420
[11/20/2020 03:15:26 PM] Run 295/420
[11/20/2020 03:15:26 PM] Run 296/420
[11/20/2020 03:15:26 PM] Run 297/420
[11/20/2020 03:15:26 PM] Run 298/420
[11/20/2020 03:15:27 PM] Run 299/420
[11/20/2020 03:15:27 PM] Run 300/420
[11/20/2020 03:15:27 PM] Run 301/420
[11/20/2020 03:15:27 PM] Run 302/420
[11/20/2020 03:15:27 PM] Run 303/420
[11/20/2020 03:15:27 PM] Run 304/420
[11/20/2020 03:15:27 PM] Run 305/420
[11/20/2020 03:15:27 PM] Run 306/420
[11/20/2020 03:15:27 PM] Run 307/420
[11/20/2020 03:15:27 PM] Run 308/420
[11/20/2020 03:15:28 PM] Run 309/420
[11/20/2020 03:15:28 PM] Run 310/420
[11/20/2020 03:15:28 PM] Run 311/420
[11/20/2020 03:15:28 PM] Run 312/420
[11/20/2020 03:15:28 PM] Run 313/420
[11/20/2020 03:15:28 PM] Run 314/420
[11/20/2020 03:15:28 PM] Run 315/420
[11/20/2020 03:15:29 PM] Run 316/420
[11/20/2020 03:15:29 PM] Run 317/420
[11/20/2020 03:15:29 PM] Run 318/420
[11/20/2020 03:15:29 PM] Run 319/420
[11/20/2020 03:15:29 PM] Run 320/420
[11/20/2020 03:15:29 PM] Run 321/420
[11/20/2020 03:15:29 PM] Run 322/420
[11/20/2020 03:15:29 PM] Run 323/420
[11/20/2020 03:15:29 PM] Run 324/420
[11/20/2020 03:15:30 PM] Run 325/420
[11/20/2020 03:15:30 PM] Run 326/420
[11/20/2020 03:15:30 PM] Run 327/420
[11/20/2020 03:15:30 PM] Run 328/420
[11/20/2020 03:15:30 PM] Run 329/420
[11/20/2020 03:15:31 PM] Run 330/420
[11/20/2020 03:15:31 PM] Run 331/420
[11/20/2020 03:15:31 PM] Run 332/420
[11/20/2020 03:15:31 PM] Run 333/420
[11/20/2020 03:15:31 PM] Run 334/420
[11/20/2020 03:15:31 PM] Run 335/420
[11/20/2020 03:15:32 PM] Run 336/420
[11/20/2020 03:15:32 PM] Run 337/420
[11/20/2020 03:15:32 PM] Run 338/420
[11/20/2020 03:15:32 PM] Run 339/420
[11/20/2020 03:15:32 PM] Run 340/420
[11/20/2020 03:15:32 PM] Run 341/420
[11/20/2020 03:15:32 PM] Run 342/420
[11/20/2020 03:15:33 PM] Run 343/420
[11/20/2020 03:15:33 PM] Run 344/420
[11/20/2020 03:15:33 PM] Run 345/420
[11/20/2020 03:15:34 PM] Run 346/420
[11/20/2020 03:15:34 PM] Run 347/420
[11/20/2020 03:15:34 PM] Run 348/420
[11/20/2020 03:15:34 PM] Run 349/420
[11/20/2020 03:15:34 PM] Run 350/420
[11/20/2020 03:15:34 PM] Run 351/420
[11/20/2020 03:15:34 PM] Run 352/420
[11/20/2020 03:15:34 PM] Run 353/420
[11/20/2020 03:15:34 PM] Run 354/420
[11/20/2020 03:15:35 PM] Run 355/420
[11/20/2020 03:15:35 PM] Run 356/420
[11/20/2020 03:15:35 PM] Run 357/420
[11/20/2020 03:15:35 PM] Run 358/420
[11/20/2020 03:15:35 PM] Run 359/420
[11/20/2020 03:15:35 PM] Run 360/420
[11/20/2020 03:15:36 PM] Run 361/420
[11/20/2020 03:15:36 PM] Run 362/420
[11/20/2020 03:15:36 PM] Run 363/420
[11/20/2020 03:15:36 PM] Run 364/420
[11/20/2020 03:15:36 PM] Run 365/420
[11/20/2020 03:15:36 PM] Run 366/420
[11/20/2020 03:15:36 PM] Run 367/420
[11/20/2020 03:15:37 PM] Run 368/420
[11/20/2020 03:15:37 PM] Run 369/420
[11/20/2020 03:15:37 PM] Run 370/420
[11/20/2020 03:15:37 PM] Run 371/420
[11/20/2020 03:15:37 PM] Run 372/420
[11/20/2020 03:15:37 PM] Run 373/420
[11/20/2020 03:15:38 PM] Run 374/420
[11/20/2020 03:15:38 PM] Run 375/420
[11/20/2020 03:15:38 PM] Run 376/420
[11/20/2020 03:15:38 PM] Run 377/420
[11/20/2020 03:15:38 PM] Run 378/420
[11/20/2020 03:15:38 PM] Run 379/420
[11/20/2020 03:15:38 PM] Run 380/420
[11/20/2020 03:15:38 PM] Run 381/420
[11/20/2020 03:15:39 PM] Run 382/420
[11/20/2020 03:15:39 PM] Run 383/420
[11/20/2020 03:15:39 PM] Run 384/420
[11/20/2020 03:15:39 PM] Run 385/420
[11/20/2020 03:15:39 PM] Run 386/420
[11/20/2020 03:15:39 PM] Run 387/420
[11/20/2020 03:15:39 PM] Run 388/420
[11/20/2020 03:15:40 PM] Run 389/420
[11/20/2020 03:15:40 PM] Run 390/420
[11/20/2020 03:15:40 PM] Run 391/420
[11/20/2020 03:15:40 PM] Run 392/420
[11/20/2020 03:15:40 PM] Run 393/420
[11/20/2020 03:15:40 PM] Run 394/420
[11/20/2020 03:15:40 PM] Run 395/420
[11/20/2020 03:15:40 PM] Run 396/420
[11/20/2020 03:15:41 PM] Run 397/420
[11/20/2020 03:15:41 PM] Run 398/420
[11/20/2020 03:15:41 PM] Run 399/420
[11/20/2020 03:15:41 PM] Run 400/420
[11/20/2020 03:15:41 PM] Run 401/420
[11/20/2020 03:15:41 PM] Run 402/420
[11/20/2020 03:15:41 PM] Run 403/420
[11/20/2020 03:15:41 PM] Run 404/420
[11/20/2020 03:15:42 PM] Run 405/420
[11/20/2020 03:15:43 PM] Run 406/420
[11/20/2020 03:15:43 PM] Run 407/420
[11/20/2020 03:15:43 PM] Run 408/420
[11/20/2020 03:15:44 PM] Run 409/420
[11/20/2020 03:15:45 PM] Run 410/420
[11/20/2020 03:15:46 PM] Run 411/420
[11/20/2020 03:15:47 PM] Run 412/420
[11/20/2020 03:15:49 PM] Run 413/420
[11/20/2020 03:15:50 PM] Run 414/420
[11/20/2020 03:15:50 PM] Run 415/420
[11/20/2020 03:19:28 PM] Run 416/420
[11/20/2020 03:19:28 PM] Run 417/420
[11/20/2020 03:19:30 PM] Run 418/420
[11/20/2020 03:19:38 PM] Run 419/420
[11/20/2020 03:19:41 PM] Run 420/420
[11/20/2020 03:19:41 PM] Processed Run 1/420
[11/20/2020 03:19:43 PM] Processed Run 2/420
[11/20/2020 03:19:43 PM] Processed Run 3/420
[11/20/2020 03:19:44 PM] Processed Run 4/420
[11/20/2020 03:19:45 PM] Processed Run 5/420
[11/20/2020 03:19:46 PM] Processed Run 6/420
[11/20/2020 03:19:46 PM] Processed Run 7/420
[11/20/2020 03:19:46 PM] Processed Run 8/420
[11/20/2020 03:19:46 PM] Processed Run 9/420
[11/20/2020 03:19:46 PM] Processed Run 10/420
[11/20/2020 03:19:46 PM] Processed Run 11/420
[11/20/2020 03:19:46 PM] Processed Run 12/420
[11/20/2020 03:19:46 PM] Processed Run 13/420
[11/20/2020 03:19:46 PM] Processed Run 14/420
[11/20/2020 03:19:46 PM] Processed Run 15/420
[11/20/2020 03:19:46 PM] Processed Run 16/420
[11/20/2020 03:19:46 PM] Processed Run 17/420
[11/20/2020 03:19:47 PM] Processed Run 18/420
[11/20/2020 03:19:47 PM] Processed Run 19/420
[11/20/2020 03:19:47 PM] Processed Run 20/420
[11/20/2020 03:19:47 PM] Processed Run 21/420
[11/20/2020 03:19:47 PM] Processed Run 22/420
[11/20/2020 03:19:47 PM] Processed Run 23/420
[11/20/2020 03:19:47 PM] Processed Run 24/420
[11/20/2020 03:19:47 PM] Processed Run 25/420
[11/20/2020 03:19:47 PM] Processed Run 26/420
[11/20/2020 03:19:47 PM] Processed Run 27/420
[11/20/2020 03:19:47 PM] Processed Run 28/420
[11/20/2020 03:19:47 PM] Processed Run 29/420
[11/20/2020 03:19:47 PM] Processed Run 30/420
[11/20/2020 03:19:47 PM] Processed Run 31/420
[11/20/2020 03:19:53 PM] Processed Run 32/420
distributed.core - INFO - Event loop was unresponsive in Worker for 3.05s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
[11/20/2020 03:20:04 PM] Processed Run 33/420
distributed.core - INFO - Event loop was unresponsive in Worker for 3.01s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
[11/20/2020 03:20:20 PM] Processed Run 34/420
distributed.core - INFO - Event loop was unresponsive in Worker for 3.15s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
[11/20/2020 03:20:32 PM] Processed Run 35/420
distributed.core - INFO - Event loop was unresponsive in Worker for 3.05s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
[11/20/2020 03:20:46 PM] Processed Run 36/420
[11/20/2020 03:20:46 PM] Processed Run 37/420
[11/20/2020 03:20:47 PM] Processed Run 38/420
[11/20/2020 03:20:47 PM] Processed Run 39/420
[11/20/2020 03:20:48 PM] Processed Run 40/420
[11/20/2020 03:20:48 PM] Processed Run 41/420
[11/20/2020 03:20:48 PM] Processed Run 42/420
[11/20/2020 03:20:48 PM] Processed Run 43/420
[11/20/2020 03:20:48 PM] Processed Run 44/420
[11/20/2020 03:20:48 PM] Processed Run 45/420
[11/20/2020 03:20:48 PM] Processed Run 46/420
[11/20/2020 03:20:48 PM] Processed Run 47/420
[11/20/2020 03:20:48 PM] Processed Run 48/420
[11/20/2020 03:20:48 PM] Processed Run 49/420
[11/20/2020 03:20:48 PM] Processed Run 50/420
[11/20/2020 03:20:48 PM] Processed Run 51/420
[11/20/2020 03:20:49 PM] Processed Run 52/420
[11/20/2020 03:20:49 PM] Processed Run 53/420
[11/20/2020 03:20:49 PM] Processed Run 54/420
[11/20/2020 03:20:49 PM] Processed Run 55/420
[11/20/2020 03:20:49 PM] Processed Run 56/420
[11/20/2020 03:20:49 PM] Processed Run 57/420
[11/20/2020 03:20:49 PM] Processed Run 58/420
[11/20/2020 03:20:49 PM] Processed Run 59/420
[11/20/2020 03:20:49 PM] Processed Run 60/420
[11/20/2020 03:20:49 PM] Processed Run 61/420
[11/20/2020 03:20:49 PM] Processed Run 62/420
[11/20/2020 03:20:50 PM] Processed Run 63/420
[11/20/2020 03:20:51 PM] Processed Run 64/420
[11/20/2020 03:20:52 PM] Processed Run 65/420
[11/20/2020 03:21:02 PM] Processed Run 66/420
[11/20/2020 03:21:02 PM] Processed Run 67/420
[11/20/2020 03:21:03 PM] Processed Run 68/420
[11/20/2020 03:21:03 PM] Processed Run 69/420
[11/20/2020 03:21:03 PM] Processed Run 70/420
[11/20/2020 03:21:03 PM] Processed Run 71/420
[11/20/2020 03:21:03 PM] Processed Run 72/420
[11/20/2020 03:21:03 PM] Processed Run 73/420
[11/20/2020 03:21:03 PM] Processed Run 74/420
[11/20/2020 03:21:03 PM] Processed Run 75/420
[11/20/2020 03:21:03 PM] Processed Run 76/420
[11/20/2020 03:21:03 PM] Processed Run 77/420
[11/20/2020 03:21:03 PM] Processed Run 78/420
[11/20/2020 03:21:03 PM] Processed Run 79/420
[11/20/2020 03:21:03 PM] Processed Run 80/420
[11/20/2020 03:21:03 PM] Processed Run 81/420
[11/20/2020 03:21:03 PM] Processed Run 82/420
[11/20/2020 03:21:03 PM] Processed Run 83/420
[11/20/2020 03:21:03 PM] Processed Run 84/420
[11/20/2020 03:21:03 PM] Processed Run 85/420
[11/20/2020 03:21:03 PM] Processed Run 86/420
[11/20/2020 03:21:03 PM] Processed Run 87/420
[11/20/2020 03:21:03 PM] Processed Run 88/420
[11/20/2020 03:21:03 PM] Processed Run 89/420
[11/20/2020 03:21:03 PM] Processed Run 90/420
[11/20/2020 03:21:03 PM] Processed Run 91/420
[11/20/2020 03:21:05 PM] Processed Run 92/420
[11/20/2020 03:21:06 PM] Processed Run 93/420
[11/20/2020 03:21:07 PM] Processed Run 94/420
[11/20/2020 03:21:07 PM] Processed Run 95/420
[11/20/2020 03:21:08 PM] Processed Run 96/420
[11/20/2020 03:21:08 PM] Processed Run 97/420
[11/20/2020 03:21:08 PM] Processed Run 98/420
[11/20/2020 03:21:09 PM] Processed Run 99/420
[11/20/2020 03:21:09 PM] Processed Run 100/420
[11/20/2020 03:21:09 PM] Processed Run 101/420
[11/20/2020 03:21:09 PM] Processed Run 102/420
[11/20/2020 03:21:09 PM] Processed Run 103/420
[11/20/2020 03:21:09 PM] Processed Run 104/420
[11/20/2020 03:21:09 PM] Processed Run 105/420
[11/20/2020 03:21:09 PM] Processed Run 106/420
[11/20/2020 03:21:09 PM] Processed Run 107/420
[11/20/2020 03:21:09 PM] Processed Run 108/420
[11/20/2020 03:21:09 PM] Processed Run 109/420
[11/20/2020 03:21:09 PM] Processed Run 110/420
[11/20/2020 03:21:09 PM] Processed Run 111/420
[11/20/2020 03:21:09 PM] Processed Run 112/420
[11/20/2020 03:21:09 PM] Processed Run 113/420
[11/20/2020 03:21:09 PM] Processed Run 114/420
[11/20/2020 03:21:09 PM] Processed Run 115/420
[11/20/2020 03:21:09 PM] Processed Run 116/420
[11/20/2020 03:21:09 PM] Processed Run 117/420
[11/20/2020 03:21:09 PM] Processed Run 118/420
[11/20/2020 03:21:09 PM] Processed Run 119/420
[11/20/2020 03:21:09 PM] Processed Run 120/420
[11/20/2020 03:21:09 PM] Processed Run 121/420
[11/20/2020 03:21:10 PM] Processed Run 122/420
[11/20/2020 03:21:22 PM] Processed Run 123/420
[11/20/2020 03:21:24 PM] Processed Run 124/420
[11/20/2020 03:21:25 PM] Processed Run 125/420
[11/20/2020 03:21:26 PM] Processed Run 126/420
[11/20/2020 03:21:26 PM] Processed Run 127/420
[11/20/2020 03:21:26 PM] Processed Run 128/420
[11/20/2020 03:21:26 PM] Processed Run 129/420
[11/20/2020 03:21:26 PM] Processed Run 130/420
[11/20/2020 03:21:26 PM] Processed Run 131/420
[11/20/2020 03:21:26 PM] Processed Run 132/420
[11/20/2020 03:21:26 PM] Processed Run 133/420
[11/20/2020 03:21:26 PM] Processed Run 134/420
[11/20/2020 03:21:26 PM] Processed Run 135/420
[11/20/2020 03:21:26 PM] Processed Run 136/420
[11/20/2020 03:21:26 PM] Processed Run 137/420
[11/20/2020 03:21:26 PM] Processed Run 138/420
[11/20/2020 03:21:27 PM] Processed Run 139/420
[11/20/2020 03:21:27 PM] Processed Run 140/420
[11/20/2020 03:21:27 PM] Processed Run 141/420
[11/20/2020 03:21:27 PM] Processed Run 142/420
[11/20/2020 03:21:27 PM] Processed Run 143/420
[11/20/2020 03:21:27 PM] Processed Run 144/420
[11/20/2020 03:21:27 PM] Processed Run 145/420
[11/20/2020 03:21:27 PM] Processed Run 146/420
[11/20/2020 03:21:27 PM] Processed Run 147/420
[11/20/2020 03:21:27 PM] Processed Run 148/420
[11/20/2020 03:21:27 PM] Processed Run 149/420
[11/20/2020 03:21:27 PM] Processed Run 150/420
[11/20/2020 03:21:27 PM] Processed Run 151/420
[11/20/2020 03:21:27 PM] Processed Run 152/420
[11/20/2020 03:21:28 PM] Processed Run 153/420
[11/20/2020 03:21:29 PM] Processed Run 154/420
[11/20/2020 03:21:30 PM] Processed Run 155/420
[11/20/2020 03:21:30 PM] Processed Run 156/420
[11/20/2020 03:21:31 PM] Processed Run 157/420
[11/20/2020 03:21:31 PM] Processed Run 158/420
[11/20/2020 03:21:31 PM] Processed Run 159/420
[11/20/2020 03:21:31 PM] Processed Run 160/420
[11/20/2020 03:21:31 PM] Processed Run 161/420
[11/20/2020 03:21:31 PM] Processed Run 162/420
[11/20/2020 03:21:31 PM] Processed Run 163/420
[11/20/2020 03:21:31 PM] Processed Run 164/420
[11/20/2020 03:21:31 PM] Processed Run 165/420
[11/20/2020 03:21:31 PM] Processed Run 166/420
[11/20/2020 03:21:31 PM] Processed Run 167/420
[11/20/2020 03:21:31 PM] Processed Run 168/420
[11/20/2020 03:21:31 PM] Processed Run 169/420
[11/20/2020 03:21:31 PM] Processed Run 170/420
[11/20/2020 03:21:31 PM] Processed Run 171/420
[11/20/2020 03:21:31 PM] Processed Run 172/420
[11/20/2020 03:21:31 PM] Processed Run 173/420
[11/20/2020 03:21:32 PM] Processed Run 174/420
[11/20/2020 03:21:32 PM] Processed Run 175/420
[11/20/2020 03:21:32 PM] Processed Run 176/420
[11/20/2020 03:21:32 PM] Processed Run 177/420
[11/20/2020 03:21:32 PM] Processed Run 178/420
[11/20/2020 03:21:32 PM] Processed Run 179/420
[11/20/2020 03:21:32 PM] Processed Run 180/420
[11/20/2020 03:21:32 PM] Processed Run 181/420
[11/20/2020 03:21:32 PM] Processed Run 182/420
[11/20/2020 03:21:46 PM] Processed Run 183/420
[11/20/2020 03:21:47 PM] Processed Run 184/420
[11/20/2020 03:21:48 PM] Processed Run 185/420
[11/20/2020 03:21:48 PM] Processed Run 186/420
[11/20/2020 03:21:48 PM] Processed Run 187/420
[11/20/2020 03:21:49 PM] Processed Run 188/420
[11/20/2020 03:21:49 PM] Processed Run 189/420
[11/20/2020 03:21:49 PM] Processed Run 190/420
[11/20/2020 03:21:49 PM] Processed Run 191/420
[11/20/2020 03:21:49 PM] Processed Run 192/420
[11/20/2020 03:21:49 PM] Processed Run 193/420
[11/20/2020 03:21:49 PM] Processed Run 194/420
[11/20/2020 03:21:49 PM] Processed Run 195/420
[11/20/2020 03:21:49 PM] Processed Run 196/420
[11/20/2020 03:21:49 PM] Processed Run 197/420
[11/20/2020 03:21:49 PM] Processed Run 198/420
[11/20/2020 03:21:49 PM] Processed Run 199/420
[11/20/2020 03:21:49 PM] Processed Run 200/420
[11/20/2020 03:21:49 PM] Processed Run 201/420
[11/20/2020 03:21:49 PM] Processed Run 202/420
[11/20/2020 03:21:49 PM] Processed Run 203/420
[11/20/2020 03:21:49 PM] Processed Run 204/420
[11/20/2020 03:21:49 PM] Processed Run 205/420
[11/20/2020 03:21:49 PM] Processed Run 206/420
[11/20/2020 03:21:49 PM] Processed Run 207/420
[11/20/2020 03:21:49 PM] Processed Run 208/420
[11/20/2020 03:21:49 PM] Processed Run 209/420
[11/20/2020 03:21:50 PM] Processed Run 210/420
[11/20/2020 03:21:50 PM] Processed Run 211/420
[11/20/2020 03:21:50 PM] Processed Run 212/420
[11/20/2020 03:21:51 PM] Processed Run 213/420
[11/20/2020 03:21:51 PM] Processed Run 214/420
[11/20/2020 03:21:52 PM] Processed Run 215/420
[11/20/2020 03:21:53 PM] Processed Run 216/420
[11/20/2020 03:21:53 PM] Processed Run 217/420
[11/20/2020 03:21:53 PM] Processed Run 218/420
[11/20/2020 03:21:53 PM] Processed Run 219/420
[11/20/2020 03:21:53 PM] Processed Run 220/420
[11/20/2020 03:21:54 PM] Processed Run 221/420
[11/20/2020 03:21:54 PM] Processed Run 222/420
[11/20/2020 03:21:54 PM] Processed Run 223/420
[11/20/2020 03:21:54 PM] Processed Run 224/420
[11/20/2020 03:21:54 PM] Processed Run 225/420
[11/20/2020 03:21:54 PM] Processed Run 226/420
[11/20/2020 03:21:54 PM] Processed Run 227/420
[11/20/2020 03:21:54 PM] Processed Run 228/420
[11/20/2020 03:21:54 PM] Processed Run 229/420
[11/20/2020 03:21:54 PM] Processed Run 230/420
[11/20/2020 03:21:54 PM] Processed Run 231/420
[11/20/2020 03:21:54 PM] Processed Run 232/420
[11/20/2020 03:21:54 PM] Processed Run 233/420
[11/20/2020 03:21:54 PM] Processed Run 234/420
[11/20/2020 03:21:54 PM] Processed Run 235/420
[11/20/2020 03:21:54 PM] Processed Run 236/420
[11/20/2020 03:21:54 PM] Processed Run 237/420
[11/20/2020 03:21:54 PM] Processed Run 238/420
[11/20/2020 03:21:54 PM] Processed Run 239/420
[11/20/2020 03:21:54 PM] Processed Run 240/420
[11/20/2020 03:21:54 PM] Processed Run 241/420
[11/20/2020 03:21:55 PM] Processed Run 242/420
[11/20/2020 03:21:56 PM] Processed Run 243/420
[11/20/2020 03:21:56 PM] Processed Run 244/420
[11/20/2020 03:21:57 PM] Processed Run 245/420
[11/20/2020 03:21:58 PM] Processed Run 246/420
[11/20/2020 03:22:15 PM] Processed Run 247/420
[11/20/2020 03:22:15 PM] Processed Run 248/420
[11/20/2020 03:22:15 PM] Processed Run 249/420
[11/20/2020 03:22:15 PM] Processed Run 250/420
[11/20/2020 03:22:15 PM] Processed Run 251/420
[11/20/2020 03:22:15 PM] Processed Run 252/420
[11/20/2020 03:22:15 PM] Processed Run 253/420
[11/20/2020 03:22:15 PM] Processed Run 254/420
[11/20/2020 03:22:15 PM] Processed Run 255/420
[11/20/2020 03:22:15 PM] Processed Run 256/420
[11/20/2020 03:22:15 PM] Processed Run 257/420
[11/20/2020 03:22:15 PM] Processed Run 258/420
[11/20/2020 03:22:15 PM] Processed Run 259/420
[11/20/2020 03:22:15 PM] Processed Run 260/420
[11/20/2020 03:22:15 PM] Processed Run 261/420
[11/20/2020 03:22:16 PM] Processed Run 262/420
[11/20/2020 03:22:16 PM] Processed Run 263/420
[11/20/2020 03:22:16 PM] Processed Run 264/420
[11/20/2020 03:22:16 PM] Processed Run 265/420
[11/20/2020 03:22:16 PM] Processed Run 266/420
[11/20/2020 03:22:16 PM] Processed Run 267/420
[11/20/2020 03:22:16 PM] Processed Run 268/420
[11/20/2020 03:22:16 PM] Processed Run 269/420
[11/20/2020 03:22:16 PM] Processed Run 270/420
[11/20/2020 03:22:16 PM] Processed Run 271/420
[11/20/2020 03:22:16 PM] Processed Run 272/420
[11/20/2020 03:22:17 PM] Processed Run 273/420
[11/20/2020 03:22:18 PM] Processed Run 274/420
[11/20/2020 03:22:19 PM] Processed Run 275/420
[11/20/2020 03:22:19 PM] Processed Run 276/420
[11/20/2020 03:22:19 PM] Processed Run 277/420
[11/20/2020 03:22:20 PM] Processed Run 278/420
[11/20/2020 03:22:20 PM] Processed Run 279/420
[11/20/2020 03:22:20 PM] Processed Run 280/420
[11/20/2020 03:22:20 PM] Processed Run 281/420
[11/20/2020 03:22:20 PM] Processed Run 282/420
[11/20/2020 03:22:20 PM] Processed Run 283/420
[11/20/2020 03:22:20 PM] Processed Run 284/420
[11/20/2020 03:22:20 PM] Processed Run 285/420
[11/20/2020 03:22:20 PM] Processed Run 286/420
[11/20/2020 03:22:20 PM] Processed Run 287/420
[11/20/2020 03:22:20 PM] Processed Run 288/420
[11/20/2020 03:22:20 PM] Processed Run 289/420
[11/20/2020 03:22:20 PM] Processed Run 290/420
[11/20/2020 03:22:20 PM] Processed Run 291/420
[11/20/2020 03:22:20 PM] Processed Run 292/420
[11/20/2020 03:22:20 PM] Processed Run 293/420
[11/20/2020 03:22:20 PM] Processed Run 294/420
[11/20/2020 03:22:20 PM] Processed Run 295/420
[11/20/2020 03:22:20 PM] Processed Run 296/420
[11/20/2020 03:22:20 PM] Processed Run 297/420
[11/20/2020 03:22:20 PM] Processed Run 298/420
[11/20/2020 03:22:20 PM] Processed Run 299/420
[11/20/2020 03:22:20 PM] Processed Run 300/420
[11/20/2020 03:22:20 PM] Processed Run 301/420
[11/20/2020 03:22:21 PM] Processed Run 302/420
[11/20/2020 03:22:22 PM] Processed Run 303/420
[11/20/2020 03:22:23 PM] Processed Run 304/420
[11/20/2020 03:22:24 PM] Processed Run 305/420
[11/20/2020 03:22:24 PM] Processed Run 306/420
[11/20/2020 03:22:24 PM] Processed Run 307/420
[11/20/2020 03:22:25 PM] Processed Run 308/420
[11/20/2020 03:22:25 PM] Processed Run 309/420
[11/20/2020 03:22:25 PM] Processed Run 310/420
[11/20/2020 03:22:25 PM] Processed Run 311/420
[11/20/2020 03:22:25 PM] Processed Run 312/420
[11/20/2020 03:22:25 PM] Processed Run 313/420
[11/20/2020 03:22:25 PM] Processed Run 314/420
[11/20/2020 03:22:25 PM] Processed Run 315/420
[11/20/2020 03:22:25 PM] Processed Run 316/420
[11/20/2020 03:22:25 PM] Processed Run 317/420
[11/20/2020 03:22:25 PM] Processed Run 318/420
[11/20/2020 03:22:25 PM] Processed Run 319/420
[11/20/2020 03:22:25 PM] Processed Run 320/420
[11/20/2020 03:22:25 PM] Processed Run 321/420
[11/20/2020 03:22:25 PM] Processed Run 322/420
[11/20/2020 03:22:25 PM] Processed Run 323/420
[11/20/2020 03:22:25 PM] Processed Run 324/420
[11/20/2020 03:22:25 PM] Processed Run 325/420
[11/20/2020 03:22:26 PM] Processed Run 326/420
[11/20/2020 03:22:26 PM] Processed Run 327/420
[11/20/2020 03:22:26 PM] Processed Run 328/420
[11/20/2020 03:22:26 PM] Processed Run 329/420
[11/20/2020 03:22:26 PM] Processed Run 330/420
[11/20/2020 03:22:26 PM] Processed Run 331/420
[11/20/2020 03:22:26 PM] Processed Run 332/420
[11/20/2020 03:22:27 PM] Processed Run 333/420
[11/20/2020 03:22:28 PM] Processed Run 334/420
[11/20/2020 03:22:48 PM] Processed Run 335/420
[11/20/2020 03:22:49 PM] Processed Run 336/420
[11/20/2020 03:22:49 PM] Processed Run 337/420
[11/20/2020 03:22:49 PM] Processed Run 338/420
[11/20/2020 03:22:50 PM] Processed Run 339/420
[11/20/2020 03:22:50 PM] Processed Run 340/420
[11/20/2020 03:22:50 PM] Processed Run 341/420
[11/20/2020 03:22:50 PM] Processed Run 342/420
[11/20/2020 03:22:50 PM] Processed Run 343/420
[11/20/2020 03:22:50 PM] Processed Run 344/420
[11/20/2020 03:22:50 PM] Processed Run 345/420
[11/20/2020 03:22:50 PM] Processed Run 346/420
[11/20/2020 03:22:50 PM] Processed Run 347/420
[11/20/2020 03:22:50 PM] Processed Run 348/420
[11/20/2020 03:22:50 PM] Processed Run 349/420
[11/20/2020 03:22:50 PM] Processed Run 350/420
[11/20/2020 03:22:50 PM] Processed Run 351/420
[11/20/2020 03:22:50 PM] Processed Run 352/420
[11/20/2020 03:22:50 PM] Processed Run 353/420
[11/20/2020 03:22:50 PM] Processed Run 354/420
[11/20/2020 03:22:50 PM] Processed Run 355/420
[11/20/2020 03:22:50 PM] Processed Run 356/420
[11/20/2020 03:22:50 PM] Processed Run 357/420
[11/20/2020 03:22:50 PM] Processed Run 358/420
[11/20/2020 03:22:50 PM] Processed Run 359/420
[11/20/2020 03:22:50 PM] Processed Run 360/420
[11/20/2020 03:22:50 PM] Processed Run 361/420
[11/20/2020 03:22:51 PM] Processed Run 362/420
[11/20/2020 03:22:52 PM] Processed Run 363/420
[11/20/2020 03:22:52 PM] Processed Run 364/420
[11/20/2020 03:22:53 PM] Processed Run 365/420
[11/20/2020 03:22:54 PM] Processed Run 366/420
[11/20/2020 03:22:54 PM] Processed Run 367/420
[11/20/2020 03:22:54 PM] Processed Run 368/420
[11/20/2020 03:22:54 PM] Processed Run 369/420
[11/20/2020 03:22:54 PM] Processed Run 370/420
[11/20/2020 03:22:54 PM] Processed Run 371/420
[11/20/2020 03:22:54 PM] Processed Run 372/420
[11/20/2020 03:22:54 PM] Processed Run 373/420
[11/20/2020 03:22:54 PM] Processed Run 374/420
[11/20/2020 03:22:54 PM] Processed Run 375/420
[11/20/2020 03:22:54 PM] Processed Run 376/420
[11/20/2020 03:22:54 PM] Processed Run 377/420
[11/20/2020 03:22:54 PM] Processed Run 378/420
[11/20/2020 03:22:54 PM] Processed Run 379/420
[11/20/2020 03:22:55 PM] Processed Run 380/420
[11/20/2020 03:22:55 PM] Processed Run 381/420
[11/20/2020 03:22:55 PM] Processed Run 382/420
[11/20/2020 03:22:55 PM] Processed Run 383/420
[11/20/2020 03:22:55 PM] Processed Run 384/420
[11/20/2020 03:22:55 PM] Processed Run 385/420
[11/20/2020 03:22:55 PM] Processed Run 386/420
[11/20/2020 03:22:55 PM] Processed Run 387/420
[11/20/2020 03:22:55 PM] Processed Run 388/420
[11/20/2020 03:22:55 PM] Processed Run 389/420
[11/20/2020 03:22:55 PM] Processed Run 390/420
[11/20/2020 03:22:55 PM] Processed Run 391/420
[11/20/2020 03:22:55 PM] Processed Run 392/420
[11/20/2020 03:22:56 PM] Processed Run 393/420
[11/20/2020 03:22:57 PM] Processed Run 394/420
[11/20/2020 03:22:57 PM] Processed Run 395/420
[11/20/2020 03:22:58 PM] Processed Run 396/420
[11/20/2020 03:22:58 PM] Processed Run 397/420
[11/20/2020 03:22:58 PM] Processed Run 398/420
[11/20/2020 03:22:59 PM] Processed Run 399/420
[11/20/2020 03:22:59 PM] Processed Run 400/420
[11/20/2020 03:22:59 PM] Processed Run 401/420
[11/20/2020 03:22:59 PM] Processed Run 402/420
[11/20/2020 03:22:59 PM] Processed Run 403/420
[11/20/2020 03:22:59 PM] Processed Run 404/420
[11/20/2020 03:22:59 PM] Processed Run 405/420
[11/20/2020 03:22:59 PM] Processed Run 406/420
[11/20/2020 03:22:59 PM] Processed Run 407/420
[11/20/2020 03:22:59 PM] Processed Run 408/420
[11/20/2020 03:22:59 PM] Processed Run 409/420
[11/20/2020 03:22:59 PM] Processed Run 410/420
[11/20/2020 03:22:59 PM] Processed Run 411/420
[11/20/2020 03:22:59 PM] Processed Run 412/420
[11/20/2020 03:22:59 PM] Processed Run 413/420
[11/20/2020 03:22:59 PM] Processed Run 414/420
[11/20/2020 03:22:59 PM] Processed Run 415/420
[11/20/2020 03:22:59 PM] Processed Run 416/420
[11/20/2020 03:22:59 PM] Processed Run 417/420
[11/20/2020 03:22:59 PM] Processed Run 418/420
[11/20/2020 03:22:59 PM] Processed Run 419/420
[11/20/2020 03:22:59 PM] Processed Run 420/420
distributed.scheduler - INFO - Receive client connection: Client-14d1f8cd-2b7f-11eb-97eb-7cd30ab15d36
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Run out-of-band function 'stop'
distributed.scheduler - INFO - Scheduler closing...
distributed.scheduler - INFO - Scheduler closing all comms
distributed.worker - INFO - Stopping worker at tcp://10.225.6.104:42101
distributed.worker - INFO - Stopping worker at tcp://10.225.6.159:37798
distributed.worker - INFO - Stopping worker at tcp://10.225.6.159:46204
distributed.worker - INFO - Stopping worker at tcp://10.225.6.104:40996
distributed.worker - INFO - Stopping worker at tcp://10.225.6.159:44953
distributed.worker - INFO - Stopping worker at tcp://10.225.6.104:40832
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.104:43113', name: 10, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.104:43113
distributed.batched - INFO - Batched Comm Closed:
distributed.worker - INFO - Connection to scheduler broken. Reconnecting...
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.104:38768', name: 8, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.104:38768
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.104:39284', name: 11, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.104:39284
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.104:34559', name: 9, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.104:34559
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.104:45376', name: 6, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.104:45376
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.104:44944', name: 16, memory: 0, processing: 0>
distributed.worker - INFO - Stopping worker at tcp://10.225.6.159:45994
distributed.core - INFO - Removing comms to tcp://10.225.6.104:44944
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.104:40013', name: 17, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.104:40013
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.104:33488', name: 21, memory: 0, processing: 0>
distributed.worker - INFO - Stopping worker at tcp://10.225.6.159:45938
distributed.core - INFO - Removing comms to tcp://10.225.6.104:33488
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.104:43447', name: 5, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.104:43447
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.104:35031', name: 7, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.104:35031
distributed.worker - INFO - Stopping worker at tcp://10.225.6.104:43447
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.104:43432', name: 20, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.104:43432
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.104:42101', name: 22, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.104:42101
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.104:33415', name: 13, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.104:33415
distributed.worker - INFO - Stopping worker at tcp://10.225.6.104:39284
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.104:33746', name: 3, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.104:33746
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.104:38000', name: 19, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.104:38000
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.104:42900', name: 23, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.104:42900
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.104:36236', name: 4, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.104:36236
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.104:36233', name: 2, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.104:36233
distributed.batched - INFO - Batched Comm Closed:
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.104:44390', name: 15, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.104:44390
distributed.worker - INFO - Stopping worker at tcp://10.225.6.159:35201
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.104:34029', name: 14, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.104:34029
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.104:40996', name: 18, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.104:40996
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.104:40832', name: 12, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.104:40832
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.159:40590', name: 34, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.159:40590
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.159:43228', name: 33, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.159:43228
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.159:44103', name: 26, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.159:44103
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.159:44927', name: 32, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.159:44927
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.159:44333', name: 39, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.159:44333
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.159:44440', name: 27, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.159:44440
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.159:44953', name: 38, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.159:44953
distributed.batched - INFO - Batched Comm Closed:
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.159:37663', name: 29, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.159:37663
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.159:41592', name: 28, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.159:41592
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.159:36325', name: 30, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.159:36325
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.159:34208', name: 44, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.159:34208
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.159:42154', name: 42, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.159:42154
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.159:34466', name: 45, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.159:34466
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.159:34808', name: 36, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.159:34808
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.159:45994', name: 46, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.159:45994
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.159:42255', name: 40, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.159:42255
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.159:37798', name: 35, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.159:37798
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.159:46204', name: 43, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.159:46204
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.159:35120', name: 37, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.159:35120
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.159:42744', name: 25, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.159:42744
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.159:33401', name: 47, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.159:33401
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.159:35201', name: 41, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.159:35201
distributed.worker - INFO - Stopping worker at tcp://10.225.6.159:34808
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.159:40599', name: 31, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.159:40599
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.225.6.159:45938', name: 24, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://10.225.6.159:45938
distributed.scheduler - INFO - Lost all workers
distributed.worker - INFO - Stopping worker at tcp://10.225.6.104:34029
distributed.batched - INFO - Batched Comm Closed:
distributed.worker - INFO - Stopping worker at tcp://10.225.6.104:33746
distributed.worker - INFO - Stopping worker at tcp://10.225.6.159:40599
distributed.batched - INFO - Batched Comm Closed:
distributed.worker - INFO - Stopping worker at tcp://10.225.6.159:35120
distributed.worker - INFO - Stopping worker at tcp://10.225.6.104:44390
distributed.worker - INFO - Stopping worker at tcp://10.225.6.159:41592
distributed.worker - INFO - Stopping worker at tcp://10.225.6.104:43432
distributed.worker - INFO - Stopping worker at tcp://10.225.6.104:33488
distributed.worker - INFO - Stopping worker at tcp://10.225.6.104:38768
distributed.worker - INFO - Stopping worker at tcp://10.225.6.104:36236
distributed.worker - INFO - Stopping worker at tcp://10.225.6.104:33415
distributed.worker - INFO - Stopping worker at tcp://10.225.6.159:40590
distributed.worker - INFO - Stopping worker at tcp://10.225.6.159:34466
distributed.worker - INFO - Stopping worker at tcp://10.225.6.159:42154
distributed.worker - INFO - Stopping worker at tcp://10.225.6.159:44333
distributed.worker - INFO - Stopping worker at tcp://10.225.6.159:37663
distributed.worker - INFO - Stopping worker at tcp://10.225.6.159:36325
distributed.worker - INFO - Stopping worker at tcp://10.225.6.159:44440
distributed.worker - INFO - Stopping worker at tcp://10.225.6.159:44927
distributed.worker - INFO - Stopping worker at tcp://10.225.6.159:34208
distributed.worker - INFO - Stopping worker at tcp://10.225.6.104:35031
distributed.worker - INFO - Stopping worker at tcp://10.225.6.104:34559
distributed.worker - INFO - Stopping worker at tcp://10.225.6.104:44944
distributed.worker - INFO - Stopping worker at tcp://10.225.6.159:42744
distributed.worker - INFO - Stopping worker at tcp://10.225.6.104:36233
distributed.worker - INFO - Stopping worker at tcp://10.225.6.159:42255
distributed.worker - INFO - Stopping worker at tcp://10.225.6.159:44103
distributed.worker - INFO - Stopping worker at tcp://10.225.6.159:43228
distributed.worker - INFO - Stopping worker at tcp://10.225.6.104:45376
distributed.worker - INFO - Stopping worker at tcp://10.225.6.104:43113
distributed.worker - INFO - Stopping worker at tcp://10.225.6.104:38000
distributed.worker - INFO - Stopping worker at tcp://10.225.6.104:42900
distributed.worker - INFO - Stopping worker at tcp://10.225.6.159:33401
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
distributed.worker - INFO - Waiting to connect to: tcp://10.225.6.104:44912
slurmstepd: error: *** STEP 6215493.0 ON shas0352 CANCELLED AT 2020-11-20T15:37:38 DUE TO TIME LIMIT ***
slurmstepd: error: *** JOB 6215493 ON shas0352 CANCELLED AT 2020-11-20T15:37:38 DUE TO TIME LIMIT ***
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
If I hit the end of a job allocation, or I OOM my cluster, or the cluster is shut down abruptly, an old scheduler.json file can be found by the workers on my next try. I could always be sure to delete this file before I launch dask-mpi but why not just pass the info over the wire?
Sound handy? We'd probably get PR that in before I figured out my other MPI cluster question...
I believe that in some simple cases, we don't need to have a rank dedicated to the Scheduler, and a rank dedicated to the Client. We should provide a way to start both within the same MPI rank in https://github.com/dask/dask-mpi/blob/master/dask_mpi/core.py#L62-L71.
Thoughts?
Describe the issue:
I'm trying out a simple hello-world style dask-mpi example, and the computation returns the right result, but I'm getting exceptions when the client finishes. I'm running the below script under Slurm as srun -n3 python repro.py, and the error is:
2022-10-05 14:05:41,510 - distributed.worker - INFO - Connection to scheduler broken. Closing without reporting. ID: Worker-a2bfbff7-5dcc-49da-868a-08c403ba78f9 Address tcp://10.250.145.105:38331 Status: Status.closing
2022-10-05 14:05:41,510 - distributed.batched - INFO - Batched Comm Closed <TCP (closed) Worker->Scheduler local=tcp://10.250.145.105:42110 remote=tcp://10.250.145.105:45101>
Traceback (most recent call last):
File "/mnt/home/lgarrison/scc/daskdistrib/venv8/lib/python3.8/site-packages/distributed/batched.py", line 115, in _background_send
nbytes = yield coro
File "/mnt/sw/nix/store/db63z7j5w4n84c625pv5b57m699bnbws-python-3.8.12-view/lib/python3.8/site-packages/tornado/gen.py", line 762, in run
value = future.result()
File "/mnt/home/lgarrison/scc/daskdistrib/venv8/lib/python3.8/site-packages/distributed/comm/tcp.py", line 269, in write
raise CommClosedError()
distributed.comm.core.CommClosedError
I thought this might be related to #87, but I'm running on Python 3.8 and there's just an exception, no hang.
Am I doing something wrong? It looks to me like the worker is complaining because the scheduler shuts down before the worker does. Is this expected? If I manually force the workers to shut down before the client and scheduler do, with:
def closeall(dask_scheduler):
for w in dask_scheduler.workers:
dask_scheduler.close_worker(w)
[...]
client.run_on_scheduler(closeall)then everything exits with no exceptions. But this feels like a hack... am I missing something?
Minimal Complete Verifiable Example:
import dask_mpi
from distributed import Client
def f(a):
return a + 1
def main():
dask_mpi.initialize()
with Client() as client:
future = client.submit(f, 1)
res = future.result()
print(f'future returned {res}')
if __name__ == '__main__':
main()Full log:
(venv8) lgarrison@scclin021:~/scc/daskdistrib$ srun -n3 python ./repro_commclosed.py
2022-10-05 14:05:40,460 - distributed.http.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy
2022-10-05 14:05:40,527 - distributed.scheduler - INFO - State start
2022-10-05 14:05:40,533 - distributed.scheduler - INFO - Scheduler at: tcp://10.250.145.105:45101
2022-10-05 14:05:40,533 - distributed.scheduler - INFO - dashboard at: :8787
2022-10-05 14:05:40,566 - distributed.worker - INFO - Start worker at: tcp://10.250.145.105:38331
2022-10-05 14:05:40,566 - distributed.worker - INFO - Listening to: tcp://10.250.145.105:38331
2022-10-05 14:05:40,566 - distributed.worker - INFO - Worker name: 2
2022-10-05 14:05:40,566 - distributed.worker - INFO - dashboard at: 10.250.145.105:34979
2022-10-05 14:05:40,566 - distributed.worker - INFO - Waiting to connect to: tcp://10.250.145.105:45101
2022-10-05 14:05:40,566 - distributed.worker - INFO - -------------------------------------------------
2022-10-05 14:05:40,566 - distributed.worker - INFO - Threads: 1
2022-10-05 14:05:40,566 - distributed.worker - INFO - Memory: 7.81 GiB
2022-10-05 14:05:40,566 - distributed.worker - INFO - Local Directory: /tmp/dask-worker-space/worker-az8e_3tm
2022-10-05 14:05:40,566 - distributed.worker - INFO - -------------------------------------------------
2022-10-05 14:05:41,354 - distributed.scheduler - INFO - Receive client connection: Client-5274d531-44d8-11ed-94ba-4cd98f221a38
2022-10-05 14:05:41,356 - distributed.core - INFO - Starting established connection
2022-10-05 14:05:41,385 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://10.250.145.105:38331', name: 2, status: init, memory: 0, processing: 0>
2022-10-05 14:05:41,386 - distributed.scheduler - INFO - Starting worker compute stream, tcp://10.250.145.105:38331
2022-10-05 14:05:41,386 - distributed.core - INFO - Starting established connection
2022-10-05 14:05:41,386 - distributed.worker - INFO - Registered to: tcp://10.250.145.105:45101
2022-10-05 14:05:41,386 - distributed.worker - INFO - -------------------------------------------------
2022-10-05 14:05:41,387 - distributed.core - INFO - Starting established connection
2022-10-05 14:05:41,500 - distributed.scheduler - INFO - Remove client Client-5274d531-44d8-11ed-94ba-4cd98f221a38
2022-10-05 14:05:41,500 - distributed.scheduler - INFO - Remove client Client-5274d531-44d8-11ed-94ba-4cd98f221a38
2022-10-05 14:05:41,500 - distributed.scheduler - INFO - Close client connection: Client-5274d531-44d8-11ed-94ba-4cd98f221a38
future returned 2
2022-10-05 14:05:41,506 - distributed.scheduler - INFO - Receive client connection: Client-53080c1f-44d8-11ed-94ba-4cd98f221a38
2022-10-05 14:05:41,507 - distributed.core - INFO - Starting established connection
2022-10-05 14:05:41,508 - distributed.worker - INFO - Run out-of-band function 'stop'
2022-10-05 14:05:41,509 - distributed.scheduler - INFO - Scheduler closing...
2022-10-05 14:05:41,509 - distributed.scheduler - INFO - Scheduler closing all comms
2022-10-05 14:05:41,509 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://10.250.145.105:38331', name: 2, status: running, memory: 0, processing: 0>
2022-10-05 14:05:41,509 - distributed.worker - INFO - Stopping worker at tcp://10.250.145.105:38331
2022-10-05 14:05:41,510 - distributed.core - INFO - Removing comms to tcp://10.250.145.105:38331
2022-10-05 14:05:41,510 - distributed.scheduler - INFO - Lost all workers
2022-10-05 14:05:41,510 - distributed.worker - INFO - Connection to scheduler broken. Closing without reporting. ID: Worker-a2bfbff7-5dcc-49da-868a-08c403ba78f9 Address tcp://10.250.145.105:38331 Status: Status.closing
2022-10-05 14:05:41,510 - distributed.batched - INFO - Batched Comm Closed <TCP (closed) Worker->Scheduler local=tcp://10.250.145.105:42110 remote=tcp://10.250.145.105:45101>
Traceback (most recent call last):
File "/mnt/home/lgarrison/scc/daskdistrib/venv8/lib/python3.8/site-packages/distributed/batched.py", line 115, in _background_send
nbytes = yield coro
File "/mnt/sw/nix/store/db63z7j5w4n84c625pv5b57m699bnbws-python-3.8.12-view/lib/python3.8/site-packages/tornado/gen.py", line 762, in run
value = future.result()
File "/mnt/home/lgarrison/scc/daskdistrib/venv8/lib/python3.8/site-packages/distributed/comm/tcp.py", line 269, in write
raise CommClosedError()
distributed.comm.core.CommClosedError
Environment:
I am trying to access the bokeh dashboard in a HPC environment managed by a Slurm scheduler.
I installed dask_mpi using conda. I ran the following in the Slurm submission script:
mpirun -np $SLURM_NTASKS dask-mpi --scheduler-file scheduler.jsonThe scheduler starts correctly, and I can also connect with a Client.
INFO: localdir at /scratch/974298.acornacchia
INFO: your job will run on local system.
2024-06-27 00:23:00,531 - distributed.scheduler - INFO - State start
2024-06-27 00:23:00,585 - distributed.scheduler - INFO - Scheduler at: tcp://192.168.7.50:8786
2024-06-27 00:23:00,585 - distributed.scheduler - INFO - dashboard at: http://192.168.7.50:8787/status
2024-06-27 00:23:00,641 - distributed.scheduler - INFO - Registering Worker plugin shuffle
2024-06-27 00:23:00,927 - distributed.nanny - INFO - Start Nanny at: 'tcp://192.168.7.69:46705'
2024-06-27 00:23:00,928 - distributed.nanny - INFO - Start Nanny at: 'tcp://192.168.7.73:41865'
2024-06-27 00:23:00,944 - distributed.nanny - INFO - Start Nanny at: 'tcp://192.168.7.70:44559'
2024-06-27 00:23:45,142 - distributed.scheduler - INFO - Receive client connection: Client-bfa9d376-340a-11ef-944a-e4434b640dd8
2024-06-27 00:24:36,898 - distributed.core - INFO - Starting established connection to tcp://192.168.7.254:49934
2024-06-27 00:24:36,898 - distributed.core - INFO - Connection to tcp://192.168.7.254:49934 has been closed.
2024-06-27 00:24:36,898 - distributed.scheduler - INFO - Remove client Client-bfa9d376-340a-11ef-944a-e4434b640dd8
However, the dashboard returns 404 HTTP error when I try to access its url
wget http://192.168.7.50:8787/status
--2024-06-27 00:26:31-- http://192.168.7.50:8787/status
Connecting to 192.168.7.50:8787... connected.
HTTP request sent, awaiting response... 404 Not Found
2024-06-27 00:26:31 ERROR 404: Not Found.
Environment:
conda list -n try-dask | grep dask
# packages in environment at /home/acornacchia/miniconda3/envs/try-dask:
dask 2024.5.0 pypi_0 pypi
dask-core 2024.5.0 py310h06a4308_0
dask-expr 1.1.0 pypi_0 pypi
dask-jobqueue 0.8.5 pyhd8ed1ab_0 conda-forge
dask-labextension 7.0.0 pypi_0 pypi
dask-mpi 2022.4.0 pypi_0 pypi
3.4.13.10.14CentOS Linux 7 (Core)Additional notes:
This does not happen using dask-jobqueue, the dashboard runs correctly.
What happened:
This may be an issue in astropy, so my apologies if this is in the wrong location. Although, this appears to only happen using dask-mpi.
I'm using dask-mpi to distribute a task with using a astropy.coordinates.Angle object. When I try to convert to the astropy.units.hourangle format, I get the following error:
File "/group/askap/athomson/miniconda3/envs/spice/lib/python3.8/site-packages/astropy/coordinates/angles.py", line 316, in to_string
values = self.hour
File "/group/askap/athomson/miniconda3/envs/spice/lib/python3.8/site-packages/astropy/units/quantity.py", line 861, in __getattr__
raise AttributeError(
AttributeError: Angle instance has no attribute 'hour'
What you expected to happen:
Using a LocalCluster and a SLURMcluster I do not get this error using otherwise identical code. Further, astropy.coordinate.Angle explicitly has an hour property (L161):
@property
def hour(self):
"""
The angle's value in hours (read-only property).
"""
return self.hourangleFurther, if I do (see MCVE):
print('hour' in dir(coord))
I get True! Something strange seems to be happening when I try to access the propety itself. I'll note something similar happened with coordinates.SkyCoord and it's hms property.
Minimal Complete Verifiable Example:
This is as close to a minimal setup as my working script. Very frustratingly, the MCVE does not produce the same error. Hair pulling abounds.
from distributed import Client
from dask_mpi import initialize
from dask import delayed
from astropy.coordinates import Angle
import astropy.units as u
@delayed
def worker(i):
coord = Angle(i*u.deg)
print('coord.to_string()',coord.to_string())
print('coord.to_string(u.hourangle)',coord.to_string(u.hourangle, sep=':', precision=3))
return
def main():
initialize(interface='ipogif0')
client = Client()
results = []
for i in range(90):
results.append(
worker(i)
)
futures = client.persist(results)
outputs = [f.compute() for f in futures]
if __name__ == "__main__":
main()Anything else we need to know?:
Environment:
dask_mpi/common.py contains the line:
from distributed.cli.utils import uri_from_host_port
However, it appears that the uri_from_host_port function has moved to distributed/comm/addressing.py.
To resolve the compile error, I think the above import statement needs to be changed to:
from distributed.comm.addressing import uri_from_host_port
What happened:
In Python 3.8 tests, the dask_mpi/tests/test_core.py and dask_mpi/tests/test_no_exit.py tests execute processes that raise exceptions at shutdown and return 0 exit codes.
What you expected to happen:
I would expect any test that raises an exception to return a non-0 exit code.
Minimal Complete Verifiable Example:
With Python 3.8 on MacOS or Linux, run the command that is executed in the dask_mpi/tests/test_core.py test:
mpirun -l -np 4 python dask_mpi/tests/test_core.pyNote the raised CommClosedError and that the exit code is 0:
echo $?[0] 2022-04-20 11:17:09,840 - distributed.scheduler - INFO - State start
[0] 2022-04-20 11:17:09,844 - distributed.scheduler - INFO - Clear task state
[0] 2022-04-20 11:17:09,844 - distributed.scheduler - INFO - Scheduler at: tcp://192.168.1.27:63569
[0] 2022-04-20 11:17:09,845 - distributed.scheduler - INFO - dashboard at: :63568
[2] 2022-04-20 11:17:09,868 - distributed.worker - INFO - Start worker at: tcp://192.168.1.27:63572
[2] 2022-04-20 11:17:09,868 - distributed.worker - INFO - Listening to: tcp://192.168.1.27:63572
[2] 2022-04-20 11:17:09,868 - distributed.worker - INFO - dashboard at: 192.168.1.27:63573
[3] 2022-04-20 11:17:09,868 - distributed.worker - INFO - Start worker at: tcp://192.168.1.27:63570
[3] 2022-04-20 11:17:09,868 - distributed.worker - INFO - Listening to: tcp://192.168.1.27:63570
[2] 2022-04-20 11:17:09,868 - distributed.worker - INFO - Waiting to connect to: tcp://192.168.1.27:63569
[3] 2022-04-20 11:17:09,868 - distributed.worker - INFO - dashboard at: 192.168.1.27:63574
[2] 2022-04-20 11:17:09,868 - distributed.worker - INFO - -------------------------------------------------
[3] 2022-04-20 11:17:09,868 - distributed.worker - INFO - Waiting to connect to: tcp://192.168.1.27:63569
[2] 2022-04-20 11:17:09,868 - distributed.worker - INFO - Threads: 1
[2] 2022-04-20 11:17:09,868 - distributed.worker - INFO - Memory: 2.00 GiB
[2] 2022-04-20 11:17:09,868 - distributed.worker - INFO - Local Directory: /Users/kpaul/Software/Development/dask-mpi/dask_mpi/tests/dask-worker-space/worker-iqjsppe2
[3] 2022-04-20 11:17:09,868 - distributed.worker - INFO - -------------------------------------------------
[3] 2022-04-20 11:17:09,868 - distributed.worker - INFO - Threads: 1
[3] 2022-04-20 11:17:09,868 - distributed.worker - INFO - Memory: 2.00 GiB
[3] 2022-04-20 11:17:09,868 - distributed.worker - INFO - Local Directory: /Users/kpaul/Software/Development/dask-mpi/dask_mpi/tests/dask-worker-space/worker-lb3qdkh6
[2] 2022-04-20 11:17:09,869 - distributed.worker - INFO - -------------------------------------------------
[3] 2022-04-20 11:17:09,869 - distributed.worker - INFO - -------------------------------------------------
[0] 2022-04-20 11:17:10,267 - distributed.scheduler - INFO - Receive client connection: Client-b626d33c-c0cd-11ec-b1b6-acde48001122
[0] 2022-04-20 11:17:10,270 - distributed.core - INFO - Starting established connection
[0] 2022-04-20 11:17:10,338 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://192.168.1.27:63570', name: 3, status: undefined, memory: 0, processing: 0>
[0] 2022-04-20 11:17:10,339 - distributed.scheduler - INFO - Starting worker compute stream, tcp://192.168.1.27:63570
[0] 2022-04-20 11:17:10,339 - distributed.core - INFO - Starting established connection
[3] 2022-04-20 11:17:10,339 - distributed.worker - INFO - Registered to: tcp://192.168.1.27:63569
[3] 2022-04-20 11:17:10,339 - distributed.worker - INFO - -------------------------------------------------
[3] 2022-04-20 11:17:10,340 - distributed.core - INFO - Starting established connection
[0] 2022-04-20 11:17:10,344 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://192.168.1.27:63572', name: 2, status: undefined, memory: 0, processing: 0>
[0] 2022-04-20 11:17:10,344 - distributed.scheduler - INFO - Starting worker compute stream, tcp://192.168.1.27:63572
[0] 2022-04-20 11:17:10,344 - distributed.core - INFO - Starting established connection
[2] 2022-04-20 11:17:10,345 - distributed.worker - INFO - Registered to: tcp://192.168.1.27:63569
[2] 2022-04-20 11:17:10,345 - distributed.worker - INFO - -------------------------------------------------
[2] 2022-04-20 11:17:10,346 - distributed.core - INFO - Starting established connection
[0] 2022-04-20 11:17:10,627 - distributed.scheduler - INFO - Remove client Client-b626d33c-c0cd-11ec-b1b6-acde48001122
[0] 2022-04-20 11:17:10,627 - distributed.scheduler - INFO - Remove client Client-b626d33c-c0cd-11ec-b1b6-acde48001122
[0] 2022-04-20 11:17:10,628 - distributed.scheduler - INFO - Close client connection: Client-b626d33c-c0cd-11ec-b1b6-acde48001122
[0] 2022-04-20 11:17:10,639 - distributed.scheduler - INFO - Receive client connection: Client-b69e5de4-c0cd-11ec-b1b6-acde48001122
[0] 2022-04-20 11:17:10,639 - distributed.core - INFO - Starting established connection
[0] 2022-04-20 11:17:10,642 - distributed.worker - INFO - Run out-of-band function 'stop'
[0] 2022-04-20 11:17:10,643 - distributed.scheduler - INFO - Scheduler closing...
[0] 2022-04-20 11:17:10,643 - distributed.scheduler - INFO - Scheduler closing all comms
[2] 2022-04-20 11:17:10,645 - distributed.worker - INFO - Stopping worker at tcp://192.168.1.27:63572
[3] 2022-04-20 11:17:10,644 - distributed.worker - INFO - Stopping worker at tcp://192.168.1.27:63570
[0] 2022-04-20 11:17:10,645 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://192.168.1.27:63570', name: 3, status: running, memory: 0, processing: 0>
[0] 2022-04-20 11:17:10,645 - distributed.core - INFO - Removing comms to tcp://192.168.1.27:63570
[0] 2022-04-20 11:17:10,645 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://192.168.1.27:63572', name: 2, status: running, memory: 0, processing: 0>
[0] 2022-04-20 11:17:10,645 - distributed.core - INFO - Removing comms to tcp://192.168.1.27:63572
[0] 2022-04-20 11:17:10,645 - distributed.scheduler - INFO - Lost all workers
[2] 2022-04-20 11:17:10,649 - distributed.batched - INFO - Batched Comm Closed <TCP (closed) Worker->Scheduler local=tcp://192.168.1.27:63576 remote=tcp://192.168.1.27:63569>
[2] Traceback (most recent call last):
[2] File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-38/lib/python3.8/site-packages/distributed/batched.py", line 94, in _background_send
[2] nbytes = yield self.comm.write(
[2] File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-38/lib/python3.8/site-packages/tornado/gen.py", line 762, in run
[2] value = future.result()
[2] File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-38/lib/python3.8/site-packages/distributed/comm/tcp.py", line 269, in write
[2] raise CommClosedError()
[2] distributed.comm.core.CommClosedError
[3] 2022-04-20 11:17:10,648 - distributed.batched - INFO - Batched Comm Closed <TCP (closed) Worker->Scheduler local=tcp://192.168.1.27:63577 remote=tcp://192.168.1.27:63569>
[3] Traceback (most recent call last):
[3] File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-38/lib/python3.8/site-packages/distributed/batched.py", line 94, in _background_send
[3] nbytes = yield self.comm.write(
[3] File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-38/lib/python3.8/site-packages/tornado/gen.py", line 762, in run
[3] value = future.result()
[3] File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-38/lib/python3.8/site-packages/distributed/comm/tcp.py", line 269, in write
[3] raise CommClosedError()
[3] distributed.comm.core.CommClosedErrorAnything else we need to know?:
After some checking, I believe that this comes from an exception being raised during the registered atexit function that dask_mpi.initialiize() uses to shutdown the scheduler and workers from the client process. One can test this easily with the following script:
import atexit
def exit_error():
raise RuntimeError('ERROR')
atexit.register(exit_error)Assuming this script is called exiterror.py, run and test and check the exit code:
> python exiterror.py
Exception ignored in atexit callback: <function exit_error at 0x10f827e20>
Traceback (most recent call last):
File ".../exiterror.py", line 5, in exit_error
raise RuntimeError('ERROR')
RuntimeError: ERROR
> echo $?
0This is a well-discussed "feature" of CPython, which you can reference in python/cpython#71222 and references therein.
So, assuming that a non-zero return code cannot be given if an error occurs in an atexit handler, then I think the best we can do is try to fix the errors that are actually occurring. With Python 3.8, this can be done by using the Client.shutdown() method in the atexit handler. That means changing:
Lines 153 to 160 in bb5e4a8
to
with Client() as c:
c.shutdown()With this change, the test above throws no exceptions at exit.
Environment:
dask-mpi looks great!
I tried to install and use it simply but things did not work as expected.
For a minimum example I prepared a binder ready repo where installed dask-mpi and I created 2 notebooks:
In a second branch, called conda I tried the same but this time using conda instead of pip.
The tests are failing using pip
FileNotFoundError: [Errno 2] No such file or directory: '/srv/conda/envs/notebook/bin/mpirun'
However mpirun is available under /usr/bin/
The tests look ok using conda, but I am not able to have a running cluster.
I would like to understand where I can improve.
Thanks!
I am interested in using dask-mpi on our cluster. Whenever I launch Dask I seem to only be getting multiple schedulers and no workers. My understanding was that rank 0 should be the scheduler and the others should be workers.
dask-mpi
from dask_mpi import initialize
from dask.distributed import Client, wait
initialize()
client = Client()sbatch:
#!/bin/bash
#SBATCH --nodes=2
#SBATCH --time=01:00:00
#SBATCH --partition=debug
module purge
ml intel-mpi/2018.0.3
mpirun -np 4 dask-mpi --scheduler-file scheduler.json --interface ib0
slurm output:
distributed.scheduler - INFO - Clear task state
distributed.scheduler - INFO - Clear task state
distributed.scheduler - INFO - Clear task state
distributed.scheduler - INFO - Clear task state
distributed.scheduler - INFO - Scheduler at: tcp://10.148.2.31:39065
distributed.scheduler - INFO - Scheduler at: tcp://10.148.2.31:45246
distributed.scheduler - INFO - Scheduler at: tcp://10.148.2.31:43614
distributed.scheduler - INFO - Scheduler at: tcp://10.148.2.31:43968
In the scheduler.json I see one of these listed as the scheduler, and no entries in the workers field. Am I missing something necessary in the setup or if this may be a site specific issue?
mpirun -np 4 --map-by node dask-mpi --no-nanny --nthreads=1 --memory-limit=0 --scheduler-file=scheduler_file.json --interface=ib0 --scheduler-port=8786 &
What happened:
I am a newbie to dask and I found dask-mpi very portable approach to setup dask cluster on multiple platforms. I have noticed that there is a CLI option to specify the scheduler port number using --scheduler-port but when I use it (example usage is given above) it does not use this port and always generate random port numbers, which is the default option.
What you expected to happen:
I suppose that my scheduler should be at the port 8786 in the above example. It would be nice to have this functionality for us.
Anything else we need to know?:
There was this PR to fix this issue, but it was closed without merging into the master.
Environment:
After the first implementation of the initialize function, making it possible to launch a Dask cluster from within the client script by using mpi4py, it seems like the natural next step is to implement an MPICluster object so that the canonical client script operational model of:
cluster = MPICluster(...args...)
client = Client(cluster)will work the same way that calling initialize works.
I'm not sure what is going on, but the latest versions of dask (1.1.1) and distributed (1.25.3) appear to break dask-mpi.
@aterrel would it be possible to have mpi.dask.org route to dask-mpi.readthedocs.io ?
I have this script dask_mpi_test.py:
from dask_mpi import initialize
initialize()
from distributed import Client
import dask
client = Client()
df = dask.datasets.timeseries()
print(df.groupby(['time', 'name']).mean().compute())
print(client)When I try to run this script with:
mpirun -np 4 python dask_mpi_test.py
~/workdir $ mpirun -np 4 python dask_mpi_test.py
distributed.scheduler - INFO - Clear task state
distributed.scheduler - INFO - Scheduler at: tcp://xxxxxx:8786
distributed.scheduler - INFO - bokeh at: :8787
distributed.worker - INFO - Start worker at: tcp://xxxxx:44712
/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/distributed/bokeh/core.py:57: UserWarning:
Port 8789 is already in use.
Perhaps you already have a cluster running?
Hosting the diagnostics dashboard on a random port instead.
warnings.warn('\n' + msg)
distributed.worker - INFO - Start worker at: tcp://xxxxxx:36782
distributed.worker - INFO - Listening to: tcp://:44712
distributed.worker - INFO - bokeh at: :8789
distributed.worker - INFO - Listening to: tcp://:36782
distributed.worker - INFO - Waiting to connect to: tcp://xxxxxx:8786
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - bokeh at: :43876
distributed.worker - INFO - Waiting to connect to: tcp://xxxxx:8786
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 3.76 GB
distributed.worker - INFO - Memory: 3.76 GB
distributed.worker - INFO - Local Directory: /gpfs/fs1/scratch/abanihi/worker-uoz0vtci
distributed.worker - INFO - Local Directory: /gpfs/fs1/scratch/abanihi/worker-bb0u_737
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - -------------------------------------------------
Traceback (most recent call last):
File "dask_mpi_test.py", line 6, in <module>
client = Client()
File "/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/distributed/client.py", line 640, in __init__
self.start(timeout=timeout)
File "/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/distributed/client.py", line 763, in start
sync(self.loop, self._start, **kwargs)
File "/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/distributed/utils.py", line 321, in sync
six.reraise(*error[0])
File "/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/six.py", line 693, in reraise
raise value
File "/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/distributed/utils.py", line 306, in f
result[0] = yield future
File "/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/tornado/gen.py", line 1133, in run
value = future.result()
File "/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/tornado/gen.py", line 1141, in run
yielded = self.gen.throw(*exc_info)
File "/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/distributed/client.py", line 851, in _start
yield self._ensure_connected(timeout=timeout)
File "/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/tornado/gen.py", line 1133, in run
value = future.result()
File "/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/tornado/gen.py", line 1141, in run
yielded = self.gen.throw(*exc_info)
File "/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/distributed/client.py", line 892, in _ensure_connected
self._update_scheduler_info())
File "/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/tornado/gen.py", line 1133, in run
value = future.result()
tornado.util.Timeout$ conda list dask
# packages in environment at /glade/work/abanihi/softwares/miniconda3/envs/analysis:
#
# Name Version Build Channel
dask 1.2.0 py_0 conda-forge
dask-core 1.2.0 py_0 conda-forge
dask-jobqueue 0.4.1+28.g5826abe pypi_0 pypi
dask-labextension 0.3.3 pypi_0 pypi
dask-mpi 1.0.2 py37_0 conda-forge
$ conda list tornado
# packages in environment at /glade/work/abanihi/softwares/miniconda3/envs/analysis:
#
# Name Version Build Channel
tornado 5.1.1 py37h14c3975_1000 conda-forge
$ conda list distributed
# packages in environment at /glade/work/abanihi/softwares/miniconda3/envs/analysis:
#
# Name Version Build Channel
distributed 1.27.0 py37_0 conda-forgeIs anyone aware of anything that must have happened in an update to dask or distributed to cause dask-mpi to break?
Ccing @kmpaul
Now that the Black code formatter is used throughout most Dask repositories, are there any objections to including pre-commit hooks for taking care of formatting/code style?
Is it fair that we drop python 2.7 tests and no longer support python 2?
I'm struggling a bit to get my first batch job with dask-mpi going, and I think it would be helpful to have a full minimal example (if possible) added to the docs.
I'm wondering specifically two things:
source activate pangeo as in the job control script below doesn't seem to work.test-mpi.py code in a main() function for dask-mpi to work?My job script looks like this:
#!/bin/bash
#SBATCH -p compute
#SBATCH -A science
#SBATCH --mem=144G
#SBATCH -t 01:00:00
#SBATCH --nodes=2 # Number of nodes
#SBATCH --ntasks-per-node=36
echo "Running Dask-MPI"
module openmpi/intel/4.0.1
source activate pangeo
mpirun -np 72 python /vortexfs1/usgs/rsignell/HRRR/HRRR-s3-download-mpi.py
Hi!
I've been reading through the docs, but I'm still a bit unsure about whether the resilience/fault tolerance in dask-distributed in maintained in dask-mpi.
As I understood it, MPI is only used to discover nodes and not for communication, but would the fact that the execution has been launched with MPI in the first place imply that it will collapse in case a node fails?
Carlos
This seems usable? Should we release?
@kmpaul I'm happy to walk you through this if you like or just handle it for you.
Need to port over the original dask-mpi documentation from distributed. Then need to add documentation for the new initialize feature of dask-mpi.
I have a .circleci/config.yml file in place, but this should be reviewed before deployment.
Also, we might need the recent changes to dask/distributed to be released before we enable CI.
I've been able to manually launch a Dask GPU cluster using Slurm successfully. The setup for that looks like:
<get allocation>
dask-scheduler --scheduler-file scheduler.json &
srun -G <number-of-gpus-in-allocation> dask-cuda-worker --scheduler-file scheduler.json
This works OK for say, 2 nodes with 4 GPUs each but I find that when I request more workers, say, 80 GPUs, some of them just never come up, or they take an inordinately long amount of time to come up. I had this problem with manual start in a CPU-only context with Dask Distributed, and the fix for that kind of on-demand scale was dask-mpi. Combined with containers, this is a very good solution we've found for reliable Dask startup on HPC. I'm hoping for the same kind of solution with GPUs.
I've tried out dask-mpi support for dask_cuda.CUDAWorker but I can't seem to find a working invocation. I tried just adapting the above, or using something similar to what I've done before:
srun -G 40 -u python -u $(which dask-mpi) --scheduler-file scheduler.json \
--nthreads 1 --worker-class dask_cuda.CUDAWorker --dashboard-address 0
The first thing I noticed is that I have to specify --nthreads 1 because otherwise you get an error which might be easy to fix:
TypeError: '<' not supported between instances of 'NoneType' and 'int'
return self.main(*args, **kwargs)
File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/site-packages/dask_mpi/cli.py", line 147, in main
asyncio.get_event_loop().run_until_complete(run_worker())
File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/site-packages/dask_mpi/cli.py", line 144, in run_worker
async with WorkerType(**opts) as worker:
File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/site-packages/dask_cuda/cuda_worker.py", line 95, in __init__
if nthreads < 1:
TypeError: '<' not supported between instances of 'NoneType' and 'int'
Once you have --nthreads 1 in place though, you hit this:
Traceback (most recent call last):
File "/pscratch/sd/r/rthomas/dask/env/bin/dask-mpi", line 8, in <module>
sys.exit(go())
File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/site-packages/dask_mpi/cli.py", line 152, in go
main()
File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/site-packages/dask_mpi/cli.py", line 147, in main
asyncio.get_event_loop().run_until_complete(run_worker())
File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/site-packages/dask_mpi/cli.py", line 144, in run_worker
async with WorkerType(**opts) as worker:
File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/site-packages/dask_cuda/cuda_worker.py", line 216, in __init__
self.nannies = [
File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/site-packages/dask_cuda/cuda_worker.py", line 217, in <listcomp>
Nanny(
TypeError: __init__() got multiple values for argument 'scheduler_ip'
as if a keyword arg is trying to overwrite a positional one. I started wondering how people were trying this out themselves and thought maybe they weren't going with a scheduler file for the use cases tested up till now, but I had trouble properly specifying options and arguments for dask_cuda.CUDAWorker via --worker-options (they seem to not matter) or specifying SCHEDULER_ADDRESS on the command line.
# packages in environment at /pscratch/sd/r/rthomas/dask/env:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main defaults
_openmp_mutex 4.5 1_gnu defaults
abseil-cpp 20210324.2 h9c3ff4c_0 conda-forge
aiohttp 3.7.4.post0 py38h497a2fe_0 conda-forge
anyio 3.3.0 py38h578d9bd_0 conda-forge
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
argon2-cffi 20.1.0 py38h497a2fe_2 conda-forge
arrow-cpp 4.0.1 py38h9a0cccc_7_cuda conda-forge
arrow-cpp-proc 3.0.0 cuda conda-forge
async-timeout 3.0.1 py_1000 conda-forge
async_generator 1.10 py_0 conda-forge
attrs 21.2.0 pyhd8ed1ab_0 conda-forge
aws-c-cal 0.5.11 h95a6274_0 conda-forge
aws-c-common 0.6.2 h7f98852_0 conda-forge
aws-c-event-stream 0.2.7 h3541f99_13 conda-forge
aws-c-io 0.10.5 hfb6a706_0 conda-forge
aws-checksums 0.1.11 ha31a3da_7 conda-forge
aws-sdk-cpp 1.8.186 hb4091e7_3 conda-forge
backcall 0.2.0 pyh9f0ad1d_0 conda-forge
backports 1.0 py_2 conda-forge
backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge
bleach 4.1.0 pyhd8ed1ab_0 conda-forge
blosc 1.21.0 h9c3ff4c_0 conda-forge
bokeh 2.3.3 py38h578d9bd_0 conda-forge
boost 1.74.0 py38hc10631b_3 conda-forge
boost-cpp 1.74.0 h312852a_4 conda-forge
brotli 1.0.9 h7f98852_5 conda-forge
brotli-bin 1.0.9 h7f98852_5 conda-forge
brotlipy 0.7.0 py38h497a2fe_1001 conda-forge
brunsli 0.1 h9c3ff4c_0 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.17.1 h7f98852_1 conda-forge
ca-certificates 2021.5.30 ha878542_0 conda-forge
cachetools 4.2.2 pyhd8ed1ab_0 conda-forge
cairo 1.16.0 h6cf1ce9_1008 conda-forge
certifi 2021.5.30 py38h578d9bd_0 conda-forge
cffi 1.14.6 py38ha65f79e_0 conda-forge
cfitsio 3.470 hb418390_7 conda-forge
chardet 4.0.0 py38h578d9bd_1 conda-forge
charls 2.2.0 h9c3ff4c_0 conda-forge
charset-normalizer 2.0.4 pyhd3eb1b0_0 defaults
click 7.1.2 pyh9f0ad1d_0 conda-forge
click-plugins 1.1.1 py_0 conda-forge
cligj 0.7.2 pyhd8ed1ab_0 conda-forge
cloudpickle 1.6.0 py_0 conda-forge
colorcet 2.0.6 pyhd8ed1ab_0 conda-forge
conda 4.10.3 py38h578d9bd_0 conda-forge
conda-package-handling 1.7.3 py38h497a2fe_0 conda-forge
cryptography 3.4.7 py38ha5dfef3_0 conda-forge
cucim 21.08.01 cuda_11.0_py38_ga89f250_0 rapidsai
cudatoolkit 11.0.221 h6bb024c_0 nvidia
cudf 21.08.02 cuda_11.0_py38_gf6d31fa95d_0 rapidsai
cudf_kafka 21.08.02 py38_gf6d31fa95d_0 rapidsai
cugraph 21.08.03 cuda11.0_py38_g9e9f1570_0 rapidsai
cuml 21.08.01 cuda11.0_py38_g5c0e99300_0 rapidsai
cupy 9.0.0 py38hc350bd8_0 conda-forge
curl 7.78.0 hea6ffbf_0 conda-forge
cusignal 21.08.00 py37_g33f663e_0 rapidsai
cuspatial 21.08.01 py38_g7c0151b_0 rapidsai
custreamz 21.08.02 py38_gf6d31fa95d_0 rapidsai
cuxfilter 21.08.00 py38_g274c584_0 rapidsai
cycler 0.10.0 py_2 conda-forge
cyrus-sasl 2.1.27 h230043b_2 conda-forge
cytoolz 0.11.0 py38h497a2fe_3 conda-forge
dask 2021.7.1 pyhd8ed1ab_0 conda-forge
dask-core 2021.7.1 pyhd8ed1ab_0 conda-forge
dask-cuda 21.08.00 py38_0 rapidsai
dask-cudf 21.08.02 py38_gf6d31fa95d_0 rapidsai
dask-mpi 2.21.0+49.gccacb62 pypi_0 pypi
datashader 0.11.1 pyh9f0ad1d_0 conda-forge
datashape 0.5.4 py_1 conda-forge
debugpy 1.4.1 py38h709712a_0 conda-forge
decorator 4.4.2 py_0 conda-forge
defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge
distributed 2021.7.1 py38h578d9bd_0 conda-forge
dlpack 0.5 h9c3ff4c_0 conda-forge
entrypoints 0.3 pyhd8ed1ab_1003 conda-forge
expat 2.4.1 h9c3ff4c_0 conda-forge
faiss-proc 1.0.0 cuda rapidsai
fastavro 1.4.4 py38h497a2fe_0 conda-forge
fastrlock 0.6 py38h709712a_1 conda-forge
fiona 1.8.20 py38hdb5a769_0 conda-forge
fontconfig 2.13.1 hba837de_1005 conda-forge
freetype 2.10.4 h0708190_1 conda-forge
freexl 1.0.6 h7f98852_0 conda-forge
fsspec 2021.8.1 pyhd8ed1ab_0 conda-forge
gdal 3.2.2 py38h507a4fd_7 conda-forge
geopandas 0.9.0 pyhd8ed1ab_1 conda-forge
geopandas-base 0.9.0 pyhd8ed1ab_1 conda-forge
geos 3.9.1 h9c3ff4c_2 conda-forge
geotiff 1.6.0 h4f31c25_6 conda-forge
gettext 0.19.8.1 h0b5b191_1005 conda-forge
gflags 2.2.2 he1b5a44_1004 conda-forge
giflib 5.2.1 h36c2ea0_2 conda-forge
glog 0.5.0 h48cff8f_0 conda-forge
grpc-cpp 1.39.0 hf1f433d_2 conda-forge
hdf4 4.2.15 h10796ff_3 conda-forge
hdf5 1.10.6 nompi_h6a2412b_1114 conda-forge
heapdict 1.0.1 py_0 conda-forge
icu 68.1 h58526e2_0 conda-forge
idna 3.2 pyhd3eb1b0_0 defaults
imagecodecs 2021.7.30 py38hb5ce8f7_0 conda-forge
imageio 2.9.0 py_0 conda-forge
importlib-metadata 4.8.1 py38h578d9bd_0 conda-forge
ipykernel 6.3.1 py38he5a9106_0 conda-forge
ipython 7.27.0 py38he5a9106_0 conda-forge
ipython_genutils 0.2.0 py_1 conda-forge
ipywidgets 7.6.4 pyhd8ed1ab_0 conda-forge
jbig 2.1 h7f98852_2003 conda-forge
jedi 0.18.0 py38h578d9bd_2 conda-forge
jinja2 3.0.1 pyhd8ed1ab_0 conda-forge
joblib 1.0.1 pyhd8ed1ab_0 conda-forge
jpeg 9d h36c2ea0_0 conda-forge
json-c 0.15 h98cffda_0 conda-forge
jsonschema 3.2.0 pyhd8ed1ab_3 conda-forge
jupyter-server-proxy 3.1.0 pyhd8ed1ab_0 conda-forge
jupyter_client 7.0.2 pyhd8ed1ab_0 conda-forge
jupyter_core 4.7.1 py38h578d9bd_0 conda-forge
jupyter_server 1.10.2 pyhd8ed1ab_0 conda-forge
jupyterlab_pygments 0.1.2 pyh9f0ad1d_0 conda-forge
jupyterlab_widgets 1.0.1 pyhd8ed1ab_0 conda-forge
jxrlib 1.1 h7f98852_2 conda-forge
kealib 1.4.14 hcc255d8_2 conda-forge
kiwisolver 1.3.1 py38h1fd1430_1 conda-forge
krb5 1.19.2 hcc1bbae_0 conda-forge
lcms2 2.12 hddcbb42_0 conda-forge
ld_impl_linux-64 2.35.1 h7274673_9 defaults
lerc 2.2.1 h9c3ff4c_0 conda-forge
libaec 1.0.5 h9c3ff4c_0 conda-forge
libblas 3.9.0 11_linux64_openblas conda-forge
libbrotlicommon 1.0.9 h7f98852_5 conda-forge
libbrotlidec 1.0.9 h7f98852_5 conda-forge
libbrotlienc 1.0.9 h7f98852_5 conda-forge
libcblas 3.9.0 11_linux64_openblas conda-forge
libcucim 21.08.01 cuda11.0_ga89f250_0 rapidsai
libcudf 21.08.02 cuda11.0_gf6d31fa95d_0 rapidsai
libcudf_kafka 21.08.02 gf6d31fa95d_0 rapidsai
libcugraph 21.08.03 cuda11.0_g9e9f1570_0 rapidsai
libcuml 21.08.01 cuda11.0_g5c0e99300_0 rapidsai
libcumlprims 21.08.00 cuda11.0_g9c188ef_0 nvidia
libcurl 7.78.0 h2574ce0_0 conda-forge
libcuspatial 21.08.01 cuda11.0_g7c0151b_0 rapidsai
libdap4 3.20.6 hd7c4107_2 conda-forge
libdeflate 1.8 h7f98852_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libevent 2.1.10 hcdb4288_3 conda-forge
libfaiss 1.7.0 cuda110h8045045_8_cuda conda-forge
libffi 3.3 he6710b0_2 defaults
libgcc-ng 9.3.0 h5101ec6_17 defaults
libgcrypt 1.9.3 h7f98852_1 conda-forge
libgdal 3.2.2 h8f005ca_7 conda-forge
libgfortran-ng 11.1.0 h69a702a_8 conda-forge
libgfortran5 11.1.0 h6c583b3_8 conda-forge
libglib 2.68.3 h3e27bee_0 conda-forge
libgomp 9.3.0 h5101ec6_17 defaults
libgpg-error 1.42 h9c3ff4c_0 conda-forge
libgsasl 1.8.0 2 conda-forge
libhwloc 2.3.0 h5e5b7d1_1 conda-forge
libiconv 1.16 h516909a_0 conda-forge
libkml 1.3.0 h238a007_1014 conda-forge
liblapack 3.9.0 11_linux64_openblas conda-forge
libllvm10 10.0.1 he513fc3_3 conda-forge
libnetcdf 4.8.0 nompi_hcd642e3_103 conda-forge
libnghttp2 1.43.0 h812cca2_0 conda-forge
libntlm 1.4 h7f98852_1002 conda-forge
libopenblas 0.3.17 pthreads_h8fe5266_1 conda-forge
libpng 1.6.37 h21135ba_2 conda-forge
libpq 13.3 hd57d9b9_0 conda-forge
libprotobuf 3.16.0 h780b84a_0 conda-forge
librdkafka 1.6.1 hc49e61c_1 conda-forge
librmm 21.08.01 cuda11.0_g66cf439_0 rapidsai
librttopo 1.1.0 h1185371_6 conda-forge
libsodium 1.0.18 h36c2ea0_1 conda-forge
libspatialindex 1.9.3 h9c3ff4c_4 conda-forge
libspatialite 5.0.1 h8694cbe_5 conda-forge
libssh2 1.9.0 ha56f1ee_6 conda-forge
libstdcxx-ng 9.3.0 hd4cf53a_17 defaults
libthrift 0.14.2 he6d91bd_1 conda-forge
libtiff 4.3.0 hf544144_0 conda-forge
libutf8proc 2.6.1 h7f98852_0 conda-forge
libuuid 2.32.1 h7f98852_1000 conda-forge
libuv 1.42.0 h7f98852_0 conda-forge
libwebp 1.2.0 h3452ae3_0 conda-forge
libwebp-base 1.2.0 h7f98852_2 conda-forge
libxcb 1.13 h7f98852_1003 conda-forge
libxgboost 1.4.2dev.rapidsai21.08 cuda11.0_0 rapidsai
libxml2 2.9.12 h72842e0_0 conda-forge
libzip 1.8.0 h4de3113_0 conda-forge
libzopfli 1.0.3 h9c3ff4c_0 conda-forge
llvmlite 0.36.0 py38h4630a5e_0 conda-forge
locket 0.2.0 py_2 conda-forge
lz4-c 1.9.3 h9c3ff4c_1 conda-forge
mapclassify 2.4.3 pyhd8ed1ab_0 conda-forge
markdown 3.3.4 pyhd8ed1ab_0 conda-forge
markupsafe 2.0.1 py38h497a2fe_0 conda-forge
matplotlib-base 3.4.2 py38hcc49a3a_0 conda-forge
matplotlib-inline 0.1.2 pyhd8ed1ab_2 conda-forge
mistune 0.8.4 py38h497a2fe_1004 conda-forge
mpi4py 3.1.1 pypi_0 pypi
msgpack-python 1.0.2 py38h1fd1430_1 conda-forge
multidict 5.1.0 py38h497a2fe_1 conda-forge
multipledispatch 0.6.0 py_0 conda-forge
munch 2.5.0 py_0 conda-forge
nbclient 0.5.4 pyhd8ed1ab_0 conda-forge
nbconvert 6.1.0 py38h578d9bd_0 conda-forge
nbformat 5.1.3 pyhd8ed1ab_0 conda-forge
nccl 2.10.3.1 h96e36e3_0 conda-forge
ncurses 6.2 he6710b0_1 defaults
nest-asyncio 1.5.1 pyhd8ed1ab_0 conda-forge
networkx 2.6.2 pyhd8ed1ab_0 conda-forge
nodejs 14.17.4 h92b4a50_0 conda-forge
notebook 6.4.3 pyha770c72_0 conda-forge
numba 0.53.1 py38h8b71fd7_1 conda-forge
numpy 1.21.1 py38h9894fe3_0 conda-forge
nvtx 0.2.3 py38h497a2fe_0 conda-forge
olefile 0.46 pyh9f0ad1d_1 conda-forge
openjpeg 2.4.0 hb52868f_1 conda-forge
openssl 1.1.1k h7f98852_0 conda-forge
orc 1.6.9 h58a87f1_0 conda-forge
packaging 21.0 pyhd8ed1ab_0 conda-forge
pandas 1.2.5 py38h1abd341_0 conda-forge
pandoc 2.14.2 h7f98852_0 conda-forge
pandocfilters 1.4.2 py_1 conda-forge
panel 0.12.1 pyhd8ed1ab_0 conda-forge
param 1.11.1 pyh6c4a22f_0 conda-forge
parquet-cpp 1.5.1 2 conda-forge
parso 0.8.2 pyhd8ed1ab_0 conda-forge
partd 1.2.0 pyhd8ed1ab_0 conda-forge
pcre 8.45 h9c3ff4c_0 conda-forge
pexpect 4.8.0 pyh9f0ad1d_2 conda-forge
pickleshare 0.7.5 py_1003 conda-forge
pillow 8.3.1 py38h8e6f84c_0 conda-forge
pip 21.2.4 pyhd8ed1ab_0 conda-forge
pixman 0.40.0 h36c2ea0_0 conda-forge
pooch 1.5.1 pyhd8ed1ab_0 conda-forge
poppler 21.03.0 h93df280_0 conda-forge
poppler-data 0.4.10 0 conda-forge
postgresql 13.3 h2510834_0 conda-forge
proj 8.0.1 h277dcde_0 conda-forge
prometheus_client 0.11.0 pyhd8ed1ab_0 conda-forge
prompt-toolkit 3.0.20 pyha770c72_0 conda-forge
protobuf 3.16.0 py38h709712a_0 conda-forge
psutil 5.8.0 py38h497a2fe_1 conda-forge
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge
py-xgboost 1.4.2dev.rapidsai21.08 cuda11.0py38_0 rapidsai
pyarrow 4.0.1 py38hdd2221d_7_cuda conda-forge
pycosat 0.6.3 py38h497a2fe_1006 conda-forge
pycparser 2.20 py_2 defaults
pyct 0.4.6 py_0 conda-forge
pyct-core 0.4.6 py_0 conda-forge
pydeck 0.5.0 pyh9f0ad1d_0 conda-forge
pygments 2.10.0 pyhd8ed1ab_0 conda-forge
pynvml 11.0.0 pyhd8ed1ab_0 conda-forge
pyopenssl 20.0.1 pyhd3eb1b0_1 defaults
pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge
pyproj 3.1.0 py38h03a1999_3 conda-forge
pyrsistent 0.17.3 py38h497a2fe_2 conda-forge
pysocks 1.7.1 py38h578d9bd_3 conda-forge
python 3.8.10 h49503c6_1_cpython conda-forge
python-confluent-kafka 1.6.0 py38h497a2fe_1 conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python_abi 3.8 2_cp38 conda-forge
pytz 2021.1 pyhd8ed1ab_0 conda-forge
pyviz_comms 2.1.0 pyhd8ed1ab_0 conda-forge
pywavelets 1.1.1 py38h5c078b8_3 conda-forge
pyyaml 5.4.1 py38h497a2fe_0 conda-forge
pyzmq 22.1.0 py38h2035c66_0 conda-forge
rapids 21.08.00 cuda11.0_py38_ga776927_74 rapidsai
rapids-xgboost 21.08.00 cuda11.0_py38_ga776927_74 rapidsai
re2 2021.08.01 h9c3ff4c_0 conda-forge
readline 8.1 h27cfd23_0 defaults
requests 2.26.0 pyhd3eb1b0_0 defaults
requests-unixsocket 0.2.0 py_0 conda-forge
rmm 21.08.01 cuda_11.0_py38_g66cf439_0 rapidsai
rtree 0.9.7 py38h02d302b_2 conda-forge
ruamel_yaml 0.15.80 py38h497a2fe_1004 conda-forge
s2n 1.0.10 h9b69904_0 conda-forge
scikit-image 0.18.1 py38h51da96c_0 conda-forge
scikit-learn 0.24.2 py38hdc147b9_0 conda-forge
scipy 1.7.0 py38h7b17777_1 conda-forge
send2trash 1.8.0 pyhd8ed1ab_0 conda-forge
setuptools 49.6.0 py38h578d9bd_3 conda-forge
shapely 1.7.1 py38haeee4fe_5 conda-forge
simpervisor 0.4 pyhd8ed1ab_0 conda-forge
six 1.16.0 pyhd3eb1b0_0 defaults
snappy 1.1.8 he1b5a44_3 conda-forge
sniffio 1.2.0 py38h578d9bd_1 conda-forge
sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge
spdlog 1.8.5 h4bd325d_0 conda-forge
sqlite 3.36.0 hc218d9a_0 defaults
streamz 0.6.2 pyh44b312d_0 conda-forge
tblib 1.7.0 pyhd8ed1ab_0 conda-forge
terminado 0.11.1 py38h578d9bd_0 conda-forge
testpath 0.5.0 pyhd8ed1ab_0 conda-forge
threadpoolctl 2.2.0 pyh8a188c0_0 conda-forge
tifffile 2021.8.30 pyhd8ed1ab_0 conda-forge
tiledb 2.3.2 he87e0bf_0 conda-forge
tk 8.6.10 hbc83047_0 defaults
toolz 0.11.1 py_0 conda-forge
tornado 6.1 py38h497a2fe_1 conda-forge
tqdm 4.62.1 pyhd3eb1b0_1 defaults
traitlets 5.1.0 pyhd8ed1ab_0 conda-forge
treelite 2.0.0 py38hc9ad5e7_0 conda-forge
treelite-runtime 2.0.0 pypi_0 pypi
typing-extensions 3.10.0.0 hd8ed1ab_0 conda-forge
typing_extensions 3.10.0.0 pyha770c72_0 conda-forge
tzcode 2021a h7f98852_2 conda-forge
tzdata 2021a h5d7bf9c_0 defaults
ucx 1.9.0+gcd9efd3 cuda11.0_0 rapidsai
ucx-proc 1.0.0 gpu rapidsai
ucx-py 0.21.0 py38_gcd9efd3_0 rapidsai
urllib3 1.26.6 pyhd3eb1b0_1 defaults
wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge
webencodings 0.5.1 py_1 conda-forge
websocket-client 0.57.0 py38h578d9bd_4 conda-forge
wheel 0.37.0 pyhd8ed1ab_1 conda-forge
widgetsnbextension 3.5.1 py38h578d9bd_4 conda-forge
xarray 0.19.0 pyhd8ed1ab_1 conda-forge
xerces-c 3.2.3 h9d8b166_2 conda-forge
xgboost 1.4.2dev.rapidsai21.08 cuda11.0py38_0 rapidsai
xorg-kbproto 1.0.7 h7f98852_1002 conda-forge
xorg-libice 1.0.10 h7f98852_0 conda-forge
xorg-libsm 1.2.3 hd9c2040_1000 conda-forge
xorg-libx11 1.7.2 h7f98852_0 conda-forge
xorg-libxau 1.0.9 h7f98852_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xorg-libxext 1.3.4 h7f98852_1 conda-forge
xorg-libxrender 0.9.10 h7f98852_1003 conda-forge
xorg-renderproto 0.11.1 h7f98852_1002 conda-forge
xorg-xextproto 7.3.0 h7f98852_1002 conda-forge
xorg-xproto 7.0.31 h7f98852_1007 conda-forge
xz 5.2.5 h7b6447c_0 defaults
yaml 0.2.5 h7b6447c_0 defaults
yarl 1.6.3 py38h497a2fe_2 conda-forge
zeromq 4.3.4 h9c3ff4c_0 conda-forge
zfp 0.5.5 h9c3ff4c_5 conda-forge
zict 2.0.0 py_0 conda-forge
zipp 3.5.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.11 h7b6447c_3 defaults
zstd 1.5.0 ha95c52a_0 conda-forge
For completeness, I did this to set up my Dask cluster env and Jupyter kernel env:
pip install --no-cache-dir --force git+https://github.com/dask/dask-mpi (to keep from overwriting my mpi4py)What happened:
When dask-mpi CLI is used, it attempts to import check_python_3 from distributed.cli.utils, which now fails because that function has been removed.
What you expected to happen:
We shouldn't need to check for Python 3 any more.
Minimal Complete Verifiable Example:
$ conda create -y --name test dask-mpi
$ dask-mpi --help
Traceback (most recent call last):
File ".../envs/daskmpi/bin/dask-mpi", line 7, in <module>
from dask_mpi.cli import go
File ".../envs/daskmpi/lib/python3.10/site-packages/dask_mpi/cli.py", line 6, in <module>
from distributed.cli.utils import check_python_3
ImportError: cannot import name 'check_python_3' from 'distributed.cli.utils' (.../envs/daskmpi/lib/python3.10/site-packages/distributed/cli/utils.py)Anything else we need to know?:
I'm working on a PR to fix this now.
Environment:
Due to changes in the Travis CI billing, the Dask org is migrating Travis CI to GitHub Actions.
This repo appears to use CircleCI. As we are putting in the effort to migrate many projects to GitHub Actions does it make sense to standardise here?
See dask/community#107 for more details.
@jrbourbeau and I are in the process of moving the default branch for this repo from master to main.
Once the name on github is changed (the first box above is Xed, or this issue closed), when you try to git pull you'll get
Your configuration specifies to merge with the ref 'refs/heads/master'
from the remote, but no such ref was fetched.First: head to your fork and rename the default branch there
Then:
git branch -m master main
git fetch origin
git branch -u origin/main mainWhat happened:
I was attempting to start a dask-mpi cluster on 20 (admittedly slow) 68-core Intel KNL nodes (1320 workers, each with a single thread). I observed the scheduler start and the workers attempt to start and connect, but eventually fail with messages like
distributed.comm.tcp - INFO - Connection from tcp://10.128.9.135:41230 closed before handshake completed
Knowing that KNL has a slow clock speed of 1.4 GHz, I attempted to increase the timeout by setting values in my ~/.config/dask/config.yaml file as recommended in the Dask docs. I also attempted to set environment variables via export DASK_DISTRIBUTED__COMM_TIMEOUTS_CONNECT=240s.
I tried a very extreme case where I set
export DASK_DISTRIBUTED__COMM_TIMEOUTS_CONNECT=1000s
export DASK_DISTRIBUTED__COMM_TIMEOUTS_TCP=1000s
but I still saw timeout failures within a minute or two while dask-mpi attempted to start my cluster, so based on that it seems like dask-mpi is not respecting these values.
What you expected to happen:
I would like/expect dask-mpi to use the configuration options advertised in the dask docs like ~/.config/dask/config.yaml. It's not clear to me if it does or should. Whatever the outcome of this issue is, it would be helpful to add a note to the docs about whether dask-mpi does support these configuration options.
Minimal Complete Verifiable Example:
I launched my dask-mpi cluster on NERSC's Cori system with the following commands inside a custom conda enviornment:
salloc --nodes=20 --ntasks=1360 --cpus-per-task=1 --time=240 --constraint=knl --qos=interactive
export OMP_NUM_THREADS=1
export PYTHONUNBUFFERED=1
export DASK_DISTRIBUTED__COMM_TIMEOUTS_CONNECT=1000s
export DASK_DISTRIBUTED__COMM_TIMEOUTS_TCP=1000s
srun -u dask-mpi --scheduler-file=scheduler.json --dashboard-address=0 --memory-limit="1.2 GiB" --nthreads=1 --no-nanny --local-directory=/tmp
After 1-2 minutes, I saw many timeout messages:
distributed.nanny - INFO - Closing Nanny at 'tcp://10.128.9.121:44235'
distributed.nanny - INFO - Closing Nanny at 'tcp://10.128.9.113:41129'
distributed.nanny - INFO - Closing Nanny at 'tcp://10.128.9.126:37758'
distributed.nanny - INFO - Closing Nanny at 'tcp://10.128.9.123:45557'
distributed.nanny - INFO - Closing Nanny at 'tcp://10.128.9.116:58355'
distributed.nanny - INFO - Closing Nanny at 'tcp://10.128.9.129:35394'
distributed.nanny - INFO - Closing Nanny at 'tcp://10.128.9.126:58587'
distributed.nanny - INFO - Closing Nanny at 'tcp://10.128.9.113:57881'
Anything else we need to know?:
Environment:
conda list | grep "dask"
dask 2021.10.0 pyhd3eb1b0_0
dask-core 2021.10.0 pyhd3eb1b0_0
dask-mpi 2.21.0 py38h4ecba47_2 conda-forge
python --version
Python 3.8.8
cat /etc/os-release
NAME="SLES"
VERSION="15"
VERSION_ID="15"
PRETTY_NAME="SUSE Linux Enterprise Server 15"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15"
condaThank you very much,
Laurie
I would like to explore the possibility of Dask starting MPI. This is sort of the reverse behavior of what the dask-mpi package does today.
To clarify the situation I'm talking about, consider the situation where we are running Dask with some other system like Kubernetes, Yarn, or SLURM, and are doing Dask's normal dynamic computations. We scale up and down, load data from disk, do some preprocessing. Then, we want to run some MPI code on the data we have in memory in the Dask worker processes. We don't currently have an MPI world set up (our workers were not started with mpirun or anything) but would like to create one. Is this possible?
To do this, Dask will have to go through whatever process mpirun/mpiexec goes through to set up an MPI communicator. What is this process?
Ideally it would be able to do this without launching new processes. Ideally the same processes currently running the Dask workers would initialize some MPI code, be told about each other, then run some MPI program, then shut down MPI and continue on with normal Dask work.
We'll need to become careful about what to do if a worker goes away during this process. We'll probably have to restart the MPI job, which will be fine. I think that I can handle that on the scheduling/resiliency side.
I think that the people who know the answer to these questions will be people who have experience not only in using MPI, but also in deploying it.
Is this supported? We have the ability (through Slurm) to srun ... dask-mpi... which will launch the cluster on a queue set aside for interactive work. From the command line it's a single command. Other facilities may not have this capability (it may require a batch script).
We're willing to take a look into how to make this work if it's needed. cc @shreddd
This is essentially a move of dask/distributed#2402 to this repository.
As a quick summary, the idea is to make it possible to launch the dask network (scheduler and workers) from within an existing MPI environment (i.e., having run the script with mpirun). Launching should be as simple as:
mpirun -np N python my_dask_script.pyAnd the dask network should be launched from within the my_dask_script.py file from an import, as described in the first option presented in @mrocklin's comment: dask/distributed#2402 (comment).
Any objections to a bugfix release?
Would this be as simple as modifying the code around https://github.com/dask/distributed/blob/master/distributed/cli/dask_mpi.py#L63 and https://github.com/dask/distributed/blob/master/distributed/cli/dask_mpi.py#L79?
@mrocklin Created this repository to move the dask-mpi code out of distributed. Obviously, it should go here.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.