Logging issue in jupyter kernel when logs are issued in a Thread about mne-python

[hoechenberger]

Not a solution, but have you considered using asyncio instead of threads?

[hoechenberger]

This works for me:

# %%
import asyncio
from mne.utils import logger

k = 0

async def foo():
    global k
    try:
        while k < 2000:
            logger.info("test %i", k)
            k += 1
            await asyncio.sleep(0.00001)
    except Exception as error:
        raise error

# Create and run the async task using the existing event loop in IPython / Jupyter
task = asyncio.create_task(foo())
await task

[mscheltienne]

Which is logic as this is now running in a single thread. I started using threading in mne-lsl and changing the structure to asyncio would be a lot of work for an unclear benefit to me. I still have no idea why this snippet is failing in interactive windows..

[mscheltienne]

But that one does work:

import logging
import sys
import threading
from concurrent.futures import ThreadPoolExecutor

global logger
logger = logging.getLogger("test")
handler = logging.StreamHandler(sys.stdout)
logger.setLevel(logging.INFO)
logger.addHandler(handler)
global k
k = 0

executor = ThreadPoolExecutor(max_workers=10)  # Limit the number of worker threads


def foo():
    global k
    logger.info("test %i", k)
    k += 1
    if k == 3000:
        return
    executor.submit(foo)  # Submit the task to the thread pool


# Start the first task
executor.submit(foo)

If I get that right, ThreadPoolExecutor is limiting the number of workers and re-using them for future tasks; while my example was spawning/destroying thread rapidly. That seems to be the issue and I don't see anything to be done on MNE's logger side to prevent this from happening. Closing..