Scheduled Jobs No Longer Running - Status "Stopped" about jupyter-scheduler

[andrii-i]

Hi @TallGibbs. Thank you for creating this issue. Most of our users are not running Windows so feedback from Windows users is especially valuable. Looking into this.

[TallGibbs]

Ok I understand and will try that next (probably tomorrow) and will let you know.

[TallGibbs]

Here is the Anaconda Prompt terminal output when the error occurs trying to schedule a job:
[E 2024-06-20 13:44:47.467 SchedulerApp] module 'fsspec' has no attribute 'open'
Traceback (most recent call last):
File "C:\Users\XXXXX\AppData\Local\anaconda3\envs\Projects\Lib\site-packages\jupyter_scheduler\handlers.py", line 127, in post
self.scheduler.create_job_definition(CreateJobDefinition(**payload))
File "C:\Users\XXXXX\AppData\Local\anaconda3\envs\Projects\Lib\site-packages\jupyter_scheduler\scheduler.py", line 544, in create_job_definition
self.copy_input_file(model.input_uri, staging_paths["input"])
File "C:\Users\XXXXX\AppData\Local\anaconda3\envs\Projects\Lib\site-packages\jupyter_scheduler\scheduler.py", line 370, in copy_input_file
with fsspec.open(input_filepath) as input_file:
^^^^^^^^^^^
AttributeError: module 'fsspec' has no attribute 'open'
[W 2024-06-20 13:44:47.469 ServerApp] 500 POST /scheduler/job_definitions?1718905487425 (::1): Unexpected error occurred during creation of job definition.
[W 2024-06-20 13:44:47.475 SchedulerApp] wrote error: 'Unexpected error occurred during creation of job definition.'
Traceback (most recent call last):
File "C:\Users\XXXXX\AppData\Local\anaconda3\envs\Projects\Lib\site-packages\jupyter_scheduler\handlers.py", line 127, in post
self.scheduler.create_job_definition(CreateJobDefinition(**payload))
File "C:\Users\XXXXX\AppData\Local\anaconda3\envs\Projects\Lib\site-packages\jupyter_scheduler\scheduler.py", line 544, in create_job_definition
self.copy_input_file(model.input_uri, staging_paths["input"])
File "C:\Users\XXXXX\AppData\Local\anaconda3\envs\Projects\Lib\site-packages\jupyter_scheduler\scheduler.py", line 370, in copy_input_file
with fsspec.open(input_filepath) as input_file:
^^^^^^^^^^^
AttributeError: module 'fsspec' has no attribute 'open'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\XXXXX\AppData\Local\anaconda3\envs\Projects\Lib\site-packages\tornado\web.py", line 1790, in _execute
    result = await result
             ^^^^^^^^^^^^
  File "C:\Users\XXXXX\AppData\Local\anaconda3\envs\Projects\Lib\site-packages\jupyter_scheduler\handlers.py", line 140, in post
    raise HTTPError(
tornado.web.HTTPError: HTTP 500: Internal Server Error (Unexpected error occurred during creation of job definition.)

[E 2024-06-20 13:44:47.478 SchedulerApp] {
"Host": "localhost:8888",
"Accept": "/",
"Referer": "http://localhost:8888/lab/tree/SCM%20DB/Policy%20%26%20User%20Guides%20%26%20Updated%20Doc/.DWH/Production%20Ready%20Jupyter%20Notebooks",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
}
[E 2024-06-20 13:44:47.478 SchedulerApp] 500 POST /scheduler/job_definitions?1718905487425 (c3adefb8021548849d573467c4ef4bb5@::1) 51.01ms referer=http://localhost:8888/lab/tree/SCM%20DB/Policy%20%26%20User%20Guides%20%26%20Updated%20Doc/.DWH/Production%20Ready%20Jupyter%20Notebooks

[TallGibbs]

@andrii-i Today I ran a test with an odd result. I have a windows server provisioned to me, so I logged into that server. I downloaded Anaconda Navigator from scratch, then created a new environment using a .yaml backup file from my computer that is giving me the "unexpected error." Using this identical environment, I was able to "run now" through Jupyter Lab's "create job" function. This surprised me that it worked using the identical environment.

What could be different? When it worked on my new server, I was using Microsoft Edge for the web browser. So, I went back to my "old" computer and opened Jupyter Lab in Edge but I got the same "unexpected error."

The server is running Windows Server 2022 Standard, so maybe that makes a differences vs. Windows 10 on my old computer?

[andrii-i]

@TallGibbs thank you for the update. The error message reads AttributeError: module 'fsspec' has no attribute 'open' so it seems problem has to do with 'fsspec' dependency in your Windows 10 environment.

Could you please try creating a fresh conda environment, installing JupyterLab and Jupyter Scheduler, and seeing if the problem persists?

[TallGibbs]

@andrii-i

Just to confirm: create the fresh conda environment that is identical to "root", and only add required packages from there. Correct? Do this on my Windows 10 machine?

Do NOT make a new environment with my .yaml file because that would/should just be identical to what I am using now on my Windows 10 machine.

[andrii-i]

@TallGibbs I wanted you to try reproducing the error in a minimal new conda environment on your Windows 10 machine. In terms of packages, please try only installing JupyterLab and Jupyter Scheduler at first and seeing if you can repro the problem. This way it would be clear if problem has to do with .yaml file and combination of packages inside it.

[TallGibbs]

@andrii-i
Doing the fresh environment with only pandas, Jupyter Lab and Jupyter Scheduler resolved the issue. Interestinly, between my original environment and this new environment it still uses the same version of fsspec (2023.6.0), so some other package is modifying that package somehow?

See attached for documentation on this new enviornments "jupyter troubleshoot" output; this is still on Windows 10 but the clean environment which now allows jobs to be scheduled.
Scheduled Jobs Documentation_v2_schedule_test.docx

[TallGibbs]

Pasting the results here in case you don't want to open an attachment:

(schedule_test) C:>jupyter troubleshoot
$PATH:
C:\Users\XXXXX\AppData\Local\anaconda3\envs\schedule_test
C:\Users\XXXXX\AppData\Local\anaconda3\envs\schedule_test\Library\mingw-w64\bin
C:\Users\XXXXX\AppData\Local\anaconda3\envs\schedule_test\Library\usr\bin
C:\Users\XXXXX\AppData\Local\anaconda3\envs\schedule_test\Library\bin
C:\Users\XXXXX\AppData\Local\anaconda3\envs\schedule_test\Scripts
C:\Users\XXXXX\AppData\Local\anaconda3\envs\schedule_test\bin
C:\Users\XXXXX\AppData\Local\anaconda3\condabin
C:\WINDOWS\system32
C:\WINDOWS
C:\WINDOWS\System32\Wbem
C:\WINDOWS\System32\WindowsPowerShell\v1.0
C:\WINDOWS\System32\OpenSSH
C:\Program Files\Citrix\System32
C:\Program Files\Citrix\ICAService
C:\app\client\32\product\19.0.0\client_1
C:\app\client\64\product\19.0.0\client_1\bin
C:\Program Files\PuTTY
C:\Program Files (x86)\Webex\Plugins
C:\ProgramData\Oracle\Java\javapath
C:\Program Files (x86)\Microsoft SQL Server\150\DTS\Binn
C:\Program Files\Azure Data Studio\bin
C:\Program Files\Python310\Scripts
C:\Program Files\Python310
C:\Program Files (x86)\Aperio\Common
C:\Program Files\dotnet
C:\Program Files (x86)\Citrix\HDX\bin
C:\Program Files\Citrix\HDX\bin
C:\Users\XXXXX\AppData\Local\Microsoft\WindowsApps
.
C:\Program Files\Azure Data Studio\bin
C:\Users\XXXXX.dotnet\tools
C:\Users\XXXXX\AppData\Local\Programs\Microsoft VS Code\bin

sys.path:
C:\Users\XXXXX\AppData\Local\anaconda3\envs\schedule_test\Scripts
C:\Users\XXXXX\AppData\Local\anaconda3\envs\schedule_test\python311.zip
C:\Users\XXXXX\AppData\Local\anaconda3\envs\schedule_test\DLLs
C:\Users\XXXXX\AppData\Local\anaconda3\envs\schedule_test\Lib
C:\Users\XXXXX\AppData\Local\anaconda3\envs\schedule_test
C:\Users\XXXXX\AppData\Roaming\Python\Python311\site-packages
C:\Users\XXXXX\AppData\Roaming\Python\Python311\site-packages\win32
C:\Users\XXXXX\AppData\Roaming\Python\Python311\site-packages\win32\lib
C:\Users\XXXXX\AppData\Roaming\Python\Python311\site-packages\Pythonwin
C:\Users\XXXXX\AppData\Local\anaconda3\envs\schedule_test\Lib\site-packages
C:\Users\XXXXX\AppData\Local\anaconda3\envs\schedule_test\Lib\site-packages\win32
C:\Users\XXXXX\AppData\Local\anaconda3\envs\schedule_test\Lib\site-packages\win32\lib
C:\Users\XXXXX\AppData\Local\anaconda3\envs\schedule_test\Lib\site-packages\Pythonwin

sys.executable:
C:\Users\XXXXX\AppData\Local\anaconda3\envs\schedule_test\python.exe

sys.version:
3.11.9 | packaged by Anaconda, Inc. | (main, Apr 19 2024, 16:40:41) [MSC v.1916 64 bit (AMD64)]

platform.platform():
Windows-10-10.0.19045-SP0

where jupyter:
C:\Users\XXXXX\AppData\Local\anaconda3\envs\schedule_test\Scripts\jupyter.exe

pip list:
Package Version
------------------------- --------------
annotated-types 0.7.0
anyio 4.2.0
argon2-cffi 21.3.0
argon2-cffi-bindings 21.2.0
arrow 1.3.0
asttokens 2.0.5
async-lru 2.0.4
attrs 23.1.0
Babel 2.11.0
beautifulsoup4 4.12.3
bleach 4.1.0
Bottleneck 1.3.7
Brotli 1.0.9
certifi 2024.6.2
cffi 1.16.0
charset-normalizer 2.0.4
colorama 0.4.6
comm 0.2.1
croniter 1.4.1
debugpy 1.6.7
decorator 5.1.1
defusedxml 0.7.1
executing 0.8.3
fastjsonschema 2.16.2
fqdn 1.5.1
fsspec 2023.6.0
greenlet 3.0.3
idna 3.7
ipykernel 6.28.0
ipython 8.25.0
isoduration 20.11.0
jedi 0.18.1
Jinja2 3.1.4
json5 0.9.6
jsonpointer 3.0.0
jsonschema 4.19.2
jsonschema-specifications 2023.7.1
jupyter_client 8.6.0
jupyter_core 5.7.2
jupyter-events 0.10.0
jupyter-lsp 2.2.0
jupyter_scheduler 2.7.1
jupyter_server 2.14.1
jupyter_server_terminals 0.4.4
jupyterlab 4.0.11
jupyterlab-pygments 0.1.2
jupyterlab_server 2.25.1
MarkupSafe 2.1.3
matplotlib-inline 0.1.6
mistune 2.0.4
mkl-fft 1.3.8
mkl-random 1.2.4
mkl-service 2.4.0
nbclient 0.8.0
nbconvert 7.10.0
nbformat 5.9.2
nest-asyncio 1.6.0
notebook_shim 0.2.3
numexpr 2.8.7
numpy 1.26.4
overrides 7.4.0
packaging 23.2
pandas 2.2.2
pandocfilters 1.5.0
parso 0.8.3
pip 24.0
platformdirs 3.10.0
prometheus-client 0.14.1
prompt-toolkit 3.0.43
psutil 5.9.0
pure-eval 0.2.2
pycparser 2.21
pydantic 2.7.4
pydantic_core 2.18.4
Pygments 2.15.1
PySocks 1.7.1
python-dateutil 2.9.0.post0
python-json-logger 2.0.7
pytz 2023.3
pywin32 306
pywinpty 2.0.10
PyYAML 6.0.1
pyzmq 25.1.2
referencing 0.30.2
requests 2.32.2
rfc3339-validator 0.1.4
rfc3986-validator 0.1.1
rpds-py 0.10.6
Send2Trash 1.8.2
setuptools 69.5.1
six 1.16.0
sniffio 1.3.0
soupsieve 2.5
SQLAlchemy 2.0.31
stack-data 0.2.0
terminado 0.17.1
tinycss2 1.2.1
tornado 6.4.1
traitlets 5.14.3
types-python-dateutil 2.9.0.20240316
typing_extensions 4.11.0
tzdata 2023.3
uri-template 1.3.0
urllib3 2.2.2
wcwidth 0.2.5
webcolors 24.6.0
webencodings 0.5.1
websocket-client 1.8.0
wheel 0.43.0
win-inet-pton 1.1.0

[andrii-i]

@TallGibbs Thank you for the update. I'm happy that Jupyter Scheduler works correctly. It's hard to say what exactly went wrong with fsspec. It also might have to do with fsspec installation.

One thing that I noticed is that in that new environment (no error) uses latest version of jupyter_scheduler 2.7.1 while old one (with error) uses jupyter_scheduler 2.4.0. Maybe updating jupyter_scheduler in your .yaml file to latest version or even removing version pins and letting conda re-resolve dependencies would help?

[TallGibbs]

@andrii-i I did "pip uninstall jupyter_scheduler", then opened a new session of jupyter lab and installed jupyter_scheduler from the "extensions" within jupyter lab. This installed the newest 2.7.1 version and when I tried to "run now" on a notebook, I got the same error. So, simply updating this 1 package is not resolving the conflict; there must be conflicts in other packages which got triggered when I did my multiple updates at once back on June 12th.

For now, I'm going to see if I can fully build out my new environment to include ALL packages needed to run my script and see if I get the scheduler to work.

[andrii-i]

Closing for now as issue is due to interaction of fsspec package with other unidentified packages / dependency(s), not with Jupyter Scheduler. Please feel free to reopen is new information appears.