Deployment on custom IP, problem with worker connection about mercury

[pplonski]

Hi @MrStrzelec,

Have you tried to start with mercury run? Additionally you can start server in verbose mode mercury run --verbose.

If you would like to run each part of architecture by your own I would recommend checking docker entrypoint script: https://github.com/mljar/mercury/blob/main/docker/mercury/entrypoint.sh

In your case, looks like you are missing: -Q celery,ws, please try to run celery -A server worker --loglevel=info -P gevent --concurrency 1 -E -Q celery,ws

Please let me know if it works for you.

[MrStrzelec]

Hey!

Server working.
I've managed also to change celery script and adjust to your advice but not working. I've got this error from celery (systemd).

Okt 30 09:22:45 Servername celery[1155]: return request("get", url, params=params, **kwargs)
Okt 30 09:22:45 Servername celery[1155]: File "/opt/jupyterhub/lib/python3.10/site-packages/requests/api.py", line 59, in request
Okt 30 09:22:45 Servername celery[1155]: return session.request(method=method, url=url, **kwargs)
Okt 30 09:22:45 Servername celery[1155]: File "/opt/jupyterhub/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
Okt 30 09:22:45 Servername celery[1155]: resp = self.send(prep, **send_kwargs)
Okt 30 09:22:45 Servername celery[1155]: File "/opt/jupyterhub/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
Okt 30 09:22:45 Servername celery[1155]: r = adapter.send(request, **kwargs)
Okt 30 09:22:45 Servername celery[1155]: File "/opt/jupyterhub/lib/python3.10/site-packages/requests/adapters.py", line 519, in send
Okt 30 09:22:45 Servername celery[1155]: raise ConnectionError(e, request=request)
Okt 30 09:22:45 Servername celery[1155]: requests.exceptions.ConnectionError: HTTPConnectionPool(host=Servername', port=5668): Max retries exceeded with url: /api/v1/worker/18db8815-183d-4f84-929b-40522256a3e2/387/12/nb (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb144005870>:Failed to establish a new connection: [Errno 111] Connection refused'))

[pplonski]

I have no idea right now what might be the issue. Is it running on 127.0.0.1?

[MrStrzelec]

It's running on a diffrent IP address. I thought i add this information.

python manage.py runserver IPADDRESS:PORT

[pplonski]

Could you please add --verbose argument to server start command? Let's check if worker is connecting to the correct IP address.

[MrStrzelec]

user@server:/opt/jupyterhub/lib/python3.10/site-packages/mercury$ python manage.py runsever --verbose IP:PORT
Traceback (most recent call last):

  File "/usr/lib/python3.10/logging/config.py", line 565, in configure
    handler = self.configure_handler(handlers[name])
  File "/usr/lib/python3.10/logging/config.py", line 746, in configure_handler
    result = factory(**kwargs)
  File "/usr/lib/python3.10/logging/__init__.py", line 1169, in __init__
    StreamHandler.__init__(self, self._open())
  File "/usr/lib/python3.10/logging/__init__.py", line 1201, in _open
    return open_func(self.baseFilename, self.mode,
PermissionError: [Errno 13] Permission denied: '/opt/jupyterhub/lib/python3.10/site-packages/mercury/django-errors.log'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

 File "/opt/jupyterhub/lib/python3.10/site-packages/mercury/manage.py", line 22, in <module>
    main()
  File "/opt/jupyterhub/lib/python3.10/site-packages/mercury/manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/opt/jupyterhub/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/opt/jupyterhub/lib/python3.10/site-packages/django/core/management/__init__.py", line 416, in execute
    django.setup()
  File "/opt/jupyterhub/lib/python3.10/site-packages/django/__init__.py", line 19, in setup
    configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
  File "/opt/jupyterhub/lib/python3.10/site-packages/django/utils/log.py", line 76, in configure_logging
    logging_config_func(logging_settings)
  File "/usr/lib/python3.10/logging/config.py", line 811, in dictConfig
    dictConfigClass(config).configure()
  File "/usr/lib/python3.10/logging/config.py", line 572, in configure
    raise ValueError('Unable to configure handler '
ValueError: Unable to configure handler 'file'

It tried to run also mercury run --verbose IP:PORT and output is similar

[pplonski]

Looks like process that is running server doesnt have permission to write logs to the directory. This might be connected to #384 I will need some time to make it configurable.

Have you tried to deploy with docker-compose?

Have you already developed the notebook with mercury and deployment is the last step?

[MrStrzelec]

PermissionError: [Errno 13] Permission denied: '/opt/jupyterhub/lib/python3.10/site-packages/mercury/django-errors.log'

I've just changed permissions to this file for everyone and now it's works. Server running correctly and worker working correctly. W/O docker or huggingface. For some reasons Mercury showing that he cannot find this file, while JupyterNotebook working correctly. U know why?

[pplonski]

Great that it is working!

Do you have data file in the same directory as notebook? It should read file if it is in the same directory.

Alternative might be to provide full path to the file.

[MrStrzelec]

Hey!

Sorry i had a day off yesterday.

Yep, data is in the same directory as a notebook. I also tried with full path and also didn't worked.

Any other idea?
Thanks a Ton :)

[pplonski]

This is strange, could you provide full error message? Maybe try to print out the working directory before loading files, just to check the path. Try to add code:

import os
print(os.getcwd())

[MrStrzelec]

I think worker was not working at all, i've checked logs and found this:

[2023-11-02 12:33:20,930: INFO/MainProcess] Task apps.ws.tasks.task_start_websocket_worker[1cfbef57-5ef1-4ea6-b307-f1380f3a20aa] received
[2023-11-02 12:33:20,976: INFO/MainProcess] Task apps.ws.tasks.task_start_websocket_worker[1cfbef57-5ef1-4ea6-b307-f1380f3a20aa] succeeded in 0.04001394798979163s: None
NB 2023-11-02 12:33:22,661 Exception when notebook load, quit
Traceback (most recent call last):
  File "/opt/jupyterhub/lib/python3.10/site-packages/mercury/apps/../apps/nbworker/rest.py", line 36, in load_notebook
    raise Exception("Cant load notebook")
Exception: Cant load notebook

I think worker wouldn't find this file due to having problem with loading notebook.
I've check everything twice and didn't find anything.

[pplonski]

Do you have notebook file in the same directory in which you start worker?