cylc / cylc-uiserver

A Jupyter Server extension that serves the cylc-ui web application for monitoring and controlling Cylc workflows.

Cylc (pronounced silk) is a general purpose workflow engine that also manages cycling systems very efficiently. It is used in production weather, climate, and environmental forecasting on HPC, but is not specialized to those domains.

Quick Start

Installation | Documentation

# install cylc
conda install cylc-flow

# extract an example to run
cylc get-resources examples/integer-cycling

# install and run it
cylc vip integer-cycling  # vip = validate, install and play

# watch it run
cylc tui integer-cycling

The Cylc Ecosystem

• cylc-flow - The core Cylc Scheduler for defining and running workflows.
• cylc-uiserver - The web-based Cylc graphical user interface for monitoring and controlling workflows.
• cylc-rose - Provides integration with Rose.

Migrating From Cylc 7

Migration Guide | Migration Support

Cylc 8 can run most Cylc 7 workflows in compatibility mode with little to no changes, go through the migration guide for more details.

Quick summary of major changes:

• Python 2 -> 3.
• Internal communications converted from HTTPS to ZMQ (TCP).
• PyGTK GUIs replaced by:
  • Terminal user interface (TUI) included in cylc-flow.
  • Web user interface provided by the cylc-uiserver package.
• A new scheduling algorithm with support for branched workflows.
• Command line changes:
  • cylc run <id> -> cylc play <id>
  • cylc restart <id> -> cylc play <id>
  • rose suite-run -> cylc install; cylc play <id>
• The core package containing Cylc scheduler program has been renamed cylc-flow.
• Cylc review has been removed, the Cylc 7 version remains Cylc 8 compatible.

Citations & Publications

Copyright and Terms of Use

Copyright (C) 2008-2024 NIWA & British Crown (Met Office) & Contributors.

Cylc is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

Cylc is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with Cylc. If not, see GNU licenses.

Contributing

Contributions welcome:

• Read the contributing page.
• Development setup instructions are in the developer docs.
• Involved change proposals can be found in the admin pages.
• Touch base in the developers chat.

This repository contains some code that was generated by GitHub Copilot.

cylc-uiserver's Issues

Automatically launch server

I am troubleshooting #44 and find that I keep running jupyterhub, then opening browser, refreshing, clicking Launch Server button, then I can debug the cylc_singleuser.py code.

It would be handy - even if just for developers - to be able to launch the server automatically when the hub initializes.

Create an endpoint for user profile

Should return the Tornado's get_current_user() response as JSON.

Part of cylc/cylc-ui#33

Release to PYPI

Created a PR for the README #62 , and with a new readme, and once @dwsutherland 's work has been merged, it would be great to get an initial release of this project in PYPI. This is a requirement for cylc/cylc-admin#27

Revisit jupyterhub_config location for Cylc UI

Developers may have different directory structures for their projects. Some checking out the code in a common directory like ~/code/, others using different parent directories (e.g. ~/javascript/, ~/python/), or even using multiple git working trees.

Cylc UI Server has a jupyterhub_config.py file used for development. That file needs a location of the static files, produced by Cylc UI. And at the moment it is expecting the developer to have both projects sharing a root directory.

We need to find another solution that accommodates these possible developer environment configurations.

Update setup.py after project rename

Just cloned while working on #15 and setup.py still mentions cylc-jupyterhub.

Make handlers for static content support routes such as `/app.js`

The current regular expressions in the Tornado handlers miss routes to assets like /app.js. This file is created as /js/app.randomvalue.js when we build the application for production.

But after #36 , we have a development mode. In this case, the produced file does not have a /js prefix. So it ends up simply returning a 404.

Better integration with the cylc-web UI

At the moment we have a directory static with the copy of the files produced in cylc-web, after running npm run build.

This is not productive, and in order to finish cylc/cylc-ui#33, we need more integration with the backend.

Might be possible to create a new npm run watch task in cylc-web, that outputs a simpler version for development as we change, in a fixed location (e.g. dist/app.js).

Then we remove the static folder, and parameterize the spawner to support passing the location of the static files too, in the same way it defines the port number to listen to.

Upgrade to Tornado 6

https://www.tornadoweb.org/en/stable/releases/v6.0.0.html

Might be a good idea to freeze pip and use specific versions in the dependencies list too.

Integrate Tornado with Cylc Web prototype

We will need to handle 404 pages (see comment about vue.js history mode).

And then implement a catch all in Tornado that will delegate it to the index.html file.

reduce the impact of FS scans

At present we are regularly performing file system (FS) scans to detect new suites. This functionality is required to pick up on suites started from the CLI.

These FS scans are quite regular and can be slow as they involve walking the directory tree.

• Is there a more light-weight alternative to a full FS scan?
  • To pick up new suites we only need to listen for directory creation.
  • However, due to hierarchical suites these directories could be created in different places.
  • Note a solution cannot be Linux-bound (though adding something like fswatch to the dependency base shouldn't be an issue)
• Can we do some kind of incremental scan to avoid re-walking trodden ground?
  • Perhaps an arrangement of "quick scans" and "full scans".

Note: For suites started via the UI we can do things in a more responsive manner (cylc/cylc-flow#3451).

Project structure

We are a very flat structure right now. Which works fine so far, but we are exporting two modules right now (cylc_singleuser, and handlers). If we add more files, we may end up exporting more modules.

And it is dangerous, as in the case we decide to have a file called job... that could be exported as job, and be mistakenly used when users did a import job.

Maybe we should use cylc_uiserver as top level package? Then organize our UI Server code within that folder? Me really bad choosing names, so CC'ing @cylc/core 🦇

Add GraphQL support

As per discussion in Riot, we are now ready to start porting the work from @dwsutherland with GraphQL + Flask to here, with GraphQL + Tornado.

As both the Flask and the Tornado implementations are provided as extensions of the Python Graphene project, it should be fairly easy to port most of the code.

Flask has an approach of a slim core, with extensions installed via pip for things like CORS, HTTP Basic, etc. Whereas Tornado includes a slim core too, but without official modules for things like CORS or some authentication mechanisms.

So this may be a bit of a challenge.

See:

• https://pypi.org/project/graphene-tornado/
• https://github.com/graphql-python/graphene-tornado

Document how to customize the Hub

Low priority, but would be nice to document

• How to change the logo / text wherever possible #55
• Test and document how to use the extra form for spawners
• Display security messages or warnings to users

Add license

No license added yet.

@cylc/core GPL v3 like Cylc I assume?

Make pytest-runner a dependency only for tests

Describe exactly what you would like to see in an upcoming release

Our Conda recipe for cylc-uiserver has pytest-runner as a requirement, even though we are not running tests there. We need to copy the approach from metomi-isodatetime here.

Additional context

Pull requests welcome!

change name of this project to cylc-ui-server

(what it says)

Add Travis-CI build

As per title, we are not building the project. At least installing it via tox or directly via pip should be a good initial build test.

Use Python 3.7

No need to stay behind Cylc 8, I believe we can start using 3.7 right now, as the 1.0 release is still a little far away.

Add unit tests

We have basically a Tornado application, which is not very simple to be tested. It still does not impede us of writing simple unit tests for the peripheral classes/objects. For example, MyApplication could be tested, at least its initial state.

zmq "too many open files" (sockets?)

After leaving the end-to-end system running for a while (with a small test suite at the back end) I get this error repeatedly:

ERROR:tornado.application:Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x7fd3aa9857b8>>, <Task finished coro=<WorkflowsManager.gather_workflows() done, defined at /home/oliverh/cylc/cylc-uiserver/workflows_mgr.py:83> exception=ZMQError('Too many open files')>)
Traceback (most recent call last):
  File "/home/oliverh/cylc/cylc-uiserver/venv/lib/python3.7/site-packages/tornado/ioloop.py", line 743, in _run_callback
    ret = callback()
  File "/home/oliverh/cylc/cylc-uiserver/venv/lib/python3.7/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
    future.result()
  File "/home/oliverh/cylc/cylc-uiserver/workflows_mgr.py", line 94, in gather_workflows
    items = await asyncio.gather(*gathers)
  File "/home/oliverh/cylc/cylc-uiserver/workflows_mgr.py", line 69, in est_workflow
    client = SuiteRuntimeClient(reg, host=host, port=port, timeout=timeout)
  File "/home/oliverh/cylc/cylc-uiserver/venv/lib/python3.7/site-packages/cylc/flow/network/client.py", line 254, in __init__
    timeout_handler=partial(self._timeout_handler, suite, host, port)
  File "/home/oliverh/cylc/cylc-uiserver/venv/lib/python3.7/site-packages/cylc/flow/network/client.py", line 104, in __init__
    self.socket = CONTEXT.socket(zmq.REQ)
  File "/home/oliverh/cylc/cylc-uiserver/venv/lib/python3.7/site-packages/zmq/sugar/context.py", line 146, in socket
    s = self._socket_class(self, socket_type, **kwargs)
  File "/home/oliverh/cylc/cylc-uiserver/venv/lib/python3.7/site-packages/zmq/_future.py", line 134, in __init__
    super(_AsyncSocket, self).__init__(context, socket_type, **kwargs)
  File "/home/oliverh/cylc/cylc-uiserver/venv/lib/python3.7/site-packages/zmq/sugar/socket.py", line 59, in __init__
    super(Socket, self).__init__(*a, **kw)
  File "zmq/backend/cython/socket.pyx", line 328, in zmq.backend.cython.socket.Socket.__init__
zmq.error.ZMQError: Too many open files

and separately:

ERROR:asyncio:Exception in callback BaseAsyncIOLoop._handle_events(9, 1)
handle: <Handle BaseAsyncIOLoop._handle_events(9, 1)>
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/asyncio/events.py", line 88, in _run
  File "/home/oliverh/cylc/cylc-uiserver/venv/lib/python3.7/site-packages/tornado/platform/asyncio.py", line 138, in _handle_events
  File "/home/oliverh/cylc/cylc-uiserver/venv/lib/python3.7/site-packages/tornado/netutil.py", line 260, in accept_handler
  File "/usr/local/lib/python3.7/socket.py", line 212, in accept
OSError: [Errno 24] Too many open files

OSError: [Errno 24] Too many open files after running the UI Server for a long time

I was running a Cylc suite from the old dev examples (cylc run --no-detach --verbose --debug --hold complex), with cylc-uiserver (i.e.jupyterhub + spawned cylc_singleuser) since around 9:30 AM.

Then started looking at Vue.JS code, had lunch, came back, to find my console full of:

ERROR:asyncio:Exception in callback BaseAsyncIOLoop._handle_events(11, 1)
handle: <Handle BaseAsyncIOLoop._handle_events(11, 1)>
Traceback (most recent call last):
  File "/home/kinow/Development/python/anaconda3/lib/python3.7/asyncio/events.py", line 88, in _run
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/tornado/platform/asyncio.py", line 138, in _handle_events
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/tornado/netutil.py", line 260, in accept_handler
  File "/home/kinow/Development/python/anaconda3/lib/python3.7/socket.py", line 212, in accept
OSError: [Errno 24] Too many open files

It appears to be known issue, reported against jupyter/notebook: jupyter/notebook#3748, with a sibling issue in pyzmq: zeromq/pyzmq#1170

Investigate if it is possible to control the redirect URL in JupyterHub

In JupyterHub, when you request a URL like http://localhost:8000/user/kinow/#/workflows/five, but the hub notices that your UI Server is down, it will redirect you to a URL with the button to "Launch Server".

This button will initiate the process to start the Cylc UI Server, and upon success, will redirect it to a URL. It is possible to see that the URL to be redirected is passed as a URL query parameter, e.g. http://localhost:8000/hub/spawn/kinow?next=%2Fhub%2Fuser%2Fkinow%2F

Which decodes to next=/hub/user/kinow/, and then gets redirected later to /user/kinow/ internally by proxy+hub.

It can be annoying to users - or at least it is annoying to devs working on views in Vue.js - to be on a page, re-launch the hub, just to be taken back to the initial page... having to go through all the work to find which workflow you were running, and find your URL again (the browser history for my case doesn't always contain the previous URL... maybe because of our Vue.js app's route configured to use hash mode).

It would be nice if instead it redirected to your previous URL.

Cheers
Bruno

Review GitHub project description and topics

I created the project some time ago under my account, and added topics. @hjoliver will probably want to review this and choose a good description and topics.

Sorry, should have brought it up in the last meeting but forgot about it. And no hurry about it, as it will probably take a while until we get our first users.

Investigate how to prevent the Tornado app of showing exception traceback info to users

Before going to production, we need to make sure users have a way to configure Cylc UI Server so that traceback are not displayed in the browser to users, as it may leak sensitive information.

• https://www.owasp.org/index.php/Exception_Handling#Displaying_Errors
• https://www.owasp.org/index.php/Error_Handling#Generic_error_messages

At the moment the user gets the following if the HTML assets are not found:

Configure logging for the Cylc UI Server app

I added some logging in main.py some time ago, while working on the initial prototype. But there was nothing important in those logging messages.

While looking at #65 , I wondered where exceptions were logged. Looks like the GraphQL client code is not logging exceptions (neither is the rest of the application to be honest).

We need to look at how JupyterHub and its spawned applications handle logging, and use it properly in Cylc UI Server. A quick look at the JupyterHub code, looks like they use some code from Tornado logging.

But I am not sure if that's the only way they are handling logging.

Cheers
Bruno

Define for how long cookies are persistent

Before I forget, I am looking at the Cylc UI Server now (because I spent some of the most frustrating hours recently troubleshooting a Webpack issue), and found a comment I left while writing the old cylc-singleuser prototype.

cylc-uiserver/cylc/uiserver/main.py

Lines 113 to 115 in ca7e65e

	# FIXME: decide (and document) whether cookies will be permanent
	# after server restart.
	cookie_secret="cylc-secret-cookie"

From memory, what I recall is that this value could be fixed or randomly generated. If a static value is used, then the cookies survive even after a server restart.

If dynamic (e.g. some random value generated on startup), then the cookie exist for as long as the application is running. It is a parameter that changes a bit security in the UI Server, but I am not sure how important it is.

But just in case, better have this logged in an issue so we don't forget before releasing to production.

Bruno

Investigate starting multiple jupyterhub instances (port binding)

Just occurred to me that when we start Cylc UI Server, we run jupyterhub. That's necessary as the Hub is actually the one responsible to spawn the UI Server.

But running jupyterhub twice in the same machine results in

    Traceback (most recent call last):
      File "/home/kinow/Development/python/workspace/jupyterhub/jupyterhub/app.py", line 2278, in launch_instance_async
        await self.start()
      File "/home/kinow/Development/python/workspace/jupyterhub/jupyterhub/app.py", line 2171, in start
        self.http_server.listen(port, address=ip)
      File "/home/kinow/Development/python/anaconda3/lib/python3.7/site-packages/tornado/tcpserver.py", line 143, in listen
        sockets = bind_sockets(port, address=address)
      File "/home/kinow/Development/python/anaconda3/lib/python3.7/site-packages/tornado/netutil.py", line 168, in bind_sockets
        sock.bind(sockaddr)
    OSError: [Errno 98] Address already in use

As the default port is already in use. We can change the Hub's port with jupyterhub --port 7777, but it still fails as the proxy uses the 8081 port by default in both.

This issue is to investigate and document how we will initialize the UI Server. I think without having to pass a parameter would be nicer.

Test JupyterHub batchspawner with Cylc 8 prototype

Documentation for the spawner: https://github.com/jupyterhub/batchspawner

We can use these Docker containers to test Cylc + PBS: https://github.com/kinow/cylc-docker/tree/master/pbs.

But will have to do a few modifications.

• Make sure both containers (pbs and cylc) have Python 3
• Include jupyterhub and batchspawner installed via pip in cylc container
• Create configuration file to load cylc-singleuser in the pbs node
• Have cylc-uiserver installed via pip install -e . in pbs, so that cylc-singleuser is available
• Have cylc-web assets in pbs node at /opt/cylc-web/dist/.
  • The cylc-singleserver is our UI server, running from the PBS node. That will start the Tornado server and serve requests for both HTTP GET/POST requests for REST API (and GraphQL in the future) and also the HTTP requests for resources such as images, CSS, HTML, etc. A copy of the files must reside in this node as well.

Good to have these for the test too:

• No selinux in the PBS node
• No firewall in the PBS node

Test LDAPAuthenticator

Low priority, but having some spare time this week, so decided to give it a try. Unfortunately I couldn't get it to work.

Used the company internal AD, but there are a few different ways to connect, and it's changing with Microsoft cloud services. Will either create an internal request to get proper access to test this, or look for a mock server (container, cloud/free by MS, etc).

So just creating this ticket here in case anyone has spare time to test it too (and easy access to an AD with test credentials).

Protect GraphQL endpoints

I am working on a Cylc UI issue (cylc/cylc-ui#130) and realized the GraphQL endpoints are open to the whole world for now.

We simply never decorated the GraphQL tornado handlers with @authenticated, which would tell the hub to make sure the user is authenticated before it can be accessed.

We must do it before the production release.

NB: I worried about it as this could break the devServer proxy in NPM, but I guess it could also cause issues to users of shell scripts, Python scripts, browser extensions, etc (unless authenticated in the hub first)

404 when going to "My Server" in the hub

With JupyterHub 1.0, it appears some change broke the link that used to work for "My Server" in the hub.

Traceback (most recent call last):
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/tornado/web.py", line 1676, in _execute
    result = self.prepare()
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/tornado/web.py", line 2431, in prepare
    raise HTTPError(self._status_code)
tornado.web.HTTPError: HTTP 404: Not Found

configurable-http-proxy start-up and /etc/hosts

On my Centos host:

$ hostname
drugs-and-money

$ cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

Jupyterhub hangs trying to start configurable-http-proxy, and connecting to localhost:8000 gives 404:

(venv) [oliverh@drugs-and-money cylc-uiserver]$ jupyterhub
[I 2019-06-27 16:49:28.516 JupyterHub app:2120] Using Authenticator: jupyterhub.auth.PAMAuthenticator-1.0.0
[I 2019-06-27 16:49:28.516 JupyterHub app:2120] Using Spawner: jupyterhub.spawner.LocalProcessSpawner-1.0.0
[I 2019-06-27 16:49:28.517 JupyterHub app:1257] Loading cookie_secret from /home/oliverh/cylc/cylc-uiserver/jupyterhub_cookie_secret
[I 2019-06-27 16:49:28.531 JupyterHub proxy:460] Generating new CONFIGPROXY_AUTH_TOKEN
[W 2019-06-27 16:49:28.532 JupyterHub app:1532] No admin users, admin interface will be unavailable.
[W 2019-06-27 16:49:28.532 JupyterHub app:1534] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2019-06-27 16:49:28.532 JupyterHub app:1563] Not using whitelist. Any authenticated user will be allowed.
[I 2019-06-27 16:49:28.560 JupyterHub app:2337] Hub API listening on http://127.0.0.1:8081/hub/
[W 2019-06-27 16:49:28.560 JupyterHub proxy:516] Found proxy pid file: /home/oliverh/cylc/cylc-uiserver/jupyterhub-proxy.pid
[W 2019-06-27 16:49:28.560 JupyterHub proxy:528] Proxy no longer running at pid=17430
[W 2019-06-27 16:49:28.561 JupyterHub proxy:642] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 2019-06-27 16:49:28.561 JupyterHub proxy:645] Starting proxy @ http://:8000
16:49:28.698 [ConfigProxy] info: Proxying http://*:8000 to (no default)
16:49:28.700 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes

16:49:40.873 [ConfigProxy] error: 404 GET / 

The Fix (not understood): add one line to /etc/hosts:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
127.0.0.1   drugs-and-money

We should understand and document this.

Settings for hub IP and Port must be dynamic

As in the case of a Notebook, the UI server that we have now assumes the hub is always running at 0.0.0.0:8000, but that may have different values in different sites.

The spawner has the hub information. So we just need to send that information to the UI server, store it, and re-use later.

List other UI servers running

We will have one UI server per user. And each UI server will look after one or more workflows.

We want to allow users to access each other's UI server and see their running workflows - subjected to proper authorization, of course.

We need to define what is the workflow going to look like.

• Is user A going to configure his/her workflow to authorize that user B can access it?
• Is user A going to share the link of his UI server with user B, or will user B have a way to automatically see what suites s/he has access to?

request for changes to graph data json schema

To allow the graph to function correctly certain data properties are required, here is an example json schema. Edge source and target are node ids. Parent is a node id, this allows the parent node to collapse child nodes. Runpercent is to allow display of runtime on a running node.

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "properties": {
    "nodes": {
      "type": "array",
      "items": [
        {
          "type": "object",
          "properties": {
            "data": {
              "type": "object",
              "properties": {
                "id": {
                  "type": "string"
                },
                "parent": {
                  "type": "string"
                },
                "label": {
                  "type": "string"
                },
                "state": {
                  "type": "string"
                },
                "runpercent": {
                  "type": "integer"
                }
              },
              "required": [
                "id",
                "parent",
                "label",
                "state",
                "runpercent"
              ]
            }
          },
          "required": [
            "data"
          ]
        }
      ]
    },
    "edges": {
      "type": "array",
      "items": [
        {
          "type": "object",
          "properties": {
            "data": {
              "type": "object",
              "properties": {
                "id": {
                  "type": "string"
                },
                "source": {
                  "type": "string"
                },
                "target": {
                  "type": "string"
                },
                "label": {
                  "type": "string"
                }
              },
              "required": [
                "id",
                "source",
                "target",
                "label"
              ]
            }
          },
          "required": [
            "data"
          ]
        }
      ]
    }
  },
  "required": [
    "nodes",
    "edges"
  ]
}

an example of the json output

{
    "nodes": [{
            "data": {
                "id": "338c432b-9f96-420c-908f-84d06539cb1d",
                "parent": "0696f731-4522-4f01-9bf1-b5d035a117a2",
                "label": "long_start_12.20150219T1200Z",
                "state": "succeeded",
                "runpercent": 0
            }
        }
    ],
    "edges": [{
        "data": {
            "id": "1095",
            "source": "338c432b-9f96-420c-908f-84d06539cb1d",
            "target": "946be865-95c9-4cf1-89f5-a06400f739b9",
            "label": "ensemble_thunderbirds_are_go_forecast_002.20150219T1200Z (interaction) ensemble_sync_happy.20150219T1200Z"
        }
    }]
}

Fix server header version warning

The Hub seems to expect a version reported by the spawned application. If no header, it displays the following warning in the logs.

[W 2019-02-22 11:26:33.705 JupyterHub _version:30] Single-user server has no version header, which means it is likely < 0.8. Expected 0.9.4

This can be easily fixed, apparently, by simply returning the JupyterHub version. IOW, if you have JupyterHub 0.9.4, and return 0.9.4, the warning is gone.

If you return something like 1.0, then the application is unavailable in the hub, and you will have an error in the logs.

[W 2019-02-22 11:34:37.321 JupyterHub web:1667] 500 GET /hub/user/kinow/?redirects=2 (::1): Redirect loop detected. Notebook has jupyterhub version 1.0, but the Hub expects 0.9.4. Try installing jupyterhub==0.9.4 in the user environment if you continue to have problems.
[E 2019-02-22 11:34:37.369 JupyterHub log:150] {
      "X-Forwarded-Host": "localhost:8000",
      "X-Forwarded-Proto": "http",
      "X-Forwarded-Port": "8000",
      "X-Forwarded-For": "::1",
      "Cookie": "jupyterhub-hub-login=\"2|1:0|10:1550786007|20:jupyterhub-hub-login|44:MzZkNzljMmE5ZDUwNGFiNGEwMGJlNmQ0ZjJjOTNiNDI=|fa6ff62d3476c652147c10138317e3bb6e65f7ca59049d6f6310c6be144d20db\"; _xsrf=2|9c1bb992|e073c22bafb9cc9ce3d33fcd34dc3814|1550716134; jupyterhub-session-id=bb91133a002142d8a3268b927e0fe1c4",
      "Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8",
      "Accept-Encoding": "gzip, deflate, br",
      "Referer": "http://localhost:8000/hub/login",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
      "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36",
      "Upgrade-Insecure-Requests": "1",
      "Cache-Control": "no-cache",
      "Pragma": "no-cache",
      "Connection": "close",
      "Host": "localhost:8000"
    }
[E 2019-02-22 11:34:37.370 JupyterHub log:158] 500 GET /hub/user/kinow/?redirects=2 (kinow@::1) 546.59ms
^C
Interrupted

cylc.flow.exceptions.ClientError: could not read suite passphrase

Not sure if due to recent logging changes, now I am seeing the following exception:

[I 2019-09-16 16:44:26.306 JupyterHub log:174] 200 GET /hub/api/authorizations/token/[secret] ([email protected]) 26.79ms
2019-09-16 16:47:06,034 cylc.uiserver.data_mgr ERROR    could not read suite passphrase
Traceback (most recent call last):
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/cylc/flow/network/client.py", line 133, in async_request
    secret = self.secret()
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/cylc/flow/network/authentication.py", line 41, in get_secret
    suite, content=True
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/cylc/flow/suite_srv_files_mgr.py", line 311, in get_auth_item
    raise SuiteServiceFileError("Couldn't get %s" % item)
cylc.flow.exceptions.SuiteServiceFileError: Couldn't get passphrase

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kinow/Development/python/workspace/cylc-uiserver/cylc/uiserver/data_mgr.py", line 36, in get_workflow_data
    pb_msg = await client.async_request(method)
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/cylc/flow/network/client.py", line 135, in async_request
    raise ClientError('could not read suite passphrase')
cylc.flow.exceptions.ClientError: could not read suite passphrase
2019-09-16 16:47:11,037 cylc.uiserver.data_mgr ERROR    could not read suite passphrase
Traceback (most recent call last):
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/cylc/flow/network/client.py", line 133, in async_request
    secret = self.secret()
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/cylc/flow/network/authentication.py", line 41, in get_secret
    suite, content=True
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/cylc/flow/suite_srv_files_mgr.py", line 311, in get_auth_item
    raise SuiteServiceFileError("Couldn't get %s" % item)
cylc.flow.exceptions.SuiteServiceFileError: Couldn't get passphrase

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kinow/Development/python/workspace/cylc-uiserver/cylc/uiserver/data_mgr.py", line 36, in get_workflow_data
    pb_msg = await client.async_request(method)
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/cylc/flow/network/client.py", line 135, in async_request
    raise ClientError('could not read suite passphrase')
cylc.flow.exceptions.ClientError: could not read suite passphrase

Interestingly, I didn't get this at work today, only now at home. But perhaps I am using some old version. At home I have:

$ git log -n 1
commit affc0b8a6dab362cbcb42b1e9499ebcf5e38e6fd (HEAD -> master, upstream/master, upstream/HEAD, kinow/master)
Author: Bruno P. Kinoshita <[email protected]>
Date:   Tue Sep 10 13:32:30 2019 +1200

    Use dictConfig with JSON instead of fileConfig with INI

Which is the latest commit.

Send server errors to the UI

At the moment, when executing GraphQL queries, it is possible to happen exceptions in the backend that are not sent back to the UI.

This means that the user won't be aware of the issue unless a component/view behaves erratically.

See #65 for an example, where there is an exception in the UI Server, but the UI server is receiving an empty [] in the GraphQL query.

(I also need to double-check if it's not the UI who is try/catch'ing the exception in returning an empty array? Though I think it doesn't do that... will confirm it later)

Several "ERROR:asyncio:Future exception was never retrieved" errors in log

Hi,

I have been running a modified workflow five with the following code:

[meta]
    title = "Inter-cycle dependence + a cold-start task"
[cylc]
    UTC mode = True
[scheduling]
    #runahead limit = 120
    initial cycle point = 20130808T00
    final cycle point = 20190812T00
    [[dependencies]]
        [[[R1]]]
            graph = "prep => foo"
        [[[PT12H]]]
            graph = "foo[-PT12H] => foo => bar"
[runtime]
[[root]]
script = "sleep 2; echo 'done!'"

I've modified it just so it would take a bit longer to finish, and give me more time to work on Vue.js code for Cylc UI, with real data.

The workflow has been running since this morning (around 10AM I think?), and around 1 PM I noticed that the UI would have the workflow tree - in a new component - disappear completely.

Checking the logs, Cylc Flow appears to be OK, as well as Cylc UI. But Cylc UI Server has the logs flooded with:

ERROR:asyncio:Future exception was never retrieved
future: <Future finished exception=ZMQError('Operation cannot be accomplished in current state')>
Traceback (most recent call last):
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/zmq/_future.py", line 363, in _add_send_event
    r = send(msg, **nowait_kwargs)
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/zmq/sugar/socket.py", line 395, in send
    return super(Socket, self).send(data, flags=flags, copy=copy, track=track)
  File "zmq/backend/cython/socket.pyx", line 733, in zmq.backend.cython.socket.Socket.send
  File "zmq/backend/cython/socket.pyx", line 780, in zmq.backend.cython.socket.Socket.send
  File "zmq/backend/cython/socket.pyx", line 247, in zmq.backend.cython.socket._send_copy
  File "zmq/backend/cython/socket.pyx", line 242, in zmq.backend.cython.socket._send_copy
  File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc
zmq.error.ZMQError: Operation cannot be accomplished in current state

Haven't investigated any further as I'm still working on some JS issues with that component. I am using master on Cylc UI Server and on Cylc Flow. And a custom branch on Cylc UI (tree-component-1).

Cheers
Bruno

Remove unused import in test_uiserver

test_uiserver.py has an import for os, but doesn't seem to be using that module. Should be pretty straightforward. Just double-check that the os module is not referenced in that module, then remove and make sure tests pass.

FileNotFoundError including reference to old repo name

I am getting a traceback that seems to stem (at least partially) from a reference to the old repository name, cylc-web:

Traceback (most recent call last):
  File "/home/h06/sbarth/miniconda3/envs/py3env/lib/python3.7/site-packages/tornado/web.py", line 1697, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "/home/h06/sbarth/miniconda3/envs/py3env/lib/python3.7/site-packages/tornado/web.py", line 3174, in wrapper
    return method(self, *args, **kwargs)
  File "/net/home/h06/sbarth/cylc-uiserver/handlers.py", line 60, in get
    self.write(open(index).read())
FileNotFoundError: [Errno 2] No such file or directory: '/net/home/h06/sbarth/cylc-web/dist/index.html'

where by grepping for cylc-web I only see one hard-coded reference outside of the README.md file:

cylc-uiserver/jupyterhub_config.py

Lines 468 to 471 in ef1ccd4

	# Some spawners allow shell-style expansion here, allowing you to use
	# environment variables here. Most, including the default, do not. Consult the
	# documentation for your spawner to verify!
	c.Spawner.args = ['-s', '../cylc-web/dist/']

so the issue must logically relate to that? Given this highly-probable origin, I would put in a "quick fix" PR, but from a little investigation, I am not sure if the final directory component of the path also needs amending, as there is no such dist/ directory? There is an index.html file, as seems to be required given the traceback, under cylc-ui/public/ (i.e. in the UI repository), but from reading that file it does not seem appropriate contextually. So while that could be the right place to point, I am not sure.

I've read through a lot of the comments in the jupyterhub_config.py file, but having just started looking at the UI & UI Server Cylc 8 components in development, I am not familiar enough yet to gain much help from them (along with some related docs reading), notably the one above the line in question. Therefore I thought it best to raise this as an Issue.

Test Cylc UI Server with SSH Spawner

The current set up used by Python UI Server and JS Vue.js UI uses the LocalProcessSpawner.

We also tested the PBS Batch Spawner. But we have not concluded the SSH Spawner tests yet.

@MartinRyan has a work-in-progress SSH spawner (private repository), and had forked the uiserver too.

As the requirement here seems to be simply validating spawning the UI Server via SSH, I think an alternative could be the existing sshspawner by NERSC (National Energy Research Scientific Computing Center, US), which appears to be an implementation that has not been deprecated (the zonca one is not being maintained).

Martin's implementation used paramiko, which is an excellent tool (I had to use it some time ago, can't remember why, but remember it worked well). While NERSC's uses asyncssh, which also implements SSHv2, but tailored for Python 3.4. I think both are OK.

Other posts seem to suggest using the BatchSpawner as replacement for SSH spawning, but that's actually as a replacement for users who were triggering HPC batch schedulers like PBS. So not exactly what we are looking for.

Accept extra parameters available for the singleuser app without crashing

When the hub starts the singleuser app, it has a series of parameters that it may send, e.g. debug.

Right now, we are only supporting a few (port, static folder), and raising an issue if any other parameter is provided.

We must ensure it works even if we don't use the values (i.e. probaly use args/kwargs).

Add GraphQL subscriptions support via WebSocket

See #11 for previous discussion. The UI Server is a Tornado project, that inherits classes from JupyterHub Tornado project (for things like authentication).

In #11 and #13 it's getting an extra layer with Graphene, an open source library for Python that implements GraphQL.

However, GraphQL subscriptions require the base layer of Graphene for GraphQL, plus an extra library to handle the query subscription (which is basically a mechanism for sending to clients the output of a query whenever if changed), the normal way being via WebSockets.

WebSockets are supported by Tornado out of the box. What we are missing, is actually the extra layer of integration with Tornado. That is done via the Graphele graphql-ws project, which has one Pull Request to add support to tornado still pending.

Once that pull request is merged, we can add that library in our setup.py, and fix this issue. An alternative in case we need that before the pull request is merged, is simply fork it and use our own until it is merged.

detecting restarted suites

The UI Server needs to detect suite shutdown events and wipe the data store.

At present if you shutdown a suite, then, very quickly restart it, the previous state remains.

Pull requests welcome!
This is an Open Source project - please consider contributing a bug fix
yourself (please read CONTRIBUTING.md before starting any work though).

Define how the authorization in the UI server will work with other components

JupyterHub has authentication, and also a thin layer of authorization in its reverse proxy. But the Cylc UI server will need to redefine parts of its authentication.

This issue is to record discussions on authorization for the UI server, which will interface with JupyterHub and probably be used by Cylc Web as a bridge to communicate with cylc and JupyterHub.

See related discussion in cylc/cylc: cylc/cylc-flow#1901

See related discussion in cylc/web: cylc/cylc-ui#60

Fix Jinja2 module import warning

From #22

BTW there's also a conflict in their Jinja2 requirements and what's installed with jupyterhub, oddly enough. It's probably used with jupyterhub's tornado server, and graphene-tornado could only use it for the graphiql interface template rendering (I think))..
graphene-tornado have is Jinja2==2.9.6 but Jupyter has no restrictions... (maybe back compat reasons?)

Anyways, this causes a failure on start-up in the package checker:

pkg_resources.DistributionNotFound: The 'Jinja2==2.9.6' distribution was not found and is required by graphene-tornado

but I changed it to Jinja2>=2.9.6 in my virtualenv ... no issues with 2.10.2

I noticed a warning too, but the rest seemed to work @dwsutherland. Let's fix it 🎉

Review the need of jupyterhub as a requirement for spawned UI server process

When users install cylc-uiserver, there is a jupyterhub dependency that is used to run the jupyterhub command line application. This is called the Hub by some of us.

The Hub then spawns what we call UI Servers. These UI Servers are accessed behind a proxy (configurable-http-proxy, a NodeJS app, by default). The Hub has REST methods ready to read cookie values and to act as an OAuth2 server.

The easiest approach to authenticate the UI Server was by re-using the JupyterHub classes to authenticate the UI Server.

This implies having jupyterhub as a dependency not only in the Hub, but also on each spawned UI Server - well noted by @MartinRyan.

We need to review this, and assess whether it would be possible to authenticate against the Hub without bringing jupyterhub as a dependency in the spawned processes (which brings other transitive dependencies with it).

user of jupyterhub_config.py

We've only made minimal changes to jupyterhub_config.py, but added our own copyright header. It seems the default config file generated by jupyterhub --generate-config has changed quite a bit at the latest version (of jupyterhub). Would it be better for us not to include the config file in the repository, but tell users to generate one, and what in it to modify to spawn our UI Server?

Test setup.py and setuptools

pip install -e . works fine, but in a Centos6 container with Anaconda 3.7, after pip install . (not editable), got the following error:

bash-4.1# cylc-singleuser 
Traceback (most recent call last):
  File "/opt/anaconda3/bin/cylc-singleuser", line 6, in <module>
    from cylc_singleuser import main
  File "/opt/anaconda3/lib/python3.7/site-packages/cylc_singleuser.py", line 11, in <module>
    from handlers import *
ModuleNotFoundError: No module named 'handlers'

The container is a bit of a monster as it's only a test for #15 . But reporting this issue just in case.