cylc / cylc-rose

A Cylc plugin providing support for the Rose rose-suite.conf file.

Cylc (pronounced silk) is a general purpose workflow engine that also manages cycling systems very efficiently. It is used in production weather, climate, and environmental forecasting on HPC, but is not specialized to those domains.

Quick Start

Installation | Documentation

# install cylc
conda install cylc-flow

# extract an example to run
cylc get-resources examples/integer-cycling

# install and run it
cylc vip integer-cycling  # vip = validate, install and play

# watch it run
cylc tui integer-cycling

The Cylc Ecosystem

• cylc-flow - The core Cylc Scheduler for defining and running workflows.
• cylc-uiserver - The web-based Cylc graphical user interface for monitoring and controlling workflows.
• cylc-rose - Provides integration with Rose.

Migrating From Cylc 7

Migration Guide | Migration Support

Cylc 8 can run most Cylc 7 workflows in compatibility mode with little to no changes, go through the migration guide for more details.

Quick summary of major changes:

• Python 2 -> 3.
• Internal communications converted from HTTPS to ZMQ (TCP).
• PyGTK GUIs replaced by:
  • Terminal user interface (TUI) included in cylc-flow.
  • Web user interface provided by the cylc-uiserver package.
• A new scheduling algorithm with support for branched workflows.
• Command line changes:
  • cylc run <id> -> cylc play <id>
  • cylc restart <id> -> cylc play <id>
  • rose suite-run -> cylc install; cylc play <id>
• The core package containing Cylc scheduler program has been renamed cylc-flow.
• Cylc review has been removed, the Cylc 7 version remains Cylc 8 compatible.

Citations & Publications

Copyright and Terms of Use

Copyright (C) 2008-2024 NIWA & British Crown (Met Office) & Contributors.

Cylc is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

Cylc is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with Cylc. If not, see GNU licenses.

Contributing

Contributions welcome:

• Read the contributing page.
• Development setup instructions are in the developer docs.
• Involved change proposals can be found in the admin pages.
• Touch base in the developers chat.

This repository contains some code that was generated by GitHub Copilot.

cylc-rose's Issues

Fail if rose options used in non-rose workflow

Issue spotted by @dpmatthews

Describe the bug
If I use -D or -S or -O without a rose-suite.conf file present these options are silently ignored.

*Expected behavior
Fail if option is ignored?

Additional context
This is probably a simple check in cylc/rose/entry_points.py

Pull requests welcome!
This is an Open Source project - please consider contributing a bug fix
yourself (please read CONTRIBUTING.md before starting any work though).

stem: provide validate functionality

Rose Stem sets some Jinja2 variables on install.

These variables are not present when trying to validate the source which makes workflow development awkward. I don't think we can "detect" rose-stem workflows because what makes them rose-stem workflows is being installed with rose-stem. Consequently I don't think we can make cylc validate play nicely by default.

We should try to do something to make stem workflow development nicer.

Pull requests welcome!

Install rose-suite.conf in cylc rose

Describe exactly what you would like to see in an upcoming release

Cylc-rose and not Cylc-flow should install the rose-suite.conf file from the source directory rather than the work directory. This will allow more complex behavior from the plugin in future, such as diffing the old and new rose-suite.conf

Additional context

You will need to add an exclusion in cylc-flow/cylc/flow/suite_files.py as a companion to changes to cylc-rose.

Pull requests welcome!

Warn if `root dir` set

The root dir setting of rose-suite.conf was made obsolete by cylc-flow PR #3688.

If it is set cylc-rose should log a warning and do nothing.

@oliver-sanders in cylc/cylc-flow#3819, does "do nothing" mean "fail with an error", or "ignore value with warning"?

Pull requests welcome!

Rose Stem Questions

[draft]

Brain dump of thoughts about rose stem.

Boilerplate logic in StemRunner.__init__:

Are there ever actually cases where the kwargs are not the default kwargs? If not, why bother with all this logic?

jinja2:suite.rc v. template variables:

SUITE_RC_PREFIX = `jinja2:suite.rc. Should it? Perhaps it should depend on Cylc version?

Cases not covered by functional tests

• self.opts.defines not empty - does anyone ever do rose stem --source=/foo --define-suites 'GREET="Hello"'

Can we check how many external function calls there are and try to cache their results?

Can value in _get_fcm_loc_layout_info ever be false?

As far as I can tell line is always in the form key:\svalue or key:\s. If this is the case value will always have at least a single space and :. be true.

     for line in fcm_loc_layout_output:
            key, value = line.split(":", 1)

            if key and value:
                ret[key] = value.strip()

@dpmatthews - is this reliable behavior from fcm loc-layout?

Why not:

     for line in fcm_loc_layout_output:
            key, value = line.split(":", 1)

-           if key and value:
-               ret[key] = value.strip()
+           ret[key] = value.strip()

Pull requests welcome!
This is an Open Source project - please consider contributing a bug fix
yourself (please read CONTRIBUTING.md before starting any work though).

consider more advanced template variable evaluation

Rose2019 dumps template variables directly into the suite.rc file which allows users to do things like this:

[jinja2:suite.rc]
FOO=range(0, 10)

Rose2 uses ast.literal_eval which will raise an error for this example.

Should we permit non-literal template variables? This use case feels like an unintended side-effect of a historical implementation that we wouldn't allow if implementing from scratch, however, if it is sufficiently widely used there might be a case for preserving it.

It would be possible to swap out ast.literal_eval with a custom eval supporting restricted syntax. There is an example of this in the Cylc codebase:

https://github.com/cylc/cylc-flow/blob/e2d361ec59e5b687403cab5695ca5099d8cd1b76/cylc/flow/host_select.py#L339

Pre configure not handling `-S` variables properly.

Describe the bug

-S 'Variable="something"' doesn't get passed to the rose config loader in the pre_configure entry point, so all Cylc commands which rely on that entry point will not load it into the rose configuration.

This has not affected the cylc install workflow; cylc play workflow recipe because the contents of -S are stored in opt/rose-suite-cylc-install.conf.

Release version(s) and/or repository branch(es) affected?

Steps to reproduce the bug

With a blank rose-suite.conf and flow.cylc thus:

[scheduling]
    initial cycle point = 1066
    [[dependencies]]
        R1 = foo

[runtime]
    [[foo]]
        script = echo {{ GREETING }}

run

> cylc validate . -S 'GREETING="Hello"'
Jinja2Error: 'GREETING' is undefined
Context lines:
    [[root]]
        script = """
            echo {{ GREETING }}	<-- UndefinedError

Expected behavior

The test above should validate.

Additional context

• A quick inspection of the code of rose/metomi/rose/config_tree.ConfigTreeLoader.load suggests that -D and -S must be merged before being passed to load(). (Double check this)
• You will probably want to change the handling of opts.rose_template_variables at around line 240 of cylc-rose/cylc/rose/utilities.py

Pull requests welcome!
This is an Open Source project - please consider contributing a bug fix
yourself (please read CONTRIBUTING.md before starting any work though).

Cylc Rose Support for Platform Groups and `platform=$subshell)`

Investigation required

• Does Cylc Rose cylc/rose/platform_utils.get_platform_from_task_config work for:
  • Tasks where [runtime][task]platform=group_name.
  • Tasks where [runtime][task]platform=$(subshell command)
• Is the typing correct for the get_platform_from_... functions? In the case of get_platform_from_task_job which interrogates the workflow db this function should never return None.

Additional context
Adding tests to tests/test_platform_utils.py would be a good place to start investigating.

Pull requests welcome!
This is an Open Source project - please consider contributing a bug fix
yourself (please read CONTRIBUTING.md before starting any work though).

track installed files

Rose fileinstall adds things into the run directory. If these things are not placed in work/ or share/ then they will be overwritten and subsequently reinstalled by cylc reinstall. This is ok but inefficient - #152.

Cylc-Rose should maintain a list of all of the stuff it has installed and provide that list to cylc reinstall where it can be merged in with the contents of the .cylcignore file (anything listed in cylcignore will be excluded from reinstall).

Pull requests welcome!

Tidy Log Directory

Describe the bug

Cylc Rose currently leaves logs in ~/cylc-run/workflow/log/conf. This should become config to be consistent with cylc/cylc-flow#4802

example

~/cylc-run/temp1/run16/log/
-|-- conf
+|-- config
 |   |-- 20220404T111205-rose-suite.conf
 `-- install
     |-- 20220404T101204Z-install.log

coverage: get coverage up

We should be able to hit 100% coverage for this project.

Pull requests welcome!

skip all plugins if the `rose-suite.conf` file is not present

At the moment even if there isn't a rose-suite.conf file in a flow the cylc-rose plugin still kicks in and creates one in the run dir along with the cylc-install opt conf and log files.

don't set teplate vars when not required

Providing [template variables] when no templating shebang is set in Cylc causes traceback by design.

We could turn this traceback into a warning (cylc/cylc-flow#4337), however, this warning will show up every time the Cylc config is read making it look to the user like they have done something wrong when they haven't.

Although a [template variables] section is otherwise harmless, perhaps the simplest solution is to turn it off when templating is not required.

Suggested logic:

is_suiterc = ...   # True if suite.rc False if flow.cylc
templating_requested = ...  # what is set in the {suite.rc,flow.cylc}
templating_section = ...  # what is set in the rose-suite.conf

if templating_section and templating_requested:
    # check the two match or raise and error
    if templating_section != templating_requested:
        raise Exception()
elif not templating_section and templating_requested:
    # use the requested templating engine
    templating = templating_requested
elif templating_section and not templating_requested and is_suiterc:
    # back-compat to Rose 2019 - auto-set the templating to match the [*:suite.rc] section
    templating = templating_section
elif not templating_section and not templating_requested:
    templating = None
else:
    raise Exception()

Would involve reverting the temporary changes made in #72

actions: check contributor list

Automatically test git shortlog against the list of contributors in order to ensure they match.

Required functionality implemented in cylc-release-actions, see the cylc-flow PR: cylc/cylc-flow#4205

investigate failing tests

Investigate test failures on master.

failing tests: tests/functional/test_pre_configure.py

example: https://github.com/cylc/cylc-rose/runs/2217786184?check_suite_focus=true

install incrementally

Because cylc install uses rsync --delete any files installed by metomi-rose file install (i.e. [file:...]) are deleted by cylc install, then re-created by cylc-rose via the post-install hook.

This is functionally ok but really inefficient. We should get fileinstall working in an incremental way (as it used to in Rose 2019).

This would involve getting cylc-rose to send back a list of the files it manages to cylc install before installation (i.e via a pre-install hook?) which cylc install would then add to .cylcignore (but without modifying the copy in the source dir please!).

Here's an example of how this could look:

# psudopython
# ask the pre_install plugin for a list of excluded files
ret = pre_install(src_dir, run_dir)

# add these to the .cylcignore in a temporary file
with TemporaryFile as temp_file:
    temp_file.write(cylc_ignore.read())
    temp_file.write('\n' + '\n'.join(ret['excluded_paths']))

# pass this temporary file to rsync
install_command = ['rsync', '--exclude-file', temp_file]

If Rose does/could support a dry-mode then this could also close #149

Pull requests welcome!

Validate Rose Stem Suites

Describe exactly what you would like to see in an upcoming release

Cylc Rose allows cylc validate, cylc graph, &c &c to access rose-suite.conf prior to workflow installation.
Rose stem cannot be used to validate a rose-stem suite.

Either by adding a rose stem --validate option, or by adding a --stem argument to the options available in Cylc enable validation, graphing &c of rose stem suites.

Pull requests welcome!

check ROSE_ORIG_HOST uses FQDN interface correctly

We have seen test failures where the ROSE_ORIG_HOST came out in a different format than anticipated suggesting that the part of the code that sets ROSE_ORIG_HOST is not using the same interface as the code that tests ROSE_ORIG_HOST.

Investigate to ensure that both parts are using the appropriate interfaces.

E         - ROSE_ORIG_HOST=fv-az397-136.digkgshymtuujfwfpq2czj2vug.jx.internal.cloudapp.net
571
E         ?                             ------------------------------
572
E         + ROSE_ORIG_HOST=fv-az397-136.internal.cloudapp.net

Originally posted by @oliver-sanders in #104 (comment)

fix breakage caused by recent cylc-flow change

Fix usage of parse_worlkflow_arg (function has recently been replaced on cylc-flow:master)

Unquoted strings passed to CLI options `-S` and `-D` do not fail validation.

Describe the bug

(probably a corollory of #122 )

Unquoted strings passed to CLI options -S and do not fail validation in the same way as for -D.

Steps to reproduce the bug

# rose-suite.conf
[template variables]
TEST = test

$ cylc install
PluginError: Error in plugin cylc.pre_configure.rose
template variables=TEST=test: Invalid template variable: test
Must be a valid Python or Jinja2 literal (note strings "must be quoted").

However, if I fix the error in rose-suite.conf and then run cylc install -S 'TEST=hello'then the install succeeds (but play fails with the plugin error). Shouldn't values set via command line options be checked in the same way?

Expected behavior
Unquoted strings passed to variables in -D or -S should raise an error.

Pull requests welcome!
This is an Open Source project - please consider contributing a bug fix
yourself (please read CONTRIBUTING.md before starting any work though).

ignore ROSE_VERSION/CYLC_VERSION in rose-suite.conf

tldr

Exclude any ROSE_VERSION and CYLC_VERSION environment variables set in the rose-suite.conf from being exported into the environment, messing with templatvars or whatever.

context

In Rose2019 there was some level of support for the [env]ROSE_VERSION and [env]CYLC_VERSION variables.

If ROSE_VERSION was set and did not match rose version then rose exited with an error.

If CYLC_VERSION was set Rose exported it to cylc executable calls (to pick up the correct version via the wrapper script).

Now that cylc-rose is a plugin and Rose2/Cylc8 are installed in the same Python environment this system is meaningless and no longer functions.

However, many Rose2019 suites will have these variables defined, they have the potential to mess with the "true" *_VERSION variables and cause problems.

CLI option to clear previous rose install opts

Description

See rose cylc install proposal: Cylc install should be able to clear existing configs saved in opt/rose-suite-cylc-install.conf thus:

cylc reinstall --clear-rose-install-options

Additional context

This will require a PR in:

• in cylc-rose to add the required functionality.
• in cylc-flow to add the command line option.

Pull requests welcome!

Set debug level from Cylc

Cylc Rose should access the LOG level being used by Cylc and set the logging level for Rose functionality called based on that setting.
This might involve passing both -v and --debug from Cylc.

Pull requests welcome!

highly verbose logging

When I install a workflow with a blank rose-suite.conf file I get some confusing log messages at the warning level:

WARNING - Overriding ROSE_VERSION: 2.0b2 -> 2.0b2
WARNING - Overriding ROSE_SUITE_VARIABLES: {'ROSE_ORIG_HOST': 'myhost',
	'ROSE_VERSION': '2.0b2', 'ROSE_SITE': '', 'ROSE_SUITE_VARIABLES': {...}} -> {'ROSE_ORIG_HOST':
	'myhost', 'ROSE_VERSION': '2.0b2', 'ROSE_SITE': '', 'ROSE_SUITE_VARIABLES':
	{...}}
WARNING - Overriding ROSE_SITE:  -> 
WARNING - Overriding ROSE_ORIG_HOST: myhost ->
	myhost

This is confusing as the values appear to be the same before as afterwards and I've not set them anywhere.

Deprecation warning for `[<engine>:suite.rc]`

Describe the bug
Setting [jinja2:suite.rc] in rose-suite.conf does not emit a deprecation warning.

Steps to reproduce the bug

Unzip
rose-dep-warnings.tar.gz into `~/cylc-src.

cylc validate rose-dep-warnings
Valid for cylc-8.0rc2.dev

Expected behavior

cylc validate rose-dep-warnings
Warning: [jinja2:suite.rc] is deprecated and will be removed: use [template variables] instead.
Valid for cylc-8.0rc2.dev

Screenshots

Additional context

It may be tidier to add an additional checking function to ultilities.py which can be called from entry_points.py.

To close this issue should also check:

• [empy:suite.rc]
• Passing both [jinja2:suite.rc] and [empy:suite.rc] to -D should also result in a warning.

Pull requests welcome!
This is an Open Source project - please consider contributing a bug fix
yourself (please read CONTRIBUTING.md before starting any work though).

Use check=True in tests (follow-up)

Describe exactly what you would like to see in an upcoming release

Follow-up from #77

Additional context

#77 (comment)

Pull requests welcome!

Consider allowable config sections in `rose-suite.conf`

Currently cylc-rose will read [jinja2:suite.rc] and [empy:suite.rc] sections.

1. We should definitely allow a generic template vars section. What should we call this? (template_vars, templatevars, template vars)
2. Should consider allowing "jinja2:flow.cylc" and "empy:flow.cylc" as a sop to consistency? I'm actually not sure that it's really necessary, and might be confusing.

what happens if `[jinja2:suite.rc]` and `[template variables]` are both set.

Describe the bug
Investigate what happens if [jinja2:suite.rc] and [template variables] are both set.

Steps to reproduce the bug

Uncompress rose-dep-warnings.tar.gz or manually create this workflow in ~/cylc-src/rose-dep-warnings:

rose-suite.conf:

[jinja2:suite.rc]
GREETING="Hello"
[template variables]
GREETING="Mittyn Da"

flow.cylc:

#!jinja2
[scheduling]
    initial cycle point = 1066
    [[dependencies]]
        R1 = foo
[runtime]
    [[foo]]
        script = echo {{ GREETING }}

Then run

cylc graph rose-dep-warnings
cylc install rose-dep-warnings
cylc play rose-dep-warnings
cat ~/cylc-run/rose-dep-warnings/log/flow-files/flow.processed.cylc

Expected behavior
A warning should be printed to the effect that you shouldn't use both sections.

Pull requests welcome!
This is an Open Source project - please consider contributing a bug fix
yourself (please read CONTRIBUTING.md before starting any work though).

`cylc reinstall` no current event loop error

Describe the bug

Updating a previously installed rose suite with cylc reinstall fails with an error message There is no current event loop in thread 'MainThread'.

Release version(s) and/or repository branch(es) affected?

Conda environment contains

# packages in environment at /g/data/access/ngm/miniconda3/envs/cylc-8.0rc2:
#
# Name                    Version                   Build  Channel
cylc-flow                 8.0rc2             pyhaef67bd_0    conda-forge
cylc-flow-base            8.0rc2             pyhd8ed1ab_0    conda-forge
cylc-rose                 1.0.2              pyhd8ed1ab_0    conda-forge
cylc-uiserver             1.0.1              pyhd8ed1ab_3    conda-forge
cylc-uiserver-base        1.0.1              pyhd8ed1ab_3    conda-forge
metomi-rose               2.0rc2             pyhd8ed1ab_0    conda-forge
metomi-rose-base          2.0rc2             pyh1a96a4e_0    conda-forge

Steps to reproduce the bug

In a rose suite directory, e.g. ~/roses/u-cm866

cylc install
cylc reinstall u-cm866

Output is

PluginError: Error in plugin cylc.post_install.rose_opts
There is no current event loop in thread 'MainThread'.

The failing function appears to be https://github.com/cylc/cylc-rose/blob/master/cylc/rose/entry_points.py#L61

Expected behavior

Configuration files in ~/cylc-run/u-cm866/run3 are updated with any changes made in the rose suite directory

Screenshots

Additional context

Pull requests welcome!
This is an Open Source project - please consider contributing a bug fix
yourself (please read CONTRIBUTING.md before starting any work though).

Only clean up files on test success

Describe exactly what you would like to see in an upcoming release

This commit causes pytest not to clean up workflows unless tests have passed. All cases where files are touched and cleaned using yield should be examined to see if the same logic makes sense.

Pull requests welcome!

coverage: get codecov integration working

We are measuring/reporting coverage but we don't seem to get notified about it.

Pull requests welcome!

Use GH Actions release process

See cylc/cylc-flow#3937

stem: install workflows hierarchically

rose stem currently installs workflows "flat".

If you're running rose stem you're likely running it many times. This means lots of old runs kicking about in the cylc-run directory.

We should consider getting rose stem to derive hierarchical workflow names by default so that all these runs go into the same hierarchy. This would help people to keep their cylc-run directories tidy, nice for the UI due to the collapsible workflow list. It would also make it easier to manage stem runs as a block e.g. cylc stop '<prefix>/*', cylc clean '<prefix>/*.

Pull requests welcome!

Changing `[jinja2:suite.rc]` to `[template variables]` causes reinstall to fail

Describe the bug

Consider a placeholder workflow with rose-suite.conf containing a [jinja2:suite.rc] section. Updating this to [template variables] and reinstalling raises the following:

PluginError: Error in plugin cylc.post_install.rose_opts
MultipleTemplatingEnginesError: You should not define more than one templating section. You defined:
	jinja2:suite.rc; template variables

Steps to reproduce the bug

cd $(mktemp -d)
touch flow.cylc
echo "[jinja2:suite.rc]" > rose-suite.conf
cylc install -n tmp
echo "[template variables]" > rose-suite.conf
cylc reinstall tmp

Expected behavior
This error should be raised only if the resulting config has both sections.

Pull requests welcome!
This is an Open Source project - please consider contributing a bug fix
yourself (please read CONTRIBUTING.md before starting any work though).

Issue raised on behalf of @GAndrsn

Review Rose Stem Documentation

Review the help built into rose stem.

Pull requests welcome!

retire ROSE_SITE

Note: This is in no way related to the rose_stem variable of the same name used on many sites. This is implemented within rose_stem via a different mechanism.

See this Discourse post - https://cylc.discourse.group/t/rose-site-is-anyone-using-it-outside-of-rose-stem/296

This functionality was quietly introduced and never gained traction. It doesn't really make sense for it to be implemented in the Rose layer, suggest moving it into Cylc as CYLC_SITE for configuration in the global.cylc file.

Should be accompanied by:

• A doc change to https://cylc.github.io/doc/build/7.8.7/html/suite-design-guide/portable-suites.html#the-jinja2-site-variable
• A change to Rose to remove the Rose configuration and a doc change to remove the env var entry.

Pull requests welcome!

pytest: Simplify usage.

Describe the bug
At commit 1fca257d4d5 it was found necessary to run pytest separately for the doctests and tests folder.

Steps to reproduce the bug
run pytest --doctest-modules.

Expected behavior
Doctests from cylc/ and tests from tests/ should both be run.

Pull requests welcome!
This is an Open Source project - please consider contributing a bug fix
yourself (please read CONTRIBUTING.md before starting any work though).

Tidy functional testing

A number of functional tests use the pattern:

@pytest.fixture
def workflow_creation_fixture:
    ...
    yield {...info...}

@pytest.fixture
def run_some_cylc_program(workflow_creation_fixture):
    ...
    yield {...more info..., 'workflow_creation_fixture': workflow_creation_fixture}

This would be better if the second fixture yielded:

    yield {...more info..., **workflow_creation_fixture}

to avoid unnecessary nesting.

Pull requests welcome!

Incorrect pyuv and Jinja2 dependencies installed?

Describe the bug

Trying cylc-rose for the very first time. Hadn't cloned this repo before, nor used it with venv etc. When installing it for the first time in a blank venv, I noticed these two errors (doesn't stop pip from installing it).

ERROR: cylc-flow 8.0b1 has requirement aiofiles==0.5.*, but you'll have aiofiles 0.7.0 which is incompatible.
ERROR: cylc-flow 8.0b1 has requirement jinja2==2.11.*, but you'll have jinja2 3.0.1 which is incompatible.

Then confirming after the command finished:

(venv) kinow@ranma:~/Development/python/workspace/cylc-rose$ pip list | grep -i -E "aiofiles|jinja"
aiofiles           0.7.0     
Jinja2             3.0.1     

At the moment I think Cylc Flow depends on Jinja2 2.11, 'jinja2==2.11.*',. But Rose accepts anything after Jinja2 2.10 "jinja2>=2.10.1",.

Release version(s) and/or repository branch(es) affected?

master branch

Steps to reproduce the bug

pip install -e . in a new venv environment.

Expected behavior

No errors logged, correct versions of compatible dependencies installed.

Screenshots

Additional context

Pull requests welcome!
This is an Open Source project - please consider contributing a bug fix
yourself (please read CONTRIBUTING.md before starting any work though).

validate rose configs against metadata on installation

Issue transferred from metomi/rose#1908

On install run validator macros against:

1. The suite config?

• And each optional suite config individually?

2. The app configs?

• And each optional app config individually?

From the copy in:

a. The source dir (pre-install)?

• Can block installation upon encountering invalid configs.
• Cannot validate any apps that are installed using Rose file installation.

b. The run dir (post-install)?

• Can't block installation (because it has already happened).
• Can quibble, log warnings and change the exit status of the install command.

When:

i. By default (with a CLI flag to opt-out)?

• Could make installation slow.

ii. When a CLI flag is used (opt in)?

• Would never be used.

iii. According to the config?

• E.G. Allow the rose-suite.conf to configure apps to exclude?
• E.G. Support the rose macro --no-warn=WARNING_TYPE opt?

Context:

• Validating the suite config should be quick.
• However, the apps may take a while.
• Not all opt configs are necessarily valid in of themselves.
  • Though it can be contested that this is poor design.
  • Note: some opt confs may be designed to be used in combination with others.
• The proportion of real world suites that pass rose macro is likely quite low.

Factor out common strings

Strings such as 'jinja2:suite.rc' and 'empy:suite.rc' should be factored out and replaced by variables to improve the maintainability of code.

Pull requests welcome!

post_install: dump the parsed configuration to conf/<timestamp>-rose-suite.conf

https://cylc.github.io/cylc-admin/proposal-rose-suite-run.html#log-file-changes

`[jinja2:flow.cylc]` should not fail silently.

Describe the bug

Given that this rose-suite.conf is legal at Cylc 7:

[jinja2:suite.rc]
SLEEP=3

... at least one user has tried:

[jinja2:flow.cylc]
SLEEP=3

Which is not legal.

We should add an error.

Pull requests welcome!
This is an Open Source project - please consider contributing a bug fix
yourself (please read CONTRIBUTING.md before starting any work though).

Rose stem installs to `~/cylc-run/rose-stem`

Describe the bug

When $PWD is not a subversion repo rose stem --source=X=$PWD is unable to get a name for the workflow install directory and installs the workflow in ~/cylc-run/rose-stem. Ideally the basename of the source directory should be used.

Expected behavior

rose stem --source=X=/home/h02/tpilling/cylc-src/example
...
- INSTALLED example/run1 from /net/home/h02/tpilling/cylc-src/example/rose-stem
+ INSTALLED rose-stem/run3 from /net/home/h02/tpilling/cylc-src/example/rose-stem

Pull requests welcome!
This is an Open Source project - please consider contributing a bug fix
yourself (please read CONTRIBUTING.md before starting any work though).

functional tests

Test the cylc-rose functionality from within the cylc-rose repository (rather than in cylc-flow).

There are reasons why this should be done here:

• Nicer (less stuff in cylc-flow).
• Ensures we have the correct Cylc/Rose version bracketing.
• Allows us to have a testing matrix with different Rose/Cylc versions to ensure inter-compatibility.

Suggested approach, use pytest, avoid the filesystem where possible:

Split the existing functions into two to separate the cylc interface and fileparsing logic from the cylc-rose internals e.g:

# the entry point
def get_rose_vars(dir_, opts):
    # illustrative and very non-functional :/
    config_tree = rose_config_tree_loader(dir_, opts)
    node = config_tree.node
    vars = _get_rose_vars(node)
    # ...
    config = {}
    config['whatever'] = vars['whatever']
    # ...
    return {'whatever': vars}

def _get_rose_vars(node):
    # ...

Then test the internal cylc-rose bits without the cylc-flow interface overheads and associated filesystem meddling:

def test_env():
    # illustrative and very non-functional :/
    node = ConfigNode()
    node.set(['env', 'foo'], '1')
    assert _get_rose_vars(node) == {
        'whatever': {
             'foo' 1
        }
    }

Make Functional Tests Fail Helpfully

Problem

Lots of tests take the form

result = subprocess.run(['some', 'command'])
assert result.returncode == 0

If these tests fail you don't get a helpful error message.

Possible Solutions:

• compare stderr, stdout and returncode tuple in test.
• log the stderr if test fails.

Pull requests welcome!

Fix the Copyright notices.

Description

Currently the copyright notices are the same as the Cylc ones, including the name.

To do list.

• Fix the name of the project in module docstrings.
• Fix the name of the project in the readme.

Warn if [template variables] set but no shebang

@kinow Tried the following: Did not get an informative error message

@wxtim I used @oliver-sanders ' example from the linked issue.

1. Created ~/cylc-src/rose1/flow.cylc with the following content:

[scheduling]
    initial cycle point = {{ ICP }}
    [[graph]]
        R1 = foo

Then created a ~/cylc-src/rose1/rose-suite.conf (not using myopt as in his example for now, will after this simpler example works).

[template variables]

ICP = "20000101T0000Z"

3. Installed the workflow

(venv) kinow@ranma:~/cylc-src/rose1$ cylc install
INSTALLED rose1/run1 from /home/kinow/cylc-src/rose1

4. Verified the files were installed/copied correctly

(venv) kinow@ranma:~/cylc-src/rose1$ ls ~/cylc-run/rose1/run1/
flow.cylc  log  opt  rose-suite.conf

5. Tried cylc validate rose1/run1

(venv) kinow@ranma:~/cylc-src/rose1$ cylc validate rose1/run1
IllegalValueError: (type=cycle point) [scheduling]initial cycle point = {{ ICP }}

Any idea what step I missed? I am trying @oliver-sanders ' example first as that was in the ticket, and looks like a simple way to > test this PR. Will try more Jinja2/empy once I get this one working :)

Thanks!!!
Bruno

I believe that Cylc Rose should warn if [template variables] section in rose-suite.conf and there is no shebang in the flow.cylc.

Pull requests welcome!
This is an Open Source project - please consider contributing a bug fix
yourself (please read CONTRIBUTING.md before starting any work though).

catch rose errors and log them with Cylc

Blocked by: metomi/rose#2427

Cylc catches all cylc.CylcError exceptions and suppresses traceback to give the user a clear error message instead.

The cylc-rose repo should extend this functionality to Rose exceptions so that they also give clear error messages rather than pages of traceback.

One, crude, but effective way to do this is to patch Rose exceptions in the cylc-rose plugin to inherit from CylcError. Crude, but effective:

>>> import cylc.flow.exceptions
>>> # this the exception to patch (e.g. from metomi.rose import ...)
>>> class Foo(Exception): pass
... 
>>> # this is the new exception which inherits both
>>> class FooError(cylc.flow.exceptions.CylcError, Foo): pass
... 
>>> # now we crudely patch it in :)
>>> cylc.flow.exceptions.CylcError = FooError
>>> # now to prove it works...
>>> e = cylc.flow.exceptions.CylcError('a')
>>> isinstance(e, CylcError)
True
>>> isinstance(e, FooError)
True

jinja2: interpretting values

The Problem

At Cylc7/Rose2019 the [jinja2:suite.rc] variables were crudely pasted into a Jinja2 {% set %} statement. This meant that variables were written in the Jinja2 format which makes perfect sense.

However, with the addition of EmPy (another Python based templating engine) and more importantly the Python future of Cylc defining workflow variables in Python makes more sense.

As extra motives:

• Cylc7 does not have typed variables, Cylc8 does, we needed a format, Python makes the most sense, it opens the door to using things like Pandas dataframes or other Python objects in the construction of workflow definitions via a future Python configuration interface.
• The previous arrangement permits not just literals but also expressions which is highly undesirable.

Literal Parsing

So the idea was to only parse Jinja2 vars from Rose configurations as Python literals since the two are very similar. Sadly, the exceptions to the rule appear to be much more plentiful than anticipated. Here follows a breakdown of the exceptions:

Jinja2 Expressions

Unintended side-effect of the way Jinja2 vars were dumped into {% set %} statements.

Example:

[jinja2:suite.rc]
LIST=[0] + range(5) | list
INT=4 + (7 * 24)

Possible workaround:

None!

This is considered bad practice and something which should not have been possible in the first place.

Multi-Line values

This refers to a subset of Jinja2 expressions that may be considered somewhat legitimate.

Examples:

[jinja2:suite.rc]
# 'foo bar baz'
MULTI_LINE_STRING='foo'
                 =' bar'
                 =' baz'
# [1,2,3]
MULTI_LINE_LIST=[
               =1,
               =2,
               =3,
               =]

Workaround:

s|\n||g

Lower-Case Booleans

Jinja2 accepts lowercase booleans (which Python does not).

Example:

[jinja2:suite.rc]
PYTHONIC_BOOL=True
LOWERCASE_BOOL=true

Possible Workaround

s/^true$/True/, s/^false$/False/.

Implicit Tuples

Example:

Jinja2 allows the declaration of tuples without the leading or trailing brackets:

[jinja2:suite.rc]
PYTHONIC_TUPLE=(1,2,3)
IMPLICIT_TUPLE=1,2,3

Workaround:

try:
    evaluate_value(value)
except:
    # attempt to parse this as an implicit tuple
    value = (
        evaluate_value(item)
        for item in value.split(',')
    )

Zero-Padded Integers

Take the value 0644, this will be parsed as:

• 420 - Bash
• 644 - Jinja2
• 420 - Python2
• SyntaxError - Python3

Bash and Python2 both parse this as an octal, Jinja2 parses it as a decimal, Python3 requires either the removal of the leading 0 to parse it as a decimal or the addition of a leading o to parse it as an octal.

Example

[jinja2:suite.rc]
# harmless (in Python2/Bash)
DECIMAL=01
# not so harmless (in Python2/Bash)
DECIMAL=09

Workaround

Since Python2 octals are invalid Python3 octals it is possible to support the two side-by-side:

s/^0+//

Rose Types

This highlights an additional issue with Rose.

It is possible to define an implicit tuple and make that editable in Rose Edit by setting the metadata length, this goes a long way to explaining why this approach has seen so much usage.

Rose does have a python_list type, but that does not allow you to specify the type of the data contained within in the manner that setting the length does.

Rose also lacks metadata types for:

• tuples
• dicts (not such an issue)
• sets (not an issue at Python2, slightly more of an issue at Python3)
• octals