cylc / cylc-doc Goto Github PK
View Code? Open in Web Editor NEWDocumentation (User Guide, Cheat Sheets, etc.) for the Cylc Workflow Engine.
Home Page: https://cylc.github.io/cylc-doc/
License: GNU General Public License v3.0
Documentation (User Guide, Cheat Sheets, etc.) for the Cylc Workflow Engine.
Home Page: https://cylc.github.io/cylc-doc/
License: GNU General Public License v3.0
To re-run tasks that already ran, users need understand the following:
Also:
Document how best to process large numbers of files generated sequentially in model runs - as described in cylc/cylc-flow#2241
cylc message
usage. #105cylc message
, cylc broadcast
"pipe".We could do with more information about advanced cycling features such as:
T-00
for start at the next 00 minutes
)and anything else that comes to mind.
After metomi/isodatetime#69 has been implemented and incorporated into cylc,
we should document that users will be able to use backslashes to escape interpretation
of characters like D
in template strings. This also applies to cycle point format specification in the suite.rc.
Document how to use Cylc in Jupyter Notebooks using nest_asyncio
.
See:
We should make the documentation available for multiple versions of Cylc and adopt a build directory structure similar to Rose.
Now that we have split Cylc into multiple decoupled components this is a little awkward but I would suggest the Cylc Flow version is the one which matters for the documentation.
Versioning documentation is pretty simple, change the directory structure so that documentation is built under a version directory, in Rose we have:
sphinx/
# the documentation source files
conf.py
index.rst
...
doc/
# the built documentation
formats.json # file listing versions and formats for the theme to present as options.
2019.01.0/
html/
index.html
...
slides/
index.html
...
pdf/
index.html
tutorial.pdf
...
2018.06.0/
...
Currently the User Guide opens with two items, 'Scheduling Forecast Suites' & 'EcoConnect', which chronicle in detailed paragraphs the NWP origins & background to Cylc.
While those sections are useful, having them as the very first sections could put off potential users who are considering Cylc for other contexts. It would be more appropriate to instead have the leading content concisely & directly summarise what Cylc is, or how to (install &) get started using it, as those are the most likely questions users will have when they consult the documentation.
We currently do not specify any information about the necessary Python version that custom xtrigger functions can be written in, just that e.g. "Trigger functions are just normal Python functions".
From a quick test, in 8.0a0
, an xtrigger written with Python 3 print function syntax works as it should, but if instead changed to Python 2 print statements, it throws a SyntaxError: Missing parentheses in call to 'print'.
From this & then going on to check in the cylc-flow
codebase, it seems that xtriggers must more specifically be compatible with the version of Python required by cylc-flow
, namely >=3.7
.
A few questions, perhaps for @hjoliver or @kinow (but if anyone else wants to respond that is fine), before I add in this information:
cylc-flow
?I was wondering if the Suite Design Guide could be updated with some details about xtriggers
. I cannot see them mentioned anywhere in there, and I would personally find it useful to adopt a standard approach to using them from the outset. I don't know what to suggest as I have not used them yet. Or are they still too fresh to include in the design guide?
Allow the docs to be built in any of the formats supported by Sphinx.
If someone wants ePub let them build it.
This is completely trivial, just pass the name of the format through to Sphinx, default to HTML if no format provided. See how it is done in Rose.
Unfortunately the sphinx doc "search engine" is very basic: it only searches individually indexed words, there's no way to search for patterns or phrases. This means searching for any multi-word phrase that has one or more common words in it will result in loads of spurious results.
[UPDATE: what's addressed so far on this issue]
Describe exactly what you would like to see in an upcoming release
Add documentation about processes spawning in Cylc 8.
Additional context
@oliver-sanders replied to a comment with an excellent explanation about some processes created in Cylc. I think that information is gold, and since we already have some backlog for documentation pre cylc-8 release, maybe we could include this task there too.
Ideally, I think we should document our intentions and what we expect to happen. Not sure if what we say will apply to all Python 3.x versions (e.g. Jython, IronPython, etc) nor to all *nix flavours (e.g. mac OS BSD variants, other BSD variants, etc). But that would still be useful to site admins and especially to developers.
Pull requests welcome!
This is an Open Source project - please consider contributing code yourself
(please read CONTRIBUTING.md
before starting any work though).
I was looking for something in the doco and noticed, what I believe are a lot of unintentional \
characters. For example, in https://cylc.github.io/doc/built-sphinx-single/index.html#scheduling-xtriggers-many
Replace \_\_MANY\_\_ with any user-defined event trigger
I assume is meant to be Replace __MANY__ with any user-defined event trigger
? I just wanted to check just in case it is intended.
Being the one with less experience on troubleshooting Cylc, it has always been challenging when a cylc workflow did not behave the way I expected it to.
Other tools include sections with tips for admins and developers to troubleshoot the tools. Maybe we could do the same? Here are some cases that could be documented (feel free to edit this or add more comments):
pip
, virtualenv
, conda
, etc. e.g. if you have a build
directory from a previous python setup.py bdist
it may interfere when using virtual environmentsWe should document that our built-in pad
filter is not needed. Just use Python-style string formatting:
foo_{{ "%05d" | format('3') }}
prints:
foo_00003
Users of suite-state polling for inter-suite triggering often hit the problem that polling tasks time out and fail if their suite is running a bit behind the clock and the upstream suite has been cold-started from scratch recently, thus wiping out its prior history of task completion.
Suite-state polling is typically used as a proxy for waiting on a file generated by a task in the upstream suite. If we checked for the file's existence and only bother to start polling the upstream generating task if it doesn't exist yet, this would largely eliminate the above problem. (And the downstream suite presumably has to know where the file lives anyway, in order to retrieve or use it).
So, to be documented:
cylc suite-state
if necessary.Finally, I considered adding a new option to cylc suite-state
to call a user-supplied conditional check (e.g. file exists) before polling or not. But I don't think there's much point when you can just use a custom polling task, and further:
Undocumented features of task job polling:
6*PT5M
has exactly the same effect as PT5M
@matthewrmshin - can you a) confirm this behaviour; and b) confirm that it is intentional? (If so, we just need to document it)
Also, the logged health-check info seems to be misleading or meaningless in light of the above behaviour. E.g. for execution time limit = PT10S
and execution timeout polling intervals = 6*PT5S
:
...INFO - [foo.1] -health check settings: execution timeout=PT40S, polling intervals=PT15S,5*PT5S,...
where PT40S is PT10S plus 6*PT5S. Misleading because the "execution timeout is really still PT10S (e.g. as translated to PBS wall time directive) and in any case (as explained above) polling will not actually stop at PT40S - it will keep on polling until the job polls as finished. So what does this PT40S represent exactly? (Or do we actually set the timeout event to go off at PT40S in this case?)
Make use of reference suite by location functionality for cylc run
.
Document use of site job-init-env.sh
to configure platform environment for Cylc etc. (And custom job error handling script too, e.g. to dump the complete job environment to stderr).
Add extension exercises to the practical sessions to keep more advanced candidates from twiddling their thumbs.
Ideally less structured and more exploitive, open ended work.
Now that we are seeing more usage of the suite run database, we should document the schema in CUG.
See also #864, #1032.
See "old tutorial" comment in #38
The old (and now removed) cylc tutorial wan't actually much of a tutorial but more, the beginnings of a user guide.
Recover any useful sections from the old tutorial and put them somewhere sensible.
The 'Installation' documentation section (quick link here) will need updating to reflect the state of Cylc 8, which has diverged greatly to Cylc 7 with respect to installation approaches & requirements.
The update should at least update the now-outdated information on:
setup.py
;The master branch is failing on Travis with
ModuleNotFoundError: No module named 'sphinx'
Remove duplicate code by making one documentation system (presumably Rose) dependent on the other (Cylc?).
Example duplication, src/ext/cylc_lang,py
Cylc language lexer.
[update]: see the sister PR https://github.com/metomi/rose/pull/2374/files
Exceptions in the asyncio loop may not be logged/displayed to the user. We should document somewhere how to troubleshoot or identify these issues. I suspect it will go to Cylc Doc, somewhere a user/site admin would read.
See cylc/cylc-uiserver#65 (comment) comment for instructions how to enable it.
The official docs for Python 3 asyncio: https://docs.python.org/3/library/asyncio-dev.html#debug-mode
Evidently some new users might misinterpret #
as a shell prompt rather than a comment, in some contexts - #2695 . It should be easy enough to make these completely unambiguous.
It is common to find "cheat sheets", condensed summaries in a single-tab-viewable or single-page-printable format, for aspects of many medium-to-large software projects, either included under the official documentation/website, or created & distributed independently by users. Some nice & fairly diverse examples are those, as linked, for Conda & pandas (official) & for Vuex & React (unofficial).
Cylc users could benefit from something similar. They may already have created their own, or primitive or non-prettified variants of, but to minimise dissemination of misinformation, deprecated content or unsuitable practices, we could maintain (an) official "cheat sheet(s)" under version control.
If we design a print out (version), we could hand it out at appropriate training or workshops, not just as a working aid but as a material resource to promote Cylc.
NB the Rose documentation has a Cheat Sheet page including standalone Cylc aspects which is very good, but minimal in scope, only covering a basic subset of the CLI for controlling & reviewing suites. We could create a more extensive complement under the Cylc project.
We could sensibly condense a variety of content, but what would be worthwhile? Possibilities could be common aspects of:
support.html
page.tree
command output in the docs tutorial.Should any cheat sheet(s) be included under (a page within) the built Sphinx documentation, as a resource under the Cylc repo for access via GitHub, or as a resource accessible from (e.g. linked from some, or its own, page under) the Cylc website?
In the former case, we are constrained to the reStructuredText format, otherwise we can use alternative markup languages or tools (see below) which might enable a more creative solution:
create in | supply in |
---|---|
reStructuredText | documentation, under built Sphinx page |
web (HTML, CSS, perhaps a little JavaScript) | separate root website page |
LaTeX | PDF form linked & downloadable from root website page |
graphics software e.g. Inkscape | image file e.g. PNG linked & downloadable from root website page |
Capability for updating & reviewing content with change history rules out graphics software as a good choice, since updated images would need to be eyeballed for visual changes, lacking a traceable code diff
(in the LaTeX case we could include the source code somewhere as well as the generated PDF).
Users have different levels of knowledge & work in different ways, so ideally we could create something that can be customised, to a sensible bare extent, to reflect that.
I envisage a one-page template into which boxes corresponding to select top-level items such as the examples listed under the above Scope section, with their own sub-categories as pluggable sub-boxes, can be dropped & dragged (to rearrange) as desired.
(Ref recent discussion with Alan on Riot.im)
Users (unfortunately) need to understand the server "task pool" and how it evolves, in certain situations:
consequences of spawn-on-submit:
foo[-P1]:submit => foo
)dynamic manipulation of workflow structure:
Spawn-on-demand (Cylc 9?) should fix these deficiencies. In the meantime, spawn to max active cycle points
may be a good workaround for many suites.
The keys in the results
dictionary returned in the satisfied signature (True, results)
from a external trigger function become appended to the xtrigger name to form an environment variable identifier, which places many restrictions on these keys that are undocumented under the 'Custom Trigger Functions' section (only perhaps alluded to in the section 'Built-in Suite State Triggers' docs section which is referenced there). All that is stated there now is that "results
is an arbitrary dictionary of information to be passed to dependent tasks"
If keys that do not conform are used, then all tasks depending on the xtrigger will fail, due to an error in exporting the variables named from the keys that get passed along. The job.err
for the downstream tasks will show, e.g. if a key -2
was used:
... 01/job: line 34: export: `breakable_xtrigger_-2': not a valid identifier
2019-07-23T21:53:21+01:00 CRITICAL - failed/EXIT
Furthermore, there is no explicit mention that the results
dict cannot be nested, to form some more complex data structure to pass through, which users could assume (I did so myself). Given the above, & after investigation, it does seem that results
must be flat, else some very strange suite behaviour followed by suite shutdown (see cylc/cylc-flow#3237) results, so this should also be explicitly documented.
So, in short, the following should be clearly documented, under the 'Custom Trigger Functions' &/or 'Current Limitations' sections, ideally with an indication of the symptoms of invalid keys to help users debug::
results
dict keys must be only contain characters that are valid within an environment variable (no easy standard to find but this thread suggests being all alphanumeric or underscores only as tested below);results
dict must be flat.Use this custom trigger function in a basic suite & change the toggle variable systematically to observe the final task state of some trivial dependent tasks for each results
case, where the dict value indicates whether the above problem emerges or not:
def breakable_script(toggle):
if toggle == "singlequote":
return (True, {'AMIBROKEN': 'no'})
elif toggle == "underscore":
return (True, {"Am_I_broken": "no"})
elif toggle == "plainnumeric":
return (True, {2: "no"})
elif toggle == "space":
return (True, {"Am I broken": "yes"}) # triggered tasks fail
elif toggle == "dash":
return (True, {"Am-I-broken": "yes"}) # triggered tasks fail
elif toggle == "signednumeric":
return (True, {-2: "yes"}) # broken: triggered tasks fail
elif toggle == "comma":
return (True, {"Am,I,broken": "yes"}) # triggered tasks fail
elif toggle == "dot":
return (True, {"Am.I.broken": "yes"}) # triggered tasks fail
elif toggle == "float":
return (True, {2.5: "yes"}) # broken: triggered tasks fail
elif toggle == "slash":
return (True, {"Am/I/broken": "yes"}) # triggered tasks fail
elif toggle == "nested":
res = {"dictA": {"keyA": "valueA"},
"dictB": {"keyB": "valueB"}}
return (True, res) # this one is broken in a much funkier way!
else:
return True, {}
We should aim towards moving all significant chunks of hard-written code out of the .rst
content files, & (if not located already in a known place, as in e.g. #65) into a separate dedicated directory as individual files, where they would be written into the built docs from the relevant file by the literalinclude:
directive.
This would have various benefits, for example it would:
suite.rc
examples;#3 moved all docs here from the cylc-flow repository, simplistically taking the relevant bits of setup.py
and cylc-check-software
with it.
We need to decide how best to build the docs here, and how best to depend (where necessary) on other project/repo versions, particularly cylc-flow. (e.g. should we use setup.py
here at all?)
See also #17
Depends on #2972
See discussion in #3037
Was looking at cylc_lang
for #43, and noticed two EMPY_BLOCK_REGEX
declarations. The second one replacing the first one:
Just wondering if it's safe to remove the first one, or if it was intended to replace the second... I think @oliver-sanders or @sadielbartholomew might know?
I noticed that when I run make html singlehtml
, the documentation appears to be generated under build/sphinx
directory, but it looks like it also creates some directories under the doc
.
If we run python setup.py bdist
, then the bdist distribution package would go under build/bdist.linux-x86_64
or so. Since we have the sphinx
folder being created under build
, and setuptools
is using that directory as well, then maybe we could stop using doc
too?
Or is there a reason for the need of another folder for building the documentation?
As discussed elsewhere, cylc-8 docs will need to span multiple components, not just cylc-flow, so best in a separate project.
Remove the figure numbering inherited from the LaTeX docs.
See "code block labels" comment in #38 (review) and the related #15
Perhaps stating the obvious, diagrams are much clearer & intuitive for explaining certain concepts than chunks of text, & they are also great for summarising content. #29 is a lovely example.
There are a lot of great schematics in the docs already, but the diagram to text ratio is still too low for my liking, & there are certain topics I read through & feel are crying out for a summarising, or substitute, diagram.
Bonus points if diagrams can be practicably auto-generated for content that can be & that is likely to change frequently, or at least is editable on an element-level so that PRs would not need to replace a whole image, where changes would be unclear.
Follow-on from #38.
Find a new home for the Cylc Tutorial suite.
There are some issues to be migrated to wherever the Cylc Tutorial lands:
The Cylc Tutorial makes use of the rose tutorial
command which is basically just a glorified rsync
which copies resources from $ROSE_HOME/etc/tutorial
to $HOME/cylc-run
for the user to work on.
The tutorial suite contains:
We will need to replace the rose tutorial
command. The question is where to put it. Options:
extra_requires
)Numbered sections, the norm in scientific publication due to the PDF format, not quite at home in the world of technical documentation in HTML format.
Master listing of all enhancements proposed for the documentation (formerly the User Guide & Suite Design Guides) under Sphinx infrastructure, after bare-bones conversion from LaTeX sourcing (#2910). Let's utilise the newly-obtained power of Sphinx! ๐ช
NB: do not use this to list proposed content improvements, only enhancements to the ultimate presentation output of the material e.g. to aspects of the infrastructure, new features, formatting, styling, bells & whistles, etc. Feel free to move items to separate issues, if appropriate, but please make annotations to that effect for tracking purposes.
cylc make-docs
, a minimal bash script, but integration into a setup.py
would be superior (see cylc/cylc-flow#2910 (comment) & cylc/cylc-flow#2910 (comment))Auto Documentation
section)To generate:
suite.rc
reference i.e. the suite configuration API;deprecation
, versionadded
& versionchanged
directives: for items in config references.seealso
directive: for many instances of plain-text "see (also) sub-section XYZ" pointers to improve internal cross-referencing.tip
, hint
, error
, for specially-rendered topics. Only some notes
& warnings
directives have been added in for obvious cases as part of #2910.tree
-command-like directory structures in remote-job-management.rst
: these were written up manually, minicylc
directive from Rose to generate graphs to ensure images are in sync with code snippetssuite.rc
code blocksminicylc
directive, as created for the Rose docs, for animated graphs;cylc graph
command to generate diagrams of example suites (see cylc/cylc-flow#2651 (comment)) (see also the minicylc
directive).minicylc
: => issue migrated to cylc/cylc-sphinx-extensions#4)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.