crate / crate-docs

Common build system, QA tools, style guides, and other meta miscellanies for doing docs at Crate.io.

---

Help us improve CrateDB by taking our User Survey!

About

CrateDB is a distributed SQL database that makes it simple to store and analyze massive amounts of data in real-time.

CrateDB offers the benefits of an SQL database and the scalability and flexibility typically associated with NoSQL databases. Modest CrateDB clusters can ingest tens of thousands of records per second without breaking a sweat. You can run ad-hoc queries using standard SQL. CrateDB's blazing-fast distributed query execution engine parallelizes query workloads across the whole cluster.

CrateDB is well suited to containerization, can be scaled horizontally using ephemeral virtual machines (e.g., Kubernetes, AWS, and Azure) with no shared state. You can deploy and run CrateDB on any sort of network — from personal computers to multi-region hybrid clouds and the edge.

Features

• Use standard SQL via the PostgreSQL wire protocol or an HTTP API.
• Dynamic table schemas and queryable objects provide document-oriented features in addition to the relational features of SQL.
• Support for time-series data, real-time full-text search, geospatial data types and search capabilities.
• Horizontally scalable, highly available and fault-tolerant clusters that run very well in virtualized and containerized environments.
• Extremely fast distributed query execution.
• Auto-partitioning, auto-sharding, and auto-replication.
• Self-healing and auto-rebalancing.
• User-defined functions (UDFs) can be used to extend the functionality of CrateDB.

Screenshots

CrateDB provides an Admin UI:

Try CrateDB

The fastest way to try CrateDB out is by running:

sh$ bash -c "$(curl -L try.crate.io)"

Or spin up the official Docker image:

sh$ docker run --publish 4200:4200 --publish 5432:5432 --env CRATE_HEAP_SIZE=1g crate -Cdiscovery.type=single-node

Visit the installation documentation to see all the available download and install options.

Once you're up and running, head over to the introductory docs. To interact with CrateDB, you can use the Admin UI sql console or the CrateDB shell CLI tool. Alternatively, review the list of recommended clients and tools that work with CrateDB.

For container-specific documentation, check out the CrateDB on Docker how-to guide or the CrateDB on Kubernetes how-to guide.

Contributing

This project is primarily maintained by Crate.io, but we welcome community contributions!

See the developer docs and the contribution docs for more information.

Security

The CrateDB team and community take security bugs seriously. We appreciate your efforts to responsibly disclose your findings, and will make every effort to acknowledge your contributions.

If you think you discovered a security flaw, please follow the guidelines at SECURITY.md.

Help

Looking for more help?

• Try one of our beginner tutorials, how-to guides, or consult the reference manual.
• Check out our support channels.
• Crate.io also offers CrateDB Cloud, a fully-managed CrateDB Database as a Service (DBaaS). The CrateDB Cloud Tutorials will get you started.

crate-docs's Issues

Vale doesn't always run

this is what we did:

• set up a new docs project from scratch using the crate-docs build system
• run make dev and saw that it worked for the first time (including linting prior to sphinx build)
• made an edit to a file and saved
• saw that sphinx-autobuild is correctly triggered

what I expected to happen:

• automatic re-linting of the changed file

Silence Vale if there is no output

at the moment, if there is nothing to report on, having lots and lots of Vale output looks sort of messy. it would be good to either shorten it to a single line (so you can see it is making progress)

Advertise `make dev` when `make html` is run

Markus runs make html

when someone runs this command, the makefile should advertise the existence of the dev target (and/or, the help target)

Translation support

Hi,

As an Open Source projects translator, I have to ask if this remarkable system and beautiful theme support i18n, or if it is planned.

TY
J

Fix/silence output: "Nothing to be done for `autobuild-deps'"

I am noticing the following at the start of output:

$ make dev
make[3]: Nothing to be done for `autobuild-deps'.

this should be fixed/silenced

Change linting error causing test failure for sentence-initial "so" into warning

Currently in the docs-utils, tests can fail due to the linter objecting to starting a sentence with 'so'. While doing so isn't terribly stylish, it doesn't seem egregious enough of an error and it seems too contextual to justify being an error as opposed to a warning. (For context, several other style issues of a similar nature are already flagged as warnings, not as errors.) I propose we change the error to a warning in this case also.

misc lint/style improvements

we should write our own linter

• figure out how to check for unused links in RTD and build test for it
• add test to docs to check for trailing whitespace
• clarify the situation with line-wrapping in docs
• figure out how to properly handle images and standardize this everywhere (_static folders)
• go through and clean up paragraph formatting in all docs
• decide on the use of "consult" or "see" (e.g., https://github.com/docker-library/docs)
• fix up all historic instances of "fix" vs "fixed" in the changelog (and other related imperative mood changes)
• replace "table_ident" in docs with "my_table"
• go through all docs and replace references to "documents" (Lucene jargon)
• replace all :doc: statements with :ref: statements
• standardize docs reference string nomenclature
• (possibly) rewrap docs to match line width of GitHub diff comments so no ugly wrapping
• go through all repos, including docker-docs, and update "the project documentation" in the readme to drop the "the" from the hyperlink for consistency
• standardise header level of "Limitations" section at end of some docs

all lint checks should be documented in the style guide

Compatibility with Python 3.9

Hi there.

This is just a quick note. When running make html within docker run -it --rm python:3.9.0 bash, it errors out with

Installing collected packages: urllib3, pytz, pyparsing, MarkupSafe, idna, chardet, certifi, tornado, sphinxcontrib-serializinghtml, sphinxcontrib-qthelp, sphinxcontrib-jsmath, sphinxcontrib-htmlhelp, sphinxcontrib-devhelp, sphinxcontrib-applehelp, snowballstemmer, six, setuptools, requests, Pygments, packaging, Jinja2, imagesize, docutils, babel, alabaster, sphinx, livereload, sphinx-autobuild
    Running setup.py install for MarkupSafe ... done
  Attempting uninstall: setuptools
    Found existing installation: setuptools 49.2.1
    Uninstalling setuptools-49.2.1:
      Successfully uninstalled setuptools-49.2.1
    Running setup.py install for livereload ... error
    ERROR: Command errored out with exit status 1:
     command: /home/crate-docs/docs/.crate-docs/.venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-jrn94hws/livereload_30ab7443dd23497e995991957799188a/setup.py'"'"'; __file__='"'"'/tmp/pip-install-jrn94hws/livereload_30ab7443dd23497e995991957799188a/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-9bgqgv7d/install-record.txt --single-version-externally-managed --compile --install-headers /home/crate-docs/docs/.crate-docs/.venv/include/site/python3.9/livereload
         cwd: /tmp/pip-install-jrn94hws/livereload_30ab7443dd23497e995991957799188a/
    Complete output (11 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/crate-docs/docs/.crate-docs/.venv/lib/python3.9/site-packages/setuptools/__init__.py", line 20, in <module>
        from setuptools.dist import Distribution, Feature
      File "/home/crate-docs/docs/.crate-docs/.venv/lib/python3.9/site-packages/setuptools/dist.py", line 35, in <module>
        from setuptools.depends import Require
      File "/home/crate-docs/docs/.crate-docs/.venv/lib/python3.9/site-packages/setuptools/depends.py", line 7, in <module>
        from .py33compat import Bytecode
      File "/home/crate-docs/docs/.crate-docs/.venv/lib/python3.9/site-packages/setuptools/py33compat.py", line 55, in <module>
        unescape = getattr(html, 'unescape', html_parser.HTMLParser().unescape)
    AttributeError: 'HTMLParser' object has no attribute 'unescape'
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/crate-docs/docs/.crate-docs/.venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-jrn94hws/livereload_30ab7443dd23497e995991957799188a/setup.py'"'"'; __file__='"'"'/tmp/pip-install-jrn94hws/livereload_30ab7443dd23497e995991957799188a/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-9bgqgv7d/install-record.txt --single-version-externally-managed --compile --install-headers /home/crate-docs/docs/.crate-docs/.venv/include/site/python3.9/livereload Check the logs for full command output.
make[1]: *** [.crate-docs/src/rules.mk:110: .crate-docs/.venv/bin/sphinx-build] Error 1
make[1]: Leaving directory '/home/crate-docs/docs'
make: *** [Makefile:69: html] Error 2

With kind regards,
Andreas.

Should we update our license boilerplate?

our license boilerplate mirrors the crate/crate boilerplate, which still contains references to commercial license. @amotl asked whether we should update the boilerplates in this repo. I responded saying I would prefer to wait until crate/crate changes the boilerplate and then we can mirror that change

see here for @amotl's comment:

#74 (comment)

see here for my question about boilerplates on crate/crate:

crate/crate#11184

Enforce use of "en/latest" in cross-links

it is going to be very easy to accidentally cross-link to a document using the URL that appears in the browser when you visit it. this URL is typically going to include the version information, e.g.:

https://crate.io/docs/crate/reference/en/4.2/sql/statements/select.html

all internal links in our docs should use "en/latest" instead of linking to a specific version. so the above link should be changed to this:

https://crate.io/docs/crate/reference/en/latest/sql/statements/select.html

we should be able to lint for this

Automate the testing of tutorials to ensure cross-platform compatibility

see mika's comments here:

crate/crate-tutorials#3

it's vital that our tutorials work. more so than the rest of our documentation, I would say -- because the tutorials are likely to be a user's first impression

we want the tutorials to work across the widest possible array of operating systems and setups

testing this manually each time is a hassle, and probably won't be done reliably. so, we should come up with a way of testing (i.e., executing) the tutorials automatically as part of the CI setup

not sure what the best approach is here. so creating this issue to collect thoughts, discussion, etc

docs-utils auto-reset

at the moment:

• if you update utils.json you must remember to run make reset to actually pull in the new version of the docs-utils files if you already have a build cache

proposal:

• when you update utils.json, make will reset the build cache itself and fetch the appropriate docs-utils files

implementation notes:

this should be possible if utils.json file is a prereq for the build cache dir. however, any update to the file will trigger a reset. but this is probably fine, given that file's singular purpose

Add spell checker

Hi there,

by integrating Vale, this project already features a prose linter. On top of that, we might also want to think about integrating a spell checker. Hereby, I am proposing PySpelling, together with unanimous.

With kind regards,
Andreas.

change WriteGood "there are..." style rule behavior

In docs-utils, the WriteGood implementation of Vale currently flags an error when sentences begin with "there are..." or "there is...". This seems absurd and should probably be disabled

The `linkcheck` target is no longer run by `make check`

see

crate-docs/src/rules.mk

Line 160 in e39edc6

# If you are having problems with the `linkcheck` target, you might

the html and linkcheck targets share the same rule but do not specify the "builder". by default html is used. so if you run the linkcheck target as currently defined, it will run sphinx-build with the default html builder

the fix is to pass in the builder using the name of the matched target, using syntax that evaluates to -b html or -b linkcheck (as appropriate)

Bump some versions

Hi again,

I just have been looking at the dependency versions here, nailed within [1] and [2]. I would suggest that we might want to upgrade some of the main artefacts.

• Sphinx is available in version 3.3.1.
• Vale is available in version 2.6.5.
• Also: docutils==0.16.

With kind regards,
Andreas.

[1] https://github.com/crate/crate-docs/blob/master/src/requirements.txt
[2] https://github.com/crate/crate-docs/blob/master/src/rules.mk

P.S.: General compatibility of the crate-docs-theme with Sphinx 3.x should be given. See also crate/crate-docs-theme#232, crate/crate-docs-theme#233 and crate/crate#10836 where we have been working on bringing this to the crate/crate repository.

Render standalone directories of RST files

from @amotl:

Add capability to instantly render whole directories instead of single files only. When detecting an index.rst within the directory, open that file in the Browser.

right now, the preview script can only render a single standalone file. if that file references image in the same directory, they will be broken when viewing the HTML

additionally, it might be possible to allow linking to other documents in the directory

the only problem I see with this is that for the standalone preview to work, we essentially build a simple sphinx project with one file (named index.rst and soft linked to your original file). to create a sphinx project on the fly from a directory with multiple files, we would have to link all the files together using a toctree directive

you could do this by placing an index.rst file in each directory and generating the toctree so that there is an automatic navigation system in place for any ad-hoc collection of files. however, the actual index.rst files would be empty aside from a table of contents

Update Makefile to use python3

As I was setting up the docs environment I encountered this issue:

Even though I have python, python3, python-sphinx, python3-sphinx, sphinx-build and everything else necessary to run Sphinx (and have run Sphinx previously without issues), after running make html I kept getting back:

nmt@nmt:~/crate.io/cloud-howtos/docs$ make html
/bin/sh: 1: python: not found
Makefile:92: *** No build version specified in `build.json`..  Stop.

@amotl recommended editing the JSON parsing to use python3.
This has fixed the issue.

Should we reference InstantRst?

@amotl suggested we reference https://github.com/gu-fan/InstantRst in our helpers directory or scripts

see #74 (comment)

opening this as a question so we can come back to it

Investigate efficiency and noise when running Vale

Hi.

Currently, Vale is invoked through ./src/bin/lint on individual files, thus making use of the feature that make will only evaluate prerequisites which changed since the last time the target was remade, which is nice in general.

However, this will lead to multiple invocations of Vale on each single file, which well incurs some overhead. On this repository, the numbers are:

$ rm -r .crate-docs/lint; time make lint

real	0m12.442s

Also the output is very long and noisy, like

make lint

.crate-docs/tools/vale --config=.crate-docs/src/_vale.ini ../CHANGES.rst
Linting: ../CHANGES.rst
---------------------------------------------------------------------------------
✔ 0 errors, 0 warnings and 0 suggestions in 1 file.


.crate-docs/tools/vale --config=.crate-docs/src/_vale.ini ../CODE_OF_CONDUCT.rst
Linting: ../CODE_OF_CONDUCT.rst
---------------------------------------------------------------------------------
✔ 0 errors, 0 warnings and 0 suggestions in 1 file.


.crate-docs/tools/vale --config=.crate-docs/src/_vale.ini ../CONTRIBUTING.rst
Linting: ../CONTRIBUTING.rst
---------------------------------------------------------------------------------
✔ 0 errors, 0 warnings and 0 suggestions in 1 file.


.crate-docs/tools/vale --config=.crate-docs/src/_vale.ini ../DEVELOP.rst
Linting: ../DEVELOP.rst
---------------------------------------------------------------------------------
✔ 0 errors, 0 warnings and 0 suggestions in 1 file.


.crate-docs/tools/vale --config=.crate-docs/src/_vale.ini ../README.rst
Linting: ../README.rst
---------------------------------------------------------------------------------
✔ 0 errors, 0 warnings and 0 suggestions in 1 file.


.crate-docs/tools/vale --config=.crate-docs/src/_vale.ini ../docs/index.rst
Linting: ../docs/index.rst
---------------------------------------------------------------------------------
✔ 0 errors, 0 warnings and 0 suggestions in 1 file.


.crate-docs/tools/vale --config=.crate-docs/src/_vale.ini ../style/README.rst
Linting: ../style/README.rst
---------------------------------------------------------------------------------
✔ 0 errors, 0 warnings and 0 suggestions in 1 file.


.crate-docs/tools/vale --config=.crate-docs/src/_vale.ini ../style/general.rst
Linting: ../style/general.rst
---------------------------------------------------------------------------------
✔ 0 errors, 0 warnings and 0 suggestions in 1 file.


.crate-docs/tools/vale --config=.crate-docs/src/_vale.ini ../style/hypertext.rst
Linting: ../style/hypertext.rst
---------------------------------------------------------------------------------
✔ 0 errors, 0 warnings and 0 suggestions in 1 file.


.crate-docs/tools/vale --config=.crate-docs/src/_vale.ini ../style/repos.rst
Linting: ../style/repos.rst
---------------------------------------------------------------------------------
✔ 0 errors, 0 warnings and 0 suggestions in 1 file.


.crate-docs/tools/vale --config=.crate-docs/src/_vale.ini ../style/rst.rst
Linting: ../style/rst.rst
---------------------------------------------------------------------------------
✔ 0 errors, 0 warnings and 0 suggestions in 1 file.


.crate-docs/tools/vale --config=.crate-docs/src/_vale.ini ../style/screenshots.rst
Linting: ../style/screenshots.rst
---------------------------------------------------------------------------------
✔ 0 errors, 0 warnings and 0 suggestions in 1 file.


.crate-docs/tools/vale --config=.crate-docs/src/_vale.ini ../style/writing.rst
Linting: ../style/writing.rst
---------------------------------------------------------------------------------
✔ 0 errors, 0 warnings and 0 suggestions in 1 file.

and I don't see a way to compress this in any way.

Given that there are 13 files to be checked

$ rm -r .crate-docs/lint; time make lint | grep file | wc -l
13

we see that it's roughly spending one second per file, mostly coming from the fact that it's invoked individually on each file.

On the other hand, invoking Vale on the whole tree is much more efficient and also reduces noise on the output.

$ time .crate-docs/tools/vale --config .crate-docs/src/_vale.ini ..
✔ 0 errors, 0 warnings and 0 suggestions in 23 files.

real	0m2.071s

So, doing it the other way round might improve stuff similar to crate/crate#10831 on the runtime and reporting side of things and will probably also help with #4. Do you have any objections on invoking Vale once on the whole tree instead?

With kind regards,
Andreas.

Create issue template that correctly sets labels

issues in this repository should set the "team: tech writing" label

Lint check: links and section header labels

• add check for anchors uses in links that go to our own docs (consider how this would change if we switched to using the Sphinx interdocs links plugin)

• add check for section headings without an explicit ID. to avoid heading rewordings from breaking existing links (e.g., https://crate.io/docs/crate/reference/en/latest/general/builtins/table-functions.html#generate-series-start-stop-step)

Always use prompts for code snippets?

Hi there,

this is a followup to @norosa's comment:

b) the sh$ prompt was designed for consistency with the frequent use of cr> prompts in the main docs. we can revisit that decision if you like, but I think that should be done as a separate thing (perhaps with a PR to crate-docs/style)

With kind regards,
Andreas.

Originally posted by @norosa in crate/cratedb-examples#3 (comment)

Spin Vale functionality out into Sphinx plugin

at the moment, the build system manually uses fswatch to run linting with Vale against Sphinx RST. it would probably make more sense to spin this out as a Sphinx plugin (which could then be interfaced with directly via sphinx-autobuild etc)

would be cool also as a Crate OSS project other people can use

Lint check: proper SQL keyword capitalization

compare against a list of SQL 99 keywords

Improve error message if Vale download fails

https://travis-ci.org/github/crate/cloud-getting-started/jobs/664347119?utm_medium=notification&utm_source=github_status

this is what happens when the vale git isn't responding. we could add a better error message here

Document build system on wiki

atm most of the build system (how it works, how it can be adapted/used, etc.) is in @mechanomi's head. let's get it out of her head and onto the wiki (in a gentle, non-destructive way. lol)

issue warning when version is out-of-date

when you run make dev or any other command, the makefile should check the version of the build tools being used against the current active release and issue a warning if the version being used is old

add note to user docs advertising make overrides

for passing custom arguments to commands like sphinx-autobuild

see https://github.com/crate/crate-docs-theme/blob/master/DEVELOP.rst#testing-on-mobile

`source_files` in rules.mk needs updating

I just realized, source_files needs updating too.

After checking, I see that this is used for computing git_log_targets, which in turn is used for the qa target. While I am not into all details of the docs build system yet (please bear with me), I believe we might have overhauled this qa/telemetry subsystem the other day by introducing lint-summary and friends on behalf of #46.

The idea behind that was to streamline all calls to Vale so that there are no different kinds of invocations and incantations. So, if you agree, we might want to remove that completely and make qa point to lint-report?

Originally posted by @amotl in #59 (comment)

Wishlist: link check that works with anchors

sometimes internal links that use anchors to link to specific sections break because the anchor ID is later changed (meaning the link takes you to the correct page, but does not bounce you to the intended subsection)

it would be great if the link checking was able to dereference anchors to make sure they exist on the target page (I suppose this could be implemented on internal pages as well as external pages. but for sure, the priority here is internal pages)

misc testing improvements

• figure out way to do "> cr" in docs and turn off test suites
• get doctests working for guide. see hello.txt

EPIC: Generate QA telemetry data from build system

the Makefile should offer a new target: report

• the qa target should generate QA telemetry data in JSON format

release history:

• 0.4.0
  • "last updated" value for each document (per Git log)
  • "last reviewed" value for each document (RST metadata manually maintained by reviewers)
  • run Vale and output the summary results per file

remaining QA telemetry to be implemented:

• count the number of documents, number of words, and so on
• count the number of internal links, external links
• count the number of headings (and maybe distribution of headings)
• more to come...

Enforce proper definition list markup syntax

our docs currently do something like this:

:Configured Shards:

  The `number of configured shards`_

the correct format is this:

**Configured Shards**:
  The `number of configured shards`_

Can the GitHub markup repository help improve our helper scripts?

@amotl was kind enough to bring my attention to https://github.com/github/markup

#74 (comment)

I believe it might be possible to use the info in this repository to make some improvements to the preview script (particularly the standalone rendering)

Add helpful message when Vale download fails

see build failure: https://travis-ci.org/github/crate/crate-admin/jobs/719651283

Build doesn't work anymore

@norosa the build doesn't work anymore, not sure if it's on my end, gonna check tomorrow:

ubuntu@dev:~/Documents/GitHub/crate-docs-theme/docs$ make SPHINX_OPTS='-W -n --host 0.0.0.0' dev
make[1]: Entering directory '/home/ubuntu/Documents/GitHub/crate-docs-theme/docs'
make[1]: Leaving directory '/home/ubuntu/Documents/GitHub/crate-docs-theme/docs'
make[1]: Entering directory '/home/ubuntu/Documents/GitHub/crate-docs-theme/docs'
Creating a Python virtual environment...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AssertionError
ERROR: Python>=3.7 is required.
.crate-docs/src/rules.mk:134: recipe for target '.crate-docs/.venv/bin/activate' failed
make[1]: *** [.crate-docs/.venv/bin/activate] Error 1
make[1]: Leaving directory '/home/ubuntu/Documents/GitHub/crate-docs-theme/docs'
Makefile:152: recipe for target 'dev' failed
make: *** [dev] Error 2

python --version gives Python 3.7.9

Originally posted by @msbt in crate/crate-docs-theme#245 (comment)

Add O'Reilly style guide to wiki

https://oreillymedia.github.io/production-resources/styleguide/

Crate.io Sphinx Inventory Labels

About

All labels and other target references of the inventories of all Sphinx-based projects at Crate.io, excluding the std:doc items. It helps you to discover relevant link targets when using intersphinx references.

Content

cratedb-sphinx-inventory-nodocs.md was generated on Fri Mar 1, 2024, using sphinx-inventories.txt.

Self-Service

Tip

You can also generate an up-to-date version of this output on your own behalf. For that, you need to set up a sandbox using the code in this repository, as outlined within the README. Then, the synopsis is:

invoke allinv --format markdown > cratedb-sphinx-inventory-nodocs.md

Add a basic QA reporting tool

DETAILS TO FOLLOW

Migrate docs utils to use GitHub actions

currently, we use Travis CI for docs CI on PRs. we should consider migrating to GitHub actions

look into act (runs github actions locally) for docs utils

CrateDB Windows Install guide

##Who
CrateDB User

##What
To know how to install CrateDB on Windows

##Why
To setup CrateDB on their local machine and use it for their project or product

---

Currently very minimal:
https://crate.io/docs/crate/tutorials/en/latest/basic/index.html#ad-hoc-unix-macos-windows
https://crate.io/docs/crate/tutorials/en/latest/basic/index.html#notes-about-microsoft-windows

Flesh out and have a separte guide how to install CrateDB on Windows

---

##Acceptance criteria

• Process is updated so it reflects the current state
• Screenshots for Windows setup are added to the documentation