roave / docbooktool Goto Github PK

View Code? Open in Web Editor NEW

52.0 52.0 4.0 9.38 MB

:books: Docbook Tool for static documentation generation from Markdown files

PHP 89.94% HTML 2.89% Gherkin 0.14% Twig 1.47% Dockerfile 4.33% Makefile 1.23%

docbooktool's People

Contributors

Stargazers

Watchers

Forkers

azjezz ciaranmcnulty bendavies herndlm

docbooktool's Issues

Inlined external PUML breaks when PUML has an include

Issue description

When an external PlantUML diagram that itself has an include, such as:

@startuml
!include external-diagram.puml
@enduml

is included by way of an inline using \Roave\DocbookTool\Formatter\InlineExternalImages, since the diagram is written to a temporary path to be rendered by PlantUML in \Roave\DocbookTool\Formatter\RenderPlantUmlDiagramInline, it becomes disconnected from the original source tree, and therefore the !include directive cannot be resolved.

Example test case

Example reproducer in our test suite:

$ git diff --cached
diff --git a/test/fixture/docbook/diagram-with-include.puml b/test/fixture/docbook/diagram-with-include.puml
new file mode 100644
index 0000000..f393672
--- /dev/null
+++ b/test/fixture/docbook/diagram-with-include.puml
@@ -0,0 +1,4 @@
+@startuml
+!include external-diagram.puml
+Bob<-Alice : hello2
+@enduml
diff --git a/test/fixture/docbook/test.md b/test/fixture/docbook/test.md
index 8f3490b..c661eb5 100644
--- a/test/fixture/docbook/test.md
+++ b/test/fixture/docbook/test.md
@@ -66,3 +66,5 @@ They are hand drawn, that's why they look rubbish.
 ## Inline PUML file
 
 ![An external PlantUML diagram](./external-diagram.puml)
+
+![An external PlantUML diagram with an include](./diagram-with-include.puml)

Workaround

Suggested workaround for consumers who encounter this bug until fixed: copy the necessary diagrams into /tmp before build, e.g.:

FROM builder AS built

# Temporary workaround for https://github.com/Roave/DocbookTool/issues/362
RUN cp -rvf /docs-src/diagrams/* /tmp

RUN bin/docbook-tool --html --pdf

Graphviz is missing

When trying to render a PlantUML that uses graphviz, the resulting image is:

I believe we just need to add Graphviz, but we'd need a test case to ensure it renders properly.

`hexagon` usage in PlantUML breaks

Creating a markdown with the following:

@startuml
Bob->Alice : hello
hexagon TestingHexagon
@enduml

breaks the PlantUML renderer:

$ vendor/bin/phpunit 
PHPUnit 9.5.19 #StandWithUkraine

E......                                                             7 / 7 (100%)

Time: 00:00.930, Memory: 10.00 MB

There was 1 error:

1) Roave\DocbookToolIntegrationTest\DocbookToolGeneratorTest::testGeneration
RuntimeException: Failed to render PUML in test - starts "
@startuml
Bob-". Output was: Error line 4 in file: /tmp/54ce0ad8736f73cf5236b6f67a827054.puml
Some diagram description contains errors

/home/james/workspace/docbook-tool/src/Formatter/RenderPlantUmlDiagramInline.php:57
/home/james/workspace/docbook-tool/src/Formatter/RenderPlantUmlDiagramInline.php:71
/home/james/workspace/docbook-tool/src/Formatter/AggregatePageFormatter.php:22
/home/james/workspace/docbook-tool/test/integration/DocbookToolGeneratorTest.php:56

ERRORS!
Tests: 7, Assertions: 6, Errors: 1.

Allow arrangement/ordering of pages

Allow the order/arrangement of pages to be redefined. Probably with some YAML front matter like order: (int) or something, or perhaps allow the path to be passed to the Twig template, so you could form a tree based on the path.

If content in the html results in a 404, wkhtmltopdf crashes

#0 15.42 [2022-10-07T15:10:59.748948+00:00] cli.DEBUG: wkhtmltopdf output: QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root' QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root' Exit with code 1 due to network error: ContentNotFoundError [] []
#0 15.42 PHP Fatal error:  Uncaught RuntimeException: Failed to generate PDF. Output was: QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
#0 15.42 QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
#0 15.42 Exit with code 1 due to network error: ContentNotFoundError in /app/src/Writer/MultiplePdfFilesWriter.php:76
#0 15.42 Stack trace:
#0 15.42 #0 /app/src/WriteAllTheOutputs.php(20): Roave\DocbookTool\Writer\MultiplePdfFilesWriter->__invoke()
#0 15.42 #1 /app/bin/docbook-tool.php(57): Roave\DocbookTool\WriteAllTheOutputs->__invoke()
#0 15.42 #2 /app/bin/docbook-tool.php(61): Roave\DocbookTool\{closure}()
#0 15.42 #3 /app/bin/docbook-tool(6): require_once('...')
#0 15.42 #4 {main}
#0 15.42   thrown in /app/src/Writer/MultiplePdfFilesWriter.php on line 76

wkhtmltopdf/wkhtmltopdf#3380 suggests parameters could be used to ignore these:

wkhtmltopdf  --load-error-handling ignore --load-media-error-handling ignore ...

Mention bearer token for Confluence auth tokens

We have a self-hosted Confluence and apparently my personal access token worked only when specified as bearer token. Should this be mentioned somehow as an example maybe? I spent a while fiddling around with basic auth and only realised that bearer tokens work by luck basically :)

Add unit tests + mutation tests

At the moment there is minimal testing; we should add unit tests and mutations for them.

Strip cache-busting parts of local image references

Just a suggestion, I didn't check yet if this is a common thing in Markdown. But for some reason we have references like

![basic example 1](images/datatables/basic-example-1.png?1)

and that breaks the tool with the following exception

[2024-02-26T17:25:15.046117+00:00] cli.DEBUG: [Roave\DocbookTool\Formatter\InlineExternalImages] Inlining image "/docs-src/book/images/datatables/basic-example-1.png?1" in page "datatables" [] []
PHP Warning:  file_get_contents(/docs-src/book/images/datatables/basic-example-1.png?1): Failed to open stream: No such file or directory in /app/vendor/thecodingmachine/safe/generated/filesystem.php on line 273
PHP Fatal error:  Uncaught Safe\Exceptions\FilesystemException: file_get_contents(/docs-src/book/images/datatables/basic-example-1.png?1): Failed to open stream: No such file or directory in /app/vendor/thecodingmachine/safe/generated/Exceptions/FilesystemException.php:9
Stack trace:
#0 /app/vendor/thecodingmachine/safe/generated/filesystem.php(276): Safe\Exceptions\FilesystemException::createFromPhpError()
#1 /app/src/Formatter/InlineExternalImages.php(46): Safe\file_get_contents()
#2 [internal function]: Roave\DocbookTool\Formatter\InlineExternalImages->Roave\DocbookTool\Formatter\{closure}()
#3 /app/src/Formatter/InlineExternalImages.php(35): preg_replace_callback()
#4 /app/src/Formatter/AggregatePageFormatter.php(22): Roave\DocbookTool\Formatter\InlineExternalImages->__invoke()
#5 [internal function]: Roave\DocbookTool\Formatter\AggregatePageFormatter->__invoke()
#6 /app/bin/docbook-tool.php(57): array_map()
#7 /app/bin/docbook-tool.php(63): Roave\DocbookTool\{closure}()
#8 /app/bin/docbook-tool(6): require_once('...')
#9 {main}
  thrown in /app/vendor/thecodingmachine/safe/generated/Exceptions/FilesystemException.php on line 9

if you don't mind I could open a PR that adds "support" for this.

If file is empty, determineTitleFromContent will crash

If the markdown file is completely empty, the determineTitleFromContent crashes, because strtok returns false presumably.

#0 5.983 [2022-10-07T15:43:58.149851+00:00] cli.INFO: Writing HTML output to /docs-package/index.html [] []
#0 5.998 PHP Fatal error:  Uncaught TypeError: str_starts_with(): Argument #1 ($haystack) must be of type string, bool given in /app/src/DocbookPage.php:70
#0 5.998 Stack trace:
#0 5.998 #0 /app/src/DocbookPage.php(70): str_starts_with()
#0 5.998 #1 /app/src/DocbookPage.php(83): Roave\DocbookTool\DocbookPage->determineTitleFromContent()
#0 5.998 #2 /app/vendor/twig/twig/src/Extension/CoreExtension.php(1607): Roave\DocbookTool\DocbookPage->title()
#0 5.998 #3 /app/vendor/twig/twig/src/Environment.php(358) : eval()'d code(226): twig_get_attribute()
#0 5.998 #4 /app/vendor/twig/twig/src/Template.php(394): __TwigTemplate_3767b10960c12aea038deadddc8f0951->doDisplay()
#0 5.998 #5 /app/vendor/twig/twig/src/Template.php(367): Twig\Template->displayWithErrorHandling()
#0 5.998 #6 /app/vendor/twig/twig/src/Template.php(379): Twig\Template->display()
#0 5.998 #7 /app/vendor/twig/twig/src/TemplateWrapper.php(40): Twig\Template->render()
#0 5.998 #8 /app/vendor/twig/twig/src/Environment.php(277): Twig\TemplateWrapper->render()

Check for invalid HTML

diff --git a/test/fixture/docbook/test.md b/test/fixture/docbook/test.md
index fe86218..0d4a9f7 100644
--- a/test/fixture/docbook/test.md
+++ b/test/fixture/docbook/test.md
@@ -22,6 +22,10 @@ This is some markdown
 }
 ```
 
+## Test invalid HTML
+
+This particular part <has> some <invalid> HTML tags. This breaks Confluence!
+
 ## Subtitle
 
 Links [here](https://www.google.com). **Bold**, _italic_, ~~strikethrough~~, `inline code`.

Applying this diff to the test fixtures will generate invalid HTML. We should add a check after generating the HTML that it is valid.

To reproduce:

Apply the above addition to the test fixture
Run make test-output to generate the build/index.html
Upload/copy the contents of build/index.html into the W3 Validator

Observation:

Perhaps there is a validator library to do this, so we can pass it through there before proceeding?

Detect line endings

At the moment, the tool crashes if it encounters a file with CRLF (\r\n) line endings. The error is misleading (it says it can't find the title of the page), but we should detect these line endings being used and reject it with a more helpful error message.

Expected behaviour

An error message saying that CRLF (\r\n) are not supported. Suggested remediation can be to use dos2unix <filename>.

Actual behaviour

[2021-10-06T16:23:13.821568+00:00] cli.INFO: Writing HTML output to /docs-package/index.html [] []
PHP Fatal error:  Uncaught RuntimeException: First line of markdown file <filename> did not start with "# "... in /app/src/DocbookPage.php:71
Stack trace:
#0 /app/src/DocbookPage.php(83): Roave\DocbookTool\DocbookPage->determineTitleFromContent()
#1 /app/vendor/twig/twig/src/Extension/CoreExtension.php(1541): Roave\DocbookTool\DocbookPage->title()
#2 /app/vendor/twig/twig/src/Environment.php(358) : eval()'d code(226): twig_get_attribute()
#3 /app/vendor/twig/twig/src/Template.php(394): __TwigTemplate_2904b74a3842d9b80ea20a5768e5df794dec35bfd57ea5620521eb0ac55c9c23->doDisplay()
#4 /app/vendor/twig/twig/src/Template.php(367): Twig\Template->displayWithErrorHandling()
#5 /app/vendor/twig/twig/src/Template.php(379): Twig\Template->display()
#6 /app/vendor/twig/twig/src/TemplateWrapper.php(40): Twig\Template->render()
#7 /app/vendor/twig/twig/src/Environment.php(277): Twig\TemplateWrapper->render()
#8 /app/src/Writer/SingleStaticHtmlWriter.php(34): Twig\Environment->render()
#9 /app/src/WriteAllTheOutputs.php(22): Roave\DocbookTool\Writer\SingleStaticHtmlWriter->__invoke()
#10 /app/bin/docbook-tool.php(55): Roave\DocbookTool\WriteAllTheOutputs->__invoke()
#11 /app/bin/docbook-tool.php(59): Roave\DocbookTool\{closure}()
#12 /app/bin/docbook-tool(6): require_once('...')
#13 {main}

Next Twig\Error\RuntimeError: An exception has been thrown during the rendering of a template ("First line of markdown file <filename> did not start with "# "..."). in /docs-src/templates/online.twig:179
Stack trace:
#0 /app/vendor/twig/twig/src/Template.php(367): Twig\Template->displayWithErrorHandling()
#1 /app/vendor/twig/twig/src/Template.php(379): Twig\Template->display()
#2 /app/vendor/twig/twig/src/TemplateWrapper.php(40): Twig\Template->render()
#3 /app/vendor/twig/twig/src/Environment.php(277): Twig\TemplateWrapper->render()
#4 /app/src/Writer/SingleStaticHtmlWriter.php(34): Twig\Environment->render()
#5 /app/src/WriteAllTheOutputs.php(22): Roave\DocbookTool\Writer\SingleStaticHtmlWriter->__invoke()
#6 /app/bin/docbook-tool.php(55): Roave\DocbookTool\WriteAllTheOutputs->__invoke()
#7 /app/bin/docbook-tool.php(59): Roave\DocbookTool\{closure}()
#8 /app/bin/docbook-tool(6): require_once('...')
#9 {main}
  thrown in /docs-src/templates/online.twig on line 179
make: *** [Makefile:64: build-docs] Error 255

Ability to embed examples from OAS

Add an ability to embed an example request or response - or a fragment thereof - from an OpenAPI spec.

Add a Twig template value for "generated time"

In generated docs, it would be useful to have a generated date/time, e.g.

Add support for SVGs

The current image inlining support introduced in #251 does not support SVGs, because GD does not understand SVGs.

Would be nice if we can support SVGs eventually too, although we don't need this immediately.

Automate PlantUML JAR download

Since the JAR filenames do not appear to be deterministic, manual update of the bundled JAR is currently needed.

You can download the latest JAR from https://github.com/plantuml/plantuml/releases.

We should figure out a way of automating the download of the JAR.

Allow relative links to be used

Currently relative links do not work with the SingleStaticHtmlWriter (this problem could exist in other writers as well). Support for relative links would be beneficial.

Add Dockerfile so we can publish a base image with this in

A handy way of running this (especially as we have external dependencies such as wkhtmltopdf and java) is to publish a Docker image (to be automated in pipeline), so something like:

FROM ubuntu:whatever

RUN <all the stuff for apt dependencies>

ADD <our source code> /app

RUN composer install etc

ENV <sensible defaults for our required/optional env vars>

ENTRYPOINT bin/docbook-tool
CMD [--html --pdf --confluence]

Then consumers would base their images off ours:

FROM roave/docbook-tool:latest

ADD ./docs/book /docs
ADD ./features /features
ADD ./docs/template /templates

CMD [--html --pdf]

Too large images result in blank page

Given an image that has a byte size over a certain amount the page renders as a blank string. You can see a demonstration of this in the fix-large-inline-image branch.

A couple things to note:

I was able to see this fail with images as small as 256kb. I cannot say for certain whether this is the actual limit, only that anything >=256kb demonstrated this problem. It is possible that a number less than 256kb would also have this problem. That has not been confirmed.
The problem is happening in the MarkdownToHml formatter. Before the Markdown parsing the $page->content() contains the proper image and after parsing the $page->content() is a blank string.

Use DOM manipulation when replacing HTML elementattributes

Originally posted by @Ocramius in #376 (comment)

We should look into getting rid of string replacements, and using DOM manipulation for HTML replacements, where possible.

This is NOT a security issue because this tool is intended for use on repository sources, NOT user input.

Add PDF comparisons to improve testing

At the moment we just assert the PDF is generated. We should also do a visual comparison to check the output matches. Note, just doing a binary diff is not sufficient, since the content DOES change every time (even if it is only /CreationDate etc.)

Might be worth taking a look at rst2pdf technique https://github.com/rst2pdf/rst2pdf/blob/master/rst2pdf/tests/compare_pdf.sh for PDF comparison

Missing license

Auto-release job failing?

https://github.com/Roave/DocbookTool/actions/runs/2947324152


In AssertException.php line 32:
                                        
  [Psl\Type\Exception\AssertException]  
  Expected "200", got "int".            
                                        

Exception trace:
  at /app/vendor/azjezz/psl/src/Psl/Type/Exception/AssertException.php:32
 Psl\Type\Exception\AssertException::withValue() at /app/vendor/azjezz/psl/src/Psl/Type/Internal/LiteralScalarType.php:109
 Psl\Type\Internal\LiteralScalarType->assert() at /app/src/Github/Api/GraphQL/RunGraphQLQuery.php:60
 Laminas\AutomaticReleases\Github\Api\GraphQL\RunGraphQLQuery->__invoke() at /app/src/Github/Api/GraphQL/Query/GetMilestoneFirst100IssuesAndPullRequests.php:85
 Laminas\AutomaticReleases\Github\Api\GraphQL\Query\GetMilestoneFirst100IssuesAndPullRequests->__invoke() at /app/src/Application/Command/ReleaseCommand.php:79
 Laminas\AutomaticReleases\Application\Command\ReleaseCommand->execute() at /app/vendor/symfony/console/Command/Command.php:298
 Symfony\Component\Console\Command\Command->run() at /app/vendor/symfony/console/Application.php:1024
 Symfony\Component\Console\Application->doRunCommand() at /app/vendor/symfony/console/Application.php:299
 Symfony\Component\Console\Application->doRun() at /app/vendor/symfony/console/Application.php:171
 Symfony\Component\Console\Application->run() at /app/bin/console.php:170
 Laminas\AutomaticReleases\WebApplication\{closure}() at /app/bin/console.php:171

laminas:automatic-releases:release

Looks like it can't query the GraphQL API of Github here (to get the issue list)

Confluence feature - Replace relative links to markdown within scope with confluence page links (if available)

Given a link like [Reporting Overview](./another-file.md), assuming another-file.md also has a confluencePageId, we could replace the link with a Confluence link, e.g. https://<confluence-url>/pages/viewpage.action?pageId=<linkedConfluencePageId>.

Image path should be resolved relative to the file including it

Given the following:

/docs-src/book
  /subdir
    some-image.png
    docs.md

If docs.md contains:

![the image](some-image.png)

It will fail; the path currently needs to be relative to the content root, so this would work:

![the image](subdir/some-image.png)

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Update all non-major dependencies (composer, docker/dockerfile, guzzlehttp/guzzle, guzzlehttp/psr7, marked, monolog/monolog, node, php, phpunit/phpunit, psalm/plugin-phpunit, symfony/yaml, twig/twig, vimeo/psalm)
Update dependency marked to v14
Update dependency phpunit/phpunit to v11
Update ubuntu Docker tag to v24
Lock file maintenance
Click on this checkbox to rebase all open PRs at once

Detected dependencies

composer

composer.json

php ~8.3.3

guzzlehttp/guzzle ^7.8.1

guzzlehttp/psr7 ^2.6.2

jasny/twig-extensions ^1.3

michelf/php-markdown ^2.0

monolog/monolog ^3.5.0

psr/log ^3.0.0

symfony/yaml ^7.0.3

thecodingmachine/safe ^2.5.0

twig/twig ^3.8.0

webmozart/assert ^1.11

doctrine/coding-standard ^12.0.0

phpunit/phpunit ^10.5.11

psalm/plugin-phpunit ^0.18.4

vimeo/psalm ^5.22.2

dockerfile

Dockerfile

docker/dockerfile 1.6

composer 2.6.6

node 22.4.0

ubuntu 22.04

github-actions

.github/workflows/ci.yml

docker/setup-buildx-action v3

docker/build-push-action v6

.github/workflows/publish-docker-image-to-github-registry.yml

actions/checkout v4

docker/metadata-action v5

docker/setup-qemu-action v3

docker/setup-buildx-action v3

docker/login-action v3

docker/build-push-action v6

.github/workflows/release-on-milestone-closed-triggering-release-event.yml

actions/checkout v4

laminas/automatic-releases v1

laminas/automatic-releases v1

laminas/automatic-releases v1

laminas/automatic-releases v1

npm

package.json

redoc-cli ^0.13.21

marked ^13.0.1

Check this box to trigger a request for Renovate to run again on this repository

Increase logging verbosity for builds

Debugging issues in this setup is quite complex at the moment: having more logging before/after operations such as wkhtmltopdf transformations, plantuml chunk rendering, etc would really be helpful.

Add integration test for Confluence

We do some basic unit testing introduced in #253 of the ConfluenceWriter, but this doesn't actually connect to a Confluence instance and verify anything. We should add some kind of integration test (which would need a running Confluence instance).

Support embedding local images in generated PDFs

While working with the tool today, I noticed that Markdown containing local imports will not work:

![An Image](./file.png)

This is because we are missing two things:

--enable-local-file-access to allow file:// usage in WkHtmlToPdf

see wkhtmltopdf/wkhtmltopdf#2660 (comment)

see wkhtmltopdf/wkhtmltopdf#4536

DocbookTool/src/Writer/MultiplePdfFilesWriter.php

Lines 54 to 66 in 00942cc

    
           exec( 
        
               escapeshellcmd(implode( 
        
                   ' ', 
        
                   [ 
        
                       $this->locationOfWkhtmltopdf, 
        
                       '-q', 
        
                       $tmpHtmlFile, 
        
                       $pdfFile, 
        
                   ], 
        
               )) . ' 2>&1', 
        
               $output, 
        
               $exitCode, 
        
           );

the relative files need to be dragged in the build workspace: currently, we copy just the HTML file to be rendered: