oxygenxml / oxygen-docbook-validate-check-completeness

Oxygen XML plugin which allows validation of DocBook files reporting broken images, internal or external links.

Oxygen XML Web Author Monitoring plugin

This plugin is used to provide monitoring information from a running Web Author instance.

Web Author collects various metrics about its runtime behavior. It can be configure to report these metrics in a number of ways.

In the browser

To access the information you should access one of the following URLs Note that you have to be logged in the Administration Page when visiting these URLs.

• http://host:port/oxygen-xml-web-author/plugins-dispatcher/monitoring/threads - to get thread dumps of all threads
• http://host:port/oxygen-xml-web-author/plugins-dispatcher/monitoring/metrics - to get various metrics about Web Author

In the server logs

By default, the monitoring information is written in JSON format in the server logs every minute. You can use various tools to parse this information and present a graphical dashboard.

To a Graphite server

You have to set the GRAPHITE_SERVER environment variable in order for Web Author to send monitoring information to this server.

To AWS CloudWatch

You have to set the following environment variables:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_DEFAULT_REGION
• https_proxy - in cases you use one

By extending this plugin you can send the collected metrics also to other metrics servers.

Custom configuration

The name of the logger used by this plugin to log the above information is NO_LAYOUT_LOGGER.METRICS

WebAuthor default configuration defines a logger named NO_LAYOUT_LOGGER. This logger has a console appender with no layout, so any child logger (with a naming scheme NO_LAYOUT_LOGGER.<LOGGER_NAME>) will log only the messages to stdout.

To override this default configuration, a NO_LAYOUT_LOGGER.METRICS logger must be defined explicitly in WebAuthor logging configuration.

Copyright and License

Copyright 2018 Syncro Soft SRL.

This project is licensed under Apache License 2.0

oxygen-docbook-validate-check-completeness's Issues

Generate a resource hierarchy

Just logging this here based on what one of our end users needed. But we'll probably not approach it for now.
The dialog used to provide settings for the validation could maybe have another setting, something like "Generate resource hierarchy report" which could generate some kind of hierarchy tree showing something like:

       topDB.xml
                         -xi:includes:
                                           module1.xml
                                            module2.xml
                          -images:
                                          image1.png
                                          image2.png
                         -internal links:
                                          idref1
                                          idref2
                          - external links:
                                         http:/...........
                                         http:/...........

and each module.xml could also be expanded and show a similar hierarchy....
The hierarchy could for example be exported to HTML and it would be built only with the content which is not excluded by the profiling set in the settings.
Problems: right now we do not quite know where an xi:include occurs in the XML as xi:includes are transparent, the validation might be done with multiple profiling settings, resulting in multiple reports being generated...

Persist cross-session settings stored in the dialog

For example persist check boxes states, list of URLs, etc.
API to use:

 		WSOptionsStorage optionsStorage = PluginWorkspaceProvider.getPluginWorkspace().getOptionsStorage();

Problem when expand external links.

The links like : "rdar://problem/21312183", create problems when the external links node is expanded

Pressing F1 in the Validation dialog could take end user to the project's readme

If you override this method:

     ro.sync.exml.workspace.api.standalone.ui.OKCancelDialog.getHelpPageID()

and provide on it the URL to the GitHub repository the dialog should take you there when pressing F1 inside it.

Code review issues

Problems found during code review:

• I do not think that the "com.oxygenxml.docbook.checker.DocBookCheckerOxygen" should exist as it does not bring any extra logic to the base class.
• com.oxygenxml.docbook.checker.OxygenInteractor.setButtonsEnable(boolean) The "state" param is not documented
• OxygenSourceDescription - use "enum" instead of constant strings.
• Spell check problems with field "com.oxygenxml.docbook.checker.OxygenSourceDescription.parrentFrame" and getter and setter methods. Maybe this particular field should not be in this "OxygenSourceDescription" class. Maybe the "OxygenInteractor" could instead contain a getter for the parent frame.
• com.oxygenxml.docbook.checker.OxygenInteractor.setButtonsEnable(boolean) Maybe you could rename the method to "setOperationInProgress(....)" and you could enable/disable the actions on it's callback like you do now.
• Maybe also rename OxygenInteractor to "ApplicationInteractor". Maybe same for "OxygenSourceDescription"
• ValidationWorker - constructor is not documented, neither are all other methods
• Maybe rename "com.oxygenxml.docbook.checker.ValidationWorkerInteractor.isSetIsCancelled()" to "isCancelled"
• CheckBoxTree - variables with friendly access but used only in current class, you can make them private. Add more javadoc to variables, methods, document each parameter used in method. Same for CheckBoxTreeModel, CheckedNode. Also in CheckedNode you can make variables final to precisely avoid them being modified by any other means than the constructor. Also in ConditionValue no Javadoc.
• ConditionValue, the equals(Object) should compare the internal values of both objects instead of relying on the hashcode(). Because it's possible that for two different strings the hashcode() returns the same integer.
• Maybe rename MapConvertor->ConditionValueUtil because it has a very generic name. Not all params are documented in Javadoc

Xi:included documents which do not contain internal/external links are not show in report

Feedback from one of our end users:

In my test, it shows only a subset of the xincluded files, apparently only those that contain external or internal links. Wouldn’t it make more sense to list all xincluded files, perhaps distinguishing those files that are included in the current document from those that are included in files that are included, but not directly depended upon.

Allow end user to specify the name of the used Docbook document type

Let's say I choose to validate using all the Docbook profiling condition sets. Right now all profiling condition sets which contain the "Docbook" name in them are gathered. But maybe the end user has defined an extension of the "Docbook" framework which has a different name.
Ideally in the dialog in the part where you choose the profiling options end users could have a combo box in which they would either pick Docbook 4 and 5 or write there the name of their Docbook extension.
If this is implemented, we can remove the similar combo box from the dialog which allows you to add a new profiling attribute to the table.

Custom icons necessary for project

1. An icon which is presented when the action is mounted on the toolbar.
2. An icon which is a link to a preferences page.

Report broken external links

For Docbook 4 and 5 the validation action should report links to web sites which do not exist or if the web site cannot serve that specific web page.
Samples available in folder "test-samples\broken-external-link".

Move contributed menu item further up in Project tree contextual menu.

To replace docbookChekerMenuItem in contextual menu,.

"Check" button not enabled when invoked from Project

Discovered by our QA, right click a folder in the project, show the checker dialog, the "Check" dialog is not enabled, you need to change between the scope radio buttons in order for it to become active.

Set proper title to contributed toolbar

If I right click the toolbar area I can see this toolbar added as "DocbookValidationToolbar". Maybe we can set a proper name to it (similar but with spaces).

Contextual menu action appears even when right clicking in DITA files

Marius from QA reported this issue.
I guess we could use the "ro.sync.exml.workspace.api.editor.WSEditorBase.getDocumentTypeInformation()" API and only add the contextual menu action if the XML has some document type information attached and if the name of the document type contains "docbook" in it. Even if we are wrong, the end user can continue using it from the toolbar so no big deal...

Report broken image links

For Docbook 4 and 5 the validation action should report links to images which to not exist.
Samples available in folder "test-samples\broken-image".

Allow end user to specify profiling filters

Docbook has quite a few profiling attributes like arch, os, condition.
We need in the UI a table in which the end user needs to specify inclusions, specify profiling attributes and their values which will be included in the output.
Something like:

 Include only elements containing profiling attributes:
    os | X;Y
    audience | Y 

For links to IDs in filtered out elements we need to have special messages like "Reference to ID X defined in filtered out content".

UI changes in checker configuration dialog

Feedback from our QA team:

1. The "Files URLs" header of the table can be hidden, there is a label above it explaining the same thing.
2. Move "From all available condition sets" radio button before the "From condition sets"
3. Check images => Check for references to missing images
4. Check external links => Check for broken links to external web sites
5. Check internal links => Check for broken links to internal targets
6. Use profiling conditions(filters) => add extra space before (

Improve readme.md

State the link to the add-on.xml, how the end users should install it, also list the set of features that the plugin has.

Validation should work even if we use xi:include in the documents

Samples in "test-samples\xi-include"

False positive from image check when xincluded file is not in the same dir as including file

For details see: #22

Mount contextual project action

We could mount the same action on the project.
In Oxygen 19.1 we will have API to add a popup menu customizer on the project view and to take from the project view the list of selected resources:

ro.sync.exml.workspace.api.standalone.StandalonePluginWorkspace.getProjectManager()

I created a 19.1 beta kit which should have this API functionality included:

https://www.oxygenxml.com/userFiles/oxygen-64bit-19.1-beta-accessibility-take-2.exe

I'm not sure how to proceed in order to have the plugin still work with older Oxygen versions.
You should try using Java reflection for the API. It seems you can also implement interfaces (like the interface which needs to be implemented to receive popup menu customizer callbacks) using java.lang.reflect.Proxy.

Translate plugin strings

Allow for plugin's labels and UI strings to be translated in multiple languages.
API to use:

 		WSOptionsStorage optionsStorage = PluginWorkspaceProvider.getPluginWorkspace().getOptionsStorage();
	 ((StandalonePluginWorkspace)PluginWorkspaceProvider.getPluginWorkspace()).getResourceBundle().getMessage(Tags.CHECK_FILE_KEY);

Decide level for all reported problems

Right now all problems seems to be reported as warnings. Maybe we can report them as errors instead.
If there are problems which completely break processing (for example XML is not wellformed) we should report them as fatal errors.
Also sometimes I get a "Validation successful" status although there are two warnings found.

Integrate validation in Oxygen

Add an action on the main "Docbook" menu which allows end user to validate the current opened Docbook document.
I would recommend this API:

       ro.sync.exml.workspace.api.standalone.StandalonePluginWorkspace.addMenusAndToolbarsContributorCustomizer(MenusAndToolbarsContributorCustomizer)

Validate also references from Docbook Assembly files

For example I may have a Docbook topic named "t1.xml":

        <?xml version="1.0" encoding="UTF-8"?>
        <?xml-model href="http://docbook.org/xml/5.1/rng/docbook.rng" schematypens="http://relaxng.org/ns/structure/1.0"?>
        <?xml-model href="http://docbook.org/xml/5.1/sch/docbook.sch" type="application/xml" schematypens="http://purl.oclc.org/dsdl/schematron"?>
        <topic xml:id="t1" xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink" version="5.1" >
            <info> <title>prob1-topic1</title> </info>
            <para>Paragraph contained in topic #1.</para>
        </topic>

and a Docbook Assembly file "assembly.xml" like:

        <?xml version="1.0" encoding="UTF-8"?>
        <?xml-model href="http://docbook.org/xml/5.1/rng/assembly.rng" schematypens="http://relaxng.org/ns/structure/1.0"?>
        <?xml-model href="http://docbook.org/xml/5.1/sch/assembly.sch" type="application/xml" schematypens="http://purl.oclc.org/dsdl/schematron"?>
        <assembly version="5.1" xmlns="http://docbook.org/ns/docbook">
            <resources>
                <resource href="t1.xml" xml:id="topic1"/>
            </resources>
            
            <structure renderas="book">
                <merge><title>Book title of sample assembly of topics</title></merge>
                
                <module renderas="chapter" xml:id="chapter1">
                    <merge><title>Title of first chapter</title></merge>
                    <module contentonly="true" resourceref="topic1" omittitles="true"/>
                </module>
            </structure>
        </assembly>

The validation should report when the "t1.xml" does not exist. It should also try to validate the "t1.xml" or any other XML referenced in the assembly.

Report broken internal links

For Docbook 4 and 5 the validation action should report links to internal IDs which do not exist or if the web site cannot serve that specific web page.
Samples available in folder "test-samples\broken-internal-link".

Report profiling attribute values conflicts

A checkbox called "Report profiling attribute values which conflict with profiling preferences" maybe followed by a button which shows the profiling preferences.
When checked, while validating, if for the profiling attributes defined in the preferences we find in the XML extra values which are not defined in the preferences, we report warnings in the same results tab with the rest of the reported problems.

Contribute plugin as add-on update

Create an add-on update site for the plugin.
Example:

http://blog.oxygenxml.com/2017/01/your-first-oxygen-add-on.html

Create GUI to allow user to select the Docbook document which will be validated

A dialog allowing the end user to select an initial URL. Maybe a couple of settings, for example if the validation should report broken links to web sites, then the dialog should start a thread, maybe show a small progress indicator, then at the end create some kind of report.

ClassCastException

Marius from QA attempted to use the completeness checker on the "samples/docbook/v5" folder from the Oxygen samples project:

    Exception in thread "AWT-EventQueue-0" java.lang.ClassCastException: java.lang.String cannot be cast to java.net.URL
    at com.oxygenxml.docbook.checker.gui.SelectFilesPanel.getFilesFromTable(SelectFilesPanel.java:214)
    at com.oxygenxml.docbook.checker.gui.DocBookCheckerDialog.doOK(DocBookCheckerDialog.java:183)
    at ro.sync.exml.workspace.api.standalone.ui.OKCancelDialog$5.actionPerformed(Unknown Source)
    at javax.swing.AbstractButton.fireActionPerformed(Unknown Source)
    at javax.swing.AbstractButton$Handler.actionPerformed(Unknown Source)
    at javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source)
    at javax.swing.DefaultButtonModel.setPressed(Unknown Source)
    at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(Unknown Source)
    at java.awt.Component.processMouseEvent(Unknown Source)
    at javax.swing.JComponent.processMouseEvent(Unknown Source)
    at java.awt.Component.processEvent(Unknown Source)
    at java.awt.Container.processEvent(Unknown Source)
    at java.awt.Component.dispatchEventImpl(Unknown Source)
    at java.awt.Container.dispatchEventImpl(Unknown Source)
    at java.awt.Component.dispatchEvent(Unknown Source)
    at java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)

Properly state in the addon.xml the minimum supported Oxygen version

The newest API we are using:
ro.sync.exml.workspace.api.PluginWorkspace.getResultsManager()

was added in Oxygen 19.0 and as it is indispensable probably Oxygen 19.0 is the lowest supported version.