svenevs / exhale Goto Github PK

Hi!

I've created a small extension which monkey patches exhale to support multiple doxygen projects.

Would you be interested in incorporating this into exhale as a default feature? If so I'll send you a pull request with the changes required.

Thanks!

Re-investigate the rules, 95% sure it still has to be outside of conf.py because of how sphinx evaluates it.

I try to use exhale for a project, but the build fails every time with the same error. To ensure that my sphinx and exhale configurations aren't the issue, I've cloned the exhale-companion repo and tried to build that. If I build it with make html or sphinx-build -b html . _build everything runs smooth until the finish of the XML generation (after lookup cache used 108/65536 hits=433 misses=110). Then this error occurs:

Extension Error: Unable to execute 'doxygen': [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\...\\UserName\\AppData\\Local\\Temp\\exhale_launched_doxygen_buff6msnj3_n'

I tried the same on linux and it worked right away. Currently the following versions are installed on the windows host: spinx==1.7.6, lxml==4.2.3, exhale==0.1.8, breathe==4.9.1, sphinx-rtd-theme==0.4.1, sphinxcontrib-apidoc==0.2.1 and -matlabdomain==0.3.3

(Being Exhale newbie, I'm not sure where the actual problem may be, so I initially posted this as AttributeError: 'str' object has no attribute 'get_display_string' to sphinx-users list .)

I'm trying to build Boost.GIL (sphinx-exhale branch) documentation. Full logs attached below.

Doxygen and Exhale seems to run fine, as I can see this logged:

[+] Exhale: doxygen ran successfully in 11.75 seconds.
[~] Exhale: adding tree view css / javascript.
[+] Exhale: added tree view css / javascript.
[~] Exhale: parsing Doxygen XML.
[+] Exhale: finished parsing Doxygen XML in 35.71 seconds.
[~] Exhale: generating reStructuredText documents.
...
[+] Exhale: generated reStructuredText documents in 23.86 seconds.

Then, Sphinx is taking over

building [html]: targets for 4515 source files that are out of date
...

and it is failing with a very long list of warnings like

WARNING: Error when parsing function declaration.
WARNING: Duplicate explicit target name
WARNING: Duplicate ID
WARNING: Duplicate declaration.
WARNING: doxygenfunction: Unable to resolve multiple matches for function ...

Eventually failing with

Exception occurred:
  File "/mnt/d/boost.wsl/libs/gil/.venv/lib/python3.6/site-packages/sphinx/domains/cpp.py", line 635, in <lambda>
    return self._stringify(lambda ast: ast.get_display_string())
AttributeError: 'str' object has no attribute 'get_display_string'

I find it difficult to figure out where is the problem.

@stefanseefeld, my team mate at GIL, posted suggestion to our mailing list, https://lists.boost.org/boost-gil/2018/11/0116.php:

Just a shot in the dark: duplicate IDs sounds less like a sphinx bug,
but a bug with the exhale extension, which generates content
that the sphinx post-processing step then can't consume.

Is this suggestion pinpointing the issue?

I'll greatly appreciate any help.

Logs

• sphinx-build-full-output.log
• sphinx-err-29i0rtgw.log

The logs are also available at https://gist.github.com/mloskot/ad6f79b9c5fd15027467e9a55429b52et

It is possible to get the graphviz code generated from doxygen and you can use graphviz in sphinx. Needs more investigation, but this would be a really nice feature. Maybe we can even filter the generated ones to make better different graphs?

Any input welcome, this is a wish-list item. Hopefully build times won't explode.

Add either a section to the README.rst or a new file with all the info needed to start contributing, i.e., which code formatting are you using, which linter, the testing process, etc.

I initially thought it was more appropriate to have this be tracked like conf.py. It's kind of a pain in terms of maintaining projects using exhale. Package it up for more convenience.

1. Decide once and for all whether you are going to keep everything in one file (probably not necessary anymore).
   • I hate the packaging system though.
2. Start using releases via tags.
3. Going to need to create some sort of autotool so that people who pip install exhale can still use the treeView stuff.
   • Pull a breathe and wrap the sphinx-quickstart tool? Sounds like a lot of work...

v0.2.0 broke the namespaced union: https://my-favorite-documentation-test.readthedocs.io/en/alabaster/api/library_root.html

api/unabridged_api.rst:157: WARNING: toctree contains reference to nonexisting document 'api/unionarbitrary_1_1NamespacedUnion'
api/unabridged_api.rst:162: WARNING: toctree contains reference to nonexisting document 'api/unionarbitrary_1_1nested_1_1NestedNamespacedUnion'
exhale-companion/docs/api/unabridged_api.rst:157: WARNING: toctree contains reference to nonexisting document 'api/unionarbitrary_1_1NamespacedUnion'
exhale-companion/docs/api/unabridged_api.rst:162: WARNING: toctree contains reference to nonexisting document 'api/unionarbitrary_1_1nested_1_1NestedNamespacedUnion'

Exception occurred:
  File "/opt/spack/opt/spack/linux-fedora25-x86_64/clang-4.0.0/python-3.6.4-uc6nu4vwxopos2avdooqutdoloypy4hw/lib/python3.6/site-packages/breathe/project.py", line 303, in retrieve_project_info_for_auto
    return self.project_info_for_auto_store[name]
KeyError: 'auto'

Currently, the programlisting is just transformed into a raw .. code-block::. This is better than nothing, but

1. Explicitly parsing the programlisting for a given refid is easy (and already partially implemented).
2. Figuring out how to write the output is the hard part.
   • Pygments is already setup for Sphinx, need to resolve generating links and highlighting the code.
   • Make sure users set primary_domain in conf.py.
   • Allow extension mappings to sphinx lexers (yet another config variable).

Ideally it would be similar to the Python viewcode extension, but it will be a challenge. This one is quite far off in the future.

You've been using stack.nl/~dimitri/doxygen which has experienced numerous outages, but apparently doxygen.org (which redirects to doxygen.nl) is still live right now. So don't be hasty next time...use the right website....

I'm using Doxygen to generate documentation from Verilog with Exhale. This seems to mostly work pretty well except here -> https://github.com/svenevs/exhale/blob/master/exhale/graph.py#L2534

            if len(f.program_listing) > 0:
                include_program_listing = True
                full_program_listing = '.. code-block:: cpp\n\n'

Exhale is forcing the program listing code highlighting to be C++. This means that Sphinx's gets very grumpy when it tries to highlight this code block and Verilog is not valid C++ :-).

It would be good if we could use one of the following for providing the code block language parameters;

• A config setting (maybe allow a function which returns the value?)
• Looking up the language via file extension?
• Read the language data from Doxygen's XML somehow?

Thoughts?

When having the verbose output enabled I ran into an issue in https://github.com/svenevs/exhale/blob/master/exhale/graph.py#L458. self.kind was interface but this wasn't present in fmt_spec so fmt_spec[self.kind] was giving a KeyError. Better check for that since it's "only" a debug message.

the following branch in my requirements.txt

git+https://github.com/svenevs/exhale.git@fix/unabridged-bury-files-directories

Dumps the following error

Collecting git+https://github.com/svenevs/exhale.git@fix/unabridged-bury-files-directories (from -r docs/requirements.txt (line 3))
  Cloning https://github.com/svenevs/exhale.git (to revision fix/unabridged-bury-files-directories) to /tmp/pip-req-build-6fnxedpx
  Running command git clone -q https://github.com/svenevs/exhale.git /tmp/pip-req-build-6fnxedpx
  WARNING: Did not find branch or tag 'fix/unabridged-bury-files-directories', assuming revision or ref.
  Running command git checkout -q fix/unabridged-bury-files-directories
  error: pathspec 'fix/unabridged-bury-files-directories' did not match any file(s) known to git.
ERROR: Command errored out with exit status 1: git checkout -q fix/unabridged-bury-files-directories Check the logs for full command output.

I assume this was merged? Should I point to master or is there a specific tag that I should be using now? Thanks

In the usage section in Start to finish for Read the Docs

In step 6 It tells you to check the option :
install your project inside a virtualenv using setup.py install

This is no more required to build the doc on RTD

Is it possible to exclude Directories and Files sections from the unabridged api?

Logfiles are hard to read, alwaysColorize=False is a lie (though the docs made that clear xD)

I refuse to change certain aspects of my style, but should conform to other standards I suppose 😉

Hi there,

I'm just using Exhale to document a small c++ library, the tool works pretty well except for overloaded functions. When it tries to parse them, it does not seem to be able to choose the right definition, as it produces the following warning message.

WARNING: doxygenfunction: Unable to resolve multiple matches for function "tsa::dimensionality::PAA" with arguments () in doxygen xml output for project "TSA" from directory: /grijander/repositories/fistro-lib/build/doc/doxygen/xml. Potential matches: - array<Point> fistro::PAA(array<Point>, int) - std::vector<Point> fistro::PAA(std::vector<Point>, int)

Actually, it includes both function definitions in the list of available functions, but If you click on any of them, I get a yellow/orange warning box with the same text as above.

Do you know any work-around to overcome this problem.
Thanks in advance, you have done a great work.
Antonio Vilches.

When functions with the same name but different signatures are present, breathe is not given enough information to disambiguate.

https://github.com/wjakob/nanogui/blob/f61c407ea4cb336aefcdd82202c6e09ff21b9e79/include/nanogui/common.h#L474-L476

and

https://github.com/wjakob/nanogui/blob/f61c407ea4cb336aefcdd82202c6e09ff21b9e79/include/nanogui/common.h#L493-L495

produce two separate nodes / reStructuredText documents, but .. doxygenfunction:: file_dialog is generated, which is incorrect and makes the docs for those members fail.

• Doxyfile is a List[Union[str, pathlib.Path]] (don't require to be in same directory as conf.py).
• Customizable doxygen launching (e.g., they need to cd somewhere other than where Doxyfile is).

It would be nice to include some unit tests. I'm not exactly sure how these tests would be performed or validate though. I will have to think a lot more about how this would go down.

I'm not sure if this is an issue with breathe or exhale but by default private member classes (which are definitely not part of the API) are displayed in the API documentation, i.e. without enabling :private-members:. To me this seems to be a bug, is there currently a way to disable this behaviour?

The sphinx_rtd_theme is what I primarily designed this around. There is no reason not to be able to have the links for each generated page link back to an actual file hosted online. Currently, generated rst docs are linked to non-existent URLs since the file is not (and is not supposed to be) tracked.

1. Ask for more information in generate
   • base_url
   • If None, disable
2. Set the right link for each page
3. See also: http://docs.readthedocs.io/en/latest/vcs.html

Functioning hack in the FAQ.

Or only functioning locally? A local build will do the redirect, but apparently still not on the actual RTD site...

Duh. That future TODO about merging file, namespace, and other into one write method needs to happen sooner. Didn't add the meta for all pages. Will fix soon ish. Fixed in v0.1.5

Whoops... I wanted to make a v1.0.0 project...

It looks like using @todo in Doxygen causes this warning to pop up during build:

(!) Unabridged API: unexpected kind 'page' (IGNORED)

The todo entry is then not visible anywhere in the final documentation.

How to reproduce

    /**
     * @todo This TODO directive causes <tt>(!) Unabridged API: unexpected kind 'page' (IGNORED)</tt>
     */
    class incomplete_class {
    };

I have a separate project where I collected this and other issues (still looking to dispatch them to the right repository); you can also check that out to reproduce.

Doxygen HTML output

Exhale HTML output

Using this containmentFolder

"containmentFolder": "./apireference/cpp",

fails with the following error:

The given containmentFolder [/home/stian/stian_sphinx_testing/apireference/cpp] must be a SUBDIRECTORY of [/home/stian/stian_sphinx_testing].

Debugging the code, I see the following values for _one, _two and _three in configs.py:

_one=False _two=False _three=True

Exhale version 0.2.3

Hi Stephen,

I think I've found a bug and a solution. I get the error
"unable to create the root api body: local variable 'has_nested_children' referenced before assignment. I solved this by moving line 1020 in exhale.py to after line 999.

Amazing work by the way, been waiting for something this intuitive that does exactly what I want for a while.

Thanks,

Sam

I'm encountering this error when processing a rather complex API on the latest exhale version via pip on Python 3. I'm quite sure this has something to do with the Doxygen XML input. Do you have a tip where I can begin to search for the problem in the data or what to do to before I start to debug in the code myself?

Below the stack trace:

(!) Exception caught while parsing:Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/exhale/deploy.py", line 390, in explode
    textRoot.parse()
  File "/usr/local/lib/python3.6/dist-packages/exhale/graph.py", line 979, in parse
    self.discoverAllNodes()
  File "/usr/local/lib/python3.6/dist-packages/exhale/graph.py", line 1201, in discoverAllNodes
    cdef = f.soup.doxygen.compounddef
AttributeError: 'ExhaleNode' object has no attribute 'soup'

• docs: don't have type hints 🙃

Hello,

I'd like to change some stuff to the generated RST, and would appreciate if I could declare a function that would be called after exhale is done generating these files.

Naturally, one could argue that instead, exhale should provide more configuration items but at the end, it seems impossible to cover all needs this way. So please allow me to be brutal and manually edit the RST after you.

I could write an extension over exhale for my own need, but I'd prefer to have something like this:

def my_postprocessing():
  # do something here

exhale_args = {
  postProcessingCallback = my_postprocessing
  ...
}

Thanks in advance,
Mt.

The tree view unordered list or javascript version needs to be minified, large-ish projects and page loads are taking an unnecessary hit. Add a configuration argument "minifyTreeView" defaulting to True. Maybe this means that a TreeView config object should be introduced as well in v1.0.0.

exhale_args = {
    # no "treeView" key => None => RST lists
    "treeView": {
        "type": "str",# "collapse" or "bootstrap"
        "minify": True # optional entry
    }
}

Can be flexible in the parsing, e.g. also allow "treeView": "collapse|bootstrap"?

See discussion here.

• RAM usage went up to ~4GB, I should be able to get rid of that in the future.
• STOP STORING THE PROGRAMLISTING EXPLICITLY

It seems that medium to large projects cause problems on RTD.

• Stop colorizing output on RTD, they don't let you have color (:rage:)
• Suppress doxygen output with /dev/null on RTD

The hacky silent branch verified that exhale is the cause of cat conf.py being the last step before failure.

If anybody ever reads this, one way to test exhale is generate the full api and track it on git (you can force-push this away, or do it on a different branch altogether and kill that branch).

Suppose you got

• docs/
  • api/
    • library_root.rst and all the others
  • doxyoutput/
  • conf.py

Then you want to locally make html once. Now git add -Af api/ doxyoutput so that all the generated reStructuredText and doxygen files are going to be available in a clone. Now edit conf.py and comment out the exhale entry in extensions and anything else exhale related (don't forget to git add conf.py changes).

Hopefully you shouldn't need to do this, but it can help identify where the problem is. In this case, doing that produced a successful build, so clearly exhale is the problem!

The bootstrap-treeview library provides significantly more customization than is currently exposed.

I got things setup far enough to satisfy my own needs, are there specific features people want that they cannot access?

Global configurations are likely the only ones that can be supported. Or at least node-level configurations in the json data would be significantly more challenging to both implement, as well as specify in exhale_args.

There currently exist quite a few pass-through options, but it seems likely people would potentially want more. This issue will basically remain open quasi-indefinitely.

• Provide better documentation on how to make development easier. As mentioned in #140 (comment) the best way to temporarily disable the generated API / rebuilds from exhale is to comment out exhale and breathe from your conf.py extensions list. This prevents everything being redocumented.
• Support only regenerating documents that have changed (#140). May cause hidden no-rebuild but should have issues.
• Harder.
  • Detecting changes in doxygen input. Can you use the cache from breathe? You want to avoid duplicating their cache.
  • Need to get a better understanding of incremental builds with doxygen on its own.
  • Relates: breathe-doc/breathe#439
    • Forcing this document to be reprocessed by sphinx pending #140 resolution.
  • Detecting files that were generated in a previous run, but changes in doxygen configs have now made it irrelevant.
  • If containmentFolder already exists, gather a list of all documents there (orig_files).
  • Generate all documents, accumulating a list of filenames that were generated (or skipped because they would be the same) (gen_files).
  • Delete any files from orig_files not found in gen_files.
  • Harder to create a test for for internal complications.

Note: progress on this is unlikely, thoughts welcome but incremental builds in this regime are complex and quite honestly intimidating. One of many reasons exhale is not well suited for large projects at this time.

Exhale is in "bugfix only" mode right now, I expect to be able to sit down and do the much needed rewrite this summer. Please continue to open issues related to build errors, or explain features you would like to see in the future. If it is a bug, a patch release can usually be obtained fairly easily.

Future requirements: Sphinx 2.0 is coming in soon, and will require python 3+. I will be requiring sphinx>=2.0 with Exhale 1.0. I also intend to require python 3.6 or later, this library needs a lot of string manipulation and f-strings make that much easier for me. If python 3.6 will present significant challenges for you or your project please speak up / explain why obtaining 3.6 or later will be problematic.

When I try to link to STL components using the cppreference.com Doxygen tag, I get the following error:

[+] Exhale: doxygen ran successfully in 0.81 seconds.
[~] Exhale: adding tree view css / javascript.
[+] Exhale: added tree view css / javascript.
[~] Exhale: parsing Doxygen XML.
[+] Exhale: finished parsing Doxygen XML in 0.04 seconds.
[~] Exhale: generating reStructuredText documents.
(!) Critical error while generating the file for [<REDACTED>].Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/exhale/graph.py", line 2574, in generateSingleNodeRST
    node.base_compounds, self.node_by_refid
  File "/usr/local/lib/python3.7/site-packages/exhale/graph.py", line 395, in baseOrDerivedListString
    link=nodeByRefid[refid].link_name
KeyError: 'cpp/error/exception'
make: *** [html] Error 1

I'm trying to exhale on a large code base that isn't particularly well documented. The doxygen XML gets created, but the .rst file generation is running into issues with file names that are too long

[+] Exhale: finished parsing Doxygen XML in 20.77 seconds.
[~] Exhale: generating reStructuredText documents.

OSError: [Errno 63] File name too long: (!) Critical error while generating the file for [docs/in/api/function_caffe2__operator_takes_as_input_a_tensor_of_floats_to_find_the_percentile_values_for_COMMA__a_2D_tensor_of_floats_COMMA__where_the_first_column_of_the_tensor_represents_sampled_values_COMMA__and_the_second_column_represents_the_percentile_labels_COMMA__and_a_tensor_of_integers_lengths._This_lengths_tensor_is_used_because_the_operator_works_on_multiple_sets_of_raw_values_at_the_same_time._For_example_COMMA__for_an_input_original_values=_COMMA__lengths=[2_COMMA__1_COMMA__1]_COMMA__value_to_pct=Our_operator_expects_that_each_column_i_of_the_input_tensor_is_sampled_from_distribution_i._Lengths_tells_us_that_the_first_two_elements_in_value_to_pct_are_sampled_from_distribution_1_COMMA__the_next_is_from_distribution_two_COMMA__and_the_last_is_from_distribution_3.We_expect_the_output_of_our_operator_to_give_us._To_calculate_the_percentile_of_an_element_COMMA__we_check_to_see_if_its_value_is_already_mapped_to_a_percentile_in_value_to_pct._If_so_COMMA__we_return_that_value._If_not_COMMA__we_linearly_interpolate_between_the_two_closest_values_in_value_to_pct._If_the_value_is_larger_than_all_values_in_value_to_pct_COMMA__we_return_1.If_it's_smaller_than_all_the_values_COMMA__we_return_0.RPAREN__DOC"RPAREN.Input.rst]

I'm not sure where to fix the specfic documentation, or the Doxygen config, or the Sphinx config to allow it to handle this. Is there some way to force it to ignore this particular file and keep building?

We have every class in our own namespace + directory, so the hierarchy looks something like this

• Namespace X

• Directory X

It would be very nice if there was a config for expanding the first level of the hierarchy by default, so that the user does not have to click it. We are using sphinx_rtd_theme, with "createTreeView": True.

in addition to the version, we should return parallel_read_safe and parallel_write_safe declarations from setup() so that we avoid this warning:

the exhale extension does not declare if it is safe for parallel reading, assuming it isn't - please ask the extension author to check and make it explicit

now i don't know enough about the internals of exhale to know if parallel reading/writing would be safe, but as a start we could just set both to false, addressing the warning and leaving room for future improvement.

what do you think? happy to open a PR as well.

So far exhale is great at recreating the doxygen hierarchy and separate pages for my huge project. My only beef with it is I'd like to edit the Full API section. I do not need it to display Files in the project when I already have a file hierarchy tree.

I'd also want to get rid of the "source" tab in my nav bar.

Is this possible? Thanks!

• replace intentional failures as TODO and just grep for those if you can't solve it right now.

Hi,

I am new for this tool, I am not sure that I can ask question here or not.
And currently I want to use this tool to generate document with following environment

Test Environment
doxygen (1.8.16)
sphinx (2.3.1)
breathe (4.13.0)
exhale (0.2.3)

And I found that when I use DOXYGEN_SHOULD_SKIP_THIS to deal with duplicate issue (same function name in header file and C source file), the function comment in the C source file cannot find. But the Doxygen web page still work.

Header File

#ifndef TEST_H
#define TEST_H

#ifdef __cplusplus
extern "C" {
#endif

#if !defined(DOXYGEN_SHOULD_SKIP_THIS)
char* at_test_action(void* data);
#endif // DOXYGEN_SHOULD_SKIP_THIS

#ifdef __cplusplus
}
#endif

#endif /* TEST_H */

Source File

/**
 * \brief Fills the data buffer
 * \param[in] data data buffer for content preparation
 * \return pointer to the data buffer
 **/
char* at_test_action(void* data)
{
    char* ptr = (char*) data;
  
    sprintf( ptr, "AT\r\n" );

    return ptr;
}

How can I deal with it?
And my conf.py setting as following,

exhale_args = {
    # These arguments are required
    "containmentFolder":     "./api",
    "rootFileName":          "library_root.rst",
    "rootFileTitle":         "Library API",
    "doxygenStripFromPath":  "..",
    # Suggested optional arguments
    "createTreeView":        True,
    # TIP: if using the sphinx-bootstrap-theme, you need
    # "treeViewIsBootstrap": True,
    "exhaleExecutesDoxygen": True,
    "exhaleDoxygenStdin": textwrap.dedent('''
        EXTRACT_ALL = YES
        SOURCE_BROWSER = YES
        EXTRACT_STATIC = YES
        OPTIMIZE_OUTPUT_FOR_C  = YES
        HIDE_SCOPE_NAMES = YES
        QUIET = YES
        INPUT = ../include ../src
        FILE_PATTERNS = *.c *.h
        EXAMPLE_RECURSIVE = YES
        GENERATE_TREEVIEW = YES
        PREDEFINED += DOXYGEN_SHOULD_SKIP_THIS
    ''')
}

Thank you so much.

sigh...

• snake_case rather than CamelCase (guarantees consumer break, which is important)

It seems that the file hierarchy option cannot handle nested directories.

For example, if I have:

Project/
   include/
      foo.h
      nestedInclude/
         bar.h

I get the error:

"Could not find directory parent of file [bar.h] with location [Project\include\nestedInclude\bar.h]."

bar.h still shows up in the file hierarchy, just without a parent directory.

Would it be possible to include functionality such that the include directory in the file hierarchy is able to have sub-directories as part of the list and not just files?