wordpress / phpdoc-parser

Documentation parser powering developer.wordpress.org

WP Parser

WP-Parser is the parser for creating the new code reference at developer.wordpress.org. It parses the inline documentation and produces custom post type entries in WordPress.

We are currently looking for contributors to help us complete the work on the parser.

There is a guide to developing for developer.wordpress.org in the WordPress documentation handbook

Requirements

• PHP 5.4+
• Composer
• WP CLI

Clone the repository into your WordPress plugins directory:

git clone https://github.com/WordPress/phpdoc-parser.git

After that install the dependencies using composer in the parser directory:

composer install

Running

Activate the plugin first:

wp plugin activate phpdoc-parser

In your site's directory:

wp parser create /path/to/source/code --user=<id|login>

phpdoc-parser's Issues

Used by and Uses information should be parsed

On the code reference we want to displayed used by and uses information. Example here: http://developer.rarst.net/reference/functions/

This information will need to be parsed so that we can display it in the theme.

Only store a single post for each hook

At present we store an entry for a hook each time it appears in the source. But hooks are only documented once, with duplicate entries marked with a comment. We should ideally only store a single entry for each hook. The entry would then have metadata/something storing the specific data for each place the hook is used (file/function/method and line numbers).

The main issue is that right now post associations are done in a parent/child manner. Since a post can only have one parent, this will have to change. Posts 2 posts has been mentioned before.

Store @since version as separate meta field

Version introduced is one of the basic facets to query by and doesn't quite work for that if buried in serialized data.

For reliable querying needs to have its own field and be normalized (some older stuff has backpress versions, etc).

End of sentences sometimes lose the space after the full stop

Sometimes, an end of sentence loses the space after the full stop. Not sure if this is a problem with the parsing or some filter in WordPress.

In the WP Text editor, it appears as "and slow on large datasets.This function"

Example: http://vip.wordpress.com/function_reference/vip_get_random_posts/
"and slow on large datasets.This function"

Function's raw phpDoc markup is at https://gist.github.com/paulgibbs/2150f1d778ae5d0876de

WP-CLI generate command output is at https://gist.github.com/paulgibbs/d51754a02ad3e31213a5 (note the \n)

Looking for feedback on where and how this should be addressed.

Untangle namespaces and prefixes

Code currently contains:

• WPFuncRef namespace
• inline unprefixed classes in it
• PSR-0, WP_ prefixed classes not in it
• wpfuncref_ functions not in it
• unprefixed functions in it
• probably exotic combinations of the above I am missing

Mildly put this is inconsistent.

Since there is actually no such project as "WP Function Reference", I am planning to untangle all of this into following:

• WP_Parser namespace for all definitions
• "flat" class files in WP coding style

Open to feedback before I get to it.

Make ['arguments'] include data from phpdoc

For a function, the ['arguments'] part of the JSON output consists of:

"arguments": [
{
  "name": "$user_id",
  "default": null,
  "type": ""
}
],

This is generated from inspecting the function parameters and any typehinting information. Here's the PHPDoc I'd typically write "@param int $user_id The User ID to delete" to document a function parameter, which in the JSON output looks like:

"tags": [
{
  "name": "param",
  "content": "int $user_id The User ID to delete"
},

The information from a PHPDoc param tag, if present, should be incorporated into the arguments object. I'm currently doing some ghetto operations to glue the two together when I'm building a template, which is messy and not the correct place to do this.

The output should end up looking like:

"arguments": [
{
  "name": "$user_id",
  "default": null,
  "type": "int",

  // Then maybe a new tag:
  "description": "The User ID to delete"
}
],

Allow periods in @since taxonomy terms slugs

http://txfx.net/wordpress-plugins/periods-in-titles/ allows periods in post and term slugs. I use it at, for example, http://genesischangelog.com/2.0.2 .

It would be good if periods would be allowed for the @since taxonomy terms, so docs.example.com/since/1-0-0 would become a friendlier looking and naturally quicker to type docs.example.com/since/1.0.0 .

Separate CPT for Methods?

Methods are bundled in to the wpapi-function post type, since there's obviously a lot of similarity to functions. However, that means that a method entitled Genesis_User_Profile_Widget-widget (which is horrible formatting btw!) appears with all of the functions on the /functions archive. Having class methods mixed in with standalone functions is confusing at best.

The query could be amended to filter on on some indicator that means it's a method not a function (maybe post_parent > 0 for methods? What about nested functions?), but that seems like extra work, just to return clean archive of functions, as the name suggests.

Methods might also want to be displayed differently, perhaps indicating and linking to the class parent in the body, formatting the title to be something more like Genesis_User_Profile_Widget::widget or showing / filtering based on visibility keywords.

Either way, I propose that there's enough potential there to at least consider doing a wpapi-method CPT, especially if constants and properties might be included later as well.

Thoughts?

Parsing hook with just variable name

While a hook name like do_action( ‘genesis_attr_’ . $context ) gets added correctly, a hook name of do_action( $args['before_sidebar_hook'] ) doesn't. It gets added to the JSON file as "name": false, which then gets imported with "(no title)" as the title, and gives a warning at the end of importing:

Warning: Problem inserting/updating post for method ""

The problem seems to come from WP_Reflection_HookReflector ::getName() or what it depends on.

Class not found in lib/runner.php

I'm getting below error while generating and importing.

Fatal error: Class 'WP_Parser\RecursiveIteratorIterator' not found in
WP-Parser/lib/runner.php on line 6

Getting end line number

End line information is available in PHP Parser Node, but it's getting buried in protected $node property of FunctionReflector object. BaseReflector only has getLinenumber() method for start line.

Any ideas on how to get to this?

WP-CLI plugin

Opening a ticket to discuss if to integrate this into a WP-CLI library.

Usable proof of concept: https://github.com/Automattic/WP-CLI-PHPDoc/

Add taxonomy for @since versions

What do you think about adding a new taxonomy for storing @SInCE versions for generated documentation? It might be nice to be able to filter when an function (etc) was added to a project.

Improve import feedback

https://github.com/rmccue/WP-Parser/blob/master/importer.php#L357

The import feedback code fails to check for a hook post type, so importing a hook results in "Imported method ...", since presumably the hook has a parent of a function.

Duplicate hook entries when re-importing

When a hook appears more than once, in two different functions (different parent_id), the importer creates multiple entries.

When the importer is run again, and checking for the existence of that second occurrence of the hook, the SELECT is limited to only the first entry. Since this is the entry for the first occurrence of the hook, it creates a second entry for that second occurrence. On the third run, it still only sees that first occurrence, and so creates a third entry for that second occurrence.

Here's what I get after parsing Genesis three times:

The hook do_action( 'genesis_loop_else' ) appears three times in Genesis, under three different functions (here, imported as IDs 372, 382 and 677). You can see that when it's run a second and third time, the second and third occurrences are duplicated, presumably because the LIMIT 1 returned only the 372 result.

I've not quite got my head around where the logic is falling down here, but it definitely is.

Improve default content function

While expand_content() can be dropped by developers in favour of a custom output, the default function should still be improved.

It is currently very focused towards how functions look, and not classes and hooks. For instance, the prototype doesn't make a lot of sense of classes or hooks.

The Arguments heading is output regardless of whether there are any argument or not.

The function name itself is also a little too generic - wpfuncref_content() might be better for instance.

Add activation / deactivation hook for flushing rewrites

support @internal tag

I'd like to update the wp-cli commands to not create posts for functions/classes/etc marked as @internal unless an override flag is passed to wp-cli.

When generating documentation from PHPDoc comments it is RECOMMENDED to hide the associated element unless the user has explicitly indicated that internal elements should be included.

An additional use of @internal is to add internal comments or additional description text inline to the Long Description. This may be done, for example, to withhold certain business-critical or confusing information when generating documentation from the source code of this piece of software.

https://github.com/phpDocumentor/phpDocumentor2/blob/develop/docs/PSR.md#78-internal

Update post type names

wpapi- prefix in CPTs is inconsistent with the rest and (as raised in chat) is confusingly similar to unrelated WP API project.

Detect deprecated files

There is number of deprecated files that seem to only be marked as deprecated by _deprecated_file() function call. Currently this is not reflected in parser output, so for example there is no way to know classes/functions defined in those files are deprecated (since they themselves are not marked as such).

Needs either:

• manually maintained list of deprecated files to use when processing parsing results
• keep track of files that call _deprecated_file() on parser level

Function long-description contains paragraph tags

When a function's [doc][long_description] is set, it's wrapped in paragraph tags.

Is this intentional? It seems inconsistent to me as the other fields don't contain HTML markup (or at least, I haven't seen anything else with).

Use Composer for dependencies

Can we please consider Composer for dependencies again?

@nacin swore to me in person at WCEU that his issue was with broken Composer install rather than Composer in general and he did not push to get rid of Composer.

2/3 devs on DevHub code reference (me & @toscho) are on Windows and Git submodules insanity hits us especially hard (which @nacin witnessed personally).

Also @wp-cli is moving towards Composer-based command distribution.

Optionally ignore deprecated structural elements

Since functions, methods and classes can have the @deprecated tag, some may want to only include the non-deprecated stuff in the final import. An --ignore-deprecated=true (default = false) flag when generating the JSON file would be handy, and would be handled in a similar way to how the @internal tag is supported.

Vaguely related: #11.

Create test source file

Running Parser against whole core lengthy process and errors thrown during are getting washed away by output.

We could use a file including representation of elements we are interested in, made up or just hacked together from some core pieces.

This is not exactly for unit testing ( #52 ), but for quicker more clear development workflow and not having to deal with thousands of posts all the time.

Treat apply_filters_ref_array() as a hook function

The do_action_ref_array() variation of do_action() is treated as a hook function, but apply_filters_ref_array() is not. Main use of this function is likely in WP core.

Import hooks in methods

Hooks in class methods are being exported (not working right now, but there is an attempt), but they aren't being imported.

phpdoc parsing has broken for function tags

If importing a function that has a @Package or a @subpackage tag set, all of the function's other tags are not imported.

This has taken a few hours ;) to track down. I suspect I broke it in a recent commit. Will fix hopefully tomorrow.

Add support for parsing PHP constants

Would be rad.

Set up package on Packagist

Moving towards easier bootstrap processes.

Can't do this myself, needs to be repo owner.

Use a hierarchical file taxonomy

It'd be cool to browse functions per directory.

Import hook type

We export the hook type (action or filter) but it isn't imported.

Multi-part tags

Tags like @see {Structural Element} {description} or @uses {Structural Element} {description} are not having the Structural Element part stored within the generated JSON file.

{
"name": "see",
"content": "Callback for sitewide box."
},

If a way could be found to store this data, and if it was imported, then automatically generated links could then start to be added at the template level.

Add tests

It would be nice to implement some kind of test suite. Thinking unit tests with PHPUnit, but others may have other suggestions.

Missing funcref CPT reference in autop_for_non_funcref()

autop_for_non_funcref() has a conditional which checks to see if the post type is a class, or function, but not a hook, before reapplying wpautop(). I think the hook post type should be in there as well.

Decouple interface from wp-cli

wp-cli is a little arbitrary as interface requirement, especially hard one.

1. Internals should be refactored into API capable of supporting arbitrary interface
2. At the minimum content for DevHub scoep should be able to be generated without wp-cli (probably via easy to use endpoint / screen in admin)

Memory allocation fatal error for hook entry

http://azzag.co.uk/hook/genesis_before gives a Fatal Error - memory allocation.

It only happens for single hook CPT entries (and the first entry of a /hooks archive). Function and Class CPT entries appear fine. CPTs from other plugins appear fine. Happens with all other plugins disabled. The suggestion from the host is that there's a memory leak for the hooks CPT (either in registration, or grabbing the data out).

Hooks with non-vaiable arguments' arguments are ignored

This may end up being a wontfix, but here goes.

If you have a hook like this:

$minimum = apply_filters( 'my_minimum', 0, $other_arg );

The arguments won't be picked up properly. Rather than being saved as an array, they are saved as a string: "0, $other_arg"

This causes an error:

PHP Warning: Invalid argument supplied for foreach() in ../WP-Parser/template.php on line 91

Changing it to this would fix the issue:
$minimum = apply_filters( 'my_minimum', $minimum = 0, $other_arg );

And that might be the best thing to do.

Invalid arguments passed in implode()

When I ran the command "wp funcref generate-and-import . --user=1" I immediately got a few warnings before the import ran.

"PHP Warning: implode(): Invalid arguments passed in [path-to-plugin]/vendor/phpdocumentor/reflection/src/phpDocumentor/Reflection/FileReflector.php on line 494."
This was followed by an identical warning, just without the "PHP" in front.

edit: Removed @SInCE error, and modified information slightly.

plugin.php - why does it need to be deactivated?

The instructions for DevHub say
When parsing is complete, disable the parsing plugin either through your admin or by using: wp plugin deactivate WP-Parser
This is important as the site fails to function correctly when it is running.

Why not just wrap the code to test for WP_CLI_ROOT

if ( defined( 'WP_CLI_ROOT' ) ) { etc

Also. I noted that it needs PHP 5.3.6 or above otherwise you get

Fatal error: Call to undefined method SplFileInfo::getExtension() in plugins\WP-Parser\lib\runner.php

Only add hooks that originate in the parsed source

This might be outside the intended scope of this plugin, but I hope it will be considered anyway.

Let's take a basic action of a theme calling do_action( 'wp_head' ). Parsing the theme, this makes it look like the hook originated in this theme, when really it is defined elsewhere (in WP itself).

For those generating docs specifically for a plugin or theme, it might make sense to be able to add a tag (@externalhook?) to the hooks documentation that identifies that a hook originates outside of the source being parsed (and so ignore those hooks like how @internal works).

A child theme might do the same with hooks that are originally defined in the parent theme.

Also ping @DrewAPicture - any thoughts on something like this being suitable for a wider recommendation via the Handbook?

/**
 * @externalhook
 */
do_action( 'wp_head' );

Indicate whether arguments are passed by reference

Would be nice to know for functions/methods and hooks.

Handling duplicate entries with the same parent ID

When the same hook is called twice within the same function, the check for an existing ID will come up as true, even though the meta (e.g. line-number) will likely be different for the second call as the first call. Equally, if functions with the same name are defined twice, perhaps in the global namespace (parent ID = 0) in different files that are not included within the same request, they too will have different meta data and should be different entries.

Perhaps a join to postmeta table on the existing ID check and also compare the file and / or line number would narrow it down?

If the same hook is being called twice on the same line in the same file, well, that's just stupid! (But technically plausible.)

Inconsistent filter names

The wpfuncref_return_type() function contains a wpfuncref_the_return_type filter, yet there is a separate wpfuncref_the_return_type() function.

The wpfuncref_return_desc() function contains a wpfuncref_the_return_desc filter, yet there is a separate wpfuncref_the_return_desc() function.

The wpfuncref_prototype() function correctly contains a wpfuncref_prototype filter (and there is a separate wpfuncref_the_prototype() function).

I'm not sure what the backwards-compatibility status is of this plugin, but having function A include a filter that is the same name as function B could lead to confusion. Can this be fixed?

Only hooks getting parsed/imported

I don't see functions and classes getting parsed or imported when I run the default command given in the readme. Is it only me facing this issue?

Notice: Undefined index when importing

The updating of post meta towards the end of multiple import_*() methods in the importer.php class need some isset() checks to potentially set the $data array values as an empty string (or delete the post meta key), to avoid some PHP Notices regarding undefined indexes.

Bundle documentation templates

I've been thinking about the problem of requiring people to build templates to display the data that this importer generates. Right now, it's not ideal:

• You have to register post types / taxonomies with the specific names that the importer uses.
• While it's a post type, you have to create template file for the single posts and archive templates, etc. While WP's template hierarchy will make things work with the default single.php template etc, this is almost certainly not going to be a decent experience for something as structured as the posts we create.

We could bundle template files with the plugin and tell users to copy into their theme, but as these will be full templates, we'll quickly find people complaining that the HTML/CSS structure is different from their themes' templates, and so on.

Proposal: create a plugin that registers the post types, and use bbPress/BuddyPress theme compatibility techniques to enable an out-of-the-box documentation experience for any site.

Split @return description from @return type

@return <"Type"> [description]

This currently ends up like so, and you have to parse it out manually:

"tags": [
{
  "name": "return",
  "content": "DPA_Achievements_Loader The one true Achievements instance"
},
{

For
@return DPA_Achievements_Loader The one true Achievements instance

A description field should be added to the tags.

Parse which functions are hooked to which hooks

I don't think there are any tickets open for this, but I believe it may have been mentioned somewhere. We should parse add_action() and add_filter() calls and export of list of which functions are hooked to which actions/filters.

Parse global variables

Definitely an under-documented portion of core. On the other hand, WP is moving away from using globals, and I'm not sure if many of them are actually documented inline anywhere. So although I'd like to see this added, I understand that it isn't going to be a priority.

Improve/remove support for changing custom post type names

In Importer::__construct() the names of the custom post types and custom taxonomies to which the data will be imported may be overridden by passing custom names in the $args parameter. This feature doesn't appear to actually be in use anywhere in the parser.

The point is, we are allowing the names of the post types and taxonomies to be customized here, and yet everywhere else they are hardcoded.

I suggest that we either remove this feature entirely, or refactor things so that it is actually more usable without having to hack the whole plugin. Thoughts?