piggypiglet / docdex

JSON API & Discord Bot for Javadocs

PiggyPiglet

Hi, I'm Piggy, a student software engineer from the south west of Australia.

Projects

Contact

Come chat to me in HelpChat, or
my personal discord.

docdex's Issues

Add preview for long javadoc texts

Currently, the bot simply posts a link to the web version of the javadoc if the requested description etc is too long. In my opinion, some sort of preview that is truncated to fit the Discord embed limits would be a good addition. The text could simply be cut off and ... or something could be appended. This could result in ugly styling issues (e.g. if this happens in the middle of a code block), so a more sophisticated strategy might be worth discussing.

Slash Commands (discord bot)

Are slash commands planned / would a slash commands PR be merged?

Don't include description in simple object serializer, move that to something else

Current page note on help message

If you use the d;help command it gives you the ability to flick between pages using reactions which is great. A suggestion would be to have a way to know which page you are currently viewing. This could be done by only remove second reaction when a new page is selected or have it shown in the message itself.

Improve display of embeds

I believe the embeds could deserve some minor restructuring...

Current:

Suggestion (Made with Embed visualizer):

Paginate embedded suggestions intead of simply listing them

Javadoc List

Javadocs hosted by the official docdex instance, https://docdex.helpch.at & https://piggypiglet.me/docdex. Please request new javadocs here, by leaving the name, and a link to the javadoc jar (preferably from a maven repo). I need the jar as these docs are hosted locally. I'll react to your message with a thumbs up when I've added your javadoc.

• jdk17/jdk
• jdk16
• jdk11
• jdk8
• jda
• 1.18.1/latest/spigot
• 1.18
• 1.17.1
• 1.17
• 1.16.5
• 1.16.4
• 1.16.3
• 1.16.2
• 1.16.1
• 1.15.2
• 1.15.1
• 1.15
• 1.14.4
• 1.14.3
• 1.14.2
• 1.14.1
• 1.14
• 1.13.2
• 1.13.1
• 1.13
• 1.12.2
• 1.12.1
• 1.12
• 1.11.2
• 1.11.1
• 1.11
• 1.10.2
• 1.10
• 1.9.4
• 1.9.2
• 1.9
• 1.8.8
• 1.8.7
• 1.8.6
• 1.8.5
• 1.8.4
• 1.8.3
• 1.8
• 1.7.10
• 1.7.9
• 1.7.8
• 1.7.5
• 1.7.2
• paper-1.18/paper
• paper-1.17
• paper-1.16
• paper-1.15
• paper-1.14
• paper-1.13
• paper-1.12
• paper-1.11
• paper-1.10
• paper-1.9
• bungeecord/bungee
• bungeecord-chat/bungee-chat
• waterfall
• waterfall-chat
• adventure
• gson
• guice
• luckperms
• snakeyaml
• sponge-8/sponge
• sponge-7
• velocity
• worldedit
• worldguard
• guava
• commons-cli
• commons-codec
• commons-collections4
• commons-compress
• commons-configuration
• commons-crypto
• commons-email
• commons-io
• commons-lang3
• commons-logging
• commons-text
• plotsquared/p2
• plotsquared-bukkit/p2-bukkit
• fawe
• fawe-bukkit
• cloud
• bkcommonlib
• traincarts
• signlink
• protocollib
• javacord
• caffeine
• helper
• helper-sql
• configurate-core
• configurate-gson
• configurate-json
• configurate-yaml
• configurate-hocon
• processing
• jetbrains annotations
• minestom

[Feature] Add Javadocs for 1.17 & Java 16

I'd raise a pr by myself, but you would need to setup the additions on your server anyway, considering you host everything yourself, a pr would be a bit useless.

Huge speed issues

Currently when searching, a similarity/distance metric has to be generated against the query and each individual element stored in memory for that javadoc & type (unless there's a direct match). The amount of searches increases drastically if it's a method, as to scan for methods AND parameters. If parameters are provided and they aren't "full" (i.e. name + type), it has to sort 5 lists just to get a result. This can result in the route taking seconds to return a result, which is far too long. I'm looking into an alternative, specifically the symmetric delete spelling correction algorithm. Problem I have with this is it uses edit distances (however virtually all of these data structures do, e.g. bk-trees), instead of finding the best match, even if that match is far away. Perhaps I'll need to add an error "No results were found" if no results were returned in a reasonable (configurable) edit distance.

Additionally, the SymSpell java implementations don't use jaro winkler, which is my preferred algorithm for searching right now. I'll need to maintain my own implementation I suppose.

Default docs selection

Ability to select the default providing docs within a discord server

Adventure not working

not sure why

Ability to add reactions for any sent embed by DocDex

Title explains it

Discord bot fails to search documentation

Documentation commands are breaking with the error message:

Something went very wrong, me.piggypiglet.docdex.documentation.IndexURLBuilder@<hashcode>

Some other commands are failing with no error message, with the command message just being deleted, not sure if that's related, though.

Tested in the Java Community server.

Auto Updater

An auto updater is essential as the javadoc list continues to grow. It's unreasonable to expect myself to manually update these javadocs at every new release, and it's also unreasonable to expect javadoc owners/requestees to have to message me everytime they update. Therefore, a program to automatically update the javadocs is necessary as this project starts gaining traction. Currently the plan is to have a relatively simple system, with different strategies of fetching the latest javadocs for different javadoc download locations. The initial implementation will be a "maven latest" strategy, which will download the javadoc based on the info in the maven-metadata.xml of a repository. With the jar, it'll place it in the documentation webroot directory, and unpack it, making sure to also assign correct permissions. Additionally it'll also seize control of the docdex config, making sure it stays up to date. i.e. this updater will be configured manually, not docdex. It'll also link into pterodactyl to restart docdex, as that's where it's currently hosted. Obviously this updater is extremely coupled to my particular setup for the public docdex instance, so it's not really of any use to the public.

Sorting improvements

There's not actually anything wrong with the performance of sorting at the moment, but science isn't about why, it's about WHY NOT, and for science I say we research the fuck out of this and find the best way to sort.

Here are the ideas that came up in the discord convo today:

• Only sort for top k results
• Truncate doubles to lower number decimals for use in an O(n) array-abusing sort
• GPU acceleration
• Combustible lemons
• Glare's cool new website - https://glare.dev/gateways/
• Radix sort

signature generation is scuffed

see jdabuilder#jdabuilder for an example.

Real ORM

I tried to make a real orm to use, but couldn't quite get it done due to it's complexity. I want to revisit this issue in the future, but I'm worried that if I spend too much time on it right now with the current rate of progress, I'll lose motivation for the project. Therefore, I'm leaving it behind in favour of the more manual (but still effective) approach I'm implementing now.

Last commit with it: 6cbe6cd

I'll probably come back to it for RPF, which would benefit greatly from an orm like the one I was attempting to implement. Sadly, such a thing is simply not needed for docdex.

Method parameters are missing

it appears that embed links have broken for some reason

Add markup support for description texts

The bot currently extracts description and other textual data from Javadoc as plain text. It would be nice if it converted the HTML used in Javadoc to Markdown, the markup format used by Discord.

This would include emphasis, strong emphasis, inline code, code blocks and hyperlinks which can all be displayed nicely in Discord's flavour of Markdown. This would also fix current ugly properties such as spaces being the only whitespace or lists being stripped away entirely (even though Discord doesn't render Markdown lists, it would still look better).

If you're looking for a HTML to MD converter, I can recommend flexmark, specifically the flexmark-html2md-converter module.

Note: This would also resolve #17 .

Queries and Discord message formatting

When using something like d;spigot NamespacedKey#minecraft it turns into an invalid query because on HelpChat, #minecraft it is a channel and DocDex takes the message literally. If possible, ignore such formatting (doesn't JDA have a method to get the raw message, with #minecraft instead of <#382856648064237568>?).

Add a way to sort out overloaded methods

Adding this will also fix #7

[API] Limit affects results

Was doing some testing, discovered that the limit of responses can affect the nature of the response.

For example, https://docdex.helpch.at/index?javadoc=jdk&query=map~getordefault(key,%20defaultvalue)&limit=10 will return one result, an exact match - as it should.

However, if you take the 0 off the end, https://docdex.helpch.at/index?javadoc=jdk&query=map~getordefault(key,%20defaultvalue)&limit=1 returns one match, but it is not an exact match, nor the same result as the query with a limit of 10.

SnakeYaml Doc bug

d;snakeyaml Yaml#loadAs
This command stated above fails to send the doc while it’s counterpart
d;snakeyaml Yaml#load
works fine this is likely due to the sneaky /./ in this:
https://docdex.helpch.at/index?javadoc=snakeyaml&query=yaml~loadas

Fail after single error

Don't keep spamming error messages, just cancel the task after a single error. e.g. when a non-matching result is returned and the message is paginated, then the bot proceeds to send 6 errors as it cant add the 6 reactions.

bukkit#getplayer isn't showing both uuid & name options

Ignore space after prefix

Add command to see the constructors of a class

Add a command like d;constructors which will display all constructors of a class. An alternative to this would be to make d;<javadoc> ClassName#ClassName return all constructors and not just one if the class has an empty constructor.

ItemStack has an empty constructor, so the rest are ignored.

To see all constructors, you have to use this:

Put Javadocs into categories

It could be helpful to categorize the list of Javadocs when using d;docs

My suggestions:

• Java for JDK docs
• Spigot for the Spigot docs
• PaperMC for the PaperMC docs
• Plugins for the various plugin docs
• Other (Or maybe Misc) for things like the commons docs

Red embed for deprecated stuff

"Since" field

Broken Javadocs

These javadocs aren't getting indexed for some reason:

• fawe
• fawe-bukkit
• waterfall
• caffeine
• waterfall-chat

Put in-description examples in code blocks

e.g. the example in plotsquared-bukkit's ChunkCoordinator

convert weird static deserialization stuff to instance bound stuff

not sure why I made it like that

Invalid JavaDoc resolution

Currently docdex resolves all documentation to the parent objects. This leads to the explicit implementation JavaDoc not being accessible.

For example:

d; jdk Set#add

returns the JavaDoc for Collection#add.

paper pathfinder doesn't send any message

Nothing is sent in discord when a metadata object is empty

When querying for, for example, sub-interfaces of a class (something straight up unpossible), nothing is returned to discord.

Would make a nicer issue but uuuh ehem lazy

NPE when saving to mongo from flatfile

Descriptions consisting of multiple blocks are not scraped correctly.

e.g. https://docs.docdex.helpch.at/jda/net/dv8tion/jda/api/events/Event.html#getResponseNumber()

Add support for enclosing interfaces

e.g. https://docs.oracle.com/en/java/javase/11/docs/api/jdk.scripting.nashorn/jdk/nashorn/api/tree/Tree.Kind.html

Broken Superinterfaces

d;super_interfaces paper player
not working

Discord Thread Support

pretty self explanatory

[Bot] Exact match not returned unless $ used in command

Had this conversation with the DocDex bot recently:

It would be ideal if, given a limit of n, when an exact match is found to build an embed around that instead of returning the same query in a list of suggestions - giving the bot a limit, I had thought, would cause it to create several paginated embeds with the first result first, but even if this is not the case when an exact match is found but a limit other than 'first' is given, the error should probably not say that no match was found 😄

Some mismatch method queries return multiple copies of the same method

Description

Some (not all) mismatched method queries suggest certain methods multiple times when there is only one of them

Examples

Spigot's ServicesManager only contains one load method yet it shows multiple:

The JDK's Map class contains in reality a single get method:

Pagination buttons

Make the pagination buttons accessible only by the message author (maybe add an option to enable / disable this?)

Results order