zsol / py.wtf

Autogenerated reference docs for PyPI packages

py.wtf

---

Wat dis?

It's autogenerated reference docs for PyPI packages. See for yourself at py.wtf.

Butwhy

py.wtf aims to be an absolutely 0 effort reference doc hosting site for all packages on PyPI: No setting up doc building pipelines, domains, hosting providers.

• Read the Docs is very nice, but it has a tutorial. py.wtf just exists.
  • It's still very nice for putting together more than just reference docs.
• MkDocs is very nice, but it has a tutorial. py.wtf just exists.
  • It's still very nice because you can write your docs in markdown and the output is pretty.
• Pycco is nice and its tutorial is pretty short too! But py.wtf just exists.

py.wtf statically (i.e. without installation) analyzes Python projects to extract modules, functions, classes, variables, their types, and docstrings. Then cross-references these across projects to display them in a single, integrated, and searchable web page.

As such, it is impossible for project authors to configure py.wtf, apart from changing docstrings themselves. This brutal inflexibility allows py.wtf to stay relatively simple¹, and attempt to do one thing, well.

Installation

Just don't.

Hacking

Please do. Python 3.12, Node 18.

cd py.wtf
pipx run hatch shell
YOUR_FAVORITE_PROJECT=click
py-wtf index --project-name $YOUR_FAVORITE_PROJECT www/public/_index/
cd www
npm install
npm run dev

Acknowledgements

This project draws inspiration heavily from the following excellent software:

• Hackage: The Haskell Package Repository
• docs.rs
• Glean

Thanks goes to these wonderful people (emoji key):

Zsolt Dollenstein 💻 🚇 🎨

Zoltán Nagy 💻 🎨

Adrienn Éva Csengeri-Pap 💻 🎨

Anuj Upadhyaya 💻

This project follows the all-contributors specification. Contributions of any kind welcome!

License

py-wtf is distributed under the terms of the MIT license.

Footnotes

1. except for the parts that deal with reStructuredText and markdown ↩

py.wtf's Issues

Inner class permalinks don't work

e.g. in the test index, go to http://localhost:3000/project-alpha/alpha.core/Helper and then click on Utils in the nav bar, it will take you to http://localhost:3000/project-alpha/alpha.core/Helper.Utils which doesn't work

Show project description on the project page

For example on https://py.wtf/click we should show https://pypi.org/project/click/#description

Add CLI command to index changes since last run

Reindexing 500+ projects can take half an hour. We only need to index changes

Support markdown in documentation

Markdown is common in project descriptions. Some md constructs cause docutils parser to raise an exception. The indexer should detect if a documentation string is markdown compatible and process it accordingly (i.e. no need for processing)

Distinguish between docstrings and comments

Currently for a function like this:

# foo
def bar():
  # lol
  """my docstring"""

The generated documentation will look like:

("foo", "lol", "my docstring")

This is suboptimal, we want my docstring to display prominently, but also keep foo and lol somewhere.

Style links coming from MyST

These links for example are unreadable:
https://py.wtf/LibCST/libcst._nodes.statement/get_docstring_impl

Index `__init__.py` as the package instead of a module named `__init__`

In the current representation, a/b/__init__.py should just be translated to a Module(name="a.b")

Set up storage for 500+epsilon projects

This will mean ~800MB of uncompressed index json files.

Make symbols clickable

Wherever a type or function or class is mentioned (esp. in a code block), it should link to the documentation for that symbol.
This needs the symbols to have URLs - see #3

This is especially tricky because of generics.

Nested modules

Like, in package foo, we could have a module foo.bar.baz. What happens with these currently? My default guess is they're skipped, without looking at the indexer. What should happen with them?

• Maintain strict hierarchy; package-level overview only shows top-level modules (a 'la Rustdoc). Module overviews show their own submodules.
• Flatten modules, show foo.bar.baz on package-level overview. Module overviews can, I guess, still show their own submodules.

Add keyboard shortcuts to search box

Triggering the search with a key (say, / and/or s), and then being able to navigate the results with the arrow keys.

Add breadcrumbs-like navigation

package > module > symbol at the top

Only parse interesting files in a project

We don't care about random .py files in the source dist:

• sphinx config files
• test files
• benchmarks

Check out https://py.wtf/numpy for a good example of what we shouldn't index.

There are some options and tools we could use:

• https://github.com/python-packaging/dowsing
• fetch pure python bdist_wheels if available
• blocklist stuff 😞

Add blocklisting capability to the indexer

Some packages like https://pypi.org/project/azure-mgmt-network/ produce several hundred megabyte index files. We don't need these for now

cross-reference base classes

Display __all__ on module pages

#33 added explicit exports to the index; let's show them! Probably as a dedicated section near the top of the module contents page.

Search box doesn't go away sometimes

Looks like on the second search, the box stays there and can't be dismissed.

Set up basic frontend (smoke) tests

• Commit a static package JSON into the repo that demonstrates all the weirdness we care about.
  • Ideally by including such a "fake" package in the source tree, and generating the JSON from that, so that we can use the same data to test the indexing part.
• Add very basic tests that verify all the various pages even render at all.
  • https://nextjs.org/docs/testing shows Next.js supports 4 testing frameworks, I only know one of them, needs research or input?
• Add somewhat less basic tests that verify the correct content is showing up (like, correct modules in the sidebar, correct items in the table, correct function signatures in detail pages)
  • This is possibly snapshot testing, though the design / layout is bound to change a whole lot and quickly, so maybe it's too early for that.
• Run all this in CI, obviously

Set up CloudFlare

Fire up a free CloudFlare account, point it at GH pages, do the URL rewriting there, get the SEO benefits, drop the horrible URL rewriting hacks?

Originally posted by @abesto in #38

Auto-generate test index into the right directory

The pre-commit hook introduced in #70 should generate the test index json files into www/__tests__/index.

Show fancy type hints

Instead of

Callable[[a, b], c]
Union[a, b]

Let's show

(a, b) -> c
a | b

And maybe even

Tuple[a, b, ...]
# should turn into
(a, b, ...)

Add a footer to every page

A gentle, non-intrusive footer would be nice to guide people to this repo, the issue tracker, etc. Something like:

Footer™: This is a hobby project, please be gentle. Send issues here. Bugs courtesy of abesto, anathien, zsol.

Index xrefs in type signatures

We want to be able to link to symbols within and across packages. Instead of a Type being a string, we'd want the indexer to emit something like

class Type:
  name: str
  xref: XRef | None
  params: None | Sequence[Type] = None

class XRef:
  pkg: Literal["__std__"] | str | None
  fqname: str

Use cases covered:

Foo

Type("Foo", XRef("mypkg", "x.y.Foo"))

dict[str, Foo]

Type(
  "dict",
  XRef("__std__", "dict"),
  params=[
    Type("str", XRef("__std__", "str")),
    Type("Foo", XRef("mypkg", "x.y.Foo")),
  ],
)

Callable[[], Mode]

Type(
  "Callable",
  XRef("__std__", "typing.Callable"),
  params=[
    Type("", xref=None, params=[]),
    Type("Mode", XRef("black", "black.Mode")),
  ],
)

Callable[[Foo], Mode]

Type(
  "Callable",
  XRef("__std__", "typing.Callable"),
  params=[
    Type("", xref=None, params=[Type("Foo", XRef("mypkg", "x.y.Foo"))]),
    Type("Mode", XRef("black", "black.Mode")),
  ],
)

MysteriousFoo (when the indexer can't determine the source)

Type("MysteriousFoo", XRef(None, "x.y.Foo"))

Run ESLint in CI

Originally posted by @zsol in #20 (comment)

(Anchor) links to symbols

Should be able to link to the docs for (and show an anchor to copy the link for)

• Modules (currently handled by normal page URLs)
• Classes (and their methods, variables, subclasses)
• Functions
• Variables
• Type definitions

Update the ClassContents sidebar to actually link to them.

Index re-exported symbols

The stuff in __all__ in packages. We probably don't want to deal with (and so, encourage) implicitly exposed symbols, like if module a imports symbol b, and so b is accessible under a.b

Add a search bar to the package pages

There should be a search bar on top of every (non-index) page that lets users search the current package's symbols.

Add a nicer theme

Not an exhaustive list, but should include:

• A nice, readable dark color scheme
• (optional) a nice, readable light color scheme
• Proper link styling
• Proper sidebar highlights
• Comfortable white space
• Readable symbol table (though maybe that should be it's own issue)

Show if a parameter has a default value

Parameter.default is set in these cases.

Next.JS -> Gatsby?

https://www.gatsbyjs.com/ uses a very similar stack to what we have now, and is explicitly designed to make building static-only sites easy. Depending on whether we foresee a world where we want an active server-side component, Gatsby could be a better fit for our needs.

Comparison maintained by Gatsby: https://www.gatsbyjs.com/features/jamstack/gatsby-vs-nextjs

This is mostly orthogonal to "do we materialize an HTML file for each symbol"; I expect to be able to use the same tricks with Gatsby that we do now with Next.JS. I also expect we'd be able to do the rewriting in the CDN config, if we moved to using a CDN.

(Which... why aren't we doing that? Fire up a free CloudFlare account, point it at GH pages, do the URL rewriting there, get the SEO benefits, drop the horrible URL rewriting hacks?)

@welcome-ui/link vs react-router-dom

We use client-side routing for ... reasons. That means react-router-dom. The Link component of react-router-dom renders an <a> tag, and you don't get a say in this matter. However, we also use WelcomeUI; the WUI Link component (that we kinda need for styling / themeing) also renders an <a> tag, and you don't get a choice in THAT matter either.

Unfortunately an <a> tag inside another <a> tag is not valid, and generates a warning. It works, but it'd be good to not be clowny.

Index project metadata

Stuff like homepage, docs URL, authors, dependencies, source code URL, issue tracker.

Type annotations formatted incorrectly

There's

• extra space before ->
• missing : in parameters

Support Sphinx cross-referencing syntax

https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#cross-referencing-syntax

make sure `ProjectRepository.get` doesn't block indefinitely on a mischievous factory

This is still a bit dodgy; if factory(key) never async yields the project with .name == key then this await will block indefinitely (or until another poor coroutine happens to fulfil the future).

I'll fix this and add a test for it in a followup PR

Originally posted by @zsol in #88 (comment)

When displaying a class, highlight the class itself in the sidebar

The sidebar has a section with all the contents of the module at the end. Let's highlight the displayed class there, same as we highlight the displayed module in the module page.

Restructure content?

I'm pretty sure that to make content legible, each "page" / view should contain docs on exactly one top-level... thing. So for example

• A module page may contain a listing of all the top-level symbols, maybe with the first line or so of its docstring
  • Maybe variable / constant docs are just fully inlined here
• Separate pages for classes (what's up with inner classes?)
• Separate pages for top-level functions?

I don't know what plans you had in mind about all this. I've hinted at this before, but I'll just drop it here explicitly so that there's a place to discuss it: my default approach would be to model the organization structure on docs.rs. Some things obviously need adapting as the languages differ, but even so that should be much less effort than trying to design this from first principles.

Link stdlib symbols to https://docs.python.org

Treat imports in `__init__.py` as exports if there's no `__all__`.

Some popular packages do this, but py.wtf/click/click is sad.
Let's humor them.

Remove ugly scrollbars from the page

Include value of function parameter defaults

Currently it's only .... For extra credits, also include what kind of value the default is so we don't need to rely on client-side heuristics for syntax highlighting (see #74)

PIL.ImageChops.darker's parameter types aren't linked properly

in https://py.wtf/pillow/PIL.ImageChops/darker
there are links to https://py.wtf/pillow/PIL.ImageChops..Image/Image

Do something with unknown MyST directives

There's plenty of them on this page for example:

• versionchanged
• versionadded

Do something better than a "Loading..." string

Add a search bar to the index page

There should be a search bar on the index page that finds packages by name. Instead of listing all packages on the index page, just list a few.

Parse, correctly display RST

Big chunks of documentation text are currently not very good-looking :D Two big things here:

• Format RST
• Detect links to symbols and make them links on py.wtf (needs #3) or standard Python docs where appropriate

Corollary: do we ever expect non-RST strings in documentation? If yes, those will need to get the same treatment.

Upgrade to Node 18

Now that we got rid of Welcome UI / Ramda (#57), we can use a modern Node version \o/