Message conversion and formatting about gh-migration

[ammaraskar]

1. Import messages within code-block markup, to render it literally:

One nice aspect of this is that bpo issues are currently displayed monospace. Without adequate conversion of code blocks such as in (3), code snippets and places where alignment is important would look broken. (3) seems quite hard to implement though, a lot of the formatting is ad-hoc (I know I've personally sometimes kept code blocks on the same level or indented them with 2 or 4 spaces occasionally). It seems like (1) might be the way to go. If we really want (2) there are mature libraries like turndown.

On bpo, links to other issues, messages, PRs, PEPs, etc. are added at rendering time using regexes.

This is probably less of a concern with https://docs.github.com/en/github/administering-a-repository/managing-repository-settings/configuring-autolinks-to-reference-external-resources which we already have set up for bpo links on the CPython repo thanks to @Mariatta in python/core-workflow#361

Issues numbers can also be remapped from the bpo to the GH numbers during the same step.

Depending on the resolution to #2, it might be nice to keep the links the same and then the bugs.python.org/issuexxx links end up redirecting you to the right Github page. But assuming we make the roundup instances read-only or mirrored then remapping the issues is probably a good idea.

[ezio-melotti]

We have:

• raw text, possibly including markdown characters
• explicit links like https://bugs.python.org/issue2771 in the msg body
• implicit links like these added while rendering the msg on bpo

We want:

• to keep the plain text as plain, possibly non-monospaced, with markdown characters escaped
• explicit links to be clickable
• implicit links to be clickable
• code, tracebacks, terminal sessions, output to be monospaced, possibly highlighted

It seems that:

• <pre> or ```...``` can be used to show the code in a monospaced block:

  • the background looks different and

  • explicit links like https://bugs.python.org/issue2771 are not clickable

  • this also doesn't work [these](https://bugs.python.org/issue2771#msg154050)

  • and neither does #3

• <samp> can be used to make the text monospaced:
  • like this
  • links like https://bugs.python.org/issue2771 work
  • this and #3 work too
  • but all other MarkDown characters are also interpreted
  • also leading spaces are stripped...
  •     ...unless replaced with &nbsp;
  • each paragraph must also be wrapped by a separate <samp>
• All ASCII punctuation can be escaped
  • this will prevent things like __init__ turning into init
  • however this will break links too
• File links in traceback will be lost if use ```...``` to get syntax highlight, but can be preserved if we use <samp> (with no highlight)

If we want to implement option 2 from the first message, we could:

• wrap each paragraph in <samp>
• replace leading whitespace with  
• escape all punctuation (except in URLs)
  This should make the messages look similar to bpo: monospaced, no fancy markup/highlight, with working links.

If we want to implement option 3, we could parse each paragraph independently and:

• if it seems mostly text, only escape all punctuation (except in URLs)
• if it looks like code, wrap it within ```...``` to have a monospaced font
  • this will break all links in the block, including ones in tracebacks
• for code highlight, we could:
  • just use py by default for all blocks
  • use a simple heuristic (e.g. contain >>> / keywords like def) to detect Python, use text otherwise
  • use pygment to detect the lang
    • only Python/C/rst should be allowed, the rest should be text
    • might be a bit slow
• we might also be able to detect and allow some additional markup, like `...` or *...*, or perhaps even lists, and leave it unescaped
• making this undoable and/or storing the raw message somewhere shouldn't be an issue, since we can always get the original messages from bpo if we ever need them

I'll do some tests on a real world sample to see if we can reach a good compromise.