dlon / html2markdown

Conservatively convert html to markdown

html2markdown

Experimental

Purpose: Converts html to markdown while preserving unsupported html markup. The goal is to generate markdown that can be converted back into html. This is the major difference between html2markdown and html2text. The latter doesn't purport to be reversible.

Usage example

import html2markdown
print html2markdown.convert('<h2>Test</h2><pre><code>Here is some code</code></pre>')

Output:

## Test

    Here is some code

Information and caveats

Does not convert the content of block-type tags other than <p> -- such as <div> tags -- into Markdown

It does convert to markdown the content of inline-type tags, e.g. <span>.

Input: <div>this is stuff. <strong>stuff</strong></div>

Result: <div>this is stuff. <strong>stuff</strong></div>

Input: <p>this is stuff. <strong>stuff</strong></p>

Result: this is stuff. __stuff__ (surrounded by a newline on either side)

Input: <span style="text-decoration:line-through;">strike <strong>through</strong> some text</span> here

Result: <span style="text-decoration:line-through;">strike __through__ some text</span> here

Except in unprocessed block-type tags, formatting characters are escaped

Input: <p>**escape me?**</p> (in html, we would use <strong> here)

Result: \*\*escape me?\*\*

Input: <span>**escape me?**</span>

Result: <span>\*\*escape me?\*\*</span>

Input: <div>**escape me?**</div>

Result: <div>**escape me?**</div> (block-type)

Attributes not supported by Markdown are kept

Example: <a href="http://myaddress" title="click me"><strong>link</strong></a>

Result: [__link__](http://myaddress "click me")

Example: <a onclick="javascript:dostuff()" href="http://myaddress" title="click me"><strong>link</strong></a>

Result: <a onclick="javascript:dostuff()" href="http://myaddress" title="click me">__link__</a> (the attribute onclick is not supported, so the tag is left alone)

Limitations

• Tables are kept as html.

Changes

0.1.7:

• Improved handling of inline tags.
• Fix: Ignore <a> tags without an href attribute.
• Improve escaping.

0.1.6: Added tests and support for Python versions below 2.7.

0.1.5: Fix Unicode issue in Python 3.

0.1.0: First version.

html2markdown's Issues

Wrong markup

When I try to convert HTML, then I get the wrong output.

Example:

In [1]: a = '<blockquote><a href="url_address">text</a></blockquote>'
In [2]: html2markdown.convert(a)
Out[2]: ' [text](url_address'

But if I add whitespace before closing tag, then everything works fine except for a space at the beginning of a line:

In [1]: a = '<blockquote><a href="url_address">text</a> </blockquote>'
In [2]: html2markdown.convert(a)
Out[2]: ' [text](url_address)'

Maybe you know how to fix it?

Getting an AttributeError in Python 2.6 with sys.version_info.major

In Python 2.6 sys.version_info doesn't have a major attribute.

Here is the stack trace and output:

>>> import html2markdown
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/rally_jira_sync/venv/lib/python2.6/site-packages/html2markdown.py", line 10, in <module>
    if sys.version_info.major > 2:
AttributeError: 'tuple' object has no attribute 'major'
>>> import sys
>>> sys.version_info
(2, 6, 6, 'final', 0)
>>> sys.version_info.major
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'tuple' object has no attribute 'major'
>>> dir(sys.version_info)
['__add__', '__class__', '__contains__', '__delattr__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__getslice__', '__gt__', '__hash__', '__init__', '__iter__', '__le__', '__len__', '__lt__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmul__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'count', 'index']

The line in question could be changed from:

if sys.version_info.major > 2:

To:

if sys.version_info[0] > 2:

To support Python 2.6.

Not sure how supported this library is, but just submitting the issue to see if I get a reply.

Thanks

Implement standard <table> (i.e. without attributes)

Preserve <pre> line formatting

Converting code blocks with formatting seems to strip line breaks and indentation. E.g. when attempting to convert this HTML:

<pre><code>import random


num = random.randint(1, 10)
guess = None

while guess != num:
    guess = input("guess a number between 1 and 10: ")
    guess = int(guess)
    
    if guess == num:
        print("congratulations! you won!")
        break
    else:
        print("nope, sorry. try again!")</code></pre>

It turns into the following Markdown:

    import random


    num = random.randint(1, 10) guess = None

    while guess != num: guess = input("guess a number between 1 and 10: ") guess = int(guess) if guess == num: print("congratulations! you won!") break else: print("nope, sorry. try again!")

Render <pre><code> constructs as fenced code blocks

Markdown that uses backticks to indicate code blocks instead of indentation has the advantage that it is very easy to add language-identifier information to it, e.g.:

```python
print("hello")
```

Is there an option to convert <pre><code> HTML into fenced code blocks, or would you be interested to add it?

Fails to encode (replace) emojis

Received this from an Apple mail client writing emojis in HTML mails. I know, I know… but the tool could handle this better.

% echo "��" | html2markdown - windows-1252
Traceback (most recent call last):              
  File "/usr/bin/html2markdown", line 33, in <module>
    sys.exit(load_entry_point('html2text==2020.1.16', 'console_scripts', 'html2text')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/html2text/cli.py", line 306, in main
    sys.stdout.write(h.handle(html))
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 0-1: surrogates not allowed

For a start, it would help to know a bit more about where the error occurs, i.e. maybe print some context?

Or have something like --encode-errors=replace?

not support python3?

not support python3?

Reference-style links?

This is great. Are reference-style links on the roadmap? Inline works great for bulk conversions, this would be more of a nice-to-have thing.

Ensure that inline-elements are handled properly when styled (or <audio> has no controls)