Bleach doesn't detect character encoding about bleach

[jsocol]

Honestly I don't think this is Bleach's responsibility. Do you have a use case where the app using Bleach doesn't know the incoming text charset?

I guess I'd call this a documentation issue and say if you know you're going to be passing in a bytestring with non-ASCII characters, convert it to a unicode object first.

[markmcclain]

Calling this a documentation issue is a reasonable resolution. This line should be changed:

"NB: Bleach accepts bytestrings or unicode, but it always returns unicode."

To me, it implies that bleach is going to attempt to do the right thing when converting to unicode. Maybe something similar to this:

NB: Given a bytestring or unicode, Bleach will always return unicode; however, the conversion from bytestring to unicode does not respect character encoding. If you are using a specific encoding, please convert the bytestring to unicode before using Bleach to ensure document integrity.

[jsocol]

Updated the README, and I also added an explicit cast to UTF8 for bytestrings, so your original example should just work now (as opposed to the implicit bytestring->unicode conversion that was happening). This makes clean() more consistent with linkify().

It would be relatively easy to add an encoding='utf-8' kwarg to both clean() and linkify(), that would be passed along to force_unicode(). Bleach still wouldn't do charset detection but you wouldn't need to do the decode() yourself.