mojibake's Introduction

mojibake

Encode and decode arbitrary bytes as a sequence of emoji optimized to produce the smallest number of graphemes.

Description

This is not a space efficient library.

Generally services(Twitter,Mastodon,etc) will restrict the number of characters you're allowed to submit based on the grapheme count, not the literal character count. Singular emoji graphemes often consist of multi byte sequences that include multiple characters.

Therefore, if you can encode more data in a smaller number of graphemes, you can transmit more information while also having far more bytes than you otherwise would.

There are at least 2048 unique emoji graphemes in the unicode specification. Therefore an emoji is actually just an 11 bit unsigned integer with extra steps.

This library packs bytes bytes into 11 bit unsigned integers, which are then mapped to sequences of unicode characters that display as a single grapheme.

Example

Original Text:
 Value: Shrek 2 was the greatest film ever made!!
 Bytes: 41,
 Characters: 41,
 Graphemes: 41

Mojibake Encoded:
 Value: 🇻🇳👌🏿🪀🔶🫳🏿🧏🏻📼🕺🏾🤛🏻🦺🤵🏽👦🏼🗄️💆🏿⚗️↗️2️⃣🧥🤵🏻🕤🙆🫚🪙😟🇦🇪🫳🏽🇸🇲😹🏴󠁧󠁢󠁳󠁣󠁴󠁿🛌🏻
 Bytes: 210,
 Characters: 55,
 Graphemes: 30

Decoded Text:
 Value: Shrek 2 was the greatest film ever made!!
 Bytes: 41,
 Characters: 41,
 Graphemes: 41

mojibake's People

Contributors

Stargazers

mojibake's Issues

Support encoding and decoding from streams

Fails to decode specific sequence

decode(encode(vec![0u8, 223u8, 124u8])) returns None.

Add a command line client

In addition to the library, I'd also like to add a command line application that can encode and decode values/files.

Add encoding optimizing for Grapheme Clusters instead of Graphemes

While services often use grapheme count for character limits, the better analog for number of visual elements is grapheme clusters.

An encoding that takes advantage of zero-width joiner (ZWG) to encode grapheme clusters made of multiple graphemes (e.g. gender, skin tone modifiers) should improve the visual density of encoded information. As a bonus, this will also increase the diversity of generated emojis.

Reference

Recommend Projects