encoding_rs an implementation of the (non-JavaScript parts of) the Encoding Standard written in Rust and used in Gecko (starting with Firefox 56).
Additionally, the mem
module provides various operations for dealing with
in-RAM text (as opposed to data that's coming from or going to an IO boundary).
The mem
module is a module instead of a separate crate due to internal
implementation detail efficiencies.
Due to the Gecko use case, encoding_rs supports decoding to and encoding from UTF-16 in addition to supporting the usual Rust use case of decoding to and encoding from UTF-8. Additionally, the API has been designed to be FFI-friendly to accommodate the C++ side of Gecko.
Specifically, encoding_rs does the following:
- Decodes a stream of bytes in an Encoding Standard-defined character encoding
into valid aligned native-endian in-RAM UTF-16 (units of
u16
/char16_t
). - Encodes a stream of potentially-invalid aligned native-endian in-RAM UTF-16
(units of
u16
/char16_t
) into a sequence of bytes in an Encoding Standard-defined character encoding as if the lone surrogates had been replaced with the REPLACEMENT CHARACTER before performing the encode. (Gecko's UTF-16 is potentially invalid.) - Decodes a stream of bytes in an Encoding Standard-defined character encoding into valid UTF-8.
- Encodes a stream of valid UTF-8 into a sequence of bytes in an Encoding Standard-defined character encoding. (Rust's UTF-8 is guaranteed-valid.)
- Does the above in streaming (input and output split across multiple buffers) and non-streaming (whole input in a single buffer and whole output in a single buffer) variants.
- Avoids copying (borrows) when possible in the non-streaming cases when decoding to or encoding from UTF-8.
- Resolves textual labels that identify character encodings in protocol text into type-safe objects representing the those encodings conceptually.
- Maps the type-safe encoding objects onto strings suitable for
returning from
document.characterSet
. - Validates UTF-8 (in common instruction set scenarios a bit faster for Web workloads than the standard library; hopefully will get upstreamed some day) and ASCII.
Additionally, encoding_rs::mem
does the following:
- Checks if a byte buffer contains only ASCII.
- Checks if a potentially-invalid UTF-16 buffer contains only Basic Latin (ASCII).
- Checks if a valid UTF-8, potentially-invalid UTF-8 or potentially-invalid UTF-16 buffer contains only Latin1 code points (below U+0100).
- Checks if a valid UTF-8, potentially-invalid UTF-8 or potentially-invalid UTF-16 buffer or a code point or a UTF-16 code unit can trigger right-to-left behavior (suitable for checking if the Unicode Bidirectional Algorithm can be optimized out).
- Combined versions of the above two checks.
- Converts valid UTF-8, potentially-invalid UTF-8 and Latin1 to UTF-16.
- Converts potentially-invalid UTF-16 and Latin1 to UTF-8.
- Converts UTF-8 and UTF-16 to Latin1 (if in range).
- Finds the first invalid code unit in a buffer of potentially-invalid UTF-16.
- Makes a mutable buffer of potential-invalid UTF-16 contain valid UTF-16.
- Copies ASCII from one buffer to another up to the first non-ASCII byte.
- Converts ASCII to UTF-16 up to the first non-ASCII byte.
- Converts UTF-16 to ASCII up to the first non-Basic Latin code unit.
Notably, the above feature list doesn't include the capability to wrap
a std::io::Read
, decode it into UTF-8 and presenting the result via
std::io::Read
. The encoding_rs_io
crate provides that capability.
Please see the file named COPYRIGHT.
Generated API documentation is available online.
An FFI layer for encoding_rs is available as a separate crate. The crate comes with a demo C++ wrapper using the C++ standard library and GSL types.
For the Gecko context, there's a C++ wrapper using the MFBT/XPCOM types.
These bindings do not cover the mem
module.
There are currently these optional cargo features:
Enables SSE2 acceleration on x86 and x86_64 and NEON acceleration on Aarch64 and ARMv7. Enabling this cargo feature is recommended when building for x86, x86_64, ARMv7 or Aarch64. The intention is for the functionality enabled by this feature to become the normal on-by-default behavior once portable SIMD becames part of stable Rust.
Enabling this feature breaks the build unless the target is x86 with SSE2
(Rust's default 32-bit x86 target, i686
, has SSE2, but Linux distros may
use an x86 target without SSE2, i.e. i586
in rustup
terms), ARMv7 or
thumbv7 with NEON (-C target_feature=+neon
), x86_64 or Aarch64.
Used by Firefox.
Enables support for serializing and deserializing &'static Encoding
-typed
struct fields using Serde.
Not used by Firefox.
A catch-all option for enabling the fastest legacy encode options. Does not affect decode speed or UTF-8 encode speed.
At present, this option is equivalent to enabling the following options:
fast-hangul-encode
fast-hanja-encode
fast-kanji-encode
fast-gb-hanzi-encode
fast-big5-hanzi-encode
Adds 176 KB to the binary size.
Not used by Firefox.
Changes encoding precomposed Hangul syllables into EUC-KR from binary search over the decode-optimized tables to lookup by index making Korean plain-text encode about 4 times as fast as without this option.
Adds 20 KB to the binary size.
Does not affect decode speed.
Not used by Firefox.
Changes encoding of Hanja into EUC-KR from linear search over the decode-optimized table to lookup by index. Since Hanja is practically absent in modern Korean text, this option doesn't affect perfomance in the common case and mainly makes sense if you want to make your application resilient agaist denial of service by someone intentionally feeding it a lot of Hanja to encode into EUC-KR.
Adds 40 KB to the binary size.
Does not affect decode speed.
Not used by Firefox.
Changes encoding of Kanji into Shift_JIS, EUC-JP and ISO-2022-JP from linear
search over the decode-optimized tables to lookup by index making Japanese
plain-text encode to legacy encodings 30 to 50 times as fast as without this
option (about 2 times as fast as with less-slow-kanji-encode
).
Takes precedence over less-slow-kanji-encode
.
Adds 36 KB to the binary size (24 KB compared to less-slow-kanji-encode
).
Does not affect decode speed.
Not used by Firefox.
Makes JIS X 0208 Level 1 Kanji (the most common Kanji in Shift_JIS, EUC-JP and ISO-2022-JP) encode less slow (binary search instead of linear search) making Japanese plain-text encode to legacy encodings 14 to 23 times as fast as without this option.
Adds 12 KB to the binary size.
Does not affect decode speed.
Not used by Firefox.
Changes encoding of Hanzi in the CJK Unified Ideographs block into GBK and
gb18030 from linear search over a part the decode-optimized tables followed
by a binary search over another part of the decode-optimized tables to lookup
by index making Simplified Chinese plain-text encode to the legacy encodings
100 to 110 times as fast as without this option (about 2.5 times as fast as
with less-slow-gb-hanzi-encode
).
Takes precedence over less-slow-gb-hanzi-encode
.
Adds 36 KB to the binary size (24 KB compared to less-slow-gb-hanzi-encode
).
Does not affect decode speed.
Not used by Firefox.
Makes GB2312 Level 1 Hanzi (the most common Hanzi in gb18030 and GBK) encode less slow (binary search instead of linear search) making Simplified Chinese plain-text encode to the legacy encodings about 40 times as fast as without this option.
Adds 12 KB to the binary size.
Does not affect decode speed.
Not used by Firefox.
Changes encoding of Hanzi in the CJK Unified Ideographs block into Big5 from
linear search over a part the decode-optimized tables to lookup by index
making Traditional Chinese plain-text encode to Big5 105 to 125 times as fast
as without this option (about 3 times as fast as with
less-slow-big5-hanzi-encode
).
Takes precedence over less-slow-big5-hanzi-encode
.
Adds 40 KB to the binary size (20 KB compared to less-slow-big5-hanzi-encode
).
Does not affect decode speed.
Not used by Firefox.
Makes Big5 Level 1 Hanzi (the most common Hanzi in Big5) encode less slow (binary search instead of linear search) making Traditional Chinese plain-text encode to Big5 about 36 times as fast as without this option.
Adds 20 KB to the binary size.
Does not affect decode speed.
Not used by Firefox.
For decoding to UTF-16, the goal is to perform at least as well as Gecko's old uconv. For decoding to UTF-8, the goal is to perform at least as well as rust-encoding. These goals have been achieved.
Encoding to UTF-8 should be fast. (UTF-8 to UTF-8 encode should be equivalent
to memcpy
and UTF-16 to UTF-8 should be fast.)
Speed is a non-goal when encoding to legacy encodings. By default, encoding to legacy encodings should not be optimized for speed at the expense of code size as long as form submission and URL parsing in Gecko don't become noticeably too slow in real-world use.
In the interest of binary size, by default, encoding_rs does not have encode-specific data tables beyond 32 bits of encode-specific data for each single-byte encoding. Therefore, encoders search the decode-optimized data tables. This is a linear search in most cases. As a result, by default, encode to legacy encodings varies from slow to extremely slow relative to other libraries. Still, with realistic work loads, this seemed fast enough not to be user-visibly slow on Raspberry Pi 3 (which stood in for a phone for testing) in the Web-exposed encoder use cases.
See the cargo features above for optionally making CJK legacy encode fast.
A framework for measuring performance is available separately.
It is a goal to support the latest stable Rust, the latest nightly Rust and the version of Rust that's used for Firefox Nightly (currently 1.29.0). These are tested on Travis.
Additionally, beta and the oldest known to work Rust version (currently
1.29.0) are tested on Travis. The oldest Rust known to work is tested as
a canary so that when the oldest known to work no longer works, the change
can be documented here. At this time, there is no firm commitment to support
a version older than what's required by Firefox. The oldest supported Rust
is expected to move forward rapidly when packed_simd
can replace the simd
crate without performance regression.
A compatibility layer that implements the rust-encoding API on top of encoding_rs is provided as a separate crate (cannot be uploaded to crates.io). The compatibility layer was originally written with the assuption that Firefox would need it, but it is not currently used in Firefox.
To regenerate the generated code:
- Have Python 2 installed.
- Clone
https://github.com/hsivonen/encoding_c
next to theencoding_rs
directory. - Clone
https://github.com/whatwg/encoding
next to theencoding_rs
directory. - Checkout revision
f381389
of theencoding
repo. - With the
encoding_rs
directory as the working directory, runpython generate-encoding-data.py
.
- Design the low-level API.
- Provide Rust-only convenience features.
- Provide an stl/gsl-flavored C++ API.
- Implement all decoders and encoders.
- Add unit tests for all decoders and encoders.
- Finish BOM sniffing variants in Rust-only convenience features.
- Document the API.
- Publish the crate on crates.io.
- Create a solution for measuring performance.
- Accelerate ASCII conversions using SSE2 on x86.
- Accelerate ASCII conversions using ALU register-sized operations on
non-x86 architectures (process an
usize
instead ofu8
at a time). - Split FFI into a separate crate so that the FFI doesn't interfere with LTO in pure-Rust usage.
- Compress CJK indices by making use of sequential code points as well as Unicode-ordered parts of indices.
- Make lookups by label or name use binary search that searches from the end of the label/name to the start.
- Make labels with non-ASCII bytes fail fast.
-
Parallelize UTF-8 validation using Rayon.(This turned out to be a pessimization in the ASCII case due to memory bandwidth reasons.) - Provide an XPCOM/MFBT-flavored C++ API.
- Investigate accelerating single-byte encode with a single fast-tracked range per encoding.
- Replace uconv with encoding_rs in Gecko.
- Implement the rust-encoding API in terms of encoding_rs.
- Add SIMD acceleration for Aarch64.
- Investigate the use of NEON on 32-bit ARM.
- Investigate Björn Höhrmann's lookup table acceleration for UTF-8 as adapted to Rust in rust-encoding.
- Add actually fast CJK encode options.
- Changed minimum Rust requirement to 1.29.0 (for the ability to refer
to the interior of a
static
when defining anotherstatic
). (Semver breaking change.) - Explicitly aligned the lookup tables for single-byte encodings to cache lines in the hope of freeing up one cache line for other data. (Perhaps the tables were already aligned and this is placebo.)
- Added 32 bits of encode-oriented data for each single-byte encoding. The change was performance-neutral for non-Latin1-ish Latin legacy encodings, improved Latin1-ish and Arabic legacy encode speed somewhat (new speed is 2.4x the old speed for German, 2.3x for Arabic, 1.7x for Portuguese and 1.4x for French) and improved non-Latin1, non-Arabic legacy single-byte encode a lot (7.2x for Thai, 6x for Greek, 5x for Russian, 4x for Hebrew).
- Added compile-time options for fast CJK legacy encode options (at the cost of binary size (up to 176 KB) and run-time memory usage). These options still retain the overall code structure instead of rewriting the CJK encoders totally, so the speed isn't as good as what could be achieved by using even more memory / making the binary even langer.
- Add method
is_single_byte()
onEncoding
.
- Disabled a unit test that tests a panic condition when the assertion being tested is disabled.
- Made
--features simd-accel
work with stable-channel compiler to simplify the Firefox build system.
- Made the
is_foo_bidi()
not treat U+FEFF (ZERO WIDTH NO-BREAK SPACE aka. BYTE ORDER MARK) as right-to-left. - Made the
is_foo_bidi()
functions reporttrue
if the input contains Hebrew presentations forms (which are right-to-left but not in a right-to-left-roadmapped block).
- Fixed a panic in the UTF-16LE/UTF-16BE decoder when decoding to UTF-8.
- Temporarily removed the debug assertion added in version 0.8.5 from
convert_utf16_to_latin1_lossy
.
- If debug assertions are enabled but fuzzing isn't enabled, lossy conversions
to Latin1 in the
mem
module assert that the input is in the range U+0000...U+00FF (inclusive). - In the
mem
module provide conversions from Latin1 and UTF-16 to UTF-8 that can deal with insufficient output space. The idea is to use them first with an allocation rounded up to jemalloc bucket size and do the worst-case allocation only if the jemalloc rounding up was insufficient as the first guess.
- Fix SSE2-specific,
simd-accel
-specific memory corruption introduced in version 0.8.1 in conversions between UTF-16 and Latin1 in themem
module.
- Removed an
#[inline(never)]
annotation that was not meant for release.
- Made non-ASCII UTF-16 to UTF-8 encode faster by manually omitting bound checks and manually adding branch prediction annotations.
- Tweaked loop unrolling and memory alignment for SSE2 conversions between
UTF-16 and Latin1 in the
mem
module to increase the performance when converting long buffers.
- Changed the minimum supported version of Rust to 1.21.0 (semver breaking change).
- Flipped around the defaults vs. optional features for controlling the size vs. speed trade-off for Kanji and Hanzi legacy encode (semver breaking change).
- Added NEON support on ARMv7.
- SIMD-accelerated x-user-defined to UTF-16 decode.
- Made UTF-16LE and UTF-16BE decode a lot faster (including SIMD acceleration).
- Add the
mem
module. - Refactor SIMD code which can affect performance outside the
mem
module.
- When encoding from invalid UTF-16, correctly handle U+DC00 followed by another low surrogate.
- Make
replacement
a label of the replacement encoding. (Spec change.) - Remove
Encoding::for_name()
. (Encoding::for_label(foo).unwrap()
is now close enough after the above label change.) - Remove the
parallel-utf8
cargo feature. - Add optional Serde support for
&'static Encoding
. - Performance tweaks for ASCII handling.
- Performance tweaks for UTF-8 validation.
- SIMD support on aarch64.
- Make
Encoder::has_pending_state()
public. - Update the
simd
crate dependency to 0.2.0.
- Reserve enough space for NCRs when encoding to ISO-2022-JP.
- Correct max length calculations for multibyte decoders.
- Correct max length calculations before BOM sniffing has been performed.
- Correctly calculate max length when encoding from UTF-16 to GBK.
- Don't prepend anything when gb18030 range decode fails. (Spec change.)
- Correcly handle the case where the first buffer contains potentially partial BOM and the next buffer is the last buffer.
- Decode byte
7F
correctly in ISO-2022-JP. - Make UTF-16 to UTF-8 encode write closer to the end of the buffer.
- Implement
Hash
forEncoding
.
- Map half-width katakana to full-width katana in ISO-2022-JP encoder. (Spec change.)
- Give
InputEmpty
correct precedence overOutputFull
when encoding with replacement and the output buffer passed in is too short or the remaining space in the output buffer is too small after a replacement.
- Correct max length calculation when a partial BOM prefix is part of the decoder's state.
- Correct max length calculation in various encoders.
- Correct max length calculation in the UTF-16 decoder.
- Derive
PartialEq
andEq
for theCoderResult
,DecoderResult
andEncoderResult
types.
- Avoid panic when encoding with replacement and the destination buffer is too short to hold one numeric character reference.
- Add support for 32-bit big-endian hosts. (For real this time.)
- Fix a panic from subslicing with bad indices in
Encoder::encode_from_utf16
. (Due to an oversight, it lacked the fix thatEncoder::encode_from_utf8
already had.) - Micro-optimize error status accumulation in non-streaming case.
- Avoid panic near integer overflow in a case that's unlikely to actually happen.
- Address Clippy lints.
- Make the methods for computing worst-case buffer size requirements check for integer overflow.
- Upgrade rayon to 0.7.0.
- Reorder methods for better documentation readability.
- Add support for big-endian hosts. (Only 64-bit case actually tested.)
- Optimize the ALU (non-SIMD) case for 32-bit ARM instead of x86_64.
- Avoid allocating an excessively long buffers in non-streaming decode.
- Fix the behavior of ISO-2022-JP and replacement decoders near the end of the output buffer.
- Annotate the result structs with
#[must_use]
.
- Split FFI into a separate crate.
- Performance tweaks.
- CJK binary size and encoding performance changes.
- Parallelize UTF-8 validation in the case of long buffers (with optional
feature
parallel-utf8
). - Borrow even with ISO-2022-JP when possible.
- Fix moving pointers to alignment in ALU-based ASCII acceleration.
- Fix errors in documentation and improve documentation.
- Fix UTF-8 to UTF-16 decode for byte sequences beginning with 0xEE.
- Make UTF-8 to UTF-8 decode SSE2-accelerated when feature
simd-accel
is used. - When decoding and encoding ASCII-only input from or to an ASCII-compatible encoding using the non-streaming API, return a borrow of the input.
- Make encode from UTF-16 to UTF-8 faster.
- Change the references to the instances of
Encoding
fromconst
tostatic
to make the referents unique across crates that use the refernces. - Introduce non-reference-typed
FOO_INIT
instances ofEncoding
to allow foreign crates to initializestatic
arrays with references toEncoding
instances even under Rust's constraints that prohibit the initialization of&'static Encoding
-typed array items with&'static Encoding
-typedstatics
. - Document that the above two points will be reverted if Rust changes
const
to work so that cross-crate usage keeps the referents unique. - Return
Cow
s from Rust-only non-streaming methods for encode and decode. Encoding::for_bom()
returns the length of the BOM.- ASCII-accelerated conversions for encodings other than UTF-16LE, UTF-16BE, ISO-2022-JP and x-user-defined.
- Add SSE2 acceleration behind the
simd-accel
feature flag. (Requires nightly Rust.) - Fix panic with long bogus labels.
- Map 0xCA to U+05BA in windows-1255. (Spec change.)
- Correct the end of the Shift_JIS EUDC range. (Spec change.)
- Polish FFI documentation.
- Fix UTF-16 to UTF-8 encode.
- Add
Encoder.encode_from_utf8_to_vec_without_replacement()
.
-
Add
Encoding.is_ascii_compatible()
. -
Add
Encoding::for_bom()
. -
Make
==
forEncoding
use name comparison instead of pointer comparison, because uses of the encoding constants in different crates result in different addresses and the constant cannot be turned into statics without breaking other things.
The initial release.