burntsushi / utf8-ranges Goto Github PK

View Code? Open in Web Editor NEW

42.0 42.0 11.0 31 KB

Convert contiguous ranges of Unicode codepoints to UTF-8 byte ranges.

License: The Unlicense

Makefile 1.28% Rust 96.40% Shell 2.04% Vim Script 0.28%

utf8-ranges's People

Contributors

Stargazers

Watchers

Forkers

behnam mgeisler ignatenkobrain zackpierce dependabot-fixtures guillaumegomez mesalock-linux eclipseo backwardn iq-scm

utf8-ranges's Issues

Discontiguous ranges?

Great package, but how does one convert discontiguous ranges to a series of byte ranges? For example, [а-мо-я]? It's not unusual for regular expressions to have ranges with holes in them.

If you convert [а-м] and [о-я] separately, the corresponding byte ranges would intersect: [{208 208} {176 188}] and [{208 208} {190 191}].

`LLVM ERROR: out of memory` when building with `cargo build --target wasm32-unknown-unknown` on nightly

This is probably an issue with rustc, but this is the only project I was able to reliably reproduce it with:

➜  utf8-ranges git:(master) cargo build --target wasm32-unknown-unknown --verbose
   Compiling utf8-ranges v1.0.0 (file:///home/udoprog/repo/utf8-ranges)
     Running `rustc --crate-name utf8_ranges src/lib.rs --crate-type lib --emit=dep-info,link -C debuginfo=2 -C metadata=90af49223e8dd833 -C extra-filename=-90af49223e8dd833 --out-dir /home/udoprog/repo/utf8-ranges/target/wasm32-unknown-unknown/debug/deps --target wasm32-unknown-unknown -C incremental=/home/udoprog/repo/utf8-ranges/target/wasm32-unknown-unknown/debug/incremental -L dependency=/home/udoprog/repo/utf8-ranges/target/wasm32-unknown-unknown/debug/deps -L dependency=/home/udoprog/repo/utf8-ranges/target/debug/deps`
LLVM ERROR: out of memory
error: Could not compile `utf8-ranges`.

Caused by:
  process didn't exit successfully: `rustc --crate-name utf8_ranges src/lib.rs --crate-type lib --emit=dep-info,link -C debuginfo=2 -C metadata=90af49223e8dd833 -C extra-filename=-90af49223e8dd833 --out-dir /home/udoprog/repo/utf8-ranges/target/wasm32-unknown-unknown/debug/deps --target wasm32-unknown-unknown -C incremental=/home/udoprog/repo/utf8-ranges/target/wasm32-unknown-unknown/debug/incremental -L dependency=/home/udoprog/repo/utf8-ranges/target/wasm32-unknown-unknown/debug/deps -L dependency=/home/udoprog/repo/utf8-ranges/target/debug/deps` (signal: 6, SIGABRT: process abort signal)

Test suite executes UB by attempting to construct invalid chars

I see this crate is deprecated but since deprecating it you've published at least one version of fst which depends on it, and the latest commit in the fst repo still depends on this crate. So I guess I'm a bit confused by the deprecated status, but I'd also understand if you just close this as a non-issue 🤷

This code in the test suite is unconditional UB because it attempts to construct an invalid char. Miri immediately flags this (which is how I ran into this).

utf8-ranges/src/lib.rs

Lines 463 to 464 in 8d38a93

    
           for cp in 0xD800..0xE000 { 
        
               let c = unsafe { ::std::mem::transmute(cp) };

But you should also be aware that the standard library is veeery slooowly adding debug assertions for preconditions which are detectable at runtime. So these tests are unlikely to break any time soon, but due to this UB these tests could start failing at any time.

approaching 1.0

This crate is rather niche and doesn't enjoy wide use, so there's been little iteration on its API. With that said, its API is very small and there's no good reason to expect usage to increase within the near term, so I think we should cut a 1.0 release within the next few weeks.

burntsushi / utf8-ranges Goto Github PK

utf8-ranges's People

Contributors

Stargazers

Watchers

Forkers

utf8-ranges's Issues

Discontiguous ranges?

`LLVM ERROR: out of memory` when building with `cargo build --target wasm32-unknown-unknown` on nightly

Test suite executes UB by attempting to construct invalid chars

approaching 1.0

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	for cp in 0xD800..0xE000 {
	let c = unsafe { ::std::mem::transmute(cp) };