unicode-org / rust-discuss Goto Github PK
View Code? Open in Web Editor NEWOmnICU-SC: For discussion of i18n in Rust.
License: Other
OmnICU-SC: For discussion of i18n in Rust.
License: Other
It's good to have a Google Groups mailing list so we can share docs and calendar events and things. Typically for these things I make an alias in the chromium.org domain. This doesn't mean we're affiliated with Chromium; it's just the most convenient GSuite account to use since a lot of us already have accounts there, and it's the same GSuite that owns the mailing lists and docs for MFWG and TC39-TG2.
Any concerns with making [email protected]?
https://github.com/zbraniecki/pluralrules is currently maintained by me, but I'd prefer to grow the maintainers group.
It's in a good shape, with a helper crates to parse the pluralrules into Rust functions and a script to generate the rust tables based on CLDR JSON data which hopefully makes it easy to maintain and update to latest CLDR at all times.
I'd like to donate the crate to an org which would make it possible for others to update the crate as needed.
I started experimenting with DateTime formatter. The codebase is very messy, I apologize for that, but I believe I can share the initial performance results.
For the minimum POC I focused on a single locale (pl
) and 10 different combinations of dateStyle/timeStyle. I omitted two which require timezone names.
Since DateTime patterns take much more space than pluralrules or locale RTL/likelySubtags data which I worked with before, I experimented with three different models of loading data, as per #5:
For the binary scenario I used bincode
crate, fetched JSON CLDR, parsed the patterns, and serialized the resulting structure to a res
file, which I then loaded into memory at runtime.
I got the following results:
I'd appreciate if someone could try to replicate my measurements and verify the results.
If the results hold, I believe this to be one more evidence that investment in Rust based crates may lead to significant performance gains.
I haven't evaluate memory use, but the ca-gregory.json
file for pl
is 18201 bytes
and pl.res
generated from it is 2187 bytes
. I'd expect the latter to take the same amount in memory.
In order to maintain compatibility between ICU and Rust Unicode crates, we'd need to develop test data which is platform independent and can be shared between at least Java, C++ and Rust.
Let's kick off with the discussion on what do we want to do with https://github.com/open-i18n
It covers some of the ICU, in particular:
unic-char
: Unicode Character Tools.
unic-normal
: Unicode Normalization Forms
(UAX#15).
unic-segment
: Unicode Text Segmentation Algorithms
(UAX#29).
unic-emoji
: Unicode Emoji
(UTS#51).
I'm not sure how up to date those are to ICU 65 and well maintained. My impression was that Behnam is looking to grow the maintainers group and/or hand them over, but I'm not sure if they should be merged together with crates such as unic-langid
or intl_pluralrules
or some future date time format, or is it ok to keep the unicode characters crates separate from the higher level intl formatting and locale management.
Thoughts?
The three crates I maintain - intl_pluralrules, unic-langid and unic-locale currently all store their data in the source code tables generated as a separate step.
This provides a very good performance, but is inflexible and potentially may lead to data duplication.
It would be good for all crates around Unicode in Rust to have cohesive data loading models. I suggest two:
We'd need to design such format, write tooling for building such resource files and API for loading them into memory.
It may be quite early to start the conversation, but I prefer to have it early than having to later chase people around.
Most of the intl-related crates in Rust use MIT/Apache dual-license model.
Now, @jfkthame released hyphenation crate which uses MLP-2.0.
I filed jfkthame/mapped_hyph#6 to ask to add MIT/Apache, but I'm also wondering if that combo is reasonable for all stakeholders to move forward with.
https://github.com/zbraniecki/unic-locale contains two crates - unic-langid
and unic-locale
. The former separates out Unicode LanguageIdentifier parsing/modifying/serializing and the latter adds Unicode Extensions.
The former is quite feature complete on the well-formed
level, with optional features such as likely subtags and layout information (character orientation - LTR/RTL) and scripts to easily update the data to latest CLDR as needed.
I'd like to donate the crates to an org that will distribute the maintenance and reduce the bus factor.
Just opening an issue to let y'all know we're working on UTS #39 Unicode Security Mechanisms in Rust here, to be used in the Rust compiler.
(Perhaps we should maintain a list of all Rust projects maintained by members of this group in the readme? Once we start consolidating things it would probably be useful)
This is similar to issue #7 in that it's calling for support for a well understood library in Rust.
Also mentioned in https://www.arewewebyet.org/topics/i18n/
Full disclosure, my project (Fuchsia OS) needs ICU today, so we started by wrapping ICU4C, learning from other work in this space. See references below for details.
It's hard to do justice to all the references to prior art, but I'll try, and will try to backfill with more information where possible.
Bindings
Related issues
With a number of crates that use data from Unicode and CLDR tables, it would be benefitial to design a policy and scripting around cohesive updates of those to minimize the scenario where one crate uses Unicode 12, while another is stuck on Unicode 11 etc.
Some open questions are like:
rust-icu 65
would depend on all crates in versions using Unicode 12 and CLDR 36)?Hi all,
I wrote up an OmnICU Charter for sanctioning under the Unicode Consortium. Please take a look:
Please leave comments. Now is the best time to amend this charter so we can all come into agreement on the goals of an industry project under Unicode.
Regarding this repo: I could either (1) move this repo to unicode-org/omnicu, (2) leave this repo alone indefinitely and make a new repo unicode-org/omnicu, or (3) make a new unicode-org/omnicu and also move this repo into something like unicode-org/rust-discuss. Thoughts?
I think there is value for supporting well thought-out I18N APIs in Rust as well as in other languages. This is because no single person can be an all-around expert on all matters I18N just by the nature of the subject, and we should be well served if we all work together to minimize the amount of wheel reinvention. ECMA-402 in particular has had tremendous amount of work invested in it, and seems like a well understood basis for this work.
A few topics that seem of interest in the Rust world as of today:
Hi folks. Now that both unic-locale and rust_icu have language ID implementations, may I propose an exercise. Let's see what it would take to factor out one or a few common traits from the two implementations so that we can have lazy and/or eager parsing of the locales and basic manipulation.
WDYT? @zbraniecki , others?
I made a doc discussing the API surface for OmnICU in several host languages. I have a section about Rust.
https://docs.google.com/document/d/1tXACn0p2EuzSCJ0Gd8Nd1RV-DgdYG54N6lDwqYeQE4Q/edit#
One big open question is about named arguments delivery. In ECMAScript, we just use object literals and it's easy. In Rust, I suggested two options:
Thoughts?
Hi ๐,
Manish suggested I introduce myself to this effort via an issue, so here goes: I work at YesLogic on Prince. I've recently been replacing our C based Unicode/UCD 9.0 related code with Rust and updating it to Unicode 12.1.
We use lookup tables that trade off some space for faster lookups than the typical binary search approach used in the Rust standard library and Unicode crates in the ecosystem. My approach has been to use ucd-generate to generate simple space efficient tables and then use a build script generate the lookup tables from those.
Where possible I've extended existing crates, but I've also published several new crates. I'm waiting for some initial feedback on my first ucd-generate
pull request before opening more, although BurntSushi seems busy at the moment. The thought is that when all the ucd-generate
changes are merged I'll open PRs against the other crates that need that newer ucd-generate
version.
So far I have:
ucd-generate
Manish suggested this work and new crates might be relevant and from our point of view an, "ICU in Rust", is also of interest.
Regards,
Wes
@jfkthame wrote mapped_hyph
which, as far as I know, is not overlapping with any other Rust Intl crate!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.