Comments (19)
If you defer to UTS29 it seems indeed this is defined, but would be good to have tests as they are a bit of an edge case.
from proposal-intl-segmenter.
Looks like lone surrogates are typically treated as control characters, which get a break on both sides. These are the Unicode semantics, but I haven't tested with various implementations yet.
from proposal-intl-segmenter.
@annevk great question. In general, I will say that 402 is underspecified when it comes to surrogates, only mentioned briefly on collator. @littledan maybe this is a good opportunity to improve that area all together.
from proposal-intl-segmenter.
I can't imagine how it could be better specified than those details in UTS29 above. If we weaken the definition of breaks to allow more diversity of conforming implementations, do we want to then go back and specify the behaviour for surrogates?
from proposal-intl-segmenter.
I will add it to my list of things to discuss in the next/first 402 meeting :) if we have time.
from proposal-intl-segmenter.
Looking again at the spec, I see that I've merged a patch to no longer reference UTS29, see ad6fa34 . This was done based on a request from Mozilla engineers (which?) to allow this to have line breaking which matched the surrounding platform, where theirs was not based on UTS29. I guess we could be more specific just about lone surrogates, but I don't really see why that's more important than, well, everything else that we're leaving just plain undefined.
from proposal-intl-segmenter.
I see, it would be nice to be a little stricter than that or at least figure out what's common, but I guess nobody wants to sign up for that.
Seems that the minimum you could require in such a case is that the algorithm in question has an answer for all code points and not just scalar values or some such.
from proposal-intl-segmenter.
@annevk I don't understand the details of why Firefox wants to use a different line breaking algorithm (for lower memory usage?) and exactly how they compare. But this is a bigger issue to discuss mostly in CSSWG.
I think the current spec basically does say that it should have an answer for all code points. Even in the more general form, it doesn't give the algorithm an opportunity to "error out" on some strings.
from proposal-intl-segmenter.
Hmm, so one thing that could be tested is that CSS and this use the same algorithm.
from proposal-intl-segmenter.
FWIW I need to fix #17 before that's a valid test.
from proposal-intl-segmenter.
cc @bakkot
from proposal-intl-segmenter.
We've discussed at a few Intl calls whether we should revert #18 or not. It seems like we're still waiting for more data from Mozilla about whether they would be able to ship this data or not. If we do decide to make normative references to UAX 29 and 14, that answers the question; otherwise, I think we should leave this implementation-dependent.
from proposal-intl-segmenter.
If there's still a question here, I think it could be resolved by adding text that just requires consistent successful handling for all input but leaves the specifics of such handling implementation-defined, inside or outside a note and with or without a non-normative reference to UAX #14.
from proposal-intl-segmenter.
What's the current status of Mozilla and ICU segmentation? Has anything changed? Cc @jswalden @zbraniecki
I don't see why we would reference UAX/UTS 14, as we no longer include line breaking. We could reference UTS 29, if Mozilla is OK with it.
from proposal-intl-segmenter.
@jfkthame you should maybe chime in here.
from proposal-intl-segmenter.
Is this only about grapheme cluster segmentation, or do we also need to consider the other types of segment breaks described in UTS 29?
For grapheme-cluster breaks, UTS 29 is clear there should be a break before and after a lone surrogate, thanks to rule GB999.
But for other kinds of segmentation, it looks like UTS 29 no longer fully addresses the issue of lone surrogate code units, as it was updated for Unicode 12 (see the Modifications section) to classify them as XX (unknown). And for word breaks, the behavior of XX is not specified in UTS 29:
characters with the Line_Break property values of [...] Unknown (XX) are assigned Word_Break property values based on criteria outside of the scope of this annex
It seems appropriate to me that there should be a word break before and after any lone surrogate, but I don't know if there's any spec that explicitly says so at this point.
As for Mozilla's implementation: I think we're fine with using UTS 29 for the definition of grapheme cluster boundaries. If our implementation fails to follow its rules, that would simply be a bug that we should fix.
I'm unsure about other levels of segmentation: I don't think we currently ship all the ICU code to support word segmentation, and our existing code is unlikely to exactly match it. (It looks quite primitive, actually.) Whether we're ready to add all the ICU segmentation code would be a product-level decision, weighing up the cost (in added download and install size) vs the benefit.
from proposal-intl-segmenter.
We'll leave implementations to do the right thing here with respect to boundary determination in any given locale, rather than overconstrain them.
from proposal-intl-segmenter.
For grapheme-cluster breaks, UTS 29 is clear there should be a break before and after a lone surrogate, thanks to rule GB999.
It doesn't seem so clear to me that GB999 means that. There is no mention of lone surrogates here, and while lone surrogates would fall under Any (as they are in none of the ranges in GraphemeBreakProperty.txt), Any is still unconditionally joined with Prepend, and possibly with SpacingMark, Extend, and ZWJ(!).
from proposal-intl-segmenter.
For grapheme-cluster breaks, UTS 29 is clear there should be a break before and after a lone surrogate, thanks to rule GB999.
It doesn't seem so clear to me that GB999 means that. There is no mention of lone surrogates here, and while lone surrogates would fall under Any (as they are in none of the ranges in GraphemeBreakProperty.txt), Any is still unconditionally joined with Prepend, and possibly with SpacingMark, Extend, and ZWJ(!).
Yes - on looking again now, that appears to be correct.
from proposal-intl-segmenter.
Related Issues (20)
- Advance to stage 3 HOT 7
- Advance to stage 4 HOT 5
- Should we throw exception when the string in Intl.Segmenter.prototype.segment ( string ) is not type string HOT 2
- Should segment data objects expose the context string? HOT 1
- FYI: ICU+WASM based polyfill ongoing work HOT 2
- Consistency with Number.range model HOT 5
- Indexed access and/or Symbol.slice support? HOT 2
- Why do we need to create a isWordLike: undefined in CreateSegmentDataObject If granularity is NOT "word" HOT 2
- Confusing fragment in README.md
- Adopt new GetOptions behavior
- Custom Dictionaries HOT 32
- Extensibility for non-ICU approaches? HOT 2
- Word segmenter with generic locale HOT 10
- Punctuation in the word segmenter
- No locale grapheme segmenter
- Line break support HOT 1
- Unicode Database and Related APIs HOT 1
- -
- Sentence break suppressions
- `granularity: "syllable"` HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from proposal-intl-segmenter.