Add hyphenate breaks at hyphens about hyphenation

[vshabanov]

Not yet sure what to do with it. Issue is more complex. Any non-alpha character could lead to hyphenation:

> H.hyphenate H.english_US "Compile.time"
["Com","pile.","time"]
> H.hyphenate H.english_US "Compile123time"
["Com","pile123","time"]

So as you asked some utility function to hyphenate whole text is needed.

Unfortunately it's quite complex task to properly separate words for hyphenation.

In my code (not yet opensource) I'm using this syntax to extract words for hyphenation:

space opening? alpha_mark+ punctuation* space

where
  space c = isSpace || c `elem` "-’—"
  opening c = c `elem` "([¡¿«‹“„‘‚”’‛»「『《〈【〔〖〘〚〝〟\"'"
  alpha_mark c = isAlpha c || isMark c

Then I'm checking that all letters in alpha_mark+ sequence are in hyphenator's language and there are no uppercase letters except first (to not break abbreviations).

And after hyphenation I'm removing hyphens between f and any char from "filjt" to not break font ligatures.

This works quite well for me but I'm not sure it worth to include such specific code to hyphenate package.

Oh, and I'm normalizing Unicode characters using text-icu (normalize NFC) before hyphenation. I don't think that hyphenate needs text-icu dependency.

Perhaps there's some generic algorithm that TeX is using to separate words for hyphenation. It could be used in hyphenate package (maybe with some settings for customization).

Or maybe it worth to hyphenate only alpha_mark blocks of characters.

[gwern]

As far as I know, browsers and most other things will not linebreak on most punctuation. For example,'/' will not linebreak and I have to have a rewrite pass to insert <wbr> at forward slashes. (I don't use a soft hyphen because there are a lot of slashes in source code snippets, and I do not want hyphens showing up, potentially misleading the reader into thinking that there's a CLI option or something there.) So adding soft hyphens at punctuation seems fine to me.

Checking for abbreviations seems potentially unnecessary given hyphenator's minimum size setting. No abbreviation of 1-3 letters could ever get a soft hyphen in my example, and I haven't seen any >3-letter ones get broken in practice.

This works quite well for me but I'm not sure it worth to include such specific code to hyphenate package.

Well, where else would it go? And such helper code is precisely what hyphenator users need. Not many people want to hyphenate a random Haskell string to admire it in ghci; anyone using hyphenator is presumably processing documents of some sort where all these considerations are what they need handled for them.