latex3 / unicode-data

Unicode case mapping and character class data for use by TeX

The expl3 (LaTeX3) Development Repository

Overview

The repository contains development material for expl3. This includes not only code to be developed into the expl3 kernel, but also a variety of test, documentation and more experimental material. All of this code works on top of LaTeX2e.

The following directories are present in the repository:

• l3kernel: code forms the expl3 kernel and all stable code. With a modern LaTeX2e kernel, this code is loaded during format creation; when using an older LaTeX2e kernel, this material is accessible using the expl3 package.
• l3backend: code for backend (driver) level interfaces across the expl3 codebase; none of this code has public interfaces, and so no distinction is made between stable and experimental code.
• l3packages: code which is written to be used on top of LaTeX2e to explore interfaces; this bundle is now made up of historical material, and the concepts have been migrated to the LaTeX2e kernel
• l3experimental: code which is written to be used on top of LaTeX2e to experiment with code and interface concepts. The interfaces for these packages are still under active discussion. Parts of this code may eventually be migrated to l3kernel.
• l3trial: material which is under very active development, for potential addition to l3kernel or l3experimental. Material in this directory may include potential replacements for existing modules, where large-scale changes are under-way. This code is not released to CTAN.
• l3leftovers: code which has been developed in the past by The LaTeX Project but is not suitable for use in its current form. Parts of this code may be used as the basis for new developments in l3kernel or l3experimental over time.

Support material for development is found in:

• support, which contains files for the automated test suite which are 'local' to the repository.

Documentation is found in:

• articles: discussion of concepts by team members for publication in TUGBoat or elsewhere.

The repository also contains the directory xpackages. This contain code which is being moved (broadly) l3experimental. Over time, xpackages is expected to be removed from the repository.

Issues

The issue tracker for expl3 is currently located on GitHub.

Build status

We use GitHub Actions as a hosted continuous integration service. For each commit, the build status is tested using the current release of TeX Live.

Current build status:

Development team

This code is developed by The LaTeX Project.

Copyright

This README file is copyright 2021-2024 The LaTeX Project.

unicode-data's Issues

CJ letters should belong to XeTeXcharclass 1

After loading load-unicode-xetex-classes.tex, we have:

• \XeTeXcharclass"3041=0 (U+3041 HIRAGANA LETTER SMALL A is a non-CJK)
• \XeTeXcharclass"3042=1 (U+3042 HIRAGANA LETTER A is a normal CJK)

But this is very strange, because U+3041 (small-a) is a full-fledged kana letter just like U+3042 (a), and at least must be treated as a CJK letter.

U+3041 belongs to the line break class ‘CJ’, and what UAX #14 states on this class is summarized as follows:

• If you want more line-break points (normal kinsoku), then treat CJ as ID.
• If you want less line-break points (strict kinsoku), then treat CJ as NS.

That is, CJ characters must have \XeTeXcharclass value 1 (normal) or 3 (strict). Of these two options, I recommend using the value 1, because the character class 3 is often seen as a group for punctuations and kana letters belonging to this group sometimes causes trouble in some package (which relies on the “old settings.”)

unicode 9.0.0 update?

seems unicode 9 is released. updates needed?
http://www.unicode.org/Public/UNIDATA/?C=M;O=A
(latest entries last) tx.

Data

Shaunicode

Adding Unicode script information

Hi,

could you add the Unicode Script Property files (Script.txt and ScriptExtension.txt, documented in Unicode Standard Annex #24) ?
They allow automatic identification of used Scripts and therefore can help with automatic font selection, especially in LuaTeX and maybe XeTeX.

OpenType Mirroring data file

For latex3/luaotfload#61 we need OpenType's "OpenType Mirroring Pairs List" and I am not sure if it should be shipped in luaotfload or in unicode-data:

It is a copy of Unicode's BidiMirroring.txt from Unicode 5.1.0 (it will never be updated). So it's not really a Unicode data file (at least not from current Unicode versions), but it still "feels" like one and doesn't really fit into luaotfload.

What do you think?

incomplete loop .. repeat while reading EastAsianWidth.txt

After the first loop of reading EastAsianWidth.txt, it never reaches the next line.
In othe words, after reading the line:

0000..001F;N     # Cc    [32] <control-0000>..<control-001F>

the loop is terminated unexpectedly. Possibly it is related to the nested loop .. repeat in parseunicodedataIII.

Again, it seems that line 188 of load-unicode-xetex-classes.tex should be corrected as follows:

\global\XeTeXcharclass\count0=\XeTeXcharclassID

as stated in the comment: class mappings may be over-ridden by defining \XeTeXcharclass<class>.