khaledhosny / harf Goto Github PK
View Code? Open in Web Editor NEWA HarfBuzz-based font loader and shaper for HarfTeX
License: GNU General Public License v2.0
A HarfBuzz-based font loader and shaper for HarfTeX
License: GNU General Public License v2.0
HarfBuzz has support for OpenType debugger that shows what lookups are applied in what order and their input/output. See if we can make any use of that.
\fontspec{Noto Color Emoji}[Scale=2] abc
in the test-latex.tex gives
! error: (file XXX/fonts/truetype/NotoColorE
moji.ttf) (type 2): there are no glyphs in the subset
! ==> Fatal error occurred, no output PDF file produced!
Once harfbuzz/harfbuzz#1354 is in, figure out what we can do with this API, if any.
If no mode or mode=harf
is used the tlig-ligatures don't work. Does that mean that if no mode is set, harf mode is enforced? And could harf mode support tlig?
(The current behaviour is a problem for the fonts preset by the latex format in the TU-fd-files, as they don't set a mode currently, but this can naturally be changed).
\documentclass{article}
\usepackage{harfload}
\begin{document}
{\small a--b}
\font\test={name:texgyretermes:+tlig;}
\test
a--b
\font\test={name:texgyrebonum:mode=harf;+tlig;}
\test
a--b
\font\test={name:texgyreheros:mode=node;+tlig;}
\test
a--b
\font\test={name:texgyreheros:mode=base;+tlig;}
\test
a--b
\end{document}
Depends on #14.
Consider using l3build and having a real test suite.
I load noto color emoji font, but it fails to generate PDF correctly
\newfontfamily{\fallbackfont}{NotoColorEmoji.ttf}[RawFeature={mode=harf}]
warning (node filter): error: ./harf-node.lua:741: attempt to compare number with boolean
I fixed it locally with this change, but not sure if this will have the correct effect in all cases:
diff --git a/src/harf-node.lua b/src/harf-node.lua
index 24f6699..2cb2690 100644
--- a/src/harf-node.lua
+++ b/src/harf-node.lua
@@ -738,12 +738,14 @@ local function tonodes(head, current, run, glyphs, color)
-- May be it is checking for the italic correction before we have had
-- loaded the glyph?
local prevchar, prevfontid = isglyph(current)
- if prevchar > 0 then
- local prevfontdata = font.getfont(prevfontid)
- local prevcharacters = prevfontdata and prevfontdata.characters
- local italic = prevcharacters and prevcharacters[prevchar].italic
- if italic then
- setkern(n, italic)
+ if type(prevchar) ~= 'boolean' then
+ if prevchar > 0 then
+ local prevfontdata = font.getfont(prevfontid)
+ local prevcharacters = prevfontdata and prevfontdata.characters
+ local italic = prevcharacters and prevcharacters[prevchar].italic
+ if italic then
+ setkern(n, italic)
+ end
end
end
head, current = insertafter(head, current, n)
I am not sure if the below is a problem in harflatex. When I compile the below latex file
\begin{filecontents*}{\jobname.xmpdata}
\Title{test}
\Author{author}
\end{filecontents*}
\RequirePackage{harfload}
\documentclass[luatex,unicode,a4paper,12pt]{article}
\usepackage[a-2u]{pdfx}
\usepackage{fontspec}
\setmainfont{Segoe UI Emoji}[
RawFeature={mode=harf;+dist;+ccmp},
BoldFont={Segoe UI Bold},
ItalicFont={Segoe UI Italic},
BoldItalicFont={Segoe UI Bold Italic}]
\begin{document}
\section{Test}
Test 😃.
\end{document}
the generated PDF fails with the PDF/A validation at
https://www.pdf-online.com/osa/validate.aspx
with error
File | pdfa-test4.pdf
-- | --
Compliance | pdfa-2u
Result | Document does not conform to PDF/A.
Details | Validating file "pdfa-test4.pdf" for conformance level pdfa-2u
The Unicode for cid 9687 is unknown.
The Unicode for cid 9688 is unknown.
The Unicode for cid 9689 is unknown.
The Unicode for cid 9690 is unknown.
The Unicode for cid 9691 is unknown.
The document does not conform to the requested standard.
The document contains fonts without appropriate character to unicode mapping information (ToUnicode maps).The document does not conform to the PDF/A-2u standard.Done.
On the other hand, lualatex and the following latex file generated a PDF without validation error.
\begin{filecontents*}{\jobname.xmpdata}
\Title{test}
\Author{author}
\end{filecontents*}
\documentclass[luatex,unicode,a4paper,12pt]{article}
\usepackage[a-2u]{pdfx}
\usepackage{fontspec}
\setmainfont{Segoe UI Emoji}[
RawFeature={+dist;+ccmp},
BoldFont={Segoe UI Bold},
ItalicFont={Segoe UI Italic},
BoldItalicFont={Segoe UI Bold Italic}]
\begin{document}
\section{Test \textnormal{😃}}
Test 😃.
\end{document}
Two generated PDF files are attached below:
pdfa-test5.pdf
pdfa-test4.pdf
The below is probably somewhat related to khaledhosny/harftex#11.
In beamer.cls, color of text can be white. After use of emoji in Segoe UI Emoji font, the color of text becomes black. A short example is below. PDF generated by harflatex 0.4.1 is attached below.
\documentclass{beamer}
\usepackage{harfload}
\usepackage{fontspec}
\setsansfont{Segoe UI Emoji}[
RawFeature={mode=harf,+dist;+ccmp}]
\usetheme{Madrid}
\begin{document}
\begin{frame}{text 😂 text after emoji}
\end{frame}
\end{document}
Fonts with Em Size 1000 work fine (Fontforge: Element→Font Info→General→Em Size), but other values have size problems. In this case Adobe Arabic has the value 2048.
\documentclass{article}
\usepackage{harfload}
\usepackage{fontspec}
\setmainfont{AdobeArabic-Regular.otf}[RawFeature={mode=harf}]
\begin{document}
åäöüñÅÄÖÜÑ
اللُّغَة العَرَبِيّة هي أكثر اللغات تحدثاً ونطقاً
\end{document}
Git revisions 7657fe3 (harf) and e11c8e8 (harftex)
./build.sh --tlopt="-C --with-system-cairo --with-system-freetype2 --with-system-gd --with-system-gmp --with-system-graphite --with-system-harfbuzz --with-system-icu --with-system-libpaper --with-system-libpng --with-system-mpfr --with-system-pixman --with-system-poppler --with-system-potrace --with-system-teckit --with-system-xpdf --with-system-zlib --with-system-zziplib --with-x-dvi-toolkit=xaw --build=x86_64-linuxmusl --host=x86_64-linuxmusl --target=x86_64-linuxmusl --datarootdir=/opt/texlive/2019 --prefix=/opt/texlive/2019 --bindir=/opt/texlive/2019/bin/ --libdir=/usr/lib --includedir=/usr/include --mandir=/usr/share/man --infodir=/usr/share/info --disable-native-texlive-build --enable-shared --disable-static --enable-largefile --disable-dependency-tracking --with-pic"
HarfBuzz does not expose much of the OS/2
table, but it is a small and simple table, so we can read it in Lua for now.
We currently calculate glyph bounding box for all glyphs in the font at font loading time, but this seems to be a waste; we should do it instead after shaping and for glyphs that are actually used in the document.
Check different space characters and whether they are handled correctly or need special handling.
Set LuaTeX’s char tounicode
or use PDF’s /ActuaText
employing a strategy similar to the one I used in LibreOffice:
Should also help with supporting COLR
/CPAL
fonts.
For missing characters we currently output .notdef
glyph (glyph 0 in the font), this is usually a box glyph or something similar which helps give visual indication that a character is not supported by the font (in contrast to traditional TeX behavior of outputting nothing).
This have the downside that messing character log messages are not shown, so we need to fix this.
This MWE works with LuaTeX but breaks the PDF with HarfTeX. The font has invalid PostScript name, but still works in all other programs I tried.
\input harfload.sty \font\HKfont"[Free-HK-Kai_4700-v1.02.ttf]:mode=harf;" \HKfont 骨窗次 \bye
I see stuff like:
Underfull \hbox (badness 1902) in paragraph at lines 874--874
[] [][][]\TU/Sanskrit2003NM(1)/bx/n/12 �
warning (print): bad raw byte to print (c=280), skipped
warning (print): bad raw byte to print (c=281), skipped
warning (print): bad raw byte to print (c=1157), skipped
Looks like LuaTeX is trying to print the glyph node’s char
fields, which (mis)use as glyph indexes and contain invalid Unicode characters. We need to figure out what is best to be shown here (by e.g. checking XeTeX) and see if we can make LuaTeX show that.
Right now we don’t populate any of the math data in the font, also the way we load the glyphs is not compatible with the engine doing layout, so we would need to conditionally change that too.
We need a way to load fonts by name and query system fonts. An easy short cut would to use luaotfload for this.
Ideally we should integrate with system font finding libraries, but linking that in LuaTeX will likely not fly with the LuaTeX team and I’m not sure what are the other options.
This shows only h.
in the PDF:
\documentclass{article}
\usepackage{url}
\usepackage{harfload}
\usepackage{fontspec}
\begin{document}
\url{https://example.com}
\end{document}
\documentclass{minimal}
\usepackage{harfload}
\font\testnoto={name:Noto Color Emoji:mode=harf}
\begin{document}
\testnoto
😄👌☃⛄
\end{document}
The above latex file produced a correctly looking PDF, but it cannot be copy-and-pasted
and it cannot be converted to a text file by, for example, pdftotext
program.
When I use other emoji fonts, e.g. Symbola or Microsoft's Segoi UI Emoji, the resulted PDF are copy-and-pastable and can be converted to text files
by pdftotext
.
I cannot exclude the possibility that it is a bug in Noto Color Emoji, and as harflatex is the only latex variant that can handle Noto Color Emoji, I cannot compare the above behavior with other variants of latex.
Currently hyphenation kinda works, but there are few issues:
1 All our font “characters” use glyph indices prefixed by the largest valid Unicode value, and mapping characters to glyphs happens during shaping not font loading.
I think it should be possible to insert images in LuaTeX from the image data we get from HarfBuzz, need to investigate.
Not sure if we can map hyphenation language to BCP 47 language tags and pass it to HarfBuzz, or if it is even a good idea.
luaotfload has a kernfactor
method which is like letterspace
only with decimal instead of percentage. The following document works fine with mode=node (both outputs are the same), but with mode=node the kernfactor version is not letterspaced:
(kernfactor is used in the microtype code. fontspec uses letterspace)
\documentclass{article}
\usepackage{harfload}
\begin{document}
\font \testA = "file:Iwona-Regular.otf:+liga;mode=harf;+tlig;kernfactor=0.125;"
\testA abc -- fi ff
\font \testA = "file:Iwona-Regular.otf:+liga;mode=harf;+tlig;letterspace=12.5;"
\testA abc -- fi ff
\end{document}
Not sure what is the current status of vertical typesetting in LuaTeX (HarfBuzz side should be usable), need to research this.
We don’t currently do any parsing of font names, so anything other than paths or file names will fail to load. At bare minimum, we should support a XeTeX compatible syntax.
It should be possible already using HarfBuzz Python-like feature syntax (e.g. -kern[4:8]
), but we need a more accessible way, something like fontspec’s \addfontfeatures
. Can be done by setting attributes/properties that we read at shaping time.
Not sure if this is new, but today the LaTeX example took so long and froze my system while consuming all available memory, which usually happens when luaotfload loads Noto Serif CJK, but it shouldn’t be loading any fonts. Might be related to the fact that with fontspec both mode=node
and mode=harf
are used.
Running a simple document like
\documentclass{article}
\usepackage{luatexbase}
\directlua{require("harf-luaotfload.lua")}
\begin{document}
\font\test={file:texgyreheros-regular.otf:mode=harf;}
\test abc
\end{document}
I get the error
warning (node filter): error: .../luaotfload/texmf/tex/luatex/luahbtex-harf/ha
rf-node.lua:339: bad argument #4 to 'shape_full' (number expected, got table)
setting shapers
to 0 in harf-node.lua (instead of a table) works, but I have naturally no idea which number the function expects.
We can’t have pdf_literal
whatsits in discretionary nodes and this is a big limitation, so we need to move inserting these nodes to after line breaking has been done and the discretionary nodes are gone.
If the same file and font index is requested again, we should not load a new a font and instead re-use the already loaded one. Even if the font size is different, we should be able to shape every thing at UPEM size and scale the output.
We should resolve script for Common and Inherited characters if no script was selected during font loading.
An open question is whether we should respect the specified script at all. Automatic script resolution should allow setting multi-script text without loading the font multiple times, but there might be corner cases where the auto-detection would fail. To be investigated.
We should be able to speed shaping considerably by caching words (splitting at spaces) if fonts do not have any substitution or positioning rules involving spaces.
I gave a try (using the extents.x_bearing
if negative) but \/
gave 0pt kerning, need to debug it.
HarfBuzz seems to support color SVG in OpenType font format from Oct. 2018 as
harfbuzz/harfbuzz#1192 (comment)
As I understand, the PDF spec. does not support SVG-in-OT, so SVG has to be converted to PDF or Type3 font to render it in PDF. If one downloads Abelone font (or anything) to current directory and process the below file by harflatex, no SVG-to-PDF or SVG-to-Type3 conversion seems taking place. Can I consider SVG-in-OT as unsupported by harftex? I have little opinion on whether or not SVG-in-OT should be supported, though use of stylish color SVG font may be fun.
Location of Abelone font: https://www.fontself.com/colorfontweek/2018#abelone
LaTeX file:
\documentclass{minimal}
\usepackage{harfload}
\font\svgcolorfont={file:./Abelone-FREE.otf:mode=harf;+svg}
\begin{document}
\noindent
\svgcolorfont ABCDEF abcdef
\end{document}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.