At begin document material from package B may end up before or after at begin document from package A requiring B about latex2e

[jfbu]

@u-fischer \ShowHook{begindocument} would have indeed facilitated my search: although I knew from the very start the problem was caused by ordering of things, this knowledge became secondary to finding out experimentally how random things influenced this ordering. Here with the \AtBeginDocument{\empty} we see that its presence influences the ordering. It is true that in most context you do your \RequirePackage's at start of your own packages. But here the situation arises because the secondary package is only conditionally loaded, depending on some user option, and many things will have happened before, inducing some \AtBeginDocument of your own. Despite the fact I knew ordering was definitely surely the root cause, of course when I first created a MWE I could not reproduce the issue because in the simplified code you write you by definition remove all unrelated things, and an \AtBeginDocument done by yourself before the \RequirePackage is definitely a priori completely unrelated. I searched for \AtBeginDocument in ltnews.pdf, there are only tow occurences. In October 2020, ltnews said nothing about \AtBeginDocument in its announcement of the hooks mechanism. It said At the moment the integration is lightweight. So at that time I did not felt needed I should devote the needed hours to go reading the full-fledged lthooks-doc.pdf and even today when searching in this PDF for the string \AtBeginDocument I do not find warnings about the behavior mentioned in this issue, it seems one has to really assimilate thoroughly the whole thing to understand the implications. As a result it is not the first time I get bitten and basically later on get responded that "this is by design" (cf #919), which I can not argue with, obviously yes it seems to be by design. The problem now is whether this design is appropriately explained to people who do not necessarily have the time to read dozens of pages.

[josephwright]

This is what the hooks mechanism is for ...

[josephwright]

A secondary issue is that I did not see in latexbug any option to tell it to accept without raising an error the packages which are part of the demonstration. Maybe some "warn" option would help the build raise an error only where it is significant

It's by-design - you can just scroll past the message if you know it's not important.

[josephwright]

If you need ordering of material in hooks, you need to use the hook rules to request a particular sort.

[jfbu]

Is such situation documented? I mean by that, naturally, the ordering of hook rules is documented, but has the impact on existing legacy usage been illustrated with examples such as this one? I am surprised the full-fledged changes of the \AtBeginDocument mechanisms did not cause more breakage in usages of legacy packages hence in legacy documents. Probably this is because the volume of LaTeX packages and their uses in real world is not as diverse as one could think. Probably old documents which get recompiled often are for example mathematical articles using amsart, hence very controlled and limited context. Most old documents do never get recompiled, hence one never realizes they would break with contemporary LaTeX.

[u-fischer]

This only happens if package A did also some innocuous \AtBeginDocument{\empty} before requiring package B! This instability where you can't trust the logical order of your own code is the problem I want to report.

Hook chunks are labelled, by default by your package name, and if you add to a hook then if there already exists code under the ⟨label⟩ then the new ⟨code⟩ is appended to the existing one.

because this is so counter-intuitive

It is only counter-intuitive if you view the hook code as unsorted, anonymous pieces of code, once you view them as labeled code chunks the logic is quite clear. It helps a lot to use \ShowHook to get a clear picture. In your case this would give

-> The hook 'begindocument':
> Code chunks:
>     kernel -> \cs_if_exist:NF \tag_if_active:T {\prg_new_conditional:Npnn \ta
g_if_active: {p,T,TF,F}{\prg_return_false: }}
>     testABDpackage -> \empty \mathcode `\,="8000\relax 
>     testABDsubpackage -> \mathchardef \mathcomma =\mathcode `\,\relax 
> Document-level (top-level) code (executed last):
>     ---
> Extra code for next invocation:
>     ---
> Rules:
>     ---
> Execution order:
>     kernel, testABDpackage, testABDsubpackage.
<recently read> }
                 
l.134 \ShowHook{begindocument}

I am surprised the full-fledged changes of the \AtBeginDocument mechanisms did not cause more breakage in usages of legacy packages hence in legacy documents.

There were a small number of cases, where order was wrong, (mostly related to the top-level label), but your case is imho rather rare as most packages load subpackages first before doing something else.

[jfbu]

The problem now is whether this design is appropriately explained to people who do not necessarily have the time to read dozens of pages.

The "dozens of pages" bit was unnecessarily worded, and to be honest lthooks-doc.pdf is only 29 pages attow. Actually I did read (substantial portions of) lthooks-doc.pdf probably first in 2021, then again in 2022, and again today. I recall that first time I read it my understanding was that I was reading (exciting) new possibilities for users of LaTeX. For sure I did not understand, although it definitely is what this doc says, that all the general things applied to legacy AtBeginDocument. And at this late stage, it has been enough that I was not occupied for a few months with LaTeX that I had again completely regressed to my initial (wrong) expectations. Perhaps also my understanding had to overcome prejudice about what the word "hook" means. The document discusses "hooks" (a notion which is defined by this LaTeX extension) on one hand and "hook code chunks" on the other, and there is also "hook label".

We could have a situation like this in the past. Package C needs to load Package A and Package B. For some reason Package B needs to be loaded after Package A. All three packages add material to be executed at begin document. It is important for Package C that some of its at begin document alteration are executed before package A. But this would create problems with Package B. So Package C needs to re-do some document alteration after Package A at begin document code is done. So in legacy situation you could have inside Packge C some code like this:

\AtBeginDocument{chunkA}
\RequirePackage{PackageA}% its at begin document code will be executed between chunkA and chunkB
\AtBeginDocument{chunkB}
\RequirePackage{PackageB}% its at begin document code will be executed between chunkB and chunkC
\AtBeginDocument{chunkC}

This legacy situation is broken by the 2020 LaTeX alterations to \AtBeginDocument. As far as I understand now this triggers this execution order

chunkA
chunkB
chunkC
code chunks from PackageA
code chunks from PackageB

and I have a doubt here if the order for the last two lines is certain.

Inside the docs I see an example like this

\DeclareHookRule{begindocument}{mypackage}{before}{babel}

but this is not enough for the above. PackageC has to use specific labels to get the desired ordering. Perhaps 2.1.7 Defining relations between hook code could add an illustration of how to do that in the description I mentioned above.

And I propose that it should explicitly state that PackageC author¹ will need to update the code and use the modern syntax, as there was a breaking change done in 2020.

Footnotes

1. the Package C author code as stated was not robust and one has to imagine it also contained code to check if PackageA and PackageB were already loaded and that the above was done only if none was loaded yet. ↩