Enclosing punctuation for Phrase Emphasis and All caps not working fully about liblouis

[krperry]

This weekend as I watch the train wreck of college football games I will try to right more tests.

[krperry]

This link takes you to the conversation on this issue on liblouis-liblouisxml free list
https://www.freelists.org/post/liblouis-liblouisxml/Emphasis-phrase-question-and-forking-till-fixed,7

[bertfrees]

See also https://www.freelists.org/post/liblouis-liblouisxml/Emphasis-phrase

[krperry]

Yes I already added it as a comment yesterday.

[bertfrees]

I'm going to need more tests to see if there is more to it, but it seems there is already one thing we can do: we can change how the counting of words to determine the length of phrases happens. Currently Liblouis only counts whole words (thereby treating unemphasisable characters at the beginning and end as spaces). But since the goal of marking phrases is to reduce the number of indicators, we could perfectly well justify counting half words too, as this won't increase the number of indicators. (Note that this true only on the condition that endemphphrase after is used, like is the case in UEB. When endemphphrase before, or begemphword, is used and the emphasized part of the last word does not span the whole word, one additional indicator is needed to cancel emphasis.)

[bertfrees]

Regarding the all caps issue: there is some code in Liblouis that was added exactly to achieve the opposite of what you say should happen, namely that if the last word of the phrase ends with non-letters (punctuation), the endemphphrase after indicator is inserted after them. The code was included specifically when the noempclass feature was added, in order to preserve the old behavior.

When I disable this code, your "'ABC ABC DEF' defg" example is translated the way you say it should, but several UEB tests start failing because they claim the opposite should happen. @krperry You may want to check them, they are in en-ueb-08-capitalization.yaml.

Apart from the UEB tests, a number of other tests are affected. Some, like Norwegian, are actually improvements, but other, such as Swedish, are regressions. In other words, different braille codes do it differently.

[krperry]

I think the three tests pretty much show everything you need the only differences are what is laidout in the ticket. The tests show the parans which are the same as the brackets, tbraces, angle brackets, and it shows the quotes whichis the same as the tilda, single quote, I guess I could make a whole file of tests for you. how many do you want? As for the way I was doing this in the source code is I changed from checking for spaces to checking for enclosing punctuation and spaces. So I think you are saying the same thing.

[krperry]

Regarding the all caps issue

I am going by what transcribers tell me at APh I guess I can disable the code but if it breaks tests what do we do but fix the tests? Does that break other languages?

[krperry]

I am going to look at the caps. With the problems of different languages. This is why I was wondering if this kind of configuration should have another op code to determine enclosing characters that should work as spaces do now.

[bertfrees]

I think the three tests pretty much show everything

They may explain the requirement, but for really good coverage we need more tests IMO:

• tests with other kinds of punctuation
• tests with punctuation that is not enclosing (only before, only after, or different punctuation before and after)
• tests with enclosing punctuation combined with other punctuation
• tests with punctuation at the beginning and/or end included in the emphasis
• etc. etc.

I am going by what transcribers tell me at APH

The existing tests are coming from the UEB rule book AFAIK. So they are probably correct. But nevertheless I think it's worth checking them out. These are the tests:

• CAUTION: WET PAINT!
• IT'S A HOAX! (APRIL FOOL!)
• V-NECK SWEATERS FOR SALE!

It would be good if you could add some clarifying comments to the tests, and also to your new tests, to explain the different expectations in different cases. The more clarifying comments are included in YAML files, the better.

[jrbowden]

Regarding the placement of the caps terminator, there are two cases:

1. When there is punctuation which does not "balance", then the caps terminator goes at the end. This is the case with the examples from RUEB 8. The examples are correct. e.g.
   CAUTION: WET PAINT!
   ⠠⠠⠠⠉⠁⠥⠰⠝⠒⠀⠺⠑⠞⠀⠏⠁⠔⠞⠖⠠⠄
   Here, the caps terminator comes after the exclamation.

2. But, when there are balancing punctuation marks (like the brackets or quotes), the capitals terminator should go before the matching punctuation, the principle of "nesting". So @krperry's examples, e.g.:
   "ABC ABC DEFG" defg
   ⠦⠠⠠⠠⠁⠃⠉⠀⠁⠃⠉⠀⠙⠑⠋⠛⠠⠄⠴⠀⠙⠑⠋⠛
   Note I have deliberately changed to double quotes to make the punctuation and caps terminator more obvious.

What I have said for capitals is equally true for typeforms (like bold):

1. Caution: wet paint!
   ⠘⠶⠠⠉⠁⠥⠰⠝⠒⠀⠺⠑⠞⠀⠏⠁⠔⠞⠖⠘⠄

2. "abc abc defg" defg
   ⠦⠘⠶⠁⠃⠉⠀⠁⠃⠉⠀⠙⠑⠋⠛⠘⠄⠴⠀⠙⠑⠋⠛

See Rules of Unified English Braille section 9.7 re typeforms and punctuation.
It would be difficult to get all the examples passing - some require "an understanding of the text".

I agree with @bertfrees more examples are needed. I will attempt to write some and add them to this ticket.

[krperry]

There is a reason I used the single quote for the caps test because the caps works with double quotes because it is a capitalized form of the quotes. And it works. If it is single quotes the caps phrase does not work. That is the only bug with caps phrase that I have found. It is a somewhat different problem with emphasis because the way the code works it only starts on a space or a character that can not be emphasized. Since all punctuation can be emphasized If you don’t emphasize the quotes or single quotes or brackets etc. Then it doesn’t count the first word as a all bold word. This is not a problem with caps. I guess this ticket should be split to one with the caps bug and one with all the emphasis because they are different problems. Caps mostly works Emphasis does not.

[bertfrees]

Thank you @jrbowden, that makes things more clear for me. And I agree with you that it might be difficult to get all the tests to pass.

The "IT'S A HOAX! (APRIL FOOL!)" example comes from RUEB section 8.6.2, which seems to be the relevant section for this issue:

The capitals terminator may precede or follow punctuation and other terminators but it is best that indicators and paired characters such as parentheses, square brackets and quotes be nested. That is, close punctuation and indicators in reverse order of opening.

Strange that this hasn't been referenced by anyone before.

I notice that it says "it is best that", not "it is required that". So that means the behavior of Liblouis should ideally be improved, but is acceptable as it is.

So I guess the "serious bug" that Ken referred to on the mailing list is the emphasis part. Regarding that part: my commit 30df4d9 fixes Ken's first and second test. I'm not sure that means it resolves the issue. It's only two tests.

[krperry]

I have been working on a brf2ebrf translator this week and had some dead lines. If you have a commit I will get it this weekend and see if I can add several more tests with different punctuation and see if it still passes. Thanks.

[bertfrees]

@krperry Great, keep me posted.

[krperry]

What branch is the development being done on. I don’t see your commit.

[bertfrees]

The only commit you need is 30df4d9. It's on the phrase-emphasis branch, but don't use the last commit of that branch.

[krperry]

Thanks perfect will give it a shot.

[krperry]

Weird I see your branch when I go to github on the web page but not when I do git branch -r strange.

[krperry]

I tried to cherry-pick 30df4d9 and it fails at the commandline. I don’t see e the branch when I do git branch -r. Is this a real branch or just a pull request from some other fork? I can’t seem to get the commit. Is there a command I am not aware of to get this?

[bertfrees]

git fetch https://github.com/liblouis/liblouis.git && git checkout 30df4d99f6

[krperry]

This seems to work for the double quotes and the single quoest with your fix. It has mostly fixed the other punctuation like (), {}, [], and <> the only problem is the enclosing ending punctuation is part of the emphasis even though it is not marked. It is hard to post it here but if you Put a paran at the start of a phrase and a paran at the end of a phrase bold the phrase but do not bold the parans. The end paran should be outside the phrase end mark. I am pretty sure that was in the tests I sent in. I will download them and check. This is the same for all enclosing punctuation. This is much better than it was now if we can only get that last bit fixed.

[krperry]

Do I need to put more tests in to get the enclosing punctuation problems fixed? I know one of the tests I put in this original ticket showed the problem but maybe more is needed? The part I am talking about is when you have text enclosed with parenthesis, brakets, braces, angle brackets, pretty much any punctuation that a person might use for enclosures other than quotes. Then if you bold the inside but not the punctuation. Liblouis still gets that wrong. If we can get that fixed then this ticket can be closed. AS it is we put the current liblouis in the current stable brailleblaster 2.1 and so far it is good other than this problem.

[bertfrees]

Hi Ken. I'm a bit confused. As far as I can tell the issue you describe is fixed. The test that I am running is the following:

table: |
  include tables/unicode.dis
  include tables/spaces.uti
  include tables/en-chardefs.cti
  include tables/en-ueb-g1.ctb
tests:
- - "(abc abc defg) defg"
  - ⠷⠘⠶⠁⠃⠉⠀⠁⠃⠉⠀⠙⠑⠋⠛⠘⠄⠾⠀⠙⠑⠋⠛
  - typeform:
      bold: ' ++++++++++++      '
- - "(abc abc abc defg) defg"
  - ⠷⠘⠶⠁⠃⠉⠀⠁⠃⠉⠀⠁⠃⠉⠀⠙⠑⠋⠛⠘⠄⠾⠀⠙⠑⠋⠛
  - typeform:
      bold: ' ++++++++++++++++      '

It corresponds with your "example 2" in your initial comment:

2. A bold phrase with non-bolded parentheses:
   (abc abc defg) defg

Can you spot a mistake in the test?

[krperry]

It looks like the test is incorrect. The end paran should not be bolded. And in this test you have the paran before the end bold. The problem is when you do not bold the opening and closing paran, bracket, braces, or angle brackets. So something like this: (abc abc abc) The opening and closing bold tags should be inside the parans. Below the dot 3 and dot 23-56 are outside the parans

[bertfrees]

I'm sorry, your last comment makes no sense to me...

In the test the parens are not bolded, and the opening and closing bold tags are inside the parens.

The test I'm using is simply taken from the file that you sent us earlier. And as far as I can tell it matches the requirement.

@jrbowden what is your take on this?

[krperry]

I miss read the dots. If this is passing. Was it passing before the release? If not I will go get the current and try again.

[bertfrees]

OK (relief).

I'm not entirely sure but I think it was passing before. In any case, it does with the current release.

[krperry]

WE have the current release in Brailleblaster and it is not working . I am compiling what is in liblouis right now and will be checking it with brailleblaster and will let you know here in maybe 30 minutes or so.

[krperry]

I just compiled and moved the latest Liblouis in to Brailleblaster and it still is not working. I am going to take your tests and run it after lunch and see if I get the same thing ou are.

[jrbowden]

Hi @krperry and @bertfrees ,
As promised, I attached a set of more extensive tests. I hope this helps.
When I run it here, all but 1 test passes. The one that fails is possibly debatable, but I think it is true.
I hope this helps.
Question is where should these tests go?

[bertfrees]

Somewhere in tests/braille-specs/en-ueb-rueb.yaml?

[bertfrees]

@krperry Do you already have more clarity?

[krperry]

Sorry I thought this was closed. I have the latest in Brailleblaster. It passes the tests and so far everything is working with all the punctuation enclosed emphases phrases working. Should we close this and if we find something broken I will make more tests and open one again.

[egli]

Yes, if it all works then let's close this :-)

[krperry]

working tested in Brailleblaster.