idpf / epub-testsuite

A collection of EPUB documents to systematically test EPUB Reading System conformance

epub-testsuite

A collection of EPUB documents to systematically test EPUB Reading System conformance

Goal

This repository contains a collection of EPUB documents used for evaluating feature coverage in EPUB Reading Systems.

The primary focus of the collection is on human-evaluated binary (pass/fail) tests. Included tests are of two types: A) required and B) optional Reading System functionality. While the tests in category A are the ones used for "formal" specification conformance testing, the tests in category B are included since they nevertheless provide useful information.

The collection uses some fairly simple markup conventions to provide easy navigation and clarity, and to allow the automated generation of a results form template.

Contributing

Following a few simple markup conventions, the process for adding a new test is:

1. Locate the, or create a new, content document to contain the test. (Note that content docs are themed by functionality areas.)
2. Add a wrapper element with an ID and class='ctest' (for required Reading System functionality) or class='otest' (for optional Reading System functionality)
3. Echo the ID value given in 2 above in a descendant element text node with class 'test-id'
4. Add the human readable description of the test intent inside the wrapper. Give the element the class 'desc'. The description typically has the form "Test whether [feature] is supported"
5. Add the evaluation criteria for the test pass/fail states inside the wrapper. This typically has the form "If [observable or invokable condition], the test passes".
6. Add a link to the test in the navigation document (typically xhtml/nav.xhtml). Note that the Navigation Document serves as the self-documenting list of available tests. Each test, regardless of whether it is required or optional, is linked to from the nav doc using the wrapper ID mentioned above.

Download

You can download this project in either zip or tar formats.

You can also clone the project with Git by running:

$ git clone git://github.com/mgylling/epub-testsuite

Licensing

Copyright © 2012-2015 International Digital Publishing Forum (IDPF)

The EPUB Test Suite by the IDPF is licensed under a Creative Commons 4.0 International License (CC-BY 4.0).

More information about this license is available at:

https://creativecommons.org/licenses/by/4.0/

More information about the IDPF is available at:

http://wwww.idpf.org

epub-testsuite's Issues

Test 0120 (Media Overlays Reflowable Tests): Introduction Wording

Current:

It uses a reflowable layout and specifies its own custom highlight style (black text with a pink background).

Proposed:

It uses a reflowable layout and specifies its own custom highlight style (black text with a pink background). If the Reading System allows overriding the highlight style, please select the "Author/User -defined style" (if available) before performing these tests.

Rationale: avoid false positive/negative.

Add build scripts

Add scripts to build the zip packages of the entire collection (can reuse bash/cmd scripts from google-code epub-samples).

The build process would be done thusly:

1. create individual .epub zips of each included document, putting the build date in the filename.
2. create a blob zip of all epubs created in 1, and include the build date in the filename
3. push the blob to /releases/ and announce

List test titles in repo

I think it would be very convenient for reference purposes to have the corresponding titles for each test suite listed somewhere in the repository. I'd be basing the titles off this link: http://epubtest.org/testsuite/

Could it be as simple as including that link in README or is there a better way of including the titles?

I'm more than happy to open a PR, I justed wanted to see what the preferred solution is.

Thanks,
Richey

Add license info (CC-BY) to readme.md and package files

tests for text decoration module support

Support was added for Selectors and Text Decoration. The latter could be applicable to the i18n tests, former might have some tests for the basic css set.

Test Issue

Test Issue

Media overlay

We have media overlay tests done by Marisa DeMeglio should we merge it with tests in first file (epub30-test-001) or keep it in separate file?
If we keep it in separate file, remove the reference to it from epub30-test-001

css changes to epub30-test-0130

This test suite makes use of the changed -epub-text-orientation and -epub-text-combine values adapted from Writing Modes in base_v.css These need to be updated to their new equivalents.

Missing "#" from refines title property in package.opf

I was running one of these EPUB files through the newly released EPUBCheck (v.4.2.5). One of the new features is that the "refines" reference in <meta... is checked. It now flags this line in the package.opf:
<meta refines="title" property="title-type">
Assuming it is correct to flag this, to avoid the error it needs to be changed to use the fragment ID:
<meta refines="#title" property="title-type">
A number of the test files have this line.

Move navigation.xhtml in epub30-test-0130 to beginning

Since navigation.xhtml tests the page progression direction, it is better to move it to the be the first chapter of this file.
It would be good to add Japanese text in it after the Hebrew test that is already there.

new events for triggers

301 added three additional events: phase, defaultAction and propagate

Probably low prio to check these are supported.

Reorganization (new branch)

I have created a new branch called "reorganization" in order to prepare the new structure that we discussed last year, which we can find in the wiki page written by Markus.

As a start, there are two subfolders: reflowable and fixed-layout. I am not sure if this is useful, but this way the organization seems easy to understand.

The 001, 003 and 005 are renamed 0100, 0120 and 0230. No more changes except the name of the folder (and the dc:title and dc:identifier).

For now, I have choose to merge 002-rtl and 004-he into 0130, because both have the same page-progression-direction. Is there a problem with this choice, @OriIdan ?

Do not forget that this is another branch, and this is not the main branch. I just want to illustrate the new structure, in ordre to show every problem we could have.

test document 100 svg-430 test is broken

it seems that there is no corresponding CSS to insert the background image.

comment from garth:

The current test has:

svg-050 {

width: 100%;
background-image: url(../svg/circle.svg);
background-repeat: no-repeat;
background-position:right bottom;
}

in "svg.css" but that's not even included by "content-svg-003.xhtml" and even if it was, it's the wrong class name.

Further there is no use of "svg-050" anywhere in the current test suite.

Ruby tests and style-410

While researching support for -epub-ruby-position, I found it interesting that some reading systems fail the Ruby tests, but pass style-410. And some reading systems pass the Ruby tests, but fail style-410. Given that style-410 is just the initial value of ruby-position, this is weird. As written, I don't think style-410 gives us much information about the level of support for ruby-position. Perhaps these style tests should be moved into the test file that checks for ruby support itself...

[OPTIONAL FEATURE] Add 'Alternate Style Tags' Tests (EPUB 3.0.1)

Alternate Style Tags is part of EPUB 3.0.1 (see CONTENT DOCUMENT 3.0.1).

There are three known examples in the IDPF EPUB3 Samples project:

• wasteland (day and night values);
• kusamakura (vertical and horizontal values);
• mymedia_lite (vertical and horizontal values).

handheld media tests

scripsit garth conboy:

I think some of the @media tests are in need of tuning.

The definition of "handheld" is so wishy washy ("typically small screen and low bandwidth") that I think this test should just be removed. At the very least the instructions for grading the test should become "If the preceding paragraph reads FAIL when displayed on a device considered a handheld, this test fails."

For TV, the definition of the device may be clearer, but the criteria for passing is wrong -- the instructions for grading the test should be "If the preceding paragraph reads FAIL when displayed on a TV device, this test fails."

Similarly the instructions for grading the portrait and landscape tests need to be fixed to be orientation-specific:

"If the preceding paragraph reads FAIL when displayed in portrait orientation, this test fails." and:

"If the preceding paragraph reads FAIL when displayed in landscape orientation, this test fails."

Best,
Garth

Test 0120 (Media Overlays Reflowable Tests): mo-playback-010 wording

Current:

This should remain highlighted during Media Overlays playback.

Proposed:

This sentence should remain highlighted (white text on green background) during Media Overlays playback.

Rationale: avoid false positive/negative.

MathML test

Added MathML tests to branch master
in content/30/epub30-test-006-mathml
This should probably go into epub30-test-0100.epub

Test 0120 (Media Overlays Reflowable Tests): mo-audio-030 Wording

Current:

If playback immediately continues to the next section after the preceding sentence is announced in full, the test passes.

Proposed:

If playback immediately continues to this sentence after the preceding sentence is announced in full, the test passes.

Rationale: this test should check whether the Reading System correctly computes the length of clip_times_tests3A.mp3, disregarding the clipEnd="00:00:20.000" in the associated SMIL file. However, the MO playback does not end with mo-5-b (the second-to-last paragraph, associated to clip_times_tests3A.mp3), but with mo-9-b (the last paragraph, as quoted above). Hence, after computing the correct clipEnd for mo-5-b, the Reading System should not jump to the next XHTML, but it should render (immediately) the last sentence instead.

add tests for 301 rendition:* properties

Add tests for the rendition:flow.

epub30-test-0130:some characters' default direction changed in UTR#50 rev11

UTR#50 rev11
http://www.unicode.org/Public/vertical/revision-11/VerticalOrientation-11.html

A)
in epub-testsuite / content / 30 / epub30-test-0130 / EPUB / xhtml / character-005.xhtml

UTR#50 rev11, some characters' default direction are "R".
even if set upright to following characters, they'll display rotate 90 degree.
° U+00B0 DEGREE SIGN
′ U+2032 PRIME
″ U+2033 DOUBLE PRIME
¢ U+00A2 CENT SIGN
£ U+00A3 POUND SIGN
Å U+00C5 LATIN CAPITAL LETTER A WITH RING ABOVE
¶ U+00B6 PILCROW SIGN

about this test, there are two ways.

1. delete the above characters and edit expected-character-v-cjk-symbols.png
2. change the purpose to "ests whether the direction of full-width math symbols is the same to UTR#50 in vertical writing.".
   to do 2)
   delete <p>followings are set text-orientation:upright</p>
   s/<span class="upright">(.*?)</span>/$1/g;
   edit expected-character-v-cjk-symbols.png to the same as UTR#50 rev11

B)
in epub-testsuite / content / 30 / epub30-test-0130 / EPUB / xhtml / character-008.xhtml

UTR#50 rev11, some characters' default direction are "R".
even if set upright to following characters, they'll display rotate 90 degree.
− U+2212 MINUS SIGN
＝ U+FF1D FULLWIDTH EQUALS SIGN
≠ U+2260 NOT EQUAL TO
＜ U+FF1C FULLWIDTH LESS-THAN SIGN
＞ U+FF1E FULLWIDTH GREATER-THAN SIGN
≦ U+2266 LESS-THAN OVER EQUAL TO
≧ U+2267 GREATER-THAN OVER EQUAL TO
∞ U+221E INFINITY
∈ U+2208 ELEMENT OF
∋ U+220B CONTAINS AS MEMBER
⊆ U+2286 SUBSET OF OR EQUAL TO
⊇ U+2287 SUPERSET OF OR EQUAL TO
⊂ U+2282 SUBSET OF
⊃ U+2283 SUPERSET OF
∪ U+222A UNION
∩ U+2229 INTERSECTION
⊄ U+2284 NOT A SUBSET OF
⊅ U+2285 NOT A SUPERSET OF
⊊ U+228A SUBSET OF WITH NOT EQUAL TO
⊋ U+228B SUPERSET OF WITH NOT EQUAL TO
∅ U+2205 EMPTY SET
∧ U+2227 LOGICAL AND
∨ U+2228 LOGICAL OR
¬ U+00AC NOT SIGN
⇒ U+21D2 RIGHTWARDS DOUBLE ARROW
⇔ U+21D4 LEFT RIGHT DOUBLE ARROW
∀ U+2200 FOR ALL
∃ U+2203 THERE EXISTS
⊖ U+2296 CIRCLED MINUS
∥ U+2225 PARALLEL TO
∦ U+2226 NOT PARALLEL TO
∠ U+2220 ANGLE
⊥ U+22A5 UP TACK
∂ U+2202 PARTIAL DIFFERENTIAL
∇ U+2207 NABLA
≡ U+2261 IDENTICAL TO
≒ U+2252 APPROXIMATELY EQUAL TO OR THE IMAGE OF
≪ U+226A MUCH LESS-THAN
≫ U+226B MUCH GREATER-THAN
√ U+221A SQUARE ROOT
∽ U+223D REVERSED TILDE
∝ U+221D PROPORTIONAL TO
∫ U+222B INTEGRAL
∬ U+222C DOUBLE INTEGRAL
≢ U+2262 NOT IDENTICAL TO
≃ U+2243 ASYMPTOTICALLY EQUAL TO
≅ U+2245 APPROXIMATELY EQUAL TO
≈ U+2248 ALMOST EQUAL TO
≶ U+2276 LESS-THAN OR GREATER-THAN
≷ U+2277 GREATER-THAN OR LESS-THAN
↔ vU+2194 LEFT RIGHT ARROW
Å U+00C5 LATIN CAPITAL LETTER A WITH RING ABOVE

There is already no reason to set "-epub-text-orientation:upright;" against full-width math symbols.

about this test, there are two ways.

1. delete all test-id character-rtl-070
2. change the purpose to "tests whether the direction of full-width math symbols is the same as UTR#50 in vertical writing.".
   to do 2)
   s/<span class="upright">(.*?)</span>/$1/g;
   edit and rename expected-character-v-math-upright.png to the same as UTR#50 rev11

I'll pull request the way 2).
How about?

add a test for media:playback-active-class

Media Overlays in EPUB 3.0.1 introduces a new metadata property called "media:playback-active-class"

need test for remote access via JSONP

The file scripting-003.xhtml has a test for reading remote resource via XMLHttpRequest. But in the context of EPUB remote resources will always be from a server in a different domain, since the EPUB package itself constitutes the implicit domain (relative URL space). So in addition to testing XMLHttpRequest, which requires servers with CORS support, the alternative JSONP technique should be tested, as this pre-CORS workaround still widely used in various JS libraries.

Test 0120 (Media Overlays Reflowable Tests): mo-nav-020 Wording

Current:

If playback continues on the destination page, the test passes.

Proposed:

If playback continues on the destination page, starting from "Section Navigation, Part 2", the test passes.

Rationale: making clearer that the playback should start from that precise point, and not from the first MO element on the landing page.

review, clarify and update the MathML tests

Review, clarify and update (where needed) the MathML tests to make sure all pass conditions are 100% clear to non-MathML experts, especially with regards to pass/fail on existing MathML implementations.

licensing of this testsuite?

Please explicitly mention - e.g. in README - the copyright holders of this project and the licensing which the copyright holders permit usage under.

Added test for Hebrew diacritics (nikud)

Hebrew does not have vowels so there are marks below and above letters.
The added test should show it.

MO test basic-010 should be simpler

Current wording:
“If there is a play/pause button that starts playing audio narration for the text on this page, then the Reading System supports Media Overlays (MO). Implied in this test is that the Reading System still recognizes MO even when the first chapter of the book does not contain it.”

Proposed change:
Remove reference to user controls as some reading systems support MO by automatically playing them (no controls though). Add another test to report whether the user has control over basic playback (stop/start).

Results form generation scripts

Add script to generate a results from the navdocs of the collection. Note that the results form must incude version info for the collection (build date)

Provide more SVG tests

The SVG part provides only simple tests for three categories: vectors (only circle and rectangle), text and bitmap.

We could do new tests with complex layout, because positioning (with transform matrix, for example) could be not really perfect.

New possible tests : other shapes, fonts, styling, other text tests and bitmap's transform.

Test 0120 (Media Overlays Reflowable Tests): mo-basic-050 Wording

Current:

Tests whether playback rate control is supported.

If you can independently adjust the MO playback rate, without warping the pitch, then the Reading System supports rate control.

Proposed:

Tests whether the playback rate of the Media Overlay playback can be independently adjusted.

If you can adjust the MO playback rate independently of the device rate setting, without warping the pitch, then the Reading System supports playback rate control.

Rationale: keeping the description of the test as close as possible to mo-basic-040 (volume).

clean up wording in tests

In some cases, it's a bit too technical:

"The site assumes a certain familiarity with EPUB specs and the test suite. This could be resolved by making some adjustments to language. For example, “Tests whether the page-list nav can be accessed” is familiar to those of us who know spec terminology. “Tests whether a paginated TOC can be accessed” might make this more usable to the broader publishing world. "

However, we should be careful to not introduce ambiguity - "Paginated TOC" might be thought to refer to a content document that is the counterpart to the print TOC (or one of the print TOCs) in the print book, which is not the same as the page-list nav.

remote file test xhr-020 should fail when CORS is not supported

scripting-003.xhtml has a simple test for getting a remote file:

// try to get a remote file
testRequest("http://idpf.org/epub/30/spec/epub30-contentdocs.html", "xhr-030-result");
};

It indicates "PASS" on Readium but IMO it should fail because the http header for that URL does not presently return the CORS specified value:
Access-Control-Allow-Origin: *
Running scripting-003.xhtml standalone in a browser for example returns "FAIL" even though of course the browser supports remote file access.

I'm not sure it's explicitly defined anywhere in EPUB 3.0 specs (perhaps a spec issue!) but I believe that the EPUB file's internal relative URL space is implicitly the origin and that requests external to that should therefore be enabled only via CORS.

This implies that this test should probably fork into two variants, one that makes a request to a known CORS-enabled URL that is expected to PASS and one that makes a request to a known non-CORS-enabled URL that is expected to FAIL.

portrait/landscape tests wording

Suggestion for the pass/fail text for the portrait/landscape tests:

"If the preceding paragraph reads “FAIL” in portrait/landscape orientation on a device that has multiple orientations, the test fails."

epub 120 (MO)

user feedback:

EPUBTEST-0120

1. mo-basic-040: Volume Control: it is not clear to me if you mean
   that the RS should have its own volume control (independent from, e.g.
   the OS volume control), or just having a way of controlling the volume
   (e.g., the OS volume control) is ok.

2. Skippability and Escapability sections: change the wording of "the
   following highlighted fragment" to something like "the following
   green-backround fragment" to avoid confusion with the MO "highlighted
   (active) fragment".

3. Clip Times: clip_times_tests.smil references 3 different audio
   files. In this way, a RS that correctly handles the missing
   clipBegin/clipEnd attributes, but it does not handle multiple audio
   references in the SMIL file, does not pass the test.
   (This observation might apply to other tests.). Moreover, in the
   clip_times_tests.smil I got, clipBegin/clipEnd are not actually
   missing from any element, so mo-audio-010 and mo-audio-020 are
   not actually checked (or I am missing something).

add remote audio/video tests

Add test for CSS namespaces

A test for CSS namespaces needs to be added.

Test 0120 (Media Overlays Reflowable Tests): mo-nav-030 assumes pagination

Current:

Let media overlay playback continue while you turn pages. The playback should be updated to reflect your current position.

Comment: This test assumes that the Reading System paginates (or, it offers an option for...) the reflowable contents. Should this test consider scrolling layout as well? In both cases, the test passes when the MO playback follows user actions (turning pages or scrolling the text column)? Is creating a new test for scrolling layout more appropriate? What if the Reading System allows the user to link/unlink the MO from the text navigation (within the same XHTML document)?

Test 0102 (Scripting Tests): xhr-030 False positive due to bad test

Current: the possibility of opening remote files via XHR is deemed available with the following code:

 request.onload = function() {
   txt = request.responseText;
   if (txt) {
     document.getElementById(testResultId).innerHTML = 'PASS';
   }

In other words, as soon as txt is truthy, the test PASSes.

This means that if the Reading System returns a spurious string in request.responseText (e.g., Cordova returns ERROR whitelist rejection: $URL if the requested URL is not in the app whitelist) the test PASSes, albeit the Reading System did not actually access the remote file.

Proposed: probing the response code instead, or -- even better -- testing against a precise (known a priori) value for txt, containing the expected text of the remote file.

Rationale: avoid false positives.

remove MO test mo-basic-240

It doesn't test anything in the spec; it's just a reading system behavior test.

use images as visual confirmation

For anything that requires visual confirmation:
Provide image example for anything that may not be commonly recognizable (e.g., Hungarian text)

Added few explanations in Hebrew

I added few more explanations and test for glyphs without specific font definitions

Add test for rendition:align-x-center property

As subject line says.

include more text in FXL tests

• Fixed layout test: It is very difficult to ascertain whether fixed layout is functional when the only thing on the screen is the page number. Including a bit more on each screen (even just some "lorem ipsum" text), especially if it were vertically centered, would be extremely helpful.

epub30-test-0100: switch-010 does not actually test if epub:switch is supported.

The content/30/epub30-test-0100/EPUB/xhtml/content-switch-001.xhtml switch-010 test is supposed to test if epub:switch is supported. However, this can be passed by a reading system that ignores text in elements it does not recognise, and recursing through the unrecognised elements.

A better test would be:

<epub:switch>
    <epub:case required-namespace="http://www.example.com/unsupported/namespace">
        <p>
            FAIL
        </p>
    </epub:case>
    <epub:default>
        <p id="fallback">
            PASS
        </p>
    </epub:default>
</epub:switch>

That is, use <p> in both branches of the epub:switch statement. This ensures the reading system is processing the epub:switch statement.

epub30-test-0100: Content Documents: SVG not supported over broad

The "Content Documents: SVG" section has a note that is over broad. It says:

Note that if no SVG tests appear after this document, SVG is not
supported in the spine and all tests should be marked Not Supported.

This should be changed to read "all tests in this section should be marked Not Supported" as this test only covers whether SVG is supported, not whether any other functionality is (i.e. failing this test does not require every test in the BISG Grid test suite to be marked as Not Supported).

epub test 120 needs new MO audio

Need to generate new MO audio for this since the text changed in several places. Unfortunately the script I was using to do this automatically seems to have issues running on the latest OSX, so I need to fix it first.

Sample 130 implements one of the worst edge cases possible

In practice, sample 130 implements one of the worst edge cases possible, an edge case that is not even discussed in the EPUB3 spec: mixed writing modes and how to deal with such a publication – so most Reading Systems will actually fail to handle it properly.

It should at least be 2 different files:

1. vertical writing;
2. RTL direction.

It’s been popping up over and over and over again in issues I’ve been dealing with while the spec doesn’t even tell how it is supposed to be handled and is left wide-open to implementers’ interpretation.

On a related note, styling for the cover doesn’t even work in web browsers, see readium/readium-css#50 (comment)