wordpress / theme-review-action Goto Github PK

View Code? Open in Web Editor NEW

28.0 12.0 11.0 9.02 MB

License: Other

PHP 8.86% JavaScript 64.93% HTML 25.27% CSS 0.93%

theme-review-action's Introduction

WordPress Theme Check Action

This code action runs various test suites to help WordPress theme development.

Requirements

NodeJs
NPM
Docker

Triggers

NPX
GitHub Actions
Locally

Running via `NPX`

Inside of your WordPress theme folder, run npx wordpress-theme-check-action.

Output Location: Console

Windows is currently not supported.

Running via `GitHub Actions`

You can run this project on GitHub. Here is an example GitHub action workflow file:

name: Test My Theme

on:
  push:
    branches: [ main ] ## Change the branch name to match yours
  pull_request:
    branches: [ main ] ## Change the branch name to match yours


jobs:
  run_tests:
    runs-on: ubuntu-latest

    steps:

    ## We need to keep this around until Gutenberg 9.6 is launched.
    - uses: actions/setup-node@v1
      with:
        node-version: '12'

    - name: Theme Test
      id: test
      uses: Wordpress/theme-review-action@trunk
      with:
        accessible-ready: true
        ui-debug: true

Github Action Inputs

Input	Type	Description
`root-folder`	string	Location of your theme's root folder
`accessible-ready`	boolean	Whether we should run the additional accessibility tests
`ui-debug`	boolean	Setting this to true will save some screenshot artifacts for debugging

Output Location: GitHub Annotations

Locally

Run git clone [email protected]:WordPress/theme-review-action.git && cd theme-review-action.
Run npm install to install dependencies.
Run npm run start.

The tests run on files within the /test-theme folder.

You can pass a relative path to your theme like so: npm run start -- --pathToTheme=../my-theme.

To see all the options run npm run start -- --help.

Output Location: /logs folder. Files are replaced on each test run.

theme-review-action's People

Contributors

Stargazers

Watchers

Forkers

stevendufresne isabella232 carolinan kafleg wenxuefeng3930 krupal-panchal garikhg arfinmilondev

theme-review-action's Issues

Required: If theme has POT file it must be up to date

Putting languages files in the theme are not required. If a theme author chooses to include a POT file in their theme it needs to be kept up to date with all translation strings.

Some of the theme authors directly download the underscores theme and work on it and forget to make their own theme POT file and keep the default underscores POT file. Also, some theme uses their own theme POT file without updating.

Some example text I found in POT file is,
Copyright 2014, # Copyright (C) 2020 Automattic

We can find those strings and the date of the POT file creation "POT-Creation-Date: 2020-04-17T21:03:15+00:00\n" and restrict the upload or give a warning message.

php file (IE: singular.php, index.php) does not contain a body class

I have been getting the following messages when I am submitting my theme, this started happening in the past three months (more or less)

"/?p=1241" (via: singular.php) does not contain a body class
"/" (via: index.php) does not contain a body class
"/?tag=alignment-2" (via: index.php) does not contain a body class

I know that I have included the body class in my theme, so I know that's not the problem that I did not forget to include the tag. So, I am submitting the issue, just in case.

Windows: Command Prompt doesn't work due to shell scripts.

Windows Command Prompt typically cannot run sh scripts unless they are specifically configured.

This means that the installation of the environment will fail.

Windows users can still use the git bash shell, however.

Not able to configure properly in windows

**$ npm run start — --pathToTheme=../polite

[email protected] start E:\laragon\www\master\wp-content\themes\theme-review-action
node bin/program.js "—"

Please ensure docker is running.

Test Version: 1.4.11
Testing Ports: 8484/8485

Steps:

Copying theme files into the environment...
× Copying theme files into the environment...
Error: Cannot copy '.' to a subdirectory of itself, 'E:\laragon\www\master\wp-content\themes\theme-review-action\test-theme'.
Checking theme's basic structure
√ Checking theme's basic structure (in 2s 137ms)
Setting up the development environment for testing...
× Setting up the development environment for testing...
Error: Command failed with exit code 1: npm run install:environment
× Error while running docker-compose command.
chown: cannot access 'wp-config.php': No such file or directory
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] wp-env: wp-env "start"
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] wp-env script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! C:\Users\KafleG\AppData\Roaming\npm-cache_logs\2021-08-02T07_10_15_788Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install:environment: npm run install:variables && npm run wp-env start && node bin/configure.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install:environment script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! C:\Users\KafleG\AppData\Roaming\npm-cache_logs\2021-08-02T07_10_15_978Z-debug.log

[email protected] install:environment E:\laragon\www\master\wp-content\themes\theme-review-action
npm run install:variables && npm run wp-env start && node bin/configure.js

[email protected] install:variables E:\laragon\www\master\wp-content\themes\theme-review-action
node bin/create-wp-env-override.js

Maybe creating .wp-env override config.
No need for an override.

[email protected] wp-env E:\laragon\www\master\wp-content\themes\theme-review-action
wp-env "start"

Tearing down the environment...
× Tearing down the environment...
Error: Command failed with exit code 1: npm run wp-env destroy
i WARNING! This will remove Docker containers, volumes, and networks associated with the WordPress instance.
× Command failed: docker volume rm $(docker volume ls | grep "a1895f5400360d5fe6cad09f327fd381" | awk '/ / { print $2 }') && docker network rm $(docker network ls | grep "a1895f5400360d5fe6cad09f327fd381" | awk '/ / { print $1 }')
Error: No such volume: $(docker
Error: No such volume: volume
Error: No such volume: ls
awk: cmd. line:1: / / { print $2 })
awk: cmd. line:1: ^ syntax error

Error: Command failed: docker volume rm $(docker volume ls | grep "a1895f5400360d5fe6cad09f327fd381" | awk '/ / { print $2 }') && docker network rm $(docker network ls | grep "a1895f5400360d5fe6cad09f327fd381" | awk '/ / { print $1 }')
Error: No such volume: $(docker
Error: No such volume: volume
Error: No such volume: ls
awk: cmd. line:1: / / { print $2 })
awk: cmd. line:1: ^ syntax error

at ChildProcess.exithandler (child_process.js:390:12)
at ChildProcess.emit (events.js:400:28)
at maybeClose (internal/child_process.js:1055:16)
at Process.ChildProcess._handle.onexit (internal/child_process.js:288:5) {

killed: false,
code: 1,
signal: null,
cmd: docker volume rm $(docker volume ls | grep "a1895f5400360d5fe6cad09f327fd381" | awk '/ / { print $2 }') && docker network rm $(docker network ls | grep "a1895f5400360d5fe6cad09f327fd381" | awk '/ / { print $1 }'),
stdout: '',
stderr: 'Error: No such volume: $(docker\n' +
'Error: No such volume: volume\n' +
'Error: No such volume: ls\n' +
'awk: cmd. line:1: / / { print $2 })\n' +
'awk: cmd. line:1: ^ syntax error\n'
}
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] wp-env: wp-env "destroy"
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] wp-env script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! C:\Users\KafleG\AppData\Roaming\npm-cache_logs\2021-08-02T07_10_44_149Z-debug.log

[email protected] wp-env E:\laragon\www\master\wp-content\themes\theme-review-action
wp-env "destroy"

? Are you sure you want to continue? Yes
Error: Unable to start environment.
We ran into an error with the test framework.
**

CC @StevenDufresne

Feedback: PHP error check

Example report:
https://themes.trac.wordpress.org/ticket/99204#comment:1

/?p=1241 contains PHP errors: Notice: Undefined variable: bool in wp-content/themes/test-theme/functions.php on line 145
See: https://github.com/WordPress/theme-review-action/blob/trunk/docs/ui-errors.md#page-should-not-have-php-errors

I find the /?p=1241 confusing. I understand that this is a page or post id but when troubleshooting, it would be more helpful to know if this view is a single post or single page.
The post id will not be the same if I was to use it to troubleshoot my theme.

wp-content/themes/test-theme/functions.php
This is also confusing. Can wp-content/themes/test-theme/ be replaced with the theme name?
(And I think whoever views the report will take it more serious if the correct theme name is used.)

Can it point and link to the Trac browser even? https://themes.trac.wordpress.org/browser/bluenest/1.0.1/functions.php#L145

The docs page says:
This test expects that the plugin does not output any PHP errors.
Is it meant to say the theme does not output any PHP errors?
This can be updated to also mention notices and warnings.

Make Theme URI and Author URI Page Message Clear

Now, the message if the theme URI is invalid. It shows,

Expected to receive a 200 status for https://www.templatesell.com/item/markup-seo-wordpress-blog-theme/. Received 404.
See: https://github.com/WordPress/theme-review-action/blob/trunk/docs/ui-errors.md#page-should-return-200-status

We can change the text or we can link to the updated guidelines.

Feedback on readme instructions

In the instructions for running the tool locally, at the end, it says:
The tests run on files within the /test-theme folder.

This needs to be listed as an action before Run npm run start. because it is easy to miss.

It needs to be clearer that the theme files needs to be placed directly in the test-theme folder:
theme-review-action/test-theme/functions.php
not theme-review-action/test-theme/twentytwentyone/functions.php

-Is there anything preventing this folder from existing when you clone the tool?
Why does the test-theme folder need to be created manually, when the folder is required?

Or is this a situation where it only needs to be created manually on Windows because of #18 ?

Improvements to communication

See: #19 (comment)

Feedback: User Interface Warnings, All page content must be contained by landmarks

All page content must be contained by landmarks,
This is correct, but there is an exception which is the skip link. Is it possible to exclude it? 🤔

TwentyTwenty - DOMException: play() failed because the user didn't interact with the document first.

I run theme-review-action locally on the Twenty Twenty Theme v1.9 and I get:

User Interface Errors:
"/" (via: index.php) contains javascript errors. Found Error: DOMException: play() failed because the user didn't interact with the document first. https://goo.gl/xX8pDD
See: https://github.com/WordPress/theme-review-action/blob/trunk/docs/ui-errors.md#browser-console-should-not-contain-errors


"/?cat=1" (via: index.php) contains javascript errors. Found Error: DOMException: play() failed because the user didn't interact with the document first. https://goo.gl/xX8pDD
See: https://github.com/WordPress/theme-review-action/blob/trunk/docs/ui-errors.md#browser-console-should-not-contain-errors


"/?tag=alignment-2" (via: index.php) contains javascript errors. Found Error: DOMException: play() failed because the user didn't interact with the document first. https://goo.gl/xX8pDD
See: https://github.com/WordPress/theme-review-action/blob/trunk/docs/ui-errors.md#browser-console-should-not-contain-errors


"/?post_format=gallery" (via: index.php) contains javascript errors. Found Error: DOMException: play() failed because the user didn't interact with the document first. https://goo.gl/xX8pDD
See: https://github.com/WordPress/theme-review-action/blob/trunk/docs/ui-errors.md#browser-console-should-not-contain-errors

I struggle to find the cause of these errors because I'm working on a theme that also gets these errors. Twenty Twenty One however does not have these errors. I know these errors are triggered if autoplay is set e.g on a <video> element and not muted. Such errors are actually content-related but there must be some way to prevent them from beeing triggered when writing and testing a theme.

Could anyone give me a hint on how to prevent these errors?

Fatal errors are not properly detected when in html attributes.

For example, take this theme run: https://themes.trac.wordpress.org/ticket/106344#comment:6

The fatal error occured within a HTML attribute (See echo esc_attr_() within https://themes.trac.wordpress.org/changeset/155597) and so wasn't matched by the fatal extraction from the DOM.

I guess this means that more might need to be done to either close the attribute before outputting errors or doing some kind of string parsing on the DOM rather than simply relying upon the DOM

Remove the "this is for informational purposes only." notice

I think that we can be confident in that the JavaScript error, HTML, PHP and block template tests are correct and
the notice can be removed.

Both theme authors and reviewers have ignored the reports that has this notice, which has led to some broken themes....

Let theme review action error results prevent theme upload

Error results from theme review action does not prevent new themes or theme updates from going live in the theme directory
or from being uploaded.

First, we should determine which errors are a risk or has the most negative consequences and needs to prevent upload.

I propose preventing upload if these two checks returns errors:

Page should not have PHP errors
Browser console should not contain errors (JavaScript errors)

Can not run locally on Windows 10

I am unable to use npm run start locally on Windows 10:

Steps:
- Copying theme files into the environment...Error: Cannot copy '.' to a subdirectory of itself, 'D:\wpreviewaction\theme-review-action\test-theme'.
× Copying theme files into the environment...
√ Checking theme's basic structure

\ Setting up the development environment for testing...Error: Command failed with exit code 1: npm run install:environment
✖ EPERM: operation not permitted, rename 'C:\Users\Carolina\.wp-env\016297c788c93c3297970932e19a5611\theme-check.temp\theme-check' -> 'C:\Users\Carolina\.wp-env\016297c788c93c3297970932e19a5611\theme-check'
[Error: EPERM: operation not permitted, rename 'C:\Users\Carolina\.wp-env\016297c788c93c3297970932e19a5611\theme-check.temp\theme-check' -> 'C:\Users\Carolina\.wp-env\016297c788c93c3297970932e19a5611\theme-check'] {
  errno: -4048,
  code: 'EPERM',
  syscall: 'rename',
  path: 'C:\\Users\\Carolina\\.wp-env\\016297c788c93c3297970932e19a5611\\theme-check.temp\\theme-check',
  dest: 'C:\\Users\\Carolina\\.wp-env\\016297c788c93c3297970932e19a5611\\theme-check'
}
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] wp-env: `wp-env "start"`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] wp-env script.

Node.js v12.18.3.
NPM 6.14.11
Command prompt as administrator.
Docker 3.1.0 (51484)

So far I have tried:

Confirmed that my user does have full control to modify the .wp-env folder and subfolders including 016297c788c93c3297970932e19a5611
Deleted the node_modules folder in the theme-review-action project folder and run npm install again

Page should not have unexpected links

"en.gravatar.com" found when viewing "/?p=1" (via: single.php) is not an approved link.
See: https://github.com/WordPress/theme-review-action/blob/trunk/docs/ui-errors.md#page-should-not-have-unexpected-links

I cannot fix it. can you help me?

Thanks

Reports should not include `test-theme`

Currently reports generated by the action include the path as themes/test-theme/ which leaves some people confused.

There's two options I see:

Always install the theme into the theme-directory name
Rewrite the output from the action to replace wp-content/themes/test-theme/ to be wp-content/themes/$THEME_SLUG/

Example: https://themes.trac.wordpress.org/ticket/105885#comment:1

Unclear Message

Test Name: Should have element focus state
Found on:
Details:
Help:
https://github.com/WordPress/theme-review-action/blob/trunk/docs/ui-warnings.md#should-have-element-focus-state

Ticket: https://themes.trac.wordpress.org/ticket/103799#comment:1

Error when trying to execute the script multiple times

Hello,

When I run npm run start -- --pathToTheme=/mnt/shared/brite second time I got and error when the script tried to execute docker-compose.

If I remove the docker images the script worked again without any issue.

Here is the output of a script execution without removing the docker images:

iulian@iulian-VirtualBox:~/theme-review-action$ npm run start -- --pathToTheme=/mnt/shared/brite

> [email protected] start /home/iulian/theme-review-action
> node bin/program.js "--pathToTheme=/mnt/shared/brite"


Test Version: 1.4.11
Testing Ports: 8484/8485

Steps:
✔ Copying theme files into the environment... (in 0s 799ms)
✔ Checking theme's basic structure (in 0s 686ms)
✖ Setting up the development environment for testing...
Error: Command failed with exit code 1: npm run --prefix /home/iulian/theme-review-action/ install:environment
✖ Error while running docker-compose command.
Building wordpress
Service 'wordpress' failed to build: The command '/bin/sh -c pecl install xdebug && docker-php-ext-enable xdebug' returned a non-zero code: 1
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] wp-env: `wp-env "start"`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] wp-env script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/iulian/.npm/_logs/2021-08-17T14_59_31_493Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install:environment: `npm run install:variables && npm run wp-env start && node bin/configure.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] install:environment script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/iulian/.npm/_logs/2021-08-17T14_59_31_522Z-debug.log

> [email protected] install:environment /home/iulian/theme-review-action
> npm run install:variables && npm run wp-env start && node bin/configure.js


> [email protected] install:variables /home/iulian/theme-review-action
> node bin/create-wp-env-override.js

Maybe creating .wp-env override config.
No need for an override.

> [email protected] wp-env /home/iulian/theme-review-action
> wp-env "start"

Step 1/6 : FROM wordpress
 ---> 8a0af18ec64a
Step 2/6 : RUN apt-get -qy install $PHPIZE_DEPS && touch /usr/local/etc/php/php.ini
 ---> Running in 512b85044810
Reading package lists...
Building dependency tree...
Reading state information...
autoconf is already the newest version (2.69-11).
dpkg-dev is already the newest version (1.19.7).
file is already the newest version (1:5.35-4+deb10u2).
g++ is already the newest version (4:8.3.0-1).
gcc is already the newest version (4:8.3.0-1).
libc6-dev is already the newest version (2.28-10).
make is already the newest version (4.2.1-1.2).
pkg-config is already the newest version (0.29-6).
re2c is already the newest version (1.1.1-1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Removing intermediate container 512b85044810
 ---> bfb6045f53e1
Step 3/6 : RUN pecl install xdebug && docker-php-ext-enable xdebug
 ---> Running in d7fa3291686e
pecl/xdebug is already installed and is the same as the released version 3.0.4
install failed
✔ Tearing down the environment... (in 12s 268ms)
Error: Unable to start environment.
We ran into an error with the test framework.

gravatar.com found on / is not an approved link.

https://github.com/WordPress/theme-review-action/blob/trunk/docs/ui-errors.md#page-should-not-have-unexpected-links

Is there a list of approved/whitelisted links? If so, gravatar.com should definitely be added to it.

Thanks

GitHub Action: Annotation Spacing

With Github's recent update to the action UI, white-spacing has changed and is no longer respecting new lines.

They probably need to add white-space: pre although they may never will.

The accessible menu test is incorrect

The menu check is incorrect; opening a sub menu on hover must not be required.
Hovering is the least accessible way.
It should not even be mentioned as recommended.

NPX: Windows: Takes forever and sometimes dies

It takes forever to run NPX on my VM and sometimes dies when trying to connect to mysql.

Validate Theme URI and Author URI

We see the theme uri and author uri are invalid because of wrong information and 404 page.

Add readme validator?

Would it be possible to also add the readme validator?

The themes has slightly different requirements for the readme though as it does not have multiple contributors or stable tag. 🤔

https://meta.trac.wordpress.org/browser/sites/trunk/wordpress.org/public_html/wp-content/plugins/plugin-directory/readme

Feedback: Unexpected links

Unexpected links

It is not 100% clear from the report or docs which requirement it refers to.
I believe it is this one?
Themes may have a single footer credit link, which is restricted to the Theme URI or Author URI defined in style.css

Is it possible to include an explanation in the report, that the link is not approved because it is not also listed as the theme or author URI?
I do not understand the code well enough to see where this is confirmed.

Example report:
https://themes.trac.wordpress.org/ticket/97704#comment:10
At the point of the report, theme did not have a theme or author URI but credit links:

themearrow.com found on /?p=1241 is not an approved link.
See: https://github.com/WordPress/theme-review-action/blob/trunk/docs/ui-errors.md#page-should-not-have-unexpected-links
themearrow.com found on / is not an approved link.
See: https://github.com/WordPress/theme-review-action/blob/trunk/docs/ui-errors.md#page-should-not-have-unexpected-links
themearrow.com found on /?cat=1 is not an approved link.
See: https://github.com/WordPress/theme-review-action/blob/trunk/docs/ui-errors.md#page-should-not-have-unexpected-links
themearrow.com found on /?tag=alignment-2 is not an approved link.
See: https://github.com/WordPress/theme-review-action/blob/trunk/docs/ui-errors.md#page-should-not-have-unexpected-links
/?post_format=gallery contains PHP errors: Notice: Undefined index: with_front in wp-content/themes/test-theme/inc/breadcrumbs.php on line 529
See: https://github.com/WordPress/theme-review-action/blob/trunk/docs/ui-errors.md#page-should-not-have-php-errors
themearrow.com found on /?post_format=gallery is not an approved link.
See: https://github.com/WordPress/theme-review-action/blob/trunk/docs/ui-errors.md#page-should-not-have-unexpected-links

I would also suggest that the requirement itself is changed to
Themes can include one single front facing credit link, which is restricted to the Theme URI or Author URI defined in style.css.
I will bring that up with the themes team.

Screenshot: Enforce ratio

Redirect on theme activation is not allowed

Redirecting the user away from the theme screen when a theme is activated is not allowed.

There are two ways to redirect on activation that are more common:

global $pagenow;

if (is_admin() && 'themes.php' == $pagenow && isset($_GET['activated'])) {
    $theme_info = wp_get_theme();
    $theme_name = $theme_info->get( 'Name' );
    if($theme_name =='THEME NAME'){
        wp_redirect(admin_url("themes.php?page=prefix_activation")); // Your admin page URL
    }
}

function prefix_after_switch_theme(){
    wp_redirect('admin.php?page=prefix-panel');
}

add_action( 'after_switch_theme', 'prefix_after_switch_theme' );

Note that neither wp_redirect or after_switch_theme on their own cause any problems.
This makes it difficult to test this requirement with a static code check.

A test could be set up to check if there is a redirect after theme activation.

Requirement: Display the correct content according to the front page setting

In the WordPress admin under reading settings, "Your homepage displays:" can be set to "Your latest posts" or "A static page".

The theme directory has a requirement that if the setting is set to the latest posts, then the latest posts must show.
If it is set to a static page, then the page and its page content must show.

In both cases, it is also allowed to show both so the requirement passes if the correct content is displayed anywhere
on the page -it doesn't have to be the only content.

This is currently tested manually.

Can this be checked with an "UI check"? (or with theme review action at all).

Question: How to handle checks that are in both Theme Check and theme review action?

Some checks are in both theme check and theme review action.
Are there any benefits with this?

For example, checking if the theme folder includes a screenshot and required files.

Trac message not escaped properly

The message sent to trac is not escaped properly, for example:
https://themes.trac.wordpress.org/ticket/99043#comment:2

Path to theme is wp-content/themes/test-theme/ could probably just strip that out and leave it as includes/template-parts.php:413
WP_Hook->do_action should be displayed WP_Hook->do_action
#1 should not be linked, this can be escaped by prefixing # with !

Another option instead of escaping would be for the details to be code blocked or inserted within a html block, which takes care of 2/3 above, but the content would need to be run through nl2br() to add the <br> tags for newlines.

Examples of those approaches:

Upgrade to PHP8.x

As themes start to require more modern PHP versions, it looks like we need to upgrade the checks in this action to run with a newer version.

See https://themes.trac.wordpress.org/ticket/132579 which requires PHP 8.0 and throws errors without it.

Structure check should probably not use PHPUnit

Replace it with something that requires less configuration, especially since it needs to be run using sudo if ran through npx.

GitHub Action: Only some errors in Log file are getting added as annotations

Not all errors/warnings that are added to the log are parsed and added as annotations.

I have switched the order of the actions to see if it's related to how the errors/warnings are printed to the log but the last action that is run appears to work regardless of which action it is.

This worked initially and appears to have changed on GitHub's end sometime around the starting of December 2020.

Rename isBlockBasedTheme to isBlockTheme [priority: low]

The term or name for themes with HTML templates is now block theme not block based theme.

Block theme specific testing

Which block-based theme-specific tests are needed?

Ideas based on documentation: https://github.com/WordPress/gutenberg/tree/trunk/docs/how-to-guides/block-based-theme

Validate experimental-theme.json
Make sure block-template-part exists if included in block-templates. (It fails quietly)
Make sure it doesn't include non-core blocks? (Edit: It can)

want to run Specific checks to save time.

Is there any way possible to run only uicheck or any other check to save time? full review is time-consuming on my windows environment.

Test the tests.

I think we should write some tests that would allow us to test the tests!!!

Unexpected result from the tabbing test

I got this result in the terminal when testing Twenty Twenty-One:

Tabbing is not working as expected
Expected: <a class="skip-link screen-reader-text" href="#content">Skip to content</a>
Current: <a href="http://localhost:8485/?cat=18">years</a>
See: https://github.com/WordPress/theme-review-action/blob/trunk/docs/ui-warnings.md#should-have-logical-tabbing

Screenshot from the output/screenshots/tabbing-test folder:

Do we know why the tab did not start from the top of the page? What went wrong?
There is a generated gif called screenshots-in-order.gif inside the folder but I am unable to open that.

Feedback on the approved link error message

The error message that displayed was:

"nayrathemes.com" found when viewing "/?p=1241" (via: single.php) is not an approved link.
See: https://github.com/WordPress/theme-review-action/blob/trunk/docs/ui-errors.md#page-should-not-have-unexpected-links

The theme author wrote: We are not understand what need to do for theme review action.

https://themes.trac.wordpress.org/ticket/103019#comment:4

The documentation says:

Page should not have unexpected links
This test expects to not find links that are not approved.

Troubleshooting
Verify that the theme only includes links in the hosts list.

I don't think it is clear enough what a not approved or unexpected link is.
The hosts list is too difficult to interpret because it is not obvious that the theme and author URI are included on the list.

Test whether links are underlined.

Links within the content must be underlined and clearly distinguishable
Heading Links
Paragraph Links
List Style Links
List Style Links
Link in Comments
Text Widget Links

Redundant test results.

Some UI tests loop through default pages and log the errors. It appears that more often than not if the issue persists on the home page, it will appear on the other pages. We should consider only running a subset of UI tests on all the pages to avoid extra noise or alternatively, find a better way to log the errors.

Example:

/ should not contain javascript errors. Found Error: TypeError: Cannot use 'in' operator to search for 'clientWidth' in null at js/prod/front-end.js?ver=1614246500:1:1442 at Array.forEach (<anonymous>) at Module.238 (js/prod/front-end.js?ver=1614246500:1:1367) at n (js/prod/front-end.js?ver=1614246500:1:110) at Object.76 (js/prod/front-end.js?ver=1614246500:1:13510) at n (js/prod/front-end.js?ver=1614246500:1:110) at js/prod/front-end.js?ver=1614246500:1:902 at js/prod/front-end.js?ver=1614246500:1:912
See: https://github.com/WordPress/theme-review-action/blob/trunk/docs/ui-errors.md#browser-console-should-not-contain-errors

/?cat=1 should not contain javascript errors. Found Error: TypeError: Cannot use 'in' operator to search for 'clientWidth' in null at js/prod/front-end.js?ver=1614246500:1:1442 at Array.forEach (<anonymous>) at Module.238 (js/prod/front-end.js?ver=1614246500:1:1367) at n (js/prod/front-end.js?ver=1614246500:1:110) at Object.76 (js/prod/front-end.js?ver=1614246500:1:13510) at n (js/prod/front-end.js?ver=1614246500:1:110) at js/prod/front-end.js?ver=1614246500:1:902 at js/prod/front-end.js?ver=1614246500:1:912
See: https://github.com/WordPress/theme-review-action/blob/trunk/docs/ui-errors.md#browser-console-should-not-contain-errors

/?tag=alignment-2 should not contain javascript errors. Found Error: TypeError: Cannot use 'in' operator to search for 'clientWidth' in null at js/prod/front-end.js?ver=1614246500:1:1442 at Array.forEach (<anonymous>) at Module.238 (js/prod/front-end.js?ver=1614246500:1:1367) at n (js/prod/front-end.js?ver=1614246500:1:110) at Object.76 (js/prod/front-end.js?ver=1614246500:1:13510) at n (js/prod/front-end.js?ver=1614246500:1:110) at js/prod/front-end.js?ver=1614246500:1:902 at js/prod/front-end.js?ver=1614246500:1:912
See: https://github.com/WordPress/theme-review-action/blob/trunk/docs/ui-errors.md#browser-console-should-not-contain-errors

/?post_format=gallery should not contain javascript errors. Found Error: TypeError: Cannot use 'in' operator to search for 'clientWidth' in null at js/prod/front-end.js?ver=1614246500:1:1442 at Array.forEach (<anonymous>) at Module.238 (js/prod/front-end.js?ver=1614246500:1:1367) at n (js/prod/front-end.js?ver=1614246500:1:110) at Object.76 (js/prod/front-end.js?ver=1614246500:1:13510) at n (js/prod/front-end.js?ver=1614246500:1:110) at js/prod/front-end.js?ver=1614246500:1:902 at js/prod/front-end.js?ver=1614246500:1:912
See: https://github.com/WordPress/theme-review-action/blob/trunk/docs/ui-errors.md#browser-console-should-not-contain-errors

Halt tests if connection is refused

I tested npm install inside the ui-check folder and there were no issues.
Then I used npm start and it runs the tests but the connection is refused:

net::ERR_CONNECTION_REFUSED at http://localhost:8889/?feed=rss2

When the connection is refused, I don't think there is any point in continuing with the next test?
I mean that the tests could stop at this error.

Experiment: Test rule sheet

This is an auto-generated markdown table. Edits here will be replaced.

ID	Category	Type		Rule	Details	Exception	Notes
safe1	Privacy	All Themes	must	disable any tracking and collection of user data by default and must be opt-in.
safe2	Privacy	All Themes	must	include documentation on how any user data is collected, and used, and needs to be included in the theme readme.txt file, preferably with a clearly stated privacy policy.
safe3	Code	All Themes	can't	have PHP or JavaScript errors, warnings or notices.
safe4	Code	All Themes	must	validate and/or sanitize untrusted data before entering it into the database
safe5	Code	All Themes	must	escape all untrusted data before output (See: Data Validation)
safe6	Code	All Themes	All Themes	Provide a unique prefix for everything the theme defines in the public namespace	including options, functions, global variables, constants, post meta, wp_enqueue_script/style handle names, add_image_size names, wp_script_add_data keys, slugs/ids for new categories created with register_block_pattern_category etc.	unless its a menu location or sidebar id.
safe7	Importing or downloading	All Themes	can't	import content to a user’s site
safe8	Importing or downloading	All Themes	can't	link directly to an XML, JSON, ZIP, or other files for direct download or import
safe9	Importing or downloading	All Themes	can't	bundle demo content via an XML, JSON, ZIP, or other files
safe10	Plugins	All Themes	can't	include plugins in the theme folder or download plugins automatically
safe11	Plugins	All Themes	can't	include plugin functionality		If you are not sure if a feature is plugin territory, contact the team and ask first. [email protected].
safe12	Plugins	All Themes	can	recommend plugins that are hosted on WordPress.org
safe13	Plugins	All Themes	must	only install plugins by installed by user action
safe14	Plugins	All Themes	can	recommend GPL compatible plugins that are not hosted on WordPress.org in the readme file or the themes information page but may not include a direct link to the download for security reasons.
safe15	Plugins	All Themes	can	include a link to the product page for the recommended plugin	but not directly to the download file.
global1	Accessibility	All Themes	must	have skip links	that include a mechanism that enables users to navigate directly to content or navigation on entering any given page	unless it is a block theme, skip links are added automatically to the element.
global2	Accessibility	All Themes	must	have skip links	that may be positioned off-screen initially but must be available to screen reader users and must be visible on focus for sighted keyboard navigators
global3	Accessibility	All Themes	must	have skip links	that are the first focusable element perceived by a user via a screen reader or keyboard navigation
global4	Accessibility	All Themes	must	have skip links	that are visible when keyboard focus moves to the link
global5	Accessibility	All Themes	must	have skip links	that move focus to the main content area of the page when activated	unless there is nothing to skip past, such as a menu or larger header section or secondary widget area before the main content.
global6	Accessibility	All Themes	must	have keyboard navigation	that provide visual keyboard focus highlighting in navigation menus and for form fields, submit buttons and text links.
global7	Accessibility	All Themes	must	have keyboard navigation	that makes all controls and links reachable by keyboard.
global8	Accessibility	All Themes	must	have keyboard navigation	that makes all controls usable with the mouse usable with the keyboard, regardless of device and screen size. Including but not limited to responsive versions for small screens, mobile and other touch screen devices. Further reading.
global9	Accessibility	All Themes	must	have underlined links within content and comments	that are distinguishes by an underlined a no other style	unless they are in navigation-like contexts (e.g. menus, lists of upcoming posts in widgets, grouped post meta data)
global10	Accessibility	All Themes	must	meet additional requirements if the theme has the tag ‘accessibility-ready’
global11	Language & internationalization	All Themes	must	use gettext for all text strings for translation
global12	Language & internationalization	All Themes	must	include the theme slug as the text-domain in style.css	that is the name of the theme in lower case, with spaces replaced by -. It is also the folder name for the theme.
global13	Language & internationalization	All Themes		If the theme uses a framework then no more than 2 unique slugs may be used (like tgmpa, redux-framework, kirki or some other allowed framework)
global14	Language & internationalization	All Themes	can	use any language for text	that only uses one language
quality9	Functionality and Features	All Themes	must	use the admin_notices API for all notifications generated by the theme.
quality10	Functionality and Features	All Themes	must	make notices dismissible.
quality11	Functionality and Features	All Themes	must	follow core UI design for everything wrapped in the admin notice
quality12	Functionality and Features	All Themes	can't	place WordPress features behind a paywall
quality13	Functionality and Features	All Themes	can't	remove, hide, or otherwise block the admin bar from appearing
quality14	Functionality and Features	All Themes	can't	redirect on theme activation or modify activation
quality15	Presentation vs Functionality	All Themes	can't	include custom post types
quality16	Presentation vs Functionality	All Themes	can't	include custom blocks
quality17	Presentation vs Functionality	All Themes	can't	include shortcodes
quality18	Presentation vs Functionality	All Themes	can't	include functionality that is not related to design and presentation.
distribute1	Selling, credits, and links	All Themes	can	include one single front facing credit link, which is restricted to the Theme URI or Author URI defined in style.css
distribute2	Selling, credits, and links	All Themes	can	have an additional footer credit link pointing to WordPress.org
distribute3	Selling, credits, and links	All Themes	must	state explicitly that the products you’re selling/distributing (free and paid) are GPL compatible	that needs to be in an easy-to-find place for visitors.
distribute4	Selling, credits, and links	All Themes	can't	display “obtrusive” upselling
distribute5	Selling, credits, and links	All Themes	can't	have affiliate URLs or links
distribute6	Licensing & copyright	All Themes	must	be compatible with the GNU General Public License			Although any GPL-compatible license is acceptable, using the same license as WordPress — “GPLv2 or later” — is strongly recommended. All code, data, and images — anything in the theme zip file — must comply with the GPL or a GPL-Compatible license.
distribute7	Licensing & copyright	All Themes	must	include third-party libraries, code, images, or otherwise that are GPL-compatible			For a specific list of compatible licenses, please read the GPL-Compatible license list on gnu.org.
distribute8	Licensing & copyright	All Themes	must	declare copyright for the theme itself.
distribute9	Licensing & copyright	All Themes	must	declare license, copyright information, and source for all resources included such as fonts or images.	that is provided in a list of all resources in one file.	unless the assets are public domain
distribute10	Licensing & copyright	All Themes	must	include code and design that are your own or legally yours			Cloning of designs is not acceptable
distribute11	Licensing & copyright	All Themes	must	only display the user’s copyright	that is not the theme author’s copyright.		Front end
distribute12	Files	All Themes	must	include all scripts, images, videos and other resources rather than hot-linking.		unless the resource is from Google Fonts.
distribute13	Files	All Themes		Main stylesheet
distribute14	Files	All Themes	must	use headers in style.css that follow the guidelines and requirements for the main stylesheet in the Theme Developer Handbook.
distribute15	Files	All Themes	can	include 'Theme URI' in style.css	that must be about the theme hosted on WordPress.org. If the URI is a demo site, the content must be about the theme itself and not test data and cannot be wordpress.org.		It's reserved for the default themes (Twenty *).
distribute16	Files	All Themes	can	include 'Author URI' in style.css	that links to a page or website about the author, author theme shop, or author project/development website.
distribute17	Files	All Themes	must	have tags in style.css that match what the theme actually does in respect to functionality and design	and doesn't use more than 3 subject tags (See: Theme Tag List).
distribute18	Files	All Themes	must	include a readme.txt file	that doesn't have empty lines in the file header
distribute19	Files	All Themes	must	include a readme.txt file	that has the 'tested up to' field
distribute20	Files	All Themes	must	include a readme.txt file	That has onle one WordPress.org username listed in the contributors field.
distribute21	Files	All Themes	can't	have minification of scripts or files		unless the original files are also in the theme folder.
distribute22	Files	All Themes	must	use WordPress’ default libraries.			WordPress includes a number of libraries such as jQuery. For security and stability reasons themes may not include those libraries in their own code. Instead themes must use the versions of those libraries packaged with WordPress. For a list of all JavaScript libraries included in WordPress, please review Default Scripts Included and Registered by WordPress.
distribute23	Files	All Themes	can't	have images that promote hate or violence or images that show children with recognizable facial or body features.
distribute24	Files	All Themes	can't	have a screenshot that looks like an advertisement			The reviewer can subjectively ask you to change screenshots if they find that it is not appropriate.
distribute25	Files	All Themes	can't	have a screenshot bigger than 1200 x 900px
distribute26	Files	All Themes	must	a ratio of width to height of 4:3
distribute33	Presentation vs Functionality	All Themes		Warning: Showing preview/demo data or manipulating the preview on WordPress.org is not allowed and can result in suspension or your user account being terminated
distribute34	Naming, spelling and trademarks	All Themes	can't	use: WordPress, Theme, Twenty* in their name
distribute35	Naming, spelling and trademarks	All Themes	must	spell “WordPress” correctly in all public-facing text: all one word, with both an uppercase W and P
distribute36	Naming, spelling and trademarks	All Themes	can't	have violation of trademarks.

System

These rules are related to uploading themes to .org.

ID	Category	Rule
system1	Theme author and theme upload restrictions	Only submit one new theme at the time. You can submit unlimited updates for your existing themes that are in the theme directory.
system2	Theme author and theme upload restrictions	A theme must be complete at the time of submission. Names cannot be “reserved” for future use or to protect brands.
system3	Theme author and theme upload restrictions	Licensing of themes distributed outside the theme directory
system4	Theme author and theme upload restrictions	If you distribute themes, you may only distribute themes that are 100% compatible with GPL.
system5	Theme author and theme upload restrictions	Otherwise you can not add themes to the WordPress.org Theme Directory (See explanation).
system6	Theme author and theme upload restrictions	Example: If you have a Themeforest account and you’re selling themes on it, all those themes need to state on their sales page that they are 100% GPL compatible (Info).
system7	Theme author and theme upload restrictions	You can have multiple accounts with the following restrictions:
system8	Theme author and theme upload restrictions	You can’t have more than one (1) open ticket in any of the trac reports/queues or under review. That means you can’t have one (1) theme ticket from an account and another one from a secondary account, simultaneously open in any of the queues.
system9	Theme author and theme upload restrictions	Failing to respect the above requirement will result in the closing of all tickets and not having the possibility to upload those themes again. Also, a 1 month no upload possibility for each ticket closed. Depending on the severity of the case, you might also end up with a permanent ban on all your accounts.
system10	Theme author and theme upload restrictions	To avoid penalties, the team requests that you disclose all your accounts by emailing us at themes[at]wordpress.org.
system11	Naming, spelling and trademarks	The themes team can decline themes based on the name and can request that the name be changed, if they deem the name inappropriate or too similar to the name of an existing theme or brand.

Failed to open stream: No such file or directory in wp-content/themes/...

I am getting this error:
"/?feed=rss2" contains PHP errors: Warning: require_once(wp-content/themes/test-theme\inc\classes\class-the-one.php): Failed to open stream: No such file or directory in wp-content/themes/the-one/inc/helpers/autoloader.php

At first glance it looks like test-theme is hardcoded, I guess that is why I am getting this error.