Switch to Playwright for automated tests about datasette-app

[simonw]

microsoft/playwright#2632 (comment) shows how to activate menu items.

[simonw]

This script worked, saved as test-playwright.js and run using node test-playwright.js:

const { _electron: electron } = require('playwright');

(async () => {
  const electronApp = await electron.launch({ args: ['main.js'] });

  // Evaluation expression in the Electron context.
  const appPath = await electronApp.evaluate(async ({ app }) => {
    // This runs in the main Electron process, parameter here is always
    // the result of the require('electron') in the main app script.
    return app.getAppPath();
  });
  console.log(appPath);

  // Get the first window that the app opens, wait if necessary.
  const window = await electronApp.firstWindow();
  console.log('Window title:', await window.title());
  await window.screenshot({ path: 'test-loading.png' });
  // Direct Electron console to Node terminal.
  window.on('console', console.log);
  // Wait for "Run SQL" link to appear
  await window.waitForSelector('#run-sql-link');
  console.log('Window title:', await window.title());
  await window.screenshot({ path: 'test-homepage.png' });
  await electronApp.close();
})();

[simonw]

I'm going to try using Mocha for that, and running it with npm test.

[simonw]

I can drop spectron as a dependency, switching in Playwright.

[simonw]

CI failed: https://github.com/simonw/datasette-app/runs/7309831752?check_suite_focus=true

  Application launch
    1) shows an initial window
  0 passing (15s)
  1 failing
  1) Application launch
       shows an initial window:
     Error: Timeout of 10000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/Users/runner/work/datasette-app/datasette-app/test/spec.js)
      at listOnTimeout (node:internal/timers:559:17)
      at processTimers (node:internal/timers:502:7)

[simonw]

https://github.com/microsoft/playwright-github-action suggests:

$ npx playwright install-deps # install dependencies for all browsers
$ npx playwright install-deps chromium # install dependencies for Chromium only

But I'm testing an Electron app, so Electron should have installed a browser already I think.

[simonw]

This demo repo looks useful: https://github.com/tanshuai/electron-playwright-e2e-test-quick-start

Relevant bits from https://github.com/tanshuai/electron-playwright-e2e-test-quick-start/blob/1e2c653bc2d0af85d98de1c58e56a888f17fe671/package.json

{
  "main": "main.js",
  "scripts": {
    "start": "electron .",
    "test": "xvfb-maybe -- playwright test"
  },
  "devDependencies": {
    "@playwright/test": "^1.16.3",
    "electron": "^15.3.1",
    "playwright": "^1.16.3",
    "xvfb-maybe": "^0.2.1"
  }
}

xvfb-maybe: https://www.npmjs.com/package/xvfb-maybe

This package runs an arbitrary executable / args under xvfb-run if the platform is Linux and DISPLAY isn't set. This is super useful for making Electron unit tests run correctly in CI environments while still working locally

Here's the commit that added xvfb-maybe there: tanshuai/electron-playwright-e2e-test-quick-start@3188344

[simonw]

On my Mac:

npm install xvfb-maybe
npx xvfb-maybe -- npm test

The tests passed for me locally.

playwright test didn't work.

[simonw]

Ran this to add it as a dev dependency:

npm install xvfb-maybe --save-dev

I'll call it with npx in the GitHub Actions workflow.

[simonw]

Failed with the same error. Maybe I need to bump up the timeout for this test:

datasette-app/test/spec.js

Lines 14 to 17 in 326150b

	it("shows an initial window", async function () {
	const window = await this.app.firstWindow();
	await window.waitForSelector('#run-sql-link');
	});

It's waiting for all of the stuff to install on the loading screen, maybe that takes longer than 10s in CI sometimes?

[simonw]

https://playwright.dev/docs/api/class-page#page-wait-for-selector says the default timeout is 30s, not 10s.

I think this is a Mocha timeout. Can be changed with this.timeout(5000);.

[simonw]

This example doesn't use Mocha, it uses Playwright test directly: https://github.com/spaceagetv/electron-playwright-example/blob/master/e2e-tests/main.spec.ts

[simonw]

Failed again, this time with the 30s timeout:

2022-07-12T20:56:15.4594910Z   Application launch
2022-07-12T20:56:48.4407420Z     1) shows an initial window
2022-07-12T20:56:49.7482580Z 
2022-07-12T20:56:49.7583040Z 
2022-07-12T20:56:49.7684270Z Failed with exit code: 1
2022-07-12T20:56:49.7685320Z   0 passing (34s)
2022-07-12T20:56:49.7785970Z Output:
2022-07-12T20:56:49.7788250Z   1 failing
2022-07-12T20:56:49.7885630Z 
2022-07-12T20:56:49.7885700Z 
2022-07-12T20:56:49.7897390Z   1) Application launch
2022-07-12T20:56:49.7900000Z        shows an initial window:
2022-07-12T20:56:49.7902670Z      Error: Timeout of 30000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/Users/runner/work/datasette-app/datasette-app/test/spec.js)
2022-07-12T20:56:49.7904680Z       at listOnTimeout (node:internal/timers:559:17)
2022-07-12T20:56:49.7906280Z       at processTimers (node:internal/timers:502:7)

https://github.com/simonw/datasette-app/runs/7310066151?check_suite_focus=true

[simonw]

https://playwright.dev/docs/intro#installation says I need to do this for playwright test to work:

npm i -D @playwright/test

[simonw]

I had to rename test/spec.js to test/spec.mjs in order to use import ... in it.

[simonw]

Same failure again: https://github.com/simonw/datasette-app/runs/7310434199?check_suite_focus=true

> playwright test
Running 1 test using 1 worker
T
  1) test/spec.mjs:4:1 › App launches and quits ====================================================
Timeout of 30000ms exceeded.
  Slow test file: test/spec.mjs (30s)
  Consider splitting slow test files to speed up parallel execution
Failed with exit code: 1
  1 failed
Output:
    test/spec.mjs:4:1 › App launches and quits =====================================================

[simonw]

I'd like to get it to record a video. Here's a clue: https://github.com/microsoft/playwright/pull/10810/files#diff-25a018cac50f8f5e3a9f2a7691a61391be2e5995e74d9cc03a058be43163a795

  const app = await playwright._electron.launch({
    args: [path.join(__dirname, 'electron-window-app.js')],
    recordVideo: { dir: testInfo.outputPath('video') }
  });

[simonw]

That didn't seem to work: https://github.com/simonw/datasette-app/actions/runs/2659525079

[simonw]

I tried running tmate and managed to get the tests to pass!

npm test

> [email protected] test
> playwright test


Running 1 test using 1 worker

·
  Slow test file: test/spec.mjs (23s)
  Consider splitting slow test files to speed up parallel execution

  1 passed (24s)

I wonder if that's because the second time you run them they take less time, because the Python virtual environment has already been setup?

[simonw]

Also, the run that I used tmate in DID produce videos.

Here's the first video, which looks like it didn't install everything within the time limit. I converted it like this:

ffmpeg -i bc74c2a51bd91fe6f6cb815e6b99b6c7.webm bc74c2a51bd91fe6f6cb815e6b99b6c7.mp4

bc74c2a51bd91fe6f6cb815e6b99b6c7.mp4

And a subsequent video showing it was much faster the second time:

a1f192a041f93158bba4e02d75139dcd.mp4

I'll try bumping up the time limit again.

[simonw]

I'm dropping xvfb-maybe because the runner is macos-latest, not Ubuntu.

[simonw]

I'm going to see if I can retry the failed test once, using this pattern: https://github.community/t/how-to-retry-a-failed-step-in-github-actions-workflow/125880/2

[simonw]

The tests passed using that trick! Wish I knew why.

[simonw]

I downloaded and installed the build from https://github.com/simonw/datasette-app/runs/7310850328?check_suite_focus=true - it worked!

I ran rm -rf ~/.datasette-app and launched it again and it took almost exactly 30s to install everything, which explains why my tests have been failing at the 30s mark.

[simonw]

I'm going to bump the time limit up to 90s and see if that means I don't need to retry the test.

[simonw]

It timed out at 30s, there must be another timeout being applied.

[simonw]

Tests passed after 57s.

I'm going to leave the test retry in though, mainly as it's a useful pattern to have recorded somewhere.

[simonw]

Wrote this up as a TIL: https://til.simonwillison.net/electron/testing-electron-playwright