replayio / playwright-tests Goto Github PK

https://lit.dev/playground/#sample=examples/directive-if-defined

Login to discord
Visit a public server

Our backend currently needs a manifest for knowing about the new tests. Can we update it to run all of the tests in the qa_wolf directory?

The yelp test occassionally get stuck finding the first restaurant

Marko is a JS framework https://markojs.com/

It might be fun to use their playground to edit some of the code and see it re-render

https://markojs.com/try-online/?file=%2Fcomponents%2Fdynamic-tagname%2Fdynamic-component.marko

1. go to https://bonobos.com/
2. click on a listing
3. add the listing to the cart

1. go to https://open.spotify.com/genre/shows_with_music-page
2. log in as a fake facebook user (if possible)
3. select some tracks

This error occurs frequently running running our playwright tests against chromium, causing the test to fail. It doesn't seem like there is a process crash anywhere and that we should be robust enough to recover if this is thrown in pageLog.

/home/ubuntu/recordreplay/playwright-tests/node_modules/playwright/lib/utils/stackTrace.js:48
    const stack = new Error().stack;
                  ^
page.evaluate: Execution context was destroyed, most likely because of a navigation.
Error:
    at Object.captureStackTrace (/home/ubuntu/recordreplay/playwright-tests/node_modules/playwright/lib/utils/stackTrace.js:48:19)
    at Connection.sendMessageToServer (/home/ubuntu/recordreplay/playwright-tests/node_modules/playwright/lib/client/connection.js:69:48)
    at Proxy.<anonymous> (/home/ubuntu/recordreplay/playwright-tests/node_modules/playwright/lib/client/channelOwner.js:64:61)
    at /home/ubuntu/recordreplay/playwright-tests/node_modules/playwright/lib/client/frame.js:179:42
    at Frame._wrapApiCall (/home/ubuntu/recordreplay/playwright-tests/node_modules/playwright/lib/client/channelOwner.js:77:34)
    at Frame.evaluate (/home/ubuntu/recordreplay/playwright-tests/node_modules/playwright/lib/client/frame.js:178:21)
    at /home/ubuntu/recordreplay/playwright-tests/node_modules/playwright/lib/client/page.js:385:60
    at Page._attributeToPage (/home/ubuntu/recordreplay/playwright-tests/node_modules/playwright/lib/client/page.js:231:20)
    at Page.evaluate (/home/ubuntu/recordreplay/playwright-tests/node_modules/playwright/lib/client/page.js:385:21)
    at pageLog (/home/ubuntu/recordreplay/playwright-tests/src/helpers.js:82:17)
undefined

/home/ubuntu/recordreplay/playwright-tests/utils.ts:55
    throw new Error(`Spawned process failed with exit code ${rv.status}`);
          ^
Error: Spawned process failed with exit code 1
    at Object.spawnChecked (/home/ubuntu/recordreplay/playwright-tests/utils.ts:55:11)
    at /home/ubuntu/recordreplay/playwright-tests/run.ts:205:3
    at step (/home/ubuntu/recordreplay/playwright-tests/run.ts:44:23)
    at Object.next (/home/ubuntu/recordreplay/playwright-tests/run.ts:25:53)
    at /home/ubuntu/recordreplay/playwright-tests/run.ts:19:71
    at new Promise (<anonymous>)
    at __awaiter (/home/ubuntu/recordreplay/playwright-tests/run.ts:15:12)
    at runTest (/home/ubuntu/recordreplay/playwright-tests/run.ts:163:12)
    at /home/ubuntu/recordreplay/playwright-tests/run.ts:230:11
    at step (/home/ubuntu/recordreplay/playwright-tests/run.ts:44:23)

1. go to https://demo.vercel.store/product/lightweight-jacket
2. click add to cart

1. go to https://www.livechat.com/
2. click the chat box in the bottom right

https://finviz.com/

Example video

Plan

1. go to https://finviz.com/
2. click maps link in the header
3. click one of the boxes in the heat map
4. click bubbles link in the left
5. mouse over some circles

The playwright test suite stopped running recently we should bring it back and ensure that we have telemetry wired up so we can be alerted in honeycomb if it stops running again

• Here are the docs on running the suite
• The suite should run on a variety of platforms, mac, linux, etc... but we can pick one for now (probably macstadium) and get that working again
• Backend runner (https://github.com/RecordReplay/backend/tree/master/test/runlive)

We have one telemetry event for when the live tests fail to create a recording

• https://github.com/RecordReplay/backend/blob/16a796be1a4d25f598cd17c2a77639e6be677ddd/docs/telemetry.md#errorrunlivenorecording
• https://github.com/RecordReplay/backend/blob/16a796be1a4d25f598cd17c2a77639e6be677ddd/src/instance/telemetryServer.ts#L149
• https://ui.honeycomb.io/replay/datasets/backend/result/cGjHmuMq8mg?hideMarkers

It'd be nice to also track attempts so that we can measure uptime and alert when the service is down and we stop trying

Our automated tests need to be able to update to the latest chromium browser.

When I click on a video in youtube with the cli, then skip ad does not skip the ad.

1. go to https://prosemirror.net/examples/basic/
2. scroll down and edit the text in the editor

1. go to https://www.yelp.com/
2. search for thai food
3. click on the first restaurant link in the results
4. wait for the page load
5. close the browser

Currently failing to record pretty regularly

We should upload some metadata about the new recording to our database so we can view the tests after the fact.

Fields:

• recording id
• url
• title
• lastScreen
• recording title (could be [PASS|FAIL] ${title}
• userId (could be a standard user like [email protected]

https://next.keystonejs.com/guides/document-fields#how-to-use-document-fields

We should have a test that

1. goes to https://natto.dev/
2. hovers in the top left and clicks weather
3. clicks run on some of the cells

similar to this replay https://app.replay.io/recording/8a34da01-a7a1-40fe-aac0-34fb2d79e2de?point=6814889627197216239043040440746101&time=3005.21325648415&hasFrames=true

https://tiptap.dev/

Example Video

Plan

1. go to https://tiptap.dev/examples/default
2. select React tab
3. select text and click on the commands like bold, ...
4. edit the text in the editor e.g. add bullets, code blocks, ...

We have a utils.js file (separate from the other utils.ts!) that is used by a single test for launching playwright and supports testing with chromium. This file should be removed, and we should support using chromium and a headful browser for tests that use the existing example() runner.

The facebook fuzzing test almost always gets stuck when trying to add a post. It either doesn't type anything into the new post box, or it doesn't click on the "post" button afterwards. Examples:

https://replay.io/view?id=936f06b8-923c-4a3a-9635-12be994b10f6
https://replay.io/view?id=2d6d6669-322e-434a-b022-c223911fee0b

Currently we store the list of playwright tests we use during automated testing in our backend repository, to avoid exposing the credentials used for the facebook test. I don't know how much that really matters but it does make it awkward to keep the list of tests in sync with the repository itself. It would be better if we had a manifest in the playwright-tests repository that includes all the other tests, in any case.

https://gist.github.com/tnolet/d3da0d7db33e6cd278b6b9195e3d78be

https://github.com/ryansolid/solid

This playground might be a good example to start with.

also the hn news site

Refactor existing tests to use smarter selectors and example helper.

• airbnb #32
• airtable
• climatescape #35
• codesandbox #34
• cypress #36
• discord
• espn #37
• firebugs #42
• framer
• monaco #45
• notion #46
• nytimes #47
• reactcharts #49
• reactvirtualized #50
• realadvisor #51
• reddit #55
• replay #60
• storybook #52
• tablecheck #53
• unsplash #56
• vscode #58
• whatsmyuseragent #57
• yelp #30

https://www.loom.com/share/38f5ecbbbc5d4ca4846887aa20c90e2f

1. go to https://quilljs.com/playground/
2. click in text editor and type
3. select text and bold it

• framer
• discord
• airtable

working on checkly reminded me that it is fine to have an authentication step

It would be nice to have a log of how many tests reached the end

The logic for downloading and updating browsers predates when we created and started using the @recordreplay/playwright package. The latter package takes care of downloading browsers, and the custom logic in this repo duplicates that effort, puts the browsers in a different place, has to download the recording driver by itself (also no longer necessary), and doesn't recognize using the RECORD_REPLAY_DIRECTORY env var to set another directory with browsers to use. This logic should be removed, so we can fully rely on the playwright package to manage the installed browsers.

Several examples use chromium instead of firefox, which we can't record yet. Also, many of the examples set { headless: false }, which doesn't work with the linux container when we run our automated tests.

It would be nice to see a dashboard of the runlive examples:

1. show example runs
2. show passes/fails defined by reaching the end
3. replays are one click away

Lets test adding an item to the cart

example

1. go to https://react-window.vercel.app/#/examples/list/memoized-list-items
2. select the examples in the examples list

https://www.loom.com/share/b53ddcd91f2643e98693eec1d804b5c9

https://www.pixijs.com/

Some of these examples will make more sense than others. We want the simplest example of a react app with minimal webgl re-renders

This is related to this issue too
replayio/gecko-dev#325

It should be easy to write a test that goes to a simple replit and views the code / output.

We might be able to do more if we copy their selenium test
https://replit.com/@sergeichestakov/selenium

1. go to https://www.denydesigns.com/collections/credenza
2. select other products below

We'll be talking with their QA team so it would be nice to have a couple playwright tests

https://www.loom.com/share/a1001c137f78427b867e290ab32cb440

1. go to https://remirror.io/
2. start typeing @j and click joe
3. Select text and apply commands like bolding

https://app.tryeraser.com/workspace/D52HdNerC78CdgEDsv3B

1. go to https://www.etsy.com/
2. search for tables in the search box
3. click on some listings
4. add a listing to the cart