replayio / playwright-tests Goto Github PK

Lets test adding an item to the cart

example

1. go to https://bonobos.com/
2. click on a listing
3. add the listing to the cart

We have a utils.js file (separate from the other utils.ts!) that is used by a single test for launching playwright and supports testing with chromium. This file should be removed, and we should support using chromium and a headful browser for tests that use the existing example() runner.

It would be nice to have a log of how many tests reached the end

It should be easy to write a test that goes to a simple replit and views the code / output.

We might be able to do more if we copy their selenium test
https://replit.com/@sergeichestakov/selenium

The facebook fuzzing test almost always gets stuck when trying to add a post. It either doesn't type anything into the new post box, or it doesn't click on the "post" button afterwards. Examples:

https://replay.io/view?id=936f06b8-923c-4a3a-9635-12be994b10f6
https://replay.io/view?id=2d6d6669-322e-434a-b022-c223911fee0b

1. go to https://www.etsy.com/
2. search for tables in the search box
3. click on some listings
4. add a listing to the cart

https://tiptap.dev/

Example Video

Plan

1. go to https://tiptap.dev/examples/default
2. select React tab
3. select text and click on the commands like bold, ...
4. edit the text in the editor e.g. add bullets, code blocks, ...

1. go to https://open.spotify.com/genre/shows_with_music-page
2. log in as a fake facebook user (if possible)
3. select some tracks

Our automated tests need to be able to update to the latest chromium browser.

Marko is a JS framework https://markojs.com/

It might be fun to use their playground to edit some of the code and see it re-render

https://markojs.com/try-online/?file=%2Fcomponents%2Fdynamic-tagname%2Fdynamic-component.marko

https://www.loom.com/share/a1001c137f78427b867e290ab32cb440

1. go to https://remirror.io/
2. start typeing @j and click joe
3. Select text and apply commands like bolding

1. go to https://react-window.vercel.app/#/examples/list/memoized-list-items
2. select the examples in the examples list

https://www.loom.com/share/b53ddcd91f2643e98693eec1d804b5c9

Currently failing to record pretty regularly

https://gist.github.com/tnolet/d3da0d7db33e6cd278b6b9195e3d78be

1. go to https://www.yelp.com/
2. search for thai food
3. click on the first restaurant link in the results
4. wait for the page load
5. close the browser

When I click on a video in youtube with the cli, then skip ad does not skip the ad.

https://www.pixijs.com/

Some of these examples will make more sense than others. We want the simplest example of a react app with minimal webgl re-renders

This is related to this issue too
replayio/gecko-dev#325

Login to discord
Visit a public server

1. go to https://www.livechat.com/
2. click the chat box in the bottom right

The logic for downloading and updating browsers predates when we created and started using the @recordreplay/playwright package. The latter package takes care of downloading browsers, and the custom logic in this repo duplicates that effort, puts the browsers in a different place, has to download the recording driver by itself (also no longer necessary), and doesn't recognize using the RECORD_REPLAY_DIRECTORY env var to set another directory with browsers to use. This logic should be removed, so we can fully rely on the playwright package to manage the installed browsers.

1. go to https://prosemirror.net/examples/basic/
2. scroll down and edit the text in the editor

https://lit.dev/playground/#sample=examples/directive-if-defined

Our backend currently needs a manifest for knowing about the new tests. Can we update it to run all of the tests in the qa_wolf directory?

We'll be talking with their QA team so it would be nice to have a couple playwright tests

The playwright test suite stopped running recently we should bring it back and ensure that we have telemetry wired up so we can be alerted in honeycomb if it stops running again

• Here are the docs on running the suite
• The suite should run on a variety of platforms, mac, linux, etc... but we can pick one for now (probably macstadium) and get that working again
• Backend runner (https://github.com/RecordReplay/backend/tree/master/test/runlive)

We have one telemetry event for when the live tests fail to create a recording

• https://github.com/RecordReplay/backend/blob/16a796be1a4d25f598cd17c2a77639e6be677ddd/docs/telemetry.md#errorrunlivenorecording
• https://github.com/RecordReplay/backend/blob/16a796be1a4d25f598cd17c2a77639e6be677ddd/src/instance/telemetryServer.ts#L149
• https://ui.honeycomb.io/replay/datasets/backend/result/cGjHmuMq8mg?hideMarkers

It'd be nice to also track attempts so that we can measure uptime and alert when the service is down and we stop trying

1. go to https://www.denydesigns.com/collections/credenza
2. select other products below

https://finviz.com/

Example video

Plan

1. go to https://finviz.com/
2. click maps link in the header
3. click one of the boxes in the heat map
4. click bubbles link in the left
5. mouse over some circles

1. go to https://demo.vercel.store/product/lightweight-jacket
2. click add to cart

The yelp test occassionally get stuck finding the first restaurant

We should have a test that

1. goes to https://natto.dev/
2. hovers in the top left and clicks weather
3. clicks run on some of the cells

similar to this replay https://app.replay.io/recording/8a34da01-a7a1-40fe-aac0-34fb2d79e2de?point=6814889627197216239043040440746101&time=3005.21325648415&hasFrames=true

This error occurs frequently running running our playwright tests against chromium, causing the test to fail. It doesn't seem like there is a process crash anywhere and that we should be robust enough to recover if this is thrown in pageLog.

/home/ubuntu/recordreplay/playwright-tests/node_modules/playwright/lib/utils/stackTrace.js:48
    const stack = new Error().stack;
                  ^
page.evaluate: Execution context was destroyed, most likely because of a navigation.
Error:
    at Object.captureStackTrace (/home/ubuntu/recordreplay/playwright-tests/node_modules/playwright/lib/utils/stackTrace.js:48:19)
    at Connection.sendMessageToServer (/home/ubuntu/recordreplay/playwright-tests/node_modules/playwright/lib/client/connection.js:69:48)
    at Proxy.<anonymous> (/home/ubuntu/recordreplay/playwright-tests/node_modules/playwright/lib/client/channelOwner.js:64:61)
    at /home/ubuntu/recordreplay/playwright-tests/node_modules/playwright/lib/client/frame.js:179:42
    at Frame._wrapApiCall (/home/ubuntu/recordreplay/playwright-tests/node_modules/playwright/lib/client/channelOwner.js:77:34)
    at Frame.evaluate (/home/ubuntu/recordreplay/playwright-tests/node_modules/playwright/lib/client/frame.js:178:21)
    at /home/ubuntu/recordreplay/playwright-tests/node_modules/playwright/lib/client/page.js:385:60
    at Page._attributeToPage (/home/ubuntu/recordreplay/playwright-tests/node_modules/playwright/lib/client/page.js:231:20)
    at Page.evaluate (/home/ubuntu/recordreplay/playwright-tests/node_modules/playwright/lib/client/page.js:385:21)
    at pageLog (/home/ubuntu/recordreplay/playwright-tests/src/helpers.js:82:17)
undefined

/home/ubuntu/recordreplay/playwright-tests/utils.ts:55
    throw new Error(`Spawned process failed with exit code ${rv.status}`);
          ^
Error: Spawned process failed with exit code 1
    at Object.spawnChecked (/home/ubuntu/recordreplay/playwright-tests/utils.ts:55:11)
    at /home/ubuntu/recordreplay/playwright-tests/run.ts:205:3
    at step (/home/ubuntu/recordreplay/playwright-tests/run.ts:44:23)
    at Object.next (/home/ubuntu/recordreplay/playwright-tests/run.ts:25:53)
    at /home/ubuntu/recordreplay/playwright-tests/run.ts:19:71
    at new Promise (<anonymous>)
    at __awaiter (/home/ubuntu/recordreplay/playwright-tests/run.ts:15:12)
    at runTest (/home/ubuntu/recordreplay/playwright-tests/run.ts:163:12)
    at /home/ubuntu/recordreplay/playwright-tests/run.ts:230:11
    at step (/home/ubuntu/recordreplay/playwright-tests/run.ts:44:23)

Several examples use chromium instead of firefox, which we can't record yet. Also, many of the examples set { headless: false }, which doesn't work with the linux container when we run our automated tests.

Refactor existing tests to use smarter selectors and example helper.

• airbnb #32
• airtable
• climatescape #35
• codesandbox #34
• cypress #36
• discord
• espn #37
• firebugs #42
• framer
• monaco #45
• notion #46
• nytimes #47
• reactcharts #49
• reactvirtualized #50
• realadvisor #51
• reddit #55
• replay #60
• storybook #52
• tablecheck #53
• unsplash #56
• vscode #58
• whatsmyuseragent #57
• yelp #30

Currently we store the list of playwright tests we use during automated testing in our backend repository, to avoid exposing the credentials used for the facebook test. I don't know how much that really matters but it does make it awkward to keep the list of tests in sync with the repository itself. It would be better if we had a manifest in the playwright-tests repository that includes all the other tests, in any case.

It would be nice to see a dashboard of the runlive examples:

1. show example runs
2. show passes/fails defined by reaching the end
3. replays are one click away

https://github.com/ryansolid/solid

This playground might be a good example to start with.

also the hn news site

https://www.loom.com/share/38f5ecbbbc5d4ca4846887aa20c90e2f

1. go to https://quilljs.com/playground/
2. click in text editor and type
3. select text and bold it

• framer
• discord
• airtable

working on checkly reminded me that it is fine to have an authentication step

https://next.keystonejs.com/guides/document-fields#how-to-use-document-fields

https://app.tryeraser.com/workspace/D52HdNerC78CdgEDsv3B

We should upload some metadata about the new recording to our database so we can view the tests after the fact.

Fields:

• recording id
• url
• title
• lastScreen
• recording title (could be [PASS|FAIL] ${title}
• userId (could be a standard user like [email protected]