Coder Social home page Coder Social logo

cypress-visual-regression / cypress-visual-regression Goto Github PK

View Code? Open in Web Editor NEW
614.0 614.0 78.0 7.48 MB

Module for adding visual regression testing to Cypress

License: MIT License

JavaScript 1.04% HTML 7.15% TypeScript 91.82%
cypress cypress-plugin image-diff visual-regression visual-regressions

cypress-visual-regression's Introduction

Cypress Visual Regression

npm

github actions

Module for adding visual regression testing to Cypress.

Installation

npm install cypress-visual-regression

Configuration

JavaScript

Configure the visual regression plugin and environment variables in your cypress.config.js file like:

const { defineConfig } = require('cypress')
const { configureVisualRegression } = require('cypress-visual-regression')

module.exports = defineConfig({
  e2e: {
    env: {
      visualRegressionType: 'regression'
    },
    screenshotsFolder: './cypress/snapshots/actual',
    setupNodeEvents(on, config) {
      configureVisualRegression(on)
    }
  }
})

Pay attention to the type option. Use 'base' to generate baseline images, and 'regression' to compare current screenshot to the base screenshot

In your support file cypress/support/e2e.js add the following:

const { addCompareSnapshotCommand } = require('cypress-visual-regression/dist/command')
addCompareSnapshotCommand()

TypeScript

If you're using TypeScript, use files with a .ts extension, as follows:

cypress.config.ts

import { defineConfig } from 'cypress'
import { configureVisualRegression } from 'cypress-visual-regression/dist/plugin'

export default defineConfig({
  e2e: {
    env: {
      visualRegressionType: 'regression'
    },
    screenshotsFolder: './cypress/snapshots/actual',
    setupNodeEvents(on, config) {
      configureVisualRegression(on)
    }
  }
})

cypress/support/e2e.ts

import { addCompareSnapshotCommand } from 'cypress-visual-regression/dist/command'
addCompareSnapshotCommand()

cypress/tsconfig.json

{
  "compilerOptions": {
    "esModuleInterop": true,
    "types": [
      "cypress",
      "cypress-visual-regression"
    ]
  }
}

For more info on how to use TypeScript with Cypress, please refer to this document.

Plugin options

All options can be configured within visualRegression namespace under env variable inside cypress.config.js file, like this:

e2e: {
  screenshotsFolder: './cypress/snapshots/actual',
  env: {
    visualRegressionType: 'regression',
    visualRegressionBaseDirectory: 'cypress/snapshot/base',
    visualRegressionDiffDirectory: 'cypress/snapshot/diff',
    visualRegressionGenerateDiff: 'always',
    visualRegressionFailSilently: true
  }
}
Variable Default Description
visualRegressionType / Either 'regression' or 'base'. Base will override any existing base images with new screenshots. Regression will compare the base to the current screenshot.
visualRegressionBaseDirectory 'cypress/snapshot/base' Path to the directory where the base snapshots will be stored.
visualRegressionDiffDirectory 'cypress/snapshot/diff' Path to the directory where the generated image differences will be stored.
visualRegressionGenerateDiff 'fail' Either 'fail', 'never' or 'always'. Determines if and when image differences are generated.
visualRegressionFailSilently false Used to decide if any error found in regression should be thrown or returned as part of the result.

You can also pass default cypress screenshot arguments to addCompareSnapshotCommand(), like this:

const { addCompareSnapshotCommand } = require('cypress-visual-regression/dist/command')
addCompareSnapshotCommand({
  capture: 'fullPage'
})

How To Use

> syntax

cy.compareSnapshot(name)
cy.compareSnapshot(name, errorThreshold)
cy.compareSnapshot(name, options)

> arguments

Arguments Default Description
name / Represents the name of the base snapshot file that the actual screenshot will be compared with.
errorThreshold 0 Threshold under which any image difference will be considered as failed test. Represented in percentages.
options {} Used to provide additional cypress screenshot options as well as failSilently and errorThreshold values.

> examples

cy.compareSnapshot('homePage') // will compare actual screenshot to current and fail if there's any difference in the images

cy.get('h1').compareSnapshot('homePage', 0.2) // will compare only the image of h1 element and fail only if the percentage of pixels that are different is bigger than 0.2%

cy.compareSnapshot('homePage', {errorThreshold: 1, failSilently: true}).then(comparisonResults => {
  console.log(comparisonResults.mismatchedPixels) // will print the number of mismatched pixels
  console.log(comparisonResults.percentage) // will print the percentage of mismatched pixels
  console.log(comparisonResults.error) // will print the visual regression error message (if any)
})

Looking for more examples? See cypress/e2e/main.cy.ts.

Example

example

Tips & Tricks

Ignore some elements

Following function creates a command that allows you to hide elements of the page based on their className:

/**
 * To be called after you setup the command, in order to add a
 * hook that does stuff before the command is triggered
 */
export function beforeCompareSnapshots(
  /** Element you want to ignore */
  ignoredElementsQuerySelector: string,
  /** Main app element (if you want for the page to be loaded before triggering the command) */
  appContentQuerySelector: string = 'body'
) {
  Cypress.Commands.overwrite('compareSnapshots', (originalFn, ...args) => {
    return (
      cy
        // wait for content to be ready
        .get(appContentQuerySelector)
        // hide ignored elements
        .then(($app) => {
          return new Cypress.Promise((resolve, reject) => {
            setTimeout(() => {
              $app.find(ignoredElementsQuerySelector).css('visibility', 'hidden')
              resolve()
              // add a very small delay to wait for the elements to be there, but you should
              // make sure your test already handles this
            }, 300)
          })
        })
        .then(() => {
          return originalFn(...args)
        })
    )
  })
}

You may then use this function like:

const { addCompareSnapshotCommand } = require('cypress-visual-regression/dist/command')
const beforeCompareSnapshots = require('./commands/beforeCompareSnapshots')
addCompareSnapshotCommand({
  errorThreshold: 0.1
})
// add a before hook to compareSnapshot (this must be called AFTER compareSnapshotCommand() so the command can be overriden)
beforeCompareSnapshots(".chromatic-ignore,[data-chromatic='ignore']", '._app-content')

In this example, we ignore the elements that are also ignored by 3rd party tool Chromatic.

Debug

set process env visualRegressionLogger to true to enable logging. ie:

visualRegressionLogger=true cypress open --e2e -b chrome -C cypress.base.config.ts

cypress-visual-regression's People

Contributors

aontas avatar area73 avatar blv-rodrigoerades avatar bodeamariuscosmin avatar d0whc3r avatar dargmuesli avatar dependabot[bot] avatar devjvao avatar gabbersepp avatar greegus avatar gummiees avatar jbutko avatar jemerald avatar karlhorky avatar lgenzelis avatar littleironman avatar mariya-gfx avatar martinhrvn avatar mighdoll avatar mjhea0 avatar mrhyde avatar msakamaki avatar nelson-amorim avatar pizzapete avatar raimund-schluessler avatar rslemos avatar shibukawa avatar skjnldsv avatar srikanthkyatham avatar zaista avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cypress-visual-regression's Issues

"actual" snapshots don't get thrashed

Hi! First of all, thanks for this plugin... it's very useful and easy to implement
One issue I am finding is that after each test execution, files in the "actual" folder for a given test start accumulating (test-actual.png / test-actual (1).png /test-actual (2).png) and, apparently, the comparison is always done against the first, so if in latter executions there is an error (diff in the image) the test will pass anyway, because it compares the base against the FIRST actual file
I suppose that the "trashAssetsBeforeRuns" option set to true should cause the "actual" files and "diff" files to be deleted before a run... but that is not happening obviously

Am I missing something??

Thanks

Property 'compareSnapshotCommand' does not exist on type 'Chainable<undefined>'

Following is my plugins/index.js

const cucumber = require('cypress-cucumber-preprocessor').default;
const browserify = require('@cypress/browserify-preprocessor');
const getCompareSnapshotsPlugin = require('cypress-visual-regression/dist/plugin');


module.exports = (on, config) => {
    const options = browserify.defaultOptions;

    options.browserifyOptions.extensions.unshift('.ts');
    options.browserifyOptions.plugin.unshift(['tsify', { project: './cypress/tsconfig.json' }]);
    on('file:preprocessor', cucumber(options));
    getCompareSnapshotsPlugin(on);
}

support/commands.js

const compareSnapshotCommand = require('cypress-visual-regression/dist/command');
compareSnapshotCommand();

cypress.json

{
    "baseUrl": "http://localhost:4200",
    "testFiles": "**/*.feature",
    "video": false,
    "env": {
        "auth_username": "admin",
        "auth_password": "admin"
    },
    "screenshotsFolder": "cypress/snapshots/actual",
    "trashAssetsBeforeRuns": true
}

Following is my feature file for test -

Scenario: Screenshot Test
    Given Sample

This is what I have in test -

Given('Sample', () => {
    cy.compareSnapshot('home');
});

Error snapshot -

Upgrade to Cypress 4.02

Hello, we are attempting to upgrade our CI to Cypress 4.0.2 but are being blocked because we are using the Visual Testing plug in. There seems to be a hard cap on using version 3.X.X with this plug in, Can you confirm we are able to have the dependancy upgraded to version 4? We have confirmed our visual tests pass on version 4.0.2

Thanks

Compatibility with Typescript

Are there any types available for Typescript compatibility?
I get an error when using this in VSCODE

Property 'compareSnapshot' does not exist on type 'cy & EventEmitter'.

e.g.

it('Matches the snapshot', () => {
    cy.compareSnapshot('login', 0.0)
})

Use errorThreshold from global options

Hi! First of all, thank for this module. It is really helpful =)

So, I have seen that here it was added errorThreshold in global options.
But as far as I tested and debugged the code, it is only took in consideration if we pass an object as second argument.
I will try to explain better with code:

compareSnapshotCommand({
  capture: "fullPage",
  errorThreshold: 0.04,
});

cy.compareSnapshot("home", 0.03); // use 0.03 as errorThreshold -> ok
cy.compareSnapshot("home"); // won't use global options errorThreshold. Instead will use 0 -> is this intentional?
cy.compareSnapshot("home", {}); // will use the global options errorThreshold

If this is not intentional, I think there is an easy way to solve that.
Let me know if I can open a PR related to this possible issue.

And, again, thanks for the module =)

When run from cypress-open, receive file names like <snapshot>-actual (1).png

First of all - great tool, much appreciated!
Have only tried it recently and usage from cypress run seems to be working ok and all is clear. However when I run cypress open as usual for TDD, I started to see that each run of test produces a new image with number at the end of the name ( apparently to not override existing one ). Is this somehow could be avoided? Maybe there is config option? This was not happening before adding the plugin.
It does not seem to be a big issue, but I do not want disk drive of our team to be stuck with lots of images..

I am using Ubuntu 20.04 and Cypress 7.5.0

Thanks for the help!

Comparisons are failing due scrolling capture being inconsistent

I'm looking for either a way to turn off scroll so compare snapshot only compares what is currently visible in the viewport, or for some advice on getting the vertical scrolling and stitching together of snapshots to be more consistent.

I had previously had an issue with the resolutions being different between base and actual image with no clear reason, and found a solution in your issue where someone mentioned setting {scale: false}. I couldn't find information on this anywhere so I'm also wondering if there are any docs that I'm missing?

UPDATE:

scale: false didn't actually work correctly so I'm looking for a way to get images to have the same resolution each run.

I'm going to attach some screenshots below where there is no difference between runs but the image resolutions are vastly different. It's very unclear why this is happening

Enhancement: "cypress open" should allow overriding baseline screenshot automatically

When I open the cypress, I am making up my tests, modifying the app to be more testable, etc... But the screenshot comparison fail "each time" (hum, yes, a bit of exaggeration here) a make a modification. So actually, I do an "rm -fr .../baseline" before each test, but that's not very top.

I would like to have a mode where Cypress-visual-regression would automatically erase screenshots. I anyway do the comparison when comiting the images in git, so I even don't look at the result at that time.

I had a look at "--env type=base", but it does not seems to work in "cypress open" mode

utils-browser.js is missing in /dist folder in release 1.5.10

The latest version is missing the utils-browser file in the dist folder from the npm package. This leads to the following error when trying to execute any Cypress tests:

Error: Webpack Compilation Error
./node_modules/cypress-visual-regression/dist/command.js
Module not found: Error: Can't resolve './utils-browser' in 'node_modules/cypress-visual-regression/dist'

image

I guess sth. went wrong while building / bundling the release. Can you please publish a new version with the file included? Thanks!

Is there a way to change the threshold?

I have an error message which color it is #d73a0f, for testing purposes I changed that to plain "purple", however Cypress is considering those colors the same giving me a false positive:

image

Is there a way to improve that?

Error: Cannot find module './utils'

Following the setup in the readme I get the following error:

The plugins file is missing or invalid.

Your `pluginsFile` is set to `/Users/_/workspace/cypress/cypress/plugins/index.js`, but either the file is missing, it contains a syntax error, or threw an error when required. The `pluginsFile` must be a `.js`, `.ts`, or `.coffee` file.

Or you might have renamed the extension of your `pluginsFile`. If that's the case, restart the test runner.

Please fix this, or set `pluginsFile` to `false` if a plugins file is not necessary for your project.

 Error: Cannot find module './utils'

Downgrading vom 1.5.3 to 1.5.2 fixes it

Should fail when images are of different sizes

The test should fail if base and actual images have different dimensions.

Currently, cypress-visual-regression just logs a message:

Error: Image sizes do not match.
    at pixelmatch (/home/.../node_modules/pixelmatch/index.js:20:15)
    at compareSnapshotsPlugin (/home/.../node_modules/cypress-visual-regression/dist/plugin.js:70:24)
    ✓ 375x667 (10081ms)

The test pass though.
At the very least the test should fail.
At best, the smallest image should be extended (transparent canvas?) before comparison, so we can inspect where elements have gone astray.

Small differences in scolling cause the tests to fail

Hi,

I'm running the tests on a website that's quite long so even in a viewport of 1920x1080 needs to be scrolled down quite a bit. The problem is, the scrolling speed or distance seems to differ ever so slightly between the tests, causing them to fail.
Here's an example. The page is https://roomkey.co/creating-a-room/

Excuse the large images.

This is the base image:
create-a-room-page-base
This is a screenshot taken in the test:
create-a-room-page-actual
And finally, the diff:
create-a-room-page-diff
As you can see, there are small differences in how far the scroll has moved and that breaks the test. It's intermittent, too.

Any ideas on how to fix that?

Error [object Object] on node_modules/cypress-visual-regression/dist/command.js:33

Hi, I'm having an error, but the logs are not helpful.
Snapshots files are in the base folder, diffs are not generated.

Cypress version 5.0.0.
Node version 10.x

Capture d’écran_2020-08-25_17-52-27

Error: [object Object]
    at Context.eval (http://localhost:36223/__cypress/tests?p=cypress/support/index.js:507:17)
From previous event:
    at Context.thenFn (http://localhost:36223/__cypress/runner/cypress_runner.js:154868:26)
    at Context.then (http://localhost:36223/__cypress/runner/cypress_runner.js:155308:21)
    at Context.<anonymous> (http://localhost:36223/__cypress/runner/cypress_runner.js:168808:21)
    at http://localhost:36223/__cypress/runner/cypress_runner.js:168241:15
From previous event:
    at runCommand (http://localhost:36223/__cypress/runner/cypress_runner.js:168220:8)
    at next (http://localhost:36223/__cypress/runner/cypress_runner.js:168366:14)
    at http://localhost:36223/__cypress/runner/cypress_runner.js:168394:16
From previous event:
    at next (http://localhost:36223/__cypress/runner/cypress_runner.js:168366:34)
From previous event:
    at http://localhost:36223/__cypress/runner/cypress_runner.js:168407:37
From previous event:
    at run (http://localhost:36223/__cypress/runner/cypress_runner.js:168400:19)
    at $Cy.cy.<computed> [as window] (http://localhost:36223/__cypress/runner/cypress_runner.js:168852:11)
    at Context.runnable.fn (http://localhost:36223/__cypress/runner/cypress_runner.js:169078:21)
    at callFn (http://localhost:36223/__cypress/runner/cypress_runner.js:103957:21)
    at Test.../driver/node_modules/mocha/lib/runnable.js.Runnable.run (http://localhost:36223/__cypress/runner/cypress_runner.js:103944:7)
    at http://localhost:36223/__cypress/runner/cypress_runner.js:174727:28
From previous event:
    at Object.onRunnableRun (http://localhost:36223/__cypress/runner/cypress_runner.js:174715:20)
    at $Cypress.action (http://localhost:36223/__cypress/runner/cypress_runner.js:165051:61)
    at Test.Runnable.run (http://localhost:36223/__cypress/runner/cypress_runner.js:173050:13)
    at Runner.../driver/node_modules/mocha/lib/runner.js.Runner.runTest (http://localhost:36223/__cypress/runner/cypress_runner.js:104616:10)
    at http://localhost:36223/__cypress/runner/cypress_runner.js:104742:12
    at next (http://localhost:36223/__cypress/runner/cypress_runner.js:104525:14)
    at http://localhost:36223/__cypress/runner/cypress_runner.js:104535:7
    at next (http://localhost:36223/__cypress/runner/cypress_runner.js:104437:14)
    at http://localhost:36223/__cypress/runner/cypress_runner.js:104503:5
    at timeslice (http://localhost:36223/__cypress/runner/cypress_runner.js:98429:27)
logError @ cypress_runner.js:199987
(anonymous) @ cypress_runner.js:199628
emit @ cypress_runner.js:51075
(anonymous) @ cypress_runner.js:183967
emit @ cypress_runner.js:51075
emit @ cypress_runner.js:184010
onPrint @ cypress_runner.js:182829
_onPrintClick @ cypress_runner.js:182834
(anonymous) @ cypress_runner.js:184221
executeAction @ cypress_runner.js:48940
n @ cypress_runner.js:48940
ca @ cypress_runner.js:59472
ja @ cypress_runner.js:59473
ka @ cypress_runner.js:59473
wa @ cypress_runner.js:59475
Aa @ cypress_runner.js:59476
ya @ cypress_runner.js:59476
Da @ cypress_runner.js:59479
Ad @ cypress_runner.js:59542
Gi @ cypress_runner.js:59708
Kb @ cypress_runner.js:59497
Dd @ cypress_runner.js:59544
(anonymous) @ cypress_runner.js:59709
../../node_modules/scheduler/cjs/scheduler.production.min.js.exports.unstable_runWithPriority @ cypress_runner.js:63856
Ii @ cypress_runner.js:59709
Cd @ cypress_runner.js:59543

cypress.json:

{
  "projectId": "3paxvy",
  "viewportWidth": 1920, 
  "viewportHeight": 1080,
  "defaultCommandTimeout": 6000,
  "experimentalComponentTesting": true,
  "componentFolder": "tests/visual",
  "nodeVersion": "system",
  "env": {
    "failSilently": false,
    "type": "actual"
  },
  "screenshotsFolder": "cypress/snapshots/actual",
  "trashAssetsBeforeRuns": true
}

Weird actual images from Jenkins CI?

When using this in our Jenkins CI pipeline, the actual images that are stored as artifacts are turning out weird. Any ideas if this is due to this plugin or due to Jenkins?

In this example, you can see a duplicated bottom section ("Campaigns").

image

Screenshot duplicates on a every test run

I create a simple test and run him several times

Test code

it('Load page corrctly', () => {
    cy.server();
    cy.route(/models\/\S+/).as('fetchMockData');

    cy.visit('/');
    cy.wait('@fetchMockData').its('status').should('eq', 200);

    cy.getByTestId('canvas').compareSnapshot('/compare/canvas');
});

All it's ok, but I saw in a project tree some screensots for this test
image

When I try to call cypress run I got this message

  (Screenshots)

  -  /tests/e2e/screenshots/open-page.spe      (689x688)
     c.js/compare/canvas-actual.png  

and all files expect canvas-actual.png was deleted.

Cypress uses canvas-actual.png but a last generated file is canvas-actual (2).png.

Could you please explain how it works?

Possible bug

Found by @ kane77:

While doing the tests I noticed that in plugin.js the percentage is initialized to 0. In case the screenshot sizes are different it throws error and returns percentage 0 so it will pass. is that expected behavior?

#28 (comment)

CypressError: cy.task('compareSnapshotsPlugin') failed with the following error:

Running ./node_modules/.bin/cypress run --env type=base --config screenshotsFolder=cypress/snapshots/base is fine but...

I am getting this error after running ./node_modules/.bin/cypress run --env type=actual

`
CypressError: cy.task('compareSnapshotsPlugin') failed with the following error:

The 'task' event has not been registered in the plugins file. You must register it before using cy.task()

Fix this in your plugins file here:
cypress/plugins/index.js

https://on.cypress.io/api/task
`

cypress/support/commands.js

const compareSnapshotCommand = require('cypress-visual-regression/dist/command'); compareSnapshotCommand();

cypress/plugins/index.js

const getCompareSnapshotsPlugin = require('cypress-visual-regression/dist/plugin'); module.exports = on => { getCompareSnapshotsPlugin(on); };

/.d.ts

declare module 'cypress-visual-regression/dist/plugin';

/cypress.json

{ "screenshotsFolder": "cypress/snapshots/actual", "trashAssetsBeforeRuns": true }

base image can't be store under specified path

When integrated cypress-visual-regression with cypress, and run command:
cypress run type=base --config screenshotsFolder=cypress/results/snapshots/base
the base image can't be placed under path 'cypress/results/snapshots/base'.
Can anybody give some suggestion?

pulgin crashes due to not able to find the actual.png file

the file location is /screenshots/actual.png, but the it's looking under /snapshot/actual/<spc.js>/actual.png.

Not sure what am i doing wrong. I added file path in the cypress.json.
"env": {
"screenshotsFolder":"./cypress/snapshots/actual",
"trashAssetsBeforeRuns": true
}

I'm on ios:

compareSnapshot is not a function

I cannot get the snapshot command to be available in my cypress.

I ran npm install cypress-visual-regression
Here's my setup (from this plugins docs):

// cypress/plugins/index.js

const getCompareSnapshotsPlugin = require('cypress-visual-regression/dist/plugin');
module.exports = (on) => {
  getCompareSnapshotsPlugin(on);
};

// cypress/support/commands.js

const compareSnapshotCommand = require('cypress-visual-regression/dist/command');
compareSnapshotCommand();

Result
Screenshot 2020-09-04 at 21 50 32

// cypress configuration

{
animationDistanceThreshold:5
fileServerFolder:""
baseUrl:null
fixturesFolder:"cypress/fixtures"
blockHosts:null
chromeWebSecurity:true
modifyObstructiveCode:true
integrationFolder:"cypress/integration"
env:failSilently
pluginsFile:"cypress/plugins"
hosts:null
screenshotsFolder:"./cypress/snapshots/actual"
numTestsKeptInMemory:50
supportFile:"cypress/support"
port:null
projectId:null
videosFolder:"cypress/videos"
reporter:"spec"
reporterOptions:null
ignoreTestFiles:"*.hot-update.js"
testFiles:"**/*.*"
defaultCommandTimeout:4000
trashAssetsBeforeRuns:true
execTimeout:60000
userAgent:null
pageLoadTimeout:60000
viewportWidth:1000
requestTimeout:5000
viewportHeight:660
responseTimeout:30000
video:true
taskTimeout:60000
videoCompression:32
videoUploadOnPasses:true
screenshotOnRunFailure:true
watchForFileChanges:true
waitForAnimations:true
nodeVersion:"default"
firefoxGcInterval:runMode, openMode
retries:runMode, openMode
componentFolder:"cypress/component"
browsers:Chrome, Firefox, Electron
experimentalSourceRewriting:false
experimentalComponentTesting:false
experimentalShadowDomSupport:false
experimentalFetchPolyfill:false
experimentalNetworkStubbing:false
}

I can't find any help on google about this.

My test always pass

My test always pass.

describe.only('Visit Home - snapshot testing', () => {
   before(() => {
      cy.viewport(2000, 1500)
      cy.visit(url)
   })

   it('Visits the home page', () => {
      cy.viewport(2000, 1500)

      cy.contains('footer', 'Need Help') // adding delay to wait till this is visible

      cy.compareSnapshot('homePage', 0.0)

      cy.scrollTo('0%', '0%') // scroll back to top
      // see
      // https://github.com/cypress-io/cypress/issues/871
   })
})
//cypress.json
{
   "video": false,
   "screenshotOnRunFailure": false,
   "screenshotsFolder": "./cypress/snapshots/actual",
   "trashAssetsBeforeRuns": true,
   "env": {
      "failSilently": false
   }
}
//commands.js
const compareSnapshotCommand = require('cypress-visual-regression/dist/command')

compareSnapshotCommand()
//cypress/plugins/index.js
const getCompareSnapshotsPlugin = require('cypress-visual-regression/dist/plugin')

module.exports = (on) => {
   getCompareSnapshotsPlugin(on)
}

In the above example, the first run it takes snapshot as expected. Then i change something (added text) on the home page and re-run. The test re-runs and passes. When i look at the screenshots, homePage-actual.png is different than homePage-actual (1).png but it passed.

I also tried

cy.compareSnapshot('homePage', 0.0)
cy.compareSnapshot('homePage', 0.1)

I am running with

    "cypress": "./node_modules/.bin/cypress open",
    "cypress-check": "./node_modules/.bin/cypress open --env type=actual",

I want the test to fail if anything is different. What am i missing?

Running regression throwing errors

  1. Navigation screenshot home screen:
    CypressError: cy.task('compareSnapshotsPlugin') failed with the following error:

Error: ENOENT: no such file or directory, mkdir '/Users/me/Documents/app/cypress/snapshots/diff/visualregression/navigation.spec.js'
at Object.fs.mkdirSync (fs.js:885:18)
at Promise (/Users/me/Documents/app/node_modules/cypress-visual-regression/dist/plugin.js:25:10)

Actual snapshot folder not found

Since #49 the actual snapshot folder is by default not found anymore and visual regression tests fail, see nextcloud-libraries/nextcloud-vue#1412.

The actual images are searched in cypress/screenshots (see https://github.com/mjhea0/cypress-visual-regression/blob/master/src/plugin.js#L109-L112 and https://github.com/mjhea0/cypress-visual-regression/blob/master/src/plugin.js#L14) whereas they reside in cypress/snapshots/actual by default. The base and diff images are correctly found by their correct paths, as their paths are constructed correctly by https://github.com/mjhea0/cypress-visual-regression/blob/master/src/plugin.js#L18-L22.

Adding

SNAPSHOT_ACTUAL_DIRECTORY = args.baseDir || path.join(process.cwd(), 'cypress', 'snapshots', 'actual');

and using this path for the actual images solves the issue. However, I don't know whether there was an intention to use the screenshots directory rather than snapshots, so please check if this solution breaks something. I will send a PR.

[Enhancement] Add retryability to compareSnapshotCommand to reduce flakiness for unstable DOM

I am mostly using this plugin to compare a THREEJS canvas which has no useable state to rely cypress waits on.
It would be good to implement retryability (based on errorThreshold) with a timeout to reduce flaky failures.

I found cypress-recurse could help with that and also tried to fork this repo to implement but I am new and not too proficient with javascript. Could someone please help implement this idea?

Cant figure why it is taking two snaps of the same page?

Ever since i implemented this library, it always has been taking two pcitures for the base, I can't find any documentation as to why this is happening?

Can you please shed some light into this? Is this is how it is supposed to work??

As you can see there are two Login page snaps and two Forgot page snaps?

image

ssh connection error

running this
npm install cypress-visual-regression --save-dev
getting this:
Error while executing:
npm ERR! C:\Program Files\Git\cmd\git.EXE ls-remote -h -t ssh://[email protected]/cypress-io/request.git
npm ERR!
npm ERR! ssh: connect to host github.com port 22: Connection timed out
npm ERR! fatal: Could not read from remote repository.

I am able to install other things just fine. any ideas? on windows.

don't fail test on screenshot diffs

There should be an env variable to stop failing whole test case when there is screenshot diffs.
like "failOnSnapshotDiff":false
This will help to carry on with rest of the cases.

Provide TypeScript definitions

thanks for the great library.
this is a re-proposal of #43 .

I'm getting the idea that I'd like to have TypeScript support for this library, as it currently has no type definitions, making it difficult to develop in VS Code.

Is there any other work that needs to be done to accept this PR? (Sorry, I gave you the PR #87 first...)

Typescript error: Property 'compareSnapshot' does not exist on type 'cy & EventEmitter'

Sorry, I overlooked the build.

Because the command.d.ts is not copied to the dist folder, typescript gave me an error when I used 1.5.9.

The error was:

Error: Webpack Compilation Error
src/integration/snapshot.spec.ts:23:10
TS2339: Property 'compareSnapshot' does not exist on type 'cy & EventEmitter'.
    21 |        * https://github.com/mjhea0/cypress-visual-regression
    22 |        */
  > 23 |       cy.compareSnapshot(`snapshot-test ${story}`);
       |          ^^^^^^^^^^^^^^^
    24 |     });
    25 |   });
    26 | });

So I would like to add a command to copy the d.ts file to dist at build time.

スクリーンショット 2021-07-24 16 49 28

I've confirmed it works and will get a PR out soon!
Thanks for the typescript support!

Enhancement: hide Chromatic ignored elements (or more globally elements based on a selector)

Hi, small enhancement suggestion: I've setup cypress-visual-regression, so it also respects Chromatic ignored elements, for those who also use Storybook for visual regression testing at the unit level.

Here is the command I've created: it overrides compareSnapshot with a "before" step that removed selected elements from the page.

/**
 * To be called after you setup the command, in order to add a
 * hook that does stuff before the command is triggered
 */
function beforeCompareSnapshotCommand(
  /** Element you want to ignore */
  ignoredElementsQuerySelector: string,
  /** Main app element (if you want for the page to be loaded before triggering the command) */
  appContentQuerySelector: string = "body"
) {
  Cypress.Commands.overwrite("compareSnapshot", (originalFn, ...args) => {
    // wait for content to be ready
    // NOTE: be sure you wait enough so that all elements are correctly present
    return cy
      .get(appContentQuerySelector)
      .then($app => {
        return new Cypress.Promise((resolve, reject) => {
          setTimeout(() => {
            $app.find(ignoredElementsQuerySelector).css("visibility", "hidden");
            resolve();
            // add a very small delay to wait for the elements to be there, but you should
            // make sure your test already handles this
          }, 300);
        });
      })
      .then(() => {
        return originalFn(...args);
      });
  });
}

module.exports = beforeCompareSnapshotCommand;

Usage:

const compareSnapshotCommand = require("./commands/compareSnapshots"); //require("cypress-visual-regression/dist/command");
const beforeCompareSnapshotCommand = require("./commands/beforeCompareSnapshots"); //require("cypress-visual-regression/dist/command");
compareSnapshotCommand({
  errorThreshold: 0.1
});
// add a before hook to compareSnapshot (this must be called AFTER compareSnapshotCommand() so the command can be overriden)
beforeCompareSnapshotCommand(
  ".chromatic-ignore,[data-chromatic='ignore']",
  "._app-content"
);

Could this be added somewhere in the doc? It can be done in user-land, so no need to update the library.

Edit: small limitation, it expects some elements to be hidden, I am reworking it a bit. It has to add some delay to the screenshot, because we need to wait for the element to be hidden, I haven't yet found an optimal way. Updated the code so it works even if you have nothing to hide.

It would be a good practice to always wait for all relevant elements to be there before screenshotting, so we can safely remove the things that should be hidden.

override precedence rules

When you set a default threshold in compareSnapshotCommand can that be updated at the compareSnapshot level? Which takes precedence?

Requires testing, documentation, and possibly changes to the library.

Plugin crashed

While running test on cypress 4.4.1 with cypress-visual-regression ^1.2.0 I'm getting following error:

The following error was thrown by a plugin. We stopped running your tests because a plugin crashed. Please check your plugins file (/Users/***/Repos/***-cypress/cypress/plugins/index.js)

Error: ENOENT: no such file or directory, open '/Users//Repos/-cypress/cypress/snapshots/base/visual/-eng-main.js/-en-home-base.png'

Actuals have different aspect ratios to base images with no discernible reason why

I am having a lot of issues with this plugin but the most frustrating by far is this. If I run with type=base then type=actual, with no other difference between the commands this issue still occurs:

The aspect ratios of the images taken for base and actual runs are different and causing the actual runs to fail when they should not.

I'll upload two snapshots below taken on identical runs besides the type being actual or base:

Base image (2212x1566):
billing-base

Actual image (2732x1574):
billing-actual (1)

Please let me know what I'm supposed to do to avoid this happening. Is there somewhere I can specify the aspect ratio/pixels/size of the snapshot images?

Snapshots are different between different systems

Hi,
I tried to make a snapshot of a bigger part, about 300x200 pixels big. On both, the development system and CI systems (both Windows), the size differs. So I made a snapshot of a smaller part (about 44x36 pixels) and disabled scaling: compareSnapshotCommand({ scale: false });. Now the size is equal but the snapshot taken at our CI server shows some pixels (about 7%) that are not equal to the snapshot taken on my development machine.

an example:
snapshot diff

Do you know if this is normal and forces us to use always a higher threshold than 0.0?

Plugin crashes when comparing actual with non-existing base screenshot

Plugin crashes and stops the whole spec when trying to compare an actual screenshot with a non-existing base. This could happen when you forgot to generate base images before, but in my opinion, it should be handled as a regular error and the test should carry on to another block within the test file.

The following error was thrown by a plugin. We stopped running your ****s because a plugin crashed. Please check your plugins file

Error: ENOENT: no such file or directory, open '/root/my-project/cypress/snapshots/base/homepage.spec.js/homepage-base.png'

Can't get the compareSnapshot to ever fail

Hi folks,

I followed the steps in your readme, added a "failSilently": false to the "env" section of my cypress.json, even tried passing this parameter in compareSnapshotCommand, no luck. I just can't ever get the tests to fail. I even tested this by taking a base image of one page, then edited the test to navigate to a different page and still try to compare it, and it doesn't fail even if the page is completely different. My test looks like this:
it('Login page visual regression', () => { cy.visit('/login') cy.compareSnapshot('login-page') })

I even verified manually that the images in base folder are completely different to those in actual folder.

Any ideas? I'd appreciate your help on this.

Thanks,
Dariusz

compareSnapshotsPlugin doesn't work when setting screenshotsFolder with non-default path

compareSnapshotsPlugin fails without any error when screenshot folder is not the same as cypress default (/cypress/snapshots)
{
"screenshotsFolder": "./src/snapshots/actual", // nx suggested path
"env": { "type": "actual" }
}

Same error as if there's not actual baseline images to compare to "- then function(){}":
image

Note that diff folder is always cypress/snapshots/diff whatever is provided to screenshotsFolder variable

Make directories configurable

I don't like the way how the snapshotdirectories are used.

I want to have a folder cypress/screenshots and a folder cypress/snapshots/base. I introduced the possibility to configure it by "env" property in cypress.json

Question: how to avoid to trash/override existing snapshots ?

Do you have a recommendation or a way to avoid to trash/override existing snapshots when running npm run --env type=base ?

I would like to generate base snapshots of the new tests or new test cases only and to avoid specifying my testFiles to the npm run --env type = base command.

Thanks for the project ;)

Taking screenshot failed due to TypeError: Cannot read property 'apply' of undefined after previously failed run

When running a test with this plugin, after a failed run (any error, such as: assertion failure), the next run will always throw error when taking screenshot. Here is the detail error:
Command: screenshot cypress_runner.js:199793 Error: TypeError: Cannot read property 'apply' of undefined at https://www.marbodal.se/:3655:16 From previous event: at takeScreenshot (https://www.marbodal.se/__cypress/runner/cypress_runner.js:159366:13) at Context.screenshot (https://www.marbodal.se/__cypress/runner/cypress_runner.js:159458:14) at Context.<anonymous> (https://www.marbodal.se/__cypress/runner/cypress_runner.js:169999:21) at https://www.marbodal.se/__cypress/runner/cypress_runner.js:169423:15 From previous event: at runCommand (https://www.marbodal.se/__cypress/runner/cypress_runner.js:169402:8) at next (https://www.marbodal.se/__cypress/runner/cypress_runner.js:169548:14) at https://www.marbodal.se/__cypress/runner/cypress_runner.js:169576:16 From previous event: at next (https://www.marbodal.se/__cypress/runner/cypress_runner.js:169548:34) From previous event: at Promise.catch.err.name (https://www.marbodal.se/__cypress/runner/cypress_runner.js:169589:37) From previous event: at run (https://www.marbodal.se/__cypress/runner/cypress_runner.js:169582:21) at $Cy.cy.<computed> [as visit] (https://www.marbodal.se/__cypress/runner/cypress_runner.js:170039:11) at Context.runnable.fn (https://www.marbodal.se/__cypress/runner/cypress_runner.js:170263:21) at callFn (https://www.marbodal.se/__cypress/runner/cypress_runner.js:104396:21) at Test.../driver/node_modules/mocha/lib/runnable.js.Runnable.run (https://www.marbodal.se/__cypress/runner/cypress_runner.js:104383:7) at https://www.marbodal.se/__cypress/runner/cypress_runner.js:175798:28 From previous event: at Object.onRunnableRun (https://www.marbodal.se/__cypress/runner/cypress_runner.js:175786:17) at $Cypress.action (https://www.marbodal.se/__cypress/runner/cypress_runner.js:166429:28) at Test.Runnable.run (https://www.marbodal.se/__cypress/runner/cypress_runner.js:174172:13) at Runner.../driver/node_modules/mocha/lib/runner.js.Runner.runTest (https://www.marbodal.se/__cypress/runner/cypress_runner.js:105055:10) at https://www.marbodal.se/__cypress/runner/cypress_runner.js:105181:12 at next (https://www.marbodal.se/__cypress/runner/cypress_runner.js:104964:14) at https://www.marbodal.se/__cypress/runner/cypress_runner.js:104974:7 at next (https://www.marbodal.se/__cypress/runner/cypress_runner.js:104876:14) at Hook.<anonymous> (https://www.marbodal.se/__cypress/runner/cypress_runner.js:104937:7) at next (https://www.marbodal.se/__cypress/runner/cypress_runner.js:175719:22) at https://www.marbodal.se/__cypress/runner/cypress_runner.js:175741:11 From previous event: at onNext (https://www.marbodal.se/__cypress/runner/cypress_runner.js:175738:57) at done (https://www.marbodal.se/__cypress/runner/cypress_runner.js:104336:5) at https://www.marbodal.se/__cypress/runner/cypress_runner.js:104401:11 From previous event: at callFn (https://www.marbodal.se/__cypress/runner/cypress_runner.js:104399:14) at Hook.../driver/node_modules/mocha/lib/runnable.js.Runnable.run (https://www.marbodal.se/__cypress/runner/cypress_runner.js:104383:7) at https://www.marbodal.se/__cypress/runner/cypress_runner.js:175798:28 From previous event: at Object.onRunnableRun (https://www.marbodal.se/__cypress/runner/cypress_runner.js:175786:17) at $Cypress.action (https://www.marbodal.se/__cypress/runner/cypress_runner.js:166429:28) at Hook.Runnable.run (https://www.marbodal.se/__cypress/runner/cypress_runner.js:174172:13) at next (https://www.marbodal.se/__cypress/runner/cypress_runner.js:104898:10) at https://www.marbodal.se/__cypress/runner/cypress_runner.js:104942:5 at timeslice (https://www.marbodal.se/__cypress/runner/cypress_runner.js:98868:27)

Restart Cypress temporarily fix the issue until the next failed run.

Intermittent Error: [object Object] on compareSnapshot command

image
All test in this screenshot are using Cypress visual regression to compare snapshot. Some of them pass and some fail because of the object Object error.

Does someone know how to fix this? I don't know what more details are relevant to be share. But please let me know if anything else is required.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.