reg-viz / reg-cli Goto Github PK
View Code? Open in Web Editor NEWπ· Visual regression test tool.
Home Page: https://reg-viz.github.io/reg-cli/
License: MIT License
π· Visual regression test tool.
Home Page: https://reg-viz.github.io/reg-cli/
License: MIT License
On registry https://registry.npmjs.org/
, the "latest" version (v6.24.1) of dependency babel-preset-es2016
has the following deprecation notice:
π Thanks for using Babel: we recommend using babel-preset-env now: please read babeljs.io/env to update!
Marking the latest version of an npm package as deprecated results in the entire package being considered deprecated, so contact the package author you think this is a mistake.
Affected package file(s): package.json
If you don't care about this, you can close this issue and not be warned about babel-preset-es2016
's deprecation again. If you would like to completely disable all future deprecation warnings then add the following to your config:
"suppressNotifications": ["deprecationWarningIssues"]
I'm trying to set up reg-cli
for Copay, an Ionic app we build for iOS, Android, and desktop (macOS, Windows, and Linux).
I'm having some trouble avoiding false positives from very slight rendering differences between Chrome screenshots taken on macOS and those taken on Linux.
Almost everything is the same, but some font and shadow antialiasing seems to be different, and it's causing false positives. There are also some platform differences causing problems (scroll bars, file upload buttons).
Here's the comparison report from our CI: https://1043-18141084-gh.circle-artifacts.com/0/home/circleci/copay/test/visual/report.html
There are a few real issues mixed in with the false positives here, but I wanted to open this issue since it may be possible to solve a lot of these with improvements to reg-cli.
I really don't know the best way to handle this issue. We're currently using the reg-cli settings: --enableAntialias --thresholdRate 0.005
.
Any recommendations for improving this? I'm afraid if I raise the thresholdRate
more, we'll start missing real issues.
It would also be helpful to understand what the diff is showing. What is yellow vs. red in the diff?
I'm also trying to figure out the best way to handle platform rendering differences like scrollbars:
Our options here:
A. Avoid any cross platform differences β e.g. render our own custom scrollbar on both platforms
- we would prefer to use the native scrollbar, so this is not really an option
B. Hide conflicting elements for e2e tests β we could inject CSS like * { ::-webkit-scrollbar { display: none; }
just to take screenshots
reg-cli
both a macOS and Linux version, so the test passes if either matches.What are your thoughts on option C
? If reg-cli
could accept one or more variant
s for each image, I think this might also offer a better solution for our Antialiasing issues
above.
For images with lots of antialiasing problems, we can provide a version for both macOS and Linux.
Maybe a new command line option like:
--variantRegex
the regex used to identify variants for each image. If omitted, (.*)-variant-[-0-9].
With the default setting, if my previous image list looked like:
test-image.png
test-image-variant-1.png
another-test.png
another-test-variant-1.png
another-test-variant-2.png
test-image-variant-1.png
would match the default variantRegex
, with a capturing group value of test-image
. And both another-test-variant-1.png
and another-test-variant-2.png
would both have a captured value of another-test
.
When testing a new test-image.png
, reg-cli
would first test against the existing test-image.png
.
If it passes we're done. If it fails, test against each variant for test-image
(removing the .png
to match the capturing groups).
If any variant passes, pass the test. If all variants fail, we use the closest variant for the diff. This allows us to specifically "pass" certain cross-platform differences, without missing real, unintentional changes.
Any thoughts on this idea?
This is a similar issue to #124. In a project like:
https://github.com/bokuweb/reg-example
If instead of test/approved/sample.png
, I have subdirectories inside of approved
, like:
test/approved/app1/sample.png
test/approved/app2/sample.png
When I run the update operation, I'll get the error:
Error: EPERM: operation not permitted, unlink 'test/visual/approved/app1'
It appears that cleanupExpectedDir
expects a flat folder. Everything else seems to work fine with subdirectories.
Hi,
I'm using this tool standalone in my visual regression set up. The report it generates is quite nice.
I was looking to improve the report by adding more image comparison tools. For an example see github's image diff in commits: cameronmcefee/Image-Diff-View-Modes@8e95f70?short_path=eda75e0#diff-eda75e03d7033f3550356ddb8363ed56
Would you be interested in having something like that? I could try implementing something like that myself and put up a pull request. I was just wondering if you already have plans for doing something like that.
For some reason (sometimes) when i run the visual tests (using storycap for capturing the images) reg-cli logs a certain diff (which is a valid change) but when i run the update script, it logs that 2 files were updated. The issue here is that the 2nd file was not even listed in the report.
When i push the changes to GitLab i can see a diff, it seems like the file size is the only thing that is different (i'm not sure why though).
It doesn't happen all the time so its hard to understand what is causing it, maybe storycap has a bug or a config i am missing that makes it generate a different image size.
Note that i am running the tests in a docker, so i doubt its an environment issue.
This is how i run it:
yarn reg-cli somePath somePath somePath -R somePath/report.html --extendedErrors --matchingThreshold 0.1
image-diff is no longer maintained, I ended up rewriting it with the latest node standard.
image-difference should run x4 faster based on the benchmark.
There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.
Error type: undefined. Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.
This is the process how we use reg-cli:
--ignoreDeleted
which will not update non existing images, but only changed ones from actual to expected dirAlternative is to manual git revert all images which are not related to the test he was running
Hi,
it would be helpful to create git tags for each new release. The latest git tag is from early 2018.
Best
I simply ran this command
$ reg-cli actual/ expected/ diff -R index.html
and found some links are broken.
The directly structure I'm seeing is like this.
Another attempt is like this. I added /
at the end of diff
$ reg-cli actual/ expected/ diff/ -R index.html
and got this.
And results in this directly structure.
When running reg-cli
with a bunch of rendered images I get an UnhandledRejection
error and the application freezes. I realized that empty pngs trigger this, which I was able to solve by first deleting those files. But it also seems that corrupt files also trigger the error. I'm not sure which file is triggering the error.
Steps to reproduce the behavior:
Invalid image files should not crash or hand the process.
(node:49896) UnhandledPromiseRejectionWarning: Error: Unexpected end of input
at module.exports.ChunkStream._end (/usr/local/lib/node_modules/reg-cli/node_modules/pngjs/lib/chunkstream.js:92:24)
at module.exports.ChunkStream.end (/usr/local/lib/node_modules/reg-cli/node_modules/pngjs/lib/chunkstream.js:81:10)
at exports.PNG.PNG.end (/usr/local/lib/node_modules/reg-cli/node_modules/pngjs/lib/png.js:100:16)
at ReadStream.onend (_stream_readable.js:677:10)
at Object.onceWrapper (events.js:421:28)
at ReadStream.emit (events.js:327:22)
at endReadableNT (_stream_readable.js:1224:12)
at processTicksAndRejections (internal/process/task_queues.js:84:21)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:49896) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:49896) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
A vulnerability has been reported in trim-newlines in the following: GHSA-7p7h-4mm5-852v
So, it looks like reg-cli is increasing the version of trim-newlines in yarn resolutions:
Line 51 in c6f6aa5
However, the latest version of meow, which reg-cli depend on, has resolved the version of trim-newlines: https://github.com/sindresorhus/meow/blob/fd0bc62ce47781e11da506b8e38e8668eb78a584/package.json#L84
Therefore, I thought that we could eliminate the resolutions by updating the version of meow.
If you need to check the behavior, etc., I will be glad to assist you.
thanks.
Search input does not filter the results
Steps to reproduce the behavior:
I'm expecting one or more images to disappear when filtering.
Filtering shows all screenshots regardless of the name (might be related to this reg-viz/reg-cli-report-ui#28) ?
π§
The comparison report doesn't work correctly since 0.17.2. The Viewer won't open after clicking a card when the image files are placed in a sub-directory.
Steps to reproduce the behavior:
https://github.com/ysgk/reg-suit-ui-repro
git clone [email protected]:ysgk/reg-suit-ui-repro.git
cd reg-suit-ui-repro
yarn
yarn reg-suit compare
npx http-server .reg
You can click to open the first item but the second won't open.
Regardless of the hierarchy the images are placed, the viewer ui can be shown.
Nothing happens after clicking a card if the image is placed in a sub-directory.
After examining the code, I found that this line was the problem.
https://github.com/reg-viz/reg-cli-report-ui/blob/465b91363112eb4f7d26f70328596809763b993e/src/containers/viewer/ViewerContainer.ts#L117
// rawId: new-sub/sample.png
// id: new-sub%2Fsample.png
const id = encodeURIComponent(rawId);
const index = allItems.findIndex((entity) => entity.id === id);
If the id has a slash, the findIndex
always returns -1 because id
is URI-encoded while entity.id
is not. Actually, entity.id
is encoded here but the slashes are left unencoded.
I didn't understand the intent of the encoding so can you take a look at that?
When i am comparing the images of two directiories using
reg-cli /path/to/actual-dir /path/to/expected-dir /path/to/diff-dir -R ./report.html
i have given the actual directory, expected directory and diff directory path.
Getting error like reg-cli command not found.
When I try to generate an HTML report I get an error that style.css
wanβt found. This worked on 0.18.1 but broke on 0.18.2
reg-cli a b c -R r
Then I see the error
Error: ENOENT: no such file or directory, open '...node_modules/.pnpm/[email protected]/node_modules/reg-cli/report/ui/dist/style.css'
Branch | Build failing π¨ |
---|---|
Dependency | vue-template-compiler |
Current Version | 2.3.3 |
Type | devDependency |
This version is covered by your current version range and after updating it in your project the build failed.
As vue-template-compiler is βonlyβ a devDependency of this project it might not break production or downstream projects, but βonlyβ your build or test tools β preventing new deploys or publishes.
I recommend you give this issue a high priority. Iβm sure you can resolve this πͺ
data.pendingInsert
There is a collection of frequently asked questions and of course you may always ask my humans.
Your Greenkeeper Bot π΄
Thanks for the excellent tool!
The CLI seems to expect activity to happen in the current working directory. This npm script causes several errors:
{
"visual:approve": "reg-cli test/latest test/approved test/diffs --report test/report.html --json test/report.json"
}
But this one works perfectly:
{
"visual:approve": "cd test && reg-cli latest approved diffs --report report.html --json report.json"
}
A sampling of the errors:
Error: ENOENT: no such file or directory, unlink './approved/[IMG]'
Error: ENOENT: no such file or directory, open './latest/[IMG]
It's also possible I'm just misunderstanding the usage.
Naive difference detection engines (e.g. pixel-matching) sometimes don't work well.
For instance, someone committed with message, "modify header component size". He or she changed only header component and never touched any other components. However the naive difference detector unfortunately misunderstands "the entire page changed!" because it can't recognize the simple Y-dimension translation.
Using local features matching based region recognition. See https://github.com/reg-viz/improve_reg_diff
regconfig.json
Right now, reg-cli highlights any different pixels in 2 images, regardless of how similar they look. It would be good if there was an option to ignore differences if they are similar colors.
For example the following 2 colors are almost indistinguishable to the human eye, but reg-cli will flag them the same as if they were 2 completely opposite colors:
Sometimes it can be useful to highlight these small differences, but other times it would be good to have a tolereance for similar colors. To achieve this, we can use pixelmatch's threshold option. Right now reg-cli always sets it to 0, but it would be good to change it via a command line argument.
Currently when tests are failing for any reason we get this message in terminal:
Inspect your code changes, re-run with
-U
to update them
The issue with this message is that in some cases running the reg-cli command is used with an npm/yarn script.
For example - yarn test:visual
.
So developers that are using the repo think they need to run yarn test:visual -U
but this will just run the test again, what they really need to run is another dedicated script for updating e.g: yarn test:visual:update
.
It would be great to pass reg-cli a fail message (we can also pass success message if you think its needed) that will be used in case of a fail. The default message can be the message we have today.
Up until releasing new version of ReportUI I could simply open generated report via file
protocol (e.g. file:///path/to/report/report.html
). Yesterday I upgraded from 0.18.1 to latest version 0.18.6 and when I tried to open the report in my browser I got only white page and an error message in console Uncaught DOMException: Failed to construct 'Worker': Script at 'file:///.../.storybook/image-snapshots/worker.js' cannot be accessed from origin 'null'.
When I use http server to serve the report everything goes fine, but we are storing the report as an artifact in our CI and I cannot run server there to see the results.
Steps to reproduce the behavior:
report.html
and Copy pathReport can be opened via file
protocol as before
Report can be opened only if it's served with http server.
There is an import of mkdirp
in report.js, but mkdirp
is not included in package.json.
The result is that reg-cli
doesn't work for my application, as I use mkdirp
in a newer major version (v3) there. As the dependency is missing in reg-cli
's depedency list, NPM doesn't install a local version for reg-cli
but uses the one installed for my application (v3). v3 and v2 are not compatible.
The solution would add mkdirp
to the dependency list.
Hi, is it possible to test various mobile devices?
If so, how can I do that?
Thank you!
A clear and concise description of what the bug is.
Steps to reproduce the behavior:
should install the module
seeing message 'found 1 high severity vulnerability'
If applicable, add screenshots to help explain your problem.
I met the following error:
TypeError: Cannot read property 'passed' of undefined
at /Users/yosuke/workspaces/javascript/reg-simple-demo/node_modules/reg-cli/dist/index.js:157:15
at Array.filter (native)
at /Users/yosuke/workspaces/javascript/reg-simple-demo/node_modules/reg-cli/dist/index.js:156:26
at process._tickDomainCallback (internal/process/next_tick.js:135:7)
at Module.runMain (module.js:606:11)
at run (bootstrap_node.js:393:7)
at startup (bootstrap_node.js:150:9)
at bootstrap_node.js:508:3
I think it's thrown when newItems
is not empty.
reg-cli currently only returns a non-zero exit code only if an image has changed. For usage in CI it would be helpful if a non-zero exit code is also returned also if an image is delete or appended.
use case
Expected folder is part of git repo. CI pipeline generates actual folder. CI pipeline should not only ensure that generated images are matching expected ones but also that there aren't any additional test cases not covered (missing expected image) and no expected test cases are skipped (missing actual image).
This is currently only supported by dirty hacks. One approach would be passing output of reg-cli. Another one is comparing the actual and expected folders using custom bash script. An example for the later one would look like:
reg-cli actual expected diff
RESULT=$?
if [ ${RESULT} -eq 0 ] &&
[ `comm -3 <(ls actual|sort) <(ls expected|sort) | wc -l` -ne 0 ];
then
RESULT=1
fi
Would be great to not require that complexity.
On registry https://registry.npmjs.org/
, the "latest" version (v6.24.1) of dependency babel-preset-es2017
has the following deprecation notice:
π Thanks for using Babel: we recommend using babel-preset-env now: please read babeljs.io/env to update!
Marking the latest version of an npm package as deprecated results in the entire package being considered deprecated, so contact the package author you think this is a mistake.
Affected package file(s): package.json
If you don't care about this, you can close this issue and not be warned about babel-preset-es2017
's deprecation again. If you would like to completely disable all future deprecation warnings then add the following to your config:
"suppressNotifications": ["deprecationWarningIssues"]
The demo site (https://reg-viz.github.io/reg-cli/) provides old version's output.
It's no longer valid UI demo and new version would be more attractive.
Update with latest version's output.
Tooling that produces the images shouldn't be producing invalid png images, but, it seems, pngjs is hypersensitive in some regard. When this happens, details about the error are displayed (stack trace & error message), but, the reg-cli tool hangs instead of completing.
/node_modules/pngjs/lib/parser.js:48
this.error(new Error("Invalid file signature"));
^
Error: Invalid file signature
at Parser._parseSignature (/Users/loginname/gitlab/tool/node_modules/pngjs/lib/parser.js:48:18)
at ChunkStream._processRead (/Users/loginname/gitlab/tool/node_modules/pngjs/lib/chunkstream.js:160:13)
at ChunkStream._process (/Users/loginname/gitlab/tool/node_modules/pngjs/lib/chunkstream.js:175:14)
at ChunkStream.write (/Users/loginname/gitlab/tool/node_modules/pngjs/lib/chunkstream.js:57:8)
at PNG.write (/Users/loginname/gitlab/tool/node_modules/pngjs/lib/png.js:95:16)
at ReadStream.ondata (node:internal/streams/readable:809:22)
at ReadStream.emit (node:events:517:28)
at addChunk (node:internal/streams/readable:368:12)
at readableAddChunk (node:internal/streams/readable:341:9)
at Readable.push (node:internal/streams/readable:278:10)
Node.js v18.20.1
Steps to reproduce the behavior:
reg-cli
tool, specifying the new image folder path, the old image folder path, a path for the temp folder, --json {path to report.json} --report {path to report.html} --ignoreChange --matchingThreshold=0.05The command should finish up with a non-zero return code and a message indicating the cause of the problem.
The tool never completes. Output does show the error being encountered, but app hangs instead of finishes.
......
threshold
stands for 2-points pixels color distance ratio. (It's derived from pixelmatch lib.)
threshold
stands for ratio between differential and matched pixels.
Imagemagick uses threthold
term in this meaning.
This value can be thresholded using a -fuzz setting to only count pixels that have a larger then the threshold.
Use i.e. child_process.spawn
to improve performance.
Some similar tools like Chromatic, Percy or Happo support approving/declining particular snapshot changes and integrate with GitHub status checks.
Generated report should have button to approve/decline all or some snapshots that should post GitHub status check update. This web hook url can be something that developer should pass to the tool manually as a parameter and these buttons could simply call these urls to publish update. When all snapshots are approved status is updated with success.
It would be nice also to be able to filter by approved/declined PRs but that would require some sort of persistence for the state of snapshot.
Right now users would have to request changes on PRs and give a feedback manually.
0.10.1
When turning -X client
, the following error is thrown:
Error: ENOENT: no such file or directory, open '/Users/yosuke/workspaces/javascript/reg-suit/node_modules/reg-cli/report/assets/cv-wasm_browser.js'
at Object.fs.openSync (fs.js:651:18)
at Object.fs.readFileSync (fs.js:553:33)
at createXimdiffWorker (/Users/yosuke/workspaces/javascript/reg-suit/node_modules/reg-cli/dist/report.js:92:35)
at exports.default (/Users/yosuke/workspaces/javascript/reg-suit/node_modules/reg-cli/dist/report.js:104:22)
at /Users/yosuke/workspaces/javascript/reg-suit/node_modules/reg-cli/dist/index.js:188:33
at tryCatcher (/Users/yosuke/workspaces/javascript/reg-suit/node_modules/bluebird/js/release/util.js:16:23)
at Promise._settlePromiseFromHandler (/Users/yosuke/workspaces/javascript/reg-suit/node_modules/bluebird/js/release/promise.js:512:31)
And the following is log of node_modules/reg-cli/report/assets
:
node_modules/reg-cli/report/assets
βββ favicon_failure.png
βββ favicon_success.png
βββ image.png
βββ reglogo.svg
0 directories, 4 files
It seems that cv-wasm_browser.js
and cv-wasm_browser.wasm
are missing.
On registry https://registry.npmjs.org/
, the "latest" version (v6.24.1) of dependency babel-preset-es2015
has the following deprecation notice:
π Thanks for using Babel: we recommend using babel-preset-env now: please read babeljs.io/env to update!
Marking the latest version of an npm package as deprecated results in the entire package being considered deprecated, so contact the package author you think this is a mistake.
Affected package file(s): package.json
If you don't care about this, you can close this issue and not be warned about babel-preset-es2015
's deprecation again. If you would like to completely disable all future deprecation warnings then add the following to your config:
"suppressNotifications": ["deprecationWarningIssues"]
Branch | Build failing π¨ |
---|---|
Dependency | vue |
Current Version | 2.3.3 |
Type | devDependency |
This version is covered by your current version range and after updating it in your project the build failed.
As vue is βonlyβ a devDependency of this project it might not break production or downstream projects, but βonlyβ your build or test tools β preventing new deploys or publishes.
I recommend you give this issue a high priority. Iβm sure you can resolve this πͺ
data.pendingInsert
The new version differs by 3 commits.
1f9416d
[release] 2.3.4
97b07be
[build] 2.3.4
33a25e7
prevent data.pendingInsert to keep reference to removed nodes (fix #5839)
See the full diff
There is a collection of frequently asked questions and of course you may always ask my humans.
Your Greenkeeper Bot π΄
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
Warning
These dependencies are deprecated:
Datasource | Name | Replacement PR? |
---|---|---|
npm | cryptiles |
|
npm | hoek |
These updates are currently rate-limited. Click on a checkbox below to force their creation now.
These updates have been manually edited so Renovate will no longer make changes. To discard all commits and start over, click on a checkbox.
These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
@babel/cli
, @babel/core
, @babel/plugin-transform-block-scoping
, @babel/plugin-transform-flow-strip-types
, @babel/preset-env
, @babel/preset-flow
).github/workflows/ci.yml
ubuntu 20.04
package.json
bluebird 3.7.2
chalk 4.1.2
cli-spinner 0.2.10
cross-spawn 7.0.3
del 6.1.1
glob 7.2.3
img-diff-js 0.5.4
jpeg-js ^0.4.4
lodash 4.17.21
make-dir 3.1.0
md5-file 4.0.0
meow ^9.0.0
mustache 4.2.0
x-img-diff-js 0.3.5
xmlbuilder2 ^3.1.1
yargs-parser ^21.0.1
@babel/cli ^7.23.4
@babel/core ^7.23.7
@babel/plugin-transform-block-scoping ^7.23.4
@babel/plugin-transform-flow-strip-types ^7.23.3
@babel/preset-env ^7.23.8
@babel/preset-flow ^7.23.3
@babel/preset-stage-2 ^7.8.3
ava 6.1.3
copyfiles 2.4.1
cross-env 7.0.3
finalhandler 1.2.0
flow-bin 0.77.0
flow-typed 3.9.0
puppeteer 13.7.0
rimraf 3.0.2
serve-static 1.15.0
node >=18
set-value 4.1.0
tar ^6.0.0
kind-of ^6.0.3
randomatic 3.1.1
deep-extend 0.6.0
cryptiles 4.1.3
lodash 4.17.21
yargs-parser ^21.0.0
minimist ^1.2.3
ajv ^8.0.0
json-schema ^0.4.0
hoek 6.1.3
ansi-regex ^5.0.1
glob-parent ^6.0.1
dot-prop ^6.0.1
node-forge ^1.0.0
serialize-javascript ^6.0.0
y18n ^5.0.0
braces ^3.0.0
trim-newlines ^3.0.1
debug ^4.3.4
.travis.yml
node 12
node 10
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.