nut-tree / nut.js Goto Github PK

Version
1.1.2

Short overview
When a given input image is detected on a smaller scale, an unscaled target region is returned.

Detailed error description
When searching for a larger image on a screen, e.g. when using input images captured on retina displays on a regular display, the returned image region is unscaled.
This results in a larger result region, than the actual image on screen, which might cause clicks outside the target.

Steps to reproduce error
Use an input image e.g. twice the scale than on screen and try to locate it.

Short overview
Add expect(...) semantics known from other tools.

Use case
Convenience.

Detailed feature description
As a user I want to work with common semantics known from other frameworks.

Additional content
e.g. `expect(nut).toProvide(similarSemantics);

Short overview
Enable matrix builds on Travis

Use case
Verify platform independence by running multi-platform matrix builds.

Detailed feature description
nut.js is meant to be used on multiple platforms. To ensure stable builds already during development, matrix builds offered by Travis should be utilized.

Additional content

Please provide any (mandatory) additional data for your desired feature

Short overview
Current installation process takes way to long

Use case
Faster installation / time to first test

Detailed description
Currently, nut.js builds OpenCV from source by default.
This installation process takes quite a while and requires external tooling (cmake).
As a user I don't want to compile a OpenCV via cmake when installing nut.js

Additional content

Please provide any (mandatory) additional data for your enhancement

Short overview
As a user I want to be able to generate screenshots of my current screen.

Use case
During a test case I want to be able to snap screenshots of the current screen, not only for error reports, but also for reporting.

Detailed feature description
It should be possible to select the screenshot format, file name and path as well as optional pre- and postfixes to the filename.

Additional content

Please provide any (mandatory) additional data for your desired feature

Short summary
npm run test fails after I install.. I can make the mouse move, but no keyboard control or anything else. I'm really new to this so I must be missing something really basic...
Detailed question
I'm on mac Mojave,
steps I did:
npm i @nut-tree/nut-js
npm run audit
npm run postinstall
npm test:
error: /trailmix-master/node_modules/execa/index.js:174:9)
error: trailmix-master/node_modules/execa/index.js:278:16

I hppe this is enough details, I would greatly appreciate the help!!!

Version
1.1.2

Short overview
nut.js does not properly perform drag & drop gestures on macOS and Windows

Detailed error description
While the mouse correctly moves from source to target, it does not keep the button state and therefore performs just a move, not a drag.

Steps to reproduce error

const {mouse, screen, straightTo, centerOf, right} = require("@nut-tree/nut-js");

(async () => {
  screen.config.resourceDirectory = "$resourceFolder";
  await mouse.move(straightTo(centerOf(screen.find("$image"))));
  await mouse.drag(right(1000));
})();

Short overview
The current CI build could be improved in terms of runtime.

Use case
Faster responses from CI

Detailed feature description
Currently OpenCV will be rebuilt on every CI run. This could be optimized by running builds inside a container with pre-installed OpenCV so no manual installation on the build system is required.

Additional content

Please provide any (mandatory) additional data for your desired feature

Version
1.0.0

Short overview
Selecting invalid regions of an image (e.g. bigger than the actual image) causes uncaught runtime errors

Detailed error description
When providing a ROI which exceeds an images size, nut.js does no checks to prevent invalid memory access.
This leads to uncaught std::runtime_errors

Steps to reproduce error

describe("Invalid ROI test", () => {
  it("should trigger a runtime error", async () => {
    // GIVEN
    const SUT = new TemplateMatchingFinder();
    const imagePath = path.resolve(__dirname, "./__mocks__/save_as.png");
    const minConfidence = 0.99;
    const vision = new ScreenAction();
    const haystack = await vision.grabScreen();
    const matchRequest = new MatchRequest(
      haystack,
      imagePath,
      new Region(0, 0, haystack.width, haystack.height),
      minConfidence
    );

    // WHEN
    await SUT.findMatch(matchRequest, true);

    // THEN
    // libc++abi.dylib: terminating with uncaught exception of type std::runtime_error
  });
});

Additional content

Please provide any (mandatory) additional data to reproduce the error (Dockerfiles etc.)

Short overview
Current installation process takes way to long

Use case
Faster installation / time to first test

Detailed description
Currently, nut.js builds OpenCV from source by default.
This installation process takes quite a while and requires external tooling (cmake).
As a user I don't want to compile a OpenCV via cmake when installing nut.js

Additional content

Please provide any (mandatory) additional data for your enhancement

Short overview
OpenCV released a new major version a while ago. In order to keep up to date, nut.js should use v4.x

Use case
Up-to-date dependencies.

As a user I'd like to have a high level method which allows me to perform e.g. mouse drag actions.

Short overview
Currently nut.js uses octalmage/robotjs, a nan module. For easier support of multiple node versions, our N-API port should be elaborated.

Use case
Easier support for multiple node modules without recompiling the native module.

Detailed description

N-API is an ABI stable C interface provided by Node.js for building native addons. It is independent from the underlying JavaScript runtime (e.g. V8 or ChakraCore) and is maintained as part of Node.js itself. It is intended to insulate native addons from changes in the underlying JavaScript engine and allow modules compiled for one version to run on later versions of Node.js without recompilation.

In @nut-tree/libnut, octalmage/robotjs has been ported to N-API, which allows to use it with any node version >= 10 without having to recompile the native module.

Short overview
Provide Docker VNC image to run UI tests in CI systems.

Use case
As a user I want to be able to run and scale my UI tests on a CI system.

Detailed feature description
Provide a new VNC container which supports nut.js.

Additional content

Please provide any (mandatory) additional data for your desired feature

The devDependency version-bump-prompt was updated from 5.0.5 to 5.0.6.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

version-bump-prompt is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Your Greenkeeper Bot 🌴

Short overview
Provide access to OS windows

Use case
As a user I want to be able to retrieve window data like it's position, title or size.

Detailed feature description

• It should be possible to retrieve a list of open windows
• It should be possible to retrieve the currently active window
• It should be possible to retrieve a window's size and position
• It should be possible to retrieve a window's title

Some config files could be updated, e.g.

• more recent compile target in tsconfig.json
• remove config options which are not really required in tsconfig.json
• set -e in pre-publish script
• move before_install to install in travis.yml
• update node versions to lts/dubnium, 11 and lts/erbium and remove exclusion of 12.13.0 in travis.yml

As a user I want to type words / sentences / single keys via keyboard.

Short overview
As a nut.js developer I want to automate and streamline build and release processes.

Use case
Less manual interaction

Detailed description
Currently every build in the Travis build matrix runs a Sonar analysis, which takes quite some time. Additionally, no automated publishing is in place.

Additional content

Please provide any (mandatory) additional data for your enhancement

Possible CI stages:

• Test on Linux with latest LTS, incl. sonar scanner
• Test on all other stages
• Publish next release on develop, stable on tags

When I try to start by electron, I got an error

Short overview
Mouse speed configuration should be revisited / overhauled.

Use case
Easier / better to understand configuration of mouse movement speed.

Detailed description
The current implementation for configuring mouse movement speed is kind of odd and not straight forward to understand.
The calculation of timesteps for pixels per second is not very accurate and limited by rounding errors.

Add support for image detection over multiple scales to allow screenshots to be taken at a (slightly) different resolution

Short overview
As a nut.js user I want to get additional information why image detection failed.

Use case
Easier debugging of failures

Detailed description
Currently, nut.js only reports a failure to detect images with an error message similar to the following:

Failed: Searching for $image failed. Reason: 'Unable to locate $image, no match!'

It would be nice to have more detailed output why detection failed, e.g. confidence values etc.

Additional content

Please provide any (mandatory) additional data for your enhancement

Short overview
Image processing and image matching are different kinds of concerns and should be separated.

Use case
Better separation of concerns allows for better code reuse.

Detailed description

Additional content

Please provide any (mandatory) additional data for your enhancement

Currently NUT doesn't handle scaling on e.g. retina displays.
This causes trouble when navigating using image matching due to wrong coordinates.

Version 1.3.1

Short overview

Detailed error description
After hours wasted trying RobotJS and getting many errors, I decided to try nut, only to also have errors... I installed node-gyp setting Python 2 path to exe. I have windows-builds-tools.
`internal/modules/cjs/loader.js:589
throw err;
^

Error: Cannot find module '@nut-tree/nut-js'`

Steps to reproduce error

Additional content

Please provide any (mandatory) additional data to reproduce the error (Dockerfiles etc.)

🚨 You need to enable Continuous Integration on all branches of this repository. 🚨

To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because it uses your CI build statuses to figure out when to notify you about breaking changes.

Since we didn’t receive a CI status on the greenkeeper/initial branch, it’s possible that you don’t have CI set up yet. We recommend using Travis CI, but Greenkeeper will work with every other CI service as well.

If you have already set up a CI for this repository, you might need to check how it’s configured. Make sure it is set to run on all new branches. If you don’t want it to run on absolutely every branch, you can whitelist branches starting with greenkeeper/.

Once you have installed and configured CI on this repository correctly, you’ll need to re-trigger Greenkeeper’s initial pull request. To do this, please delete the greenkeeper/initial branch in this repository, and then remove and re-add this repository to the Greenkeeper App’s white list on Github. You'll find this list on your repo or organization’s settings page, under Installed GitHub Apps.

Short overview
Refactor code to get rid of unnecessary awaits or Promise wrapping.

Use case
Clean code

Detailed description

Additional content

Please provide any (mandatory) additional data for your enhancement

The devDependency version-bump-prompt was updated from 5.0.4 to 5.0.5.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

version-bump-prompt is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Your Greenkeeper Bot 🌴

Short overview
Add native "highlight" method similar to the Sikuli implementation

Use case
E.g. to highlight screen regions for image based matching.

Detailed feature description
When performing screen based actions it would be great to have a "highlight" functionality (red border around a given region) to visually track which regions has been detected via screenshot matching.
One simple approach would be to display a window showing a modified screenshot.

Drawback: The window blocks the screen, so underlying actions are not captured. Might be acceptable for debugging purposes.

Additional content

Please provide any (mandatory) additional data for your desired feature

robot-js seems to be pretty inactive.

Before waiting too long, it might be better to switch to an alternative.

Keyboard, mouse and screen parts can be replaced with robotjs, clipboard support could be implemented via clipboardy

🚨 You need to enable Continuous Integration on all branches of this repository. 🚨

To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because it uses your CI build statuses to figure out when to notify you about breaking changes.

Since we didn’t receive a CI status on the greenkeeper/initial branch, it’s possible that you don’t have CI set up yet. We recommend using Travis CI, but Greenkeeper will work with every other CI service as well.

If you have already set up a CI for this repository, you might need to check how it’s configured. Make sure it is set to run on all new branches. If you don’t want it to run on absolutely every branch, you can whitelist branches starting with greenkeeper/.

Once you have installed and configured CI on this repository correctly, you’ll need to re-trigger Greenkeeper’s initial pull request. To do this, please delete the greenkeeper/initial branch in this repository, and then remove and re-add this repository to the Greenkeeper App’s white list on Github. You'll find this list on your repo or organization’s settings page, under Installed GitHub Apps.

The MouseActionInterface should be renamed to MouseActionProvider for consistency.

when i give a image ,it is unable to locate the image and says No match.

screen.config.resourceDirectory = downloadsPath;
await mouse.move(straightTo(centerOf(screen.find("down.PNG"))));

Failed: Searching for down.PNG failed. Reason: 'Unable to locate C:\down.PNG, no match!'

Please ,tell me how to find the image and move to it and click()

Short overview
Some APIs, e.g. mouse actions could be refactored for better usability.

Use case
Increased usability of APIs

Detailed description
Some APIs, e.g. mouse actions in combination with image search are quite cumbersome.

await mouse.move(await movement.straightTo(Location.centerOf(await screen.find("plus.png"))));

vs.

await mouse.move(straightTo(centerOf(screen.find("plus.png"))));

Additional content

Please provide any (mandatory) additional data for your enhancement

C:_git\testing>npm i @nut-tree/nut-js

[email protected] install C:_git\testing\node_modules\opencv4nodejs-prebuilt
prebuild-install || (node ./install/dependencies.js && node ./install/install.js)

prebuild-install WARN install No prebuilt binaries found (target=10.15.3 runtime=node arch=ia32 libc= platform=win32)
Unsupported platform, only x64 is supported.
npm ERR! code ELIFECYCLE
npm ERR! errno 4294967295
npm ERR! [email protected] install: prebuild-install || (node ./install/dependencies.js && node ./install/install.js)
npm ERR! Exit status 4294967295
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

P.S. robotjs was installed normally.

Short overview
nut.js should provide a waitFor method to search for an image until a timeout is reached

Use case
Dynamic waits on images.

Detailed feature description

Additional content

Please provide any (mandatory) additional data for your desired feature

Short overview
Speed up installation process of nut.js

Use case
Faster build / install cycle

Detailed feature description
When running npm ci, opencv4nodejs will recompile its bindings, which takes an unpleasant amount of time.

Additional content

Please provide any (mandatory) additional data for your desired feature

Version: 1.3.1

Whenever Travis CI or GitHub Actions on macOS hit the first test that uses nut.js keyboard, I get a segmentation fault:
/Users/travis/.travis/functions: line 113: 6934 Segmentation fault: 11 npm run test
(full Travis log here)

Even going back to my last passing build and re-running it, the same build now fails with a Segmentation fault. Same is true with GitHub actions (see e.g. here).

I played around with moving nut.js calls to keyboard, and each time my tests would fail with a seg fault whenever it hit the first call.

I've been busting my head about this for two days! Any ideas?

For context, all I really want to do is to be able to send keyboard events like "Cmd+S" and "Alt+Cmd+Shift+Y" on macOS from my node.js integration tests. Other libraries have worked for Linux & Windows, but none has worked consistently on macOS.

Documentation would be helpful

Short overview
I am looking for a way to find all match on the screen for a template image

Use case
Same as the existing screen.find(...) method

Detailed description
I am thinking about something like screen.findAll(...): Promise<Region[]>
A maxMatch parameter might be needed for performance

Short overview
Button enum should be exported

Use case
mouse.pressButton(...) and mouse.releaseButton(...)

Short overview
Find hooks would allow to execute actions whenever a certain image if found on screen

Use case
When using a find hook it would be possible to provide a callback to handle e.g. pop-ups etc.
So before resolving possible hooks should be awaited.

Detailed feature description
A find hook should be registered using the search needle string passed to find.
Hooks have to implement the following signature: (target: MatchResult) => Promise<void>

Additional content

Please provide any (mandatory) additional data for your desired feature

Short overview
As a nut.js developer I’d like to have stable tests without flaky testcases

Use case
Reduced flakiness and less uncertainty

Detailed description
Currently some cases related to timing are quite flaky. This flakiness could be reduced by introducing a small error margin.

nut.js provider classes currently contain code which is split based on whether it's native / os specific or image based.

This should be further refined based on keyboard, mouse, screen, clipboard actions etc.

Short overview
As a nut.js user, I'd like to use OCR

Use case
Extracting text from images, PDF files etc.

Detailed feature description
nut.js should support OCR to be able to extract text from images of any kind. This would allow for additional validation logic, e.g. to check whether a PDF has been generated successfully or not.

As a user I want to use copy & paste functionality provided by system clipboards.

Version
latest dev

Short overview
Target regions currently do not account for pixel density in search results.

Detailed error description
When searching for an image on screen the current implementation does not take pixel density into account for width / height values. This leads to wrong locations for Location.centerOf

Steps to reproduce error

Additional content

Please provide any (mandatory) additional data to reproduce the error (Dockerfiles etc.)

Version
1.1.1

Short overview
ClipboardAction#copy does not resolve

Detailed error description
The copy() method of Clipboardy ClipboardActionProvider does not call resolve and therefore blocks.

Steps to reproduce error

Additional content

Please provide any (mandatory) additional data to reproduce the error (Dockerfiles etc.)

Short overview

Right now there is no way to invoke something like mouse.buttonDown() or mouse.buttonRelease() in the public mouse api

Use case

More granularity and control especially in complex drag and drop scenarios.