raiman / sikulix4python

Use SikuliX from real Python via py4j

sikulix4python

(currently paused --- will be continued 2022 hopefully;-)

Use SikuliX from real Python via py4j (but no need to know, how it works ;-)

under development -- no guarentee for anything :-)

... but you might post issues (questions, bugs, ideas, requests, ...)

What you need

• the latest sikulixapi.jar (2.0.5+) (download from here)
• a Python installation 2.7 up to 3.9 (testing on 3.9)
• run pip install py4j to get the support on the Python side
• a mature Python IDE (I use IntelliJ's PyCharm) or just a Python REPL.

How to test

Currently it is recommended, to fork this project and work in the project context.

But this should also be possible:

• copy the folder sikulix4python from inside the project to a place, where your Python will find it, when imported

In both cases:

• put from sikulix4phyton import * at the beginning of your test script
• use stuff from sxundotted just as is
• use stuff from sxclasses to access features of the SikuliX API classes like Region, Screen, Location ...

Currently you have to look into the Python files, to find out, what is possible.

How is the SikuliX Java API accessed from Python

Before running Python scripts this way, you have to start a SikuliX server instance manually:

java -jar path-to/sikulixapi.jar -p

When you see this in the commandline window, it is running:

[info] SikulixAPI: Running py4j server on port 25333

To stop the server you can use the known stop hotkey (Windows: ALT-SHIFT-C, Mac: CMD-SHIFT-C).

What is the plan

• have the complete official SikuliX API with docs available at time of script development (autocomplete, docs, help, ...)
• be able to additionally access every public method from the SikuliX Java API (you must know the method signature)
• have a handy solution for handling image path's of imported scripts/modules
• run existing SikuliX scripts without changes (... but there will surely be exceptions)

Things you should be aware off

• you might get any problem at any time, since the current state is proof of concept
• anything might be changed at any time without notice
• errors and warnings you get in the terminal window can be ignored, as long as it works as intended on the Python side
• output coming from the SikuliX Java API is currently only going to the terminal window
• my current development work is on macOS 11, Windows 10, Java 11/15, Python 3.9 with PyCharm
• no tests on Linux yet

An example

# https://www.python.org should be opened in Safari

# build the bridge to SikuliX
from sikulix4python import *

# make images available in the folder of the script
addImagePath()

switchApp("Safari")

hover() # undotted uses SCREEN object (sxundotted)

scr = Screen()

# getCenter() is found auto-magically,
# though not defined in the Python class Screen
# see sxclasses::__getattr__
# one gets method missing, if signatures do not fit
scr.getCenter().grow(100).highlight(2)

img = "img.png" # located via ImagePath

# a Match object is completely handled at the Java level
# not defined at the Python level
match = scr.exists(img, 3) # method missing, wrong signature
match = scr.exists(img, 3.0) # number must be float/double
if match:
    match.highlight(2)
    match.click()

sikulix4python's Issues

Detecting Ui controls like button, textbox, checkbox

@RaiMan
thanks for your work.

I want to highlight another application UI controls from my application based on the mouse position.

example : when mouse moves over the button then I want to highlight the button region(rect)?

How can I do the same with the help of your library?

ref:
https://dsp.stackexchange.com/questions/1471/how-can-i-detect-buttons-in-a-gui-of-an-app-with-pattern-detection-using-python

please advise.

How to run search within screenshot?

Hello! Is is possible to run search within the screenshot?

Best Regards,
Alex

it does not works on linux

when I try to run it on linux ,it can not find the image,here is the exception below
Traceback (most recent call last):
File "test.py", line 15, in
click("desktopShow.png")
File "/home/xuman/Desktop/sikulix4python-master/sikulix4python-master/sikulix4python/sxundotted.py", line 32, in click
return SCREEN.click(*args)
File "/home/xuman/Desktop/sikulix4python-master/sikulix4python-master/sikulix4python/sxregion.py", line 35, in click
return self.instance.click(convertArgs(args))
File "/home/xuman/.local/lib/python3.7/site-packages/py4j/java_gateway.py", line 1310, in call
answer, self.gateway_client, self.target_id, self.name)
File "/home/xuman/.local/lib/python3.7/site-packages/py4j/protocol.py", line 328, in get_return_value
format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling o33.click.
: FindFailed: desktopShow.png: (0x0) in R[0,0 1920x1080]@s(0)
Line 2226, in file Region.java

    at org.sikuli.script.Region.wait(Region.java:2226)
    at org.sikuli.script.Region.wait(Region.java:2244)
    at org.sikuli.script.Region.getLocationFromTarget(Region.java:3298)
    at org.sikuli.script.Region.click(Region.java:3916)
    at org.sikuli.script.Region.click(Region.java:3892)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.base/java.lang.Thread.run(Thread.java:834)

hi RaiMan，i just use sikulix4python on my linux systerm,and i find a question

i use pycharm in my linux systerm for AutoTest, now i find a question
click(img), can't find the img . the screenshot below:

the same code works very well on MacOS and Win autoText ~

so thx u help
~ tips:poor english don't mind~ 😄

highlight does not exist?

when I run the example,
scr.getCenter().grow(100).highlight(2)
I get the Trace:
py4j.Py4JException: Method highlight([class java.lang.Integer]) does not exist
and I cannot understand what is the mean of"
one gets method missing, if signatures do not fit
"

Tested Py4J feature in the new RF library and it works fine on any OS

Hi @RaiMan

As you proposed last year in this topic discussion jpype-project/jpype#911, I enabled both Py4J and JPype on my Robot Framework library proposal (https://github.com/adrian-evo/robotframework-sikulixlibrary). From my testing, SikuliX with Java bridge works fine now also on MacOS with Py4J, and on Windows and Linux with both Py4J and Jython.

I think I didn't used any sikulix4python experimental features you described in your project but I guess I used just your enabled Py4J server and standard Java features. Hopefully this approach will work also in the future SikuliX releases.

However what I noticed might not work yet is some get features, e.g. ImagePath.get(), and the known App open issue with Ubuntu. Also tested the library extensively in Windows project for several hours and didn't encountered any relevant issue.

Changing settings in a bundle Sikulix + py4j

Hey! At the moment I am writing my mini framework using Sikuli + py4j. But I just can't seem to solve one problem
How can I change the settings? For example, I want to change the search precision, but changing "JavaGW.jvm.org.sikuli.basic.Settings.MinSimilarity" does nothing.

RaiMan I would really appreciate a hint on how to get around this. I think the problem is in Settings.synchronized init () but I'm not sure.

Code example:

from py4j.java_gateway import JavaGateway

JavaGW = JavaGateway()
SXPKG = JavaGW.jvm.org.sikuli
SXPKG.basic.Settings.AutoWaitTimeout = 100.0

Screen = SXPKG.script.Screen()

print(SXPKG.basic.Settings.AutoWaitTimeout) # <py4j.java_gateway.JavaPackage object at 0x7f3a09724a30>
print(Screen.getAutoWaitTimeout()) # 3.0

Class not prepared for SikuliX in MacOS

Class not prepared for SikuliX in MacOS

Traceback (most recent call last): File "Main.py", line 17, in start from Client.SocketClient import Client File "/Users/vt100i/Desktop/ST/dcs1000_AutoTest/PointW_Mac/Win/Client/SocketClient/Client.py", line 5, in <module> from Client.SocketClient import Exe_keyword File "/Users/vt100i/Desktop/ST/dcs1000_AutoTest/PointW_Mac/Win/Client/SocketClient/Exe_keyword.py", line 10, in <module> from Client.Libraries import WhiteBoardKeywords File "/Users/vt100i/Desktop/ST/dcs1000_AutoTest/PointW_Mac/Win/Client/Libraries/WhiteBoardKeywords.py", line 4, in <module> from Client.Libraries.Recognise.sikulix4python import * File "/Users/vt100i/Desktop/ST/dcs1000_AutoTest/PointW_Mac/Win/Client/Libraries/Recognise/sikulix4python/__init__.py", line 3, in <module> from . sxundotted import * File "/Users/vt100i/Desktop/ST/dcs1000_AutoTest/PointW_Mac/Win/Client/Libraries/Recognise/sikulix4python/sxundotted.py", line 3, in <module> SCREEN = Screen() File "/Users/vt100i/Desktop/ST/dcs1000_AutoTest/PointW_Mac/Win/Client/Libraries/Recognise/sikulix4python/sxbase.py", line 24, in __init__ raise Exception("Class not prepared for SikuliX") Exception: Class not prepared for SikuliX

java: "1.8.0_251"
python: 2.7.16
sikulixapi: 2.0.5

please help me~ @RaiMan

[error] Python server: py4j not on classpath

Python 32-bit 3.8.7
Java 15 version 15 vm 15.0.1+9 class 59.0 arch 64
java -jar sikulixapi.jar -p

[error] Python server: py4j not on classpath

Verbose

S D:\Backup\Projects\SikuliX\sikulix4python> java -jar sikulixapi.jar -v
[DEBUG]startUp: API: Running: D:\Backup\Projects\voice recognition toolbox\Caster projects\SikuliX\sikulix4python\sikulixapi.jar
[DEBUG]startUp: API: AppData: C:\Users\Main\AppData\Roaming\Sikulix
[DEBUG]startUp: API: starting with classpath: sikulixapi.jar ...
[DEBUG]startUp: API: no extensions.txt nor valid content
[46 debug] RunTime:temp folder ok: C:\Users\Main\AppData\Local\Temp\Sikulix_1318413536
Options: *** options dump
Options: testing =
Options: OptionsFile = C:\Users\Main\AppData\Roaming\Sikulix\SikulixStore\SikulixOptions.txt
Options: *** options dump end
***** show environment for 2.0.4-2020-03-14_08:01 API
user.home: C:\Users\Main
user.dir (work dir): D:\Backup\Projects\SikuliX\sikulix4python
user.name: Main
java.io.tmpdir: C:\Users\Main\AppData\Local\Temp
running 64Bit(amd64) on w (10.0) from a jar
java 15 version 15 vm 15.0.1+9 class 59.0 arch 64
app data folder: C:\Users\Main\AppData\Roaming\Sikulix
executing jar: D:\Backup\Projects\SikuliX\sikulix4python\sikulixapi.jar
*** classpath dump sikulix
  0: sikulixapi.jar
*** classpath dump end
***** show environment end
[DEBUG]startUp: API: *********************** leaving start
[29 debug] Sikulix: starting API
[47 debug] RunTime:temp folder ok: C:\Users\Main\AppData\Local\Temp\Sikulix_273099905
Options: *** options dump
Options: testing =
Options: OptionsFile = C:\Users\Main\AppData\Roaming\Sikulix\SikulixStore\SikulixOptions.txt
Options: *** options dump end
***** show environment for 2.0.4-2020-03-14_08:01 API
user.home: C:\Users\Main
user.dir (work dir): D:\Backup\Projects\SikuliX\sikulix4python
user.name: Main
java.io.tmpdir: C:\Users\Main\AppData\Local\Temp
running 64Bit(amd64) on w (10.0) from a jar
java 15 version 15 vm 15.0.1+9 class 59.0 arch 64
app data folder: C:\Users\Main\AppData\Roaming\Sikulix
executing jar: D:\Backup\Projects\SikuliX\sikulix4python\sikulixapi.jar
*** classpath dump sikulix
  0: D:\Backup\Projects\SikuliX\sikulix4python
  1: sikulixapi.jar
*** classpath dump end
***** show environment end
[287 debug] RunTime:loadlib: trying opencv_java342
[442 debug] RunTime:addToWindowsSystemPath: added to systempath:
C:\Users\Main\AppData\Roaming\Sikulix\SikulixLibs
[444 debug] RunTime:libsExport: folder exists: C:\Users\Main\AppData\Roaming\Sikulix\SikulixLibs (2.0.4 - 202003140801)
[461 debug] RunTime:loadLib: opencv_java342.dll (success)
[461 debug] OCR: start: Tess4J 4.4.1 using Tesseract 4.1.0
SikuliX API: nothing to do
[478 debug] RunTime:***** final cleanup at System.exit() *****
[491 debug] FileManager: deleteFileOrFolder:
C:\Users\Main\AppData\Local\Temp\Sikulix_273099905
[1083 debug] RunTime:***** final cleanup at System.exit() *****
[1106 debug] FileManager: deleteFileOrFolder:
C:\Users\Main\AppData\Local\Temp\Sikulix_1318413536

how to work with Key and KeyModifier classes

hi

i want to type ALT+A

my code is:

reg = Region()
reg.type("a", Key.ALT())

and i define Key class by add this code to sxclasses.py :

class Key(SXBase):
    SXClass = SXKey

that SXKey = SXPKG.script.Key

but after run :

Traceback (most recent call last):
  File "testSikulix4python.py", line 22, in <module>
    reg.type("a", Key.ALT())
AttributeError: type object 'Key' has no attribute 'ALT'

Tanks 🌹🙏🏻

The relative path of img.png does not work.

Win10+jdk1.8+sikulix 1.1.4

Try to run the given code, the relative path does not work as expected.
Here is the debug info from sikulixapi:

corresponding to the code:

img = "img.png"
scr.exists(img,3.0)

White dot in the center

There will be a white dot in the center of the screen when using
java -jar path-to/sikulixapi.jar -p
on Windows 10.
This may cause OCR failure.
Not tested with pattern recognition. But I think it's best to remove it.