testartool / testar_dev Goto Github PK
View Code? Open in Web Editor NEWTESTAR, automated testing through the Graphical User Interface
Home Page: http://www.testar.org
License: BSD 3-Clause "New" or "Revised" License
TESTAR, automated testing through the Graphical User Interface
Home Page: http://www.testar.org
License: BSD 3-Clause "New" or "Revised" License
When Testar opens a link in a new tab, the webdriver doesn't switch to that tab.
Instead it continues to test the current page.
Generate sequence, oracles, and logs with new names when replaying a sequence, so that you can compare the results between different executions of the same test sequence.
Creating a "Wait for TESTAR" icon, when SUT is showing but TESTAR not yet ready
The new Record-mode does not have visualization on (so that it does not cover the SUT for user actions) - now user does not know when to start doing the actions - maybe TESTAR is not yet ready to record.
When we execute a Windows 10 store application like Calculator (C:/Windows/System32/calc.exe), the resulting process is not only a calc.exe process (TESTAR prepare a SUT with this name).
Is a combination of several process (the main one is ApplicationFrameHost: https://www.howtogeek.com/325127/what-is-application-frame-host-and-why-is-it-running-on-my-pc/)
So we can't link correctly TESTAR with this kind of applications using the Command_line.
To link to the SUT we must execute manually the calculator and use the SUT_Windows_title mode.
A part of the code (the part that stores the widgets) is implemented in the specific protocol class (for now). I made this choice since it was the only location that new the available widgets which are used to select the next action. I will update the documentation
gradlew run tries to read about ./testar/settings, ./testar/output, /testar/windows.dll etc, path that does not exist.
This problem does not occur with installDist because the directory created is correct ./bin/settings, ./bin/output, ./bin/windows.dll, etc.
If you move the files ./testar/target/scripts in ./testar you can work with gradlew run, without having to run installDist every time.
Option 1: Move the commented files (2 folders and 4 files)
Option 2: Modify Main.java code, SettingsDialog.java, Windows.java and the path structure of test.settings,
However, this does not allow working with installDist, since the read path is modified.
Option ?: Can this be solved from some build.gradle file?
Think about the possibility of integrate Jenkins and TESTAR Github, to automate the testing of pull requests.
The main idea would be the execution of a virtual machine that contains the TESTAR version of the pull request, and design a battery of test cases that can be executed from the command line.
Example: testar SUTConnectorValue=C:\Windows\System32\notepad.exe Sequences=20 SequenceLength=100
After each execution check that the result was correct.
(nothing has failed and there are the corresponding logs?)
This capture represents the detection of the different widgets when we use the native VLC interface.
VLC can be tested using the native interface, however, if we use the Skins that modify the VLC GUI, the widgets are not recognized. Skins components are not detected by TESTAR or by the Windows OS.
Tools -> preferences -> use custom skin -> Select the downloaded skin
(I have tried the "eDark", "NIce" and "Debian Red Coast" skins on the Windows 10 OS)
This test was executed on pekka dev version, as the master has other issues.
This bug does not happen on Win10.
The HTML report generates the first state and selects an action, but executing action has some bug.
Trace:
Protocol_desktop_generic: selectAction(), (Overriding Urko's Grapher and AbstractProtocol implementation)
AbstractProtocol: saving state into HTML Sequence Report
class nl.ou.testar.GuiStateGraphWithVisitedActions: new state, selecting randomly from 5 available actions
class nl.ou.testar.IdBasedGuiState: addVisitedAction: action removed from the unvisited actions
[END] Running processes (14):
ScreenshotSerialiser finished
TestSerialiser finished
TESTAR sequence graph dump on way ...
Sync XML graph: synced
... finished TESTAR sequence graph dump
TESTAR grapher reset
Exception <Unable to update layered window!> has been caught; Stack trace:
[1] org.fruit.alayer.windows.Windows.UpdateLayeredWindow(Native Method)
[2] org.fruit.alayer.windows.GDIScreenCanvas.end(GDIScreenCanvas.java:180)
[3] org.fruit.monkey.AbstractProtocol.runAction(AbstractProtocol.java:905)
[4] org.fruit.monkey.AbstractProtocol.runTest(AbstractProtocol.java:1196)
[5] org.fruit.monkey.AbstractProtocol.run(AbstractProtocol.java:1512)
[6] org.fruit.monkey.AbstractProtocol.run(AbstractProtocol.java:131)
[7] org.fruit.monkey.Main.main(Main.java:143)
If generating many sequences in one TESTAR session, only one HTML sequence report file is created. Should be one HTML file per sequence (preferably in sync with sequence file numbering).
2- If a tab or windows closes, the execution of the webdriver fails:
org.openqa.selenium.NoSuchWindowException: no such window: target window already closed
I dont know if this is a limitation of the webdriver or it can be activated in some way in the protocol,
but sometimes it may be necessary to read the url of new tabs or windows and close them.
3-If SUT closes unexpected the execution of the webdriver fails:
org.openqa.selenium.WebDriverException: chrome not reachable
Is it possible to add some way to detect that the chrome browser is no longer reachable and save that sequence like unexpectedclose? And avoid TESTAR fail.
Suggestion:
If for any reason TESTAR & webdriver fails, sometimes the chromedriver.exe process stays alive.
Would it be possible to add in the beginning of the webdriver protocol, some way to find the existence of chromedriver processes (old executions) and close them to perform a clean invocation of the chromedriver?
SOLVED:
1- The problems seems something into my environment
The position and interaction of the mouse and actions with the widgets seems wrong. However the widgets are represented correctly by the green dots.
It could be something from my personal environment, but with other protocols and applications it doesn't happen, neither with the previous version of webdriver.
4-Casting Objects into Runtime execution error (check if object its Double or Long):
java.util.concurrent.ExecutionException: java.lang.ClassCastException: java.lang.Double cannot be cast to java.lang.Long
Spy Mode on "http:/angularexampleapp.com" and playing with "Heroes List", I try to replicate but cant do it
5- Frameset web containers problem (Finally not Login & Webdriver):
Making a login on a web page that doesn't change the url, it seems that the widget tree doesn't refresh correctly.
This is maybe unwanted behaviour instead of a bug, but it messes up the action selection, because random chance of selecting editable widgets is much higher than clickable widgets.
TESTAR generates multiple edit actions with different text for each editable widget, for example text box.
A better behaviour would be generating one edit action for each editable widget, then generating the input for the edit if it is selected, or just generating one input for the action.
The protocol_filter.xml is now in the testar/bin directory in stead of the related settings direcory of the used protocol.
When using this binary zip TESTAR of Ramon (test.settings and protocol included) https://testar.org/alpro/
I tried to filter some of the URLs that are on the right. First I tried with the CAPS_LOCK + CNTRL which does not work. Then I tried filtering with the regular expression that gives some strange behavior.
Here are two videos that shows it:
https://www.dropbox.com/sh/8z71n9ac8o8quaw/AABHCju_igNYLClQkwvqjCbpa?dl=0
For an experiment, to observe execution traces for temporal behavior (e.g. 'eventually'), I created a very simple java application: a single window java application that simulates a stack with a depth of 7.
I deliberately wanted to limit the actions to 'control' TESTAR.
When inspecting the log, i noticed that the abstract ID's of the executed Actions showed up in the output (logs, XML, .DOT)
conforming to the TESTAR_manual 1.3 page 51:
- Concrete
Action ID: the ID for an action a in state s is calculated as id(a) =f(id(s) + p1 + โฆ + pk) where pi are the action properties.- Abstract
Action ID: similar to the concrete action ID, but discarding properties of the action (e.g. texts typed, keys pressed). For example, we can refer to writing in a concrete textbox, no matter of the text used to write
In other words: Abstract ID's for actions are derived from the concrete state id. This functionality is implemented in TESTAR. So, this is by design and as such this is not a code bug.
However, i would have expected that an abstract action-id would be derived from the abstract state-id. Comparable to the way an abstract state is derived from the abstract widget tree.
The current implementation does not condense the number of actions when they are abstracted, which is usually a reason for applying abstraction.
( see
AbstractActions 20180529 CSS.pdf
)
Is this worth changing?
Idea: having a way to define RegExp for titles of closing actions (in a similar way to suspicious titles), and not reporting failing sequence when such an action closes the SUT.
These closing actions could be used for "graceful shutdown" of the SUT in the end of sequence.
Design a layout of the datamodel which will be used to store the data
Maakt het bouwen misschien nog wat soepeler voor mensen die van Ant komen.
When TESTAR finds a failure, it should automatically try to find the shortest sequence to reproduce the failure.
This way invalid regular expressions are not accepted through the dialog. Now what happens is that no warning whatsoever is given and TESTAR just does not apply it.
Study the benefits of the Java 9 ,10 and 11 environments in relation to TESTAR.
Prepare an execution environment and perform compilation + execution tests.
UPDATE:
Branch "testar_new_jdk" allows the compilation "gradlew installdist" and execution of TESTAR into a VM with Java JDK12.
c7d480a
[ OK ] Compiles and allow the execution of Notepad + Windows UIAutomation
[ OK ] Compiles and allow the execution of Calculator.jar + Windows UIAutomation
[ Swing ERROR Temporally SOLVED] c11ee17
Access Bridge method:
Windows.GetHWNDFromAccessibleContext(vmidAC[0],vmidAC[1]);
Test settings is <./settings/desktop_generic/test.settings>
Saved <./settings/desktop_generic/test.settings>
Saved current settings to <./settings/desktop_generic/test.settings>
java.lang.ClassNotFoundException: Protocol_desktop_generic
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.fruit.monkey.Main.main(Main.java:328)
Right now, the settings object is passed around to all methods that need it.
But passing it to other methods is painful.
Also, this ensures there is only one settings object.
Because they will then get built, so we can spot errors.
Eclipse already adds them when importing with gradle eclipse
, but the folder structure is wrong:
Description Resource Path Location Type
The declared package "" does not match the expected package "settings.desktop_generic" Protocol_desktop_generic.java /testar/resources/settings/desktop_generic line 1 Java Problem
We can:
In either case it seems to make sense to automatically build the protocols with Gradle.
To remember, an environment running now on Ramons server.
When you hover over an element in the GUI you get a help box with useful information.
In some cases there is no help box.
For example in General Settings tab no info for:
When using the webdriver module, the above exception is thrown.
It originates in sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method).
If we use the Java Swing version, the native elements created at the operating system level are not correctly recognized.
The buttons Close, Minimize, Maximize and others, not belonging to the Java Swing interface and their JVM associated, are not detected by TESTAR.
Happened with IE but not with Chrome:
SUT has a modal popup that is another process, and TESTAR tries to bring original process to foreground and that does not work because of modal popup.
In addition to filtering actions with RegExp, there could be another field for defining "closing actions". Those actions could be automatically used for "graful shutdown" of the SUT, as is possible in stopSystem(), instead of just killing the process.
In Record-mode: input text is not captured correctly, sometimes some text is missing on the captured test sequence
After build and first start up of TESTAR, the amount of sequences is per default 1, but when doing a GENERATE TEST, this amount is not enforced.
When changing the amount of sequences, this does not solve this.
However, shen adapting the amount of actions, the amount of sequences is also taking into account.
This must be a bug somewhere in saving and/or overwritting the settings file.
TESTAR creates an empty sequence in SPY mode and that cannot be executed in Replay / View mode - throws TESTAR exception.
An idea:
How about providing a list of all the running processes that user can select from, bringing the selected process to foreground and letting user to confirm that is the SUT.
When starting Spy mode Shift+3 will do nothing, only after Shift+2.
Shift+1 is a toggle key for actions.
Shift+2 is a toggle key for some information.
Suggestion to make Shift+3 a toggle key for detailed information.
OR perhaps better
Shift+1 is a toggle key for actions.
Shits+2 is a toggle key for some information->detailed information->no information-> some information....
Shift+3 no function.
I have seen this in VLC (but also in the APRO web site). It recovers and starts selecting other actions after a while, but it repeats the same over and over again.
I will try to make a video next time I see it.
Carlos Sengers was playing around with the Graph DB and he found something that looks like a visualization of this behavior.
Sometimes if TESTAR runs with several sequences, it can be seen that the last or last actions that appear as performed inside the log (output / log / sequenceX.log), it doesn't appear as executed in their corresponding sequence file (output / sequence / sequenceX)
This can be compared by opening the log from a text editor and opening the corresponding sequence from the "View" mode offered by TESTAR.
In TESTAR protocol it is possible to define startSUT() and beginSequence() specific actions before starting the normal testing. It would be useful if the user could record a script (as now in GenerateManual mode), and tell TESTAR to run that script (from sequence file) when starting the SUT or a sequence.
The webdriver protcol /webdriver layer designates slide actions incorrectly, resulting in unexecutable actions.
And this also slows down the action visualization.
Example : elements of the lists in the footer on https://www.ou.nl
it says:
After installation of these tools the file compile_w10.bat and clean_w10.bat need to be adapted. CALL "C:\2017\BuildTools\Common7\Tools\VsDevCmd.bat" -arch=x64
Where are compile_w10.bat and clean_w10.bat?
Itt should be made clearer that you need to edit them.
I have adapted the README.rd in tanja_dev branch. See if you like this better.
By mentioning Close in de Filters tab in the Testar GUI you see no green dot in Spy mode after shift+1.
That is correct.
By using the clickfilter-feature I'am trying to deselect Close, so that it is now a possible action again.
shift_Ctrl.
The result is a green box around Close. Why?
I expected a green dot again and Close deleted from the test.settings file.
The green box around close will not be selected during Generate.
The instructions have this:
After installation of these tools the file compile_w10.bat and clean_w10.bat need to be adapted. CALL "C:\2017\BuildTools\Common7\Tools\VsDevCmd.bat" -arch=x64
I can find w7 of both files and I can find w10.cpp but I can't find the files that are supposed to be adapted.
In Spy-mode, with a specific TESTAR run-time control, you could add application specific oracles into specific widgets (a bit like click filter is working).
It should be able to listen during testing (ONLINE oracle) and it should stores it all in a file that can be inspected after testing (OFFLINE oracle).
System.out and System.err listens to the StdOut/Err of TESTAR itself. We should somehow catch the StdOUT/ERR of the SUT when it is started up.
HINTS on a possible solution within Java: https://stackoverflow.com/questions/3643939/java-process-with-input-output-stream
HINTS on a possible solution in cpp WinAPI:
https://msdn.microsoft.com/en-us/library/windows/desktop/ms682425(v=vs.85).aspx
https://msdn.microsoft.com/en-us/library/windows/desktop/ms682499(v=vs.85).aspx
By disabling all the notepad buttons, leaving only the text field and running about one hundred actions, this issue can be observed using the View mode of TESTAR or using the desktop_simple_stategraph protocol (with the html report).
Sometimes the screenshot of the state that transits from an action performed is not shown correctly in these captures.
Current status: we have the text "161541072" in the field
We perform action: Type text "-1408540088"
New state to observe: we have the text "-1408540088" in the field
However, a capture of another different state is shown:
Investigate if it occurs only with the text input fields or also with other widgets
Problem checked with JEdit and Freemind.
Java applications executed as .exe, dont link correctly with TESTAR using Command_line.
The created SUT is defined as JEdit.exe / Freemind.exe, but the resulting process in the OS is a java.exe / javaw.exe process.
To link to the SUT we must run the .jar application, fortunately JEdit & Freemind contain it.
Just a few lines in README.MD to get someone started.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.