sea-lab-wm / burt Goto Github PK

In this way, the user cannot re-select screenshots or re-click on buttons

If the user says "this was the last step", the chatbot should be able to identify that and move on with the conversation.

Right now, this is not working, so the conversation is stuck asking for the next S2R every time

For BURT's evaluation, we need to ask for the ID that we will use to identify the participants.

The chat should ask for this after the user says hi.

I have a question about the TraceReplayer indexing of the screenshots.

This issue is related to the differences between the regular and augmented screenshots of Mileage that @ojcchar mentioned.

The screenshot 'com.evancharlton.mileage.User-Trace.1.com.evancharlton.mileage_3.1.1_mileage9_augmented.png' is an augmented screenshot which tells us the component that was interacted in 'com.evancharlton.mileage.User-Trace.1.com.evancharlton.mileage_3.1.1_mileage8.png'. So, I think these two images will correspond to the same screen. So, for screen '........i.png', the augmented screenshot will be '.......(i+1).png'

I think Crashscope has the same implementation. I am referring to one example from the Crashscope data for mileage. Here, for the screenshot 'com.evancharlton.mileage.Top_Down.Expected..1.com.evancharlton.mileage_3.1.1_36.png', the augmented screenshot is 'com.evancharlton.mileage.Top_Down.Expected..1.com.evancharlton.mileage_3.1.1_37_augmented.png'. Another example of the images are 'com.evancharlton.mileage.Top_Down.Expected..1.com.evancharlton.mileage_3.1.1_47.png' and 'com.evancharlton.mileage.Top_Down.Expected..1.com.evancharlton.mileage_3.1.1_48_augmented.png'

@kpmoran and @ojcchar Please let me know your suggestion.

Some S2Rs can be long. When that happens, the "enlarge buttons" get misaligned and overlap with the screenshots right below. See the attached image.

It could be annoying to scroll up every time to know what the chatbot said. The chatbot should stay at the initial textual messages.

The problem is shown below.

I suspect the bug happens when the user says "no" when confirming the OB screen and then she says "yes" when the chatbot asks the question again.

The chatbot cannot recognize the "close app" step as seen below:

I have done the task of creating the execution.json file for the trace-replayer. I have pushed the changes to Github. I will make some changes to make a more clean code. There are some current scenarios that I want to discuss with you. For example,

i. For searching activity (image attached), I cannot extract the GUI component because to come to this page and to write something in the search text component, we clicked a search button which is a different GUI component. When we type something in the search text button, we select one of the predictive texts from those options. However, we cannot extract the predictive text from the uiautomator. Therefore, I did not take screenshots for the search option.

ii. For the email or password, I took the screenshots twice: (i) when clicking on the GUI component , (ii) after writing the texts. Please let me know if I need to make any change to that.

iii. For the swipe option, we took the screenshots where the screen is clicked initially with the cursor. If the initially selected option is a GUI component, we get the screenshot of the GUI component. However, in some cases, we got a cropped image, because the GUI component is not selected properly.

iv. For typing texts, each character is considered as a click event using coordinates. I did not take screenshots when typing all characters. I took the screenshots only before and after writing texts in a GUI component.

I have attached one result for the trace-replayer here. Please let me know your opinion on the above mentioned scenarios.

traceReplayer.zip

When the browser is refreshed in the middle of a conversation, there is a crash on the client-side (Javascript).

Whenever the chatbot is refreshed, after the bug report has been generated, the link of the bug report is incorrect, it is http://localhost:8081/undefined rather than the valid one (eg., http://localhost:8081/P23-Mileage-3.1.1-e0b150f1-6980-4db6-b577-b1e5293e4e6f.html)

When no picture is selected and the done button is clicked, there is no validation whatsoever and null selection seems to be sent to the server.

This seems to be happening in any of the widget messages where there is a screenshot selection.

The chatbot should check that a selection is done, before sending the request to the server.

We should externalize the messages of the chatbot to a JSON file.

For each action, we should have one or more templates of the messages.

I am thinking the following format:

{ "ACTION1": [ "template1", "template2" , ...], "ACTION2": [ "template1", "template2" , ...], ... }

If we have multiple templates, we can randomly pick one in the action classes. In this way, the conversation looks more natural.

Please provide me the list of components that we need from the execution step.

The chatbot should give specific feedback.

For example, for the S2R "I tapped on submit button":

The chatbot should say that it didn't match any "submit button" and should prompt the user to rephrase S2R, clarifying that part specifically.

Alternatively, the chatbot should offer a list of suggestions to "submit button" either textually or graphically.

As seen in the image below, some of the screenshots for the S2R (e.g., I added fillup options) are incorrect.

Right now, if a step does not have a screenshot, which happens when a specific action from the graph is not matched, the chatbot tries to infer this step by matching the GUI components of the screens.

I suspect that the step inference is happening for an incorrect screen, for the one shown, rather than for the "setting screens", which is the one I targeted.

To reproduce this bug, we can input the S2R seen in the image on the chatbot.

1. When we click the screenshot, the popup window should show up with the image enlarged
2. Add arrows between S2R
3. The icon of the screenshot popup window is not shown properly (a ? is current shown)

As seen in the image, part of the screenshot is cropped because it is too large.
It would great if it can fit the height of the browser window. We should make the screenshot smaller.

When the user says "no" to the question asked by burt that says that "you selected [OB Screen], is that right?" (this is not the exact message)., an exception occurs in the server and it is not caught, leading to the chatbot not responding to the user (an error is thrown in the background).

For the evaluation, we will collect the time the participants take for reporting the bugs. We need a way to collect the time in BURT

If we take a look at the first augmented image, TraceReplayer can identify the GUI component easily.

Example:
Augmented Image 1:

GUI Image 1:

However, TraceReplayer cannot identify the correct GUI component in all cases. If we consider the second image, the correct GUI component should be "Departments". However, TraceReplayer incorrectly identifies "Clothing" as the correct GUI component. We cannot even click this component and that component must not be considered. The screenshot is actually considering "Clothing" as the GUI component. But the screenshot is taken from the current screen and therefore, we got an incorrect GUI image. I think one possible solution can be to consider only the current actionable GUI components. What will the best approach to solving this problem?

Augmented Image 2:

GUI Image 2:

@kpmoran please push the data for GnuCash to the repo.

Please push it on data/CrashScope-Data, inside a folder named as "package_name-app_version" like for the other apps there.

Thanks.

bugs: APOD-RB, TIME-CC1, GNU-RC, GNU-CC9

this happens for both apps Mileage and Droit Weight

If I enter "yes" after BURT welcomes me, there is an exception on the server side and nothing happens in the browser (see the pics below).

1. The chatbot should not crash
2. If it crashes, it should show a message indicating there was an internal error
3. for "yes", burt should respond as if the user said "hi" (for now).

Task: Set up a web-based Android emulator and apps (buggy and non buggy)
apps: APOD, TIME, GNU
bugs: APOD-RB, TIME-CC1, GNU-RC, GNU-CC9

Implement report generation based on the conversation.

Initially, the report should look like the Quality Report generated by EULER (see the FSE'19 paper).

The chatbot can only identify welcome message like hi, hello. It should respond to the user messages like "I want to report a bug" or something else

Right now, we are supporting basic utterances for greetings, yes/no, etc.

We should support more (Ideally, we should support more complex language such "I clicked this, and this is the last step"

This is going to be based on the common usage scenarios embedded in the graph.

the input weight 3 is just for reference, not the input from the users

This is not a major issue, but the name of the screenshot files shouldn't contain "gnucash" as seen below.

I mean the hover cursor (aka the hand cursor)

We need to improve the GUI in this way:

• Use a darker background
• Change the icon of the chatbot
• The chatbot conversation box should be wider (not super wide though)
• The chatbot conversation box should reach the bottom of the browser for any screenresolution