At the moment, anyone can open SERVER/token . This is a potential security risk.

One possible solution: Make page accessible only to localhost.

If one chat-partner has a blocked "typing area", he can still put his cursor into it, type on his keyboard (nothing happens on the typing area, of course), and the other user see "player is typing".

If one chat-partner has a blocked "typing area", he should not trigger the "is typing" sign on the other users interface.

At the moment, there is no central repository (that I am aware of) of the various plugins that we have written. Should we collect them here, together with their bots? Or use the standard format for experiments? The latter seems a bit overkill, if we just want to document our general, abstract experiment frameworks (rather than actual experiments).

The core functionality is working, but I would like to see minor improvements:

• If a user press "enter" (or in general: send a message), the typing-bit should be set to 0. Because its likely that he stops typing now. If he restarts typing, the bit is set to 1 again anyway.

• If the second user joins my room, he triggers "is typing" even if he do not press anything. This also happens if a user rejoins the chat.

• Make the "is typing" string dynamic: "is typing ." "is typing .." "is typing ..." "is typing ." (Little animation of the dots). This makes it more visible for turker

When using multiple scripts for the same trigger, only the last one is applied.

The syntax of scripts should be extended to accept a list of scripts or a single scripts to keep backwards compatibility

Currently, the Concierge bot uses the markdown plugin.

• Mention this in the docs.

We don't have any tests at the moment. That's bad, because we rely on it to just work!

It would be good if in the near future we could build a test framework bit by bit.

The first step here is unit testing. Later, smoke tests or even integration tests should be feasible.

For terminology, see this answer on Stackoverflow.

@dsg-bielefeld/slurkers

Split up events.py:

• Login-related (connection, tokens, etc.)
• Client-related (message, image, etc.)
  • User specific (set_attribute, set_class, etc)
  • Bot specific (command, task_room, etc.)
• Custom (see below)

Do not pass all information to every command to every client. For example, if we currently ask for a user, we get information about the user, their current room, their previous rooms, and all related information for each room. This can grow pretty fast. Adding some commands to retrieve certain information would make more sense here. However, there would be two disadvantages: It would break compatibility (so it would introduce a major version increase), and it would be a bit more work on the bot and client side. IMO, these are just minor drawbacks and the extra work could be done very elegantly using the async/await syntax.

We may need a plugin system for the server itself. It would be convenient to add arbitrary events to the events.py.

We used to have a message appear when the chat partner was typing (as it is done in Slack), but then removed it, because it was the culprit for increased latency.

It would be very useful to have such a functionality, though, as it would be likely to reduce the amount of overlapping talk between parties.

Hence it would be good to try again, with more event parsimonious implementation this time. Perhaps by moving much of the logic to the client and just sending "has started typing", "has stopped typing" from there?

At the moment, the relative sizes of the areas (display, chat history, entry) is fixed. We should at least have an option where the relative size can be changed by dragging the border. (Many web apps allow this for changing the size of panes.)

With the move of the lab, the name of the organisation will have to change as well. Which means that the URLs change. My understanding is that at least for a while, github will automatically forward requests to the old URL.

But still, things that are under our control should be adapted, if and where we see them. That is, should there be any absolute links in the documentation, they will need to be adapted. (Ideally, to relative URLs, otherwise to contain the new organisation name, once I know it.)

Repos for bots (= one experiment) should follow same format. Maybe integrate slurk as git submodule.

Summary: The startup scripts do not seem to be working correctly for me. Server starts up, as do clients, but clients are then stuck in waiting room, stuck at "waiting for other player".

Steps to reproduce:
In terminal A: python start_server_and_bot.py ../sample_bots/parrot.py
In terminal B: python start_clients.py

Expected results: The client webrowsers open, clients are moved to a room where they can interact, and I can type something / demo an interaction.

Actual results: The clients do open, but they are forever stuck at "Waiting for other player..."

More info:

(py36) bash-3.2$ python start_server_and_bot.py ../sample_bots/turntaking/turntaking_bot.py 
Directory: /Users/das/mbp16_synced/work/svn/Gits/a_Projects/Projects/Infrastructure/slurk/slurk-myfork
Bots:  ['/Users/das/mbp16_synced/work/svn/Gits/a_Projects/Projects/Infrastructure/slurk/slurk-myfork/sample_bots/turntaking/turntaking_bot.py'] 

server started
starting /Users/das/mbp16_synced/work/svn/Gits/a_Projects/Projects/Infrastructure/slurk/slurk-myfork/sample_bots/turntaking/turntaking_bot.py 
token: a16bc3f7-fd30-4376-98e7-2b3f420bb42e
TTBot connected
loading layout from /Users/das/mbp16_synced/work/svn/Gits/a_Projects/Projects/Infrastructure/slurk/slurk-myfork/app/main/../static/layouts/waiting_room.json
TTBot joined room: Waiting Room
message handler error
Traceback (most recent call last):
  File "/Users/das/anaconda2/envs/py36/lib/python3.6/site-packages/engineio/server.py", line 405, in _trigger_event
    return self.handlers[event](*args)
  File "/Users/das/anaconda2/envs/py36/lib/python3.6/site-packages/socketio/server.py", line 518, in _handle_eio_message
    self._handle_connect(sid, pkt.namespace)
  File "/Users/das/anaconda2/envs/py36/lib/python3.6/site-packages/socketio/server.py", line 421, in _handle_connect
    self.environ[sid]) is False:
  File "/Users/das/anaconda2/envs/py36/lib/python3.6/site-packages/socketio/server.py", line 490, in _trigger_event
    return self.handlers[namespace][event](*args)
  File "/Users/das/anaconda2/envs/py36/lib/python3.6/site-packages/flask_socketio/__init__.py", line 243, in _handler
    *args)
  File "/Users/das/anaconda2/envs/py36/lib/python3.6/site-packages/flask_socketio/__init__.py", line 624, in _handle_event
    ret = handler()
  File "/Users/das/anaconda2/envs/py36/lib/python3.6/site-packages/flask_login/utils.py", line 260, in decorated_view
    return current_app.login_manager.unauthorized()
  File "/Users/das/anaconda2/envs/py36/lib/python3.6/site-packages/flask_login/login_manager.py", line 160, in unauthorized
    abort(401)
  File "/Users/das/anaconda2/envs/py36/lib/python3.6/site-packages/werkzeug/exceptions.py", line 707, in abort
    return _aborter(status, *args, **kwargs)
  File "/Users/das/anaconda2/envs/py36/lib/python3.6/site-packages/werkzeug/exceptions.py", line 687, in __call__
    raise self.mapping[code](*args, **kwargs)
werkzeug.exceptions.Unauthorized: 401 Unauthorized: The server could not verify that you are authorized to access the URL requested.  You either supplied the wrong credentials (e.g. a bad password), or your browser doesn't understand how to supply the credentials required.

(That's actually new... Before it started without errors.)

For some use cases (smaller experiments, piloting with a known set of users (e.g., participants in a class), it would be useful to be able to distribute an experiment (= task room type) to several users with the same URL.

When someone clicks on this URL, what should happen is that they get assigned a unique name, get put into the waiting room, and then, when the usual criteria are met (number of participants that are required), they get moved to a pre-determined task room.

In other words, the idea is that the URL does not encode a token that needs to be unique for each participant.

I'm fairly sure this is already possible (by putting a token into the URL that is re-usable? but does that result in new names for everyone who clicks on the link?), but it would be good if it could be documented prominently how to do it.

After the execution of get_admin_token.sh, the environment variable $ADMIN_TOKEN is still empty. Docker seems to be installed correctly and start_slurk_server.sh seems to work as expected ($SLURK_SERVER_ID is not empty after execution).

docker logs $SLURK_SERVER_ID
returns

[2020-09-21 13:51:18 +0000] [1] [INFO] Starting gunicorn 19.9.0
[2020-09-21 13:51:18 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)  
[2020-09-21 13:51:18 +0000] [1] [INFO] Using worker: geventwebsocket.gunicorn.workers.GeventWebSocketWorker  
[2020-09-21 13:51:18 +0000] [9] [INFO] Booting worker with pid: 9  
[2020-09-21 13:51:18 +0000] [9] [ERROR] Exception in worker process  
Traceback (most recent call last):  
  File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()  
  File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/ggevent.py", line 196, in init_process
    self.patch()  
  File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/ggevent.py", line 75, in patch
    fileno=s.sock.fileno()))  
  File "/usr/local/lib/python3.7/site-packages/gevent/_socket3.py", line 114, in __init__
    self._sock = self._gevent_sock_class(family, type, proto, fileno)  
OSError: [Errno 92] Protocol not available: 'protocol'  
[2020-09-21 13:51:18 +0000] [9] [INFO] Worker exiting (pid: 9)  
[2020-09-21 13:51:18 +0000] [1] [INFO] Shutting down: Master  
[2020-09-21 13:51:18 +0000] [1] [INFO] Reason: Worker failed to boot.

This might have something to do with using Ubuntu on Windows Subsystem for Linux, but I am not sure. Many thanks in advance.

Currently, only the default layout is supported. We need a way to modify any content, which is injected by layouts.

The python socket io lib has integrated the socketio_client lib.
https://python-socketio.readthedocs.io/en/latest/client.html

But using the latest version (5.1.0) like in "Client Examples" resolves into an error "Unexpected server response" from slurk. https://python-socketio.readthedocs.io/en/latest/intro.html

import socketio

sio = socketio.Client()

@sio.event
def connect():
    print('connection established')

@sio.event
def my_message(data):
    print('message received with ', data)
    sio.emit('my response', {'response': 'my response'})

@sio.event
def disconnect():
    print('disconnected from server')

sio.connect('http://localhost:5000')
sio.wait()

The problem might be that the message returned by slurk is somehow zero bytes long (thats visible in the logs). Usually the message is at least two bytes long (using socketio_client 0.7.2).

Layouts are a pretty big topic and needs to be added to the documentation.

Extracting from the docs, these things are missing so far:

General

• #7 (solved in #57)

Start page (solved in #53)

• Revise everything below bots.
• Add section on structure of database.
• Group technical stuff (events, commands, database, etc.) under one section.

Quickstart

• Double check. Is that true? Or is there another way to start with a fresh empty test room w/o killing the chat server?
• In the window of the chat bot, you will see some JSON structures that will be explained
• Provide example script that extracts the utterances in a parlAI style representation?

Pairing up participants and running multiple rooms (and bots)

• everything?

Layouts and Plugins

• #14

Tools (solved in #53)

• Add usage instructions for tools added in #44 (Also mention, that the lxml package is needed)

Writing your own bots (solved in #53)

• Disecting the minimal bot
• Disecting the multi bot

What else is missing?

There should be a text file in the top level directory with the names of all contributors! (That is, all of our wonderful HiWis.)

HTML tags passed as plain text in chat as well as display area.

External links in the documentation should open a new tab. Then it is easier for the user to follow the documentation.

Current state: Text in chat bubble is not word wrapped. Long lines are broken up within words.

Should be: Text should be wrapped only at spaces. Use python textwrap or similar.

The right side is way to static. To solve this, we want to introduces layouts. The idea:

• Layouts should be loaded dynamically at entering a Room
• One layout consists of a .json file with a tree of DOM and CSS elements.
• Some DOM elements may be the same as in HTML, some may be different (like a dynamic clock/timer)
• An element can be a node or a link to an external .json file to realize a plugin structure.

These types have to be injected (non-exhaustive):

• DOM tree (done in #2)
• Style sheet (done in #2)
• Scripting (done in #9)

When the pairup bot moves player around, plugins, which relies on a function, are not properly loaded.

Current workaround: put the plugins in the waiting room too 😆

Requirements (non-exhaustive)

• Send audio to server via
  • streaming
  • record and send
• Distribute audio to clients
• Save audio and link it in log files

Unresolved questions

• Which framework makes the most sense for our use-case?
• Which security requirements to we need here?
• Where do we want to safe the audio?
• What legal constraints must we take into account?

When entering chat messages like <script>alert();</script>, this will be executed by any user. This also applies to images, status messages and usernames.

The REST API has not many endpoints, but the GET and POST endpoints already differ in names. They should have the same endpoint name, but only differ in the HTTP method as it is common to design such an API.

For example

GET /api/v2/tokens
POST /api/v2/token

shoud be

GET /api/v2/tokens
POST /api/v2/tokens

the POST request can be interpreted as "add a new token to the collection of tokens". The same has to be done with rooms, tasks and users.

This is no breaking change, because this actually adds new endpoints to the API and the old endpoints might be only marked deprecated.

one docker file that starts nginx, gunicorn, slurk...

derive from that docker container that also starts bot.

Advantage: test deployment environment on your own machine.

The title already is says it. The main purpose of this tool is to enable data collections, so the documentation should talk more about the logs — how to get them, and how to process them. Starting from the QuickStart page.

slurk should log the initial browser window size of clients, and all size change events. This would allow us to detect whether users experienced the game in the way that we want them to.

The current pattern (DATE TIME-NAME NUMBER.log) makes it unnecessarily hard to parse filenames. (Because "-" is in DATE and TIME as well.)

Should be something like DATE-TIME_NAME-NUMBER.log.

It might be good to allow other characters to mark commands. For example, we thought that for a particular setting, marking comands with # instead of / would be more intuitive for people.

(More concretely, in Nazia's setting that would be
A belongs to category 1, because it is green. #answer
instead of
\answer A belongs to category 1, because it is green. Which shows that here maybe the parsing also needs to be different, as the syntax is not "command argument(s)", but rather that the command follwos the "arguments".)

Solution could be to make the set of "command indicator characters" a configurable option?

Our next milestone is accessibility and usability. I can't speak for others, but I find the usability of testing bots quite cumbersome (despite Simeon's tools. Don't get me wrong, it's easier, but still... uncomfortable? :Smile: ).

However, there is one thing that prevents us from using only Slurk from a container (Docker) or Dyno (Heroku): the configuration. With our current system config.ini and config.ini.template we can't just use our container, because we have the same container everywhere, so the same secret key, SSL certificate, host, port, etc. We want configurations per instance. A while ago I postponed #43, now it's time to get back to it, but better!

To summarize the milestone plan:

• We want a central repository with only the server (this repository).
• the server can be hosted anywhere without changing the configuration. (Remember: we can specify different tasks on the same server, the pairup bot is already aware of this feature).
• we want several other repositories with only bots.
• The bot repositories should use the one and only server as it is.

For the last point, the bot must provide its requirements such as rooms that must exist.

Long story short, that leads me to this roadmap, which I want to release on May 17th:

1. refactoring code to allow further steps for the milestone (#67)
2. reevaluate database
3. accept requirements from bots when logging in
4. make server self-contained (#52)
5. split bots into separate repositories with the same format (#45)

Write example of bot that can join a conversation that is managed by openVidu and can do something with the stream. (For now, it should be enough to work with the audio stream.)

E.g., the bot could play a soundfile into the ongoing conversation.

Or, much more challenging (but much closer to our eventual use cases), the bot could direct the audio stream to the google ASR API and send the resulting recognition results back to the (slurk) conversation as text messages.

This is to be the starting point for later work where we may want to hook up a dialogue agent to a conversation that is initiated via slurk and managed (on the webRTC part) by the video bot.

All messages are compared to the list of currently registered commands. If a message could be a misspelling of one of those (e.g., leading slash missing), then a private message is sent to the originator of that message.

E.g., in meetup, someone types "\done". This can be recognised as an attempt to type the command "done" (i.e., to type "/done"). A private message is sent: "Did you mean to type "/done"? Remember that commands can only be recognised if written correctly, with a leading forward slash "/"."

This requires some fuzzy string matching, but should be relatively straightforward. Should only be tested for single word messages, and then go through list of all currently registered command.

(I've used fuzzywuzzy for something like this before: https://github.com/seatgeek/fuzzywuzzy . Might even be overkill here.)

This could be done via the intercept mechanism and on the level of individual bots, but since commands are a server-side thing, and a very general thing, this seems better placed (as an option) in the server.

The HTML tag <audio> has the attribute type.

We need to prefix our custom attributes in order to prevent name clashes.

So maybe this is the best solution: We create another repository, let say slurk-bots, in which we collect all bots that we have created so far (beyond the pairup and minimal bot, which should stay here). These bots should live in a separate directories and be self-contained, and ideally be abstracted away a bit from any specific data that was used for specific experiments with them.

Off the top of my head, I can think of: the meetup bot ( @nilinykh ), the matchup bot ( @nilinykh ), the agreement game bot (@nattari , @atuefekci ), the mouse tracking bot / layout ( @simeonschuez ) , the viewport / zooming on images bot / layout ( @simeonschuez ). These should all be collected here.

My understanding is that to test turntaking_bot.py (and presumably others), one has to edit files that are under version control (in app/static/layouts/). That is not ideal. It should be possible to distribute required layouts together with the bot, so that trying out / demoing a bot really is self-contained and can work just with start_server_and_bots.py.

We want to achieve that a test environment can be created as easily as possible. A test environment will consist at least of slurk and a bot, and possibly additionally also nginx. To keep these components separate, but connected, it might be a good idea to run each in a separate container. However, these containers must be able to have network connections.

It seems to me that docker container is exactly the solution for this. It would allow to specify how the two (or three) containers are started up, and how they get connected.

Currently, external libraries has to be loaded dynamically, which result in an asynchronous call and is hard use in callbacks emitted by SocketIO.

A plugin trigger, which injects a script loading call directly into the header would be very handy.

cc #77

The documentation on emitted events is a bit sparse. In particular, it is very unclear when on_status, joined_room and new_room is emitted (if at all). Furthermore, there are events intended for the server and ones for the client which differ only in word order e.g. "class_add" vs. "add_class".

We should have an overview page with each event and its data structure and the circumenstances when it is triggered.

This is what I could summarize so far:

Slurk listens to

Events sent by layouts

• keypress

  • emits start_typing or stop_typing for each current_user room

• typed_message

  • emits user_message for each current_user room

• text

  • emits text_message for the room where the user typed (contained in the payload)
  • emits stop_typing for each current_user room

• message_command

  • emits command for the room where the user typed (contained in the payload)
  • emits stop_typing for each current_user room

• image

  • emits image_message for the room where the user typed (contained in the payload)
  • emits stop_typing for each current_user room

Events sent by bots programmatically via programmer

• set_attribute

  • emits attribute_update for the room

• set_text

  • emits text_update for the room

• add_class

  • emits class_add for the room

• remove_class

  • emits class_removed for the room

• room_created

  • emits new_room for everyone (broadcast)
  • emits new_task_room for everyone (broadcast) if task in data

• join_room (might also be emitted by the layout)

  • emits joined_room for the room

• leave_room (might also be emitted by the layout)

  • emits left_room for the room

Events sent by bots automatically via socketio framework

• connect
  • emits status for each current_user room
• disconnect
  • emits status for each current_user room
  • emits left_room for each current_user room
• ready
  • emits joined_room for each current_user room

Events sent by socketio

• log
  • emits nothing

Notes

• the intended receiver for most of the events is the javascript client manipulating the layout

• bots might listen to:

  • new_room (possibly as a callback to a rest api call)
  • new_task_room (possibly as a callback to a rest api call)
  • status
  • left_room (possibly to catch user logouts)
  • joined_room (possibly to catch new user logins)
  • user_message
  • text_message
  • command

• there are special messages:

  • starting "@" for private messages to user
  • starting "/" for commands
  • starting "image:" for image messages
  • starting "@ image:" for private image message

Previously it was possible to use HTML tags in chat, but this way it was possible to inject <script> tags.

As we want formatting (at least something like bold, emphasize and line breaks) a markup language should be supported.

One option would be Markdown.

This is needed, to let bots intercept messages

Requirements (non-exhaustive)

• Send video to server via
  • streaming
• Distribute video to clients
• Save video and link it in log files

Unresolved questions

• Which framework makes the most sense for our use-case?
• Which security requirements to we need here?
• Where do we want to safe the video?
• What legal constraints must we take into account?