A simulation-based implementation of the attentive support robot introduced in the paper
To Help or Not to Help: LLM-based Attentive Support for Human-Robot Group Interactions.
See the project website for an overview.
Prerequisites for building the simulator workspace: g++, cmake, Libxml2, Qt5, qwt, OpenSceneGraph, Bullet Physics
Ubuntu
libxml2-dev, qt5-default, libqwt-qt5-dev, libopenscenegraph-dev, libbullet-devFedora
cmake, gcc-c++, OpenSceneGraph-devel, libxml2, qwt-qt5-devel, bullet-devel, asio-devel, cppzmq-devel, python3-develClone this repo and change into it: git clone [email protected]:HRI-EU/AttentiveSupport.git && cd AttentiveSupport
You can either run the setup script: bash build.sh
or follow these steps:
- Get the submodules:
git submodule update --init --recursive
- Create a
build
directory in the AttentiveSupport directory:mkdir -p build
and change into itcd build
- Install the smile workspace:
cmake ../src/Smile/ -DCMAKE_INSTALL_PREFIX=../install; make -j; make install
Note that if you have the Smile workspace installed somewhere else, you have to change the relative path inconfig.yaml
accordingly. For details, check here - Install the Python dependencies:
python -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt
- Make sure you have an OpenAI API key set up, see the official instructions
- Enjoy ๐น๏ธ
- Run the agent in interactive mode, from the AttentiveSupport directory:
python -i src/tool_agent.py
- Provide commands:
agent.plan_with_functions("Move the red glass to Felix")
- Reset
- The simulation:
agent.sim.reset()
- The agent:
agent.reset()
- The simulation:
- Change the agent's character:
- Either via the
system_prompt
variable ingpt_config
- Or directly, note that this is not persistent:
agent.character = "You are a whiny but helpful robot."
- Either via the
- Provide the agent with further tools:
- Define tools as Python functions in
tools.py
- Make sure to use type hints and add docstrings in the Sphinx notation. This is important so that the
function_analyzer.py
can generate the function descriptions for openai automagically - For inspiration, check out some more examples in
src/tool_variants/extended_tools.py
- Define tools as Python functions in
- Change generic settings such as the model used and its temperature via
gpt_config
- Enable text to speech
- Run the agent
agent.sim.addTTS("native")
agent.sim.callEvent("Start")
Running the simulation with "Move the red glass to Felix"
:
For reproducing the situated interaction scenario run the following:
agent.plan_with_functions("Felix -> Daniel: Hey Daniel, what do we have to drink?")
Robot should do nothing because Daniel is available to answer.agent.plan_with_functions("Daniel -> Felix: We have two options, cola and fanta.")
Robot should correct Daniel.agent.plan_with_functions("Felix -> Daniel: Daniel, please hand me the red glass.")
Robot should help because the red glass is out of reach for Daniel- Manually set Daniel to busy with the mobile:
agent.set_busy("Daniel", "iphone5")
agent.plan_with_functions("Felix -> Daniel: Daniel, could you fill some coca cola into my glass?")
Robot should help as Daniel is busy.agent.plan_with_functions("Daniel -> Felix: Felix, can you give me a full glass of the same, but without sugar?")
Robot should help as Felix cannot see or reach the coke zero.agent.plan_with_functions("Felix -> Robot: What do you know about mixing coke and fanta?")
Robot should answer.