rtr-nettest / open-rmbt

RTR-NetTest Open Source repository

Open-RMBT

Open-RMBT is an open source, multi-threaded bandwidth measurement system.

This is the original, mostly historic repository for RTR-Netztest (RTR-NetTest, aka Open-RMBT). Current code can be found in separate repositories as detailed below.

Open-RMBT consists of the following components:

• Web site +
• JavaScript client
• Android client +
• iOS client +
• Measurement server +
• QoS measurement server (in this repository)
• Control server +
• Statistics server +
• Map server (in this repository, a renewed implementation is under development)
• Desktop app +

+) These components are available in separate repositories. This repository still contains outdated and incompatible versions of these components. They shall not be used for production purposes. Plese contact us if you are uncertain about the compatiblity of repositories.

Open-RMBT is released under the Apache License, Version 2.0. It was developed by the Austrian Regulatory Authority for Broadcasting and Telecommunications (RTR-GmbH).

Related material

• RMBT specification
• RTR-NetTest/rmbt-server - Test Server for conducting measurements based on the RMBT protocol
• RTR-NetTest/rmbtws - JavaScript client for conducting RMBT-based speed measurements
• RTR-NetTest/open-rmbt-control - Control server
• RTR-NetTest/open-rmbt-statistics - Statistics server
• RTR-NetTest/open-rmbt-ios - iOS app
• RTR-NetTest/open-rmbt-android - Android app
• RTR-NetTest/open-rmbt-website - Web site
• RTR-NetTest/open-rmbt-desktop - Desktop app

System requirements

• 1-3 servers

• Everything can be installed on a single server

• The test server (RMBT-Websocket) should run on a physical machine

• Base system Debian 12 or newer (or similar)

• At least a single static public IPv4 address (IPv6 support recommended, more addresses allow to run more services on port 443)

  NOTE: other Linux distributions can also be used, but commands and package names may be different

Installation

For each server:

1. Setup IP/DNS/hostname
2. firewall (e.g. iptables)
3. Install git
4. Install and configure sshd
5. Install and configure ntp
6. dpkg-reconfigure locales (database requires en_US.UTF-8)
7. dpkg-reconfigure tzdata
8. Install and configure letsencrypt

Database Server

1. Install:

   • postgresql-13
   • postgresql-contrib
   • postgresql-13-cron
   • postgis
   • postgresql-13-postgis-3
   • for quantile extension; Install:
     • devscripts
     • sudo
     • postgresql-server-dev-13
     • pgxnclient
     • Run: pgxn install quantile

2. Configure pg_cron, add in /etc/postgresql/13/main/postgresql.conf

   # -- extension pg_cron
   # add to postgresql.conf
   
   # required to load pg_cron background worker on start-up
   shared_preload_libraries = 'pg_cron'
   # optionally, specify the database in which the pg_cron background worker should
   cron.database_name = 'rmbt'
   
   # Schedule jobs via background workers instead of localhost connections
   cron.use_background_workers = on
   

3. Run:

   # Restart database to enable pg_cron
   systemctl restart postgresql
   su - postgres
   # Create database users
   createuser -lSRDP rmbt     # (set password)
   createuser -lSRDP rmbt_control     # (set password)
   createuser -LSRD rmbt_group_control
   createuser -LSRD rmbt_group_read_only
   createuser -LSRD rmbt_group_read_only
   # The following two users are within schema, but not required
   # for basic functionality
   createuser -LSRD kibana # (for export)
   createuser -LSRD nagios # (for monitoring) 
   # Additional users might be required for replication
   echo 'GRANT rmbt_group_read_only TO rmbt_group_control;' | psql
   echo 'GRANT rmbt_group_control TO rmbt_control;' | psql
   
   # Create database
   createdb -O rmbt rmbt 
   
   # import database scheme    
   cat rmbt.sql | psql rmbt -1
   # import basic configuration (modifiy according to your needs)
   cat rmbt_init.sql | psql rmbt -1

   (optional: add additional 3rd party open data, eg. Corine)

4. Edit table "test_server"

   You need to add the test server key to the test_server table.

5. Edit /etc/postgresql/13/main/postgresql.conf to allow remote access for the ControlServer and other servers (change listen_addresses) and configure /etc/postgresql/13/main/pg_hba.conf accordingly

6. Optimise postgres settings

   Check the values of

   • shared_buffers (approx. 25% of available memory)
   • work_mem (approx25% of mem/max_connections)
   • effective_cache_size (approx. 50% of mem)
   • maintenance_work_mem (approx. 5% of mem)
   • max_worker_processes (as number of CPUs)
   • max_parallel_workers_per_gather (as number of CPUs)
   • max_parallel_workers (as number of CPUs)

MapServer

1. Install:

• Apache Tomcat 9 (do not use a higher version)
• nginx (optional, highly recommended)
• openjdk-11-jre (do not use a higher version)
• libservlet3.1-java

2. Edit /etc/tomcat9/context.xml (substitute parts with []), add to <Context>:

   <Context>
   <!-- [...] -->
   <Resource 
      name="jdbc/rmbtro" 
      auth="Container"
      type="javax.sql.DataSource"
      maxActive="200" maxIdle="10" maxWait="10000"
      url="jdbc:postgresql://[db host]/rmbt"
      driverClassName="org.postgresql.Driver"
      username="rmbt" password="[read only pass]"
      description="DB RO Connection" />
   <!-- [...] -->
   </Context>
    

3. Build the server:

   The map server can be built with gradle:

   ./gradlew :RMBTMapServer:war 

   The war file is located in RMBTMapServer/build/lib.

4. Copy RMBTMapServer.war to /var/lib/tomcat9/webapps/

5. Add the package libpostgresql-jdbc-java from Postgresql JDBC and restart tomcat9.

6. Optimize tomcat settings

   Check the values in /etc/default/tomcat9

   • JAVA_OPTS -Xmms MEM -Xmx MEM

Get in Touch

• RTR-Netztest on the web

open-rmbt's Issues

Connection to test server cannot be established

Hello guys, me again :)

Now when my servers are working I am trying to make them work with Android app.

There is no specific manual so I have looked into code and found some possible places where should I put my IP address of Control Server. Possibly defaults.xml in res/values/defaults/ and res/values/xml/preferences_dev.xml. I have changed in defaults value of mapserver to my server IP address and also controlServer to the same server IP (they are all on 1 notebook as solved in my previous issue).
Also I have left empty strings default_control_host_ipv4_only , default_control_host_ipv6_only , default_control_check_ipv4_url and default_control_check_ipv6_url. Is it necessary to fill that strings?

After building app and running it, when I press start button, app showes me error: Connection to test server cannot be established.

Is there any more settings which needs to be set?
What about key needed by database? In databse table test_server I have put to the column "key" some number, but nothing else. Now there is some uuid and my key, should I put there something esle or is it handled by the app?

Client error

I keep receiving an error from the control server during the POST for /RMBTControlServer/settings.

{"error":["Your client version is not supported."]}

I am not looking for individual support. I just want you to give me a hint on what could be causing this problem. Thank you!

(Errors) not working.

I am using postgresql 14.

I receive these errors when trying to import the rmbt.sql into the database.

psql:/home/root_control/open-rmbt/install/postgresql/rmbt.sql:2793: ERROR: type "public.pgis_abs" does not exist
psql:/home/root_control/open-rmbt/install/postgresql/rmbt.sql:2796: ERROR: aggregate public.accum(public.geometry) does not exist
psql:/home/root_control/open-rmbt/install/postgresql/rmbt.sql:2806: ERROR: function public.st_combine_bbox(public.box3d, public.geometry) does not exist
psql:/home/root_control/open-rmbt/install/postgresql/rmbt.sql:2809: ERROR: aggregate public.extent(public.geometry) does not exist
psql:/home/root_control/open-rmbt/install/postgresql/rmbt.sql:2819: ERROR: type "public.pgis_abs" does not exist
psql:/home/root_control/open-rmbt/install/postgresql/rmbt.sql:2822: ERROR: aggregate public.makeline(public.geometry) does not exist
psql:/home/root_control/open-rmbt/install/postgresql/rmbt.sql:2833: ERROR: function array_append(anyarray, anyelement) does not exist
psql:/home/root_control/open-rmbt/install/postgresql/rmbt.sql:2836: ERROR: aggregate public.median(anyelement) does not exist
psql:/home/root_control/open-rmbt/install/postgresql/rmbt.sql:2845: ERROR: function "memcollect" already exists with same argument types
ALTER AGGREGATE
psql:/home/root_control/open-rmbt/install/postgresql/rmbt.sql:2857: ERROR: function public.st_combine_bbox(public.box3d, public.geometry) does not exist
psql:/home/root_control/open-rmbt/install/postgresql/rmbt.sql:2860: ERROR: aggregate public.st_extent3d(public.geometry) does not exist

These errors are hindering A LOT of the functionalities of this repository.

test-server does not accept client token

When the client receive token from control-server and then send to test-server the test-server does not accept the token .
I add secret.key's content as value of RMBT_SECRETKEY parmameter in tomcat8 and run test-server by this command: ./rmbtd -c "server.crt" -k "server.key" -l 443 -w.
the control-server generate a token successfully but the test-server doesn't accept this token.
Can you help me about this?

Client version 0.3 with loop mode

Hello,

I have found weird behaviour using loop mode. I have measured my LTE in my kitchen two days in a row.
Now I have found some blank lines (not filled in database), where the app didnt record any measurments.
Measurement was every 15 minutes. From 207 measuements ( druing 2 days) there are 9 empty ( or missing) rows. There is no patern, one missing is at 02:56 (in the morning) other 13:52, previous day at 14:36 and 22:54 and so on.

But they all have in common different client version. For good measurements client version is 1.2.1 and for these "wrong" measurements is client version 0.3. How can there be different client version?

Also in column status is written "STARTED", the rows contain some more information like time, and some couple informations, but geolocation is missing, IP osf server is missing, no spead or ping measurement. I can provide output in CSV from database if wanted.

nginx

Is it possible to share with us the nginx conf files that netztest.at runs on?
Been having a lot of troubles setting up them.

Thank you

Please add the used protocol ( ipv4/ipv6) to recordings and later to the filters statistics and opendata

https://www.netztest.at/doc/#_measurement_data

The following information is transmitted to the Control Server for each completed test:

• Latency
  

• Uplink data rate
  

• Downlink data rate
  

• Test UUID
  

• Date and time
  

• The number n of concurrent TCP connections actually used
  

  (optionally) the individual tuples (t(j)k,b(j)k)(tk(j),bk(j)) measured during uplink and downlink measurement. In order to keep the data size small, not all tuples might be transmitted. In these cases an exact recalculation of the measured data rates based on the transmitted tuples might not possible.

Note	Further parameters might be collected and transmitted additionally to these information. These additional parameters are out of scope for this specification.

Maybe the used protocol ( IPv4/Ipv6) can be added to statistics and so shown which ISP or Mobile ISP uses already IPv6 in Austria and to which quality

outdated gradle

The files cannot be built with a gradle 7+ version. Please update it. Thank you!

web.xml issue

Wachutka Martin asked me if i could mace a little Testing-Device fort the FiberServiece and the Breitbandbüro, soI tried to compile the CLI-Client myself by cloning the repository, but everything has to be compilable in order for the client to compile.
So I fixed a lot of dependency issues on my system, but now im stuck on an error within the web.xml file.
For the 2 rows it tells me that
"The listener-class element declares a class in the application must be registered as a web application listener bean. The
value is the fully qualified classname of the listener class."
and
"cvc-complex-type.2.4.d: Invalid content was found starting with element 'listener-class'. No child element is expected at this point."

web.txt

Im pretty new to this whole Java-Eclipse Stuff and i would be glad for some help.

public.dhm error

I started receiving an error that the relation public "dhm" does not exist. I noticed that when I ran the "rmbt.sql" file, the table dhm was not created. I tried creating it again but I keep receiving an error the type "public.raster" does not exist. What could be the solution for this problem?

Random incompleted measurements in loop mode

Hello after long time :) Its quarantine time and I finally found some time to come back to this :)

So I tried to make again server like last time. now everything works as last time and I tried to measure LTE in my SIM card. So I ran loop test every 30 minutes and 0 meters GPS range ( since I am nonstop in my flat).
Weird thing is that again like last time I posted in issue, there are some random not finnished tests, I thought it was because of my last setup, but now I dont have VPN like last time so I guess it wasnt that.
I will attach CSV generated from my database(as .txt because github doesnt support csv).

Any idea why there are random not finnished tests?

database.txt

problem with provided rmbt_init.sql

Hi,

Thank you for your support and provide us the needed SQL files.
But now, i get in some troubles with import the rmbt_init.sql file. Below you will se the error messages.

Any suggestions about that syntax error?

br
Bernhard

Guide improvement.

Is there a possibility to give better and more thorough instructions on how to build and run a fully working version?
The given instructions are incomplete and very vague.
Thank you.

cat rmbt_init.sql | psql rmbt -1 error

Hello!

I'm getting an error while running

cat rmbt_init.sql | psql rmbt -1

ERROR:  syntax error at or near "SELECT"
LINE 7: SELECT pg_catalog.setval('public.test_server_uid_seq', 86, t...

this is the last line in the rmbt_init.sql file.

how I can fix it?

Initialization scripts collide with each other.

Hello,
I'm trying to set up my own RMBT servers simply by following instructions provided in readme.MD. In order to set up DB server I have to run two initialization scripts (rmbt.sql & rmbt_init.sql), but they collide with each other (ie. both of them are creating same objects or in rmbt.sql some constrains are created which are violated when rmbt_init.sql tries to insert data). Therefore I'm not able to execute second script.
Any suggestions what should I do? Could you try to dump your DB again? I would like to avoid changing scripts manually.

BR, Filip

Map crash

Hello again me with another issue or actually two separate issues.

---

Edit#1: Looks like server connection problem is only on android 9 and android 10, even in emulator. Android 8 and bellow has no problem to connect to server.

Edit#2: Found solution.
Starting with Android 9 (API level 28), cleartext support is disabled by default.

Thats why it doesnt worked with Android 9 and above. As I wrote in previuos issue in AndroidManifest.xml I had to add to next line after this line android:networkSecurityConfig="@xml/network_security_config
this code android:usesCleartextTraffic="true"

---

The main issue: Everytime I click on Map, the app always crash. It doesnt matter if its in phone or in emultaor, it crashers everytime.

And also I noticed that measured is speed is showing me around 30 Mbit per second but my real speed is 10 times higher, around 300-500 Mbit . I testet original app from RTR-netztest downlaoded from google play and it shows correct values, around 350 Mbit.

Any idea why is it so slow? Or why it isnt showing correct values? Is there some sort of limitation when I have server on my laptop?

Connect ControlServer with Android App

[Question] Hello open-rmbt masters and enthusiasts :)

I want to measure 4G and QoS via open-rmbt android app and store data to my database but I am not that much experienced in this area so I have some questions about how to get Android App working together with server running on my laptop - if it is even possible :).

---

I did follow README.MD guide step by step.
Databse server: I have created everything, added secret key to the table, everything seems to be working here.

---

Control-, Map- and StatisticServer - this is something I am a little confused about. I installed every dependency and stuff and changed /etc/tomcat8/context.xml. I have copied those two Contexts from README.MD and substitued parts with []. Then I built servers and restarted service tomcat8. But after trying to reach server via browser it didnt work. Then I tried delete one of those contexts and in /etc/tomcat8/context.xml is context only for ControlServer. Now when I test it via web browser for example with version of server it returns {"version":"master_822cccf-dirty"}. Well I guess it works as it suppose to work.

Now I am confused where should be located that other context for statistic/map server??

---

I have made my ControlServer public via tunnel and it can be reached from internet, so I have tried to put address to the Android App but app after build and run says it cannot find host.

cp /var/lib/tomcat7/webapps/RMBTControlServer/WEB-INF/lib/postgresql-XXX.jdbcYY.jar /var/lib/postgresql/netztest/RMBTSharedCode/lib/postgresql-XXX.jdbcYY.jar

I have tried to copy the following .jar but I dont have the folder "netztest" in/var/lib/postgresql/I only have there my versoin of my postgres -> /var/libpostgresql/10/ and so on but there is no folder RMBTSharedCode/lib , so should I make that folder or copy it from source code or it should have been there somehow created by commands I didnt use??

---

Thank you in advance for reply and like I said I am noe much experienced. If I am doing somwthing wrong can you point me to right direction ? I would really appreciate some detailed instruction how to make it but I guess every programmer is so too busy to make something like that :D I will be happy even if you tell me what am I doing wrong :) Best regards.

Good

Compiled download

At the moment, it is not possible to download a compiled program, like the console version, which means it's only possible to download the whole project and compile it yourself. At the moment, I am working on getting the Netztest to run on my Raspberry Pi, but I can't compile the files (most likely a problem with the IDE). It would be great if someone could provide me a compiled form of the Linux console application.

Help on running RMBT servers

Hello everyone,

Thank you for building and open-sourcing the program.
For research purpose, I want to add some tests to the RMBT. I am trying to run the control server and the accompanies, but have faced some issues. Would you mind to help to explain some points, I am sorry if these are naive questions:

• In step 2 of DB server, I don't get where the rmbt_init.sql and rmbt.sql files come from.
• Is the RMBT_SECRETKEY mandatory and how could I create it?

Thank you in advance,

Hoang