scrin / ruuvicollector

Utility to collect measurements from RuuviTags and store them in InfluxDB

RuuviCollector

RuuviCollector is an application for collecting sensor measurements from RuuviTags and storing them to InfluxDB. For more about how and for what this is used for, see this post.

Do you have a Ruuvi Gateway? You might be insterested in RuuviBridge instead. You can also use ruuvi-go-gateway if you want to upgrade to the "new stack" without needing a physical Ruuvi Gateway, or want to use a mix of both.

Note: This tool is primarily intended for advanced users, so some knowledge in Linux and Java might be necessary for fully understanding how to use this. However there is a more beginner friendly setup "guide" here

Features

Supports following RuuviTag Data Formats:

• Data Format 2: Eddystone-URL, URL-safe base64 -encoded, kickstarter edition
• Data Format 3: "RAW v1" BLE Manufacturer specific data, all current sensor readings
• Data Format 4: Eddystone-URL, URL-safe base64 -encoded, with tag id
• Data Format 5: "RAW v2" BLE Manufacturer specific data, all current sensor readings + extra

Additionally basic support for iBeacon and Eddystone exists:

• iBeacon: MAC, RSSI and other receiver-side generated data
• Eddystone UID: MAC, RSSI and other receiver-side generated data
• Eddystone TLM: temperature, battery voltage, MAC, RSSI and other receiver-side generated data

Supports following data from the tag (depending on tag firmware):

• Temperature (Celsius)
• Relative humidity (0-100%)
• Air pressure (Pascal)
• Acceleration for X, Y and Z axes (g)
• Battery voltage (Volts)
• TX power (dBm)
• RSSI (Signal strength at the receiver, dBm)
• Movement counter (Running counter incremented each time a motion detection interrupt is received)
• Measurement sequence number (Running counter incremented each time a new measurement is taken on the tag)

Ability to calculate following values in addition to the raw data (the accuracy of these values are approximations):

• Total acceleration (g)
• Absolute humidity (g/m³)
• Dew point (Celsius)
• Equilibrium vapor pressure (Pascal)
• Air density (Accounts for humidity in the air, kg/m³)
• Acceleration angle from X, Y and Z axes (Degrees)

See MEASUREMENTS.md for additional details about the measurements.

Requirements

• Linux-based OS (this application uses the bluez stack for Bluetooth which is not available for Windows for example)
• Bluetooth adapter supporting Bluetooth Low Energy
• bluez and bluez-hcidump at least version 5.41 (For running the application, versions prior to 5.41 have a bug which causes the bluetooth to hang occasionally while doing BLE scanning)
• Maven (For building from sources)
• JDK8 (For building from sources, JRE8 is enough for just running the built JAR)

Building

Execute

mvn clean package

Installation

Automatic Setup

Service scripts and other necessary stuff for "properly installing" this are available in the service-setup directory.

Manual Setup

• hcitool and hcidump require additional capabilities to be run as a normal user, so you need to execute the following commands or run the application as root

sudo setcap 'cap_net_raw,cap_net_admin+eip' `which hcitool`
sudo setcap 'cap_net_raw,cap_net_admin+eip' `which hcidump`

• Run the built JAR-file with java -jar ruuvi-collector-*.jar
  • Note: it's recommended to run this for example inside screen to avoid the application being killed when terminal session ends
• To configure the settings, copy the ruuvi-collector.properties.example to ruuvi-collector.properties and place it in the same directory as the JAR file and edit the file according to your needs.

Configuration

The default configuration which works without a config file assumes InfluxDB is running locally with default settings, with a database called 'ruuvi'. To change the default settings, copy the ruuvi-collector.properties.example file as ruuvi-collector.properties in the same directory as the collector jar file and change the settings you want. Most up-to-date information about the supported configuration options can be found in the ruuvi-collector.properties.example file.

To give human readable friendly names to tags (based on their MAC address), copy the ruuvi-names.properties.example file as ruuvi-names.properties in the same directory as the collector jar file and set the names in this file according to the examples there.

Running

For built version (while in the "root" of the project):

java -jar target/ruuvi-collector-*.jar

Easily compile and run while developing:

mvn compile exec:java

Docker

Dockerized installation is possible with the bundled Dockerfile, which is particularly useful for "server-grade" installations. The Docker image can be built with for example:

docker build -t ruuvi-collector .

Note: if you have configuration files present in the current directory, they will be added to the built image. Alternatively they can be mounted inside the container while running.

Depending on the configuration, it may be necessary to use --net=host (to access the host network stack directly) and/or --privileged (to access a local BLE adapter directly), for example:

docker run --name ruuvi-collector --privileged --net=host -d ruuvi-collector

ruuvicollector's Issues

Exit uncleanly when database connection is lost

Describe the feature you'd like to have implemented
When the database connection is lost, I'd like the software to exit with a non-zero return code.

Describe alternatives you've considered
Instead of exiting, just keep trying to reconnect forever. This was discussed in #8 but the decision was to exit uncleanly and allow the supervisor program (e.g., systemd) to deal with restarting failed services.

The scope of #8 was only if the service could not make the initial database connection (e.g., if the service were started before the database was ready). It does not handle the case where the database connection is established but then later lost.

Do you want to implement this feature?
I'd be happy to, but if someone else wanted to that'd be fine too.

Additional info
When a database connection currently drops, the error message printed is this one:

Uncaught exception while handling measurements from MAC address "00FF00AABBCC", if this repeats and this is not a Ruuvitag, try blacklisting it

This proposed change would have a more specific error message in the event that the database save was what failed.

BLE scanning service

Describe the feature you'd like to have implemented
The core desire here is to ensure that the scanning command is always running. My proposed implementation to achieve this is to run the lescan as a service, rather than being run by RuuviCollector.

I'm having troubles with the scan command (hcitool lescan --duplicates --passive) dying on me. This results in no data coming in, no error messages being printed, and the service does not exit, which makes the recovery process difficult to automate.

Describe alternatives you've considered
The ProcessBuilder.start() method does return a Process object. It's be possible to spin up our own thread that would waitFor() the process to exit and then just immediately restart the process.

I have two objections to this alternative approach:

1. It's more code that would need to be written and maintained in RuuviCollector, which seems inferior to using existing code
2. A philosophical objection that the process supervisor (systemv, systemd, runit should be responsible for making sure processes that are supposed to be running, actually are. To have RuuviCollector take on this task feels like a side quest, whereas it could just delegate this task to a system that was specifically designed for it.

Do you want to implement this feature?
Yes

Additional info
I've already implemented this as a systemd service in my deployment. My question is if I can and how I should contribute this back to the project. I also have a .service file for RuuviCollector that I'd be willing to contribute as well.

Please let me know if these service files should go in a directory by themselves, whether you'd like an install shell script to set this up, if it's OK to update the README with instructions, and so forth. Also, systemd requires an absolute path to the executable, so if there are any suggestions on where that should go (/usr/local/bin?), I'd be happy to incorporate them.

This merge request would not be any changes to any existing code. The user could change their ruuvi-collector.properties file to make the scan command an empty string (or not, it wouldn't really hurt to have two lescans running at once, though it'd be unnecessary) and then set up and start the lescan service.

FYI: Humidity Formulas

FYI ... a good link to humidity formulas is:
https://www.vaisala.com/sites/default/files/documents/Humidity_Conversion_Formulas_B210973EN-F.pdf

You mention that you lost the source of many of the formulas in measurements.md.

Whitelist/Blacklist MAC address filtering on first message

Describe the feature you'd like to have implemented
Today the MAC filtering is performed after the message has been parsed

Describe alternatives you've considered
MAC filtering should be before any subsequent message parsing

Do you want to implement this feature?
Yes

Additional info
This should leverage the existing code to whitelist/blacklist MAC addresses

How to enable changes in ruuvi-names.properties correctly?

Hi!

I edited ruuvi-names.properties with my MAC & new tags and placed it beside the .jar, but they don't seem to change the tag names in grafana from the MAC format? Service start/stop service does not seem to do the job either. Is there rebuild needed or how the configuration should work?

br,
Markus

Feature: HttpDBConnection

A HttpDBConnection that will do a HTTP POST to a configurable http/https url, with the measurement data encoded as a JSON object with the field names as described in https://github.com/Scrin/RuuviCollector/blob/master/MEASUREMENTS.md

Calibration of Temp and Humidity in Ruuvi Collector?

I have a bit more than 20 Ruuvitags. While in the same small container with a calibration pack (salt solution) for a few days they deviate up to around 5-6% relative humidity and about 0.8°C from eachother. The tags closest to the real RH still display 2% below the real relative humidity (using calibration pack with a salt solution and cross checking using other high end devices).

Wouldn't it make sense to have an option in RuuviCollector to calibrate these, by having the possibility to set an offset temp and/or humidity for each device in the config files?

Of course this would have to be updated or checked frequently, but that's not difference than doing that somewhere else. I understand (but didn't test) the RuuviStation mobile apps now have the same feature in the meantime.

Feature request/idea: temperatureOffset.MAC=nn

Hi!

It would be nice to be able to fix temperature readings of a tag by some offset based on a reference temperature. I know that's not fully accurate either because the offset may vary by temperature, but this would be a start, doesn't it?

I can implement it if no one else wants to. :-)

How about a new configuration property: temperatureOffset.MAC=nn ?
I think it's better use MACs here rather than names because names can be changed by tag use but this property is strictly tag hardware -specific.

Cheers,
Anssi Kanninen

Add flexibility to storing just "raw" or "extended" values

I'm working on a branch to allow the user to specify the exact fields they want to store into InfluxDB. This should work the same way as the current whitelisting/blacklisting of MAC addresses: I plan to use the storage.values property, which currently supports the values raw and extended, and to add the new possible values of whitelist and blacklist. The actual list would be provided by some new configuration property, such as storage.values.list.

Feature request/idea: put receiver MAC or other unique identifier as tag

Describe the feature you'd like to have implemented

It would be nice if the database contained information about the receiver (such as a custom string identifier or receiver MAC), in case a person has multiple receivers. Currently many receivers may receiver partially the same data except maybe for the RSSI value (barring manual white/blacklisting).

It would also allow interesting applications like estimating location via means of triangulating the RSSI values (now that it would be known where it was received).

Describe alternatives you've considered

An alternative to this would be using multiple databases, one for each receiver. While doable, I think this would complicate the query part needlessly. Perhaps this data can be optional similarly as other collected values are now configurable.

Another alternative would be to use different names on different hosts. Still, I think a global identifier would be easier.

Do you want to implement this feature?

Yes.

No.

Not until I find myself installing more tags, anyway..

Feature request/idea: InfluxDB2 support

Describe the feature you'd like to have implemented
Support for InfluxDB2
InfluxDB2 has a new java client library, a bit different internal structure with organizations and buckets, and a new connection setup which uses access token instead of username/password.

Describe alternatives you've considered
Alternative is of course continue using v1.x, which works perfectly fine.

Do you want to implement this feature?
I kinda already did

Additional info
I decided to see if I can make a db converter and connector with the new java client library, and it seems to run now. I used as much the same structure as possible with the existing InfluxDBConnector and Converter, and then added new configuration items for v2 support.

I did this only to see if I could still write something, as I haven't really programmed anything in a very long time. Anyway, if this would be useful in anyway, I could for once contribute something back

log addition and revision

It would be helpful to have the log include a startup message including the version in Main
something like
LOG.info("Version: 0.2.4 begins with args:"+args+":");

Additionally, since configuration only occurs at startup perhaps using
LOG.info in config/Config.java (instead of .debug) would be helpful and not clutter logs.

For that matter LOG.info of EVERY getProperty in readConfigFromProperties
LOG.info would not only document properties but also assist in diagnosing incorrect/misspelled property names.

Thanks

Feature request/idea: TagSocket support

Describe the feature you'd like to have implemented
tagSocket support for sharing RuuviTag poll data to multiple applications, ie. RuuviCollector and homebridge-ruuvitag receiving data at the same time.

Describe alternatives you've considered

Do you want to implement this feature?
No

Additional info
https://github.com/klaalo/ifData/tree/master/tagSocket
https://github.com/pakastin/homebridge-ruuvitag

BUG: java.net.UnknownHostException: <domain>

Describe the bug
Jul 25, 2020 12:38:49 AM org.influxdb.impl.BatchProcessor write
SEVERE: Batch could not be sent. Data will be lost
org.influxdb.InfluxDBIOException: java.net.UnknownHostException: darkball.net
at org.influxdb.impl.InfluxDBImpl.execute(InfluxDBImpl.java:800)
at org.influxdb.impl.InfluxDBImpl.write(InfluxDBImpl.java:455)
at org.influxdb.impl.OneShotBatchWriter.write(OneShotBatchWriter.java:22)
at org.influxdb.impl.BatchProcessor.write(BatchProcessor.java:321)
at org.influxdb.impl.BatchProcessor$1.run(BatchProcessor.java:269)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: darkball.net
at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
at java.net.InetAddress.getAllByName(InetAddress.java:1193)
at java.net.InetAddress.getAllByName(InetAddress.java:1127)
at okhttp3.Dns$1.lookup(Dns.java:40)
at okhttp3.internal.connection.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:185)
at okhttp3.internal.connection.RouteSelector.nextProxy(RouteSelector.java:149)
at okhttp3.internal.connection.RouteSelector.next(RouteSelector.java:84)
at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:214)
at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at org.influxdb.impl.BasicAuthInterceptor.intercept(BasicAuthInterceptor.java:22)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at org.influxdb.impl.GzipRequestInterceptor.intercept(GzipRequestInterceptor.java:53)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at okhttp3.logging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.java:144)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200)
at okhttp3.RealCall.execute(RealCall.java:77)
at retrofit2.OkHttpCall.execute(OkHttpCall.java:180)
at org.influxdb.impl.InfluxDBImpl.execute(InfluxDBImpl.java:788)
... 11 more

To Reproduce
It happens randomly

Expected behavior
This error shouldn't happen

Environment
OS and device (ie. Raspbian on RaspberryPI): OctoPi 0.17.0 on raspberry pi 4
Java version (check with java -version):
java version "1.8.0_212"
Java(TM) SE Runtime Environment (build 1.8.0_212-b10)
Java HotSpot(TM) Client VM (build 25.212-b10, mixed mode)

Additional information
Ruuvi-collector is running in screen. InfluxDB and Grafana are hosted on my linode server and not locally on rpi.
RPI4 is connected to the internet through wifi. It is also running octoprint for my 3d printer, that's why octopi distro. It is based on raspbian

Humidity Conversion Formula -broken link -fix provided

I have retrieved Humidity_Conversion_Formulas_B210973EN-F.pdf and put it at

MyBeacons.info/Humidity_Conversion_Formulas_B210973EN-F.pdf

Please retrieve it from there and correct the link at:

https://github.com/Scrin/RuuviCollector/blob/master/MEASUREMENTS.md

BUG: RuuviCollector crashes after couple of hours

Describe the bug
RuuviCollector crashes after couple of hours usage ''connection refused'' Java print this:
sudo systemctl status ruuvicollector.service
● ruuvicollector.service - RuuviCollector Service
Loaded: loaded (/etc/systemd/system/ruuvicollector.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2019-06-18 15:38:35 EEST; 1h 55min ago
Main PID: 466 (java)
CGroup: /system.slice/ruuvicollector.service
├─466 /usr/bin/java -jar /home/pi/ruuvi-collector-0.2.jar
├─728 hcitool lescan --duplicates --passive
└─744 hcidump --raw

Jun 18 15:38:59 raspberrypi java[466]: Caused by: java.net.ConnectException: Connection refused
Jun 18 15:38:59 raspberrypi java[466]: at java.net.PlainSocketImpl.socketConnect(Native Method)
Jun 18 15:38:59 raspberrypi java[466]: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
Jun 18 15:38:59 raspberrypi java[466]: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
Jun 18 15:38:59 raspberrypi java[466]: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
Jun 18 15:38:59 raspberrypi java[466]: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
Jun 18 15:38:59 raspberrypi java[466]: at java.net.Socket.connect(Socket.java:589)
Jun 18 15:38:59 raspberrypi java[466]: at okhttp3.internal.platform.Platform.connectSocket(Platform.java:129)
Jun 18 15:38:59 raspberrypi java[466]: at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:245)
Jun 18 15:38:59 raspberrypi java[466]:

Environment
raspberry 3, Rasbian
java version "1.8.0_65"
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
Java HotSpot(TM) Client VM (build 25.65-b01, mixed mode)

All files should use LF line-endings

Hi,

currently the files in this repository have a mixture of LF and CRLF line-endings. Should all the files have LF line-endings?
Based on the last comment to this pull request PR 10 comment it seems that the desire would be to have only LF line-endings.

There are many instructions about how to convert repository to only have LF line-endings:

• https://help.github.com/en/articles/dealing-with-line-endings
  Contains good documentation but not so good commands to perform the conversion
• https://stackoverflow.com/a/15646791
  Shows working commands for many different git client versions

The simplest way to achieve this is to add .gitattributes file and then running "git add --renormalize ." to convert all files accordingly.

Feature request/idea: ability to define used BT adapter in the config

Describe the feature you'd like to have implemented
Ability to choose the BT adapter used, instead of defaulting to hci0.

Describe alternatives you've considered
I'm running this on an RPI3B+ with an integrated BT adapter to monitor a few Ruuvitags and a couple MiFlora plant sensors. Running 2 BTLE daemons simultaneously on the same adapter causes conflicts, and I added an extra usb BT dongle. I would very much like the ability to choose which adapter each collector daemon is running on, I would guess this would be fairly simple to implement with a config option.

Do you want to implement this feature?
No

Feature request/idea: Insert to two influxUrl's

Describe the feature you'd like to have implemented
By default the collector writes to one influxUrl.

Describe alternatives you've considered
Making RuuviCollector to write two separate influxUrl's would help with displaying collected data on Grafana when ruuvitags are on two different geolocations. RuuvICollector on remote site would write to local influx and also to remote (Grafana-site influx. Local RuuvCollector would write to local influx. Remote RuuviCollector would continue writing to its local influx if the connection to remote influx is lost.

Sadly I don't have skills to make this happen

The collector incorrectly tries to parse certain BLE packets that are similar to those sent by ruuvi

Hi! I got this warning around every 3 second and no data was stored in the Influxdb:
WARN [Main] Uncaught exception while handling measurements java.lang.ArrayIndexOutOfBoundsException: 5 at fi.tkgwf.ruuvi.handler.impl.AbstractEddystoneURL.handleMeasurement(AbstractEddystoneURL.java:74) at fi.tkgwf.ruuvi.handler.impl.AbstractEddystoneURL.read(AbstractEddystoneURL.java:36) at fi.tkgwf.ruuvi.Main.lambda$run$5(Main.java:122) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.LinkedList$LLSpliterator.forEachRemaining(LinkedList.java:1235) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) at fi.tkgwf.ruuvi.Main.run(Main.java:124) at fi.tkgwf.ruuvi.Main.main(Main.java:88)

Do you have any clues? Thanks!

BUG: No data received anymore from Ruuvitags after apt-get update/upgrade

No data received anymore after upgrade of Bluetooth related packages on Raspberry Pi3 Raspbian Buster.

Bluetooth packages were upgraded, history.log:
Start-Date: 2020-04-02 07:59:14
Commandline: apt-get upgrade
Requested-By: pi (1000)
Upgrade: bluez:armhf (5.50-1+rpt1, 5.50-1.2deb10u1+rpt1), bluez-hcidump:armhf (5.50-1+rpt1, 5.50-1.2deb10u1+rpt1), fi
rmware-realtek:armhf (1:20190114-1+rpt4, 1:20190114-1+rpt5), rpi-eeprom-images:armhf (5.0-2, 5.2-1), libicu63:armhf (63
.1-6, 63.1-6+deb10u1), rpi-eeprom:armhf (5.0-2, 5.2-1), firmware-atheros:armhf (1:20190114-1+rpt4, 1:20190114-1+rpt5),
firmware-libertas:armhf (1:20190114-1+rpt4, 1:20190114-1+rpt5), firmware-brcm80211:armhf (1:20190114-1+rpt4, 1:20190114
-1+rpt5), firmware-misc-nonfree:armhf (1:20190114-1+rpt4, 1:20190114-1+rpt5)
End-Date: 2020-04-02 08:01:19

After the upgrade no data has been received from RuuviTags.

Environment
Running Raspberry Pi3b+ Rasbian Buster latest.
Ruuvitag macs are visible in hcitool lescan

Restarting, running disbling/enabling scans through bluetoothctl/hcitool does not seem to help.

Utils.hexToBytes - java.lang.StringIndexOutOfBoundsException

Been running for a few days - then this occured

2019-04-28 18:54:26.448 INFO  [Main] Successfully reading data from hcidump
2019-05-01 11:22:55.300 WARN  [Main] Uncaught exception while handling measurements from MAC address 
"C9B7EB45C0EF", if this repeats and this is not a Ruuvitag, consider blacklisting it
java.lang.StringIndexOutOfBoundsException: String index out of range: 19
	at java.lang.String.charAt(String.java:658)
	at fi.tkgwf.ruuvi.utils.Utils.hexToBytes(Utils.java:33)
	at fi.tkgwf.ruuvi.utils.HCIParser.readLine(HCIParser.java:51)
	at fi.tkgwf.ruuvi.Main.run(Main.java:104)
	at fi.tkgwf.ruuvi.Main.run(Main.java:80)
	at fi.tkgwf.ruuvi.Main.main(Main.java:43)
2019-05-01 11:22:55.330 DEBUG [Main] Offending line: device: disconnected
2019-05-01 11:22:55.342 INFO  [Main] Clean exit

I'm going to try and trap the error - then look at handling input - prefer the service to keep running and log the bad data. Thoughts?

Documentation on stored values

I'd like to see (and possibly contribute to) documentation on the saved values, so I could understand what I'm actually storing. At least the following information would be helpful:

• Do you store latest value from the tag or an average over a sampling period?
• How are the extra values (air density, abs humidity, etc.) calculated (equations)?
• If there are some experimental constants in the equations, where have they been taken from (citation)?

As the quality of this software feels like professional level, why not bring the documentation to the same level? :)

Local development

How are you running locally to test it? (mac or pc?)
Googling/searching around Bluez isnt supported on Mac or PC - have you stored locally some streamed data that your replay, via a test class back into the main class?

Trying to work out how you built this project :)

Feature: Multiple-input-multiple-output

• Support for multiple simultaneous input sources of the same type
• Support for multiple simultaneous outputs of the same type
• Redesign some of the config format to support configuring things like this

Config tests fail if there's a normal config file present in the project root

Steps to reproduce:

• Create config files: ruuvi-collector.properties and ruuvi-names.properties into the project root
• Run tests with mvn tests
• Observe that some of the config related tests will fail

Likely cause is that while running the tests, the collector picks up the "normal" config files rather than the test config files.

Compiling the project in case of failing tests can be worked around with -DskipTests for now, ie.

mvn package -DskipTests

Different limiting strategies

Currently, by default, the app records new measurements every ten seconds. There should of course be some limiting strategy like this, as otherwise one would fill their database too quickly.

However, if you intend to track something that takes less than 10 second, such as some movements, there's a great chance you won't catch the packet(s) where the accelerometer values are interesting.

So I propose that the application should have different methods of limiting the number of saved measurements. Maybe the app could decide to store a measurement after all even if 10 seconds hasn't elapsed yet, provided that some threshold of some sensor values are exceeded. Or maybe it should remember, say, the maximum values of the 10 second period and record those at the end of it.

I volunteer myself for working on this improvement as I've already started on it in my own fork. :)

Retry/Reconnect for InfluxDB

InfluxDB may not be available at all times, especially during startup.
Create retry/reconnect logic to InfluxDBConnection.java

The retry logic should retry a configurable number of times (default maybe 10) before giving up and throwing errors/exiting

Provide better logging on unexpected behavior

Currently RuuviCollector does not handle and/or log certain unexpected output from hcitool/hcidump. These kind of messages should be parsed and logged in a more user-friendly way. Examples:

device: disconnected when the bluetooth adapter is either externally disabled or physically disconnected during operation

Can't attach to device hci0. No such device(19) when trying to run hcidump with no available/enabled bluetooth adapters present

Service scripts & installer

Create service script and something to build a .deb installer package which will install the collector and set up it as a service.
The installer package should have properly marked dependencies (ie. Java 8 or newer).
The installer package should set up a service user for the collector with permissions to run hcitool and hcidump as needed.

RuuviCollector gives up after BT HW error

Every time this appears in dmesg, RuuviCollector stops collecting data: [ke marras 27 20:51:50 2019] Bluetooth: hci0: hardware error 0x43. The process itself does not crash, it is still there but nothing appears in logs about the situation.

BUG: Capabilties are not enough after raspberry pi os 12/2022 updates

Describe the bug
RuuviCollector does not work when run without root privileges after raspberry pi os 12/2022 updates.

To Reproduce

• Make a fresh install with latest updates of raspberry pi os 32-bit
• Install RuuviCollector as per readme
• Run RuuviCollector without root privileges

Expected behavior
RuuviCollector works.

Environment
OS and device (ie. Raspbian on RaspberryPI): Raspberry pi OS (32-bit) & Raspberry Pi ZeroW 2
Java version (check with java -version): openjdk version "17.0.4" 2022-07-19

Additional information
Issue was resolved with adding cap_sys_rawio to capabilities.

Feature: INVALID values should be stored as null

Newest version of ruuvitag_fw (as of Fix #157, 9/3/19, 2.5.5) will be reporting several INVALID values (Actually they indicate UNAVAILABLE sensors).

In addition the humidity will now include an invalid constant (120) when the compensated (i.e. reading adjusted by the correction table equations results in an error
(see https://github.com/ruuvi/ruuvitag_fw/blob/master/drivers/bme280/bme280.c compensate_H_int32 ( where 503316480 is actually 120<<10<<12 i.e. 120% ))

Can these values be (optionally?) treated as null?

Feature request/idea: Offset values to collected measurements

As the Ruuvi tags have some differences in the measured values, it would be good to be able to use some offset values to the data that is stored in the database. For example, when checking for the air pressure difference between tags outdoors and indoors, using normalized values is quite essential.

Feature request/idea: output items from ruuvi-collector.properties,

in ruuvi/config/Config.java/readConfig.java , It would be helpful if item encountered in ruuvi-collector.properties were output to LOG.info.

Crash on Utils.java

I am beginner programmer, but wanting to contribute

Is there anything easy I could do for this project?

Latest official version of Weather Station firmware has different header

The latest version of the Weather Station DFU image has a single byte difference in the header to the sensortag firmware:

    private static final String SENSORTAG_BEGINS = "> 04 3E 25 02 01 03 01 ";

I'm not sure exactly what difference in the DFU image causes this change in the packet, but I've now got a working Grafana dashboard looking at an InfluxDB data source with three tags logging successfully after changing this string.

raw packet storing

I wonder if you would consider creating an option for ruuviCollector (or a totally independent application)
which very simply inserts any messages with the manufacturer ID of 0x0499 (or a list of configurable ID) into an influxDB without interpretation.
An option to blacklist (ignore) packets of certain formats intrepreted elsewhare would be nice.

This would "future proof" the RuuviCollector allowing new formats to be stored without need to modify the collector instead places the responsibility of the interpretation of the message on a subsequent application retrieving the data from the database.

This would also allow custom packets to be stored as created by custom firmware.

Consider :

1. The currently preliminary proposed format 08.
   https://f.ruuvi.com/t/proposal-for-secure-data-format/3756

2. Your own proposal for dynamic data format
   https://f.ruuvi.com/t/proposal-for-secure-data-format/3756

3. A lower resolution format reducing the size most measurement fields allowing for increase in size of count fields like movement counter, additional fields for additional sensors or previous values while still remaining within the advertisement length limit.

4. theBASTI0N proposed alternating PacketA and Packet B

5. Secure Shuffling schedules beacons to change the data that they broadcast (major, minor, MAC-address) on regular basis.
   https://support.kontakt.io/hc/en-gb/articles/206762009

I am considering implementing :
A) A C0 format which reports the configuration to be transmitted every 100 packets.
as discussed at
https://f.ruuvi.com/t/packet-formats-new-tag-status-configuration-packet/3797

B) A 06 format a one byte extension to 05 containing temperature/humidity delta indicators as discussed at https://f.ruuvi.com/t/packet-formats-new-tag-status-configuration-packet/3797

C) A variant of an existing format with solar incidence as reported by a GA1A1s202wp log sensor replacing some combination of atmospheric pressure, batteryVoltage&eTXpower or most significant bytes of MAC (leaving remaining bytes for beacon identification)
https://cdn-learn.adafruit.com/downloads/pdf/adafruit-ga1a12s202-log-scale-analog-light-sensor.pdf

Thank you for your consideration

[Feature request] configurable retention policy

If I understand correctly, the retention policy is currently assumed to be "autogen". I assume this is the InfluxDB java library default since it doesn't seem to be set explicitly.

Following the helpful suggestion in MEASUREMENTS.md, I'm trying to set retention policy and continuous query to automatically downsample data in Influxdb. For this purpose, it would be useful to be able to set retention policy in the configuration file.

Feature request/idea: measurementUpdateLimit.NAME=mmmmm

It might be useful to provide a measurementUpdateLimit on a tag specific basis.

For example tags know to be located where environmental changes are infrequent might have an update limit of 30000ms (every 5 minutes) and others might be better with 10000(10 seconds).

Useful for very long term data samples and very small databases (on feeble processors like RaspberryPi Zero).

influx trouble causes looping

If there is a problem with influxdb (broken configuration, out of space..). RuuviCollector will crash continuously sucking CPU and perhaps filling log. See #8

Please add (something like)

RestartSec=300
to
/lib/systemd/system/ruuvicollector.service
++++++++++++++
When running influxDB on the same machine “requires=influxdb.service after=influxdb.service” should be specified.

PS I consider this as deserving a BUG label as "..Malfunctions that should be fixed to comply with the expected behavior" If influx is not running,

Feature: filter.mode=named

Currently there 3 different filter modes configurable:

# none      = Allows any source to be stored (default)
# blacklist = Allows all sources EXCEPT those listed
# whitelist = Allows ONLY sources that are listed

and a 4th mode should be added called named which will operate like the whitelist, but only allow tags (macs) that have an associated name in ruuvi-names.properties.

It should ignore the filter.macs configuration property (like none does), and it should throw an IllegalStateException if no names are defined when using this filter mode, just like the whitelist does now when using it and no macs are defined.

Feature request/idea: replace deprecated hcitool/hcidump

Describe the feature you'd like to have implemented
Replace deprecated bluetooth utilities with supported versions, maybe bluetoothctl?

Describe alternatives you've considered
N/A

Do you want to implement this feature?
No

Additional info
hcitool/hcidump deprecated since 2017
https://git.kernel.org/pub/scm/bluetooth/bluez.git/commit/?id=b1eb2c4cd057624312e0412f6c4be000f7fc3617

Running on OS X?

It seems this Ruuvi collector app can't be run on OS X due to lack of Bluez on OS X? Right?

BUG:named-filter does not recognise ruuvi-names.properties

Describe the bug

named-filter does not recognise ruuvi-names.properties

To Reproduce
Configure ruuvi-names.properties:
miha@ruuvitag:~/RuuviCollector/target$ tail -2 ruuvi-names.properties
EA:D6:63:18:F3:82=Fridge
C7:60:16:8C:BB:4F=Freezer

Configure ruuvi-collector.properties:
#filter.mode=none
filter.mode=named

filter.macs=Fridge,Freezer

Expected behavior
Collector should send data to remote influxd-machine. That does not happen (followed with netstat -nputwc) However, with filter.mode=none and filer.macs commented out, collector works.

Environment
OS and device (ie. Raspbian on RaspberryPI):
NanoPI Neo AIr
Java version (check with java -version):
openjdk version "1.8.0_232"
OpenJDK Runtime Environment (build 1.8.0_232-8u232-b09-1~deb9u1-b09)
OpenJDK Client VM (build 25.232-b09, mixed mode)

Additional information
Any other information that may help in diagnosing this issue
miha@ruuvitag:/RuuviCollector/target$ pwd
/home/miha/RuuviCollector/target
miha@ruuvitag:/RuuviCollector/target$ ls -la
total 4864
drwxr-xr-x 9 miha miha 4096 May 9 14:42 .
drwxr-xr-x 6 miha miha 4096 May 9 09:29 ..
drwxr-xr-x 3 miha miha 4096 May 9 09:29 classes
drwxr-xr-x 3 miha miha 4096 May 9 09:29 generated-sources
drwxr-xr-x 3 miha miha 4096 May 9 09:29 generated-test-sources
drwxr-xr-x 2 miha miha 4096 May 9 09:29 maven-archiver
drwxr-xr-x 3 miha miha 4096 May 9 09:29 maven-status
-rw-r--r-- 1 miha miha 53771 May 9 09:29 original-ruuvi-collector-0.2.jar
-rw-r--r-- 1 miha miha 4867034 May 9 09:29 ruuvi-collector-0.2.jar
-rw-r--r-- 1 miha miha 5328 May 9 14:41 ruuvi-collector.properties
-rw-r--r-- 1 miha miha 479 May 9 14:42 ruuvi-names.properties
drwxr-xr-x 2 miha miha 4096 May 9 09:29 surefire-reports
drwxr-xr-x 3 miha miha 4096 May 9 09:29 test-classes
miha@ruuvitag:/RuuviCollector/target$ tail -2 ruuvi-names.properties
EA:D6:63:18:F3:82=Fridge
C7:60:16:8C:BB:4F=Freezer
miha@ruuvitag:/RuuviCollector/target$ grep named ruuvi-collector.properties
filter.mode=named
miha@ruuvitag:/RuuviCollector/target$ sudo java -jar -Xdebug ruuvi-collector-0.2.jar
2020-05-09 14:48:09.184 DEBUG [Config] Tag names: /home/miha/RuuviCollector/target/ruuvi-names.properties
2020-05-09 14:48:09.217 DEBUG [Config] Config: /home/miha/RuuviCollector/target/ruuvi-collector.properties
Exception in thread "main" java.lang.ExceptionInInitializerError
at fi.tkgwf.ruuvi.Main.startHciListeners(Main.java:41)
at fi.tkgwf.ruuvi.Main.run(Main.java:63)
at fi.tkgwf.ruuvi.Main.main(Main.java:30)
Caused by: java.lang.IllegalStateException: You have set filter.mode=named but left ruuvi-names.properties empty. Please select a different filter.mode value or populate ruuvi-names.properties.
at fi.tkgwf.ruuvi.config.Config.parseFilterMode(Config.java:260)
at fi.tkgwf.ruuvi.config.Config.readConfigFromProperties(Config.java:145)
at fi.tkgwf.ruuvi.config.Config.readConfig(Config.java:127)
at fi.tkgwf.ruuvi.config.Config.reload(Config.java:88)
at fi.tkgwf.ruuvi.config.Config.reload(Config.java:81)
at fi.tkgwf.ruuvi.config.Config.(Config.java:77)
... 3 more
miha@ruuvitag:~/RuuviCollector/target$