bellrichm / weewx-mqttsubscribe Goto Github PK

View Code? Open in Web Editor NEW

49.0 49.0 13.0 1.23 MB

A WeeWX service and driver that receives data from MQTT.

License: GNU General Public License v3.0

Python 95.63% Shell 4.37%

mqtt python weewx

weewx-mqttsubscribe's People

Contributors

Stargazers

Watchers

Forkers

tkotz8105 joepfeiffer czechmark tixel1 nickr01 nikallian ss109 edi-x derekk19 nifoc hoetzgit lloydr jlrgraham

weewx-mqttsubscribe's Issues

global name 'settingd' is not defined

Hello,
When i execute MQTTSubscribe_install_type=DRIVER wee_extension --install=v1.2.0.tar.gz
it is fine but after wee_config --reconfig
and picking the installed drivers include: 0 for MQTTSubscibeDriver
i get this Traceback:

File "/usr/bin/wee_config", line 121, in <module>
  main()
File "/usr/bin/wee_config", line 115, in main
  config_mgr.run(args, options)
File "/usr/share/weewx/weecfg/config.py", line 103, in run
  self.modify_config(config_dict, options)
File "/usr/share/weewx/weecfg/config.py", line 119, in modify_config
  stn_info = self.get_stn_info(config_dict, options)
File "/usr/share/weewx/weecfg/config.py", line 150, in get_stn_info
  stn_info.update(weecfg.prompt_for_driver_settings(driver, config_dict))
File "/usr/share/weewx/weecfg/__init__.py", line 1438, in prompt_for_driver_settings
  settings[driver_module.DRIVER_NAME] = editor.prompt_for_settings()
File "/usr/share/weewx/user/MQTTSubscribe.py", line 823, in prompt_for_settings
  settingd['topics'] = {}

Im sorry if thats no issue but would be nice if you can help me.

Handle wind data in separate topics

Wind data needs to be collected into a single packet. The code currently does this if all the data is in a single topic.
See, #58 for additional information.

Options:

Move the Collector outside of the get data method.

This seems simple and straight forward.
This will result in the get data having a somewhat strange side effect of populating and emptying the Collector.
It probably won’t robustly solve the problem because the topics will be processed sequentially. So all windDir data will be processed, then windSpeed, etc.

Queue up the wind data in a separate queue.

Wind data will be queued as it comes in. So it will be processed independently of original topics.
How should units be handled? To be robust, probably need units at the “field level”.
Still need to do some type of collection when getting the data.

When getting data, put wind data into a new queue.

It probably won’t robustly solve the problem because the topics will be processed sequentially. So all windDir data will be processed, then windSpeed, etc.

Improve the Collector to just collect and then separate method at end to output.

Still somewhat strange interaction wit get data method, but better.

In indivdual mode, handle same "field names" in different topics

For example, Indoor/temp = inTemp and Outdoor/temp = outTemp
For additional information see,
https://groups.google.com/d/msg/weewx-user/RpHzL70Svlw/92-1VGvrDAAJ

Documentation says unit_system_name should be unit_system

when running standalone allow [[message_callback]]/type to be overriden

Continue to improve MQTT payload handling

Doing some experimenting with the keyword payload handing. Running weewx under python 3 it is a byte array when running my pubmqtt.py utility.

In this issue I will fix the keyword handling. Probably need to do deep dive at somepoint

Here is a snippet of code
https://github.com/eclipse/paho.mqtt.python/blob/master/src/paho/mqtt/client.py

    if isinstance(payload, unicode):
        local_payload = payload.encode('utf-8')
    elif isinstance(payload, (bytes, bytearray)):
        local_payload = payload
    elif isinstance(payload, (int, float)):
        local_payload = str(payload).encode('ascii')
    elif payload is None:
        local_payload = b''
    else:
        raise TypeError(
            'payload must be a string, bytearray, int, float or None.')

Move the overlap option into the queue management

With #24 and #25, various ‘manipulations’ of datetimes are done while retrieving data from the queue. For consistency, it makes sense to move the overlap processing into this code.

Replace optparse with argparse

optparse has been deprecated.

MQTTSubscribe value error: start time is greater than stop time

I installed MQTTSubscribe a couple of days ago. It is running in weewx 3.9.2 with weatherflowudp and mqtt. Weewx has shut down a few times today with this type of error:

Jun 20 09:32:42 MITX-6930 weewx[14061]: MQTTSubscribeService: Packet after update is: 2019-06-20 09:32:39 PDT (1561048359) dateTime: 1561048359, usUnits: 17, windDir: 0, windSpeed: 0.0
Jun 20 09:32:42 MITX-6930 weewx[14061]: restx: MQTT: Published record 2019-06-20 09:32:39 PDT (1561048359)
Jun 20 09:32:45 MITX-6930 weewx[14061]: MQTTSubscribeService: Packet prior to update is: 2019-06-20 09:32:42 PDT (1561048362) dateTime: 1561048362, usUnits: 17, windDir: 0, windSpeed: 0.0
Jun 20 09:32:45 MITX-6930 weewx[14061]: MQTTSubscribeService: Processing interval: 1561048359.000000 1561048362.000000
Jun 20 09:32:45 MITX-6930 weewx[14061]: MQTTSubscribe: TopicManager queue size is: 0
Jun 20 09:32:45 MITX-6930 weewx[14061]: MQTTSubscribeService: Queue was empty
Jun 20 09:32:45 MITX-6930 weewx[14061]: MQTTSubscribeService: Packet after update is: 2019-06-20 09:32:42 PDT (1561048362) dateTime: 1561048362, usUnits: 17, windDir: 0, windSpeed: 0.0
Jun 20 09:32:45 MITX-6930 weewx[14061]: restx: MQTT: Published record 2019-06-20 09:32:42 PDT (1561048362)
Jun 20 09:32:45 MITX-6930 weewx[14061]: MQTTSubscribeService: Packet prior to update is: 2019-06-20 09:32:17 PDT (1561048337) dateTime: 1561048337, illuminance: 40134, precipitationType: 0, radiation: 334, rain: 0.0, skyBatteryVoltage: 3.37, usUnits: 17, UV: 3.77, windBatteryStatus: 3.37, windLull: 0.0
Jun 20 09:32:45 MITX-6930 weewx[14061]: MQTTSubscribeService: Processing interval: 1561048362.000000 1561048337.000000
Jun 20 09:32:45 MITX-6930 weewx[14061]: engine: Main loop exiting. Shutting engine down.
Jun 20 09:32:45 MITX-6930 weewx[14061]: engine: Shutting down StdReport thread
Jun 20 09:32:45 MITX-6930 weewx[14061]: engine: StdReport thread has been terminated
Jun 20 09:32:45 MITX-6930 weewx[14061]: restx: Shut down MQTT thread.
Jun 20 09:32:45 MITX-6930 weewx[14061]: restx: Shut down Wunderground-RF thread.
Jun 20 09:32:45 MITX-6930 weewx[14061]: MQTTSubscribe: Disconnected with result code 0
Jun 20 09:32:45 MITX-6930 weewx[14061]: engine: Caught unrecoverable exception in engine:
Jun 20 09:32:45 MITX-6930 weewx[14061]:     ****  start time (1561048362) is greater than stop time (1561048337)
Jun 20 09:32:45 MITX-6930 weewx[14061]:     ****  Traceback (most recent call last):
Jun 20 09:32:45 MITX-6930 weewx[14061]:     ****    File "/usr/share/weewx/weewx/engine.py", line 890, in main
Jun 20 09:32:45 MITX-6930 weewx[14061]:     ****      engine.run()
Jun 20 09:32:45 MITX-6930 weewx[14061]:     ****    File "/usr/share/weewx/weewx/engine.py", line 191, in run
Jun 20 09:32:45 MITX-6930 weewx[14061]:     ****      self.dispatchEvent(weewx.Event(weewx.NEW_LOOP_PACKET, packet=packet))
Jun 20 09:32:45 MITX-6930 weewx[14061]:     ****    File "/usr/share/weewx/weewx/engine.py", line 224, in dispatchEvent
Jun 20 09:32:45 MITX-6930 weewx[14061]:     ****      callback(event)
Jun 20 09:32:45 MITX-6930 weewx[14061]:     ****    File "/usr/share/weewx/user/MQTTSubscribe.py", line 609, in new_loop_packet
Jun 20 09:32:45 MITX-6930 weewx[14061]:     ****      target_data = self.subscriber.get_accumulated_data(topic, start_ts, self.end_ts, event.packet['usUnits'])
Jun 20 09:32:45 MITX-6930 weewx[14061]:     ****    File "/usr/share/weewx/user/MQTTSubscribe.py", line 529, in get_accumulated_data
Jun 20 09:32:45 MITX-6930 weewx[14061]:     ****      return self.manager.get_accumulated_data(topic, start_ts, end_ts, units)
Jun 20 09:32:45 MITX-6930 weewx[14061]:     ****    File "/usr/share/weewx/user/MQTTSubscribe.py", line 277, in get_accumulated_data
Jun 20 09:32:45 MITX-6930 weewx[14061]:     ****      accumulator = weewx.accum.Accum(weeutil.weeutil.TimeSpan(start_ts, end_ts))
Jun 20 09:32:45 MITX-6930 weewx[14061]:     ****    File "/usr/share/weewx/weeutil/weeutil.py", line 256, in __new__
Jun 20 09:32:45 MITX-6930 weewx[14061]:     ****      raise ValueError("start time (%d) is greater than stop time (%d)" % (args[0], args[1]))
Jun 20 09:32:45 MITX-6930 weewx[14061]:     ****  ValueError: start time (1561048362) is greater than stop time (1561048337)
Jun 20 09:32:45 MITX-6930 weewx[14061]:     ****  Exiting.

I have added more debugging to syslog and hope to get more info. Previous to installing MQTTSubscribe, weewx was accepting that data without issue from weatherflowudp.

Support subscribing to multiple topics

In individual mode, support subscribing to multiple topics and map to fieldnames.

Support keyword payload

Now that the unit test environment is in place, the rest of this can be developed.

Investigate if the service really handles MQTT payload with odd intervals and partial data

Investigate unicode/str and the _bytefy routine in python 3 and 2

Add a test when processing json that the keys are not unicode (or are str)

Unit tests should ensure that the data dictionary has data types that are supported by WeeWX. Currently I caught this via the smoke tests, which is awkward at best.

“Accumulate” the MQTT messages into archive data

When processing the queue elements, accumulate the data - like software generations. Aggregates, averages, last value, mins, maxes, etc.

Clean up encoding of MQTT message topics when processing individual payload

Currently encoding under both python 2 and 3 from what I learned in #33, probably only need to encode in python 2 (because the data is unicode). Also for consistency, should encode to utf-8 like in the json payload.

Update Frequency

Now that I've fixed #63, I have an issue with my air quality sensor. It publishes every 10 minutes, whereas all my other sensors do so more frequently, which means they show up as N/A on the reporting when generated.

In other contexts with MQTT, there is software that has an expires_after parameter, that will retain the value for x number of seconds/minutes/etc. Would this be possible here?

Support WeeWX StdCalibrate

For software generation, the calibrate is only done on the loop - so it is not done twice
Possible solutions:
Create loop packets when it is software generation.
Perform the calibration prior to augmenting the archive record.
Assume that the data is always correct in the MQTT payload.

Unit tests

Nested Json

Hi Rich,
I'm using MQTTSubcribe now in a 'keyword' configuration. Works great, thanks!
I would like to use it with nested Json messages (Json -not nested- works fine).
But i'm struggeling with the configuration of the nested fields.

What I would like to do:
Input:
{ ..
"gas": {
"totalUse": 2352.545,
"ts": 1588096800
},
..
}

Requested mapping (from MQTT to loop)
gas_totalUse -> gas
gas_ts -> TsGas

I have this mapping:

    [[[fields]]]
        [[[[gas_totalUse]]]]
        name = gas
        [[[[gas_ts]]]]
        name = TsGas
        conversion_type = int

But I don't get is to work.
mqtt-log1.txt

Regards,
Jaap

Optionally convert json payload to float

Some json has the values as strings instead of numbers. These need to be converted to floats. Add an option convert_to_float. Default will be False.

Add option when running standalone to print out the version

When running as a driver, it defaults the archive interval to 300

The undocumented config option, archive_interval has a default of 300. This means that if the user sets the option at the WeeWX level, it will still use the MQTTSubscribe interval.
I think the same logic that is used in genArchiveRecords needs to be implemented in the archive_interval property. Meaning that if this no archive_topic set, NotImplementedError should be raised.

Develop installer

Need to handle installing one or both.

In individual mode, handle multi-level topics

Should handle something like
topic1/.../topicN/fieldName

JSON value pair mapping

I have been adding sensors to WeeWX via WeeWX-MQTTSubscribe and previously I have been republishing the topics to "conform" to existing WeeWX database fields. I am dealing with topics like this:

{
  "Time": "2019-07-11T10:00:38",
  "BME280": {
    "Temperature": 70.6,
    "Humidity": 61.4,
    "Pressure": 1013.1
  },
  "ADS1115": {
    "A0": 135.78
    "Unit": "PSI"
  },
  "SI1145": {
    "Illuminance": 262,
    "Infrared": 281,
    "UvIndex": 0.3
  },
  "PressureUnit": "hPa",
  "TempUnit": "F"
}

I would like to be able to use this time stamp value pair format directly. The second issue is mapping these value pairs to the WeeWX database fields. What I'm thinking is adding something like this to weewx.conf:

[living_room_temperature] = [BME280.Temperature]
[living_room_humidity] = [BME280.Humidity]
[living_room_barometer] = ([BME280.Pressure] * 0.02953) + 0.2 
[pump_water_pressure] = [ADS1115.A0]
[living_room_illuminance] = [SI1145.Illuminance]
[living_room_infrared] = [SI1145.Infrared]
[living_room_UV] = [SI1145.UvIndex]

Thanks,
Patrick

Run service “standalone”

For development and debugging, it would be nice to run the service “standalone”

Develop a functional/end to end testbed

I have a set of unit tests, but as issue #44 shows, it is not enough. Current thought is to test the class MQTTSubscribe. Specifically the get_data and get_accumulated_data methods. I believe if I capture the on_message callback, I could call it with known data and skip the MQTT processing.

Capturing it would look something like this

self.on_message = message_callback_provider.get_callback() # This is new
self.client.on_message = self.on_message

Then something like this should be possible

SUT = MQTTSubscribe(config, logger)
SUT.on_message((client, userdata, msg)

And then this

data = SUT.get_accumulated_data(topic, start_ts, end_ts, units)
Perform test

Or this

for data in SUT.get_data(topic):
    Capture data
Perform test

Trying to convert properties I did not specify in the fields parameter

I'm using a payload as described in #28.

ERROR user.MQTTSubscribe: MessageCallbackProvider on_message_json failed with: could not convert string to float: 'hPa'
ERROR user.MQTTSubscribe: **** MessageCallbackProvider Ignoring topic=tele/barometer/SENSOR and payload=b'{"Time":1588297297,"BME280":{"Temperature":24.3,"Humidity":38.9,"DewPoint":9.4,"Pressure":1007.1},"PressureUnit":"hPa","TempUnit":"C"}'

I understand it wants to convert fields to floats, but I only care about the fields in the BME280 nested JSON, the temperature units don't need to be converted. Nothing I can see in the documentation says how to tell it to ignore certain properties in the JSON.

I was previously using this plugin and a custom script that just sent 'barometer' as a topic by itself, but I switched to this embedded Tasmota idea to make things easier and allow me to deploy more sensors in an easier manner.

unit scale / conversion

loving this driver BTW

unless I am doing something stupid , I cant see a way to do unit conversion / scaling when processing incoming data

I have mastered the labels, and that is working great.

but would like to convert the scale of units during processing
using RTL_433 the bresser 5 in 1 outputs

'rain' is mm, weewx wants cm ( * 0.1)
'windSpeed' is m/s, weewx wants km/h (* 3.6)
'windGust' is m/s, weewx wants km/h (* 3.6)

Wind data is not being ‘collected’ across MQTT payloads

From my understanding of how WeeWX handles wind data, the loop packet has to have the windSpeed along with the windDir. If it doesn’t, the windDir will be ignored.
Code was developed to collect the wind data across multiple MQTT payloads into a single loop packet. But due to a bug, it was never called.
In addition, the code attempted to handle multiple data fields in a MQTT payload. The code has been simplified to only work with ‘individual’ payloads, with the assumption that payloads with multiple fields should have the necessary data in the payload.

Add option to bind to loop packet instead of archive record

I think I could bind to loop packet. When a new loop packet arrives, just generate loop packets from every queue element.
Note, this would software generation to then process the loop packets into archive data. But, would make it available to other services.

Write a small utility that publishes MQTT messages

This could be useful to debug specific payloads.

Support only one topic that is passed in s a parameter.
Read messages from a file. Each line is a message.
Support passing in an interval between publishing or prompt for next message.
Perhaps support a range of random time intervals between messages.
Support adding current date/time to message.

Support additional MQTT payload formats

Add a config option that will change the on_message call back function.
Have different functions for the different payload formats.

Support datetime formats other than unix epoch

Splitting this out from issue #28.

When running standalone allow topics to be overridden via command line

See mqtt_test.py as an example

Handle cumulative data

Some data may come in as cumulative instead of increments,

Improved documentation

What is the difference between a driver and a service?
why would I choose one over the other?

How can I troubleshoot the installation when no data appears in the database?

Add an option to ignore the datetime in the MQTT payload and set it to the weewx server time

This is for the service, just need to make sure to not break the driver.

Add an option to ignore the MQTT payload datetime and process the queue until it is empty

This is for the service, just need to make sure to not break the driver.
Also, if data comes in very frequently from MQTT, there is a chance that we could get stuck in a loop just pulling data from MQTT.

Create a driver

With some refactoring, should be easy to have a driver

Create a Logger class

Wrap the logging in a Logger class. Pass an instance into the callback provider. This will hide the logging details from the callback providers.
Note, the tests will need to be updated to pass in a mock logger.

Python 3 support

Unit Conversion - Some sensors report in US and some in Metric within the same topic

I have an Acurite 5in1 1036. I'm using rtl_433 to publish to MQTT. The Acurite sends data in both US and Metric formats. The unit_system seems to be global. Can you provide functionality to select the unit_system per field?

[[message_callback]]
        type = individual
    #    keyword_separator = :
        
        [[[fields]]]
            [[[[wind_dir_deg]]]]
                name = windDir
                conversion_type = int
            [[[[wind_avg_km_h]]]]
                name = windSpeed
            [[[[temperature_F]]]]
                name = outTemp
            [[[[humidity]]]]
                name = outHumidity
            [[[[battery_ok]]]]
                name = rainBatteryStatus
                conversion_type = bool
            [[[[rain_in]]]]
                name = rain
                contains_total = True
    
    # The topics to subscribe to.
    [[topics]]
        unit_system = US
        [[[sensors/rtl_433/Acurite-5n1/1011/temperature_F]]]
        [[[sensors/rtl_433/Acurite-5n1/1011/wind_dir_deg]]]
        [[[sensors/rtl_433/Acurite-5n1/1011/wind_avg_km_h]]]
        [[[sensors/rtl_433/Acurite-5n1/1011/humidity]]]
        [[[sensors/rtl_433/Acurite-5n1/1011/battery_ok]]]
        [[[sensors/rtl_433/Acurite-5n1/1011/rain_in]]]

Here is a sample of weewxd output:

LOOP:   2020-04-25 19:30:40 UTC (1587843040) battery_ok: 1.0, dateTime: 1587843040.9, maxSolarRad: None, rain_in: 2.4, rainRate: 0, usUnits: 1, wind_avg_km_h: 10.934, wind_dir_deg: 225.0, windDir: 225
LOOP:   2020-04-25 19:30:40 UTC (1587843040) dateTime: 1587843040.91, maxSolarRad: None, rainRate: 0, usUnits: 1, windDir: 225
LOOP:   2020-04-25 19:30:40 UTC (1587843040) battery_ok: 1.0, dateTime: 1587843040.91, maxSolarRad: None, rain_in: 2.4, rainRate: 0, usUnits: 1, wind_avg_km_h: 10.934, wind_dir_deg: 225.0, windDir: 225
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843040.902822 which is prior to previous packet 1587843040.912873
LOOP:   2020-04-25 19:30:40 UTC (1587843040) dateTime: 1587843040.9, maxSolarRad: None, rainRate: 0, usUnits: 1, windSpeed: 10.934
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843040.906305 which is prior to previous packet 1587843040.912873
LOOP:   2020-04-25 19:30:40 UTC (1587843040) dateTime: 1587843040.91, maxSolarRad: None, rainRate: 0, usUnits: 1, windSpeed: 10.934
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843040.911563 which is prior to previous packet 1587843040.912873
LOOP:   2020-04-25 19:30:40 UTC (1587843040) dateTime: 1587843040.91, maxSolarRad: None, rainRate: 0, usUnits: 1, windSpeed: 10.934
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843040.901169 which is prior to previous packet 1587843040.912873
LOOP:   2020-04-25 19:30:40 UTC (1587843040) dateTime: 1587843040.9, maxSolarRad: None, rainBatteryStatus: True, rainRate: 0, usUnits: 1
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843040.905568 which is prior to previous packet 1587843040.912873
LOOP:   2020-04-25 19:30:40 UTC (1587843040) dateTime: 1587843040.91, maxSolarRad: None, rainBatteryStatus: True, rainRate: 0, usUnits: 1
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843040.910828 which is prior to previous packet 1587843040.912873
LOOP:   2020-04-25 19:30:40 UTC (1587843040) dateTime: 1587843040.91, maxSolarRad: None, rainBatteryStatus: True, rainRate: 0, usUnits: 1
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843040.904643 which is prior to previous packet 1587843040.912873
LOOP:   2020-04-25 19:30:40 UTC (1587843040) dateTime: 1587843040.9, maxSolarRad: None, rain: 0.0, rainRate: 0, usUnits: 1
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843040.909670 which is prior to previous packet 1587843040.912873
LOOP:   2020-04-25 19:30:40 UTC (1587843040) dateTime: 1587843040.91, maxSolarRad: None, rain: 0.0, rainRate: 0, usUnits: 1
LOOP:   2020-04-25 19:30:40 UTC (1587843040) battery_ok: 1.0, dateTime: 1587843040.91, maxSolarRad: None, rain: 0.0, rainRate: 0, usUnits: 1, wind_avg_km_h: 10.934
LOOP:   2020-04-25 19:30:59 UTC (1587843059) battery_ok: 1.0, dateTime: 1587843059.25, maxSolarRad: None, outTemp: 85.8, rain_in: 2.4, rainRate: 0, temperature_F: 85.8, usUnits: 1, wind_avg_km_h: 8.45, wind_dir_deg: 225.0
LOOP:   2020-04-25 19:30:59 UTC (1587843059) battery_ok: 1.0, dateTime: 1587843059.26, humidity: 70.0, maxSolarRad: None, outTemp: 85.8, rainRate: 0, temperature_F: 85.8, usUnits: 1, wind_avg_km_h: 8.45
LOOP:   2020-04-25 19:30:59 UTC (1587843059) dateTime: 1587843059.26, maxSolarRad: None, outTemp: 85.8, rainRate: 0, usUnits: 1
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843059.247852 which is prior to previous packet 1587843059.260362
LOOP:   2020-04-25 19:30:59 UTC (1587843059) dateTime: 1587843059.25, maxSolarRad: None, rainRate: 0, usUnits: 1, windSpeed: 8.45
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843059.254031 which is prior to previous packet 1587843059.260362
LOOP:   2020-04-25 19:30:59 UTC (1587843059) dateTime: 1587843059.25, maxSolarRad: None, rainRate: 0, usUnits: 1, windSpeed: 8.45
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843059.259809 which is prior to previous packet 1587843059.260362
LOOP:   2020-04-25 19:30:59 UTC (1587843059) dateTime: 1587843059.26, maxSolarRad: None, rainRate: 0, usUnits: 1, windSpeed: 8.45
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843059.247136 which is prior to previous packet 1587843059.260362
LOOP:   2020-04-25 19:30:59 UTC (1587843059) dateTime: 1587843059.25, maxSolarRad: None, rainBatteryStatus: True, rainRate: 0, usUnits: 1
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843059.252173 which is prior to previous packet 1587843059.260362
LOOP:   2020-04-25 19:30:59 UTC (1587843059) dateTime: 1587843059.25, maxSolarRad: None, rainBatteryStatus: True, rainRate: 0, usUnits: 1
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843059.259330 which is prior to previous packet 1587843059.260362
LOOP:   2020-04-25 19:30:59 UTC (1587843059) dateTime: 1587843059.26, maxSolarRad: None, rainBatteryStatus: True, rainRate: 0, usUnits: 1
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843059.251268 which is prior to previous packet 1587843059.260362
LOOP:   2020-04-25 19:30:59 UTC (1587843059) dateTime: 1587843059.25, maxSolarRad: None, outHumidity: 70.0, rainRate: 0, usUnits: 1
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843059.258833 which is prior to previous packet 1587843059.260362
LOOP:   2020-04-25 19:30:59 UTC (1587843059) dateTime: 1587843059.26, maxSolarRad: None, outHumidity: 70.0, rainRate: 0, usUnits: 1
LOOP:   2020-04-25 19:30:59 UTC (1587843059) dateTime: 1587843059.26, maxSolarRad: None, outHumidity: 70.0, rainRate: 0, usUnits: 1
LOOP:   2020-04-25 19:31:17 UTC (1587843077) battery_ok: 1.0, dateTime: 1587843077.6, maxSolarRad: None, rainRate: 0, usUnits: 1, wind_avg_km_h: 10.106, windDir: 270
LOOP:   2020-04-25 19:31:17 UTC (1587843077) battery_ok: 1.0, dateTime: 1587843077.61, maxSolarRad: None, rain_in: 2.4, rainRate: 0, usUnits: 1, wind_avg_km_h: 10.106, wind_dir_deg: 270.0, windDir: 270
LOOP:   2020-04-25 19:31:17 UTC (1587843077) battery_ok: 1.0, dateTime: 1587843077.65, maxSolarRad: None, rain_in: 2.4, rainRate: 0, usUnits: 1, wind_avg_km_h: 10.106, wind_dir_deg: 270.0, windDir: 270
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843077.600203 which is prior to previous packet 1587843077.645759
LOOP:   2020-04-25 19:31:17 UTC (1587843077) dateTime: 1587843077.6, maxSolarRad: None, rainRate: 0, usUnits: 1, windSpeed: 10.106
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843077.605252 which is prior to previous packet 1587843077.645759
LOOP:   2020-04-25 19:31:17 UTC (1587843077) dateTime: 1587843077.61, maxSolarRad: None, rainRate: 0, usUnits: 1, windSpeed: 10.106
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843077.643581 which is prior to previous packet 1587843077.645759
LOOP:   2020-04-25 19:31:17 UTC (1587843077) dateTime: 1587843077.64, maxSolarRad: None, rainRate: 0, usUnits: 1, windSpeed: 10.106
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843077.598757 which is prior to previous packet 1587843077.645759
LOOP:   2020-04-25 19:31:17 UTC (1587843077) dateTime: 1587843077.6, maxSolarRad: None, rainBatteryStatus: True, rainRate: 0, usUnits: 1
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843077.604045 which is prior to previous packet 1587843077.645759
LOOP:   2020-04-25 19:31:17 UTC (1587843077) dateTime: 1587843077.6, maxSolarRad: None, rainBatteryStatus: True, rainRate: 0, usUnits: 1
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843077.641589 which is prior to previous packet 1587843077.645759
LOOP:   2020-04-25 19:31:17 UTC (1587843077) dateTime: 1587843077.64, maxSolarRad: None, rainBatteryStatus: True, rainRate: 0, usUnits: 1
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843077.602780 which is prior to previous packet 1587843077.645759
LOOP:   2020-04-25 19:31:17 UTC (1587843077) dateTime: 1587843077.6, maxSolarRad: None, rain: 0.0, rainRate: 0, usUnits: 1
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843077.608067 which is prior to previous packet 1587843077.645759
LOOP:   2020-04-25 19:31:17 UTC (1587843077) dateTime: 1587843077.61, maxSolarRad: None, rain: 0.0, rainRate: 0, usUnits: 1
LOOP:   2020-04-25 19:31:17 UTC (1587843077) dateTime: 1587843077.65, maxSolarRad: None, rain: 0.0, rainRate: 0, usUnits: 1, wind_dir_deg: 270.0
LOOP:   2020-04-25 19:31:36 UTC (1587843096) dateTime: 1587843096.47, maxSolarRad: None, outTemp: 86.0, rain_in: 2.4, rainRate: 0, usUnits: 1
LOOP:   2020-04-25 19:31:36 UTC (1587843096) battery_ok: 1.0, dateTime: 1587843096.48, humidity: 70.0, maxSolarRad: None, outTemp: 86.0, rainRate: 0, temperature_F: 86.0, usUnits: 1, wind_avg_km_h: 8.45
LOOP:   2020-04-25 19:31:36 UTC (1587843096) battery_ok: 1.0, dateTime: 1587843096.48, humidity: 70.0, maxSolarRad: None, outTemp: 86.0, rainRate: 0, temperature_F: 86.0, usUnits: 1, wind_avg_km_h: 8.45
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843096.468349 which is prior to previous packet 1587843096.482884
LOOP:   2020-04-25 19:31:36 UTC (1587843096) dateTime: 1587843096.47, maxSolarRad: None, rainRate: 0, usUnits: 1, windSpeed: 8.45
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843096.476654 which is prior to previous packet 1587843096.482884
LOOP:   2020-04-25 19:31:36 UTC (1587843096) dateTime: 1587843096.48, maxSolarRad: None, rainRate: 0, usUnits: 1, windSpeed: 8.45
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843096.481598 which is prior to previous packet 1587843096.482884
LOOP:   2020-04-25 19:31:36 UTC (1587843096) dateTime: 1587843096.48, maxSolarRad: None, rainRate: 0, usUnits: 1, windSpeed: 8.45
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843096.466990 which is prior to previous packet 1587843096.482884
LOOP:   2020-04-25 19:31:36 UTC (1587843096) dateTime: 1587843096.47, maxSolarRad: None, rainBatteryStatus: True, rainRate: 0, usUnits: 1
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843096.474472 which is prior to previous packet 1587843096.482884
LOOP:   2020-04-25 19:31:36 UTC (1587843096) dateTime: 1587843096.47, maxSolarRad: None, rainBatteryStatus: True, rainRate: 0, usUnits: 1
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843096.480391 which is prior to previous packet 1587843096.482884
LOOP:   2020-04-25 19:31:36 UTC (1587843096) dateTime: 1587843096.48, maxSolarRad: None, rainBatteryStatus: True, rainRate: 0, usUnits: 1
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843096.469637 which is prior to previous packet 1587843096.482884
LOOP:   2020-04-25 19:31:36 UTC (1587843096) dateTime: 1587843096.47, maxSolarRad: None, outHumidity: 70.0, rainRate: 0, usUnits: 1
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843096.478957 which is prior to previous packet 1587843096.482884
LOOP:   2020-04-25 19:31:36 UTC (1587843096) dateTime: 1587843096.48, maxSolarRad: None, outHumidity: 70.0, rainRate: 0, usUnits: 1
LOOP:   2020-04-25 19:31:36 UTC (1587843096) dateTime: 1587843096.48, maxSolarRad: None, outHumidity: 70.0, rainRate: 0, temperature_F: 86.0, usUnits: 1
LOOP:   2020-04-25 19:31:54 UTC (1587843114) dateTime: 1587843114.82, humidity: 70.0, maxSolarRad: None, rainRate: 0, usUnits: 1, windDir: 202
LOOP:   2020-04-25 19:31:54 UTC (1587843114) dateTime: 1587843114.82, maxSolarRad: None, rainRate: 0, usUnits: 1, windDir: 202
LOOP:   2020-04-25 19:31:54 UTC (1587843114) battery_ok: 1.0, dateTime: 1587843114.83, maxSolarRad: None, rainRate: 0, usUnits: 1, windDir: 202
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843114.821543 which is prior to previous packet 1587843114.828766
LOOP:   2020-04-25 19:31:54 UTC (1587843114) dateTime: 1587843114.82, maxSolarRad: None, rainRate: 0, usUnits: 1, windSpeed: 6.795
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843114.824284 which is prior to previous packet 1587843114.828766
LOOP:   2020-04-25 19:31:54 UTC (1587843114) dateTime: 1587843114.82, maxSolarRad: None, rainRate: 0, usUnits: 1, windSpeed: 6.795
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843114.827832 which is prior to previous packet 1587843114.828766
LOOP:   2020-04-25 19:31:54 UTC (1587843114) dateTime: 1587843114.83, maxSolarRad: None, rainRate: 0, usUnits: 1, windSpeed: 6.795
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843114.820626 which is prior to previous packet 1587843114.828766
LOOP:   2020-04-25 19:31:54 UTC (1587843114) dateTime: 1587843114.82, maxSolarRad: None, rainBatteryStatus: True, rainRate: 0, usUnits: 1
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843114.823686 which is prior to previous packet 1587843114.828766
LOOP:   2020-04-25 19:31:54 UTC (1587843114) dateTime: 1587843114.82, maxSolarRad: None, rainBatteryStatus: True, rainRate: 0, usUnits: 1
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843114.826141 which is prior to previous packet 1587843114.828766
LOOP:   2020-04-25 19:31:54 UTC (1587843114) dateTime: 1587843114.83, maxSolarRad: None, rainBatteryStatus: True, rainRate: 0, usUnits: 1
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843114.822950 which is prior to previous packet 1587843114.828766
LOOP:   2020-04-25 19:31:54 UTC (1587843114) dateTime: 1587843114.82, maxSolarRad: None, rain: 0.0, rainRate: 0, usUnits: 1
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843114.825548 which is prior to previous packet 1587843114.828766
LOOP:   2020-04-25 19:31:54 UTC (1587843114) dateTime: 1587843114.83, maxSolarRad: None, rain: 0.0, rainRate: 0, usUnits: 1
LOOP:   2020-04-25 19:31:54 UTC (1587843114) dateTime: 1587843114.83, maxSolarRad: None, rain: 0.0, rainRate: 0, usUnits: 1, wind_avg_km_h: 6.795
LOOP:   2020-04-25 19:32:13 UTC (1587843133) battery_ok: 1.0, dateTime: 1587843133.17, maxSolarRad: None, outTemp: 86.0, rain_in: 2.4, rainRate: 0, usUnits: 1, wind_avg_km_h: 6.795, wind_dir_deg: 202.5
LOOP:   2020-04-25 19:32:13 UTC (1587843133) dateTime: 1587843133.17, maxSolarRad: None, outTemp: 86.0, rainRate: 0, usUnits: 1
LOOP:   2020-04-25 19:32:13 UTC (1587843133) battery_ok: 1.0, dateTime: 1587843133.21, humidity: 70.0, maxSolarRad: None, outTemp: 86.0, rainRate: 0, temperature_F: 86.0, usUnits: 1, wind_avg_km_h: 7.622
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843133.172036 which is prior to previous packet 1587843133.214794
LOOP:   2020-04-25 19:32:13 UTC (1587843133) dateTime: 1587843133.17, maxSolarRad: None, rainRate: 0, usUnits: 1, windSpeed: 7.622
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843133.174369 which is prior to previous packet 1587843133.214794
LOOP:   2020-04-25 19:32:13 UTC (1587843133) dateTime: 1587843133.17, maxSolarRad: None, rainRate: 0, usUnits: 1, windSpeed: 7.622
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843133.214184 which is prior to previous packet 1587843133.214794
LOOP:   2020-04-25 19:32:13 UTC (1587843133) dateTime: 1587843133.21, maxSolarRad: None, rainRate: 0, usUnits: 1, windSpeed: 7.622
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843133.171374 which is prior to previous packet 1587843133.214794
LOOP:   2020-04-25 19:32:13 UTC (1587843133) dateTime: 1587843133.17, maxSolarRad: None, rainBatteryStatus: True, rainRate: 0, usUnits: 1
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843133.173821 which is prior to previous packet 1587843133.214794
LOOP:   2020-04-25 19:32:13 UTC (1587843133) dateTime: 1587843133.17, maxSolarRad: None, rainBatteryStatus: True, rainRate: 0, usUnits: 1
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843133.213546 which is prior to previous packet 1587843133.214794
LOOP:   2020-04-25 19:32:13 UTC (1587843133) dateTime: 1587843133.21, maxSolarRad: None, rainBatteryStatus: True, rainRate: 0, usUnits: 1
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843133.173261 which is prior to previous packet 1587843133.214794
LOOP:   2020-04-25 19:32:13 UTC (1587843133) dateTime: 1587843133.17, maxSolarRad: None, outHumidity: 70.0, rainRate: 0, usUnits: 1
weewx[323]: user.MQTTSubscribe: MQTTSubscribeService ignoring packet has dateTime of 1587843133.177543 which is prior to previous packet 1587843133.214794
LOOP:   2020-04-25 19:32:13 UTC (1587843133) dateTime: 1587843133.18, maxSolarRad: None, outHumidity: 70.0, rainRate: 0, usUnits: 1
LOOP:   2020-04-25 19:32:13 UTC (1587843133) dateTime: 1587843133.22, maxSolarRad: None, outHumidity: 70.0, rainRate: 0, usUnits: 1

Cleanup the configuration

Possibly have a shared section that could be overridden by the service and driver.
Pass a dictionary to the parent class instead of all the individual parameters.

Handle queueing of archive data when service calling driver is in software generation mode

Currently if the archive_topic is set, this payload is added to the archive queue and then removed in the genArchiveRecords method. If the controlling service never calls this method (for example, software generation setting of stdArchive), the archive queue will grow infinitely.
Typically the calling service is stdArchive, but it doesn’t have to be. Additionally, the driver usually does not have access to the service’s configuration.
So, the approach will be to set an archive queue limit. When this is reached, the queue will be emptied, the new payload add, and an error message logged.

A quick hack to test/learn is decode the MQTT payload before calling json.loads.

            self._log_message(msg)

            msg.payload = msg.payload.decode('utf-8') # Hack to experiment with type of data

            data = self._byteify(json.loads(msg.payload, object_hook=self._byteify), ignore_dicts=True)

More learning is needed to see how this might impact/simplify the _byteify routine.

Feature Request: [[deltas]] functionality similar to weewx-sdr

I'm using the MQTTSubscribe driver to ingest topics from my Acurite Atlas.

RTL_433's (feat-atlas branch) mqtt option is being used to capture and send the sensor data.

The atlas only reports rainfall accumulation in inches, so I can't use it directly with WeeWX.

It would be great to have a feature similar to [[deltas]] in WeeWX-SDR that would take the cumulative measures and split into delta measures.

Config example:

  [[deltas]]
        rain = rain_total