Update Frequency about weewx-mqttsubscribe

[bellrichm]

I’m not thrilled about adding caching, but do understand the need/value. I need to cleanup some other things before looking at this.
rich

[dshanske]

I haven't looked at the architecture too closely...I mostly code in PHP. But I figured it was worth suggesting due the use case. The other alternative is a RETAIN on the MQTT statement, but I think that is potentially dangerous to the values as well.

[bellrichm]

Sorry for the whining. I get it and do plan on implementing something. It will probably be a dirt cheap cache that right before updating the packet will pull any missing data from the cache.
Data to be cached and some type of expiration would be configurable. It would be a new configuration section under the MQTTSubscribe section.
I’d probably develop in a separate branch and if you could beta test, that would be a big help.

[dshanske]

Didn't see it as whining. But happy to beta test. Anything I can do.

[orrious]

@bellrichm Sounds like this is the same request I made with the cache / hold timers. Would it be possible to add a "cache" in seconds under [[[fields]]]?

[[message_callback]]
    type = json
    full_topic_fieldname = True
    [[[fields]]]
        [[[[sensors/rtl_433/time]]]]
            ignore = True
        [[[[sensors/rtl_433/model]]]]
            ignore = True
        [[[[sensors/rtl_433/channel]]]]
            ignore = True
        [[[[sensors/rtl_433/mic]]]]
            ignore = True
        [[[[sensors/rtl_433/wind_dir_deg]]]]
            name = windDir
            conversion_type = int
            cache = 60
        [[[[sensors/rtl_433/wind_avg_km_h]]]]
            name = windSpeed
            units = km_per_hour
        [[[[sensors/rtl_433/temperature_F]]]]
            name = outTemp
        [[[[sensors/rtl_433/humidity]]]]
            name = outHumidity
        [[[[sensors/rtl_433/battery_ok]]]]
            name = rainBatteryStatus
            conversion_type = bool
        [[[[sensors/rtl_433/rain_in]]]]
            name = rain
            contains_total = True
        [[[[sensors/office/bme280/time_UTC]]]]
            ignore = True
        [[[[sensors/office/bme280/pressure_inHg]]]]    
            name = pressure            
        [[[[sensors/office/bme280/humidity]]]]    
            name = inHumidity
        [[[[sensors/office/bme280/temperature_F]]]]    
            name = inTemp                                            
    
[[topics]]
    [[[sensors/rtl_433]]]
    [[[sensors/office/bme280]]]

[bellrichm]

Hmm, I think there are two problems/use cases here.

1. Messages are published ‘frequently’, but content varies. This is @orrious‘ problem with windDir missing.
2. Messages are published infrequently, such that WeeWX ‘sees’ the data as missing. I believe this is @dshanske‘s problem.
   The way MQTTSubscribe works, (1) seems to be easy to fix via the additional fields option.
   The second is bit more challenging because the gathering of the data is triggered off a new message appearing. One possible approach is to keep track of when the last message for a topic was received. If it has been too long, take the cached value and create a packet with it. Another is to just add the cached value to every packet that doesn’t have the value and let WeeWX do its magic. I was leaning toward option 2, even though it is ugly. Option 1 will require some additional topic configuration and effort. But the more I think about it, the more I think that of these two options, (1) is the way to go.
   rich

[dshanske]

My problem is as identified. Regrettably, 1 probably won't solve it.

[dshanske]

I think I will have to set retain on the mqtt side to retain the last version of the message until a new one refreshes it.

[bellrichm]

@dshanske, thanks for the confirmation.
If it is reasonable to cache via MQTT, I’d prefer that... But, after thinking more, I think I have a simple solution that will meet your needs. So let me know how it goes.
I’ll continue to flush out the details.

[dshanske]

@bellrichm Always appreciate help and hope I can return the favor someday.

[orrious]

@bellrichm what are your thoughts on building an "auto-loop" based on a timer. So every x seconds a loop will be generated with the fields included, regardless of trigger new mqtt event. Only fields with cached entries would be announced.

[[message_callback]]
    type = json
    full_topic_fieldname = True
    auto-loop = 30
    [[[fields]]]
        [[[[sensors/rtl_433/time]]]]
            ignore = True
        [[[[sensors/rtl_433/model]]]]
            ignore = True
        [[[[sensors/rtl_433/channel]]]]
            ignore = True
        [[[[sensors/rtl_433/mic]]]]
            ignore = True
        [[[[sensors/rtl_433/wind_dir_deg]]]]
            name = windDir
            conversion_type = int
            cache = 60
        [[[[sensors/rtl_433/wind_avg_km_h]]]]
            name = windSpeed
            units = km_per_hour
        [[[[sensors/rtl_433/temperature_F]]]]
            name = outTemp
        [[[[sensors/rtl_433/humidity]]]]
            name = outHumidity
        [[[[sensors/rtl_433/battery_ok]]]]
            name = rainBatteryStatus
            conversion_type = bool
        [[[[sensors/rtl_433/rain_in]]]]
            name = rain
            contains_total = True
        [[[[sensors/office/bme280/time_UTC]]]]
            ignore = True
        [[[[sensors/office/bme280/pressure_inHg]]]]    
            name = pressure            
        [[[[sensors/office/bme280/humidity]]]]    
            name = inHumidity
        [[[[sensors/office/bme280/temperature_F]]]]    
            name = inTemp   
        [[[[sensors/office/bme280/airquality]]]]      
            auto-loop = true
            cache = 1200                                     
    
[[topics]]
    [[[sensors/rtl_433]]]
    [[[sensors/office/bme280]]]

[bellrichm]

@dshanske , would you mind sharing your [topics] and [fields] configuration sections? Make sure to include your air quality info.
Thanks. rich

[bellrichm]

@orrious, interesting idea.
The current topic/field handling might make it a challenge, will have to some more investigation. I was almost ready to revamp it, but then remembered that the topics one subscribes to can be different than the topics in the message. This happens if one use wild carding, such as the #. It does make adding a topic section under the message handler section (as you mentioned in #69) more palatable to me.
I need to think a bit more. rich

[dshanske]

[[[fields]]] [[[[BME280_Pressure]]]] name = pressure conversion_type = float contains_total = False [[[[BME280_Temperature]]]] name = extraTemp1 conversion_type = float contains_total = False [[[[BME280_Humidity]]]] name = extraHumid1 conversion_type = float contains_total = False [[[[BME280_DewPoint]]]] ignore = true [[[[PressureUnit]]]] ignore = true [[[[TempUnit]]]] ignore = true [[[[pm2_5_ug_m3]]]] name = pm2_5 conversion_type = int contains_total = False [[[[pm10_0_ug_m3]]]] name = pm10_0 conversion_type = int contains_total = False [[[[model]]]] ignore = true [[[[mic]]]] ignore = true [[topics]] unit_system = METRIC [[[rtl_915/events]]] [[[tele/barometer/SENSOR]]]

[bellrichm]

@orrious , If I understand your thoughts correctly, i don’t think it would work for wind data. This is because the auto-loop packet would only have windDir data.
But, stepping back, I really don’t think we need to do anything. WeeWX is designed to handle packets with partial data. Yes, wind data is special and has to be in the same packet. But we fixed that special case when subscribed to individual topics.
rich

[bellrichm]

@dshanske
Would something like this work?

   [[topics]]
        [[[topic1]]]
        [[[topic2]]]
            [[[[cache]]]]
                polling = x
                expiration = y

If a message has not been received in x seconds, the previous message will be used. If the previous message is older than y seconds, it will not be used.
There is a bit more going on under the covers, but that is essentially what would happen.
rich

[dshanske]

That's exactly what I was thinking. Although the two terms I'm not sure would reflect the meaning to me.

Would you consider expiration changing to expire_after ? Maybe polling to use_stale ? Just throwing out the first names that come to mind

[bellrichm]

I’m open to any naming improvements! We’ll go with what you suggested. We can change it up to release time.
rich

[orrious]

@bellrichm I needed to add one more variable, "peer". Variable names can and should be changed.. ;)

Here is my logic that solves for both scenario's.

1.) Cache windDir for 60 seconds. If you don't get an update in 60 seconds, windDir=null. "peer" ties windDir to windSpeed. When a LOOP with windSpeed is generated, (auto loop or when an mqtt event comes in) the cached value of windDir will be included in the LOOP.

Notice neither windDir nor windSpeed are not included in auto-loop, so therefore when an auto-loop is generated these values will not be included in this config, however if [[[[sensors/rtl_433/wind_avg_km_h]]]] had auto-loop = true, then windSpeed, windDir, and airquality would all be included in auto-loop.

2.) Auto-loop - This will generate a LOOP on a timer. Any field with auto-loop = true will be included in the timed LOOP generation. If you auto-loop = true and cache is not set, then your auto-loop will contain NULL for the values. Don't use auto-loop=true without setting cache.

[[message_callback]]
    type = json
    full_topic_fieldname = True
    auto-loop = 30
    [[[fields]]]
        [[[[sensors/rtl_433/time]]]]
            ignore = True
        [[[[sensors/rtl_433/model]]]]
            ignore = True
        [[[[sensors/rtl_433/channel]]]]
            ignore = True
        [[[[sensors/rtl_433/mic]]]]
            ignore = True
        [[[[sensors/rtl_433/wind_dir_deg]]]]
            name = windDir
            conversion_type = int
            cache = 60
            peer = windSpeed
        [[[[sensors/rtl_433/wind_avg_km_h]]]]
            name = windSpeed
            units = km_per_hour
        [[[[sensors/rtl_433/temperature_F]]]]
            name = outTemp
        [[[[sensors/rtl_433/humidity]]]]
            name = outHumidity
        [[[[sensors/rtl_433/battery_ok]]]]
            name = rainBatteryStatus
            conversion_type = bool
        [[[[sensors/rtl_433/rain_in]]]]
            name = rain
            contains_total = True
        [[[[sensors/office/bme280/time_UTC]]]]
            ignore = True
        [[[[sensors/office/bme280/pressure_inHg]]]]    
            name = pressure            
        [[[[sensors/office/bme280/humidity]]]]    
            name = inHumidity
        [[[[sensors/office/bme280/temperature_F]]]]    
            name = inTemp   
        [[[[sensors/office/bme280/airquality]]]]      
            auto-loop = true
            cache = 1200                                     
    
[[topics]]
    [[[sensors/rtl_433]]]
    [[[sensors/office/bme280]]]

Thoughts?

[orrious]

Btw, I know we are talking about two subjects in the same thread, but IMO they are and should be handled as separate items.

1. Including cached values during event driven LOOPS
2. Generating LOOPS based on a timer.

[bellrichm]

Just capturing this for myself. Based on this discussion, I could see the possibility that some would want a None value instead of the last value

https://groups.google.com/d/msg/weewx-user/_Jq3rmxEsdY/jwZEc5-sBAAJ

[bellrichm]

@orrious, are you seeing problems with the versions of MQTTSubscribe in your docker containers? Last I looked, both the individual payloads and json payloads seemed to be working

[orrious]

In my approach, if no cache value is specified, then you would get the None for the value with either the peer or auto-loop set.

With peer set without cache set, when an event came in for its peer, the subsequent LOOP generated would contain a None value.

Same for auto-loop. It's all in what is triggering the LOOP.

[orrious]

@bellrichm I saw that as well. I didn't know if you were done before I went poking around. If you feel it's ready, I'll move it to my prod instance

[bellrichm]

Let me package it up in a prerelease before you do that. That way if there is a problem, we will know exactly what you have.

[dshanske]

@bellrichm Tell me when you want me to test something.

[bellrichm]

@orrious here is the prerelease. If we have need to discuss anything related to it, we can move that offline.

[bellrichm]

While I don't think it is the end of the world, the following could happen with all our proposals. We create a loop with cached values and right after it the 'real' value comes in. When WeeWX creates the archive record, it will use the data from both loop packets...

So, after pondering this more, I now have a different perspective on it. The problem is when the data is missing from the archive records. This is the data that goes in the DB, drives reports, etc.

So far all of our thoughts proposals have used the loop packets to get the cached data into the archive record. I am leaning toward actually caching at the archive record. The logic would be, "is it missing from the archive record?, then get it from the cache."

Just struggling a bit if it really belongs in MQTTSubscribe or if it should be a super lightweight separate service.

[bellrichm]

@dshanske, in this branch, https://github.com/bellrichm/WeeWX-MQTTSubscribe/tree/archive-cache I have some alpha code that caches archive record fields. It is supported only when running as a service.
Under the [MQTTSubscribe] section, the current configuration looks like this.

    # Fields in this section will be cached.
    # If the field is not in the current archive record, its value will be retrieved from the cache.
    # Only used by the service.
    # EXPERIMENTAL - may be removed
    [[archive_field_cache]]
        # The unit system of the cache.
        # This must match the unit_system of the archive record.
        # Default is US.
        unit_system = US

        # The WeeWX fields to cache.
        [[[fields]]]
            # The name of the field to cache.
            [[[[fieldname]]]]
                # In seconds how long the cache is valid.
                # Value of 0 means the cache is always expired.
                # Useful if missing fields should have a value of None instead of the previous value.
                # Value of None means the cache never expires.
                # Default is None.
                expires_after = None

[orrious]

@bellrichm I've been using your prerelease with both individual and json and seems to be working pretty well on Weewx 3.9.2. I'm trying to get some time to start testing Weewx 4.

In regards to your concerns about caching and sending stale data, after reading other contributors points of view, it may not be the best idea to cache values. Someone may really need to for some fringe case and wouldn't hurt to have the functionality available, but more than likely I won't be leveraging the feature myself.

[bellrichm]

@orrious, thanks for the feedback!! Let me know if I can do anything else.
I’m feeling pretty comfortable with ‘caching’ at the archive record. My understanding is it should have all fields. So providing a mechanism to ensure that does seem necessary. The user can decide if missing values should be None or the previous value.
Re: WeeWX v4 and logging - have you seen this?
https://groups.google.com/d/msgid/weewx-development/2e9d6f6b-9324-46b3-815b-f4dbde3f7df7%40googlegroups.com?utm_medium=email&utm_source=footer
Thanks for your help. rich

[bellrichm]

@dshanske Have you had a chance to try the alpha code?

[dshanske]

@bellrichm Sorry, I thought I responded. Installed it, but no effect.

[bellrichm]

@dshanske, No problem. I fixed a bug in the latest commit on that branch. Could you give it a try when convenient?

[dshanske]

@bellrichm Still nothing. I added expires_after = 540 to the appropriate fields.

[bellrichm]

@dshanske, Thanks for testing. I can look into trying to reproduce. It will probably take awhile.
In the meantime, if you still up for testing/debugging, the next step would be to set debug = 2 and capture the log from start through an archive record with the air quality value and the following archive record that should retrieve it from the cache. This will result in a lot of logging. Only subscribing to the air quality topic should greatly reduce it and hopefully still be a valid test.
Again, thanks.

[bellrichm]

@dshanske , I found another silly bug. I’ll post back when a fix has been committed

[bellrichm]

@dshanske
The latest commit fixes the configuration bug.

The only way I could think to more fully test was, to write a small service that adds two fields to the archive record every third time. I then configured MQTTSubscribe to cache these fields, extraTemp1 and newObservation. In the output below, you can see the first two records have a value of None, because no value has been received yet. Then records 3, 4, and 5 all have the value of 3. Then records 6, 7, and 8 all have the value of 6.

REC:    2020-05-23 11:23:00 EDT (1590247380) barometer: 31.099999333773486, consBatteryVoltage: 12.23846372734675, dateTime: 1590247380, extraTemp1: None, heatingVoltage: 12.02870415754863, inHumidity: 29.999973350955027, inTemp: 63.000013324522506, inTempBatteryStatus: 0.0, interval: 1.0, newObservation: None, outHumidity: 79.9999800132043, outTemp: 32.659539544098685, outTempBatteryStatus: 0.0, pressure: 31.099999333773486, radiation: 973.4754881073069, rain: 0.0, rainBatteryStatus: 0.0, referenceVoltage: 11.921173820749413, rxCheckPercent: 94.43551632086655, supplyVoltage: 11.938077650424617, txBatteryStatus: 0.0, usUnits: 1, UV: 13.628656833502292, windBatteryStatus: 0.0, windDir: 359.99978429136166, windGust: 1.1459094846344442e-05, windGustDir: 359.9996562271546, windSpeed: 3.331132617826168e-06
REC:    2020-05-23 11:24:00 EDT (1590247440) barometer: 31.099995025856696, consBatteryVoltage: 11.846511429210915, dateTime: 1590247440, extraTemp1: None, heatingVoltage: 11.901495148457585, inHumidity: 29.999801034838825, inTemp: 63.00009948258059, inTempBatteryStatus: 0.0, interval: 1.0, newObservation: None, outHumidity: 79.99985077570078, outTemp: 32.618023240331155, outTempBatteryStatus: 0.0, pressure: 31.099995025856696, radiation: 974.8024877470854, rain: 0.0, rainBatteryStatus: 0.0, referenceVoltage: 12.113962328826497, rxCheckPercent: 98.08485836522107, supplyVoltage: 11.970669894416547, txBatteryStatus: 0.0, usUnits: 1, UV: 13.647234828459196, windBatteryStatus: 0.0, windDir: 359.9989666575507, windGust: 5.132107049909962e-05, windGustDir: 359.99846036788506, windSpeed: 2.487071653563729e-05
REC:    2020-05-23 11:25:00 EDT (1590247500) barometer: 31.09998590322979, consBatteryVoltage: 11.933837091963793, dateTime: 1590247500, extraTemp1: 3, heatingVoltage: 11.90253451357684, inHumidity: 29.999436133386737, inTemp: 63.00028193330663, inTempBatteryStatus: 0.0, interval: 1.0, newObservation: 3, outHumidity: 79.9995770968937, outTemp: 32.57502363190307, outTempBatteryStatus: 0.0, pressure: 31.09998590322979, radiation: 976.1520272405422, rain: 0.0, rainBatteryStatus: 0.0, referenceVoltage: 12.220582490289901, rxCheckPercent: 98.58671497757011, supplyVoltage: 12.184590583896698, txBatteryStatus: 0.0, usUnits: 1, UV: 13.66612838136759, windBatteryStatus: 0.0, windDir: 359.99732164440377, windGust: 0.00011974067365905228, windGustDir: 359.9964077797902, windSpeed: 7.048385104901911e-05
REC:    2020-05-23 11:26:00 EDT (1590247560) barometer: 31.099972021024687, consBatteryVoltage: 11.921100697582311, dateTime: 1590247560, extraTemp1: 3, heatingVoltage: 11.962612484326007, inHumidity: 29.998880857084554, inTemp: 63.00055957145773, inTempBatteryStatus: 0.0, interval: 1.0, newObservation: 3, outHumidity: 79.99916063074062, outTemp: 32.53235576990749, outTempBatteryStatus: 0.0, pressure: 31.099972021024687, radiation: 977.465305913627, rain: 0.0, rainBatteryStatus: 0.0, referenceVoltage: 12.240712959897158, rxCheckPercent: 93.32437243516682, supplyVoltage: 12.042436576158956, txBatteryStatus: 0.0, usUnits: 1, UV: 13.684514282790778, windBatteryStatus: 0.0, windDir: 359.9948220528405, windGust: 0.00021671757867380848, windGustDir: 359.99349847263977, windSpeed: 0.0001398948765658862
REC:    2020-05-23 11:27:00 EDT (1590247620) barometer: 31.09995337930746, consBatteryVoltage: 11.793935828535595, dateTime: 1590247620, extraTemp1: 3, heatingVoltage: 11.965771102221845, inHumidity: 29.998135216503915, inTemp: 63.000932391748044, inTempBatteryStatus: 0.0, interval: 1.0, newObservation: 3, outHumidity: 79.99860137922371, outTemp: 32.49002046667897, outTempBatteryStatus: 0.0, pressure: 31.09995337930746, radiation: 978.7422237550844, rain: 0.0, rainBatteryStatus: 0.0, referenceVoltage: 12.183610162749455, rxCheckPercent: 96.13805910885928, supplyVoltage: 11.919129605063281, txBatteryStatus: 0.0, usUnits: 1, UV: 13.702391132571185, windBatteryStatus: 0.0, windDir: 359.99146622200885, windGust: 0.00034225132396770164, windGustDir: 359.98973246028095, windSpeed: 0.0002331034627144346
REC:    2020-05-23 11:28:00 EDT (1590247680) barometer: 31.099929978166838, consBatteryVoltage: 11.92112280962858, dateTime: 1590247680, extraTemp1: 6, heatingVoltage: 12.092961195028757, inHumidity: 29.99719922584076, inTemp: 63.00140038707962, inTempBatteryStatus: 0.0, interval: 1.0, newObservation: 6, outHumidity: 79.99789934500488, outTemp: 32.448018528220686, outTempBatteryStatus: 0.0, pressure: 31.099929978166838, radiation: 979.9826835226771, rain: 0.0, rainBatteryStatus: 0.0, referenceVoltage: 12.08530931059938, rxCheckPercent: 90.99489729755648, supplyVoltage: 11.915879226725005, txBatteryStatus: 0.0, usUnits: 1, UV: 13.719757569317478, windBatteryStatus: 0.0, windDir: 359.9872538563203, windGust: 0.0004963413120435689, windGustDir: 359.9851097606387, windSpeed: 0.0003501091658547247
REC:    2020-05-23 11:29:00 EDT (1590247740) barometer: 31.09990181771418, consBatteryVoltage: 11.992836443624116, dateTime: 1590247740, extraTemp1: 6, heatingVoltage: 12.127755144789042, inHumidity: 29.996072902914985, inTemp: 63.0019635485425, inTempBatteryStatus: 0.0, interval: 1.0, newObservation: 6, outHumidity: 79.99705453142552, outTemp: 32.40635075418897, outTempBatteryStatus: 0.0, pressure: 31.09990181771418, radiation: 981.1865907505909, rain: 0.0, rainBatteryStatus: 0.0, referenceVoltage: 11.983082658225605, rxCheckPercent: 82.71527675118185, supplyVoltage: 11.986935758293539, txBatteryStatus: 0.0, usUnits: 1, UV: 13.736612270508273, windBatteryStatus: 0.0, windDir: 359.9821848842579, windGust: 0.0006789868094898566, windGustDir: 359.9796303957153, windSpeed: 0.0004909114290810868
REC:    2020-05-23 11:30:00 EDT (1590247800) barometer: 31.099868898083557, consBatteryVoltage: 12.219006485525107, dateTime: 1590247800, extraTemp1: 6, heatingVoltage: 12.357882431374586, inHumidity: 29.994756269170185, inTemp: 63.00262186541491, inTempBatteryStatus: 0.0, interval: 1.0, newObservation: 6, outHumidity: 79.99606694250664, outTemp: 32.365017937878214, outTempBatteryStatus: 0.0, pressure: 31.099868898083557, radiation: 982.3538537566297, rain: 0.0, rainBatteryStatus: 0.0, referenceVoltage: 12.040907599275302, rxCheckPercent: 89.39773428987591, supplyVoltage: 12.029866221853167, txBatteryStatus: 0.0, usUnits: 1, UV: 13.752953952592817, windBatteryStatus: 0.0, windDir: 359.9762592952788, windGust: 0.0008901869469779555, windGustDir: 359.9732943915907, windSpeed: 0.0006555095822248594

[bellrichm]

@dshanske, I’m guessing it’s still not working for you. If you want me ton continue to pursue this, I’ll need to see the log.

[dshanske]

Will do... Have a different problem with the main driver I need to fix. I am at a secondary site...and installed the same barometer you helped me get working and that's doing fine... need to drive back...I think the batteries on my weather station died.

[dshanske]

Replaced the batteries, which means things are now back online. Which lines, as the debug is quite long?

Jun 02 08:47:32 nuc python3[3651105]: weewx[3651105] DEBUG user.MQTTSubscribe: MessageCallbackProvider data-> incoming topic: rtl_915/events, QOS: 0, retain: 0, payload: b'{"time":"1591102052","model":"Fineoffset-WH0290","id":203,"pm2_5_ug_m3":6,"pm10_0_ug_m3":7,"mic":"CRC"}'
Jun 02 08:47:32 nuc python3[3651105]: weewx[3651105] DEBUG user.MQTTSubscribe: TopicManager data-> incoming rtl_915/events: id: 203.0, pm10_0: 7, pm2_5: 6, time: 1591102052.0
Jun 02 08:47:32 nuc python3[3651105]: weewx[3651105] DEBUG user.MQTTSubscribe: MessageCallbackProvider data-> incoming topic: rtl_915/events, QOS: 0, retain: 0, payload: b'{"time":"1591102052","model":"Fineoffset-WH0290","id":203,"pm2_5_ug_m3":6,"pm10_0_ug_m3":7,"mic":"CRC"}'
Jun 02 08:47:32 nuc python3[3651105]: weewx[3651105] DEBUG user.MQTTSubscribe: TopicManager data-> incoming rtl_915/events: id: 203.0, pm10_0: 7, pm2_5: 6, time: 1591102052.0
Jun 02 08:47:35 nuc python3[3651105]: weewx[3651105] DEBUG user.sdr: lines=[]

And again, this is how I have it set...sensor sends every 10 minutes.

Jun 02 08:47:32 nuc python3[3651105]: weewx[3651105] DEBUG user.MQTTSubscribe: MessageCallbackProvider data-> incoming topic: rtl_915/events, QOS: 0, retain: 0, payload: b'{"time":"1591102052","model":"Fineoffset-WH0290","id":203,"pm2_5_ug_m3":6,"pm10_0_ug_m3":7,"mic":"CRC"}'
Jun 02 08:47:32 nuc python3[3651105]: weewx[3651105] DEBUG user.MQTTSubscribe: TopicManager data-> incoming rtl_915/events: id: 203.0, pm10_0: 7, pm2_5: 6, time: 1591102052.0
Jun 02 08:47:32 nuc python3[3651105]: weewx[3651105] DEBUG user.MQTTSubscribe: MessageCallbackProvider data-> incoming topic: rtl_915/events, QOS: 0, retain: 0, payload: b'{"time":"1591102052","model":"Fineoffset-WH0290","id":203,"pm2_5_ug_m3":6,"pm10_0_ug_m3":7,"mic":"CRC"}'
Jun 02 08:47:32 nuc python3[3651105]: weewx[3651105] DEBUG user.MQTTSubscribe: TopicManager data-> incoming rtl_915/events: id: 203.0, pm10_0: 7, pm2_5: 6, time: 1591102052.0
Jun 02 08:47:35 nuc python3[3651105]: weewx[3651105] DEBUG user.sdr: lines=[]

[bellrichm]

I knew it was going to be large. If you attach it, i’d still be happy to look at it.
Or...
Not sure exactly what i’m looking for. But start by looking for “ MQTTSubscribeService data-> final record is”. This is what the archive record is and should have your air quality field in it.
After verifying that, “ MQTTSubscribeService adding” and “ MQTTSubscribeService target_data after cache lookup is:” would give some insight into the cached data.

[dshanske]

@bellrichm I think I might have figured out the problem. I was putting expires_after under the message_callback area, under the normal fields. If I reread your config...it shouldn't go there, should it?

[dshanske]

Okay...now it's working. Sometimes it is the stupidest things.

[bellrichm]

Good to know. I will package it up into a formal release.
Yeah, the configuration file has evolved into a bit of a mess. I am thinking about cleaning it up, but it would involve breaking changes to existing configs.