Comments (15)
What do you think of adding a ‘conversion function’ for fields? Then you could something like this.
[[[topic]]]
[[[[field]]]]
name = weewx_name
conversion_func = lambda x: True if x == ‘ON’ else False
from weewx-mqttsubscribe.
Yes, that sounds promising.
I would surround the call of that function by try:
except:
as the value of x comes from an external source and we cannot control it from a WeeWX point of view. It could be None or something really unexpected.
from weewx-mqttsubscribe.
Good point about the try/except. I had to that for the simple calls to convert the values. But, I am thinking that I might have ‘catch’ more (all?) exceptions, because who knows what the lambda function might raise....
from weewx-mqttsubscribe.
Doing some experimentation, I was able to get SyntaxError and NameError exceptions raised at ‘compile’ time. Since the expression is ‘compiled’ at initialization of WeeWX, if they are not handled it results in WeeWX not starting. So at this time, I see no reason to do anything more for these.
At execution time, I managed to get ValueError, NameError, and TypeError exceptions raised. The NameError and TypeError seemed to be related to ‘bad’ expressions and the ValueError to bad input. So, the plan is to continue handle the ValueError when ‘converting’ the data and either setting the converted data to ‘None’ or letting the exception bubble up. The NameError and TypeError will bubble up to MQTTSubscribe’s generic exception handler that logs the exception and ignores the MQTT message.
Based on real world experiences, this can easily be refined/changed.
from weewx-mqttsubscribe.
@roe-dl
Issue 126 is functionally complete. All that is left is to improve the logging when ‘running the conversion function’ raises an exception. If you want to do some beta testing, it should be safe to install from this branch, https://github.com/bellrichm/WeeWX-MQTTSubscribe/tree/issue126. Or if you prefer, I can create a prerelease.
rich
from weewx-mqttsubscribe.
Yes, that would be nice.
from weewx-mqttsubscribe.
Great. I have created this release.
https://github.com/bellrichm/WeeWX-MQTTSubscribe/releases/tag/v2.0.0-rc08
Thanks for any and all feedback and testing.
from weewx-mqttsubscribe.
I added the following configuration to the end of the [[topic]] section:
[[[wetter/regnet]]]
name = isRaining
conversion_func = lambda x: True if x == 'ON' else False
conversion_type = float
I subscribed to the MQTT broker by MQTT Explorer. The topic "wetter/regnet" is there, and the sensor publishs to it once every minute.
I switched on logging for MQTTSubscribeService and set logging_level to "DEBUG". Every record is logged to syslog. I see other topics received by MQTTSubscribe, but the topic above I do not see.
grep isRaining /var/log/syslog
displays:
Apr 5 19:50:47 LokalWiki weewx[1892476] DEBUG user.MQTTSubscribe: (Service) MQTTSUBscriber sanitized_service_dict is {'username': 'xxxx', 'host': '192.168.xxx.xxx', 'topics': {'unit_system': 'METRIC', 'use_server_datatime': 'True', 'ignore_start_time': 'True', 'ignore_end_time': 'True', 'use_topic_as_fieldname': 'True', 'pegel/566055/W_cm': {'name': 'W566055v', 'conversion_type': 'float', 'expires_after': '4000'}, 'pegel/566055/Wtrend_cmph': {'name': 'W566055t', 'conversion_type': 'float', 'expires_after': '4000'}, 'pegel/566055/Q_m3ps': {'name': 'Q566055v', 'conversion_type': 'float', 'expires_after': '4000'}, 'pegel/566055/Qtrend_m3psph': {'name': 'Q566055t', 'conversion_type': 'float', 'expires_after': '4000'}, 'pegel/567470/W_cm': {'name': 'W567470v', 'conversion_type': 'float', 'expires_after': '4000'}, 'pegel/567470/Wtrend_cmph': {'name': 'W567470t', 'conversion_type': 'float', 'expires_after': '4000'}, 'pegel/567470/Q_m3ps': {'name': 'Q567470v', 'conversion_type': 'float', 'expires_after': '4000'}, 'pegel/567470/Qtrend_m3psph': {'name': 'Q567470t', 'conversion_type': 'float', 'expires_after': '4000'}, 'pegel/567470/alarmcolor': {'name': 'W567470c', 'conversion_type': 'None', 'expires_after': '4000'}, 'pegel/566055/alarmcolor': {'name': 'W566055c', 'conversion_type': 'None', 'expires_after': '4000'}, 'wetter/regnet': {'name': 'isRaining', 'conversion_func': "lambda x: True if x == 'ON' else False", 'conversion_type': 'float'}}, 'keepalive': '60', 'logging_level': 'DEBUG', 'message_callback': {'type': 'individual'}, 'enable': 'True', 'port': '1883', 'log': 'True', 'binding': 'loop'}
This is the only record where "isRaining" is logged between start and stop of WeeWX. All the loop records do not include it, but they include other topics received by MQTTSubscribe.
I tried different values for "conversion_type" including "bool".
Now I am not sure what to do next.
from weewx-mqttsubscribe.
Try grepping on the topic, wetter/regent. The early processing of MQTT messages includes the topic in the log. I believe there should also be some messages about subscribing to it.
from weewx-mqttsubscribe.
You are right:
Apr 5 19:50:47 LokalWiki weewx[1892476] DEBUG user.MQTTSubscribe: (Service) MQTTSubscribe MQTT: Sending SUBSCRIBE (d0, m11) [(b'wetter/regnet', 0)]
Apr 5 19:50:47 LokalWiki weewx[1892476] INFO user.MQTTSubscribe: (Service) Subscribing to wetter/regnet has a mid 11 and rc 0
Apr 5 20:15:38 LokalWiki weewx[1894069] INFO user.MQTTSubscribe: (Service) Subscribing to wetter/regnet has a mid 11 and rc 0
from weewx-mqttsubscribe.
By looking into the code I found that there is a log level "TRACE", too. I tried it, but to no other result.
MQTTSub.log
Additionally I changed the configuration a little bit. I commented out the line "conversion_type". No difference. I changed True to 1.0 and False to 0.0. No difference.
Actually I see no further way to debug.
from weewx-mqttsubscribe.
The code snippet is implying that the subscription is never fully completed. If you grep on a different topic, you should see something like
Subscribing to...
Subscribed to...
and the when the message is received
Incoming....
I can poke around here, but without seeing more of the log I don’t think I will be able to help much.
The TRACE level just logs the internal processing in even more detail. For this topic, we haven’t yet gotten the subscription to work.
from weewx-mqttsubscribe.
I thought TRACE is more than DEBUG and includes all the debug messages. But that seems not the case, if I consider your reply. So I repeated the test with setting DEBUG. Especially there is no "incoming" message, if I set TRACE, but there are such messages if I set DEBUG.
For another topic I saw:
Apr 6 14:55:54 LokalWiki weewx[1962589] INFO user.MQTTSubscribe: (Service) Subscribing to pegel/566055/Wtrend_cmph has a mid 2 and rc 0
Apr 6 14:55:54 LokalWiki weewx[1962589] INFO user.MQTTSubscribe: (Service) Subscribed to mid: 2 is size 1 has a QOS of 0
For our topic "wetter/regnet" I see:
Apr 6 14:55:54 LokalWiki weewx[1962589] INFO user.MQTTSubscribe: (Service) Subscribing to wetter/regnet has a mid 11 and rc 0
Apr 6 14:55:54 LokalWiki weewx[1962589] INFO user.MQTTSubscribe: (Service) Subscribed to mid: 11 is size 1 has a QOS of 0
The "subscribed" message never includes a topic but a value called "mid", that is also found within the "subscribing" message. So I guess the message with the same "mid" value belong together.
I can't see a real difference between the two pairs of messages.
But your comment led me into another direction. Indeed, there are "incoming" messages for the other topics, and for "wetter/regnet" they are not. So I looked at the access rights of the MQTT broker. While anonymous clients can subscribe to all topics, named clients cannot. And for the user name that the MQTTSubscribe service uses, I had not granted the right to read "wetter/regnet". After changing that, the value is received.
Now every minute there is an "incoming" message for "wetter/regnet". And the value is processed within WeeWX. The value is converted as said by the conversion function. It is working as expected.
Thank you very much for all your work.
And sorry for wasting your time. It is the old rule: the problem is situated between monitor and chair.
from weewx-mqttsubscribe.
No apology needed. Only seeing part of the picture can make it a challenge. Glad I could help.
Yes, the ‘mid’ ties ‘subscribing’ and ‘subscribed’ log messages together. I believe it is short for ‘message id’. It is the information that MQTT client provides. I suppose I could do some mapping, but for debugging it doesn’t seem worth the effort.
The ‘difference’ is subtle, if at all. The ‘subscribing’ message is just logging that a ‘subscription request’ has been ‘submitted’. The ‘subscribed’ means that the subscription was ‘accepted/completed’. I know the wording could be improved, any suggestions are welcome. (I am also working on a wiki page...)
Your understanding of how TRACE level debugging should work is correct. Seems like a bug. I’ve opened #128 to track.
Once I get the logging on exceptions improved, I should be able to package this up into a formal release. Thanks for your help.
from weewx-mqttsubscribe.
So I guess, this issue can be considered solved and can be closed.
Thank you again.
from weewx-mqttsubscribe.
Related Issues (20)
- Data not processed into reports HOT 3
- Setting log level to TRACE ‘turns off’ DEBUG messages HOT 1
- add apt-get commands to paho-mqtt installation instructions HOT 2
- typos in examples HOT 3
- WeeWxIOError: ('Certificate subject does not match remote hostname.',) when using AWS IoT Core HOT 3
- Support TLS setup for MQTT brokers HOT 4
- Help needed in configuring JSON HOT 2
- 5 Minute Record Ignored rc2 HOT 5
- Rain Cumulative, Should Be Total HOT 9
- Ignore
- Do not run codecov and sonar on pull requests
- Error with date fields HOT 6
- expires_after = None is ignored HOT 1
- Unsubscribed Topics HOT 8
- Need to migrate off of nose HOT 2
- Int Values Stored As Floats in DB HOT 29
- Possible problem with rain and contains_total HOT 2
- problem while installing... HOT 4
- problem reading values... HOT 7
- WeeWX crashes because MQTTSubscribe fails network connect on cold start or reboot HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from weewx-mqttsubscribe.