Readings or compensation factors for temperature and humidity seem way off about enviro-monitor

[roscoe81]

The variables within my temperature and humidity compensation algorithms are set based on the calibration and regression analysis that I undertook with my code and my case design. It's therefore not surprising that the results vary when you change either of those factors. My suggestion is that you either stick to using my case design or go through your own calibration and regression exercise with your preferred case. You can then use that to adjust the variables in my temperature and humidity compensation algorithms to suit your case. I wish i could offer a simpler solution but I found that the temperature and humidity sensor on the Enviro+ is greatly impacted by the heat generated by the Raspberry Pi and by the board itself.

[mrfocus22]

I had already planned on printing the case, both indoor and outdoor version eventually.

In the meantime, I'm looking at your regression analysis code and having no experience in machine learning, I'm trying to understand what source you used as the 'Real Temperature'.

From looking at the code, essentially I create a json file with a pair of both data output from the Enviro+ as well as some verifiable source for temperature and humidity, gather enough data and run it through that code to come up with my own factors?

[vistvanv]

Hey, I've also just got started with an enviro+ and trying to set it up for outdoor use. and would like to use the regression analyser but that level of code is beyond my knowledge?.
Would you be kind to give some hints on how to use it?

[roscoe81]

Hey, I've also just got started with an enviro+ and trying to set it up for outdoor use. and would like to use the regression analyser but that level of code is beyond my knowledge?. Would you be kind to give some hints on how to use it?

Here are a few hints @vistvanv but it's process that does have quite a few steps and possibly requires you to do some customisation to suit your setup. The first (and most time consuming) step, requires the logging of "real" temperature, humidity and air pressure levels against the levels being measured by the Enviro+. To do that, you will need some calibrated external sensors that can be read by the Enviro Monitor, set up the Enviro Monitor code to read those sensors and vary the temperature, humidity and air pressure over the ranges you wish to calibrate. In my case, I used some Aeotec Z-wave Multisensors that had their data captured by Domoticz and sent via mqtt messages to the Enviro Monitor. The variables "use_external_temp_hum" and "use_external_barometer" in lines 1993 and 1994 need to be set to True during the logging process. "enable_receive_data_from_homemanager" and "enable_climate_and_gas_logging" also need to be set to true and the mqtt fields set up in the config.json file to complete the setup for data logging. You should be able to replicate this process if you're using identical external sensors and Domoticz, but you'll likely need to make extensive code modifications to the ExternalSensors Class starting in line 987 if you wish to use something different. The data logging is triggered in line 2405 and created by the code starting in line 422. It creates a text file that you then need to convert to a json format before processing it with the Regression Analyser code.

Running the Regression Analyser code is a straightforward process, once you have your captured data in json format and stored in the same directory as the code. It produces linear, quad and cubic fit coefficients and their coefficients of determination (r-squared). You can then use the coefficients in the Enviro Monitor code staring in line 1819. Once you insert those coefficients, you can then use the Regression Analyser code to provide a report on their effectiveness in producing accurate measurements. It will also show graphs of how the data fits the regression curves and how ineffective it is to just use CPU temperature to undertake compensation.

I've pasted an example of a report from the Regression Analyser below.

Hope that helps and good luck.

{'Temp and Hum Performance': {}, 'Gas Sensor Performance': {}}

Output Temp and Real Temperature
Slope: 1.1189
Intercept: -2.7532
R2: 0.96
Quad Fit Inliers coef:[[0.0000 1.0936 0.0006]] - b:-2.4723
R2: 0.96
Cubic Fit Inliers coef:[[0.00000 3.50211 -0.10663 0.00157]] - b:-20.33691
R2: 0.96

Output Humidity and Real Humidity
Slope: 1.1145
Intercept: -5.8361
R2: 0.95
Quad Fit Inliers coef:[[0.0000 0.8776 0.0019]] - b:1.1927
R2: 0.95
Cubic Fit Inliers coef:[[0.00000 3.64018 -0.04687 0.00028]] - b:-49.21718
R2: 0.96

Raw Temperature and Real Temperature
Slope: 0.9093
Intercept: -3.2561
R2: 0.97
Quad Fit Inliers coef:[[0.0000 0.9630 -0.0010]] - b:-4.0045
R2: 0.97
Cubic Fit Inliers coef:[[0.00000 2.97577 -0.07229 0.00083]] - b:-22.75657
R2: 0.97

Raw Humidity and Real Humidity
Slope: 1.3340
Intercept: 9.7933
R2: 0.95
Quad Fit Inliers coef:[[0.0000 1.2818 0.0007]] - b:10.7836
R2: 0.95
Cubic Fit Inliers coef:[[0.00000 2.85507 -0.04330 0.00039]] - b:-6.90191
R2: 0.95

Real Temperature and RedRS
Slope: 2195.1415
Intercept: 242899.2639
R2: 0.03
Quad Fit Inliers coef:[[0.0000 1004.9090 -851.9276]] - b:248357.0657
R2: 0.10
Cubic Fit Inliers coef:[[0.00000 1307.83249 -859.84186 -14.18742]] - b:248607.53965
R2: 0.10

Raw Temperature and RedRS
Slope: 1194.2788
Intercept: 234408.7781
R2: 0.01
Quad Fit Inliers coef:[[0.0000 8291.8374 -684.4509]] - b:221667.1764
R2: 0.07
Cubic Fit Inliers coef:[[0.00000 9225.10682 -915.54203 14.01874]] - b:220945.53238
R2: 0.07

Real Humidity and RedRS
Slope: -9.6332
Intercept: 239390.0799
R2: 0.00
Quad Fit Inliers coef:[[0.0000 1075.8942 -42.8993]] - b:236975.0586
R2: 0.04
Cubic Fit Inliers coef:[[0.00000 1226.77042 6.78459 -1.73431]] - b:230668.60861
R2: 0.06

Raw Humidity and RedRS
Slope: 271.0004
Intercept: 241201.7089
R2: 0.00
Quad Fit Inliers coef:[[0.0000 -1195.9233 -69.1059]] - b:237393.1230
R2: 0.03
Cubic Fit Inliers coef:[[0.00000 -2609.26848 -250.75567 -4.80692]] - b:236948.60235
R2: 0.05

Bar and RedRS
Slope: -536.5062
Intercept: 243586.0350
R2: 0.01
Quad Fit Inliers coef:[[0.0000 2941.8684 -248.1485]] - b:236031.1015
R2: 0.04
Cubic Fit Inliers coef:[[0.00000 2020.54934 -17.84328 -12.07118]] - b:235433.15426
R2: 0.04

Real Temperature and OxiRS
Slope: 22168.9990
Intercept: 264566.4666
R2: 0.17
Quad Fit Inliers coef:[[0.0000 22033.8486 -96.7360]] - b:265186.1978
R2: 0.17
Cubic Fit Inliers coef:[[0.00000 29213.82347 -284.32190 -336.27409]] - b:271122.99978
R2: 0.18

Raw Temperature and OxiRS
Slope: 15950.3565
Intercept: 163124.7843
R2: 0.10
Quad Fit Inliers coef:[[0.0000 12409.2309 341.4874]] - b:169481.8452
R2: 0.10
Cubic Fit Inliers coef:[[0.00000 2392.61941 2821.74700 -150.46055]] - b:177227.12010
R2: 0.11

Real Humidity and OxiRS
Slope: -6767.3665
Intercept: 341063.7141
R2: 0.23
Quad Fit Inliers coef:[[0.0000 -3506.2472 -128.8771]] - b:333808.5560
R2: 0.25
Cubic Fit Inliers coef:[[0.00000 -3560.29362 -146.67475 0.62126]] - b:336067.63382
R2: 0.25

Raw Humidity and OxiRS
Slope: -7763.9161
Intercept: 170961.0970
R2: 0.16
Quad Fit Inliers coef:[[0.0000 -11869.0266 -193.3892]] - b:160302.9655
R2: 0.17
Cubic Fit Inliers coef:[[0.00000 -13287.76169 -375.73175 -4.82525]] - b:159856.74960
R2: 0.17

Bar and OxiRS
Slope: -12312.1872
Intercept: 327496.3745
R2: 0.15
Quad Fit Inliers coef:[[0.0000 2972.6147 -1090.4233]] - b:294298.2069
R2: 0.18
Cubic Fit Inliers coef:[[0.00000 637.51504 -506.71059 -30.59463]] - b:292782.69887
R2: 0.18

Real Temperature and NH3RS
Slope: 14016.8714
Intercept: 359749.8102
R2: 0.31
Quad Fit Inliers coef:[[0.0000 14192.5576 125.7501]] - b:358944.2026
R2: 0.31
Cubic Fit Inliers coef:[[0.00000 13771.01993 136.76334 19.74271]] - b:358595.65189
R2: 0.31

Raw Temperature and NH3RS
Slope: 10933.5341
Intercept: 292186.5230
R2: 0.22
Quad Fit Inliers coef:[[0.0000 8005.6670 282.3479]] - b:297442.6564
R2: 0.22
Cubic Fit Inliers coef:[[0.00000 13380.07948 -1048.43526 80.72960]] - b:293286.92950
R2: 0.22

Real Humidity and NH3RS
Slope: -4396.7028
Intercept: 410095.1859
R2: 0.44
Quad Fit Inliers coef:[[0.0000 -3316.6905 -42.6813]] - b:407692.4341
R2: 0.45
Cubic Fit Inliers coef:[[0.00000 -3224.13727 -12.20336 -1.06389]] - b:403823.81781
R2: 0.45

Raw Humidity and NH3RS
Slope: -5376.6422
Intercept: 297159.8782
R2: 0.35
Quad Fit Inliers coef:[[0.0000 -6788.9795 -66.5343]] - b:293493.0156
R2: 0.36
Cubic Fit Inliers coef:[[0.00000 -8514.86096 -288.35278 -5.86989]] - b:292950.19709
R2: 0.36

Bar and NH3RS
Slope: -6281.7876
Intercept: 387331.9891
R2: 0.18
Quad Fit Inliers coef:[[0.0000 -2547.3457 -266.4164]] - b:379220.8845
R2: 0.19
Cubic Fit Inliers coef:[[0.00000 -12024.61965 2102.64961 -124.17187]] - b:373070.01867
R2: 0.23

Temp and Hum Performance Summary
Output Temp Real Temperature {'Slope': array([1.11888864]), 'Intercept': array([-2.75316622]), 'R2': 0.9618026588956703}
Output Humidity Real Humidity {'Slope': array([1.11447276]), 'Intercept': array([-5.836107]), 'R2': 0.9542486305856879}
Raw Temperature Real Temperature {'Slope': array([0.90931205]), 'Intercept': array([-3.25606213]), 'R2': 0.9658834067738279}
Raw Humidity Real Humidity {'Slope': array([1.3340499]), 'Intercept': array([9.79334964]), 'R2': 0.9528821924010881}

Gas Sensor Performance Summary
Real Temperature RedRS {'Slope': array([2195.14153691]), 'Intercept': array([242899.26386365]), 'R2': 0.032356948693239906}
Raw Temperature RedRS {'Slope': array([1194.27876931]), 'Intercept': array([234408.77805682]), 'R2': 0.011188007330676264}
Real Humidity RedRS {'Slope': array([-9.63318654]), 'Intercept': array([239390.07993833]), 'R2': 9.088063978701655e-06}
Raw Humidity RedRS {'Slope': array([271.0004355]), 'Intercept': array([241201.70888197]), 'R2': 0.0038509375338504626}
Bar RedRS {'Slope': array([-536.50615097]), 'Intercept': array([243586.03497138]), 'R2': 0.005667526738205475}
Real Temperature OxiRS {'Slope': array([22168.99899349]), 'Intercept': array([264566.46660772]), 'R2': 0.16979240541749274}
Raw Temperature OxiRS {'Slope': array([15950.35652211]), 'Intercept': array([163124.7843364]), 'R2': 0.10267523291691893}
Real Humidity OxiRS {'Slope': array([-6767.36647378]), 'Intercept': array([341063.71410945]), 'R2': 0.23075675056797473}
Raw Humidity OxiRS {'Slope': array([-7763.91609168]), 'Intercept': array([170961.09696277]), 'R2': 0.16261917416672467}
Bar OxiRS {'Slope': array([-12312.18724159]), 'Intercept': array([327496.37454883]), 'R2': 0.1535670983997286}
Real Temperature NH3RS {'Slope': array([14016.87138671]), 'Intercept': array([359749.81020654]), 'R2': 0.3062711681919147}
Raw Temperature NH3RS {'Slope': array([10933.53405726]), 'Intercept': array([292186.52299448]), 'R2': 0.21768268006252545}
Real Humidity NH3RS {'Slope': array([-4396.70284153]), 'Intercept': array([410095.18592539]), 'R2': 0.43948807600027895}
Raw Humidity NH3RS {'Slope': array([-5376.64221651]), 'Intercept': array([297159.87823158]), 'R2': 0.35189265747179943}
Bar NH3RS {'Slope': array([-6281.78756879]), 'Intercept': array([387331.98910214]), 'R2': 0.1803732337783318}

Improved CPU Temp Factor
Slope: -0.0526
Intercept: 16.8176
R2: 0.02
Quad Fit Inliers coef:[[0.0000 0.5946 -0.0115]] - b:7.7916
R2: 0.04
Cubic Fit Inliers coef:[[0.00000 0.19347 0.00274 -0.00017]] - b:11.52897
R2: 0.04

Improved Comp Temp
Slope: 0.9101
Intercept: 11.2909
R2: 0.96
Quad Fit Inliers coef:[[0.0000 0.9073 0.0001]] - b:11.3076
R2: 0.96
Cubic Fit Inliers coef:[[0.00000 1.28120 -0.03146 0.00084]] - b:9.90493
R2: 0.04

Improved Temp Performance
Slope: 1.0000
Intercept: -0.0000
R2: 0.96
Quad Fit Inliers coef:[[0.0000 0.9936 0.0001]] - b:0.0708
R2: 0.96
Cubic Fit Inliers coef:[[0.00000 2.69322 -0.07588 0.00112]] - b:-12.44134
R2: 0.96

Improved Output Humidity
Slope: 0.9782
Intercept: 7.9983
R2: 0.95
Quad Fit Inliers coef:[[0.0000 0.8950 0.0008]] - b:10.2181
R2: 0.95
Cubic Fit Inliers coef:[[0.00000 2.19971 -0.02516 0.00016]] - b:-10.48673
R2: 0.95

IMPROVED TEMP AND HUM COMPENSATION SUMMARY
CPU Temp Factor. Slope: [-0.05258922] Intercept: [16.81756342] R2: 0.020789784151703072
Improved Comp Temp. Slope: [0.9101445] Intercept: [11.29094449] R2: 0.9626563606631459
Improved Temp. Slope: [1.] Intercept: [-2.48689958e-14] R2: 0.9626563606631459
Improved Comp Hum. Slope: [0.97816389] Intercept: [7.99830796] R2: 0.953407253415441

[Sindar-sudo]

One question - have you ever tried your setup in cold temperatures? I'm using this setup with the case and everything for a few months now (outside with weather cover), however, it would seem that as long as temperature is in positive Celsius, it seems to read correctly, but as soon as it slips into colder weather, it drastically overestimates the coldness. If it's like -5C it would show -12C or so...
So I set up 3 different sensors right next to enviro with DHT22, HDC1080 and BMP280 connected to ESP32. All of them read raw temperatures and all of them are slightly warmer than the real temps outside (physical termomether was also set up next to these), but they are all "in tune" with one another while enviro with your script reads similarly when it's warm, but starts floating much colder temps when it gets into negative C.
So... I also tried reading raw temps from Enviro's BME280 via python and raw values for temperature are just WAAAY off - it reads about 7C when it's really ~0C outside. Your script brings that to -0.7C which is pretty much true. So the calculation IS doing something very useful, but I don't understand why it goes crazy when it's negative C and also why raw values are so wrong. Perhaps BME280 is just broken on my device, I may look into replacing it. Or perhaps I need to adjust the calculation somehow when it's negative C.
My question summed up here is - did you ever try it in negative C and how did it work out?
I attached an image of the graph of all the temperature sensors. It's not THAT cold at the moment, but you can see what I mean - sensors are mostly "in tune" when it's warmer, but the purple line (enviro sensor) goes rogue when it gets colder. It's much worse when it gets even colder.

[roscoe81]

Thanks for the feedback @Sindar-sudo. No, I didn't run the calibration exercise at negative C temperatures. I had no easy way of replicating those temperatures at the time, so it's not surprising that the compensation algorithm isn't correct. I expect that the BME 280's raw temperature reading is being affected by heating from the Pi Zero and Enviro+ components and the algorithm is over-compensating.

I see two potential solutions. Either you use an external sensor and capture its data via mqtt messages for display on your Enviro Monitor or provide me with some negative C data points (raw BME280 temp and hum vs. real temp and hum) that I can add to my existing data points (0C to 40C), rerun the correlation code and produce new coefficients.