Jabber module does not work about fhem-docker

[git-developer]

The v4 images do not contain Authen::SASL::Cyrus. I can confirm that Jabber is working in 4.0.0-beta8-bullseye and latest dev images (both minimal and extended).

[sidey79]

@elrozone

Which Image variant/version ist affected?

[elronzo]

ghcr.io/fhem/fhem/fhem-docker:bullseye

[git-developer]

Probably related: #40

[sidey79]

Does this also occure with the v4 Beta 8 Image?

[git-developer]

I tried v4-beta8, but it does not properly boot here. I'm coming from v3.3.

USB

Error message in the log:

Can't locate Device/SerialPort.pm in @INC (you may need to install the Device::SerialPort module) (@INC contains: ./lib ./FHEM . /usr/local/lib/perl5/site_perl/5.36.3/arm-linux-gnueabihf-64int /usr/local/lib/perl5/site_perl/5.36.3 /usr/local/lib/perl5/vendor_perl/5.36.3/arm-linux-gnueabihf-64int /usr/local/lib/perl5/vendor_perl/5.36.3 /usr/local/lib/perl5/5.36.3/arm-linux-gnueabihf-64int /usr/local/lib/perl5/5.36.3 ./FHEM/lib) at FHEM/DevIo.pm line 612, <$fh> line 405.

This breaks all USB devices.

Jabber

Error message in the log:

E: Unable to locate package libauthen-sasl-cyrus-perl

This is coming from my pre-init.sh where I run apt-get remove libauthen-sasl-cyrus-perl. So I guess the package is not installed which could mean that the Jabber issue does not apply to v4-beta8. I can't confirm that for sure because I had to revert to v3 because of the USB problem.

Image size

What happened to the minimal image? It's twice as big as before:

$ docker images | grep fhem
ghcr.io/fhem/fhem-minimal-docker   4.0.0-beta8-bullseye                 8947fca5f21e   5 days ago      1.34GB
ghcr.io/fhem/fhem-minimal-docker   3.3.1-bullseye                       c9b1f0c873e7   4 months ago    635MB

[sidey79]

The usb problem seems to be specific to arm Images.

[sidey79]

Can you please test the recent development image.
The image should support serial devices also on arm plattform.

docker pull ghcr.io/fhem/fhem-docker:dev-bullseye

[git-developer]

The behavior is unchanged for ghcr.io/fhem/fhem-minimal-docker:dev-bullseye.

$ docker images | grep fhem
ghcr.io/fhem/fhem-minimal-docker   dev-bullseye                         cf28244b08a3   12 hours ago    905MB
ghcr.io/fhem/fhem-minimal-docker   4.0.0-beta8-bullseye                 8947fca5f21e   7 days ago      1.34GB
ghcr.io/fhem/fhem-minimal-docker   3.3.1-bullseye                       c9b1f0c873e7   4 months ago    635MB

$ docker inspect ghcr.io/fhem/fhem-minimal-docker:dev-bullseye | grep 2024
        "Created": "2024-04-11T15:04:23.351924779Z",
                "org.opencontainers.image.created": "2024-04-11T15:04:19.486Z",

[git-developer]

I tried to install the packages manually in pre-init.sh:

apt-get update && apt-get install -y libdevice-serialport-perl libxml-bare-perl

According to the log, they are installed, but still missing for FHEM.

[git-developer]

Looks like perl libraries installed by apt are not available.

From a shell within ghcr.io/fhem/fhem-minimal-docker:dev-bullseye (using XML::Bare as an example, may be replaced by any other lib):

$ perl -e 'use XML::Bare;'
Can't locate XML/Bare.pm in @INC (you may need to install the XML::Bare module) (@INC contains: /usr/local/lib/perl5/site_perl/5.36.3/arm-linux-gnueabihf-64int /usr/local/lib/perl5/site_perl/5.36.3 /usr/local/lib/perl5/vendor_perl/5.36.3/arm-linux-gnueabihf-64int /usr/local/lib/perl5/vendor_perl/5.36.3 /usr/local/lib/perl5/5.36.3/arm-linux-gnueabihf-64int /usr/local/lib/perl5/5.36.3) at -e line 1.

$ apt update && apt install -qqy libxml-bare-perl
[...]
Preparing to unpack .../libxml-bare-perl_0.53-1+b8_armhf.deb ...
Unpacking libxml-bare-perl (0.53-1+b8) ...
Setting up libxml-bare-perl (0.53-1+b8) ...
$ perl -e 'use XML::Bare;'
Can't locate XML/Bare.pm in @INC (you may need to install the XML::Bare module) (@INC contains: /usr/local/lib/perl5/site_perl/5.36.3/arm-linux-gnueabihf-64int /usr/local/lib/perl5/site_perl/5.36.3 /usr/local/lib/perl5/vendor_perl/5.36.3/arm-linux-gnueabihf-64int /usr/local/lib/perl5/vendor_perl/5.36.3 /usr/local/lib/perl5/5.36.3/arm-linux-gnueabihf-64int /usr/local/lib/perl5/5.36.3) at -e line 1.

$ dpkg -S Bare.pm
libxml-bare-perl: /usr/lib/arm-linux-gnueabihf/perl5/5.32/XML/Bare.pm
$ PERL5LIB=/usr/lib/arm-linux-gnueabihf/perl5/5.32 perl -e 'use XML::Bare;'
Can't load '/usr/lib/arm-linux-gnueabihf/perl5/5.32/auto/XML/Bare/Bare.so' for module XML::Bare: /usr/lib/arm-linux-gnueabihf/perl5/5.32/auto/XML/Bare/Bare.so: undefined symbol: PL_thr_key at /usr/local/lib/perl5/5.36.3/arm-linux-gnueabihf-64int/DynaLoader.pm line 206.
 at -e line 1.
Compilation failed in require at -e line 1.
BEGIN failed--compilation aborted at -e line 1.

$ cpanm XML::Bare
[...]
$ perl -e 'use XML::Bare;'
$ echo $?
0

[git-developer]

In v3.3 image, the library path is (different from v4 above):

@INC contains: /etc/perl /usr/local/lib/arm-linux-gnueabihf/perl/5.32.1 /usr/local/share/perl/5.32.1 /usr/lib/arm-linux-gnueabihf/perl5/5.32 /usr/share/perl5 /usr/lib/arm-linux-gnueabihf/perl-base /usr/lib/arm-linux-gnueabihf/perl/5.32 /usr/share/perl/5.32 /usr/local/lib/site_perl

[sidey79]

Can you please try the full Image?

I think, the Net::XML package isn't required by any of the modules available in the svn repo.

I checked, that it ist available in the full Image.

Installing perl packages via apt / pre-init.sh isn't the way to go in v4.

[git-developer]

Thanks for caring about this issue. I'm ready to give feedback in case you don't have arm hardware available.

1. I tried the full image. Device::SerialPort is still missing.

2. I used the module XML::Base above just as an example to show that the installation of a perl library via apt has no effect. The module has no relevance for this issue.

3. Installing perl packages via apt / pre-init.sh isn't the way to go in v4.

   • I tried to find out the (breaking) changes between v3 and v4. The README has a section about removal of env variables, but I didn't find anything about 'apt / pre-init.sh'. Is there any other documentation about the changes, and how to migrate?
   • I noticed that a perl library (nmap-parser) is installed via apt in the Dockerfile. Is that OK for v4?

4. I read your suggestion to add an inline Dockerfile for customizations. I understand that an additional layer is better from the performance perspective because the installation occurs once at build time and not on every container start. But this means I'm no longer just a user but maintainer of a layer. The layer is able to outdate, so I have to think about building the image (my layer) from time to time. I prefer using the pre-init.sh. Since it is running automatically on container start, I don't have to think about anything. I don't care about longer boot time of the container. Another point is that using an inline Dockerfile ties the image to docker compose. May not be a problem, but I don't like the idea.

5. Thanks for having a look at the image size. It is now down from 1.3 GB to 900 MB. Previously, the minimal image was ~600 MB. I don't use the features of the full image. Is there any chance to further reduce the image size? I'd appreciate that.

[git-developer]

Device::SerialPort is still missing.

I'm curious why this is the case. It looks like parse-METAJson.pl is collecting all required perl dependencies by parsing the FHEM source files. Some of them contain a META.json section that declares dependencies:

Modules with META.json

# grep -H -r -l '=for :application/json;q=META.json' src/fhem/trunk/fhem/FHEM/ | sort
src/fhem/trunk/fhem/FHEM/00_OW2S0SMSGUARD.pm
src/fhem/trunk/fhem/FHEM/00_SIGNALduino.pm
src/fhem/trunk/fhem/FHEM/00_THZ.pm
src/fhem/trunk/fhem/FHEM/10_FS10.pm
src/fhem/trunk/fhem/FHEM/10_RESIDENTS.pm
src/fhem/trunk/fhem/FHEM/10_SD_GT.pm
src/fhem/trunk/fhem/FHEM/14_FLAMINGO.pm
src/fhem/trunk/fhem/FHEM/14_Hideki.pm
src/fhem/trunk/fhem/FHEM/14_SD_BELL.pm
src/fhem/trunk/fhem/FHEM/14_SD_UT.pm
src/fhem/trunk/fhem/FHEM/14_SD_WS07.pm
src/fhem/trunk/fhem/FHEM/14_SD_WS09.pm
src/fhem/trunk/fhem/FHEM/14_SD_WS_Maverick.pm
src/fhem/trunk/fhem/FHEM/14_SD_WS.pm
src/fhem/trunk/fhem/FHEM/20_GUEST.pm
src/fhem/trunk/fhem/FHEM/20_PET.pm
src/fhem/trunk/fhem/FHEM/20_ROOMMATE.pm
src/fhem/trunk/fhem/FHEM/30_HUEBridge.pm
src/fhem/trunk/fhem/FHEM/30_tradfri.pm
src/fhem/trunk/fhem/FHEM/31_HUEDevice.pm
src/fhem/trunk/fhem/FHEM/38_BEOK.pm
src/fhem/trunk/fhem/FHEM/39_alexa.pm
src/fhem/trunk/fhem/FHEM/41_OREGON.pm
src/fhem/trunk/fhem/FHEM/42_AptToDate.pm
src/fhem/trunk/fhem/FHEM/42_npmjs.pm
src/fhem/trunk/fhem/FHEM/46_TeslaPowerwall2AC.pm
src/fhem/trunk/fhem/FHEM/49_SSCam.pm
src/fhem/trunk/fhem/FHEM/49_SSCamSTRM.pm
src/fhem/trunk/fhem/FHEM/50_HP1000.pm
src/fhem/trunk/fhem/FHEM/50_SSChatBot.pm
src/fhem/trunk/fhem/FHEM/50_SSFile.pm
src/fhem/trunk/fhem/FHEM/57_SSCal.pm
src/fhem/trunk/fhem/FHEM/58_HVAC_DaikinAC.pm
src/fhem/trunk/fhem/FHEM/59_Twilight.pm
src/fhem/trunk/fhem/FHEM/59_Weather.pm
src/fhem/trunk/fhem/FHEM/59_Wunderground.pm
src/fhem/trunk/fhem/FHEM/59_WUup.pm
src/fhem/trunk/fhem/FHEM/60_Watches.pm
src/fhem/trunk/fhem/FHEM/70_ENIGMA2.pm
src/fhem/trunk/fhem/FHEM/70_LaMetric2.pm
src/fhem/trunk/fhem/FHEM/70_PHTV.pm
src/fhem/trunk/fhem/FHEM/70_Pushover.pm
src/fhem/trunk/fhem/FHEM/70_PylonLowVoltage.pm
src/fhem/trunk/fhem/FHEM/73_AMADCommBridge.pm
src/fhem/trunk/fhem/FHEM/73_AutoShuttersControl.pm
src/fhem/trunk/fhem/FHEM/73_DoorBird.pm
src/fhem/trunk/fhem/FHEM/73_ElectricityCalculator.pm
src/fhem/trunk/fhem/FHEM/73_GardenaSmartBridge.pm
src/fhem/trunk/fhem/FHEM/73_GasCalculator.pm
src/fhem/trunk/fhem/FHEM/73_km200.pm
src/fhem/trunk/fhem/FHEM/73_NUKIBridge.pm
src/fhem/trunk/fhem/FHEM/73_WaterCalculator.pm
src/fhem/trunk/fhem/FHEM/74_AMADDevice.pm
src/fhem/trunk/fhem/FHEM/74_GardenaSmartDevice.pm
src/fhem/trunk/fhem/FHEM/74_NUKIDevice.pm
src/fhem/trunk/fhem/FHEM/74_THINKINGCLEANER.pm
src/fhem/trunk/fhem/FHEM/74_XiaomiBTLESens.pm
src/fhem/trunk/fhem/FHEM/75_msgConfig.pm
src/fhem/trunk/fhem/FHEM/75_MSG.pm
src/fhem/trunk/fhem/FHEM/76_SMAInverter.pm
src/fhem/trunk/fhem/FHEM/76_SolarForecast.pm
src/fhem/trunk/fhem/FHEM/77_SMAEM.pm
src/fhem/trunk/fhem/FHEM/77_UWZ.pm
src/fhem/trunk/fhem/FHEM/82_LGTV_WebOS.pm
src/fhem/trunk/fhem/FHEM/93_DbLog.pm
src/fhem/trunk/fhem/FHEM/93_DbRep.pm
src/fhem/trunk/fhem/FHEM/93_Log2Syslog.pm
src/fhem/trunk/fhem/FHEM/95_Astro.pm
src/fhem/trunk/fhem/FHEM/95_Dashboard.pm
src/fhem/trunk/fhem/FHEM/98_backup.pm
src/fhem/trunk/fhem/FHEM/98_DSBMobile.pm
src/fhem/trunk/fhem/FHEM/98_freezemon.pm
src/fhem/trunk/fhem/FHEM/98_GEOFANCY.pm
src/fhem/trunk/fhem/FHEM/98_Installer.pm
src/fhem/trunk/fhem/FHEM/98_RandomTimer.pm
src/fhem/trunk/fhem/FHEM/98_readingsWatcher.pm
src/fhem/trunk/fhem/FHEM/98_search.pm
src/fhem/trunk/fhem/FHEM/98_WeekdayTimer.pm
src/fhem/trunk/fhem/FHEM/HOMESTATEtk.pm
src/fhem/trunk/fhem/FHEM/Meta.pm
src/fhem/trunk/fhem/FHEM/msgSchema.pm
src/fhem/trunk/fhem/FHEM/RESIDENTStk.pm
src/fhem/trunk/fhem/FHEM/UConv.pm
src/fhem/trunk/fhem/FHEM/Unit.pm

But none of the modules that require Device::SerialPort has such a META.json section:

Modules using Device::SerialPort

``` # grep -H -r 'use Device::SerialPort' src/fhem/trunk/fhem/FHEM/ | sort src/fhem/trunk/fhem/FHEM/19_VBUSIF.pm:use Device::SerialPort; src/fhem/trunk/fhem/FHEM/44_S7_S5Client.pm: #eval ("use Device::SerialPort;"); src/fhem/trunk/fhem/FHEM/50_WS300.pm: eval ("use Device::SerialPort;"); src/fhem/trunk/fhem/FHEM/60_EM.pm: eval ("use Device::SerialPort;"); src/fhem/trunk/fhem/FHEM/70_SCIVT.pm:use Device::SerialPort; src/fhem/trunk/fhem/FHEM/70_USBWX.pm:use Device::SerialPort; src/fhem/trunk/fhem/FHEM/70_WS3600.pm:#use Device::SerialPort; src/fhem/trunk/fhem/FHEM/80_M232.pm: eval ("use Device::SerialPort;"); src/fhem/trunk/fhem/FHEM/87_WS2000.pm: eval ("use Device::SerialPort;"); ```

Looks like the META.json approach does not cover all modules. I don't understand why didn't show up earlier because Device::Serial is required for USB devices, one of FHEM's core features ever since.

I guess the best case would be to ask module maintainers to add a META.json section. Are there any options beyond to improve the situation?

[sidey79]

I think the missing Device::Serial is a caching issue, the cpanfile hasn't changed, because it was always there, but removed just before the build is executed.
As soon as i found the correct cache layer i can delete it and it should be included.

[sidey79]

Something is failing during the build for device-serialport.

[git-developer]

I created separate issues for Device::SerialPort and the image size because they do not relate to Jabber integration.