ambient light sensors work, at least on macbook 13,1 about mbp-2016-linux

[cb22]

@Dunedan good news! It's exactly that. I did a quick decode of the packets from Windows and added what I thought was controlling the keyboard backlight as an init_command to the driver, and the keyboard indeed lights up.

If you add:

"\x40\x01\x00\x00\x00\x00\x10\x00\x51\xB0\x00\x26\x06\x00\x06\x00\xB0\x01\x3E\x00\xF4\x01\x96\xC5\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x3E\x59"

to applespi_init_commands - it should light up on the lowest brightness level 🎉

I captured the trace for 3 different settings, level 1, level 2 and off (there are many more, but I didn't feel like going through them all - there's probably a pattern too). In that same order, they are:

400100000000100051B0002606000600B0013E00F40196C500000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003E59
400100000000100051B0002706000600B0015C00F4018D8100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003E59
400100000000100051B0002F06000600B00100000100332100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003E59

[roadrunner2]

@Dunedan, you were right of course about the ALS being part of the iBridge. I was reading up on USB HID report descriptors etc a short while back, and then took a closer look at those that the iBridge HID interfaces declared, and lo and behold I found a HID sensors descriptor containing the reports for the ambient light sensor. I've now implemented the basic functionality for that in the appletb driver (though I'm thinking of splitting that driver up a bit), and it works reasonably well. One surprise is that the touchbar is now autodimming based on the ALS' readings - looks like interacting with the ALS "activates" it.

One slight ugliness is that while I've set things up so that the sensor only reports changes over a certain threshold (hysteresis), that threshold is a fixed, absolute value (defaults to 25, the value I saw being used in the USB traces from windows). But tracing the USB interface under MacOS it looks like they set up a relative threshold of around 15% to 20% - this gives better sensitivity at low values and fewer events at large values. I just can't figure out how MacOS sets this up, though (the sensor feature report only allows setting an absolute threshold, which is what I use, so this must be somewhere in one of the apple-specific reports, but no clue which).

Lastly, there's a bug in iio-sensor-proxy that is triggered by the way Fedora builds their rpm's (see hadess/iio-sensor-proxy#238), so that on Fedora the light readings are always basically 0.

[roadrunner2]

@Dunedan wrote:

I've now implemented the basic functionality for that in the appletb driver (though I'm thinking of splitting that driver up a bit)

I thought about that as well and it probably makes sense to have separate drivers for each function of the iBridge, as it fulfills multiple roles, which might also differ from device to device. E.g. I expect the iMac Pro to have its ambient light sensor exposed in the same way. So the ambient light sensor would probably fit in some kind of "applesmc2"-driver.

After looking at stuff a bit, I'm planning on creating an mfd driver called apple-ibridge with sub-drivers for the touchbar and ALS.

I've split up the appletb driver into 3 separate drivers now and pushed that on my repo. If folks can test that (but check the upgrading section in the README), that would be great, as I'd like to start trying to upstream this stuff.

[cb22]

@Dunedan that's an interesting point about the keyboard backlight perhaps being controlled by SPI. I'll try do some SPI captures in Windows and see if changing the keyboard backlight leads to any activity.

[Dunedan]

OMG, this is awesome! 🎉 🍾

I'll try it out the next days and will report back if it works the same way for the MacBookPro13,x as well.

[christophgysin]

I can confirm this:

[   46.117610] applesmc: key=745 fan=1 temp=33 index=33 acc=0 lux=2 kbd=0

(notice the lux=2)

$ cat /sys/class/hwmon/hwmon2/device/light
(26,0)
$ cat /sys/class/hwmon/hwmon2/device/light
(5,0)

[Dunedan]

That's interesting. The MacBookPro13,2 shows none:

[   11.310374] applesmc: key=820 fan=2 temp=41 index=40 acc=0 lux=0 kbd=0

So one more piece of hardware which differs between the models. 😫

[Dunedan]

From some MacBook8,1 dmesg output (https://bugzilla.kernel.org/attachment.cgi?id=202821) I noticed that applesmc there also doesn't detect accelerometer and keyboard backlight, so it's likely the same situation as for the MacBookPro13,1, which is good to know.

Given that the MacBook8,1 and the MacBookPro13,x were rather large redesigns my current guess is:

1. they don't include an accelerometer anymore, as they were never designed with mechanical HDDs in mind
2. as they are the first models with per-key keyboard backlight and the keyboard solely connected via SPI, keyboard backlight could be controlled via SPI as well (maybe even per key?)
3. ambient light sensors in the MacBookPro13,2 and MacBookPro13,3 could be connected via the iBridge device, as that device already connects the webcam and the ambient light sensors should be right next to the webcam

[Dunedan]

I can confirm it works on a MacBookPro13,2 as well. 👍

[cb22]

Great. I should add this to the driver then; I don't have much familiarity with backlight devices on Linux, but I see the applesmc driver uses a LED class device type (instead of a backlight class device - presumably that's only meant for screens).

[Dunedan]

Maybe @roadrunner2 is familiar with this area as well.

[roadrunner2]

@cb22 Awesome! I spent some time a few weeks ago looking at the smc's items and came to the conclusion that neither the light sensor nor the keyboard backlights were controlled/attached to it, and so started suspecting the keyboard backlight might be controllable via SPI - this is fantastic that you figured it out.

Also, yes, the keyboard backlight goes under LED's class, not the backlight.

As to the format of the packets sent and received, I have some info on the details (figured out several fields) - will post more info soon, as well as a pull request to create/use appropriate struct's rather than these raw char[]'s.

[WhyNotHugo]

FWIW, on some older macbooks the light sensor doesn't work either. On some it returns values between (0,255), and others (0,3). The latter ones are probably broken, but I tried (and failed) to debug them.

My guess is that there's a problem decoding the data the sensor sent (it being two bytes), but it was slightly beyond me.

[Dunedan]

So, I finally looked a bit into my theory of the ambient light sensor being connected to the T1 chip and exposed via iBridge in the TouchBar-models. That motivation was partially based on recent statements, that Apple's T2 chip in the new iMac Pro takes over some of the SMC functionality. So why shouldn't the T1 already do that? 😉

And I got at least something interesting:
When covering the ambient light sensor there gets a single interrupt package (see ambient-light-sensor-interrupt.zip) sent via USB from the iBridge to the host, but only if the status of the TouchBar changed (switch on, switched off or dimmed) since the last package. That's what I could observe by running macOS in a VM under Linux. I can imagine that macOS would somehow react to that package if it would detect a keyboard with backlight functionality, which isn't the case when running macOS as VM, as there is no way to pass through the SPI stuff.

I believe the fact that there is a reaction at all from the ambient light sensor via USB proves my point that it's connected through the iBridge instead of the traditional SMC-way.

[Dunedan]

@l1k has another theory (which sounds pretty plausible):

BTW, for the folks with an MBP13,3 and similar models, there's a device called apple-uart-soc-als attached to URT2, could be the adaptive light sensor, someone will probably have to reverse-engineer the protocol and write a hwmon or iio driver. There's also an apple-uart-soc device below URT1, no idea what this could be.

[roadrunner2]

@Dunedan I took a look at your usb traces: the "problem" is that they involve one of the new usb devices that show up when the extended mode is used. As such I'm a bit sceptical about them. l1k's theory, however, sounds quite plausible - now we need somebody to get traces (if possible) from Windows so we can reverse engineer and implement the protocol.

[l1k]

I tried to find the macOS driver among the 10.12 kernel extensions to no avail. If you invoke ioreg -l on macOS and search for MALS, what do you see below it, any CFBundleIdentifier?

[roadrunner2]

Here's the full ioreg -l output: ioreg-l.txt Searching for 'soc-als' shows several drivers involved, including com.apple.driver.AppleIntelLpssUARTv1 and com.apple.iokit.IOSerialFamily. I tried dtrace'ing those (via fbt) but didn't see any activity.

[l1k]

There's no ALS driver attached to the UART slave, it's merely exported to userspace via:

    | |   |   |       "IOCalloutDevice" = "/dev/cu.lpss-serial2"
    | |   |   |       "IODialinDevice" = "/dev/tty.lpss-serial2"

Is there a daemon that has those files opened? lsof /dev/cu.lpss-serial2 /dev/tty.lpss-serial2?

There's something else which seems to support @Dunedan's theory that ALS is handled by the iBridge:

    | |   |   |   |   +-o IOHIDInterface  <class IOHIDInterface, id 0x10000046f, registered, matched, active, busy 0 (8 ms), retain 7>
    | |   |   |   |   | | {
...
    | |   |   |   |   | |   "Product" = "iBridge"
...
    | |   |   |   |   | | }
    | |   |   |   |   | | 
    | |   |   |   |   | +-o AppleUSBALSService  <class AppleUSBALSService, id 0x100000478, registered, matched, active, busy 0 (0 ms), retain 5>
    | |   |   |   |   |     {
    | |   |   |   |   |       "IOClass" = "AppleUSBALSService"
    | |   |   |   |   |       "CFBundleIdentifier" = "com.apple.driver.AppleHIDALSService"
    | |   |   |   |   |       "IOProviderClass" = "IOHIDInterface"
    | |   |   |   |   |       "IOCFPlugInTypes" = {"0516B563-B15B-11DA-96EB-0014519758EF"="AppleHIDALSService.kext/Contents/PlugIns/AppleHIDALS.plugin"}
...
    | |   |   |   |   |     }

I did a quick grep over the extracted 10.12.5 combo update I had lying around here and noticed something else of interest: The file /System/Library/PrivateFrameworks/EmbeddedOSInstall.framework/EmbeddedOSInstall contains the string /dev/cu.lpss-serial1. Is this the OS updater for the iBridge? If so it would mean there's a serial backdoor to this allegedly highly secure chip. Bit odd.

You may want to repeat the grep for cu.lpss-serial over your OS installation, I don't have a full OS install here, only this combo updater.

[roadrunner2]

@1lk Sorry, forgot to mention that: no, lsof doesn't show anything having either of those open.

[l1k]

@roadrunner2:

So one theory would be that the ALS can be accessed in two ways, via USB and via UART. They've done things like that before. E.g. on the MacBookPro12,1 the keyboard is attached to both, SPI and USB. The MacBook8,1 which came out 1 month later left the USB pins on the keyboard controller unconnected. I'm not sure but I think that on the MBP12,1, the BIOS switches to USB if Windows is booted and to SPI if macOS is booted, subject to the custom EFI OS identification protocol that's also used to turn off the integrated GPU on MBP11,3+ if anything else than macOS is booted.

You're writing above:

they involve one of the new usb devices that show up when the extended mode is used

Is this extended mode also available on Windows or is it macOS-only? If the latter, it would mean that Windows probably uses the UART interface to access the ALS.

But because the ALS is part of the iBridge, it would mean there's not one but two serial backdoors. I find it odd that Apple would increase the attack surface like this. It is also weird that they hardcoded the tty into the EmbeddedOSInstall.framework. Seems sloppy from a maintainability point of view, what if they want to attach the iBridge to a different UART than lpss-serial1 on future models? It would have been more logical to search for the UART labeled "apple-uart-soc" in the I/O registry to find out its tty. They can't do that without shipping an updated DSDT because they botched the _CID names on the MBP13,2 and MBP14,2 where this UART is labeled "apple-uart-ssdc".

[l1k]

Looking at EmbeddedOSInstall.framework in the disassembler right now, the device is used by an EOSSerialLog class. The baudrate etc. are all hardcoded. Hm, this looks like a debug facility.

[Dunedan]

Is this extended mode also available on Windows or is it macOS-only?

macOS only.

If the latter, it would mean that Windows probably uses the UART interface to access the ALS.

But only if the Windows drivers provide access to the ambient light sensor. Do we know if they do?

[Dunedan]

Great work @roadrunner2 for figuring out all these details! 👍

I did try to get it to work on my MacBook Pro. One problem I stumbled upon was that appletb also requires industrialio-triggered-buffer and kfifo_buf now, which I needed to manually add to the modules available in the initramfs to get the Touch Bar working during boot again. It'd be probably worth documenting that in the README.

After that appletb works fine again and udevadm info --export-db shows the ambient light sensor as well, however I'm also getting only a value of 0 out of it, so I'm probably also affected by hadess/iio-sensor-proxy#238 with Debian/unstable.

I've now implemented the basic functionality for that in the appletb driver (though I'm thinking of splitting that driver up a bit)

I thought about that as well and it probably makes sense to have separate drivers for each function of the iBridge, as it fulfills multiple roles, which might also differ from device to device. E.g. I expect the iMac Pro to have its ambient light sensor exposed in the same way. So the ambient light sensor would probably fit in some kind of "applesmc2"-driver.

One surprise is that the touchbar is now autodimming based on the ALS' readings - looks like interacting with the ALS "activates" it.

That sounds odd, at least if there is no obvious way to manually override that automatic behavior as that's possible with macOS. Maybe that's related to the other unknown configuration interface, which could allow manual control over that as well.

[Dunedan]

FYI: With KDE the ambient light sensor doesn't do anything, as using its data isn't implemented yet. There is a feature request for that though: https://bugs.kde.org/show_bug.cgi?id=353463

[roadrunner2]

@Dunedan wrote:

I did try to get it to work on my MacBook Pro. One problem I stumbled upon was that appletb also requires industrialio-triggered-buffer and kfifo_buf now, which I needed to manually add to the modules available in the initramfs to get the Touch Bar working during boot again. It'd be probably worth documenting that in the README.

Thanks for pointing that out! I updated the README now (you actually just want to sudo modprobe industrialio-triggered-buffer).

After that appletb works fine again and udevadm info --export-db shows the ambient light sensor as well, however I'm also getting only a value of 0 out of it, so I'm probably also affected by hadess/iio-sensor-proxy#238 with Debian/unstable.

Yes. The fix has now been applied and is part of the just released version 2.5, so I suggest grabbing that.

I've now implemented the basic functionality for that in the appletb driver (though I'm thinking of splitting that driver up a bit)

I thought about that as well and it probably makes sense to have separate drivers for each function of the iBridge, as it fulfills multiple roles, which might also differ from device to device. E.g. I expect the iMac Pro to have its ambient light sensor exposed in the same way. So the ambient light sensor would probably fit in some kind of "applesmc2"-driver.

After looking at stuff a bit, I'm planning on creating an mfd driver called apple-ibridge with sub-drivers for the touchbar and ALS.

One surprise is that the touchbar is now autodimming based on the ALS' readings - looks like interacting with the ALS "activates" it.

That sounds odd, at least if there is no obvious way to manually override that automatic behavior as that's possible with macOS. Maybe that's related to the other unknown configuration interface, which could allow manual control over that as well.

There are 5 proprietary HID reports on interface 3, only one of which we have partial knowledge of (for dimming and turning off the touchbar display) - the rest I have no clue what they do. I agree, it's quite probable that one of them allows controlling this.

[Dunedan]

Thanks @roadrunner2 for adding support for the ambient light sensor as apple-ib-als kernel module. As the ambient light sensor now works on all models and how to get it working is documented in the README, I'm closing this now.