HT to @wdormann here - somebody has backdoored the open source project XZ which has downstream impacts.
For example, although OpenSSH doesn’t use XZ, Debian patch OpenSSH and introduced a dependency which translates as the XZ changes introducing a sshd authentication bypass backdoor it appears.
One dude bothered to investigate in his free time about why ssh was running slow, so it was caught fairly early - i.e. hopefully before distros started bundling it.
Worryingly it looks like the backdoor comes via one of the two main devs and dates back over a month from their GitHub account, with legit commits too - XZ is used in systemd so this one might play out for a while.
I suspect distros probably want to roll XZ back to around January 2024, stop bundling updates until the developer is removed in GitHub or a logical explanation can be given, and somebody needs to fund a code review of it.
@GossiTheDog We're incredibly lucky that it wasn't more sophisticated, or it may have gone unnoticed for a long time.
@emaste @GossiTheDog Pretty wild that the implication is that this particular backdoor was only caught because it was coded badly enough to have a significant performance impact. What if it’d been better written, I wonder?
@GossiTheDog either that or whatever build infra they're using for the release tarballs has been compromised.
@GossiTheDog
The lesson to be learned from all of this is...
Threat actors should make sure that their code doesn't throw any Valgrind errors?
😬
As I said, the impact here will be very limited due to how quick it was caught. Everybody owes the finder a beer.
Eckhofer
@wdormann @GossiTheDog ... or slows down user facing interactions
Really just incredible luck that this was found. One has to wonder how many malicious changes are never detected.
Postgres developer @AndresFreundTec saving Linux security from backdoors as a side of desk activity
CISA advisory: Reported Supply Chain Compromise Affecting XZ Utils Data Compression Library, CVE-2024-3094 https://www.cisa.gov/news-events/alerts/2024/03/29/reported-supply-chain-compromise-affecting-xz-utils-data-compression-library-cve-2024-3094
The person/account on XZ repo also altered the security disclosure policy on that and other repos they author in months prior.
@GossiTheDog That's interesting. So basically removed all of the requests for specific information, and left reporting as: "send us an email or something"?
Interesting find by @fuomag9 - the XZ repo person tried getting Ubuntu to update yesterday by filing a bug report https://bugs.launchpad.net/bugs/2059417
@fuomag9 @GossiTheDog no, it's actually harmless
The PR he made can be found here: https://github.com/MicrosoftDocs/cpp-docs/pull/4716/files
No code change, just documentation
@fuomag9 @GossiTheDog that specific one seems to be a benign doc change.
@GossiTheDog @fuomag9
However, this is more worrying
https://infosec.exchange/@fuomag9@lime.fuo.fi/112180741553647804
Compromised account used to promote update of xz on other projects
The Twilight zone time - a bug from 2015 comes back around in XZ incident, it appears https://github.com/google/sanitizers/issues/342
@GossiTheDog @wdormann
What is it with Debian and patces that end up being security holes?
OpenSSH, OpenSSL...
@leeloo @GossiTheDog @wdormann Maybe read the announcements before spouting stupid hot takes?
@sjuvonen @GossiTheDog @wdormann
Are you trying to say that I missed something? Then say it, rather that throwing accusations.
As I understand it, Debian took a heavily audited piece of code (OpenSSH), added a patch to use a library from an untrusted source, who then (allegedly deliberately) added a back door.
What am I missing?
@leeloo @sjuvonen @GossiTheDog @wdormann
(obligatory "I'm not an expert, but...")
The patch to OpenSSH makes it work with systemd-notify, which comes from a trusted source, which relies on lzma, also from a trusted source, which is used as a vector to load the backdoor from xz (also from a supposedly-trusted-until-now source) into sshd.
The same or similar patch is applied to OpenSSH in most systemd-based distros. It was noticed on Debian but affects pretty much any distro using deb or rpm packages. Other distros patch OpenSSH but the exploit only triggers on deb/rpm distros since most servers use one of the two package formats.
This is a long con, and honestly the only people at fault are the bad actors themselves. Assuming Jia Tan's GitHub identity and pgp key weren't compromised by someone else, this backdoor appears to be the culmination of three years of work.
@GossiTheDog @emberquill I am just learning about this same as every one else, but these kinds of attacks seem to be high effort, high risk since its public with a good audit trail and all it takes is one curious soul to say "hmm that's weird" to blow the whole thing up. I don't know if there are other better hidden ops out there, but the ones that are confirmed seem to get detected within days and weeks, rarely months and years, or at least that's the layman's impression I have. Is it worth it?
@GossiTheDog @fuomag9 Interesting that Debian and other distros already knew about the issue (on embargo) and were already reverting, but Ubuntu didn’t seem to know about it yet from these comments
Also, reading these comments apparently 5.6.0 was already on ubuntu?
@GossiTheDog @emberquill @leeloo @sjuvonen @wdormann once again, some weird nerd’s obsession saves the day.
Weird nerds are the heroes we need, but definitely don’t deserve.
Multiple different XZ repos and website have been suspended by GitHub.
Back in 2022 a host of characters appeared and basically bullied the creator of the XZ project to hand it over to somebody else - at the time the guy cited mental health issues around not updating the project quickly.
At the time he was already talking about maybe handing over to the account who years later introduced the backdoor.
In mid 2023 said account introduced a change to Google’s OSS Fuzzer to weaken detection for XZ.
Somebody played a years long game of Jenga and lost.
Before everybody high fives each other, this is how the backdoor was found: somebody happened to look at why CPU usage had increased in sshd, and did all the research and notification work themselves. By this point the backdoor had been there for a month unnoticed.
I’ve made the joke before that if GCHQ aren’t introducing backdoors and vulns in open source that I want a tax refund. It wasn’t a joke. And it won’t be just be GCHQ.
@GossiTheDog I'm sure the finder is basking in kudos right now! Good on them.
Another two thoughts on XZ -
- sshd itself has no dependency on the XZ utils library. The streams got crossed in a way I don’t think anybody understood (except the threat actor).
- had that backdoor been performant with sshd, I don’t think anybody would have spotted it.
The way this played out opens a window of opportunity to go back and look at both issues.
@GossiTheDog
One could argue that your tax money should also be spent by GCHQ to happen to look into increased CPU usage after some weird lib update in all the places where they didn't plant anything.
@GossiTheDog
Denying foreign actors access to UK companies secrets isn't in under economic advantage?
Ours actually has that in there (especially for govt entities and suppliers), as does (theoretically) NSA.
Really good timeline of what is known to have happened so far. It looks like the rogue developer deliberately introduced a vulnerability in other package, too - I haven’t seen anybody else mention this.
Reading the dev’s GitHub history, they’ve been making changes to other open source projects too around compression. It also appears they/somebody involved has other accounts, too.
https://boehs.org/node/everything-i-know-about-the-xz-backdoor
How far the rabbit hole goes - back in 2021 they deliberately introduced a risky change in the compression library libarchive. Nobody noticed. This is shipped in a ton of systems:
https://github.com/libarchive/libarchive/pull/1609
Whoever the threat actor is knows what they are doing as they’ve gone after chained dependencies around compression.
If anybody thinks this kind of thing is unique, it isn’t.
Example - CVE-2021-44529 in Ivanti Endpoint Manager. The cause?
Backdoor in open source code, was there for 7 years.
@GossiTheDog It looks like a very minor and completely innocuous change to start building a history for the account, not like another compromised package. Statements like "deliberately introduced an obvious vulnerability in the compression library libarchive. Nobody noticed. This is shipped in a ton of systems" from trusted sources are going to lead to people running those systems panicking about whether they've been compromised for years.
@GossiTheDog this link has a better detail on the actual obfuscated code https://www.labs.greynoise.io/grimoire/2024-02-what-is-this-old-ivanti-exploit/
@GossiTheDog Feel free to correct me, but the code with the backdoor was only a mirrored repository. The actual repository never contained it.
XZ Embedded Linux kernel module for IoT devices, 10 days ago had a change submitted to add Jia Tan (backdoor author) as a maintainer.
https://lore.kernel.org/lkml/202403201[email protected]/
Linux kernel documentation: https://docs.kernel.org/staging/xz.html
The GitHub repository for XZ Embedded kernel module has also been disabled: https://github.com/tukaani-project/xz-embedded/
Original maintainer of XZ repos has posted a short update:
https://tukaani.org/xz-backdoor/
HT @SamantazFox
@GossiTheDog if ivantis codebase had been open source someone may have noticed it. Since it’s very obvious.
It’s a bit pedantic from me, but open source was not the security weakness.
I don't know which is the biggest crime: naming a product or service with a word of the English language, like "upstream", or mentioning that product or service in an article without capitalizing, changing the font, or qualifiying it -- as "software from upstream" rather than "software from Upstream" or "software from the /upstream/ site".
@GossiTheDog The very recent zstd fork branch updates agree with the assessment that the compression ecosystem as a whole was the domain this threat actor was playing in:
Another weakening of XZ found: https://hachyderm.io/@danderson/112185746000358589
Also since there’s a lot going on here, up thread I mentioned a 2015 minor bug in Google’s OSS Fuzzer (security testing tool) - the threat actor deliberately introduced the bugged function into XZ, then used that to get an exception in OSS Fuzzer’s code to stop scanning of XZ.
I’ve just been looking at the actual backdoor for a few hours with greater minds than me, it’s incredibly complex - it basically piggy backs RSA key RCE inside sshd as a Trojan horse. Somebody/bodies spent $$ on this.
Also, to be super clear nobody should panic about #XZ as the Postgres developer who found this basically caught it quick enough that almost no businesses or devices will be running the code.
So everybody should be chill about this specific issue as that guy saved everybody’s bacon.
To give an idea of the scale of OpenSSH usage, it’s absolutely huge, it dwarfs RDP by a huge margin (think ten times), and had this survived for a long period of time it would have been unbelievably bad.
The sshd backdoor in #XZ is just way beyond my technical ability. There’s so much there, I imagine more than a few conference talks are going to be submitted for it.
My amateur hour view is it’s really well put together (eg you can only execute commands if you have a private key that only the attacker has) and appears to allow remote removal of the backdoor, too. There’s a whole bunch of features which I’m too dumb to get.
Also for me, performance isn’t that bad - I wouldn’t have noticed it.
- replies
- 1
- announces
- 0
- likes
- 0
Kinda interesting - a change made by the threat actor has ended up in Windows OS. Redmond bundled libarchive into the OS, which the TA had been tinkering with.
This is the code change which has been imported: https://github.com/libarchive/libarchive/pull/1609
Edit: reworded this as the scope looks bigger than Win 11.
@GossiTheDog I think this was for the BSD tar executable, so you have to run the code deliberately:
C:\>tar.exe --version
bsdtar 3.6.2 - libarchive 3.6.2 zlib/1.2.5.f-ipp liblzma/5.2.5 bz2lib/1.0.8 libzstd/1.5.4
@GossiTheDog @carey FWIW, libarchive (as well as bsdtar) has been bundled with Windows since 2018. :)
@0daystolive @GossiTheDog @SamantazFox
Blush. OK. Then the criminal charges are hereby reduced to a mere misdemeanor: use of confusing jargon. A few weeks of community service will do.
I gotta say, the open source community response to the XZ issue and auditing code changes for the specific threat actor has been incredibly good overall.
4 days since XZ backdoor became public knowledge and most major Linux AV and EDR security vendors still have zero detections.. they haven’t even set the static file hashes as malicious.
Can’t wait for all the vendor blogs in a week saying they fully protect against the threat. 👍
@GossiTheDog “Probability of exploitation activity in the next 30 days: 0.05%” so it seems efforts spent somewhere else would most likely wield better results despite this being a CVSS 10.0.
@GossiTheDog That'd blow up systems that haven't downgraded though, no? Detection would be nice, but would risk flooding analysts with alerts they can't do anything about 🤔 (not working on the EDR space, extrapolating)
@GossiTheDog thinking of the “one death is a tragedy, a thousand is a statistic” quote here, but I don’t necessarily disagree. Just thinking that if you have 1000s of vulnerable hosts, getting flooded by alerts wouldn’t be very useful. With that said, I never was on the receiving end so I’m mostly guessing
The libarchive change has been rolled back as it introduced a low severity security issue, will go through usual process for updates.
Linux distribution versions impacted by #XZ backdoor (or not), best list I’ve seen: https://www.rapid7.com/blog/post/2024/04/01/etr-backdoored-xz-utils-cve-2024-3094/
If you use Microsoft Vulnerability Management, it is false positiving on CVE-2024-3094 aka #XZ backdoor - it is picking up the Cygwin version of XZ as vuln on Windows systems.
The Cygwin packages predate the backdoor and it doesn’t impact Windows, also the file it flags isn’t the backdoor but lzmadec.exe
Really good timeline of #XZ backdoor, laying out everything known about what the threat actor was up to: https://research.swtch.com/xz-timeline
Re #XZ attacker - the known threat actor account made various changes across multiple open source projects and documentation.
Library maintainers should not look at those changes in isolation of just that line change, or assume the threat actor only became malicious later. Assume they are very well resourced and acting with broad objectives.
In at least one case they made an existing unknown vulnerability exploitable, and we know they were socially engineering the XZ maintainer years ago.
‘They’ are very likely a multi million dollar operation - see also just the shell script analysis, before you even get to the backdoor (which is much more nuts) https://research.swtch.com/xz-script
The actual SSH backdoor is cryptographically signed so only the threat actor can use it. If you work in threat intelligence and write “foreign” intelligence agency, you might want to look at your bias training.
@GossiTheDog not sure if this statement is entirely true. I suppose you have seen this ? https://github.com/R4GN4R0K-SEC/xzbot
@GossiTheDog I am amazed how they took all that time to prepare and effort to hide all this, and then impatiently started pushing the distros in such a suspicious way.
@GossiTheDog doesn’t this all seem still a bit extreme though? Backdooring most of enterprise Linux systems on one go? Once the attacks started it would be discovered reasonably fast that the problem was in ssh so what’s the point? Do mass-wiper or mass-ransomware attack on one go and move somewhere warm with fat bitcoin wallet?
@GossiTheDog “The actual SSH backdoor is cryptographically signed so only the threat actor can use it”
@GossiTheDog Have you seen the Tukanni project installations on windows devices?