2024-03-29 18:46:46.611727+01 by Dan Lyke 19 comments
Uh oh: xz, and the liblzma compression library, may have been backdoored.
openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma.
If you (or your upstream providers on Debian based platforms it's in Red Hat Fedora Linux 40 and Fedora Rawhide, too) built from tarballs after 5.6.0, shit just got real.
[ related topics: Language Free Software Interactive Drama Books Open Source ]
comments in ascending chronological order (reverse):
#Comment Re: xz backdoored made: 2024-03-29 19:14:27.06058+01 by: brainopener
I'm seeing some chatter that this is wider spread than those packages. It's early, fog of war, misinformation flying, all of that. But, this does appear to be a big damn deal.
#Comment Re: xz backdoored made: 2024-03-29 19:46:54.032732+01 by: Dan Lyke
Yeah, and I was wrong, it's not just Debian based distros: Red Hat: Urgent security alert for Fedora Linux 40 and Fedora Rawhide users
And I think this is shades of stuff that we've kinda seen looming with the Node package repos, but... yeah. This is serious "reflections on trusting trust" level.
#Comment Re: xz backdoored made: 2024-03-29 20:27:36.159461+01 by: Dan Lyke
RT AndresFreundTec @AndresFreundTec@mastodon.social
I was doing some micro-benchmarking at the time, needed to quiesce the system to reduce noise. Saw sshd processes were using a surprising amount of CPU, despite immediately failing because of wrong usernames etc. Profiled sshd, showing lots of cpu time in liblzma, with perf unable to attribute it to a symbol. Got suspicious. Recalled that I had seen an odd valgrind complaint in automated testing of postgres, a few weeks earlier, after package updates.
Really required a lot of coincidences.
The first signs of the backdoor were introduced in a February 23 update that added obfuscated code, officials from Red Hat said in an email. An update the following day introduced functions for deobfuscating that code and injecting it into code libraries as they were being built during the xz Utils update process. The malicious code has resided only in the archived releases—known as tarballs—which are released upstream. So-called GIT code available in repositories aren’t affected, although they do contain second-stage artifacts allowing the injection during the build time. In the event the obfuscated code introduced on February 23 is present, the artifacts in the GIT version allow the backdoor to operate.
The malicious changes were submitted by JiaT75, one of the two main xz Utils developers with years of contributions to the project.
And it's at least an SSH backdoor attempt.
#Comment Re: xz backdoored made: 2024-03-29 23:42:40.33404+01 by: Dan Lyke
RT Glyph @glyph@mastodon.social
@eb I really hope that this causes an industry-wide reckoning with the common practice of letting your entire goddamn product rest on the shoulders of one overworked person having a slow mental health crisis without financially or operationally supporting them whatsoever. I want everyone who has an open source dependency to read this message https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.html
That email message reads, in part:
I haven't lost interest but my ability to care has been fairly limited mostly due to longterm mental health issues but also due to some other things. Recently I've worked off-list a bit with Jia Tan on XZ Utils and perhaps he will have a bigger role in the future, we'll see.
It's also good to keep in mind that this is an unpaid hobby project.
#Comment Re: xz backdoored made: 2024-03-30 00:22:52.02695+01 by: Dan Lyke
Good rundown, being actively updated, at https://boehs.org/node/everything-i-know-about-the-xz-backdoor
#Comment Re: xz backdoored made: 2024-03-30 05:56:27.755849+01 by: Dan Lyke
lcamtuf's thing: Technologist vs spy: the xz backdoor debate
#Comment Re: xz backdoored made: 2024-03-30 06:00:48.784007+01 by: Dan Lyke
RT Irenes (many) @irenes@mastodon.social
so, the github copy of the xz repo and all associated discussion history has been hidden, and possibly deleted, as a policy enforcement action
you know how we all spent this morning understanding the nuances of how the attacker went to great trouble to deploy the final trigger of the attack in a way that lets them avoid durable records of it?
next time they won't bother. github will do that part for them.
#Comment Re: xz backdoored made: 2024-03-30 12:25:14.348942+01 by: meuon
Decent scale coordinated compromise of a complex ecosystem of mostly volunteers... and that ONE has been caught. Evans write up is pretty good.
Icamtufs is decent commentary. Which leads to a problem he points at: So much of the modern technology infrastructure is built on layers and layers of code and dependencies whose core were created by people that are aging out of the scene. The modern "cut and paste cowboys" I'm running into doing web dev don't have the skills, ethos or interest.
It does re-enforce my distrust of people with iterative usernames and github in general. Modern packaging of docker containers, flatpacks, snaps.. make it easier to introduce malicious code. Not enough serious skilled paranoid people in the world to monitor it all.
#Comment Re: xz backdoored made: 2024-03-30 16:30:22.479254+01 by: Dan Lyke
The original author of xz has an state of the world post https://tukaani.org/xz-backdoor/
And, yeah: commercial code is nefariously trying tp up sell us to subscription services, and backdooring to cooperate with coercive state actors. Open source is abusing its authors, and creating social situations that allow for backdooring. It's hard to figure a path forward...
#Comment Re: xz backdoored made: 2024-03-30 23:21:27.218516+01 by: Dan Lyke
Good summary of.how the social expectations normalized around Open Source created.the conditions for this: https://robmensching.com/blog/...actions-in-open-source-projects/
#Comment Re: xz backdoored made: 2024-03-31 05:02:17.9212+02 by: Dan Lyke
And another good summary https://gist.github.com/gonoph/c41630716d594e61a69477760ac045ae
#Comment Re: xz backdoored made: 2024-03-31 05:47:55.016981+02 by: flushy
@meuon - this is a trusted supply chain issue at it's core.
I've had the privilege of talking to our (Red Hat) product security folks over the years. There's a lot that goes on behind the scenes - a lot of stuff that's just boring and hidden. Stuff most admins and procurement folks really don't care about. However, a lot of 3 letter agencies do care about it. Public sector is an area where they take provenance very seriously. SBOM (software bill of materials) is one thing that's in the works. Basically, an API that you would query to obtain a bunch of metadata about the stuff you've obtained from us. All of our stuff is signed, and we document where every commit and line from upstream came from.
https://www.redhat.com/en/blog/future-red-hat-security-data
I personally find the work the security folks do fascinating, but I don't have the attention to detail to be good at it!
This won't solve the current xz issue - which is basically a rogue actor obtaining admin rights to a project, coupled with the other maintainer taking a "leave of absence" around the same time.
I personally wish we could hire the guy, and make it his "job" to maintain xz along with the full backing of our engineering.
But.. that's above my pay grade.
#Comment Re: xz backdoored made: 2024-03-31 12:23:44.421436+02 by: meuon
@flushy and all.. and we need more of that "hire" those devs mentality.. I'll give RedHat a big thank you for as much of that as they do.
Sadly, our open source world needs such a kick every now and then.
#Comment Re: xz backdoored made: 2024-03-31 18:50:12.364879+02 by: Dan Lyke
Okay, I'm now learing things about ldd and binutils that I kinda wish I didn't know, but now I do. And I'm gonna take this thread and print it out and read it regularly...
RT Carol (Nichols || Goulding) ꙮ @carol@crabby.fyi
the lesson *I'm* choosing to take from xz, as an oss maintainer, is that anyone trying to pressure or guilt me into doing something should immediately be told no, for security reasons
RT mybarkingdogs @mybarkingdogs@freeradical.zone
@carol This is literally a good lesson for EVERYONE in *anything,* not even just software.
Giving into pressure/guilt is DANGEROUS
In personal relationships, it's one of the worst mistakes: it tells an abuser/manipulator you're a target.
In anything financial, it's often a baited hook for a scam
In politics it gets you pulled into anything from outright far-right fascist bullshit like qanon to "left" (but not really left, obviously!) groups that are state-sponsored ops or personality cults
RT mybarkingdogs @mybarkingdogs@freeradical.zone
@carol (As an example, I've actually seen this exact threat model used to turn reddits/FB groups/discords/etc into fascist or tankie hellpits
The fascist or tankie joins, behaves, gets in close to the main admins or maintainers or leaders or whomever, contributes, essentially digs in - and then starts attacking other members to run them out of the space, posting propaganda, etc
And then they either become sole admins or have gotten the owners/admins on their side
and the place is theirs)
RT mybarkingdogs @mybarkingdogs@freeradical.zone
@carol (and that's how, for example, a generally center-leftish gaming community gets infiltrated by fash and turns hard far right, or an anarchocommunist group gets turned into another spigot for the Kremlin firehose rather than actual anti-oppression organizing, etc.
Someone gets in, ratchets up the pressure, makes people feel guilty/afraid to speak up about the changes or ask the asshat to leave, etc...)
#Comment Re: xz backdoored made: 2024-04-01 05:46:59.11304+02 by: flushy [edit history]
It's like the Michael Tager's story about kicking out folks before it becomes a Nazi bar. You error on the side of decorum, and they use your own arguments of tolerance against you. Before you know it, they're entrenched, and it's now a Nazi Bar.
Michael left twitter though, so can't link to his tweet(s).
#Comment Re: xz backdoored made: 2024-04-01 19:19:19.65012+02 by: Dan Lyke
RT nikki@topspicy.social Nikki @nikki@topspicy.social
this whole xz thing was very nearly an amazingly successful attempt to own the libs
#Comment Re: xz backdoored made: 2024-04-02 02:58:07.280606+02 by: flushy
This is a nice write up of the timezones of the commits.
https://rheaeve.substack.com/p...-backdoor-times-damned-times-and
#Comment Re: xz backdoored made: 2024-04-03 01:07:45.954575+02 by: flushy [edit history]
Also, here's a quick container version of a POC of the backdoor:
https://github.com/dguerri/exploits-collection/tree/main/xz-5.6.1- backdoor
Based on work done by https://github.com/amlweems/xzbot
#Comment Re: xz backdoored made: 2024-04-05 22:52:54.616406+02 by: Dan Lyke
Interesting read-through from the perspective of a BSD developer: https://marc.info/?l=openbsd-misc&m=171227941117852&w=2
(If anyone ever actually uses Webmention/indie-action to post here, please email me)
We will not edit your comments. However, we may delete your comments, or cause them to be hidden behind another link, if we feel they detract from the conversation. Commercial plugs are fine, if they are relevant to the conversation, and if you don't try to pretend to be a consumer. Annoying endorsements will be deleted if you're lucky, if you're not a whole bunch of people smarter and more articulate than you will ridicule you, and we will leave such ridicule in place.
Flutterby™ is a trademark claimed by
Dan Lyke for the web publications at www.flutterby.com and www.flutterby.net.