What we know about the xz Utils tailgate that almost took the world by storm

Researchers on Friday revealed the discovery of a backdoor that was deliberately planted in xz Utils, an open source data compression utility available on almost all installations of Linux and other Unix-like operating systems. The person or people behind this project have likely spent years working on it. They were probably very close to seeing the backdoor update integrated into Debian and Red Hat, the two largest Linux distributions, when one eagle-eyed developer spotted something fishy.

“This may be the best-executed supply chain attack we've ever seen described in public, and it's a nightmare scenario: malicious, efficient, and authorized in a widely used library,” said software and cryptographic engineer Filippo Valsorda. He said An effort that came frighteningly close to success.

Researchers spent the weekend collecting evidence. Here's what we know so far.

What is xz utility?

xz Utils is almost everywhere in Linux. It provides lossless data compression on almost all Unix-like operating systems, including Linux. xz Utils provides important functions for compressing and decompressing data during all types of operations. xz Utils also supports the legacy .lzma format, which makes this component even more important.

What happened?

Anders Freund, a developer and engineer who works on Microsoft's PostgreSQL offering, was recently troubleshooting the performance issues Debian was experiencing with SSH, the protocol most commonly used to remotely log in to machines over the Internet. Specifically, SSH logins were consuming too many CPU cycles and were throwing errors with valgrind, a computer memory monitoring utility.

Through a combination of sheer luck and Freund's keen eye, he eventually discovered that the problems were the result of updates made to the xz Utils. On Friday, Freund took to the open source security menu to reveal that the updates were the result of someone intentionally planting a backdoor in the compression software.

What does a back door do?

Malicious code has been added to versions 5.6.0 and 5.6.1 of xz Utils to modify the way the program behaves when performing operations related to lzma compression or decompression. When these functions involve SSH, they allow malicious code to be executed with root privileges. This token allows anyone with a pre-defined encryption key to log into the back-end system via SSH. From now on, this person will have the same level of control as any authorized administrator.

How did this back door come to be?

It appears this backdoor has been under construction for years. In 2021, a person with the username JiaT575 created their own account First known commit To an open source project. Later, It changes To the libarchive project is suspicious, because it replaced the Safe_fprint function with a variable that has long been recognized as less secure. No one noticed at the time.

The following year, JiaT575 submitted a patch on the xz Utils mailing list, and almost immediately, an unseen participant named Jigar Kumar joined the discussion and argued that Lasse Collin, the longtime xz Utils maintainer, wasn't updating the software more often than not. Often or quickly enough. Kumar, with the support of Dennis Ince and several others who had not previously been on the list, lobbied Colin to bring in an additional developer to maintain the project.

In January 2023, JiaT75 manufactured it First commitment To xz Utils. In the following months, JiaT75, who used the name Jia Tan, became increasingly involved in the affairs of xz Utils. For example, Tan replaced Collins' contact information with his own on Microsoft's oss-fuzz project, a project that scans open source software for signs of maliciousness. Tan also asked oss-fuzz to disable the ifunc function during testing, a change that prevented him from detecting the malicious changes Tan would soon make to xz Utils.

In February this year, Tan released commits for versions 5.6.0 and 5.6.1 of xz Utils. Implemented backdoor updates. In the following weeks, Tan or others implored Ubuntu, Red Hat and Debian developers to integrate the updates into their operating systems. Eventually, one of the two updates made its way to subsequent releases, according to Defensible Security Company:

Can you say more about what this backdoor does?

In short, it allows anyone with the right private key to hijack sshd, the executable file responsible for making SSH connections, and from there to execute malicious commands. The tailgate is implemented through a five-stage loader that uses a series of simple but clever techniques to conceal itself. It also provides the means to deliver new payloads without requiring major changes.

Many people who have reverse engineered the updates have a lot to say about the backdoor.

Introduced by developer Sam James This overview:

This backdoor has several components. At a high level:

• Versions of tarballs published upstream do not contain the same code as in GitHub. This is common in C projects so that downstream consumers do not need to remember how to run autotools and autoconferences. The version of build-to-host.m4 found in the tarballs differs significantly from the original version on GitHub.
• There are built test files in the tests/ folder inside the git repository as well. These files are located in the following commits:
• A script called by build-to-host.m4 unpacks this malicious test data and uses it to modify the build process.
• IFUNC, a mechanism in glibc that allows indirect function calls, is used to perform runtime binding/redirection of OpenSSH authentication procedures. IFUNC is a tool that is usually used for legitimate things, but in this case it is being exploited for this attack path.

Upstream usually publishes tarballs that are different from those automatically generated in GitHub. In these modified tarballs, a malicious version of build-to-host.m4 is included to execute a script during the build process.

This script (at least in versions 5.6.0 and 5.6.1) checks various conditions such as device architecture. Here is an excerpt of the malicious script decompiled by build-to-host.m4 and an explanation of what it does:

if ! (echo "$build" | grep -Eq "^x86_64" > /dev/null 2>&1) && (echo "$build" | grep -Eq "linux-gnu$" > /dev/null 2>&1);then

• If amd64/x86_64 is the build target
• And if the target uses the name linux-gnu (mostly checks for use of glibc)

It also checks the tool chain used:

if test "x$GCC" != 'xyes' > /dev/null 2>&1;then
exit 0
fi
if test "x$CC" != 'xgcc' > /dev/null 2>&1;then
exit 0
fi
LDv=$LD" -v"
if ! $LDv 2>&1 | grep -qs 'GNU ld' > /dev/null 2>&1;then
exit 0

And if you are trying to build a Debian or Red Hat package:

if test -f "$srcdir/debian/rules" || test "x$RPM_ARCH" = "xx86_64";then

This attack thus appears to target AMD64 systems running glibc using distributions derived from Debian or Red Hat. Other systems may be vulnerable at this time, but we don't know.

In an online interview, developer and reverse engineer HD Moore confirmed Sam James' suspicions that the backdoor is targeting Debian or Red Hat distributions.

“The attack was sneaky because it only took the last steps of the backdoor if you were building the library on AMD64 (Intel x86 64-bit) and you were creating a Debian or RPM package (instead of using it for a local installation).” ,” he wrote.

He continued by paraphrasing the observations of researchers who collectively spent the weekend analyzing the malicious updates:

When verifying an SSH public key, if the public key matches a certain fingerprint function, the contents of the key will be decrypted using a pre-shared key before the public key is actually verified. The decrypted contents are then passed directly to the system.

If the fingerprint doesn't match or the decrypted contents don't match a certain format, it's back to normal key verification and no one is the wiser.

The back door is very sneaky. Uses an unknown feature of glibc to bind a function. It is only run when the backdoored xz library is loaded by the /usr/bin/sshd process on one of the affected distributions. There may be many other backdoors, but the backdoor everyone is talking about uses a non-directional function to add the hook. The payload was encrypted in fake xz test files and effectively ran as shellcode, altering the SSH RSA key verification code so that the magic public key (sent during normal authentication) would allow the attacker to access

Their grand plan was:

1) Invisibly backdoors the release balls, but not the source code

2) Use Sockpuppet accounts to convince various Linux distributions to pull and package the latest version

3) Once these distros ship, they can take over any user/company system/etc

Additional technical analysis is available from above Bluesky thread from Valsorda, Researcher Kevin Beaumont And Freund revealed on Friday.

What do we know more about Jia Tan?

Right now, that's very little, especially for someone tasked with overseeing a piece of software as widespread and sensitive as xz Utils. This developer personality has touched on dozens of other open source software in the past few years. At the moment, it is unknown if there is a real person behind this username or if Jia Tan is a completely fabricated individual.