Could TPRM Have Foreseen the XZ Utils (CVE-2024-3094) Crisis?

Written by: Ferhat Dikbiyik

The recent CVE-2024-3094 vulnerability in XZ Utils has sent shockwaves through the cybersecurity world, unraveling a narrative that seems straight out of a cyber thriller. The backdoor discovered in the open-source compression software that allows attackers to execute remote code execution definitely rings the alarm bells. At its core, this incident isn’t just a technical loophole; it’s a masterclass in manipulation and trust exploitation. In a world increasingly reliant on open-source components, the incident shines a glaring spotlight on the vulnerabilities that can arise from overlooked corners of our digital infrastructure. This blog post aims to dissect the incident from a Third-Party Risk Management (TPRM) perspective, highlighting why it matters profoundly to professionals in the field and what lessons can be drawn to safeguard against future threats.

What Is the XZ Backdoor Vulnerability?

Last Friday marked the unveiling of a sophisticated cybersecurity breach, specifically the XZ Backdoor vulnerability, bringing the digital world to a standstill. This discovery was not merely a technical hiccup but a saga of deception, laying bare the vulnerabilities lurking within our trusted open-source components. As someone deeply entrenched in the cybersecurity field, I watched as experts meticulously unraveled the layers, mapping out a timeline that began with unassuming contributions to a popular compression utility, XZ Utils, and culminated in a complex backdoor capable of compromising countless systems.

CISA has also issued an alert regarding a supply chain compromise targeting the XZ Utils data compression library, identified as CVE-2024-3094, urging affected entities to apply recommended mitigations and report any suspicious activity.

The vulnerability is assigned to the highest severity level, 10.0 out of 10.0, based on CVSSv3.

One cannot help but remember Log4Shell vulnerabilities. The XZ Backdoor vulnerability mirrors the Log4Shell crisis in its stealth and potential for widespread disruption. So, is it the next Log4Shell, the infamous Log4J vulnerability?  Both vulnerabilities exploit ubiquitous components deeply embedded in system operations—Log4Shell with its logging library and XZ with its compression tools. Yet, while Log4Shell was a direct exploit of a specific software flaw, CVE-2024-3094 unravels a more complex narrative of social engineering entwined with technical exploitation. Unlike the immediate execution of code with Log4Shell, the XZ Backdoor reflects a calculated campaign of trust exploitation, which allowed malicious actors to systematically plant a backdoor in a tool integral to Linux distributions. Confined primarily to Linux distributions, its scope might seem narrower than Log4J. Yet, its potential to disrupt the software supply chain is undeniable and worrisome.

(If you are not into the technical details and are just interested in the TPRM perspective, you can skip here.)

How Did the XZ Backdoor Unfold?

The timeline of the XZ backdoor vulnerability reveals a meticulously planned attack, stretching more thantwo years. It began in early 2021 with the creation of the “Jia Tan” persona, followed by suspicious initial contributions to other projects to gain credibility within the open-source community.

By mid-2022, another persona, “Jigar Kumar,” emerged, pressuring for changes that paved the way for the backdoor’s insertion into XZ Utils. This strategic orchestration of social engineering and technical manipulation culminated in the compromise of widely used compression software, illustrating a deep understanding of open-source dynamics and the trust mechanisms within.

2021

• Jia Tan (JiaT75) creates GitHub account: Begins suspicious activity, including a potentially harmful commit to libarchive.

2022

• April: Jia Tan engages in questionable correspondence and contributes to pressuring for changes in XZ’s maintenance.
• Soon after: Jigar Kumar disappears after influencing XZ maintenance. JiaT75 starts contributing to XZ regularly.

2023

• January 7: JiaT75 merges a significant commit, indicating gained trust within the XZ community.
• March: Jia Tan becomes the primary contact for Google’s oss-fuzz, concerning XZ.
• June: A key testing infrastructure component for the exploit is committed.
• July: Efforts to mask the exploit’s preparation through PRs and issues.

2024

• Early 2024: Changes to Google’s oss-fuzz URL and key commits introduce the final pieces of the backdoor.
• Discovery and Reaction: Discovery of the backdoor leads to public disclosure, analysis, and urgent remediation efforts. GitHub suspends JiaT75 and Lasse Collin’s, the maintainer’s, accounts, hindering further audit efforts.

How Was Social Engineering Employed in This Incident?

The backbone of the XZ backdoor incident was a social engineering masterclass. Attackers meticulously crafted personas like “Jia Tan” and “Jigar Kumar,” weaving a narrative of trustworthiness and technical prowess over months. They exploited the open-source culture of mutual aid, targeting the maintainer’s burnout and eagerness for support. This scenario highlights a critical vulnerability not in code but in human nature.

The mental health of the maintainer played a significant role in this situation for a few reasons, painting a broader picture of vulnerability in the open-source community:

• Single Point of Failure: The entire incident underscores how the reliance on a single, overworked individual for critical project maintenance creates a fragile security posture.
• Pressure and Stress: Open-source maintainers often work in their spare time, without financial compensation, juggling their personal lives, full-time jobs, and their roles in maintaining software. This can lead to burnout, decreased mental health, and a compromised decision-making process. In this context, the maintainer’s ability to resist pressure to make hasty additions or changes to the codebase might weaken, especially if they’re experiencing a mental health crisis.
• Exploitation of Trust: Bad actors can exploit these vulnerabilities, pressuring maintainers to accept changes or add new contributors. When a maintainer is struggling, they might be less likely to push back on these requests, especially if they feel overwhelmed or are seeking to reduce their workload.
• Community Support Systems: The incident highlights a lack of support structures for maintainers in the open-source ecosystem. Without adequate backup, financial support, or resources to manage the project, maintainers are left to deal with the immense pressure on their own. This can exacerbate mental health issues and lead to situations where security is not given the priority it needs.

In summary, the mental health of a maintainer is crucial not just for their well-being but for the security and integrity of the open-source projects they oversee. It emphasizes the need for a more supportive, collaborative approach within the community to safeguard both the people and the projects they dedicate their time to.

As a cybersecurity researcher, I see this as a cautionary tale emphasizing the need for community support for maintainers to prevent burnout, making them less susceptible to such deceptions. Additionally, this highlights the significance of a Software Bill of Materials (SBOM) in tracking and managing components of software, enhancing transparency and security in the supply chain. In events like this, an SBOM could play a pivotal role in quickly identifying and mitigating compromised components.

What Lies Beneath the XZ Backdoor?

The technical orchestration of the XZ backdoor involved sophisticated manipulation of the build process. Attackers injected malicious code into the auto-build system using an ‘m4’ macro, then crafted a complex chain of commands that were executed during the building of ‘liblzma’, effectively concealing the backdoor’s presence. It involved obfuscating and replacing normal code execution flows with their malicious counterparts.

What made this attack notably cunning was the use of legitimate testing infrastructures as the Trojan horse for the malicious payload, demonstrating an intricate understanding of the system’s architecture. This level of technical prowess not only facilitated unauthorized access but also positioned the backdoor to evade initial detection.

The blog post from Gynvael Coldwind provides an in-depth look at the obfuscation techniques used in the bash stages of the backdoor found in xz/liblzma, which affected OpenSSH servers. Here’s a summary of the stages and obfuscation methods detailed in the post:

Stage 0

• Initiated in ‵m4/build-to-host.m4‵.
• Malicious code is injected during the build process, modifying the ‵configure‵ script.
  • Transforms bytes read from ‵tests/files/bad-3-corrupt_lzma2.xz‵ and substitutes certain byte values to ‘uncorrupt’ the file and form a proper xz stream.

Stage 1

• The ‵configure‵ script, now containing obfuscated malicious code, manipulates linker and compiler flags.
  • Contains binary bytes in comments, with different bytes between versions 5.6.0 and 5.6.1.
  • The script includes a check for Linux, repeated five times in 5.6.1.
  • Utilizes a large ‵export‵ command that defines a function that, when invoked, processes data from ‵tests/files/good-large_compressed.lzma‵.
  • The function chains numerous ‵head‵ calls to skip or output specific byte sequences.
• Ultimately, this results in a deciphered data stream, which is then decompressed and executed.
• This causes the Makefile to interfere with the symbols resolution process, effectively priming the system for the subsequent stages of the attack.

Stage 2

• Modifies the actual compilation process, weaving in a .o file into the process.
• The compromised Makefile is executed, altering the symbol ‵RSA_public_decrypt‵ to point to malicious code that will be executed at runtime.
  • The script uses ‵sed‵ to put each byte of the input on a new line, preparing it for an RC4-like decryption implemented in ‵awk‵.
• Post decryption, data is decompressed and then saved as ‵liblzma_la-crc64-fast.o‵, which is the binary backdoor.
• The compiled ‵liblzma‵ library now contains a backdoor, allowing for an authentication bypass in the SSHD process.

Extension Mechanism in Stage 2 (5.6.1)

• Searches for specific byte signatures in files to allow for future scripts to be run without modifying original payload-carrying test files.
• No files with the specified signatures were found in the examined TAR archives, suggesting this system was designed for potential future use.

SSHD Authentication Bypass:

• When SSHD calls ‵RSA_public_decrypt‵, due to the symbol resolution manipulation, the call is redirected to the malicious code.
• This results in an authentication bypass within the SSH service.

The meticulously orchestrated attack not only led to an authentication bypass but, more critically, enabled remote code execution with root privileges. This allowed the attackers to execute commands and control the system with unfettered access. It’s imperative to understand the severity of this—once the backdoor is exploited, the attacker has the same level of control over the system as the legitimate administrator.

Beyond the immediate threat of unauthorized root-level access, the CVE-2024-3094 vulnerability introduces another layer of stealth that compounds its danger—the manipulation of the system’s logging mechanisms. By interfering with the ‵setlogmask‵ function, the malicious code effectively blinds the eyes that would normally watch for signs of intrusion. Researchers have pointed out a particularly insidious consequence of this manipulation: successful exploitation of the backdoor leaves behind no traditional forensic footprint.

Unlike failed authentication attempts or similar security events, which are typically logged and alert administrators, attacks leveraging this vulnerability pass undetected. This creates a significant challenge for cybersecurity professionals and forensic analysts, making it exceedingly difficult to ascertain if, when, and how a system was compromised.

The backdoor is highly sophisticated, utilizing standard command-line tools to create a multistage, inconspicuous attack. Techniques include file carving, substitution ciphers, and a custom decryption method. The system even includes provisions for future extensions without the need to alter the initial malicious files.

Below is also an infographic generated by Thomas Roccia that explains the vulnerability in detail.

Who Orchestrated the XZ Backdoor, and Where Does the Blame Lie?

Determining who is truly behind the XZ backdoor attack is complex. While the investigation points to the creation of suspicious personas such as “Jia Tan,” the true identities and affiliations remain shrouded in anonymity.

Hints strewn across the timeline of the XZ backdoor incident point toward the sophistication typical of state-sponsored threat actors. The careful crafting of personas and long-term planning suggest resources and motivations beyond ordinary cybercriminals. The possibility of state involvement is a theory I firmly believe cannot be dismissed. While attribution in cyberattacks is notoriously difficult, the complexity and subtlety of this operation have all the hallmarks of a campaign orchestrated by actors with significant backing, potentially at the state level.

Pinning blame solely on the maintainers would be unfair; they, too, are victims of an elaborate and well-orchestrated deception. The incident serves as a reminder that the open-source ecosystem relies heavily on trust and the tireless work of often unsupported maintainers. As such, it is the collective responsibility of the cybersecurity community to foster a culture of support and robust security practices to protect against such vulnerabilities.

What’s the Scope and Impact of the XZ Backdoor on Linux Distributions?

The CVE-2024-3094 vulnerability embedded within XZ Utils versions 5.6.0 and 5.6.1 presents a severe threat, potentially affecting numerous Linux distributions that utilize these versions for data compression tasks. The breach, characterized by its complexity, could lead to unauthorized system access and a host of malicious activities.

Affected Linux variants likely include widely-used distributions such as Fedora, Ubuntu^(*), and possibly others that deploy XZ Utils for package compression. Several analyses show that Linux distributions are susceptible to XZ backdoor vulnerabilities.

The table below reflects the Linux distributions and their respective branches and packages affected by the backdoor vulnerability, CVE-2024-3094, along with the remediation steps and comments on the status or specifics of the affected versions.

*Note on Ubuntu: Not listed due to the vulnerability affecting its internal build pipeline, not released packages. Servers updated post-25 Feb 2024 should audit packages linked to xz or liblzma.”

The identified vulnerability not only represents a critical challenge for the security infrastructure of the affected Linux distributions but also signals a broader concern for organizations relying on these systems.As organizations race to understand the depth of this issue, they must also consider the implications it has from a Third-Party Risk Management (TPRM) standpoint. Let’s explore how TPRM professionals should approach the situation, evaluate the risk posed by their vendors, and ensure that their supply chain remains secure in the wake of CVE-2024-3094.

How Can TPRM Professionals Address the XZ Backdoor Incident’s Wide-Reaching Implications?

TPRM professionals should pay close attention to the XZ backdoor incident due to its extensive ripple effect. The infiltration of trusted open-source components can lead to a chain reaction of security breaches across numerous systems and networks. Understanding the depth and breadth of this incident can help TPRM professionals gauge the potential impact on their organization’s security posture and the necessity for robust vendor risk assessments. 

TPRM professionals should initiate dialogues with vendors about potentially impacted Linux distributions as a priority, recognizing that vendors might lack detailed insights into specific components like XZ Utils. This strategic approach aims to manage the broader implications of supported systems while minimizing questionnaire fatigue. Given the intricate nature and detection challenges of the XZ backdoor, leveraging risk intelligence is essential for TPRM professionals to conduct efficient and effective inquiries, ensuring a clear understanding of the threat landscape and the steps vendors are taking to mitigate risks.

Supply Chain Risk Intelligence is crucial for avoiding vendor questionnaire fatigue and responding quickly to risks associated with vulnerabilities like the XZ Backdoor. These risks can have ripple effects in the supply chain. The Black Kite Supply Chain Module illuminates supply chain risks with the right intelligence, giving organizations the ability to identify cyber risks not only in third-party vendors but also in 4th- and 5th-parties.

Check out the Black Kite Supply Chain Module here

Questions to ask vendors:

As a TPRM (Third-Party Risk Management) professional addressing potential impacts of CVE-2024-3094, it’s crucial to ask pointed questions that cover both the extent of exposure and the steps vendors have taken to mitigate risks.

1. How have you verified the integrity of patches applied to mitigate the CVE-2024-3094 vulnerability in XZ Utils within your Linux distributions?
2. What specific tools or processes have you employed to detect the presence and ensure the removal of the compromised XZ Utils versions 5.6.0 and 5.6.1 from your systems?
3. Can you provide a detailed and current Software Bill of Materials (SBOM) that outlines the presence of XZ Utils or any related dependencies within your infrastructure post-incident?
4. In response to the CVE-2024-3094 backdoor discovery, what adjustments have you made to your open-source component security vetting process to prevent similar incidents in the future?

Surely, more technically detailed questions may help vendors to understand what is expected from them and even guide them towards remediation.

1. Detection and System Inventory: Can you confirm whether you have conducted a comprehensive scan of your systems using the command ‵strings $(which xz) | grep ‘5\.6\.[01]’‵ to identify affected versions of xz-utils, especially for Linux distributions prone to CVE-2024-3094, and what was the outcome? For Alpine Linux, did you utilize the apk package manager to check for the exact version installed?
2. Immediate Remediation Steps: In response to CVE-2024-3094, have you downgraded xz to the safe version 5.4.6 across all systems, including those running vulnerable Linux distributions, and can you provide details on the process and confirmation of a complete restart of the OpenSSH server post-remediation?
3. Mitigation Verification: For systems where immediate upgrade or downgrade of xz is not feasible, have you implemented the recommended ‘kill switch’ by adding ‵yolAbejyiejuvnup=Evjtgvsh5okmkAvj‵ to /etc/environment, and how have you verified the deactivation of the backdoor functionality?
4. Ongoing Risk Management: What long-term measures are being put in place to ensure such vulnerabilities are promptly identified and remediated in the future, particularly for the Linux distributions directly affected by CVE-2024-3094, and how is this integrated into your vulnerability management program?

Some organizations have thousands of vendors. Surely, without proper risk intelligence, reaching out to all those vendors will not be effective. Suppose vendor outreach is done with proper risk intelligence, such as asset information of the Linux distribution used on the vendor’s system susceptible to XZ-backdoor. In that case, vendors tend to respond more rapidly.

How can Vendors Detect and Remediate CVE-2024-3094

Vendors can detect the XZ backdoor by scrutinizing their systems for the specific versions of XZ Utils that are compromised (5.6.0 and 5.6.1), employing tools to scan for signatures of the backdoor’s code, and conducting a thorough audit of their software supply chain to ensure the integrity of their components. Utilizing updated security software capable of identifying such sophisticated threats is also crucial. For a detailed approach, consulting security advisories that outline detection methodologies is recommended.

Vendors can utilize the detection script created by Fabio Baroni to identify the presence of the CVE-2024-3094 vulnerability in XZ Utils on Linux systems. This script checks for vulnerable versions and assists in uninstalling them while attempting to install a safe version automatically.

• Immediate Action:
  • Conduct a full system inventory to identify installations of the affected xz-utils versions (5.6.0 or 5.6.1).
  • Run the detection command provided (‵strings $(which xz) | grep ‘5\.6\.[01]’‵) on all systems, and pay special attention to Alpine Linux where the output may not reflect patched versions.
• Patch Management:
  • Downgrade xz-utils to the last known secure version (5.4.6) on all affected systems as soon as possible.
  • For Alpine Linux, ensure the use of Alpine Package Manager (‵apk list xz‵) to confirm the exact version of xz installed post-patch.
• Service Restart:
  • After downgrading, restart the OpenSSH server (‵sudo systemctl restart ssh‵) to remove potentially patched code from memory.
• Kill Switch Activation:
  • If downgrading is not immediately possible, implement the backdoor’s “kill switch” by adding the specific string (‵yolAbejyiejuvnup=Evjtgvsh5okmkAvj‵) to ‵/etc/environment‵ and restart SSH and Systemd.
• Verification:
  • Post-remediation, re-run the detection command to ensure no affected versions remain active.
  • Confirm that the kill switch is active by checking for the absence of the malicious functionality.
• Communication:
  • Inform all stakeholders of the vulnerability and the actions taken.
  • Communicate continuously throughout the remediation process, providing updates as new information becomes available.

What to expect next

The cybersecurity community is still unraveling the impact of the XZ backdoor incident. As multiple analyses are ongoing, new revelations are expected to emerge, potentially altering our understanding of the breach’s scope and sophistication. TPRM professionals and organizations must remain vigilant and be ready to adjust their strategies as more information becomes available.

The discovery of the XZ Backdoor highlights the hidden dangers within the open-source software supply chain. It demonstrates how attackers, potentially with state sponsorship, exploit the trust that has been built over the years to embed vulnerabilities. This incident, fortunately, uncovered, raises questions about what other undetected threats may still be lurking in open-source components, emphasizing the sophisticated strategies attackers use to breach systems.

What are the good resources to get a more detailed analysis of XZ-Backdoor

Below are the resources that were extremely helpful to me when I analyzed the incident and wrote this article. I am certain that many other good analyses will surface in the following days.

A detailed analysis of the vulnerability along with exploit code and detection (First discovery by Andres Freund). 

CISA Alert: https://www.cisa.gov/news-events/alerts/2024/03/29/reported-supply-chain-compromise-affecting-xz-utils-data-compression-library-cve-2024-3094 

Good threads on X (formerly Twitter) by Rachel Tobac and Grugq about the social engineering side of the attack. ​​  

Also, Rob Mensching, CEO and a long-time open-source maintainer, provides a good perspective here.

A detailed technical analysis by Sam James.

A detailed backdoor analysis is provided here.

Detection Script by Fabio Baroni.

Malwar3Ninja also provides some detection mechanisms on X.

A nice Threat Hunt Query is provided by Bert-Jan. 

Threat Brief by Palo Alto Networks that also provides impact analysis.

A chronological analysis of the incident by Evan Boehs.

A very good analysis by Gynvael Coldwind. 

A good blog about the overall vulnerability by JFrog.
Redhat’s alert for Fedora Linux 40 and Fedora Rawhide users.

Conclusion

As the dust settles on the CVE-2024-3094 crisis, it’s clear that we’ve all got a bit more homework to do when it comes to keeping our digital doors locked tight, especially in the open-source neighborhood we all rely on. This isn’t just a hiccup in the tech world; it’s a serious nudge for everyone, especially those in Third-Party Risk Management, to stay on their toes.

The subtlety of CVE-2024-3094 extends beyond its initial compromise, affecting the very tools we rely on for security assurance. The vulnerability’s ability to bypass conventional logging not only facilitates the stealthy execution of unauthorized actions but also poses profound questions for the field of digital forensics. Identifying the root cause of an intrusion becomes a daunting task when the usual evidentiary trails vanish into thin air.

The story here isn’t just about a sneaky bit of code; it’s about learning how to better watch each other’s backs. From here on out, it’s about asking the tough questions, sharing what we know, and making sure a quick fix isn’t just a Band-Aid over a larger problem. We’re all in this together, and it’s on us to learn from these slip-ups so we can build a safer, savvier cyber world.

Special thanks to Colin van Niekerk from BlckRhino for his invaluable feedback, which helped refine and clarify this post.