Critical Security Flaws Discovered in NVIDIA Container Toolkit

Nvidia Container Toolkit found to have worrying security flaws

Vulnerability could allow hackers to escape the container and cause havoc NVIDIA Container Toolkit and GPU Operator were carrying a critical vulnerability that allowed threat actors access to the underlying host's file system, experts have warned. Cybersecurity researchers at Wiz discovered and reported the flaw, tracked as CVE-2024-0132, and carries a vulnerability score of 9.0/10 - critical, to Nvidia on September 1, 2024. It is described as a Time-of-Check Time-of-Use (TOCTOU) vulnerability. To be abused the tools need to be set up in default configurations - then, a threat actor could craft a special container image that grants them access to the host file system. "A successful exploit of this vulnerability may lead to code execution, denial of service, escalation of privileges, information disclosure, and data tampering," the company said in a security advisory. The bug affected all NVIDIA Container Toolkit versions to v.1.16.2, and all NVIDIA GPU Operator versions until 24.6.2, which were the first ones to have addressed the flaw. It is also worth mentioning that the vulnerability does not work when Container Device Interface (CDI) is used. "The urgency with which you should fix the vulnerability depends on the architecture of your environment and the level of trust you place in running images," the researchers said in their technical write-up. "Any environment that allows the use of third party container images or AI models - either internally or as-a-service - is at higher risk given that this vulnerability can be exploited via a malicious image." They stressed that single-tenant compute environments could be at risk if a user downloads a malicious container image from an untrusted source, giving the crooks access to the workstation. In orchestrated environments such as Kubernetes (K8), an attacker with permission to deploy a container could access data and secrets of other applications running on the same node or cluster.

Critical NVIDIA Container Toolkit Vulnerability Could Grant Full Host Access to Attackers

A critical security flaw has been disclosed in the NVIDIA Container Toolkit that, if successfully exploited, could allow threat actors to break out of the confines of a container and gain full access to the underlying host. The vulnerability, tracked as CVE-2024-0132, carries a CVSS score of 9.0 out of a maximum of 10.0. It has been addressed in NVIDIA Container Toolkit version v1.16.2 and NVIDIA GPU Operator version 24.6.2. "NVIDIA Container Toolkit 1.16.1 or earlier contains a Time-of-check Time-of-Use (TOCTOU) vulnerability when used with default configuration where a specifically crafted container image may gain access to the host file system," NVIDIA said in an advisory. "A successful exploit of this vulnerability may lead to code execution, denial of service, escalation of privileges, information disclosure, and data tampering." The issue impacts all versions of NVIDIA Container Toolkit up to and including v1.16.1, and Nvidia GPU Operator up to and including 24.6.1. However, it does not affect use cases where Container Device Interface (CDI) is used. Cloud security firm Wiz, which discovered and reported the flaw to NVIDIA on September 1, 2024, said it would allow an attacker who controls the container images run by the Toolkit to perform a container escape and gain full access to the underlying host. In an hypothetical attack scenario, a threat actor could weaponize the shortcoming by creating a rogue container image that, when run on the target platform either directly or indirectly, grants them full access to the file system. This could materialize in the form of a supply chain attack where the victim is tricked into running the malicious image, or, alternatively, via services that allow shared GPU resources. "With this access, the attacker can now reach the Container Runtime Unix sockets (docker.sock/containerd.sock)," security researchers Shir Tamari, Ronen Shustin, and Andres Riancho said. "These sockets can be used to execute arbitrary commands on the host system with root privileges, effectively taking control of the machine." The problem poses a severe risk to orchestrated, multi-tenant environments, as it could permit an attacker to escape the container and obtain access to data and secrets of other applications running on the same node, and even the same cluster. Technical aspects of the attack have been withheld at this stage to prevent exploitation efforts. It's highly recommended that users take steps to apply the patches to safeguard against potential threats. "While the hype concerning AI security risks tends to focus on futuristic AI-based attacks, 'old-school' infrastructure vulnerabilities in the ever-growing AI tech stack remain the immediate risk that security teams should prioritize and protect against," the researchers said.

Critical Nvidia bug allows container escape, host takeover

A critical bug in Nvidia's widely used Container Toolkit could allow a rogue user or software to escape their containers and ultimately take complete control of the underlying host. The flaw, tracked as CVE-2024-0132, earned a 9.0 out of 10 CVSS severity rating, and affects all versions of Container Toolkit up to and including v1.16.1, and Nvidia GPU Operator up to and including 24.6.1. Nvidia issued a fix on Wednesday with the latest version of Container Toolkit (v1.16.2) and Nvidia GPU Operator (v24.6.2). The vulnerability does not impact use cases where Container Device Interface (CDI) is used. This particular library is used across clouds and AI workloads. According to infosec house Wiz, 33 percent of cloud environments have a buggy version of Nvidia Container Toolkit installed, rendering them vulnerable. Wiz security researchers found and disclosed the bug on September 1, and the GPU giant has confirmed it is as concerning as the cloud security shop makes it out to be. "A successful exploit of this vulnerability may lead to code execution, denial of service, escalation of privileges, information disclosure, and data tampering," Nvidia warned in its security advisory. Again, this is exploitable by someone or something that's been allowed to or managed to run or run within a container on a vulnerable host. CVE-2024-0132 is a Time of Check Time of Use (TOCTOU) vulnerability, a type of race condition. This can allow the attacker to gain access to resources that they should not have access to. Specific to Nvidia Container Toolkit: "Any environment that allows the use of third party container images or AI models - either internally or as-a-service - is at higher risk given that this vulnerability can be exploited via a malicious image," Wiz kids Shir Tamari, Ronen Shustin, Andres Riancho said in a write-up about the bug. To exploit CVE-2024-0132, an attacker would need to craft a specially designed image and then get the image to run on the target platform, either indirectly, by convincing/tricking the user into running the malicious image, or directly, if the attacker has access to shared GPU resources. In a single-tenant compute environment, this could happen if a user downloads a malicious container image -- say, via a social engineering attack where the user believes the container image is coming from a trusted source. In this scenario, the attacker could then take over the user's workstation. In a shared environment, such as Kubernetes-powered one, however, a miscreant with permission to deploy a container could escape it and then access data or secrets of other applications on the same node or cluster, the researchers noted. This second scenario "is especially relevant for AI service providers that allow customers to run their own GPU-enabled container images," they warned. "An attacker could deploy a harmful container, break out of it, and use the host machine's secrets to target the cloud service's control systems," the researchers continued. "This could give the attacker access to sensitive information, like the source code, data, and secrets of other customers using the same service." Wiz isn't providing too many technical details about how to exploit the vuln because the security shop wants to ensure that vulnerable organizations have time to deploy the fix -- and not have their host system taken over with root privileges. But the researchers promised more to come soon, including exploit details, so it's a good idea to get ahead of the would-be attackers on this one. ®