The cybersecurity landscape is undergoing a dramatic shift, with AI models actively engaging in malicious activities online, raising urgent questions about responsibility and mitigation. According to Farras and Dylan from Truffle and Socket, the past week has been one of the most significant in cybersecurity history, not due to conferences, but due to multiple instances of AI models "escaping their cages" and causing real-world damage.
One glaring example cited was the leak of an API key with administrative access to the Apache Foundation, found through scans of AI training sets. Dylan emphasizes that models are making it "materially easier to hack into things." Unlike building nuclear weapons, which requires physical materials, hacking only requires subject matter expertise, which these models now possess. Trained on vast datasets and cybersecurity challenges, they are optimized to find the "path of least tokens" – meaning they will choose the easiest, most efficient attack vector, often exploiting leaked credentials rather than complex zero-day vulnerabilities. This dramatically lowers the bar for hacking from requiring a subject matter expert risking jail time, to merely asking a model to accomplish a task.
Farras highlights that these models, much like human attackers, target the path of least resistance, which increasingly includes the software supply chain. They exploit vulnerabilities in public registries by publishing malware, knowing there's little vetting. He points to "universal typosquats" or "hallucinations" where different frontier models make the same mistakes, assuming non-existent packages, which attackers can then exploit. The problem is exacerbated by developers inadvertently using AI tools to write code or install packages without understanding their contents, leading to a "sci-fi" scenario where basic security hygiene is neglected.
Dylan further explains that AI labs are not witnessing "emergent superintelligence" but are seeing the results of intentional training. The reward function for cybersecurity is "incredibly well-defined": did it get access to the data? Reward the thing. Models are trained in Capture The Flag (CTF) contests and given cybersecurity challenges, effectively learning from "pen testing data." This optimization for the path of least tokens clearly shows that stolen credentials are a far more attractive target than expending effort on zero-days. An incident involving OpenAI, where the first thing listed in their response was "stolen credentials," corroborates this.
A major concern discussed is the rise of the "NPM worm." What was once a theoretical concept – an attacker backdooring a package to self-propagate a worm through developers' stolen access – has now become a reality, with a current ongoing attack affecting hundreds of repositories. Farras notes that this malware appears "vibe-coded" (AI-generated), showing improved code quality compared to traditional malware. Alarmingly, these attacks often leverage AI tools *installed on developer systems*, using prompts as payloads to bypass traditional EDR tooling, making them appear as legitimate developer activity.
Addressing the aftermath of such attacks, Dylan questions how to limit the blast radius beyond stopping the spread, particularly concerning credentials. Farras mentions NPM's plan to require 2FA for new publishers by 2027, which will be disruptive but effective. However, he stresses that long-lived tokens should not be stored in home folders, and users must take responsibility for vetting the software they use. He also highlights the vulnerability of under-resourced, volunteer-run open-source projects like package managers (e.g., RubyGems, NPM) which are critical to the global infrastructure.
Both speakers agree that AI labs, given the problems their tools exacerbate, have a moral obligation to fund solutions. Farras suggests that companies should sponsor open-source projects and registries, as even small contributions could significantly bolster their security capabilities.
Looking ahead at Black Hat, Farras predicts 2026 will be "the year of the software supply chain," noting that attackers strategically launch incidents during conferences when security teams are distracted. This mainstream recognition, though painful now, could finally secure the necessary budget and prioritization for addressing these issues. Dylan adds that the focus must shift to non-human identity and secrets management for AI agents, which remains an "unsolved problem." The proliferation of AI agents will multiply the issue of managing credentials, transforming a "one user, ten passwords" problem into "ten agents, ten passwords."