Vibe coding, shadow IT, and the $600,000 lesson from METR's own dashboard

In March 2026, a researcher at METR, an organisation whose entire job is evaluating how capable and how risky frontier AI models are, built a small agent dashboard the way a lot of people build things now: by describing what they wanted to an AI coding assistant. It worked. It also had a fail-open authentication bug, meaning the login screen could be bypassed rather than enforced, and it was sitting on a public instance. An attacker found it, asked the exposed agent directly for its API key, added a persistent SSH key, and spent three weeks quietly burning through roughly $600,000 in AI credits before anyone noticed. METR disclosed the whole thing on 31 August 2026.

Nobody at METR was careless in the way that word usually implies. Someone solved a problem, quickly, with a tool that made it easy. That’s precisely what makes this worth writing about. AI has collapsed the barrier to building software far faster than it has collapsed the barrier to understanding how to build it safely. Vibe coding lets people reach “it works” before they understand “it is safe.” That gap is where the risk sits, and it’s showing up in real incidents, not hypotheticals.

Key takeaways

• AI-assisted development separates the ability to build software from the knowledge traditionally required to build it safely: authentication, trust boundaries, secrets handling, dependency risk, and secure architecture.
• The METR incident is a concrete, dated example: a fail-open auth flaw in a vibe-coded dashboard, three weeks undetected, roughly $600,000 in stolen AI credits.
• Open source supply chain risk has become a live attack path, not a background vulnerability count. Sonatype had logged 1.8 million malicious packages by Q2 2026, and Google’s threat intelligence team has tracked large, coordinated campaigns across PyPI, npm, and Docker Hub since March 2026.
• The more subtle risk is psychological: AI removes the friction that used to stop someone who didn’t understand a concept. A blocked implementation used to be a warning sign. A working one, generated in seconds, no longer is.
• This is “shadow IT” evolved: employees no longer just adopt unapproved software, they build it. The fix isn’t banning the practice, it’s extending proper security guardrails to everyone doing it.

“It works” is not the same claim as “it is safe”

A novice builder tends to think in terms of features: “I need a login,” “the user enters a value here,” “I’ll connect this to the database.” A security practitioner asks a different set of questions about the exact same feature: fail-open or fail-closed, what identity does this run under, who else can reach this, what happens if the input is malicious. Those are fundamentally different mental models, and AI is very good at producing syntactically competent code for the first without the person driving it ever encountering the second.

This shows up as recognisable categories of failure. Authentication and access control mistakes, where “add a login” doesn’t come with an understanding of session handling, privilege separation, or secure defaults, is exactly what happened at METR. Input validation gaps follow the same pattern: a builder thinks “the user enters a value here,” while an attacker treats everything arriving at that point as hostile until proven otherwise, which is how SQL injection, path traversal, server-side request forgery, and insecure deserialisation keep reappearing in new code despite being decades-old problems. AI can generate defensive-looking code in response to a prompt. It’s much less reliable at knowing, and telling you, when that defence is incomplete.

Secrets end up everywhere

API keys, database passwords, and cloud tokens increasingly get embedded directly in source code, environment files, build logs, or client-side JavaScript, because to someone building quickly, a key looks like “the thing the API needs” rather than a credential with a blast radius extending well past the small tool it was pasted into. That’s exactly the mechanism behind the METR loss: one exposed API key, reachable because authentication silently failed open, turned into a persistent foothold and a five-figure-a-week bill. It also matches what Google’s threat intelligence team is seeing industrially: supply chain campaigns hitting npm, PyPI, and Docker Hub through April and May 2026 targeted secrets specifically, API keys, cloud credentials, SSH keys, and CI/CD tokens, because a stolen credential is more valuable and more reusable than a single compromised machine.

The dependency problem is now an attack path, not a vulnerability count

Someone vibe coding a “2,000-line application” may have no visibility into the hundreds or thousands of transitive dependencies that application actually pulls in, who maintains them, what runs during installation, or whether a maintainer account has been compromised. NIST’s Secure Software Development Framework treats this as its own risk category precisely because open source provenance, integrity, and maintenance status vary wildly from one project to the next and are rarely obvious from the package name alone.

The scale is no longer marginal. Sonatype’s research had logged 1.8 million malicious open source packages by the end of Q2 2026 alone, with npm accounting for 96.6% of that quarter’s malicious package activity. Google’s threat intelligence team has been tracking a specific actor, UNC6780 , running large-scale compromises across PyPI, npm, and Docker Hub since March 2026, including abuse of the pull_request_target GitHub Actions trigger to obtain repository secrets and write access, followed by credential-stealing malware monetised through direct sale or ransomware partnerships. Twenty years ago the open source security question was “has a vulnerability been found in this library.” The current question is “is this actually the library we think it is, and can we trust the route by which it arrived.” CISA’s response is correspondingly practical: pin dependencies to specific trusted versions, pull packages only from known sources, and assess risk based on how a component was produced rather than simply who published it.

Secure architecture is invisible until someone asks about it

AI will happily help connect a browser to an API, a database, cloud storage, an AI model, and a third-party service, and produce something that functions correctly end to end. A security review of the identical diagram asks a different set of questions: who can talk to the database directly, what identity does the API run under, can the browser invoke anything privileged, where do credentials actually live, what happens if the AI’s own output is malicious, are tenants properly separated, what happens if someone edits an identifier in the URL. The person who commissioned the app, and quite possibly the AI that built it, may never have surfaced any of those questions, because nothing about “it works” forces them to.

Crossing the line from a tool to an information system

A common and underappreciated version of this problem shows up in data protection rather than infrastructure. Someone solving a genuinely useful business problem builds an internal tool and feeds it employee records, customer data, or commercially sensitive material. Without necessarily realising it, they’ve created a new processing activity, a new data store, potentially a new processor relationship if it touches a third-party AI API, a retention question nobody has answered, and possibly an international transfer. None of that gets considered, because the person building it is thinking “I made a small tool,” not “I created an information system.”

That distinction matters more than it sounds like it should. The greatest risk in vibe coding may not be bad code at all. It’s that people don’t realise they’ve crossed the line from creating a useful tool into operating an information system, and an entirely different set of obligations, Records of Processing Activities , a DPIA where the processing is high risk, a defined lawful basis , applies the moment they do, whether or not anyone noticed the line being crossed.

Nobody decided this was now production software

A developer’s instinct is “it runs.” An operations or security team’s instinct is “what happens when it doesn’t.” Who patches it, who monitors it, who restores it from backup, what happens when a dependency disappears or the AI platform changes its API, what happens when the person who built it leaves. This becomes genuinely dangerous through a very plausible, very ordinary lifecycle: experiment becomes useful tool, useful tool gets adopted by a team, a team’s dependency becomes something customers rely on, and at no point does anyone formally decide “we are now operating production software” or apply the controls that decision should trigger.

AI removes the point where ignorance used to become visible

Traditional programming has friction built into it. Someone who doesn’t understand OAuth typically gets stuck trying to implement it, and getting stuck is itself a signal, often the moment someone stops, asks for help, or scopes the feature down. AI frequently removes that friction. Typing “add Google authentication” and getting something that appears to work thirty seconds later is a genuine productivity gain, but it also means a person can succeed at execution without ever passing through the point where their own lack of understanding would previously have stopped them. Successful execution gets mistaken for successful engineering, and the two are not the same claim.

Shadow IT 2.0

Traditional shadow IT was employees signing up for SaaS tools without approval. The AI-assisted version is employees building software without approval; a department lead saying “I built us a little customer portal,” which IT later discovers is running on a personal hosting account, authenticating against a database the employee created themselves, calling an AI API with a token sitting in a plaintext environment file, pulling in a dozen open source packages nobody has reviewed, and storing real customer data. Nobody behaved maliciously or even carelessly by their own lights. They were solving a problem with the tools available to them, which is precisely what makes this harder to respond to than the old version of shadow IT. “Nobody is allowed to build anything” isn’t a credible policy when the productivity upside is this large, and it won’t be followed anyway.

It’s also shadow AI, not just shadow IT, and that’s where ISO 42001 becomes directly relevant rather than a stretch. An organisation running an AIMS is meant to maintain an AI system inventory covering every AI system “developed in house, deployed to staff, or embedded in procured tools.” An AI coding assistant an employee uses to vibe code an internal tool is an AI system in exactly that sense, and so, arguably, is any AI feature the assistant happily bolts onto the output. If the inventory only captures customer-facing AI products, it’s missing the AI usage most likely to be ungoverned.

What actually works: guardrails, not a ban

The realistic governance answer is giving citizen developers the same security defaults a professional engineering team would already have, rather than trying to stop the practice outright. In concrete terms, that means:

• Approved hosting and identity, so tools don’t end up on personal cloud accounts authenticated against nothing in particular.
• Managed secrets, so API keys and credentials never live in source code, environment files, or client-side JavaScript in the first place.
• Automatic dependency and code scanning (SCA and SAST) on anything that gets deployed, catching known-vulnerable or suspicious packages before they ship rather than after.
• Approved package sources and pinned versions, matching CISA’s guidance directly: trusted registries only, specific pinned versions rather than always-latest.
• A lightweight, fast review gate before anything reaches real customers or real data, not a six-week change advisory board, something proportionate enough that people actually use it.
• Basic data classification rules, so a tool touching personal or commercially sensitive data gets flagged for the RoPA and DPIA questions before it’s live, not after a regulator asks.
• Logging and backup standards applied by default, so “who patches this” and “where’s the backup” have answers from day one.
• An AI system inventory that actually includes staff-used coding assistants, not just customer-facing AI products, so the tools behind vibe-coded software get the same visibility as everything else.

Several of these map directly onto existing ISO 27001 Annex A controls, A.8.25 (secure development life cycle), A.8.28 (secure coding), A.8.29 (security testing in development), A.5.23 (information security for cloud services), and A.5.9 (inventory of assets), which means an organisation with a mature ISMS is closer to being ready for this than it might think. Where the AI coding assistant itself is the concern rather than just its output, that’s an ISO 42001 inventory and AI risk assessment question, and the two standards are designed to run alongside each other rather than compete for the same evidence. The gap usually isn’t the control framework, it’s that nobody has yet decided those controls apply to a department’s internally built tool the same way they apply to the main product.

Practical questions to ask now

• If an employee built a working internal tool this month using an AI coding assistant, would your security or IT team even know it exists?
• Do any internally built tools store customer or employee data, and if so, does a RoPA entry and lawful basis exist for that processing?
• Are API keys and credentials for these tools stored in a secrets manager, or are they sitting in code, config files, or someone’s notes?
• Is there a fast, lightweight review step before an internally built tool reaches real customers or real data, or does adoption happen silently?
• If the person who built a now-relied-upon internal tool left tomorrow, does anyone else know how to patch, monitor, or restore it?

Closing

None of this is an argument against AI-assisted development, or open source, or letting non-engineers build things. All three are producing real value, and trying to ban any of them is both the wrong call and not one that will actually hold. The argument is narrower and harder to ignore: the identity of “the developer” has changed faster than the security model built around that role, and the organisations getting caught out aren’t the ones acting recklessly, they’re the ones who never noticed a line had been crossed. METR’s own postmortem makes that point better than any hypothetical could.

If you’d like help putting security guardrails around AI-assisted and citizen development in your organisation, or governing the tools your teams have already built, get in touch . Our AI Governance and Technical Security services both cover this ground.