AI assistants now write a large share of new code in many teams. That code goes through the same compilers and the same production servers as human code, but it fails in different ways. It is fluent, confident and often correct — which makes the incorrect parts harder to spot. This article summarises what security research has found about AI-generated code, explains the new supply-chain risk of hallucinated packages, and lays out the controls we use so that faster coding does not mean more incidents.

What the research says

Several studies have looked at the security of code produced with AI assistance:

  • Vulnerable suggestions are common in risky scenarios. Researchers prompted GitHub Copilot with 89 scenarios designed around CWE weaknesses; about 40% of the 1,689 generated programs contained vulnerabilities (Pearce et al., "Asleep at the Keyboard?"). The models have improved since, but the scenarios — SQL queries, path handling, crypto — are still where mistakes cluster.
  • Developers with assistants write less secure code and feel more confident about it. In a user study at Stanford, participants with access to an AI assistant wrote significantly less secure code on several tasks and were more likely to believe their code was secure (Perry et al., 2023).
  • Package hallucinations are systematic. Across 576,000 code samples from 16 models, 19.7% of recommended packages did not exist; the researchers counted over 205,000 unique hallucinated package names, and many were repeated consistently across runs (Spracklen et al., 2024).

The second finding is the most important for process design: the risk is not only in the code, it is in the reduced scrutiny. Controls must compensate for overconfidence.

The vulnerability classes to watch

AI-generated code tends to reproduce patterns that are common in public code, including insecure ones. The usual suspects map directly to the OWASP Top 10:2025:

Weakness Typical AI-generated form Control
Injection (SQL, command, template) String-built queries, shell=True, unescaped templates Parameterised APIs, Semgrep rules, review checklist
Broken access control New endpoint copied from a public one without auth check Auth-by-default middleware, route tests
Cryptographic failures MD5/SHA1 for passwords, hard-coded IVs, Math.random() tokens Approved crypto wrappers, linters
Security misconfiguration CORS: *, debug mode on, permissive cookie flags Config templates, IaC scanning
SSRF Fetching user-supplied URLs server-side Allow-lists, egress controls
Hard-coded secrets Example API keys left in code Secret scanning, push protection
Deserialisation pickle.loads, Java native serialisation on input Banned API list

None of these are new. What is new is the volume: when an agent writes 500 lines in a minute, a single copied pattern can appear in a dozen places.

Slopsquatting: the hallucinated dependency attack

When a model suggests pip install fastapi-auth-helpers and the package does not exist, an attacker can register it with malicious code. Because models repeat the same hallucinations, attackers can harvest likely names by prompting models themselves. This attack has been nicknamed slopsquatting, and it is a twist on typosquatting that targets machines rather than humans.

Agentic tools make it worse: an agent that sees an import error may run the install command itself. Defences:

  1. Never auto-approve dependency installation in agent permissions.
  2. Verify every new dependency: does it exist, who maintains it, how old is it, how many downloads, does the repository link match? Tools such as OpenSSF Scorecard and deps.dev help.
  3. Use a private registry proxy (Artifactory, Nexus, Verdaccio, a pip/npm mirror) with allow-listing or a minimum package age policy for new packages.
  4. Lock files and integrity hashes must be committed and verified in CI.
  5. Generate an SBOM and scan it on every build, as described in container supply chain security.

Layered controls for AI-assisted development

Think of controls in four layers, from the developer's keyboard to production.

Layer 1: The assistant itself

  • Security instructions in the project file. Add a short section to CLAUDE.md or AGENTS.md: "Use db.query() with parameters; never build SQL strings. All new routes go through withAuth(). Do not add dependencies without asking." Assistants follow concrete rules far better than generic "write secure code" prompts.
  • Restricted permissions. No network, no installs and no access to credential files without confirmation. See agentic coding best practices.
  • Hooks that block edits to sensitive paths (auth, crypto, payments) or require a human for them.

Layer 2: Before commit

  • Pre-commit secret scanning with Gitleaks or similar.
  • Fast SAST with Semgrep rules tuned to your stack. Custom rules for your internal APIs ("never call rawQuery") catch exactly the patterns an assistant copies.
  • Developer review of every line. Ownership stays with the human who commits.

Layer 3: Pull request and CI

  • Full SAST with CodeQL or equivalent, plus dependency scanning.
  • AI security review as an extra pass, prompted with your threat model and banned patterns — see AI code review. It is good at explaining why something is dangerous, which helps developers learn.
  • New-dependency gate: a CI job that fails when a lock file adds a package, until a human approves it.
  • Tests for security properties: authorisation tests for every route, input validation tests for parsers.

Layer 4: Runtime

  • Least-privilege service accounts and network egress controls limit the blast radius of any missed vulnerability.
  • WAF and rate limits in front of public endpoints.
  • Observability and alerting on auth failures, unusual queries and error spikes; see OpenTelemetry.

A Semgrep rule example

Custom rules are where you encode your codebase's specific dangers. This rule flags string-built SQL with a project-specific client:

rules:
  - id: no-string-built-sql
    languages: [typescript]
    severity: ERROR
    message: >
      Build SQL with db.query(sql, params). String concatenation or template
      literals with variables enable SQL injection.
    patterns:
      - pattern-either:
          - pattern: db.query(`...${$X}...`)
          - pattern: db.query("..." + $X + "...")

Rules like this cost minutes to write and run in seconds, and they turn a lesson learned once into a check that runs forever.

Prompting for security, without relying on it

Security-focused prompting helps at the margin. Asking an assistant to "list the security assumptions of this function and the inputs an attacker controls" before implementation produces better code than asking for the implementation directly. Asking it to write negative tests — invalid input, missing permissions, oversized payloads — catches classes of bugs early.

But prompting is not a control. Models can be steered by malicious content in the context — a poisoned README, an issue description, a dependency's docstring — which is the prompt injection problem described in AI agent security. Deterministic checks in layers 2–4 are what actually enforce policy.

Licensing and provenance

Security also includes legal exposure. Some vendors offer filters that block suggestions matching public code, or show references to the source. Enable them for proprietary products, keep license scanning in CI, and record in the PR description when large blocks of code came from an assistant. For regulated environments, this provenance trail also supports audit requirements.

A checklist for teams

  • Project instruction file contains concrete security rules for your stack
  • Agents cannot install packages, access the network or read secrets without approval
  • Pre-commit secret scanning is enabled for every developer
  • SAST with custom rules runs on every PR
  • New dependencies require explicit human approval
  • Lock files and SBOMs are generated and scanned on every build
  • Every route has an authorisation test
  • AI review runs with your threat model, and humans still approve
  • Security training covers AI-specific risks: overconfidence, hallucinated packages, prompt injection

FAQ

Is AI-generated code less secure than human code? Research suggests developers using assistants are more likely to accept insecure patterns and less likely to notice them. The code itself is not inherently worse; the review around it often is.

Should we ban AI assistants for security-critical modules? Many teams restrict agents from editing auth, crypto and payment code without senior review. That is reasonable. A blanket ban usually drives usage underground.

Do newer models still hallucinate packages? Less often, but not never, and attackers only need it to happen occasionally. Verification is cheap; keep it.

Can we let the AI fix vulnerabilities found by scanners? Yes, and it works well for known patterns. Review the fix, and add a test that would fail on the vulnerable version.

Sources

  1. Pearce et al. (2021). Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions.
  2. Perry et al. (2023). Do Users Write More Insecure Code with AI Assistants?
  3. Spracklen et al. (2024). We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs.
  4. OWASP. OWASP Top 10:2025 and Top 10 for LLM Applications.
  5. OpenSSF. Scorecard; Google. deps.dev.
  6. Semgrep. Writing rules.