How to Red-Team an LLM in 2026 - A Step-by-Step Workflow

Most teams “red-team” their LLM the same lazy way - someone opens the chat, types a few jailbreak prompts they saw on social media, watches it refuse, and calls it secure. That is not red-teaming. That is a demo.

Real red-teaming is a repeatable pipeline. You define an attack taxonomy, generate an adversarial dataset from it, run automated scans, score the results by attack type, fix the failures, and re-run the whole battery on every change. This guide walks that workflow end to end and points to the two open-source tools that ship the attacks, so you are scoring against a known threat model instead of improvising jailbreaks from a blank prompt box.

Step 1: Define your attack taxonomy

You cannot test for failures you have not named. Before any tooling, write down the categories of attack that actually matter for your application. A generic list is a fine starting point, but the risks that bite are the ones specific to what your app does.

A practical taxonomy usually spans four buckets:

• Prompt injection - can hidden instructions, in the user message or in retrieved content, override your system prompt. This is the top risk for any RAG or agent system. If you want the concept first, see the prompt injection glossary entry.
• Jailbreaks - roleplay framing, encoding tricks, or multi-turn manipulation that coaxes the model past its guardrails into disallowed content.
• Data extraction - attempts to leak the system prompt, training data, other users’ data, or secrets the app has access to.
• Harmful output - bias, toxicity, misinformation, or unsafe instructions, whether provoked or spontaneous.

The industry-standard reference to anchor this is the OWASP Top 10 for LLM Applications, which both tools below map their attacks to. Start from that framework, then add the risks unique to your domain - a medical chatbot and a coding agent have very different worst cases.

Step 2: Build an adversarial dataset

Once you know what you are testing for, you need concrete attack inputs to test with. An adversarial dataset is the red-team equivalent of a golden eval set - a collection of malicious inputs, each tagged with the attack type it represents and the behavior that would count as a failure.

You build it three ways, in increasing order of value:

1. Curated known attacks - published jailbreaks and injection patterns, grouped by category.
2. Generated variations - take each seed attack and have a model rewrite it into dozens of paraphrases, encodings and framings, because a filter that blocks one wording rarely blocks all of them.
3. Your own domain attacks - the ones that target your specific tools, data and system prompt.

The important habit is tagging. Every probe carries its attack category so your results roll up into “we block 95 percent of injection but only 40 percent of jailbreaks” rather than one meaningless pass rate. That per-category breakdown is what tells you where to spend fixing time.

Step 3: Run automated scans with a red-teaming tool

This is where you stop hand-writing attacks. Two open-source tools do the heavy lifting, and they suit different teams.

Promptfoo is the de-facto OSS red-teaming CLI. It ships 50+ attack plugins and has the strongest OWASP mapping in the category, with built-in OWASP LLM Top 10, OWASP Agentic and NIST presets. You describe the scan in declarative YAML, point it at your endpoint, and it fires the probes and reports failures by category. Because the config is YAML, your red-team suite lives in version control next to your app. It is MIT-licensed and free, with the Community edition capped at 10,000 red-team probes per month - enough for most teams to run real scans at no cost. One thing to know for 2026: Promptfoo was acquired by OpenAI in March 2026 and folded into OpenAI’s “Frontier” security work, with a stated commitment to stay open source under MIT.

DeepEval covers the same ground through DeepTeam, its separate red-teaming framework. DeepTeam is also Apache-2.0 and free, and ships 50+ vulnerabilities and 20+ attack vectors with OWASP Top 10 and NIST AI RMF coverage. If your team already lives in DeepEval’s pytest-style world, DeepTeam fits the same programmatic, Python-first shape - you script the red-team run the way you script your other tests.

Here is the shape of a Promptfoo red-team config (illustrative):

# promptfooconfig.yaml - illustrative
redteam:
  plugins:
    - owasp:llm          # the full OWASP LLM Top 10 preset
    - prompt-injection
    - jailbreak
    - pii
  numTests: 25           # variations generated per plugin
targets:
  - id: https
    config:
      url: https://your-app.example/chat

The point of the tool is coverage you could never write by hand - hundreds of tagged, generated probes fired automatically and scored by category.

Step 4: Score, triage and fix

A scan returns a pass/fail per probe. Now turn that into action. Roll the results up by attack category, then triage by severity times exposure - a data-extraction hole in a system with access to customer records outranks a mild toxicity slip in a sandbox.

Fixes usually come from three levers, applied in order:

• Input and output guardrails - a classifier or filter layer that catches injection and harmful content before or after the model. Our roundup of the best LLM guardrails tools covers the options, and the guardrails glossary entry explains the concept.
• System-prompt hardening - explicit instructions on what to refuse and how to treat retrieved content as untrusted data, not commands.
• Architecture - least-privilege tool access, so even a successful injection cannot reach data or actions it should not.

For the injection category specifically, our dedicated guide on how to detect prompt injection goes deeper on the guardrail-and-eval-test combination.

Step 5: Gate it in CI and re-run forever

A red-team scan you run once is a pentest with an expiry date. The model updates, the system prompt gets edited, a new tool gets wired in - and any of those can silently reopen a vulnerability you already closed.

So wire the scans into your pipeline. Run a fast subset of the attack battery on every pull request that touches prompts, tools or retrieval, and run the full suite on every release and on a weekly cron. Promptfoo’s YAML slots into CI cleanly, and DeepTeam runs in the same CI job as your DeepEval tests. When a new probe passes that used to fail, or a fixed one regresses, you find out at the PR - not from a security researcher’s email.

Bottom line

Red-teaming an LLM is a workflow, not a vibe. Name your attack taxonomy, build a tagged adversarial dataset, run automated scans, score by category, fix with guardrails and least privilege, then gate it in CI and re-run forever. Reach for Promptfoo if you want the strongest OWASP mapping in a YAML-in-git config, and for DeepEval’s DeepTeam if you want it Python-first alongside your existing test suite. Both are free and open source, so there is no excuse to still be testing your model’s safety by typing jailbreaks into a chat box by hand.