Red Team & Evaluation Tools

Attacking an agent by hand teaches you what is possible. Attacking it with tooling teaches you what is repeatable. The difference matters, because a finding you cannot reproduce on demand is an anecdote, and an anecdote does not get engineering time allocated to fix it.

These tools mostly do one of three things: generate adversarial inputs at scale, score the outputs against a policy, or block the bad ones at runtime. Know which category you are in at any moment.

NVIDIA garak

• Link: https://github.com/NVIDIA/garak
• What it is: An open-source LLM vulnerability scanner. It runs a catalog of probes (prompt injection, jailbreaks, toxicity, data leakage, encoding tricks) against a target model or endpoint and reports which probes succeeded.
• Install: pip install garak
• Why it matters for security: This is the closest thing the field has to nmap for models: broad, opinionated, and a fast way to get a first inventory of where a target is weak. Run it early so that your later hand-crafted attacks start from a baseline rather than from zero.

Microsoft PyRIT

• Link: https://github.com/Azure/PyRIT
• What it is: The Python Risk Identification Toolkit for generative AI. It provides orchestrators, converters, targets, and scorers so you can build automated multi-turn attack campaigns rather than one-off prompts.
• Install: pip install pyrit
• Why it matters for security: Real agent compromises are rarely single-turn. PyRIT’s orchestrator model lets you express “keep escalating until the scorer says the objective was met,” which is how you find the multi-step paths that single-shot scanners miss entirely.

promptfoo

• Link: https://github.com/promptfoo/promptfoo
• What it is: A test and evaluation framework for prompts, models, and agents, with a red-teaming mode that generates adversarial test cases and grades responses. Configuration is declarative YAML.
• Install: npm install -g promptfoo (or run it with npx promptfoo@latest)
• Why it matters for security: It is the easiest of these to put in CI. Once a prompt injection is expressed as a promptfoo test case, a regression in the system prompt fails the build instead of shipping. That transition, from finding to gate, is the thing hiring managers listen for.

Giskard

• Link: https://github.com/Giskard-AI/giskard
• What it is: An open-source testing library for ML and LLM applications, including automated scans for RAG and LLM systems that surface hallucination, harmfulness, and prompt-injection issues as a test report.
• Install: pip install giskard
• Why it matters for security: Giskard’s framing is quality-and-safety rather than pure attack, which is exactly the vocabulary a product team will accept. Use it when you need a report that a non-security stakeholder will actually read and act on.

Adversarial Robustness Toolbox (ART)

• Link: https://github.com/Trusted-AI/adversarial-robustness-toolbox
• What it is: A Python library for adversarial machine learning, covering evasion, poisoning, extraction, and inference attacks against classifiers and other classical models, plus defenses.
• Install: pip install adversarial-robustness-toolbox
• Why it matters for security: Agentic systems still contain classical models: the classifier that routes a request, the moderation model that gates output, the embedding model behind retrieval. ART is how you learn that those components are attackable too, and it gives you the vocabulary MITRE ATLAS uses.

Foolbox

• Link: https://github.com/bethgelab/foolbox
• What it is: A Python library for creating adversarial examples against machine learning models, with a clean implementation of the standard attack algorithms.
• Install: pip install foolbox
• Why it matters for security: Smaller and more focused than ART, which makes it the better teaching tool. Running one evasion attack yourself permanently changes how you read a vendor claim about model robustness.

LLM Guard

• Link: https://github.com/protectai/llm-guard
• What it is: A security toolkit that wraps LLM input and output with composable scanners: prompt-injection detection, PII redaction, secret detection, topic and toxicity filters.
• Install: pip install llm-guard
• Why it matters for security: Every red team engagement ends with someone asking “so what do we do about it.” Guardrail libraries are the honest answer and the honest limitation: they raise cost for the attacker without being a boundary. Deploy one, then bypass it, and you will be able to explain both halves credibly.

Supporting tooling

You will use these alongside the AI-specific scanners on almost every engagement. They are not agent tools; they are the reason your findings hold up.

An intercepting HTTP proxy

• Link: https://mitmproxy.org (Burp Suite Community and OWASP ZAP are the common alternatives)
• What it is: A proxy that intercepts, inspects, and rewrites HTTP/HTTPS traffic, including the API calls an agent makes to model providers and to its own tools.
• Install: pip install mitmproxy
• Why it matters for security: Agent frameworks hide the wire. A proxy is how you prove what was actually sent, catch an API key in a header where it should not be, and confirm that a tool call carried the arguments you think it did rather than the ones the framework logged.

gitleaks

• Link: https://github.com/gitleaks/gitleaks
• What it is: A scanner that detects hardcoded secrets, API keys, and tokens in git repositories and their history.
• Install: Install the binary via Homebrew (brew install gitleaks), a release archive, or the official Docker image.
• Why it matters for security: AI projects leak credentials at an unusual rate because notebooks, prompt files, and quick demos all accumulate keys. Run it against your own lab repositories before you publish them; a portfolio repo with a live key in the history is a bad first impression.

Syft and pip-audit

• Link: https://github.com/anchore/syft and https://github.com/pypa/pip-audit
• What it is: Syft generates a software bill of materials from a directory, image, or archive. pip-audit checks an installed Python environment or requirements file against known vulnerability data.
• Install: pip install pip-audit; install Syft from its official release artifacts or Homebrew.
• Why it matters for security: The AI dependency tree is enormous, fast-moving, and full of packages nobody has reviewed. Supply chain is a first-class agentic attack path, not a compliance checkbox, and an SBOM is the artifact that makes an incident answerable in hours instead of weeks.

How to use this page

Pair these tools with the threat taxonomies rather than running them blind. Map every finding to a technique in MITRE ATLAS so it lands in a language defenders already speak, and practice the full loop in Lab 3. The frameworks you will be attacking are on the agent frameworks page; the evidence you produce belongs in the observability toolchain. Full set at the tools index.