Bonus: the open-source red-teaming toolbox, and how to choose

A dated survey of the actively maintained open-source tools for scanning, attacking, evaluating and defending AI systems, with how to choose between them and what to check before you adopt one.

Diese Seite gibt es vorerst auf Englisch.

The tool landscape changes quickly, so this lesson gives you a method as well as a list. The table below is a snapshot of public repository metadata read from GitHub’s interface on 12 October 2026. The descriptions are each project’s own words. “Last push” is the date of the most recent push to the repository. Stars measure attention, not quality, and a recent push shows that the code is being worked on, not that it is good. Read each project’s documentation before you adopt it.

Public GitHub repository metadata, read 12 October 2026. Sorted by stars, except where noted.
Project	What it says it is	Licence	Stars	Last push
promptfoo	Test your prompts, agents and RAGs; red teaming, pentesting and vulnerability scanning for AI	MIT	25,928	2026-10-12
garak (NVIDIA)	The LLM vulnerability scanner	Apache-2.0	9,536	2026-10-10
NeMo Guardrails (NVIDIA)	An open-source toolkit for adding programmable guardrails to LLM-based conversational systems	Apache-2.0	7,280	2026-10-12
Adversarial Robustness Toolbox (ART)	A Python library for machine-learning security: evasion, poisoning, extraction and more	MIT	6,266	2026-10-08
Giskard (giskard-oss)	Open-source evaluation and testing library for LLM agents	Apache-2.0	5,888	2026-10-12
PyRIT (Microsoft)	The Python Risk Identification Tool for generative AI	MIT	4,605	2026-10-12
PurpleLlama (Meta)	A set of tools to assess and improve LLM security	No single licence reported by GitHub: read the repository	4,425	2026-09-29
agent-scan (Snyk)	A security scanner for AI agents, MCP servers and agent skills	Apache-2.0	3,132	2026-10-09
DeepTeam (Confident AI)	A framework to red team LLMs and AI agents	Apache-2.0	3,008	2026-10-01
Inspect (UKGovernmentBEIS)	A framework for large language model evaluations	MIT	2,987	2026-10-11
AI Red Teaming Playground Labs (Microsoft)	Labs to run AI red-teaming training, including infrastructure	MIT	2,094	2026-02-13
Augustus (Praetorian)	An LLM vulnerability scanner written in Go	Apache-2.0	302	2026-10-10
LLM Guard (Protect AI)	The security toolkit for LLM interactions	MIT	3,218	Archived; last push 2026-07-08
HarmBench	A standardised evaluation framework for automated red teaming and robust refusal	MIT	1,068	2024-08-16

Which tool for which job

Job	Start with	Why
Broad scan of a model or endpoint from the command line	garak or Augustus	Both organise probes, detectors, generators and buffs, with reports you can keep (lesson 11)
Red teaming an application and running it in CI	promptfoo	Its documentation covers plugins, strategies, targets and grading, with CI/CD integration and JUnit export
Scripted attacks, multi-turn conversations and other modalities	PyRIT	Orchestrators, converters, targets, scorers and memory, as shown in the series (lesson 12)
A general evaluation harness for models and agents	Inspect or Giskard	Built for evaluation and testing rather than only attacks
Agents and MCP servers	agent-scan	Describes itself as a scanner for agents, MCP servers and agent skills
Guardrails to test and compare	NeMo Guardrails, and Llama Guard as a classifier model	Open building blocks for the filtering layer (lesson 15)
Machine-learning-level attacks on classifiers	ART	Evasion, poisoning and extraction for classic models
Learning and training	The Playground Labs	Twelve challenges you can run locally, with a planted objective to score

Five questions before adopting a tool

Does it match the job? A model scanner, an application tester and an evaluation harness are different things.
Is it maintained? Check the last push and whether the project is archived. LLM Guard, for example, is archived at the time of reading.
What is the licence? Check it yourself. GitHub could not report a single licence for some projects.
What leaves your network? Most tools send prompts to a model API. Know which provider sees your data and your attack content.
Can you reproduce a result? Look for versioned releases, fixed seeds where offered, and exportable logs.

Using several tools

Different scanners ship different probes. Running two gives wider coverage, but the raw numbers are not comparable, because their probes, detectors and defaults differ (lesson 14). Pick one as your trend line, keep it fixed, and use the others to look for gaps in the coverage matrix from lesson 7.

Das Wichtigste

Choose a tool by the job: scanning, application testing, scripted attacks, evaluation, agents, guardrails or classic machine learning.
Stars show attention. Recent activity, archival status and licence tell you more.
Check what data a tool sends to which provider before you run it.
Use one tool as a fixed trend line and others to find coverage gaps.

Testen Sie sich

1. What does a recent push to a repository show?
- The code is being worked on, not that it is good — Richtig.
- The tool has been independently audited
- The tool is free of bugs
- The licence changed
Activity is a maintenance signal, not a quality guarantee.
2. Which tool describes itself as a scanner for AI agents and MCP servers?
- agent-scan — Richtig.
- HarmBench
- ART
- MLflow
That is how its repository describes it.
3. Why check what leaves your network?
- Many tools send prompts to a model API — Richtig.
- Tools never use the network
- It speeds up scans
- Licences require it
Attack content and your data may go to a provider.
4. Why are raw failure rates from two scanners not comparable?
- Their probes, detectors and defaults differ — Richtig.
- They use different colours
- One is faster
- They both report zero
Compare within one fixed setup, not across tools.

Mit FireAI ausprobieren

Wenden Sie diese Lektion auf Ihrem eigenen Mac an.

Quellen

Setzen Sie es auf Ihrem Mac um

Teste alle Funktionen 17 Tage kostenlos, ohne Karte.

Download für Mac Doku