Giskard Review (2026)
Apache-2.0 testing and red-teaming library for LLM agents, strongest on adversarial security testing rather than quality metrics. The v3 rewrite requires Python 3.12+, which quietly rules it out for a lot of infrastructure.
Rating
Starting Price
Not published (Hub)
Free Plan
Yes
SDKs & Frameworks
2
Deployment
2
Best For
Teams that need adversarial testing and red teaming for LLM agents, especially in security-conscious or regulated settings, and who are on Python 3.12 or later.
Last Updated:
10 Things You Should Know About Giskard
- 1 Licensed Apache 2.0, an OSI-approved permissive license
- 2 The v3 architecture requires Python 3.12 or later
- 3 v3 is a modular set of focused packages, each carrying only the dependencies it needs
- 4 Built-in evals include string matching, comparisons, regex, semantic similarity and LLM-as-judge checks
- 5 LLM-as-judge checks available include Groundedness, Conformity and LLMJudge
- 6 The repository sits at roughly 5,414 GitHub stars and was updated in May 2026
- 7 Giskard Hub is the enterprise tier and its pricing is not publicly listed
Pros & Cons
Pros
- ✓ Genuinely strong at adversarial and security testing, which most eval frameworks barely attempt
- ✓ Apache 2.0, permissive with no copyleft complications
- ✓ The modular v3 packaging means you install only what you need rather than a monolithic dependency
- ✓ Works against black-box agents, so you can test systems you did not build and cannot instrument
- ✓ Healthy project with roughly 5,400 GitHub stars and updates through May 2026
Cons
- ✕ v3 requires Python 3.12 or later, which silently excludes any infrastructure pinned to 3.10 or 3.11
- ✕ The v3 rewrite is still maturing and some v2 features are not yet fully ported
- ✕ LLM-as-judge checks may require external API calls, which is a problem for air-gapped deployments
- ✕ Includes optional telemetry, worth reviewing before deploying in a sensitive environment
- ✕ Giskard Hub pricing is not published, so the commercial tier cannot be cost-compared
- ✕ Weaker than Ragas or DeepEval on conventional quality metrics - this is a security tool first
Features
A security tool wearing an eval framework’s clothes
Most of this category measures quality. Giskard attacks your system.
It is an Apache-2.0 Python library for testing and evaluating agentic systems, and its centre of gravity is automated red teaming and adversarial vulnerability scanning. Where Ragas asks whether your RAG answer was grounded and DeepEval asks whether you should block the deploy, Giskard asks what an adversary could make your agent do.
That is genuinely underserved. Most eval frameworks treat safety as one metric among many, if they address it at all.
It also works against black-box systems. Giskard wraps an LLM, an agent, or a multi-step pipeline behind one interface, which means you can test something you did not build and cannot instrument - a vendor’s agent, another team’s service, a third-party API. For security testing that matters, because the systems you most want to probe are frequently the ones you can see least of.
The Python 3.12 requirement
This deserves to be the first practical thing you check, and we have not seen it mentioned in a single comparison of eval frameworks.
Giskard v3 requires Python 3.12 or later. If your infrastructure is pinned to 3.10 or 3.11, it will not run.
That sounds like a footnote. It is not. Plenty of production environments are pinned - by a platform dependency, a base image, a data science stack that has not certified newer Python, or an internal policy. Upgrading the interpreter across a real deployment is often a substantially larger project than adopting an eval library was ever going to be.
It is also exactly the kind of constraint that surfaces after a team has decided, when someone actually tries to install it.
Check your Python version before you evaluate anything else here.
What the v3 architecture gets right
Credit where due: v3 is a modular set of focused packages, each carrying only the dependencies it needs.
Anyone who has installed a monolithic ML library into a production service and watched the dependency tree explode will appreciate this. You pull in the checks you use rather than everything the project has ever supported.
The check library spans string matching, regex, comparisons, semantic similarity, and LLM-as-judge checks including Groundedness, Conformity and LLMJudge. That is a sensible ladder from cheap deterministic checks to expensive model-graded ones - the cheap ones catch a surprising amount and cost nothing to run.
The caveat: the v3 rewrite is still maturing and some v2 features are not fully ported. Starting fresh, target v3. Migrating from v2, verify your specific capability exists rather than assuming parity.
Air-gapped deployment needs care
Two things to audit before deploying somewhere restricted.
Giskard includes optional telemetry. Review the setting.
More importantly, LLM-as-judge checks may require external API calls. The judge model has to run somewhere, and if that is a hosted provider then your prompts and outputs leave the network - which in an air-gapped context defeats the purpose entirely.
The deterministic checks (string matching, regex, comparisons) and semantic similarity are far less likely to need egress than the LLM-as-judge family. If you need judge-based checks in a sensitive environment, route them to a locally hosted model and verify no traffic escapes before you trust it.
Commercial tier
Giskard Hub is the enterprise platform - shared workspaces, collaborative annotation workflows, role-based access control, and continuous red teaming. Pricing is not published; contact sales.
One point in its favour: the open-source library is complete enough to be genuinely useful alone. Hub adds team workflow rather than unlocking core capability, so you are not being funnelled toward a paid tier to get basic functionality. That is a more honest open-core split than several competitors manage.
Project health looks fine - roughly 5,414 GitHub stars, updated May 2026, Apache 2.0 with no support or warranty guarantee (production support requires negotiating directly).
Should you use it?
Use Giskard if you need adversarial testing and red teaming for LLM agents, you are in a security-conscious or regulated environment, and you are on Python 3.12 or later.
Don’t use it if your Python is pinned below 3.12, you need conventional quality metrics as the primary output, or you are fully air-gapped and cannot resolve the judge-call question.
Bottom line: the best open-source option for adversarially testing LLM agents, and it fills a gap the quality-focused frameworks leave open. Run it alongside Ragas or DeepEval rather than instead of them - they answer different questions. Just confirm your Python version first, because that single line will decide it for a meaningful number of teams.
License, version requirements, architecture and repository activity verified against vendor documentation and GitHub on 31 July 2026. Giskard Hub pricing is not published and has not been estimated. This is a researched directory entry - we have not yet instrumented this framework with our reference application.
Pricing Plans
Open-source library
$0
- Apache 2.0 licensed
- Full testing and red-teaming library
- No support or warranty
- You pay only for judge model calls
Giskard Hub
Custom
- Enterprise platform for LLM agent testing
- Shared workspaces and collaborative annotation
- Role-based access control
- Continuous red teaming
- Pricing not publicly listed
SDKs & Frameworks
Deployment
Eval Methods
Focus
Our Verdict
Giskard occupies a different niche from most of this category and does it well. Where Ragas measures RAG quality and DeepEval gates CI, Giskard attacks your system - automated red teaming and adversarial vulnerability scanning against LLM agents, including black-box systems you did not build. That is a genuinely underserved capability, and it is Apache 2.0. Two things will decide whether you can use it. The v3 rewrite requires Python 3.12 or later, which quietly disqualifies any infrastructure pinned to 3.10 or 3.11 - a constraint we have not seen flagged in a single comparison, and one that will surface after you have already committed. And LLM-as-judge checks may require external API calls, so air-gapped environments need to check carefully what leaves the network. The v3 rewrite is also still maturing, with some v2 features not fully ported. Treat it as a security testing tool that complements a quality-focused framework, not as a replacement for one.
Similar Tools
LM Evaluation Harness
Anyone benchmarking base models, comparing fine-tunes against published baselines, or producing numbers that need to line up with academic literature and the Open LLM Leaderboard.
Patronus AI
Enterprises that want evaluation backed by purpose-trained judge models rather than prompted general LLMs, particularly for hallucination detection and agent debugging, and who can work with enterprise procurement.
Promptfoo
Security and CI teams who want config-driven LLM eval plus serious red-teaming, from an OSS tool with no seat cost
Ragas
Teams building RAG pipelines who want fast, meaningful retrieval and generation metrics during development, and who already have or want a separate tracing platform.
Frequently Asked Questions
What is the Python 3.12 problem?
Giskard v3 requires Python 3.12 or later, and if your infrastructure is pinned to 3.10 or 3.11 the library will not run. This sounds minor and is not. Plenty of production environments are pinned by a platform dependency, a base image, or a data science stack that has not certified newer Python, and upgrading the interpreter is frequently a much larger project than adopting an eval library. We have not seen this flagged in any comparison of eval frameworks, and it is exactly the kind of constraint that surfaces after a team has already decided. Check your Python version before you evaluate anything else about this tool.
How is Giskard different from Ragas or DeepEval?
It attacks your system rather than measuring it. Ragas scores RAG quality, DeepEval gates your CI on quality thresholds, and both are fundamentally about whether the output is good. Giskard's centre of gravity is adversarial - automated red teaming and vulnerability scanning that probes for ways to make your agent misbehave. Those are complementary rather than competing concerns, and most teams that care about both will run one of each. Judged purely as a quality metrics library Giskard is weaker than the alternatives, which is not a criticism so much as a description of what it is for.
Can I use it in an air-gapped environment?
Only carefully, and you need to check. Giskard includes optional telemetry, and its LLM-as-judge checks may require external API calls - the judge has to run somewhere, and if that is a hosted model your prompts leave the network. Neither is unusual, but both matter in restricted environments where the whole point is that data does not leave. The string matching, regex, comparison and semantic similarity checks are less likely to require egress than the LLM-as-judge ones. If you are deploying somewhere sensitive, audit the telemetry setting and route judge calls to a locally hosted model before you rely on it.
Is the v3 rewrite stable?
Maturing rather than mature. The v3 architecture is a modular set of focused packages, each carrying only the dependencies it needs, which is a genuinely better design than a monolithic install. But the rewrite is still in progress and some v2 features are not yet fully ported. If you are starting fresh, v3 is the right target. If you depend on a specific v2 capability, verify it exists in v3 before migrating rather than assuming feature parity. The project is healthy - roughly 5,400 stars and updates through May 2026 - so the gaps are likely to close, but check rather than assume.
What does Giskard Hub cost?
Not published. Hub is the enterprise platform, adding shared workspaces, collaborative annotation workflows, role-based access control and continuous red teaming on top of the open-source library, and pricing requires contacting sales. We are recording it as not published rather than guessing. The open-source library is genuinely complete enough to be useful on its own, so unlike some open-core products you are not being funnelled toward the paid tier to get basic functionality - Hub is about team workflow rather than core capability.
Can it test systems I did not build?
Yes, and this is one of its better properties. Giskard is designed to wrap an LLM, a black-box agent or a multi-step pipeline, which means you can test a system you have no ability to instrument - a vendor's agent, an internal service owned by another team, or a third-party API. Most eval frameworks assume you control the code and can add tracing to it. For security testing specifically, black-box capability is close to essential, since the systems you most want to probe are often the ones you cannot see inside.