WORK WITH BUSINESS CLIENTS?
Sell AI employees under your own brand.
ARIA Evaluator
Overview
Open-source, self-hosted platform for evaluating conversational AI with judge-panel scoring, safety tests, observability, and reports.
ARIA Evaluator is an open-source platform for testing, observing, and governing conversational AI agents. Teams define evaluation scenarios in YAML, connect ARIA to an agent endpoint, and run conversations through supported adapters such as Amazon Connect, Amazon Lex, Azure Bot Service, Microsoft Copilot, OpenAPI/REST, or WebSocket. ARIA records the transcript, scores it with a panel of independent LLM judges across quality, task completion, safety, customer experience, escalation, and vulnerability dimensions, and produces dashboard views plus shareable HTML and JSON reports with scores and judge reasoning. It is designed for engineering, security, product, and governance teams that want self-hosted AI agent evaluation workflows, adversarial testing, and human review when judge results disagree.
Product overview
Capabilities, pricing and fit
ARIA Evaluator is an open-source, self-hosted platform for evaluating conversational agents with scenario-based runs, judge-panel scoring, adversarial safety tests, observability dashboards, and shareable reports.
Information as of November 14, 2026
What it can do
- Panel-based LLM judging
Scores every conversation with several independent AI judges rather than a single evaluator, with disagreements escalated to a human reviewer.[1], [2]
- 15-dimension conversation scoring
Evaluates transcripts across response quality, task completion, safety and security, customer experience, and escalation and vulnerability dimensions.[1], [2]
- Scenario-driven evaluation pipeline
Runs YAML scenarios through an adapter and conversation runner, records an immutable transcript, sends it to LLM judges, and produces an evaluation result and report.[2]
- Adversarial security testing
Supports security scenarios such as prompt injection, jailbreaks, and social engineering to test whether agent guardrails hold under pressure.[1], [2]
- Agent platform adapters
Connects to Amazon Connect, Amazon Lex, Azure Bot Service, Microsoft Copilot, OpenAPI/REST endpoints, and WebSocket-based custom chat bots.[1], [2]
- Real-time observability
Lets teams watch evaluation runs stream live, inspect full conversations turn by turn, and track scores, latency, and cost for judge invocations.[1]
- Shareable reports with reasoning
Generates HTML and JSON reports containing scores, evidence, and judge reasoning, and also displays results live in the dashboard.[2]
- Governance workflow support
Includes human review queues, scheduled regression runs, audit-logged overrides, and release-readiness style reporting for product and security sign-off.[1]
Where it fits best
- Engineering, security, and product teams preparing conversational AI agents for production, especially in regulated or safety-critical settings where verifiable quality and safety scoring is needed.[1], [2]
- Teams evaluating agents on Amazon Connect, Amazon Lex, Azure Bot Service, Microsoft Copilot, or custom HTTP/WebSocket endpoints without adding instrumentation to the agent under test.[1], [2]
- Organizations that prefer self-hosted evaluation workflows, Apache-2.0 open-source software, and control over infrastructure and model-provider credentials.[1], [2]
Pricing and availability
Self-hosted ARIA Evaluator is presented as open source under the Apache-2.0 license with no account or signup required for local use. The terms page also references Free, Individual, Enterprise Starter, Enterprise Pro, and Enterprise Unlimited service tiers, but paid tier prices are not shown in the supplied public material.[1], [3]
Platforms: Web dashboard, CLI, Amazon Connect adapter, Amazon Lex adapter, Azure Bot Service adapter, Microsoft Copilot adapter, OpenAPI / REST adapter, WebSocket adapter[1], [2]
Access: Self-hosted, Local development, Own infrastructure[1], [2]
Important considerations
- Setup is infrastructure-led rather than account-led: teams clone the project, install dependencies, configure their model provider, and define scenarios in YAML.[1], [2]
- Human review remains part of the workflow because judge disagreement is escalated to a person for the final call.[1]
- Public commercial details are limited: the product page emphasizes open-source local use, while the terms page references Free, Individual, Enterprise Starter, Enterprise Pro, and Enterprise Unlimited service tiers without the supplied material showing published prices.[1], [3]
Sources (3)
- ARIA EvaluatorOfficial site · checked 2026-11-14
- ARIA Evaluator DocsDocumentation · checked 2026-11-14
- Terms of Service - ARIA EvaluatorPricing page · checked 2026-11-14
Autonomy level
64/100
Our subjective rating of how independently the product carries out its documented tasks, considering the human setup, input, approval and review it needs. This is not a measured success rate.
Comparisons
Custom Comparisons
Some of the use cases of ARIA Evaluator:
- Testing conversational AI agents before release
- Running regression checks against agent scenarios
- Evaluating prompt-injection, jailbreak, and social-engineering resistance
- Reviewing transcripts, scores, and judge reasoning
- Producing evidence for AI safety, quality, and compliance reviews
- Monitoring agent quality and safety drift over time
Loading Community Opinions...
Pricing model:
Code access:
Popularity level: 14%
Tags:
Turn this profile into a trust signal
Show prospects that ARIA Evaluator has a public place where they can check product details, pricing, ratings, and reviews.
Build confidence
Give buyers a third-party profile to explore.
Reduce hesitation
Put validation beside your strongest CTA.
Earn discovery
Every badge links prospects to your listing.
Choose your style
Preview it, then copy the complete embed code.
Shows buyers where to validate your product, pricing, and reputation.
Plain HTML. No signup, script, or maintenance required.
FOR AGENT BUILDERS
Run your agents in a persistent workspace.
Run Hermes and OpenClaw with saved context, connected tools and delegated coordination.