EvalCore
Records real AI application behavior and replays it offline in CI, helping you find regressions caused by changes before users do.
About EvalCore
EvalCore is an Apache-2.0 open-source AI application evaluation tool: after changing prompts, switching models, or upgrading dependencies, it records real AI behavior and compares it with a baseline to find regressions before users do. Any service that supports HTTP or shell can be an evaluation target, with no SDK and no dependency on a programming language. On the first run, it calls the real model and records requests and responses in a local SQLite cassette; thereafter, each PR replays it offline in CI, without an API key, so identical inputs produce identical conclusions, and with --baseline main it fails only on regressions. The tool is a single dependency-free binary with no telemetry, registration, or server required; the official page lists CI use as $0.
Best for
Best use cases
Check whether AI application behavior has regressed after changing a prompt or model
Set up an offline regression gate for PRs so CI fails only when degradation appears
Build evaluation datasets for services accessed over HTTP or shell
Track token usage and cost for each evaluation run
Features
Behavior recording
On the first run, it calls the real model, records every request and response in a local SQLite cassette, and summarizes token usage and cost.
Offline replay
In CI, it replays the cassette without network access or an API key; identical inputs always produce identical conclusions.
Baseline gate
Compare against the baseline with --baseline main and fail only on regressions; CI uses the exit code as a gate.
Broad compatibility
Supports OpenAI, vLLM, Ollama, any OpenAI-compatible gateway, your own REST API, shell commands, and OTel / OpenInference tracing.
Pros
- Single dependency-free binary usable with any language
- Offline replay makes identical inputs produce identical conclusions, avoiding evaluation flakiness
- Apache-2.0 open source; the official page lists CI use as $0
- No telemetry, registration, or server required