EvalCore buying guide: Does passing a recorded case show the new model would answer the same way?

Three paper evaluation cards separate a passing replay, a missing recording, and evidence from a fresh model response.

Source: github.com
October 24, 2026
2 min

A text classification team wants to know whether it can recheck the same cases under the same conditions after changing its scoring rules. EvalCore is worth comparing for its recording and replay. The v0.7.5 project README separates replay of existing responses from a run that obtains new ones. The replay documentation says that a missing matching recording is reported as a failure in replay-only mode; it does not quietly ask the model. A fixed answer sheet is useful for reviewing the question and scoring rules. It is not an answer the model just produced.

Record replay separately from a new answer

In a fictional paper-craft exhibition, exhibits get labels by shape. Case A says “three straight edges” and expects “triangle.” The last recording did say “triangle.” Case B says “one curved edge and one straight edge”; it was added later and has no recorded answer. Card C asks a separate question: the team changed the model and wants to know if A still gets “triangle.”

Article img
Original paper cards: A has an old answer, B's old-answer slot is blank, and C needs a new answer rather than a copy from A.

Fictional review cards: A's old answer is available for replay, B has no old answer, and C requires a fresh answer obtained separately.

Give the cards to a colleague who did not take part in the evaluation. They should use only what is on the cards: A can show whether “triangle” still receives credit; for B the replay framework lacks a match, not an incorrect model answer; C still lacks evidence and cannot pass because A did. This is a paper exercise; no software or model was run.

A change in model identity can also prevent an old recording from matching. Where cases are fixed and reviewed repeatedly, EvalCore's explicit miss is useful, but keeping a representative recording set takes maintenance and review. To find out how a new model behaves, evaluate its own answers separately. If you're checking a dozen rules by hand, start with a table separating existing answers, scores, and whether the answer is new; another product may not be needed.

Content note:This article is based on publicly available information and is intended as a reference for evaluating tools. Features, pricing, and related policies may change at any time; please refer to the latest official information.

Original source:github.com