Choosing AI21 Maestro: Can it abstain when the evidence is missing?
Test dynamic verification against fixed retrieval on twelve synthetic questions, and judge evidence gaps, delay and review effort.
If your knowledge base retrieves several contradictory files, AI21 Maestro’s dynamic planning and rule-based validation are worth testing. AI21 describes configurable output requirements and compute budgets, along with an execution graph and validation report. Those records show a reviewer where to look; they do not establish that the evidence supports the answer. If most requests are clear lookups in one current handbook, fixed retrieval with citations is simpler to maintain.
Include requests that should get no answer
Build a blind comparison using fictional warehouse rules. The fixed workflow retrieves, answers and cites from the same files Maestro receives. Require each key claim to cite its document version and passage, and ask for an explicit abstention when the evidence is missing or the conflict cannot be resolved. Use twelve questions: four with one current source, four returning old and current versions, and four deliberately lacking the needed record. In the synthetic example, the old rule permits four rolls of spare labels and the current one permits two. A request about a special training day with no governing record should not inherit the ordinary limit. Keep the same files and timing method for both routes. Do not connect production knowledge, grant action permissions or permit writes.

Figure 1. Cite v2 for an ordinary day. For the special day, explain which rule is missing rather than borrowing the ordinary limit. Editorial illustration, not a product result.
Count usable answers, not report scores
Have reviewers use an answer key written before the test. Accept a correct answer only with a specific passage; an appropriate abstention also passes if it identifies the missing record. Record each group separately, and mark stale-policy answers and unsupported guesses as distinct failures. Measure response wait, compute usage and manual review time. AI21 published budget and accuracy experiments; those results do not predict performance on your archive. A larger budget earns its place only if fewer errors and less review outweigh the extra wait and cost.
Set an explicit bar: no confidently wrong answers in the conflict or evidence-gap groups, and ordinary queries must meet the team's response target. Fill document gaps first if missing records are the main problem. If fixed retrieval already follows versions, abstains and leaves a usable record, keep it. Discuss integration and quotes only if difficult cases improve and reviewers can retrace the result from citations and the execution record.
Content note:This article is based on publicly available information and is intended as a reference for evaluating tools. Features, pricing, and related policies may change at any time; please refer to the latest official information.
Original source:www.ai21.com