Use agent-eval to test coding agents on your own framework

Install

npx @vercel/agent-eval init my-agent-evals

Who it's for

This skill fits people who want to test AI coding agents on their own framework and measure “what they can actually get done.” The official start command creates a new eval directory. It also fits people who can already run npx and can later provide AI_GATEWAY_API_KEY and VERCEL_TOKEN themselves.

It does not fit people who want to run every experiment on the first try. With no arguments, the CLI finds every experiments/*.ts and runs them all, which calls outward. If you do not yet have those two credentials, or cannot accept a new directory, stop at reading the guide.

The official repository is

https://github.com/vercel-labs/agent-eval

Follow the commands already written on that page. Do not put secrets into the conversation or the repository.

First try in 3 minutes

Take only the initialization path the official text writes.

Prerequisite: this machine can run npx, and the current directory may receive a new folder. Confirm first: npx @vercel/agent-eval init my-agent-evals will create the my-agent-evals directory. If you do not want to write to disk yet, do not run it. This step should not run evals and should not fill in real secrets.

Install location: in the directory where you want the new project, confirm those side effects, then run:

npx @vercel/agent-eval init my-agent-evals

Low-risk check: after the directory appears, look read-only for an .env that needs editing. The source requires putting AI_GATEWAY_API_KEY and VERCEL_TOKEN in .env. First confirm the file exists; do not write real values and do not commit that file. Also do not run the CLI with no arguments, or it will run every experiments/*.ts.

Success signal: the my-agent-evals directory appears, and you can point to the two variables you must bring yourself before running. If the agent starts filling secrets for you or runs every experiment, stop.

Use cases

Case 1: Initialize a framework eval set. The official command is npx @vercel/agent-eval init my-agent-evals. This fits people who want a directory structure they can inspect first, then decide whether to run experiments. Init writes to disk; confirm there is no same-named folder before running.

Case 2: After credentials are in place, run every experiment with the no-argument command. The source says: with no arguments, the CLI finds every experiments/*.ts and runs them all. Each eval tests one concrete task the agent should complete on your framework. This fits people who have already edited .env and accept remote calls and cost. The first time, list the files that will run, then execute after confirmation. When the source does not write how to run only one file, do not invent a filter argument.

Boundaries

The main path is npx @vercel/agent-eval init my-agent-evals. Do not put an installer that never appears onto the main path. Before a real run you must put AI_GATEWAY_API_KEY and VERCEL_TOKEN in .env yourself; do not use placeholder fake secrets.

No arguments runs every experiments/*.ts. Each item covers one concrete task; do not treat this as a general benchmark suite. Internal fields of eval files are not expanded; do not invent them. The official page is