Skills

Repro Lens: Reproducibility Checks for ML Agents

When you want to confirm experiments still reproduce after changing ML code, Repro Lens is best suited for static risk scanning, experiment replay, and output comparison; prerequisites are Python 3.11+ and Git locally, static checking needs no ML dependencies or API keys, but experiment replay requires the experiment's own dependencies and must be explicitly triggered.

Official source · Repro Lens

Install

uv tool install 'git+https://github.com/00200200/[email protected]'

Repro Lens is a reproducibility checking tool for ML and coding agents: it scans code for reproducibility risks, replays experiments, and compares outputs before and after changes. Its core use case is confirming that experiment results have not been broken after you modify ML code; static checking requires only Python 3.11+ and Git, with no ML dependencies or API keys, while experiment replay requires the experiment's own dependencies and must be explicitly triggered by you.

Who It's For

  • ML/research engineers who need to verify experiment reproducibility before and after code changes, and teams setting up reproducibility audit workflows for coding agents.
  • Users who only want a static scan: no ML dependencies, no API keys, and the scanned code is never imported or executed; output consists of static findings with file locations.
  • Situations where it is not suitable: without a Git environment, neither installation path is available; on Python below 3.11, the static checker cannot run. Beyond these requirements, the source material does not state any other system prerequisites, and whether platforms such as Windows are supported is not officially stated.
  • Do not assume anything the evidence does not state: it does not declare which text or code languages are supported, nor which agent clients it works in.

First Try in 3 Minutes

Prerequisites: Python 3.11+, Git, and uv installed locally. The recommended main path is to install a pinned version from the official repository using uv:

uv tool install 'git+https://github.com/00200200/[email protected]'

An alternative is to install the same git URL with pip inside a virtual environment (this also requires Git); it is only a secondary option and does not constitute another main entry point.

After installation, start with a low-risk, read-only static check: in your project, give the agent or terminal the task "Use repro-lens check to inspect this file for known reproducibility risks." The corresponding core operation is check: it produces static findings with file locations, and the scanned code is never imported. The success signal is that the output lists findings with file locations (even a conclusion of "no risks found" indicates a successful run).

Note: check being a diagnostic operation only means it does not import the scanned code; whether it reads or writes files, executes commands, or accesses the network is not stated in the source material and cannot be confirmed. Experiment replay (verify, which requires the declared outputs of two runs to match) actually executes the experiment and is a side-effecting operation—do not use it for your first try; if you do use it, first confirm all experiment dependencies are in place and explicitly confirm before triggering.

Use Cases

Case 1: Static audit before modifying a training script. Ask the agent to "run repro-lens check on the training script." The tool executes the check operation, returns static findings with file locations, and does not import your code. You can use this to judge whether the change introduces known reproducibility risks; if details beyond the source material (such as a specific list of rules) are not provided, the only thing you can confirm is that you "obtained findings with locations."

Case 2: Output comparison after a fix. Prerequisite: you have already saved a report via the official workflow before making changes. Ask the agent to "replay the experiment with repro-lens verify, then compare the two saved reports with compare." verify requires the declared outputs of the two runs to match exactly, and compare diffs saved reports. Expected results: a passing verify means the output matches the declaration, and compare reports the differences between the reports. Replay requires the experiment's own dependencies and is an explicit command—the tool will never replay automatically on your behalf.

Usage Boundaries

  • File and command access: check does not import the scanned code; verify executes the experiment (requires dependencies), and replay is always an explicit command. The pre-commit hook (repo repro-lens, rev v0.3.0, id repro-lens) only screens code and project policies, and does not replay automatically. Whether network access is involved is not officially stated.
  • Confirmation point: any operation that executes an experiment (verify) should be preceded by confirmation of the experiment configuration and side effects.
  • Installation differences: installing the pinned version v0.3.0 with uv is recommended; pip installation in a virtual environment is the officially provided alternative. The source material does not mention marketplace or other platform installation methods—do not assume they exist.
  • Agent integration: skills/reproducibility/SKILL.md in the repository can guide an existing coding agent through audits or authorized fixes; its launcher and the CLI share the same engine. The full repository is also a Codex plugin (containing .codex-plugin/plugin.json). Beyond this, the officially supported list of agent clients is not specified.
  • Key unconfirmed information: network behavior, the range of supported code languages, and the read/write boundaries of check/compare all lack official statements—defer to the repository documentation before use.