Soup CLI
All-in-one CLI for model post-training, evaluation, and release gates
About Soup CLI
Soup CLI brings the post-training workflow into one command-line tool: it checks data before training, selects methods by rule and generates configuration, derives evaluations from owned data, gates every save, and self-corrects reward hacking during runs. It supports 23 methods, 163 recipes, and 17 quantization formats; a frozen base model can be streamed layer by layer from CPU memory or NVMe into GPU memory and quantized to 4-bit. soup eval generates SHA-pinned evaluation suites, while soup ship rejects models whose task score improves but general knowledge is harmed.
Best for
Best use cases
Pre-training data checks and configuration generation
Fine-tuning models on low-memory devices
Building offline evaluations from owned data
Validating general knowledge before release
Features
Data precheck and config generation
Checks data before training, selects methods by rule, and generates training configuration.
Layer streaming and quantization
Streams the frozen base model one decoder layer at a time from CPU memory or NVMe, with 4-bit quantization.
Owned-data evaluation derivation
Derives SHA-pinned evaluation suites from owned data and gates every save.
Model release gate
soup ship scores 8 offline suites and rejects models whose task score improves while general knowledge is harmed.
Pros
- Free and Apache-2.0 licensed
- Runs offline with no signup or credit card
- Covers many methods, recipes, and quantization formats
- Release-gate evidence can be submitted
Cons
- The layer-streaming reference implementation reads layers about 1.52 times
- No measured tok/s data is available for preference loss
Pricing Plans
Free
- Apache-2.0 license
- Offline operation
- No signup
- No credit card
* Pricing information is for reference only. Please refer to the latest announcement on Soup CLI's official website.