# mlv-eval: AI Evaluation Harness

> A released Go evaluation harness that scores golden datasets, targets any HTTP endpoint or command, and turns quality thresholds into CI exit codes.

- URL: https://mandalsuraj.com/blog/mlv-eval
- Author: Suraj Mandal (https://mandalsuraj.com)
- Published: 2026-08-29
- Tags: go, ai, evaluation, cli, ci, testing, opensource
- Source: https://github.com/surajmandalcell/mlv-eval
- v0.1.0 release: https://github.com/surajmandalcell/mlv-eval/releases/tag/v0.1.0

![mlv-eval: AI Evaluation Harness cover](https://mandalsuraj.com/images/blog/mlv-eval/cover.png)

`mlv-eval` is a released, open-source Go harness that turns golden datasets into CI quality gates. It treats the system under test as a black box behind an HTTP endpoint or shell command.

The integration stays language-neutral: no SDK inside the target, no hosted dashboard, and no model required to test the harness itself.

## Quickstart

Install the single binary, scaffold one of the bundled evaluation projects, and run it offline:

```bash
go install github.com/surajmandalcell/mlv-eval/cmd/mlv-eval@latest
mlv-eval init image2meta
mlv-eval run
```


![Terminal showing mlv-eval scaffolding the image2meta preset, scoring six cases, and printing GATE PASS](https://mandalsuraj.com/images/blog/mlv-eval/demo.svg?v=2)

  The image2meta scaffold runs against a bundled offline target before it is pointed at a real system.

## Targets

Every case is one JSON object with `input` and `expected` fields. JSONL keeps datasets reviewable in Git, while content-derived case IDs stay stable when lines are reordered or reformatted.

| Target | Contract |
|---|---|
| HTTP | POST one case as JSON and read one JSON object from the response. |
| Command | Write one case to standard input and read one JSON object from standard output. |

HTTP targets receive the case ID in `X-Mlv-Case`. Command targets receive the same value in `MLV_CASE`, making failures easy to correlate with target-side logs or traces.

Runs support bounded concurrency, per-case timeouts, retries, and suite selection. A failed target call scores zero instead of disappearing from the aggregate.

## Scorers

The three presets cover common AI workloads without locking the harness to one model type:


![Three mlv-eval presets: image2meta for labelled bounding boxes, query2rank for ranked identifiers, and text2json for structured extraction](https://mandalsuraj.com/images/blog/mlv-eval/presets.svg?v=2)

  Each preset pairs a target shape with scorers suited to that workload.

| Workload | Preset | Built-in scorers |
|---|---|---|
| Vision | `image2meta` | JSON Schema, field accuracy, and bounding-box IoU. |
| Retrieval | `query2rank` | hit@k and mean reciprocal rank. |
| Extraction | `text2json` | JSON Schema, exact match, field accuracy, and fuzzy text similarity. |

Built-in scorers cover overlap, ranking quality, schema validity, field accuracy, exact matches, and fuzzy text. A custom scorer can be any executable that accepts a case on standard input and returns a scored JSON verdict.

## CI Quality Gates

Thresholds turn aggregate scores into CI outcomes. A run exits `0` when every gate passes, `1` when quality falls below a gate, and `2` when configuration or execution fails.


![mlv-eval quality gate: passing thresholds exit 0, a broken quality gate exits 1, and configuration or runtime errors exit 2](https://mandalsuraj.com/images/blog/mlv-eval/gate.svg?v=2)

  Stable exit codes let the same evaluation run act as a CI quality gate.

| Exit code | Meaning |
|---|---|
| `0` | Every configured quality gate passed. |
| `1` | At least one score fell below its threshold. |
| `2` | Configuration, dataset, target, or runtime failure. |

Use `-json` to capture suite metrics and failed-case reasons for build artifacts or pull-request comments.

## Release

The v0.1.0 offline examples cover 30 cases across all three presets:

| Example | Cases | Result |
|---|---:|---|
| `image2meta` | 6 | Schema 1.000, fields 1.000, IoU 1.000. |
| `query2rank` | 16 | hit@5 0.938, MRR 0.875. |
| `text2json` | 8 | Schema 1.000, fields 0.875, fuzzy 0.875. |