{"kind":"post","path":"/blog/mlv-eval","post":{"metadata":{"title":"mlv-eval: AI Evaluation Harness","summary":"A released Go evaluation harness that scores golden datasets, targets any HTTP endpoint or command, and turns quality thresholds into CI exit codes.","publishedAt":"2026-08-29","ascent":"#23C8A5","github":"surajmandalcell/mlv-eval","textCoverPreset":"t2a-teal-dither","links":[{"label":"GitHub","href":"https://github.com/surajmandalcell/mlv-eval","type":"github"},{"label":"v0.1.0 release","href":"https://github.com/surajmandalcell/mlv-eval/releases/tag/v0.1.0","type":"package"}],"tags":["go","ai","evaluation","cli","ci","testing","opensource"],"weight":95},"slug":"mlv-eval","content":"\u003c!-- cover: /images/blog/mlv-eval/cover.png --\u003e\n\n`mlv-eval` is a released, open-source Go harness that turns golden datasets into CI quality gates. It treats the system under test as a black box behind an HTTP endpoint or shell command.\n\nThe integration stays language-neutral: no SDK inside the target, no hosted dashboard, and no model required to test the harness itself.\n\n## Quickstart\n\nInstall the single binary, scaffold one of the bundled evaluation projects, and run it offline:\n\n```bash\ngo install github.com/surajmandalcell/mlv-eval/cmd/mlv-eval@latest\nmlv-eval init image2meta\nmlv-eval run\n```\n\n\u003cfigure className=\"bp-native-media w-full max-w-[620px]\" data-preview-gallery=\"true\"\u003e\n \u003cimg src=\"/images/blog/mlv-eval/demo.svg?v=2\" alt=\"Terminal showing mlv-eval scaffolding the image2meta preset, scoring six cases, and printing GATE PASS\" width=\"700\" height=\"378\" /\u003e\n \u003cfigcaption\u003eThe image2meta scaffold runs against a bundled offline target before it is pointed at a real system.\u003c/figcaption\u003e\n\u003c/figure\u003e\n\n## Targets\n\nEvery case is one JSON object with `input` and `expected` fields. JSONL keeps datasets reviewable in Git, while content-derived case IDs stay stable when lines are reordered or reformatted.\n\n| Target | Contract |\n|---|---|\n| HTTP | POST one case as JSON and read one JSON object from the response. |\n| Command | Write one case to standard input and read one JSON object from standard output. |\n\nHTTP targets receive the case ID in `X-Mlv-Case`. Command targets receive the same value in `MLV_CASE`, making failures easy to correlate with target-side logs or traces.\n\nRuns support bounded concurrency, per-case timeouts, retries, and suite selection. A failed target call scores zero instead of disappearing from the aggregate.\n\n## Scorers\n\nThe three presets cover common AI workloads without locking the harness to one model type:\n\n\u003cfigure className=\"bp-native-hero bp-native-media\" data-preview-gallery=\"true\"\u003e\n \u003cimg src=\"/images/blog/mlv-eval/presets.svg?v=2\" alt=\"Three mlv-eval presets: image2meta for labelled bounding boxes, query2rank for ranked identifiers, and text2json for structured extraction\" width=\"860\" height=\"308\" loading=\"lazy\" /\u003e\n \u003cfigcaption\u003eEach preset pairs a target shape with scorers suited to that workload.\u003c/figcaption\u003e\n\u003c/figure\u003e\n\n| Workload | Preset | Built-in scorers |\n|---|---|---|\n| Vision | `image2meta` | JSON Schema, field accuracy, and bounding-box IoU. |\n| Retrieval | `query2rank` | hit@k and mean reciprocal rank. |\n| Extraction | `text2json` | JSON Schema, exact match, field accuracy, and fuzzy text similarity. |\n\nBuilt-in scorers cover overlap, ranking quality, schema validity, field accuracy, exact matches, and fuzzy text. A custom scorer can be any executable that accepts a case on standard input and returns a scored JSON verdict.\n\n## CI Quality Gates\n\nThresholds turn aggregate scores into CI outcomes. A run exits `0` when every gate passes, `1` when quality falls below a gate, and `2` when configuration or execution fails.\n\n\u003cfigure className=\"bp-native-hero bp-native-media\" data-preview-gallery=\"true\"\u003e\n \u003cimg src=\"/images/blog/mlv-eval/gate.svg?v=2\" alt=\"mlv-eval quality gate: passing thresholds exit 0, a broken quality gate exits 1, and configuration or runtime errors exit 2\" width=\"860\" height=\"300\" loading=\"lazy\" /\u003e\n \u003cfigcaption\u003eStable exit codes let the same evaluation run act as a CI quality gate.\u003c/figcaption\u003e\n\u003c/figure\u003e\n\n| Exit code | Meaning |\n|---|---|\n| `0` | Every configured quality gate passed. |\n| `1` | At least one score fell below its threshold. |\n| `2` | Configuration, dataset, target, or runtime failure. |\n\nUse `-json` to capture suite metrics and failed-case reasons for build artifacts or pull-request comments.\n\n## Release\n\nThe v0.1.0 offline examples cover 30 cases across all three presets:\n\n| Example | Cases | Result |\n|---|---:|---|\n| `image2meta` | 6 | Schema 1.000, fields 1.000, IoU 1.000. |\n| `query2rank` | 16 | hit@5 0.938, MRR 0.875. |\n| `text2json` | 8 | Schema 1.000, fields 0.875, fuzzy 0.875. |"}}