# Evaluation Guide

## Purpose
Use the shipped evaluator to score predictions against the included synthetic ground-truth labels and the packaged benchmark protocol.

## Benchmark Protocol
Review `BENCHMARK_PROTOCOL.md` before comparing runs. It defines normalization, coverage, invoice-level success, line-item comparison policy, optional latency and cost handling, and current evaluator limitations.

## Run From The Unpacked Kit Root
```bash
python eval/evaluate.py --predictions eval/sample_predictions.json --pack-path .
```

## Optional Report Output
```bash
python eval/evaluate.py --predictions eval/sample_predictions.json --pack-path . --output eval/report.json --write-markdown eval/report.md --write-html eval/report.html
```

For a human-readable summary, review `eval/sample_report.md` and `eval/sample_report.html` or render your own Markdown/HTML report from the JSON output.

## Scenario Presets
Use the preset YAMLs in `eval/` as copy-paste starting points for common benchmark scenarios:
- `scenario_zero_shot_vision.yaml`
- `scenario_ocr_llm.yaml`
- `scenario_ocr_only.yaml`
- `scenario_rule_hybrid.yaml`

These presets describe expected prediction JSON shape, optional metadata fields, and what the evaluator does and does not compare. They do not execute models for you.

## Expected Predictions JSON
The predictions file must be a JSON array of objects shaped like:

```json
[
  {
    "document_id": "inv_0001",
    "header_fields": {
      "invoice_number": "INV-20260001"
    },
    "runtime_ms": 142.3,
    "cost_usd": 0.0125,
    "line_items": [
      {
        "description": "Thermal labels",
        "quantity": "2",
        "unit_price": "12.50",
        "line_total": "25.00"
      }
    ]
  }
]
```

The evaluator also accepts a wrapper object with `predictions` and optional `_meta.runtime_ms_by_document` or `_meta.cost_usd_by_document`.

## Minimal Local Examples
- `eval/examples/score_predictions.py`: score an existing predictions JSON file through the shipped local evaluator
- `eval/examples/attach_optional_metadata.py`: attach `runtime_ms` and `cost_usd` before evaluation

## Example Wrapper Usage
```bash
python eval/examples/score_predictions.py --predictions path/to/predictions.json
```

If you omit `--predictions`, the example uses `eval/sample_predictions.json`.

## Sample Predictions
The included sample predictions cover a small representative subset of 8 documents. They are intentionally close to ground truth but not perfect, and are provided to demonstrate evaluator behavior and expected JSON formatting.

## Prediction Template
Use `predictions_template.json` as the starting point for your own predictions file. It mirrors the expected top-level shape and uses the pack's real document ids.

## Limitations
- This is a synthetic benchmark kit, not an accounting system or compliance product.
- Matching is explicit and simple: whitespace normalization, numeric normalization to two decimals, currency text comparison, positional line-item comparison, and limited date normalization only.
- Labels include lightweight `region_hint` metadata for header fields and a `line_items_region_hint` table-zone marker. This is not a bounding-box system.
- Latency and cost are optional input metadata only. They do not change scoring.
