# Benchmark Protocol

This benchmark kit uses bounded exact-match scoring after lightweight normalization. It is designed for repeatable extractor testing, regression checks, and buyer-safe benchmark review.

## Task Definition
- Input: one invoice PDF from this pack.
- Expected output: the pack schema's header fields plus positional line items in JSON.
- The evaluator compares predictions only against shipped ground-truth labels in `labels/`.

## Schema And Matching Summary
- Header fields are matched by field name against the expected label keys.
- Extra predicted header fields are counted separately and reduce invoice-level success.
- Line items are compared positionally in listed order. The evaluator does not reorder, align, or fuzzy-match rows.
- Line-item count mismatches are recorded separately and prevent invoice-level success.

## Field Normalization Rules
- Text fields: whitespace is normalized before comparison.
- Date fields: `YYYY-MM-DD`, `MM/DD/YYYY`, `Mon DD, YYYY`, and `Month DD, YYYY` normalize to ISO date form when parsing succeeds.
- Amount fields: numeric values normalize to two decimal places via decimal parsing.
- Amount tolerance: no extra tolerance band is applied beyond that two-decimal normalization.
- Currency fields: compared as normalized text. Current shipped packs use explicit `USD`.
- Empty strings or omitted expected fields count as not attempted for coverage and do not score as correct.

## Invoice-Level Success
- An invoice counts as success only if every expected header field matches, no extra header fields are present, the predicted line-item count matches the expected count, at least one expected line-item field exists, and line-item accuracy for that invoice is `1.0`.

## Line-Item Comparison Policy
- Each compared line-item field uses the same bounded normalization rules as header fields.
- Only expected line-item fields are scored.
- Extra predicted rows or fields do not increase coverage or accuracy.

## Coverage Definition
- Header-field coverage: proportion of expected header fields that were present with a non-empty predicted value.
- Line-item coverage: proportion of expected line-item fields that were present with a non-empty predicted value in the compared positional rows.
- Overall coverage: combined attempted expected fields across headers and line items, reported separately from accuracy.

## Optional Latency And Cost
- Predictions may include per-document `runtime_ms` and `cost_usd`.
- A wrapper object with `predictions` and optional `_meta.runtime_ms_by_document` or `_meta.cost_usd_by_document` is also accepted.
- Latency and cost are summarized when present and never change scoring.

## Known Limitations
- Synthetic invoices only.
- Lightweight `region_hint` metadata only; no bounding boxes.
- No fuzzy matching, semantic equivalence, semantic row alignment, or OCR confidence scoring.
- No hosted leaderboard, train/dev/test split management, or experiment tracking.

## Common Benchmark Pitfalls
- Do not compare runs without stating whether predictions were normalized upstream or emitted raw.
- High accuracy with low coverage usually means the model skipped fields instead of extracting them.
- This evaluator is intentionally explicit and bounded; do not oversell it as real-world accounting validation.
