# Invoice Extraction Benchmark Kit — US English / Free Sample

This pack is a ready-to-run invoice extraction benchmark kit for local evaluation, regression checks, and comparison workflows. It includes synthetic US English invoice PDFs, labels, a standalone evaluator, and supporting benchmark docs.

## What This Is
- A synthetic benchmark kit for invoice field extraction and regression testing
- A packaged customer artifact that can be used locally after unzip
- A compact free sample pack to inspect the schema, labels, evaluator, and benchmark structure.

## What This Is Not
- An accounting system
- A compliance product
- A tax engine
- A full OCR platform
- A source of real company or personal data

## Contents
- `pdf/`: rendered invoice PDFs
- `labels/`: per-document ground-truth JSON labels
- `manifest.json`: pack-level summary
- `index.csv`: flat index for downstream tooling
- `DATASET_REPORT.md`: human-readable dataset and QC summary
- `BENCHMARK_CARD.md`: compact buyer-facing summary of scope and coverage
- `PREVIEW.md`: where to start and which documents to inspect first
- `MODEL_COMPARISON_GUIDE.md`: practical instructions for fair model-to-model comparison
- `BENCHMARK_PROTOCOL.md`: packaged scoring and normalization protocol
- `predictions_template.json`: starter template for customer predictions
- `eval/`: evaluator and sample prediction/report files
- `eval/scenario_*.yaml`: scenario presets for common benchmarking setups
- `eval/examples/`: tiny local integration examples
- `EVAL_GUIDE.md`: evaluator usage guide
- `qc/verification_report.json`: packaged verification summary
- `qc/visual_qc_report.json`: advisory visual QA summary
- `qc/visual_flags.csv`: advisory visual QA flag list
- `LICENSE.txt`: tier-specific license summary and restrictions
- `SUPPORT.md`: issuer and support contact details

## Profile
- Language: English
- Currency: USD
- License tier: `free`
- Versioning: official releases use consecutive pack versions such as `1.0`, `1.1`

## Quickstart
1. Open `PREVIEW.md` for the fastest orientation path through the pack.
2. Review `BENCHMARK_CARD.md` for the customer-facing summary of what this benchmark covers.
3. Review `BENCHMARK_PROTOCOL.md` for scoring, normalization, and invoice-level success rules.
4. Review `MODEL_COMPARISON_GUIDE.md` before comparing two model runs.
5. Copy `predictions_template.json` and fill it with your model outputs.
6. If useful, start from one of the `eval/scenario_*.yaml` presets to define your run contract.
7. Run the shipped evaluator from the unpacked kit root:
   `python eval/evaluate.py --predictions your_predictions.json --pack-path .`
8. Inspect the JSON, Markdown, or HTML report for overall accuracy, invoice-level success, coverage, slice summaries, and mismatch examples.

## Run Evaluation From The Unpacked Kit Root
```bash
python eval/evaluate.py --predictions eval/sample_predictions.json --pack-path .
```

## Predictions Format
Predictions must be a JSON array. Each entry should contain:
- `document_id`
- `header_fields`
- `line_items`
- Optional: `runtime_ms`
- Optional: `cost_usd`

## Sample Predictions
`eval/sample_predictions.json` is an example submission covering a small representative subset of 8 documents from the pack labels, with a small number of intentional mistakes so the evaluator produces a non-perfect sample report.

## Scenario Presets
The `eval/scenario_*.yaml` files are compact presets for zero-shot vision, OCR+LLM, OCR-only, and rule-hybrid benchmarking. They document expected prediction shape, optional metadata, and what this evaluator compares.

## Pack Verification
The kit also includes `qc/verification_report.json`, a compact verification summary covering pack composition checks, artifact presence checks, and sample evaluator consistency.
`qc/visual_qc_report.json` and `qc/visual_flags.csv` add advisory visual QA signals for manual spot-checking only.

## Support
- Contact: `info@ocurka.com`
- Issuer: `Filip Barcík`
- Address: Cheb, Lesní 1411/44, 35002, Czech Republic

## Known Limitations
- Synthetic data only
- Single-page invoices only
- English and USD only
- No bounding boxes
- No fuzzy matching or advanced alignment logic
