# report/

`summary.json`, `tables.md` and `chart.svg` are generated by `grb report` from `results/`; do not edit them.
This README is written by hand: `grb report` does not touch it.

## summary.json (schema version 1)

Strict JSON (no NaN), ids and numbers only, keys without spaces: the rules of the result files apply.

Version rule: `schema_version` stays 1 as long as a change only adds fields or entries and leaves every existing key
with its meaning, type and unit. Changing or removing a key, or changing its meaning, type or unit, makes it schema 2.

Which result is used: of all result files with the same `kind` and `name`, the newest counts, by `created_utc` and then
by `run_id`. The files it supersedes stay in `results/` (result files are never overwritten). These are all of them:

| kind | name | superseded | counts instead |
|---|---|---|---|
| `costs` | `round1` | `costs-round1-20261011T105830Z-61f9d4` | `costs-round1-20261011T164212Z-95734e` |
| `first_stage` | `hybrid-71k` | `first_stage-hybrid-71k-20261009T233420Z-a2085e` | `first_stage-hybrid-71k-20261010T000729Z-108e4c` |
| `first_stage` | `keyword-71k` | `first_stage-keyword-71k-20261009T233413Z-fbd2cc` | `first_stage-keyword-71k-20261010T000721Z-52e3c2` |
| `first_stage` | `keyword-plain-71k` | `first_stage-keyword-plain-71k-20261009T233424Z-1e3c1f` | `first_stage-keyword-plain-71k-20261010T000733Z-b317ea` |
| `first_stage` | `vector-71k` | `first_stage-vector-71k-20261009T233417Z-a81640` | `first_stage-vector-71k-20261010T000725Z-1fc254` |
| `judgments` | `judged-sample` | `judgments-judged-sample-20261010T172706Z-d8a837` | `judgments-judged-sample-20261010T182100Z-ca8dbb` |
| `quality` | `summary-hybrid-71k` | `quality-summary-hybrid-71k-20261010T140103Z-499aff`, `quality-summary-hybrid-71k-20261010T182051Z-40e1f5` | `quality-summary-hybrid-71k-20261012T080507Z-95eada` |
| `quality` | `summary-keyword-71k` | `quality-summary-keyword-71k-20261010T140103Z-971b7f`, `quality-summary-keyword-71k-20261010T182050Z-0c262d` | `quality-summary-keyword-71k-20261012T080507Z-ab7e07` |
| `quality` | `summary-keyword-full` | `quality-summary-keyword-full-20261010T140104Z-5bfc00`, `quality-summary-keyword-full-20261010T182051Z-37dc14` | `quality-summary-keyword-full-20261012T080508Z-368030` |
| `quality` | `summary-vector-71k` | `quality-summary-vector-71k-20261010T140103Z-31553a`, `quality-summary-vector-71k-20261010T182051Z-cf6364` | `quality-summary-vector-71k-20261012T080507Z-e8fbcc` |
| `validation` | `mxbai-large-v2` | `validation-mxbai-large-v2-20261010T033738Z-cd4a53` | `validation-mxbai-large-v2-20261010T040457Z-eac0e2` |

The four first stages were built on 2026-10-09 and again on 2026-10-10. The four `quality-summary-*` files and the
judged sample were written again on 2026-10-10, with the same numbers; the four summaries once more on 2026-10-12,
with the corrected contamination flag of bge-v2-m3 and the same numbers (DEVIATIONS.md). The first equality check of mxbai-large-v2 was not
rated, the second passed. The costs were computed again on 2026-10-11, with the break-even against both Voyage models,
the saving by Recall and the peak price of DeepSeek added; the numbers of the first computation are unchanged. A
`run_id` is the name of its file; the folder is the UTC date of `created_utc`.

Top-level keys:

| key | content |
|---|---|
| `schema_version` | 1 |
| `measured_from`, `measured` | UTC dates of the oldest and the newest result file of the kinds the report reads (`first_stage`, `quality`, `latency`, `validation`, `costs`, `judgments`, `pitfall`), superseded files included, the quality summaries not (they evaluate measured runs): the period of the round |
| `repo_commit`, `repo_dirty` | the commit the report was computed at, and whether the tree had changes besides `results/`, `runs/` and `report/` |
| `plan` | `commit` that added PLAN.md, `timestamp_file` (its OpenTimestamps proof) |
| `bm25_reference` | BM25 values of the MIRACL paper (not run) |
| `hardware` | `gpu`: the GPU of the GPU latency results |
| `models` | per model id: registry data, `contamination`, `rated`, `validation` (the equality check in short) |
| `first_stage` | per first stage: `metrics`, `counts`, `latency` (search), `query_embedding`, `contamination`, `exploratory` |
| `quality` | per rerank stage: `models`, `best`, `winner`, `not_measured` |
| `latency` | reranker latency per device group (below) |
| `cpu_indicative` | desktop CPU pre-measurement: `conditions` and `models`; indicative only |
| `checks` | `determinism`, `dtype`, `pointwise`, `evaluation`: the latest check results |
| `costs` | `costs`, `context`, `checks` and `notes` of the costs result |
| `judgments` | `metrics`, `checks` and `notes` of the judged sample |
| `pitfall` | `metrics`, `checks`, `model` and `stage` of the pitfalls experiment |

Composite keys:

- `latency`: `cuda:desktop-se` (the GPU) and `api:desktop-se:<time of day>` (Voyage; morning, afternoon, evening), then
  model id, then `k@<k>` with `p50_ms`, `p95_ms`, `mean_ms` and `n`. Besides the `k@<k>` keys an entry holds the fields
  described below; a reader that wants the timings takes the keys that start with `k@`.
- `quality.<stage>.models.<model>`: `depth@20`, `depth@50` and `depth@100` with the metrics, and `comparison`.
- `costs.costs.api`: model id, then stage, then `k@<k>`.
- `costs.costs.break_even`: `<model>-cuda-desktop-se-k@<k>`; every entry also holds `model` and `k`, so nobody has to
  take the key apart.
- `costs.context`: `<stage>--<model>`.
- `judgments.metrics`: `<stage>:<model>` for the sampled groups, `control-relevant` and `control-irrelevant`.

`cuda` in `summary.json` is called `gpu` in `chart.svg` (its classes `grb-gpu` and `grb-api`).

## Where a value comes from

`run_id` and `created_utc` name the result file whose values stand beside them. They are in `quality.<stage>` (the
stage summary: `best`, `winner`, `not_measured` and the `comparison` of every model), in
`quality.<stage>.models.<model>` (the `depth@<k>` numbers), in `first_stage.<stage>` (`metrics`, `counts`,
`contamination`), in `latency.<group>.<model>`, `cpu_indicative.models.<model>`, `costs`, `judgments` and `pitfall`. In
`first_stage.<stage>` the `latency` and `query_embedding` come from the latency result of that stage:
`latency_run_id` and `latency_created_utc` name it. The entries of `checks` and `models.<id>.validation` carry a
`run_id` only.

The Voyage entries of `latency` hold in addition:

| field | content |
|---|---|
| `finished_local` | the local time at which the result file was written, at the end of the run: ISO 8601 with UTC offset, in Europe/Stockholm, for example `2026-10-11T12:05:17+02:00`; the run began a few minutes earlier |
| `weekday` | the day of the week of `finished_local` (English name) |
| `tcp_connect_ms` | median TCP connect time to the address of the API host, in ms: the baseline of the run |
| `training_opt_out` | status of the opt-out from model training of the Voyage account when the run was made: `unconfirmed`, `confirmed` or `not set` |
| `training_opt_out_checked` | the date (YYYY-MM-DD) on which that status was set |

The result file records one time stamp, `created_utc`, which is written with the file at the end of a run; it records
no start. `finished_local` is that time stamp in local time, and the run began a few minutes earlier. Time of day, in
local time (Europe/Stockholm) at the start of a run: before 12:00 morning, 12:00 to 18:00 afternoon, from 18:00
evening. The three times of day were measured on a Saturday (evening) and a Sunday (morning and afternoon), by the
weekday of `finished_local`; no weekday was measured. The opt-out was `unconfirmed` until the afternoon run and
`confirmed` at that run.

## Units and ranges

- Keys ending in `_ms` are milliseconds. `capacity_*` and `break_even_*` are queries per month. `tokens_per_query*` are
  the tokens that Voyage billed per query; `tokenizer_deviation_pct` is a percentage.
- Metrics (`ndcg@10`, `hit@5`, `recall@20`, `rate`, ...) are fractions between 0 and 1. A key with `points` in its name
  (`delta_vs_first_stage_points`, `difference_points`, `tolerance_points`, ...) is a difference in points: nDCG × 100,
  so 0.01 of nDCG is 1 point. `ci_low` and `ci_high` are the 95 % bootstrap interval; `p_*` are p-values of the paired
  randomisation tests (`..._holm`: Holm-adjusted).
- Money is in EUR (`eur_per_month`, `eur_per_1000`: EUR per 1,000 queries, `llm_eur_saved_per_1000*`) or in USD
  (`usd_per_eur`, `usd_per_million_tokens` and `usd_per_million_input*` in `costs.costs.prices`). The monthly price
  of the server (`eur_per_month`) is 212.30 EUR plus the extra monthly fee of 1.70 EUR, without the one-off setup fee.
- `wording_vs_first_stage`, `wording_best_vs_model` and `wording_vs_correct` are one of `clearly better`,
  `measurably better`, `on par`, `measurably worse`, `clearly worse` (the wording rule of PLAN.md, applied to worse
  differences as well: DEVIATIONS.md, 2026-10-10).
- `status_chat` and `status_live`, also with the suffix `_vs_rerank-3-lite`, are `reachable` or `not reachable`. A
  `status` alone is `ok` or `unusable` in `models.<id>.validation` (`unusable`: the deciding reference could not run),
  `not measurable at depth 100` in a `costs.context` entry without a value and `not measurable in 30 GB` in a
  `cpu_indicative` entry without a value.
- `models.<id>.hf_id`, `revision` and `params_m`: a local model has its Hugging Face id, the 40-character revision of
  its snapshot and its parameters in millions. The two API models (Voyage) hold placeholders: `hf_id` is the empty
  string and `params_m` is 0 (not applicable). Their `revision` is the date of the measurement (YYYY-MM-DD; PLAN.md:
  "API, measurement date"): the Voyage quality scores were requested on 2026-10-10.

## Costs

- `costs.costs.fx`: the exchange rate (`usd_per_eur`) with its `date`, the `weekday` of that date, its `source` and a
  `note` that says why it is the rate of that day: the last ECB reference rate before the measurement days, because the
  ECB publishes no rate on weekends and the measurement days were a Saturday and a Sunday. `costs.costs.usd_per_eur`
  and `costs.costs.prices.fx` hold the same rate.
- `costs.costs.break_even.<key>`: `server`, `model`, `k`, `eur_per_month`, `capacity_full`, `capacity_50` (at 50 %
  load), `approximation` (true: the GPU is the desktop's, not the server's) and, against each Voyage model, the
  break-even and a chat and a live status: `break_even_vs_rerank-3` with `status_chat` and `status_live`, and
  `break_even_vs_rerank-3-lite` with `status_chat_vs_rerank-3-lite` and `status_live_vs_rerank-3-lite`. The statuses
  without a suffix are against `rerank-3`.
- `costs.context.<stage>--<model>`: `hit@5_reranked` and `recall@5_reranked`, the smallest first-stage depth that
  reaches the value (`k_needed_hit`, `k_needed_recall`), the passages saved (`saved_passages_hit`,
  `saved_passages_recall`: depth minus 5) and the saving per 1,000 queries by LLM id: `llm_eur_saved_per_1000` (by
  Hit) and `llm_eur_saved_per_1000_recall` at the input price of the LLM (DeepSeek: its off-peak price),
  `llm_eur_saved_per_1000_peak` and `llm_eur_saved_per_1000_recall_peak` at the peak price, for the LLMs that have one
  (DeepSeek).

## cpu_indicative

`conditions` holds `cpu`, `threads`, `n_queries`, `warmup_calls` and `passes` of the desktop CPU results, and
`memory_cap` with `gb` and `source`. The cap of the container (30 GB) is in no result file: it stands in DEVIATIONS.md
(2026-10-11), as `source` says. `models.<id>` has `k@20` and `k@100` (a missing one holds `status`), `loadavg_before`,
`run_id` and `created_utc`. qwen3-0.6b ran out of memory at k = 100 under the cap; its k = 20 value was measured again
(DEVIATIONS.md, 2026-10-11).

## Optional keys

- `quality.<stage>.models.<id>.comparison`: `delta_best_minus_model_points`, `p_vs_best_holm` and
  `wording_best_vs_model` are missing for the best model of the stage.
- `quality.<stage>.models.<id>.depth@100` is missing for jina-v3.5 on keyword-full (not measurable on 16 GB).
- `pitfall.metrics.qwen3.correct` has no `delta_vs_correct_points`, `p_vs_correct_holm` and `wording_vs_correct`: it is
  the reference of the other variants.
- `costs.costs.api.<model>.<stage>.k@<k>`: `tokenizer_estimate`, `tokenizer_deviation_pct`, `n_results` and
  `tokens_per_query_min` and `_max` only for the stage of the latency bundle (hybrid-71k); `n_queries` only for the
  other stages, whose tokens come from the quality runs.
- `costs.context.<key>` has only `status` where there is no depth-100 run (jina-v3.5 on keyword-full). The `_peak`
  dictionaries name only the LLMs that have a peak price.
- `latency.<group>.<model>`: `k@50` for the GPU entries only (Voyage was measured at k = 20 and k = 100); the Voyage
  fields above for the API entries only.

Where a value is `null`:

- `repo_commit`, `repo_dirty` and `plan.commit` without git;
- `models.<id>.validation` for the API models (no equality check);
- `models.<id>.validation.reason` for a check that passed (a check that did not pass holds the reason as text), and
  `status`, `mean_spearman` and `min_spearman` there if the deciding reference did not run;
- `first_stage.<stage>.latency`, `latency_run_id` and `latency_created_utc` for a stage that was not timed
  (`keyword-71k-ltr`), and `query_embedding` for a stage without vector search;
- `quality.<stage>.winner` if no model is clearly better than all others;
- `quality.<stage>.models.<id>.comparison` for a model without a depth-100 result (jina-v3.5 on keyword-full);
- `checks.determinism.<id>.batch_size` and `checks.dtype.<id>.batch_size_gpu` for jina-v3.5 (it forms its own blocks).

No top-level block is `null`: `grb report` stops if the costs, judgments, pitfall, checks or desktop CPU results
are missing.

## chart.svg

Quality (nDCG@10) against reranker latency (p50, 100 candidates) on the first stage of the latency bundle. The latency
axis is logarithmic, labelled in ms below a second and in s from a second on. The API points are those of the
afternoon run, which the `<desc>` says. Classes: `grb-point` with `grb-gpu` or `grb-api` per data point, `grb-label`
per label, `grb-legend` for the legend markers. Colours only through `currentColor`. A dagger after a model name marks
MIRACL among its training data (MTEB registry); the `<desc>` says so.