# Measurement plan: German rerankers on MIRACL-de (round 1)

This plan was fixed before any measurement. It is the first commit of this repository and is timestamped with
OpenTimestamps (`PLAN.md.ots`). Later deviations are logged in `DEVIATIONS.md`; every change of that file is
timestamped as well. Where a value can only be found by measuring, this plan fixes the rule and the result file
records the value.

## Question
Which rerankers improve German question-answering retrieval (knowledge base, RAG), at what latency and cost, on a
GPU, on a CPU server and via API?

## Data
- MIRACL German dev: 305 queries, 3,144 judgments, 811 relevant. `miracl/miracl@5be20db9509754dadad47689368639fcec739c00`.
- Corpus "71k": `mteb/MIRACLRetrievalHardNegatives@332a9acb49f5e83d5397683f79d23e588f685916`, subset de,
  71,277 passages (pooled top 250 of BM25, mE5-large and e5-mistral-instruct; all 811 relevant passages included).
- Corpus "full": `miracl/miracl-corpus@d921ec7e349ce0d28daf30b2da9da5ee698bef0d`, de, 15,866,222 passages. Keyword
  stage only.
- `mteb/MIRACLReranking@ab6f54eff185a84bc1f6ab96b56bc7df87433228` (de, 304 queries x 100 BM25 candidates, mapped back
  to MIRACL ids via `miracl/mmteb-miracl-reranking@6d1962c527217f8927fca80f890f14f36b2802af` `data/bm25/de.gz`):
  input for adapter equality tests only, not a quality reference.
- Passage text is `title + "\n" + text` everywhere; equality tests use the MIRACLReranking text as provided.
- Files are checked against `datasets.lock` (SHA-256; row counts were checked when the lock was written) at the start
  of every measuring command.

## First stages (Typesense 30.2, `typesense/typesense:30.2@sha256:610f2d34b1f93d00762869da2c67736775e5798d19a2c8b91b014b8a0cc1e110`)
| stage | corpus | method |
|---|---|---|
| keyword-71k | 71k | Typesense text search, German settings |
| vector-71k | 71k | Arctic embeddings, Typesense vector search |
| hybrid-71k | 71k | Typesense rank fusion, alpha 0.3 |
| keyword-full | full | Typesense text search, German settings |
| keyword-plain-71k | 71k | Typesense without locale, stemming and stopwords, default search parameters (reference only, no rerankers) |

German settings: fields `title` and `text` with `locale: "de"` and `stem: true`; stopword set = Snowball German stop
list (BSD) plus the Snowball German stem of every entry (Typesense stems the query before it removes stopwords;
270 entries, SHA-256 recorded with every result);
`query_by=title,text`, `prefix=false`, `drop_tokens_threshold=100`, `num_typos=2`, `typo_tokens_threshold=1`,
`per_page=100`; documents imported in ascending passage-id order (score ties are broken by insertion order).
Embeddings: `Snowflake/snowflake-arctic-embed-l-v2.0@ac6544c8a46e00af67e330e85a9028c66b8cfd9a`, queries with prompt
`query: `, L2-normalised, 1024 dimensions, computed outside Typesense on the GPU. Vector search: HNSW; `ef` is raised
until the top-100 overlap with exact search is at least 99 % (rule).
Nothing is tuned for relevance on the test queries; `ef` is chosen by overlap with exact search, which uses no
judgments. BM25 reference (not run): MIRACL paper, full corpus, nDCG@10 0.226,
Recall@100 0.572.
Reported per stage: Recall@100, ceiling (nDCG@10 of a perfect reordering of the top 100), first-stage latency.

## Rerankers
| id | model | revision | implementation | max input |
|---|---|---|---|---|
| gte-base | Alibaba-NLP/gte-multilingual-reranker-base | 8215cf04918ba6f7b6a62bb44238ce2953d8831c (code Alibaba-NLP/new-impl@40ced75c3017eb27626c9d4ea981bde21a2662f4) | sentence-transformers CrossEncoder, non-persistent buffers restored after loading | 512 (model card) |
| bge-v2-m3 | BAAI/bge-reranker-v2-m3 | 953dc6f6f85a1b2dbfca4c34a2796e7dde08d41e | sentence-transformers CrossEncoder | 512 (model card) |
| jina-v3.5 | jinaai/jina-reranker-v3.5 | e8a93f33f0b22108f8c2364f8484ce3422552fbc | `model.rerank` (listwise), loaded from the pinned snapshot | fixed by vendor code (query 1024, document 8192) |
| qwen3-0.6b | Qwen/Qwen3-Reranker-0.6B | e61197ed45024b0ed8a2d74b80b4d909f1255473 | sentence-transformers CrossEncoder, raw logit score | vendor default |
| mxbai-large-v2 | mixedbread-ai/mxbai-rerank-large-v2 | ca7e1ee484c37c0ddd8d178a9a5c33cec575c5e6 | sentence-transformers CrossEncoder, raw logit score | vendor default |
| qwen3-4b | Qwen/Qwen3-Reranker-4B | 22e683669bc0f0bd69640a1354a6d0aebcfeede5 | sentence-transformers CrossEncoder, raw logit score | vendor default |
| querit-4b | Querit/Querit-4B | e557bb5ce6555f16b33dbb7d2ca62e8d27c14c13 | input format of mteb 2.24.3 `QueritWrapper`, model class loaded explicitly | 1024 (the model pads every input to the maximum) |
| voyage-rerank-3 | Voyage API `rerank-3` | API, measurement date | HTTP API | 32k |
| voyage-rerank-3-lite | Voyage API `rerank-3-lite` | API, measurement date | HTTP API | 32k |

Rules: per local model the longest observed input (tokens of the real model input) and the number of truncated pairs
are recorded; dtype per vendor recommendation (recorded); Qwen3 uses the model-card instruction "Given a web search
query, retrieve relevant passages that answer the query"; jina receives the candidates in first-stage order; score
ties are broken by first-stage rank; Voyage is called with `truncation: true` and `return_documents: false`. All
local models run with transformers 5.5.4 and sentence-transformers 6.1.0.
Contamination flags (MTEB model registry, `training_datasets`, read 2026-10-09): Arctic-l-v2.0: MIRACLRetrieval,
MIRACLRetrievalHardNegatives, MIRACLReranking; Qwen3-Reranker 0.6B and 4B: MIRACLReranking; Querit-4B:
MIRACLReranking; bge and jina: unknown; mxbai: no MIRACL; gte: not listed. Flagged models are marked in every table;
a flagged model is never called a winner without this note.

## Adapter checks (before a model is scored)
1. Equality: on the first 20 MIRACLReranking queries (100 fixed candidates each) the adapter's scores match one
   deciding reference per model with Spearman >= 0.99 per query. The deciding reference does not share the adapter's
   code path: bge-v2-m3 and Qwen3 the model-card code in plain transformers; mxbai the vendor package
   `mxbai-rerank==0.1.6` from the model card; gte, jina and Querit (remote code) the model-card code under
   transformers 4.57, which catches transformers-5 initialisation bugs (Querit has no model-card code: input format of
   mteb's `QueritWrapper`). Further references (mteb wrappers) are reported, not deciding. If the deciding reference
   cannot run, the model is not rated.
2. Evaluation: our nDCG@10 equals pytrec_eval `ndcg_cut_10` within 0.0001 on a real run.
3. Voyage scores count as pointwise only if the same candidate gets the same score (within 0.001) at k = 20 and
   k = 100; otherwise every depth is requested separately.
A model failing check 1 is reported as "not rated" with the reason.

## Quality
Runs: every reranker on keyword-71k, vector-71k, hybrid-71k and keyword-full, depth 100. Pointwise depths 20 and 50
are derived from the same scores (only the first k candidates are reordered); jina is run separately per depth.
Metrics: nDCG@10 (main), nDCG'@10 on condensed lists (unjudged passages removed), Hit@5, Hit@10, Recall@5, Recall@10,
Recall@20, MRR@10, judged@10.
Uncertainty: 95 % bootstrap intervals (10,000 resamples over queries, seed 20261009) for nDCG@10 and Hit@5.
Significance: paired randomisation test (10,000 sign flips, seed 20261009) against the first stage and against the
best model of the stage; Holm correction per stage and comparison family (all models against the first stage; all
models against the best model); alpha 0.05.
Wording rule: "clearly better" = significant after Holm and at least 2.0 nDCG points; "measurably better" =
significant but less than 2.0 points; otherwise "on par". A "winner" exists only if a model is clearly better than all
others. Rationale for 2 points: smaller differences are rarely noticeable in practice. The threshold is not changed
after seeing results.
Hit rates are lower bounds because unjudged relevant passages count as misses. Numbers from different corpora are
never compared directly.
Determinism: bge-v2-m3, qwen3-0.6b and jina-v3.5 are run a second time on hybrid-71k; nDCG@10 must agree within
0.1 points.
Data type check: gte, bge, jina and qwen3-0.6b on 20 queries, desktop CPU fp32 against GPU; if nDCG@10 differs by
less than 0.5 points, the GPU value is used for CPU points.

## Judged sample of unjudged passages
Hybrid-71k: the three best rated models by nDCG@10, 15 random unjudged passages each from ranks 1 to 3; keyword-full:
15 unjudged passages from ranks 1 to 3 of the best rated model there; 15 judged control pairs (8 relevant, 7 not
relevant). Duplicates removed, order shuffled, model names hidden. Judge: an LLM, Anthropic Claude Opus 5.5
(`claude-opus-5-5`), in a fresh session that sees only the shuffled query and passage texts (no ids, model names,
ranks, stages or existing judgments) and labels each pair "the passage answers the question" (1) or not (0)
(MIRACL scheme), or "unclear" (excluded from the rates, counted separately). Output: estimated rate of false
negatives per group with a Wilson 95 % interval and the agreement with the MIRACL judgments on the control pairs. If
fewer than 12 of the 15 control pairs agree, the rates are published only with this warning. Caveat stated with the
result: an LLM judge may favour the same passages as LLM-based rerankers; the control pairs measure agreement with
the human MIRACL annotators, not this bias. A later human judgment of the same pairs is reported as agreement with
this one; the LLM judgment stays the published one. Published: ids, labels and the judge's model id only. Context for
the main metric; it never changes the judgments behind it.

## Pitfalls experiment
Qwen3-Reranker-0.6B on hybrid-71k with a plain transformers implementation: (a) correct per model card; (b) empty
instruction; (c) right padding; (d) maximum length 256 with correct truncation (passage only) and with wrong
truncation (end of the whole input cut, yes/no suffix lost). Typesense: keyword-plain-71k against keyword-71k.

## Latency
All timings are taken inside the benchmark container (host network).
Unit: one query with k candidates, from texts in to scores out, tokenisation included, model already loaded; all k
candidates in one forward pass if memory allows, otherwise in chunks whose size is recorded; GPU timing uses
`torch.cuda.synchronize()`; before each k, 20 warm-up calls (the first 20 queries) are discarded, then every query is
measured; p50 and p95.
GPU (RTX 5070 Ti, desktop idle, `nvidia-smi` logging clocks, temperature and power): all seven local models, 305
queries, 3 passes, k = 20, 50, 100.
CPU pre-measurement (desktop Ryzen 9 5950X, `taskset` to 8 threads): not published.
CPU, published (Hetzner CCX33, 8 dedicated vCPU, FSN1, NBG1 or HEL1): gte-base, bge-v2-m3, jina-v3.5 and qwen3-0.6b with
PyTorch fp32 and bge-v2-m3 with OpenVINO int8; fixed sample of 100 queries (seed 20261009), k = 20 and 100, 1 pass;
qwen3-4b: 5 queries, k = 20, 1 warm-up call. `lscpu` and torch threads recorded. int8 scores on the server must match the desktop
(Spearman >= 0.99 on 20 queries).
API (Voyage): from the desktop in Sweden over a wired connection, 305 queries, k = 20 and 100, at three times of day;
full HTTP time; TCP connect time to the endpoint recorded as baseline; billed tokens recorded per k without warm-up;
the account's training opt-out status is recorded with every Voyage result.
First stage: Typesense query time on the desktop for every stage.

## OpenVINO int8 variant
bge-reranker-v2-m3, static int8 quantisation with OpenVINO, calibrated on 300 (query, passage) pairs from MIRACL test-B
queries (no judgments, disjoint from dev) and random full-corpus passages outside the 71k corpus. Quality measured on
the desktop (305 queries, hybrid-71k, k = 100) against the fp32 model.

## Costs
API: billed tokens times list price on the measurement day, USD to EUR at the ECB reference rate of that day.
Servers: fixed monthly price; Hetzner CCX33 138.49 EUR/month plus IPv4; Hetzner GEX45 (RTX PRO 4000 Blackwell SFF,
24 GB) 212.30 EUR/month plus 1.70 EUR plus 209 EUR setup. GPU latency is measured on an RTX 5070 Ti (same generation);
GEX45 values are approximations.
Capacity = 2,592,000 s per month / mean latency (one query at a time), at full load and at 50 % utilisation; for the
GPU this is a lower bound (no batching across queries). Latency budgets (assumption): live search p95 <= 500 ms in
total (first stage plus reranker); chat p95 <= 2,000 ms for the reranking step. Break-even = monthly price / API cost
per query, valid
only if capacity and p95 fit, otherwise "not reachable".
Context calculation: the smallest k without reranker whose Hit@k (and Recall@k) reaches the reranked value at k = 5;
saved passages times mean passage tokens (Voyage tokenizer as approximation) times the input price of two reference
LLMs (list prices with date) gives the saved LLM cost per 1,000 queries, compared with the reranker cost. Answer
quality is not measured.

## Reproducibility and licences
Software: all Python code runs in one container (`docker/bench.Dockerfile`, base images pinned by digest; Python
3.14.8 and every package pinned by `uv.lock`); Typesense runs in Docker as well.
Run files (TREC, ids and scores) and result files (JSON, ids and numbers only, no texts) are committed under
CC BY 4.0; code under Apache-2.0. A fresh clone must reproduce nDCG@10 of two rerankers from the committed
first-stage runs on the same GPU within 0.1 points.