# Results (generated by `grb report`)

Measured: 2026-10-09 to 2026-10-11 (UTC). Repository commit: `e715d27a1bf73df51f6cbddc99e936b293488590` (clean tree). Plan commit: `dec01291a2ce67e259385f3a69145b6b5f041f50`. BM25 reference (not run): nDCG@10 0.226, Recall@100 0.572; source: MIRACL paper (TACL 2023), Table 5, full corpus.

Flags: contamination flags from the MTEB registry (read 2026-10-09): flagged models are marked in every table; a flagged model is never called a winner without this note. A flag names the MIRACL datasets in the training data of the model, or of the Arctic embeddings that a first stage searches with. No flag does not mean clean: the training data of gte-base, jina-v3.5, voyage-rerank-3, voyage-rerank-3-lite is unknown or not listed in the registry.

## keyword-71k

| model | nDCG@10 (95 % CI) | Hit@5 | Recall@20 | vs first stage | flags |
|---|---|---|---|---|---|
| first stage (keyword-71k) | 0.0743 (0.0543–0.0961) | 0.134 | 0.145 | - |  |
| qwen3-4b | 0.2193 (0.1809–0.2587) | 0.334 | 0.236 | clearly better | contaminated: MIRACLReranking |
| jina-v3.5 | 0.2161 (0.1786–0.2545) | 0.344 | 0.233 | clearly better |  |
| voyage-rerank-3 | 0.2147 (0.1766–0.2537) | 0.328 | 0.229 | clearly better |  |
| qwen3-0.6b | 0.2116 (0.1748–0.2493) | 0.331 | 0.234 | clearly better | contaminated: MIRACLReranking |
| querit-4b | 0.2093 (0.1716–0.2481) | 0.318 | 0.232 | clearly better | contaminated: MIRACLReranking |
| bge-v2-m3 | 0.2080 (0.1709–0.2458) | 0.331 | 0.229 | clearly better | contaminated: MIRACLReranking, MIRACLRetrieval, MIRACLRetrievalHardNegatives |
| voyage-rerank-3-lite | 0.2062 (0.1701–0.2439) | 0.311 | 0.230 | clearly better |  |
| gte-base | 0.2023 (0.1661–0.2396) | 0.315 | 0.223 | clearly better |  |
| mxbai-large-v2 | 0.1925 (0.1580–0.2284) | 0.305 | 0.226 | clearly better |  |

Best: qwen3-4b (contaminated: MIRACLReranking). Winner: none (wording rule).

Equal scores are ordered by the rank in the first stage; trec_eval would reorder them by docid.

## vector-71k

| model | nDCG@10 (95 % CI) | Hit@5 | Recall@20 | vs first stage | flags |
|---|---|---|---|---|---|
| first stage (vector-71k) | 0.5903 (0.5533–0.6265) | 0.820 | 0.805 | - | Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| qwen3-4b | 0.6823 (0.6477–0.7165) | 0.882 | 0.891 | clearly better | contaminated: MIRACLReranking |
| voyage-rerank-3 | 0.6584 (0.6221–0.6938) | 0.852 | 0.864 | clearly better |  |
| jina-v3.5 | 0.6415 (0.6056–0.6772) | 0.852 | 0.845 | clearly better |  |
| qwen3-0.6b | 0.6378 (0.6038–0.6717) | 0.872 | 0.886 | clearly better | contaminated: MIRACLReranking |
| bge-v2-m3 | 0.6378 (0.6001–0.6741) | 0.852 | 0.847 | clearly better | contaminated: MIRACLReranking, MIRACLRetrieval, MIRACLRetrievalHardNegatives |
| querit-4b | 0.6333 (0.5958–0.6708) | 0.830 | 0.828 | clearly better | contaminated: MIRACLReranking |
| voyage-rerank-3-lite | 0.6272 (0.5912–0.6631) | 0.839 | 0.851 | clearly better |  |
| gte-base | 0.5675 (0.5298–0.6048) | 0.784 | 0.802 | on par |  |
| mxbai-large-v2 | 0.5398 (0.5022–0.5779) | 0.764 | 0.800 | clearly worse |  |

Best: qwen3-4b (contaminated: MIRACLReranking). Winner: qwen3-4b (contaminated: MIRACLReranking).

Caution: qwen3-4b, the winner of this stage, was trained on MIRACL data according to the MTEB registry (MIRACLReranking), like the Arctic embeddings of the first stage (MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking); read its nDCG@10 with this in mind.

Equal scores are ordered by the rank in the first stage; trec_eval would reorder them by docid.

## hybrid-71k

| model | nDCG@10 (95 % CI) | Hit@5 | Recall@20 | vs first stage | flags |
|---|---|---|---|---|---|
| first stage (hybrid-71k) | 0.1750 (0.1477–0.2037) | 0.338 | 0.329 | - | Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| qwen3-4b | 0.3790 (0.3336–0.4233) | 0.518 | 0.431 | clearly better | contaminated: MIRACLReranking |
| voyage-rerank-3 | 0.3788 (0.3338–0.4244) | 0.508 | 0.425 | clearly better |  |
| jina-v3.5 | 0.3766 (0.3317–0.4211) | 0.518 | 0.426 | clearly better |  |
| voyage-rerank-3-lite | 0.3664 (0.3224–0.4109) | 0.502 | 0.425 | clearly better |  |
| qwen3-0.6b | 0.3647 (0.3215–0.4069) | 0.508 | 0.431 | clearly better | contaminated: MIRACLReranking |
| querit-4b | 0.3622 (0.3185–0.4060) | 0.498 | 0.419 | clearly better | contaminated: MIRACLReranking |
| bge-v2-m3 | 0.3601 (0.3166–0.4034) | 0.511 | 0.425 | clearly better | contaminated: MIRACLReranking, MIRACLRetrieval, MIRACLRetrievalHardNegatives |
| gte-base | 0.3383 (0.2965–0.3807) | 0.485 | 0.409 | clearly better |  |
| mxbai-large-v2 | 0.3309 (0.2897–0.3718) | 0.482 | 0.418 | clearly better |  |

Best: qwen3-4b (contaminated: MIRACLReranking). Winner: none (wording rule).

Equal scores are ordered by the rank in the first stage; trec_eval would reorder them by docid.

## keyword-full

| model | nDCG@10 (95 % CI) | Hit@5 | Recall@20 | vs first stage | flags |
|---|---|---|---|---|---|
| first stage (keyword-full) | 0.0615 (0.0424–0.0825) | 0.108 | 0.102 | - |  |
| qwen3-4b | 0.1612 (0.1272–0.1971) | 0.246 | 0.165 | clearly better | contaminated: MIRACLReranking |
| voyage-rerank-3 | 0.1586 (0.1251–0.1940) | 0.236 | 0.163 | clearly better |  |
| querit-4b | 0.1555 (0.1226–0.1899) | 0.239 | 0.165 | clearly better | contaminated: MIRACLReranking |
| qwen3-0.6b | 0.1546 (0.1217–0.1888) | 0.239 | 0.164 | clearly better | contaminated: MIRACLReranking |
| bge-v2-m3 | 0.1532 (0.1207–0.1876) | 0.236 | 0.163 | clearly better | contaminated: MIRACLReranking, MIRACLRetrieval, MIRACLRetrievalHardNegatives |
| voyage-rerank-3-lite | 0.1529 (0.1202–0.1871) | 0.230 | 0.163 | clearly better |  |
| mxbai-large-v2 | 0.1480 (0.1163–0.1814) | 0.233 | 0.162 | clearly better |  |
| gte-base | 0.1471 (0.1154–0.1805) | 0.223 | 0.156 | clearly better |  |
| jina-v3.5 | not measurable at depth 100 | - | - | - |  |

Not measurable at depth 100 — jina-v3.5: vendor listwise block exceeds 16 GB VRAM for at least 13 of 304 queries (prompts >= 45.2k tokens OOM; 35.5k fit).

Best: qwen3-4b (contaminated: MIRACLReranking). Winner: none (wording rule).

Equal scores are ordered by the rank in the first stage; trec_eval would reorder them by docid.

## Checks

Checks of the quality measurements. determinism: a second run of the same model gives the same nDCG@10 (difference and tolerance in points). dtype: nDCG@10 on the GPU in the recommended data type against the CPU in fp32; it passes if the difference stays under the tolerance, so that the GPU value holds for the CPU. pointwise: the score of a Voyage candidate does not depend on the other candidates of the list. evaluation: our nDCG@10 against pytrec_eval on the first-stage run. The determinism and dtype checks ran on hybrid-71k (Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking).

| check | subject | result | passed | flags |
|---|---|---|---|---|
| determinism | bge-v2-m3 | nDCG@10 0.3601 and 0.3601 in two runs on 305 queries; difference 0.00 points (tolerance 0.1); 305 of 305 rankings identical | yes | contaminated: MIRACLReranking, MIRACLRetrieval, MIRACLRetrievalHardNegatives |
| determinism | jina-v3.5 | nDCG@10 0.3766 and 0.3766 in two runs on 305 queries; difference 0.00 points (tolerance 0.1); 305 of 305 rankings identical | yes |  |
| determinism | qwen3-0.6b | nDCG@10 0.3647 and 0.3647 in two runs on 305 queries; difference 0.00 points (tolerance 0.1); 305 of 305 rankings identical | yes | contaminated: MIRACLReranking |
| dtype | bge-v2-m3 | nDCG@10 0.7611 on the GPU and 0.7611 on the CPU (fp32), 20 queries; difference 0.00 points (tolerance 0.5); 1 of 20 rankings identical | yes | contaminated: MIRACLReranking, MIRACLRetrieval, MIRACLRetrievalHardNegatives |
| dtype | gte-base | nDCG@10 0.6865 on the GPU and 0.6865 on the CPU (fp32), 20 queries; difference 0.00 points (tolerance 0.5); 2 of 20 rankings identical | yes |  |
| dtype | jina-v3.5 | nDCG@10 0.7724 on the GPU and 0.7712 on the CPU (fp32), 20 queries; difference 0.13 points (tolerance 0.5); 0 of 20 rankings identical | yes |  |
| dtype | qwen3-0.6b | nDCG@10 0.7305 on the GPU and 0.7305 on the CPU (fp32), 20 queries; difference 0.00 points (tolerance 0.5); 0 of 20 rankings identical | yes | contaminated: MIRACLReranking |
| pointwise | voyage-rerank-3 | same score for the same candidate at k = 20 and 100; largest difference 0 (tolerance 0.001), 5 queries | yes |  |
| pointwise | voyage-rerank-3-lite | same score for the same candidate at k = 20 and 100; largest difference 0 (tolerance 0.001), 5 queries | yes |  |
| evaluation | hybrid-71k run | nDCG@10 0.1750 with our code and 0.1750 with pytrec_eval, 305 queries; difference 0 (tolerance 0.0001) | yes | Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |

## First stage (no reranker)

Wall time of one search request to Typesense on the desktop (p50 / p95 in ms). The query embedding (Arctic, GPU, batch 1) is measured apart and comes on top for the stages that search with vectors. The ceiling nDCG@10 is the nDCG@10 of a perfect reordering of the top 100: the most a reranker could reach.

| first stage | nDCG@10 (95 % CI) | Recall@100 | ceiling nDCG@10 | search p50 / p95 (ms) | query embedding p95 (ms) | flags |
|---|---|---|---|---|---|---|
| keyword-71k | 0.0743 (0.0543–0.0961) | 0.238 | 0.2639 | 36.8 / 55.0 | - |  |
| vector-71k | 0.5903 (0.5533–0.6265) | 0.965 | 0.9716 | 31.7 / 41.0 | 11.3 | Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| hybrid-71k | 0.1750 (0.1477–0.2037) | 0.435 | 0.4612 | 35.0 / 56.0 | 11.3 | Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| keyword-full | 0.0615 (0.0424–0.0825) | 0.166 | 0.1857 | 56.5 / 764.9 | - |  |
| keyword-71k-ltr (exploratory) | 0.1225 (0.0984–0.1488) | 0.463 | 0.5014 | - | - |  |
| keyword-plain-71k | 0.0121 (0.0049–0.0208) | 0.012 | 0.0157 | 21.4 / 76.7 | - |  |

Exploratory (keyword-71k-ltr): added after the first stages had been measured (DEVIATIONS.md, 2026-10-10); in no pre-registered comparison.

## Latency (p50 / p95 ms)

Time for one query with k candidates, text in and scores out. GPU: NVIDIA GeForce RTX 5070 Ti; API: Voyage, from the desktop in Sweden at 3 times of day (morning, afternoon, evening). Candidates: the first k of hybrid-71k (Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking).

| device | model | k=20 | k=50 | k=100 | flags |
|---|---|---|---|---|---|
| cuda:desktop-se | bge-v2-m3 | 85.9 / 108.0 | 259.3 / 264.3 | 496.5 / 516.6 | contaminated: MIRACLReranking, MIRACLRetrieval, MIRACLRetrievalHardNegatives |
| cuda:desktop-se | gte-base | 41.4 / 50.1 | 123.1 / 125.4 | 244.5 / 247.3 |  |
| cuda:desktop-se | jina-v3.5 | 186.3 / 337.7 | 835.8 / 1,588.8 | 2,957.4 / 5,466.7 |  |
| cuda:desktop-se | mxbai-large-v2 | 515.2 / 996.3 | 1,220.3 / 2,221.4 | 2,095.4 / 4,342.6 |  |
| cuda:desktop-se | querit-4b | 2,639.5 / 2,687.8 | 6,578.1 / 6,651.6 | 13,118.4 / 13,134.3 | contaminated: MIRACLReranking |
| cuda:desktop-se | qwen3-0.6b | 322.8 / 692.6 | 693.5 / 1,269.0 | 1,166.9 / 2,647.6 | contaminated: MIRACLReranking |
| cuda:desktop-se | qwen3-4b | 864.1 / 1,418.8 | 1,976.4 / 2,918.8 | 3,799.2 / 5,509.1 | contaminated: MIRACLReranking |
| api:desktop-se:morning | voyage-rerank-3 | 278.2 / 328.9 | - | 340.6 / 433.7 |  |
| api:desktop-se:morning | voyage-rerank-3-lite | 286.7 / 361.4 | - | 350.0 / 414.3 |  |
| api:desktop-se:afternoon | voyage-rerank-3 | 278.5 / 366.0 | - | 341.3 / 456.9 |  |
| api:desktop-se:afternoon | voyage-rerank-3-lite | 287.0 / 356.1 | - | 344.4 / 435.1 |  |
| api:desktop-se:evening | voyage-rerank-3 | 278.5 / 355.5 | - | 342.2 / 504.9 |  |
| api:desktop-se:evening | voyage-rerank-3-lite | 289.3 / 352.5 | - | 353.2 / 481.8 |  |

Time of day is the local time of the desktop (Europe/Stockholm): before 12:00 morning, 12:00 to 18:00 afternoon, from 18:00 evening.

Result files of the Voyage runs were written at (local time): evening Saturday 2026-10-10 at 18:29 (voyage-rerank-3) and 18:32 (voyage-rerank-3-lite); morning Sunday 2026-10-11 at 08:34 (voyage-rerank-3) and 08:37 (voyage-rerank-3-lite); afternoon Sunday 2026-10-11 at 12:05 (voyage-rerank-3) and 12:08 (voyage-rerank-3-lite). Each run started a few minutes earlier.

All 3 times of day fall on a weekend (Saturday and Sunday); no weekday was measured, so the load of the API on a working day is not covered.

The training opt-out of the Voyage account, as recorded with each run: unconfirmed (checked 2026-10-09) for the evening and morning runs, confirmed (checked 2026-10-11) for the afternoon runs.

## Costs: API (EUR per 1,000 queries)

Tokens that Voyage billed per query (measured). List prices per million tokens: rerank-3 0.05 USD (2026-10-11), rerank-3-lite 0.02 USD (2026-10-11). Exchange rate 1.1206 USD per EUR: ECB reference rate of Friday 2026-10-09, the last one before the measurement days (Saturday 2026-10-10, Sunday 2026-10-11); the ECB publishes no rate on weekends.

| model | first stage | k | tokens per query | EUR per 1,000 queries | flags |
|---|---|---|---|---|---|
| voyage-rerank-3 | keyword-71k | 100 | 21,704 | 0.968 |  |
| voyage-rerank-3 | vector-71k | 100 | 17,494 | 0.781 | Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| voyage-rerank-3 | hybrid-71k | 20 | 4,248 | 0.190 | Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| voyage-rerank-3 | hybrid-71k | 100 | 21,368 | 0.953 | Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| voyage-rerank-3 | keyword-full | 100 | 27,486 | 1.226 |  |
| voyage-rerank-3-lite | keyword-71k | 100 | 21,704 | 0.387 |  |
| voyage-rerank-3-lite | vector-71k | 100 | 17,494 | 0.312 | Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| voyage-rerank-3-lite | hybrid-71k | 20 | 4,248 | 0.076 | Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| voyage-rerank-3-lite | hybrid-71k | 100 | 21,368 | 0.381 | Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| voyage-rerank-3-lite | keyword-full | 100 | 27,486 | 0.491 |  |

## Costs: GPU server against the API

Server gex45 at 214.00 EUR per month against the API (rerank-3 and rerank-3-lite), for k = 20 and 100: the candidate counts of the Voyage measurement. Capacity in queries per month from the mean latency, at full load and at 50 % load. A status is reachable if the break-even is within the capacity at 50 % load and the p95 fits the budget: 2000 ms for the reranking step of a chat, 500 ms for a live search with its first stage (67.2 ms p95). Candidates: the first k of hybrid-71k (Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking).

| model | k | capacity per month (full / 50 % load) | break-even vs rerank-3 (queries per month) | chat vs rerank-3 | live vs rerank-3 | break-even vs rerank-3-lite (queries per month) | chat vs rerank-3-lite | live vs rerank-3-lite | flags |
|---|---|---|---|---|---|---|---|---|---|
| bge-v2-m3 | 20 | 31,190,898 / 15,595,449 | 1,128,984 | reachable | reachable | 2,822,460 | reachable | reachable | contaminated: MIRACLReranking, MIRACLRetrieval, MIRACLRetrievalHardNegatives |
| bge-v2-m3 | 100 | 5,330,770 / 2,665,385 | 224,451 | reachable | not reachable | 561,127 | reachable | not reachable | contaminated: MIRACLReranking, MIRACLRetrieval, MIRACLRetrievalHardNegatives |
| gte-base | 20 | 65,339,774 / 32,669,887 | 1,128,984 | reachable | reachable | 2,822,460 | reachable | reachable |  |
| gte-base | 100 | 10,977,994 / 5,488,997 | 224,451 | reachable | reachable | 561,127 | reachable | reachable |  |
| jina-v3.5 | 20 | 12,750,989 / 6,375,495 | 1,128,984 | reachable | reachable | 2,822,460 | reachable | reachable |  |
| jina-v3.5 | 100 | 837,351 / 418,675 | 224,451 | not reachable | not reachable | 561,127 | not reachable | not reachable |  |
| mxbai-large-v2 | 20 | 4,543,088 / 2,271,544 | 1,128,984 | reachable | not reachable | 2,822,460 | not reachable | not reachable |  |
| mxbai-large-v2 | 100 | 1,136,688 / 568,344 | 224,451 | not reachable | not reachable | 561,127 | not reachable | not reachable |  |
| querit-4b | 20 | 978,844 / 489,422 | 1,128,984 | not reachable | not reachable | 2,822,460 | not reachable | not reachable | contaminated: MIRACLReranking |
| querit-4b | 100 | 197,438 / 98,719 | 224,451 | not reachable | not reachable | 561,127 | not reachable | not reachable | contaminated: MIRACLReranking |
| qwen3-0.6b | 20 | 6,964,359 / 3,482,180 | 1,128,984 | reachable | not reachable | 2,822,460 | reachable | not reachable | contaminated: MIRACLReranking |
| qwen3-0.6b | 100 | 2,003,913 / 1,001,956 | 224,451 | not reachable | not reachable | 561,127 | not reachable | not reachable | contaminated: MIRACLReranking |
| qwen3-4b | 20 | 2,804,435 / 1,402,217 | 1,128,984 | reachable | not reachable | 2,822,460 | not reachable | not reachable | contaminated: MIRACLReranking |
| qwen3-4b | 100 | 666,445 / 333,222 | 224,451 | not reachable | not reachable | 561,127 | not reachable | not reachable | contaminated: MIRACLReranking |

Statuses that differ between the two Voyage models: chat of mxbai-large-v2 at k = 20 (reachable against rerank-3, not reachable against rerank-3-lite); chat of qwen3-4b at k = 20 (reachable against rerank-3, not reachable against rerank-3-lite).

Notes: GPU latency was measured on an RTX 5070 Ti, not on the GEX45 (RTX PRO 4000 Blackwell SFF 24 GB): its break-even values are an approximation; the GEX45 monthly price is 214.00 EUR; the one-off setup fee of 209 EUR is not part of it.

The RTX PRO 4000 Blackwell SFF 24 GB of the GEX45 is expected to be slower than the NVIDIA GeForce RTX 5070 Ti that was measured (an expectation, not a measurement): a status close to a budget could flip.

The capacity of the GPU is a lower bound: queries are served one at a time, not batched across queries.

## Context cost: passages saved by reranking (EUR per 1,000 queries)

Without a reranker the first stage alone must hand the LLM k passages to reach the Hit@5 (or the Recall@5) that the top 5 passages of the reranker reach; with the reranker 5 passages are enough, so k - 5 passages are saved in the LLM context. A cell with two values shows Hit@5 first, Recall@5 second: Hit asks for one relevant passage in the top, Recall for the share of all relevant passages of a query. Savings use the mean passage length per stage (tokens: keyword-71k 209, vector-71k 167, hybrid-71k 205, keyword-full 267) and the list prices per million input tokens: anthropic/claude-sonnet-5.5 2.0 USD (2026-10-11), deepseek/deepseek-flash 0.15 USD (2026-10-11). For comparison, what the Voyage model of a row costs for 100 candidates (voyage-rerank-3 for the other rows).

| first stage | reranker | Hit@5 / Recall@5 reranked | depth without reranker (Hit / Recall) | passages saved (Hit / Recall) | saving anthropic/claude-sonnet-5.5 (Hit / Recall) | saving deepseek/deepseek-flash, off-peak (Hit / Recall) | saving deepseek/deepseek-flash, peak (Hit / Recall) | compared with | its cost for 100 candidates | flags |
|---|---|---|---|---|---|---|---|---|---|---|
| keyword-71k | bge-v2-m3 | 0.331 / 0.202 | 60 / 57 | 55 / 52 | 20.535 / 19.415 | 1.540 / 1.456 | 3.080 / 2.912 | voyage-rerank-3 | 0.968 | contaminated: MIRACLReranking, MIRACLRetrieval, MIRACLRetrievalHardNegatives |
| keyword-71k | gte-base | 0.315 / 0.186 | 57 / 48 | 52 / 43 | 19.415 / 16.054 | 1.456 / 1.204 | 2.912 / 2.408 | voyage-rerank-3 | 0.968 |  |
| keyword-71k | jina-v3.5 | 0.344 / 0.213 | 75 / 62 | 70 / 57 | 26.135 / 21.281 | 1.960 / 1.596 | 3.920 / 3.192 | voyage-rerank-3 | 0.968 |  |
| keyword-71k | mxbai-large-v2 | 0.305 / 0.183 | 53 / 47 | 48 / 42 | 17.921 / 15.681 | 1.344 / 1.176 | 2.688 / 2.352 | voyage-rerank-3 | 0.968 |  |
| keyword-71k | querit-4b | 0.318 / 0.196 | 58 / 56 | 53 / 51 | 19.788 / 19.041 | 1.484 / 1.428 | 2.968 / 2.856 | voyage-rerank-3 | 0.968 | contaminated: MIRACLReranking |
| keyword-71k | qwen3-0.6b | 0.331 / 0.199 | 60 / 57 | 55 / 52 | 20.535 / 19.415 | 1.540 / 1.456 | 3.080 / 2.912 | voyage-rerank-3 | 0.968 | contaminated: MIRACLReranking |
| keyword-71k | qwen3-4b | 0.334 / 0.210 | 62 / 60 | 57 / 55 | 21.281 / 20.535 | 1.596 / 1.540 | 3.192 / 3.080 | voyage-rerank-3 | 0.968 | contaminated: MIRACLReranking |
| keyword-71k | voyage-rerank-3 | 0.328 / 0.203 | 60 / 57 | 55 / 52 | 20.535 / 19.415 | 1.540 / 1.456 | 3.080 / 2.912 | voyage-rerank-3 | 0.968 |  |
| keyword-71k | voyage-rerank-3-lite | 0.311 / 0.194 | 57 / 55 | 52 / 50 | 19.415 / 18.668 | 1.456 / 1.400 | 2.912 / 2.800 | voyage-rerank-3-lite | 0.387 |  |
| vector-71k | bge-v2-m3 | 0.852 / 0.632 | 8 / 7 | 3 / 2 | 0.892 / 0.595 | 0.067 / 0.045 | 0.134 / 0.089 | voyage-rerank-3 | 0.781 | contaminated: MIRACLReranking, MIRACLRetrieval, MIRACLRetrievalHardNegatives; Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| vector-71k | gte-base | 0.784 / 0.552 | 4 / 5 | 0 / 0 | 0.000 / 0.000 | 0.000 / 0.000 | 0.000 / 0.000 | voyage-rerank-3 | 0.781 | Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| vector-71k | jina-v3.5 | 0.852 / 0.644 | 8 / 8 | 3 / 3 | 0.892 / 0.892 | 0.067 / 0.067 | 0.134 / 0.134 | voyage-rerank-3 | 0.781 | Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| vector-71k | mxbai-large-v2 | 0.764 / 0.532 | 4 / 5 | 0 / 0 | 0.000 / 0.000 | 0.000 / 0.000 | 0.000 / 0.000 | voyage-rerank-3 | 0.781 | Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| vector-71k | querit-4b | 0.830 / 0.626 | 6 / 7 | 1 / 2 | 0.297 / 0.595 | 0.022 / 0.045 | 0.045 / 0.089 | voyage-rerank-3 | 0.781 | contaminated: MIRACLReranking; Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| vector-71k | qwen3-0.6b | 0.872 / 0.660 | 9 / 8 | 4 / 3 | 1.190 / 0.892 | 0.089 / 0.067 | 0.178 / 0.134 | voyage-rerank-3 | 0.781 | contaminated: MIRACLReranking; Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| vector-71k | qwen3-4b | 0.882 / 0.684 | 10 / 9 | 5 / 4 | 1.487 / 1.190 | 0.112 / 0.089 | 0.223 / 0.178 | voyage-rerank-3 | 0.781 | contaminated: MIRACLReranking; Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| vector-71k | voyage-rerank-3 | 0.852 / 0.648 | 8 / 8 | 3 / 3 | 0.892 / 0.892 | 0.067 / 0.067 | 0.134 / 0.134 | voyage-rerank-3 | 0.781 | Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| vector-71k | voyage-rerank-3-lite | 0.839 / 0.612 | 7 / 7 | 2 / 2 | 0.595 / 0.595 | 0.045 / 0.045 | 0.089 / 0.089 | voyage-rerank-3-lite | 0.312 | Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| hybrid-71k | bge-v2-m3 | 0.511 / 0.365 | 26 / 32 | 21 / 27 | 7.695 / 9.893 | 0.577 / 0.742 | 1.154 / 1.484 | voyage-rerank-3 | 0.953 | contaminated: MIRACLReranking, MIRACLRetrieval, MIRACLRetrievalHardNegatives; Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| hybrid-71k | gte-base | 0.485 / 0.351 | 20 / 27 | 15 / 22 | 5.496 / 8.061 | 0.412 / 0.605 | 0.824 / 1.209 | voyage-rerank-3 | 0.953 | Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| hybrid-71k | jina-v3.5 | 0.518 / 0.373 | 30 / 38 | 25 / 33 | 9.160 / 12.092 | 0.687 / 0.907 | 1.374 / 1.814 | voyage-rerank-3 | 0.953 | Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| hybrid-71k | mxbai-large-v2 | 0.482 / 0.336 | 20 / 23 | 15 / 18 | 5.496 / 6.596 | 0.412 / 0.495 | 0.824 / 0.989 | voyage-rerank-3 | 0.953 | Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| hybrid-71k | querit-4b | 0.498 / 0.357 | 24 / 29 | 19 / 24 | 6.962 / 8.794 | 0.522 / 0.660 | 1.044 / 1.319 | voyage-rerank-3 | 0.953 | contaminated: MIRACLReranking; Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| hybrid-71k | qwen3-0.6b | 0.508 / 0.367 | 24 / 33 | 19 / 28 | 6.962 / 10.260 | 0.522 / 0.769 | 1.044 / 1.539 | voyage-rerank-3 | 0.953 | contaminated: MIRACLReranking; Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| hybrid-71k | qwen3-4b | 0.518 / 0.377 | 30 / 38 | 25 / 33 | 9.160 / 12.092 | 0.687 / 0.907 | 1.374 / 1.814 | voyage-rerank-3 | 0.953 | contaminated: MIRACLReranking; Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| hybrid-71k | voyage-rerank-3 | 0.508 / 0.372 | 24 / 37 | 19 / 32 | 6.962 / 11.725 | 0.522 / 0.879 | 1.044 / 1.759 | voyage-rerank-3 | 0.953 | Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| hybrid-71k | voyage-rerank-3-lite | 0.502 / 0.361 | 24 / 32 | 19 / 27 | 6.962 / 9.893 | 0.522 / 0.742 | 1.044 / 1.484 | voyage-rerank-3-lite | 0.381 | Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| keyword-full | bge-v2-m3 | 0.236 / 0.145 | 72 / 71 | 67 / 66 | 31.868 / 31.393 | 2.390 / 2.354 | 4.780 / 4.709 | voyage-rerank-3 | 1.226 | contaminated: MIRACLReranking, MIRACLRetrieval, MIRACLRetrievalHardNegatives |
| keyword-full | gte-base | 0.223 / 0.135 | 61 / 54 | 56 / 49 | 26.636 / 23.307 | 1.998 / 1.748 | 3.995 / 3.496 | voyage-rerank-3 | 1.226 |  |
| keyword-full | jina-v3.5 | not measurable at depth 100 | - | - | - | - | - | voyage-rerank-3 | 1.226 |  |
| keyword-full | mxbai-large-v2 | 0.233 / 0.141 | 71 / 65 | 66 / 60 | 31.393 / 28.539 | 2.354 / 2.140 | 4.709 / 4.281 | voyage-rerank-3 | 1.226 |  |
| keyword-full | querit-4b | 0.239 / 0.145 | 76 / 71 | 71 / 66 | 33.771 / 31.393 | 2.533 / 2.354 | 5.066 / 4.709 | voyage-rerank-3 | 1.226 | contaminated: MIRACLReranking |
| keyword-full | qwen3-0.6b | 0.239 / 0.145 | 76 / 71 | 71 / 66 | 33.771 / 31.393 | 2.533 / 2.354 | 5.066 / 4.709 | voyage-rerank-3 | 1.226 | contaminated: MIRACLReranking |
| keyword-full | qwen3-4b | 0.246 / 0.153 | 78 / 76 | 73 / 71 | 34.722 / 33.771 | 2.604 / 2.533 | 5.208 / 5.066 | voyage-rerank-3 | 1.226 | contaminated: MIRACLReranking |
| keyword-full | voyage-rerank-3 | 0.236 / 0.146 | 72 / 71 | 67 / 66 | 31.868 / 31.393 | 2.390 / 2.354 | 4.780 / 4.709 | voyage-rerank-3 | 1.226 |  |
| keyword-full | voyage-rerank-3-lite | 0.230 / 0.139 | 71 / 64 | 66 / 59 | 31.393 / 28.063 | 2.354 / 2.105 | 4.709 / 4.209 | voyage-rerank-3-lite | 0.491 |  |

An entry without a depth-100 run is not measurable and marked so.

deepseek/deepseek-flash is shown at its off-peak price of 0.15 USD per million input tokens and at its peak price of 0.30 USD (2x).

Passage tokens are approximated with the Voyage tokenizer (mean_passage_tokens: per stage, voyageai/rerank-3 at revision 2d2c29517b5c3bbf130c879fda12b1678e598158 on title and text of the top 100 candidates of all queries).

Answer quality is not measured.

## CPU latency (indicative, desktop pre-measurement)

AMD Ryzen 9 5950X 16-Core Processor, 8 threads, container memory capped at 30 GB (DEVIATIONS.md 2026-10-11), 10 queries, 2 warm-up calls, 1 pass. qwen3-0.6b was measured again at k = 20 after its run ran out of memory at k = 100 (DEVIATIONS.md, 2026-10-11); the other models were measured once and not repeated. The load average before a run shows the after-effect of the run before. The values give an order of magnitude only. They are not part of the latency table, the chart or the costs. Candidates: the first k of hybrid-71k (Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking).

| model | k=20 mean / p50 / p95 (ms) | k=100 mean / p50 / p95 (ms) | load average (1 min) before | flags |
|---|---|---|---|---|
| bge-v2-m3 | 9,977 / 10,242 / 12,917 | 61,410 / 62,147 / 62,643 | 7.18 | contaminated: MIRACLReranking, MIRACLRetrieval, MIRACLRetrievalHardNegatives |
| gte-base | 4,224 / 4,512 / 5,357 | 28,298 / 28,612 / 28,820 | 1.12 |  |
| jina-v3.5 | 19,276 / 18,283 / 34,406 | 188,551 / 195,979 / 283,558 | 6.37 |  |
| qwen3-0.6b | 44,060 / 35,534 / 98,837 | not measurable in 30 GB | 0.76 | contaminated: MIRACLReranking |

## Pitfalls: integration

qwen3-0.6b on hybrid-71k: the correct integration (the model card's code) against 4 common mistakes. max256 is a maximum input length of 256 tokens that cuts the passage (correct) or the end of the whole input (tail-cut). Differences in nDCG@10 points against the correct integration; the mistakes are one Holm family of paired randomisation tests, worded by the wording rule. The correct integration differs from the sentence-transformers main run by 0.05 points (tolerance 0.2).

| variant | reranker | first stage | nDCG@10 (95 % CI) | difference to correct (points) | p (Holm) | wording | flags |
|---|---|---|---|---|---|---|---|
| correct | qwen3-0.6b | hybrid-71k | 0.3652 (0.3220–0.4075) | - | - | - | contaminated: MIRACLReranking; Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| no-instruction | qwen3-0.6b | hybrid-71k | 0.3756 (0.3315–0.4182) | +1.04 | 0.0428 | measurably better | contaminated: MIRACLReranking; Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| max256-correct | qwen3-0.6b | hybrid-71k | 0.3584 (0.3162–0.3998) | -0.68 | 0.0428 | measurably worse | contaminated: MIRACLReranking; Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| max256-tail-cut | qwen3-0.6b | hybrid-71k | 0.2049 (0.1717–0.2390) | -16.03 | 0.0004 | clearly worse | contaminated: MIRACLReranking; Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| right-padding | qwen3-0.6b | hybrid-71k | 0.1357 (0.1099–0.1633) | -22.95 | 0.0004 | clearly worse | contaminated: MIRACLReranking; Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |

## Pitfalls: Typesense

Typesense with its defaults (keyword-plain-71k) against the German setup (keyword-71k): pre-registered, a family of one. keyword-71k-ltr is exploratory: a variant of keyword-71k, added after the first stages had been measured (DEVIATIONS.md, 2026-10-10); its p-value is raw and belongs to no Holm family.

| first stage | nDCG@10 (95 % CI) | difference (points) | p | wording | flags |
|---|---|---|---|---|---|
| keyword-plain-71k | 0.0121 (0.0049–0.0208) | - | - | - |  |
| keyword-71k | 0.0743 (0.0543–0.0961) | +6.22 vs keyword-plain-71k | 0.0001 | clearly better |  |
| keyword-71k-ltr (exploratory) | 0.1225 (0.0984–0.1488) | +4.82 vs keyword-71k | 0.0001 (raw) | exploratory: raw p, in no family |  |

## Judged sample

Unjudged passages from ranks 1 to 3 of the best rated models, judged blind by an LLM (claude-opus-5-5), and control pairs with a known MIRACL judgment. The rate is the share judged relevant, an estimate of the false negatives among the unjudged passages, with a Wilson 95 % interval; unclear answers are left out (0 of 72).

| group | first stage | reranker | judged | relevant | rate (Wilson 95 % interval) | flags |
|---|---|---|---|---|---|---|
| unjudged top 3 | hybrid-71k | jina-v3.5 | 15 | 6 | 0.400 (0.198–0.643) | Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| unjudged top 3 | hybrid-71k | qwen3-4b | 15 | 6 | 0.400 (0.198–0.643) | contaminated: MIRACLReranking; Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| unjudged top 3 | hybrid-71k | voyage-rerank-3 | 15 | 7 | 0.467 (0.248–0.699) | Arctic embeddings, contaminated: MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLReranking |
| unjudged top 3 | keyword-full | qwen3-4b | 15 | 1 | 0.067 (0.012–0.298) | contaminated: MIRACLReranking |
| control-relevant | - | - | 8 | 8 | 1.000 (0.676–1.000) |  |
| control-irrelevant | - | - | 7 | 2 | 0.286 (0.082–0.641) |  |

Controls: the judge agreed with the MIRACL judgments on 13 of 15 control pairs (87%).

Caveats: 15 pairs per group, so the intervals are wide; an LLM judge may favour the same passages as LLM-based rerankers; the control pairs measure agreement with the MIRACL annotators, not this bias; 2 of the 7 control pairs that MIRACL marks not relevant were judged relevant (false alarms).