# Deviations from PLAN.md Each entry: date (UTC), what changed and why. Every change of this file is timestamped with OpenTimestamps (`DEVIATIONS.md.ots`). ## 2026-10-09 (UTC) — clarification, not a deviation: Arctic dtype PLAN.md fixes model, revision, prompt `query: `, L2 normalisation and 1024 dimensions for the Arctic embeddings but no dtype. They are computed in float16 on the GPU and renormalised to unit length in float32 (exact search then equals cosine; top-100 overlap with a float64 cosine reference 1.0 on 71k). The dtype and prompt are recorded with every first-stage result. Checked before any measurement: on 71k, float32 gives nDCG@10 0.5895 (the MTEB reference value) and float16 0.5903; top-100 overlap between the two 0.9989. ## 2026-10-10 (UTC) — mxbai-large-v2: vendor input format instead of the sentence-transformers template PLAN.md lists mxbai-large-v2 as "sentence-transformers CrossEncoder, raw logit score" and names the vendor package `mxbai-rerank==0.1.6` as its deciding equality reference. Before any measurement, the code review of the adapters found that the two feed the model different tokens: the sentence-transformers path renders the chat template into one string and tokenises it at once, which merges boundary tokens such as `?\n`; mxbai-rerank tokenises query, separators, document and task prompt piece by piece. On the three fixture queries all 300 pairs differ (e.g. 327 vs 329 tokens); Spearman between the two inputs per query 0.992 / 0.965 / 0.992, scores differ by up to 1.31 logits. The adapter therefore uses the vendor's input format (ported from mxbai-rerank 0.1.6, `MxbaiRerankV2`), with the same weights, dtype and raw score logit("1") - logit("0"). The equality check against the vendor package decides as planned. The sentence-transformers difference is reported as an integration pitfall. Clarification in the same review: "vendor default" maximum input for qwen3-0.6b, qwen3-4b and mxbai is 8192 tokens (model-card and vendor code cut; sentence-transformers would allow 40960 / 32768). The value is recorded with every result. ## 2026-10-10 (UTC) — container isolation for local models; torch native JIT disabled PLAN.md says all timings are taken inside the benchmark container with the host network. Models that execute vendor code from Hugging Face (gte, jina, Querit) now run only in an isolated mode of the same container: no network (`--network none`), repository and cache read-only, no secrets visible; only result and run directories and a work directory are writable. To keep conditions equal, every local-model run (adapter checks, quality runs, GPU and CPU latency) uses this mode; no local model sends or receives data over the network, so the computation is unchanged. Voyage API latency and first-stage (Typesense) latency use the host network as planned. torch 2.14.1 routes CUDA `bmm` with an inner dimension of 1 (the rotary embeddings of the Qwen2/Qwen3-based models) to a Triton kernel that compiles C code at first use; the container has no C compiler. `TORCH_DISABLE_NATIVE_JIT=1` is therefore set for every run, which restores the stock ATen kernel. In this environment Triton is the only native kernel package, so only this outer-product kernel is affected; a check gave bit-identical results. ## 2026-10-10 (UTC) — equality references: environments outside uv.lock; mxbai reference at batch size 8 PLAN.md says Python 3.14.8 and every package are pinned by `uv.lock`. The equality references for gte, jina, Querit and mxbai run in two separate virtual environments, because they need transformers 4.57: Python 3.13.16, torch 2.14.1, transformers 4.57.6, huggingface-hub 0.36.2, and for mxbai additionally mxbai-rerank 0.1.6 and batched 0.1.5. They are pinned by exact versions, built before the runs, and every validation result records the installed versions. mxbai's deciding reference is the vendor package `mxbai-rerank` 0.1.6. With its default `batch_size=32` it computes full-vocabulary logits at every position and needs 12.8 GiB more than the 16 GB GPU has, so the first equality run recorded mxbai as not rated (the result is kept). The reference was re-run with the package's own `batch_size=8` parameter, otherwise unchanged; the batch size is recorded in the result. Minimum Spearman 0.9966 (mean 0.9991) over the 20 queries; mxbai is rated. ## 2026-10-10 (UTC) — jina-v3.5 on keyword-full: depth 100 not measurable on 16 GB; CPU pre-measurement shortened PLAN.md runs every reranker on every stage at depth 100, jina separately per depth. jina-v3.5 ranks all candidates of a query jointly; its vendor code packs up to 125 documents into one block (131k-token context). On keyword-full, where passages from the full corpus are long, the depth-100 block of at least 13 of 304 queries (prompts of 45.2k tokens and more) does not fit the 16 GB GPU; prompts up to 35.5k tokens fit. The vendor's block logic is not changed. jina on keyword-full is measured at depths 20 and 50; depth 100 is recorded as not measurable on this GPU and jina is left out of the depth-100 comparison of that stage. The desktop CPU pre-measurement (not published; it only estimates the runtime of the published CPU run) would take about 25 hours with 100 queries on 8 threads. It is shortened to the first 10 queries of the fixed latency sample with 2 warm-up calls per k. The published CPU protocol is unchanged. ## 2026-10-10 (UTC) — latency: bundle manifest instead of the dataset check; additions to the protocol PLAN.md checks the files against `datasets.lock` at the start of every measuring command. The reranker latency runs read no dataset file: they measure from a bundle (the first 100 first-stage candidates of every hybrid-71k query with their texts, kept outside the repository), which `grb latency bundle` builds after the `datasets.lock` check. The committed `configs/latency_bundles.json` lists every bundle with the SHA-256 of the file and of its content, of `datasets.lock` and of the first-stage run file. A latency run refuses a bundle that is not listed, or whose lock or run file no longer matches the checkout, before a model loads, and records the entry with its result. The check also holds on the CPU server, which receives only the bundles. Additions, decided before any latency measurement: - On the GPU, one untimed call per model initialises CUDA before the 20 warm-up calls. - A call that needed an HTTP retry is not timed; retried calls are counted per k. A share above 5 % of a k is flagged in the result; above 20 % no result is written and the run is repeated, because p50 and p95 would be biased. - The TCP connect baseline goes to the address the API host name resolves to, which is the provider's edge, not its data centre; address and spread are recorded. - First-stage latency is the client wall time of one search request over a keep-alive connection to Typesense on the same machine, with the query vectors computed beforehand. The embedding of the query (Arctic, GPU, batch 1) is measured separately and reported apart, so that the total time of the vector and hybrid stages can be estimated. - The `nvidia-smi` log is summarised per k; samples at 10 % GPU utilisation or more count as busy (the idle desktop shows 1–3 %). ## 2026-10-10 (UTC) — pitfalls experiment: exploratory Typesense variant keyword-71k-ltr, added after the fact PLAN.md compares keyword-plain-71k against keyword-71k. After the first stages had been measured, a diagnosis of the weak keyword stage (nDCG@10 0.0743 and Recall@100 0.2376 on 71k, against 0.2316 and 0.6747 for an independent BM25 on the same data) found the cause in Typesense's default `drop_tokens_mode` (`right_to_left`): when no passage matches all words of a German question, it drops words from the end, so it keeps the leading question word (wann, wer, warum; not in the stop word list) and drops the content words. The pitfalls experiment therefore adds keyword-71k-ltr: keyword-71k with `drop_tokens_mode=left_to_right`, otherwise identical, without reranking. The diagnosis had shown nDCG@10 0.1225 and Recall@100 0.4626; the experiment's run reproduces these values. The variant was chosen after its effect was seen (decided on 2026-10-09, before the experiment ran; this entry is written after the run). It is reported as exploratory: its p-value against keyword-71k is raw and belongs to no Holm family. No rerank stage, pre-registered comparison or metric changes; its run file and parameters are separate, and the committed first-stage files are unchanged. ## 2026-10-10 (UTC) — GPU latency: chunk sizes; measurement conditions and repeat rule; clarifications **Chunk sizes on the GPU.** PLAN.md puts all k candidates into one forward pass if memory allows, otherwise into chunks whose size is recorded. A first GPU latency run took the chunk sizes of the quality runs, which the long passages of keyword-full had forced (querit-4b 4, qwen3-4b 2, mxbai-large-v2 8, qwen3-0.6b 8). The latency bundle holds hybrid-71k candidates, where larger chunks fit, so that run was stopped after 30 minutes, before any result was written. The chunk size per model is now the largest that fits the worst case of the bundle: b copies of its longest pair (padding makes no real chunk of size b larger), with 1.5 GiB of GPU memory left free, probed from large to small sizes. Result: bge-v2-m3 and gte-base 100 (every k in one pass), qwen3-0.6b and mxbai-large-v2 32, querit-4b 16, qwen3-4b 7; jina-v3.5 splits through its vendor code. A k at or below the chunk size runs in one pass. Every result records the chunk size and the passes per call. **Measurement conditions.** PLAN.md asks for an idle desktop. During the latency runs the desktop session stayed open (browser, mail and messenger idle in the background), and other work on the machine occasionally ran test suites on the CPU. Every latency result records the load average before and after the run and, on the GPU, the memory in use and the utilisation before the model loads; a host log notes GPU processes, load and memory before and after each model. Rule, fixed before the GPU runs: a model is measured again if the median latencies of its passes differ by more than 15 % or the load average exceeds 4. The repeated result then counts; both stay committed and the repetition is logged here. **Clarifications.** - Wording rule: PLAN.md defines "clearly better", "measurably better" and "on par"; the same thresholds apply symmetrically to a worse model ("clearly worse", "measurably worse"), e.g. a reranker against its first stage. - Pitfalls experiment: the four integration mistakes are compared with the correct integration as one Holm family, worded by the wording rule; keyword-71k against keyword-plain-71k is a family of one. - dtype check: the 20 queries are the first 20 dev queries by id with at least one relevant passage among their hybrid-71k candidates, so that every query is informative. - The Voyage quality scores were requested on 2026-10-10 between 06:03 and 06:21 UTC; this is the measurement date PLAN.md lists for the API models. The quality results record the time of their evaluation. ## 2026-10-10 (UTC) — clarification of the latency repeat rule, before any GPU latency result exists The load average in the repeat rule above is the 1-minute value recorded before the model loads (`context.loadavg_before`). The value after the run includes the measured process itself (one to two cores) and is not used for the rule. A disturbance during a run shows in the medians of its passes, which the 15 % rule checks per k. ## 2026-10-11 (UTC) — GPU latency of qwen3-0.6b at chunk size 24 The GPU latency run of qwen3-0.6b at chunk size 32 ran out of GPU memory at k=50, after its k=20 pass had finished (2026-10-11 04:19 UTC): PyTorch's caching allocator held 3.45 GiB reserved but unallocated, in pieces too small for the next request, more than the 1.5 GiB the chunk-size probe leaves free. No result was written. The model is measured again at chunk size 24, the next smaller size, at which k=20 still runs in one pass; should 24 run out of memory as well, 16 is used. The other models keep their sizes. ## 2026-10-11 (UTC) — CPU server run and OpenVINO int8 variant dropped; desktop CPU pre-measurement published as indicative values Decided by the operator after the desktop CPU pre-measurement. On 8 desktop threads one query with 100 candidates took 28 s (gte-base) to 189 s (jina-v3.5) on average, and at least 4.2 s with 20 candidates: far above both latency budgets (500 ms for live search, 2,000 ms for the reranking step of a chat). A CCX33 with 8 vCPU would not change that conclusion. - The published CPU run on the Hetzner CCX33 is not carried out. The OpenVINO int8 variant of bge-v2-m3, which served that run, is dropped as well; neither its quality nor its latency is measured. - The desktop CPU pre-measurement is published as indicative values instead of "not published": host label `desktop-se-cpu8`, Ryzen 9 5950X with 8 threads via `taskset`, container capped at 30 GB, the first 10 queries of the fixed latency sample, 2 warm-up calls per k, 1 pass, k = 20 and 100. bge-v2-m3 and jina-v3.5 started at a 1-minute load average of 7.2 and 6.4, the after-effect of the run that had ended just before. They are not repeated: the values only serve as an order of magnitude. - qwen3-0.6b ran out of memory at k = 100 under the 30 GB cap, with all 100 pairs in one pass, and wrote no result. Its k = 20 value is measured again under the same conditions; k = 100 is reported as not measurable in 30 GB. - Costs, capacity and break-even cover the Voyage API and the GPU server (GEX45); the CPU server is not costed. The report lists the desktop CPU values in a separate table marked as indicative. ## 2026-10-11 (UTC) — clarifications: costs, times of day, reference rate Written after the final review of the cost and report code, before the results of round 1 are published. No measured value changes. - Context calculation: PLAN.md names Hit@k and Recall@k. The saving is reported for both: the smallest k without reranker whose Hit@k, respectively Recall@k, reaches the reranked value at k = 5. Depending on the stage either one saves more passages. - Break-even: PLAN.md does not name the API it is compared with. It is reported against rerank-3 and, in addition, against rerank-3-lite. A status is "reachable" when the break-even fits within the capacity at 50 % utilisation and the p95 fits the budget; at full load no status would differ. - The reference LLM prices are the vendors' list prices with date (checked 2026-10-11). DeepSeek charges a peak price on weekdays from 01:00 to 04:00 and from 06:00 to 10:00 UTC (https://api-docs.deepseek.com/quick_start/pricing). The saving is computed at the off-peak price and, in addition, at the peak price, which doubles it. - Reference rate: the ECB publishes no rates on weekends. The measurement days 2026-10-10 and 2026-10-11 use the last reference rate before them, of Friday 2026-10-09. - Times of day of the API latency are local time in Europe/Stockholm: morning before 12:00, afternoon from 12:00 to 18:00, evening from 18:00; each run started within its window. All three runs fell on a weekend (Saturday evening, Sunday morning, Sunday afternoon), and the afternoon and evening runs started shortly after their windows opened. The result files record the end of each run (`created_utc`); the report gives that time and says so. - With the CPU server run, its qwen3-4b part (5 queries, k = 20) is dropped as well. ## 2026-10-12 (UTC) — correction of a contamination flag; diagnostics for the article Written after an independent review of the article, before the results of round 1 are published. No measured value changes. - Correction: PLAN.md lists the contamination flag of bge-reranker-v2-m3 as unknown ("bge and jina: unknown", read 2026-10-09). That was a misreading. The MTEB registry (mteb 2.24.3, `rerankers_custom.py`) gives the model `bge_m3_training_data`, which contains MIRACLRetrieval, MIRACLRetrievalHardNegatives and MIRACLReranking. `configs/models.toml` now flags it, and a test checks every flag against the pinned registry (`tests/test_registry.py`). MIRACL has no German training split, so for German the exposure is indirect. jina stays unknown (the registry lists the model without training data); the Querit-4B entry spells MIRACLReranking as "MIRACLRanking", its flag stands. - The four `quality-summary-*` files are written again on 2026-10-12 (`grb evaluate --summary`) so that they carry the corrected flag. Their numbers are unchanged; the earlier files stay in `results/`. A quality summary evaluates measured runs and measures nothing: the measurement period of the report no longer counts these files. - keyword-plain-71k uses Typesense's default search parameters, as PLAN.md says, not only no locale, stemming and stop words: among them `drop_tokens_threshold` 1 instead of 100 and prefix search on instead of off. Its difference to keyword-71k is the effect of all of these together (for example 287 against 3 queries with fewer than 100 hits); the share of each setting was not measured. - Diagnostics outside the plan, cited by the article; they are not results of the round: - Hybrid (71k): on average 94.2 of the 100 hybrid-71k hits are in the keyword-71k top 100 and 16.4 in the vector-71k top 100, computed from the committed first-stage runs with `scripts/hybrid_overlap.py`. Looked at on 2026-10-10 in the raw Typesense responses, which are not committed: the fusion `0.7/rank_text + 0.3/rank_vector` ranks the keyword side densely over few distinct `text_match` values (on average 20.9 in the top 100), and 67 % of adjacent pairs share the same fusion score. alpha = 0.3 (the Typesense default) stays as fixed; other values were not measured. - Stop words (71k, 2026-10-09, ad hoc against Typesense 30.2, not committed): entries that only match through their stem (for example "wurde" through "wurd") are applied in the first pass with all tokens, but not once `drop_tokens_threshold` drops tokens. This affects 72 of the 305 queries; with those words removed beforehand, nDCG@10 was 0.0826 instead of 0.0743. The fixed configuration stays.