What does reranking cost on a CPU, and what does it actually buy?

Document-grounded AI can fail before a single word is generated. The retriever misses the passage that holds the answer, ranks a near miss first, or works from text that came out of a scanned page garbled.

For document collections that can’t leave the building, the usual remedies (a hosted embedding API, a managed vector database, a bigger model) may not be available at all. LocalRAG++ asks a narrower, practical question: how much retrieval quality can you get locally, on a CPU, and when does extra computation stop paying for itself?

• Offline. Models are downloaded once; evaluation calls no external service.
• CPU only. No GPU acceleration in any measured run.
• Small models on purpose. A 33M-parameter embedder and a 22M-parameter reranker.
• A fixed, inspectable benchmark. 114 questions, scored by the same rule for every configuration. That rule is document-level; see the scoring note above.

Every configuration reads from the same chunk corpus. PDFs pass through a cascade of converters: Docling first for layout-aware conversion, Marker as a fallback for math-heavy content, PyMuPDF for fast baseline extraction. A confidence score over six text signals decides whether a page is re-extracted with Tesseract or PaddleOCR.

Text is split on section boundaries (up to 1,500 characters, 150 overlap) and each chunk gets a deterministic SHA-1-based identifier, so runs stay comparable. Two indexes sit over those chunks: BM25 with code-aware tokenisation, and an exact FAISS index of bge-small embeddings. Hybrid retrieval fuses the two ranked lists with reciprocal rank fusion.

For the reranking runs, hybrid retrieval supplies c candidates to a MiniLM cross-encoder, sweeping c across 10, 20, 40, 60 and 80, and the final top 10 is scored. That sweep is the experiment: it isolates what candidate depth buys and what it costs.

LocalRAG++ is co-authored with Dr Salaja S, and I don’t want this page to blur that. What I can point to publicly is the repository under my GitHub account: the evaluation harness, its configuration, the question set and the raw outputs behind every number below.

A precise split of who did what isn’t written up here yet. Until it is, I’d rather under-claim than over-claim.

Read the code and outputs on GitHub

Each configuration returns a top-10 list for each of the 114 questions. Each metric has a name that promises one thing and, on this question set, an implementation that measures something narrower. Both columns matter:

Read from src/eval/metrics.py andquestions.jsonl at commit ce5909e: all 94 evidence records have page: null. This describes the code; no metric was recomputed for this page.

What the 114 questions are

Counted from the repository’s questions.jsonl, the benchmark covers 2 PDFs:

• 70 section questions on a public corporate annual report
• 12 section questions on an arXiv-style preprint
• 12 factual questions on the same preprint
• 20 deliberately unanswerable questions spanning both

The questions were generated deterministically from the extracted chunks by a script in the repository: no LLM, fixed seed. That makes them reproducible. It also makes them templated (“What is discussed in the section …?” is a typical form), and in my reading that favours lexical matching, because the question shares its words with the heading it came from. Keep that in mind when you get to the BM25 result.

Does it come out the same twice?

The repository includes three repeat runs of the unreranked hybrid configuration. Quality metrics were identical each time, as you’d expect from deterministic retrieval. Latency was not:

Repo files metrics_run1–3.json, hybrid without reranking, n = 114. Latency is retrieval-only, so it differs slightly from the 34 ms in Table 1, which comes from a separate run.

Cold start is almost entirely the embedding model loading (about 18–21 s); the indexes load in about 0.2 s. In practice that argues for keeping the process warm for a session rather than starting it per query.

Two kinds of number appear below. Latency is a reported measurement the scoring rule doesn’t touch. Retrieval scores are the originally reported results under document-level matching (see the scoring note), so they say whether a configuration reached the right PDF, not the right passage.

The cost side is clear

Warm mean latency climbs from 6 ms for BM25 and 34 ms for hybrid to 395 ms with 10 reranked candidates and 901 ms with 20, then 1,572, 3,338 and 4,040 ms at 40, 60 and 80. The difference over hybrid is the reranking stage, a cross-encoder scoring every candidate on a CPU, so it grows with candidate count.

The averages hide the tail. The repository also records the 95th percentile, and for c=20 it’s 1,605 ms. If an interface has a hard one-second budget, c=20 doesn’t meet it for every query.

The benefit side, as reported, is document-level

In the reported table, reranking 10 candidates changes nothing: every score matches plain hybrid. At 20, reported P@1 goes from 0.754 to 0.789, MRR from 0.754 to 0.794 and EH@10 from 0.930 to 0.974. P@1 then stays at 0.789 through c=80, while EH@10 reaches 0.982 at c=40 and 0.991 at c=80.

Under the code’s matching rule, those four questions are cases where reranking moved a chunk from the correct PDF into first place. That is a real change in the ranking. It isn’t evidence that the reranker found the annotated passage more often, and with two PDFs and four questions it is a thin signal either way. The manuscript reads c=20 as the knee of the curve. On these scores that’s where the reported numbers stop moving; whether passage-level quality behaves the same way is what a chunk-level re-run would show.

The cheap baseline belongs in the same table

As reported, BM25 alone reached the same EvidenceHit@10 as the reranked pipeline, 0.974, in 6 ms instead of 901. Read with the scoring note, that is a narrower statement than it looks. It means that for the same share of questions, at least one of the top ten chunks came from the right PDF, and 20 of the 114 count as hits automatically. It does not show BM25 finding the evidence passage as often as the reranker. What does survive is the method point: the fastest configuration was measured at the same n, under the same rule, and it was competitive on the coverage score. “Hybrid with reranking wins everything” isn’t what the table says, even before the caveat.

The quality problem starts upstream

Separately from the retrieval benchmark, the quality layer scored 11 documents (2,432 pages) on 18 deterministic page features: character, token and line statistics, OCR-confusion patterns and a bigram-entropy proxy. Each page got one of five classes. No trained model, no GPU, no network.

89.56% of pages came out digital_clean. Only 2 of the 11 documents did. The denominators differ, and both numbers are true: scan-derived documents with readable OCR pass page-level checks while sitting at a systematically lower mean quality score.

The document-level rule was calibrated on a four-document, human-labelled pilot. A single-threshold rule agreed with the human labels on 2 of 4 documents; the three-guard rule agreed on all 4. That’s a useful sanity check, and the margin it rests on is narrow: mean quality 0.780 for the best scan-derived document, 0.787 for the weakest born-digital one.

Manuscript Table I. Calibration pilot, four human-labelled documents. q̄ = mean page quality score.

• Passage-level retrieval quality. The reported scores match relevance at the document level and give unanswerable questions automatic EvidenceHit@10 credit (see the scoring note). They don’t show how often the annotated passage was retrieved.
• Independent verification. Every number here is reported by the authors from the repository outputs and the manuscript. None was re-run for this page.
• Answer quality. Nothing here measures generated-answer accuracy, hallucination or faithfulness. No answers were generated.
• Generality. 114 templated questions over two PDFs is enough to compare these configurations against each other. It is not enough to rank retrieval methods in general, and legal, medical or multilingual corpora may behave differently.
• Statistical certainty. The P@1 gain is four questions. The repository includes a bootstrap confidence-interval script, but no intervals are reported for these results.
• Cause and effect. Retrieval metrics were not broken down by document quality class, so this work does not show that low-quality pages caused specific misses.
• Hardware. “Commodity CPU”, without a model number. A GPU would move the latency curve, and possibly the knee.

Read the scorer, not just the scores. A metric’s name says what it was meant to measure; the code says what it does. On this project the two came apart, and the gap only shows up in the source.

Keep the cheap baseline in the same table, at the same n. The most useful comparison here is one that could easily have been left out.

Report latency as a distribution. The mean said “under a second”; the P95 said “not always”. Both belong on the page.

Pages and documents answer different questions. A corpus can be mostly clean page by page and still mostly mixed document by document.

Next

• Re-score at the chunk level, using the chunk_id already stored in every evidence record, and report unanswerable questions separately instead of folding them into EvidenceHit@10. This is tracked as its own piece of work; the numbers on this page will change only when it’s done and reviewed.
• Break retrieval metrics down by document quality class, which is the manuscript’s own stated next step.
• Report bootstrap intervals alongside every row of the main table.
• Try query-adaptive candidate counts: rerank deeply only when the first stage is unsure.
• Add human-written questions with exact evidence spans next to the templated set.
• Only then evaluate generated answers end to end.

Sources: the co-authored manuscript (Tables I–III, §III–VIII) and the public repository at commit ce5909e. Figures are redrawn from those numbers; none are screenshots. Retrieval scores are reproduced as originally reported and were not re-run.