A walkthrough of how the visual-rag demo retrieves pages from the Norwegian Government Pension Fund Global (GPFG) report corpus, using ColPali's per-patch embeddings, BM25 over OCR'd page text, and Gemini-generated query expansions — all served from a single Vespa application with a binary HNSW index in the recall tier and full-precision MaxSim in the rerank tier.
~4,670 individual pages, drawn from the GPFG's annual reports, quarterly updates and policy documents (2010-2024). Each page lives in Vespa as one document carrying: the rendered page image (full and blurred for fast load), the raw single-page PDF, the OCR'd text, and the bit-packed ColPali multi-vector — 1,024 patches × 128 dimensions each, quantised to int8 for the on-disk attribute and binarised (16 bytes per patch) for HNSW recall over hamming distance.
Classic OCR+BM25 misses anything that lives in the layout — charts, infoboxes, tables, the visual rhythm of a page. A single CLIP-style embedding flattens the page back to one token, which throws away the spatial structure that often makes a page recognisable in the first place. ColPali embeds each ~128×128 patch separately; ranking ("MaxSim") is the sum, over query tokens, of each token's best match across the page's patches. So a query like "equity allocation by region 2019" can match the chart title in one patch and the bar-graph axis label in another, even if the OCR happened to miss the title text entirely.
Computing MaxSim over float tensors on every candidate is prohibitively expensive (~6 ms per doc on a warm cache, × tens of thousands of candidates). The fix is a binary first-phase: each patch's 128-dim vector is sign-quantised to a 16-byte signature, indexed in HNSW with hamming distance. Recall is rough but cheap (~0.4 ms per doc); the top 10 are then re-ranked with the full float MaxSim. With BM25 mixed in at the second phase for the 'hybrid' profile, the gap to a flat full-precision MaxSim collapses to within recall@10 noise.
During ingestion every page also gets three Gemini-generated search queries written for it: a broad-topical, a specific-detail and a visual-element query. These power the suggestion chips on the search page (so a cold-start user has something to click) and they're also what the eval harness uses as ground-truth queries. Storing them inline keeps the suggestion endpoint a single Vespa fetch rather than a Gemini round-trip per session.
Recall@K averaged over a sampled validation set; see /eval for the full table and the raw JSON.
| Rank profile | R@1 | R@5 | R@10 | R@20 |
|---|---|---|---|---|
bm25 | — | — | — | — |
colpali | — | — | — | — |
hybrid | — | — | — | — |
broad_topical_query field generated for each page at ingest time. That biases absolute recall upwards (the query was derived from the page) but doesn't affect relative ordering of the rank profiles.max-links-per-node=32, neighbors-to-explore-at-insert=400 over hamming distance, matching the Vespa ColPali scaling guide's defaults.