About the demo

This demo showcases using Vespa for performing combined textual and visual search through a large database of PDF documents. It uses a combination of textual search using the BM25 ranking algorithm along with vector similarity search of image embeddings generated using the ColPali vision model.

• JavaScript web application: This is the website you are currently visiting. It is responsible for providing a user interface, and visualization of the matching documents.
• Python Backend: Provides a HTTP API the web application can speak to. It is responsible for generating the token similarity (how relevant each token is in the PDF page) and querying both Vespa and Google Gemini for results.
• Vespa Cloud: Hosts our Vespa document store which has been fed the 2770 documents that you can search through. Vespa Cloud is the easiest way to get started with Vespa, and we provide a free $300 in credits to new customers.
• Google Gemini: Is used to demonstrate a real-world retrieval augmented generation (RAG) workflow. Based on the documents ranked by Vespa, the generative AI model is capable of providing more precise answers.

Document ranking

Vespa enables the developer to fully customize how the documents are ranked, allowing you to define ranking functions that fit your use-case perfectly. In this demo we have selected three ranking functions; Okapi BM25 for textual ranking, max-similarity of vector embeddings, and a combination of the two.

Dataset

This demo uses open data from the United Nations. The materials are available under the Creative Commons Attribution 3.0 IGO license. Full license terms are available at the United Nations HDR website.