Retrieval Semantics: A Democracy of Queries (RAG-Fusion and RRF)

Last time, a poorly built retrieval layer could condemn your RAG system to an execution by hanging. This time, we are holding an election (and we are all hoping for no hanging chads).

Instead of trusting one query to find the right context, RAG-Fusion asks the question several different ways, lets each version cast its own ranked ballot of documents, and then relies on Reciprocal Rank Fusion (RRF) to count the votes. Documents that keep showing up near the top across many queries win the seat. Documents that only one query believes in rarely survive the count.

The polls are open!

Okay, Let's Backtrack...

We discovered, in the last article, how that HyDE and Query Rewriting help reduce the danger of irrelevant retrieval.

How that HyDE enables the RAG system to derive a hypothetical answer that closely match the shape of the documents in the knowledge base (or the vector database), which produces more relevant documents when embedded and searched against the vector database.

How that query rewriting involved taking the user's question or prompt and deriving a new question from it that includes useful context improving retrieval. We uncovered how useful this was for an ongoing chat conversation, as it reduced the risks of shallow or incorrect responses.

There were a couple of query rewriting techniques we discussed. One of them required the system to ask a clarifying question. Particularly useful when that is the first user question or prompt within the chat session. The other takes the existing conversation, summarizes it, then use it to create a new query from the user's question that contains more context.

Both techniques lean on the same idea. A retriever fails when it has too few observations about what the user actually wants, and HyDE and query rewriting each supply some. HyDE hands it a hypothetical answer to search with. Query rewriting hands it the context the question was missing.

But look at how much trust we placed in a single thing each time. One hypothetical answer. One rewritten question. Query rewriting is a bet on one rewrite, and when that rewrite is wrong, it is wrong with full confidence, because the retriever has no way of knowing the question it received is not the one the user asked.

So what if we stopped betting on one?

Then there was RAG-Fusion... The "bad boy" that deserved its own article. This article.

What is RAG-Fusion?

In a traditional RAG system, the indexing stage involves the following steps:

Sourcing

There is always a source that acts as the knowledge base for any RAG system.

The source could be anything from documents containing product information, policies, and so much more to sites like documentation sites, but it must be a source of truth. Its contents thoroughly verified.

For any retrieval system to work, this source must be a constant. Of course, information within the source can be updated, but once done, will affect every other step in this stage.

How this source is curated can differ from one RAG system to the next. Oftentimes, it is already available. Most of the use cases of RAG revolve around taking dense source of information like datasets or companies' help centers and making them more accessible to people without having them read or skim through these documents (sometimes using a lot of Ctrl + F that often get them nowhere without exact keywords).

Other times, the source has to be created from other sources through data scraping or data mining to improve the quality of the source. For this, the scope of information that should be in the source has to be defined. Then per several strategies, existing fragments like meeting notes, internal wikis, support tickets, and so much can be curated either by web scraping, leveraging APIs, or downloading. Usually source curation (in the case where it is not readily available) is down by a different data pipeline.

What a RAG system sees at the end is a source, the knowledge base it can then chunk, embed, and store into a vector database.

Chunking

After curating the source, they need to be chunked. Remember that the source is very dense with information. To make this information easier to access, they are chunked into smaller strips or snippets of information using various approaches. These approaches can be divided into two groups:

1. Basic strategies such as fixed-sized chunking, recursive character splitting, and document-based chunking.

   Fixed-sized chunking sets a point-blank character or token limit for each chunk. In this case, the RAG system takes a large corpus of information (simply called a document) and splits it into fixed-size chunks. Assuming I have a 5,000-word document and I split it into 1,000-character chunks. A 5,000-word document typically contains between 25,000 to 32,500 characters. Hence, I could get between 25 to 33 1,000-character chunks from a 5,000-word document.

   Recursive-character splitting sets an hierarchy of separators used to prevent mid-sentence cutoff, which is the main disadvantage of fixed-sized chunking. Separators include paragraphs, lines, and white-spaces. It is usually the recommended starting point for most RAG systems dealing with text content today.

   Document-based chunking uses documents or logical sections (such as PDF pages or Markdown headers) as chunks, which helps preserve full context but reduced retrieval precision for long documents.

2. Advanced and Semantic strategies such as semantic chunking, agentic chunking, and late chunking.

   Based on this article by Unstructured, semantic chunking captures topic shifts regardless of document structure. The way it works is by embedding each sentence or paragraph or a document, and comparing its embedding vector with the next sentence's or paragraph's using similarity algorithms like cosine similarity or Euclidean distance. You can configure a similarity threshold that serve as a classifier that determines where a chunk ends and another begins.

   This article has been in the works for a while, shortly after I published HyDE vs Query Rewriting. Yet, I got to learn about late chunking today, from Jina. In October 2023, Jina launched the first embedding model with an 8k context length (Günther & Xiao, 2024). Before that, long-context embedding didn't make sense at all, hence chunks were kept within a fixed-sized tokens or characters window. Aside from this, there is also the concern that contextual information get diluted or lost when long documents are embedded (Günther & Xiao, 2024). Late chunking works by first embedding the entire text (or as much as it is possibly allowed by the embedding model, 8k tokens in the case of Jina, jina-embeddings-v2-small-en, accommodates up to 10 standard pages of text [Günther & Xiao, 2024]), then combine some sequence of token embeddings (or vector representation of tokens) into chunks.

Embedding

Embedding is a way to represent text as dense vectors in high dimensional spaces (Kumar, 2024). The sole purpose of embedding is to capture the semantic meaning of the text in a way that phrases or sentences with similar meaning stay close to one another in the vector space.

Retrieval Stage...

Beyond the indexing stage (which helps source for information for the knowledge base), the retrieval and generation stages help answer a user's question. The retrieval stage involves embedding the user query, checking the query vector against all store chunks stored in a vector database like Chroma, Pinecone, and Qdrant, and ranking the chunks based on similarity score.

The retrieval stage looks quite simple and it really is, until your RAG system is deployed to production and it frantically tries to keep up with users' needs. A safe fallback, whenever a question is asked that has no relevant chunks, could be to let them know that there was no answer to their question. But...

• Is that true or merely a consequence of a poorly handled edge case?
• What if the answer to the user's question is present in the knowledge base but retrieval missed it?
• How much context does the user have to omit before your RAG system gives up?
• Should your RAG system ever give up?

It is usually not easy to predict when a user question's would lack enough context to be answerable in one go, but it is easier to prepare for it, by simply upgrading from a traditional RAG system to a more robust RAG system.

Introducing RAG-Fusion, a query rewrite techique.

With RAG-Fusion, the user's question is sent to the LLM to generate a number of new queries (or questions) based on the original question with the intention of capturing as much contextual dimensions as possible to unlock more relevant chunks with depth that could have been missed by the original question (Rackauckas, 2024).

Since there are a number of new queries generated (usually between 3 and 5), each of them must pass through the traditional retrieval flow: query -> embedding -> similarity search -> ranking. Each is casting its own ranked ballot of documents and relying on RRF to count the votes.

So, instead of the user query determining what chunks are sent to the generation pipeline, different variants of the user query decide on what the relevant chunks should be and then wait for RRF (see it as INEC) to decide what chunks truly matter across the queries.

Here is the whole journey, from question to answer:

1. The user asks a question.
2. An LLM writes several variants of it. I keep the original question in the mix, so the user’s own words always get a ballot too. In Rackauckas’s (2024) example, a request like “Tell me about MEMs microphones” turns into questions about what they are, how they work, and what advantages they offer.
3. Each query goes through retrieval on its own: embed it, search the vector database, rank the chunks. Every query returns one ballot, a ranked list of chunks.
4. RRF counts the ballots and produces a single final ranking.
5. The top chunks go to the LLM, which writes the answer. In Rackauckas’s setup, the model receives the fused results together with the generated queries and the original query (Rackauckas, 2024).

Noticed the difference?

In a regular election, a citizen can only place one vote. Here, each citizen of the user query (each generated variant) must place a ballot of documents as vote. Basically, each is agreeing to the chunks that came up as relevant and marking them as their winners. Reciprocal Rank Fusion (RRF) is the judge that decides whether the documents (or chunks) they suggest are worthy of the positive testimonials.

At this point, I feel like I am repeating myself. But I believe there is power in repetition. I hope you can pause now and think about it. If you do this deeply, a question should surface in your subconscious:

Why Can’t We Just Merge the Ballots?

One important note you should keep closer to your heart than your yesterday's meals is that each generated variant of the user query proposes a ballot, not just a single vote.

During a presidential election, like the one coming up in January 2027, all citizens have polling units in their various states, where they go to vote. While each citizen is an important unit, they do not necessarily matter in the bigger abstraction. What matters is the number of votes within each state.

It is true that without the citizens, there wouldn't be a state. Yet they are abstracted away. INEC does not really care about who the voters are (beyond that they are qualified citizens), but they care about how many of them voted for that state.

This is a great analogy of what happens in Rag-Fusion. Each query proposes a ballot of chunks that are certified to be relevant for that query by the retrieval engine. All queries must propose their respective ballots.

In an election, the results across all 36 states are merged together by counting and summing. But for Rag-Fusion, we don't just merge the ballots. Why?

No two citizens are ever the same. Even if they are identical twins. Hence, in an election, there is no duplication danger, where a voter is counted twice. If a voter is in Lagos, they cannot be in Enugu. Hence, you simply (or not so simply) need to count the number of voters per state, sum it up with the counts from other states, and share the results per presidential candidate (or party).

This is not the case with Rag-Fusion. Because a query is suggesting a ballot of relevant chunks, these chunks have a tendency of reappearing in another query's ballot, simply because they were also relevant to answer that query too.

So when the ballots come in, they overlap. One chunk may sit on three ballots, another on one, a few on two. A naive merge runs into three problems.

The duplicates. A chunk on four ballots lands in your context four times. You pay for those tokens, and the model reads the same paragraph repeated like a rumor at a naming ceremony.

The scores. Each ballot comes with similarity scores, and it is tempting to add or average them. But a score tends to say as much about the query’s wording as about the chunk. A short, vague variant may never score above 0.6, even when its top result is exactly right, while a long, specific one can hand out 0.9 to a so-so match. Averaging them is like averaging exam results from two teachers, one stingy and one generous.

The mess that follows. What do we do with a chunk that is missing from a ballot? Give it a zero? Ignore it? Either way we have made a quiet decision, and missing from a top-ten list does not mean irrelevant. It may simply have missed the cut.

So we need a fair way to combine rankings without trusting the raw scores.

What is Reciprocal Rank Fusion?

In plain words: every document earns 1 divided by (k + its rank) from each ballot it appears on, and the earnings are added up. A document missing from a ballot earns nothing from it.

score = sum of 1 / (k + rank), across every ballot the document appears on

The intuition is that RRF counts position, not how loudly the retriever shouted. INEC collates the number of votes. It does not average how loudly each polling unit celebrated.

The usual default is k = 60. The original paper fixed it there during a pilot and never changed it (Cormack et al., 2009), and most production systems ship it as their default (BigDataBoutique, n.d.). Ranks start at one, so first place earns 1/61.

One fun detail: that original paper is titled "Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods" (Cormack et al., 2009). Condorcet is a voting method, so the election metaphor was never really ours.

Let’s Count the Votes

Three ballots, k = 60:

Document	Ballot 1	Ballot 2	Ballot 3	Score
A	rank 2	rank 2	rank 2	0.0484
B	rank 3	rank 1	absent	0.0323
C	rank 1	absent	absent	0.0164

Without telling you the actual scores per rank, you should be able to decipher it using the formula above. But I will make it plainer. The count, step by step:

• A: 1/62 + 1/62 + 1/62 = 0.0484
• B: 1/63 + 1/61 = 0.0159 + 0.0164 = 0.0323
• C: 1/61 = 0.0164

A never came first on a single list, and A still wins the election. C topped Ballot 1 and B topped Ballot 2, yet neither could catch the document that showed up everywhere. A wins by showing up consistently.

What Does k Actually Do?

The constant k decides how much the top spot is worth. A small k rewards top ranks heavily. A large k flattens the differences between ranks, so the number of ballots a document appears on starts to matter more.

Here is the same contest at two settings:

Document	Where it appears	Score at k = 1	Score at k = 60
X	rank 1 in one list only	0.5000	0.0164
Y	rank 10 in three lists	0.2727	0.0429

At k = 1, X wins on the strength of one first place. At k = 60, Y wins by turning up three times.

This is where I need to correct something I said earlier. I told you a high rank in a few lists can beat a mediocre rank in many. With the default k = 60 it is usually the opposite: appearing across many lists tends to outweigh a single first place. In fact, if your ballots only hold the top 10 or 20 chunks, any document that makes two ballots earns more than a lone first place. Only a small k favors top ranks that strongly. Cormack et al. (2009) found that k = 60 was near optimal but that the exact choice was not critical.

Is RAG-Fusion Situational Awareness?

Last time we said retrieval fails when it lacks observations. HyDE and query rewriting each add one. A single rewrite is a single glance at the situation, and a single glance can be wrong.

RAG-Fusion looks several times, from several angles, and then asks what survives across the glances. That is what Rackauckas (2024) credits for the quality of the answers: the generated queries contextualize the original from different perspectives. Think of asking three danfo conductors for directions and trusting the two who agree. A wrong rewrite now has to survive being outvoted. Consensus reduces your dependence on any single observation, though it does not remove it.

Where Does RAG-Fusion Fail Us?

Every election has its failures. Here are four.

It is slower and costlier. Several LLM calls and several retrievals happen before any answer begins. The searches can run in parallel, but the variants must be written first, and every extra query adds a search.

Bad variants vote too. Rackauckas (2024) found that some answers strayed off topic when the generated queries were not relevant enough to the original. RRF softens this, but the flattening from section 6 cuts both ways. An off-target document that lands on a couple of ballots can outrank a lone excellent chunk.

A popular document is not necessarily a correct one. Here your democracy metaphor gives us "tyranny of the majority." The variants are not independent citizens, since the same LLM wrote them from the same question. If it misreads the question, every variant misreads it together and the majority is confidently wrong. Generic overview chunks also match almost everything, so they turn up on every ballot just because they are popular.

Rank-only fusion throws away confidence. A weak match at rank 1 earns exactly what a strong match at rank 1 earns. If every ballot’s best result is weak, RRF still crowns a winner. One fix worth trying is dropping matches below a similarity threshold before the count.

HyDE, Query Rewriting, or RAG-Fusion?

Extend last article’s test, "why did retrieval fail?", to all three:

Technique	Retrieval failed because...	What it adds	Cost	Main risk
HyDE	The question does not look like the documents that answer it	A hypothetical answer to search with	One LLM call	A confidently wrong hypothetical steers the search
Query Rewriting	The question is missing context, as in follow-ups in a chat	A rewritten, context-rich question	One LLM call	One rewrite, and a wrong one fails with confidence
RAG-Fusion	One phrasing is a gamble	Several phrasings and a fair count	Several LLM calls and several retrievals	Bad variants vote too

A bonus point: RRF is not only for rewrites. The original paper fused the rankings of many different retrieval systems, not many phrasings of one question (Cormack et al., 2009). Today it is also a standard way to fuse keyword and vector search in hybrid setups, where the two scores live on different scales and only the ranks can be compared (BigDataBoutique, n.d.).

Closing

We counted every ballot, and nobody needed a magnifying glass for a hanging chad. The user’s question no longer decides alone. A small group of versions of it votes, and RRF makes sure a document that keeps showing up near the top beats one that was merely loud once.

But a fair count is not the same as a correct result. Our winner is the most agreed-upon chunk, and agreement can be wrong. In the next article titled Retrieval Semantics: Rerankers, the Election Tribunal of RAG," we meet the judge who reads the top candidates properly before the result is declared.

Till then, think about the different ways the Nigerian presidential candidates could rig 2027 election if we were using RRF.

References

BigDataBoutique. (n.d.). Reciprocal rank fusion (RRF): How it works and when to use it. https://bigdataboutique.com/blog/reciprocal-rank-fusion-how-it-works-and-when-to-use-it

Cormack, G. V., Clarke, C. L. A., & Büttcher, S. (2009). Reciprocal rank fusion outperforms Condorcet and individual rank learning methods. Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval. https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf

Günther, M., & Xiao, H. (2024, August 22). Late chunking in long-context embedding models. Jina AI. https://jina.ai/news/late-chunking-in-long-context-embedding-models/

Kumar, S. (2024, August 18). Mastering RAG: A deep dive into embeddings. Medium. https://medium.com/@shravankoninti/mastering-rag-a-deep-dive-into-embeddings-b78782aa1259

Rackauckas, Z. (2024). RAG-Fusion: A new take on retrieval-augmented generation. International Journal on Natural Language Computing, 13(1), 37–47. https://doi.org/10.5121/ijnlc.2024.13103