Semantic plus keyword: how we rank passages

Every document-QA system lives or dies on one question: given what the user asked, which handful of passages should the model read? Get that wrong and no amount of model quality saves the answer. Here's how AskMyDocs does it, in plain engineering terms.

Two searches, because queries come in two kinds

Ask "how long do I have to hand back the flat?" and the answer lives in a passage about "notice of intention to vacate" — no shared words at all. That's what semantic search is for: passages are embedded as vectors, and nearness in that space captures meaning, not spelling.

But ask for invoice INV-2024-0117 and semantic nearness is useless — you need the passage containing exactly that token. That's classic keyword (full-text) search. Neither search alone covers both kinds of question, so every query runs both.

Fusing the two lists

The two searches return two differently-ordered lists. We merge them by rank rather than by score — a passage that shows up near the top of either list rises, and one that appears in both rises further. Rank-based fusion sidesteps the apples-to-oranges problem of comparing a cosine similarity to a text-match score.

Re-ranking the survivors

Fusion is cheap and a little naive, so the fused shortlist gets a second, harder look: a re-ranking model reads the actual question against each candidate passage and scores how well it really answers. This is where near-misses die — the passage that mentions your keywords but answers a different question drops out here.

Re-ranking also gives us a relevance gate. If nothing clears the bar, AskMyDocs doesn't pad an answer from the least-bad matches — it tells you it found nothing relevant. An honest empty result is a feature; admins even get a "knowledge gaps" list built from these misses.

Permissions run inside all of this

One design decision ties it together: access control is applied inside the candidate search, before either list is cut off. Documents you can't see aren't candidates in the vector arm, the keyword arm, or the re-rank — so a restricted passage can't crowd out an accessible one, and can't leak into your answer by any path.

What you see at the end

The handful of passages that survive all of the above are exactly what the model is allowed to answer from — and exactly what your numbered citations point to. The ranking machinery is invisible; the receipts aren't.