The topics

1. Intro to databases
2. Embeddings
3. FAISS
4. SQLite3
5. POstgresSQL pygvector
6. SQLite and Ollama
7. Complete RAG system app
8. Vector Query Language.

shiny and nice topics.

Vector database Book Review.

When you read the book you would feel it’s not connected with each other in a nice way. it’s like listing some information from GitHub pages without the sense of teaching.

a lot of chapters listing info that is specific to Project which if i need it i will read GitHub page.

The Charts are not that helpful they don’t add more value just an overview it’s more like AI generated one. It maybe not AI but not much helpful.

You feel there is a gab between the depth of knowledge and the titles. shiny titles.

I hoped it was more detailed more to concepts with nice explaining that works with any vector database.

• Example the FAISS chapter :
  • it gives all the indexing types in FAISS and quick overview about quantization that is bad then benefit and trade offs this is style of book!

My Recommendation.

Any Vector database Articles are far helpful and much better like :

Quarto Blogs.

Weaviate

What is HyperRun ?

HyperRun lets developers add a semantic code search chat widget to their docs.

Built on ColGrep for indexing, Lisette for LLM orchestration, and FastHTML for the UI.

Why ColGrep

Most coding agents still use grep to search codebases.

It works — but it’s pure pattern matching.

If you don’t know the exact function name, you’re stuck guessing.

Semantic search (RAG) solves this but introduces problems:

• Requires remote storage of your code (security concern)

• Needs a separate vector DB service running

• Keeping the index in sync with a fast-moving codebase is hard

ColGrep takes a completely different approach. It’s a Rust CLI tool built by LightOn that:

• Mirrors the grep interface — agents already know how to use it

• Runs entirely locally — no remote storage, no separate API service

• Uses late-interaction retrieval (ColBERT-style) via LateOn-Code models — not traditional embeddings

  • We are using their specialized coding retrieval model 17M

• Supports hybrid queries — regex filtering first, then semantic ranking on the filtered results

• Incremental index updates — only re-indexes changed files, not the whole repo

  • They are using a nice hashing trick 😎

• Tree-sitter parsing — extracts functions, classes, signatures, call graphs — not just raw text chunks

The results speak for themselves:

• Won 70% of head-to-head comparisons against vanilla grep with Claude Code

• Cut token usage by 15.7% on average

• 56% fewer search operations needed to find the right code

• Hard conceptual questions (where you describe behavior, not function names) benefit the most

Traditional embeddings compress an entire code block into a single vector — losing detail.

Late-interaction models (like ColBERT) keep per-token vectors, so they can do soft matching between query terms and code tokens.

This is why ColGrep handles “find the function that inserts articles into the database” better than a single-vector approach.

For HyperRun, ColGrep is the retrieval layer — it finds the relevant code semantically, then Lisette sends it to the LLM for a conversational answer.

How It Works

HyperRun has two sides — the Doc Author who sets it up, and the End User who asks questions.

<script src="https://your-server/embed"></script>

What’s Next

HyperRun is still early — here’s what I’m working on:

• Citations — show which code snippets the LLM used to answer, so users can verify

• GitHub Actions — auto re-index on push, auto deploy the server — zero manual steps

• Better nbdev support — ColGrep struggles with .ipynb files, so we index the exported .py package for now

• Secure BYOK — move API keys from session cookies to server-side encrypted storage

• Switching embedding models — let doc authors choose which ColGrep model to use

For a deep dive into the streaming UI architecture, see HyperRun Deep Dive: FastHTML, HTMX, and SSE.

If you want to try it or contribute: github.com/abdelkareemkobo/hyperrun

FastHTML App Setup

  User types "what does insert_article do?"
        │
        ▼
  ┌──────────┐  POST /ask   ┌──────────────┐
  │  Browser │────────────►│  FastHTML     │
  │  (HTMX)  │             │  /ask route   │
  └──────────┘              └──────┬───────┘
        ▲                          │
        │                          ▼
        │                   ┌──────────────┐
        │                   │ Returns:     │
        │                   │ • user bubble│
        │                   │ • ai bubble  │
        │                   │   (with SSE) │
        │                   │ • new input  │
        │                   │   (OOB swap) │
        │                   └──────┬───────┘
        │                          │
        │  SSE connects to         │
        │  /stream?query=...       ▼
        │                   ┌──────────────┐
        │  sse: message     │  /stream     │
        │◄──────────────────│  async gen   │
        │  sse: message     │      │       │
        │◄──────────────────│      ▼       │
        │  sse: message     │  ┌────────┐  │
        │◄──────────────────│  │ColGrep │  │
        │  sse: close       │  │search  │  │
        │◄──────────────────│  └───┬────┘  │
        │                   │      │       │
        ▼                   │      ▼       │
    marked.parse()          │  ┌────────┐  │
    renders markdown        │  │Lisette │  │
                            │  │AsyncChat│ │
                            │  │+ tools │  │
                            │  └────────┘  │
                            └──────────────┘

Setting up the app means picking your CSS framework and loading the right headers:

from fasthtml.common import *
from fhdaisy import *
from fastlucide import SvgSprites, SvgStyle

icons = SvgSprites()

app, rt = fast_app(
    pico=False,
    hdrs=(
        *daisy_hdrs,
        SvgStyle(),
        CHAT_CSS,
        Script(src="https://unpkg.com/htmx-ext-sse@2.2.3/sse.js"),
        Script(src="https://cdn.jsdelivr.net/npm/marked/marked.min.js"),
    ))

A few things to note:

• pico=False — FastHTML includes PicoCSS by default, we swap it for DaisyUI
• daisy_hdrs — from fhdaisy, loads Tailwind + DaisyUI via CDN
• SvgSprites() — from fastlucide, gives us Lucide icons as inline SVGs
• SSE extension and marked.js are the only external JS we load

That’s the whole frontend stack.

No bundler needed.

The Chat Widget Architecture

def chat_widget():
    return Div(
        icons,
        Label(
            Span(cls="inline-flex items-center gap-2")(
                icons("message-circle", sz=20),
                Span("Ask AI", cls="font-semibold tracking-tight")),
            fr="chat-modal",
            cls="btn btn-primary rounded-full fixed bottom-6 right-6 z-50"),
        Input(type="checkbox", id="chat-modal", cls="modal-toggle"),
        Div(cls="modal")(
            chat_content(),
            Label(cls="modal-backdrop", fr="chat-modal")))

def chat_content(standalone=False):
    box_cls = "w-full h-screen p-0 bg-base-100" if standalone else "modal-box w-11/12 max-w-4xl p-0 ..."
    return Div(
        # header, messages, input form
        cls=box_cls)

def user_bubble(msg):
    return Div(cls="chat chat-end")(
        Div("You", cls="chat-header"),
        Div(msg, cls="chat-bubble chat-bubble-primary"))

Streaming with SSE + HTMX

**Streaming Life Cycle**

  Browser (HTMX)                          Server (FastHTML)
       │                                        │
       │  ── SSE connect /stream ──────────►    │
       │                                        │
       │                              AsyncChat(query, stream=True)
       │                                        │
       │  ◄── sse: message (Span("I'll")) ──    │
       │  ◄── sse: message (Span(" search"))──  │
       │  ◄── sse: message (Span(" the")) ──    │
       │  ◄── sse: message (Span(" code"))──    │
       │           ...                          │
       │  ◄── sse: message (Span("...")) ──     │
       │  ◄── sse: close (Span("")) ────────    │
       │                                        │
       │  disconnect                            │
       │                                        │
       ▼                                        
  hx_on__sse_close fires:
  this.innerHTML = marked.parse(this.textContent)
  
  Raw text ──► Rendered markdown HTML

Why SSE over WebSockets

Chat is one-directional — the server streams tokens to the client.

WebSockets are bidirectional, which is overkill here.

SSE is:

• Built into browsers natively
• Supported by HTMX via a small extension
• Dead simple in FastHTML — just return an EventStream

def ai_bubble_stream(query):
    return Div(
        Span(cls="loading loading-dots loading-sm"),
        hx_ext="sse",
        sse_connect=stream_response.to(query=query),
        sse_swap="message",
        sse_close="close",
        hx_swap="beforeend",
        hx_on__sse_close="this.innerHTML=marked.parse(this.textContent)",
        cls="chat-bubble bg-base-200")

@rt("/stream")
async def stream_response(query: str, sess):
    chat = chats[sid]

    async def generate():
        async for chunk in await chat(query, stream=True):
            text = chunk.choices[0].delta.content
            if text:
                yield sse_message(Span(text))
        yield sse_message(Span(""), event="close")

    return EventStream(generate())

HTMX Patterns We Used

@rt("/ask")
def ask(query: str, sess):
    new_inp = Input(id="query", placeholder="Ask a follow-up question...",
                    hx_swap_oob="true")
    return Div(user_bubble(query), ai_bubble_stream(query)), new_inp

Form(cls="join w-full", hx_post=ask, hx_target="#messages", hx_swap="beforeend")(
    Input(id="query", name="query"),
    Button(icons("send", sz=18)))

@rt("/stream")
async def stream_response(query: str, sess):
    model = sess.get("model", "claude-sonnet-4-20250514")
    api_key = sess.get("api_key", None)

The Embed Trick

@rt("/embed")
def embed_js(req):
    server = f"{req.url.scheme}://{req.url.netloc}"
    js = f"""document.addEventListener('DOMContentLoaded',function(){{
      // creates floating button + iframe pointing to {server}/chat-standalone
    }});"""
    return Response(js, media_type='application/javascript; charset=utf-8')

@rt("/chat-standalone")
def chat_standalone():
    return Html(
        Head(*daisy_hdrs, SvgStyle(), CHAT_CSS,
             Script(src="<https://unpkg.com/htmx.org>"),
             Script(src="<https://unpkg.com/htmx-ext-sse@2.2.3/sse.js>"),
             Script(src="<https://cdn.jsdelivr.net/npm/marked/marked.min.js>")),
        Body(icons, chat_content(standalone=True)))

window.addEventListener('message', function(e) {
    if (e.data === 'hyperrun-close')
        document.getElementById('hr-frame').style.display = 'none'
});

Lessons Learned Using Streaming HTMX

Building HyperRun was mostly smooth — but a few things bit us.

Here’s what to watch for.

yield sse_message(Span(""), event="close")

Input(cls="modal-toggle")  # ✓ checkbox
Label(...)                  # ✗ this breaks the sibling selector
Div(cls="modal")(...)       # modal never opens

Label(...)                  # button can go anywhere before
Input(cls="modal-toggle")  # ✓ immediately before modal
Div(cls="modal")(...)       # ✓ works

Final Thoughts

The stack — FastHTML + HTMX + SSE + DaisyUI — turned out to be surprisingly productive for building a real-time chat widget. No build step, no React, no client-side state management. Just Python functions returning HTML.

You can read more about other related blogs here:

1. HyperRun

2. SeoRat

3. BM25 Search

4. BM25 Explained

5. BM25 Benchmark

Dataset Benchmark

I will continue exploring the bm25 and how to use with a huggingface dataset with multiple implementations and compare their speed

The methods i will use:

1. Normal Python Calculations
2. Spicy sparse matrix with python loops
3. Spicy sparse matrix with Numpy Vectorization
4. Rank_bm25
5. bm25-sparse
6. Polars optimization

If there is any mistake please tell me!

it’s first time for me to create a benchmark for anything

from datasets import load_dataset
import time
import random

# Load dataset
dataset = load_dataset("rotten_tomatoes", split="train")
corpus = [item["text"] for item in dataset]
print(f"Corpus size: {len(corpus)} reviews")


def generate_random_queries(corpus, num_queries=1000):
    queries = []
    for _ in range(num_queries):
        doc = random.choice(corpus)
        words = tokenize(doc)
        num_words = random.randint(2, 4)
        if len(words) >= num_words:
            query_words = random.sample(words, num_words)
        else:
            query_words = words
        queries.append(" ".join(query_words))
    return queries


test_queries = generate_random_queries(corpus, 1000)
print(f"Generated {len(test_queries)} test queries")

Corpus size: 8530 reviews
Generated 1000 test queries

def benchmark_python_loops():
    print("=== 1. NORMAL PYTHON (LOOPS) ===")
    start = time.time()

    # Tokenize all documents
    all_docs_terms_py = []
    for doc in corpus:
        tokens = tokenize(doc)
        all_docs_terms_py.append(Counter(tokens))

    # Compute DF
    df_py = {}
    for doc_tf in all_docs_terms_py:
        for term in doc_tf.keys():
            df_py[term] = df_py.get(term, 0) + 1

    # Average doc length
    total_len = sum(sum(doc_tf.values()) for doc_tf in all_docs_terms_py)
    avg_doc_len_py = total_len / len(corpus)

    # IDF function
    def compute_idf_py(term, df_dict, N):
        if term not in df_dict:
            return 0
        return np.log((N - df_dict[term] + 0.5) / (df_dict[term] + 0.5))

    # BM25 function
    def bm25_score_py(term, doc_tf, idf_val, doc_len, avg_len, k1=1.5, b=0.75):
        tf = doc_tf.get(term, 0)
        if tf == 0:
            return 0
        return idf_val * tf * (k1 + 1) / (tf + k1 * (1 - b + b * (doc_len / avg_len)))

    index_time = time.time() - start

    # Query
    start_query = time.time()
    for query in test_queries:
        query_tokens = tokenize(query)
        doc_scores = [0] * len(corpus)
        for token in query_tokens:
            if token in df_py:
                idf_val = compute_idf_py(token, df_py, len(corpus))
                for doc_idx, doc_tf in enumerate(all_docs_terms_py):
                    doc_len = sum(doc_tf.values())
                    doc_scores[doc_idx] += bm25_score_py(
                        token, doc_tf, idf_val, doc_len, avg_doc_len_py
                    )
        top_k = sorted(
            range(len(doc_scores)), key=lambda i: doc_scores[i], reverse=True
        )[:10]

    query_time = time.time() - start_query
    qps = len(test_queries) / query_time

    print(f"Indexing: {index_time:.4f}s")
    print(f"Query time: {query_time:.4f}s")
    print(f"QPS: {qps:.2f}\n")

    return index_time, query_time, qps


benchmark_python_loops()

=== 1. NORMAL PYTHON (LOOPS) ===
Indexing: 0.0801s
Query time: 23.6460s
QPS: 42.29

(0.08005261421203613, 23.645965099334717, 42.29051323551751)

Why Sparse Matrices?

Right now, to search for “python programming”, you’d need to: 1. Loop through each document 2. Calculate BM25 score for “python” 3. Calculate BM25 score for “programming” 4. Add them up

That’s slow for large datasets!

The key insight: We can pre-compute ALL BM25 scores for ALL words in ALL documents and store them in a matrix. Then searching becomes just looking up rows and adding them.

Here’s what the matrix looks like conceptually:

               Doc0   Doc1   Doc2
python         -0.51  -0.56   0.00
programming    -1.95  -2.14  -1.79
java            0.00   0.00   2.20

def benchmark_sparse_python_loops():
    print("=== 2. SPARSE MATRIX WITH PYTHON LOOPS ===")
    start = time.time()

    # Tokenize
    all_docs_terms_sp = []
    for doc in corpus:
        tokens = tokenize(doc)
        all_docs_terms_sp.append(Counter(tokens))

    # Get all words and create word_to_idx
    all_words_sp = list(set(word for doc in all_docs_terms_sp for word in doc))
    word_to_idx_sp = {word: idx for idx, word in enumerate(all_words_sp)}

    # Compute DF
    df_sp = {}
    for doc_tf in all_docs_terms_sp:
        for term in doc_tf.keys():
            df_sp[term] = df_sp.get(term, 0) + 1

    # Average doc length
    avg_doc_len_sp = sum(sum(doc_tf.values()) for doc_tf in all_docs_terms_sp) / len(
        corpus
    )

    # Build sparse matrix
    rows_idx = []
    cols_idx = []
    data = []

    for word_idx, word in enumerate(all_words_sp):
        idf_val = np.log((len(corpus) - df_sp[word] + 0.5) / (df_sp[word] + 0.5))
        for doc_idx, doc_tf in enumerate(all_docs_terms_sp):
            doc_len = sum(doc_tf.values())
            tf = doc_tf.get(word, 0)
            if tf != 0:
                score = (
                    idf_val
                    * tf
                    * 2.5
                    / (tf + 1.5 * (1 - 0.75 + 0.75 * (doc_len / avg_doc_len_sp)))
                )
                rows_idx.append(word_idx)
                cols_idx.append(doc_idx)
                data.append(score)

    sparse_matrix_sp = csr_matrix(
        (data, (rows_idx, cols_idx)), shape=(len(all_words_sp), len(corpus))
    )

    index_time = time.time() - start

    # Query
    start_query = time.time()
    for query in test_queries:
        query_indices = [
            word_to_idx_sp[word] for word in tokenize(query) if word in word_to_idx_sp
        ]
        if query_indices:
            doc_scores = np.array(
                sparse_matrix_sp[query_indices, :].sum(axis=0)
            ).flatten()
            top_k = np.argsort(doc_scores)[-10:][::-1]

    query_time = time.time() - start_query
    qps = len(test_queries) / query_time

    print(f"Indexing: {index_time:.4f}s")
    print(f"Query time: {query_time:.4f}s")
    print(f"QPS: {qps:.2f}\n")

    return index_time, query_time, qps


benchmark_sparse_python_loops()

=== 2. SPARSE MATRIX WITH PYTHON LOOPS ===
Indexing: 77.9146s
Query time: 0.4709s
QPS: 2123.64

(77.91458225250244, 0.4708900451660156, 2123.6380133019015)

Performance Optimization with Numpy

My code has python loops everywhere. Python loops are very slow. The goal here it to use Numpy Vectorization: doing operations on entiry arrays at once instead of looping :

for word_1 in all_words:
for doc_idx, doc_1 in enumerate(all_docs_terms):
    idf_python_1 = compute_idf(word_1, 3, df)
    doc_len_1 = sum(all_docs_terms[doc_idx].values())
    score_1 = bm25_score(word_1, doc_1, idf_python_1, doc_len_1, avg_doc_len)

This has nested loops => very slow for large datasets. Key Insight: Instead of computing one BM25 score at a time, we can compute all scores at once using matrix opertions.

def benchmark_sparse_numpy():
    print("=== 3. SPARSE MATRIX WITH NUMPY VECTORIZATION ===")
    start = time.time()

    # Tokenize
    all_docs_terms_np = []
    for doc in corpus:
        tokens = tokenize(doc)
        all_docs_terms_np.append(Counter(tokens))

    all_words_np = list(set(word for doc in all_docs_terms_np for word in doc))
    word_to_idx_np = {word: idx for idx, word in enumerate(all_words_np)}

    # Build TF matrix
    rows = []
    cols = []
    tf_data = []

    for doc_idx, doc_tf in enumerate(all_docs_terms_np):
        for word, tf in doc_tf.items():
            rows.append(word_to_idx_np[word])
            cols.append(doc_idx)
            tf_data.append(tf)

    tf_matrix_np = csr_matrix(
        (tf_data, (rows, cols)), shape=(len(all_words_np), len(corpus))
    )

    # Compute DF array
    df_array_np = np.array((tf_matrix_np > 0).sum(axis=1)).flatten()

    # Document lengths
    doc_lengths_np = np.array(tf_matrix_np.sum(axis=0)).flatten()
    avg_doc_len_np = doc_lengths_np.mean()

    # Compute IDF array
    N = len(corpus)
    idf_array_np = np.log((N - df_array_np + 0.5) / (df_array_np + 0.5))

    # Vectorized BM25
    k1 = 1.5
    b = 0.75
    tf_dense = tf_matrix_np.toarray()

    numerator = tf_dense * (k1 + 1)
    length_norm = 1 - b + b * (doc_lengths_np / avg_doc_len_np)
    denominator = tf_dense + k1 * length_norm

    idf_column = idf_array_np.reshape(-1, 1)
    bm25_scores = idf_column * (numerator / (denominator + 1e-10))

    bm25_matrix_np = csr_matrix(bm25_scores)

    index_time = time.time() - start

    # Query
    start_query = time.time()
    for query in test_queries:
        query_indices = [
            word_to_idx_np[word] for word in tokenize(query) if word in word_to_idx_np
        ]
        if query_indices:
            doc_scores = np.array(
                bm25_matrix_np[query_indices, :].sum(axis=0)
            ).flatten()
            top_k = np.argsort(doc_scores)[-10:][::-1]

    query_time = time.time() - start_query
    qps = len(test_queries) / query_time

    print(f"Indexing: {index_time:.4f}s")
    print(f"Query time: {query_time:.4f}s")
    print(f"QPS: {qps:.2f}\n")

    return index_time, query_time, qps


benchmark_sparse_numpy()

=== 3. SPARSE MATRIX WITH NUMPY VECTORIZATION ===
Indexing: 6.9885s
Query time: 0.5250s
QPS: 1904.79

(6.988537311553955, 0.5249927043914795, 1904.7883744577036)

def benchmark_rank_bm25():
    from rank_bm25 import BM25Okapi

    print("=== 4. RANK-BM25 ===")

    tokenized_corpus_rank = [tokenize(doc) for doc in corpus]

    start = time.time()
    bm25_rank = BM25Okapi(tokenized_corpus_rank)
    index_time = time.time() - start

    start_query = time.time()
    for query in test_queries:
        query_tokens = tokenize(query)
        scores = bm25_rank.get_scores(query_tokens)
        top_k = np.argsort(scores)[-10:][::-1]

    query_time = time.time() - start_query
    qps = len(test_queries) / query_time

    print(f"Indexing: {index_time:.4f}s")
    print(f"Query time: {query_time:.4f}s")
    print(f"QPS: {qps:.2f}\n")

    return index_time, query_time, qps


benchmark_rank_bm25()

=== 4. RANK-BM25 ===
Indexing: 0.0813s
Query time: 5.0695s
QPS: 197.26

(0.08132147789001465, 5.069519281387329, 197.2573619892297)

def benchmark_bm25s():
    import bm25s

    print("=== 5. BM25S ===")

    start = time.time()
    retriever_bm25s = bm25s.BM25()
    corpus_tokens_bm25s = bm25s.tokenize(corpus, show_progress=False)
    retriever_bm25s.index(corpus_tokens_bm25s, show_progress=False)
    index_time = time.time() - start

    start_query = time.time()
    for query in test_queries:
        query_tokens_bm25s = bm25s.tokenize(query, show_progress=False)
        results, scores = retriever_bm25s.retrieve(
            query_tokens_bm25s, k=10, show_progress=False
        )

    query_time = time.time() - start_query
    qps = len(test_queries) / query_time

    print(f"Indexing: {index_time:.4f}s")
    print(f"Query time: {query_time:.4f}s")
    print(f"QPS: {qps:.2f}\n")

    return index_time, query_time, qps


benchmark_bm25s()

=== 5. BM25S ===
Indexing: 0.5329s
Query time: 0.7049s
QPS: 1418.74

(0.5328564643859863, 0.7048521041870117, 1418.7373408687158)

def benchmark_polars():
    import polars as pl

    print("=== 6. POLARS OPTIMIZATION ===")
    start = time.time()

    k1 = 1.5
    b = 0.75
    N = len(corpus)

    # Create LazyFrame
    lf = pl.LazyFrame({"doc_id": range(N), "text": corpus})

    # Tokenize and explode
    lf_tokens = (
        lf.with_columns(
            pl.col("text").str.to_lowercase().str.split(" ").alias("tokens")
        )
        .explode("tokens")
        .filter(pl.col("tokens").str.len_chars() > 0)
    )

    # Build vocab
    vocab_df = (
        lf_tokens.select(pl.col("tokens").unique()).collect().with_row_index("word_idx")
    )
    vocab_size = len(vocab_df)

    # Join to map tokens to indices
    lf_tokens = lf_tokens.join(vocab_df.lazy(), on="tokens", how="left")

    # Compute TF
    lf_tf = lf_tokens.group_by(["doc_id", "word_idx"]).agg(pl.len().alias("tf"))

    # Compute document lengths
    lf_doc_lens = lf_tf.group_by("doc_id").agg(pl.col("tf").sum().alias("doc_len"))
    avg_doc_len_pl = lf_doc_lens.select(pl.col("doc_len").mean()).collect().item()

    # Compute DF and IDF
    lf_idf = (
        lf_tf.group_by("word_idx")
        .agg(pl.col("doc_id").n_unique().alias("df"))
        .with_columns(
            ((pl.lit(N) - pl.col("df") + 0.5) / (pl.col("df") + 0.5)).log().alias("idf")
        )
    )

    # Calculate BM25
    df_bm25_pl = (
        lf_tf.join(lf_doc_lens, on="doc_id", how="left")
        .join(lf_idf, on="word_idx", how="left")
        .with_columns(
            (
                pl.col("idf")
                * pl.col("tf")
                * (k1 + 1)
                / (
                    pl.col("tf")
                    + k1 * (1 - b + b * (pl.col("doc_len") / avg_doc_len_pl))
                )
            ).alias("bm25_score")
        )
        .select(["doc_id", "word_idx", "bm25_score"])
        .collect()
    )

    # Build sparse matrix
    rows_pl = df_bm25_pl["word_idx"].to_numpy()
    cols_pl = df_bm25_pl["doc_id"].to_numpy()
    data_pl = df_bm25_pl["bm25_score"].to_numpy()
    word_to_idx_pl = dict(zip(vocab_df["tokens"], vocab_df["word_idx"]))
    bm25_matrix_pl = csr_matrix((data_pl, (rows_pl, cols_pl)), shape=(vocab_size, N))

    index_time = time.time() - start

    # Query
    start_query = time.time()
    for query in test_queries:
        query_indices = [
            word_to_idx_pl[word] for word in tokenize(query) if word in word_to_idx_pl
        ]
        if query_indices:
            doc_scores = np.array(
                bm25_matrix_pl[query_indices, :].sum(axis=0)
            ).flatten()
            top_k = np.argsort(doc_scores)[-10:][::-1]

    query_time = time.time() - start_query
    qps = len(test_queries) / query_time

    print(f"Indexing: {index_time:.4f}s")
    print(f"Query time: {query_time:.4f}s")
    print(f"QPS: {qps:.2f}\n")

    return index_time, query_time, qps


benchmark_polars()

=== 6. POLARS OPTIMIZATION ===
Indexing: 0.1580s
Query time: 0.6143s
QPS: 1627.94

(0.1579906940460205, 0.6142714023590088, 1627.9449053946898)

🏆 BM25 Benchmark Results (8,530 documents, 1000 queries)

Key Findings:

The Problem

When building a real estate search system, I ran into a frustrating issue: dense vector search kept returning the wrong locations. A query for “6th Settlement Apartment” would return properties in “5th Settlement” or “New Cairo” instead.

I was using Gemini’s embedding model, which performs well for both English and Arabic. But even strong embeddings struggle with domain-specific data where names are nearly identical:

• El Patio Vera
• El Patio Solo
• El Patio Casa

These project names fall outside the embedding model’s training distribution.

The semantic similarity between them is too high for dense search to distinguish.

My property chunks contain full descriptions like “Villa in El Patio Vera, 3 bedrooms, 1,000,000 EGP”. Semantic search handles most queries well, but some searches are fundamentally lexical. I needed both approaches working together.

When Dense Search Fails

My dataset contained 12,877 real estate properties across Egypt with location names like:

• 6th Settlement, 5th Settlement
• 6th of October, 5th of October
• Sheikh Zayed, New Zayed
• El Alamein, North Coast

When a user searched for “6th Settlement Apartment less than 10 million”, the dense vector results looked like this:

5th Settlement, New Cairo - 9.5M
5th Settlement, New Cairo - 9.3M
El Alamein, North Coast - 5.6M
5th Settlement, New Cairo - 9.4M
El Alamein, North Coast - 5.7M

Zero results from 6th Settlement. The embedding model treated “6th” and “5th” as semantically similar because they are both ordinal numbers. The word “Settlement” matched in both cases, pushing the semantic similarity even higher. This is exactly where lexical matching would help.

Why Hybrid Search Works

Dense vectors and BM25 have complementary strengths.

Dense vectors understand meaning. They know that “apartment”, “flat”, and “unit” refer to similar things. They handle typos and variations gracefully. But they struggle with exact matches where surface-level differences matter, like distinguishing “6th” from “5th”.

BM25 sparse vectors match tokens exactly. The token “6th” will never match “5th”. This works across languages without additional configuration. The downside is that BM25 has no semantic understanding. It cannot recognize that “apartment” and “flat” mean the same thing.

By combining both approaches, you get semantic understanding when you need it and exact matching when that matters more.

The architecture looks like this:

User Query: "6th Settlement Apartment"
                    |
         +----------+----------+
         |                     |
   Dense Search          BM25 Search
   (100 candidates)      (30 candidates)
         |                     |
         +----------+----------+
                    |
              RRF Fusion
                    |
             Final Results

Reciprocal Rank Fusion combines the rankings from both approaches, giving weight to results that appear highly ranked in either or both lists.

Challenge 1: Reducing Token Noise

The first problem I encountered was tokenization noise. Raw property chunks contained everything: URLs, metadata IDs, field names, and numeric values.

A typical chunk looked like:

"Palm Hills, Palm Hills New Cairo, Apartment, 15.3M EGP,
154 sqm, metadata_id_123, https://example.com/file.pdf, ..."

Tokenizing this produced over 13,000 unique tokens across the corpus. Most of these were useless for search: URL fragments, random IDs, and field names that would never appear in user queries.

The solution was to extract only the fields that users would actually search for:

• Developer name
• Project name
• Unit type
• Location and sublocation

After filtering, a chunk became:

"Palm Hills Palm Hills New Cairo Apartment 5th Settlement New Cairo"

This reduced the vocabulary from 13,000 tokens to 1,941, an 86% reduction. The BM25 index became faster and more accurate because every remaining token was meaningful.

Challenge 2: Handling Stopwords

Common words like “of”, “in”, and “the” added noise to the BM25 scores. Since this was a bilingual system supporting Arabic and English, I needed stopwords for both languages:

STOPWORDS = {
    # English
    'the', 'a', 'an', 'of', 'in', 'on', 'at', 'to', 'for', 'and', 'or',
    # Arabic
    'في', 'من', 'إلى', 'على', 'عن', 'مع', 'هذا', 'هذه', 'ذلك', 'التي', 'الذي'
}

Removing these ensured that queries like “apartment in 6th Settlement” focused on the meaningful tokens rather than matching every document containing “in”.

Challenge 3: N-gram Tokenization

With basic tokenization, similar location names still overlapped too much:

"6th Settlement" → ['6th', 'settlement']
"5th Settlement" → ['5th', 'settlement']

Both share the token “settlement”, so BM25 would give partial credit to 5th Settlement results even when the user explicitly asked for 6th Settlement.

The solution was to add bigrams, two-word phrases joined with an underscore:

def tokenize_ngram(text: str) -> List[str]:
    words = [w for w in text.split() if w not in STOPWORDS]
    tokens = words.copy()
    
    for i in range(len(words) - 1):
        tokens.append(f"{words[i]}_{words[i+1]}")
    
    return tokens

Now the tokenization produces:

"6th Settlement" → ['6th', 'settlement', '6th_settlement']
"5th Settlement" → ['5th', 'settlement', '5th_settlement']

The bigram “6th_settlement” is unique and will only match documents containing that exact phrase. This dramatically improved precision for location-specific queries.

Challenge 4: Data Imbalance

Even with proper tokenization, some brands dominated the results unfairly. My dataset had:

• Palm Hills: 718 properties
• El Patio: 49 properties

When searching for “El Patio apartment”, BM25 would often return Palm Hills results first because the sheer frequency of Palm Hills in the corpus gave it higher term frequency scores.

The solution was to adjust the balance between dense and sparse search in the hybrid approach. By retrieving more candidates from dense search (100) and fewer from BM25 (30), I gave semantic understanding more influence in the final ranking. This allowed the dense embeddings, which correctly understood “El Patio” as a distinct entity, to override BM25’s frequency bias.

RRF (Reciprocal Rank Fusion) Explained

RRF is a method for combining rankings from multiple search systems.

c How It Works

Instead of averaging scores (which can be misleading when different systems use different scales), RRF uses rank positions.

Formula:

RRF_score = Σ (1 / (k + rank))

Where: - k = constant (usually 60) - rank = position in that ranking (1st, 2nd, 3rd…)

Results

After implementing hybrid search with these optimizations, the same query that previously failed now worked correctly.

Query: “6th Settlement Apartment less than 10 million”

Before (Dense Only):

5th Settlement - 9.5M
5th Settlement - 9.3M
El Alamein - 5.6M

After (Hybrid with BM25):

6th Settlement - 8.0M
6th Settlement - 9.5M
6th Settlement - 7.2M

Key Takeaways

1. Filter your tokens aggressively. Remove everything that users would never search for.

2. Use n-grams for multi-word entities. Bigrams turn “6th Settlement” into a unique, matchable token.

3. Handle stopwords in all supported languages. A bilingual system needs bilingual stopword lists.

4. Balance dense and sparse weights based on your data. If one brand dominates your corpus, give more weight to semantic search.

5. Hybrid search is not always necessary. If pure semantic search works for your use case, the added complexity of BM25 may not be worth it. Use hybrid when exact matches matter and when you have domain-specific vocabulary that embeddings handle poorly.

# Step 1: Install dependencies
# pip install qdrant-client rank-bm25

import re
from typing import List
import pickle
from rank_bm25 import BM25Okapi
from qdrant_client import QdrantClient, models

# Step 2: Define tokenizer with stopwords and bigrams
STOPWORDS = {
    "the",
    "a",
    "an",
    "of",
    "in",
    "on",
    "at",
    "to",
    "for",
    "and",
    "or",
    "في",
    "من",
    "إلى",
    "على",
    "عن",
    "مع",
}


def tokenize_ngram(text: str) -> List[str]:
    text = text.lower()
    text = re.sub(r"[^\w\s]", " ", text)
    words = [w for w in text.split() if w not in STOPWORDS]
    tokens = words.copy()
    for i in range(len(words) - 1):
        tokens.append(f"{words[i]}_{words[i + 1]}")
    return tokens

# Step 3: Build BM25 index from your filtered chunks
filtered_chunks = ["palm hills apartment 5th settlement new cairo", ...]  # Your data
tokenized_corpus = [tokenize_ngram(chunk) for chunk in filtered_chunks]
bm25 = BM25Okapi(tokenized_corpus)

# Save for later use
with open("bm25_index.pkl", "wb") as f:
    pickle.dump(bm25, f)

# Step 4: Create hybrid collection in Qdrant
client = QdrantClient(url="your-url", api_key="your-key")

client.create_collection(
    collection_name="hybrid_collection",
    vectors_config={
        "dense": models.VectorParams(size=3072, distance=models.Distance.COSINE)
    },
    sparse_vectors_config={"text-sparse": models.SparseVectorParams()},
)

# Step 5: Convert text to sparse vector using BM25 scores
def text_to_sparse_vector(text: str, bm25_index) -> models.SparseVector:
    tokens = tokenize_ngram(text)
    scores = bm25_index.get_scores(tokens)

    indices = []
    values = []
    for idx, score in enumerate(scores):
        if score > 0:
            indices.append(idx)
            values.append(float(score))

    return models.SparseVector(indices=indices, values=values)


# Step 6: Upload documents with both vectors
for i, (chunk, dense_embedding, metadata) in enumerate(
    zip(filtered_chunks, embeddings, metadata_list)
):
    client.upsert(
        collection_name="hybrid_collection",
        points=[
            models.PointStruct(
                id=i,
                payload=metadata,
                vector={
                    "dense": dense_embedding,
                    "text-sparse": text_to_sparse_vector(chunk, bm25),
                },
            )
        ],
    )

# Step 7: Hybrid search with RRF fusion
def hybrid_search(query_text: str, query_dense_vector: list, limit: int = 10):
    query_sparse = text_to_sparse_vector(query_text, bm25)

    results = client.query_points(
        collection_name="hybrid_collection",
        prefetch=[
            models.Prefetch(query=query_dense_vector, using="dense", limit=100),
            models.Prefetch(query=query_sparse, using="text-sparse", limit=30),
        ],
        query=models.FusionQuery(fusion=models.Fusion.RRF),
        limit=limit,
    )
    return results.points


# Usage
results = hybrid_search("6th Settlement Apartment", your_query_embedding)

1. Sample Documents

Let’s start with a small collection of documents to search through.

documents = [
    "Python is a programming language",
    "I love Python programming",
    "Java is also a programming language",
]

2. Tokenization

First, we need to break text into individual words (tokens). We’ll convert to lowercase for case-insensitive matching.

def tokenize(text):
    """Convert text to lowercase and split into words."""
    return text.lower().split()


# Test tokenization
print("Example:", tokenize(documents[0]))

Example: ['python', 'is', 'a', 'programming', 'language']

3. Term Frequency (TF)

For each document, we count how many times each word appears.

We’ll use Python’s Counter for this.

from collections import Counter


def compute_term_frequencies(documents):
    """Compute term frequency for each document."""
    doc_term_freqs = []
    for doc in documents:
        tokens = tokenize(doc)
        term_freq = Counter(tokens)
        doc_term_freqs.append(term_freq)
    return doc_term_freqs


all_docs_terms = compute_term_frequencies(documents)

# Display term frequencies for each document
for i, tf in enumerate(all_docs_terms):
    print(f"Document {i}: {dict(tf)}")

Document 0: {'python': 1, 'is': 1, 'a': 1, 'programming': 1, 'language': 1}
Document 1: {'i': 1, 'love': 1, 'python': 1, 'programming': 1}
Document 2: {'java': 1, 'is': 1, 'also': 1, 'a': 1, 'programming': 1, 'language': 1}

4. Document Frequency (DF)

Document frequency counts in how many documents each term appears. This helps identify common vs. rare words.

def compute_document_frequency(doc_term_freqs):
    """Count how many documents each term appears in."""
    df = {}
    for doc_tf in doc_term_freqs:
        for term in doc_tf.keys():
            df[term] = df.get(term, 0) + 1
    return df


df = compute_document_frequency(all_docs_terms)
print("Document Frequencies:", df)

Document Frequencies: {'python': 2, 'is': 2, 'a': 2, 'programming': 3, 'language': 2, 'i': 1, 'love': 1, 'java': 1, 'also': 1}

5. Inverse Document Frequency (IDF)

IDF measures how rare or common a word is across all documents. Rare words get higher scores.

The formula is:

where:

• = total number of documents

• = document frequency of word

import math


def compute_idf(term, df, num_docs):
    """Calculate IDF score for a term."""
    if term not in df:
        return 0
    df_term = df[term]
    return math.log((num_docs - df_term + 0.5) / (df_term + 0.5))


# Test IDF scores
print(f"IDF('python'): {compute_idf('python', df, len(documents)):.4f}")
print(f"IDF('java'): {compute_idf('java', df, len(documents)):.4f}")
print(f"IDF('programming'): {compute_idf('programming', df, len(documents)):.4f}")

IDF('python'): -0.5108
IDF('java'): 0.5108
IDF('programming'): -1.9459

6. Average Document Length

BM25 normalizes scores by document length to avoid bias toward longer documents.

def compute_avg_doc_length(doc_term_freqs):
    """Calculate average document length (in terms)."""
    total_length = sum(len(doc_tf) for doc_tf in doc_term_freqs)
    return total_length / len(doc_term_freqs)


avg_doc_len = compute_avg_doc_length(all_docs_terms)
print(f"Average document length: {avg_doc_len:.2f} unique terms")

Average document length: 5.00 unique terms

7. BM25 Score for a Single Term

The BM25 score for a single term in a document is:

where:

• = term frequency of word in document

• = length of document

• = average document length

• = term frequency saturation parameter (typically 1.2-2.0)

• = length normalization parameter (typically 0.75)

The Role of Parameters and

Parameter	Controls	Low Value	High Value
(1.2-2.0)	Term frequency saturation	TF matters less	TF matters more
(0-1)	Length normalization	Ignore length	Penalize long docs

• : Only IDF matters, TF is ignored

• : Document length is ignored

• : Full length normalization

def bm25_term_score(term, doc_tf, idf_score, doc_length, avg_doc_len, k1=1.5, b=0.75):
    """Calculate BM25 score for a single term in a document."""
    tf = doc_tf.get(term, 0)
    if tf == 0:
        return 0

    numerator = idf_score * tf * (k1 + 1)
    denominator = tf + k1 * (1 - b + b * (doc_length / avg_doc_len))
    return numerator / denominator

8. Full BM25 Search

To score a query against all documents:

1. Tokenize the query

2. For each document, sum the BM25 scores of all query terms

3. Return scores for all documents

def bm25_search(query, documents, doc_term_freqs, df, avg_doc_len, k1=1.5, b=0.75):
    """
    Score all documents for a given query using BM25.

    Returns a list of (document_index, score) tuples sorted by relevance.
    """
    query_tokens = tokenize(query)
    num_docs = len(documents)
    scores = []

    for doc_idx, doc_tf in enumerate(doc_term_freqs):
        doc_length = sum(doc_tf.values())  # Total terms in document
        doc_score = 0

        for term in query_tokens:
            idf_score = compute_idf(term, df, num_docs)
            doc_score += bm25_term_score(
                term, doc_tf, idf_score, doc_length, avg_doc_len, k1, b
            )

        scores.append((doc_idx, doc_score))

    # Sort by score (descending)
    scores.sort(key=lambda x: x[1], reverse=True)
    return scores

9. Test the Search Engine

def display_results(query, results, documents):
    """Pretty print search results."""
    print(f"\nQuery: '{query}'")
    print("-" * 60)
    for rank, (doc_idx, score) in enumerate(results, 1):
        print(f"{rank}. [Score: {score:6.3f}] {documents[doc_idx]}")


# Test different queries
queries = ["python programming", "java programming", "love python"]

for query in queries:
    results = bm25_search(query, documents, all_docs_terms, df, avg_doc_len)
    display_results(query, results, documents)

Query: 'python programming'
------------------------------------------------------------
1. [Score: -1.785] Java is also a programming language
2. [Score: -2.457] Python is a programming language
3. [Score: -2.700] I love Python programming

Query: 'java programming'
------------------------------------------------------------
1. [Score: -1.317] Java is also a programming language
2. [Score: -1.946] Python is a programming language
3. [Score: -2.138] I love Python programming

Query: 'love python'
------------------------------------------------------------
1. [Score:  0.000] I love Python programming
2. [Score:  0.000] Java is also a programming language
3. [Score: -0.511] Python is a programming language

10. Understanding the Results

• Negative scores can occur when query terms appear in most documents (high DF)
• Higher scores indicate better relevance
• With small document collections, common words dominate; BM25 works best with larger collections
• The parameters and can be tuned for different applications

The Three Ideas Behind BM25

1. Term Frequency: Words appearing more often in a doc → more relevant
   • BUT with diminishing returns (5 mentions isn’t 5x better than 1)
2. Inverse Document Frequency: Rare words matter more
   • “quantum” is more useful than “the”
3. Document Length Normalization: Shorter docs are often more focused
   • A 100-word doc mentioning “python” once may be more relevant than a 10,000-word doc mentioning it once

Here is Part 2: BM25 with Qdrant: A real use case for using BM25 with Gemini embeddings to improve search results for real estate.

Also the Marimo BM25 Explained

A benchmark: BM25 Benchmark

1. Bi-Encoders (The Speed Kings)

Bi-encoders (like standard BERT embeddings) encode the query and the document independently into a single vector. Search is just a cosine similarity between these two points. It’s incredibly fast (sub-millisecond) but loses fine-grained details because the entire document is compressed into one fixed-size vector.

2. Cross-Encoders (The Accuracy Masters)

Cross-encoders feed both the query and the document into the model simultaneously (Early Interaction). The model can attend to every word in the query relative to every word in the document. This is highly accurate but computationally expensive because you must run the model for every single query-document pair. You can’t pre-compute embeddings.

3. Late Interaction: The Best of Both Worlds

Late Interaction models, pioneered by ColBERT, bridge this gap. Instead of one vector per document, they store a vector for every single token in the document.

When a query comes in: 1. The query is encoded into token-level embeddings. 2. A MaxSim (Maximum Similarity) operation is performed: for each query token, we find the document token that matches it best. 3. We sum these maximum similarities to get the final score.

This allows the model to perform fine-grained matching (like a cross-encoder) while still allowing document embeddings to be pre-computed (like a bi-encoder).

ColPali: Retrieval Without OCR

One of the most exciting recent developments is ColPali. Traditional PDF retrieval requires a complex pipeline: OCR the text, chunk it, and then embed it. This often fails on tables, charts, and complex layouts.

ColPali applies the Late Interaction principle to vision models (PaliGemma). It treats image patches of a PDF page as “tokens.” Instead of reading text, it “looks” at the page and matches query tokens directly to visual features.

Key Benefits of ColPali: - Layout Aware: It understands that a caption belongs to a specific image. - OCR-Free: No more messy text extraction from scanned documents. - Superior Retrieval: It outperforms traditional text-based RAG on visually rich documents.

Ecosystem and Tools

If you want to implement Late Interaction today, these are the projects to watch: - ColBERTv2: The optimized version of the original late interaction model. - PyLate: A flexible Python library for training and using late interaction models. - PLAID: An extremely fast engine for searching ColBERT vectors. - Model2Vec: While focused on static embeddings, it shows the trend towards more efficient representation learning.

Late interaction is transforming how we think about retrieval, moving us away from “one vector fits all” towards a more nuanced, token-aware future.

Practical Example: Using ColBERT with PyLate

Here’s how to use a late interaction model for retrieval in Python:

from pylate import ColBERT, Indexes, retrieve

# Load a pre-trained ColBERT model
model = ColBERT("lightonai/colbertv2")

# Encode documents into token-level embeddings
documents = [
    "Late interaction models store a vector per token instead of one per document.",
    "Bi-encoders compress documents into a single vector for fast retrieval.",
    "Cross-encoders process query and document together for higher accuracy."
]

document_embeddings = model.encode(documents, convert_to_tensor=True)

# Build an index for fast retrieval
index = Indexes.FlatIndex()
index.add_documents(document_embeddings, documents=documents)

# Encode the query
query = "What is the difference between bi-encoders and cross-encoders?"
query_embedding = model.encode([query], convert_to_tensor=True)

# Retrieve top-k results
results = index.search(query_embedding, k=3)
for doc, score in results:
    print(f"{score:.3f}: {doc}")

FAQ: Late Interaction Models

Related Posts

• BM25 Benchmarking — Comparing sparse retrieval methods
• MTEB Benchmark Explained — How embedding models are evaluated
• Tiny-GTE Transformer Model — Efficient transformer architecture for embeddings