local rag · zero paid apis · open book, closed box

RAG, for people new here

How Scrappy reads its own notes and answers questions without ever guessing.

This is the explainer for Scrappy's local RAG pipeline. RAG stands for retrieval-augmented generation, and in plain words it means: before the computer answers a question, it looks things up in its own notes, then writes an answer only from what those notes say. No guessing. If the notes don't cover the question, it says so out loud. Everything here runs on this machine. No cloud, no paid API, no metered call.

see the pipeline → watch it answer

960 chunks in the store pieces of notes the pipeline can search

952 distinct sources 868 model docs · 82 provider docs · 2 scraped pages

768 embedding dimensions the size of the number list per chunk

$0 paid API spend all inference local after one-time free downloads

The verdict The pipeline works. You ask it a question about what Scrappy has read, it pulls the right notes off the shelf, quotes them, and answers a few seconds later. The whole thing cost nothing to run. Read on and every word below is translated into plain English.

The legend, every term in plain english

glossary

This page uses a handful of words from the RAG and machine-learning world. Each one gets three things here: what it means, a plain-English translation, and a working analogy. If you meet a word you don't recognize, it lives in this table.

amber = highlight, numbers, the answer lime = it works, a win, an honest "I don't know" red = a limit, a known weakness grey = background, metadata, the quiet parts

term	what it is	plain english	working analogy
`RAG`	retrieval-augmented generation	Before answering, the program looks up relevant notes, then answers only from those notes	An open-book exam. You don't memorize everything; you search the book for the right pages, then write your answer from what those pages say.
`chunk`	a piece of a document, cut small enough to search and quote	a short slice of a page, usually a paragraph	Cutting a long book into bookmark-sized pages so you can find the exact page instead of hauling the whole book.
`embedding`	a list of numbers that describes the meaning of a piece of text	a numeric fingerprint of what a sentence means	Giving every sentence a map coordinate, so "expensive" and "costs a lot" land close together on the map.
`vector store`	the database that keeps the number lists and finds the closest ones	a library filed by meaning, not by title	One here is LanceDB. Like a librarian who finds the book closest in spirit to your question.
`cosine similarity`	a score for how close two meanings are, from -1 to 1	a "how alike are these two?" number	Measuring the angle between two arrows. A small angle means they point the same way, so they're similar.
`BM25`	a keyword-matching search method	finds chunks by the exact words they share	A smarter version of ctrl+F that also ranks the hits by how rare and important the matched words are.
`dense retrieval`	meaning-based search using embeddings	finds chunks by idea, even when the words differ	Searching by what the text means, so a question rephrased with different words still finds the right note.
`RRF fusion`	reciprocal rank fusion; blends two ranked lists into one	merges the keyword results and the meaning results into one final list	Two friends each rank the same candidates, then you blend both lists so someone who ranks high on both wins.
`top-k`	how many of the best chunks you hand the model	how many notes you bring to the open-book desk	The default here is 5. You bring the 5 best-matching notes, no more, no less.
`context`	the text you give the model to read before it answers	the open pages of the book the model is allowed to use	The answer must come from the context. What is not in context is off limits.
`citation`	a [n] marker pointing to the chunk used	a footnote saying "this came from page 12"	Each claim is tagged with which note it was copied from, so you can trace it back.
`abstain`	the model saying "I don't know" instead of guessing	a refusal to answer when the notes have nothing	Refusing to answer a question the book doesn't cover, instead of making one up.
`hallucination`	the model inventing a confident false answer	a made-up fact stated with total confidence	The thing abstaining exists to prevent. A narrator who fills gaps with a confident lie.
`LLM`	large language model	a program trained to read and write text	A very well-read autocomplete that can also follow instructions like "answer from these notes only."
`tokens`	the small pieces of text an LLM reads and writes	words and parts of words	Pricing like "$0.20 per 1M tokens" means per roughly a million word-pieces.
`local model`	the LLM runs on your own computer	the brain lives on this machine, not on a company server	A book on your desk, not one you must rent online. No data leaves the room.
`GGUF`	a file format that stores a compressed model	a packed model file that runs on a laptop	A zipped library book that opens fast on your machine instead of in a warehouse server.
`LanceDB`	the vector store this pipeline uses	the meaning-filed library behind the scenes	The shelf in the warehouse where every chunk's number list is stored and searched.

In plain english Every term above is just a label for an everyday idea: look something up, weigh which notes fit best, quote them, and refuse to answer when nothing fits. The rest of this page uses those ideas.

The idea in one picture

8 steps

The pipeline has two halves. First it builds the store once, from Scrappy's own data. Then, every time you ask a question, it searches the store and lets a local model answer from what it finds. Here are all eight steps, in order.

Scrappy data

The pipeline starts with what Scrappy already collects: 868 model documentation pages, 82 provider documentation pages, and 2 scraped pages from the engine cache. Together these are 952 distinct sources.

input: the mirror's markdown and pricing docs

Chop into chunks

Each page is cut into short pieces called chunks, small enough to search, quote, and cite. The 952 sources become 960 chunks. A few pages are short enough to stay one chunk; most split into several.

output: 960 chunks

Turn each chunk into numbers

An embedding model, nomic-embed-text-v1.5, reads each chunk and writes a list of 768 numbers that capture its meaning. Same idea, close numbers. The model file is 139 MB (Q8_0 quantized).

model: nomic-embed-text-v1.5 · 768 dimensions · 139 MB

Store the numbers in LanceDB

Each chunk, its 768-number list, and its source URL go into a vector store called LanceDB. This is the built library. Building it once is called ingest.

store: rag/vector_store/chunks.lance

Your question gets the same treatment

When you ask a question, it does not sit beside the library. It gets turned into numbers too, using the same embedding model, so the pipeline can compare meanings.

Hybrid search finds the best chunks

Two searches run at once: BM25 matches exact words, and dense retrieval matches meaning. RRF fusion blends the two ranked lists into one, and the top-k best chunks are pulled. The default top-k is 5. A flag, --no-hybrid, turns off BM25 for dense-only search.

hybrid: BM25 + dense, fused with RRF · top-k default 5

A local model writes an answer from those chunks

The best chunks become the context, and the reader model, Qwen2.5-3B-Instruct, writes an answer using only that context. Each claim carries a citation, a [n] pointing back to the chunk it came from. The reader runs through llama-cpp-python with Metal on an Apple M4 with 24 GB RAM. Its GGUF file is 2.0 GB (Q4_K_M).

model: Qwen2.5-3B-Instruct · Q4_K_M · 2.0 GB · on-device

If nothing fits, it says so

When the context has no answer for the question, the model abstains. It answers with "I don't know based on the provided context" instead of inventing a fact. That refusal is the guard against hallucination.

output: an answer with citations, or an honest "I don't know"

In plain english Steps 1 to 4 build the library once. Steps 5 to 8 happen on every question: turn the question into numbers, find the closest notes, hand a small local model exactly those notes, and let it answer only from them. The model is a reader, not a knower.

III

How one question flows

walkthrough

Take a real question that was asked of the pipeline: "How much does gpt-5.6-luna cost?" Here is the path it takes, end to end, in plain words.

The question is read in. Its meaning is turned into a 768-number list by the same embedding model the library uses. The pipeline runs hybrid search: BM25 catches the exact words "gpt-5.6-luna" and "cost", and dense retrieval catches the idea of "pricing for that model". RRF fusion blends both lists, and the best few chunks are pulled. One of them is the OpenRouter page for gpt-5.6-luna, which holds its prices.

Those chunks become the context handed to the reader. The reader writes an answer using only that context and tags each claim with a citation. It returns: GPT-5.6 Luna costs $0.20 per 1M tokens, with a [1] pointing back to that page. The whole run happened on this machine.

In plain english The pipeline never answers from memory. It answers from whatever notes it just pulled, and it tells you which note each claim came from. If the note weren't there, it would say so instead.

Live demo, verbatim answers to real questions

verbatim

These four question and answer pairs are the pipeline's actual output, quoted word for word. Nothing is trimmed or prettied. The honest notes under each answer are part of the point: this is a small model with real limits, and we show them.

Q"How much does gpt-5.6-luna cost?"

GPT-5.6 Luna costs $0.20 per 1M tokens. [1] Honest note. The pipeline retrieved the openrouter.ai/openai/gpt-5.6-luna page, the right one. The reader quotes the input price, and the same chunk also holds the output price, $1.20. It reports one of the two.

Q"Which provider serves the cheapest Gemini model?"

Google AI Studio serves the cheapest Gemini model, as indicated by the lowest input cost per 1M tokens for all models listed: [4] Google: Gemini 3.6 Flash at $0.75 per 1M tokens, [2] Google Gemini Pro Latest at $2.00 per 1M tokens, [3] Google Gemini Flash Latest at $0.38 per 1M tokens, and [1] Google Nano Banana 2 Lite (Gemini 3.1 Flash Lite Image) at $0.25 per 1M tokens. Honest note. All the numbers are real and come from the context, but the small local reader muddles which one is actually cheapest, $0.25, while anchoring on [4]. A known v1 limitation, shown honestly.

Q"What kind of license is Python developed under?"

Python is developed under an OSI-approved open source license. [1]

Q"Who won the 1998 World Cup?"

I don't know based on the provided context. The abstain behavior working. The store has no World Cup data, so the model refuses to guess. That refusal is the honest answer.

In plain english Two of these answers are exactly right. One gets the right prices but names the wrong winner of "cheapest", and one honestly says "I don't know." That mix is what a small local reader actually produces, and the page shows all of it rather than just the wins.

Quirks and honest limits

the ugly numbers

No pipeline this size is flawless. Here is what does not work yet, stated plainly, including the bug that fixed itself by complaining.

honest bug

The first build ate the wrong table

The first ingest pulled the wrong table. Prices were missing from the store, so price questions abstained even when the right pages were retrieved. The fix added the mirror's real catalog table, with per-token prices converted to $/1M tokens. The store went from 508 to 960 chunks, and price questions started answering.

The abstain behavior did its job. When there was no price data, the model refused to guess instead of inventing one. That refusal is what pointed us at the bug.

we lose

Thin page corpus

The store is nearly all model and provider docs: 868 model docs, 82 provider docs, and only 2 scraped pages. Ask about the real web and the library has almost nothing, so the model abstains. Its reach is the size of its notes.

we lose

The 3B reader trips on multi-hop questions

Qwen2.5-3B-Instruct is a small model. Its weakness shows when an answer needs several facts brought together, as in the Gemini "cheapest" question: it quotes all the real numbers but names the wrong cheapest one. Comparing across rows is harder than reading a number off one row.

we lose

The comparison muddle, shown instead of hidden

The Gemini answer is the clearest example. Every price in it is real and from context, yet the reader anchors on [4] and misses that $0.25 is the true low. A bigger reader would likely fix this; for now the limitation is printed in the demo on purpose.

How to run it

three commands

The pipeline lives in the rag/ folder of the Scrappy repo. Two virtual environments: .venv for building the store, and .venv-b for asking questions. The models download once from HuggingFace, then everything runs locally.

build the store

# one-time: reads Scrappy's data, chops
# it, embeds it, writes the LanceDB library
./rag/.venv/bin/python rag/ingest.py

This is steps 1 to 4. Run it once, or again when the mirror grows. It re-reads the data and rebuilds the library.

ask a question

./rag/.venv-b/bin/python rag/query.py "your question"

This is steps 5 to 8. The store is already built, so each question is fast. Windows, doors, no paid calls.

two knobs

./rag/.venv-b/bin/python rag/query.py "your question" --top-k 10
./rag/.venv-b/bin/python rag/query.py "your question" --no-hybrid

# --top-k N   how many chunks the model may read (default 5)
# --no-hybrid turn off BM25, use meaning-only search

--top-k changes how many notes the reader may use. --no-hybrid drops the keyword search and keeps only the meaning search. You can tweak either per question.

In plain english One command builds the library. One command asks it questions. Two flags let you tune how many notes the model reads and whether it uses meaning-only search. That is the whole surface.

VII

What could come next

levers, not promises

The pipeline works now. These are the concrete levers that would push its weak spots, each tied to a limit named above.

Add a reranker

A reranker re-scores the pulled chunks so the best ones rank first. It is the usual cure for the "right page retrieved, wrong fact quoted" class of errors the demo exposes.

Bring in a bigger reader

Qwen2.5-3B trips on multi-hop comparison questions. A larger local model would almost certainly fix the Gemini "cheapest" muddle.

Scrape more pages into the store

The corpus is 868 model docs, 82 provider docs, and only 2 scraped pages. Scraping more pages broadens what the library knows, which shrinks how often the model has to abstain.

Re-ingest on the hourly mirror

Once an hour the mirror becomes more complete. Re-running ingest keeps the library in step, so new model and price data become answerable.

In plain english Every next step targets a named weakness: rank the finds better, read with a bigger brain, own more pages, and keep the library current. None of them touches the core idea, which already works.

The keys, every spec has a door

Where everything lives, and the exact commands to run it.

file	what it does
`rag/ingest.py`	builds the store
`rag/chunking.py`	cuts pages into chunks
`rag/embedding.py`	turns chunks and questions into numbers
`rag/retriever.py`	runs hybrid BM25 + dense search with RRF fusion
`rag/db.py`	talks to the LanceDB store
`rag/llm_local.py`	loads and runs the local reader model
`rag/query.py`	the CLI you ask questions with
`rag/CONTRACT.md` · `rag/README.md`	the contract and the usage notes

ingest

./rag/.venv/bin/python rag/ingest.py

ask

./rag/.venv-b/bin/python rag/query.py "your question" --top-k 5