The Case for a Local RAG

Ask a frontier AI model a question about the Peace of Westphalia, the Krebs cycle or the semantics of Rust lifetimes, and it will usually give you an excellent answer. Ask it what your landlord's contract says about the notice period, which supplier quoted the lowest price in last spring's tender, or what you concluded in your own notes on a project three years ago, and it has nothing to offer. It has never seen those documents. The knowledge that matters most to your daily work is precisely the knowledge a general-purpose model does not have.

There are two ways to close that gap. You can send your documents to the model, or you can bring the model to your documents. The second option, a retrieval system running entirely on your own hardware, has quietly become practical. For personal knowledge and for a meaningful share of professional work, it is the better choice.

What retrieval-augmented generation actually does

Retrieval-augmented generation, usually shortened to RAG, was formalised in a 2020 research paper and has since become the standard way to connect language models to private knowledge. The idea is simple. Instead of hoping a model memorised the right facts during training, you retrieve the relevant passages from your own collection at question time and hand them to the model along with the question. The model's job shifts from recalling to reading.

A RAG system has four moving parts:

1. Ingestion. Documents are read, cleaned and split into passages, typically a few hundred words each.
2. Embedding. Each passage is converted into a vector, a list of numbers that captures its meaning, by a small specialised model.
3. Retrieval. When you ask a question, it is embedded the same way, and the passages whose vectors are closest in meaning are pulled from the index. Good systems combine this with traditional keyword search, because exact terms like invoice numbers or names are where pure semantic search stumbles.
4. Generation. A language model receives the question and the retrieved passages and writes an answer, ideally citing which documents it used.

Every one of those steps can run on a laptop.

Why local changes the equation

The obvious benefit is privacy, but it is worth being precise about why.

When you use a cloud assistant over your files, three things leave your control: the documents themselves (uploaded or retrieved through a connector), the index built from them, and every question you ask. Questions are revealing in their own right. A sequence of queries about a medical condition, a divorce settlement or an acquisition target discloses as much as the documents do.

It is also a mistake to think the index is harmless because it is "just numbers". Research published in 2023 showed that the original text can be reconstructed from embeddings with high fidelity. An embedding index of sensitive documents is a sensitive asset, and a cloud-hosted one is a copy of your data held by someone else.

A local system removes all three exposures at once. No processing agreement is needed because there is no processor. No transfer assessment is needed because nothing is transferred. Retention policy is whatever you decide, enforced by your own disk. For individuals, this is peace of mind. For professionals bound by confidentiality, such as lawyers, doctors, bankers and advisers, it can be the difference between being allowed to use AI on client material and not.

There are less obvious benefits too.

It works offline. On a train, on a plane, in a secure facility, or when a provider has an outage.

It is stable. Cloud models are updated, re-tuned and retired on the provider's schedule. A local model behaves the same way next year as it does today unless you choose to change it. For anything resembling a professional workflow, that predictability is worth a great deal.

The marginal cost is zero. Once the hardware exists, re-indexing a hundred thousand documents or asking a thousand questions costs electricity, not tokens. That changes how freely you use the tool.

It is honest about its sources. Because a local RAG answers from passages it can show you, it encourages a habit cloud chat often discourages: checking. A good local system cites the file and the passage behind every claim, so you can open the original in one click.

"But local models are weaker"

They are, in absolute terms. The largest open-weight models now rival closed frontier systems, but those run on data-centre hardware, not laptops. What fits comfortably on a modern laptop or desktop, typically models in the 4 to 14 billion parameter range, is noticeably less capable at open-ended reasoning, advanced mathematics and complex multi-step tasks than the best cloud models.

The point is that RAG changes what "capable" needs to mean. A model answering from retrieved passages does not need to know much; it needs to read carefully, follow instructions, stay faithful to the source and say "the documents don't cover this" when they don't. Those are exactly the skills small instruction-tuned models have improved at fastest. A compact model given the right three paragraphs will often outperform a giant model guessing from memory, and it will hallucinate less because it has something to anchor to.

The real limit is retrieval. If the right passage never reaches the model, no amount of intelligence downstream will save the answer. That is where the engineering effort belongs.

Where local RAG succeeds or fails

From experience building these systems, the same handful of factors decide whether a local RAG feels magical or frustrating:

Parsing. PDFs, scanned documents, spreadsheets and slide decks are hostile formats. Text that comes out of the parser jumbled or incomplete cannot be retrieved well. Budget time for this; it is unglamorous and it is half the job.

Chunking with context. Passages split blindly at fixed lengths lose their meaning. A paragraph that says "the fee is waived in this case" is useless without the heading and document it came from. Keeping titles, headings and file paths attached to every chunk improves retrieval dramatically.

Hybrid search. Combine semantic vectors with keyword matching. Ask a pure vector system for "invoice 4471" and it may return invoices in general.

Metadata and filters. Dates, folders and document types let you ask "in last year's tax documents" rather than searching everything.

Scope discipline. Start with one well-understood folder rather than an entire disk. Indexing everything at once produces a noisy index and a poor first impression. Expand deliberately.

Evaluation. Write twenty questions whose answers you know, and run them every time you change a setting. Without this you are tuning by feel, and feel is unreliable.

Refresh strategy. Decide whether the index updates continuously or on a schedule. Scheduled re-indexing is simpler and usually sufficient for personal knowledge; live watching matters more for shared, fast-changing collections.

A sensible starting stack

The tooling has matured to the point where a capable local RAG can be assembled in an afternoon. A typical setup today uses a local model runtime such as Ollama or LM Studio, a small embedding model, a lightweight vector store that lives in a single file, and a compact instruction-tuned language model of 4 to 8 billion parameters for answers. On recent hardware with 16 gigabytes or more of memory, this runs comfortably alongside everyday work. Quantised builds of models such as Switzerland's Apertus need roughly 6 gigabytes, which puts a fully open, multilingual model within reach of an ordinary laptop.

The interface can be as simple as a command line or as polished as a private web page reachable only from your own devices. What matters is that every answer shows its sources.

The bigger point

For two years, the default assumption has been that useful AI lives in someone else's data centre and that using it means handing over your data. Local RAG is the clearest demonstration that this assumption is no longer universally true. For the large class of tasks that consist of finding, reading and synthesising what you already have, a private system on your own machine is not a compromise. It is often the better product: more private, more stable, cheaper to run and more honest about where its answers come from.

The frontier models will keep pulling ahead at the top end, and there will always be problems that need them. But the assistant that knows your work, and keeps it to itself, may be the one you end up using most.