How to Build a Privacy-First AI Stack

Most organisations handle AI and confidential data in one of two ways. They ban it, and staff quietly use personal accounts anyway. Or they approve one vendor and send everything through it, including the things that should never have left the building.

There is a middle path, and it is not complicated. Decide which of your data actually cannot leave your control, then architect only that part carefully and stop over-engineering the rest.

This is a practical build guide for that middle path, layer by layer.

Step 1: Classify before you architect

Almost every failed AI privacy project starts by choosing tools. Start by sorting data instead.

Three tiers is usually enough. Public or low-sensitivity material, where the only real question is output quality. Internal material, where a contractual assurance and a decent vendor is proportionate. And restricted material — unreleased product work, personal data, legally privileged documents, anything under a customer NDA.

Most teams discover the restricted tier is a small fraction of what they touch daily. That is the useful finding, because it means you only need to build the expensive architecture around a narrow slice, and can use ordinary hosted tools for everything else without anxiety.

Write the classification down and make it specific to your business. A generic policy that says "be careful with sensitive data" produces no behaviour change whatsoever.

Step 2: The model layer

For tier one and two, a hosted API is fine and you should not talk yourself into infrastructure you do not need.

For tier three, running open-weight models on hardware you control removes the question entirely.

The practical requirements are more modest than people assume. At four-bit quantisation, an 8B model occupies roughly 6GB of VRAM at an 8k context and runs comfortably on an 8GB card. A 14B model needs around 11GB, so realistically a 12GB card. A 70B model wants about 46GB, which means a workstation card or multiple GPUs.

Remember that the KV cache grows linearly with context length, so a long-context workload needs materially more memory than the model file suggests. Size for your actual context window, not the model.

Be honest about the capability gap. A quantised mid-size open model is genuinely useful for summarisation, extraction, classification and routine drafting. It is not equivalent to a frontier hosted model on hard reasoning, and teams that pretend otherwise abandon the local setup within a month.

Step 3: The assistant layer

The model is not what your staff interact with. The interface is, and this is where most stacks leak.

Self-hosted chat front ends solve this cleanly. Open WebUI and LibreChat both run on your own infrastructure and can point at either a local model or a hosted API, which means you can route by data tier from a single interface — restricted work to the local model, everything else to a hosted one.

For teams that want a hosted experience without operating infrastructure, an encrypted ChatGPT alternative addresses the same concern from the other direction, though you trade away the routing flexibility that self-hosting gives you.

Whichever you pick, the question to ask is what gets logged. Prompt logs, error traces and analytics are the most common place confidential text ends up somewhere nobody intended.

Step 4: Retrieval, without handing over your documents

Retrieval is where teams accidentally undo everything they did in step two.

The trap is embeddings. If you send a confidential document to a hosted embedding API to vectorise it, the document has left your control, regardless of what you do with the vectors afterwards. Run embeddings locally for anything in tier three — embedding models are small and cheap to run, far more so than generation models.

Keep the vector store local too. Options that run on your own infrastructure are mature now, and for most corpora the storage requirement is trivial.

Then make the index respect the permissions of the source system. An assistant that can retrieve any document in the company is a privilege escalation tool wearing a friendly interface. If a user cannot open the file directly, retrieval should not surface its contents to them.

Step 5: Agents and tool access

This is the layer that turns a privacy question into a security question, and it deserves more caution than it usually gets.

Give every agent scoped credentials, never a standing admin token. Scope by what the task genuinely needs — read-only where reading is enough, and a separate credential per integration so you can revoke one without breaking everything.

Require human approval for writes, deletions, payments and anything else you cannot undo.

Autonomy is appropriate for retrieval and analysis, and rarely appropriate for irreversible actions.

Log every tool call with its inputs, outputs and the identity that triggered it. When something goes wrong you will need to reconstruct what the agent did, and without that log you cannot.

Treat everything retrieved as untrusted input. If an agent reads a web page, an email or a PDF, that content can contain instructions aimed at the model rather than the reader. Prompt injection is a live problem, not a theoretical one, and the mitigation is architectural — keep the agent's permissions small enough that a successful injection cannot do much.

Step 6: Storage, transit and keys

Encrypt at rest and in transit as a baseline, then ask the question that matters: who holds the keys. If the vendor holds them, your protection is contractual. If you hold them, it is architectural. Both are defensible; know which one you have.

Watch the periphery. Backups, log aggregation, error reporting and observability tooling routinely copy prompt content into systems with weaker controls than the primary one. This is the most common real leak in an otherwise well-designed stack.

Set retention deliberately. Conversation history is useful and it is also a growing archive of everything your staff have ever pasted. Decide how long it lives.

Step 7: The layer nobody builds

None of the above matters if your staff are using personal accounts on their phones, and in most organisations some of them are.

The reason is almost always friction. If the sanctioned tool is slower, worse or harder to reach than the unsanctioned one, people route around it, and no policy document will stop them.

So make the approved path the easy path. Single sign-on, no extra logins, available where they already work. Then say clearly what may go where — a one-page rule people can remember beats a fifteen-page policy nobody opens.

A realistic sequence

This week: classify your data into three tiers and write the one-page rule. Audit who is currently using personal accounts for work. Move any team on consumer plans to business plans, which changes the default terms immediately and costs almost nothing in effort.

This month: stand up a self-hosted front end and connect it to a hosted model for ordinary work.

Get people used to a single interface before you change what sits behind it.

This quarter: add a local model for tier three, move embeddings and the vector store in-house, and put scoped credentials and approval gates around any agent that can act rather than just read.

The honest trade-off

A privacy-first stack costs you something. Local models are weaker than frontier ones, self-hosting takes maintenance, and approval gates slow agents down. Anyone claiming otherwise is selling something.

What you get is the ability to say precisely where each class of your data goes and who can read it, which is increasingly what customers, regulators and enterprise procurement all want to hear.

The mistake is applying that cost to everything. Build the careful architecture around the narrow slice that needs it, use ordinary tools for the rest, and the whole thing stays affordable enough that people actually use it.