A chatbot is one model behind a text box. What I built this week is not that. It is a single machine in the garage running a fleet of models — a dozen of them by the time I stopped — each one good at a different, narrow thing, orchestrated so the whole is more useful than any one part. The interesting engineering was never “which model is best.” It was deciding what the team should be, and how the pieces hand off to each other, on hardware that costs less than a single new graphics card and never sends a byte to anyone’s cloud.
The box is two second-hand datacenter GPUs — Pascal-era cards, chosen for having a lot of memory for their price rather than a lot of speed. That one trade shapes every decision downstream, so it is worth stating plainly: these cards are generous with memory and stingy with modern compute. Anything that leans on the newer number formats a 2020s GPU expects runs at a crawl. Anything that streams weights from memory — which is most language models — runs fine. So the first job was sorting what thrives here from what merely limps, and building the fleet entirely out of the former.
These are passively cooled cards — no fans of their own, because in a real server they sit in a hurricane of chassis airflow. In a quiet box they would simply cook. So keeping them alive meant printing custom ducts to bolt a high-static-pressure fan onto the intake of each card — the one bit of the build that had to be fabricated rather than bought.
The constraint that designs the system
Forty-eight gigabytes of video memory sounds like a lot until you notice the good models are thirty to thirty-seven gigabytes each. Two won’t fit. So the core rule of the whole architecture is: one large model is resident at a time. Ask for a different one and it swaps in from disk, a twenty-to-forty-second cost. This sounds like a limitation, and it is, but it also forces a clarifying discipline — you don’t get to be sloppy and keep six giant models hot. You pick a default that stays loaded, and you switch deliberately.
Everything else in the design falls out of that. The small specialist models have to be small — a few gigabytes or less — so they can run in the sliver of memory left over, or on the CPU, without evicting the big model doing the actual reasoning. That budget is the reason the fleet looks the way it does.
The generalists
Two big models do the heavy thinking, and they split along a line that turns out to matter more than raw benchmark scores.
The default, always-loaded model is the agent brain — the one chosen for how reliably it makes tool calls. An agent is only as good as its ability to decide “call this function with these arguments” and get the arguments right, every time, across a long back-and-forth. That’s a different skill from writing a nice paragraph, and the model that wins at it isn’t the one that tops the general leaderboards. It’s resident by default precisely because the assistant that drives everything else leans on it constantly, and I didn’t want to pay the swap cost on every routine request.
The second big model is the daily driver — general reasoning, coding, and reading images. It’s a mixture-of-experts design, which is the shape that suits these old cards beautifully: a model with tens of billions of parameters that only activates a few billion per word, so it reads from memory at a rate the hardware can actually sustain. It generates at about fifty words per second, which is faster than you can read, and it beat the dedicated coding model I’d earmarked at both speed and coding benchmarks — so the coding model got dropped and this one absorbed the role. One fewer thing to maintain.
There is a third slot, an uncensored build of the same daily driver, kept for research questions where a stock model reflexively refuses things that are perfectly legitimate to ask locally. It runs at identical speed — the modification costs nothing in capability — and it exists because the entire point of owning the hardware is that what I ask my own machine is nobody else’s business.
The measured numbers are the whole argument for the mixture-of-experts approach on this hardware. The models that only activate a few billion parameters per word run several times faster than the dense ones — which is exactly why the dense candidates were tested and set aside rather than kept.
The specialists
Around the generalists orbit a set of small, fast, single-purpose models. None of them chats. Each does one thing and hands off.
Ears. A speech-to-text model transcribes audio — voice notes, recordings, anything spoken. On the GPU it runs at roughly fourteen times real time; a minute of audio becomes text in about four seconds. It’s tiny enough to load on demand without disturbing the resident language model.
A voice. A text-to-speech model, small enough to run on the processor alone with zero claim on the GPUs, turns any text back into natural speech in real time. Ears plus a voice plus the language model is, of course, the shape of a genuinely private voice assistant — one where the audio of your own home never leaves the building.
A memory. This is the piece I’m most pleased with. On its own, a language model knows the world but not your world. So there’s now a retrieval layer: a small embedding model turns my personal knowledge base — notes on the house, the network, every project — into searchable vectors, and when I ask a question the system finds the relevant passages and hands them to the big model to answer from my own sources, with citations. The satisfying part is that when the answer genuinely isn’t in my notes, it says so rather than inventing one. Grounded, and honest about the edges of what it knows.
Eyes. A small image classifier can screen a photo library for content that shouldn’t be sitting in a family-readable share — running on the processor, read-only, flagging things for a human to review rather than ever deleting anything itself. The same class of model could tag a decade of photo backups by what’s actually in them. This is the one job that deliberately stays off the cloud services that would happily do it, because a photo library is exactly the kind of thing you don’t upload to be “analyzed.”
A flourish. And because the hardware had a spare corner, a music-generation model that turns a text prompt into a short original clip — “calm ambient synth pad” becomes six seconds of audio in about seven. Not useful, exactly. But it runs on demand, gives the memory back when it’s done, and it’s a nice reminder that all of this is mine to play with.
System one and system two
The pattern underneath is borrowed from how people are said to think: fast, intuitive reflexes versus slow, deliberate reasoning. The small specialists are the reflexes — transcribe this, is this image safe, which of these things did the user mean. The big generalists are the deliberation. A good assistant spends most of its time in the cheap, fast layer and only escalates to the expensive one when a question actually needs it. Built this way, the swap-one-model-at-a-time constraint stops being a wart: the thing that stays resident is the reasoning brain, and the reflexes cost almost nothing to keep on call beside it.
It is, deliberately, a team of small honest specialists rather than one enormous model pretending to be good at everything.
Why local
Every part of this runs offline. No API keys to a company, no per-token bill, no terms of service, no telemetry, nothing indexed by someone else. For the routine stuff that’s merely nice. For the parts that touch my documents, my photos, my home’s own audio, or research questions I’d rather not have logged against my name, it’s the entire point. The models are public and the hardware is cheap and second-hand; what’s private is that it all happens in my garage and stays there.
It is not finished — the memory layer’s recall needs sharpening, and the one-model-at-a-time swap is a real cost I’m still deciding how to hide. But the shape is right: not a chatbot, but a small, private institution of specialists, each doing the one thing it’s good at, on hardware nobody else wanted.