The Self-Hosted AI Platform
Most “AI products” are a thin wrapper around someone else’s API, billed by the token, governed
by someone else’s terms, and one pricing change away from being unviable. I went the other way. The
platform behind everything I ship runs on three machines I own, in a room I can walk into. The model
weights are on my disk. The inference bill is a power meter. When a request leaves the building it
does so deliberately, not by default.
That is the whole thesis, and it is worth being precise about why it matters. Self-hosting buys
three things that a cloud endpoint structurally cannot. You own your data and your IP — prompts,
fine-tuning corpora, and customer content never become training fodder or a subpoena target on
infrastructure you don’t control. Your marginal cost of inference trends to near zero, which changes
what you are willing to build: you stop rationing tokens and start running agents in loops. And you
get production control end to end — the model version, the quantization, the routing, the retry
logic, the failure modes are all yours to reason about. The tradeoff is real and I’ll be honest about
it at the end. First, how it’s built.
The three-node architecture
The platform is three nodes, each with one job, connected over a private LAN. Splitting by role
rather than cramming everything onto one box keeps the failure domains separate: an inference job that
pins the GPU can’t take the public websites down with it, and a hosting node reboot doesn’t interrupt
a model that’s mid-render.
inky prokopton sg1
inference & automation hosting estate GPU media & voice
┌────────────────────┐ ┌────────────────────┐ ┌────────────────────┐
│ RTX 3090 / 64GB │ │ 5x WordPress + │ │ RTX 4070 │
│ Qwenopus 3.6 (27B) │◄───►│ MariaDB (Docker) │◄───►│ Voicebox neural TTS│
│ llama.cpp + proxy │ │ Caddy reverse proxy│ │ Twilio + Grok agent│
│ model hot-swap │ │ Cloudflare tunnels │ │ Moodle LMS │
│ Hermes gateway │ │ git = source of │ │ │
│ agent + Remotion │ │ truth per site │ │ │
└────────────────────┘ └────────────────────┘ └────────────────────┘
private LAN · no public inboundinky — inference & automation
inky is the brain — a Xeon workstation with 64 GB of RAM and an RTX 3090. It serves a
27B-parameter model (Qwenopus 3.6) through llama.cpp on that 3090,
GGUF-quantized so it fits inside the card’s 24 GB of VRAM without falling off a quality cliff. Because one card can’t hold every model I want resident at once, a
systemd-driven hot-swap layer tears down one model and brings up another on demand, so a chat model,
a coding model, and an embedding model can share the same VRAM without me babysitting processes.
A local model is, by definition, frozen at its training cutoff — so inky fronts every model call
with a web-search proxy that injects fresh search results into the context before the tokens are
generated. The model stays current without being retrained. On top of that sits Hermes, a gateway
that bridges Telegram and WhatsApp to the local models so I can talk to the platform from my phone; a
Remotion render server for programmatic video; and an agent control plane that orchestrates
CEO/CTO-style agents which plan and dispatch work across the estate.
prokopton — hosting estate
prokopton runs the public-facing business. Five isolated WordPress + MariaDB stacks live in Docker
Compose — one network and one database per site, so a compromise or a runaway query in one is
contained to that site. Caddy sits in front as the reverse proxy, and public access comes through
Cloudflare tunnels: the node makes an outbound connection to the edge, which means there are zero open
inbound ports on the host. Nothing is portforwarded; there is no public IP to scan. Each site’s theme
is a git repository, so deployment is a pull and the running state is always reconstructable from
source.
sg1 — GPU media & voice
sg1 carries an RTX 4070 and handles everything media- and voice-shaped. It runs Voicebox for neural
text-to-speech, and a voice agent built on Twilio and x.ai’s Grok that screens forwarded phone calls —
it answers, figures out who’s calling and why, and drops a transcript into Telegram so I can decide
whether a call was worth interrupting my day for. It also hosts a Moodle LMS with a full automated
test harness behind it.
A war story: the timeouts that weren’t the firewall
Here is the kind of thing that “Real and Works” actually means. Shortly after standing up the
hosting estate, containers on prokopton started failing to reach each other. WordPress would
intermittently lose its database; a site would 500 for thirty seconds and then heal itself. The
symptom screamed firewall — that’s the obvious suspect for east-west traffic dropping — so the obvious
move is to go add rules. I didn’t, because the pattern was wrong for a firewall: a firewall block is
consistent, and this was intermittent and self-healing. Firewalls don’t flap.
So I watched the actual interfaces instead of guessing. tcpdump on the Docker bridge
showed the SYNs leaving and simply never arriving at the neighbour. The bridge looked healthy one
second and amnesiac the next. The tell was in ip link: the veth pairs Docker creates for
each container were briefly going DOWN and getting renamed under me. Something else on the
box was claiming ownership of interfaces it didn’t create.
That something was systemd-networkd. By default it tries to manage every
interface it sees, including the virtual veths the Docker daemon spins up per container. The two
daemons were fighting over the same NICs: Docker would create a veth, networkd would notice an
“unconfigured” interface and reset it, the link would bounce, and the container’s path to its
neighbours would drop until Docker repaired it. Not the firewall. Not Docker’s bridge config. A
governance fight between two pieces of init-adjacent plumbing.
The fix was to tell networkd to keep its hands off the virtual interfaces and leave them to
Docker:
# /etc/systemd/network/10-docker-veth.network
# Tell systemd-networkd to ignore Docker-managed virtual interfaces.
[Match]
Name=veth* docker* br-*
[Link]
Unmanaged=yesThe lesson isn’t “disable networkd.” It’s that the obvious suspect — the firewall — was the wrong
one, and the evidence said so before I touched it. Intermittent and self-healing is a different
signature than blocked. Watch the interfaces, not your assumptions.
GPU work has its own version of this. On sg1 the recurring gotcha is driver/CUDA version skew after
an unattended upgrade quietly bumps the kernel: the card is fine, but the userspace driver and the
kernel module disagree and the GPU vanishes from the runtime until they’re realigned. The pattern is
the same — the loud symptom and the real cause are two different layers, and the discipline is to
prove which layer before you change anything.
What it costs versus what it buys
I won’t pretend self-hosting is free. The honest costs are real: capital up front for the hardware,
a power bill that runs whether or not anything is using the GPU, and — the big one — you are now the
ops team. There is no managed-service SLA to fall back on. When networkd hijacks a veth at 1am, that’s
your pager. You trade a predictable monthly invoice for occasional sharp debugging sessions and the
standing responsibility of keeping three machines healthy.
What you get back is worth it for the work I do. Inference cost collapses from a per-token meter to
a power draw, which means agents can run in long loops and render pipelines can churn without anyone
doing arithmetic about the bill. Data and IP stay on hardware I own — nothing leaves unless I send it.
And control is total: I choose the model, the quantization, the routing, the retry behaviour, and I
can reason about every failure because there’s no opaque layer between me and the metal. For a
platform whose entire pitch is “production AI that runs on your own hardware,” that ownership isn’t a
nice-to-have. It’s the product.
Live on the platform
This isn’t a lab demo. Real sites serve real traffic off this estate right now:
- www.inkypyrus.com
- afrecosoil-wp.inkypyrus.com — AfrEcoSoil, mirrored on this estate