Hermes

My self-hosted models live on a GPU in a room I can walk into. Hermes is how they live in my
pocket too. It’s a messaging gateway that bridges Telegram and WhatsApp to my local Qwen-class models, so
I can talk to my own private inference from the same chat apps I already use — and no third-party AI
service ever sees a word of it, and there’s no per-token bill at the end of the month.

The motivation is plain. The convenience of chatting with an assistant from your phone shouldn’t
require shipping every message to someone else’s servers under someone else’s terms. If the model is
mine and the hardware is mine, the only thing standing between “private model on a GPU” and “private
model in my hand” is a gateway. Hermes is that gateway, and it turns out the model is the easy part.

What Hermes actually does

Hermes presents two chat surfaces — Telegram and WhatsApp — and routes each to the local inference
node. The two channels are not treated identically. Each has its own profile, because the right model and
the right context settings differ by use: a quick mobile back-and-forth wants a snappy chat model with a
modest context window; a longer working session wants a different target. So Hermes does per-channel model
targeting with pinned context windows rather than pretending one configuration fits both.

  Telegram ─┐
            ├─► Hermes gateway ─► proxy ─► local Qwen-class models (inference node)
  WhatsApp ─┘     per-channel        │
                  profiles           └─ pinned context window per channel

A profile is just a small declarative block — which model to hit, what context window to pin, which
channel it belongs to. Keeping it declarative means adding a channel or retargeting a model is a config
change, not a code change.

# hermes profiles — per-channel model targeting + pinned context
[channel.telegram]
model        = "qwen-chat-27b"
context_pin  = 8192          # cap below the model's train length on purpose
system       = "concise mobile assistant"

[channel.whatsapp]
model        = "qwen-chat-27b"
context_pin  = 4096          # tighter — WA sessions run long and chatty
system       = "concise mobile assistant"

The honest part: WhatsApp is the hard one

Telegram has a clean bot API. WhatsApp does not, and that asymmetry is most of the engineering. The
WhatsApp side rides on a session that has to be paired to a device and kept alive, and
that session is fragile in ways a bot token never is. Sessions expire. A pairing can be invalidated
server-side without warning. A reconnect can come back in a state that looks alive but silently isn’t
delivering. None of these announce themselves cleanly, so you can’t trust “the process is running” as a
proxy for “the channel works.”

That fragility is why the gateway needs a real watchdog rather than just a restart policy. A bare
restart=on-failure only catches a process that crashes. The failure mode that bites
is the gateway that’s still up, still holding a socket, but whose WhatsApp session has gone stale — it
fails the liveness expectation without ever exiting. So the watchdog checks channel health, not just
process liveness, and restarts on a stale session as aggressively as on a crash.

# /etc/systemd/system/hermes.service
[Unit]
Description=Hermes messaging gateway
After=network-online.target

[Service]
ExecStart=/opt/hermes/bin/hermes --config /etc/hermes/config.toml
Restart=always
RestartSec=5
# liveness: watchdog must be pinged or systemd restarts the unit
WatchdogSec=60
StartLimitIntervalSec=0

[Install]
WantedBy=multi-user.target

Inside the service, the loop pets the watchdog only when the channels are genuinely healthy — a sent
heartbeat that round-trips — so a stale-but-running gateway gets reaped and reconnected instead of sitting
there looking fine while dropping messages.

Context windows vs train length

A subtle trap with local models is conflating the context window you can set with the one you
should. A model trained on sequences up to some length will technically accept more, but quality
degrades as you push past where it actually learned to attend — coherence frays, it loses the thread,
recall gets unreliable. So Hermes pins context windows deliberately below the train length rather than
maxing them out. A pinned window is also a predictable memory footprint, which matters when several
channels share one GPU.

The lesson that generalises: the limit the config lets you set is not the limit the model was built
for. Pinning context below train length trades a bit of headroom for output you can actually trust —
which, on a model you’re putting your own name behind, is the right trade.

Plumbing

The connective tissue between the gateway and the inference node is deliberately boring — a thin proxy
layer (socat and friends) that brokers connections between Hermes and the local model
endpoints. Boring is the point: the interesting reliability work lives in the watchdog and the session
handling, not in the wiring, and the wiring should never be the thing that wakes me up.

What it buys

Two things, both structural. Privacy: messages go from my phone to my GPU and back,
and no third-party AI provider is in the loop — there’s no terms-of-service change that can suddenly start
training on my conversations. Cost: there’s no per-token meter running. The marginal cost
of a message is a flicker on a power bill, which means I use my own assistant freely instead of rationing
prompts against an invoice. That’s the whole reason to put your own models in your pocket.