The Call Screener
When my phone rings from a number I don’t recognise, I have a choice: interrupt whatever I’m doing
to find out if it matters, or ignore it and risk missing something that does. The Call Screener removes
the choice. I forward the call, an AI voice agent answers, talks to the caller, figures out who they
are and what they want, and drops a transcript into Telegram. I decide on my own schedule whether it
was worth interrupting my day for. It runs on my own hardware and I dogfood it daily.
This is not a voicemail box with a transcript bolted on. Voicemail is passive — it waits, records a
monologue, and hopes the caller leaves something useful. A screener is an agent: it holds a real
two-way conversation, asks clarifying questions, and applies judgment about what to surface. That
distinction is the whole point, and it is also where all the engineering difficulty lives.
The chain
An inbound call passes through four moving parts before it becomes a message in my pocket. Each one
is a separate failure domain, which matters because telephony is unreliable in ways that web requests
are not.
caller ──► Twilio ──► voice-gateway ──► grok-bridge ──► Telegram
(telephony) (FastAPI/uvicorn) (realtime voice) (transcript)
│
▼
Voicebox TTS ◄── Hermes voice brain
(GPU, sg1) (OpenAI-compatible proxy)thing that leaves the building is the call itself, over Twilio.
Twilio handles the telephony — it owns the PSTN relationship and streams the call’s audio over a
media WebSocket. The voice-gateway, a small FastAPI/uvicorn service on the GPU node, is the
broker: it accepts Twilio’s media stream, forwards caller audio to the voice model, and pipes synthesised
speech back. The grok-bridge is the agent itself — a realtime voice loop built on x.ai’s
Grok that actually converses with the caller as a screener. Behind it, an OpenAI-compatible
chat-completions proxy I call the Hermes voice brain shapes the screening logic, and Voicebox neural TTS
gives the agent a voice. On hangup, the bridge ships the transcript to Telegram.
Latency is the product
People will forgive an AI a great deal, but they will not forgive a pause. In text, a half-second of
“thinking” is invisible. On a phone call, 700ms of silence after someone finishes a sentence reads as
“the line dropped” or “this thing is broken,” and the caller starts talking over the agent. Turn-taking
is the hard problem of voice agents, and it is unforgiving.
The screener has to do three timing-sensitive things well. It needs end-of-speech
detection — knowing the caller has actually finished, not just paused mid-sentence to think —
because cutting someone off is worse than waiting. It needs barge-in — when the caller
starts speaking while the agent is mid-sentence, the agent must stop talking immediately, the way a
human would, rather than steamrolling through its scripted line. And it needs the whole
listen-think-speak loop to feel like sub-second turnaround, which means streaming partial results rather
than waiting for a complete response before the first phoneme of speech.
Getting barge-in right is mostly about being willing to throw away work. The naive loop generates a
full response, synthesises it, and plays it. The responsive loop has to be interruptible at every stage:
the moment inbound audio energy crosses a threshold, you cancel in-flight TTS and the model generation,
flush the output buffer, and start listening again. Below is the shape of that interrupt handling,
stripped to its essentials.
# voice-gateway: barge-in handling on the Twilio media stream
async def on_caller_audio(chunk: bytes, session: CallSession):
if session.agent_is_speaking and is_speech(chunk):
# Caller interrupted — kill our output immediately.
await session.cancel_tts() # stop Voicebox playback
await session.cancel_generation() # abort the in-flight model turn
session.agent_is_speaking = False
session.vad.push(chunk)
if session.vad.end_of_utterance(): # debounced silence, not a mid-sentence pause
text = await session.transcribe()
await session.respond(text) # streams tokens -> TTS -> TwilioThe end_of_utterance() call hides the real tuning work. Too aggressive and you cut
people off; too lenient and the agent feels slow and dim. There is no universally correct silence
threshold — it depends on the caller’s cadence, the carrier’s jitter, and how much background noise the
line carries.
What a screener should and shouldn’t decide
The temptation with an autonomous agent is to let it do more: book the meeting, answer the question,
make the call. I deliberately kept the screener narrow. Its job is to understand and relay, not
to act. It establishes who is calling and why, and it hands me the decision. It does not commit
me to anything, quote a price, or make a promise on my behalf, because an agent that can make commitments
on a phone line is an agent that can be socially engineered into making the wrong one.
The right autonomy boundary for a screener is “gather and summarise,” not “decide and act.” The cost
of the agent being too cautious is that I read a transcript I didn’t strictly need. The cost of it being
too eager is that it speaks for me. Those costs are not symmetric, so I tuned toward caution.
Failure modes I actually hit
Telephony breaks in ways that surprise web engineers. The recurring problems weren’t in the AI —
they were at the edges:
- Carrier quirks: some carriers inject a second or two of dead air or comfort noise
at call setup, which a naive end-of-speech detector reads as the caller having finished before they’ve
said a word. The agent greets, hears “silence,” and awkwardly greets again. - Silent callers: robocallers and misdials connect and say nothing. Without a
timeout-and-graceful-exit path, the session hangs holding a GPU-backed model open for a caller who was
never there. - Media-stream drops: the WebSocket from Twilio can stall without closing cleanly, so
the gateway needs liveness checks rather than trusting the socket to tell you it’s dead.
None of these are glamorous. All of them are the difference between a demo and something you actually
leave pointed at your real phone line.
Live / proof
It’s live — and you can call it yourself. Dial the screener and have a conversation with the agent:
It will screen you the way it screens anyone who reaches my line — work out who you are and what you
want — and drop a transcript into my Telegram the moment you hang up. It runs every day on hardware I
own; the only thing that leaves the building is the call itself.