Bedrock RAG Post-Mortem

This is the honest one. I built a private RAG expert on AWS Bedrock — a real, globally fast,
IP-controlled knowledge assistant over a proprietary methodology — and then I turned it off. Not because
it didn’t work. Because the ongoing cost didn’t justify the value at that stage, and a managed tool did a
good-enough version for a fraction of the effort. Knowing when not to keep something running is
a senior instinct, and this is the post-mortem that documents the judgment call.

What I built

The brief was a private “expert” over roughly 35,000 words of proprietary methodology — a body of
knowledge the owner wanted queryable, but emphatically not leaked into a public model or sitting on
infrastructure they couldn’t reason about. The build was a fairly clean managed-RAG stack:

  user ──► Cloudflare Worker (edge proxy) ──► Bedrock Knowledge Base
                                                    │  retrieve top-k chunks
                                                    ▼
                                              Claude 3.5 Haiku
                                                    │  grounded answer
                                                    ▼
                                              response (globally fast)

The Bedrock Knowledge Base handled ingestion, chunking, embedding, and vector retrieval — the managed
pieces I didn’t want to hand-roll for a corpus this size. Claude 3.5 Haiku was the
generation model: cheap, fast, and more than capable of grounded answers over retrieved chunks. A
Cloudflare Worker sat at the edge as a proxy, which bought two things — global low
latency regardless of where the user was, and a control point where I owned auth and rate limiting rather
than exposing Bedrock directly.

What worked

Quite a lot, honestly. The retrieval quality over a focused 35k-word corpus was good — small,
coherent knowledge bases are where RAG genuinely shines, because there’s little to confuse the retriever.
Haiku’s answers were fast and well-grounded. The edge proxy made it feel instant globally. As a piece of
engineering it did exactly what it was specified to do, and the IP stayed controlled: the corpus lived in
infrastructure the owner could account for, not pasted into a public chat tool.

Why I turned it off

RAG has a cost structure that’s easy to underestimate when you’re admiring the architecture. There are
three meters running, and only one of them is the obvious one:

• Embedding: a one-time cost to ingest the corpus, plus re-embedding whenever the
  source changes. Modest here, given the size.
• Storage / vector index: an ongoing cost that accrues whether or not anyone
  asks a single question. The knowledge base bills for existing, not just for being used.
• Inference: per-query generation. Cheap on Haiku, but non-zero and scaling with use.

Add it up against the actual usage at that stage — real but not high-volume — and the maths stopped
making sense. I was paying a standing monthly cost for a managed knowledge base and the surrounding
plumbing to serve a query volume that didn’t warrant it. The system wasn’t failing; it was
over-built for its demand. That’s a quieter problem than an outage, and an easier one to let run
out of sunk-cost inertia, which is exactly why it’s worth being disciplined about.

The hard part of build-vs-buy isn’t the initial decision — it’s the courage to reverse it. I built a
thing that worked and then admitted it shouldn’t keep running. Sunsetting a working system on cost is not
a failure; refusing to is.

The pragmatic follow-on

The need didn’t disappear, so I replaced the bespoke stack with the right-sized tool. A live knowledge
assistant — a tax-regulations helper — now runs on Google NotebookLM. I want to be
precise about what that is, because the easy move here would be to overclaim. NotebookLM is a
managed tool. It is not hand-built RAG. I did not engineer its retrieval pipeline; I chose it,
loaded the source material, and configured it. The engineering decision was the selection, not
the construction.

And that was the correct decision. For a knowledge assistant at this scale and budget, NotebookLM
delivers most of what the Bedrock stack delivered — grounded answers over a controlled document set —
with effectively none of the standing infrastructure cost and none of the ops burden. The cost/effort
tradeoff lands clearly in its favour. Picking it was deliberate, not lazy.

When to build, and when to buy

The whole point of this write-up is the judgment, so here it is explicitly. Build a custom RAG stack
when the constraints actually demand it — strict data residency or IP control a managed tool can’t meet,
volume high enough that per-query economics favour owning the inference, deep integration that off-the-
shelf tools can’t reach, or retrieval behaviour you need to control precisely. Buy a managed tool when
the corpus is small and stable, the volume is modest, and the value is in answers rather than in owning
the pipeline.

The Bedrock build sat in an awkward middle: the IP-control argument was real, but the volume never grew
into the cost, and the data-residency bar turned out to be one a managed tool could clear acceptably. So
the senior move wasn’t to defend the thing I’d built — it was to read the meter honestly and pick the
boring, cheaper option that was good enough. The most valuable instinct here isn’t knowing how to build
RAG. It’s knowing when not to.

Live / proof

• The Bedrock expert: built, run, and deliberately sunset on cost — documented above.
• The tax-regulations assistant: live today on Google NotebookLM, a managed tool chosen on purpose.