There are two ways to ship an AI feature inside a product. The first — and still the most common — is to take an existing app and bolt an LLM call onto a button somewhere. Click the button, send the user's input to OpenAI, show the response. The second is to rebuild the product so that AI is assumed in the data model, the UI, and the infrastructure from day one.
The first path is faster to demo. The second is cheaper, more reliable, and more defensible. We have shipped both, watched both in production, and have the invoices to prove the difference. Here is what changes when you build AI-native instead of AI-retrofit.
Retrieval is a product decision, not an infrastructure decision
Most retrofit AI features stuff user input, a system prompt, and a rushed RAG call into a single LLM request. The LLM is doing three jobs at once: understanding the question, retrieving the answer, and composing the response. It is bad at all three when it does them together.
An AI-native architecture separates retrieval from generation explicitly. A typical request goes:
- Classify the intent — is this a factual question, a generation task, or a tool call?
- Retrieve relevant data from a vector database (we use pgvector in Postgres — no extra infrastructure) with an actual reranking step
- Compose the response with structured output (Claude's tool-use or OpenAI's structured outputs)
This breaks one slow, expensive, unreliable call into three fast, cheap, testable calls. In a recent engagement, the total cost per request dropped from $0.11 (retrofit) to $0.03 (native), and latency dropped from 6 seconds to 1.8 seconds. The feature got cheaper and faster at the same time, because we were no longer asking the LLM to be smart about things it did not need to be smart about.
Evaluation comes first, not last
Retrofit AI features ship when they "seem fine" to the product manager. AI-native features ship when they pass an evaluation harness.
The evaluation harness is a set of representative inputs, expected outputs, and metrics. In our builds it lives in the repo as evals/ with 30–50 test cases per feature. Every pull request that touches AI code runs the evals in CI. Regressions block the merge.
The metrics depend on the feature. For document Q&A it is retrieval precision and recall on a labelled set. For summarisation it is ROUGE plus human spot-checks on a rotating 10% sample. For structured extraction it is field-level accuracy against a golden set.
This sounds like heavy process. It is not. Writing 30 evals takes a junior engineer one day. The payoff is that you catch an hallucination regression the first time it appears, instead of when a customer emails you about it three weeks later. We have seen one engagement where a model version bump silently broke a feature for 11 days before a customer noticed. That did not happen because the team did not care — it happened because there was no eval harness to catch it.
Cost control happens in code, not in the billing portal
Every retrofit AI feature we have audited has one thing in common: the OpenAI bill is larger than it needs to be, often by 3–5x. The reasons are always the same:
- The same system prompt is appended to every request with no caching
- Chat history grows unbounded and gets resent in full each time
- The biggest, most expensive model is used by default even for trivial classification tasks
- Retries happen on HTTP timeouts without any backoff or deduplication
- Batch-able requests run sequentially
An AI-native architecture puts a thin "LLM gateway" layer between your code and the provider. We use a custom wrapper — it could be LiteLLM or your own — that:
- Caches idempotent responses for 24 hours
- Routes small requests to small models (Haiku or GPT-4o mini) and big requests to big models (Sonnet or GPT-5)
- Enforces context-window budgets and refuses requests that would exceed them
- Batches when the provider supports it
- Emits cost metrics to your observability stack, tagged by feature
In every engagement where we have retrofitted this layer onto an existing product, we have seen 40–70% cost reduction with no quality loss. The reason is not that the retrofit team was incompetent — it is that the original product was not designed with these controls in mind.
Observability is non-negotiable
Retrofit AI features are observed through the web app's existing tools — Sentry for errors, Mixpanel for product analytics. That is not enough for AI.
AI-native features emit structured traces to a dedicated AI observability tool. We use Langfuse (open source, self-hostable) or Braintrust (commercial, hosted). Every LLM call gets a trace with:
- Full input and output
- Model name and version
- Token counts and cost
- Latency broken down by component
- User-level tagging so you can query "show me all slow requests for Enterprise customers last week"
This turns debugging from "that feature seems slow sometimes" into "here is the exact request, here is the prompt we sent, here is the response we got, and here is why it took 14 seconds." Without this, you are debugging blind and every production incident eats a day.
Data model changes are the biggest unlock
The deepest architectural difference between retrofit and native is in the data model itself.
A retrofit AI feature on an e-commerce product might ingest product descriptions by calling an embedding API every time a product is viewed. It works, it is slow, and it is expensive. An AI-native version adds embedding vector(1536) as a column on the products table, populates it with a trigger on insert and update, and does retrieval as a normal SQL query with ORDER BY embedding <=> :query_embedding. The AI feature becomes a database query.
The same pattern applies to conversation history, cached generations, user preferences learned from behaviour, and dozens of other places where "AI state" needs to live. When you treat AI data as a first-class citizen of your schema, you get indexing, querying, backup, migration, and transactional consistency for free. When you shove it into a separate vector database or an object store, you pay for all of that again.
What it looks like in practice
A concrete example from a recent build. The client runs a legal-tech SaaS with about 400 paying customers. Their retrofit AI feature — a document Q&A — cost $0.08 per question, took 4 seconds, and had a user-reported hallucination rate of roughly 6%. We rebuilt it native over three weeks. The result:
- Cost per question: $0.021 (74% reduction)
- Median latency: 900ms (77% reduction)
- User-reported hallucination rate: under 1% (measured over 8 weeks post-launch)
- Monthly AI bill: dropped from $4,800 to $1,260
The work included migrating to pgvector from a standalone Pinecone instance, rebuilding the retrieval pipeline with reranking, adding an LLM gateway with caching and model routing, and shipping 40 evals. The client recovered the engagement cost through their reduced monthly AI bill in under a month.
Where retrofit is still the right call
Not every AI feature justifies going native. Retrofit is still the right call when:
- The feature is experimental and will be cut if it doesn't land in 30 days
- Your user base is small enough that the bill will never matter
- You are pre-product-market-fit and the cost of rewriting later is lower than the cost of delaying now
- The LLM is doing a genuinely one-off task that won't repeat or scale
For everything else — any AI feature you expect to still be in production in six months — building native from the start is cheaper than retrofitting later. Most retrofit-to-native migrations we take on could have been avoided if the original team had invested the extra week up front.
If you are evaluating whether an AI feature you are planning should be built native or retrofit, we run a 2-week paid spike that ends with a functional prototype, a cost projection, and a recommendation. Most clients use it as a decision gate before committing larger engineering budget.