Shipping AI features without the hype

AI features are easy to demo and hard to ship. The gap between an impressive prototype and something people rely on daily is filled with unglamorous work: evaluation, guardrails, latency, cost, and honest interface design about what the system cannot do.

Start with friction, not with the technology

The best AI features are usually invisible. A smarter default, a search that understands what was meant, a first draft that saves someone ten minutes.

The worst are a chat box bolted onto a product nobody wanted to have a conversation with. "Where could we add AI" is the wrong question. "Where do our users do something tedious that a model is genuinely good at" is the right one.

Build the evaluation before the feature

This is the step most teams skip and the one that decides whether the feature survives contact with users.

If you cannot measure whether an answer is good, you cannot tell whether a prompt change improved things or quietly broke them. A useful harness is not elaborate:

1. Collect thirty to fifty real examples, including the awkward ones.
2. Write down what a good answer looks like for each.
3. Run them automatically on every change.
4. Track the failures over time rather than the successes.

Without this you are tuning by vibes, and vibes do not survive a model version change.

Design for the model being wrong

It will be. The question is what happens then.

• Make it obvious which content was generated rather than authored.
• Make correcting it faster than doing it manually, or people will do it manually.
• Show the source where there is one, so a claim can be checked.
• Never let a model take an irreversible action without confirmation.

An assistant that is right most of the time and easy to correct is useful. One that is right most of the time and hard to correct is worse than nothing, because it transfers the checking burden onto the user without telling them.

Latency and cost are product decisions

A model call that takes eight seconds changes the interface around it — you now need progressive output, a meaningful loading state, and a plan for timeouts.

Cost scales with usage, unlike most features. A generous free tier plus a per-token model is a combination worth modelling before launch rather than discovering at the end of the first busy month.

Know when not to

Plenty of problems presented as AI problems are better solved by a database query, a decent search index or a well-designed form. Those are cheaper, faster, deterministic and never make things up.

Reaching for a model where a lookup would do is the most common and least discussed failure in this area.

If you are scoping something in this space, our MVP cost breakdown is a reasonable starting point — and the discipline it describes, building the narrowest thing that answers your riskiest question, applies doubly when the technology itself is the risk.

How do you know if an AI feature is worth building?

Ask where your users already do something tedious that a model is genuinely good at, rather than where AI would look most impressive. The best AI features are usually invisible — a smarter default, a search that understands intent, a draft that saves ten minutes. A chat box bolted onto a product nobody wanted to converse with is the common failure.

Why build the evaluation before the feature?

Because without it you cannot tell whether a prompt change improved things or quietly broke them, and you will be tuning by intuition. A useful harness is not elaborate: thirty to fifty real examples including the awkward ones, a written description of what a good answer looks like, and an automatic run on every change. Track the failures over time rather than the successes.