AI in production

Moving from a prototype to a production AI system requires treating the model as one component of a larger distributed system.

Important concerns include:

Reliability

LLM output is probabilistic. Critical workflows should therefore use validation, retries, deterministic checks, and explicit failure handling.

Security

Never assume model-generated instructions are trustworthy.

Important controls include:

• Authentication
• Authorization
• Tool-level permissions
• Input validation
• Output validation
• Secret isolation
• Audit logging
• Prompt-injection defenses

Observability

Measure more than API latency.

Useful metrics include:

• Request latency
• Token usage
• Model cost
• Retrieval precision
• Tool-call success rate
• Agent completion rate
• Failure rate
• Human intervention rate

Cost

An agent may make multiple model and tool calls for a single user request.

A useful cost model is:

Total Cost
=
LLM Calls
+ Embedding Calls
+ Retrieval
+ Reranking
+ Tool Execution
+ Infrastructure

Evaluation

Traditional unit tests are not sufficient for many AI systems.

Production systems should evaluate:

• Answer correctness
• Retrieval quality
• Tool selection
• Tool arguments
• Safety behavior
• Hallucination rate
• Task completion
• Regression across model changes