LLM application integration

LLM application integration means shipping product features backed by language models: chat, summarization, classification, retrieval-augmented generation (RAG), and **tool use** (function calling). Model output is probabilistic—design for bounded behavior, observability, evaluation, and safe failure modes.

| Layer | Design focus | |-------|--------------| | Context | Token budgets, retrieval quality, prompt templates, conversation memory | | Tools | Narrow APIs with schemas, authz, idempotency, audit logs | | Streaming | Partial UX, cancellation, backpressure, error mid-stream | | Reliability | Retries, timeouts, fallbacks, circuit breakers, degraded responses | | Safety | PII filtering, injection resistance, refusal policies, human escalation | | Cost | Model routing, caching embeddings, batching, usage caps |

					User message → guardrails → retrieve context → LLM → validate structured output →
authorized tool call → persist audit event → stream to client
				

RAG interviews: chunking strategy, embedding refresh, citation of sources, handling stale docs, and measuring retrieval precision—not only "we use a vector DB."

Evaluations: regression suites for safety, tool selection, latency, cost, and golden prompts before rollout; shadow traffic in production when possible.

On interviews: design a support bot that can read orders but not issue refunds without confirmation. Name authorization boundaries, eval plan, and what happens when the model hallucinates an order ID.

Common pitfalls: unbounded admin tools; trusting system prompts for security; no logging of prompts/responses (with redaction); single model for all tasks; ignoring latency p95 on mobile.

The trade-off is capability (more tools, longer context) versus attack surface, cost, and operational complexity.

Checklist:

• Scope tools to least privilege.
• Structured outputs with validation before side effects.
• Offline and online eval gates.
• Redacted observability and cost dashboards.