A practical field manual for engineering teams who want AI features that survive real users, incidents, and budgets — not just demo day.
AI Best Practices for Engineering Teams: From Prompt Experiments to Platform Discipline
Most AI initiatives don't fail because the model is bad. They fail because the surrounding system is missing: no clear contracts, no evaluation, no governance, and no path from one clever notebook to a repeatable product capability.
This guide is a field manual for engineering teams who want AI features that survive real users, real incidents, and real budgets — not just demo day.
An AI contract is a one-pager every feature must have before any prompt work begins. It defines:
- Task statement — What the AI is allowed to do (and not do).
- Inputs — Exact fields, types, and sources (for example,
sanitized Markdown, max 4k chars).
- Output schema — JSON / markdown / text shape your code will parse.
- Quality target — What “good enough” means (accuracy, coverage, refusal rate).
- Latency budget — For example,
P95 ≤ 2.5s.
- Cost budget — For example,
≤ $0.005 per request at forecast traffic.
A team shipping an “AI incident summary” feature skipped this step. They let the model “summarize any incident data” with no schema. Within weeks:
- Some outputs were bullet lists, others were paragraphs, some included tables.
- The notifications system broke whenever the model decided to “be helpful” and add Markdown headings.
- Product managers couldn’t tell whether things were improving — there was no quality target.
They rewrote the feature around a contract:
Task: Summarize a production incident for engineers.
Inputs: Root cause, impact, timeline (ISO timestamps), remediation, incident tags.
Output: JSON with fields: title, short_summary (≤ 240 chars), long_summary (≤ 800 chars), impact_level (LOW|MEDIUM|HIGH).
Quality: 90% of summaries accepted without edit by on-call engineer.
Latency: P95 ≤ 2s.
Cost: ≤ $0.003 per incident.
Breakage stopped overnight because downstream code finally knew what to expect.
- Make the contract a template in your repo (for example,
docs/ai/contracts/incident-summary.md) and require it in PRs for new AI features.
- Add a checkbox in PR descriptions: “AI contract attached and reviewed by product + eng.”
- Revisit contracts every quarter when usage, data, or risk changes.
Prompts live in a vendor UI, people tweak them during incidents, and nobody knows why behavior changed last week.
Real-world example
A support team had a “reply draft” assistant. A well-meaning engineer changed the tone phrasing in the prompt from “professional and concise” to “friendly and conversational.” Conversion dipped on enterprise accounts — customers started complaining about “too casual” responses — but there was:
- No commit,
- No diff,
- No quick rollback.
- Store prompts in your repo (for example,
prompts/support/reply-draft.prompt.md).
- Reference them by versioned ID in code, like
SUPPORT_REPLY_V3.
- For every prompt change, require:
- A short changelog note (for example, “tightened escalation instructions for security issues”).
- A link to evaluation results (see section 4).
- Create a
prompts/ folder with:
*.prompt.md — raw prompt text.*.policy.md — safety/guardrail instructions.CHANGELOG.md — bullet list of meaningful changes.
- Enforce a prompt diff review: prompt changes must be reviewed by at least one engineer and one domain owner (for example, a support lead).
For retrieval-augmented generation (RAG), prompt tuning on top of bad retrieval is like polishing a cracked lens.
A team shipped a RAG “runbook assistant” for on-call engineers. It hallucinated outdated mitigation steps. Root cause was not the model; it was:
- Runbooks older than 18 months were never re-embedded after updates.
- Chunking split tables across chunks, losing context.
- Ranking ignored incident severity; archived incidents were surfacing over fresh ones.
After they fixed retrieval:
- Freshness rule: re-embed within 5 minutes of document updates.
- Chunking benchmarks: tuned for whole procedures, not arbitrary token sizes.
- Ranking: boosted docs tagged with the active service and last 6 months of incidents.
Hallucinations dropped by more than 40%, with no model change.
- Freshness
- Chunking
- Ranking
- Evaluation
Goal: your system never crashes because of AI output shape.
- Test that every output parses into your schema.
- Test refusal handling (nulls, fallback messages) without a real model call.
Goal: ensure the model is “good enough” for key tasks.
- Keep a set of labeled examples (input + ideal output).
- Score them automatically (for example, with another model or rule-based scoring).
- Track per-change: “Did this prompt or model version improve or regress?”
Goal: check end-to-end behavior in real user flows.
Example: for an “incident summary + Slack notification” flow, tests should verify:
- The summary is generated.
- The Slack message uses the short summary.
- The incident link is correct.
- For “LOW” incidents, notifications don’t ping the exec channel.
- Start with 10–20 curated test cases per critical feature instead of aiming for hundreds.
- Wire evaluation into CI so any prompt or model change runs the suite before merge.
Accuracy alone is vanity if latency, cost, or policy risk are unacceptable.
A customer-facing “architecture review assistant” had great semantic scores but:
- P95 latency was 8–10 seconds.
- Token burn was 3–4× budget for big accounts.
- Security flagged it because logs were sending full code samples to a third-party.
Once they built a scorecard per release, they exposed:
- Technical metrics — accuracy, latency, error rate.
- Business metrics — daily active users, completion rate, support escalations.
- Risk metrics — PII exposure incidents, policy violations, manual escalations.
They then explicitly traded a slight drop in accuracy for 2× faster responses and 50% lower cost, which turned out to be the right business call.
- Quality: evaluation score, acceptance / edit-rate.
- Reliability: P95 latency, timeout rate, fallback activation rate.
- Cost: average cost per request, per active user, per month.
- Safety: policy violation rate, manual escalations.
- Business: user adoption, impact metric (for example, tickets resolved per agent).
You don’t need a committee; you need clear ownership.
- AI platform owner — chooses providers, manages SDKs, enforces contracts.
- Risk/Security contact — signs off on data flow and vendors.
- Product owner — signs off on UX and user impact.
Keep a single ai/systems.yaml file listing:
- Each AI workflow (for example,
incident_summary_v2).
- Owner, data sources, model(s) used, and fallback behavior.
- Links to prompt, contract, and evaluation dashboards.
This file becomes your source of truth during incidents, audits, and vendor changes.
Blindly using the “best” model for everything is wasteful and risky.
- Tier 0 (low risk, high volume) — log classification, tag suggestions, deduping. Use small, cost-efficient models.
- Tier 1 (medium risk) — internal summaries, query rewriting. Use mid-tier models with guardrails.
- Tier 2 (high risk, external) — customer-visible decisions, security advice, compliance-sensitive output. Use premium models, stricter evaluation, and possibly human-in-the-loop.
Implement routing via a config file (for example, ai/routing.json) rather than conditionals scattered through code. That makes it easy to change behavior without redeploying everything.
- Clearly label AI outputs (“AI-generated summary — last updated 2 min ago”).
- Show confidence and completeness hints (for example, “Based on 3 incidents and 2 runbooks”).
- Provide simple controls:
- “Regenerate with more detail”
- “Include logs from the last 30 minutes”
- “Escalate to human reviewer”
In an internal incident tool, adding a small “What data was used?” toggle reduced complaints about hallucinations. Engineers could see exactly which runbooks and incidents the assistant used.
Assume:
- Any external content might be prompt injection.
- Any tool call could be abused if permissions are too broad.
- Input sanitization — strip or sandbox user-provided instructions before passing them to the model.
- Tool allowlists — for example, the “architecture advisor” can read docs and schemas but cannot run CI jobs or modify configs.
- Response guards — never directly execute shell commands or SQL produced by the model; require explicit human confirmation for dangerous actions.
A team added a “deploy with AI” helper. They blocked direct shell execution and forced:
- AI suggests a deploy command.
- Human sees a diff and the command.
- Human clicks “Approve & run.”
This added 1–2 seconds but avoided an entire class of “the model misread the ticket” incidents.
- Short feedback loops — weekly review of eval metrics and incident learnings.
- Recorded “prompt post-mortems” — for every major issue, capture:
- The bad input, trace, and output.
- Which contract or guardrail failed.
- The new test case added to prevent recurrence.
- Shared playbook — lightweight
ai/runbooks/*.md for:
- Model provider outages.
- Cost spikes.
- Elevated policy violations.
Over time, this turns AI work from “clever prompts” into a disciplined engineering practice — the real competitive advantage in 2026.
- Day 1–2
- Write contracts for your top one or two AI features.
- Move prompts into source control.
- Day 3–4
- Build a 10–20 case eval suite for each feature.
- Add at least one retrieval benchmark if you’re using RAG.
- Day 5
- Create a minimal
ai/systems.yaml and routing config.
- Agree on a monthly AI scorecard review.
If you do only this, you’ll already be ahead of most teams shipping AI in production.
