Operations practice library32 practices
Service level and cycle time
Monitor end-to-end completion time and stage-level latency against the approved service level.
Volume capacity and backlog
Track demand, throughput, concurrency, backlog age, and peak-event capacity.
Handoff and exception flow
Verify routing, referrals, exceptions, and downstream acceptance.
Customer communication delivery
Monitor whether approved communications are timely, complete, accessible, and correctly delivered.
Production quality sampling
Review representative live outputs and decisions using a versioned rubric.
Error rework and reopen rate
Measure corrections, rework, reopened cases, downstream defects, and error severity.
Outcome and counter-metric monitoring
Track intended business benefit together with quality, fairness, and consumer counter-metrics.
Segment and edge-case outcomes
Compare results across products, states, channels, groups, severity bands, and difficult cases.
Schema and data-quality monitoring
Detect missing fields, type changes, failed joins, duplicates, staleness, and source reconciliation breaks.
Input and feature drift
Measure distribution change in material inputs and embeddings.
Output score and calibration drift
Track score, class, amount, confidence, and observed-to-expected behavior.
Concept and outcome drift
Detect changes in the relationship between inputs, decisions, and actual outcomes.
Grounding and factuality monitoring
Sample material statements for support by authoritative sources and record unsupported claims.
Retrieval and citation health
Monitor retrieval success, source freshness, access controls, broken links, and citation fidelity.
Prompt output and policy compliance
Check required format, prohibited content, tone, disclosure, and instruction adherence.
Foundation model and vendor change watch
Track provider releases, model deprecations, behavior changes, pricing, limits, and incidents.
End-to-end task success
Measure completion of the approved objective, including final state and side effects.
Tool call and action accuracy
Monitor tool selection, parameters, target records, sequence, and consequential actions.
Authority approval and segregation
Verify permissions, approval gates, delegated limits, identity, and separation of duties.
Loop budget and termination control
Detect repeated actions, runaway loops, excess tokens or cost, timeouts, and failure to stop.
Override and disagreement monitoring
Track human disagreement, override reason, outcome, and whether the AI influenced the reviewer.
Escalation quality and workload
Monitor uncertainty routing, escalation appropriateness, queue age, reviewer capacity, and closure.
Automation bias and reviewer effectiveness
Sample accepted recommendations to detect rubber-stamping and assess information quality.
Complaint appeal and correction signals
Connect complaints, appeals, disputes, corrected outcomes, and accessibility issues to AI-influenced work.
Security and misuse signal monitoring
Detect prompt injection, malicious content, extraction attempts, access anomalies, and unsafe requests.
AI incident triage and response
Classify events by consumer, financial, legal, privacy, security, and operational impact.
Dependency health and safe fallback
Monitor models, retrieval, tools, networks, identity, and upstream data with tested degraded modes.
Recovery reconciliation and rollback
Verify restoration, rollback, duplicate prevention, record reconciliation, and manual continuity.
Change intake and materiality assessment
Record model, prompt, data, tool, workflow, policy, authority, and vendor changes before release.
Regression revalidation and approval
Run the test suite proportionate to impact and renew required approvals.
Controlled release and canary monitoring
Use staged exposure, version control, release evidence, enhanced monitoring, and rollback triggers.
Vendor service exit and decommissioning
Monitor contractual service, incidents, concentration, audit evidence, exit readiness, and safe retirement.