Learn how to monitor AI models in production. Track performance, detect drift, and ensure model reliability with comprehensive observability strategies.
AI Observability and Monitoring: Tracking Model Performance in Production
Monitoring AI models in production is crucial for maintaining quality and detecting issues early. This guide covers essential observability strategies.
- Latency: Response time per prediction
- Throughput: Predictions per second
- Error Rate: Failed predictions
- Success Rate: Successful predictions
import time
from prometheus_client import Counter, Histogram
prediction_latency = Histogram(
'model_prediction_latency_seconds',
'Time spent processing predictions'
)
prediction_errors = Counter(
'model_prediction_errors_total',
'Total prediction errors'
)
def track_prediction(func):
def wrapper(*args, **kwargs):
start = time.time()
try:
result = func(*args, **kwargs)
prediction_latency.observe(time.time() - start)
return result
except Exception as e:
prediction_errors.inc()
raise
return wrapper
- Accuracy: Overall correctness
- Precision/Recall: Per-class performance
- F1 Score: Balanced metric
- Confidence Scores: Prediction confidence distribution
from evidently import ColumnDriftMetric, DataDriftProfile
def detect_drift(reference_data, current_data):
drift_profile = DataDriftProfile()
drift_profile.calculate(reference_data, current_data)
drift_metrics = drift_profile.get_metrics()
for metric in drift_metrics:
if metric.drift_detected:
alert(f"Drift detected in {metric.column_name}")
Track prediction distribution changes:
from scipy import stats
def detect_model_drift(reference_predictions, current_predictions):
# Statistical test for distribution change
statistic, p_value = stats.ks_2samp(
reference_predictions,
current_predictions
)
if p_value < 0.05:
alert("Model drift detected")
import logging
import json
logger = logging.getLogger("ai_model")
def log_prediction(input_data, prediction, metadata):
log_entry = {
"timestamp": datetime.now().isoformat(),
"input": input_data,
"prediction": prediction,
"metadata": metadata
}
logger.info(json.dumps(log_entry))
from prometheus_client import start_http_server, Gauge
# Model version gauge
model_version = Gauge(
'model_version',
'Current model version',
['version']
)
# Feature importance tracking
feature_importance = Gauge(
'feature_importance',
'Feature importance scores',
['feature_name']
)
def check_metrics():
if error_rate > 0.05:
send_alert("Error rate exceeded 5%")
if latency_p99 > 1.0:
send_alert("P99 latency exceeded 1 second")
if accuracy < 0.90:
send_alert("Model accuracy dropped below 90%")
from sklearn.ensemble import IsolationForest
def detect_anomalies(features):
model = IsolationForest(contamination=0.1)
model.fit(training_features)
predictions = model.predict(features)
anomalies = features[predictions == -1]
if len(anomalies) > 0:
alert(f"Detected {len(anomalies)} anomalies")
- Prometheus: Metrics collection
- Grafana: Visualization
- Evidently AI: Data drift detection
- MLflow: Experiment tracking
- Weights & Biases: Experiment tracking
- Arize AI: Model monitoring
- Fiddler AI: Explainability and monitoring
- Baseline Metrics: Establish baseline before deployment
- Continuous Monitoring: Monitor 24/7, not just during deployment
- Automated Alerts: Set up alerts for critical metrics
- Regular Reviews: Weekly model performance reviews
- Version Tracking: Track model versions and their performance
# Grafana dashboard configuration
dashboard = {
"panels": [
{
"title": "Prediction Latency",
"targets": [{"expr": "histogram_quantile(0.95, model_latency)"}]
},
{
"title": "Error Rate",
"targets": [{"expr": "rate(model_errors[5m])"}]
},
{
"title": "Model Accuracy",
"targets": [{"expr": "model_accuracy"}]
}
]
}
Comprehensive observability is essential for production AI systems. Start with basic metrics and gradually add more sophisticated monitoring as your system matures.
For AI Observability and Monitoring: Tracking Model Performance in Production, define pre-deploy checks, rollout gates, and rollback triggers before release. Track p95 latency, error rate, and cost per request for at least 24 hours after deployment. If the trend regresses from baseline, revert quickly and document the decision in the runbook.
Keep the operating model simple under pressure: one owner per change, one decision channel, and clear stop conditions. Review alert quality regularly to remove noise and ensure on-call engineers can distinguish urgent failures from routine variance.
Repeatability is the goal. Convert successful interventions into standard operating procedures and version them in the repository so future responders can execute the same flow without ambiguity.
For AI Observability and Monitoring: Tracking Model Performance in Production, define pre-deploy checks, rollout gates, and rollback triggers before release. Track p95 latency, error rate, and cost per request for at least 24 hours after deployment. If the trend regresses from baseline, revert quickly and document the decision in the runbook.
Keep the operating model simple under pressure: one owner per change, one decision channel, and clear stop conditions. Review alert quality regularly to remove noise and ensure on-call engineers can distinguish urgent failures from routine variance.
Repeatability is the goal. Convert successful interventions into standard operating procedures and version them in the repository so future responders can execute the same flow without ambiguity.
For AI Observability and Monitoring: Tracking Model Performance in Production, define pre-deploy checks, rollout gates, and rollback triggers before release. Track p95 latency, error rate, and cost per request for at least 24 hours after deployment. If the trend regresses from baseline, revert quickly and document the decision in the runbook.
Keep the operating model simple under pressure: one owner per change, one decision channel, and clear stop conditions. Review alert quality regularly to remove noise and ensure on-call engineers can distinguish urgent failures from routine variance.
Repeatability is the goal. Convert successful interventions into standard operating procedures and version them in the repository so future responders can execute the same flow without ambiguity.
