Monitoring & Observability implementation checklist

Configure a global error handler (e.g., Sentry, Highlight.io) at the application entry point to capture unhandled exceptions and promise rejections.

Integrate source map uploads into the CI/CD pipeline to ensure stack traces map back to original source code rather than minified bundles.

Attach non-PII user identifiers and session IDs to error reports to facilitate reproduction and impact assessment of user-specific bugs.

Tag every error with a specific commit SHA or version number to identify regressions introduced in specific deployments.

Configure automatic capture of network requests, console logs, and state changes leading up to a crash for debugging context.

Implement middleware (e.g., Helicone, LangSmith) to log prompt and completion tokens for every LLM provider call to track unit costs.

Measure Time-To-First-Token (TTFT) and total request duration for all LLM API calls to identify provider-side performance degradation.

Explicitly tag metrics with specific model versions (e.g., gpt-4-0613 vs gpt-4-turbo) to evaluate performance and cost deltas.

Securely log prompt inputs and model outputs for a sampled percentage of requests to monitor for hallucinations or drift.

Monitor the latency of vector database queries and embedding generation steps in RAG pipelines.

Set up Prometheus or Datadog alerts for CPU utilization >85% and Memory utilization >90% sustained for over 5 minutes.

Configure alerts for server-side error rates exceeding 1% of total traffic over a 60-second window.

Monitor active vs. maximum available database connections to prevent application hangs during traffic spikes.

Establish alerts for disk space usage exceeding 80% and high I/O wait times on database volumes.

For worker-based systems, monitor the number of pending messages and the time-to-process for background jobs.

Ensure trace headers (e.g., W3C Trace Context) are passed across all service boundaries, including internal microservices and external proxies.

Enable auto-instrumentation for database drivers to capture slow queries as spans within a distributed trace.

Configure trace sampling rates (e.g., 10% for high-volume, 100% for errors) to balance visibility with storage costs.

Instrument all outgoing HTTP calls to third-party services to identify which external dependency is causing latency.

Ensure that background jobs inherit the trace context from the triggering HTTP request for end-to-end visibility.

Configure external probes (e.g., BetterStack) to check the /health endpoint from multiple geographic regions every 60 seconds.

Define a clear escalation path in an incident management tool to ensure critical alerts page a human responder within 5 minutes.

Distinguish between 'Critical' (page responder) and 'Warning' (Slack notification only) to prevent alert fatigue.

Script a critical user path (e.g., login or checkout) to run every 5-15 minutes to verify functional correctness.

Establish a standardized template for documenting root causes and action items after every high-severity incident.