Advanced PromQL

Range Vectors for Historical Comparison¶

Range vectors allow you to query data over a time range, enabling comparisons between current and historical metrics. This is critical for detecting anomalies like sudden traffic spikes or gradual performance degradation.

Example: Detecting CPU Usage Anomalies¶

Compare current CPU usage with the same time last week using offset() for alignment:

avg(cpu_utilization{job="app-server"}[5m]) > 
  avg(cpu_utilization{job="app-server"}[5m])[1w] offset 1w

This query checks if the current 5-minute average exceeds the same metric from one week ago, flagging potential anomalies.

Example: Trend Analysis with `avg_over_time`¶

Analyze CPU usage trends over a 7-day window:

avg_over_time(cpu_utilization{job="app-server"}[7d])

Use this to identify gradual increases that might indicate resource exhaustion.

Alignment Joins for Synchronized Metrics¶

When comparing metrics from different sources (e.g., HTTP latency vs. database query latency), alignment joins ensure they are evaluated at the same time steps. Use align() to synchronize time series.

Example: Aligning HTTP and Database Metrics¶

align(
  avg(http_request_duration_seconds{job="api"}),
  avg(db_query_latency_seconds{job="db"})
)[1m]

This aligns both HTTP and database metrics to 1-minute intervals, enabling direct comparison at the same time steps.

Diagram: Alignment Join Workflow¶

[HTTP Latency]        [Database Latency]
       |                  |
       v                  v
align() -------------------> [Aligned Metrics]
       |                  |
       v                  v
[Correlation Analysis]   [Performance Comparison]

Rate Calculations for Anomaly Detection¶

The rate() function calculates the per-second average rate of increase of a counter, ideal for detecting sudden error spikes or throughput changes.

Example: Detecting Error Rate Anomalies¶

rate(http_error_count{job="api"}[5m]) > 0.05

This query flags instances where the error rate exceeds 5% per second, indicating potential service degradation.

Example: Combining Rate with Historical Baselines¶

rate(http_error_count{job="api"}[5m]) > 
  avg_over_time(http_error_count{job="api"}[5m])[1w] * 1.5

Compares current error rates to historical baselines, alerting on 150% increases.

Combining Techniques for Comprehensive Analysis¶

Advanced queries often blend multiple techniques. For example, use align() to synchronize metrics, then apply rate() to detect anomalies in their relationship.

Example: Correlated Anomaly Detection¶

(
  rate(
    align(
      avg(http_request_duration_seconds{job="api"}),
      avg(db_query_latency_seconds{job="db"})
    )[1m]
  )
  - 
  avg_over_time(http_request_duration_seconds{job="api"}[5m])[1w]
) > 0.2

This identifies sudden increases in request latency compared to historical averages after aligning metrics to a common time interval.

Key takeaways¶

Range vectors enable historical comparisons, critical for identifying gradual or sudden anomalies.
Alignment joins synchronize metrics from different sources, ensuring accurate cross-metric analysis.
Rate calculations reveal per-second changes in counters, ideal for detecting transient issues like error spikes.
Combine these techniques to build robust anomaly detection pipelines that align with SLOs and error budget management.