deployment/docker/rules: add RequestErrorsToUnknownPaths alert which focuses on errors without

path set (i.e. invalid authentication) A new alert would help us separating two cases: * API errors on known paths, indicating an application issue * errors where path is not recorded - this indicates authentication problems
deployment/docker/rules: update RequestErrorsToAPI alert definition
2026-07-25 10:09:43 +03:00 · 2026-07-24 09:57:54 +02:00 · 2026-07-24 09:57:48 +02:00
3 changed files with 15 additions and 3 deletions
--- a/app/vmalert/config/testdata/rules/rules-replay-good.rules
+++ b/app/vmalert/config/testdata/rules/rules-replay-good.rules
@@ -18,7 +18,7 @@ groups:
    concurrency: 2
    rules:
      - alert: RequestErrorsToAPI
-        expr: increase(vm_http_request_errors_total[5m]) > 0
+        expr: increase(vm_http_request_errors_total{path=~".+"}[5m]) > 0
        for: 15m
        labels:
          severity: warning
--- a/deployment/docker/rules/alerts-cluster.yml
+++ b/deployment/docker/rules/alerts-cluster.yml
@@ -75,7 +75,7 @@ groups:
            Consider to limit the ingestion rate, decrease retention or scale the disk space if possible."
      - alert: RequestErrorsToAPI
-        expr: increase(vm_http_request_errors_total[5m]) > 0
+        expr: increase(vm_http_request_errors_total{path=~".+", path!="*"}[5m]) > 0
        for: 15m
        labels:
          severity: warning
@@ -86,6 +86,18 @@ groups:
          description: "Requests to path {{ $labels.path }} are receiving errors.
            Please verify if clients are sending correct requests."
      - alert: RequestErrorsToUnknownPaths
        expr: sum(increase(vm_http_request_errors_total{path="*"}[5m])) by(job, instance, reason) > 0
        for: 15m
        labels:
          severity: warning
          show_at: dashboard
        annotations:
          dashboard: "{{ $externalURL }}/d/oS7Bi_0Wz?viewPanel=52&var-instance={{ $labels.instance }}"
          summary: "Too many errors served for {{ $labels.job }} with reason {{ $labels.reason }} (instance {{ $labels.instance }})"
          description: "Requests are failing with reason {{ $labels.reason }}.
            Please verify if clients are sending correct requests."
      - alert: RPCErrors
        expr: |
          (
--- a/deployment/docker/rules/alerts-single-node.yml
+++ b/deployment/docker/rules/alerts-single-node.yml
@@ -75,7 +75,7 @@ groups:
            Consider to limit the ingestion rate, decrease retention or scale the disk space if possible."
      - alert: RequestErrorsToAPI
-        expr: increase(vm_http_request_errors_total[5m]) > 0
+        expr: increase(vm_http_request_errors_total{path=~".+"}[5m]) > 0
        for: 15m
        labels:
          severity: warning
Author	SHA1	Message	Date
Vadim Rutkovsky	4a08ac7532	deployment/docker/rules: add RequestErrorsToUnknownPaths alert which focuses on errors without path set (i.e. invalid authentication) A new alert would help us separating two cases: * API errors on known paths, indicating an application issue * errors where path is not recorded - this indicates authentication problems	2026-07-24 09:57:54 +02:00
Vadim Rutkovsky	911c48fc92	deployment/docker/rules: update RequestErrorsToAPI alert definition	2026-07-24 09:57:48 +02:00