deployment/docker/rules: add VMSelectConcurrentQueriesExceedMemoryLimit alert

Warn users when cluster is misconfigured to allow too many concurrent selects

app/vmalert/config/testdata/rules/rules-replay-good.rules

@@ -18,7 +18,7 @@ groups:
     concurrency: 2
     rules:
       - alert: RequestErrorsToAPI
-        expr: increase(vm_http_request_errors_total{path=~".+"}[5m]) > 0
+        expr: increase(vm_http_request_errors_total[5m]) > 0
         for: 15m
         labels:
           severity: warning
@@ -37,4 +37,4 @@ groups:
           dashboard: "http://localhost:3000/d/wNf0q_kZk?viewPanel=67&var-instance={{ $labels.instance }}"
           summary: "Too many logs printed for job \"{{ $labels.job }}\" ({{ $labels.instance }})"
           description: "Logging rate for job \"{{ $labels.job }}\" ({{ $labels.instance }}) is {{ $value }} for last 15m.\n
-           Worth to check logs for specific error messages."
+           Worth to check logs for specific error messages."

app/vmselect/promql/eval.go

@@ -1687,6 +1687,10 @@ func assertInstantValues(tss []*timeseries) {
 
 var memoryIntensiveQueries = metrics.NewCounter(`vm_memory_intensive_queries_total`)
 
+var _ = metrics.NewGauge(`vm_max_memory_per_query`, func() float64 {
+	return float64(maxMemoryPerQuery.N)
+})
+
 func evalRollupFuncWithMetricExpr(qt *querytracer.Tracer, ec *EvalConfig, funcName string, rf rollupFunc,
 	expr metricsql.Expr, me *metricsql.MetricExpr, iafc *incrementalAggrFuncContext, windowExpr *metricsql.DurationExpr,
 ) ([]*timeseries, error) {

deployment/docker/rules/alerts-cluster.yml

@@ -75,7 +75,7 @@ groups:
             Consider to limit the ingestion rate, decrease retention or scale the disk space if possible."
 
       - alert: RequestErrorsToAPI
-        expr: increase(vm_http_request_errors_total{path=~".+"}[5m]) > 0
+        expr: increase(vm_http_request_errors_total[5m]) > 0
         for: 15m
         labels:
           severity: warning
@@ -207,9 +207,9 @@ groups:
         annotations:
           summary: "IndexDB skipped registering items during data ingestion with reason={{ $labels.reason }}."
           description: |
-            VictoriaMetrics could skip registering new timeseries during ingestion if they fail the validation process.
+            VictoriaMetrics could skip registering new timeseries during ingestion if they fail the validation process. 
-            For example, `reason=too_long_item` means that time series cannot exceed 64KB. Please, reduce the number
+            For example, `reason=too_long_item` means that time series cannot exceed 64KB. Please, reduce the number 
-            of labels or label values for such series. Or enforce these limits via `-maxLabelsPerTimeseries` and
+            of labels or label values for such series. Or enforce these limits via `-maxLabelsPerTimeseries` and 
             `-maxLabelValueLen` command-line flags.
 
       - alert: TooManyTSIDMisses
@@ -224,3 +224,15 @@ groups:
             If this happens after unclean shutdown of VictoriaMetrics process (via \"kill -9\", OOM or power off),
             then this is OK - the alert must go away in a few minutes after the restart.
             Otherwise this may point to the corruption of index data.
+
+      - alert: VMSelectConcurrentQueriesExceedMemoryLimit
+        expr: (vm_max_memory_per_query * on(job, instance) vm_concurrent_select_capacity) > on(job, instance) vm_available_memory_bytes
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "vmselect ({{ $labels.instance }}) concurrent query memory may exceed pod limit"
+          description: "Current concurrent queries ({{ $value | humanize1024 }} combined max memory) exceed
+            the available memory on instance {{ $labels.instance }}.
+            This may result in OOM kills. Consider reducing -maxConcurrentRequests,
+            lowering -maxMemoryPerQuery, or scaling up pod memory limits."

deployment/docker/rules/alerts-single-node.yml

@@ -75,7 +75,7 @@ groups:
             Consider to limit the ingestion rate, decrease retention or scale the disk space if possible."
 
       - alert: RequestErrorsToAPI
-        expr: increase(vm_http_request_errors_total{path=~".+"}[5m]) > 0
+        expr: increase(vm_http_request_errors_total[5m]) > 0
         for: 15m
         labels:
           severity: warning
@@ -173,9 +173,9 @@ groups:
         annotations:
           summary: "IndexDB skipped registering items during data ingestion with reason={{ $labels.reason }}."
           description: |
-            VictoriaMetrics could skip registering new timeseries during ingestion if they fail the validation process.
+            VictoriaMetrics could skip registering new timeseries during ingestion if they fail the validation process. 
-            For example, `reason=too_long_item` means that time series cannot exceed 64KB. Please, reduce the number
+            For example, `reason=too_long_item` means that time series cannot exceed 64KB. Please, reduce the number 
-            of labels or label values for such series. Or enforce these limits via `-maxLabelsPerTimeseries` and
+            of labels or label values for such series. Or enforce these limits via `-maxLabelsPerTimeseries` and 
             `-maxLabelValueLen` command-line flags.
 
       - alert: TooManyTSIDMisses
@@ -189,4 +189,4 @@ groups:
             Unexpected TSID misses for \"{{ $labels.job }}\" ({{ $labels.instance }}) for the last 15 minutes.
             If this happens after unclean shutdown of VictoriaMetrics process (via \"kill -9\", OOM or power off),
             then this is OK - the alert must go away in a few minutes after the restart.
-            Otherwise this may point to the corruption of index data.
+            Otherwise this may point to the corruption of index data.

docs/victoriametrics/changelog/CHANGELOG.md

@@ -28,6 +28,8 @@ See also [LTS releases](https://docs.victoriametrics.com/victoriametrics/lts-rel
 
 * SECURITY: upgrade base docker image (Alpine) from 3.23.4 to 3.24.1. See [Alpine 3.24.1 release notes](https://www.alpinelinux.org/posts/Alpine-3.24.1-released.html).
 
+* FEATURE: `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): expose `vm_max_memory_per_query` metric reflecting the `-search.maxMemoryPerQuery` limit. Create `VMSelectConcurrentQueriesExceedMemoryLimit` alert to warn when OOMs are possible due to misconfiguration of `-search.maxMemoryPerQuery` and max concurrent queries.
+
 * FEATURE: [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/): add `default_vm_access_claim` field into `jwt` section of auth config. It could be used at [JWT claim placeholders](https://docs.victoriametrics.com/victoriametrics/vmauth/#jwt-claim-based-request-templating), if `JWT` token doesn't have `vm_access` claim. See [#11054](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11054).
 * FEATURE: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/): reduces CPU usage by 10% at [sharding among remote storages](https://docs.victoriametrics.com/victoriametrics/vmagent/#sharding-among-remote-storages). See [#11113](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11113). Thanks to @bennf for contribution.
 * FEATURE: [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): add `optimize_repeated_binary_op_subexprs=1` query arg to [/api/v1/query_range](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#range-query) for executing binary operator sides sequentially when they share the same optimized aggregate rollup result expression. This allows the second side to reuse rollup result cache populated by the first side. See [#10575](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10575).