app/vmselect: fail the query request directly when there is not enough disk space

Previously, vmselect may crash if there is not enough free disk space to store tmp files from vmstorage. vmselect stores data blocks received from vmstorage if response size exceeds in-memory limit.
 It reduced vmselect availability, because it makes impossible to serve small queries. And huge query may crash vmselect for all users.

 So this commit adds a error check instead. If there is not enough disk space, vmselect will stop query execution and return error back to user.

fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4688

README.md

@@ -1,6 +1,6 @@
 # VictoriaMetrics
 
-[![Latest Release](https://img.shields.io/github/v/release/VictoriaMetrics/VictoriaMetrics?sort=semver&label=&filter=!*-victorialogs&logo=github&labelColor=gray&color=gray&link=https%3A%2F%2Fgithub.com%2FVictoriaMetrics%2FVictoriaMetrics%2Freleases%2Flatest)](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)
+[![Latest Release](https://img.shields.io/github/v/release/VictoriaMetrics/VictoriaMetrics?logo=github&labelColor=gray&color=gray&label=Release)](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)
 [![Docker Pulls](https://img.shields.io/docker/pulls/victoriametrics/victoria-metrics?label=&logo=docker&logoColor=white&labelColor=2496ED&color=2496ED&link=https%3A%2F%2Fhub.docker.com%2Fr%2Fvictoriametrics%2Fvictoria-metrics)](https://hub.docker.com/u/victoriametrics)
 [![Build Status](https://github.com/VictoriaMetrics/VictoriaMetrics/actions/workflows/build.yml/badge.svg?branch=master&link=https%3A%2F%2Fgithub.com%2FVictoriaMetrics%2FVictoriaMetrics%2Factions)](https://github.com/VictoriaMetrics/VictoriaMetrics/actions/workflows/build.yml)
 [![License](https://img.shields.io/github/license/VictoriaMetrics/VictoriaMetrics?labelColor=green&label=&link=https%3A%2F%2Fgithub.com%2FVictoriaMetrics%2FVictoriaMetrics%2Fblob%2Fmaster%2FLICENSE)](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/LICENSE)
@@ -41,8 +41,8 @@ VictoriaMetrics is optimized for timeseries data, even when old time series are
 * **Ideal for big data**: Works well with large amounts of time series data from APM, Kubernetes, IoT sensors, connected cars, industrial telemetry, financial data and various [Enterprise workloads](https://docs.victoriametrics.com/victoriametrics/enterprise/).
 * **Query language**: Supports both PromQL and the more performant MetricsQL.
 * **Easy to setup**: No dependencies, single [small binary](https://medium.com/@valyala/stripping-dependency-bloat-in-victoriametrics-docker-image-983fb5912b0d), configuration through command-line flags, but the default is also fine-tuned; backup and restore with [instant snapshots](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282).
-* **Global query view**: Multiple Prometheus instances or any other data sources may ingest data into VictoriaMetrics and queried via a single query.
-* **Various Protocols**: Support metric scraping, ingestion and backfilling in various protocol.
+* **Global query view**: Multiple Prometheus instances or any other data sources may ingest data into VictoriaMetrics and be queried via a single query.
+* **Various Protocols**: Support metric scraping, ingestion and backfilling in various protocols.
     * [Prometheus exporters](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-scrape-prometheus-exporters-such-as-node-exporter), [Prometheus remote write API](https://docs.victoriametrics.com/victoriametrics/integrations/prometheus/), [Prometheus exposition format](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-import-data-in-prometheus-exposition-format).
     * [InfluxDB line protocol](https://docs.victoriametrics.com/victoriametrics/integrations/influxdb/) over HTTP, TCP and UDP.
     * [Graphite plaintext protocol](https://docs.victoriametrics.com/victoriametrics/integrations/graphite/#ingesting) with [tags](https://graphite.readthedocs.io/en/latest/tags.html#carbon).
@@ -78,7 +78,7 @@ We strictly apply security measures in everything we do. VictoriaMetrics has ach
 Some good benchmarks VictoriaMetrics achieved:
 
 * **Minimal memory footprint**: handling millions of unique timeseries with [10x less RAM](https://medium.com/@valyala/insert-benchmarks-with-inch-influxdb-vs-victoriametrics-e31a41ae2893) than InfluxDB, up to [7x less RAM](https://valyala.medium.com/prometheus-vs-victoriametrics-benchmark-on-node-exporter-metrics-4ca29c75590f) than Prometheus, Thanos or Cortex.
-* **Highly scalable and performance** for [data ingestion](https://medium.com/@valyala/high-cardinality-tsdb-benchmarks-victoriametrics-vs-timescaledb-vs-influxdb-13e6ee64dd6b) and [querying](https://medium.com/@valyala/when-size-matters-benchmarking-victoriametrics-vs-timescale-and-influxdb-6035811952d4), [20x outperforms](https://medium.com/@valyala/insert-benchmarks-with-inch-influxdb-vs-victoriametrics-e31a41ae2893) InfluxDB and TimescaleDB.
+* **Highly scalable and performant** for [data ingestion](https://medium.com/@valyala/high-cardinality-tsdb-benchmarks-victoriametrics-vs-timescaledb-vs-influxdb-13e6ee64dd6b) and [querying](https://medium.com/@valyala/when-size-matters-benchmarking-victoriametrics-vs-timescale-and-influxdb-6035811952d4), [20x outperforms](https://medium.com/@valyala/insert-benchmarks-with-inch-influxdb-vs-victoriametrics-e31a41ae2893) InfluxDB and TimescaleDB.
 * **High data compression**: [70x more data points](https://medium.com/@valyala/when-size-matters-benchmarking-victoriametrics-vs-timescale-and-influxdb-6035811952d4) may be stored into limited storage than TimescaleDB, [7x less storage](https://valyala.medium.com/prometheus-vs-victoriametrics-benchmark-on-node-exporter-metrics-4ca29c75590f) space is required than Prometheus, Thanos or Cortex.
 * **Reducing storage costs**: [10x more effective](https://docs.victoriametrics.com/victoriametrics/casestudies/#grammarly) than Graphite according to the Grammarly case study.
 * **A single-node VictoriaMetrics** can replace medium-sized clusters built with competing solutions such as Thanos, M3DB, Cortex, InfluxDB or TimescaleDB. See [VictoriaMetrics vs Thanos](https://medium.com/@valyala/comparing-thanos-to-victoriametrics-cluster-b193bea1683), [Measuring vertical scalability](https://medium.com/@valyala/measuring-vertical-scalability-for-time-series-databases-in-google-cloud-92550d78d8ae), [Remote write storage wars - PromCon 2019](https://promcon.io/2019-munich/talks/remote-write-storage-wars/).
@@ -86,7 +86,7 @@ Some good benchmarks VictoriaMetrics achieved:
 
 ## Community and contributions
 
-Feel free asking any questions regarding VictoriaMetrics:
+Feel free to ask any questions regarding VictoriaMetrics:
 
 * [Slack Inviter](https://slack.victoriametrics.com/) and [Slack channel](https://victoriametrics.slack.com/)
 * [X (Twitter)](https://x.com/VictoriaMetrics/)

app/victoria-metrics/main.go

@@ -13,6 +13,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/appmetrics"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/buildinfo"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/cgroup"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/envflag"
@@ -34,9 +35,10 @@ var (
 		"This can be changed with -promscrape.config.strictParse=false command-line flag")
 	maxIngestionRate = flag.Int("maxIngestionRate", 0, "The maximum number of samples vmsingle can receive per second. Data ingestion is paused when the limit is exceeded. "+
 		"By default there are no limits on samples ingestion rate.")
-	vmselectMaxConcurrentRequests = flag.Int("search.maxConcurrentRequests", getDefaultMaxConcurrentRequests(), "The maximum number of concurrent search requests. "+
-		"It shouldn't be high, since a single request can saturate all the CPU cores, while many concurrently executed requests may require high amounts of memory. "+
-		"See also -search.maxQueueDuration and -search.maxMemoryPerQuery")
+	vmselectMaxConcurrentRequests = flagutil.NewIntWithDynamicDefault("search.maxConcurrentRequests", getDefaultMaxConcurrentRequests(), "vmselect.getDefaultMaxConcurrentRequests()",
+		"The maximum number of concurrent search requests. "+
+			"It shouldn't be high, since a single request can saturate all the CPU cores, while many concurrently executed requests may require high amounts of memory. "+
+			"See also -search.maxQueueDuration and -search.maxMemoryPerQuery")
 	vmselectMaxQueueDuration = flag.Duration("search.maxQueueDuration", 10*time.Second, "The maximum time the request waits for execution when -search.maxConcurrentRequests "+
 		"limit is reached; see also -search.maxQueryDuration")
 )
@@ -63,6 +65,7 @@ func main() {
 	flag.CommandLine.SetOutput(os.Stdout)
 	flag.Usage = usage
 	envflag.Parse()
+	initSecretFlags()
 	buildinfo.Init()
 	logger.Init()
 
@@ -89,7 +92,9 @@ func main() {
 	}
 	logger.Infof("starting VictoriaMetrics at %q...", listenAddrs)
 	startTime := time.Now()
+
 	vmstorage.Init(*vmselectMaxConcurrentRequests, *vmselectMaxQueueDuration, promql.ResetRollupResultCacheIfNeeded)
+	appmetrics.MustCreateUncleanShutdownMarker(vmstorage.DataPath())
 	vmselect.Init(*vmselectMaxConcurrentRequests, *vmselectMaxQueueDuration)
 	vminsertcommon.StartIngestionRateLimiter(*maxIngestionRate)
 	vminsert.Init()
@@ -119,6 +124,7 @@ func main() {
 
 	vmstorage.Stop()
 	vmselect.Stop()
+	appmetrics.MustRemoveUncleanShutdownMarker(vmstorage.DataPath())
 
 	logger.Infof("the VictoriaMetrics has been stopped in %.3f seconds", time.Since(startTime).Seconds())
 }
@@ -170,3 +176,9 @@ See the docs at https://docs.victoriametrics.com/victoriametrics/
 `
 	flagutil.Usage(s)
 }
+
+// initSecretFlags manages the secret flags for this app and must be called after flag parsing and before logger init.
+func initSecretFlags() {
+	pushmetrics.InitSecretFlags()
+	vmselect.InitSecretFlags()
+}

app/vmagent/main.go

@@ -84,7 +84,7 @@ var (
 	maxLabelNameLen        = flag.Int("maxLabelNameLen", 0, "The maximum length of label names in the accepted time series. Series with longer label name are ignored. In this case the vm_rows_ignored_total{reason=\"too_long_label_name\"} metric at /metrics page is incremented")
 	maxLabelValueLen       = flag.Int("maxLabelValueLen", 0, "The maximum length of label values in the accepted time series. Series with longer label value are ignored. In this case the vm_rows_ignored_total{reason=\"too_long_label_value\"} metric at /metrics page is incremented")
 
-	enableMultitenancyViaHeaders = flag.Bool("enableMultitenancyViaHeaders", false, "Enables multitenancy via HTTP headers. "+
+	enableMultitenancyViaHeaders = flag.Bool("enableMultitenancyViaHeaders", true, "Enables multitenancy via HTTP headers. "+
 		"See https://docs.victoriametrics.com/victoriametrics/vmagent/#multitenancy")
 )
 
@@ -115,7 +115,7 @@ func main() {
 	flag.CommandLine.SetOutput(os.Stdout)
 	flag.Usage = usage
 	envflag.Parse()
-	remotewrite.InitSecretFlags()
+	initSecretFlags()
 	buildinfo.Init()
 	logger.Init()
 	opentelemetry.Init()
@@ -843,3 +843,9 @@ See the docs at https://docs.victoriametrics.com/victoriametrics/vmagent/ .
 `
 	flagutil.Usage(s)
 }
+
+// initSecretFlags manages the secret flags for this app and must be called after flag parsing and before logger init.
+func initSecretFlags() {
+	remotewrite.InitSecretFlags()
+	pushmetrics.InitSecretFlags()
+}

app/vmagent/prometheusimport/request_handler_test.go

@@ -52,6 +52,7 @@ func setUp() {
 
 func tearDown() {
 	protoparserutil.StopUnmarshalWorkers()
+	remotewrite.Stop()
 	srv.Close()
 	logger.ResetOutputForTest()
 	tmpDataDir := flag.Lookup("remoteWrite.tmpDataPath").Value.String()

app/vmagent/remotewrite/remotewrite.go

@@ -15,6 +15,7 @@ import (
 	"github.com/VictoriaMetrics/metrics"
 	"github.com/cespare/xxhash/v2"
 
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/appmetrics"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/auth"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bloomfilter"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/cgroup"
@@ -62,9 +63,10 @@ var (
 		"See also -remoteWrite.maxDiskUsagePerURL and -remoteWrite.disableOnDiskQueue")
 	keepDanglingQueues = flag.Bool("remoteWrite.keepDanglingQueues", false, "Keep persistent queues contents at -remoteWrite.tmpDataPath in case there are no matching -remoteWrite.url. "+
 		"Useful when -remoteWrite.url is changed temporarily and persistent queue files will be needed later on.")
-	queues = flagutil.NewArrayInt("remoteWrite.queues", cgroup.AvailableCPUs()*2, "The number of concurrent queues to each -remoteWrite.url. Set more queues if default number of queues "+
-		"isn't enough for sending high volume of collected data to remote storage. "+
-		"Default value depends on the number of available CPU cores. It should work fine in most cases since it minimizes resource usage")
+	queues = flagutil.NewArrayIntWithDynamicDefault("remoteWrite.queues", cgroup.AvailableCPUs()*2, "2*cgroup.AvailableCPUs()",
+		"The number of concurrent queues to each -remoteWrite.url. Set more queues if default number of queues "+
+			"isn't enough for sending high volume of collected data to remote storage. "+
+			"Default value depends on the number of available CPU cores. It should work fine in most cases since it minimizes resource usage")
 	inmemoryQueues = flagutil.NewArrayInt("remoteWrite.inmemoryQueues", 0, "The number of additional workers per each -remoteWrite.url, which send only recently ingested data from the in-memory queue, "+
 		"while the file-based queue at -remoteWrite.tmpDataPath is drained by workers configured via -remoteWrite.queues. "+
 		"This reduces delivery lag for fresh samples when the file-based queue contains a backlog accumulated during remote storage outages.")
@@ -156,7 +158,8 @@ var maxQueues = cgroup.AvailableCPUs() * 16
 
 const persistentQueueDirname = "persistent-queue"
 
-// InitSecretFlags must be called after flag.Parse and before any logging.
+// InitSecretFlags manages the secret flags for this pkg and must be called by app-level initSecretFlags.
+// It should run before logger initialization and package Init() (if exists).
 func InitSecretFlags() {
 	if !*showRemoteWriteURL {
 		// remoteWrite.url can contain authentication codes, so hide it at `/metrics` output.
@@ -232,6 +235,7 @@ func Init() {
 	initStreamAggrConfigGlobal()
 
 	initRemoteWriteCtxs(*remoteWriteURLs)
+	appmetrics.MustCreateUncleanShutdownMarker(*tmpDataPath)
 
 	disableOnDiskQueues := []bool(*disableOnDiskQueue)
 	disableOnDiskQueueAny = slices.Contains(disableOnDiskQueues, true)
@@ -245,6 +249,8 @@ func Init() {
 	dropDanglingQueues()
 
 	// Start config reloader.
+	configReloaderStopCh = make(chan struct{})
+	configReloaderWG = sync.WaitGroup{}
 	configReloaderWG.Go(func() {
 		for {
 			select {
@@ -331,7 +337,7 @@ func initRemoteWriteCtxs(urls []string) {
 }
 
 var (
-	configReloaderStopCh = make(chan struct{})
+	configReloaderStopCh chan struct{}
 	configReloaderWG     sync.WaitGroup
 )
 
@@ -388,6 +394,8 @@ func Stop() {
 	if sl := dailySeriesLimiter; sl != nil {
 		sl.MustStop()
 	}
+
+	appmetrics.MustRemoveUncleanShutdownMarker(*tmpDataPath)
 }
 
 // PushDropSamplesOnFailure pushes wr to the configured remote storage systems set via -remoteWrite.url

app/vmalert/config/testdata/rules/rules-replay-good.rules

@@ -18,7 +18,7 @@ groups:
     concurrency: 2
     rules:
       - alert: RequestErrorsToAPI
-        expr: increase(vm_http_request_errors_total{path=~".+"}[5m]) > 0
+        expr: increase(vm_http_request_errors_total[5m]) > 0
         for: 15m
         labels:
           severity: warning

app/vmalert/datasource/init.go

@@ -60,7 +60,8 @@ var (
 		`Only valid for VictoriaMetrics as the datasource.`)
 )
 
-// InitSecretFlags must be called after flag.Parse and before any logging
+// InitSecretFlags manages the secret flags for this pkg and must be called by app-level initSecretFlags.
+// It should run before logger initialization and package Init() (if exists).
 func InitSecretFlags() {
 	if !*showDatasourceURL {
 		flagutil.RegisterSecretFlag("datasource.url")

app/vmalert/main.go

@@ -88,10 +88,7 @@ func main() {
 	flag.CommandLine.SetOutput(os.Stdout)
 	flag.Usage = usage
 	envflag.Parse()
-	remoteread.InitSecretFlags()
-	remotewrite.InitSecretFlags()
-	datasource.InitSecretFlags()
-	notifier.InitSecretFlags()
+	initSecretFlags()
 	buildinfo.Init()
 	logger.Init()
 
@@ -438,3 +435,12 @@ func getLastConfigError() error {
 	defer lastConfigErrMu.RUnlock()
 	return lastConfigErr
 }
+
+// initSecretFlags manages the secret flags for this app and must be called after flag parsing and before logger init.
+func initSecretFlags() {
+	remoteread.InitSecretFlags()
+	remotewrite.InitSecretFlags()
+	datasource.InitSecretFlags()
+	notifier.InitSecretFlags()
+	pushmetrics.InitSecretFlags()
+}

app/vmalert/notifier/init.go

@@ -189,7 +189,8 @@ func Init(extLabels map[string]string, extURL string) error {
 	return nil
 }
 
-// InitSecretFlags must be called after flag.Parse and before any logging
+// InitSecretFlags manages the secret flags for this pkg and must be called by app-level initSecretFlags.
+// It should run before logger initialization and package Init() (if exists).
 func InitSecretFlags() {
 	if !*showNotifierURL {
 		flagutil.RegisterSecretFlag("notifier.url")

app/vmalert/remoteread/init.go

@@ -56,7 +56,8 @@ var (
 	oauth2Scopes   = flag.String("remoteRead.oauth2.scopes", "", "Optional OAuth2 scopes to use for -remoteRead.url. Scopes must be delimited by ';'.")
 )
 
-// InitSecretFlags must be called after flag.Parse and before any logging
+// InitSecretFlags manages the secret flags for this pkg and must be called by app-level initSecretFlags.
+// It should run before logger initialization and package Init() (if exists).
 func InitSecretFlags() {
 	if !*showRemoteReadURL {
 		flagutil.RegisterSecretFlag("remoteRead.url")

app/vmalert/remotewrite/client.go

@@ -284,7 +284,15 @@ func (c *Client) flush(ctx context.Context, wr *prompb.WriteRequest) {
 	bb := writeRequestBufPool.Get()
 	bb.B = wr.MarshalProtobuf(bb.B[:0])
 	zb := compressBufPool.Get()
-	defer compressBufPool.Put(zb)
+	// A failed send may leave the http transport still reading zb.B in a separate goroutine
+	// even after send returns, so zb is returned to the pool only if no send attempt has failed.
+	// See https://pkg.go.dev/net/http#RoundTripper
+	sendFailed := false
+	defer func() {
+		if !sendFailed {
+			compressBufPool.Put(zb)
+		}
+	}()
 	if c.isVMRemoteWrite.Load() {
 		zb.B = zstd.CompressLevel(zb.B[:0], bb.B, 0)
 	} else {
@@ -303,10 +311,13 @@ func (c *Client) flush(ctx context.Context, wr *prompb.WriteRequest) {
 L:
 	for {
 		err := c.send(ctx, zb.B)
-		if err != nil && (errors.Is(err, io.EOF) || netutil.IsTrivialNetworkError(err)) {
-			// Something in the middle between client and destination might be closing
-			// the connection. So we do a one more attempt in hope request will succeed.
-			err = c.send(ctx, zb.B)
+		if err != nil {
+			sendFailed = true
+			if errors.Is(err, io.EOF) || netutil.IsTrivialNetworkError(err) {
+				// Something in the middle between client and destination might be closing
+				// the connection. So we do a one more attempt in hope request will succeed.
+				err = c.send(ctx, zb.B)
+			}
 		}
 		if err == nil {
 			sentRows.Add(len(wr.Timeseries))

app/vmalert/remotewrite/debug_client.go

@@ -36,6 +36,13 @@ func NewDebugClient() (*DebugClient, error) {
 	if err != nil {
 		return nil, fmt.Errorf("failed to create transport for -remoteWrite.url=%q: %w", *addr, err)
 	}
+	tr.IdleConnTimeout = *idleConnectionTimeout
+	// DebugClient sends every series in a separate request, so it needs more idle
+	// connections than the two http.DefaultTransport keeps per host.
+	tr.MaxIdleConnsPerHost = *maxIdleConnections
+	if tr.MaxIdleConns != 0 && tr.MaxIdleConns < tr.MaxIdleConnsPerHost {
+		tr.MaxIdleConns = tr.MaxIdleConnsPerHost
+	}
 	c := &DebugClient{
 		c: &http.Client{
 			Timeout:   *sendTimeout,

app/vmalert/remotewrite/debug_client_conns_test.go

@@ -0,0 +1,45 @@
+package remotewrite
+
+import (
+	"net/http"
+	"testing"
+)
+
+// TestDebugClient_IdleConns makes sure DebugClient keeps enough idle connections
+// to -remoteWrite.url. Every series is pushed in a separate request, so with the
+// two idle connections per host of http.DefaultTransport most of the concurrent
+// requests would open a new connection and leave a socket in TIME_WAIT state.
+func TestDebugClient_IdleConns(t *testing.T) {
+	f := func(maxIdle int) {
+		t.Helper()
+
+		oldAddr, oldMaxIdle := *addr, *maxIdleConnections
+		*addr, *maxIdleConnections = "http://localhost:8428", maxIdle
+		defer func() {
+			*addr, *maxIdleConnections = oldAddr, oldMaxIdle
+		}()
+
+		client, err := NewDebugClient()
+		if err != nil {
+			t.Fatalf("failed to create debug client: %s", err)
+		}
+		tr, ok := client.c.Transport.(*http.Transport)
+		if !ok {
+			t.Fatalf("unexpected transport type %T", client.c.Transport)
+		}
+		if tr.MaxIdleConnsPerHost != maxIdle {
+			t.Fatalf("unexpected MaxIdleConnsPerHost; got %d; want %d", tr.MaxIdleConnsPerHost, maxIdle)
+		}
+		if tr.MaxIdleConns != 0 && tr.MaxIdleConns < maxIdle {
+			t.Fatalf("MaxIdleConns=%d is lower than MaxIdleConnsPerHost=%d", tr.MaxIdleConns, maxIdle)
+		}
+		if tr.IdleConnTimeout != *idleConnectionTimeout {
+			t.Fatalf("unexpected IdleConnTimeout; got %s; want %s", tr.IdleConnTimeout, *idleConnectionTimeout)
+		}
+	}
+
+	f(100)
+
+	// the number of idle connections must be raised together with the total limit
+	f(1000)
+}

app/vmalert/remotewrite/init.go

@@ -34,10 +34,12 @@ var (
 	bearerTokenFile = flag.String("remoteWrite.bearerTokenFile", "", "Optional path to bearer token file to use for -remoteWrite.url.")
 
 	idleConnectionTimeout = flag.Duration("remoteWrite.idleConnTimeout", 50*time.Second, `Defines a duration for idle (keep-alive connections) to exist. Consider settings this value less to the value of "-http.idleConnTimeout". It must prevent possible "write: broken pipe" and "read: connection reset by peer" errors.`)
+	maxIdleConnections    = flag.Int("remoteWrite.maxIdleConnections", 100, `Defines the number of idle (keep-alive connections) to -remoteWrite.url for the vmalert-tool debug writer, which sends every series in a separate request. Too low a value may result in a high number of sockets in TIME_WAIT state.`)
 
-	maxQueueSize  = flag.Int("remoteWrite.maxQueueSize", defaultMaxQueueSize, "Defines the max number of pending datapoints to remote write endpoint")
-	maxBatchSize  = flag.Int("remoteWrite.maxBatchSize", defaultMaxBatchSize, "Defines max number of timeseries to be flushed at once")
-	concurrency   = flag.Int("remoteWrite.concurrency", defaultConcurrency, "Defines number of writers for concurrent writing into remote write endpoint. Default value depends on the number of available CPU cores.")
+	maxQueueSize = flag.Int("remoteWrite.maxQueueSize", defaultMaxQueueSize, "Defines the max number of pending datapoints to remote write endpoint")
+	maxBatchSize = flag.Int("remoteWrite.maxBatchSize", defaultMaxBatchSize, "Defines max number of timeseries to be flushed at once")
+	concurrency  = flagutil.NewIntWithDynamicDefault("remoteWrite.concurrency", defaultConcurrency, "2*cgroup.AvailableCPUs()",
+		"Defines number of writers for concurrent writing into remote write endpoint. Default value depends on the number of available CPU cores.")
 	flushInterval = flag.Duration("remoteWrite.flushInterval", defaultFlushInterval, "Defines interval of flushes to remote write endpoint")
 
 	tlsInsecureSkipVerify = flag.Bool("remoteWrite.tlsInsecureSkipVerify", false, "Whether to skip tls verification when connecting to -remoteWrite.url")
@@ -57,7 +59,8 @@ var (
 	oauth2Scopes   = flag.String("remoteWrite.oauth2.scopes", "", "Optional OAuth2 scopes to use for -notifier.url. Scopes must be delimited by ';'.")
 )
 
-// InitSecretFlags must be called after flag.Parse and before any logging
+// InitSecretFlags manages the secret flags for this pkg and must be called by app-level initSecretFlags.
+// It should run before logger initialization and package Init() (if exists).
 func InitSecretFlags() {
 	if !*showRemoteWriteURL {
 		flagutil.RegisterSecretFlag("remoteWrite.url")

app/vmalert/replay.go

@@ -30,7 +30,8 @@ var (
 		"Progress bar rendering might be verbose or break the logs parsing, so it is recommended to be disabled when not used in interactive mode.")
 	ruleEvaluationConcurrency = flag.Int("replay.ruleEvaluationConcurrency", 1, "The maximum number of concurrent '/query_range' requests when replay recording rule or alerting rule with for=0. "+
 		"Increasing this value when replaying for a long time, since each request is limited by -replay.maxDatapointsPerQuery.")
-	continueWithExecutionErr = flag.Bool("replay.continueWithExecutionErr", false, "Whether to continue replaying other rules if a rule execution fails with a 422 response code, which can happen due to an expression syntax error or a resource limit being hit.")
+	continueWithExecutionErr = flag.Bool("replay.continueWithExecutionErr", false, "Whether to continue replaying other rules if a rule execution fails with a 400 or 422 response code, "+
+		"which can happen due to an expression syntax error or a resource limit being hit.")
 )
 
 func replay(groupsCfg []config.Group, qb datasource.QuerierBuilder, rw remotewrite.RWClient) (totalRows, droppedRows int, err error) {

app/vmalert/rule/alerting.go

@@ -462,7 +462,11 @@ func (ar *AlertingRule) exec(ctx context.Context, ts time.Time, limit int) ([]pr
 	}
 
 	isPartial := isPartialResponse(res)
-	ar.logDebugf(ts, nil, "query returned %d series (elapsed: %s, isPartial: %t)", curState.Samples, curState.Duration, isPartial)
+	seriesFetched := 0
+	if res.SeriesFetched != nil {
+		seriesFetched = *res.SeriesFetched
+	}
+	ar.logDebugf(ts, nil, "query returned %d series (series_fetched: %d, elapsed: %s, isPartial: %t)", curState.Samples, seriesFetched, curState.Duration, isPartial)
 	qFn := func(query string) ([]datasource.Metric, error) {
 		res, _, err := ar.q.Query(ctx, query, ts)
 		return res.Data, err

app/vmalert/rule/group.go

@@ -290,6 +290,8 @@ func (g *Group) updateWith(newGroup *Group) error {
 	g.Headers = newGroup.Headers
 	g.NotifierHeaders = newGroup.NotifierHeaders
 	g.Labels = newGroup.Labels
+	g.EvalDelay = newGroup.EvalDelay
+	g.evalAlignment = newGroup.evalAlignment
 	g.Limit = newGroup.Limit
 	g.checksum = newGroup.checksum
 	g.Rules = newRules
@@ -337,7 +339,7 @@ func (g *Group) Init() {
 		i := g.Interval.Seconds()
 		return i
 	})
-	g.metrics.iterationLimit = g.metrics.set.NewGauge(fmt.Sprintf(`vmalert_rule_group_results_limit{%s}`, labels), func() float64 {
+	g.metrics.iterationLimit = g.metrics.set.NewGauge(fmt.Sprintf(`vmalert_group_rule_results_limit{%s}`, labels), func() float64 {
 		g.mu.RLock()
 		limit := g.Limit
 		g.mu.RUnlock()
@@ -373,7 +375,7 @@ func (g *Group) Start(ctx context.Context, rw remotewrite.RWClient, rr datasourc
 				g.mu.Lock()
 				err := g.updateWith(ng)
 				if err != nil {
-					logger.Errorf("group %q: failed to update: %s", g.Name, err)
+					logger.Errorf("group %q (file=%q): failed to update: %s", g.Name, g.File, err)
 					g.mu.Unlock()
 					continue
 				}
@@ -412,7 +414,7 @@ func (g *Group) Start(ctx context.Context, rw remotewrite.RWClient, rr datasourc
 		errs := e.execConcurrently(ctx, g.Rules, ts, g.Concurrency, resolveDuration, g.Limit)
 		for err := range errs {
 			if err != nil {
-				logger.Errorf("group %q: %s", g.Name, err)
+				logger.Errorf("group %q (file=%q): %s", g.Name, g.File, err)
 			}
 		}
 		g.metrics.iterationDuration.UpdateDuration(start)
@@ -441,17 +443,17 @@ func (g *Group) Start(ctx context.Context, rw remotewrite.RWClient, rr datasourc
 	if rr != nil {
 		err := g.restore(ctx, rr, realEvalTS, *remoteReadLookBack)
 		if err != nil {
-			logger.Errorf("error while restoring ruleState for group %q: %s", g.Name, err)
+			logger.Errorf("error while restoring ruleState for group %q (file=%q): %s", g.Name, g.File, err)
 		}
 	}
 
 	for {
 		select {
 		case <-ctx.Done():
-			logger.Infof("group %q: context cancelled", g.Name)
+			logger.Infof("group %q (file=%q): context cancelled", g.Name, g.File)
 			return
 		case <-g.doneCh:
-			logger.Infof("group %q: received stop signal", g.Name)
+			logger.Infof("group %q (file=%q): received stop signal", g.Name, g.File)
 			return
 		case ng := <-g.updateCh:
 			g.mu.Lock()
@@ -465,7 +467,7 @@ func (g *Group) Start(ctx context.Context, rw remotewrite.RWClient, rr datasourc
 
 			err := g.updateWith(ng)
 			if err != nil {
-				logger.Errorf("group %q: failed to update: %s", g.Name, err)
+				logger.Errorf("group %q (file=%q): failed to update: %s", g.Name, g.File, err)
 				g.mu.Unlock()
 				continue
 			}
@@ -543,8 +545,8 @@ func (g *Group) delayBeforeStart(ts time.Time, maxDelay time.Duration) time.Dura
 
 func (g *Group) infof(format string, args ...any) {
 	msg := fmt.Sprintf(format, args...)
-	logger.Infof("group %q %s; interval=%v; eval_offset=%v; concurrency=%d",
-		g.Name, msg, g.Interval, g.EvalOffset, g.Concurrency)
+	logger.Infof("group %q (file=%q; interval=%v; eval_offset=%v; concurrency=%d) %s",
+		g.Name, g.File, g.Interval, g.EvalOffset, g.Concurrency, msg)
 }
 
 // Replay performs group replay

app/vmalert/rule/group_test.go

@@ -78,6 +78,12 @@ func TestUpdateWith(t *testing.T) {
 		if g.Debug != expect.Debug {
 			t.Fatalf("expected to have debug %v; got %v", expect.Debug, g.Debug)
 		}
+		if !durationPtrEqual(g.EvalDelay, expect.EvalDelay) {
+			t.Fatalf("expected to have eval_delay %v; got %v", expect.EvalDelay, g.EvalDelay)
+		}
+		if !boolPtrEqual(g.evalAlignment, expect.evalAlignment) {
+			t.Fatalf("expected to have eval_alignment %v; got %v", expect.evalAlignment, g.evalAlignment)
+		}
 	}
 
 	// new rule
@@ -237,6 +243,37 @@ func TestUpdateWith(t *testing.T) {
 			{Alert: "foo1", Debug: &debug},
 		},
 	})
+
+	// update group evaluation settings
+	evalDelay := promutil.NewDuration(time.Minute)
+	evalAlignment := false
+	f(config.Group{
+		Rules: []config.Rule{{
+			Record: "foo",
+			Expr:   "max(up)",
+		}},
+	}, config.Group{
+		EvalDelay:     evalDelay,
+		EvalAlignment: &evalAlignment,
+		Rules: []config.Rule{{
+			Record: "foo",
+			Expr:   "min(up)",
+		}},
+	})
+}
+
+func durationPtrEqual(a, b *time.Duration) bool {
+	if a == nil || b == nil {
+		return a == b
+	}
+	return *a == *b
+}
+
+func boolPtrEqual(a, b *bool) bool {
+	if a == nil || b == nil {
+		return a == b
+	}
+	return *a == *b
 }
 
 func TestUpdateDuringRandSleep(t *testing.T) {

app/vmalert/rule/recording.go

@@ -208,7 +208,11 @@ func (rr *RecordingRule) exec(ctx context.Context, ts time.Time, limit int) ([]p
 		return nil, curState.Err
 	}
 
-	rr.logDebugf(ts, "query returned %d samples (elapsed: %s, isPartial: %t)", curState.Samples, curState.Duration, isPartialResponse(res))
+	seriesFetched := 0
+	if res.SeriesFetched != nil {
+		seriesFetched = *res.SeriesFetched
+	}
+	rr.logDebugf(ts, "query returned %d samples (series_fetched: %d, elapsed: %s, isPartial: %t)", curState.Samples, seriesFetched, curState.Duration, isPartialResponse(res))
 
 	qMetrics := res.Data
 	numSeries := len(qMetrics)

app/vmalert/rule/rule.go

@@ -132,9 +132,10 @@ func replayRule(r Rule, start, end time.Time, rw remotewrite.RWClient, replayRul
 		var esc *httpserver.ErrorWithStatusCode
 		if errors.As(err, &esc) {
 			statusCode := esc.StatusCode
-			// if the status code is 422, it means that the query was executed but failed due to an expression syntax error or a the resource limit being hit,
-			// continue replaying but skip the problematic execution if continueWithExecutionErr is true, otherwise, return the error without retry.
-			if statusCode == http.StatusUnprocessableEntity {
+			// if the status code is 400 or 422, the query failed due to reasons such as an expression syntax error or a resource limit being hit,
+			// rather than datasource unavailability.
+			// Continue replaying but skip the problematic execution if continueWithExecutionErr is true, otherwise, return the error without retry.
+			if statusCode == http.StatusUnprocessableEntity || statusCode == http.StatusBadRequest {
 				if continueWithExecutionErr {
 					logger.Errorf("rule %q: %s", r, err)
 					return 0, nil

app/vmauth/main.go

@@ -96,6 +96,7 @@ func main() {
 	flag.CommandLine.SetOutput(os.Stdout)
 	flag.Usage = usage
 	envflag.Parse()
+	initSecretFlags()
 	buildinfo.Init()
 	logger.Init()
 
@@ -911,3 +912,8 @@ func slowdownUnauthorizedResponse(r *http.Request) {
 	}
 	timerpool.Put(t)
 }
+
+// initSecretFlags manages the secret flags for this app and must be called after flag parsing and before logger init.
+func initSecretFlags() {
+	pushmetrics.InitSecretFlags()
+}

app/vmbackup/main.go

@@ -47,9 +47,8 @@ func main() {
 	// Write flags and help message to stdout, since it is easier to grep or pipe.
 	flag.CommandLine.SetOutput(os.Stdout)
 	flag.Usage = usage
-	flagutil.RegisterSecretFlag("snapshot.createURL")
-	flagutil.RegisterSecretFlag("snapshot.deleteURL")
 	envflag.Parse()
+	initSecretFlags()
 	buildinfo.Init()
 	logger.Init()
 
@@ -273,3 +272,10 @@ func newRemoteOriginFS(ctx context.Context) (common.RemoteFS, error) {
 	}
 	return fs, nil
 }
+
+// initSecretFlags manages the secret flags for this app and must be called after flag parsing and before logger init.
+func initSecretFlags() {
+	flagutil.RegisterSecretFlag("snapshot.createURL")
+	flagutil.RegisterSecretFlag("snapshot.deleteURL")
+	pushmetrics.InitSecretFlags()
+}

app/vmctl/main.go

@@ -47,6 +47,7 @@ func main() {
 	start := time.Now()
 	beforeFn := func(c *cli.Context) error {
 		flag.Parse()
+		initSecretFlags()
 		logger.Init()
 		isSilent = c.Bool(globalSilent)
 		if c.Bool(globalDisableProgressBar) {
@@ -619,3 +620,8 @@ func initConfigVM(c *cli.Context) (vm.Config, error) {
 		Backoff:            bf,
 	}, nil
 }
+
+// initSecretFlags manages the secret flags for this app and must be called after flag parsing and before logger init.
+func initSecretFlags() {
+	pushmetrics.InitSecretFlags()
+}

app/vmrestore/main.go

@@ -38,6 +38,7 @@ func main() {
 	flag.CommandLine.SetOutput(os.Stdout)
 	flag.Usage = usage
 	envflag.Parse()
+	initSecretFlags()
 	buildinfo.Init()
 	logger.Init()
 
@@ -112,3 +113,8 @@ func newSrcFS(ctx context.Context) (common.RemoteFS, error) {
 	}
 	return fs, nil
 }
+
+// initSecretFlags manages the secret flags for this app and must be called after flag parsing and before logger init.
+func initSecretFlags() {
+	pushmetrics.InitSecretFlags()
+}

app/vmselect/main.go

@@ -59,6 +59,12 @@ func Init(vmselectMaxConcurrentRequests int, vmselectMaxQueueDuration time.Durat
 	initVMUIConfig()
 
 	vmalertproxy.Init(*vmalertProxyURL)
+
+}
+
+// InitSecretFlags manages the secret flags for this pkg and must be called by app-level initSecretFlags.
+// It should run before logger initialization and package Init() (if exists).
+func InitSecretFlags() {
 	flagutil.RegisterSecretFlag("vmalert.proxyURL")
 }
 

app/vmselect/netstorage/netstorage.go

@@ -20,6 +20,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/cgroup"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fasttime"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/querytracer"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/storage"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/metricnamestats"
@@ -30,11 +31,12 @@ var (
 	maxSamplesPerSeries = flag.Int("search.maxSamplesPerSeries", 30e6, "The maximum number of raw samples a single query can scan per each time series. This option allows limiting memory usage")
 	maxSamplesPerQuery  = flag.Int("search.maxSamplesPerQuery", 1e9, "The maximum number of raw samples a single query can process across all time series. "+
 		"This protects from heavy queries, which select unexpectedly high number of raw samples. See also -search.maxSamplesPerSeries")
-	maxWorkersPerQuery = flag.Int("search.maxWorkersPerQuery", defaultMaxWorkersPerQuery, "The maximum number of CPU cores a single query can use. "+
-		"The default value should work good for most cases. "+
-		"The flag can be set to lower values for improving performance of big number of concurrently executed queries. "+
-		"The flag can be set to bigger values for improving performance of heavy queries, which scan big number of time series (>10K) and/or big number of samples (>100M). "+
-		"There is no sense in setting this flag to values bigger than the number of CPU cores available on the system")
+	maxWorkersPerQuery = flagutil.NewIntWithDynamicDefault("search.maxWorkersPerQuery", defaultMaxWorkersPerQuery, "netstorage.defaultMaxWorkersPerQuery()",
+		"The maximum number of CPU cores a single query can use. "+
+			"The default value should work good for most cases. "+
+			"The flag can be set to lower values for improving performance of big number of concurrently executed queries. "+
+			"The flag can be set to bigger values for improving performance of heavy queries, which scan big number of time series (>10K) and/or big number of samples (>100M). "+
+			"There is no sense in setting this flag to values bigger than the number of CPU cores available on the system")
 )
 
 // Result is a single timeseries result.

app/vmselect/netstorage/tmp_blocks_file.go

@@ -62,6 +62,9 @@ type tmpBlocksFile struct {
 	r *fs.ReaderAt
 
 	offset uint64
+
+	// err stores the first error occurred while writing the temporary blocks file.
+	err error
 }
 
 func getTmpBlocksFile() *tmpBlocksFile {
@@ -82,6 +85,7 @@ func putTmpBlocksFile(tbf *tmpBlocksFile) {
 	tbf.f = nil
 	tbf.r = nil
 	tbf.offset = 0
+	tbf.err = nil
 	tmpBlocksFilePool.Put(tbf)
 }
 
@@ -109,6 +113,10 @@ var (
 // and this must be handled.
 func (tbf *tmpBlocksFile) WriteBlockRefData(b []byte) (tmpBlockAddr, error) {
 	var addr tmpBlockAddr
+	if tbf.err != nil {
+		// Do not write anything to the tbf after the first failed write
+		return addr, tbf.err
+	}
 	addr.offset = tbf.offset
 	addr.size = len(b)
 	tbf.offset += uint64(addr.size)
@@ -122,7 +130,8 @@ func (tbf *tmpBlocksFile) WriteBlockRefData(b []byte) (tmpBlockAddr, error) {
 	if tbf.f == nil {
 		f, err := os.CreateTemp(tmpBlocksDir, "")
 		if err != nil {
-			return addr, err
+			tbf.err = fmt.Errorf("cannot create temporary blocks file at %q: %w", tmpBlocksDir, err)
+			return addr, tbf.err
 		}
 		tbf.f = f
 		tmpBlocksFilesCreated.Inc()
@@ -130,7 +139,9 @@ func (tbf *tmpBlocksFile) WriteBlockRefData(b []byte) (tmpBlockAddr, error) {
 	_, err := tbf.f.Write(tbf.buf)
 	tbf.buf = append(tbf.buf[:0], b...)
 	if err != nil {
-		return addr, fmt.Errorf("cannot write block to %q: %w", tbf.f.Name(), err)
+		// The blocks buffered at tbf.buf could be partially lost, mark the tbf as unusable.
+		tbf.err = fmt.Errorf("cannot write block to %q: %w", tbf.f.Name(), err)
+		return addr, tbf.err
 	}
 	return addr, nil
 }
@@ -141,12 +152,16 @@ func (tbf *tmpBlocksFile) Len() uint64 {
 }
 
 func (tbf *tmpBlocksFile) Finalize() error {
+	if tbf.err != nil {
+		return tbf.err
+	}
 	if tbf.f == nil {
 		return nil
 	}
 	fname := tbf.f.Name()
 	if _, err := tbf.f.Write(tbf.buf); err != nil {
-		return fmt.Errorf("cannot write the remaining %d bytes to %q: %w", len(tbf.buf), fname, err)
+		tbf.err = fmt.Errorf("cannot write the remaining %d bytes to %q: %w", len(tbf.buf), fname, err)
+		return tbf.err
 	}
 	tbf.buf = tbf.buf[:0]
 	r := fs.NewReaderAt(tbf.f)
@@ -166,6 +181,10 @@ func (tbf *tmpBlocksFile) Finalize() error {
 }
 
 func (tbf *tmpBlocksFile) MustReadBlockRefAt(partRef storage.PartRef, addr tmpBlockAddr) storage.BlockRef {
+	if tbf.err != nil {
+		// This should never happen, since Finalize() already returns the error for such a tbf.
+		logger.Panicf("BUG: cannot read block at %s from the temporary blocks file with the failed write: %s", addr, tbf.err)
+	}
 	var buf []byte
 	if tbf.r == nil {
 		buf = tbf.buf[addr.offset : addr.offset+uint64(addr.size)]

app/vmselect/prometheus/prometheus.go

@@ -516,7 +516,7 @@ func DeleteHandler(startTime time.Time, r *http.Request) error {
 	cp.deadline = searchutil.GetDeadlineForDelete(r, startTime)
 
 	if !cp.IsDefaultTimeRange() {
-		return fmt.Errorf("start=%d and end=%d args aren't supported. Remove these args from the query in order to delete all the matching metrics", cp.start, cp.end)
+		return fmt.Errorf("delete API does not support specific time ranges using start and end args, the series can only be deleted completely")
 	}
 	sq := storage.NewSearchQuery(cp.start, cp.end, cp.filterss, *maxDeleteSeries)
 	deletedCount, err := netstorage.DeleteSeries(nil, sq, cp.deadline)
@@ -540,11 +540,11 @@ func LabelValuesHandler(qt *querytracer.Tracer, startTime time.Time, labelName s
 
 	cp, err := getCommonParamsForLabelsAPI(r, startTime, false)
 	if err != nil {
-		return err
+		return httpserver.InvalidParamError(err)
 	}
 	limit, err := httputil.GetInt(r, "limit")
 	if err != nil {
-		return err
+		return httpserver.InvalidParamError(err)
 	}
 	sq := storage.NewSearchQuery(cp.start, cp.end, cp.filterss, *maxLabelsAPISeries)
 
@@ -584,7 +584,7 @@ func TSDBStatusHandler(qt *querytracer.Tracer, startTime time.Time, w http.Respo
 
 	cp, err := getCommonParams(r, startTime, false)
 	if err != nil {
-		return err
+		return httpserver.InvalidParamError(err)
 	}
 	cp.deadline = searchutil.GetDeadlineForStatusRequest(r, startTime)
 
@@ -596,7 +596,7 @@ func TSDBStatusHandler(qt *querytracer.Tracer, startTime time.Time, w http.Respo
 		} else {
 			t, err := time.Parse("2006-01-02", dateStr)
 			if err != nil {
-				return fmt.Errorf("cannot parse `date` arg %q: %w", dateStr, err)
+				return httpserver.InvalidParamError(fmt.Errorf("cannot parse `date` arg %q: %w", dateStr, err))
 			}
 			date = uint64(t.Unix()) / secsPerDay
 		}
@@ -607,7 +607,7 @@ func TSDBStatusHandler(qt *querytracer.Tracer, startTime time.Time, w http.Respo
 	if len(topNStr) > 0 {
 		n, err := strconv.Atoi(topNStr)
 		if err != nil {
-			return fmt.Errorf("cannot parse `topN` arg %q: %w", topNStr, err)
+			return httpserver.InvalidParamError(fmt.Errorf("cannot parse `topN` arg %q: %w", topNStr, err))
 		}
 		if n <= 0 {
 			n = 1
@@ -645,11 +645,11 @@ func LabelsHandler(qt *querytracer.Tracer, startTime time.Time, w http.ResponseW
 
 	cp, err := getCommonParamsForLabelsAPI(r, startTime, false)
 	if err != nil {
-		return err
+		return httpserver.InvalidParamError(err)
 	}
 	limit, err := httputil.GetInt(r, "limit")
 	if err != nil {
-		return err
+		return httpserver.InvalidParamError(err)
 	}
 	sq := storage.NewSearchQuery(cp.start, cp.end, cp.filterss, *maxLabelsAPISeries)
 	labels, err := netstorage.LabelNames(qt, sq, limit, cp.deadline)
@@ -671,10 +671,9 @@ func LabelsHandler(qt *querytracer.Tracer, startTime time.Time, w http.ResponseW
 //
 // See https://prometheus.io/docs/prometheus/latest/querying/api/#querying-metric-metadata
 func MetadataHandler(qt *querytracer.Tracer, startTime time.Time, w http.ResponseWriter, r *http.Request) error {
-
 	limit, err := httputil.GetInt(r, "limit")
 	if err != nil {
-		return err
+		return httpserver.InvalidParamError(err)
 	}
 	if limit < 0 {
 		limit = 0
@@ -734,11 +733,11 @@ func SeriesHandler(qt *querytracer.Tracer, startTime time.Time, w http.ResponseW
 	// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/91
 	cp, err := getCommonParamsForLabelsAPI(r, startTime, true)
 	if err != nil {
-		return err
+		return httpserver.InvalidParamError(err)
 	}
 	limit, err := httputil.GetInt(r, "limit")
 	if err != nil {
-		return err
+		return httpserver.InvalidParamError(err)
 	}
 
 	sq := storage.NewSearchQuery(cp.start, cp.end, cp.filterss, *maxSeriesLimit)
@@ -772,19 +771,19 @@ func QueryHandler(qt *querytracer.Tracer, startTime time.Time, w http.ResponseWr
 	mayCache := !httputil.GetBool(r, "nocache")
 	query := r.FormValue("query")
 	if len(query) == 0 {
-		return fmt.Errorf("missing `query` arg")
+		return httpserver.InvalidParamError(fmt.Errorf("missing `query` arg"))
 	}
 	start, err := httputil.GetTime(r, "time", ct)
 	if err != nil {
-		return err
+		return httpserver.InvalidParamError(err)
 	}
 	lookbackDelta, err := getMaxLookback(r)
 	if err != nil {
-		return err
+		return httpserver.InvalidParamError(err)
 	}
 	step, err := httputil.GetDuration(r, "step", lookbackDelta)
 	if err != nil {
-		return err
+		return httpserver.InvalidParamError(err)
 	}
 	if step <= 0 {
 		step = defaultStep
@@ -792,16 +791,16 @@ func QueryHandler(qt *querytracer.Tracer, startTime time.Time, w http.ResponseWr
 
 	maxLen := searchutil.GetMaxQueryLen()
 	if len(query) > maxLen {
-		return fmt.Errorf("too long query; got %d bytes; mustn't exceed `-search.maxQueryLen=%d` bytes", len(query), maxLen)
+		return httpserver.InvalidParamError(fmt.Errorf("too long query; got %d bytes; mustn't exceed `-search.maxQueryLen=%d` bytes", len(query), maxLen))
 	}
 	etfs, err := searchutil.GetExtraTagFilters(r)
 	if err != nil {
-		return err
+		return httpserver.InvalidParamError(err)
 	}
 	if childQuery, windowExpr, offsetExpr := promql.IsMetricSelectorWithRollup(query); childQuery != "" {
 		window, err := windowExpr.NonNegativeDuration(step)
 		if err != nil {
-			return fmt.Errorf("cannot parse lookbehind window in square brackets at %s: %w", query, err)
+			return httpserver.InvalidParamError(fmt.Errorf("cannot parse lookbehind window in square brackets at %s: %w", query, err))
 		}
 		offset := offsetExpr.Duration(step)
 		start -= offset
@@ -815,7 +814,7 @@ func QueryHandler(qt *querytracer.Tracer, startTime time.Time, w http.ResponseWr
 
 		tagFilterss, err := getTagFilterssFromMatches([]string{childQuery})
 		if err != nil {
-			return err
+			return httpserver.InvalidParamError(err)
 		}
 		filterss := searchutil.JoinTagFilterss(tagFilterss, etfs)
 
@@ -831,22 +830,25 @@ func QueryHandler(qt *querytracer.Tracer, startTime time.Time, w http.ResponseWr
 		return nil
 	}
 	if childQuery, windowExpr, stepExpr, offsetExpr := promql.IsRollup(query); childQuery != "" {
+		if len(childQuery) > maxLen {
+			return httpserver.InvalidParamError(fmt.Errorf("too long query; got %d bytes; mustn't exceed `-search.maxQueryLen=%d` bytes", len(childQuery), maxLen))
+		}
 		newStep, err := stepExpr.NonNegativeDuration(step)
 		if err != nil {
-			return fmt.Errorf("cannot parse step in square brackets at %s: %w", query, err)
+			return httpserver.InvalidParamError(fmt.Errorf("cannot parse step in square brackets at %s: %w", query, err))
 		}
 		if newStep > 0 {
 			step = newStep
 		}
 		window, err := windowExpr.NonNegativeDuration(step)
 		if err != nil {
-			return fmt.Errorf("cannot parse lookbehind window in square brackets at %s: %w", query, err)
+			return httpserver.InvalidParamError(fmt.Errorf("cannot parse lookbehind window in square brackets at %s: %w", query, err))
 		}
 		offset := offsetExpr.Duration(step)
 		start -= offset
 		end := start
 		start = end - window
-		if err := queryRangeHandler(qt, startTime, w, childQuery, start, end, step, r, ct, etfs); err != nil {
+		if err := queryRangeHandler(qt, startTime, w, childQuery, start, end, step, lookbackDelta, r, ct, etfs); err != nil {
 			return fmt.Errorf("error when executing query=%q on the time range (start=%d, end=%d, step=%d): %w", childQuery, start, end, step, err)
 		}
 		return nil
@@ -854,7 +856,7 @@ func QueryHandler(qt *querytracer.Tracer, startTime time.Time, w http.ResponseWr
 
 	queryOffset, err := getLatencyOffsetMilliseconds(r)
 	if err != nil {
-		return err
+		return httpserver.InvalidParamError(err)
 	}
 	if !httputil.GetBool(r, "nocache") && ct-start < queryOffset && start-ct < queryOffset {
 		// Adjust start time only if `nocache` arg isn't set.
@@ -928,45 +930,43 @@ func QueryRangeHandler(qt *querytracer.Tracer, startTime time.Time, w http.Respo
 	ct := startTime.UnixNano() / 1e6
 	query := r.FormValue("query")
 	if len(query) == 0 {
-		return fmt.Errorf("missing `query` arg")
+		return httpserver.InvalidParamError(fmt.Errorf("missing `query` arg"))
+	}
+	maxLen := searchutil.GetMaxQueryLen()
+	if len(query) > maxLen {
+		return httpserver.InvalidParamError(fmt.Errorf("too long query; got %d bytes; mustn't exceed `-search.maxQueryLen=%d` bytes", len(query), maxLen))
 	}
 	start, err := httputil.GetTime(r, "start", ct-defaultStep)
 	if err != nil {
-		return err
+		return httpserver.InvalidParamError(err)
 	}
 	end, err := httputil.GetTime(r, "end", ct)
 	if err != nil {
-		return err
+		return httpserver.InvalidParamError(err)
 	}
 	step, err := httputil.GetDuration(r, "step", defaultStep)
 	if err != nil {
-		return err
+		return httpserver.InvalidParamError(err)
 	}
 	etfs, err := searchutil.GetExtraTagFilters(r)
 	if err != nil {
-		return err
+		return httpserver.InvalidParamError(err)
 	}
-	if err := queryRangeHandler(qt, startTime, w, query, start, end, step, r, ct, etfs); err != nil {
+	lookbackDelta, err := getMaxLookback(r)
+	if err != nil {
+		return httpserver.InvalidParamError(err)
+	}
+	if err := queryRangeHandler(qt, startTime, w, query, start, end, step, lookbackDelta, r, ct, etfs); err != nil {
 		return fmt.Errorf("error when executing query=%q on the time range (start=%d, end=%d, step=%d): %w", query, start, end, step, err)
 	}
 	return nil
 }
 
 func queryRangeHandler(qt *querytracer.Tracer, startTime time.Time, w http.ResponseWriter, query string,
-	start, end, step int64, r *http.Request, ct int64, etfs [][]storage.TagFilter) error {
+	start, end, step, lookbackDelta int64, r *http.Request, ct int64, etfs [][]storage.TagFilter) error {
 	deadline := searchutil.GetDeadlineForQuery(r, startTime)
 	mayCache := !httputil.GetBool(r, "nocache")
 	optimizeRepeatedBinaryOpSubexprs := httputil.GetBool(r, "optimize_repeated_binary_op_subexprs")
-	lookbackDelta, err := getMaxLookback(r)
-	if err != nil {
-		return err
-	}
-
-	// Validate input args.
-	maxLen := searchutil.GetMaxQueryLen()
-	if len(query) > maxLen {
-		return fmt.Errorf("too long query; got %d bytes; mustn't exceed `-search.maxQueryLen=%d` bytes", len(query), maxLen)
-	}
 	if start > end {
 		end = start + defaultStep
 	}
@@ -1005,7 +1005,7 @@ func queryRangeHandler(qt *querytracer.Tracer, startTime time.Time, w http.Respo
 	if step < maxStepForPointsAdjustment.Milliseconds() {
 		queryOffset, err := getLatencyOffsetMilliseconds(r)
 		if err != nil {
-			return err
+			return httpserver.InvalidParamError(err)
 		}
 		if ct-queryOffset < end {
 			result = adjustLastPoints(result, ct-queryOffset, ct+step)
@@ -1156,13 +1156,13 @@ func QueryStatsHandler(w http.ResponseWriter, r *http.Request) error {
 	if len(topNStr) > 0 {
 		n, err := strconv.Atoi(topNStr)
 		if err != nil {
-			return fmt.Errorf("cannot parse `topN` arg %q: %w", topNStr, err)
+			return httpserver.InvalidParamError(fmt.Errorf("cannot parse `topN` arg %q: %w", topNStr, err))
 		}
 		topN = n
 	}
 	maxLifetimeMsecs, err := httputil.GetDuration(r, "maxLifetime", 10*60*1000)
 	if err != nil {
-		return fmt.Errorf("cannot parse `maxLifetime` arg: %w", err)
+		return httpserver.InvalidParamError(fmt.Errorf("cannot parse `maxLifetime` arg: %w", err))
 	}
 	maxLifetime := time.Duration(maxLifetimeMsecs) * time.Millisecond
 	w.Header().Set("Content-Type", "application/json")

app/vmselect/promql/exec.go

@@ -11,6 +11,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/netstorage"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/querystats"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/decimal"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/querytracer"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/storage"
@@ -46,15 +47,15 @@ func Exec(qt *querytracer.Tracer, ec *EvalConfig, q string, isFirstPointOnly boo
 
 	e, err := parsePromQLWithCache(q)
 	if err != nil {
-		return nil, err
+		return nil, httpserver.InvalidParamError(err)
 	}
 
 	if *disableImplicitConversion || *logImplicitConversion {
 		isInvalid := metricsql.IsLikelyInvalid(e)
 		if isInvalid && *disableImplicitConversion {
 			// we don't add query=%q to err message as it will be added by the caller
-			return nil, fmt.Errorf("query requires implicit conversion and is rejected according to -search.disableImplicitConversion command-line flag. " +
-				"See https://docs.victoriametrics.com/victoriametrics/metricsql/#implicit-query-conversions for details")
+			return nil, httpserver.InvalidParamError(fmt.Errorf("query requires implicit conversion and is rejected according to -search.disableImplicitConversion command-line flag. " +
+				"See https://docs.victoriametrics.com/victoriametrics/metricsql/#implicit-query-conversions for details"))
 		}
 		if isInvalid && *logImplicitConversion {
 			logger.Warnf("query=%q requires implicit conversion, see https://docs.victoriametrics.com/victoriametrics/metricsql/#implicit-query-conversions for details", e.AppendString(nil))

app/vmselect/promql/transform.go

@@ -2,6 +2,7 @@ package promql
 
 import (
 	"bytes"
+	"errors"
 	"fmt"
 	"math"
 	"math/rand"
@@ -2566,6 +2567,11 @@ func isDecimalChar(ch byte) bool {
 func mustParseNum(s string) float64 {
 	f, err := strconv.ParseFloat(s, 64)
 	if err != nil {
+		if errors.Is(err, strconv.ErrRange) {
+			// The number is too large to fit into float64; ParseFloat returns ±Inf in this case.
+			// Use ±Inf for sorting purposes — it is semantically correct.
+			return f
+		}
 		logger.Panicf("BUG: unexpected error when parsing the number %q: %s", s, err)
 	}
 	return f

app/vmselect/promql/transform_test.go

@@ -385,4 +385,12 @@ func TestNumericLess(t *testing.T) {
 	f("12.9", "12.56", false)
 	f("12.56", "12.9", true)
 	f("12.9", "12.9", false)
+
+	// 309-digit numbers - must not panic (regression test for GHSA-9g98-8jgr-x2vv)
+	big := strings.Repeat("9", 309)
+	f(big, "1", false)
+	f("1", big, true)
+	f(big, big, false)
+	f("-"+big, big, true)
+	f(big, "-"+big, false)
 }

app/vmselect/vmui/assets/favicon.svg

@@ -0,0 +1,5 @@
+<svg width="48" height="48" fill="#020202" xmlns="http://www.w3.org/2000/svg">
+    <path d="M24.5475 0C10.3246.0265251 1.11379 3.06365 4.40623 6.10077c0 0 12.32997 11.23333 16.58217 14.84083.8131.6896 2.1728 1.1936 3.5191 1.2201h.1199c1.3463-.0265 2.706-.5305 3.5191-1.2201 4.2522-3.5942 16.5422-14.84083 16.5422-14.84083C48.0478 3.06365 38.8636.0265251 24.6674 0"/>
+    <path d="M28.1579 27.0159c-.8131.6896-2.1728 1.1936-3.5191 1.2201h-.12c-1.3463-.0265-2.7059-.5305-3.519-1.2201-2.9725-2.5067-13.35639-11.87-17.26201-15.3979v5.4112c0 .5968.22661 1.3793.6265 1.7506C7.00358 21.1936 17.2675 30.5437 20.9731 33.6737c.8132.6896 2.1728 1.1936 3.5191 1.2201h.12c1.3463-.0265 2.7059-.5305 3.519-1.2201 3.679-3.13 13.9429-12.4536 16.6089-14.8939.4132-.3713.6265-1.1538.6265-1.7506V11.618c-3.9323 3.5411-14.3162 12.931-17.2354 15.3979h.0267Z"/>
+    <path d="M28.1579 39.748c-.8131.6897-2.1728 1.1937-3.5191 1.2202h-.12c-1.3463-.0265-2.7059-.5305-3.519-1.2202-2.9725-2.4933-13.35639-11.8567-17.26201-15.3978v5.4111c0 .5969.22661 1.3793.6265 1.7507C7.00358 33.9258 17.2675 43.2759 20.9731 46.4058c.8132.6897 2.1728 1.1937 3.5191 1.2202h.12c1.3463-.0265 2.7059-.5305 3.519-1.2202 3.679-3.1299 13.9429-12.4535 16.6089-14.8938.4132-.3714.6265-1.1538.6265-1.7507v-5.4111c-3.9323 3.5411-14.3162 12.931-17.2354 15.3978h.0267Z"/>
+</svg>

app/vmselect/vmui/favicon.svg

@@ -1 +0,0 @@
-<svg width="48" height="48" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M24.5475 0C10.3246.0265251 1.11379 3.06365 4.40623 6.10077c0 0 12.32997 11.23333 16.58217 14.84083.8131.6896 2.1728 1.1936 3.5191 1.2201h.1199c1.3463-.0265 2.706-.5305 3.5191-1.2201 4.2522-3.5942 16.5422-14.84083 16.5422-14.84083C48.0478 3.06365 38.8636.0265251 24.6674 0" fill="#020202"/><path d="M28.1579 27.0159c-.8131.6896-2.1728 1.1936-3.5191 1.2201h-.12c-1.3463-.0265-2.7059-.5305-3.519-1.2201-2.9725-2.5067-13.35639-11.87-17.26201-15.3979v5.4112c0 .5968.22661 1.3793.6265 1.7506C7.00358 21.1936 17.2675 30.5437 20.9731 33.6737c.8132.6896 2.1728 1.1936 3.5191 1.2201h.12c1.3463-.0265 2.7059-.5305 3.519-1.2201 3.679-3.13 13.9429-12.4536 16.6089-14.8939.4132-.3713.6265-1.1538.6265-1.7506V11.618c-3.9323 3.5411-14.3162 12.931-17.2354 15.3979h.0267Z" fill="#020202"/><path d="M28.1579 39.748c-.8131.6897-2.1728 1.1937-3.5191 1.2202h-.12c-1.3463-.0265-2.7059-.5305-3.519-1.2202-2.9725-2.4933-13.35639-11.8567-17.26201-15.3978v5.4111c0 .5969.22661 1.3793.6265 1.7507C7.00358 33.9258 17.2675 43.2759 20.9731 46.4058c.8132.6897 2.1728 1.1937 3.5191 1.2202h.12c1.3463-.0265 2.7059-.5305 3.519-1.2202 3.679-3.1299 13.9429-12.4535 16.6089-14.8938.4132-.3714.6265-1.1538.6265-1.7507v-5.4111c-3.9323 3.5411-14.3162 12.931-17.2354 15.3978h.0267Z" fill="#020202"/></svg>

app/vmselect/vmui/index.html

@@ -2,9 +2,9 @@
 <html lang="en">
 <head>
   <meta charset="utf-8"/>
-  <link rel="icon" href="./favicon.svg"/>
-  <link rel="apple-touch-icon" href="./favicon.svg"/>
-  <link rel="mask-icon" href="./favicon.svg" color="#000000">
+  <link id="favicon" rel="icon" href="./assets/favicon.svg" />
+  <link rel="apple-touch-icon" href="./assets/favicon.svg" />
+  <link id="mask-icon" rel="mask-icon" href="./assets/favicon.svg" color="#000000">
 
   <meta name="robots" content="noindex">
   <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=5"/>
@@ -37,11 +37,11 @@
   <meta property="og:title" content="UI for VictoriaMetrics">
   <meta property="og:url" content="https://victoriametrics.com/">
   <meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data">
-  <script type="module" crossorigin src="./assets/index-B1dXK3k7.js"></script>
+  <script type="module" crossorigin src="./assets/index-BiDX4bB6.js"></script>
   <link rel="modulepreload" crossorigin href="./assets/rolldown-runtime-CNC7AqOf.js">
   <link rel="modulepreload" crossorigin href="./assets/vendor-DwJYpOdw.js">
   <link rel="stylesheet" crossorigin href="./assets/vendor-CnsZ1jie.css">
-  <link rel="stylesheet" crossorigin href="./assets/index-BJqoElx2.css">
+  <link rel="stylesheet" crossorigin href="./assets/index-CymA7XYg.css">
 </head>
 <body>
 <noscript>You need to enable JavaScript to run this app.</noscript>

app/vmselect/vmui/manifest.json

@@ -3,7 +3,7 @@
   "name": "vmui",
   "icons": [
     {
-      "src": "favicon.svg",
+      "src": "./assets/favicon.svg",
       "sizes": "any",
       "type": "image/svg+xml"
     }

app/vmui/Dockerfile-web

@@ -1,4 +1,4 @@
-FROM golang:1.26.5 AS build-web-stage
+FROM golang:1.26.6 AS build-web-stage
 COPY build /build
 
 WORKDIR /build

app/vmui/packages/vmui/src/components/Configurators/GlobalSettings/GlobalSettings.tsx

@@ -1,4 +1,4 @@
-import { FC, useRef } from "preact/compat";
+import { forwardRef, useImperativeHandle, useRef } from "preact/compat";
 import ServerConfigurator from "./ServerConfigurator/ServerConfigurator";
 import { ArrowDownIcon, SettingsIcon } from "../../Main/Icons";
 import Button from "../../Main/Button/Button";
@@ -21,7 +21,11 @@ export interface ChildComponentHandle {
   handleApply: () => void;
 }
 
-const GlobalSettings: FC = () => {
+export interface GlobalSettingsHandle {
+  open: () => void;
+}
+
+const GlobalSettings = forwardRef<GlobalSettingsHandle>((_, ref) => {
   const { isMobile } = useDeviceDetect();
 
   const appModeEnable = getAppModeEnable();
@@ -74,6 +78,10 @@ const GlobalSettings: FC = () => {
     },
   ].filter(control => control.show);
 
+  useImperativeHandle(ref, () => ({
+    open: handleOpen,
+  }));
+
   return <>
     {isMobile ? (
       <div
@@ -139,6 +147,6 @@ const GlobalSettings: FC = () => {
       </Modal>
     )}
   </>;
-};
+});
 
 export default GlobalSettings;

app/vmui/packages/vmui/src/components/Configurators/GlobalSettings/TimeZonePreview/TimeZonePreview.tsx

@@ -0,0 +1,52 @@
+import { FC } from "preact/compat";
+import Button from "../../../Main/Button/Button";
+import { useTimeState } from "../../../../state/time/TimeStateContext";
+import useDeviceDetect from "../../../../hooks/useDeviceDetect";
+import { getUTCByTimezone } from "../../../../utils/time";
+import { useMemo } from "react";
+import { ArrowDownIcon, PlanetIcon } from "../../../Main/Icons";
+
+type Props = {
+  onOpenSettings?: () => void;
+}
+
+const TimeZonePreview: FC<Props> = ({ onOpenSettings }) => {
+  const { isMobile } = useDeviceDetect();
+
+  const { timezone } = useTimeState();
+  const utcOffset = useMemo(() => getUTCByTimezone(timezone), [timezone]);
+
+  const handleOpenSettings = () => {
+    onOpenSettings && onOpenSettings();
+  };
+
+
+  if (isMobile) {
+    return (
+      <button
+        className="vm-mobile-option"
+        onClick={handleOpenSettings}
+      >
+        <span className="vm-mobile-option__icon"><PlanetIcon/></span>
+        <div className="vm-mobile-option-text">
+          <span className="vm-mobile-option-text__label">Time zone</span>
+          <span className="vm-mobile-option-text__value">{utcOffset}</span>
+        </div>
+        <span className="vm-mobile-option__arrow"><ArrowDownIcon/></span>
+      </button>
+    );
+  }
+
+  return (
+    <Button
+      className="vm-header-button"
+      onClick={handleOpenSettings}
+      startIcon={<PlanetIcon/>}
+    >
+      {utcOffset}
+    </Button>
+  );
+};
+
+
+export default TimeZonePreview;

app/vmui/packages/vmui/src/components/Configurators/StepConfigurator/StepConfigurator.tsx

@@ -113,6 +113,8 @@ const StepConfigurator: FC = () => {
     setError("");
   }, [defaultStep, prevDefaultStep, value, graphDispatch]);
 
+  const textValue = isAutoStep ? `auto (${customStep})` : customStep;
+
   return (
     <div
       className="vm-step-control"
@@ -126,7 +128,7 @@ const StepConfigurator: FC = () => {
           <span className="vm-mobile-option__icon"><TimelineIcon/></span>
           <div className="vm-mobile-option-text">
             <span className="vm-mobile-option-text__label">Step</span>
-            <span className="vm-mobile-option-text__value">{customStep}</span>
+            <span className="vm-mobile-option-text__value">{textValue}</span>
           </div>
           <span className="vm-mobile-option__arrow"><ArrowDownIcon/></span>
         </div>
@@ -138,7 +140,7 @@ const StepConfigurator: FC = () => {
           startIcon={<TimelineIcon/>}
           onClick={toggleOpenOptions}
         >
-          Step: {isAutoStep ? `auto (${customStep})` : customStep}
+          Step: {textValue}
         </Button>
       )}
       <Popper

app/vmui/packages/vmui/src/components/Configurators/TimeRangeSettings/TimeSelector/TimeSelector.tsx

@@ -19,7 +19,11 @@ import useBoolean from "../../../../hooks/useBoolean";
 import useWindowSize from "../../../../hooks/useWindowSize";
 import usePrevious from "../../../../hooks/usePrevious";
 
-export const TimeSelector: FC = () => {
+type Props = {
+  onOpenSettings?: () => void;
+}
+
+export const TimeSelector: FC<Props> = ({ onOpenSettings }) => {
   const { isMobile } = useDeviceDetect();
   const { isDarkTheme } = useAppState();
   const wrapperRef = useRef<HTMLDivElement>(null);
@@ -53,7 +57,7 @@ export const TimeSelector: FC = () => {
     setFrom(formatDateForNativeInput(dateFromSeconds(start)));
   }, [timezone, start]);
 
-  const setDuration = ({ duration, until, id }: {duration: string, until: Date, id: string}) => {
+  const setDuration = ({ duration, until, id }: { duration: string, until: Date, id: string }) => {
     dispatch({ type: "SET_RELATIVE_TIME", payload: { duration, until, id } });
     handleCloseOptions();
   };
@@ -75,16 +79,23 @@ export const TimeSelector: FC = () => {
 
   const setTimeAndClosePicker = () => {
     if (from && until) {
-      dispatch({ type: "SET_PERIOD", payload: {
-        from: dayjs.tz(from).toDate(),
-        to: dayjs.tz(until).toDate()
-      } });
+      dispatch({
+        type: "SET_PERIOD", payload: {
+          from: dayjs.tz(from).toDate(),
+          to: dayjs.tz(until).toDate()
+        }
+      });
     }
     handleCloseOptions();
   };
 
   const onSwitchToNow = () => dispatch({ type: "RUN_QUERY_TO_NOW" });
 
+  const handleOpenSettings = () => {
+    onOpenSettings && onOpenSettings();
+    handleCloseOptions();
+  };
+
   const onCancelClick = () => {
     setUntil(formatDateForNativeInput(dateFromSeconds(end)));
     setFrom(formatDateForNativeInput(dateFromSeconds(start)));
@@ -140,6 +151,7 @@ export const TimeSelector: FC = () => {
         </Tooltip>
       )}
     </div>
+
     <Popper
       open={openOptions}
       buttonRef={buttonRef}
@@ -179,13 +191,17 @@ export const TimeSelector: FC = () => {
               onEnter={setTimeAndClosePicker}
             />
           </div>
-          <div className="vm-time-selector-left-timezone">
-            <div className="vm-time-selector-left-timezone__title">{activeTimezone.region}</div>
-            <div className="vm-time-selector-left-timezone__utc">{activeTimezone.utc}</div>
-          </div>
+          <button
+            type="button"
+            className="vm-time-selector-left-timezone"
+            onClick={handleOpenSettings}
+          >
+            <span className="vm-time-selector-left-timezone__title">{activeTimezone.region}</span>
+            <span className="vm-time-selector-left-timezone__utc">{activeTimezone.utc}</span>
+          </button>
           <Button
             variant="text"
-            startIcon={<AlarmIcon />}
+            startIcon={<AlarmIcon/>}
             onClick={onSwitchToNow}
           >
             switch to now

app/vmui/packages/vmui/src/components/Configurators/TimeRangeSettings/TimeSelector/style.scss

@@ -40,8 +40,13 @@
       gap: $padding-small;
       font-size: $font-size-small;
       margin-bottom: $padding-small;
+      color: $color-text;
+      cursor: pointer;
 
-      &__title {}
+      &:hover {
+        color: $color-primary;
+        text-decoration: underline;
+      }
 
       &__utc {
         display: inline-flex;

app/vmui/packages/vmui/src/components/Main/Icons/index.tsx

@@ -634,6 +634,17 @@ export const DebugIcon = () => (
   </svg>
 );
 
+export const PlanetIcon = () => (
+  <svg
+    viewBox="0 0 24 24"
+    fill="currentColor"
+  >
+    <path
+      d="M12 2C6.48 2 2 6.48 2 12s4.48 10 10 10 10-4.48 10-10S17.52 2 12 2M4 12c0-.61.08-1.21.21-1.78L8.99 15v1c0 1.1.9 2 2 2v1.93C7.06 19.43 4 16.07 4 12m13.89 5.4c-.26-.81-1-1.4-1.9-1.4h-1v-3c0-.55-.45-1-1-1h-6v-2h2c.55 0 1-.45 1-1V7h2c1.1 0 2-.9 2-2v-.41C17.92 5.77 20 8.65 20 12c0 2.08-.81 3.98-2.11 5.4"
+    ></path>
+  </svg>
+);
+
 export const SystemIcon = () => (
   <svg
     viewBox="0 0 24 24"

app/vmui/packages/vmui/src/layouts/Header/HeaderControls/style.scss

@@ -11,6 +11,7 @@
   &_mobile {
     display: grid;
     grid-template-columns: 1fr;
+    gap: 0;
     padding: 0;
     flex-grow: initial;
 

app/vmui/packages/vmui/src/layouts/MainLayout/ControlsMainLayout.tsx

@@ -6,9 +6,11 @@ import StepConfigurator from "../../components/Configurators/StepConfigurator/St
 import { TimeSelector } from "../../components/Configurators/TimeRangeSettings/TimeSelector/TimeSelector";
 import CardinalityDatePicker from "../../components/Configurators/CardinalityDatePicker/CardinalityDatePicker";
 import { ExecutionControls } from "../../components/Configurators/TimeRangeSettings/ExecutionControls/ExecutionControls";
-import GlobalSettings from "../../components/Configurators/GlobalSettings/GlobalSettings";
+import GlobalSettings, { GlobalSettingsHandle } from "../../components/Configurators/GlobalSettings/GlobalSettings";
 import ShortcutKeys from "../../components/Main/ShortcutKeys/ShortcutKeys";
 import { ControlsProps } from "../Header/HeaderControls/HeaderControls";
+import { useRef } from "react";
+import TimeZonePreview from "../../components/Configurators/GlobalSettings/TimeZonePreview/TimeZonePreview";
 
 const ControlsMainLayout: FC<ControlsProps> = ({
   displaySidebar,
@@ -17,6 +19,7 @@ const ControlsMainLayout: FC<ControlsProps> = ({
   accountIds,
   closeModal,
 }) => {
+  const settingsRef = useRef<GlobalSettingsHandle>(null);
 
   return (
     <div
@@ -27,14 +30,15 @@ const ControlsMainLayout: FC<ControlsProps> = ({
     >
       {headerSetup?.tenant && <TenantsConfiguration accountIds={accountIds || []}/>}
       {headerSetup?.stepControl && <StepConfigurator/>}
-      {headerSetup?.timeSelector && <TimeSelector/>}
+      {headerSetup?.timeSelector && <TimeSelector onOpenSettings={() => settingsRef.current?.open()}/>}
       {headerSetup?.cardinalityDatePicker && <CardinalityDatePicker/>}
+      <TimeZonePreview onOpenSettings={() => settingsRef.current?.open()}/>
       {headerSetup?.executionControls && <ExecutionControls
         tooltip={headerSetup?.executionControls?.tooltip}
         useAutorefresh={headerSetup?.executionControls?.useAutorefresh}
         closeModal={closeModal}
       />}
-      <GlobalSettings/>
+      <GlobalSettings ref={settingsRef}/>
       {!displaySidebar && <ShortcutKeys/>}
     </div>
   );

app/vmui/packages/vmui/src/styles/components/mobile-option.scss

@@ -1,11 +1,12 @@
 @use "src/styles/variables" as *;
 
 .vm-mobile-option {
-  display: flex;
+  display: grid;
+  grid-template-columns: auto 1fr auto;
   align-items: center;
   justify-content: flex-start;
-  gap: $padding-small;
-  padding: calc($padding-medium/2) 0;
+  gap: $padding-global;
+  padding: $padding-global $padding-small;
   width: 100%;
   user-select: none;
 
@@ -17,14 +18,33 @@
   }
 
   &__icon {
-    width: 22px;
-    height: 22px;
+    position: relative;
+    display: flex;
+    width: 40px;
+    height: 40px;
     color: $color-primary;
+
+    &:after {
+      content: "";
+      position: absolute;
+      top: 0;
+      left: 0;
+      width: 100%;
+      height: 100%;
+      opacity: 0.1;
+      background-color: currentColor;
+      border-radius: $border-radius-medium;
+    }
+
+    svg {
+      width: 21px;
+      height: auto;
+    }
   }
 
   &__arrow {
-    width: 14px;
-    height: 14px;
+    width: 20px;
+    height: 20px;
     transform: rotate(-90deg);
     color: $color-primary;
   }
@@ -32,11 +52,13 @@
   &-text {
     display: grid;
     align-items: center;
-    gap: 2px;
+    height: 100%;
+    gap: calc($padding-small / 2);
     flex-grow: 1;
+    text-align: left;
 
     &__label {
-      font-weight: bold;
+      font-weight: 600;
     }
 
     &__value {

apptest/client.go

@@ -135,7 +135,7 @@ func tenantViaURL(addr, prefix, tenant, suffix string) string {
 }
 
 // tenantViaHeaders returns path in cluster's URL format where tenant is omitted in URL
-// Only supported if -enableMultitenancyViaHeaders is specified
+// Only supported if -enableMultitenancyViaHeaders is enabled
 func tenantViaHeaders(addr, prefix, suffix string) string {
 	return fmt.Sprintf("http://%s/%s/%s", addr, prefix, suffix)
 }

apptest/tests/multitenancy_via_headers_test.go

@@ -25,12 +25,10 @@ func TestClusterMultiTenantSelectViaHeaders(t *testing.T) {
 	})
 	vminsert := tc.MustStartVminsert("vminsert", []string{
 		"-storageNode=" + vmstorage.VminsertAddr(),
-		"-enableMultitenancyViaHeaders",
 	})
 	vmselect := tc.MustStartVmselect("vmselect", []string{
 		"-storageNode=" + vmstorage.VmselectAddr(),
 		"-search.tenantCacheExpireDuration=0",
-		"-enableMultitenancyViaHeaders",
 	})
 
 	multitenant := make(http.Header)

apptest/tests/vmagent_remotewrite_test.go

@@ -594,7 +594,6 @@ func TestSingleVMAgentMultitenancy(t *testing.T) {
 		fmt.Sprintf(`-remoteWrite.url=%s/api/v1/write`, remoteWriteSrv.URL),
 		"-remoteWrite.tmpDataPath=" + tc.Dir() + "/vmagent-multitenancy",
 		"-enableMultitenantHandlers",
-		"-enableMultitenancyViaHeaders",
 	})
 
 	vmagent.APIV1ImportPrometheus(t, []string{

dashboards/backupmanager.json

@@ -59,6 +59,19 @@
         },
         "type": "dashboard"
       },
+      {
+        "datasource": {
+          "type": "prometheus",
+          "uid": "$ds"
+        },
+        "enable": true,
+        "expr": "sum by(version) (\n    label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n    OR\n    vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n) \nunless \n(\n    sum by(version) (\n        label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n        OR\n        vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n    ) offset $__interval\n)",
+        "hide": true,
+        "iconColor": "dark-blue",
+        "name": "version",
+        "textFormat": "{{version}}",
+        "titleFormat": "Version change"
+      },
       {
         "datasource": {
           "type": "prometheus",

dashboards/victoriametrics-cluster.json

@@ -37,7 +37,7 @@
           "uid": "$ds"
         },
         "enable": true,
-        "expr": "sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"}) by(version) unless (sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"} offset $__interval) by(version))",
+        "expr": "sum by(version) (\n    label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n    OR\n    vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n) \nunless \n(\n    sum by(version) (\n        label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n        OR\n        vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n    ) offset $__interval\n)",
         "hide": true,
         "iconColor": "dark-blue",
         "name": "version change",

dashboards/victoriametrics.json

@@ -37,7 +37,7 @@
           "uid": "$ds"
         },
         "enable": true,
-        "expr": "sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"}) by(version) unless (sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"} offset $__interval) by(version))",
+        "expr": "sum by(version) (\n    label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n    OR\n    vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n) \nunless \n(\n    sum by(version) (\n        label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n        OR\n        vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n    ) offset $__interval\n)",
         "hide": true,
         "iconColor": "dark-blue",
         "name": "version",

dashboards/vm/backupmanager.json

@@ -60,6 +60,19 @@
         },
         "type": "dashboard"
       },
+      {
+        "datasource": {
+          "type": "victoriametrics-metrics-datasource",
+          "uid": "$ds"
+        },
+        "enable": true,
+        "expr": "sum by(version) (\n    label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n    OR\n    vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n) \nunless \n(\n    sum by(version) (\n        label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n        OR\n        vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n    ) offset $__interval\n)",
+        "hide": true,
+        "iconColor": "dark-blue",
+        "name": "version",
+        "textFormat": "{{version}}",
+        "titleFormat": "Version change"
+      },
       {
         "datasource": {
           "type": "victoriametrics-metrics-datasource",

dashboards/vm/victoriametrics-cluster.json

@@ -38,7 +38,7 @@
           "uid": "$ds"
         },
         "enable": true,
-        "expr": "sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"}) by(version) unless (sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"} offset $__interval) by(version))",
+        "expr": "sum by(version) (\n    label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n    OR\n    vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n) \nunless \n(\n    sum by(version) (\n        label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n        OR\n        vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n    ) offset $__interval\n)",
         "hide": true,
         "iconColor": "dark-blue",
         "name": "version change",

dashboards/vm/victoriametrics.json

@@ -38,7 +38,7 @@
           "uid": "$ds"
         },
         "enable": true,
-        "expr": "sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"}) by(version) unless (sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"} offset $__interval) by(version))",
+        "expr": "sum by(version) (\n    label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n    OR\n    vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n) \nunless \n(\n    sum by(version) (\n        label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n        OR\n        vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n    ) offset $__interval\n)",
         "hide": true,
         "iconColor": "dark-blue",
         "name": "version",

dashboards/vm/vmagent.json

@@ -26,11 +26,11 @@
           "uid": "$ds"
         },
         "enable": true,
-        "expr": "sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"}) by(short_version) unless (sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"} offset $__interval) by(short_version))",
+        "expr": "sum by(version) (\n    label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n    OR\n    vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n) \nunless \n(\n    sum by(version) (\n        label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n        OR\n        vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n    ) offset $__interval\n)",
         "hide": true,
         "iconColor": "dark-blue",
         "name": "version",
-        "textFormat": "{{short_version}}",
+        "textFormat": "{{version}}",
         "titleFormat": "Version change"
       },
       {

dashboards/vm/vmalert.json

@@ -26,11 +26,11 @@
           "uid": "$ds"
         },
         "enable": true,
-        "expr": "sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"}) by(short_version) unless (sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"} offset $__interval) by(short_version))",
+        "expr": "sum by(version) (\n    label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n    OR\n    vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n) \nunless \n(\n    sum by(version) (\n        label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n        OR\n        vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n    ) offset $__interval\n)",
         "hide": true,
         "iconColor": "dark-blue",
         "name": "version",
-        "textFormat": "{{short_version}}",
+        "textFormat": "{{version}}",
         "titleFormat": "Version change"
       },
       {

dashboards/vm/vmauth.json

@@ -54,6 +54,19 @@
         },
         "type": "dashboard"
       },
+      {
+        "datasource": {
+          "type": "victoriametrics-metrics-datasource",
+          "uid": "$ds"
+        },
+        "enable": true,
+        "expr": "sum by(version) (\n    label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n    OR\n    vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n) \nunless \n(\n    sum by(version) (\n        label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n        OR\n        vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n    ) offset $__interval\n)",
+        "hide": true,
+        "iconColor": "dark-blue",
+        "name": "version",
+        "textFormat": "{{version}}",
+        "titleFormat": "Version change"
+      },
       {
         "datasource": {
           "type": "victoriametrics-metrics-datasource",

dashboards/vmagent.json

@@ -25,11 +25,11 @@
           "uid": "$ds"
         },
         "enable": true,
-        "expr": "sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"}) by(short_version) unless (sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"} offset $__interval) by(short_version))",
+        "expr": "sum by(version) (\n    label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n    OR\n    vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n) \nunless \n(\n    sum by(version) (\n        label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n        OR\n        vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n    ) offset $__interval\n)",
         "hide": true,
         "iconColor": "dark-blue",
         "name": "version",
-        "textFormat": "{{short_version}}",
+        "textFormat": "{{version}}",
         "titleFormat": "Version change"
       },
       {

dashboards/vmalert.json

@@ -25,11 +25,11 @@
           "uid": "$ds"
         },
         "enable": true,
-        "expr": "sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"}) by(short_version) unless (sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"} offset $__interval) by(short_version))",
+        "expr": "sum by(version) (\n    label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n    OR\n    vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n) \nunless \n(\n    sum by(version) (\n        label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n        OR\n        vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n    ) offset $__interval\n)",
         "hide": true,
         "iconColor": "dark-blue",
         "name": "version",
-        "textFormat": "{{short_version}}",
+        "textFormat": "{{version}}",
         "titleFormat": "Version change"
       },
       {

dashboards/vmauth.json

@@ -53,6 +53,19 @@
         },
         "type": "dashboard"
       },
+      {
+        "datasource": {
+          "type": "prometheus",
+          "uid": "$ds"
+        },
+        "enable": true,
+        "expr": "sum by(version) (\n    label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n    OR\n    vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n) \nunless \n(\n    sum by(version) (\n        label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n        OR\n        vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n    ) offset $__interval\n)",
+        "hide": true,
+        "iconColor": "dark-blue",
+        "name": "version",
+        "textFormat": "{{version}}",
+        "titleFormat": "Version change"
+      },
       {
         "datasource": {
           "type": "prometheus",

deployment/docker/Makefile

@@ -7,7 +7,7 @@ ROOT_IMAGE ?= alpine:3.24.1
 ROOT_IMAGE_SCRATCH ?= scratch
 CERTS_IMAGE := alpine:3.24.1
 
-GO_BUILDER_IMAGE := golang:1.26.5
+GO_BUILDER_IMAGE := golang:1.26.6
 
 BUILDER_IMAGE := local/builder:2.0.0-$(shell echo $(GO_BUILDER_IMAGE) | tr :/ __)-1
 BASE_IMAGE := local/base:1.1.4-$(shell echo $(ROOT_IMAGE) | tr :/ __)-$(shell echo $(CERTS_IMAGE) | tr :/ __)

deployment/docker/compose-vm-cluster.yml

@@ -3,7 +3,7 @@ services:
   #  It scrapes targets defined in --promscrape.config
   #  And forward them to --remoteWrite.url
   vmagent:
-    image: victoriametrics/vmagent:v1.149.0
+    image: victoriametrics/vmagent:v1.150.0
     depends_on:
       - "vmauth"
     ports:
@@ -42,14 +42,14 @@ services:
   # vmstorage shards. Each shard receives 1/N of all metrics sent to vminserts,
   # where N is number of vmstorages (2 in this case).
   vmstorage-1:
-    image: victoriametrics/vmstorage:v1.149.0-cluster
+    image: victoriametrics/vmstorage:v1.150.0-cluster
     volumes:
       - strgdata-1:/storage
     command:
       - "--storageDataPath=/storage"
     restart: always
   vmstorage-2:
-    image: victoriametrics/vmstorage:v1.149.0-cluster
+    image: victoriametrics/vmstorage:v1.150.0-cluster
     volumes:
       - strgdata-2:/storage
     command:
@@ -59,7 +59,7 @@ services:
   # vminsert is ingestion frontend. It receives metrics pushed by vmagent,
   # pre-process them and distributes across configured vmstorage shards.
   vminsert-1:
-    image: victoriametrics/vminsert:v1.149.0-cluster
+    image: victoriametrics/vminsert:v1.150.0-cluster
     depends_on:
       - "vmstorage-1"
       - "vmstorage-2"
@@ -68,7 +68,7 @@ services:
       - "--storageNode=vmstorage-2:8400"
     restart: always
   vminsert-2:
-    image: victoriametrics/vminsert:v1.149.0-cluster
+    image: victoriametrics/vminsert:v1.150.0-cluster
     depends_on:
       - "vmstorage-1"
       - "vmstorage-2"
@@ -80,7 +80,7 @@ services:
   # vmselect is a query fronted. It serves read queries in MetricsQL or PromQL.
   # vmselect collects results from configured `--storageNode` shards.
   vmselect-1:
-    image: victoriametrics/vmselect:v1.149.0-cluster
+    image: victoriametrics/vmselect:v1.150.0-cluster
     depends_on:
       - "vmstorage-1"
       - "vmstorage-2"
@@ -90,7 +90,7 @@ services:
       - "--vmalert.proxyURL=http://vmalert:8880"
     restart: always
   vmselect-2:
-    image: victoriametrics/vmselect:v1.149.0-cluster
+    image: victoriametrics/vmselect:v1.150.0-cluster
     depends_on:
       - "vmstorage-1"
       - "vmstorage-2"
@@ -105,7 +105,7 @@ services:
   # read requests from Grafana, vmui, vmalert among vmselects.
   # It can be used as an authentication proxy.
   vmauth:
-    image: victoriametrics/vmauth:v1.149.0
+    image: victoriametrics/vmauth:v1.150.0
     depends_on:
       - "vmselect-1"
       - "vmselect-2"
@@ -119,7 +119,7 @@ services:
 
   # vmalert executes alerting and recording rules
   vmalert:
-    image: victoriametrics/vmalert:v1.149.0
+    image: victoriametrics/vmalert:v1.150.0
     depends_on:
       - "vmauth"
     ports:

deployment/docker/compose-vm-single.yml

@@ -3,7 +3,7 @@ services:
   #  It scrapes targets defined in --promscrape.config
   #  And forward them to --remoteWrite.url
   vmagent:
-    image: victoriametrics/vmagent:v1.149.0
+    image: victoriametrics/vmagent:v1.150.0
     depends_on:
       - "victoriametrics"
     ports:
@@ -18,7 +18,7 @@ services:
   # VictoriaMetrics instance, a single process responsible for
   # storing metrics and serve read requests.
   victoriametrics:
-    image: victoriametrics/victoria-metrics:v1.149.0
+    image: victoriametrics/victoria-metrics:v1.150.0
     ports:
       - 8428:8428
       - 8089:8089
@@ -59,7 +59,7 @@ services:
 
   # vmalert executes alerting and recording rules
   vmalert:
-    image: victoriametrics/vmalert:v1.149.0
+    image: victoriametrics/vmalert:v1.150.0
     depends_on:
       - "victoriametrics"
       - "alertmanager"

deployment/docker/rules/alerts-cluster.yml

@@ -75,7 +75,7 @@ groups:
             Consider to limit the ingestion rate, decrease retention or scale the disk space if possible."
 
       - alert: RequestErrorsToAPI
-        expr: increase(vm_http_request_errors_total{path=~".+", path!="*"}[5m]) > 0
+        expr: increase(vm_http_request_errors_total[5m]) > 0
         for: 15m
         labels:
           severity: warning
@@ -83,24 +83,8 @@ groups:
         annotations:
           dashboard: "{{ $externalURL }}/d/oS7Bi_0Wz?viewPanel=52&var-instance={{ $labels.instance }}"
           summary: "Too many errors served for {{ $labels.job }} path {{ $labels.path }} (instance {{ $labels.instance }})"
-          description: |
-              Requests to path {{ $labels.path }} are receiving errors.
-              Please verify if clients are sending correct requests.
-
-      # Auth errors and unknown paths should be handled by a different alert
-      # See https://github.com/VictoriaMetrics/VictoriaMetrics/blob/fdd9a221df835daa378ae2e6c9f12e4e3be79c76/lib/httpserver/httpserver.go#L589-L591
-      - alert: RequestErrorsToUnknownPaths
-        expr: sum(increase(vm_http_request_errors_total{path=~"^(\*|)$"}[5m])) by(job, instance, reason) > 0
-        for: 15m
-        labels:
-          severity: warning
-          show_at: dashboard
-        annotations:
-          dashboard: "{{ $externalURL }}/d/oS7Bi_0Wz?viewPanel=52&var-instance={{ $labels.instance }}"
-          summary: "Too many errors served for {{ $labels.job }} with reason {{ $labels.reason }} (instance {{ $labels.instance }})"
-          description: |
-              Requests are failing with reason {{ $labels.reason }}.
-              Please verify if clients are sending correct requests.
+          description: "Requests to path {{ $labels.path }} are receiving errors.
+            Please verify if clients are sending correct requests."
 
       - alert: RPCErrors
         expr: |

deployment/docker/rules/alerts-health.yml

@@ -16,6 +16,19 @@ groups:
             Job {{ $labels.job }} (instance {{ $labels.instance }}) has restarted more than twice in the last 15 minutes.
             It might be crashlooping.
 
+      - alert: UncleanShutdown
+        expr: vm_app_prev_shutdown_unclean == 1 and time() - vm_app_start_timestamp < 600
+        labels:
+          severity: warning
+        annotations:
+          summary: "{{ $labels.job }} on instance {{ $labels.instance }} started after an unclean shutdown"
+          description: |
+            The previous process run didn't shut down cleanly. Check the logs for OOM, SIGKILL,
+            a host failure, or another unexpected termination. In Kubernetes, a pod may be forcefully
+            killed with SIGKILL if the shutdown takes longer than terminationGracePeriodSeconds.
+            This alert stops firing 10 minutes after startup.
+            See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8443 for more details.
+
       - alert: ServiceDown
         expr: up{job=~".*(victoriametrics|vmselect|vminsert|vmstorage|vmagent|vmalert|vmsingle|vmalertmanager|vmauth).*"} == 0
         for: 2m

deployment/docker/rules/alerts-single-node.yml

@@ -75,7 +75,7 @@ groups:
             Consider to limit the ingestion rate, decrease retention or scale the disk space if possible."
 
       - alert: RequestErrorsToAPI
-        expr: increase(vm_http_request_errors_total{path=~".+"}[5m]) > 0
+        expr: increase(vm_http_request_errors_total[5m]) > 0
         for: 15m
         labels:
           severity: warning
@@ -85,21 +85,6 @@ groups:
           description: "Requests to path {{ $labels.path }} are receiving errors.
             Please verify if clients are sending correct requests."
 
-      # Auth errors and unknown paths should be handled by a different alert
-      # See https://github.com/VictoriaMetrics/VictoriaMetrics/blob/fdd9a221df835daa378ae2e6c9f12e4e3be79c76/lib/httpserver/httpserver.go#L589-L591
-      - alert: RequestErrorsToUnknownPaths
-        expr: sum(increase(vm_http_request_errors_total{path=~"^(\*|)$"}[5m])) by(job, instance, reason) > 0
-        for: 15m
-        labels:
-          severity: warning
-          show_at: dashboard
-        annotations:
-          dashboard: "{{ $externalURL }}/d/oS7Bi_0Wz?viewPanel=52&var-instance={{ $labels.instance }}"
-          summary: "Too many errors served for {{ $labels.job }} with reason {{ $labels.reason }} (instance {{ $labels.instance }})"
-          description: |
-            Requests are failing with reason {{ $labels.reason }}.
-            Please verify if clients are sending correct requests.
-
       - alert: TooHighChurnRate
         expr: |
           (

deployment/docker/vmanomaly/vmanomaly-integration/compose.yml

@@ -1,6 +1,6 @@
 services:
   vmagent:
-    image: victoriametrics/vmagent:v1.149.0
+    image: victoriametrics/vmagent:v1.150.0
     depends_on:
       - "victoriametrics"
     ports:
@@ -14,7 +14,7 @@ services:
     restart: always
 
   victoriametrics:
-    image: victoriametrics/victoria-metrics:v1.149.0
+    image: victoriametrics/victoria-metrics:v1.150.0
     ports:
       - 8428:8428
     volumes:
@@ -40,7 +40,7 @@ services:
     restart: always
 
   vmalert:
-    image: victoriametrics/vmalert:v1.149.0
+    image: victoriametrics/vmalert:v1.150.0
     depends_on:
       - "victoriametrics"
     ports:
@@ -59,7 +59,7 @@ services:
       - '--external.alert.source=explore?orgId=1&left=["now-1h","now","VictoriaMetrics",{"expr": },{"mode":"Metrics"},{"ui":[true,true,true,"none"]}]'
     restart: always
   vmanomaly:
-    image: victoriametrics/vmanomaly:v1.30.1
+    image: victoriametrics/vmanomaly:v1.30.2
     depends_on:
       - "victoriametrics"
     ports:

deployment/docker/vmanomaly/vmanomaly-integration/vmanomaly_config.yml

@@ -1,7 +1,7 @@
 schedulers:
   periodic:
     infer_every: "1m"
-    fit_every: "100w" # the online model keeps learning during inference
+    fit_every: "1000d" # bootstrap-only schedule; use a finite cadence if accumulated state must be reset
     fit_window: "2w"
 
 models:

docs/Makefile

@@ -90,12 +90,9 @@ endif
 	sed -i 's/\t/ /g' docs/victoriametrics/victoria_metrics_common_flags.md
 	sed -i 's/\t/ /g' docs/victoriametrics/victoria_metrics_enterprise_flags.md
 
-	# adjust flags with dynamic default values
-	# remove after https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9680 implemented
-	sed -i '/The maximum number of concurrent insert requests/ s/(default [0-9]\+)/(default 2*cgroup.AvailableCPUs())/' docs/victoriametrics/victoria_metrics_common_flags.md
-	sed -i '/The maximum number of concurrent search requests\./ s/(default [0-9]\+)/(default vmselect.getDefaultMaxConcurrentRequests())/' docs/victoriametrics/victoria_metrics_common_flags.md
-	sed -i '/The maximum number of CPU cores a single query can use\./ s/(default [0-9]\+)/(default netstorage.defaultMaxWorkersPerQuery())/' docs/victoriametrics/victoria_metrics_common_flags.md
-	sed -i '/The maximum number of concurrent goroutines to work with files;/ s/(default [0-9]\+)/(default fsutil.getDefaultConcurrency())/' docs/victoriametrics/victoria_metrics_common_flags.md
+	# hide the machine-specific value of dynamic defaults, keeping the formula.
+	# the flagutil.New*WithDynamicDefault constructors print them as "(default <value> = <formula>)".
+	sed -i 's/(default [0-9]\+ = \(.*\))$$/(default \1)/' docs/victoriametrics/victoria_metrics_common_flags.md
 
 docs-update-vmauth-flags:
 ifndef TAG
@@ -119,9 +116,9 @@ endif
 	sed -i 's/\t/ /g' docs/victoriametrics/vmauth_common_flags.md
 	sed -i 's/\t/ /g' docs/victoriametrics/vmauth_enterprise_flags.md
 
-	# adjust flags with dynamic default values
-	# remove after https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9680 implemented
-	sed -i '/The maximum number of concurrent goroutines to work with files;/ s/(default [0-9]\+)/(default fsutil.getDefaultConcurrency())/' docs/victoriametrics/vmauth_common_flags.md
+	# hide the machine-specific value of dynamic defaults, keeping the formula.
+	# the flagutil.New*WithDynamicDefault constructors print them as "(default <value> = <formula>)".
+	sed -i 's/(default [0-9]\+ = \(.*\))$$/(default \1)/' docs/victoriametrics/vmauth_common_flags.md
 
 docs-update-vmagent-flags:
 ifndef TAG
@@ -145,11 +142,9 @@ endif
 	sed -i 's/\t/ /g' docs/victoriametrics/vmagent_common_flags.md
 	sed -i 's/\t/ /g' docs/victoriametrics/vmagent_enterprise_flags.md
 
-	# adjust flags with dynamic default values
-	# remove after https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9680 implemented
-	sed -i '/The maximum number of concurrent insert requests/ s/(default [0-9]\+)/(default 2*cgroup.AvailableCPUs())/' docs/victoriametrics/vmagent_common_flags.md
-	sed -i '/The number of concurrent queues to each -remoteWrite.url./ s/(default [0-9]\+)/(default 2*cgroup.AvailableCPUs())/' docs/victoriametrics/vmagent_common_flags.md
-	sed -i '/The maximum number of concurrent goroutines to work with files;/ s/(default [0-9]\+)/(default fsutil.getDefaultConcurrency())/' docs/victoriametrics/vmagent_common_flags.md
+	# hide the machine-specific value of dynamic defaults, keeping the formula.
+	# the flagutil.New*WithDynamicDefault constructors print them as "(default <value> = <formula>)".
+	sed -i 's/(default [0-9]\+ = \(.*\))$$/(default \1)/' docs/victoriametrics/vmagent_common_flags.md
 
 docs-update-vmalert-flags:
 ifndef TAG
@@ -173,10 +168,9 @@ endif
 	sed -i 's/\t/ /g' docs/victoriametrics/vmalert_common_flags.md
 	sed -i 's/\t/ /g' docs/victoriametrics/vmalert_enterprise_flags.md
 
-	# adjust flags with dynamic default values
-	# remove after https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9680 implemented
-	sed -i '/Defines number of writers for concurrent writing into remote write endpoint./ s/(default [0-9]\+)/(default 2*cgroup.AvailableCPUs())/' docs/victoriametrics/vmalert_common_flags.md
-	sed -i '/The maximum number of concurrent goroutines to work with files;/ s/(default [0-9]\+)/(default fsutil.getDefaultConcurrency())/' docs/victoriametrics/vmalert_common_flags.md
+	# hide the machine-specific value of dynamic defaults, keeping the formula.
+	# the flagutil.New*WithDynamicDefault constructors print them as "(default <value> = <formula>)".
+	sed -i 's/(default [0-9]\+ = \(.*\))$$/(default \1)/' docs/victoriametrics/vmalert_common_flags.md
 
 docs-update-vmselect-flags:
 ifndef TAG

docs/ai-tools/_index.md

@@ -1,5 +1,6 @@
 ---
 title: AI tools
+description: "MCP servers, skills, and AI assistant integrations for querying metrics, logs, and traces with natural language."
 weight: 61
 menu:
   docs:

docs/anomaly-detection/CHANGELOG.md

@@ -1,6 +1,7 @@
 ---
 weight: 7
 title: CHANGELOG
+description: "Release history for vmanomaly."
 menu:
   docs:
     identifier: "vmanomaly-changelog"
@@ -16,6 +17,23 @@ Please find the changelog for VictoriaMetrics Anomaly Detection below.
 
 {{% collapse name="2026" open=true %}}
 
+## v1.30.2
+Released: 2026-08-13
+
+- UI: Updated [vmanomaly UI](https://docs.victoriametrics.com/anomaly-detection/ui/) from [v1.8.1](https://docs.victoriametrics.com/anomaly-detection/ui/#v181) to [v1.8.2](https://docs.victoriametrics.com/anomaly-detection/ui/#v182), fixing tenant discovery and switching for multitenant VictoriaMetrics datasources.
+
+- FEATURE: Added **query**-level [`data_range`, `detection_direction`, `min_dev_from_expected`, and `min_rel_dev_from_expected`](https://docs.victoriametrics.com/anomaly-detection/components/reader/#per-query-parameters). Model-level placement is deprecated but remains a compatible fallback.
+
+- IMPROVEMENT: Added [`reader.workers`](https://docs.victoriametrics.com/anomaly-detection/components/reader/#config-parameters) to cap concurrent datasource requests and disk-streamed query chunks; `0` selects an automatic bound.
+
+- IMPROVEMENT: Added [`settings.native_threads_per_worker`](https://docs.victoriametrics.com/anomaly-detection/components/settings/#parallelization) to reduce [native-thread oversubscription](https://scikit-learn.org/stable/computing/parallelism.html#oversubscription-spawning-too-many-threads), throttling risk, fit latency, and memory. For example, with 16 CPUs/workers, [Temporal Envelope](https://docs.victoriametrics.com/anomaly-detection/components/models/#temporal-envelope) fit time fell 70.6% for 1,000 univariate models and 11.5% for 100 x 10-channel grouped models; inference was unchanged.
+
+- IMPROVEMENT: Removed temporary fit-data generations after all dependent models finish and commit, while safely retaining failed or overlapping generations.
+
+- IMPROVEMENT: Reduced disk-backed grouped multivariate memory and fit latency without model or state migration. For example, 100 x 100-channel four-week fits cut peak PSS/fit time by 63%/56% for [Temporal Envelope](https://docs.victoriametrics.com/anomaly-detection/components/models/#temporal-envelope).
+
+- BUGFIX: Made [multivariate models](https://docs.victoriametrics.com/anomaly-detection/components/models/#multivariate-models) independent of input channel order when the fitted channel set matches; missing, extra, or duplicate channels remain rejected.
+
 ## v1.30.1
 Released: 2026-08-06
 

docs/anomaly-detection/FAQ.md

@@ -1,6 +1,7 @@
 ---
 weight: 6
 title: FAQ
+description: "Frequently asked questions about vmanomaly."
 menu:
   docs:
     identifier: "vmanomaly-faq"
@@ -33,7 +34,7 @@ Please see example graph illustrating this logic below:
 
 ![anomaly-score-calculation-example](vmanomaly-prophet-example.webp)
 
-> p.s. please note that additional post-processing logic might be applied to produced anomaly scores, if common arguments like [`min_dev_from_expected`](https://docs.victoriametrics.com/anomaly-detection/components/models/#minimal-deviation-from-expected) or [`detection_direction`](https://docs.victoriametrics.com/anomaly-detection/components/models/#detection-direction) are enabled for a particular model. Follow the links above for the explanations.
+> Additional post-processing logic may be applied to produced anomaly scores when query policies such as [`min_dev_from_expected`](https://docs.victoriametrics.com/anomaly-detection/components/models/#minimal-deviation-from-expected) or [`detection_direction`](https://docs.victoriametrics.com/anomaly-detection/components/models/#detection-direction) are configured. Follow the links for details.
 
 
 ## How does vmanomaly work?
@@ -135,7 +136,7 @@ Still not 100% sure what to use? We are [here to help](https://docs.victoriametr
 
 ## Incorporating domain knowledge
 
-Anomaly detection models can significantly improve when incorporating business-specific assumptions about the data and what constitutes an anomaly. `vmanomaly` supports various [business-side configuration parameters](https://docs.victoriametrics.com/anomaly-detection/components/models/#common-args) across all built-in models to **reduce [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-positive)** and **align model behavior with business needs**, for example:
+Anomaly detection models can significantly improve when incorporating business-specific assumptions about the data and what constitutes an anomaly. `vmanomaly` supports [business policies](https://docs.victoriametrics.com/anomaly-detection/components/models/#common-args) across built-in models to **reduce [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-positive)** and **align model behavior with business needs**, for example:
 
 - **Setting `detection_direction`** - use [`detection_direction`](https://docs.victoriametrics.com/anomaly-detection/components/models/#detection-direction) to specify whether anomalies occur **above or below expectations**:
   - Set to `above_expected` for metrics like error rates, where spikes indicate anomalies.
@@ -163,7 +164,7 @@ Then, the following config may be used to benefit from incorporating domain know
 schedulers:
   periodic_http:
     class: periodic
-    fit_every: 12w
+    fit_every: 1000d
     fit_window: 1w
     infer_every: 1m
   # other schedulers ...
@@ -172,18 +173,19 @@ reader:
   queries:
     percentage_4xx:
       expr: respective_metricsQL_expr
-      data_range: [0, 0.05]  # to automatically trigger anomaly score > 1 for error rates > 5%
+      data_range: [0, 0.05]  # query-level business policy from v1.30.2; error rates >5% trigger anomaly score >1
+      detection_direction: 'above_expected'  # query-level from v1.30.2; only spikes are anomalous
+      min_dev_from_expected: [0, 0.005]  # query-level from v1.30.2; ignore upward deviations below 0.5%
+      min_rel_dev_from_expected: [0, 10]  # query-level from v1.30.2; ignore upward deviations below 10%
       step: 1m
 models:
   # other models ...
   zscore:  # let it be online Z-score, for simplicity
     class: zscore_online  # online model update itself each infer call, resulting in resource-efficient setups
     z_threshold: 3.0
+    decay: 0.99  # give more weight to recent data while using the bootstrap-only fit schedule
     schedulers: ['periodic_http']
     queries: ['percentage_4xx']
-    detection_direction: 'above_expected'  # as interested only in spikes, drops are OK
-    min_dev_from_expected: [0, 0.005]  # <0.5% deviations vs expected values should be neglected, generating anomaly score == 0
-    min_rel_dev_from_expected: [0, 0.1]  # <10% relative deviations vs expected values should be neglected, generating anomaly score == 0
     # to align predictions to be within [0, 5%] interval, defined in reader.queries.percentage_4xx.data_range
     clip_predictions: True
     # specify output series produced by vmanomaly to be written to VictoriaMetrics in `writer`
@@ -229,7 +231,7 @@ models:
     schedulers: ['scheduler_alias']  # if omitted, all the defined schedulers will be attached
     queries: ['query_alias1']  # if omitted, all the defined queries will be attached
     # https://docs.victoriametrics.com/anomaly-detection/components/models/#provide-series
-    provide_series: ['anomaly_score']  
+    provide_series: ['anomaly_score']
   # ... other models
 
 reader:
@@ -255,6 +257,7 @@ Configuration above will produce N intervals of full length (`fit_window`=14d +
 
 `vmanomaly` can generate future forecasts with [Temporal Envelope](https://docs.victoriametrics.com/anomaly-detection/components/models/#temporal-envelope) {{% available_from "v1.30.0" anomaly %}}, the preferred online forecasting model. [ProphetModel](https://docs.victoriametrics.com/anomaly-detection/components/models/#prophet) {{% available_from "v1.25.3" anomaly %}} also supports forecasting for existing offline configurations. Forecasts help with capacity planning, resource allocation, or trend analysis when the underlying data is complex and exceeds what inline MetricsQL queries, including [predict_linear](https://docs.victoriametrics.com/victoriametrics/metricsql/#predict_linear), can handle.
 
+> [!WARNING]
 > However, please note that this mode should be used with care, as the model will produce `yhat_{h}` (and probably `yhat_lower_{h}`, and `yhat_upper_{h}`) time series **for each timeseries returned by input queries and for each forecasting horizon specified in `forecast_at` argument, which can lead to a significant increase in the number of active timeseries in VictoriaMetrics TSDB**.
 
 Here's an example of how to produce forecasts using `vmanomaly` and combine it with the regular model, e.g. to estimate daily outcomes for a disk usage metric:
@@ -264,12 +267,12 @@ Here's an example of how to produce forecasts using `vmanomaly` and combine it w
 schedulers:
   periodic_5m:  # this scheduler will be used to produce anomaly scores each 5 minutes using "regular" simple model
     class: 'periodic'
-    fit_every: '100w'
+    fit_every: '1000d'
     fit_window: '3d'
     infer_every: '5m'
   periodic_forecast:  # this scheduler will be used to produce forecasts each 24h using "daily" model
     class: 'periodic'
-    fit_every: '1000w'
+    fit_every: '1000d'
     fit_window: '730d' # to fit the model on 2 years of data to account for seasonality and holidays
     infer_every: '24h'
 # https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader
@@ -289,6 +292,7 @@ reader:
           1h
         )
       data_range: [0, 1]
+      detection_direction: 'above_expected'  # query-level from v1.30.2
       # step: '1m'  # default will be inherited from sampling_period
     disk_usage_perc_1d:
       expr: |
@@ -300,14 +304,15 @@ reader:
         )
       step: '1d'  # override default step to 1d, as we want to produce daily forecasts
       data_range: [0, 1]
+      detection_direction: 'above_expected'  # query-level from v1.30.2
 # https://docs.victoriametrics.com/anomaly-detection/components/models/
 models:
   quantile_5m:
     class: 'quantile_online'  # online model, which updates itself each infer call
     queries: ['disk_usage_perc_5m']
     schedulers: ['periodic_5m']
+    decay: 0.99  # give more weight to recent data while using the bootstrap-only fit schedule
     clip_predictions: True
-    detection_direction: 'above_expected'  # as we are interested in spikes in capacity planning
     quantiles: [0.25, 0.5, 0.75]  # to produce median and upper quartiles
     iqr_threshold: 2.0
 
@@ -315,8 +320,9 @@ models:
     class: 'temporal_envelope'
     queries: ['disk_usage_perc_1d']
     schedulers: ['periodic_forecast']
+    alpha: 0.005  # capture the changes faster if increased
+    loss_reactivity: 3  # allow new deviations to update the envelope
     clip_predictions: True
-    detection_direction: 'above_expected'  # as we are interested in spikes in capacity planning
     forecast_at: ['3d', '7d']  # this will produce forecasts for 3 and 7 days ahead
     provide_series: ['yhat', 'yhat_upper']  # to write forecasts back to VictoriaMetrics, omitting `yhat_lower` as it is not needed in this example
     seasonalities: [dow_smooth]
@@ -425,13 +431,15 @@ For information on migrating between different versions of `vmanomaly`, please r
 
 > {{% available_from "v1.24.0" anomaly %}} This feature is best used in conjunction with [stateful mode](https://docs.victoriametrics.com/anomaly-detection/components/settings/#state-restoration) to ensure that the model state is preserved across service restarts.
 
+> {{% available_from "v1.30.2" anomaly %}} Scheduler-managed fit data is **temporary**. It is removed after every dependent univariate or multivariate model completes fitting and commits its state, rather than being retained until the next `fit_every` cycle. Model dumps and state metadata remain available for restoration.
+
 Here's an example of how to set it up in docker-compose using volumes:
 ```yaml
 services:
   # ...
   vmanomaly:
     container_name: vmanomaly
-    image: victoriametrics/vmanomaly:v1.30.1
+    image: victoriametrics/vmanomaly:v1.30.2
     # ...
     restart: always
     volumes:
@@ -502,7 +510,7 @@ settings:
 schedulers:
   periodic:
     class: 'periodic'
-    fit_every: '180d'  # we need only initial fit to start
+    fit_every: '1000d'
     fit_window: '4h'  # reduced window, especially if the data doesn't have strong seasonality
     infer_every: '1m'  # the model will be updated during each infer call
   # other schedulers ...
@@ -510,7 +518,7 @@ models:
   zscore_example:
     class: 'zscore_online'
     min_n_samples_seen: 120  # i.e. minimal relevant seasonality or (initial) fit_window / sampling_period
-    decay: 0.999  # decay factor to control how fast the model adapts to new data, the lower, the faster it adapts
+    decay: 0.99  # decay factor to control how fast the model adapts to new data, the lower, the faster it adapts
     schedulers: ['periodic']
     # other model params ...
 # other config sections ...
@@ -524,11 +532,11 @@ As a result, switching from the offline Z-score model to the Online Z-score mode
 
 **New configuration**: 
 - `fit_window`: 4 hours
-- `fit_every`: 180 days ( >1 week)
+- `fit_every`: 1000 days ( >1 week)
 
 The old configuration would perform 168 (hours in a week) `fit` calls, each using 2 days (48 hours) of data, totaling 168 * 48 = 8064 hours of data for each timeseries returned.
 
-The new configuration performs only 1 `fit` call in 180 days, using 4 hours of data initially, totaling 4 hours of data, which is **magnitudes smaller**.
+The new configuration performs only 1 `fit` call in 1000 days, using 4 hours of data initially, totaling 4 hours of data, which is **magnitudes smaller**.
 
 P.s. `infer` data volume will remain the same for both models, so it does not affect the overall calculations.
 
@@ -556,9 +564,7 @@ models:
     temporal_envelope:
       class: temporal_envelope
       # other model args
-      queries: [
-        'sum_alerts',
-      ]
+      queries: ['sum_alerts']
 # other config sections
 ```
 
@@ -578,9 +584,7 @@ models:
     temporal_envelope:
       class: temporal_envelope
       # other model args
-      queries: [
-        'sum_alerts',
-      ]
+      queries: ['sum_alerts']
 # other config sections
 ```
 
@@ -598,10 +602,7 @@ models:
     temporal_envelope:
       class: temporal_envelope
       # other model args
-      queries: [
-        'sum_alerts_pending',
-        'sum_alerts_firing',
-      ]
+      queries: ['sum_alerts_pending', 'sum_alerts_firing']
 # other config sections
 ```
 
@@ -651,10 +652,12 @@ options:
                         Minimum level to log. Default: INFO
 ```
 
+For a side-by-side comparison of all split modes and their resulting sub-configurations, see [splitting strategies](https://docs.victoriametrics.com/anomaly-detection/scaling-vmanomaly/#splitting-strategies).
+
 Here’s an example of using the config splitter to divide configurations based on the `extra_filters` argument from the reader section:
 
 ```sh
-docker pull victoriametrics/vmanomaly:v1.30.1 && docker image tag victoriametrics/vmanomaly:v1.30.1 vmanomaly
+docker pull victoriametrics/vmanomaly:v1.30.2 && docker image tag victoriametrics/vmanomaly:v1.30.2 vmanomaly
 ```
 
 ```sh
@@ -687,10 +690,11 @@ reader:
   # ...
   queries:
     extra_big_query: metricsql_expression_returning_too_many_timeseries
-    extra_filters:
+    extra_filters: [
       # suppose you have a label `region` with values to deterministically define such subsets
-      - '{env="region_name_1"}'
+      '{env="region_name_1"}',
       # ...
+    ]
 ```
 
 ```yaml
@@ -700,10 +704,11 @@ reader:
   # ...
   queries:
     extra_big_query: metricsql_expression_returning_too_many_timeseries
-    extra_filters:
+    extra_filters: [
       # suppose you have a label `region` with values to deterministically define such subsets
-      - '{region="region_name_2"}'
+      '{region="region_name_2"}',
       # ...
+    ]
 ```
 
 ## Monitoring vmanomaly

docs/anomaly-detection/Migration.md

@@ -1,6 +1,7 @@
 ---
 weight: 5
 title: Migration
+description: "Migration guide to the latest vmanomaly version."
 menu:
   docs:
     identifier: "vmanomaly-migration"
@@ -45,7 +46,7 @@ There are 2 types of compatibility to consider when migrating in stateful mode:
 
 | Group start | Group end | Compatibility | Notes |
 |---------|--------- |------------|-------|
-| [v1.29.1](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1291) | [v1.30.1](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1301) | Fully Compatible | v1.30.0 adds new [Temporal Envelope](https://docs.victoriametrics.com/anomaly-detection/components/models/#temporal-envelope) model state without changing the compatibility of existing model and data artifacts. v1.30.1 remains compatible with v1.30.0 state and its compatible predecessors. |
+| [v1.29.1](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1291) | [v1.30.2](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1302) | Fully Compatible | v1.30.0 adds new [Temporal Envelope](https://docs.victoriametrics.com/anomaly-detection/components/models/#temporal-envelope) model state without changing the compatibility of existing model and data artifacts. v1.30.2 remains compatible with v1.30.1 state and its compatible predecessors; no persisted-state migration is required. |
 | [v1.28.7](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1287) | [v1.29.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1290) | Partially compatible* | Dumped models of class [prophet](https://docs.victoriametrics.com/anomaly-detection/components/models/#prophet) and [seasonal quantile](https://docs.victoriametrics.com/anomaly-detection/components/models/#online-seasonal-quantile) have problems with loading to [v1.29.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1290) due to dropped `pytz` library. **Upgrading directly from v1.28.7 to [v1.29.1](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1291) with a fix is suggested** |
 | [v1.26.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1262) | [v1.28.7](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1287) | Fully Compatible | [v1.28.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1280) introduced [rolling](https://docs.victoriametrics.com/anomaly-detection/components/models/#rolling-models) model class drop in favor of [online](https://docs.victoriametrics.com/anomaly-detection/components/models/#online-models) models (`rolling_quantile` and `std` models), however, it does not impact compatibility, as artifacts were not produced by default for rolling models. Also, offline `mad` and `zscore` models are redirecting to their respective online counterparts since [v1.28.4](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1284). |
 | [v1.25.3](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1253) | [v1.26.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1270) | Partially Compatible* | [v1.25.3](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1253) introduced `forecast_at` argument for base [univariate](https://docs.victoriametrics.com/anomaly-detection/components/models/#univariate-models) and `Prophet` [models](https://docs.victoriametrics.com/anomaly-detection/components/models/#prophet), however, itself remains backward-reversible from newer states like [v1.26.2](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1262), [v1.27.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1270). (All models except `isolation_forest_multivariate` class will be dropped) |

docs/anomaly-detection/Presets.md

@@ -1,6 +1,7 @@
 ---
 weight: 2
 title: Presets
+description: "Preconfigured anomaly detection configurations for widely-recognized metrics (e.g., node_exporter)"
 menu:
   docs:
     parent: "anomaly-detection"

docs/anomaly-detection/QuickStart.md

@@ -1,6 +1,7 @@
 ---
 weight: 1
 title: Quick Start
+description: "Get started with vmanomaly. Install, configure, and run anomaly detection."
 menu:
   docs:
     parent: "anomaly-detection"
@@ -137,7 +138,7 @@ Below are the steps to get `vmanomaly` up and running inside a Docker container:
 1. Pull Docker image:
 
 ```sh
-docker pull victoriametrics/vmanomaly:v1.30.1
+docker pull victoriametrics/vmanomaly:v1.30.2
 ```
 
 2. Create the license file with your license key.
@@ -157,7 +158,7 @@ docker run -it \
     -v ./license:/license \
     -v ./config.yaml:/config.yaml \
     -p 8490:8490 \
-    victoriametrics/vmanomaly:v1.30.1 \
+    victoriametrics/vmanomaly:v1.30.2 \
     /config.yaml \
     --licenseFile=/license \
     --loggerLevel=INFO \
@@ -174,7 +175,7 @@ docker run -it \
     -e VMANOMALY_DATA_DUMPS_DIR=/tmp/vmanomaly/data \
     -e VMANOMALY_MODEL_DUMPS_DIR=/tmp/vmanomaly/models \
     -p 8490:8490 \
-    victoriametrics/vmanomaly:v1.30.1 \
+    victoriametrics/vmanomaly:v1.30.2 \
     /config.yaml \
     --licenseFile=/license \
     --loggerLevel=INFO \
@@ -187,7 +188,7 @@ services:
   # ...
   vmanomaly:
     container_name: vmanomaly
-    image: victoriametrics/vmanomaly:v1.30.1
+    image: victoriametrics/vmanomaly:v1.30.2
     # ...
     restart: always
     volumes:
@@ -250,12 +251,13 @@ Before deploying, check the correctness of your configuration validate config fi
 
 ### Example
 
-Here is an example of a config file that runs the online [Temporal Envelope](https://docs.victoriametrics.com/anomaly-detection/components/models/#temporal-envelope) model on a CPU metric. The scheduler runs inference every five minutes and performs a full refit only every 100 weeks; between refits the model updates causally from each inference batch. The initial fit uses four weeks of data. The model produces `anomaly_score`, `yhat`, `yhat_lower`, and `yhat_upper` [series](https://docs.victoriametrics.com/anomaly-detection/components/models/#vmanomaly-output) for debugging, and its hour-of-day and day-of-week profiles follow the query timezone and daylight-saving-time changes.
+Here is an example of a config file that runs the online [Temporal Envelope](https://docs.victoriametrics.com/anomaly-detection/components/models/#temporal-envelope) model on a CPU metric. The scheduler runs inference every five minutes and uses the fit only for initial bootstrap; between fits the model updates causally from each inference batch. The initial fit uses four weeks of data. The model produces `anomaly_score`, `yhat`, `yhat_lower`, and `yhat_upper` [series](https://docs.victoriametrics.com/anomaly-detection/components/models/#vmanomaly-output) for debugging, and its hour-of-day and day-of-week profiles follow the query timezone and daylight-saving-time changes.
 
 ```yaml
 settings:
   # https://docs.victoriametrics.com/anomaly-detection/components/settings/
   n_workers: 2  # number of workers to run workload in parallel, set to 0 or negative number to use all available CPU cores
+  native_threads_per_worker: 0  # automatically divide container-aware CPU capacity across workers
   anomaly_score_outside_data_range: 5.0  # default anomaly score for anomalies outside expected data range
   restore_state: true  # restore state from previous run, available since v1.24.0
   # https://docs.victoriametrics.com/anomaly-detection/components/settings/#logger-levels
@@ -268,13 +270,12 @@ settings:
     model.online.temporal_envelope: WARNING
 
 schedulers:
-  100w_5m:
+  online_5m:
     # https://docs.victoriametrics.com/anomaly-detection/components/scheduler/#periodic-scheduler
     class: 'periodic'
     infer_every: '5m'
     scatter_infer_jobs: true
-    # Temporal Envelope learns online between full refits.
-    fit_every: '100w'
+    fit_every: '1000d'
     fit_window: '4w'
 
 models:
@@ -282,7 +283,7 @@ models:
   temporal_envelope_model:
     class: 'temporal_envelope'
     queries: ['cpu_user']
-    schedulers: ['100w_5m']
+    schedulers: ['online_5m']
     provide_series: ['anomaly_score', 'yhat', 'yhat_lower', 'yhat_upper']  # for debugging
     seasonalities: ['hod_smooth', 'dow_smooth']
     alpha: 0.005  # trend reactivity; try 0.0025-0.02
@@ -297,12 +298,15 @@ reader:
   tenant_id: '0:0'
   sampling_period: "5m"
   tz: 'UTC'  # set the IANA timezone that defines local calendar patterns, e.g. 'America/New_York'
+  workers: 0  # automatically choose bounded datasource concurrency
   series_processing_batch_size: 8  # number of time series to process together while preparing data for fit or infer stages
   queries:
     # define your queries with MetricsQL - https://docs.victoriametrics.com/victoriametrics/metricsql/
     cpu_user:
       expr: 'sum(rate(node_cpu_seconds_total{mode=~"user"}[10m])) by (container)'
-      max_datapoints_per_query: 15000  # to deal with longer queries hitting search.MaxPointsPerTimeseries
+      data_range: [0, 'inf']  # query-level business policy from v1.30.2
+      detection_direction: 'above_expected'  # query-level from v1.30.2; only spikes are anomalous
+      max_points_per_query: 15000  # to deal with longer queries hitting search.maxPointsPerTimeseries
     # other queries ...
 
 writer:

docs/anomaly-detection/Scaling-vmanomaly.md

@@ -1,6 +1,7 @@
 ---
 weight: 3
 title: Scaling vmanomaly
+description: "High availability and horizontal scaling for vmanomaly."
 menu:
   docs:
     identifier: "vmanomaly-scaling"
@@ -32,14 +33,15 @@ schedulers:
   periodic_1d:  # alias
     class: 'periodic' # scheduler class
     infer_every: "30s"
-    fit_every: "1h"
+    fit_every: "1000d"
     fit_window: "24h"
 
 # https://docs.victoriametrics.com/anomaly-detection/components/models/
 models:
   zscore:  # we can set up alias for model
-    class: 'zscore'  # model class
+    class: 'zscore_online'  # online model class
     z_threshold: 3.5
+    decay: 0.99  # give more weight to recent data while using the bootstrap-only fit schedule
     queries: ['cpu_seconds_total', 'host_network_receive_errors']
 
 # https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader
@@ -80,6 +82,7 @@ Additionally, a replication factor `R ≥ 1` ensures [high availability](#high-a
 
 {{% content "vmanomaly-sharding-ha-diagram.md" %}}
 
+> [!WARNING]
 > Please [refer to deployment options section](#deployment-options) for the examples (Docker, Docker Compose, Helm). To avoid duplicate metrics being reported from each vmanomaly service used in sharded mode, make sure that [deduplication](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#deduplication) is configured on vmsingle or vmselect and vmstorage for the VictoriaMetrics instance used in the [writer section of the configuration](https://docs.victoriametrics.com/anomaly-detection/components/writer/).
 
 Sharding configuration can be controlled by using the following environment variables:
@@ -87,7 +90,94 @@ Sharding configuration can be controlled by using the following environment vari
 - **`VMANOMALY_MEMBERS_COUNT`**: Defines the total number of shards (i.e., available nodes to distribute [sub-configurations](#sub-configuration) to). <br>Defaults to `1` for backward compatibility.
 - **`VMANOMALY_MEMBER_NUM`**: Specifies the shard index (`0` to `VMANOMALY_MEMBERS_COUNT - 1`), determining the subset of [sub-configurations](#sub-configuration) to run on a specific node. Defaults to `0`. Supports automatic **pod name discovery** in Kubernetes [StatefulSets](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) (e.g., if set to `vmanomaly-node-exporter-7`, shard `7` will be extracted).
 - **`VMANOMALY_REPLICATION_FACTOR`**: If `R > 1`, enables [high availability](#high-availability) by ensuring each [sub-configuration](#sub-configuration) is assigned to exactly `R` shards. Defaults to `1` (no replication).
-- **`VMANOMALY_SPLIT_BY`**: Defines the logical entity used to split the global config into [sub-configurations](#sub-configuration). Defaults to `complete`, which provides the most granular distribution (1 model per [sub-config](#sub-configuration), mapped to 1 query and attached to 1 scheduler) for balanced workloads.
+- **`VMANOMALY_SPLIT_BY`**: Defines the logical entity used to split the global config into [sub-configurations](#sub-configuration). The accepted values are `SCHEDULERS`, `MODELS`, `QUERIES`, `EXTRA_FILTERS`, and `COMPLETE` (case-insensitive). It defaults to `COMPLETE`, which usually provides the most granular and balanced distribution.
+
+The split strategies differ as follows:
+
+| `VMANOMALY_SPLIT_BY` | Unit of work in each sub-configuration | Recommended use |
+| --- | --- | --- |
+| `SCHEDULERS` | One scheduler and the workload attached to it | Separate workloads by fit and inference cadence. The number of sub-configurations is limited by the number of referenced schedulers. |
+| `MODELS` | One configured model alias with its attached schedulers and queries | Isolate computationally different models or distribute several models that process the same queries. |
+| `QUERIES` | One query for [univariate models](https://docs.victoriametrics.com/anomaly-detection/components/models/#univariate-models); the complete attached query set for each [multivariate model](https://docs.victoriametrics.com/anomaly-detection/components/models/#multivariate-models) | Distribute independent query workloads. Queries belonging to one multivariate model remain together because the model needs all channels. This option does not split the series returned by one query. |
+| `EXTRA_FILTERS` | One configured `reader.extra_filters` selector, with the full model/query/scheduler topology retained | Partition the series returned by large queries, for example by region, cluster, another stable label, or by [VictoriaMetrics tenant](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy-via-labels) using `vm_account_id` and `vm_project_id` selectors with the multitenant endpoint. The filters must already be defined in the global configuration. |
+| `COMPLETE` | One valid scheduler/model/query combination; [multivariate](https://docs.victoriametrics.com/anomaly-detection/components/models/#multivariate-models) query sets remain together | Obtain the finest general-purpose split and the default choice for balanced sharding. `reader.extra_filters` are intentionally not expanded by this strategy. |
+
+After the selected strategy creates the sub-configurations, they are assigned to members in deterministic round-robin order and then replicated according to `VMANOMALY_REPLICATION_FACTOR`.
+
+### Splitting strategies
+
+{{% collapse name="Configuration and resulting sub-configurations" %}}
+
+The following abbreviated global configuration contains two schedulers, two models, four queries, and two data partitions:
+
+```yaml
+schedulers:
+  fast:
+    class: periodic
+    infer_every: 1m
+    fit_every: 1000d
+    fit_window: 1d
+  seasonal:
+    class: periodic
+    infer_every: 5m
+    fit_every: 1000d
+    fit_window: 2w
+
+models:
+  cpu_zscore:
+    class: zscore_online
+    schedulers: [fast]
+    queries: [cpu, error_rate]
+    decay: 0.99
+  gpu_envelope:
+    class: temporal_envelope_multivariate
+    schedulers: [seasonal]
+    queries: [temperature, power]
+    seasonalities: [hod_smooth, dow_smooth]
+
+reader:
+  class: vm
+  datasource_url: http://victoriametrics:8428/
+  sampling_period: 1m
+  queries:
+    cpu:
+      expr: avg(rate(node_cpu_seconds_total[5m])) by (instance)
+    error_rate:
+      expr: rate(application_errors_total[5m])
+    temperature:
+      expr: avg(gpu_temperature_celsius) by (gpu)
+    power:
+      expr: avg(gpu_power_watts) by (gpu)
+  extra_filters: ['{region="us-east"}', '{region="eu-west"}']
+
+writer:
+  class: vm
+  datasource_url: http://victoriametrics:8428/
+```
+
+For this configuration, each strategy produces the following logical units before they are assigned to shards:
+
+| Value | Resulting sub-configurations |
+| --- | --- |
+| `SCHEDULERS` | `fast`; `seasonal` |
+| `MODELS` | `cpu_zscore`; `gpu_envelope` |
+| `QUERIES` | `cpu`; `error_rate`; the multivariate set `power,temperature` |
+| `EXTRA_FILTERS` | `{region="us-east"}`; `{region="eu-west"}`; each retains all schedulers, models, and queries, while the query context is restricted by its selector |
+| `COMPLETE` | `fast:cpu_zscore:cpu`; `fast:cpu_zscore:error_rate`; `seasonal:gpu_envelope:power,temperature` |
+
+For example, choose the query split with:
+
+```yaml
+environment:
+  VMANOMALY_MEMBERS_COUNT: 3
+  VMANOMALY_MEMBER_NUM: 0
+  VMANOMALY_REPLICATION_FACTOR: 1
+  VMANOMALY_SPLIT_BY: QUERIES
+```
+
+To partition the timeseries returned by the same large query instead, define non-overlapping selectors in `reader.extra_filters` and use `VMANOMALY_SPLIT_BY: EXTRA_FILTERS`. Each generated sub-configuration keeps one selector, for example `{region="us-east"}` or `{region="eu-west"}`.
+
+{{% /collapse %}}
 
 ---
 
@@ -130,6 +220,7 @@ When `VMANOMALY_REPLICATION_FACTOR` > 1, each [sub-config](#sub-configuration) `
 
 {{% content "vmanomaly-sharding-ha-diagram.md" %}}
 
+> [!WARNING]
 > Please [refer to deployment options section](#deployment-options) for the examples (Docker, Docker Compose, Helm). To avoid duplicate metrics being reported from each vmanomaly service used in sharded mode, make sure that [deduplication](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#deduplication) is configured on vmsingle or vmselect and vmstorage for the VictoriaMetrics instance used in the [writer section of the configuration](https://docs.victoriametrics.com/anomaly-detection/components/writer/).
 
 ### Example
@@ -198,7 +289,11 @@ services:
     user: "1000:1000"
     restart: always
     healthcheck:
-      test: ["CMD", "curl", "-f", "http://127.0.0.1:8490/health"]
+      test:
+        - "CMD"
+        - "curl"
+        - "-f"
+        - "http://127.0.0.1:8490/health"
       interval: 30s
       timeout: 10s
       retries: 5
@@ -218,7 +313,11 @@ services:
     user: "1000:1000"
     restart: always
     healthcheck:
-      test: ["CMD", "curl", "-f", "http://127.0.0.1:8490/health"]
+      test:
+        - "CMD"
+        - "curl"
+        - "-f"
+        - "http://127.0.0.1:8490/health"
       interval: 30s
       timeout: 10s
       retries: 5

docs/anomaly-detection/Self-monitoring.md

@@ -1,6 +1,7 @@
 ---
 weight: 4
 title: Self-monitoring
+description: "Track vmanomaly health and operational performance."
 menu:
   docs:
     identifier: "vmanomaly-self-monitoring"

docs/anomaly-detection/UI.md

@@ -1,6 +1,7 @@
 ---
 weight: 2
 title: UI
+description: "Built-in vmui-like UI for exploring anomaly detection results."
 menu:
   docs:
     parent: "anomaly-detection"
@@ -137,13 +138,13 @@ users:
     password: '<password>'
     url_map:
       - src_hosts:
-        - "metrics.local.some-domain.net"
+          - "metrics.local.some-domain.net"
         url_prefix: "http://victoriametrics:8428"
       - src_hosts:
-        - "vl.local.some-domain.net"
+          - "vl.local.some-domain.net"
         url_prefix: "http://victorialogs:9428"
       - src_hosts:
-        - "vmanomaly.local.some-domain.net"
+          - "vmanomaly.local.some-domain.net"
         url_prefix: "http://vmanomaly:8490"
         keep_original_host: true
 ```
@@ -316,7 +317,7 @@ docker run -it --rm \
   -e VMANOMALY_MCP_SERVER_URL=http://mcp-vmanomaly:8081/mcp \
   -p 8080:8080 \
   -p 8490:8490 \
-  victoriametrics/vmanomaly:v1.30.1 \
+  victoriametrics/vmanomaly:v1.30.2 \
   vmanomaly_config.yaml
 ```
 
@@ -645,6 +646,13 @@ If the **results** look good and the **model configuration should be deployed in
 
 {{% collapse name="Release history" %}}
 
+### v1.8.2
+Released: 2026-08-13
+
+vmanomaly version: [v1.30.2](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1302)
+
+- BUGFIX: Fixed tenant discovery for VictoriaMetrics datasource URLs containing `/select/multitenant/prometheus`. The UI now loads available numeric tenants from `/admin/tenants` and can switch the datasource URL from `multitenant` to the selected tenant.
+
 ### v1.8.1
 Released: 2026-08-06
 

docs/anomaly-detection/_index.md

@@ -1,5 +1,6 @@
 ---
 title: Anomaly Detection
+description: "Use vmanomaly to detect anomalies in metrics and logs. Configure models, run inference, monitor the service, and connect results to alerts and dashboards."
 weight: 50
 menu:
   docs:

docs/anomaly-detection/components/README.md

@@ -37,6 +37,7 @@ The following minimal configuration demonstrates current many-to-many model, que
 ```yaml
 settings:
   n_workers: 4  # number of workers to run models in parallel
+  native_threads_per_worker: 0  # automatically divide container-aware CPU capacity across workers
   anomaly_score_outside_data_range: 5.0  # default anomaly score for anomalies outside expected data range
   restore_state: True  # restore state from previous run, if available
   retention:  # how long to keep stale models on disk/in memory
@@ -50,15 +51,15 @@ schedulers:
     class: 'periodic' # scheduler class
     infer_every: "30s"  # how often to produce anomaly scores for new data
     scatter_infer_jobs: true  # distribute infer jobs evenly across the infer interval to reduce synchronized bursts
-    fit_every: "365d"  # how often to re-fit the models, for online models used effectively once, then they are updated with new data and won't require re-fit
+    fit_every: "1000d"  # bootstrap-only schedule; use a finite cadence if accumulated state must be reset
     fit_window: "3d"  # how much historical data to use for fit stage
-    start_from: "00:00"  # align the annual fit schedule to midnight in the configured timezone
+    start_from: "00:00"  # align the bootstrap fit to midnight in the configured timezone
     tz: "Europe/Kyiv"  # timezone to use for start_from
   periodic_online_weekly:
     class: 'periodic'
     infer_every: "15m"
     scatter_infer_jobs: true
-    fit_every: "365d"  # online state continues adapting between infrequent full re-fits
+    fit_every: "1000d"  # bootstrap-only schedule; use a finite cadence if accumulated state must be reset
     fit_window: "14d"
     # if no start_from is specified, jobs will start immediately after service starts
 
@@ -72,17 +73,15 @@ models:
     provide_series: ['anomaly_score', 'y', 'yhat', 'yhat_upper']  # what series to produce as output of the model
     queries: ['host_network_receive_errors']  # what queries to run particular model on
     schedulers: ['periodic_online']  # will be fit once, used for infer every 30s
-    min_dev_from_expected: 0.0  # turned off. if |y - yhat| < min_dev_from_expected, anomaly score will be 0
-    detection_direction: 'above_expected' # detect anomalies only when y > yhat, "peaks"
     clip_predictions: True  # clip predictions to expected data range, i.e. [0, inf] for this query `host_network_receive_errors
   envelope_weekly: # we can set up alias for model
     class: 'temporal_envelope'
+    alpha: 0.005  # adapt the trend while using the bootstrap-only fit schedule
+    loss_reactivity: 3  # allow new deviations to update the envelope
     provide_series: ['anomaly_score', 'y', 'yhat', 'yhat_lower', 'yhat_upper']
     queries: ['cpu_seconds_total']
     schedulers: ['periodic_online_weekly']  # fit on two weekly cycles, then update online every 15m
-    min_dev_from_expected: [0.01, 0.01]  # minimum deviation from expected value to be even considered as anomaly
     anomaly_score_outside_data_range: 1.5  # override default anomaly score outside expected data range
-    detection_direction: 'above_expected'
     clip_predictions: True  # clip predictions to expected data range, i.e. [0, inf] for this query `cpu_seconds_total`
     seasonalities: ['hod_smooth', 'dow_smooth']
 
@@ -93,6 +92,7 @@ reader:
   datasource_url: "https://play.victoriametrics.com/"
   tenant_id: "0:0"
   sampling_period: "30s"  # what data resolution to fetch from VictoriaMetrics' /query_range endpoint
+  workers: 0  # automatically choose bounded datasource concurrency
   latency_offset: '1ms'
   query_from_last_seen_timestamp: False
   tz: "UTC"  # timezone to use for queries without explicit timezone
@@ -101,11 +101,15 @@ reader:
     cpu_seconds_total:
       expr: 'avg(rate(node_cpu_seconds_total[5m])) by (mode)' 
       # step: '30s'  # if not set, will be equal to reader-level sampling_period
-      data_range: [0, 'inf']  # expected value range, anomaly_score = anomaly_score_outside_data_range if y (real value) is outside
+      data_range: [0, 'inf']  # query-level business policy from v1.30.2
+      detection_direction: 'above_expected'  # query-level from v1.30.2; detect spikes only
+      min_dev_from_expected: [0.01, 0.01]  # query-level from v1.30.2
     host_network_receive_errors:
       expr: 'rate(node_network_receive_errs_total[3m]) / rate(node_network_receive_packets_total[3m])'
       step: '15m'  # here we override per-query `sampling_period` to request way less data from VM TSDB
-      data_range: [0, 'inf']
+      data_range: [0, 'inf']  # query-level business policy from v1.30.2
+      detection_direction: 'above_expected'  # query-level from v1.30.2; detect spikes only
+      min_dev_from_expected: 0.0  # query-level from v1.30.2; absolute-deviation filtering is disabled
 
 # where to write data to
 # https://docs.victoriametrics.com/anomaly-detection/components/writer/
@@ -146,7 +150,7 @@ server:
 
 {{% available_from "v1.25.0" anomaly %}} The service supports hot reload of configuration files, applying changes without an explicit restart. Enable it with the `--watch` [CLI argument](https://docs.victoriametrics.com/anomaly-detection/quickstart/#command-line-arguments). The `vmanomaly_config_reload_enabled` [self-monitoring metric](https://docs.victoriametrics.com/anomaly-detection/components/monitoring/#startup-metrics) is `1` when hot reload is enabled and `0` otherwise.
 
-> [!NOTE]
+> [!WARNING]
 > {{% deprecated_from "v1.29.5" anomaly %}} File system event-based hot reload has been deprecated in favor of content-based polling with configurable `-configCheckInterval` due to reliability issues with Kubernetes ConfigMap symlink rotations and other filesystems where event delivery can be inconsistent. If you were using file system event-based hot reload, please switch to content-based polling by enabling `--watch` flag and configuring `-configCheckInterval` as needed.
 
 ### How it works
@@ -173,7 +177,7 @@ schedulers:
   periodic:
     class: 'periodic'
     infer_every: "30s"
-    fit_every: "365d"
+    fit_every: "1000d"  # bootstrap-only schedule; use a finite cadence if accumulated state must be reset
     fit_window: "24h"
 
 reader:

docs/anomaly-detection/components/_index.md

@@ -1,5 +1,6 @@
 ---
 title: Components
+description: "Architecture overview. Models, reader, writer, scheduler, monitoring, settings, server."
 weight: 3
 menu:
   docs:

docs/anomaly-detection/components/models.md

@@ -1,5 +1,6 @@
 ---
 title: Models
+description: "Model types and configuration. Built-in and custom anomaly detection models."
 weight: 1
 menu:
   docs:
@@ -65,6 +66,9 @@ models:
 
 Common arguments supported by every model were introduced in [v1.10.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1100).
 
+> [!WARNING]
+> Configuring `data_range`, `detection_direction`, `min_dev_from_expected`, or `min_rel_dev_from_expected` at model level is deprecated {{% deprecated_from "v1.30.2" anomaly %}}. These stable KPI policies belong under [`reader.queries.<alias>`](https://docs.victoriametrics.com/anomaly-detection/components/reader/#per-query-parameters), where they remain consistent across every [univariate](#univariate-models) or [multivariate](#multivariate-models) model that uses the query. Existing model-level values remain compatible as model-local fallbacks when an attached query does not define the corresponding field; an explicit query value is authoritative.
+
 <div class="collapse-group">
 
 {{% collapse name="Queries" %}}
@@ -145,61 +149,46 @@ models:
 {{% collapse name="Detection direction" %}}
 
 ### Detection direction
-The `detection_direction` argument{{% available_from "v1.13.0" anomaly %}} can reduce [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-positive) when domain knowledge indicates that only values above or below the expected value are anomalous. Available values are `both`, `above_expected`, and `below_expected`.
+The `detection_direction` argument{{% available_from "v1.13.0" anomaly %}} can reduce [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-positive) when domain knowledge indicates that only values above or below the expected value are anomalous. Available values are `both`, `above_expected`, and `below_expected`. Configure it on the input query; model-level placement is {{% deprecated_from "v1.30.2" anomaly %}}.
 
-Here's how default (backward-compatible) behavior looks like - anomalies will be tracked in `both` directions (`y > yhat` or `y < yhat`). This is useful when there is no domain expertise to filter the required direction.
+Here's how the three options differ:
 
-![schema_detection_direction=both](schema_detection_direction_both.webp)
+![detection_direction comparison](schema_detection_direction.webp)
 
+With the default, backward-compatible `both` value, anomalies are tracked in both directions (`y > yhat` or `y < yhat`). This is useful when there is no domain expertise to filter the required direction.
 
 When set to `above_expected`, anomalies are tracked only when `y > yhat`.
 
 *Example metrics*: Error rate, response time, page load time, number of failed transactions - metrics where *lower values are better*, so **higher** values are typically tracked.
 
-![schema_detection_direction=above_expected](schema_detection_direction_above_expected.webp)
-
-
 When set to `below_expected`, anomalies are tracked only when `y < yhat`. 
 
 *Example metrics*: Service Level Agreement (SLA) compliance, conversion rate, Customer Satisfaction Score (CSAT) - metrics where *higher values are better*, so **lower** values are typically tracked.
 
-![schema_detection_direction=below_expected](schema_detection_direction_below_expected.webp)
-
-
-Config with a split example:
+One model can use multiple queries with different directions because the policy belongs to each query:
 
 ```yaml
-models:
-  model_above_expected:
-    class: 'zscore_online'
-    z_threshold: 3.0
-    # track only cases when y > yhat, otherwise anomaly_score would be explicitly set to 0
-    detection_direction: 'above_expected'
-    # for this query we do not need to track lower values, thus, set anomaly detection tracking for y > yhat (above_expected)
-    queries: ['query_values_the_lower_the_better']
-  model_below_expected:
-    class: 'zscore_online'
-    z_threshold: 3.0
-    # track only cases when y < yhat, otherwise anomaly_score would be explicitly set to 0
-    detection_direction: 'below_expected'
-    # for this query we do not need to track higher values, thus, set anomaly detection tracking for y < yhat (above_expected)
-    queries: ['query_values_the_higher_the_better']
-  model_bidirectional_default:
-    class: 'zscore_online'
-    z_threshold: 3.0
-    # track in both direction, same backward-compatible behavior in case this arg is missing
-    detection_direction: 'both'
-    # for this query both directions can be equally important for anomaly detection, thus, setting it bidirectional (both)
-    queries: ['query_values_both_direction_matters']
 reader:
   # ...
   queries:
-    query_values_the_lower_the_better: 
+    query_values_the_lower_the_better:
       expr: metricsql_expression1
-    query_values_the_higher_the_better: 
+      detection_direction: 'above_expected'  # query-level from v1.30.2; only y > yhat can be anomalous
+    query_values_the_higher_the_better:
       expr: metricsql_expression2
-    query_values_both_direction_matters: 
+      detection_direction: 'below_expected'  # query-level from v1.30.2; only y < yhat can be anomalous
+    query_values_both_direction_matters:
       expr: metricsql_expression3
+      detection_direction: 'both'  # query-level from v1.30.2; the default when omitted
+models:
+  model_all_directions:
+    class: 'zscore_online'
+    z_threshold: 3.0
+    queries: [
+      'query_values_the_lower_the_better',
+      'query_values_the_higher_the_better',
+      'query_values_both_direction_matters',
+    ]
 # other components like writer, schedule, monitoring
 ```
 
@@ -209,7 +198,7 @@ reader:
 
 ### Minimal deviation from expected
 
-`min_dev_from_expected`{{% available_from "v1.13.0" anomaly %}} argument is designed to **reduce [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-positive)** in scenarios where deviations between the actual value (`y`) and the expected value (`yhat`) are **relatively** high. Such deviations can cause models to generate high [anomaly scores](https://docs.victoriametrics.com/anomaly-detection/faq/#what-is-anomaly-score). However, these deviations may not be significant enough in **absolute values** from a business perspective to be considered anomalies. This parameter ensures that anomaly scores for data points where `|y - yhat| < min_dev_from_expected` are explicitly set to 0. By default, if this parameter is not set, it is set to `0` to maintain backward compatibility.
+`min_dev_from_expected`{{% available_from "v1.13.0" anomaly %}} argument is designed to **reduce [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-positive)** in scenarios where deviations between the actual value (`y`) and the expected value (`yhat`) are **relatively** high. Such deviations can cause models to generate high [anomaly scores](https://docs.victoriametrics.com/anomaly-detection/faq/#what-is-anomaly-score). However, these deviations may not be significant enough in **absolute values** from a business perspective to be considered anomalies. This parameter ensures that anomaly scores for data points where `|y - yhat| < min_dev_from_expected` are explicitly set to 0. By default, if this parameter is not set, it is set to `0` to maintain backward compatibility. Configure it on the input query; model-level placement is {{% deprecated_from "v1.30.2" anomaly %}}.
 
 > [!NOTE]
 {{% available_from "v1.23.0" anomaly %}} The `min_dev_from_expected` argument can be a list of two float values, allowing separate thresholds for upper and lower deviations. This is useful when the acceptable deviation varies in different directions (e.g., `min_dev_from_expected: [0.01, 0.02]` means that the lower bound is `0.01` when `y` is less than `yhat` and the upper bound is `0.02` when `y` is greater than `yhat`). If only one value is provided, it is broadcasted to both directions, meaning that the same threshold is applied for both upper and lower deviations (e.g., `min_dev_from_expected: 0.01` means that the lower bound is `0.01` when `y` is less than `yhat` and the upper bound is also `0.01` when `y` is greater than `yhat`).
@@ -218,15 +207,9 @@ reader:
 
 *Example*: Consider a scenario where CPU utilization in specific mode is low and oscillates around 0.3% (0.003). A sudden spike to 1.3% (0.013) represents a +333% increase in **relative** terms, but only a +1 percentage point (0.01) increase in **absolute** terms, which may be negligible and not warrant an alert. Setting the `min_dev_from_expected` argument to `0.01` (1%) will ensure that all anomaly scores for deviations <= `0.01` are set to 0.
 
-Visualizations below demonstrate this concept; the green zone defined as the `[yhat - min_dev_from_expected, yhat + min_dev_from_expected]` range excludes actual data points (`y`) from generating anomaly scores if they fall within that range.
+The visualization below demonstrates this concept. The narrow blue model prediction boundary is nested inside the wider green business protection boundary. Actual values outside the prediction boundary but still within `[yhat - min_dev_from_expected, yhat + min_dev_from_expected]` receive `anomaly_score = 0`; only values outside the green boundary remain anomalous.
 
-![min_dev_from_expected-default](schema_min_dev_from_expected_0.webp)
-
-
-![min_dev_from_expected-small](schema_min_dev_from_expected_1_0.webp)
-
-
-![min_dev_from_expected-big](schema_min_dev_from_expected_5_0.webp)
+![min_dev_from_expected](schema_min_dev_from_expected.webp)
 
 Example config of how to use this param based on query results:
 
@@ -236,23 +219,17 @@ reader:
   # ...
   queries:
     # the usage of min_dev should reduce false positives here
-    need_to_include_min_dev: 
+    need_to_include_min_dev:
       expr: small_abs_values_metricsql_expression
+      min_dev_from_expected: [5.0, 5.0]  # query-level from v1.30.2
     # min_dev is not really needed here
-    normal_behavior: 
+    normal_behavior:
       expr: no_need_to_exclude_small_deviations_metricsql_expression
 models:
-  zscore_with_min_dev:
+  zscore:
     class: 'zscore_online'
     z_threshold: 3
-    min_dev_from_expected: [5.0, 5.0]  # set the same threshold for both directions, meaning that deviations less than 5.0 in absolute values won't be considered anomalous, even if they are relatively significant
-    queries: ['need_to_include_min_dev']  # use such models on queries where domain experience confirm usefulness
-  zscore_wo_min_dev:
-    class: 'zscore_online'
-    z_threshold: 3
-    # if not set, equals to setting min_dev_from_expected == 0 (meaning no filtering is applied)
-    # min_dev_from_expected: [0.0, 0.0]
-    queries: ['normal_behavior']  # use the default where it's not needed
+    queries: ['need_to_include_min_dev', 'normal_behavior']
 ```
 
 {{% /collapse %}}
@@ -261,13 +238,17 @@ models:
 
 ### Minimal relative deviation from expected
 
-{{% available_from "v1.29.1" anomaly %}} `min_rel_dev_from_expected` argument serves a similar purpose to `min_dev_from_expected` (see [section above](#minimal-deviation-from-expected)), but focuses on **relative deviations** rather than absolute ones. It is designed to reduce [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-positive) in scenarios where the relative deviation between the actual value (`y`) and the expected value (`yhat`) is high, but the absolute deviation is not significant enough to be considered an anomaly from a business perspective. This parameter ensures that anomaly scores for data points where `|y - yhat| / |yhat| < min_rel_dev_from_expected` are explicitly set to 0. By default, if this parameter is not set, it is set to `0` to maintain backward compatibility. 
+{{% available_from "v1.29.1" anomaly %}} `min_rel_dev_from_expected` argument serves a similar purpose to `min_dev_from_expected` (see [section above](#minimal-deviation-from-expected)), but focuses on **relative deviations** rather than absolute ones. It is designed to reduce [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-positive) in scenarios where the relative deviation between the actual value (`y`) and the expected value (`yhat`) is high, but the absolute deviation is not significant enough to be considered an anomaly from a business perspective. This parameter ensures that anomaly scores for data points where `|y - yhat| / |yhat| < min_rel_dev_from_expected` are explicitly set to 0. By default, if this parameter is not set, it is set to `0` to maintain backward compatibility. Configure it on the input query; model-level placement is {{% deprecated_from "v1.30.2" anomaly %}}.
 
 Parameter can be a list of two float values, *allowing separate thresholds for upper and lower relative deviations*. If only one value is provided, it is broadcasted to both directions.
 
 > [!NOTE]
 If both `min_dev_from_expected` [arg](#minimal-deviation-from-expected) and `min_rel_dev_from_expected` are set, the model will combine both filters. A data point will be considered anomalous (i.e., have an anomaly score != 0) only if it exceeds **both** the *absolute* deviation threshold defined by `min_dev_from_expected` and the *relative* deviation threshold defined by `min_rel_dev_from_expected`. This allows for more granular control over anomaly detection, ensuring that only significant deviations in both absolute and relative terms are flagged as anomalies.
 
+The green business protection boundary below scales with `|yhat|`, while the model prediction boundary remains visible inside it. Actual values outside the blue boundary but inside the proportional green boundary receive `anomaly_score = 0`.
+
+![min_rel_dev_from_expected](schema_min_rel_dev_from_expected.webp)
+
 
 *Example*: Consider a scenario of monitoring incoming traffic to websites that typically receives *unknown in advance* requests per second (from tens to thousands). Setting absolute deviation threshold with `min_dev_from_expected` *may not be effective in reducing false positives*, as even a small increase in traffic (e.g., from 10 to 20 requests per second) can represent a 100% relative increase, which may be significant for that website. Instead, setting `min_rel_dev_from_expected` to smaller relative value - `[20, 40]` (20/40%) - will ensure that traffic drop from 10 to 8 requests per second (20% decrease) and traffic spike from 10 to 14 requests per second (40% increase) won't be considered anomalous, even if they exceed confidence intervals, thus, reducing false positives for small absolute deviations that are relatively significant.
 
@@ -279,23 +260,17 @@ reader:
   # ...
   queries:
     # the usage of min_rel_dev should reduce false positives here
-    need_to_include_min_rel_dev: 
+    need_to_include_min_rel_dev:
       expr: small_abs_values_metricsql_expression
+      min_rel_dev_from_expected: [10, 20]  # query-level from v1.30.2
     # min_rel_dev is not really needed here
-    normal_behavior: 
+    normal_behavior:
       expr: no_need_to_exclude_small_deviations_metricsql_expression
 models:
-  zscore_with_min_rel_dev:
+  zscore:
     class: 'zscore_online'
     z_threshold: 3
-    min_rel_dev_from_expected: [10, 20]  # set different thresholds for both directions, meaning that relative deviations less than 10% when y < yhat and less than 20% when y > yhat won't be considered anomalous, even if they exceed confidence intervals, thus, reducing false positives for small absolute deviations that are relatively significant
-    queries: ['need_to_include_min_rel_dev']  # use such models on queries where domain experience confirm usefulness
-  zscore_wo_min_rel_dev:
-    class: 'zscore_online'
-    z_threshold: 3
-    # if not set, equals to setting min_rel_dev_from_expected == 0 (meaning no filtering is applied)
-    # min_rel_dev_from_expected: [0, 0]
-    queries: ['normal_behavior']  # use the default where it's not needed
+    queries: ['need_to_include_min_rel_dev', 'normal_behavior']
 ```
   
 
@@ -318,17 +293,29 @@ reader:
   # assume there are M unique hosts identified by the `host` label
   queries:
     # return one timeseries for each CPU mode per host, total = N*M timeseries
-    cpu: sum(rate(node_cpu_seconds_total[5m])) by (host, mode)
+    cpu:
+      expr: sum(rate(node_cpu_seconds_total[5m])) by (host, mode)
+      data_range: [0, 'inf']
+      detection_direction: both
+      min_rel_dev_from_expected: [15, 15]
     # return one timeseries per host, total = 1*M timeseries
-    ram: | 
-      (
-       (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) 
-       / node_memory_MemTotal_bytes
-      ) * 100 by (host)
+    ram:
+      expr: |
+        100 * (
+          1 - node_memory_MemAvailable_bytes
+          / node_memory_MemTotal_bytes
+        )
+      data_range: [0, 100]
+      detection_direction: above_expected
+      min_rel_dev_from_expected: [0, 15]
     # return one timeseries per host for both network receive and transmit data, total = 1*M timeseries
-    network: |
-      sum(rate(node_network_receive_bytes_total[5m])) by (host) 
-      + sum(rate(node_network_transmit_bytes_total[5m])) by (host)
+    network:
+      expr: |
+        sum(rate(node_network_receive_bytes_total[5m])) by (host)
+        + sum(rate(node_network_transmit_bytes_total[5m])) by (host)
+      data_range: [0, 'inf']
+      detection_direction: below_expected
+      min_rel_dev_from_expected: [20, 0]
 
 models:
   envelope: # alias for the model
@@ -342,6 +329,9 @@ models:
     groupby: [host]
 ```
 
+> [!TIP]
+> {{% available_from "v1.30.2" anomaly %}} Multivariate Temporal Envelope applies each query's [`data_range`, `detection_direction`, and minimum relative deviation](https://docs.victoriametrics.com/anomaly-detection/components/reader/#per-query-parameters) to every channel returned by that query before aggregating the joint anomaly score. The example detects CPU deviations in either direction, RAM increases of at least 15%, and network drops of at least 20% within each host model.
+
 {{% /collapse %}}
 
 {{% collapse name="Scale" %}}
@@ -359,6 +349,10 @@ For example, setting `scale: [1.2, 0.75]` for particular model will:
 - **Increase** the width of the lower confidence interval by **20%**.
 - **Decrease** the width of the upper confidence boundary by **25%**.
 
+Alternative visualization:
+
+![two-sided scale comparison](schema_scale_overview_v2.webp)
+
 The most common **use case** is when there is a preference to **widen one side** to blacklist smaller false positives (which otherwise would have [anomaly scores](https://docs.victoriametrics.com/anomaly-detection/faq/#how-is-anomaly-score-calculated) **only slightly higher than 1.0**, still making such data points **anomalous**), while **tightening the other side** to avoid missing true positives due to an overly loose margin (leading to [anomaly scores](https://docs.victoriametrics.com/anomaly-detection/faq/#how-is-anomaly-score-calculated) being slightly less than 1.0, making such data points **non-anomalous**).
 
 ```yaml
@@ -557,6 +551,8 @@ For a multivariate model, **one shared model instance** is fitted and used acros
 
 For example, if you have some **multivariate** model to use 3 [MetricQL queries](https://docs.victoriametrics.com/victoriametrics/metricsql/), each returning 5 time series, there will be one shared model created in total. Once fit, this model will expect **exactly 15 time series with exact same labelsets as an input**. This model will produce **one shared [output](#vmanomaly-output)**.
 
+> {{% available_from "v1.30.2" anomaly %}} Multivariate Temporal Envelope and Isolation Forest accept matching input channels in any order. The channel set must still match the fitted model exactly: missing, extra, and duplicate channels are rejected, while a matching set is restored to learned fit order before inference or online updates.
+
 > {{% available_from "v1.16.0" anomaly %}} N models — one for each N unique combinations of label values specified in the `groupby` [common argument](#group-by) — can be trained. This allows for context separation (e.g., one model per host, region, or other relevant grouping label), leading to improved accuracy and faster training. See an example [here](#group-by).
 
 If during an inference, you got a **different amount of series** or some series having a **new labelset** (not present in any of fitted models), the inference will be skipped until you get a model, trained particularly for such labelset during forthcoming re-fit step. 
@@ -685,7 +681,7 @@ Selecting model [hyperparameters](https://en.wikipedia.org/wiki/Hyperparameter_(
 - `tuned_class_name` (string) - [Built-in model class](#built-in-models) to wrap, i.e. `zscore_online`
 - `optimization_params` (dict) - Optimization parameters for *unsupervised* model tuning. Control percentage of found anomalies, as well as a tradeoff between time spent and the accuracy. The higher `timeout` and `n_trials` are, the better model configuration can be found for `tuned_class_name`, but the longer it takes and vice versa. Set `n_jobs` to `-1` to use all the CPUs available, it makes sense if only you have a big dataset to train on during `fit` calls, otherwise overhead isn't worth it.
   - `anomaly_percentage` (float) - Expected percentage of anomalies that can be seen in training data, from `[0, 0.5)` interval (i.e. 0.01 means it's expected ~ 1% of anomalies to be present in training data). This is a *required* parameter.
-  - `optimized_business_params` (list[string]) - {{% available_from "v1.15.0" anomaly %}} this argument allows particular [business-specific parameters](#common-args) such as [`detection_direction`](https://docs.victoriametrics.com/anomaly-detection/components/models/#detection-direction) or [`min_dev_from_expected`](https://docs.victoriametrics.com/anomaly-detection/components/models/#minimal-deviation-from-expected) to remain **unchanged during optimizations, retaining their initial values**. I.e. setting `optimized_business_params` to  `['detection_direction']` will allow to optimize only `detection_direction` business-specific arg, while `min_dev_from_expected` will retain its default value of (e.g. [1, 2] if set to that value in model config). By default and if not set, will be equal to `[]` (empty list), meaning no business params will be optimized. **A recommended option is to leave it empty** as this feature is still experimental and may lead to unexpected results.
+  - `optimized_business_params` (list[string]) - {{% available_from "v1.15.0" anomaly %}} Experimental optimization of model-level business parameters is {{% deprecated_from "v1.30.2" anomaly %}}. Keep this list empty and configure stable `detection_direction`, `min_dev_from_expected`, and `min_rel_dev_from_expected` policies on [`reader.queries.<alias>`](https://docs.victoriametrics.com/anomaly-detection/components/reader/#per-query-parameters) instead.
   - `seed` (int) - Random seed for reproducibility and deterministic nature of underlying optimizations.
   - `validation_scheme` (string) - {{% available_from "v1.25.1" anomaly %}} the validation scheme to use for hyperparameter tuning, either `regular` (time-based default) or `leaky` (regular cross-validation with  `n_splits` folds, where each fold is a time-based split of the data). The `leaky` scheme is recommended for `anomaly_percentage` ~ 0%, as it allows the model to "see" all the datapoints at least once during the optimization process, which can lead to better results in such cases. Defaults to `regular`.
   - `n_splits` (int) - How many folds to create for hyperparameter tuning out of your data. The higher, the longer it takes but the better the results can be. Defaults to 3.
@@ -809,7 +805,7 @@ For simple profiles without strong trend or seasonality, prefer [Online MAD](#on
 
 Preset suffixes describe expected profile shape: `smooth` represents gradual recurring curves, `spiky` represents narrow phase peaks, and `plateau` represents sustained calendar levels. Choose only profiles supported by the data. Calendar and holiday features use civil time from the configured query timezone, so hour/day profiles remain aligned across daylight-saving-time transitions.
 
-Temporal Envelope also supports the [common model arguments](#common-args), including `queries`, `schedulers`, `provide_series`, `detection_direction`, `scale`, `clip_predictions`, `min_dev_from_expected`, and `min_rel_dev_from_expected`. Input `data_range` and query timezone are configured on the [reader](https://docs.victoriametrics.com/anomaly-detection/components/reader/#config-parameters).
+Temporal Envelope also supports the [common model arguments](#common-args), including `queries`, `schedulers`, `provide_series`, `scale`, and `clip_predictions`. Configure `data_range`, `detection_direction`, `min_dev_from_expected`, `min_rel_dev_from_expected`, and query timezone under the corresponding [reader query](https://docs.victoriametrics.com/anomaly-detection/components/reader/#per-query-parameters). The multivariate variant applies these business policies independently to each input channel {{% available_from "v1.30.2" anomaly %}}, so one model can represent combinations such as temperature above expected, power above expected, and clock below expected.
 
 The multivariate variant uses `class: temporal_envelope_multivariate` or `model.online.TemporalEnvelopeMultivariateModel` and adds:
 
@@ -900,10 +896,13 @@ models:
     # See https://docs.victoriametrics.com/anomaly-detection/components/models/#common-args
     #
     # provide_series: ['anomaly_score', 'yhat', 'yhat_lower', 'yhat_upper']
-    # schedulers: [all scheduler aliases defined in `scheduler` section]
-    # queries: [all query aliases defined in `reader.queries` section]
-    # detection_direction: 'both'  # meaning both drops and spikes will be captured
-    # min_dev_from_expected: [0.0, 0.0]  # meaning, no minimal threshold is applied to prevent smaller anomalies
+    # schedulers: [
+    #   all scheduler aliases defined in `scheduler` section,
+    # ]
+    # queries: [
+    #   all query aliases defined in `reader.queries` section,
+    # ]
+    # Configure detection_direction and minimum-deviation policies under reader.queries.<alias> (query-level from v1.30.2).
     # scale: [1.0, 1.0]  # if needed, prediction intervals' width can be increased (>1) or narrowed (<1)
     # clip_predictions: False  # if data_range for respective `queries` is set in reader, `yhat.*` columns will be clipped
     # anomaly_score_outside_data_range: 1.01  # auto anomaly score (1.01) if `y` (real value) is outside of data_range, if set
@@ -967,10 +966,13 @@ models:
     # See https://docs.victoriametrics.com/anomaly-detection/components/models/#common-args
     #
     # provide_series: ['anomaly_score', 'yhat', 'yhat_lower', 'yhat_upper']
-    # schedulers: [all scheduler aliases defined in `scheduler` section]
-    # queries: [all query aliases defined in `reader.queries` section]
-    # detection_direction: 'both'  # meaning both drops and spikes will be captured
-    # min_dev_from_expected: [0.0, 0.0]  # meaning, no minimal threshold is applied to prevent smaller anomalies
+    # schedulers: [
+    #   all scheduler aliases defined in `scheduler` section,
+    # ]
+    # queries: [
+    #   all query aliases defined in `reader.queries` section,
+    # ]
+    # Configure detection_direction and minimum-deviation policies under reader.queries.<alias> (query-level from v1.30.2).
     # scale: [1.0, 1.0]  # if needed, prediction intervals' width can be increased (>1) or narrowed (<1)
     # clip_predictions: False  # if data_range for respective `queries` is set in reader, `yhat.*` columns will be clipped
     # anomaly_score_outside_data_range: 1.01  # auto anomaly score (1.01) if `y` (real value) is outside of data_range, if set
@@ -1014,10 +1016,13 @@ models:
     # See https://docs.victoriametrics.com/anomaly-detection/components/models/#common-args
     #
     # provide_series: ['anomaly_score', 'yhat', 'yhat_lower', 'yhat_upper']
-    # schedulers: [all scheduler aliases defined in `scheduler` section]
-    # queries: [all query aliases defined in `reader.queries` section]
-    # detection_direction: 'both'  # meaning both drops and spikes will be captured
-    # min_dev_from_expected: [0.0, 0.0]  # meaning, no minimal threshold is applied to prevent smaller anomalies
+    # schedulers: [
+    #   all scheduler aliases defined in `scheduler` section,
+    # ]
+    # queries: [
+    #   all query aliases defined in `reader.queries` section,
+    # ]
+    # Configure detection_direction and minimum-deviation policies under reader.queries.<alias> (query-level from v1.30.2).
     # scale: [1.0, 1.0]  # if needed, prediction intervals' width can be increased (>1) or narrowed (<1)
     # clip_predictions: False  # if data_range for respective `queries` is set in reader, `yhat.*` columns will be clipped
     # anomaly_score_outside_data_range: 1.01  # auto anomaly score (1.01) if `y` (real value) is outside of data_range, if set
@@ -1060,10 +1065,13 @@ models:
     # See https://docs.victoriametrics.com/anomaly-detection/components/models/#common-args
     #
     # provide_series: ['anomaly_score', 'yhat', 'yhat_lower', 'yhat_upper']
-    # schedulers: [all scheduler aliases defined in `scheduler` section]
-    # queries: [all query aliases defined in `reader.queries` section]
-    # detection_direction: 'both'  # meaning both drops and spikes will be captured
-    # min_dev_from_expected: [0.0, 0.0]  # meaning, no minimal threshold is applied to prevent smaller anomalies
+    # schedulers: [
+    #   all scheduler aliases defined in `scheduler` section,
+    # ]
+    # queries: [
+    #   all query aliases defined in `reader.queries` section,
+    # ]
+    # Configure detection_direction and minimum-deviation policies under reader.queries.<alias> (query-level from v1.30.2).
     # scale: [1.0, 1.0]  # if needed, prediction intervals' width can be increased (>1) or narrowed (<1)
     # clip_predictions: False  # if data_range for respective `queries` is set in reader, `yhat.*` columns will be clipped
     # anomaly_score_outside_data_range: 1.01  # auto anomaly score (1.01) if `y` (real value) is outside of data_range, if set
@@ -1099,6 +1107,7 @@ Resulting metrics of the model are described [here](#vmanomaly-output).
 - `tz_use_cyclical_encoding`{{% available_from "v1.18.0" anomaly %}} (bool): If set to `True`, applies [cyclical encoding technique](https://www.kaggle.com/code/avanwyk/encoding-cyclical-features-for-deep-learning) to timezone-aware seasonalities. Should be used with `tz_aware=True` and `tz_seasonalities`.
 - `forecast_at`{{% available_from "v1.25.3" anomaly %}} (list[str]): Specifies future relative offsets for which forecasts should be generated (e.g., `['1h', '1d']`). Works similarly to [predict_linear](https://docs.victoriametrics.com/victoriametrics/metricsql/#predict_linear) in MetricQL, but with more flexibility and seasonality support - produced series will have *the same timestamp* as the other [output](#vmanomaly-output) series, but with the forecasted value for the *future timestamp*. Defaults to `[]` (empty list, meaning no future forecasts are produced). If set, `provide_series` must include at least `yhat` for point-wise forecasts (and `yhat_lower` or/and `yhat_upper` for respective confidence intervals). For example, if `forecast_at` is set to `['1h', '1d']`, the model will produce forecasts for both the next hour and the next day, and these series can be accessed by `yhat_1h`, `yhat_lower_1h`, `yhat_upper_1h`, `yhat_1d`, `yhat_lower_1d`, and `yhat_upper_1d` in the output, respectively. See [FAQ](https://docs.victoriametrics.com/anomaly-detection/faq/#forecasting) for more details.
 
+> [!WARNING]
 > `forecast_at` parameter can lead to **significant increase in active timeseries** if you have a lot of time series returned by your queries, as it will produce additional series for each of the future timestamps specified in `forecast_at` (optionally multiplied by 1-3 if interval forecasts are included). For example, if you have 1000 time series returned by your query and set `forecast_at` to `[1h, 1d, 1w]`, and `provide_series` includes `yhat_lower` and `yhat_upper`, it will produce 1000 (series) * 3 (intervals) * 3 (predictions, point + interval) = 9000 additional timeseries. Consider using it only on small subset of metrics (e.g. grouped by `host` or `region`) to avoid this issue, as it also **proportionally (to the number of `forecast_at` elements) increases the timings of inference calls**.
 
 - `compression` {{% available_from "v1.28.1" anomaly %}} (dict, optional): Configuration for downsampling input data before fitting the model. Useful for high-frequency data to reduce CPU and RAM/disk load and improve model performance. The `compression` block supports the following parameters:
@@ -1121,10 +1130,13 @@ models:
     # See https://docs.victoriametrics.com/anomaly-detection/components/models/#common-args
     #
     # provide_series: ['anomaly_score', 'yhat', 'yhat_lower', 'yhat_upper', 'trend']
-    # schedulers: [all scheduler aliases defined in `scheduler` section]
-    # queries: [all query aliases defined in `reader.queries` section]
-    # detection_direction: 'both'  # meaning both drops and spikes will be captured
-    # min_dev_from_expected: [0.0, 0.0]  # meaning, no minimal threshold is applied to prevent smaller anomalies
+    # schedulers: [
+    #   all scheduler aliases defined in `scheduler` section,
+    # ]
+    # queries: [
+    #   all query aliases defined in `reader.queries` section,
+    # ]
+    # Configure detection_direction and minimum-deviation policies under reader.queries.<alias> (query-level from v1.30.2).
     # scale: [1.0, 1.0]  # if needed, prediction intervals' width can be increased (>1) or narrowed (<1)
     # clip_predictions: False  # if data_range for respective `queries` is set in reader, `yhat.*` columns will be clipped
     # anomaly_score_outside_data_range: 1.01  # auto anomaly score (1.01) if `y` (real value) is outside of data_range, if set
@@ -1155,10 +1167,13 @@ models:
     # See https://docs.victoriametrics.com/anomaly-detection/components/models/#common-args
     #
     # provide_series: ['anomaly_score', 'yhat', 'yhat_lower', 'yhat_upper', 'trend']
-    # schedulers: [all scheduler aliases defined in `scheduler` section]
-    # queries: [all query aliases defined in `reader.queries` section]
-    # detection_direction: 'both'  # meaning both drops and spikes will be captured
-    # min_dev_from_expected: [0.0, 0.0]  # meaning, no minimal threshold is applied to prevent smaller anomalies
+    # schedulers: [
+    #   all scheduler aliases defined in `scheduler` section,
+    # ]
+    # queries: [
+    #   all query aliases defined in `reader.queries` section,
+    # ]
+    # Configure detection_direction and minimum-deviation policies under reader.queries.<alias> (query-level from v1.30.2).
     # scale: [1.0, 1.0]  # if needed, prediction intervals' width can be increased (>1) or narrowed (<1)
     # clip_predictions: False  # if data_range for respective `queries` is set in reader, `yhat.*` columns will be clipped
     # anomaly_score_outside_data_range: 1.01  # auto anomaly score (1.01) if `y` (real value) is outside of data_range, if set
@@ -1254,8 +1269,12 @@ models:
     # See https://docs.victoriametrics.com/anomaly-detection/components/models/#common-args
     #
     # provide_series: ['anomaly_score', 'yhat', 'yhat_lower', 'yhat_upper']
-    # schedulers: [all scheduler aliases defined in `scheduler` section]
-    # queries: [all query aliases defined in `reader.queries` section]
+    # schedulers: [
+    #   all scheduler aliases defined in `scheduler` section,
+    # ]
+    # queries: [
+    #   all query aliases defined in `reader.queries` section,
+    # ]
     # anomaly_score_outside_data_range: 1.01  # auto anomaly score (1.01) if `y` (real value) is outside of data_range, if set
 ```
 
@@ -1316,10 +1335,13 @@ models:
     # See https://docs.victoriametrics.com/anomaly-detection/components/models/#common-args
     #
     # provide_series: ['anomaly_score', 'yhat', 'yhat_lower', 'yhat_upper']
-    # schedulers: [all scheduler aliases defined in `scheduler` section]
-    # queries: [all query aliases defined in `reader.queries` section]
-    # detection_direction: 'both'  # meaning both drops and spikes will be captured
-    # min_dev_from_expected: [0.0, 0.0]  # meaning, no minimal threshold is applied to prevent smaller anomalies
+    # schedulers: [
+    #   all scheduler aliases defined in `scheduler` section,
+    # ]
+    # queries: [
+    #   all query aliases defined in `reader.queries` section,
+    # ]
+    # Configure detection_direction and minimum-deviation policies under reader.queries.<alias> (query-level from v1.30.2).
     # scale: [1.0, 1.0]  # if needed, prediction intervals' width can be increased (>1) or narrowed (<1)
     # clip_predictions: False  # if data_range for respective `queries` is set in reader, `yhat.*` columns will be clipped
     # anomaly_score_outside_data_range: 1.01  # auto anomaly score (1.01) if `y` (real value) is outside of data_range, if set
@@ -1359,10 +1381,13 @@ models:
     # See https://docs.victoriametrics.com/anomaly-detection/components/models/#common-args
     #
     # provide_series: ['anomaly_score', 'yhat', 'yhat_lower', 'yhat_upper']
-    # schedulers: [all scheduler aliases defined in `scheduler` section]
-    # queries: [all query aliases defined in `reader.queries` section]
-    # detection_direction: 'both'  # meaning both drops and spikes will be captured
-    # min_dev_from_expected: [0.0, 0.0]  # meaning, no minimal threshold is applied to prevent smaller anomalies
+    # schedulers: [
+    #   all scheduler aliases defined in `scheduler` section,
+    # ]
+    # queries: [
+    #   all query aliases defined in `reader.queries` section,
+    # ]
+    # Configure detection_direction and minimum-deviation policies under reader.queries.<alias> (query-level from v1.30.2).
     # scale: [1.0, 1.0]  # if needed, prediction intervals' width can be increased (>1) or narrowed (<1)
     # clip_predictions: False  # if data_range for respective `queries` is set in reader, `yhat.*` columns will be clipped
     # anomaly_score_outside_data_range: 1.01  # auto anomaly score (1.01) if `y` (real value) is outside of data_range, if set
@@ -1433,7 +1458,7 @@ Create `custom_model.py` with a `CustomModel` class derived from `Model`. A conc
 - `serialize`, which returns `bytes` suitable for on-disk storage;
 - `deserialize`, which restores the same model from bytes or a file path.
 
-Model-specific configuration is passed through the `args` mapping. The example below learns a stationary normal interval. It emits the standard forecast columns and uses the base-class anomaly-score calculation, so common settings such as `detection_direction`, `data_range`, `scale`, and minimum deviations continue to work.
+Model-specific configuration is passed through the `args` mapping. The example below learns a stationary normal interval. It emits the standard forecast columns and uses the base-class anomaly-score calculation, so query policies such as `detection_direction`, `data_range`, and minimum deviations, together with model settings such as `scale`, continue to work.
 
 ```python
 from pickle import dumps
@@ -1561,7 +1586,7 @@ See the [component configuration reference](https://docs.victoriametrics.com/ano
 Pull the `vmanomaly` image:
 
 ```sh
-docker pull victoriametrics/vmanomaly:v1.30.1
+docker pull victoriametrics/vmanomaly:v1.30.2
 ```
 
 Mount the module at `/vmanomaly/src/model/custom.py`, which matches the configured import path `model.custom.CustomModel`. Validate the complete configuration with `--dryRun` before starting the long-running service.
@@ -1571,7 +1596,7 @@ docker run --rm \
   -v "$PWD/license:/license:ro" \
   -v "$PWD/custom_model.py:/vmanomaly/src/model/custom.py:ro" \
   -v "$PWD/config.yaml:/config.yaml:ro" \
-  victoriametrics/vmanomaly:v1.30.1 \
+  victoriametrics/vmanomaly:v1.30.2 \
   /config.yaml \
   --licenseFile=/license \
   --dryRun
@@ -1667,10 +1692,13 @@ models:
     # See https://docs.victoriametrics.com/anomaly-detection/components/models/#common-args
     #
     # provide_series: ['anomaly_score', 'yhat', 'yhat_lower', 'yhat_upper']
-    # schedulers: [all scheduler aliases defined in `scheduler` section]
-    # queries: [all query aliases defined in `reader.queries` section]
-    # detection_direction: 'both'  # meaning both drops and spikes will be captured
-    # min_dev_from_expected: [0.0, 0.0]  # meaning, no minimal threshold is applied to prevent smaller anomalies
+    # schedulers: [
+    #   all scheduler aliases defined in `scheduler` section,
+    # ]
+    # queries: [
+    #   all query aliases defined in `reader.queries` section,
+    # ]
+    # Configure detection_direction and minimum-deviation policies under reader.queries.<alias> (query-level from v1.30.2).
     # scale: [1.0, 1.0]  # if needed, prediction intervals' width can be increased (>1) or narrowed (<1)
     # clip_predictions: False  # if data_range for respective `queries` is set in reader, `yhat.*` columns will be clipped
     # anomaly_score_outside_data_range: 1.01  # auto anomaly score (1.01) if `y` (real value) is outside of data_range, if set
@@ -1709,10 +1737,13 @@ models:
     # See https://docs.victoriametrics.com/anomaly-detection/components/models/#common-args
     #
     # provide_series: ['anomaly_score', 'yhat', 'yhat_lower', 'yhat_upper']
-    # schedulers: [all scheduler aliases defined in `scheduler` section]
-    # queries: [all query aliases defined in `reader.queries` section]
-    # detection_direction: 'both'  # meaning both drops and spikes will be captured
-    # min_dev_from_expected: [0.0, 0.0]  # meaning, no minimal threshold is applied to prevent smaller anomalies
+    # schedulers: [
+    #   all scheduler aliases defined in `scheduler` section,
+    # ]
+    # queries: [
+    #   all query aliases defined in `reader.queries` section,
+    # ]
+    # Configure detection_direction and minimum-deviation policies under reader.queries.<alias> (query-level from v1.30.2).
     # scale: [1.0, 1.0]  # if needed, prediction intervals' width can be increased (>1) or narrowed (<1)
     # clip_predictions: False  # if data_range for respective `queries` is set in reader, `yhat.*` columns will be clipped
     # anomaly_score_outside_data_range: 1.01  # auto anomaly score (1.01) if `y` (real value) is outside of data_range, if set

docs/anomaly-detection/components/monitoring.md

@@ -1,5 +1,6 @@
 ---
 title: Monitoring
+description: "Self-monitoring via push and pull models."
 weight: 5
 menu:
   docs:
@@ -333,6 +334,14 @@ For detailed guidance on configuring mTLS parameters such as `verify_tls`, `tls_
         <tr>
             <td>
 
+<span style="white-space: nowrap;">`vmanomaly_native_threads_per_worker`</span>
+            </td>
+            <td>Gauge</td>
+            <td>Effective maximum native numerical-library threads per model worker{{% available_from "v1.30.2" anomaly %}} after resolving [`settings.native_threads_per_worker`](https://docs.victoriametrics.com/anomaly-detection/components/settings/#parallelization) against the effective worker count and container-aware CPU capacity.</td>
+        </tr>
+        <tr>
+            <td>
+
 <span style="white-space: nowrap;">`vmanomaly_config_entities`</span>
             </td>
             <td>Gauge</td>

docs/anomaly-detection/components/reader.md

@@ -1,5 +1,6 @@
 ---
 title: Reader
+description: "Data reader configuration. MetricsQL queries from VictoriaMetrics or LogsQL from VictoriaLogs/VictoriaTraces."
 weight: 2
 menu:
   docs:
@@ -59,7 +60,7 @@ reader:
       step: '10s'  # individual step for this query, will be filled with `sampling_period` from the root level
       data_range: ['-inf', 'inf']  # by default, no constraints applied on data range
       tz: 'UTC'  # by default, tz-free data is used throughout the model lifecycle
-      # new query-level arguments will be added in backward-compatible way in future releases
+      # from v1.30.2, explicitly add detection_direction and minimum-deviation policies here when needed
 ```
 {{% /collapse %}}
 
@@ -85,6 +86,16 @@ There is change {{% available_from "v1.13.0" anomaly %}} of [`queries`](https://
 
   > If not set explicitly (or if older config style prior to [v1.13.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1130)) is used, then it is set to reader-level `data_range` arg{{% available_from "v1.18.1" anomaly %}}
 
+  > Configuring `data_range` in a model is {{% deprecated_from "v1.30.2" anomaly %}}. Configure it under `reader.queries.<alias>` so the KPI domain remains the same when the query is attached to different models. Existing model-level values remain compatible as model-local fallbacks when the query does not define an explicit value.
+
+- `detection_direction`{{% available_from "v1.30.2" anomaly %}} (`both`, `above_expected`, or `below_expected`): controls whether deviations on both sides, only above the expected value, or only below it can produce anomaly scores. The default is `both`. See [detection direction](https://docs.victoriametrics.com/anomaly-detection/components/models/#detection-direction) for behavior details.
+
+- `min_dev_from_expected`{{% available_from "v1.30.2" anomaly %}} (float or one/two-element list[float]): ignores deviations smaller than the configured absolute threshold. A scalar or one-element list applies to both directions; a two-element list configures lower and upper deviations separately. See [minimal deviation from expected](https://docs.victoriametrics.com/anomaly-detection/components/models/#minimal-deviation-from-expected).
+
+- `min_rel_dev_from_expected`{{% available_from "v1.30.2" anomaly %}} (float or one/two-element list[float]): ignores deviations smaller than the configured percentage of the absolute expected value. A scalar or one-element list applies to both directions; a two-element list configures lower and upper percentages separately. See [minimal relative deviation from expected](https://docs.victoriametrics.com/anomaly-detection/components/models/#minimal-relative-deviation-from-expected).
+
+  > Configuring `detection_direction`, `min_dev_from_expected`, or `min_rel_dev_from_expected` in a model is {{% deprecated_from "v1.30.2" anomaly %}}. Query-level values are authoritative. Existing model-level values remain compatible only as model-local fallbacks for attached queries that do not define the corresponding policy.
+
 - `max_points_per_query`{{% available_from "v1.17.0" anomaly %}} (int): Optional arg, overrides how `search.maxPointsPerTimeseries` flag{{% available_from "v1.14.1" anomaly %}} impacts `vmanomaly` on splitting long `fit_window` [queries](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader) into smaller sub-intervals. This helps users avoid hitting the `search.maxQueryDuration` limit for individual queries by distributing initial query across multiple subquery requests with minimal overhead. Set less than `search.maxPointsPerTimeseries` if hitting `maxQueryDuration` limits. If set on a query-level, it overrides the global `max_points_per_query` (reader-level).
 
 - `tz`{{% available_from "v1.18.0" anomaly %}} (string): this optional argument enables timezone specification per query, overriding the reader’s default `tz`. This setting helps to account for local timezone shifts, such as [DST](https://en.wikipedia.org/wiki/Daylight_saving_time), in models that are sensitive to seasonal variations (e.g., [`TemporalEnvelopeModel`](https://docs.victoriametrics.com/anomaly-detection/components/models/#temporal-envelope) or [`OnlineQuantileModel`](https://docs.victoriametrics.com/anomaly-detection/components/models/#online-seasonal-quantile)).
@@ -115,7 +126,10 @@ reader:
     ingestion_rate_t1:
       expr: 'sum(rate(vm_rows_inserted_total[5m])) by (type) > 0'
       step: '2m'  # overrides global `sampling_period` of 1m
-      data_range: [10, 'inf']  # meaning only positive values > 10 are expected, i.e. a value `y` < 10 will trigger anomaly score > 1
+      data_range: [10, 'inf']  # query-level business policy from v1.30.2; y < 10 triggers anomaly score > 1
+      detection_direction: 'above_expected'  # query-level from v1.30.2; only spikes can be anomalous
+      min_dev_from_expected: [0, 5]  # query-level from v1.30.2; ignore upward deviations smaller than 5
+      min_rel_dev_from_expected: [0, 15]  # query-level from v1.30.2; ignore upward deviations below 15%
       max_points_per_query: 5000 # overrides reader-level value of 10000 for `ingestion_rate` query
       tz: 'America/New_York'  # to override reader-wise `tz`
       tenant_id: '1:0'  # overriding tenant_id to isolate data
@@ -302,6 +316,19 @@ Optional timeout {{% available_from "v1.30.0" anomaly %}} for post-fetch process
         <tr>
             <td>
 
+<span style="white-space: nowrap;">`workers`</span>
+            </td>
+            <td>
+
+`0`
+            </td>
+            <td>
+Maximum concurrent datasource fetch threads {{% available_from "v1.30.2" anomaly %}}. `0` selects a bounded value automatically from the number of queries and available CPUs. A positive value sets an explicit cap for queries and disk-streamed split-query chunks.
+            </td>
+        </tr>
+        <tr>
+            <td>
+
 <span style="white-space: nowrap;">`verify_tls`</span>
             </td>
             <td>
@@ -510,6 +537,7 @@ reader:
   timeout: '30s'  # backward-compatible default for both phases
   fetch_timeout: '30s'  # timeout for each datasource request, overrides `timeout` if set
   processing_timeout: '1m'  # timeout for preparing fetched series for fit/infer, overrides `timeout` if set
+  workers: 0  # automatic bounded datasource concurrency; set a positive value for an explicit cap
   query_from_last_seen_timestamp: True  # false by default
   latency_offset: '1ms'
   series_processing_batch_size: 8
@@ -896,6 +924,19 @@ Optional timeout {{% available_from "v1.30.0" anomaly %}} for post-fetch process
         <tr>
             <td>
 
+<span style="white-space: nowrap;">`workers`</span>
+            </td>
+            <td>
+
+`0`
+            </td>
+            <td>
+Maximum concurrent datasource fetch threads {{% available_from "v1.30.2" anomaly %}}. `0` selects a bounded value automatically from the number of queries and available CPUs. A positive value sets an explicit cap for queries and disk-streamed split-query chunks.
+            </td>
+        </tr>
+        <tr>
+            <td>
+
 <span style="white-space: nowrap;">`verify_tls`</span>
             </td>
             <td>
@@ -1037,12 +1078,15 @@ reader:
   timeout: '30s'  # backward-compatible default for both phases
   fetch_timeout: '30s'  # timeout for each datasource request, overrides `timeout` if set
   processing_timeout: '1m'  # timeout for preparing fetched series for fit/infer, overrides `timeout` if set
+  workers: 0  # automatic bounded datasource concurrency; set a positive value for an explicit cap
   queries:
     # one query returning 1 result fields (avg_duration), it will have __name__ label (series name) as `duration_30m__avg`
     duration_avg_30m:
       expr: "* | stats avg(duration) as avg"  # initial LogsQL expression
       step: '2m'  # overrides global `sampling_period` of 1m
-      data_range: [0, 'inf']  # meaning only positive values > 0 are expected, i.e. a value `y` < 0 will trigger anomaly score > 1
+      data_range: [0, 'inf']  # query-level business policy from v1.30.2; y < 0 triggers anomaly score > 1
+      detection_direction: 'above_expected'  # query-level from v1.30.2
+      min_rel_dev_from_expected: [0, 20]  # query-level from v1.30.2; ignore upward deviations below 20%
       tz: 'America/New_York'  # to override reader-wise `tz`
       # tenant_id: '1:0'  # overriding tenant_id to isolate data
       # offset: '-15s'  # to override reader-wise `offset` and query data 15 seconds earlier to account for data collection delays

docs/anomaly-detection/components/scheduler.md

@@ -1,5 +1,6 @@
 ---
 title: Scheduler
+description: "Scheduling configuration. Inference frequency and training time range."
 weight: 3
 menu:
   docs:
@@ -70,6 +71,7 @@ options={`"scheduler.periodic.PeriodicScheduler"`, `"scheduler.oneoff.OneoffSche
 
 ## Periodic scheduler 
 
+> [!WARNING]
 > If `start_from` [parameter](#parameters-1) is used, it's suggested to also set `restore_state: true` in the [Settings section](https://docs.victoriametrics.com/anomaly-detection/components/settings/#state-restoration) of a config, so that the scheduler can restore its state from the previous run **if terminated or restarted in between scheduled runs** and continue producing anomaly scores without interruptions, otherwise the service will be idle until future `start_from` time is reached. E.g. if `start_from` is set to `20:00` and the service is started and then terminated and restarted at `20:30`, it will not produce any anomaly scores until the next day's `20:00` is reached (+23:30 of being idle), which introduces inconvenience for the users.
 
 > {{% available_from "v1.30.0" anomaly %}} If a periodic scheduler worker exits unexpectedly, the service attempts bounded restarts with exponential backoff instead of shutting down unrelated schedulers. Monitor [`vmanomaly_scheduler_alive`](https://docs.victoriametrics.com/anomaly-detection/components/monitoring/#startup-metrics) and `vmanomaly_scheduler_restarts_total` to alert on persistent failures.
@@ -196,6 +198,7 @@ This configuration specifies that `vmanomaly` will calculate a 14-day time windo
 
 ## Oneoff scheduler
 
+> [!WARNING]
 > As of latest version, the Oneoff scheduler can't be explicitly used with a combination of [stateful service](https://docs.victoriametrics.com/anomaly-detection/components/settings/#state-restoration). It is designed to run once and exit, so it does not maintain state across runs. A warning will be raised in logs and internal state for such scheduler will not be saved and restored upon restart. If you need to run the scheduler periodically and/or maintain state, consider using the [Periodic scheduler](#periodic-scheduler) instead.
 
 ### Parameters
@@ -367,6 +370,7 @@ schedulers:
 
 > {{% available_from "v1.26.0" anomaly %}} `BacktestingScheduler` in [inference-only](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/#inference-only-mode) mode is used in UI for backtesting configurations on historical data to verify that it works as expected before it goes live. See [vmanomaly UI](https://docs.victoriametrics.com/anomaly-detection/ui/) on how to access and use the UI.
 
+> [!WARNING]
 > As of latest version, the Backtesting scheduler can't be explicitly used with a combination of [state restoration](https://docs.victoriametrics.com/anomaly-detection/components/settings/#state-restoration). It is designed to run once and exit, so it does not maintain state across runs. A warning will be raised in logs and internal state for such scheduler will not be saved and restored upon restart. If you need to run the scheduler periodically and/or maintain state, consider using the [Periodic scheduler](#periodic-scheduler) instead.
 
 > A new, more intuitive backtesting mode is available {{% available_from "v1.22.1" anomaly %}}. In **Inference only** mode, the window you specify via `[from, to]` (or `[from_iso, to_iso]`) is used *solely for inference*, and the corresponding training (“fit”) windows are determined automatically. To enable this behavior, set:

docs/anomaly-detection/components/server.md

@@ -1,5 +1,6 @@
 ---
 title: Server
+description: "HTTP server. REST API, /metrics endpoint, and web UI."
 weight: 7
 menu:
   docs: