include group first evaluation into interval ticker

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>

.github/copilot-instructions.md

@@ -1,23 +0,0 @@
-# Project Overview
-
-VictoriaMetrics is a fast, cost-saving, and scalable solution for monitoring and managing time series data. It delivers high performance and reliability, making it an ideal choice for businesses of all sizes.
-
-## Folder Structure
-
-- `/app`: Contains the compilable binaries.
-- `/lib`: Contains the golang reusable libraries
-- `/docs/victoriametrics`: Contains documentation for the project.
-- `/apptest/tests`: Contains integration tests.
-
-## Libraries and Frameworks
-
-- Backend: Golang, no framework. Use third-party libraries sparingly.
-- Frontend: React.
-
-## Code review guidelines
-
-Ensure the feature or bugfix includes a changelog entry in /docs/victoriametrics/changelog/CHANGELOG.md.
-Verify the entry is under the ## tip section and matches the structure and style of existing entries.
-Chore-only changes may be omitted from the changelog.
-
-

.github/workflows/check-commit-signed.yml

@@ -27,11 +27,21 @@ jobs:
             exit 0
           fi
           
-          unsigned=$(git log --pretty="%H %G?" $RANGE | grep -vE " (G|E)$" || true)
+          # Check raw commit objects for a "gpgsig" header as a fast early signal for
+          # contributors. Both GPG and SSH signatures use this header. 
+          # This avoids relying on %G? which returns N for SSH commits.
+          # This check is not a security enforcement — unsigned commits cannot be merged
+          # anyway due to the GitHub repository merge policy.
+          unsigned=""
+          for sha in $(git rev-list $RANGE); do
+            if ! git cat-file commit "$sha" | grep -q "^gpgsig"; then
+              unsigned="$unsigned $sha"
+            fi
+          done
           if [ -n "$unsigned" ]; then
             echo "Found unsigned commits:"
             echo "$unsigned"
             exit 1
           fi
-          
-          echo "All commits in PR are signed (G or E)"
+
+          echo "All commits in PR are signed (GPG or SSH)"

.github/workflows/docs.yaml

@@ -28,7 +28,7 @@ jobs:
           path: __vm-docs
 
       - name: Import GPG key
-        uses: crazy-max/ghaction-import-gpg@v6
+        uses: crazy-max/ghaction-import-gpg@v7
         id: import-gpg
         with:
           gpg_private_key: ${{ secrets.VM_BOT_GPG_PRIVATE_KEY }}

.github/workflows/test.yml

@@ -89,7 +89,7 @@ jobs:
         run: make ${{ matrix.scenario}}
 
       - name: Publish coverage
-        uses: codecov/codecov-action@v5
+        uses: codecov/codecov-action@v6
         with:
           files: ./coverage.txt
 

README.md

@@ -1,12 +1,12 @@
 # VictoriaMetrics
 
 [![Latest Release](https://img.shields.io/github/v/release/VictoriaMetrics/VictoriaMetrics?sort=semver&label=&filter=!*-victorialogs&logo=github&labelColor=gray&color=gray&link=https%3A%2F%2Fgithub.com%2FVictoriaMetrics%2FVictoriaMetrics%2Freleases%2Flatest)](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)
-![Docker Pulls](https://img.shields.io/docker/pulls/victoriametrics/victoria-metrics?label=&logo=docker&logoColor=white&labelColor=2496ED&color=2496ED&link=https%3A%2F%2Fhub.docker.com%2Fr%2Fvictoriametrics%2Fvictoria-metrics)
+[![Docker Pulls](https://img.shields.io/docker/pulls/victoriametrics/victoria-metrics?label=&logo=docker&logoColor=white&labelColor=2496ED&color=2496ED&link=https%3A%2F%2Fhub.docker.com%2Fr%2Fvictoriametrics%2Fvictoria-metrics)](https://hub.docker.com/u/victoriametrics)
 [![Go Report](https://goreportcard.com/badge/github.com/VictoriaMetrics/VictoriaMetrics?link=https%3A%2F%2Fgoreportcard.com%2Freport%2Fgithub.com%2FVictoriaMetrics%2FVictoriaMetrics)](https://goreportcard.com/report/github.com/VictoriaMetrics/VictoriaMetrics)
 [![Build Status](https://github.com/VictoriaMetrics/VictoriaMetrics/actions/workflows/build.yml/badge.svg?branch=master&link=https%3A%2F%2Fgithub.com%2FVictoriaMetrics%2FVictoriaMetrics%2Factions)](https://github.com/VictoriaMetrics/VictoriaMetrics/actions/workflows/build.yml)
 [![codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/branch/master/graph/badge.svg?link=https%3A%2F%2Fcodecov.io%2Fgh%2FVictoriaMetrics%2FVictoriaMetrics)](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics)
 [![License](https://img.shields.io/github/license/VictoriaMetrics/VictoriaMetrics?labelColor=green&label=&link=https%3A%2F%2Fgithub.com%2FVictoriaMetrics%2FVictoriaMetrics%2Fblob%2Fmaster%2FLICENSE)](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/LICENSE)
-![Slack](https://img.shields.io/badge/Join-4A154B?logo=slack&link=https%3A%2F%2Fslack.victoriametrics.com)
+[![Join Slack](https://img.shields.io/badge/Join%20Slack-4A154B?logo=slack)](https://slack.victoriametrics.com)
 [![X](https://img.shields.io/twitter/follow/VictoriaMetrics?style=flat&label=Follow&color=black&logo=x&labelColor=black&link=https%3A%2F%2Fx.com%2FVictoriaMetrics)](https://x.com/VictoriaMetrics/)
 [![Reddit](https://img.shields.io/reddit/subreddit-subscribers/VictoriaMetrics?style=flat&label=Join&labelColor=red&logoColor=white&logo=reddit&link=https%3A%2F%2Fwww.reddit.com%2Fr%2FVictoriaMetrics)](https://www.reddit.com/r/VictoriaMetrics/)
 

app/vmagent/datadogsketches/request_handler.go

@@ -49,6 +49,11 @@ func insertRows(at *auth.Token, sketches []*datadogsketches.Sketch, extraLabels
 				Name:  "__name__",
 				Value: m.Name,
 			})
+			// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10557
+			labels = append(labels, prompb.Label{
+				Name:  "host",
+				Value: sketch.Host,
+			})
 			for _, label := range m.Labels {
 				labels = append(labels, prompb.Label{
 					Name:  label.Name,
@@ -57,9 +62,6 @@ func insertRows(at *auth.Token, sketches []*datadogsketches.Sketch, extraLabels
 			}
 			for _, tag := range sketch.Tags {
 				name, value := datadogutil.SplitTag(tag)
-				if name == "host" {
-					name = "exported_host"
-				}
 				labels = append(labels, prompb.Label{
 					Name:  name,
 					Value: value,

app/vmalert/config/config.go

@@ -81,12 +81,9 @@ func (g *Group) Validate(validateTplFn ValidateTplFn, validateExpressions bool)
 	if g.Interval.Duration() < 0 {
 		return fmt.Errorf("interval shouldn't be lower than 0")
 	}
-	if g.EvalOffset.Duration() < 0 {
-		return fmt.Errorf("eval_offset shouldn't be lower than 0")
-	}
-	// if `eval_offset` is set, interval won't use global evaluationInterval flag and must bigger than offset.
-	if g.EvalOffset.Duration() > g.Interval.Duration() {
-		return fmt.Errorf("eval_offset should be smaller than interval; now eval_offset: %v, interval: %v", g.EvalOffset.Duration(), g.Interval.Duration())
+	// if `eval_offset` is set, the group interval must be specified explicitly(instead of inherited from global evaluationInterval flag) and must bigger than offset.
+	if g.EvalOffset.Duration().Abs() > g.Interval.Duration() {
+		return fmt.Errorf("the abs value of eval_offset should be smaller than interval; now eval_offset: %v, interval: %v", g.EvalOffset.Duration(), g.Interval.Duration())
 	}
 	if g.EvalOffset != nil && g.EvalDelay != nil {
 		return fmt.Errorf("eval_offset cannot be used with eval_delay")

app/vmalert/config/config_test.go

@@ -176,11 +176,17 @@ func TestGroupValidate_Failure(t *testing.T) {
 	}, false, "interval shouldn't be lower than 0")
 
 	f(&Group{
-		Name:       "wrong eval_offset",
+		Name:       "too big eval_offset",
 		Interval:   promutil.NewDuration(time.Minute),
 		EvalOffset: promutil.NewDuration(2 * time.Minute),
 	}, false, "eval_offset should be smaller than interval")
 
+	f(&Group{
+		Name:       "too big negative eval_offset",
+		Interval:   promutil.NewDuration(time.Minute),
+		EvalOffset: promutil.NewDuration(-2 * time.Minute),
+	}, false, "eval_offset should be smaller than interval")
+
 	limit := -1
 	f(&Group{
 		Name:  "wrong limit",

app/vmalert/main.go

@@ -56,7 +56,7 @@ absolute path to all .tpl files in root.
  -rule.templates="dir/**/*.tpl". Includes all the .tpl files in "dir" subfolders recursively.
 `)
 
-	configCheckInterval = flag.Duration("configCheckInterval", 0, "Interval for checking for changes in '-rule' or '-notifier.config' files. "+
+	configCheckInterval = flag.Duration("configCheckInterval", 0, "Interval for checking for changes in '-rule', '-rule.templates' and '-notifier.config' files. "+
 		"By default, the checking is disabled. Send SIGHUP signal in order to force config check for changes.")
 
 	httpListenAddrs  = flagutil.NewArrayString("httpListenAddr", "Address to listen for incoming http requests. See also -tls and -httpListenAddr.useProxyProtocol")

app/vmalert/manager.go

@@ -98,7 +98,7 @@ func (m *manager) close() {
 	m.wg.Wait()
 }
 
-func (m *manager) startGroup(ctx context.Context, g *rule.Group, restore bool) error {
+func (m *manager) startGroup(ctx context.Context, g *rule.Group, restore bool) {
 	id := g.GetID()
 	g.Init()
 	m.wg.Go(func() {
@@ -110,7 +110,6 @@ func (m *manager) startGroup(ctx context.Context, g *rule.Group, restore bool) e
 	})
 
 	m.groups[id] = g
-	return nil
 }
 
 func (m *manager) update(ctx context.Context, groupsCfg []config.Group, restore bool) error {
@@ -119,7 +118,7 @@ func (m *manager) update(ctx context.Context, groupsCfg []config.Group, restore
 	for _, cfg := range groupsCfg {
 		for _, r := range cfg.Rules {
 			if rrPresent && arPresent {
-				continue
+				break
 			}
 			if r.Record != "" {
 				rrPresent = true
@@ -162,10 +161,7 @@ func (m *manager) update(ctx context.Context, groupsCfg []config.Group, restore
 		}
 	}
 	for _, ng := range groupsRegistry {
-		if err := m.startGroup(ctx, ng, restore); err != nil {
-			m.groupsMu.Unlock()
-			return err
-		}
+		m.startGroup(ctx, ng, restore)
 	}
 	m.groupsMu.Unlock()
 

app/vmalert/remotewrite/client.go

@@ -186,6 +186,11 @@ func (c *Client) run(ctx context.Context) {
 				return
 			case <-ticker.C:
 				c.flush(ctx, wr)
+				// drain the potential stale tick to avoid small or empty flushes after a slow flush.
+				select {
+				case <-ticker.C:
+				default:
+				}
 			case ts, ok := <-c.input:
 				if !ok {
 					continue

app/vmalert/rule/alerting.go

@@ -789,16 +789,7 @@ func firingAlertStaleTimeSeries(ls map[string]string, timestamp int64) []prompb.
 
 // restore restores the value of ActiveAt field for active alerts,
 // based on previously written time series `alertForStateMetricName`.
-// Only rules with For > 0 can be restored.
 func (ar *AlertingRule) restore(ctx context.Context, q datasource.Querier, ts time.Time, lookback time.Duration) error {
-	if ar.For < 1 {
-		return nil
-	}
-
-	if len(ar.alerts) < 1 {
-		return nil
-	}
-
 	nameStr := fmt.Sprintf("%s=%q", alertNameLabel, ar.Name)
 	if !*disableAlertGroupLabel {
 		nameStr = fmt.Sprintf("%s=%q,%s=%q", alertGroupNameLabel, ar.GroupName, alertNameLabel, ar.Name)

app/vmalert/rule/group.go

@@ -8,6 +8,7 @@ import (
 	"hash/fnv"
 	"maps"
 	"net/url"
+	"os"
 	"sync"
 	"time"
 
@@ -213,6 +214,7 @@ func (g *Group) CreateID() uint64 {
 // restore restores alerts state for group rules
 func (g *Group) restore(ctx context.Context, qb datasource.QuerierBuilder, ts time.Time, lookback time.Duration) error {
 	for _, rule := range g.Rules {
+		// Only alerting rule with for > 0 and has active alerts from the first evaluation can be restored
 		ar, ok := rule.(*AlertingRule)
 		if !ok {
 			continue
@@ -220,6 +222,9 @@ func (g *Group) restore(ctx context.Context, qb datasource.QuerierBuilder, ts ti
 		if ar.For < 1 {
 			continue
 		}
+		if len(ar.alerts) < 1 {
+			return nil
+		}
 		q := qb.BuildWithParams(datasource.QuerierParams{
 			EvaluationInterval: g.Interval,
 			QueryParams:        g.Params,
@@ -333,6 +338,11 @@ func (g *Group) Init() {
 // Start starts group's evaluation
 func (g *Group) Start(ctx context.Context, rw remotewrite.RWClient, rr datasource.QuerierBuilder) {
 	defer func() { close(g.finishedCh) }()
+	e := &executor{
+		Rw:              rw,
+		notifierHeaders: g.NotifierHeaders,
+	}
+
 	evalTS := time.Now()
 	// sleep random duration to spread group rules evaluation
 	// over maxStartDelay to reduce the load on datasource.
@@ -367,11 +377,6 @@ func (g *Group) Start(ctx context.Context, rw remotewrite.RWClient, rr datasourc
 		evalTS = evalTS.Add(sleepBeforeStart)
 	}
 
-	e := &executor{
-		Rw:              rw,
-		notifierHeaders: g.NotifierHeaders,
-	}
-
 	g.infof("started")
 
 	eval := func(ctx context.Context, ts time.Time) time.Time {
@@ -381,7 +386,9 @@ func (g *Group) Start(ctx context.Context, rw remotewrite.RWClient, rr datasourc
 
 		if len(g.Rules) < 1 {
 			g.metrics.iterationDuration.UpdateDuration(start)
+			g.mu.Lock()
 			g.LastEvaluation = start
+			g.mu.Unlock()
 			return ts
 		}
 
@@ -395,7 +402,32 @@ func (g *Group) Start(ctx context.Context, rw remotewrite.RWClient, rr datasourc
 			}
 		}
 		g.metrics.iterationDuration.UpdateDuration(start)
+		g.mu.Lock()
 		g.LastEvaluation = start
+		g.mu.Unlock()
+		if g.EvalOffset != nil && e.Rw != nil {
+			hostname, err := os.Hostname()
+			if err != nil {
+				hostname = "unknown"
+			}
+			labels := map[string]string{
+				"__name__": "vmalert_eval_timestamp",
+				"host":     hostname,
+				"group":    g.Name,
+				"file":     g.File,
+			}
+			var ls []prompb.Label
+			for k, v := range labels {
+				ls = append(ls, prompb.Label{
+					Name:  k,
+					Value: v,
+				})
+			}
+			ts := newTimeSeries([]float64{float64(ts.Unix())}, []int64{start.Unix()}, ls)
+			if err := e.Rw.Push(ts); err != nil {
+				logger.Errorf("group %q: failed to push evaluation timestamp: %s", g.Name, err)
+			}
+		}
 		return ts
 	}
 
@@ -405,11 +437,11 @@ func (g *Group) Start(ctx context.Context, rw remotewrite.RWClient, rr datasourc
 	g.mu.Unlock()
 	defer g.evalCancel()
 
-	realEvalTS := eval(evalCtx, evalTS)
-
 	t := time.NewTicker(g.Interval)
 	defer t.Stop()
 
+	realEvalTS := eval(evalCtx, evalTS)
+
 	// restore the rules state after the first evaluation
 	// so only active alerts can be restored.
 	if rr != nil {
@@ -484,8 +516,15 @@ func (g *Group) UpdateWith(newGroup *Group) {
 // delayBeforeStart calculates delay based on Group ID, so all groups will start at different moments of time.
 func (g *Group) delayBeforeStart(ts time.Time, maxDelay time.Duration) time.Duration {
 	if g.EvalOffset != nil {
+		offset := *g.EvalOffset
+		// adjust the offset for negative evalOffset, the rule is:
+		// `eval_offset: -x` is equivalent to `eval_offset: y` for `interval: x+y`.
+		// For example, `eval_offset: -6m` is equivalent to `eval_offset: 4m` for `interval: 10m`.
+		if offset < 0 {
+			offset += g.Interval
+		}
 		// if offset is specified, ignore the maxDelay and return a duration aligned with offset
-		currentOffsetPoint := ts.Truncate(g.Interval).Add(*g.EvalOffset)
+		currentOffsetPoint := ts.Truncate(g.Interval).Add(offset)
 		if currentOffsetPoint.Before(ts) {
 			// wait until the next offset point
 			return currentOffsetPoint.Add(g.Interval).Sub(ts)

app/vmalert/rule/group_test.go

@@ -606,6 +606,15 @@ func TestGroupStartDelay(t *testing.T) {
 	f("2023-01-01T00:03:30.000+00:00", "2023-01-01T00:08:00.000+00:00")
 	f("2023-01-01T00:08:00.000+00:00", "2023-01-01T00:08:00.000+00:00")
 
+	// test group with negative offset -2min, which is equivalent to 3min offset for 5min interval
+	offset = -2 * time.Minute
+	g.EvalOffset = &offset
+
+	f("2023-01-01T00:00:15.000+00:00", "2023-01-01T00:03:00.000+00:00")
+	f("2023-01-01T00:01:00.000+00:00", "2023-01-01T00:03:00.000+00:00")
+	f("2023-01-01T00:03:30.000+00:00", "2023-01-01T00:08:00.000+00:00")
+	f("2023-01-01T00:08:00.000+00:00", "2023-01-01T00:08:00.000+00:00")
+
 	maxDelay = time.Minute * 1
 	g.EvalOffset = nil
 

app/vmalert/rule/web.go

@@ -57,12 +57,8 @@ type ApiGroup struct {
 	EvalOffset float64 `json:"eval_offset,omitempty"`
 	// EvalDelay will adjust the `time` parameter of rule evaluation requests to compensate intentional query delay from datasource.
 	EvalDelay float64 `json:"eval_delay,omitempty"`
-	// Unhealthy unhealthy rules count
-	Unhealthy int
-	// Healthy passing rules count
-	Healthy int
-	// NoMatch not matching rules count
-	NoMatch int
+	// States represents counts per each rule state
+	States map[string]int `json:"states"`
 }
 
 // APILink returns a link to the group's JSON representation.
@@ -134,6 +130,11 @@ type ApiRule struct {
 	Updates []StateEntry `json:"-"`
 }
 
+// IsNoMatch returns true if rule is in nomatch state
+func (r *ApiRule) IsNoMatch() bool {
+	return r.LastSamples == 0 && r.LastSeriesFetched != nil && *r.LastSeriesFetched == 0
+}
+
 // ApiAlert represents a notifier.AlertingRule state
 // for WEB view
 // https://github.com/prometheus/compliance/blob/main/alert_generator/specification.md#get-apiv1rules
@@ -235,6 +236,20 @@ func NewAlertAPI(ar *AlertingRule, a *notifier.Alert) *ApiAlert {
 	return aa
 }
 
+func (r *ApiRule) ExtendState() {
+	if len(r.Alerts) > 0 {
+		return
+	}
+	if r.State == "" {
+		r.State = "ok"
+	}
+	if r.Health != "ok" {
+		r.State = "unhealthy"
+	} else if r.IsNoMatch() {
+		r.State = "nomatch"
+	}
+}
+
 // ToAPI returns ApiGroup representation of g
 func (g *Group) ToAPI() *ApiGroup {
 	g.mu.RLock()
@@ -252,6 +267,7 @@ func (g *Group) ToAPI() *ApiGroup {
 		Headers:         headersToStrings(g.Headers),
 		NotifierHeaders: headersToStrings(g.NotifierHeaders),
 		Labels:          g.Labels,
+		States:          make(map[string]int),
 	}
 	if g.EvalOffset != nil {
 		ag.EvalOffset = g.EvalOffset.Seconds()
@@ -259,9 +275,10 @@ func (g *Group) ToAPI() *ApiGroup {
 	if g.EvalDelay != nil {
 		ag.EvalDelay = g.EvalDelay.Seconds()
 	}
-	ag.Rules = make([]ApiRule, 0)
+	ag.Rules = make([]ApiRule, 0, len(g.Rules))
 	for _, r := range g.Rules {
-		ag.Rules = append(ag.Rules, r.ToAPI())
+		ar := r.ToAPI()
+		ag.Rules = append(ag.Rules, ar)
 	}
 	return &ag
 }

app/vmalert/static/icons/icons.svg

@@ -11,7 +11,7 @@
     <path d="M224.163 175.27a1.9 1.9 0 0 0 2.8 0l6-5.9a2.1 2.1 0 0 0 .2-2.7 1.9 1.9 0 0 0-3-.2l-2.6 2.6v-5.2c0-1.54-1.667-2.502-3-1.732-.619.357-1 1.017-1 1.732v5.2l-2.6-2.6a1.9 1.9 0 0 0-3 .2 2.1 2.1 0 0 0 .2 2.7zm-16.459-23.297h36c1.54 0 2.502-1.667 1.732-3a2 2 0 0 0-1.732-1h-36c-1.54 0-2.502 1.667-1.732 3 .357.619 1.017 1 1.732 1m36 4h-36c-1.54 0-2.502 1.667-1.732 3 .357.619 1.017 1 1.732 1h36c1.54 0 2.502-1.667 1.732-3a2 2 0 0 0-1.732-1m-16.59-23.517a1.9 1.9 0 0 0-2.8 0l-6 5.9a2.1 2.1 0 0 0-.2 2.7 1.9 1.9 0 0 0 3 .2l2.6-2.6v5.2c0 1.54 1.667 2.502 3 1.732.619-.357 1-1.017 1-1.732v-5.2l2.6 2.6a1.9 1.9 0 0 0 3-.2 2.1 2.1 0 0 0-.2-2.7z"/>
   </symbol>
 
-  <symbol id="filter" viewBox="-10 -10 320 310">
+  <symbol id="state" viewBox="-10 -10 320 310">
     <path d="M288.953 0h-277c-5.522 0-10 4.478-10 10v49.531c0 5.522 4.478 10 10 10h12.372l91.378 107.397v113.978a10 10 0 0 0 15.547 8.32l49.5-33a10 10 0 0 0 4.453-8.32v-80.978l91.378-107.397h12.372c5.522 0 10-4.478 10-10V10c0-5.522-4.477-10-10-10M167.587 166.77a10 10 0 0 0-2.384 6.48v79.305l-29.5 19.666V173.25a10 10 0 0 0-2.384-6.48L50.585 69.531h199.736zM278.953 49.531h-257V20h257z"/>
   </symbol>
 

app/vmalert/static/js/custom.js

@@ -8,9 +8,9 @@ function actionAll(isCollapse) {
     });
 }
 
-function groupFilter(key) {
+function groupForState(key) {
     if (key) {
-        location.href = `?filter=${key}`;
+        location.href = `?state=${key}`;
     } else {
         window.location = window.location.pathname;
     }

app/vmalert/web.go

@@ -1,9 +1,11 @@
 package main
 
 import (
+	"cmp"
 	"embed"
 	"encoding/json"
 	"fmt"
+	"math"
 	"net/http"
 	"slices"
 	"strconv"
@@ -50,6 +52,7 @@ var (
 		"alert":  rule.TypeAlerting,
 		"record": rule.TypeRecording,
 	}
+	ruleStates = []string{"ok", "nomatch", "inactive", "firing", "pending", "unhealthy"}
 )
 
 type requestHandler struct {
@@ -63,6 +66,14 @@ var (
 	staticServer  = http.StripPrefix("/vmalert", staticHandler)
 )
 
+func marshalJson(v any, kind string) ([]byte, *httpserver.ErrorWithStatusCode) {
+	data, err := json.Marshal(v)
+	if err != nil {
+		return nil, errResponse(fmt.Errorf("failed to marshal %s: %s", kind, err), http.StatusInternalServerError)
+	}
+	return data, nil
+}
+
 func (rh *requestHandler) handler(w http.ResponseWriter, r *http.Request) bool {
 	if strings.HasPrefix(r.URL.Path, "/vmalert/static") {
 		staticServer.ServeHTTP(w, r)
@@ -94,40 +105,32 @@ func (rh *requestHandler) handler(w http.ResponseWriter, r *http.Request) bool {
 			httpserver.Errorf(w, r, "%s", err)
 			return true
 		}
-		WriteRuleDetails(w, r, rule)
+		WriteRule(w, r, rule)
 		return true
-	case "/vmalert/groups":
+	// current used by old vmalert UI and Grafana Alerts
+	case "/vmalert/groups", "/rules":
 		rf, err := newRulesFilter(r)
 		if err != nil {
 			httpserver.Errorf(w, r, "%s", err)
 			return true
 		}
-		data := rh.groups(rf)
-		WriteListGroups(w, r, data, rf.filter)
+		// only support filtering by a single state
+		state := ""
+		if len(rf.states) > 0 {
+			state = rf.states[0]
+			rf.states = rf.states[:1]
+		}
+		lr := rh.groups(rf)
+		WriteListGroups(w, r, lr.Data.Groups, state)
 		return true
 	case "/vmalert/notifiers":
 		WriteListTargets(w, r, notifier.GetTargets())
 		return true
 
-	// special cases for Grafana requests,
-	// served without `vmalert` prefix:
-	case "/rules":
-		// Grafana makes an extra request to `/rules`
-		// handler in addition to `/api/v1/rules` calls in alerts UI
-		var data []*rule.ApiGroup
-		rf, err := newRulesFilter(r)
-		if err != nil {
-			httpserver.Errorf(w, r, "%s", err)
-			return true
-		}
-		data = rh.groups(rf)
-		WriteListGroups(w, r, data, rf.filter)
-		return true
-
 	case "/vmalert/api/v1/notifiers", "/api/v1/notifiers":
 		data, err := rh.listNotifiers()
 		if err != nil {
-			httpserver.Errorf(w, r, "%s", err)
+			errJson(w, r, err)
 			return true
 		}
 		w.Header().Set("Content-Type", "application/json")
@@ -135,15 +138,14 @@ func (rh *requestHandler) handler(w http.ResponseWriter, r *http.Request) bool {
 		return true
 	case "/vmalert/api/v1/rules", "/api/v1/rules":
 		// path used by Grafana for ng alerting
-		var data []byte
 		rf, err := newRulesFilter(r)
 		if err != nil {
-			httpserver.Errorf(w, r, "%s", err)
+			errJson(w, r, err)
 			return true
 		}
-		data, err = rh.listGroups(rf)
+		data, err := rh.listGroups(rf)
 		if err != nil {
-			httpserver.Errorf(w, r, "%s", err)
+			errJson(w, r, err)
 			return true
 		}
 		w.Header().Set("Content-Type", "application/json")
@@ -152,14 +154,14 @@ func (rh *requestHandler) handler(w http.ResponseWriter, r *http.Request) bool {
 
 	case "/vmalert/api/v1/alerts", "/api/v1/alerts":
 		// path used by Grafana for ng alerting
-		rf, err := newRulesFilter(r)
+		gf, err := newGroupsFilter(r)
 		if err != nil {
-			httpserver.Errorf(w, r, "%s", err)
+			errJson(w, r, err)
 			return true
 		}
-		data, err := rh.listAlerts(rf)
+		data, err := rh.listAlerts(gf)
 		if err != nil {
-			httpserver.Errorf(w, r, "%s", err)
+			errJson(w, r, err)
 			return true
 		}
 		w.Header().Set("Content-Type", "application/json")
@@ -168,12 +170,12 @@ func (rh *requestHandler) handler(w http.ResponseWriter, r *http.Request) bool {
 	case "/vmalert/api/v1/alert", "/api/v1/alert":
 		alert, err := rh.getAlert(r)
 		if err != nil {
-			httpserver.Errorf(w, r, "%s", err)
+			errJson(w, r, err)
 			return true
 		}
-		data, err := json.Marshal(alert)
+		data, err := marshalJson(alert, "alert")
 		if err != nil {
-			httpserver.Errorf(w, r, "failed to marshal alert: %s", err)
+			errJson(w, r, err)
 			return true
 		}
 		w.Header().Set("Content-Type", "application/json")
@@ -182,16 +184,16 @@ func (rh *requestHandler) handler(w http.ResponseWriter, r *http.Request) bool {
 	case "/vmalert/api/v1/rule", "/api/v1/rule":
 		apiRule, err := rh.getRule(r)
 		if err != nil {
-			httpserver.Errorf(w, r, "%s", err)
+			errJson(w, r, err)
 			return true
 		}
 		rwu := rule.ApiRuleWithUpdates{
 			ApiRule:      apiRule,
 			StateUpdates: apiRule.Updates,
 		}
-		data, err := json.Marshal(rwu)
+		data, err := marshalJson(rwu, "rule")
 		if err != nil {
-			httpserver.Errorf(w, r, "failed to marshal rule: %s", err)
+			errJson(w, r, err)
 			return true
 		}
 		w.Header().Set("Content-Type", "application/json")
@@ -200,12 +202,12 @@ func (rh *requestHandler) handler(w http.ResponseWriter, r *http.Request) bool {
 	case "/vmalert/api/v1/group", "/api/v1/group":
 		group, err := rh.getGroup(r)
 		if err != nil {
-			httpserver.Errorf(w, r, "%s", err)
+			errJson(w, r, err)
 			return true
 		}
-		data, err := json.Marshal(group)
+		data, err := marshalJson(group, "group")
 		if err != nil {
-			httpserver.Errorf(w, r, "failed to marshal group: %s", err)
+			errJson(w, r, err)
 			return true
 		}
 		w.Header().Set("Content-Type", "application/json")
@@ -225,10 +227,10 @@ func (rh *requestHandler) handler(w http.ResponseWriter, r *http.Request) bool {
 	}
 }
 
-func (rh *requestHandler) getGroup(r *http.Request) (*rule.ApiGroup, error) {
+func (rh *requestHandler) getGroup(r *http.Request) (*rule.ApiGroup, *httpserver.ErrorWithStatusCode) {
 	groupID, err := strconv.ParseUint(r.FormValue(rule.ParamGroupID), 10, 64)
 	if err != nil {
-		return nil, fmt.Errorf("failed to read %q param: %w", rule.ParamGroupID, err)
+		return nil, errResponse(fmt.Errorf("failed to read %q param: %w", rule.ParamGroupID, err), http.StatusBadRequest)
 	}
 	obj, err := rh.m.groupAPI(groupID)
 	if err != nil {
@@ -237,14 +239,14 @@ func (rh *requestHandler) getGroup(r *http.Request) (*rule.ApiGroup, error) {
 	return obj, nil
 }
 
-func (rh *requestHandler) getRule(r *http.Request) (rule.ApiRule, error) {
+func (rh *requestHandler) getRule(r *http.Request) (rule.ApiRule, *httpserver.ErrorWithStatusCode) {
 	groupID, err := strconv.ParseUint(r.FormValue(rule.ParamGroupID), 10, 64)
 	if err != nil {
-		return rule.ApiRule{}, fmt.Errorf("failed to read %q param: %w", rule.ParamGroupID, err)
+		return rule.ApiRule{}, errResponse(fmt.Errorf("failed to read %q param: %w", rule.ParamGroupID, err), http.StatusBadRequest)
 	}
 	ruleID, err := strconv.ParseUint(r.FormValue(rule.ParamRuleID), 10, 64)
 	if err != nil {
-		return rule.ApiRule{}, fmt.Errorf("failed to read %q param: %w", rule.ParamRuleID, err)
+		return rule.ApiRule{}, errResponse(fmt.Errorf("failed to read %q param: %w", rule.ParamRuleID, err), http.StatusBadRequest)
 	}
 	obj, err := rh.m.ruleAPI(groupID, ruleID)
 	if err != nil {
@@ -253,14 +255,14 @@ func (rh *requestHandler) getRule(r *http.Request) (rule.ApiRule, error) {
 	return obj, nil
 }
 
-func (rh *requestHandler) getAlert(r *http.Request) (*rule.ApiAlert, error) {
+func (rh *requestHandler) getAlert(r *http.Request) (*rule.ApiAlert, *httpserver.ErrorWithStatusCode) {
 	groupID, err := strconv.ParseUint(r.FormValue(rule.ParamGroupID), 10, 64)
 	if err != nil {
-		return nil, fmt.Errorf("failed to read %q param: %w", rule.ParamGroupID, err)
+		return nil, errResponse(fmt.Errorf("failed to read %q param: %w", rule.ParamGroupID, err), http.StatusBadRequest)
 	}
 	alertID, err := strconv.ParseUint(r.FormValue(rule.ParamAlertID), 10, 64)
 	if err != nil {
-		return nil, fmt.Errorf("failed to read %q param: %w", rule.ParamAlertID, err)
+		return nil, errResponse(fmt.Errorf("failed to read %q param: %w", rule.ParamAlertID, err), http.StatusBadRequest)
 	}
 	a, err := rh.m.alertAPI(groupID, alertID)
 	if err != nil {
@@ -270,28 +272,76 @@ func (rh *requestHandler) getAlert(r *http.Request) (*rule.ApiAlert, error) {
 }
 
 type listGroupsResponse struct {
-	Status string `json:"status"`
-	Data   struct {
+	Status      string `json:"status"`
+	Page        int    `json:"page,omitempty"`
+	TotalPages  int    `json:"total_pages,omitempty"`
+	TotalGroups int    `json:"total_groups,omitempty"`
+	TotalRules  int    `json:"total_rules,omitempty"`
+	Data        struct {
 		Groups []*rule.ApiGroup `json:"groups"`
 	} `json:"data"`
 }
 
-// see https://prometheus.io/docs/prometheus/latest/querying/api/#rules
-type rulesFilter struct {
-	files         []string
-	groupNames    []string
-	ruleNames     []string
-	ruleType      string
-	excludeAlerts bool
-	filter        string
-	dsType        config.Type
+type groupsFilter struct {
+	groupNames []string
+	files      []string
+	dsType     config.Type
 }
 
-func newRulesFilter(r *http.Request) (*rulesFilter, error) {
-	rf := &rulesFilter{}
-	query := r.URL.Query()
+func newGroupsFilter(r *http.Request) (*groupsFilter, *httpserver.ErrorWithStatusCode) {
+	_ = r.ParseForm()
+	vs := r.Form
+	gf := &groupsFilter{
+		groupNames: vs["rule_group[]"],
+		files:      vs["file[]"],
+	}
+	dsType := vs.Get("datasource_type")
+	if len(dsType) > 0 {
+		if config.SupportedType(dsType) {
+			gf.dsType = config.NewRawType(dsType)
+		} else {
+			return nil, errResponse(fmt.Errorf(`invalid parameter "datasource_type": not supported value %q`, dsType), http.StatusBadRequest)
+		}
+	}
+	return gf, nil
+}
 
-	ruleTypeParam := query.Get("type")
+func (gf *groupsFilter) matches(group *rule.Group) bool {
+	if len(gf.groupNames) > 0 && !slices.Contains(gf.groupNames, group.Name) {
+		return false
+	}
+	if len(gf.files) > 0 && !slices.Contains(gf.files, group.File) {
+		return false
+	}
+	if len(gf.dsType.Name) > 0 && gf.dsType.String() != group.Type.String() {
+		return false
+	}
+	return true
+}
+
+// see https://prometheus.io/docs/prometheus/latest/querying/api/#rules
+type rulesFilter struct {
+	gf             *groupsFilter
+	ruleNames      []string
+	ruleType       string
+	excludeAlerts  bool
+	states         []string
+	maxGroups      int
+	pageNum        int
+	search         string
+	extendedStates bool
+}
+
+func newRulesFilter(r *http.Request) (*rulesFilter, *httpserver.ErrorWithStatusCode) {
+	gf, err := newGroupsFilter(r)
+	if err != nil {
+		return nil, err
+	}
+
+	var rf rulesFilter
+	rf.gf = gf
+	vs := r.Form
+	ruleTypeParam := vs.Get("type")
 	if len(ruleTypeParam) > 0 {
 		if ruleType, ok := ruleTypeMap[ruleTypeParam]; ok {
 			rf.ruleType = ruleType
@@ -300,102 +350,146 @@ func newRulesFilter(r *http.Request) (*rulesFilter, error) {
 		}
 	}
 
-	dsType := query.Get("datasource_type")
-	if len(dsType) > 0 {
-		if config.SupportedType(dsType) {
-			rf.dsType = config.NewRawType(dsType)
-		} else {
-			return nil, errResponse(fmt.Errorf(`invalid parameter "datasource_type": not supported value %q`, dsType), http.StatusBadRequest)
-		}
+	states := vs["state"]
+	if len(states) == 0 {
+		states = vs["filter"]
 	}
-
-	filter := strings.ToLower(query.Get("filter"))
-	if len(filter) > 0 {
-		if filter == "nomatch" || filter == "unhealthy" {
-			rf.filter = filter
-		} else {
-			return nil, errResponse(fmt.Errorf(`invalid parameter "filter": not supported value %q`, filter), http.StatusBadRequest)
+	for _, s := range states {
+		values := strings.Split(s, ",")
+		for _, v := range values {
+			if len(v) == 0 {
+				continue
+			}
+			if !slices.Contains(ruleStates, v) {
+				return nil, errResponse(fmt.Errorf(`invalid parameter "state": contains not supported value %q`, v), http.StatusBadRequest)
+			}
+			rf.states = append(rf.states, v)
 		}
 	}
 
 	rf.excludeAlerts = httputil.GetBool(r, "exclude_alerts")
-	rf.ruleNames = append([]string{}, r.Form["rule_name[]"]...)
-	rf.groupNames = append([]string{}, r.Form["rule_group[]"]...)
-	rf.files = append([]string{}, r.Form["file[]"]...)
-	return rf, nil
+	rf.extendedStates = httputil.GetBool(r, "extended_states")
+	rf.ruleNames = append([]string{}, vs["rule_name[]"]...)
+	rf.search = strings.ToLower(vs.Get("search"))
+
+	pageNum := vs.Get("page_num")
+	maxGroups := vs.Get("group_limit")
+	if pageNum != "" {
+		if maxGroups == "" {
+			return nil, errResponse(fmt.Errorf(`"group_limit" needs to be present in order to paginate over the groups`), http.StatusBadRequest)
+		}
+		v, err := strconv.Atoi(pageNum)
+		if err != nil || v <= 0 {
+			return nil, errResponse(fmt.Errorf(`"page_num" is expected to be a positive number, found %q`, pageNum), http.StatusBadRequest)
+		}
+		rf.pageNum = v
+	}
+	if maxGroups != "" {
+		v, err := strconv.Atoi(maxGroups)
+		if err != nil || v <= 0 {
+			return nil, errResponse(fmt.Errorf(`"group_limit" is expected to be a positive number, found %q`, maxGroups), http.StatusBadRequest)
+		}
+		rf.maxGroups = v
+	}
+	return &rf, nil
 }
 
-func (rf *rulesFilter) matchesGroup(group *rule.Group) bool {
-	if len(rf.groupNames) > 0 && !slices.Contains(rf.groupNames, group.Name) {
+func (rf *rulesFilter) matchesRule(r *rule.ApiRule) bool {
+	if rf.ruleType != "" && rf.ruleType != r.Type {
 		return false
 	}
-	if len(rf.files) > 0 && !slices.Contains(rf.files, group.File) {
+	if len(rf.ruleNames) > 0 && !slices.Contains(rf.ruleNames, r.Name) {
 		return false
 	}
-	if len(rf.dsType.Name) > 0 && rf.dsType.String() != group.Type.String() {
-		return false
+	if len(rf.states) == 0 {
+		return true
 	}
-	return true
+	return slices.Contains(rf.states, r.State)
 }
 
-func (rh *requestHandler) groups(rf *rulesFilter) []*rule.ApiGroup {
+func (rh *requestHandler) groups(rf *rulesFilter) *listGroupsResponse {
 	rh.m.groupsMu.RLock()
 	defer rh.m.groupsMu.RUnlock()
 
-	groups := make([]*rule.ApiGroup, 0)
+	skipGroups := (rf.pageNum - 1) * rf.maxGroups
+	lr := &listGroupsResponse{
+		Status: "success",
+	}
+	lr.Data.Groups = make([]*rule.ApiGroup, 0)
+	if skipGroups >= len(rh.m.groups) {
+		return lr
+	}
+	// sort list of groups for deterministic output
+	groups := make([]*rule.Group, 0, len(rh.m.groups))
 	for _, group := range rh.m.groups {
-		if !rf.matchesGroup(group) {
+		groups = append(groups, group)
+	}
+
+	slices.SortFunc(groups, func(a, b *rule.Group) int {
+		nameCmp := cmp.Compare(a.Name, b.Name)
+		if nameCmp != 0 {
+			return nameCmp
+		}
+		return cmp.Compare(a.File, b.File)
+	})
+	for _, group := range groups {
+		if !rf.gf.matches(group) {
 			continue
 		}
+		groupFound := len(rf.search) == 0 || strings.Contains(strings.ToLower(group.Name), rf.search) || strings.Contains(strings.ToLower(group.File), rf.search)
 		g := group.ToAPI()
 		// the returned list should always be non-nil
 		// https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4221
 		filteredRules := make([]rule.ApiRule, 0)
 		for _, rule := range g.Rules {
-			if rf.ruleType != "" && rf.ruleType != rule.Type {
+			if !groupFound && !strings.Contains(strings.ToLower(rule.Name), rf.search) {
 				continue
 			}
-			if len(rf.ruleNames) > 0 && !slices.Contains(rf.ruleNames, rule.Name) {
-				continue
+			if rf.extendedStates {
+				rule.ExtendState()
 			}
-			if (rule.LastError == "" && rf.filter == "unhealthy") || (!isNoMatch(rule) && rf.filter == "nomatch") {
+			if !rf.matchesRule(&rule) {
 				continue
 			}
 			if rf.excludeAlerts {
 				rule.Alerts = nil
 			}
-			if rule.LastError != "" {
-				g.Unhealthy++
-			} else {
-				g.Healthy++
-			}
-			if isNoMatch(rule) {
-				g.NoMatch++
-			}
+			g.States[rule.State]++
 			filteredRules = append(filteredRules, rule)
 		}
-		g.Rules = filteredRules
-		groups = append(groups, g)
-	}
-	// sort list of groups for deterministic output
-	slices.SortFunc(groups, func(a, b *rule.ApiGroup) int {
-		if a.Name != b.Name {
-			return strings.Compare(a.Name, b.Name)
+		if len(g.Rules) == 0 || len(filteredRules) > 0 {
+			if rf.maxGroups > 0 {
+				lr.TotalGroups++
+				lr.TotalRules += len(filteredRules)
+			}
+			if skipGroups > 0 {
+				skipGroups--
+				continue
+			}
+			if rf.maxGroups == 0 || len(lr.Data.Groups) < rf.maxGroups {
+				g.Rules = filteredRules
+				lr.Data.Groups = append(lr.Data.Groups, g)
+			}
 		}
-		return strings.Compare(a.File, b.File)
-	})
-	return groups
+	}
+	if rf.maxGroups > 0 {
+		lr.Page = rf.pageNum
+		lr.TotalPages = max(int(math.Ceil(float64(lr.TotalGroups)/float64(rf.maxGroups))), 1)
+	}
+	return lr
 }
 
-func (rh *requestHandler) listGroups(rf *rulesFilter) ([]byte, error) {
-	lr := listGroupsResponse{Status: "success"}
-	lr.Data.Groups = rh.groups(rf)
+func (rh *requestHandler) listGroups(rf *rulesFilter) ([]byte, *httpserver.ErrorWithStatusCode) {
+	lr := rh.groups(rf)
+	if rf.pageNum > 1 && len(lr.Data.Groups) == 0 {
+		return nil, errResponse(fmt.Errorf(`page_num exceeds total amount of pages`), http.StatusBadRequest)
+	}
+	if lr.Page > lr.TotalPages {
+		return nil, errResponse(fmt.Errorf(`page_num=%d exceeds total amount of pages in result=%d`, lr.Page, lr.TotalPages), http.StatusBadRequest)
+	}
 	b, err := json.Marshal(lr)
 	if err != nil {
-		return nil, &httpserver.ErrorWithStatusCode{
-			Err:        fmt.Errorf(`error encoding list of active alerts: %w`, err),
-			StatusCode: http.StatusInternalServerError,
-		}
+		return nil, errResponse(fmt.Errorf(`error encoding list of groups: %w`, err), http.StatusInternalServerError)
 	}
 	return b, nil
 }
@@ -434,14 +528,14 @@ func (rh *requestHandler) groupAlerts() []rule.GroupAlerts {
 	return gAlerts
 }
 
-func (rh *requestHandler) listAlerts(rf *rulesFilter) ([]byte, error) {
+func (rh *requestHandler) listAlerts(gf *groupsFilter) ([]byte, *httpserver.ErrorWithStatusCode) {
 	rh.m.groupsMu.RLock()
 	defer rh.m.groupsMu.RUnlock()
 
 	lr := listAlertsResponse{Status: "success"}
 	lr.Data.Alerts = make([]*rule.ApiAlert, 0)
 	for _, group := range rh.m.groups {
-		if !rf.matchesGroup(group) {
+		if !gf.matches(group) {
 			continue
 		}
 		g := group.ToAPI()
@@ -460,10 +554,7 @@ func (rh *requestHandler) listAlerts(rf *rulesFilter) ([]byte, error) {
 
 	b, err := json.Marshal(lr)
 	if err != nil {
-		return nil, &httpserver.ErrorWithStatusCode{
-			Err:        fmt.Errorf(`error encoding list of active alerts: %w`, err),
-			StatusCode: http.StatusInternalServerError,
-		}
+		return nil, errResponse(fmt.Errorf(`error encoding list of active alerts: %w`, err), http.StatusInternalServerError)
 	}
 	return b, nil
 }
@@ -475,7 +566,7 @@ type listNotifiersResponse struct {
 	} `json:"data"`
 }
 
-func (rh *requestHandler) listNotifiers() ([]byte, error) {
+func (rh *requestHandler) listNotifiers() ([]byte, *httpserver.ErrorWithStatusCode) {
 	targets := notifier.GetTargets()
 
 	lr := listNotifiersResponse{Status: "success"}
@@ -497,10 +588,7 @@ func (rh *requestHandler) listNotifiers() ([]byte, error) {
 
 	b, err := json.Marshal(lr)
 	if err != nil {
-		return nil, &httpserver.ErrorWithStatusCode{
-			Err:        fmt.Errorf(`error encoding list of notifiers: %w`, err),
-			StatusCode: http.StatusInternalServerError,
-		}
+		return nil, errResponse(fmt.Errorf(`error encoding list of notifiers: %w`, err), http.StatusInternalServerError)
 	}
 	return b, nil
 }
@@ -511,3 +599,8 @@ func errResponse(err error, sc int) *httpserver.ErrorWithStatusCode {
 		StatusCode: sc,
 	}
 }
+
+func errJson(w http.ResponseWriter, r *http.Request, err *httpserver.ErrorWithStatusCode) {
+	w.Header().Set("Content-Type", "application/json")
+	httpserver.Errorf(w, r, `{"error":%q,"errorType":%d}`, err, err.StatusCode)
+}

app/vmalert/web.qtpl

@@ -12,7 +12,7 @@
     "github.com/VictoriaMetrics/VictoriaMetrics/lib/buildinfo"
 ) %}
 
-{% func Controls(prefix, currentIcon, currentText string, icons, filters map[string]string, search bool) %}
+{% func Controls(prefix, currentIcon, currentText string, icons, states map[string]string, search bool) %}
     <div class="btn-toolbar mb-3" role="toolbar">
         <div class="d-flex gap-2 justify-content-between w-100">
             <div class="d-flex gap-2 align-items-center">
@@ -28,10 +28,10 @@
                         <use href="{%s prefix %}static/icons/icons.svg#expand"/>
                     </svg>
                 </a>
-                {% if len(filters) > 0 %}
+                {% if len(states) > 0 %}
                     <span class="d-none d-md-inline-block">Filter by status:</span>
                     <svg class="d-md-none" width="20" height="20">
-                        <use href="{%s prefix %}static/icons/icons.svg#filter">
+                        <use href="{%s prefix %}static/icons/icons.svg#state">
                     </svg>
                     <div class="dropdown">
                         <button
@@ -46,10 +46,10 @@
                             </svg>
                         </button>
                         <ul class="dropdown-menu">
-                            {% for key, title := range filters %}
+                            {% for key, title := range states %}
                                 {% if title != currentText %}
                                     <li>
-                                        <a class="dropdown-item" onclick="groupFilter('{%s key %}')">
+                                        <a class="dropdown-item" onclick="groupForState('{%s key %}')">
                                             <span class="d-none d-md-inline-block">{%s title %}</span>
                                             <svg class="d-md-none" width="22" height="22">
                                                 <use href="{%s prefix %}static/icons/icons.svg#{%s icons[key] %}"/>
@@ -97,10 +97,10 @@
     {%= tpl.Footer(r) %}
 {% endfunc %}
 
-{% func ListGroups(r *http.Request, groups []*rule.ApiGroup, filter string) %}
+{% func ListGroups(r *http.Request, groups []*rule.ApiGroup, state string) %}
     {%code
         prefix := vmalertutil.Prefix(r.URL.Path)
-        filters := map[string]string{
+        states := map[string]string{
             "":          "All",
             "unhealthy": "Unhealthy",
             "nomatch":   "No Match",
@@ -110,14 +110,14 @@
             "unhealthy": "unhealthy",
             "nomatch":   "nomatch",
         }
-        currentText := filters[filter]
-        currentIcon := icons[filter]
+        currentText := states[state]
+        currentIcon := icons[state]
     %}
     {%= tpl.Header(r, navItems, "Groups", getLastConfigError()) %}
-        {%= Controls(prefix, currentIcon, currentText, icons, filters, true) %}
+        {%= Controls(prefix, currentIcon, currentText, icons, states, true) %}
         {%  if len(groups) > 0 %}
             {% for _, g := range groups %}
-                <div id="group-{%s g.ID %}" class="w-100 border-0 flex-column vm-group{% if g.Unhealthy > 0 %} alert-danger{% endif %}">
+                <div id="group-{%s g.ID %}" class="w-100 border-0 flex-column vm-group{% if g.States["unhealthy"] > 0 %} alert-danger{% endif %}">
                     <span class="d-flex justify-content-between">
                         <a
                             class="vm-group-search"
@@ -130,9 +130,9 @@
                             data-bs-target="#item-{%s g.ID %}"
                         >
                             <span class="d-flex gap-2">
-                                {% if g.Unhealthy > 0 %}<span class="badge bg-danger" title="Number of rules with status Error">{%d g.Unhealthy %}</span> {% endif %}
-                                {% if g.NoMatch > 0 %}<span class="badge bg-warning" title="Number of rules with status NoMatch">{%d g.NoMatch %}</span> {% endif %}
-                                <span class="badge bg-success" title="Number of rules with status Ok">{%d g.Healthy %}</span>
+                                {% if g.States["unhealthy"] > 0 %}<span class="badge bg-danger" title="Number of rules with status Error">{%d g.States["unhealthy"] %}</span> {% endif %}
+                                {% if g.States["nomatch"] > 0 %}<span class="badge bg-warning" title="Number of rules with status NoMatch">{%d g.States["nomatch"] %}</span> {% endif %}
+                                <span class="badge bg-success" title="Number of rules with status Ok">{%d g.States["ok"] %}</span>
                             </span>
                         </span>
                     </span>
@@ -189,7 +189,7 @@
                                                         <b>record:</b> {%s r.Name %}
                                                     {% endif %}
                                                     |
-                                                    {%= seriesFetchedWarn(prefix, r) %}
+                                                    {%= seriesFetchedWarn(prefix, &r) %}
                                                     <span><a target="_blank" href="{%s prefix+r.WebLink() %}">Details</a></span>
                                                 </div>
                                                 <div class="col-12">
@@ -476,7 +476,7 @@
 {% endfunc %}
 
 
-{% func RuleDetails(r *http.Request, rule rule.ApiRule) %}
+{% func Rule(r *http.Request, rule rule.ApiRule) %}
     {%code prefix := vmalertutil.Prefix(r.URL.Path) %}
     {%= tpl.Header(r, navItems, "", getLastConfigError()) %}
     {%code
@@ -661,8 +661,8 @@
 <span class="badge bg-warning text-dark" title="This firing state is kept because of `keep_firing_for`">stabilizing</span>
 {% endfunc %}
 
-{% func seriesFetchedWarn(prefix string, r rule.ApiRule) %}
-{% if isNoMatch(r) %}
+{% func seriesFetchedWarn(prefix string, r *rule.ApiRule) %}
+{% if r.IsNoMatch() %}
 <svg
     data-bs-toggle="tooltip"
     title="No match! This rule's last evaluation hasn't selected any time series from the datasource.
@@ -673,9 +673,3 @@
 </svg>
 {% endif %}
 {% endfunc %}
-
-{%code
-  func isNoMatch (r rule.ApiRule) bool {
-    return r.LastSamples == 0 && r.LastSeriesFetched != nil && *r.LastSeriesFetched == 0
-  }
-%}

app/vmalert/web.qtpl.go

@@ -31,7 +31,7 @@ var (
 )
 
 //line app/vmalert/web.qtpl:15
-func StreamControls(qw422016 *qt422016.Writer, prefix, currentIcon, currentText string, icons, filters map[string]string, search bool) {
+func StreamControls(qw422016 *qt422016.Writer, prefix, currentIcon, currentText string, icons, states map[string]string, search bool) {
 //line app/vmalert/web.qtpl:15
 	qw422016.N().S(`
     <div class="btn-toolbar mb-3" role="toolbar">
@@ -59,7 +59,7 @@ func StreamControls(qw422016 *qt422016.Writer, prefix, currentIcon, currentText
                 </a>
                 `)
 //line app/vmalert/web.qtpl:31
-	if len(filters) > 0 {
+	if len(states) > 0 {
 //line app/vmalert/web.qtpl:31
 		qw422016.N().S(`
                     <span class="d-none d-md-inline-block">Filter by status:</span>
@@ -68,7 +68,7 @@ func StreamControls(qw422016 *qt422016.Writer, prefix, currentIcon, currentText
 //line app/vmalert/web.qtpl:34
 		qw422016.E().S(prefix)
 //line app/vmalert/web.qtpl:34
-		qw422016.N().S(`static/icons/icons.svg#filter">
+		qw422016.N().S(`static/icons/icons.svg#state">
                     </svg>
                     <div class="dropdown">
                         <button
@@ -97,7 +97,7 @@ func StreamControls(qw422016 *qt422016.Writer, prefix, currentIcon, currentText
                         <ul class="dropdown-menu">
                             `)
 //line app/vmalert/web.qtpl:49
-		for key, title := range filters {
+		for key, title := range states {
 //line app/vmalert/web.qtpl:49
 			qw422016.N().S(`
                                 `)
@@ -106,7 +106,7 @@ func StreamControls(qw422016 *qt422016.Writer, prefix, currentIcon, currentText
 //line app/vmalert/web.qtpl:50
 				qw422016.N().S(`
                                     <li>
-                                        <a class="dropdown-item" onclick="groupFilter('`)
+                                        <a class="dropdown-item" onclick="groupForState('`)
 //line app/vmalert/web.qtpl:52
 				qw422016.E().S(key)
 //line app/vmalert/web.qtpl:52
@@ -176,22 +176,22 @@ func StreamControls(qw422016 *qt422016.Writer, prefix, currentIcon, currentText
 }
 
 //line app/vmalert/web.qtpl:77
-func WriteControls(qq422016 qtio422016.Writer, prefix, currentIcon, currentText string, icons, filters map[string]string, search bool) {
+func WriteControls(qq422016 qtio422016.Writer, prefix, currentIcon, currentText string, icons, states map[string]string, search bool) {
 //line app/vmalert/web.qtpl:77
 	qw422016 := qt422016.AcquireWriter(qq422016)
 //line app/vmalert/web.qtpl:77
-	StreamControls(qw422016, prefix, currentIcon, currentText, icons, filters, search)
+	StreamControls(qw422016, prefix, currentIcon, currentText, icons, states, search)
 //line app/vmalert/web.qtpl:77
 	qt422016.ReleaseWriter(qw422016)
 //line app/vmalert/web.qtpl:77
 }
 
 //line app/vmalert/web.qtpl:77
-func Controls(prefix, currentIcon, currentText string, icons, filters map[string]string, search bool) string {
+func Controls(prefix, currentIcon, currentText string, icons, states map[string]string, search bool) string {
 //line app/vmalert/web.qtpl:77
 	qb422016 := qt422016.AcquireByteBuffer()
 //line app/vmalert/web.qtpl:77
-	WriteControls(qb422016, prefix, currentIcon, currentText, icons, filters, search)
+	WriteControls(qb422016, prefix, currentIcon, currentText, icons, states, search)
 //line app/vmalert/web.qtpl:77
 	qs422016 := string(qb422016.B)
 //line app/vmalert/web.qtpl:77
@@ -324,13 +324,13 @@ func Welcome(r *http.Request) string {
 }
 
 //line app/vmalert/web.qtpl:100
-func StreamListGroups(qw422016 *qt422016.Writer, r *http.Request, groups []*rule.ApiGroup, filter string) {
+func StreamListGroups(qw422016 *qt422016.Writer, r *http.Request, groups []*rule.ApiGroup, state string) {
 //line app/vmalert/web.qtpl:100
 	qw422016.N().S(`
     `)
 //line app/vmalert/web.qtpl:102
 	prefix := vmalertutil.Prefix(r.URL.Path)
-	filters := map[string]string{
+	states := map[string]string{
 		"":          "All",
 		"unhealthy": "Unhealthy",
 		"nomatch":   "No Match",
@@ -340,8 +340,8 @@ func StreamListGroups(qw422016 *qt422016.Writer, r *http.Request, groups []*rule
 		"unhealthy": "unhealthy",
 		"nomatch":   "nomatch",
 	}
-	currentText := filters[filter]
-	currentIcon := icons[filter]
+	currentText := states[state]
+	currentIcon := icons[state]
 
 //line app/vmalert/web.qtpl:115
 	qw422016.N().S(`
@@ -352,7 +352,7 @@ func StreamListGroups(qw422016 *qt422016.Writer, r *http.Request, groups []*rule
 	qw422016.N().S(`
         `)
 //line app/vmalert/web.qtpl:117
-	StreamControls(qw422016, prefix, currentIcon, currentText, icons, filters, true)
+	StreamControls(qw422016, prefix, currentIcon, currentText, icons, states, true)
 //line app/vmalert/web.qtpl:117
 	qw422016.N().S(`
         `)
@@ -371,7 +371,7 @@ func StreamListGroups(qw422016 *qt422016.Writer, r *http.Request, groups []*rule
 //line app/vmalert/web.qtpl:120
 			qw422016.N().S(`" class="w-100 border-0 flex-column vm-group`)
 //line app/vmalert/web.qtpl:120
-			if g.Unhealthy > 0 {
+			if g.States["unhealthy"] > 0 {
 //line app/vmalert/web.qtpl:120
 				qw422016.N().S(` alert-danger`)
 //line app/vmalert/web.qtpl:120
@@ -418,11 +418,11 @@ func StreamListGroups(qw422016 *qt422016.Writer, r *http.Request, groups []*rule
                             <span class="d-flex gap-2">
                                 `)
 //line app/vmalert/web.qtpl:133
-			if g.Unhealthy > 0 {
+			if g.States["unhealthy"] > 0 {
 //line app/vmalert/web.qtpl:133
 				qw422016.N().S(`<span class="badge bg-danger" title="Number of rules with status Error">`)
 //line app/vmalert/web.qtpl:133
-				qw422016.N().D(g.Unhealthy)
+				qw422016.N().D(g.States["unhealthy"])
 //line app/vmalert/web.qtpl:133
 				qw422016.N().S(`</span> `)
 //line app/vmalert/web.qtpl:133
@@ -431,11 +431,11 @@ func StreamListGroups(qw422016 *qt422016.Writer, r *http.Request, groups []*rule
 			qw422016.N().S(`
                                 `)
 //line app/vmalert/web.qtpl:134
-			if g.NoMatch > 0 {
+			if g.States["nomatch"] > 0 {
 //line app/vmalert/web.qtpl:134
 				qw422016.N().S(`<span class="badge bg-warning" title="Number of rules with status NoMatch">`)
 //line app/vmalert/web.qtpl:134
-				qw422016.N().D(g.NoMatch)
+				qw422016.N().D(g.States["nomatch"])
 //line app/vmalert/web.qtpl:134
 				qw422016.N().S(`</span> `)
 //line app/vmalert/web.qtpl:134
@@ -444,7 +444,7 @@ func StreamListGroups(qw422016 *qt422016.Writer, r *http.Request, groups []*rule
 			qw422016.N().S(`
                                 <span class="badge bg-success" title="Number of rules with status Ok">`)
 //line app/vmalert/web.qtpl:135
-			qw422016.N().D(g.Healthy)
+			qw422016.N().D(g.States["ok"])
 //line app/vmalert/web.qtpl:135
 			qw422016.N().S(`</span>
                             </span>
@@ -617,7 +617,7 @@ func StreamListGroups(qw422016 *qt422016.Writer, r *http.Request, groups []*rule
                                                     |
                                                     `)
 //line app/vmalert/web.qtpl:192
-				streamseriesFetchedWarn(qw422016, prefix, r)
+				streamseriesFetchedWarn(qw422016, prefix, &r)
 //line app/vmalert/web.qtpl:192
 				qw422016.N().S(`
                                                     <span><a target="_blank" href="`)
@@ -750,22 +750,22 @@ func StreamListGroups(qw422016 *qt422016.Writer, r *http.Request, groups []*rule
 }
 
 //line app/vmalert/web.qtpl:234
-func WriteListGroups(qq422016 qtio422016.Writer, r *http.Request, groups []*rule.ApiGroup, filter string) {
+func WriteListGroups(qq422016 qtio422016.Writer, r *http.Request, groups []*rule.ApiGroup, state string) {
 //line app/vmalert/web.qtpl:234
 	qw422016 := qt422016.AcquireWriter(qq422016)
 //line app/vmalert/web.qtpl:234
-	StreamListGroups(qw422016, r, groups, filter)
+	StreamListGroups(qw422016, r, groups, state)
 //line app/vmalert/web.qtpl:234
 	qt422016.ReleaseWriter(qw422016)
 //line app/vmalert/web.qtpl:234
 }
 
 //line app/vmalert/web.qtpl:234
-func ListGroups(r *http.Request, groups []*rule.ApiGroup, filter string) string {
+func ListGroups(r *http.Request, groups []*rule.ApiGroup, state string) string {
 //line app/vmalert/web.qtpl:234
 	qb422016 := qt422016.AcquireByteBuffer()
 //line app/vmalert/web.qtpl:234
-	WriteListGroups(qb422016, r, groups, filter)
+	WriteListGroups(qb422016, r, groups, state)
 //line app/vmalert/web.qtpl:234
 	qs422016 := string(qb422016.B)
 //line app/vmalert/web.qtpl:234
@@ -1462,7 +1462,7 @@ func Alert(r *http.Request, alert *rule.ApiAlert) string {
 }
 
 //line app/vmalert/web.qtpl:479
-func StreamRuleDetails(qw422016 *qt422016.Writer, r *http.Request, rule rule.ApiRule) {
+func StreamRule(qw422016 *qt422016.Writer, r *http.Request, rule rule.ApiRule) {
 //line app/vmalert/web.qtpl:479
 	qw422016.N().S(`
     `)
@@ -1859,22 +1859,22 @@ func StreamRuleDetails(qw422016 *qt422016.Writer, r *http.Request, rule rule.Api
 }
 
 //line app/vmalert/web.qtpl:642
-func WriteRuleDetails(qq422016 qtio422016.Writer, r *http.Request, rule rule.ApiRule) {
+func WriteRule(qq422016 qtio422016.Writer, r *http.Request, rule rule.ApiRule) {
 //line app/vmalert/web.qtpl:642
 	qw422016 := qt422016.AcquireWriter(qq422016)
 //line app/vmalert/web.qtpl:642
-	StreamRuleDetails(qw422016, r, rule)
+	StreamRule(qw422016, r, rule)
 //line app/vmalert/web.qtpl:642
 	qt422016.ReleaseWriter(qw422016)
 //line app/vmalert/web.qtpl:642
 }
 
 //line app/vmalert/web.qtpl:642
-func RuleDetails(r *http.Request, rule rule.ApiRule) string {
+func Rule(r *http.Request, rule rule.ApiRule) string {
 //line app/vmalert/web.qtpl:642
 	qb422016 := qt422016.AcquireByteBuffer()
 //line app/vmalert/web.qtpl:642
-	WriteRuleDetails(qb422016, r, rule)
+	WriteRule(qb422016, r, rule)
 //line app/vmalert/web.qtpl:642
 	qs422016 := string(qb422016.B)
 //line app/vmalert/web.qtpl:642
@@ -2015,12 +2015,12 @@ func badgeStabilizing() string {
 }
 
 //line app/vmalert/web.qtpl:664
-func streamseriesFetchedWarn(qw422016 *qt422016.Writer, prefix string, r rule.ApiRule) {
+func streamseriesFetchedWarn(qw422016 *qt422016.Writer, prefix string, r *rule.ApiRule) {
 //line app/vmalert/web.qtpl:664
 	qw422016.N().S(`
 `)
 //line app/vmalert/web.qtpl:665
-	if isNoMatch(r) {
+	if r.IsNoMatch() {
 //line app/vmalert/web.qtpl:665
 		qw422016.N().S(`
 <svg
@@ -2045,7 +2045,7 @@ func streamseriesFetchedWarn(qw422016 *qt422016.Writer, prefix string, r rule.Ap
 }
 
 //line app/vmalert/web.qtpl:675
-func writeseriesFetchedWarn(qq422016 qtio422016.Writer, prefix string, r rule.ApiRule) {
+func writeseriesFetchedWarn(qq422016 qtio422016.Writer, prefix string, r *rule.ApiRule) {
 //line app/vmalert/web.qtpl:675
 	qw422016 := qt422016.AcquireWriter(qq422016)
 //line app/vmalert/web.qtpl:675
@@ -2056,7 +2056,7 @@ func writeseriesFetchedWarn(qq422016 qtio422016.Writer, prefix string, r rule.Ap
 }
 
 //line app/vmalert/web.qtpl:675
-func seriesFetchedWarn(prefix string, r rule.ApiRule) string {
+func seriesFetchedWarn(prefix string, r *rule.ApiRule) string {
 //line app/vmalert/web.qtpl:675
 	qb422016 := qt422016.AcquireByteBuffer()
 //line app/vmalert/web.qtpl:675
@@ -2069,8 +2069,3 @@ func seriesFetchedWarn(prefix string, r rule.ApiRule) string {
 	return qs422016
 //line app/vmalert/web.qtpl:675
 }
-
-//line app/vmalert/web.qtpl:678
-func isNoMatch(r rule.ApiRule) bool {
-	return r.LastSamples == 0 && r.LastSeriesFetched != nil && *r.LastSeriesFetched == 0
-}

app/vmalert/web_test.go

@@ -210,7 +210,7 @@ func TestHandler(t *testing.T) {
 		}
 	})
 
-	t.Run("/api/v1/rules&filters", func(t *testing.T) {
+	t.Run("/api/v1/rules&states", func(t *testing.T) {
 		check := func(url string, statusCode, expGroups, expRules int) {
 			t.Helper()
 			lr := listGroupsResponse{}
@@ -252,9 +252,15 @@ func TestHandler(t *testing.T) {
 		check("/api/v1/rules?rule_group[]=group&file[]=foo", 200, 0, 0)
 		check("/api/v1/rules?rule_group[]=group&file[]=rules.yaml", 200, 3, 6)
 
-		check("/api/v1/rules?rule_group[]=group&file[]=rules.yaml&rule_name[]=foo", 200, 3, 0)
+		check("/api/v1/rules?rule_group[]=group&file[]=rules.yaml&rule_name[]=foo", 200, 0, 0)
 		check("/api/v1/rules?rule_group[]=group&file[]=rules.yaml&rule_name[]=alert", 200, 3, 3)
 		check("/api/v1/rules?rule_group[]=group&file[]=rules.yaml&rule_name[]=alert&rule_name[]=record", 200, 3, 6)
+
+		check("/api/v1/rules?group_limit=1", 200, 1, 2)
+		check("/api/v1/rules?group_limit=1&type=alert", 200, 1, 1)
+		check("/api/v1/rules?group_limit=1&type=record", 200, 1, 1)
+		check("/api/v1/rules?group_limit=2", 200, 2, 4)
+		check(fmt.Sprintf("/api/v1/rules?group_limit=1&page_num=%d", 1), 200, 1, 2)
 	})
 	t.Run("/api/v1/rules&exclude_alerts=true", func(t *testing.T) {
 		// check if response returns active alerts by default

app/vmauth/auth_config.go

@@ -117,7 +117,10 @@ type AccessLogFilters struct {
 	SkipStatusCodes []int `yaml:"skip_status_codes"`
 }
 
-func (ui *UserInfo) logRequest(r *http.Request, userName string, statusCode int) {
+func (ui *UserInfo) logRequest(r *http.Request, userName string, statusCode int, duration time.Duration) {
+	if ui.AccessLog == nil {
+		return
+	}
 	filters := ui.AccessLog.Filters
 	if filters != nil && len(filters.SkipStatusCodes) > 0 {
 		if slices.Contains(filters.SkipStatusCodes, statusCode) {
@@ -127,8 +130,8 @@ func (ui *UserInfo) logRequest(r *http.Request, userName string, statusCode int)
 
 	remoteAddr := httpserver.GetQuotedRemoteAddr(r)
 	requestURI := httpserver.GetRequestURI(r)
-	logger.Infof("access_log request_host=%q request_uri=%q status_code=%d remote_addr=%s user_agent=%q referer=%q username=%q",
-		r.Host, requestURI, statusCode, remoteAddr, r.UserAgent(), r.Referer(), userName)
+	logger.Infof("access_log request_host=%q request_uri=%q status_code=%d remote_addr=%s user_agent=%q referer=%q duration_ms=%d username=%q",
+		r.Host, requestURI, statusCode, remoteAddr, r.UserAgent(), r.Referer(), duration.Milliseconds(), userName)
 }
 
 // HeadersConf represents config for request and response headers.
@@ -144,7 +147,7 @@ func (ui *UserInfo) beginConcurrencyLimit(ctx context.Context) error {
 	case ui.concurrencyLimitCh <- struct{}{}:
 		return nil
 	default:
-		// The number of concurrently executed requests for the given user equals the limt.
+		// The number of concurrently executed requests for the given user equals the limit.
 		// Wait until some of the currently executed requests are finished, so the current request could be executed.
 		// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10078
 		select {
@@ -632,7 +635,7 @@ func getLeastLoadedBackendURL(bus []*backendURL, atomicCounter *atomic.Uint32) *
 		// The Load() in front of CompareAndSwap() avoids CAS overhead for items with values bigger than 0.
 		if bu.concurrentRequests.Load() == 0 && bu.concurrentRequests.CompareAndSwap(0, 1) {
 			atomicCounter.CompareAndSwap(n+1, idx+1)
-			// There is no need in the call bu.get(), because we already incremented bu.concrrentRequests above.
+			// There is no need in the call bu.get(), because we already incremented bu.concurrentRequests above.
 			return bu
 		}
 	}
@@ -875,12 +878,14 @@ func reloadAuthConfigData(data []byte) (bool, error) {
 		return false, fmt.Errorf("failed to parse auth config: %w", err)
 	}
 
-	jui, err := parseJWTUsers(ac)
+	jui, oidcDP, err := parseJWTUsers(ac)
 	if err != nil {
 		return false, fmt.Errorf("failed to parse JWT users from auth config: %w", err)
 	}
+	oidcDP.startDiscovery()
 	jwtc := &jwtCache{
-		users: jui,
+		users:  jui,
+		oidcDP: oidcDP,
 	}
 
 	m, err := parseAuthConfigUsers(ac)
@@ -899,6 +904,11 @@ func reloadAuthConfigData(data []byte) (bool, error) {
 	}
 	metrics.RegisterSet(ac.ms)
 
+	jwtcPrev := jwtAuthCache.Load()
+	if jwtcPrev != nil {
+		jwtcPrev.oidcDP.stopDiscovery()
+	}
+
 	authConfig.Store(ac)
 	authConfigData.Store(&data)
 	authUsers.Store(&m)

app/vmauth/auth_config_test.go

@@ -4,8 +4,11 @@ import (
 	"bytes"
 	"fmt"
 	"net"
+	"net/http"
 	"net/url"
+	"strings"
 	"testing"
+	"time"
 
 	"gopkg.in/yaml.v2"
 
@@ -993,6 +996,41 @@ func TestDiscoverBackendIPsWithIPV6(t *testing.T) {
 
 }
 
+func TestLogRequest(t *testing.T) {
+	ui := &UserInfo{AccessLog: &AccessLog{}}
+
+	testOutput := &bytes.Buffer{}
+	logger.SetOutputForTests(testOutput)
+	defer logger.ResetOutputForTest()
+
+	req, err := http.NewRequest("GET", "http://localhost:8080/select/0/prometheus", nil)
+	if err != nil {
+		t.Fatalf("unexpected error: %s", err)
+	}
+
+	f := func(user string, status int, duration time.Duration, expectedLog string) {
+		t.Helper()
+
+		testOutput.Reset()
+		ui.logRequest(req, user, status, duration)
+
+		got := testOutput.String()
+		if expectedLog == "" && got != "" {
+			t.Fatalf("expected empty log, got %q", got)
+		}
+		if !strings.Contains(got, expectedLog) {
+			t.Fatalf("output \n%q \nshould contain \n%q", testOutput.String(), expectedLog)
+		}
+	}
+
+	f("foo", 200, 10*time.Millisecond, `access_log request_host="localhost:8080" request_uri="" status_code=200 remote_addr="" user_agent="" referer="" duration_ms=10 username="foo"`)
+	f("foo", 404, time.Second, `access_log request_host="localhost:8080" request_uri="" status_code=404 remote_addr="" user_agent="" referer="" duration_ms=1000 username="foo"`)
+
+	ui.AccessLog.Filters = &AccessLogFilters{SkipStatusCodes: []int{200}}
+	f("foo", 200, 10*time.Millisecond, ``)
+	f("foo", 404, 10*time.Millisecond, `access_log request_host="localhost:8080" request_uri="" status_code=404 remote_addr="" user_agent="" referer="" duration_ms=10 username="foo"`)
+}
+
 func getRegexs(paths []string) []*Regex {
 	var sps []*Regex
 	for _, path := range paths {

app/vmauth/example_config.yml

@@ -116,6 +116,20 @@ users:
   - "http://default1:8888/unsupported_url_handler"
   - "http://default2:8888/unsupported_url_handler"
 
+# A JWT token based routing:
+# - Requests with JWT token that has the following structure:
+# {"team": "ops", "security": {"read_access": "1"}, "vm_access": {"metrics_account_id": 1000,"metrics_project_id":5}}
+# is routed to vmselect nodes and request url placeholder replaced with metrics tenant identificators
+- name: jwt-opts-team
+  jwt:
+    match_claims:
+     team: ops
+     security.read_access: "1"
+    skip_verify: true
+  url_prefix:
+  - "http://vmselect1:8481/select/{{.MetricsTenant}}/prometheus"
+  - "http://vmselect2:8481/select/{{.MetricsTenant}}/prometheus"
+
 # Requests without Authorization header are proxied according to `unauthorized_user` section.
 # Requests are proxied in round-robin fashion between `url_prefix` backends.
 # The deny_partial_response query arg is added to all the proxied requests.

app/vmauth/jwt.go

@@ -5,7 +5,10 @@ import (
 	"net/url"
 	"os"
 	"slices"
+	"sort"
 	"strings"
+	"sync"
+	"sync/atomic"
 	"time"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/jwt"
@@ -44,38 +47,69 @@ var urlPathPlaceHolders = []string{
 type jwtCache struct {
 	// users contain UserInfo`s from AuthConfig with JWTConfig set
 	users []*UserInfo
+
+	oidcDP *oidcDiscovererPool
 }
 
 type JWTConfig struct {
-	PublicKeys     []string `yaml:"public_keys,omitempty"`
-	PublicKeyFiles []string `yaml:"public_key_files,omitempty"`
-	SkipVerify     bool     `yaml:"skip_verify,omitempty"`
+	PublicKeys        []string          `yaml:"public_keys,omitempty"`
+	PublicKeyFiles    []string          `yaml:"public_key_files,omitempty"`
+	SkipVerify        bool              `yaml:"skip_verify,omitempty"`
+	OIDC              *oidcConfig       `yaml:"oidc,omitempty"`
+	MatchClaims       map[string]string `yaml:"match_claims,omitempty"`
+	parsedMatchClaims []*jwt.Claim
 
-	verifierPool *jwt.VerifierPool
+	// verifierPool is used to verify JWT tokens.
+	// It is initialized from PublicKeys and/or PublicKeyFiles.
+	// In this case, it is initialized once at config reload and never updated until next reload
+	// In case of OIDC, it is initialized on config reload and periodically updated by discovery process.
+	verifierPool atomic.Pointer[jwt.VerifierPool]
 }
 
-func parseJWTUsers(ac *AuthConfig) ([]*UserInfo, error) {
+func parseJWTUsers(ac *AuthConfig) ([]*UserInfo, *oidcDiscovererPool, error) {
 	jui := make([]*UserInfo, 0, len(ac.Users))
-	for _, ui := range ac.Users {
+	oidcDP := &oidcDiscovererPool{}
+
+	uniqClaims := make(map[string]*UserInfo)
+	var sortedClaims []string
+	for idx, ui := range ac.Users {
 		jwtToken := ui.JWT
 		if jwtToken == nil {
 			continue
 		}
 
 		if ui.AuthToken != "" || ui.BearerToken != "" || ui.Username != "" || ui.Password != "" {
-			return nil, fmt.Errorf("auth_token, bearer_token, username and password cannot be specified if jwt is set")
+			return nil, nil, fmt.Errorf("auth_token, bearer_token, username and password cannot be specified if jwt is set")
 		}
-		if len(jwtToken.PublicKeys) == 0 && len(jwtToken.PublicKeyFiles) == 0 && !jwtToken.SkipVerify {
-			return nil, fmt.Errorf("jwt must contain at least a single public key, public_key_files or have skip_verify=true")
+		if len(jwtToken.PublicKeys) == 0 && len(jwtToken.PublicKeyFiles) == 0 && !jwtToken.SkipVerify && jwtToken.OIDC == nil {
+			return nil, nil, fmt.Errorf("jwt must contain at least a single public key, public_key_files, oidc or have skip_verify=true")
 		}
+		var claimsString string
+		sortedClaims = sortedClaims[:0]
+		parsedClaims := make([]*jwt.Claim, 0, len(jwtToken.MatchClaims))
+		for ck, cv := range jwtToken.MatchClaims {
+			sortedClaims = append(sortedClaims, fmt.Sprintf("%s=%s", ck, cv))
+			pc, err := jwt.NewClaim(ck, cv)
+			if err != nil {
+				return nil, nil, fmt.Errorf("incorrect match claim, key=%q, value regex=%q: %w", ck, cv, err)
+			}
+			parsedClaims = append(parsedClaims, pc)
+		}
+		ui.JWT.parsedMatchClaims = parsedClaims
+		sort.Strings(sortedClaims)
+		claimsString = strings.Join(sortedClaims, ",")
 
+		if oldUI, ok := uniqClaims[claimsString]; ok {
+			return nil, nil, fmt.Errorf("duplicate match claims=%q found for name=%q at idx=%d; the previous one is set for name=%q", claimsString, ui.Name, idx, oldUI.Name)
+		}
+		uniqClaims[claimsString] = &ui
 		if len(jwtToken.PublicKeys) > 0 || len(jwtToken.PublicKeyFiles) > 0 {
 			keys := make([]any, 0, len(jwtToken.PublicKeys)+len(jwtToken.PublicKeyFiles))
 
 			for i := range jwtToken.PublicKeys {
 				k, err := jwt.ParseKey([]byte(jwtToken.PublicKeys[i]))
 				if err != nil {
-					return nil, err
+					return nil, nil, err
 				}
 				keys = append(keys, k)
 			}
@@ -83,33 +117,52 @@ func parseJWTUsers(ac *AuthConfig) ([]*UserInfo, error) {
 			for _, filePath := range jwtToken.PublicKeyFiles {
 				keyData, err := os.ReadFile(filePath)
 				if err != nil {
-					return nil, fmt.Errorf("cannot read public key from file %q: %w", filePath, err)
+					return nil, nil, fmt.Errorf("cannot read public key from file %q: %w", filePath, err)
 				}
 				k, err := jwt.ParseKey(keyData)
 				if err != nil {
-					return nil, fmt.Errorf("cannot parse public key from file %q: %w", filePath, err)
+					return nil, nil, fmt.Errorf("cannot parse public key from file %q: %w", filePath, err)
 				}
 				keys = append(keys, k)
 			}
 
 			vp, err := jwt.NewVerifierPool(keys)
 			if err != nil {
-				return nil, err
+				return nil, nil, err
 			}
 
-			jwtToken.verifierPool = vp
+			jwtToken.verifierPool.Store(vp)
 		}
+		if jwtToken.OIDC != nil {
+			if len(jwtToken.PublicKeys) > 0 || len(jwtToken.PublicKeyFiles) > 0 || jwtToken.SkipVerify {
+				return nil, nil, fmt.Errorf("jwt with oidc cannot contain public keys or have skip_verify=true")
+			}
+
+			if jwtToken.OIDC.Issuer == "" {
+				return nil, nil, fmt.Errorf("oidc issuer cannot be empty")
+			}
+			isserURL, err := url.Parse(jwtToken.OIDC.Issuer)
+			if err != nil {
+				return nil, nil, fmt.Errorf("oidc issuer %q must be a valid URL", jwtToken.OIDC.Issuer)
+			}
+			if isserURL.Scheme != "https" && isserURL.Scheme != "http" {
+				return nil, nil, fmt.Errorf("oidc issuer %q must have http or https scheme", jwtToken.OIDC.Issuer)
+			}
+
+			oidcDP.createOrAdd(ui.JWT.OIDC.Issuer, &ui.JWT.verifierPool)
+		}
+
 		if err := parseJWTPlaceholdersForUserInfo(&ui, true); err != nil {
-			return nil, err
+			return nil, nil, err
 		}
 
 		if err := ui.initURLs(); err != nil {
-			return nil, err
+			return nil, nil, err
 		}
 
 		metricLabels, err := ui.getMetricLabels()
 		if err != nil {
-			return nil, fmt.Errorf("cannot parse metric_labels: %w", err)
+			return nil, nil, fmt.Errorf("cannot parse metric_labels: %w", err)
 		}
 		ui.requests = ac.ms.GetOrCreateCounter(`vmauth_user_requests_total` + metricLabels)
 		ui.requestErrors = ac.ms.GetOrCreateCounter(`vmauth_user_request_errors_total` + metricLabels)
@@ -128,36 +181,53 @@ func parseJWTUsers(ac *AuthConfig) ([]*UserInfo, error) {
 
 		rt, err := newRoundTripper(ui.TLSCAFile, ui.TLSCertFile, ui.TLSKeyFile, ui.TLSServerName, ui.TLSInsecureSkipVerify)
 		if err != nil {
-			return nil, fmt.Errorf("cannot initialize HTTP RoundTripper: %w", err)
+			return nil, nil, fmt.Errorf("cannot initialize HTTP RoundTripper: %w", err)
 		}
 		ui.rt = rt
 
 		jui = append(jui, &ui)
 	}
 
-	// TODO: the limitation will be lifted once claim based matching will be implemented
-	if len(jui) > 1 {
-		return nil, fmt.Errorf("multiple users with JWT tokens are not supported; found %d users", len(jui))
-	}
+	// sort by amount of matching claims
+	// it allows to more specific claim win in case of clash
+	sort.SliceStable(jui, func(i, j int) bool {
+		return len(jui[i].JWT.MatchClaims) > len(jui[j].JWT.MatchClaims)
+	})
 
-	return jui, nil
+	return jui, oidcDP, nil
 }
 
-func getUserInfoByJWTToken(ats []string) (*UserInfo, *jwt.Token) {
+var tokenPool sync.Pool
+
+func getToken() *jwt.Token {
+	tkn := tokenPool.Get()
+	if tkn == nil {
+		return &jwt.Token{}
+	}
+	return tkn.(*jwt.Token)
+}
+
+func putToken(tkn *jwt.Token) {
+	tkn.Reset()
+	tokenPool.Put(tkn)
+}
+
+func getJWTUserInfo(ats []string) (*UserInfo, *jwt.Token) {
 	js := *jwtAuthCache.Load()
 	if len(js.users) == 0 {
 		return nil, nil
 	}
 
+	tkn := getToken()
+
 	for _, at := range ats {
 		if strings.Count(at, ".") != 2 {
 			continue
 		}
 
 		at, _ = strings.CutPrefix(at, `http_auth:`)
-
-		tkn, err := jwt.NewToken(at, true)
-		if err != nil {
+		tkn.Reset()
+		if err := tkn.Parse(at, true); err != nil {
 			if *logInvalidAuthTokens {
 				logger.Infof("cannot parse jwt token: %s", err)
 			}
@@ -172,25 +242,68 @@ func getUserInfoByJWTToken(ats []string) (*UserInfo, *jwt.Token) {
 			continue
 		}
 
-		for _, ui := range js.users {
-			if ui.JWT.SkipVerify {
-				return ui, tkn
-			}
-
-			if err := ui.JWT.verifierPool.Verify(tkn); err != nil {
-				if *logInvalidAuthTokens {
-					logger.Infof("cannot verify jwt token: %s", err)
-				}
-				continue
-			}
-
+		if ui := getUserInfoByJWTToken(tkn, js.users); ui != nil {
 			return ui, tkn
 		}
 	}
 
+	putToken(tkn)
 	return nil, nil
 }
 
+func getUserInfoByJWTToken(tkn *jwt.Token, users []*UserInfo) *UserInfo {
+	for _, ui := range users {
+		if !tkn.MatchClaims(ui.JWT.parsedMatchClaims) {
+			continue
+		}
+
+		if ui.JWT.SkipVerify {
+			return ui
+		}
+
+		if ui.JWT.OIDC != nil {
+			// OIDC requires iss claim.
+			// It must match the discovery issuer URL set in OIDC config.
+			// https://openid.net/specs/openid-connect-discovery-1_0.html#ProviderMetadata
+			if tkn.Issuer() == "" {
+				if *logInvalidAuthTokens {
+					logger.Infof("jwt token must have issuer filed")
+				}
+				return nil
+			}
+			if tkn.Issuer() != ui.JWT.OIDC.Issuer {
+				if *logInvalidAuthTokens {
+					logger.Infof("jwt token issuer: %q does not match oidc issuer: %q", tkn.Issuer(), ui.JWT.OIDC.Issuer)
+				}
+				return nil
+			}
+		}
+
+		vp := ui.JWT.verifierPool.Load()
+		if vp == nil {
+			if *logInvalidAuthTokens {
+				logger.Infof("jwt verifier not initialed")
+			}
+			return nil
+		}
+
+		if err := vp.Verify(tkn); err != nil {
+			if *logInvalidAuthTokens {
+				logger.Infof("cannot verify jwt token: %s", err)
+			}
+			return nil
+		}
+
+		return ui
+	}
+
+	if *logInvalidAuthTokens {
+		logger.Infof("no user match jwt token")
+	}
+
+	return nil
+}
+
 func replaceJWTPlaceholders(bu *backendURL, hc HeadersConf, vma *jwt.VMAccessClaim) (*url.URL, HeadersConf) {
 	if !bu.hasPlaceHolders && !hc.hasAnyPlaceHolders {
 		return bu.url, hc

app/vmauth/jwt_test.go

@@ -1,7 +1,10 @@
 package main
 
 import (
+	"encoding/json"
 	"fmt"
+	"net/http"
+	"net/http/httptest"
 	"os"
 	"path/filepath"
 	"testing"
@@ -36,14 +39,16 @@ XOtclIk1uhc03oL9nOQ=
 			}
 			return
 		}
-		users, err := parseJWTUsers(ac)
-		if err != nil {
-			if expErr != err.Error() {
-				t.Fatalf("unexpected error; got\n%q\nwant \n%q", err.Error(), expErr)
-			}
-			return
+		users, oidcDP, err := parseJWTUsers(ac)
+		if err == nil {
+			t.Fatalf("expecting non-nil error; got %v", users)
+		}
+		if expErr != err.Error() {
+			t.Fatalf("unexpected error; got\n%q\nwant \n%q", err.Error(), expErr)
+		}
+		if oidcDP != nil {
+			t.Fatalf("expecting nil oidcDP; got %v", oidcDP)
 		}
-		t.Fatalf("expecting non-nil error; got %v", users)
 	}
 
 	// unauthorized_user cannot be used with jwt
@@ -80,28 +85,28 @@ users:
 users:
 - jwt: {}
   url_prefix: http://foo.bar
-`, `jwt must contain at least a single public key, public_key_files or have skip_verify=true`)
+`, `jwt must contain at least a single public key, public_key_files, oidc or have skip_verify=true`)
 
 	// jwt public_keys or skip_verify must be set, part 2
 	f(`
 users:
 - jwt: {public_keys: null}
   url_prefix: http://foo.bar
-`, `jwt must contain at least a single public key, public_key_files or have skip_verify=true`)
+`, `jwt must contain at least a single public key, public_key_files, oidc or have skip_verify=true`)
 
 	// jwt public_keys or skip_verify must be set, part 3
 	f(`
 users:
 - jwt: {public_keys: []}
   url_prefix: http://foo.bar
-`, `jwt must contain at least a single public key, public_key_files or have skip_verify=true`)
+`, `jwt must contain at least a single public key, public_key_files, oidc or have skip_verify=true`)
 
 	// jwt public_keys, public_key_files or skip_verify must be set
 	f(`
 users:
 - jwt: {public_key_files: []}
   url_prefix: http://foo.bar
-`, `jwt must contain at least a single public key, public_key_files or have skip_verify=true`)
+`, `jwt must contain at least a single public key, public_key_files, oidc or have skip_verify=true`)
 
 	// invalid public key, part 1
 	f(`
@@ -140,7 +145,7 @@ users:
     public_keys:
     - %q
   url_prefix: http://foo.bar
-`, validRSAPublicKey, validECDSAPublicKey), `multiple users with JWT tokens are not supported; found 2 users`)
+`, validRSAPublicKey, validECDSAPublicKey), `duplicate match claims="" found for name="" at idx=1; the previous one is set for name=""`)
 
 	// public key file doesn't exist
 	f(`
@@ -196,6 +201,90 @@ users:
 `,
 		"request header: \"AccountID\" has unsupported placeholder: \"{{ .LogsAccountID }}\", supported values are: {{.MetricsTenant}}, {{.MetricsExtraLabels}}, {{.MetricsExtraFilters}}, {{.LogsAccountID}}, {{.LogsProjectID}}, {{.LogsExtraFilters}}, {{.LogsExtraStreamFilters}}",
 	)
+
+	// oidc is not an object
+	f(`
+users:
+- jwt: 
+    oidc: "not an object"
+  url_prefix: http://foo.bar
+`,
+		"cannot unmarshal AuthConfig data: yaml: unmarshal errors:\n  line 4: cannot unmarshal !!str `not an ...` into main.oidcConfig",
+	)
+
+	// oidc issuer empty
+	f(`
+users:
+- jwt: 
+    oidc: {}
+  url_prefix: http://foo.bar
+`,
+		"oidc issuer cannot be empty",
+	)
+
+	// oidc issuer invalid urls
+	f(`
+users:
+- jwt: 
+    oidc:
+      issuer: "::invalid-url"
+  url_prefix: http://foo.bar
+`,
+		"oidc issuer \"::invalid-url\" must be a valid URL",
+	)
+
+	// oidc issuer invalid urls
+	f(`
+users:
+- jwt: 
+    oidc:
+      issuer: "invalid-url"
+  url_prefix: http://foo.bar
+`,
+		"oidc issuer \"invalid-url\" must have http or https scheme",
+	)
+
+	// oidc and public_keys are not allowed
+	f(fmt.Sprintf(`
+users:
+- jwt: 
+    public_keys:
+    - %q
+    oidc: 
+      issuer: https://example.com
+  url_prefix: http://foo.bar
+`, validRSAPublicKey),
+		"jwt with oidc cannot contain public keys or have skip_verify=true",
+	)
+
+	// oidc and skip_verify are not allowed
+	f(`
+users:
+- jwt: 
+    skip_verify: true
+    oidc: 
+      issuer: https://example.com
+  url_prefix: http://foo.bar
+`,
+		"jwt with oidc cannot contain public keys or have skip_verify=true",
+	)
+	// duplicate claims
+	f(`
+users:
+- jwt:
+    skip_verify: true
+    match_claims:
+     team: ops
+  name: user-1
+  url_prefix: http://foo.bar
+- jwt:
+    skip_verify: true
+    match_claims:
+     team: ops
+  name: user-2
+  url_prefix: http://foo.bar`,
+		"duplicate match claims=\"team=ops\" found for name=\"user-2\" at idx=1; the previous one is set for name=\"user-1\"",
+	)
 }
 
 func TestJWTParseAuthConfigSuccess(t *testing.T) {
@@ -225,10 +314,12 @@ XOtclIk1uhc03oL9nOQ=
 			t.Fatalf("unexpected error: %s", err)
 		}
 
-		jui, err := parseJWTUsers(ac)
+		jui, oidcDP, err := parseJWTUsers(ac)
 		if err != nil {
 			t.Fatalf("unexpected error: %s", err)
 		}
+		oidcDP.startDiscovery()
+		defer oidcDP.stopDiscovery()
 
 		for _, ui := range jui {
 			if ui.JWT == nil {
@@ -236,13 +327,13 @@ XOtclIk1uhc03oL9nOQ=
 			}
 
 			if ui.JWT.SkipVerify {
-				if ui.JWT.verifierPool != nil {
+				if ui.JWT.verifierPool.Load() != nil {
 					t.Fatalf("unexpected non-nil verifier pool for skip_verify=true")
 				}
 				continue
 			}
 
-			if ui.JWT.verifierPool == nil {
+			if ui.JWT.verifierPool.Load() == nil {
 				t.Fatalf("unexpected nil verifier pool for non-empty public keys")
 			}
 		}
@@ -333,4 +424,80 @@ users:
     - %q
   url_prefix: http://foo.bar
 `, validECDSAPublicKey, rsaKeyFile))
+
+	// oidc stub server
+	var ipSrv *httptest.Server
+	ipSrv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.URL.Path == "/.well-known/openid-configuration" {
+			w.Header().Set("Content-Type", "application/json")
+			_ = json.NewEncoder(w).Encode(map[string]string{
+				"issuer":   ipSrv.URL,
+				"jwks_uri": fmt.Sprintf("%s/jwks", ipSrv.URL),
+			})
+			return
+		}
+		if r.URL.Path == "/jwks" {
+			// resp generated by https://jwkset.com/generate
+			w.Header().Set("Content-Type", "application/json")
+			w.Write([]byte(`
+{
+  "keys": [
+    {
+      "kty": "RSA",
+      "kid": "f13eee91-f566-4829-80fa-fca847c21f0e",
+      "d": "Ua1llEFz3LZ05CrK5a2JxKMUEWJGXhBPPF20hHQjzxd1w0IEJK_mhPZQG8dNtBROBNIi1FC9l6QRw-RTnVIVat5Xy4yDFNKXXL3ZLXejOHY8SXrNEIDqQ-cSwIpK9cK7Umib0PcPeEeeAED5mqDH75D8_YssWFF18kLbNB5Z9pZmn6Fshiht7l2Sh4GN-KcReOW6eiQQwckDte3OGmZCRbtEriLWJt5TUGUvfZVIlcclqNMycNB6jGa9E1pO5Up7Ki3ZbI_-6XmRgZPtqnR9oLJ1zn3fj3hYpCXo-zcqLuOu3qxcslsq5igsfBzgGtfIJHY9LfWmHUsaDEa5cAX1gQ",
+      "n": "xbLXXBTNREk70UCMiqZ53_mTzYh89W-UaPU61GZ-RZ5lYcLgyWOb5mdyRbvJpcgfZpsOeGAUWbk3GkQ4vqn8kUMnnWhUum2Qk9kGubOJGLW6yaURd00j3E-ilQ5xO2R_Hzz8bAojxV8GKdGTQ-iTf8z8nsSHH8kR2SERbNJCFFtwtFU7vyFWyoH4Lmvu2UpICTHFCR9RqwQVjyoKB1JjJ6Dh1L4zPTlsvQEnqoeFQHPYr0QcQSMYXdfPvlt_FiLOAOE89fX_9T2r9WbFAoda3uTRE5_aal0jxUU2cFyeVSIgauNtF07fp422XFb4XPkWQWrdNx0KX53laSIYQ9HOpw",
+      "e": "AQAB",
+      "p": "2JT57AD-Q2lamgjgyn0wL7DgYZ3OoCTTrDm5_NHg6h13uDvyIlXSukuUeWm4tzPSDedpstbS7dgXkLw5eQXBHwPYtByTcEZS8Z37CBnhMOOhfo_U1aNIPPanJACvWBgz47-TxHsxW1YhztZqghRoicBZPSSBAj49MgANJ4jF0zc",
+      "q": "6a4MkeSXJI-ZzQ-bgP8hwJqpLFr0AiNGQcjZMH4Nn4CPGdnGiqqe6flhfLimgbNhbb67B0-8fLIji8zGhGKDL_JSIpAAdmfs2vzeEsY2hScrqVbd1VbfRcRh0J6lsn7obxkbvQthp9sX2DQbeDcEeaFEvd9gDKQSATYEqWo7eBE",
+      "dp": "haL2yu6Z9RJuuxi7S3YPY33qFZF_y0St71j3L854zzw7gMxMTW9TRWwZQwk-1pv9AmNFzvnK0MNDVyUs-UXZsb932TrApshdqYRnPsppLvdl0GgDVYcYrbUr0IUzrFHSwraVAOlavRbaaXvX4EejcUvkRFvf1nh83fs2Iqy8E-U",
+      "dq": "Cnf5qC-Ndd3ZDg688LJ9WJuVKJ-Kfu4Fn7zXvgxnn9Wqk4XmFyA9rk21yFidXQIkQz5gMpun3g48-W5bFmMzbVp1w4af_q35NnZNnJm0p5Jxqkxx87TIm9-IYkg5NB3rW87MJ1PzNAnkr5LmCCSu1qQa6Eaxjt9qzxMUcmKH94E",
+      "qi": "saAeU11iaKHmye3cwCAYkegcyWbXV3xIXEVJtS9Af_yM19UhspwY2VhuwRaajcwYZwtvR9_ITmX9M-ea7uLdd7aDYO1fujC8NGbopeC4Hkr7yb5vTly3pfKf4h-3LwGGUucJUetdz1lmMIYiyuG4_gSf1yIEtPDLKzXiedgEMdI"
+    }
+  ]
+}
+`))
+			return
+		}
+
+		http.NotFound(w, r)
+	}))
+	defer ipSrv.Close()
+
+	f(`
+users:
+- jwt:
+    oidc:
+      issuer: ` + ipSrv.URL + `
+  url_prefix: http://foo.bar
+`)
+	// multiple match claims
+	f(fmt.Sprintf(`
+users:
+- jwt:
+   match_claims:
+    role: ro
+    team: dev
+   public_keys:
+    - %q
+  url_prefix: http://foo.bar
+- jwt:
+   match_claims:
+      role: admin
+      team: dev
+   public_key_files:
+    - %q
+    - %q
+  url_prefix: http://foo.bar
+- jwt:
+   match_claims:
+     role: viewer
+     team: dev
+     department: ceo
+   skip_verify: true
+  url_prefix: http://foo.bar
+
+
+`, validRSAPublicKey, rsaKeyFile, ecdsaKeyFile))
+
 }

app/vmauth/main.go

@@ -48,7 +48,7 @@ var (
 	responseTimeout = flag.Duration("responseTimeout", 5*time.Minute, "The timeout for receiving a response from backend")
 
 	requestBufferSize = flagutil.NewBytes("requestBufferSize", 32*1024, "The size of the buffer for reading the request body before proxying the request to backends. "+
-		"This allows reducing the comsumption of backend resources when processing requests from clients connected via slow networks. "+
+		"This allows reducing the consumption of backend resources when processing requests from clients connected via slow networks. "+
 		"Set to 0 to disable request buffering. See https://docs.victoriametrics.com/victoriametrics/vmauth/#request-body-buffering")
 	maxRequestBodySizeToRetry = flagutil.NewBytes("maxRequestBodySizeToRetry", 16*1024, "The maximum request body size to buffer in memory for potential retries at other backends. "+
 		"Request bodies larger than this size cannot be retried if the backend fails. Zero or negative value disables request body buffering and retries. "+
@@ -186,11 +186,11 @@ func requestHandler(w http.ResponseWriter, r *http.Request) bool {
 		processUserRequest(w, r, ui, nil)
 		return true
 	}
-	if ui, tkn := getUserInfoByJWTToken(ats); ui != nil {
+	if ui, tkn := getJWTUserInfo(ats); ui != nil {
 		if tkn == nil {
 			logger.Panicf("BUG: unexpected nil jwt token for user %q", ui.name())
 		}
-
+		defer putToken(tkn)
 		processUserRequest(w, r, ui, tkn)
 		return true
 	}
@@ -274,7 +274,8 @@ func processUserRequest(w http.ResponseWriter, r *http.Request, ui *UserInfo, tk
 		w = &responseWriterWithStatus{ResponseWriter: w}
 		defer func() {
 			rws := w.(*responseWriterWithStatus)
-			ui.logRequest(r, userName, rws.status)
+			duration := time.Since(startTime)
+			ui.logRequest(r, userName, rws.status, duration)
 		}()
 	}
 
@@ -427,9 +428,11 @@ func processRequest(w http.ResponseWriter, r *http.Request, ui *UserInfo, tkn *j
 		}
 		if isDefault {
 			// Don't change path and add request_path query param for default route.
+			targetURLCopy := *targetURL
 			query := targetURL.Query()
 			query.Set("request_path", u.String())
-			targetURL.RawQuery = query.Encode()
+			targetURLCopy.RawQuery = query.Encode()
+			targetURL = &targetURLCopy
 		} else {
 			// Update path for regular routes.
 			targetURL = mergeURLs(targetURL, u, up.dropSrcPathPrefixParts, up.mergeQueryArgs)

app/vmauth/main_test.go

@@ -12,6 +12,7 @@ import (
 	"encoding/pem"
 	"fmt"
 	"io"
+	"math/big"
 	"net"
 	"net/http"
 	"net/http/httptest"
@@ -102,6 +103,35 @@ User-Agent: vmauth
 X-Forwarded-For: 12.34.56.78, 42.2.3.84`
 	f(cfgStr, requestURL, backendHandler, responseExpected)
 
+	// with default_url
+	cfgStr = `
+unauthorized_user:
+  default_url: {BACKEND}/default
+  url_map:
+  - src_paths:
+    - /empty
+    url_prefix: {BACKEND}/empty`
+	requestURL = "http://some-host.com/abc/def?some_arg=some_value"
+	backendHandler = func(w http.ResponseWriter, r *http.Request) {
+		h := w.Header()
+		h.Set("Connection", "close")
+		h.Set("Foo", "bar")
+
+		var bb bytes.Buffer
+		if err := r.Header.Write(&bb); err != nil {
+			panic(fmt.Errorf("unexpected error when marshaling headers: %w", err))
+		}
+		fmt.Fprintf(w, "requested_url=http://%s%s\n%s", r.Host, r.URL, bb.String())
+	}
+	responseExpected = `
+statusCode=200
+Foo: bar
+requested_url={BACKEND}/default?request_path=http%3A%2F%2Fsome-host.com%2Fabc%2Fdef%3Fsome_arg%3Dsome_value
+Pass-Header: abc
+User-Agent: vmauth
+X-Forwarded-For: 12.34.56.78, 42.2.3.84`
+	f(cfgStr, requestURL, backendHandler, responseExpected)
+
 	// routing of all failed to authorize requests to unauthorized_user (issue #7543)
 	cfgStr = `
 unauthorized_user:
@@ -1235,11 +1265,275 @@ users:
 		request,
 		responseExpected,
 	)
+	nestedToken := genToken(t, map[string]any{
+		"exp":  time.Now().Add(10 * time.Minute).Unix(),
+		"team": "dev",
+		"nested": map[string]any{
+			"department_id": 0,
+			"scopes":        []string{"metrics", "logs"},
+			"team_permissions": map[string]any{
+				"read":  0,
+				"write": 1,
+			},
+		},
+		"vm_access": map[string]any{
+			"metrics_account_id": 123,
+			"metrics_project_id": 234,
+			"metrics_extra_labels": []string{
+				"label1=value1",
+				"label2=value2",
+			},
+			"metrics_extra_filters": []string{
+				`{label3="value3"}`,
+				`{label4="value4"}`,
+			},
+			"logs_account_id": 345,
+			"logs_project_id": 456,
+			"logs_extra_filters": []string{
+				`{"namespace":"my-app","env":"prod"}`,
+			},
+			"logs_extra_stream_filters": []string{
+				`{"team":"dev"}`,
+			},
+		},
+	}, true)
+
+	// use claim for routing, must specific match wins
+	request = httptest.NewRequest(`GET`, "http://some-host.com/route", nil)
+	request.Header.Set(`Authorization`, `Bearer `+nestedToken)
+	responseExpected = `
+statusCode=200
+path: /dev/route
+query:
+headers:
+`
+	f(`
+users:
+- jwt:
+    skip_verify: true
+    match_claims:
+     team: dev
+     nested.scopes.1: "logs"
+     nested.department_id: "0"
+  url_map:
+    - src_paths: ["/route"]
+      url_prefix: {BACKEND}/dev
+- jwt:
+    skip_verify: true
+    match_claims:
+     team: dev
+     nested.scopes.1: "logs"
+  url_map:
+    - src_paths: ["/route"]
+      url_prefix: {BACKEND}/ops
+`,
+		request,
+		responseExpected,
+	)
+
+	// use claim for routing, most specific not matching
+	request = httptest.NewRequest(`GET`, "http://some-host.com/route", nil)
+	request.Header.Set(`Authorization`, `Bearer `+nestedToken)
+	responseExpected = `
+statusCode=200
+path: /less_claims/route
+query:
+headers:
+`
+	f(`
+users:
+- jwt:
+    skip_verify: true
+    match_claims:
+     team: ops
+     nested.scopes.1: "logs"
+     nested.department_id: "0"
+  url_map:
+    - src_paths: ["/route"]
+      url_prefix: {BACKEND}/more_claims
+- jwt:
+    skip_verify: true
+    match_claims:
+     team: dev
+     nested.team_permissions.write: "1"
+  url_map:
+    - src_paths: ["/route"]
+      url_prefix: {BACKEND}/less_claims
+`,
+		request,
+		responseExpected,
+	)
+
+	// use claim for routing, empty claim match
+	request = httptest.NewRequest(`GET`, "http://some-host.com/route", nil)
+	request.Header.Set(`Authorization`, `Bearer `+nestedToken)
+	responseExpected = `
+statusCode=200
+path: /empty/route
+query:
+headers:
+`
+	f(`
+users:
+- jwt:
+    skip_verify: true
+  url_map:
+    - src_paths: ["/route"]
+      url_prefix: {BACKEND}/empty
+- jwt:
+    skip_verify: true
+    match_claims:
+     team: ops
+     nested.team_permissions.write: "1"
+  url_map:
+    - src_paths: ["/route"]
+      url_prefix: {BACKEND}/ops
+`,
+		request,
+		responseExpected,
+	)
 
 }
 
+func TestOIDCRequestHandler(t *testing.T) {
+	privateKey, err := rsa.GenerateKey(rand.Reader, 2048)
+	if err != nil {
+		t.Fatalf("cannot generate RSA key: %s", err)
+	}
+
+	var oidcSrv *httptest.Server
+	oidcRespOK := atomic.Bool{}
+	oidcRespOK.Store(true)
+
+	oidcSrv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch r.URL.Path {
+		case "/.well-known/openid-configuration":
+			w.Header().Set("Content-Type", "application/json")
+			if err := json.NewEncoder(w).Encode(map[string]string{
+				"issuer":   oidcSrv.URL,
+				"jwks_uri": oidcSrv.URL + "/jwks",
+			}); err != nil {
+				panic(fmt.Errorf("cannot write openid-configuration response: %w", err))
+			}
+		case "/jwks":
+			if !oidcRespOK.Load() {
+				http.Error(w, "internal server error", http.StatusInternalServerError)
+				return
+			}
+
+			// Encode the RSA public key in JWK format (base64url, no padding)
+			nBytes := privateKey.N.Bytes()
+			eBytes := big.NewInt(int64(privateKey.E)).Bytes()
+			jwksBody := fmt.Sprintf(`{"keys":[{"kty":"RSA","kid":%q,"n":%q,"e":%q}]}`,
+				`test-key-id`,
+				base64.RawURLEncoding.EncodeToString(nBytes),
+				base64.RawURLEncoding.EncodeToString(eBytes),
+			)
+
+			w.Header().Set("Content-Type", "application/json")
+			if _, err := w.Write([]byte(jwksBody)); err != nil {
+				panic(fmt.Errorf("cannot write jwks response: %w", err))
+			}
+		default:
+			http.NotFound(w, r)
+		}
+	}))
+	defer oidcSrv.Close()
+
+	headerJSON, err := json.Marshal(map[string]any{
+		"alg": "RS256",
+		"typ": "JWT",
+		"iss": oidcSrv.URL,
+		"kid": `test-key-id`,
+	})
+	if err != nil {
+		t.Fatalf("cannot marshal JWT header: %s", err)
+	}
+	headerB64 := base64.RawURLEncoding.EncodeToString(headerJSON)
+
+	bodyJSON, err := json.Marshal(map[string]any{
+		"exp":       time.Now().Add(time.Minute).Unix(),
+		"iss":       oidcSrv.URL,
+		"vm_access": map[string]any{},
+	})
+	if err != nil {
+		t.Fatalf("cannot marshal JWT body: %s", err)
+	}
+	bodyB64 := base64.RawURLEncoding.EncodeToString(bodyJSON)
+
+	payload := headerB64 + "." + bodyB64
+
+	var signatureB64 string
+	hash := crypto.SHA256
+	h := hash.New()
+	h.Write([]byte(payload))
+	digest := h.Sum(nil)
+
+	signature, err := rsa.SignPKCS1v15(rand.Reader, privateKey, hash, digest)
+	if err != nil {
+		t.Fatalf("cannot sign JWT token: %s", err)
+	}
+	signatureB64 = base64.RawURLEncoding.EncodeToString(signature)
+
+	tkn := payload + "." + signatureB64
+
+	backSrv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+	}))
+	defer backSrv.Close()
+
+	f := func(responseExpected string) {
+		t.Helper()
+
+		cfgStr := `
+users:
+- jwt:
+    oidc:
+      issuer: ` + oidcSrv.URL + `
+  url_prefix: ` + backSrv.URL + `/
+`
+
+		cfgOrigP := authConfigData.Load()
+		if _, err := reloadAuthConfigData([]byte(cfgStr)); err != nil {
+			t.Fatalf("cannot load config data: %s", err)
+		}
+		defer func() {
+			cfgOrig := []byte("unauthorized_user:\n  url_prefix: http://foo/bar")
+			if cfgOrigP != nil {
+				cfgOrig = *cfgOrigP
+			}
+			if _, err := reloadAuthConfigData(cfgOrig); err != nil {
+				t.Fatalf("cannot restore original config: %s", err)
+			}
+		}()
+
+		r := httptest.NewRequest("GET", "http://some-host.com/api/v1/query", nil)
+		r.Header.Set("Authorization", "Bearer "+tkn)
+
+		w := &fakeResponseWriter{}
+		if !requestHandlerWithInternalRoutes(w, r) {
+			t.Fatalf("unexpected false returned from requestHandler")
+		}
+
+		if response := w.getResponse(); response != responseExpected {
+			t.Fatalf("unexpected response\ngot\n%s\nwant\n%s", response, responseExpected)
+		}
+	}
+
+	// successful
+	f(`statusCode=200
+`)
+
+	oidcRespOK.Store(false)
+	// OIDC server error
+	f(`statusCode=401
+Unauthorized
+`)
+}
+
 type fakeResponseWriter struct {
-	h http.Header
+	statusCode int
+	h          http.Header
 
 	bb bytes.Buffer
 }
@@ -1265,6 +1559,7 @@ func (w *fakeResponseWriter) Write(p []byte) (int, error) {
 }
 
 func (w *fakeResponseWriter) WriteHeader(statusCode int) {
+	w.statusCode = statusCode
 	fmt.Fprintf(&w.bb, "statusCode=%d\n", statusCode)
 	if w.h == nil {
 		return
@@ -1285,6 +1580,12 @@ func (w *fakeResponseWriter) SetReadDeadline(deadline time.Time) error {
 	return nil
 }
 
+func (w *fakeResponseWriter) reset() {
+	w.bb.Reset()
+	w.statusCode = 0
+	clear(w.h)
+}
+
 func TestBufferRequestBody_Success(t *testing.T) {
 	defaultRequestBufferSize := requestBufferSize.String()
 	defer func() {

app/vmauth/main_timing_test.go

@@ -0,0 +1,194 @@
+package main
+
+import (
+	"crypto"
+	"crypto/rand"
+	"crypto/rsa"
+	"crypto/x509"
+	"encoding/base64"
+	"encoding/json"
+	"encoding/pem"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+	"time"
+)
+
+func BenchmarkJWTRequestHandler(b *testing.B) {
+	// Generate RSA key pair for testing
+	privateKey, err := rsa.GenerateKey(rand.Reader, 2048)
+	if err != nil {
+		b.Fatalf("cannot generate RSA key: %s", err)
+	}
+
+	// Generate public key PEM
+	publicKeyBytes, err := x509.MarshalPKIXPublicKey(&privateKey.PublicKey)
+	if err != nil {
+		b.Fatalf("cannot marshal public key: %s", err)
+	}
+	publicKeyPEM := pem.EncodeToMemory(&pem.Block{
+		Type:  "PUBLIC KEY",
+		Bytes: publicKeyBytes,
+	})
+
+	genToken := func(t *testing.B, body map[string]any, valid bool) string {
+		t.Helper()
+
+		headerJSON, err := json.Marshal(map[string]any{
+			"alg": "RS256",
+			"typ": "JWT",
+		})
+		if err != nil {
+			t.Fatalf("cannot marshal header: %s", err)
+		}
+		headerB64 := base64.RawURLEncoding.EncodeToString(headerJSON)
+
+		bodyJSON, err := json.Marshal(body)
+		if err != nil {
+			t.Fatalf("cannot marshal body: %s", err)
+		}
+		bodyB64 := base64.RawURLEncoding.EncodeToString(bodyJSON)
+
+		payload := headerB64 + "." + bodyB64
+
+		var signatureB64 string
+		if valid {
+			// Create real RSA signature
+			hash := crypto.SHA256
+			h := hash.New()
+			h.Write([]byte(payload))
+			digest := h.Sum(nil)
+
+			signature, err := rsa.SignPKCS1v15(rand.Reader, privateKey, hash, digest)
+			if err != nil {
+				t.Fatalf("cannot sign token: %s", err)
+			}
+			signatureB64 = base64.RawURLEncoding.EncodeToString(signature)
+		} else {
+			signatureB64 = base64.RawURLEncoding.EncodeToString([]byte("invalid_signature"))
+		}
+
+		return payload + "." + signatureB64
+	}
+
+	f := func(name string, cfgStr string, r *http.Request, statusCodeExpected int) {
+		b.Helper()
+
+		b.ReportAllocs()
+		b.ResetTimer()
+		ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			w.WriteHeader(http.StatusOK)
+			if _, err := w.Write([]byte("path: " + r.URL.Path + "\n")); err != nil {
+				panic(fmt.Errorf("cannot write response: %w", err))
+			}
+		}))
+		defer ts.Close()
+
+		cfgStr = strings.ReplaceAll(cfgStr, "{BACKEND}", ts.URL)
+
+		cfgOrigP := authConfigData.Load()
+		if _, err := reloadAuthConfigData([]byte(cfgStr)); err != nil {
+			b.Fatalf("cannot load config data: %s", err)
+		}
+		defer func() {
+			cfgOrig := []byte("unauthorized_user:\n  url_prefix: http://foo/bar")
+			if cfgOrigP != nil {
+				cfgOrig = *cfgOrigP
+			}
+			_, err := reloadAuthConfigData(cfgOrig)
+			if err != nil {
+				b.Fatalf("cannot load the original config: %s", err)
+			}
+		}()
+
+		b.Run(name, func(b *testing.B) {
+			b.ResetTimer()
+			b.ReportAllocs()
+			b.RunParallel(func(pb *testing.PB) {
+				w := &fakeResponseWriter{}
+				for pb.Next() {
+					w.reset()
+					if !requestHandlerWithInternalRoutes(w, r) {
+						b.Fatalf("unexpected false is returned from requestHandler")
+					}
+					if w.statusCode != statusCodeExpected {
+						b.Fatalf("unexpected response code (-%d;+%d)", statusCodeExpected, w.statusCode)
+					}
+
+				}
+			})
+		})
+	}
+
+	simpleCfgStr := fmt.Sprintf(`
+users:
+- jwt:
+    public_keys:
+    - %q
+  url_prefix: {BACKEND}/foo`, string(publicKeyPEM))
+	noVMAccessClaimToken := genToken(b, nil, true)
+	expiredToken := genToken(b, map[string]any{
+		"exp":       10,
+		"vm_access": map[string]any{},
+	}, true)
+
+	fullToken := genToken(b, map[string]any{
+		"exp":   time.Now().Add(10 * time.Minute).Unix(),
+		"scope": "email id",
+		"vm_access": map[string]any{
+			"extra_labels": map[string]string{
+				"label":  "value1",
+				"label2": "value3",
+			},
+			"extra_filters":      []string{"stream_filter1", "stream_filter2"},
+			"metrics_account_id": 123,
+			"metrics_project_id": 234,
+			"metrics_extra_labels": []string{
+				"label1=value1",
+				"label2=value2",
+			},
+			"metrics_extra_filters": []string{
+				`{label3="value3"}`,
+				`{label4="value4"}`,
+			},
+			"logs_account_id": 345,
+			"logs_project_id": 456,
+			"logs_extra_filters": []string{
+				`{"namespace":"my-app","env":"prod"}`,
+			},
+			"logs_extra_stream_filters": []string{
+				`{"team":"dev"}`,
+			},
+		},
+	}, true)
+
+	// tenant headers are overwritten if set as placeholders
+	// extra_filters extra_stream_filters from vm_access claim merged with statically defined
+	request := httptest.NewRequest(`GET`, "http://some-host.com/query", nil)
+	request.Header.Set(`Authorization`, `Bearer `+fullToken)
+	f("full_template",
+		fmt.Sprintf(`
+users:
+- jwt:
+    public_keys:
+    - %q
+  headers:
+    - "AccountID: {{.LogsAccountID}}"
+    - "ProjectID: {{.LogsProjectID}}"
+  url_prefix: {BACKEND}/select/logsql/?extra_filters=aStaticFilter&extra_stream_filters=aStaticStreamFilter&extra_filters={{.LogsExtraFilters}}&extra_stream_filters={{.LogsExtraStreamFilters}}`, string(publicKeyPEM)),
+		request,
+		http.StatusOK,
+	)
+
+	// token without vm_access claim
+	request = httptest.NewRequest(`GET`, "http://some-host.com/abc", nil)
+	request.Header.Set(`Authorization`, `Bearer `+noVMAccessClaimToken)
+	f("token_without_claim", simpleCfgStr, request, http.StatusUnauthorized)
+
+	// expired token
+	request = httptest.NewRequest(`GET`, "http://some-host.com/abc", nil)
+	request.Header.Set(`Authorization`, `Bearer `+expiredToken)
+	f("expired_token", simpleCfgStr, request, http.StatusUnauthorized)
+}

app/vmauth/oidc.go

@@ -0,0 +1,195 @@
+package main
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"strings"
+	"sync"
+	"sync/atomic"
+	"time"
+
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/jwt"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/timeutil"
+)
+
+type oidcConfig struct {
+	Issuer string `yaml:"issuer"`
+}
+
+type oidcDiscovererPool struct {
+	ds map[string]*oidcDiscoverer
+
+	context context.Context
+	cancel  func()
+	wg      *sync.WaitGroup
+}
+
+func (dp *oidcDiscovererPool) createOrAdd(issuer string, vp *atomic.Pointer[jwt.VerifierPool]) {
+	if dp.ds == nil {
+		dp.ds = make(map[string]*oidcDiscoverer)
+		dp.context, dp.cancel = context.WithCancel(context.Background())
+		dp.wg = &sync.WaitGroup{}
+	}
+
+	ds, found := dp.ds[issuer]
+	if !found {
+		ds = &oidcDiscoverer{
+			issuer: issuer,
+		}
+		dp.ds[issuer] = ds
+	}
+
+	ds.vps = append(ds.vps, vp)
+}
+
+func (dp *oidcDiscovererPool) startDiscovery() {
+	if len(dp.ds) == 0 {
+		return
+	}
+
+	for _, d := range dp.ds {
+		dp.wg.Go(func() {
+			if err := d.refreshVerifierPools(dp.context); err != nil {
+				logger.Errorf("failed to initialize OIDC verifier pool at start for issuer %q: %s", d.issuer, err)
+			}
+		})
+	}
+	dp.wg.Wait()
+
+	for _, d := range dp.ds {
+		dp.wg.Go(func() {
+			d.run(dp.context)
+		})
+	}
+}
+
+func (dp *oidcDiscovererPool) stopDiscovery() {
+	if len(dp.ds) == 0 {
+		return
+	}
+
+	dp.cancel()
+	dp.wg.Wait()
+}
+
+type oidcDiscoverer struct {
+	issuer string
+	vps    []*atomic.Pointer[jwt.VerifierPool]
+}
+
+func (d *oidcDiscoverer) run(ctx context.Context) {
+	t := time.NewTimer(timeutil.AddJitterToDuration(time.Second * 10))
+	defer t.Stop()
+
+	for {
+		select {
+		case <-t.C:
+			if err := d.refreshVerifierPools(ctx); errors.Is(err, context.Canceled) {
+				return
+			} else if err != nil {
+				t.Reset(timeutil.AddJitterToDuration(time.Second * 10))
+				logger.Errorf("failed to refresh OIDC verifier pool for issuer %q: %v", d.issuer, err)
+				continue
+			}
+			// OIDC may return Cache-Control header with max-age directive.
+			// It could be used as time range for next refresh.
+			// https://openid.net/specs/openid-connect-core-1_0.html#RotateEncKeys
+			t.Reset(timeutil.AddJitterToDuration(time.Minute * 5))
+		case <-ctx.Done():
+			return
+		}
+	}
+}
+
+func (d *oidcDiscoverer) refreshVerifierPools(ctx context.Context) error {
+	cfg, err := getOpenIDConfiguration(ctx, d.issuer)
+	if err != nil {
+		return err
+	}
+	// The issuer in the OIDC configuration must match the expected issuer.
+	// https://openid.net/specs/openid-connect-core-1_0.html#RotateEncKeys
+	if cfg.Issuer != d.issuer {
+		return fmt.Errorf("openid configuration issuer %q does not match expected issuer %q", cfg.Issuer, d.issuer)
+	}
+
+	verifierPool, err := fetchAndParseJWKs(ctx, cfg.JWKsURI)
+	if err != nil {
+		return err
+	}
+
+	for _, vp := range d.vps {
+		vp.Store(verifierPool)
+	}
+	return nil
+}
+
+// See https://openid.net/specs/openid-connect-discovery-1_0.html#ProviderMetadata for details.
+type openidConfig struct {
+	Issuer  string `json:"issuer"`
+	JWKsURI string `json:"jwks_uri"`
+}
+
+var oidcHTTPClient = &http.Client{
+	Timeout: time.Second * 5,
+}
+
+func fetchAndParseJWKs(ctx context.Context, jwksURI string) (*jwt.VerifierPool, error) {
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, jwksURI, nil)
+	if err != nil {
+		return nil, fmt.Errorf("failed to create request for fetching jwks keys from %q: %w", jwksURI, err)
+	}
+
+	resp, err := oidcHTTPClient.Do(req)
+	if err != nil {
+		return nil, fmt.Errorf("failed to fetch jwks keys from %q: %w", jwksURI, err)
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		return nil, fmt.Errorf("unexpected status code %d when fetching jwks keys from %q", resp.StatusCode, jwksURI)
+	}
+
+	b, err := io.ReadAll(resp.Body)
+	if err != nil {
+		return nil, fmt.Errorf("failed to read response body from %q: %w", jwksURI, err)
+	}
+
+	vp, err := jwt.ParseJWKs(b)
+	if err != nil {
+		return nil, fmt.Errorf("failed to parse jwks keys from %q: %v", jwksURI, err)
+	}
+
+	return vp, nil
+}
+
+func getOpenIDConfiguration(ctx context.Context, issuer string) (openidConfig, error) {
+	issuer, _ = strings.CutSuffix(issuer, "/")
+	configURL := fmt.Sprintf("%s/.well-known/openid-configuration", issuer)
+
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, configURL, nil)
+	if err != nil {
+		return openidConfig{}, fmt.Errorf("failed to create request for fetching openid config from %q: %w", configURL, err)
+	}
+
+	resp, err := oidcHTTPClient.Do(req)
+	if err != nil {
+		return openidConfig{}, fmt.Errorf("failed to fetch openid config from %q: %w", configURL, err)
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		return openidConfig{}, fmt.Errorf("unexpected status code %d when fetching openid config from %q", resp.StatusCode, configURL)
+	}
+
+	var cfg openidConfig
+	if err := json.NewDecoder(resp.Body).Decode(&cfg); err != nil {
+		return openidConfig{}, fmt.Errorf("failed to decode openid config from %q: %s", configURL, err)
+	}
+
+	return cfg, nil
+}

app/vminsert/common/streamaggr.go

@@ -55,7 +55,7 @@ var (
 	deduplicator *streamaggr.Deduplicator
 )
 
-// CheckStreamAggrConfig checks config pointed by -stramaggr.config
+// CheckStreamAggrConfig checks config pointed by -streamaggr.config
 func CheckStreamAggrConfig() error {
 	if *streamAggrConfig == "" {
 		return nil

app/vminsert/datadogsketches/request_handler.go

@@ -45,15 +45,14 @@ func insertRows(sketches []*datadogsketches.Sketch, extraLabels []prompb.Label)
 		ms := sketch.ToSummary()
 		for _, m := range ms {
 			ctx.Labels = ctx.Labels[:0]
+			// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10557
+			ctx.AddLabel("host", sketch.Host) // newly added
 			ctx.AddLabel("", m.Name)
 			for _, label := range m.Labels {
 				ctx.AddLabel(label.Name, label.Value)
 			}
 			for _, tag := range sketch.Tags {
 				name, value := datadogutil.SplitTag(tag)
-				if name == "host" {
-					name = "exported_host"
-				}
 				ctx.AddLabel(name, value)
 			}
 			for j := range extraLabels {

app/vminsert/prompush/push.go

@@ -77,7 +77,7 @@ func push(ctx *common.InsertCtx, tss []prompb.TimeSeries) {
 			r := &ts.Samples[i]
 			metricNameRaw, err = ctx.WriteDataPointExt(metricNameRaw, ctx.Labels, r.Timestamp, r.Value)
 			if err != nil {
-				logger.Errorf("cannot write promscape data to storage: %s", err)
+				logger.Errorf("cannot write promscrape data to storage: %s", err)
 				return
 			}
 		}

app/vmrestore/main.go

@@ -30,6 +30,7 @@ var (
 	concurrency             = flag.Int("concurrency", 10, "The number of concurrent workers. Higher concurrency may reduce restore duration")
 	maxBytesPerSecond       = flagutil.NewBytes("maxBytesPerSecond", 0, "The maximum download speed. There is no limit if it is set to 0")
 	skipBackupCompleteCheck = flag.Bool("skipBackupCompleteCheck", false, "Whether to skip checking for 'backup complete' file in -src. This may be useful for restoring from old backups, which were created without 'backup complete' file")
+	SkipPreallocation       = flag.Bool("skipFilePreallocation", false, "Whether to skip pre-allocated files. This will likely be slower in most cases, but allows restores to resume mid file on failure")
 )
 
 func main() {
@@ -63,6 +64,7 @@ func main() {
 		Src:                     srcFS,
 		Dst:                     dstFS,
 		SkipBackupCompleteCheck: *skipBackupCompleteCheck,
+		SkipPreallocation:       *SkipPreallocation,
 	}
 	pushmetrics.Init()
 	if err := a.Run(ctx); err != nil {

app/vmselect/main.go

@@ -321,19 +321,23 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 		return true
 	case "/tags/tagSeries":
 		graphiteTagsTagSeriesRequests.Inc()
-		if err := graphite.TagsTagSeriesHandler(startTime, w, r); err != nil {
-			graphiteTagsTagSeriesErrors.Inc()
-			httpserver.Errorf(w, r, "%s", err)
-			return true
+		err := &httpserver.ErrorWithStatusCode{
+			Err: fmt.Errorf("graphite tag registration has been disabled and is planned to be removed in future. " +
+				"See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10544"),
+			StatusCode: http.StatusNotImplemented,
 		}
+		graphiteTagsTagSeriesErrors.Inc()
+		httpserver.Errorf(w, r, "%s", err)
 		return true
 	case "/tags/tagMultiSeries":
 		graphiteTagsTagMultiSeriesRequests.Inc()
-		if err := graphite.TagsTagMultiSeriesHandler(startTime, w, r); err != nil {
-			graphiteTagsTagMultiSeriesErrors.Inc()
-			httpserver.Errorf(w, r, "%s", err)
-			return true
+		err := &httpserver.ErrorWithStatusCode{
+			Err: fmt.Errorf("graphite tag registration has been disabled and is planned to be removed in future. " +
+				"See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10544"),
+			StatusCode: http.StatusNotImplemented,
 		}
+		graphiteTagsTagMultiSeriesErrors.Inc()
+		httpserver.Errorf(w, r, "%s", err)
 		return true
 	case "/tags":
 		graphiteTagsRequests.Inc()
@@ -739,6 +743,26 @@ func proxyVMAlertRequests(w http.ResponseWriter, r *http.Request, path string) {
 	req := r.Clone(r.Context())
 	req.URL.Path = strings.TrimPrefix(path, "prometheus")
 	req.Host = vmalertProxyHost
+
+	if strings.HasPrefix(r.Header.Get(`User-Agent`), `Grafana`) {
+		// Grafana currently supports only Prometheus-style alerts. If other alert types
+		// (e.g. logs or traces) are returned, it may fail with "Error loading alerts".
+		//
+		// Grafana queries the vmalert API directly, bypassing the VictoriaMetrics datasource,
+		// so query params (such as datasource_type) cannot be enforced on the Grafana side.
+		//
+		// To ensure compatibility, we detect Grafana requests via the User-Agent and enforce
+		// `datasource_type=prometheus`.
+		//
+		// See:
+		// - https://github.com/VictoriaMetrics/victoriametrics-datasource/issues/329#issuecomment-3847585443
+		// - https://github.com/VictoriaMetrics/victoriametrics-datasource/issues/59
+		q := req.URL.Query()
+		q.Set("datasource_type", "prometheus")
+		req.URL.RawQuery = q.Encode()
+		req.RequestURI = ""
+	}
+
 	vmalertProxy.ServeHTTP(w, req)
 }
 

app/vmselect/vmui/assets/rolldown-runtime-COnpUsM8.js

@@ -0,0 +1 @@
+var e=Object.create,t=Object.defineProperty,n=Object.getOwnPropertyDescriptor,r=Object.getOwnPropertyNames,i=Object.getPrototypeOf,a=Object.prototype.hasOwnProperty,o=(e,t)=>()=>(e&&(t=e(e=0)),t),s=(e,t)=>()=>(t||e((t={exports:{}}).exports,t),t.exports),c=(e,n)=>{let r={};for(var i in e)t(r,i,{get:e[i],enumerable:!0});return n||t(r,Symbol.toStringTag,{value:`Module`}),r},l=(e,i,o,s)=>{if(i&&typeof i==`object`||typeof i==`function`)for(var c=r(i),l=0,u=c.length,d;l<u;l++)d=c[l],!a.call(e,d)&&d!==o&&t(e,d,{get:(e=>i[e]).bind(null,d),enumerable:!(s=n(i,d))||s.enumerable});return e},u=(n,r,a)=>(a=n==null?{}:e(i(n)),l(r||!n||!n.__esModule?t(a,`default`,{value:n,enumerable:!0}):a,n)),d=e=>a.call(e,`module.exports`)?e[`module.exports`]:l(t({},`__esModule`,{value:!0}),e);export{u as a,d as i,o as n,c as r,s as t};

app/vmselect/vmui/assets/vendor-CnsZ1jie.css

@@ -0,0 +1 @@
+.uplot,.uplot *,.uplot :before,.uplot :after{box-sizing:border-box}.uplot{width:min-content;font-family:system-ui,-apple-system,Segoe UI,Roboto,Helvetica Neue,Arial,Noto Sans,sans-serif,Apple Color Emoji,Segoe UI Emoji,Segoe UI Symbol,Noto Color Emoji;line-height:1.5}.u-title{text-align:center;font-size:18px;font-weight:700}.u-wrap{-webkit-user-select:none;user-select:none;position:relative}.u-over,.u-under{position:absolute}.u-under{overflow:hidden}.uplot canvas{width:100%;height:100%;display:block;position:relative}.u-axis{position:absolute}.u-legend{text-align:center;margin:auto;font-size:14px}.u-inline{display:block}.u-inline *{display:inline-block}.u-inline tr{margin-right:16px}.u-legend th{font-weight:600}.u-legend th>*{vertical-align:middle;display:inline-block}.u-legend .u-marker{width:1em;height:1em;margin-right:4px;background-clip:padding-box!important}.u-inline.u-live th:after{content:":";vertical-align:middle}.u-inline:not(.u-live) .u-value{display:none}.u-series>*{padding:4px}.u-series th{cursor:pointer}.u-legend .u-off>*{opacity:.3}.u-select{pointer-events:none;background:#00000012;position:absolute}.u-cursor-x,.u-cursor-y{pointer-events:none;will-change:transform;position:absolute;top:0;left:0}.u-hz .u-cursor-x,.u-vt .u-cursor-y{border-right:1px dashed #607d8b;height:100%}.u-hz .u-cursor-y,.u-vt .u-cursor-x{border-bottom:1px dashed #607d8b;width:100%}.u-cursor-pt{pointer-events:none;will-change:transform;border:0 solid;border-radius:50%;position:absolute;top:0;left:0;background-clip:padding-box!important}.u-axis.u-off,.u-select.u-off,.u-cursor-x.u-off,.u-cursor-y.u-off,.u-cursor-pt.u-off{display:none}

app/vmselect/vmui/assets/vendor-D1GxaB_c.css

@@ -1 +0,0 @@
-.uplot,.uplot *,.uplot *:before,.uplot *:after{box-sizing:border-box}.uplot{font-family:system-ui,-apple-system,Segoe UI,Roboto,Helvetica Neue,Arial,Noto Sans,sans-serif,"Apple Color Emoji","Segoe UI Emoji",Segoe UI Symbol,"Noto Color Emoji";line-height:1.5;width:min-content}.u-title{text-align:center;font-size:18px;font-weight:700}.u-wrap{position:relative;-webkit-user-select:none;user-select:none}.u-over,.u-under{position:absolute}.u-under{overflow:hidden}.uplot canvas{display:block;position:relative;width:100%;height:100%}.u-axis{position:absolute}.u-legend{font-size:14px;margin:auto;text-align:center}.u-inline{display:block}.u-inline *{display:inline-block}.u-inline tr{margin-right:16px}.u-legend th{font-weight:600}.u-legend th>*{vertical-align:middle;display:inline-block}.u-legend .u-marker{width:1em;height:1em;margin-right:4px;background-clip:padding-box!important}.u-inline.u-live th:after{content:":";vertical-align:middle}.u-inline:not(.u-live) .u-value{display:none}.u-series>*{padding:4px}.u-series th{cursor:pointer}.u-legend .u-off>*{opacity:.3}.u-select{background:#00000012;position:absolute;pointer-events:none}.u-cursor-x,.u-cursor-y{position:absolute;left:0;top:0;pointer-events:none;will-change:transform}.u-hz .u-cursor-x,.u-vt .u-cursor-y{height:100%;border-right:1px dashed #607D8B}.u-hz .u-cursor-y,.u-vt .u-cursor-x{width:100%;border-bottom:1px dashed #607D8B}.u-cursor-pt{position:absolute;top:0;left:0;border-radius:50%;border:0 solid;pointer-events:none;will-change:transform;background-clip:padding-box!important}.u-axis.u-off,.u-select.u-off,.u-cursor-x.u-off,.u-cursor-y.u-off,.u-cursor-pt.u-off{display:none}

app/vmselect/vmui/index.html

@@ -37,10 +37,11 @@
   <meta property="og:title" content="UI for VictoriaMetrics">
   <meta property="og:url" content="https://victoriametrics.com/">
   <meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data">
-  <script type="module" crossorigin src="./assets/index-DIRuq0ns.js"></script>
-  <link rel="modulepreload" crossorigin href="./assets/vendor-BR6Q0Fin.js">
-  <link rel="stylesheet" crossorigin href="./assets/vendor-D1GxaB_c.css">
-  <link rel="stylesheet" crossorigin href="./assets/index-D7CzMv1O.css">
+  <script type="module" crossorigin src="./assets/index-KEOgEEMl.js"></script>
+  <link rel="modulepreload" crossorigin href="./assets/rolldown-runtime-COnpUsM8.js">
+  <link rel="modulepreload" crossorigin href="./assets/vendor-Mr0bmX1E.js">
+  <link rel="stylesheet" crossorigin href="./assets/vendor-CnsZ1jie.css">
+  <link rel="stylesheet" crossorigin href="./assets/index-D2OEy8Ra.css">
 </head>
 <body>
 <noscript>You need to enable JavaScript to run this app.</noscript>

app/vmstorage/main.go

@@ -319,6 +319,7 @@ func Stop() {
 	Storage.MustClose()
 	logger.Infof("successfully closed the storage in %.3f seconds", time.Since(startTime).Seconds())
 
+	fs.MustStopDirRemover()
 	logger.Infof("the storage has been stopped")
 }
 

app/vmui/Dockerfile-web

@@ -1,4 +1,4 @@
-FROM golang:1.26.0 AS build-web-stage
+FROM golang:1.26.1 AS build-web-stage
 COPY build /build
 
 WORKDIR /build

app/vmui/packages/vmui/package.json

@@ -21,43 +21,42 @@
   },
   "dependencies": {
     "classnames": "^2.5.1",
-    "dayjs": "^1.11.19",
+    "dayjs": "^1.11.20",
     "lodash.debounce": "^4.0.8",
-    "marked": "^17.0.1",
-    "preact": "^10.28.3",
-    "qs": "^6.14.1",
+    "marked": "^17.0.5",
+    "preact": "^10.29.0",
+    "qs": "^6.15.0",
     "react-input-mask": "^2.0.4",
-    "react-router-dom": "^7.13.0",
+    "react-router-dom": "^7.13.2",
     "uplot": "^1.6.32",
-    "vite": "^7.3.1",
+    "vite": "^8.0.2",
     "web-vitals": "^5.1.0"
   },
   "devDependencies": {
-    "@eslint/eslintrc": "^3.3.3",
+    "@eslint/eslintrc": "^3.3.5",
     "@eslint/js": "^9.39.2",
-    "@preact/preset-vite": "^2.10.3",
+    "@preact/preset-vite": "^2.10.5",
     "@testing-library/jest-dom": "^6.9.1",
     "@testing-library/preact": "^3.2.4",
     "@types/lodash.debounce": "^4.0.9",
-    "@types/node": "^25.2.0",
-    "@types/qs": "^6.14.0",
-    "@types/react": "^19.2.10",
+    "@types/node": "^25.5.0",
+    "@types/qs": "^6.15.0",
+    "@types/react": "^19.2.14",
     "@types/react-input-mask": "^3.0.6",
     "@types/react-router-dom": "^5.3.3",
-    "@typescript-eslint/eslint-plugin": "^8.54.0",
-    "@typescript-eslint/parser": "^8.54.0",
+    "@typescript-eslint/eslint-plugin": "^8.57.2",
+    "@typescript-eslint/parser": "^8.57.2",
     "cross-env": "^10.1.0",
     "eslint": "^9.39.2",
     "eslint-plugin-react": "^7.37.5",
-    "eslint-plugin-unused-imports": "^4.3.0",
-    "globals": "^17.3.0",
+    "eslint-plugin-unused-imports": "^4.4.1",
+    "globals": "^17.4.0",
     "http-proxy-middleware": "^3.0.5",
-    "jsdom": "^28.0.0",
-    "postcss": "^8.5.6",
-    "rollup-plugin-visualizer": "^6.0.5",
-    "sass-embedded": "^1.97.3",
+    "jsdom": "^29.0.1",
+    "postcss": "^8.5.8",
+    "sass-embedded": "^1.98.0",
     "typescript": "^5.9.3",
-    "vitest": "^4.0.18"
+    "vitest": "^4.1.1"
   },
   "browserslist": {
     "production": [

app/vmui/packages/vmui/src/api/explore-alerts.ts

@@ -1,5 +1,5 @@
-export const getGroupsUrl = (server: string): string => {
-  return `${server}/vmalert/api/v1/rules?datasource_type=prometheus`;
+export const getGroupsUrl = (server: string, search: string, type: string, states: string[], maxGroups: number): string => {
+  return `${server}/vmalert/api/v1/rules?datasource_type=prometheus&search=${encodeURIComponent(search)}&type=${encodeURIComponent(type)}&state=${states.map(encodeURIComponent).join(",")}&group_limit=${maxGroups}&extended_states=true`;
 };
 
 export const getItemUrl = (

app/vmui/packages/vmui/src/components/Configurators/QueryEditor/QueryEditorAutocomplete.tsx

@@ -60,7 +60,7 @@ const QueryEditorAutocomplete: FC<QueryEditorAutocompleteProps> = ({
   const options = useMemo(() => {
     switch (context) {
       case QueryContextType.metricsql:
-        return [...metrics, ...metricsqlFunctions];
+        return includeFunctions ? [...metrics, ...metricsqlFunctions] : metrics;
       case QueryContextType.label:
         return labels;
       case QueryContextType.labelValue:
@@ -68,7 +68,7 @@ const QueryEditorAutocomplete: FC<QueryEditorAutocompleteProps> = ({
       default:
         return [];
     }
-  }, [context, metrics, labels, labelValues, metricsqlFunctions]);
+  }, [context, metrics, labels, labelValues, metricsqlFunctions, includeFunctions]);
 
   const handleSelect = useCallback((insert: string) => {
     // Find the start and end of valueByContext in the query string

app/vmui/packages/vmui/src/components/ExploreAlerts/Badges/index.tsx

@@ -1,7 +1,7 @@
 import "./style.scss";
 import { ReactNode } from "react";
 
-export type BadgeColor = "firing" | "inactive" | "pending" | "no-match" | "unhealthy" | "ok" | "passive";
+export type BadgeColor = "firing" | "inactive" | "pending" | "nomatch" | "unhealthy" | "ok" | "passive";
 
 interface BadgeItem {
   value?: number | string;

app/vmui/packages/vmui/src/components/ExploreAlerts/Badges/style.scss

@@ -4,7 +4,7 @@ $badge-colors: (
   "firing": $color-error,
   "inactive": $color-success,
   "pending": $color-warning,
-  "no-match": $color-notice,
+  "nomatch": $color-notice,
   "unhealthy": $color-broken,
   "ok": $color-info,
   "passive": $color-passive,

app/vmui/packages/vmui/src/components/ExploreAlerts/BaseGroup/index.tsx

@@ -1,7 +1,8 @@
 import { useMemo } from "preact/compat";
 import "./style.scss";
 import { Group as APIGroup } from "../../../types";
-import { formatDuration, formatEventTime } from "../helpers";
+import ItemHeader from "../ItemHeader";
+import { getStates, formatDuration, formatEventTime } from "../helpers";
 import Badges, { BadgeColor } from "../Badges";
 
 interface BaseGroupProps {
@@ -117,6 +118,21 @@ const BaseGroup = ({ group }: BaseGroupProps) => {
           )}
         </tbody>
       </table>
+      <div className="vm-explore-alerts-rule-item">
+        <span className="vm-alerts-title">Rules</span>
+        {group.rules.map((rule) => (
+          <ItemHeader
+            classes={["vm-badge-item", rule.state]}
+            key={rule.id}
+            entity="rule"
+            type={rule.type}
+            groupId={rule.group_id}
+            states={getStates(rule)}
+            id={rule.id}
+            name={rule.name}
+          />
+        ))}
+      </div>
     </div>
   );
 };

app/vmui/packages/vmui/src/components/ExploreAlerts/ItemHeader/index.tsx

@@ -18,6 +18,7 @@ import {
 import Button from "../../Main/Button/Button";
 
 interface ItemHeaderControlsProps {
+  classes?: string[];
   entity: string;
   type?: string;
   groupId: string;
@@ -27,12 +28,19 @@ interface ItemHeaderControlsProps {
   onClose?: () => void;
 }
 
-const ItemHeader: FC<ItemHeaderControlsProps> = ({ name, id, groupId, entity, type, states, onClose }) => {
+const ItemHeader: FC<ItemHeaderControlsProps> = ({ name, id, groupId, entity, type, states, onClose, classes }) => {
   const { isMobile } = useDeviceDetect();
   const { serverUrl } = useAppState();
   const navigate = useNavigate();
   const copyToClipboard = useCopyToClipboard();
 
+  const openGroupLink = () => {
+    navigate({
+      pathname: "/rules",
+      search:   `group_id=${groupId}`,
+    });
+  };
+
   const openItemLink = () => {
     navigate({
       pathname: "/rules",
@@ -49,7 +57,7 @@ const ItemHeader: FC<ItemHeaderControlsProps> = ({ name, id, groupId, entity, ty
   const headerClasses = classNames({
     "vm-explore-alerts-item-header": true,
     "vm-explore-alerts-item-header_mobile": isMobile,
-  });
+  }, classes);
 
   const renderIcon = () => {
     switch(entity) {
@@ -105,16 +113,30 @@ const ItemHeader: FC<ItemHeaderControlsProps> = ({ name, id, groupId, entity, ty
           items={badgesItems}
         />
         {onClose ? (
-          <Button
-            className="vm-back-button"
-            size="small"
-            variant="outlined"
-            color="gray"
-            startIcon={<LinkIcon />}
-            onClick={copyLink}
-          >
-            <span className="vm-button-text">Copy Link</span>
-          </Button>
+          <>
+            {id && (
+              <Button
+                className="vm-back-button"
+                size="small"
+                variant="outlined"
+                color="gray"
+                startIcon={<GroupIcon />}
+                onClick={openGroupLink}
+              >
+                <span className="vm-button-text">Open Group</span>
+              </Button>
+            )}
+            <Button
+              className="vm-back-button"
+              size="small"
+              variant="outlined"
+              color="gray"
+              startIcon={<LinkIcon />}
+              onClick={copyLink}
+            >
+              <span className="vm-button-text">Copy Link</span>
+            </Button>
+          </>
         ) : (
           <Button
             className="vm-button-borderless"

app/vmui/packages/vmui/src/components/ExploreAlerts/ItemHeader/style.scss

@@ -6,6 +6,10 @@
   justify-content: space-between;
   gap: $padding-global;
 
+  &:is(.vm-badge-item) {
+    padding: 6px 0 6px 6px;
+  }
+
   .vm-button_small {
     padding: 4px;
   }

app/vmui/packages/vmui/src/components/ExploreAlerts/Pagination/index.tsx

@@ -0,0 +1,94 @@
+import Button from "../../Main/Button/Button";
+import { ArrowDownIcon } from "../../Main/Icons";
+import "./style.scss";
+import classNames from "classnames";
+
+interface PaginationProps {
+  page: number;
+  totalPages: number;
+  totalRules: number;
+  totalGroups: number;
+  pageRules: number;
+  pageGroups: number;
+  onPageChange: (num: number) => () => void;
+}
+
+const getButtons = (page: number, totalPages: number) => {
+  const result: number[] = [];
+  if (totalPages < 2) return result;
+  result.push(1);
+  if (page > 3) result.push(0);
+  if (page > 2) result.push(page - 1);
+  if (page > 1 && page < totalPages) result.push(page);
+  if (page > 0 && page < totalPages - 1) result.push(page + 1);
+  if (totalPages - page > 2) result.push(0);
+  result.push(totalPages);
+  return result;
+};
+
+const Pagination = ({
+  page,
+  totalPages,
+  onPageChange,
+  totalGroups,
+  totalRules,
+  pageGroups,
+  pageRules,
+}: PaginationProps) => {
+
+  const buttons = getButtons(page, totalPages);
+  return (
+    <>
+      <div
+        className="vm-pagination"
+      >
+        <span className="vm-pagination-stats">
+          <span>Page rules/groups:</span> <b>{pageRules}</b> / <b>{pageGroups}</b>
+        </span>
+        {!!buttons.length && (
+          <div className="vm-pagination-buttons">
+            <Button
+              className="vm-button-borderless vm-pagination-prev"
+              size="small"
+              color="gray"
+              disabled={page == 1}
+              variant="outlined"
+              startIcon={<ArrowDownIcon />}
+              onClick={onPageChange(page-1)}
+            />
+            {buttons.map((button, index) => {
+              return button ? (
+                <Button
+                  className={classNames({
+                    "vm-button-borderless": page !== button,
+                  })}
+                  key={index}
+                  size="small"
+                  color="gray"
+                  variant="outlined"
+                  onClick={onPageChange(button)}
+                >{button}</Button>
+              ) : (
+                <span className="vm-pagination-more">...</span>
+              );
+            })}
+            <Button
+              className="vm-button-borderless vm-pagination-next"
+              size="small"
+              color="gray"
+              disabled={page==totalPages}
+              variant="outlined"
+              startIcon={<ArrowDownIcon />}
+              onClick={onPageChange(page+1)}
+            />
+          </div>
+        )}
+        <span className="vm-pagination-stats">
+          <span>Total rules/groups:</span> <b>{totalRules}</b> / <b>{totalGroups}</b>
+        </span>
+      </div>
+    </>
+  );
+};
+
+export default Pagination;

app/vmui/packages/vmui/src/components/ExploreAlerts/Pagination/style.scss

@@ -0,0 +1,33 @@
+@use "src/styles/variables" as *;
+
+.vm-pagination {
+  display: flex;
+  min-height: 24px;
+  justify-content: space-between;
+  &-stats {
+    display: flex;
+    align-items: center;
+    color: var(--color-text-secondary);
+    column-gap: $padding-tiny;
+  }
+  &-buttons {
+    display: flex;
+    column-gap: $padding-small;
+  }
+  .vm-button-borderless {
+    border: 0;
+  }
+  &-more {
+    align-self: center;
+  }
+  &-prev {
+    svg {
+      transform: rotate(90deg);
+    }
+  }
+  &-next {
+    svg {
+      transform: rotate(-90deg);
+    }
+  }
+}

app/vmui/packages/vmui/src/components/ExploreAlerts/RulesHeader/index.tsx

@@ -1,4 +1,4 @@
-import { FC, useMemo } from "preact/compat";
+import { useMemo } from "preact/compat";
 import Select from "../../Main/Select/Select";
 import { SearchIcon } from "../../Main/Icons";
 import TextField from "../../Main/TextField/TextField";
@@ -8,25 +8,25 @@ import useDeviceDetect from "../../../hooks/useDeviceDetect";
 
 interface RulesHeaderProps {
   types: string[];
-  allTypes: string[];
+  allRuleTypes: string[];
   allStates: string[];
   states: string[];
   search: string;
-  onChangeTypes: (input: string) => void;
+  onChangeRuleType: (input: string) => void;
   onChangeStates: (input: string) => void;
   onChangeSearch: (input: string) => void;
 }
 
-const RulesHeader: FC<RulesHeaderProps> = ({
+const RulesHeader = ({
   types,
-  allTypes,
+  allRuleTypes,
   allStates,
   states,
   search,
-  onChangeTypes,
+  onChangeRuleType,
   onChangeStates,
   onChangeSearch,
-}) => {
+}: RulesHeaderProps) => {
   const noStateText = useMemo(
     () => (types.length ? "" : "No states. Please select rule states"),
     [types],
@@ -46,10 +46,10 @@ const RulesHeader: FC<RulesHeaderProps> = ({
         <div className="vm-explore-alerts-header__rule_type">
           <Select
             value={types}
-            list={allTypes}
-            label="Rules type"
+            list={allRuleTypes}
+            label="Rule type"
             placeholder="Please select rule type"
-            onChange={onChangeTypes}
+            onChange={onChangeRuleType}
             autofocus={!!types.length && !isMobile}
             includeAll
             searchable

app/vmui/packages/vmui/src/components/ExploreAlerts/helpers.ts

@@ -1,4 +1,5 @@
 import dayjs from "dayjs";
+import { Rule } from "../../types";
 
 export const formatDuration = (raw: number) => {
   const duration = dayjs.duration(Math.round(raw * 1000));
@@ -18,3 +19,13 @@ export const formatEventTime = (raw: string) => {
   const t = dayjs(raw);
   return t.year() <= 1 ? "Never" : t.format("DD MMM YYYY HH:mm:ss");
 };
+
+export const getStates = (rule: Rule) => {
+  if (!rule.alerts?.length) {
+    return { [rule.state]: 1 };
+  }
+  return rule.alerts.reduce((acc, alert) => {
+    acc[alert.state] = (acc[alert.state] ?? 0) + 1;
+    return acc;
+  }, {} as Record<string, number>);
+};

app/vmui/packages/vmui/src/components/ExploreMetrics/ExploreMetricGraph/ExploreMetricItemGraph.tsx

@@ -55,7 +55,7 @@ const ExploreMetricItem: FC<ExploreMetricItemGraphProps> = ({
 
     const base = `{${params.join(",")}}`;
     if (isBucket) {
-      return [`sum(rate(${base})) by (vmrange, le)`];
+      return [`sum(increase_pure(${base})) by (vmrange, le)`];
     }
     const queryBase = rateEnabled ? `rollup_rate(${base})` : `rollup(${base})`;
     return [`

app/vmui/packages/vmui/src/components/Main/TextField/TextField.tsx

@@ -27,6 +27,7 @@ interface TextFieldProps {
   endIcon?: ReactNode
   startIcon?: ReactNode
   disabled?: boolean
+  readonly?: boolean
   autofocus?: boolean
   helperText?: string
   inputmode?: "search" | "text" | "email" | "tel" | "url" | "none" | "numeric" | "decimal"
@@ -50,6 +51,7 @@ const TextField: FC<TextFieldProps> = ({
   endIcon,
   startIcon,
   disabled = false,
+  readonly = false,
   autofocus = false,
   inputmode = "text",
   caretPosition,
@@ -148,6 +150,7 @@ const TextField: FC<TextFieldProps> = ({
         <textarea
           className={inputClasses}
           disabled={disabled}
+          readOnly={readonly}
           ref={textareaRef}
           value={value}
           rows={1}
@@ -166,6 +169,7 @@ const TextField: FC<TextFieldProps> = ({
         <input
           className={inputClasses}
           disabled={disabled}
+          readOnly={readonly}
           ref={inputRef}
           value={value}
           type={type}

app/vmui/packages/vmui/src/hooks/useGetMetricsQL.tsx

@@ -72,9 +72,9 @@ const useGetMetricsQL = (includeFunctions: boolean) => {
       }
     };
     fetchMarkdown();
-  }, []);
+  }, [includeFunctions, metricsQLFunctions.length, queryDispatch]);
 
-  return includeFunctions ? metricsQLFunctions : [];
+  return metricsQLFunctions;
 };
 
 export default useGetMetricsQL;

app/vmui/packages/vmui/src/pages/CardinalityPanel/appConfigurator.ts

@@ -80,7 +80,7 @@ export default class AppConfigurator {
 
     let keys: string[] = [];
     if (focusLabel || isMetricWithLabel) {
-      keys = keys.concat("seriesCountByFocusLabelValue");
+      keys = keys.concat("seriesCountByMetricName", "seriesCountByFocusLabelValue");
     } else if (isMetric) {
       keys = keys.concat("labelValueCountByLabelName");
     } else if (isLabel) {

app/vmui/packages/vmui/src/pages/DownsamplingFilters/index.tsx

@@ -115,16 +115,20 @@ const DownsamplingFilters: FC = () => {
         </div>
         <div className="vm-downsampling-filters-body-top">
           <a
-            className="vm-link vm-link_with-icon"
             target="_blank"
             href="https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#downsampling"
             rel="help noreferrer"
           >
-            <WikiIcon/>
-            Documentation
+            <Button
+              variant="text"
+              color="gray"
+              startIcon={<WikiIcon/>}
+            >
+              Documentation
+            </Button>
           </a>
           <Button
-            variant="text"
+            variant="outlined"
             onClick={handleRunExample}
           >
             Try example
@@ -134,7 +138,7 @@ const DownsamplingFilters: FC = () => {
             onClick={handleApplyFilters}
             startIcon={<PlayIcon/>}
           >
-            Apply
+            Preview
           </Button>
         </div>
       </div>

app/vmui/packages/vmui/src/pages/ExploreAlerts/ExploreRule.tsx

@@ -6,7 +6,7 @@ import { Rule as APIRule } from "../../types";
 import ItemHeader from "../../components/ExploreAlerts/ItemHeader";
 import BaseRule from "../../components/ExploreAlerts/BaseRule";
 import Modal from "../../components/Main/Modal/Modal";
-import { getStates } from "./helpers";
+import { getStates } from "../../components/ExploreAlerts/helpers";
 
 interface ExploreRuleProps {
   groupId: string;

app/vmui/packages/vmui/src/pages/ExploreAlerts/ExploreRules.tsx

@@ -7,30 +7,36 @@ import Accordion from "../../components/Main/Accordion/Accordion";
 import { useFetchGroups } from "./hooks/useFetchGroups";
 import "./style.scss";
 import RulesHeader from "../../components/ExploreAlerts/RulesHeader";
+import Pagination from "../../components/ExploreAlerts/Pagination";
 import GroupHeader from "../../components/ExploreAlerts/GroupHeader";
 import Rule from "../../components/ExploreAlerts/Rule";
 import ExploreRule from "../../pages/ExploreAlerts/ExploreRule";
 import ExploreAlert from "../../pages/ExploreAlerts/ExploreAlert";
 import ExploreGroup from "../../pages/ExploreAlerts/ExploreGroup";
 import { getQueryStringValue } from "../../utils/query-string";
-import { getStates, getChanges, filterGroups } from "./helpers";
+import { getChanges } from "./helpers";
 import debounce from "lodash.debounce";
+import { getStates } from "../../components/ExploreAlerts/helpers";
 
-const defaultTypesStr = getQueryStringValue("types", "") as string;
-const defaultTypes = defaultTypesStr.split("&").filter((rt) => rt) as string[];
+const defaultRuleType = getQueryStringValue("type", "") as string;
 const defaultStatesStr = getQueryStringValue("states", "") as string;
 const defaultStates = defaultStatesStr.split("&").filter((s) => s) as string[];
 const defaultSearchInput = getQueryStringValue("search", "") as string;
+const TYPE_STATES: Record<string, string[]> = {
+  alert:  ["inactive", "firing", "nomatch", "pending", "unhealthy"],
+  record: ["unhealthy", "nomatch", "ok"],
+};
 
 const ExploreRules: FC = () => {
+  const pageNum = getQueryStringValue("page_num", "1") as string;
   const groupId = getQueryStringValue("group_id", "") as string;
   const ruleId = getQueryStringValue("rule_id", "") as string;
   const alertId = getQueryStringValue("alert_id", "") as string;
 
   const [searchInput, setSearchInput] = useState(defaultSearchInput);
-  const [types, setTypes] = useState(defaultTypes);
+  const [ruleType, setRuleType] = useState(defaultRuleType);
   const [states, setStates] = useState(defaultStates);
-  const [modalOpen, setModalOpen] = useState(true);
+  const [modalOpen, setModalOpen] = useState(false);
   const [searchParams, setSearchParams] = useSearchParams();
 
   useEffect(() => {
@@ -38,7 +44,7 @@ const ExploreRules: FC = () => {
   }, [groupId]);
 
   useSetQueryParams({
-    types: types.join("&"),
+    type: ruleType,
     states: states.join("&"),
     search: searchInput,
     group_id: groupId,
@@ -47,12 +53,11 @@ const ExploreRules: FC = () => {
   });
 
   const handleChangeSearch = useCallback((input: string) => {
-    if (!input) {
-      setSearchInput("");
-    } else {
-      setSearchInput(input);
-    }
-  }, [searchInput]);
+    const newParams = new URLSearchParams(searchParams);
+    newParams.set("page_num", "1");
+    setSearchParams(newParams);
+    setSearchInput(input || "");
+  }, [searchInput, searchParams]);
 
   const getModal = () => {
     if (ruleId) {
@@ -94,55 +99,79 @@ const ExploreRules: FC = () => {
     setModalOpen(false);
   };
 
+  const onPageChange = (num: number) => {
+    return () => {
+      const newParams = new URLSearchParams(searchParams);
+      newParams.set("page_num", num.toString());
+      setSearchParams(newParams);
+    };
+  };
+
+  const allRuleTypes = Object.keys(TYPE_STATES);
+  const allStates = useMemo(
+    () => Array.from(ruleType === "" ? new Set(Object.values(TYPE_STATES).flat()) : TYPE_STATES[ruleType] || []),
+    [ruleType]
+  );
+  const selectedRuleTypes = [ruleType].filter(Boolean);
+  useEffect(() => {
+    if (!states.every(v => allStates.includes(v))) {
+      setStates([]);
+    }
+  }, [states, allStates]);
+
+  const pageNumInt: number = Math.max(1, parseInt(pageNum, 10) || 1);
   const {
     groups,
     isLoading,
     error,
-  } = useFetchGroups({ blockFetch: modalOpen });
-
-  const { filteredGroups, allTypes, allStates } = useMemo(
-    () => filterGroups(groups || [], types, states, searchInput),
-    [groups, types, states, searchInput]
-  );
-
-  if (!types.every(v => allTypes.has(v))) {
-    setTypes([]);
-  }
-  const selectedTypes = allTypes.size === types.length ? [] : types;
-
-  if (!states.every(v => allStates.has(v))) {
-    setStates([]);
-  }
-  const selectedStates = allStates.size === states.length ? [] : states;
+    pageInfo,
+  } = useFetchGroups({ blockFetch: modalOpen, search: searchInput, ruleType, states, pageNum: pageNumInt, onPageChange });
 
   const handleChangeStates = useCallback((title: string) => {
-    setStates(getChanges(title, selectedStates));
-  }, [states]);
+    const newParams = new URLSearchParams(searchParams);
+    newParams.set("page_num", "1");
+    setSearchParams(newParams);
+    const changes = getChanges(title, states);
+    setStates(changes.length == allStates.length ? [] : changes);
+  }, [states, searchParams]);
 
-  const handleChangeTypes = useCallback((title: string) => {
-    setTypes(getChanges(title, selectedTypes));
-  }, [types]);
+  const handleChangeRuleType = useCallback((title: string) => {
+    const newParams = new URLSearchParams(searchParams);
+    newParams.set("page_num", "1");
+    setSearchParams(newParams);
+    const changes = getChanges(title, selectedRuleTypes);
+    setRuleType(changes.length && changes.length !== allRuleTypes.length ? changes[0] : "");
+  }, [ruleType, searchParams]);
 
   return (
     <>
       {modalOpen && getModal()}
-      {(!modalOpen || !!allStates?.size) && (
+      {(!modalOpen || !!allStates?.length) && (
         <div className="vm-explore-alerts">
           <RulesHeader
-            types={selectedTypes}
-            allTypes={Array.from(allTypes)}
-            states={selectedStates}
-            allStates={Array.from(allStates)}
+            types={selectedRuleTypes}
+            allRuleTypes={allRuleTypes}
+            states={states}
+            allStates={allStates}
             search={searchInput}
-            onChangeTypes={handleChangeTypes}
+            onChangeRuleType={handleChangeRuleType}
             onChangeStates={handleChangeStates}
             onChangeSearch={debounce(handleChangeSearch, 500)}
           />
+          <Pagination
+            page={pageInfo.page}
+            totalPages={pageInfo.total_pages}
+            pageRules={groups.reduce((total, g) => total + g?.rules.length, 0)}
+            pageGroups={groups.length}
+            totalRules={pageInfo.total_rules}
+            totalGroups={pageInfo.total_groups}
+            onPageChange={onPageChange}
+          />
           {(isLoading && <Spinner />) || (error && <Alert variant="error">{error}</Alert>) || (
-            !filteredGroups.length && <Alert variant="info">{noRuleFound}</Alert>
+            !groups.length && <Alert variant="info">{noRuleFound}</Alert>
           ) || (
             <div className="vm-explore-alerts-body">
-              {filteredGroups.map((group) => (
+              {groups.map((group) => (
                 <div
                   key={group.id}
                   className="vm-explore-alert-group vm-block vm-block_empty-padding"

app/vmui/packages/vmui/src/pages/ExploreAlerts/helpers.ts

@@ -1,5 +1,3 @@
-import { Rule, Group } from "../../types";
-
 export const getChanges = (title: string, prevValues: string[]): string[] => {
   if (title === "All") return [];
 
@@ -12,77 +10,3 @@ export const getChanges = (title: string, prevValues: string[]): string[] => {
 
   return Array.from(newValues);
 };
-
-export const getState = (rule: Rule) => {
-  let state = rule?.state || "ok";
-  if (rule?.health !== "ok") {
-    state = "unhealthy";
-  } else if (!rule?.lastSamples && !rule?.lastSeriesFetched) {
-    state = "no match";
-  }
-  return state;
-};
-
-export const getStates = (rule: Rule) => {
-  const output: Record<string, number> = {};
-  const alertsCount = rule?.alerts?.length || 0;
-  if (alertsCount > 0) {
-    rule.alerts.forEach((alert) => {
-      if (alert.state in output) {
-        output[alert.state] += 1;
-      } else {
-        output[alert.state] = 1;
-      }
-    });
-  } else {
-    output[getState(rule)] = 1;
-  }
-  return output;
-};
-
-export const filterGroups = (groups: Group[], types: string[], states: string[], searchInput: string) => {
-  const allTypes: Set<string> = new Set();
-  const allStates: Set<string> = new Set();
-  const filteredGroups: Group[] = [];
-
-  groups.forEach((group) => {
-    const filteredRules: Rule[] = [];
-    const statesPerGroup: Record<string, number> = {};
-    group.rules.forEach((rule) => {
-      const ruleType = rule.type.charAt(0).toUpperCase() + rule.type.slice(1);
-      allTypes.add(ruleType);
-      if (types?.length && !types.includes(ruleType)) return;
-
-      const state = getState(rule);
-      const stateName = state.charAt(0).toUpperCase() + state.slice(1);
-      allStates.add(stateName);
-      if (states?.length && !states.includes(stateName)) return;
-
-      if (
-        searchInput &&
-        !rule.name.toLowerCase().includes(searchInput.toLowerCase()) &&
-        !group.name.toLowerCase().includes(searchInput.toLowerCase()) &&
-        !group.file.toLowerCase().includes(searchInput.toLowerCase())
-      )
-        return;
-
-      filteredRules.push(rule);
-      if (state !== "no match" && state !== "unhealthy" && state !== "firing" && state !== "pending")
-        return;
-
-      const count = state === "firing" || state === "pending" ? rule?.alerts?.length : 1;
-      if (stateName in statesPerGroup) {
-        statesPerGroup[stateName] += count;
-      } else {
-        statesPerGroup[stateName] = count;
-      }
-    });
-    if (filteredRules.length) {
-      const g = Object.assign({}, group);
-      g.rules = filteredRules;
-      g.states = statesPerGroup;
-      filteredGroups.push(g);
-    }
-  });
-  return { filteredGroups, allTypes, allStates };
-};

app/vmui/packages/vmui/src/pages/ExploreAlerts/hooks/useFetchGroups.ts

@@ -1,46 +1,75 @@
-import { useTimeState } from "../../../state/time/TimeStateContext";
-import { useEffect, useMemo, useState } from "preact/compat";
+import { useMemo, useEffect, useState } from "preact/compat";
 import { getGroupsUrl } from "../../../api/explore-alerts";
 import { useAppState } from "../../../state/common/StateContext";
 import { ErrorTypes, Group } from "../../../types";
+import { useTimeState } from "../../../state/time/TimeStateContext";
 
 interface FetchGroupsReturn {
   groups: Group[];
   isLoading: boolean;
   error?: ErrorTypes | string;
+  pageInfo: PageInfo;
 }
 
 interface FetchGroupsProps {
-  blockFetch: boolean
+  blockFetch: boolean;
+  search: string;
+  ruleType: string;
+  states: string[];
+  pageNum: number;
+  onPageChange: (num: number) => () => void;
 }
 
-export const useFetchGroups = ({ blockFetch }: FetchGroupsProps): FetchGroupsReturn => {
+interface PageInfo {
+  page: number;
+  total_pages: number;
+  total_groups: number;
+  total_rules: number;
+}
+
+const MAX_GROUPS = 100;
+
+export const useFetchGroups = ({ blockFetch, pageNum, search, ruleType, states, onPageChange }: FetchGroupsProps): FetchGroupsReturn => {
   const { serverUrl } = useAppState();
   const { period } = useTimeState();
 
   const [groups, setGroups] = useState<Group[]>([]);
   const [isLoading, setIsLoading] = useState(false);
+  const [pageInfo, setPageInfo] = useState<PageInfo>({
+    page: pageNum,
+    total_pages: 1,
+    total_groups: 0,
+    total_rules: 0,
+  });
   const [error, setError] = useState<ErrorTypes | string>();
 
   const fetchUrl = useMemo(
-    () => getGroupsUrl(serverUrl),
-    [serverUrl],
+    () => getGroupsUrl(serverUrl, search, ruleType, states, MAX_GROUPS),
+    [serverUrl, search, ruleType, states],
   );
 
   const loaded = !!groups.length || !blockFetch;
-
   useEffect(() => {
     if (blockFetch) return;
     const fetchData = async () => {
       setIsLoading(true);
       try {
-        const response = await fetch(fetchUrl);
+        const url = `${fetchUrl}&page_num=${pageNum}`;
+        const response = await fetch(url);
         const resp = await response.json();
-
         if (response.ok) {
-          const data = (resp.data.groups || []) as Group[];
-          setGroups(data.sort((a, b) => a.name.localeCompare(b.name)));
+          const loadedGroups = (resp.data.groups || []) as Group[];
+          setGroups(loadedGroups);
+          setPageInfo({
+            page: resp.page || 1,
+            total_pages: resp.total_pages || 1,
+            total_groups: resp.total_groups || 0,
+            total_rules: resp.total_rules || 0,
+          });
           setError(undefined);
+        } else if (response.status === 400 && resp?.error?.includes("exceeds total amount of pages")) {
+          onPageChange(1)();
+          setError(`${resp.errorType}\r\n${resp?.error}`);
         } else {
           setError(`${resp.errorType}\r\n${resp?.error}`);
         }
@@ -51,9 +80,8 @@ export const useFetchGroups = ({ blockFetch }: FetchGroupsProps): FetchGroupsRet
       }
       setIsLoading(false);
     };
-
     fetchData().catch(console.error);
-  }, [fetchUrl, period, loaded]);
+  }, [fetchUrl, period, loaded, pageNum]);
 
-  return { groups, isLoading, error };
+  return { groups, isLoading, error, pageInfo };
 };

app/vmui/packages/vmui/src/pages/ExploreAlerts/hooks/useSetQueryParams.ts

@@ -3,7 +3,7 @@ import { compactObject } from "../../../utils/object";
 import useSearchParamsFromObject from "../../../hooks/useSearchParamsFromObject";
 
 interface rulesQueryProps {
-  types?: string;
+  type?: string;
   states?: string;
   search?: string;
   rule_id: string;
@@ -12,7 +12,7 @@ interface rulesQueryProps {
 }
 
 export const useRulesSetQueryParams = ({
-  types,
+  type,
   states,
   search,
   rule_id,
@@ -23,7 +23,7 @@ export const useRulesSetQueryParams = ({
 
   const setSearchParamsFromState = () => {
     const params = compactObject({
-      types,
+      type,
       states,
       search,
       alert_id,
@@ -35,7 +35,7 @@ export const useRulesSetQueryParams = ({
   };
 
   useEffect(setSearchParamsFromState, [
-    types,
+    type,
     states,
     search,
     rule_id,

app/vmui/packages/vmui/src/pages/ExploreAlerts/style.scss

@@ -17,6 +17,19 @@
   }
 }
 
+.vm-explore-alerts-load {
+  text-align: center;
+  color: var(--color-text-disabled);
+  button {
+    border: none;
+  }
+  &-before {
+    svg {
+      transform: rotate(180deg);
+    }
+  }
+}
+
 .vm-list-item-inner {
   display: flex;
   align-items: center;

app/vmui/packages/vmui/src/pages/Relabel/index.tsx

@@ -90,25 +90,33 @@ const Relabel: FC = () => {
         </div>
         <div className="vm-relabeling-header-bottom">
           <a
-            className="vm-link vm-link_with-icon"
             target="_blank"
             href="https://docs.victoriametrics.com/victoriametrics/relabeling/"
             rel="help noreferrer"
           >
-            <InfoIcon/>
-            Relabeling cookbook
+            <Button
+              variant="text"
+              color="gray"
+              startIcon={<InfoIcon/>}
+            >
+              Relabeling cookbook
+            </Button>
           </a>
           <a
-            className="vm-link vm-link_with-icon"
             target="_blank"
             href="https://docs.victoriametrics.com/victoriametrics/relabeling/"
             rel="help noreferrer"
           >
-            <WikiIcon/>
-            Documentation
+            <Button
+              variant="text"
+              color="gray"
+              startIcon={<WikiIcon/>}
+            >
+              Documentation
+            </Button>
           </a>
           <Button
-            variant="text"
+            variant="outlined"
             onClick={handleRunExample}
           >
             Try example
@@ -118,7 +126,7 @@ const Relabel: FC = () => {
             onClick={handleRunQuery}
             startIcon={<PlayIcon/>}
           >
-            Submit
+            Preview
           </Button>
         </div>
       </div>

app/vmui/packages/vmui/src/pages/Relabel/style.scss

@@ -33,7 +33,7 @@
       display: flex;
       align-items: center;
       justify-content: flex-end;
-      gap: $padding-global;
+      gap: $padding-small;
 
       a {
         color: $color-text-secondary;

app/vmui/packages/vmui/src/pages/RetentionFilters/index.tsx

@@ -107,16 +107,20 @@ const RetentionFilters: FC = () => {
         </div>
         <div className="vm-retention-filters-body-top">
           <a
-            className="vm-link vm-link_with-icon"
             target="_blank"
             href="https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#retention-filters"
             rel="help noreferrer"
           >
-            <WikiIcon/>
-            Documentation
+            <Button
+              variant="text"
+              color="gray"
+              startIcon={<WikiIcon/>}
+            >
+              Documentation
+            </Button>
           </a>
           <Button
-            variant="text"
+            variant="outlined"
             onClick={handleRunExample}
           >
             Try example
@@ -126,7 +130,7 @@ const RetentionFilters: FC = () => {
             onClick={handleApplyFilters}
             startIcon={<PlayIcon/>}
           >
-            Apply
+            Preview
           </Button>
         </div>
       </div>

app/vmui/packages/vmui/src/pages/WithTemplate/index.tsx

@@ -48,7 +48,7 @@ const WithTemplate: FC = () => {
             type="textarea"
             label="MetricsQL query after expanding WITH expressions and applying other optimizations"
             value={data}
-            disabled
+            readonly
           />
         </div>
         <div className="vm-with-template-body-top">

app/vmui/packages/vmui/src/types/index.ts

@@ -230,6 +230,7 @@ export interface Rule {
   debug: boolean;
   updates: RuleUpdate[];
   max_updates_entries: number;
+  states: Record<string, number>;
 }
 
 interface RuleUpdate {

app/vmui/packages/vmui/vite.config.ts

@@ -21,7 +21,7 @@ const getProxy = (): Record<string, ProxyOptions> | undefined => {
   };
 
   return {
-    "^/prometheus/(api|vmalert)/.*": { ...commonProxy },
+    "^/prometheus/.*": { ...commonProxy },
     "/prometheus/vmui/config.json": { ...commonProxy },
   };
 };

apptest/model.go

@@ -33,6 +33,8 @@ type PrometheusQuerier interface {
 	// separate interface or rename this interface to allow for multiple querier
 	// types.
 	GraphiteMetricsIndex(t *testing.T, opts QueryOpts) GraphiteMetricsIndexResponse
+	GraphiteTagsTagSeries(t *testing.T, record string, opts QueryOpts)
+	GraphiteTagsTagMultiSeries(t *testing.T, records []string, opts QueryOpts)
 }
 
 // Writer contains methods for writing new data

apptest/tests/graphite_test.go

@@ -60,3 +60,60 @@ func TestClusterMetricsIndex(t *testing.T) {
 
 	testMetricsIndex(tc.T(), sut)
 }
+
+// testTagSeries tests the registration of new time series in index.
+//
+// See https://graphite.readthedocs.io/en/stable/tags.html#adding-series-to-the-tagdb.
+func testTagSeries(tc *apptest.TestCase, sut apptest.PrometheusWriteQuerier, getStorageMetric func(string) int) {
+	t := tc.T()
+
+	assertNewTimeseriesCreatedTotal := func(want int) {
+		tc.Assert(&apptest.AssertOptions{
+			Msg: "unexpected vm_new_timeseries_created_total",
+			Got: func() any {
+				return getStorageMetric("vm_new_timeseries_created_total")
+			},
+			Want: want,
+		})
+	}
+
+	rec := "disk.used;rack=a1;datacenter=dc1;server=web01"
+	sut.GraphiteTagsTagSeries(t, rec, apptest.QueryOpts{})
+	assertNewTimeseriesCreatedTotal(0)
+
+	recs := []string{
+		"metric.yyy;t2=a;t1=b;t3=c",
+		"metric.zzz;t5=d;t4=e;t6=f",
+		"metric.xxx;t8=g;t7=h;t9=i",
+	}
+	sut.GraphiteTagsTagMultiSeries(t, recs, apptest.QueryOpts{})
+	assertNewTimeseriesCreatedTotal(0)
+}
+
+func TestSingleTagSeries(t *testing.T) {
+	tc := apptest.NewTestCase(t)
+	defer tc.Stop()
+
+	sut := tc.MustStartDefaultVmsingle()
+	getStorageMetric := func(name string) int {
+		return sut.GetIntMetric(t, name)
+	}
+
+	testTagSeries(tc, sut, getStorageMetric)
+}
+
+func TestClusterTagSeries(t *testing.T) {
+	tc := apptest.NewTestCase(t)
+	defer tc.Stop()
+
+	sut := tc.MustStartDefaultCluster()
+	getStorageMetric := func(name string) int {
+		var v int
+		for _, s := range sut.Vmstorages {
+			v += s.GetIntMetric(t, name)
+		}
+		return v
+	}
+
+	testTagSeries(tc, sut, getStorageMetric)
+}

apptest/tests/prometheus_mock_storage.go

@@ -22,7 +22,7 @@ func NewPrometheusMockStorage(series []*prompb.TimeSeries) *PrometheusMockStorag
 	return &PrometheusMockStorage{store: series}
 }
 
-// ReadMultiple implemnets the storage.ReadClient interface for reading time series data.
+// ReadMultiple implements the storage.ReadClient interface for reading time series data.
 func (ms *PrometheusMockStorage) ReadMultiple(ctx context.Context, queries []*prompb.Query, sortSeries bool) (storage.SeriesSet, error) {
 	if len(queries) != 1 {
 		panic(fmt.Errorf("reading multiple queries isn't implemented"))

apptest/vmselect.go

@@ -307,6 +307,37 @@ func (app *Vmselect) GraphiteMetricsIndex(t *testing.T, opts QueryOpts) Graphite
 	return index
 }
 
+// GraphiteTagsTagSeries is a test helper function that registers Graphite tags
+// for a single time series by sending a HTTP POST request to
+// /graphite/tags/tagSeries vmsingle endpoint.
+func (app *Vmselect) GraphiteTagsTagSeries(t *testing.T, record string, opts QueryOpts) {
+	t.Helper()
+
+	url := fmt.Sprintf("http://%s/select/%s/graphite/tags/tagSeries", app.httpListenAddr, opts.getTenant())
+	values := opts.asURLValues()
+	values.Add("path", record)
+
+	_, statusCode := app.cli.PostForm(t, url, values)
+	if got, want := statusCode, http.StatusNotImplemented; got != want {
+		t.Fatalf("unexpected status code: got %d, want %d", got, want)
+	}
+}
+
+func (app *Vmselect) GraphiteTagsTagMultiSeries(t *testing.T, records []string, opts QueryOpts) {
+	t.Helper()
+
+	url := fmt.Sprintf("http://%s/select/%s/graphite/tags/tagMultiSeries", app.httpListenAddr, opts.getTenant())
+	values := opts.asURLValues()
+	for _, rec := range records {
+		values.Add("path", rec)
+	}
+
+	_, statusCode := app.cli.PostForm(t, url, values)
+	if got, want := statusCode, http.StatusNotImplemented; got != want {
+		t.Fatalf("unexpected status code: got %d, want %d", got, want)
+	}
+}
+
 // APIV1AdminTenants sends a query to a /admin/tenants endpoint
 func (app *Vmselect) APIV1AdminTenants(t *testing.T) *AdminTenantsResponse {
 	t.Helper()

apptest/vmsingle.go

@@ -414,6 +414,37 @@ func (app *Vmsingle) GraphiteMetricsIndex(t *testing.T, _ QueryOpts) GraphiteMet
 	return index
 }
 
+// GraphiteTagsTagSeries is a test helper function that registers Graphite tags
+// for a single time series by sending a HTTP POST request to
+// /graphite/tags/tagSeries vmsingle endpoint.
+func (app *Vmsingle) GraphiteTagsTagSeries(t *testing.T, record string, opts QueryOpts) {
+	t.Helper()
+
+	url := fmt.Sprintf("http://%s/graphite/tags/tagSeries", app.httpListenAddr)
+	values := opts.asURLValues()
+	values.Add("path", record)
+
+	_, statusCode := app.cli.PostForm(t, url, values)
+	if got, want := statusCode, http.StatusNotImplemented; got != want {
+		t.Fatalf("unexpected status code: got %d, want %d", got, want)
+	}
+}
+
+func (app *Vmsingle) GraphiteTagsTagMultiSeries(t *testing.T, records []string, opts QueryOpts) {
+	t.Helper()
+
+	url := fmt.Sprintf("http://%s/graphite/tags/tagMultiSeries", app.httpListenAddr)
+	values := opts.asURLValues()
+	for _, rec := range records {
+		values.Add("path", rec)
+	}
+
+	_, statusCode := app.cli.PostForm(t, url, values)
+	if got, want := statusCode, http.StatusNotImplemented; got != want {
+		t.Fatalf("unexpected status code: got %d, want %d", got, want)
+	}
+}
+
 // APIV1StatusMetricNamesStats sends a query to a /api/v1/status/metric_names_stats endpoint
 // and returns the statistics response for given params.
 //

dashboards/vm/vmagent.json

@@ -51,7 +51,7 @@
   "editable": true,
   "fiscalYearStartMonth": 0,
   "graphTooltip": 1,
-  "id": 2,
+  "id": 3,
   "links": [
     {
       "icon": "doc",
@@ -1769,7 +1769,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 698
+            "y": 141
           },
           "id": 111,
           "options": {
@@ -1884,7 +1884,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 698
+            "y": 141
           },
           "id": 157,
           "options": {
@@ -1996,7 +1996,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 758
+            "y": 196
           },
           "id": 155,
           "options": {
@@ -2103,7 +2103,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 758
+            "y": 196
           },
           "id": 158,
           "options": {
@@ -2226,7 +2226,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 766
+            "y": 204
           },
           "id": 156,
           "options": {
@@ -2370,7 +2370,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 766
+            "y": 204
           },
           "id": 81,
           "options": {
@@ -2497,7 +2497,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 774
+            "y": 212
           },
           "id": 39,
           "options": {
@@ -2603,7 +2603,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 774
+            "y": 212
           },
           "id": 159,
           "options": {
@@ -2729,7 +2729,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 782
+            "y": 220
           },
           "id": 41,
           "options": {
@@ -2849,7 +2849,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 782
+            "y": 220
           },
           "id": 7,
           "options": {
@@ -2971,7 +2971,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 790
+            "y": 228
           },
           "id": 135,
           "options": {
@@ -3081,7 +3081,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 790
+            "y": 228
           },
           "id": 149,
           "options": {
@@ -3187,7 +3187,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 798
+            "y": 236
           },
           "id": 154,
           "options": {
@@ -3297,7 +3297,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 798
+            "y": 236
           },
           "id": 83,
           "options": {
@@ -3386,6 +3386,7 @@
                   "type": "linear"
                 },
                 "showPoints": "never",
+                "showValues": false,
                 "spanNulls": false,
                 "stacking": {
                   "group": "A",
@@ -3400,7 +3401,8 @@
                 "mode": "absolute",
                 "steps": [
                   {
-                    "color": "green"
+                    "color": "green",
+                    "value": 0
                   },
                   {
                     "color": "red",
@@ -3416,7 +3418,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 3116
+            "y": 142
           },
           "id": 92,
           "options": {
@@ -3438,7 +3440,7 @@
               "sort": "desc"
             }
           },
-          "pluginVersion": "11.5.0",
+          "pluginVersion": "12.2.0",
           "targets": [
             {
               "datasource": {
@@ -3454,7 +3456,7 @@
               "refId": "A"
             }
           ],
-          "title": "Top 10 jobs by unique samples",
+          "title": "Top 10 jobs by newly added series",
           "type": "timeseries"
         },
         {
@@ -3462,7 +3464,7 @@
             "type": "victoriametrics-metrics-datasource",
             "uid": "$ds"
           },
-          "description": "Shows top 10 instances by the number of new series registered by vmagent over the 5min range. These instances generate the most of the churn rate.",
+          "description": "Shows top 10 targets by the number of new series registered by vmagent over the 5min range. These instances generate the most of the churn rate.",
           "fieldConfig": {
             "defaults": {
               "color": {
@@ -3492,6 +3494,7 @@
                   "type": "linear"
                 },
                 "showPoints": "never",
+                "showValues": false,
                 "spanNulls": false,
                 "stacking": {
                   "group": "A",
@@ -3506,7 +3509,8 @@
                 "mode": "absolute",
                 "steps": [
                   {
-                    "color": "green"
+                    "color": "green",
+                    "value": 0
                   },
                   {
                     "color": "red",
@@ -3522,7 +3526,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 3116
+            "y": 142
           },
           "id": 95,
           "options": {
@@ -3544,7 +3548,7 @@
               "sort": "desc"
             }
           },
-          "pluginVersion": "11.5.0",
+          "pluginVersion": "12.2.0",
           "targets": [
             {
               "datasource": {
@@ -3553,14 +3557,14 @@
               },
               "editorMode": "code",
               "exemplar": false,
-              "expr": "topk(10, sum(sum_over_time(scrape_series_added[5m])) by (instance)) > 0",
+              "expr": "topk(10, sum(sum_over_time(scrape_series_added[5m])) by (job,instance)) > 0",
               "interval": "",
-              "legendFormat": "__auto",
+              "legendFormat": "{{job}}-{{instance}}",
               "range": true,
               "refId": "A"
             }
           ],
-          "title": "Top 10 instances by unique samples",
+          "title": "Top 10 targets by newly added series",
           "type": "timeseries"
         },
         {
@@ -3599,6 +3603,7 @@
                   "type": "linear"
                 },
                 "showPoints": "never",
+                "showValues": false,
                 "spanNulls": false,
                 "stacking": {
                   "group": "A",
@@ -3615,7 +3620,8 @@
                 "mode": "absolute",
                 "steps": [
                   {
-                    "color": "transparent"
+                    "color": "transparent",
+                    "value": 0
                   },
                   {
                     "color": "red",
@@ -3631,7 +3637,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 3124
+            "y": 150
           },
           "id": 98,
           "options": {
@@ -3653,7 +3659,7 @@
               "sort": "desc"
             }
           },
-          "pluginVersion": "11.5.0",
+          "pluginVersion": "12.2.0",
           "targets": [
             {
               "datasource": {
@@ -3708,6 +3714,7 @@
                   "type": "linear"
                 },
                 "showPoints": "never",
+                "showValues": false,
                 "spanNulls": false,
                 "stacking": {
                   "group": "A",
@@ -3724,7 +3731,8 @@
                 "mode": "absolute",
                 "steps": [
                   {
-                    "color": "transparent"
+                    "color": "transparent",
+                    "value": 0
                   },
                   {
                     "color": "red",
@@ -3740,7 +3748,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 3124
+            "y": 150
           },
           "id": 99,
           "options": {
@@ -3762,7 +3770,7 @@
               "sort": "none"
             }
           },
-          "pluginVersion": "11.5.0",
+          "pluginVersion": "12.2.0",
           "targets": [
             {
               "datasource": {
@@ -3816,6 +3824,7 @@
                   "type": "linear"
                 },
                 "showPoints": "never",
+                "showValues": false,
                 "spanNulls": false,
                 "stacking": {
                   "group": "A",
@@ -3832,7 +3841,8 @@
                 "mode": "absolute",
                 "steps": [
                   {
-                    "color": "green"
+                    "color": "green",
+                    "value": 0
                   },
                   {
                     "color": "red",
@@ -3848,7 +3858,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 3132
+            "y": 158
           },
           "id": 79,
           "options": {
@@ -3870,7 +3880,7 @@
               "sort": "none"
             }
           },
-          "pluginVersion": "11.5.0",
+          "pluginVersion": "12.2.0",
           "targets": [
             {
               "datasource": {
@@ -3924,6 +3934,7 @@
                   "type": "linear"
                 },
                 "showPoints": "never",
+                "showValues": false,
                 "spanNulls": false,
                 "stacking": {
                   "group": "A",
@@ -3940,7 +3951,8 @@
                 "mode": "absolute",
                 "steps": [
                   {
-                    "color": "green"
+                    "color": "green",
+                    "value": 0
                   },
                   {
                     "color": "red",
@@ -3956,7 +3968,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 3132
+            "y": 158
           },
           "id": 18,
           "links": [
@@ -3985,7 +3997,7 @@
               "sort": "none"
             }
           },
-          "pluginVersion": "11.5.0",
+          "pluginVersion": "12.2.0",
           "targets": [
             {
               "datasource": {
@@ -4070,7 +4082,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 3140
+            "y": 166
           },
           "id": 127,
           "options": {
@@ -4176,7 +4188,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 3140
+            "y": 166
           },
           "id": 50,
           "options": {
@@ -4278,7 +4290,7 @@
             "h": 7,
             "w": 12,
             "x": 0,
-            "y": 3148
+            "y": 174
           },
           "id": 129,
           "options": {
@@ -4413,7 +4425,7 @@
             "h": 7,
             "w": 12,
             "x": 12,
-            "y": 3148
+            "y": 174
           },
           "id": 150,
           "options": {
@@ -4516,7 +4528,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 3155
+            "y": 181
           },
           "id": 151,
           "options": {
@@ -4637,7 +4649,7 @@
             "h": 7,
             "w": 12,
             "x": 0,
-            "y": 3361
+            "y": 4209
           },
           "id": 48,
           "options": {
@@ -4745,7 +4757,7 @@
             "h": 7,
             "w": 12,
             "x": 12,
-            "y": 3361
+            "y": 4209
           },
           "id": 76,
           "options": {
@@ -4851,7 +4863,7 @@
             "h": 7,
             "w": 12,
             "x": 0,
-            "y": 3368
+            "y": 4216
           },
           "id": 132,
           "options": {
@@ -4959,7 +4971,7 @@
             "h": 7,
             "w": 12,
             "x": 12,
-            "y": 3368
+            "y": 4216
           },
           "id": 133,
           "options": {
@@ -5066,7 +5078,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 3375
+            "y": 4223
           },
           "id": 20,
           "options": {
@@ -5172,7 +5184,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 3375
+            "y": 4223
           },
           "id": 126,
           "options": {
@@ -5277,7 +5289,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 3383
+            "y": 4231
           },
           "id": 46,
           "options": {
@@ -5382,7 +5394,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 3383
+            "y": 4231
           },
           "id": 148,
           "options": {
@@ -5487,7 +5499,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 3391
+            "y": 4239
           },
           "id": 31,
           "options": {
@@ -5654,7 +5666,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 3083
+            "y": 3931
           },
           "id": 73,
           "options": {
@@ -5771,7 +5783,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 3083
+            "y": 3931
           },
           "id": 131,
           "options": {
@@ -5875,7 +5887,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 3359
+            "y": 4207
           },
           "id": 130,
           "options": {
@@ -5992,7 +6004,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 3359
+            "y": 4207
           },
           "id": 77,
           "options": {
@@ -6117,7 +6129,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 3406
+            "y": 4254
           },
           "id": 146,
           "options": {
@@ -6219,7 +6231,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 3406
+            "y": 4254
           },
           "id": 143,
           "options": {
@@ -6315,7 +6327,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 3414
+            "y": 4262
           },
           "id": 147,
           "options": {
@@ -6418,7 +6430,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 3414
+            "y": 4262
           },
           "id": 139,
           "options": {
@@ -6529,7 +6541,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 3422
+            "y": 4270
           },
           "id": 142,
           "options": {
@@ -6626,7 +6638,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 3422
+            "y": 4270
           },
           "id": 137,
           "options": {
@@ -6739,7 +6751,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 3462
+            "y": 4310
           },
           "id": 141,
           "options": {
@@ -6869,7 +6881,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 1411
+            "y": 2259
           },
           "id": 60,
           "options": {
@@ -6977,7 +6989,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 1411
+            "y": 2259
           },
           "id": 66,
           "options": {
@@ -7085,7 +7097,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 1419
+            "y": 2267
           },
           "id": 61,
           "options": {
@@ -7193,7 +7205,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 1419
+            "y": 2267
           },
           "id": 65,
           "options": {
@@ -7300,7 +7312,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 1427
+            "y": 2275
           },
           "id": 88,
           "options": {
@@ -7404,7 +7416,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 1427
+            "y": 2275
           },
           "id": 84,
           "options": {
@@ -7511,7 +7523,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 1435
+            "y": 2283
           },
           "id": 90,
           "options": {
@@ -7569,7 +7581,7 @@
             "h": 2,
             "w": 24,
             "x": 0,
-            "y": 70
+            "y": 918
           },
           "id": 115,
           "options": {
@@ -7651,7 +7663,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 72
+            "y": 920
           },
           "id": 119,
           "options": {
@@ -7759,7 +7771,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 72
+            "y": 920
           },
           "id": 117,
           "options": {
@@ -7869,7 +7881,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 80
+            "y": 928
           },
           "id": 125,
           "links": [
@@ -7995,7 +8007,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 80
+            "y": 928
           },
           "id": 123,
           "options": {
@@ -8129,7 +8141,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 88
+            "y": 936
           },
           "id": 121,
           "options": {
@@ -8256,7 +8268,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 88
+            "y": 936
           },
           "id": 161,
           "links": [
@@ -8378,9 +8390,9 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 461
+            "y": 1309
           },
-          "id": 154,
+          "id": 162,
           "options": {
             "legend": {
               "calcs": [
@@ -8537,4 +8549,4 @@
   "title": "VictoriaMetrics - vmagent (VM)",
   "uid": "G7Z9GzMGz_vm",
   "version": 1
-}
+}

dashboards/vm/vmauth.json

@@ -1612,7 +1612,7 @@
                 "type": "victoriametrics-metrics-datasource",
                 "uid": "$ds"
               },
-              "expr": "sum(go_memstats_sys_bytes{job=~\"$job\", instance=~\"$instance\"}) + sum(vm_cache_size_bytes{job=~\"$job\", instance=~\"$instance\"})",
+              "expr": "sum(go_memstats_sys_bytes{job=~\"$job\", instance=~\"$instance\"})",
               "format": "time_series",
               "hide": false,
               "intervalFactor": 1,
@@ -1624,7 +1624,7 @@
                 "type": "victoriametrics-metrics-datasource",
                 "uid": "$ds"
               },
-              "expr": "sum(go_memstats_heap_inuse_bytes{job=~\"$job\", instance=~\"$instance\"}) + sum(vm_cache_size_bytes{job=~\"$job\", instance=~\"$instance\"})",
+              "expr": "sum(go_memstats_heap_inuse_bytes{job=~\"$job\", instance=~\"$instance\"})",
               "format": "time_series",
               "hide": false,
               "intervalFactor": 1,

dashboards/vmagent.json

@@ -50,7 +50,7 @@
   "editable": true,
   "fiscalYearStartMonth": 0,
   "graphTooltip": 1,
-  "id": 2,
+  "id": 3,
   "links": [
     {
       "icon": "doc",
@@ -1768,7 +1768,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 698
+            "y": 141
           },
           "id": 111,
           "options": {
@@ -1883,7 +1883,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 698
+            "y": 141
           },
           "id": 157,
           "options": {
@@ -1995,7 +1995,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 758
+            "y": 196
           },
           "id": 155,
           "options": {
@@ -2102,7 +2102,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 758
+            "y": 196
           },
           "id": 158,
           "options": {
@@ -2225,7 +2225,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 766
+            "y": 204
           },
           "id": 156,
           "options": {
@@ -2369,7 +2369,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 766
+            "y": 204
           },
           "id": 81,
           "options": {
@@ -2496,7 +2496,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 774
+            "y": 212
           },
           "id": 39,
           "options": {
@@ -2602,7 +2602,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 774
+            "y": 212
           },
           "id": 159,
           "options": {
@@ -2728,7 +2728,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 782
+            "y": 220
           },
           "id": 41,
           "options": {
@@ -2848,7 +2848,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 782
+            "y": 220
           },
           "id": 7,
           "options": {
@@ -2970,7 +2970,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 790
+            "y": 228
           },
           "id": 135,
           "options": {
@@ -3080,7 +3080,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 790
+            "y": 228
           },
           "id": 149,
           "options": {
@@ -3186,7 +3186,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 798
+            "y": 236
           },
           "id": 154,
           "options": {
@@ -3296,7 +3296,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 798
+            "y": 236
           },
           "id": 83,
           "options": {
@@ -3385,6 +3385,7 @@
                   "type": "linear"
                 },
                 "showPoints": "never",
+                "showValues": false,
                 "spanNulls": false,
                 "stacking": {
                   "group": "A",
@@ -3399,7 +3400,8 @@
                 "mode": "absolute",
                 "steps": [
                   {
-                    "color": "green"
+                    "color": "green",
+                    "value": 0
                   },
                   {
                     "color": "red",
@@ -3415,7 +3417,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 3116
+            "y": 142
           },
           "id": 92,
           "options": {
@@ -3437,7 +3439,7 @@
               "sort": "desc"
             }
           },
-          "pluginVersion": "11.5.0",
+          "pluginVersion": "12.2.0",
           "targets": [
             {
               "datasource": {
@@ -3453,7 +3455,7 @@
               "refId": "A"
             }
           ],
-          "title": "Top 10 jobs by unique samples",
+          "title": "Top 10 jobs by newly added series",
           "type": "timeseries"
         },
         {
@@ -3461,7 +3463,7 @@
             "type": "prometheus",
             "uid": "$ds"
           },
-          "description": "Shows top 10 instances by the number of new series registered by vmagent over the 5min range. These instances generate the most of the churn rate.",
+          "description": "Shows top 10 targets by the number of new series registered by vmagent over the 5min range. These instances generate the most of the churn rate.",
           "fieldConfig": {
             "defaults": {
               "color": {
@@ -3491,6 +3493,7 @@
                   "type": "linear"
                 },
                 "showPoints": "never",
+                "showValues": false,
                 "spanNulls": false,
                 "stacking": {
                   "group": "A",
@@ -3505,7 +3508,8 @@
                 "mode": "absolute",
                 "steps": [
                   {
-                    "color": "green"
+                    "color": "green",
+                    "value": 0
                   },
                   {
                     "color": "red",
@@ -3521,7 +3525,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 3116
+            "y": 142
           },
           "id": 95,
           "options": {
@@ -3543,7 +3547,7 @@
               "sort": "desc"
             }
           },
-          "pluginVersion": "11.5.0",
+          "pluginVersion": "12.2.0",
           "targets": [
             {
               "datasource": {
@@ -3552,14 +3556,14 @@
               },
               "editorMode": "code",
               "exemplar": false,
-              "expr": "topk(10, sum(sum_over_time(scrape_series_added[5m])) by (instance)) > 0",
+              "expr": "topk(10, sum(sum_over_time(scrape_series_added[5m])) by (job,instance)) > 0",
               "interval": "",
-              "legendFormat": "__auto",
+              "legendFormat": "{{job}}-{{instance}}",
               "range": true,
               "refId": "A"
             }
           ],
-          "title": "Top 10 instances by unique samples",
+          "title": "Top 10 targets by newly added series",
           "type": "timeseries"
         },
         {
@@ -3598,6 +3602,7 @@
                   "type": "linear"
                 },
                 "showPoints": "never",
+                "showValues": false,
                 "spanNulls": false,
                 "stacking": {
                   "group": "A",
@@ -3614,7 +3619,8 @@
                 "mode": "absolute",
                 "steps": [
                   {
-                    "color": "transparent"
+                    "color": "transparent",
+                    "value": 0
                   },
                   {
                     "color": "red",
@@ -3630,7 +3636,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 3124
+            "y": 150
           },
           "id": 98,
           "options": {
@@ -3652,7 +3658,7 @@
               "sort": "desc"
             }
           },
-          "pluginVersion": "11.5.0",
+          "pluginVersion": "12.2.0",
           "targets": [
             {
               "datasource": {
@@ -3707,6 +3713,7 @@
                   "type": "linear"
                 },
                 "showPoints": "never",
+                "showValues": false,
                 "spanNulls": false,
                 "stacking": {
                   "group": "A",
@@ -3723,7 +3730,8 @@
                 "mode": "absolute",
                 "steps": [
                   {
-                    "color": "transparent"
+                    "color": "transparent",
+                    "value": 0
                   },
                   {
                     "color": "red",
@@ -3739,7 +3747,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 3124
+            "y": 150
           },
           "id": 99,
           "options": {
@@ -3761,7 +3769,7 @@
               "sort": "none"
             }
           },
-          "pluginVersion": "11.5.0",
+          "pluginVersion": "12.2.0",
           "targets": [
             {
               "datasource": {
@@ -3815,6 +3823,7 @@
                   "type": "linear"
                 },
                 "showPoints": "never",
+                "showValues": false,
                 "spanNulls": false,
                 "stacking": {
                   "group": "A",
@@ -3831,7 +3840,8 @@
                 "mode": "absolute",
                 "steps": [
                   {
-                    "color": "green"
+                    "color": "green",
+                    "value": 0
                   },
                   {
                     "color": "red",
@@ -3847,7 +3857,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 3132
+            "y": 158
           },
           "id": 79,
           "options": {
@@ -3869,7 +3879,7 @@
               "sort": "none"
             }
           },
-          "pluginVersion": "11.5.0",
+          "pluginVersion": "12.2.0",
           "targets": [
             {
               "datasource": {
@@ -3923,6 +3933,7 @@
                   "type": "linear"
                 },
                 "showPoints": "never",
+                "showValues": false,
                 "spanNulls": false,
                 "stacking": {
                   "group": "A",
@@ -3939,7 +3950,8 @@
                 "mode": "absolute",
                 "steps": [
                   {
-                    "color": "green"
+                    "color": "green",
+                    "value": 0
                   },
                   {
                     "color": "red",
@@ -3955,7 +3967,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 3132
+            "y": 158
           },
           "id": 18,
           "links": [
@@ -3984,7 +3996,7 @@
               "sort": "none"
             }
           },
-          "pluginVersion": "11.5.0",
+          "pluginVersion": "12.2.0",
           "targets": [
             {
               "datasource": {
@@ -4069,7 +4081,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 3140
+            "y": 166
           },
           "id": 127,
           "options": {
@@ -4175,7 +4187,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 3140
+            "y": 166
           },
           "id": 50,
           "options": {
@@ -4277,7 +4289,7 @@
             "h": 7,
             "w": 12,
             "x": 0,
-            "y": 3148
+            "y": 174
           },
           "id": 129,
           "options": {
@@ -4412,7 +4424,7 @@
             "h": 7,
             "w": 12,
             "x": 12,
-            "y": 3148
+            "y": 174
           },
           "id": 150,
           "options": {
@@ -4515,7 +4527,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 3155
+            "y": 181
           },
           "id": 151,
           "options": {
@@ -4636,7 +4648,7 @@
             "h": 7,
             "w": 12,
             "x": 0,
-            "y": 3361
+            "y": 4209
           },
           "id": 48,
           "options": {
@@ -4744,7 +4756,7 @@
             "h": 7,
             "w": 12,
             "x": 12,
-            "y": 3361
+            "y": 4209
           },
           "id": 76,
           "options": {
@@ -4850,7 +4862,7 @@
             "h": 7,
             "w": 12,
             "x": 0,
-            "y": 3368
+            "y": 4216
           },
           "id": 132,
           "options": {
@@ -4958,7 +4970,7 @@
             "h": 7,
             "w": 12,
             "x": 12,
-            "y": 3368
+            "y": 4216
           },
           "id": 133,
           "options": {
@@ -5065,7 +5077,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 3375
+            "y": 4223
           },
           "id": 20,
           "options": {
@@ -5171,7 +5183,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 3375
+            "y": 4223
           },
           "id": 126,
           "options": {
@@ -5276,7 +5288,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 3383
+            "y": 4231
           },
           "id": 46,
           "options": {
@@ -5381,7 +5393,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 3383
+            "y": 4231
           },
           "id": 148,
           "options": {
@@ -5486,7 +5498,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 3391
+            "y": 4239
           },
           "id": 31,
           "options": {
@@ -5653,7 +5665,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 3083
+            "y": 3931
           },
           "id": 73,
           "options": {
@@ -5770,7 +5782,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 3083
+            "y": 3931
           },
           "id": 131,
           "options": {
@@ -5874,7 +5886,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 3359
+            "y": 4207
           },
           "id": 130,
           "options": {
@@ -5991,7 +6003,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 3359
+            "y": 4207
           },
           "id": 77,
           "options": {
@@ -6116,7 +6128,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 3406
+            "y": 4254
           },
           "id": 146,
           "options": {
@@ -6218,7 +6230,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 3406
+            "y": 4254
           },
           "id": 143,
           "options": {
@@ -6314,7 +6326,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 3414
+            "y": 4262
           },
           "id": 147,
           "options": {
@@ -6417,7 +6429,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 3414
+            "y": 4262
           },
           "id": 139,
           "options": {
@@ -6528,7 +6540,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 3422
+            "y": 4270
           },
           "id": 142,
           "options": {
@@ -6625,7 +6637,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 3422
+            "y": 4270
           },
           "id": 137,
           "options": {
@@ -6738,7 +6750,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 3462
+            "y": 4310
           },
           "id": 141,
           "options": {
@@ -6868,7 +6880,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 1411
+            "y": 2259
           },
           "id": 60,
           "options": {
@@ -6976,7 +6988,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 1411
+            "y": 2259
           },
           "id": 66,
           "options": {
@@ -7084,7 +7096,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 1419
+            "y": 2267
           },
           "id": 61,
           "options": {
@@ -7192,7 +7204,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 1419
+            "y": 2267
           },
           "id": 65,
           "options": {
@@ -7299,7 +7311,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 1427
+            "y": 2275
           },
           "id": 88,
           "options": {
@@ -7403,7 +7415,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 1427
+            "y": 2275
           },
           "id": 84,
           "options": {
@@ -7510,7 +7522,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 1435
+            "y": 2283
           },
           "id": 90,
           "options": {
@@ -7568,7 +7580,7 @@
             "h": 2,
             "w": 24,
             "x": 0,
-            "y": 70
+            "y": 918
           },
           "id": 115,
           "options": {
@@ -7650,7 +7662,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 72
+            "y": 920
           },
           "id": 119,
           "options": {
@@ -7758,7 +7770,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 72
+            "y": 920
           },
           "id": 117,
           "options": {
@@ -7868,7 +7880,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 80
+            "y": 928
           },
           "id": 125,
           "links": [
@@ -7994,7 +8006,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 80
+            "y": 928
           },
           "id": 123,
           "options": {
@@ -8128,7 +8140,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 88
+            "y": 936
           },
           "id": 121,
           "options": {
@@ -8255,7 +8267,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 88
+            "y": 936
           },
           "id": 161,
           "links": [
@@ -8377,9 +8389,9 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 461
+            "y": 1309
           },
-          "id": 154,
+          "id": 162,
           "options": {
             "legend": {
               "calcs": [
@@ -8536,4 +8548,4 @@
   "title": "VictoriaMetrics - vmagent",
   "uid": "G7Z9GzMGz",
   "version": 1
-}
+}

dashboards/vmauth.json

@@ -1611,7 +1611,7 @@
                 "type": "prometheus",
                 "uid": "$ds"
               },
-              "expr": "sum(go_memstats_sys_bytes{job=~\"$job\", instance=~\"$instance\"}) + sum(vm_cache_size_bytes{job=~\"$job\", instance=~\"$instance\"})",
+              "expr": "sum(go_memstats_sys_bytes{job=~\"$job\", instance=~\"$instance\"})",
               "format": "time_series",
               "hide": false,
               "intervalFactor": 1,
@@ -1623,7 +1623,7 @@
                 "type": "prometheus",
                 "uid": "$ds"
               },
-              "expr": "sum(go_memstats_heap_inuse_bytes{job=~\"$job\", instance=~\"$instance\"}) + sum(vm_cache_size_bytes{job=~\"$job\", instance=~\"$instance\"})",
+              "expr": "sum(go_memstats_heap_inuse_bytes{job=~\"$job\", instance=~\"$instance\"})",
               "format": "time_series",
               "hide": false,
               "intervalFactor": 1,

deployment/docker/Makefile

@@ -7,7 +7,7 @@ ROOT_IMAGE ?= alpine:3.23.3
 ROOT_IMAGE_SCRATCH ?= scratch
 CERTS_IMAGE := alpine:3.23.3
 
-GO_BUILDER_IMAGE := golang:1.26.0
+GO_BUILDER_IMAGE := golang:1.26.1
 
 BUILDER_IMAGE := local/builder:2.0.0-$(shell echo $(GO_BUILDER_IMAGE) | tr :/ __)-1
 BASE_IMAGE := local/base:1.1.4-$(shell echo $(ROOT_IMAGE) | tr :/ __)-$(shell echo $(CERTS_IMAGE) | tr :/ __)

deployment/docker/compose-vm-cluster.yml

@@ -3,7 +3,7 @@ services:
   #  It scrapes targets defined in --promscrape.config
   #  And forward them to --remoteWrite.url
   vmagent:
-    image: victoriametrics/vmagent:v1.136.0
+    image: victoriametrics/vmagent:v1.138.0
     depends_on:
       - "vmauth"
     ports:
@@ -25,27 +25,31 @@ services:
     ports:
       - 3000:3000
     restart: always
+    environment:
+      - GF_PLUGINS_PREINSTALL=yesoreyeram-infinity-datasource
     volumes:
       - grafanadata:/var/lib/grafana
-      - ./provisioning/datasources/prometheus-datasource/cluster.yml:/etc/grafana/provisioning/datasources/cluster.yml
+      - ./provisioning/datasources/prometheus/cluster.yml:/etc/grafana/provisioning/datasources/cluster.yml
+      - ./provisioning/datasources/infinity/cluster.yml:/etc/grafana/provisioning/datasources/infinity-cluster.yml
       - ./provisioning/dashboards:/etc/grafana/provisioning/dashboards
       - ./../../dashboards/victoriametrics-cluster.json:/var/lib/grafana/dashboards/vm.json
       - ./../../dashboards/vmagent.json:/var/lib/grafana/dashboards/vmagent.json
       - ./../../dashboards/vmalert.json:/var/lib/grafana/dashboards/vmalert.json
       - ./../../dashboards/vmauth.json:/var/lib/grafana/dashboards/vmauth.json
       - ./../../dashboards/alert-statistics.json:/var/lib/grafana/dashboards/alert-statistics.json
+      - ./../../dashboards/metrics-explorer.json:/var/lib/grafana/dashboards/metrics-explorer.json
 
   # vmstorage shards. Each shard receives 1/N of all metrics sent to vminserts,
   # where N is number of vmstorages (2 in this case).
   vmstorage-1:
-    image: victoriametrics/vmstorage:v1.136.0-cluster
+    image: victoriametrics/vmstorage:v1.138.0-cluster
     volumes:
       - strgdata-1:/storage
     command:
       - "--storageDataPath=/storage"
     restart: always
   vmstorage-2:
-    image: victoriametrics/vmstorage:v1.136.0-cluster
+    image: victoriametrics/vmstorage:v1.138.0-cluster
     volumes:
       - strgdata-2:/storage
     command:
@@ -55,7 +59,7 @@ services:
   # vminsert is ingestion frontend. It receives metrics pushed by vmagent,
   # pre-process them and distributes across configured vmstorage shards.
   vminsert-1:
-    image: victoriametrics/vminsert:v1.136.0-cluster
+    image: victoriametrics/vminsert:v1.138.0-cluster
     depends_on:
       - "vmstorage-1"
       - "vmstorage-2"
@@ -64,7 +68,7 @@ services:
       - "--storageNode=vmstorage-2:8400"
     restart: always
   vminsert-2:
-    image: victoriametrics/vminsert:v1.136.0-cluster
+    image: victoriametrics/vminsert:v1.138.0-cluster
     depends_on:
       - "vmstorage-1"
       - "vmstorage-2"
@@ -76,7 +80,7 @@ services:
   # vmselect is a query fronted. It serves read queries in MetricsQL or PromQL.
   # vmselect collects results from configured `--storageNode` shards.
   vmselect-1:
-    image: victoriametrics/vmselect:v1.136.0-cluster
+    image: victoriametrics/vmselect:v1.138.0-cluster
     depends_on:
       - "vmstorage-1"
       - "vmstorage-2"
@@ -86,7 +90,7 @@ services:
       - "--vmalert.proxyURL=http://vmalert:8880"
     restart: always
   vmselect-2:
-    image: victoriametrics/vmselect:v1.136.0-cluster
+    image: victoriametrics/vmselect:v1.138.0-cluster
     depends_on:
       - "vmstorage-1"
       - "vmstorage-2"
@@ -101,7 +105,7 @@ services:
   # read requests from Grafana, vmui, vmalert among vmselects.
   # It can be used as an authentication proxy.
   vmauth:
-    image: victoriametrics/vmauth:v1.136.0
+    image: victoriametrics/vmauth:v1.138.0
     depends_on:
       - "vmselect-1"
       - "vmselect-2"
@@ -115,7 +119,7 @@ services:
 
   # vmalert executes alerting and recording rules
   vmalert:
-    image: victoriametrics/vmalert:v1.136.0
+    image: victoriametrics/vmalert:v1.138.0
     depends_on:
       - "vmauth"
     ports:
@@ -127,8 +131,17 @@ services:
       - ./rules/alerts-vmalert.yml:/etc/alerts/alerts-vmalert.yml
     command:
       - "--datasource.url=http://vmauth:8427/select/0/prometheus"
+      - "--datasource.basicAuth.username=foo"
+      - "--datasource.basicAuth.password=bar"
+
       - "--remoteRead.url=http://vmauth:8427/select/0/prometheus"
+      - "--remoteRead.basicAuth.username=foo"
+      - "--remoteRead.basicAuth.password=bar"
+
       - "--remoteWrite.url=http://vmauth:8427/insert/0/prometheus"
+      - "--remoteWrite.basicAuth.username=foo"
+      - "--remoteWrite.basicAuth.password=bar"
+
       - "--notifier.url=http://alertmanager:9093/"
       - "--rule=/etc/alerts/*.yml"
       # display source of alerts in grafana

deployment/docker/compose-vm-single.yml

@@ -3,7 +3,7 @@ services:
   #  It scrapes targets defined in --promscrape.config
   #  And forward them to --remoteWrite.url
   vmagent:
-    image: victoriametrics/vmagent:v1.136.0
+    image: victoriametrics/vmagent:v1.138.0
     depends_on:
       - "victoriametrics"
     ports:
@@ -18,7 +18,7 @@ services:
   # VictoriaMetrics instance, a single process responsible for
   # storing metrics and serve read requests.
   victoriametrics:
-    image: victoriametrics/victoria-metrics:v1.136.0
+    image: victoriametrics/victoria-metrics:v1.138.0
     ports:
       - 8428:8428
       - 8089:8089
@@ -43,19 +43,23 @@ services:
       - "victoriametrics"
     ports:
       - 3000:3000
+    restart: always
+    environment:
+      - GF_PLUGINS_PREINSTALL=yesoreyeram-infinity-datasource
     volumes:
       - grafanadata:/var/lib/grafana
-      - ./provisioning/datasources/prometheus-datasource/single.yml:/etc/grafana/provisioning/datasources/single.yml
+      - ./provisioning/datasources/prometheus/single.yml:/etc/grafana/provisioning/datasources/single.yml
+      - ./provisioning/datasources/infinity/single.yml:/etc/grafana/provisioning/datasources/infinity-single.yml
       - ./provisioning/dashboards:/etc/grafana/provisioning/dashboards
       - ./../../dashboards/victoriametrics.json:/var/lib/grafana/dashboards/vm.json
       - ./../../dashboards/vmagent.json:/var/lib/grafana/dashboards/vmagent.json
       - ./../../dashboards/vmalert.json:/var/lib/grafana/dashboards/vmalert.json
       - ./../../dashboards/alert-statistics.json:/var/lib/grafana/dashboards/alert-statistics.json
-    restart: always
+      - ./../../dashboards/metrics-explorer.json:/var/lib/grafana/dashboards/metrics-explorer.json
 
   # vmalert executes alerting and recording rules
   vmalert:
-    image: victoriametrics/vmalert:v1.136.0
+    image: victoriametrics/vmalert:v1.138.0
     depends_on:
       - "victoriametrics"
       - "alertmanager"

deployment/docker/provisioning/datasources/infinity/cluster.yml

@@ -0,0 +1,10 @@
+apiVersion: 1
+
+datasources:
+  - name: VictoriaMetrics-Infinity
+    type: yesoreyeram-infinity-datasource
+    url: "http://vmauth:8427/select/0/prometheus"
+    basicAuth: true
+    basicAuthUser: foo
+    secureJsonData:
+      basicAuthPassword: bar

deployment/docker/provisioning/datasources/infinity/single.yml

@@ -0,0 +1,6 @@
+apiVersion: 1
+
+datasources:
+  - name: VictoriaMetrics-Infinity
+    type: yesoreyeram-infinity-datasource
+    url: "http://victoriametrics:8428"

deployment/docker/vmanomaly/vmanomaly-integration/compose.yml

@@ -1,6 +1,6 @@
 services:
   vmagent:
-    image: victoriametrics/vmagent:v1.136.0
+    image: victoriametrics/vmagent:v1.138.0
     depends_on:
       - "victoriametrics"
     ports:
@@ -14,7 +14,7 @@ services:
     restart: always
 
   victoriametrics:
-    image: victoriametrics/victoria-metrics:v1.136.0
+    image: victoriametrics/victoria-metrics:v1.138.0
     ports:
       - 8428:8428
     volumes:
@@ -40,7 +40,7 @@ services:
     restart: always
 
   vmalert:
-    image: victoriametrics/vmalert:v1.136.0
+    image: victoriametrics/vmalert:v1.138.0
     depends_on:
       - "victoriametrics"
     ports:
@@ -59,7 +59,7 @@ services:
       - '--external.alert.source=explore?orgId=1&left=["now-1h","now","VictoriaMetrics",{"expr": },{"mode":"Metrics"},{"ui":[true,true,true,"none"]}]'
     restart: always
   vmanomaly:
-    image: victoriametrics/vmanomaly:v1.28.7
+    image: victoriametrics/vmanomaly:v1.29.1
     depends_on:
       - "victoriametrics"
     ports:

docs/ai-tools/README.md

@@ -0,0 +1,93 @@
+VictoriaMetrics Observability Stack integrates with AI assistants through MCP servers and agent skills.
+These integrations allow AI agents and automation tools to query metrics, logs, and traces, analyze telemetry data, 
+and assist engineers with debugging and observability tasks.
+
+# MCP Servers
+
+MCP (Model Context Protocol) servers expose observability data and operational capabilities to AI assistants in a structured way.
+This allows AI agents to query telemetry data, analyze system behavior, and assist engineers in troubleshooting and investigation workflows.
+
+## VictoriaMetrics MCP Server
+
+[VictoriaMetrics MCP Server](https://github.com/VictoriaMetrics/mcp-victoriametrics) provides access to VictoriaMetrics
+instances, seamless integration with [VictoriaMetrics APIs](https://docs.victoriametrics.com/victoriametrics/url-examples/) 
+and [documentation](https://docs.victoriametrics.com/). 
+
+It offers a comprehensive interface for monitoring, observability, and debugging tasks related to VictoriaMetrics, 
+enabling advanced automation and interaction capabilities for engineers and tools.
+
+Capabilities include:
+- Query metrics and exploring data (even drawing graphs if your client supports it)
+- List and exporting available metrics, labels, labels values and entire time series
+- Analyze and testing your alerting and recording rules and alerts
+- Show parameters of your VictoriaMetrics instances
+- Explore cardinality of your data and metrics usage statistics
+- Analyze, trace, prettify and explain your queries
+- Debug your relabeling rules, downsampling and retention policy configurations
+- Integrate with [VictoriaMetrics Cloud](https://docs.victoriametrics.com/victoriametrics-cloud/)
+ 
+> On YouTube: [How to Use an AI Assistant with Your Monitoring System – VictoriaMetrics MCP Server](https://www.youtube.com/watch?v=1k7xgbRi1k0).
+
+See more details at [VictoriaMetrics/mcp-victoriametrics](https://github.com/VictoriaMetrics/mcp-victoriametrics).
+
+## VictoriaLogs MCP Server
+
+[VictoriaLogs MCP Server](https://github.com/VictoriaMetrics/mcp-victorialogs) provides access to VictoriaLogs instances,
+integration with [VictoriaLogs APIs](https://docs.victoriametrics.com/victorialogs/querying/#http-api) and [documentation](https://docs.victoriametrics.com/victorialogs/).
+
+It provides a comprehensive interface for working with logs and performing observability and debugging tasks related to VictoriaLogs.
+
+Capabilities include:
+- Querying logs and exploring logs data
+- Showing parameters of your VictoriaLogs instances
+- Listing available streams, fields, field values
+- Query statistics for the logs as metrics
+
+See more details at [VictoriaMetrics/mcp-victorialogs](https://github.com/VictoriaMetrics/mcp-victorialogs).
+
+## VictoriaTraces MCP Server
+
+[VictoriaTraces MCP Server](https://github.com/VictoriaMetrics/mcp-victoriatraces) provides access to VictoriaTraces instances,
+integration with [VictoriaTraces APIs](https://docs.victoriametrics.com/victoriatraces/querying/#http-api) and [documentation](https://docs.victoriametrics.com/victoriatraces/).
+
+It enables AI assistants and tools to interact with distributed tracing data for observability and debugging tasks.
+
+Capabilities include:
+- Get services and operations (span names)
+- Query traces, explore and analyze traces data
+
+See more details at [VictoriaMetrics/mcp-victoriatraces](https://github.com/VictoriaMetrics/mcp-victoriatraces).
+
+## vmanomaly MCP Server
+
+[vmanomaly MCP Server](https://github.com/VictoriaMetrics/mcp-vmanomaly) provides seamless integration with vmanomaly
+REST API and documentation for AI-assisted anomaly detection, model management, and observability insights.
+
+Capabilities include:
+- Health Monitoring: Check `vmanomaly` server health and build information
+- Model Management: List, validate, and configure anomaly detection models (like `zscore_online`, `prophet`, and more)
+- Configuration Generation: Generate complete `vmanomaly` YAML configurations
+- Alert Rule Generation: Generate [`vmalert`](https://docs.victoriametrics.com/victoriametrics/vmalert/) [alerting rules](https://docs.victoriametrics.com/victoriametrics/vmalert/#alerting-rules) based on [anomaly score metrics](https://docs.victoriametrics.com/anomaly-detection/faq/#what-is-anomaly-score) to simplify alerting setup
+- Documentation Search: Full-text search across embedded `vmanomaly` documentation with fuzzy matching
+
+See more details at [VictoriaMetrics/mcp-vmanomaly](https://github.com/VictoriaMetrics/mcp-vmanomaly).
+
+
+# Agent Skills
+
+[Agent skills](https://github.com/VictoriaMetrics/skills) help AI agents and automation tools understand, operate, 
+and troubleshoot VictoriaMetrics observability components, including metrics, logs, and traces.
+
+These skills provide predefined workflows and capabilities such as:
+* Query metrics, logs, traces and alerts
+* Query trace analysis
+* Multi-signal investigations 
+* Cardinality optimization 
+* Unused metric detection
+
+To install the available skills for AI agents, run:
+```sh
+npx skills add VictoriaMetrics/skills
+```
+
+See more details at [VictoriaMetrics/skills](https://github.com/VictoriaMetrics/skills).

docs/ai-tools/_index.md

@@ -0,0 +1,19 @@
+---
+title: AI tools
+weight: 61
+menu:
+  docs:
+    weight: 61
+    identifier: ai-tools
+tags:
+  - metrics
+  - logs
+  - traces
+  - AI
+  - AI integration
+  - agent
+  - assistant
+  - MCP server
+
+---
+{{% content "README.md" %}}

docs/anomaly-detection/CHANGELOG.md

@@ -14,6 +14,34 @@ aliases:
 ---
 Please find the changelog for VictoriaMetrics Anomaly Detection below.
 
+## v1.29.1
+Released: 2026-03-25
+
+- FEATURE: Added `min_rel_dev_from_expected` [common model argument](https://docs.victoriametrics.com/anomaly-detection/components/models/#minimal-relative-deviation-from-expected) to support relative business gating based on percentage deviation from expected values. This allows users to specify a relative threshold for ignoring small deviations that are not significant in the context of the expected value, particularly useful for heterogeneous series with varying unknown magnitudes, returned from the same query.
+
+- UI: Updated [vmanomaly UI](https://docs.victoriametrics.com/anomaly-detection/ui/) from [v1.5.0](https://docs.victoriametrics.com/anomaly-detection/ui/#v150) to [v1.5.1](https://docs.victoriametrics.com/anomaly-detection/ui/#v151), see respective [release notes](https://docs.victoriametrics.com/anomaly-detection/ui/#v151) for details.
+
+- IMPROVEMENT: Optimized [`VmWriter`](https://docs.victoriametrics.com/anomaly-detection/components/writer/#vm-writer) hot path by 2-3x in terms of infer-write latency
+
+- IMPROVEMENT: Optimized backbone of [t-digest](https://www.sciencedirect.com/science/article/pii/S2665963820300403) data structures to reduce the memory usage and speed up the fit/infer calls for underlying models that use it (e.g. [OnlineQuantileModel](https://docs.victoriametrics.com/anomaly-detection/components/models/#online-seasonal-quantile) or [MAD](https://docs.victoriametrics.com/anomaly-detection/components/models/#online-mad)).
+
+- BUGFIX: Fixed forward compatibility issues with the persisted state from [v1.28.5](#v1285) for [ProphetModel](https://docs.victoriametrics.com/anomaly-detection/components/models/#prophet) and [QuantileModel](https://docs.victoriametrics.com/anomaly-detection/components/models/#online-seasonal-quantile) models, which could lead to "util.files - ERROR - Unexpected error while loading model from ..." in [stateful mode](https://docs.victoriametrics.com/anomaly-detection/components/settings/#state-restoration) after the upgrade to [v1.29.0](#v1290). Now the service can properly load the persisted state from [v1.28.5](#v1285) and continue functioning without requiring retraining of the affected models.
+
+## v1.29.0
+Released: 2026-03-05
+
+- UI: Updated [vmanomaly UI](https://docs.victoriametrics.com/anomaly-detection/ui/) from [v1.4.3](https://docs.victoriametrics.com/anomaly-detection/ui/#v143) to [v1.5.0](https://docs.victoriametrics.com/anomaly-detection/ui/#v150), see respective [release notes](https://docs.victoriametrics.com/anomaly-detection/ui/#v150) for details. Notable changes include [AI assistance](https://docs.victoriametrics.com/anomaly-detection/ui/#ai-assistance) support capable of applying model configuration changes, generating VMAlert rules, and providing general guidance on using the product.
+
+- IMPROVEMENT: Optimized internal data structures for readers when `query_from_last_seen_timestamp` [parameter](https://docs.victoriametrics.com/anomaly-detection/components/reader/#config-parameters) is enabled, resulting in reduced memory usage and improved performance for large datasets.
+
+- IMPROVEMENT: Hardened [hot reload](https://docs.victoriametrics.com/anomaly-detection/components/#hot-reload) with staged snapshot apply and automatic rollback. Reload now validates once and applies the same snapshot, preventing re-read race conditions and avoiding same-port conflicts during restart; failures keep previous runtime and are reflected in [startup metrics](https://docs.victoriametrics.com/anomaly-detection/components/monitoring/#startup-metrics).
+
+- BUGFIX: Config file read/parse failures are now non-fatal in [hot reload](https://docs.victoriametrics.com/anomaly-detection/components/#hot-reload) mode (service keeps running), while initial startup remains fatal for invalid/broken config files.
+
+- BUGFIX: Fixed missing datapoints in [BacktestingScheduler](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/#backtesting-scheduler) windows used in [exact mode](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/#defining-inference-timeframe-1), leading to "gaps" in plotted predictions and scores.
+
+- BUGFIX: Fixed a model state update issue in [BacktestingScheduler exact mode](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/#defining-inference-timeframe-1) when parallelization (`settings.n_workers > 1`) was enabled, causing [online models](https://docs.victoriametrics.com/anomaly-detection/components/models/#online-models) to produce stale/flat `yhat`, `yhat_lower`, and `yhat_upper` lines.
+
 ## v1.28.7
 Released: 2026-02-09
 
@@ -43,7 +71,7 @@ Released: 2026-01-12
 ## v1.28.3
 Released: 2025-12-17
 
-- IMPROVEMENT: Aligned service endpoints for `vmanomaly` [MCP Server](https://github.com/VictoriaMetrics-Community/mcp-vmanomaly) integration.
+- IMPROVEMENT: Aligned service endpoints for `vmanomaly` [MCP Server](https://github.com/VictoriaMetrics/mcp-vmanomaly) integration.
 
 ## v1.28.2
 Released: 2025-12-11
@@ -103,7 +131,7 @@ Released: 2025-10-09
   ```
   This happened in scenarios with a large number of queries (e.g., in non-sharded deployments). Now the pool size is set dynamically to prevent such warnings and retain efficient connection reuse.
 
-## v1.26.1  
+## v1.26.1
 Released: 2025-10-08
 
 - IMPROVEMENT: Enriched lifecycle logs with the deterministic labelset hash for each query result (metric). This allows correlating model training, inference runs/skips, and on-disk artifacts presence or cleanup during incident triage.
@@ -113,7 +141,7 @@ Released: 2025-10-02
 
 - FEATURE: Introduced vmui-like [UI](https://docs.victoriametrics.com/anomaly-detection/ui/) for `vmanomaly` service to simplify the configuration and backtesting of anomaly detection models before it goes to production. It provides an intuitive interface to finetune model configurations, visualize its predictions and anomaly scores, and perform backtesting on historical data. The UI is accessible via a web browser and can be run as a [standalone service](https://docs.victoriametrics.com/anomaly-detection/ui/#preset-usage) or [integrated with productionalized deployments](https://docs.victoriametrics.com/anomaly-detection/ui/#mixed-usage). For more details, refer to the [documentation](https://docs.victoriametrics.com/anomaly-detection/ui/).
 
-- FEATURE: Added support for reading data from [VictoriaLogs stats queries](https://docs.victoriametrics.com/victorialogs/querying/#querying-log-range-stats) with `VLogsReader`. This reader allows quering and analyzing log data stored in VictoriaLogs, enabling anomaly detection on metrics generated from logs. It supports similar configuration options as `VmReader`, including `datasource_url`, `tenant_id`, `queries`, etc. For more details, refer to the [documentation](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vlogs-reader). It can be also used in [UI mode](https://docs.victoriametrics.com/anomaly-detection/ui/) for backtesting log-based anomaly detection configurations.
+- FEATURE: Added support for reading data from [VictoriaLogs stats queries](https://docs.victoriametrics.com/victorialogs/querying/#querying-log-range-stats) with `VLogsReader`. This reader allows querying and analyzing log data stored in VictoriaLogs, enabling anomaly detection on metrics generated from logs. It supports similar configuration options as `VmReader`, including `datasource_url`, `tenant_id`, `queries`, etc. For more details, refer to the [documentation](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vlogs-reader). It can be also used in [UI mode](https://docs.victoriametrics.com/anomaly-detection/ui/) for backtesting log-based anomaly detection configurations.
 
 - IMPROVEMENT: Resolved the case in the [`IsolationForestModel`](https://docs.victoriametrics.com/anomaly-detection/components/models/#isolation-forest-multivariate) with `provide_series` common model [argument](https://docs.victoriametrics.com/anomaly-detection/components/models/#provide-series) including `yhat.*` series (prediction and confidence boundaries), which are not produced by this model. Now config validation will fail with a clear error message if such series names are requested.
 
@@ -168,7 +196,7 @@ Released: 2025-07-17
 
 - FEATURE: Added an option to reference environment variables in [configuration files](https://docs.victoriametrics.com/anomaly-detection/components/) using scalar string placeholders `%{ENV_NAME}`. See the [environment variables](https://docs.victoriametrics.com/anomaly-detection/components/#environment-variables) section for more details and examples. This feature is particularly useful for managing sensitive information like API keys or database credentials while still making it accessible to the service.
 
-- IMPROVEMENT: Added `iqr_threshold` to [OnlineQuantileModel](https://docs.victoriametrics.com/anomaly-detection/components/models/#online-seasonal-quantile) to refine the prediction boundaries without the need to manually adjusting `scale` [argument](https://docs.victoriametrics.com/anomaly-detection/components/models/#scale). Best set as >= 2 and used with smaller, robust quantiles (e.g. `(0.25, 0.5, 0.75)`) to both reduce the impact of outliers on the prediction boundaries and increase the likelyhood of having "non-anomalous" data within updated boundaries.
+- IMPROVEMENT: Added `iqr_threshold` to [OnlineQuantileModel](https://docs.victoriametrics.com/anomaly-detection/components/models/#online-seasonal-quantile) to refine the prediction boundaries without the need to manually adjusting `scale` [argument](https://docs.victoriametrics.com/anomaly-detection/components/models/#scale). Best set as >= 2 and used with smaller, robust quantiles (e.g. `(0.25, 0.5, 0.75)`) to both reduce the impact of outliers on the prediction boundaries and increase the likelihood of having "non-anomalous" data within updated boundaries.
 
 - IMPROVEMENT: Fixed duplicated calls to VictoriaMetrics' in [reader](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader) for queries in `reader.queries` that are attached to multiple models in `models` [section](https://docs.victoriametrics.com/anomaly-detection/components/models/#queries) where previously, each model would independently fetch for the same query, leading to unnecessary load on the reader and VictoriaMetrics TSDB. Now, the reader will only be called once per unique (scheduler_alias, query_key) pair, and the results will be shared across all models that use the same query in the same scheduler.
 
@@ -270,7 +298,7 @@ Released: 2025-03-03
 > This release contains a bug introduced in [v1.18.7](#v1187) - [`PeriodicScheduler`](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/#periodic-scheduler) where configurations with `fit_every` > `fit_window` could cause inference to be skipped for |fit_every - fit_window| time, until the next `fit_every` call happens. For `fit_every` > `fit_window` configurations we recommend upgrading to [v1.20.1](#v1201), which resolves this issue.
 
 - FEATURE: The `scale` argument is now a [common argument](https://docs.victoriametrics.com/anomaly-detection/components/models/#scale), previously supported only by [`ProphetModel`](https://docs.victoriametrics.com/anomaly-detection/components/models/#prophet) and [`OnlineQuantileModel`](https://docs.victoriametrics.com/anomaly-detection/components/models/#online-seasonal-quantile). Additionally, `scale` is now **two-sided**, represented as `[scale_lb, scale_ub]`. The previous format (`scale: x`) remains supported and will be automatically converted to `scale: [x, x]`.
-  
+
 - FEATURE: Introduced a post-processing step to clip `yhat`, `yhat_lower`, and `yhat_upper` to the configured `data_range` [values](https://docs.victoriametrics.com/anomaly-detection/components/reader/) in `VmReader`, if defined. This feature is disabled by default for backward compatibility. It can be enabled for models that generate predictions and estimates, such as [`ProphetModel`](https://docs.victoriametrics.com/anomaly-detection/components/models/#prophet), by setting the [common argument](https://docs.victoriametrics.com/anomaly-detection/components/models/#clip-predictions) `clip_predictions` to `True`.
 
 - IMPROVEMENT: Introduced the `anomaly_score_outside_data_range` [parameter](https://docs.victoriametrics.com/anomaly-detection/components/models/#score-outside-data-range) to allow overriding the default anomaly score (`1.01`) assigned when input values (`y`) fall outside the defined `data_range` (data domain violation). It improves flexibility for alerting rules and enables clearer visual distinction between different anomaly scenarios. Override can be configured at the **service level** (`settings`) or per **model instance** (`models.model_xxx`), with model-level values taking priority. If not explicitly set, the default anomaly score remains `1.01` for backward compatibility.
@@ -305,8 +333,8 @@ Released: 2025-01-20
 > This release contains a bug introduced in [v1.18.7](#v1187) - [`PeriodicScheduler`](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/#periodic-scheduler) where configurations with `fit_every` > `fit_window` could cause inference to be skipped for |fit_every - fit_window| time, until the next `fit_every` call happens. For `fit_every` > `fit_window` configurations we recommend upgrading to [v1.20.1](#v1201), which resolves this issue.
 
 - FEATURE: Added support for per-query `tenant_id` in the [`VmReader`](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader). This allows overriding the reader-level `tenant_id` within a single global `vmanomaly` configuration on a *per-query* basis, enabling isolation of data for different tenants in separate queries when querying the [VictoriaMetrics cluster version](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/). For details, see the [documentation](https://docs.victoriametrics.com/anomaly-detection/components/reader/#per-query-parameters).
-- IMPROVEMEMT: Speedup the model infer stage on multicore systems.
-- IMPROVEMEMT: Speedup the model fitting stage by 1.25-3x, depending on configuration complexity.
+- IMPROVEMENT: Speedup the model infer stage on multicore systems.
+- IMPROVEMENT: Speedup the model fitting stage by 1.25-3x, depending on configuration complexity.
 - IMPROVEMENT: Reduced service RAM usage by 5-10%, depending on configuration complexity.
 - BUGFIX: Now [`VmReader`](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader) properly handles the cases where the number of queries processed in parallel (up to `reader.queries` cardinality) exceeds the default limit of 10 HTTP(S) connections, preventing potential data loss from discarded queries. The pool limit will automatically adjust to match `reader.queries` cardinality.
 - BUGFIX: Corrected the construction of write endpoints for cluster VictoriaMetrics `url`s (`tenant_id` arg is set) in `monitoring.push` [section configurations](https://docs.victoriametrics.com/anomaly-detection/components/monitoring/#push-config-parameters).
@@ -484,7 +512,7 @@ Released: 2024-08-26
 
 ## v1.15.5
 Released: 2024-08-19
-- BUGFIX: following [v1.15.2](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1152) online model enhancement, now `data_range` parameter is correctly initialized for online models, created (for new time series returned by particular query) during `infer` calls. 
+- BUGFIX: following [v1.15.2](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1152) online model enhancement, now `data_range` parameter is correctly initialized for online models, created (for new time series returned by particular query) during `infer` calls.
 
 ## v1.15.4
 Released: 2024-08-15
@@ -524,7 +552,7 @@ Released: 2024-08-06
 
 - FEATURE: Introduced the `optimized_business_params` key (list of strings) to the [`AutoTuned`](https://docs.victoriametrics.com/anomaly-detection/components/models/#autotuned) `optimization_params`. This allows particular business-specific parameters such as [`detection_direction`](https://docs.victoriametrics.com/anomaly-detection/components/models/#detection-direction) and [`min_dev_from_expected`](https://docs.victoriametrics.com/anomaly-detection/components/models/#minimal-deviation-from-expected) to remain **unchanged during optimizations, retaining their default values**.
 - IMPROVEMENT: Optimized the [`AutoTuned`](https://docs.victoriametrics.com/anomaly-detection/components/models/#autotuned) model logic to minimize deviations from the expected `anomaly_percentage` specified in the configuration and the detected percentage in the data, while also reducing discrepancies between the actual values (`y`) and the predictions (`yhat`).
-- IMPROVEMENT: Allow [`ProphetModel`](https://docs.victoriametrics.com/anomaly-detection/components/models/#prophet) to fit with multiple seasonalities when used in [`AutoTuned`](https://docs.victoriametrics.com/anomaly-detection/components/models/#autotuned) mode. 
+- IMPROVEMENT: Allow [`ProphetModel`](https://docs.victoriametrics.com/anomaly-detection/components/models/#prophet) to fit with multiple seasonalities when used in [`AutoTuned`](https://docs.victoriametrics.com/anomaly-detection/components/models/#autotuned) mode.
 
 ## v1.14.2
 Released: 2024-07-26
@@ -575,10 +603,10 @@ Released: 2024-03-31
 Released: 2024-02-22
 - FEATURE: Multi-scheduler support. Now users can use multiple [model specs](https://docs.victoriametrics.com/anomaly-detection/components/models/) in a single config (via aliasing), each spec can be run with its own (even multiple) [schedulers](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/).
   - Introduction of `schedulers` arg in model spec:
-    - It allows each model to be managed by 1 (or more) schedulers, so overall resource usage is optimized and flexibility is preserved. 
+    - It allows each model to be managed by 1 (or more) schedulers, so overall resource usage is optimized and flexibility is preserved.
     - Passing an empty list or not specifying this param implies that each model is run in **all** the schedulers, which is a backward-compatible behavior.
     - Please find more details in docs on [Model section](https://docs.victoriametrics.com/anomaly-detection/components/models/#schedulers)
-- DEPRECATION: slight refactor of a scheduler config section 
+- DEPRECATION: slight refactor of a scheduler config section
   - Now schedulers are passed as a mapping of `scheduler_alias: scheduler_spec` under [scheduler](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/) sections. Using old format (< [1.11.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1110)) will produce warnings for now and will be removed in future versions.
 - DEPRECATION: The `--watch` CLI option for config file reloads is deprecated and will be ignored in the future.
 
@@ -586,11 +614,11 @@ Released: 2024-02-22
 Released: 2024-02-15
 - FEATURE: Multi-model support. Now users can specify multiple [model specs](https://docs.victoriametrics.com/anomaly-detection/components/models/) in a single config (via aliasing), as well as to reference what [queries from VmReader](https://docs.victoriametrics.com/anomaly-detection/components/reader/#config-parameters) it should be run on.
   - Introduction of `queries` arg in model spec:
-    - It allows the model to be executed only on a particular query subset from `reader` section. 
+    - It allows the model to be executed only on a particular query subset from `reader` section.
     - Passing an empty list or not specifying this param implies that each model is run on results from **all** queries, which is a backward-compatible behavior.
     - Please find more details in docs on [Model section](https://docs.victoriametrics.com/anomaly-detection/components/models/#queries)
 
-- DEPRECATION: slight refactor of a model config section 
+- DEPRECATION: slight refactor of a model config section
   - Now models are passed as a mapping of `model_alias: model_spec` under [model](https://docs.victoriametrics.com/anomaly-detection/components/models/) sections. Using old format (<= [1.9.2](https://docs.victoriametrics.com/anomaly-detection/changelog/#v192)) will produce warnings for now and will be removed in future versions.
   - Please find more details in docs on [Model section](https://docs.victoriametrics.com/anomaly-detection/components/models/)
 - IMPROVEMENT: now logs from [`monitoring.pull`](https://docs.victoriametrics.com/anomaly-detection/components/monitoring/#monitoring-section-config-example) GET requests to `/metrics` endpoint are shown only in DEBUG mode
@@ -641,7 +669,7 @@ Released: 2023-12-21
 
 ## v1.6.0
 Released: 2023-10-30
-- IMPROVEMENT: 
+- IMPROVEMENT:
   - now all the produced healthcheck metrics have `vmanomaly_` prefix for easier accessing.
   - updated docs for monitoring.
   > This is an backward-incompatible change, as metric names will be changed, resulting in new metrics creation, i.e. `model_datapoints_produced` will become `vmanomaly_model_datapoints_produced`
@@ -654,19 +682,19 @@ Released: 2023-10-30
 
 ## v1.5.1
 Released: 2023-09-18
-- IMPROVEMENT: Infer from the latest seen datapoint for each query. Handles the case datapoints arrive late. 
+- IMPROVEMENT: Infer from the latest seen datapoint for each query. Handles the case datapoints arrive late.
 
 
 ## v1.5.0
 Released: 2023-08-11
-- FEATURE: add `--license` and `--license-file` command-line flags for license code verification. 
+- FEATURE: add `--license` and `--license-file` command-line flags for license code verification.
 - IMPROVEMENT: Updated Python to 3.11.4 and updated dependencies.
 - IMPROVEMENT: Guide documentation for Custom Model usage.
 
 
 ## v1.4.2
 Released: 2023-06-09
-- BUGFIX: Fix case with received metric labels overriding generated.  
+- BUGFIX: Fix case with received metric labels overriding generated.
 
 
 ## v1.4.1

docs/anomaly-detection/FAQ.md

@@ -139,7 +139,7 @@ For information on migrating between different versions of `vmanomaly`, please r
 
 ## Choosing the right model for vmanomaly
 
-> {{% available_from "v1.28.3" anomaly %}} Try our [MCP Server](https://github.com/VictoriaMetrics-Community/mcp-vmanomaly) to get AI-assisted recommendations on selecting the best model and its configuration for your use case. See [installation guide](https://github.com/VictoriaMetrics-Community/mcp-vmanomaly#installation) for more details.
+> {{% available_from "v1.28.3" anomaly %}} Try our [MCP Server](https://github.com/VictoriaMetrics/mcp-vmanomaly) to get AI-assisted recommendations on selecting the best model and its configuration for your use case. See [installation guide](https://github.com/VictoriaMetrics/mcp-vmanomaly#installation) for more details.
 
 Selecting the best model for `vmanomaly` depends on the data's nature and the [types of anomalies](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-2/#categories-of-anomalies) to detect. For instance, [Z-score](https://docs.victoriametrics.com/anomaly-detection/components/models/#online-z-score) is suitable for data without trends or seasonality, while more complex patterns might require models like [Prophet](https://docs.victoriametrics.com/anomaly-detection/components/models/#prophet).
 
@@ -151,7 +151,8 @@ Still not 100% sure what to use? We are [here to help](https://docs.victoriametr
 
 ## Incorporating domain knowledge
 
-> {{% available_from "v1.28.3" anomaly %}} Try our [MCP Server](https://github.com/VictoriaMetrics-Community/mcp-vmanomaly) to get AI-assisted recommendations on incorporating domain knowledge into your anomaly detection models. See [installation guide](https://github.com/VictoriaMetrics-Community/mcp-vmanomaly#installation) for more details.
+> [!TIP]
+> {{% available_from "v1.28.3" anomaly %}} Try our [MCP Server](https://github.com/VictoriaMetrics/mcp-vmanomaly) to get AI-assisted recommendations on incorporating domain knowledge into your anomaly detection models. See [installation guide](https://github.com/VictoriaMetrics/mcp-vmanomaly#installation) for more details. {{% available_from "v1.29.0" anomaly %}} Connect MCP server to the [vmanomaly UI](https://docs.victoriametrics.com/anomaly-detection/ui/) to benefit from better response quality and tool access in the UI Copilot, which provides AI-assisted configuration generation and debugging capabilities. See the [UI documentation](https://docs.victoriametrics.com/anomaly-detection/ui/#ai-assistance) for instructions on how to set it up.
 
 Anomaly detection models can significantly improve when incorporating business-specific assumptions about the data and what constitutes an anomaly. `vmanomaly` supports various [business-side configuration parameters](https://docs.victoriametrics.com/anomaly-detection/components/models/#common-args) across all built-in models to **reduce [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-positive)** and **align model behavior with business needs**, for example:
 
@@ -161,7 +162,7 @@ Anomaly detection models can significantly improve when incorporating business-s
 
 - **Defining a `data_range`** - configure [`data_range`](https://docs.victoriametrics.com/anomaly-detection/components/reader/#config-parameters) for the model’s input query to **automatically assign anomaly scores > 1** for values (`y`) that fall outside the defined range.
 
-- **Filtering minor fluctuations with `min_dev_from_expected`** – use [`min_dev_from_expected`](https://docs.victoriametrics.com/anomaly-detection/components/models/#minimal-deviation-from-expected) to **ignore insignificant deviations** and prevent small fluctuations from triggering [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-positive).
+- **Filtering minor fluctuations with absolute (`min_dev_from_expected`)** or **relative (`min_rel_dev_from_expected`)** thresholding – use [`min_dev_from_expected`](https://docs.victoriametrics.com/anomaly-detection/components/models/#minimal-deviation-from-expected) and [`min_rel_dev_from_expected`](https://docs.victoriametrics.com/anomaly-detection/components/models/#minimal-relative-deviation-from-expected) to **ignore insignificant deviations** and prevent alerting rules from triggering [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-positive).
 
 - **Applying `scale` for asymmetric confidence adjustments** - use [`scale`](https://docs.victoriametrics.com/anomaly-detection/components/models/#scale) to adjust confidence intervals **differently for spikes and drops**, ensuring more appropriate anomaly detection.
 
@@ -172,7 +173,7 @@ Consider a metric tracking the percentage of HTTP 4xx status codes for a specifi
 - **Expected data range**: The percentage naturally falls between `0%` and `100%` (`[0, 1]`).
 - **Threshold-based anomaly detection**: If the error rate exceeds `5%`, it should be **automatically flagged as an anomaly** ([anomaly score](#what-is-anomaly-score) > 1), encouraging an incident investigation.
 - **Regime shift detection**: A **continuous increase** in error rates (e.g., from `1.5%` to `3%`) should also be considered **anomalous**, as regime change may indicate underlying system problem, e.g. with a new release.
-- **Avoiding false positives**: **Small, infrequent deviations** (e.g., from `1%` to `1.3%`) should **not** trigger alerts to **prevent unnecessary SRE escalations**. Let it be on the level of 0.5%.
+- **Avoiding false positives**: **Small, infrequent deviations** (e.g., from `1%` to `1.3%` on a scale of 0-100%) should **not** trigger alerts to **prevent unnecessary SRE escalations**. Let it be on the level of 0.5%. Also, relative deviations of less than 10% (e.g., from `1%` to `1.1%`) should be ignored, as they may not represent significant changes in the context of the metric vs its average fluctuation.
 
 Then, the following config may be used to benefit from incorporating domain knowledge into model behavior:
 
@@ -200,7 +201,8 @@ models:
     schedulers: ['periodic_http']
     queries: ['percentage_4xx']
     detection_direction: 'above_expected'  # as interested only in spikes, drops are OK
-    min_dev_from_expected: 0.005  # <0.5% deviations vs expected values should be neglected, generating anomaly score == 0
+    min_dev_from_expected: [0, 0.005]  # <0.5% deviations vs expected values should be neglected, generating anomaly score == 0
+    min_rel_dev_from_expected: [0, 0.1]  # <10% relative deviations vs expected values should be neglected, generating anomaly score == 0
     # to align predictions to be within [0, 5%] interval, defined in reader.queries.percentage_4xx.data_range
     clip_predictions: True
     # specify output series produced by vmanomaly to be written to VictoriaMetrics in `writer`
@@ -236,7 +238,7 @@ groups:
 
 > {{% available_from "v1.27.0" anomaly %}} You can also use the [vmanomaly UI](https://docs.victoriametrics.com/anomaly-detection/ui/) to generate alerting rules automatically based on your model configurations and selected thresholds.
 
-> {{% available_from "v1.28.3" anomaly %}} Check out our [MCP Server](https://github.com/VictoriaMetrics-Community/mcp-vmanomaly) to get AI-assisted recommendations on setting up alerting rules based on produced anomaly scores. See [installation guide](https://github.com/VictoriaMetrics-Community/mcp-vmanomaly#installation) for more details.
+> {{% available_from "v1.28.3" anomaly %}} Check out our [MCP Server](https://github.com/VictoriaMetrics/mcp-vmanomaly) to get AI-assisted recommendations on setting up alerting rules based on produced anomaly scores. See [installation guide](https://github.com/VictoriaMetrics/mcp-vmanomaly#installation) for more details.
 
 ## Preventing alert fatigue
 Produced anomaly scores are designed in such a way that values from 0.0 to 1.0 indicate non-anomalous data, while a value greater than 1.0 is generally classified as an anomaly. However, there are no perfect models for anomaly detection, that's why reasonable defaults expressions like `anomaly_score > 1` may not work 100% of the time. However, anomaly scores, produced by `vmanomaly` are written back as metrics to VictoriaMetrics, where tools like [`vmalert`](https://docs.victoriametrics.com/victoriametrics/vmalert/) can use [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) expressions to fine-tune alerting thresholds and conditions, balancing between avoiding [false negatives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-negative) and reducing [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-positive).
@@ -419,7 +421,7 @@ services:
   # ...
   vmanomaly:
     container_name: vmanomaly
-    image: victoriametrics/vmanomaly:v1.28.7
+    image: victoriametrics/vmanomaly:v1.29.1
     # ...
     restart: always
     volumes:
@@ -637,7 +639,7 @@ options:
 Here’s an example of using the config splitter to divide configurations based on the `extra_filters` argument from the reader section:
 
 ```sh
-docker pull victoriametrics/vmanomaly:v1.28.6 && docker image tag victoriametrics/vmanomaly:v1.28.7 vmanomaly
+docker pull victoriametrics/vmanomaly:v1.29.1 && docker image tag victoriametrics/vmanomaly:v1.29.1 vmanomaly
 ```
 
 ```sh

docs/anomaly-detection/Migration.md

@@ -39,13 +39,14 @@ This section outlines the compatibility of different `vmanomaly` versions with v
 
 > Used if `settings.restore_state` is set to `true`. See argument details in the [configuration documentation](https://docs.victoriametrics.com/anomaly-detection/components/settings/#state-restoration).
 
-There are 2 types of compatibilitity to consider when migrating in stateful mode:
+There are 2 types of compatibility to consider when migrating in stateful mode:
 - **Global (in)compatibility**: The new version can seamlessly read and utilize the existing state without any modifications or data loss. Or, in case of incompatibility, the existing state must be dropped completely to proceed with the migration.
 - **Component (in)compatibility**: The new version may introduce changes that affect specific components (e.g., specific models, data formats) but can still operate with the existing state with some adjustments or drop of incompatible on disk artifacts.
 
 | Group start | Group end | Compatibility | Notes |
 |---------|--------- |------------|-------|
-| [v1.28.7](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1287) | Latest* | Fully Compatible | Just a placeholder for new releases |
+| [v1.29.1](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1291) | Latest* | Fully Compatible | Just a placeholder for new releases |
+| [v1.28.7](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1287) | [v1.29.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1290) | Partially compatible* | Dumped models of class [prophet](https://docs.victoriametrics.com/anomaly-detection/components/models/#prophet) and [seasonal quantile](https://docs.victoriametrics.com/anomaly-detection/components/models/#online-seasonal-quantile) have problems with loading to [v1.29.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1290) due to dropped `pytz` library. **Upgrading directly from v1.28.7 to [v1.29.1](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1291) with a fix is suggested** |
 | [v1.26.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1262) | [v1.28.7](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1287) | Fully Compatible | [v1.28.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1280) introduced [rolling](https://docs.victoriametrics.com/anomaly-detection/components/models/#rolling-models) model class drop in favor of [online](https://docs.victoriametrics.com/anomaly-detection/components/models/#online-models) models (`rolling_quantile` and `std` models), however, it does not impact compatibility, as artifacts were not produced by default for rolling models. Also, offline `mad` and `zscore` models are redirecting to their respective online counterparts since [v1.28.4](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1284). |
 | [v1.25.3](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1253) | [v1.26.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1270) | Partially Compatible* | [v1.25.3](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1253) introduced `forecast_at` argument for base [univariate](https://docs.victoriametrics.com/anomaly-detection/components/models/#univariate-models) and `Prophet` [models](https://docs.victoriametrics.com/anomaly-detection/components/models/#prophet), however, itself remains backward-reversible from newer states like [v1.26.2](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1262), [v1.27.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1270). (All models except `isolation_forest_multivariate` class will be dropped) |
 | [v1.25.1](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1251) | [v1.25.2](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1252) | Fully Compatible | In [v1.25.1](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1251) there was a change to `vmanomaly.db` metadata database format, so migrating from v1.24.0-v1.25.0 requires deletion of a state, see note above the table |