docs/CHANGELOG.md: cut v1.83.0

The issue was in the `labels := dst[offset:]` line in the beginning of appendExtraLabels() function.
The `dst` may be re-allocated when adding extra labels to it. In this case the addition of `exported_`
prefix to labels inside `labels` slice become invisible in the returned `dst` labels.

While at it, properly handle some corner cases:

- Add additional `exported_` prefix to clashing metric labels with already existing `exported_` prefix.
- Store scraped metric names in `exported___name__` label if scrape target contains `__name__` label.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3278

Thanks to @jplanckeel for the initial attempt to fix this issue
at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3281

.github/workflows/check-licenses.yml

@@ -17,7 +17,7 @@ jobs:
       - name: Setup Go
         uses: actions/setup-go@main
         with:
-          go-version: 1.19.1
+          go-version: 1.19.2
         id: go
       - name: Code checkout
         uses: actions/checkout@master

.github/workflows/codeql-analysis.yml

@@ -40,6 +40,12 @@ jobs:
     - name: Checkout repository
       uses: actions/checkout@v3
 
+    - name: Set up Go
+      uses: actions/setup-go@v2
+      with:
+        go-version: 1.19
+      if: ${{ matrix.language == 'go' }}
+
     # Initializes the CodeQL tools for scanning.
     - name: Initialize CodeQL
       uses: github/codeql-action/init@v2

.github/workflows/main.yml

@@ -19,7 +19,7 @@ jobs:
       - name: Setup Go
         uses: actions/setup-go@main
         with:
-          go-version: 1.19.1
+          go-version: 1.19.2
         id: go
       - name: Code checkout
         uses: actions/checkout@master

README.md

@@ -24,7 +24,7 @@ Learn more about [key concepts](https://docs.victoriametrics.com/keyConcepts.htm
 [quick start guide](https://docs.victoriametrics.com/Quick-Start.html) for a better experience.
 
 [Contact us](mailto:info@victoriametrics.com) if you need enterprise support for VictoriaMetrics. 
-See [features available in enterprise package](https://victoriametrics.com/products/enterprise/).
+See [features available in enterprise package](https://docs.victoriametrics.com/enterprise.html).
 Enterprise binaries can be downloaded and evaluated for free 
 from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
 
@@ -65,7 +65,7 @@ VictoriaMetrics has the following prominent features:
   * [DataDog agent or DogStatsD](#how-to-send-data-from-datadog-agent).
 * It supports metrics [relabeling](#relabeling).
 * It can deal with [high cardinality issues](https://docs.victoriametrics.com/FAQ.html#what-is-high-cardinality) and [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate) issues via [series limiter](#cardinality-limiter).
-* It ideally works with big amounts of time series data from APM, Kubernetes, IoT sensors, connected cars, industrial telemetry, financial data and various [Enterprise workloads](https://victoriametrics.com/products/enterprise/).
+* It ideally works with big amounts of time series data from APM, Kubernetes, IoT sensors, connected cars, industrial telemetry, financial data and various [Enterprise workloads](https://docs.victoriametrics.com/enterprise.html).
 * It has open source [cluster version](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/cluster).
 * It can store data on [NFS-based storages](https://en.wikipedia.org/wiki/Network_File_System) such as [Amazon EFS](https://aws.amazon.com/efs/) and [Google Filestore](https://cloud.google.com/filestore).
 
@@ -132,7 +132,15 @@ VictoriaMetrics is developed at a fast pace, so it is recommended periodically c
 
 ### Environment variables
 
-Each flag value can be set via environment variables according to these rules:
+All the VictoriaMetrics components allow referring environment variables in command-line flags via `%{ENV_VAR}` syntax.
+For example, `-metricsAuthKey=%{METRICS_AUTH_KEY}` is automatically expanded to `-metricsAuthKey=top-secret`
+if `METRICS_AUTH_KEY=top-secret` environment variable exists at VictoriaMetrics startup.
+This expansion is performed by VictoriaMetrics itself.
+
+VictoriaMetrics recursively expands `%{ENV_VAR}` references in environment variables on startup.
+For example, `FOO=%{BAR}` environment variable is expanded to `FOO=abc` if `BAR=a%{BAZ}` and `BAZ=bc`.
+
+Additionally, all the VictoriaMetrics components allow setting flag values via environment variables according to these rules:
 
 * The `-envflag.enable` flag must be set.
 * Each `.` char in flag name must be substituted with `_` (for example `-insert.maxQueueDuration <duration>` will translate to `insert_maxQueueDuration=<duration>`).
@@ -277,7 +285,7 @@ Multi-line queries can be entered by pressing `Shift-Enter` in query input field
 
 When querying the [backfilled data](https://docs.victoriametrics.com/#backfilling) or during [query troubleshooting](https://docs.victoriametrics.com/Troubleshooting.html#unexpected-query-results), it may be useful disabling response cache by clicking `Disable cache` checkbox.
 
-VMUI automatically adjusts the interval between datapoints on the graph depending on the horizontal resolution and on the selected time range. The step value can be customized by clickhing `Override step value` checkbox.
+VMUI automatically adjusts the interval between datapoints on the graph depending on the horizontal resolution and on the selected time range. The step value can be customized by changing `Step value` input.
 
 VMUI allows investigating correlations between multiple queries on the same graph. Just click `Add Query` button, enter an additional query in the newly appeared input field and press `Ctrl+Enter`. Results for all the queries should be displayed simultaneously on the same graph.
 
@@ -321,7 +329,7 @@ VictoriaMetrics is configured via command-line flags, so it must be restarted wh
 * Wait until the process stops. This can take a few seconds.
 * Start VictoriaMetrics with the new command-line flags.
 
-Prometheus doesn't drop data during VictoriaMetrics restart. See [this article](https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/) for details. The same applies alos to [vmagent](https://docs.victoriametrics.com/vmagent.html).
+Prometheus doesn't drop data during VictoriaMetrics restart. See [this article](https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/) for details. The same applies also to [vmagent](https://docs.victoriametrics.com/vmagent.html).
 
 ## How to scrape Prometheus exporters such as [node-exporter](https://github.com/prometheus/node_exporter)
 
@@ -731,7 +739,7 @@ VictoriaMetrics supports `__graphite__` pseudo-label for filtering time series w
 
 ### Graphite Render API usage
 
-[VictoriaMetrics Enterprise](https://victoriametrics.com/products/enterprise/) supports [Graphite Render API](https://graphite.readthedocs.io/en/stable/render_api.html) subset
+[VictoriaMetrics Enterprise](https://docs.victoriametrics.com/enterprise.html) supports [Graphite Render API](https://graphite.readthedocs.io/en/stable/render_api.html) subset
 at `/render` endpoint, which is used by [Graphite datasource in Grafana](https://grafana.com/docs/grafana/latest/datasources/graphite/).
 When configuring Graphite datasource in Grafana, the `Storage-Step` http request header must be set to a step between Graphite data points stored in VictoriaMetrics. For example, `Storage-Step: 10s` would mean 10 seconds distance between Graphite datapoints stored in VictoriaMetrics.
 Enterprise binaries can be downloaded and evaluated for free from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
@@ -772,7 +780,7 @@ to your needs or when testing bugfixes.
 
 ### Development build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.2.
 2. Run `make victoria-metrics` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `victoria-metrics` binary and puts it into the `bin` folder.
 
@@ -788,7 +796,7 @@ ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://b
 
 ### Development ARM build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.2.
 2. Run `make victoria-metrics-linux-arm` or `make victoria-metrics-linux-arm64` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `victoria-metrics-linux-arm` or `victoria-metrics-linux-arm64` binary respectively and puts it into the `bin` folder.
 
@@ -802,7 +810,7 @@ ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://b
 
 `Pure Go` mode builds only Go code without [cgo](https://golang.org/cmd/cgo/) dependencies.
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.2.
 2. Run `make victoria-metrics-pure` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `victoria-metrics-pure` binary and puts it into the `bin` folder.
 
@@ -1263,10 +1271,11 @@ See also [resource usage limits docs](#resource-usage-limits).
 
 By default VictoriaMetrics is tuned for an optimal resource usage under typical workloads. Some workloads may need fine-grained resource usage limits. In these cases the following command-line flags may be useful:
 
-- `-memory.allowedPercent` and `-search.allowedBytes` limit the amounts of memory, which may be used for various internal caches at VictoriaMetrics. Note that VictoriaMetrics may use more memory, since these flags don't limit additional memory, which may be needed on a per-query basis.
+- `-memory.allowedPercent` and `-memory.allowedBytes` limit the amounts of memory, which may be used for various internal caches at VictoriaMetrics. Note that VictoriaMetrics may use more memory, since these flags don't limit additional memory, which may be needed on a per-query basis.
+- `-search.maxMemoryPerQuery` limits the amounts of memory, which can be used for processing a single query. Queries, which need more memory, are rejected. Heavy queries, which select big number of time series, may exceed the per-query memory limit by a small percent. The total memory limit for concurrently executed queries can be estimated as `-search.maxMemoryPerQuery` multiplied by `-search.maxConcurrentRequests`.
 - `-search.maxUniqueTimeseries` limits the number of unique time series a single query can find and process. VictoriaMetrics keeps in memory some metainformation about the time series located by each query and spends some CPU time for processing the found time series. This means that the maximum memory usage and CPU usage a single query can use is proportional to `-search.maxUniqueTimeseries`.
 - `-search.maxQueryDuration` limits the duration of a single query. If the query takes longer than the given duration, then it is canceled. This allows saving CPU and RAM when executing unexpected heavy queries.
-- `-search.maxConcurrentRequests` limits the number of concurrent requests VictoriaMetrics can process. Bigger number of concurrent requests usually means bigger memory usage. For example, if a single query needs 100 MiB of additional memory during its execution, then 100 concurrent queries may need `100 * 100 MiB = 10 GiB` of additional memory. So it is better to limit the number of concurrent queries, while suspending additional incoming queries if the concurrency limit is reached. VictoriaMetrics provides `-search.maxQueueDuration` command-line flag for limiting the max wait time for suspended queries.
+- `-search.maxConcurrentRequests` limits the number of concurrent requests VictoriaMetrics can process. Bigger number of concurrent requests usually means bigger memory usage. For example, if a single query needs 100 MiB of additional memory during its execution, then 100 concurrent queries may need `100 * 100 MiB = 10 GiB` of additional memory. So it is better to limit the number of concurrent queries, while suspending additional incoming queries if the concurrency limit is reached. VictoriaMetrics provides `-search.maxQueueDuration` command-line flag for limiting the max wait time for suspended queries. See also `-search.maxMemoryPerQuery` command-line flag.
 - `-search.maxSamplesPerSeries` limits the number of raw samples the query can process per each time series. VictoriaMetrics sequentially processes raw samples per each found time series during the query. It unpacks raw samples on the selected time range per each time series into memory and then applies the given [rollup function](https://docs.victoriametrics.com/MetricsQL.html#rollup-functions). The `-search.maxSamplesPerSeries` command-line flag allows limiting memory usage in the case when the query is executed on a time range, which contains hundreds of millions of raw samples per each located time series.
 - `-search.maxSamplesPerQuery` limits the number of raw samples a single query can process. This allows limiting CPU usage for heavy queries.
 - `-search.maxPointsPerTimeseries` limits the number of calculated points, which can be returned per each matching time series from [range query](https://docs.victoriametrics.com/keyConcepts.html#range-query).
@@ -1400,7 +1409,11 @@ VictoriaMetrics does not support indefinite retention, but you can specify an ar
 
 ## Multiple retentions
 
-A single instance of VictoriaMetrics supports only a single retention, which can be configured via `-retentionPeriod` command-line flag. If you need multiple retentions, then you may start multiple VictoriaMetrics instances with distinct values for the following flags:
+Distinct retentions for distinct time series can be configured via [retention filters](#retention-filters)
+in [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise.html).
+
+Community version of VictoriaMetrics supports only a single retention, which can be configured via [-retentionPeriod](#retention) command-line flag.
+If you need multiple retentions in community version of VictoriaMetrics, then you may start multiple VictoriaMetrics instances with distinct values for the following flags:
 
 * `-retentionPeriod`
 * `-storageDataPath`, so the data for each retention period is saved in a separate directory
@@ -1408,12 +1421,44 @@ A single instance of VictoriaMetrics supports only a single retention, which can
 
 Then set up [vmauth](https://docs.victoriametrics.com/vmauth.html) in front of VictoriaMetrics instances,
 so it could route requests from particular user to VictoriaMetrics with the desired retention.
-The same scheme could be implemented for multiple tenants in [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html).
+
+Similar scheme can be applied for multiple tenants in [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html).
 See [these docs](https://docs.victoriametrics.com/guides/guide-vmcluster-multiple-retention-setup.html) for multi-retention setup details.
 
+## Retention filters
+
+[Enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/enterprise.html) supports e.g. `retention filters`,
+which allow configuring multiple retentions for distinct sets of time series matching the configured [series filters](https://docs.victoriametrics.com/keyConcepts.html#filtering)
+via `-retentionFilter` command-line flag. This flag accepts `filter:duration` options, where `filter` must be
+a valid [series filter](https://docs.victoriametrics.com/keyConcepts.html#filtering), while the `duration`
+must contain valid [retention](#retention) for time series matching the given `filter`. If series doesn't match
+any configured `-retentionFilter`, then the retention configured via [-retentionPeriod](#retention) command-line flag is applied to it.
+If series matches multiple configured retention filters, then the smallest retention is applied.
+
+For example, the following config sets 3 days retention for time series with `team="juniors"` label,
+30 days retention for time series with `env="dev"` or `env="staging"` label and 1 year retention for the remaining time series:
+
+```
+-retentionFilter='{team="juniors"}:3d' -retentionFilter='{env=~"dev|staging"}:30d' -retentionPeriod=1y
+```
+
+Important notes:
+
+- The data outside of the configured retention isn't deleted instantly - it is deleted eventually during [background merges](https://docs.victoriametrics.com/#storage).
+- The `-retentionFilter` doesn't remove old data from `indexdb` (aka inverted index) until the configured [-retentionPeriod](#retention).
+  So the `indexdb` size can grow big under [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate)
+  even for small retentions configured via `-retentionFilter`.
+
+It is safe updating `-retentionFilter` during VictoriaMetrics restarts - the updated retention filters are applied eventually
+to historical data.
+
+See [how to configure multiple retentions in VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#retention-filters).
+
+Retention filters can be evaluated for free by downloading and using enterprise binaries from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
+
 ## Downsampling
 
-[VictoriaMetrics Enterprise](https://victoriametrics.com/products/enterprise/) supports multi-level downsampling with `-downsampling.period` command-line flag. For example:
+[VictoriaMetrics Enterprise](https://docs.victoriametrics.com/enterprise.html) supports multi-level downsampling with `-downsampling.period` command-line flag. For example:
 
 * `-downsampling.period=30d:5m` instructs VictoriaMetrics to [deduplicate](#deduplication) samples older than 30 days with 5 minutes interval.
 
@@ -2205,8 +2250,11 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      Optional path to a file with relabeling rules, which are applied to all the ingested metrics. The path can point either to local file or to http url. See https://docs.victoriametrics.com/#relabeling for details. The config is reloaded on SIGHUP signal
   -relabelDebug
      Whether to log metrics before and after relabeling with -relabelConfig. If the -relabelDebug is enabled, then the metrics aren't sent to storage. This is useful for debugging the relabeling configs
+  -retentionFilter array
+     Retention filter in the format 'filter:retention'. For example, '{env="dev"}:3d' configures the retention for time series with env="dev" label to 3 days. See https://docs.victoriametrics.com/#retention-filters for details. This flag is available only in enterprise version of VictoriaMetrics
+     Supports an array of values separated by comma or specified via multiple flags.
   -retentionPeriod value
-     Data with timestamps outside the retentionPeriod is automatically deleted
+     Data with timestamps outside the retentionPeriod is automatically deleted. See also -retentionFilter
      The following optional suffixes are supported: h (hour), d (day), w (week), y (year). If suffix isn't set, then the duration is counted in months (default 1)
   -retentionTimezoneOffset duration
      The offset for performing indexdb rotation. If set to 0, then the indexdb rotation is performed at 4am UTC time per each -retentionPeriod. If set to 2h, then the indexdb rotation is performed at 4am EET time (the timezone with +2h offset)
@@ -2225,7 +2273,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
   -search.logSlowQueryDuration duration
      Log queries with execution time exceeding this value. Zero disables slow query logging (default 5s)
   -search.maxConcurrentRequests int
-     The maximum number of concurrent search requests. It shouldn't be high, since a single request can saturate all the CPU cores. See also -search.maxQueueDuration (default 8)
+     The maximum number of concurrent search requests. It shouldn't be high, since a single request can saturate all the CPU cores, while many concurrently executed requests may require high amounts of memory. See also -search.maxQueueDuration and -search.maxMemoryPerQuery (default 8)
   -search.maxExportDuration duration
      The maximum duration for /api/v1/export call (default 720h0m0s)
   -search.maxExportSeries int
@@ -2236,6 +2284,9 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      The maximum number of time series, which can be scanned during queries to Graphite Render API. See https://docs.victoriametrics.com/#graphite-render-api-usage . This flag is available only in enterprise version of VictoriaMetrics (default 300000)
   -search.maxLookback duration
      Synonym to -search.lookback-delta from Prometheus. The value is dynamically detected from interval between time series datapoints if not set. It can be overridden on per-query basis via max_lookback arg. See also '-search.maxStalenessInterval' flag, which has the same meaining due to historical reasons
+  -search.maxMemoryPerQuery size
+     The maximum amounts of memory a single query may consume. Queries requiring more memory are rejected. The total memory limit for concurrently executed queries can be estimated as -search.maxMemoryPerQuery multiplied by -search.maxConcurrentRequests
+     Supports the following optional suffixes for size values: KB, MB, GB, KiB, MiB, GiB (default 0)
   -search.maxPointsPerTimeseries int
      The maximum points per a single timeseries returned from /api/v1/query_range. This option doesn't limit the number of scanned raw samples in the database. The main purpose of this option is to limit the number of per-series points returned to graphing UI such as VMUI or Grafana. There is no sense in setting this limit to values bigger than the horizontal resolution of the graph (default 30000)
   -search.maxPointsSubqueryPerTimeseries int
@@ -2305,7 +2356,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      Overrides max size for indexdb/indexBlocks cache. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning
      Supports the following optional suffixes for size values: KB, MB, GB, KiB, MiB, GiB (default 0)
   -storage.cacheSizeIndexDBTagFilters size
-     Overrides max size for indexdb/tagFilters cache. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning
+     Overrides max size for indexdb/tagFiltersToMetricIDs cache. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning
      Supports the following optional suffixes for size values: KB, MB, GB, KiB, MiB, GiB (default 0)
   -storage.cacheSizeStorageTSID size
      Overrides max size for storage/tsid cache. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning

app/vmagent/README.md

@@ -382,7 +382,7 @@ Extra labels can be added to metrics collected by `vmagent` via the following me
   ```
 
   `vmagent` sets `scrape_series_added` to zero when it runs with `-promscrape.noStaleMarkers` command-line option
-  (e.g. when [staleness markers](#prometheus-staleness-markers) are disabled).
+  or when it scrapes target with `no_stale_markers: true` option, e.g. when [staleness markers](#prometheus-staleness-markers) are disabled.
 
 * `scrape_series_limit` - the limit on the number of unique time series the given target can expose according to [these docs](#cardinality-limiter).
   This metric is exposed only if the series limit is set.
@@ -604,9 +604,13 @@ Additionally, the `action: graphite` relabeling rules usually work much faster t
 * If the scrape target is removed from the list of targets, then stale markers are sent for all the metrics scraped from this target.
 
 Prometheus staleness markers' tracking needs additional memory, since it must store the previous response body per each scrape target
-in order to compare it to the current response body. The memory usage may be reduced by passing `-promscrape.noStaleMarkers`
-command-line flag to `vmagent`. This disables staleness tracking. This also disables tracking the number of new time series
-per each scrape with the auto-generated `scrape_series_added` metric. See [these docs](#automatically-generated-metrics) for details.
+in order to compare it to the current response body. The memory usage may be reduced by disabling staleness tracking in the following ways:
+
+* By passing `-promscrape.noStaleMarkers` command-line flag to `vmagent`. This disables staleness tracking across all the targets.
+* By specifying `no_stale_markers: true` option in the [scrape_config](https://docs.victoriametrics.com/sd_configs.html#scrape_configs) for the corresponding target.
+
+When staleness tracking is disabled, then `vmagent` doesn't track the number of new time series per each scrape,
+e.g. it sets `scrape_series_added` metric to zero. See [these docs](#automatically-generated-metrics) for details.
 
 ## Stream parsing mode
 
@@ -904,7 +908,7 @@ See also [troubleshooting docs](https://docs.victoriametrics.com/Troubleshooting
 
 ## Kafka integration
 
-[Enterprise version](https://victoriametrics.com/products/enterprise/) of `vmagent` can read and write metrics from / to Kafka:
+[Enterprise version](https://docs.victoriametrics.com/enterprise.html) of `vmagent` can read and write metrics from / to Kafka:
 
 * [Reading metrics from Kafka](#reading-metrics-from-kafka)
 * [Writing metrics to Kafka](#writing-metrics-to-kafka)
@@ -914,7 +918,7 @@ in `vmutils-...-enteprise.tar.gz` archives and in [docker images](https://hub.do
 
 ### Reading metrics from Kafka
 
-[Enterprise version](https://victoriametrics.com/products/enterprise/) of `vmagent` can read metrics in various formats from Kafka messages.
+[Enterprise version](https://docs.victoriametrics.com/enterprise.html) of `vmagent` can read metrics in various formats from Kafka messages.
 These formats can be configured with `-kafka.consumer.topic.defaultFormat` or `-kafka.consumer.topic.format` command-line options. The following formats are supported:
 
 * `promremotewrite` - [Prometheus remote_write](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write).
@@ -956,7 +960,7 @@ data_format = "influx"
 
 #### Command-line flags for Kafka consumer
 
-These command-line flags are available only in [enterprise](https://victoriametrics.com/products/enterprise/) version of `vmagent`,
+These command-line flags are available only in [enterprise](https://docs.victoriametrics.com/enterprise.html) version of `vmagent`,
 which can be downloaded for evaluation from [releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) page
 (see `vmutils-...-enteprise.tar.gz` archives) and from [docker images](https://hub.docker.com/r/victoriametrics/vmagent/tags) with tags containing `enterprise` suffix.
 
@@ -991,7 +995,7 @@ which can be downloaded for evaluation from [releases](https://github.com/Victor
 
 ### Writing metrics to Kafka
 
-[Enterprise version](https://victoriametrics.com/products/enterprise/) of `vmagent` writes data to Kafka with `at-least-once`
+[Enterprise version](https://docs.victoriametrics.com/enterprise.html) of `vmagent` writes data to Kafka with `at-least-once`
 semantics if `-remoteWrite.url` contains e.g. Kafka url. For example, if `vmagent` is started with `-remoteWrite.url=kafka://localhost:9092/?topic=prom-rw`,
 then it would send Prometheus remote_write messages to Kafka bootstrap server at `localhost:9092` with the topic `prom-rw`.
 These messages can be read later from Kafka by another `vmagent` - see [these docs](#reading-metrics-from-kafka) for details.
@@ -1023,7 +1027,7 @@ It may be needed to build `vmagent` from source code when developing or testing
 
 ### Development build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.2.
 2. Run `make vmagent` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds the `vmagent` binary and puts it into the `bin` folder.
 
@@ -1052,7 +1056,7 @@ ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://b
 
 ### Development ARM build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.2.
 2. Run `make vmagent-linux-arm` or `make vmagent-linux-arm64` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics)
    It builds `vmagent-linux-arm` or `vmagent-linux-arm64` binary respectively and puts it into the `bin` folder.
 

app/vmagent/remotewrite/relabel.go

@@ -123,7 +123,8 @@ func (rctx *relabelCtx) applyRelabeling(tss []prompbmarshal.TimeSeries, extraLab
 				}
 			}
 		}
-		labels = pcs.Apply(labels, labelsLen, true)
+		labels = pcs.Apply(labels, labelsLen)
+		labels = promrelabel.FinalizeLabels(labels[:labelsLen], labels[labelsLen:])
 		if len(labels) == labelsLen {
 			// Drop the current time series, since relabeling removed all the labels.
 			continue

app/vmalert/README.md

@@ -213,13 +213,51 @@ The following variables are available in templating:
 | $labels or .Labels                 | The list of labels of the current alert. Use as ".Labels.<label_name>".                                   | {% raw %}Too high number of connections for {{ .Labels.instance }}{% endraw %}                                                                                         |
 | $alertID or .AlertID               | The current alert's ID generated by vmalert.                                                              | {% raw %}Link: vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}{% endraw %}                                                                                   |
 | $groupID or .GroupID               | The current alert's group ID generated by vmalert.                                                        | {% raw %}Link: vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}{% endraw %}                                                                                   |
-| $expr or .Expr                     | Alert's expression. Can be used for generating links to Grafana or other systems.                         | {% raw %}/api/v1/query?query={{ $expr&#124;quotesEscape&#124;queryEscape }}{% endraw %}                                                                                |
+| $expr or .Expr                     | Alert's expression. Can be used for generating links to Grafana or other systems.                         | {% raw %}/api/v1/query?query={{ $expr&#124;queryEscape }}{% endraw %}                                                                                |
 | $externalLabels or .ExternalLabels | List of labels configured via `-external.label` command-line flag.                                        | {% raw %}Issues with {{ $labels.instance }} (datacenter-{{ $externalLabels.dc }}){% endraw %}                                                                          |
 | $externalURL or .ExternalURL       | URL configured via `-external.url` command-line flag. Used for cases when vmalert is hidden behind proxy. | {% raw %}Visit {{ $externalURL }} for more details{% endraw %}                                                                                                         |
 
-Additionally, `vmalert` provides some extra templating functions
-listed [here](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmalert/templates/template.go)
-and [reusable templates](#reusable-templates).
+Additionally, `vmalert` provides some extra templating functions listed [here](#template-functions) and [reusable templates](#reusable-templates).
+
+#### Template functions
+
+`vmalert` provides the following template functions, which can be used during [templating](#templating):
+
+- `args arg0 ... argN` - converts the input args into a map with `arg0`, ..., `argN` keys.
+- `externalURL` - returns the value of `-external.url` command-line flag.
+- `first` - returns the first result from the input query results returned by `query` function.
+- `htmlEscape` - escapes special chars in input string, so it can be safely embedded as a plaintext into HTML.
+- `humanize` - converts the input number into human-readable format by adding [metric prefixes](https://en.wikipedia.org/wiki/Metric_prefix).
+  For example, `100000` is converted into `100K`.
+- `humanize1024` - converts the input number into human-readable format with 1024 base.
+  For example, `1024` is converted into 1ki`.
+- `humanizeDuration` - converts the input number in seconds into human-readable duration.
+- `humanizePercentage` - converts the input number to percentage. For example, `0.123` is converted into `12.3%`.
+- `humanizeTimestamp` - converts the input unix timestamp into human-readable time.
+- `jsonEscape` - JSON-encodes the input string.
+- `label name` - returns the value of the label with the given `name` from the input query result.
+- `match regex` - matches the input string against the provided `regex`.
+- `parseDuration` - parses the input string into duration in seconds. For example, `1h` is parsed into `3600`.
+- `parseDurationTime` - parses the input string into [time.Duration](https://pkg.go.dev/time#Duration).
+- `pathEscape` - escapes the input string, so it can be safely put inside path part of URL.
+- `pathPrefix` - returns the path part of the `-external.url` command-line flag.
+- `query` - executes the [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html) query against `-datasource.url` and returns the query result.
+  For example, {% raw %}`{{ query "sort_desc(process_resident_memory_bytes)" | first | value }}`{% endraw %} executes the `sort_desc(process_resident_memory_bytes)`
+  query at `-datasource.url` and returns the first result.
+- `queryEscape` - escapes the input string, so it can be safely put inside [query arg](https://en.wikipedia.org/wiki/Percent-encoding) part of URL.
+- `quotesEscape` - escapes the input string, so it can be safely embedded into JSON string.
+- `reReplaceAll regex repl` - replaces all the occurences of the `regex` in input string with the `repl`.
+- `safeHtml` - marks the input string as safe to use in HTML context without the need to html-escape it.
+- `sortByLabel name` - sorts the input query results by the label with the given `name`.
+- `stripDomain` - leaves the first part of the domain. For example, `foo.bar.baz` is converted to `foo`.
+  The port part is left in the output string. E.g. `foo.bar:1234` is converted into `foo:1234`.
+- `stripPort` - strips `port` part from `host:port` input string.
+- `strvalue` - returns the metric name from the input query result.
+- `title` - converts the first letters of every input word to uppercase.
+- `toLower` - converts all the chars in the input string to lowercase.
+- `toTime` - converts the input unix timestamp to [time.Time](https://pkg.go.dev/time#Time).
+- `toUpper` - converts all the chars in the input string to uppercase.
+- `value` - returns the numeric value from the input query result.
 
 #### Reusable templates
 
@@ -315,7 +353,7 @@ There are the following approaches exist for alerting and recording rules across
   rules to `AccountID=123`.
 
 * To specify `tenant` parameter per each alerting and recording group if
-  [enterprise version of vmalert](https://victoriametrics.com/products/enterprise/) is used
+  [enterprise version of vmalert](https://docs.victoriametrics.com/enterprise.html) is used
   with `-clusterMode` command-line flag. For example:
 
 ```yaml
@@ -665,9 +703,10 @@ Try the following recommendations in such cases:
 are delivered to the datasource;
 * If you know in advance, that data in datasource is delayed - try changing vmalert's `-datasource.lookback`
 command-line flag to add a time shift for evaluations;
-* If time intervals between datapoints in datasource are irregular - try changing vmalert's `-datasource.queryStep`
-command-line flag to specify how far search query can lookback for the recent datapoint. By default, this value
-is equal to group's evaluation interval.
+* If time intervals between datapoints in datasource are irregular or `>=5min` - try changing vmalert's
+`-datasource.queryStep` command-line flag to specify how far search query can lookback for the recent datapoint. 
+The recommendation is to have the step at least two times bigger than `scrape_interval`, since
+there are no guarantees that scrape will not fail.
 
 Sometimes, it is not clear why some specific alert fired or didn't fire. It is very important to remember, that
 alerts with `for: 0` fire immediately when their expression becomes true. And alerts with `for > 0` will fire only
@@ -814,7 +853,7 @@ The shortlist of configuration flags is the following:
   -evaluationInterval duration
      How often to evaluate the rules (default 1m0s)
   -external.alert.source string
-     External Alert Source allows to override the Source link for alerts sent to AlertManager for cases where you want to build a custom link to Grafana, Prometheus or any other service. Supports templating - see https://docs.victoriametrics.com/vmalert.html#templating . For example, link to Grafana: -external.alert.source='explore?orgId=1&left=[\"now-1h\",\"now\",\"VictoriaMetrics\",{\"expr\": \"{{$expr|quotesEscape|crlfEscape|queryEscape}}\"},{\"mode\":\"Metrics\"},{\"ui\":[true,true,true,\"none\"]}]' . If empty 'vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}' is used
+     External Alert Source allows to override the Source link for alerts sent to AlertManager for cases where you want to build a custom link to Grafana, Prometheus or any other service. Supports templating - see https://docs.victoriametrics.com/vmalert.html#templating . For example, link to Grafana: -external.alert.source='explore?orgId=1&left=["now-1h","now","VictoriaMetrics",{"expr":{{$expr|jsonEscape|queryEscape}} },{"mode":"Metrics"},{"ui":[true,true,true,"none"]}]' . If empty 'vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}' is used
      If empty 'vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}' is used.
   -external.label array
      Optional label in the form 'Name=value' to add to all generated recording rules and alerts. Pass multiple -label flags in order to add multiple label sets.
@@ -1275,7 +1314,7 @@ spec:
 
 ### Development build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.2.
 2. Run `make vmalert` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `vmalert` binary and puts it into the `bin` folder.
 
@@ -1291,7 +1330,7 @@ ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://b
 
 ### Development ARM build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.2.
 2. Run `make vmalert-linux-arm` or `make vmalert-linux-arm64` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `vmalert-linux-arm` or `vmalert-linux-arm64` binary respectively and puts it into the `bin` folder.
 

app/vmalert/config/config.go

@@ -243,9 +243,12 @@ func Parse(pathPatterns []string, validateTplFn ValidateTplFn, validateExpressio
 func parseFile(path string) ([]Group, error) {
 	data, err := os.ReadFile(path)
 	if err != nil {
-		return nil, fmt.Errorf("error reading alert rule file: %w", err)
+		return nil, fmt.Errorf("error reading alert rule file %q: %w", path, err)
+	}
+	data, err = envtemplate.ReplaceBytes(data)
+	if err != nil {
+		return nil, fmt.Errorf("cannot expand environment vars in %q: %w", path, err)
 	}
-	data = envtemplate.Replace(data)
 	g := struct {
 		Groups []Group `yaml:"groups"`
 		// Catches all undefined fields and must be empty after parsing.

app/vmalert/main.go

@@ -62,8 +62,8 @@ absolute path to all .tpl files in root.`)
 	externalAlertSource = flag.String("external.alert.source", "", `External Alert Source allows to override the Source link for alerts sent to AlertManager `+
 		`for cases where you want to build a custom link to Grafana, Prometheus or any other service. `+
 		`Supports templating - see https://docs.victoriametrics.com/vmalert.html#templating . `+
-		`For example, link to Grafana: -external.alert.source='explore?orgId=1&left=[\"now-1h\",\"now\",\"VictoriaMetrics\",{\"expr\": \"{{$expr|quotesEscape|crlfEscape|queryEscape}}\"},{\"mode\":\"Metrics\"},{\"ui\":[true,true,true,\"none\"]}]' . `+
-		`If empty 'vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}' is used`)
+		`For example, link to Grafana: -external.alert.source='explore?orgId=1&left=["now-1h","now","VictoriaMetrics",{"expr":{{$expr|jsonEscape|queryEscape}} },{"mode":"Metrics"},{"ui":[true,true,true,"none"]}]' . `+
+		`If empty 'vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}' is used.`)
 	externalLabels = flagutil.NewArrayString("external.label", "Optional label in the form 'Name=value' to add to all generated recording rules and alerts. "+
 		"Pass multiple -label flags in order to add multiple label sets.")
 

app/vmalert/notifier/alert.go

@@ -183,9 +183,9 @@ func (a Alert) toPromLabels(relabelCfg *promrelabel.ParsedConfigs) []prompbmarsh
 			Value: v,
 		})
 	}
-	promrelabel.SortLabels(labels)
 	if relabelCfg != nil {
-		return relabelCfg.Apply(labels, 0, false)
+		labels = relabelCfg.Apply(labels, 0)
 	}
+	promrelabel.SortLabels(labels)
 	return labels
 }

app/vmalert/notifier/alert_test.go

@@ -67,15 +67,21 @@ func TestAlert_ExecTemplate(t *testing.T) {
 		{
 			name: "expression-template",
 			alert: &Alert{
-				Expr: `vm_rows{"label"="bar"}>0`,
+				Expr: `vm_rows{"label"="bar"}<0`,
 			},
 			annotations: map[string]string{
-				"exprEscapedQuery": "{{ $expr|quotesEscape|queryEscape }}",
-				"exprEscapedPath":  "{{ $expr|quotesEscape|pathEscape }}",
+				"exprEscapedQuery":  "{{ $expr|queryEscape }}",
+				"exprEscapedPath":   "{{ $expr|pathEscape }}",
+				"exprEscapedJSON":   "{{ $expr|jsonEscape }}",
+				"exprEscapedQuotes": "{{ $expr|quotesEscape }}",
+				"exprEscapedHTML":   "{{ $expr|htmlEscape }}",
 			},
 			expTpl: map[string]string{
-				"exprEscapedQuery": "vm_rows%7B%5C%22label%5C%22%3D%5C%22bar%5C%22%7D%3E0",
-				"exprEscapedPath":  "vm_rows%7B%5C%22label%5C%22=%5C%22bar%5C%22%7D%3E0",
+				"exprEscapedQuery":  "vm_rows%7B%22label%22%3D%22bar%22%7D%3C0",
+				"exprEscapedPath":   "vm_rows%7B%22label%22=%22bar%22%7D%3C0",
+				"exprEscapedJSON":   `"vm_rows{\"label\"=\"bar\"}\u003c0"`,
+				"exprEscapedQuotes": `vm_rows{\"label\"=\"bar\"}\u003c0`,
+				"exprEscapedHTML":   "vm_rows{&quot;label&quot;=&quot;bar&quot;}&lt;0",
 			},
 		},
 		{

app/vmalert/notifier/alertmanager_request.qtpl

@@ -15,10 +15,10 @@
     "endsAt":{%q= alert.End.Format(time.RFC3339Nano) %},
     {% endif %}
     "labels": {
-        "alertname":{%q= alert.Name %}
         {% code lbls := alert.toPromLabels(relabelCfg) %}
-        {% for _, l := range lbls %}
-            ,{%q= l.Name %}:{%q= l.Value %}
+        {% code ll := len(lbls) %}
+        {% for idx, l := range lbls %}
+            {%q= l.Name %}:{%q= l.Value %}{% if idx != ll-1 %}, {% endif %}
         {% endfor %}
     },
     "annotations": {

app/vmalert/notifier/alertmanager_request.qtpl.go

@@ -51,22 +51,27 @@ func streamamRequest(qw422016 *qt422016.Writer, alerts []Alert, generatorURL fun
 //line app/vmalert/notifier/alertmanager_request.qtpl:16
 		}
 //line app/vmalert/notifier/alertmanager_request.qtpl:16
-		qw422016.N().S(`"labels": {"alertname":`)
+		qw422016.N().S(`"labels": {`)
 //line app/vmalert/notifier/alertmanager_request.qtpl:18
-		qw422016.N().Q(alert.Name)
-//line app/vmalert/notifier/alertmanager_request.qtpl:19
 		lbls := alert.toPromLabels(relabelCfg)
 
+//line app/vmalert/notifier/alertmanager_request.qtpl:19
+		ll := len(lbls)
+
 //line app/vmalert/notifier/alertmanager_request.qtpl:20
-		for _, l := range lbls {
-//line app/vmalert/notifier/alertmanager_request.qtpl:20
-			qw422016.N().S(`,`)
+		for idx, l := range lbls {
 //line app/vmalert/notifier/alertmanager_request.qtpl:21
 			qw422016.N().Q(l.Name)
 //line app/vmalert/notifier/alertmanager_request.qtpl:21
 			qw422016.N().S(`:`)
 //line app/vmalert/notifier/alertmanager_request.qtpl:21
 			qw422016.N().Q(l.Value)
+//line app/vmalert/notifier/alertmanager_request.qtpl:21
+			if idx != ll-1 {
+//line app/vmalert/notifier/alertmanager_request.qtpl:21
+				qw422016.N().S(`,`)
+//line app/vmalert/notifier/alertmanager_request.qtpl:21
+			}
 //line app/vmalert/notifier/alertmanager_request.qtpl:22
 		}
 //line app/vmalert/notifier/alertmanager_request.qtpl:22

app/vmalert/notifier/alertmanager_test.go

@@ -67,9 +67,6 @@ func TestAlertManager_Send(t *testing.T) {
 			if a[0].GeneratorURL != "0/0" {
 				t.Errorf("expected 0/0 as generatorURL got %s", a[0].GeneratorURL)
 			}
-			if a[0].Labels["alertname"] != "alert0" {
-				t.Errorf("expected alert0 as alert name got %s", a[0].Labels["alertname"])
-			}
 			if a[0].StartsAt.IsZero() {
 				t.Errorf("expected non-zero start time")
 			}

app/vmalert/notifier/config.go

@@ -132,8 +132,9 @@ func parseConfig(path string) (*Config, error) {
 
 func parseLabels(target string, metaLabels map[string]string, cfg *Config) (string, []prompbmarshal.Label, error) {
 	labels := mergeLabels(target, metaLabels, cfg)
-	labels = cfg.parsedRelabelConfigs.Apply(labels, 0, false)
+	labels = cfg.parsedRelabelConfigs.Apply(labels, 0)
 	labels = promrelabel.RemoveMetaLabels(labels[:0], labels)
+	promrelabel.SortLabels(labels)
 	// Remove references to already deleted labels, so GC could clean strings for label name and label value past len(labels).
 	// This should reduce memory usage when relabeling creates big number of temporary labels with long names and/or values.
 	// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825 for details.

app/vmalert/remotewrite/remotewrite.go

@@ -218,7 +218,7 @@ func (c *Client) flush(ctx context.Context, wr *prompbmarshal.WriteRequest) {
 			return
 		}
 
-		logger.Errorf("attempt %d to send request failed: %s", i+1, err)
+		logger.Warnf("attempt %d to send request failed: %s", i+1, err)
 		// sleeping to avoid remote db hammering
 		time.Sleep(time.Second)
 		continue

app/vmalert/templates/funcs.qtpl

@@ -0,0 +1,15 @@
+{% stripspace %}
+
+{% func quotesEscape(s string) %}
+	{%j= s %}
+{% endfunc %}
+
+{% func jsonEscape(s string) %}
+	{%q= s %}
+{% endfunc %}
+
+{% func htmlEscape(s string) %}
+	{%s s %}
+{% endfunc %}
+
+{% endstripspace %}

app/vmalert/templates/funcs.qtpl.go

@@ -0,0 +1,117 @@
+// Code generated by qtc from "funcs.qtpl". DO NOT EDIT.
+// See https://github.com/valyala/quicktemplate for details.
+
+//line app/vmalert/templates/funcs.qtpl:3
+package templates
+
+//line app/vmalert/templates/funcs.qtpl:3
+import (
+	qtio422016 "io"
+
+	qt422016 "github.com/valyala/quicktemplate"
+)
+
+//line app/vmalert/templates/funcs.qtpl:3
+var (
+	_ = qtio422016.Copy
+	_ = qt422016.AcquireByteBuffer
+)
+
+//line app/vmalert/templates/funcs.qtpl:3
+func streamquotesEscape(qw422016 *qt422016.Writer, s string) {
+//line app/vmalert/templates/funcs.qtpl:4
+	qw422016.N().J(s)
+//line app/vmalert/templates/funcs.qtpl:5
+}
+
+//line app/vmalert/templates/funcs.qtpl:5
+func writequotesEscape(qq422016 qtio422016.Writer, s string) {
+//line app/vmalert/templates/funcs.qtpl:5
+	qw422016 := qt422016.AcquireWriter(qq422016)
+//line app/vmalert/templates/funcs.qtpl:5
+	streamquotesEscape(qw422016, s)
+//line app/vmalert/templates/funcs.qtpl:5
+	qt422016.ReleaseWriter(qw422016)
+//line app/vmalert/templates/funcs.qtpl:5
+}
+
+//line app/vmalert/templates/funcs.qtpl:5
+func quotesEscape(s string) string {
+//line app/vmalert/templates/funcs.qtpl:5
+	qb422016 := qt422016.AcquireByteBuffer()
+//line app/vmalert/templates/funcs.qtpl:5
+	writequotesEscape(qb422016, s)
+//line app/vmalert/templates/funcs.qtpl:5
+	qs422016 := string(qb422016.B)
+//line app/vmalert/templates/funcs.qtpl:5
+	qt422016.ReleaseByteBuffer(qb422016)
+//line app/vmalert/templates/funcs.qtpl:5
+	return qs422016
+//line app/vmalert/templates/funcs.qtpl:5
+}
+
+//line app/vmalert/templates/funcs.qtpl:7
+func streamjsonEscape(qw422016 *qt422016.Writer, s string) {
+//line app/vmalert/templates/funcs.qtpl:8
+	qw422016.N().Q(s)
+//line app/vmalert/templates/funcs.qtpl:9
+}
+
+//line app/vmalert/templates/funcs.qtpl:9
+func writejsonEscape(qq422016 qtio422016.Writer, s string) {
+//line app/vmalert/templates/funcs.qtpl:9
+	qw422016 := qt422016.AcquireWriter(qq422016)
+//line app/vmalert/templates/funcs.qtpl:9
+	streamjsonEscape(qw422016, s)
+//line app/vmalert/templates/funcs.qtpl:9
+	qt422016.ReleaseWriter(qw422016)
+//line app/vmalert/templates/funcs.qtpl:9
+}
+
+//line app/vmalert/templates/funcs.qtpl:9
+func jsonEscape(s string) string {
+//line app/vmalert/templates/funcs.qtpl:9
+	qb422016 := qt422016.AcquireByteBuffer()
+//line app/vmalert/templates/funcs.qtpl:9
+	writejsonEscape(qb422016, s)
+//line app/vmalert/templates/funcs.qtpl:9
+	qs422016 := string(qb422016.B)
+//line app/vmalert/templates/funcs.qtpl:9
+	qt422016.ReleaseByteBuffer(qb422016)
+//line app/vmalert/templates/funcs.qtpl:9
+	return qs422016
+//line app/vmalert/templates/funcs.qtpl:9
+}
+
+//line app/vmalert/templates/funcs.qtpl:11
+func streamhtmlEscape(qw422016 *qt422016.Writer, s string) {
+//line app/vmalert/templates/funcs.qtpl:12
+	qw422016.E().S(s)
+//line app/vmalert/templates/funcs.qtpl:13
+}
+
+//line app/vmalert/templates/funcs.qtpl:13
+func writehtmlEscape(qq422016 qtio422016.Writer, s string) {
+//line app/vmalert/templates/funcs.qtpl:13
+	qw422016 := qt422016.AcquireWriter(qq422016)
+//line app/vmalert/templates/funcs.qtpl:13
+	streamhtmlEscape(qw422016, s)
+//line app/vmalert/templates/funcs.qtpl:13
+	qt422016.ReleaseWriter(qw422016)
+//line app/vmalert/templates/funcs.qtpl:13
+}
+
+//line app/vmalert/templates/funcs.qtpl:13
+func htmlEscape(s string) string {
+//line app/vmalert/templates/funcs.qtpl:13
+	qb422016 := qt422016.AcquireByteBuffer()
+//line app/vmalert/templates/funcs.qtpl:13
+	writehtmlEscape(qb422016, s)
+//line app/vmalert/templates/funcs.qtpl:13
+	qs422016 := string(qb422016.B)
+//line app/vmalert/templates/funcs.qtpl:13
+	qt422016.ReleaseByteBuffer(qb422016)
+//line app/vmalert/templates/funcs.qtpl:13
+	return qs422016
+//line app/vmalert/templates/funcs.qtpl:13
+}

app/vmalert/templates/template.go

@@ -207,23 +207,10 @@ func FuncsWithExternalURL(externalURL *url.URL) textTpl.FuncMap {
 // templateFuncs initiates template helper functions
 func templateFuncs() textTpl.FuncMap {
 	// See https://prometheus.io/docs/prometheus/latest/configuration/template_reference/
+	// and https://github.com/prometheus/prometheus/blob/fa6e05903fd3ce52e374a6e1bf4eb98c9f1f45a7/template/template.go#L150
 	return textTpl.FuncMap{
 		/* Strings */
 
-		// reReplaceAll ReplaceAllString returns a copy of src, replacing matches of the Regexp with
-		// the replacement string repl. Inside repl, $ signs are interpreted as in Expand,
-		// so for instance $1 represents the text of the first submatch.
-		// alias for https://golang.org/pkg/regexp/#Regexp.ReplaceAllString
-		"reReplaceAll": func(pattern, repl, text string) string {
-			re := regexp.MustCompile(pattern)
-			return re.ReplaceAllString(text, repl)
-		},
-
-		// match reports whether the string s
-		// contains any match of the regular expression pattern.
-		// alias for https://golang.org/pkg/regexp/#MatchString
-		"match": regexp.MatchString,
-
 		// title returns a copy of the string s with all Unicode letters
 		// that begin words mapped to their Unicode title case.
 		// alias for https://golang.org/pkg/strings/#Title
@@ -237,6 +224,31 @@ func templateFuncs() textTpl.FuncMap {
 		// alias for https://golang.org/pkg/strings/#ToLower
 		"toLower": strings.ToLower,
 
+		// crlfEscape replaces '\n' and '\r' chars with `\\n` and `\\r`.
+		// This funcion is deprectated.
+		//
+		// It is better to use quotesEscape, jsonEscape, queryEscape or pathEscape instead -
+		// these functions properly escape `\n` and `\r` chars according to their purpose.
+		"crlfEscape": func(q string) string {
+			q = strings.Replace(q, "\n", `\n`, -1)
+			return strings.Replace(q, "\r", `\r`, -1)
+		},
+
+		// quotesEscape escapes the string, so it can be safely put inside JSON string.
+		//
+		// See also jsonEscape.
+		"quotesEscape": quotesEscape,
+
+		// jsonEscape converts the string to properly encoded JSON string.
+		//
+		// See also quotesEscape.
+		"jsonEscape": jsonEscape,
+
+		// htmlEscape applies html-escaping to q, so it can be safely embedded as plaintext into html.
+		//
+		// See also safeHtml.
+		"htmlEscape": htmlEscape,
+
 		// stripPort splits string into host and port, then returns only host.
 		"stripPort": func(hostPort string) string {
 			host, _, err := net.SplitHostPort(hostPort)
@@ -246,6 +258,37 @@ func templateFuncs() textTpl.FuncMap {
 			return host
 		},
 
+		// stripDomain removes the domain part of a FQDN. Leaves port untouched.
+		"stripDomain": func(hostPort string) string {
+			host, port, err := net.SplitHostPort(hostPort)
+			if err != nil {
+				host = hostPort
+			}
+			ip := net.ParseIP(host)
+			if ip != nil {
+				return hostPort
+			}
+			host = strings.Split(host, ".")[0]
+			if port != "" {
+				return net.JoinHostPort(host, port)
+			}
+			return host
+		},
+
+		// match reports whether the string s
+		// contains any match of the regular expression pattern.
+		// alias for https://golang.org/pkg/regexp/#MatchString
+		"match": regexp.MatchString,
+
+		// reReplaceAll ReplaceAllString returns a copy of src, replacing matches of the Regexp with
+		// the replacement string repl. Inside repl, $ signs are interpreted as in Expand,
+		// so for instance $1 represents the text of the first submatch.
+		// alias for https://golang.org/pkg/regexp/#Regexp.ReplaceAllString
+		"reReplaceAll": func(pattern, repl, text string) string {
+			re := regexp.MustCompile(pattern)
+			return re.ReplaceAllString(text, repl)
+		},
+
 		// parseDuration parses a duration string such as "1h" into the number of seconds it represents
 		"parseDuration": func(s string) (float64, error) {
 			d, err := promutils.ParseDuration(s)
@@ -421,31 +464,15 @@ func templateFuncs() textTpl.FuncMap {
 			return ""
 		},
 
-		// pathEscape escapes the string so it can be safely placed inside a URL path segment,
-		// replacing special characters (including /) with %XX sequences as needed.
-		// alias for https://golang.org/pkg/net/url/#PathEscape
-		"pathEscape": func(u string) string {
-			return url.PathEscape(u)
-		},
+		// pathEscape escapes the string so it can be safely placed inside a URL path segment.
+		//
+		// See also queryEscape.
+		"pathEscape": url.PathEscape,
 
-		// queryEscape escapes the string so it can be safely placed
-		// inside a URL query.
-		// alias for https://golang.org/pkg/net/url/#QueryEscape
-		"queryEscape": func(q string) string {
-			return url.QueryEscape(q)
-		},
-
-		// crlfEscape replaces new line chars to skip URL encoding.
-		// see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/890
-		"crlfEscape": func(q string) string {
-			q = strings.Replace(q, "\n", `\n`, -1)
-			return strings.Replace(q, "\r", `\r`, -1)
-		},
-
-		// quotesEscape escapes quote char
-		"quotesEscape": func(q string) string {
-			return strings.Replace(q, `"`, `\"`, -1)
-		},
+		// queryEscape escapes the string so it can be safely placed inside a query arg in URL.
+		//
+		// See also queryEscape.
+		"queryEscape": url.QueryEscape,
 
 		// query executes the MetricsQL/PromQL query against
 		// configured `datasource.url` address.
@@ -477,6 +504,17 @@ func templateFuncs() textTpl.FuncMap {
 			return m.Labels[label]
 		},
 
+		// value returns the value of the given metric.
+		// usually used alongside with `query` template function.
+		"value": func(m metric) float64 {
+			return m.Value
+		},
+
+		// strvalue returns metric name.
+		"strvalue": func(m metric) string {
+			return m.Labels["__name__"]
+		},
+
 		// sortByLabel sorts the given metrics by provided label key
 		"sortByLabel": func(label string, metrics []metric) []metric {
 			sort.SliceStable(metrics, func(i, j int) bool {
@@ -485,12 +523,6 @@ func templateFuncs() textTpl.FuncMap {
 			return metrics
 		},
 
-		// value returns the value of the given metric.
-		// usually used alongside with `query` template function.
-		"value": func(m metric) float64 {
-			return m.Value
-		},
-
 		/* Helpers */
 
 		// Converts a list of objects to a map with keys arg0, arg1 etc.
@@ -504,6 +536,8 @@ func templateFuncs() textTpl.FuncMap {
 		},
 
 		// safeHtml marks string as HTML not requiring auto-escaping.
+		//
+		// See also htmlEscape.
 		"safeHtml": func(text string) htmlTpl.HTML {
 			return htmlTpl.HTML(text)
 		},

app/vmalert/templates/template_test.go

@@ -6,6 +6,52 @@ import (
 	textTpl "text/template"
 )
 
+func TestTemplateFuncs(t *testing.T) {
+	funcs := templateFuncs()
+	f := func(funcName, s, resultExpected string) {
+		t.Helper()
+		v := funcs[funcName]
+		fLocal := v.(func(s string) string)
+		result := fLocal(s)
+		if result != resultExpected {
+			t.Fatalf("unexpected result for %s(%q); got\n%s\nwant\n%s", funcName, s, result, resultExpected)
+		}
+	}
+	f("title", "foo bar", "Foo Bar")
+	f("toUpper", "foo", "FOO")
+	f("toLower", "FOO", "foo")
+	f("pathEscape", "foo/bar\n+baz", "foo%2Fbar%0A+baz")
+	f("queryEscape", "foo+bar\n+baz", "foo%2Bbar%0A%2Bbaz")
+	f("jsonEscape", `foo{bar="baz"}`+"\n + 1", `"foo{bar=\"baz\"}\n + 1"`)
+	f("quotesEscape", `foo{bar="baz"}`+"\n + 1", `foo{bar=\"baz\"}\n + 1`)
+	f("htmlEscape", "foo < 10\nabc", "foo &lt; 10\nabc")
+	f("crlfEscape", "foo\nbar\rx", `foo\nbar\rx`)
+	f("stripPort", "foo", "foo")
+	f("stripPort", "foo:1234", "foo")
+	f("stripDomain", "foo.bar.baz", "foo")
+	f("stripDomain", "foo.bar:123", "foo:123")
+
+	// check "match" func
+	matchFunc := funcs["match"].(func(pattern, s string) (bool, error))
+	if _, err := matchFunc("invalid[regexp", "abc"); err == nil {
+		t.Fatalf("expecting non-nil error on invalid regexp")
+	}
+	ok, err := matchFunc("abc", "def")
+	if err != nil {
+		t.Fatalf("unexpected error")
+	}
+	if ok {
+		t.Fatalf("unexpected match")
+	}
+	ok, err = matchFunc("a.+b", "acsdb")
+	if err != nil {
+		t.Fatalf("unexpected error")
+	}
+	if !ok {
+		t.Fatalf("unexpected mismatch")
+	}
+}
+
 func mkTemplate(current, replacement interface{}) textTemplate {
 	tmpl := textTemplate{}
 	if current != nil {

app/vmalert/web_types.go

@@ -5,11 +5,11 @@ import (
 	"time"
 )
 
-// APIAlert represents a notifier.AlertingRule ruleState
+// APIAlert represents a notifier.AlertingRule state
 // for WEB view
 // https://github.com/prometheus/compliance/blob/main/alert_generator/specification.md#get-apiv1rules
 type APIAlert struct {
-	State       string            `json:"ruleState"`
+	State       string            `json:"state"`
 	Name        string            `json:"name"`
 	Value       string            `json:"value"`
 	Labels      map[string]string `json:"labels,omitempty"`
@@ -30,7 +30,7 @@ type APIAlert struct {
 	// SourceLink contains a link to a system which should show
 	// why Alert was generated
 	SourceLink string `json:"source"`
-	// Restored shows whether Alert's ruleState was restored on restart
+	// Restored shows whether Alert's state was restored on restart
 	Restored bool `json:"restored"`
 }
 
@@ -86,10 +86,10 @@ type GroupAlerts struct {
 // see https://github.com/prometheus/compliance/blob/main/alert_generator/specification.md#get-apiv1rules
 type APIRule struct {
 	// State must be one of these under following scenarios
-	//  "pending": at least 1 alert in the rule in pending ruleState and no other alert in firing ruleState.
-	//  "firing": at least 1 alert in the rule in firing ruleState.
-	//  "inactive": no alert in the rule in firing or pending ruleState.
-	State string `json:"ruleState"`
+	//  "pending": at least 1 alert in the rule in pending state and no other alert in firing ruleState.
+	//  "firing": at least 1 alert in the rule in firing state.
+	//  "inactive": no alert in the rule in firing or pending state.
+	State string `json:"state"`
 	Name  string `json:"name"`
 	// Query represents Rule's `expression` field
 	Query string `json:"query"`
@@ -121,7 +121,7 @@ type APIRule struct {
 	// GroupID is an unique Group's ID
 	GroupID string `json:"group_id"`
 
-	// TODO:
+	// Updates contains the ordered list of recorded ruleStateEntry objects
 	Updates []ruleStateEntry `json:"updates"`
 }
 

app/vmauth/README.md

@@ -167,7 +167,7 @@ It is recommended using [binary releases](https://github.com/VictoriaMetrics/Vic
 
 ### Development build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.2.
 2. Run `make vmauth` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `vmauth` binary and puts it into the `bin` folder.
 

app/vmauth/auth_config.go

@@ -250,7 +250,11 @@ func readAuthConfig(path string) (map[string]*UserInfo, error) {
 }
 
 func parseAuthConfig(data []byte) (map[string]*UserInfo, error) {
-	data = envtemplate.Replace(data)
+	var err error
+	data, err = envtemplate.ReplaceBytes(data)
+	if err != nil {
+		return nil, fmt.Errorf("cannot expand environment vars: %w", err)
+	}
 	var ac AuthConfig
 	if err := yaml.UnmarshalStrict(data, &ac); err != nil {
 		return nil, fmt.Errorf("cannot unmarshal AuthConfig data: %w", err)

app/vmbackup/README.md

@@ -2,14 +2,6 @@
 
 `vmbackup` creates VictoriaMetrics data backups from [instant snapshots](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-work-with-snapshots).
 
-Supported storage systems for backups:
-
-* [GCS](https://cloud.google.com/storage/). Example: `gs://<bucket>/<path/to/backup>`
-* [S3](https://aws.amazon.com/s3/). Example: `s3://<bucket>/<path/to/backup>`
-* [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs/). Example: `azblob://<bucket>/<path/to/backup>`
-* Any S3-compatible storage such as [MinIO](https://github.com/minio/minio), [Ceph](https://docs.ceph.com/en/pacific/radosgw/s3/) or [Swift](https://platform.swiftstack.com/docs/admin/middleware/s3_middleware.html). See [these docs](#advanced-usage) for details.
-* Local filesystem. Example: `fs://</absolute/path/to/backup>`. Note that `vmbackup` prevents from storing the backup into the directory pointed by `-storageDataPath` command-line flag, since this directory should be managed solely by VictoriaMetrics or `vmstorage`.
-
 `vmbackup` supports incremental and full backups. Incremental backups are created automatically if the destination path already contains data from the previous backup.
 Full backups can be sped up with `-origin` pointing to an already existing backup on the same remote storage. In this case `vmbackup` makes server-side copy for the shared
 data between the existing backup and new backup. It saves time and costs on data transfer.
@@ -23,6 +15,16 @@ See [this article](https://medium.com/@valyala/speeding-up-backups-for-big-time-
 See also [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager.html) tool built on top of `vmbackup`. This tool simplifies
 creation of hourly, daily, weekly and monthly backups.
 
+## Supported storage types
+
+`vmbackup` supports the following `-dst` storage types:
+
+* [GCS](https://cloud.google.com/storage/). Example: `gs://<bucket>/<path/to/backup>`
+* [S3](https://aws.amazon.com/s3/). Example: `s3://<bucket>/<path/to/backup>`
+* [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs/). Example: `azblob://<bucket>/<path/to/backup>`
+* Any S3-compatible storage such as [MinIO](https://github.com/minio/minio), [Ceph](https://docs.ceph.com/en/pacific/radosgw/s3/) or [Swift](https://platform.swiftstack.com/docs/admin/middleware/s3_middleware.html). See [these docs](#advanced-usage) for details.
+* Local filesystem. Example: `fs://</absolute/path/to/backup>`. Note that `vmbackup` prevents from storing the backup into the directory pointed by `-storageDataPath` command-line flag, since this directory should be managed solely by VictoriaMetrics or `vmstorage`.
+
 ## Use cases
 
 ### Regular backups
@@ -30,7 +32,7 @@ creation of hourly, daily, weekly and monthly backups.
 Regular backup can be performed with the following command:
 
 ```console
-vmbackup -storageDataPath=</path/to/victoria-metrics-data> -snapshot.createURL=http://localhost:8428/snapshot/create -dst=gs://<bucket>/<path/to/new/backup>
+./vmbackup -storageDataPath=</path/to/victoria-metrics-data> -snapshot.createURL=http://localhost:8428/snapshot/create -dst=gs://<bucket>/<path/to/new/backup>
 ```
 
 * `</path/to/victoria-metrics-data>` - path to VictoriaMetrics data pointed by `-storageDataPath` command-line flag in single-node VictoriaMetrics or in cluster `vmstorage`.
@@ -75,7 +77,7 @@ The command will upload only changed data to `gs://<bucket>/latest`.
 * Run the following command once a day:
 
 ```console
-vmbackup -storageDataPath=</path/to/victoria-metrics-data> -snapshot.createURL=http://localhost:8428/snapshot/create -dst=gs://<bucket>/<YYYYMMDD> -origin=gs://<bucket>/latest
+./vmbackup -storageDataPath=</path/to/victoria-metrics-data> -snapshot.createURL=http://localhost:8428/snapshot/create -dst=gs://<bucket>/<YYYYMMDD> -origin=gs://<bucket>/latest
 ```
 
 Where `<daily-snapshot>` is the snapshot for the last day `<YYYYMMDD>`.
@@ -152,6 +154,11 @@ See [this article](https://medium.com/@valyala/speeding-up-backups-for-big-time-
            "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/service-account-email"
     }
     ```
+* Obtaining credentials from env variables.
+    - For AWS S3 compatible storages set env variable `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.
+      Also you can set env variable `AWS_SHARED_CREDENTIALS_FILE` with path to credentials file.
+    - For GCE cloud storage set env variable `GOOGLE_APPLICATION_CREDENTIALS` with path to credentials file.
+    - For Azure storage either set env variables `AZURE_STORAGE_ACCOUNT_NAME` and `AZURE_STORAGE_ACCOUNT_KEY`, or `AZURE_STORAGE_ACCOUNT_CONNECTION_STRING`.
 
 * Usage with s3 custom url endpoint. It is possible to use `vmbackup` with s3 compatible storages like minio, cloudian, etc.
   You have to add a custom url endpoint via flag:
@@ -279,7 +286,7 @@ It is recommended using [binary releases](https://github.com/VictoriaMetrics/Vic
 
 ### Development build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.2.
 2. Run `make vmbackup` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `vmbackup` binary and puts it into the `bin` folder.
 

app/vmbackupmanager/README.md

@@ -1,17 +1,23 @@
 ## vmbackupmanager
 
-***vmbackupmanager is a part of [enterprise package](https://victoriametrics.com/products/enterprise/). It is available for download and evaluation at [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)***
+***vmbackupmanager is a part of [enterprise package](https://docs.victoriametrics.com/enterprise.html). It is available for download and evaluation at [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)***
 
-The VictoriaMetrics backup manager automates regular backup procedures. It supports the following backup intervals: **hourly**, **daily**, **weekly** and **monthly**. Multiple backup intervals may be configured simultaneously. I.e. the backup manager creates hourly backups every hour, while it creates daily backups every day, etc. Backup manager must have read access to the storage data, so best practice is to install it on the same machine (or as a sidecar) where the storage node is installed.
-The backup service makes a backup every hour and puts it to the latest folder and then copies data to the folders which represent the backup intervals (hourly, daily, weekly and monthly)
+The VictoriaMetrics backup manager automates regular backup procedures. It supports the following backup intervals: **hourly**, **daily**, **weekly** and **monthly**.
+Multiple backup intervals may be configured simultaneously. I.e. the backup manager creates hourly backups every hour, while it creates daily backups every day, etc.
+Backup manager must have read access to the storage data, so best practice is to install it on the same machine (or as a sidecar) where the storage node is installed.
+The backup service makes a backup every hour and puts it to the latest folder and then copies data to the folders
+which represent the backup intervals (hourly, daily, weekly and monthly)
 
 The required flags for running the service are as follows:
 
-* -eula - should be true and means that you have the legal right to run a backup manager. That can either be a signed contract or an email with confirmation to run the service in a trial period
-* -storageDataPath - path to VictoriaMetrics or vmstorage data path to make backup from
+* -eula - should be true and means that you have the legal right to run a backup manager. That can either be a signed contract or an email
+  with confirmation to run the service in a trial period.
+* -storageDataPath - path to VictoriaMetrics or vmstorage data path to make backup from.
 * -snapshot.createURL - VictoriaMetrics creates snapshot URL which will automatically be created during backup. Example: <http://victoriametrics:8428/snapshot/create>
-* -dst - backup destination at s3, gcs or local filesystem
-* -credsFilePath - path to file with GCS or S3 credentials. Credentials are loaded from default locations if not set. See [https://cloud.google.com/iam/docs/creating-managing-service-account-keys](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) and [https://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html](https://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html)
+* -dst - backup destination at [the supported storage types](https://docs.victoriametrics.com/vmbackup.html#supported-storage-types).
+* -credsFilePath - path to file with GCS or S3 credentials. Credentials are loaded from default locations if not set.
+  See [https://cloud.google.com/iam/docs/creating-managing-service-account-keys](https://cloud.google.com/iam/docs/creating-managing-service-account-keys)
+  and [https://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html](https://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html).
 
 Backup schedule is controlled by the following flags:
 
@@ -36,7 +42,11 @@ To get the full list of supported flags please run the following command:
 ./vmbackupmanager --help
 ```
 
-The service creates a **full** backup each run. This means that the system can be restored fully from any particular backup using vmrestore. Backup manager uploads only the data that has been changed or created since the most recent backup (incremental backup).
+The service creates a **full** backup each run. This means that the system can be restored fully
+from any particular backup using [vmrestore](https://docs.victoriametrics.com/vmrestore.html).
+Backup manager uploads only the data that has been changed or created since the most recent backup (incremental backup).
+This reduces the consumed network traffic and the time needed for performing the backup.
+See [this article](https://medium.com/@valyala/speeding-up-backups-for-big-time-series-databases-533c1a927883) for details.
 
 *Please take into account that the first backup upload could take a significant amount of time as it needs to upload all of the data.*
 
@@ -47,7 +57,7 @@ There are two flags which could help with performance tuning:
 
 ## Example of Usage
 
-GCS and cluster version. You need to have a credentials file in json format with following structure
+GCS and cluster version. You need to have a credentials file in json format with following structure:
 
 credentials.json
 
@@ -141,6 +151,142 @@ The result on the GCS bucket. We see only 3 daily backups:
 ![daily](vmbackupmanager_rp_daily_2.png)
 
 
+## API methods
+
+`vmbackupmanager` exposes the following API methods:
+
+* GET `/api/v1/backups` - returns list of backups in remote storage.
+  Example output:
+  ```json
+  ["daily/2022-10-06","daily/2022-10-10","hourly/2022-10-04:13","hourly/2022-10-06:12","hourly/2022-10-06:13","hourly/2022-10-10:14","hourly/2022-10-10:16","monthly/2022-10","weekly/2022-40","weekly/2022-41"]
+  ```
+
+* POST `/api/v1/restore` - saves backup name to restore when [performing restore](#restore-commands).
+  Example request body:
+  ```json
+  {"backup":"daily/2022-10-06"}
+  ```
+
+* GET `/api/v1/restore` - returns backup name from restore mark if it exists.
+  Example response:
+  ```json
+  {"backup":"daily/2022-10-06"}
+  ```
+
+* DELETE `/api/v1/restore` - delete restore mark.
+
+## CLI
+
+`vmbackupmanager` exposes CLI commands to work with [API methods](#api-methods) without external dependencies.
+
+Supported commands:
+```console
+vmbackupmanager backup 
+
+  vmbackupmanager backup list 
+    List backups in remote storage
+
+vmbackupmanager restore 
+  Restore backup specified by restore mark if it exists
+
+  vmbackupmanager restore get 
+    Get restore mark if it exists
+
+  vmbackupmanager restore delete 
+    Delete restore mark if it exists
+
+  vmbackupmanager restore create [backup_name]
+    Create restore mark
+```
+
+By default, CLI commands are using `http://127.0.0.1:8300` endpoint to reach `vmbackupmanager` API.
+It can be changed by using flag:
+```
+-apiURL string
+      vmbackupmanager address to perform API requests (default "http://127.0.0.1:8300")
+```
+
+### Backup commands
+
+`vmbackupmanager backup list` lists backups in remote storage:
+```console
+$ ./vmbackupmanager backup list
+["daily/2022-10-06","daily/2022-10-10","hourly/2022-10-04:13","hourly/2022-10-06:12","hourly/2022-10-06:13","hourly/2022-10-10:14","hourly/2022-10-10:16","monthly/2022-10","weekly/2022-40","weekly/2022-41"]
+```
+
+### Restore commands
+
+Restore commands are used to create, get and delete restore mark.
+Restore mark is used by `vmbackupmanager` to store backup name to restore when running restore.
+
+
+Create restore mark:
+```console
+$ ./vmbackupmanager restore create daily/2022-10-06
+```
+
+Get restore mark if it exists:
+```console
+$ ./vmbackupmanager restore get
+{"backup":"daily/2022-10-06"}
+```
+
+Delete restore mark if it exists:
+```console
+$ ./vmbackupmanager restore delete
+```
+
+Perform restore:
+```console
+$ /vmbackupmanager-prod restore -dst=gs://vmstorage-data/$NODE_IP -credsFilePath=credentials.json -storageDataPath=/vmstorage-data
+```
+Note that `vmsingle` or `vmstorage` should be stopped before performing restore.
+
+If restore mark doesn't exist at `storageDataPath`(restore wasn't requested) `vmbackupmanager restore` will exit with successful status code.
+
+### How to restore backup via CLI
+
+1. Run `vmbackupmanager backup list` to get list of available backups:
+  ```console
+  $ /vmbackupmanager-prod backup list
+  ["daily/2022-10-06","daily/2022-10-10","hourly/2022-10-04:13","hourly/2022-10-06:12","hourly/2022-10-06:13","hourly/2022-10-10:14","hourly/2022-10-10:16","monthly/2022-10","weekly/2022-40","weekly/2022-41"]
+  ```
+2. Run `vmbackupmanager restore create` to create restore mark:
+   - Use relative path to backup to restore from currently used remote storage:
+     ```console
+     $ /vmbackupmanager-prod restore create daily/2022-10-06
+     ```
+   - Use full path to backup to restore from any remote storage:
+     ```console
+     $ /vmbackupmanager-prod restore create azblob://test1/vmbackupmanager/daily/2022-10-06
+     ```
+3. Stop `vmstorage` or `vmsingle` node
+4. Run `vmbackupmanager restore` to restore backup:
+  ```console
+  $ /vmbackupmanager-prod restore -credsFilePath=credentials.json -storageDataPath=/vmstorage-data
+  ```
+5. Start `vmstorage` or `vmsingle` node
+
+
+### How to restore in Kubernetes
+
+1. Enter container running `vmbackupmanager`
+2. Use `vmbackupmanager backup list` to get list of available backups:
+  ```console
+  $ /vmbackupmanager-prod backup list
+  ["daily/2022-10-06","daily/2022-10-10","hourly/2022-10-04:13","hourly/2022-10-06:12","hourly/2022-10-06:13","hourly/2022-10-10:14","hourly/2022-10-10:16","monthly/2022-10","weekly/2022-40","weekly/2022-41"]
+  ```
+3. Use `vmbackupmanager restore create` to create restore mark:
+  - Use relative path to backup to restore from currently used remote storage:
+    ```console
+    $ /vmbackupmanager-prod restore create daily/2022-10-06
+    ```
+  - Use full path to backup to restore from any remote storage:
+    ```console
+    $ /vmbackupmanager-prod restore create azblob://test1/vmbackupmanager/daily/2022-10-06
+    ```
+4. Restart pod
+
 ## Configuration
 
 ### Flags
@@ -246,6 +392,8 @@ vmbackupmanager performs regular backups according to the provided configs.
   -pushmetrics.url array
      Optional URL to push metrics exposed at /metrics page. See https://docs.victoriametrics.com/#push-metrics . By default metrics exposed at /metrics page aren't pushed to any remote storage
      Supports an array of values separated by comma or specified via multiple flags.
+  -restoreOnStart
+     Check if backup restore was requested and restore requested backup.
   -runOnStart
      Upload backups immediately after start of the service. Otherwise the backup starts on new hour
   -s3ForcePathStyle

app/vmctl/README.md

@@ -700,7 +700,7 @@ It is recommended using [binary releases](https://github.com/VictoriaMetrics/Vic
 
 ### Development build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.2.
 2. Run `make vmctl` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `vmctl` binary and puts it into the `bin` folder.
 
@@ -729,7 +729,7 @@ ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://b
 
 #### Development ARM build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.2.
 2. Run `make vmctl-linux-arm` or `make vmctl-linux-arm64` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `vmctl-linux-arm` or `vmctl-linux-arm64` binary respectively and puts it into the `bin` folder.
 

app/vmgateway/README.md

@@ -1,6 +1,6 @@
 # vmgateway
 
-***vmgateway is a part of [enterprise package](https://victoriametrics.com/products/enterprise/). It is available for download and evaluation at [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)***
+***vmgateway is a part of [enterprise package](https://docs.victoriametrics.com/enterprise.html). It is available for download and evaluation at [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)***
 
 <img alt="vmgateway" src="vmgateway-overview.jpeg">
 
@@ -13,7 +13,7 @@
   * Provides access by tenantID in the Cluster version
   * Allows for separate write/read/admin access to data
 
-`vmgateway` is included in our [enterprise packages](https://victoriametrics.com/products/enterprise/).
+`vmgateway` is included in our [enterprise packages](https://docs.victoriametrics.com/enterprise.html).
 
 ## Access Control
 

app/vminsert/relabel/relabel.go

@@ -123,7 +123,8 @@ func (ctx *Ctx) ApplyRelabeling(labels []prompb.Label) []prompb.Label {
 
 	if pcs.Len() > 0 {
 		// Apply relabeling
-		tmpLabels = pcs.Apply(tmpLabels, 0, true)
+		tmpLabels = pcs.Apply(tmpLabels, 0)
+		tmpLabels = promrelabel.FinalizeLabels(tmpLabels[:0], tmpLabels)
 		if len(tmpLabels) == 0 {
 			metricsDropped.Inc()
 		}

app/vmrestore/README.md

@@ -12,7 +12,7 @@ VictoriaMetrics must be stopped during the restore process.
 Run the following command to restore backup from the given `-src` into the given `-storageDataPath`:
 
 ```console
-vmrestore -src=<storageType>://<path/to/backup> -storageDataPath=<local/path/to/restore>
+./vmrestore -src=<storageType>://<path/to/backup> -storageDataPath=<local/path/to/restore>
 ```
 
 * `<storageType>://<path/to/backup>` is the path to backup made with [vmbackup](https://docs.victoriametrics.com/vmbackup.html).
@@ -186,7 +186,7 @@ It is recommended using [binary releases](https://github.com/VictoriaMetrics/Vic
 
 ### Development build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.2.
 2. Run `make vmrestore` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `vmrestore` binary and puts it into the `bin` folder.
 

app/vmselect/main.go

@@ -29,7 +29,8 @@ import (
 var (
 	deleteAuthKey         = flag.String("deleteAuthKey", "", "authKey for metrics' deletion via /api/v1/admin/tsdb/delete_series and /tags/delSeries")
 	maxConcurrentRequests = flag.Int("search.maxConcurrentRequests", getDefaultMaxConcurrentRequests(), "The maximum number of concurrent search requests. "+
-		"It shouldn't be high, since a single request can saturate all the CPU cores. See also -search.maxQueueDuration")
+		"It shouldn't be high, since a single request can saturate all the CPU cores, while many concurrently executed requests may require high amounts of memory. "+
+		"See also -search.maxQueueDuration and -search.maxMemoryPerQuery")
 	maxQueueDuration = flag.Duration("search.maxQueueDuration", 10*time.Second, "The maximum time the request waits for execution when -search.maxConcurrentRequests "+
 		"limit is reached; see also -search.maxQueryDuration")
 	resetCacheAuthKey    = flag.String("search.resetCacheAuthKey", "", "Optional authKey for resetting rollup cache via /internal/resetRollupResultCache call")

app/vmselect/promql/eval.go

@@ -15,6 +15,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/cgroup"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/decimal"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/memory"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/querytracer"
@@ -27,7 +28,11 @@ var (
 	disableCache                   = flag.Bool("search.disableCache", false, "Whether to disable response caching. This may be useful during data backfilling")
 	maxPointsSubqueryPerTimeseries = flag.Int("search.maxPointsSubqueryPerTimeseries", 100e3, "The maximum number of points per series, which can be generated by subquery. "+
 		"See https://valyala.medium.com/prometheus-subqueries-in-victoriametrics-9b1492b720b3")
-	noStaleMarkers = flag.Bool("search.noStaleMarkers", false, "Set this flag to true if the database doesn't contain Prometheus stale markers, so there is no need in spending additional CPU time on its handling. Staleness markers may exist only in data obtained from Prometheus scrape targets")
+	maxMemoryPerQuery = flagutil.NewBytes("search.maxMemoryPerQuery", 0, "The maximum amounts of memory a single query may consume. "+
+		"Queries requiring more memory are rejected. The total memory limit for concurrently executed queries can be estimated "+
+		"as -search.maxMemoryPerQuery multiplied by -search.maxConcurrentRequests")
+	noStaleMarkers = flag.Bool("search.noStaleMarkers", false, "Set this flag to true if the database doesn't contain Prometheus stale markers, "+
+		"so there is no need in spending additional CPU time on its handling. Staleness markers may exist only in data obtained from Prometheus scrape targets")
 )
 
 // The minimum number of points per timeseries for enabling time rounding.
@@ -1051,7 +1056,17 @@ func evalRollupFuncWithMetricExpr(qt *querytracer.Tracer, ec *EvalConfig, funcNa
 		}
 	}
 	rollupPoints := mulNoOverflow(pointsPerTimeseries, int64(timeseriesLen*len(rcs)))
-	rollupMemorySize = mulNoOverflow(rollupPoints, 16)
+	rollupMemorySize = sumNoOverflow(mulNoOverflow(int64(rssLen), 1000), mulNoOverflow(rollupPoints, 16))
+	if maxMemory := int64(maxMemoryPerQuery.N); maxMemory > 0 && rollupMemorySize > maxMemory {
+		rss.Cancel()
+		return nil, &UserReadableError{
+			Err: fmt.Errorf("not enough memory for processing %d data points across %d time series with %d points in each time series "+
+				"according to -search.maxMemoryPerQuery=%d; requested memory: %d bytes; "+
+				"possible solutions are: reducing the number of matching time series; increasing `step` query arg (step=%gs); "+
+				"increasing -search.maxMemoryPerQuery",
+				rollupPoints, timeseriesLen*len(rcs), pointsPerTimeseries, maxMemory, rollupMemorySize, float64(ec.Step)/1e3),
+		}
+	}
 	rml := getRollupMemoryLimiter()
 	if !rml.Get(uint64(rollupMemorySize)) {
 		rss.Cancel()
@@ -1059,8 +1074,8 @@ func evalRollupFuncWithMetricExpr(qt *querytracer.Tracer, ec *EvalConfig, funcNa
 			Err: fmt.Errorf("not enough memory for processing %d data points across %d time series with %d points in each time series; "+
 				"total available memory for concurrent requests: %d bytes; "+
 				"requested memory: %d bytes; "+
-				"possible solutions are: reducing the number of matching time series; switching to node with more RAM; "+
-				"increasing -memory.allowedPercent; increasing `step` query arg (%gs)",
+				"possible solutions are: reducing the number of matching time series; increasing `step` query arg (step=%gs); "+
+				"switching to node with more RAM; increasing -memory.allowedPercent",
 				rollupPoints, timeseriesLen*len(rcs), pointsPerTimeseries, rml.MaxSize, uint64(rollupMemorySize), float64(ec.Step)/1e3),
 		}
 	}
@@ -1227,6 +1242,14 @@ func mulNoOverflow(a, b int64) int64 {
 	return a * b
 }
 
+func sumNoOverflow(a, b int64) int64 {
+	if math.MaxInt64-a < b {
+		// Overflow
+		return math.MaxInt64
+	}
+	return a + b
+}
+
 func dropStaleNaNs(funcName string, values []float64, timestamps []int64) ([]float64, []int64) {
 	if *noStaleMarkers || funcName == "default_rollup" || funcName == "stale_samples_over_time" {
 		// Do not drop Prometheus staleness marks (aka stale NaNs) for default_rollup() function,

app/vmselect/promql/exec_test.go

@@ -3789,6 +3789,27 @@ func TestExecSuccess(t *testing.T) {
 		resultExpected := []netstorage.Result{r}
 		f(q, resultExpected)
 	})
+	t.Run(`histogram_quantile(duplicate-le)`, func(t *testing.T) {
+		// See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3225
+		t.Parallel()
+		q := `round(sort(histogram_quantile(0.6,
+			label_set(90, "foo", "bar", "le", "5")
+			or label_set(100, "foo", "bar", "le", "5.0")
+			or label_set(200, "foo", "bar", "le", "6.0")
+			or label_set(300, "foo", "bar", "le", "+Inf")
+		)), 0.1)`
+		r1 := netstorage.Result{
+			MetricName: metricNameExpected,
+			Values:     []float64{4.7, 4.7, 4.7, 4.7, 4.7, 4.7},
+			Timestamps: timestampsExpected,
+		}
+		r1.MetricName.Tags = []storage.Tag{{
+			Key:   []byte("foo"),
+			Value: []byte("bar"),
+		}}
+		resultExpected := []netstorage.Result{r1}
+		f(q, resultExpected)
+	})
 	t.Run(`histogram_quantile(valid)`, func(t *testing.T) {
 		t.Parallel()
 		q := `sort(histogram_quantile(0.6,

app/vmselect/promql/rollup_result_cache.go

@@ -152,6 +152,9 @@ func InitRollupResultCache(cachePath string) {
 	metrics.GetOrCreateGauge(`vm_cache_size_bytes{type="promql/rollupResult"}`, func() float64 {
 		return float64(fcs().BytesSize)
 	})
+	metrics.GetOrCreateGauge(`vm_cache_size_max_bytes{type="promql/rollupResult"}`, func() float64 {
+		return float64(fcs().MaxBytesSize)
+	})
 	metrics.GetOrCreateGauge(`vm_cache_requests_total{type="promql/rollupResult"}`, func() float64 {
 		return float64(fcs().GetCalls)
 	})
@@ -430,8 +433,7 @@ func mustLoadRollupResultCacheKeyPrefix(path string) {
 func mustSaveRollupResultCacheKeyPrefix(path string) {
 	path = path + ".key.prefix"
 	data := encoding.MarshalUint64(nil, rollupResultCacheKeyPrefix)
-	fs.MustRemoveAll(path)
-	if err := fs.WriteFileAtomically(path, data); err != nil {
+	if err := fs.WriteFileAtomically(path, data, true); err != nil {
 		logger.Fatalf("cannot store rollupResult cache key prefix to %q: %s", path, err)
 	}
 }

app/vmselect/promql/transform.go

@@ -665,6 +665,7 @@ func transformHistogramShare(tfa *transformFuncArg) ([]*timeseries, error) {
 		sort.Slice(xss, func(i, j int) bool {
 			return xss[i].le < xss[j].le
 		})
+		xss = mergeSameLE(xss)
 		dst := xss[0].ts
 		var tsLower, tsUpper *timeseries
 		if len(boundsLabel) > 0 {
@@ -945,6 +946,7 @@ func transformHistogramQuantile(tfa *transformFuncArg) ([]*timeseries, error) {
 		sort.Slice(xss, func(i, j int) bool {
 			return xss[i].le < xss[j].le
 		})
+		xss = mergeSameLE(xss)
 		dst := xss[0].ts
 		var tsLower, tsUpper *timeseries
 		if len(boundsLabel) > 0 {
@@ -1012,6 +1014,7 @@ func fixBrokenBuckets(i int, xss []leTimeseries) {
 	if len(xss) < 2 {
 		return
 	}
+	// Fill NaN in upper buckets with the first non-NaN value found in lower buckets.
 	for j := len(xss) - 1; j >= 0; j-- {
 		v := xss[j].ts.Values[i]
 		if !math.IsNaN(v) {
@@ -1023,6 +1026,8 @@ func fixBrokenBuckets(i int, xss []leTimeseries) {
 			break
 		}
 	}
+	// Substitute lower bucket values with upper values if the lower values are NaN
+	// or are bigger than the upper bucket values.
 	vNext := xss[len(xss)-1].ts.Values[i]
 	for j := len(xss) - 2; j >= 0; j-- {
 		v := xss[j].ts.Values[i]
@@ -1034,6 +1039,26 @@ func fixBrokenBuckets(i int, xss []leTimeseries) {
 	}
 }
 
+func mergeSameLE(xss []leTimeseries) []leTimeseries {
+	// Merge buckets with identical le values.
+	// See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3225
+	xsDst := xss[0]
+	dst := xss[:1]
+	for j := 1; j < len(xss); j++ {
+		xs := xss[j]
+		if xs.le != xsDst.le {
+			dst = append(dst, xs)
+			xsDst = xs
+			continue
+		}
+		dstValues := xsDst.ts.Values
+		for k, v := range xs.ts.Values {
+			dstValues[k] += v
+		}
+	}
+	return dst
+}
+
 func transformHour(t time.Time) int {
 	return t.Hour()
 }

app/vmselect/vmui/asset-manifest.json

@@ -1,12 +1,12 @@
 {
   "files": {
-    "main.css": "./static/css/main.ba692000.css",
-    "main.js": "./static/js/main.623b88d4.js",
+    "main.css": "./static/css/main.07bcc4ad.css",
+    "main.js": "./static/js/main.1293a99e.js",
     "static/js/27.939f971b.chunk.js": "./static/js/27.939f971b.chunk.js",
     "index.html": "./index.html"
   },
   "entrypoints": [
-    "static/css/main.ba692000.css",
-    "static/js/main.623b88d4.js"
+    "static/css/main.07bcc4ad.css",
+    "static/js/main.1293a99e.js"
   ]
 }

app/vmselect/vmui/index.html

@@ -1 +1 @@
-<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="VM-UI is a metric explorer for Victoria Metrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,500,700&display=swap"/><script src="./dashboards/index.js" type="module"></script><script defer="defer" src="./static/js/main.623b88d4.js"></script><link href="./static/css/main.ba692000.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>
+<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="VM-UI is a metric explorer for Victoria Metrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><link rel="preconnect" href="https://fonts.googleapis.com"><link rel="preconnect" href="https://fonts.gstatic.com" crossorigin><link href="https://fonts.googleapis.com/css2?family=Lato:ital,wght@0,300;0,400;0,700;1,300;1,400;1,700&display=swap" rel="stylesheet"><script src="./dashboards/index.js" type="module"></script><script defer="defer" src="./static/js/main.1293a99e.js"></script><link href="./static/css/main.07bcc4ad.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>

app/vmselect/vmui/static/css/main.07bcc4ad.css

@@ -0,0 +1 @@
+body{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;font-family:Lato,sans-serif}code{font-family:source-code-pro,Menlo,Monaco,Consolas,Courier New,monospace}.MuiAccordionSummary-content{margin:0!important}.shortcut-key{align-items:center;border:1px solid #dedede;border-radius:4px;cursor:default;display:inline-flex;font-size:10px;justify-content:center;line-height:22px;padding:2px 6px 0;text-align:center;white-space:nowrap}input[type=number]::-webkit-inner-spin-button,input[type=number]::-webkit-outer-spin-button{-webkit-appearance:none;margin:0}.uplot,.uplot *,.uplot :after,.uplot :before{box-sizing:border-box}.uplot{font-family:system-ui,-apple-system,Segoe UI,Roboto,Helvetica Neue,Arial,Noto Sans,sans-serif,Apple Color Emoji,Segoe UI Emoji,Segoe UI Symbol,Noto Color Emoji;line-height:1.5;width:-webkit-min-content;width:min-content}.u-title{font-size:18px;font-weight:700;text-align:center}.u-wrap{position:relative;-webkit-user-select:none;-ms-user-select:none;user-select:none}.u-over,.u-under{position:absolute}.u-under{overflow:hidden}.uplot canvas{display:block;height:100%;position:relative;width:100%}.u-axis{position:absolute}.u-legend{font-size:14px;margin:auto;text-align:center}.u-inline{display:block}.u-inline *{display:inline-block}.u-inline tr{margin-right:16px}.u-legend th{font-weight:600}.u-legend th>*{display:inline-block;vertical-align:middle}.u-legend .u-marker{background-clip:padding-box!important;height:1em;margin-right:4px;width:1em}.u-inline.u-live th:after{content:":";vertical-align:middle}.u-inline:not(.u-live) .u-value{display:none}.u-series>*{padding:4px}.u-series th{cursor:pointer}.u-legend .u-off>*{opacity:.3}.u-select{background:rgba(0,0,0,.07)}.u-cursor-x,.u-cursor-y,.u-select{pointer-events:none;position:absolute}.u-cursor-x,.u-cursor-y{left:0;top:0;will-change:transform;z-index:100}.u-hz .u-cursor-x,.u-vt .u-cursor-y{border-right:1px dashed #607d8b;height:100%}.u-hz .u-cursor-y,.u-vt .u-cursor-x{border-bottom:1px dashed #607d8b;width:100%}.u-cursor-pt{background-clip:padding-box!important;border:0 solid;border-radius:50%;left:0;pointer-events:none;position:absolute;top:0;will-change:transform;z-index:100}.u-axis.u-off,.u-cursor-pt.u-off,.u-cursor-x.u-off,.u-cursor-y.u-off,.u-select.u-off,.u-tooltip{display:none}.u-tooltip{grid-gap:12px;word-wrap:break-word;background:rgba(57,57,57,.9);border-radius:4px;color:#fff;font-family:monospace;font-size:10px;font-weight:700;line-height:1.4em;max-width:300px;padding:8px;pointer-events:none;position:absolute;z-index:100}.u-tooltip-data{align-items:center;display:flex;flex-wrap:wrap;font-size:11px;line-height:150%}.u-tooltip-data__value{font-weight:700;padding:4px}.u-tooltip__info{grid-gap:4px;display:grid}.u-tooltip__marker{height:12px;margin-right:4px;width:12px}.legendWrapper{cursor:default;display:flex;flex-wrap:wrap;margin-top:20px;position:relative}.legendGroup{margin:0 12px 0 0;padding:10px 6px}.legendGroupTitle{align-items:center;border-bottom:1px solid #ecebe6;display:flex;font-size:11px;margin-bottom:5px;padding:0 10px 5px}.legendGroupQuery{font-weight:700;margin-right:4px}.legendGroupLine{margin-right:10px}.legendItem{grid-gap:6px;align-items:start;background-color:#fff;cursor:pointer;display:grid;grid-template-columns:auto auto;justify-content:start;padding:7px 50px 7px 10px;transition:.2s ease}.legendItemHide{opacity:.5;text-decoration:line-through}.legendItem:hover{background-color:rgba(0,0,0,.1)}.legendMarker{box-sizing:border-box;height:12px;transition:.2s ease;width:12px}.legendLabel{font-size:11px;font-weight:400;line-height:12px}.legendFreeFields{cursor:pointer;padding:3px}.legendFreeFields:hover{text-decoration:underline}.legendFreeFields:not(:last-child):after{content:","}.panelDescription ul{line-height:2.2}.panelDescription a{color:#fff}.panelDescription code{background-color:rgba(0,0,0,.3);border-radius:2px;color:#fff;display:inline;font-size:inherit;font-weight:400;max-width:100%;padding:4px 6px}

app/vmselect/vmui/static/css/main.ba692000.css

@@ -1 +0,0 @@
-body{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;font-family:-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Oxygen,Ubuntu,Cantarell,Fira Sans,Droid Sans,Helvetica Neue,sans-serif}code{font-family:source-code-pro,Menlo,Monaco,Consolas,Courier New,monospace}.MuiAccordionSummary-content{margin:0!important}.shortcut-key{align-items:center;border:1px solid #dedede;border-radius:4px;cursor:default;display:inline-flex;font-size:10px;justify-content:center;line-height:22px;padding:2px 6px 0;text-align:center;white-space:nowrap}.uplot,.uplot *,.uplot :after,.uplot :before{box-sizing:border-box}.uplot{font-family:system-ui,-apple-system,Segoe UI,Roboto,Helvetica Neue,Arial,Noto Sans,sans-serif,Apple Color Emoji,Segoe UI Emoji,Segoe UI Symbol,Noto Color Emoji;line-height:1.5;width:-webkit-min-content;width:min-content}.u-title{font-size:18px;font-weight:700;text-align:center}.u-wrap{position:relative;-webkit-user-select:none;-ms-user-select:none;user-select:none}.u-over,.u-under{position:absolute}.u-under{overflow:hidden}.uplot canvas{display:block;height:100%;position:relative;width:100%}.u-axis{position:absolute}.u-legend{font-size:14px;margin:auto;text-align:center}.u-inline{display:block}.u-inline *{display:inline-block}.u-inline tr{margin-right:16px}.u-legend th{font-weight:600}.u-legend th>*{display:inline-block;vertical-align:middle}.u-legend .u-marker{background-clip:padding-box!important;height:1em;margin-right:4px;width:1em}.u-inline.u-live th:after{content:":";vertical-align:middle}.u-inline:not(.u-live) .u-value{display:none}.u-series>*{padding:4px}.u-series th{cursor:pointer}.u-legend .u-off>*{opacity:.3}.u-select{background:rgba(0,0,0,.07)}.u-cursor-x,.u-cursor-y,.u-select{pointer-events:none;position:absolute}.u-cursor-x,.u-cursor-y{left:0;top:0;will-change:transform;z-index:100}.u-hz .u-cursor-x,.u-vt .u-cursor-y{border-right:1px dashed #607d8b;height:100%}.u-hz .u-cursor-y,.u-vt .u-cursor-x{border-bottom:1px dashed #607d8b;width:100%}.u-cursor-pt{background-clip:padding-box!important;border:0 solid;border-radius:50%;left:0;pointer-events:none;position:absolute;top:0;will-change:transform;z-index:100}.u-axis.u-off,.u-cursor-pt.u-off,.u-cursor-x.u-off,.u-cursor-y.u-off,.u-select.u-off,.u-tooltip{display:none}.u-tooltip{grid-gap:12px;word-wrap:break-word;background:rgba(57,57,57,.9);border-radius:4px;color:#fff;font-family:monospace;font-size:10px;font-weight:500;line-height:1.4em;max-width:300px;padding:8px;pointer-events:none;position:absolute;z-index:100}.u-tooltip-data{align-items:center;display:flex;flex-wrap:wrap;font-size:11px;line-height:150%}.u-tooltip-data__value{font-weight:700;padding:4px}.u-tooltip__info{grid-gap:4px;display:grid}.u-tooltip__marker{height:12px;margin-right:4px;width:12px}.legendWrapper{cursor:default;display:flex;flex-wrap:wrap;margin-top:20px;position:relative}.legendGroup{margin:0 12px 0 0;padding:10px 6px}.legendGroupTitle{align-items:center;border-bottom:1px solid #ecebe6;display:flex;font-size:11px;margin-bottom:5px;padding:0 10px 5px}.legendGroupQuery{font-weight:700;margin-right:4px}.legendGroupLine{margin-right:10px}.legendItem{grid-gap:6px;align-items:start;background-color:#fff;cursor:pointer;display:grid;grid-template-columns:auto auto;justify-content:start;padding:7px 50px 7px 10px;transition:.2s ease}.legendItemHide{opacity:.5;text-decoration:line-through}.legendItem:hover{background-color:rgba(0,0,0,.1)}.legendMarker{box-sizing:border-box;height:12px;transition:.2s ease;width:12px}.legendLabel{font-size:11px;font-weight:400;line-height:12px}.legendFreeFields{cursor:pointer;padding:3px}.legendFreeFields:hover{text-decoration:underline}.legendFreeFields:not(:last-child):after{content:","}.panelDescription ul{line-height:2.2}.panelDescription a{color:#fff}.panelDescription code{background-color:rgba(0,0,0,.3);border-radius:2px;color:#fff;display:inline;font-size:inherit;font-weight:400;max-width:100%;padding:4px 6px}

app/vmstorage/main.go

@@ -23,7 +23,7 @@ import (
 )
 
 var (
-	retentionPeriod   = flagutil.NewDuration("retentionPeriod", "1", "Data with timestamps outside the retentionPeriod is automatically deleted")
+	retentionPeriod   = flagutil.NewDuration("retentionPeriod", "1", "Data with timestamps outside the retentionPeriod is automatically deleted. See also -retentionFilter")
 	snapshotAuthKey   = flag.String("snapshotAuthKey", "", "authKey, which must be passed in query string to /snapshot* pages")
 	forceMergeAuthKey = flag.String("forceMergeAuthKey", "", "authKey, which must be passed in query string to /internal/force_merge pages")
 	forceFlushAuthKey = flag.String("forceFlushAuthKey", "", "authKey, which must be passed in query string to /internal/force_flush pages")
@@ -57,10 +57,14 @@ var (
 
 	minFreeDiskSpaceBytes = flagutil.NewBytes("storage.minFreeDiskSpaceBytes", 10e6, "The minimum free disk space at -storageDataPath after which the storage stops accepting new data")
 
-	cacheSizeStorageTSID        = flagutil.NewBytes("storage.cacheSizeStorageTSID", 0, "Overrides max size for storage/tsid cache. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning")
-	cacheSizeIndexDBIndexBlocks = flagutil.NewBytes("storage.cacheSizeIndexDBIndexBlocks", 0, "Overrides max size for indexdb/indexBlocks cache. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning")
-	cacheSizeIndexDBDataBlocks  = flagutil.NewBytes("storage.cacheSizeIndexDBDataBlocks", 0, "Overrides max size for indexdb/dataBlocks cache. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning")
-	cacheSizeIndexDBTagFilters  = flagutil.NewBytes("storage.cacheSizeIndexDBTagFilters", 0, "Overrides max size for indexdb/tagFilters cache. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning")
+	cacheSizeStorageTSID = flagutil.NewBytes("storage.cacheSizeStorageTSID", 0, "Overrides max size for storage/tsid cache. "+
+		"See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning")
+	cacheSizeIndexDBIndexBlocks = flagutil.NewBytes("storage.cacheSizeIndexDBIndexBlocks", 0, "Overrides max size for indexdb/indexBlocks cache. "+
+		"See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning")
+	cacheSizeIndexDBDataBlocks = flagutil.NewBytes("storage.cacheSizeIndexDBDataBlocks", 0, "Overrides max size for indexdb/dataBlocks cache. "+
+		"See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning")
+	cacheSizeIndexDBTagFilters = flagutil.NewBytes("storage.cacheSizeIndexDBTagFilters", 0, "Overrides max size for indexdb/tagFiltersToMetricIDs cache. "+
+		"See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning")
 )
 
 // CheckTimeRange returns true if the given tr is denied for querying.
@@ -100,7 +104,7 @@ func InitWithoutMetrics(resetCacheIfNeeded func(mrs []storage.MetricRow)) {
 	storage.SetRetentionTimezoneOffset(*retentionTimezoneOffset)
 	storage.SetFreeDiskSpaceLimit(minFreeDiskSpaceBytes.N)
 	storage.SetTSIDCacheSize(cacheSizeStorageTSID.N)
-	storage.SetTagFilterCacheSize(cacheSizeIndexDBTagFilters.N)
+	storage.SetTagFiltersCacheSize(cacheSizeIndexDBTagFilters.N)
 	mergeset.SetIndexBlocksCacheSize(cacheSizeIndexDBIndexBlocks.N)
 	mergeset.SetDataBlocksCacheSize(cacheSizeIndexDBDataBlocks.N)
 
@@ -601,19 +605,6 @@ func registerStorageMetrics(strg *storage.Storage) {
 		return float64(m().AddRowsConcurrencyCurrent)
 	})
 
-	metrics.NewGauge(`vm_concurrent_search_tsids_limit_reached_total`, func() float64 {
-		return float64(m().SearchTSIDsConcurrencyLimitReached)
-	})
-	metrics.NewGauge(`vm_concurrent_search_tsids_limit_timeout_total`, func() float64 {
-		return float64(m().SearchTSIDsConcurrencyLimitTimeout)
-	})
-	metrics.NewGauge(`vm_concurrent_search_tsids_capacity`, func() float64 {
-		return float64(m().SearchTSIDsConcurrencyCapacity)
-	})
-	metrics.NewGauge(`vm_concurrent_search_tsids_current`, func() float64 {
-		return float64(m().SearchTSIDsConcurrencyCurrent)
-	})
-
 	metrics.NewGauge(`vm_search_delays_total`, func() float64 {
 		return float64(m().SearchDelays)
 	})
@@ -717,8 +708,8 @@ func registerStorageMetrics(strg *storage.Storage) {
 	metrics.NewGauge(`vm_cache_entries{type="indexdb/indexBlocks"}`, func() float64 {
 		return float64(idbm().IndexBlocksCacheSize)
 	})
-	metrics.NewGauge(`vm_cache_entries{type="indexdb/tagFilters"}`, func() float64 {
-		return float64(idbm().TagFiltersCacheSize)
+	metrics.NewGauge(`vm_cache_entries{type="indexdb/tagFiltersToMetricIDs"}`, func() float64 {
+		return float64(idbm().TagFiltersToMetricIDsCacheSize)
 	})
 	metrics.NewGauge(`vm_cache_entries{type="storage/regexps"}`, func() float64 {
 		return float64(storage.RegexpCacheSize())
@@ -758,8 +749,8 @@ func registerStorageMetrics(strg *storage.Storage) {
 	metrics.NewGauge(`vm_cache_size_bytes{type="storage/next_day_metric_ids"}`, func() float64 {
 		return float64(m().NextDayMetricIDCacheSizeBytes)
 	})
-	metrics.NewGauge(`vm_cache_size_bytes{type="indexdb/tagFilters"}`, func() float64 {
-		return float64(idbm().TagFiltersCacheSizeBytes)
+	metrics.NewGauge(`vm_cache_size_bytes{type="indexdb/tagFiltersToMetricIDs"}`, func() float64 {
+		return float64(idbm().TagFiltersToMetricIDsCacheSizeBytes)
 	})
 	metrics.NewGauge(`vm_cache_size_bytes{type="storage/regexps"}`, func() float64 {
 		return float64(storage.RegexpCacheSizeBytes())
@@ -789,8 +780,8 @@ func registerStorageMetrics(strg *storage.Storage) {
 	metrics.NewGauge(`vm_cache_size_max_bytes{type="indexdb/indexBlocks"}`, func() float64 {
 		return float64(idbm().IndexBlocksCacheSizeMaxBytes)
 	})
-	metrics.NewGauge(`vm_cache_size_max_bytes{type="indexdb/tagFilters"}`, func() float64 {
-		return float64(idbm().TagFiltersCacheSizeMaxBytes)
+	metrics.NewGauge(`vm_cache_size_max_bytes{type="indexdb/tagFiltersToMetricIDs"}`, func() float64 {
+		return float64(idbm().TagFiltersToMetricIDsCacheSizeMaxBytes)
 	})
 	metrics.NewGauge(`vm_cache_size_max_bytes{type="storage/regexps"}`, func() float64 {
 		return float64(storage.RegexpCacheMaxSizeBytes())
@@ -817,8 +808,8 @@ func registerStorageMetrics(strg *storage.Storage) {
 	metrics.NewGauge(`vm_cache_requests_total{type="indexdb/indexBlocks"}`, func() float64 {
 		return float64(idbm().IndexBlocksCacheRequests)
 	})
-	metrics.NewGauge(`vm_cache_requests_total{type="indexdb/tagFilters"}`, func() float64 {
-		return float64(idbm().TagFiltersCacheRequests)
+	metrics.NewGauge(`vm_cache_requests_total{type="indexdb/tagFiltersToMetricIDs"}`, func() float64 {
+		return float64(idbm().TagFiltersToMetricIDsCacheRequests)
 	})
 	metrics.NewGauge(`vm_cache_requests_total{type="storage/regexps"}`, func() float64 {
 		return float64(storage.RegexpCacheRequests())
@@ -845,8 +836,8 @@ func registerStorageMetrics(strg *storage.Storage) {
 	metrics.NewGauge(`vm_cache_misses_total{type="indexdb/indexBlocks"}`, func() float64 {
 		return float64(idbm().IndexBlocksCacheMisses)
 	})
-	metrics.NewGauge(`vm_cache_misses_total{type="indexdb/tagFilters"}`, func() float64 {
-		return float64(idbm().TagFiltersCacheMisses)
+	metrics.NewGauge(`vm_cache_misses_total{type="indexdb/tagFiltersToMetricIDs"}`, func() float64 {
+		return float64(idbm().TagFiltersToMetricIDsCacheMisses)
 	})
 	metrics.NewGauge(`vm_cache_misses_total{type="storage/regexps"}`, func() float64 {
 		return float64(storage.RegexpCacheMisses())

app/vmui/Dockerfile-web

@@ -1,4 +1,4 @@
-FROM golang:1.19.1 as build-web-stage
+FROM golang:1.19.2 as build-web-stage
 COPY build /build
 
 WORKDIR /build

app/vmui/packages/vmui/README.md

@@ -44,3 +44,59 @@ You don’t have to ever use `eject`. The curated feature set is suitable for sm
 You can learn more in the [Create React App documentation](https://facebook.github.io/create-react-app/docs/getting-started).
 
 To learn React, check out the [React documentation](https://reactjs.org/).
+
+---
+
+# VMUI config options
+VMUI can be used to paste into other applications
+
+1. Go to file `index.html`
+2. Find root element `<div id="root"></div>`
+3. Add `data-params` with the options
+
+#### Options (JSON):
+
+| Name                    |    Default     |                                                                           Description |
+|:------------------------|:--------------:|--------------------------------------------------------------------------------------:|
+| serverURL               |  domain name   |                                                          Can't be changed from the UI |
+| inputTenantID           |       -        |                            If the flag is present, the "Tenant ID" field is displayed |
+| headerStyles.background |   `#FFFFFF`    |                                                               Header background color |
+| headerStyles.color      | `primary.main` |                                                                     Header font color |
+| palette.primary         |   `#3F51B5`    |                               used to represent primary interface elements for a user |
+| palette.secondary       |   `#F50057`    |                             used to represent secondary interface elements for a user |
+| palette.error           |   `#FF4141`    |            used to represent interface elements that the user should be made aware of |
+| palette.warning         |   `#FF9800`    |                 used to represent potentially dangerous actions or important messages |
+| palette.success         |   `#4CAF50`    |           used to indicate the successful completion of an action that user triggered |
+| palette.info            |   `#03A9F4`    | used to present information to the user that is neutral and not necessarily important |
+
+#### JSON example:
+```json
+{
+  "serverURL": "http://localhost:8428",
+  "inputTenantID": "true",
+  "headerStyles": {
+    "background": "#fff",
+    "color": "primary.main"
+  },
+  "palette": {
+    "primary": "#538DE8",
+    "secondary": "#F76F8E",
+    "error": "#FD151B",
+    "warning": "#FFB30F",
+    "success": "#7BE622",
+    "info": "#0F5BFF"
+  }
+}
+```
+
+
+#### HTML example:
+```html
+<div id="root" data-params='{"serverURL":"http://localhost:8428","inputTenantID":"true","headerStyles":{"background":"#fff","color":"primary.main"},"palette":{"primary":"#538DE8","secondary":"#F76F8E","error":"#FD151B","warning":"#FFB30F","success":"#7BE622","info":"#0F5BFF"}}'></div>
+```
+
+
+
+
+
+

app/vmui/packages/vmui/public/index.html

@@ -26,7 +26,9 @@
       Learn how to configure a non-root public URL by running `npm run build`.
     -->
     <title>VM UI</title>
-    <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,500,700&display=swap" />
+    <link rel="preconnect" href="https://fonts.googleapis.com">
+    <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
+    <link href="https://fonts.googleapis.com/css2?family=Lato:ital,wght@0,300;0,400;0,700;1,300;1,400;1,700&display=swap" rel="stylesheet">
     <script src="%PUBLIC_URL%/dashboards/index.js" type="module"></script>
   </head>
   <body>

app/vmui/packages/vmui/src/components/CustomPanel/Configurator/Graph/GraphSettings.tsx

@@ -21,10 +21,10 @@ const classes = {
     display: "flex",
     alignItems: "center",
     justifyContent: "space-between",
-    background: "#3F51B5",
+    backgroundColor: "primary.main",
     padding: "6px 6px 6px 12px",
     borderRadius: "4px 4px 0 0",
-    color: "#FFF",
+    color: "primary.contrastText",
   },
   popoverBody: {
     display: "grid",

app/vmui/packages/vmui/src/components/CustomPanel/Configurator/Query/AdditionalSettings.tsx

@@ -5,12 +5,14 @@ import {saveToStorage} from "../../../../utils/storage";
 import {useAppDispatch, useAppState} from "../../../../state/common/StateContext";
 import BasicSwitch from "../../../../theme/switch";
 import StepConfigurator from "./StepConfigurator";
-import {useGraphDispatch, useGraphState} from "../../../../state/graph/GraphStateContext";
+import {useGraphDispatch} from "../../../../state/graph/GraphStateContext";
+import {getAppModeParams} from "../../../../utils/app-mode";
+import TenantsConfiguration from "../Settings/TenantsConfiguration";
 
 const AdditionalSettings: FC = () => {
 
-  const {customStep} = useGraphState();
   const graphDispatch = useGraphDispatch();
+  const {inputTenantID} = getAppModeParams();
 
   const {queryControls: {autocomplete, nocache, isTracingEnabled}, time: {period: {step}}} = useAppState();
   const dispatch = useAppDispatch();
@@ -30,31 +32,30 @@ const AdditionalSettings: FC = () => {
     saveToStorage("QUERY_TRACING", !isTracingEnabled);
   };
 
-  return <Box display="flex" alignItems="center">
+  return <Box display="flex" alignItems="center" flexWrap="wrap" gap={2}>
     <Box>
-      <FormControlLabel label="Autocomplete"
+      <FormControlLabel label="Autocomplete" sx={{m: 0}}
         control={<BasicSwitch checked={autocomplete} onChange={onChangeAutocomplete}/>}
       />
     </Box>
-    <Box ml={2}>
-      <FormControlLabel label="Disable cache"
+    <Box>
+      <FormControlLabel label="Disable cache" sx={{m: 0}}
         control={<BasicSwitch checked={nocache} onChange={onChangeCache}/>}
       />
     </Box>
-    <Box ml={2}>
-      <FormControlLabel label="Trace query"
+    <Box>
+      <FormControlLabel label="Trace query" sx={{m: 0}}
         control={<BasicSwitch checked={isTracingEnabled} onChange={onChangeQueryTracing} />}
       />
     </Box>
-    <Box ml={2} mr={2}>
-      <StepConfigurator defaultStep={step} customStepEnable={customStep.enable}
+    <Box ml={2}>
+      <StepConfigurator defaultStep={step}
         setStep={(value) => {
           graphDispatch({type: "SET_CUSTOM_STEP", payload: value});
         }}
-        toggleEnableStep={() => {
-          graphDispatch({type: "TOGGLE_CUSTOM_STEP"});
-        }}/>
+      />
     </Box>
+    {!!inputTenantID && <Box ml={2}><TenantsConfiguration/></Box>}
   </Box>;
 };
 

app/vmui/packages/vmui/src/components/CustomPanel/Configurator/Query/QueryConfigurator.tsx

@@ -1,4 +1,4 @@
-import React, {FC, useState} from "preact/compat";
+import React, {FC, useState, useEffect} from "preact/compat";
 import Box from "@mui/material/Box";
 import IconButton from "@mui/material/IconButton";
 import Tooltip from "@mui/material/Tooltip";
@@ -11,18 +11,20 @@ import AdditionalSettings from "./AdditionalSettings";
 import {ErrorTypes} from "../../../../types";
 import Button from "@mui/material/Button";
 import Typography from "@mui/material/Typography";
+import usePrevious from "../../../../hooks/usePrevious";
+import {MAX_QUERY_FIELDS} from "../../../../config";
 
 export interface QueryConfiguratorProps {
   error?: ErrorTypes | string;
   queryOptions: string[]
 }
 
-export const MAX_QUERY_FIELDS = 4;
 
 const QueryConfigurator: FC<QueryConfiguratorProps> = ({error, queryOptions}) => {
 
   const {query, queryHistory, queryControls: {autocomplete}} = useAppState();
   const [stateQuery, setStateQuery] = useState(query || []);
+  const prevStateQuery = usePrevious(stateQuery) as (undefined | string[]);
   const dispatch = useAppDispatch();
 
   const updateHistory = () => {
@@ -66,7 +68,14 @@ const QueryConfigurator: FC<QueryConfiguratorProps> = ({error, queryOptions}) =>
       payload: {value: {values, index: newIndexHistory}, queryNumber: indexQuery}
     });
   };
-  return <Box boxShadow="rgba(99, 99, 99, 0.2) 0px 2px 8px 0px;" p={4} pb={2} m={-4} mb={2}>
+
+  useEffect(() => {
+    if (prevStateQuery && (stateQuery.length < prevStateQuery.filter(q => q).length)) {
+      onRunQuery();
+    }
+  }, [stateQuery]);
+
+  return <Box>
     <Box>
       {stateQuery.map((q, i) =>
         <Box key={i} display="grid" gridTemplateColumns="1fr auto" gap="4px" width="100%" position="relative"
@@ -82,11 +91,11 @@ const QueryConfigurator: FC<QueryConfiguratorProps> = ({error, queryOptions}) =>
           </Tooltip>}
         </Box>)}
     </Box>
-    <Box mt={3} display="grid" gridTemplateColumns="1fr auto" alignItems="center">
+    <Box mt={3} display="grid" gridTemplateColumns="1fr auto" alignItems="start" gap={4}>
       <AdditionalSettings/>
-      <Box>
+      <Box display="grid" gridTemplateColumns="repeat(2, auto)" gap={1}>
         {stateQuery.length < MAX_QUERY_FIELDS && (
-          <Button variant="outlined" onClick={onAddQuery} startIcon={<AddIcon/>} sx={{mr: 2}}>
+          <Button variant="outlined" onClick={onAddQuery} startIcon={<AddIcon/>}>
             <Typography lineHeight={"20px"} fontWeight="500">Add Query</Typography>
           </Button>
         )}

app/vmui/packages/vmui/src/components/CustomPanel/Configurator/Query/StepConfigurator.tsx

@@ -1,60 +1,67 @@
-import React, {FC, useEffect, useState} from "preact/compat";
-import {ChangeEvent} from "react";
-import Box from "@mui/material/Box";
-import FormControlLabel from "@mui/material/FormControlLabel";
+import React, {FC, useCallback, useState} from "preact/compat";
+import {ChangeEvent, useEffect} from "react";
 import TextField from "@mui/material/TextField";
-import BasicSwitch from "../../../../theme/switch";
+import debounce from "lodash.debounce";
+import InputAdornment from "@mui/material/InputAdornment";
+import Tooltip from "@mui/material/Tooltip";
+import RestartAltIcon from "@mui/icons-material/RestartAlt";
+import IconButton from "@mui/material/IconButton";
 
 interface StepConfiguratorProps {
   defaultStep?: number,
-  customStepEnable: boolean,
   setStep: (step: number) => void,
-  toggleEnableStep: () => void
 }
 
-const StepConfigurator: FC<StepConfiguratorProps> = ({
-  defaultStep, customStepEnable, setStep, toggleEnableStep
-}) => {
+const StepConfigurator: FC<StepConfiguratorProps> = ({defaultStep, setStep}) => {
 
   const [customStep, setCustomStep] = useState(defaultStep);
   const [error, setError] = useState(false);
 
-  useEffect(() => {
-    setStep(customStep || 1);
-  }, [customStep]);
+  const handleApply = (step: number) => setStep(step || 1);
+  const debouncedHandleApply = useCallback(debounce(handleApply, 700), []);
 
   const onChangeStep = (e: ChangeEvent<HTMLInputElement | HTMLTextAreaElement>) => {
-    if (!customStepEnable) return;
     const value = +e.target.value;
+    if (!value) return;
+    handleSetStep(value);
+  };
+
+  const handleSetStep = (value: number) => {
     if (value > 0) {
       setCustomStep(value);
+      debouncedHandleApply(value);
       setError(false);
     } else {
       setError(true);
     }
   };
 
-  const onChangeEnableStep = () => {
-    setError(false);
-    toggleEnableStep();
-  };
+  useEffect(() => {
+    if (defaultStep) handleSetStep(defaultStep);
+  }, [defaultStep]);
 
-  return <Box display="grid" gridTemplateColumns="auto 120px" alignItems="center">
-    <FormControlLabel
-      control={<BasicSwitch checked={customStepEnable} onChange={onChangeEnableStep}/>}
-      label="Override step value"
-    />
-    <TextField
-      label="Step value"
-      type="number"
-      size="small"
-      variant="outlined"
-      value={customStep}
-      disabled={!customStepEnable}
-      error={error}
-      helperText={error ? "step is out of allowed range" : " "}
-      onChange={onChangeStep}/>
-  </Box>;
+  return <TextField
+    label="Step value"
+    type="number"
+    size="small"
+    variant="outlined"
+    value={customStep}
+    error={error}
+    helperText={error ? "step is out of allowed range" : " "}
+    onChange={onChangeStep}
+    InputProps={{
+      inputProps: {min: 0},
+      endAdornment: (
+        <InputAdornment position="start" sx={{mr: -0.5, cursor: "pointer"}}>
+          <Tooltip title={"Reset step to default"}>
+            <IconButton size={"small"} onClick={() => handleSetStep(defaultStep || 1)}>
+              <RestartAltIcon fontSize={"small"} />
+            </IconButton>
+          </Tooltip>
+        </InputAdornment>
+      ),
+    }}
+  />;
 };
 
-export default StepConfigurator;
+export default StepConfigurator;

app/vmui/packages/vmui/src/components/CustomPanel/Configurator/Settings/GlobalSettings.tsx

@@ -9,7 +9,6 @@ import Typography from "@mui/material/Typography";
 import CloseIcon from "@mui/icons-material/Close";
 import IconButton from "@mui/material/IconButton";
 import {useAppDispatch, useAppState} from "../../../../state/common/StateContext";
-import {getAppModeEnable} from "../../../../utils/app-mode";
 
 const modalStyle = {
   position: "absolute" as const,
@@ -27,13 +26,12 @@ const title = "Setting Server URL";
 
 const GlobalSettings: FC = () => {
 
-  const appModeEnable = getAppModeEnable();
   const {serverUrl} = useAppState();
   const dispatch = useAppDispatch();
   const [changedServerUrl, setChangedServerUrl] = useState(serverUrl);
 
-  const setServer = () => {
-    if (!appModeEnable) dispatch({type: "SET_SERVER", payload: changedServerUrl});
+  const setServer = (url?: string) => {
+    dispatch({type: "SET_SERVER", payload: url || changedServerUrl});
     handleClose();
   };
 
@@ -65,12 +63,12 @@ const GlobalSettings: FC = () => {
             <CloseIcon/>
           </IconButton>
         </Box>
-        <ServerConfigurator setServer={setChangedServerUrl}/>
+        <ServerConfigurator setServer={setChangedServerUrl} onEnter={setServer}/>
         <Box display="grid" gridTemplateColumns="auto auto" gap={1} justifyContent="end" mt={4}>
           <Button variant="outlined" onClick={handleClose}>
             Cancel
           </Button>
-          <Button variant="contained" onClick={setServer}>
+          <Button variant="contained" onClick={() => setServer()}>
             apply
           </Button>
         </Box>
@@ -79,4 +77,4 @@ const GlobalSettings: FC = () => {
   </>;
 };
 
-export default GlobalSettings;
+export default GlobalSettings;

app/vmui/packages/vmui/src/components/CustomPanel/Configurator/Settings/ServerConfigurator.tsx

@@ -1,41 +1,44 @@
-import React, {FC, useEffect, useState} from "preact/compat";
+import React, {FC, useState} from "preact/compat";
 import TextField from "@mui/material/TextField";
-import {useAppDispatch, useAppState} from "../../../../state/common/StateContext";
+import {useAppState} from "../../../../state/common/StateContext";
 import {ErrorTypes} from "../../../../types";
-import {getAppModeEnable, getAppModeParams} from "../../../../utils/app-mode";
-import {ChangeEvent} from "react";
+import {ChangeEvent, KeyboardEvent} from "react";
 
 export interface ServerConfiguratorProps {
   error?: ErrorTypes | string;
   setServer: (url: string) => void
+  onEnter: (url: string) => void
 }
 
-const ServerConfigurator: FC<ServerConfiguratorProps> = ({error, setServer}) => {
-
-  const appModeEnable = getAppModeEnable();
-  const {serverURL: appServerUrl} = getAppModeParams();
+const ServerConfigurator: FC<ServerConfiguratorProps> = ({error, setServer, onEnter}) => {
 
   const {serverUrl} = useAppState();
-  const dispatch = useAppDispatch();
   const [changedServerUrl, setChangedServerUrl] = useState(serverUrl);
 
-  useEffect(() => {
-    if (appModeEnable) {
-      dispatch({type: "SET_SERVER", payload: appServerUrl});
-      setChangedServerUrl(appServerUrl);
-    }
-  }, [appServerUrl]);
-
   const onChangeServer = (e: ChangeEvent<HTMLTextAreaElement | HTMLInputElement>) => {
     const value = e.target.value || "";
     setChangedServerUrl(value);
     setServer(value);
   };
 
-  return  <TextField variant="outlined" fullWidth label="Server URL" value={changedServerUrl || ""} disabled={appModeEnable}
+  const onKeyDown = (e: KeyboardEvent) => {
+    if (e.key === "Enter") {
+      e.preventDefault();
+      onEnter(changedServerUrl);
+    }
+  };
+
+  return <TextField
+    autoFocus
+    fullWidth
+    variant="outlined"
+    label="Server URL"
+    value={changedServerUrl || ""}
     error={error === ErrorTypes.validServer || error === ErrorTypes.emptyServer}
     inputProps={{style: {fontFamily: "Monospace"}}}
-    onChange={onChangeServer}/>;
+    onChange={onChangeServer}
+    onKeyDown={onKeyDown}
+  />;
 };
 
-export default ServerConfigurator;
+export default ServerConfigurator;

app/vmui/packages/vmui/src/components/CustomPanel/Configurator/Settings/TenantsConfiguration.tsx

@@ -0,0 +1,60 @@
+import React, {FC, useState, useEffect, useCallback} from "preact/compat";
+import TextField from "@mui/material/TextField";
+import InputAdornment from "@mui/material/InputAdornment";
+import Tooltip from "@mui/material/Tooltip";
+import InfoIcon from "@mui/icons-material/Info";
+import {useAppDispatch, useAppState} from "../../../../state/common/StateContext";
+import {ChangeEvent} from "react";
+import debounce from "lodash.debounce";
+import {getAppModeParams} from "../../../../utils/app-mode";
+
+const TenantsConfiguration: FC = () => {
+  const {serverURL} = getAppModeParams();
+  const {tenantId: tenantIdState} = useAppState();
+  const dispatch = useAppDispatch();
+
+  const [tenantId, setTenantId] = useState<string | number>(tenantIdState || 0);
+
+  const handleApply = (value: string | number) => {
+    const tenantId = Number(value);
+    dispatch({type: "SET_TENANT_ID", payload: tenantId});
+    if (serverURL) {
+      const updateServerUrl = serverURL.replace(/(\/select\/)([\d]+)(\/prometheus)/gmi, `$1${tenantId}$3`);
+      dispatch({type: "SET_SERVER", payload: updateServerUrl});
+      dispatch({type: "RUN_QUERY"});
+    }
+  };
+
+  const debouncedHandleApply = useCallback(debounce(handleApply, 700), []);
+
+  const handleChange = (e: ChangeEvent<HTMLInputElement | HTMLTextAreaElement>) => {
+    setTenantId(e.target.value);
+    debouncedHandleApply(e.target.value);
+  };
+
+  useEffect(() => {
+    if (tenantId === tenantIdState) return;
+    setTenantId(tenantIdState);
+  }, [tenantIdState]);
+
+  return <TextField
+    label="Tenant ID"
+    type="number"
+    size="small"
+    variant="outlined"
+    value={tenantId}
+    onChange={handleChange}
+    InputProps={{
+      inputProps: {min: 0},
+      startAdornment: (
+        <InputAdornment position="start">
+          <Tooltip title={"Define tenant id if you need request to another storage"}>
+            <InfoIcon fontSize={"small"} />
+          </Tooltip>
+        </InputAdornment>
+      ),
+    }}
+  />;
+};
+
+export default TenantsConfiguration;

app/vmui/packages/vmui/src/components/CustomPanel/Configurator/Time/ExecutionControls.tsx

@@ -11,6 +11,8 @@ import List from "@mui/material/List";
 import ListItem from "@mui/material/ListItem";
 import ListItemText from "@mui/material/ListItemText";
 import {useLocation} from "react-router-dom";
+import {getAppModeEnable} from "../../../../utils/app-mode";
+import Box from "@mui/material/Box";
 
 interface AutoRefreshOption {
   seconds: number
@@ -35,6 +37,7 @@ const delayOptions: AutoRefreshOption[] = [
 export const ExecutionControls: FC = () => {
 
   const dispatch = useAppDispatch();
+  const appModeEnable = getAppModeEnable();
   const {queryControls: {autoRefresh}} = useAppState();
 
   const location = useLocation();
@@ -53,12 +56,16 @@ export const ExecutionControls: FC = () => {
     setAnchorEl(null);
   };
 
+  const handleUpdate = () => {
+    dispatch({type: "RUN_QUERY"});
+  };
+
   useEffect(() => {
     const delay = selectedDelay.seconds;
     let timer: number;
     if (autoRefresh) {
       timer = setInterval(() => {
-        dispatch({type: "RUN_QUERY_TO_NOW"});
+        dispatch({type: "RUN_QUERY"});
       }, delay * 1000) as unknown as number;
     } else {
       setSelectedDelay(delayOptions[0]);
@@ -72,22 +79,33 @@ export const ExecutionControls: FC = () => {
   const open = Boolean(anchorEl);
 
   return <>
-    <Tooltip title="Auto-refresh control">
-      <Button variant="contained" color="primary"
-        sx={{
-          minWidth: "110px",
-          color: "white",
-          border: "1px solid rgba(0, 0, 0, 0.2)",
-          justifyContent: "space-between",
-          boxShadow: "none",
-        }}
-        startIcon={<AutorenewIcon/>}
-        endIcon={<KeyboardArrowDownIcon sx={{transform: open ? "rotate(180deg)" : "none"}}/>}
-        onClick={(e) => setAnchorEl(e.currentTarget)}
-      >
-        {selectedDelay.title}
-      </Button>
-    </Tooltip>
+    <Box sx={{
+      minWidth: "110px",
+      color: "white",
+      border: appModeEnable ? "none" : "1px solid rgba(0, 0, 0, 0.2)",
+      justifyContent: "space-between",
+      boxShadow: "none",
+      borderRadius: "4px",
+      display: "grid",
+      gridTemplateColumns: "auto 1fr"
+    }}>
+      <Tooltip title="Refresh dashboard">
+        <Button variant="contained" color="primary"
+          sx={{color: "white", minWidth: "34px", boxShadow: "none", borderRadius: "3px 0 0 3px", p: "6px 6px"}}
+          startIcon={<AutorenewIcon fontSize={"small"} style={{marginRight: "-8px", marginLeft: "4px"}}/>}
+          onClick={handleUpdate}
+        >
+        </Button>
+      </Tooltip>
+      <Tooltip title="Auto-refresh control">
+        <Button variant="contained" color="primary" sx={{boxShadow: "none", borderRadius: "0 3px 3px 0"}} fullWidth
+          endIcon={<KeyboardArrowDownIcon sx={{transform: open ? "rotate(180deg)" : "none"}}/>}
+          onClick={(e) => setAnchorEl(e.currentTarget)}
+        >
+          {selectedDelay.title}
+        </Button>
+      </Tooltip>
+    </Box>
     <Popper
       open={open}
       anchorEl={anchorEl}
@@ -95,7 +113,7 @@ export const ExecutionControls: FC = () => {
       modifiers={[{name: "offset", options: {offset: [0, 6]}}]}>
       <ClickAwayListener onClickAway={() => setAnchorEl(null)}>
         <Paper elevation={3}>
-          <List style={{minWidth: "110px",maxHeight: "208px", overflow: "auto", padding: "20px 0"}}>
+          <List style={{minWidth: "110px", maxHeight: "208px", overflow: "auto", padding: "20px 0"}}>
             {delayOptions.map(d =>
               <ListItem key={d.seconds} button onClick={() => handleChange(d)}>
                 <ListItemText primary={d.title}/>
@@ -104,4 +122,4 @@ export const ExecutionControls: FC = () => {
         </Paper>
       </ClickAwayListener></Popper>
   </>;
-};
+};

app/vmui/packages/vmui/src/components/CustomPanel/Configurator/Time/TimeSelector.tsx

@@ -15,6 +15,8 @@ import Divider from "@mui/material/Divider";
 import ClickAwayListener from "@mui/material/ClickAwayListener";
 import Tooltip from "@mui/material/Tooltip";
 import AlarmAdd from "@mui/icons-material/AlarmAdd";
+import useMediaQuery from "@mui/material/useMediaQuery";
+import {getAppModeEnable} from "../../../../utils/app-mode";
 
 const formatDate = "YYYY-MM-DD HH:mm:ss";
 
@@ -37,12 +39,15 @@ const classes = {
 
 export const TimeSelector: FC = () => {
 
+  const displayFullDate = useMediaQuery("(min-width: 1120px)");
+
   const [anchorEl, setAnchorEl] = useState<HTMLButtonElement | null>(null);
   const [until, setUntil] = useState<string>();
   const [from, setFrom] = useState<string>();
 
   const {time: {period: {end, start}, relativeTime}} = useAppState();
   const dispatch = useAppDispatch();
+  const appModeEnable = getAppModeEnable();
 
   useEffect(() => {
     setUntil(formatDateForNativeInput(dateFromSeconds(end)));
@@ -68,11 +73,8 @@ export const TimeSelector: FC = () => {
 
   const open = Boolean(anchorEl);
   const setTimeAndClosePicker = () => {
-    if (from) {
-      dispatch({type: "SET_FROM", payload: new Date(from)});
-    }
-    if (until) {
-      dispatch({type: "SET_UNTIL", payload: new Date(until)});
+    if (from && until) {
+      dispatch({type: "SET_PERIOD", payload: {from: new Date(from), to: new Date(until)}});
     }
     setAnchorEl(null);
   };
@@ -96,14 +98,18 @@ export const TimeSelector: FC = () => {
       <Button variant="contained" color="primary"
         sx={{
           color: "white",
-          border: "1px solid rgba(0, 0, 0, 0.2)",
-          boxShadow: "none"
+          border: appModeEnable ? "none" : "1px solid rgba(0, 0, 0, 0.2)",
+          boxShadow: "none",
+          minWidth: "34px",
+          padding: displayFullDate ? "" : "6px 8px",
         }}
-        startIcon={<QueryBuilderIcon/>}
+        startIcon={<QueryBuilderIcon style={displayFullDate ? {} : {marginRight: "-8px", marginLeft: "4px"}}/>}
         onClick={(e) => setAnchorEl(e.currentTarget)}>
-        {relativeTime && relativeTime !== "none"
-          ? relativeTime.replace(/_/g, " ")
-          : `${formatRange.start} - ${formatRange.end}`}
+        {displayFullDate && <span>
+          {relativeTime && relativeTime !== "none"
+            ? relativeTime.replace(/_/g, " ")
+            : `${formatRange.start} - ${formatRange.end}`}
+        </span>}
       </Button>
     </Tooltip>
     <Popper

app/vmui/packages/vmui/src/components/CustomPanel/CustomPanel.tsx

@@ -40,7 +40,7 @@ const CustomPanel: FC = () => {
   };
 
   const {queryOptions} = useFetchQueryOptions();
-  const {isLoading, liveData, graphData, error, traces} = useFetchQuery({
+  const {isLoading, liveData, graphData, error, warning, traces} = useFetchQuery({
     visible: true,
     customStep
   });
@@ -62,7 +62,9 @@ const CustomPanel: FC = () => {
 
   return (
     <Box p={4} display="grid" gridTemplateRows="auto 1fr" style={{minHeight: "calc(100vh - 64px)"}}>
-      <QueryConfigurator error={error} queryOptions={queryOptions}/>
+      <Box boxShadow="rgba(99, 99, 99, 0.2) 0px 2px 8px 0px;" p={4} pb={2} m={-4} mb={2}>
+        <QueryConfigurator error={error} queryOptions={queryOptions}/>
+      </Box>
       <Box height="100%">
         {isLoading && <Spinner isLoading={isLoading} height={"500px"}/>}
         {<Box height={"100%"} bgcolor={"#fff"}>
@@ -83,6 +85,7 @@ const CustomPanel: FC = () => {
             </Box>
           </Box>
           {error && <Alert color="error" severity="error" sx={{whiteSpace: "pre-wrap", mt: 2}}>{error}</Alert>}
+          {warning && <Alert color="warning" severity="warning" sx={{whiteSpace: "pre-wrap", my: 2}}>{warning}</Alert>}
           {graphData && period && (displayType === "chart") && <>
             {isTracingEnabled && <TracingsView
               traces={tracesState}

app/vmui/packages/vmui/src/components/CustomPanel/Views/GraphView.tsx

@@ -4,15 +4,16 @@ import LineChart from "../../LineChart/LineChart";
 import {AlignedData as uPlotData, Series as uPlotSeries} from "uplot";
 import Legend from "../../Legend/Legend";
 import {getHideSeries, getLegendItem, getSeriesItem} from "../../../utils/uplot/series";
-import {getLimitsYAxis, getTimeSeries} from "../../../utils/uplot/axes";
+import {getLimitsYAxis, getMinMaxBuffer, getTimeSeries} from "../../../utils/uplot/axes";
 import {LegendItem} from "../../../utils/uplot/types";
 import {TimeParams} from "../../../types";
-import {AxisRange, CustomStep, YaxisState} from "../../../state/graph/reducer";
+import {AxisRange, YaxisState} from "../../../state/graph/reducer";
+import {getAvgFromArray, getMaxFromArray, getMinFromArray} from "../../../utils/math";
 
 export interface GraphViewProps {
   data?: MetricResult[];
   period: TimeParams;
-  customStep: CustomStep;
+  customStep: number;
   query: string[];
   alias?: string[],
   yaxis: YaxisState;
@@ -20,6 +21,7 @@ export interface GraphViewProps {
   showLegend?: boolean;
   setYaxisLimits: (val: AxisRange) => void
   setPeriod: ({from, to}: {from: Date, to: Date}) => void
+  fullWidth?: boolean
 }
 
 const promValueToNumber = (s: string): number => {
@@ -47,9 +49,10 @@ const GraphView: FC<GraphViewProps> = ({
   showLegend= true,
   setYaxisLimits,
   setPeriod,
-  alias = []
+  alias = [],
+  fullWidth = true
 }) => {
-  const currentStep = useMemo(() => customStep.enable ? customStep.value : period.step || 1, [period.step, customStep]);
+  const currentStep = useMemo(() => customStep || period.step || 1, [period.step, customStep]);
 
   const [dataChart, setDataChart] = useState<uPlotData>([[]]);
   const [series, setSeries] = useState<uPlotSeries[]>([]);
@@ -69,16 +72,13 @@ const GraphView: FC<GraphViewProps> = ({
     const tempTimes: number[] = [];
     const tempValues: {[key: string]: number[]} = {};
     const tempLegend: LegendItem[] = [];
-    const tempSeries: uPlotSeries[] = [];
+    const tempSeries: uPlotSeries[] = [{}];
 
     data?.forEach((d) => {
       const seriesItem = getSeriesItem(d, hideSeries, alias);
       tempSeries.push(seriesItem);
       tempLegend.push(getLegendItem(seriesItem, d.group));
-      let tmpValues = tempValues[d.group];
-      if (!tmpValues) {
-        tmpValues = [];
-      }
+      const tmpValues = tempValues[d.group] || [];
       for (const v of d.values) {
         tempTimes.push(v[0]);
         tmpValues.push(promValueToNumber(v[1]));
@@ -87,14 +87,15 @@ const GraphView: FC<GraphViewProps> = ({
     });
 
     const timeSeries = getTimeSeries(tempTimes, currentStep, period);
-    setDataChart([timeSeries, ...data.map(d => {
+    const timeDataSeries = data.map(d => {
       const results = [];
       const values = d.values;
+      const length = values.length;
       let j = 0;
       for (const t of timeSeries) {
-        while (j < values.length && values[j][0] < t) j++;
+        while (j < length && values[j][0] < t) j++;
         let v = null;
-        if (j < values.length && values[j][0] == t) {
+        if (j < length && values[j][0] == t) {
           v = promValueToNumber(values[j][1]);
           if (!Number.isFinite(v)) {
             // Treat special values as nulls in order to satisfy uPlot.
@@ -104,33 +105,38 @@ const GraphView: FC<GraphViewProps> = ({
         }
         results.push(v);
       }
-      return results;
-    })] as uPlotData);
-    setLimitsYaxis(tempValues);
 
-    const newSeries = [{}, ...tempSeries];
-    if (JSON.stringify(newSeries) !== JSON.stringify(series)) {
-      setSeries(newSeries);
-      setLegend(tempLegend);
-    }
+      // stabilize float numbers
+      const resultAsNumber = results.filter(s => s !== null) as number[];
+      const avg = Math.abs(getAvgFromArray(resultAsNumber));
+      const range = getMinMaxBuffer(getMinFromArray(resultAsNumber), getMaxFromArray(resultAsNumber));
+      const rangeStep = Math.abs(range[1] - range[0]);
+
+      return (avg > rangeStep * 1e10) ? results.map(() => avg) : results;
+    });
+    timeDataSeries.unshift(timeSeries);
+    setLimitsYaxis(tempValues);
+    setDataChart(timeDataSeries as uPlotData);
+    setSeries(tempSeries);
+    setLegend(tempLegend);
   }, [data]);
 
   useEffect(() => {
     const tempLegend: LegendItem[] = [];
-    const tempSeries: uPlotSeries[] = [];
+    const tempSeries: uPlotSeries[] = [{}];
     data?.forEach(d => {
       const seriesItem = getSeriesItem(d, hideSeries, alias);
       tempSeries.push(seriesItem);
       tempLegend.push(getLegendItem(seriesItem, d.group));
     });
-    setSeries([{}, ...tempSeries]);
+    setSeries(tempSeries);
     setLegend(tempLegend);
   }, [hideSeries]);
 
   const containerRef = useRef<HTMLDivElement>(null);
 
   return <>
-    <div style={{width: "100%"}} ref={containerRef}>
+    <div style={{width: fullWidth ? "calc(100vw - 68px)" : "100%"}} ref={containerRef}>
       {containerRef?.current &&
           <LineChart data={dataChart} series={series} metrics={data} period={period} yaxis={yaxis} unit={unit}
             setPeriod={setPeriod} container={containerRef?.current}/>}

app/vmui/packages/vmui/src/components/Header/Header.tsx

@@ -3,7 +3,6 @@ import AppBar from "@mui/material/AppBar";
 import Box from "@mui/material/Box";
 import Link from "@mui/material/Link";
 import Toolbar from "@mui/material/Toolbar";
-import Typography from "@mui/material/Typography";
 import {ExecutionControls} from "../CustomPanel/Configurator/Time/ExecutionControls";
 import Logo from "../common/Logo";
 import {setQueryStringWithoutPageReload} from "../../utils/query-string";
@@ -17,6 +16,7 @@ import DatePicker from "../Main/DatePicker/DatePicker";
 import {useCardinalityState, useCardinalityDispatch} from "../../state/cardinality/CardinalityStateContext";
 import {useEffect} from "react";
 import ShortcutKeys from "../ShortcutKeys/ShortcutKeys";
+import {getAppModeEnable, getAppModeParams} from "../../utils/app-mode";
 
 const classes = {
   logo: {
@@ -25,9 +25,8 @@ const classes = {
     alignItems: "center",
     color: "#fff",
     cursor: "pointer",
-    "&:hover": {
-      textDecoration: "underline"
-    }
+    width: "100%",
+    marginBottom: "2px"
   },
   issueLink: {
     textAlign: "center",
@@ -36,6 +35,7 @@ const classes = {
     color: "inherit",
     textDecoration: "underline",
     transition: ".2s opacity",
+    whiteSpace: "nowrap",
     "&:hover": {
       opacity: ".8",
     }
@@ -58,12 +58,18 @@ const classes = {
 
 const Header: FC = () => {
 
+  const appModeEnable = getAppModeEnable();
+  const {headerStyles: {
+    background = appModeEnable ? "#FFF" : "primary.main",
+    color = appModeEnable ? "primary.main" : "#FFF",
+  } = {}} = getAppModeParams();
+
   const {date} = useCardinalityState();
   const cardinalityDispatch = useCardinalityDispatch();
 
   const navigate = useNavigate();
   const {search, pathname} = useLocation();
-  const routes = [
+  const routes = useMemo(() => ([
     {
       label: "Custom panel",
       value: router.home,
@@ -71,6 +77,7 @@ const Header: FC = () => {
     {
       label: "Dashboards",
       value: router.dashboards,
+      hide: appModeEnable
     },
     {
       label: "Cardinality",
@@ -80,7 +87,7 @@ const Header: FC = () => {
       label: "Top queries",
       value: router.topQueries,
     }
-  ];
+  ]), [appModeEnable]);
 
   const [activeMenu, setActiveMenu] = useState(pathname);
 
@@ -102,36 +109,35 @@ const Header: FC = () => {
     setActiveMenu(pathname);
   }, [pathname]);
 
-  return <AppBar position="static" sx={{px: 1, boxShadow: "none"}}>
+  return <AppBar position="static" sx={{px: 1, boxShadow: "none", bgcolor: background, color}}>
     <Toolbar>
-      <Box display="grid" alignItems="center" justifyContent="center">
-        <Box onClick={onClickLogo} sx={classes.logo}>
-          <Logo style={{color: "inherit", marginRight: "6px"}}/>
-          <Typography variant="h5">
-            <span style={{fontWeight: "bolder"}}>VM</span>
-            <span style={{fontWeight: "lighter"}}>UI</span>
-          </Typography>
+      {!appModeEnable && (
+        <Box display="grid" alignItems="center" justifyContent="center">
+          <Box onClick={onClickLogo} sx={classes.logo}>
+            <Logo style={{color: "inherit", width: "100%"}}/>
+          </Box>
+          <Link sx={classes.issueLink} target="_blank"
+            href="https://github.com/VictoriaMetrics/VictoriaMetrics/issues/new">
+            create an issue
+          </Link>
         </Box>
-        <Link sx={classes.issueLink} target="_blank"
-          href="https://github.com/VictoriaMetrics/VictoriaMetrics/issues/new">
-          create an issue
-        </Link>
-      </Box>
-      <Box sx={{ml: 8}}>
-        <Tabs value={activeMenu} textColor="inherit" TabIndicatorProps={{style: {background: "white"}}}
+      )}
+      <Box ml={appModeEnable ? 0 : 8} flexGrow={1}>
+        <Tabs value={activeMenu} textColor="inherit" TabIndicatorProps={{style: {background: color}}}
           onChange={(e, val) => setActiveMenu(val)}>
-          {routes.map(r => (
+          {routes.filter(r => !r.hide).map(r => (
             <Tab
               key={`${r.label}_${r.value}`}
               label={r.label}
               value={r.value}
               component={RouterLink}
               to={`${r.value}${search}`}
+              sx={{color}}
             />
           ))}
         </Tabs>
       </Box>
-      <Box display="flex" gap={1} alignItems="center" ml="auto" mr={0}>
+      <Box display="flex" gap={1} alignItems="center" mr={0} ml={4}>
         {headerSetup?.timeSelector && <TimeSelector/>}
         {headerSetup?.datePicker && (
           <DatePicker
@@ -140,7 +146,7 @@ const Header: FC = () => {
           />
         )}
         {headerSetup?.executionControls && <ExecutionControls/>}
-        {headerSetup?.globalSettings && <GlobalSettings/>}
+        {headerSetup?.globalSettings && !appModeEnable && <GlobalSettings/>}
         <ShortcutKeys/>
       </Box>
     </Toolbar>

app/vmui/packages/vmui/src/components/LineChart/tooltip.css

@@ -9,7 +9,7 @@
     color: #fff;
     font-size: 10px;
     line-height: 1.4em;
-    font-weight: 500;
+    font-weight: bold;
     word-wrap: break-word;
     font-family: monospace;
     pointer-events: none;
@@ -38,4 +38,4 @@
     width: 12px;
     height: 12px;
     margin-right: 4px;
-}
+}

app/vmui/packages/vmui/src/components/Main/DatePicker/DatePicker.tsx

@@ -10,6 +10,7 @@ import Popper from "@mui/material/Popper";
 import ClickAwayListener from "@mui/material/ClickAwayListener";
 import Paper from "@mui/material/Paper";
 import EventIcon from "@mui/icons-material/Event";
+import {getAppModeEnable} from "../../../utils/app-mode";
 
 const formatDate = "YYYY-MM-DD";
 
@@ -20,6 +21,7 @@ interface DatePickerProps {
 
 const DatePicker: FC<DatePickerProps> = ({date, onChange}) => {
 
+  const appModeEnable = getAppModeEnable();
   const dateFormatted = date ? dayjs(date).format(formatDate) : null;
 
   const [anchorEl, setAnchorEl] = useState<HTMLButtonElement | null>(null);
@@ -30,7 +32,7 @@ const DatePicker: FC<DatePickerProps> = ({date, onChange}) => {
       <Button variant="contained" color="primary"
         sx={{
           color: "white",
-          border: "1px solid rgba(0, 0, 0, 0.2)",
+          border: appModeEnable ? "none" : "1px solid rgba(0, 0, 0, 0.2)",
           boxShadow: "none"
         }}
         startIcon={<EventIcon/>}

app/vmui/packages/vmui/src/components/PredefinedPanels/PredefinedPanels.tsx

@@ -12,7 +12,6 @@ import {useFetchQuery} from "../../hooks/useFetchQuery";
 import Spinner from "../common/Spinner";
 import StepConfigurator from "../CustomPanel/Configurator/Query/StepConfigurator";
 import GraphSettings from "../CustomPanel/Configurator/Graph/GraphSettings";
-import {CustomStep} from "../../state/graph/reducer";
 import {marked} from "marked";
 import "./dashboard.css";
 
@@ -36,7 +35,7 @@ const PredefinedPanels: FC<PredefinedPanelsProps> = ({
 
   const containerRef = useRef<HTMLDivElement>(null);
   const [visible, setVisible] = useState(true);
-  const [customStep, setCustomStep] = useState<CustomStep>({enable: false, value: period.step || 1});
+  const [customStep, setCustomStep] = useState<number>(period.step || 1);
   const [yaxis, setYaxis] = useState<YaxisState>({
     limits: {
       enable: false,
@@ -46,7 +45,7 @@ const PredefinedPanels: FC<PredefinedPanelsProps> = ({
 
   const validExpr = useMemo(() => Array.isArray(expr) && expr.every(q => q), [expr]);
 
-  const {isLoading, graphData, error} = useFetchQuery({
+  const {isLoading, graphData, error, warning} = useFetchQuery({
     predefinedQuery: validExpr ? expr : [],
     display: "chart",
     visible,
@@ -106,15 +105,17 @@ const PredefinedPanels: FC<PredefinedPanelsProps> = ({
         {title || ""}
       </Typography>
       <Box mr={2} py={1}>
-        <StepConfigurator defaultStep={period.step} customStepEnable={customStep.enable}
-          setStep={(value) => setCustomStep({...customStep, value: value})}
-          toggleEnableStep={() => setCustomStep({...customStep, enable: !customStep.enable})}/>
+        <StepConfigurator
+          defaultStep={period.step}
+          setStep={(value) => setCustomStep(value)}
+        />
       </Box>
       <GraphSettings yaxis={yaxis} setYaxisLimits={setYaxisLimits} toggleEnableLimits={toggleEnableLimits}/>
     </Box>
     <Box px={2} pb={2}>
       {isLoading && <Spinner isLoading={true} height={"500px"}/>}
       {error && <Alert color="error" severity="error" sx={{whiteSpace: "pre-wrap", mt: 2}}>{error}</Alert>}
+      {warning && <Alert color="warning" severity="warning" sx={{whiteSpace: "pre-wrap", my: 2}}>{warning}</Alert>}
       {graphData && <GraphView
         data={graphData}
         period={period}
@@ -125,7 +126,9 @@ const PredefinedPanels: FC<PredefinedPanelsProps> = ({
         alias={alias}
         showLegend={showLegend}
         setYaxisLimits={setYaxisLimits}
-        setPeriod={setPeriod}/>
+        setPeriod={setPeriod}
+        fullWidth={false}
+      />
       }
     </Box>
   </Box>;

app/vmui/packages/vmui/src/components/ShortcutKeys/ShortcutKeys.tsx

@@ -9,6 +9,7 @@ import KeyboardIcon from "@mui/icons-material/Keyboard";
 import CloseIcon from "@mui/icons-material/Close";
 import Divider from "@mui/material/Divider";
 import {isMacOs} from "../../utils/detect-os";
+import {getAppModeEnable} from "../../utils/app-mode";
 
 const modalStyle = {
   position: "absolute" as const,
@@ -80,13 +81,14 @@ const keyList = [
 
 const ShortcutKeys: FC = () => {
   const [openList, setOpenList] = useState(false);
+  const appModeEnable = getAppModeEnable();
 
   return <>
     <Tooltip title={"Shortcut keys"}>
       <Button variant="contained" color="primary"
         sx={{
           color: "white",
-          border: "1px solid rgba(0, 0, 0, 0.2)",
+          border: appModeEnable ? "none" : "1px solid rgba(0, 0, 0, 0.2)",
           minWidth: "34px",
           padding: "6px 8px",
           boxShadow: "none",

app/vmui/packages/vmui/src/components/Table/TableSettings.tsx

@@ -26,10 +26,10 @@ const classes = {
     display: "flex",
     alignItems: "center",
     justifyContent: "space-between",
-    background: "#3F51B5",
+    backgroundColor: "primary.main",
     padding: "6px 6px 6px 12px",
     borderRadius: "4px 4px 0 0",
-    color: "#FFF",
+    color: "primary.contrastText",
   },
   popoverBody: {
     display: "grid",
@@ -80,7 +80,7 @@ const TableSettings: FC<TableSettingsProps> = ({data, defaultColumns, onChange})
 
   return <Box>
     <Tooltip title={title}>
-      <IconButton onClick={(e) => setAnchorEl(e.currentTarget)}>
+      <IconButton onClick={(e) => setAnchorEl(e.currentTarget)} disabled={!columns.length}>
         <SettingsIcon/>
       </IconButton>
     </Tooltip>

app/vmui/packages/vmui/src/components/common/Logo.tsx

@@ -7,13 +7,15 @@ interface LogoProps {
 }
 
 const Logo: FC<LogoProps> = ({style}) => (
-  <SvgIcon style={style} viewBox="0 0 20 24">
+  <SvgIcon style={style}  viewBox="0 0 74 20">
+    <path fillRule="evenodd" clipRule="evenodd"
+      d="M6.11771 9.47563C6.4774 9.7554 6.91935 9.90875 7.37507 9.9119H7.42685C7.9076 9.90451 8.38836 9.71964 8.67681 9.46823C10.1856 8.18898 14.5568 4.18115 14.5568 4.18115C15.7254 3.09415 12.4637 2.00716 7.42685 1.99977H7.36768C2.33084 2.00716 -0.930893 3.09415 0.237711 4.18115C0.237711 4.18115 4.60888 8.18898 6.11771 9.47563ZM8.67681 11.6422C8.31807 11.9246 7.87603 12.0806 7.41945 12.0859H7.37507C6.91849 12.0806 6.47645 11.9246 6.11771 11.6422C5.08224 10.7549 1.38413 7.41995 0.00103198 6.14809V8.07806C0.00103198 8.2925 0.0823905 8.57349 0.222919 8.70659L0.293358 8.77097L0.293386 8.77099C1.33788 9.72556 4.83907 12.9253 6.11771 14.0159C6.47645 14.2983 6.91849 14.4543 7.37507 14.4595H7.41945C7.9076 14.4447 8.38096 14.2599 8.67681 14.0159C9.98594 12.9067 13.6249 9.57175 14.5642 8.70659C14.7121 8.57349 14.7861 8.2925 14.7861 8.07806V6.14809C12.7662 7.99781 10.7297 9.82926 8.67681 11.6422ZM7.41945 16.6261C7.87517 16.623 8.31712 16.4696 8.67681 16.1898C10.7298 14.3744 12.7663 12.5405 14.7861 10.6883V12.6257C14.7861 12.8327 14.7121 13.1137 14.5642 13.2468C13.6249 14.1194 9.98594 17.4469 8.67681 18.5561C8.38096 18.8075 7.9076 18.9924 7.41945 18.9998H7.37507C6.91935 18.9966 6.4774 18.8433 6.11771 18.5635C4.91431 17.5371 1.74223 14.6362 0.502336 13.5023C0.3934 13.4027 0.299379 13.3167 0.222919 13.2468C0.0823905 13.1137 0.00103198 12.8327 0.00103198 12.6257V10.6883C1.38413 11.9528 5.08224 15.2951 6.11771 16.1825C6.47645 16.4649 6.91849 16.6209 7.37507 16.6261H7.41945Z"
+      fill="currentColor"
+    />
     <path
-      d="M8.27 10.58a2.8 2.8 0 0 0 1.7.59h.07c.65-.01 1.3-.26 1.69-.6 2.04-1.73 7.95-7.15 7.95-7.15C21.26 1.95 16.85.48 10.04.47h-.08C3.15.48-1.26 1.95.32 3.42c0 0 5.91 5.42 7.95 7.16"/>
-    <path
-      d="M11.73 13.51a2.8 2.8 0 0 1-1.7.6h-.06a2.8 2.8 0 0 1-1.7-.6C6.87 12.31 1.87 7.8 0 6.08v2.61c0 .29.11.67.3.85 1.28 1.17 6.2 5.67 7.97 7.18a2.8 2.8 0 0 0 1.7.6h.06c.66-.02 1.3-.27 1.7-.6 1.77-1.5 6.69-6.01 7.96-7.18.2-.18.3-.56.3-.85V6.08a615.27 615.27 0 0 1-8.26 7.43"/>
-    <path
-      d="M11.73 19.66a2.8 2.8 0 0 1-1.7.59h-.06a2.8 2.8 0 0 1-1.7-.6c-1.4-1.2-6.4-5.72-8.27-7.43v2.62c0 .28.11.66.3.84 1.28 1.17 6.2 5.68 7.97 7.19a2.8 2.8 0 0 0 1.7.59h.06c.66-.01 1.3-.26 1.7-.6 1.77-1.5 6.69-6 7.96-7.18.2-.18.3-.56.3-.84v-2.62a614.96 614.96 0 0 1-8.26 7.44"/>
+      d="M35 3.54L29.16 18H26.73L20.89 3.54H23.05C23.2833 3.54 23.4733 3.59667 23.62 3.71C23.7667 3.82333 23.8767 3.97 23.95 4.15L27.36 12.97C27.4733 13.2567 27.58 13.5733 27.68 13.92C27.7867 14.26 27.8867 14.6167 27.98 14.99C28.06 14.6167 28.1467 14.26 28.24 13.92C28.3333 13.5733 28.4367 13.2567 28.55 12.97L31.94 4.15C31.9933 3.99667 32.0967 3.85667 32.25 3.73C32.41 3.60333 32.6033 3.54 32.83 3.54H35ZM52.1767 3.54V18H49.8067V8.66C49.8067 8.28667 49.8267 7.88333 49.8667 7.45L45.4967 15.66C45.2901 16.0533 44.9734 16.25 44.5467 16.25H44.1667C43.7401 16.25 43.4234 16.0533 43.2167 15.66L38.7967 7.42C38.8167 7.64 38.8334 7.85667 38.8467 8.07C38.8601 8.28333 38.8667 8.48 38.8667 8.66V18H36.4967V3.54H38.5267C38.6467 3.54 38.7501 3.54333 38.8367 3.55C38.9234 3.55667 39.0001 3.57333 39.0667 3.6C39.1401 3.62667 39.2034 3.67 39.2567 3.73C39.3167 3.79 39.3734 3.87 39.4267 3.97L43.7567 12C43.8701 12.2133 43.9734 12.4333 44.0667 12.66C44.1667 12.8867 44.2634 13.12 44.3567 13.36C44.4501 13.1133 44.5467 12.8767 44.6467 12.65C44.7467 12.4167 44.8534 12.1933 44.9667 11.98L49.2367 3.97C49.2901 3.87 49.3467 3.79 49.4067 3.73C49.4667 3.67 49.5301 3.62667 49.5967 3.6C49.6701 3.57333 49.7501 3.55667 49.8367 3.55C49.9234 3.54333 50.0267 3.54 50.1467 3.54H52.1767ZM61.063 17.27C61.743 17.27 62.3496 17.1533 62.883 16.92C63.423 16.68 63.8796 16.35 64.253 15.93C64.6263 15.51 64.9096 15.0167 65.103 14.45C65.303 13.8767 65.403 13.26 65.403 12.6V3.85H66.423V12.6C66.423 13.38 66.2996 14.11 66.053 14.79C65.8063 15.4633 65.4496 16.0533 64.983 16.56C64.523 17.06 63.9596 17.4533 63.293 17.74C62.633 18.0267 61.8896 18.17 61.063 18.17C60.2363 18.17 59.4896 18.0267 58.823 17.74C58.163 17.4533 57.5996 17.06 57.133 16.56C56.673 16.0533 56.3196 15.4633 56.073 14.79C55.8263 14.11 55.703 13.38 55.703 12.6V3.85H56.733V12.59C56.733 13.25 56.8296 13.8667 57.023 14.44C57.223 15.0067 57.5063 15.5 57.873 15.92C58.2463 16.34 58.6996 16.67 59.233 16.91C59.773 17.15 60.383 17.27 61.063 17.27ZM71.4442 18H70.4142V3.85H71.4442V18Z"
+      fill="currentColor"
+    />
   </SvgIcon>
 );
 

app/vmui/packages/vmui/src/config.tsx

@@ -0,0 +1,6 @@
+export const MAX_QUERY_FIELDS = 4;
+export const MAX_SERIES = {
+  table: 100,
+  chart: 20,
+  code: Infinity,
+};

app/vmui/packages/vmui/src/hooks/useCardinalityFetch.ts

@@ -2,14 +2,10 @@ import {ErrorTypes} from "../types";
 import {useAppState} from "../state/common/StateContext";
 import {useEffect, useState} from "preact/compat";
 import {CardinalityRequestsParams, getCardinalityInfo} from "../api/tsdb";
-import {getAppModeEnable, getAppModeParams} from "../utils/app-mode";
 import {TSDBStatus} from "../components/CardinalityPanel/types";
 import {useCardinalityState} from "../state/cardinality/CardinalityStateContext";
 import AppConfigurator from "../components/CardinalityPanel/appConfigurator";
 
-const appModeEnable = getAppModeEnable();
-const {serverURL: appServerUrl} = getAppModeParams();
-
 export const useFetchQuery = (): {
   fetchUrl?: string[],
   isLoading: boolean,
@@ -32,12 +28,11 @@ export const useFetchQuery = (): {
   }, [error]);
 
   const fetchCardinalityInfo = async (requestParams: CardinalityRequestsParams) => {
-    const server = appModeEnable ? appServerUrl : serverUrl;
-    if (!server) return;
+    if (!serverUrl) return;
     setError("");
     setIsLoading(true);
     setTSDBStatus(appConfigurator.defaultTSDBStatus);
-    const url = getCardinalityInfo(server, requestParams);
+    const url = getCardinalityInfo(serverUrl, requestParams);
 
     try {
       const response = await fetch(url);

app/vmui/packages/vmui/src/hooks/useFetchQuery.ts

@@ -4,30 +4,25 @@ import {useAppState} from "../state/common/StateContext";
 import {InstantMetricResult, MetricBase, MetricResult} from "../api/types";
 import {isValidHttpUrl} from "../utils/url";
 import {ErrorTypes} from "../types";
-import {getAppModeEnable, getAppModeParams} from "../utils/app-mode";
 import debounce from "lodash.debounce";
 import {DisplayType} from "../components/CustomPanel/Configurator/DisplayTypeSwitch";
-import {CustomStep} from "../state/graph/reducer";
-import usePrevious from "./usePrevious";
-import {arrayEquals} from "../utils/array";
 import Trace from "../components/CustomPanel/Trace/Trace";
+import {MAX_SERIES} from "../config";
 
 interface FetchQueryParams {
   predefinedQuery?: string[]
   visible: boolean
   display?: DisplayType,
-  customStep: CustomStep,
+  customStep: number,
 }
 
-const appModeEnable = getAppModeEnable();
-const {serverURL: appServerUrl} = getAppModeParams();
-
 export const useFetchQuery = ({predefinedQuery, visible, display, customStep}: FetchQueryParams): {
   fetchUrl?: string[],
   isLoading: boolean,
   graphData?: MetricResult[],
   liveData?: InstantMetricResult[],
   error?: ErrorTypes | string,
+  warning?: string,
   traces?: Trace[],
 } => {
   const {query, displayType, serverUrl, time: {period}, queryControls: {nocache, isTracingEnabled}} = useAppState();
@@ -37,6 +32,7 @@ export const useFetchQuery = ({predefinedQuery, visible, display, customStep}: F
   const [liveData, setLiveData] = useState<InstantMetricResult[]>();
   const [traces, setTraces] = useState<Trace[]>();
   const [error, setError] = useState<ErrorTypes | string>();
+  const [warning, setWarning] = useState<string>();
   const [fetchQueue, setFetchQueue] = useState<AbortController[]>([]);
 
   useEffect(() => {
@@ -50,29 +46,42 @@ export const useFetchQuery = ({predefinedQuery, visible, display, customStep}: F
   const fetchData = async (fetchUrl: string[], fetchQueue: AbortController[], displayType: DisplayType, query: string[]) => {
     const controller = new AbortController();
     setFetchQueue([...fetchQueue, controller]);
-    const isDisplayChart = displayType === "chart";
     try {
-      const responses = await Promise.all(fetchUrl.map(url => fetch(url, {signal: controller.signal})));
+      const isDisplayChart = displayType === "chart";
+      const seriesLimit = MAX_SERIES[displayType];
       const tempData: MetricBase[] = [];
       const tempTraces: Trace[] = [];
       let counter = 1;
-      for await (const response of responses) {
+      let totalLength = 0;
+
+      for await (const url of fetchUrl) {
+        const response = await fetch(url, {signal: controller.signal});
         const resp = await response.json();
+
         if (response.ok) {
           setError(undefined);
+
           if (resp.trace) {
-            const trace = new Trace(resp.trace, query[counter-1]);
+            const trace = new Trace(resp.trace, query[counter - 1]);
             tempTraces.push(trace);
           }
-          resp.data.result.forEach((d: MetricBase) => {
+
+          const freeTempSize = seriesLimit - tempData.length;
+          resp.data.result.slice(0, freeTempSize).forEach((d: MetricBase) => {
             d.group = counter;
             tempData.push(d);
           });
+
+          totalLength += resp.data.result.length;
           counter++;
         } else {
           setError(`${resp.errorType}\r\n${resp?.error}`);
         }
       }
+
+      const limitText = `Showing ${seriesLimit} series out of ${totalLength} series due to performance reasons. Please narrow down the query, so it returns less series`;
+      setWarning(totalLength > seriesLimit ? limitText : "");
+
       isDisplayChart ? setGraphData(tempData as MetricResult[]) : setLiveData(tempData as InstantMetricResult[]);
       setTraces(tempTraces);
     } catch (e) {
@@ -83,33 +92,30 @@ export const useFetchQuery = ({predefinedQuery, visible, display, customStep}: F
     setIsLoading(false);
   };
 
-  const throttledFetchData = useCallback(debounce(fetchData, 600), []);
+  const throttledFetchData = useCallback(debounce(fetchData, 800), []);
 
   const fetchUrl = useMemo(() => {
-    const server = appModeEnable ? appServerUrl : serverUrl;
     const expr = predefinedQuery ?? query;
     const displayChart = (display || displayType) === "chart";
     if (!period) return;
-    if (!server) {
+    if (!serverUrl) {
       setError(ErrorTypes.emptyServer);
     } else if (expr.every(q => !q.trim())) {
       setError(ErrorTypes.validQuery);
-    } else if (isValidHttpUrl(server)) {
+    } else if (isValidHttpUrl(serverUrl)) {
       const updatedPeriod = {...period};
-      if (customStep.enable) updatedPeriod.step = customStep.value;
+      updatedPeriod.step = customStep;
       return expr.filter(q => q.trim()).map(q => displayChart
-        ? getQueryRangeUrl(server, q, updatedPeriod, nocache, isTracingEnabled)
-        : getQueryUrl(server, q, updatedPeriod, isTracingEnabled));
+        ? getQueryRangeUrl(serverUrl, q, updatedPeriod, nocache, isTracingEnabled)
+        : getQueryUrl(serverUrl, q, updatedPeriod, isTracingEnabled));
     } else {
       setError(ErrorTypes.validServer);
     }
   },
   [serverUrl, period, displayType, customStep]);
 
-  const prevFetchUrl = usePrevious(fetchUrl);
-
   useEffect(() => {
-    if (!visible || (fetchUrl && prevFetchUrl && arrayEquals(fetchUrl, prevFetchUrl)) || !fetchUrl?.length) return;
+    if (!visible || !fetchUrl?.length) return;
     setIsLoading(true);
     const expr = predefinedQuery ?? query;
     throttledFetchData(fetchUrl, fetchQueue, (display || displayType), expr);
@@ -122,5 +128,5 @@ export const useFetchQuery = ({predefinedQuery, visible, display, customStep}: F
     setFetchQueue(fetchQueue.filter(f => !f.signal.aborted));
   }, [fetchQueue]);
 
-  return {fetchUrl, isLoading, graphData, liveData, error, traces};
+  return {fetchUrl, isLoading, graphData, liveData, error, warning, traces};
 };

app/vmui/packages/vmui/src/hooks/useFetchQueryOptions.ts

@@ -1,10 +1,6 @@
 import {useEffect, useState} from "preact/compat";
 import {getQueryOptions} from "../api/query-range";
 import {useAppState} from "../state/common/StateContext";
-import {getAppModeEnable, getAppModeParams} from "../utils/app-mode";
-
-const appModeEnable = getAppModeEnable();
-const {serverURL: appServerUrl} = getAppModeParams();
 
 export const useFetchQueryOptions = (): {
   queryOptions: string[],
@@ -14,9 +10,8 @@ export const useFetchQueryOptions = (): {
   const [queryOptions, setQueryOptions] = useState([]);
 
   const fetchOptions = async () => {
-    const server = appModeEnable ? appServerUrl : serverUrl;
-    if (!server) return;
-    const url = getQueryOptions(server);
+    if (!serverUrl) return;
+    const url = getQueryOptions(serverUrl);
 
     try {
       const response = await fetch(url);

app/vmui/packages/vmui/src/hooks/useFetchTopQueries.ts

@@ -1,6 +1,5 @@
 import { useEffect, useState } from "react";
 import {ErrorTypes} from "../types";
-import {getAppModeEnable, getAppModeParams} from "../utils/app-mode";
 import {useAppState} from "../state/common/StateContext";
 import {useMemo} from "preact/compat";
 import {getTopQueries} from "../api/top-queries";
@@ -8,8 +7,6 @@ import {TopQueriesData} from "../types";
 import {useTopQueriesState} from "../state/topQueries/TopQueriesStateContext";
 
 export const useFetchTopQueries = () => {
-  const appModeEnable = getAppModeEnable();
-  const {serverURL: appServerUrl} = getAppModeParams();
   const {serverUrl} = useAppState();
   const {topN, maxLifetime, runQuery} = useTopQueriesState();
 
@@ -17,9 +14,7 @@ export const useFetchTopQueries = () => {
   const [loading, setLoading] = useState(false);
   const [error, setError] = useState<ErrorTypes | string>();
 
-  const server = useMemo(() => appModeEnable ? appServerUrl : serverUrl,
-    [appModeEnable, serverUrl, appServerUrl]);
-  const fetchUrl = useMemo(() => getTopQueries(server, topN, maxLifetime), [server, topN, maxLifetime]);
+  const fetchUrl = useMemo(() => getTopQueries(serverUrl, topN, maxLifetime), [serverUrl, topN, maxLifetime]);
 
   const fetchData = async () => {
     setLoading(true);

app/vmui/packages/vmui/src/index.css

@@ -1,7 +1,5 @@
 body {
-  font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', 'Roboto', 'Oxygen',
-    'Ubuntu', 'Cantarell', 'Fira Sans', 'Droid Sans', 'Helvetica Neue',
-    sans-serif;
+  font-family: 'Lato', sans-serif;
   -webkit-font-smoothing: antialiased;
   -moz-osx-font-smoothing: grayscale;
 }
@@ -29,3 +27,9 @@ code {
   border: 1px solid #dedede;
   cursor: default;
 }
+
+input[type=number]::-webkit-inner-spin-button,
+input[type=number]::-webkit-outer-spin-button {
+  -webkit-appearance: none;
+  margin: 0;
+}

app/vmui/packages/vmui/src/state/common/reducer.ts

@@ -31,7 +31,8 @@ export interface AppState {
   displayType: DisplayType;
   query: string[];
   time: TimeState;
-  queryHistory: QueryHistory[],
+  tenantId: number;
+  queryHistory: QueryHistory[];
   queryControls: {
     autoRefresh: boolean;
     autocomplete: boolean;
@@ -51,6 +52,7 @@ export type Action =
     | { type: "SET_UNTIL", payload: Date }
     | { type: "SET_FROM", payload: Date }
     | { type: "SET_PERIOD", payload: TimePeriod }
+    | { type: "SET_TENANT_ID", payload: number }
     | { type: "RUN_QUERY"}
     | { type: "RUN_QUERY_TO_NOW"}
     | { type: "TOGGLE_AUTOREFRESH"}
@@ -72,6 +74,7 @@ export const initialState: AppState = {
   displayType: (displayType?.value || "chart") as DisplayType,
   query: query, // demo_memory_usage_bytes
   queryHistory: query.map(q => ({index: 0, values: [q]})),
+  tenantId: Number(getQueryStringValue("g0.tenantID", 0)),
   time: {
     duration,
     period: getTimeperiodForDuration(duration, endInput),
@@ -174,6 +177,11 @@ export function reducer(state: AppState, action: Action): AppState {
           relativeTime: "none"
         }
       };
+    case "SET_TENANT_ID":
+      return {
+        ...state,
+        tenantId: action.payload
+      };
     case "TOGGLE_AUTOREFRESH":
       return {
         ...state,

app/vmui/packages/vmui/src/state/graph/reducer.ts

@@ -1,3 +1,5 @@
+import {getQueryStringValue} from "../../utils/query-string";
+
 export interface AxisRange {
   [key: string]: [number, number]
 }
@@ -9,24 +11,18 @@ export interface YaxisState {
   }
 }
 
-export interface CustomStep {
-    enable: boolean,
-    value: number
-}
-
 export interface GraphState {
-  customStep: CustomStep
+  customStep: number
   yaxis: YaxisState
 }
 
 export type GraphAction =
   | { type: "TOGGLE_ENABLE_YAXIS_LIMITS" }
   | { type: "SET_YAXIS_LIMITS", payload: AxisRange }
-  | { type: "TOGGLE_CUSTOM_STEP" }
   | { type: "SET_CUSTOM_STEP", payload: number}
 
 export const initialGraphState: GraphState = {
-  customStep: {enable: false, value: 1},
+  customStep: parseFloat(getQueryStringValue("g0.step_input", "0") as string),
   yaxis: {
     limits: {enable: false, range: {"1": [0, 0]}}
   }
@@ -45,21 +41,10 @@ export function reducer(state: GraphState, action: GraphAction): GraphState {
           }
         }
       };
-    case "TOGGLE_CUSTOM_STEP":
-      return {
-        ...state,
-        customStep: {
-          ...state.customStep,
-          enable: !state.customStep.enable
-        }
-      };
     case "SET_CUSTOM_STEP":
       return {
         ...state,
-        customStep: {
-          ...state.customStep,
-          value: action.payload
-        }
+        customStep: action.payload
       };
     case "SET_YAXIS_LIMITS":
       return {

app/vmui/packages/vmui/src/theme/theme.ts

@@ -1,16 +1,28 @@
 import {createTheme} from "@mui/material/styles";
+import {getAppModeParams} from "../utils/app-mode";
+
+const {palette} = getAppModeParams();
 
 const THEME = createTheme({
   palette: {
     primary: {
-      main: "#3F51B5",
+      main: palette?.primary || "#3F51B5",
       light: "#e3f2fd"
     },
     secondary: {
-      main: "#F50057"
+      main: palette?.secondary || "#F50057"
     },
     error: {
-      main: "#FF4141"
+      main: palette?.error || "#FF4141"
+    },
+    warning: {
+      main: palette?.warning || "#ff9800"
+    },
+    info: {
+      main: palette?.info || "#03a9f4"
+    },
+    success: {
+      main: palette?.success || "#4caf50"
     }
   },
   components: {
@@ -104,10 +116,25 @@ const THEME = createTheme({
           boxShadow: "rgba(0, 0, 0, 0.08) 0px 4px 12px"
         }
       }
+    },
+    MuiTableCell: {
+      styleOverrides: {
+        head: {
+          fontWeight: 600
+        }
+      }
+    },
+    MuiTab: {
+      styleOverrides: {
+        root: {
+          fontWeight: 600
+        }
+      }
     }
   },
   typography: {
-    "fontSize": 10
+    "fontSize": 10,
+    fontFamily: "'Lato', sans-serif"
   }
 });
 

app/vmui/packages/vmui/src/utils/app-mode.ts

@@ -1,10 +1,28 @@
 export interface AppParams {
-  serverURL: string
+  serverURL?: string
+  inputTenantID?: boolean
+  headerStyles?: {
+    background?: string
+    color?: string
+  }
+  palette?: {
+    primary?: string
+    secondary?: string
+    error?: string
+    warning?: string
+    info?: string
+    success?: string
+  }
 }
 
 const getAppModeParams = (): AppParams => {
   const dataParams = document.getElementById("root")?.dataset.params || "{}";
-  return JSON.parse(dataParams);
+  try {
+    return JSON.parse(dataParams);
+  } catch (e) {
+    console.error(e);
+    return {};
+  }
 };
 
 const getAppModeEnable = (): boolean => !!Object.keys(getAppModeParams()).length;

app/vmui/packages/vmui/src/utils/default-server-url.ts

@@ -1,3 +1,6 @@
+import {getAppModeParams} from "./app-mode";
+
 export const getDefaultServer = (): string => {
-  return window.location.href.replace(/\/(?:prometheus\/)?(?:graph|vmui)\/.*/, "/prometheus");
+  const {serverURL} = getAppModeParams();
+  return serverURL || window.location.href.replace(/\/(?:prometheus\/)?(?:graph|vmui)\/.*/, "/prometheus");
 };

app/vmui/packages/vmui/src/utils/math.ts

@@ -21,3 +21,5 @@ export const getMinFromArray = (a: number[]) => {
   }
   return Number.isFinite(min) ? min : null;
 };
+
+export const getAvgFromArray = (a: number[]) => a.reduce((a,b) => a+b) / a.length;

app/vmui/packages/vmui/src/utils/query-string.ts

@@ -1,7 +1,7 @@
 import qs from "qs";
 import get from "lodash.get";
 import router from "../router";
-import {MAX_QUERY_FIELDS} from "../components/CustomPanel/Configurator/Query/QueryConfigurator";
+import {MAX_QUERY_FIELDS} from "../config";
 
 const graphStateToUrlParams = {
   "time.duration": "range_input",
@@ -9,6 +9,7 @@ const graphStateToUrlParams = {
   "time.period.step": "step_input",
   "time.relativeTime": "relative_time",
   "displayType": "tab",
+  "tenantId": "tenantID",
 };
 
 const stateToUrlParams = {

app/vmui/packages/vmui/src/utils/uplot/axes.ts

@@ -48,7 +48,7 @@ export const getMinMaxBuffer = (min: number | null, max: number | null): [number
   }
   const valueRange = Math.abs(max - min) || Math.abs(min) || 1;
   const padding = 0.02*valueRange;
-  return [Math.floor(min - padding), Math.ceil(max + padding)];
+  return [min - padding, max + padding];
 };
 
 export const getLimitsYAxis = (values: { [key: string]: number[] }): AxisRange => {

deployment/docker/docker-compose-cluster.yml

@@ -100,7 +100,7 @@ services:
       # display source of alerts in grafana
       - '-external.url=http://127.0.0.1:3000' #grafana outside container
       # when copypaste the line below be aware of '$$' for escaping in '$expr'
-      - '--external.alert.source=explore?orgId=1&left=["now-1h","now","VictoriaMetrics",{"expr":"{{$$expr|quotesEscape|crlfEscape|queryEscape}}"},{"mode":"Metrics"},{"ui":[true,true,true,"none"]}]'
+      - '--external.alert.source=explore?orgId=1&left=["now-1h","now","VictoriaMetrics",{"expr":{{$$expr|jsonEscape|queryEscape}} },{"mode":"Metrics"},{"ui":[true,true,true,"none"]}]'
     restart: always
 
   alertmanager:
@@ -118,4 +118,4 @@ volumes:
   vmagentdata: {}
   strgdata-1: {}
   strgdata-2: {}
-  grafanadata: {}
+  grafanadata: {}

deployment/docker/docker-compose.yml

@@ -75,7 +75,7 @@ services:
       # display source of alerts in grafana
       - "--external.url=http://127.0.0.1:3000" #grafana outside container
       # when copypaste the line be aware of '$$' for escaping in '$expr'
-      - '--external.alert.source=explore?orgId=1&left=["now-1h","now","VictoriaMetrics",{"expr":"{{$$expr|quotesEscape|crlfEscape|queryEscape}}"},{"mode":"Metrics"},{"ui":[true,true,true,"none"]}]'
+      - '--external.alert.source=explore?orgId=1&left=["now-1h","now","VictoriaMetrics",{"expr":{{$$expr|jsonEscape|queryEscape}} },{"mode":"Metrics"},{"ui":[true,true,true,"none"]}]'
     networks:
       - vm_net
     restart: always

docs/BestPractices.md

@@ -66,4 +66,4 @@ There are the following channels for providing technical support for VictoriaMet
 * [Slack channel](https://slack.victoriametrics.com/)
 * [Telegram channel](https://t.me/VictoriaMetrics_en)
 
-We also provide [Enterprise support](https://victoriametrics.com/products/enterprise/).
+We also provide [Enterprise support](https://docs.victoriametrics.com/enterprise.html).

docs/CHANGELOG.md

@@ -16,10 +16,79 @@ The following tip changes can be tested by building VictoriaMetrics components f
 ## tip
 
 
+## [v1.83.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.83.0)
+
+Released at 29-10-2022
+
+**Update note 1:** the `indexdb/tagFilters` cache type at [/metrics](https://docs.victoriametrics.com/#monitoring) has been renamed to `indexdb/tagFiltersToMetricIDs` in order to make its puropose more clear.
+
+**Update note 2:** [vmalert](https://docs.victoriametrics.com/vmalert.html): the `crlfEscape` [template function](https://docs.victoriametrics.com/vmalert.html#template-functions) becames obsolete starting from this release. It can be safely removed from alerting templates, since `\n` chars are properly escaped with other `*Escape` functions now. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3139) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/890) issue for details.
+
+
+* FEATURE: [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise.html): add support for automatic `vmstorage` nodes discovering and updating at `vmselect` and `vminsert`. See [these docs](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#automatic-vmstorage-discovery).
+* FEATURE: [VictoriaMetric enterprise](https://docs.victoriametrics.com/enterprise.html): allow configuring multiple retentions for distinct sets of time series. See [these docs](https://docs.victoriametrics.com/#retention-filters), [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/143) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/289) feature request.
+* FEATURE: [VictoriaMetric cluster enterprise](https://docs.victoriametrics.com/enterprise.html): add support for multiple retentions for distinct tenants - see [these docs](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#retention-filters) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/143) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/289) feature request.
+* FEATURE: allow limiting memory usage on a per-query basis with `-search.maxMemoryPerQuery` command-line flag. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3203).
+* FEATURE: allow referring environment variables inside command-line flags via `%{ENV_VAR}` syntax. For example, if `AUTH_KEY=top-secret` environment variable is set, then `-metricsAuthKey=%{AUTH_KEY}` command-line flag is automatically expanded to `-storageDataPath=top-secret` at VictoriaMetrics startup. See [these docs](https://docs.victoriametrics.com/#environment-variables) for details.
+* FEATURE: allow referring environment variables inside other environment variables via `%{ENV_VAR}` syntax. For example, if `A=a-%{B}`, `B=b-%{C}` and 'C=c` env vars are set, then VictoriaMetrics components automatically expand them to `A=a-b-c`, `B=b-c` and `C=c` on startup.
+* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): drop all the labels with `__` prefix from discovered targets in the same way as Prometheus does according to [this article](https://www.robustperception.io/life-of-a-label/). Previously the following labels were available during [metric-level relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#metric_relabel_configs): `__address__`, `__scheme__`, `__metrics_path__`, `__scrape_interval__`, `__scrape_timeout__`, `__param_*`. Now these labels are available only during [target-level relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config). This should reduce CPU usage and memory usage for `vmagent` setups, which scrape big number of targets.
+* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): improve the performance for metric-level [relabeling](https://docs.victoriametrics.com/vmagent.html#relabeling), which can be applied via `metric_relabel_configs` section at [scrape_configs](https://docs.victoriametrics.com/sd_configs.html#scrape_configs), via `-remoteWrite.relabelConfig` or via `-remoteWrite.urlRelabelConfig` command-line options.
+* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): allow specifying full url in scrape target addresses (aka `__address__` label). This makes valid the following `-promscrape.config`:
+
+  ```yml
+  scrape_configs:
+  - job_name: abc
+    metrics_path: /foo/bar
+    scheme: https
+    static_configs:
+    - targets:
+      # the following targets are scraped by the provided full urls
+      - 'http://host1/metric/path1'
+      - 'https://host2/metric/path2'
+      - 'http://host3:1234/metric/path3?arg1=value1'
+      # the following target is scraped by <scheme>://host4:1234<metrics_path>
+      - host4:1234
+  ```
+
+  See [the corresponding issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3208).
+
+* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): allow controlling [staleness tracking](https://docs.victoriametrics.com/vmagent.html#prometheus-staleness-markers) on a per-[scrape_config](https://docs.victoriametrics.com/sd_configs.html#scrape_configs) basis by specifying `no_stale_markers: true` or `no_stale_markers: false` option in the corresponding [scrape_config](https://docs.victoriametrics.com/sd_configs.html#scrape_configs).
+* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): add `strvalue` and `stripDomain` [template functions](https://docs.victoriametrics.com/vmalert.html#template-functions) in order to improve compatibility with Prometheus.
+* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): add `jsonEscape` and `htmlEscape` [template functions](https://docs.victoriametrics.com/vmalert.html#template-functions).
+* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): limit the number of plotted series. This should prevent from browser crashes or hangs when the query returns big number of time series. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3155).
+* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): reduce memory usage when querying big number of time series. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3240).
+* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add responsive styles for small screens. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3239) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3256).
+* FEATURE: log error if some environment variables referred at `-promscrape.config` via `%{ENV_VAR}` aren't found. This should prevent from silent using incorrect config files.
+* FEATURE: immediately shut down VictoriaMetrics apps on the second SIGINT or SIGTERM signal if they couldn't be finished gracefully for some reason after receiving the first signal.
+* FEATURE: improve the performance of [/api/v1/series](https://docs.victoriametrics.com/url-examples.html#apiv1series) endpoint by eliminating loading of unused `TSID` data during the API call.
+* FEATURE: [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager.html): add functionality for automated restore from backup. See [these docs](https://docs.victoriametrics.com/vmbackupmanager.html#how-to-restore-backup-via-cli).
+
+* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html): properly merge buckets with identical `le` values, but with different string representation of these values when calculating [histogram_quantile](https://docs.victoriametrics.com/MetricsQL.html#histogram_quantile) and [histogram_share](https://docs.victoriametrics.com/MetricsQL.html#histogram_share). For example, `http_request_duration_seconds_bucket{le="5"}` and `http_requests_duration_seconds_bucket{le="5.0"}`. Such buckets may be returned from distinct targets. Thanks to @647-coder for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3225).
+* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert.html): change severity level for log messages about failed attempts for sending data to remote storage from `error` to `warn`. The message for about all failed send attempts remains at `error` severity level.
+* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert.html): fix panic if `vmalert` runs with `-clusterMode` command-line flag in [multitenant mode](https://docs.victoriametrics.com/vmalert.html#multitenancy). The issue has been introduced in [v1.82.0](https://docs.victoriametrics.com/CHANGELOG.html#v1820).
+* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert.html): properly escape string passed to `quotesEscape` [template function](https://docs.victoriametrics.com/vmalert.html#template-functions), so it can be safely embedded into JSON string. This makes obsolete the `crlfEscape` function. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3139) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/890) issue.
+* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): do not show invalid error message in Kubernetes service discovery: `cannot parse WatchEvent json response: EOF`. The invalid error message has been appeared in [v1.82.0](https://docs.victoriametrics.com/CHANGELOG.html#v1820).
+* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): properly add `exported_` prefix to metric labels, which clashing with scrape target labels if `honor_labels: true` option isn't set in [scrape_config](https://docs.victoriametrics.com/sd_configs.html#scrape_configs). Previously some `exported_` prefixes were missing in the resulting metric labels. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3278). The issue has been introduced in [v1.82.0](https://docs.victoriametrics.com/CHANGELOG.html#v1820).
+* BUGFIX: `vmselect`: expose missing metric `vm_cache_size_max_bytes{type="promql/rollupResult"}` . This metric is used for monitoring rollup cache usage with the query `vm_cache_size_bytes{type="promql/rollupResult"} / vm_cache_size_max_bytes{type="promql/rollupResult"}` in the same way as this is done for other cache types.
+
+## [v1.82.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.82.1)
+
+Released at 14-10-2022
+
+* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): automatically update graph, legend and url after the removal of query field. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3169) and [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3196#issuecomment-1269765205).
+* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert.html): remove duplicate `alertname` JSON entry from generated alerts. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3053). Thanks to @Howie59 for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3182)!
+* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert.html): fix integration with Grafana via `-vmalert.proxyURL`, which has been broken in [v1.82.0](https://docs.victoriametrics.com/CHANGELOG.html#v1820). See [this issue](https://github.com/VictoriaMetrics/helm-charts/issues/391).
+* BUGFIX: [vmbackup](https://docs.victoriametrics.com/vmbackup.html): set default region to `us-east-1` if `AWS_REGION` environment variable isn't set. The issue was introduced in [vmbackup v1.82.0](https://docs.victoriametrics.com/CHANGELOG.html#v1820). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3224).
+* BUGFIX: [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager.html): fix deletion of old backups at [Azure blob storage](https://azure.microsoft.com/en-us/products/storage/blobs/).
+* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html): properly apply regex filters when searching for time series. Previously unexpected time series could be returned from regex filter. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3227). The issue was introduced in [v1.82.0](https://docs.victoriametrics.com/CHANGELOG.html#v1820).
+* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmbagent.html): properly apply `if` section with regex filters. Previously unexpected metrics could be returned from `if` section. The issue was introduced in [v1.82.0](https://docs.victoriametrics.com/CHANGELOG.html#v1820).
+
 ## [v1.82.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.82.0)
 
 Released at 07-10-2022
 
+**It isn't recommended to use VictoriaMetrics and vmagent v1.82.0 because of [the bug](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3227), which may result in incorrect query results and [relabeling](https://docs.victoriametrics.com/vmagent.html#relabeling) results. Upgrade to [v1.82.1](https://docs.victoriametrics.com/CHANGELOG.html#v1821) instead.**
+
 **Update note 1:** this release changes data format for [/api/v1/export/native](https://docs.victoriametrics.com/#how-to-export-data-in-native-format) in incompatible way, so it cannot be imported into older version of VictoriaMetrics via [/api/v1/import/native](https://docs.victoriametrics.com/#how-to-import-data-in-native-format).
 
 **Update note 2:** [vmalert](https://docs.victoriametrics.com/vmalert.html) changes default value for command-line flag `-datasource.queryStep` from `0s` to `5m`. The change supposed to improve reliability of the rules evaluation when evaluation interval is lower than scraping interval.
@@ -48,7 +117,7 @@ See [these docs](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#m
 * FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): add experimental feature for displaying last 10 states of the rule (recording or alerting) evaluation. The state is available on the Rule page, which can be opened by clicking on `Details` link next to Rule's name on the `/groups` page.
 * FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): allow using extra labels in annotiations. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3013).
 * FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): allow configuring authorization params per list of targets in vmalert's notifier config for `static_configs`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2690).
-* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): allow using `{{$labels}}` for templating in command-line flag `-external.alert.source`. The change supposed to provide additional flexibility for generating alert's source link based on labels values.
+* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): allow using {% raw %}`{{$labels}}`{% endraw %} for templating in command-line flag `-external.alert.source`. The change supposed to provide additional flexibility for generating alert's source link based on labels values.
 * FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): add `vm_account_id` and `vm_project_id` labels to results of alerting and recording rules if `-clusterMode` is enabled. This improves [multitenant support in vmalert](https://docs.victoriametrics.com/vmalert.html#multitenancy).
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): minimize the time needed for reading large responses from scrape targets in [stream parsing mode](https://docs.victoriametrics.com/vmagent.html#stream-parsing-mode). This should reduce scrape durations for such targets as [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) running in a big Kubernetes cluster.
 * FEATURE: [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html): add [sort_by_label_numeric](https://docs.victoriametrics.com/MetricsQL.html#sort_by_label_numeric) and [sort_by_label_numeric_desc](https://docs.victoriametrics.com/MetricsQL.html#sort_by_label_numeric_desc) functions for [numeric sort](https://www.gnu.org/software/coreutils/manual/html_node/Version-sort-is-not-the-same-as-numeric-sort.html) of input time series by the specified labels. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2938).
@@ -344,7 +413,7 @@ Released at 20-06-2022
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add `-promscrape.suppressScrapeErrorsDelay` command-line flag, which can be used for delaying and aggregating the logging of per-target scrape errors. This may reduce the amounts of logs when `vmagent` scrapes many unreliable targets. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2575). Thanks to @jelmd for [the initial implementation](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2576).
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add `-promscrape.cluster.name` command-line flag, which allows proper data de-duplication when the same target is scraped from multiple [vmagent clusters](https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2679).
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add `action: graphite` relabeling rules optimized for extracting labels from Graphite-style metric names. See [these docs](https://docs.victoriametrics.com/vmagent.html#graphite-relabeling) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2737).
-* FEATURE: [VictoriaMetrics enterprise](https://victoriametrics.com/products/enterprise/): expose `vm_downsampling_partitions_scheduled` and `vm_downsampling_partitions_scheduled_size_bytes` metrics, which can be used for tracking the progress of initial [downsampling](https://docs.victoriametrics.com/#downsampling) for historical data. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2612).
+* FEATURE: [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise.html): expose `vm_downsampling_partitions_scheduled` and `vm_downsampling_partitions_scheduled_size_bytes` metrics, which can be used for tracking the progress of initial [downsampling](https://docs.victoriametrics.com/#downsampling) for historical data. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2612).
 * FEATURE: [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html): do not spend up to 5 seconds when trying to connect to unavailable `vmstorage` nodes. This should improve query latency when some of `vmstorage` nodes aren't available. Expose `vm_tcpdialer_addr_available{addr="..."}` metric at `http://vmselect:8481/metrics` for determining whether the given `addr` is available for establishing new connections. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711#issuecomment-1160363187).
 * FEATURE: [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html): add `-vmstorageDialTimeout` command-line flags to `vmselect` and `vminsert` for tuning the maximum duration for connection estabilishing to `vmstorage` nodes. This should help resolving [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711).
 
@@ -658,7 +727,7 @@ into increase of CPU usage until merges are finished.
 
 We recommend updating in "off-peak" time when load on the VictoriaMetrics is on its minimum.
 
-* FEATURE: [VictoriaMetrics enterprise](https://victoriametrics.com/products/enterprise/): add multi-level downsampling support. See [these docs](https://docs.victoriametrics.com/#downsampling) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/36).
+* FEATURE: [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise.html): add multi-level downsampling support. See [these docs](https://docs.victoriametrics.com/#downsampling) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/36).
 * FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add ability to analyze the correlation between two queries on a single graph. Just click `+Query` button, enter the second query in the newly appeared input field and press `Ctrl+Enter`. Results for both queries should be displayed simultaneously on the same graph. Every query has its own vertical scale, which is displayed on the left and the right side of the graph. Lines for the second query are dashed. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1916).
 * FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add ability to override the interval between returned datapoints. By default it is automatically calculated depending on the selected time range and horizontal resolution of the graph. Now it is possible to override it with custom values. This may be useful during data exploration and debugging.
 * FEATURE: accept optional `extra_filters[]=series_selector` query args at Prometheus query APIs additionally to `extra_label` query args. This allows enforcing additional filters for all the Prometheus query APIs by using [vmgateway](https://docs.victoriametrics.com/vmgateway.html) or [vmauth](https://docs.victoriametrics.com/vmauth.html). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1863).
@@ -700,7 +769,7 @@ Released at 02-12-2021
 * FEATURE: expose `/-/healthy` and `/-/ready` endpoints as Prometheus does. This is needed for improving integration with third-party solutions, which rely on these endpoints. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1833).
 
 * BUGFIX: vmagent: prevent from scraping duplicate targets if `-promscrape.dropOriginalLabels` command-line flag is set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1830). Thanks to @guidao for the fix.
-* BUGFIX: vmstorage [enterprise](https://victoriametrics.com/products/enterprise/): added missing `vm_tenant_used_tenant_bytes` metric, which shows the approximate per-tenant disk usage. See [these docs](https://docs.victoriametrics.com/PerTenantStatistic.html) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1605).
+* BUGFIX: vmstorage [enterprise](https://docs.victoriametrics.com/enterprise.html): added missing `vm_tenant_used_tenant_bytes` metric, which shows the approximate per-tenant disk usage. See [these docs](https://docs.victoriametrics.com/PerTenantStatistic.html) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1605).
 * BUGFIX: vmauth: properly take into account the value passed to `-maxIdleConnsPerBackend` command-line flag. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1300).
 * BUGFIX: vmagent: fix [reading data from Kafka](https://docs.victoriametrics.com/vmagent.html#reading-metrics-from-kafka).
 * BUGFIX: vmalert: fix [replay mode](https://docs.victoriametrics.com/vmalert.html#rules-backfilling) in enterprise version.
@@ -764,7 +833,7 @@ Released at 22-10-2021
 Released at 08-10-2021
 
 * FEATURE: add ability to accept metrics from [DataDog agent](https://docs.datadoghq.com/agent/) and [DogStatsD](https://docs.datadoghq.com/developers/dogstatsd/). See [these docs](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-send-data-from-datadog-agent). This option simplifies the migration path from DataDog to VictoriaMetrics. See also [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/206).
-* FEATURE: vmagent [enterprise](https://victoriametrics.com/products/enterprise/): add support for data reading and writing from/to [Apache Kafka](https://kafka.apache.org/). See [these docs](https://docs.victoriametrics.com/vmagent.html#kafka-integration).
+* FEATURE: vmagent [enterprise](https://docs.victoriametrics.com/enterprise.html): add support for data reading and writing from/to [Apache Kafka](https://kafka.apache.org/). See [these docs](https://docs.victoriametrics.com/vmagent.html#kafka-integration).
 * FEATURE: vmui: switch to [μPlot](https://github.com/leeoniya/uPlot) and add ability to naturally scroll and zoom graphs. See [these docs](https://docs.victoriametrics.com/#vmui). Thanks to @Loori-R.
 * FEATURE: vmstorage: stop accepting new data if `-storageDataPath` directory contains less than `-storage.minFreeDiskSpaceBytes` of free space. This should prevent from `out of disk space` crashes. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/269).
 * FEATURE: calculate quantiles in the same way as Prometheus does in such functions as [quantile_over_time](https://docs.victoriametrics.com/MetricsQL.html#quantile_over_time) and [quantile](https://docs.victoriametrics.com/MetricsQL.html#quantile). Previously results from VictoriaMetrics could be slightly different than results from Prometheus. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1625) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1612) issues.
@@ -820,7 +889,7 @@ Released at 20-09-2021
 * FEATURE: add [outliers_mad(tolerance, q)](https://docs.victoriametrics.com/MetricsQL.html#outliers_mad) function to [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html). It returns time series with peaks outside the [Median absolute deviation](https://en.wikipedia.org/wiki/Median_absolute_deviation) multiplied by `tolerance`.
 * FEATURE: add `histogram_quantiles("phiLabel", phi1, ..., phiN, buckets)` function to [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html). It calculates the given `phi*`-quantiles over the given `buckets` and returns time series per each quantile with the corresponding `{phiLabel="phi*"}` label.
 * FEATURE: add `quantiles_over_time("phiLabel", phi1, ..., phiN, series_selector[d])` function to [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html). It calculates the given `phi*`-quantiles over raw samples selected by `series_selector` on the given lookbehind window `d`. It returns time series per each quantile with the corresponding `{phiLabel="phi*"}` label.
-* FEATURE: [enterprise](https://victoriametrics.com/products/enterprise/): do not ask for `-eula` flag if `-version` flag is passed to enteprise app. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1621).
+* FEATURE: [enterprise](https://docs.victoriametrics.com/enterprise.html): do not ask for `-eula` flag if `-version` flag is passed to enteprise app. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1621).
 
 * BUGFIX: properly handle queries with multiple filters matching empty labels such as `metric{label1=~"foo|",label2="bar|"}`. This filter must match the following series: `metric`, `metric{label1="foo"}`, `metric{label2="bar"}` and `metric{label1="foo",label2="bar"}`. Previously it was matching only `metric{label1="foo",label2="bar"}`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1601).
 * BUGFIX: vmselect: reset connection timeouts after each request to `vmstorage`. This should prevent from `cannot read data in 0.000 seconds: unexpected EOF` warning in logs. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1562). Thanks to @mxlxm .
@@ -1171,7 +1240,7 @@ Released at 18-02-2021
 * FEATURE: remove dependency on external programs such as `cat`, `grep` and `cut` when detecting cpu and memory limits inside Docker or LXC container.
 * FEATURE: vmagent: add `__meta_kubernetes_endpoints_label_*`, `__meta_kubernetes_endpoints_labelpresent_*`, `__meta_kubernetes_endpoints_annotation_*` and `__meta_kubernetes_endpoints_annotationpresent_*` labels for `role: endpoints` in Kubernetes service discovery. These labels where added in Prometheus 2.25.
 * FEATURE: reduce the minimum supported retention period for inverted index (aka `indexdb`) from one month to one day. This should reduce disk space usage for `<-storageDataPath>/indexdb` folder if `-retentionPeriod` is set to values smaller than one month.
-* FEATURE: vmselect: export per-tenant metrics `vm_vmselect_http_requests_total` and `vm_vmselect_http_requests_duration_ms_total` . Other per-tenant metrics are available as a part of [enterprise package](https://victoriametrics.com/products/enterprise/). See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/932> for details.
+* FEATURE: vmselect: export per-tenant metrics `vm_vmselect_http_requests_total` and `vm_vmselect_http_requests_duration_ms_total` . Other per-tenant metrics are available as a part of [enterprise package](https://docs.victoriametrics.com/enterprise.html). See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/932> for details.
 
 * BUGFIX: properly convert regexp tag filters containing escaped dots to non-regexp tag filters. For example, `{foo=~"bar\.baz"}` should be converted to `{foo="bar.baz"}`. Previously it was incorrectly converted to `{foo="bar\.baz"}`, which could result in missing time series for this tag filter.
 * BUGFIX: do not spam error logs when discovering Docker Swarm targets without dedicated IP. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1028> .

docs/Cluster-VictoriaMetrics.md

@@ -159,6 +159,9 @@ A minimal cluster must contain the following nodes:
 - a single `vminsert` node with `-storageNode=<vmstorage_host>`
 - a single `vmselect` node with `-storageNode=<vmstorage_host>`
 
+[Enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/enterprise.html) supports automatic discoverying and updating of `vmstorage` nodes.
+See [these docs](#automatic-vmstorage-discovery) for details.
+
 It is recommended to run at least two nodes for each service for high availability purposes. In this case the cluster continues working when a single node is temporarily unavailable and the remaining nodes can handle the increased workload. The node may be temporarily unavailable when the underlying hardware breaks, during software upgrades, migration or other maintenance tasks.
 
 It is preferred to run many small `vmstorage` nodes over a few big `vmstorage` nodes, since this reduces the workload increase on the remaining `vmstorage` nodes when some of `vmstorage` nodes become temporarily unavailable.
@@ -184,16 +187,39 @@ It is possible manualy setting up a toy cluster on a single host. In this case e
 
 ### Environment variables
 
-Each flag values can be set through environment variables by following these rules:
+All the VictoriaMetrics components allow referring environment variables in command-line flags via `%{ENV_VAR}` syntax.
+For example, `-metricsAuthKey=%{METRICS_AUTH_KEY}` is automatically expanded to `-metricsAuthKey=top-secret`
+if `METRICS_AUTH_KEY=top-secret` environment variable exists at VictoriaMetrics startup.
+This expansion is performed by VictoriaMetrics itself.
+
+VictoriaMetrics recursively expands `%{ENV_VAR}` references in environment variables on startup.
+For example, `FOO=%{BAR}` environment variable is expanded to `FOO=abc` if `BAR=a%{BAZ}` and `BAZ=bc`.
+
+Additionally, all the VictoriaMetrics components allow setting flag values via environment variables according to these rules:
 
 - The `-envflag.enable` flag must be set
 - Each `.` in flag names must be substituted by `_` (for example `-insert.maxQueueDuration <duration>` will translate to `insert_maxQueueDuration=<duration>`)
 - For repeating flags, an alternative syntax can be used by joining the different values into one using `,` as separator (for example `-storageNode <nodeA> -storageNode <nodeB>` will translate to `storageNode=<nodeA>,<nodeB>`)
 - It is possible setting prefix for environment vars with `-envflag.prefix`. For instance, if `-envflag.prefix=VM_`, then env vars must be prepended with `VM_`
 
+## Automatic vmstorage discovery
+
+[Entrprise version of VictoriaMetrics](https://docs.victoriametrics.com/enterprise.html) supports [dns+srv](https://en.wikipedia.org/wiki/SRV_record) names
+at `-storageNode` command-line flag passed to `vminsert` and `vmstorage`. In this case the provided `dns+srv` names are resolved
+into tcp addresses of `vmstorage` nodes to connect to. The list of discovered `vmstorage` nodes is automatically updated at `vminsert` and `vmstorage`
+when it changes behind the corresponding `dns+srv` names. The `dns+srv` names must be prefixed with `dns+srv:` prefix.
+
+It is possible passing multiple `dns+srv` names to `-storageNode` command-line flag. In this case all these names are resolved to tcp addresses of `vmstorage` nodes to connect to.
+For example, `-storageNode='dns+srv:vmstorage-hot' -storageNode='dns+srv:vmstorage-cold'` .
+
+It is OK to pass regular static `vmstorage` addresses together with `dns+srv` addresses at `-storageNode`. For example,
+`-storageNode=vmstorage1,vmstorage2 -storageNode='dns+srv:vmstorage-autodiscovery'`.
+
+The currently discovered `vmstorage` nodes can be [monitored](#monitoring) vith `vm_rpc_vmstorage_is_reachable` and `vm_rpc_vmstorage_is_read_only` metrics.
+
 ## mTLS protection
 
-By default `vminsert` and `vmselect` nodes use unencrypted connections to `vmstorage` nodes, since it is assumed that all the cluster components run in a protected environment. [Enterprise version of VictoriaMetrics](https://victoriametrics.com/products/enterprise/) provides optional support for [mTLS connections](https://en.wikipedia.org/wiki/Mutual_authentication#mTLS) between cluster components. Pass `-cluster.tls=true` command-line flag to `vminsert`, `vmselect` and `vmstorage` nodes in order to enable mTLS protection. Additionally, `vminsert`, `vmselect` and `vmstorage` must be configured with mTLS certificates via `-cluster.tlsCertFile`, `-cluster.tlsKeyFile` command-line options. These certificates are mutually verified when `vminsert` and `vmselect` dial `vmstorage`.
+By default `vminsert` and `vmselect` nodes use unencrypted connections to `vmstorage` nodes, since it is assumed that all the cluster components run in a protected environment. [Enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/enterprise.html) provides optional support for [mTLS connections](https://en.wikipedia.org/wiki/Mutual_authentication#mTLS) between cluster components. Pass `-cluster.tls=true` command-line flag to `vminsert`, `vmselect` and `vmstorage` nodes in order to enable mTLS protection. Additionally, `vminsert`, `vmselect` and `vmstorage` must be configured with mTLS certificates via `-cluster.tlsCertFile`, `-cluster.tlsKeyFile` command-line options. These certificates are mutually verified when `vminsert` and `vmselect` dial `vmstorage`.
 
 The following optional command-line flags related to mTLS are supported:
 
@@ -205,7 +231,7 @@ When `vmselect` runs with `-clusternativeListenAddr` command-line option, then i
 
 See [these docs](https://gist.github.com/f41gh7/76ed8e5fb1ebb9737fe746bae9175ee6) on how to set up mTLS in VictoriaMetrics cluster.
 
-[Enterprise version of VictoriaMetrics](https://victoriametrics.com/products/enterprise/) can be downloaded and evaluated for free from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
+[Enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/enterprise.html) can be downloaded and evaluated for free from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
 
 ## Monitoring
 
@@ -281,7 +307,7 @@ See [troubleshooting docs](https://docs.victoriametrics.com/Troubleshooting.html
 - URLs for [Graphite Metrics API](https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api): `http://<vmselect>:8481/select/<accountID>/graphite/<suffix>`, where:
   - `<accountID>` is an arbitrary number identifying data namespace for query (aka tenant)
   - `<suffix>` may have the following values:
-    - `render` - implements Graphite Render API. See [these docs](https://graphite.readthedocs.io/en/stable/render_api.html). This functionality is available in [Enterprise package](https://victoriametrics.com/products/enterprise/). Enterprise binaries can be downloaded and evaluated for free from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
+    - `render` - implements Graphite Render API. See [these docs](https://graphite.readthedocs.io/en/stable/render_api.html). This functionality is available in [Enterprise package](https://docs.victoriametrics.com/enterprise.html). Enterprise binaries can be downloaded and evaluated for free from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
     - `metrics/find` - searches Graphite metrics. See [these docs](https://graphite-api.readthedocs.io/en/latest/api.html#metrics-find).
     - `metrics/expand` - expands Graphite metrics. See [these docs](https://graphite-api.readthedocs.io/en/latest/api.html#metrics-expand).
     - `metrics/index.json` - returns all the metric names. See [these docs](https://graphite-api.readthedocs.io/en/latest/api.html#metrics-index-json).
@@ -469,10 +495,11 @@ See also [resource usage limits docs](#resource-usage-limits).
 
 By default cluster components of VictoriaMetrics are tuned for an optimal resource usage under typical workloads. Some workloads may need fine-grained resource usage limits. In these cases the following command-line flags may be useful:
 
-- `-memory.allowedPercent` and `-search.allowedBytes` limit the amounts of memory, which may be used for various internal caches at all the cluster components of VictoriaMetrics - `vminsert`, `vmselect` and `vmstorage`. Note that VictoriaMetrics components may use more memory, since these flags don't limit additional memory, which may be needed on a per-query basis.
+- `-memory.allowedPercent` and `-memory.allowedBytes` limit the amounts of memory, which may be used for various internal caches at all the cluster components of VictoriaMetrics - `vminsert`, `vmselect` and `vmstorage`. Note that VictoriaMetrics components may use more memory, since these flags don't limit additional memory, which may be needed on a per-query basis.
+- `-search.maxMemoryPerQuery` limits the amounts of memory, which can be used for processing a single query at `vmselect` node. Queries, which need more memory, are rejected. Heavy queries, which select big number of time series, may exceed the per-query memory limit by a small percent. The total memory limit for concurrently executed queries can be estimated as `-search.maxMemoryPerQuery` multiplied by `-search.maxConcurrentRequests`.
 - `-search.maxUniqueTimeseries` at `vmselect` component limits the number of unique time series a single query can find and process. `vmselect` passes the limit to `vmstorage` component, which keeps in memory some metainformation about the time series located by each query and spends some CPU time for processing the found time series. This means that the maximum memory usage and CPU usage a single query can use at `vmstorage` is proportional to `-search.maxUniqueTimeseries`.
 - `-search.maxQueryDuration` at `vmselect` limits the duration of a single query. If the query takes longer than the given duration, then it is canceled. This allows saving CPU and RAM at `vmselect` and `vmstorage` when executing unexpected heavy queries.
-- `-search.maxConcurrentRequests` at `vmselect` limits the number of concurrent requests a single `vmselect` node can process. Bigger number of concurrent requests usually means bigger memory usage at both `vmselect` and `vmstorage`. For example, if a single query needs 100 MiB of additional memory during its execution, then 100 concurrent queries may need `100 * 100 MiB = 10 GiB` of additional memory. So it is better to limit the number of concurrent queries, while suspending additional incoming queries if the concurrency limit is reached. `vmselect` provides `-search.maxQueueDuration` command-line flag for limiting the max wait time for suspended queries.
+- `-search.maxConcurrentRequests` at `vmselect` limits the number of concurrent requests a single `vmselect` node can process. Bigger number of concurrent requests usually means bigger memory usage at both `vmselect` and `vmstorage`. For example, if a single query needs 100 MiB of additional memory during its execution, then 100 concurrent queries may need `100 * 100 MiB = 10 GiB` of additional memory. So it is better to limit the number of concurrent queries, while suspending additional incoming queries if the concurrency limit is reached. `vmselect` provides `-search.maxQueueDuration` command-line flag for limiting the max wait time for suspended queries. See also `-search.maxMemoryPerQuery` command-line flag.
 - `-search.maxSamplesPerSeries` at `vmselect` limits the number of raw samples the query can process per each time series. `vmselect` sequentially processes raw samples per each found time series during the query. It unpacks raw samples on the selected time range per each time series into memory and then applies the given [rollup function](https://docs.victoriametrics.com/MetricsQL.html#rollup-functions). The `-search.maxSamplesPerSeries` command-line flag allows limiting memory usage at `vmselect` in the case when the query is executed on a time range, which contains hundreds of millions of raw samples per each located time series.
 - `-search.maxSamplesPerQuery` at `vmselect` limits the number of raw samples a single query can process. This allows limiting CPU usage at `vmselect` for heavy queries.
 - `-search.maxPointsPerTimeseries` limits the number of calculated points, which can be returned per each matching time series from [range query](https://docs.victoriametrics.com/keyConcepts.html#range-query).
@@ -568,9 +595,37 @@ Restoring from backup:
 2. Restore data from backup using [vmrestore](https://docs.victoriametrics.com/vmrestore.html) into `-storageDataPath` directory.
 3. Start `vmstorage` node.
 
+## Retention filters
+
+[VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise.html) supports configuring multiple retentions for distinct sets of time series
+by passing `-retentionFilter` command-line flag to `vmstorage` nodes. See [these docs](https://docs.victoriametrics.com/#retention-filters) for details on this feature.
+
+Additionally, enterprise version of VictoriaMetrics cluster supports multiple retentions for distinct sets of [tenants](#multitenancy)
+by specifying filters on `vm_account_id` and/or `vm_project_id` pseudo-labels in `-retentionFilter` command-line flag.
+If the tenant doesn't match specified `-retentionFilter` options, then the global `-retentionPeriod` is used for it.
+
+For example, the following config sets retention to 1 day for [tenants](#multitenancy) with `accountID` starting from '42',
+then sets retention to 3 days for time series with label `env="dev"` or `env="prod"` from any tenant,
+while the rest of tenants will have 4 weeks retention:
+
+```
+-retentionFilter='{vm_account_id=~"42.*"}:1d' -retentionFilter='{env=~"dev|staging"}:3d' -retentionPeriod=4w
+```
+
+It is OK to mix filters on real labels with filters on `vm_account_id` and `vm_project_id` pseudo-labels.
+For example, the following config sets retention to 5 days for time series with `env="dev"` label from [tenant](#multitenancy) `accountID=5`:
+
+```
+-retentionFilter='{vm_account_id="5",env="dev"}:5d'
+```
+
+See also [these docs](https://docs.victoriametrics.com/#retention-filters) for additional details on retention filters.
+
+Enterprise binaries can be downloaded and evaluated for free from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
+
 ## Downsampling
 
-Downsampling is available in [enterprise version of VictoriaMetrics](https://victoriametrics.com/products/enterprise/). It is configured with `-downsampling.period` command-line flag. The same flag value must be passed to both `vmstorage` and `vmselect` nodes. See [these docs](https://docs.victoriametrics.com/#downsampling) for details.
+Downsampling is available in [enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/enterprise.html). It is configured with `-downsampling.period` command-line flag. The same flag value must be passed to both `vmstorage` and `vmselect` nodes. See [these docs](https://docs.victoriametrics.com/#downsampling) for details.
 
 Enterprise binaries can be downloaded and evaluated for free from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
 
@@ -797,7 +852,7 @@ Below is the output for `/path/to/vminsert -help`:
   -sortLabels
      Whether to sort labels for incoming samples before writing them to storage. This may be needed for reducing memory usage at storage when the order of labels in incoming samples is random. For example, if m{k1="v1",k2="v2"} may be sent as m{k2="v2",k1="v1"}. Enabled sorting for labels can slow down ingestion performance a bit
   -storageNode array
-     Comma-separated addresses of vmstorage nodes; usage: -storageNode=vmstorage-host1,...,vmstorage-hostN
+     Comma-separated addresses of vmstorage nodes; usage: -storageNode=vmstorage-host1,...,vmstorage-hostN . Enterprise version of VictoriaMetrics supports automatic discovery of vmstorage addresses via dns+srv records. For example, -storageNode=dns+srv:vmstorage.addrs . See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#automatic-vmstorage-discovery
      Supports an array of values separated by comma or specified via multiple flags.
   -tls
      Whether to enable TLS for incoming HTTP requests at -httpListenAddr (aka https). -tlsCertFile and -tlsKeyFile must be set if -tls is set
@@ -945,7 +1000,7 @@ Below is the output for `/path/to/vmselect -help`:
   -search.logSlowQueryDuration duration
      Log queries with execution time exceeding this value. Zero disables slow query logging (default 5s)
   -search.maxConcurrentRequests int
-     The maximum number of concurrent search requests. It shouldn't be high, since a single request can saturate all the CPU cores. See also -search.maxQueueDuration (default 8)
+     The maximum number of concurrent search requests. It shouldn't be high, since a single request can saturate all the CPU cores, while many concurrently executed requests may require high amounts of memory. See also -search.maxQueueDuration and -search.maxMemoryPerQuery (default 8)
   -search.maxExportDuration duration
      The maximum duration for /api/v1/export call (default 720h0m0s)
   -search.maxExportSeries int
@@ -956,6 +1011,9 @@ Below is the output for `/path/to/vmselect -help`:
      The maximum number of time series, which can be scanned during queries to Graphite Render API. See https://docs.victoriametrics.com/#graphite-render-api-usage . This flag is available only in enterprise version of VictoriaMetrics (default 300000)
   -search.maxLookback duration
      Synonym to -search.lookback-delta from Prometheus. The value is dynamically detected from interval between time series datapoints if not set. It can be overridden on per-query basis via max_lookback arg. See also '-search.maxStalenessInterval' flag, which has the same meaining due to historical reasons
+  -search.maxMemoryPerQuery size
+     The maximum amounts of memory a single query may consume. Queries requiring more memory are rejected. The total memory limit for concurrently executed queries can be estimated as -search.maxMemoryPerQuery multiplied by -search.maxConcurrentRequests
+     Supports the following optional suffixes for size values: KB, MB, GB, KiB, MiB, GiB (default 0)
   -search.maxPointsPerTimeseries int
      The maximum points per a single timeseries returned from /api/v1/query_range. This option doesn't limit the number of scanned raw samples in the database. The main purpose of this option is to limit the number of per-series points returned to graphing UI such as VMUI or Grafana. There is no sense in setting this limit to values bigger than the horizontal resolution of the graph (default 30000)
   -search.maxPointsSubqueryPerTimeseries int
@@ -1003,7 +1061,7 @@ Below is the output for `/path/to/vmselect -help`:
      Comma-separated addresses of vmselect nodes; usage: -selectNode=vmselect-host1,...,vmselect-hostN
      Supports an array of values separated by comma or specified via multiple flags.
   -storageNode array
-     Comma-separated addresses of vmstorage nodes; usage: -storageNode=vmstorage-host1,...,vmstorage-hostN
+     Comma-separated addresses of vmstorage nodes; usage: -storageNode=vmstorage-host1,...,vmstorage-hostN . Enterprise version of VictoriaMetrics supports automatic discovery of vmstorage addresses via dns+srv records. For example, -storageNode=dns+srv:vmstorage.addrs .See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#automatic-vmstorage-discovery
      Supports an array of values separated by comma or specified via multiple flags.
   -tls
      Whether to enable TLS for incoming HTTP requests at -httpListenAddr (aka https). -tlsCertFile and -tlsKeyFile must be set if -tls is set
@@ -1124,8 +1182,11 @@ Below is the output for `/path/to/vmstorage -help`:
   -pushmetrics.url array
      Optional URL to push metrics exposed at /metrics page. See https://docs.victoriametrics.com/#push-metrics . By default metrics exposed at /metrics page aren't pushed to any remote storage
      Supports an array of values separated by comma or specified via multiple flags.
+  -retentionFilter array
+     Retention filter in the format 'filter:retention'. For example, '{env="dev"}:3d' configures the retention for time series with env="dev" label to 3 days. See https://docs.victoriametrics.com/#retention-filters for details. This flag is available only in enterprise version of VictoriaMetrics
+     Supports an array of values separated by comma or specified via multiple flags.
   -retentionPeriod value
-     Data with timestamps outside the retentionPeriod is automatically deleted
+     Data with timestamps outside the retentionPeriod is automatically deleted. See also -retentionFilter
      The following optional suffixes are supported: h (hour), d (day), w (week), y (year). If suffix isn't set, then the duration is counted in months (default 1)
   -retentionTimezoneOffset duration
      The offset for performing indexdb rotation. If set to 0, then the indexdb rotation is performed at 4am UTC time per each -retentionPeriod. If set to 2h, then the indexdb rotation is performed at 4am EET time (the timezone with +2h offset)
@@ -1153,7 +1214,7 @@ Below is the output for `/path/to/vmstorage -help`:
      Overrides max size for indexdb/indexBlocks cache. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning
      Supports the following optional suffixes for size values: KB, MB, GB, KiB, MiB, GiB (default 0)
   -storage.cacheSizeIndexDBTagFilters size
-     Overrides max size for indexdb/tagFilters cache. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning
+     Overrides max size for indexdb/tagFiltersToMetricIDs cache. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning
      Supports the following optional suffixes for size values: KB, MB, GB, KiB, MiB, GiB (default 0)
   -storage.cacheSizeStorageTSID size
      Overrides max size for storage/tsid cache. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning

docs/Makefile

@@ -2,6 +2,8 @@ docs-install:
 	gem install jekyll bundler
 	bundle install --gemfile=Gemfile
 
+# To run localy you need to use ruby version =< 2.7.6, but not >=3.x , see https://bbs.archlinux.org/viewtopic.php?pid=1976408#p1976408
+#
 # run local server for documentation website
 # at http://127.0.0.1:4000/
 # On first use, please run `make docs-install`

docs/MetricsQL.md

@@ -1022,14 +1022,18 @@ per each `job` over the last 5 minutes.
 
 #### histogram_quantile
 
-`histogram_quantile(phi, buckets)` is a [transform function](#transform-functions), which calculates `phi`-quantile over the given
-[histogram buckets](https://valyala.medium.com/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350).
+`histogram_quantile(phi, buckets)` is a [transform function](#transform-functions), which calculates `phi`-[percentile](https://en.wikipedia.org/wiki/Percentile)
+over the given [histogram buckets](https://valyala.medium.com/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350).
 `phi` must be in the range `[0...1]`. For example, `histogram_quantile(0.5, sum(rate(http_request_duration_seconds_bucket[5m]) by (le))`
 would return median request duration for all the requests during the last 5 minutes.
 
 The function accepts optional third arg - `boundsLabel`. In this case it returns `lower` and `upper` bounds for the estimated percentile with the given `boundsLabel` label.
 See [this issue for details](https://github.com/prometheus/prometheus/issues/5706).
 
+When the [percentile](https://en.wikipedia.org/wiki/Percentile) is calculated over multiple histograms,
+then all the input histograms **must** have buckets with identical boundaries, e.g. they must have the same set of `le` or `vmrange` labels.
+Otherwise the returned result may be invalid. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3231) for details.
+
 This function is supported by PromQL (except of the `boundLabel` arg). See also [histogram_quantiles](#histogram_quantiles), [histogram_share](#histogram_share)
 and [quantile](#quantile).
 

docs/PerTenantStatistic.md

@@ -4,7 +4,7 @@ sort: 19
 
 # VictoriaMetrics Cluster Per Tenant Statistic
 
-***The per-tenant statistic is a part of [enterprise package](https://victoriametrics.com/products/enterprise/). It is available for download and evaluation at [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)***
+***The per-tenant statistic is a part of [enterprise package](https://docs.victoriametrics.com/enterprise.html). It is available for download and evaluation at [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)***
 
 <img alt="cluster-per-tenant-stat" src="PerTenantStatistic-stats.jpg">
 

docs/README.md

@@ -25,7 +25,7 @@ Learn more about [key concepts](https://docs.victoriametrics.com/keyConcepts.htm
 [quick start guide](https://docs.victoriametrics.com/Quick-Start.html) for a better experience.
 
 [Contact us](mailto:info@victoriametrics.com) if you need enterprise support for VictoriaMetrics. 
-See [features available in enterprise package](https://victoriametrics.com/products/enterprise/).
+See [features available in enterprise package](https://docs.victoriametrics.com/enterprise.html).
 Enterprise binaries can be downloaded and evaluated for free 
 from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
 
@@ -66,7 +66,7 @@ VictoriaMetrics has the following prominent features:
   * [DataDog agent or DogStatsD](#how-to-send-data-from-datadog-agent).
 * It supports metrics [relabeling](#relabeling).
 * It can deal with [high cardinality issues](https://docs.victoriametrics.com/FAQ.html#what-is-high-cardinality) and [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate) issues via [series limiter](#cardinality-limiter).
-* It ideally works with big amounts of time series data from APM, Kubernetes, IoT sensors, connected cars, industrial telemetry, financial data and various [Enterprise workloads](https://victoriametrics.com/products/enterprise/).
+* It ideally works with big amounts of time series data from APM, Kubernetes, IoT sensors, connected cars, industrial telemetry, financial data and various [Enterprise workloads](https://docs.victoriametrics.com/enterprise.html).
 * It has open source [cluster version](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/cluster).
 * It can store data on [NFS-based storages](https://en.wikipedia.org/wiki/Network_File_System) such as [Amazon EFS](https://aws.amazon.com/efs/) and [Google Filestore](https://cloud.google.com/filestore).
 
@@ -133,7 +133,15 @@ VictoriaMetrics is developed at a fast pace, so it is recommended periodically c
 
 ### Environment variables
 
-Each flag value can be set via environment variables according to these rules:
+All the VictoriaMetrics components allow referring environment variables in command-line flags via `%{ENV_VAR}` syntax.
+For example, `-metricsAuthKey=%{METRICS_AUTH_KEY}` is automatically expanded to `-metricsAuthKey=top-secret`
+if `METRICS_AUTH_KEY=top-secret` environment variable exists at VictoriaMetrics startup.
+This expansion is performed by VictoriaMetrics itself.
+
+VictoriaMetrics recursively expands `%{ENV_VAR}` references in environment variables on startup.
+For example, `FOO=%{BAR}` environment variable is expanded to `FOO=abc` if `BAR=a%{BAZ}` and `BAZ=bc`.
+
+Additionally, all the VictoriaMetrics components allow setting flag values via environment variables according to these rules:
 
 * The `-envflag.enable` flag must be set.
 * Each `.` char in flag name must be substituted with `_` (for example `-insert.maxQueueDuration <duration>` will translate to `insert_maxQueueDuration=<duration>`).
@@ -278,7 +286,7 @@ Multi-line queries can be entered by pressing `Shift-Enter` in query input field
 
 When querying the [backfilled data](https://docs.victoriametrics.com/#backfilling) or during [query troubleshooting](https://docs.victoriametrics.com/Troubleshooting.html#unexpected-query-results), it may be useful disabling response cache by clicking `Disable cache` checkbox.
 
-VMUI automatically adjusts the interval between datapoints on the graph depending on the horizontal resolution and on the selected time range. The step value can be customized by clickhing `Override step value` checkbox.
+VMUI automatically adjusts the interval between datapoints on the graph depending on the horizontal resolution and on the selected time range. The step value can be customized by changing `Step value` input.
 
 VMUI allows investigating correlations between multiple queries on the same graph. Just click `Add Query` button, enter an additional query in the newly appeared input field and press `Ctrl+Enter`. Results for all the queries should be displayed simultaneously on the same graph.
 
@@ -322,7 +330,7 @@ VictoriaMetrics is configured via command-line flags, so it must be restarted wh
 * Wait until the process stops. This can take a few seconds.
 * Start VictoriaMetrics with the new command-line flags.
 
-Prometheus doesn't drop data during VictoriaMetrics restart. See [this article](https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/) for details. The same applies alos to [vmagent](https://docs.victoriametrics.com/vmagent.html).
+Prometheus doesn't drop data during VictoriaMetrics restart. See [this article](https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/) for details. The same applies also to [vmagent](https://docs.victoriametrics.com/vmagent.html).
 
 ## How to scrape Prometheus exporters such as [node-exporter](https://github.com/prometheus/node_exporter)
 
@@ -732,7 +740,7 @@ VictoriaMetrics supports `__graphite__` pseudo-label for filtering time series w
 
 ### Graphite Render API usage
 
-[VictoriaMetrics Enterprise](https://victoriametrics.com/products/enterprise/) supports [Graphite Render API](https://graphite.readthedocs.io/en/stable/render_api.html) subset
+[VictoriaMetrics Enterprise](https://docs.victoriametrics.com/enterprise.html) supports [Graphite Render API](https://graphite.readthedocs.io/en/stable/render_api.html) subset
 at `/render` endpoint, which is used by [Graphite datasource in Grafana](https://grafana.com/docs/grafana/latest/datasources/graphite/).
 When configuring Graphite datasource in Grafana, the `Storage-Step` http request header must be set to a step between Graphite data points stored in VictoriaMetrics. For example, `Storage-Step: 10s` would mean 10 seconds distance between Graphite datapoints stored in VictoriaMetrics.
 Enterprise binaries can be downloaded and evaluated for free from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
@@ -773,7 +781,7 @@ to your needs or when testing bugfixes.
 
 ### Development build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.2.
 2. Run `make victoria-metrics` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `victoria-metrics` binary and puts it into the `bin` folder.
 
@@ -789,7 +797,7 @@ ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://b
 
 ### Development ARM build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.2.
 2. Run `make victoria-metrics-linux-arm` or `make victoria-metrics-linux-arm64` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `victoria-metrics-linux-arm` or `victoria-metrics-linux-arm64` binary respectively and puts it into the `bin` folder.
 
@@ -803,7 +811,7 @@ ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://b
 
 `Pure Go` mode builds only Go code without [cgo](https://golang.org/cmd/cgo/) dependencies.
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.2.
 2. Run `make victoria-metrics-pure` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `victoria-metrics-pure` binary and puts it into the `bin` folder.
 
@@ -1264,10 +1272,11 @@ See also [resource usage limits docs](#resource-usage-limits).
 
 By default VictoriaMetrics is tuned for an optimal resource usage under typical workloads. Some workloads may need fine-grained resource usage limits. In these cases the following command-line flags may be useful:
 
-- `-memory.allowedPercent` and `-search.allowedBytes` limit the amounts of memory, which may be used for various internal caches at VictoriaMetrics. Note that VictoriaMetrics may use more memory, since these flags don't limit additional memory, which may be needed on a per-query basis.
+- `-memory.allowedPercent` and `-memory.allowedBytes` limit the amounts of memory, which may be used for various internal caches at VictoriaMetrics. Note that VictoriaMetrics may use more memory, since these flags don't limit additional memory, which may be needed on a per-query basis.
+- `-search.maxMemoryPerQuery` limits the amounts of memory, which can be used for processing a single query. Queries, which need more memory, are rejected. Heavy queries, which select big number of time series, may exceed the per-query memory limit by a small percent. The total memory limit for concurrently executed queries can be estimated as `-search.maxMemoryPerQuery` multiplied by `-search.maxConcurrentRequests`.
 - `-search.maxUniqueTimeseries` limits the number of unique time series a single query can find and process. VictoriaMetrics keeps in memory some metainformation about the time series located by each query and spends some CPU time for processing the found time series. This means that the maximum memory usage and CPU usage a single query can use is proportional to `-search.maxUniqueTimeseries`.
 - `-search.maxQueryDuration` limits the duration of a single query. If the query takes longer than the given duration, then it is canceled. This allows saving CPU and RAM when executing unexpected heavy queries.
-- `-search.maxConcurrentRequests` limits the number of concurrent requests VictoriaMetrics can process. Bigger number of concurrent requests usually means bigger memory usage. For example, if a single query needs 100 MiB of additional memory during its execution, then 100 concurrent queries may need `100 * 100 MiB = 10 GiB` of additional memory. So it is better to limit the number of concurrent queries, while suspending additional incoming queries if the concurrency limit is reached. VictoriaMetrics provides `-search.maxQueueDuration` command-line flag for limiting the max wait time for suspended queries.
+- `-search.maxConcurrentRequests` limits the number of concurrent requests VictoriaMetrics can process. Bigger number of concurrent requests usually means bigger memory usage. For example, if a single query needs 100 MiB of additional memory during its execution, then 100 concurrent queries may need `100 * 100 MiB = 10 GiB` of additional memory. So it is better to limit the number of concurrent queries, while suspending additional incoming queries if the concurrency limit is reached. VictoriaMetrics provides `-search.maxQueueDuration` command-line flag for limiting the max wait time for suspended queries. See also `-search.maxMemoryPerQuery` command-line flag.
 - `-search.maxSamplesPerSeries` limits the number of raw samples the query can process per each time series. VictoriaMetrics sequentially processes raw samples per each found time series during the query. It unpacks raw samples on the selected time range per each time series into memory and then applies the given [rollup function](https://docs.victoriametrics.com/MetricsQL.html#rollup-functions). The `-search.maxSamplesPerSeries` command-line flag allows limiting memory usage in the case when the query is executed on a time range, which contains hundreds of millions of raw samples per each located time series.
 - `-search.maxSamplesPerQuery` limits the number of raw samples a single query can process. This allows limiting CPU usage for heavy queries.
 - `-search.maxPointsPerTimeseries` limits the number of calculated points, which can be returned per each matching time series from [range query](https://docs.victoriametrics.com/keyConcepts.html#range-query).
@@ -1401,7 +1410,11 @@ VictoriaMetrics does not support indefinite retention, but you can specify an ar
 
 ## Multiple retentions
 
-A single instance of VictoriaMetrics supports only a single retention, which can be configured via `-retentionPeriod` command-line flag. If you need multiple retentions, then you may start multiple VictoriaMetrics instances with distinct values for the following flags:
+Distinct retentions for distinct time series can be configured via [retention filters](#retention-filters)
+in [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise.html).
+
+Community version of VictoriaMetrics supports only a single retention, which can be configured via [-retentionPeriod](#retention) command-line flag.
+If you need multiple retentions in community version of VictoriaMetrics, then you may start multiple VictoriaMetrics instances with distinct values for the following flags:
 
 * `-retentionPeriod`
 * `-storageDataPath`, so the data for each retention period is saved in a separate directory
@@ -1409,12 +1422,44 @@ A single instance of VictoriaMetrics supports only a single retention, which can
 
 Then set up [vmauth](https://docs.victoriametrics.com/vmauth.html) in front of VictoriaMetrics instances,
 so it could route requests from particular user to VictoriaMetrics with the desired retention.
-The same scheme could be implemented for multiple tenants in [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html).
+
+Similar scheme can be applied for multiple tenants in [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html).
 See [these docs](https://docs.victoriametrics.com/guides/guide-vmcluster-multiple-retention-setup.html) for multi-retention setup details.
 
+## Retention filters
+
+[Enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/enterprise.html) supports e.g. `retention filters`,
+which allow configuring multiple retentions for distinct sets of time series matching the configured [series filters](https://docs.victoriametrics.com/keyConcepts.html#filtering)
+via `-retentionFilter` command-line flag. This flag accepts `filter:duration` options, where `filter` must be
+a valid [series filter](https://docs.victoriametrics.com/keyConcepts.html#filtering), while the `duration`
+must contain valid [retention](#retention) for time series matching the given `filter`. If series doesn't match
+any configured `-retentionFilter`, then the retention configured via [-retentionPeriod](#retention) command-line flag is applied to it.
+If series matches multiple configured retention filters, then the smallest retention is applied.
+
+For example, the following config sets 3 days retention for time series with `team="juniors"` label,
+30 days retention for time series with `env="dev"` or `env="staging"` label and 1 year retention for the remaining time series:
+
+```
+-retentionFilter='{team="juniors"}:3d' -retentionFilter='{env=~"dev|staging"}:30d' -retentionPeriod=1y
+```
+
+Important notes:
+
+- The data outside of the configured retention isn't deleted instantly - it is deleted eventually during [background merges](https://docs.victoriametrics.com/#storage).
+- The `-retentionFilter` doesn't remove old data from `indexdb` (aka inverted index) until the configured [-retentionPeriod](#retention).
+  So the `indexdb` size can grow big under [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate)
+  even for small retentions configured via `-retentionFilter`.
+
+It is safe updating `-retentionFilter` during VictoriaMetrics restarts - the updated retention filters are applied eventually
+to historical data.
+
+See [how to configure multiple retentions in VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#retention-filters).
+
+Retention filters can be evaluated for free by downloading and using enterprise binaries from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
+
 ## Downsampling
 
-[VictoriaMetrics Enterprise](https://victoriametrics.com/products/enterprise/) supports multi-level downsampling with `-downsampling.period` command-line flag. For example:
+[VictoriaMetrics Enterprise](https://docs.victoriametrics.com/enterprise.html) supports multi-level downsampling with `-downsampling.period` command-line flag. For example:
 
 * `-downsampling.period=30d:5m` instructs VictoriaMetrics to [deduplicate](#deduplication) samples older than 30 days with 5 minutes interval.
 
@@ -2206,8 +2251,11 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      Optional path to a file with relabeling rules, which are applied to all the ingested metrics. The path can point either to local file or to http url. See https://docs.victoriametrics.com/#relabeling for details. The config is reloaded on SIGHUP signal
   -relabelDebug
      Whether to log metrics before and after relabeling with -relabelConfig. If the -relabelDebug is enabled, then the metrics aren't sent to storage. This is useful for debugging the relabeling configs
+  -retentionFilter array
+     Retention filter in the format 'filter:retention'. For example, '{env="dev"}:3d' configures the retention for time series with env="dev" label to 3 days. See https://docs.victoriametrics.com/#retention-filters for details. This flag is available only in enterprise version of VictoriaMetrics
+     Supports an array of values separated by comma or specified via multiple flags.
   -retentionPeriod value
-     Data with timestamps outside the retentionPeriod is automatically deleted
+     Data with timestamps outside the retentionPeriod is automatically deleted. See also -retentionFilter
      The following optional suffixes are supported: h (hour), d (day), w (week), y (year). If suffix isn't set, then the duration is counted in months (default 1)
   -retentionTimezoneOffset duration
      The offset for performing indexdb rotation. If set to 0, then the indexdb rotation is performed at 4am UTC time per each -retentionPeriod. If set to 2h, then the indexdb rotation is performed at 4am EET time (the timezone with +2h offset)
@@ -2226,7 +2274,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
   -search.logSlowQueryDuration duration
      Log queries with execution time exceeding this value. Zero disables slow query logging (default 5s)
   -search.maxConcurrentRequests int
-     The maximum number of concurrent search requests. It shouldn't be high, since a single request can saturate all the CPU cores. See also -search.maxQueueDuration (default 8)
+     The maximum number of concurrent search requests. It shouldn't be high, since a single request can saturate all the CPU cores, while many concurrently executed requests may require high amounts of memory. See also -search.maxQueueDuration and -search.maxMemoryPerQuery (default 8)
   -search.maxExportDuration duration
      The maximum duration for /api/v1/export call (default 720h0m0s)
   -search.maxExportSeries int
@@ -2237,6 +2285,9 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      The maximum number of time series, which can be scanned during queries to Graphite Render API. See https://docs.victoriametrics.com/#graphite-render-api-usage . This flag is available only in enterprise version of VictoriaMetrics (default 300000)
   -search.maxLookback duration
      Synonym to -search.lookback-delta from Prometheus. The value is dynamically detected from interval between time series datapoints if not set. It can be overridden on per-query basis via max_lookback arg. See also '-search.maxStalenessInterval' flag, which has the same meaining due to historical reasons
+  -search.maxMemoryPerQuery size
+     The maximum amounts of memory a single query may consume. Queries requiring more memory are rejected. The total memory limit for concurrently executed queries can be estimated as -search.maxMemoryPerQuery multiplied by -search.maxConcurrentRequests
+     Supports the following optional suffixes for size values: KB, MB, GB, KiB, MiB, GiB (default 0)
   -search.maxPointsPerTimeseries int
      The maximum points per a single timeseries returned from /api/v1/query_range. This option doesn't limit the number of scanned raw samples in the database. The main purpose of this option is to limit the number of per-series points returned to graphing UI such as VMUI or Grafana. There is no sense in setting this limit to values bigger than the horizontal resolution of the graph (default 30000)
   -search.maxPointsSubqueryPerTimeseries int
@@ -2306,7 +2357,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      Overrides max size for indexdb/indexBlocks cache. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning
      Supports the following optional suffixes for size values: KB, MB, GB, KiB, MiB, GiB (default 0)
   -storage.cacheSizeIndexDBTagFilters size
-     Overrides max size for indexdb/tagFilters cache. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning
+     Overrides max size for indexdb/tagFiltersToMetricIDs cache. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning
      Supports the following optional suffixes for size values: KB, MB, GB, KiB, MiB, GiB (default 0)
   -storage.cacheSizeStorageTSID size
      Overrides max size for storage/tsid cache. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning

docs/Single-server-VictoriaMetrics.md

@@ -28,7 +28,7 @@ Learn more about [key concepts](https://docs.victoriametrics.com/keyConcepts.htm
 [quick start guide](https://docs.victoriametrics.com/Quick-Start.html) for a better experience.
 
 [Contact us](mailto:info@victoriametrics.com) if you need enterprise support for VictoriaMetrics. 
-See [features available in enterprise package](https://victoriametrics.com/products/enterprise/).
+See [features available in enterprise package](https://docs.victoriametrics.com/enterprise.html).
 Enterprise binaries can be downloaded and evaluated for free 
 from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
 
@@ -69,7 +69,7 @@ VictoriaMetrics has the following prominent features:
   * [DataDog agent or DogStatsD](#how-to-send-data-from-datadog-agent).
 * It supports metrics [relabeling](#relabeling).
 * It can deal with [high cardinality issues](https://docs.victoriametrics.com/FAQ.html#what-is-high-cardinality) and [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate) issues via [series limiter](#cardinality-limiter).
-* It ideally works with big amounts of time series data from APM, Kubernetes, IoT sensors, connected cars, industrial telemetry, financial data and various [Enterprise workloads](https://victoriametrics.com/products/enterprise/).
+* It ideally works with big amounts of time series data from APM, Kubernetes, IoT sensors, connected cars, industrial telemetry, financial data and various [Enterprise workloads](https://docs.victoriametrics.com/enterprise.html).
 * It has open source [cluster version](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/cluster).
 * It can store data on [NFS-based storages](https://en.wikipedia.org/wiki/Network_File_System) such as [Amazon EFS](https://aws.amazon.com/efs/) and [Google Filestore](https://cloud.google.com/filestore).
 
@@ -136,7 +136,15 @@ VictoriaMetrics is developed at a fast pace, so it is recommended periodically c
 
 ### Environment variables
 
-Each flag value can be set via environment variables according to these rules:
+All the VictoriaMetrics components allow referring environment variables in command-line flags via `%{ENV_VAR}` syntax.
+For example, `-metricsAuthKey=%{METRICS_AUTH_KEY}` is automatically expanded to `-metricsAuthKey=top-secret`
+if `METRICS_AUTH_KEY=top-secret` environment variable exists at VictoriaMetrics startup.
+This expansion is performed by VictoriaMetrics itself.
+
+VictoriaMetrics recursively expands `%{ENV_VAR}` references in environment variables on startup.
+For example, `FOO=%{BAR}` environment variable is expanded to `FOO=abc` if `BAR=a%{BAZ}` and `BAZ=bc`.
+
+Additionally, all the VictoriaMetrics components allow setting flag values via environment variables according to these rules:
 
 * The `-envflag.enable` flag must be set.
 * Each `.` char in flag name must be substituted with `_` (for example `-insert.maxQueueDuration <duration>` will translate to `insert_maxQueueDuration=<duration>`).
@@ -281,7 +289,7 @@ Multi-line queries can be entered by pressing `Shift-Enter` in query input field
 
 When querying the [backfilled data](https://docs.victoriametrics.com/#backfilling) or during [query troubleshooting](https://docs.victoriametrics.com/Troubleshooting.html#unexpected-query-results), it may be useful disabling response cache by clicking `Disable cache` checkbox.
 
-VMUI automatically adjusts the interval between datapoints on the graph depending on the horizontal resolution and on the selected time range. The step value can be customized by clickhing `Override step value` checkbox.
+VMUI automatically adjusts the interval between datapoints on the graph depending on the horizontal resolution and on the selected time range. The step value can be customized by changing `Step value` input.
 
 VMUI allows investigating correlations between multiple queries on the same graph. Just click `Add Query` button, enter an additional query in the newly appeared input field and press `Ctrl+Enter`. Results for all the queries should be displayed simultaneously on the same graph.
 
@@ -325,7 +333,7 @@ VictoriaMetrics is configured via command-line flags, so it must be restarted wh
 * Wait until the process stops. This can take a few seconds.
 * Start VictoriaMetrics with the new command-line flags.
 
-Prometheus doesn't drop data during VictoriaMetrics restart. See [this article](https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/) for details. The same applies alos to [vmagent](https://docs.victoriametrics.com/vmagent.html).
+Prometheus doesn't drop data during VictoriaMetrics restart. See [this article](https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/) for details. The same applies also to [vmagent](https://docs.victoriametrics.com/vmagent.html).
 
 ## How to scrape Prometheus exporters such as [node-exporter](https://github.com/prometheus/node_exporter)
 
@@ -735,7 +743,7 @@ VictoriaMetrics supports `__graphite__` pseudo-label for filtering time series w
 
 ### Graphite Render API usage
 
-[VictoriaMetrics Enterprise](https://victoriametrics.com/products/enterprise/) supports [Graphite Render API](https://graphite.readthedocs.io/en/stable/render_api.html) subset
+[VictoriaMetrics Enterprise](https://docs.victoriametrics.com/enterprise.html) supports [Graphite Render API](https://graphite.readthedocs.io/en/stable/render_api.html) subset
 at `/render` endpoint, which is used by [Graphite datasource in Grafana](https://grafana.com/docs/grafana/latest/datasources/graphite/).
 When configuring Graphite datasource in Grafana, the `Storage-Step` http request header must be set to a step between Graphite data points stored in VictoriaMetrics. For example, `Storage-Step: 10s` would mean 10 seconds distance between Graphite datapoints stored in VictoriaMetrics.
 Enterprise binaries can be downloaded and evaluated for free from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
@@ -776,7 +784,7 @@ to your needs or when testing bugfixes.
 
 ### Development build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.2.
 2. Run `make victoria-metrics` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `victoria-metrics` binary and puts it into the `bin` folder.
 
@@ -792,7 +800,7 @@ ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://b
 
 ### Development ARM build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.2.
 2. Run `make victoria-metrics-linux-arm` or `make victoria-metrics-linux-arm64` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `victoria-metrics-linux-arm` or `victoria-metrics-linux-arm64` binary respectively and puts it into the `bin` folder.
 
@@ -806,7 +814,7 @@ ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://b
 
 `Pure Go` mode builds only Go code without [cgo](https://golang.org/cmd/cgo/) dependencies.
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.2.
 2. Run `make victoria-metrics-pure` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `victoria-metrics-pure` binary and puts it into the `bin` folder.
 
@@ -1267,10 +1275,11 @@ See also [resource usage limits docs](#resource-usage-limits).
 
 By default VictoriaMetrics is tuned for an optimal resource usage under typical workloads. Some workloads may need fine-grained resource usage limits. In these cases the following command-line flags may be useful:
 
-- `-memory.allowedPercent` and `-search.allowedBytes` limit the amounts of memory, which may be used for various internal caches at VictoriaMetrics. Note that VictoriaMetrics may use more memory, since these flags don't limit additional memory, which may be needed on a per-query basis.
+- `-memory.allowedPercent` and `-memory.allowedBytes` limit the amounts of memory, which may be used for various internal caches at VictoriaMetrics. Note that VictoriaMetrics may use more memory, since these flags don't limit additional memory, which may be needed on a per-query basis.
+- `-search.maxMemoryPerQuery` limits the amounts of memory, which can be used for processing a single query. Queries, which need more memory, are rejected. Heavy queries, which select big number of time series, may exceed the per-query memory limit by a small percent. The total memory limit for concurrently executed queries can be estimated as `-search.maxMemoryPerQuery` multiplied by `-search.maxConcurrentRequests`.
 - `-search.maxUniqueTimeseries` limits the number of unique time series a single query can find and process. VictoriaMetrics keeps in memory some metainformation about the time series located by each query and spends some CPU time for processing the found time series. This means that the maximum memory usage and CPU usage a single query can use is proportional to `-search.maxUniqueTimeseries`.
 - `-search.maxQueryDuration` limits the duration of a single query. If the query takes longer than the given duration, then it is canceled. This allows saving CPU and RAM when executing unexpected heavy queries.
-- `-search.maxConcurrentRequests` limits the number of concurrent requests VictoriaMetrics can process. Bigger number of concurrent requests usually means bigger memory usage. For example, if a single query needs 100 MiB of additional memory during its execution, then 100 concurrent queries may need `100 * 100 MiB = 10 GiB` of additional memory. So it is better to limit the number of concurrent queries, while suspending additional incoming queries if the concurrency limit is reached. VictoriaMetrics provides `-search.maxQueueDuration` command-line flag for limiting the max wait time for suspended queries.
+- `-search.maxConcurrentRequests` limits the number of concurrent requests VictoriaMetrics can process. Bigger number of concurrent requests usually means bigger memory usage. For example, if a single query needs 100 MiB of additional memory during its execution, then 100 concurrent queries may need `100 * 100 MiB = 10 GiB` of additional memory. So it is better to limit the number of concurrent queries, while suspending additional incoming queries if the concurrency limit is reached. VictoriaMetrics provides `-search.maxQueueDuration` command-line flag for limiting the max wait time for suspended queries. See also `-search.maxMemoryPerQuery` command-line flag.
 - `-search.maxSamplesPerSeries` limits the number of raw samples the query can process per each time series. VictoriaMetrics sequentially processes raw samples per each found time series during the query. It unpacks raw samples on the selected time range per each time series into memory and then applies the given [rollup function](https://docs.victoriametrics.com/MetricsQL.html#rollup-functions). The `-search.maxSamplesPerSeries` command-line flag allows limiting memory usage in the case when the query is executed on a time range, which contains hundreds of millions of raw samples per each located time series.
 - `-search.maxSamplesPerQuery` limits the number of raw samples a single query can process. This allows limiting CPU usage for heavy queries.
 - `-search.maxPointsPerTimeseries` limits the number of calculated points, which can be returned per each matching time series from [range query](https://docs.victoriametrics.com/keyConcepts.html#range-query).
@@ -1404,7 +1413,11 @@ VictoriaMetrics does not support indefinite retention, but you can specify an ar
 
 ## Multiple retentions
 
-A single instance of VictoriaMetrics supports only a single retention, which can be configured via `-retentionPeriod` command-line flag. If you need multiple retentions, then you may start multiple VictoriaMetrics instances with distinct values for the following flags:
+Distinct retentions for distinct time series can be configured via [retention filters](#retention-filters)
+in [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise.html).
+
+Community version of VictoriaMetrics supports only a single retention, which can be configured via [-retentionPeriod](#retention) command-line flag.
+If you need multiple retentions in community version of VictoriaMetrics, then you may start multiple VictoriaMetrics instances with distinct values for the following flags:
 
 * `-retentionPeriod`
 * `-storageDataPath`, so the data for each retention period is saved in a separate directory
@@ -1412,12 +1425,44 @@ A single instance of VictoriaMetrics supports only a single retention, which can
 
 Then set up [vmauth](https://docs.victoriametrics.com/vmauth.html) in front of VictoriaMetrics instances,
 so it could route requests from particular user to VictoriaMetrics with the desired retention.
-The same scheme could be implemented for multiple tenants in [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html).
+
+Similar scheme can be applied for multiple tenants in [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html).
 See [these docs](https://docs.victoriametrics.com/guides/guide-vmcluster-multiple-retention-setup.html) for multi-retention setup details.
 
+## Retention filters
+
+[Enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/enterprise.html) supports e.g. `retention filters`,
+which allow configuring multiple retentions for distinct sets of time series matching the configured [series filters](https://docs.victoriametrics.com/keyConcepts.html#filtering)
+via `-retentionFilter` command-line flag. This flag accepts `filter:duration` options, where `filter` must be
+a valid [series filter](https://docs.victoriametrics.com/keyConcepts.html#filtering), while the `duration`
+must contain valid [retention](#retention) for time series matching the given `filter`. If series doesn't match
+any configured `-retentionFilter`, then the retention configured via [-retentionPeriod](#retention) command-line flag is applied to it.
+If series matches multiple configured retention filters, then the smallest retention is applied.
+
+For example, the following config sets 3 days retention for time series with `team="juniors"` label,
+30 days retention for time series with `env="dev"` or `env="staging"` label and 1 year retention for the remaining time series:
+
+```
+-retentionFilter='{team="juniors"}:3d' -retentionFilter='{env=~"dev|staging"}:30d' -retentionPeriod=1y
+```
+
+Important notes:
+
+- The data outside of the configured retention isn't deleted instantly - it is deleted eventually during [background merges](https://docs.victoriametrics.com/#storage).
+- The `-retentionFilter` doesn't remove old data from `indexdb` (aka inverted index) until the configured [-retentionPeriod](#retention).
+  So the `indexdb` size can grow big under [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate)
+  even for small retentions configured via `-retentionFilter`.
+
+It is safe updating `-retentionFilter` during VictoriaMetrics restarts - the updated retention filters are applied eventually
+to historical data.
+
+See [how to configure multiple retentions in VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#retention-filters).
+
+Retention filters can be evaluated for free by downloading and using enterprise binaries from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
+
 ## Downsampling
 
-[VictoriaMetrics Enterprise](https://victoriametrics.com/products/enterprise/) supports multi-level downsampling with `-downsampling.period` command-line flag. For example:
+[VictoriaMetrics Enterprise](https://docs.victoriametrics.com/enterprise.html) supports multi-level downsampling with `-downsampling.period` command-line flag. For example:
 
 * `-downsampling.period=30d:5m` instructs VictoriaMetrics to [deduplicate](#deduplication) samples older than 30 days with 5 minutes interval.
 
@@ -2209,8 +2254,11 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      Optional path to a file with relabeling rules, which are applied to all the ingested metrics. The path can point either to local file or to http url. See https://docs.victoriametrics.com/#relabeling for details. The config is reloaded on SIGHUP signal
   -relabelDebug
      Whether to log metrics before and after relabeling with -relabelConfig. If the -relabelDebug is enabled, then the metrics aren't sent to storage. This is useful for debugging the relabeling configs
+  -retentionFilter array
+     Retention filter in the format 'filter:retention'. For example, '{env="dev"}:3d' configures the retention for time series with env="dev" label to 3 days. See https://docs.victoriametrics.com/#retention-filters for details. This flag is available only in enterprise version of VictoriaMetrics
+     Supports an array of values separated by comma or specified via multiple flags.
   -retentionPeriod value
-     Data with timestamps outside the retentionPeriod is automatically deleted
+     Data with timestamps outside the retentionPeriod is automatically deleted. See also -retentionFilter
      The following optional suffixes are supported: h (hour), d (day), w (week), y (year). If suffix isn't set, then the duration is counted in months (default 1)
   -retentionTimezoneOffset duration
      The offset for performing indexdb rotation. If set to 0, then the indexdb rotation is performed at 4am UTC time per each -retentionPeriod. If set to 2h, then the indexdb rotation is performed at 4am EET time (the timezone with +2h offset)
@@ -2229,7 +2277,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
   -search.logSlowQueryDuration duration
      Log queries with execution time exceeding this value. Zero disables slow query logging (default 5s)
   -search.maxConcurrentRequests int
-     The maximum number of concurrent search requests. It shouldn't be high, since a single request can saturate all the CPU cores. See also -search.maxQueueDuration (default 8)
+     The maximum number of concurrent search requests. It shouldn't be high, since a single request can saturate all the CPU cores, while many concurrently executed requests may require high amounts of memory. See also -search.maxQueueDuration and -search.maxMemoryPerQuery (default 8)
   -search.maxExportDuration duration
      The maximum duration for /api/v1/export call (default 720h0m0s)
   -search.maxExportSeries int
@@ -2240,6 +2288,9 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      The maximum number of time series, which can be scanned during queries to Graphite Render API. See https://docs.victoriametrics.com/#graphite-render-api-usage . This flag is available only in enterprise version of VictoriaMetrics (default 300000)
   -search.maxLookback duration
      Synonym to -search.lookback-delta from Prometheus. The value is dynamically detected from interval between time series datapoints if not set. It can be overridden on per-query basis via max_lookback arg. See also '-search.maxStalenessInterval' flag, which has the same meaining due to historical reasons
+  -search.maxMemoryPerQuery size
+     The maximum amounts of memory a single query may consume. Queries requiring more memory are rejected. The total memory limit for concurrently executed queries can be estimated as -search.maxMemoryPerQuery multiplied by -search.maxConcurrentRequests
+     Supports the following optional suffixes for size values: KB, MB, GB, KiB, MiB, GiB (default 0)
   -search.maxPointsPerTimeseries int
      The maximum points per a single timeseries returned from /api/v1/query_range. This option doesn't limit the number of scanned raw samples in the database. The main purpose of this option is to limit the number of per-series points returned to graphing UI such as VMUI or Grafana. There is no sense in setting this limit to values bigger than the horizontal resolution of the graph (default 30000)
   -search.maxPointsSubqueryPerTimeseries int
@@ -2309,7 +2360,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      Overrides max size for indexdb/indexBlocks cache. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning
      Supports the following optional suffixes for size values: KB, MB, GB, KiB, MiB, GiB (default 0)
   -storage.cacheSizeIndexDBTagFilters size
-     Overrides max size for indexdb/tagFilters cache. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning
+     Overrides max size for indexdb/tagFiltersToMetricIDs cache. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning
      Supports the following optional suffixes for size values: KB, MB, GB, KiB, MiB, GiB (default 0)
   -storage.cacheSizeStorageTSID size
      Overrides max size for storage/tsid cache. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning

docs/_includes/img.html

@@ -1,5 +0,0 @@
-<p style="text-align: center">
-    <a href="{{ include.href }}" target="_blank" rel="noopener">
-        <img src="{{ include.href }}">
-    </a>
-</p>

docs/enterprise.md

@@ -0,0 +1,55 @@
+---
+sort: 99
+---
+
+# VictoriaMetrics enterprise
+
+VictoriaMetrics components are provided in two kinds - [community edition](https://victoriametrics.com/products/open-source/)
+and [enterprise edition](https://victoriametrics.com/products/enterprise/).
+
+VictoriaMetrics community components are open source and are free to use - see [the source code](https://github.com/VictoriaMetrics/VictoriaMetrics/)
+and [the license](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/LICENSE).
+
+The use of VictoriaMetrics enteprise components is permitted in the following cases:
+
+- Evaluation use in non-production setups. Just download and run enterprise binaries or packages of VictoriaMetrics
+  components from usual places - [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) and [docker hub](https://hub.docker.com/u/victoriametrics).
+  Enterprise binaries and packages have `enterprise` suffix in their names.
+
+- Production use if you have a valid enterprise contract or valid permit from VictoriaMetrics company.
+  [Contact us](mailto:info@victoriametrics.com) if you need such contract.
+
+- [Managed VictoriaMetrics](https://docs.victoriametrics.com/managed_victoriametrics/) is built on top of enterprise binaries of VictoriaMetrics.
+
+All the enterprise apps require `-eula` command-line flag to be passed to them. This flag acknowledges that your usage fits one of the cases listed above.
+
+## VictoriaMetrics enterprise features
+
+VictoriaMetrics enterprise includes [all the features of the community edition](https://docs.victoriametrics.com/#prominent-features),
+plus the following additional features:
+
+- [Downsampling](https://docs.victoriametrics.com/#downsampling) - this feature allows reducing storage costs
+  and increasing performance for queries over historical data.
+- [Multiple retentions](https://docs.victoriametrics.com/#retention-filters) - this feature allows reducing storage costs
+  by specifying different retentions to different datasets.
+- [Automatic discovery of vmstorage nodes](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#automatic-vmstorage-discovery) -
+  this feature allows updating the list of `vmstorage` nodes at `vminsert` and `vmselect` without the need to restart these services.
+- [Graphite querying](https://docs.victoriametrics.com/#graphite-render-api-usage) - this feature allows seamless
+  transition from Graphite to VictoriaMetrics without the need to modify queries at dashboards and alerts.
+  VictoriaMetrics allows reducing infrastructure costs by more than 10x comparing to Graphite -
+  see [this case study](https://docs.victoriametrics.com/CaseStudies.html#grammarly).
+- [Backup automation](https://docs.victoriametrics.com/vmbackupmanager.html).
+- [Advanced per-tenant stats](https://docs.victoriametrics.com/PerTenantStatistic.html).
+- [Advanced auth and rate limiter](https://docs.victoriametrics.com/vmgateway.html).
+- [mTLS for cluster components](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection).
+- [Kafka integration](https://docs.victoriametrics.com/vmagent.html#kafka-integration).
+- [Multitenant support in vmalert](https://docs.victoriametrics.com/vmalert.html#multitenancy).
+
+On top of this, enterprise package of VictoriaMetrics usually includes the following important Enterprise points:
+
+- First-class consulting and technical support provided by the core dev team.
+- [Monitoring of monitoring](https://victoriametrics.com/products/mom/) - this feature allows forecasting
+  and preventing possible issues in VictoriaMetrics setups.
+- [Enterprise security compliance](https://victoriametrics.com/security/).
+
+[Contact us](mailto:info@victoriametrics.com) if you are interested in VictoriaMetrics enterprise.

docs/guides/README.md

@@ -11,3 +11,4 @@ sort: 26
 5. [Multi Retention Setup within VictoriaMetrics Cluster](https://docs.victoriametrics.com/guides/guide-vmcluster-multiple-retention-setup.html)
 6. [Migrate from InfluxDB to VictoriaMetrics](https://docs.victoriametrics.com/guides/migrate-from-influx.html)
 7. [Multi-regional setup with VictoriaMetrics: Dedicated regions for monitoring](https://docs.victoriametrics.com/guides/multi-regional-setup-dedicated-regions.html)
+8. [How to delete or replace metrics in VictoriaMetrics](https://docs.victoriametrics.com/guides/guide-delete-and-replace-metrics.html)

docs/guides/guide-delete-or-replace-metrics.md

@@ -0,0 +1,236 @@
+# How to delete or replace metrics in VictoriaMetrics 
+
+Data deletion is an operation people expect a database to have. [VictoriaMetrics](https://victoriametrics.com) supports 
+[delete operation](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-delete-time-series) but to a limited extent. Due to implementation details, VictoriaMetrics remains an [append-only database](https://en.wikipedia.org/wiki/Append-only), which perfectly fits the case for storing time series data. But the drawback of such architecture is that it is extremely expensive to mutate the data. Hence, `delete` or `update` operations support is very limited. In this guide, we'll walk through the possible workarounds for deleting or changing already written data in VictoriaMetrics.
+
+### Precondition
+
+- [Single-node VictoriaMetrics](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html);
+- [Cluster version of VictoriaMetrics](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html);
+- [curl](https://curl.se/docs/manual.html)
+- [jq tool](https://stedolan.github.io/jq/)
+
+## How to delete metrics
+
+_Warning: time series deletion is not recommended to use on a regular basis. Each call to delete API could have a performance penalty. The API was provided for one-off operations to deleting malformed data or to satisfy GDPR compliance._
+
+[Delete API](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-delete-time-series) expects from user to specify [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors). So the first thing to do before the deletion is to verify whether the selector matches the correct series.
+
+To check that metrics are present in **VictoriaMetrics Cluster** run the following command:
+
+_Warning: response can return many metrics, so be careful with series selector._
+
+<div class="with-copy" markdown="1">
+
+```console
+curl -s 'http://vmselect:8481/select/0/prometheus/api/v1/series?match[]=process_cpu_cores_available' | jq
+```
+
+</div>
+
+The expected output:
+
+```json
+{
+  "status": "success",
+  "isPartial": false,
+  "data": [
+    {
+      "__name__": "process_cpu_cores_available",
+      "job": "vmagent",
+      "instance": "vmagent:8429"
+    },
+    {
+      "__name__": "process_cpu_cores_available",
+      "job": "vmalert",
+      "instance": "vmalert:8880"
+    },
+    {
+      "__name__": "process_cpu_cores_available",
+      "job": "vminsert",
+      "instance": "vminsert:8480"
+    },
+    {
+      "__name__": "process_cpu_cores_available",
+      "job": "vmselect",
+      "instance": "vmselect:8481"
+    },
+    {
+      "__name__": "process_cpu_cores_available",
+      "job": "vmstorage",
+      "instance": "vmstorage-1:8482"
+    },
+    {
+      "__name__": "process_cpu_cores_available",
+      "job": "vmstorage",
+      "instance": "vmstorage-2:8482"
+    }
+  ]
+}
+
+```
+
+When you're sure [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) is correct, send a POST request to [delete API](https://docs.victoriametrics.com/url-examples.html#apiv1admintsdbdelete_series) with [`match[]=<time-series-selector>`](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) argument. For example:
+
+<div class="with-copy" markdown="1">
+
+```console
+curl -s 'http://vmselect:8481/select/0/prometheus/api/v1/series?match[]=process_cpu_cores_available'
+```
+
+</div>
+
+If operation was successful, the deleted series will stop being [queryable](https://docs.victoriametrics.com/keyConcepts.html#query-data). Storage space for the deleted time series isn't freed instantly - it is freed during subsequent [background merges of data files](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282). The background merges may never occur for data from previous months, so storage space won't be freed for historical data. In this case [forced merge](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#forced-merge) may help freeing up storage space.
+
+To trigger [forced merge](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#forced-merge) on VictoriaMetrics Cluster run the following command:
+
+<div class="with-copy" markdown="1">
+
+```console
+curl -v -X POST http://vmstorage:8482/internal/force_merge
+```
+
+After the merge is complete, the data will be permanently deleted from the disk.
+
+## How to update metrics
+
+By default, VictoriaMetrics doesn't provide a mechanism for replacing or updating data. As a workaround, take the following actions:
+
+- [export time series to a file](https://docs.victoriametrics.com/url-examples.html#apiv1export);
+- change the values of time serie in the file and save it;
+- [delete time series from a database](https://docs.victoriametrics.com/url-examples.html#apiv1admintsdbdelete_series);
+- [import saved file to VictoriaMetrics](https://docs.victoriametrics.com/url-examples.html#apiv1import).
+
+### Export metrics
+
+For example, let's export metric for `node_memory_MemTotal_bytes` with labels `instance="node-exporter:9100"` and `job="hostname.com"`:
+
+<div class="with-copy" markdown="1">
+
+```console
+curl -X POST -g http://vmselect:8481/select/0/prometheus/api/v1/export -d 'match[]=node_memory_MemTotal_bytes{instance="node-exporter:9100", job="hostname.com"}' > data.jsonl
+```
+
+</div>
+
+To check that exported file contains time series we can use [cat](https://man7.org/linux/man-pages/man1/cat.1.html) and [jq](https://stedolan.github.io/jq/download/)
+
+<div class="with-copy" markdown="1">
+
+```console
+cat data.jsonl | jq
+```
+
+</div>
+
+The expected output will look like:
+
+```json
+{
+  "metric": {
+    "__name__": "node_memory_MemTotal_bytes",
+    "job": "hostname.com",
+    "instance": "node-exporter:9100"
+  },
+  "values": [
+    33604390912,
+    33604390912,
+    33604390912,
+    33604390912
+  ],
+  "timestamps": [
+    1656669031378,
+    1656669032378,
+    1656669033378,
+    1656669034378
+  ]
+}
+
+```
+
+In this example, we will replace the values of `node_memory_MemTotal_bytes` from `33604390912` to `17179869184` (from 32Gb to 16Gb) via [sed](https://linux.die.net/man/1/sed), but it can be done in any of the available ways.
+
+```console
+sed -i 's/33604390912/17179869184/g' data.jsonl
+```
+
+Let's check the changes in data.jsonl with `cat`:
+
+```console
+cat data.jsonl | jq
+```
+
+The expected output will be the next:
+
+```json
+{
+  "metric": {
+    "__name__": "node_memory_MemTotal_bytes",
+    "job": "hostname.com",
+    "instance": "node-exporter:9100"
+  },
+  "values": [
+    17179869184,
+    17179869184,
+    17179869184,
+    17179869184
+  ],
+  "timestamps": [
+    1656669031378,
+    1656669032378,
+    1656669033378,
+    1656669034378
+  ]
+}
+```
+
+### Delete metrics
+
+See [How-to-delete-metrics](https://docs.victoriametrics.com/guides/guide-delete-or-replace-metrics.html#how-to-delete-metrics) from the previous paragraph
+
+### Import metrics
+
+Victoriametrics supports a lot of [ingestion protocols](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-import-time-series-data) and we will use [import from JSON line format](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-import-data-in-json-line-format).
+
+The next command will import metrics from `data.jsonl` to VictoriaMetrics:
+
+<div class="with-copy" markdown="1">
+
+```console
+curl -v -X POST http://vminsert:8480/insert/0/prometheus/api/v1/import -T data.jsonl
+```
+</div>
+
+### Check imported metrics
+
+<div class="with-copy" markdown="1">
+
+```console
+curl -X POST -g http://vmselect:8481/select/0/prometheus/api/v1/export -d match[]=node_memory_MemTotal_bytes
+```
+
+</div>
+
+The expected output will look like:
+
+```json
+{
+  "metric": {
+    "__name__": "node_memory_MemTotal_bytes",
+    "job": "hostname.com",
+    "instance": "node-exporter:9100"
+  },
+  "values": [
+    17179869184,
+    17179869184,
+    17179869184,
+    17179869184
+  ],
+  "timestamps": [
+    1656669031378,
+    1656669032378,
+    1656669033378,
+    1656669034378
+  ]
+}
+```

docs/guides/guide-vmcluster-multiple-retention-setup.md

@@ -3,12 +3,16 @@
 
 **Objective**
 
-Setup Victoria Metrics TSDB with support of multiple retention periods within one installation.
+Setup Victoria Metrics Cluster with support of multiple retention periods within one installation.
 
 **Challenge**
 
-VictoriaMetrics instance (single node or vmstorage node) supports only one retention period.
+If you use [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise.html), then you can use
+[retention filters](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#retention-filters) for applying multiple retentions
+to distinct sets of time series and/or [tenants](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multitenancy).
 
+Community version of VictoriaMetrics supports only one retention period via [-retentionPeriod](https://docs.victoriametrics.com/#retention) command-line flag.
+The following solution allows supporting multiple retentions for community version of VictoriaMetrics.
 
 **Solution**
 
@@ -16,7 +20,7 @@ A multi-retention setup can be implemented by dividing a [victoriametrics cluste
 
 Example:
 Setup should handle 3 different retention groups 3months, 1year and 3 years.
-Solution contains 3 groups of vmstorages + vminserst and one group of vmselects. Routing is done by [vmagent](https://docs.victoriametrics.com/vmagent.html) and [relabeling configuration](https://docs.victoriametrics.com/vmagent.html#relabeling) 
+Solution contains 3 groups of vmstorages + vminserst and one group of vmselects. Routing is done by [vmagent](https://docs.victoriametrics.com/vmagent.html) and [relabeling configuration](https://docs.victoriametrics.com/vmagent.html#relabeling). The [-retentionPeriod](https://docs.victoriametrics.com/#retention) sets how long to keep the metrics. 
 
 The diagram below shows a proposed solution
 
@@ -25,11 +29,11 @@ The diagram below shows a proposed solution
 </p>
 
 **Implementation Details**
-  1. Groups of vminserts A know about only vmstorages A and this is explicitly specified in [-storageNode configuration](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#cluster-setup). 
-  2. Groups of vminserts B know about only vmstorages B and this is explicitly specified in `-storageNode` configuration. 
-  3. Groups of vminserts C know about only vmstorages A and this is explicitly specified in `-storageNode` configuration. 
-  4. Vmselect reads data from all vmstorage nodes.
-  5. Vmagent routes incoming metrics to the given set of `vminsert` nodes using relabeling rules specified at `-remoteWrite.urlRelabelConfig`. See [these docs](https://docs.victoriametrics.com/vmagent.html#relabeling).
+  1. Groups of vminserts A know about only vmstorages A and this is explicitly specified via `-storageNode` [configuration](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#cluster-setup). 
+  2. Groups of vminserts B know about only vmstorages B and this is explicitly specified via `-storageNode` [configuration](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#cluster-setup). 
+  3. Groups of vminserts C know about only vmstorages A and this is explicitly specified via `-storageNode` [configuration](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#cluster-setup). 
+  4. Vmselect reads data from all vmstorage nodes via `-storageNode` [configuration](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#cluster-setup). 
+  5. Vmagent routes incoming metrics to the given set of `vminsert` nodes using relabeling rules specified at `-remoteWrite.urlRelabelConfig` [configuration](https://docs.victoriametrics.com/vmagent.html#relabeling).
 
 **Multi-Tenant Setup**
 

docs/guides/k8s-monitoring-via-vm-cluster.md

@@ -89,7 +89,7 @@ vmstorage:
   podAnnotations:
       prometheus.io/scrape: "true"
       prometheus.io/port: "8482"
-EOF     
+EOF
 ```
 </div>
 

docs/guides/migrate-from-influx.md

@@ -130,7 +130,7 @@ for serving read queries. This API is used in various integrations such as
 by [VMUI](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#vmui) - a graphical User Interface for
 querying and visualizing metrics:
 
-{% include img.html href="migrate-from-influx-vmui.png" %}
+<img src="migrate-from-influx-vmui.png">
 
 See more about [how to query data in VictoriaMetrics](https://docs.victoriametrics.com/keyConcepts.html#query-data).
 
@@ -159,7 +159,7 @@ The data sample consists data points for a measurement `foo`
 and a field `bar` with additional tag `instance=localhost`. If we would like plot this data as a time series in Grafana
 it might have the following look:
 
-{% include img.html href="migrate-from-influx-data-sample-in-influx.png" %}
+<img src="migrate-from-influx-data-sample-in-influx.png">
 
 The query used for this panel is written in
 [InfluxQL](https://docs.influxdata.com/influxdb/v1.8/query_language/):
@@ -194,7 +194,7 @@ InfluxQL query might be translated to MetricsQL let's break it into components f
 In result, executing the `foo_bar{instance="localhost"}` MetricsQL expression with `step=1m` for the same set of data in
 Grafana will have the following form:
 
-{% include img.html href="migrate-from-influx-data-sample-in-vm.png" %}
+<img src="migrate-from-influx-data-sample-in-vm.png">
 
 Visualizations from both databases are a bit different - VictoriaMetrics shows some extra points
 filling the gaps in the graph. This behavior is described in more

docs/keyConcepts.md

@@ -93,7 +93,7 @@ So, the `counter` metric shows the number of observed events since the service s
 
 In programming, `counter` is a variable that you **increment** each time something happens.
 
-{% include img.html href="keyConcepts_counter.png" %}
+<img src="keyConcepts_counter.png">
 
 `vm_http_requests_total` is a typical example of a counter. The interpretation of a graph
 above is that time series `vm_http_requests_total{instance="localhost:8428", job="victoriametrics", path="api/v1/query_range"}`
@@ -119,7 +119,7 @@ by humans from other metric types.
 
 Gauge is used for measuring a value that can go up and down:
 
-{% include img.html href="keyConcepts_gauge.png" %}
+<img src="keyConcepts_gauge.png">
 
 The metric `process_resident_memory_anon_bytes` on the graph shows the memory usage of the application at every given time.
 It is changing frequently, going up and down showing how the process allocates and frees the memory.
@@ -219,7 +219,7 @@ Such a combination of `counter` metrics allows
 plotting [Heatmaps in Grafana](https://grafana.com/docs/grafana/latest/visualizations/heatmap/)
 and calculating [quantiles](https://prometheus.io/docs/practices/histograms/#quantiles):
 
-{% include img.html href="keyConcepts_histogram.png" %}
+<img src="keyConcepts_histogram.png">
 
 Grafana doesn't understand buckets with `vmrange` labels, so the [prometheus_buckets](https://docs.victoriametrics.com/MetricsQL.html#prometheus_buckets)
 function must be used for converting buckets with `vmrange` labels to buckets with `le` labels before building heatmaps in Grafana.
@@ -261,7 +261,7 @@ go_gc_duration_seconds_count 83
 
 The visualisation of summaries is pretty straightforward:
 
-{% include img.html href="keyConcepts_summary.png" %}
+<img src="keyConcepts_summary.png">
 
 Such an approach makes summaries easier to use but also puts significant limitations compared to [histograms](#histogram):
 
@@ -320,7 +320,7 @@ VictoriaMetrics supports both models used in modern monitoring applications: [pu
 
 Client regularly sends the collected metrics to the server in the push model:
 
-{% include img.html href="keyConcepts_push_model.png" %}
+<img src="keyConcepts_push_model.png">
 
 The client (application) decides when and where to send its metrics. VictoriaMetrics supports the following protocols
 for data ingestion (aka `push protocols`):
@@ -378,7 +378,7 @@ The cons of push protocol:
 Pull model is an approach popularized by [Prometheus](https://prometheus.io/), where the monitoring system decides when
 and where to pull metrics from:
 
-{% include img.html href="keyConcepts_pull_model.png" %}
+<img src="keyConcepts_pull_model.png">
 
 In pull model, the monitoring system needs to be aware of all the applications it needs to monitor. The metrics are
 scraped (pulled) from the known applications (aka `scrape targets`) via HTTP protocol on a regular basis (aka `scrape_interval`).
@@ -409,7 +409,7 @@ models for data collection. Many installations use exclusively one of these mode
 
 The most common approach for data collection is using both models:
 
-{% include img.html href="keyConcepts_data_collection.png" %}
+<img src="keyConcepts_data_collection.png">
 
 In this approach the additional component is used - [vmagent](https://docs.victoriametrics.com/vmagent.html). Vmagent is
 a lightweight agent whose main purpose is to collect, filter, relabel and deliver metrics to VictoriaMetrics.
@@ -424,7 +424,7 @@ installation for querying collected data.
 
 VictoriaMetrics components allow building more advanced topologies. For example, vmagents can push metrics from separate datacenters to the central VictoriaMetrics:
 
-{% include img.html href="keyConcepts_two_dcs.png" %}
+<img src="keyConcepts_two_dcs.png">
 
 VictoriaMetrics in this example may be either [single-node VictoriaMetrics](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html)
 or [VictoriaMetrics Cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html). Vmagent also allows
@@ -854,7 +854,7 @@ VictoriaMetrics has a built-in graphical User Interface for querying and visuali
 [VMUI](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#vmui).
 Open `http://victoriametrics:8428/vmui` page, type the query and see the results:
 
-{% include img.html href="keyConcepts_vmui.png" %}
+<img src="keyConcepts_vmui.png">
 
 VictoriaMetrics supports [Prometheus HTTP API](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#prometheus-querying-api-usage)
 which makes it possible to [query it with Grafana](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#grafana-setup)

docs/relabeling.md

@@ -249,7 +249,7 @@ See also [useful tips for target relabeling](#useful-tips-for-target-relabeling)
 Single-node VictoriaMetrics and [vmagent](https://docs.victoriametrics.com/vmagent.html) automatically add `instance` and `job` labels per each discovered target:
 
 * The `job` label is set to `job_name` value specified in the corresponding [scrape_config](https://docs.victoriametrics.com/sd_configs.html#scrape_configs).
-* The `instance` label is set to the final `__address__` label value after target-level relabeling.
+* The `instance` label is set to the `host:port` part of `__address__` label value after target-level relabeling.
   The `__address__` label value is automatically set to the most suitable value depending
   on the used [service discovery type](https://docs.victoriametrics.com/sd_configs.html#supported-service-discovery-configs).
   The `__address__` label can be overriden during relabeling - see [these docs](#how-to-modify-scrape-urls-in-targets).
@@ -284,8 +284,10 @@ URLs for scrape targets are composed of the following parts:
   just update the `__address__` label during relabeling to the needed value.
   The port part is optional. If it is missing, then it is automatically set either to `80` or `443` depending
   on the used scheme (`http` or `https`).
-  The final `__address__` label is automatically converted into `instance` label per each target unless the `instance`
+  The `host:port` part from the final `__address__` label is automatically set to `instance` label unless the `instance`
   label is explicitly set during relabeling.
+  The `__address__` label can contain the full scrape url, e.g. `http://host:port/metrics/path?query_args`.
+  In this case the `__scheme__` and `__metrics_path__` labels are ignored.
 * URL path (e.g. `/metrics`). This information is available during target relabeling in a special label - `__metrics_path__`.
   By default the `__metrics_path__` is set to `/metrics`. It can be overriden either by specifying the `metrics_path`
   option at [scrape_config](https://docs.victoriametrics.com/sd_configs.html#scrape_configs)

docs/sd_configs.md

@@ -987,6 +987,8 @@ scrape_configs:
     #
     # Alternatively the scheme and path can be changed via `relabel_configs` section at `scrape_config` level.
     # See https://docs.victoriametrics.com/vmagent.html#relabeling .
+    #
+    # It is also possible specifying full target urls here, e.g. "http://host:port/metrics/path?query_args"
   - targets:
     - "vmsingle1:8428"
     - "vmsingleN:8428"
@@ -1087,6 +1089,9 @@ scrape_configs:
   # Example values:
   # - "30s" - 30 seconds
   # - "2m" - 2 minutes
+  # The scrape_interval can be set on a per-target basis by specifying `__scrape_interval__`
+  # label during target relabeling phase.
+  # See https://docs.victoriametrics.com/vmagent.html#relabeling
   # scrape_interval: <duration>
 
   # scrape_timeout is an optional timeout when scraping the targets.
@@ -1098,6 +1103,9 @@ scrape_configs:
   # - "30s" - 30 seconds
   # - "2m" - 2 minutes
   # The `scrape_timeout` cannot exceed the `scrape_interval`.
+  # The scrape_timeout can be set on a per-target basis by specifying `__scrape_timeout__`
+  # label during target relabeling phase.
+  # See https://docs.victoriametrics.com/vmagent.html#relabeling
   # scrape_timeout: <duration>
 
   # metrics_path is the path to fetch metrics from targets.
@@ -1186,8 +1194,11 @@ scrape_configs:
   # disable_keepalive: <boolean>
 
   # stream_parse allows enabling stream parsing mode when scraping targets.
-  # By default stram parsing mode is disabled for targets which return up to a few thosands samples.
+  # By default stream parsing mode is disabled for targets which return up to a few thosands samples.
   # See https://docs.victoriametrics.com/vmagent.html#stream-parsing-mode .
+  # The stream_parse can be set on a per-target basis by specifying `__stream_parse__`
+  # label during target relabeling phase.
+  # See https://docs.victoriametrics.com/vmagent.html#relabeling
   # stream_parse: <boolean>
 
   # scrape_align_interval allows aligning scrapes to the given interval.
@@ -1208,8 +1219,16 @@ scrape_configs:
   # a single target can expose during all the scrapes.
   # By default there is no limit on the number of exposed series.
   # See https://docs.victoriametrics.com/vmagent.html#cardinality-limiter .
+  # The series_limit can be set on a per-target basis by specifying `__series_limit__`
+  # label during target relabeling phase.
+  # See https://docs.victoriametrics.com/vmagent.html#relabeling
   # series_limit: ...
 
+  # no_stale_markers allows disabling staleness tracking.
+  # By default staleness tracking is enabled for all the discovered scrape targets.
+  # See https://docs.victoriametrics.com/vmagent.html#prometheus-staleness-markers
+  # no_stale_markers: <boolean>
+
   # Additional HTTP client options for target scraping can be specified here.
   # See https://docs.victoriametrics.com/sd_configs.html#http-api-client-options
 ```

docs/vmagent.md

@@ -386,7 +386,7 @@ Extra labels can be added to metrics collected by `vmagent` via the following me
   ```
 
   `vmagent` sets `scrape_series_added` to zero when it runs with `-promscrape.noStaleMarkers` command-line option
-  (e.g. when [staleness markers](#prometheus-staleness-markers) are disabled).
+  or when it scrapes target with `no_stale_markers: true` option, e.g. when [staleness markers](#prometheus-staleness-markers) are disabled.
 
 * `scrape_series_limit` - the limit on the number of unique time series the given target can expose according to [these docs](#cardinality-limiter).
   This metric is exposed only if the series limit is set.
@@ -608,9 +608,13 @@ Additionally, the `action: graphite` relabeling rules usually work much faster t
 * If the scrape target is removed from the list of targets, then stale markers are sent for all the metrics scraped from this target.
 
 Prometheus staleness markers' tracking needs additional memory, since it must store the previous response body per each scrape target
-in order to compare it to the current response body. The memory usage may be reduced by passing `-promscrape.noStaleMarkers`
-command-line flag to `vmagent`. This disables staleness tracking. This also disables tracking the number of new time series
-per each scrape with the auto-generated `scrape_series_added` metric. See [these docs](#automatically-generated-metrics) for details.
+in order to compare it to the current response body. The memory usage may be reduced by disabling staleness tracking in the following ways:
+
+* By passing `-promscrape.noStaleMarkers` command-line flag to `vmagent`. This disables staleness tracking across all the targets.
+* By specifying `no_stale_markers: true` option in the [scrape_config](https://docs.victoriametrics.com/sd_configs.html#scrape_configs) for the corresponding target.
+
+When staleness tracking is disabled, then `vmagent` doesn't track the number of new time series per each scrape,
+e.g. it sets `scrape_series_added` metric to zero. See [these docs](#automatically-generated-metrics) for details.
 
 ## Stream parsing mode
 
@@ -908,7 +912,7 @@ See also [troubleshooting docs](https://docs.victoriametrics.com/Troubleshooting
 
 ## Kafka integration
 
-[Enterprise version](https://victoriametrics.com/products/enterprise/) of `vmagent` can read and write metrics from / to Kafka:
+[Enterprise version](https://docs.victoriametrics.com/enterprise.html) of `vmagent` can read and write metrics from / to Kafka:
 
 * [Reading metrics from Kafka](#reading-metrics-from-kafka)
 * [Writing metrics to Kafka](#writing-metrics-to-kafka)
@@ -918,7 +922,7 @@ in `vmutils-...-enteprise.tar.gz` archives and in [docker images](https://hub.do
 
 ### Reading metrics from Kafka
 
-[Enterprise version](https://victoriametrics.com/products/enterprise/) of `vmagent` can read metrics in various formats from Kafka messages.
+[Enterprise version](https://docs.victoriametrics.com/enterprise.html) of `vmagent` can read metrics in various formats from Kafka messages.
 These formats can be configured with `-kafka.consumer.topic.defaultFormat` or `-kafka.consumer.topic.format` command-line options. The following formats are supported:
 
 * `promremotewrite` - [Prometheus remote_write](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write).
@@ -960,7 +964,7 @@ data_format = "influx"
 
 #### Command-line flags for Kafka consumer
 
-These command-line flags are available only in [enterprise](https://victoriametrics.com/products/enterprise/) version of `vmagent`,
+These command-line flags are available only in [enterprise](https://docs.victoriametrics.com/enterprise.html) version of `vmagent`,
 which can be downloaded for evaluation from [releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) page
 (see `vmutils-...-enteprise.tar.gz` archives) and from [docker images](https://hub.docker.com/r/victoriametrics/vmagent/tags) with tags containing `enterprise` suffix.
 
@@ -995,7 +999,7 @@ which can be downloaded for evaluation from [releases](https://github.com/Victor
 
 ### Writing metrics to Kafka
 
-[Enterprise version](https://victoriametrics.com/products/enterprise/) of `vmagent` writes data to Kafka with `at-least-once`
+[Enterprise version](https://docs.victoriametrics.com/enterprise.html) of `vmagent` writes data to Kafka with `at-least-once`
 semantics if `-remoteWrite.url` contains e.g. Kafka url. For example, if `vmagent` is started with `-remoteWrite.url=kafka://localhost:9092/?topic=prom-rw`,
 then it would send Prometheus remote_write messages to Kafka bootstrap server at `localhost:9092` with the topic `prom-rw`.
 These messages can be read later from Kafka by another `vmagent` - see [these docs](#reading-metrics-from-kafka) for details.
@@ -1027,7 +1031,7 @@ It may be needed to build `vmagent` from source code when developing or testing
 
 ### Development build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.2.
 2. Run `make vmagent` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds the `vmagent` binary and puts it into the `bin` folder.
 
@@ -1056,7 +1060,7 @@ ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://b
 
 ### Development ARM build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.2.
 2. Run `make vmagent-linux-arm` or `make vmagent-linux-arm64` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics)
    It builds `vmagent-linux-arm` or `vmagent-linux-arm64` binary respectively and puts it into the `bin` folder.
 

docs/vmalert.md

@@ -217,13 +217,51 @@ The following variables are available in templating:
 | $labels or .Labels                 | The list of labels of the current alert. Use as ".Labels.<label_name>".                                   | {% raw %}Too high number of connections for {{ .Labels.instance }}{% endraw %}                                                                                         |
 | $alertID or .AlertID               | The current alert's ID generated by vmalert.                                                              | {% raw %}Link: vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}{% endraw %}                                                                                   |
 | $groupID or .GroupID               | The current alert's group ID generated by vmalert.                                                        | {% raw %}Link: vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}{% endraw %}                                                                                   |
-| $expr or .Expr                     | Alert's expression. Can be used for generating links to Grafana or other systems.                         | {% raw %}/api/v1/query?query={{ $expr&#124;quotesEscape&#124;queryEscape }}{% endraw %}                                                                                |
+| $expr or .Expr                     | Alert's expression. Can be used for generating links to Grafana or other systems.                         | {% raw %}/api/v1/query?query={{ $expr&#124;queryEscape }}{% endraw %}                                                                                |
 | $externalLabels or .ExternalLabels | List of labels configured via `-external.label` command-line flag.                                        | {% raw %}Issues with {{ $labels.instance }} (datacenter-{{ $externalLabels.dc }}){% endraw %}                                                                          |
 | $externalURL or .ExternalURL       | URL configured via `-external.url` command-line flag. Used for cases when vmalert is hidden behind proxy. | {% raw %}Visit {{ $externalURL }} for more details{% endraw %}                                                                                                         |
 
-Additionally, `vmalert` provides some extra templating functions
-listed [here](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmalert/templates/template.go)
-and [reusable templates](#reusable-templates).
+Additionally, `vmalert` provides some extra templating functions listed [here](#template-functions) and [reusable templates](#reusable-templates).
+
+#### Template functions
+
+`vmalert` provides the following template functions, which can be used during [templating](#templating):
+
+- `args arg0 ... argN` - converts the input args into a map with `arg0`, ..., `argN` keys.
+- `externalURL` - returns the value of `-external.url` command-line flag.
+- `first` - returns the first result from the input query results returned by `query` function.
+- `htmlEscape` - escapes special chars in input string, so it can be safely embedded as a plaintext into HTML.
+- `humanize` - converts the input number into human-readable format by adding [metric prefixes](https://en.wikipedia.org/wiki/Metric_prefix).
+  For example, `100000` is converted into `100K`.
+- `humanize1024` - converts the input number into human-readable format with 1024 base.
+  For example, `1024` is converted into 1ki`.
+- `humanizeDuration` - converts the input number in seconds into human-readable duration.
+- `humanizePercentage` - converts the input number to percentage. For example, `0.123` is converted into `12.3%`.
+- `humanizeTimestamp` - converts the input unix timestamp into human-readable time.
+- `jsonEscape` - JSON-encodes the input string.
+- `label name` - returns the value of the label with the given `name` from the input query result.
+- `match regex` - matches the input string against the provided `regex`.
+- `parseDuration` - parses the input string into duration in seconds. For example, `1h` is parsed into `3600`.
+- `parseDurationTime` - parses the input string into [time.Duration](https://pkg.go.dev/time#Duration).
+- `pathEscape` - escapes the input string, so it can be safely put inside path part of URL.
+- `pathPrefix` - returns the path part of the `-external.url` command-line flag.
+- `query` - executes the [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html) query against `-datasource.url` and returns the query result.
+  For example, {% raw %}`{{ query "sort_desc(process_resident_memory_bytes)" | first | value }}`{% endraw %} executes the `sort_desc(process_resident_memory_bytes)`
+  query at `-datasource.url` and returns the first result.
+- `queryEscape` - escapes the input string, so it can be safely put inside [query arg](https://en.wikipedia.org/wiki/Percent-encoding) part of URL.
+- `quotesEscape` - escapes the input string, so it can be safely embedded into JSON string.
+- `reReplaceAll regex repl` - replaces all the occurences of the `regex` in input string with the `repl`.
+- `safeHtml` - marks the input string as safe to use in HTML context without the need to html-escape it.
+- `sortByLabel name` - sorts the input query results by the label with the given `name`.
+- `stripDomain` - leaves the first part of the domain. For example, `foo.bar.baz` is converted to `foo`.
+  The port part is left in the output string. E.g. `foo.bar:1234` is converted into `foo:1234`.
+- `stripPort` - strips `port` part from `host:port` input string.
+- `strvalue` - returns the metric name from the input query result.
+- `title` - converts the first letters of every input word to uppercase.
+- `toLower` - converts all the chars in the input string to lowercase.
+- `toTime` - converts the input unix timestamp to [time.Time](https://pkg.go.dev/time#Time).
+- `toUpper` - converts all the chars in the input string to uppercase.
+- `value` - returns the numeric value from the input query result.
 
 #### Reusable templates
 
@@ -319,7 +357,7 @@ There are the following approaches exist for alerting and recording rules across
   rules to `AccountID=123`.
 
 * To specify `tenant` parameter per each alerting and recording group if
-  [enterprise version of vmalert](https://victoriametrics.com/products/enterprise/) is used
+  [enterprise version of vmalert](https://docs.victoriametrics.com/enterprise.html) is used
   with `-clusterMode` command-line flag. For example:
 
 ```yaml
@@ -669,9 +707,10 @@ Try the following recommendations in such cases:
 are delivered to the datasource;
 * If you know in advance, that data in datasource is delayed - try changing vmalert's `-datasource.lookback`
 command-line flag to add a time shift for evaluations;
-* If time intervals between datapoints in datasource are irregular - try changing vmalert's `-datasource.queryStep`
-command-line flag to specify how far search query can lookback for the recent datapoint. By default, this value
-is equal to group's evaluation interval.
+* If time intervals between datapoints in datasource are irregular or `>=5min` - try changing vmalert's
+`-datasource.queryStep` command-line flag to specify how far search query can lookback for the recent datapoint. 
+The recommendation is to have the step at least two times bigger than `scrape_interval`, since
+there are no guarantees that scrape will not fail.
 
 Sometimes, it is not clear why some specific alert fired or didn't fire. It is very important to remember, that
 alerts with `for: 0` fire immediately when their expression becomes true. And alerts with `for > 0` will fire only
@@ -818,7 +857,7 @@ The shortlist of configuration flags is the following:
   -evaluationInterval duration
      How often to evaluate the rules (default 1m0s)
   -external.alert.source string
-     External Alert Source allows to override the Source link for alerts sent to AlertManager for cases where you want to build a custom link to Grafana, Prometheus or any other service. Supports templating - see https://docs.victoriametrics.com/vmalert.html#templating . For example, link to Grafana: -external.alert.source='explore?orgId=1&left=[\"now-1h\",\"now\",\"VictoriaMetrics\",{\"expr\": \"{{$expr|quotesEscape|crlfEscape|queryEscape}}\"},{\"mode\":\"Metrics\"},{\"ui\":[true,true,true,\"none\"]}]' . If empty 'vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}' is used
+     External Alert Source allows to override the Source link for alerts sent to AlertManager for cases where you want to build a custom link to Grafana, Prometheus or any other service. Supports templating - see https://docs.victoriametrics.com/vmalert.html#templating . For example, link to Grafana: -external.alert.source='explore?orgId=1&left=["now-1h","now","VictoriaMetrics",{"expr":{{$expr|jsonEscape|queryEscape}} },{"mode":"Metrics"},{"ui":[true,true,true,"none"]}]' . If empty 'vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}' is used
      If empty 'vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}' is used.
   -external.label array
      Optional label in the form 'Name=value' to add to all generated recording rules and alerts. Pass multiple -label flags in order to add multiple label sets.
@@ -1279,7 +1318,7 @@ spec:
 
 ### Development build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.2.
 2. Run `make vmalert` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `vmalert` binary and puts it into the `bin` folder.
 
@@ -1295,7 +1334,7 @@ ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://b
 
 ### Development ARM build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.2.
 2. Run `make vmalert-linux-arm` or `make vmalert-linux-arm64` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `vmalert-linux-arm` or `vmalert-linux-arm64` binary respectively and puts it into the `bin` folder.
 

docs/vmanomaly.md

@@ -4,7 +4,7 @@ sort: 11
 
 # vmanomaly
 
-**_vmanomaly is a part of [enterprise package](https://victoriametrics.com/products/enterprise/).
+**_vmanomaly is a part of [enterprise package](https://docs.victoriametrics.com/enterprise.html).
 Please [contact us](https://victoriametrics.com/contact-us/) to find out more._**
 
 ## About