vendor: update github.com/klauspost/compress from v1.15.12 to v1.15.13

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3427

.github/ISSUE_TEMPLATE/bug_report.md

@@ -11,7 +11,7 @@ A clear and concise description of what the bug is.
 It would be great to [upgrade](https://docs.victoriametrics.com/#how-to-upgrade) 
 to [the latest available release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)
 and verify whether the bug is reproducible there.
-It's also recommended to read the [troubleshooting docs](https://docs.victoriametrics.com/#troubleshooting).
+It's also recommended to read the [troubleshooting docs](https://docs.victoriametrics.com/Troubleshooting.html).
 
 **To Reproduce**
 Steps to reproduce the behavior.

.github/workflows/check-licenses.yml

@@ -17,7 +17,7 @@ jobs:
       - name: Setup Go
         uses: actions/setup-go@main
         with:
-          go-version: 1.19.1
+          go-version: 1.19.4
         id: go
       - name: Code checkout
         uses: actions/checkout@master

.github/workflows/codeql-analysis.yml

@@ -40,6 +40,12 @@ jobs:
     - name: Checkout repository
       uses: actions/checkout@v3
 
+    - name: Set up Go
+      uses: actions/setup-go@v2
+      with:
+        go-version: 1.19
+      if: ${{ matrix.language == 'go' }}
+
     # Initializes the CodeQL tools for scanning.
     - name: Initialize CodeQL
       uses: github/codeql-action/init@v2

.github/workflows/main.yml

@@ -19,7 +19,7 @@ jobs:
       - name: Setup Go
         uses: actions/setup-go@main
         with:
-          go-version: 1.19.1
+          go-version: 1.19.4
         id: go
       - name: Code checkout
         uses: actions/checkout@master

Makefile

@@ -2,7 +2,7 @@ PKG_PREFIX := github.com/VictoriaMetrics/VictoriaMetrics
 
 DATEINFO_TAG ?= $(shell date -u +'%Y%m%d-%H%M%S')
 BUILDINFO_TAG ?= $(shell echo $$(git describe --long --all | tr '/' '-')$$( \
-	      git diff-index --quiet HEAD -- || echo '-dirty-'$$(git diff-index -u HEAD | openssl sha1 | cut -c 10-17)))
+	      git diff-index --quiet HEAD -- || echo '-dirty-'$$(git diff-index -u HEAD | openssl sha1 | cut -d' ' -f2 | cut -c 1-8)))
 
 PKG_TAG ?= $(shell git tag -l --points-at HEAD)
 ifeq ($(PKG_TAG),)

README.md

@@ -24,12 +24,14 @@ Learn more about [key concepts](https://docs.victoriametrics.com/keyConcepts.htm
 [quick start guide](https://docs.victoriametrics.com/Quick-Start.html) for a better experience.
 
 [Contact us](mailto:info@victoriametrics.com) if you need enterprise support for VictoriaMetrics. 
-See [features available in enterprise package](https://victoriametrics.com/products/enterprise/).
+See [features available in enterprise package](https://docs.victoriametrics.com/enterprise.html).
 Enterprise binaries can be downloaded and evaluated for free 
 from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
 
 VictoriaMetrics is developed at a fast pace, so it is recommended periodically checking the [CHANGELOG](https://docs.victoriametrics.com/CHANGELOG.html) and performing [regular upgrades](#how-to-upgrade-victoriametrics).
 
+VictoriaMetrics has achieved security certifications for Database Software Development and Software-Based Monitoring Services. We apply strict security measures in everything we do. See our [Security page](https://victoriametrics.com/security/) for more details.
+
 ## Prominent features
 
 VictoriaMetrics has the following prominent features:
@@ -65,7 +67,7 @@ VictoriaMetrics has the following prominent features:
   * [DataDog agent or DogStatsD](#how-to-send-data-from-datadog-agent).
 * It supports metrics [relabeling](#relabeling).
 * It can deal with [high cardinality issues](https://docs.victoriametrics.com/FAQ.html#what-is-high-cardinality) and [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate) issues via [series limiter](#cardinality-limiter).
-* It ideally works with big amounts of time series data from APM, Kubernetes, IoT sensors, connected cars, industrial telemetry, financial data and various [Enterprise workloads](https://victoriametrics.com/products/enterprise/).
+* It ideally works with big amounts of time series data from APM, Kubernetes, IoT sensors, connected cars, industrial telemetry, financial data and various [Enterprise workloads](https://docs.victoriametrics.com/enterprise.html).
 * It has open source [cluster version](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/cluster).
 * It can store data on [NFS-based storages](https://en.wikipedia.org/wiki/Network_File_System) such as [Amazon EFS](https://aws.amazon.com/efs/) and [Google Filestore](https://cloud.google.com/filestore).
 
@@ -132,7 +134,15 @@ VictoriaMetrics is developed at a fast pace, so it is recommended periodically c
 
 ### Environment variables
 
-Each flag value can be set via environment variables according to these rules:
+All the VictoriaMetrics components allow referring environment variables in command-line flags via `%{ENV_VAR}` syntax.
+For example, `-metricsAuthKey=%{METRICS_AUTH_KEY}` is automatically expanded to `-metricsAuthKey=top-secret`
+if `METRICS_AUTH_KEY=top-secret` environment variable exists at VictoriaMetrics startup.
+This expansion is performed by VictoriaMetrics itself.
+
+VictoriaMetrics recursively expands `%{ENV_VAR}` references in environment variables on startup.
+For example, `FOO=%{BAR}` environment variable is expanded to `FOO=abc` if `BAR=a%{BAZ}` and `BAZ=bc`.
+
+Additionally, all the VictoriaMetrics components allow setting flag values via environment variables according to these rules:
 
 * The `-envflag.enable` flag must be set.
 * Each `.` char in flag name must be substituted with `_` (for example `-insert.maxQueueDuration <duration>` will translate to `insert_maxQueueDuration=<duration>`).
@@ -265,7 +275,7 @@ It also provides the following features:
 - [query tracer](#query-tracing)
 - [top queries explorer](#top-queries)
 
-Graphs in vmui support scrolling and zooming:
+Graphs in `vmui` support scrolling and zooming:
 
 * Select the needed time range on the graph in order to zoom in into the selected time range. Hold `ctrl` (or `cmd` on MacOS) and scroll down in order to zoom out.
 * Hold `ctrl` (or `cmd` on MacOS) and scroll up in order to zoom in the area under cursor.
@@ -277,9 +287,14 @@ Multi-line queries can be entered by pressing `Shift-Enter` in query input field
 
 When querying the [backfilled data](https://docs.victoriametrics.com/#backfilling) or during [query troubleshooting](https://docs.victoriametrics.com/Troubleshooting.html#unexpected-query-results), it may be useful disabling response cache by clicking `Disable cache` checkbox.
 
-VMUI automatically adjusts the interval between datapoints on the graph depending on the horizontal resolution and on the selected time range. The step value can be customized by clickhing `Override step value` checkbox.
+VMUI automatically adjusts the interval between datapoints on the graph depending on the horizontal resolution and on the selected time range. The step value can be customized by changing `Step value` input.
 
-VMUI allows investigating correlations between multiple queries on the same graph. Just click `Add Query` button, enter an additional query in the newly appeared input field and press `Ctrl+Enter`. Results for all the queries should be displayed simultaneously on the same graph.
+VMUI allows investigating correlations between multiple queries on the same graph. Just click `Add Query` button,
+enter an additional query in the newly appeared input field and press `Enter`.
+Results for all the queries are displayed simultaneously on the same graph.
+Graphs for a particular query can be temporarily hidden by clicking the `eye` icon on the right side of the input field.
+When the `eye` icon is clicked while holding the `ctrl` key, then query results for the rest of queries become hidden
+except of the current query results.
 
 See the [example VMUI at VictoriaMetrics playground](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/graph/?g0.expr=100%20*%20sum(rate(process_cpu_seconds_total))%20by%20(job)&g0.range_input=1d).
 
@@ -321,7 +336,7 @@ VictoriaMetrics is configured via command-line flags, so it must be restarted wh
 * Wait until the process stops. This can take a few seconds.
 * Start VictoriaMetrics with the new command-line flags.
 
-Prometheus doesn't drop data during VictoriaMetrics restart. See [this article](https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/) for details. The same applies alos to [vmagent](https://docs.victoriametrics.com/vmagent.html).
+Prometheus doesn't drop data during VictoriaMetrics restart. See [this article](https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/) for details. The same applies also to [vmagent](https://docs.victoriametrics.com/vmagent.html).
 
 ## How to scrape Prometheus exporters such as [node-exporter](https://github.com/prometheus/node_exporter)
 
@@ -731,7 +746,7 @@ VictoriaMetrics supports `__graphite__` pseudo-label for filtering time series w
 
 ### Graphite Render API usage
 
-[VictoriaMetrics Enterprise](https://victoriametrics.com/products/enterprise/) supports [Graphite Render API](https://graphite.readthedocs.io/en/stable/render_api.html) subset
+[VictoriaMetrics Enterprise](https://docs.victoriametrics.com/enterprise.html) supports [Graphite Render API](https://graphite.readthedocs.io/en/stable/render_api.html) subset
 at `/render` endpoint, which is used by [Graphite datasource in Grafana](https://grafana.com/docs/grafana/latest/datasources/graphite/).
 When configuring Graphite datasource in Grafana, the `Storage-Step` http request header must be set to a step between Graphite data points stored in VictoriaMetrics. For example, `Storage-Step: 10s` would mean 10 seconds distance between Graphite datapoints stored in VictoriaMetrics.
 Enterprise binaries can be downloaded and evaluated for free from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
@@ -772,7 +787,7 @@ to your needs or when testing bugfixes.
 
 ### Development build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.
 2. Run `make victoria-metrics` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `victoria-metrics` binary and puts it into the `bin` folder.
 
@@ -788,7 +803,7 @@ ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://b
 
 ### Development ARM build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.
 2. Run `make victoria-metrics-linux-arm` or `make victoria-metrics-linux-arm64` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `victoria-metrics-linux-arm` or `victoria-metrics-linux-arm64` binary respectively and puts it into the `bin` folder.
 
@@ -802,7 +817,7 @@ ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://b
 
 `Pure Go` mode builds only Go code without [cgo](https://golang.org/cmd/cgo/) dependencies.
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.
 2. Run `make victoria-metrics-pure` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `victoria-metrics-pure` binary and puts it into the `bin` folder.
 
@@ -864,8 +879,9 @@ Steps for restoring from a snapshot:
 
 Send a request to `http://<victoriametrics-addr>:8428/api/v1/admin/tsdb/delete_series?match[]=<timeseries_selector_for_delete>`,
 where `<timeseries_selector_for_delete>` may contain any [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
-for metrics to delete. After that all the time series matching the given selector are deleted. Storage space for
-the deleted time series isn't freed instantly - it is freed during subsequent [background merges of data files](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282).
+for metrics to delete. Delete API doesn't support the deletion of specific time ranges, the series can only be deleted completely. 
+Storage space for the deleted time series isn't freed instantly - it is freed during subsequent
+[background merges of data files](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282).
 
 Note that background merges may never occur for data from previous months, so storage space won't be freed for historical data.
 In this case [forced merge](#forced-merge) may help freeing up storage space.
@@ -1034,7 +1050,8 @@ Time series data can be imported into VictoriaMetrics via any supported data ing
 * `/api/v1/import/native` for importing data obtained from [/api/v1/export/native](#how-to-export-data-in-native-format).
   See [these docs](#how-to-import-data-in-native-format) for details.
 * `/api/v1/import/csv` for importing arbitrary CSV data. See [these docs](#how-to-import-csv-data) for details.
-* `/api/v1/import/prometheus` for importing data in Prometheus exposition format. See [these docs](#how-to-import-data-in-prometheus-exposition-format) for details.
+* `/api/v1/import/prometheus` for importing data in Prometheus exposition format and in [Pushgateway format](https://github.com/prometheus/pushgateway#url).
+  See [these docs](#how-to-import-data-in-prometheus-exposition-format) for details.
 
 ### How to import data in JSON line format
 
@@ -1139,9 +1156,11 @@ Note that it could be required to flush response cache after importing historica
 
 ### How to import data in Prometheus exposition format
 
-VictoriaMetrics accepts data in [Prometheus exposition format](https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#text-based-format)
-and in [OpenMetrics format](https://github.com/OpenObservability/OpenMetrics/blob/master/specification/OpenMetrics.md)
-via `/api/v1/import/prometheus` path. For example, the following line imports a single line in Prometheus exposition format into VictoriaMetrics:
+VictoriaMetrics accepts data in [Prometheus exposition format](https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#text-based-format),
+in [OpenMetrics format](https://github.com/OpenObservability/OpenMetrics/blob/master/specification/OpenMetrics.md)
+and in [Pushgateway format](https://github.com/prometheus/pushgateway#url) via `/api/v1/import/prometheus` path.
+
+For example, the following command imports a single line in Prometheus exposition format into VictoriaMetrics:
 
 <div class="with-copy" markdown="1">
 
@@ -1167,6 +1186,16 @@ It should return something like the following:
 {"metric":{"__name__":"foo","bar":"baz"},"values":[123],"timestamps":[1594370496905]}
 ```
 
+The following command imports a single metric via [Pushgateway format](https://github.com/prometheus/pushgateway#url) with `{job="my_app",instance="host123"}` labels:
+
+<div class="with-copy" markdown="1">
+
+```console
+curl -d 'metric{label="abc"} 123' -X POST 'http://localhost:8428/api/v1/import/prometheus/metrics/job/my_app/instance/host123'
+```
+
+</div>
+
 Pass `Content-Encoding: gzip` HTTP request header to `/api/v1/import/prometheus` for importing gzipped data:
 
 <div class="with-copy" markdown="1">
@@ -1178,8 +1207,8 @@ curl -X POST -H 'Content-Encoding: gzip' http://destination-victoriametrics:8428
 
 </div>
 
-Extra labels may be added to all the imported metrics by passing `extra_label=name=value` query args.
-For example, `/api/v1/import/prometheus?extra_label=foo=bar` would add `{foo="bar"}` label to all the imported metrics.
+Extra labels may be added to all the imported metrics either via [Pushgateway format](https://github.com/prometheus/pushgateway#url)
+or by passing `extra_label=name=value` query args. For example, `/api/v1/import/prometheus?extra_label=foo=bar` would add `{foo="bar"}` label to all the imported metrics.
 
 If timestamp is missing in `<metric> <value> <timestamp>` Prometheus exposition format line, then the current timestamp is used during data ingestion.
 It can be overridden by passing unix timestamp in *milliseconds* via `timestamp` query arg. For example, `/api/v1/import/prometheus?timestamp=1594370496905`.
@@ -1216,7 +1245,11 @@ Example contents for `-relabelConfig` file:
   regex: true
 ```
 
-VictoriaMetrics provides additional relabeling features such as Graphite-style relabeling. See [these docs](https://docs.victoriametrics.com/vmagent.html#relabeling) for more details.
+VictoriaMetrics provides additional relabeling features such as Graphite-style relabeling.
+See [these docs](https://docs.victoriametrics.com/vmagent.html#relabeling) for more details.
+
+The relabeling can be debugged at `http://victoriametrics:8428/metric-relabel-debug` page.
+See [these docs](https://docs.victoriametrics.com/vmagent.html#relabel-debug) for more details.
 
 
 ## Federation
@@ -1263,10 +1296,11 @@ See also [resource usage limits docs](#resource-usage-limits).
 
 By default VictoriaMetrics is tuned for an optimal resource usage under typical workloads. Some workloads may need fine-grained resource usage limits. In these cases the following command-line flags may be useful:
 
-- `-memory.allowedPercent` and `-search.allowedBytes` limit the amounts of memory, which may be used for various internal caches at VictoriaMetrics. Note that VictoriaMetrics may use more memory, since these flags don't limit additional memory, which may be needed on a per-query basis.
+- `-memory.allowedPercent` and `-memory.allowedBytes` limit the amounts of memory, which may be used for various internal caches at VictoriaMetrics. Note that VictoriaMetrics may use more memory, since these flags don't limit additional memory, which may be needed on a per-query basis.
+- `-search.maxMemoryPerQuery` limits the amounts of memory, which can be used for processing a single query. Queries, which need more memory, are rejected. Heavy queries, which select big number of time series, may exceed the per-query memory limit by a small percent. The total memory limit for concurrently executed queries can be estimated as `-search.maxMemoryPerQuery` multiplied by `-search.maxConcurrentRequests`.
 - `-search.maxUniqueTimeseries` limits the number of unique time series a single query can find and process. VictoriaMetrics keeps in memory some metainformation about the time series located by each query and spends some CPU time for processing the found time series. This means that the maximum memory usage and CPU usage a single query can use is proportional to `-search.maxUniqueTimeseries`.
 - `-search.maxQueryDuration` limits the duration of a single query. If the query takes longer than the given duration, then it is canceled. This allows saving CPU and RAM when executing unexpected heavy queries.
-- `-search.maxConcurrentRequests` limits the number of concurrent requests VictoriaMetrics can process. Bigger number of concurrent requests usually means bigger memory usage. For example, if a single query needs 100 MiB of additional memory during its execution, then 100 concurrent queries may need `100 * 100 MiB = 10 GiB` of additional memory. So it is better to limit the number of concurrent queries, while suspending additional incoming queries if the concurrency limit is reached. VictoriaMetrics provides `-search.maxQueueDuration` command-line flag for limiting the max wait time for suspended queries.
+- `-search.maxConcurrentRequests` limits the number of concurrent requests VictoriaMetrics can process. Bigger number of concurrent requests usually means bigger memory usage. For example, if a single query needs 100 MiB of additional memory during its execution, then 100 concurrent queries may need `100 * 100 MiB = 10 GiB` of additional memory. So it is better to limit the number of concurrent queries, while suspending additional incoming queries if the concurrency limit is reached. VictoriaMetrics provides `-search.maxQueueDuration` command-line flag for limiting the max wait time for suspended queries. See also `-search.maxMemoryPerQuery` command-line flag.
 - `-search.maxSamplesPerSeries` limits the number of raw samples the query can process per each time series. VictoriaMetrics sequentially processes raw samples per each found time series during the query. It unpacks raw samples on the selected time range per each time series into memory and then applies the given [rollup function](https://docs.victoriametrics.com/MetricsQL.html#rollup-functions). The `-search.maxSamplesPerSeries` command-line flag allows limiting memory usage in the case when the query is executed on a time range, which contains hundreds of millions of raw samples per each located time series.
 - `-search.maxSamplesPerQuery` limits the number of raw samples a single query can process. This allows limiting CPU usage for heavy queries.
 - `-search.maxPointsPerTimeseries` limits the number of calculated points, which can be returned per each matching time series from [range query](https://docs.victoriametrics.com/keyConcepts.html#range-query).
@@ -1321,7 +1355,12 @@ with the enabled de-duplication. See [this section](#deduplication) for details.
 
 ## Deduplication
 
-VictoriaMetrics leaves a single raw sample with the biggest timestamp per each `-dedup.minScrapeInterval` discrete interval if `-dedup.minScrapeInterval` is set to positive duration. For example, `-dedup.minScrapeInterval=60s` would leave a single raw sample with the biggest timestamp per each discrete 60s interval. If multiple raw samples have the same biggest timestamp on the given `-dedup.minScrapeInterval` discrete interval, then an arbitrary sample out of these samples is left. This aligns with the [staleness rules in Prometheus](https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness).
+VictoriaMetrics leaves a single raw sample with the biggest timestamp per each `-dedup.minScrapeInterval` discrete interval
+if `-dedup.minScrapeInterval` is set to positive duration. For example, `-dedup.minScrapeInterval=60s` would leave a single
+raw sample with the biggest timestamp per each discrete 60s interval.
+This aligns with the [staleness rules in Prometheus](https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness).
+
+If multiple raw samples have the same biggest timestamp on the given `-dedup.minScrapeInterval` discrete interval, then the sample with the biggest value is left.
 
 The `-dedup.minScrapeInterval=D` is equivalent to `-downsampling.period=0s:D` if [downsampling](#downsampling) is enabled. So it is safe to use deduplication and downsampling simultaneously.
 
@@ -1335,18 +1374,50 @@ It is recommended passing different `-promscrape.cluster.name` values to HA pair
 
 ## Storage
 
-VictoriaMetrics stores time series data in [MergeTree](https://en.wikipedia.org/wiki/Log-structured_merge-tree)-like
-data structures. On insert, VictoriaMetrics accumulates up to 1s of data and dumps it on disk to
-`<-storageDataPath>/data/small/YYYY_MM/` subdirectory forming a `part` with the following
-name pattern: `rowsCount_blocksCount_minTimestamp_maxTimestamp`. Each part consists of two "columns":
-values and timestamps. These are sorted and compressed raw time series values. Additionally, part contains
-index files for searching for specific series in the values and timestamps files.
+VictoriaMetrics buffers the ingested data in memory for up to a second. Then the buffered data is written to in-memory `parts`,
+which can be searched during queries. The in-memory `parts` are periodically persisted to disk, so they could survive unclean shutdown
+such as out of memory crash, hardware power loss or `SIGKILL` signal. The interval for flushing the in-memory data to disk
+can be configured with the `-inmemoryDataFlushInterval` command-line flag (note that too short flush interval may significantly increase disk IO).
 
-`Parts` are periodically merged into the bigger parts. The resulting `part` is constructed
-under `<-storageDataPath>/data/{small,big}/YYYY_MM/tmp` subdirectory.
-When the resulting `part` is complete, it is atomically moved from the `tmp`
-to its own subdirectory, while the source parts are atomically removed. The end result is that the source
-parts are substituted by a single resulting bigger `part` in the `<-storageDataPath>/data/{small,big}/YYYY_MM/` directory.
+In-memory parts are persisted to disk into `part` directories under the `<-storageDataPath>/data/small/YYYY_MM/` folder,
+where `YYYY_MM` is the month partition for the stored data. For example, `2022_11` is the partition for `parts`
+with [raw samples](https://docs.victoriametrics.com/keyConcepts.html#raw-samples) from `November 2022`.
+
+The `part` directory has the following name pattern: `rowsCount_blocksCount_minTimestamp_maxTimestamp`, where:
+
+- `rowsCount` - the number of [raw samples](https://docs.victoriametrics.com/keyConcepts.html#raw-samples) stored in the part
+- `blocksCount` - the number of blocks stored in the part (see details about blocks below)
+- `minTimestamp` and `maxTimestamp` - minimum and maximum timestamps across raw samples stored in the part
+
+Each `part` consists of `blocks` sorted by internal time series id (aka `TSID`).
+Each `block` contains up to 8K [raw samples](https://docs.victoriametrics.com/keyConcepts.html#raw-samples),
+which belong to a single [time series](https://docs.victoriametrics.com/keyConcepts.html#time-series).
+Raw samples in each block are sorted by `timestamp`. Blocks for the same time series are sorted
+by the `timestamp` of the first sample. Timestamps and values for all the blocks
+are stored in [compressed form](https://faun.pub/victoriametrics-achieving-better-compression-for-time-series-data-than-gorilla-317bc1f95932)
+in separate files under `part` directory - `timestamps.bin` and `values.bin`.
+
+The `part` directory also contains `index.bin` and `metaindex.bin` files - these files contain index
+for fast block lookups, which belong to the given `TSID` and cover the given time range.
+
+`Parts` are periodically merged into bigger parts in background. The background merge provides the following benefits:
+
+* keeping the number of data files under control, so they don't exceed limits on open files
+* improved data compression, since bigger parts are usually compressed better than smaller parts
+* improved query speed, since queries over smaller number of parts are executed faster
+* various background maintenance tasks such as [de-duplication](#deduplication), [downsampling](#downsampling)
+  and [freeing up disk space for the deleted time series](#how-to-delete-time-series) are performed during the merge
+
+Newly added `parts` either successfully appear in the storage or fail to appear.
+The newly added `parts` are being created in a temporary directory under `<-storageDataPath>/data/{small,big}/YYYY_MM/tmp` folder.
+When the newly added `part` is fully written and [fsynced](https://man7.org/linux/man-pages/man2/fsync.2.html)
+to a temporary directory, then it is atomically moved to the storage directory.
+Thanks to this alogrithm, storage never contains partially created parts, even if hardware power off
+occurrs in the middle of writing the `part` to disk - such incompletely written `parts`
+are automatically deleted on the next VictoriaMetrics start.
+
+The same applies to merge process — `parts` are either fully merged into a new `part` or fail to merge,
+leaving the source `parts` untouched.
 
 VictoriaMetrics doesn't merge parts if their summary size exceeds free disk space.
 This prevents from potential out of disk space errors during merge.
@@ -1355,24 +1426,10 @@ This increases overhead during data querying, since VictoriaMetrics needs to rea
 bigger number of parts per each request. That's why it is recommended to have at least 20%
 of free disk space under directory pointed by `-storageDataPath` command-line flag.
 
-Information about merging process is available in [single-node VictoriaMetrics](https://grafana.com/dashboards/10229)
-and [clustered VictoriaMetrics](https://grafana.com/grafana/dashboards/11176) Grafana dashboards.
+Information about merging process is available in [the dashboard for single-node VictoriaMetrics](https://grafana.com/dashboards/10229)
+and [the dashboard for VictoriaMetrics cluster](https://grafana.com/grafana/dashboards/11176).
 See more details in [monitoring docs](#monitoring).
 
-The `merge` process improves compression rate and keeps number of `parts` on disk relatively low.
-Benefits of doing the merge process are the following:
-
-* it improves query performance, since lower number of `parts` are inspected with each query
-* it reduces the number of data files, since each `part` contains fixed number of files
-* various background maintenance tasks such as [de-duplication](#deduplication), [downsampling](#downsampling)
-  and [freeing up disk space for the deleted time series](#how-to-delete-time-series) are performed during the merge.
-
-Newly added `parts` either appear in the storage or fail to appear.
-Storage never contains partially created parts. The same applies to merge process — `parts` are either fully
-merged into a new `part` or fail to merge. MergeTree doesn't contain partially merged `parts`.
-`Part` contents in MergeTree never change. Parts are immutable. They may be only deleted after the merge
-to a bigger `part` or when the `part` contents goes outside the configured `-retentionPeriod`.
-
 See [this article](https://valyala.medium.com/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282) for more details.
 
 See also [how to work with snapshots](#how-to-work-with-snapshots).
@@ -1400,7 +1457,11 @@ VictoriaMetrics does not support indefinite retention, but you can specify an ar
 
 ## Multiple retentions
 
-A single instance of VictoriaMetrics supports only a single retention, which can be configured via `-retentionPeriod` command-line flag. If you need multiple retentions, then you may start multiple VictoriaMetrics instances with distinct values for the following flags:
+Distinct retentions for distinct time series can be configured via [retention filters](#retention-filters)
+in [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise.html).
+
+Community version of VictoriaMetrics supports only a single retention, which can be configured via [-retentionPeriod](#retention) command-line flag.
+If you need multiple retentions in community version of VictoriaMetrics, then you may start multiple VictoriaMetrics instances with distinct values for the following flags:
 
 * `-retentionPeriod`
 * `-storageDataPath`, so the data for each retention period is saved in a separate directory
@@ -1408,12 +1469,44 @@ A single instance of VictoriaMetrics supports only a single retention, which can
 
 Then set up [vmauth](https://docs.victoriametrics.com/vmauth.html) in front of VictoriaMetrics instances,
 so it could route requests from particular user to VictoriaMetrics with the desired retention.
-The same scheme could be implemented for multiple tenants in [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html).
+
+Similar scheme can be applied for multiple tenants in [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html).
 See [these docs](https://docs.victoriametrics.com/guides/guide-vmcluster-multiple-retention-setup.html) for multi-retention setup details.
 
+## Retention filters
+
+[Enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/enterprise.html) supports e.g. `retention filters`,
+which allow configuring multiple retentions for distinct sets of time series matching the configured [series filters](https://docs.victoriametrics.com/keyConcepts.html#filtering)
+via `-retentionFilter` command-line flag. This flag accepts `filter:duration` options, where `filter` must be
+a valid [series filter](https://docs.victoriametrics.com/keyConcepts.html#filtering), while the `duration`
+must contain valid [retention](#retention) for time series matching the given `filter`. If series doesn't match
+any configured `-retentionFilter`, then the retention configured via [-retentionPeriod](#retention) command-line flag is applied to it.
+If series matches multiple configured retention filters, then the smallest retention is applied.
+
+For example, the following config sets 3 days retention for time series with `team="juniors"` label,
+30 days retention for time series with `env="dev"` or `env="staging"` label and 1 year retention for the remaining time series:
+
+```
+-retentionFilter='{team="juniors"}:3d' -retentionFilter='{env=~"dev|staging"}:30d' -retentionPeriod=1y
+```
+
+Important notes:
+
+- The data outside of the configured retention isn't deleted instantly - it is deleted eventually during [background merges](https://docs.victoriametrics.com/#storage).
+- The `-retentionFilter` doesn't remove old data from `indexdb` (aka inverted index) until the configured [-retentionPeriod](#retention).
+  So the `indexdb` size can grow big under [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate)
+  even for small retentions configured via `-retentionFilter`.
+
+It is safe updating `-retentionFilter` during VictoriaMetrics restarts - the updated retention filters are applied eventually
+to historical data.
+
+See [how to configure multiple retentions in VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#retention-filters).
+
+Retention filters can be evaluated for free by downloading and using enterprise binaries from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
+
 ## Downsampling
 
-[VictoriaMetrics Enterprise](https://victoriametrics.com/products/enterprise/) supports multi-level downsampling with `-downsampling.period` command-line flag. For example:
+[VictoriaMetrics Enterprise](https://docs.victoriametrics.com/enterprise.html) supports multi-level downsampling with `-downsampling.period` command-line flag. For example:
 
 * `-downsampling.period=30d:5m` instructs VictoriaMetrics to [deduplicate](#deduplication) samples older than 30 days with 5 minutes interval.
 
@@ -1472,6 +1565,8 @@ VictoriaMetrics provides the following security-related command-line flags:
 Explicitly set internal network interface for TCP and UDP ports for data ingestion with Graphite and OpenTSDB formats.
 For example, substitute `-graphiteListenAddr=:2003` with `-graphiteListenAddr=<internal_iface_ip>:2003`. This protects from unexpected requests from untrusted network interfaces.
 
+VictoriaMetrics has achieved security certifications for Database Software Development and Software-Based Monitoring Services. We apply strict security measures in everything we do. See our [Security page](https://victoriametrics.com/security/) for more details.
+
 ## Tuning
 
 * No need in tuning for VictoriaMetrics - it uses reasonable defaults for command-line flags,
@@ -1591,7 +1686,9 @@ All the durations and timestamps in traces are in milliseconds.
 
 Query tracing is allowed by default. It can be denied by passing `-denyQueryTracing` command-line flag to VictoriaMetrics.
 
-[VMUI](#vmui) provides an UI for query tracing - just click `Trace query` checkbox and re-run the query in order to investigate its' trace.
+[VMUI](#vmui) provides an UI:
+- for query tracing - just click `Trace query` checkbox and re-run the query in order to investigate its' trace.
+- for exploring custom trace - go to the tab `Trace analyzer` and upload or paste JSON with trace information.
 
 
 ## Cardinality limiter
@@ -1655,10 +1752,11 @@ and [cardinality explorer docs](#cardinality-explorer).
 
 * VictoriaMetrics buffers incoming data in memory for up to a few seconds before flushing it to persistent storage.
   This may lead to the following "issues":
-  * Data becomes available for querying in a few seconds after inserting. It is possible to flush in-memory buffers to persistent storage
+  * Data becomes available for querying in a few seconds after inserting. It is possible to flush in-memory buffers to searchable parts
     by requesting `/internal/force_flush` http handler. This handler is mostly needed for testing and debugging purposes.
   * The last few seconds of inserted data may be lost on unclean shutdown (i.e. OOM, `kill -9` or hardware reset).
-    See [this article for technical details](https://valyala.medium.com/wal-usage-looks-broken-in-modern-time-series-databases-b62a627ab704).
+    The `-inmemoryDataFlushInterval` command-line flag allows controlling the frequency of in-memory data flush to persistent storage.
+    See [storage docs](#storage) and [this article](https://valyala.medium.com/wal-usage-looks-broken-in-modern-time-series-databases-b62a627ab704) for more details.
 
 * If VictoriaMetrics works slowly and eats more than a CPU core per 100K ingested data points per second,
   then it is likely you have too many [active time series](https://docs.victoriametrics.com/FAQ.html#what-is-an-active-time-series) for the current amount of RAM.
@@ -1980,6 +2078,8 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
 ```
   -bigMergeConcurrency int
      The maximum number of CPU cores to use for big merges. Default value is used if set to 0
+  -cacheExpireDuration duration
+     Items are removed from in-memory caches after they aren't accessed for this duration. Lower values may reduce memory usage at the cost of higher CPU usage. See also -prevCacheRemovalPercent (default 30m0s)
   -configAuthKey string
      Authorization key for accessing /config page. It must be passed via authKey query arg
   -csvTrimTimestamp duration
@@ -1998,7 +2098,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
   -denyQueryTracing
      Whether to disable the ability to trace queries. See https://docs.victoriametrics.com/#query-tracing
   -downsampling.period array
-     Comma-separated downsampling periods in the format 'offset:period'. For example, '30d:10m' instructs to leave a single sample per 10 minutes for samples older than 30 days. See https://docs.victoriametrics.com/#downsampling for details. This flag is available only in enterprise version of VictoriaMetrics
+     Comma-separated downsampling periods in the format 'offset:period'. For example, '30d:10m' instructs to leave a single sample per 10 minutes for samples older than 30 days. See https://docs.victoriametrics.com/#downsampling for details. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
      Supports an array of values separated by comma or specified via multiple flags.
   -dryRun
      Whether to check only -promscrape.config and then exit. Unknown config entries aren't allowed in -promscrape.config by default. This can be changed with -promscrape.config.strictParse=false command-line flag
@@ -2009,7 +2109,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
   -envflag.prefix string
      Prefix for environment variables if -envflag.enable is set
   -eula
-     By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf . This flag is available only in enterprise version of VictoriaMetrics
+     By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
   -finalMergeDelay duration
      The delay before starting final merge for per-month partition after no new data is ingested into it. Final merge may require additional disk IO and CPU resources. Final merge may increase query speed and reduce disk space usage in some cases. Zero value disables final merge
   -flagsAuthKey string
@@ -2063,6 +2163,8 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      Uses '{measurement}' instead of '{measurement}{separator}{field_name}' for metic name if InfluxDB line contains only a single field
   -influxTrimTimestamp duration
      Trim timestamps for InfluxDB line protocol data to this duration. Minimum practical duration is 1ms. Higher duration (i.e. 1s) may be used for reducing disk space usage for timestamp data (default 1ms)
+  -inmemoryDataFlushInterval duration
+     The interval for guaranteed saving of in-memory data to disk. The saved data survives unclean shutdown such as OOM crash, hardware reset, SIGKILL, etc. Bigger intervals may help increasing lifetime of flash storage with limited write cycles (e.g. Raspberry PI). Smaller intervals increase disk IO load. Minimum supported value is 1s (default 5s)
   -insert.maxQueueDuration duration
      The maximum duration for waiting in the queue for insert requests due to -maxConcurrentInserts (default 1m0s)
   -logNewSeries
@@ -2112,6 +2214,8 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      Auth key for /debug/pprof/* endpoints. It must be passed via authKey query arg. It overrides httpAuth.* settings
   -precisionBits int
      The number of precision bits to store per each value. Lower precision bits improves data compression at the cost of precision loss (default 64)
+  -prevCacheRemovalPercent float
+     Items in the previous caches are removed when the percent of requests it serves becomes lower than this value. Higher values reduce memory usage at the cost of higher CPU usage. See also -cacheExpireDuration (default 0.1)
   -promscrape.azureSDCheckInterval duration
      Interval for checking for changes in Azure. This works only if azure_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs.html#azure_sd_configs for details (default 1m0s)
   -promscrape.cluster.memberNum string
@@ -2203,10 +2307,11 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      Supports an array of values separated by comma or specified via multiple flags.
   -relabelConfig string
      Optional path to a file with relabeling rules, which are applied to all the ingested metrics. The path can point either to local file or to http url. See https://docs.victoriametrics.com/#relabeling for details. The config is reloaded on SIGHUP signal
-  -relabelDebug
-     Whether to log metrics before and after relabeling with -relabelConfig. If the -relabelDebug is enabled, then the metrics aren't sent to storage. This is useful for debugging the relabeling configs
+  -retentionFilter array
+     Retention filter in the format 'filter:retention'. For example, '{env="dev"}:3d' configures the retention for time series with env="dev" label to 3 days. See https://docs.victoriametrics.com/#retention-filters for details. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
+     Supports an array of values separated by comma or specified via multiple flags.
   -retentionPeriod value
-     Data with timestamps outside the retentionPeriod is automatically deleted
+     Data with timestamps outside the retentionPeriod is automatically deleted. See also -retentionFilter
      The following optional suffixes are supported: h (hour), d (day), w (week), y (year). If suffix isn't set, then the duration is counted in months (default 1)
   -retentionTimezoneOffset duration
      The offset for performing indexdb rotation. If set to 0, then the indexdb rotation is performed at 4am UTC time per each -retentionPeriod. If set to 2h, then the indexdb rotation is performed at 4am EET time (the timezone with +2h offset)
@@ -2217,15 +2322,15 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
   -search.disableCache
      Whether to disable response caching. This may be useful during data backfilling
   -search.graphiteMaxPointsPerSeries int
-     The maximum number of points per series Graphite render API can return. This flag is available only in enterprise version of VictoriaMetrics (default 1000000)
+     The maximum number of points per series Graphite render API can return. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html (default 1000000)
   -search.graphiteStorageStep duration
-     The interval between datapoints stored in the database. It is used at Graphite Render API handler for normalizing the interval between datapoints in case it isn't normalized. It can be overridden by sending 'storage_step' query arg to /render API or by sending the desired interval via 'Storage-Step' http header during querying /render API. This flag is available only in enterprise version of VictoriaMetrics (default 10s)
+     The interval between datapoints stored in the database. It is used at Graphite Render API handler for normalizing the interval between datapoints in case it isn't normalized. It can be overridden by sending 'storage_step' query arg to /render API or by sending the desired interval via 'Storage-Step' http header during querying /render API. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html (default 10s)
   -search.latencyOffset duration
      The time when data points become visible in query results after the collection. Too small value can result in incomplete last points for query results (default 30s)
   -search.logSlowQueryDuration duration
      Log queries with execution time exceeding this value. Zero disables slow query logging (default 5s)
   -search.maxConcurrentRequests int
-     The maximum number of concurrent search requests. It shouldn't be high, since a single request can saturate all the CPU cores. See also -search.maxQueueDuration (default 8)
+     The maximum number of concurrent search requests. It shouldn't be high, since a single request can saturate all the CPU cores, while many concurrently executed requests may require high amounts of memory. See also -search.maxQueueDuration and -search.maxMemoryPerQuery (default 8)
   -search.maxExportDuration duration
      The maximum duration for /api/v1/export call (default 720h0m0s)
   -search.maxExportSeries int
@@ -2233,9 +2338,12 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
   -search.maxFederateSeries int
      The maximum number of time series, which can be returned from /federate. This option allows limiting memory usage (default 1000000)
   -search.maxGraphiteSeries int
-     The maximum number of time series, which can be scanned during queries to Graphite Render API. See https://docs.victoriametrics.com/#graphite-render-api-usage . This flag is available only in enterprise version of VictoriaMetrics (default 300000)
+     The maximum number of time series, which can be scanned during queries to Graphite Render API. See https://docs.victoriametrics.com/#graphite-render-api-usage . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html (default 300000)
   -search.maxLookback duration
      Synonym to -search.lookback-delta from Prometheus. The value is dynamically detected from interval between time series datapoints if not set. It can be overridden on per-query basis via max_lookback arg. See also '-search.maxStalenessInterval' flag, which has the same meaining due to historical reasons
+  -search.maxMemoryPerQuery size
+     The maximum amounts of memory a single query may consume. Queries requiring more memory are rejected. The total memory limit for concurrently executed queries can be estimated as -search.maxMemoryPerQuery multiplied by -search.maxConcurrentRequests
+     Supports the following optional suffixes for size values: KB, MB, GB, KiB, MiB, GiB (default 0)
   -search.maxPointsPerTimeseries int
      The maximum points per a single timeseries returned from /api/v1/query_range. This option doesn't limit the number of scanned raw samples in the database. The main purpose of this option is to limit the number of per-series points returned to graphing UI such as VMUI or Grafana. There is no sense in setting this limit to values bigger than the horizontal resolution of the graph (default 30000)
   -search.maxPointsSubqueryPerTimeseries int
@@ -2305,7 +2413,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      Overrides max size for indexdb/indexBlocks cache. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning
      Supports the following optional suffixes for size values: KB, MB, GB, KiB, MiB, GiB (default 0)
   -storage.cacheSizeIndexDBTagFilters size
-     Overrides max size for indexdb/tagFilters cache. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning
+     Overrides max size for indexdb/tagFiltersToMetricIDs cache. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning
      Supports the following optional suffixes for size values: KB, MB, GB, KiB, MiB, GiB (default 0)
   -storage.cacheSizeStorageTSID size
      Overrides max size for storage/tsid cache. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning

app/victoria-metrics/main.go

@@ -29,6 +29,10 @@ var (
 		"equal to -dedup.minScrapeInterval > 0. See https://docs.victoriametrics.com/#deduplication and https://docs.victoriametrics.com/#downsampling")
 	dryRun = flag.Bool("dryRun", false, "Whether to check only -promscrape.config and then exit. "+
 		"Unknown config entries aren't allowed in -promscrape.config by default. This can be changed with -promscrape.config.strictParse=false command-line flag")
+	inmemoryDataFlushInterval = flag.Duration("inmemoryDataFlushInterval", 5*time.Second, "The interval for guaranteed saving of in-memory data to disk. "+
+		"The saved data survives unclean shutdown such as OOM crash, hardware reset, SIGKILL, etc. "+
+		"Bigger intervals may help increasing lifetime of flash storage with limited write cycles (e.g. Raspberry PI). "+
+		"Smaller intervals increase disk IO load. Minimum supported value is 1s")
 )
 
 func main() {
@@ -54,6 +58,7 @@ func main() {
 	logger.Infof("starting VictoriaMetrics at %q...", *httpListenAddr)
 	startTime := time.Now()
 	storage.SetDedupInterval(*minScrapeInterval)
+	storage.SetDataFlushInterval(*inmemoryDataFlushInterval)
 	vmstorage.Init(promql.ResetRollupResultCacheIfNeeded)
 	vmselect.Init()
 	vminsert.Init()
@@ -96,6 +101,7 @@ func requestHandler(w http.ResponseWriter, r *http.Request) bool {
 			{"vmui", "Web UI"},
 			{"targets", "status for discovered active targets"},
 			{"service-discovery", "labels before and after relabeling for discovered targets"},
+			{"metric-relabel-debug", "debug metric relabeling"},
 			{"api/v1/targets", "advanced information about discovered targets in JSON format"},
 			{"config", "-promscrape.config contents"},
 			{"metrics", "available service metrics"},

app/vmagent/README.md

@@ -245,8 +245,6 @@ scrape_configs:
 * `scrape_align_interval: duration` for aligning scrapes to the given interval instead of using random offset
   in the range `[0 ... scrape_interval]` for scraping each target. The random offset helps spreading scrapes evenly in time.
 * `scrape_offset: duration` for specifying the exact offset for scraping instead of using random offset in the range `[0 ... scrape_interval]`.
-* `relabel_debug: true` for enabling debug logging during relabeling of the discovered targets. See [these docs](#relabeling).
-* `metric_relabel_debug: true` for enabling debug logging during relabeling of the scraped metrics. See [these docs](#relabeling).
 
 See [scrape_configs docs](https://docs.victoriametrics.com/sd_configs.html#scrape_configs) for more details on all the supported options.
 
@@ -317,7 +315,8 @@ Extra labels can be added to metrics collected by `vmagent` via the following me
 
 ## Automatically generated metrics
 
-`vmagent` automatically generates the following metrics per each scrape of every [Prometheus-compatible target](#how-to-collect-metrics-in-prometheus-format):
+`vmagent` automatically generates the following metrics per each scrape of every [Prometheus-compatible target](#how-to-collect-metrics-in-prometheus-format)
+and attaches target-specific `instance` and `job` labels to these metrics:
 
 * `up` - this metric exposes `1` value on successful scrape and `0` value on unsuccessful scrape. This allows monitoring
   failing scrapes with the following [MetricsQL query](https://docs.victoriametrics.com/MetricsQL.html):
@@ -382,7 +381,7 @@ Extra labels can be added to metrics collected by `vmagent` via the following me
   ```
 
   `vmagent` sets `scrape_series_added` to zero when it runs with `-promscrape.noStaleMarkers` command-line option
-  (e.g. when [staleness markers](#prometheus-staleness-markers) are disabled).
+  or when it scrapes target with `no_stale_markers: true` option, e.g. when [staleness markers](#prometheus-staleness-markers) are disabled.
 
 * `scrape_series_limit` - the limit on the number of unique time series the given target can expose according to [these docs](#cardinality-limiter).
   This metric is exposed only if the series limit is set.
@@ -405,6 +404,9 @@ Extra labels can be added to metrics collected by `vmagent` via the following me
   sum_over_time(scrape_series_limit_samples_dropped[1h]) > 0
   ```
 
+If the target exports metrics with names clashing with the automatically generated metric names, then `vmagent` automatically
+adds `exported_` prefix to these metric names, so they don't clash with automatically generated metric names.
+
 
 ## Relabeling
 
@@ -415,26 +417,25 @@ with [additional enhancements](#relabeling-enhancements). The relabeling can be
   This relabeling is used for modifying labels in discovered targets and for dropping unneded targets.
   See [relabeling cookbook](https://docs.victoriametrics.com/relabeling.html) for details.
 
-  This relabeling can be debugged by passing `relabel_debug: true` option to the corresponding `scrape_config` section.
-  In this case `vmagent` logs target labels before and after the relabeling and then drops the logged target.
+  This relabeling can be debugged by clicking the `debug` link at the corresponding target on the `http://vmagent:8429/targets` page
+  or on the `http://vmagent:8429/service-discovery` page. See [these docs](#relabel-debug) for details.
 
 * At the `scrape_config -> metric_relabel_configs` section in `-promscrape.config` file.
   This relabeling is used for modifying labels in scraped metrics and for dropping unneeded metrics.
   See [relabeling cookbook](https://docs.victoriametrics.com/relabeling.html) for details.
 
-  This relabeling can be debugged by passing `metric_relabel_debug: true` option to the corresponding `scrape_config` section.
-  In this case `vmagent` logs metrics before and after the relabeling and then drops the logged metrics.
+  This relabeling can be debugged via `http://vmagent:8429/metric-relabel-debug` page. See [these docs](#relabel-debug) for details.
 
 * At the `-remoteWrite.relabelConfig` file. This relabeling is used for modifying labels for all the collected metrics
-  (inluding [metrics obtained via push-based protocols](#how-to-push-data-to-vmagent)) and for dropping unneeded metrics
+  (including [metrics obtained via push-based protocols](#how-to-push-data-to-vmagent)) and for dropping unneeded metrics
   before sending them to all the configured `-remoteWrite.url` addresses.
-  This relabeling can be debugged by passing `-remoteWrite.relabelDebug` command-line option to `vmagent`.
-  In this case `vmagent` logs metrics before and after the relabeling and then drops all the logged metrics instead of sending them to remote storage.
+
+  This relabeling can be debugged via `http://vmagent:8429/metric-relabel-debug` page. See [these docs](#relabel-debug) for details.
 
 * At the `-remoteWrite.urlRelabelConfig` files. This relabeling is used for modifying labels for metrics
   and for dropping unneeded metrics before sending them to a particular `-remoteWrite.url`.
-  This relabeling can be debugged by passing `-remoteWrite.urlRelabelDebug` command-line options to `vmagent`.
-  In this case `vmagent` logs metrics before and after the relabeling and then drops all the logged metrics instead of sending them to the corresponding `-remoteWrite.url`.
+
+  This relabeling can be debugged via `http://vmagent:8429/metric-relabel-debug` page. See [these docs](#relabel-debug) for details.
 
 All the files with relabeling configs can contain special placeholders in the form `%{ENV_VAR}`,
 which are replaced by the corresponding environment variable values.
@@ -449,9 +450,6 @@ The following articles contain useful information about Prometheus relabeling:
 * [Extracting labels from legacy metric names](https://www.robustperception.io/extracting-labels-from-legacy-metric-names)
 * [relabel_configs vs metric_relabel_configs](https://www.robustperception.io/relabel_configs-vs-metric_relabel_configs)
 
-[This relabeler playground](https://relabeler.promlabs.com/) can help debugging issues related to relabeling.
-
-
 ## Relabeling enhancements
 
 `vmagent` provides the following enhancements on top of Prometheus-compatible relabeling:
@@ -545,7 +543,7 @@ The following articles contain useful information about Prometheus relabeling:
 
   * `keep_metrics`: keeps all the metrics with names matching the given `regex`,
     while dropping all the other metrics. For example, the following relabeling config keeps metrics
-    with `fo` and `bar` names, while dropping all the other metrics:
+    with `foo` and `bar` names, while dropping all the other metrics:
 
     ```yaml
     - action: keep_metrics
@@ -593,6 +591,28 @@ Important notes about `action: graphite` relabeling rules:
 The `action: graphite` relabeling rules are easier to write and maintain than `action: replace` for labels extraction from Graphite-style metric names.
 Additionally, the `action: graphite` relabeling rules usually work much faster than the equivalent `action: replace` rules.
 
+## Relabel debug
+
+`vmagent` and [single-node VictoriaMetrics](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html)
+provide the following tools for debugging target-level and metric-level relabeling:
+
+- Target-level relabeling (e.g. `relabel_configs` section at [scrape_configs](https://docs.victoriametrics.com/sd_configs.html#scrape_configs))
+  can be performed by navigating to `http://vmagent:8429/targets` page (`http://victoriametrics:8428/targets` page for single-node VictoriaMetrics)
+  and clicking the `debug` link at the target, which must be debugged.
+  The opened page will show step-by-step results for the actual relabeling rules applied to the target labels.
+
+  The `http://vmagent:8429/targets` page shows only active targets. If you need to understand why some target
+  is dropped during the relabeling, then navigate to `http://vmagent:8428/service-discovery` page
+  (`http://victoriametrics:8428/service-discovery` for single-node VictoriaMetrics), find the dropped target
+  and click the `debug` link there. The opened page will show step-by-step results for the actual relabeling rules,
+  which result to target drop.
+
+- Metric-level relabeling (e.g. `metric_relabel_configs` section at [scrape_configs](https://docs.victoriametrics.com/sd_configs.html#scrape_configs)
+  and all the relabeling, which can be set up via `-relabelConfig`, `-remoteWrite.relabelConfig` and `-remoteWrite.urlRelabelConfig`
+  command-line flags) can be performed by navigating to `http://vmagent:8429/metric-relabel-debug` page
+  (`http://victoriametrics:8428/metric-relabel-debug` page for single-node VictoriaMetrics)
+  and submitting there relabeling rules together with the metric to be relabeled.
+  The page will show step-by-step results for the entered relabeling rules executed against the entered metric.
 
 ## Prometheus staleness markers
 
@@ -604,9 +624,13 @@ Additionally, the `action: graphite` relabeling rules usually work much faster t
 * If the scrape target is removed from the list of targets, then stale markers are sent for all the metrics scraped from this target.
 
 Prometheus staleness markers' tracking needs additional memory, since it must store the previous response body per each scrape target
-in order to compare it to the current response body. The memory usage may be reduced by passing `-promscrape.noStaleMarkers`
-command-line flag to `vmagent`. This disables staleness tracking. This also disables tracking the number of new time series
-per each scrape with the auto-generated `scrape_series_added` metric. See [these docs](#automatically-generated-metrics) for details.
+in order to compare it to the current response body. The memory usage may be reduced by disabling staleness tracking in the following ways:
+
+* By passing `-promscrape.noStaleMarkers` command-line flag to `vmagent`. This disables staleness tracking across all the targets.
+* By specifying `no_stale_markers: true` option in the [scrape_config](https://docs.victoriametrics.com/sd_configs.html#scrape_configs) for the corresponding target.
+
+When staleness tracking is disabled, then `vmagent` doesn't track the number of new time series per each scrape,
+e.g. it sets `scrape_series_added` metric to zero. See [these docs](#automatically-generated-metrics) for details.
 
 ## Stream parsing mode
 
@@ -646,8 +670,9 @@ scrape_configs:
     'match[]': ['{__name__!=""}']
 ```
 
-Note that `sample_limit` and `series_limit` [scrape_config options](https://docs.victoriametrics.com/sd_configs.html#scrape_configs)
-cannot be used in stream parsing mode because the parsed data is pushed to remote storage as soon as it is parsed.
+Note that `vmagent` in stream parsing mode stores up to `sample_limit` samples to the configured `-remoteStorage.url`
+instead of droping all the samples read from the target, because the parsed data is sent to the remote storage
+as soon as it is parsed in stream parsing mode.
 
 ## Scraping big number of targets
 
@@ -736,8 +761,8 @@ By default `vmagent` doesn't limit the number of time series each scrape target
 
 * Via `-promscrape.seriesLimitPerTarget` command-line option. This limit is applied individually
   to all the scrape targets defined in the file pointed by `-promscrape.config`.
-* Via `series_limit` config option at `scrape_config` section. This limit is applied individually
-  to all the scrape targets defined in the given `scrape_config`.
+* Via `series_limit` config option at [scrape_config](https://docs.victoriametrics.com/sd_configs.html#scrape_configs) section.
+  This limit is applied individually to all the scrape targets defined in the given `scrape_config`.
 * Via `__series_limit__` label, which can be set with [relabeling](#relabeling) at `relabel_configs` section.
   This limit is applied to the corresponding scrape targets. Typical use case: to set the limit
   via [Kubernetes annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/) for targets,
@@ -904,7 +929,7 @@ See also [troubleshooting docs](https://docs.victoriametrics.com/Troubleshooting
 
 ## Kafka integration
 
-[Enterprise version](https://victoriametrics.com/products/enterprise/) of `vmagent` can read and write metrics from / to Kafka:
+[Enterprise version](https://docs.victoriametrics.com/enterprise.html) of `vmagent` can read and write metrics from / to Kafka:
 
 * [Reading metrics from Kafka](#reading-metrics-from-kafka)
 * [Writing metrics to Kafka](#writing-metrics-to-kafka)
@@ -914,7 +939,7 @@ in `vmutils-...-enteprise.tar.gz` archives and in [docker images](https://hub.do
 
 ### Reading metrics from Kafka
 
-[Enterprise version](https://victoriametrics.com/products/enterprise/) of `vmagent` can read metrics in various formats from Kafka messages.
+[Enterprise version](https://docs.victoriametrics.com/enterprise.html) of `vmagent` can read metrics in various formats from Kafka messages.
 These formats can be configured with `-kafka.consumer.topic.defaultFormat` or `-kafka.consumer.topic.format` command-line options. The following formats are supported:
 
 * `promremotewrite` - [Prometheus remote_write](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write).
@@ -956,7 +981,7 @@ data_format = "influx"
 
 #### Command-line flags for Kafka consumer
 
-These command-line flags are available only in [enterprise](https://victoriametrics.com/products/enterprise/) version of `vmagent`,
+These command-line flags are available only in [enterprise](https://docs.victoriametrics.com/enterprise.html) version of `vmagent`,
 which can be downloaded for evaluation from [releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) page
 (see `vmutils-...-enteprise.tar.gz` archives) and from [docker images](https://hub.docker.com/r/victoriametrics/vmagent/tags) with tags containing `enterprise` suffix.
 
@@ -991,7 +1016,7 @@ which can be downloaded for evaluation from [releases](https://github.com/Victor
 
 ### Writing metrics to Kafka
 
-[Enterprise version](https://victoriametrics.com/products/enterprise/) of `vmagent` writes data to Kafka with `at-least-once`
+[Enterprise version](https://docs.victoriametrics.com/enterprise.html) of `vmagent` writes data to Kafka with `at-least-once`
 semantics if `-remoteWrite.url` contains e.g. Kafka url. For example, if `vmagent` is started with `-remoteWrite.url=kafka://localhost:9092/?topic=prom-rw`,
 then it would send Prometheus remote_write messages to Kafka bootstrap server at `localhost:9092` with the topic `prom-rw`.
 These messages can be read later from Kafka by another `vmagent` - see [these docs](#reading-metrics-from-kafka) for details.
@@ -1023,7 +1048,7 @@ It may be needed to build `vmagent` from source code when developing or testing
 
 ### Development build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.
 2. Run `make vmagent` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds the `vmagent` binary and puts it into the `bin` folder.
 
@@ -1052,7 +1077,7 @@ ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://b
 
 ### Development ARM build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.
 2. Run `make vmagent-linux-arm` or `make vmagent-linux-arm64` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics)
    It builds `vmagent-linux-arm` or `vmagent-linux-arm64` binary respectively and puts it into the `bin` folder.
 
@@ -1103,6 +1128,8 @@ vmagent collects metrics data via popular data ingestion protocols and routes th
 
 See the docs at https://docs.victoriametrics.com/vmagent.html .
 
+  -cacheExpireDuration duration
+     Items are removed from in-memory caches after they aren't accessed for this duration. Lower values may reduce memory usage at the cost of higher CPU usage. See also -prevCacheRemovalPercent (default 30m0s)
   -configAuthKey string
      Authorization key for accessing /config page. It must be passed via authKey query arg
   -csvTrimTimestamp duration
@@ -1123,7 +1150,7 @@ See the docs at https://docs.victoriametrics.com/vmagent.html .
   -envflag.prefix string
      Prefix for environment variables if -envflag.enable is set
   -eula
-     By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf . This flag is available only in enterprise version of VictoriaMetrics
+     By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
   -flagsAuthKey string
      Auth key for /flags endpoint. It must be passed via authKey query arg. It overrides httpAuth.* settings
   -fs.disableMmap
@@ -1174,30 +1201,30 @@ See the docs at https://docs.victoriametrics.com/vmagent.html .
   -insert.maxQueueDuration duration
      The maximum duration for waiting in the queue for insert requests due to -maxConcurrentInserts (default 1m0s)
   -kafka.consumer.topic array
-     Kafka topic names for data consumption. This flag is available only in enterprise version of VictoriaMetrics
+     Kafka topic names for data consumption. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
      Supports an array of values separated by comma or specified via multiple flags.
   -kafka.consumer.topic.basicAuth.password array
-     Optional basic auth password for -kafka.consumer.topic. Must be used in conjunction with any supported auth methods for kafka client, specified by flag -kafka.consumer.topic.options='security.protocol=SASL_SSL;sasl.mechanisms=PLAIN' . This flag is available only in enterprise version of VictoriaMetrics
+     Optional basic auth password for -kafka.consumer.topic. Must be used in conjunction with any supported auth methods for kafka client, specified by flag -kafka.consumer.topic.options='security.protocol=SASL_SSL;sasl.mechanisms=PLAIN' . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
      Supports an array of values separated by comma or specified via multiple flags.
   -kafka.consumer.topic.basicAuth.username array
-     Optional basic auth username for -kafka.consumer.topic. Must be used in conjunction with any supported auth methods for kafka client, specified by flag -kafka.consumer.topic.options='security.protocol=SASL_SSL;sasl.mechanisms=PLAIN' . This flag is available only in enterprise version of VictoriaMetrics
+     Optional basic auth username for -kafka.consumer.topic. Must be used in conjunction with any supported auth methods for kafka client, specified by flag -kafka.consumer.topic.options='security.protocol=SASL_SSL;sasl.mechanisms=PLAIN' . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
      Supports an array of values separated by comma or specified via multiple flags.
   -kafka.consumer.topic.brokers array
-     List of brokers to connect for given topic, e.g. -kafka.consumer.topic.broker=host-1:9092;host-2:9092 . This flag is available only in enterprise version of VictoriaMetrics
+     List of brokers to connect for given topic, e.g. -kafka.consumer.topic.broker=host-1:9092;host-2:9092 . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
      Supports an array of values separated by comma or specified via multiple flags.
   -kafka.consumer.topic.defaultFormat string
-     Expected data format in the topic if -kafka.consumer.topic.format is skipped. This flag is available only in enterprise version of VictoriaMetrics (default "promremotewrite")
+     Expected data format in the topic if -kafka.consumer.topic.format is skipped. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html (default "promremotewrite")
   -kafka.consumer.topic.format array
-     data format for corresponding kafka topic. Valid formats: influx, prometheus, promremotewrite, graphite, jsonline . This flag is available only in enterprise version of VictoriaMetrics
+     data format for corresponding kafka topic. Valid formats: influx, prometheus, promremotewrite, graphite, jsonline . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
      Supports an array of values separated by comma or specified via multiple flags.
   -kafka.consumer.topic.groupID array
-     Defines group.id for topic. This flag is available only in enterprise version of VictoriaMetrics
+     Defines group.id for topic. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
      Supports an array of values separated by comma or specified via multiple flags.
   -kafka.consumer.topic.isGzipped array
-     Enables gzip setting for topic messages payload. Only prometheus, jsonline and influx formats accept gzipped messages.This flag is available only in enterprise version of VictoriaMetrics
+     Enables gzip setting for topic messages payload. Only prometheus, jsonline and influx formats accept gzipped messages.This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
      Supports array of values separated by comma or specified via multiple flags.
   -kafka.consumer.topic.options array
-     Optional key=value;key1=value2 settings for topic consumer. See full configuration options at https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md . This flag is available only in enterprise version of VictoriaMetrics
+     Optional key=value;key1=value2 settings for topic consumer. See full configuration options at https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
      Supports an array of values separated by comma or specified via multiple flags.
   -loggerDisableTimestamps
      Whether to disable writing timestamps in logs
@@ -1238,6 +1265,8 @@ See the docs at https://docs.victoriametrics.com/vmagent.html .
      Trim timestamps for OpenTSDB HTTP data to this duration. Minimum practical duration is 1ms. Higher duration (i.e. 1s) may be used for reducing disk space usage for timestamp data (default 1ms)
   -pprofAuthKey string
      Auth key for /debug/pprof/* endpoints. It must be passed via authKey query arg. It overrides httpAuth.* settings
+  -prevCacheRemovalPercent float
+     Items in the previous caches are removed when the percent of requests it serves becomes lower than this value. Higher values reduce memory usage at the cost of higher CPU usage. See also -cacheExpireDuration (default 0.1)
   -promscrape.azureSDCheckInterval duration
      Interval for checking for changes in Azure. This works only if azure_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs.html#azure_sd_configs for details (default 1m0s)
   -promscrape.cluster.memberNum string
@@ -1415,8 +1444,6 @@ See the docs at https://docs.victoriametrics.com/vmagent.html .
      Supports array of values separated by comma or specified via multiple flags.
   -remoteWrite.relabelConfig string
      Optional path to file with relabel_config entries. The path can point either to local file or to http url. These entries are applied to all the metrics before sending them to -remoteWrite.url. See https://docs.victoriametrics.com/vmagent.html#relabeling for details
-  -remoteWrite.relabelDebug
-     Whether to log metrics before and after relabeling with -remoteWrite.relabelConfig. If the -remoteWrite.relabelDebug is enabled, then the metrics aren't sent to remote storage. This is useful for debugging the relabeling configs
   -remoteWrite.roundDigits array
      Round metric values to this number of decimal digits after the point before writing them to remote storage. Examples: -remoteWrite.roundDigits=2 would round 1.236 to 1.24, while -remoteWrite.roundDigits=-1 would round 126.78 to 130. By default digits rounding is disabled. Set it to 100 for disabling it for a particular remote storage. This option may be used for improving data compression for the stored metrics
      Supports array of values separated by comma or specified via multiple flags.
@@ -1451,9 +1478,6 @@ See the docs at https://docs.victoriametrics.com/vmagent.html .
   -remoteWrite.urlRelabelConfig array
      Optional path to relabel config for the corresponding -remoteWrite.url. The path can point either to local file or to http url
      Supports an array of values separated by comma or specified via multiple flags.
-  -remoteWrite.urlRelabelDebug array
-     Whether to log metrics before and after relabeling with -remoteWrite.urlRelabelConfig. If the -remoteWrite.urlRelabelDebug is enabled, then the metrics aren't sent to the corresponding -remoteWrite.url. This is useful for debugging the relabeling configs
-     Supports array of values separated by comma or specified via multiple flags.
   -sortLabels
      Whether to sort labels for incoming samples before writing them to all the configured remote storage systems. This may be needed for reducing memory usage at remote storage when the order of labels in incoming samples is random. For example, if m{k1="v1",k2="v2"} may be sent as m{k2="v2",k1="v1"}Enabled sorting for labels can slow down ingestion performance a bit
   -tls

app/vmagent/datadog/request_handler.go

@@ -52,10 +52,18 @@ func insertRows(at *auth.Token, series []parser.Series, extraLabels []prompbmars
 			Name:  "__name__",
 			Value: ss.Metric,
 		})
-		labels = append(labels, prompbmarshal.Label{
-			Name:  "host",
-			Value: ss.Host,
-		})
+		if ss.Host != "" {
+			labels = append(labels, prompbmarshal.Label{
+				Name:  "host",
+				Value: ss.Host,
+			})
+		}
+		if ss.Device != "" {
+			labels = append(labels, prompbmarshal.Label{
+				Name:  "device",
+				Value: ss.Device,
+			})
+		}
 		for _, tag := range ss.Tags {
 			name, value := parser.SplitTag(tag)
 			if name == "host" {

app/vmagent/main.go

@@ -207,6 +207,7 @@ func requestHandler(w http.ResponseWriter, r *http.Request) bool {
 		httpserver.WriteAPIHelp(w, [][2]string{
 			{"targets", "status for discovered active targets"},
 			{"service-discovery", "labels before and after relabeling for discovered targets"},
+			{"metric-relabel-debug", "debug metric relabeling"},
 			{"api/v1/targets", "advanced information about discovered targets in JSON format"},
 			{"config", "-promscrape.config contents"},
 			{"metrics", "available service metrics"},
@@ -217,6 +218,16 @@ func requestHandler(w http.ResponseWriter, r *http.Request) bool {
 	}
 
 	path := strings.Replace(r.URL.Path, "//", "/", -1)
+	if strings.HasPrefix(path, "/prometheus/api/v1/import/prometheus") || strings.HasPrefix(path, "/api/v1/import/prometheus") {
+		prometheusimportRequests.Inc()
+		if err := prometheusimport.InsertHandler(nil, r); err != nil {
+			prometheusimportErrors.Inc()
+			httpserver.Errorf(w, r, "%s", err)
+			return true
+		}
+		w.WriteHeader(http.StatusNoContent)
+		return true
+	}
 	if strings.HasPrefix(path, "datadog/") {
 		// Trim suffix from paths starting from /datadog/ in order to support legacy DataDog agent.
 		// See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2670
@@ -250,15 +261,6 @@ func requestHandler(w http.ResponseWriter, r *http.Request) bool {
 		}
 		w.WriteHeader(http.StatusNoContent)
 		return true
-	case "/prometheus/api/v1/import/prometheus", "/api/v1/import/prometheus":
-		prometheusimportRequests.Inc()
-		if err := prometheusimport.InsertHandler(nil, r); err != nil {
-			prometheusimportErrors.Inc()
-			httpserver.Errorf(w, r, "%s", err)
-			return true
-		}
-		w.WriteHeader(http.StatusNoContent)
-		return true
 	case "/prometheus/api/v1/import/native", "/api/v1/import/native":
 		nativeimportRequests.Inc()
 		if err := native.InsertHandler(nil, r); err != nil {
@@ -324,6 +326,14 @@ func requestHandler(w http.ResponseWriter, r *http.Request) bool {
 		promscrapeServiceDiscoveryRequests.Inc()
 		promscrape.WriteServiceDiscovery(w, r)
 		return true
+	case "/prometheus/metric-relabel-debug", "/metric-relabel-debug":
+		promscrapeMetricRelabelDebugRequests.Inc()
+		promscrape.WriteMetricRelabelDebug(w, r)
+		return true
+	case "/prometheus/target-relabel-debug", "/target-relabel-debug":
+		promscrapeTargetRelabelDebugRequests.Inc()
+		promscrape.WriteTargetRelabelDebug(w, r)
+		return true
 	case "/prometheus/api/v1/targets", "/api/v1/targets":
 		promscrapeAPIV1TargetsRequests.Inc()
 		w.Header().Set("Content-Type", "application/json")
@@ -409,6 +419,16 @@ func processMultitenantRequest(w http.ResponseWriter, r *http.Request, path stri
 		httpserver.Errorf(w, r, "cannot obtain auth token: %s", err)
 		return true
 	}
+	if strings.HasPrefix(p.Suffix, "prometheus/api/v1/import/prometheus") {
+		prometheusimportRequests.Inc()
+		if err := prometheusimport.InsertHandler(at, r); err != nil {
+			prometheusimportErrors.Inc()
+			httpserver.Errorf(w, r, "%s", err)
+			return true
+		}
+		w.WriteHeader(http.StatusNoContent)
+		return true
+	}
 	if strings.HasPrefix(p.Suffix, "datadog/") {
 		// Trim suffix from paths starting from /datadog/ in order to support legacy DataDog agent.
 		// See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2670
@@ -442,15 +462,6 @@ func processMultitenantRequest(w http.ResponseWriter, r *http.Request, path stri
 		}
 		w.WriteHeader(http.StatusNoContent)
 		return true
-	case "prometheus/api/v1/import/prometheus":
-		prometheusimportRequests.Inc()
-		if err := prometheusimport.InsertHandler(at, r); err != nil {
-			prometheusimportErrors.Inc()
-			httpserver.Errorf(w, r, "%s", err)
-			return true
-		}
-		w.WriteHeader(http.StatusNoContent)
-		return true
 	case "prometheus/api/v1/import/native":
 		nativeimportRequests.Inc()
 		if err := native.InsertHandler(at, r); err != nil {
@@ -544,7 +555,11 @@ var (
 
 	promscrapeTargetsRequests          = metrics.NewCounter(`vmagent_http_requests_total{path="/targets"}`)
 	promscrapeServiceDiscoveryRequests = metrics.NewCounter(`vmagent_http_requests_total{path="/service-discovery"}`)
-	promscrapeAPIV1TargetsRequests     = metrics.NewCounter(`vmagent_http_requests_total{path="/api/v1/targets"}`)
+
+	promscrapeMetricRelabelDebugRequests = metrics.NewCounter(`vmagent_http_requests_total{path="/metric-relabel-debug"}`)
+	promscrapeTargetRelabelDebugRequests = metrics.NewCounter(`vmagent_http_requests_total{path="/target-relabel-debug"}`)
+
+	promscrapeAPIV1TargetsRequests = metrics.NewCounter(`vmagent_http_requests_total{path="/api/v1/targets"}`)
 
 	promscrapeTargetResponseRequests = metrics.NewCounter(`vmagent_http_requests_total{path="/target_response"}`)
 	promscrapeTargetResponseErrors   = metrics.NewCounter(`vmagent_http_request_errors_total{path="/target_response"}`)

app/vmagent/remotewrite/relabel.go

@@ -18,13 +18,8 @@ var (
 	relabelConfigPathGlobal = flag.String("remoteWrite.relabelConfig", "", "Optional path to file with relabel_config entries. "+
 		"The path can point either to local file or to http url. These entries are applied to all the metrics "+
 		"before sending them to -remoteWrite.url. See https://docs.victoriametrics.com/vmagent.html#relabeling for details")
-	relabelDebugGlobal = flag.Bool("remoteWrite.relabelDebug", false, "Whether to log metrics before and after relabeling with -remoteWrite.relabelConfig. "+
-		"If the -remoteWrite.relabelDebug is enabled, then the metrics aren't sent to remote storage. This is useful for debugging the relabeling configs")
 	relabelConfigPaths = flagutil.NewArrayString("remoteWrite.urlRelabelConfig", "Optional path to relabel config for the corresponding -remoteWrite.url. "+
 		"The path can point either to local file or to http url")
-	relabelDebug = flagutil.NewArrayBool("remoteWrite.urlRelabelDebug", "Whether to log metrics before and after relabeling with -remoteWrite.urlRelabelConfig. "+
-		"If the -remoteWrite.urlRelabelDebug is enabled, then the metrics aren't sent to the corresponding -remoteWrite.url. "+
-		"This is useful for debugging the relabeling configs")
 
 	usePromCompatibleNaming = flag.Bool("usePromCompatibleNaming", false, "Whether to replace characters unsupported by Prometheus with underscores "+
 		"in the ingested metric names and label names. For example, foo.bar{a.b='c'} is transformed into foo_bar{a_b='c'} during data ingestion if this flag is set. "+
@@ -42,7 +37,7 @@ func CheckRelabelConfigs() error {
 func loadRelabelConfigs() (*relabelConfigs, error) {
 	var rcs relabelConfigs
 	if *relabelConfigPathGlobal != "" {
-		global, err := promrelabel.LoadRelabelConfigs(*relabelConfigPathGlobal, *relabelDebugGlobal)
+		global, err := promrelabel.LoadRelabelConfigs(*relabelConfigPathGlobal)
 		if err != nil {
 			return nil, fmt.Errorf("cannot load -remoteWrite.relabelConfig=%q: %w", *relabelConfigPathGlobal, err)
 		}
@@ -58,7 +53,7 @@ func loadRelabelConfigs() (*relabelConfigs, error) {
 			// Skip empty relabel config.
 			continue
 		}
-		prc, err := promrelabel.LoadRelabelConfigs(path, relabelDebug.GetOptionalArg(i))
+		prc, err := promrelabel.LoadRelabelConfigs(path)
 		if err != nil {
 			return nil, fmt.Errorf("cannot load relabel configs from -remoteWrite.urlRelabelConfig=%q: %w", path, err)
 		}
@@ -123,7 +118,8 @@ func (rctx *relabelCtx) applyRelabeling(tss []prompbmarshal.TimeSeries, extraLab
 				}
 			}
 		}
-		labels = pcs.Apply(labels, labelsLen, true)
+		labels = pcs.Apply(labels, labelsLen)
+		labels = promrelabel.FinalizeLabels(labels[:labelsLen], labels[labelsLen:])
 		if len(labels) == labelsLen {
 			// Drop the current time series, since relabeling removed all the labels.
 			continue

app/vmagent/remotewrite/remotewrite.go

@@ -13,6 +13,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bloomfilter"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/cgroup"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fasttime"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/memory"
@@ -139,6 +140,8 @@ func Init() {
 		logger.Fatalf("cannot load relabel configs: %s", err)
 	}
 	allRelabelConfigs.Store(rcs)
+	configSuccess.Set(1)
+	configTimestamp.Set(fasttime.UnixTimestamp())
 
 	if len(*remoteWriteURLs) > 0 {
 		rwctxsDefault = newRemoteWriteCtxs(nil, *remoteWriteURLs)
@@ -154,18 +157,31 @@ func Init() {
 			case <-stopCh:
 				return
 			}
+			configReloads.Inc()
 			logger.Infof("SIGHUP received; reloading relabel configs pointed by -remoteWrite.relabelConfig and -remoteWrite.urlRelabelConfig")
 			rcs, err := loadRelabelConfigs()
 			if err != nil {
+				configReloadErrors.Inc()
+				configSuccess.Set(0)
 				logger.Errorf("cannot reload relabel configs; preserving the previous configs; error: %s", err)
 				continue
 			}
+
 			allRelabelConfigs.Store(rcs)
+			configSuccess.Set(1)
+			configTimestamp.Set(fasttime.UnixTimestamp())
 			logger.Infof("Successfully reloaded relabel configs")
 		}
 	}()
 }
 
+var (
+	configReloads      = metrics.NewCounter(`vmagent_relabel_config_reloads_total`)
+	configReloadErrors = metrics.NewCounter(`vmagent_relabel_config_reloads_errors_total`)
+	configSuccess      = metrics.NewCounter(`vmagent_relabel_config_last_reload_successful`)
+	configTimestamp    = metrics.NewCounter(`vmagent_relabel_config_last_reload_success_timestamp_seconds`)
+)
+
 func newRemoteWriteCtxs(at *auth.Token, urls []string) []*remoteWriteCtx {
 	if len(urls) == 0 {
 		logger.Panicf("BUG: urls must be non-empty")
@@ -235,7 +251,7 @@ func Stop() {
 // Push sends wr to remote storage systems set via `-remoteWrite.url`.
 //
 // If at is nil, then the data is pushed to the configured `-remoteWrite.url`.
-// If at isn't nil, the the data is pushed to the configured `-remoteWrite.multitenantURL`.
+// If at isn't nil, the data is pushed to the configured `-remoteWrite.multitenantURL`.
 //
 // Note that wr may be modified by Push due to relabeling and rounding.
 func Push(at *auth.Token, wr *prompbmarshal.WriteRequest) {

app/vmalert/README.md

@@ -213,13 +213,51 @@ The following variables are available in templating:
 | $labels or .Labels                 | The list of labels of the current alert. Use as ".Labels.<label_name>".                                   | {% raw %}Too high number of connections for {{ .Labels.instance }}{% endraw %}                                                                                         |
 | $alertID or .AlertID               | The current alert's ID generated by vmalert.                                                              | {% raw %}Link: vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}{% endraw %}                                                                                   |
 | $groupID or .GroupID               | The current alert's group ID generated by vmalert.                                                        | {% raw %}Link: vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}{% endraw %}                                                                                   |
-| $expr or .Expr                     | Alert's expression. Can be used for generating links to Grafana or other systems.                         | {% raw %}/api/v1/query?query={{ $expr&#124;quotesEscape&#124;queryEscape }}{% endraw %}                                                                                |
+| $expr or .Expr                     | Alert's expression. Can be used for generating links to Grafana or other systems.                         | {% raw %}/api/v1/query?query={{ $expr&#124;queryEscape }}{% endraw %}                                                                                |
 | $externalLabels or .ExternalLabels | List of labels configured via `-external.label` command-line flag.                                        | {% raw %}Issues with {{ $labels.instance }} (datacenter-{{ $externalLabels.dc }}){% endraw %}                                                                          |
 | $externalURL or .ExternalURL       | URL configured via `-external.url` command-line flag. Used for cases when vmalert is hidden behind proxy. | {% raw %}Visit {{ $externalURL }} for more details{% endraw %}                                                                                                         |
 
-Additionally, `vmalert` provides some extra templating functions
-listed [here](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmalert/templates/template.go)
-and [reusable templates](#reusable-templates).
+Additionally, `vmalert` provides some extra templating functions listed [here](#template-functions) and [reusable templates](#reusable-templates).
+
+#### Template functions
+
+`vmalert` provides the following template functions, which can be used during [templating](#templating):
+
+- `args arg0 ... argN` - converts the input args into a map with `arg0`, ..., `argN` keys.
+- `externalURL` - returns the value of `-external.url` command-line flag.
+- `first` - returns the first result from the input query results returned by `query` function.
+- `htmlEscape` - escapes special chars in input string, so it can be safely embedded as a plaintext into HTML.
+- `humanize` - converts the input number into human-readable format by adding [metric prefixes](https://en.wikipedia.org/wiki/Metric_prefix).
+  For example, `100000` is converted into `100K`.
+- `humanize1024` - converts the input number into human-readable format with 1024 base.
+  For example, `1024` is converted into 1ki`.
+- `humanizeDuration` - converts the input number in seconds into human-readable duration.
+- `humanizePercentage` - converts the input number to percentage. For example, `0.123` is converted into `12.3%`.
+- `humanizeTimestamp` - converts the input unix timestamp into human-readable time.
+- `jsonEscape` - JSON-encodes the input string.
+- `label name` - returns the value of the label with the given `name` from the input query result.
+- `match regex` - matches the input string against the provided `regex`.
+- `parseDuration` - parses the input string into duration in seconds. For example, `1h` is parsed into `3600`.
+- `parseDurationTime` - parses the input string into [time.Duration](https://pkg.go.dev/time#Duration).
+- `pathEscape` - escapes the input string, so it can be safely put inside path part of URL.
+- `pathPrefix` - returns the path part of the `-external.url` command-line flag.
+- `query` - executes the [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html) query against `-datasource.url` and returns the query result.
+  For example, {% raw %}`{{ query "sort_desc(process_resident_memory_bytes)" | first | value }}`{% endraw %} executes the `sort_desc(process_resident_memory_bytes)`
+  query at `-datasource.url` and returns the first result.
+- `queryEscape` - escapes the input string, so it can be safely put inside [query arg](https://en.wikipedia.org/wiki/Percent-encoding) part of URL.
+- `quotesEscape` - escapes the input string, so it can be safely embedded into JSON string.
+- `reReplaceAll regex repl` - replaces all the occurences of the `regex` in input string with the `repl`.
+- `safeHtml` - marks the input string as safe to use in HTML context without the need to html-escape it.
+- `sortByLabel name` - sorts the input query results by the label with the given `name`.
+- `stripDomain` - leaves the first part of the domain. For example, `foo.bar.baz` is converted to `foo`.
+  The port part is left in the output string. E.g. `foo.bar:1234` is converted into `foo:1234`.
+- `stripPort` - strips `port` part from `host:port` input string.
+- `strvalue` - returns the metric name from the input query result.
+- `title` - converts the first letters of every input word to uppercase.
+- `toLower` - converts all the chars in the input string to lowercase.
+- `toTime` - converts the input unix timestamp to [time.Time](https://pkg.go.dev/time#Time).
+- `toUpper` - converts all the chars in the input string to uppercase.
+- `value` - returns the numeric value from the input query result.
 
 #### Reusable templates
 
@@ -315,7 +353,7 @@ There are the following approaches exist for alerting and recording rules across
   rules to `AccountID=123`.
 
 * To specify `tenant` parameter per each alerting and recording group if
-  [enterprise version of vmalert](https://victoriametrics.com/products/enterprise/) is used
+  [enterprise version of vmalert](https://docs.victoriametrics.com/enterprise.html) is used
   with `-clusterMode` command-line flag. For example:
 
 ```yaml
@@ -635,10 +673,12 @@ See full description for these flags in `./vmalert -help`.
 ## Monitoring
 
 `vmalert` exports various metrics in Prometheus exposition format at `http://vmalert-host:8880/metrics` page.
+The default list of alerting rules for these metric can be found [here](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker).
 We recommend setting up regular scraping of this page either through `vmagent` or by Prometheus so that the exported
 metrics may be analyzed later.
 
-Use the official [Grafana dashboard](https://grafana.com/grafana/dashboards/14950) for `vmalert` overview. Graphs on this dashboard contain useful hints - hover the `i` icon at the top left corner of each graph in order to read it.
+Use the official [Grafana dashboard](https://grafana.com/grafana/dashboards/14950) for `vmalert` overview. 
+Graphs on this dashboard contain useful hints - hover the `i` icon in the top left corner of each graph in order to read it.
 If you have suggestions for improvements or have found a bug - please open an issue on github or add
 a review to the dashboard.
 
@@ -665,9 +705,10 @@ Try the following recommendations in such cases:
 are delivered to the datasource;
 * If you know in advance, that data in datasource is delayed - try changing vmalert's `-datasource.lookback`
 command-line flag to add a time shift for evaluations;
-* If time intervals between datapoints in datasource are irregular - try changing vmalert's `-datasource.queryStep`
-command-line flag to specify how far search query can lookback for the recent datapoint. By default, this value
-is equal to group's evaluation interval.
+* If time intervals between datapoints in datasource are irregular or `>=5min` - try changing vmalert's
+`-datasource.queryStep` command-line flag to specify how far search query can lookback for the recent datapoint. 
+The recommendation is to have the step at least two times bigger than `scrape_interval`, since
+there are no guarantees that scrape will not fail.
 
 Sometimes, it is not clear why some specific alert fired or didn't fire. It is very important to remember, that
 alerts with `for: 0` fire immediately when their expression becomes true. And alerts with `for > 0` will fire only
@@ -742,7 +783,7 @@ The shortlist of configuration flags is the following:
 {% raw  %}
 ```
   -clusterMode
-     If clusterMode is enabled, then vmalert automatically adds the tenant specified in config groups to -datasource.url, -remoteWrite.url and -remoteRead.url. See https://docs.victoriametrics.com/vmalert.html#multitenancy . This flag is available only in enterprise version of VictoriaMetrics
+     If clusterMode is enabled, then vmalert automatically adds the tenant specified in config groups to -datasource.url, -remoteWrite.url and -remoteRead.url. See https://docs.victoriametrics.com/vmalert.html#multitenancy . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
   -configCheckInterval duration
      Interval for checking for changes in '-rule' or '-notifier.config' files. By default the checking is disabled. Send SIGHUP signal in order to force config check for changes.
   -datasource.appendTypePrefix
@@ -766,11 +807,11 @@ The shortlist of configuration flags is the following:
   -datasource.maxIdleConnections int
      Defines the number of idle (keep-alive connections) to each configured datasource. Consider setting this value equal to the value: groups_total * group.concurrency. Too low a value may result in a high number of sockets in TIME_WAIT state. (default 100)
   -datasource.oauth2.clientID string
-     Optional OAuth2 clientID to use for -datasource.url.
+     Optional OAuth2 clientID to use for -datasource.url. 
   -datasource.oauth2.clientSecret string
      Optional OAuth2 clientSecret to use for -datasource.url.
   -datasource.oauth2.clientSecretFile string
-     Optional OAuth2 clientSecretFile to use for -datasource.url.
+     Optional OAuth2 clientSecretFile to use for -datasource.url. 
   -datasource.oauth2.scopes string
      Optional OAuth2 scopes to use for -datasource.url. Scopes must be delimited by ';'
   -datasource.oauth2.tokenUrl string
@@ -794,11 +835,11 @@ The shortlist of configuration flags is the following:
   -datasource.tlsServerName string
      Optional TLS server name to use for connections to -datasource.url. By default, the server name from -datasource.url is used
   -datasource.url string
-     Datasource compatible with Prometheus HTTP API. It can be single node VictoriaMetrics or vmselect URL. Required parameter. E.g. http://127.0.0.1:8428 . See also '-datasource.disablePathAppend', '-datasource.showURL'.
+     Datasource compatible with Prometheus HTTP API. It can be single node VictoriaMetrics or vmselect URL. Required parameter. E.g. http://127.0.0.1:8428 . See also -remoteRead.disablePathAppend and -datasource.showURL
   -defaultTenant.graphite string
-     Default tenant for Graphite alerting groups. See https://docs.victoriametrics.com/vmalert.html#multitenancy .This flag is available only in enterprise version of VictoriaMetrics
+     Default tenant for Graphite alerting groups. See https://docs.victoriametrics.com/vmalert.html#multitenancy .This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
   -defaultTenant.prometheus string
-     Default tenant for Prometheus alerting groups. See https://docs.victoriametrics.com/vmalert.html#multitenancy . This flag is available only in enterprise version of VictoriaMetrics
+     Default tenant for Prometheus alerting groups. See https://docs.victoriametrics.com/vmalert.html#multitenancy . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
   -disableAlertgroupLabel
      Whether to disable adding group's Name as label to generated alerts and time series.
   -dryRun
@@ -810,12 +851,11 @@ The shortlist of configuration flags is the following:
   -envflag.prefix string
      Prefix for environment variables if -envflag.enable is set
   -eula
-     By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf . This flag is available only in enterprise version of VictoriaMetrics
+     By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
   -evaluationInterval duration
      How often to evaluate the rules (default 1m0s)
   -external.alert.source string
-     External Alert Source allows to override the Source link for alerts sent to AlertManager for cases where you want to build a custom link to Grafana, Prometheus or any other service. Supports templating - see https://docs.victoriametrics.com/vmalert.html#templating . For example, link to Grafana: -external.alert.source='explore?orgId=1&left=[\"now-1h\",\"now\",\"VictoriaMetrics\",{\"expr\": \"{{$expr|quotesEscape|crlfEscape|queryEscape}}\"},{\"mode\":\"Metrics\"},{\"ui\":[true,true,true,\"none\"]}]' . If empty 'vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}' is used
-     If empty 'vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}' is used.
+     External Alert Source allows to override the Source link for alerts sent to AlertManager for cases where you want to build a custom link to Grafana, Prometheus or any other service. Supports templating - see https://docs.victoriametrics.com/vmalert.html#templating . For example, link to Grafana: -external.alert.source='explore?orgId=1&left=["now-1h","now","VictoriaMetrics",{"expr":{{$expr|jsonEscape|queryEscape}} },{"mode":"Metrics"},{"ui":[true,true,true,"none"]}]' . If empty 'vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}' is used.
   -external.label array
      Optional label in the form 'Name=value' to add to all generated recording rules and alerts. Pass multiple -label flags in order to add multiple label sets.
      Supports an array of values separated by comma or specified via multiple flags.
@@ -963,7 +1003,7 @@ The shortlist of configuration flags is the following:
   -remoteRead.oauth2.scopes string
      Optional OAuth2 scopes to use for -remoteRead.url. Scopes must be delimited by ';'.
   -remoteRead.oauth2.tokenUrl string
-     Optional OAuth2 tokenURL to use for -remoteRead.url.
+     Optional OAuth2 tokenURL to use for -remoteRead.url. 
   -remoteRead.showURL
      Whether to show -remoteRead.url in the exported metrics. It is hidden by default, since it can contain sensitive info such as auth key
   -remoteRead.tlsCAFile string
@@ -1010,6 +1050,8 @@ The shortlist of configuration flags is the following:
      Optional OAuth2 scopes to use for -notifier.url. Scopes must be delimited by ';'.
   -remoteWrite.oauth2.tokenUrl string
      Optional OAuth2 tokenURL to use for -notifier.url.
+  -remoteWrite.sendTimeout duration
+     Timeout for sending data to the configured -remoteWrite.url. (default 30s)
   -remoteWrite.showURL
      Whether to show -remoteWrite.url in the exported metrics. It is hidden by default, since it can contain sensitive info such as auth key
   -remoteWrite.tlsCAFile string
@@ -1275,7 +1317,7 @@ spec:
 
 ### Development build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.
 2. Run `make vmalert` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `vmalert` binary and puts it into the `bin` folder.
 
@@ -1291,7 +1333,7 @@ ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://b
 
 ### Development ARM build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.
 2. Run `make vmalert-linux-arm` or `make vmalert-linux-arm64` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `vmalert-linux-arm` or `vmalert-linux-arm64` binary respectively and puts it into the `bin` folder.
 

app/vmalert/alerting.go

@@ -284,7 +284,7 @@ func (ar *AlertingRule) Exec(ctx context.Context, ts time.Time, limit int) ([]pr
 		duration: time.Since(start),
 		samples:  len(qMetrics),
 		err:      err,
-		req:      req,
+		curl:     requestToCurl(req),
 	}
 
 	defer func() {

app/vmalert/config/config.go

@@ -243,9 +243,12 @@ func Parse(pathPatterns []string, validateTplFn ValidateTplFn, validateExpressio
 func parseFile(path string) ([]Group, error) {
 	data, err := os.ReadFile(path)
 	if err != nil {
-		return nil, fmt.Errorf("error reading alert rule file: %w", err)
+		return nil, fmt.Errorf("error reading alert rule file %q: %w", path, err)
+	}
+	data, err = envtemplate.ReplaceBytes(data)
+	if err != nil {
+		return nil, fmt.Errorf("cannot expand environment vars in %q: %w", path, err)
 	}
-	data = envtemplate.Replace(data)
 	g := struct {
 		Groups []Group `yaml:"groups"`
 		// Catches all undefined fields and must be empty after parsing.

app/vmalert/datasource/datasource.go

@@ -54,6 +54,19 @@ func (m *Metric) SetLabel(key, value string) {
 	m.AddLabel(key, value)
 }
 
+// SetLabels sets the given map as Metric labels
+func (m *Metric) SetLabels(ls map[string]string) {
+	var i int
+	m.Labels = make([]Label, len(ls))
+	for k, v := range ls {
+		m.Labels[i] = Label{
+			Name:  k,
+			Value: v,
+		}
+		i++
+	}
+}
+
 // AddLabel appends the given label to the label set
 func (m *Metric) AddLabel(key, value string) {
 	m.Labels = append(m.Labels, Label{Name: key, Value: value})

app/vmalert/datasource/init.go

@@ -14,7 +14,7 @@ import (
 
 var (
 	addr = flag.String("datasource.url", "", "Datasource compatible with Prometheus HTTP API. It can be single node VictoriaMetrics or vmselect URL. Required parameter. "+
-		"E.g. http://127.0.0.1:8428 . See also '-datasource.disablePathAppend', '-datasource.showURL'.")
+		"E.g. http://127.0.0.1:8428 . See also -remoteRead.disablePathAppend and -datasource.showURL")
 	appendTypePrefix  = flag.Bool("datasource.appendTypePrefix", false, "Whether to add type prefix to -datasource.url based on the query type. Set to true if sending different query types to the vmselect URL.")
 	showDatasourceURL = flag.Bool("datasource.showURL", false, "Whether to show -datasource.url in the exported metrics. "+
 		"It is hidden by default, since it can contain sensitive info such as auth key")

app/vmalert/datasource/vm_prom_api.go

@@ -32,19 +32,17 @@ type promInstant struct {
 }
 
 func (r promInstant) metrics() ([]Metric, error) {
-	var result []Metric
+	result := make([]Metric, len(r.Result))
 	for i, res := range r.Result {
 		f, err := strconv.ParseFloat(res.TV[1].(string), 64)
 		if err != nil {
 			return nil, fmt.Errorf("metric %v, unable to parse float64 from %s: %w", res, res.TV[1], err)
 		}
 		var m Metric
-		for k, v := range r.Result[i].Labels {
-			m.AddLabel(k, v)
-		}
+		m.SetLabels(res.Labels)
 		m.Timestamps = append(m.Timestamps, int64(res.TV[0].(float64)))
 		m.Values = append(m.Values, f)
-		result = append(result, m)
+		result[i] = m
 	}
 	return result, nil
 }
@@ -149,6 +147,16 @@ func (s *VMStorage) setPrometheusInstantReqParams(r *http.Request, query string,
 		timestamp = timestamp.Truncate(s.evaluationInterval)
 	}
 	q.Set("time", fmt.Sprintf("%d", timestamp.Unix()))
+	if s.evaluationInterval > 0 { // set step as evaluationInterval by default
+		// always convert to seconds to keep compatibility with older
+		// Prometheus versions. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1943
+		q.Set("step", fmt.Sprintf("%ds", int(s.evaluationInterval.Seconds())))
+	}
+	if s.queryStep > 0 { // override step with user-specified value
+		// always convert to seconds to keep compatibility with older
+		// Prometheus versions. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1943
+		q.Set("step", fmt.Sprintf("%ds", int(s.queryStep.Seconds())))
+	}
 	r.URL.RawQuery = q.Encode()
 	s.setPrometheusReqParams(r, query)
 }
@@ -163,6 +171,11 @@ func (s *VMStorage) setPrometheusRangeReqParams(r *http.Request, query string, s
 	q := r.URL.Query()
 	q.Add("start", fmt.Sprintf("%d", start.Unix()))
 	q.Add("end", fmt.Sprintf("%d", end.Unix()))
+	if s.evaluationInterval > 0 { // set step as evaluationInterval by default
+		// always convert to seconds to keep compatibility with older
+		// Prometheus versions. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1943
+		q.Set("step", fmt.Sprintf("%ds", int(s.evaluationInterval.Seconds())))
+	}
 	r.URL.RawQuery = q.Encode()
 	s.setPrometheusReqParams(r, query)
 }
@@ -178,15 +191,5 @@ func (s *VMStorage) setPrometheusReqParams(r *http.Request, query string) {
 		}
 	}
 	q.Set("query", query)
-	if s.evaluationInterval > 0 { // set step as evaluationInterval by default
-		// always convert to seconds to keep compatibility with older
-		// Prometheus versions. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1943
-		q.Set("step", fmt.Sprintf("%ds", int(s.evaluationInterval.Seconds())))
-	}
-	if s.queryStep > 0 { // override step with user-specified value
-		// always convert to seconds to keep compatibility with older
-		// Prometheus versions. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1943
-		q.Set("step", fmt.Sprintf("%ds", int(s.queryStep.Seconds())))
-	}
 	r.URL.RawQuery = q.Encode()
 }

app/vmalert/datasource/vm_prom_api_test.go

@@ -0,0 +1,20 @@
+package datasource
+
+import (
+	"encoding/json"
+	"testing"
+)
+
+func BenchmarkMetrics(b *testing.B) {
+	payload := []byte(`[{"metric":{"__name__":"vm_rows"},"value":[1583786142,"13763"]},{"metric":{"__name__":"vm_requests", "foo":"bar", "baz": "qux"},"value":[1583786140,"2000"]}]`)
+
+	var pi promInstant
+	if err := json.Unmarshal(payload, &pi.Result); err != nil {
+		b.Fatalf(err.Error())
+	}
+	b.Run("Instant", func(b *testing.B) {
+		for i := 0; i < b.N; i++ {
+			_, _ = pi.metrics()
+		}
+	})
+}

app/vmalert/datasource/vm_test.go

@@ -7,6 +7,7 @@ import (
 	"net/http/httptest"
 	"net/url"
 	"reflect"
+	"sort"
 	"strconv"
 	"strings"
 	"testing"
@@ -74,7 +75,7 @@ func TestVMInstantQuery(t *testing.T) {
 		case 5:
 			w.Write([]byte(`{"status":"success","data":{"resultType":"matrix"}}`))
 		case 6:
-			w.Write([]byte(`{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"vm_rows"},"value":[1583786142,"13763"]},{"metric":{"__name__":"vm_requests"},"value":[1583786140,"2000"]}]}}`))
+			w.Write([]byte(`{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"vm_rows","foo":"bar"},"value":[1583786142,"13763"]},{"metric":{"__name__":"vm_requests","foo":"baz"},"value":[1583786140,"2000"]}]}}`))
 		case 7:
 			w.Write([]byte(`{"status":"success","data":{"resultType":"scalar","result":[1583786142, "1"]}}`))
 		}
@@ -115,19 +116,17 @@ func TestVMInstantQuery(t *testing.T) {
 	}
 	expected := []Metric{
 		{
-			Labels:     []Label{{Value: "vm_rows", Name: "__name__"}},
+			Labels:     []Label{{Value: "vm_rows", Name: "__name__"}, {Value: "bar", Name: "foo"}},
 			Timestamps: []int64{1583786142},
 			Values:     []float64{13763},
 		},
 		{
-			Labels:     []Label{{Value: "vm_requests", Name: "__name__"}},
+			Labels:     []Label{{Value: "vm_requests", Name: "__name__"}, {Value: "baz", Name: "foo"}},
 			Timestamps: []int64{1583786140},
 			Values:     []float64{2000},
 		},
 	}
-	if !reflect.DeepEqual(m, expected) {
-		t.Fatalf("unexpected metric %+v want %+v", m, expected)
-	}
+	metricsEqual(t, m, expected)
 
 	m, req, err := pq.Query(ctx, query, ts) // 7 - scalar
 	if err != nil {
@@ -158,13 +157,36 @@ func TestVMInstantQuery(t *testing.T) {
 	if len(m) != 1 {
 		t.Fatalf("expected 1 metric  got %d in %+v", len(m), m)
 	}
-	exp := Metric{
-		Labels:     []Label{{Value: "constantLine(10)", Name: "name"}},
-		Timestamps: []int64{1611758403},
-		Values:     []float64{10},
+	exp := []Metric{
+		{
+			Labels:     []Label{{Value: "constantLine(10)", Name: "name"}},
+			Timestamps: []int64{1611758403},
+			Values:     []float64{10},
+		},
 	}
-	if !reflect.DeepEqual(m[0], exp) {
-		t.Fatalf("unexpected metric %+v want %+v", m[0], expected)
+	metricsEqual(t, m, exp)
+}
+
+func metricsEqual(t *testing.T, gotM, expectedM []Metric) {
+	for i, exp := range expectedM {
+		got := gotM[i]
+		gotTS, expTS := got.Timestamps, exp.Timestamps
+		if !reflect.DeepEqual(gotTS, expTS) {
+			t.Fatalf("unexpected timestamps %+v want %+v", gotTS, expTS)
+		}
+		gotV, expV := got.Values, exp.Values
+		if !reflect.DeepEqual(gotV, expV) {
+			t.Fatalf("unexpected values %+v want %+v", gotV, expV)
+		}
+		sort.Slice(got.Labels, func(i, j int) bool {
+			return got.Labels[i].Name < got.Labels[j].Name
+		})
+		sort.Slice(exp.Labels, func(i, j int) bool {
+			return exp.Labels[i].Name < exp.Labels[j].Name
+		})
+		if !reflect.DeepEqual(exp.Labels, got.Labels) {
+			t.Fatalf("unexpected labels %+v want %+v", got.Labels, exp.Labels)
+		}
 	}
 }
 
@@ -199,6 +221,10 @@ func TestVMRangeQuery(t *testing.T) {
 		if _, err := strconv.ParseInt(endTS, 10, 64); err != nil {
 			t.Errorf("failed to parse 'end' query param: %s", err)
 		}
+		step := r.URL.Query().Get("step")
+		if step != "15s" {
+			t.Errorf("expected 'step' query param to be 15s; got %q instead", step)
+		}
 		switch c {
 		case 0:
 			w.Write([]byte(`{"status":"success","data":{"resultType":"matrix","result":[{"metric":{"__name__":"vm_rows"},"values":[[1583786142,"13763"]]}]}}`))
@@ -212,7 +238,7 @@ func TestVMRangeQuery(t *testing.T) {
 	if err != nil {
 		t.Fatalf("unexpected: %s", err)
 	}
-	s := NewVMStorage(srv.URL, authCfg, time.Minute, 0, false, srv.Client())
+	s := NewVMStorage(srv.URL, authCfg, time.Minute, *queryStep, false, srv.Client())
 
 	pq := s.BuildWithParams(QuerierParams{DataSourceType: string(datasourcePrometheus), EvaluationInterval: 15 * time.Second})
 

app/vmalert/group.go

@@ -168,9 +168,12 @@ func (g *Group) Restore(ctx context.Context, qb datasource.QuerierBuilder, lookb
 		if rr.For < 1 {
 			continue
 		}
-		// ignore g.ExtraFilterLabels on purpose, so it
-		// won't affect the restore procedure.
-		q := qb.BuildWithParams(datasource.QuerierParams{})
+		// ignore QueryParams on purpose, because they could contain
+		// query filters. This may affect the restore procedure.
+		q := qb.BuildWithParams(datasource.QuerierParams{
+			DataSourceType: g.Type.String(),
+			Headers:        g.Headers,
+		})
 		if err := rr.Restore(ctx, q, lookback, labels); err != nil {
 			return fmt.Errorf("error while restoring rule %q: %w", rule, err)
 		}
@@ -418,20 +421,26 @@ func (e *executor) exec(ctx context.Context, rule Rule, ts time.Time, resolveDur
 		return fmt.Errorf("rule %q: failed to execute: %w", rule, err)
 	}
 
-	errGr := new(utils.ErrGroup)
 	if e.rw != nil {
-		pushToRW := func(tss []prompbmarshal.TimeSeries) {
+		pushToRW := func(tss []prompbmarshal.TimeSeries) error {
+			var lastErr error
 			for _, ts := range tss {
 				remoteWriteTotal.Inc()
 				if err := e.rw.Push(ts); err != nil {
 					remoteWriteErrors.Inc()
-					errGr.Add(fmt.Errorf("rule %q: remote write failure: %w", rule, err))
+					lastErr = fmt.Errorf("rule %q: remote write failure: %w", rule, err)
 				}
 			}
+			return lastErr
 		}
-		pushToRW(tss)
+		if err := pushToRW(tss); err != nil {
+			return err
+		}
+
 		staleSeries := e.getStaleSeries(rule, tss, ts)
-		pushToRW(staleSeries)
+		if err := pushToRW(staleSeries); err != nil {
+			return err
+		}
 	}
 
 	ar, ok := rule.(*AlertingRule)
@@ -445,6 +454,7 @@ func (e *executor) exec(ctx context.Context, rule Rule, ts time.Time, resolveDur
 	}
 
 	wg := sync.WaitGroup{}
+	errGr := new(utils.ErrGroup)
 	for _, nt := range e.notifiers() {
 		wg.Add(1)
 		go func(nt notifier.Notifier) {

app/vmalert/group_test.go

@@ -3,8 +3,6 @@ package main
 import (
 	"context"
 	"fmt"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/decimal"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
 	"reflect"
 	"sort"
 	"testing"
@@ -12,6 +10,9 @@ import (
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/config"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/notifier"
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/remotewrite"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/decimal"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promutils"
 )
 
@@ -452,3 +453,24 @@ func TestFaultyNotifier(t *testing.T) {
 	}
 	t.Fatalf("alive notifier didn't receive notification by %v", deadline)
 }
+
+func TestFaultyRW(t *testing.T) {
+	fq := &fakeQuerier{}
+	fq.add(metricWithValueAndLabels(t, 1, "__name__", "foo", "job", "bar"))
+
+	r := &RecordingRule{
+		Name:  "test",
+		state: newRuleState(),
+		q:     fq,
+	}
+
+	e := &executor{
+		rw:                       &remotewrite.Client{},
+		previouslySentSeriesToRW: make(map[uint64]map[string][]prompbmarshal.Label),
+	}
+
+	err := e.exec(context.Background(), r, time.Now(), 0, 10)
+	if err == nil {
+		t.Fatalf("expected to get an error from faulty RW client, got nil instead")
+	}
+}

app/vmalert/main.go

@@ -62,8 +62,8 @@ absolute path to all .tpl files in root.`)
 	externalAlertSource = flag.String("external.alert.source", "", `External Alert Source allows to override the Source link for alerts sent to AlertManager `+
 		`for cases where you want to build a custom link to Grafana, Prometheus or any other service. `+
 		`Supports templating - see https://docs.victoriametrics.com/vmalert.html#templating . `+
-		`For example, link to Grafana: -external.alert.source='explore?orgId=1&left=[\"now-1h\",\"now\",\"VictoriaMetrics\",{\"expr\": \"{{$expr|quotesEscape|crlfEscape|queryEscape}}\"},{\"mode\":\"Metrics\"},{\"ui\":[true,true,true,\"none\"]}]' . `+
-		`If empty 'vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}' is used`)
+		`For example, link to Grafana: -external.alert.source='explore?orgId=1&left=["now-1h","now","VictoriaMetrics",{"expr":{{$expr|jsonEscape|queryEscape}} },{"mode":"Metrics"},{"ui":[true,true,true,"none"]}]' . `+
+		`If empty 'vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}' is used.`)
 	externalLabels = flagutil.NewArrayString("external.label", "Optional label in the form 'Name=value' to add to all generated recording rules and alerts. "+
 		"Pass multiple -label flags in order to add multiple label sets.")
 

app/vmalert/notifier/alert.go

@@ -183,9 +183,9 @@ func (a Alert) toPromLabels(relabelCfg *promrelabel.ParsedConfigs) []prompbmarsh
 			Value: v,
 		})
 	}
-	promrelabel.SortLabels(labels)
 	if relabelCfg != nil {
-		return relabelCfg.Apply(labels, 0, false)
+		labels = relabelCfg.Apply(labels, 0)
 	}
+	promrelabel.SortLabels(labels)
 	return labels
 }

app/vmalert/notifier/alert_test.go

@@ -67,15 +67,21 @@ func TestAlert_ExecTemplate(t *testing.T) {
 		{
 			name: "expression-template",
 			alert: &Alert{
-				Expr: `vm_rows{"label"="bar"}>0`,
+				Expr: `vm_rows{"label"="bar"}<0`,
 			},
 			annotations: map[string]string{
-				"exprEscapedQuery": "{{ $expr|quotesEscape|queryEscape }}",
-				"exprEscapedPath":  "{{ $expr|quotesEscape|pathEscape }}",
+				"exprEscapedQuery":  "{{ $expr|queryEscape }}",
+				"exprEscapedPath":   "{{ $expr|pathEscape }}",
+				"exprEscapedJSON":   "{{ $expr|jsonEscape }}",
+				"exprEscapedQuotes": "{{ $expr|quotesEscape }}",
+				"exprEscapedHTML":   "{{ $expr|htmlEscape }}",
 			},
 			expTpl: map[string]string{
-				"exprEscapedQuery": "vm_rows%7B%5C%22label%5C%22%3D%5C%22bar%5C%22%7D%3E0",
-				"exprEscapedPath":  "vm_rows%7B%5C%22label%5C%22=%5C%22bar%5C%22%7D%3E0",
+				"exprEscapedQuery":  "vm_rows%7B%22label%22%3D%22bar%22%7D%3C0",
+				"exprEscapedPath":   "vm_rows%7B%22label%22=%22bar%22%7D%3C0",
+				"exprEscapedJSON":   `"vm_rows{\"label\"=\"bar\"}\u003c0"`,
+				"exprEscapedQuotes": `vm_rows{\"label\"=\"bar\"}\u003c0`,
+				"exprEscapedHTML":   "vm_rows{&quot;label&quot;=&quot;bar&quot;}&lt;0",
 			},
 		},
 		{
@@ -233,7 +239,7 @@ func TestAlert_toPromLabels(t *testing.T) {
   replacement: "aaa"
 - action: labeldrop
   regex: "env.*"
-`), false)
+`))
 	if err != nil {
 		t.Fatalf("unexpected error: %s", err)
 	}

app/vmalert/notifier/alertmanager_request.qtpl

@@ -15,10 +15,10 @@
     "endsAt":{%q= alert.End.Format(time.RFC3339Nano) %},
     {% endif %}
     "labels": {
-        "alertname":{%q= alert.Name %}
         {% code lbls := alert.toPromLabels(relabelCfg) %}
-        {% for _, l := range lbls %}
-            ,{%q= l.Name %}:{%q= l.Value %}
+        {% code ll := len(lbls) %}
+        {% for idx, l := range lbls %}
+            {%q= l.Name %}:{%q= l.Value %}{% if idx != ll-1 %}, {% endif %}
         {% endfor %}
     },
     "annotations": {

app/vmalert/notifier/alertmanager_request.qtpl.go

@@ -51,22 +51,27 @@ func streamamRequest(qw422016 *qt422016.Writer, alerts []Alert, generatorURL fun
 //line app/vmalert/notifier/alertmanager_request.qtpl:16
 		}
 //line app/vmalert/notifier/alertmanager_request.qtpl:16
-		qw422016.N().S(`"labels": {"alertname":`)
+		qw422016.N().S(`"labels": {`)
 //line app/vmalert/notifier/alertmanager_request.qtpl:18
-		qw422016.N().Q(alert.Name)
-//line app/vmalert/notifier/alertmanager_request.qtpl:19
 		lbls := alert.toPromLabels(relabelCfg)
 
+//line app/vmalert/notifier/alertmanager_request.qtpl:19
+		ll := len(lbls)
+
 //line app/vmalert/notifier/alertmanager_request.qtpl:20
-		for _, l := range lbls {
-//line app/vmalert/notifier/alertmanager_request.qtpl:20
-			qw422016.N().S(`,`)
+		for idx, l := range lbls {
 //line app/vmalert/notifier/alertmanager_request.qtpl:21
 			qw422016.N().Q(l.Name)
 //line app/vmalert/notifier/alertmanager_request.qtpl:21
 			qw422016.N().S(`:`)
 //line app/vmalert/notifier/alertmanager_request.qtpl:21
 			qw422016.N().Q(l.Value)
+//line app/vmalert/notifier/alertmanager_request.qtpl:21
+			if idx != ll-1 {
+//line app/vmalert/notifier/alertmanager_request.qtpl:21
+				qw422016.N().S(`,`)
+//line app/vmalert/notifier/alertmanager_request.qtpl:21
+			}
 //line app/vmalert/notifier/alertmanager_request.qtpl:22
 		}
 //line app/vmalert/notifier/alertmanager_request.qtpl:22

app/vmalert/notifier/alertmanager_test.go

@@ -67,9 +67,6 @@ func TestAlertManager_Send(t *testing.T) {
 			if a[0].GeneratorURL != "0/0" {
 				t.Errorf("expected 0/0 as generatorURL got %s", a[0].GeneratorURL)
 			}
-			if a[0].Labels["alertname"] != "alert0" {
-				t.Errorf("expected alert0 as alert name got %s", a[0].Labels["alertname"])
-			}
 			if a[0].StartsAt.IsZero() {
 				t.Errorf("expected non-zero start time")
 			}

app/vmalert/notifier/config.go

@@ -12,7 +12,6 @@ import (
 	"time"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promauth"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promrelabel"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape/discovery/consul"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape/discovery/dns"
@@ -84,12 +83,12 @@ func (cfg *Config) UnmarshalYAML(unmarshal func(interface{}) error) error {
 	if cfg.Timeout.Duration() == 0 {
 		cfg.Timeout = promutils.NewDuration(time.Second * 10)
 	}
-	rCfg, err := promrelabel.ParseRelabelConfigs(cfg.RelabelConfigs, false)
+	rCfg, err := promrelabel.ParseRelabelConfigs(cfg.RelabelConfigs)
 	if err != nil {
 		return fmt.Errorf("failed to parse relabeling config: %w", err)
 	}
 	cfg.parsedRelabelConfigs = rCfg
-	arCfg, err := promrelabel.ParseRelabelConfigs(cfg.AlertRelabelConfigs, false)
+	arCfg, err := promrelabel.ParseRelabelConfigs(cfg.AlertRelabelConfigs)
 	if err != nil {
 		return fmt.Errorf("failed to parse alert relabeling config: %w", err)
 	}
@@ -130,23 +129,24 @@ func parseConfig(path string) (*Config, error) {
 	return cfg, nil
 }
 
-func parseLabels(target string, metaLabels map[string]string, cfg *Config) (string, []prompbmarshal.Label, error) {
+func parseLabels(target string, metaLabels *promutils.Labels, cfg *Config) (string, *promutils.Labels, error) {
 	labels := mergeLabels(target, metaLabels, cfg)
-	labels = cfg.parsedRelabelConfigs.Apply(labels, 0, false)
-	labels = promrelabel.RemoveMetaLabels(labels[:0], labels)
+	labels.Labels = cfg.parsedRelabelConfigs.Apply(labels.Labels, 0)
+	labels.RemoveMetaLabels()
+	labels.Sort()
 	// Remove references to already deleted labels, so GC could clean strings for label name and label value past len(labels).
 	// This should reduce memory usage when relabeling creates big number of temporary labels with long names and/or values.
 	// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825 for details.
-	labels = append([]prompbmarshal.Label{}, labels...)
+	labels = labels.Clone()
 
-	if len(labels) == 0 {
+	if labels.Len() == 0 {
 		return "", nil, nil
 	}
-	schemeRelabeled := promrelabel.GetLabelValueByName(labels, "__scheme__")
+	schemeRelabeled := labels.Get("__scheme__")
 	if len(schemeRelabeled) == 0 {
 		schemeRelabeled = "http"
 	}
-	addressRelabeled := promrelabel.GetLabelValueByName(labels, "__address__")
+	addressRelabeled := labels.Get("__address__")
 	if len(addressRelabeled) == 0 {
 		return "", nil, nil
 	}
@@ -154,7 +154,7 @@ func parseLabels(target string, metaLabels map[string]string, cfg *Config) (stri
 		return "", nil, nil
 	}
 	addressRelabeled = addMissingPort(schemeRelabeled, addressRelabeled)
-	alertsPathRelabeled := promrelabel.GetLabelValueByName(labels, "__alerts_path__")
+	alertsPathRelabeled := labels.Get("__alerts_path__")
 	if !strings.HasPrefix(alertsPathRelabeled, "/") {
 		alertsPathRelabeled = "/" + alertsPathRelabeled
 	}
@@ -178,21 +178,12 @@ func addMissingPort(scheme, target string) string {
 	return target
 }
 
-func mergeLabels(target string, metaLabels map[string]string, cfg *Config) []prompbmarshal.Label {
+func mergeLabels(target string, metaLabels *promutils.Labels, cfg *Config) *promutils.Labels {
 	// See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
-	m := make(map[string]string)
-	m["__address__"] = target
-	m["__scheme__"] = cfg.Scheme
-	m["__alerts_path__"] = path.Join("/", cfg.PathPrefix, alertManagerPath)
-	for k, v := range metaLabels {
-		m[k] = v
-	}
-	result := make([]prompbmarshal.Label, 0, len(m))
-	for k, v := range m {
-		result = append(result, prompbmarshal.Label{
-			Name:  k,
-			Value: v,
-		})
-	}
-	return result
+	m := promutils.NewLabels(3 + metaLabels.Len())
+	m.Add("__address__", target)
+	m.Add("__scheme__", cfg.Scheme)
+	m.Add("__alerts_path__", path.Join("/", cfg.PathPrefix, alertManagerPath))
+	m.AddFrom(metaLabels)
+	return m
 }

app/vmalert/notifier/config_watcher.go

@@ -9,6 +9,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promauth"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape/discovery/consul"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape/discovery/dns"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promutils"
 )
 
 // configWatcher supports dynamic reload of Notifier objects
@@ -123,7 +124,7 @@ func targetsFromLabels(labelsFn getLabels, cfg *Config, genFn AlertURLGenerator)
 	var errors []error
 	duplicates := make(map[string]struct{})
 	for _, labels := range metaLabels {
-		target := labels["__address__"]
+		target := labels.Get("__address__")
 		u, processedLabels, err := parseLabels(target, labels, cfg)
 		if err != nil {
 			errors = append(errors, err)
@@ -156,7 +157,7 @@ func targetsFromLabels(labelsFn getLabels, cfg *Config, genFn AlertURLGenerator)
 	return targets, errors
 }
 
-type getLabels func() ([]map[string]string, error)
+type getLabels func() ([]*promutils.Labels, error)
 
 func (cw *configWatcher) start() error {
 	if len(cw.cfg.StaticConfigs) > 0 {
@@ -182,8 +183,8 @@ func (cw *configWatcher) start() error {
 	}
 
 	if len(cw.cfg.ConsulSDConfigs) > 0 {
-		err := cw.add(TargetConsul, *consul.SDCheckInterval, func() ([]map[string]string, error) {
-			var labels []map[string]string
+		err := cw.add(TargetConsul, *consul.SDCheckInterval, func() ([]*promutils.Labels, error) {
+			var labels []*promutils.Labels
 			for i := range cw.cfg.ConsulSDConfigs {
 				sdc := &cw.cfg.ConsulSDConfigs[i]
 				targetLabels, err := sdc.GetLabels(cw.cfg.baseDir)
@@ -200,8 +201,8 @@ func (cw *configWatcher) start() error {
 	}
 
 	if len(cw.cfg.DNSSDConfigs) > 0 {
-		err := cw.add(TargetDNS, *dns.SDCheckInterval, func() ([]map[string]string, error) {
-			var labels []map[string]string
+		err := cw.add(TargetDNS, *dns.SDCheckInterval, func() ([]*promutils.Labels, error) {
+			var labels []*promutils.Labels
 			for i := range cw.cfg.DNSSDConfigs {
 				sdc := &cw.cfg.DNSSDConfigs[i]
 				targetLabels, err := sdc.GetLabels(cw.cfg.baseDir)

app/vmalert/notifier/init.go

@@ -10,7 +10,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/templates"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promauth"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promutils"
 )
 
 var (
@@ -159,7 +159,7 @@ func notifiersFromFlags(gen AlertURLGenerator) ([]Notifier, error) {
 // list of labels added during discovery.
 type Target struct {
 	Notifier
-	Labels []prompbmarshal.Label
+	Labels *promutils.Labels
 }
 
 // TargetType defines how the Target was discovered

app/vmalert/recording.go

@@ -121,7 +121,7 @@ func (rr *RecordingRule) Exec(ctx context.Context, ts time.Time, limit int) ([]p
 		at:       ts,
 		duration: time.Since(start),
 		samples:  len(qMetrics),
-		req:      req,
+		curl:     requestToCurl(req),
 	}
 
 	defer func() {

app/vmalert/remotewrite/init.go

@@ -77,13 +77,12 @@ func Init(ctx context.Context) (*Client, error) {
 	}
 
 	return NewClient(ctx, Config{
-		Addr:              *addr,
-		AuthCfg:           authCfg,
-		Concurrency:       *concurrency,
-		MaxQueueSize:      *maxQueueSize,
-		MaxBatchSize:      *maxBatchSize,
-		FlushInterval:     *flushInterval,
-		DisablePathAppend: *disablePathAppend,
-		Transport:         t,
+		Addr:          *addr,
+		AuthCfg:       authCfg,
+		Concurrency:   *concurrency,
+		MaxQueueSize:  *maxQueueSize,
+		MaxBatchSize:  *maxBatchSize,
+		FlushInterval: *flushInterval,
+		Transport:     t,
 	})
 }

app/vmalert/remotewrite/remotewrite.go

@@ -22,6 +22,7 @@ import (
 
 var (
 	disablePathAppend = flag.Bool("remoteWrite.disablePathAppend", false, "Whether to disable automatic appending of '/api/v1/write' path to the configured -remoteWrite.url.")
+	sendTimeout       = flag.Duration("remoteWrite.sendTimeout", 30*time.Second, "Timeout for sending data to the configured -remoteWrite.url.")
 )
 
 // Client is an asynchronous HTTP client for writing
@@ -57,13 +58,8 @@ type Config struct {
 	MaxQueueSize int
 	// FlushInterval defines time interval for flushing batches
 	FlushInterval time.Duration
-	// WriteTimeout defines timeout for HTTP write request
-	// to remote storage
-	WriteTimeout time.Duration
 	// Transport will be used by the underlying http.Client
 	Transport *http.Transport
-	// DisablePathAppend can be used to not automatically append '/api/v1/write' to the remote write url
-	DisablePathAppend bool
 }
 
 const (
@@ -89,9 +85,6 @@ func NewClient(ctx context.Context, cfg Config) (*Client, error) {
 	if cfg.FlushInterval == 0 {
 		cfg.FlushInterval = defaultFlushInterval
 	}
-	if cfg.WriteTimeout == 0 {
-		cfg.WriteTimeout = defaultWriteTimeout
-	}
 	if cfg.Transport == nil {
 		cfg.Transport = http.DefaultTransport.(*http.Transport).Clone()
 	}
@@ -101,7 +94,7 @@ func NewClient(ctx context.Context, cfg Config) (*Client, error) {
 	}
 	c := &Client{
 		c: &http.Client{
-			Timeout:   cfg.WriteTimeout,
+			Timeout:   *sendTimeout,
 			Transport: cfg.Transport,
 		},
 		addr:          strings.TrimSuffix(cfg.Addr, "/"),
@@ -218,7 +211,7 @@ func (c *Client) flush(ctx context.Context, wr *prompbmarshal.WriteRequest) {
 			return
 		}
 
-		logger.Errorf("attempt %d to send request failed: %s", i+1, err)
+		logger.Warnf("attempt %d to send request failed: %s", i+1, err)
 		// sleeping to avoid remote db hammering
 		time.Sleep(time.Second)
 		continue

app/vmalert/rule.go

@@ -3,7 +3,6 @@ package main
 import (
 	"context"
 	"errors"
-	"net/http"
 	"sync"
 	"time"
 
@@ -54,8 +53,8 @@ type ruleStateEntry struct {
 	// stores the number of samples returned during
 	// the last evaluation
 	samples int
-	// stores the HTTP request used by datasource during rule.Exec
-	req *http.Request
+	// stores the curl command reflecting the HTTP request used during rule.Exec
+	curl string
 }
 
 const defaultStateEntriesLimit = 20

app/vmalert/templates/funcs.qtpl

@@ -0,0 +1,15 @@
+{% stripspace %}
+
+{% func quotesEscape(s string) %}
+	{%j= s %}
+{% endfunc %}
+
+{% func jsonEscape(s string) %}
+	{%q= s %}
+{% endfunc %}
+
+{% func htmlEscape(s string) %}
+	{%s s %}
+{% endfunc %}
+
+{% endstripspace %}

app/vmalert/templates/funcs.qtpl.go

@@ -0,0 +1,117 @@
+// Code generated by qtc from "funcs.qtpl". DO NOT EDIT.
+// See https://github.com/valyala/quicktemplate for details.
+
+//line app/vmalert/templates/funcs.qtpl:3
+package templates
+
+//line app/vmalert/templates/funcs.qtpl:3
+import (
+	qtio422016 "io"
+
+	qt422016 "github.com/valyala/quicktemplate"
+)
+
+//line app/vmalert/templates/funcs.qtpl:3
+var (
+	_ = qtio422016.Copy
+	_ = qt422016.AcquireByteBuffer
+)
+
+//line app/vmalert/templates/funcs.qtpl:3
+func streamquotesEscape(qw422016 *qt422016.Writer, s string) {
+//line app/vmalert/templates/funcs.qtpl:4
+	qw422016.N().J(s)
+//line app/vmalert/templates/funcs.qtpl:5
+}
+
+//line app/vmalert/templates/funcs.qtpl:5
+func writequotesEscape(qq422016 qtio422016.Writer, s string) {
+//line app/vmalert/templates/funcs.qtpl:5
+	qw422016 := qt422016.AcquireWriter(qq422016)
+//line app/vmalert/templates/funcs.qtpl:5
+	streamquotesEscape(qw422016, s)
+//line app/vmalert/templates/funcs.qtpl:5
+	qt422016.ReleaseWriter(qw422016)
+//line app/vmalert/templates/funcs.qtpl:5
+}
+
+//line app/vmalert/templates/funcs.qtpl:5
+func quotesEscape(s string) string {
+//line app/vmalert/templates/funcs.qtpl:5
+	qb422016 := qt422016.AcquireByteBuffer()
+//line app/vmalert/templates/funcs.qtpl:5
+	writequotesEscape(qb422016, s)
+//line app/vmalert/templates/funcs.qtpl:5
+	qs422016 := string(qb422016.B)
+//line app/vmalert/templates/funcs.qtpl:5
+	qt422016.ReleaseByteBuffer(qb422016)
+//line app/vmalert/templates/funcs.qtpl:5
+	return qs422016
+//line app/vmalert/templates/funcs.qtpl:5
+}
+
+//line app/vmalert/templates/funcs.qtpl:7
+func streamjsonEscape(qw422016 *qt422016.Writer, s string) {
+//line app/vmalert/templates/funcs.qtpl:8
+	qw422016.N().Q(s)
+//line app/vmalert/templates/funcs.qtpl:9
+}
+
+//line app/vmalert/templates/funcs.qtpl:9
+func writejsonEscape(qq422016 qtio422016.Writer, s string) {
+//line app/vmalert/templates/funcs.qtpl:9
+	qw422016 := qt422016.AcquireWriter(qq422016)
+//line app/vmalert/templates/funcs.qtpl:9
+	streamjsonEscape(qw422016, s)
+//line app/vmalert/templates/funcs.qtpl:9
+	qt422016.ReleaseWriter(qw422016)
+//line app/vmalert/templates/funcs.qtpl:9
+}
+
+//line app/vmalert/templates/funcs.qtpl:9
+func jsonEscape(s string) string {
+//line app/vmalert/templates/funcs.qtpl:9
+	qb422016 := qt422016.AcquireByteBuffer()
+//line app/vmalert/templates/funcs.qtpl:9
+	writejsonEscape(qb422016, s)
+//line app/vmalert/templates/funcs.qtpl:9
+	qs422016 := string(qb422016.B)
+//line app/vmalert/templates/funcs.qtpl:9
+	qt422016.ReleaseByteBuffer(qb422016)
+//line app/vmalert/templates/funcs.qtpl:9
+	return qs422016
+//line app/vmalert/templates/funcs.qtpl:9
+}
+
+//line app/vmalert/templates/funcs.qtpl:11
+func streamhtmlEscape(qw422016 *qt422016.Writer, s string) {
+//line app/vmalert/templates/funcs.qtpl:12
+	qw422016.E().S(s)
+//line app/vmalert/templates/funcs.qtpl:13
+}
+
+//line app/vmalert/templates/funcs.qtpl:13
+func writehtmlEscape(qq422016 qtio422016.Writer, s string) {
+//line app/vmalert/templates/funcs.qtpl:13
+	qw422016 := qt422016.AcquireWriter(qq422016)
+//line app/vmalert/templates/funcs.qtpl:13
+	streamhtmlEscape(qw422016, s)
+//line app/vmalert/templates/funcs.qtpl:13
+	qt422016.ReleaseWriter(qw422016)
+//line app/vmalert/templates/funcs.qtpl:13
+}
+
+//line app/vmalert/templates/funcs.qtpl:13
+func htmlEscape(s string) string {
+//line app/vmalert/templates/funcs.qtpl:13
+	qb422016 := qt422016.AcquireByteBuffer()
+//line app/vmalert/templates/funcs.qtpl:13
+	writehtmlEscape(qb422016, s)
+//line app/vmalert/templates/funcs.qtpl:13
+	qs422016 := string(qb422016.B)
+//line app/vmalert/templates/funcs.qtpl:13
+	qt422016.ReleaseByteBuffer(qb422016)
+//line app/vmalert/templates/funcs.qtpl:13
+	return qs422016
+//line app/vmalert/templates/funcs.qtpl:13
+}

app/vmalert/templates/template.go

@@ -207,23 +207,10 @@ func FuncsWithExternalURL(externalURL *url.URL) textTpl.FuncMap {
 // templateFuncs initiates template helper functions
 func templateFuncs() textTpl.FuncMap {
 	// See https://prometheus.io/docs/prometheus/latest/configuration/template_reference/
+	// and https://github.com/prometheus/prometheus/blob/fa6e05903fd3ce52e374a6e1bf4eb98c9f1f45a7/template/template.go#L150
 	return textTpl.FuncMap{
 		/* Strings */
 
-		// reReplaceAll ReplaceAllString returns a copy of src, replacing matches of the Regexp with
-		// the replacement string repl. Inside repl, $ signs are interpreted as in Expand,
-		// so for instance $1 represents the text of the first submatch.
-		// alias for https://golang.org/pkg/regexp/#Regexp.ReplaceAllString
-		"reReplaceAll": func(pattern, repl, text string) string {
-			re := regexp.MustCompile(pattern)
-			return re.ReplaceAllString(text, repl)
-		},
-
-		// match reports whether the string s
-		// contains any match of the regular expression pattern.
-		// alias for https://golang.org/pkg/regexp/#MatchString
-		"match": regexp.MatchString,
-
 		// title returns a copy of the string s with all Unicode letters
 		// that begin words mapped to their Unicode title case.
 		// alias for https://golang.org/pkg/strings/#Title
@@ -237,6 +224,31 @@ func templateFuncs() textTpl.FuncMap {
 		// alias for https://golang.org/pkg/strings/#ToLower
 		"toLower": strings.ToLower,
 
+		// crlfEscape replaces '\n' and '\r' chars with `\\n` and `\\r`.
+		// This funcion is deprectated.
+		//
+		// It is better to use quotesEscape, jsonEscape, queryEscape or pathEscape instead -
+		// these functions properly escape `\n` and `\r` chars according to their purpose.
+		"crlfEscape": func(q string) string {
+			q = strings.Replace(q, "\n", `\n`, -1)
+			return strings.Replace(q, "\r", `\r`, -1)
+		},
+
+		// quotesEscape escapes the string, so it can be safely put inside JSON string.
+		//
+		// See also jsonEscape.
+		"quotesEscape": quotesEscape,
+
+		// jsonEscape converts the string to properly encoded JSON string.
+		//
+		// See also quotesEscape.
+		"jsonEscape": jsonEscape,
+
+		// htmlEscape applies html-escaping to q, so it can be safely embedded as plaintext into html.
+		//
+		// See also safeHtml.
+		"htmlEscape": htmlEscape,
+
 		// stripPort splits string into host and port, then returns only host.
 		"stripPort": func(hostPort string) string {
 			host, _, err := net.SplitHostPort(hostPort)
@@ -246,6 +258,37 @@ func templateFuncs() textTpl.FuncMap {
 			return host
 		},
 
+		// stripDomain removes the domain part of a FQDN. Leaves port untouched.
+		"stripDomain": func(hostPort string) string {
+			host, port, err := net.SplitHostPort(hostPort)
+			if err != nil {
+				host = hostPort
+			}
+			ip := net.ParseIP(host)
+			if ip != nil {
+				return hostPort
+			}
+			host = strings.Split(host, ".")[0]
+			if port != "" {
+				return net.JoinHostPort(host, port)
+			}
+			return host
+		},
+
+		// match reports whether the string s
+		// contains any match of the regular expression pattern.
+		// alias for https://golang.org/pkg/regexp/#MatchString
+		"match": regexp.MatchString,
+
+		// reReplaceAll ReplaceAllString returns a copy of src, replacing matches of the Regexp with
+		// the replacement string repl. Inside repl, $ signs are interpreted as in Expand,
+		// so for instance $1 represents the text of the first submatch.
+		// alias for https://golang.org/pkg/regexp/#Regexp.ReplaceAllString
+		"reReplaceAll": func(pattern, repl, text string) string {
+			re := regexp.MustCompile(pattern)
+			return re.ReplaceAllString(text, repl)
+		},
+
 		// parseDuration parses a duration string such as "1h" into the number of seconds it represents
 		"parseDuration": func(s string) (float64, error) {
 			d, err := promutils.ParseDuration(s)
@@ -421,31 +464,15 @@ func templateFuncs() textTpl.FuncMap {
 			return ""
 		},
 
-		// pathEscape escapes the string so it can be safely placed inside a URL path segment,
-		// replacing special characters (including /) with %XX sequences as needed.
-		// alias for https://golang.org/pkg/net/url/#PathEscape
-		"pathEscape": func(u string) string {
-			return url.PathEscape(u)
-		},
+		// pathEscape escapes the string so it can be safely placed inside a URL path segment.
+		//
+		// See also queryEscape.
+		"pathEscape": url.PathEscape,
 
-		// queryEscape escapes the string so it can be safely placed
-		// inside a URL query.
-		// alias for https://golang.org/pkg/net/url/#QueryEscape
-		"queryEscape": func(q string) string {
-			return url.QueryEscape(q)
-		},
-
-		// crlfEscape replaces new line chars to skip URL encoding.
-		// see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/890
-		"crlfEscape": func(q string) string {
-			q = strings.Replace(q, "\n", `\n`, -1)
-			return strings.Replace(q, "\r", `\r`, -1)
-		},
-
-		// quotesEscape escapes quote char
-		"quotesEscape": func(q string) string {
-			return strings.Replace(q, `"`, `\"`, -1)
-		},
+		// queryEscape escapes the string so it can be safely placed inside a query arg in URL.
+		//
+		// See also queryEscape.
+		"queryEscape": url.QueryEscape,
 
 		// query executes the MetricsQL/PromQL query against
 		// configured `datasource.url` address.
@@ -477,6 +504,17 @@ func templateFuncs() textTpl.FuncMap {
 			return m.Labels[label]
 		},
 
+		// value returns the value of the given metric.
+		// usually used alongside with `query` template function.
+		"value": func(m metric) float64 {
+			return m.Value
+		},
+
+		// strvalue returns metric name.
+		"strvalue": func(m metric) string {
+			return m.Labels["__name__"]
+		},
+
 		// sortByLabel sorts the given metrics by provided label key
 		"sortByLabel": func(label string, metrics []metric) []metric {
 			sort.SliceStable(metrics, func(i, j int) bool {
@@ -485,12 +523,6 @@ func templateFuncs() textTpl.FuncMap {
 			return metrics
 		},
 
-		// value returns the value of the given metric.
-		// usually used alongside with `query` template function.
-		"value": func(m metric) float64 {
-			return m.Value
-		},
-
 		/* Helpers */
 
 		// Converts a list of objects to a map with keys arg0, arg1 etc.
@@ -504,6 +536,8 @@ func templateFuncs() textTpl.FuncMap {
 		},
 
 		// safeHtml marks string as HTML not requiring auto-escaping.
+		//
+		// See also htmlEscape.
 		"safeHtml": func(text string) htmlTpl.HTML {
 			return htmlTpl.HTML(text)
 		},

app/vmalert/templates/template_test.go

@@ -6,6 +6,52 @@ import (
 	textTpl "text/template"
 )
 
+func TestTemplateFuncs(t *testing.T) {
+	funcs := templateFuncs()
+	f := func(funcName, s, resultExpected string) {
+		t.Helper()
+		v := funcs[funcName]
+		fLocal := v.(func(s string) string)
+		result := fLocal(s)
+		if result != resultExpected {
+			t.Fatalf("unexpected result for %s(%q); got\n%s\nwant\n%s", funcName, s, result, resultExpected)
+		}
+	}
+	f("title", "foo bar", "Foo Bar")
+	f("toUpper", "foo", "FOO")
+	f("toLower", "FOO", "foo")
+	f("pathEscape", "foo/bar\n+baz", "foo%2Fbar%0A+baz")
+	f("queryEscape", "foo+bar\n+baz", "foo%2Bbar%0A%2Bbaz")
+	f("jsonEscape", `foo{bar="baz"}`+"\n + 1", `"foo{bar=\"baz\"}\n + 1"`)
+	f("quotesEscape", `foo{bar="baz"}`+"\n + 1", `foo{bar=\"baz\"}\n + 1`)
+	f("htmlEscape", "foo < 10\nabc", "foo &lt; 10\nabc")
+	f("crlfEscape", "foo\nbar\rx", `foo\nbar\rx`)
+	f("stripPort", "foo", "foo")
+	f("stripPort", "foo:1234", "foo")
+	f("stripDomain", "foo.bar.baz", "foo")
+	f("stripDomain", "foo.bar:123", "foo:123")
+
+	// check "match" func
+	matchFunc := funcs["match"].(func(pattern, s string) (bool, error))
+	if _, err := matchFunc("invalid[regexp", "abc"); err == nil {
+		t.Fatalf("expecting non-nil error on invalid regexp")
+	}
+	ok, err := matchFunc("abc", "def")
+	if err != nil {
+		t.Fatalf("unexpected error")
+	}
+	if ok {
+		t.Fatalf("unexpected match")
+	}
+	ok, err = matchFunc("a.+b", "acsdb")
+	if err != nil {
+		t.Fatalf("unexpected error")
+	}
+	if !ok {
+		t.Fatalf("unexpected mismatch")
+	}
+}
+
 func mkTemplate(current, replacement interface{}) textTemplate {
 	tmpl := textTemplate{}
 	if current != nil {

app/vmalert/utils.go

@@ -72,7 +72,7 @@ func (cw *curlWriter) add(str string) {
 }
 
 func requestToCurl(req *http.Request) string {
-	if req.URL == nil {
+	if req == nil || req.URL == nil {
 		return ""
 	}
 

app/vmalert/web.go

@@ -8,7 +8,6 @@ import (
 	"sort"
 	"strconv"
 	"strings"
-	"sync"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/notifier"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/tpl"
@@ -18,20 +17,14 @@ import (
 )
 
 var (
-	once     = sync.Once{}
-	apiLinks [][2]string
-	navItems []tpl.NavItem
-)
-
-func initLinks() {
 	apiLinks = [][2]string{
 		// api links are relative since they can be used by external clients,
 		// such as Grafana, and proxied via vmselect.
 		{"api/v1/rules", "list all loaded groups and rules"},
 		{"api/v1/alerts", "list all active alerts"},
 		{fmt.Sprintf("api/v1/alert?%s=<int>&%s=<int>", paramGroupID, paramAlertID), "get alert status by group and alert ID"},
-
-		// system links
+	}
+	systemLinks = [][2]string{
 		{"/flags", "command-line flags"},
 		{"/metrics", "list of application metrics"},
 		{"/-/reload", "reload configuration"},
@@ -43,7 +36,7 @@ func initLinks() {
 		{Name: "Notifiers", Url: "notifiers"},
 		{Name: "Docs", Url: "https://docs.victoriametrics.com/vmalert.html"},
 	}
-}
+)
 
 type requestHandler struct {
 	m *manager
@@ -57,10 +50,6 @@ var (
 )
 
 func (rh *requestHandler) handler(w http.ResponseWriter, r *http.Request) bool {
-	once.Do(func() {
-		initLinks()
-	})
-
 	if strings.HasPrefix(r.URL.Path, "/vmalert/static") {
 		staticServer.ServeHTTP(w, r)
 		return true

app/vmalert/web.qtpl

@@ -16,11 +16,16 @@
     <p>
         API:<br>
         {% for _, p := range apiLinks  %}
-            {%code
-                p, doc := p[0], p[1]
-            %}
-        	<a href="{%s p %}">{%s p %}</a> - {%s doc %}<br/>
+            {%code p, doc := p[0], p[1] %}
+            <a href="{%s p %}">{%s p %}</a> - {%s doc %}<br/>
         {% endfor %}
+        {% if r.Header.Get("X-Forwarded-For") == "" %}
+            System:<br>
+            {% for _, p := range systemLinks  %}
+                {%code p, doc := p[0], p[1] %}
+                <a href="{%s p %}">{%s p %}</a> - {%s doc %}<br/>
+            {% endfor %}
+        {% endif %}
     </p>
     {%= tpl.Footer(r) %}
 {% endfunc %}
@@ -248,7 +253,7 @@
                  {% for _, n := range ns %}
                      <tr>
                          <td>
-                              {% for _, l := range n.Labels %}
+                              {% for _, l := range n.Labels.GetLabels() %}
                                       <span class="ms-1 badge bg-primary">{%s l.Name %}={%s l.Value %}</span>
                               {% endfor %}
                           </td>
@@ -457,7 +462,7 @@
                  <td class="text-center">{%f.3 u.duration.Seconds() %}s</td>
                  <td class="text-center">{%s u.at.Format(time.RFC3339) %}</td>
                  <td>
-                    <textarea class="curl-area" rows="1" onclick="this.focus();this.select()">{%s requestToCurl(u.req) %}</textarea>
+                    <textarea class="curl-area" rows="1" onclick="this.focus();this.select()">{%s u.curl %}</textarea>
                 </td>
              </tr>
           </li>

app/vmalert/web_types.go

@@ -5,11 +5,11 @@ import (
 	"time"
 )
 
-// APIAlert represents a notifier.AlertingRule ruleState
+// APIAlert represents a notifier.AlertingRule state
 // for WEB view
 // https://github.com/prometheus/compliance/blob/main/alert_generator/specification.md#get-apiv1rules
 type APIAlert struct {
-	State       string            `json:"ruleState"`
+	State       string            `json:"state"`
 	Name        string            `json:"name"`
 	Value       string            `json:"value"`
 	Labels      map[string]string `json:"labels,omitempty"`
@@ -30,7 +30,7 @@ type APIAlert struct {
 	// SourceLink contains a link to a system which should show
 	// why Alert was generated
 	SourceLink string `json:"source"`
-	// Restored shows whether Alert's ruleState was restored on restart
+	// Restored shows whether Alert's state was restored on restart
 	Restored bool `json:"restored"`
 }
 
@@ -86,10 +86,10 @@ type GroupAlerts struct {
 // see https://github.com/prometheus/compliance/blob/main/alert_generator/specification.md#get-apiv1rules
 type APIRule struct {
 	// State must be one of these under following scenarios
-	//  "pending": at least 1 alert in the rule in pending ruleState and no other alert in firing ruleState.
-	//  "firing": at least 1 alert in the rule in firing ruleState.
-	//  "inactive": no alert in the rule in firing or pending ruleState.
-	State string `json:"ruleState"`
+	//  "pending": at least 1 alert in the rule in pending state and no other alert in firing ruleState.
+	//  "firing": at least 1 alert in the rule in firing state.
+	//  "inactive": no alert in the rule in firing or pending state.
+	State string `json:"state"`
 	Name  string `json:"name"`
 	// Query represents Rule's `expression` field
 	Query string `json:"query"`
@@ -121,7 +121,7 @@ type APIRule struct {
 	// GroupID is an unique Group's ID
 	GroupID string `json:"group_id"`
 
-	// TODO:
+	// Updates contains the ordered list of recorded ruleStateEntry objects
 	Updates []ruleStateEntry `json:"updates"`
 }
 

app/vmauth/README.md

@@ -167,7 +167,7 @@ It is recommended using [binary releases](https://github.com/VictoriaMetrics/Vic
 
 ### Development build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.
 2. Run `make vmauth` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `vmauth` binary and puts it into the `bin` folder.
 
@@ -239,7 +239,7 @@ See the docs at https://docs.victoriametrics.com/vmauth.html .
   -envflag.prefix string
      Prefix for environment variables if -envflag.enable is set
   -eula
-     By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf . This flag is available only in enterprise version of VictoriaMetrics
+     By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
   -flagsAuthKey string
      Auth key for /flags endpoint. It must be passed via authKey query arg. It overrides httpAuth.* settings
   -fs.disableMmap

app/vmauth/auth_config.go

@@ -250,7 +250,11 @@ func readAuthConfig(path string) (map[string]*UserInfo, error) {
 }
 
 func parseAuthConfig(data []byte) (map[string]*UserInfo, error) {
-	data = envtemplate.Replace(data)
+	var err error
+	data, err = envtemplate.ReplaceBytes(data)
+	if err != nil {
+		return nil, fmt.Errorf("cannot expand environment vars: %w", err)
+	}
 	var ac AuthConfig
 	if err := yaml.UnmarshalStrict(data, &ac); err != nil {
 		return nil, fmt.Errorf("cannot unmarshal AuthConfig data: %w", err)

app/vmbackup/README.md

@@ -2,14 +2,6 @@
 
 `vmbackup` creates VictoriaMetrics data backups from [instant snapshots](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-work-with-snapshots).
 
-Supported storage systems for backups:
-
-* [GCS](https://cloud.google.com/storage/). Example: `gs://<bucket>/<path/to/backup>`
-* [S3](https://aws.amazon.com/s3/). Example: `s3://<bucket>/<path/to/backup>`
-* [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs/). Example: `azblob://<bucket>/<path/to/backup>`
-* Any S3-compatible storage such as [MinIO](https://github.com/minio/minio), [Ceph](https://docs.ceph.com/en/pacific/radosgw/s3/) or [Swift](https://platform.swiftstack.com/docs/admin/middleware/s3_middleware.html). See [these docs](#advanced-usage) for details.
-* Local filesystem. Example: `fs://</absolute/path/to/backup>`. Note that `vmbackup` prevents from storing the backup into the directory pointed by `-storageDataPath` command-line flag, since this directory should be managed solely by VictoriaMetrics or `vmstorage`.
-
 `vmbackup` supports incremental and full backups. Incremental backups are created automatically if the destination path already contains data from the previous backup.
 Full backups can be sped up with `-origin` pointing to an already existing backup on the same remote storage. In this case `vmbackup` makes server-side copy for the shared
 data between the existing backup and new backup. It saves time and costs on data transfer.
@@ -23,6 +15,16 @@ See [this article](https://medium.com/@valyala/speeding-up-backups-for-big-time-
 See also [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager.html) tool built on top of `vmbackup`. This tool simplifies
 creation of hourly, daily, weekly and monthly backups.
 
+## Supported storage types
+
+`vmbackup` supports the following `-dst` storage types:
+
+* [GCS](https://cloud.google.com/storage/). Example: `gs://<bucket>/<path/to/backup>`
+* [S3](https://aws.amazon.com/s3/). Example: `s3://<bucket>/<path/to/backup>`
+* [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs/). Example: `azblob://<bucket>/<path/to/backup>`
+* Any S3-compatible storage such as [MinIO](https://github.com/minio/minio), [Ceph](https://docs.ceph.com/en/pacific/radosgw/s3/) or [Swift](https://platform.swiftstack.com/docs/admin/middleware/s3_middleware.html). See [these docs](#advanced-usage) for details.
+* Local filesystem. Example: `fs://</absolute/path/to/backup>`. Note that `vmbackup` prevents from storing the backup into the directory pointed by `-storageDataPath` command-line flag, since this directory should be managed solely by VictoriaMetrics or `vmstorage`.
+
 ## Use cases
 
 ### Regular backups
@@ -30,7 +32,7 @@ creation of hourly, daily, weekly and monthly backups.
 Regular backup can be performed with the following command:
 
 ```console
-vmbackup -storageDataPath=</path/to/victoria-metrics-data> -snapshot.createURL=http://localhost:8428/snapshot/create -dst=gs://<bucket>/<path/to/new/backup>
+./vmbackup -storageDataPath=</path/to/victoria-metrics-data> -snapshot.createURL=http://localhost:8428/snapshot/create -dst=gs://<bucket>/<path/to/new/backup>
 ```
 
 * `</path/to/victoria-metrics-data>` - path to VictoriaMetrics data pointed by `-storageDataPath` command-line flag in single-node VictoriaMetrics or in cluster `vmstorage`.
@@ -75,7 +77,7 @@ The command will upload only changed data to `gs://<bucket>/latest`.
 * Run the following command once a day:
 
 ```console
-vmbackup -storageDataPath=</path/to/victoria-metrics-data> -snapshot.createURL=http://localhost:8428/snapshot/create -dst=gs://<bucket>/<YYYYMMDD> -origin=gs://<bucket>/latest
+./vmbackup -storageDataPath=</path/to/victoria-metrics-data> -snapshot.createURL=http://localhost:8428/snapshot/create -dst=gs://<bucket>/<YYYYMMDD> -origin=gs://<bucket>/latest
 ```
 
 Where `<daily-snapshot>` is the snapshot for the last day `<YYYYMMDD>`.
@@ -152,6 +154,11 @@ See [this article](https://medium.com/@valyala/speeding-up-backups-for-big-time-
            "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/service-account-email"
     }
     ```
+* Obtaining credentials from env variables.
+    - For AWS S3 compatible storages set env variable `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.
+      Also you can set env variable `AWS_SHARED_CREDENTIALS_FILE` with path to credentials file.
+    - For GCE cloud storage set env variable `GOOGLE_APPLICATION_CREDENTIALS` with path to credentials file.
+    - For Azure storage either set env variables `AZURE_STORAGE_ACCOUNT_NAME` and `AZURE_STORAGE_ACCOUNT_KEY`, or `AZURE_STORAGE_ACCOUNT_CONNECTION_STRING`.
 
 * Usage with s3 custom url endpoint. It is possible to use `vmbackup` with s3 compatible storages like minio, cloudian, etc.
   You have to add a custom url endpoint via flag:
@@ -189,7 +196,7 @@ See [this article](https://medium.com/@valyala/speeding-up-backups-for-big-time-
   -envflag.prefix string
      Prefix for environment variables if -envflag.enable is set
   -eula
-     By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf . This flag is available only in enterprise version of VictoriaMetrics
+     By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
   -flagsAuthKey string
      Auth key for /flags endpoint. It must be passed via authKey query arg. It overrides httpAuth.* settings
   -fs.disableMmap
@@ -279,7 +286,7 @@ It is recommended using [binary releases](https://github.com/VictoriaMetrics/Vic
 
 ### Development build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.
 2. Run `make vmbackup` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `vmbackup` binary and puts it into the `bin` folder.
 

app/vmbackupmanager/README.md

@@ -1,17 +1,23 @@
 ## vmbackupmanager
 
-***vmbackupmanager is a part of [enterprise package](https://victoriametrics.com/products/enterprise/). It is available for download and evaluation at [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)***
+***vmbackupmanager is a part of [enterprise package](https://docs.victoriametrics.com/enterprise.html). It is available for download and evaluation at [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)***
 
-The VictoriaMetrics backup manager automates regular backup procedures. It supports the following backup intervals: **hourly**, **daily**, **weekly** and **monthly**. Multiple backup intervals may be configured simultaneously. I.e. the backup manager creates hourly backups every hour, while it creates daily backups every day, etc. Backup manager must have read access to the storage data, so best practice is to install it on the same machine (or as a sidecar) where the storage node is installed.
-The backup service makes a backup every hour and puts it to the latest folder and then copies data to the folders which represent the backup intervals (hourly, daily, weekly and monthly)
+The VictoriaMetrics backup manager automates regular backup procedures. It supports the following backup intervals: **hourly**, **daily**, **weekly** and **monthly**.
+Multiple backup intervals may be configured simultaneously. I.e. the backup manager creates hourly backups every hour, while it creates daily backups every day, etc.
+Backup manager must have read access to the storage data, so best practice is to install it on the same machine (or as a sidecar) where the storage node is installed.
+The backup service makes a backup every hour and puts it to the latest folder and then copies data to the folders
+which represent the backup intervals (hourly, daily, weekly and monthly)
 
 The required flags for running the service are as follows:
 
-* -eula - should be true and means that you have the legal right to run a backup manager. That can either be a signed contract or an email with confirmation to run the service in a trial period
-* -storageDataPath - path to VictoriaMetrics or vmstorage data path to make backup from
+* -eula - should be true and means that you have the legal right to run a backup manager. That can either be a signed contract or an email
+  with confirmation to run the service in a trial period.
+* -storageDataPath - path to VictoriaMetrics or vmstorage data path to make backup from.
 * -snapshot.createURL - VictoriaMetrics creates snapshot URL which will automatically be created during backup. Example: <http://victoriametrics:8428/snapshot/create>
-* -dst - backup destination at s3, gcs or local filesystem
-* -credsFilePath - path to file with GCS or S3 credentials. Credentials are loaded from default locations if not set. See [https://cloud.google.com/iam/docs/creating-managing-service-account-keys](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) and [https://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html](https://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html)
+* -dst - backup destination at [the supported storage types](https://docs.victoriametrics.com/vmbackup.html#supported-storage-types).
+* -credsFilePath - path to file with GCS or S3 credentials. Credentials are loaded from default locations if not set.
+  See [https://cloud.google.com/iam/docs/creating-managing-service-account-keys](https://cloud.google.com/iam/docs/creating-managing-service-account-keys)
+  and [https://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html](https://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html).
 
 Backup schedule is controlled by the following flags:
 
@@ -36,7 +42,11 @@ To get the full list of supported flags please run the following command:
 ./vmbackupmanager --help
 ```
 
-The service creates a **full** backup each run. This means that the system can be restored fully from any particular backup using vmrestore. Backup manager uploads only the data that has been changed or created since the most recent backup (incremental backup).
+The service creates a **full** backup each run. This means that the system can be restored fully
+from any particular backup using [vmrestore](https://docs.victoriametrics.com/vmrestore.html).
+Backup manager uploads only the data that has been changed or created since the most recent backup (incremental backup).
+This reduces the consumed network traffic and the time needed for performing the backup.
+See [this article](https://medium.com/@valyala/speeding-up-backups-for-big-time-series-databases-533c1a927883) for details.
 
 *Please take into account that the first backup upload could take a significant amount of time as it needs to upload all of the data.*
 
@@ -47,7 +57,7 @@ There are two flags which could help with performance tuning:
 
 ## Example of Usage
 
-GCS and cluster version. You need to have a credentials file in json format with following structure
+GCS and cluster version. You need to have a credentials file in json format with following structure:
 
 credentials.json
 
@@ -141,6 +151,142 @@ The result on the GCS bucket. We see only 3 daily backups:
 ![daily](vmbackupmanager_rp_daily_2.png)
 
 
+## API methods
+
+`vmbackupmanager` exposes the following API methods:
+
+* GET `/api/v1/backups` - returns list of backups in remote storage.
+  Example output:
+  ```json
+  ["daily/2022-10-06","daily/2022-10-10","hourly/2022-10-04:13","hourly/2022-10-06:12","hourly/2022-10-06:13","hourly/2022-10-10:14","hourly/2022-10-10:16","monthly/2022-10","weekly/2022-40","weekly/2022-41"]
+  ```
+
+* POST `/api/v1/restore` - saves backup name to restore when [performing restore](#restore-commands).
+  Example request body:
+  ```json
+  {"backup":"daily/2022-10-06"}
+  ```
+
+* GET `/api/v1/restore` - returns backup name from restore mark if it exists.
+  Example response:
+  ```json
+  {"backup":"daily/2022-10-06"}
+  ```
+
+* DELETE `/api/v1/restore` - delete restore mark.
+
+## CLI
+
+`vmbackupmanager` exposes CLI commands to work with [API methods](#api-methods) without external dependencies.
+
+Supported commands:
+```console
+vmbackupmanager backup 
+
+  vmbackupmanager backup list 
+    List backups in remote storage
+
+vmbackupmanager restore 
+  Restore backup specified by restore mark if it exists
+
+  vmbackupmanager restore get 
+    Get restore mark if it exists
+
+  vmbackupmanager restore delete 
+    Delete restore mark if it exists
+
+  vmbackupmanager restore create [backup_name]
+    Create restore mark
+```
+
+By default, CLI commands are using `http://127.0.0.1:8300` endpoint to reach `vmbackupmanager` API.
+It can be changed by using flag:
+```
+-apiURL string
+      vmbackupmanager address to perform API requests (default "http://127.0.0.1:8300")
+```
+
+### Backup commands
+
+`vmbackupmanager backup list` lists backups in remote storage:
+```console
+$ ./vmbackupmanager backup list
+["daily/2022-10-06","daily/2022-10-10","hourly/2022-10-04:13","hourly/2022-10-06:12","hourly/2022-10-06:13","hourly/2022-10-10:14","hourly/2022-10-10:16","monthly/2022-10","weekly/2022-40","weekly/2022-41"]
+```
+
+### Restore commands
+
+Restore commands are used to create, get and delete restore mark.
+Restore mark is used by `vmbackupmanager` to store backup name to restore when running restore.
+
+
+Create restore mark:
+```console
+$ ./vmbackupmanager restore create daily/2022-10-06
+```
+
+Get restore mark if it exists:
+```console
+$ ./vmbackupmanager restore get
+{"backup":"daily/2022-10-06"}
+```
+
+Delete restore mark if it exists:
+```console
+$ ./vmbackupmanager restore delete
+```
+
+Perform restore:
+```console
+$ /vmbackupmanager-prod restore -dst=gs://vmstorage-data/$NODE_IP -credsFilePath=credentials.json -storageDataPath=/vmstorage-data
+```
+Note that `vmsingle` or `vmstorage` should be stopped before performing restore.
+
+If restore mark doesn't exist at `storageDataPath`(restore wasn't requested) `vmbackupmanager restore` will exit with successful status code.
+
+### How to restore backup via CLI
+
+1. Run `vmbackupmanager backup list` to get list of available backups:
+  ```console
+  $ /vmbackupmanager-prod backup list
+  ["daily/2022-10-06","daily/2022-10-10","hourly/2022-10-04:13","hourly/2022-10-06:12","hourly/2022-10-06:13","hourly/2022-10-10:14","hourly/2022-10-10:16","monthly/2022-10","weekly/2022-40","weekly/2022-41"]
+  ```
+2. Run `vmbackupmanager restore create` to create restore mark:
+   - Use relative path to backup to restore from currently used remote storage:
+     ```console
+     $ /vmbackupmanager-prod restore create daily/2022-10-06
+     ```
+   - Use full path to backup to restore from any remote storage:
+     ```console
+     $ /vmbackupmanager-prod restore create azblob://test1/vmbackupmanager/daily/2022-10-06
+     ```
+3. Stop `vmstorage` or `vmsingle` node
+4. Run `vmbackupmanager restore` to restore backup:
+  ```console
+  $ /vmbackupmanager-prod restore -credsFilePath=credentials.json -storageDataPath=/vmstorage-data
+  ```
+5. Start `vmstorage` or `vmsingle` node
+
+
+### How to restore in Kubernetes
+
+1. Enter container running `vmbackupmanager`
+2. Use `vmbackupmanager backup list` to get list of available backups:
+  ```console
+  $ /vmbackupmanager-prod backup list
+  ["daily/2022-10-06","daily/2022-10-10","hourly/2022-10-04:13","hourly/2022-10-06:12","hourly/2022-10-06:13","hourly/2022-10-10:14","hourly/2022-10-10:16","monthly/2022-10","weekly/2022-40","weekly/2022-41"]
+  ```
+3. Use `vmbackupmanager restore create` to create restore mark:
+  - Use relative path to backup to restore from currently used remote storage:
+    ```console
+    $ /vmbackupmanager-prod restore create daily/2022-10-06
+    ```
+  - Use full path to backup to restore from any remote storage:
+    ```console
+    $ /vmbackupmanager-prod restore create azblob://test1/vmbackupmanager/daily/2022-10-06
+    ```
+4. Restart pod
+
 ## Configuration
 
 ### Flags
@@ -153,6 +299,13 @@ The shortlist of configuration flags is the following:
 ```
 vmbackupmanager performs regular backups according to the provided configs.
 
+subcommands:
+ backup: provides auxiliary backup-related commands
+ restore: restores backup specified by restore mark if it exists
+
+command-line flags:
+  -apiURL string
+     vmbackupmanager address to perform API requests (default "http://127.0.0.1:8300")
   -concurrency int
      The number of concurrent workers. Higher concurrency may reduce backup duration (default 10)
   -configFilePath string
@@ -182,7 +335,7 @@ vmbackupmanager performs regular backups according to the provided configs.
   -envflag.prefix string
      Prefix for environment variables if -envflag.enable is set
   -eula
-     By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf . This flag is available only in enterprise version of VictoriaMetrics
+     By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
   -flagsAuthKey string
      Auth key for /flags endpoint. It must be passed via authKey query arg. It overrides httpAuth.* settings
   -fs.disableMmap

app/vmctl/README.md

@@ -7,9 +7,12 @@ vmctl provides various useful actions with VictoriaMetrics components.
 Features:
 - migrate data from [Prometheus](#migrating-data-from-prometheus) to VictoriaMetrics using snapshot API
 - migrate data from [Thanos](#migrating-data-from-thanos) to VictoriaMetrics
+- migrate data from [Cortex](#migrating-data-from-cortex) to VictoriaMetrics
+- migrate data from [Mimir](#migrating-data-from-mimir) to VictoriaMetrics
 - migrate data from [InfluxDB](#migrating-data-from-influxdb-1x) to VictoriaMetrics
 - migrate data from [OpenTSDB](#migrating-data-from-opentsdb) to VictoriaMetrics
 - migrate data between [VictoriaMetrics](#migrating-data-from-victoriametrics) single or cluster version.
+- migrate data by [Prometheus remote read protocol](#migrating-data-by-remote-read-protocol) to VictoriaMetrics
 - [verify](#verifying-exported-blocks-from-victoriametrics) exported blocks from VictoriaMetrics single or cluster version.
 
 To see the full list of supported modes
@@ -28,6 +31,7 @@ COMMANDS:
    influx      Migrate timeseries from InfluxDB
    prometheus  Migrate timeseries from Prometheus
    vm-native   Migrate time series between VictoriaMetrics installations via native binary format
+   remote-read Migrate timeseries by Prometheus remote read protocol
    verify-block  Verifies correctness of data blocks exported via VictoriaMetrics Native format. See https://docs.victoriametrics.com/#how-to-export-data-in-native-format
 ```
 
@@ -432,6 +436,64 @@ Found 2 blocks to import. Continue? [Y/n] y
 2020/02/23 15:51:07 Total time: 7.153158218s
 ```
 
+## Migrating data by remote read protocol
+
+`vmctl` supports the `remote-read` mode for migrating data from databases which support 
+[Prometheus remote read API](https://prometheus.io/docs/prometheus/latest/querying/remote_read_api/)
+
+See `./vmctl remote-read --help` for details and full list of flags.
+
+To start the migration process configure the following flags:
+1. `--remote-read-src-addr` - data source address to read from;
+2. `--vm-addr` - VictoriaMetrics address to write to. For single-node VM is usually equal to `--httpListenAddr`, 
+and for cluster version is equal to `--httpListenAddr` flag of vminsert component (for example `http://<vminsert>:8480/insert/<accountID>/prometheus`);
+3. `--remote-read-filter-time-start` - the time filter in RFC3339 format to select time series with timestamp equal or higher than provided value. E.g. '2020-01-01T20:07:00Z';
+4. `--remote-read-filter-time-end` - the time filter in RFC3339 format to select time series with timestamp equal or smaller than provided value. E.g. '2020-01-01T20:07:00Z'. Current time is used when omitted.;
+5. `--remote-read-step-interval` - split export data into chunks. Valid values are `month, day, hour, minute`;
+
+The importing process example for local installation of Prometheus
+and single-node VictoriaMetrics(`http://localhost:8428`):
+
+```
+./vmctl remote-read \
+--remote-read-src-addr=http://127.0.0.1:9091 \
+--remote-read-filter-time-start=2021-10-18T00:00:00Z \
+--remote-read-step-interval=hour \
+--vm-addr=http://127.0.0.1:8428 \
+--vm-concurrency=6
+
+Split defined times into 8798 ranges to import. Continue? [Y/n]
+VM worker 0:↘ 127177 samples/s
+VM worker 1:↘ 140137 samples/s
+VM worker 2:↘ 151606 samples/s
+VM worker 3:↘ 130765 samples/s
+VM worker 4:↘ 131904 samples/s
+VM worker 5:↘ 132693 samples/s
+Processing ranges: 8798 / 8798 [█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████] 100.00%
+2022/10/19 16:45:37 Import finished!
+2022/10/19 16:45:37 VictoriaMetrics importer stats:
+  idle duration: 6m57.793987511s;
+  time spent while importing: 1m18.463744801s;
+  total samples: 25348208;
+  samples/s: 323056.31;
+  total bytes: 669.7 MB;
+  bytes/s: 8.5 MB;
+  import requests: 127;
+  import requests retries: 0;
+2022/10/19 16:45:37 Total time: 1m19.406283424s
+```
+
+### Filtering
+
+The filtering consists of two parts: by labels and time.
+
+Filtering by time can be configured via flags `--remote-read-filter-time-start` and `--remote-read-filter-time-end`
+in RFC3339 format.
+
+Filtering by labels can be configured via flags `--remote-read-filter-label` and `--remote-read-filter-label-value`.
+For example, `--remote-read-filter-label=tenant` and `--remote-read-filter-label-value="team-eu"` will select only series
+with `tenant="team-eu"` label-value pair.
+
 ## Migrating data from Thanos
 
 Thanos uses the same storage engine as Prometheus and the data layout on-disk should be the same. That means
@@ -478,6 +540,187 @@ then import it into VM using `vmctl` in `prometheus` mode.
     vmctl prometheus --prom-snapshot thanos-data --vm-addr http://victoria-metrics:8428
     ```
 
+### Remote read protocol
+
+Currently, Thanos doesn't support streaming remote read protocol. It is [recommended](https://thanos.io/tip/thanos/integrations.md/#storeapi-as-prometheus-remote-read)
+to use [thanos-remote-read](https://github.com/G-Research/thanos-remote-read) a proxy, that allows exposing any Thanos 
+service (or anything that exposes gRPC StoreAPI e.g. Querier) via Prometheus remote read protocol.
+
+If you want to migrate data, you should run [thanos-remote-read](https://github.com/G-Research/thanos-remote-read) proxy
+and define the Thanos store address `./thanos-remote-read -store 127.0.0.1:19194`. 
+It is important to know that `store` flag is Thanos Store API gRPC endpoint. 
+Also, it is important to know that thanos-remote-read proxy  doesn't support `STREAMED_XOR_CHUNKS` mode.
+When you run thanos-remote-read proxy, it exposes port to serve HTTP on `10080 by default`.
+
+The importing process example for local installation of Thanos
+and single-node VictoriaMetrics(`http://localhost:8428`):
+
+```
+./vmctl remote-read \
+--remote-read-src-addr=http://127.0.0.1:10080 \
+--remote-read-filter-time-start=2021-10-18T00:00:00Z \
+--remote-read-step-interval=hour \
+--vm-addr=http://127.0.0.1:8428 \
+--vm-concurrency=6
+```
+
+On the [thanos-remote-read](https://github.com/G-Research/thanos-remote-read) proxy side you will see logs like:
+```
+ts=2022-10-19T15:05:04.193916Z caller=main.go:278 level=info traceID=00000000000000000000000000000000 msg="thanos request" request="min_time:1666180800000 max_time:1666184399999 matchers:<type:RE value:\".*\" > aggregates:RAW "
+ts=2022-10-19T15:05:04.468852Z caller=main.go:278 level=info traceID=00000000000000000000000000000000 msg="thanos request" request="min_time:1666184400000 max_time:1666187999999 matchers:<type:RE value:\".*\" > aggregates:RAW "
+ts=2022-10-19T15:05:04.553914Z caller=main.go:278 level=info traceID=00000000000000000000000000000000 msg="thanos request" request="min_time:1666188000000 max_time:1666191364863 matchers:<type:RE value:\".*\" > aggregates:RAW "
+```
+
+And when process will finish you will see:
+```
+Split defined times into 8799 ranges to import. Continue? [Y/n]
+VM worker 0:↓ 98183 samples/s
+VM worker 1:↓ 114640 samples/s
+VM worker 2:↓ 131710 samples/s
+VM worker 3:↓ 114256 samples/s
+VM worker 4:↓ 105671 samples/s
+VM worker 5:↓ 124000 samples/s
+Processing ranges: 8799 / 8799 [█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████] 100.00%
+2022/10/19 18:05:07 Import finished!
+2022/10/19 18:05:07 VictoriaMetrics importer stats:
+  idle duration: 52m13.987637229s;
+  time spent while importing: 9m1.728983776s;
+  total samples: 70836111;
+  samples/s: 130759.32;
+  total bytes: 2.2 GB;
+  bytes/s: 4.0 MB;
+  import requests: 356;
+  import requests retries: 0;
+2022/10/19 18:05:07 Total time: 9m2.607521618s
+```
+
+## Migrating data from Cortex
+
+Cortex has an implementation of the Prometheus remote read protocol. That means
+`vmctl` in mode `remote-read` may also be used for Cortex historical data migration.
+These instructions may vary based on the details of your Cortex configuration.
+Please read carefully and verify as you go.
+
+### Remote read protocol
+
+If you want to migrate data, you should check your cortex configuration in the section
+```yaml
+api:
+  prometheus_http_prefix:
+```
+
+If you defined some prometheus prefix, you should use it when you define flag `--remote-read-src-addr=http://127.0.0.1:9009/{prometheus_http_prefix}`.
+By default, Cortex uses the `prometheus` path prefix, so you should define the flag `--remote-read-src-addr=http://127.0.0.1:9009/prometheus`.
+
+It is important to know that Cortex doesn't support the `STREAMED_XOR_CHUNKS` mode.
+When you run Cortex, it exposes a port to serve HTTP on `9009 by default`.
+
+The importing process example for the local installation of Cortex
+and single-node VictoriaMetrics(`http://localhost:8428`):
+
+```
+./vmctl remote-read \ 
+--remote-read-src-addr=http://127.0.0.1:9009/prometheus \
+--remote-read-filter-time-start=2021-10-18T00:00:00Z \
+--remote-read-step-interval=hour \
+--remote-read-src-check-alive=false \
+--vm-addr=http://127.0.0.1:8428 \
+--vm-concurrency=6 
+```
+And when the process finishes, you will see the following:
+
+```
+Split defined times into 8842 ranges to import. Continue? [Y/n]
+VM worker 0:↗ 3863 samples/s
+VM worker 1:↗ 2686 samples/s
+VM worker 2:↗ 2620 samples/s
+VM worker 3:↗ 2705 samples/s
+VM worker 4:↗ 2643 samples/s
+VM worker 5:↗ 2593 samples/s
+Processing ranges: 8842 / 8842 [█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████] 100.00%
+2022/10/21 12:09:49 Import finished!
+2022/10/21 12:09:49 VictoriaMetrics importer stats:
+  idle duration: 0s;
+  time spent while importing: 3.82640757s;
+  total samples: 160232;
+  samples/s: 41875.31;
+  total bytes: 11.3 MB;
+  bytes/s: 3.0 MB;
+  import requests: 6;
+  import requests retries: 0;
+2022/10/21 12:09:49 Total time: 4.71824253s
+```
+It is important to know that if you run your Cortex installation in multi-tenant mode, remote read protocol
+requires an Authentication header like `X-Scope-OrgID`. You can define it via the flag `--remote-read-headers=X-Scope-OrgID:demo`
+
+## Migrating data from Mimir
+
+Mimir has similar implemintation as Cortex and also support of the Prometheus remote read protocol. That means
+`vmctl` in mode `remote-read` may also be used for Mimir historical data migration.
+These instructions may vary based on the details of your Mimir configuration.
+Please read carefully and verify as you go.
+
+### Remote read protocol
+
+If you want to migrate data, you should check your Mimir configuration in the section
+```yaml
+api:
+  prometheus_http_prefix:
+```
+
+If you defined some prometheus prefix, you should use it when you define flag `--remote-read-src-addr=http://127.0.0.1:9009/{prometheus_http_prefix}`.
+By default, Mimir uses the `prometheus` path prefix, so you should define the flag `--remote-read-src-addr=http://127.0.0.1:9009/prometheus`.
+
+Mimir supports both remote read mode, so you can use `STREAMED_XOR_CHUNKS` mode and `SAMPLES` mode.
+When you run Mimir, it exposes a port to serve HTTP on `8080 by default`.
+
+Next example of the local installation was in multi-tenant mode (3 instances of mimir) with nginx as load balancer.
+Load balancer expose single port `:9090`.
+
+As you can see in the example we call `:9009` instead of `:8080` because of proxy.
+
+The importing process example for the local installation of Mimir
+and single-node VictoriaMetrics(`http://localhost:8428`):
+
+```
+./vmctl remote-read 
+--remote-read-src-addr=http://127.0.0.1:9009/prometheus \
+--remote-read-filter-time-start=2021-10-18T00:00:00Z \
+--remote-read-step-interval=hour \
+--remote-read-src-check-alive=false \
+--remote-read-headers=X-Scope-OrgID:demo \
+--remote-read-use-stream=true \
+--vm-addr=http://127.0.0.1:8428 \
+--vm-concurrency=6
+```
+
+And when the process finishes, you will see the following:
+
+```
+Split defined times into 8847 ranges to import. Continue? [Y/n]
+VM worker 0:→ 12176 samples/s
+VM worker 1:→ 11918 samples/s
+VM worker 2:→ 11261 samples/s
+VM worker 3:→ 12861 samples/s
+VM worker 4:→ 11096 samples/s
+VM worker 5:→ 11575 samples/s
+Processing ranges: 8847 / 8847 [█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████] 100.00%
+2022/10/21 17:22:23 Import finished!
+2022/10/21 17:22:23 VictoriaMetrics importer stats:
+  idle duration: 0s;
+  time spent while importing: 15.379614356s;
+  total samples: 81243;
+  samples/s: 5282.51;
+  total bytes: 6.1 MB;
+  bytes/s: 397.8 kB;
+  import requests: 6;
+  import requests retries: 0;
+2022/10/21 17:22:23 Total time: 16.287405248s
+```
+
+It is important to know that if you run your Mimir installation in multi-tenant mode, remote read protocol
+requires an Authentication header like `X-Scope-OrgID`. You can define it via the flag `--remote-read-headers=X-Scope-OrgID:demo`
+
 ## Migrating data from VictoriaMetrics
 
 ### Native protocol
@@ -590,6 +833,80 @@ Total: 16 B ↗ Speed: 186.32 KiB p/s
 2022/08/30 19:48:24 Total time: 12.680582ms
 ```
 
+#### Cluster-to-cluster migration mode
+
+Using cluster-to-cluster migration mode helps to migrate all tenants data in a single `vmctl` run.
+
+Cluster-to-cluster uses `/admin/tenants` endpoint (available starting from [v1.84.0](https://docs.victoriametrics.com/CHANGELOG.html#v1840)) to discover list of tenants from source cluster.
+
+To use this mode you need to set `--vm-intercluster` flag to `true`, `--vm-native-src-addr` flag to 'http://vmselect:8481/' and `--vm-native-dst-addr` value to http://vminsert:8480/:
+
+```console
+./bin/vmctl vm-native --vm-intercluster=true --vm-native-src-addr=http://localhost:8481/ --vm-native-dst-addr=http://172.17.0.3:8480/
+VictoriaMetrics Native import mode
+2022/12/05 21:20:06 Discovered tenants: [123:1 12812919:1 1289198:1 1289:1283 12:1 1:0 1:1 1:1231231 1:1271727 1:12819 1:281 812891298:1]
+2022/12/05 21:20:06 Initing export pipe from "http://localhost:8481/select/123:1/prometheus/api/v1/export/native" with filters: 
+        filter: match[]={__name__!=""}
+Initing import process to "http://172.17.0.3:8480/insert/123:1/prometheus/api/v1/import/native":
+Total: 61.13 MiB ↖ Speed: 2.05 MiB p/s 
+Total: 61.13 MiB ↗ Speed: 2.30 MiB p/s 
+2022/12/05 21:20:33 Initing export pipe from "http://localhost:8481/select/12812919:1/prometheus/api/v1/export/native" with filters: 
+        filter: match[]={__name__!=""}
+Initing import process to "http://172.17.0.3:8480/insert/12812919:1/prometheus/api/v1/import/native":
+Total: 43.14 MiB ↘ Speed: 1.86 MiB p/s  
+Total: 43.14 MiB ↙ Speed: 2.36 MiB p/s 
+2022/12/05 21:20:51 Initing export pipe from "http://localhost:8481/select/1289198:1/prometheus/api/v1/export/native" with filters: 
+        filter: match[]={__name__!=""}
+Initing import process to "http://172.17.0.3:8480/insert/1289198:1/prometheus/api/v1/import/native":
+Total: 16.64 MiB ↗ Speed: 2.66 MiB p/s 
+Total: 16.64 MiB ↘ Speed: 2.19 MiB p/s 
+2022/12/05 21:20:59 Initing export pipe from "http://localhost:8481/select/1289:1283/prometheus/api/v1/export/native" with filters: 
+        filter: match[]={__name__!=""}
+Initing import process to "http://172.17.0.3:8480/insert/1289:1283/prometheus/api/v1/import/native":
+Total: 43.33 MiB ↙ Speed: 1.94 MiB p/s 
+Total: 43.33 MiB ↖ Speed: 2.35 MiB p/s 
+2022/12/05 21:21:18 Initing export pipe from "http://localhost:8481/select/12:1/prometheus/api/v1/export/native" with filters: 
+        filter: match[]={__name__!=""}
+Initing import process to "http://172.17.0.3:8480/insert/12:1/prometheus/api/v1/import/native":
+Total: 63.78 MiB ↙ Speed: 1.96 MiB p/s 
+Total: 63.78 MiB ↖ Speed: 2.28 MiB p/s 
+2022/12/05 21:21:46 Initing export pipe from "http://localhost:8481/select/1:0/prometheus/api/v1/export/native" with filters: 
+        filter: match[]={__name__!=""}
+Initing import process to "http://172.17.0.3:8480/insert/1:0/prometheus/api/v1/import/native":
+2022/12/05 21:21:46 Import finished!
+Total: 330 B ↗ Speed: 3.53 MiB p/s 
+2022/12/05 21:21:46 Initing export pipe from "http://localhost:8481/select/1:1/prometheus/api/v1/export/native" with filters: 
+        filter: match[]={__name__!=""}
+Initing import process to "http://172.17.0.3:8480/insert/1:1/prometheus/api/v1/import/native":
+Total: 63.81 MiB ↙ Speed: 1.96 MiB p/s 
+Total: 63.81 MiB ↖ Speed: 2.28 MiB p/s 
+2022/12/05 21:22:14 Initing export pipe from "http://localhost:8481/select/1:1231231/prometheus/api/v1/export/native" with filters: 
+        filter: match[]={__name__!=""}
+Initing import process to "http://172.17.0.3:8480/insert/1:1231231/prometheus/api/v1/import/native":
+Total: 63.84 MiB ↙ Speed: 1.93 MiB p/s 
+Total: 63.84 MiB ↖ Speed: 2.29 MiB p/s 
+2022/12/05 21:22:42 Initing export pipe from "http://localhost:8481/select/1:1271727/prometheus/api/v1/export/native" with filters: 
+        filter: match[]={__name__!=""}
+Initing import process to "http://172.17.0.3:8480/insert/1:1271727/prometheus/api/v1/import/native":
+Total: 54.37 MiB ↘ Speed: 1.90 MiB p/s 
+Total: 54.37 MiB ↙ Speed: 2.37 MiB p/s 
+2022/12/05 21:23:05 Initing export pipe from "http://localhost:8481/select/1:12819/prometheus/api/v1/export/native" with filters: 
+        filter: match[]={__name__!=""}
+Initing import process to "http://172.17.0.3:8480/insert/1:12819/prometheus/api/v1/import/native":
+Total: 17.01 MiB ↙ Speed: 1.75 MiB p/s 
+Total: 17.01 MiB ↖ Speed: 2.15 MiB p/s 
+2022/12/05 21:23:13 Initing export pipe from "http://localhost:8481/select/1:281/prometheus/api/v1/export/native" with filters: 
+        filter: match[]={__name__!=""}
+Initing import process to "http://172.17.0.3:8480/insert/1:281/prometheus/api/v1/import/native":
+Total: 63.89 MiB ↘ Speed: 1.90 MiB p/s 
+Total: 63.89 MiB ↙ Speed: 2.29 MiB p/s 
+2022/12/05 21:23:42 Initing export pipe from "http://localhost:8481/select/812891298:1/prometheus/api/v1/export/native" with filters: 
+        filter: match[]={__name__!=""}
+Initing import process to "http://172.17.0.3:8480/insert/812891298:1/prometheus/api/v1/import/native":
+Total: 63.84 MiB ↖ Speed: 1.99 MiB p/s 
+Total: 63.84 MiB ↗ Speed: 2.26 MiB p/s 
+2022/12/05 21:24:10 Total time: 4m4.1466565s
+```
 
 ## Verifying exported blocks from VictoriaMetrics
 
@@ -700,7 +1017,7 @@ It is recommended using [binary releases](https://github.com/VictoriaMetrics/Vic
 
 ### Development build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.
 2. Run `make vmctl` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `vmctl` binary and puts it into the `bin` folder.
 
@@ -729,7 +1046,7 @@ ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://b
 
 #### Development ARM build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.
 2. Run `make vmctl-linux-arm` or `make vmctl-linux-arm64` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `vmctl-linux-arm` or `vmctl-linux-arm64` binary respectively and puts it into the `bin` folder.
 

app/vmctl/flags.go

@@ -2,6 +2,7 @@ package main
 
 import (
 	"fmt"
+	"time"
 
 	"github.com/urfave/cli/v2"
 
@@ -43,6 +44,8 @@ const (
 	// also used in vm-native
 	vmExtraLabel = "vm-extra-label"
 	vmRateLimit  = "vm-rate-limit"
+
+	vmInterCluster = "vm-intercluster"
 )
 
 var (
@@ -397,6 +400,99 @@ var (
 			Usage: "Optional data transfer rate limit in bytes per second.\n" +
 				"By default the rate limit is disabled. It can be useful for limiting load on source or destination databases.",
 		},
+		&cli.BoolFlag{
+			Name: vmInterCluster,
+			Usage: "Enables cluster-to-cluster migration mode with automatic tenants data migration.\n" +
+				fmt.Sprintf(" In this mode --%s flag format is: 'http://vmselect:8481/'. --%s flag format is: http://vminsert:8480/. \n", vmNativeSrcAddr, vmNativeDstAddr) +
+				" TenantID will be appended automatically after discovering tenants from src.",
+		},
+	}
+)
+
+const (
+	remoteRead                 = "remote-read"
+	remoteReadUseStream        = "remote-read-use-stream"
+	remoteReadConcurrency      = "remote-read-concurrency"
+	remoteReadFilterTimeStart  = "remote-read-filter-time-start"
+	remoteReadFilterTimeEnd    = "remote-read-filter-time-end"
+	remoteReadFilterLabel      = "remote-read-filter-label"
+	remoteReadFilterLabelValue = "remote-read-filter-label-value"
+	remoteReadStepInterval     = "remote-read-step-interval"
+	remoteReadSrcAddr          = "remote-read-src-addr"
+	remoteReadUser             = "remote-read-user"
+	remoteReadPassword         = "remote-read-password"
+	remoteReadHTTPTimeout      = "remote-read-http-timeout"
+	remoteReadHeaders          = "remote-read-headers"
+)
+
+var (
+	remoteReadFlags = []cli.Flag{
+		&cli.IntFlag{
+			Name:  remoteReadConcurrency,
+			Usage: "Number of concurrently running remote read readers",
+			Value: 1,
+		},
+		&cli.TimestampFlag{
+			Name:   remoteReadFilterTimeStart,
+			Usage:  "The time filter in RFC3339 format to select timeseries with timestamp equal or higher than provided value. E.g. '2020-01-01T20:07:00Z'",
+			Layout: time.RFC3339,
+		},
+		&cli.TimestampFlag{
+			Name:   remoteReadFilterTimeEnd,
+			Usage:  "The time filter in RFC3339 format to select timeseries with timestamp equal or lower than provided value. E.g. '2020-01-01T20:07:00Z'",
+			Layout: time.RFC3339,
+		},
+		&cli.StringFlag{
+			Name:  remoteReadFilterLabel,
+			Usage: "Prometheus label name to filter timeseries by. E.g. '__name__' will filter timeseries by name.",
+			Value: "__name__",
+		},
+		&cli.StringFlag{
+			Name:  remoteReadFilterLabelValue,
+			Usage: fmt.Sprintf("Prometheus regular expression to filter label from %q flag.", remoteReadFilterLabelValue),
+			Value: ".*",
+		},
+		&cli.BoolFlag{
+			Name:  remoteRead,
+			Usage: "Use Prometheus remote read protocol",
+			Value: false,
+		},
+		&cli.BoolFlag{
+			Name:  remoteReadUseStream,
+			Usage: "Defines whether to use SAMPLES or STREAMED_XOR_CHUNKS mode. By default is uses SAMPLES mode. See https://prometheus.io/docs/prometheus/latest/querying/remote_read_api/#streamed-chunks",
+			Value: false,
+		},
+		&cli.StringFlag{
+			Name:     remoteReadStepInterval,
+			Usage:    fmt.Sprintf("Split export data into chunks. Requires setting --%s. Valid values are %q,%q,%q,%q.", remoteReadFilterTimeStart, stepper.StepMonth, stepper.StepDay, stepper.StepHour, stepper.StepMinute),
+			Required: true,
+		},
+		&cli.StringFlag{
+			Name:     remoteReadSrcAddr,
+			Usage:    "Remote read address to perform read from.",
+			Required: true,
+		},
+		&cli.StringFlag{
+			Name:    remoteReadUser,
+			Usage:   "Remote read username for basic auth",
+			EnvVars: []string{"REMOTE_READ_USERNAME"},
+		},
+		&cli.StringFlag{
+			Name:    remoteReadPassword,
+			Usage:   "Remote read password for basic auth",
+			EnvVars: []string{"REMOTE_READ_PASSWORD"},
+		},
+		&cli.DurationFlag{
+			Name:  remoteReadHTTPTimeout,
+			Usage: "Timeout defines timeout for HTTP write request to remote storage",
+		},
+		&cli.StringFlag{
+			Name:  remoteReadHeaders,
+			Value: "",
+			Usage: "Optional HTTP headers to send with each request to the corresponding remote source storage \n" +
+				"For example, --remote-read-headers='My-Auth:foobar' would send 'My-Auth: foobar' HTTP header with every request to the corresponding remote source storage. \n" +
+				"Multiple headers must be delimited by '^^': --remote-read-headers='header1:value1^^header2:value2'",
+		},
 	}
 )
 

app/vmctl/main.go

@@ -11,6 +11,7 @@ import (
 	"syscall"
 	"time"
 
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/remoteread"
 	"github.com/urfave/cli/v2"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/influx"
@@ -34,10 +35,13 @@ func main() {
 		Name:    "vmctl",
 		Usage:   "VictoriaMetrics command-line tool",
 		Version: buildinfo.Version,
+		// Disable `-version` flag to avoid conflict with lib/buildinfo flags
+		// see https://github.com/urfave/cli/issues/1560
+		HideVersion: true,
 		Commands: []*cli.Command{
 			{
 				Name:  "opentsdb",
-				Usage: "Migrate timeseries from OpenTSDB",
+				Usage: "Migrate time series from OpenTSDB",
 				Flags: mergeFlags(globalFlags, otsdbFlags, vmFlags),
 				Action: func(c *cli.Context) error {
 					fmt.Println("OpenTSDB import mode")
@@ -72,7 +76,7 @@ func main() {
 			},
 			{
 				Name:  "influx",
-				Usage: "Migrate timeseries from InfluxDB",
+				Usage: "Migrate time series from InfluxDB",
 				Flags: mergeFlags(globalFlags, influxFlags, vmFlags),
 				Action: func(c *cli.Context) error {
 					fmt.Println("InfluxDB import mode")
@@ -111,9 +115,48 @@ func main() {
 					return processor.run(c.Bool(globalSilent), c.Bool(globalVerbose))
 				},
 			},
+			{
+				Name:  "remote-read",
+				Usage: "Migrate time series via Prometheus remote-read protocol",
+				Flags: mergeFlags(globalFlags, remoteReadFlags, vmFlags),
+				Action: func(c *cli.Context) error {
+					rr, err := remoteread.NewClient(remoteread.Config{
+						Addr:       c.String(remoteReadSrcAddr),
+						Username:   c.String(remoteReadUser),
+						Password:   c.String(remoteReadPassword),
+						Timeout:    c.Duration(remoteReadHTTPTimeout),
+						UseStream:  c.Bool(remoteReadUseStream),
+						Headers:    c.String(remoteReadHeaders),
+						LabelName:  c.String(remoteReadFilterLabel),
+						LabelValue: c.String(remoteReadFilterLabelValue),
+					})
+					if err != nil {
+						return fmt.Errorf("error create remote read client: %s", err)
+					}
+
+					vmCfg := initConfigVM(c)
+
+					importer, err := vm.NewImporter(vmCfg)
+					if err != nil {
+						return fmt.Errorf("failed to create VM importer: %s", err)
+					}
+
+					rmp := remoteReadProcessor{
+						src: rr,
+						dst: importer,
+						filter: remoteReadFilter{
+							timeStart: c.Timestamp(remoteReadFilterTimeStart),
+							timeEnd:   c.Timestamp(remoteReadFilterTimeEnd),
+							chunk:     c.String(remoteReadStepInterval),
+						},
+						cc: c.Int(remoteReadConcurrency),
+					}
+					return rmp.run(ctx, c.Bool(globalSilent), c.Bool(globalVerbose))
+				},
+			},
 			{
 				Name:  "prometheus",
-				Usage: "Migrate timeseries from Prometheus",
+				Usage: "Migrate time series from Prometheus",
 				Flags: mergeFlags(globalFlags, promFlags, vmFlags),
 				Action: func(c *cli.Context) error {
 					fmt.Println("Prometheus import mode")
@@ -157,7 +200,8 @@ func main() {
 					}
 
 					p := vmNativeProcessor{
-						rateLimit: c.Int64(vmRateLimit),
+						rateLimit:    c.Int64(vmRateLimit),
+						interCluster: c.Bool(vmInterCluster),
 						filter: filter{
 							match:     c.String(vmNativeFilterMatch),
 							timeStart: c.String(vmNativeFilterTimeStart),

app/vmctl/prometheus.go

@@ -9,6 +9,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/prometheus"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/vm"
 	"github.com/prometheus/prometheus/tsdb"
+	"github.com/prometheus/prometheus/tsdb/chunkenc"
 )
 
 type prometheusProcessor struct {
@@ -123,7 +124,15 @@ func (pp *prometheusProcessor) do(b tsdb.BlockReader) error {
 		var timestamps []int64
 		var values []float64
 		it := series.Iterator()
-		for it.Next() {
+		for {
+			typ := it.Next()
+			if typ == chunkenc.ValNone {
+				break
+			}
+			if typ != chunkenc.ValFloat {
+				// Skip unsupported values
+				continue
+			}
 			t, v := it.At()
 			timestamps = append(timestamps, t)
 			values = append(values, v)

app/vmctl/prometheus/prometheus.go

@@ -4,7 +4,7 @@ import (
 	"fmt"
 	"time"
 
-	"github.com/prometheus/prometheus/pkg/labels"
+	"github.com/prometheus/prometheus/model/labels"
 	"github.com/prometheus/prometheus/storage"
 	"github.com/prometheus/prometheus/tsdb"
 )

app/vmctl/remoteread.go

@@ -0,0 +1,127 @@
+package main
+
+import (
+	"context"
+	"fmt"
+	"log"
+	"sync"
+	"time"
+
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/barpool"
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/remoteread"
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/stepper"
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/vm"
+	"github.com/cheggaaa/pb/v3"
+)
+
+type remoteReadProcessor struct {
+	filter remoteReadFilter
+
+	dst *vm.Importer
+	src *remoteread.Client
+
+	cc int
+}
+
+type remoteReadFilter struct {
+	timeStart *time.Time
+	timeEnd   *time.Time
+	chunk     string
+}
+
+func (rrp *remoteReadProcessor) run(ctx context.Context, silent, verbose bool) error {
+	rrp.dst.ResetStats()
+	if rrp.filter.timeEnd == nil {
+		t := time.Now().In(rrp.filter.timeStart.Location())
+		rrp.filter.timeEnd = &t
+	}
+	if rrp.cc < 1 {
+		rrp.cc = 1
+	}
+
+	ranges, err := stepper.SplitDateRange(*rrp.filter.timeStart, *rrp.filter.timeEnd, rrp.filter.chunk)
+	if err != nil {
+		return fmt.Errorf("failed to create date ranges for the given time filters: %v", err)
+	}
+
+	question := fmt.Sprintf("Selected time range %q - %q will be split into %d ranges according to %q step. Continue?",
+		rrp.filter.timeStart.String(), rrp.filter.timeEnd.String(), len(ranges), rrp.filter.chunk)
+	if !silent && !prompt(question) {
+		return nil
+	}
+
+	var bar *pb.ProgressBar
+	if !silent {
+		bar = barpool.AddWithTemplate(fmt.Sprintf(barTpl, "Processing ranges"), len(ranges))
+		if err := barpool.Start(); err != nil {
+			return err
+		}
+	}
+	defer func() {
+		if !silent {
+			barpool.Stop()
+		}
+		log.Println("Import finished!")
+		log.Print(rrp.dst.Stats())
+	}()
+
+	rangeC := make(chan *remoteread.Filter)
+	errCh := make(chan error)
+
+	var wg sync.WaitGroup
+	wg.Add(rrp.cc)
+	for i := 0; i < rrp.cc; i++ {
+		go func() {
+			defer wg.Done()
+			for r := range rangeC {
+				if err := rrp.do(ctx, r); err != nil {
+					errCh <- fmt.Errorf("request failed for: %s", err)
+					return
+				}
+				if bar != nil {
+					bar.Increment()
+				}
+			}
+		}()
+	}
+
+	for _, r := range ranges {
+		select {
+		case infErr := <-errCh:
+			return fmt.Errorf("remote read error: %s", infErr)
+		case vmErr := <-rrp.dst.Errors():
+			return fmt.Errorf("import process failed: %s", wrapErr(vmErr, verbose))
+		case rangeC <- &remoteread.Filter{
+			StartTimestampMs: r[0].UnixMilli(),
+			EndTimestampMs:   r[1].UnixMilli(),
+		}:
+		}
+	}
+
+	close(rangeC)
+	wg.Wait()
+	rrp.dst.Close()
+	close(errCh)
+	// drain import errors channel
+	for vmErr := range rrp.dst.Errors() {
+		if vmErr.Err != nil {
+			return fmt.Errorf("import process failed: %s", wrapErr(vmErr, verbose))
+		}
+	}
+	for err := range errCh {
+		return fmt.Errorf("import process failed: %s", err)
+	}
+
+	return nil
+}
+
+func (rrp *remoteReadProcessor) do(ctx context.Context, filter *remoteread.Filter) error {
+	return rrp.src.Read(ctx, filter, func(series *vm.TimeSeries) error {
+		if err := rrp.dst.Input(series); err != nil {
+			return fmt.Errorf(
+				"failed to read data for time range start: %d, end: %d, %s",
+				filter.StartTimestampMs, filter.EndTimestampMs, err)
+		}
+		return nil
+	})
+}

app/vmctl/remoteread/remoteread.go

@@ -0,0 +1,349 @@
+package remoteread
+
+import (
+	"bytes"
+	"context"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"strings"
+	"time"
+
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/vm"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
+	"github.com/gogo/protobuf/proto"
+	"github.com/golang/snappy"
+	"github.com/prometheus/prometheus/prompb"
+	"github.com/prometheus/prometheus/storage/remote"
+	"github.com/prometheus/prometheus/tsdb/chunkenc"
+)
+
+const (
+	defaultReadTimeout = 30 * time.Second
+	remoteReadPath     = "/api/v1/read"
+	healthPath         = "/-/healthy"
+)
+
+// StreamCallback is a callback function for processing time series
+type StreamCallback func(series *vm.TimeSeries) error
+
+// Client is an HTTP client for reading
+// time series via remote read protocol.
+type Client struct {
+	addr      string
+	c         *http.Client
+	user      string
+	password  string
+	useStream bool
+	headers   []keyValue
+	matchers  []*prompb.LabelMatcher
+}
+
+// Config is config for remote read.
+type Config struct {
+	// Addr of remote storage
+	Addr string
+	// Timeout defines timeout for HTTP requests
+	// made by remote read client
+	Timeout time.Duration
+	// Username is the remote read username, optional.
+	Username string
+	// Password is the remote read password, optional.
+	Password string
+	// UseStream defines whether to use SAMPLES or STREAMED_XOR_CHUNKS mode
+	// see https://prometheus.io/docs/prometheus/latest/querying/remote_read_api/#samples
+	// https://prometheus.io/docs/prometheus/latest/querying/remote_read_api/#streamed-chunks
+	UseStream bool
+	// Headers optional HTTP headers to send with each request to the corresponding remote storage
+	Headers string
+	// LabelName, LabelValue stands for label=~value pair used for read requests.
+	// Is optional.
+	LabelName, LabelValue string
+}
+
+// Filter defines a list of filters applied to requested data
+type Filter struct {
+	StartTimestampMs int64
+	EndTimestampMs   int64
+}
+
+// NewClient returns client for
+// reading time series via remote read protocol.
+func NewClient(cfg Config) (*Client, error) {
+	if cfg.Addr == "" {
+		return nil, fmt.Errorf("config.Addr can't be empty")
+	}
+	if cfg.Timeout == 0 {
+		cfg.Timeout = defaultReadTimeout
+	}
+
+	var hdrs []string
+	if cfg.Headers != "" {
+		hdrs = strings.Split(cfg.Headers, "^^")
+	}
+
+	headers, err := parseHeaders(hdrs)
+	if err != nil {
+		return nil, err
+	}
+
+	var m *prompb.LabelMatcher
+	if cfg.LabelName != "" && cfg.LabelValue != "" {
+		m = &prompb.LabelMatcher{
+			Type:  prompb.LabelMatcher_RE,
+			Name:  cfg.LabelName,
+			Value: cfg.LabelValue,
+		}
+	}
+
+	c := &Client{
+		c: &http.Client{
+			Timeout:   cfg.Timeout,
+			Transport: http.DefaultTransport.(*http.Transport).Clone(),
+		},
+		addr:      strings.TrimSuffix(cfg.Addr, "/"),
+		user:      cfg.Username,
+		password:  cfg.Password,
+		useStream: cfg.UseStream,
+		headers:   headers,
+		matchers:  []*prompb.LabelMatcher{m},
+	}
+	return c, nil
+}
+
+// Read fetch data from remote read source
+func (c *Client) Read(ctx context.Context, filter *Filter, streamCb StreamCallback) error {
+	req := &prompb.ReadRequest{
+		Queries: []*prompb.Query{
+			{
+				StartTimestampMs: filter.StartTimestampMs,
+				EndTimestampMs:   filter.EndTimestampMs - 1,
+				Matchers:         c.matchers,
+			},
+		},
+	}
+	if c.useStream {
+		req.AcceptedResponseTypes = []prompb.ReadRequest_ResponseType{prompb.ReadRequest_STREAMED_XOR_CHUNKS}
+	}
+	data, err := proto.Marshal(req)
+	if err != nil {
+		return fmt.Errorf("unable to marshal read request: %w", err)
+	}
+
+	b := snappy.Encode(nil, data)
+	if err := c.fetch(ctx, b, streamCb); err != nil {
+		if errors.Is(err, context.Canceled) {
+			return fmt.Errorf("fetch request has ben cancelled")
+		}
+		return fmt.Errorf("error while fetching data from remote storage: %s", err)
+	}
+	return nil
+}
+
+func (c *Client) do(req *http.Request) (*http.Response, error) {
+	if c.user != "" {
+		req.SetBasicAuth(c.user, c.password)
+	}
+	for _, h := range c.headers {
+		req.Header.Add(h.key, h.value)
+	}
+	return c.c.Do(req)
+}
+
+// Ping checks the health of the read source
+func (c *Client) Ping() error {
+	url := c.addr + healthPath
+	req, err := http.NewRequest("GET", url, nil)
+	if err != nil {
+		return fmt.Errorf("cannot create request to %q: %s", url, err)
+	}
+	resp, err := c.do(req)
+	if err != nil {
+		return err
+	}
+	if resp.StatusCode != http.StatusOK {
+		return fmt.Errorf("bad status code: %d", resp.StatusCode)
+	}
+	return nil
+}
+
+func (c *Client) fetch(ctx context.Context, data []byte, streamCb StreamCallback) error {
+	r := bytes.NewReader(data)
+	url := c.addr + remoteReadPath
+	req, err := http.NewRequest("POST", url, r)
+	if err != nil {
+		return fmt.Errorf("failed to create new HTTP request: %w", err)
+	}
+
+	req.Header.Add("Content-Encoding", "snappy")
+	req.Header.Add("Accept-Encoding", "snappy")
+	req.Header.Set("Content-Type", "application/x-protobuf")
+	if c.useStream {
+		req.Header.Set("Content-Type", "application/x-streamed-protobuf; proto=prometheus.ChunkedReadResponse")
+	}
+	req.Header.Set("X-Prometheus-Remote-Read-Version", "0.1.0")
+
+	resp, err := c.do(req.WithContext(ctx))
+	if err != nil {
+		return fmt.Errorf("error while sending request to %s: %w; Data len %d(%d)",
+			req.URL.Redacted(), err, len(data), r.Size())
+	}
+	defer func() { _ = resp.Body.Close() }()
+
+	if resp.StatusCode != http.StatusNoContent && resp.StatusCode != http.StatusOK {
+		body, _ := io.ReadAll(resp.Body)
+		return fmt.Errorf("unexpected response code %d for %s. Response body %q",
+			resp.StatusCode, req.URL.Redacted(), body)
+	}
+
+	if c.useStream {
+		return processStreamResponse(resp.Body, streamCb)
+	}
+
+	return processResponse(resp.Body, streamCb)
+}
+
+func processResponse(body io.ReadCloser, callback StreamCallback) error {
+	d, err := io.ReadAll(body)
+	if err != nil {
+		return fmt.Errorf("error reading response: %w", err)
+	}
+	uncompressed, err := snappy.Decode(nil, d)
+	if err != nil {
+		return fmt.Errorf("error decoding response: %w", err)
+	}
+	var readResp prompb.ReadResponse
+	err = proto.Unmarshal(uncompressed, &readResp)
+	if err != nil {
+		return fmt.Errorf("unable to unmarshal response body: %w", err)
+	}
+	// response could have no results for the given filter, but that
+	// shouldn't be accounted as an error.
+	for _, res := range readResp.Results {
+		for _, ts := range res.Timeseries {
+			vmTs := convertSamples(ts.Samples, ts.Labels)
+			if err := callback(vmTs); err != nil {
+				return err
+			}
+		}
+	}
+
+	return nil
+}
+
+var bbPool bytesutil.ByteBufferPool
+
+func processStreamResponse(body io.ReadCloser, callback StreamCallback) error {
+	bb := bbPool.Get()
+	defer func() { bbPool.Put(bb) }()
+
+	stream := remote.NewChunkedReader(body, remote.DefaultChunkedReadLimit, bb.B)
+	for {
+		res := &prompb.ChunkedReadResponse{}
+		err := stream.NextProto(res)
+		if err == io.EOF {
+			break
+		}
+		if err != nil {
+			return err
+		}
+
+		for _, series := range res.ChunkedSeries {
+			samples := make([]prompb.Sample, 0)
+			for _, chunk := range series.Chunks {
+				s, err := parseSamples(chunk.Data)
+				if err != nil {
+					return err
+				}
+				samples = append(samples, s...)
+			}
+
+			ts := convertSamples(samples, series.Labels)
+			if err := callback(ts); err != nil {
+				return err
+			}
+		}
+	}
+	return nil
+}
+
+func parseSamples(chunk []byte) ([]prompb.Sample, error) {
+	c, err := chunkenc.FromData(chunkenc.EncXOR, chunk)
+	if err != nil {
+		return nil, fmt.Errorf("error read chunk: %w", err)
+	}
+
+	var samples []prompb.Sample
+	it := c.Iterator(nil)
+	for {
+		typ := it.Next()
+		if typ == chunkenc.ValNone {
+			break
+		}
+		if typ != chunkenc.ValFloat {
+			// Skip unsupported values
+			continue
+		}
+		if it.Err() != nil {
+			return nil, fmt.Errorf("error iterate over chunks: %w", it.Err())
+		}
+
+		ts, v := it.At()
+		s := prompb.Sample{
+			Timestamp: ts,
+			Value:     v,
+		}
+		samples = append(samples, s)
+	}
+
+	return samples, it.Err()
+}
+
+type keyValue struct {
+	key   string
+	value string
+}
+
+func parseHeaders(headers []string) ([]keyValue, error) {
+	if len(headers) == 0 {
+		return nil, nil
+	}
+	kvs := make([]keyValue, len(headers))
+	for i, h := range headers {
+		n := strings.IndexByte(h, ':')
+		if n < 0 {
+			return nil, fmt.Errorf(`missing ':' in header %q; expecting "key: value" format`, h)
+		}
+		kv := &kvs[i]
+		kv.key = strings.TrimSpace(h[:n])
+		kv.value = strings.TrimSpace(h[n+1:])
+	}
+	return kvs, nil
+}
+
+func convertSamples(samples []prompb.Sample, labels []prompb.Label) *vm.TimeSeries {
+	labelPairs := make([]vm.LabelPair, 0, len(labels))
+	nameValue := ""
+	for _, label := range labels {
+		if label.Name == "__name__" {
+			nameValue = label.Value
+			continue
+		}
+		labelPairs = append(labelPairs, vm.LabelPair{Name: label.Name, Value: label.Value})
+	}
+
+	n := len(samples)
+	values := make([]float64, 0, n)
+	timestamps := make([]int64, 0, n)
+	for _, sample := range samples {
+		values = append(values, sample.Value)
+		timestamps = append(timestamps, sample.Timestamp)
+	}
+	return &vm.TimeSeries{
+		Name:       nameValue,
+		LabelPairs: labelPairs,
+		Timestamps: timestamps,
+		Values:     values,
+	}
+}

app/vmctl/stepper/split.go

@@ -12,6 +12,8 @@ const (
 	StepDay string = "day"
 	// StepHour represents a one hour interval
 	StepHour string = "hour"
+	// StepMinute represents a one minute interval
+	StepMinute string = "minute"
 )
 
 // SplitDateRange splits start-end range in a subset of ranges respecting the given step
@@ -42,8 +44,13 @@ func SplitDateRange(start, end time.Time, step string) ([][]time.Time, error) {
 		nextStep = func(t time.Time) (time.Time, time.Time) {
 			return t, t.Add(time.Hour * 1)
 		}
+	case StepMinute:
+		nextStep = func(t time.Time) (time.Time, time.Time) {
+			return t, t.Add(time.Minute * 1)
+		}
 	default:
-		return nil, fmt.Errorf("failed to parse step value, valid values are: '%s', '%s', '%s'. provided: '%s'", StepMonth, StepDay, StepHour, step)
+		return nil, fmt.Errorf("failed to parse step value, valid values are: '%s', '%s', '%s', '%s'. provided: '%s'",
+			StepMonth, StepDay, StepHour, StepMinute, step)
 	}
 
 	currentStep := start

app/vmctl/vm/vm.go

@@ -182,6 +182,8 @@ func (im *Importer) Errors() chan *ImportError { return im.errors }
 // that need to be imported
 func (im *Importer) Input(ts *TimeSeries) error {
 	select {
+	case <-im.close:
+		return fmt.Errorf("importer is closed")
 	case im.input <- ts:
 		return nil
 	case err := <-im.errors:
@@ -197,6 +199,7 @@ func (im *Importer) Input(ts *TimeSeries) error {
 func (im *Importer) Close() {
 	im.once.Do(func() {
 		close(im.close)
+		close(im.input)
 		im.wg.Wait()
 		close(im.errors)
 	})
@@ -209,6 +212,10 @@ func (im *Importer) startWorker(bar *pb.ProgressBar, batchSize, significantFigur
 	for {
 		select {
 		case <-im.close:
+			for ts := range im.input {
+				ts = roundTimeseriesValue(ts, significantFigures, roundDigits)
+				batch = append(batch, ts)
+			}
 			exitErr := &ImportError{
 				Batch: batch,
 			}
@@ -217,24 +224,17 @@ func (im *Importer) startWorker(bar *pb.ProgressBar, batchSize, significantFigur
 			}
 			im.errors <- exitErr
 			return
-		case ts := <-im.input:
+		case ts, ok := <-im.input:
+			if !ok {
+				continue
+			}
 			// init waitForBatch when first
 			// value was received
 			if waitForBatch.IsZero() {
 				waitForBatch = time.Now()
 			}
 
-			if significantFigures > 0 {
-				for i, v := range ts.Values {
-					ts.Values[i] = decimal.RoundToSignificantFigures(v, significantFigures)
-				}
-			}
-			if roundDigits < 100 {
-				for i, v := range ts.Values {
-					ts.Values[i] = decimal.RoundToDecimalDigits(v, roundDigits)
-				}
-			}
-
+			ts = roundTimeseriesValue(ts, significantFigures, roundDigits)
 			batch = append(batch, ts)
 			dataPoints += len(ts.Values)
 
@@ -418,3 +418,18 @@ func byteCountSI(b int64) string {
 	return fmt.Sprintf("%.1f %cB",
 		float64(b)/float64(div), "kMGTPE"[exp])
 }
+
+func roundTimeseriesValue(ts *TimeSeries, significantFigures, roundDigits int) *TimeSeries {
+	if significantFigures > 0 {
+		for i, v := range ts.Values {
+			ts.Values[i] = decimal.RoundToSignificantFigures(v, significantFigures)
+		}
+	}
+	if roundDigits < 100 {
+		for i, v := range ts.Values {
+			ts.Values[i] = decimal.RoundToDecimalDigits(v, roundDigits)
+		}
+	}
+
+	return ts
+}

app/vmctl/vm_native.go

@@ -2,6 +2,7 @@ package main
 
 import (
 	"context"
+	"encoding/json"
 	"fmt"
 	"io"
 	"log"
@@ -19,8 +20,9 @@ type vmNativeProcessor struct {
 	filter    filter
 	rateLimit int64
 
-	dst *vmNativeClient
-	src *vmNativeClient
+	dst          *vmNativeClient
+	src          *vmNativeClient
+	interCluster bool
 }
 
 type vmNativeClient struct {
@@ -49,15 +51,16 @@ func (f filter) String() string {
 }
 
 const (
-	nativeExportAddr = "api/v1/export/native"
-	nativeImportAddr = "api/v1/import/native"
+	nativeExportAddr  = "api/v1/export/native"
+	nativeImportAddr  = "api/v1/import/native"
+	nativeTenantsAddr = "admin/tenants"
 
 	nativeBarTpl = `Total: {{counters . }} {{ cycle . "↖" "↗" "↘" "↙" }} Speed: {{speed . }} {{string . "suffix"}}`
 )
 
 func (p *vmNativeProcessor) run(ctx context.Context) error {
 	if p.filter.chunk == "" {
-		return p.runSingle(ctx, p.filter)
+		return p.runWithFilter(ctx, p.filter)
 	}
 
 	startOfRange, err := time.Parse(time.RFC3339, p.filter.timeStart)
@@ -89,7 +92,7 @@ func (p *vmNativeProcessor) run(ctx context.Context) error {
 			timeStart: formattedStartTime,
 			timeEnd:   formattedEndTime,
 		}
-		err := p.runSingle(ctx, f)
+		err := p.runWithFilter(ctx, f)
 
 		if err != nil {
 			log.Printf("processing failed for range %d/%d: %s - %s \n", rangeIdx+1, len(ranges), formattedStartTime, formattedEndTime)
@@ -99,25 +102,52 @@ func (p *vmNativeProcessor) run(ctx context.Context) error {
 	return nil
 }
 
-func (p *vmNativeProcessor) runSingle(ctx context.Context, f filter) error {
-	pr, pw := io.Pipe()
+func (p *vmNativeProcessor) runWithFilter(ctx context.Context, f filter) error {
+	nativeImportAddr, err := vm.AddExtraLabelsToImportPath(nativeImportAddr, p.dst.extraLabels)
 
-	log.Printf("Initing export pipe from %q with filters: %s\n", p.src.addr, f)
-	exportReader, err := p.exportPipe(ctx, f)
+	if err != nil {
+		return fmt.Errorf("failed to add labels to import path: %s", err)
+	}
+
+	if !p.interCluster {
+		srcURL := fmt.Sprintf("%s/%s", p.src.addr, nativeExportAddr)
+		dstURL := fmt.Sprintf("%s/%s", p.dst.addr, nativeImportAddr)
+
+		return p.runSingle(ctx, f, srcURL, dstURL)
+	}
+
+	tenants, err := p.getSourceTenants(ctx, f)
+	if err != nil {
+		return fmt.Errorf("failed to get source tenants: %s", err)
+	}
+
+	log.Printf("Discovered tenants: %v", tenants)
+	for _, tenant := range tenants {
+		// src and dst expected formats: http://vminsert:8480/ and http://vmselect:8481/
+		srcURL := fmt.Sprintf("%s/select/%s/prometheus/%s", p.src.addr, tenant, nativeExportAddr)
+		dstURL := fmt.Sprintf("%s/insert/%s/prometheus/%s", p.dst.addr, tenant, nativeImportAddr)
+
+		if err := p.runSingle(ctx, f, srcURL, dstURL); err != nil {
+			return fmt.Errorf("failed to migrate data for tenant %q: %s", tenant, err)
+		}
+	}
+
+	return nil
+}
+
+func (p *vmNativeProcessor) runSingle(ctx context.Context, f filter, srcURL, dstURL string) error {
+	log.Printf("Initing export pipe from %q with filters: %s\n", srcURL, f)
+
+	exportReader, err := p.exportPipe(ctx, srcURL, f)
 	if err != nil {
 		return fmt.Errorf("failed to init export pipe: %s", err)
 	}
 
-	nativeImportAddr, err := vm.AddExtraLabelsToImportPath(nativeImportAddr, p.dst.extraLabels)
-	if err != nil {
-		return err
-	}
-
+	pr, pw := io.Pipe()
 	sync := make(chan struct{})
 	go func() {
 		defer func() { close(sync) }()
-		u := fmt.Sprintf("%s/%s", p.dst.addr, nativeImportAddr)
-		req, err := http.NewRequestWithContext(ctx, "POST", u, pr)
+		req, err := http.NewRequestWithContext(ctx, "POST", dstURL, pr)
 		if err != nil {
 			log.Fatalf("cannot create import request to %q: %s", p.dst.addr, err)
 		}
@@ -130,7 +160,7 @@ func (p *vmNativeProcessor) runSingle(ctx context.Context, f filter) error {
 		}
 	}()
 
-	fmt.Printf("Initing import process to %q:\n", p.dst.addr)
+	fmt.Printf("Initing import process to %q:\n", dstURL)
 	pool := pb.NewPool()
 	bar := pb.ProgressBarTemplate(nativeBarTpl).New(0)
 	pool.Add(bar)
@@ -166,9 +196,43 @@ func (p *vmNativeProcessor) runSingle(ctx context.Context, f filter) error {
 	return nil
 }
 
-func (p *vmNativeProcessor) exportPipe(ctx context.Context, f filter) (io.ReadCloser, error) {
-	u := fmt.Sprintf("%s/%s", p.src.addr, nativeExportAddr)
+func (p *vmNativeProcessor) getSourceTenants(ctx context.Context, f filter) ([]string, error) {
+	u := fmt.Sprintf("%s/%s", p.src.addr, nativeTenantsAddr)
 	req, err := http.NewRequestWithContext(ctx, "GET", u, nil)
+	if err != nil {
+		return nil, fmt.Errorf("cannot create request to %q: %s", u, err)
+	}
+
+	params := req.URL.Query()
+	if f.timeStart != "" {
+		params.Set("start", f.timeStart)
+	}
+	if f.timeEnd != "" {
+		params.Set("end", f.timeEnd)
+	}
+	req.URL.RawQuery = params.Encode()
+
+	resp, err := p.src.do(req, http.StatusOK)
+	if err != nil {
+		return nil, fmt.Errorf("tenants request failed: %s", err)
+	}
+
+	var r struct {
+		Tenants []string `json:"data"`
+	}
+	if err := json.NewDecoder(resp.Body).Decode(&r); err != nil {
+		return nil, fmt.Errorf("cannot decode tenants response: %s", err)
+	}
+
+	if err := resp.Body.Close(); err != nil {
+		return nil, fmt.Errorf("cannot close tenants response body: %s", err)
+	}
+
+	return r.Tenants, nil
+}
+
+func (p *vmNativeProcessor) exportPipe(ctx context.Context, url string, f filter) (io.ReadCloser, error) {
+	req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
 	if err != nil {
 		return nil, fmt.Errorf("cannot create request to %q: %s", p.src.addr, err)
 	}

app/vmgateway/README.md

@@ -1,6 +1,6 @@
 # vmgateway
 
-***vmgateway is a part of [enterprise package](https://victoriametrics.com/products/enterprise/). It is available for download and evaluation at [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)***
+***vmgateway is a part of [enterprise package](https://docs.victoriametrics.com/enterprise.html). It is available for download and evaluation at [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)***
 
 <img alt="vmgateway" src="vmgateway-overview.jpeg">
 
@@ -13,7 +13,7 @@
   * Provides access by tenantID in the Cluster version
   * Allows for separate write/read/admin access to data
 
-`vmgateway` is included in our [enterprise packages](https://victoriametrics.com/products/enterprise/).
+`vmgateway` is included in our [enterprise packages](https://docs.victoriametrics.com/enterprise.html).
 
 ## Access Control
 
@@ -171,13 +171,54 @@ curl 'http://localhost:8431/api/v1/labels' -H 'Authorization: Bearer eyJhbGciOiJ
 # check rate limit
 ```
 
+## JWT signature verification
+
+`vmgateway` supports JWT signature verification.
+
+Supported algorithms are `RS256`, `RS384`, `RS512`, `ES256`, `ES384`, `ES512`, `PS256`, `PS384`, `PS512`.
+Tokens with unsupported algorithms will be rejected.
+
+In order to enable JWT signature verification, you need to specify keys for signature verification.
+The following flags are used to specify keys:
+- `-auth.publicKeyFiles` - allows to pass file path to file with public key.
+- `-auth.publicKeys` - allows to pass public key directly.
+
+Note that both flags support passing multiple keys and also can be used together.
+
+Example usage:
+```console
+./bin/vmgateway -eula \
+  -enable.auth \
+  -write.url=http://localhost:8480 \
+  -read.url=http://localhost:8481 \
+  -auth.publicKeyFiles=public_key.pem \
+  -auth.publicKeyFiles=public_key2.pem \
+  -auth.publicKeys=`-----BEGIN PUBLIC KEY-----
+MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAu1SU1LfVLPHCozMxH2Mo
+4lgOEePzNm0tRgeLezV6ffAt0gunVTLw7onLRnrq0/IzW7yWR7QkrmBL7jTKEn5u
++qKhbwKfBstIs+bMY2Zkp18gnTxKLxoS2tFczGkPLPgizskuemMghRniWaoLcyeh
+kd3qqGElvW/VDL5AaWTg0nLVkjRo9z+40RQzuVaE8AkAFmxZzow3x+VJYKdjykkJ
+0iT9wCS0DRTXu269V264Vf/3jvredZiKRkgwlL9xNAwxXFg0x/XFw005UWVRIkdg
+cKWTjpBP2dPwVZ4WWC+9aGVd+Gyn1o0CLelf4rEjGoXbAAEgAqeGUxrcIlbjXfbc
+mwIDAQAB
+-----END PUBLIC KEY-----
+`
+```
+This command will result in 3 keys loaded: 2 keys from files and 1 from command line.
+
 ## Configuration
 
 The shortlist of configuration flags include the following:
 
 ```console
-  -auth.httpHeader
-    HTTP header name to look for JWT authorization token
+  -auth.httpHeader string
+     HTTP header name to look for JWT authorization token (default "Authorization")
+  -auth.publicKeyFiles array
+     Path file with public key to verify JWT token signature
+     Supports an array of values separated by comma or specified via multiple flags.
+  -auth.publicKeys array
+     Public keys to verify JWT token signature
+     Supports an array of values separated by comma or specified via multiple flags.
   -clusterMode
      enable this for the cluster version
   -datasource.appendTypePrefix
@@ -201,17 +242,17 @@ The shortlist of configuration flags include the following:
   -datasource.maxIdleConnections int
      Defines the number of idle (keep-alive connections) to each configured datasource. Consider setting this value equal to the value: groups_total * group.concurrency. Too low a value may result in a high number of sockets in TIME_WAIT state. (default 100)
   -datasource.oauth2.clientID string
-     Optional OAuth2 clientID to use for -datasource.url.
+     Optional OAuth2 clientID to use for -datasource.url. 
   -datasource.oauth2.clientSecret string
      Optional OAuth2 clientSecret to use for -datasource.url.
   -datasource.oauth2.clientSecretFile string
-     Optional OAuth2 clientSecretFile to use for -datasource.url.
+     Optional OAuth2 clientSecretFile to use for -datasource.url. 
   -datasource.oauth2.scopes string
      Optional OAuth2 scopes to use for -datasource.url. Scopes must be delimited by ';'
   -datasource.oauth2.tokenUrl string
      Optional OAuth2 tokenURL to use for -datasource.url.
   -datasource.queryStep duration
-     queryStep defines how far a value can fallback to when evaluating queries. For example, if datasource.queryStep=15s then param "step" with value "15s" will be added to every query.If queryStep isn't specified, rule's evaluationInterval will be used instead.
+     How far a value can fallback to when evaluating queries. For example, if -datasource.queryStep=15s then param "step" with value "15s" will be added to every query. If set to 0, rule's evaluation interval will be used instead. (default 5m0s)
   -datasource.queryTimeAlignment
      Whether to align "time" parameter with evaluation interval.Alignment supposed to produce deterministic results despite of number of vmalert replicas or time they were started. See more details here https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1257 (default true)
   -datasource.roundDigits int
@@ -229,7 +270,7 @@ The shortlist of configuration flags include the following:
   -datasource.tlsServerName string
      Optional TLS server name to use for connections to -datasource.url. By default, the server name from -datasource.url is used
   -datasource.url string
-     Datasource compatible with Prometheus HTTP API. It can be single node VictoriaMetrics or vmselect URL. Required parameter. E.g. http://127.0.0.1:8428 . See also '-datasource.disablePathAppend', '-datasource.showURL'.
+     Datasource compatible with Prometheus HTTP API. It can be single node VictoriaMetrics or vmselect URL. Required parameter. E.g. http://127.0.0.1:8428 . See also -remoteRead.disablePathAppend and -datasource.showURL
   -enable.auth
      enables auth with jwt token
   -enable.rateLimit
@@ -241,7 +282,7 @@ The shortlist of configuration flags include the following:
   -envflag.prefix string
      Prefix for environment variables if -envflag.enable is set
   -eula
-     By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf . This flag is available only in enterprise version of VictoriaMetrics
+     By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
   -flagsAuthKey string
      Auth key for /flags endpoint. It must be passed via authKey query arg. It overrides httpAuth.* settings
   -fs.disableMmap
@@ -336,7 +377,7 @@ The shortlist of configuration flags include the following:
 ## Limitations
 
 * Access Control:
-  * `jwt` token must be validated by external system, currently `vmgateway` can't validate the signature.
+  * `jwt` token signature verification for `HMAC` algorithms is not supported.
 * RateLimiting:
   * limits applied based on queries to `datasource.url`
   * only cluster version can be rate-limited.

app/vminsert/datadog/request_handler.go

@@ -54,7 +54,12 @@ func insertRows(series []parser.Series, extraLabels []prompbmarshal.Label) error
 		rowsTotal += len(ss.Points)
 		ctx.Labels = ctx.Labels[:0]
 		ctx.AddLabel("", ss.Metric)
-		ctx.AddLabel("host", ss.Host)
+		if ss.Host != "" {
+			ctx.AddLabel("host", ss.Host)
+		}
+		if ss.Device != "" {
+			ctx.AddLabel("device", ss.Device)
+		}
 		for _, tag := range ss.Tags {
 			name, value := parser.SplitTag(tag)
 			if name == "host" {

app/vminsert/main.go

@@ -120,6 +120,16 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 		staticServer.ServeHTTP(w, r)
 		return true
 	}
+	if strings.HasPrefix(path, "/prometheus/api/v1/import/prometheus") || strings.HasPrefix(path, "/api/v1/import/prometheus") {
+		prometheusimportRequests.Inc()
+		if err := prometheusimport.InsertHandler(r); err != nil {
+			prometheusimportErrors.Inc()
+			httpserver.Errorf(w, r, "%s", err)
+			return true
+		}
+		w.WriteHeader(http.StatusNoContent)
+		return true
+	}
 	if strings.HasPrefix(path, "/datadog/") {
 		// Trim suffix from paths starting from /datadog/ in order to support legacy DataDog agent.
 		// See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2670
@@ -153,15 +163,6 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 		}
 		w.WriteHeader(http.StatusNoContent)
 		return true
-	case "/prometheus/api/v1/import/prometheus", "/api/v1/import/prometheus":
-		prometheusimportRequests.Inc()
-		if err := prometheusimport.InsertHandler(r); err != nil {
-			prometheusimportErrors.Inc()
-			httpserver.Errorf(w, r, "%s", err)
-			return true
-		}
-		w.WriteHeader(http.StatusNoContent)
-		return true
 	case "/prometheus/api/v1/import/native", "/api/v1/import/native":
 		nativeimportRequests.Inc()
 		if err := native.InsertHandler(r); err != nil {
@@ -332,7 +333,8 @@ var (
 
 	promscrapeTargetsRequests          = metrics.NewCounter(`vm_http_requests_total{path="/targets"}`)
 	promscrapeServiceDiscoveryRequests = metrics.NewCounter(`vm_http_requests_total{path="/service-discovery"}`)
-	promscrapeAPIV1TargetsRequests     = metrics.NewCounter(`vm_http_requests_total{path="/api/v1/targets"}`)
+
+	promscrapeAPIV1TargetsRequests = metrics.NewCounter(`vm_http_requests_total{path="/api/v1/targets"}`)
 
 	promscrapeTargetResponseRequests = metrics.NewCounter(`vm_http_requests_total{path="/target_response"}`)
 	promscrapeTargetResponseErrors   = metrics.NewCounter(`vm_http_request_errors_total{path="/target_response"}`)

app/vminsert/relabel/relabel.go

@@ -6,6 +6,7 @@ import (
 	"sync/atomic"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fasttime"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/procutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompb"
@@ -18,8 +19,6 @@ var (
 	relabelConfig = flag.String("relabelConfig", "", "Optional path to a file with relabeling rules, which are applied to all the ingested metrics. "+
 		"The path can point either to local file or to http url. "+
 		"See https://docs.victoriametrics.com/#relabeling for details. The config is reloaded on SIGHUP signal")
-	relabelDebug = flag.Bool("relabelDebug", false, "Whether to log metrics before and after relabeling with -relabelConfig. If the -relabelDebug is enabled, "+
-		"then the metrics aren't sent to storage. This is useful for debugging the relabeling configs")
 
 	usePromCompatibleNaming = flag.Bool("usePromCompatibleNaming", false, "Whether to replace characters unsupported by Prometheus with underscores "+
 		"in the ingested metric names and label names. For example, foo.bar{a.b='c'} is transformed into foo_bar{a_b='c'} during data ingestion if this flag is set. "+
@@ -38,30 +37,45 @@ func Init() {
 		logger.Fatalf("cannot load relabelConfig: %s", err)
 	}
 	pcsGlobal.Store(pcs)
+	configSuccess.Set(1)
+	configTimestamp.Set(fasttime.UnixTimestamp())
+
 	if len(*relabelConfig) == 0 {
 		return
 	}
 	go func() {
 		for range sighupCh {
+			configReloads.Inc()
 			logger.Infof("received SIGHUP; reloading -relabelConfig=%q...", *relabelConfig)
 			pcs, err := loadRelabelConfig()
 			if err != nil {
+				configReloadErrors.Inc()
+				configSuccess.Set(0)
 				logger.Errorf("cannot load the updated relabelConfig: %s; preserving the previous config", err)
 				continue
 			}
 			pcsGlobal.Store(pcs)
+			configSuccess.Set(1)
+			configTimestamp.Set(fasttime.UnixTimestamp())
 			logger.Infof("successfully reloaded -relabelConfig=%q", *relabelConfig)
 		}
 	}()
 }
 
+var (
+	configReloads      = metrics.NewCounter(`vm_relabel_config_reloads_total`)
+	configReloadErrors = metrics.NewCounter(`vm_relabel_config_reloads_errors_total`)
+	configSuccess      = metrics.NewCounter(`vm_relabel_config_last_reload_successful`)
+	configTimestamp    = metrics.NewCounter(`vm_relabel_config_last_reload_success_timestamp_seconds`)
+)
+
 var pcsGlobal atomic.Value
 
 func loadRelabelConfig() (*promrelabel.ParsedConfigs, error) {
 	if len(*relabelConfig) == 0 {
 		return nil, nil
 	}
-	pcs, err := promrelabel.LoadRelabelConfigs(*relabelConfig, *relabelDebug)
+	pcs, err := promrelabel.LoadRelabelConfigs(*relabelConfig)
 	if err != nil {
 		return nil, fmt.Errorf("error when reading -relabelConfig=%q: %w", *relabelConfig, err)
 	}
@@ -123,7 +137,8 @@ func (ctx *Ctx) ApplyRelabeling(labels []prompb.Label) []prompb.Label {
 
 	if pcs.Len() > 0 {
 		// Apply relabeling
-		tmpLabels = pcs.Apply(tmpLabels, 0, true)
+		tmpLabels = pcs.Apply(tmpLabels, 0)
+		tmpLabels = promrelabel.FinalizeLabels(tmpLabels[:0], tmpLabels)
 		if len(tmpLabels) == 0 {
 			metricsDropped.Inc()
 		}

app/vmrestore/README.md

@@ -12,7 +12,7 @@ VictoriaMetrics must be stopped during the restore process.
 Run the following command to restore backup from the given `-src` into the given `-storageDataPath`:
 
 ```console
-vmrestore -src=<storageType>://<path/to/backup> -storageDataPath=<local/path/to/restore>
+./vmrestore -src=<storageType>://<path/to/backup> -storageDataPath=<local/path/to/restore>
 ```
 
 * `<storageType>://<path/to/backup>` is the path to backup made with [vmbackup](https://docs.victoriametrics.com/vmbackup.html).
@@ -100,7 +100,7 @@ i.e. the end result would be similar to [rsync --delete](https://askubuntu.com/q
   -envflag.prefix string
      Prefix for environment variables if -envflag.enable is set
   -eula
-     By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf . This flag is available only in enterprise version of VictoriaMetrics
+     By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
   -flagsAuthKey string
      Auth key for /flags endpoint. It must be passed via authKey query arg. It overrides httpAuth.* settings
   -fs.disableMmap
@@ -186,7 +186,7 @@ It is recommended using [binary releases](https://github.com/VictoriaMetrics/Vic
 
 ### Development build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.1.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.
 2. Run `make vmrestore` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `vmrestore` binary and puts it into the `bin` folder.
 

app/vmselect/main.go

@@ -21,6 +21,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fs"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/querytracer"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/timerpool"
 	"github.com/VictoriaMetrics/metrics"
@@ -29,7 +30,8 @@ import (
 var (
 	deleteAuthKey         = flag.String("deleteAuthKey", "", "authKey for metrics' deletion via /api/v1/admin/tsdb/delete_series and /tags/delSeries")
 	maxConcurrentRequests = flag.Int("search.maxConcurrentRequests", getDefaultMaxConcurrentRequests(), "The maximum number of concurrent search requests. "+
-		"It shouldn't be high, since a single request can saturate all the CPU cores. See also -search.maxQueueDuration")
+		"It shouldn't be high, since a single request can saturate all the CPU cores, while many concurrently executed requests may require high amounts of memory. "+
+		"See also -search.maxQueueDuration and -search.maxMemoryPerQuery")
 	maxQueueDuration = flag.Duration("search.maxQueueDuration", 10*time.Second, "The maximum time the request waits for execution when -search.maxConcurrentRequests "+
 		"limit is reached; see also -search.maxQueryDuration")
 	resetCacheAuthKey    = flag.String("search.resetCacheAuthKey", "", "Optional authKey for resetting rollup cache via /internal/resetRollupResultCache call")
@@ -214,8 +216,8 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 	}
 
 	if path == "/vmalert" {
-		// vmalert access via incomplete url without `/` in the end. Redirecto to complete url.
-		// Use relative redirect, since, since the hostname and path prefix may be incorrect if VictoriaMetrics
+		// vmalert access via incomplete url without `/` in the end. Redirect to complete url.
+		// Use relative redirect, since the hostname and path prefix may be incorrect if VictoriaMetrics
 		// is hidden behind vmauth or similar proxy.
 		httpserver.Redirect(w, "vmalert/")
 		return true
@@ -422,6 +424,14 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 			return true
 		}
 		return true
+	case "/metric-relabel-debug":
+		promscrapeMetricRelabelDebugRequests.Inc()
+		promscrape.WriteMetricRelabelDebug(w, r)
+		return true
+	case "/target-relabel-debug":
+		promscrapeTargetRelabelDebugRequests.Inc()
+		promscrape.WriteTargetRelabelDebug(w, r)
+		return true
 	case "/api/v1/rules", "/rules":
 		rulesRequests.Inc()
 		if len(*vmalertProxyURL) > 0 {
@@ -586,6 +596,9 @@ var (
 	graphiteTagsDelSeriesRequests = metrics.NewCounter(`vm_http_requests_total{path="/tags/delSeries"}`)
 	graphiteTagsDelSeriesErrors   = metrics.NewCounter(`vm_http_request_errors_total{path="/tags/delSeries"}`)
 
+	promscrapeMetricRelabelDebugRequests = metrics.NewCounter(`vm_http_requests_total{path="/metric-relabel-debug"}`)
+	promscrapeTargetRelabelDebugRequests = metrics.NewCounter(`vm_http_requests_total{path="/target-relabel-debug"}`)
+
 	graphiteFunctionsRequests = metrics.NewCounter(`vm_http_requests_total{path="/functions"}`)
 
 	vmalertRequests = metrics.NewCounter(`vm_http_requests_total{path="/vmalert"}`)

app/vmselect/netstorage/netstorage.go

@@ -1011,7 +1011,10 @@ func ProcessSearchQuery(qt *querytracer.Tracer, sq *storage.SearchQuery, deadlin
 	startTime := time.Now()
 	maxSeriesCount := sr.Init(qt, vmstorage.Storage, tfss, tr, sq.MaxMetrics, deadline.Deadline())
 	indexSearchDuration.UpdateDuration(startTime)
-	m := make(map[string][]blockRef, maxSeriesCount)
+	type blockRefs struct {
+		brs []blockRef
+	}
+	m := make(map[string]*blockRefs, maxSeriesCount)
 	orderedMetricNames := make([]string, 0, maxSeriesCount)
 	blocksRead := 0
 	samples := 0
@@ -1040,13 +1043,14 @@ func ProcessSearchQuery(qt *querytracer.Tracer, sq *storage.SearchQuery, deadlin
 		}
 		metricName := sr.MetricBlockRef.MetricName
 		brs := m[string(metricName)]
-		brs = append(brs, blockRef{
+		if brs == nil {
+			brs = &blockRefs{}
+		}
+		brs.brs = append(brs.brs, blockRef{
 			partRef: br.PartRef(),
 			addr:    addr,
 		})
-		if len(brs) > 1 {
-			m[string(metricName)] = brs
-		} else {
+		if len(brs.brs) == 1 {
 			// An optimization for big number of time series with long metricName values:
 			// use only a single copy of metricName for both orderedMetricNames and m.
 			orderedMetricNames = append(orderedMetricNames, string(metricName))
@@ -1075,7 +1079,7 @@ func ProcessSearchQuery(qt *querytracer.Tracer, sq *storage.SearchQuery, deadlin
 	for i, metricName := range orderedMetricNames {
 		pts[i] = packedTimeseries{
 			metricName: metricName,
-			brs:        m[metricName],
+			brs:        m[metricName].brs,
 		}
 	}
 	rss.packedTimeseries = pts

app/vmselect/netstorage/netstorage_test.go

@@ -144,7 +144,7 @@ func TestMergeSortBlocks(t *testing.T) {
 		},
 	}, 1, &Result{
 		Timestamps: []int64{1, 2, 4, 5, 10, 11, 12},
-		Values:     []float64{21, 22, 23, 7, 24, 5, 26},
+		Values:     []float64{21, 22, 23, 7, 24, 25, 26},
 	})
 
 	// Multiple blocks with identical timestamp ranges, no deduplication.

app/vmselect/promql/aggr.go

@@ -748,7 +748,7 @@ func getIntK(k float64, kMax int) int {
 	if math.IsNaN(k) {
 		return 0
 	}
-	kn := int(k)
+	kn := floatToIntBounded(k)
 	if kn < 0 {
 		return 0
 	}
@@ -999,14 +999,10 @@ func aggrFuncLimitK(afa *aggrFuncArg) ([]*timeseries, error) {
 	if err := expectTransformArgsNum(args, 2); err != nil {
 		return nil, err
 	}
-	limits, err := getScalar(args[0], 0)
+	limit, err := getIntNumber(args[0], 0)
 	if err != nil {
 		return nil, fmt.Errorf("cannot obtain limit arg: %w", err)
 	}
-	limit := 0
-	if len(limits) > 0 {
-		limit = int(limits[0])
-	}
 	if limit < 0 {
 		limit = 0
 	}
@@ -1155,3 +1151,13 @@ func lessWithNaNs(a, b float64) bool {
 	}
 	return a < b
 }
+
+func floatToIntBounded(f float64) int {
+	if f > math.MaxInt {
+		return math.MaxInt
+	}
+	if f < math.MinInt {
+		return math.MinInt
+	}
+	return int(f)
+}

app/vmselect/promql/eval.go

@@ -15,6 +15,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/cgroup"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/decimal"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/memory"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/querytracer"
@@ -27,7 +28,11 @@ var (
 	disableCache                   = flag.Bool("search.disableCache", false, "Whether to disable response caching. This may be useful during data backfilling")
 	maxPointsSubqueryPerTimeseries = flag.Int("search.maxPointsSubqueryPerTimeseries", 100e3, "The maximum number of points per series, which can be generated by subquery. "+
 		"See https://valyala.medium.com/prometheus-subqueries-in-victoriametrics-9b1492b720b3")
-	noStaleMarkers = flag.Bool("search.noStaleMarkers", false, "Set this flag to true if the database doesn't contain Prometheus stale markers, so there is no need in spending additional CPU time on its handling. Staleness markers may exist only in data obtained from Prometheus scrape targets")
+	maxMemoryPerQuery = flagutil.NewBytes("search.maxMemoryPerQuery", 0, "The maximum amounts of memory a single query may consume. "+
+		"Queries requiring more memory are rejected. The total memory limit for concurrently executed queries can be estimated "+
+		"as -search.maxMemoryPerQuery multiplied by -search.maxConcurrentRequests")
+	noStaleMarkers = flag.Bool("search.noStaleMarkers", false, "Set this flag to true if the database doesn't contain Prometheus stale markers, "+
+		"so there is no need in spending additional CPU time on its handling. Staleness markers may exist only in data obtained from Prometheus scrape targets")
 )
 
 // The minimum number of points per timeseries for enabling time rounding.
@@ -483,6 +488,12 @@ func execBinaryOpArgs(qt *querytracer.Tracer, ec *EvalConfig, exprFirst, exprSec
 	if err != nil {
 		return nil, nil, err
 	}
+	if len(tssFirst) == 0 && strings.ToLower(be.Op) != "or" {
+		// Fast path: there is no sense in executing the exprSecond when exprFirst returns an empty result,
+		// since the "exprFirst op exprSecond" would return an empty result in any case.
+		// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3349
+		return nil, nil, nil
+	}
 	lfs := getCommonLabelFilters(tssFirst)
 	lfs = metricsql.TrimFiltersByGroupModifier(lfs, be)
 	exprSecond = metricsql.PushdownBinaryOpFilters(exprSecond, lfs)
@@ -1051,7 +1062,17 @@ func evalRollupFuncWithMetricExpr(qt *querytracer.Tracer, ec *EvalConfig, funcNa
 		}
 	}
 	rollupPoints := mulNoOverflow(pointsPerTimeseries, int64(timeseriesLen*len(rcs)))
-	rollupMemorySize = mulNoOverflow(rollupPoints, 16)
+	rollupMemorySize = sumNoOverflow(mulNoOverflow(int64(rssLen), 1000), mulNoOverflow(rollupPoints, 16))
+	if maxMemory := int64(maxMemoryPerQuery.N); maxMemory > 0 && rollupMemorySize > maxMemory {
+		rss.Cancel()
+		return nil, &UserReadableError{
+			Err: fmt.Errorf("not enough memory for processing %d data points across %d time series with %d points in each time series "+
+				"according to -search.maxMemoryPerQuery=%d; requested memory: %d bytes; "+
+				"possible solutions are: reducing the number of matching time series; increasing `step` query arg (step=%gs); "+
+				"increasing -search.maxMemoryPerQuery",
+				rollupPoints, timeseriesLen*len(rcs), pointsPerTimeseries, maxMemory, rollupMemorySize, float64(ec.Step)/1e3),
+		}
+	}
 	rml := getRollupMemoryLimiter()
 	if !rml.Get(uint64(rollupMemorySize)) {
 		rss.Cancel()
@@ -1059,8 +1080,8 @@ func evalRollupFuncWithMetricExpr(qt *querytracer.Tracer, ec *EvalConfig, funcNa
 			Err: fmt.Errorf("not enough memory for processing %d data points across %d time series with %d points in each time series; "+
 				"total available memory for concurrent requests: %d bytes; "+
 				"requested memory: %d bytes; "+
-				"possible solutions are: reducing the number of matching time series; switching to node with more RAM; "+
-				"increasing -memory.allowedPercent; increasing `step` query arg (%gs)",
+				"possible solutions are: reducing the number of matching time series; increasing `step` query arg (step=%gs); "+
+				"switching to node with more RAM; increasing -memory.allowedPercent",
 				rollupPoints, timeseriesLen*len(rcs), pointsPerTimeseries, rml.MaxSize, uint64(rollupMemorySize), float64(ec.Step)/1e3),
 		}
 	}
@@ -1227,6 +1248,14 @@ func mulNoOverflow(a, b int64) int64 {
 	return a * b
 }
 
+func sumNoOverflow(a, b int64) int64 {
+	if math.MaxInt64-a < b {
+		// Overflow
+		return math.MaxInt64
+	}
+	return a + b
+}
+
 func dropStaleNaNs(funcName string, values []float64, timestamps []int64) ([]float64, []int64) {
 	if *noStaleMarkers || funcName == "default_rollup" || funcName == "stale_samples_over_time" {
 		// Do not drop Prometheus staleness marks (aka stale NaNs) for default_rollup() function,

app/vmselect/promql/exec_test.go

@@ -2281,6 +2281,16 @@ func TestExecSuccess(t *testing.T) {
 		resultExpected := []netstorage.Result{r}
 		f(q, resultExpected)
 	})
+	t.Run(`limit_offset(too-big-offset)`, func(t *testing.T) {
+		t.Parallel()
+		q := `limit_offset(1, 10, sort_by_label((
+			label_set(time()*1, "foo", "y"),
+			label_set(time()*2, "foo", "a"),
+			label_set(time()*3, "foo", "x"),
+		), "foo"))`
+		resultExpected := []netstorage.Result{}
+		f(q, resultExpected)
+	})
 	t.Run(`limit_offset NaN`, func(t *testing.T) {
 		t.Parallel()
 		// q returns 3 time series, where foo=3 contains only NaN values
@@ -3789,6 +3799,27 @@ func TestExecSuccess(t *testing.T) {
 		resultExpected := []netstorage.Result{r}
 		f(q, resultExpected)
 	})
+	t.Run(`histogram_quantile(duplicate-le)`, func(t *testing.T) {
+		// See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3225
+		t.Parallel()
+		q := `round(sort(histogram_quantile(0.6,
+			label_set(90, "foo", "bar", "le", "5")
+			or label_set(100, "foo", "bar", "le", "5.0")
+			or label_set(200, "foo", "bar", "le", "6.0")
+			or label_set(300, "foo", "bar", "le", "+Inf")
+		)), 0.1)`
+		r1 := netstorage.Result{
+			MetricName: metricNameExpected,
+			Values:     []float64{4.7, 4.7, 4.7, 4.7, 4.7, 4.7},
+			Timestamps: timestampsExpected,
+		}
+		r1.MetricName.Tags = []storage.Tag{{
+			Key:   []byte("foo"),
+			Value: []byte("bar"),
+		}}
+		resultExpected := []netstorage.Result{r1}
+		f(q, resultExpected)
+	})
 	t.Run(`histogram_quantile(valid)`, func(t *testing.T) {
 		t.Parallel()
 		q := `sort(histogram_quantile(0.6,
@@ -5518,6 +5549,30 @@ func TestExecSuccess(t *testing.T) {
 		resultExpected := []netstorage.Result{r1, r2}
 		f(q, resultExpected)
 	})
+	t.Run(`limitk(inf)`, func(t *testing.T) {
+		t.Parallel()
+		q := `sort(limitk(inf, label_set(10, "foo", "bar") or label_set(time()/150, "baz", "sss")))`
+		r1 := netstorage.Result{
+			MetricName: metricNameExpected,
+			Values:     []float64{10, 10, 10, 10, 10, 10},
+			Timestamps: timestampsExpected,
+		}
+		r1.MetricName.Tags = []storage.Tag{{
+			Key:   []byte("foo"),
+			Value: []byte("bar"),
+		}}
+		r2 := netstorage.Result{
+			MetricName: metricNameExpected,
+			Values:     []float64{6.666666666666667, 8, 9.333333333333334, 10.666666666666666, 12, 13.333333333333334},
+			Timestamps: timestampsExpected,
+		}
+		r2.MetricName.Tags = []storage.Tag{{
+			Key:   []byte("baz"),
+			Value: []byte("sss"),
+		}}
+		resultExpected := []netstorage.Result{r1, r2}
+		f(q, resultExpected)
+	})
 	t.Run(`any()`, func(t *testing.T) {
 		t.Parallel()
 		q := `any(label_set(10, "__name__", "x", "foo", "bar") or label_set(time()/150, "__name__", "y", "baz", "sss"))`
@@ -6354,24 +6409,55 @@ func TestExecSuccess(t *testing.T) {
 		resultExpected := []netstorage.Result{r1, r2}
 		f(q, resultExpected)
 	})
+	t.Run(`range_trim_spikes()`, func(t *testing.T) {
+		t.Parallel()
+		q := `range_trim_spikes(0.2, time())`
+		r := netstorage.Result{
+			MetricName: metricNameExpected,
+			Values:     []float64{nan, 1200, 1400, 1600, 1800, nan},
+			Timestamps: timestampsExpected,
+		}
+		resultExpected := []netstorage.Result{r}
+		f(q, resultExpected)
+	})
 	t.Run(`range_quantile(0.5)`, func(t *testing.T) {
 		t.Parallel()
 		q := `range_quantile(0.5, time())`
 		r := netstorage.Result{
 			MetricName: metricNameExpected,
-			// time() results in [1000 1200 1400 1600 1800 2000]
 			Values:     []float64{1500, 1500, 1500, 1500, 1500, 1500},
 			Timestamps: timestampsExpected,
 		}
 		resultExpected := []netstorage.Result{r}
 		f(q, resultExpected)
 	})
+	t.Run(`range_stddev()`, func(t *testing.T) {
+		t.Parallel()
+		q := `round(range_stddev(time()),0.01)`
+		r := netstorage.Result{
+			MetricName: metricNameExpected,
+			Values:     []float64{341.57, 341.57, 341.57, 341.57, 341.57, 341.57},
+			Timestamps: timestampsExpected,
+		}
+		resultExpected := []netstorage.Result{r}
+		f(q, resultExpected)
+	})
+	t.Run(`range_stdvar()`, func(t *testing.T) {
+		t.Parallel()
+		q := `round(range_stdvar(time()),0.01)`
+		r := netstorage.Result{
+			MetricName: metricNameExpected,
+			Values:     []float64{116666.67, 116666.67, 116666.67, 116666.67, 116666.67, 116666.67},
+			Timestamps: timestampsExpected,
+		}
+		resultExpected := []netstorage.Result{r}
+		f(q, resultExpected)
+	})
 	t.Run(`range_median()`, func(t *testing.T) {
 		t.Parallel()
 		q := `range_median(time())`
 		r := netstorage.Result{
 			MetricName: metricNameExpected,
-			// time() results in [1000 1200 1400 1600 1800 2000]
 			Values:     []float64{1500, 1500, 1500, 1500, 1500, 1500},
 			Timestamps: timestampsExpected,
 		}
@@ -6851,6 +6937,23 @@ func TestExecSuccess(t *testing.T) {
 		resultExpected := []netstorage.Result{r}
 		f(q, resultExpected)
 	})
+	t.Run(`range_normalize(time(),alias(-time(),"negative"))`, func(t *testing.T) {
+		t.Parallel()
+		q := `range_normalize(time(),alias(-time(), "negative"))`
+		r1 := netstorage.Result{
+			MetricName: metricNameExpected,
+			Values:     []float64{0, 0.2, 0.4, 0.6, 0.8, 1},
+			Timestamps: timestampsExpected,
+		}
+		r2 := netstorage.Result{
+			MetricName: metricNameExpected,
+			Values:     []float64{1, 0.8, 0.6, 0.4, 0.2, 0},
+			Timestamps: timestampsExpected,
+		}
+		r2.MetricName.MetricGroup = []byte("negative")
+		resultExpected := []netstorage.Result{r1, r2}
+		f(q, resultExpected)
+	})
 	t.Run(`range_first(time())`, func(t *testing.T) {
 		t.Parallel()
 		q := `range_first(time())`
@@ -6884,6 +6987,51 @@ func TestExecSuccess(t *testing.T) {
 		resultExpected := []netstorage.Result{r}
 		f(q, resultExpected)
 	})
+	t.Run(`range_linear_regression(time())`, func(t *testing.T) {
+		t.Parallel()
+		q := `range_linear_regression(time())`
+		r := netstorage.Result{
+			MetricName: metricNameExpected,
+			Values:     []float64{1000, 1200, 1400, 1600, 1800, 2000},
+			Timestamps: timestampsExpected,
+		}
+		resultExpected := []netstorage.Result{r}
+		f(q, resultExpected)
+	})
+	t.Run(`range_linear_regression(-time())`, func(t *testing.T) {
+		t.Parallel()
+		q := `range_linear_regression(-time())`
+		r := netstorage.Result{
+			MetricName: metricNameExpected,
+			Values:     []float64{-1000, -1200, -1400, -1600, -1800, -2000},
+			Timestamps: timestampsExpected,
+		}
+		resultExpected := []netstorage.Result{r}
+		f(q, resultExpected)
+	})
+	t.Run(`range_linear_regression(100/time())`, func(t *testing.T) {
+		t.Parallel()
+		q := `sort_desc(round((
+				alias(range_linear_regression(100/time()), "regress"),
+				alias(100/time(), "orig"),
+			),
+			0.001
+		))`
+		r1 := netstorage.Result{
+			MetricName: metricNameExpected,
+			Values:     []float64{0.1, 0.083, 0.071, 0.062, 0.056, 0.05},
+			Timestamps: timestampsExpected,
+		}
+		r1.MetricName.MetricGroup = []byte("orig")
+		r2 := netstorage.Result{
+			MetricName: metricNameExpected,
+			Values:     []float64{0.095, 0.085, 0.075, 0.066, 0.056, 0.046},
+			Timestamps: timestampsExpected,
+		}
+		r2.MetricName.MetricGroup = []byte("regress")
+		resultExpected := []netstorage.Result{r1, r2}
+		f(q, resultExpected)
+	})
 	t.Run(`deriv(N)`, func(t *testing.T) {
 		t.Parallel()
 		q := `deriv(1000)`
@@ -8013,6 +8161,8 @@ func TestExecError(t *testing.T) {
 	f(`nonexisting()`)
 
 	// Invalid number of args
+	f(`range_stddev()`)
+	f(`range_stdvar()`)
 	f(`range_quantile()`)
 	f(`range_quantile(1, 2, 3)`)
 	f(`range_median()`)
@@ -8074,8 +8224,10 @@ func TestExecError(t *testing.T) {
 	f(`step(1)`)
 	f(`running_sum(1, 2)`)
 	f(`range_sum(1, 2)`)
+	f(`range_trim_spikes()`)
 	f(`range_first(1,  2)`)
 	f(`range_last(1, 2)`)
+	f(`range_linear_regression(1, 2)`)
 	f(`smooth_exponential()`)
 	f(`smooth_exponential(1)`)
 	f(`remove_resets()`)

app/vmselect/promql/rollup.go

@@ -894,7 +894,7 @@ func newRollupPredictLinear(args []interface{}) (rollupFunc, error) {
 		return nil, err
 	}
 	rf := func(rfa *rollupFuncArg) float64 {
-		v, k := linearRegression(rfa)
+		v, k := linearRegression(rfa.values, rfa.timestamps, rfa.currTimestamp)
 		if math.IsNaN(v) {
 			return nan
 		}
@@ -904,13 +904,8 @@ func newRollupPredictLinear(args []interface{}) (rollupFunc, error) {
 	return rf, nil
 }
 
-func linearRegression(rfa *rollupFuncArg) (float64, float64) {
-	// There is no need in handling NaNs here, since they must be cleaned up
-	// before calling rollup funcs.
-	values := rfa.values
-	timestamps := rfa.timestamps
-	n := float64(len(values))
-	if n == 0 {
+func linearRegression(values []float64, timestamps []int64, interceptTime int64) (float64, float64) {
+	if len(values) == 0 {
 		return nan, nan
 	}
 	if areConstValues(values) {
@@ -918,25 +913,32 @@ func linearRegression(rfa *rollupFuncArg) (float64, float64) {
 	}
 
 	// See https://en.wikipedia.org/wiki/Simple_linear_regression#Numerical_example
-	interceptTime := rfa.currTimestamp
 	vSum := float64(0)
 	tSum := float64(0)
 	tvSum := float64(0)
 	ttSum := float64(0)
+	n := 0
 	for i, v := range values {
+		if math.IsNaN(v) {
+			continue
+		}
 		dt := float64(timestamps[i]-interceptTime) / 1e3
 		vSum += v
 		tSum += dt
 		tvSum += dt * v
 		ttSum += dt * dt
+		n++
+	}
+	if n == 0 {
+		return nan, nan
 	}
 	k := float64(0)
-	tDiff := ttSum - tSum*tSum/n
+	tDiff := ttSum - tSum*tSum/float64(n)
 	if math.Abs(tDiff) >= 1e-6 {
 		// Prevent from incorrect division for too small tDiff values.
-		k = (tvSum - tSum*vSum/n) / tDiff
+		k = (tvSum - tSum*vSum/float64(n)) / tDiff
 	}
-	v := vSum/n - k*tSum/n
+	v := vSum/float64(n) - k*tSum/float64(n)
 	return v, k
 }
 
@@ -1473,16 +1475,20 @@ func rollupStaleSamples(rfa *rollupFuncArg) float64 {
 }
 
 func rollupStddev(rfa *rollupFuncArg) float64 {
-	stdvar := rollupStdvar(rfa)
-	return math.Sqrt(stdvar)
+	return stddev(rfa.values)
 }
 
 func rollupStdvar(rfa *rollupFuncArg) float64 {
-	// See `Rapid calculation methods` at https://en.wikipedia.org/wiki/Standard_deviation
+	return stdvar(rfa.values)
+}
 
-	// There is no need in handling NaNs here, since they must be cleaned up
-	// before calling rollup funcs.
-	values := rfa.values
+func stddev(values []float64) float64 {
+	v := stdvar(values)
+	return math.Sqrt(v)
+}
+
+func stdvar(values []float64) float64 {
+	// See `Rapid calculation methods` at https://en.wikipedia.org/wiki/Standard_deviation
 	if len(values) == 0 {
 		return nan
 	}
@@ -1494,11 +1500,17 @@ func rollupStdvar(rfa *rollupFuncArg) float64 {
 	var count float64
 	var q float64
 	for _, v := range values {
+		if math.IsNaN(v) {
+			continue
+		}
 		count++
 		avgNew := avg + (v-avg)/count
 		q += (v - avg) * (v - avgNew)
 		avg = avgNew
 	}
+	if count == 0 {
+		return nan
+	}
 	return q / count
 }
 
@@ -1605,7 +1617,7 @@ func rollupIdelta(rfa *rollupFuncArg) float64 {
 func rollupDerivSlow(rfa *rollupFuncArg) float64 {
 	// Use linear regression like Prometheus does.
 	// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/73
-	_, k := linearRegression(rfa)
+	_, k := linearRegression(rfa.values, rfa.timestamps, rfa.currTimestamp)
 	return k
 }
 
@@ -2115,7 +2127,7 @@ func getIntNumber(arg interface{}, argNum int) (int, error) {
 	}
 	n := 0
 	if len(v) > 0 {
-		n = int(v[0])
+		n = floatToIntBounded(v[0])
 	}
 	return n, nil
 }

app/vmselect/promql/rollup_result_cache.go

@@ -152,6 +152,9 @@ func InitRollupResultCache(cachePath string) {
 	metrics.GetOrCreateGauge(`vm_cache_size_bytes{type="promql/rollupResult"}`, func() float64 {
 		return float64(fcs().BytesSize)
 	})
+	metrics.GetOrCreateGauge(`vm_cache_size_max_bytes{type="promql/rollupResult"}`, func() float64 {
+		return float64(fcs().MaxBytesSize)
+	})
 	metrics.GetOrCreateGauge(`vm_cache_requests_total{type="promql/rollupResult"}`, func() float64 {
 		return float64(fcs().GetCalls)
 	})
@@ -430,8 +433,7 @@ func mustLoadRollupResultCacheKeyPrefix(path string) {
 func mustSaveRollupResultCacheKeyPrefix(path string) {
 	path = path + ".key.prefix"
 	data := encoding.MarshalUint64(nil, rollupResultCacheKeyPrefix)
-	fs.MustRemoveAll(path)
-	if err := fs.WriteFileAtomically(path, data); err != nil {
+	if err := fs.WriteFileAtomically(path, data, true); err != nil {
 		logger.Fatalf("cannot store rollupResult cache key prefix to %q: %s", path, err)
 	}
 }

app/vmselect/promql/rollup_test.go

@@ -388,12 +388,7 @@ func TestRollupPredictLinear(t *testing.T) {
 func TestLinearRegression(t *testing.T) {
 	f := func(values []float64, timestamps []int64, expV, expK float64) {
 		t.Helper()
-		rfa := &rollupFuncArg{
-			values:        values,
-			timestamps:    timestamps,
-			currTimestamp: timestamps[0] + 100,
-		}
-		v, k := linearRegression(rfa)
+		v, k := linearRegression(values, timestamps, timestamps[0]+100)
 		if err := compareValues([]float64{v}, []float64{expV}); err != nil {
 			t.Fatalf("unexpected v err: %s", err)
 		}

app/vmselect/promql/transform.go

@@ -88,10 +88,15 @@ var transformFuncs = map[string]transformFunc{
 	"range_avg":                  newTransformFuncRange(runningAvg),
 	"range_first":                transformRangeFirst,
 	"range_last":                 transformRangeLast,
+	"range_linear_regression":    transformRangeLinearRegression,
 	"range_max":                  newTransformFuncRange(runningMax),
 	"range_min":                  newTransformFuncRange(runningMin),
+	"range_normalize":            transformRangeNormalize,
 	"range_quantile":             transformRangeQuantile,
+	"range_stddev":               transformRangeStddev,
+	"range_stdvar":               transformRangeStdvar,
 	"range_sum":                  newTransformFuncRange(runningSum),
+	"range_trim_spikes":          transformRangeTrimSpikes,
 	"remove_resets":              transformRemoveResets,
 	"round":                      transformRound,
 	"running_avg":                newTransformFuncRunning(runningAvg),
@@ -125,25 +130,29 @@ var transformFuncs = map[string]transformFunc{
 // These functions don't change physical meaning of input time series,
 // so they don't drop metric name
 var transformFuncsKeepMetricName = map[string]bool{
-	"ceil":               true,
-	"clamp":              true,
-	"clamp_max":          true,
-	"clamp_min":          true,
-	"floor":              true,
-	"interpolate":        true,
-	"keep_last_value":    true,
-	"keep_next_value":    true,
-	"range_avg":          true,
-	"range_first":        true,
-	"range_last":         true,
-	"range_max":          true,
-	"range_min":          true,
-	"range_quantile":     true,
-	"round":              true,
-	"running_avg":        true,
-	"running_max":        true,
-	"running_min":        true,
-	"smooth_exponential": true,
+	"ceil":                    true,
+	"clamp":                   true,
+	"clamp_max":               true,
+	"clamp_min":               true,
+	"floor":                   true,
+	"interpolate":             true,
+	"keep_last_value":         true,
+	"keep_next_value":         true,
+	"range_avg":               true,
+	"range_first":             true,
+	"range_last":              true,
+	"range_linear_regression": true,
+	"range_max":               true,
+	"range_min":               true,
+	"range_normalize":         true,
+	"range_quantile":          true,
+	"range_stdvar":            true,
+	"range_sddev":             true,
+	"round":                   true,
+	"running_avg":             true,
+	"running_max":             true,
+	"running_min":             true,
+	"smooth_exponential":      true,
 }
 
 func getTransformFunc(s string) transformFunc {
@@ -362,14 +371,10 @@ func transformBucketsLimit(tfa *transformFuncArg) ([]*timeseries, error) {
 	if err := expectTransformArgsNum(args, 2); err != nil {
 		return nil, err
 	}
-	limits, err := getScalar(args[0], 1)
+	limit, err := getIntNumber(args[0], 0)
 	if err != nil {
 		return nil, err
 	}
-	limit := 0
-	if len(limits) > 0 {
-		limit = int(limits[0])
-	}
 	if limit <= 0 {
 		return nil, nil
 	}
@@ -381,6 +386,7 @@ func transformBucketsLimit(tfa *transformFuncArg) ([]*timeseries, error) {
 	if len(tss) == 0 {
 		return nil, nil
 	}
+	pointsCount := len(tss[0].Values)
 
 	// Group timeseries by all MetricGroup+tags excluding `le` tag.
 	type x struct {
@@ -428,7 +434,7 @@ func transformBucketsLimit(tfa *transformFuncArg) ([]*timeseries, error) {
 		sort.Slice(leGroup, func(i, j int) bool {
 			return leGroup[i].le < leGroup[j].le
 		})
-		for n := range limits {
+		for n := 0; n < pointsCount; n++ {
 			prevValue := float64(0)
 			for i := range leGroup {
 				xx := &leGroup[i]
@@ -665,6 +671,7 @@ func transformHistogramShare(tfa *transformFuncArg) ([]*timeseries, error) {
 		sort.Slice(xss, func(i, j int) bool {
 			return xss[i].le < xss[j].le
 		})
+		xss = mergeSameLE(xss)
 		dst := xss[0].ts
 		var tsLower, tsUpper *timeseries
 		if len(boundsLabel) > 0 {
@@ -945,6 +952,7 @@ func transformHistogramQuantile(tfa *transformFuncArg) ([]*timeseries, error) {
 		sort.Slice(xss, func(i, j int) bool {
 			return xss[i].le < xss[j].le
 		})
+		xss = mergeSameLE(xss)
 		dst := xss[0].ts
 		var tsLower, tsUpper *timeseries
 		if len(boundsLabel) > 0 {
@@ -1012,6 +1020,7 @@ func fixBrokenBuckets(i int, xss []leTimeseries) {
 	if len(xss) < 2 {
 		return
 	}
+	// Fill NaN in upper buckets with the first non-NaN value found in lower buckets.
 	for j := len(xss) - 1; j >= 0; j-- {
 		v := xss[j].ts.Values[i]
 		if !math.IsNaN(v) {
@@ -1023,6 +1032,8 @@ func fixBrokenBuckets(i int, xss []leTimeseries) {
 			break
 		}
 	}
+	// Substitute lower bucket values with upper values if the lower values are NaN
+	// or are bigger than the upper bucket values.
 	vNext := xss[len(xss)-1].ts.Values[i]
 	for j := len(xss) - 2; j >= 0; j-- {
 		v := xss[j].ts.Values[i]
@@ -1034,6 +1045,26 @@ func fixBrokenBuckets(i int, xss []leTimeseries) {
 	}
 }
 
+func mergeSameLE(xss []leTimeseries) []leTimeseries {
+	// Merge buckets with identical le values.
+	// See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3225
+	xsDst := xss[0]
+	dst := xss[:1]
+	for j := 1; j < len(xss); j++ {
+		xs := xss[j]
+		if xs.le != xsDst.le {
+			dst = append(dst, xs)
+			xsDst = xs
+			continue
+		}
+		dstValues := xsDst.ts.Values
+		for k, v := range xs.ts.Values {
+			dstValues[k] += v
+		}
+	}
+	return dst
+}
+
 func transformHour(t time.Time) int {
 	return t.Hour()
 }
@@ -1209,6 +1240,139 @@ func newTransformFuncRange(rf func(a, b float64, idx int) float64) transformFunc
 	}
 }
 
+func transformRangeNormalize(tfa *transformFuncArg) ([]*timeseries, error) {
+	args := tfa.args
+	var rvs []*timeseries
+	for _, tss := range args {
+		for _, ts := range tss {
+			values := ts.Values
+			vMin := inf
+			vMax := -inf
+			for _, v := range values {
+				if math.IsNaN(v) {
+					continue
+				}
+				if v < vMin {
+					vMin = v
+				}
+				if v > vMax {
+					vMax = v
+				}
+			}
+			d := vMax - vMin
+			if math.IsInf(d, 0) {
+				continue
+			}
+			for i, v := range values {
+				values[i] = (v - vMin) / d
+			}
+			rvs = append(rvs, ts)
+		}
+	}
+	return rvs, nil
+}
+
+func transformRangeTrimSpikes(tfa *transformFuncArg) ([]*timeseries, error) {
+	args := tfa.args
+	if err := expectTransformArgsNum(args, 2); err != nil {
+		return nil, err
+	}
+	phis, err := getScalar(args[0], 0)
+	if err != nil {
+		return nil, err
+	}
+	phi := float64(0)
+	if len(phis) > 0 {
+		phi = phis[0]
+	}
+	// Trim 100% * (phi / 2) samples with the lowest / highest values per each time series
+	phi /= 2
+	phiUpper := 1 - phi
+	phiLower := phi
+	rvs := args[1]
+	a := getFloat64s()
+	values := a.A[:0]
+	for _, ts := range rvs {
+		values := values[:0]
+		originValues := ts.Values
+		for _, v := range originValues {
+			if math.IsNaN(v) {
+				continue
+			}
+			values = append(values, v)
+		}
+		sort.Float64s(values)
+		vMax := quantileSorted(phiUpper, values)
+		vMin := quantileSorted(phiLower, values)
+		for i, v := range originValues {
+			if math.IsNaN(v) {
+				continue
+			}
+			if v > vMax {
+				originValues[i] = nan
+			} else if v < vMin {
+				originValues[i] = nan
+			}
+		}
+	}
+	a.A = values
+	putFloat64s(a)
+	return rvs, nil
+}
+
+func transformRangeLinearRegression(tfa *transformFuncArg) ([]*timeseries, error) {
+	args := tfa.args
+	if err := expectTransformArgsNum(args, 1); err != nil {
+		return nil, err
+	}
+	rvs := args[0]
+	for _, ts := range rvs {
+		values := ts.Values
+		timestamps := ts.Timestamps
+		if len(timestamps) == 0 {
+			continue
+		}
+		interceptTimestamp := timestamps[0]
+		v, k := linearRegression(values, timestamps, interceptTimestamp)
+		for i, t := range timestamps {
+			values[i] = v + k*float64(t-interceptTimestamp)/1e3
+		}
+	}
+	return rvs, nil
+}
+
+func transformRangeStddev(tfa *transformFuncArg) ([]*timeseries, error) {
+	args := tfa.args
+	if err := expectTransformArgsNum(args, 1); err != nil {
+		return nil, err
+	}
+	rvs := args[0]
+	for _, ts := range rvs {
+		values := ts.Values
+		v := stddev(values)
+		for i := range values {
+			values[i] = v
+		}
+	}
+	return rvs, nil
+}
+
+func transformRangeStdvar(tfa *transformFuncArg) ([]*timeseries, error) {
+	args := tfa.args
+	if err := expectTransformArgsNum(args, 1); err != nil {
+		return nil, err
+	}
+	rvs := args[0]
+	for _, ts := range rvs {
+		values := ts.Values
+		v := stdvar(values)
+		for i := range values {
+			values[i] = v
+		}
+	}
+	return rvs, nil
+}
+
 func transformRangeQuantile(tfa *transformFuncArg) ([]*timeseries, error) {
 	args := tfa.args
 	if err := expectTransformArgsNum(args, 2); err != nil {
@@ -1859,6 +2023,8 @@ func transformLimitOffset(tfa *transformFuncArg) ([]*timeseries, error) {
 	rvs := removeEmptySeries(args[2])
 	if len(rvs) >= offset {
 		rvs = rvs[offset:]
+	} else {
+		rvs = nil
 	}
 	if len(rvs) > limit {
 		rvs = rvs[:limit]

app/vmselect/searchutils/searchutils.go

@@ -209,7 +209,7 @@ func (d *Deadline) String() string {
 	startTime := time.Unix(int64(d.deadline), 0).Add(-d.timeout)
 	elapsed := time.Since(startTime)
 	msg := fmt.Sprintf("%.3f seconds (elapsed %.3f seconds)", d.timeout.Seconds(), elapsed.Seconds())
-	if d.flagHint != "" {
+	if float64(elapsed)/float64(d.timeout) > 0.9 && d.flagHint != "" {
 		msg += fmt.Sprintf("; the timeout can be adjusted with `%s` command-line flag", d.flagHint)
 	}
 	return msg

app/vmselect/vmui/asset-manifest.json

@@ -1,12 +1,12 @@
 {
   "files": {
-    "main.css": "./static/css/main.ba692000.css",
-    "main.js": "./static/js/main.623b88d4.js",
-    "static/js/27.939f971b.chunk.js": "./static/js/27.939f971b.chunk.js",
+    "main.css": "./static/css/main.89abca0f.css",
+    "main.js": "./static/js/main.c552245f.js",
+    "static/js/27.c1ccfd29.chunk.js": "./static/js/27.c1ccfd29.chunk.js",
     "index.html": "./index.html"
   },
   "entrypoints": [
-    "static/css/main.ba692000.css",
-    "static/js/main.623b88d4.js"
+    "static/css/main.89abca0f.css",
+    "static/js/main.c552245f.js"
   ]
 }

app/vmselect/vmui/index.html

@@ -1 +1 @@
-<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="VM-UI is a metric explorer for Victoria Metrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,500,700&display=swap"/><script src="./dashboards/index.js" type="module"></script><script defer="defer" src="./static/js/main.623b88d4.js"></script><link href="./static/css/main.ba692000.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>
+<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="VM-UI is a metric explorer for Victoria Metrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><link rel="preconnect" href="https://fonts.googleapis.com"><link rel="preconnect" href="https://fonts.gstatic.com" crossorigin><link href="https://fonts.googleapis.com/css2?family=JetBrains+Mono&family=Lato:wght@300;400;700&display=swap" rel="stylesheet"><script src="./dashboards/index.js" type="module"></script><script defer="defer" src="./static/js/main.c552245f.js"></script><link href="./static/css/main.89abca0f.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>

app/vmselect/vmui/static/css/main.ba692000.css

@@ -1 +0,0 @@
-body{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;font-family:-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Oxygen,Ubuntu,Cantarell,Fira Sans,Droid Sans,Helvetica Neue,sans-serif}code{font-family:source-code-pro,Menlo,Monaco,Consolas,Courier New,monospace}.MuiAccordionSummary-content{margin:0!important}.shortcut-key{align-items:center;border:1px solid #dedede;border-radius:4px;cursor:default;display:inline-flex;font-size:10px;justify-content:center;line-height:22px;padding:2px 6px 0;text-align:center;white-space:nowrap}.uplot,.uplot *,.uplot :after,.uplot :before{box-sizing:border-box}.uplot{font-family:system-ui,-apple-system,Segoe UI,Roboto,Helvetica Neue,Arial,Noto Sans,sans-serif,Apple Color Emoji,Segoe UI Emoji,Segoe UI Symbol,Noto Color Emoji;line-height:1.5;width:-webkit-min-content;width:min-content}.u-title{font-size:18px;font-weight:700;text-align:center}.u-wrap{position:relative;-webkit-user-select:none;-ms-user-select:none;user-select:none}.u-over,.u-under{position:absolute}.u-under{overflow:hidden}.uplot canvas{display:block;height:100%;position:relative;width:100%}.u-axis{position:absolute}.u-legend{font-size:14px;margin:auto;text-align:center}.u-inline{display:block}.u-inline *{display:inline-block}.u-inline tr{margin-right:16px}.u-legend th{font-weight:600}.u-legend th>*{display:inline-block;vertical-align:middle}.u-legend .u-marker{background-clip:padding-box!important;height:1em;margin-right:4px;width:1em}.u-inline.u-live th:after{content:":";vertical-align:middle}.u-inline:not(.u-live) .u-value{display:none}.u-series>*{padding:4px}.u-series th{cursor:pointer}.u-legend .u-off>*{opacity:.3}.u-select{background:rgba(0,0,0,.07)}.u-cursor-x,.u-cursor-y,.u-select{pointer-events:none;position:absolute}.u-cursor-x,.u-cursor-y{left:0;top:0;will-change:transform;z-index:100}.u-hz .u-cursor-x,.u-vt .u-cursor-y{border-right:1px dashed #607d8b;height:100%}.u-hz .u-cursor-y,.u-vt .u-cursor-x{border-bottom:1px dashed #607d8b;width:100%}.u-cursor-pt{background-clip:padding-box!important;border:0 solid;border-radius:50%;left:0;pointer-events:none;position:absolute;top:0;will-change:transform;z-index:100}.u-axis.u-off,.u-cursor-pt.u-off,.u-cursor-x.u-off,.u-cursor-y.u-off,.u-select.u-off,.u-tooltip{display:none}.u-tooltip{grid-gap:12px;word-wrap:break-word;background:rgba(57,57,57,.9);border-radius:4px;color:#fff;font-family:monospace;font-size:10px;font-weight:500;line-height:1.4em;max-width:300px;padding:8px;pointer-events:none;position:absolute;z-index:100}.u-tooltip-data{align-items:center;display:flex;flex-wrap:wrap;font-size:11px;line-height:150%}.u-tooltip-data__value{font-weight:700;padding:4px}.u-tooltip__info{grid-gap:4px;display:grid}.u-tooltip__marker{height:12px;margin-right:4px;width:12px}.legendWrapper{cursor:default;display:flex;flex-wrap:wrap;margin-top:20px;position:relative}.legendGroup{margin:0 12px 0 0;padding:10px 6px}.legendGroupTitle{align-items:center;border-bottom:1px solid #ecebe6;display:flex;font-size:11px;margin-bottom:5px;padding:0 10px 5px}.legendGroupQuery{font-weight:700;margin-right:4px}.legendGroupLine{margin-right:10px}.legendItem{grid-gap:6px;align-items:start;background-color:#fff;cursor:pointer;display:grid;grid-template-columns:auto auto;justify-content:start;padding:7px 50px 7px 10px;transition:.2s ease}.legendItemHide{opacity:.5;text-decoration:line-through}.legendItem:hover{background-color:rgba(0,0,0,.1)}.legendMarker{box-sizing:border-box;height:12px;transition:.2s ease;width:12px}.legendLabel{font-size:11px;font-weight:400;line-height:12px}.legendFreeFields{cursor:pointer;padding:3px}.legendFreeFields:hover{text-decoration:underline}.legendFreeFields:not(:last-child):after{content:","}.panelDescription ul{line-height:2.2}.panelDescription a{color:#fff}.panelDescription code{background-color:rgba(0,0,0,.3);border-radius:2px;color:#fff;display:inline;font-size:inherit;font-weight:400;max-width:100%;padding:4px 6px}

app/vmselect/vmui/static/js/27.939f971b.chunk.js

@@ -1 +0,0 @@
-"use strict";(self.webpackChunkvmui=self.webpackChunkvmui||[]).push([[27],{4027:function(e,t,n){n.r(t),n.d(t,{getCLS:function(){return y},getFCP:function(){return g},getFID:function(){return C},getLCP:function(){return P},getTTFB:function(){return D}});var i,r,a,o,u=function(e,t){return{name:e,value:void 0===t?-1:t,delta:0,entries:[],id:"v2-".concat(Date.now(),"-").concat(Math.floor(8999999999999*Math.random())+1e12)}},c=function(e,t){try{if(PerformanceObserver.supportedEntryTypes.includes(e)){if("first-input"===e&&!("PerformanceEventTiming"in self))return;var n=new PerformanceObserver((function(e){return e.getEntries().map(t)}));return n.observe({type:e,buffered:!0}),n}}catch(e){}},f=function(e,t){var n=function n(i){"pagehide"!==i.type&&"hidden"!==document.visibilityState||(e(i),t&&(removeEventListener("visibilitychange",n,!0),removeEventListener("pagehide",n,!0)))};addEventListener("visibilitychange",n,!0),addEventListener("pagehide",n,!0)},s=function(e){addEventListener("pageshow",(function(t){t.persisted&&e(t)}),!0)},m=function(e,t,n){var i;return function(r){t.value>=0&&(r||n)&&(t.delta=t.value-(i||0),(t.delta||void 0===i)&&(i=t.value,e(t)))}},v=-1,p=function(){return"hidden"===document.visibilityState?0:1/0},d=function(){f((function(e){var t=e.timeStamp;v=t}),!0)},l=function(){return v<0&&(v=p(),d(),s((function(){setTimeout((function(){v=p(),d()}),0)}))),{get firstHiddenTime(){return v}}},g=function(e,t){var n,i=l(),r=u("FCP"),a=function(e){"first-contentful-paint"===e.name&&(f&&f.disconnect(),e.startTime<i.firstHiddenTime&&(r.value=e.startTime,r.entries.push(e),n(!0)))},o=window.performance&&performance.getEntriesByName&&performance.getEntriesByName("first-contentful-paint")[0],f=o?null:c("paint",a);(o||f)&&(n=m(e,r,t),o&&a(o),s((function(i){r=u("FCP"),n=m(e,r,t),requestAnimationFrame((function(){requestAnimationFrame((function(){r.value=performance.now()-i.timeStamp,n(!0)}))}))})))},h=!1,T=-1,y=function(e,t){h||(g((function(e){T=e.value})),h=!0);var n,i=function(t){T>-1&&e(t)},r=u("CLS",0),a=0,o=[],v=function(e){if(!e.hadRecentInput){var t=o[0],i=o[o.length-1];a&&e.startTime-i.startTime<1e3&&e.startTime-t.startTime<5e3?(a+=e.value,o.push(e)):(a=e.value,o=[e]),a>r.value&&(r.value=a,r.entries=o,n())}},p=c("layout-shift",v);p&&(n=m(i,r,t),f((function(){p.takeRecords().map(v),n(!0)})),s((function(){a=0,T=-1,r=u("CLS",0),n=m(i,r,t)})))},E={passive:!0,capture:!0},w=new Date,L=function(e,t){i||(i=t,r=e,a=new Date,F(removeEventListener),S())},S=function(){if(r>=0&&r<a-w){var e={entryType:"first-input",name:i.type,target:i.target,cancelable:i.cancelable,startTime:i.timeStamp,processingStart:i.timeStamp+r};o.forEach((function(t){t(e)})),o=[]}},b=function(e){if(e.cancelable){var t=(e.timeStamp>1e12?new Date:performance.now())-e.timeStamp;"pointerdown"==e.type?function(e,t){var n=function(){L(e,t),r()},i=function(){r()},r=function(){removeEventListener("pointerup",n,E),removeEventListener("pointercancel",i,E)};addEventListener("pointerup",n,E),addEventListener("pointercancel",i,E)}(t,e):L(t,e)}},F=function(e){["mousedown","keydown","touchstart","pointerdown"].forEach((function(t){return e(t,b,E)}))},C=function(e,t){var n,a=l(),v=u("FID"),p=function(e){e.startTime<a.firstHiddenTime&&(v.value=e.processingStart-e.startTime,v.entries.push(e),n(!0))},d=c("first-input",p);n=m(e,v,t),d&&f((function(){d.takeRecords().map(p),d.disconnect()}),!0),d&&s((function(){var a;v=u("FID"),n=m(e,v,t),o=[],r=-1,i=null,F(addEventListener),a=p,o.push(a),S()}))},k={},P=function(e,t){var n,i=l(),r=u("LCP"),a=function(e){var t=e.startTime;t<i.firstHiddenTime&&(r.value=t,r.entries.push(e),n())},o=c("largest-contentful-paint",a);if(o){n=m(e,r,t);var v=function(){k[r.id]||(o.takeRecords().map(a),o.disconnect(),k[r.id]=!0,n(!0))};["keydown","click"].forEach((function(e){addEventListener(e,v,{once:!0,capture:!0})})),f(v,!0),s((function(i){r=u("LCP"),n=m(e,r,t),requestAnimationFrame((function(){requestAnimationFrame((function(){r.value=performance.now()-i.timeStamp,k[r.id]=!0,n(!0)}))}))}))}},D=function(e){var t,n=u("TTFB");t=function(){try{var t=performance.getEntriesByType("navigation")[0]||function(){var e=performance.timing,t={entryType:"navigation",startTime:0};for(var n in e)"navigationStart"!==n&&"toJSON"!==n&&(t[n]=Math.max(e[n]-e.navigationStart,0));return t}();if(n.value=n.delta=t.responseStart,n.value<0||n.value>performance.now())return;n.entries=[t],e(n)}catch(e){}},"complete"===document.readyState?setTimeout(t,0):addEventListener("load",(function(){return setTimeout(t,0)}))}}}]);

app/vmselect/vmui/static/js/27.c1ccfd29.chunk.js

@@ -0,0 +1 @@
+"use strict";(self.webpackChunkvmui=self.webpackChunkvmui||[]).push([[27],{27:function(e,t,n){n.r(t),n.d(t,{getCLS:function(){return y},getFCP:function(){return g},getFID:function(){return C},getLCP:function(){return P},getTTFB:function(){return D}});var i,r,a,o,u=function(e,t){return{name:e,value:void 0===t?-1:t,delta:0,entries:[],id:"v2-".concat(Date.now(),"-").concat(Math.floor(8999999999999*Math.random())+1e12)}},c=function(e,t){try{if(PerformanceObserver.supportedEntryTypes.includes(e)){if("first-input"===e&&!("PerformanceEventTiming"in self))return;var n=new PerformanceObserver((function(e){return e.getEntries().map(t)}));return n.observe({type:e,buffered:!0}),n}}catch(e){}},f=function(e,t){var n=function n(i){"pagehide"!==i.type&&"hidden"!==document.visibilityState||(e(i),t&&(removeEventListener("visibilitychange",n,!0),removeEventListener("pagehide",n,!0)))};addEventListener("visibilitychange",n,!0),addEventListener("pagehide",n,!0)},s=function(e){addEventListener("pageshow",(function(t){t.persisted&&e(t)}),!0)},m=function(e,t,n){var i;return function(r){t.value>=0&&(r||n)&&(t.delta=t.value-(i||0),(t.delta||void 0===i)&&(i=t.value,e(t)))}},v=-1,p=function(){return"hidden"===document.visibilityState?0:1/0},d=function(){f((function(e){var t=e.timeStamp;v=t}),!0)},l=function(){return v<0&&(v=p(),d(),s((function(){setTimeout((function(){v=p(),d()}),0)}))),{get firstHiddenTime(){return v}}},g=function(e,t){var n,i=l(),r=u("FCP"),a=function(e){"first-contentful-paint"===e.name&&(f&&f.disconnect(),e.startTime<i.firstHiddenTime&&(r.value=e.startTime,r.entries.push(e),n(!0)))},o=window.performance&&performance.getEntriesByName&&performance.getEntriesByName("first-contentful-paint")[0],f=o?null:c("paint",a);(o||f)&&(n=m(e,r,t),o&&a(o),s((function(i){r=u("FCP"),n=m(e,r,t),requestAnimationFrame((function(){requestAnimationFrame((function(){r.value=performance.now()-i.timeStamp,n(!0)}))}))})))},h=!1,T=-1,y=function(e,t){h||(g((function(e){T=e.value})),h=!0);var n,i=function(t){T>-1&&e(t)},r=u("CLS",0),a=0,o=[],v=function(e){if(!e.hadRecentInput){var t=o[0],i=o[o.length-1];a&&e.startTime-i.startTime<1e3&&e.startTime-t.startTime<5e3?(a+=e.value,o.push(e)):(a=e.value,o=[e]),a>r.value&&(r.value=a,r.entries=o,n())}},p=c("layout-shift",v);p&&(n=m(i,r,t),f((function(){p.takeRecords().map(v),n(!0)})),s((function(){a=0,T=-1,r=u("CLS",0),n=m(i,r,t)})))},E={passive:!0,capture:!0},w=new Date,L=function(e,t){i||(i=t,r=e,a=new Date,F(removeEventListener),S())},S=function(){if(r>=0&&r<a-w){var e={entryType:"first-input",name:i.type,target:i.target,cancelable:i.cancelable,startTime:i.timeStamp,processingStart:i.timeStamp+r};o.forEach((function(t){t(e)})),o=[]}},b=function(e){if(e.cancelable){var t=(e.timeStamp>1e12?new Date:performance.now())-e.timeStamp;"pointerdown"==e.type?function(e,t){var n=function(){L(e,t),r()},i=function(){r()},r=function(){removeEventListener("pointerup",n,E),removeEventListener("pointercancel",i,E)};addEventListener("pointerup",n,E),addEventListener("pointercancel",i,E)}(t,e):L(t,e)}},F=function(e){["mousedown","keydown","touchstart","pointerdown"].forEach((function(t){return e(t,b,E)}))},C=function(e,t){var n,a=l(),v=u("FID"),p=function(e){e.startTime<a.firstHiddenTime&&(v.value=e.processingStart-e.startTime,v.entries.push(e),n(!0))},d=c("first-input",p);n=m(e,v,t),d&&f((function(){d.takeRecords().map(p),d.disconnect()}),!0),d&&s((function(){var a;v=u("FID"),n=m(e,v,t),o=[],r=-1,i=null,F(addEventListener),a=p,o.push(a),S()}))},k={},P=function(e,t){var n,i=l(),r=u("LCP"),a=function(e){var t=e.startTime;t<i.firstHiddenTime&&(r.value=t,r.entries.push(e),n())},o=c("largest-contentful-paint",a);if(o){n=m(e,r,t);var v=function(){k[r.id]||(o.takeRecords().map(a),o.disconnect(),k[r.id]=!0,n(!0))};["keydown","click"].forEach((function(e){addEventListener(e,v,{once:!0,capture:!0})})),f(v,!0),s((function(i){r=u("LCP"),n=m(e,r,t),requestAnimationFrame((function(){requestAnimationFrame((function(){r.value=performance.now()-i.timeStamp,k[r.id]=!0,n(!0)}))}))}))}},D=function(e){var t,n=u("TTFB");t=function(){try{var t=performance.getEntriesByType("navigation")[0]||function(){var e=performance.timing,t={entryType:"navigation",startTime:0};for(var n in e)"navigationStart"!==n&&"toJSON"!==n&&(t[n]=Math.max(e[n]-e.navigationStart,0));return t}();if(n.value=n.delta=t.responseStart,n.value<0||n.value>performance.now())return;n.entries=[t],e(n)}catch(e){}},"complete"===document.readyState?setTimeout(t,0):addEventListener("load",(function(){return setTimeout(t,0)}))}}}]);

app/vmselect/vmui/static/js/main.623b88d4.js.LICENSE.txt

@@ -1,45 +0,0 @@
-/**
- * React Router DOM v6.3.0
- *
- * Copyright (c) Remix Software Inc.
- *
- * This source code is licensed under the MIT license found in the
- * LICENSE.md file in the root directory of this source tree.
- *
- * @license MIT
- */
-
-/**
- * React Router v6.3.0
- *
- * Copyright (c) Remix Software Inc.
- *
- * This source code is licensed under the MIT license found in the
- * LICENSE.md file in the root directory of this source tree.
- *
- * @license MIT
- */
-
-/** @license MUI v5.6.1
- *
- * This source code is licensed under the MIT license found in the
- * LICENSE file in the root directory of this source tree.
- */
-
-/** @license React v16.13.1
- * react-is.production.min.js
- *
- * Copyright (c) Facebook, Inc. and its affiliates.
- *
- * This source code is licensed under the MIT license found in the
- * LICENSE file in the root directory of this source tree.
- */
-
-/** @license React v17.0.2
- * react-is.production.min.js
- *
- * Copyright (c) Facebook, Inc. and its affiliates.
- *
- * This source code is licensed under the MIT license found in the
- * LICENSE file in the root directory of this source tree.
- */

app/vmselect/vmui/static/js/main.c552245f.js.LICENSE.txt

@@ -0,0 +1,29 @@
+/*!
+	Copyright (c) 2018 Jed Watson.
+	Licensed under the MIT License (MIT), see
+	http://jedwatson.github.io/classnames
+*/
+
+/*! regenerator-runtime -- Copyright (c) 2014-present, Facebook, Inc. -- license (MIT): https://github.com/facebook/regenerator/blob/main/LICENSE */
+
+/**
+ * @remix-run/router v1.0.3
+ *
+ * Copyright (c) Remix Software Inc.
+ *
+ * This source code is licensed under the MIT license found in the
+ * LICENSE.md file in the root directory of this source tree.
+ *
+ * @license MIT
+ */
+
+/**
+ * React Router v6.4.3
+ *
+ * Copyright (c) Remix Software Inc.
+ *
+ * This source code is licensed under the MIT license found in the
+ * LICENSE.md file in the root directory of this source tree.
+ *
+ * @license MIT
+ */

app/vmstorage/main.go

@@ -23,7 +23,7 @@ import (
 )
 
 var (
-	retentionPeriod   = flagutil.NewDuration("retentionPeriod", "1", "Data with timestamps outside the retentionPeriod is automatically deleted")
+	retentionPeriod   = flagutil.NewDuration("retentionPeriod", "1", "Data with timestamps outside the retentionPeriod is automatically deleted. See also -retentionFilter")
 	snapshotAuthKey   = flag.String("snapshotAuthKey", "", "authKey, which must be passed in query string to /snapshot* pages")
 	forceMergeAuthKey = flag.String("forceMergeAuthKey", "", "authKey, which must be passed in query string to /internal/force_merge pages")
 	forceFlushAuthKey = flag.String("forceFlushAuthKey", "", "authKey, which must be passed in query string to /internal/force_flush pages")
@@ -57,10 +57,14 @@ var (
 
 	minFreeDiskSpaceBytes = flagutil.NewBytes("storage.minFreeDiskSpaceBytes", 10e6, "The minimum free disk space at -storageDataPath after which the storage stops accepting new data")
 
-	cacheSizeStorageTSID        = flagutil.NewBytes("storage.cacheSizeStorageTSID", 0, "Overrides max size for storage/tsid cache. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning")
-	cacheSizeIndexDBIndexBlocks = flagutil.NewBytes("storage.cacheSizeIndexDBIndexBlocks", 0, "Overrides max size for indexdb/indexBlocks cache. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning")
-	cacheSizeIndexDBDataBlocks  = flagutil.NewBytes("storage.cacheSizeIndexDBDataBlocks", 0, "Overrides max size for indexdb/dataBlocks cache. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning")
-	cacheSizeIndexDBTagFilters  = flagutil.NewBytes("storage.cacheSizeIndexDBTagFilters", 0, "Overrides max size for indexdb/tagFilters cache. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning")
+	cacheSizeStorageTSID = flagutil.NewBytes("storage.cacheSizeStorageTSID", 0, "Overrides max size for storage/tsid cache. "+
+		"See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning")
+	cacheSizeIndexDBIndexBlocks = flagutil.NewBytes("storage.cacheSizeIndexDBIndexBlocks", 0, "Overrides max size for indexdb/indexBlocks cache. "+
+		"See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning")
+	cacheSizeIndexDBDataBlocks = flagutil.NewBytes("storage.cacheSizeIndexDBDataBlocks", 0, "Overrides max size for indexdb/dataBlocks cache. "+
+		"See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning")
+	cacheSizeIndexDBTagFilters = flagutil.NewBytes("storage.cacheSizeIndexDBTagFilters", 0, "Overrides max size for indexdb/tagFiltersToMetricIDs cache. "+
+		"See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning")
 )
 
 // CheckTimeRange returns true if the given tr is denied for querying.
@@ -96,11 +100,11 @@ func InitWithoutMetrics(resetCacheIfNeeded func(mrs []storage.MetricRow)) {
 	storage.SetLogNewSeries(*logNewSeries)
 	storage.SetFinalMergeDelay(*finalMergeDelay)
 	storage.SetBigMergeWorkersCount(*bigMergeConcurrency)
-	storage.SetSmallMergeWorkersCount(*smallMergeConcurrency)
+	storage.SetMergeWorkersCount(*smallMergeConcurrency)
 	storage.SetRetentionTimezoneOffset(*retentionTimezoneOffset)
 	storage.SetFreeDiskSpaceLimit(minFreeDiskSpaceBytes.N)
 	storage.SetTSIDCacheSize(cacheSizeStorageTSID.N)
-	storage.SetTagFilterCacheSize(cacheSizeIndexDBTagFilters.N)
+	storage.SetTagFiltersCacheSize(cacheSizeIndexDBTagFilters.N)
 	mergeset.SetIndexBlocksCacheSize(cacheSizeIndexDBIndexBlocks.N)
 	mergeset.SetDataBlocksCacheSize(cacheSizeIndexDBDataBlocks.N)
 
@@ -329,12 +333,27 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 	case "/delete":
 		w.Header().Set("Content-Type", "application/json")
 		snapshotName := r.FormValue("snapshot")
-		if err := Storage.DeleteSnapshot(snapshotName); err != nil {
-			err = fmt.Errorf("cannot delete snapshot %q: %w", snapshotName, err)
+
+		snapshots, err := Storage.ListSnapshots()
+		if err != nil {
+			err = fmt.Errorf("cannot list snapshots: %w", err)
 			jsonResponseError(w, err)
 			return true
 		}
-		fmt.Fprintf(w, `{"status":"ok"}`)
+		for _, snName := range snapshots {
+			if snName == snapshotName {
+				if err := Storage.DeleteSnapshot(snName); err != nil {
+					err = fmt.Errorf("cannot delete snapshot %q: %w", snName, err)
+					jsonResponseError(w, err)
+					return true
+				}
+				fmt.Fprintf(w, `{"status":"ok"}`)
+				return true
+			}
+		}
+
+		err = fmt.Errorf("cannot find snapshot %q: %w", snapshotName, err)
+		jsonResponseError(w, err)
 		return true
 	case "/delete_all":
 		w.Header().Set("Content-Type", "application/json")
@@ -434,56 +453,80 @@ func registerStorageMetrics(strg *storage.Storage) {
 		return 0
 	})
 
-	metrics.NewGauge(`vm_active_merges{type="storage/big"}`, func() float64 {
-		return float64(tm().ActiveBigMerges)
+	metrics.NewGauge(`vm_active_merges{type="storage/inmemory"}`, func() float64 {
+		return float64(tm().ActiveInmemoryMerges)
 	})
 	metrics.NewGauge(`vm_active_merges{type="storage/small"}`, func() float64 {
 		return float64(tm().ActiveSmallMerges)
 	})
-	metrics.NewGauge(`vm_active_merges{type="indexdb"}`, func() float64 {
-		return float64(idbm().ActiveMerges)
+	metrics.NewGauge(`vm_active_merges{type="storage/big"}`, func() float64 {
+		return float64(tm().ActiveBigMerges)
+	})
+	metrics.NewGauge(`vm_active_merges{type="indexdb/inmemory"}`, func() float64 {
+		return float64(idbm().ActiveInmemoryMerges)
+	})
+	metrics.NewGauge(`vm_active_merges{type="indexdb/file"}`, func() float64 {
+		return float64(idbm().ActiveFileMerges)
 	})
 
-	metrics.NewGauge(`vm_merges_total{type="storage/big"}`, func() float64 {
-		return float64(tm().BigMergesCount)
+	metrics.NewGauge(`vm_merges_total{type="storage/inmemory"}`, func() float64 {
+		return float64(tm().InmemoryMergesCount)
 	})
 	metrics.NewGauge(`vm_merges_total{type="storage/small"}`, func() float64 {
 		return float64(tm().SmallMergesCount)
 	})
-	metrics.NewGauge(`vm_merges_total{type="indexdb"}`, func() float64 {
-		return float64(idbm().MergesCount)
+	metrics.NewGauge(`vm_merges_total{type="storage/big"}`, func() float64 {
+		return float64(tm().BigMergesCount)
+	})
+	metrics.NewGauge(`vm_merges_total{type="indexdb/inmemory"}`, func() float64 {
+		return float64(idbm().InmemoryMergesCount)
+	})
+	metrics.NewGauge(`vm_merges_total{type="indexdb/file"}`, func() float64 {
+		return float64(idbm().FileMergesCount)
 	})
 
-	metrics.NewGauge(`vm_rows_merged_total{type="storage/big"}`, func() float64 {
-		return float64(tm().BigRowsMerged)
+	metrics.NewGauge(`vm_rows_merged_total{type="storage/inmemory"}`, func() float64 {
+		return float64(tm().InmemoryRowsMerged)
 	})
 	metrics.NewGauge(`vm_rows_merged_total{type="storage/small"}`, func() float64 {
 		return float64(tm().SmallRowsMerged)
 	})
-	metrics.NewGauge(`vm_rows_merged_total{type="indexdb"}`, func() float64 {
-		return float64(idbm().ItemsMerged)
+	metrics.NewGauge(`vm_rows_merged_total{type="storage/big"}`, func() float64 {
+		return float64(tm().BigRowsMerged)
+	})
+	metrics.NewGauge(`vm_rows_merged_total{type="indexdb/inmemory"}`, func() float64 {
+		return float64(idbm().InmemoryItemsMerged)
+	})
+	metrics.NewGauge(`vm_rows_merged_total{type="indexdb/file"}`, func() float64 {
+		return float64(idbm().FileItemsMerged)
 	})
 
-	metrics.NewGauge(`vm_rows_deleted_total{type="storage/big"}`, func() float64 {
-		return float64(tm().BigRowsDeleted)
+	metrics.NewGauge(`vm_rows_deleted_total{type="storage/inmemory"}`, func() float64 {
+		return float64(tm().InmemoryRowsDeleted)
 	})
 	metrics.NewGauge(`vm_rows_deleted_total{type="storage/small"}`, func() float64 {
 		return float64(tm().SmallRowsDeleted)
 	})
-
-	metrics.NewGauge(`vm_references{type="storage/big", name="parts"}`, func() float64 {
-		return float64(tm().BigPartsRefCount)
+	metrics.NewGauge(`vm_rows_deleted_total{type="storage/big"}`, func() float64 {
+		return float64(tm().BigRowsDeleted)
 	})
-	metrics.NewGauge(`vm_references{type="storage/small", name="parts"}`, func() float64 {
+
+	metrics.NewGauge(`vm_part_references{type="storage/inmemory"}`, func() float64 {
+		return float64(tm().InmemoryPartsRefCount)
+	})
+	metrics.NewGauge(`vm_part_references{type="storage/small"}`, func() float64 {
 		return float64(tm().SmallPartsRefCount)
 	})
-	metrics.NewGauge(`vm_references{type="storage", name="partitions"}`, func() float64 {
+	metrics.NewGauge(`vm_part_references{type="storage/big"}`, func() float64 {
+		return float64(tm().BigPartsRefCount)
+	})
+	metrics.NewGauge(`vm_partition_references{type="storage"}`, func() float64 {
 		return float64(tm().PartitionsRefCount)
 	})
-	metrics.NewGauge(`vm_references{type="indexdb", name="objects"}`, func() float64 {
+	metrics.NewGauge(`vm_object_references{type="indexdb"}`, func() float64 {
 		return float64(idbm().IndexDBRefCount)
 	})
-	metrics.NewGauge(`vm_references{type="indexdb", name="parts"}`, func() float64 {
+	metrics.NewGauge(`vm_part_references{type="indexdb"}`, func() float64 {
 		return float64(idbm().PartsRefCount)
 	})
 
@@ -512,11 +555,11 @@ func registerStorageMetrics(strg *storage.Storage) {
 		return float64(idbm().CompositeFilterMissingConversions)
 	})
 
-	metrics.NewGauge(`vm_assisted_merges_total{type="storage/small"}`, func() float64 {
-		return float64(tm().SmallAssistedMerges)
+	metrics.NewGauge(`vm_assisted_merges_total{type="storage/inmemory"}`, func() float64 {
+		return float64(tm().InmemoryAssistedMerges)
 	})
-	metrics.NewGauge(`vm_assisted_merges_total{type="indexdb"}`, func() float64 {
-		return float64(idbm().AssistedMerges)
+	metrics.NewGauge(`vm_assisted_merges_total{type="indexdb/inmemory"}`, func() float64 {
+		return float64(idbm().AssistedInmemoryMerges)
 	})
 
 	metrics.NewGauge(`vm_indexdb_items_added_total`, func() float64 {
@@ -527,11 +570,8 @@ func registerStorageMetrics(strg *storage.Storage) {
 	})
 
 	// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686
-	metrics.NewGauge(`vm_merge_need_free_disk_space{type="storage/small"}`, func() float64 {
-		return float64(tm().SmallMergeNeedFreeDiskSpace)
-	})
-	metrics.NewGauge(`vm_merge_need_free_disk_space{type="storage/big"}`, func() float64 {
-		return float64(tm().BigMergeNeedFreeDiskSpace)
+	metrics.NewGauge(`vm_merge_need_free_disk_space`, func() float64 {
+		return float64(tm().MergeNeedFreeDiskSpace)
 	})
 
 	metrics.NewGauge(`vm_pending_rows{type="storage"}`, func() float64 {
@@ -541,34 +581,52 @@ func registerStorageMetrics(strg *storage.Storage) {
 		return float64(idbm().PendingItems)
 	})
 
-	metrics.NewGauge(`vm_parts{type="storage/big"}`, func() float64 {
-		return float64(tm().BigPartsCount)
+	metrics.NewGauge(`vm_parts{type="storage/inmemory"}`, func() float64 {
+		return float64(tm().InmemoryPartsCount)
 	})
 	metrics.NewGauge(`vm_parts{type="storage/small"}`, func() float64 {
 		return float64(tm().SmallPartsCount)
 	})
-	metrics.NewGauge(`vm_parts{type="indexdb"}`, func() float64 {
-		return float64(idbm().PartsCount)
+	metrics.NewGauge(`vm_parts{type="storage/big"}`, func() float64 {
+		return float64(tm().BigPartsCount)
+	})
+	metrics.NewGauge(`vm_parts{type="indexdb/inmemory"}`, func() float64 {
+		return float64(idbm().InmemoryPartsCount)
+	})
+	metrics.NewGauge(`vm_parts{type="indexdb/file"}`, func() float64 {
+		return float64(idbm().FilePartsCount)
 	})
 
-	metrics.NewGauge(`vm_blocks{type="storage/big"}`, func() float64 {
-		return float64(tm().BigBlocksCount)
+	metrics.NewGauge(`vm_blocks{type="storage/inmemory"}`, func() float64 {
+		return float64(tm().InmemoryBlocksCount)
 	})
 	metrics.NewGauge(`vm_blocks{type="storage/small"}`, func() float64 {
 		return float64(tm().SmallBlocksCount)
 	})
-	metrics.NewGauge(`vm_blocks{type="indexdb"}`, func() float64 {
-		return float64(idbm().BlocksCount)
+	metrics.NewGauge(`vm_blocks{type="storage/big"}`, func() float64 {
+		return float64(tm().BigBlocksCount)
+	})
+	metrics.NewGauge(`vm_blocks{type="indexdb/inmemory"}`, func() float64 {
+		return float64(idbm().InmemoryBlocksCount)
+	})
+	metrics.NewGauge(`vm_blocks{type="indexdb/file"}`, func() float64 {
+		return float64(idbm().FileBlocksCount)
 	})
 
-	metrics.NewGauge(`vm_data_size_bytes{type="storage/big"}`, func() float64 {
-		return float64(tm().BigSizeBytes)
+	metrics.NewGauge(`vm_data_size_bytes{type="storage/inmemory"}`, func() float64 {
+		return float64(tm().InmemorySizeBytes)
 	})
 	metrics.NewGauge(`vm_data_size_bytes{type="storage/small"}`, func() float64 {
 		return float64(tm().SmallSizeBytes)
 	})
-	metrics.NewGauge(`vm_data_size_bytes{type="indexdb"}`, func() float64 {
-		return float64(idbm().SizeBytes)
+	metrics.NewGauge(`vm_data_size_bytes{type="storage/big"}`, func() float64 {
+		return float64(tm().BigSizeBytes)
+	})
+	metrics.NewGauge(`vm_data_size_bytes{type="indexdb/inmemory"}`, func() float64 {
+		return float64(idbm().InmemorySizeBytes)
+	})
+	metrics.NewGauge(`vm_data_size_bytes{type="indexdb/file"}`, func() float64 {
+		return float64(idbm().FileSizeBytes)
 	})
 
 	metrics.NewGauge(`vm_rows_added_to_storage_total`, func() float64 {
@@ -601,19 +659,6 @@ func registerStorageMetrics(strg *storage.Storage) {
 		return float64(m().AddRowsConcurrencyCurrent)
 	})
 
-	metrics.NewGauge(`vm_concurrent_search_tsids_limit_reached_total`, func() float64 {
-		return float64(m().SearchTSIDsConcurrencyLimitReached)
-	})
-	metrics.NewGauge(`vm_concurrent_search_tsids_limit_timeout_total`, func() float64 {
-		return float64(m().SearchTSIDsConcurrencyLimitTimeout)
-	})
-	metrics.NewGauge(`vm_concurrent_search_tsids_capacity`, func() float64 {
-		return float64(m().SearchTSIDsConcurrencyCapacity)
-	})
-	metrics.NewGauge(`vm_concurrent_search_tsids_current`, func() float64 {
-		return float64(m().SearchTSIDsConcurrencyCurrent)
-	})
-
 	metrics.NewGauge(`vm_search_delays_total`, func() float64 {
 		return float64(m().SearchDelays)
 	})
@@ -659,14 +704,20 @@ func registerStorageMetrics(strg *storage.Storage) {
 		return float64(m().TimestampsBytesSaved)
 	})
 
-	metrics.NewGauge(`vm_rows{type="storage/big"}`, func() float64 {
-		return float64(tm().BigRowsCount)
+	metrics.NewGauge(`vm_rows{type="storage/inmemory"}`, func() float64 {
+		return float64(tm().InmemoryRowsCount)
 	})
 	metrics.NewGauge(`vm_rows{type="storage/small"}`, func() float64 {
 		return float64(tm().SmallRowsCount)
 	})
-	metrics.NewGauge(`vm_rows{type="indexdb"}`, func() float64 {
-		return float64(idbm().ItemsCount)
+	metrics.NewGauge(`vm_rows{type="storage/big"}`, func() float64 {
+		return float64(tm().BigRowsCount)
+	})
+	metrics.NewGauge(`vm_rows{type="indexdb/inmemory"}`, func() float64 {
+		return float64(idbm().InmemoryItemsCount)
+	})
+	metrics.NewGauge(`vm_rows{type="indexdb/file"}`, func() float64 {
+		return float64(idbm().FileItemsCount)
 	})
 
 	metrics.NewGauge(`vm_date_range_search_calls_total`, func() float64 {
@@ -717,8 +768,8 @@ func registerStorageMetrics(strg *storage.Storage) {
 	metrics.NewGauge(`vm_cache_entries{type="indexdb/indexBlocks"}`, func() float64 {
 		return float64(idbm().IndexBlocksCacheSize)
 	})
-	metrics.NewGauge(`vm_cache_entries{type="indexdb/tagFilters"}`, func() float64 {
-		return float64(idbm().TagFiltersCacheSize)
+	metrics.NewGauge(`vm_cache_entries{type="indexdb/tagFiltersToMetricIDs"}`, func() float64 {
+		return float64(idbm().TagFiltersToMetricIDsCacheSize)
 	})
 	metrics.NewGauge(`vm_cache_entries{type="storage/regexps"}`, func() float64 {
 		return float64(storage.RegexpCacheSize())
@@ -758,8 +809,8 @@ func registerStorageMetrics(strg *storage.Storage) {
 	metrics.NewGauge(`vm_cache_size_bytes{type="storage/next_day_metric_ids"}`, func() float64 {
 		return float64(m().NextDayMetricIDCacheSizeBytes)
 	})
-	metrics.NewGauge(`vm_cache_size_bytes{type="indexdb/tagFilters"}`, func() float64 {
-		return float64(idbm().TagFiltersCacheSizeBytes)
+	metrics.NewGauge(`vm_cache_size_bytes{type="indexdb/tagFiltersToMetricIDs"}`, func() float64 {
+		return float64(idbm().TagFiltersToMetricIDsCacheSizeBytes)
 	})
 	metrics.NewGauge(`vm_cache_size_bytes{type="storage/regexps"}`, func() float64 {
 		return float64(storage.RegexpCacheSizeBytes())
@@ -789,8 +840,8 @@ func registerStorageMetrics(strg *storage.Storage) {
 	metrics.NewGauge(`vm_cache_size_max_bytes{type="indexdb/indexBlocks"}`, func() float64 {
 		return float64(idbm().IndexBlocksCacheSizeMaxBytes)
 	})
-	metrics.NewGauge(`vm_cache_size_max_bytes{type="indexdb/tagFilters"}`, func() float64 {
-		return float64(idbm().TagFiltersCacheSizeMaxBytes)
+	metrics.NewGauge(`vm_cache_size_max_bytes{type="indexdb/tagFiltersToMetricIDs"}`, func() float64 {
+		return float64(idbm().TagFiltersToMetricIDsCacheSizeMaxBytes)
 	})
 	metrics.NewGauge(`vm_cache_size_max_bytes{type="storage/regexps"}`, func() float64 {
 		return float64(storage.RegexpCacheMaxSizeBytes())
@@ -817,8 +868,8 @@ func registerStorageMetrics(strg *storage.Storage) {
 	metrics.NewGauge(`vm_cache_requests_total{type="indexdb/indexBlocks"}`, func() float64 {
 		return float64(idbm().IndexBlocksCacheRequests)
 	})
-	metrics.NewGauge(`vm_cache_requests_total{type="indexdb/tagFilters"}`, func() float64 {
-		return float64(idbm().TagFiltersCacheRequests)
+	metrics.NewGauge(`vm_cache_requests_total{type="indexdb/tagFiltersToMetricIDs"}`, func() float64 {
+		return float64(idbm().TagFiltersToMetricIDsCacheRequests)
 	})
 	metrics.NewGauge(`vm_cache_requests_total{type="storage/regexps"}`, func() float64 {
 		return float64(storage.RegexpCacheRequests())
@@ -845,8 +896,8 @@ func registerStorageMetrics(strg *storage.Storage) {
 	metrics.NewGauge(`vm_cache_misses_total{type="indexdb/indexBlocks"}`, func() float64 {
 		return float64(idbm().IndexBlocksCacheMisses)
 	})
-	metrics.NewGauge(`vm_cache_misses_total{type="indexdb/tagFilters"}`, func() float64 {
-		return float64(idbm().TagFiltersCacheMisses)
+	metrics.NewGauge(`vm_cache_misses_total{type="indexdb/tagFiltersToMetricIDs"}`, func() float64 {
+		return float64(idbm().TagFiltersToMetricIDsCacheMisses)
 	})
 	metrics.NewGauge(`vm_cache_misses_total{type="storage/regexps"}`, func() float64 {
 		return float64(storage.RegexpCacheMisses())

app/vmui/Dockerfile-web

@@ -1,4 +1,4 @@
-FROM golang:1.19.1 as build-web-stage
+FROM golang:1.19.4 as build-web-stage
 COPY build /build
 
 WORKDIR /build
@@ -6,7 +6,7 @@ COPY web/ /build/
 RUN GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o web-amd64 github.com/VictoriMetrics/vmui/ && \
     GOOS=windows GOARCH=amd64 CGO_ENABLED=0 go build -o web-windows github.com/VictoriMetrics/vmui/
 
-FROM alpine:3.16.2
+FROM alpine:3.17.0
 USER root
 
 COPY --from=build-web-stage /build/web-amd64 /app/web

app/vmui/packages/vmui/.eslintrc.js

@@ -1,3 +1,4 @@
+// eslint-disable-next-line no-undef
 module.exports = {
   "env": {
     "browser": true,
@@ -10,9 +11,7 @@ module.exports = {
   ],
   "parser": "@typescript-eslint/parser",
   "parserOptions": {
-    "ecmaFeatures": {
-      "jsx": true
-    },
+    "ecmaFeatures": { "jsx": true },
     "ecmaVersion": 12,
     "sourceType": "module"
   },
@@ -21,32 +20,15 @@ module.exports = {
     "@typescript-eslint"
   ],
   "rules": {
-    "indent": [
-      "error",
-      2,
-      { "SwitchCase": 1 }
-    ],
-    "linebreak-style": [
-      "error",
-      "unix"
-    ],
-    "quotes": [
-      "error",
-      "double"
-    ],
-    "semi": [
-      "error",
-      "always"
-    ],
-    "react/prop-types": 0,
-    "max-lines": [
-      "error",
-      {
-        "max": 1000,
-        "skipBlankLines": true,
-        "skipComments": true,
-      }
-    ]
+    "react/jsx-closing-bracket-location": [1, "line-aligned"],
+    "react/jsx-max-props-per-line":[1, { "maximum": 1 }],
+    "react/jsx-first-prop-new-line": [1, "multiline"],
+    "object-curly-spacing": [2, "always"],
+    "indent": ["error", 2, { "SwitchCase": 1 }],
+    "linebreak-style": ["error", "unix"],
+    "quotes": ["error", "double"],
+    "semi": ["error", "always"],
+    "react/prop-types": 0
   },
   "settings": {
     "react": {
@@ -56,7 +38,10 @@ module.exports = {
     "linkComponents": [
       // Components used as alternatives to <a> for linking, eg. <Link to={ url } />
       "Hyperlink",
-      {"name": "Link", "linkAttribute": "to"}
+      {
+        "name": "Link", "linkAttribute": "to"
+      }
     ]
   }
 };
+

app/vmui/packages/vmui/README.md

@@ -44,3 +44,59 @@ You don’t have to ever use `eject`. The curated feature set is suitable for sm
 You can learn more in the [Create React App documentation](https://facebook.github.io/create-react-app/docs/getting-started).
 
 To learn React, check out the [React documentation](https://reactjs.org/).
+
+---
+
+# VMUI config options
+VMUI can be used to paste into other applications
+
+1. Go to file `index.html`
+2. Find root element `<div id="root"></div>`
+3. Add `data-params` with the options
+
+#### Options (JSON):
+
+| Name                    |   Default   |                                                                           Description |
+|:------------------------|:-----------:|--------------------------------------------------------------------------------------:|
+| serverURL               | domain name |                                                          Can't be changed from the UI |
+| inputTenantID           |      -      |                            If the flag is present, the "Tenant ID" field is displayed |
+| headerStyles.background |  `#FFFFFF`  |                                                               Header background color |
+| headerStyles.color      |  `#3F51B5`  |                                                                     Header font color |
+| palette.primary         |  `#3F51B5`  |                               used to represent primary interface elements for a user |
+| palette.secondary       |  `#F50057`  |                             used to represent secondary interface elements for a user |
+| palette.error           |  `#FF4141`  |            used to represent interface elements that the user should be made aware of |
+| palette.warning         |  `#FF9800`  |                 used to represent potentially dangerous actions or important messages |
+| palette.success         |  `#4CAF50`  |           used to indicate the successful completion of an action that user triggered |
+| palette.info            |  `#03A9F4`  | used to present information to the user that is neutral and not necessarily important |
+
+#### JSON example:
+```json
+{
+  "serverURL": "http://localhost:8428",
+  "inputTenantID": "true",
+  "headerStyles": {
+    "background": "#FFFFFF",
+    "color": "#538DE8"
+  },
+  "palette": {
+    "primary": "#538DE8",
+    "secondary": "#F76F8E",
+    "error": "#FD151B",
+    "warning": "#FFB30F",
+    "success": "#7BE622",
+    "info": "#0F5BFF"
+  }
+}
+```
+
+
+#### HTML example:
+```html
+<div id="root" data-params='{"serverURL":"http://localhost:8428","inputTenantID":"true","headerStyles":{"background":"#FFFFFF","color":"#538DE8"},"palette":{"primary":"#538DE8","secondary":"#F76F8E","error":"#FD151B","warning":"#FFB30F","success":"#7BE622","info":"#0F5BFF"}}'></div>
+```
+
+
+
+
+
+

app/vmui/packages/vmui/TESTCASES.md

@@ -0,0 +1,98 @@
+# Test cases
+
+----
+
+**Name:** Force execution of a queries
+
+**Steps:**
+1. click to button `Execute query`
+2. click to icon `Refresh dashboard`
+3. press `enter` on the query field
+
+**Expected Result:**  
+For each step sends a request and render new data
+
+----
+
+**Name:** Time Range with auto refresh
+
+**Steps:**
+1. Set absolute time range
+2. Enable auto refresh
+3. Change delay auto refresh
+4. Disable auto refresh
+
+**Expected Result:**  
+Time range has not changed
+
+----
+
+**Name:** Query history
+
+**Steps:**
+1. Run query one by one: `1`, `2`, `3`
+2. Press `Ctrl + ArrowUp`/`Ctrl + ArrowDown` when the query field focus
+
+**Expected Result:**  
+Query value changes according to execution order (Preserve execution order). 
+<br/>
+`Ctrl + ArrowUp` - set prev value, `Ctrl + ArrowDown` - set next value 
+
+----
+
+**Name:** Absolute time range fields
+
+**Steps:**
+1. Open `Time range controls`
+2. Change `From` or `Until` time value
+3. Click to `Apply`
+
+**Expected Result:**  
+When you change one of the fields, the second does not change
+
+----
+
+**Name:** Auto update after query delete
+
+**Steps:**
+1. Add multiple query
+2. Execute queries
+3. Delete one of the queries
+
+**Expected Result:**  
+Graph is automatically updated after the query delete
+
+----
+
+**Name:** Query URL params
+
+**Steps:**
+1. [Open graph](http://localhost:3000/?g0.range_input=1d&g0.end_input=2022-10-26T14%3A00%3A00&g0.step_input=180&g0.relative_time=none&g0.tab=chart&g0.expr=1&g1.range_input=1d&g1.end_input=2022-10-26T14%3A00%3A00&g1.step_input=180&g1.relative_time=none&g1.tab=chart&g1.expr=2#/) with params:
+> ?g0.range_input=1d&g0.end_input=2022-10-26T14%3A00%3A00&g0.step_input=180&g0.relative_time=none&g0.tab=chart&g0.expr=1&g1.range_input=1d&g1.end_input=2022-10-26T14%3A00%3A00&g1.step_input=180&g1.relative_time=none&g1.tab=chart&g1.expr=2#/
+
+**Expected Result:**  
+Executed two query with params: 
+```
+query: 1 and 2
+start: 1666706400
+end: 1666792800
+step: from "Step value" field (depends on screen width)
+```
+- Display two queries: `1` and `2`
+- Time range from `2022-10-25 16:00:00` to `2022-10-26 16:00:00` (:warning: by UTC +2)
+- Display tab `Table`
+
+----
+
+**Name:** Prometheus query URL params
+
+**Steps:**
+1. [Open graph](http://localhost:3000/?g0.expr=node_arp_entries&g0.tab=1&g0.stacked=0&g0.range_input=30m&g0.end_input=2021-09-11%2000%3A00%3A00&g0.moment_input=2021-09-11%2000%3A00%3A00&g0.step_input=6&g1.expr=node_cpu_guest_seconds_total&g1.tab=1&g1.stacked=0&g1.range_input=30m&g1.end_input=2022-12-01%2014%3A00%3A00&g1.moment_input=2022-12-01%2014%3A00%3A00&g1.step_input=6) with params:
+> ?g0.expr=node_arp_entries&g0.tab=1&g0.stacked=0&g0.range_input=30m&g0.end_input=2021-09-11%2000%3A00%3A00&g0.moment_input=2021-09-11%2000%3A00%3A00&g0.step_input=6&g1.expr=node_cpu_guest_seconds_total&g1.tab=1&g1.stacked=0&g1.range_input=30m&g1.end_input=2022-12-01%2014%3A00%3A00&g1.moment_input=2022-12-01%2014%3A00%3A00&g1.step_input=6
+
+**Expected Result:**
+- Display two queries: `node_arp_entries` and `node_cpu_guest_seconds_total`
+- Time range from `2021-09-11 01:30:00` to `2021-09-11 02:00:00` (:warning: by UTC +2)
+- Display tab `Table`
+
+----

app/vmui/packages/vmui/package.json

@@ -4,11 +4,6 @@
   "private": true,
   "homepage": "./",
   "dependencies": {
-    "@date-io/dayjs": "^2.13.1",
-    "@emotion/styled": "^11.8.1",
-    "@mui/icons-material": "^5.6.0",
-    "@mui/lab": "^5.0.0-alpha.73",
-    "@mui/material": "^5.5.1",
     "@testing-library/jest-dom": "^5.16.2",
     "@testing-library/react": "^13.0.0",
     "@testing-library/user-event": "^14.0.4",
@@ -21,6 +16,7 @@
     "@types/qs": "^6.9.7",
     "@types/react-router-dom": "^5.3.3",
     "@types/webpack-env": "^1.16.3",
+    "classnames": "^2.3.2",
     "dayjs": "^1.11.0",
     "lodash.debounce": "^4.0.8",
     "lodash.get": "^4.4.2",
@@ -29,6 +25,7 @@
     "preact": "^10.7.1",
     "qs": "^6.10.3",
     "react-router-dom": "^6.3.0",
+    "sass": "^1.56.0",
     "typescript": "~4.6.2",
     "uplot": "^1.6.19",
     "web-vitals": "^2.1.4"

app/vmui/packages/vmui/public/index.html

@@ -26,7 +26,9 @@
       Learn how to configure a non-root public URL by running `npm run build`.
     -->
     <title>VM UI</title>
-    <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,500,700&display=swap" />
+    <link rel="preconnect" href="https://fonts.googleapis.com">
+    <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
+    <link href="https://fonts.googleapis.com/css2?family=JetBrains+Mono&family=Lato:wght@300;400;700&display=swap" rel="stylesheet">
     <script src="%PUBLIC_URL%/dashboards/index.js" type="module"></script>
   </head>
   <body>

app/vmui/packages/vmui/src/App.test.tsx

@@ -1,5 +1,5 @@
 import React from "preact/compat";
-import {render, screen} from "@testing-library/react";
+import { render, screen } from "@testing-library/react";
 import App from "./App";
 
 test("renders header", () => {

app/vmui/packages/vmui/src/App.tsx

@@ -1,56 +1,58 @@
-import React, {FC} from "preact/compat";
-import {HashRouter, Route, Routes} from "react-router-dom";
-import {SnackbarProvider} from "./contexts/Snackbar";
-import {StateProvider} from "./state/common/StateContext";
-import {AuthStateProvider} from "./state/auth/AuthStateContext";
-import {GraphStateProvider} from "./state/graph/GraphStateContext";
-import {CardinalityStateProvider} from "./state/cardinality/CardinalityStateContext";
-import {TopQueriesStateProvider} from "./state/topQueries/TopQueriesStateContext";
-import THEME from "./theme/theme";
-import { ThemeProvider, StyledEngineProvider } from "@mui/material/styles";
-import CssBaseline from "@mui/material/CssBaseline";
-import LocalizationProvider from "@mui/lab/LocalizationProvider";
-import DayjsUtils from "@date-io/dayjs";
-import router from "./router/index";
-
-import CustomPanel from "./components/CustomPanel/CustomPanel";
+import React, { FC, useState } from "preact/compat";
+import { HashRouter, Route, Routes } from "react-router-dom";
+import router from "./router";
+import AppContextProvider from "./contexts/AppContextProvider";
 import HomeLayout from "./components/Home/HomeLayout";
-import DashboardsLayout from "./components/PredefinedPanels/DashboardsLayout";
-import CardinalityPanel from "./components/CardinalityPanel/CardinalityPanel";
-import TopQueries from "./components/TopQueries/TopQueries";
-
+import CustomPanel from "./pages/CustomPanel";
+import DashboardsLayout from "./pages/PredefinedPanels";
+import CardinalityPanel from "./pages/CardinalityPanel";
+import TopQueries from "./pages/TopQueries";
+import ThemeProvider from "./components/Main/ThemeProvider/ThemeProvider";
+import Spinner from "./components/Main/Spinner/Spinner";
+import TracePage from "./pages/TracePage";
 
 const App: FC = () => {
 
+  const [loadingTheme, setLoadingTheme] = useState(true);
+
+  if (loadingTheme) return (
+    <>
+      <Spinner/>
+      <ThemeProvider setLoadingTheme={setLoadingTheme}/>;
+    </>
+  );
+
   return <>
     <HashRouter>
-      <CssBaseline /> {/* CSS Baseline: kind of normalize.css made by materialUI team - can be scoped */}
-      <LocalizationProvider dateAdapter={DayjsUtils}> {/* Allows datepicker to work with DayJS */}
-        <StyledEngineProvider injectFirst>
-          <ThemeProvider theme={THEME}>  {/* Material UI theme customization */}
-            <StateProvider> {/* Serialized into query string, common app settings */}
-              <AuthStateProvider> {/* Auth related info - optionally persisted to Local Storage */}
-                <GraphStateProvider> {/* Graph settings */}
-                  <CardinalityStateProvider> {/* Cardinality settings */}
-                    <TopQueriesStateProvider> {/* Top Queries settings */}
-                      <SnackbarProvider> {/* Display various snackbars */}
-                        <Routes>
-                          <Route path={"/"} element={<HomeLayout/>}>
-                            <Route path={router.home} element={<CustomPanel/>}/>
-                            <Route path={router.dashboards} element={<DashboardsLayout/>}/>
-                            <Route path={router.cardinality} element={<CardinalityPanel/>} />
-                            <Route path={router.topQueries} element={<TopQueries/>} />
-                          </Route>
-                        </Routes>
-                      </SnackbarProvider>
-                    </TopQueriesStateProvider>
-                  </CardinalityStateProvider>
-                </GraphStateProvider>
-              </AuthStateProvider>
-            </StateProvider>
-          </ThemeProvider>
-        </StyledEngineProvider>
-      </LocalizationProvider>
+      <AppContextProvider>
+        <Routes>
+          <Route
+            path={"/"}
+            element={<HomeLayout/>}
+          >
+            <Route
+              path={router.home}
+              element={<CustomPanel/>}
+            />
+            <Route
+              path={router.dashboards}
+              element={<DashboardsLayout/>}
+            />
+            <Route
+              path={router.cardinality}
+              element={<CardinalityPanel/>}
+            />
+            <Route
+              path={router.topQueries}
+              element={<TopQueries/>}
+            />
+            <Route
+              path={router.trace}
+              element={<TracePage/>}
+            />
+          </Route>
+        </Routes>
+      </AppContextProvider>
     </HashRouter>
   </>;
 };

app/vmui/packages/vmui/src/api/query-range.ts

@@ -1,4 +1,4 @@
-import {TimeParams} from "../types";
+import { TimeParams } from "../types";
 
 export const getQueryRangeUrl = (server: string, query: string, period: TimeParams, nocache: boolean, queryTracing: boolean): string =>
   `${server}/api/v1/query_range?query=${encodeURIComponent(query)}&start=${period.start}&end=${period.end}&step=${period.step}${nocache ? "&nocache=1" : ""}${queryTracing ? "&trace=1" : ""}`;

app/vmui/packages/vmui/src/components/BorderLineProgress/BorderLinearProgress.tsx

@@ -1,27 +0,0 @@
-import React from "preact/compat";
-import { styled } from "@mui/material/styles";
-import LinearProgressWithLabel, {linearProgressClasses, LinearProgressProps} from "@mui/material/LinearProgress";
-import {Box, Typography} from "@mui/material";
-
-export const BorderLinearProgress = styled(LinearProgressWithLabel)(({ theme }) => ({
-  height: 20,
-  borderRadius: 5,
-  [`&.${linearProgressClasses.colorPrimary}`]: {
-    backgroundColor: theme.palette.grey[theme.palette.mode === "light" ? 200 : 800],
-  },
-  [`& .${linearProgressClasses.bar}`]: {
-    borderRadius: 5,
-    backgroundColor: theme.palette.mode === "light" ? "#1a90ff" : "#308fe8",
-  },
-}));
-
-export const BorderLinearProgressWithLabel = (props: LinearProgressProps & { value: number }) => (
-  <Box sx={{ display: "flex", alignItems: "center" }}>
-    <Box sx={{ width: "100%", mr: 1 }}>
-      <BorderLinearProgress variant="determinate" {...props} />
-    </Box>
-    <Box sx={{ minWidth: 35 }}>
-      <Typography variant="body2" color="text.secondary">{`${props.value.toFixed(2)}%`}</Typography>
-    </Box>
-  </Box>
-);

app/vmui/packages/vmui/src/components/CardinalityPanel/CardinalityConfigurator/CardinalityConfigurator.tsx

@@ -1,103 +0,0 @@
-import React, {ChangeEvent, FC} from "react";
-import Box from "@mui/material/Box";
-import QueryEditor from "../../CustomPanel/Configurator/Query/QueryEditor";
-import Tooltip from "@mui/material/Tooltip";
-import IconButton from "@mui/material/IconButton";
-import PlayCircleOutlineIcon from "@mui/icons-material/PlayCircleOutline";
-import {useFetchQueryOptions} from "../../../hooks/useFetchQueryOptions";
-import {useAppDispatch, useAppState} from "../../../state/common/StateContext";
-import FormControlLabel from "@mui/material/FormControlLabel";
-import BasicSwitch from "../../../theme/switch";
-import {saveToStorage} from "../../../utils/storage";
-import TextField from "@mui/material/TextField";
-import {ErrorTypes} from "../../../types";
-
-export interface CardinalityConfiguratorProps {
-  onSetHistory: (step: number, index: number) => void;
-  onSetQuery: (query: string, index: number) => void;
-  onRunQuery: () => void;
-  onTopNChange: (e: ChangeEvent<HTMLTextAreaElement|HTMLInputElement>) => void;
-  onFocusLabelChange: (e: ChangeEvent<HTMLTextAreaElement|HTMLInputElement>) => void;
-  query: string;
-  topN: number;
-  error?: ErrorTypes | string;
-  totalSeries: number;
-  totalLabelValuePairs: number;
-  date: string | null;
-  match: string | null;
-  focusLabel: string | null;
-}
-
-const CardinalityConfigurator: FC<CardinalityConfiguratorProps> = ({
-  topN,
-  error,
-  query,
-  onSetHistory,
-  onRunQuery,
-  onSetQuery,
-  onTopNChange,
-  onFocusLabelChange,
-  totalSeries,
-  totalLabelValuePairs,
-  date,
-  match,
-  focusLabel
-}) => {
-  const dispatch = useAppDispatch();
-  const {queryControls: {autocomplete}} = useAppState();
-  const {queryOptions} = useFetchQueryOptions();
-
-  const onChangeAutocomplete = () => {
-    dispatch({type: "TOGGLE_AUTOCOMPLETE"});
-    saveToStorage("AUTOCOMPLETE", !autocomplete);
-  };
-
-  return <Box boxShadow="rgba(99, 99, 99, 0.2) 0px 2px 8px 0px;" p={4} pb={2} mb={2}>
-    <Box>
-      <Box display="grid" gridTemplateColumns="1fr auto auto auto auto" gap="4px" width="100%" mb={4}>
-        <QueryEditor
-          query={query} index={0} autocomplete={autocomplete} queryOptions={queryOptions}
-          error={error} setHistoryIndex={onSetHistory} runQuery={onRunQuery} setQuery={onSetQuery}
-          label={"Time series selector"}
-        />
-        <Box mr={2}>
-          <TextField
-            label="Number of entries per table"
-            type="number"
-            size="medium"
-            variant="outlined"
-            value={topN}
-            error={topN < 1}
-            helperText={topN < 1 ? "Number must be bigger than zero" : " "}
-            onChange={onTopNChange}/>
-        </Box>
-        <Box mr={2}>
-          <TextField
-            label="Focus label"
-            type="text"
-            size="medium"
-            variant="outlined"
-            value={focusLabel}
-            onChange={onFocusLabelChange} />
-        </Box>
-        <Box>
-          <FormControlLabel label="Autocomplete"
-            control={<BasicSwitch checked={autocomplete} onChange={onChangeAutocomplete}/>}
-          />
-        </Box>
-        <Tooltip title="Execute Query">
-          <IconButton onClick={onRunQuery} sx={{height: "49px", width: "49px"}}>
-            <PlayCircleOutlineIcon/>
-          </IconButton>
-        </Tooltip>
-      </Box>
-    </Box>
-    <Box>
-      Analyzed <b>{totalSeries}</b> series with <b>{totalLabelValuePairs}</b> &quot;label=value&quot; pairs
-      at <b>{date}</b> {match && <span>for series selector <b>{match}</b></span>}.
-      Show top {topN} entries per table.
-    </Box>
-  </Box>;
-};
-
-export default CardinalityConfigurator;

app/vmui/packages/vmui/src/components/CardinalityPanel/CardinalityPanel.tsx

@@ -1,119 +0,0 @@
-import React, {ChangeEvent, FC, useState} from "react";
-import {SyntheticEvent} from "react";
-import {Alert} from "@mui/material";
-import {useFetchQuery} from "../../hooks/useCardinalityFetch";
-import {queryUpdater} from "./helpers";
-import {Data} from "../Table/types";
-import CardinalityConfigurator from "./CardinalityConfigurator/CardinalityConfigurator";
-import Spinner from "../common/Spinner";
-import {useCardinalityDispatch, useCardinalityState} from "../../state/cardinality/CardinalityStateContext";
-import MetricsContent from "./MetricsContent/MetricsContent";
-import {DefaultActiveTab, Tabs, TSDBStatus, Containers} from "./types";
-
-const spinnerContainerStyles = (height: string) =>  {
-  return {
-    width: "100%",
-    maxWidth: "100%",
-    position: "absolute",
-    height: height ?? "50%",
-    background: "rgba(255, 255, 255, 0.7)",
-    pointerEvents: "none",
-    zIndex: 1000,
-  };
-};
-
-const CardinalityPanel: FC = () => {
-  const cardinalityDispatch = useCardinalityDispatch();
-
-  const {topN, match, date, focusLabel} = useCardinalityState();
-  const configError = "";
-  const [query, setQuery] = useState(match || "");
-  const [queryHistoryIndex, setQueryHistoryIndex] = useState(0);
-  const [queryHistory, setQueryHistory] = useState<string[]>([]);
-
-  const onRunQuery = () => {
-    setQueryHistory(prev => [...prev, query]);
-    setQueryHistoryIndex(prev => prev + 1);
-    cardinalityDispatch({type: "SET_MATCH", payload: query});
-    cardinalityDispatch({type: "RUN_QUERY"});
-  };
-
-  const onSetQuery = (query: string) => {
-    setQuery(query);
-  };
-
-  const onSetHistory = (step: number) => {
-    const newIndexHistory = queryHistoryIndex + step;
-    if (newIndexHistory < 0 || newIndexHistory >= queryHistory.length) return;
-    setQueryHistoryIndex(newIndexHistory);
-    setQuery(queryHistory[newIndexHistory]);
-  };
-
-  const onTopNChange = (e: ChangeEvent<HTMLTextAreaElement|HTMLInputElement>) => {
-    cardinalityDispatch({type: "SET_TOP_N", payload: +e.target.value});
-  };
-
-  const onFocusLabelChange = (e: ChangeEvent<HTMLTextAreaElement|HTMLInputElement>) => {
-    cardinalityDispatch({type: "SET_FOCUS_LABEL", payload: e.target.value});
-  };
-
-  const {isLoading, appConfigurator, error} = useFetchQuery();
-  const [stateTabs, setTab] = useState(appConfigurator.defaultState.defaultActiveTab);
-  const {tsdbStatusData, defaultState, tablesHeaders} = appConfigurator;
-  const handleTabChange = (e: SyntheticEvent, newValue: number) => {
-    // eslint-disable-next-line @typescript-eslint/ban-ts-comment
-    // @ts-ignore
-    setTab({...stateTabs, [e.target.id]: newValue});
-  };
-
-  const handleFilterClick = (key: string) => (e: SyntheticEvent) => {
-    const name = e.currentTarget.id;
-    const query = queryUpdater[key](focusLabel, name);
-    setQuery(query);
-    setQueryHistory(prev => [...prev, query]);
-    setQueryHistoryIndex(prev => prev + 1);
-    cardinalityDispatch({type: "SET_MATCH", payload: query});
-    let newFocusLabel = "";
-    if (key === "labelValueCountByLabelName" || key == "seriesCountByLabelName") {
-      newFocusLabel = name;
-    }
-    cardinalityDispatch({type: "SET_FOCUS_LABEL", payload: newFocusLabel});
-    cardinalityDispatch({type: "RUN_QUERY"});
-  };
-
-  return (
-    <>
-      {isLoading && <Spinner
-        isLoading={isLoading}
-        height={"800px"}
-        containerStyles={spinnerContainerStyles("100%")}
-        title={<Alert color="error" severity="error" sx={{whiteSpace: "pre-wrap", mt: 2}}>
-          Please wait while cardinality stats is calculated. This may take some time if the db contains big number of time series
-        </Alert>}
-      />}
-      <CardinalityConfigurator error={configError} query={query} onRunQuery={onRunQuery} onSetQuery={onSetQuery}
-        onSetHistory={onSetHistory} onTopNChange={onTopNChange} topN={topN} date={date} match={match}
-        totalSeries={tsdbStatusData.totalSeries} totalLabelValuePairs={tsdbStatusData.totalLabelValuePairs}
-        focusLabel={focusLabel} onFocusLabelChange={onFocusLabelChange}
-      />
-      {error && <Alert color="error" severity="error" sx={{whiteSpace: "pre-wrap", mt: 2}}>{error}</Alert>}
-      {appConfigurator.keys(focusLabel).map((keyName) => (
-        <MetricsContent
-          key={keyName}
-          sectionTitle={appConfigurator.sectionsTitles(focusLabel)[keyName]}
-          activeTab={stateTabs[keyName as keyof DefaultActiveTab]}
-          rows={tsdbStatusData[keyName as keyof TSDBStatus] as unknown as Data[]}
-          onChange={handleTabChange}
-          onActionClick={handleFilterClick(keyName)}
-          tabs={defaultState.tabs[keyName as keyof Tabs]}
-          chartContainer={defaultState.containerRefs[keyName as keyof Containers<HTMLDivElement>]}
-          totalSeries={appConfigurator.totalSeries(keyName)}
-          tabId={keyName}
-          tableHeaderCells={tablesHeaders[keyName]}
-        />
-      ))}
-    </>
-  );
-};
-
-export default CardinalityPanel;

app/vmui/packages/vmui/src/components/CardinalityPanel/MetricsContent/MetricsContent.tsx

@@ -1,96 +0,0 @@
-import {FC} from "react";
-import {Box, Grid, Tab, Tabs, Typography} from "@mui/material";
-import TableChartIcon from "@mui/icons-material/TableChart";
-import ShowChartIcon from "@mui/icons-material/ShowChart";
-import TabPanel from "../../TabPanel/TabPanel";
-import EnhancedTable from "../../Table/Table";
-import TableCells from "../TableCells/TableCells";
-import BarChart from "../../BarChart/BarChart";
-import {barOptions} from "../../BarChart/consts";
-import React, {SyntheticEvent} from "react";
-import {Data, HeadCell} from "../../Table/types";
-import {MutableRef} from "preact/hooks";
-
-interface MetricsProperties {
-  rows: Data[];
-  activeTab: number;
-  onChange: (e: SyntheticEvent, newValue: number) => void;
-  onActionClick: (e: SyntheticEvent) => void;
-  tabs: string[];
-  chartContainer: MutableRef<HTMLDivElement> | undefined;
-  totalSeries: number,
-  tabId: string;
-  sectionTitle: string;
-  tableHeaderCells: HeadCell[];
-}
-
-const MetricsContent: FC<MetricsProperties> = ({
-  rows,
-  activeTab,
-  onChange,
-  tabs,
-  chartContainer,
-  totalSeries,
-  tabId,
-  onActionClick,
-  sectionTitle,
-  tableHeaderCells,
-}) => {
-  const tableCells = (row: Data) => (
-    <TableCells
-      row={row}
-      totalSeries={totalSeries}
-      onActionClick={onActionClick}
-    />
-  );
-  return (
-    <>
-      <Grid container spacing={2} sx={{px: 2}}>
-        <Grid item xs={12} md={12} lg={12}>
-          <Typography gutterBottom variant="h5" component="h5">{sectionTitle}</Typography>
-          <Box sx={{ borderBottom: 1, borderColor: "divider" }}>
-            <Tabs
-              value={activeTab}
-              onChange={onChange} aria-label="basic tabs example">
-              {tabs.map((title: string, i: number) =>
-                <Tab
-                  key={title}
-                  label={title}
-                  aria-controls={`tabpanel-${i}`}
-                  id={tabId}
-                  iconPosition={"start"}
-                  icon={ i === 0 ? <TableChartIcon /> : <ShowChartIcon /> } />
-              )}
-            </Tabs>
-          </Box>
-          {tabs.map((_,idx) =>
-            <div
-              ref={chartContainer}
-              style={{width: "100%", paddingRight: idx !== 0 ? "40px" : 0 }} key={`chart-${idx}`}>
-              <TabPanel value={activeTab} index={idx}>
-                {activeTab === 0 ? <EnhancedTable
-                  rows={rows}
-                  headerCells={tableHeaderCells}
-                  defaultSortColumn={"value"}
-                  tableCells={tableCells}
-                />: <BarChart
-                  data={[
-                    // eslint-disable-next-line @typescript-eslint/ban-ts-comment
-                    // @ts-ignore
-                    rows.map((v) => v.name),
-                    rows.map((v) => v.value),
-                    rows.map((_, i) => i % 12 == 0 ? 1 : i % 10 == 0 ? 2 : 0),
-                  ]}
-                  container={chartContainer?.current || null}
-                  configs={barOptions}
-                />}
-              </TabPanel>
-            </div>
-          )}
-        </Grid>
-      </Grid>
-    </>
-  );
-};
-
-export default MetricsContent;

app/vmui/packages/vmui/src/components/CardinalityPanel/TableCells/TableCells.tsx

@@ -1,39 +0,0 @@
-import {SyntheticEvent} from "react";
-import React, {FC} from "preact/compat";
-import {TableCell, ButtonGroup} from "@mui/material";
-import {Data} from "../../Table/types";
-import {BorderLinearProgressWithLabel} from "../../BorderLineProgress/BorderLinearProgress";
-import IconButton from "@mui/material/IconButton";
-import PlayCircleOutlineIcon from "@mui/icons-material/PlayCircleOutline";
-import Tooltip from "@mui/material/Tooltip";
-
-interface CardinalityTableCells {
-  row: Data,
-  totalSeries: number;
-  onActionClick: (e: SyntheticEvent) => void;
-}
-
-const TableCells: FC<CardinalityTableCells> = ({ row, totalSeries, onActionClick }) => {
-  const progress = totalSeries > 0 ? row.value / totalSeries * 100 : -1;
-  return <>
-    <TableCell key={row.name}>{row.name}</TableCell>
-    <TableCell key={row.value}>{row.value}</TableCell>
-    {progress > 0 ? <TableCell key={row.progressValue}>
-      <BorderLinearProgressWithLabel variant="determinate" value={progress} />
-    </TableCell> : null}
-    <TableCell key={"action"}>
-      <ButtonGroup variant="contained">
-        <Tooltip title={`Filter by ${row.name}`}>
-          <IconButton
-            id={row.name}
-            onClick={onActionClick}
-            sx={{height: "20px", width: "20px"}}>
-            <PlayCircleOutlineIcon/>
-          </IconButton>
-        </Tooltip>
-      </ButtonGroup>
-    </TableCell>
-  </>;
-};
-
-export default TableCells;