deployment/docker: upgrade base Docker image from Alpine 3.19.0 to 3.19.1

See https://www.alpinelinux.org/posts/Alpine-3.19.1-released.html

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -60,8 +60,8 @@ body:
   
         For VictoriaMetrics health-state issues please provide full-length screenshots
         of Grafana dashboards if possible:
-          * [Grafana dashboard for single-node VictoriaMetrics](https://grafana.com/grafana/dashboards/10229-victoriametrics-single-node/)
-          * [Grafana dashboard for VictoriaMetrics cluster](https://grafana.com/grafana/dashboards/11176-victoriametrics-cluster/)
+          * [Grafana dashboard for single-node VictoriaMetrics](https://grafana.com/grafana/dashboards/10229/)
+          * [Grafana dashboard for VictoriaMetrics cluster](https://grafana.com/grafana/dashboards/11176/)
         
         See how to setup monitoring here:
           * [monitoring for single-node VictoriaMetrics](https://docs.victoriametrics.com/#monitoring)

.github/workflows/codeql-analysis-js.yml

@@ -36,11 +36,11 @@ jobs:
         uses: actions/checkout@v4
 
       - name: Initialize CodeQL
-        uses: github/codeql-action/init@v2
+        uses: github/codeql-action/init@v3
         with:
           languages: ${{ matrix.language }}
 
       - name: Perform CodeQL Analysis
-        uses: github/codeql-action/analyze@v2
+        uses: github/codeql-action/analyze@v3
         with:
           category: "javascript"

.github/workflows/codeql-analysis.yml

@@ -75,7 +75,7 @@ jobs:
 
       # Initializes the CodeQL tools for scanning.
       - name: Initialize CodeQL
-        uses: github/codeql-action/init@v2
+        uses: github/codeql-action/init@v3
         with:
           languages: ${{ matrix.language }}
           # If you wish to specify custom queries, you can do so here or in a config file.
@@ -86,7 +86,7 @@ jobs:
       # Autobuild attempts to build any compiled languages  (C/C++, C#, or Java).
       # If this step fails, then you should remove it and run the build manually (see below)
       - name: Autobuild
-        uses: github/codeql-action/autobuild@v2
+        uses: github/codeql-action/autobuild@v3
 
       # ℹ️ Command-line programs to run using the OS shell.
       # 📚 https://git.io/JvXDl
@@ -100,4 +100,4 @@ jobs:
       #   make release
 
       - name: Perform CodeQL Analysis
-        uses: github/codeql-action/analyze@v2
+        uses: github/codeql-action/analyze@v3

.gitignore

@@ -22,3 +22,4 @@ Gemfile.lock
 /_site
 _site
 *.tmp
+/docs/.jekyll-metadata

Makefile

@@ -1,6 +1,6 @@
 PKG_PREFIX := github.com/VictoriaMetrics/VictoriaMetrics
 
-MAKE_CONCURRENCY ?= $(shell cat /proc/cpuinfo | grep -c processor)
+MAKE_CONCURRENCY ?= $(shell getconf _NPROCESSORS_ONLN)
 MAKE_PARALLEL := $(MAKE) -j $(MAKE_CONCURRENCY)
 DATEINFO_TAG ?= $(shell date -u +'%Y%m%d-%H%M%S')
 BUILDINFO_TAG ?= $(shell echo $$(git describe --long --all | tr '/' '-')$$( \

README.md

@@ -196,7 +196,7 @@ Snap package for VictoriaMetrics is available [here](https://snapcraft.io/victor
 
 Command-line flags for Snap package can be set with following command:
 
-```text
+```sh
 echo 'FLAGS="-selfScrapeInterval=10s -search.logSlowQueryDuration=20s"' > $SNAP_DATA/var/snap/victoriametrics/current/extra_flags
 snap restart victoriametrics
 ```
@@ -205,7 +205,7 @@ Do not change value for `-storageDataPath` flag, because snap package has limite
 
 Changing scrape configuration is possible with text editor:
 
-```text
+```sh
 vi $SNAP_DATA/var/snap/victoriametrics/current/etc/victoriametrics-scrape-config.yaml
 ```
 
@@ -258,7 +258,7 @@ and then install it as a service according to the following guide:
 
 1. Install VictoriaMetrics as a service by running the following from elevated PowerShell:
 
-    ```console
+    ```sh
     winsw install VictoriaMetrics.xml
     Get-Service VictoriaMetrics | Start-Service
     ```
@@ -270,25 +270,21 @@ See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3781)
 
 Add the following lines to Prometheus config file (it is usually located at `/etc/prometheus/prometheus.yml`) in order to send data to VictoriaMetrics:
 
-<div class="with-copy" markdown="1">
 
-```yml
+```yaml
 remote_write:
   - url: http://<victoriametrics-addr>:8428/api/v1/write
 ```
 
-</div>
 
 Substitute `<victoriametrics-addr>` with hostname or IP address of VictoriaMetrics.
 Then apply new config via the following command:
 
-<div class="with-copy" markdown="1">
 
-```console
+```sh
 kill -HUP `pidof prometheus`
 ```
 
-</div>
 
 Prometheus writes incoming data to local storage and replicates it to remote storage in parallel.
 This means that data remains available in local storage for `--storage.tsdb.retention.time` duration
@@ -297,7 +293,7 @@ even if remote storage is unavailable.
 If you plan sending data to VictoriaMetrics from multiple Prometheus instances, then add the following lines into `global` section
 of [Prometheus config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file):
 
-```yml
+```yaml
 global:
   external_labels:
     datacenter: dc-123
@@ -309,7 +305,6 @@ across Prometheus instances, so time series could be filtered and grouped by thi
 
 For highly loaded Prometheus instances (200k+ samples per second) the following tuning may be applied:
 
-<div class="with-copy" markdown="1">
 
 ```yaml
 remote_write:
@@ -320,7 +315,6 @@ remote_write:
       max_shards: 30
 ```
 
-</div>
 
 Using remote write increases memory usage for Prometheus by up to ~25%. If you are experiencing issues with
 too high memory consumption of Prometheus, then try to lower `max_samples_per_send` and `capacity` params. 
@@ -523,30 +517,24 @@ via ["submit metrics" API](https://docs.datadoghq.com/api/latest/metrics/#submit
 DataDog agent allows configuring destinations for metrics sending via ENV variable `DD_DD_URL` 
 or via [configuration file](https://docs.datadoghq.com/agent/guide/agent-configuration-files/) in section `dd_url`.
 
-<p align="center">
-  <img src="docs/Single-server-VictoriaMetrics-sending_DD_metrics_to_VM.webp" width="800">
-</p>
+<img src="docs/Single-server-VictoriaMetrics-sending_DD_metrics_to_VM.webp">
 
 To configure DataDog agent via ENV variable add the following prefix:
 
-<div class="with-copy" markdown="1">
 
-```
+```sh
 DD_DD_URL=http://victoriametrics:8428/datadog
 ```
 
-</div>
 
 _Choose correct URL for VictoriaMetrics [here](https://docs.victoriametrics.com/url-examples.html#datadog)._
 
 To configure DataDog agent via [configuration file](https://github.com/DataDog/datadog-agent/blob/878600ef7a55c5ef0efb41ed0915f020cf7e3bd0/pkg/config/config_template.yaml#L33)
 add the following line:
 
-<div class="with-copy" markdown="1">
-```
+```yaml
 dd_url: http://victoriametrics:8428/datadog
 ```
-</div>
 
 [vmagent](https://docs.victoriametrics.com/vmagent.html) also can accept Datadog metrics format. Depending on where vmagent will forward data,
 pick [single-node or cluster URL](https://docs.victoriametrics.com/url-examples.html#datadog) formats.
@@ -556,36 +544,26 @@ pick [single-node or cluster URL](https://docs.victoriametrics.com/url-examples.
 DataDog allows configuring [Dual Shipping](https://docs.datadoghq.com/agent/guide/dual-shipping/) for metrics 
 sending via ENV variable `DD_ADDITIONAL_ENDPOINTS` or via configuration file `additional_endpoints`.
  
-<p align="center">
-  <img src="docs/Single-server-VictoriaMetrics-sending_DD_metrics_to_VM_and_DD.webp" width="800">
-</p>
+<img src="docs/Single-server-VictoriaMetrics-sending_DD_metrics_to_VM_and_DD.webp">
  
 Run DataDog using the following ENV variable with VictoriaMetrics as additional metrics receiver:
 
-<div class="with-copy" markdown="1">
-
-```
+```sh
 DD_ADDITIONAL_ENDPOINTS='{\"http://victoriametrics:8428/datadog\": [\"apikey\"]}'
-
 ```
 
-</div>
-
 _Choose correct URL for VictoriaMetrics [here](https://docs.victoriametrics.com/url-examples.html#datadog)._
 
 
 To configure DataDog Dual Shipping via [configuration file](https://docs.datadoghq.com/agent/guide/agent-configuration-files)
 add the following line:
 
-<div class="with-copy" markdown="1">
-
-```
+```yaml
 additional_endpoints:
   "http://victoriametrics:8428/datadog":
   - apikey
 ```
 
-</div>
 
 ### Send via cURL
 
@@ -654,24 +632,20 @@ foo_field2{tag1="value1", tag2="value2"} 40
 Example for writing data with [InfluxDB line protocol](https://docs.influxdata.com/influxdb/v1.7/write_protocols/line_protocol_tutorial/)
 to local VictoriaMetrics using `curl`:
 
-<div class="with-copy" markdown="1">
 
-```console
+```sh
 curl -d 'measurement,tag1=value1,tag2=value2 field1=123,field2=1.23' -X POST 'http://localhost:8428/write'
 ```
 
-</div>
 
 An arbitrary number of lines delimited by '\n' (aka newline char) can be sent in a single request.
 After that the data may be read via [/api/v1/export](#how-to-export-data-in-json-line-format) endpoint:
 
-<div class="with-copy" markdown="1">
 
-```console
+```sh
 curl -G 'http://localhost:8428/api/v1/export' -d 'match={__name__=~"measurement_.*"}'
 ```
 
-</div>
 
 The `/api/v1/export` endpoint should return the following response:
 
@@ -698,13 +672,11 @@ VictoriaMetrics exposes endpoint for InfluxDB v2 HTTP API at `/influx/api/v2/wri
 
 In order to write data with InfluxDB line protocol to local VictoriaMetrics using `curl`:
 
-<div class="with-copy" markdown="1">
 
-```console
+```sh
 curl -d 'measurement,tag1=value1,tag2=value2 field1=123,field2=1.23' -X POST 'http://localhost:8428/api/v2/write'
 ```
 
-</div>
 
 The `/api/v1/export` endpoint should return the following response:
 
@@ -718,7 +690,7 @@ The `/api/v1/export` endpoint should return the following response:
 Enable Graphite receiver in VictoriaMetrics by setting `-graphiteListenAddr` command line flag. For instance,
 the following command will enable Graphite receiver in VictoriaMetrics on TCP and UDP port `2003`:
 
-```console
+```sh
 /path/to/victoria-metrics-prod -graphiteListenAddr=:2003
 ```
 
@@ -727,7 +699,7 @@ to the VictoriaMetrics host in `StatsD` configs.
 
 Example for writing data with Graphite plaintext protocol to local VictoriaMetrics using `nc`:
 
-```console
+```sh
 echo "foo.bar.baz;tag1=value1;tag2=value2 123 `date +%s`" | nc -N localhost 2003
 ```
 
@@ -735,13 +707,11 @@ VictoriaMetrics sets the current time if the timestamp is omitted.
 An arbitrary number of lines delimited by `\n` (aka newline char) can be sent in one go.
 After that the data may be read via [/api/v1/export](#how-to-export-data-in-json-line-format) endpoint:
 
-<div class="with-copy" markdown="1">
 
-```console
+```sh
 curl -G 'http://localhost:8428/api/v1/export' -d 'match=foo.bar.baz'
 ```
 
-</div>
 
 The `/api/v1/export` endpoint should return the following response:
 
@@ -778,7 +748,7 @@ The same protocol is used for [ingesting data in KairosDB](https://kairosdb.gith
 Enable OpenTSDB receiver in VictoriaMetrics by setting `-opentsdbListenAddr` command line flag. For instance,
 the following command enables OpenTSDB receiver in VictoriaMetrics on TCP and UDP port `4242`:
 
-```console
+```sh
 /path/to/victoria-metrics-prod -opentsdbListenAddr=:4242
 ```
 
@@ -786,24 +756,20 @@ Send data to the given address from OpenTSDB-compatible agents.
 
 Example for writing data with OpenTSDB protocol to local VictoriaMetrics using `nc`:
 
-<div class="with-copy" markdown="1">
 
-```console
+```sh
 echo "put foo.bar.baz `date +%s` 123 tag1=value1 tag2=value2" | nc -N localhost 4242
 ```
 
-</div>
 
 An arbitrary number of lines delimited by `\n` (aka newline char) can be sent in one go.
 After that the data may be read via [/api/v1/export](#how-to-export-data-in-json-line-format) endpoint:
 
-<div class="with-copy" markdown="1">
 
-```console
+```sh
 curl -G 'http://localhost:8428/api/v1/export' -d 'match=foo.bar.baz'
 ```
 
-</div>
 
 The `/api/v1/export` endpoint should return the following response:
 
@@ -816,7 +782,7 @@ The `/api/v1/export` endpoint should return the following response:
 Enable HTTP server for OpenTSDB `/api/put` requests by setting `-opentsdbHTTPListenAddr` command line flag. For instance,
 the following command enables OpenTSDB HTTP server on port `4242`:
 
-```console
+```sh
 /path/to/victoria-metrics-prod -opentsdbHTTPListenAddr=:4242
 ```
 
@@ -824,33 +790,26 @@ Send data to the given address from OpenTSDB-compatible agents.
 
 Example for writing a single data point:
 
-<div class="with-copy" markdown="1">
 
-```console
+```sh
 curl -H 'Content-Type: application/json' -d '{"metric":"x.y.z","value":45.34,"tags":{"t1":"v1","t2":"v2"}}' http://localhost:4242/api/put
 ```
 
-</div>
 
 Example for writing multiple data points in a single request:
 
-<div class="with-copy" markdown="1">
-
-```console
+```sh
 curl -H 'Content-Type: application/json' -d '[{"metric":"foo","value":45.34},{"metric":"bar","value":43}]' http://localhost:4242/api/put
 ```
 
-</div>
 
 After that the data may be read via [/api/v1/export](#how-to-export-data-in-json-line-format) endpoint:
 
-<div class="with-copy" markdown="1">
 
-```console
+```sh
 curl -G 'http://localhost:8428/api/v1/export' -d 'match[]=x.y.z' -d 'match[]=foo' -d 'match[]=bar'
 ```
 
-</div>
 
 The `/api/v1/export` endpoint should return the following response:
 
@@ -876,7 +835,7 @@ The `COLLECTOR_URL` must point to `/newrelic` HTTP endpoint at VictoriaMetrics,
 which can be obtained [here](https://newrelic.com/signup).
 For example, if VictoriaMetrics runs at `localhost:8428`, then the following command can be used for running NewRelic infrastructure agent:
 
-```console
+```sh
 COLLECTOR_URL="http://localhost:8428/newrelic" NRIA_LICENSE_KEY="NEWRELIC_LICENSE_KEY" ./newrelic-infra
 ```
 
@@ -917,13 +876,13 @@ For example, let's import the following NewRelic Events request to VictoriaMetri
 
 Save this JSON into `newrelic.json` file and then use the following command in order to import it into VictoriaMetrics:
 
-```console
+```sh
 curl -X POST -H 'Content-Type: application/json' --data-binary @newrelic.json http://localhost:8428/newrelic/infra/v2/metrics/events/bulk
 ```
 
 Let's fetch the ingested data via [data export API](#how-to-export-data-in-json-line-format):
 
-```console
+```sh
 curl http://localhost:8428/api/v1/export -d 'match={eventType="SystemSample"}'
 {"metric":{"__name__":"cpuStealPercent","entityKey":"macbook-pro.local","eventType":"SystemSample"},"values":[0],"timestamps":[1697407970000]}
 {"metric":{"__name__":"loadAverageFiveMinute","entityKey":"macbook-pro.local","eventType":"SystemSample"},"values":[4.099609375],"timestamps":[1697407970000]}
@@ -1131,7 +1090,7 @@ The base docker image is [alpine](https://hub.docker.com/_/alpine) but it is pos
 by setting it via `<ROOT_IMAGE>` environment variable.
 For example, the following command builds the image on top of [scratch](https://hub.docker.com/_/scratch) image:
 
-```console
+```sh
 ROOT_IMAGE=scratch make package-victoria-metrics
 ```
 
@@ -1274,7 +1233,7 @@ Optional `start` and `end` args may be added to the request in order to limit th
 See [allowed formats](#timestamp-formats) for these args.
 
 For example:
-```console
+```sh
 curl http://<victoriametrics-addr>:8428/api/v1/export -d 'match[]=<timeseries_selector_for_export>' -d 'start=1654543486' -d 'end=1654543486'
 curl http://<victoriametrics-addr>:8428/api/v1/export -d 'match[]=<timeseries_selector_for_export>' -d 'start=2022-06-06T19:25:48' -d 'end=2022-06-06T19:29:07'
 ```
@@ -1286,13 +1245,11 @@ In this case the output may contain multiple lines with samples for the same tim
 Pass `Accept-Encoding: gzip` HTTP header in the request to `/api/v1/export` in order to reduce network bandwidth during exporting big amounts
 of time series data. This enables gzip compression for the exported data. Example for exporting gzipped data:
 
-<div class="with-copy" markdown="1">
 
-```console
+```sh
 curl -H 'Accept-Encoding: gzip' http://localhost:8428/api/v1/export -d 'match[]={__name__!=""}' > data.jsonl.gz
 ```
 
-</div>
 
 The maximum duration for each request to `/api/v1/export` is limited by `-search.maxExportDuration` command-line flag.
 
@@ -1323,7 +1280,7 @@ Optional `start` and `end` args may be added to the request in order to limit th
 See [allowed formats](#timestamp-formats) for these args.
 
 For example:
-```console
+```sh
 curl http://<victoriametrics-addr>:8428/api/v1/export/csv -d 'format=<format>' -d 'match[]=<timeseries_selector_for_export>' -d 'start=1654543486' -d 'end=1654543486'
 curl http://<victoriametrics-addr>:8428/api/v1/export/csv -d 'format=<format>' -d 'match[]=<timeseries_selector_for_export>' -d 'start=2022-06-06T19:25:48' -d 'end=2022-06-06T19:29:07'
 ```
@@ -1340,7 +1297,7 @@ for metrics to export. Use `{__name__=~".*"}` selector for fetching all the time
 
 On large databases you may experience problems with limit on the number of time series, which can be exported. In this case you need to adjust `-search.maxExportSeries` command-line flag:
 
-```console
+```sh
 # count unique time series in database
 wget -O- -q 'http://your_victoriametrics_instance:8428/api/v1/series/count' | jq '.data[0]'
 
@@ -1351,7 +1308,7 @@ Optional `start` and `end` args may be added to the request in order to limit th
 See [allowed formats](#timestamp-formats) for these args.
 
 For example:
-```console
+```sh
 curl http://<victoriametrics-addr>:8428/api/v1/export/native -d 'match[]=<timeseries_selector_for_export>' -d 'start=1654543486' -d 'end=1654543486'
 curl http://<victoriametrics-addr>:8428/api/v1/export/native -d 'match[]=<timeseries_selector_for_export>' -d 'start=2022-06-06T19:25:48' -d 'end=2022-06-06T19:29:07'
 ```
@@ -1396,7 +1353,7 @@ VictoriaMetrics accepts metrics data in JSON line format at `/api/v1/import` end
 
 Example for importing data obtained via [/api/v1/export](#how-to-export-data-in-json-line-format):
 
-```console
+```sh
 # Export the data from <source-victoriametrics>:
 curl http://source-victoriametrics:8428/api/v1/export -d 'match={__name__!=""}' > exported_data.jsonl
 
@@ -1406,7 +1363,7 @@ curl -X POST http://destination-victoriametrics:8428/api/v1/import -T exported_d
 
 Pass `Content-Encoding: gzip` HTTP request header to `/api/v1/import` for importing gzipped data:
 
-```console
+```sh
 # Export gzipped data from <source-victoriametrics>:
 curl -H 'Accept-Encoding: gzip' http://source-victoriametrics:8428/api/v1/export -d 'match={__name__!=""}' > exported_data.jsonl.gz
 
@@ -1432,7 +1389,7 @@ The specification of VictoriaMetrics' native format may yet change and is not fo
 
 If you have a native format file obtained via [/api/v1/export/native](#how-to-export-data-in-native-format) however this is the most efficient protocol for importing data in.
 
-```console
+```sh
 # Export the data from <source-victoriametrics>:
 curl http://source-victoriametrics:8428/api/v1/export/native -d 'match={__name__!=""}' > exported_data.bin
 
@@ -1450,7 +1407,7 @@ Note that it could be required to flush response cache after importing historica
 Arbitrary CSV data can be imported via `/api/v1/import/csv`. The CSV data is imported according to the provided `format` query arg.
 The `format` query arg must contain comma-separated list of parsing rules for CSV fields. Each rule consists of three parts delimited by a colon:
 
-```
+```text
 <column_pos>:<type>:<context>
 ```
 
@@ -1473,14 +1430,14 @@ Each request to `/api/v1/import/csv` may contain arbitrary number of CSV lines.
 
 Example for importing CSV data via `/api/v1/import/csv`:
 
-```console
+```sh
 curl -d "GOOG,1.23,4.56,NYSE" 'http://localhost:8428/api/v1/import/csv?format=2:metric:ask,3:metric:bid,1:label:ticker,4:label:market'
 curl -d "MSFT,3.21,1.67,NASDAQ" 'http://localhost:8428/api/v1/import/csv?format=2:metric:ask,3:metric:bid,1:label:ticker,4:label:market'
 ```
 
 After that the data may be read via [/api/v1/export](#how-to-export-data-in-json-line-format) endpoint:
 
-```console
+```sh
 curl -G 'http://localhost:8428/api/v1/export' -d 'match[]={ticker!=""}'
 ```
 
@@ -1506,23 +1463,19 @@ and in [Pushgateway format](https://github.com/prometheus/pushgateway#url) via `
 
 For example, the following command imports a single line in Prometheus exposition format into VictoriaMetrics:
 
-<div class="with-copy" markdown="1">
 
-```console
+```sh
 curl -d 'foo{bar="baz"} 123' -X POST 'http://localhost:8428/api/v1/import/prometheus'
 ```
 
-</div>
 
 The following command may be used for verifying the imported data:
 
-<div class="with-copy" markdown="1">
 
-```console
+```sh
 curl -G 'http://localhost:8428/api/v1/export' -d 'match={__name__=~"foo"}'
 ```
 
-</div>
 
 It should return something like the following:
 
@@ -1532,24 +1485,20 @@ It should return something like the following:
 
 The following command imports a single metric via [Pushgateway format](https://github.com/prometheus/pushgateway#url) with `{job="my_app",instance="host123"}` labels:
 
-<div class="with-copy" markdown="1">
 
-```console
+```sh
 curl -d 'metric{label="abc"} 123' -X POST 'http://localhost:8428/api/v1/import/prometheus/metrics/job/my_app/instance/host123'
 ```
 
-</div>
 
 Pass `Content-Encoding: gzip` HTTP request header to `/api/v1/import/prometheus` for importing gzipped data:
 
-<div class="with-copy" markdown="1">
 
-```console
+```sh
 # Import gzipped data to <destination-victoriametrics>:
 curl -X POST -H 'Content-Encoding: gzip' http://destination-victoriametrics:8428/api/v1/import/prometheus -T prometheus_data.gz
 ```
 
-</div>
 
 Extra labels may be added to all the imported metrics either via [Pushgateway format](https://github.com/prometheus/pushgateway#url)
 or by passing `extra_label=name=value` query args. For example, `/api/v1/import/prometheus?extra_label=foo=bar` would add `{foo="bar"}` label to all the imported metrics.
@@ -1577,7 +1526,7 @@ and exports data in this format at [/api/v1/export](#how-to-export-data-in-json-
 
 The format follows [JSON streaming concept](http://ndjson.org/), e.g. each line contains JSON object with metrics data in the following format:
 
-```
+```json
 {
   // metric contans metric name plus labels for a particular time series
   "metric":{
@@ -1629,7 +1578,7 @@ The `-relabelConfig` files can contain special placeholders in the form `%{ENV_V
 
 Example contents for `-relabelConfig` file:
 
-```yml
+```yaml
 # Add {cluster="dev"} label.
 - target_label: cluster
   replacement: dev
@@ -1657,7 +1606,7 @@ Optional `start` and `end` args may be added to the request in order to scrape t
 See [allowed formats](#timestamp-formats) for these args.
 
 For example:
-```console
+```sh
 curl http://<victoriametrics-addr>:8428/federate -d 'match[]=<timeseries_selector_for_export>' -d 'start=1654543486' -d 'end=1654543486'
 curl http://<victoriametrics-addr>:8428/federate -d 'match[]=<timeseries_selector_for_export>' -d 'start=2022-06-06T19:25:48' -d 'end=2022-06-06T19:29:07'
 ```
@@ -1734,7 +1683,7 @@ then it can be configured with multiple `-remoteWrite.url` command-line flags, w
 instance in a particular availability zone, in order to replicate the collected data to all the VictoriaMetrics instances.
 For example, the following command instructs `vmagent` to replicate data to `vm-az1` and `vm-az2` instances of VictoriaMetrics:
 
-```console
+```sh
 /path/to/vmagent \
   -remoteWrite.url=http://<vm-az1>:8428/api/v1/write \
   -remoteWrite.url=http://<vm-az2>:8428/api/v1/write
@@ -1744,7 +1693,7 @@ If you use Prometheus for collecting and writing the data to VictoriaMetrics,
 then the following [`remote_write`](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write) section
 in Prometheus config can be used for replicating the collected data to `vm-az1` and `vm-az2` VictoriaMetrics instances:
 
-```yml
+```yaml
 remote_write:
   - url: http://<vm-az1>:8428/api/v1/write
   - url: http://<vm-az2>:8428/api/v1/write
@@ -1855,8 +1804,8 @@ This increases overhead during data querying, since VictoriaMetrics needs to rea
 bigger number of parts per each request. That's why it is recommended to have at least 20%
 of free disk space under directory pointed by `-storageDataPath` command-line flag.
 
-Information about merging process is available in [the dashboard for single-node VictoriaMetrics](https://grafana.com/grafana/dashboards/10229-victoriametrics-single-node/)
-and [the dashboard for VictoriaMetrics cluster](https://grafana.com/grafana/dashboards/11176-victoriametrics-cluster/).
+Information about merging process is available in [the dashboard for single-node VictoriaMetrics](https://grafana.com/grafana/dashboards/10229)
+and [the dashboard for VictoriaMetrics cluster](https://grafana.com/grafana/dashboards/11176).
 See more details in [monitoring docs](#monitoring).
 
 See [this article](https://valyala.medium.com/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282) for more details.
@@ -1918,7 +1867,7 @@ command-line flag is applied to it. If series matches multiple configured retent
 For example, the following config sets 3 days retention for time series with `team="juniors"` label,
 30 days retention for time series with `env="dev"` or `env="staging"` label and 1 year retention for the remaining time series:
 
-```
+```sh
 -retentionFilter='{team="juniors"}:3d' -retentionFilter='{env=~"dev|staging"}:30d' -retentionPeriod=1y
 ```
 
@@ -2013,8 +1962,10 @@ VictoriaMetrics provides the following security-related command-line flags:
   with [HTTP Basic Authentication](https://en.wikipedia.org/wiki/Basic_access_authentication).
 * `-deleteAuthKey` for protecting `/api/v1/admin/tsdb/delete_series` endpoint. See [how to delete time series](#how-to-delete-time-series).
 * `-snapshotAuthKey` for protecting `/snapshot*` endpoints. See [how to work with snapshots](#how-to-work-with-snapshots).
+* `-forceFlushAuthKey` for protecting `/internal/force_flush` endpoint. See [these docs](#troubleshooting).
 * `-forceMergeAuthKey` for protecting `/internal/force_merge` endpoint. See [force merge docs](#forced-merge).
 * `-search.resetCacheAuthKey` for protecting `/internal/resetRollupResultCache` endpoint. See [backfilling](#backfilling) for more details.
+* `-reloadAuthKey` for protecting `/-/reload` endpoint, which is used for force reloading of [`-promscrape.config`](#how-to-scrape-prometheus-exporters-such-as-node-exporter).
 * `-configAuthKey` for protecting `/config` endpoint, since it may contain sensitive information such as passwords.
 * `-flagsAuthKey` for protecting `/flags` endpoint.
 * `-pprofAuthKey` for protecting `/debug/pprof/*` endpoints, which can be used for [profiling](#profiling).
@@ -2044,7 +1995,7 @@ and [the general security page at VictoriaMetrics website](https://victoriametri
   If you plan to store more than 1TB of data on `ext4` partition or plan extending it to more than 16TB,
   then the following options are recommended to pass to `mkfs.ext4`:
 
-```console
+```sh
 mkfs.ext4 ... -O 64bit,huge_file,extent -T huge
 ```
 
@@ -2058,9 +2009,9 @@ with 10 seconds interval.
 
 _Please note, never use loadbalancer address for scraping metrics. All monitored components should be scraped directly by their address._
 
-Official Grafana dashboards available for [single-node](https://grafana.com/grafana/dashboards/10229-victoriametrics-single-node/) 
-and [clustered](https://grafana.com/grafana/dashboards/11176-victoriametrics-cluster/) VictoriaMetrics. 
-See an [alternative dashboard for clustered VictoriaMetrics](https://grafana.com/grafana/dashboards/11831) 
+Official Grafana dashboards available for [single-node](https://grafana.com/grafana/dashboards/10229)
+and [clustered](https://grafana.com/grafana/dashboards/11176) VictoriaMetrics.
+See an [alternative dashboard for clustered VictoriaMetrics](https://grafana.com/grafana/dashboards/11831)
 created by community.
 
 Graphs on the dashboards contain useful hints - hover the `i` icon in the top left corner of each graph to read it.
@@ -2102,7 +2053,7 @@ In this case VictoriaMetrics puts query trace into `trace` field in the output J
 
 For example, the following command:
 
-```console
+```sh
 curl http://localhost:8428/api/v1/query_range -d 'query=2*rand()' -d 'start=-1h' -d 'step=1m' -d 'trace=1' | jq '.trace'
 ```
 
@@ -2301,7 +2252,7 @@ For example, the following command instructs VictoriaMetrics to push metrics fro
 with `user:pass` [Basic auth](https://en.wikipedia.org/wiki/Basic_access_authentication). The `instance="foobar"` and `job="vm"` labels
 are added to all the metrics before sending them to the remote storage:
 
-```console
+```sh
 /path/to/victoria-metrics \
   -pushmetrics.url=https://user:pass@maas.victoriametrics.com/api/v1/import/prometheus \
   -pushmetrics.extraLabel='instance="foobar"' \
@@ -2329,8 +2280,8 @@ The following metrics for each type of cache are exported at [`/metrics` page](#
 * `vm_cache_misses_total` - the number of cache misses
 * `vm_cache_entries` - the number of entries in the cache
 
-Both Grafana dashboards for [single-node VictoriaMetrics](https://grafana.com/grafana/dashboards/10229-victoriametrics-single-node/)
-and [clustered VictoriaMetrics](https://grafana.com/grafana/dashboards/11176-victoriametrics-cluster/)
+Both Grafana dashboards for [single-node VictoriaMetrics](https://grafana.com/grafana/dashboards/10229)
+and [clustered VictoriaMetrics](https://grafana.com/grafana/dashboards/11176)
 contain `Caches` section with cache metrics visualized. The panels show the current
 memory usage by each type of cache, and also a cache hit rate. If hit rate is close to 100%
 then cache efficiency is already very high and does not need any tuning.
@@ -2444,23 +2395,19 @@ VictoriaMetrics provides handlers for collecting the following [Go profiles](htt
 
 * Memory profile. It can be collected with the following command (replace `0.0.0.0` with hostname if needed):
 
-<div class="with-copy" markdown="1">
 
-```console
+```sh
 curl http://0.0.0.0:8428/debug/pprof/heap > mem.pprof
 ```
 
-</div>
 
 * CPU profile. It can be collected with the following command (replace `0.0.0.0` with hostname if needed):
 
-<div class="with-copy" markdown="1">
 
-```console
+```sh
 curl http://0.0.0.0:8428/debug/pprof/profile > cpu.pprof
 ```
 
-</div>
 
 The command for collecting CPU profile waits for 30 seconds before returning.
 
@@ -2528,7 +2475,7 @@ If the page needs to have many images, consider using WEB-optimized image format
 When adding a new doc with many images use `webp` format right away. Or use a Makefile command below to
 convert already existing images at `docs` folder automatically to `web` format:
 
-```console
+```sh
 make docs-images-to-webp
 ```
 
@@ -2568,15 +2515,16 @@ Files included in each folder:
 
 Pass `-help` to VictoriaMetrics in order to see the list of supported command-line flags with their description:
 
-```
+```sh
   -bigMergeConcurrency int
-     Deprecated: this flag does nothing. Please use -smallMergeConcurrency for controlling the concurrency of background merges. See https://docs.victoriametrics.com/#storage
+     Deprecated: this flag does nothing
   -blockcache.missesBeforeCaching int
      The number of cache misses before putting the block into cache. Higher values may reduce indexdb/dataBlocks cache size at the cost of higher CPU and disk read usage (default 2)
   -cacheExpireDuration duration
      Items are removed from in-memory caches after they aren't accessed for this duration. Lower values may reduce memory usage at the cost of higher CPU usage. See also -prevCacheRemovalPercent (default 30m0s)
-  -configAuthKey string
+  -configAuthKey value
      Authorization key for accessing /config page. It must be passed via authKey query arg
+     Flag value can be read from the given file when using -configAuthKey=file:///abs/path/to/file or -configAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -configAuthKey=http://host/path or -configAuthKey=https://host/path
   -csvTrimTimestamp duration
      Trim timestamps when importing csv data to this duration. Minimum practical duration is 1ms. Higher duration (i.e. 1s) may be used for reducing disk space usage for timestamp data (default 1ms)
   -datadog.maxInsertRequestSize size
@@ -2586,8 +2534,9 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      Sanitize metric names for the ingested DataDog data to comply with DataDog behaviour described at https://docs.datadoghq.com/metrics/custom_metrics/#naming-custom-metrics (default true)
   -dedup.minScrapeInterval duration
      Leave only the last sample in every time series per each discrete interval equal to -dedup.minScrapeInterval > 0. See https://docs.victoriametrics.com/#deduplication and https://docs.victoriametrics.com/#downsampling
-  -deleteAuthKey string
+  -deleteAuthKey value
      authKey for metrics' deletion via /api/v1/admin/tsdb/delete_series and /tags/delSeries
+     Flag value can be read from the given file when using -deleteAuthKey=file:///abs/path/to/file or -deleteAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -deleteAuthKey=http://host/path or -deleteAuthKey=https://host/path
   -denyQueriesOutsideRetention
      Whether to deny queries outside the configured -retentionPeriod. When set, then /api/v1/query_range would return '503 Service Unavailable' error for queries with 'from' value outside -retentionPeriod. This may be useful when multiple data sources with distinct retentions are hidden behind query-tee
   -denyQueryTracing
@@ -2608,13 +2557,16 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
   -filestream.disableFadvise
      Whether to disable fadvise() syscall when reading large data files. The fadvise() syscall prevents from eviction of recently accessed data from OS page cache during background merges and backups. In some rare cases it is better to disable the syscall if it uses too much CPU
   -finalMergeDelay duration
-     The delay before starting final merge for per-month partition after no new data is ingested into it. Final merge may require additional disk IO and CPU resources. Final merge may increase query speed and reduce disk space usage in some cases. Zero value disables final merge
-  -flagsAuthKey string
+     Deprecated: this flag does nothing
+  -flagsAuthKey value
      Auth key for /flags endpoint. It must be passed via authKey query arg. It overrides httpAuth.* settings
-  -forceFlushAuthKey string
+     Flag value can be read from the given file when using -flagsAuthKey=file:///abs/path/to/file or -flagsAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -flagsAuthKey=http://host/path or -flagsAuthKey=https://host/path
+  -forceFlushAuthKey value
      authKey, which must be passed in query string to /internal/force_flush pages
-  -forceMergeAuthKey string
+     Flag value can be read from the given file when using -forceFlushAuthKey=file:///abs/path/to/file or -forceFlushAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -forceFlushAuthKey=http://host/path or -forceFlushAuthKey=https://host/path
+  -forceMergeAuthKey value
      authKey, which must be passed in query string to /internal/force_merge pages
+     Flag value can be read from the given file when using -forceMergeAuthKey=file:///abs/path/to/file or -forceMergeAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -forceMergeAuthKey=http://host/path or -forceMergeAuthKey=https://host/path
   -fs.disableMmap
      Whether to use pread() instead of mmap() for reading data files. By default, mmap() is used for 64-bit arches and pread() is used for 32-bit arches, since they cannot read data files bigger than 2^32 bytes in memory. mmap() is usually faster for reading small data chunks than pread()
   -graphiteListenAddr string
@@ -2641,8 +2593,9 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      An optional prefix to add to all the paths handled by http server. For example, if '-http.pathPrefix=/foo/bar' is set, then all the http requests will be handled on '/foo/bar/*' paths. This may be useful for proxied requests. See https://www.robustperception.io/using-external-urls-and-proxies-with-prometheus
   -http.shutdownDelay duration
      Optional delay before http server shutdown. During this delay, the server returns non-OK responses from /health page, so load balancers can route new requests to other servers
-  -httpAuth.password string
+  -httpAuth.password value
      Password for HTTP server's Basic Auth. The authentication is disabled if -httpAuth.username is empty
+     Flag value can be read from the given file when using -httpAuth.password=file:///abs/path/to/file or -httpAuth.password=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -httpAuth.password=http://host/path or -httpAuth.password=https://host/path
   -httpAuth.username string
      Username for HTTP server's Basic Auth. The authentication is disabled if empty. See also -httpAuth.password
   -httpListenAddr string
@@ -2724,8 +2677,9 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      Allowed percent of system memory VictoriaMetrics caches may occupy. See also -memory.allowedBytes. Too low a value may increase cache miss rate usually resulting in higher CPU and disk IO usage. Too high a value may evict too much data from the OS page cache which will result in higher disk IO usage (default 60)
   -metrics.exposeMetadata
      Whether to expose TYPE and HELP metadata at the /metrics page, which is exposed at -httpListenAddr . The metadata may be needed when the /metrics page is consumed by systems, which require this information. For example, Managed Prometheus in Google Cloud - https://cloud.google.com/stackdriver/docs/managed-prometheus/troubleshooting#missing-metric-type
-  -metricsAuthKey string
+  -metricsAuthKey value
      Auth key for /metrics endpoint. It must be passed via authKey query arg. It overrides httpAuth.* settings
+     Flag value can be read from the given file when using -metricsAuthKey=file:///abs/path/to/file or -metricsAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -metricsAuthKey=http://host/path or -metricsAuthKey=https://host/path
   -newrelic.maxInsertRequestSize size
      The maximum size in bytes of a single NewRelic request to /newrelic/infra/v2/metrics/events/bulk
      Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 67108864)
@@ -2744,8 +2698,9 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 33554432)
   -opentsdbhttpTrimTimestamp duration
      Trim timestamps for OpenTSDB HTTP data to this duration. Minimum practical duration is 1ms. Higher duration (i.e. 1s) may be used for reducing disk space usage for timestamp data (default 1ms)
-  -pprofAuthKey string
+  -pprofAuthKey value
      Auth key for /debug/pprof/* endpoints. It must be passed via authKey query arg. It overrides httpAuth.* settings
+     Flag value can be read from the given file when using -pprofAuthKey=file:///abs/path/to/file or -pprofAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -pprofAuthKey=http://host/path or -pprofAuthKey=https://host/path
   -precisionBits int
      The number of precision bits to store per each value. Lower precision bits improves data compression at the cost of precision loss (default 64)
   -prevCacheRemovalPercent float
@@ -2808,6 +2763,8 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      Interval for checking for changes in http endpoint service discovery. This works only if http_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs.html#http_sd_configs for details (default 1m0s)
   -promscrape.kubernetes.apiServerTimeout duration
      How frequently to reload the full state from Kubernetes API server (default 30m0s)
+  -promscrape.kubernetes.attachNodeMetadataAll
+     Whether to set attach_metadata.node=true for all the kubernetes_sd_configs at -promscrape.config . It is possible to set attach_metadata.node=false individually per each kubernetes_sd_configs . See https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs
   -promscrape.kubernetesSDCheckInterval duration
      Interval for checking for changes in Kubernetes API server. This works only if kubernetes_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs for details (default 30s)
   -promscrape.kumaSDCheckInterval duration
@@ -2858,6 +2815,9 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      Supports an array of values separated by comma or specified via multiple flags.
   -relabelConfig string
      Optional path to a file with relabeling rules, which are applied to all the ingested metrics. The path can point either to local file or to http url. See https://docs.victoriametrics.com/#relabeling for details. The config is reloaded on SIGHUP signal
+ -reloadAuthKey value
+     Auth key for /-/reload http endpoint. It must be passed as authKey=...
+     Flag value can be read from the given file when using -reloadAuthKey=file:///abs/path/to/file or -reloadAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -reloadAuthKey=http://host/path or -reloadAuthKey=https://host/path
   -retentionFilter array
      Retention filter in the format 'filter:retention'. For example, '{env="dev"}:3d' configures the retention for time series with env="dev" label to 3 days. See https://docs.victoriametrics.com/#retention-filters for details. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
      Supports an array of values separated by comma or specified via multiple flags.
@@ -2952,8 +2912,9 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      Query stats for /api/v1/status/top_queries is tracked on this number of last queries. Zero value disables query stats tracking (default 20000)
   -search.queryStats.minQueryDuration duration
      The minimum duration for queries to track in query stats at /api/v1/status/top_queries. Queries with lower duration are ignored in query stats (default 1ms)
-  -search.resetCacheAuthKey string
+  -search.resetCacheAuthKey value
      Optional authKey for resetting rollup cache via /internal/resetRollupResultCache call
+     Flag value can be read from the given file when using -search.resetCacheAuthKey=file:///abs/path/to/file or -search.resetCacheAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -search.resetCacheAuthKey=http://host/path or -search.resetCacheAuthKey=https://host/path
   -search.setLookbackToStep
      Whether to fix lookback interval to 'step' query arg value. If set to true, the query model becomes closer to InfluxDB data model. If set to true, then -search.maxLookback and -search.maxStalenessInterval are ignored
   -search.treatDotsAsIsInRegexps
@@ -2965,9 +2926,10 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
   -selfScrapeJob string
      Value for 'job' label, which is added to self-scraped metrics (default "victoria-metrics")
   -smallMergeConcurrency int
-     The maximum number of workers for background merges. See https://docs.victoriametrics.com/#storage . It isn't recommended tuning this flag in general case, since this may lead to uncontrolled increase in the number of parts and increased CPU usage during queries
-  -snapshotAuthKey string
+     Deprecated: this flag does nothing
+  -snapshotAuthKey value
      authKey, which must be passed in query string to /snapshot* pages
+     Flag value can be read from the given file when using -snapshotAuthKey=file:///abs/path/to/file or -snapshotAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -snapshotAuthKey=http://host/path or -snapshotAuthKey=https://host/path
   -snapshotCreateTimeout duration
      The timeout for creating new snapshot. If set, make sure that timeout is lower than backup period
   -snapshotsMaxAge value
@@ -3023,4 +2985,6 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      Optional URL for proxying requests to vmalert. For example, if -vmalert.proxyURL=http://vmalert:8880 , then alerting API requests such as /api/v1/rules from Grafana will be proxied to http://vmalert:8880/api/v1/rules
   -vmui.customDashboardsPath string
      Optional path to vmui dashboards. See https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmui/packages/vmui/public/dashboards
+  -vmui.defaultTimezone string
+     The default timezone to be used in vmui. Timezone must be a valid IANA Time Zone. For example: America/New_York, Europe/Berlin, Etc/GMT+3 or Local. See https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmui#timezone-configuration
 ```

app/vmagent/main.go

@@ -69,7 +69,8 @@ var (
 		"See also -opentsdbHTTPListenAddr.useProxyProtocol")
 	opentsdbHTTPUseProxyProtocol = flag.Bool("opentsdbHTTPListenAddr.useProxyProtocol", false, "Whether to use proxy protocol for connections accepted "+
 		"at -opentsdbHTTPListenAddr . See https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt")
-	configAuthKey = flag.String("configAuthKey", "", "Authorization key for accessing /config page. It must be passed via authKey query arg")
+	configAuthKey = flagutil.NewPassword("configAuthKey", "Authorization key for accessing /config page. It must be passed via authKey query arg")
+	reloadAuthKey = flagutil.NewPassword("reloadAuthKey", "Auth key for /-/reload http endpoint. It must be passed as authKey=...")
 	dryRun        = flag.Bool("dryRun", false, "Whether to check config files without running vmagent. The following files are checked: "+
 		"-promscrape.config, -remoteWrite.relabelConfig, -remoteWrite.urlRelabelConfig, -remoteWrite.streamAggr.config . "+
 		"Unknown config entries aren't allowed in -promscrape.config by default. This can be changed by passing -promscrape.config.strictParse=false command-line flag")
@@ -421,7 +422,7 @@ func requestHandler(w http.ResponseWriter, r *http.Request) bool {
 		}
 		return true
 	case "/prometheus/config", "/config":
-		if !httpserver.CheckAuthFlag(w, r, *configAuthKey, "configAuthKey") {
+		if !httpserver.CheckAuthFlag(w, r, configAuthKey.Get(), "configAuthKey") {
 			return true
 		}
 		promscrapeConfigRequests.Inc()
@@ -430,7 +431,7 @@ func requestHandler(w http.ResponseWriter, r *http.Request) bool {
 		return true
 	case "/prometheus/api/v1/status/config", "/api/v1/status/config":
 		// See https://prometheus.io/docs/prometheus/latest/querying/api/#config
-		if !httpserver.CheckAuthFlag(w, r, *configAuthKey, "configAuthKey") {
+		if !httpserver.CheckAuthFlag(w, r, configAuthKey.Get(), "configAuthKey") {
 			return true
 		}
 		promscrapeStatusConfigRequests.Inc()
@@ -440,6 +441,9 @@ func requestHandler(w http.ResponseWriter, r *http.Request) bool {
 		fmt.Fprintf(w, `{"status":"success","data":{"yaml":%q}}`, bb.B)
 		return true
 	case "/prometheus/-/reload", "/-/reload":
+		if !httpserver.CheckAuthFlag(w, r, reloadAuthKey.Get(), "reloadAuthKey") {
+			return true
+		}
 		promscrapeConfigReloadRequests.Inc()
 		procutil.SelfSIGHUP()
 		w.WriteHeader(http.StatusOK)

app/vmagent/remotewrite/client.go

@@ -18,6 +18,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promauth"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/common"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/timerpool"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/timeutil"
 	"github.com/VictoriaMetrics/metrics"
 )
 
@@ -395,7 +396,8 @@ func (c *client) newRequest(url string, body []byte) (*http.Request, error) {
 // Otherwise it tries sending the block to remote storage indefinitely.
 func (c *client) sendBlockHTTP(block []byte) bool {
 	c.rl.register(len(block), c.stopCh)
-	retryDuration := time.Second
+	maxRetryDuration := timeutil.AddJitterToDuration(time.Minute)
+	retryDuration := timeutil.AddJitterToDuration(time.Second)
 	retriesCount := 0
 
 again:
@@ -405,8 +407,8 @@ again:
 	if err != nil {
 		c.errorsCount.Inc()
 		retryDuration *= 2
-		if retryDuration > time.Minute {
-			retryDuration = time.Minute
+		if retryDuration > maxRetryDuration {
+			retryDuration = maxRetryDuration
 		}
 		logger.Warnf("couldn't send a block with size %d bytes to %q: %s; re-sending the block in %.3f seconds",
 			len(block), c.sanitizedURL, err, retryDuration.Seconds())
@@ -452,8 +454,8 @@ again:
 	// Unexpected status code returned
 	retriesCount++
 	retryDuration *= 2
-	if retryDuration > time.Minute {
-		retryDuration = time.Minute
+	if retryDuration > maxRetryDuration {
+		retryDuration = maxRetryDuration
 	}
 	body, err := io.ReadAll(resp.Body)
 	_ = resp.Body.Close()

app/vmagent/remotewrite/pendingseries.go

@@ -7,6 +7,7 @@ import (
 	"time"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/cgroup"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/decimal"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/encoding/zstd"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fasttime"
@@ -15,6 +16,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/persistentqueue"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promrelabel"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/timeutil"
 	"github.com/VictoriaMetrics/metrics"
 	"github.com/golang/snappy"
 )
@@ -69,7 +71,8 @@ func (ps *pendingSeries) periodicFlusher() {
 	if flushSeconds <= 0 {
 		flushSeconds = 1
 	}
-	ticker := time.NewTicker(*flushInterval)
+	d := timeutil.AddJitterToDuration(*flushInterval)
+	ticker := time.NewTicker(d)
 	defer ticker.Stop()
 	for {
 		select {
@@ -107,11 +110,12 @@ type writeRequest struct {
 
 	wr prompbmarshal.WriteRequest
 
-	tss []prompbmarshal.TimeSeries
-
+	tss     []prompbmarshal.TimeSeries
 	labels  []prompbmarshal.Label
 	samples []prompbmarshal.Sample
-	buf     []byte
+
+	// buf holds labels data
+	buf []byte
 }
 
 func (wr *writeRequest) reset() {
@@ -222,33 +226,45 @@ func (wr *writeRequest) copyTimeSeries(dst, src *prompbmarshal.TimeSeries) {
 	wr.buf = buf
 }
 
+// marshalConcurrency limits the maximum number of concurrent workers, which marshal and compress WriteRequest.
+var marshalConcurrencyCh = make(chan struct{}, cgroup.AvailableCPUs())
+
 func tryPushWriteRequest(wr *prompbmarshal.WriteRequest, tryPushBlock func(block []byte) bool, isVMRemoteWrite bool) bool {
 	if len(wr.Timeseries) == 0 {
 		// Nothing to push
 		return true
 	}
+
+	marshalConcurrencyCh <- struct{}{}
+
 	bb := writeRequestBufPool.Get()
 	bb.B = wr.MarshalProtobuf(bb.B[:0])
 	if len(bb.B) <= maxUnpackedBlockSize.IntN() {
-		zb := snappyBufPool.Get()
+		zb := compressBufPool.Get()
 		if isVMRemoteWrite {
 			zb.B = zstd.CompressLevel(zb.B[:0], bb.B, *vmProtoCompressLevel)
 		} else {
 			zb.B = snappy.Encode(zb.B[:cap(zb.B)], bb.B)
 		}
 		writeRequestBufPool.Put(bb)
+
+		<-marshalConcurrencyCh
+
 		if len(zb.B) <= persistentqueue.MaxBlockSize {
-			if !tryPushBlock(zb.B) {
-				return false
+			zbLen := len(zb.B)
+			ok := tryPushBlock(zb.B)
+			compressBufPool.Put(zb)
+			if ok {
+				blockSizeRows.Update(float64(len(wr.Timeseries)))
+				blockSizeBytes.Update(float64(zbLen))
 			}
-			blockSizeRows.Update(float64(len(wr.Timeseries)))
-			blockSizeBytes.Update(float64(len(zb.B)))
-			snappyBufPool.Put(zb)
-			return true
+			return ok
 		}
-		snappyBufPool.Put(zb)
+		compressBufPool.Put(zb)
 	} else {
 		writeRequestBufPool.Put(bb)
+
+		<-marshalConcurrencyCh
 	}
 
 	// Too big block. Recursively split it into smaller parts if possible.
@@ -294,5 +310,7 @@ var (
 	blockSizeRows  = metrics.NewHistogram(`vmagent_remotewrite_block_size_rows`)
 )
 
-var writeRequestBufPool bytesutil.ByteBufferPool
-var snappyBufPool bytesutil.ByteBufferPool
+var (
+	writeRequestBufPool bytesutil.ByteBufferPool
+	compressBufPool     bytesutil.ByteBufferPool
+)

app/vmalert/Makefile

@@ -68,6 +68,7 @@ publish-vmalert:
 
 test-vmalert:
 	go test -v -race -cover ./app/vmalert -loggerLevel=ERROR
+	go test -v -race -cover ./app/vmalert/rule
 	go test -v -race -cover ./app/vmalert/templates
 	go test -v -race -cover ./app/vmalert/datasource
 	go test -v -race -cover ./app/vmalert/notifier

app/vmalert/config/testdata/rules/rules2-good.rules

@@ -22,6 +22,7 @@ groups:
               {{ . | first | value }}
             {{ end }}
           description: "It is {{ $value }} connections for {{$labels.instance}}"
+          link: http://localhost:3000/d/wNf0q_kZk?viewPanel=51&from={{($activeAt.Add (parseDurationTime "1h")).UnixMilli}}&to={{($activeAt.Add (parseDurationTime "-1h")).UnixMilli}}
       - alert: ExampleAlertAlwaysFiring
         update_entries_limit: -1
         expr: sum by(job)

app/vmalert/manager.go

@@ -156,11 +156,14 @@ func (m *manager) update(ctx context.Context, groupsCfg []config.Group, restore
 		var wg sync.WaitGroup
 		for _, item := range toUpdate {
 			wg.Add(1)
+			// cancel evaluation so the Update will be applied as fast as possible.
+			// it is important to call InterruptEval before the update, because cancel fn
+			// can be re-assigned during the update.
+			item.old.InterruptEval()
 			go func(old *rule.Group, new *rule.Group) {
 				old.UpdateWith(new)
 				wg.Done()
 			}(item.old, item.new)
-			item.old.InterruptEval()
 		}
 		wg.Wait()
 	}

app/vmalert/rule/alerting.go

@@ -324,16 +324,6 @@ func (ar *AlertingRule) execRange(ctx context.Context, start, end time.Time) ([]
 			return nil, fmt.Errorf("failed to create alert: %w", err)
 		}
 
-		// if alert is instant, For: 0
-		if ar.For == 0 {
-			a.State = notifier.StateFiring
-			for i := range s.Values {
-				result = append(result, ar.alertToTimeSeries(a, s.Timestamps[i])...)
-			}
-			continue
-		}
-
-		// if alert with For > 0
 		prevT := time.Time{}
 		for i := range s.Values {
 			at := time.Unix(s.Timestamps[i], 0)
@@ -354,6 +344,10 @@ func (ar *AlertingRule) execRange(ctx context.Context, start, end time.Time) ([]
 				a.Start = at
 			}
 			prevT = at
+			if ar.For == 0 {
+				// rules with `for: 0` are always firing when they have Value
+				a.State = notifier.StateFiring
+			}
 			result = append(result, ar.alertToTimeSeries(a, s.Timestamps[i])...)
 
 			// save alert's state on last iteration, so it can be used on the next execRange call
@@ -446,14 +440,13 @@ func (ar *AlertingRule) exec(ctx context.Context, ts time.Time, limit int) ([]pr
 			a.KeepFiringSince = time.Time{}
 			continue
 		}
-		a, err := ar.newAlert(m, ls, start, qFn)
+		a, err := ar.newAlert(m, ls, ts, qFn)
 		if err != nil {
 			curState.Err = fmt.Errorf("failed to create alert: %w", err)
 			return nil, curState.Err
 		}
 		a.ID = h
 		a.State = notifier.StatePending
-		a.ActiveAt = ts
 		ar.alerts[h] = a
 		ar.logDebugf(ts, a, "created in state PENDING")
 	}
@@ -479,7 +472,7 @@ func (ar *AlertingRule) exec(ctx context.Context, ts time.Time, limit int) ([]pr
 				}
 				// alerts with ar.KeepFiringFor>0 may remain FIRING
 				// even if their expression isn't true anymore
-				if ts.Sub(a.KeepFiringSince) > ar.KeepFiringFor {
+				if ts.Sub(a.KeepFiringSince) >= ar.KeepFiringFor {
 					a.State = notifier.StateInactive
 					a.ResolvedAt = ts
 					ar.logDebugf(ts, a, "FIRING => INACTIVE: is absent in current evaluation round")
@@ -559,9 +552,9 @@ func (ar *AlertingRule) newAlert(m datasource.Metric, ls *labelSet, start time.T
 }
 
 const (
-	// alertMetricName is the metric name for synthetic alert timeseries.
+	// alertMetricName is the metric name for time series reflecting the alert state.
 	alertMetricName = "ALERTS"
-	// alertForStateMetricName is the metric name for 'for' state of alert.
+	// alertForStateMetricName is the metric name for time series reflecting the moment of time when alert became active.
 	alertForStateMetricName = "ALERTS_FOR_STATE"
 
 	// alertNameLabel is the label name indicating the name of an alert.
@@ -576,12 +569,10 @@ const (
 
 // alertToTimeSeries converts the given alert with the given timestamp to time series
 func (ar *AlertingRule) alertToTimeSeries(a *notifier.Alert, timestamp int64) []prompbmarshal.TimeSeries {
-	var tss []prompbmarshal.TimeSeries
-	tss = append(tss, alertToTimeSeries(a, timestamp))
-	if ar.For > 0 {
-		tss = append(tss, alertForToTimeSeries(a, timestamp))
+	return []prompbmarshal.TimeSeries{
+		alertToTimeSeries(a, timestamp),
+		alertForToTimeSeries(a, timestamp),
 	}
-	return tss
 }
 
 func alertToTimeSeries(a *notifier.Alert, timestamp int64) prompbmarshal.TimeSeries {

app/vmalert/rule/alerting_test.go

@@ -28,20 +28,26 @@ func TestAlertingRule_ToTimeSeries(t *testing.T) {
 	}{
 		{
 			newTestAlertingRule("instant", 0),
-			&notifier.Alert{State: notifier.StateFiring},
+			&notifier.Alert{State: notifier.StateFiring, ActiveAt: timestamp.Add(time.Second)},
 			[]prompbmarshal.TimeSeries{
 				newTimeSeries([]float64{1}, []int64{timestamp.UnixNano()}, map[string]string{
 					"__name__":      alertMetricName,
 					alertStateLabel: notifier.StateFiring.String(),
 				}),
+				newTimeSeries([]float64{float64(timestamp.Add(time.Second).Unix())},
+					[]int64{timestamp.UnixNano()},
+					map[string]string{
+						"__name__": alertForStateMetricName,
+					}),
 			},
 		},
 		{
 			newTestAlertingRule("instant extra labels", 0),
-			&notifier.Alert{State: notifier.StateFiring, Labels: map[string]string{
-				"job":      "foo",
-				"instance": "bar",
-			}},
+			&notifier.Alert{State: notifier.StateFiring, ActiveAt: timestamp.Add(time.Second),
+				Labels: map[string]string{
+					"job":      "foo",
+					"instance": "bar",
+				}},
 			[]prompbmarshal.TimeSeries{
 				newTimeSeries([]float64{1}, []int64{timestamp.UnixNano()}, map[string]string{
 					"__name__":      alertMetricName,
@@ -49,19 +55,33 @@ func TestAlertingRule_ToTimeSeries(t *testing.T) {
 					"job":           "foo",
 					"instance":      "bar",
 				}),
+				newTimeSeries([]float64{float64(timestamp.Add(time.Second).Unix())},
+					[]int64{timestamp.UnixNano()},
+					map[string]string{
+						"__name__": alertForStateMetricName,
+						"job":      "foo",
+						"instance": "bar",
+					}),
 			},
 		},
 		{
 			newTestAlertingRule("instant labels override", 0),
-			&notifier.Alert{State: notifier.StateFiring, Labels: map[string]string{
-				alertStateLabel: "foo",
-				"__name__":      "bar",
-			}},
+			&notifier.Alert{State: notifier.StateFiring, ActiveAt: timestamp.Add(time.Second),
+				Labels: map[string]string{
+					alertStateLabel: "foo",
+					"__name__":      "bar",
+				}},
 			[]prompbmarshal.TimeSeries{
 				newTimeSeries([]float64{1}, []int64{timestamp.UnixNano()}, map[string]string{
 					"__name__":      alertMetricName,
 					alertStateLabel: notifier.StateFiring.String(),
 				}),
+				newTimeSeries([]float64{float64(timestamp.Add(time.Second).Unix())},
+					[]int64{timestamp.UnixNano()},
+					map[string]string{
+						"__name__":      alertForStateMetricName,
+						alertStateLabel: "foo",
+					}),
 			},
 		},
 		{
@@ -308,14 +328,17 @@ func TestAlertingRule_Exec(t *testing.T) {
 			fq := &datasource.FakeQuerier{}
 			tc.rule.q = fq
 			tc.rule.GroupID = fakeGroup.ID()
+			ts := time.Now()
 			for i, step := range tc.steps {
 				fq.Reset()
 				fq.Add(step...)
-				if _, err := tc.rule.exec(context.TODO(), time.Now(), 0); err != nil {
+				if _, err := tc.rule.exec(context.TODO(), ts, 0); err != nil {
 					t.Fatalf("unexpected err: %s", err)
 				}
-				// artificial delay between applying steps
-				time.Sleep(defaultStep)
+
+				// shift the execution timestamp before the next iteration
+				ts = ts.Add(defaultStep)
+
 				if _, ok := tc.expAlerts[i]; !ok {
 					continue
 				}
@@ -367,7 +390,7 @@ func TestAlertingRule_ExecRange(t *testing.T) {
 				{Values: []float64{1}, Timestamps: []int64{1}},
 			},
 			[]*notifier.Alert{
-				{State: notifier.StateFiring},
+				{State: notifier.StateFiring, ActiveAt: time.Unix(1, 0)},
 			},
 			nil,
 		},
@@ -378,8 +401,9 @@ func TestAlertingRule_ExecRange(t *testing.T) {
 			},
 			[]*notifier.Alert{
 				{
-					Labels: map[string]string{"name": "foo"},
-					State:  notifier.StateFiring,
+					Labels:   map[string]string{"name": "foo"},
+					State:    notifier.StateFiring,
+					ActiveAt: time.Unix(1, 0),
 				},
 			},
 			nil,
@@ -390,9 +414,9 @@ func TestAlertingRule_ExecRange(t *testing.T) {
 				{Values: []float64{1, 1, 1}, Timestamps: []int64{1e3, 2e3, 3e3}},
 			},
 			[]*notifier.Alert{
-				{State: notifier.StateFiring},
-				{State: notifier.StateFiring},
-				{State: notifier.StateFiring},
+				{State: notifier.StateFiring, ActiveAt: time.Unix(1e3, 0)},
+				{State: notifier.StateFiring, ActiveAt: time.Unix(2e3, 0)},
+				{State: notifier.StateFiring, ActiveAt: time.Unix(3e3, 0)},
 			},
 			nil,
 		},
@@ -460,6 +484,20 @@ func TestAlertingRule_ExecRange(t *testing.T) {
 				For:         time.Second,
 			}},
 		},
+		{
+			newTestAlertingRuleWithEvalInterval("firing=>inactive=>inactive=>firing=>firing", 0, time.Second),
+			[]datasource.Metric{
+				{Values: []float64{1, 1, 1, 1}, Timestamps: []int64{1, 4, 5, 6}},
+			},
+			[]*notifier.Alert{
+				{State: notifier.StateFiring, ActiveAt: time.Unix(1, 0)},
+				// It is expected for ActiveAT to remain the same while rule continues to fire in each iteration
+				{State: notifier.StateFiring, ActiveAt: time.Unix(4, 0)},
+				{State: notifier.StateFiring, ActiveAt: time.Unix(4, 0)},
+				{State: notifier.StateFiring, ActiveAt: time.Unix(4, 0)},
+			},
+			nil,
+		},
 		{
 			newTestAlertingRule("for=>pending=>firing=>pending=>firing=>pending", time.Second),
 			[]datasource.Metric{
@@ -534,21 +572,25 @@ func TestAlertingRule_ExecRange(t *testing.T) {
 				},
 			},
 			[]*notifier.Alert{
-				{State: notifier.StateFiring, Labels: map[string]string{
-					"source": "vm",
-				}},
-				{State: notifier.StateFiring, Labels: map[string]string{
-					"source": "vm",
-				}},
+				{State: notifier.StateFiring, ActiveAt: time.Unix(1, 0),
+					Labels: map[string]string{
+						"source": "vm",
+					}},
+				{State: notifier.StateFiring, ActiveAt: time.Unix(100, 0),
+					Labels: map[string]string{
+						"source": "vm",
+					}},
 				//
-				{State: notifier.StateFiring, Labels: map[string]string{
-					"foo":    "bar",
-					"source": "vm",
-				}},
-				{State: notifier.StateFiring, Labels: map[string]string{
-					"foo":    "bar",
-					"source": "vm",
-				}},
+				{State: notifier.StateFiring, ActiveAt: time.Unix(1, 0),
+					Labels: map[string]string{
+						"foo":    "bar",
+						"source": "vm",
+					}},
+				{State: notifier.StateFiring, ActiveAt: time.Unix(5, 0),
+					Labels: map[string]string{
+						"foo":    "bar",
+						"source": "vm",
+					}},
 			},
 			nil,
 		},
@@ -1095,6 +1137,12 @@ func newTestAlertingRule(name string, waitFor time.Duration) *AlertingRule {
 	return &rule
 }
 
+func newTestAlertingRuleWithEvalInterval(name string, waitFor, evalInterval time.Duration) *AlertingRule {
+	rule := newTestAlertingRule(name, waitFor)
+	rule.EvalInterval = evalInterval
+	return rule
+}
+
 func newTestAlertingRuleWithKeepFiring(name string, waitFor, keepFiringFor time.Duration) *AlertingRule {
 	rule := newTestAlertingRule(name, waitFor)
 	rule.KeepFiringFor = keepFiringFor

app/vmalert/web.go

@@ -12,11 +12,14 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/notifier"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/rule"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/tpl"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/procutil"
 )
 
+var reloadAuthKey = flagutil.NewPassword("reloadAuthKey", "Auth key for /-/reload http endpoint. It must be passed as authKey=...")
+
 var (
 	apiLinks = [][2]string{
 		// api links are relative since they can be used by external clients,
@@ -151,6 +154,9 @@ func (rh *requestHandler) handler(w http.ResponseWriter, r *http.Request) bool {
 		w.Write(data)
 		return true
 	case "/-/reload":
+		if !httpserver.CheckAuthFlag(w, r, reloadAuthKey.Get(), "reloadAuthKey") {
+			return true
+		}
 		logger.Infof("api config reload was called, sending sighup")
 		procutil.SelfSIGHUP()
 		w.WriteHeader(http.StatusOK)

app/vmauth/auth_config.go

@@ -9,6 +9,7 @@ import (
 	"net/url"
 	"os"
 	"regexp"
+	"sort"
 	"strconv"
 	"strings"
 	"sync"
@@ -16,12 +17,13 @@ import (
 	"time"
 
 	"github.com/VictoriaMetrics/metrics"
+	"github.com/cespare/xxhash/v2"
 	"gopkg.in/yaml.v2"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/envtemplate"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fasttime"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fs"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fs/fscore"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/procutil"
 )
@@ -60,12 +62,15 @@ type UserInfo struct {
 	TLSInsecureSkipVerify  *bool       `yaml:"tls_insecure_skip_verify,omitempty"`
 	TLSCAFile              string      `yaml:"tls_ca_file,omitempty"`
 
+	MetricLabels map[string]string `yaml:"metric_labels,omitempty"`
+
 	concurrencyLimitCh      chan struct{}
 	concurrencyLimitReached *metrics.Counter
 
 	httpTransport *http.Transport
 
 	requests         *metrics.Counter
+	backendErrors    *metrics.Counter
 	requestsDuration *metrics.Summary
 }
 
@@ -266,8 +271,10 @@ func (up *URLPrefix) getLeastLoadedBackendURL() *backendURL {
 		if bu.isBroken() {
 			continue
 		}
-		if atomic.CompareAndSwapInt32(&bu.concurrentRequests, 0, 1) {
+		if atomic.LoadInt32(&bu.concurrentRequests) == 0 {
 			// Fast path - return the backend with zero concurrently executed requests.
+			// Do not use atomic.CompareAndSwapInt32(), since it is much slower on systems with many CPU cores.
+			atomic.AddInt32(&bu.concurrentRequests, 1)
 			return bu
 		}
 	}
@@ -462,17 +469,19 @@ func authConfigReloader(sighupCh <-chan os.Signal) {
 // authConfigData needs to be updated each time authConfig is updated.
 var authConfigData atomic.Pointer[[]byte]
 
-var authConfig atomic.Pointer[AuthConfig]
-var authUsers atomic.Pointer[map[string]*UserInfo]
-var authConfigWG sync.WaitGroup
-var stopCh chan struct{}
+var (
+	authConfig   atomic.Pointer[AuthConfig]
+	authUsers    atomic.Pointer[map[string]*UserInfo]
+	authConfigWG sync.WaitGroup
+	stopCh       chan struct{}
+)
 
 // loadAuthConfig loads and applies the config from *authConfigPath.
 // It returns bool value to identify if new config was applied.
 // The config can be not applied if there is a parsing error
 // or if there are no changes to the current authConfig.
 func loadAuthConfig() (bool, error) {
-	data, err := fs.ReadFileOrHTTP(*authConfigPath)
+	data, err := fscore.ReadFileOrHTTP(*authConfigPath)
 	if err != nil {
 		return false, fmt.Errorf("failed to read -auth.config=%q: %w", *authConfigPath, err)
 	}
@@ -527,16 +536,23 @@ func parseAuthConfig(data []byte) (*AuthConfig, error) {
 		if err := ui.initURLs(); err != nil {
 			return nil, err
 		}
-		ui.requests = metrics.GetOrCreateCounter(`vmauth_unauthorized_user_requests_total`)
-		ui.requestsDuration = metrics.GetOrCreateSummary(`vmauth_unauthorized_user_request_duration_seconds`)
+
+		metricLabels, err := ui.getMetricLabels()
+		if err != nil {
+			return nil, fmt.Errorf("cannot parse metric_labels for unauthorized_user: %w", err)
+		}
+		ui.requests = metrics.GetOrCreateCounter(`vmauth_unauthorized_user_requests_total` + metricLabels)
+		ui.backendErrors = metrics.GetOrCreateCounter(`vmauth_unauthorized_user_request_backend_errors_total` + metricLabels)
+		ui.requestsDuration = metrics.GetOrCreateSummary(`vmauth_unauthorized_user_request_duration_seconds` + metricLabels)
 		ui.concurrencyLimitCh = make(chan struct{}, ui.getMaxConcurrentRequests())
-		ui.concurrencyLimitReached = metrics.GetOrCreateCounter(`vmauth_unauthorized_user_concurrent_requests_limit_reached_total`)
-		_ = metrics.GetOrCreateGauge(`vmauth_unauthorized_user_concurrent_requests_capacity`, func() float64 {
+		ui.concurrencyLimitReached = metrics.GetOrCreateCounter(`vmauth_unauthorized_user_concurrent_requests_limit_reached_total` + metricLabels)
+		_ = metrics.GetOrCreateGauge(`vmauth_unauthorized_user_concurrent_requests_capacity`+metricLabels, func() float64 {
 			return float64(cap(ui.concurrencyLimitCh))
 		})
-		_ = metrics.GetOrCreateGauge(`vmauth_unauthorized_user_concurrent_requests_current`, func() float64 {
+		_ = metrics.GetOrCreateGauge(`vmauth_unauthorized_user_concurrent_requests_current`+metricLabels, func() float64 {
 			return float64(len(ui.concurrencyLimitCh))
 		})
+
 		tr, err := getTransport(ui.TLSInsecureSkipVerify, ui.TLSCAFile)
 		if err != nil {
 			return nil, fmt.Errorf("cannot initialize HTTP transport: %w", err)
@@ -572,25 +588,24 @@ func parseAuthConfigUsers(ac *AuthConfig) (map[string]*UserInfo, error) {
 			return nil, err
 		}
 
-		name := ui.name()
-		if ui.BearerToken != "" {
-			if ui.Password != "" {
-				return nil, fmt.Errorf("password shouldn't be set for bearer_token %q", ui.BearerToken)
-			}
-			ui.requests = metrics.GetOrCreateCounter(fmt.Sprintf(`vmauth_user_requests_total{username=%q}`, name))
-			ui.requestsDuration = metrics.GetOrCreateSummary(fmt.Sprintf(`vmauth_user_request_duration_seconds{username=%q}`, name))
+		if ui.BearerToken != "" && ui.Password != "" {
+			return nil, fmt.Errorf("password shouldn't be set for bearer_token %q", ui.BearerToken)
 		}
-		if ui.Username != "" {
-			ui.requests = metrics.GetOrCreateCounter(fmt.Sprintf(`vmauth_user_requests_total{username=%q}`, name))
-			ui.requestsDuration = metrics.GetOrCreateSummary(fmt.Sprintf(`vmauth_user_request_duration_seconds{username=%q}`, name))
+
+		metricLabels, err := ui.getMetricLabels()
+		if err != nil {
+			return nil, fmt.Errorf("cannot parse metric_labels: %w", err)
 		}
+		ui.requests = metrics.GetOrCreateCounter(`vmauth_user_requests_total` + metricLabels)
+		ui.backendErrors = metrics.GetOrCreateCounter(`vmauth_user_request_backend_errors_total` + metricLabels)
+		ui.requestsDuration = metrics.GetOrCreateSummary(`vmauth_user_request_duration_seconds` + metricLabels)
 		mcr := ui.getMaxConcurrentRequests()
 		ui.concurrencyLimitCh = make(chan struct{}, mcr)
-		ui.concurrencyLimitReached = metrics.GetOrCreateCounter(fmt.Sprintf(`vmauth_user_concurrent_requests_limit_reached_total{username=%q}`, name))
-		_ = metrics.GetOrCreateGauge(fmt.Sprintf(`vmauth_user_concurrent_requests_capacity{username=%q}`, name), func() float64 {
+		ui.concurrencyLimitReached = metrics.GetOrCreateCounter(`vmauth_user_concurrent_requests_limit_reached_total` + metricLabels)
+		_ = metrics.GetOrCreateGauge(`vmauth_user_concurrent_requests_capacity`+metricLabels, func() float64 {
 			return float64(cap(ui.concurrencyLimitCh))
 		})
-		_ = metrics.GetOrCreateGauge(fmt.Sprintf(`vmauth_user_concurrent_requests_current{username=%q}`, name), func() float64 {
+		_ = metrics.GetOrCreateGauge(`vmauth_user_concurrent_requests_current`+metricLabels, func() float64 {
 			return float64(len(ui.concurrencyLimitCh))
 		})
 
@@ -606,6 +621,29 @@ func parseAuthConfigUsers(ac *AuthConfig) (map[string]*UserInfo, error) {
 	return byAuthToken, nil
 }
 
+var labelNameRegexp = regexp.MustCompile("^[a-zA-Z_:.][a-zA-Z0-9_:.]*$")
+
+func (ui *UserInfo) getMetricLabels() (string, error) {
+	name := ui.name()
+	if len(name) == 0 && len(ui.MetricLabels) == 0 {
+		// fast path
+		return "", nil
+	}
+	labels := make([]string, 0, len(ui.MetricLabels)+1)
+	if len(name) > 0 {
+		labels = append(labels, fmt.Sprintf(`username=%q`, name))
+	}
+	for k, v := range ui.MetricLabels {
+		if !labelNameRegexp.MatchString(k) {
+			return "", fmt.Errorf("incorrect label name=%q, it must match regex=%q for user=%q", k, labelNameRegexp, name)
+		}
+		labels = append(labels, fmt.Sprintf(`%s=%q`, k, v))
+	}
+	sort.Strings(labels)
+	labelsStr := "{" + strings.Join(labels, ",") + "}"
+	return labelsStr, nil
+}
+
 func (ui *UserInfo) initURLs() error {
 	retryStatusCodes := defaultRetryStatusCodes.Values()
 	loadBalancingPolicy := *defaultLoadBalancingPolicy
@@ -676,7 +714,8 @@ func (ui *UserInfo) name() string {
 		return ui.Username
 	}
 	if ui.BearerToken != "" {
-		return "bearer_token"
+		h := xxhash.Sum64([]byte(ui.BearerToken))
+		return fmt.Sprintf("bearer_token:hash:%016X", h)
 	}
 	return ""
 }

app/vmauth/auth_config_test.go

@@ -229,6 +229,14 @@ users:
     url_prefix: http://foobar
     headers:
       aaa: bbb
+`)
+	// Invalid metric label name
+	f(`
+users:
+- username: foo
+  url_prefix: http://foo.bar
+  metric_labels:
+    not-prometheus-compatible: value
 `)
 }
 
@@ -489,7 +497,41 @@ users:
 			}),
 		},
 	})
-
+	// With metric_labels
+	f(`
+users:
+- username: foo-same
+  password: baz
+  url_prefix: http://foo
+  metric_labels:
+    dc: eu
+    team: dev
+- username: foo-same
+  password: bar
+  url_prefix: https://bar/x///
+  metric_labels:
+    backend_env: test
+    team: accounting
+`, map[string]*UserInfo{
+		getAuthToken("", "foo-same", "baz"): {
+			Username:  "foo-same",
+			Password:  "baz",
+			URLPrefix: mustParseURL("http://foo"),
+			MetricLabels: map[string]string{
+				"dc":   "eu",
+				"team": "dev",
+			},
+		},
+		getAuthToken("", "foo-same", "bar"): {
+			Username:  "foo-same",
+			Password:  "bar",
+			URLPrefix: mustParseURL("https://bar/x"),
+			MetricLabels: map[string]string{
+				"backend_env": "test",
+				"team":        "accounting",
+			},
+		},
+	})
 }
 
 func TestParseAuthConfigPassesTLSVerificationConfig(t *testing.T) {
@@ -526,6 +568,86 @@ unauthorized_user:
 	}
 }
 
+func TestUserInfoGetMetricLabels(t *testing.T) {
+	t.Run("empty-labels", func(t *testing.T) {
+		ui := &UserInfo{
+			Username: "user1",
+		}
+		labels, err := ui.getMetricLabels()
+		if err != nil {
+			t.Fatalf("unexpected error: %s", err)
+		}
+		labelsExpected := `{username="user1"}`
+		if labels != labelsExpected {
+			t.Fatalf("unexpected labels; got %s; want %s", labels, labelsExpected)
+		}
+	})
+	t.Run("non-empty-username", func(t *testing.T) {
+		ui := &UserInfo{
+			Username: "user1",
+			MetricLabels: map[string]string{
+				"env":        "prod",
+				"datacenter": "dc1",
+			},
+		}
+		labels, err := ui.getMetricLabels()
+		if err != nil {
+			t.Fatalf("unexpected error: %s", err)
+		}
+		labelsExpected := `{datacenter="dc1",env="prod",username="user1"}`
+		if labels != labelsExpected {
+			t.Fatalf("unexpected labels; got %s; want %s", labels, labelsExpected)
+		}
+	})
+	t.Run("non-empty-name", func(t *testing.T) {
+		ui := &UserInfo{
+			Name:        "user1",
+			BearerToken: "abc",
+			MetricLabels: map[string]string{
+				"env":        "prod",
+				"datacenter": "dc1",
+			},
+		}
+		labels, err := ui.getMetricLabels()
+		if err != nil {
+			t.Fatalf("unexpected error: %s", err)
+		}
+		labelsExpected := `{datacenter="dc1",env="prod",username="user1"}`
+		if labels != labelsExpected {
+			t.Fatalf("unexpected labels; got %s; want %s", labels, labelsExpected)
+		}
+	})
+	t.Run("non-empty-bearer-token", func(t *testing.T) {
+		ui := &UserInfo{
+			BearerToken: "abc",
+			MetricLabels: map[string]string{
+				"env":        "prod",
+				"datacenter": "dc1",
+			},
+		}
+		labels, err := ui.getMetricLabels()
+		if err != nil {
+			t.Fatalf("unexpected error: %s", err)
+		}
+		labelsExpected := `{datacenter="dc1",env="prod",username="bearer_token:hash:44BC2CF5AD770999"}`
+		if labels != labelsExpected {
+			t.Fatalf("unexpected labels; got %s; want %s", labels, labelsExpected)
+		}
+	})
+	t.Run("invalid-label", func(t *testing.T) {
+		ui := &UserInfo{
+			Username: "foo",
+			MetricLabels: map[string]string{
+				",{": "aaaa",
+			},
+		}
+		_, err := ui.getMetricLabels()
+		if err == nil {
+			t.Fatalf("expecting non-nil error")
+		}
+	})
+}
+
 func isSetBool(boolP *bool, expectedValue bool) bool {
 	if boolP == nil {
 		return false

app/vmauth/example_config.yml

@@ -10,6 +10,11 @@ users:
 - bearer_token: "XXXX"
   url_prefix: "http://localhost:8428"
 
+  # Adds labels to the exported metrics for given user section
+  # label name must be prometheus compatible and match regex: `^[a-zA-Z_:.][a-zA-Z0-9_:.]*$`
+  metric_labels:
+    backend_dc: eu
+    access_team: dev
   # Requests with the 'Authorization: Bearer YYY' header are proxied to http://localhost:8428 ,
   # The `X-Scope-OrgID: foobar` http header is added to every proxied request.
   # The `X-Server-Hostname:` http header is removed from the proxied response.

app/vmauth/main.go

@@ -24,7 +24,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/encoding"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/envflag"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fs"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fs/fscore"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/netutil"
@@ -45,7 +45,7 @@ var (
 	maxConcurrentPerUserRequests = flag.Int("maxConcurrentPerUserRequests", 300, "The maximum number of concurrent requests vmauth can process per each configured user. "+
 		"Other requests are rejected with '429 Too Many Requests' http status code. See also -maxConcurrentRequests command-line option and max_concurrent_requests option "+
 		"in per-user config")
-	reloadAuthKey        = flag.String("reloadAuthKey", "", "Auth key for /-/reload http endpoint. It must be passed as authKey=...")
+	reloadAuthKey        = flagutil.NewPassword("reloadAuthKey", "Auth key for /-/reload http endpoint. It must be passed as authKey=...")
 	logInvalidAuthTokens = flag.Bool("logInvalidAuthTokens", false, "Whether to log requests with invalid auth tokens. "+
 		`Such requests are always counted at vmauth_http_request_errors_total{reason="invalid_auth_token"} metric, which is exposed at /metrics page`)
 	failTimeout               = flag.Duration("failTimeout", 3*time.Second, "Sets a delay period for load balancing to skip a malfunctioning backend")
@@ -89,7 +89,7 @@ func main() {
 func requestHandler(w http.ResponseWriter, r *http.Request) bool {
 	switch r.URL.Path {
 	case "/-/reload":
-		if !httpserver.CheckAuthFlag(w, r, *reloadAuthKey, "reloadAuthKey") {
+		if !httpserver.CheckAuthFlag(w, r, reloadAuthKey.Get(), "reloadAuthKey") {
 			return true
 		}
 		configReloadRequests.Inc()
@@ -150,12 +150,20 @@ func processUserRequest(w http.ResponseWriter, r *http.Request, ui *UserInfo) {
 		if err := ui.beginConcurrencyLimit(); err != nil {
 			handleConcurrencyLimitError(w, r, err)
 			<-concurrencyLimitCh
+
+			// Requests failed because of concurrency limit must be counted as errors,
+			// since this usually means the backend cannot keep up with the current load.
+			ui.backendErrors.Inc()
 			return
 		}
 	default:
 		concurrentRequestsLimitReached.Inc()
 		err := fmt.Errorf("cannot serve more than -maxConcurrentRequests=%d concurrent requests", cap(concurrencyLimitCh))
 		handleConcurrencyLimitError(w, r, err)
+
+		// Requests failed because of concurrency limit must be counted as errors,
+		// since this usually means the backend cannot keep up with the current load.
+		ui.backendErrors.Inc()
 		return
 	}
 	processRequest(w, r, ui)
@@ -201,7 +209,7 @@ func processRequest(w http.ResponseWriter, r *http.Request, ui *UserInfo) {
 		} else { // Update path for regular routes.
 			targetURL = mergeURLs(targetURL, u, up.dropSrcPathPrefixParts)
 		}
-		ok := tryProcessingRequest(w, r, targetURL, hc, up.retryStatusCodes, ui.httpTransport)
+		ok := tryProcessingRequest(w, r, targetURL, hc, up.retryStatusCodes, ui)
 		bu.put()
 		if ok {
 			return
@@ -213,15 +221,16 @@ func processRequest(w http.ResponseWriter, r *http.Request, ui *UserInfo) {
 		StatusCode: http.StatusServiceUnavailable,
 	}
 	httpserver.Errorf(w, r, "%s", err)
+	ui.backendErrors.Inc()
 }
 
-func tryProcessingRequest(w http.ResponseWriter, r *http.Request, targetURL *url.URL, hc HeadersConf, retryStatusCodes []int, transport *http.Transport) bool {
+func tryProcessingRequest(w http.ResponseWriter, r *http.Request, targetURL *url.URL, hc HeadersConf, retryStatusCodes []int, ui *UserInfo) bool {
 	// This code has been copied from net/http/httputil/reverseproxy.go
 	req := sanitizeRequestHeaders(r)
 	req.URL = targetURL
 	req.Host = targetURL.Host
 	updateHeadersByConfig(req.Header, hc.RequestHeaders)
-	res, err := transport.RoundTrip(req)
+	res, err := ui.httpTransport.RoundTrip(req)
 	rtb, rtbOK := req.Body.(*readTrackingBody)
 	if err != nil {
 		if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
@@ -229,15 +238,20 @@ func tryProcessingRequest(w http.ResponseWriter, r *http.Request, targetURL *url
 			remoteAddr := httpserver.GetQuotedRemoteAddr(r)
 			requestURI := httpserver.GetRequestURI(r)
 			logger.Warnf("remoteAddr: %s; requestURI: %s; error when proxying response body from %s: %s", remoteAddr, requestURI, targetURL, err)
+			if errors.Is(err, context.DeadlineExceeded) {
+				// Timed out request must be counted as errors, since this usually means that the backend is slow.
+				ui.backendErrors.Inc()
+			}
 			return true
 		}
 		if !rtbOK || !rtb.canRetry() {
 			// Request body cannot be re-sent to another backend. Return the error to the client then.
 			err = &httpserver.ErrorWithStatusCode{
-				Err:        fmt.Errorf("cannot proxy the request to %q: %w", targetURL, err),
+				Err:        fmt.Errorf("cannot proxy the request to %s: %w", targetURL, err),
 				StatusCode: http.StatusServiceUnavailable,
 			}
 			httpserver.Errorf(w, r, "%s", err)
+			ui.backendErrors.Inc()
 			return true
 		}
 		// Retry the request if its body wasn't read yet. This usually means that the backend isn't reachable.
@@ -247,7 +261,20 @@ func tryProcessingRequest(w http.ResponseWriter, r *http.Request, targetURL *url
 		logger.Warnf("remoteAddr: %s; requestURI: %s; retrying the request to %s because of response error: %s", remoteAddr, req.URL, targetURL, err)
 		return false
 	}
-	if (rtbOK && rtb.canRetry()) && hasInt(retryStatusCodes, res.StatusCode) {
+	if hasInt(retryStatusCodes, res.StatusCode) {
+		_ = res.Body.Close()
+		if !rtbOK || !rtb.canRetry() {
+			// If we get an error from the retry_status_codes list, but cannot execute retry,
+			// we consider such a request an error as well.
+			err := &httpserver.ErrorWithStatusCode{
+				Err: fmt.Errorf("got response status code=%d from %s, but cannot retry the request on another backend, because the request has been already consumed",
+					res.StatusCode, targetURL),
+				StatusCode: http.StatusServiceUnavailable,
+			}
+			httpserver.Errorf(w, r, "%s", err)
+			ui.backendErrors.Inc()
+			return true
+		}
 		// Retry requests at other backends if it matches retryStatusCodes.
 		// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4893
 		remoteAddr := httpserver.GetQuotedRemoteAddr(r)
@@ -266,6 +293,7 @@ func tryProcessingRequest(w http.ResponseWriter, r *http.Request, targetURL *url
 	copyBuf.B = bytesutil.ResizeNoCopyNoOverallocate(copyBuf.B, 16*1024)
 	_, err = io.CopyBuffer(w, res.Body, copyBuf.B)
 	copyBufPool.Put(copyBuf)
+	_ = res.Body.Close()
 	if err != nil && !netutil.IsTrivialNetworkError(err) {
 		remoteAddr := httpserver.GetQuotedRemoteAddr(r)
 		requestURI := httpserver.GetRequestURI(r)
@@ -393,8 +421,10 @@ func getTransport(insecureSkipVerifyP *bool, caFile string) (*http.Transport, er
 	return tr, nil
 }
 
-var transportMap = make(map[string]*http.Transport)
-var transportMapLock sync.Mutex
+var (
+	transportMap     = make(map[string]*http.Transport)
+	transportMapLock sync.Mutex
+)
 
 func appendTransportKey(dst []byte, insecureSkipVerify bool, caFile string) []byte {
 	dst = encoding.MarshalBool(dst, insecureSkipVerify)
@@ -422,7 +452,7 @@ func newTransport(insecureSkipVerify bool, caFile string) (*http.Transport, erro
 		tlsCfg.ClientSessionCache = tls.NewLRUClientSessionCache(0)
 		tlsCfg.InsecureSkipVerify = insecureSkipVerify
 		if caFile != "" {
-			data, err := fs.ReadFileOrHTTP(caFile)
+			data, err := fscore.ReadFileOrHTTP(caFile)
 			if err != nil {
 				return nil, fmt.Errorf("cannot read tls_ca_file: %w", err)
 			}

app/vmctl/backoff/backoff_test.go

@@ -15,7 +15,6 @@ func TestRetry_Do(t *testing.T) {
 		backoffFactor      float64
 		backoffMinDuration time.Duration
 		retryableFunc      retryableFunc
-		ctx                context.Context
 		cancelTimeout      time.Duration
 		want               uint64
 		wantErr            bool
@@ -25,7 +24,6 @@ func TestRetry_Do(t *testing.T) {
 			retryableFunc: func() error {
 				return ErrBadRequest
 			},
-			ctx:     context.Background(),
 			want:    0,
 			wantErr: true,
 		},
@@ -35,7 +33,6 @@ func TestRetry_Do(t *testing.T) {
 				time.Sleep(time.Millisecond * 100)
 				return nil
 			},
-			ctx:     context.Background(),
 			want:    0,
 			wantErr: true,
 		},
@@ -58,7 +55,6 @@ func TestRetry_Do(t *testing.T) {
 				}
 				return nil
 			},
-			ctx:     context.Background(),
 			want:    1,
 			wantErr: false,
 		},
@@ -75,7 +71,6 @@ func TestRetry_Do(t *testing.T) {
 				}
 				return nil
 			},
-			ctx:     context.Background(),
 			want:    5,
 			wantErr: true,
 		},
@@ -85,14 +80,8 @@ func TestRetry_Do(t *testing.T) {
 			backoffFactor:      1.7,
 			backoffMinDuration: time.Millisecond * 10,
 			retryableFunc: func() error {
-				t := time.NewTicker(time.Millisecond * 5)
-				defer t.Stop()
-				for range t.C {
-					return fmt.Errorf("got some error")
-				}
-				return nil
+				return fmt.Errorf("got some error")
 			},
-			ctx:           context.Background(),
 			cancelTimeout: time.Millisecond * 40,
 			want:          3,
 			wantErr:       true,
@@ -101,12 +90,13 @@ func TestRetry_Do(t *testing.T) {
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
 			r := &Backoff{retries: tt.backoffRetries, factor: tt.backoffFactor, minDuration: tt.backoffMinDuration}
+			ctx := context.Background()
 			if tt.cancelTimeout != 0 {
-				newCtx, cancelFn := context.WithTimeout(tt.ctx, tt.cancelTimeout)
-				tt.ctx = newCtx
+				newCtx, cancelFn := context.WithTimeout(context.Background(), tt.cancelTimeout)
+				ctx = newCtx
 				defer cancelFn()
 			}
-			got, err := r.Retry(tt.ctx, tt.retryableFunc)
+			got, err := r.Retry(ctx, tt.retryableFunc)
 			if (err != nil) != tt.wantErr {
 				t.Errorf("Retry() error = %v, wantErr %v", err, tt.wantErr)
 				return

app/vminsert/main.go

@@ -29,6 +29,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vminsert/vmimport"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/auth"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/influxutils"
 	graphiteserver "github.com/VictoriaMetrics/VictoriaMetrics/lib/ingestserver/graphite"
@@ -62,7 +63,8 @@ var (
 		"See also -opentsdbHTTPListenAddr.useProxyProtocol")
 	opentsdbHTTPUseProxyProtocol = flag.Bool("opentsdbHTTPListenAddr.useProxyProtocol", false, "Whether to use proxy protocol for connections accepted "+
 		"at -opentsdbHTTPListenAddr . See https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt")
-	configAuthKey          = flag.String("configAuthKey", "", "Authorization key for accessing /config page. It must be passed via authKey query arg")
+	configAuthKey          = flagutil.NewPassword("configAuthKey", "Authorization key for accessing /config page. It must be passed via authKey query arg")
+	reloadAuthKey          = flagutil.NewPassword("reloadAuthKey", "Auth key for /-/reload http endpoint. It must be passed as authKey=...")
 	maxLabelsPerTimeseries = flag.Int("maxLabelsPerTimeseries", 30, "The maximum number of labels accepted per time series. Superfluous labels are dropped. In this case the vm_metrics_with_dropped_labels_total metric at /metrics page is incremented")
 	maxLabelValueLen       = flag.Int("maxLabelValueLen", 16*1024, "The maximum length of label values in the accepted time series. Longer label values are truncated. In this case the vm_too_long_label_values_total metric at /metrics page is incremented")
 )
@@ -315,7 +317,7 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 		}
 		return true
 	case "/prometheus/config", "/config":
-		if !httpserver.CheckAuthFlag(w, r, *configAuthKey, "configAuthKey") {
+		if !httpserver.CheckAuthFlag(w, r, configAuthKey.Get(), "configAuthKey") {
 			return true
 		}
 		promscrapeConfigRequests.Inc()
@@ -324,7 +326,7 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 		return true
 	case "/prometheus/api/v1/status/config", "/api/v1/status/config":
 		// See https://prometheus.io/docs/prometheus/latest/querying/api/#config
-		if !httpserver.CheckAuthFlag(w, r, *configAuthKey, "configAuthKey") {
+		if !httpserver.CheckAuthFlag(w, r, configAuthKey.Get(), "configAuthKey") {
 			return true
 		}
 		promscrapeStatusConfigRequests.Inc()
@@ -334,6 +336,9 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 		fmt.Fprintf(w, `{"status":"success","data":{"yaml":%q}}`, bb.B)
 		return true
 	case "/prometheus/-/reload", "/-/reload":
+		if !httpserver.CheckAuthFlag(w, r, reloadAuthKey.Get(), "reloadAuthKey") {
+			return true
+		}
 		promscrapeConfigReloadRequests.Inc()
 		procutil.SelfSIGHUP()
 		w.WriteHeader(http.StatusNoContent)

app/vmselect/main.go

@@ -18,6 +18,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/searchutils"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/cgroup"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fs"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/httputils"
@@ -29,13 +30,13 @@ import (
 )
 
 var (
-	deleteAuthKey         = flag.String("deleteAuthKey", "", "authKey for metrics' deletion via /api/v1/admin/tsdb/delete_series and /tags/delSeries")
+	deleteAuthKey         = flagutil.NewPassword("deleteAuthKey", "authKey for metrics' deletion via /api/v1/admin/tsdb/delete_series and /tags/delSeries")
 	maxConcurrentRequests = flag.Int("search.maxConcurrentRequests", getDefaultMaxConcurrentRequests(), "The maximum number of concurrent search requests. "+
 		"It shouldn't be high, since a single request can saturate all the CPU cores, while many concurrently executed requests may require high amounts of memory. "+
 		"See also -search.maxQueueDuration and -search.maxMemoryPerQuery")
 	maxQueueDuration = flag.Duration("search.maxQueueDuration", 10*time.Second, "The maximum time the request waits for execution when -search.maxConcurrentRequests "+
 		"limit is reached; see also -search.maxQueryDuration")
-	resetCacheAuthKey    = flag.String("search.resetCacheAuthKey", "", "Optional authKey for resetting rollup cache via /internal/resetRollupResultCache call")
+	resetCacheAuthKey    = flagutil.NewPassword("search.resetCacheAuthKey", "Optional authKey for resetting rollup cache via /internal/resetRollupResultCache call")
 	logSlowQueryDuration = flag.Duration("search.logSlowQueryDuration", 5*time.Second, "Log queries with execution time exceeding this value. Zero disables slow query logging. "+
 		"See also -search.logQueryMemoryUsage")
 	vmalertProxyURL = flag.String("vmalert.proxyURL", "", "Optional URL for proxying requests to vmalert. For example, if -vmalert.proxyURL=http://vmalert:8880 , then alerting API requests such as /api/v1/rules from Grafana will be proxied to http://vmalert:8880/api/v1/rules")
@@ -148,8 +149,9 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 					"are executed. Possible solutions: to reduce query load; to add more compute resources to the server; "+
 					"to increase -search.maxQueueDuration=%s; to increase -search.maxQueryDuration; to increase -search.maxConcurrentRequests",
 					d.Seconds(), *maxConcurrentRequests, maxQueueDuration),
-				StatusCode: http.StatusServiceUnavailable,
+				StatusCode: http.StatusTooManyRequests,
 			}
+			w.Header().Add("Retry-After", "10")
 			httpserver.Errorf(w, r, "%s", err)
 			return true
 		}
@@ -170,7 +172,7 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 	}
 
 	if path == "/internal/resetRollupResultCache" {
-		if !httpserver.CheckAuthFlag(w, r, *resetCacheAuthKey, "resetCacheAuthKey") {
+		if !httpserver.CheckAuthFlag(w, r, resetCacheAuthKey.Get(), "resetCacheAuthKey") {
 			return true
 		}
 		promql.ResetRollupResultCache()
@@ -367,7 +369,7 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 		}
 		return true
 	case "/tags/delSeries":
-		if !httpserver.CheckAuthFlag(w, r, *deleteAuthKey, "deleteAuthKey") {
+		if !httpserver.CheckAuthFlag(w, r, deleteAuthKey.Get(), "deleteAuthKey") {
 			return true
 		}
 		graphiteTagsDelSeriesRequests.Inc()
@@ -386,7 +388,7 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 		}
 		return true
 	case "/api/v1/admin/tsdb/delete_series":
-		if !httpserver.CheckAuthFlag(w, r, *deleteAuthKey, "deleteAuthKey") {
+		if !httpserver.CheckAuthFlag(w, r, deleteAuthKey.Get(), "deleteAuthKey") {
 			return true
 		}
 		deleteRequests.Inc()
@@ -425,6 +427,14 @@ func handleStaticAndSimpleRequests(w http.ResponseWriter, r *http.Request, path
 		}
 		return true
 	}
+	if path == "/vmui/timezone" {
+		httpserver.EnableCORS(w, r)
+		if err := handleVMUITimezone(w); err != nil {
+			httpserver.Errorf(w, r, "%s", err)
+			return true
+		}
+		return true
+	}
 	if strings.HasPrefix(path, "/vmui/") {
 		if strings.HasPrefix(path, "/vmui/static/") {
 			// Allow clients caching static contents for long period of time, since it shouldn't change over time.

app/vmselect/netstorage/netstorage.go

@@ -5,10 +5,12 @@ import (
 	"errors"
 	"flag"
 	"fmt"
+	"reflect"
 	"sort"
 	"sync"
 	"sync/atomic"
 	"time"
+	"unsafe"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/searchutils"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage"
@@ -88,30 +90,6 @@ type timeseriesWork struct {
 	rowsProcessed int
 }
 
-func (tsw *timeseriesWork) reset() {
-	tsw.mustStop = nil
-	tsw.rss = nil
-	tsw.pts = nil
-	tsw.f = nil
-	tsw.err = nil
-	tsw.rowsProcessed = 0
-}
-
-func getTimeseriesWork() *timeseriesWork {
-	v := tswPool.Get()
-	if v == nil {
-		v = &timeseriesWork{}
-	}
-	return v.(*timeseriesWork)
-}
-
-func putTimeseriesWork(tsw *timeseriesWork) {
-	tsw.reset()
-	tswPool.Put(tsw)
-}
-
-var tswPool sync.Pool
-
 func (tsw *timeseriesWork) do(r *Result, workerID uint) error {
 	if atomic.LoadUint32(tsw.mustStop) != 0 {
 		return nil
@@ -270,22 +248,20 @@ func (rss *Results) runParallel(qt *querytracer.Tracer, f func(rs *Result, worke
 	maxWorkers := MaxWorkers()
 	if maxWorkers == 1 || tswsLen == 1 {
 		// It is faster to process time series in the current goroutine.
-		tsw := getTimeseriesWork()
+		var tsw timeseriesWork
 		tmpResult := getTmpResult()
 		rowsProcessedTotal := 0
 		var err error
 		for i := range rss.packedTimeseries {
-			initTimeseriesWork(tsw, &rss.packedTimeseries[i])
+			initTimeseriesWork(&tsw, &rss.packedTimeseries[i])
 			err = tsw.do(&tmpResult.rs, 0)
 			rowsReadPerSeries.Update(float64(tsw.rowsProcessed))
 			rowsProcessedTotal += tsw.rowsProcessed
 			if err != nil {
 				break
 			}
-			tsw.reset()
 		}
 		putTmpResult(tmpResult)
-		putTimeseriesWork(tsw)
 
 		return rowsProcessedTotal, err
 	}
@@ -295,11 +271,9 @@ func (rss *Results) runParallel(qt *querytracer.Tracer, f func(rs *Result, worke
 	// which reduces the scalability on systems with many CPU cores.
 
 	// Prepare the work for workers.
-	tsws := make([]*timeseriesWork, len(rss.packedTimeseries))
+	tsws := make([]timeseriesWork, len(rss.packedTimeseries))
 	for i := range rss.packedTimeseries {
-		tsw := getTimeseriesWork()
-		initTimeseriesWork(tsw, &rss.packedTimeseries[i])
-		tsws[i] = tsw
+		initTimeseriesWork(&tsws[i], &rss.packedTimeseries[i])
 	}
 
 	// Prepare worker channels.
@@ -314,9 +288,9 @@ func (rss *Results) runParallel(qt *querytracer.Tracer, f func(rs *Result, worke
 	}
 
 	// Spread work among workers.
-	for i, tsw := range tsws {
+	for i := range tsws {
 		idx := i % len(workChs)
-		workChs[idx] <- tsw
+		workChs[idx] <- &tsws[i]
 	}
 	// Mark worker channels as closed.
 	for _, workCh := range workChs {
@@ -339,14 +313,14 @@ func (rss *Results) runParallel(qt *querytracer.Tracer, f func(rs *Result, worke
 	// Collect results.
 	var firstErr error
 	rowsProcessedTotal := 0
-	for _, tsw := range tsws {
+	for i := range tsws {
+		tsw := &tsws[i]
 		if tsw.err != nil && firstErr == nil {
 			// Return just the first error, since other errors are likely duplicate the first error.
 			firstErr = tsw.err
 		}
 		rowsReadPerSeries.Update(float64(tsw.rowsProcessed))
 		rowsProcessedTotal += tsw.rowsProcessed
-		putTimeseriesWork(tsw)
 	}
 	return rowsProcessedTotal, firstErr
 }
@@ -1044,7 +1018,7 @@ func ExportBlocks(qt *querytracer.Tracer, sq *storage.SearchQuery, deadline sear
 			for xw := range workCh {
 				if err := f(&xw.mn, &xw.b, tr, workerID); err != nil {
 					errGlobalLock.Lock()
-					if errGlobal != nil {
+					if errGlobal == nil {
 						errGlobal = err
 						atomic.StoreUint32(&mustStop, 1)
 					}
@@ -1169,15 +1143,44 @@ func ProcessSearchQuery(qt *querytracer.Tracer, sq *storage.SearchQuery, deadlin
 	maxSeriesCount := sr.Init(qt, vmstorage.Storage, tfss, tr, sq.MaxMetrics, deadline.Deadline())
 	indexSearchDuration.UpdateDuration(startTime)
 	type blockRefs struct {
-		brsPrealloc [4]blockRef
-		brs         []blockRef
+		brs []blockRef
 	}
-	m := make(map[string]*blockRefs, maxSeriesCount)
-	orderedMetricNames := make([]string, 0, maxSeriesCount)
+
 	blocksRead := 0
 	samples := 0
 	tbf := getTmpBlocksFile()
 	var buf []byte
+	var metricNamePrev []byte
+
+	// metricNamesBuf is used for holding all the loaded unique metric names at m and orderedMetricNames.
+	// It should reduce pressure on Go GC by reducing the number of string allocations
+	// when constructing metricName string from byte slice.
+	metricNamesBufCap := maxSeriesCount * 100
+	if metricNamesBufCap > maxFastAllocBlockSize {
+		metricNamesBufCap = maxFastAllocBlockSize
+	}
+	metricNamesBuf := make([]byte, 0, metricNamesBufCap)
+
+	// brssPool is used for holding all the blockRefs objects across all the loaded time series.
+	// It should reduce pressure on Go GC by reducing the number of blockRefs allocations.
+	brssPool := make([]blockRefs, 0, maxSeriesCount)
+
+	// brsPool is used for holding the most of blockRefs.brs slices across all the loaded time series.
+	// It should reduce pressure on Go GC by reducing the number of allocations for blockRefs.brs slices.
+	brsPoolCap := uintptr(maxSeriesCount)
+	if brsPoolCap > maxFastAllocBlockSize/unsafe.Sizeof(blockRef{}) {
+		brsPoolCap = maxFastAllocBlockSize / unsafe.Sizeof(blockRef{})
+	}
+	brsPool := make([]blockRef, 0, brsPoolCap)
+
+	// m maps from metricName to the index of blockRefs inside brssPool
+	m := make(map[string]int, maxSeriesCount)
+
+	// orderedMetricNames contains the list of loaded unique metric names
+	// in the load order. This order is important for triggering sequential data reading.
+	orderedMetricNames := make([]string, 0, maxSeriesCount)
+
+	var brsIdx int
 	for sr.NextMetricBlock() {
 		blocksRead++
 		if deadline.Exceeded() {
@@ -1190,8 +1193,10 @@ func ProcessSearchQuery(qt *querytracer.Tracer, sq *storage.SearchQuery, deadlin
 		if *maxSamplesPerQuery > 0 && samples > *maxSamplesPerQuery {
 			putTmpBlocksFile(tbf)
 			putStorageSearch(sr)
-			return nil, fmt.Errorf("cannot select more than -search.maxSamplesPerQuery=%d samples; possible solutions: to increase the -search.maxSamplesPerQuery; to reduce time range for the query; to use more specific label filters in order to select lower number of series", *maxSamplesPerQuery)
+			return nil, fmt.Errorf("cannot select more than -search.maxSamplesPerQuery=%d samples; possible solutions: to increase the -search.maxSamplesPerQuery; "+
+				"to reduce time range for the query; to use more specific label filters in order to select lower number of series", *maxSamplesPerQuery)
 		}
+
 		buf = br.Marshal(buf[:0])
 		addr, err := tbf.WriteBlockRefData(buf)
 		if err != nil {
@@ -1199,24 +1204,59 @@ func ProcessSearchQuery(qt *querytracer.Tracer, sq *storage.SearchQuery, deadlin
 			putStorageSearch(sr)
 			return nil, fmt.Errorf("cannot write %d bytes to temporary file: %w", len(buf), err)
 		}
-		// Do not intern mb.MetricName, since it leads to increased memory usage.
-		// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3692
+
 		metricName := sr.MetricBlockRef.MetricName
-		brs := m[string(metricName)]
-		if brs == nil {
-			brs = &blockRefs{}
-			brs.brs = brs.brsPrealloc[:0]
+		if metricNamePrev == nil || string(metricName) != string(metricNamePrev) {
+			idx, ok := m[string(metricName)]
+			if !ok {
+				if cap(brssPool) > len(brssPool) {
+					brssPool = brssPool[:len(brssPool)+1]
+				} else {
+					brssPool = append(brssPool, blockRefs{})
+				}
+				idx = len(brssPool) - 1
+			}
+			brsIdx = idx
+			metricNamePrev = append(metricNamePrev[:0], metricName...)
+		}
+
+		brs := &brssPool[brsIdx]
+		partRef := br.PartRef()
+		if uintptr(cap(brsPool)) >= maxFastAllocBlockSize/unsafe.Sizeof(blockRef{}) && len(brsPool) == cap(brsPool) {
+			// Allocate a new brsPool in order to avoid slow allocation of an object
+			// bigger than maxFastAllocBlockSize bytes at append() below.
+			brsPool = make([]blockRef, 0, maxFastAllocBlockSize/unsafe.Sizeof(blockRef{}))
+		}
+		if brs.brs == nil || haveSameBlockRefTails(brs.brs, brsPool) {
+			// It is safe appending blockRef to brsPool, since there are no other items added there yet.
+			brsPool = append(brsPool, blockRef{
+				partRef: partRef,
+				addr:    addr,
+			})
+			brs.brs = brsPool[len(brsPool)-len(brs.brs)-1 : len(brsPool) : len(brsPool)]
+		} else {
+			// It is unsafe appending blockRef to brsPool, since there are other items added there.
+			// So just append it to brs.brs.
+			brs.brs = append(brs.brs, blockRef{
+				partRef: partRef,
+				addr:    addr,
+			})
 		}
-		brs.brs = append(brs.brs, blockRef{
-			partRef: br.PartRef(),
-			addr:    addr,
-		})
 		if len(brs.brs) == 1 {
-			metricNameStr := string(metricName)
+			if cap(metricNamesBuf) >= maxFastAllocBlockSize && len(metricNamesBuf)+len(metricName) > cap(metricNamesBuf) {
+				// Allocate a new metricNamesBuf in order to avoid slow allocation of byte slice
+				// bigger than maxFastAllocBlockSize bytes at append() below.
+				metricNamesBuf = make([]byte, 0, maxFastAllocBlockSize)
+			}
+			metricNamesBufLen := len(metricNamesBuf)
+			metricNamesBuf = append(metricNamesBuf, metricName...)
+			metricNameStr := bytesutil.ToUnsafeString(metricNamesBuf[metricNamesBufLen:])
+
 			orderedMetricNames = append(orderedMetricNames, metricNameStr)
-			m[metricNameStr] = brs
+			m[metricNameStr] = brsIdx
 		}
 	}
+
 	if err := sr.Error(); err != nil {
 		putTmpBlocksFile(tbf)
 		putStorageSearch(sr)
@@ -1239,7 +1279,7 @@ func ProcessSearchQuery(qt *querytracer.Tracer, sq *storage.SearchQuery, deadlin
 	for i, metricName := range orderedMetricNames {
 		pts[i] = packedTimeseries{
 			metricName: metricName,
-			brs:        m[metricName].brs,
+			brs:        brssPool[m[metricName]].brs,
 		}
 	}
 	rss.packedTimeseries = pts
@@ -1255,6 +1295,12 @@ type blockRef struct {
 	addr    tmpBlockAddr
 }
 
+func haveSameBlockRefTails(a, b []blockRef) bool {
+	sha := (*reflect.SliceHeader)(unsafe.Pointer(&a))
+	shb := (*reflect.SliceHeader)(unsafe.Pointer(&b))
+	return sha.Data+uintptr(sha.Len)*unsafe.Sizeof(blockRef{}) == shb.Data+uintptr(shb.Len)*unsafe.Sizeof(blockRef{})
+}
+
 func setupTfss(qt *querytracer.Tracer, tr storage.TimeRange, tagFilterss [][]storage.TagFilter, maxMetrics int, deadline searchutils.Deadline) ([]*storage.TagFilters, error) {
 	tfss := make([]*storage.TagFilters, 0, len(tagFilterss))
 	for _, tagFilters := range tagFilterss {
@@ -1300,3 +1346,8 @@ func applyGraphiteRegexpFilter(filter string, ss []string) ([]string, error) {
 	}
 	return dst, nil
 }
+
+// Go uses fast allocations for block sizes up to 32Kb.
+//
+// See https://github.com/golang/go/blob/704401ffa06c60e059c9e6e4048045b4ff42530a/src/runtime/malloc.go#L11
+const maxFastAllocBlockSize = 32 * 1024

app/vmselect/prometheus/expand-with-exprs.qtpl

@@ -82,7 +82,7 @@ textarea { margin: 1em }
 <h3>Tutorial for WITH expressions in <a href="https://docs.victoriametrics.com/MetricsQL.html">MetricsQL</a></h3>
 
 <p>
-    Let's look at the following real query from <a href="https://grafana.com/grafana/dashboards/1860-node-exporter-full/">Node Exporter Full</a> dashboard:
+    Let's look at the following real query from <a href="https://grafana.com/grafana/dashboards/1860">Node Exporter Full</a> dashboard:
 </p>
 
 <pre>
@@ -146,7 +146,7 @@ my_resource_utilization(
 </p>
 
 <p>
-    Let's take another nice query from <a href="https://grafana.com/grafana/dashboards/1860-node-exporter-full/">Node Exporter Full</a> dashboard:
+    Let's take another nice query from <a href="https://grafana.com/grafana/dashboards/1860">Node Exporter Full</a> dashboard:
 </p>
 
 <pre>

app/vmselect/prometheus/expand-with-exprs.qtpl.go

@@ -195,7 +195,7 @@ func streamwithExprsTutorial(qw422016 *qt422016.Writer) {
 <h3>Tutorial for WITH expressions in <a href="https://docs.victoriametrics.com/MetricsQL.html">MetricsQL</a></h3>
 
 <p>
-    Let's look at the following real query from <a href="https://grafana.com/grafana/dashboards/1860-node-exporter-full/">Node Exporter Full</a> dashboard:
+    Let's look at the following real query from <a href="https://grafana.com/grafana/dashboards/1860">Node Exporter Full</a> dashboard:
 </p>
 
 <pre>
@@ -259,7 +259,7 @@ my_resource_utilization(
 </p>
 
 <p>
-    Let's take another nice query from <a href="https://grafana.com/grafana/dashboards/1860-node-exporter-full/">Node Exporter Full</a> dashboard:
+    Let's take another nice query from <a href="https://grafana.com/grafana/dashboards/1860">Node Exporter Full</a> dashboard:
 </p>
 
 <pre>

app/vmselect/prometheus/prometheus.go

@@ -321,7 +321,9 @@ func exportHandler(qt *querytracer.Tracer, w http.ResponseWriter, cp *commonPara
 		firstLineSent := uint32(0)
 		writeLineFunc = func(xb *exportBlock, workerID uint) error {
 			bb := sw.getBuffer(workerID)
-			if atomic.CompareAndSwapUint32(&firstLineOnce, 0, 1) {
+			// Use atomic.LoadUint32() in front of atomic.CompareAndSwapUint32() in order to avoid slow inter-CPU synchronization
+			// in fast path after the first line has been already sent.
+			if atomic.LoadUint32(&firstLineOnce) == 0 && atomic.CompareAndSwapUint32(&firstLineOnce, 0, 1) {
 				// Send the first line to sw.bw
 				WriteExportPromAPILine(bb, xb)
 				_, err := sw.bw.Write(bb.B)
@@ -497,7 +499,10 @@ func LabelValuesHandler(qt *querytracer.Tracer, startTime time.Time, labelName s
 	if err != nil {
 		return err
 	}
-	sq := storage.NewSearchQuery(cp.start, cp.end, cp.filterss, *maxUniqueTimeseries)
+	// Do not limit the number of unique time series, which could be scanned
+	// during the search for matching label values, since users expect this API
+	// must always work.
+	sq := storage.NewSearchQuery(cp.start, cp.end, cp.filterss, -1)
 	labelValues, err := netstorage.LabelValues(qt, labelName, sq, limit, cp.deadline)
 	if err != nil {
 		return fmt.Errorf("cannot obtain values for label %q: %w", labelName, err)
@@ -594,7 +599,10 @@ func LabelsHandler(qt *querytracer.Tracer, startTime time.Time, w http.ResponseW
 	if err != nil {
 		return err
 	}
-	sq := storage.NewSearchQuery(cp.start, cp.end, cp.filterss, *maxUniqueTimeseries)
+	// Do not limit the number of unique time series, which could be scanned
+	// during the search for matching label values, since users expect this API
+	// must always work.
+	sq := storage.NewSearchQuery(cp.start, cp.end, cp.filterss, -1)
 	labels, err := netstorage.LabelNames(qt, sq, limit, cp.deadline)
 	if err != nil {
 		return fmt.Errorf("cannot obtain labels: %w", err)
@@ -718,9 +726,6 @@ func QueryHandler(qt *querytracer.Tracer, startTime time.Time, w http.ResponseWr
 		start -= offset
 		end := start
 		start = end - window
-		if start < 0 {
-			start = 0
-		}
 		// Do not include data point with a timestamp matching the lower boundary of the window as Prometheus does.
 		start++
 		if end < start {

app/vmselect/promql/aggr.go

@@ -111,41 +111,48 @@ func aggrFuncExt(afe func(tss []*timeseries, modifier *metricsql.ModifierExpr) [
 	modifier *metricsql.ModifierExpr, maxSeries int, keepOriginal bool) ([]*timeseries, error) {
 	m := aggrPrepareSeries(argOrig, modifier, maxSeries, keepOriginal)
 	rvs := make([]*timeseries, 0, len(m))
-	for _, tss := range m {
-		rv := afe(tss, modifier)
+	for _, tssl := range m {
+		rv := afe(tssl.tss, modifier)
 		rvs = append(rvs, rv...)
 	}
 	return rvs, nil
 }
 
-func aggrPrepareSeries(argOrig []*timeseries, modifier *metricsql.ModifierExpr, maxSeries int, keepOriginal bool) map[string][]*timeseries {
+func aggrPrepareSeries(argOrig []*timeseries, modifier *metricsql.ModifierExpr, maxSeries int, keepOriginal bool) map[string]*tssList {
 	// Remove empty time series, e.g. series with all NaN samples,
 	// since such series are ignored by aggregate functions.
 	argOrig = removeEmptySeries(argOrig)
 	arg := copyTimeseriesMetricNames(argOrig, keepOriginal)
 
 	// Perform grouping.
-	m := make(map[string][]*timeseries)
+	m := make(map[string]*tssList)
 	bb := bbPool.Get()
 	for i, ts := range arg {
 		removeGroupTags(&ts.MetricName, modifier)
 		bb.B = marshalMetricNameSorted(bb.B[:0], &ts.MetricName)
-		k := string(bb.B)
+		k := bb.B
 		if keepOriginal {
 			ts = argOrig[i]
 		}
-		tss := m[k]
-		if tss == nil && maxSeries > 0 && len(m) >= maxSeries {
-			// We already reached time series limit after grouping. Skip other time series.
-			continue
+		tssl := m[string(k)]
+		if tssl == nil {
+			if maxSeries > 0 && len(m) >= maxSeries {
+				// We already reached time series limit after grouping. Skip other time series.
+				continue
+			}
+			tssl = &tssList{}
+			m[string(k)] = tssl
 		}
-		tss = append(tss, ts)
-		m[k] = tss
+		tssl.tss = append(tssl.tss, ts)
 	}
 	bbPool.Put(bb)
 	return m
 }
 
+type tssList struct {
+	tss []*timeseries
+}
+
 func aggrFuncAny(afa *aggrFuncArg) ([]*timeseries, error) {
 	tss, err := getAggrTimeseries(afa.args)
 	if err != nil {
@@ -626,8 +633,8 @@ func aggrFuncCountValues(afa *aggrFuncArg) ([]*timeseries, error) {
 
 	m := aggrPrepareSeries(args[1], &afa.ae.Modifier, afa.ae.Limit, false)
 	rvs := make([]*timeseries, 0, len(m))
-	for _, tss := range m {
-		rv, err := afe(tss, modifier)
+	for _, tssl := range m {
+		rv, err := afe(tssl.tss, modifier)
 		if err != nil {
 			return nil, err
 		}

app/vmselect/promql/aggr_incremental.go

@@ -111,8 +111,8 @@ func (iafc *incrementalAggrFuncContext) updateTimeseries(tsOrig *timeseries, wor
 	removeGroupTags(&ts.MetricName, &iafc.ae.Modifier)
 	bb := bbPool.Get()
 	bb.B = marshalMetricNameSorted(bb.B[:0], &ts.MetricName)
-	k := string(bb.B)
-	iac := m[k]
+	k := bb.B
+	iac := m[string(k)]
 	if iac == nil {
 		if iafc.ae.Limit > 0 && len(m) >= iafc.ae.Limit {
 			// Skip this time series, since the limit on the number of output time series has been already reached.
@@ -131,7 +131,7 @@ func (iafc *incrementalAggrFuncContext) updateTimeseries(tsOrig *timeseries, wor
 			ts:     tsAggr,
 			values: make([]float64, len(ts.Values)),
 		}
-		m[k] = iac
+		m[string(k)] = iac
 	}
 	bbPool.Put(bb)
 	iafc.callbacks.updateAggrFunc(iac, ts.Values)

app/vmselect/promql/eval.go

@@ -1265,7 +1265,7 @@ func evalInstantRollup(qt *querytracer.Tracer, ec *EvalConfig, funcName string,
 			return evalAt(qtChild, timestamp, window)
 		}
 		// Calculate the result
-		tss, ok := getMaxInstantValues(qtChild, tssCached, tssStart, tssEnd)
+		tss, ok := getMaxInstantValues(qtChild, tssCached, tssStart, tssEnd, timestamp)
 		if !ok {
 			qtChild.Printf("cannot apply instant rollup optimization, since tssEnd contains bigger values than tssCached")
 			deleteCachedSeries(qtChild)
@@ -1327,7 +1327,7 @@ func evalInstantRollup(qt *querytracer.Tracer, ec *EvalConfig, funcName string,
 			return evalAt(qtChild, timestamp, window)
 		}
 		// Calculate the result
-		tss, ok := getMinInstantValues(qtChild, tssCached, tssStart, tssEnd)
+		tss, ok := getMinInstantValues(qtChild, tssCached, tssStart, tssEnd, timestamp)
 		if !ok {
 			qtChild.Printf("cannot apply instant rollup optimization, since tssEnd contains smaller values than tssCached")
 			deleteCachedSeries(qtChild)
@@ -1392,7 +1392,7 @@ func evalInstantRollup(qt *querytracer.Tracer, ec *EvalConfig, funcName string,
 			return evalAt(qtChild, timestamp, window)
 		}
 		// Calculate the result
-		tss := getSumInstantValues(qtChild, tssCached, tssStart, tssEnd)
+		tss := getSumInstantValues(qtChild, tssCached, tssStart, tssEnd, timestamp)
 		return tss, nil
 	default:
 		qt.Printf("instant rollup optimization isn't implemented for %s()", funcName)
@@ -1419,8 +1419,8 @@ func hasDuplicateSeries(tss []*timeseries) bool {
 	return false
 }
 
-func getMinInstantValues(qt *querytracer.Tracer, tssCached, tssStart, tssEnd []*timeseries) ([]*timeseries, bool) {
-	qt = qt.NewChild("calculate the minimum for instant values across series; cached=%d, start=%d, end=%d", len(tssCached), len(tssStart), len(tssEnd))
+func getMinInstantValues(qt *querytracer.Tracer, tssCached, tssStart, tssEnd []*timeseries, timestamp int64) ([]*timeseries, bool) {
+	qt = qt.NewChild("calculate the minimum for instant values across series; cached=%d, start=%d, end=%d, timestamp=%d", len(tssCached), len(tssStart), len(tssEnd), timestamp)
 	defer qt.Done()
 
 	getMin := func(a, b float64) float64 {
@@ -1429,13 +1429,13 @@ func getMinInstantValues(qt *querytracer.Tracer, tssCached, tssStart, tssEnd []*
 		}
 		return b
 	}
-	tss, ok := getMinMaxInstantValues(tssCached, tssStart, tssEnd, getMin)
+	tss, ok := getMinMaxInstantValues(tssCached, tssStart, tssEnd, timestamp, getMin)
 	qt.Printf("resulting series=%d; ok=%v", len(tss), ok)
 	return tss, ok
 }
 
-func getMaxInstantValues(qt *querytracer.Tracer, tssCached, tssStart, tssEnd []*timeseries) ([]*timeseries, bool) {
-	qt = qt.NewChild("calculate the maximum for instant values across series; cached=%d, start=%d, end=%d", len(tssCached), len(tssStart), len(tssEnd))
+func getMaxInstantValues(qt *querytracer.Tracer, tssCached, tssStart, tssEnd []*timeseries, timestamp int64) ([]*timeseries, bool) {
+	qt = qt.NewChild("calculate the maximum for instant values across series; cached=%d, start=%d, end=%d, timestamp=%d", len(tssCached), len(tssStart), len(tssEnd), timestamp)
 	defer qt.Done()
 
 	getMax := func(a, b float64) float64 {
@@ -1444,12 +1444,12 @@ func getMaxInstantValues(qt *querytracer.Tracer, tssCached, tssStart, tssEnd []*
 		}
 		return b
 	}
-	tss, ok := getMinMaxInstantValues(tssCached, tssStart, tssEnd, getMax)
+	tss, ok := getMinMaxInstantValues(tssCached, tssStart, tssEnd, timestamp, getMax)
 	qt.Printf("resulting series=%d", len(tss))
 	return tss, ok
 }
 
-func getMinMaxInstantValues(tssCached, tssStart, tssEnd []*timeseries, f func(a, b float64) float64) ([]*timeseries, bool) {
+func getMinMaxInstantValues(tssCached, tssStart, tssEnd []*timeseries, timestamp int64, f func(a, b float64) float64) ([]*timeseries, bool) {
 	assertInstantValues(tssCached)
 	assertInstantValues(tssStart)
 	assertInstantValues(tssEnd)
@@ -1500,12 +1500,16 @@ func getMinMaxInstantValues(tssCached, tssStart, tssEnd []*timeseries, f func(a,
 	for _, ts := range m {
 		rvs = append(rvs, ts)
 	}
+
+	setInstantTimestamp(rvs, timestamp)
+
 	return rvs, true
 }
 
-// getSumInstantValues calculates tssCached + tssStart - tssEnd
-func getSumInstantValues(qt *querytracer.Tracer, tssCached, tssStart, tssEnd []*timeseries) []*timeseries {
-	qt = qt.NewChild("calculate the sum for instant values across series; cached=%d, start=%d, end=%d", len(tssCached), len(tssStart), len(tssEnd))
+// getSumInstantValues aggregates tssCached, tssStart, tssEnd time series
+// into a new time series with value = tssCached + tssStart - tssEnd
+func getSumInstantValues(qt *querytracer.Tracer, tssCached, tssStart, tssEnd []*timeseries, timestamp int64) []*timeseries {
+	qt = qt.NewChild("calculate the sum for instant values across series; cached=%d, start=%d, end=%d, timestamp=%d", len(tssCached), len(tssStart), len(tssEnd), timestamp)
 	defer qt.Done()
 
 	assertInstantValues(tssCached)
@@ -1550,10 +1554,19 @@ func getSumInstantValues(qt *querytracer.Tracer, tssCached, tssStart, tssEnd []*
 	for _, ts := range m {
 		rvs = append(rvs, ts)
 	}
+
+	setInstantTimestamp(rvs, timestamp)
+
 	qt.Printf("resulting series=%d", len(rvs))
 	return rvs
 }
 
+func setInstantTimestamp(tss []*timeseries, timestamp int64) {
+	for _, ts := range tss {
+		ts.Timestamps[0] = timestamp
+	}
+}
+
 func assertInstantValues(tss []*timeseries) {
 	for _, ts := range tss {
 		if len(ts.Values) != 1 {
@@ -1681,9 +1694,6 @@ func evalRollupFuncNoCache(qt *querytracer.Tracer, ec *EvalConfig, funcName stri
 	} else {
 		minTimestamp -= ec.Step
 	}
-	if minTimestamp < 0 {
-		minTimestamp = 0
-	}
 	sq := storage.NewSearchQuery(minTimestamp, ec.End, tfss, ec.MaxSeries)
 	rss, err := netstorage.ProcessSearchQuery(qt, sq, ec.Deadline)
 	if err != nil {

app/vmselect/promql/eval_test.go

@@ -1,6 +1,7 @@
 package promql
 
 import (
+	"reflect"
 	"testing"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/prometheus"
@@ -95,3 +96,77 @@ func TestQueryStats_addSeriesFetched(t *testing.T) {
 		t.Fatalf("expected to get 4; got %d instead", qs.SeriesFetched)
 	}
 }
+
+func TestGetSumInstantValues(t *testing.T) {
+	f := func(cached, start, end []*timeseries, timestamp int64, expectedResult []*timeseries) {
+		t.Helper()
+
+		result := getSumInstantValues(nil, cached, start, end, timestamp)
+		if !reflect.DeepEqual(result, expectedResult) {
+			t.Errorf("unexpected result; got\n%v\nwant\n%v", result, expectedResult)
+		}
+	}
+	ts := func(name string, timestamp int64, value float64) *timeseries {
+		return &timeseries{
+			MetricName: storage.MetricName{
+				MetricGroup: []byte(name),
+			},
+			Timestamps: []int64{timestamp},
+			Values:     []float64{value},
+		}
+	}
+
+	// start - end + cached = 1
+	f(
+		nil,
+		[]*timeseries{ts("foo", 42, 1)},
+		nil,
+		100,
+		[]*timeseries{ts("foo", 100, 1)},
+	)
+
+	// start - end + cached = 0
+	f(
+		nil,
+		[]*timeseries{ts("foo", 100, 1)},
+		[]*timeseries{ts("foo", 10, 1)},
+		100,
+		[]*timeseries{ts("foo", 100, 0)},
+	)
+
+	// start - end + cached = 2
+	f(
+		[]*timeseries{ts("foo", 10, 1)},
+		[]*timeseries{ts("foo", 100, 1)},
+		nil,
+		100,
+		[]*timeseries{ts("foo", 100, 2)},
+	)
+
+	// start - end + cached = 1
+	f(
+		[]*timeseries{ts("foo", 50, 1)},
+		[]*timeseries{ts("foo", 100, 1)},
+		[]*timeseries{ts("foo", 10, 1)},
+		100,
+		[]*timeseries{ts("foo", 100, 1)},
+	)
+
+	// start - end + cached = 0
+	f(
+		[]*timeseries{ts("foo", 50, 1)},
+		nil,
+		[]*timeseries{ts("foo", 10, 1)},
+		100,
+		[]*timeseries{ts("foo", 100, 0)},
+	)
+
+	// start - end + cached = 1
+	f(
+		[]*timeseries{ts("foo", 50, 1)},
+		nil,
+		nil,
+		100,
+		[]*timeseries{ts("foo", 100, 1)},
+	)
+}

app/vmselect/promql/exec_test.go

@@ -8050,10 +8050,10 @@ func TestExecSuccess(t *testing.T) {
 	})
 	t.Run(`rollup_rate()`, func(t *testing.T) {
 		t.Parallel()
-		q := `rollup_rate((2000-time())[600s])`
+		q := `rollup_rate((2200-time())[600s])`
 		r1 := netstorage.Result{
 			MetricName: metricNameExpected,
-			Values:     []float64{5, 4, 3, 2, 1, 0},
+			Values:     []float64{6, 5, 4, 3, 2, 1},
 			Timestamps: timestampsExpected,
 		}
 		r1.MetricName.Tags = []storage.Tag{{
@@ -8062,7 +8062,7 @@ func TestExecSuccess(t *testing.T) {
 		}}
 		r2 := netstorage.Result{
 			MetricName: metricNameExpected,
-			Values:     []float64{6, 5, 4, 3, 2, 1},
+			Values:     []float64{7, 6, 5, 4, 3, 2},
 			Timestamps: timestampsExpected,
 		}
 		r2.MetricName.Tags = []storage.Tag{{
@@ -8071,7 +8071,7 @@ func TestExecSuccess(t *testing.T) {
 		}}
 		r3 := netstorage.Result{
 			MetricName: metricNameExpected,
-			Values:     []float64{4, 3, 2, 1, 0, -1},
+			Values:     []float64{5, 4, 3, 2, 1, 0},
 			Timestamps: timestampsExpected,
 		}
 		r3.MetricName.Tags = []storage.Tag{{
@@ -8083,10 +8083,10 @@ func TestExecSuccess(t *testing.T) {
 	})
 	t.Run(`rollup_rate(q, "max")`, func(t *testing.T) {
 		t.Parallel()
-		q := `rollup_rate((2000-time())[600s], "max")`
+		q := `rollup_rate((2200-time())[600s], "max")`
 		r := netstorage.Result{
 			MetricName: metricNameExpected,
-			Values:     []float64{6, 5, 4, 3, 2, 1},
+			Values:     []float64{7, 6, 5, 4, 3, 2},
 			Timestamps: timestampsExpected,
 		}
 		resultExpected := []netstorage.Result{r}
@@ -8094,10 +8094,10 @@ func TestExecSuccess(t *testing.T) {
 	})
 	t.Run(`rollup_rate(q, "avg")`, func(t *testing.T) {
 		t.Parallel()
-		q := `rollup_rate((2000-time())[600s], "avg")`
+		q := `rollup_rate((2200-time())[600s], "avg")`
 		r := netstorage.Result{
 			MetricName: metricNameExpected,
-			Values:     []float64{5, 4, 3, 2, 1, 0},
+			Values:     []float64{6, 5, 4, 3, 2, 1},
 			Timestamps: timestampsExpected,
 		}
 		resultExpected := []netstorage.Result{r}

app/vmselect/promql/rollup.go

@@ -859,6 +859,11 @@ func removeCounterResets(values []float64) {
 		}
 		prevValue = v
 		values[i] = v + correction
+		// Check again, there could be precision error in float operations,
+		// see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5571
+		if i > 0 && values[i] < values[i-1] {
+			values[i] = values[i-1]
+		}
 	}
 }
 

app/vmselect/promql/rollup_result_cache.go

@@ -43,6 +43,11 @@ func ResetRollupResultCacheIfNeeded(mrs []storage.MetricRow) {
 		rollupResultResetMetricRowSample.Store(&storage.MetricRow{})
 		go checkRollupResultCacheReset()
 	})
+	if atomic.LoadUint32(&needRollupResultCacheReset) != 0 {
+		// The cache has been already instructed to reset.
+		return
+	}
+
 	minTimestamp := int64(fasttime.UnixTimestamp()*1000) - cacheTimestampOffset.Milliseconds() + checkRollupResultCacheResetInterval.Milliseconds()
 	needCacheReset := false
 	for i := range mrs {

app/vmselect/promql/rollup_test.go

@@ -125,7 +125,7 @@ func TestRemoveCounterResets(t *testing.T) {
 	// removeCounterResets doesn't expect negative values, so it doesn't work properly with them.
 	values = []float64{-100, -200, -300, -400}
 	removeCounterResets(values)
-	valuesExpected = []float64{-100, -300, -600, -1000}
+	valuesExpected = []float64{-100, -100, -100, -100}
 	timestampsExpected := []int64{0, 1, 2, 3}
 	testRowsEqual(t, values, timestampsExpected, valuesExpected, timestampsExpected)
 
@@ -136,6 +136,17 @@ func TestRemoveCounterResets(t *testing.T) {
 	valuesExpected = []float64{100, 100, 125, 125, 145, 195}
 	timestampsExpected = []int64{0, 1, 2, 3, 4, 5}
 	testRowsEqual(t, values, timestampsExpected, valuesExpected, timestampsExpected)
+
+	// verify results always increase monotonically with possible float operations precision error
+	values = []float64{34.094223, 2.7518, 2.140669, 0.044878, 1.887095, 2.546569, 2.490149, 0.045, 0.035684, 0.062454, 0.058296}
+	removeCounterResets(values)
+	var prev float64
+	for i, v := range values {
+		if v < prev {
+			t.Fatalf("error: unexpected value keep getting bigger %d; cur %v; pre %v\n", i, v, prev)
+		}
+		prev = v
+	}
 }
 
 func TestDeltaValues(t *testing.T) {

app/vmselect/vmui.go

@@ -7,6 +7,7 @@ import (
 	"net/http"
 	"os"
 	"path/filepath"
+	"time"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fs"
 )
@@ -14,6 +15,8 @@ import (
 var (
 	vmuiCustomDashboardsPath = flag.String("vmui.customDashboardsPath", "", "Optional path to vmui dashboards. "+
 		"See https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmui/packages/vmui/public/dashboards")
+	vmuiDefaultTimezone = flag.String("vmui.defaultTimezone", "", "The default timezone to be used in vmui."+
+		"Timezone must be a valid IANA Time Zone. For example: America/New_York, Europe/Berlin, Etc/GMT+3 or Local")
 )
 
 // dashboardSettings represents dashboard settings file struct.
@@ -65,6 +68,16 @@ func handleVMUICustomDashboards(w http.ResponseWriter) error {
 	return nil
 }
 
+func handleVMUITimezone(w http.ResponseWriter) error {
+	tz, err := time.LoadLocation(*vmuiDefaultTimezone)
+	if err != nil {
+		return fmt.Errorf("cannot load timezone %q: %w", *vmuiDefaultTimezone, err)
+	}
+	response := fmt.Sprintf(`{"timezone": %q}`, tz)
+	writeSuccessResponse(w, []byte(response))
+	return nil
+}
+
 func writeSuccessResponse(w http.ResponseWriter, data []byte) {
 	w.WriteHeader(http.StatusOK)
 	w.Header().Set("Content-Type", "application/json")

app/vmselect/vmui/asset-manifest.json

@@ -1,13 +1,13 @@
 {
   "files": {
-    "main.css": "./static/css/main.fb353c1e.css",
-    "main.js": "./static/js/main.5bcddddc.js",
+    "main.css": "./static/css/main.2f6793ba.css",
+    "main.js": "./static/js/main.bcd188b5.js",
     "static/js/522.da77e7b3.chunk.js": "./static/js/522.da77e7b3.chunk.js",
     "static/media/MetricsQL.md": "./static/media/MetricsQL.b64c4dbf91f4fa581621.md",
     "index.html": "./index.html"
   },
   "entrypoints": [
-    "static/css/main.fb353c1e.css",
-    "static/js/main.5bcddddc.js"
+    "static/css/main.2f6793ba.css",
+    "static/js/main.bcd188b5.js"
   ]
 }

app/vmselect/vmui/index.html

@@ -1 +1 @@
-<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=5"/><meta name="theme-color" content="#000000"/><meta name="description" content="UI for VictoriaMetrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><script src="./dashboards/index.js" type="module"></script><meta name="twitter:card" content="summary_large_image"><meta name="twitter:image" content="./preview.jpg"><meta name="twitter:title" content="UI for VictoriaMetrics"><meta name="twitter:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta name="twitter:site" content="@VictoriaMetrics"><meta property="og:title" content="Metric explorer for VictoriaMetrics"><meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta property="og:image" content="./preview.jpg"><meta property="og:type" content="website"><script defer="defer" src="./static/js/main.5bcddddc.js"></script><link href="./static/css/main.fb353c1e.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>
+<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=5"/><meta name="theme-color" content="#000000"/><meta name="description" content="UI for VictoriaMetrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><script src="./dashboards/index.js" type="module"></script><meta name="twitter:card" content="summary_large_image"><meta name="twitter:image" content="./preview.jpg"><meta name="twitter:title" content="UI for VictoriaMetrics"><meta name="twitter:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta name="twitter:site" content="@VictoriaMetrics"><meta property="og:title" content="Metric explorer for VictoriaMetrics"><meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta property="og:image" content="./preview.jpg"><meta property="og:type" content="website"><script defer="defer" src="./static/js/main.bcd188b5.js"></script><link href="./static/css/main.2f6793ba.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>

app/vmstorage/main.go

@@ -10,6 +10,8 @@ import (
 	"sync"
 	"time"
 
+	"github.com/VictoriaMetrics/metrics"
+
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/encoding"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fasttime"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
@@ -20,14 +22,14 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/querytracer"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/storage"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/syncwg"
-	"github.com/VictoriaMetrics/metrics"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/timeutil"
 )
 
 var (
 	retentionPeriod       = flagutil.NewDuration("retentionPeriod", "1", "Data with timestamps outside the retentionPeriod is automatically deleted. The minimum retentionPeriod is 24h or 1d. See also -retentionFilter")
-	snapshotAuthKey       = flag.String("snapshotAuthKey", "", "authKey, which must be passed in query string to /snapshot* pages")
-	forceMergeAuthKey     = flag.String("forceMergeAuthKey", "", "authKey, which must be passed in query string to /internal/force_merge pages")
-	forceFlushAuthKey     = flag.String("forceFlushAuthKey", "", "authKey, which must be passed in query string to /internal/force_flush pages")
+	snapshotAuthKey       = flagutil.NewPassword("snapshotAuthKey", "authKey, which must be passed in query string to /snapshot* pages")
+	forceMergeAuthKey     = flagutil.NewPassword("forceMergeAuthKey", "authKey, which must be passed in query string to /internal/force_merge pages")
+	forceFlushAuthKey     = flagutil.NewPassword("forceFlushAuthKey", "authKey, which must be passed in query string to /internal/force_flush pages")
 	snapshotsMaxAge       = flagutil.NewDuration("snapshotsMaxAge", "0", "Automatically delete snapshots older than -snapshotsMaxAge if it is set to non-zero duration. Make sure that backup process has enough time to finish the backup before the corresponding snapshot is automatically deleted")
 	snapshotCreateTimeout = flag.Duration("snapshotCreateTimeout", 0, "The timeout for creating new snapshot. If set, make sure that timeout is lower than backup period")
 
@@ -36,13 +38,10 @@ var (
 	// DataPath is a path to storage data.
 	DataPath = flag.String("storageDataPath", "victoria-metrics-data", "Path to storage data")
 
-	finalMergeDelay = flag.Duration("finalMergeDelay", 0, "The delay before starting final merge for per-month partition after no new data is ingested into it. "+
-		"Final merge may require additional disk IO and CPU resources. Final merge may increase query speed and reduce disk space usage in some cases. "+
-		"Zero value disables final merge")
-	_ = flag.Int("bigMergeConcurrency", 0, "Deprecated: this flag does nothing. Please use -smallMergeConcurrency "+
-		"for controlling the concurrency of background merges. See https://docs.victoriametrics.com/#storage")
-	smallMergeConcurrency = flag.Int("smallMergeConcurrency", 0, "The maximum number of workers for background merges. See https://docs.victoriametrics.com/#storage . "+
-		"It isn't recommended tuning this flag in general case, since this may lead to uncontrolled increase in the number of parts and increased CPU usage during queries")
+	_ = flag.Duration("finalMergeDelay", 0, "Deprecated: this flag does nothing")
+	_ = flag.Int("bigMergeConcurrency", 0, "Deprecated: this flag does nothing")
+	_ = flag.Int("smallMergeConcurrency", 0, "Deprecated: this flag does nothing")
+
 	retentionTimezoneOffset = flag.Duration("retentionTimezoneOffset", 0, "The offset for performing indexdb rotation. "+
 		"If set to 0, then the indexdb rotation is performed at 4am UTC time per each -retentionPeriod. "+
 		"If set to 2h, then the indexdb rotation is performed at 4am EET time (the timezone with +2h offset)")
@@ -94,8 +93,6 @@ func Init(resetCacheIfNeeded func(mrs []storage.MetricRow)) {
 
 	resetResponseCacheIfNeeded = resetCacheIfNeeded
 	storage.SetLogNewSeries(*logNewSeries)
-	storage.SetFinalMergeDelay(*finalMergeDelay)
-	storage.SetMergeWorkersCount(*smallMergeConcurrency)
 	storage.SetRetentionTimezoneOffset(*retentionTimezoneOffset)
 	storage.SetFreeDiskSpaceLimit(minFreeDiskSpaceBytes.N)
 	storage.SetTSIDCacheSize(cacheSizeStorageTSID.IntN())
@@ -259,7 +256,7 @@ func Stop() {
 func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 	path := r.URL.Path
 	if path == "/internal/force_merge" {
-		if !httpserver.CheckAuthFlag(w, r, *forceMergeAuthKey, "forceMergeAuthKey") {
+		if !httpserver.CheckAuthFlag(w, r, forceMergeAuthKey.Get(), "forceMergeAuthKey") {
 			return true
 		}
 		// Run force merge in background
@@ -277,7 +274,7 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 		return true
 	}
 	if path == "/internal/force_flush" {
-		if !httpserver.CheckAuthFlag(w, r, *forceFlushAuthKey, "forceFlushAuthKey") {
+		if !httpserver.CheckAuthFlag(w, r, forceFlushAuthKey.Get(), "forceFlushAuthKey") {
 			return true
 		}
 		logger.Infof("flushing storage to make pending data available for reading")
@@ -293,7 +290,7 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 	if !strings.HasPrefix(path, "/snapshot") {
 		return false
 	}
-	if !httpserver.CheckAuthFlag(w, r, *snapshotAuthKey, "snapshotAuthKey") {
+	if !httpserver.CheckAuthFlag(w, r, snapshotAuthKey.Get(), "snapshotAuthKey") {
 		return true
 	}
 	path = path[len("/snapshot"):]
@@ -400,7 +397,8 @@ func initStaleSnapshotsRemover(strg *storage.Storage) {
 	staleSnapshotsRemoverWG.Add(1)
 	go func() {
 		defer staleSnapshotsRemoverWG.Done()
-		t := time.NewTicker(11 * time.Second)
+		d := timeutil.AddJitterToDuration(time.Second * 11)
+		t := time.NewTicker(d)
 		defer t.Stop()
 		for {
 			select {
@@ -493,11 +491,8 @@ func writeStorageMetrics(w io.Writer, strg *storage.Storage) {
 	metrics.WriteCounterUint64(w, `vm_composite_filter_success_conversions_total`, idbm.CompositeFilterSuccessConversions)
 	metrics.WriteCounterUint64(w, `vm_composite_filter_missing_conversions_total`, idbm.CompositeFilterMissingConversions)
 
-	metrics.WriteCounterUint64(w, `vm_assisted_merges_total{type="storage/inmemory"}`, tm.InmemoryAssistedMerges)
-	metrics.WriteCounterUint64(w, `vm_assisted_merges_total{type="storage/small"}`, tm.SmallAssistedMerges)
-
-	metrics.WriteCounterUint64(w, `vm_assisted_merges_total{type="indexdb/inmemory"}`, idbm.InmemoryAssistedMerges)
-	metrics.WriteCounterUint64(w, `vm_assisted_merges_total{type="indexdb/file"}`, idbm.FileAssistedMerges)
+	// vm_assisted_merges_total name is used for backwards compatibility.
+	metrics.WriteCounterUint64(w, `vm_assisted_merges_total{type="indexdb/inmemory"}`, idbm.InmemoryPartsLimitReachedCount)
 
 	metrics.WriteCounterUint64(w, `vm_indexdb_items_added_total`, idbm.ItemsAdded)
 	metrics.WriteCounterUint64(w, `vm_indexdb_items_added_size_bytes_total`, idbm.ItemsAddedSizeBytes)

app/vmui/README.md

@@ -7,6 +7,7 @@ Web UI for VictoriaMetrics
 * [Updating vmui embedded into VictoriaMetrics](#updating-vmui-embedded-into-victoriametrics)
 * [Predefined dashboards](#predefined-dashboards)
 * [App mode config options](#app-mode-config-options)
+* [Timezone configuration](#timezone-configuration)
 
 ----
 
@@ -246,3 +247,39 @@ vmui can be used to paste into other applications
 ```html
 <div id="root" data-params='{"serverURL":"http://localhost:8428","useTenantID":true,"headerStyles":{"background":"#FFFFFF","color":"#538DE8"},"palette":{"primary":"#538DE8","secondary":"#F76F8E","error":"#FD151B","warning":"#FFB30F","success":"#7BE622","info":"#0F5BFF"}}'></div>
 ```
+
+----
+
+## Timezone configuration
+
+vmui's timezone setting offers flexibility in displaying time data. It can be set through a configuration flag and is adjustable within the vmui interface. This feature caters to various user preferences and time zones.
+
+### Default Timezone Setting
+
+#### Via Configuration Flag
+
+- Set the default timezone using the `--vmui.defaultTimezone` flag.
+- Accepts a valid IANA Time Zone string (e.g., `America/New_York`, `Europe/Berlin`, `Etc/GMT+3`).
+- If the flag is unset or invalid, vmui defaults to the browser's local timezone.
+
+#### User Interface Adjustments
+
+- Users can change the timezone in the vmui interface.
+- Any changed setting in the interface overrides the flag's default, persisting for the user.
+- The timezone specified in the `--vmui.defaultTimezone` flag is included in the vmui's timezone selection dropdown, aiding user choice.
+
+### Key Points
+
+- **Fallback to Browser's Local Timezone**: If the flag is not set or an invalid timezone is specified, vmui uses the local timezone of the user's browser.
+- **User Preference Priority**: User-selected timezones in vmui take precedence over the default set by the flag.
+- **Cluster Consistency**: Ensure uniform timezone settings across cluster nodes, but individual user interface selections will always override these defaults.
+
+### Examples
+
+Setting a default timezone, with user options to change:
+
+```
+./victoria-metrics --vmui.defaultTimezone="America/New_York"
+```
+
+In this scenario, if a user in Berlin accesses vmui without changing settings, it will default to their browser's local timezone (CET). If they select a different timezone in vmui, this choice will override the `"America/New_York"` setting for that user.

app/vmui/packages/vmui/src/App.tsx

@@ -14,6 +14,7 @@ import PreviewIcons from "./components/Main/Icons/PreviewIcons";
 import WithTemplate from "./pages/WithTemplate";
 import Relabel from "./pages/Relabel";
 import ActiveQueries from "./pages/ActiveQueries";
+import QueryAnalyzer from "./pages/QueryAnalyzer";
 
 const App: FC = () => {
   const [loadedTheme, setLoadedTheme] = useState(false);
@@ -49,6 +50,10 @@ const App: FC = () => {
                   path={router.trace}
                   element={<TracePage/>}
                 />
+                <Route
+                  path={router.queryAnalyzer}
+                  element={<QueryAnalyzer/>}
+                />
                 <Route
                   path={router.dashboards}
                   element={<DashboardsLayout/>}

app/vmui/packages/vmui/src/api/query-range.ts

@@ -4,4 +4,4 @@ export const getQueryRangeUrl = (server: string, query: string, period: TimePara
   `${server}/api/v1/query_range?query=${encodeURIComponent(query)}&start=${period.start}&end=${period.end}&step=${period.step}${nocache ? "&nocache=1" : ""}${queryTracing ? "&trace=1" : ""}`;
 
 export const getQueryUrl = (server: string, query: string, period: TimeParams, nocache: boolean, queryTracing: boolean): string =>
-  `${server}/api/v1/query?query=${encodeURIComponent(query)}&time=${period.end}${nocache ? "&nocache=1" : ""}${queryTracing ? "&trace=1" : ""}`;
+  `${server}/api/v1/query?query=${encodeURIComponent(query)}&time=${period.end}&step=${period.step}${nocache ? "&nocache=1" : ""}${queryTracing ? "&trace=1" : ""}`;

app/vmui/packages/vmui/src/components/Configurators/GlobalSettings/GlobalSettings.tsx

@@ -29,7 +29,7 @@ const GlobalSettings: FC = () => {
 
   const appModeEnable = getAppModeEnable();
   const { serverUrl: stateServerUrl, theme } = useAppState();
-  const { timezone: stateTimezone } = useTimeState();
+  const { timezone: stateTimezone, defaultTimezone } = useTimeState();
   const { seriesLimits } = useCustomPanelState();
 
   const dispatch = useAppDispatch();
@@ -78,6 +78,10 @@ const GlobalSettings: FC = () => {
     setServerUrl(stateServerUrl);
   }, [stateServerUrl]);
 
+  useEffect(() => {
+    setTimezone(stateTimezone);
+  }, [stateTimezone]);
+
   const controls = [
     {
       show: !appModeEnable && !isLogsApp,
@@ -100,6 +104,7 @@ const GlobalSettings: FC = () => {
       show: true,
       component: <Timezones
         timezoneState={timezone}
+        defaultTimezone={defaultTimezone}
         onChange={setTimezone}
       />
     },

app/vmui/packages/vmui/src/components/Configurators/GlobalSettings/ServerConfigurator/ServerConfigurator.tsx

@@ -51,6 +51,12 @@ const ServerConfigurator: FC<ServerConfiguratorProps> = ({
     }
   }, [enabledStorage]);
 
+  useEffect(() => {
+    if (enabledStorage) {
+      saveToStorage("SERVER_URL", serverUrl);
+    }
+  }, [serverUrl]);
+
   return (
     <div>
       <div className="vm-server-configurator__title">

app/vmui/packages/vmui/src/components/Configurators/GlobalSettings/Timezones/Timezones.tsx

@@ -12,11 +12,16 @@ import useDeviceDetect from "../../../../hooks/useDeviceDetect";
 import useBoolean from "../../../../hooks/useBoolean";
 
 interface TimezonesProps {
-  timezoneState: string
-  onChange: (val: string) => void
+  timezoneState: string;
+  defaultTimezone?: string;
+  onChange: (val: string) => void;
 }
 
-const Timezones: FC<TimezonesProps> = ({ timezoneState, onChange }) => {
+interface PinnedTimezone extends Timezone {
+  title: string
+}
+
+const Timezones: FC<TimezonesProps> = ({ timezoneState, defaultTimezone, onChange }) => {
   const { isMobile } = useDeviceDetect();
   const timezones = getTimezoneList();
 
@@ -29,6 +34,24 @@ const Timezones: FC<TimezonesProps> = ({ timezoneState, onChange }) => {
     setFalse: handleCloseList,
   } = useBoolean(false);
 
+  const pinnedTimezones = useMemo(() => [
+    {
+      title: `Default time (${defaultTimezone})`,
+      region: defaultTimezone,
+      utc: defaultTimezone ? getUTCByTimezone(defaultTimezone) : "UTC"
+    },
+    {
+      title: `Browser Time (${dayjs.tz.guess()})`,
+      region: dayjs.tz.guess(),
+      utc: getUTCByTimezone(dayjs.tz.guess())
+    },
+    {
+      title: "UTC (Coordinated Universal Time)",
+      region: "UTC",
+      utc: "UTC"
+    },
+  ].filter(t => t.region) as PinnedTimezone[], [defaultTimezone]);
+
   const searchTimezones = useMemo(() => {
     if (!search) return timezones;
     try {
@@ -40,11 +63,6 @@ const Timezones: FC<TimezonesProps> = ({ timezoneState, onChange }) => {
 
   const timezonesGroups = useMemo(() => Object.keys(searchTimezones), [searchTimezones]);
 
-  const localTimezone = useMemo(() => ({
-    region: dayjs.tz.guess(),
-    utc: getUTCByTimezone(dayjs.tz.guess())
-  }), []);
-
   const activeTimezone = useMemo(() => ({
     region: timezoneState,
     utc: getUTCByTimezone(timezoneState)
@@ -108,13 +126,16 @@ const Timezones: FC<TimezonesProps> = ({ timezoneState, onChange }) => {
                 onChange={handleChangeSearch}
               />
             </div>
-            <div
-              className="vm-timezones-item vm-timezones-list-group-options__item"
-              onClick={createHandlerSetTimezone(localTimezone)}
-            >
-              <div className="vm-timezones-item__title">Browser Time ({localTimezone.region})</div>
-              <div className="vm-timezones-item__utc">{localTimezone.utc}</div>
-            </div>
+            {pinnedTimezones.map((t, i) => t && (
+              <div
+                key={`${i}_${t.region}`}
+                className="vm-timezones-item vm-timezones-list-group-options__item"
+                onClick={createHandlerSetTimezone(t)}
+              >
+                <div className="vm-timezones-item__title">{t.title}</div>
+                <div className="vm-timezones-item__utc">{t.utc}</div>
+              </div>
+            ))}
           </div>
           {timezonesGroups.map(t => (
             <div

app/vmui/packages/vmui/src/components/Configurators/QueryEditor/QueryAutocompleteCache.ts

@@ -24,7 +24,7 @@ export class QueryAutocompleteCache {
       const equalRange = cacheItem.start === key.start && cacheItem.end === key.end;
       const equalType = cacheItem.type === key.type;
       const isIncluded = key.value && cacheItem.value && key.value.includes(cacheItem.value);
-      const isSimilar = cacheItem.match === key.match || isIncluded;
+      const isSimilar = (cacheItem.match === key.match) || isIncluded;
       const isUnderLimit = cacheValue.length < AUTOCOMPLETE_LIMITS.queryLimit;
       if (isSimilar && equalRange && equalType && isUnderLimit) {
         return cacheValue;

app/vmui/packages/vmui/src/components/Configurators/QueryEditor/QueryEditor.tsx

@@ -1,5 +1,5 @@
-import React, { FC, useRef, useState } from "preact/compat";
-import { KeyboardEvent, useEffect } from "react";
+import React, { FC, useEffect, useRef, useState } from "preact/compat";
+import { KeyboardEvent } from "react";
 import { ErrorTypes } from "../../../types";
 import TextField from "../../Main/TextField/TextField";
 import QueryEditorAutocomplete from "./QueryEditorAutocomplete";
@@ -7,7 +7,8 @@ import "./style.scss";
 import { QueryStats } from "../../../api/types";
 import { partialWarning, seriesFetchedWarning } from "./warningText";
 import { AutocompleteOptions } from "../../Main/Autocomplete/Autocomplete";
-import { useQueryDispatch } from "../../../state/query/QueryStateContext";
+import useDeviceDetect from "../../../hooks/useDeviceDetect";
+import { useQueryState } from "../../../state/query/QueryStateContext";
 
 export interface QueryEditorProps {
   onChange: (query: string) => void;
@@ -35,11 +36,12 @@ const QueryEditor: FC<QueryEditorProps> = ({
   label,
   disabled = false
 }) => {
+  const { autocompleteQuick } = useQueryState();
+  const { isMobile } = useDeviceDetect();
 
   const [openAutocomplete, setOpenAutocomplete] = useState(false);
   const [caretPosition, setCaretPosition] = useState([0, 0]);
   const autocompleteAnchorEl = useRef<HTMLInputElement>(null);
-  const queryDispatch = useQueryDispatch();
 
   const warning = [
     {
@@ -103,8 +105,8 @@ const QueryEditor: FC<QueryEditorProps> = ({
   };
 
   useEffect(() => {
-    queryDispatch({ type: "SET_AUTOCOMPLETE_QUICK", payload: false });
-  }, [value]);
+    setOpenAutocomplete(autocomplete);
+  }, [autocompleteQuick]);
 
   return (
     <div
@@ -115,7 +117,7 @@ const QueryEditor: FC<QueryEditorProps> = ({
         value={value}
         label={label}
         type={"textarea"}
-        autofocus={!!value}
+        autofocus={!isMobile}
         error={error}
         warning={warning}
         onKeyDown={handleKeyDown}

app/vmui/packages/vmui/src/components/Configurators/QueryEditor/QueryEditorAutocomplete.tsx

@@ -88,13 +88,6 @@ const QueryEditorAutocomplete: FC<QueryEditorAutocompleteProps> = ({
     const beforeValueByContext = value.substring(0, startIndexOfValueByContext);
     const afterValueByContext = value.substring(endIndexOfValueByContext);
 
-    // Add quotes around the value if the context is labelValue
-    if (context === QueryContextType.labelValue) {
-      const quote = "\"";
-      const needsQuote = !beforeValueByContext.endsWith(quote);
-      insert = `${needsQuote ? quote : ""}${insert}${quote}`;
-    }
-
     // Assemble the new value with the inserted text
     const newVal = `${beforeValueByContext}${insert}${afterValueByContext}`;
     onSelect(newVal);

app/vmui/packages/vmui/src/components/ExploreMetrics/ExploreMetricsHeader/ExploreMetricsHeader.tsx

@@ -71,7 +71,7 @@ const ExploreMetricsHeader: FC<ExploreMetricsHeaderProps> = ({
             label="Job"
             placeholder="Please select job"
             onChange={onChangeJob}
-            autofocus={!job}
+            autofocus={!job && !!jobs.length && !isMobile}
             searchable
           />
         </div>

app/vmui/packages/vmui/src/components/Main/Autocomplete/Autocomplete.tsx

@@ -85,6 +85,10 @@ const Autocomplete: FC<AutocompleteProps> = ({
     }
   }, [openAutocomplete, options, value]);
 
+  const hideFoundedOptions = useMemo(() => {
+    return foundOptions.length === 1 && foundOptions[0]?.value === value;
+  }, [foundOptions]);
+
   const displayNoOptionsText = useMemo(() => {
     return noOptionsText && !foundOptions.length;
   }, [noOptionsText,foundOptions]);
@@ -159,8 +163,8 @@ const Autocomplete: FC<AutocompleteProps> = ({
   }, [openAutocomplete]);
 
   useEffect(() => {
-    onFoundOptions && onFoundOptions(foundOptions);
-  }, [foundOptions]);
+    onFoundOptions && onFoundOptions(hideFoundedOptions ? [] : foundOptions);
+  }, [foundOptions, hideFoundedOptions]);
 
   return (
     <Popper
@@ -181,7 +185,7 @@ const Autocomplete: FC<AutocompleteProps> = ({
         ref={wrapperEl}
       >
         {displayNoOptionsText && <div className="vm-autocomplete__no-options">{noOptionsText}</div>}
-        {!(foundOptions.length === 1 && foundOptions[0]?.value === value) && foundOptions.map((option, i) =>
+        {!hideFoundedOptions && foundOptions.map((option, i) =>
           <div
             className={classNames({
               "vm-list-item": true,

app/vmui/packages/vmui/src/components/Main/Button/Button.tsx

@@ -5,7 +5,7 @@ import "./style.scss";
 
 interface ButtonProps {
   variant?: "contained" | "outlined" | "text"
-  color?: "primary" | "secondary" | "success" | "error" | "gray"  | "warning"
+  color?: "primary" | "secondary" | "success" | "error" | "gray"  | "warning" | "white"
   size?: "small" | "medium" | "large"
   ariaLabel?: string // https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/Attributes/aria-label
   endIcon?: ReactNode

app/vmui/packages/vmui/src/components/Main/Button/style.scss

@@ -9,7 +9,7 @@ $button-radius: 6px;
   justify-content: center;
   padding: 6px 14px;
   font-size: $font-size-small;
-  line-height: 1.3;
+  line-height: calc($font-size + 1px);
   font-weight: normal;
   min-height: 31px;
   border-radius: $button-radius;
@@ -56,7 +56,7 @@ $button-radius: 6px;
     transform: translateZ(1px);
 
     svg {
-      width: 15px;
+      width: calc($font-size + 1px);
     }
   }
 
@@ -180,6 +180,10 @@ $button-radius: 6px;
     color: $color-text-secondary;
   }
 
+  &_text_white {
+    color: $color-white;
+  }
+
   &_text_warning {
     color: $color-warning;
   }
@@ -211,6 +215,11 @@ $button-radius: 6px;
     color: $color-text-secondary;
   }
 
+  &_outlined_white {
+    border: 1px solid $color-white;
+    color: $color-white;
+  }
+
   &_outlined_warning {
     border: 1px solid $color-warning;
     color: $color-warning;

app/vmui/packages/vmui/src/components/Main/Icons/index.tsx

@@ -511,3 +511,12 @@ export const ValueIcon = () => (
     />
   </svg>
 );
+
+export const DownloadIcon = () => (
+  <svg
+    viewBox="0 0 24 24"
+    fill="currentColor"
+  >
+    <path d="M19 9h-4V3H9v6H5l7 7 7-7zM5 18v2h14v-2H5z"></path>
+  </svg>
+);

app/vmui/packages/vmui/src/components/Main/Popper/Popper.tsx

@@ -23,6 +23,7 @@ interface PopperProps {
   fullWidth?: boolean
   title?: string
   disabledFullScreen?: boolean
+  variant?: "default" | "dark"
 }
 
 const Popper: FC<PopperProps> = ({
@@ -35,7 +36,8 @@ const Popper: FC<PopperProps> = ({
   clickOutside = true,
   fullWidth,
   title,
-  disabledFullScreen
+  disabledFullScreen,
+  variant
 }) => {
   const { isMobile } = useDeviceDetect();
   const navigate = useNavigate();
@@ -52,18 +54,16 @@ const Popper: FC<PopperProps> = ({
 
   useEffect(() => {
     setIsOpen(open);
-  }, [open]);
 
-  useEffect(() => {
-    if (!isOpen && onClose) onClose();
-    if (isOpen && isMobile && !disabledFullScreen) {
+    if (!open && onClose) onClose();
+    if (open && isMobile && !disabledFullScreen) {
       document.body.style.overflow = "hidden";
     }
 
     return () => {
       document.body.style.overflow = "auto";
     };
-  }, [isOpen]);
+  }, [open]);
 
   useEffect(() => {
     setPopperSize({
@@ -149,6 +149,7 @@ const Popper: FC<PopperProps> = ({
         <div
           className={classNames({
             "vm-popper": true,
+            [`vm-popper_${variant}`]: variant,
             "vm-popper_mobile": isMobile && !disabledFullScreen,
             "vm-popper_open": (isMobile || Object.keys(popperStyle).length) && isOpen,
           })}
@@ -160,6 +161,7 @@ const Popper: FC<PopperProps> = ({
               <p className="vm-popper-header__title">{title}</p>
               <Button
                 variant="text"
+                color={variant === "dark" ? "white" : "primary"}
                 size="small"
                 onClick={handleClickClose}
                 ariaLabel="close"

app/vmui/packages/vmui/src/components/Main/Popper/style.scss

@@ -49,6 +49,16 @@
       user-select: none;
     }
   }
+
+  &_dark {
+    background-color: $color-background-tooltip;
+    color: $color-white;
+  }
+
+  &_dark &-header {
+    background-color: transparent;
+    color: $color-white;
+  }
 }
 
 @keyframes vm-slider {

app/vmui/packages/vmui/src/components/TraceQuery/NestedNav/NestedNav.tsx

@@ -14,13 +14,14 @@ interface RecursiveProps {
   isRoot?: boolean;
   trace: Trace;
   totalMsec: number;
+  isExpandedAll? : boolean;
 }
 
 interface OpenLevels {
   [x: number]: boolean
 }
 
-const NestedNav: FC<RecursiveProps> = ({ isRoot, trace, totalMsec })  => {
+const NestedNav: FC<RecursiveProps> = ({ isRoot, trace, totalMsec, isExpandedAll })  => {
   const { isDarkTheme } = useAppState();
   const { isMobile } = useDeviceDetect();
   const [openLevels, setOpenLevels] = useState({} as OpenLevels);
@@ -53,6 +54,26 @@ const NestedNav: FC<RecursiveProps> = ({ isRoot, trace, totalMsec })  => {
     });
   };
 
+  const getIdsFromChildren = (tracingData: Trace) => {
+    const ids = [tracingData.idValue];
+    tracingData?.children?.forEach((child) => {
+      ids.push(...getIdsFromChildren(child));
+    });
+    return ids;
+  };
+
+  useEffect(() => {
+    if (!isExpandedAll) {
+      setOpenLevels([]);
+      return;
+    }
+
+    const allIds = getIdsFromChildren(trace);
+    const openLevels = {} as OpenLevels;
+    allIds.forEach(id => { openLevels[id] = true; });
+    setOpenLevels(openLevels);
+  }, [isExpandedAll]);
+
   return (
     <div
       className={classNames({
@@ -106,13 +127,14 @@ const NestedNav: FC<RecursiveProps> = ({ isRoot, trace, totalMsec })  => {
           )}
         </div>
       </div>
-      {openLevels[trace.idValue] && (
+      {(openLevels[trace.idValue]) && (
         <div className="vm-nested-nav__childrens">
           {hasChildren && trace.children.map((trace) => (
             <NestedNav
               key={trace.duration}
               trace={trace}
               totalMsec={totalMsec}
+              isExpandedAll={isExpandedAll}
             />
           ))}
         </div>

app/vmui/packages/vmui/src/components/TraceQuery/NestedNav/style.scss

@@ -2,6 +2,8 @@
 
 $color-base-nested-nav: $color-tropical-blue;
 $color-base-nested-nav-dark: $color-background-body;
+$width-line: 2px;
+$left-position: calc(-1 * $padding-small);
 
 .vm-nested-nav {
   position: relative;
@@ -48,19 +50,19 @@ $color-base-nested-nav-dark: $color-background-body;
       content: "";
       position: absolute;
       top: calc(50% - 1px);
-      height: 2px;
+      height: $width-line;
       width: $padding-small;
       background-color: $color-base-nested-nav;
-      left: calc(-1 * $padding-small);
+      left: $left-position;
     }
 
     &:before {
       content: "";
       position: absolute;
       bottom: 50%;
-      left: calc(-1 * $padding-small);
+      left: $left-position;
       height: calc(50% + $padding-small);
-      width: 2px;
+      width: $width-line;
       background-color: $color-base-nested-nav;
     }
 
@@ -122,9 +124,9 @@ $color-base-nested-nav-dark: $color-background-body;
       content: "";
       position: absolute;
       top: 0;
-      left: calc(-1 * $padding-small);
+      left: $left-position;
       height: 100%;
-      width: 2px;
+      width: $width-line;
       background-color: $color-base-nested-nav;
     }
   }

app/vmui/packages/vmui/src/components/TraceQuery/TracingsView.tsx

@@ -1,7 +1,7 @@
 import React, { FC, useState } from "preact/compat";
 import Trace from "./Trace";
 import Button from "../Main/Button/Button";
-import { CodeIcon, DeleteIcon } from "../Main/Icons";
+import { ArrowDownIcon, CodeIcon, DeleteIcon, DownloadIcon } from "../Main/Icons";
 import "./style.scss";
 import NestedNav from "./NestedNav/NestedNav";
 import Alert from "../Main/Alert/Alert";
@@ -20,6 +20,7 @@ interface TraceViewProps {
 const TracingsView: FC<TraceViewProps> = ({ traces, jsonEditor = false, onDeleteClick }) => {
   const { isMobile } = useDeviceDetect();
   const [openTrace, setOpenTrace] = useState<Trace | null>(null);
+  const [expandedTraces, setExpandedTraces] = useState<number[]>([]);
 
   const handleCloseJson = () => {
     setOpenTrace(null);
@@ -52,6 +53,27 @@ const TracingsView: FC<TraceViewProps> = ({ traces, jsonEditor = false, onDelete
     setOpenTrace(tracingData);
   };
 
+  const handleSaveToFile = (tracingData: Trace) => () => {
+    const blob = new Blob([tracingData.originalJSON], { type: "application/json" });
+    const href = URL.createObjectURL(blob);
+
+    const link = document.createElement("a");
+    link.href = href;
+    link.download = `vmui_trace_${tracingData.queryValue}.json`;
+    document.body.appendChild(link);
+    link.click();
+
+    document.body.removeChild(link);
+    URL.revokeObjectURL(href);
+  };
+
+  const handleExpandAll = (tracingData: Trace) => () => {
+    setExpandedTraces(prev => prev.includes(tracingData.idValue)
+      ? prev.filter(n => n !== tracingData.idValue)
+      : [...prev, tracingData.idValue]
+    );
+  };
+
   return (
     <>
       <div className="vm-tracings-view">
@@ -64,6 +86,28 @@ const TracingsView: FC<TraceViewProps> = ({ traces, jsonEditor = false, onDelete
               <h3 className="vm-tracings-view-trace-header-title">
               Trace for <b className="vm-tracings-view-trace-header-title__query">{trace.queryValue}</b>
               </h3>
+              <Tooltip title={expandedTraces.includes(trace.idValue) ? "Collapse All" : "Expand All"}>
+                <Button
+                  variant="text"
+                  startIcon={(
+                    <div
+                      className={classNames({
+                        "vm-tracings-view-trace-header__expand-icon": true,
+                        "vm-tracings-view-trace-header__expand-icon_open": expandedTraces.includes(trace.idValue) })}
+                    ><ArrowDownIcon/></div>
+                  )}
+                  onClick={handleExpandAll(trace)}
+                  ariaLabel={expandedTraces.includes(trace.idValue) ? "Collapse All" : "Expand All"}
+                />
+              </Tooltip>
+              <Tooltip title={"Save Trace to JSON"}>
+                <Button
+                  variant="text"
+                  startIcon={<DownloadIcon/>}
+                  onClick={handleSaveToFile(trace)}
+                  ariaLabel="Save trace to JSON"
+                />
+              </Tooltip>
               <Tooltip title={"Open JSON"}>
                 <Button
                   variant="text"
@@ -92,6 +136,7 @@ const TracingsView: FC<TraceViewProps> = ({ traces, jsonEditor = false, onDelete
                 isRoot
                 trace={trace}
                 totalMsec={trace.duration}
+                isExpandedAll={expandedTraces.includes(trace.idValue)}
               />
             </nav>
           </div>

app/vmui/packages/vmui/src/components/TraceQuery/style.scss

@@ -21,6 +21,20 @@
           font-weight: bold;
         }
       }
+
+      &__expand-icon {
+        display: flex;
+        align-items: center;
+        justify-content: center;
+        width: 20px;
+        transition: transform 200ms ease-in-out;
+        transform: rotate(-90deg);
+        color: $color-text-secondary;
+
+        &_open {
+          transform: rotate(0);
+        }
+      }
     }
 
     &__nav {

app/vmui/packages/vmui/src/components/UploadJsonButtons/UploadJsonButtons.tsx

@@ -0,0 +1,33 @@
+import React, { FC } from "preact/compat";
+import { ChangeEvent } from "react";
+import Button from "../Main/Button/Button";
+import "./style.scss";
+
+interface Props {
+  onOpenModal: () => void;
+  onChange: (e: ChangeEvent<HTMLInputElement>) => void;
+}
+
+const UploadJsonButtons: FC<Props> = ({ onOpenModal, onChange }) => (
+  <div className="vm-upload-json-buttons">
+    <Button
+      variant="outlined"
+      onClick={onOpenModal}
+    >
+      Paste JSON
+    </Button>
+    <Button>
+      Upload Files
+      <input
+        id="json"
+        type="file"
+        accept="application/json"
+        multiple
+        title=" "
+        onChange={onChange}
+      />
+    </Button>
+  </div>
+);
+
+export default UploadJsonButtons;

app/vmui/packages/vmui/src/components/UploadJsonButtons/style.scss

@@ -0,0 +1,9 @@
+@use "src/styles/variables" as *;
+
+.vm-upload-json-buttons {
+  display: grid;
+  grid-template-columns: 1fr 1fr;
+  gap: $padding-global;
+  align-items: center;
+  justify-content: center;
+}

app/vmui/packages/vmui/src/constants/date.ts

@@ -2,3 +2,4 @@ export const DATE_FORMAT = "YYYY-MM-DD";
 export const DATE_TIME_FORMAT = "YYYY-MM-DD HH:mm:ss";
 export const DATE_FULL_TIMEZONE_FORMAT = "YYYY-MM-DD HH:mm:ss:SSS (Z)";
 export const DATE_ISO_FORMAT = "YYYY-MM-DD[T]HH:mm:ss";
+export const DATE_FILENAME_FORMAT = "YYYY-MM-DD_HHmmss";

app/vmui/packages/vmui/src/constants/navigation.ts

@@ -36,6 +36,10 @@ const tools = {
       label: routerOptions[router.trace].title,
       value: router.trace,
     },
+    {
+      label: routerOptions[router.queryAnalyzer].title,
+      value: router.queryAnalyzer,
+    },
     {
       label: routerOptions[router.withTemplate].title,
       value: router.withTemplate,

app/vmui/packages/vmui/src/hooks/useFetchDefaultTimezone.ts

@@ -0,0 +1,59 @@
+import { useEffect, useState } from "preact/compat";
+import { ErrorTypes } from "../types";
+import { useAppState } from "../state/common/StateContext";
+import { useTimeDispatch } from "../state/time/TimeStateContext";
+import { getFromStorage } from "../utils/storage";
+import dayjs from "dayjs";
+
+const disabledDefaultTimezone = Boolean(getFromStorage("DISABLED_DEFAULT_TIMEZONE"));
+
+const useFetchDefaultTimezone = () => {
+  const { serverUrl } = useAppState();
+  const timeDispatch = useTimeDispatch();
+
+  const [isLoading, setIsLoading] = useState(false);
+  const [error, setError] = useState<ErrorTypes | string>("");
+
+  const setTimezone = (timezoneStr: string) => {
+    const timezone = timezoneStr.toLowerCase() === "local" ?  dayjs.tz.guess() : timezoneStr;
+    try {
+      dayjs().tz(timezone).isValid();
+      timeDispatch({ type: "SET_DEFAULT_TIMEZONE", payload: timezone });
+      if (disabledDefaultTimezone) return;
+      timeDispatch({ type: "SET_TIMEZONE", payload: timezone });
+    } catch (e) {
+      if (e instanceof Error) setError(`${e.name}: ${e.message}`);
+    }
+  };
+
+  const fetchDefaultTimezone = async () => {
+    if (!serverUrl || process.env.REACT_APP_TYPE) return;
+    setError("");
+    setIsLoading(true);
+
+    try {
+      const response = await fetch(`${serverUrl}/vmui/timezone`);
+      const resp = await response.json();
+
+      if (response.ok) {
+        setTimezone(resp.timezone);
+        setIsLoading(false);
+      } else {
+        setError(resp.error);
+        setIsLoading(false);
+      }
+    } catch (e) {
+      setIsLoading(false);
+      if (e instanceof Error) setError(`${e.name}: ${e.message}`);
+    }
+  };
+
+  useEffect(() => {
+    fetchDefaultTimezone();
+  }, [serverUrl]);
+
+  return { isLoading, error };
+};
+
+export default useFetchDefaultTimezone;
+

app/vmui/packages/vmui/src/hooks/useFetchQuery.ts

@@ -212,5 +212,17 @@ export const useFetchQuery = ({
     if (defaultStep === customStep) setGraphData([]);
   }, [isHistogram]);
 
-  return { fetchUrl, isLoading, graphData, liveData, error, queryErrors, setQueryErrors, queryStats, warning, traces, isHistogram };
+  return {
+    fetchUrl,
+    isLoading,
+    graphData,
+    liveData,
+    error,
+    queryErrors,
+    setQueryErrors,
+    queryStats,
+    warning,
+    traces,
+    isHistogram
+  };
 };

app/vmui/packages/vmui/src/hooks/useFetchQueryOptions.tsx

@@ -10,6 +10,7 @@ import { useQueryDispatch, useQueryState } from "../state/query/QueryStateContex
 import { QueryContextType } from "../types";
 import { AUTOCOMPLETE_LIMITS } from "../constants/queryAutocomplete";
 import { escapeDoubleQuotes, escapeRegexp } from "../utils/regexp";
+import dayjs from "dayjs";
 
 enum TypeData {
   metric = "metric",
@@ -59,11 +60,14 @@ export const useFetchQueryOptions = ({ valueByContext, metric, label, context }:
   const abortControllerRef = useRef(new AbortController());
 
   const getQueryParams = useCallback((params?: Record<string, string>) => {
+    const startDay = dayjs(start * 1000).startOf("day").valueOf() / 1000;
+    const endDay = dayjs(end * 1000).endOf("day").valueOf() / 1000;
+
     return new URLSearchParams({
       ...(params || {}),
       limit: `${AUTOCOMPLETE_LIMITS.queryLimit}`,
-      start: `${start}`,
-      end: `${end}`
+      start: `${startDay}`,
+      end: `${endDay}`
     });
   }, [start, end]);
 
@@ -76,6 +80,7 @@ export const useFetchQueryOptions = ({ valueByContext, metric, label, context }:
   };
 
   const fetchData = async ({ value, urlSuffix, setter, type, params }: FetchDataArgs) => {
+    if (!value) return;
     abortControllerRef.current.abort();
     abortControllerRef.current = new AbortController();
     const { signal } = abortControllerRef.current;

app/vmui/packages/vmui/src/layouts/AnomalyLayout/AnomalyLayout.tsx

@@ -7,7 +7,7 @@ import { getAppModeEnable } from "../../utils/app-mode";
 import classNames from "classnames";
 import Footer from "../Footer/Footer";
 import { routerOptions } from "../../router";
-import { useFetchDashboards } from "../../pages/PredefinedPanels/hooks/useFetchDashboards";
+import useFetchDefaultTimezone from "../../hooks/useFetchDefaultTimezone";
 import useDeviceDetect from "../../hooks/useDeviceDetect";
 import ControlsAnomalyLayout from "./ControlsAnomalyLayout";
 
@@ -17,7 +17,7 @@ const AnomalyLayout: FC = () => {
   const { pathname } = useLocation();
   const [searchParams, setSearchParams] = useSearchParams();
 
-  useFetchDashboards();
+  useFetchDefaultTimezone();
 
   const setDocumentTitle = () => {
     const defaultTitle = "vmui for vmanomaly";

app/vmui/packages/vmui/src/layouts/LogsLayout/LogsLayout.tsx

@@ -8,12 +8,15 @@ import Footer from "../Footer/Footer";
 import router, { routerOptions } from "../../router";
 import useDeviceDetect from "../../hooks/useDeviceDetect";
 import ControlsLogsLayout from "./ControlsLogsLayout";
+import useFetchDefaultTimezone from "../../hooks/useFetchDefaultTimezone";
 
 const LogsLayout: FC = () => {
   const appModeEnable = getAppModeEnable();
   const { isMobile } = useDeviceDetect();
   const { pathname } = useLocation();
 
+  useFetchDefaultTimezone();
+
   const setDocumentTitle = () => {
     const defaultTitle = "vmui for VictoriaLogs";
     const routeTitle = routerOptions[router.logs]?.title;

app/vmui/packages/vmui/src/layouts/MainLayout/MainLayout.tsx

@@ -10,6 +10,7 @@ import { routerOptions } from "../../router";
 import { useFetchDashboards } from "../../pages/PredefinedPanels/hooks/useFetchDashboards";
 import useDeviceDetect from "../../hooks/useDeviceDetect";
 import ControlsMainLayout from "./ControlsMainLayout";
+import useFetchDefaultTimezone from "../../hooks/useFetchDefaultTimezone";
 
 const MainLayout: FC = () => {
   const appModeEnable = getAppModeEnable();
@@ -18,6 +19,7 @@ const MainLayout: FC = () => {
   const [searchParams, setSearchParams] = useSearchParams();
 
   useFetchDashboards();
+  useFetchDefaultTimezone();
 
   const setDocumentTitle = () => {
     const defaultTitle = "vmui";

app/vmui/packages/vmui/src/pages/CustomPanel/DownloadReport/DownloadReport.tsx

@@ -0,0 +1,225 @@
+import React, { FC, useCallback, useEffect, useMemo, useRef, useState } from "preact/compat";
+import { DownloadIcon } from "../../../components/Main/Icons";
+import Button from "../../../components/Main/Button/Button";
+import Tooltip from "../../../components/Main/Tooltip/Tooltip";
+import useBoolean from "../../../hooks/useBoolean";
+import "./style.scss";
+import Checkbox from "../../../components/Main/Checkbox/Checkbox";
+import Modal from "../../../components/Main/Modal/Modal";
+import dayjs from "dayjs";
+import { DATE_FILENAME_FORMAT } from "../../../constants/date";
+import TextField from "../../../components/Main/TextField/TextField";
+import { useQueryState } from "../../../state/query/QueryStateContext";
+import { ErrorTypes } from "../../../types";
+import Alert from "../../../components/Main/Alert/Alert";
+import qs from "qs";
+import Popper from "../../../components/Main/Popper/Popper";
+import helperText from "./helperText";
+
+type Props = {
+  fetchUrl?: string[];
+}
+
+const getDefaultReportName = () => `vmui_report_${dayjs().utc().format(DATE_FILENAME_FORMAT)}`;
+
+const DownloadReport: FC<Props> = ({ fetchUrl }) => {
+  const { query } = useQueryState();
+
+  const [filename, setFilename] = useState(getDefaultReportName());
+  const [comment, setComment] = useState("");
+  const [trace, setTrace] = useState(true);
+  const [error, setError] = useState<ErrorTypes | string>();
+  const [isLoading, setIsLoading] = useState(false);
+
+  const filenameRef = useRef<HTMLDivElement>(null);
+  const commentRef = useRef<HTMLDivElement>(null);
+  const traceRef = useRef<HTMLDivElement>(null);
+  const generateRef = useRef<HTMLDivElement>(null);
+  const helperRefs = [filenameRef, commentRef, traceRef, generateRef];
+  const [stepHelper, setStepHelper] = useState(0);
+
+  const {
+    value: openModal,
+    toggle: toggleOpen,
+    setFalse: handleClose,
+  } = useBoolean(false);
+
+  const {
+    value: openHelper,
+    toggle: toggleHelper,
+    setFalse: handleCloseHelper,
+  } = useBoolean(false);
+
+  const fetchUrlReport = useMemo(() => {
+    if (!fetchUrl) return;
+    return fetchUrl.map((str, i) => {
+      const url = new URL(str);
+      trace ? url.searchParams.set("trace", "1") : url.searchParams.delete("trace");
+      return { id: i, url: url };
+    });
+  }, [fetchUrl, trace]);
+
+  const generateFile = useCallback((data: unknown) => {
+    const json = JSON.stringify(data, null, 2);
+    const blob = new Blob([json], { type: "application/json" });
+    const href = URL.createObjectURL(blob);
+
+    const link = document.createElement("a");
+    link.href = href;
+    link.download = `${filename || getDefaultReportName()}.json`;
+    document.body.appendChild(link);
+    link.click();
+
+    document.body.removeChild(link);
+    URL.revokeObjectURL(href);
+    handleClose();
+  }, [filename]);
+
+  const handleGenerateReport = useCallback(async () => {
+    if (!fetchUrlReport) {
+      setError(ErrorTypes.validQuery);
+      return;
+    }
+
+    setError("");
+    setIsLoading(true);
+
+    try {
+      const result = [];
+      for await (const { url, id } of fetchUrlReport) {
+        const response = await fetch(url);
+        const resp = await response.json();
+        if (response.ok) {
+          resp.vmui = {
+            id,
+            comment,
+            params: qs.parse(new URL(url).search.replace(/^\?/, ""))
+          };
+          result.push(resp);
+        } else {
+          const errorType = resp.errorType ? `${resp.errorType}\r\n` : "";
+          setError(`${errorType}${resp?.error || resp?.message || "unknown error"}`);
+        }
+      }
+      result.length && generateFile(result);
+    } catch (e) {
+      if (e instanceof Error && e.name !== "AbortError") {
+        setError(`${e.name}: ${e.message}`);
+      }
+    } finally {
+      setIsLoading(false);
+    }
+  }, [fetchUrlReport, comment, generateFile, query]);
+
+  const handleChangeHelp = (step: number) => () => {
+    setStepHelper(prevStep => prevStep + step);
+  };
+
+  useEffect(() => {
+    setError("");
+    setFilename(getDefaultReportName());
+    setComment("");
+  }, [openModal]);
+
+  useEffect(() => {
+    setStepHelper(0);
+  }, [openHelper]);
+
+  return (
+    <>
+      <Tooltip title={"Export query"}>
+        <Button
+          variant="text"
+          startIcon={<DownloadIcon/>}
+          onClick={toggleOpen}
+          ariaLabel="export query"
+        />
+      </Tooltip>
+      {openModal && (
+        <Modal
+          title={"Export query"}
+          onClose={handleClose}
+          isOpen={openModal}
+        >
+          <div className="vm-download-report">
+            <div className="vm-download-report-settings">
+              <div ref={filenameRef}>
+                <TextField
+                  label="Filename"
+                  value={filename}
+                  onChange={setFilename}
+                />
+              </div>
+              <div ref={commentRef}>
+                <TextField
+                  type="textarea"
+                  label="Comment"
+                  value={comment}
+                  onChange={setComment}
+                />
+              </div>
+              <div ref={traceRef}>
+                <Checkbox
+                  checked={trace}
+                  onChange={setTrace}
+                  label={"Include query trace"}
+                />
+              </div>
+            </div>
+            {error && <Alert variant="error">{error}</Alert>}
+            <div className="vm-download-report__buttons">
+              <Button
+                variant="text"
+                onClick={toggleHelper}
+              >
+                Help
+              </Button>
+              <div ref={generateRef}>
+                <Button
+                  onClick={handleGenerateReport}
+                  disabled={isLoading}
+                >
+                  {isLoading ? "Loading data..." : "Generate Report"}
+                </Button>
+              </div>
+            </div>
+            <Popper
+              open={openHelper}
+              buttonRef={helperRefs[stepHelper]}
+              placement="top-left"
+              variant="dark"
+              onClose={handleCloseHelper}
+            >
+              <div className="vm-download-report-helper">
+                <div className="vm-download-report-helper__description">
+                  {helperText[stepHelper]}
+                </div>
+                <div className="vm-download-report-helper__buttons">
+                  {stepHelper !== 0 && (
+                    <Button
+                      onClick={handleChangeHelp(-1)}
+                      size="small"
+                      color={"white"}
+                    >
+                      Prev
+                    </Button>
+                  )}
+                  <Button
+                    onClick={stepHelper === helperRefs.length - 1 ? handleCloseHelper : handleChangeHelp(1)}
+                    size="small"
+                    color={"white"}
+                    variant={"text"}
+                  >
+                    {stepHelper === helperRefs.length - 1 ? "Close" : "Next"}
+                  </Button>
+                </div>
+              </div>
+            </Popper>
+          </div>
+        </Modal>
+      )}
+    </>
+  );
+};
+
+export default DownloadReport;

app/vmui/packages/vmui/src/pages/CustomPanel/DownloadReport/helperText.tsx

@@ -0,0 +1,45 @@
+import React from "preact/compat";
+import { DATE_FILENAME_FORMAT } from "../../../constants/date";
+import router, { routerOptions } from "../../../router";
+import { Link } from "react-router-dom";
+
+const filename = (
+  <>
+    <p>Filename - specify the name for your report file.</p>
+    <p>Default format: <code>vmui_report_${DATE_FILENAME_FORMAT}.json</code>.</p>
+    <p>This name will be used when saving your report on your device.</p>
+  </>
+);
+
+const comment = (
+  <>
+    <p>Comment (optional) - add a comment to your report.</p>
+    <p>This can be any additional information that will be useful when reviewing the report later.</p>
+  </>
+);
+
+const trace = (
+  <>
+    <p>Query trace - enable this option to include a query trace in your report.</p>
+    <p>This will assist in analyzing and diagnosing the query processing.</p>
+  </>
+);
+
+const generate = (
+  <>
+    <p>Generate Report - click this button to generate and save your report. </p>
+    <p>After creation, the report can be downloaded and examined on the <Link
+      to={router.queryAnalyzer}
+      target="_blank"
+      rel="noreferrer"
+      className="vm-link vm-link_underlined"
+    >{routerOptions[router.queryAnalyzer].title}</Link> page.</p>
+  </>
+);
+
+export default [
+  filename,
+  comment,
+  trace,
+  generate,
+];

app/vmui/packages/vmui/src/pages/CustomPanel/DownloadReport/style.scss

@@ -0,0 +1,47 @@
+@use "src/styles/variables" as *;
+
+.vm-download-report {
+  display: grid;
+  gap: $padding-large;
+  padding-top: calc($padding-large - $padding-global);
+  min-width: 400px;
+
+  &-settings {
+    display: grid;
+    gap: $padding-global;
+
+    textarea {
+      min-height: 100px;
+    }
+  }
+
+  &__buttons {
+    display: flex;
+    align-items: center;
+    justify-content: flex-end;
+    gap: $padding-global;
+  }
+
+  &-helper {
+    display: grid;
+    gap: $padding-small;
+    padding: $padding-global;
+
+    &__description {
+      max-width: 400px;
+      white-space: pre-line;
+      line-height: 1.3;
+
+      p {
+        margin-bottom: calc($padding-small/2);
+      }
+    }
+
+    &__buttons {
+      display: flex;
+      align-items: center;
+      justify-content: flex-end;
+      gap: $padding-small;
+    }
+  }
+}

app/vmui/packages/vmui/src/pages/CustomPanel/QueryConfigurator/QueryConfigurator.tsx

@@ -128,6 +128,7 @@ const QueryConfigurator: FC<QueryConfiguratorProps> = ({
 
   const createHandlerChangeQuery = (i: number) => (value: string) => {
     handleChangeQuery(value, i);
+    queryDispatch({ type: "SET_AUTOCOMPLETE_QUICK", payload: false });
   };
 
   const createHandlerRemoveQuery = (i: number) => () => {

app/vmui/packages/vmui/src/pages/CustomPanel/index.tsx

@@ -18,6 +18,7 @@ import CustomPanelTraces from "./CustomPanelTraces/CustomPanelTraces";
 import WarningLimitSeries from "./WarningLimitSeries/WarningLimitSeries";
 import CustomPanelTabs from "./CustomPanelTabs";
 import { DisplayType } from "../../types";
+import DownloadReport from "./DownloadReport/DownloadReport";
 
 const CustomPanel: FC = () => {
   useSetQueryParams();
@@ -35,6 +36,7 @@ const CustomPanel: FC = () => {
   const controlsRef = useRef<HTMLDivElement>(null);
 
   const {
+    fetchUrl,
     isLoading,
     liveData,
     graphData,
@@ -111,7 +113,10 @@ const CustomPanel: FC = () => {
           className="vm-custom-panel-body-header"
           ref={controlsRef}
         >
-          {<DisplayTypeSwitch/>}
+          <div className="vm-custom-panel-body-header__tabs">
+            <DisplayTypeSwitch/>
+          </div>
+          {(graphData || liveData) && <DownloadReport fetchUrl={fetchUrl}/>}
         </div>
         <CustomPanelTabs
           graphData={graphData}

app/vmui/packages/vmui/src/pages/CustomPanel/style.scss

@@ -33,13 +33,19 @@
       position: relative;
       display: flex;
       align-items: center;
-      justify-content: space-between;
+      justify-content: flex-start;
       font-size: $font-size-small;
       margin: -$padding-medium 0-$padding-medium $padding-medium;
       padding: 0 $padding-medium;
       border-bottom: $border-divider;
       z-index: 1;
 
+      &__tabs {
+        display: flex;
+        justify-content: flex-start;
+        flex-grow: 1;
+      }
+
       &__graph-controls {
         display: flex;
         align-items: center;

app/vmui/packages/vmui/src/pages/QueryAnalyzer/JsonForm/JsonForm.tsx

@@ -0,0 +1,77 @@
+import React, { FC, useState, useMemo } from "preact/compat";
+import TextField from "../../../components/Main/TextField/TextField";
+import "./style.scss";
+import Button from "../../../components/Main/Button/Button";
+import classNames from "classnames";
+import useDeviceDetect from "../../../hooks/useDeviceDetect";
+
+interface JsonFormProps {
+  onUpload: (json: string) => void
+  onClose: () => void
+}
+
+const JsonForm: FC<JsonFormProps> = ({ onClose, onUpload }) => {
+  const { isMobile } = useDeviceDetect();
+
+  const [json, setJson] = useState("");
+  const [error, setError] = useState("");
+
+  const errorJson = useMemo(() => {
+    try {
+      JSON.parse(json);
+      return "";
+    } catch (e) {
+      return e instanceof Error ? e.message : "Unknown error";
+    }
+  }, [json]);
+
+  const handleChangeJson = (val: string) => {
+    setError("");
+    setJson(val);
+  };
+
+  const handleApply = () => {
+    setError(errorJson);
+    if (errorJson) return;
+    onUpload(json);
+    onClose();
+  };
+
+  return (
+    <div
+      className={classNames({
+        "vm-json-form vm-json-form_one-field": true,
+        "vm-json-form_mobile vm-json-form_one-field_mobile": isMobile,
+      })}
+    >
+      <TextField
+        value={json}
+        label="JSON"
+        type="textarea"
+        error={error}
+        autofocus
+        onChange={handleChangeJson}
+        onEnter={handleApply}
+      />
+      <div className="vm-json-form-footer">
+        <div className="vm-json-form-footer__controls vm-json-form-footer__controls_right">
+          <Button
+            variant="outlined"
+            color="error"
+            onClick={onClose}
+          >
+            Cancel
+          </Button>
+          <Button
+            variant="contained"
+            onClick={handleApply}
+          >
+            apply
+          </Button>
+        </div>
+      </div>
+    </div>
+  );
+};
+
+export default JsonForm;

app/vmui/packages/vmui/src/pages/QueryAnalyzer/JsonForm/style.scss

@@ -0,0 +1,73 @@
+@use "src/styles/variables" as *;
+
+.vm-json-form {
+  display: grid;
+  grid-template-rows: auto calc(($vh * 70) - 78px - ($padding-medium*3)) auto;
+  gap: $padding-global;
+  width: 70vw;
+  max-width: 1000px;
+  max-height: 900px;
+  overflow: hidden;
+
+  &_mobile {
+    width: 100%;
+    min-height: 100%;
+    grid-template-rows: auto calc(($vh * 100) - 200px - ($padding-global*3)) auto;
+  }
+
+  &_one-field {
+    grid-template-rows: calc(($vh * 70) - 78px - ($padding-medium*3)) auto;
+
+    &_mobile {
+      grid-template-rows: calc(($vh * 100) - 160px - ($padding-global*2)) auto;
+    }
+  }
+
+  textarea {
+    overflow: auto;
+    width: 100%;
+    height: 100%;
+    max-height: 900px;
+  }
+
+  &-footer {
+    display: flex;
+    align-items: center;
+    justify-content: space-between;
+    gap: $padding-small;
+
+    @media (max-width: 500px) {
+      flex-direction: column;
+
+      button {
+        flex-grow: 1;
+      }
+    }
+
+    &__controls {
+      flex-grow: 1;
+      display: flex;
+      align-items: center;
+      justify-content: flex-start;
+      gap: $padding-small;
+
+      @media (max-width: 500px) {
+        grid-template-columns: repeat(2, 1fr);
+        justify-content: center;
+        width: 100%;
+      }
+
+      &_right {
+        display: grid;
+        grid-template-columns: repeat(2, 90px);
+        justify-content: flex-end;
+
+        @media (max-width: 500px) {
+          grid-template-columns: repeat(2, 1fr);
+          justify-content: center;
+          width: 100%;
+        }
+      }
+    }
+  }
+}

app/vmui/packages/vmui/src/pages/QueryAnalyzer/QueryAnalyzerInfo/QueryAnalyzerInfo.tsx

@@ -0,0 +1,97 @@
+import React, { FC, useMemo } from "preact/compat";
+import { DataAnalyzerType } from "../index";
+import Button from "../../../components/Main/Button/Button";
+import { ClockIcon, InfoIcon, TimelineIcon } from "../../../components/Main/Icons";
+import useBoolean from "../../../hooks/useBoolean";
+import Modal from "../../../components/Main/Modal/Modal";
+import { TimeParams } from "../../../types";
+import "./style.scss";
+import dayjs from "dayjs";
+import { DATE_TIME_FORMAT } from "../../../constants/date";
+
+type Props = {
+  data: DataAnalyzerType[];
+  period?: TimeParams;
+}
+
+const QueryAnalyzerInfo: FC<Props> = ({ data, period }) => {
+  const dataWithStats = useMemo(() => data.filter(d => d.stats && d.data.resultType === "matrix"), [data]);
+  const comment = useMemo(() => data.find(d => d?.vmui?.comment)?.vmui?.comment, [data]);
+
+  const timeRange = useMemo(() => {
+    if (!period) return "";
+    const start = dayjs(period.start * 1000).tz().format(DATE_TIME_FORMAT);
+    const end = dayjs(period.end * 1000).tz().format(DATE_TIME_FORMAT);
+    return `${start} - ${end}`;
+  }, [period]);
+
+  const {
+    value: openModal,
+    setTrue: handleOpenModal,
+    setFalse: handleCloseModal,
+  } = useBoolean(false);
+
+  return (
+    <>
+      <div className="vm-query-analyzer-info-header">
+        <Button
+          startIcon={<InfoIcon/>}
+          variant="outlined"
+          color="warning"
+          onClick={handleOpenModal}
+        >
+            Show report info
+        </Button>
+        {period && (
+          <>
+            <div className="vm-query-analyzer-info-header__period">
+              <TimelineIcon/> step: {period.step}
+            </div>
+            <div className="vm-query-analyzer-info-header__period">
+              <ClockIcon/> {timeRange}
+            </div>
+          </>
+        )}
+      </div>
+
+      {openModal && (
+        <Modal
+          title="Report info"
+          onClose={handleCloseModal}
+        >
+          <div className="vm-query-analyzer-info">
+            {comment && (
+              <div className="vm-query-analyzer-info-item vm-query-analyzer-info-item_comment">
+                <div className="vm-query-analyzer-info-item__title">Comment:</div>
+                <div className="vm-query-analyzer-info-item__text">{comment}</div>
+              </div>
+            )}
+            {dataWithStats.map((d, i) => (
+              <div
+                className="vm-query-analyzer-info-item"
+                key={i}
+              >
+                <div className="vm-query-analyzer-info-item__title">
+                  {dataWithStats.length > 1 ? `Query ${i + 1}:` : "Stats:"}
+                </div>
+                <div className="vm-query-analyzer-info-item__text">
+                  {Object.entries(d.stats || {}).map(([key, value]) => (
+                    <div key={key}>
+                      {key}: {value ?? "-"}
+                    </div>
+                  ))}
+                  isPartial: {String(d.isPartial ?? "-")}
+                </div>
+              </div>
+            ))}
+            <div className="vm-query-analyzer-info-type">
+              {dataWithStats[0]?.vmui?.params ? "The report was created using vmui" : "The report was created manually"}
+            </div>
+          </div>
+        </Modal>
+      )}
+    </>
+  );
+};
+
+export default QueryAnalyzerInfo;

app/vmui/packages/vmui/src/pages/QueryAnalyzer/QueryAnalyzerInfo/style.scss

@@ -0,0 +1,47 @@
+@use "src/styles/variables" as *;
+
+.vm-query-analyzer-info-header {
+  display: flex;
+  gap: $padding-global;
+
+  &__period {
+    display: flex;
+    align-items: center;
+    gap: $padding-small;
+    border: $border-divider;
+    border-radius: $border-radius-small;
+    padding: 6px $padding-global;
+
+    svg {
+      width: calc($font-size-small + 1px);
+      color: $color-primary;
+    }
+  }
+}
+
+.vm-query-analyzer-info {
+  display: grid;
+  gap: $padding-large;
+  min-width: 300px;
+
+  &-type {
+    text-align: center;
+    font-style: italic;
+    color: $color-text-secondary;
+  }
+
+  &-item {
+    display: grid;
+    padding-bottom: $padding-large;
+    border-bottom: $border-divider;
+    line-height: 130%;
+
+    &__title {
+      font-weight: bold;
+    }
+
+    &__text {
+      white-space: pre-wrap;
+    }
+  }
+}

app/vmui/packages/vmui/src/pages/QueryAnalyzer/QueryAnalyzerView/QueryAnalyzerView.tsx

@@ -0,0 +1,180 @@
+import React, { FC, useMemo, useState, useEffect } from "preact/compat";
+import Trace from "../../../components/TraceQuery/Trace";
+import { DataAnalyzerType } from "../index";
+import classNames from "classnames";
+import { displayTypeTabs } from "../../CustomPanel/DisplayTypeSwitch";
+import GraphTips from "../../../components/Chart/GraphTips/GraphTips";
+import GraphSettings from "../../../components/Configurators/GraphSettings/GraphSettings";
+import useDeviceDetect from "../../../hooks/useDeviceDetect";
+import { AxisRange } from "../../../state/graph/reducer";
+import { useGraphDispatch, useGraphState } from "../../../state/graph/GraphStateContext";
+import Tabs from "../../../components/Main/Tabs/Tabs";
+import TracingsView from "../../../components/TraceQuery/TracingsView";
+import "./style.scss";
+import GraphView from "../../../components/Views/GraphView/GraphView";
+import JsonView from "../../../components/Views/JsonView/JsonView";
+import { InstantMetricResult, MetricResult } from "../../../api/types";
+import { isHistogramData } from "../../../utils/metric";
+import { DisplayType, TimeParams } from "../../../types";
+import TableSettings from "../../../components/Table/TableSettings/TableSettings";
+import { getColumns } from "../../../hooks/useSortedCategories";
+import { useCustomPanelDispatch, useCustomPanelState } from "../../../state/customPanel/CustomPanelStateContext";
+import TableView from "../../../components/Views/TableView/TableView";
+
+type Props = {
+  data: DataAnalyzerType[];
+  period?: TimeParams;
+}
+
+const QueryAnalyzerView: FC<Props> = ({ data, period }) => {
+  const { isMobile } = useDeviceDetect();
+  const { tableCompact } = useCustomPanelState();
+  const customPanelDispatch = useCustomPanelDispatch();
+
+  const [traces, setTraces] = useState<Trace[]>([]);
+  const [graphData, setGraphData] = useState<MetricResult[]>();
+  const [liveData, setLiveData] = useState<InstantMetricResult[]>();
+  const [isHistogram, setIsHistogram] = useState(false);
+  const [queries, setQueries] = useState<string[]>([]);
+  const [displayColumns, setDisplayColumns] = useState<string[]>();
+
+  const columns = useMemo(() => getColumns(liveData || []).map(c => c.key), [liveData]);
+
+  const tabs = useMemo(() => {
+    const hasQueryRange = data.some(d => d.data.resultType === "matrix");
+    const hasInstantQuery = data.some(d => d.data.resultType === "vector");
+    if (hasInstantQuery && hasQueryRange) return displayTypeTabs;
+    if (!hasQueryRange) return displayTypeTabs.filter(t => t.value !== "chart");
+    return displayTypeTabs.filter(t => t.value === "chart");
+  }, [data]);
+  const [displayType, setDisplayType] = useState(tabs[0].value);
+
+  const { yaxis } = useGraphState();
+  const graphDispatch = useGraphDispatch();
+
+  const setYaxisLimits = (limits: AxisRange) => {
+    graphDispatch({ type: "SET_YAXIS_LIMITS", payload: limits });
+  };
+
+  const toggleEnableLimits = () => {
+    graphDispatch({ type: "TOGGLE_ENABLE_YAXIS_LIMITS" });
+  };
+
+  const handleChangeDisplayType = (newValue: string) => {
+    setDisplayType(newValue as DisplayType);
+  };
+
+  const handleTraceDelete = (trace: Trace) => {
+    setTraces(prev => prev.filter((data) => data.idValue !== trace.idValue));
+  };
+
+  const toggleTableCompact = () => {
+    customPanelDispatch({ type: "TOGGLE_TABLE_COMPACT" });
+  };
+
+  useEffect(() => {
+    const resultType = displayType === "chart" ? "matrix" : "vector";
+    const traces = data.filter(d => d.data.resultType === resultType && d.trace)
+      .map(d => d.trace ? new Trace(d.trace, d?.vmui?.params?.query || "Query") : null);
+    setTraces(traces.filter(Boolean) as Trace[]);
+  }, [data, displayType]);
+
+  useEffect(() => {
+    const tempQueries: string[] = [];
+    const tempGraphData: MetricResult[] = [];
+    const tempLiveData: InstantMetricResult[] = [];
+
+    data.forEach((d, i) => {
+      const result = d.data.result.map((r) => ({ ...r, group: Number(d.vmui?.params?.id ?? i) + 1 }));
+      if (d.data.resultType === "matrix") {
+        tempGraphData.push(...result as MetricResult[]);
+        tempQueries.push(d.vmui?.params?.query || "Query");
+      } else {
+        tempLiveData.push(...result as InstantMetricResult[]);
+      }
+    });
+
+    setQueries(tempQueries);
+    setGraphData(tempGraphData);
+    setLiveData(tempLiveData);
+  }, [data]);
+
+  useEffect(() => {
+    setIsHistogram(!!graphData && isHistogramData(graphData));
+  }, [graphData]);
+
+  return (
+    <div
+      className={classNames({
+        "vm-query-analyzer-view": true,
+        "vm-query-analyzer-view_mobile": isMobile,
+      })}
+    >
+      {!!traces.length && (
+        <TracingsView
+          traces={traces}
+          onDeleteClick={handleTraceDelete}
+        />
+      )}
+      <div
+        className={classNames({
+          "vm-block": true,
+          "vm-block_mobile": isMobile,
+        })}
+      >
+        <div className="vm-custom-panel-body-header">
+          <div className="vm-custom-panel-body-header__tabs">
+            <Tabs
+              activeItem={displayType}
+              items={tabs}
+              onChange={handleChangeDisplayType}
+            />
+          </div>
+          <div className="vm-custom-panel-body-header__graph-controls">
+            {displayType === "chart" && <GraphTips/>}
+            {displayType === "chart" && (
+              <GraphSettings
+                yaxis={yaxis}
+                setYaxisLimits={setYaxisLimits}
+                toggleEnableLimits={toggleEnableLimits}
+              />
+            )}
+            {displayType === "table" && (
+              <TableSettings
+                columns={columns}
+                defaultColumns={displayColumns}
+                onChangeColumns={setDisplayColumns}
+                tableCompact={tableCompact}
+                toggleTableCompact={toggleTableCompact}
+              />
+            )}
+          </div>
+        </div>
+        {graphData && period && (displayType === "chart") && (
+          <GraphView
+            data={graphData}
+            period={period}
+            customStep={period.step || "1s"}
+            query={queries}
+            yaxis={yaxis}
+            setYaxisLimits={setYaxisLimits}
+            setPeriod={() => null}
+            height={isMobile ? window.innerHeight * 0.5 : 500}
+            isHistogram={isHistogram}
+          />
+        )}
+        {liveData && (displayType === "code") && (
+          <JsonView data={liveData}/>
+        )}
+        {liveData && (displayType === "table") && (
+          <TableView
+            data={liveData}
+            displayColumns={displayColumns}
+          />
+        )}
+      </div>
+    </div>
+  );
+};
+
+export default QueryAnalyzerView;

app/vmui/packages/vmui/src/pages/QueryAnalyzer/QueryAnalyzerView/style.scss

@@ -0,0 +1,30 @@
+@use "src/styles/variables" as *;
+
+.vm-query-analyzer-view {
+  display: grid;
+  gap: $padding-global;
+  position: relative;
+
+  &-header {
+    position: relative;
+    display: flex;
+    align-items: center;
+    justify-content: space-between;
+    font-size: $font-size-small;
+    margin: -$padding-medium 0-$padding-medium $padding-medium;
+    padding: 0 $padding-medium;
+    border-bottom: $border-divider;
+    z-index: 1;
+
+    &__left {
+      display: flex;
+      align-items: center;
+      gap: $padding-small;
+    }
+  }
+
+  &_mobile &-header {
+    margin: -$padding-global 0-$padding-global $padding-global;
+    padding: 0 $padding-global;
+  }
+}

app/vmui/packages/vmui/src/pages/QueryAnalyzer/QueryAnalyzerView/utils.ts

@@ -0,0 +1,20 @@
+export const findMostCommonStep = (numbers: number[]) => {
+  const differences: number[] = numbers.slice(1).map((num, i) => num - numbers[i]);
+
+  const counts: { [key: string]: number } = {};
+  differences.forEach(diff => {
+    const key = diff.toString();
+    counts[key] = (counts[key] || 0) + 1;
+  });
+
+  let mostCommonStep = 0;
+  let maxCount = 0;
+  for (const diff in counts) {
+    if (counts[diff] > maxCount) {
+      maxCount = counts[diff];
+      mostCommonStep = Number(diff);
+    }
+  }
+
+  return mostCommonStep;
+};

app/vmui/packages/vmui/src/pages/QueryAnalyzer/index.tsx

@@ -0,0 +1,205 @@
+import React, { FC, useEffect, useMemo, useState } from "preact/compat";
+import { ChangeEvent } from "react";
+import Button from "../../components/Main/Button/Button";
+import Alert from "../../components/Main/Alert/Alert";
+import { CloseIcon } from "../../components/Main/Icons";
+import Modal from "../../components/Main/Modal/Modal";
+import useDropzone from "../../hooks/useDropzone";
+import useBoolean from "../../hooks/useBoolean";
+import UploadJsonButtons from "../../components/UploadJsonButtons/UploadJsonButtons";
+import JsonForm from "./JsonForm/JsonForm";
+import "../TracePage/style.scss";
+import QueryAnalyzerView from "./QueryAnalyzerView/QueryAnalyzerView";
+import { InstantMetricResult, MetricResult, TracingData } from "../../api/types";
+import QueryAnalyzerInfo from "./QueryAnalyzerInfo/QueryAnalyzerInfo";
+import { TimeParams } from "../../types";
+import { dateFromSeconds, formatDateToUTC, humanizeSeconds } from "../../utils/time";
+import { findMostCommonStep } from "./QueryAnalyzerView/utils";
+
+export type DataAnalyzerType = {
+  data: {
+    resultType: "vector" | "matrix";
+    result: MetricResult[] | InstantMetricResult[]
+  };
+  stats?: {
+    seriesFetched?: string;
+    executionTimeMsec?: number
+  };
+  vmui?: {
+    id: number;
+    comment: string;
+    params: Record<string, string>;
+  };
+  status: string;
+  trace?: TracingData;
+  isPartial?: boolean;
+}
+
+const QueryAnalyzer: FC = () => {
+  const [data, setData] = useState<DataAnalyzerType[]>([]);
+  const [error, setError] = useState("");
+  const hasData = useMemo(() => !!data.length, [data]);
+
+  const {
+    value: openModal,
+    setTrue: handleOpenModal,
+    setFalse: handleCloseModal,
+  } = useBoolean(false);
+
+  const period: TimeParams | undefined = useMemo(() => {
+    if (!data) return;
+    const params = data[0]?.vmui?.params;
+
+    const result = {
+      start: +(params?.start || 0),
+      end: +(params?.end || 0),
+      step: params?.step,
+      date: ""
+    };
+
+    if (!params) {
+      const dataResult = data.filter(d => d.data.resultType === "matrix").map(d => d.data.result).flat();
+      const times = dataResult.map(r => r.values ? r.values?.map(v => v[0]) : [0]).flat();
+      const uniqTimes = Array.from(new Set(times.filter(Boolean))).sort((a, b) => a - b);
+      result.start = uniqTimes[0];
+      result.end = uniqTimes[uniqTimes.length - 1];
+      result.step = humanizeSeconds(findMostCommonStep(uniqTimes));
+    }
+
+    result.date = formatDateToUTC(dateFromSeconds(result.end));
+    return result;
+  }, [data]);
+
+  const isValidResponse = (response: unknown[]): boolean => {
+    return response.every(element => {
+      if (typeof element === "object" && element !== null) {
+        const data = (element as { data?: unknown }).data;
+        if (typeof data === "object" && data !== null) {
+          const result = (data as { result?: unknown }).result;
+          const resultType = (data as { resultType?: unknown }).resultType;
+          return Array.isArray(result) && typeof resultType === "string";
+        }
+      }
+      return false;
+    });
+  };
+
+  const handleOnload = (result: string) => {
+    try {
+      const obj = JSON.parse(result);
+      const response = Array.isArray(obj) ? obj : [obj];
+      if (isValidResponse(response)) {
+        setData(response);
+      } else {
+        setError("Invalid structure - JSON does not match the expected format");
+      }
+    } catch (e) {
+      if (e instanceof Error) {
+        setError(`${e.name}: ${e.message}`);
+      }
+    }
+  };
+
+  const handleReadFiles = (files: File[]) => {
+    files.map(f => {
+      const reader = new FileReader();
+      reader.onload = (e) => {
+        const result = String(e.target?.result);
+        handleOnload(result);
+      };
+      reader.readAsText(f);
+    });
+  };
+
+  const handleChange = (e: ChangeEvent<HTMLInputElement>) => {
+    setError("");
+    const files = Array.from(e.target.files || []);
+    handleReadFiles(files);
+    e.target.value = "";
+  };
+
+  const handleCloseError = () => {
+    setError("");
+  };
+
+  const { files, dragging } = useDropzone();
+
+  useEffect(() => {
+    handleReadFiles(files);
+  }, [files]);
+
+  return (
+    <div className="vm-trace-page">
+      {hasData && (
+        <div className="vm-trace-page-header">
+          <div className="vm-trace-page-header-errors">
+            <QueryAnalyzerInfo
+              data={data}
+              period={period}
+            />
+          </div>
+          <div>
+            <UploadJsonButtons
+              onOpenModal={handleOpenModal}
+              onChange={handleChange}
+            />
+          </div>
+        </div>
+      )}
+
+      {error && (
+        <div className="vm-trace-page-header-errors-item vm-trace-page-header-errors-item_margin-bottom">
+          <Alert variant="error">{error}</Alert>
+          <Button
+            className="vm-trace-page-header-errors-item__close"
+            startIcon={<CloseIcon/>}
+            variant="text"
+            color="error"
+            onClick={handleCloseError}
+          />
+        </div>
+      )}
+
+      {hasData && (
+        <QueryAnalyzerView
+          data={data}
+          period={period}
+        />
+      )}
+
+      {!hasData && (
+        <div className="vm-trace-page-preview">
+          <p className="vm-trace-page-preview__text">
+            Please, upload file with JSON response content.
+            {"\n"}
+            The file must contain query information in JSON format.
+            {"\n"}
+            Graph will be displayed after file upload.
+            {"\n"}
+            Attach files by dragging & dropping, selecting or pasting them.
+          </p>
+          <UploadJsonButtons
+            onOpenModal={handleOpenModal}
+            onChange={handleChange}
+          />
+        </div>
+      )}
+
+      {openModal && (
+        <Modal
+          title="Paste JSON"
+          onClose={handleCloseModal}
+        >
+          <JsonForm
+            onClose={handleCloseModal}
+            onUpload={handleOnload}
+          />
+        </Modal>
+      )}
+
+      {dragging && <div className="vm-trace-page__dropzone"/>}
+    </div>
+  );
+};
+
+export default QueryAnalyzer;

app/vmui/packages/vmui/src/pages/TracePage/TraceUploadButtons/TraceUploadButtons.tsx

@@ -1,35 +0,0 @@
-import React, { FC } from "preact/compat";
-import Button from "../../../components/Main/Button/Button";
-import Tooltip from "../../../components/Main/Tooltip/Tooltip";
-import { ChangeEvent } from "react";
-
-interface TraceUploadButtonsProps {
-  onOpenModal: () => void;
-  onChange: (e: ChangeEvent<HTMLInputElement>) => void;
-}
-
-const TraceUploadButtons: FC<TraceUploadButtonsProps> = ({ onOpenModal, onChange }) => (
-  <div className="vm-trace-page-controls">
-    <Button
-      variant="outlined"
-      onClick={onOpenModal}
-    >
-      Paste JSON
-    </Button>
-    <Tooltip title="The file must contain tracing information in JSON format">
-      <Button>
-        Upload Files
-        <input
-          id="json"
-          type="file"
-          accept="application/json"
-          multiple
-          title=" "
-          onChange={onChange}
-        />
-      </Button>
-    </Tooltip>
-  </div>
-);
-
-export default TraceUploadButtons;

app/vmui/packages/vmui/src/pages/TracePage/index.tsx

@@ -10,7 +10,7 @@ import Modal from "../../components/Main/Modal/Modal";
 import JsonForm from "./JsonForm/JsonForm";
 import { ErrorTypes } from "../../types";
 import useDropzone from "../../hooks/useDropzone";
-import TraceUploadButtons from "./TraceUploadButtons/TraceUploadButtons";
+import UploadJsonButtons from "../../components/UploadJsonButtons/UploadJsonButtons";
 import useBoolean from "../../hooks/useBoolean";
 
 const TracePage: FC = () => {
@@ -106,7 +106,7 @@ const TracePage: FC = () => {
         </div>
         <div>
           {hasTraces && (
-            <TraceUploadButtons
+            <UploadJsonButtons
               onOpenModal={handleOpenModal}
               onChange={handleChange}
             />
@@ -145,7 +145,7 @@ const TracePage: FC = () => {
             {"\n"}
             Attach files by dragging & dropping, selecting or pasting them.
           </p>
-          <TraceUploadButtons
+          <UploadJsonButtons
             onOpenModal={handleOpenModal}
             onChange={handleChange}
           />

app/vmui/packages/vmui/src/pages/TracePage/style.scss

@@ -9,14 +9,6 @@
     padding: $padding-medium 0;
   }
 
-  &-controls {
-    display: grid;
-    grid-template-columns: 1fr 1fr;
-    gap: $padding-global;
-    align-items: center;
-    justify-content: center;
-  }
-
   &-header {
     display: grid;
     grid-template-columns: 1fr auto;
@@ -46,6 +38,10 @@
         align-items: center;
         justify-content: stretch;
 
+        &_margin-bottom {
+          margin-bottom: $padding-global;
+        }
+
         &__filename {
           min-height: 20px;
         }

app/vmui/packages/vmui/src/router/index.ts

@@ -11,6 +11,7 @@ const router = {
   relabel: "/relabeling",
   logs: "/logs",
   activeQueries: "/active-queries",
+  queryAnalyzer: "/query-analyzer",
   icons: "/icons",
   anomaly: "/anomaly",
   query: "/query",
@@ -72,6 +73,10 @@ export const routerOptions: {[key: string]: RouterOptions} = {
     title: "Trace analyzer",
     header: {}
   },
+  [router.queryAnalyzer]: {
+    title: "Query analyzer",
+    header: {}
+  },
   [router.dashboards]: {
     title: "Dashboards",
     ...routerOptionsDefault,

app/vmui/packages/vmui/src/state/time/reducer.ts

@@ -17,6 +17,7 @@ export interface TimeState {
   period: TimeParams;
   relativeTime?: string;
   timezone: string;
+  defaultTimezone?: string;
 }
 
 export type TimeAction =
@@ -26,6 +27,7 @@ export type TimeAction =
   | { type: "RUN_QUERY"}
   | { type: "RUN_QUERY_TO_NOW"}
   | { type: "SET_TIMEZONE", payload: string }
+  | { type: "SET_DEFAULT_TIMEZONE", payload: string }
 
 const timezone = getFromStorage("TIMEZONE") as string || dayjs.tz.guess();
 setTimezone(timezone);
@@ -90,10 +92,16 @@ export function reducer(state: TimeState, action: TimeAction): TimeState {
     case "SET_TIMEZONE":
       setTimezone(action.payload);
       saveToStorage("TIMEZONE", action.payload);
+      if (state.defaultTimezone) saveToStorage("DISABLED_DEFAULT_TIMEZONE", action.payload !== state.defaultTimezone);
       return {
         ...state,
         timezone: action.payload
       };
+    case "SET_DEFAULT_TIMEZONE":
+      return {
+        ...state,
+        defaultTimezone: action.payload
+      };
     default:
       throw new Error();
   }

app/vmui/packages/vmui/src/utils/storage.ts

@@ -4,6 +4,7 @@ export type StorageKeys = "AUTOCOMPLETE"
     | "SERIES_LIMITS"
     | "TABLE_COMPACT"
     | "TIMEZONE"
+    | "DISABLED_DEFAULT_TIMEZONE"
     | "THEME"
     | "LOGS_LIMIT"
     | "EXPLORE_METRICS_TIPS"

dashboards/Makefile

@@ -10,7 +10,7 @@ dashboard-copy:
 # Copies listed dashboards to vm/* but changes the datasource type from Prometheus to VictoriaMetrics.
 # The command should be called before commiting changes to dashboards/* files.
 dashboards-sync:
-	SRC=victoriametrics.json D_UID=wNf0q_kZk TITLE="VictoriaMetrics" $(MAKE) dashboard-copy
+	SRC=victoriametrics.json D_UID=wNf0q_kZk TITLE="VictoriaMetrics - single-node" $(MAKE) dashboard-copy
 	SRC=victoriametrics-cluster.json D_UID=oS7Bi_0Wz TITLE="VictoriaMetrics - cluster" $(MAKE) dashboard-copy
 	SRC=vmagent.json D_UID=G7Z9GzMGz TITLE="VictoriaMetrics - vmagent" $(MAKE) dashboard-copy
 	SRC=vmalert.json D_UID=LzldHAVnz TITLE="VictoriaMetrics - vmalert" $(MAKE) dashboard-copy

dashboards/victoriametrics-cluster.json

@@ -3313,7 +3313,7 @@
             "type": "prometheus",
             "uid": "$ds"
           },
-          "description": "VictoriaMetrics limits the number of labels per each metric with `-maxLabelsPerTimeseries` command-line flag.\n\nThis prevents from ingesting metrics with too many labels. The value of `maxLabelsPerTimeseries` must be adjusted for your workload.\n\nWhen limit is exceeded (graph is > 0) - extra labels are dropped, which could result in unexpected identical time series.",
+          "description": "VictoriaMetrics limits the number of labels per each metric with `-maxLabelsPerTimeseries` command-line flag.\n\nThis prevents from ingesting metrics with too many labels. The value of `maxLabelsPerTimeseries` must be adjusted for your workload.\n\nWhen limit is exceeded (graph is > 0) - extra labels are dropped, which could result in unexpected identical time series. See more details about dropped labels in vminsert logs.",
           "fieldConfig": {
             "defaults": {
               "color": {

dashboards/victoriametrics.json

@@ -3700,7 +3700,7 @@
             "type": "prometheus",
             "uid": "$ds"
           },
-          "description": "VictoriaMetrics limits the number of labels per each metric with `-maxLabelsPerTimeseries` command-line flag.\n\nThis prevents from ingesting metrics with too many labels. The value of `maxLabelsPerTimeseries` must be adjusted for your workload.\n\nWhen limit is exceeded (graph is > 0) - extra labels are dropped, which could result in unexpected identical time series.",
+          "description": "VictoriaMetrics limits the number of labels per each metric with `-maxLabelsPerTimeseries` command-line flag.\n\nThis prevents from ingesting metrics with too many labels. The value of `maxLabelsPerTimeseries` must be adjusted for your workload.\n\nWhen limit is exceeded (graph is > 0) - extra labels are dropped, which could result in unexpected identical time series. See more details about dropped labels in logs.",
           "fieldConfig": {
             "defaults": {
               "color": {

dashboards/vm/victoriametrics-cluster.json

@@ -3314,7 +3314,7 @@
             "type": "victoriametrics-datasource",
             "uid": "$ds"
           },
-          "description": "VictoriaMetrics limits the number of labels per each metric with `-maxLabelsPerTimeseries` command-line flag.\n\nThis prevents from ingesting metrics with too many labels. The value of `maxLabelsPerTimeseries` must be adjusted for your workload.\n\nWhen limit is exceeded (graph is > 0) - extra labels are dropped, which could result in unexpected identical time series.",
+          "description": "VictoriaMetrics limits the number of labels per each metric with `-maxLabelsPerTimeseries` command-line flag.\n\nThis prevents from ingesting metrics with too many labels. The value of `maxLabelsPerTimeseries` must be adjusted for your workload.\n\nWhen limit is exceeded (graph is > 0) - extra labels are dropped, which could result in unexpected identical time series. See more details about dropped labels in vminsert logs.",
           "fieldConfig": {
             "defaults": {
               "color": {

dashboards/vm/victoriametrics.json

@@ -3701,7 +3701,7 @@
             "type": "victoriametrics-datasource",
             "uid": "$ds"
           },
-          "description": "VictoriaMetrics limits the number of labels per each metric with `-maxLabelsPerTimeseries` command-line flag.\n\nThis prevents from ingesting metrics with too many labels. The value of `maxLabelsPerTimeseries` must be adjusted for your workload.\n\nWhen limit is exceeded (graph is > 0) - extra labels are dropped, which could result in unexpected identical time series.",
+          "description": "VictoriaMetrics limits the number of labels per each metric with `-maxLabelsPerTimeseries` command-line flag.\n\nThis prevents from ingesting metrics with too many labels. The value of `maxLabelsPerTimeseries` must be adjusted for your workload.\n\nWhen limit is exceeded (graph is > 0) - extra labels are dropped, which could result in unexpected identical time series. See more details about dropped labels in logs.",
           "fieldConfig": {
             "defaults": {
               "color": {
@@ -5581,7 +5581,7 @@
     ]
   },
   "timezone": "",
-  "title": "VictoriaMetrics - single-node",
+  "title": "VictoriaMetrics - single-node (VM)",
   "uid": "wNf0q_kZk_vm",
   "version": 1,
   "weekStart": ""

deployment/docker/Makefile

@@ -2,8 +2,8 @@
 
 DOCKER_NAMESPACE ?= victoriametrics
 
-ROOT_IMAGE ?= alpine:3.19.0
-CERTS_IMAGE := alpine:3.19.0
+ROOT_IMAGE ?= alpine:3.19.1
+CERTS_IMAGE := alpine:3.19.1
 
 GO_BUILDER_IMAGE := golang:1.21.6-alpine
 BUILDER_IMAGE := local/builder:2.0.0-$(shell echo $(GO_BUILDER_IMAGE) | tr :/ __)-1
@@ -187,8 +187,20 @@ docker-single-up:
 docker-single-down:
 	$(DOCKER_COMPOSE) -f deployment/docker/docker-compose.yml down -v
 
+docker-single-vm-datasource-up:
+	$(DOCKER_COMPOSE) -f deployment/docker/docker-compose.yml -f deployment/docker/vm-datasource/docker-compose.yml up -d
+
+docker-single-vm-datasource-down:
+	$(DOCKER_COMPOSE) -f deployment/docker/docker-compose.yml -f deployment/docker/vm-datasource/docker-compose.yml down -v
+
 docker-cluster-up:
 	$(DOCKER_COMPOSE) -f deployment/docker/docker-compose-cluster.yml up -d
 
 docker-cluster-down:
 	$(DOCKER_COMPOSE) -f deployment/docker/docker-compose-cluster.yml down -v
+
+docker-cluster-vm-datasource-up:
+	$(DOCKER_COMPOSE) -f deployment/docker/docker-compose-cluster.yml -f deployment/docker/vm-datasource/docker-compose-cluster.yml up -d
+
+docker-cluster-vm-datasource-down:
+	$(DOCKER_COMPOSE) -f deployment/docker/docker-compose-cluster.yml -f deployment/docker/vm-datasource/docker-compose-cluster.yml down -v

deployment/docker/README.md

@@ -7,24 +7,28 @@ and [Grafana](https://grafana.com/).
 For starting the docker-compose environment ensure you have docker installed and running and access to the Internet.
 **All commands should be executed from the root directory of [the repo](https://github.com/VictoriaMetrics/VictoriaMetrics).**
 
-To spin-up environment for single server VictoriaMetrics run the following command:
+To spin-up environment with VictoriaMetrics components run one of the following commands:
 ```
-make docker-single-up
+make docker-single-up  # start single server VictoriaMetrics 
+ or
+make docker-cluster-up # start cluster VictoriaMetrics 
 ```
 
-To shut down the docker-compose environment for single server run the following command:
+To shut down the docker-compose environment run one the following commands:
 ```
-make docker-single-down
+make docker-single-down # shutdown single server VictoriaMetrics 
+ or
+make docker-cluster-down # shutdown cluster VictoriaMetrics 
 ```
 
-For cluster version the command will be the following:
-```
-make docker-cluster-up
+Optionally, environment with [VictoriaMetrics Grafana datasource](https://github.com/VictoriaMetrics/grafana-datasource)
+can be started with the following commands:
 ```
+make docker-single-vm-datasource-up    # start single server
+make docker-single-vm-datasource-down  # shut down single server
 
-To shut down the docker compose environment for cluster version run the following command:
-```
-make docker-cluster-down
+make docker-cluster-vm-datasource-up   # start cluster
+make docker-cluster-vm-datasource-down # shutdown cluster
 ```
 
 ## VictoriaMetrics single server
@@ -123,6 +127,10 @@ Grafana is provisioned by default with following entities:
 
 Remember to pick `VictoriaMetrics - cluster` datasource when viewing `VictoriaMetrics - cluster` dashboard.
 
+If environment was started via `docker-single-vm-datasource-up` or `docker-cluster-vm-datasource-up`, then
+Grafana will have [VictoriaMetrics Grafana datasource](https://github.com/VictoriaMetrics/grafana-datasource)
+installed by default.
+
 ## Alerts
 
 See below a list of recommended alerting rules for various VictoriaMetrics components for running in production.