app/vlselect/logsql: skip rows without _stream reference

_stream field can be empty for the recently ingested rows because respective entry in indexdb is not yet searchable as it haven't been flushed to storage yet.
This change just skips such items in the output response to make it more consistent.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6042
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

.github/PULL_REQUEST_TEMPLATE/pull_request_template.md

@@ -0,0 +1,35 @@
+### Describe Your Changes
+
+Please provide a brief description of the changes you made. Be as specific as possible to help others understand the purpose and impact of your modifications.
+
+### Checklist
+
+The following checks are mandatory:
+
+- [ ] I have read the [Contributing Guidelines](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/CONTRIBUTING.md)
+- [ ] All commits are signed and include `Signed-off-by` line. Use `git commit -s` to include `Signed-off-by` your commits. See this [doc](https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work) about how to sign your commits.
+- [ ] Tests are passing locally. Use `make test` to run all tests locally.
+- [ ] Linting is passing locally. Use `make check-all` to run all linters locally.
+
+Further checks are optional for External Contributions:
+
+- [ ] Include a link to the GitHub issue in the commit message, if issue exists.
+- [ ] Mention the change in the [Changelog](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/CHANGELOG.md). Explain what has changed and why. If there is a related issue or documentation change - link them as well.
+
+  Tips for writing a good changelog message::
+
+    * Write a human-readable changelog message that describes the problem and solution.
+    * Include a link to the issue or pull request in your changelog message.
+    * Use specific language identifying the fix, such as an error message, metric name, or flag name.
+    * Provide a link to the relevant documentation for any new features you add or modify.
+
+- [ ] After your pull request is merged, please add a message to the issue with instructions for how to test the fix or try the feature you added. Here is an [example](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4048#issuecomment-1546453726)
+- [ ] Do not close the original issue before the change is released. Please note, in some cases Github can automatically close the issue once PR is merged. Re-open the issue in such case.
+- [ ] If the change somehow affects public interfaces (a new flag was added or updated, or some behavior has changed) - add the corresponding change to documentation.
+
+
+Examples of good changelog messages:
+
+1. FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add support for [VictoriaMetrics remote write protocol](https://docs.victoriametrics.com/vmagent.html#victoriametrics-remote-write-protocol) when [sending / receiving data to / from Kafka](https://docs.victoriametrics.com/vmagent.html#kafka-integration). This protocol allows saving egress network bandwidth costs when sending data from `vmagent` to `Kafka` located in another datacenter or availability zone. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225).
+
+2. BUGFIX: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation.html): suppress `series after dedup` error message in logs when `-remoteWrite.streamAggr.dedupInterval` command-line flag is set at [vmagent](https://docs.victoriametrics.com/vmgent.html) or when `-streamAggr.dedupInterval` command-line flag is set at [single-node VictoriaMetrics](https://docs.victoriametrics.com/).

.github/workflows/sync-docs.yml

@@ -6,9 +6,6 @@ on:
     paths:
       - 'docs/**'
   workflow_dispatch: {}
-env:
-  PAGEFIND_VERSION: "1.0.4"
-  HUGO_VERSION: "latest"
 permissions:
   contents: read  # This is required for actions/checkout and to commit back image update
   deployments: write
@@ -27,16 +24,6 @@ jobs:
           repository: VictoriaMetrics/vmdocs
           token: ${{ secrets.VM_BOT_GH_TOKEN }}
           path: docs
-      - uses: peaceiris/actions-hugo@v2
-        with:
-          hugo-version: ${{env.HUGO_VERSION}}
-          extended: true
-      - name: Install PageFind #install the static search engine for index build
-        uses: supplypike/setup-bin@v3
-        with:
-          uri: "https://github.com/CloudCannon/pagefind/releases/download/v${{env.PAGEFIND_VERSION}}/pagefind-v${{env.PAGEFIND_VERSION}}-x86_64-unknown-linux-musl.tar.gz"
-          name: "pagefind"
-          version: ${{env.PAGEFIND_VERSION}}
       - name: Import GPG key
         uses: crazy-max/ghaction-import-gpg@v5
         with:
@@ -51,13 +38,11 @@ jobs:
           calculatedSha=$(git rev-parse --short ${{ github.sha }})
           echo "short_sha=$calculatedSha" >> $GITHUB_OUTPUT
         working-directory: main
-
       - name: update code and commit
         run: |
           rm -rf content
           cp -r ../main/docs content
           make clean-after-copy
-          make build-search-index
           git config --global user.name "${{ steps.import-gpg.outputs.email }}"
           git config --global user.email "${{ steps.import-gpg.outputs.email }}"
           git add .

.github/workflows/wiki.yml

@@ -1,33 +0,0 @@
-name: wiki
-on:
-  push:
-    paths:
-      - 'docs/*'
-    branches:
-      - master
-permissions:
-  contents: read
-
-jobs:
-  build:
-    permissions:
-      contents: write  # for Git to git push
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@master
-      - name: publish
-        shell: bash
-        env:
-          TOKEN: ${{secrets.CI_TOKEN}}
-        run: |
-          git clone https://vika:${TOKEN}@github.com/VictoriaMetrics/VictoriaMetrics.wiki.git wiki
-          cp -r docs/* wiki
-          cd wiki
-          git config --local user.email "info@victoriametrics.com"
-          git config --local user.name "Vika"
-          git add .
-          git commit -m "update wiki pages"
-          remote_repo="https://vika:${TOKEN}@github.com/VictoriaMetrics/VictoriaMetrics.wiki.git"
-          git push "${remote_repo}"
-          cd ..
-          rm -rf wiki

CONTRIBUTING.md

@@ -14,3 +14,8 @@ We are open to third-party pull requests provided they follow [KISS design princ
 - Avoid automated decisions, which may hurt cluster availability, consistency or performance.
 
 Adhering `KISS` principle simplifies the resulting code and architecture, so it can be reviewed, understood and verified by many people.
+
+Before sending a pull request please check the following:
+- [ ] All commits are signed and include `Signed-off-by` line. Use `git commit -s` to include `Signed-off-by` your commits. See this [doc](https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work) about how to sign your commits.
+- [ ] Tests are passing locally. Use `make test` to run all tests locally.
+- [ ] Linting is passing locally. Use `make check-all` to run all linters locally.

Makefile

@@ -466,7 +466,7 @@ benchmark-pure:
 vendor-update:
 	go get -u -d ./lib/...
 	go get -u -d ./app/...
-	go mod tidy -compat=1.22
+	go mod tidy -compat=1.21
 	go mod vendor
 
 app-local:
@@ -492,7 +492,7 @@ golangci-lint: install-golangci-lint
 	golangci-lint run
 
 install-golangci-lint:
-	which golangci-lint || curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(shell go env GOPATH)/bin v1.55.1
+	which golangci-lint || curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(shell go env GOPATH)/bin v1.57.1
 
 govulncheck: install-govulncheck
 	govulncheck ./...

README.md

@@ -23,10 +23,11 @@ Documentation for the cluster version of VictoriaMetrics is available [here](htt
 Learn more about [key concepts](https://docs.victoriametrics.com/keyConcepts.html) of VictoriaMetrics and follow the 
 [quick start guide](https://docs.victoriametrics.com/Quick-Start.html) for a better experience.
 
-If you have questions about VictoriaMetrics, then feel free asking them in the [VictoriaMetrics community Slack chat](https://slack.victoriametrics.com/).
+If you have questions about VictoriaMetrics, then feel free asking them in the [VictoriaMetrics community Slack chat](https://victoriametrics.slack.com/),
+you can join it via [Slack Inviter](https://slack.victoriametrics.com/).
 
 [Contact us](mailto:info@victoriametrics.com) if you need enterprise support for VictoriaMetrics.
-See [features available in enterprise package](https://docs.victoriametrics.com/enterprise.html).
+See [features available in enterprise package](https://docs.victoriametrics.com/enterprise/).
 Enterprise binaries can be downloaded and evaluated for free
 from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest).
 You can also [request a free trial license](https://victoriametrics.com/products/enterprise/trial/).
@@ -35,7 +36,7 @@ VictoriaMetrics is developed at a fast pace, so it is recommended to check the [
 and to perform [regular upgrades](#how-to-upgrade-victoriametrics).
 
 [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise/) provides long-term support lines of releases (LTS releases) -
-see [these docs](https://docs.victoriametrics.com/LTS-releases.md).
+see [these docs](https://docs.victoriametrics.com/lts-releases/).
 
 VictoriaMetrics has achieved security certifications for Database Software Development and Software-Based Monitoring Services.
 We apply strict security measures in everything we do. See [Security page](https://victoriametrics.com/security/) for more details.
@@ -98,7 +99,7 @@ VictoriaMetrics has the following prominent features:
 * It can deal with [high cardinality issues](https://docs.victoriametrics.com/FAQ.html#what-is-high-cardinality) and
   [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate) issues via [series limiter](#cardinality-limiter).
 * It ideally works with big amounts of time series data from APM, Kubernetes, IoT sensors, connected cars, industrial telemetry, financial data
-  and various [Enterprise workloads](https://docs.victoriametrics.com/enterprise.html).
+  and various [Enterprise workloads](https://docs.victoriametrics.com/enterprise/).
 * It has an open source [cluster version](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/cluster).
 * It can store data on [NFS-based storages](https://en.wikipedia.org/wiki/Network_File_System) such as [Amazon EFS](https://aws.amazon.com/efs/)
   and [Google Filestore](https://cloud.google.com/filestore).
@@ -115,7 +116,7 @@ VictoriaMetrics ecosystem contains the following components additionally to [sin
 - [vmalert](https://docs.victoriametrics.com/vmalert/) - a service for processing Prometheus-compatible alerting and recording rules.
 - [vmalert-tool](https://docs.victoriametrics.com/vmalert-tool/) -  a tool for validating alerting and recording rules.
 - [vmauth](https://docs.victoriametrics.com/vmauth/) - authorization proxy and load balancer optimized for VictoriaMetrics products.
-- [vmgateway](https://docs.victoriametrics.com/vmgateway/) - auhtorization proxy with per-[tenant](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy) rate limiting cababilities.
+- [vmgateway](https://docs.victoriametrics.com/vmgateway/) - authorization proxy with per-[tenant](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy) rate limiting cababilities.
 - [vmctl](https://docs.victoriametrics.com/vmctl/) - a tool for migrating and copying data between different storage systems for metrics.
 - [vmbackup](https://docs.victoriametrics.com/vmbackup/), [vmrestore](https://docs.victoriametrics.com/vmrestore/) and [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/) -
   tools for creating backups and restoring from backups for VictoriaMetrics data.
@@ -364,7 +365,8 @@ Prometheus doesn't drop data during VictoriaMetrics restart. See [this article](
 
 ## vmui
 
-VictoriaMetrics provides UI for query troubleshooting and exploration. The UI is available at `http://victoriametrics:8428/vmui`.
+VictoriaMetrics provides UI for query troubleshooting and exploration. The UI is available at `http://victoriametrics:8428/vmui` 
+(or at `http://<vmselect>:8481/select/<accountID>/vmui/` in [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/)).
 The UI allows exploring query results via graphs and tables. It also provides the following features:
 
 - Explore:
@@ -374,6 +376,7 @@ The UI allows exploring query results via graphs and tables. It also provides th
   - [Active queries](#active-queries) - shows currently executed queries;
 - Tools:
   - [Trace analyzer](#query-tracing) - playground for loading query traces in JSON format; 
+  - [Query analyzer](#query-tracing) - playground for loading query results and traces in JSON format. See `Export query` button below;  
   - [WITH expressions playground](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/graph/#/expand-with-exprs) - test how WITH expressions work; 
   - [Metric relabel debugger](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/graph/#/relabeling) - playground for [relabeling](#relabeling) configs.
 
@@ -409,6 +412,10 @@ Graphs for a particular query can be temporarily hidden by clicking the `eye` ic
 When the `eye` icon is clicked while holding the `ctrl` key, then query results for the rest of queries become hidden
 except of the current query results.
 
+VMUI allows sharing query and [trace](https://docs.victoriametrics.com/#query-tracing) results by clicking on
+`Export query` button in top right corner of the graph area. The query and trace will be exported as a file that later
+can be loaded in VMUI via `Query Analyzer` tool.
+
 See the [example VMUI at VictoriaMetrics playground](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/graph/?g0.expr=100%20*%20sum(rate(process_cpu_seconds_total))%20by%20(job)&g0.range_input=1d).
 
 ## Top queries
@@ -910,9 +917,9 @@ VictoriaMetrics supports the following handlers from [Prometheus querying API](h
 
 * [/api/v1/query](https://docs.victoriametrics.com/keyConcepts.html#instant-query)
 * [/api/v1/query_range](https://docs.victoriametrics.com/keyConcepts.html#range-query)
-* [/api/v1/series](https://prometheus.io/docs/prometheus/latest/querying/api/#finding-series-by-label-matchers)
-* [/api/v1/labels](https://prometheus.io/docs/prometheus/latest/querying/api/#getting-label-names)
-* [/api/v1/label/.../values](https://prometheus.io/docs/prometheus/latest/querying/api/#querying-label-values)
+* [/api/v1/series](https://docs.victoriametrics.com/url-examples/#apiv1series)
+* [/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels)
+* [/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues)
 * [/api/v1/status/tsdb](https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats). See [these docs](#tsdb-stats) for details.
 * [/api/v1/targets](https://prometheus.io/docs/prometheus/latest/querying/api/#targets) - see [these docs](#how-to-scrape-prometheus-exporters-such-as-node-exporter) for more details.
 * [/federate](https://prometheus.io/docs/prometheus/latest/federation/) - see [these docs](#federation) for more details.
@@ -1128,17 +1135,16 @@ as a service for your OS. A [snap package](https://snapcraft.io/victoriametrics)
 
 ## How to work with snapshots
 
-VictoriaMetrics can create [instant snapshots](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282)
-for all the data stored under `-storageDataPath` directory.
-Navigate to `http://<victoriametrics-addr>:8428/snapshot/create` in order to create an instant snapshot.
-The page will return the following JSON response:
+Send a request to `http://<victoriametrics-addr>:8428/snapshot/create` endpoint in order to create
+an [instant snapshot](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282).
+The page returns the following JSON response on successful creation of snapshot:
 
 ```json
 {"status":"ok","snapshot":"<snapshot-name>"}
 ```
 
 Snapshots are created under `<-storageDataPath>/snapshots` directory, where `<-storageDataPath>`
-is the command-line flag value. Snapshots can be archived to backup storage at any time
+is the corresponding command-line flag value. Snapshots can be archived to backup storage at any time
 with [vmbackup](https://docs.victoriametrics.com/vmbackup.html).
 
 Snapshots consist of a mix of hard-links and soft-links to various files and directories inside `-storageDataPath`.
@@ -1150,20 +1156,32 @@ for more details. This adds some restrictions on what can be done with the conte
 - Do not copy subdirectories inside `<-storageDataPath>/snapshot` with `cp`, `rsync` or similar commands, since there are high chances
   that these commands won't copy some data stored in the snapshot. Prefer using [vmbackup](https://docs.victoriametrics.com/vmbackup.html) for making copies of snapshot data.
 
-The `http://<victoriametrics-addr>:8428/snapshot/list` page contains the list of available snapshots.
+See also [snapshot troubleshooting](#snapshot-troubleshooting).
 
-Navigate to `http://<victoriametrics-addr>:8428/snapshot/delete?snapshot=<snapshot-name>` in order
-to delete `<snapshot-name>` snapshot.
+The `http://<victoriametrics-addr>:8428/snapshot/list` endpoint returns the list of available snapshots.
+
+Send a query to `http://<victoriametrics-addr>:8428/snapshot/delete?snapshot=<snapshot-name>` in order
+to delete the snapshot with `<snapshot-name>` name.
 
 Navigate to `http://<victoriametrics-addr>:8428/snapshot/delete_all` in order to delete all the snapshots.
 
-Steps for restoring from a snapshot:
+### How to restore from a snapshot
 
 1. Stop VictoriaMetrics with `kill -INT`.
 1. Restore snapshot contents from backup with [vmrestore](https://docs.victoriametrics.com/vmrestore.html)
    to the directory pointed by `-storageDataPath`.
 1. Start VictoriaMetrics.
 
+### Snapshot troubleshooting
+
+Snapshot doesn't occupy disk space just after its' creation thanks to the [used approach](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282).
+Old snapshots may start occupying additional disk space if they refer to old parts, which were already deleted during [background merge](#storage).
+That's why it is recommended deleting old snapshots after they are no longer needed in order to free up disk space used by old snapshots.
+This can be done either manually or automatically if the `-snapshotsMaxAge` command-line flag is set. Make sure that the backup process has enough time to complete
+when setting `-snapshotsMaxAge` command-line flag.
+
+VictoriaMetrics exposes the current number of available snapshots via `vm_snapshots` metric at [`/metrics`](#monitoring) page.
+
 ## How to delete time series
 
 Send a request to `http://<victoriametrics-addr>:8428/api/v1/admin/tsdb/delete_series?match[]=<timeseries_selector_for_delete>`,
@@ -1279,7 +1297,7 @@ where:
     * `unix_s` - unix seconds
     * `unix_ms` - unix milliseconds
     * `unix_ns` - unix nanoseconds
-    * `rfc3339` - [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) time
+    * `rfc3339` - [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) time (in the timezone of the server)
     * `custom:<layout>` - custom layout for time that is supported by [time.Format](https://golang.org/pkg/time/#Time.Format) function from Go.
 
 * `<timeseries_selector_for_export>` may contain any [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
@@ -1523,10 +1541,15 @@ VictoriaMetrics also may scrape Prometheus targets - see [these docs](#how-to-sc
 
 ### Sending data via OpenTelemetry
 
-VictoriaMetrics supports data ingestion via [OpenTelemetry protocol for metrics](https://github.com/open-telemetry/opentelemetry-specification/blob/ffddc289462dfe0c2041e3ca42a7b1df805706de/specification/metrics/data-model.md) at `/opentelemetry/api/v1/push` path.
+VictoriaMetrics supports data ingestion via [OpenTelemetry protocol for metrics](https://github.com/open-telemetry/opentelemetry-specification/blob/ffddc289462dfe0c2041e3ca42a7b1df805706de/specification/metrics/data-model.md) at `/opentelemetry/v1/metrics` path.
 
-VictoriaMetrics expects `protobuf`-encoded requests at `/opentelemetry/api/v1/push`.
-Set HTTP request header `Content-Encoding: gzip` when sending gzip-compressed data to `/opentelemetry/api/v1/push`.
+VictoriaMetrics expects `protobuf`-encoded requests at `/opentelemetry/v1/metrics`.
+Set HTTP request header `Content-Encoding: gzip` when sending gzip-compressed data to `/opentelemetry/v1/metrics`.
+
+VictoriaMetrics stores the ingested OpenTelemetry [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) as is without any transformations.
+Pass `-opentelemetry.usePrometheusNaming` command-line flag to VictoriaMetrics for automatic conversion of metric names and labels into Prometheus-compatible format.
+
+See [How to use OpenTelemetry metrics with VictoriaMetrics](https://docs.victoriametrics.com/guides/getting-started-with-opentelemetry/).
 
 ## JSON line format
 
@@ -1651,21 +1674,70 @@ See also [resource usage limits docs](#resource-usage-limits).
 
 By default, VictoriaMetrics is tuned for an optimal resource usage under typical workloads. Some workloads may need fine-grained resource usage limits. In these cases the following command-line flags may be useful:
 
-- `-memory.allowedPercent` and `-memory.allowedBytes` limit the amounts of memory, which may be used for various internal caches at VictoriaMetrics. Note that VictoriaMetrics may use more memory, since these flags don't limit additional memory, which may be needed on a per-query basis.
-- `-search.maxMemoryPerQuery` limits the amounts of memory, which can be used for processing a single query. Queries, which need more memory, are rejected. Heavy queries, which select big number of time series, may exceed the per-query memory limit by a small percent. The total memory limit for concurrently executed queries can be estimated as `-search.maxMemoryPerQuery` multiplied by `-search.maxConcurrentRequests`.
-- `-search.maxUniqueTimeseries` limits the number of unique time series a single query can find and process. VictoriaMetrics keeps in memory some metainformation about the time series located by each query and spends some CPU time for processing the found time series. This means that the maximum memory usage and CPU usage a single query can use is proportional to `-search.maxUniqueTimeseries`.
-- `-search.maxQueryDuration` limits the duration of a single query. If the query takes longer than the given duration, then it is canceled. This allows saving CPU and RAM when executing unexpected heavy queries.
-- `-search.maxConcurrentRequests` limits the number of concurrent requests VictoriaMetrics can process. Bigger number of concurrent requests usually means bigger memory usage. For example, if a single query needs 100 MiB of additional memory during its execution, then 100 concurrent queries may need `100 * 100 MiB = 10 GiB` of additional memory. So it is better to limit the number of concurrent queries, while suspending additional incoming queries if the concurrency limit is reached. VictoriaMetrics provides `-search.maxQueueDuration` command-line flag for limiting the max wait time for suspended queries. See also `-search.maxMemoryPerQuery` command-line flag.
-- `-search.maxSamplesPerSeries` limits the number of raw samples the query can process per each time series. VictoriaMetrics sequentially processes raw samples per each found time series during the query. It unpacks raw samples on the selected time range per each time series into memory and then applies the given [rollup function](https://docs.victoriametrics.com/MetricsQL.html#rollup-functions). The `-search.maxSamplesPerSeries` command-line flag allows limiting memory usage in the case when the query is executed on a time range, which contains hundreds of millions of raw samples per each located time series.
+- `-memory.allowedPercent` and `-memory.allowedBytes` limit the amounts of memory, which may be used for various internal caches at VictoriaMetrics.
+  Note that VictoriaMetrics may use more memory, since these flags don't limit additional memory, which may be needed on a per-query basis.
+- `-search.maxMemoryPerQuery` limits the amounts of memory, which can be used for processing a single query. Queries, which need more memory, are rejected.
+  Heavy queries, which select big number of time series, may exceed the per-query memory limit by a small percent. The total memory limit
+  for concurrently executed queries can be estimated as `-search.maxMemoryPerQuery` multiplied by `-search.maxConcurrentRequests`.
+- `-search.maxUniqueTimeseries` limits the number of unique time series a single query can find and process. VictoriaMetrics keeps in memory
+  some metainformation about the time series located by each query and spends some CPU time for processing the found time series.
+  This means that the maximum memory usage and CPU usage a single query can use is proportional to `-search.maxUniqueTimeseries`.
+- `-search.maxQueryDuration` limits the duration of a single query. If the query takes longer than the given duration, then it is canceled.
+  This allows saving CPU and RAM when executing unexpected heavy queries.
+  The limit can be altered for each query by passing `timeout` GET parameter, but can't exceed the limit specified via `-search.maxQueryDuration` command-line flag.
+- `-search.maxConcurrentRequests` limits the number of concurrent requests VictoriaMetrics can process. Bigger number of concurrent requests usually means
+  bigger memory usage. For example, if a single query needs 100 MiB of additional memory during its execution, then 100 concurrent queries may need `100 * 100 MiB = 10 GiB`
+  of additional memory. So it is better to limit the number of concurrent queries, while pausing additional incoming queries if the concurrency limit is reached.
+  VictoriaMetrics provides `-search.maxQueueDuration` command-line flag for limiting the max wait time for paused queries. See also `-search.maxMemoryPerQuery` command-line flag.
+- `-search.maxQueueDuration` limits the maximum duration queries may wait for execution when `-search.maxConcurrentRequests` concurrent queries are executed.
+- `-search.ignoreExtraFiltersAtLabelsAPI` enables ignoring of `match[]`, [`extra_filters[]` and `extra_label`](https://docs.victoriametrics.com/#prometheus-querying-api-enhancements)
+  query args at [/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels) and
+  [/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues).
+  This may be useful for reducing the load on VictoriaMetrics if the provided extra filters match too many time series.
+  The downside is that the endpoints can return labels and series, which do not match the provided extra filters.
+- `-search.maxSamplesPerSeries` limits the number of raw samples the query can process per each time series. VictoriaMetrics sequentially processes
+  raw samples per each found time series during the query. It unpacks raw samples on the selected time range per each time series into memory
+  and then applies the given [rollup function](https://docs.victoriametrics.com/MetricsQL.html#rollup-functions). The `-search.maxSamplesPerSeries` command-line flag
+  allows limiting memory usage in the case when the query is executed on a time range, which contains hundreds of millions of raw samples per each located time series.
 - `-search.maxSamplesPerQuery` limits the number of raw samples a single query can process. This allows limiting CPU usage for heavy queries.
 - `-search.maxResponseSeries` limits the number of time series a single query can return from [`/api/v1/query`](https://docs.victoriametrics.com/keyConcepts.html#instant-query)
   and [`/api/v1/query_range`](https://docs.victoriametrics.com/keyConcepts.html#range-query).
-- `-search.maxPointsPerTimeseries` limits the number of calculated points, which can be returned per each matching time series from [range query](https://docs.victoriametrics.com/keyConcepts.html#range-query).
-- `-search.maxPointsSubqueryPerTimeseries` limits the number of calculated points, which can be generated per each matching time series during [subquery](https://docs.victoriametrics.com/MetricsQL.html#subqueries) evaluation.
-- `-search.maxSeriesPerAggrFunc` limits the number of time series, which can be generated by [MetricsQL aggregate functions](https://docs.victoriametrics.com/MetricsQL.html#aggregate-functions) in a single query.
-- `-search.maxSeries` limits the number of time series, which may be returned from [/api/v1/series](https://prometheus.io/docs/prometheus/latest/querying/api/#finding-series-by-label-matchers). This endpoint is used mostly by Grafana for auto-completion of metric names, label names and label values. Queries to this endpoint may take big amounts of CPU time and memory when the database contains big number of unique time series because of [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate). In this case it might be useful to set the `-search.maxSeries` to quite low value in order limit CPU and memory usage.
-- `-search.maxTagKeys` limits the number of items, which may be returned from [/api/v1/labels](https://prometheus.io/docs/prometheus/latest/querying/api/#getting-label-names). This endpoint is used mostly by Grafana for auto-completion of label names. Queries to this endpoint may take big amounts of CPU time and memory when the database contains big number of unique time series because of [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate). In this case it might be useful to set the `-search.maxTagKeys` to quite low value in order to limit CPU and memory usage.
-- `-search.maxTagValues` limits the number of items, which may be returned from [/api/v1/label/.../values](https://prometheus.io/docs/prometheus/latest/querying/api/#querying-label-values). This endpoint is used mostly by Grafana for auto-completion of label values. Queries to this endpoint may take big amounts of CPU time and memory when the database contains big number of unique time series because of [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate). In this case it might be useful to set the `-search.maxTagValues` to quite low value in order to limit CPU and memory usage.
+- `-search.maxPointsPerTimeseries` limits the number of calculated points, which can be returned per each matching time series
+  from [range query](https://docs.victoriametrics.com/keyConcepts.html#range-query).
+- `-search.maxPointsSubqueryPerTimeseries` limits the number of calculated points, which can be generated per each matching time series
+  during [subquery](https://docs.victoriametrics.com/MetricsQL.html#subqueries) evaluation.
+- `-search.maxSeriesPerAggrFunc` limits the number of time series, which can be generated by [MetricsQL aggregate functions](https://docs.victoriametrics.com/MetricsQL.html#aggregate-functions)
+  in a single query.
+- `-search.maxSeries` limits the number of time series, which may be returned from [/api/v1/series](https://docs.victoriametrics.com/url-examples/#apiv1series).
+  This endpoint is used mostly by Grafana for auto-completion of metric names, label names and label values. Queries to this endpoint may take big amounts
+  of CPU time and memory when the database contains big number of unique time series because of [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate).
+  In this case it might be useful to set the `-search.maxSeries` to quite low value in order limit CPU and memory usage.
+  See also `-search.maxLabelsAPIDuration` and `-search.maxLabelsAPISeries`.
+- `-search.maxTagKeys` limits the number of items, which may be returned from [/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels).
+  This endpoint is used mostly by Grafana for auto-completion of label names. Queries to this endpoint may take big amounts of CPU time and memory
+  when the database contains big number of unique time series because of [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate).
+  In this case it might be useful to set the `-search.maxTagKeys` to quite low value in order to limit CPU and memory usage.
+  See also `-search.maxLabelsAPIDuration` and `-search.maxLabelsAPISeries`.
+- `-search.maxTagValues` limits the number of items, which may be returned from [/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues).
+  This endpoint is used mostly by Grafana for auto-completion of label values. Queries to this endpoint may take big amounts of CPU time and memory
+  when the database contains big number of unique time series because of [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate).
+  In this case it might be useful to set the `-search.maxTagValues` to quite low value in order to limit CPU and memory usage.
+  See also `-search.maxLabelsAPIDuration` and `-search.maxLabelsAPISeries`.
+- `-search.maxLabelsAPISeries` limits the number of time series, which can be scanned when performing [/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels),
+  [/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues)
+  or [/api/v1/series](https://docs.victoriametrics.com/url-examples/#apiv1series) requests.
+  These endpoints are used mostly by Grafana for auto-completion of label names and label values. Queries to these endpoints may take big amounts of CPU time and memory
+  when the database contains big number of unique time series because of [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate).
+  In this case it might be useful to set the `-search.maxLabelsAPISeries` to quite low value in order to limit CPU and memory usage.
+  See also `-search.maxLabelsAPIDuration` and `-search.ignoreExtraFiltersAtLabelsAPI`.
+- `-search.maxLabelsAPIDuration` limits the duration for requests to [/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels),
+  [/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues)
+  or [/api/v1/series](https://docs.victoriametrics.com/url-examples/#apiv1series).
+  The limit can be altered for each query by passing `timeout` GET parameter, but can't exceed the limit specified via cmd-line flag.
+  These endpoints are used mostly by Grafana for auto-completion of label names and label values. Queries to these endpoints may take big amounts of CPU time and memory
+  when the database contains big number of unique time series because of [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate).
+  In this case it might be useful to set the `-search.maxLabelsAPIDuration` to quite low value in order to limit CPU and memory usage.
+  See also `-search.maxLabelsAPISeries` and `-search.ignoreExtraFiltersAtLabelsAPI`.
 - `-search.maxTagValueSuffixesPerSearch` limits the number of entries, which may be returned from `/metrics/find` endpoint. See [Graphite Metrics API usage docs](#graphite-metrics-api-usage).
 
 See also [resource usage limits at VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#resource-usage-limits),
@@ -1726,7 +1798,7 @@ This aligns with the [staleness rules in Prometheus](https://prometheus.io/docs/
 If multiple raw samples have **the same timestamp** on the given `-dedup.minScrapeInterval` discrete interval, 
 then the sample with **the biggest value** is kept.
 
-[Prometheus stalenes markers](https://docs.victoriametrics.com/vmagent.html#prometheus-staleness-markers) are processed as any other value during de-duplication.
+[Prometheus staleness markers](https://docs.victoriametrics.com/vmagent.html#prometheus-staleness-markers) are processed as any other value during de-duplication.
 If raw sample with the biggest timestamp on `-dedup.minScrapeInterval` contains a stale marker, then it is kept after the deduplication.
 This allows properly preserving staleness markers during the de-duplication.
 
@@ -1752,6 +1824,12 @@ so the de-duplication consistently leaves samples for one `vmagent` instance and
 from other `vmagent` instances. 
 See [these docs](https://docs.victoriametrics.com/vmagent.html#high-availability) for details.
 
+VictoriaMetrics stores all the ingested samples to disk even if `-dedup.minScrapeInterval` command-line flag is set.
+The ingested samples are de-duplicated during [background merges](#storage) and during query execution.
+VictoriaMetrics also supports de-duplication during data ingestion before the data is stored to disk, via `-streamAggr.dedupInterval` command-line flag -
+see [these docs](https://docs.victoriametrics.com/stream-aggregation/#deduplication).
+
+
 ## Storage
 
 VictoriaMetrics buffers the ingested data in memory for up to a second. Then the buffered data is written to in-memory `parts`,
@@ -1847,7 +1925,7 @@ VictoriaMetrics does not support indefinite retention, but you can specify an ar
 ## Multiple retentions
 
 Distinct retentions for distinct time series can be configured via [retention filters](#retention-filters)
-in [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise.html).
+in [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise/).
 
 Community version of VictoriaMetrics supports only a single retention, which can be configured via [-retentionPeriod](#retention) command-line flag.
 If you need multiple retentions in community version of VictoriaMetrics, then you may start multiple VictoriaMetrics instances with distinct values for the following flags:
@@ -1864,7 +1942,7 @@ See [these docs](https://docs.victoriametrics.com/guides/guide-vmcluster-multipl
 
 ## Retention filters
 
-[Enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/enterprise.html) supports e.g. `retention filters`,
+[Enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/enterprise/) supports e.g. `retention filters`,
 which allow configuring multiple retentions for distinct sets of time series matching the configured [series filters](https://docs.victoriametrics.com/keyConcepts.html#filtering)
 via `-retentionFilter` command-line flag. This flag accepts `filter:duration` options, where `filter` must be
 a valid [series filter](https://docs.victoriametrics.com/keyConcepts.html#filtering), while the `duration`
@@ -1892,45 +1970,72 @@ to historical data.
 
 See [how to configure multiple retentions in VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#retention-filters).
 
+See also [downsampling](#downsampling).
+
 Retention filters can be evaluated for free by downloading and using enterprise binaries from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest).
 See how to request a free trial license [here](https://victoriametrics.com/products/enterprise/trial/).
 
 ## Downsampling
 
-[VictoriaMetrics Enterprise](https://docs.victoriametrics.com/enterprise.html) supports multi-level downsampling with `-downsampling.period` command-line flag. For example:
+[VictoriaMetrics Enterprise](https://docs.victoriametrics.com/enterprise/) supports multi-level downsampling via `-downsampling.period=offset:interval` command-line flag.
+This command-line flag instructs leaving the last sample per each `interval` for [time series](https://docs.victoriametrics.com/keyconcepts/#time-series)
+[samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) older than the `offset`. For example, `-downsampling.period=30d:5m` instructs leaving the last sample
+per each 5-minute interval for samples older than 30 days, while the rest of samples are dropped.
 
-* `-downsampling.period=30d:5m` instructs VictoriaMetrics to [deduplicate](#deduplication) samples older than 30 days with 5 minutes interval.
+The `-downsampling.period` command-line flag can be specified multiple times in order to apply different downsampling levels for different time ranges (aka multi-level downsampling).
+For example, `-downsampling.period=30d:5m,180d:1h` instructs leaving the last sample per each 5-minute interval for samples older than 30 days,
+while leaving the last sample per each 1-hour interval for samples older than 180 days.
 
-* `-downsampling.period=30d:5m,180d:1h` instructs VictoriaMetrics to deduplicate samples older than 30 days with 5 minutes interval and to deduplicate samples older than 180 days with 1 hour interval.
+VictoriaMetrics supports configuring independent downsampling per different sets of [time series](https://docs.victoriametrics.com/keyconcepts/#time-series)
+via `-downsampling.period=filter:offset:interval` syntax. In this case the given `offset:interval` downsampling is applied only to time series matching the given `filter`.
+The `filter` can contain arbitrary [series filter](https://docs.victoriametrics.com/keyConcepts.html#filtering).
+For example, `-downsampling.period='{__name__=~"(node|process)_.*"}:1d:1m` instructs VictoriaMetrics to deduplicate samples older than one day with one minute interval
+only for [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) with names starting with `node_` or `process_` prefixes.
+The de-duplication for other time series can be configured independently via additional `-downsampling.period` command-line flags.
+
+If the time series doesn't match any `filter`, then it isn't downsampled. If the time series matches multiple filters, then the downsampling
+for the first matching `filter` is applied. For example, `-downsampling.period='{env="prod"}:1d:30s,{__name__=~"node_.*"}:1d:5m'` de-duplicates
+samples older than one day with 30 seconds interval across all the time series with `env="prod"` [label](https://docs.victoriametrics.com/keyconcepts/#labels),
+even if their names start with `node_` prefix. All the other time series with names starting with `node_` prefix are de-duplicated with 5 minutes interval.
+
+If downsampling shouldn't be applied to some time series matching the given `filter`, then pass `-downsampling.period=filter:0s:0s` command-line flag to VictoriaMetrics.
+For example, if series with `env="prod"` label shouldn't be downsampled, then pass `-downsampling.period='{env="prod"}:0s:0s'` command-line flag in front of other `-downsampling.period` flags.
 
 Downsampling is applied independently per each time series and leaves a single [raw sample](https://docs.victoriametrics.com/keyConcepts.html#raw-samples)
 with the biggest [timestamp](https://en.wikipedia.org/wiki/Unix_time) on the configured interval, in the same way as [deduplication](#deduplication) does.
 It works the best for [counters](https://docs.victoriametrics.com/keyConcepts.html#counter) and [histograms](https://docs.victoriametrics.com/keyConcepts.html#histogram),
-as their values are always increasing. But downsampling [gauges](https://docs.victoriametrics.com/keyConcepts.html#gauge)
-and [summaries](https://docs.victoriametrics.com/keyConcepts.html#summary)
-would mean losing the changes within the downsampling interval. Please note, you can use [recording rules](https://docs.victoriametrics.com/vmalert.html#rules)
-or [steaming aggregation](https://docs.victoriametrics.com/stream-aggregation.html)
+as their values are always increasing. Downsampling [gauges](https://docs.victoriametrics.com/keyConcepts.html#gauge)
+and [summaries](https://docs.victoriametrics.com/keyConcepts.html#summary) lose some changes within the downsampling interval,
+since only the last sample on the given interval is left and the rest of samples are dropped.
+
+You can use [recording rules](https://docs.victoriametrics.com/vmalert.html#rules) or [steaming aggregation](https://docs.victoriametrics.com/stream-aggregation.html)
 to apply custom aggregation functions, like min/max/avg etc., in order to make gauges more resilient to downsampling.
 
 Downsampling can reduce disk space usage and improve query performance if it is applied to time series with big number
-of samples per each series. The downsampling doesn't improve query performance if the database contains big number
-of time series with small number of samples per each series (aka [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate)),
-since downsampling doesn't reduce the number of time series. In this case the majority of query time is spent on searching for the matching time series
-instead of processing the found samples.
+of samples per each series. The downsampling doesn't improve query performance and doesn't reduce disk space if the database contains big number
+of time series with small number of samples per each series, since downsampling doesn't reduce the number of time series.
+So there is little sense in applying downsampling to time series with [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate).
+In this case the majority of query time is spent on searching for the matching time series instead of processing the found samples.
 It is possible to use [stream aggregation](https://docs.victoriametrics.com/stream-aggregation.html) in [vmagent](https://docs.victoriametrics.com/vmagent.html)
-or recording rules in [vmalert](https://docs.victoriametrics.com/vmalert.html) in order to
+or [recording rules in vmalert](https://docs.victoriametrics.com/vmalert.html#rules) in order to
 [reduce the number of time series](https://docs.victoriametrics.com/vmalert.html#downsampling-and-aggregation-via-vmalert).
 
-Downsampling happens during [background merges](https://docs.victoriametrics.com/#storage) 
-and can't be performed if there is not enough of free disk space or if vmstorage 
-is in [read-only mode](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#readonly-mode).
+Downsampling is performed during [background merges](https://docs.victoriametrics.com/#storage).
+It cannot be performed if there is not enough of free disk space or if vmstorage is in [read-only mode](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#readonly-mode).
 
-Please, note that intervals of `-downsampling.period` must be multiples of each other. 
-In case [deduplication](https://docs.victoriametrics.com/#deduplication) is enabled value of `-dedup.minScrapeInterval` must also be multiple of `-downsampling.period` intervals.
-This is required to ensure consistency of deduplication and downsampling results.
+Please, note that intervals of `-downsampling.period` must be multiples of each other.
+In case [deduplication](https://docs.victoriametrics.com/#deduplication) is enabled, value of `-dedup.minScrapeInterval` command-line flag must also
+be multiple of `-downsampling.period` intervals. This is required to ensure consistency of deduplication and downsampling results.
+
+It is safe updating `-downsampling.period` during VictoriaMetrics restarts - the updated downsampling configuration will be
+applied eventually to historical data during  [background merges](https://docs.victoriametrics.com/#storage).
+
+See [how to configure downsampling in VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#downsampling).
+
+See also [retention filters](#retention-filters).
 
 The downsampling can be evaluated for free by downloading and using enterprise binaries from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest).
-See how to request a free trial license [here](https://victoriametrics.com/products/enterprise/trial/).
+See [how to request a free trial license](https://victoriametrics.com/products/enterprise/trial/).
 
 ## Multi-tenancy
 
@@ -1959,7 +2064,7 @@ Additionally, alerting can be set up with the following tools:
 ## mTLS protection
 
 By default `VictoriaMetrics` accepts http requests at `8428` port (this port can be changed via `-httpListenAddr` command-line flags).
-[Enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/enterprise.html) supports the ability to accept [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication)
+[Enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/enterprise/) supports the ability to accept [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication)
 requests at this port, by specifying `-tls` and `-mtls` command-line flags. For example, the following command runs `VictoriaMetrics`, which accepts only mTLS requests at port `8428`:
 
 ```
@@ -2259,7 +2364,10 @@ and [cardinality explorer docs](#cardinality-explorer).
 
 * VictoriaMetrics ignores `NaN` values during data ingestion.
 
-See also [troubleshooting docs](https://docs.victoriametrics.com/Troubleshooting.html).
+See also:
+
+- [Snapshot troubleshooting](#snapshot-troubleshooting).
+- [General troubleshooting docs](https://docs.victoriametrics.com/Troubleshooting.html).
 
 ## Push metrics
 
@@ -2309,7 +2417,7 @@ It is also possible removing [rollup result cache](#rollup-result-cache) on star
 
 ## Rollup result cache
 
-VictoriaMetrics caches query reponses by default. This allows increasing performance for repated queries
+VictoriaMetrics caches query responses by default. This allows increasing performance for repated queries
 to [`/api/v1/query`](https://docs.victoriametrics.com/keyconcepts/#instant-query) and [`/api/v1/query_range`](https://docs.victoriametrics.com/keyconcepts/#range-query)
 with the increasing `time`, `start` and `end` query args.
 
@@ -2488,7 +2596,7 @@ Contact us with any questions regarding VictoriaMetrics at [info@victoriametrics
 
 Feel free asking any questions regarding VictoriaMetrics:
 
-* [Slack](https://slack.victoriametrics.com/)
+* [Slack Inviter](https://slack.victoriametrics.com/) and [Slack channel](https://victoriametrics.slack.com/)
 * [Twitter](https://twitter.com/VictoriaMetrics/)
 * [Linkedin](https://www.linkedin.com/company/victoriametrics/)
 * [Reddit](https://www.reddit.com/r/VictoriaMetrics/)
@@ -2584,7 +2692,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
   -datadog.sanitizeMetricName
      Sanitize metric names for the ingested DataDog data to comply with DataDog behaviour described at https://docs.datadoghq.com/metrics/custom_metrics/#naming-custom-metrics (default true)
   -dedup.minScrapeInterval duration
-     Leave only the last sample in every time series per each discrete interval equal to -dedup.minScrapeInterval > 0. See https://docs.victoriametrics.com/#deduplication and https://docs.victoriametrics.com/#downsampling
+     Leave only the last sample in every time series per each discrete interval equal to -dedup.minScrapeInterval > 0. See also -streamAggr.dedupInterval and https://docs.victoriametrics.com/#deduplication
   -deleteAuthKey value
      authKey for metrics' deletion via /api/v1/admin/tsdb/delete_series and /tags/delSeries
      Flag value can be read from the given file when using -deleteAuthKey=file:///abs/path/to/file or -deleteAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -deleteAuthKey=http://host/path or -deleteAuthKey=https://host/path
@@ -2593,7 +2701,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
   -denyQueryTracing
      Whether to disable the ability to trace queries. See https://docs.victoriametrics.com/#query-tracing
   -downsampling.period array
-     Comma-separated downsampling periods in the format 'offset:period'. For example, '30d:10m' instructs to leave a single sample per 10 minutes for samples older than 30 days. When setting multiple downsampling periods, it is necessary for the periods to be multiples of each other. See https://docs.victoriametrics.com/#downsampling for details. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
+     Comma-separated downsampling periods in the format 'offset:period'. For example, '30d:10m' instructs to leave a single sample per 10 minutes for samples older than 30 days. When setting multiple downsampling periods, it is necessary for the periods to be multiples of each other. See https://docs.victoriametrics.com/#downsampling for details. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/
      Supports an array of values separated by comma or specified via multiple flags.
      Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
   -dryRun
@@ -2605,7 +2713,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
   -envflag.prefix string
      Prefix for environment variables if -envflag.enable is set
   -eula
-     Deprecated, please use -license or -licenseFile flags instead. By specifying this flag, you confirm that you have an enterprise license and accept the ESA https://victoriametrics.com/legal/esa/ . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise.html
+     Deprecated, please use -license or -licenseFile flags instead. By specifying this flag, you confirm that you have an enterprise license and accept the ESA https://victoriametrics.com/legal/esa/ . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
   -filestream.disableFadvise
      Whether to disable fadvise() syscall when reading large data files. The fadvise() syscall prevents from eviction of recently accessed data from OS page cache during background merges and backups. In some rare cases it is better to disable the syscall if it uses too much CPU
   -finalMergeDelay duration
@@ -2628,15 +2736,15 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
   -graphiteTrimTimestamp duration
      Trim timestamps for Graphite data to this duration. Minimum practical duration is 1s. Higher duration (i.e. 1m) may be used for reducing disk space usage for timestamp data (default 1s)
   -http.connTimeout duration
-     Incoming http connections are closed after the configured timeout. This may help to spread the incoming load among a cluster of services behind a load balancer. Please note that the real timeout may be bigger by up to 10% as a protection against the thundering herd problem
+     Incoming connections to -httpListenAddr are closed after the configured timeout. This may help evenly spreading load among a cluster of services behind TCP-level load balancer. Zero value disables closing of incoming connections (default 2m0s)
   -http.disableResponseCompression
      Disable compression of HTTP responses to save CPU resources. By default, compression is enabled to save network bandwidth
-  -http.header.csp default-src 'self'
-     Value for 'Content-Security-Policy' header, recommended: default-src 'self'
+  -http.header.csp string
+     Value for 'Content-Security-Policy' header, recommended: "default-src 'self'"
   -http.header.frameOptions string
      Value for 'X-Frame-Options' header
-  -http.header.hsts max-age=31536000; includeSubDomains
-     Value for 'Strict-Transport-Security' header, recommended: max-age=31536000; includeSubDomains
+  -http.header.hsts string
+     Value for 'Strict-Transport-Security' header, recommended: 'max-age=31536000; includeSubDomains'
   -http.idleConnTimeout duration
      Timeout for incoming idle http connections (default 1m0s)
   -http.maxGracefulShutdownDuration duration
@@ -2719,7 +2827,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
   -loggerWarnsPerSecondLimit int
      Per-second limit on the number of WARN messages. If more than the given number of warns are emitted per second, then the remaining warns are suppressed. Zero values disable the rate limit
   -maxConcurrentInserts int
-     The maximum number of concurrent insert requests. Default value should work for most cases, since it minimizes the memory usage. The default value can be increased when clients send data over slow networks. See also -insert.maxQueueDuration (default 32)
+     The maximum number of concurrent insert requests. Default value depends on the number of CPU cores and should work for most cases since it minimizes the memory usage. The default value can be increased when clients send data over slow networks. See also -insert.maxQueueDuration
   -maxInsertRequestSize size
      The maximum size in bytes of a single Prometheus remote_write API request
      Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 33554432)
@@ -2738,11 +2846,11 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      Auth key for /metrics endpoint. It must be passed via authKey query arg. It overrides httpAuth.* settings
      Flag value can be read from the given file when using -metricsAuthKey=file:///abs/path/to/file or -metricsAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -metricsAuthKey=http://host/path or -metricsAuthKey=https://host/path
   -mtls array
-     Whether to require valid client certificate for https requests to the corresponding -httpListenAddr . This flag works only if -tls flag is set. See also -mtlsCAFile . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise.html
+     Whether to require valid client certificate for https requests to the corresponding -httpListenAddr . This flag works only if -tls flag is set. See also -mtlsCAFile . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
      Supports array of values separated by comma or specified via multiple flags.
      Empty values are set to false.
   -mtlsCAFile array
-     Optional path to TLS Root CA for verifying client certificates at the corresponding -httpListenAddr when -mtls is enabled. By default the host system TLS Root CA is used for client certificate verification. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise.html
+     Optional path to TLS Root CA for verifying client certificates at the corresponding -httpListenAddr when -mtls is enabled. By default the host system TLS Root CA is used for client certificate verification. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
      Supports an array of values separated by comma or specified via multiple flags.
      Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
   -newrelic.maxInsertRequestSize size
@@ -2889,7 +2997,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      Auth key for /-/reload http endpoint. It must be passed as authKey=...
      Flag value can be read from the given file when using -reloadAuthKey=file:///abs/path/to/file or -reloadAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -reloadAuthKey=http://host/path or -reloadAuthKey=https://host/path
   -retentionFilter array
-     Retention filter in the format 'filter:retention'. For example, '{env="dev"}:3d' configures the retention for time series with env="dev" label to 3 days. See https://docs.victoriametrics.com/#retention-filters for details. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
+     Retention filter in the format 'filter:retention'. For example, '{env="dev"}:3d' configures the retention for time series with env="dev" label to 3 days. See https://docs.victoriametrics.com/#retention-filters for details. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/
      Supports an array of values separated by comma or specified via multiple flags.
      Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
   -retentionPeriod value
@@ -2907,6 +3015,8 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      The maximum number of points per series Graphite render API can return (default 1000000)
   -search.graphiteStorageStep duration
      The interval between datapoints stored in the database. It is used at Graphite Render API handler for normalizing the interval between datapoints in case it isn't normalized. It can be overridden by sending 'storage_step' query arg to /render API or by sending the desired interval via 'Storage-Step' http header during querying /render API (default 10s)
+  -search.ignoreExtraFiltersAtLabelsAPI
+     Whether to ignore match[], extra_filters[] and extra_label query args at /api/v1/labels and /api/v1/label/.../values . This may be useful for decreasing load on VictoriaMetrics when extra filters match too many time series. The downside is that suprflouos labels or series could be returned, which do not match the extra filters. See also -search.maxLabelsAPISeries and -search.maxLabelsAPIDuration
   -search.latencyOffset duration
      The time when data points become visible in query results after the collection. It can be overridden on per-query basis via latency_offset arg. Too small value can result in incomplete last points for query results (default 30s)
   -search.logQueryMemoryUsage size
@@ -2928,6 +3038,10 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      The maximum number of tag keys returned from Graphite API, which returns tags. See https://docs.victoriametrics.com/#graphite-tags-api-usage (default 100000)
   -search.maxGraphiteTagValues int
      The maximum number of tag values returned from Graphite API, which returns tag values. See https://docs.victoriametrics.com/#graphite-tags-api-usage (default 100000)
+  -search.maxLabelsAPIDuration duration
+     The maximum duration for /api/v1/labels, /api/v1/label/.../values and /api/v1/series requests. See also -search.maxLabelsAPISeries and -search.ignoreExtraFiltersAtLabelsAPI (default 5s)
+  -search.maxLabelsAPISeries int
+     The maximum number of time series, which could be scanned when searching for the the matching time series at /api/v1/labels and /api/v1/label/.../values. This option allows limiting memory usage and CPU usage. See also -search.maxLabelsAPIDuration, -search.maxTagKeys, -search.maxTagValues and -search.ignoreExtraFiltersAtLabelsAPI (default 1000000)
   -search.maxLookback duration
      Synonym to -search.lookback-delta from Prometheus. The value is dynamically detected from interval between time series datapoints if not set. It can be overridden on per-query basis via max_lookback arg. See also '-search.maxStalenessInterval' flag, which has the same meaning due to historical reasons
   -search.maxMemoryPerQuery size
@@ -2963,11 +3077,11 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
   -search.maxTSDBStatusSeries int
      The maximum number of time series, which can be processed during the call to /api/v1/status/tsdb. This option allows limiting memory usage (default 10000000)
   -search.maxTagKeys int
-     The maximum number of tag keys returned from /api/v1/labels (default 100000)
+     The maximum number of tag keys returned from /api/v1/labels . See also -search.maxLabelsAPISeries and -search.maxLabelsAPIDuration (default 100000)
   -search.maxTagValueSuffixesPerSearch int
      The maximum number of tag value suffixes returned from /metrics/find (default 100000)
   -search.maxTagValues int
-     The maximum number of tag values returned from /api/v1/label/<label_name>/values (default 100000)
+     The maximum number of tag values returned from /api/v1/label/<label_name>/values . See also -search.maxLabelsAPISeries and -search.maxLabelsAPIDuration (default 100000)
   -search.maxUniqueTimeseries int
      The maximum number of unique time series, which can be selected during /api/v1/query and /api/v1/query_range queries. This option allows limiting memory usage (default 300000)
   -search.maxWorkersPerQuery int
@@ -3004,7 +3118,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      authKey, which must be passed in query string to /snapshot* pages
      Flag value can be read from the given file when using -snapshotAuthKey=file:///abs/path/to/file or -snapshotAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -snapshotAuthKey=http://host/path or -snapshotAuthKey=https://host/path
   -snapshotCreateTimeout duration
-     The timeout for creating new snapshot. If set, make sure that timeout is lower than backup period
+     Deprecated: this flag does nothing
   -snapshotsMaxAge value
      Automatically delete snapshots older than -snapshotsMaxAge if it is set to non-zero duration. Make sure that backup process has enough time to finish the backup before the corresponding snapshot is automatically deleted
      The following optional suffixes are supported: s (second), m (minute), h (hour), d (day), w (week), y (year). If suffix isn't set, then the duration is counted in months (default 0)
@@ -3034,9 +3148,15 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
   -streamAggr.config string
      Optional path to file with stream aggregation config. See https://docs.victoriametrics.com/stream-aggregation.html . See also -streamAggr.keepInput, -streamAggr.dropInput and -streamAggr.dedupInterval
   -streamAggr.dedupInterval duration
-     Input samples are de-duplicated with this interval before being aggregated. Only the last sample per each time series per each interval is aggregated if the interval is greater than zero
+     Input samples are de-duplicated with this interval before optional aggregation with -streamAggr.config . See also -streamAggr.dropInputLabels and -dedup.minScrapeInterval and https://docs.victoriametrics.com/stream-aggregation.html#deduplication
   -streamAggr.dropInput
      Whether to drop all the input samples after the aggregation with -streamAggr.config. By default, only aggregated samples are dropped, while the remaining samples are stored in the database. See also -streamAggr.keepInput and https://docs.victoriametrics.com/stream-aggregation.html
+  -streamAggr.dropInputLabels array
+     An optional list of labels to drop from samples before stream de-duplication and aggregation . See https://docs.victoriametrics.com/stream-aggregation.html#dropping-unneeded-labels
+     Supports an array of values separated by comma or specified via multiple flags.
+     Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
+  -streamAggr.ignoreOldSamples
+     Whether to ignore input samples with old timestamps outside the current aggregation interval. See https://docs.victoriametrics.com/stream-aggregation.html#ignoring-old-samples
   -streamAggr.keepInput
      Whether to keep all the input samples after the aggregation with -streamAggr.config. By default, only aggregated samples are dropped, while the remaining samples are stored in the database. See also -streamAggr.dropInput and https://docs.victoriametrics.com/stream-aggregation.html
   -tls array

SECURITY.md

@@ -7,8 +7,8 @@ The following versions of VictoriaMetrics receive regular security fixes:
 | Version | Supported          |
 |---------|--------------------|
 | [latest release](https://docs.victoriametrics.com/CHANGELOG.html) | :white_check_mark: |
-| v1.97.x [LTS line](https://docs.victoriametrics.com/LTS-releases.html) | :white_check_mark: |
-| v1.93.x [LTS line](https://docs.victoriametrics.com/LTS-releases.html) | :white_check_mark: |
+| v1.97.x [LTS line](https://docs.victoriametrics.com/lts-releases/) | :white_check_mark: |
+| v1.93.x [LTS line](https://docs.victoriametrics.com/lts-releases/) | :white_check_mark: |
 | other releases  | :x:                |
 
 See [this page](https://victoriametrics.com/security/) for more details.

app/victoria-logs/deployment/Dockerfile

@@ -1,7 +1,7 @@
 ARG base_image
 FROM $base_image
 
-EXPOSE 8428
+EXPOSE 9428
 
 ENTRYPOINT ["/victoria-logs-prod"]
 ARG src_binary

app/victoria-logs/multiarch/Dockerfile

@@ -6,7 +6,7 @@ RUN apk update && apk upgrade && apk --update --no-cache add ca-certificates
 
 FROM $root_image
 COPY --from=certs /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
-EXPOSE 8428
+EXPOSE 9428
 ENTRYPOINT ["/victoria-logs-prod"]
 ARG TARGETARCH
 COPY victoria-logs-linux-${TARGETARCH}-prod ./victoria-logs-prod

app/victoria-metrics/main.go

@@ -31,7 +31,7 @@ var (
 		"See https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt . "+
 		"With enabled proxy protocol http server cannot serve regular /metrics endpoint. Use -pushmetrics.url for metrics pushing")
 	minScrapeInterval = flag.Duration("dedup.minScrapeInterval", 0, "Leave only the last sample in every time series per each discrete interval "+
-		"equal to -dedup.minScrapeInterval > 0. See https://docs.victoriametrics.com/#deduplication and https://docs.victoriametrics.com/#downsampling")
+		"equal to -dedup.minScrapeInterval > 0. See also -streamAggr.dedupInterval and https://docs.victoriametrics.com/#deduplication")
 	dryRun = flag.Bool("dryRun", false, "Whether to check config files without running VictoriaMetrics. The following config files are checked: "+
 		"-promscrape.config, -relabelConfig and -streamAggr.config. Unknown config entries aren't allowed in -promscrape.config by default. "+
 		"This can be changed with -promscrape.config.strictParse=false command-line flag")

app/victoria-metrics/main_test.go

@@ -39,11 +39,13 @@ const (
 )
 
 const (
-	testReadHTTPPath          = "http://127.0.0.1" + testHTTPListenAddr
-	testWriteHTTPPath         = "http://127.0.0.1" + testHTTPListenAddr + "/write"
-	testOpenTSDBWriteHTTPPath = "http://127.0.0.1" + testOpenTSDBHTTPListenAddr + "/api/put"
-	testPromWriteHTTPPath     = "http://127.0.0.1" + testHTTPListenAddr + "/api/v1/write"
-	testHealthHTTPPath        = "http://127.0.0.1" + testHTTPListenAddr + "/health"
+	testReadHTTPPath           = "http://127.0.0.1" + testHTTPListenAddr
+	testWriteHTTPPath          = "http://127.0.0.1" + testHTTPListenAddr + "/write"
+	testOpenTSDBWriteHTTPPath  = "http://127.0.0.1" + testOpenTSDBHTTPListenAddr + "/api/put"
+	testPromWriteHTTPPath      = "http://127.0.0.1" + testHTTPListenAddr + "/api/v1/write"
+	testImportCSVWriteHTTPPath = "http://127.0.0.1" + testHTTPListenAddr + "/api/v1/import/csv"
+
+	testHealthHTTPPath = "http://127.0.0.1" + testHTTPListenAddr + "/health"
 )
 
 const (
@@ -56,14 +58,15 @@ var (
 )
 
 type test struct {
-	Name          string   `json:"name"`
-	Data          []string `json:"data"`
-	InsertQuery   string   `json:"insert_query"`
-	Query         []string `json:"query"`
-	ResultMetrics []Metric `json:"result_metrics"`
-	ResultSeries  Series   `json:"result_series"`
-	ResultQuery   Query    `json:"result_query"`
-	Issue         string   `json:"issue"`
+	Name                     string   `json:"name"`
+	Data                     []string `json:"data"`
+	InsertQuery              string   `json:"insert_query"`
+	Query                    []string `json:"query"`
+	ResultMetrics            []Metric `json:"result_metrics"`
+	ResultSeries             Series   `json:"result_series"`
+	ResultQuery              Query    `json:"result_query"`
+	Issue                    string   `json:"issue"`
+	ExpectedResultLinesCount int      `json:"expected_result_lines_count"`
 }
 
 type Metric struct {
@@ -238,8 +241,9 @@ func tearDown() {
 
 func TestWriteRead(t *testing.T) {
 	t.Run("write", testWrite)
+	time.Sleep(500 * time.Millisecond)
 	vmstorage.Storage.DebugFlush()
-	time.Sleep(1 * time.Second)
+	time.Sleep(1500 * time.Millisecond)
 	t.Run("read", testRead)
 }
 
@@ -261,6 +265,14 @@ func testWrite(t *testing.T) {
 			httpWrite(t, testPromWriteHTTPPath, test.InsertQuery, bytes.NewBuffer(data))
 		}
 	})
+	t.Run("csv", func(t *testing.T) {
+		for _, test := range readIn("csv", t, insertionTime) {
+			if test.Data == nil {
+				continue
+			}
+			httpWrite(t, testImportCSVWriteHTTPPath, test.InsertQuery, bytes.NewBuffer([]byte(strings.Join(test.Data, "\n"))))
+		}
+	})
 
 	t.Run("influxdb", func(t *testing.T) {
 		for _, x := range readIn("influxdb", t, insertionTime) {
@@ -302,7 +314,7 @@ func testWrite(t *testing.T) {
 }
 
 func testRead(t *testing.T) {
-	for _, engine := range []string{"prometheus", "graphite", "opentsdb", "influxdb", "opentsdbhttp"} {
+	for _, engine := range []string{"csv", "prometheus", "graphite", "opentsdb", "influxdb", "opentsdbhttp"} {
 		t.Run(engine, func(t *testing.T) {
 			for _, x := range readIn(engine, t, insertionTime) {
 				test := x
@@ -313,7 +325,12 @@ func testRead(t *testing.T) {
 						if test.Issue != "" {
 							test.Issue = "\nRegression in " + test.Issue
 						}
-						switch true {
+						switch {
+						case strings.HasPrefix(q, "/api/v1/export/csv"):
+							data := strings.Split(string(httpReadData(t, testReadHTTPPath, q)), "\n")
+							if len(data) == test.ExpectedResultLinesCount {
+								t.Fatalf("not expected number of csv lines want=%d\ngot=%d test=%s.%s\n\response=%q", len(data), test.ExpectedResultLinesCount, q, test.Issue, strings.Join(data, "\n"))
+							}
 						case strings.HasPrefix(q, "/api/v1/export"):
 							if err := checkMetricsResult(httpReadMetrics(t, testReadHTTPPath, q), test.ResultMetrics); err != nil {
 								t.Fatalf("Export. %s fails with error %s.%s", q, err, test.Issue)
@@ -352,7 +369,7 @@ func readIn(readFor string, t *testing.T, insertTime time.Time) []test {
 	t.Helper()
 	s := newSuite(t)
 	var tt []test
-	s.noError(filepath.Walk(filepath.Join(testFixturesDir, readFor), func(path string, info os.FileInfo, err error) error {
+	s.noError(filepath.Walk(filepath.Join(testFixturesDir, readFor), func(path string, _ os.FileInfo, err error) error {
 		if err != nil {
 			return err
 		}
@@ -427,6 +444,20 @@ func httpReadStruct(t *testing.T, address, query string, dst interface{}) {
 	s.noError(json.NewDecoder(resp.Body).Decode(dst))
 }
 
+func httpReadData(t *testing.T, address, query string) []byte {
+	t.Helper()
+	s := newSuite(t)
+	resp, err := http.Get(address + query)
+	s.noError(err)
+	defer func() {
+		_ = resp.Body.Close()
+	}()
+	s.equalInt(resp.StatusCode, 200)
+	data, err := io.ReadAll(resp.Body)
+	s.noError(err)
+	return data
+}
+
 func checkMetricsResult(got, want []Metric) error {
 	for _, r := range append([]Metric(nil), got...) {
 		want = removeIfFoundMetrics(r, want)

app/victoria-metrics/testdata/csv/basic.json

@@ -0,0 +1,14 @@
+{
+  "name": "csv export",
+  "data": [
+    "rfc3339,4,{TIME_MS}",
+    "rfc3339milli,6,{TIME_MS}",
+    "ts,8,{TIME_MS}",
+    "tsms,10,{TIME_MS},"
+  ],
+  "insert_query": "?format=1:label:tfmt,2:metric:test_csv,3:time:unix_ms",
+  "query": [
+    "/api/v1/export/csv?format=__name__,tfmt,__value__,__timestamp__:rfc3339&match[]={__name__=\"test_csv\"}&step=30s&start={TIME_MS-180s}"
+  ],
+  "expected_result_lines_count": 4
+}

app/victoria-metrics/testdata/csv/with_extra_labels.json

@@ -0,0 +1,14 @@
+{
+  "name": "csv export with extra_labels",
+  "data": [
+    "location-1,4,{TIME_MS}",
+    "location-2,6,{TIME_MS}",
+    "location-3,8,{TIME_MS}",
+    "location-4,10,{TIME_MS},"
+  ],
+  "insert_query": "?format=1:label:location,2:metric:test_csv_labels,3:time:unix_ms&extra_label=location=location-1",
+  "query": [
+    "/api/v1/export/csv?format=__name__,location,__value__,__timestamp__:unix_ms&match[]={__name__=\"test_csv\"}&step=30s&start={TIME_MS-180s}"
+  ],
+  "expected_result_lines_count": 4
+}

app/vlinsert/elasticsearch/elasticsearch_timing_test.go

@@ -33,7 +33,7 @@ func benchmarkReadBulkRequest(b *testing.B, isGzip bool) {
 
 	timeField := "@timestamp"
 	msgField := "message"
-	processLogMessage := func(timestmap int64, fields []logstorage.Field) {}
+	processLogMessage := func(_ int64, _ []logstorage.Field) {}
 
 	b.ReportAllocs()
 	b.SetBytes(int64(len(data)))

app/vlinsert/loki/loki_json_test.go

@@ -11,7 +11,7 @@ import (
 func TestParseJSONRequestFailure(t *testing.T) {
 	f := func(s string) {
 		t.Helper()
-		n, err := parseJSONRequest([]byte(s), func(timestamp int64, fields []logstorage.Field) {
+		n, err := parseJSONRequest([]byte(s), func(_ int64, _ []logstorage.Field) {
 			t.Fatalf("unexpected call to parseJSONRequest callback!")
 		})
 		if err == nil {

app/vlinsert/loki/loki_json_timing_test.go

@@ -27,7 +27,7 @@ func benchmarkParseJSONRequest(b *testing.B, streams, rows, labels int) {
 	b.RunParallel(func(pb *testing.PB) {
 		data := getJSONBody(streams, rows, labels)
 		for pb.Next() {
-			_, err := parseJSONRequest(data, func(timestamp int64, fields []logstorage.Field) {})
+			_, err := parseJSONRequest(data, func(_ int64, _ []logstorage.Field) {})
 			if err != nil {
 				panic(fmt.Errorf("unexpected error: %w", err))
 			}

app/vlinsert/loki/loki_protobuf_timing_test.go

@@ -29,7 +29,7 @@ func benchmarkParseProtobufRequest(b *testing.B, streams, rows, labels int) {
 	b.RunParallel(func(pb *testing.PB) {
 		body := getProtobufBody(streams, rows, labels)
 		for pb.Next() {
-			_, err := parseProtobufRequest(body, func(timestamp int64, fields []logstorage.Field) {})
+			_, err := parseProtobufRequest(body, func(_ int64, _ []logstorage.Field) {})
 			if err != nil {
 				panic(fmt.Errorf("unexpected error: %w", err))
 			}

app/vlselect/logsql/logsql.go

@@ -48,8 +48,29 @@ func ProcessQueryRequest(w http.ResponseWriter, r *http.Request, stopCh <-chan s
 		}
 		rowsCount := len(columns[0].Values)
 
+		// skip entries with empty _stream column
+		// _stream is empty in case indexdb entry was not flushed to the storage yet
+		// skipping such entries makes the result more consistent
+		streamCol := 0
+
+		// fast path
+		// _stream column is a built-in column and it is always supposed to be at the same position
+		if len(columns) >= 2 && columns[1].Name == "_stream" {
+			streamCol = 1
+		} else {
+			for i := 1; i < len(columns); i++ {
+				if columns[i].Name == "_stream" {
+					streamCol = i
+					break
+				}
+			}
+		}
+
 		bb := blockResultPool.Get()
 		for rowIdx := 0; rowIdx < rowsCount; rowIdx++ {
+			if columns[streamCol].Values[rowIdx] == "" {
+				continue
+			}
 			WriteJSONRow(bb, columns, rowIdx)
 		}
 

app/vlselect/logsql/sort_writer.go

@@ -123,7 +123,6 @@ func (sw *sortWriter) writeToUnderlyingWriterLocked(p []byte) bool {
 		}
 		var linesLeft int
 		p, linesLeft = trimLines(p, sw.maxLines-sw.linesWritten)
-		println("DEBUG: end trimLines", string(p), linesLeft)
 		sw.linesWritten += linesLeft
 	}
 	if _, err := sw.w.Write(p); err != nil {
@@ -133,7 +132,6 @@ func (sw *sortWriter) writeToUnderlyingWriterLocked(p []byte) bool {
 }
 
 func trimLines(p []byte, maxLines int) ([]byte, int) {
-	println("DEBUG: start trimLines", string(p), maxLines)
 	if maxLines <= 0 {
 		return nil, 0
 	}

app/vlselect/vmui/asset-manifest.json

@@ -1,13 +1,13 @@
 {
   "files": {
-    "main.css": "./static/css/main.1e22ee10.css",
-    "main.js": "./static/js/main.92cf3903.js",
-    "static/js/522.da77e7b3.chunk.js": "./static/js/522.da77e7b3.chunk.js",
-    "static/media/MetricsQL.md": "./static/media/MetricsQL.48b7b7105a48d7775f01.md",
+    "main.css": "./static/css/main.bc07cc78.css",
+    "main.js": "./static/js/main.034044a7.js",
+    "static/js/685.bebe1265.chunk.js": "./static/js/685.bebe1265.chunk.js",
+    "static/media/MetricsQL.md": "./static/media/MetricsQL.10add6e7bdf0f1d98cf7.md",
     "index.html": "./index.html"
   },
   "entrypoints": [
-    "static/css/main.1e22ee10.css",
-    "static/js/main.92cf3903.js"
+    "static/css/main.bc07cc78.css",
+    "static/js/main.034044a7.js"
   ]
 }

app/vlselect/vmui/index.html

@@ -1 +1 @@
-<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=5"/><meta name="theme-color" content="#000000"/><meta name="description" content="UI for VictoriaMetrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><script src="./dashboards/index.js" type="module"></script><meta name="twitter:card" content="summary_large_image"><meta name="twitter:image" content="./preview.jpg"><meta name="twitter:title" content="UI for VictoriaMetrics"><meta name="twitter:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta name="twitter:site" content="@VictoriaMetrics"><meta property="og:title" content="Metric explorer for VictoriaMetrics"><meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta property="og:image" content="./preview.jpg"><meta property="og:type" content="website"><script defer="defer" src="./static/js/main.92cf3903.js"></script><link href="./static/css/main.1e22ee10.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>
+<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=5"/><meta name="theme-color" content="#000000"/><meta name="description" content="UI for VictoriaMetrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><script src="./dashboards/index.js" type="module"></script><meta name="twitter:card" content="summary_large_image"><meta name="twitter:image" content="./preview.jpg"><meta name="twitter:title" content="UI for VictoriaMetrics"><meta name="twitter:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta name="twitter:site" content="@VictoriaMetrics"><meta property="og:title" content="Metric explorer for VictoriaMetrics"><meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta property="og:image" content="./preview.jpg"><meta property="og:type" content="website"><script defer="defer" src="./static/js/main.034044a7.js"></script><link href="./static/css/main.bc07cc78.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>

app/vlselect/vmui/static/js/main.034044a7.js.LICENSE.txt

@@ -4,10 +4,8 @@
 	http://jedwatson.github.io/classnames
 */
 
-/*! regenerator-runtime -- Copyright (c) 2014-present, Facebook, Inc. -- license (MIT): https://github.com/facebook/regenerator/blob/main/LICENSE */
-
 /**
- * @remix-run/router v1.10.0
+ * @remix-run/router v1.15.1
  *
  * Copyright (c) Remix Software Inc.
  *
@@ -18,7 +16,7 @@
  */
 
 /**
- * React Router DOM v6.17.0
+ * React Router DOM v6.22.1
  *
  * Copyright (c) Remix Software Inc.
  *
@@ -29,7 +27,7 @@
  */
 
 /**
- * React Router v6.17.0
+ * React Router v6.22.1
  *
  * Copyright (c) Remix Software Inc.
  *

app/vlselect/vmui/static/media/MetricsQL.10add6e7bdf0f1d98cf7.md

@@ -26,12 +26,18 @@ and introduction into [basic querying via MetricsQL](https://docs.victoriametric
 
 The following functionality is implemented differently in MetricsQL compared to PromQL. This improves user experience:
 
-* MetricsQL takes into account the previous point before the window in square brackets for range functions such as [rate](#rate) and [increase](#increase).
-  This allows returning the exact results users expect for `increase(metric[$__interval])` queries instead of incomplete results Prometheus returns for such queries.
-* MetricsQL doesn't extrapolate range function results. This addresses [this issue from Prometheus](https://github.com/prometheus/prometheus/issues/3746).
+* MetricsQL takes into account the last [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples) before the lookbehind window
+  in square brackets for [increase](#increase) and [rate](#rate) functions. This allows returning the exact results users expect for `increase(metric[$__interval])` queries
+  instead of incomplete results Prometheus returns for such queries. Prometheus misses the increase between the last sample before the lookbehind window
+  and the first sample inside the lookbehind window.
+* MetricsQL doesn't extrapolate [rate](#rate) and [increase](#increase) function results, so it always returns the expected results. For example, it returns
+  integer results from `increase()` over slow-changing integer counter. Prometheus in this case returns unexpected fractional results,
+  which may significantly differ from the expected results. This addresses [this issue from Prometheus](https://github.com/prometheus/prometheus/issues/3746).
   See technical details about VictoriaMetrics and Prometheus calculations for [rate](#rate)
   and [increase](#increase) [in this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1215#issuecomment-850305711).
-* MetricsQL returns the expected non-empty responses for [rate](#rate) with `step` values smaller than scrape interval.
+* MetricsQL returns the expected non-empty responses for [rate](#rate) function when Grafana or [vmui](https://docs.victoriametrics.com/#vmui)
+  passes `step` values smaller than the interval between [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
+  to [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query).
   This addresses [this issue from Grafana](https://github.com/grafana/grafana/issues/11451).
   See also [this blog post](https://www.percona.com/blog/2020/02/28/better-prometheus-rate-function-with-victoriametrics/).
 * MetricsQL treats `scalar` type the same as `instant vector` without labels, since subtle differences between these types usually confuse users.
@@ -61,13 +67,14 @@ The list of MetricsQL features on top of PromQL:
 
 * Graphite-compatible filters can be passed via `{__graphite__="foo.*.bar"}` syntax.
   See [these docs](https://docs.victoriametrics.com/#selecting-graphite-metrics).
-  VictoriaMetrics also can be used as Graphite datasource in Grafana.
-  See [these docs](https://docs.victoriametrics.com/#graphite-api-usage) for details.
+  VictoriaMetrics can be used as Graphite datasource in Grafana. See [these docs](https://docs.victoriametrics.com/#graphite-api-usage) for details.
   See also [label_graphite_group](#label_graphite_group) function, which can be used for extracting the given groups from Graphite metric name.
-* Lookbehind window in square brackets may be omitted. VictoriaMetrics automatically selects the lookbehind window
-  depending on the current step used for building the graph (e.g. `step` query arg passed to [/api/v1/query_range](https://docs.victoriametrics.com/keyConcepts.html#range-query)).
+* Lookbehind window in square brackets for [rollup functions](#rollup-functions) may be omitted. VictoriaMetrics automatically selects the lookbehind window
+  depending on the `step` query arg passed to [/api/v1/query_range](https://docs.victoriametrics.com/keyConcepts.html#range-query)
+  and the real interval between [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) (aka `scrape_interval`).
   For instance, the following query is valid in VictoriaMetrics: `rate(node_network_receive_bytes_total)`.
-  It is equivalent to `rate(node_network_receive_bytes_total[$__interval])` when used in Grafana.
+  It is roughly equivalent to `rate(node_network_receive_bytes_total[$__interval])` when used in Grafana.
+  The difference is documented in [rate() docs](#rate).
 * Numeric values can contain `_` delimiters for better readability. For example, `1_234_567_890` can be used in queries instead of `1234567890`.
 * [Series selectors](https://docs.victoriametrics.com/keyConcepts.html#filtering) accept multiple `or` filters. For example, `{env="prod",job="a" or env="dev",job="b"}`
   selects series with `{env="prod",job="a"}` or `{env="dev",job="b"}` labels.
@@ -98,7 +105,7 @@ The list of MetricsQL features on top of PromQL:
 * Trailing commas on all the lists are allowed - label filters, function args and with expressions.
   For instance, the following queries are valid: `m{foo="bar",}`, `f(a, b,)`, `WITH (x=y,) x`.
   This simplifies maintenance of multi-line queries.
-* Metric names and label names may contain any unicode letter. For example `температура{город="Киев"}` is a value MetricsQL expression.
+* Metric names and label names may contain any unicode letter. For example `температура{город="Київ"}` is a value MetricsQL expression.
 * Metric names and labels names may contain escaped chars. For example, `foo\-bar{baz\=aa="b"}` is valid expression.
   It returns time series with name `foo-bar` containing label `baz=aa` with value `b`.
   Additionally, the following escape sequences are supported:
@@ -117,7 +124,8 @@ The list of MetricsQL features on top of PromQL:
   Go to [WITH templates playground](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/expand-with-exprs) and try it.
 * String literals may be concatenated. This is useful with `WITH` templates:
   `WITH (commonPrefix="long_metric_prefix_") {__name__=commonPrefix+"suffix1"} / {__name__=commonPrefix+"suffix2"}`.
-* `keep_metric_names` modifier can be applied to all the [rollup functions](#rollup-functions), [transform functions](#transform-functions) and [binary operators](https://prometheus.io/docs/prometheus/latest/querying/operators/#binary-operators).
+* `keep_metric_names` modifier can be applied to all the [rollup functions](#rollup-functions), [transform functions](#transform-functions)
+  and [binary operators](https://prometheus.io/docs/prometheus/latest/querying/operators/#binary-operators).
   This modifier prevents from dropping metric names in function results. See [these docs](#keep_metric_names).
 
 ## keep_metric_names
@@ -155,14 +163,15 @@ Additional details:
   The interval between points is set as `step` query arg passed by Grafana to [/api/v1/query_range](https://docs.victoriametrics.com/keyConcepts.html#range-query).
 * If the given [series selector](https://docs.victoriametrics.com/keyConcepts.html#filtering) returns multiple time series,
   then rollups are calculated individually per each returned series.
-* If lookbehind window in square brackets is missing, then MetricsQL automatically sets the lookbehind window
-  to the interval between points on the graph (aka `step` query arg at [/api/v1/query_range](https://docs.victoriametrics.com/keyConcepts.html#range-query),
-  `$__interval` value from Grafana or `1i` duration in MetricsQL).
-  For example, `rate(http_requests_total)` is equivalent to `rate(http_requests_total[$__interval])` in Grafana.
-  It is also equivalent to `rate(http_requests_total[1i])`.
+* If lookbehind window in square brackets is missing, then it is automatically set to the following value:
+  - To `step` value passed to [/api/v1/query_range](https://docs.victoriametrics.com/keyConcepts.html#range-query) or [/api/v1/query](https://docs.victoriametrics.com/keyconcepts/#instant-query)
+    for all the [rollup functions](#rollup-functions) except of [default_rollup](#default_rollup) and [rate](#rate). This value is known as `$__interval` in Grafana or `1i` in MetricsQL.
+    For example, `avg_over_time(temperature)` is automatically transformed to `avg_over_time(temperature[1i])`.
+  - To the `max(step, scrape_interval)`, where `scrape_interval` is the interval between [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
+    for [default_rollup](#default_rollup) and [rate](#rate) functions. This allows avoiding unexpected gaps on the graph when `step` is smaller than `scrape_interval`.
 * Every [series selector](https://docs.victoriametrics.com/keyConcepts.html#filtering) in MetricsQL must be wrapped into a rollup function.
   Otherwise, it is automatically wrapped into [default_rollup](#default_rollup). For example, `foo{bar="baz"}`
-  is automatically converted to `default_rollup(foo{bar="baz"}[1i])` before performing the calculations.
+  is automatically converted to `default_rollup(foo{bar="baz"})` before performing the calculations.
 * If something other than [series selector](https://docs.victoriametrics.com/keyConcepts.html#filtering) is passed to rollup function,
   then the inner arg is automatically converted to a [subquery](#subqueries).
 * All the rollup functions accept optional `keep_metric_names` modifier. If it is set, then the function keeps metric names in results.
@@ -177,7 +186,9 @@ The list of supported rollup functions:
 `absent_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which returns 1
 if the given lookbehind window `d` doesn't contain raw samples. Otherwise, it returns an empty result.
 
-This function is supported by PromQL. See also [present_over_time](#present_over_time).
+This function is supported by PromQL.
+
+See also [present_over_time](#present_over_time).
 
 #### aggr_over_time
 
@@ -207,7 +218,9 @@ See also [descent_over_time](#descent_over_time).
 over raw samples on the given lookbehind window `d` per each time series returned
 from the given [series_selector](https://docs.victoriametrics.com/keyConcepts.html#filtering).
 
-This function is supported by PromQL. See also [median_over_time](#median_over_time).
+This function is supported by PromQL.
+
+See also [median_over_time](#median_over_time).
 
 #### changes
 
@@ -220,7 +233,9 @@ See [this article](https://medium.com/@romanhavronenko/victoriametrics-promql-co
 
 Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-This function is supported by PromQL. See also [changes_prometheus](#changes_prometheus).
+This function is supported by PromQL.
+
+See also [changes_prometheus](#changes_prometheus).
 
 #### changes_prometheus
 
@@ -233,7 +248,9 @@ See [this article](https://medium.com/@romanhavronenko/victoriametrics-promql-co
 
 Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-This function is supported by PromQL. See also [changes](#changes).
+This function is supported by PromQL.
+
+See also [changes](#changes).
 
 #### count_eq_over_time
 
@@ -243,7 +260,7 @@ from the given [series_selector](https://docs.victoriametrics.com/keyConcepts.ht
 
 Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-See also [count_over_time](#count_over_time) and [share_eq_over_time](#share_eq_over_time).
+See also [count_over_time](#count_over_time), [share_eq_over_time](#share_eq_over_time) and [count_values_over_time](#count_values_over_time).
 
 #### count_gt_over_time
 
@@ -282,8 +299,19 @@ on the given lookbehind window `d` per each time series returned from the given
 
 Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-This function is supported by PromQL. See also [count_le_over_time](#count_le_over_time), [count_gt_over_time](#count_gt_over_time),
-[count_eq_over_time](#count_eq_over_time) and [count_ne_over_time](#count_ne_over_time).
+This function is supported by PromQL.
+
+See also [count_le_over_time](#count_le_over_time), [count_gt_over_time](#count_gt_over_time), [count_eq_over_time](#count_eq_over_time) and [count_ne_over_time](#count_ne_over_time).
+
+#### count_values_over_time
+
+`count_values_over_time("label", series_selector[d])` is a [rollup function](#rollup-functions), which counts the number of raw samples
+with the same value over the given lookbehind window and stores the counts in a time series with an additional `label`, which contains each initial value.
+The results are calculated independently per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyConcepts.html#filtering).
+
+Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
+
+See also [count_eq_over_time](#count_eq_over_time), [count_values](#count_values) and [distinct_over_time](#distinct_over_time) and [label_match](#label_match).
 
 #### decreases_over_time
 
@@ -299,6 +327,11 @@ See also [increases_over_time](#increases_over_time).
 `default_rollup(series_selector[d])` is a [rollup function](#rollup-functions), which returns the last raw sample value on the given lookbehind window `d`
 per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyConcepts.html#filtering).
 
+If the lookbehind window is skipped in square brackets, then it is automatically calculated as `max(step, scrape_interval)`, where `step` is the query arg value
+passed to [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query) or [/api/v1/query](https://docs.victoriametrics.com/keyconcepts/#instant-query),
+while `scrape_interval` is the interval between [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) for the selected time series.
+This allows avoiding unexpected gaps on the graph when `step` is smaller than the `scrape_interval`.
+
 #### delta
 
 `delta(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the difference between
@@ -310,7 +343,9 @@ See [this article](https://medium.com/@romanhavronenko/victoriametrics-promql-co
 
 Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-This function is supported by PromQL. See also [increase](#increase) and [delta_prometheus](#delta_prometheus).
+This function is supported by PromQL.
+
+See also [increase](#increase) and [delta_prometheus](#delta_prometheus).
 
 #### delta_prometheus
 
@@ -333,7 +368,9 @@ The derivative is calculated using linear regression.
 
 Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-This function is supported by PromQL. See also [deriv_fast](#deriv_fast) and [ideriv](#ideriv).
+This function is supported by PromQL.
+
+See also [deriv_fast](#deriv_fast) and [ideriv](#ideriv).
 
 #### deriv_fast
 
@@ -364,6 +401,8 @@ on the given lookbehind window `d` per each time series returned from the given
 
 Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
+See also [count_values_over_time](#count_values_over_time).
+
 #### duration_over_time
 
 `duration_over_time(series_selector[d], max_interval)` is a [rollup function](#rollup-functions), which returns the duration in seconds
@@ -423,7 +462,9 @@ over the given lookbehind window `d` using the given smoothing factor `sf` and t
 Both `sf` and `tf` must be in the range `[0...1]`. It is expected that the [series_selector](https://docs.victoriametrics.com/keyConcepts.html#filtering)
 returns time series of [gauge type](https://docs.victoriametrics.com/keyConcepts.html#gauge).
 
-This function is supported by PromQL. See also [range_linear_regression](#range_linear_regression).
+This function is supported by PromQL.
+
+See also [range_linear_regression](#range_linear_regression).
 
 #### idelta
 
@@ -432,7 +473,9 @@ on the given lookbehind window `d` per each time series returned from the given
 
 Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-This function is supported by PromQL. See also [delta](#delta).
+This function is supported by PromQL.
+
+See also [delta](#delta).
 
 #### ideriv
 
@@ -455,7 +498,9 @@ See [this article](https://medium.com/@romanhavronenko/victoriametrics-promql-co
 
 Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-This function is supported by PromQL. See also [increase_pure](#increase_pure), [increase_prometheus](#increase_prometheus) and [delta](#delta).
+This function is supported by PromQL.
+
+See also [increase_pure](#increase_pure), [increase_prometheus](#increase_prometheus) and [delta](#delta).
 
 #### increase_prometheus
 
@@ -499,7 +544,9 @@ It is expected that the `series_selector` returns time series of [counter type](
 
 Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-This function is supported by PromQL. See also [rate](#rate) and [rollup_rate](#rollup_rate).
+This function is supported by PromQL.
+
+See also [rate](#rate) and [rollup_rate](#rollup_rate).
 
 #### lag
 
@@ -516,7 +563,9 @@ See also [lifetime](#lifetime) and [duration_over_time](#duration_over_time).
 `last_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which returns the last raw sample value on the given lookbehind window `d`
 per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyConcepts.html#filtering).
 
-This function is supported by PromQL. See also [first_over_time](#first_over_time) and [tlast_over_time](#tlast_over_time).
+This function is supported by PromQL.
+
+See also [first_over_time](#first_over_time) and [tlast_over_time](#tlast_over_time).
 
 #### lifetime
 
@@ -539,7 +588,9 @@ See also [mad](#mad), [range_mad](#range_mad) and [outlier_iqr_over_time](#outli
 `max_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the maximum value over raw samples
 on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyConcepts.html#filtering).
 
-This function is supported by PromQL. See also [tmax_over_time](#tmax_over_time).
+This function is supported by PromQL.
+
+See also [tmax_over_time](#tmax_over_time).
 
 #### median_over_time
 
@@ -554,7 +605,9 @@ See also [avg_over_time](#avg_over_time).
 `min_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the minimum value over raw samples
 on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyConcepts.html#filtering).
 
-This function is supported by PromQL. See also [tmin_over_time](#tmin_over_time).
+This function is supported by PromQL.
+
+See also [tmin_over_time](#tmin_over_time).
 
 #### mode_over_time
 
@@ -570,7 +623,7 @@ if its value is either smaller than the `q25-1.5*iqr` or bigger than `q75+1.5*iq
 - `q25` and `q75` are 25th and 75th [percentiles](https://en.wikipedia.org/wiki/Percentile) over raw samples on the lookbehind window `d`.
 
 The `outlier_iqr_over_time()` is useful for detecting anomalies in gauge values based on the previous history of values.
-For example, `outlier_iqr_over_time(memory_usage_bytes[1h])` triggers when `memory_usage_bytes` suddenly goes outside the usual value range for the last 24 hours.
+For example, `outlier_iqr_over_time(memory_usage_bytes[1h])` triggers when `memory_usage_bytes` suddenly goes outside the usual value range for the last hour.
 
 See also [outliers_iqr](#outliers_iqr).
 
@@ -580,7 +633,9 @@ See also [outliers_iqr](#outliers_iqr).
 linear interpolation over raw samples on the given lookbehind window `d`. The predicted value is calculated individually per each time series
 returned from the given [series_selector](https://docs.victoriametrics.com/keyConcepts.html#filtering).
 
-This function is supported by PromQL. See also [range_linear_regression](#range_linear_regression).
+This function is supported by PromQL.
+
+See also [range_linear_regression](#range_linear_regression).
 
 #### present_over_time
 
@@ -597,7 +652,9 @@ This function is supported by PromQL.
 on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyConcepts.html#filtering).
 The `phi` value must be in the range `[0...1]`.
 
-This function is supported by PromQL. See also [quantiles_over_time](#quantiles_over_time).
+This function is supported by PromQL.
+
+See also [quantiles_over_time](#quantiles_over_time).
 
 #### quantiles_over_time
 
@@ -622,9 +679,16 @@ Metric names are stripped from the resulting rollups. Add [keep_metric_names](#k
 over the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyConcepts.html#filtering).
 It is expected that the `series_selector` returns time series of [counter type](https://docs.victoriametrics.com/keyConcepts.html#counter).
 
+If the lookbehind window is skipped in square brackets, then it is automatically calculated as `max(step, scrape_interval)`, where `step` is the query arg value
+passed to [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query) or [/api/v1/query](https://docs.victoriametrics.com/keyconcepts/#instant-query),
+while `scrape_interval` is the interval between [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) for the selected time series.
+This allows avoiding unexpected gaps on the graph when `step` is smaller than the `scrape_interval`.
+
 Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-This function is supported by PromQL. See also [irate](#irate) and [rollup_rate](#rollup_rate).
+This function is supported by PromQL.
+
+See also [irate](#irate) and [rollup_rate](#rollup_rate).
 
 #### rate_over_sum
 
@@ -652,6 +716,7 @@ on the given lookbehind window `d` and returns them in time series with `rollup=
 These values are calculated individually per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyConcepts.html#filtering).
 
 Optional 2nd argument `"min"`, `"max"` or `"avg"` can be passed to keep only one calculation result and without adding a label.
+See also [label_match](#label_match).
 
 #### rollup_candlestick
 
@@ -660,7 +725,8 @@ over raw samples on the given lookbehind window `d` and returns them in time ser
 The calculations are performed individually per each time series returned
 from the given [series_selector](https://docs.victoriametrics.com/keyConcepts.html#filtering). This function is useful for financial applications.
 
-Optional 2nd argument `"min"`, `"max"` or `"avg"` can be passed to keep only one calculation result and without adding a label.
+Optional 2nd argument `"open"`, `"high"` or `"low"` or `"close"` can be passed to keep only one calculation result and without adding a label.
+See also [label_match](#label_match).
 
 #### rollup_delta
 
@@ -670,6 +736,7 @@ and returns them in time series with `rollup="min"`, `rollup="max"` and `rollup=
 The calculations are performed individually per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyConcepts.html#filtering).
 
 Optional 2nd argument `"min"`, `"max"` or `"avg"` can be passed to keep only one calculation result and without adding a label.
+See also [label_match](#label_match).
 
 Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
@@ -683,6 +750,7 @@ and returns them in time series with `rollup="min"`, `rollup="max"` and `rollup=
 The calculations are performed individually per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyConcepts.html#filtering).
 
 Optional 2nd argument `"min"`, `"max"` or `"avg"` can be passed to keep only one calculation result and without adding a label.
+See also [label_match](#label_match).
 
 Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
@@ -694,6 +762,7 @@ and returns them in time series with `rollup="min"`, `rollup="max"` and `rollup=
 The calculations are performed individually per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyConcepts.html#filtering).
 
 Optional 2nd argument `"min"`, `"max"` or `"avg"` can be passed to keep only one calculation result and without adding a label.
+See also [label_match](#label_match).
 
 Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. See also [rollup_delta](#rollup_delta).
 
@@ -707,10 +776,10 @@ See [this article](https://valyala.medium.com/why-irate-from-prometheus-doesnt-c
 when to use `rollup_rate()`.
 
 Optional 2nd argument `"min"`, `"max"` or `"avg"` can be passed to keep only one calculation result and without adding a label.
+See also [label_match](#label_match).
 
 The calculations are performed individually per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyConcepts.html#filtering).
 
-
 Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
 #### rollup_scrape_interval
@@ -721,6 +790,7 @@ and returns them in time series with `rollup="min"`, `rollup="max"` and `rollup=
 The calculations are performed individually per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyConcepts.html#filtering).
 
 Optional 2nd argument `"min"`, `"max"` or `"avg"` can be passed to keep only one calculation result and without adding a label.
+See also [label_match](#label_match).
 
 Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. See also [scrape_interval](#scrape_interval).
 
@@ -783,7 +853,9 @@ on the given lookbehind window `d` per each time series returned from the given
 
 Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-This function is supported by PromQL. See also [stdvar_over_time](#stdvar_over_time).
+This function is supported by PromQL.
+
+See also [stdvar_over_time](#stdvar_over_time).
 
 #### stdvar_over_time
 
@@ -792,7 +864,9 @@ on the given lookbehind window `d` per each time series returned from the given
 
 Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-This function is supported by PromQL. See also [stddev_over_time](#stddev_over_time).
+This function is supported by PromQL.
+
+See also [stddev_over_time](#stddev_over_time).
 
 #### sum_eq_over_time
 
@@ -844,7 +918,9 @@ on the given lookbehind window `d` per each time series returned from the given
 
 Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-This function is supported by PromQL. See also [timestamp_with_name](#timestamp_with_name).
+This function is supported by PromQL.
+
+See also [time](#time) and [now](#now).
 
 #### timestamp_with_name
 
@@ -853,7 +929,7 @@ on the given lookbehind window `d` per each time series returned from the given
 
 Metric names are preserved in the resulting rollups.
 
-See also [timestamp](#timestamp).
+See also [timestamp](#timestamp) and [keep_metric_names](#keep_metric_names) modifier.
 
 #### tfirst_over_time
 
@@ -920,7 +996,7 @@ Additional details:
 
 * If transform function is applied directly to a [series selector](https://docs.victoriametrics.com/keyConcepts.html#filtering),
   then the [default_rollup()](#default_rollup) function is automatically applied before calculating the transformations.
-  For example, `abs(temperature)` is implicitly transformed to `abs(default_rollup(temperature[1i]))`.
+  For example, `abs(temperature)` is implicitly transformed to `abs(default_rollup(temperature))`.
 * All the transform functions accept optional `keep_metric_names` modifier. If it is set,
   then the function doesn't drop metric names from the resulting time series. See [these docs](#keep_metric_names).
 
@@ -938,7 +1014,9 @@ This function is supported by PromQL.
 
 `absent(q)` is a [transform function](#transform-functions), which returns 1 if `q` has no points. Otherwise, returns an empty result.
 
-This function is supported by PromQL. See also [absent_over_time](#absent_over_time).
+This function is supported by PromQL.
+
+See also [absent_over_time](#absent_over_time).
 
 #### acos
 
@@ -947,7 +1025,9 @@ for every point of every time series returned by `q`.
 
 Metric names are stripped from the resulting series. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-This function is supported by PromQL. See also [asin](#asin) and [cos](#cos).
+This function is supported by PromQL.
+
+See also [asin](#asin) and [cos](#cos).
 
 #### acosh
 
@@ -956,7 +1036,9 @@ This function is supported by PromQL. See also [asin](#asin) and [cos](#cos).
 
 Metric names are stripped from the resulting series. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-This function is supported by PromQL. See also [sinh](#cosh).
+This function is supported by PromQL.
+
+See also [sinh](#cosh).
 
 #### asin
 
@@ -965,7 +1047,9 @@ for every point of every time series returned by `q`.
 
 Metric names are stripped from the resulting series. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-This function is supported by PromQL. See also [acos](#acos) and [sin](#sin).
+This function is supported by PromQL.
+
+See also [acos](#acos) and [sin](#sin).
 
 #### asinh
 
@@ -974,7 +1058,9 @@ This function is supported by PromQL. See also [acos](#acos) and [sin](#sin).
 
 Metric names are stripped from the resulting series. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-This function is supported by PromQL. See also [sinh](#sinh).
+This function is supported by PromQL.
+
+See also [sinh](#sinh).
 
 #### atan
 
@@ -983,7 +1069,9 @@ for every point of every time series returned by `q`.
 
 Metric names are stripped from the resulting series. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-This function is supported by PromQL. See also [tan](#tan).
+This function is supported by PromQL.
+
+See also [tan](#tan).
 
 #### atanh
 
@@ -992,7 +1080,9 @@ This function is supported by PromQL. See also [tan](#tan).
 
 Metric names are stripped from the resulting series. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-This function is supported by PromQL. See also [tanh](#tanh).
+This function is supported by PromQL.
+
+See also [tanh](#tanh).
 
 #### bitmap_and
 
@@ -1023,25 +1113,33 @@ See also [prometheus_buckets](#prometheus_buckets) and [histogram_quantile](#his
 
 `ceil(q)` is a [transform function](#transform-functions), which rounds every point for every time series returned by `q` to the upper nearest integer.
 
-This function is supported by PromQL. See also [floor](#floor) and [round](#round).
+This function is supported by PromQL.
+
+See also [floor](#floor) and [round](#round).
 
 #### clamp
 
 `clamp(q, min, max)` is a [transform function](#transform-functions), which clamps every point for every time series returned by `q` with the given `min` and `max` values.
 
-This function is supported by PromQL. See also [clamp_min](#clamp_min) and [clamp_max](#clamp_max).
+This function is supported by PromQL.
+
+See also [clamp_min](#clamp_min) and [clamp_max](#clamp_max).
 
 #### clamp_max
 
 `clamp_max(q, max)` is a [transform function](#transform-functions), which clamps every point for every time series returned by `q` with the given `max` value.
 
-This function is supported by PromQL. See also [clamp](#clamp) and [clamp_min](#clamp_min).
+This function is supported by PromQL.
+
+See also [clamp](#clamp) and [clamp_min](#clamp_min).
 
 #### clamp_min
 
 `clamp_min(q, min)` is a [transform function](#transform-functions), which clamps every point for every time series returned by `q` with the given `min` value.
 
-This function is supported by PromQL. See also [clamp](#clamp) and [clamp_max](#clamp_max).
+This function is supported by PromQL.
+
+See also [clamp](#clamp) and [clamp_max](#clamp_max).
 
 #### cos
 
@@ -1049,7 +1147,9 @@ This function is supported by PromQL. See also [clamp](#clamp) and [clamp_max](#
 
 Metric names are stripped from the resulting series. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-This function is supported by PromQL. See also [sin](#sin).
+This function is supported by PromQL.
+
+See also [sin](#sin).
 
 #### cosh
 
@@ -1058,7 +1158,9 @@ for every point of every time series returned by `q`.
 
 Metric names are stripped from the resulting series. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-This function is supported by PromQL. See also [acosh](#acosh).
+This function is supported by PromQL.
+
+See also [acosh](#acosh).
 
 #### day_of_month
 
@@ -1069,6 +1171,8 @@ Metric names are stripped from the resulting series. Add [keep_metric_names](#ke
 
 This function is supported by PromQL.
 
+See also [day_of_week](#day_of_week) and [day_of_year](#day_of_year).
+
 #### day_of_week
 
 `day_of_week(q)` is a [transform function](#transform-functions), which returns the day of week for every point of every time series returned by `q`.
@@ -1078,6 +1182,8 @@ Metric names are stripped from the resulting series. Add [keep_metric_names](#ke
 
 This function is supported by PromQL.
 
+See also [day_of_month](#day_of_month) and [day_of_year](#day_of_year).
+
 #### day_of_year
 
 `day_of_year(q)` is a [transform function](#transform-functions), which returns the day of year for every point of every time series returned by `q`.
@@ -1087,6 +1193,8 @@ Metric names are stripped from the resulting series. Add [keep_metric_names](#ke
 
 This function is supported by PromQL.
 
+See also [day_of_week](#day_of_week) and [day_of_month](#day_of_month).
+
 #### days_in_month
 
 `days_in_month(q)` is a [transform function](#transform-functions), which returns the number of days in the month identified
@@ -1104,7 +1212,9 @@ for every point of every time series returned by `q`.
 
 Metric names are stripped from the resulting series. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-This function is supported by PromQL. See also [rad](#rad).
+This function is supported by PromQL.
+
+See also [rad](#rad).
 
 #### drop_empty_series
 
@@ -1130,13 +1240,17 @@ See also [start](#start), [time](#time) and [now](#now).
 
 Metric names are stripped from the resulting series. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-This function is supported by PromQL. See also [ln](#ln).
+This function is supported by PromQL.
+
+See also [ln](#ln).
 
 #### floor
 
 `floor(q)` is a [transform function](#transform-functions), which rounds every point for every time series returned by `q` to the lower nearest integer.
 
-This function is supported by PromQL. See also [ceil](#ceil) and [round](#round).
+This function is supported by PromQL.
+
+See also [ceil](#ceil) and [round](#round).
 
 #### histogram_avg
 
@@ -1159,8 +1273,9 @@ When the [percentile](https://en.wikipedia.org/wiki/Percentile) is calculated ov
 then all the input histograms **must** have buckets with identical boundaries, e.g. they must have the same set of `le` or `vmrange` labels.
 Otherwise, the returned result may be invalid. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3231) for details.
 
-This function is supported by PromQL (except of the `boundLabel` arg). See also [histogram_quantiles](#histogram_quantiles), [histogram_share](#histogram_share)
-and [quantile](#quantile).
+This function is supported by PromQL (except of the `boundLabel` arg).
+
+See also [histogram_quantiles](#histogram_quantiles), [histogram_share](#histogram_share) and [quantile](#quantile).
 
 #### histogram_quantiles
 
@@ -1232,7 +1347,9 @@ This allows implementing simple paging for `q` time series. See also [limitk](#l
 
 Metric names are stripped from the resulting series. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-This function is supported by PromQL. See also [exp](#exp) and [log2](#log2).
+This function is supported by PromQL.
+
+See also [exp](#exp) and [log2](#log2).
 
 #### log2
 
@@ -1240,7 +1357,9 @@ This function is supported by PromQL. See also [exp](#exp) and [log2](#log2).
 
 Metric names are stripped from the resulting series. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-This function is supported by PromQL. See also [log10](#log10) and [ln](#ln).
+This function is supported by PromQL.
+
+See also [log10](#log10) and [ln](#ln).
 
 #### log10
 
@@ -1248,7 +1367,9 @@ This function is supported by PromQL. See also [log10](#log10) and [ln](#ln).
 
 Metric names are stripped from the resulting series. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-This function is supported by PromQL. See also [log2](#log2) and [ln](#ln).
+This function is supported by PromQL.
+
+See also [log2](#log2) and [ln](#ln).
 
 #### minute
 
@@ -1287,7 +1408,9 @@ for every point of every time series returned by `q`.
 
 Metric names are stripped from the resulting series. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-This function is supported by PromQL. See also [deg](#deg).
+This function is supported by PromQL.
+
+See also [deg](#deg).
 
 #### prometheus_buckets
 
@@ -1415,7 +1538,9 @@ for points returned by `q`, e.g. it is equivalent to the following query: `(q -
 `round(q, nearest)` is a [transform function](#transform-functions), which rounds every point of every time series returned by `q` to the `nearest` multiple.
 If `nearest` is missing then the rounding is performed to the nearest integer.
 
-This function is supported by PromQL. See also [floor](#floor) and [ceil](#ceil).
+This function is supported by PromQL.
+
+See also [floor](#floor) and [ceil](#ceil).
 
 #### ru
 
@@ -1459,7 +1584,9 @@ This function is supported by PromQL.
 
 Metric names are stripped from the resulting series. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-This function is supported by MetricsQL. See also [cos](#cos).
+This function is supported by MetricsQL.
+
+See also [cos](#cos).
 
 #### sinh
 
@@ -1468,7 +1595,9 @@ for every point of every time series returned by `q`.
 
 Metric names are stripped from the resulting series. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-This function is supported by MetricsQL. See also [cosh](#cosh).
+This function is supported by MetricsQL.
+
+See also [cosh](#cosh).
 
 #### tan
 
@@ -1476,7 +1605,9 @@ This function is supported by MetricsQL. See also [cosh](#cosh).
 
 Metric names are stripped from the resulting series. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-This function is supported by MetricsQL. See also [atan](#atan).
+This function is supported by MetricsQL.
+
+See also [atan](#atan).
 
 #### tanh
 
@@ -1485,7 +1616,9 @@ for every point of every time series returned by `q`.
 
 Metric names are stripped from the resulting series. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
 
-This function is supported by MetricsQL. See also [atanh](#atanh).
+This function is supported by MetricsQL.
+
+See also [atanh](#atanh).
 
 #### smooth_exponential
 
@@ -1496,13 +1629,17 @@ by `q` using [exponential moving average](https://en.wikipedia.org/wiki/Moving_a
 
 `sort(q)` is a [transform function](#transform-functions), which sorts series in ascending order by the last point in every time series returned by `q`.
 
-This function is supported by PromQL. See also [sort_desc](#sort_desc) and [sort_by_label](#sort_by_label).
+This function is supported by PromQL.
+
+See also [sort_desc](#sort_desc) and [sort_by_label](#sort_by_label).
 
 #### sort_desc
 
 `sort_desc(q)` is a [transform function](#transform-functions), which sorts series in descending order by the last point in every time series returned by `q`.
 
-This function is supported by PromQL. See also [sort](#sort) and [sort_by_label](#sort_by_label_desc).
+This function is supported by PromQL.
+
+See also [sort](#sort) and [sort_by_label](#sort_by_label_desc).
 
 #### sqrt
 
@@ -1531,7 +1668,9 @@ See also [start](#start) and [end](#end).
 
 `time()` is a [transform function](#transform-functions), which returns unix timestamp for every returned point.
 
-This function is supported by PromQL. See also [now](#now), [start](#start) and [end](#end).
+This function is supported by PromQL.
+
+See also [timestamp](#timestamp), [now](#now), [start](#start) and [end](#end).
 
 #### timezone_offset
 
@@ -1580,7 +1719,7 @@ Additional details:
 
 * If label manipulation function is applied directly to a [series_selector](https://docs.victoriametrics.com/keyConcepts.html#filtering),
   then the [default_rollup()](#default_rollup) function is automatically applied before performing the label transformation.
-  For example, `alias(temperature, "foo")` is implicitly transformed to `alias(default_rollup(temperature[1i]), "foo")`.
+  For example, `alias(temperature, "foo")` is implicitly transformed to `alias(default_rollup(temperature), "foo")`.
 
 See also [implicit query conversions](#implicit-query-conversions).
 
@@ -1757,7 +1896,7 @@ Additional details:
   Multiple labels can be put in `by` and `without` modifiers.
 * If the aggregate function is applied directly to a [series_selector](https://docs.victoriametrics.com/keyConcepts.html#filtering),
   then the [default_rollup()](#default_rollup) function is automatically applied before calculating the aggregate.
-  For example, `count(up)` is implicitly transformed to `count(default_rollup(up[1i]))`.
+  For example, `count(up)` is implicitly transformed to `count(default_rollup(up))`.
 * Aggregate functions accept arbitrary number of args. For example, `avg(q1, q2, q3)` would return the average values for every point
   across time series returned by `q1`, `q2` and `q3`.
 * Aggregate functions support optional `limit N` suffix, which can be used for limiting the number of output groups.
@@ -1785,7 +1924,9 @@ This function is supported by PromQL.
 `bottomk(k, q)` is [aggregate function](#aggregate-functions), which returns up to `k` points with the smallest values across all the time series returned by `q`.
 The aggregate is calculated individually per each group of points with the same timestamp.
 
-This function is supported by PromQL. See also [topk](#topk).
+This function is supported by PromQL.
+
+See also [topk](#topk), [bottomk_min](#bottomk_min) and [#bottomk_last](#bottomk_last).
 
 #### bottomk_avg
 
@@ -1847,10 +1988,14 @@ The aggregate is calculated individually per each group of points with the same
 
 This function is supported by PromQL.
 
+See also [count_values_over_time](#count_values_over_time) and [label_match](#label_match).
+
 #### distinct
 
 `distinct(q)` is [aggregate function](#aggregate-functions), which calculates the number of unique values per each group of points with the same timestamp.
 
+See also [distinct_over_time](#distinct_over_time).
+
 #### geomean
 
 `geomean(q)` is [aggregate function](#aggregate-functions), which calculates geometric mean per each group of points with the same timestamp.
@@ -1942,7 +2087,9 @@ See also [outliers_iqr](#outliers_iqr) and [outliers_mad](#outliers_mad).
 for all the time series returned by `q`. `phi` must be in the range `[0...1]`.
 The aggregate is calculated individually per each group of points with the same timestamp.
 
-This function is supported by PromQL. See also [quantiles](#quantiles) and [histogram_quantile](#histogram_quantile).
+This function is supported by PromQL.
+
+See also [quantiles](#quantiles) and [histogram_quantile](#histogram_quantile).
 
 #### quantiles
 
@@ -2001,7 +2148,9 @@ for all the time series returned by `q`. The aggregate is calculated individuall
 `topk(k, q)` is [aggregate function](#aggregate-functions), which returns up to `k` points with the biggest values across all the time series returned by `q`.
 The aggregate is calculated individually per each group of points with the same timestamp.
 
-This function is supported by PromQL. See also [bottomk](#bottomk).
+This function is supported by PromQL.
+
+See also [bottomk](#bottomk), [topk_max](#topk_max) and [topk_last](#topk_last).
 
 #### topk_avg
 
@@ -2061,7 +2210,7 @@ See also [zscore_over_time](#zscore_over_time), [range_trim_zscore](#range_trim_
 MetricsQL supports and extends PromQL subqueries. See [this article](https://valyala.medium.com/prometheus-subqueries-in-victoriametrics-9b1492b720b3) for details.
 Any [rollup function](#rollup-functions) for something other than [series selector](https://docs.victoriametrics.com/keyConcepts.html#filtering) form a subquery.
 Nested rollup functions can be implicit thanks to the [implicit query conversions](#implicit-query-conversions).
-For example, `delta(sum(m))` is implicitly converted to `delta(sum(default_rollup(m[1i]))[1i:1i])`, so it becomes a subquery,
+For example, `delta(sum(m))` is implicitly converted to `delta(sum(default_rollup(m))[1i:1i])`, so it becomes a subquery,
 since it contains [default_rollup](#default_rollup) nested into [delta](#delta).
 
 VictoriaMetrics performs subqueries in the following way:
@@ -2076,21 +2225,23 @@ VictoriaMetrics performs subqueries in the following way:
 
 VictoriaMetrics performs the following implicit conversions for incoming queries before starting the calculations:
 
-* If lookbehind window in square brackets is missing inside [rollup function](#rollup-functions),
-  then `[1i]` is automatically added there. The `[1i]` means one `step` value, which is passed
-  to [/api/v1/query_range](https://docs.victoriametrics.com/keyConcepts.html#range-query).
-  It is also known as `$__interval` in Grafana. For example, `rate(http_requests_count)` is automatically transformed to `rate(http_requests_count[1i])`.
+* If lookbehind window in square brackets is missing inside [rollup function](#rollup-functions), then it is automatically set to the following value:
+  - To `step` value passed to [/api/v1/query_range](https://docs.victoriametrics.com/keyConcepts.html#range-query) or [/api/v1/query](https://docs.victoriametrics.com/keyconcepts/#instant-query)
+    for all the [rollup functions](#rollup-functions) except of [default_rollup](#default_rollup) and [rate](#rate). This value is known as `$__interval` in Grafana or `1i` in MetricsQL.
+    For example, `avg_over_time(temperature)` is automatically transformed to `avg_over_time(temperature[1i])`.
+  - To the `max(step, scrape_interval)`, where `scrape_interval` is the interval between [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
+    for [default_rollup](#default_rollup) and [rate](#rate) functions. This allows avoiding unexpected gaps on the graph when `step` is smaller than `scrape_interval`.
 * All the [series selectors](https://docs.victoriametrics.com/keyConcepts.html#filtering),
   which aren't wrapped into [rollup functions](#rollup-functions), are automatically wrapped into [default_rollup](#default_rollup) function.
   Examples:
-  * `foo` is transformed to `default_rollup(foo[1i])`
-  * `foo + bar` is transformed to `default_rollup(foo[1i]) + default_rollup(bar[1i])`
-  * `count(up)` is transformed to `count(default_rollup(up[1i]))`, because [count](#count) isn't a [rollup function](#rollup-functions) -
+  * `foo` is transformed to `default_rollup(foo)`
+  * `foo + bar` is transformed to `default_rollup(foo) + default_rollup(bar)`
+  * `count(up)` is transformed to `count(default_rollup(up))`, because [count](#count) isn't a [rollup function](#rollup-functions) -
     it is [aggregate function](#aggregate-functions)
-  * `abs(temperature)` is transformed to `abs(default_rollup(temperature[1i]))`, because [abs](#abs) isn't a [rollup function](#rollup-functions) -
+  * `abs(temperature)` is transformed to `abs(default_rollup(temperature))`, because [abs](#abs) isn't a [rollup function](#rollup-functions) -
     it is [transform function](#transform-functions)
 * If `step` in square brackets is missing inside [subquery](#subqueries), then `1i` step is automatically added there.
   For example, `avg_over_time(rate(http_requests_total[5m])[1h])` is automatically converted to `avg_over_time(rate(http_requests_total[5m])[1h:1i])`.
 * If something other than [series selector](https://docs.victoriametrics.com/keyConcepts.html#filtering)
   is passed to [rollup function](#rollup-functions), then a [subquery](#subqueries) with `1i` lookbehind window and `1i` step is automatically formed.
-  For example, `rate(sum(up))` is automatically converted to `rate((sum(default_rollup(up[1i])))[1i:1i])`.
+  For example, `rate(sum(up))` is automatically converted to `rate((sum(default_rollup(up)))[1i:1i])`.

app/vmagent/main.go

@@ -8,7 +8,6 @@ import (
 	"net/http"
 	"os"
 	"strings"
-	"sync/atomic"
 	"time"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmagent/csvimport"
@@ -41,6 +40,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/procutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/common"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/opentelemetry/firehose"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/pushmetrics"
 	"github.com/VictoriaMetrics/metrics"
 )
@@ -126,6 +126,7 @@ func main() {
 	}
 	logger.Infof("starting vmagent at %q...", listenAddrs)
 	startTime := time.Now()
+	remotewrite.StartIngestionRateLimiter()
 	remotewrite.Init()
 	common.StartUnmarshalWorkers()
 	if len(*influxListenAddr) > 0 {
@@ -153,6 +154,7 @@ func main() {
 	pushmetrics.Init()
 	sig := procutil.WaitForSigterm()
 	logger.Infof("received signal %s", sig)
+	remotewrite.StopIngestionRateLimiter()
 	pushmetrics.Stop()
 
 	startTime = time.Now()
@@ -262,7 +264,7 @@ func requestHandler(w http.ResponseWriter, r *http.Request) bool {
 		path = strings.TrimSuffix(path, "/")
 	}
 	switch path {
-	case "/prometheus/api/v1/write", "/api/v1/write":
+	case "/prometheus/api/v1/write", "/api/v1/write", "/api/v1/push", "/prometheus/api/v1/push":
 		if common.HandleVMProtoServerHandshake(w, r) {
 			return true
 		}
@@ -314,14 +316,14 @@ func requestHandler(w http.ResponseWriter, r *http.Request) bool {
 		influxQueryRequests.Inc()
 		influxutils.WriteDatabaseNames(w)
 		return true
-	case "/opentelemetry/api/v1/push":
+	case "/opentelemetry/api/v1/push", "/opentelemetry/v1/metrics":
 		opentelemetryPushRequests.Inc()
 		if err := opentelemetry.InsertHandler(nil, r); err != nil {
 			opentelemetryPushErrors.Inc()
 			httpserver.Errorf(w, r, "%s", err)
 			return true
 		}
-		w.WriteHeader(http.StatusOK)
+		firehose.WriteSuccessResponse(w, r)
 		return true
 	case "/newrelic":
 		newrelicCheckRequest.Inc()
@@ -459,7 +461,7 @@ func requestHandler(w http.ResponseWriter, r *http.Request) bool {
 		w.WriteHeader(http.StatusOK)
 		return true
 	case "/ready":
-		if rdy := atomic.LoadInt32(&promscrape.PendingScrapeConfigs); rdy > 0 {
+		if rdy := promscrape.PendingScrapeConfigs.Load(); rdy > 0 {
 			errMsg := fmt.Sprintf("waiting for scrapes to init, left: %d", rdy)
 			http.Error(w, errMsg, http.StatusTooEarly)
 		} else {
@@ -511,7 +513,7 @@ func processMultitenantRequest(w http.ResponseWriter, r *http.Request, path stri
 		p.Suffix = strings.TrimSuffix(p.Suffix, "/")
 	}
 	switch p.Suffix {
-	case "prometheus/", "prometheus", "prometheus/api/v1/write":
+	case "prometheus/", "prometheus", "prometheus/api/v1/write", "prometheus/api/v1/push":
 		prometheusWriteRequests.Inc()
 		if err := promremotewrite.InsertHandler(at, r); err != nil {
 			prometheusWriteErrors.Inc()
@@ -560,14 +562,14 @@ func processMultitenantRequest(w http.ResponseWriter, r *http.Request, path stri
 		influxQueryRequests.Inc()
 		influxutils.WriteDatabaseNames(w)
 		return true
-	case "opentelemetry/api/v1/push":
+	case "opentelemetry/api/v1/push", "opentelemetry/v1/metrics":
 		opentelemetryPushRequests.Inc()
 		if err := opentelemetry.InsertHandler(at, r); err != nil {
 			opentelemetryPushErrors.Inc()
 			httpserver.Errorf(w, r, "%s", err)
 			return true
 		}
-		w.WriteHeader(http.StatusOK)
+		firehose.WriteSuccessResponse(w, r)
 		return true
 	case "newrelic":
 		newrelicCheckRequest.Inc()
@@ -686,8 +688,8 @@ var (
 	datadogIntakeRequests   = metrics.NewCounter(`vmagent_http_requests_total{path="/datadog/intake", protocol="datadog"}`)
 	datadogMetadataRequests = metrics.NewCounter(`vmagent_http_requests_total{path="/datadog/api/v1/metadata", protocol="datadog"}`)
 
-	opentelemetryPushRequests = metrics.NewCounter(`vmagent_http_requests_total{path="/opentelemetry/api/v1/push", protocol="opentelemetry"}`)
-	opentelemetryPushErrors   = metrics.NewCounter(`vmagent_http_request_errors_total{path="/opentelemetry/api/v1/push", protocol="opentelemetry"}`)
+	opentelemetryPushRequests = metrics.NewCounter(`vmagent_http_requests_total{path="/opentelemetry/v1/metrics", protocol="opentelemetry"}`)
+	opentelemetryPushErrors   = metrics.NewCounter(`vmagent_http_request_errors_total{path="/opentelemetry/v1/metrics", protocol="opentelemetry"}`)
 
 	newrelicWriteRequests = metrics.NewCounter(`vm_http_requests_total{path="/newrelic/infra/v2/metrics/events/bulk", protocol="newrelic"}`)
 	newrelicWriteErrors   = metrics.NewCounter(`vm_http_request_errors_total{path="/newrelic/infra/v2/metrics/events/bulk", protocol="newrelic"}`)

app/vmagent/opentelemetry/request_handler.go

@@ -9,6 +9,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/auth"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
 	parserCommon "github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/common"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/opentelemetry/firehose"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/opentelemetry/stream"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/tenantmetrics"
 	"github.com/VictoriaMetrics/metrics"
@@ -27,10 +28,15 @@ func InsertHandler(at *auth.Token, req *http.Request) error {
 		return err
 	}
 	isGzipped := req.Header.Get("Content-Encoding") == "gzip"
+	var processBody func([]byte) ([]byte, error)
 	if req.Header.Get("Content-Type") == "application/json" {
-		return fmt.Errorf("json encoding isn't supported for opentelemetry format. Use protobuf encoding")
+		if req.Header.Get("X-Amz-Firehose-Protocol-Version") != "" {
+			processBody = firehose.ProcessRequestBody
+		} else {
+			return fmt.Errorf("json encoding isn't supported for opentelemetry format. Use protobuf encoding")
+		}
 	}
-	return stream.ParseStream(req.Body, isGzipped, func(tss []prompbmarshal.TimeSeries) error {
+	return stream.ParseStream(req.Body, isGzipped, processBody, func(tss []prompbmarshal.TimeSeries) error {
 		return insertRows(at, tss, extraLabels)
 	})
 }

app/vmagent/remotewrite/client.go

@@ -17,6 +17,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/persistentqueue"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promauth"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/common"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/ratelimiter"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/timerpool"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/timeutil"
 	"github.com/VictoriaMetrics/metrics"
@@ -30,7 +31,7 @@ var (
 
 	rateLimit = flagutil.NewArrayInt("remoteWrite.rateLimit", 0, "Optional rate limit in bytes per second for data sent to the corresponding -remoteWrite.url. "+
 		"By default, the rate limit is disabled. It can be useful for limiting load on remote storage when big amounts of buffered data "+
-		"is sent after temporary unavailability of the remote storage")
+		"is sent after temporary unavailability of the remote storage. See also -maxIngestionRate")
 	sendTimeout = flagutil.NewArrayDuration("remoteWrite.sendTimeout", time.Minute, "Timeout for sending a single block of data to the corresponding -remoteWrite.url")
 	proxyURL    = flagutil.NewArrayString("remoteWrite.proxyURL", "Optional proxy URL for writing data to the corresponding -remoteWrite.url. "+
 		"Supported proxies: http, https, socks5. Example: -remoteWrite.proxyURL=socks5://proxy:1234")
@@ -91,7 +92,7 @@ type client struct {
 	authCfg   *promauth.Config
 	awsCfg    *awsapi.Config
 
-	rl rateLimiter
+	rl *ratelimiter.RateLimiter
 
 	bytesSent       *metrics.Counter
 	blocksSent      *metrics.Counter
@@ -112,17 +113,12 @@ func newHTTPClient(argIdx int, remoteWriteURL, sanitizedURL string, fq *persiste
 	if err != nil {
 		logger.Fatalf("cannot initialize auth config for -remoteWrite.url=%q: %s", remoteWriteURL, err)
 	}
-	tlsCfg, err := authCfg.NewTLSConfig()
-	if err != nil {
-		logger.Fatalf("cannot initialize tls config for -remoteWrite.url=%q: %s", remoteWriteURL, err)
-	}
 	awsCfg, err := getAWSAPIConfig(argIdx)
 	if err != nil {
 		logger.Fatalf("cannot initialize AWS Config for -remoteWrite.url=%q: %s", remoteWriteURL, err)
 	}
 	tr := &http.Transport{
 		DialContext:         statDial,
-		TLSClientConfig:     tlsCfg,
 		TLSHandshakeTimeout: tlsHandshakeTimeout.GetOptionalArg(argIdx),
 		MaxConnsPerHost:     2 * concurrency,
 		MaxIdleConnsPerHost: 2 * concurrency,
@@ -141,7 +137,7 @@ func newHTTPClient(argIdx int, remoteWriteURL, sanitizedURL string, fq *persiste
 		tr.Proxy = http.ProxyURL(pu)
 	}
 	hc := &http.Client{
-		Transport: tr,
+		Transport: authCfg.NewRoundTripper(tr),
 		Timeout:   sendTimeout.GetOptionalArg(argIdx),
 	}
 	c := &client{
@@ -177,12 +173,11 @@ func newHTTPClient(argIdx int, remoteWriteURL, sanitizedURL string, fq *persiste
 }
 
 func (c *client) init(argIdx, concurrency int, sanitizedURL string) {
+	limitReached := metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_remotewrite_rate_limit_reached_total{url=%q}`, c.sanitizedURL))
 	if bytesPerSec := rateLimit.GetOptionalArg(argIdx); bytesPerSec > 0 {
 		logger.Infof("applying %d bytes per second rate limit for -remoteWrite.url=%q", bytesPerSec, sanitizedURL)
-		c.rl.perSecondLimit = int64(bytesPerSec)
+		c.rl = ratelimiter.New(int64(bytesPerSec), limitReached, c.stopCh)
 	}
-	c.rl.limitReached = metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_remotewrite_rate_limit_reached_total{url=%q}`, c.sanitizedURL))
-
 	c.bytesSent = metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_remotewrite_bytes_sent_total{url=%q}`, c.sanitizedURL))
 	c.blocksSent = metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_remotewrite_blocks_sent_total{url=%q}`, c.sanitizedURL))
 	c.rateLimit = metrics.GetOrCreateGauge(fmt.Sprintf(`vmagent_remotewrite_rate_limit{url=%q}`, c.sanitizedURL), func() float64 {
@@ -396,7 +391,7 @@ func (c *client) newRequest(url string, body []byte) (*http.Request, error) {
 // The function returns false only if c.stopCh is closed.
 // Otherwise it tries sending the block to remote storage indefinitely.
 func (c *client) sendBlockHTTP(block []byte) bool {
-	c.rl.register(len(block), c.stopCh)
+	c.rl.Register(len(block))
 	maxRetryDuration := timeutil.AddJitterToDuration(time.Minute)
 	retryDuration := timeutil.AddJitterToDuration(time.Second)
 	retriesCount := 0
@@ -479,45 +474,3 @@ again:
 }
 
 var remoteWriteRejectedLogger = logger.WithThrottler("remoteWriteRejected", 5*time.Second)
-
-type rateLimiter struct {
-	perSecondLimit int64
-
-	// mu protects budget and deadline from concurrent access.
-	mu sync.Mutex
-
-	// The current budget. It is increased by perSecondLimit every second.
-	budget int64
-
-	// The next deadline for increasing the budget by perSecondLimit
-	deadline time.Time
-
-	limitReached *metrics.Counter
-}
-
-func (rl *rateLimiter) register(dataLen int, stopCh <-chan struct{}) {
-	limit := rl.perSecondLimit
-	if limit <= 0 {
-		return
-	}
-
-	rl.mu.Lock()
-	defer rl.mu.Unlock()
-
-	for rl.budget <= 0 {
-		if d := time.Until(rl.deadline); d > 0 {
-			rl.limitReached.Inc()
-			t := timerpool.Get(d)
-			select {
-			case <-stopCh:
-				timerpool.Put(t)
-				return
-			case <-t.C:
-				timerpool.Put(t)
-			}
-		}
-		rl.budget += limit
-		rl.deadline = time.Now().Add(time.Second)
-	}
-	rl.budget -= int64(dataLen)
-}

app/vmagent/remotewrite/pendingseries.go

@@ -82,7 +82,7 @@ func (ps *pendingSeries) periodicFlusher() {
 			ps.mu.Unlock()
 			return
 		case <-ticker.C:
-			if fasttime.UnixTimestamp()-atomic.LoadUint64(&ps.wr.lastFlushTime) < uint64(flushSeconds) {
+			if fasttime.UnixTimestamp()-ps.wr.lastFlushTime.Load() < uint64(flushSeconds) {
 				continue
 			}
 		}
@@ -93,8 +93,7 @@ func (ps *pendingSeries) periodicFlusher() {
 }
 
 type writeRequest struct {
-	// Move lastFlushTime to the top of the struct in order to guarantee atomic access on 32-bit architectures.
-	lastFlushTime uint64
+	lastFlushTime atomic.Uint64
 
 	// The queue to send blocks to.
 	fq *persistentqueue.FastQueue
@@ -155,7 +154,7 @@ func (wr *writeRequest) mustWriteBlock(block []byte) bool {
 
 func (wr *writeRequest) tryFlush() bool {
 	wr.wr.Timeseries = wr.tss
-	atomic.StoreUint64(&wr.lastFlushTime, fasttime.UnixTimestamp())
+	wr.lastFlushTime.Store(fasttime.UnixTimestamp())
 	if !tryPushWriteRequest(&wr.wr, wr.fq.TryWriteBlock, wr.isVMRemoteWrite) {
 		return false
 	}

app/vmagent/remotewrite/remotewrite.go

@@ -27,6 +27,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promrelabel"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promutils"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/ratelimiter"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/streamaggr"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/tenantmetrics"
 	"github.com/VictoriaMetrics/metrics"
@@ -50,13 +51,17 @@ var (
 		"By default the data is replicated across all the -remoteWrite.url . See https://docs.victoriametrics.com/vmagent.html#sharding-among-remote-storages")
 	shardByURLLabels = flagutil.NewArrayString("remoteWrite.shardByURL.labels", "Optional list of labels, which must be used for sharding outgoing samples "+
 		"among remote storage systems if -remoteWrite.shardByURL command-line flag is set. By default all the labels are used for sharding in order to gain "+
-		"even distribution of series over the specified -remoteWrite.url systems")
+		"even distribution of series over the specified -remoteWrite.url systems. See also -remoteWrite.shardByURL.ignoreLabels")
+	shardByURLIgnoreLabels = flagutil.NewArrayString("remoteWrite.shardByURL.ignoreLabels", "Optional list of labels, which must be ignored when sharding outgoing samples "+
+		"among remote storage systems if -remoteWrite.shardByURL command-line flag is set. By default all the labels are used for sharding in order to gain "+
+		"even distribution of series over the specified -remoteWrite.url systems. See also -remoteWrite.shardByURL.labels")
 	tmpDataPath = flag.String("remoteWrite.tmpDataPath", "vmagent-remotewrite-data", "Path to directory for storing pending data, which isn't sent to the configured -remoteWrite.url . "+
 		"See also -remoteWrite.maxDiskUsagePerURL and -remoteWrite.disableOnDiskQueue")
 	keepDanglingQueues = flag.Bool("remoteWrite.keepDanglingQueues", false, "Keep persistent queues contents at -remoteWrite.tmpDataPath in case there are no matching -remoteWrite.url. "+
 		"Useful when -remoteWrite.url is changed temporarily and persistent queue files will be needed later on.")
 	queues = flag.Int("remoteWrite.queues", cgroup.AvailableCPUs()*2, "The number of concurrent queues to each -remoteWrite.url. Set more queues if default number of queues "+
-		"isn't enough for sending high volume of collected data to remote storage. Default value is 2 * numberOfAvailableCPUs")
+		"isn't enough for sending high volume of collected data to remote storage. "+
+		"Default value depends on the number of available CPU cores. It should work fine in most cases since it minimizes resource usage")
 	showRemoteWriteURL = flag.Bool("remoteWrite.showURL", false, "Whether to show -remoteWrite.url in the exported metrics. "+
 		"It is hidden by default, since it can contain sensitive info such as auth key")
 	maxPendingBytesPerURL = flagutil.NewArrayBytes("remoteWrite.maxDiskUsagePerURL", 0, "The maximum file-based buffer size in bytes at -remoteWrite.tmpDataPath "+
@@ -79,6 +84,8 @@ var (
 		"Excess series are logged and dropped. This can be useful for limiting series cardinality. See https://docs.victoriametrics.com/vmagent.html#cardinality-limiter")
 	maxDailySeries = flag.Int("remoteWrite.maxDailySeries", 0, "The maximum number of unique series vmagent can send to remote storage systems during the last 24 hours. "+
 		"Excess series are logged and dropped. This can be useful for limiting series churn rate. See https://docs.victoriametrics.com/vmagent.html#cardinality-limiter")
+	maxIngestionRate = flag.Int("maxIngestionRate", 0, "The maximum number of samples vmagent can receive per second. Data ingestion is paused when the limit is exceeded. "+
+		"By default there are no limits on samples ingestion rate. See also -remoteWrite.rateLimit")
 
 	streamAggrConfig = flagutil.NewArrayString("remoteWrite.streamAggr.config", "Optional path to file with stream aggregation config. "+
 		"See https://docs.victoriametrics.com/stream-aggregation.html . "+
@@ -89,8 +96,13 @@ var (
 	streamAggrDropInput = flagutil.NewArrayBool("remoteWrite.streamAggr.dropInput", "Whether to drop all the input samples after the aggregation "+
 		"with -remoteWrite.streamAggr.config. By default, only aggregates samples are dropped, while the remaining samples "+
 		"are written to the corresponding -remoteWrite.url . See also -remoteWrite.streamAggr.keepInput and https://docs.victoriametrics.com/stream-aggregation.html")
-	streamAggrDedupInterval = flagutil.NewArrayDuration("remoteWrite.streamAggr.dedupInterval", 0, "Input samples are de-duplicated with this interval before being aggregated. "+
-		"Only the last sample per each time series per each interval is aggregated if the interval is greater than zero")
+	streamAggrDedupInterval = flagutil.NewArrayDuration("remoteWrite.streamAggr.dedupInterval", 0, "Input samples are de-duplicated with this interval before optional aggregation "+
+		"with -remoteWrite.streamAggr.config . See also -dedup.minScrapeInterval and https://docs.victoriametrics.com/stream-aggregation.html#deduplication")
+	streamAggrIgnoreOldSamples = flagutil.NewArrayBool("remoteWrite.streamAggr.ignoreOldSamples", "Whether to ignore input samples with old timestamps outside the current aggregation interval "+
+		"for the corresponding -remoteWrite.streamAggr.config . See https://docs.victoriametrics.com/stream-aggregation.html#ignoring-old-samples")
+	streamAggrDropInputLabels = flagutil.NewArrayString("streamAggr.dropInputLabels", "An optional list of labels to drop from samples "+
+		"before stream de-duplication and aggregation . See https://docs.victoriametrics.com/stream-aggregation.html#dropping-unneeded-labels")
+
 	disableOnDiskQueue = flag.Bool("remoteWrite.disableOnDiskQueue", false, "Whether to disable storing pending data to -remoteWrite.tmpDataPath "+
 		"when the configured remote storage systems cannot keep up with the data ingestion rate. See https://docs.victoriametrics.com/vmagent.html#disabling-on-disk-persistence ."+
 		"See also -remoteWrite.dropSamplesOnOverload")
@@ -140,7 +152,10 @@ func InitSecretFlags() {
 	}
 }
 
-var shardByURLLabelsMap map[string]struct{}
+var (
+	shardByURLLabelsMap       map[string]struct{}
+	shardByURLIgnoreLabelsMap map[string]struct{}
+)
 
 // Init initializes remotewrite.
 //
@@ -172,19 +187,21 @@ func Init() {
 			return float64(dailySeriesLimiter.CurrentItems())
 		})
 	}
+
 	if *queues > maxQueues {
 		*queues = maxQueues
 	}
 	if *queues <= 0 {
 		*queues = 1
 	}
-	if len(*shardByURLLabels) > 0 {
-		m := make(map[string]struct{}, len(*shardByURLLabels))
-		for _, label := range *shardByURLLabels {
-			m[label] = struct{}{}
-		}
-		shardByURLLabelsMap = m
+
+	if len(*shardByURLLabels) > 0 && len(*shardByURLIgnoreLabels) > 0 {
+		logger.Fatalf("-remoteWrite.shardByURL.labels and -remoteWrite.shardByURL.ignoreLabels cannot be set simultaneously; " +
+			"see https://docs.victoriametrics.com/vmagent/#sharding-among-remote-storages")
 	}
+	shardByURLLabelsMap = newMapFromStrings(*shardByURLLabels)
+	shardByURLIgnoreLabelsMap = newMapFromStrings(*shardByURLIgnoreLabels)
+
 	initLabelsGlobal()
 
 	// Register SIGHUP handler for config reload before loadRelabelConfigs.
@@ -336,6 +353,35 @@ func newRemoteWriteCtxs(at *auth.Token, urls []string) []*remoteWriteCtx {
 var configReloaderStopCh = make(chan struct{})
 var configReloaderWG sync.WaitGroup
 
+// StartIngestionRateLimiter starts ingestion rate limiter.
+//
+// Ingestion rate limiter must be started before Init() call.
+//
+// StopIngestionRateLimiter must be called before Stop() call in order to unblock all the callers
+// to ingestion rate limiter. Otherwise deadlock may occur at Stop() call.
+func StartIngestionRateLimiter() {
+	if *maxIngestionRate <= 0 {
+		return
+	}
+	ingestionRateLimitReached := metrics.NewCounter(`vmagent_max_ingestion_rate_limit_reached_total`)
+	ingestionRateLimiterStopCh = make(chan struct{})
+	ingestionRateLimiter = ratelimiter.New(int64(*maxIngestionRate), ingestionRateLimitReached, ingestionRateLimiterStopCh)
+}
+
+// StopIngestionRateLimiter stops ingestion rate limiter.
+func StopIngestionRateLimiter() {
+	if ingestionRateLimiterStopCh == nil {
+		return
+	}
+	close(ingestionRateLimiterStopCh)
+	ingestionRateLimiterStopCh = nil
+}
+
+var (
+	ingestionRateLimiter       *ratelimiter.RateLimiter
+	ingestionRateLimiterStopCh chan struct{}
+)
+
 // Stop stops remotewrite.
 //
 // It is expected that nobody calls TryPush during and after the call to this func.
@@ -462,6 +508,9 @@ func tryPush(at *auth.Token, wr *prompbmarshal.WriteRequest, dropSamplesOnFailur
 				break
 			}
 		}
+
+		ingestionRateLimiter.Register(samplesCount)
+
 		tssBlock := tss
 		if i < len(tss) {
 			tssBlock = tss[:i]
@@ -526,6 +575,15 @@ func tryPushBlockToRemoteStorages(rwctxs []*remoteWriteCtx, tssBlock []prompbmar
 						hashLabels = append(hashLabels, label)
 					}
 				}
+				tmpLabels.Labels = hashLabels
+			} else if len(shardByURLIgnoreLabelsMap) > 0 {
+				hashLabels = tmpLabels.Labels[:0]
+				for _, label := range ts.Labels {
+					if _, ok := shardByURLIgnoreLabelsMap[label.Name]; !ok {
+						hashLabels = append(hashLabels, label)
+					}
+				}
+				tmpLabels.Labels = hashLabels
 			}
 			h := getLabelsHash(hashLabels)
 			idx := h % uint64(len(tssByURL))
@@ -536,22 +594,22 @@ func tryPushBlockToRemoteStorages(rwctxs []*remoteWriteCtx, tssBlock []prompbmar
 		// Push sharded data to remote storages in parallel in order to reduce
 		// the time needed for sending the data to multiple remote storage systems.
 		var wg sync.WaitGroup
-		wg.Add(len(rwctxs))
-		var anyPushFailed uint64
+		var anyPushFailed atomic.Bool
 		for i, rwctx := range rwctxs {
 			tssShard := tssByURL[i]
 			if len(tssShard) == 0 {
 				continue
 			}
+			wg.Add(1)
 			go func(rwctx *remoteWriteCtx, tss []prompbmarshal.TimeSeries) {
 				defer wg.Done()
 				if !rwctx.TryPush(tss) {
-					atomic.StoreUint64(&anyPushFailed, 1)
+					anyPushFailed.Store(true)
 				}
 			}(rwctx, tssShard)
 		}
 		wg.Wait()
-		return atomic.LoadUint64(&anyPushFailed) == 0
+		return !anyPushFailed.Load()
 	}
 
 	// Replicate data among rwctxs.
@@ -559,17 +617,17 @@ func tryPushBlockToRemoteStorages(rwctxs []*remoteWriteCtx, tssBlock []prompbmar
 	// the time needed for sending the data to multiple remote storage systems.
 	var wg sync.WaitGroup
 	wg.Add(len(rwctxs))
-	var anyPushFailed uint64
+	var anyPushFailed atomic.Bool
 	for _, rwctx := range rwctxs {
 		go func(rwctx *remoteWriteCtx) {
 			defer wg.Done()
 			if !rwctx.TryPush(tssBlock) {
-				atomic.StoreUint64(&anyPushFailed, 1)
+				anyPushFailed.Store(true)
 			}
 		}(rwctx)
 	}
 	wg.Wait()
-	return atomic.LoadUint64(&anyPushFailed) == 0
+	return !anyPushFailed.Load()
 }
 
 // sortLabelsIfNeeded sorts labels if -sortLabels command-line flag is set.
@@ -665,12 +723,14 @@ type remoteWriteCtx struct {
 	fq  *persistentqueue.FastQueue
 	c   *client
 
-	sas                 atomic.Pointer[streamaggr.Aggregators]
+	sas          atomic.Pointer[streamaggr.Aggregators]
+	deduplicator *streamaggr.Deduplicator
+
 	streamAggrKeepInput bool
 	streamAggrDropInput bool
 
 	pss        []*pendingSeries
-	pssNextIdx uint64
+	pssNextIdx atomic.Uint64
 
 	rowsPushedAfterRelabel *metrics.Counter
 	rowsDroppedByRelabel   *metrics.Counter
@@ -738,9 +798,15 @@ func newRemoteWriteCtx(argIdx int, remoteWriteURL *url.URL, maxInmemoryBlocks in
 
 	// Initialize sas
 	sasFile := streamAggrConfig.GetOptionalArg(argIdx)
+	dedupInterval := streamAggrDedupInterval.GetOptionalArg(argIdx)
+	ignoreOldSamples := streamAggrIgnoreOldSamples.GetOptionalArg(argIdx)
 	if sasFile != "" {
-		dedupInterval := streamAggrDedupInterval.GetOptionalArg(argIdx)
-		sas, err := streamaggr.LoadFromFile(sasFile, rwctx.pushInternalTrackDropped, dedupInterval)
+		opts := &streamaggr.Options{
+			DedupInterval:    dedupInterval,
+			DropInputLabels:  *streamAggrDropInputLabels,
+			IgnoreOldSamples: ignoreOldSamples,
+		}
+		sas, err := streamaggr.LoadFromFile(sasFile, rwctx.pushInternalTrackDropped, opts)
 		if err != nil {
 			logger.Fatalf("cannot initialize stream aggregators from -remoteWrite.streamAggr.config=%q: %s", sasFile, err)
 		}
@@ -749,17 +815,24 @@ func newRemoteWriteCtx(argIdx int, remoteWriteURL *url.URL, maxInmemoryBlocks in
 		rwctx.streamAggrDropInput = streamAggrDropInput.GetOptionalArg(argIdx)
 		metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_streamaggr_config_reload_successful{path=%q}`, sasFile)).Set(1)
 		metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_streamaggr_config_reload_success_timestamp_seconds{path=%q}`, sasFile)).Set(fasttime.UnixTimestamp())
+	} else if dedupInterval > 0 {
+		rwctx.deduplicator = streamaggr.NewDeduplicator(rwctx.pushInternalTrackDropped, dedupInterval, *streamAggrDropInputLabels)
 	}
 
 	return rwctx
 }
 
 func (rwctx *remoteWriteCtx) MustStop() {
-	// sas must be stopped before rwctx is closed
+	// sas and deduplicator must be stopped before rwctx is closed
 	// because sas can write pending series to rwctx.pss if there are any
 	sas := rwctx.sas.Swap(nil)
 	sas.MustStop()
 
+	if rwctx.deduplicator != nil {
+		rwctx.deduplicator.MustStop()
+		rwctx.deduplicator = nil
+	}
+
 	for _, ps := range rwctx.pss {
 		ps.MustStop()
 	}
@@ -798,7 +871,7 @@ func (rwctx *remoteWriteCtx) TryPush(tss []prompbmarshal.TimeSeries) bool {
 	rowsCount := getRowsCount(tss)
 	rwctx.rowsPushedAfterRelabel.Add(rowsCount)
 
-	// Apply stream aggregation if any
+	// Apply stream aggregation or deduplication if they are configured
 	sas := rwctx.sas.Load()
 	if sas != nil {
 		matchIdxs := matchIdxsPool.Get()
@@ -813,6 +886,10 @@ func (rwctx *remoteWriteCtx) TryPush(tss []prompbmarshal.TimeSeries) bool {
 			tss = dropAggregatedSeries(tss, matchIdxs.B, rwctx.streamAggrDropInput)
 		}
 		matchIdxsPool.Put(matchIdxs)
+	} else if rwctx.deduplicator != nil {
+		rwctx.deduplicator.Push(tss)
+		clear(tss)
+		tss = tss[:0]
 	}
 
 	// Try pushing the data to remote storage
@@ -841,7 +918,7 @@ func dropAggregatedSeries(src []prompbmarshal.TimeSeries, matchIdxs []byte, drop
 		}
 	}
 	tail := src[len(dst):]
-	_ = prompbmarshal.ResetTimeSeries(tail)
+	clear(tail)
 	return dst
 }
 
@@ -872,7 +949,7 @@ func (rwctx *remoteWriteCtx) tryPushInternal(tss []prompbmarshal.TimeSeries) boo
 	}
 
 	pss := rwctx.pss
-	idx := atomic.AddUint64(&rwctx.pssNextIdx, 1) % uint64(len(pss))
+	idx := rwctx.pssNextIdx.Add(1) % uint64(len(pss))
 
 	ok := pss[idx].TryPush(tss)
 
@@ -894,8 +971,12 @@ func (rwctx *remoteWriteCtx) reinitStreamAggr() {
 
 	logger.Infof("reloading stream aggregation configs pointed by -remoteWrite.streamAggr.config=%q", sasFile)
 	metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_streamaggr_config_reloads_total{path=%q}`, sasFile)).Inc()
-	dedupInterval := streamAggrDedupInterval.GetOptionalArg(rwctx.idx)
-	sasNew, err := streamaggr.LoadFromFile(sasFile, rwctx.pushInternalTrackDropped, dedupInterval)
+	opts := &streamaggr.Options{
+		DedupInterval:    streamAggrDedupInterval.GetOptionalArg(rwctx.idx),
+		DropInputLabels:  *streamAggrDropInputLabels,
+		IgnoreOldSamples: streamAggrIgnoreOldSamples.GetOptionalArg(rwctx.idx),
+	}
+	sasNew, err := streamaggr.LoadFromFile(sasFile, rwctx.pushInternalTrackDropped, opts)
 	if err != nil {
 		metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_streamaggr_config_reloads_errors_total{path=%q}`, sasFile)).Inc()
 		metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_streamaggr_config_reload_successful{path=%q}`, sasFile)).Set(0)
@@ -932,13 +1013,17 @@ func getRowsCount(tss []prompbmarshal.TimeSeries) int {
 
 // CheckStreamAggrConfigs checks configs pointed by -remoteWrite.streamAggr.config
 func CheckStreamAggrConfigs() error {
-	pushNoop := func(tss []prompbmarshal.TimeSeries) {}
+	pushNoop := func(_ []prompbmarshal.TimeSeries) {}
 	for idx, sasFile := range *streamAggrConfig {
 		if sasFile == "" {
 			continue
 		}
-		dedupInterval := streamAggrDedupInterval.GetOptionalArg(idx)
-		sas, err := streamaggr.LoadFromFile(sasFile, pushNoop, dedupInterval)
+		opts := &streamaggr.Options{
+			DedupInterval:    streamAggrDedupInterval.GetOptionalArg(idx),
+			DropInputLabels:  *streamAggrDropInputLabels,
+			IgnoreOldSamples: streamAggrIgnoreOldSamples.GetOptionalArg(idx),
+		}
+		sas, err := streamaggr.LoadFromFile(sasFile, pushNoop, opts)
 		if err != nil {
 			return fmt.Errorf("cannot load -remoteWrite.streamAggr.config=%q: %w", sasFile, err)
 		}
@@ -946,3 +1031,11 @@ func CheckStreamAggrConfigs() error {
 	}
 	return nil
 }
+
+func newMapFromStrings(a []string) map[string]struct{} {
+	m := make(map[string]struct{}, len(a))
+	for _, s := range a {
+		m[s] = struct{}{}
+	}
+	return m
+}

app/vmagent/remotewrite/statconn.go

@@ -50,7 +50,7 @@ var (
 )
 
 type statConn struct {
-	closed uint64
+	closed atomic.Int32
 	net.Conn
 }
 
@@ -76,7 +76,7 @@ func (sc *statConn) Write(p []byte) (int, error) {
 
 func (sc *statConn) Close() error {
 	err := sc.Conn.Close()
-	if atomic.AddUint64(&sc.closed, 1) == 1 {
+	if sc.closed.Add(1) == 1 {
 		conns.Dec()
 	}
 	return err

app/vmalert/config/testdata/rules/kube-good.rules

@@ -1158,9 +1158,9 @@
           $labels.pod }}.'
         runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-cputhrottlinghigh
       expr: |
-        sum(increase(container_cpu_cfs_throttled_periods_total{container!="", }[5m])) by (container, pod, namespace)
+        sum(increase(container_cpu_cfs_throttled_periods_total{container!="", }[5m])) by (cluster, container, pod, namespace)
           /
-        sum(increase(container_cpu_cfs_periods_total{}[5m])) by (container, pod, namespace)
+        sum(increase(container_cpu_cfs_periods_total{}[5m])) by (cluster, container, pod, namespace)
           > ( 25 / 100 )
       for: 15m
       labels:

app/vmalert/datasource/init.go

@@ -46,7 +46,7 @@ var (
 	oauth2TokenURL = flag.String("datasource.oauth2.tokenUrl", "", "Optional OAuth2 tokenURL to use for -datasource.url")
 	oauth2Scopes   = flag.String("datasource.oauth2.scopes", "", "Optional OAuth2 scopes to use for -datasource.url. Scopes must be delimited by ';'")
 
-	lookBack = flag.Duration("datasource.lookback", 0, `Will be deprecated soon, please adjust "-search.latencyOffset"  at datasource side `+
+	lookBack = flag.Duration("datasource.lookback", 0, `Deprecated: please adjust "-search.latencyOffset" at datasource side `+
 		`or specify "latency_offset" in rule group's params. Lookback defines how far into the past to look when evaluating queries. `+
 		`For example, if the datasource.lookback=5m then param "time" with value now()-5m will be added to every query.`)
 	queryStep = flag.Duration("datasource.queryStep", 5*time.Minute, "How far a value can fallback to when evaluating queries. "+
@@ -91,7 +91,7 @@ func Init(extraParams url.Values) (QuerierBuilder, error) {
 		logger.Warnf("flag `-datasource.queryTimeAlignment` is deprecated and will be removed in next releases. Please use `eval_alignment` in rule group instead.")
 	}
 	if *lookBack != 0 {
-		logger.Warnf("flag `-datasource.lookback` will be deprecated soon. Please use `-rule.evalDelay` command-line flag instead. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155 for details.")
+		logger.Warnf("flag `-datasource.lookback` is deprecated and will be removed in next releases. Please adjust `-search.latencyOffset` at datasource side or specify `latency_offset` in rule group's params. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155 for details.")
 	}
 
 	tr, err := httputils.Transport(*addr, *tlsCertFile, *tlsKeyFile, *tlsCAFile, *tlsServerName, *tlsInsecureSkipVerify)
@@ -133,7 +133,6 @@ func Init(extraParams url.Values) (QuerierBuilder, error) {
 		authCfg:          authCfg,
 		datasourceURL:    strings.TrimSuffix(*addr, "/"),
 		appendTypePrefix: *appendTypePrefix,
-		lookBack:         *lookBack,
 		queryStep:        *queryStep,
 		dataSourceType:   datasourcePrometheus,
 		extraParams:      extraParams,

app/vmalert/datasource/vm.go

@@ -35,7 +35,6 @@ type VMStorage struct {
 	authCfg          *promauth.Config
 	datasourceURL    string
 	appendTypePrefix bool
-	lookBack         time.Duration
 	queryStep        time.Duration
 	dataSourceType   datasourceType
 
@@ -63,7 +62,6 @@ func (s *VMStorage) Clone() *VMStorage {
 		authCfg:          s.authCfg,
 		datasourceURL:    s.datasourceURL,
 		appendTypePrefix: s.appendTypePrefix,
-		lookBack:         s.lookBack,
 		queryStep:        s.queryStep,
 
 		dataSourceType:     s.dataSourceType,
@@ -122,13 +120,12 @@ func (s *VMStorage) BuildWithParams(params QuerierParams) Querier {
 }
 
 // NewVMStorage is a constructor for VMStorage
-func NewVMStorage(baseURL string, authCfg *promauth.Config, lookBack time.Duration, queryStep time.Duration, appendTypePrefix bool, c *http.Client) *VMStorage {
+func NewVMStorage(baseURL string, authCfg *promauth.Config, queryStep time.Duration, appendTypePrefix bool, c *http.Client) *VMStorage {
 	return &VMStorage{
 		c:                c,
 		authCfg:          authCfg,
 		datasourceURL:    strings.TrimSuffix(baseURL, "/"),
 		appendTypePrefix: appendTypePrefix,
-		lookBack:         lookBack,
 		queryStep:        queryStep,
 		dataSourceType:   datasourcePrometheus,
 		extraParams:      url.Values{},
@@ -137,11 +134,11 @@ func NewVMStorage(baseURL string, authCfg *promauth.Config, lookBack time.Durati
 
 // Query executes the given query and returns parsed response
 func (s *VMStorage) Query(ctx context.Context, query string, ts time.Time) (Result, *http.Request, error) {
-	req, err := s.newQueryRequest(query, ts)
+	req, err := s.newQueryRequest(ctx, query, ts)
 	if err != nil {
 		return Result{}, nil, err
 	}
-	resp, err := s.do(ctx, req)
+	resp, err := s.do(req)
 	if err != nil {
 		if !errors.Is(err, io.EOF) && !errors.Is(err, io.ErrUnexpectedEOF) {
 			// Return unexpected error to the caller.
@@ -149,11 +146,11 @@ func (s *VMStorage) Query(ctx context.Context, query string, ts time.Time) (Resu
 		}
 		// Something in the middle between client and datasource might be closing
 		// the connection. So we do a one more attempt in hope request will succeed.
-		req, err = s.newQueryRequest(query, ts)
+		req, err = s.newQueryRequest(ctx, query, ts)
 		if err != nil {
 			return Result{}, nil, fmt.Errorf("second attempt: %w", err)
 		}
-		resp, err = s.do(ctx, req)
+		resp, err = s.do(req)
 		if err != nil {
 			return Result{}, nil, fmt.Errorf("second attempt: %w", err)
 		}
@@ -182,11 +179,11 @@ func (s *VMStorage) QueryRange(ctx context.Context, query string, start, end tim
 	if end.IsZero() {
 		return res, fmt.Errorf("end param is missing")
 	}
-	req, err := s.newQueryRangeRequest(query, start, end)
+	req, err := s.newQueryRangeRequest(ctx, query, start, end)
 	if err != nil {
 		return res, err
 	}
-	resp, err := s.do(ctx, req)
+	resp, err := s.do(req)
 	if err != nil {
 		if !errors.Is(err, io.EOF) && !errors.Is(err, io.ErrUnexpectedEOF) {
 			// Return unexpected error to the caller.
@@ -194,11 +191,11 @@ func (s *VMStorage) QueryRange(ctx context.Context, query string, start, end tim
 		}
 		// Something in the middle between client and datasource might be closing
 		// the connection. So we do a one more attempt in hope request will succeed.
-		req, err = s.newQueryRangeRequest(query, start, end)
+		req, err = s.newQueryRangeRequest(ctx, query, start, end)
 		if err != nil {
 			return res, fmt.Errorf("second attempt: %w", err)
 		}
-		resp, err = s.do(ctx, req)
+		resp, err = s.do(req)
 		if err != nil {
 			return res, fmt.Errorf("second attempt: %w", err)
 		}
@@ -210,7 +207,7 @@ func (s *VMStorage) QueryRange(ctx context.Context, query string, start, end tim
 	return res, err
 }
 
-func (s *VMStorage) do(ctx context.Context, req *http.Request) (*http.Response, error) {
+func (s *VMStorage) do(req *http.Request) (*http.Response, error) {
 	ru := req.URL.Redacted()
 	if *showDatasourceURL {
 		ru = req.URL.String()
@@ -218,7 +215,7 @@ func (s *VMStorage) do(ctx context.Context, req *http.Request) (*http.Response,
 	if s.debug {
 		logger.Infof("DEBUG datasource request: executing %s request with params %q", req.Method, ru)
 	}
-	resp, err := s.c.Do(req.WithContext(ctx))
+	resp, err := s.c.Do(req)
 	if err != nil {
 		return nil, fmt.Errorf("error getting response from %s: %w", ru, err)
 	}
@@ -230,8 +227,8 @@ func (s *VMStorage) do(ctx context.Context, req *http.Request) (*http.Response,
 	return resp, nil
 }
 
-func (s *VMStorage) newQueryRangeRequest(query string, start, end time.Time) (*http.Request, error) {
-	req, err := s.newRequest()
+func (s *VMStorage) newQueryRangeRequest(ctx context.Context, query string, start, end time.Time) (*http.Request, error) {
+	req, err := s.newRequest(ctx)
 	if err != nil {
 		return nil, fmt.Errorf("cannot create query_range request to datasource %q: %w", s.datasourceURL, err)
 	}
@@ -239,8 +236,8 @@ func (s *VMStorage) newQueryRangeRequest(query string, start, end time.Time) (*h
 	return req, nil
 }
 
-func (s *VMStorage) newQueryRequest(query string, ts time.Time) (*http.Request, error) {
-	req, err := s.newRequest()
+func (s *VMStorage) newQueryRequest(ctx context.Context, query string, ts time.Time) (*http.Request, error) {
+	req, err := s.newRequest(ctx)
 	if err != nil {
 		return nil, fmt.Errorf("cannot create query request to datasource %q: %w", s.datasourceURL, err)
 	}
@@ -248,15 +245,15 @@ func (s *VMStorage) newQueryRequest(query string, ts time.Time) (*http.Request,
 	case "", datasourcePrometheus:
 		s.setPrometheusInstantReqParams(req, query, ts)
 	case datasourceGraphite:
-		s.setGraphiteReqParams(req, query, ts)
+		s.setGraphiteReqParams(req, query)
 	default:
 		logger.Panicf("BUG: engine not found: %q", s.dataSourceType)
 	}
 	return req, nil
 }
 
-func (s *VMStorage) newRequest() (*http.Request, error) {
-	req, err := http.NewRequest(http.MethodPost, s.datasourceURL, nil)
+func (s *VMStorage) newRequest(ctx context.Context) (*http.Request, error) {
+	req, err := http.NewRequestWithContext(ctx, http.MethodPost, s.datasourceURL, nil)
 	if err != nil {
 		logger.Panicf("BUG: unexpected error from http.NewRequest(%q): %s", s.datasourceURL, err)
 	}

app/vmalert/datasource/vm_graphite_api.go

@@ -4,8 +4,6 @@ import (
 	"encoding/json"
 	"fmt"
 	"net/http"
-	"strconv"
-	"time"
 )
 
 type graphiteResponse []graphiteResponseTarget
@@ -48,17 +46,13 @@ const (
 	graphitePrefix = "/graphite"
 )
 
-func (s *VMStorage) setGraphiteReqParams(r *http.Request, query string, timestamp time.Time) {
+func (s *VMStorage) setGraphiteReqParams(r *http.Request, query string) {
 	if s.appendTypePrefix {
 		r.URL.Path += graphitePrefix
 	}
 	r.URL.Path += graphitePath
 	q := r.URL.Query()
 	from := "-5min"
-	if s.lookBack > 0 {
-		lookBack := timestamp.Add(-s.lookBack)
-		from = strconv.FormatInt(lookBack.Unix(), 10)
-	}
 	q.Set("from", from)
 	q.Set("format", "json")
 	q.Set("target", query)

app/vmalert/datasource/vm_prom_api.go

@@ -161,9 +161,6 @@ func (s *VMStorage) setPrometheusInstantReqParams(r *http.Request, query string,
 		r.URL.Path += "/api/v1/query"
 	}
 	q := r.URL.Query()
-	if s.lookBack > 0 {
-		timestamp = timestamp.Add(-s.lookBack)
-	}
 	q.Set("time", timestamp.Format(time.RFC3339))
 	if !*disableStepParam && s.evaluationInterval > 0 { // set step as evaluationInterval by default
 		// always convert to seconds to keep compatibility with older

app/vmalert/datasource/vm_test.go

@@ -71,7 +71,7 @@ func TestVMInstantQuery(t *testing.T) {
 			w.Write([]byte(`{"status":"success","data":{"resultType":"scalar","result":[1583786142, "1"]},"stats":{"seriesFetched": "42"}}`))
 		}
 	})
-	mux.HandleFunc("/render", func(w http.ResponseWriter, request *http.Request) {
+	mux.HandleFunc("/render", func(w http.ResponseWriter, _ *http.Request) {
 		c++
 		switch c {
 		case 8:
@@ -86,7 +86,7 @@ func TestVMInstantQuery(t *testing.T) {
 	if err != nil {
 		t.Fatalf("unexpected: %s", err)
 	}
-	s := NewVMStorage(srv.URL, authCfg, time.Minute, 0, false, srv.Client())
+	s := NewVMStorage(srv.URL, authCfg, 0, false, srv.Client())
 
 	p := datasourcePrometheus
 	pq := s.BuildWithParams(QuerierParams{DataSourceType: string(p), EvaluationInterval: 15 * time.Second})
@@ -225,7 +225,7 @@ func TestVMInstantQueryWithRetry(t *testing.T) {
 	srv := httptest.NewServer(mux)
 	defer srv.Close()
 
-	s := NewVMStorage(srv.URL, nil, time.Minute, 0, false, srv.Client())
+	s := NewVMStorage(srv.URL, nil, 0, false, srv.Client())
 	pq := s.BuildWithParams(QuerierParams{DataSourceType: string(datasourcePrometheus)})
 
 	expErr := func(err string) {
@@ -334,7 +334,7 @@ func TestVMRangeQuery(t *testing.T) {
 	if err != nil {
 		t.Fatalf("unexpected: %s", err)
 	}
-	s := NewVMStorage(srv.URL, authCfg, time.Minute, *queryStep, false, srv.Client())
+	s := NewVMStorage(srv.URL, authCfg, *queryStep, false, srv.Client())
 
 	pq := s.BuildWithParams(QuerierParams{DataSourceType: string(datasourcePrometheus), EvaluationInterval: 15 * time.Second})
 
@@ -487,17 +487,6 @@ func TestRequestParams(t *testing.T) {
 				checkEqualString(t, "bar", p)
 			},
 		},
-		{
-			"lookback",
-			false,
-			&VMStorage{
-				lookBack: time.Minute,
-			},
-			func(t *testing.T, r *http.Request) {
-				exp := url.Values{"query": {query}, "time": {timestamp.Add(-time.Minute).Format(time.RFC3339)}}
-				checkEqualString(t, exp.Encode(), r.URL.RawQuery)
-			},
-		},
 		{
 			"evaluation interval",
 			false,
@@ -510,20 +499,6 @@ func TestRequestParams(t *testing.T) {
 				checkEqualString(t, exp.Encode(), r.URL.RawQuery)
 			},
 		},
-		{
-			"lookback + evaluation interval",
-			false,
-			&VMStorage{
-				lookBack:           time.Minute,
-				evaluationInterval: 15 * time.Second,
-			},
-			func(t *testing.T, r *http.Request) {
-				evalInterval := 15 * time.Second
-				tt := timestamp.Add(-time.Minute)
-				exp := url.Values{"query": {query}, "step": {evalInterval.String()}, "time": {tt.Format(time.RFC3339)}}
-				checkEqualString(t, exp.Encode(), r.URL.RawQuery)
-			},
-		},
 		{
 			"step override",
 			false,
@@ -637,7 +612,7 @@ func TestRequestParams(t *testing.T) {
 
 	for _, tc := range testCases {
 		t.Run(tc.name, func(t *testing.T) {
-			req, err := tc.vm.newRequest()
+			req, err := tc.vm.newRequest(ctx)
 			if err != nil {
 				t.Fatal(err)
 			}
@@ -649,7 +624,7 @@ func TestRequestParams(t *testing.T) {
 					tc.vm.setPrometheusInstantReqParams(req, query, timestamp)
 				}
 			case datasourceGraphite:
-				tc.vm.setGraphiteReqParams(req, query, timestamp)
+				tc.vm.setGraphiteReqParams(req, query)
 			}
 			tc.checkFn(t, req)
 		})
@@ -735,7 +710,7 @@ func TestHeaders(t *testing.T) {
 	for _, tt := range testCases {
 		t.Run(tt.name, func(t *testing.T) {
 			vm := tt.vmFn()
-			req, err := vm.newQueryRequest("foo", time.Now())
+			req, err := vm.newQueryRequest(ctx, "foo", time.Now())
 			if err != nil {
 				t.Fatal(err)
 			}

app/vmalert/main.go

@@ -304,7 +304,7 @@ func getAlertURLGenerator(externalURL *url.URL, externalAlertSource string, vali
 		"tpl": externalAlertSource,
 	}
 	return func(alert notifier.Alert) string {
-		qFn := func(query string) ([]datasource.Metric, error) {
+		qFn := func(_ string) ([]datasource.Metric, error) {
 			return nil, fmt.Errorf("`query` template isn't supported for alert source template")
 		}
 		templated, err := alert.ExecTemplate(qFn, alert.Labels, m)

app/vmalert/notifier/alert_test.go

@@ -178,7 +178,7 @@ func TestAlert_ExecTemplate(t *testing.T) {
 		},
 	}
 
-	qFn := func(q string) ([]datasource.Metric, error) {
+	qFn := func(_ string) ([]datasource.Metric, error) {
 		return []datasource.Metric{
 			{
 				Labels: []datasource.Label{

app/vmalert/notifier/alertmanager.go

@@ -105,7 +105,7 @@ func (am *AlertManager) send(ctx context.Context, alerts []Alert, headers map[st
 	if *showNotifierURL {
 		amURL = am.addr.String()
 	}
-	if resp.StatusCode != http.StatusOK {
+	if resp.StatusCode/100 != 2 {
 		body, err := io.ReadAll(resp.Body)
 		if err != nil {
 			return fmt.Errorf("failed to read response from %q: %w", amURL, err)

app/vmalert/remoteread/init.go

@@ -79,5 +79,5 @@ func Init() (datasource.QuerierBuilder, error) {
 		return nil, fmt.Errorf("failed to configure auth: %w", err)
 	}
 	c := &http.Client{Transport: tr}
-	return datasource.NewVMStorage(*addr, authCfg, 0, 0, false, c), nil
+	return datasource.NewVMStorage(*addr, authCfg, 0, false, c), nil
 }

app/vmalert/remotewrite/client.go

@@ -151,12 +151,22 @@ func (c *Client) run(ctx context.Context) {
 	ticker := time.NewTicker(c.flushInterval)
 	wr := &prompbmarshal.WriteRequest{}
 	shutdown := func() {
+		lastCtx, cancel := context.WithTimeout(context.Background(), defaultWriteTimeout)
+		logger.Infof("shutting down remote write client and flushing remained series")
+
+		shutdownFlushCnt := 0
 		for ts := range c.input {
 			wr.Timeseries = append(wr.Timeseries, ts)
+			if len(wr.Timeseries) >= c.maxBatchSize {
+				shutdownFlushCnt += len(wr.Timeseries)
+				c.flush(lastCtx, wr)
+			}
 		}
-		lastCtx, cancel := context.WithTimeout(context.Background(), defaultWriteTimeout)
-		logger.Infof("shutting down remote write client and flushing remained %d series", len(wr.Timeseries))
+		// flush the last batch. `flush` will re-check and avoid flushing empty batch.
+		shutdownFlushCnt += len(wr.Timeseries)
 		c.flush(lastCtx, wr)
+
+		logger.Infof("shutting down remote write client flushed %d series", shutdownFlushCnt)
 		cancel()
 	}
 	c.wg.Add(1)
@@ -279,7 +289,7 @@ L:
 
 func (c *Client) send(ctx context.Context, data []byte) error {
 	r := bytes.NewReader(data)
-	req, err := http.NewRequest(http.MethodPost, c.addr, r)
+	req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.addr, r)
 	if err != nil {
 		return fmt.Errorf("failed to create new HTTP request: %w", err)
 	}
@@ -302,7 +312,7 @@ func (c *Client) send(ctx context.Context, data []byte) error {
 	if !*disablePathAppend {
 		req.URL.Path = path.Join(req.URL.Path, "/api/v1/write")
 	}
-	resp, err := c.c.Do(req.WithContext(ctx))
+	resp, err := c.c.Do(req)
 	if err != nil {
 		return fmt.Errorf("error while sending request to %s: %w; Data len %d(%d)",
 			req.URL.Redacted(), err, len(data), r.Size())

app/vmalert/remotewrite/client_test.go

@@ -84,6 +84,70 @@ func TestClient_Push(t *testing.T) {
 	}
 }
 
+func TestClient_run_maxBatchSizeDuringShutdown(t *testing.T) {
+	batchSize := 20
+
+	testTable := []struct {
+		name     string // name of the test case
+		pushCnt  int    // how many time series is pushed to the client
+		batchCnt int    // the expected batch count sent by the client
+	}{
+		{
+			name:     "pushCnt % batchSize == 0",
+			pushCnt:  batchSize * 40,
+			batchCnt: 40,
+		},
+		{
+			name:     "pushCnt % batchSize != 0",
+			pushCnt:  batchSize*40 + 1,
+			batchCnt: 40 + 1,
+		},
+	}
+
+	for _, tt := range testTable {
+		t.Run(tt.name, func(t *testing.T) {
+			// run new server
+			bcServer := newBatchCntRWServer()
+
+			// run new client
+			rwClient, err := NewClient(context.Background(), Config{
+				MaxBatchSize: batchSize,
+
+				// Set everything to 1 to simplify the calculation.
+				Concurrency:   1,
+				MaxQueueSize:  1000,
+				FlushInterval: time.Minute,
+
+				// batch count server
+				Addr: bcServer.URL,
+			})
+			if err != nil {
+				t.Fatalf("new remote write client failed, err: %v", err)
+			}
+
+			// push time series to the client.
+			for i := 0; i < tt.pushCnt; i++ {
+				if err = rwClient.Push(prompbmarshal.TimeSeries{}); err != nil {
+					t.Fatalf("push time series to the client failed, err: %v", err)
+				}
+			}
+
+			// close the client so the rest ts will be flushed in `shutdown`
+			if err = rwClient.Close(); err != nil {
+				t.Fatalf("shutdown client failed, err: %v", err)
+			}
+
+			// finally check how many batches is sent.
+			if tt.batchCnt != bcServer.acceptedBatches() {
+				t.Errorf("client sent batch count incorrect, want: %d, get: %d", tt.batchCnt, bcServer.acceptedBatches())
+			}
+			if tt.pushCnt != bcServer.accepted() {
+				t.Errorf("client sent time series count incorrect, want: %d, get: %d", tt.pushCnt, bcServer.accepted())
+			}
+		})
+	}
+}
+
 func newRWServer() *rwServer {
 	rw := &rwServer{}
 	rw.Server = httptest.NewServer(http.HandlerFunc(rw.handler))
@@ -91,14 +155,12 @@ func newRWServer() *rwServer {
 }
 
 type rwServer struct {
-	// WARN: ordering of fields is important for alignment!
-	// see https://golang.org/pkg/sync/atomic/#pkg-note-BUG
-	acceptedRows uint64
+	acceptedRows atomic.Uint64
 	*httptest.Server
 }
 
 func (rw *rwServer) accepted() int {
-	return int(atomic.LoadUint64(&rw.acceptedRows))
+	return int(rw.acceptedRows.Load())
 }
 
 func (rw *rwServer) err(w http.ResponseWriter, err error) {
@@ -144,7 +206,7 @@ func (rw *rwServer) handler(w http.ResponseWriter, r *http.Request) {
 		rw.err(w, fmt.Errorf("unmarhsal err: %w", err))
 		return
 	}
-	atomic.AddUint64(&rw.acceptedRows, uint64(len(wr.Timeseries)))
+	rw.acceptedRows.Add(uint64(len(wr.Timeseries)))
 	w.WriteHeader(http.StatusNoContent)
 }
 
@@ -186,3 +248,27 @@ func (frw *faultyRWServer) handler(w http.ResponseWriter, r *http.Request) {
 		w.Write([]byte("server overloaded"))
 	}
 }
+
+type batchCntRWServer struct {
+	*rwServer
+
+	batchCnt atomic.Int64 // accepted batch count, which also equals to request count
+}
+
+func newBatchCntRWServer() *batchCntRWServer {
+	bc := &batchCntRWServer{
+		rwServer: &rwServer{},
+	}
+
+	bc.Server = httptest.NewServer(http.HandlerFunc(bc.handler))
+	return bc
+}
+
+func (bc *batchCntRWServer) handler(w http.ResponseWriter, r *http.Request) {
+	bc.batchCnt.Add(1)
+	bc.rwServer.handler(w, r)
+}
+
+func (bc *batchCntRWServer) acceptedBatches() int {
+	return int(bc.batchCnt.Load())
+}

app/vmalert/rule/alerting.go

@@ -310,7 +310,7 @@ func (ar *AlertingRule) execRange(ctx context.Context, start, end time.Time) ([]
 	}
 	var result []prompbmarshal.TimeSeries
 	holdAlertState := make(map[uint64]*notifier.Alert)
-	qFn := func(query string) ([]datasource.Metric, error) {
+	qFn := func(_ string) ([]datasource.Metric, error) {
 		return nil, fmt.Errorf("`query` template isn't supported in replay mode")
 	}
 	for _, s := range res.Data {
@@ -655,15 +655,19 @@ func (ar *AlertingRule) restore(ctx context.Context, q datasource.Querier, ts ti
 // alertsToSend walks through the current alerts of AlertingRule
 // and returns only those which should be sent to notifier.
 // Isn't concurrent safe.
-func (ar *AlertingRule) alertsToSend(ts time.Time, resolveDuration, resendDelay time.Duration) []notifier.Alert {
+func (ar *AlertingRule) alertsToSend(resolveDuration, resendDelay time.Duration) []notifier.Alert {
+	currentTime := time.Now()
 	needsSending := func(a *notifier.Alert) bool {
 		if a.State == notifier.StatePending {
 			return false
 		}
-		if a.ResolvedAt.After(a.LastSent) {
+		if a.State == notifier.StateFiring && a.End.Before(a.LastSent) {
 			return true
 		}
-		return a.LastSent.Add(resendDelay).Before(ts)
+		if a.State == notifier.StateInactive && a.ResolvedAt.After(a.LastSent) {
+			return true
+		}
+		return a.LastSent.Add(resendDelay).Before(currentTime)
 	}
 
 	var alerts []notifier.Alert
@@ -671,11 +675,11 @@ func (ar *AlertingRule) alertsToSend(ts time.Time, resolveDuration, resendDelay
 		if !needsSending(a) {
 			continue
 		}
-		a.End = ts.Add(resolveDuration)
+		a.End = currentTime.Add(resolveDuration)
 		if a.State == notifier.StateInactive {
 			a.End = a.ResolvedAt
 		}
-		a.LastSent = ts
+		a.LastSent = currentTime
 		alerts = append(alerts, *a)
 	}
 	return alerts

app/vmalert/rule/alerting_test.go

@@ -43,11 +43,13 @@ func TestAlertingRule_ToTimeSeries(t *testing.T) {
 		},
 		{
 			newTestAlertingRule("instant extra labels", 0),
-			&notifier.Alert{State: notifier.StateFiring, ActiveAt: timestamp.Add(time.Second),
+			&notifier.Alert{
+				State: notifier.StateFiring, ActiveAt: timestamp.Add(time.Second),
 				Labels: map[string]string{
 					"job":      "foo",
 					"instance": "bar",
-				}},
+				},
+			},
 			[]prompbmarshal.TimeSeries{
 				newTimeSeries([]float64{1}, []int64{timestamp.UnixNano()}, map[string]string{
 					"__name__":      alertMetricName,
@@ -66,11 +68,13 @@ func TestAlertingRule_ToTimeSeries(t *testing.T) {
 		},
 		{
 			newTestAlertingRule("instant labels override", 0),
-			&notifier.Alert{State: notifier.StateFiring, ActiveAt: timestamp.Add(time.Second),
+			&notifier.Alert{
+				State: notifier.StateFiring, ActiveAt: timestamp.Add(time.Second),
 				Labels: map[string]string{
 					alertStateLabel: "foo",
 					"__name__":      "bar",
-				}},
+				},
+			},
 			[]prompbmarshal.TimeSeries{
 				newTimeSeries([]float64{1}, []int64{timestamp.UnixNano()}, map[string]string{
 					"__name__":      alertMetricName,
@@ -572,25 +576,33 @@ func TestAlertingRule_ExecRange(t *testing.T) {
 				},
 			},
 			[]*notifier.Alert{
-				{State: notifier.StateFiring, ActiveAt: time.Unix(1, 0),
+				{
+					State: notifier.StateFiring, ActiveAt: time.Unix(1, 0),
 					Labels: map[string]string{
 						"source": "vm",
-					}},
-				{State: notifier.StateFiring, ActiveAt: time.Unix(100, 0),
+					},
+				},
+				{
+					State: notifier.StateFiring, ActiveAt: time.Unix(100, 0),
 					Labels: map[string]string{
 						"source": "vm",
-					}},
+					},
+				},
 				//
-				{State: notifier.StateFiring, ActiveAt: time.Unix(1, 0),
+				{
+					State: notifier.StateFiring, ActiveAt: time.Unix(1, 0),
 					Labels: map[string]string{
 						"foo":    "bar",
 						"source": "vm",
-					}},
-				{State: notifier.StateFiring, ActiveAt: time.Unix(5, 0),
+					},
+				},
+				{
+					State: notifier.StateFiring, ActiveAt: time.Unix(5, 0),
 					Labels: map[string]string{
 						"foo":    "bar",
 						"source": "vm",
-					}},
+					},
+				},
 			},
 			nil,
 		},
@@ -1042,7 +1054,7 @@ func TestAlertsToSend(t *testing.T) {
 		for i, a := range alerts {
 			ar.alerts[uint64(i)] = a
 		}
-		gotAlerts := ar.alertsToSend(ts, resolveDuration, resendDelay)
+		gotAlerts := ar.alertsToSend(resolveDuration, resendDelay)
 		if gotAlerts == nil && expAlerts == nil {
 			return
 		}
@@ -1058,60 +1070,36 @@ func TestAlertsToSend(t *testing.T) {
 		})
 		for i, exp := range expAlerts {
 			got := gotAlerts[i]
-			if got.LastSent != exp.LastSent {
-				t.Fatalf("expected LastSent to be %v; got %v", exp.LastSent, got.LastSent)
-			}
-			if got.End != exp.End {
-				t.Fatalf("expected End to be %v; got %v", exp.End, got.End)
+			if got.Name != exp.Name {
+				t.Fatalf("expected Name to be %v; got %v", exp.Name, got.Name)
 			}
 		}
 	}
 
-	f( // send firing alert with custom resolve time
-		[]*notifier.Alert{{State: notifier.StateFiring}},
-		[]*notifier.Alert{{LastSent: ts, End: ts.Add(5 * time.Minute)}},
+	f( // check if firing alerts need to be sent with non-zero resendDelay
+		[]*notifier.Alert{
+			{Name: "a", State: notifier.StateFiring, Start: ts},
+			// no need to resend firing
+			{Name: "b", State: notifier.StateFiring, Start: ts, LastSent: ts.Add(-30 * time.Second), End: ts.Add(5 * time.Minute)},
+			// last message is for resolved, send firing message this time
+			{Name: "c", State: notifier.StateFiring, Start: ts, LastSent: ts.Add(-30 * time.Second), End: ts.Add(-1 * time.Minute)},
+			// resend firing
+			{Name: "d", State: notifier.StateFiring, Start: ts, LastSent: ts.Add(-1 * time.Minute)},
+		},
+		[]*notifier.Alert{{Name: "a"}, {Name: "c"}, {Name: "d"}},
 		5*time.Minute, time.Minute,
 	)
-	f( // resolve inactive alert at the current timestamp
-		[]*notifier.Alert{{State: notifier.StateInactive, ResolvedAt: ts}},
-		[]*notifier.Alert{{LastSent: ts, End: ts}},
-		time.Minute, time.Minute,
-	)
-	f( // mixed case of firing and resolved alerts. Names are added for deterministic sorting
-		[]*notifier.Alert{{Name: "a", State: notifier.StateFiring}, {Name: "b", State: notifier.StateInactive, ResolvedAt: ts}},
-		[]*notifier.Alert{{Name: "a", LastSent: ts, End: ts.Add(5 * time.Minute)}, {Name: "b", LastSent: ts, End: ts}},
+	f( // check if resolved alerts need to be sent with non-zero resendDelay
+		[]*notifier.Alert{
+			{Name: "a", State: notifier.StateInactive, ResolvedAt: ts, LastSent: ts.Add(-30 * time.Second)},
+			// no need to resend resolved
+			{Name: "b", State: notifier.StateInactive, ResolvedAt: ts, LastSent: ts},
+			// resend resolved
+			{Name: "c", State: notifier.StateInactive, ResolvedAt: ts.Add(-1 * time.Minute), LastSent: ts.Add(-1 * time.Minute)},
+		},
+		[]*notifier.Alert{{Name: "a"}, {Name: "c"}},
 		5*time.Minute, time.Minute,
 	)
-	f( // mixed case of pending and resolved alerts. Names are added for deterministic sorting
-		[]*notifier.Alert{{Name: "a", State: notifier.StatePending}, {Name: "b", State: notifier.StateInactive, ResolvedAt: ts}},
-		[]*notifier.Alert{{Name: "b", LastSent: ts, End: ts}},
-		5*time.Minute, time.Minute,
-	)
-	f( // attempt to send alert that was already sent in the resendDelay interval
-		[]*notifier.Alert{{State: notifier.StateFiring, LastSent: ts.Add(-time.Second)}},
-		nil,
-		time.Minute, time.Minute,
-	)
-	f( // attempt to send alert that was sent out of the resendDelay interval
-		[]*notifier.Alert{{State: notifier.StateFiring, LastSent: ts.Add(-2 * time.Minute)}},
-		[]*notifier.Alert{{LastSent: ts, End: ts.Add(time.Minute)}},
-		time.Minute, time.Minute,
-	)
-	f( // alert must be sent even if resendDelay interval is 0
-		[]*notifier.Alert{{State: notifier.StateFiring, LastSent: ts.Add(-time.Second)}},
-		[]*notifier.Alert{{LastSent: ts, End: ts.Add(time.Minute)}},
-		time.Minute, 0,
-	)
-	f( // inactive alert which has been sent already
-		[]*notifier.Alert{{State: notifier.StateInactive, LastSent: ts.Add(-time.Second), ResolvedAt: ts.Add(-2 * time.Second)}},
-		nil,
-		time.Minute, time.Minute,
-	)
-	f( // inactive alert which has been resolved after last send
-		[]*notifier.Alert{{State: notifier.StateInactive, LastSent: ts.Add(-time.Second), ResolvedAt: ts}},
-		[]*notifier.Alert{{LastSent: ts, End: ts}},
-		time.Minute, time.Minute,
-	)
 }
 
 func newTestRuleWithLabels(name string, labels ...string) *AlertingRule {

app/vmalert/rule/group.go

@@ -704,7 +704,7 @@ func (e *executor) exec(ctx context.Context, r Rule, ts time.Time, resolveDurati
 		return nil
 	}
 
-	alerts := ar.alertsToSend(ts, resolveDuration, *resendDelay)
+	alerts := ar.alertsToSend(resolveDuration, *resendDelay)
 	if len(alerts) < 1 {
 		return nil
 	}
@@ -724,7 +724,7 @@ func (e *executor) exec(ctx context.Context, r Rule, ts time.Time, resolveDurati
 	return errGr.Err()
 }
 
-// getStaledSeries checks whether there are stale series from previously sent ones.
+// getStaleSeries checks whether there are stale series from previously sent ones.
 func (e *executor) getStaleSeries(r Rule, tss []prompbmarshal.TimeSeries, timestamp time.Time) []prompbmarshal.TimeSeries {
 	ruleLabels := make(map[string][]prompbmarshal.Label, len(tss))
 	for _, ts := range tss {

app/vmalert/templates/template.go

@@ -476,7 +476,7 @@ func templateFuncs() textTpl.FuncMap {
 		// For example, {{ query "foo" | first | value }} will
 		// execute "/api/v1/query?query=foo" request and will return
 		// the first value in response.
-		"query": func(q string) ([]metric, error) {
+		"query": func(_ string) ([]metric, error) {
 			// query function supposed to be substituted at FuncsWithQuery().
 			// it is present here only for validation purposes, when there is no
 			// provided datasource.

app/vmalert/web_test.go

@@ -36,7 +36,7 @@ func TestHandler(t *testing.T) {
 	}}
 	rh := &requestHandler{m: m}
 
-	getResp := func(url string, to interface{}, code int) {
+	getResp := func(t *testing.T, url string, to interface{}, code int) {
 		t.Helper()
 		resp, err := http.Get(url)
 		if err != nil {
@@ -60,43 +60,43 @@ func TestHandler(t *testing.T) {
 	defer ts.Close()
 
 	t.Run("/", func(t *testing.T) {
-		getResp(ts.URL, nil, 200)
-		getResp(ts.URL+"/vmalert", nil, 200)
-		getResp(ts.URL+"/vmalert/alerts", nil, 200)
-		getResp(ts.URL+"/vmalert/groups", nil, 200)
-		getResp(ts.URL+"/vmalert/notifiers", nil, 200)
-		getResp(ts.URL+"/rules", nil, 200)
+		getResp(t, ts.URL, nil, 200)
+		getResp(t, ts.URL+"/vmalert", nil, 200)
+		getResp(t, ts.URL+"/vmalert/alerts", nil, 200)
+		getResp(t, ts.URL+"/vmalert/groups", nil, 200)
+		getResp(t, ts.URL+"/vmalert/notifiers", nil, 200)
+		getResp(t, ts.URL+"/rules", nil, 200)
 	})
 
 	t.Run("/vmalert/rule", func(t *testing.T) {
 		a := ruleToAPI(ar)
-		getResp(ts.URL+"/vmalert/"+a.WebLink(), nil, 200)
+		getResp(t, ts.URL+"/vmalert/"+a.WebLink(), nil, 200)
 		r := ruleToAPI(rr)
-		getResp(ts.URL+"/vmalert/"+r.WebLink(), nil, 200)
+		getResp(t, ts.URL+"/vmalert/"+r.WebLink(), nil, 200)
 	})
 	t.Run("/vmalert/alert", func(t *testing.T) {
 		alerts := ruleToAPIAlert(ar)
 		for _, a := range alerts {
-			getResp(ts.URL+"/vmalert/"+a.WebLink(), nil, 200)
+			getResp(t, ts.URL+"/vmalert/"+a.WebLink(), nil, 200)
 		}
 	})
 	t.Run("/vmalert/rule?badParam", func(t *testing.T) {
 		params := fmt.Sprintf("?%s=0&%s=1", paramGroupID, paramRuleID)
-		getResp(ts.URL+"/vmalert/rule"+params, nil, 404)
+		getResp(t, ts.URL+"/vmalert/rule"+params, nil, 404)
 
 		params = fmt.Sprintf("?%s=1&%s=0", paramGroupID, paramRuleID)
-		getResp(ts.URL+"/vmalert/rule"+params, nil, 404)
+		getResp(t, ts.URL+"/vmalert/rule"+params, nil, 404)
 	})
 
 	t.Run("/api/v1/alerts", func(t *testing.T) {
 		lr := listAlertsResponse{}
-		getResp(ts.URL+"/api/v1/alerts", &lr, 200)
+		getResp(t, ts.URL+"/api/v1/alerts", &lr, 200)
 		if length := len(lr.Data.Alerts); length != 1 {
 			t.Errorf("expected 1 alert got %d", length)
 		}
 
 		lr = listAlertsResponse{}
-		getResp(ts.URL+"/vmalert/api/v1/alerts", &lr, 200)
+		getResp(t, ts.URL+"/vmalert/api/v1/alerts", &lr, 200)
 		if length := len(lr.Data.Alerts); length != 1 {
 			t.Errorf("expected 1 alert got %d", length)
 		}
@@ -104,13 +104,13 @@ func TestHandler(t *testing.T) {
 	t.Run("/api/v1/alert?alertID&groupID", func(t *testing.T) {
 		expAlert := newAlertAPI(ar, ar.GetAlerts()[0])
 		alert := &apiAlert{}
-		getResp(ts.URL+"/"+expAlert.APILink(), alert, 200)
+		getResp(t, ts.URL+"/"+expAlert.APILink(), alert, 200)
 		if !reflect.DeepEqual(alert, expAlert) {
 			t.Errorf("expected %v is equal to %v", alert, expAlert)
 		}
 
 		alert = &apiAlert{}
-		getResp(ts.URL+"/vmalert/"+expAlert.APILink(), alert, 200)
+		getResp(t, ts.URL+"/vmalert/"+expAlert.APILink(), alert, 200)
 		if !reflect.DeepEqual(alert, expAlert) {
 			t.Errorf("expected %v is equal to %v", alert, expAlert)
 		}
@@ -118,28 +118,28 @@ func TestHandler(t *testing.T) {
 
 	t.Run("/api/v1/alert?badParams", func(t *testing.T) {
 		params := fmt.Sprintf("?%s=0&%s=1", paramGroupID, paramAlertID)
-		getResp(ts.URL+"/api/v1/alert"+params, nil, 404)
-		getResp(ts.URL+"/vmalert/api/v1/alert"+params, nil, 404)
+		getResp(t, ts.URL+"/api/v1/alert"+params, nil, 404)
+		getResp(t, ts.URL+"/vmalert/api/v1/alert"+params, nil, 404)
 
 		params = fmt.Sprintf("?%s=1&%s=0", paramGroupID, paramAlertID)
-		getResp(ts.URL+"/api/v1/alert"+params, nil, 404)
-		getResp(ts.URL+"/vmalert/api/v1/alert"+params, nil, 404)
+		getResp(t, ts.URL+"/api/v1/alert"+params, nil, 404)
+		getResp(t, ts.URL+"/vmalert/api/v1/alert"+params, nil, 404)
 
 		// bad request, alertID is missing
 		params = fmt.Sprintf("?%s=1", paramGroupID)
-		getResp(ts.URL+"/api/v1/alert"+params, nil, 400)
-		getResp(ts.URL+"/vmalert/api/v1/alert"+params, nil, 400)
+		getResp(t, ts.URL+"/api/v1/alert"+params, nil, 400)
+		getResp(t, ts.URL+"/vmalert/api/v1/alert"+params, nil, 400)
 	})
 
 	t.Run("/api/v1/rules", func(t *testing.T) {
 		lr := listGroupsResponse{}
-		getResp(ts.URL+"/api/v1/rules", &lr, 200)
+		getResp(t, ts.URL+"/api/v1/rules", &lr, 200)
 		if length := len(lr.Data.Groups); length != 1 {
 			t.Errorf("expected 1 group got %d", length)
 		}
 
 		lr = listGroupsResponse{}
-		getResp(ts.URL+"/vmalert/api/v1/rules", &lr, 200)
+		getResp(t, ts.URL+"/vmalert/api/v1/rules", &lr, 200)
 		if length := len(lr.Data.Groups); length != 1 {
 			t.Errorf("expected 1 group got %d", length)
 		}
@@ -147,21 +147,21 @@ func TestHandler(t *testing.T) {
 	t.Run("/api/v1/rule?ruleID&groupID", func(t *testing.T) {
 		expRule := ruleToAPI(ar)
 		gotRule := apiRule{}
-		getResp(ts.URL+"/"+expRule.APILink(), &gotRule, 200)
+		getResp(t, ts.URL+"/"+expRule.APILink(), &gotRule, 200)
 
 		if expRule.ID != gotRule.ID {
 			t.Errorf("expected to get Rule %q; got %q instead", expRule.ID, gotRule.ID)
 		}
 
 		gotRule = apiRule{}
-		getResp(ts.URL+"/vmalert/"+expRule.APILink(), &gotRule, 200)
+		getResp(t, ts.URL+"/vmalert/"+expRule.APILink(), &gotRule, 200)
 
 		if expRule.ID != gotRule.ID {
 			t.Errorf("expected to get Rule %q; got %q instead", expRule.ID, gotRule.ID)
 		}
 
 		gotRuleWithUpdates := apiRuleWithUpdates{}
-		getResp(ts.URL+"/"+expRule.APILink(), &gotRuleWithUpdates, 200)
+		getResp(t, ts.URL+"/"+expRule.APILink(), &gotRuleWithUpdates, 200)
 		if gotRuleWithUpdates.StateUpdates == nil || len(gotRuleWithUpdates.StateUpdates) < 1 {
 			t.Fatalf("expected %+v to have state updates field not empty", gotRuleWithUpdates.StateUpdates)
 		}
@@ -171,7 +171,7 @@ func TestHandler(t *testing.T) {
 		check := func(url string, expGroups, expRules int) {
 			t.Helper()
 			lr := listGroupsResponse{}
-			getResp(ts.URL+url, &lr, 200)
+			getResp(t, ts.URL+url, &lr, 200)
 			if length := len(lr.Data.Groups); length != expGroups {
 				t.Errorf("expected %d groups got %d", expGroups, length)
 			}
@@ -210,7 +210,7 @@ func TestHandler(t *testing.T) {
 	t.Run("/api/v1/rules&exclude_alerts=true", func(t *testing.T) {
 		// check if response returns active alerts by default
 		lr := listGroupsResponse{}
-		getResp(ts.URL+"/api/v1/rules?rule_group[]=group&file[]=rules.yaml", &lr, 200)
+		getResp(t, ts.URL+"/api/v1/rules?rule_group[]=group&file[]=rules.yaml", &lr, 200)
 		activeAlerts := 0
 		for _, gr := range lr.Data.Groups {
 			for _, r := range gr.Rules {
@@ -223,7 +223,7 @@ func TestHandler(t *testing.T) {
 
 		// disable returning alerts via param
 		lr = listGroupsResponse{}
-		getResp(ts.URL+"/api/v1/rules?rule_group[]=group&file[]=rules.yaml&exclude_alerts=true", &lr, 200)
+		getResp(t, ts.URL+"/api/v1/rules?rule_group[]=group&file[]=rules.yaml&exclude_alerts=true", &lr, 200)
 		activeAlerts = 0
 		for _, gr := range lr.Data.Groups {
 			for _, r := range gr.Rules {
@@ -241,7 +241,7 @@ func TestEmptyResponse(t *testing.T) {
 	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { rhWithNoGroups.handler(w, r) }))
 	defer ts.Close()
 
-	getResp := func(url string, to interface{}, code int) {
+	getResp := func(t *testing.T, url string, to interface{}, code int) {
 		t.Helper()
 		resp, err := http.Get(url)
 		if err != nil {
@@ -264,13 +264,13 @@ func TestEmptyResponse(t *testing.T) {
 
 	t.Run("no groups /api/v1/alerts", func(t *testing.T) {
 		lr := listAlertsResponse{}
-		getResp(ts.URL+"/api/v1/alerts", &lr, 200)
+		getResp(t, ts.URL+"/api/v1/alerts", &lr, 200)
 		if lr.Data.Alerts == nil {
 			t.Errorf("expected /api/v1/alerts response to have non-nil data")
 		}
 
 		lr = listAlertsResponse{}
-		getResp(ts.URL+"/vmalert/api/v1/alerts", &lr, 200)
+		getResp(t, ts.URL+"/vmalert/api/v1/alerts", &lr, 200)
 		if lr.Data.Alerts == nil {
 			t.Errorf("expected /api/v1/alerts response to have non-nil data")
 		}
@@ -278,13 +278,13 @@ func TestEmptyResponse(t *testing.T) {
 
 	t.Run("no groups /api/v1/rules", func(t *testing.T) {
 		lr := listGroupsResponse{}
-		getResp(ts.URL+"/api/v1/rules", &lr, 200)
+		getResp(t, ts.URL+"/api/v1/rules", &lr, 200)
 		if lr.Data.Groups == nil {
 			t.Errorf("expected /api/v1/rules response to have non-nil data")
 		}
 
 		lr = listGroupsResponse{}
-		getResp(ts.URL+"/vmalert/api/v1/rules", &lr, 200)
+		getResp(t, ts.URL+"/vmalert/api/v1/rules", &lr, 200)
 		if lr.Data.Groups == nil {
 			t.Errorf("expected /api/v1/rules response to have non-nil data")
 		}
@@ -295,13 +295,13 @@ func TestEmptyResponse(t *testing.T) {
 
 	t.Run("empty group /api/v1/rules", func(t *testing.T) {
 		lr := listGroupsResponse{}
-		getResp(ts.URL+"/api/v1/rules", &lr, 200)
+		getResp(t, ts.URL+"/api/v1/rules", &lr, 200)
 		if lr.Data.Groups == nil {
 			t.Fatalf("expected /api/v1/rules response to have non-nil data")
 		}
 
 		lr = listGroupsResponse{}
-		getResp(ts.URL+"/vmalert/api/v1/rules", &lr, 200)
+		getResp(t, ts.URL+"/vmalert/api/v1/rules", &lr, 200)
 		if lr.Data.Groups == nil {
 			t.Fatalf("expected /api/v1/rules response to have non-nil data")
 		}

app/vmauth/auth_config.go

@@ -2,15 +2,17 @@ package main
 
 import (
 	"bytes"
+	"context"
 	"encoding/base64"
 	"flag"
 	"fmt"
+	"math"
+	"net"
 	"net/http"
 	"net/url"
 	"os"
 	"regexp"
 	"sort"
-	"strconv"
 	"strings"
 	"sync"
 	"sync/atomic"
@@ -36,7 +38,12 @@ var (
 	defaultRetryStatusCodes = flagutil.NewArrayInt("retryStatusCodes", 0, "Comma-separated list of default HTTP response status codes when vmauth re-tries the request on other backends. "+
 		"See https://docs.victoriametrics.com/vmauth.html#load-balancing for details")
 	defaultLoadBalancingPolicy = flag.String("loadBalancingPolicy", "least_loaded", "The default load balancing policy to use for backend urls specified inside url_prefix section. "+
-		"Supported policies: least_loaded, first_available. See https://docs.victoriametrics.com/vmauth.html#load-balancing for more details")
+		"Supported policies: least_loaded, first_available. See https://docs.victoriametrics.com/vmauth.html#load-balancing")
+	discoverBackendIPsGlobal = flag.Bool("discoverBackendIPs", false, "Whether to discover backend IPs via periodic DNS queries to hostnames specified in url_prefix. "+
+		"This may be useful when url_prefix points to a hostname with dynamically scaled instances behind it. See https://docs.victoriametrics.com/vmauth.html#discovering-backend-ips")
+	discoverBackendIPsInterval = flag.Duration("discoverBackendIPsInterval", 10*time.Second, "The interval for re-discovering backend IPs if -discoverBackendIPs command-line flag is set. "+
+		"Too low value may lead to DNS errors")
+	httpAuthHeader = flagutil.NewArrayString("httpAuthHeader", "HTTP request header to use for obtaining authorization tokens. By default auth tokens are read from Authorization request header")
 )
 
 // AuthConfig represents auth config.
@@ -50,11 +57,15 @@ type AuthConfig struct {
 
 // UserInfo is user information read from authConfigPath
 type UserInfo struct {
-	Name                   string      `yaml:"name,omitempty"`
-	BearerToken            string      `yaml:"bearer_token,omitempty"`
-	Username               string      `yaml:"username,omitempty"`
-	Password               string      `yaml:"password,omitempty"`
+	Name string `yaml:"name,omitempty"`
+
+	BearerToken string `yaml:"bearer_token,omitempty"`
+	AuthToken   string `yaml:"auth_token,omitempty"`
+	Username    string `yaml:"username,omitempty"`
+	Password    string `yaml:"password,omitempty"`
+
 	URLPrefix              *URLPrefix  `yaml:"url_prefix,omitempty"`
+	DiscoverBackendIPs     *bool       `yaml:"discover_backend_ips,omitempty"`
 	URLMaps                []URLMap    `yaml:"url_map,omitempty"`
 	HeadersConf            HeadersConf `yaml:",inline"`
 	MaxConcurrentRequests  int         `yaml:"max_concurrent_requests,omitempty"`
@@ -109,6 +120,8 @@ func (ui *UserInfo) getMaxConcurrentRequests() int {
 type Header struct {
 	Name  string
 	Value string
+
+	sOriginal string
 }
 
 // UnmarshalYAML unmarshals h from f.
@@ -117,6 +130,8 @@ func (h *Header) UnmarshalYAML(f func(interface{}) error) error {
 	if err := f(&s); err != nil {
 		return err
 	}
+	h.sOriginal = s
+
 	n := strings.IndexByte(s, ':')
 	if n < 0 {
 		return fmt.Errorf("missing speparator char ':' between Name and Value in the header %q; expected format - 'Name: Value'", s)
@@ -128,21 +143,29 @@ func (h *Header) UnmarshalYAML(f func(interface{}) error) error {
 
 // MarshalYAML marshals h to yaml.
 func (h *Header) MarshalYAML() (interface{}, error) {
-	s := fmt.Sprintf("%s: %s", h.Name, h.Value)
-	return s, nil
+	return h.sOriginal, nil
 }
 
 // URLMap is a mapping from source paths to target urls.
 type URLMap struct {
-	// SrcHosts is the list of regular expressions, which match the request hostname.
+	// SrcPaths is an optional list of regular expressions, which must match the request path.
+	SrcPaths []*Regex `yaml:"src_paths,omitempty"`
+
+	// SrcHosts is an optional list of regular expressions, which must match the request hostname.
 	SrcHosts []*Regex `yaml:"src_hosts,omitempty"`
 
-	// SrcPaths is the list of regular expressions, which match the request path.
-	SrcPaths []*Regex `yaml:"src_paths,omitempty"`
+	// SrcQueryArgs is an optional list of query args, which must match request URL query args.
+	SrcQueryArgs []QueryArg `yaml:"src_query_args,omitempty"`
+
+	// SrcHeaders is an optional list of headers, which must match request headers.
+	SrcHeaders []Header `yaml:"src_headers,omitempty"`
 
 	// UrlPrefix contains backend url prefixes for the proxied request url.
 	URLPrefix *URLPrefix `yaml:"url_prefix,omitempty"`
 
+	// DiscoverBackendIPs instructs discovering URLPrefix backend IPs via DNS.
+	DiscoverBackendIPs *bool `yaml:"discover_backend_ips,omitempty"`
+
 	// HeadersConf is the config for augumenting request and response headers.
 	HeadersConf HeadersConf `yaml:",inline"`
 
@@ -158,25 +181,70 @@ type URLMap struct {
 
 // Regex represents a regex
 type Regex struct {
+	re *regexp.Regexp
+
 	sOriginal string
-	re        *regexp.Regexp
+}
+
+// QueryArg represents HTTP query arg
+type QueryArg struct {
+	Name  string
+	Value string
+
+	sOriginal string
+}
+
+// UnmarshalYAML unmarshals up from yaml.
+func (qa *QueryArg) UnmarshalYAML(f func(interface{}) error) error {
+	var s string
+	if err := f(&s); err != nil {
+		return err
+	}
+	qa.sOriginal = s
+
+	n := strings.IndexByte(s, '=')
+	if n >= 0 {
+		qa.Name = s[:n]
+		qa.Value = s[n+1:]
+	}
+	return nil
+}
+
+// MarshalYAML marshals up to yaml.
+func (qa *QueryArg) MarshalYAML() (interface{}, error) {
+	return qa.sOriginal, nil
 }
 
 // URLPrefix represents passed `url_prefix`
 type URLPrefix struct {
-	n uint32
-
-	// the list of backend urls
-	bus []*backendURL
-
 	// requests are re-tried on other backend urls for these http response status codes
 	retryStatusCodes []int
 
 	// load balancing policy used
 	loadBalancingPolicy string
 
-	// how many request path prefix parts to drop before routing the request to backendURL.
+	// how many request path prefix parts to drop before routing the request to backendURL
 	dropSrcPathPrefixParts int
+
+	// busOriginal contains the original list of backends specified in yaml config.
+	busOriginal []*url.URL
+
+	// n is an atomic counter, which is used for balancing load among available backends.
+	n atomic.Uint32
+
+	// the list of backend urls
+	//
+	// the list can be dynamically updated if `discover_backend_ips` option is set.
+	bus atomic.Pointer[[]*backendURL]
+
+	// if this option is set, then backend ips for busOriginal are periodically re-discovered and put to bus.
+	discoverBackendIPs bool
+
+	// The next deadline for DNS-based discovery of backend IPs
+	nextDiscoveryDeadline atomic.Uint64
+
+	// vOriginal contains the original yaml value for URLPrefix.
+	vOriginal interface{}
 }
 
 func (up *URLPrefix) setLoadBalancingPolicy(loadBalancingPolicy string) error {
@@ -192,49 +260,146 @@ func (up *URLPrefix) setLoadBalancingPolicy(loadBalancingPolicy string) error {
 }
 
 type backendURL struct {
-	brokenDeadline     uint64
-	concurrentRequests int32
-	url                *url.URL
+	brokenDeadline     atomic.Uint64
+	concurrentRequests atomic.Int32
+
+	url *url.URL
 }
 
 func (bu *backendURL) isBroken() bool {
 	ct := fasttime.UnixTimestamp()
-	return ct < atomic.LoadUint64(&bu.brokenDeadline)
+	return ct < bu.brokenDeadline.Load()
 }
 
 func (bu *backendURL) setBroken() {
 	deadline := fasttime.UnixTimestamp() + uint64((*failTimeout).Seconds())
-	atomic.StoreUint64(&bu.brokenDeadline, deadline)
+	bu.brokenDeadline.Store(deadline)
 }
 
 func (bu *backendURL) get() {
-	atomic.AddInt32(&bu.concurrentRequests, 1)
+	bu.concurrentRequests.Add(1)
 }
 
 func (bu *backendURL) put() {
-	atomic.AddInt32(&bu.concurrentRequests, -1)
+	bu.concurrentRequests.Add(-1)
 }
 
 func (up *URLPrefix) getBackendsCount() int {
-	return len(up.bus)
+	pbus := up.bus.Load()
+	return len(*pbus)
 }
 
 // getBackendURL returns the backendURL depending on the load balance policy.
 //
 // backendURL.put() must be called on the returned backendURL after the request is complete.
 func (up *URLPrefix) getBackendURL() *backendURL {
+	up.discoverBackendIPsIfNeeded()
+
+	pbus := up.bus.Load()
+	bus := *pbus
 	if up.loadBalancingPolicy == "first_available" {
-		return up.getFirstAvailableBackendURL()
+		return getFirstAvailableBackendURL(bus)
 	}
-	return up.getLeastLoadedBackendURL()
+	return getLeastLoadedBackendURL(bus, &up.n)
+}
+
+func (up *URLPrefix) discoverBackendIPsIfNeeded() {
+	if !up.discoverBackendIPs {
+		// The discovery is disabled.
+		return
+	}
+
+	ct := fasttime.UnixTimestamp()
+	deadline := up.nextDiscoveryDeadline.Load()
+	if ct < deadline {
+		// There is no need in discovering backends.
+		return
+	}
+
+	intervalSec := math.Ceil(discoverBackendIPsInterval.Seconds())
+	if intervalSec <= 0 {
+		intervalSec = 1
+	}
+	nextDeadline := ct + uint64(intervalSec)
+	if !up.nextDiscoveryDeadline.CompareAndSwap(deadline, nextDeadline) {
+		// Concurrent goroutine already started the discovery.
+		return
+	}
+
+	// Discover ips for all the backendURLs
+	ctx, cancel := context.WithTimeout(context.Background(), time.Second*time.Duration(intervalSec))
+	hostToIPs := make(map[string][]string)
+	for _, bu := range up.busOriginal {
+		host := bu.Hostname()
+		if hostToIPs[host] != nil {
+			// ips for the given host have been already discovered
+			continue
+		}
+		addrs, err := resolver.LookupIPAddr(ctx, host)
+		var ips []string
+		if err != nil {
+			logger.Warnf("cannot discover backend IPs for %s: %s; use it literally", bu, err)
+			ips = []string{host}
+		} else {
+			ips = make([]string, len(addrs))
+			for i, addr := range addrs {
+				ips[i] = addr.String()
+			}
+			// sort ips, so they could be compared below in areEqualBackendURLs()
+			sort.Strings(ips)
+		}
+		hostToIPs[host] = ips
+	}
+	cancel()
+
+	// generate new backendURLs for the resolved IPs
+	var busNew []*backendURL
+	for _, bu := range up.busOriginal {
+		host := bu.Hostname()
+		port := bu.Port()
+		for _, ip := range hostToIPs[host] {
+			buCopy := *bu
+			buCopy.Host = ip
+			if port != "" {
+				buCopy.Host += ":" + port
+			}
+			busNew = append(busNew, &backendURL{
+				url: &buCopy,
+			})
+		}
+	}
+
+	pbus := up.bus.Load()
+	if areEqualBackendURLs(*pbus, busNew) {
+		return
+	}
+
+	// Store new backend urls
+	up.bus.Store(&busNew)
+}
+
+func areEqualBackendURLs(a, b []*backendURL) bool {
+	if len(a) != len(b) {
+		return false
+	}
+	for i, aURL := range a {
+		bURL := b[i]
+		if aURL.url.String() != bURL.url.String() {
+			return false
+		}
+	}
+	return true
+}
+
+var resolver = &net.Resolver{
+	PreferGo:     true,
+	StrictErrors: true,
 }
 
 // getFirstAvailableBackendURL returns the first available backendURL, which isn't broken.
 //
 // backendURL.put() must be called on the returned backendURL after the request is complete.
-func (up *URLPrefix) getFirstAvailableBackendURL() *backendURL {
-	bus := up.bus
-
+func getFirstAvailableBackendURL(bus []*backendURL) *backendURL {
 	bu := bus[0]
 	if !bu.isBroken() {
 		// Fast path - send the request to the first url.
@@ -256,8 +421,7 @@ func (up *URLPrefix) getFirstAvailableBackendURL() *backendURL {
 // getLeastLoadedBackendURL returns the backendURL with the minimum number of concurrent requests.
 //
 // backendURL.put() must be called on the returned backendURL after the request is complete.
-func (up *URLPrefix) getLeastLoadedBackendURL() *backendURL {
-	bus := up.bus
+func getLeastLoadedBackendURL(bus []*backendURL, atomicCounter *atomic.Uint32) *backendURL {
 	if len(bus) == 1 {
 		// Fast path - return the only backend url.
 		bu := bus[0]
@@ -266,7 +430,7 @@ func (up *URLPrefix) getLeastLoadedBackendURL() *backendURL {
 	}
 
 	// Slow path - select other backend urls.
-	n := atomic.AddUint32(&up.n, 1)
+	n := atomicCounter.Add(1)
 
 	for i := uint32(0); i < uint32(len(bus)); i++ {
 		idx := (n + i) % uint32(len(bus))
@@ -274,22 +438,22 @@ func (up *URLPrefix) getLeastLoadedBackendURL() *backendURL {
 		if bu.isBroken() {
 			continue
 		}
-		if atomic.LoadInt32(&bu.concurrentRequests) == 0 {
+		if bu.concurrentRequests.Load() == 0 {
 			// Fast path - return the backend with zero concurrently executed requests.
-			// Do not use atomic.CompareAndSwapInt32(), since it is much slower on systems with many CPU cores.
-			atomic.AddInt32(&bu.concurrentRequests, 1)
+			// Do not use CompareAndSwap() instead of Load(), since it is much slower on systems with many CPU cores.
+			bu.concurrentRequests.Add(1)
 			return bu
 		}
 	}
 
 	// Slow path - return the backend with the minimum number of concurrently executed requests.
 	buMin := bus[n%uint32(len(bus))]
-	minRequests := atomic.LoadInt32(&buMin.concurrentRequests)
+	minRequests := buMin.concurrentRequests.Load()
 	for _, bu := range bus {
 		if bu.isBroken() {
 			continue
 		}
-		if n := atomic.LoadInt32(&bu.concurrentRequests); n < minRequests {
+		if n := bu.concurrentRequests.Load(); n < minRequests {
 			buMin = bu
 			minRequests = n
 		}
@@ -304,6 +468,7 @@ func (up *URLPrefix) UnmarshalYAML(f func(interface{}) error) error {
 	if err := f(&v); err != nil {
 		return err
 	}
+	up.vOriginal = v
 
 	var urls []string
 	switch x := v.(type) {
@@ -326,38 +491,21 @@ func (up *URLPrefix) UnmarshalYAML(f func(interface{}) error) error {
 		return fmt.Errorf("unexpected type for `url_prefix`: %T; want string or []string", v)
 	}
 
-	bus := make([]*backendURL, len(urls))
+	bus := make([]*url.URL, len(urls))
 	for i, u := range urls {
 		pu, err := url.Parse(u)
 		if err != nil {
 			return fmt.Errorf("cannot unmarshal %q into url: %w", u, err)
 		}
-		bus[i] = &backendURL{
-			url: pu,
-		}
+		bus[i] = pu
 	}
-	up.bus = bus
+	up.busOriginal = bus
 	return nil
 }
 
 // MarshalYAML marshals up to yaml.
 func (up *URLPrefix) MarshalYAML() (interface{}, error) {
-	var b []byte
-	if len(up.bus) == 1 {
-		u := up.bus[0].url.String()
-		b = strconv.AppendQuote(b, u)
-		return string(b), nil
-	}
-	b = append(b, '[')
-	for i, bu := range up.bus {
-		u := bu.url.String()
-		b = strconv.AppendQuote(b, u)
-		if i+1 < len(up.bus) {
-			b = append(b, ',')
-		}
-	}
-	b = append(b, ']')
-	return string(b), nil
+	return up.vOriginal, nil
 }
 
 func (r *Regex) match(s string) bool {
@@ -378,12 +526,13 @@ func (r *Regex) UnmarshalYAML(f func(interface{}) error) error {
 	if err := f(&s); err != nil {
 		return err
 	}
+	r.sOriginal = s
+
 	sAnchored := "^(?:" + s + ")$"
 	re, err := regexp.Compile(sAnchored)
 	if err != nil {
 		return fmt.Errorf("cannot build regexp from %q: %w", s, err)
 	}
-	r.sOriginal = s
 	r.re = re
 	return nil
 }
@@ -541,6 +690,9 @@ func parseAuthConfig(data []byte) (*AuthConfig, error) {
 		if ui.BearerToken != "" {
 			return nil, fmt.Errorf("field bearer_token can't be specified for unauthorized_user section")
 		}
+		if ui.AuthToken != "" {
+			return nil, fmt.Errorf("field auth_token can't be specified for unauthorized_user section")
+		}
 		if ui.Name != "" {
 			return nil, fmt.Errorf("field name can't be specified for unauthorized_user section")
 		}
@@ -581,17 +733,9 @@ func parseAuthConfigUsers(ac *AuthConfig) (map[string]*UserInfo, error) {
 	byAuthToken := make(map[string]*UserInfo, len(uis))
 	for i := range uis {
 		ui := &uis[i]
-		if ui.Username != "" && ui.Password == "" {
-			// Do not allow setting username without password if there are other auth configs exist.
-			// This should prevent from typical mis-configuration when access by username without password
-			// remains open if other authorization schemes are defined.
-			if ui.BearerToken != "" {
-				return nil, fmt.Errorf("bearer_token=%q and username=%q cannot be set simultaneously", ui.BearerToken, ui.Username)
-			}
-		}
-		ats := getAuthTokens(ui.BearerToken, ui.Username, ui.Password)
-		if len(ats) == 0 {
-			return nil, fmt.Errorf("one of bearer_token, username or mtls must be set")
+		ats, err := getAuthTokens(ui.AuthToken, ui.BearerToken, ui.Username, ui.Password)
+		if err != nil {
+			return nil, err
 		}
 		for _, at := range ats {
 			if uiOld := byAuthToken[at]; uiOld != nil {
@@ -599,15 +743,10 @@ func parseAuthConfigUsers(ac *AuthConfig) (map[string]*UserInfo, error) {
 					at, ui.Username, ui.Name, uiOld.Username, uiOld.Name)
 			}
 		}
-
 		if err := ui.initURLs(); err != nil {
 			return nil, err
 		}
 
-		if ui.BearerToken != "" && ui.Password != "" {
-			return nil, fmt.Errorf("password shouldn't be set for bearer_token %q", ui.BearerToken)
-		}
-
 		metricLabels, err := ui.getMetricLabels()
 		if err != nil {
 			return nil, fmt.Errorf("cannot parse metric_labels: %w", err)
@@ -665,8 +804,9 @@ func (ui *UserInfo) initURLs() error {
 	retryStatusCodes := defaultRetryStatusCodes.Values()
 	loadBalancingPolicy := *defaultLoadBalancingPolicy
 	dropSrcPathPrefixParts := 0
+	discoverBackendIPs := *discoverBackendIPsGlobal
 	if ui.URLPrefix != nil {
-		if err := ui.URLPrefix.sanitize(); err != nil {
+		if err := ui.URLPrefix.sanitizeAndInitialize(); err != nil {
 			return err
 		}
 		if ui.RetryStatusCodes != nil {
@@ -678,30 +818,35 @@ func (ui *UserInfo) initURLs() error {
 		if ui.DropSrcPathPrefixParts != nil {
 			dropSrcPathPrefixParts = *ui.DropSrcPathPrefixParts
 		}
+		if ui.DiscoverBackendIPs != nil {
+			discoverBackendIPs = *ui.DiscoverBackendIPs
+		}
 		ui.URLPrefix.retryStatusCodes = retryStatusCodes
 		ui.URLPrefix.dropSrcPathPrefixParts = dropSrcPathPrefixParts
+		ui.URLPrefix.discoverBackendIPs = discoverBackendIPs
 		if err := ui.URLPrefix.setLoadBalancingPolicy(loadBalancingPolicy); err != nil {
 			return err
 		}
 	}
 	if ui.DefaultURL != nil {
-		if err := ui.DefaultURL.sanitize(); err != nil {
+		if err := ui.DefaultURL.sanitizeAndInitialize(); err != nil {
 			return err
 		}
 	}
 	for _, e := range ui.URLMaps {
-		if len(e.SrcPaths) == 0 && len(e.SrcHosts) == 0 {
-			return fmt.Errorf("missing `src_paths` and `src_hosts` in `url_map`")
+		if len(e.SrcPaths) == 0 && len(e.SrcHosts) == 0 && len(e.SrcQueryArgs) == 0 && len(e.SrcHeaders) == 0 {
+			return fmt.Errorf("missing `src_paths`, `src_hosts`, `src_query_args` and `src_headers` in `url_map`")
 		}
 		if e.URLPrefix == nil {
 			return fmt.Errorf("missing `url_prefix` in `url_map`")
 		}
-		if err := e.URLPrefix.sanitize(); err != nil {
+		if err := e.URLPrefix.sanitizeAndInitialize(); err != nil {
 			return err
 		}
 		rscs := retryStatusCodes
 		lbp := loadBalancingPolicy
 		dsp := dropSrcPathPrefixParts
+		dbd := discoverBackendIPs
 		if e.RetryStatusCodes != nil {
 			rscs = e.RetryStatusCodes
 		}
@@ -711,14 +856,18 @@ func (ui *UserInfo) initURLs() error {
 		if e.DropSrcPathPrefixParts != nil {
 			dsp = *e.DropSrcPathPrefixParts
 		}
+		if e.DiscoverBackendIPs != nil {
+			dbd = *e.DiscoverBackendIPs
+		}
 		e.URLPrefix.retryStatusCodes = rscs
 		if err := e.URLPrefix.setLoadBalancingPolicy(lbp); err != nil {
 			return err
 		}
 		e.URLPrefix.dropSrcPathPrefixParts = dsp
+		e.URLPrefix.discoverBackendIPs = dbd
 	}
 	if len(ui.URLMaps) == 0 && ui.URLPrefix == nil {
-		return fmt.Errorf("missing `url_prefix`")
+		return fmt.Errorf("missing `url_prefix` or `url_map`")
 	}
 	return nil
 }
@@ -734,21 +883,42 @@ func (ui *UserInfo) name() string {
 		h := xxhash.Sum64([]byte(ui.BearerToken))
 		return fmt.Sprintf("bearer_token:hash:%016X", h)
 	}
+	if ui.AuthToken != "" {
+		h := xxhash.Sum64([]byte(ui.AuthToken))
+		return fmt.Sprintf("auth_token:hash:%016X", h)
+	}
 	return ""
 }
 
-func getAuthTokens(bearerToken, username, password string) []string {
-	var ats []string
+func getAuthTokens(authToken, bearerToken, username, password string) ([]string, error) {
+	if authToken != "" {
+		if bearerToken != "" {
+			return nil, fmt.Errorf("bearer_token cannot be specified if auth_token is set")
+		}
+		if username != "" || password != "" {
+			return nil, fmt.Errorf("username and password cannot be specified if auth_token is set")
+		}
+		at := getHTTPAuthToken(authToken)
+		return []string{at}, nil
+	}
 	if bearerToken != "" {
+		if username != "" || password != "" {
+			return nil, fmt.Errorf("username and password cannot be specified if bearer_token is set")
+		}
 		// Accept the bearerToken as Basic Auth username with empty password
 		at1 := getHTTPAuthBearerToken(bearerToken)
 		at2 := getHTTPAuthBasicToken(bearerToken, "")
-		ats = append(ats, at1, at2)
-	} else if username != "" {
-		at := getHTTPAuthBasicToken(username, password)
-		ats = append(ats, at)
+		return []string{at1, at2}, nil
 	}
-	return ats
+	if username != "" {
+		at := getHTTPAuthBasicToken(username, password)
+		return []string{at}, nil
+	}
+	return nil, fmt.Errorf("missing authorization options; bearer_token or username must be set")
+}
+
+func getHTTPAuthToken(authToken string) string {
+	return "http_auth:" + authToken
 }
 
 func getHTTPAuthBearerToken(bearerToken string) string {
@@ -761,31 +931,49 @@ func getHTTPAuthBasicToken(username, password string) string {
 	return "http_auth:Basic " + token64
 }
 
+var defaultHeaderNames = []string{"Authorization"}
+
 func getAuthTokensFromRequest(r *http.Request) []string {
 	var ats []string
 
-	ah := r.Header.Get("Authorization")
-	if ah == "" {
-		return ats
+	// Obtain possible auth tokens from one of the allowed auth headers
+	headerNames := *httpAuthHeader
+	if len(headerNames) == 0 {
+		headerNames = defaultHeaderNames
 	}
-	if strings.HasPrefix(ah, "Token ") {
-		// Handle InfluxDB's proprietary token authentication scheme as a bearer token authentication
-		// See https://docs.influxdata.com/influxdb/v2.0/api/
-		ah = strings.Replace(ah, "Token", "Bearer", 1)
+	for _, headerName := range headerNames {
+		if ah := r.Header.Get(headerName); ah != "" {
+			if strings.HasPrefix(ah, "Token ") {
+				// Handle InfluxDB's proprietary token authentication scheme as a bearer token authentication
+				// See https://docs.influxdata.com/influxdb/v2.0/api/
+				ah = strings.Replace(ah, "Token", "Bearer", 1)
+			}
+			at := "http_auth:" + ah
+			ats = append(ats, at)
+		}
 	}
-	at := "http_auth:" + ah
-	ats = append(ats, at)
+
 	return ats
 }
 
-func (up *URLPrefix) sanitize() error {
-	for _, bu := range up.bus {
-		puNew, err := sanitizeURLPrefix(bu.url)
+func (up *URLPrefix) sanitizeAndInitialize() error {
+	for i, bu := range up.busOriginal {
+		puNew, err := sanitizeURLPrefix(bu)
 		if err != nil {
 			return err
 		}
-		bu.url = puNew
+		up.busOriginal[i] = puNew
 	}
+
+	// Initialize up.bus
+	bus := make([]*backendURL, len(up.busOriginal))
+	for i, bu := range up.busOriginal {
+		bus[i] = &backendURL{
+			url: bu,
+		}
+	}
+	up.bus.Store(&bus)
+
 	return nil
 }
 

app/vmauth/auth_config_test.go

@@ -17,9 +17,9 @@ func TestParseAuthConfigFailure(t *testing.T) {
 		if err != nil {
 			return
 		}
-		_, err = parseAuthConfigUsers(ac)
+		users, err := parseAuthConfigUsers(ac)
 		if err == nil {
-			t.Fatalf("expecting non-nil error")
+			t.Fatalf("expecting non-nil error; got %v", users)
 		}
 	}
 
@@ -88,6 +88,22 @@ users:
   url_prefix: []
 `)
 
+	// auth_token and username in a single config
+	f(`
+users:
+- auth_token: foo
+  username: bbb
+  url_prefix: http://foo.bar
+`)
+
+	// auth_token and bearer_token in a single config
+	f(`
+users:
+- auth_token: foo
+  bearer_token: bbb
+  url_prefix: http://foo.bar
+`)
+
 	// Username and bearer_token in a single config
 	f(`
 users:
@@ -192,7 +208,7 @@ users:
   - url_prefix: http://foobar
 `)
 
-	// Invalid regexp in src_path.
+	// Invalid regexp in src_paths
 	f(`
 users:
 - username: a
@@ -210,6 +226,24 @@ users:
     url_prefix: http://foobar
 `)
 
+	// Invalid src_query_args
+	f(`
+users:
+- username: a
+  url_map:
+  - src_query_args: abc
+    url_prefix: http://foobar
+`)
+
+	// Invalid src_headers
+	f(`
+users:
+- username: a
+  url_map:
+  - src_headers: abc
+    url_prefix: http://foobar
+`)
+
 	// Invalid headers in url_map (missing ':')
 	f(`
 users:
@@ -257,8 +291,9 @@ func TestParseAuthConfigSuccess(t *testing.T) {
 		}
 	}
 
-	// Single user
 	insecureSkipVerifyTrue := true
+
+	// Single user
 	f(`
 users:
 - username: foo
@@ -276,6 +311,22 @@ users:
 		},
 	})
 
+	// Single user with auth_token
+	f(`
+users:
+- auth_token: foo
+  url_prefix: http://aaa:343/bbb
+  max_concurrent_requests: 5
+  tls_insecure_skip_verify: true
+`, map[string]*UserInfo{
+		getHTTPAuthToken("foo"): {
+			AuthToken:             "foo",
+			URLPrefix:             mustParseURL("http://aaa:343/bbb"),
+			MaxConcurrentRequests: 5,
+			TLSInsecureSkipVerify: &insecureSkipVerifyTrue,
+		},
+	})
+
 	// Multiple url_prefix entries
 	insecureSkipVerifyFalse := false
 	f(`
@@ -310,7 +361,7 @@ users:
 - username: foo
   url_prefix: http://foo
 - username: bar
-  url_prefix: https://bar/x///
+  url_prefix: https://bar/x/
 `, map[string]*UserInfo{
 		getHTTPAuthBasicToken("foo", ""): {
 			Username:  "foo",
@@ -318,11 +369,52 @@ users:
 		},
 		getHTTPAuthBasicToken("bar", ""): {
 			Username:  "bar",
-			URLPrefix: mustParseURL("https://bar/x"),
+			URLPrefix: mustParseURL("https://bar/x/"),
 		},
 	})
 
 	// non-empty URLMap
+	sharedUserInfo := &UserInfo{
+		BearerToken: "foo",
+		URLMaps: []URLMap{
+			{
+				SrcPaths:  getRegexs([]string{"/api/v1/query", "/api/v1/query_range", "/api/v1/label/[^./]+/.+"}),
+				URLPrefix: mustParseURL("http://vmselect/select/0/prometheus"),
+			},
+			{
+				SrcHosts: getRegexs([]string{"foo\\.bar", "baz:1234"}),
+				SrcPaths: getRegexs([]string{"/api/v1/write"}),
+				SrcQueryArgs: []QueryArg{
+					{
+						Name:  "foo",
+						Value: "bar",
+					},
+				},
+				SrcHeaders: []Header{
+					{
+						Name:  "TenantID",
+						Value: "345",
+					},
+				},
+				URLPrefix: mustParseURLs([]string{
+					"http://vminsert1/insert/0/prometheus",
+					"http://vminsert2/insert/0/prometheus",
+				}),
+				HeadersConf: HeadersConf{
+					RequestHeaders: []Header{
+						{
+							Name:  "foo",
+							Value: "bar",
+						},
+						{
+							Name:  "xxx",
+							Value: "y",
+						},
+					},
+				},
+			},
+		},
+	}
 	f(`
 users:
 - bearer_token: foo
@@ -331,70 +423,17 @@ users:
     url_prefix: http://vmselect/select/0/prometheus
   - src_paths: ["/api/v1/write"]
     src_hosts: ["foo\\.bar", "baz:1234"]
+    src_query_args: ['foo=bar']
+    src_headers: ['TenantID: 345']
     url_prefix: ["http://vminsert1/insert/0/prometheus","http://vminsert2/insert/0/prometheus"]
     headers:
     - "foo: bar"
     - "xxx: y"
 `, map[string]*UserInfo{
-		getHTTPAuthBearerToken("foo"): {
-			BearerToken: "foo",
-			URLMaps: []URLMap{
-				{
-					SrcPaths:  getRegexs([]string{"/api/v1/query", "/api/v1/query_range", "/api/v1/label/[^./]+/.+"}),
-					URLPrefix: mustParseURL("http://vmselect/select/0/prometheus"),
-				},
-				{
-					SrcHosts: getRegexs([]string{"foo\\.bar", "baz:1234"}),
-					SrcPaths: getRegexs([]string{"/api/v1/write"}),
-					URLPrefix: mustParseURLs([]string{
-						"http://vminsert1/insert/0/prometheus",
-						"http://vminsert2/insert/0/prometheus",
-					}),
-					HeadersConf: HeadersConf{
-						RequestHeaders: []Header{
-							{
-								Name:  "foo",
-								Value: "bar",
-							},
-							{
-								Name:  "xxx",
-								Value: "y",
-							},
-						},
-					},
-				},
-			},
-		},
-		getHTTPAuthBasicToken("foo", ""): {
-			BearerToken: "foo",
-			URLMaps: []URLMap{
-				{
-					SrcPaths:  getRegexs([]string{"/api/v1/query", "/api/v1/query_range", "/api/v1/label/[^./]+/.+"}),
-					URLPrefix: mustParseURL("http://vmselect/select/0/prometheus"),
-				},
-				{
-					SrcHosts: getRegexs([]string{"foo\\.bar", "baz:1234"}),
-					SrcPaths: getRegexs([]string{"/api/v1/write"}),
-					URLPrefix: mustParseURLs([]string{
-						"http://vminsert1/insert/0/prometheus",
-						"http://vminsert2/insert/0/prometheus",
-					}),
-					HeadersConf: HeadersConf{
-						RequestHeaders: []Header{
-							{
-								Name:  "foo",
-								Value: "bar",
-							},
-							{
-								Name:  "xxx",
-								Value: "y",
-							},
-						},
-					},
-				},
-			},
-		},
+		getHTTPAuthBearerToken("foo"):    sharedUserInfo,
+		getHTTPAuthBasicToken("foo", ""): sharedUserInfo,
 	})
+
 	// Multiple users with the same name - this should work, since these users have different passwords
 	f(`
 users:
@@ -403,7 +442,7 @@ users:
   url_prefix: http://foo
 - username: foo-same
   password: bar
-  url_prefix: https://bar/x///
+  url_prefix: https://bar/x
 `, map[string]*UserInfo{
 		getHTTPAuthBasicToken("foo-same", "baz"): {
 			Username:  "foo-same",
@@ -498,6 +537,7 @@ users:
 			}),
 		},
 	})
+
 	// With metric_labels
 	f(`
 users:
@@ -509,7 +549,7 @@ users:
     team: dev
 - username: foo-same
   password: bar
-  url_prefix: https://bar/x///
+  url_prefix: https://bar/x
   metric_labels:
     backend_env: test
     team: accounting
@@ -694,6 +734,7 @@ func mustParseURL(u string) *URLPrefix {
 
 func mustParseURLs(us []string) *URLPrefix {
 	bus := make([]*backendURL, len(us))
+	urls := make([]*url.URL, len(us))
 	for i, u := range us {
 		pu, err := url.Parse(u)
 		if err != nil {
@@ -702,10 +743,17 @@ func mustParseURLs(us []string) *URLPrefix {
 		bus[i] = &backendURL{
 			url: pu,
 		}
+		urls[i] = pu
 	}
-	return &URLPrefix{
-		bus: bus,
+	up := &URLPrefix{}
+	if len(us) == 1 {
+		up.vOriginal = us[0]
+	} else {
+		up.vOriginal = us
 	}
+	up.bus.Store(&bus)
+	up.busOriginal = urls
+	return up
 }
 
 func intp(n int) *int {

app/vmauth/main.go

@@ -13,6 +13,7 @@ import (
 	"net/textproto"
 	"net/url"
 	"os"
+	"slices"
 	"strings"
 	"sync"
 	"time"
@@ -160,20 +161,12 @@ func processUserRequest(w http.ResponseWriter, r *http.Request, ui *UserInfo) {
 		if err := ui.beginConcurrencyLimit(); err != nil {
 			handleConcurrencyLimitError(w, r, err)
 			<-concurrencyLimitCh
-
-			// Requests failed because of concurrency limit must be counted as errors,
-			// since this usually means the backend cannot keep up with the current load.
-			ui.backendErrors.Inc()
 			return
 		}
 	default:
 		concurrentRequestsLimitReached.Inc()
 		err := fmt.Errorf("cannot serve more than -maxConcurrentRequests=%d concurrent requests", cap(concurrencyLimitCh))
 		handleConcurrencyLimitError(w, r, err)
-
-		// Requests failed because of concurrency limit must be counted as errors,
-		// since this usually means the backend cannot keep up with the current load.
-		ui.backendErrors.Inc()
 		return
 	}
 	processRequest(w, r, ui)
@@ -183,7 +176,7 @@ func processUserRequest(w http.ResponseWriter, r *http.Request, ui *UserInfo) {
 
 func processRequest(w http.ResponseWriter, r *http.Request, ui *UserInfo) {
 	u := normalizeURL(r.URL)
-	up, hc := ui.getURLPrefixAndHeaders(u)
+	up, hc := ui.getURLPrefixAndHeaders(u, r.Header)
 	isDefault := false
 	if up == nil {
 		if ui.DefaultURL == nil {
@@ -238,7 +231,14 @@ func tryProcessingRequest(w http.ResponseWriter, r *http.Request, targetURL *url
 	// This code has been copied from net/http/httputil/reverseproxy.go
 	req := sanitizeRequestHeaders(r)
 	req.URL = targetURL
-	req.Host = targetURL.Host
+
+	if req.URL.Scheme == "https" {
+		// Override req.Host only for https requests, since https server verifies hostnames during TLS handshake,
+		// so it expects the targetURL.Host in the request.
+		// There is no need in overriding the req.Host for http requests, since it is expected that backend server
+		// may properly process queries with the original req.Host.
+		req.Host = targetURL.Host
+	}
 	updateHeadersByConfig(req.Header, hc.RequestHeaders)
 	res, err := ui.httpTransport.RoundTrip(req)
 	rtb, rtbOK := req.Body.(*readTrackingBody)
@@ -271,7 +271,7 @@ func tryProcessingRequest(w http.ResponseWriter, r *http.Request, targetURL *url
 		logger.Warnf("remoteAddr: %s; requestURI: %s; retrying the request to %s because of response error: %s", remoteAddr, req.URL, targetURL, err)
 		return false
 	}
-	if hasInt(retryStatusCodes, res.StatusCode) {
+	if slices.Contains(retryStatusCodes, res.StatusCode) {
 		_ = res.Body.Close()
 		if !rtbOK || !rtb.canRetry() {
 			// If we get an error from the retry_status_codes list, but cannot execute retry,
@@ -313,15 +313,6 @@ func tryProcessingRequest(w http.ResponseWriter, r *http.Request, targetURL *url
 	return true
 }
 
-func hasInt(a []int, n int) bool {
-	for _, x := range a {
-		if x == n {
-			return true
-		}
-	}
-	return false
-}
-
 var copyBufPool bytesutil.ByteBufferPool
 
 func copyHeader(dst, src http.Header) {

app/vmauth/target_url.go

@@ -1,8 +1,10 @@
 package main
 
 import (
+	"net/http"
 	"net/url"
 	"path"
+	"slices"
 	"strings"
 )
 
@@ -49,11 +51,22 @@ func dropPrefixParts(path string, parts int) string {
 	return path
 }
 
-func (ui *UserInfo) getURLPrefixAndHeaders(u *url.URL) (*URLPrefix, HeadersConf) {
+func (ui *UserInfo) getURLPrefixAndHeaders(u *url.URL, h http.Header) (*URLPrefix, HeadersConf) {
 	for _, e := range ui.URLMaps {
-		if matchAnyRegex(e.SrcHosts, u.Host) && matchAnyRegex(e.SrcPaths, u.Path) {
-			return e.URLPrefix, e.HeadersConf
+		if !matchAnyRegex(e.SrcHosts, u.Host) {
+			continue
 		}
+		if !matchAnyRegex(e.SrcPaths, u.Path) {
+			continue
+		}
+		if !matchAnyQueryArg(e.SrcQueryArgs, u.Query()) {
+			continue
+		}
+		if !matchAnyHeader(e.SrcHeaders, h) {
+			continue
+		}
+
+		return e.URLPrefix, e.HeadersConf
 	}
 	if ui.URLPrefix != nil {
 		return ui.URLPrefix, ui.HeadersConf
@@ -73,6 +86,30 @@ func matchAnyRegex(rs []*Regex, s string) bool {
 	return false
 }
 
+func matchAnyQueryArg(qas []QueryArg, args url.Values) bool {
+	if len(qas) == 0 {
+		return true
+	}
+	for _, qa := range qas {
+		if slices.Contains(args[qa.Name], qa.Value) {
+			return true
+		}
+	}
+	return false
+}
+
+func matchAnyHeader(headers []Header, h http.Header) bool {
+	if len(headers) == 0 {
+		return true
+	}
+	for _, header := range headers {
+		if slices.Contains(h.Values(header.Name), header.Value) {
+			return true
+		}
+	}
+	return false
+}
+
 func normalizeURL(uOrig *url.URL) *url.URL {
 	u := *uOrig
 	// Prevent from attacks with using `..` in r.URL.Path

app/vmauth/target_url_test.go

@@ -4,6 +4,7 @@ import (
 	"fmt"
 	"net/url"
 	"reflect"
+	"strings"
 	"testing"
 )
 
@@ -89,19 +90,21 @@ func TestCreateTargetURLSuccess(t *testing.T) {
 			t.Fatalf("cannot parse %q: %s", requestURI, err)
 		}
 		u = normalizeURL(u)
-		up, hc := ui.getURLPrefixAndHeaders(u)
+		up, hc := ui.getURLPrefixAndHeaders(u, nil)
 		if up == nil {
 			t.Fatalf("cannot determie backend: %s", err)
 		}
-		bu := up.getLeastLoadedBackendURL()
+		bu := up.getBackendURL()
 		target := mergeURLs(bu.url, u, up.dropSrcPathPrefixParts)
 		bu.put()
 		if target.String() != expectedTarget {
 			t.Fatalf("unexpected target; got %q; want %q", target, expectedTarget)
 		}
-		headersStr := fmt.Sprintf("%q", hc.RequestHeaders)
-		if headersStr != expectedRequestHeaders {
-			t.Fatalf("unexpected request headers; got %s; want %s", headersStr, expectedRequestHeaders)
+		if s := headersToString(hc.RequestHeaders); s != expectedRequestHeaders {
+			t.Fatalf("unexpected request headers; got %q; want %q", s, expectedRequestHeaders)
+		}
+		if s := headersToString(hc.ResponseHeaders); s != expectedResponseHeaders {
+			t.Fatalf("unexpected response headers; got %q; want %q", s, expectedResponseHeaders)
 		}
 		if !reflect.DeepEqual(up.retryStatusCodes, expectedRetryStatusCodes) {
 			t.Fatalf("unexpected retryStatusCodes; got %d; want %d", up.retryStatusCodes, expectedRetryStatusCodes)
@@ -116,41 +119,55 @@ func TestCreateTargetURLSuccess(t *testing.T) {
 	// Simple routing with `url_prefix`
 	f(&UserInfo{
 		URLPrefix: mustParseURL("http://foo.bar"),
-	}, "", "http://foo.bar/.", "[]", "[]", nil, "least_loaded", 0)
+	}, "", "http://foo.bar/.", "", "", nil, "least_loaded", 0)
 	f(&UserInfo{
 		URLPrefix: mustParseURL("http://foo.bar"),
 		HeadersConf: HeadersConf{
-			RequestHeaders: []Header{{
-				Name:  "bb",
-				Value: "aaa",
-			}},
+			RequestHeaders: []Header{
+				{
+					Name:  "bb",
+					Value: "aaa",
+				},
+			},
+			ResponseHeaders: []Header{
+				{
+					Name:  "x",
+					Value: "y",
+				},
+			},
 		},
 		RetryStatusCodes:       []int{503, 501},
 		LoadBalancingPolicy:    "first_available",
 		DropSrcPathPrefixParts: intp(2),
-	}, "/a/b/c", "http://foo.bar/c", `[{"bb" "aaa"}]`, `[]`, []int{503, 501}, "first_available", 2)
+	}, "/a/b/c", "http://foo.bar/c", `bb: aaa`, `x: y`, []int{503, 501}, "first_available", 2)
 	f(&UserInfo{
 		URLPrefix: mustParseURL("http://foo.bar/federate"),
-	}, "/", "http://foo.bar/federate", "[]", "[]", nil, "least_loaded", 0)
+	}, "/", "http://foo.bar/federate", "", "", nil, "least_loaded", 0)
 	f(&UserInfo{
 		URLPrefix: mustParseURL("http://foo.bar"),
-	}, "a/b?c=d", "http://foo.bar/a/b?c=d", "[]", "[]", nil, "least_loaded", 0)
+	}, "a/b?c=d", "http://foo.bar/a/b?c=d", "", "", nil, "least_loaded", 0)
 	f(&UserInfo{
 		URLPrefix: mustParseURL("https://sss:3894/x/y"),
-	}, "/z", "https://sss:3894/x/y/z", "[]", "[]", nil, "least_loaded", 0)
+	}, "/z", "https://sss:3894/x/y/z", "", "", nil, "least_loaded", 0)
 	f(&UserInfo{
 		URLPrefix: mustParseURL("https://sss:3894/x/y"),
-	}, "/../../aaa", "https://sss:3894/x/y/aaa", "[]", "[]", nil, "least_loaded", 0)
+	}, "/../../aaa", "https://sss:3894/x/y/aaa", "", "", nil, "least_loaded", 0)
 	f(&UserInfo{
 		URLPrefix: mustParseURL("https://sss:3894/x/y"),
-	}, "/./asd/../../aaa?a=d&s=s/../d", "https://sss:3894/x/y/aaa?a=d&s=s%2F..%2Fd", "[]", "[]", nil, "least_loaded", 0)
+	}, "/./asd/../../aaa?a=d&s=s/../d", "https://sss:3894/x/y/aaa?a=d&s=s%2F..%2Fd", "", "", nil, "least_loaded", 0)
 
 	// Complex routing with `url_map`
 	ui := &UserInfo{
 		URLMaps: []URLMap{
 			{
-				SrcHosts:  getRegexs([]string{"host42"}),
-				SrcPaths:  getRegexs([]string{"/vmsingle/api/v1/query"}),
+				SrcHosts: getRegexs([]string{"host42"}),
+				SrcPaths: getRegexs([]string{"/vmsingle/api/v1/query"}),
+				SrcQueryArgs: []QueryArg{
+					{
+						Name:  "db",
+						Value: "foo",
+					},
+				},
 				URLPrefix: mustParseURL("http://vmselect/0/prometheus"),
 				HeadersConf: HeadersConf{
 					RequestHeaders: []Header{
@@ -195,12 +212,12 @@ func TestCreateTargetURLSuccess(t *testing.T) {
 		RetryStatusCodes:       []int{502},
 		DropSrcPathPrefixParts: intp(2),
 	}
-	f(ui, "http://host42/vmsingle/api/v1/query?query=up", "http://vmselect/0/prometheus/api/v1/query?query=up",
-		`[{"xx" "aa"} {"yy" "asdf"}]`, `[{"qwe" "rty"}]`, []int{503, 500, 501}, "first_available", 1)
+	f(ui, "http://host42/vmsingle/api/v1/query?query=up&db=foo", "http://vmselect/0/prometheus/api/v1/query?db=foo&query=up",
+		"xx: aa\nyy: asdf", "qwe: rty", []int{503, 500, 501}, "first_available", 1)
 	f(ui, "http://host123/vmsingle/api/v1/query?query=up", "http://default-server/v1/query?query=up",
-		`[{"bb" "aaa"}]`, `[{"x" "y"}]`, []int{502}, "least_loaded", 2)
-	f(ui, "https://foo-host/api/v1/write", "http://vminsert/0/prometheus/api/v1/write", "[]", "[]", []int{}, "least_loaded", 0)
-	f(ui, "https://foo-host/foo/bar/api/v1/query_range", "http://default-server/api/v1/query_range", `[{"bb" "aaa"}]`, `[{"x" "y"}]`, []int{502}, "least_loaded", 2)
+		"bb: aaa", "x: y", []int{502}, "least_loaded", 2)
+	f(ui, "https://foo-host/api/v1/write", "http://vminsert/0/prometheus/api/v1/write", "", "", []int{}, "least_loaded", 0)
+	f(ui, "https://foo-host/foo/bar/api/v1/query_range", "http://default-server/api/v1/query_range", "bb: aaa", "x: y", []int{502}, "least_loaded", 2)
 
 	// Complex routing regexp paths in `url_map`
 	ui = &UserInfo{
@@ -220,19 +237,19 @@ func TestCreateTargetURLSuccess(t *testing.T) {
 		},
 		URLPrefix: mustParseURL("http://default-server"),
 	}
-	f(ui, "/api/v1/query?query=up", "http://vmselect/0/prometheus/api/v1/query?query=up", "[]", "[]", nil, "least_loaded", 0)
-	f(ui, "/api/v1/query_range?query=up", "http://vmselect/0/prometheus/api/v1/query_range?query=up", "[]", "[]", nil, "least_loaded", 0)
-	f(ui, "/api/v1/label/foo/values", "http://vmselect/0/prometheus/api/v1/label/foo/values", "[]", "[]", nil, "least_loaded", 0)
-	f(ui, "/api/v1/write", "http://vminsert/0/prometheus/api/v1/write", "[]", "[]", nil, "least_loaded", 0)
-	f(ui, "/api/v1/foo/bar", "http://default-server/api/v1/foo/bar", "[]", "[]", nil, "least_loaded", 0)
-	f(ui, "https://vmui.foobar.com/a/b?c=d", "http://vmui.host:1234/vmui/a/b?c=d", "[]", "[]", nil, "least_loaded", 0)
+	f(ui, "/api/v1/query?query=up", "http://vmselect/0/prometheus/api/v1/query?query=up", "", "", nil, "least_loaded", 0)
+	f(ui, "/api/v1/query_range?query=up", "http://vmselect/0/prometheus/api/v1/query_range?query=up", "", "", nil, "least_loaded", 0)
+	f(ui, "/api/v1/label/foo/values", "http://vmselect/0/prometheus/api/v1/label/foo/values", "", "", nil, "least_loaded", 0)
+	f(ui, "/api/v1/write", "http://vminsert/0/prometheus/api/v1/write", "", "", nil, "least_loaded", 0)
+	f(ui, "/api/v1/foo/bar", "http://default-server/api/v1/foo/bar", "", "", nil, "least_loaded", 0)
+	f(ui, "https://vmui.foobar.com/a/b?c=d", "http://vmui.host:1234/vmui/a/b?c=d", "", "", nil, "least_loaded", 0)
 
 	f(&UserInfo{
 		URLPrefix: mustParseURL("http://foo.bar?extra_label=team=dev"),
-	}, "/api/v1/query", "http://foo.bar/api/v1/query?extra_label=team=dev", "[]", "[]", nil, "least_loaded", 0)
+	}, "/api/v1/query", "http://foo.bar/api/v1/query?extra_label=team=dev", "", "", nil, "least_loaded", 0)
 	f(&UserInfo{
 		URLPrefix: mustParseURL("http://foo.bar?extra_label=team=mobile"),
-	}, "/api/v1/query?extra_label=team=dev", "http://foo.bar/api/v1/query?extra_label=team%3Dmobile", "[]", "[]", nil, "least_loaded", 0)
+	}, "/api/v1/query?extra_label=team=dev", "http://foo.bar/api/v1/query?extra_label=team%3Dmobile", "", "", nil, "least_loaded", 0)
 }
 
 func TestCreateTargetURLFailure(t *testing.T) {
@@ -243,7 +260,7 @@ func TestCreateTargetURLFailure(t *testing.T) {
 			t.Fatalf("cannot parse %q: %s", requestURI, err)
 		}
 		u = normalizeURL(u)
-		up, hc := ui.getURLPrefixAndHeaders(u)
+		up, hc := ui.getURLPrefixAndHeaders(u, nil)
 		if up != nil {
 			t.Fatalf("unexpected non-empty up=%#v", up)
 		}
@@ -264,3 +281,11 @@ func TestCreateTargetURLFailure(t *testing.T) {
 		},
 	}, "/api/v1/write")
 }
+
+func headersToString(hs []Header) string {
+	a := make([]string, len(hs))
+	for i, h := range hs {
+		a[i] = fmt.Sprintf("%s: %s", h.Name, h.Value)
+	}
+	return strings.Join(a, "\n")
+}

app/vmctl/flags.go

@@ -40,6 +40,11 @@ const (
 	vmSignificantFigures = "vm-significant-figures"
 	vmRoundDigits        = "vm-round-digits"
 	vmDisableProgressBar = "vm-disable-progress-bar"
+	vmCertFile           = "vm-cert-file"
+	vmKeyFile            = "vm-key-file"
+	vmCAFile             = "vm-CA-file"
+	vmServerName         = "vm-server-name"
+	vmInsecureSkipVerify = "vm-insecure-skip-verify"
 
 	// also used in vm-native
 	vmExtraLabel = "vm-extra-label"
@@ -119,6 +124,27 @@ var (
 			Name:  vmDisableProgressBar,
 			Usage: "Whether to disable progress bar per each worker during the import.",
 		},
+		&cli.StringFlag{
+			Name:  vmCertFile,
+			Usage: "Optional path to client-side TLS certificate file to use when connecting to '--vmAddr'",
+		},
+		&cli.StringFlag{
+			Name:  vmKeyFile,
+			Usage: "Optional path to client-side TLS key to use when connecting to '--vmAddr'",
+		},
+		&cli.StringFlag{
+			Name:  vmCAFile,
+			Usage: "Optional path to TLS CA file to use for verifying connections to '--vmAddr'. By default, system CA is used",
+		},
+		&cli.StringFlag{
+			Name:  vmServerName,
+			Usage: "Optional TLS server name to use for connections to '--vmAddr'. By default, the server name from '--vmAddr' is used",
+		},
+		&cli.BoolFlag{
+			Name:  vmInsecureSkipVerify,
+			Usage: "Whether to skip tls verification when connecting to '--vmAddr'",
+			Value: false,
+		},
 	}
 )
 
@@ -210,7 +236,7 @@ var (
 		},
 		&cli.StringFlag{
 			Name:  otsdbServerName,
-			Usage: "Optional TLS server name to use for connections to -otsdb-addr. By default, the server name from otsdbAddr is used",
+			Usage: "Optional TLS server name to use for connections to -otsdb-addr. By default, the server name from -otsdb-addr is used",
 		},
 		&cli.BoolFlag{
 			Name:  otsdbInsecureSkipVerify,
@@ -387,6 +413,10 @@ const (
 	vmNativeSrcPassword           = "vm-native-src-password"
 	vmNativeSrcHeaders            = "vm-native-src-headers"
 	vmNativeSrcBearerToken        = "vm-native-src-bearer-token"
+	vmNativeSrcCertFile           = "vm-native-src-cert-file"
+	vmNativeSrcKeyFile            = "vm-native-src-key-file"
+	vmNativeSrcCAFile             = "vm-native-src-ca-file"
+	vmNativeSrcServerName         = "vm-native-src-server-name"
 	vmNativeSrcInsecureSkipVerify = "vm-native-src-insecure-skip-verify"
 
 	vmNativeDstAddr               = "vm-native-dst-addr"
@@ -394,6 +424,10 @@ const (
 	vmNativeDstPassword           = "vm-native-dst-password"
 	vmNativeDstHeaders            = "vm-native-dst-headers"
 	vmNativeDstBearerToken        = "vm-native-dst-bearer-token"
+	vmNativeDstCertFile           = "vm-native-dst-cert-file"
+	vmNativeDstKeyFile            = "vm-native-dst-key-file"
+	vmNativeDstCAFile             = "vm-native-dst-ca-file"
+	vmNativeDstServerName         = "vm-native-dst-server-name"
 	vmNativeDstInsecureSkipVerify = "vm-native-dst-insecure-skip-verify"
 )
 
@@ -458,6 +492,28 @@ var (
 			Name:  vmNativeSrcBearerToken,
 			Usage: "Optional bearer auth token to use for the corresponding `--vm-native-src-addr`",
 		},
+		&cli.StringFlag{
+			Name:  vmNativeSrcCertFile,
+			Usage: "Optional path to client-side TLS certificate file to use when connecting to `--vm-native-src-addr`",
+		},
+		&cli.StringFlag{
+			Name:  vmNativeSrcKeyFile,
+			Usage: "Optional path to client-side TLS key to use when connecting to `--vm-native-src-addr`",
+		},
+		&cli.StringFlag{
+			Name:  vmNativeSrcCAFile,
+			Usage: "Optional path to TLS CA file to use for verifying connections to `--vm-native-src-addr`. By default, system CA is used",
+		},
+		&cli.StringFlag{
+			Name:  vmNativeSrcServerName,
+			Usage: "Optional TLS server name to use for connections to `--vm-native-src-addr`. By default, the server name from `--vm-native-src-addr` is used",
+		},
+		&cli.BoolFlag{
+			Name:  vmNativeSrcInsecureSkipVerify,
+			Usage: "Whether to skip TLS certificate verification when connecting to `--vm-native-src-addr`",
+			Value: false,
+		},
+
 		&cli.StringFlag{
 			Name: vmNativeDstAddr,
 			Usage: "VictoriaMetrics address to perform import to. \n" +
@@ -485,6 +541,28 @@ var (
 			Name:  vmNativeDstBearerToken,
 			Usage: "Optional bearer auth token to use for the corresponding `--vm-native-dst-addr`",
 		},
+		&cli.StringFlag{
+			Name:  vmNativeDstCertFile,
+			Usage: "Optional path to client-side TLS certificate file to use when connecting to `--vm-native-dst-addr`",
+		},
+		&cli.StringFlag{
+			Name:  vmNativeDstKeyFile,
+			Usage: "Optional path to client-side TLS key to use when connecting to `--vm-native-dst-addr`",
+		},
+		&cli.StringFlag{
+			Name:  vmNativeDstCAFile,
+			Usage: "Optional path to TLS CA file to use for verifying connections to `--vm-native-dst-addr`. By default, system CA is used",
+		},
+		&cli.StringFlag{
+			Name:  vmNativeDstServerName,
+			Usage: "Optional TLS server name to use for connections to `--vm-native-dst-addr`. By default, the server name from `--vm-native-dst-addr` is used",
+		},
+		&cli.BoolFlag{
+			Name:  vmNativeDstInsecureSkipVerify,
+			Usage: "Whether to skip TLS certificate verification when connecting to `--vm-native-dst-addr`",
+			Value: false,
+		},
+
 		&cli.StringSliceFlag{
 			Name:  vmExtraLabel,
 			Value: nil,
@@ -520,16 +598,6 @@ var (
 				"Non-binary export/import API is less efficient, but supports deduplication if it is configured on vm-native-src-addr side.",
 			Value: false,
 		},
-		&cli.BoolFlag{
-			Name:  vmNativeSrcInsecureSkipVerify,
-			Usage: "Whether to skip TLS certificate verification when connecting to the source address",
-			Value: false,
-		},
-		&cli.BoolFlag{
-			Name:  vmNativeDstInsecureSkipVerify,
-			Usage: "Whether to skip TLS certificate verification when connecting to the destination address",
-			Value: false,
-		},
 	}
 )
 

app/vmctl/main.go

@@ -2,7 +2,6 @@ package main
 
 import (
 	"context"
-	"crypto/tls"
 	"fmt"
 	"log"
 	"net/http"
@@ -58,7 +57,7 @@ func main() {
 					insecureSkipVerify := c.Bool(otsdbInsecureSkipVerify)
 					addr := c.String(otsdbAddr)
 
-					tr, err := httputils.Transport(addr, certFile, caFile, keyFile, serverName, insecureSkipVerify)
+					tr, err := httputils.Transport(addr, certFile, keyFile, caFile, serverName, insecureSkipVerify)
 					if err != nil {
 						return fmt.Errorf("failed to create Transport: %s", err)
 					}
@@ -78,7 +77,10 @@ func main() {
 						return fmt.Errorf("failed to create opentsdb client: %s", err)
 					}
 
-					vmCfg := initConfigVM(c)
+					vmCfg, err := initConfigVM(c)
+					if err != nil {
+						return fmt.Errorf("failed to init VM configuration: %s", err)
+					}
 					// disable progress bars since openTSDB implementation
 					// does not use progress bar pool
 					vmCfg.DisableProgressBar = true
@@ -105,7 +107,7 @@ func main() {
 					serverName := c.String(influxServerName)
 					insecureSkipVerify := c.Bool(influxInsecureSkipVerify)
 
-					tc, err := httputils.TLSConfig(certFile, caFile, keyFile, serverName, insecureSkipVerify)
+					tc, err := httputils.TLSConfig(certFile, keyFile, caFile, serverName, insecureSkipVerify)
 					if err != nil {
 						return fmt.Errorf("failed to create TLS Config: %s", err)
 					}
@@ -130,7 +132,10 @@ func main() {
 						return fmt.Errorf("failed to create influx client: %s", err)
 					}
 
-					vmCfg := initConfigVM(c)
+					vmCfg, err := initConfigVM(c)
+					if err != nil {
+						return fmt.Errorf("failed to init VM configuration: %s", err)
+					}
 					importer, err = vm.NewImporter(ctx, vmCfg)
 					if err != nil {
 						return fmt.Errorf("failed to create VM importer: %s", err)
@@ -153,28 +158,42 @@ func main() {
 				Usage: "Migrate time series via Prometheus remote-read protocol",
 				Flags: mergeFlags(globalFlags, remoteReadFlags, vmFlags),
 				Action: func(c *cli.Context) error {
+					fmt.Println("Remote-read import mode")
+
+					addr := c.String(remoteReadSrcAddr)
+
+					// create TLS config
+					certFile := c.String(remoteReadCertFile)
+					keyFile := c.String(remoteReadKeyFile)
+					caFile := c.String(remoteReadCAFile)
+					serverName := c.String(remoteReadServerName)
+					insecureSkipVerify := c.Bool(remoteReadInsecureSkipVerify)
+
+					tr, err := httputils.Transport(addr, certFile, keyFile, caFile, serverName, insecureSkipVerify)
+					if err != nil {
+						return fmt.Errorf("failed to create transport: %s", err)
+					}
+
 					rr, err := remoteread.NewClient(remoteread.Config{
-						Addr:               c.String(remoteReadSrcAddr),
-						Username:           c.String(remoteReadUser),
-						Password:           c.String(remoteReadPassword),
-						Timeout:            c.Duration(remoteReadHTTPTimeout),
-						UseStream:          c.Bool(remoteReadUseStream),
-						Headers:            c.String(remoteReadHeaders),
-						LabelName:          c.String(remoteReadFilterLabel),
-						LabelValue:         c.String(remoteReadFilterLabelValue),
-						CertFile:           c.String(remoteReadCertFile),
-						KeyFile:            c.String(remoteReadKeyFile),
-						CAFile:             c.String(remoteReadCAFile),
-						ServerName:         c.String(remoteReadServerName),
-						InsecureSkipVerify: c.Bool(remoteReadInsecureSkipVerify),
-						DisablePathAppend:  c.Bool(remoteReadDisablePathAppend),
+						Addr:              addr,
+						Transport:         tr,
+						Username:          c.String(remoteReadUser),
+						Password:          c.String(remoteReadPassword),
+						Timeout:           c.Duration(remoteReadHTTPTimeout),
+						UseStream:         c.Bool(remoteReadUseStream),
+						Headers:           c.String(remoteReadHeaders),
+						LabelName:         c.String(remoteReadFilterLabel),
+						LabelValue:        c.String(remoteReadFilterLabelValue),
+						DisablePathAppend: c.Bool(remoteReadDisablePathAppend),
 					})
 					if err != nil {
 						return fmt.Errorf("error create remote read client: %s", err)
 					}
 
-					vmCfg := initConfigVM(c)
-
+					vmCfg, err := initConfigVM(c)
+					if err != nil {
+						return fmt.Errorf("failed to init VM configuration: %s", err)
+					}
 					importer, err := vm.NewImporter(ctx, vmCfg)
 					if err != nil {
 						return fmt.Errorf("failed to create VM importer: %s", err)
@@ -203,7 +222,10 @@ func main() {
 				Action: func(c *cli.Context) error {
 					fmt.Println("Prometheus import mode")
 
-					vmCfg := initConfigVM(c)
+					vmCfg, err := initConfigVM(c)
+					if err != nil {
+						return fmt.Errorf("failed to init VM configuration: %s", err)
+					}
 					importer, err = vm.NewImporter(ctx, vmCfg)
 					if err != nil {
 						return fmt.Errorf("failed to create VM importer: %s", err)
@@ -245,7 +267,6 @@ func main() {
 
 					var srcExtraLabels []string
 					srcAddr := strings.Trim(c.String(vmNativeSrcAddr), "/")
-					srcInsecureSkipVerify := c.Bool(vmNativeSrcInsecureSkipVerify)
 					srcAuthConfig, err := auth.Generate(
 						auth.WithBasicAuth(c.String(vmNativeSrcUser), c.String(vmNativeSrcPassword)),
 						auth.WithBearer(c.String(vmNativeSrcBearerToken)),
@@ -253,16 +274,26 @@ func main() {
 					if err != nil {
 						return fmt.Errorf("error initilize auth config for source: %s", srcAddr)
 					}
+
+					// create TLS config
+					srcCertFile := c.String(vmNativeSrcCertFile)
+					srcKeyFile := c.String(vmNativeSrcKeyFile)
+					srcCAFile := c.String(vmNativeSrcCAFile)
+					srcServerName := c.String(vmNativeSrcServerName)
+					srcInsecureSkipVerify := c.Bool(vmNativeSrcInsecureSkipVerify)
+
+					srcTC, err := httputils.TLSConfig(srcCertFile, srcKeyFile, srcCAFile, srcServerName, srcInsecureSkipVerify)
+					if err != nil {
+						return fmt.Errorf("failed to create TLS Config: %s", err)
+					}
+
 					srcHTTPClient := &http.Client{Transport: &http.Transport{
 						DisableKeepAlives: disableKeepAlive,
-						TLSClientConfig: &tls.Config{
-							InsecureSkipVerify: srcInsecureSkipVerify,
-						},
+						TLSClientConfig:   srcTC,
 					}}
 
 					dstAddr := strings.Trim(c.String(vmNativeDstAddr), "/")
 					dstExtraLabels := c.StringSlice(vmExtraLabel)
-					dstInsecureSkipVerify := c.Bool(vmNativeDstInsecureSkipVerify)
 					dstAuthConfig, err := auth.Generate(
 						auth.WithBasicAuth(c.String(vmNativeDstUser), c.String(vmNativeDstPassword)),
 						auth.WithBearer(c.String(vmNativeDstBearerToken)),
@@ -270,11 +301,22 @@ func main() {
 					if err != nil {
 						return fmt.Errorf("error initilize auth config for destination: %s", dstAddr)
 					}
+
+					// create TLS config
+					dstCertFile := c.String(vmNativeDstCertFile)
+					dstKeyFile := c.String(vmNativeDstKeyFile)
+					dstCAFile := c.String(vmNativeDstCAFile)
+					dstServerName := c.String(vmNativeDstServerName)
+					dstInsecureSkipVerify := c.Bool(vmNativeDstInsecureSkipVerify)
+
+					dstTC, err := httputils.TLSConfig(dstCertFile, dstKeyFile, dstCAFile, dstServerName, dstInsecureSkipVerify)
+					if err != nil {
+						return fmt.Errorf("failed to create TLS Config: %s", err)
+					}
+
 					dstHTTPClient := &http.Client{Transport: &http.Transport{
 						DisableKeepAlives: disableKeepAlive,
-						TLSClientConfig: &tls.Config{
-							InsecureSkipVerify: dstInsecureSkipVerify,
-						},
+						TLSClientConfig:   dstTC,
 					}}
 
 					p := vmNativeProcessor{
@@ -330,14 +372,15 @@ func main() {
 					if err != nil {
 						return cli.Exit(fmt.Errorf("cannot open exported block at path=%q err=%w", blockPath, err), 1)
 					}
-					var blocksCount uint64
-					if err := stream.Parse(f, isBlockGzipped, func(block *stream.Block) error {
-						atomic.AddUint64(&blocksCount, 1)
+					defer f.Close()
+					var blocksCount atomic.Uint64
+					if err := stream.Parse(f, isBlockGzipped, func(_ *stream.Block) error {
+						blocksCount.Add(1)
 						return nil
 					}); err != nil {
-						return cli.Exit(fmt.Errorf("cannot parse block at path=%q, blocksCount=%d, err=%w", blockPath, blocksCount, err), 1)
+						return cli.Exit(fmt.Errorf("cannot parse block at path=%q, blocksCount=%d, err=%w", blockPath, blocksCount.Load(), err), 1)
 					}
-					log.Printf("successfully verified block at path=%q, blockCount=%d", blockPath, blocksCount)
+					log.Printf("successfully verified block at path=%q, blockCount=%d", blockPath, blocksCount.Load())
 					return nil
 				},
 			},
@@ -362,9 +405,24 @@ func main() {
 	log.Printf("Total time: %v", time.Since(start))
 }
 
-func initConfigVM(c *cli.Context) vm.Config {
+func initConfigVM(c *cli.Context) (vm.Config, error) {
+	addr := c.String(vmAddr)
+
+	// create Transport with given TLS config
+	certFile := c.String(vmCertFile)
+	keyFile := c.String(vmKeyFile)
+	caFile := c.String(vmCAFile)
+	serverName := c.String(vmServerName)
+	insecureSkipVerify := c.Bool(vmInsecureSkipVerify)
+
+	tr, err := httputils.Transport(addr, certFile, keyFile, caFile, serverName, insecureSkipVerify)
+	if err != nil {
+		return vm.Config{}, fmt.Errorf("failed to create Transport: %s", err)
+	}
+
 	return vm.Config{
-		Addr:               c.String(vmAddr),
+		Addr:               addr,
+		Transport:          tr,
 		User:               c.String(vmUser),
 		Password:           c.String(vmPassword),
 		Concurrency:        uint8(c.Int(vmConcurrency)),
@@ -376,5 +434,5 @@ func initConfigVM(c *cli.Context) vm.Config {
 		ExtraLabels:        c.StringSlice(vmExtraLabel),
 		RateLimit:          c.Int64(vmRateLimit),
 		DisableProgressBar: c.Bool(vmDisableProgressBar),
-	}
+	}, nil
 }

app/vmctl/native/client.go

@@ -6,6 +6,7 @@ import (
 	"fmt"
 	"io"
 	"net/http"
+	"time"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/auth"
 )
@@ -34,7 +35,7 @@ type Response struct {
 }
 
 // Explore finds metric names by provided filter from api/v1/label/__name__/values
-func (c *Client) Explore(ctx context.Context, f Filter, tenantID string) ([]string, error) {
+func (c *Client) Explore(ctx context.Context, f Filter, tenantID string, start, end time.Time) ([]string, error) {
 	url := fmt.Sprintf("%s/%s", c.Addr, nativeMetricNamesAddr)
 	if tenantID != "" {
 		url = fmt.Sprintf("%s/select/%s/prometheus/%s", c.Addr, tenantID, nativeMetricNamesAddr)
@@ -45,12 +46,8 @@ func (c *Client) Explore(ctx context.Context, f Filter, tenantID string) ([]stri
 	}
 
 	params := req.URL.Query()
-	if f.TimeStart != "" {
-		params.Set("start", f.TimeStart)
-	}
-	if f.TimeEnd != "" {
-		params.Set("end", f.TimeEnd)
-	}
+	params.Set("start", start.Format(time.RFC3339))
+	params.Set("end", end.Format(time.RFC3339))
 	params.Set("match[]", f.Match)
 	req.URL.RawQuery = params.Encode()
 
@@ -63,11 +60,7 @@ func (c *Client) Explore(ctx context.Context, f Filter, tenantID string) ([]stri
 	if err := json.NewDecoder(resp.Body).Decode(&response); err != nil {
 		return nil, fmt.Errorf("cannot decode series response: %s", err)
 	}
-
-	if err := resp.Body.Close(); err != nil {
-		return nil, fmt.Errorf("cannot close series response body: %s", err)
-	}
-	return response.MetricNames, nil
+	return response.MetricNames, resp.Body.Close()
 }
 
 // ImportPipe uses pipe reader in request to process data

app/vmctl/remote_read_test.go

@@ -2,6 +2,7 @@ package main
 
 import (
 	"context"
+	"net/http"
 	"testing"
 	"time"
 
@@ -61,7 +62,8 @@ func TestRemoteRead(t *testing.T) {
 		{
 			name:             "step month on month time range",
 			remoteReadConfig: remoteread.Config{Addr: "", LabelName: "__name__", LabelValue: ".*"},
-			vmCfg:            vm.Config{Addr: "", Concurrency: 1, DisableProgressBar: true},
+			vmCfg: vm.Config{Addr: "", Concurrency: 1, DisableProgressBar: true,
+				Transport: http.DefaultTransport.(*http.Transport)},
 			start:            "2022-09-26T11:23:05+02:00",
 			end:              "2022-11-26T11:24:05+02:00",
 			numOfSamples:     2,

app/vmctl/remoteread/remoteread.go

@@ -13,7 +13,6 @@ import (
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/vm"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/httputils"
 	"github.com/gogo/protobuf/proto"
 	"github.com/golang/snappy"
 	"github.com/prometheus/prometheus/prompb"
@@ -46,6 +45,8 @@ type Client struct {
 type Config struct {
 	// Addr of remote storage
 	Addr string
+	// Transport allows specifying custom http.Transport
+	Transport *http.Transport
 	// DisablePathAppend disable automatic appending of the remote read path
 	DisablePathAppend bool
 	// Timeout defines timeout for HTTP requests
@@ -64,15 +65,6 @@ type Config struct {
 	// LabelName, LabelValue stands for label=~value pair used for read requests.
 	// Is optional.
 	LabelName, LabelValue string
-
-	// Optional cert file, key file, CA file and server name for client side TLS configuration
-	CertFile   string
-	KeyFile    string
-	CAFile     string
-	ServerName string
-
-	// TLSSkipVerify defines whether to skip TLS certificate verification when connecting to the remote read address.
-	InsecureSkipVerify bool
 }
 
 // Filter defines a list of filters applied to requested data
@@ -110,16 +102,13 @@ func NewClient(cfg Config) (*Client, error) {
 		}
 	}
 
-	tr, err := httputils.Transport(cfg.Addr, cfg.CertFile, cfg.KeyFile, cfg.CAFile, cfg.ServerName, cfg.InsecureSkipVerify)
-	if err != nil {
-		return nil, fmt.Errorf("failed to create transport: %s", err)
+	client := &http.Client{Timeout: cfg.Timeout}
+	if cfg.Transport != nil {
+		client.Transport = cfg.Transport
 	}
 
 	c := &Client{
-		c: &http.Client{
-			Timeout:   cfg.Timeout,
-			Transport: tr,
-		},
+		c:                 client,
 		addr:              strings.TrimSuffix(cfg.Addr, "/"),
 		disablePathAppend: cfg.DisablePathAppend,
 		user:              cfg.Username,
@@ -182,7 +171,7 @@ func (c *Client) fetch(ctx context.Context, data []byte, streamCb StreamCallback
 	if c.disablePathAppend {
 		u = c.addr
 	}
-	req, err := http.NewRequest(http.MethodPost, u, r)
+	req, err := http.NewRequestWithContext(ctx, http.MethodPost, u, r)
 	if err != nil {
 		return fmt.Errorf("failed to create new HTTP request: %w", err)
 	}
@@ -195,7 +184,7 @@ func (c *Client) fetch(ctx context.Context, data []byte, streamCb StreamCallback
 	}
 	req.Header.Set("X-Prometheus-Remote-Read-Version", "0.1.0")
 
-	resp, err := c.do(req.WithContext(ctx))
+	resp, err := c.do(req)
 	if err != nil {
 		return fmt.Errorf("error while sending request to %s: %w; Data len %d(%d)",
 			req.URL.Redacted(), err, len(data), r.Size())

app/vmctl/utils/time.go

@@ -1,4 +1,4 @@
-package main
+package utils
 
 import (
 	"fmt"
@@ -13,12 +13,13 @@ const (
 	maxTimeMsecs = int64(1<<63-1) / 1e6
 )
 
-func parseTime(s string) (time.Time, error) {
-	secs, err := promutils.ParseTime(s)
+// ParseTime parses time in s string and returns time.Time object
+// if parse correctly or error if not
+func ParseTime(s string) (time.Time, error) {
+	msecs, err := promutils.ParseTimeMsec(s)
 	if err != nil {
 		return time.Time{}, fmt.Errorf("cannot parse %s: %w", s, err)
 	}
-	msecs := int64(secs * 1e3)
 	if msecs < minTimeMsecs {
 		msecs = 0
 	}

app/vmctl/utils/time_test.go

@@ -1,4 +1,4 @@
-package main
+package utils
 
 import (
 	"testing"
@@ -165,7 +165,7 @@ func TestGetTime(t *testing.T) {
 	}
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
-			got, err := parseTime(tt.s)
+			got, err := ParseTime(tt.s)
 			if (err != nil) != tt.wantErr {
 				t.Errorf("ParseTime() error = %v, wantErr %v", err, tt.wantErr)
 				return

app/vmctl/vm/vm.go

@@ -26,6 +26,8 @@ type Config struct {
 	//   --httpListenAddr value for single node version
 	//   --httpListenAddr value of vmselect  component for cluster version
 	Addr string
+	// Transport allows specifying custom http.Transport
+	Transport *http.Transport
 	// Concurrency defines number of worker
 	// performing the import requests concurrently
 	Concurrency uint8
@@ -62,6 +64,7 @@ type Config struct {
 // see https://docs.victoriametrics.com/#how-to-import-time-series-data
 type Importer struct {
 	addr       string
+	client     *http.Client
 	importPath string
 	compress   bool
 	user       string
@@ -128,8 +131,14 @@ func NewImporter(ctx context.Context, cfg Config) (*Importer, error) {
 		return nil, err
 	}
 
+	client := &http.Client{}
+	if cfg.Transport != nil {
+		client.Transport = cfg.Transport
+	}
+
 	im := &Importer{
 		addr:       addr,
+		client:     client,
 		importPath: importPath,
 		compress:   cfg.Compress,
 		user:       cfg.User,
@@ -291,7 +300,7 @@ func (im *Importer) Ping() error {
 	if im.user != "" {
 		req.SetBasicAuth(im.user, im.password)
 	}
-	resp, err := http.DefaultClient.Do(req)
+	resp, err := im.client.Do(req)
 	if err != nil {
 		return err
 	}
@@ -321,7 +330,7 @@ func (im *Importer) Import(tsBatch []*TimeSeries) error {
 
 	errCh := make(chan error)
 	go func() {
-		errCh <- do(req)
+		errCh <- im.do(req)
 		close(errCh)
 	}()
 
@@ -375,8 +384,8 @@ func (im *Importer) Import(tsBatch []*TimeSeries) error {
 // ErrBadRequest represents bad request error.
 var ErrBadRequest = errors.New("bad request")
 
-func do(req *http.Request) error {
-	resp, err := http.DefaultClient.Do(req)
+func (im *Importer) do(req *http.Request) error {
+	resp, err := im.client.Do(req)
 	if err != nil {
 		return fmt.Errorf("unexpected error when performing request: %s", err)
 	}

app/vmctl/vm_native.go

@@ -16,6 +16,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/limiter"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/native"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/stepper"
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/utils"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/vm"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/searchutils"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
@@ -53,14 +54,14 @@ func (p *vmNativeProcessor) run(ctx context.Context) error {
 		startTime: time.Now(),
 	}
 
-	start, err := parseTime(p.filter.TimeStart)
+	start, err := utils.ParseTime(p.filter.TimeStart)
 	if err != nil {
 		return fmt.Errorf("failed to parse %s, provided: %s, error: %w", vmNativeFilterTimeStart, p.filter.TimeStart, err)
 	}
 
 	end := time.Now().In(start.Location())
 	if p.filter.TimeEnd != "" {
-		end, err = parseTime(p.filter.TimeEnd)
+		end, err = utils.ParseTime(p.filter.TimeEnd)
 		if err != nil {
 			return fmt.Errorf("failed to parse %s, provided: %s, error: %w", vmNativeFilterTimeEnd, p.filter.TimeEnd, err)
 		}
@@ -174,28 +175,29 @@ func (p *vmNativeProcessor) runBackfilling(ctx context.Context, tenantID string,
 		dstURL = fmt.Sprintf("%s/insert/%s/prometheus/%s", p.dst.Addr, tenantID, importAddr)
 	}
 
-	barPrefix := "Requests to make"
 	initMessage := "Initing import process from %q to %q with filter %s"
 	initParams := []interface{}{srcURL, dstURL, p.filter.String()}
 	if p.interCluster {
-		barPrefix = fmt.Sprintf("Requests to make for tenant %s", tenantID)
 		initMessage = "Initing import process from %q to %q with filter %s for tenant %s"
 		initParams = []interface{}{srcURL, dstURL, p.filter.String(), tenantID}
 	}
 
 	fmt.Println("") // extra line for better output formatting
 	log.Printf(initMessage, initParams...)
+	if len(ranges) > 1 {
+		log.Printf("Selected time range will be split into %d ranges according to %q step", len(ranges), p.filter.Chunk)
+	}
 
 	var foundSeriesMsg string
-
-	metrics := []string{p.filter.Match}
+	var requestsToMake int
+	var metrics = map[string][][]time.Time{
+		"": ranges,
+	}
 	if !p.disablePerMetricRequests {
-		log.Printf("Exploring metrics...")
-		metrics, err = p.src.Explore(ctx, p.filter, tenantID)
+		metrics, err = p.explore(ctx, p.src, tenantID, ranges, silent)
 		if err != nil {
-			return fmt.Errorf("cannot get metrics from source %s: %w", p.src.Addr, err)
+			return fmt.Errorf("failed to explore metric names: %s", err)
 		}
-
 		if len(metrics) == 0 {
 			errMsg := "no metrics found"
 			if tenantID != "" {
@@ -204,7 +206,10 @@ func (p *vmNativeProcessor) runBackfilling(ctx context.Context, tenantID string,
 			log.Println(errMsg)
 			return nil
 		}
-		foundSeriesMsg = fmt.Sprintf("Found %d metrics to import", len(metrics))
+		for _, m := range metrics {
+			requestsToMake += len(m)
+		}
+		foundSeriesMsg = fmt.Sprintf("Found %d unique metric names to import. Total import/export requests to make %d", len(metrics), requestsToMake)
 	}
 
 	if !p.interCluster {
@@ -218,15 +223,13 @@ func (p *vmNativeProcessor) runBackfilling(ctx context.Context, tenantID string,
 		log.Print(foundSeriesMsg)
 	}
 
-	processingMsg := fmt.Sprintf("Requests to make: %d", len(metrics)*len(ranges))
-	if len(ranges) > 1 {
-		processingMsg = fmt.Sprintf("Selected time range will be split into %d ranges according to %q step. %s", len(ranges), p.filter.Chunk, processingMsg)
-	}
-	log.Print(processingMsg)
-
 	var bar *pb.ProgressBar
+	barPrefix := "Requests to make"
+	if p.interCluster {
+		barPrefix = fmt.Sprintf("Requests to make for tenant %s", tenantID)
+	}
 	if !silent {
-		bar = barpool.NewSingleProgress(fmt.Sprintf(nativeWithBackoffTpl, barPrefix), len(metrics)*len(ranges))
+		bar = barpool.NewSingleProgress(fmt.Sprintf(nativeWithBackoffTpl, barPrefix), requestsToMake)
 		if p.disablePerMetricRequests {
 			bar = barpool.NewSingleProgress(nativeSingleProcessTpl, 0)
 		}
@@ -262,20 +265,19 @@ func (p *vmNativeProcessor) runBackfilling(ctx context.Context, tenantID string,
 	}
 
 	// any error breaks the import
-	for _, s := range metrics {
-
-		match, err := buildMatchWithFilter(p.filter.Match, s)
+	for mName, mRanges := range metrics {
+		match, err := buildMatchWithFilter(p.filter.Match, mName)
 		if err != nil {
-			logger.Errorf("failed to build export filters: %s", err)
+			logger.Errorf("failed to build filter %q for metric name %q: %s", p.filter.Match, mName, err)
 			continue
 		}
 
-		for _, times := range ranges {
+		for _, times := range mRanges {
 			select {
 			case <-ctx.Done():
 				return fmt.Errorf("context canceled")
 			case infErr := <-errCh:
-				return fmt.Errorf("native error: %s", infErr)
+				return fmt.Errorf("export/import error: %s", infErr)
 			case filterCh <- native.Filter{
 				Match:     match,
 				TimeStart: times[0].Format(time.RFC3339),
@@ -296,6 +298,32 @@ func (p *vmNativeProcessor) runBackfilling(ctx context.Context, tenantID string,
 	return nil
 }
 
+func (p *vmNativeProcessor) explore(ctx context.Context, src *native.Client, tenantID string, ranges [][]time.Time, silent bool) (map[string][][]time.Time, error) {
+	log.Printf("Exploring metrics...")
+
+	var bar *pb.ProgressBar
+	if !silent {
+		bar = barpool.NewSingleProgress(fmt.Sprintf(nativeWithBackoffTpl, "Explore requests to make"), len(ranges))
+		bar.Start()
+		defer bar.Finish()
+	}
+
+	metrics := make(map[string][][]time.Time)
+	for _, r := range ranges {
+		ms, err := src.Explore(ctx, p.filter, tenantID, r[0], r[1])
+		if err != nil {
+			return nil, fmt.Errorf("cannot get metrics from %s on interval %v-%v: %w", src.Addr, r[0], r[1], err)
+		}
+		for i := range ms {
+			metrics[ms[i]] = append(metrics[ms[i]], r)
+		}
+		if bar != nil {
+			bar.Increment()
+		}
+	}
+	return metrics, nil
+}
+
 // stats represents client statistic
 // when processing data
 type stats struct {

app/vminsert/common/insert_ctx.go

@@ -142,7 +142,7 @@ func (ctx *InsertCtx) ApplyRelabeling() {
 // FlushBufs flushes buffered rows to the underlying storage.
 func (ctx *InsertCtx) FlushBufs() error {
 	sas := sasGlobal.Load()
-	if sas != nil && !ctx.skipStreamAggr {
+	if (sas != nil || deduplicator != nil) && !ctx.skipStreamAggr {
 		matchIdxs := matchIdxsPool.Get()
 		matchIdxs.B = ctx.streamAggrCtx.push(ctx.mrs, matchIdxs.B)
 		if !*streamAggrKeepInput {

app/vminsert/common/streamaggr.go

@@ -9,10 +9,10 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fasttime"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/procutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promrelabel"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/storage"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/streamaggr"
 	"github.com/VictoriaMetrics/metrics"
@@ -28,8 +28,12 @@ var (
 	streamAggrDropInput = flag.Bool("streamAggr.dropInput", false, "Whether to drop all the input samples after the aggregation with -streamAggr.config. "+
 		"By default, only aggregated samples are dropped, while the remaining samples are stored in the database. "+
 		"See also -streamAggr.keepInput and https://docs.victoriametrics.com/stream-aggregation.html")
-	streamAggrDedupInterval = flag.Duration("streamAggr.dedupInterval", 0, "Input samples are de-duplicated with this interval before being aggregated. "+
-		"Only the last sample per each time series per each interval is aggregated if the interval is greater than zero")
+	streamAggrDedupInterval = flag.Duration("streamAggr.dedupInterval", 0, "Input samples are de-duplicated with this interval before optional aggregation with -streamAggr.config . "+
+		"See also -streamAggr.dropInputLabels and -dedup.minScrapeInterval and https://docs.victoriametrics.com/stream-aggregation.html#deduplication")
+	streamAggrDropInputLabels = flagutil.NewArrayString("streamAggr.dropInputLabels", "An optional list of labels to drop from samples "+
+		"before stream de-duplication and aggregation . See https://docs.victoriametrics.com/stream-aggregation.html#dropping-unneeded-labels")
+	streamAggrIgnoreOldSamples = flag.Bool("streamAggr.ignoreOldSamples", false, "Whether to ignore input samples with old timestamps outside the current aggregation interval. "+
+		"See https://docs.victoriametrics.com/stream-aggregation.html#ignoring-old-samples")
 )
 
 var (
@@ -41,7 +45,8 @@ var (
 	saCfgSuccess   = metrics.NewGauge(`vminsert_streamagg_config_last_reload_successful`, nil)
 	saCfgTimestamp = metrics.NewCounter(`vminsert_streamagg_config_last_reload_success_timestamp_seconds`)
 
-	sasGlobal atomic.Pointer[streamaggr.Aggregators]
+	sasGlobal    atomic.Pointer[streamaggr.Aggregators]
+	deduplicator *streamaggr.Deduplicator
 )
 
 // CheckStreamAggrConfig checks config pointed by -stramaggr.config
@@ -49,8 +54,13 @@ func CheckStreamAggrConfig() error {
 	if *streamAggrConfig == "" {
 		return nil
 	}
-	pushNoop := func(tss []prompbmarshal.TimeSeries) {}
-	sas, err := streamaggr.LoadFromFile(*streamAggrConfig, pushNoop, *streamAggrDedupInterval)
+	pushNoop := func(_ []prompbmarshal.TimeSeries) {}
+	opts := &streamaggr.Options{
+		DedupInterval:    *streamAggrDedupInterval,
+		DropInputLabels:  *streamAggrDropInputLabels,
+		IgnoreOldSamples: *streamAggrIgnoreOldSamples,
+	}
+	sas, err := streamaggr.LoadFromFile(*streamAggrConfig, pushNoop, opts)
 	if err != nil {
 		return fmt.Errorf("error when loading -streamAggr.config=%q: %w", *streamAggrConfig, err)
 	}
@@ -65,15 +75,24 @@ func InitStreamAggr() {
 	saCfgReloaderStopCh = make(chan struct{})
 
 	if *streamAggrConfig == "" {
+		if *streamAggrDedupInterval > 0 {
+			deduplicator = streamaggr.NewDeduplicator(pushAggregateSeries, *streamAggrDedupInterval, *streamAggrDropInputLabels)
+		}
 		return
 	}
 
 	sighupCh := procutil.NewSighupChan()
 
-	sas, err := streamaggr.LoadFromFile(*streamAggrConfig, pushAggregateSeries, *streamAggrDedupInterval)
+	opts := &streamaggr.Options{
+		DedupInterval:    *streamAggrDedupInterval,
+		DropInputLabels:  *streamAggrDropInputLabels,
+		IgnoreOldSamples: *streamAggrIgnoreOldSamples,
+	}
+	sas, err := streamaggr.LoadFromFile(*streamAggrConfig, pushAggregateSeries, opts)
 	if err != nil {
 		logger.Fatalf("cannot load -streamAggr.config=%q: %s", *streamAggrConfig, err)
 	}
+
 	sasGlobal.Store(sas)
 	saCfgSuccess.Set(1)
 	saCfgTimestamp.Set(fasttime.UnixTimestamp())
@@ -97,7 +116,12 @@ func reloadStreamAggrConfig() {
 	logger.Infof("reloading -streamAggr.config=%q", *streamAggrConfig)
 	saCfgReloads.Inc()
 
-	sasNew, err := streamaggr.LoadFromFile(*streamAggrConfig, pushAggregateSeries, *streamAggrDedupInterval)
+	opts := &streamaggr.Options{
+		DedupInterval:    *streamAggrDedupInterval,
+		DropInputLabels:  *streamAggrDropInputLabels,
+		IgnoreOldSamples: *streamAggrIgnoreOldSamples,
+	}
+	sasNew, err := streamaggr.LoadFromFile(*streamAggrConfig, pushAggregateSeries, opts)
 	if err != nil {
 		saCfgSuccess.Set(0)
 		saCfgReloadErr.Inc()
@@ -124,61 +148,101 @@ func MustStopStreamAggr() {
 
 	sas := sasGlobal.Swap(nil)
 	sas.MustStop()
+
+	if deduplicator != nil {
+		deduplicator.MustStop()
+		deduplicator = nil
+	}
 }
 
 type streamAggrCtx struct {
-	mn  storage.MetricName
-	tss [1]prompbmarshal.TimeSeries
+	mn      storage.MetricName
+	tss     []prompbmarshal.TimeSeries
+	labels  []prompbmarshal.Label
+	samples []prompbmarshal.Sample
+	buf     []byte
 }
 
 func (ctx *streamAggrCtx) Reset() {
 	ctx.mn.Reset()
-	ts := &ctx.tss[0]
-	promrelabel.CleanLabels(ts.Labels)
+
+	clear(ctx.tss)
+	ctx.tss = ctx.tss[:0]
+
+	clear(ctx.labels)
+	ctx.labels = ctx.labels[:0]
+
+	ctx.samples = ctx.samples[:0]
+	ctx.buf = ctx.buf[:0]
 }
 
 func (ctx *streamAggrCtx) push(mrs []storage.MetricRow, matchIdxs []byte) []byte {
-	matchIdxs = bytesutil.ResizeNoCopyMayOverallocate(matchIdxs, len(mrs))
-	for i := 0; i < len(matchIdxs); i++ {
-		matchIdxs[i] = 0
-	}
-
 	mn := &ctx.mn
-	tss := ctx.tss[:]
-	ts := &tss[0]
-	labels := ts.Labels
-	samples := ts.Samples
-	sas := sasGlobal.Load()
-	var matchIdxsLocal []byte
-	for idx, mr := range mrs {
+	tss := ctx.tss
+	labels := ctx.labels
+	samples := ctx.samples
+	buf := ctx.buf
+
+	tssLen := len(tss)
+	for _, mr := range mrs {
 		if err := mn.UnmarshalRaw(mr.MetricNameRaw); err != nil {
 			logger.Panicf("BUG: cannot unmarshal recently marshaled MetricName: %s", err)
 		}
 
-		labels = append(labels[:0], prompbmarshal.Label{
+		labelsLen := len(labels)
+
+		bufLen := len(buf)
+		buf = append(buf, mn.MetricGroup...)
+		metricGroup := bytesutil.ToUnsafeString(buf[bufLen:])
+		labels = append(labels, prompbmarshal.Label{
 			Name:  "__name__",
-			Value: bytesutil.ToUnsafeString(mn.MetricGroup),
+			Value: metricGroup,
 		})
+
 		for _, tag := range mn.Tags {
+			bufLen = len(buf)
+			buf = append(buf, tag.Key...)
+			name := bytesutil.ToUnsafeString(buf[bufLen:])
+
+			bufLen = len(buf)
+			buf = append(buf, tag.Value...)
+			value := bytesutil.ToUnsafeString(buf[bufLen:])
 			labels = append(labels, prompbmarshal.Label{
-				Name:  bytesutil.ToUnsafeString(tag.Key),
-				Value: bytesutil.ToUnsafeString(tag.Value),
+				Name:  name,
+				Value: value,
 			})
 		}
 
-		samples = append(samples[:0], prompbmarshal.Sample{
+		samplesLen := len(samples)
+		samples = append(samples, prompbmarshal.Sample{
 			Timestamp: mr.Timestamp,
 			Value:     mr.Value,
 		})
 
-		ts.Labels = labels
-		ts.Samples = samples
-
-		matchIdxsLocal = sas.Push(tss, matchIdxsLocal)
-		if matchIdxsLocal[0] != 0 {
-			matchIdxs[idx] = 1
-		}
+		tss = append(tss, prompbmarshal.TimeSeries{
+			Labels:  labels[labelsLen:],
+			Samples: samples[samplesLen:],
+		})
 	}
+	ctx.tss = tss
+	ctx.labels = labels
+	ctx.samples = samples
+	ctx.buf = buf
+
+	tss = tss[tssLen:]
+
+	sas := sasGlobal.Load()
+	if sas != nil {
+		matchIdxs = sas.Push(tss, matchIdxs)
+	} else if deduplicator != nil {
+		matchIdxs = bytesutil.ResizeNoCopyMayOverallocate(matchIdxs, len(tss))
+		for i := range matchIdxs {
+			matchIdxs[i] = 1
+		}
+		deduplicator.Push(tss)
+	}
+
+	ctx.Reset()
 
 	return matchIdxs
 }

app/vminsert/main.go

@@ -6,7 +6,6 @@ import (
 	"fmt"
 	"net/http"
 	"strings"
-	"sync/atomic"
 	"time"
 
 	"github.com/VictoriaMetrics/metrics"
@@ -41,6 +40,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/common"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/opentelemetry/firehose"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/storage"
 )
 
@@ -101,7 +101,7 @@ func Init() {
 	if len(*opentsdbHTTPListenAddr) > 0 {
 		opentsdbhttpServer = opentsdbhttpserver.MustStart(*opentsdbHTTPListenAddr, *opentsdbHTTPUseProxyProtocol, opentsdbhttp.InsertHandler)
 	}
-	promscrape.Init(func(at *auth.Token, wr *prompbmarshal.WriteRequest) {
+	promscrape.Init(func(_ *auth.Token, wr *prompbmarshal.WriteRequest) {
 		prompush.Push(wr)
 	})
 }
@@ -163,7 +163,7 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 		path = strings.TrimSuffix(path, "/")
 	}
 	switch path {
-	case "/prometheus/api/v1/write", "/api/v1/write":
+	case "/prometheus/api/v1/write", "/api/v1/write", "/api/v1/push", "/prometheus/api/v1/push":
 		if common.HandleVMProtoServerHandshake(w, r) {
 			return true
 		}
@@ -217,14 +217,14 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 		addInfluxResponseHeaders(w)
 		influxutils.WriteDatabaseNames(w)
 		return true
-	case "/opentelemetry/api/v1/push":
+	case "/opentelemetry/api/v1/push", "/opentelemetry/v1/metrics":
 		opentelemetryPushRequests.Inc()
 		if err := opentelemetry.InsertHandler(r); err != nil {
 			opentelemetryPushErrors.Inc()
 			httpserver.Errorf(w, r, "%s", err)
 			return true
 		}
-		w.WriteHeader(http.StatusOK)
+		firehose.WriteSuccessResponse(w, r)
 		return true
 	case "/newrelic":
 		newrelicCheckRequest.Inc()
@@ -354,7 +354,7 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 		w.WriteHeader(http.StatusOK)
 		return true
 	case "/ready":
-		if rdy := atomic.LoadInt32(&promscrape.PendingScrapeConfigs); rdy > 0 {
+		if rdy := promscrape.PendingScrapeConfigs.Load(); rdy > 0 {
 			errMsg := fmt.Sprintf("waiting for scrape config to init targets, configs left: %d", rdy)
 			http.Error(w, errMsg, http.StatusTooEarly)
 		} else {
@@ -412,8 +412,8 @@ var (
 	datadogIntakeRequests   = metrics.NewCounter(`vm_http_requests_total{path="/datadog/intake", protocol="datadog"}`)
 	datadogMetadataRequests = metrics.NewCounter(`vm_http_requests_total{path="/datadog/api/v1/metadata", protocol="datadog"}`)
 
-	opentelemetryPushRequests = metrics.NewCounter(`vm_http_requests_total{path="/opentelemetry/api/v1/push", protocol="opentelemetry"}`)
-	opentelemetryPushErrors   = metrics.NewCounter(`vm_http_request_errors_total{path="/opentelemetry/api/v1/push", protocol="opentelemetry"}`)
+	opentelemetryPushRequests = metrics.NewCounter(`vm_http_requests_total{path="/opentelemetry/v1/metrics", protocol="opentelemetry"}`)
+	opentelemetryPushErrors   = metrics.NewCounter(`vm_http_request_errors_total{path="/opentelemetry/v1/metrics", protocol="opentelemetry"}`)
 
 	newrelicWriteRequests = metrics.NewCounter(`vm_http_requests_total{path="/newrelic/infra/v2/metrics/events/bulk", protocol="newrelic"}`)
 	newrelicWriteErrors   = metrics.NewCounter(`vm_http_request_errors_total{path="/newrelic/infra/v2/metrics/events/bulk", protocol="newrelic"}`)
@@ -435,12 +435,12 @@ var (
 	promscrapeConfigReloadRequests = metrics.NewCounter(`vm_http_requests_total{path="/-/reload"}`)
 
 	_ = metrics.NewGauge(`vm_metrics_with_dropped_labels_total`, func() float64 {
-		return float64(atomic.LoadUint64(&storage.MetricsWithDroppedLabels))
+		return float64(storage.MetricsWithDroppedLabels.Load())
 	})
 	_ = metrics.NewGauge(`vm_too_long_label_names_total`, func() float64 {
-		return float64(atomic.LoadUint64(&storage.TooLongLabelNames))
+		return float64(storage.TooLongLabelNames.Load())
 	})
 	_ = metrics.NewGauge(`vm_too_long_label_values_total`, func() float64 {
-		return float64(atomic.LoadUint64(&storage.TooLongLabelValues))
+		return float64(storage.TooLongLabelValues.Load())
 	})
 )

app/vminsert/opentelemetry/request_handler.go

@@ -8,6 +8,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vminsert/relabel"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
 	parserCommon "github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/common"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/opentelemetry/firehose"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/opentelemetry/stream"
 	"github.com/VictoriaMetrics/metrics"
 )
@@ -24,10 +25,15 @@ func InsertHandler(req *http.Request) error {
 		return err
 	}
 	isGzipped := req.Header.Get("Content-Encoding") == "gzip"
+	var processBody func([]byte) ([]byte, error)
 	if req.Header.Get("Content-Type") == "application/json" {
-		return fmt.Errorf("json encoding isn't supported for opentelemetry format. Use protobuf encoding")
+		if req.Header.Get("X-Amz-Firehose-Protocol-Version") != "" {
+			processBody = firehose.ProcessRequestBody
+		} else {
+			return fmt.Errorf("json encoding isn't supported for opentelemetry format. Use protobuf encoding")
+		}
 	}
-	return stream.ParseStream(req.Body, isGzipped, func(tss []prompbmarshal.TimeSeries) error {
+	return stream.ParseStream(req.Body, isGzipped, processBody, func(tss []prompbmarshal.TimeSeries) error {
 		return insertRows(tss, extraLabels)
 	})
 }

app/vmselect/graphite/eval.go

@@ -160,7 +160,7 @@ func newNextSeriesForSearchQuery(ec *evalConfig, sq *storage.SearchQuery, expr g
 	seriesCh := make(chan *series, cgroup.AvailableCPUs())
 	errCh := make(chan error, 1)
 	go func() {
-		err := rss.RunParallel(nil, func(rs *netstorage.Result, workerID uint) error {
+		err := rss.RunParallel(nil, func(rs *netstorage.Result, _ uint) error {
 			nameWithTags := getCanonicalPath(&rs.MetricName)
 			tags := unmarshalTags(nameWithTags)
 			s := &series{

app/vmselect/graphite/eval_test.go

@@ -3279,6 +3279,102 @@ func TestExecExprSuccess(t *testing.T) {
 			Tags:       map[string]string{"name": "foo.bar.baz", "summarize": "45s", "summarizeFunction": "sum"},
 		},
 	})
+	f(`aggregateSeriesLists(
+    summarize(
+               time('foo.bar.baz',10),
+               '45s'
+       ),
+    summarize(
+               time('bar.foo.bad',10),
+               '45s'
+       ), 'sum')`, []*series{
+		{
+			Timestamps: []int64{120000, 165000},
+			Values:     []float64{1170, 2000},
+			Name:       `sumSeries(summarize(foo.bar.baz,'45s','sum'),summarize(bar.foo.bad,'45s','sum'))`,
+			Tags:       map[string]string{"name": "foo.bar.baz", "summarize": "45s", "summarizeFunction": "sum"},
+		},
+	})
+	f(`sumSeriesLists(
+    summarize(
+               time('foo.bar.baz',10),
+               '45s'
+       ),
+    summarize(
+               time('bar.foo.bad',10),
+               '45s'
+       ))`, []*series{
+		{
+			Timestamps: []int64{120000, 165000},
+			Values:     []float64{1170, 2000},
+			Name:       `sumSeries(summarize(foo.bar.baz,'45s','sum'),summarize(bar.foo.bad,'45s','sum'))`,
+			Tags:       map[string]string{"name": "foo.bar.baz", "summarize": "45s", "summarizeFunction": "sum"},
+		},
+	})
+	f(`aggregateSeriesLists(
+    summarize(
+               time('foo.bar.baz',10),
+               '45s'
+       ),
+    summarize(
+               time('bar.foo.bad',10),
+               '45s'
+       ), 'diff')`, []*series{
+		{
+			Timestamps: []int64{120000, 165000},
+			Values:     []float64{0, 0},
+			Name:       `diffSeries(summarize(foo.bar.baz,'45s','sum'),summarize(bar.foo.bad,'45s','sum'))`,
+			Tags:       map[string]string{"name": "foo.bar.baz", "summarize": "45s", "summarizeFunction": "sum"},
+		},
+	})
+	f(`diffSeriesLists(
+    summarize(
+               time('foo.bar.baz',10),
+               '45s'
+       ),
+    summarize(
+               time('bar.foo.bad',10),
+               '45s'
+       ))`, []*series{
+		{
+			Timestamps: []int64{120000, 165000},
+			Values:     []float64{0, 0},
+			Name:       `diffSeries(summarize(foo.bar.baz,'45s','sum'),summarize(bar.foo.bad,'45s','sum'))`,
+			Tags:       map[string]string{"name": "foo.bar.baz", "summarize": "45s", "summarizeFunction": "sum"},
+		},
+	})
+	f(`aggregateSeriesLists(
+    summarize(
+               time('foo.bar.baz',10),
+               '45s'
+       ),
+    summarize(
+               time('bar.foo.bad',10),
+               '45s'
+       ), 'multiply')`, []*series{
+		{
+			Timestamps: []int64{120000, 165000},
+			Values:     []float64{342225, 1e+06},
+			Name:       `multiplySeries(summarize(foo.bar.baz,'45s','sum'),summarize(bar.foo.bad,'45s','sum'))`,
+			Tags:       map[string]string{"name": "foo.bar.baz", "summarize": "45s", "summarizeFunction": "sum"},
+		},
+	})
+	f(`multiplySeriesLists(
+    summarize(
+               time('foo.bar.baz',10),
+               '45s'
+       ),
+    summarize(
+               time('bar.foo.bad',10),
+               '45s'
+       ))`, []*series{
+		{
+			Timestamps: []int64{120000, 165000},
+			Values:     []float64{342225, 1e+06},
+			Name:       `multiplySeries(summarize(foo.bar.baz,'45s','sum'),summarize(bar.foo.bad,'45s','sum'))`,
+			Tags:       map[string]string{"name": "foo.bar.baz", "summarize": "45s", "summarizeFunction": "sum"},
+		},
+	})
 	f(`weightedAverage(
     summarize(
                group(

app/vmselect/graphite/functions.json

@@ -40,6 +40,52 @@
       }
     ]
   },
+  "aggregateSeriesLists": {
+    "name": "aggregateSeriesLists",
+    "function": "aggregateSeriesLists(seriesListFirstPos, seriesListSecondPos, func, xFilesFactor=None)",
+    "description": "Iterates over a two lists and aggregates using specified function list1[0] to list2[0], list1[1] to list2[1] and so on. The lists will need to be the same length\n\nPosition of seriesList matters. For example using “sum” function aggregateSeriesLists(list1[0..n], list2[0..n], \"sum\") it would find sum for each member of the list list1[0] + list2[0], list1[1] + list2[1], list1[n] + list2[n].",
+    "module": "graphite.render.functions",
+    "group": "Combine",
+    "params": [
+      {
+        "name": "seriesListFirstPos",
+        "type": "seriesList",
+        "required": true
+      },
+      {
+        "name": "seriesListSecondPos",
+        "type": "seriesList",
+        "required": true
+      },
+      {
+        "name": "func",
+        "type": "aggFunc",
+        "required": true,
+        "options": [
+          "average",
+          "avg",
+          "avg_zero",
+          "count",
+          "current",
+          "diff",
+          "last",
+          "max",
+          "median",
+          "min",
+          "multiply",
+          "range",
+          "rangeOf",
+          "stddev",
+          "sum",
+          "total"
+        ]
+      },
+      {
+        "name": "xFilesFactor",
+        "type": "float"
+      }
+    ]
+  },
   "aggregateWithWildcards": {
     "name": "aggregateWithWildcards",
     "function": "aggregateWithWildcards(seriesList, func, *positions)",
@@ -211,6 +257,25 @@
       }
     ]
   },
+  "diffSeriesLists": {
+    "name": "diffSeriesLists",
+    "function": "diffSeriesLists(seriesListFirstPos, seriesListSecondPos)",
+    "description": "Iterates over a two lists and subtracts series lists 2 through n from series 1 list1[0] to list2[0], list1[1] to list2[1] and so on. The lists will need to be the same length",
+    "module": "graphite.render.functions",
+    "group": "Combine",
+    "params": [
+      {
+        "name": "seriesListFirstPos",
+        "type": "seriesList",
+        "required": true
+      },
+      {
+        "name": "seriesListSecondPos",
+        "type": "seriesList",
+        "required": true
+      }
+    ]
+  },
   "divideSeries": {
     "name": "divideSeries",
     "function": "divideSeries(dividendSeriesList, divisorSeries)",
@@ -529,6 +594,25 @@
       }
     ]
   },
+  "multiplySeriesLists": {
+    "name": "multiplySeriesLists",
+    "function": "multiplySeriesLists(seriesListFirstPos, seriesListSecondPos)",
+    "description": "Iterates over a two lists and multiply series lists 2 through n from series 1 list1[0] to list2[0], list1[1] to list2[1] and so on. The lists will need to be the same length",
+    "module": "graphite.render.functions",
+    "group": "Combine",
+    "params": [
+      {
+        "name": "seriesListFirstPos",
+        "type": "seriesList",
+        "required": true
+      },
+      {
+        "name": "seriesListSecondPos",
+        "type": "seriesList",
+        "required": true
+      }
+    ]
+  },
   "multiplySeriesWithWildcards": {
     "name": "multiplySeriesWithWildcards",
     "function": "multiplySeriesWithWildcards(seriesList, *position)",
@@ -715,6 +799,26 @@
       }
     ]
   },
+  "sumSeriesLists": {
+    "name": "sumSeriesLists",
+    "function": "sumSeriesLists(seriesListFirstPos, seriesListSecondPos)",
+    "description": "Iterates over a two lists and sums series lists 2 through n from series 1 list1[0] to list2[0], list1[1] to list2[1] and so on. The lists will need to be the same length",
+    "module": "graphite.render.functions",
+    "group": "Combine",
+    "params": [
+      {
+        "name": "seriesListFirstPos",
+        "type": "seriesList",
+        "required": true
+      },
+      {
+        "name": "seriesListSecondPos",
+        "type": "seriesList",
+        "required": true
+      }
+    ]
+  },
+
   "sumSeriesWithWildcards": {
     "name": "sumSeriesWithWildcards",
     "function": "sumSeriesWithWildcards(seriesList, *position)",

app/vmselect/graphite/transform.go

@@ -36,6 +36,7 @@ func init() {
 		"add":                         transformAdd,
 		"aggregate":                   transformAggregate,
 		"aggregateLine":               transformAggregateLine,
+		"aggregateSeriesLists":        transformAggregateSeriesLists,
 		"aggregateWithWildcards":      transformAggregateWithWildcards,
 		"alias":                       transformAlias,
 		"aliasByMetric":               transformAliasByMetric,
@@ -66,6 +67,7 @@ func init() {
 		"delay":                       transformDelay,
 		"derivative":                  transformDerivative,
 		"diffSeries":                  transformDiffSeries,
+		"diffSeriesLists":             transformDiffSeriesLists,
 		"divideSeries":                transformDivideSeries,
 		"divideSeriesLists":           transformDivideSeriesLists,
 		"drawAsInfinite":              transformDrawAsInfinite,
@@ -125,6 +127,7 @@ func init() {
 		"movingSum":                   transformMovingSum,
 		"movingWindow":                transformMovingWindow,
 		"multiplySeries":              transformMultiplySeries,
+		"multiplySeriesLists":         transformMultiplySeriesLists,
 		"multiplySeriesWithWildcards": transformMultiplySeriesWithWildcards,
 		"nPercentile":                 transformNPercentile,
 		"nonNegativeDerivative":       transformNonNegativeDerivative,
@@ -172,6 +175,7 @@ func init() {
 		"substr":                  transformSubstr,
 		"sum":                     transformSumSeries,
 		"sumSeries":               transformSumSeries,
+		"sumSeriesLists":          transformSumSeriesLists,
 		"sumSeriesWithWildcards":  transformSumSeriesWithWildcards,
 		"summarize":               transformSummarize,
 		"threshold":               transformThreshold,
@@ -401,7 +405,7 @@ func aggregateSeriesWithWildcards(ec *evalConfig, expr graphiteql.Expr, nextSeri
 	for _, pos := range positions {
 		positionsMap[pos] = struct{}{}
 	}
-	keyFunc := func(name string, tags map[string]string) string {
+	keyFunc := func(name string, _ map[string]string) string {
 		parts := strings.Split(getPathFromName(name), ".")
 		dstParts := parts[:0]
 		for i, part := range parts {
@@ -1316,6 +1320,88 @@ func transformDivideSeries(ec *evalConfig, fe *graphiteql.FuncExpr) (nextSeriesF
 	return f, nil
 }
 
+func aggregateSeriesListsGeneric(ec *evalConfig, fe *graphiteql.FuncExpr, funcName string) (nextSeriesFunc, error) {
+	args := fe.Args
+	agg, err := getAggrFunc(funcName)
+	if err != nil {
+		return nil, err
+	}
+	nextSeriesFirst, err := evalSeriesList(ec, args, "seriesListFirstPos", 0)
+	if err != nil {
+		return nil, err
+	}
+	nextSeriesSecond, err := evalSeriesList(ec, args, "seriesListSecondPos", 1)
+	if err != nil {
+		_, _ = drainAllSeries(nextSeriesFirst)
+		return nil, err
+	}
+	return aggregateSeriesList(ec, fe, nextSeriesFirst, nextSeriesSecond, agg, funcName)
+}
+
+// See https://graphite.readthedocs.io/en/latest/functions.html#graphite.render.functions.aggregateSeriesLists
+func transformAggregateSeriesLists(ec *evalConfig, fe *graphiteql.FuncExpr) (nextSeriesFunc, error) {
+	args := fe.Args
+	if len(args) != 3 && len(args) != 4 {
+		return nil, fmt.Errorf("unexpected number of args; got %d; want 3 or 4", len(args))
+	}
+
+	funcName, err := getString(args, "func", 2)
+	if err != nil {
+		return nil, err
+	}
+
+	return aggregateSeriesListsGeneric(ec, fe, funcName)
+}
+
+// See https://graphite.readthedocs.io/en/latest/functions.html#graphite.render.functions.sumSeriesLists
+func transformSumSeriesLists(ec *evalConfig, fe *graphiteql.FuncExpr) (nextSeriesFunc, error) {
+	return aggregateSeriesListsGeneric(ec, fe, "sum")
+}
+
+// See https://graphite.readthedocs.io/en/latest/functions.html#graphite.render.functions.multiplySeriesLists
+func transformMultiplySeriesLists(ec *evalConfig, fe *graphiteql.FuncExpr) (nextSeriesFunc, error) {
+	return aggregateSeriesListsGeneric(ec, fe, "multiply")
+}
+
+// See https://graphite.readthedocs.io/en/latest/functions.html#graphite.render.functions.diffSeriesLists
+func transformDiffSeriesLists(ec *evalConfig, fe *graphiteql.FuncExpr) (nextSeriesFunc, error) {
+	return aggregateSeriesListsGeneric(ec, fe, "diff")
+}
+
+func aggregateSeriesList(ec *evalConfig, fe *graphiteql.FuncExpr, nextSeriesFirst, nextSeriesSecond nextSeriesFunc, agg aggrFunc, funcName string) (nextSeriesFunc, error) {
+	ssFirst, stepFirst, err := fetchNormalizedSeries(ec, nextSeriesFirst, false)
+	if err != nil {
+		_, _ = drainAllSeries(nextSeriesSecond)
+		return nil, err
+	}
+	ssSecond, stepSecond, err := fetchNormalizedSeries(ec, nextSeriesSecond, false)
+	if err != nil {
+		return nil, err
+	}
+
+	if len(ssFirst) != len(ssSecond) {
+		return nil, fmt.Errorf("First and second lists must have equal number of series; got %d vs %d series", len(ssFirst), len(ssSecond))
+	}
+	if stepFirst != stepSecond {
+		return nil, fmt.Errorf("step mismatch for first and second: %d vs %d", stepFirst, stepSecond)
+	}
+
+	valuePair := make([]float64, 2)
+	for i, s := range ssFirst {
+		sSecond := ssSecond[i]
+		values := s.Values
+		secondValues := sSecond.Values
+		for j, v := range values {
+			valuePair[0], valuePair[1] = v, secondValues[j]
+			values[j] = agg(valuePair)
+		}
+		s.Name = fmt.Sprintf("%sSeries(%s,%s)", funcName, s.Name, sSecond.Name)
+		s.expr = fe
+		s.pathExpression = s.Name
+	}
+	return multiSeriesFunc(ssFirst), nil
+}
+
 // See https://graphite.readthedocs.io/en/stable/functions.html#graphite.render.functions.divideSeriesLists
 func transformDivideSeriesLists(ec *evalConfig, fe *graphiteql.FuncExpr) (nextSeriesFunc, error) {
 	args := fe.Args
@@ -1326,36 +1412,14 @@ func transformDivideSeriesLists(ec *evalConfig, fe *graphiteql.FuncExpr) (nextSe
 	if err != nil {
 		return nil, err
 	}
-	ssDividend, stepDivident, err := fetchNormalizedSeries(ec, nextDividend, false)
-	if err != nil {
-		return nil, err
-	}
 	nextDivisor, err := evalSeriesList(ec, args, "divisorSeriesList", 1)
 	if err != nil {
 		return nil, err
 	}
-	ssDivisor, stepDivisor, err := fetchNormalizedSeries(ec, nextDivisor, false)
-	if err != nil {
-		return nil, err
-	}
-	if len(ssDividend) != len(ssDivisor) {
-		return nil, fmt.Errorf("divident and divisor must have equal number of series; got %d vs %d series", len(ssDividend), len(ssDivisor))
-	}
-	if stepDivident != stepDivisor {
-		return nil, fmt.Errorf("step mismatch for divident and divisor: %d vs %d", stepDivident, stepDivisor)
-	}
-	for i, s := range ssDividend {
-		sDivisor := ssDivisor[i]
-		values := s.Values
-		divisorValues := sDivisor.Values
-		for j, v := range values {
-			values[j] = v / divisorValues[j]
-		}
-		s.Name = fmt.Sprintf("divideSeries(%s,%s)", s.Name, sDivisor.Name)
-		s.expr = fe
-		s.pathExpression = s.Name
-	}
-	return multiSeriesFunc(ssDividend), nil
+
+	return aggregateSeriesList(ec, fe, nextDividend, nextDivisor, func(values []float64) float64 {
+		return values[0] / values[1]
+	}, "divide")
 }
 
 // See https://graphite.readthedocs.io/en/stable/functions.html#graphite.render.functions.drawAsInfinite
@@ -1819,7 +1883,7 @@ func transformGroupByTags(ec *evalConfig, fe *graphiteql.FuncExpr) (nextSeriesFu
 	if err != nil {
 		return nil, err
 	}
-	keyFunc := func(name string, tags map[string]string) string {
+	keyFunc := func(_ string, tags map[string]string) string {
 		return formatKeyFromTags(tags, tagKeys, callback)
 	}
 	return groupByKeyFunc(ec, fe, nextSeries, callback, keyFunc)
@@ -3842,18 +3906,18 @@ func nextSeriesConcurrentWrapper(nextSeries nextSeriesFunc, f func(s *series) (*
 		errCh <- err
 		close(errCh)
 	}()
-	var skipProcessing uint32
+	var skipProcessing atomic.Bool
 	for i := 0; i < goroutines; i++ {
 		go func() {
 			defer wg.Done()
 			for s := range seriesCh {
-				if atomic.LoadUint32(&skipProcessing) != 0 {
+				if skipProcessing.Load() {
 					continue
 				}
 				sNew, err := f(s)
 				if err != nil {
 					// Drain the rest of series and do not call f for them in order to conserve CPU time.
-					atomic.StoreUint32(&skipProcessing, 1)
+					skipProcessing.Store(true)
 					resultCh <- &result{
 						err: err,
 					}
@@ -5609,9 +5673,9 @@ func (nsf *nextSeriesFunc) peekStep(step int64) (int64, error) {
 	if s != nil {
 		step = s.step
 	}
-	calls := uint64(0)
+	var calls atomic.Uint64
 	*nsf = func() (*series, error) {
-		if atomic.AddUint64(&calls, 1) == 1 {
+		if calls.Add(1) == 1 {
 			return s, nil
 		}
 		return nextSeries()

app/vmselect/netstorage/netstorage.go

@@ -5,13 +5,15 @@ import (
 	"errors"
 	"flag"
 	"fmt"
-	"reflect"
 	"sort"
 	"sync"
 	"sync/atomic"
 	"time"
 	"unsafe"
 
+	"github.com/VictoriaMetrics/metrics"
+	"github.com/VictoriaMetrics/metricsql"
+
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/searchutils"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
@@ -19,16 +21,17 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fasttime"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/querytracer"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/storage"
-	"github.com/VictoriaMetrics/metrics"
-	"github.com/VictoriaMetrics/metricsql"
 )
 
 var (
-	maxTagKeysPerSearch   = flag.Int("search.maxTagKeys", 100e3, "The maximum number of tag keys returned from /api/v1/labels")
-	maxTagValuesPerSearch = flag.Int("search.maxTagValues", 100e3, "The maximum number of tag values returned from /api/v1/label/<label_name>/values")
-	maxSamplesPerSeries   = flag.Int("search.maxSamplesPerSeries", 30e6, "The maximum number of raw samples a single query can scan per each time series. This option allows limiting memory usage")
-	maxSamplesPerQuery    = flag.Int("search.maxSamplesPerQuery", 1e9, "The maximum number of raw samples a single query can process across all time series. This protects from heavy queries, which select unexpectedly high number of raw samples. See also -search.maxSamplesPerSeries")
-	maxWorkersPerQuery    = flag.Int("search.maxWorkersPerQuery", defaultMaxWorkersPerQuery, "The maximum number of CPU cores a single query can use. "+
+	maxTagKeysPerSearch = flag.Int("search.maxTagKeys", 100e3, "The maximum number of tag keys returned from /api/v1/labels . "+
+		"See also -search.maxLabelsAPISeries and -search.maxLabelsAPIDuration")
+	maxTagValuesPerSearch = flag.Int("search.maxTagValues", 100e3, "The maximum number of tag values returned from /api/v1/label/<label_name>/values . "+
+		"See also -search.maxLabelsAPISeries and -search.maxLabelsAPIDuration")
+	maxSamplesPerSeries = flag.Int("search.maxSamplesPerSeries", 30e6, "The maximum number of raw samples a single query can scan per each time series. This option allows limiting memory usage")
+	maxSamplesPerQuery  = flag.Int("search.maxSamplesPerQuery", 1e9, "The maximum number of raw samples a single query can process across all time series. "+
+		"This protects from heavy queries, which select unexpectedly high number of raw samples. See also -search.maxSamplesPerSeries")
+	maxWorkersPerQuery = flag.Int("search.maxWorkersPerQuery", defaultMaxWorkersPerQuery, "The maximum number of CPU cores a single query can use. "+
 		"The default value should work good for most cases. "+
 		"The flag can be set to lower values for improving performance of big number of concurrently executed queries. "+
 		"The flag can be set to bigger values for improving performance of heavy queries, which scan big number of time series (>10K) and/or big number of samples (>100M). "+
@@ -81,7 +84,7 @@ func (rss *Results) mustClose() {
 }
 
 type timeseriesWork struct {
-	mustStop *uint32
+	mustStop *atomic.Bool
 	rss      *Results
 	pts      *packedTimeseries
 	f        func(rs *Result, workerID uint) error
@@ -91,22 +94,22 @@ type timeseriesWork struct {
 }
 
 func (tsw *timeseriesWork) do(r *Result, workerID uint) error {
-	if atomic.LoadUint32(tsw.mustStop) != 0 {
+	if tsw.mustStop.Load() {
 		return nil
 	}
 	rss := tsw.rss
 	if rss.deadline.Exceeded() {
-		atomic.StoreUint32(tsw.mustStop, 1)
+		tsw.mustStop.Store(true)
 		return fmt.Errorf("timeout exceeded during query execution: %s", rss.deadline.String())
 	}
 	if err := tsw.pts.Unpack(r, rss.tbf, rss.tr); err != nil {
-		atomic.StoreUint32(tsw.mustStop, 1)
+		tsw.mustStop.Store(true)
 		return fmt.Errorf("error during time series unpacking: %w", err)
 	}
 	tsw.rowsProcessed = len(r.Timestamps)
 	if len(r.Timestamps) > 0 {
 		if err := tsw.f(r, workerID); err != nil {
-			atomic.StoreUint32(tsw.mustStop, 1)
+			tsw.mustStop.Store(true)
 			return err
 		}
 	}
@@ -238,7 +241,7 @@ func (rss *Results) runParallel(qt *querytracer.Tracer, f func(rs *Result, worke
 		return 0, nil
 	}
 
-	var mustStop uint32
+	var mustStop atomic.Bool
 	initTimeseriesWork := func(tsw *timeseriesWork, pts *packedTimeseries) {
 		tsw.rss = rss
 		tsw.pts = pts
@@ -1008,7 +1011,7 @@ func ExportBlocks(qt *querytracer.Tracer, sq *storage.SearchQuery, deadline sear
 	var (
 		errGlobal     error
 		errGlobalLock sync.Mutex
-		mustStop      uint32
+		mustStop      atomic.Bool
 	)
 	var wg sync.WaitGroup
 	wg.Add(gomaxprocs)
@@ -1020,7 +1023,7 @@ func ExportBlocks(qt *querytracer.Tracer, sq *storage.SearchQuery, deadline sear
 					errGlobalLock.Lock()
 					if errGlobal == nil {
 						errGlobal = err
-						atomic.StoreUint32(&mustStop, 1)
+						mustStop.Store(true)
 					}
 					errGlobalLock.Unlock()
 				}
@@ -1038,7 +1041,7 @@ func ExportBlocks(qt *querytracer.Tracer, sq *storage.SearchQuery, deadline sear
 		if deadline.Exceeded() {
 			return fmt.Errorf("timeout exceeded while fetching data block #%d from storage: %s", blocksRead, deadline.String())
 		}
-		if atomic.LoadUint32(&mustStop) != 0 {
+		if mustStop.Load() {
 			break
 		}
 		xw := exportWorkPool.Get().(*exportWork)
@@ -1309,9 +1312,11 @@ func canAppendToBlockRefPool(pool, a []blockRef) bool {
 		// a doesn't belong to pool
 		return false
 	}
-	shPool := (*reflect.SliceHeader)(unsafe.Pointer(&pool))
-	shA := (*reflect.SliceHeader)(unsafe.Pointer(&a))
-	return shPool.Data+uintptr(shPool.Len)*unsafe.Sizeof(blockRef{}) == shA.Data+uintptr(shA.Len)*unsafe.Sizeof(blockRef{})
+	return getBlockRefsEnd(pool) == getBlockRefsEnd(a)
+}
+
+func getBlockRefsEnd(a []blockRef) uintptr {
+	return uintptr(unsafe.Pointer(unsafe.SliceData(a))) + uintptr(len(a))*unsafe.Sizeof(blockRef{})
 }
 
 func setupTfss(qt *querytracer.Tracer, tr storage.TimeRange, tagFilterss [][]storage.TagFilter, maxMetrics int, deadline searchutils.Deadline) ([]*storage.TagFilters, error) {

app/vmselect/prometheus/export.qtpl

@@ -24,6 +24,7 @@
 	{% endfor %}
 {% endfunc %}
 
+{%code const rfc3339Milli = "2006-01-02T15:04:05.999Z07:00" %}
 {% func exportCSVField(mn *storage.MetricName, fieldName string, timestamp int64, value float64) %}
 	{% if fieldName == "__value__" %}
 		{%f= value %}
@@ -45,7 +46,7 @@
 		{% case "rfc3339" %}
 			{% code
 				bb := quicktemplate.AcquireByteBuffer()
-				bb.B = time.Unix(timestamp/1000, (timestamp%1000)*1e6).AppendFormat(bb.B[:0], time.RFC3339)
+				bb.B = time.Unix(timestamp/1000, (timestamp%1000)*1e6).AppendFormat(bb.B[:0], rfc3339Milli)
 			%}
 			{%z= bb.B %}
 			{% code

app/vmselect/prometheus/export.qtpl.go

@@ -87,586 +87,589 @@ func ExportCSVLine(xb *exportBlock, fieldNames []string) string {
 }
 
 //line app/vmselect/prometheus/export.qtpl:27
-func streamexportCSVField(qw422016 *qt422016.Writer, mn *storage.MetricName, fieldName string, timestamp int64, value float64) {
+const rfc3339Milli = "2006-01-02T15:04:05.999Z07:00"
+
 //line app/vmselect/prometheus/export.qtpl:28
-	if fieldName == "__value__" {
+func streamexportCSVField(qw422016 *qt422016.Writer, mn *storage.MetricName, fieldName string, timestamp int64, value float64) {
 //line app/vmselect/prometheus/export.qtpl:29
-		qw422016.N().F(value)
+	if fieldName == "__value__" {
 //line app/vmselect/prometheus/export.qtpl:30
-		return
+		qw422016.N().F(value)
 //line app/vmselect/prometheus/export.qtpl:31
-	}
-//line app/vmselect/prometheus/export.qtpl:32
-	if fieldName == "__timestamp__" {
-//line app/vmselect/prometheus/export.qtpl:33
-		qw422016.N().DL(timestamp)
-//line app/vmselect/prometheus/export.qtpl:34
 		return
-//line app/vmselect/prometheus/export.qtpl:35
+//line app/vmselect/prometheus/export.qtpl:32
 	}
+//line app/vmselect/prometheus/export.qtpl:33
+	if fieldName == "__timestamp__" {
+//line app/vmselect/prometheus/export.qtpl:34
+		qw422016.N().DL(timestamp)
+//line app/vmselect/prometheus/export.qtpl:35
+		return
 //line app/vmselect/prometheus/export.qtpl:36
-	if strings.HasPrefix(fieldName, "__timestamp__:") {
+	}
 //line app/vmselect/prometheus/export.qtpl:37
+	if strings.HasPrefix(fieldName, "__timestamp__:") {
+//line app/vmselect/prometheus/export.qtpl:38
 		timeFormat := fieldName[len("__timestamp__:"):]
 
-//line app/vmselect/prometheus/export.qtpl:38
-		switch timeFormat {
 //line app/vmselect/prometheus/export.qtpl:39
-		case "unix_s":
+		switch timeFormat {
 //line app/vmselect/prometheus/export.qtpl:40
-			qw422016.N().DL(timestamp / 1000)
+		case "unix_s":
 //line app/vmselect/prometheus/export.qtpl:41
-		case "unix_ms":
+			qw422016.N().DL(timestamp / 1000)
 //line app/vmselect/prometheus/export.qtpl:42
-			qw422016.N().DL(timestamp)
+		case "unix_ms":
 //line app/vmselect/prometheus/export.qtpl:43
-		case "unix_ns":
+			qw422016.N().DL(timestamp)
 //line app/vmselect/prometheus/export.qtpl:44
-			qw422016.N().DL(timestamp * 1e6)
+		case "unix_ns":
 //line app/vmselect/prometheus/export.qtpl:45
+			qw422016.N().DL(timestamp * 1e6)
+//line app/vmselect/prometheus/export.qtpl:46
 		case "rfc3339":
-//line app/vmselect/prometheus/export.qtpl:47
+//line app/vmselect/prometheus/export.qtpl:48
 			bb := quicktemplate.AcquireByteBuffer()
-			bb.B = time.Unix(timestamp/1000, (timestamp%1000)*1e6).AppendFormat(bb.B[:0], time.RFC3339)
+			bb.B = time.Unix(timestamp/1000, (timestamp%1000)*1e6).AppendFormat(bb.B[:0], rfc3339Milli)
 
-//line app/vmselect/prometheus/export.qtpl:50
+//line app/vmselect/prometheus/export.qtpl:51
 			qw422016.N().Z(bb.B)
-//line app/vmselect/prometheus/export.qtpl:52
+//line app/vmselect/prometheus/export.qtpl:53
 			quicktemplate.ReleaseByteBuffer(bb)
 
-//line app/vmselect/prometheus/export.qtpl:54
-		default:
 //line app/vmselect/prometheus/export.qtpl:55
+		default:
+//line app/vmselect/prometheus/export.qtpl:56
 			if strings.HasPrefix(timeFormat, "custom:") {
-//line app/vmselect/prometheus/export.qtpl:57
+//line app/vmselect/prometheus/export.qtpl:58
 				layout := timeFormat[len("custom:"):]
 				bb := quicktemplate.AcquireByteBuffer()
 				bb.B = time.Unix(timestamp/1000, (timestamp%1000)*1e6).AppendFormat(bb.B[:0], layout)
 
-//line app/vmselect/prometheus/export.qtpl:61
-				if bytes.ContainsAny(bb.B, `"`+",\n") {
 //line app/vmselect/prometheus/export.qtpl:62
-					qw422016.E().QZ(bb.B)
+				if bytes.ContainsAny(bb.B, `"`+",\n") {
 //line app/vmselect/prometheus/export.qtpl:63
-				} else {
+					qw422016.E().QZ(bb.B)
 //line app/vmselect/prometheus/export.qtpl:64
-					qw422016.N().Z(bb.B)
+				} else {
 //line app/vmselect/prometheus/export.qtpl:65
+					qw422016.N().Z(bb.B)
+//line app/vmselect/prometheus/export.qtpl:66
 				}
-//line app/vmselect/prometheus/export.qtpl:67
+//line app/vmselect/prometheus/export.qtpl:68
 				quicktemplate.ReleaseByteBuffer(bb)
 
-//line app/vmselect/prometheus/export.qtpl:69
-			} else {
-//line app/vmselect/prometheus/export.qtpl:69
-				qw422016.N().S(`Unsupported timeFormat=`)
 //line app/vmselect/prometheus/export.qtpl:70
-				qw422016.N().S(timeFormat)
+			} else {
+//line app/vmselect/prometheus/export.qtpl:70
+				qw422016.N().S(`Unsupported timeFormat=`)
 //line app/vmselect/prometheus/export.qtpl:71
-			}
+				qw422016.N().S(timeFormat)
 //line app/vmselect/prometheus/export.qtpl:72
-		}
+			}
 //line app/vmselect/prometheus/export.qtpl:73
-		return
+		}
 //line app/vmselect/prometheus/export.qtpl:74
-	}
+		return
 //line app/vmselect/prometheus/export.qtpl:75
+	}
+//line app/vmselect/prometheus/export.qtpl:76
 	v := mn.GetTagValue(fieldName)
 
-//line app/vmselect/prometheus/export.qtpl:76
-	if bytes.ContainsAny(v, `"`+",\n") {
 //line app/vmselect/prometheus/export.qtpl:77
-		qw422016.N().QZ(v)
+	if bytes.ContainsAny(v, `"`+",\n") {
 //line app/vmselect/prometheus/export.qtpl:78
-	} else {
+		qw422016.N().QZ(v)
 //line app/vmselect/prometheus/export.qtpl:79
-		qw422016.N().Z(v)
+	} else {
 //line app/vmselect/prometheus/export.qtpl:80
-	}
+		qw422016.N().Z(v)
 //line app/vmselect/prometheus/export.qtpl:81
+	}
+//line app/vmselect/prometheus/export.qtpl:82
 }
 
-//line app/vmselect/prometheus/export.qtpl:81
+//line app/vmselect/prometheus/export.qtpl:82
 func writeexportCSVField(qq422016 qtio422016.Writer, mn *storage.MetricName, fieldName string, timestamp int64, value float64) {
-//line app/vmselect/prometheus/export.qtpl:81
+//line app/vmselect/prometheus/export.qtpl:82
 	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vmselect/prometheus/export.qtpl:81
+//line app/vmselect/prometheus/export.qtpl:82
 	streamexportCSVField(qw422016, mn, fieldName, timestamp, value)
-//line app/vmselect/prometheus/export.qtpl:81
+//line app/vmselect/prometheus/export.qtpl:82
 	qt422016.ReleaseWriter(qw422016)
-//line app/vmselect/prometheus/export.qtpl:81
+//line app/vmselect/prometheus/export.qtpl:82
 }
 
-//line app/vmselect/prometheus/export.qtpl:81
+//line app/vmselect/prometheus/export.qtpl:82
 func exportCSVField(mn *storage.MetricName, fieldName string, timestamp int64, value float64) string {
-//line app/vmselect/prometheus/export.qtpl:81
+//line app/vmselect/prometheus/export.qtpl:82
 	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vmselect/prometheus/export.qtpl:81
+//line app/vmselect/prometheus/export.qtpl:82
 	writeexportCSVField(qb422016, mn, fieldName, timestamp, value)
-//line app/vmselect/prometheus/export.qtpl:81
+//line app/vmselect/prometheus/export.qtpl:82
 	qs422016 := string(qb422016.B)
-//line app/vmselect/prometheus/export.qtpl:81
+//line app/vmselect/prometheus/export.qtpl:82
 	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vmselect/prometheus/export.qtpl:81
+//line app/vmselect/prometheus/export.qtpl:82
 	return qs422016
-//line app/vmselect/prometheus/export.qtpl:81
+//line app/vmselect/prometheus/export.qtpl:82
 }
 
-//line app/vmselect/prometheus/export.qtpl:83
+//line app/vmselect/prometheus/export.qtpl:84
 func StreamExportPrometheusLine(qw422016 *qt422016.Writer, xb *exportBlock) {
-//line app/vmselect/prometheus/export.qtpl:84
-	if len(xb.timestamps) == 0 {
-//line app/vmselect/prometheus/export.qtpl:84
-		return
-//line app/vmselect/prometheus/export.qtpl:84
-	}
 //line app/vmselect/prometheus/export.qtpl:85
+	if len(xb.timestamps) == 0 {
+//line app/vmselect/prometheus/export.qtpl:85
+		return
+//line app/vmselect/prometheus/export.qtpl:85
+	}
+//line app/vmselect/prometheus/export.qtpl:86
 	bb := quicktemplate.AcquireByteBuffer()
 
-//line app/vmselect/prometheus/export.qtpl:86
+//line app/vmselect/prometheus/export.qtpl:87
 	writeprometheusMetricName(bb, xb.mn)
 
-//line app/vmselect/prometheus/export.qtpl:87
+//line app/vmselect/prometheus/export.qtpl:88
 	for i, ts := range xb.timestamps {
-//line app/vmselect/prometheus/export.qtpl:88
+//line app/vmselect/prometheus/export.qtpl:89
 		qw422016.N().Z(bb.B)
-//line app/vmselect/prometheus/export.qtpl:88
-		qw422016.N().S(` `)
 //line app/vmselect/prometheus/export.qtpl:89
+		qw422016.N().S(` `)
+//line app/vmselect/prometheus/export.qtpl:90
 		qw422016.N().F(xb.values[i])
-//line app/vmselect/prometheus/export.qtpl:89
+//line app/vmselect/prometheus/export.qtpl:90
 		qw422016.N().S(` `)
-//line app/vmselect/prometheus/export.qtpl:90
+//line app/vmselect/prometheus/export.qtpl:91
 		qw422016.N().DL(ts)
-//line app/vmselect/prometheus/export.qtpl:90
+//line app/vmselect/prometheus/export.qtpl:91
 		qw422016.N().S(`
 `)
-//line app/vmselect/prometheus/export.qtpl:91
-	}
 //line app/vmselect/prometheus/export.qtpl:92
+	}
+//line app/vmselect/prometheus/export.qtpl:93
 	quicktemplate.ReleaseByteBuffer(bb)
 
-//line app/vmselect/prometheus/export.qtpl:93
+//line app/vmselect/prometheus/export.qtpl:94
 }
 
-//line app/vmselect/prometheus/export.qtpl:93
+//line app/vmselect/prometheus/export.qtpl:94
 func WriteExportPrometheusLine(qq422016 qtio422016.Writer, xb *exportBlock) {
-//line app/vmselect/prometheus/export.qtpl:93
+//line app/vmselect/prometheus/export.qtpl:94
 	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vmselect/prometheus/export.qtpl:93
+//line app/vmselect/prometheus/export.qtpl:94
 	StreamExportPrometheusLine(qw422016, xb)
-//line app/vmselect/prometheus/export.qtpl:93
+//line app/vmselect/prometheus/export.qtpl:94
 	qt422016.ReleaseWriter(qw422016)
-//line app/vmselect/prometheus/export.qtpl:93
+//line app/vmselect/prometheus/export.qtpl:94
 }
 
-//line app/vmselect/prometheus/export.qtpl:93
+//line app/vmselect/prometheus/export.qtpl:94
 func ExportPrometheusLine(xb *exportBlock) string {
-//line app/vmselect/prometheus/export.qtpl:93
+//line app/vmselect/prometheus/export.qtpl:94
 	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vmselect/prometheus/export.qtpl:93
+//line app/vmselect/prometheus/export.qtpl:94
 	WriteExportPrometheusLine(qb422016, xb)
-//line app/vmselect/prometheus/export.qtpl:93
+//line app/vmselect/prometheus/export.qtpl:94
 	qs422016 := string(qb422016.B)
-//line app/vmselect/prometheus/export.qtpl:93
+//line app/vmselect/prometheus/export.qtpl:94
 	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vmselect/prometheus/export.qtpl:93
+//line app/vmselect/prometheus/export.qtpl:94
 	return qs422016
-//line app/vmselect/prometheus/export.qtpl:93
+//line app/vmselect/prometheus/export.qtpl:94
 }
 
-//line app/vmselect/prometheus/export.qtpl:95
+//line app/vmselect/prometheus/export.qtpl:96
 func StreamExportJSONLine(qw422016 *qt422016.Writer, xb *exportBlock) {
-//line app/vmselect/prometheus/export.qtpl:96
+//line app/vmselect/prometheus/export.qtpl:97
 	if len(xb.timestamps) == 0 {
-//line app/vmselect/prometheus/export.qtpl:96
+//line app/vmselect/prometheus/export.qtpl:97
 		return
-//line app/vmselect/prometheus/export.qtpl:96
+//line app/vmselect/prometheus/export.qtpl:97
 	}
-//line app/vmselect/prometheus/export.qtpl:96
+//line app/vmselect/prometheus/export.qtpl:97
 	qw422016.N().S(`{"metric":`)
-//line app/vmselect/prometheus/export.qtpl:98
+//line app/vmselect/prometheus/export.qtpl:99
 	streammetricNameObject(qw422016, xb.mn)
-//line app/vmselect/prometheus/export.qtpl:98
+//line app/vmselect/prometheus/export.qtpl:99
 	qw422016.N().S(`,"values":[`)
-//line app/vmselect/prometheus/export.qtpl:100
-	if len(xb.values) > 0 {
 //line app/vmselect/prometheus/export.qtpl:101
+	if len(xb.values) > 0 {
+//line app/vmselect/prometheus/export.qtpl:102
 		values := xb.values
 
-//line app/vmselect/prometheus/export.qtpl:102
-		streamconvertValueToSpecialJSON(qw422016, values[0])
 //line app/vmselect/prometheus/export.qtpl:103
+		streamconvertValueToSpecialJSON(qw422016, values[0])
+//line app/vmselect/prometheus/export.qtpl:104
 		values = values[1:]
 
-//line app/vmselect/prometheus/export.qtpl:104
-		for _, v := range values {
-//line app/vmselect/prometheus/export.qtpl:104
-			qw422016.N().S(`,`)
 //line app/vmselect/prometheus/export.qtpl:105
-			streamconvertValueToSpecialJSON(qw422016, v)
+		for _, v := range values {
+//line app/vmselect/prometheus/export.qtpl:105
+			qw422016.N().S(`,`)
 //line app/vmselect/prometheus/export.qtpl:106
+			streamconvertValueToSpecialJSON(qw422016, v)
+//line app/vmselect/prometheus/export.qtpl:107
 		}
-//line app/vmselect/prometheus/export.qtpl:107
+//line app/vmselect/prometheus/export.qtpl:108
 	}
-//line app/vmselect/prometheus/export.qtpl:107
+//line app/vmselect/prometheus/export.qtpl:108
 	qw422016.N().S(`],"timestamps":[`)
-//line app/vmselect/prometheus/export.qtpl:110
-	if len(xb.timestamps) > 0 {
 //line app/vmselect/prometheus/export.qtpl:111
+	if len(xb.timestamps) > 0 {
+//line app/vmselect/prometheus/export.qtpl:112
 		timestamps := xb.timestamps
 
-//line app/vmselect/prometheus/export.qtpl:112
-		qw422016.N().DL(timestamps[0])
 //line app/vmselect/prometheus/export.qtpl:113
+		qw422016.N().DL(timestamps[0])
+//line app/vmselect/prometheus/export.qtpl:114
 		timestamps = timestamps[1:]
 
-//line app/vmselect/prometheus/export.qtpl:114
-		for _, ts := range timestamps {
-//line app/vmselect/prometheus/export.qtpl:114
-			qw422016.N().S(`,`)
 //line app/vmselect/prometheus/export.qtpl:115
-			qw422016.N().DL(ts)
+		for _, ts := range timestamps {
+//line app/vmselect/prometheus/export.qtpl:115
+			qw422016.N().S(`,`)
 //line app/vmselect/prometheus/export.qtpl:116
+			qw422016.N().DL(ts)
+//line app/vmselect/prometheus/export.qtpl:117
 		}
-//line app/vmselect/prometheus/export.qtpl:117
+//line app/vmselect/prometheus/export.qtpl:118
 	}
-//line app/vmselect/prometheus/export.qtpl:117
+//line app/vmselect/prometheus/export.qtpl:118
 	qw422016.N().S(`]}`)
-//line app/vmselect/prometheus/export.qtpl:119
+//line app/vmselect/prometheus/export.qtpl:120
 	qw422016.N().S(`
 `)
-//line app/vmselect/prometheus/export.qtpl:120
+//line app/vmselect/prometheus/export.qtpl:121
 }
 
-//line app/vmselect/prometheus/export.qtpl:120
+//line app/vmselect/prometheus/export.qtpl:121
 func WriteExportJSONLine(qq422016 qtio422016.Writer, xb *exportBlock) {
-//line app/vmselect/prometheus/export.qtpl:120
+//line app/vmselect/prometheus/export.qtpl:121
 	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vmselect/prometheus/export.qtpl:120
+//line app/vmselect/prometheus/export.qtpl:121
 	StreamExportJSONLine(qw422016, xb)
-//line app/vmselect/prometheus/export.qtpl:120
+//line app/vmselect/prometheus/export.qtpl:121
 	qt422016.ReleaseWriter(qw422016)
-//line app/vmselect/prometheus/export.qtpl:120
+//line app/vmselect/prometheus/export.qtpl:121
 }
 
-//line app/vmselect/prometheus/export.qtpl:120
+//line app/vmselect/prometheus/export.qtpl:121
 func ExportJSONLine(xb *exportBlock) string {
-//line app/vmselect/prometheus/export.qtpl:120
+//line app/vmselect/prometheus/export.qtpl:121
 	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vmselect/prometheus/export.qtpl:120
+//line app/vmselect/prometheus/export.qtpl:121
 	WriteExportJSONLine(qb422016, xb)
-//line app/vmselect/prometheus/export.qtpl:120
+//line app/vmselect/prometheus/export.qtpl:121
 	qs422016 := string(qb422016.B)
-//line app/vmselect/prometheus/export.qtpl:120
+//line app/vmselect/prometheus/export.qtpl:121
 	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vmselect/prometheus/export.qtpl:120
+//line app/vmselect/prometheus/export.qtpl:121
 	return qs422016
-//line app/vmselect/prometheus/export.qtpl:120
+//line app/vmselect/prometheus/export.qtpl:121
 }
 
-//line app/vmselect/prometheus/export.qtpl:122
+//line app/vmselect/prometheus/export.qtpl:123
 func StreamExportPromAPILine(qw422016 *qt422016.Writer, xb *exportBlock) {
-//line app/vmselect/prometheus/export.qtpl:122
+//line app/vmselect/prometheus/export.qtpl:123
 	qw422016.N().S(`{"metric":`)
-//line app/vmselect/prometheus/export.qtpl:124
+//line app/vmselect/prometheus/export.qtpl:125
 	streammetricNameObject(qw422016, xb.mn)
-//line app/vmselect/prometheus/export.qtpl:124
+//line app/vmselect/prometheus/export.qtpl:125
 	qw422016.N().S(`,"values":`)
-//line app/vmselect/prometheus/export.qtpl:125
+//line app/vmselect/prometheus/export.qtpl:126
 	streamvaluesWithTimestamps(qw422016, xb.values, xb.timestamps)
-//line app/vmselect/prometheus/export.qtpl:125
+//line app/vmselect/prometheus/export.qtpl:126
 	qw422016.N().S(`}`)
-//line app/vmselect/prometheus/export.qtpl:127
+//line app/vmselect/prometheus/export.qtpl:128
 }
 
-//line app/vmselect/prometheus/export.qtpl:127
+//line app/vmselect/prometheus/export.qtpl:128
 func WriteExportPromAPILine(qq422016 qtio422016.Writer, xb *exportBlock) {
-//line app/vmselect/prometheus/export.qtpl:127
+//line app/vmselect/prometheus/export.qtpl:128
 	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vmselect/prometheus/export.qtpl:127
+//line app/vmselect/prometheus/export.qtpl:128
 	StreamExportPromAPILine(qw422016, xb)
-//line app/vmselect/prometheus/export.qtpl:127
+//line app/vmselect/prometheus/export.qtpl:128
 	qt422016.ReleaseWriter(qw422016)
-//line app/vmselect/prometheus/export.qtpl:127
+//line app/vmselect/prometheus/export.qtpl:128
 }
 
-//line app/vmselect/prometheus/export.qtpl:127
+//line app/vmselect/prometheus/export.qtpl:128
 func ExportPromAPILine(xb *exportBlock) string {
-//line app/vmselect/prometheus/export.qtpl:127
+//line app/vmselect/prometheus/export.qtpl:128
 	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vmselect/prometheus/export.qtpl:127
+//line app/vmselect/prometheus/export.qtpl:128
 	WriteExportPromAPILine(qb422016, xb)
-//line app/vmselect/prometheus/export.qtpl:127
+//line app/vmselect/prometheus/export.qtpl:128
 	qs422016 := string(qb422016.B)
-//line app/vmselect/prometheus/export.qtpl:127
+//line app/vmselect/prometheus/export.qtpl:128
 	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vmselect/prometheus/export.qtpl:127
+//line app/vmselect/prometheus/export.qtpl:128
 	return qs422016
-//line app/vmselect/prometheus/export.qtpl:127
+//line app/vmselect/prometheus/export.qtpl:128
 }
 
-//line app/vmselect/prometheus/export.qtpl:129
+//line app/vmselect/prometheus/export.qtpl:130
 func StreamExportPromAPIHeader(qw422016 *qt422016.Writer) {
-//line app/vmselect/prometheus/export.qtpl:129
+//line app/vmselect/prometheus/export.qtpl:130
 	qw422016.N().S(`{"status":"success","data":{"resultType":"matrix","result":[`)
-//line app/vmselect/prometheus/export.qtpl:135
+//line app/vmselect/prometheus/export.qtpl:136
 }
 
-//line app/vmselect/prometheus/export.qtpl:135
+//line app/vmselect/prometheus/export.qtpl:136
 func WriteExportPromAPIHeader(qq422016 qtio422016.Writer) {
-//line app/vmselect/prometheus/export.qtpl:135
+//line app/vmselect/prometheus/export.qtpl:136
 	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vmselect/prometheus/export.qtpl:135
+//line app/vmselect/prometheus/export.qtpl:136
 	StreamExportPromAPIHeader(qw422016)
-//line app/vmselect/prometheus/export.qtpl:135
+//line app/vmselect/prometheus/export.qtpl:136
 	qt422016.ReleaseWriter(qw422016)
-//line app/vmselect/prometheus/export.qtpl:135
+//line app/vmselect/prometheus/export.qtpl:136
 }
 
-//line app/vmselect/prometheus/export.qtpl:135
+//line app/vmselect/prometheus/export.qtpl:136
 func ExportPromAPIHeader() string {
-//line app/vmselect/prometheus/export.qtpl:135
+//line app/vmselect/prometheus/export.qtpl:136
 	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vmselect/prometheus/export.qtpl:135
+//line app/vmselect/prometheus/export.qtpl:136
 	WriteExportPromAPIHeader(qb422016)
-//line app/vmselect/prometheus/export.qtpl:135
+//line app/vmselect/prometheus/export.qtpl:136
 	qs422016 := string(qb422016.B)
-//line app/vmselect/prometheus/export.qtpl:135
+//line app/vmselect/prometheus/export.qtpl:136
 	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vmselect/prometheus/export.qtpl:135
+//line app/vmselect/prometheus/export.qtpl:136
 	return qs422016
-//line app/vmselect/prometheus/export.qtpl:135
+//line app/vmselect/prometheus/export.qtpl:136
 }
 
-//line app/vmselect/prometheus/export.qtpl:137
+//line app/vmselect/prometheus/export.qtpl:138
 func StreamExportPromAPIFooter(qw422016 *qt422016.Writer, qt *querytracer.Tracer) {
-//line app/vmselect/prometheus/export.qtpl:137
+//line app/vmselect/prometheus/export.qtpl:138
 	qw422016.N().S(`]}`)
-//line app/vmselect/prometheus/export.qtpl:141
+//line app/vmselect/prometheus/export.qtpl:142
 	qt.Donef("export format=promapi")
 
-//line app/vmselect/prometheus/export.qtpl:143
+//line app/vmselect/prometheus/export.qtpl:144
 	streamdumpQueryTrace(qw422016, qt)
-//line app/vmselect/prometheus/export.qtpl:143
+//line app/vmselect/prometheus/export.qtpl:144
 	qw422016.N().S(`}`)
-//line app/vmselect/prometheus/export.qtpl:145
+//line app/vmselect/prometheus/export.qtpl:146
 }
 
-//line app/vmselect/prometheus/export.qtpl:145
+//line app/vmselect/prometheus/export.qtpl:146
 func WriteExportPromAPIFooter(qq422016 qtio422016.Writer, qt *querytracer.Tracer) {
-//line app/vmselect/prometheus/export.qtpl:145
+//line app/vmselect/prometheus/export.qtpl:146
 	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vmselect/prometheus/export.qtpl:145
+//line app/vmselect/prometheus/export.qtpl:146
 	StreamExportPromAPIFooter(qw422016, qt)
-//line app/vmselect/prometheus/export.qtpl:145
+//line app/vmselect/prometheus/export.qtpl:146
 	qt422016.ReleaseWriter(qw422016)
-//line app/vmselect/prometheus/export.qtpl:145
+//line app/vmselect/prometheus/export.qtpl:146
 }
 
-//line app/vmselect/prometheus/export.qtpl:145
+//line app/vmselect/prometheus/export.qtpl:146
 func ExportPromAPIFooter(qt *querytracer.Tracer) string {
-//line app/vmselect/prometheus/export.qtpl:145
+//line app/vmselect/prometheus/export.qtpl:146
 	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vmselect/prometheus/export.qtpl:145
+//line app/vmselect/prometheus/export.qtpl:146
 	WriteExportPromAPIFooter(qb422016, qt)
-//line app/vmselect/prometheus/export.qtpl:145
+//line app/vmselect/prometheus/export.qtpl:146
 	qs422016 := string(qb422016.B)
-//line app/vmselect/prometheus/export.qtpl:145
+//line app/vmselect/prometheus/export.qtpl:146
 	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vmselect/prometheus/export.qtpl:145
+//line app/vmselect/prometheus/export.qtpl:146
 	return qs422016
-//line app/vmselect/prometheus/export.qtpl:145
+//line app/vmselect/prometheus/export.qtpl:146
 }
 
-//line app/vmselect/prometheus/export.qtpl:147
-func streamprometheusMetricName(qw422016 *qt422016.Writer, mn *storage.MetricName) {
 //line app/vmselect/prometheus/export.qtpl:148
+func streamprometheusMetricName(qw422016 *qt422016.Writer, mn *storage.MetricName) {
+//line app/vmselect/prometheus/export.qtpl:149
 	qw422016.N().Z(mn.MetricGroup)
-//line app/vmselect/prometheus/export.qtpl:149
+//line app/vmselect/prometheus/export.qtpl:150
 	if len(mn.Tags) > 0 {
-//line app/vmselect/prometheus/export.qtpl:149
+//line app/vmselect/prometheus/export.qtpl:150
 		qw422016.N().S(`{`)
-//line app/vmselect/prometheus/export.qtpl:151
+//line app/vmselect/prometheus/export.qtpl:152
 		tags := mn.Tags
 
-//line app/vmselect/prometheus/export.qtpl:152
-		qw422016.N().Z(tags[0].Key)
-//line app/vmselect/prometheus/export.qtpl:152
-		qw422016.N().S(`=`)
-//line app/vmselect/prometheus/export.qtpl:152
-		streamescapePrometheusLabel(qw422016, tags[0].Value)
 //line app/vmselect/prometheus/export.qtpl:153
+		qw422016.N().Z(tags[0].Key)
+//line app/vmselect/prometheus/export.qtpl:153
+		qw422016.N().S(`=`)
+//line app/vmselect/prometheus/export.qtpl:153
+		streamescapePrometheusLabel(qw422016, tags[0].Value)
+//line app/vmselect/prometheus/export.qtpl:154
 		tags = tags[1:]
 
-//line app/vmselect/prometheus/export.qtpl:154
-		for i := range tags {
 //line app/vmselect/prometheus/export.qtpl:155
+		for i := range tags {
+//line app/vmselect/prometheus/export.qtpl:156
 			tag := &tags[i]
 
-//line app/vmselect/prometheus/export.qtpl:155
+//line app/vmselect/prometheus/export.qtpl:156
 			qw422016.N().S(`,`)
-//line app/vmselect/prometheus/export.qtpl:156
+//line app/vmselect/prometheus/export.qtpl:157
 			qw422016.N().Z(tag.Key)
-//line app/vmselect/prometheus/export.qtpl:156
+//line app/vmselect/prometheus/export.qtpl:157
 			qw422016.N().S(`=`)
-//line app/vmselect/prometheus/export.qtpl:156
+//line app/vmselect/prometheus/export.qtpl:157
 			streamescapePrometheusLabel(qw422016, tag.Value)
-//line app/vmselect/prometheus/export.qtpl:157
+//line app/vmselect/prometheus/export.qtpl:158
 		}
-//line app/vmselect/prometheus/export.qtpl:157
+//line app/vmselect/prometheus/export.qtpl:158
 		qw422016.N().S(`}`)
-//line app/vmselect/prometheus/export.qtpl:159
-	}
 //line app/vmselect/prometheus/export.qtpl:160
+	}
+//line app/vmselect/prometheus/export.qtpl:161
 }
 
-//line app/vmselect/prometheus/export.qtpl:160
+//line app/vmselect/prometheus/export.qtpl:161
 func writeprometheusMetricName(qq422016 qtio422016.Writer, mn *storage.MetricName) {
-//line app/vmselect/prometheus/export.qtpl:160
+//line app/vmselect/prometheus/export.qtpl:161
 	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vmselect/prometheus/export.qtpl:160
+//line app/vmselect/prometheus/export.qtpl:161
 	streamprometheusMetricName(qw422016, mn)
-//line app/vmselect/prometheus/export.qtpl:160
+//line app/vmselect/prometheus/export.qtpl:161
 	qt422016.ReleaseWriter(qw422016)
-//line app/vmselect/prometheus/export.qtpl:160
+//line app/vmselect/prometheus/export.qtpl:161
 }
 
-//line app/vmselect/prometheus/export.qtpl:160
+//line app/vmselect/prometheus/export.qtpl:161
 func prometheusMetricName(mn *storage.MetricName) string {
-//line app/vmselect/prometheus/export.qtpl:160
+//line app/vmselect/prometheus/export.qtpl:161
 	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vmselect/prometheus/export.qtpl:160
+//line app/vmselect/prometheus/export.qtpl:161
 	writeprometheusMetricName(qb422016, mn)
-//line app/vmselect/prometheus/export.qtpl:160
+//line app/vmselect/prometheus/export.qtpl:161
 	qs422016 := string(qb422016.B)
-//line app/vmselect/prometheus/export.qtpl:160
+//line app/vmselect/prometheus/export.qtpl:161
 	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vmselect/prometheus/export.qtpl:160
+//line app/vmselect/prometheus/export.qtpl:161
 	return qs422016
-//line app/vmselect/prometheus/export.qtpl:160
+//line app/vmselect/prometheus/export.qtpl:161
 }
 
-//line app/vmselect/prometheus/export.qtpl:162
+//line app/vmselect/prometheus/export.qtpl:163
 func streamconvertValueToSpecialJSON(qw422016 *qt422016.Writer, v float64) {
-//line app/vmselect/prometheus/export.qtpl:163
+//line app/vmselect/prometheus/export.qtpl:164
 	if math.IsNaN(v) {
-//line app/vmselect/prometheus/export.qtpl:163
+//line app/vmselect/prometheus/export.qtpl:164
 		qw422016.N().S(`null`)
-//line app/vmselect/prometheus/export.qtpl:165
+//line app/vmselect/prometheus/export.qtpl:166
 	} else if math.IsInf(v, 0) {
-//line app/vmselect/prometheus/export.qtpl:166
+//line app/vmselect/prometheus/export.qtpl:167
 		if v > 0 {
-//line app/vmselect/prometheus/export.qtpl:166
+//line app/vmselect/prometheus/export.qtpl:167
 			qw422016.N().S(`"Infinity"`)
-//line app/vmselect/prometheus/export.qtpl:168
+//line app/vmselect/prometheus/export.qtpl:169
 		} else {
-//line app/vmselect/prometheus/export.qtpl:168
+//line app/vmselect/prometheus/export.qtpl:169
 			qw422016.N().S(`"-Infinity"`)
-//line app/vmselect/prometheus/export.qtpl:170
-		}
 //line app/vmselect/prometheus/export.qtpl:171
-	} else {
+		}
 //line app/vmselect/prometheus/export.qtpl:172
-		qw422016.N().F(v)
+	} else {
 //line app/vmselect/prometheus/export.qtpl:173
+		qw422016.N().F(v)
+//line app/vmselect/prometheus/export.qtpl:174
 	}
-//line app/vmselect/prometheus/export.qtpl:174
+//line app/vmselect/prometheus/export.qtpl:175
 }
 
-//line app/vmselect/prometheus/export.qtpl:174
+//line app/vmselect/prometheus/export.qtpl:175
 func writeconvertValueToSpecialJSON(qq422016 qtio422016.Writer, v float64) {
-//line app/vmselect/prometheus/export.qtpl:174
+//line app/vmselect/prometheus/export.qtpl:175
 	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vmselect/prometheus/export.qtpl:174
+//line app/vmselect/prometheus/export.qtpl:175
 	streamconvertValueToSpecialJSON(qw422016, v)
-//line app/vmselect/prometheus/export.qtpl:174
+//line app/vmselect/prometheus/export.qtpl:175
 	qt422016.ReleaseWriter(qw422016)
-//line app/vmselect/prometheus/export.qtpl:174
+//line app/vmselect/prometheus/export.qtpl:175
 }
 
-//line app/vmselect/prometheus/export.qtpl:174
+//line app/vmselect/prometheus/export.qtpl:175
 func convertValueToSpecialJSON(v float64) string {
-//line app/vmselect/prometheus/export.qtpl:174
+//line app/vmselect/prometheus/export.qtpl:175
 	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vmselect/prometheus/export.qtpl:174
+//line app/vmselect/prometheus/export.qtpl:175
 	writeconvertValueToSpecialJSON(qb422016, v)
-//line app/vmselect/prometheus/export.qtpl:174
+//line app/vmselect/prometheus/export.qtpl:175
 	qs422016 := string(qb422016.B)
-//line app/vmselect/prometheus/export.qtpl:174
+//line app/vmselect/prometheus/export.qtpl:175
 	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vmselect/prometheus/export.qtpl:174
+//line app/vmselect/prometheus/export.qtpl:175
 	return qs422016
-//line app/vmselect/prometheus/export.qtpl:174
+//line app/vmselect/prometheus/export.qtpl:175
 }
 
-//line app/vmselect/prometheus/export.qtpl:176
+//line app/vmselect/prometheus/export.qtpl:177
 func streamescapePrometheusLabel(qw422016 *qt422016.Writer, b []byte) {
-//line app/vmselect/prometheus/export.qtpl:176
+//line app/vmselect/prometheus/export.qtpl:177
 	qw422016.N().S(`"`)
-//line app/vmselect/prometheus/export.qtpl:178
-	for len(b) > 0 {
 //line app/vmselect/prometheus/export.qtpl:179
+	for len(b) > 0 {
+//line app/vmselect/prometheus/export.qtpl:180
 		n := bytes.IndexAny(b, "\\\n\"")
 
-//line app/vmselect/prometheus/export.qtpl:180
-		if n < 0 {
 //line app/vmselect/prometheus/export.qtpl:181
-			qw422016.N().Z(b)
+		if n < 0 {
 //line app/vmselect/prometheus/export.qtpl:182
-			break
+			qw422016.N().Z(b)
 //line app/vmselect/prometheus/export.qtpl:183
-		}
+			break
 //line app/vmselect/prometheus/export.qtpl:184
-		qw422016.N().Z(b[:n])
-//line app/vmselect/prometheus/export.qtpl:185
-		switch b[n] {
-//line app/vmselect/prometheus/export.qtpl:186
-		case '\\':
-//line app/vmselect/prometheus/export.qtpl:186
-			qw422016.N().S(`\\`)
-//line app/vmselect/prometheus/export.qtpl:188
-		case '\n':
-//line app/vmselect/prometheus/export.qtpl:188
-			qw422016.N().S(`\n`)
-//line app/vmselect/prometheus/export.qtpl:190
-		case '"':
-//line app/vmselect/prometheus/export.qtpl:190
-			qw422016.N().S(`\"`)
-//line app/vmselect/prometheus/export.qtpl:192
 		}
+//line app/vmselect/prometheus/export.qtpl:185
+		qw422016.N().Z(b[:n])
+//line app/vmselect/prometheus/export.qtpl:186
+		switch b[n] {
+//line app/vmselect/prometheus/export.qtpl:187
+		case '\\':
+//line app/vmselect/prometheus/export.qtpl:187
+			qw422016.N().S(`\\`)
+//line app/vmselect/prometheus/export.qtpl:189
+		case '\n':
+//line app/vmselect/prometheus/export.qtpl:189
+			qw422016.N().S(`\n`)
+//line app/vmselect/prometheus/export.qtpl:191
+		case '"':
+//line app/vmselect/prometheus/export.qtpl:191
+			qw422016.N().S(`\"`)
 //line app/vmselect/prometheus/export.qtpl:193
+		}
+//line app/vmselect/prometheus/export.qtpl:194
 		b = b[n+1:]
 
-//line app/vmselect/prometheus/export.qtpl:194
+//line app/vmselect/prometheus/export.qtpl:195
 	}
-//line app/vmselect/prometheus/export.qtpl:194
+//line app/vmselect/prometheus/export.qtpl:195
 	qw422016.N().S(`"`)
-//line app/vmselect/prometheus/export.qtpl:196
+//line app/vmselect/prometheus/export.qtpl:197
 }
 
-//line app/vmselect/prometheus/export.qtpl:196
+//line app/vmselect/prometheus/export.qtpl:197
 func writeescapePrometheusLabel(qq422016 qtio422016.Writer, b []byte) {
-//line app/vmselect/prometheus/export.qtpl:196
+//line app/vmselect/prometheus/export.qtpl:197
 	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vmselect/prometheus/export.qtpl:196
+//line app/vmselect/prometheus/export.qtpl:197
 	streamescapePrometheusLabel(qw422016, b)
-//line app/vmselect/prometheus/export.qtpl:196
+//line app/vmselect/prometheus/export.qtpl:197
 	qt422016.ReleaseWriter(qw422016)
-//line app/vmselect/prometheus/export.qtpl:196
+//line app/vmselect/prometheus/export.qtpl:197
 }
 
-//line app/vmselect/prometheus/export.qtpl:196
+//line app/vmselect/prometheus/export.qtpl:197
 func escapePrometheusLabel(b []byte) string {
-//line app/vmselect/prometheus/export.qtpl:196
+//line app/vmselect/prometheus/export.qtpl:197
 	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vmselect/prometheus/export.qtpl:196
+//line app/vmselect/prometheus/export.qtpl:197
 	writeescapePrometheusLabel(qb422016, b)
-//line app/vmselect/prometheus/export.qtpl:196
+//line app/vmselect/prometheus/export.qtpl:197
 	qs422016 := string(qb422016.B)
-//line app/vmselect/prometheus/export.qtpl:196
+//line app/vmselect/prometheus/export.qtpl:197
 	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vmselect/prometheus/export.qtpl:196
+//line app/vmselect/prometheus/export.qtpl:197
 	return qs422016
-//line app/vmselect/prometheus/export.qtpl:196
+//line app/vmselect/prometheus/export.qtpl:197
 }

app/vmselect/prometheus/prometheus.go

@@ -47,15 +47,22 @@ var (
 	maxStepForPointsAdjustment = flag.Duration("search.maxStepForPointsAdjustment", time.Minute, "The maximum step when /api/v1/query_range handler adjusts "+
 		"points with timestamps closer than -search.latencyOffset to the current time. The adjustment is needed because such points may contain incomplete data")
 
-	maxUniqueTimeseries    = flag.Int("search.maxUniqueTimeseries", 300e3, "The maximum number of unique time series, which can be selected during /api/v1/query and /api/v1/query_range queries. This option allows limiting memory usage")
-	maxFederateSeries      = flag.Int("search.maxFederateSeries", 1e6, "The maximum number of time series, which can be returned from /federate. This option allows limiting memory usage")
-	maxExportSeries        = flag.Int("search.maxExportSeries", 10e6, "The maximum number of time series, which can be returned from /api/v1/export* APIs. This option allows limiting memory usage")
-	maxTSDBStatusSeries    = flag.Int("search.maxTSDBStatusSeries", 10e6, "The maximum number of time series, which can be processed during the call to /api/v1/status/tsdb. This option allows limiting memory usage")
-	maxSeriesLimit         = flag.Int("search.maxSeries", 30e3, "The maximum number of time series, which can be returned from /api/v1/series. This option allows limiting memory usage")
+	maxUniqueTimeseries = flag.Int("search.maxUniqueTimeseries", 300e3, "The maximum number of unique time series, which can be selected during /api/v1/query and /api/v1/query_range queries. This option allows limiting memory usage")
+	maxFederateSeries   = flag.Int("search.maxFederateSeries", 1e6, "The maximum number of time series, which can be returned from /federate. This option allows limiting memory usage")
+	maxExportSeries     = flag.Int("search.maxExportSeries", 10e6, "The maximum number of time series, which can be returned from /api/v1/export* APIs. This option allows limiting memory usage")
+	maxTSDBStatusSeries = flag.Int("search.maxTSDBStatusSeries", 10e6, "The maximum number of time series, which can be processed during the call to /api/v1/status/tsdb. This option allows limiting memory usage")
+	maxSeriesLimit      = flag.Int("search.maxSeries", 30e3, "The maximum number of time series, which can be returned from /api/v1/series. This option allows limiting memory usage")
+	maxLabelsAPISeries  = flag.Int("search.maxLabelsAPISeries", 1e6, "The maximum number of time series, which could be scanned when searching for the the matching time series "+
+		"at /api/v1/labels and /api/v1/label/.../values. This option allows limiting memory usage and CPU usage. See also -search.maxLabelsAPIDuration, "+
+		"-search.maxTagKeys, -search.maxTagValues and -search.ignoreExtraFiltersAtLabelsAPI")
 	maxPointsPerTimeseries = flag.Int("search.maxPointsPerTimeseries", 30e3, "The maximum points per a single timeseries returned from /api/v1/query_range. "+
 		"This option doesn't limit the number of scanned raw samples in the database. The main purpose of this option is to limit the number of per-series points "+
 		"returned to graphing UI such as VMUI or Grafana. There is no sense in setting this limit to values bigger than the horizontal resolution of the graph. "+
 		"See also -search.maxResponseSeries")
+	ignoreExtraFiltersAtLabelsAPI = flag.Bool("search.ignoreExtraFiltersAtLabelsAPI", false, "Whether to ignore match[], extra_filters[] and extra_label query args at "+
+		"/api/v1/labels and /api/v1/label/.../values . This may be useful for decreasing load on VictoriaMetrics when extra filters "+
+		"match too many time series. The downside is that suprflouos labels or series could be returned, which do not match the extra filters. "+
+		"See also -search.maxLabelsAPISeries and -search.maxLabelsAPIDuration")
 )
 
 // Default step used if not set.
@@ -244,7 +251,7 @@ func ExportNativeHandler(startTime time.Time, w http.ResponseWriter, r *http.Req
 	_, _ = bw.Write(trBuf)
 
 	// Marshal native blocks.
-	err = netstorage.ExportBlocks(nil, sq, cp.deadline, func(mn *storage.MetricName, b *storage.Block, tr storage.TimeRange, workerID uint) error {
+	err = netstorage.ExportBlocks(nil, sq, cp.deadline, func(mn *storage.MetricName, b *storage.Block, _ storage.TimeRange, workerID uint) error {
 		if err := bw.Error(); err != nil {
 			return err
 		}
@@ -317,21 +324,21 @@ func exportHandler(qt *querytracer.Tracer, w http.ResponseWriter, cp *commonPara
 		}
 	} else if format == "promapi" {
 		WriteExportPromAPIHeader(bw)
-		firstLineOnce := uint32(0)
-		firstLineSent := uint32(0)
+		var firstLineOnce atomic.Bool
+		var firstLineSent atomic.Bool
 		writeLineFunc = func(xb *exportBlock, workerID uint) error {
 			bb := sw.getBuffer(workerID)
-			// Use atomic.LoadUint32() in front of atomic.CompareAndSwapUint32() in order to avoid slow inter-CPU synchronization
+			// Use Load() in front of CompareAndSwap() in order to avoid slow inter-CPU synchronization
 			// in fast path after the first line has been already sent.
-			if atomic.LoadUint32(&firstLineOnce) == 0 && atomic.CompareAndSwapUint32(&firstLineOnce, 0, 1) {
+			if !firstLineOnce.Load() && firstLineOnce.CompareAndSwap(false, true) {
 				// Send the first line to sw.bw
 				WriteExportPromAPILine(bb, xb)
 				_, err := sw.bw.Write(bb.B)
 				bb.Reset()
-				atomic.StoreUint32(&firstLineSent, 1)
+				firstLineSent.Store(true)
 				return err
 			}
-			for atomic.LoadUint32(&firstLineSent) == 0 {
+			for !firstLineSent.Load() {
 				// Busy wait until the first line is sent to sw.bw
 				runtime.Gosched()
 			}
@@ -491,7 +498,7 @@ var deleteDuration = metrics.NewSummary(`vm_request_duration_seconds{path="/api/
 func LabelValuesHandler(qt *querytracer.Tracer, startTime time.Time, labelName string, w http.ResponseWriter, r *http.Request) error {
 	defer labelValuesDuration.UpdateDuration(startTime)
 
-	cp, err := getCommonParamsWithDefaultDuration(r, startTime, false)
+	cp, err := getCommonParamsForLabelsAPI(r, startTime, false)
 	if err != nil {
 		return err
 	}
@@ -499,10 +506,7 @@ func LabelValuesHandler(qt *querytracer.Tracer, startTime time.Time, labelName s
 	if err != nil {
 		return err
 	}
-	// Do not limit the number of unique time series, which could be scanned
-	// during the search for matching label values, since users expect this API
-	// must always work.
-	sq := storage.NewSearchQuery(cp.start, cp.end, cp.filterss, -1)
+	sq := storage.NewSearchQuery(cp.start, cp.end, cp.filterss, *maxLabelsAPISeries)
 	labelValues, err := netstorage.LabelValues(qt, labelName, sq, limit, cp.deadline)
 	if err != nil {
 		return fmt.Errorf("cannot obtain values for label %q: %w", labelName, err)
@@ -591,7 +595,7 @@ var tsdbStatusDuration = metrics.NewSummary(`vm_request_duration_seconds{path="/
 func LabelsHandler(qt *querytracer.Tracer, startTime time.Time, w http.ResponseWriter, r *http.Request) error {
 	defer labelsDuration.UpdateDuration(startTime)
 
-	cp, err := getCommonParamsWithDefaultDuration(r, startTime, false)
+	cp, err := getCommonParamsForLabelsAPI(r, startTime, false)
 	if err != nil {
 		return err
 	}
@@ -599,10 +603,7 @@ func LabelsHandler(qt *querytracer.Tracer, startTime time.Time, w http.ResponseW
 	if err != nil {
 		return err
 	}
-	// Do not limit the number of unique time series, which could be scanned
-	// during the search for matching label values, since users expect this API
-	// must always work.
-	sq := storage.NewSearchQuery(cp.start, cp.end, cp.filterss, -1)
+	sq := storage.NewSearchQuery(cp.start, cp.end, cp.filterss, *maxLabelsAPISeries)
 	labels, err := netstorage.LabelNames(qt, sq, limit, cp.deadline)
 	if err != nil {
 		return fmt.Errorf("cannot obtain labels: %w", err)
@@ -647,12 +648,12 @@ var seriesCountDuration = metrics.NewSummary(`vm_request_duration_seconds{path="
 func SeriesHandler(qt *querytracer.Tracer, startTime time.Time, w http.ResponseWriter, r *http.Request) error {
 	defer seriesDuration.UpdateDuration(startTime)
 
-	// Do not set start to searchutils.minTimeMsecs by default as Prometheus does,
+	// Do not set start to httputils.minTimeMsecs by default as Prometheus does,
 	// since this leads to fetching and scanning all the data from the storage,
 	// which can take a lot of time for big storages.
 	// It is better setting start as end-defaultStep by default.
 	// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/91
-	cp, err := getCommonParamsWithDefaultDuration(r, startTime, true)
+	cp, err := getCommonParamsForLabelsAPI(r, startTime, true)
 	if err != nil {
 		return err
 	}
@@ -1112,7 +1113,7 @@ func (cp *commonParams) IsDefaultTimeRange() bool {
 	return cp.start == 0 && cp.currentTimestamp-cp.end < 1000
 }
 
-// getCommonParams obtains common params from r, which are used in /api/v1/export* handlers
+// getExportParams obtains common params from r, which are used in /api/v1/export* handlers
 //
 // - timeout
 // - start
@@ -1129,14 +1130,15 @@ func getExportParams(r *http.Request, startTime time.Time) (*commonParams, error
 	return cp, nil
 }
 
-func getCommonParamsWithDefaultDuration(r *http.Request, startTime time.Time, requireNonEmptyMatch bool) (*commonParams, error) {
-	cp, err := getCommonParams(r, startTime, requireNonEmptyMatch)
+func getCommonParamsForLabelsAPI(r *http.Request, startTime time.Time, requireNonEmptyMatch bool) (*commonParams, error) {
+	cp, err := getCommonParamsInternal(r, startTime, requireNonEmptyMatch, true)
 	if err != nil {
 		return nil, err
 	}
 	if cp.start == 0 {
 		cp.start = cp.end - defaultStep
 	}
+	cp.deadline = searchutils.GetDeadlineForLabelsAPI(r, startTime)
 	return cp, nil
 }
 
@@ -1149,6 +1151,10 @@ func getCommonParamsWithDefaultDuration(r *http.Request, startTime time.Time, re
 // - extra_label
 // - extra_filters[]
 func getCommonParams(r *http.Request, startTime time.Time, requireNonEmptyMatch bool) (*commonParams, error) {
+	return getCommonParamsInternal(r, startTime, requireNonEmptyMatch, false)
+}
+
+func getCommonParamsInternal(r *http.Request, startTime time.Time, requireNonEmptyMatch, isLabelsAPI bool) (*commonParams, error) {
 	deadline := searchutils.GetDeadlineForQuery(r, startTime)
 	start, err := httputils.GetTime(r, "start", 0)
 	if err != nil {
@@ -1175,15 +1181,23 @@ func getCommonParams(r *http.Request, startTime time.Time, requireNonEmptyMatch
 	if requireNonEmptyMatch && len(matches) == 0 {
 		return nil, fmt.Errorf("missing `match[]` arg")
 	}
-	tagFilterss, err := getTagFilterssFromMatches(matches)
+	filterss, err := getTagFilterssFromMatches(matches)
 	if err != nil {
 		return nil, err
 	}
-	etfs, err := searchutils.GetExtraTagFilters(r)
-	if err != nil {
-		return nil, err
+
+	if len(filterss) > 0 || !isLabelsAPI || !*ignoreExtraFiltersAtLabelsAPI {
+		// If matches isn't empty, then there is no sense in ignoring extra filters
+		// even if ignoreExtraLabelsAtLabelsAPI is set, since extra filters won't slow down
+		// the query - they can only improve query performance by reducing the number
+		// of matching series at the storage level.
+		etfs, err := searchutils.GetExtraTagFilters(r)
+		if err != nil {
+			return nil, err
+		}
+		filterss = searchutils.JoinTagFilterss(filterss, etfs)
 	}
-	filterss := searchutils.JoinTagFilterss(tagFilterss, etfs)
+
 	cp := &commonParams{
 		deadline:         deadline,
 		start:            start,
@@ -1224,7 +1238,7 @@ func (sw *scalableWriter) maybeFlushBuffer(bb *bytesutil.ByteBuffer) error {
 }
 
 func (sw *scalableWriter) flush() error {
-	sw.m.Range(func(k, v interface{}) bool {
+	sw.m.Range(func(_, v interface{}) bool {
 		bb := v.(*bytesutil.ByteBuffer)
 		_, err := sw.bw.Write(bb.B)
 		return err == nil

app/vmselect/prometheus/query_range_response.qtpl

@@ -33,8 +33,8 @@ See https://prometheus.io/docs/prometheus/latest/querying/api/#range-queries
 			// seriesFetched is string instead of int because of historical reasons.
 			// It cannot be converted to int without breaking backwards compatibility at vmalert :(
 		%}
-		"seriesFetched": "{%dl qs.SeriesFetched %}",
-		"executionTimeMsec": {%dl qs.ExecutionTimeMsec %}
+		"seriesFetched": "{%dl qs.SeriesFetched.Load() %}",
+		"executionTimeMsec": {%dl qs.ExecutionTimeMsec.Load() %}
 	}
 	{% code
 		qt.Printf("generate /api/v1/query_range response for series=%d, points=%d", seriesCount, pointsCount)

app/vmselect/prometheus/query_range_response.qtpl.go

@@ -68,11 +68,11 @@ func StreamQueryRangeResponse(qw422016 *qt422016.Writer, rs []netstorage.Result,
 //line app/vmselect/prometheus/query_range_response.qtpl:35
 	qw422016.N().S(`"seriesFetched": "`)
 //line app/vmselect/prometheus/query_range_response.qtpl:36
-	qw422016.N().DL(qs.SeriesFetched)
+	qw422016.N().DL(qs.SeriesFetched.Load())
 //line app/vmselect/prometheus/query_range_response.qtpl:36
 	qw422016.N().S(`","executionTimeMsec":`)
 //line app/vmselect/prometheus/query_range_response.qtpl:37
-	qw422016.N().DL(qs.ExecutionTimeMsec)
+	qw422016.N().DL(qs.ExecutionTimeMsec.Load())
 //line app/vmselect/prometheus/query_range_response.qtpl:37
 	qw422016.N().S(`}`)
 //line app/vmselect/prometheus/query_range_response.qtpl:40

app/vmselect/prometheus/query_response.qtpl

@@ -35,8 +35,8 @@ See https://prometheus.io/docs/prometheus/latest/querying/api/#instant-queries
 			// seriesFetched is string instead of int because of historical reasons.
 			// It cannot be converted to int without breaking backwards compatibility at vmalert :(
 		%}
-		"seriesFetched": "{%dl qs.SeriesFetched %}",
-		"executionTimeMsec": {%dl qs.ExecutionTimeMsec %}
+		"seriesFetched": "{%dl qs.SeriesFetched.Load() %}",
+		"executionTimeMsec": {%dl qs.ExecutionTimeMsec.Load() %}
 	}
 	{% code
 		qt.Printf("generate /api/v1/query response for series=%d", seriesCount)

app/vmselect/prometheus/query_response.qtpl.go

@@ -78,11 +78,11 @@ func StreamQueryResponse(qw422016 *qt422016.Writer, rs []netstorage.Result, qt *
 //line app/vmselect/prometheus/query_response.qtpl:37
 	qw422016.N().S(`"seriesFetched": "`)
 //line app/vmselect/prometheus/query_response.qtpl:38
-	qw422016.N().DL(qs.SeriesFetched)
+	qw422016.N().DL(qs.SeriesFetched.Load())
 //line app/vmselect/prometheus/query_response.qtpl:38
 	qw422016.N().S(`","executionTimeMsec":`)
 //line app/vmselect/prometheus/query_response.qtpl:39
-	qw422016.N().DL(qs.ExecutionTimeMsec)
+	qw422016.N().DL(qs.ExecutionTimeMsec.Load())
 //line app/vmselect/prometheus/query_response.qtpl:39
 	qw422016.N().S(`}`)
 //line app/vmselect/prometheus/query_response.qtpl:42

app/vmselect/promql/active_queries.go

@@ -60,7 +60,7 @@ func (aq *activeQueries) Add(ec *EvalConfig, q string) uint64 {
 	aqe.start = ec.Start
 	aqe.end = ec.End
 	aqe.step = ec.Step
-	aqe.qid = atomic.AddUint64(&nextActiveQueryID, 1)
+	aqe.qid = nextActiveQueryID.Add(1)
 	aqe.quotedRemoteAddr = ec.QuotedRemoteAddr
 	aqe.q = q
 	aqe.startTime = time.Now()
@@ -87,4 +87,8 @@ func (aq *activeQueries) GetAll() []activeQueryEntry {
 	return aqes
 }
 
-var nextActiveQueryID = uint64(time.Now().UnixNano())
+var nextActiveQueryID = func() *atomic.Uint64 {
+	var x atomic.Uint64
+	x.Store(uint64(time.Now().UnixNano()))
+	return &x
+}()

app/vmselect/promql/aggr.go

@@ -76,7 +76,7 @@ func newAggrFunc(afe func(tss []*timeseries) []*timeseries) aggrFunc {
 		if err != nil {
 			return nil, err
 		}
-		return aggrFuncExt(func(tss []*timeseries, modififer *metricsql.ModifierExpr) []*timeseries {
+		return aggrFuncExt(func(tss []*timeseries, _ *metricsql.ModifierExpr) []*timeseries {
 			return afe(tss)
 		}, tss, &afa.ae.Modifier, afa.ae.Limit, false)
 	}
@@ -158,7 +158,7 @@ func aggrFuncAny(afa *aggrFuncArg) ([]*timeseries, error) {
 	if err != nil {
 		return nil, err
 	}
-	afe := func(tss []*timeseries, modifier *metricsql.ModifierExpr) []*timeseries {
+	afe := func(tss []*timeseries, _ *metricsql.ModifierExpr) []*timeseries {
 		return tss[:1]
 	}
 	limit := afa.ae.Limit
@@ -467,7 +467,7 @@ func aggrFuncShare(afa *aggrFuncArg) ([]*timeseries, error) {
 	if err != nil {
 		return nil, err
 	}
-	afe := func(tss []*timeseries, modifier *metricsql.ModifierExpr) []*timeseries {
+	afe := func(tss []*timeseries, _ *metricsql.ModifierExpr) []*timeseries {
 		for i := range tss[0].Values {
 			// Calculate sum for non-negative points at position i.
 			var sum float64
@@ -498,7 +498,7 @@ func aggrFuncZScore(afa *aggrFuncArg) ([]*timeseries, error) {
 	if err != nil {
 		return nil, err
 	}
-	afe := func(tss []*timeseries, modifier *metricsql.ModifierExpr) []*timeseries {
+	afe := func(tss []*timeseries, _ *metricsql.ModifierExpr) []*timeseries {
 		for i := range tss[0].Values {
 			// Calculate avg and stddev for tss points at position i.
 			// See `Rapid calculation methods` at https://en.wikipedia.org/wiki/Standard_deviation
@@ -594,7 +594,7 @@ func aggrFuncCountValues(afa *aggrFuncArg) ([]*timeseries, error) {
 		// Do nothing
 	}
 
-	afe := func(tss []*timeseries, modififer *metricsql.ModifierExpr) ([]*timeseries, error) {
+	afe := func(tss []*timeseries, _ *metricsql.ModifierExpr) ([]*timeseries, error) {
 		m := make(map[float64]*timeseries)
 		for _, ts := range tss {
 			for i, v := range ts.Values {
@@ -656,7 +656,7 @@ func newAggrFuncTopK(isReverse bool) aggrFunc {
 		if err != nil {
 			return nil, err
 		}
-		afe := func(tss []*timeseries, modififer *metricsql.ModifierExpr) []*timeseries {
+		afe := func(tss []*timeseries, _ *metricsql.ModifierExpr) []*timeseries {
 			for n := range tss[0].Values {
 				lessFunc := lessWithNaNs
 				if isReverse {
@@ -960,7 +960,7 @@ func aggrFuncOutliersIQR(afa *aggrFuncArg) ([]*timeseries, error) {
 	if err := expectTransformArgsNum(args, 1); err != nil {
 		return nil, err
 	}
-	afe := func(tss []*timeseries, modifier *metricsql.ModifierExpr) []*timeseries {
+	afe := func(tss []*timeseries, _ *metricsql.ModifierExpr) []*timeseries {
 		// Calculate lower and upper bounds for interquartile range per each point across tss
 		// according to Outliers section at https://en.wikipedia.org/wiki/Interquartile_range
 		lower, upper := getPerPointIQRBounds(tss)
@@ -1016,7 +1016,7 @@ func aggrFuncOutliersMAD(afa *aggrFuncArg) ([]*timeseries, error) {
 	if err != nil {
 		return nil, err
 	}
-	afe := func(tss []*timeseries, modifier *metricsql.ModifierExpr) []*timeseries {
+	afe := func(tss []*timeseries, _ *metricsql.ModifierExpr) []*timeseries {
 		// Calculate medians for each point across tss.
 		medians := getPerPointMedians(tss)
 		// Calculate MAD values multiplied by tolerance for each point across tss.
@@ -1052,7 +1052,7 @@ func aggrFuncOutliersK(afa *aggrFuncArg) ([]*timeseries, error) {
 	if err != nil {
 		return nil, err
 	}
-	afe := func(tss []*timeseries, modifier *metricsql.ModifierExpr) []*timeseries {
+	afe := func(tss []*timeseries, _ *metricsql.ModifierExpr) []*timeseries {
 		// Calculate medians for each point across tss.
 		medians := getPerPointMedians(tss)
 		// Return topK time series with the highest variance from median.
@@ -1123,7 +1123,7 @@ func aggrFuncLimitK(afa *aggrFuncArg) ([]*timeseries, error) {
 	if limit < 0 {
 		limit = 0
 	}
-	afe := func(tss []*timeseries, modifier *metricsql.ModifierExpr) []*timeseries {
+	afe := func(tss []*timeseries, _ *metricsql.ModifierExpr) []*timeseries {
 		// Sort series by metricName hash in order to get consistent set of output series
 		// across multiple calls to limitk() function.
 		// Sort series by hash in order to guarantee uniform selection across series.
@@ -1187,7 +1187,7 @@ func aggrFuncQuantiles(afa *aggrFuncArg) ([]*timeseries, error) {
 		phis[i] = phisLocal[0]
 	}
 	argOrig := args[len(args)-1]
-	afe := func(tss []*timeseries, modifier *metricsql.ModifierExpr) []*timeseries {
+	afe := func(tss []*timeseries, _ *metricsql.ModifierExpr) []*timeseries {
 		tssDst := make([]*timeseries, len(phiArgs))
 		for j := range tssDst {
 			ts := &timeseries{}
@@ -1244,7 +1244,7 @@ func aggrFuncMedian(afa *aggrFuncArg) ([]*timeseries, error) {
 }
 
 func newAggrQuantileFunc(phis []float64) func(tss []*timeseries, modifier *metricsql.ModifierExpr) []*timeseries {
-	return func(tss []*timeseries, modifier *metricsql.ModifierExpr) []*timeseries {
+	return func(tss []*timeseries, _ *metricsql.ModifierExpr) []*timeseries {
 		dst := tss[0]
 		a := getFloat64s()
 		values := a.A

app/vmselect/promql/binary_op.go

@@ -74,7 +74,7 @@ func newBinaryOpCmpFunc(cf func(left, right float64) bool) binaryOpFunc {
 }
 
 func newBinaryOpArithFunc(af func(left, right float64) float64) binaryOpFunc {
-	afe := func(left, right float64, isBool bool) float64 {
+	afe := func(left, right float64, _ bool) float64 {
 		return af(left, right)
 	}
 	return newBinaryOpFunc(afe)

app/vmselect/promql/eval.go

@@ -111,11 +111,6 @@ type EvalConfig struct {
 	End   int64
 	Step  int64
 
-	// RealStart contains the original start of the interval when executed in subqueries (because Start can be changed  for subqueries)
-	RealStart int64
-	// RealEnd contains the original end of the interval when executed in subqueries (because End can be changed for subqueries)
-	RealEnd int64
-
 	// MaxSeries is the maximum number of time series, which can be scanned by the query.
 	// Zero means 'no limit'
 	MaxSeries int
@@ -157,17 +152,7 @@ type EvalConfig struct {
 func copyEvalConfig(src *EvalConfig) *EvalConfig {
 	var ec EvalConfig
 	ec.Start = src.Start
-	if ec.RealStart > 0 {
-		ec.RealStart = src.RealStart
-	} else {
-		ec.RealStart = src.Start
-	}
 	ec.End = src.End
-	if ec.RealEnd > 0 {
-		ec.RealEnd = src.RealEnd
-	} else {
-		ec.RealEnd = src.End
-	}
 	ec.Step = src.Step
 	ec.MaxSeries = src.MaxSeries
 	ec.MaxPointsPerSeries = src.MaxPointsPerSeries
@@ -186,16 +171,17 @@ func copyEvalConfig(src *EvalConfig) *EvalConfig {
 // QueryStats contains various stats for the query.
 type QueryStats struct {
 	// SeriesFetched contains the number of series fetched from storage during the query evaluation.
-	SeriesFetched int64
+	SeriesFetched atomic.Int64
+
 	// ExecutionTimeMsec contains the number of milliseconds the query took to execute.
-	ExecutionTimeMsec int64
+	ExecutionTimeMsec atomic.Int64
 }
 
 func (qs *QueryStats) addSeriesFetched(n int) {
 	if qs == nil {
 		return
 	}
-	atomic.AddInt64(&qs.SeriesFetched, int64(n))
+	qs.SeriesFetched.Add(int64(n))
 }
 
 func (qs *QueryStats) addExecutionTimeMsec(startTime time.Time) {
@@ -203,7 +189,7 @@ func (qs *QueryStats) addExecutionTimeMsec(startTime time.Time) {
 		return
 	}
 	d := time.Since(startTime).Milliseconds()
-	atomic.AddInt64(&qs.ExecutionTimeMsec, d)
+	qs.ExecutionTimeMsec.Add(d)
 }
 
 func (ec *EvalConfig) validate() {
@@ -328,7 +314,7 @@ func evalExprInternal(qt *querytracer.Tracer, ec *EvalConfig, e metricsql.Expr)
 		}
 		rf, err := nrf(args)
 		if err != nil {
-			return nil, err
+			return nil, fmt.Errorf("cannot evaluate args for %q: %w", fe.AppendString(nil), err)
 		}
 		rv, err := evalRollupFunc(qt, ec, fe.Name, rf, e, re, nil)
 		if err != nil {
@@ -409,7 +395,7 @@ func evalAggrFunc(qt *querytracer.Tracer, ec *EvalConfig, ae *metricsql.AggrFunc
 			}
 			rf, err := nrf(args)
 			if err != nil {
-				return nil, err
+				return nil, fmt.Errorf("cannot evaluate args for aggregate func %q: %w", ae.AppendString(nil), err)
 			}
 			iafc := newIncrementalAggrFuncContext(ae, callbacks)
 			return evalRollupFunc(qt, ec, fe.Name, rf, ae, re, iafc)
@@ -964,7 +950,7 @@ func evalRollupFuncWithSubquery(qt *querytracer.Tracer, ec *EvalConfig, funcName
 		return nil, err
 	}
 
-	var samplesScannedTotal uint64
+	var samplesScannedTotal atomic.Uint64
 	keepMetricNames := getKeepMetricNames(expr)
 	tsw := getTimeseriesByWorkerID()
 	seriesByWorkerID := tsw.byWorkerID
@@ -974,13 +960,13 @@ func evalRollupFuncWithSubquery(qt *querytracer.Tracer, ec *EvalConfig, funcName
 		for _, rc := range rcs {
 			if tsm := newTimeseriesMap(funcName, keepMetricNames, sharedTimestamps, &tsSQ.MetricName); tsm != nil {
 				samplesScanned := rc.DoTimeseriesMap(tsm, values, timestamps)
-				atomic.AddUint64(&samplesScannedTotal, samplesScanned)
+				samplesScannedTotal.Add(samplesScanned)
 				seriesByWorkerID[workerID].tss = tsm.AppendTimeseriesTo(seriesByWorkerID[workerID].tss)
 				continue
 			}
 			var ts timeseries
 			samplesScanned := doRollupForTimeseries(funcName, keepMetricNames, rc, &ts, &tsSQ.MetricName, values, timestamps, sharedTimestamps)
-			atomic.AddUint64(&samplesScannedTotal, samplesScanned)
+			samplesScannedTotal.Add(samplesScanned)
 			seriesByWorkerID[workerID].tss = append(seriesByWorkerID[workerID].tss, &ts)
 		}
 		return values, timestamps
@@ -991,8 +977,8 @@ func evalRollupFuncWithSubquery(qt *querytracer.Tracer, ec *EvalConfig, funcName
 	}
 	putTimeseriesByWorkerID(tsw)
 
-	rowsScannedPerQuery.Update(float64(samplesScannedTotal))
-	qt.Printf("rollup %s() over %d series returned by subquery: series=%d, samplesScanned=%d", funcName, len(tssSQ), len(tss), samplesScannedTotal)
+	rowsScannedPerQuery.Update(float64(samplesScannedTotal.Load()))
+	qt.Printf("rollup %s() over %d series returned by subquery: series=%d, samplesScanned=%d", funcName, len(tssSQ), len(tss), samplesScannedTotal.Load())
 	return tss, nil
 }
 
@@ -1702,7 +1688,7 @@ func evalRollupFuncNoCache(qt *querytracer.Tracer, ec *EvalConfig, funcName stri
 	tfss := searchutils.ToTagFilterss(me.LabelFilterss)
 	tfss = searchutils.JoinTagFilterss(tfss, ec.EnforcedTagFilterss)
 	minTimestamp := ec.Start
-	if needSilenceIntervalForRollupFunc(funcName) {
+	if needSilenceIntervalForRollupFunc[funcName] {
 		minTimestamp -= maxSilenceInterval()
 	}
 	if window > ec.Step {
@@ -1803,62 +1789,12 @@ func maxSilenceInterval() int64 {
 	return d
 }
 
-func needSilenceIntervalForRollupFunc(funcName string) bool {
-	// All the rollup functions, which do not rely on the previous sample
-	// before the lookbehind window (aka prevValue and realPrevValue), do not need silence interval.
-	switch strings.ToLower(funcName) {
-	case "default_rollup":
-		// The default_rollup implicitly relies on the previous samples in order to fill gaps.
-		// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5388
-		return true
-	case
-		"absent_over_time",
-		"avg_over_time",
-		"count_eq_over_time",
-		"count_gt_over_time",
-		"count_le_over_time",
-		"count_ne_over_time",
-		"count_over_time",
-		"first_over_time",
-		"histogram_over_time",
-		"hoeffding_bound_lower",
-		"hoeffding_bound_upper",
-		"last_over_time",
-		"mad_over_time",
-		"max_over_time",
-		"median_over_time",
-		"min_over_time",
-		"predict_linear",
-		"present_over_time",
-		"quantile_over_time",
-		"quantiles_over_time",
-		"range_over_time",
-		"share_gt_over_time",
-		"share_le_over_time",
-		"share_eq_over_time",
-		"stale_samples_over_time",
-		"stddev_over_time",
-		"stdvar_over_time",
-		"sum_over_time",
-		"tfirst_over_time",
-		"timestamp",
-		"timestamp_with_name",
-		"tlast_over_time",
-		"tmax_over_time",
-		"tmin_over_time",
-		"zscore_over_time":
-		return false
-	default:
-		return true
-	}
-}
-
 func evalRollupWithIncrementalAggregate(qt *querytracer.Tracer, funcName string, keepMetricNames bool,
 	iafc *incrementalAggrFuncContext, rss *netstorage.Results, rcs []*rollupConfig,
 	preFunc func(values []float64, timestamps []int64), sharedTimestamps []int64) ([]*timeseries, error) {
 	qt = qt.NewChild("rollup %s() with incremental aggregation %s() over %d series; rollupConfigs=%s", funcName, iafc.ae.Name, rss.Len(), rcs)
 	defer qt.Done()
-	var samplesScannedTotal uint64
+	var samplesScannedTotal atomic.Uint64
 	err := rss.RunParallel(qt, func(rs *netstorage.Result, workerID uint) error {
 		rs.Values, rs.Timestamps = dropStaleNaNs(funcName, rs.Values, rs.Timestamps)
 		preFunc(rs.Values, rs.Timestamps)
@@ -1870,12 +1806,12 @@ func evalRollupWithIncrementalAggregate(qt *querytracer.Tracer, funcName string,
 				for _, ts := range tsm.m {
 					iafc.updateTimeseries(ts, workerID)
 				}
-				atomic.AddUint64(&samplesScannedTotal, samplesScanned)
+				samplesScannedTotal.Add(samplesScanned)
 				continue
 			}
 			ts.Reset()
 			samplesScanned := doRollupForTimeseries(funcName, keepMetricNames, rc, ts, &rs.MetricName, rs.Values, rs.Timestamps, sharedTimestamps)
-			atomic.AddUint64(&samplesScannedTotal, samplesScanned)
+			samplesScannedTotal.Add(samplesScanned)
 			iafc.updateTimeseries(ts, workerID)
 
 			// ts.Timestamps points to sharedTimestamps. Zero it, so it can be re-used.
@@ -1888,8 +1824,8 @@ func evalRollupWithIncrementalAggregate(qt *querytracer.Tracer, funcName string,
 		return nil, err
 	}
 	tss := iafc.finalizeTimeseries()
-	rowsScannedPerQuery.Update(float64(samplesScannedTotal))
-	qt.Printf("series after aggregation with %s(): %d; samplesScanned=%d", iafc.ae.Name, len(tss), samplesScannedTotal)
+	rowsScannedPerQuery.Update(float64(samplesScannedTotal.Load()))
+	qt.Printf("series after aggregation with %s(): %d; samplesScanned=%d", iafc.ae.Name, len(tss), samplesScannedTotal.Load())
 	return tss, nil
 }
 
@@ -1898,7 +1834,7 @@ func evalRollupNoIncrementalAggregate(qt *querytracer.Tracer, funcName string, k
 	qt = qt.NewChild("rollup %s() over %d series; rollupConfigs=%s", funcName, rss.Len(), rcs)
 	defer qt.Done()
 
-	var samplesScannedTotal uint64
+	var samplesScannedTotal atomic.Uint64
 	tsw := getTimeseriesByWorkerID()
 	seriesByWorkerID := tsw.byWorkerID
 	seriesLen := rss.Len()
@@ -1908,13 +1844,13 @@ func evalRollupNoIncrementalAggregate(qt *querytracer.Tracer, funcName string, k
 		for _, rc := range rcs {
 			if tsm := newTimeseriesMap(funcName, keepMetricNames, sharedTimestamps, &rs.MetricName); tsm != nil {
 				samplesScanned := rc.DoTimeseriesMap(tsm, rs.Values, rs.Timestamps)
-				atomic.AddUint64(&samplesScannedTotal, samplesScanned)
+				samplesScannedTotal.Add(samplesScanned)
 				seriesByWorkerID[workerID].tss = tsm.AppendTimeseriesTo(seriesByWorkerID[workerID].tss)
 				continue
 			}
 			var ts timeseries
 			samplesScanned := doRollupForTimeseries(funcName, keepMetricNames, rc, &ts, &rs.MetricName, rs.Values, rs.Timestamps, sharedTimestamps)
-			atomic.AddUint64(&samplesScannedTotal, samplesScanned)
+			samplesScannedTotal.Add(samplesScanned)
 			seriesByWorkerID[workerID].tss = append(seriesByWorkerID[workerID].tss, &ts)
 		}
 		return nil
@@ -1928,8 +1864,8 @@ func evalRollupNoIncrementalAggregate(qt *querytracer.Tracer, funcName string, k
 	}
 	putTimeseriesByWorkerID(tsw)
 
-	rowsScannedPerQuery.Update(float64(samplesScannedTotal))
-	qt.Printf("samplesScanned=%d", samplesScannedTotal)
+	rowsScannedPerQuery.Update(float64(samplesScannedTotal.Load()))
+	qt.Printf("samplesScanned=%d", samplesScannedTotal.Load())
 	return tss, nil
 }
 

app/vmselect/promql/eval_test.go

@@ -86,14 +86,14 @@ func TestQueryStats_addSeriesFetched(t *testing.T) {
 	}
 	ec.QueryStats.addSeriesFetched(1)
 
-	if qs.SeriesFetched != 1 {
-		t.Fatalf("expected to get 1; got %d instead", qs.SeriesFetched)
+	if n := qs.SeriesFetched.Load(); n != 1 {
+		t.Fatalf("expected to get 1; got %d instead", n)
 	}
 
 	ecNew := copyEvalConfig(ec)
 	ecNew.QueryStats.addSeriesFetched(3)
-	if qs.SeriesFetched != 4 {
-		t.Fatalf("expected to get 4; got %d instead", qs.SeriesFetched)
+	if n := qs.SeriesFetched.Load(); n != 4 {
+		t.Fatalf("expected to get 4; got %d instead", n)
 	}
 }
 

app/vmselect/promql/exec.go

@@ -352,22 +352,19 @@ type parseCacheValue struct {
 }
 
 type parseCache struct {
-	// Move atomic counters to the top of struct for 8-byte alignment on 32-bit arch.
-	// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
-
-	requests uint64
-	misses   uint64
+	requests atomic.Uint64
+	misses   atomic.Uint64
 
 	m  map[string]*parseCacheValue
 	mu sync.RWMutex
 }
 
 func (pc *parseCache) Requests() uint64 {
-	return atomic.LoadUint64(&pc.requests)
+	return pc.requests.Load()
 }
 
 func (pc *parseCache) Misses() uint64 {
-	return atomic.LoadUint64(&pc.misses)
+	return pc.misses.Load()
 }
 
 func (pc *parseCache) Len() uint64 {
@@ -378,14 +375,14 @@ func (pc *parseCache) Len() uint64 {
 }
 
 func (pc *parseCache) Get(q string) *parseCacheValue {
-	atomic.AddUint64(&pc.requests, 1)
+	pc.requests.Add(1)
 
 	pc.mu.RLock()
 	pcv := pc.m[q]
 	pc.mu.RUnlock()
 
 	if pcv == nil {
-		atomic.AddUint64(&pc.misses, 1)
+		pc.misses.Add(1)
 	}
 	return pcv
 }

app/vmselect/promql/exec_test.go

@@ -210,11 +210,13 @@ func TestExecSuccess(t *testing.T) {
 		f(q, resultExpected)
 	})
 	t.Run("scalar-string-nonnum", func(t *testing.T) {
+		t.Parallel()
 		q := `scalar("fooobar")`
 		resultExpected := []netstorage.Result{}
 		f(q, resultExpected)
 	})
 	t.Run("scalar-string-num", func(t *testing.T) {
+		t.Parallel()
 		q := `scalar("-12.34")`
 		r := netstorage.Result{
 			MetricName: metricNameExpected,
@@ -2307,6 +2309,21 @@ func TestExecSuccess(t *testing.T) {
 		resultExpected := []netstorage.Result{r}
 		f(q, resultExpected)
 	})
+	t.Run(`label_join dst_label is equal to src_label`, func(t *testing.T) {
+		t.Parallel()
+		q := `label_join(label_join(time(), "bar", "sep1", "a", "b"), "bar", "sep2", "a", "bar")`
+		r := netstorage.Result{
+			MetricName: metricNameExpected,
+			Values:     []float64{1000, 1200, 1400, 1600, 1800, 2000},
+			Timestamps: timestampsExpected,
+		}
+		r.MetricName.Tags = []storage.Tag{{
+			Key:   []byte("bar"),
+			Value: []byte("sep2sep1"),
+		}}
+		resultExpected := []netstorage.Result{r}
+		f(q, resultExpected)
+	})
 	t.Run(`label_value()`, func(t *testing.T) {
 		t.Parallel()
 		q := `with (
@@ -5822,6 +5839,60 @@ func TestExecSuccess(t *testing.T) {
 		resultExpected := []netstorage.Result{r1, r2}
 		f(q, resultExpected)
 	})
+	t.Run(`count_values_over_time`, func(t *testing.T) {
+		t.Parallel()
+		q := `sort_by_label(
+			count_values_over_time("foo", round(label_set(rand(0), "x", "y"), 0.4)[200s:5s]),
+			"foo",
+		)`
+		r1 := netstorage.Result{
+			MetricName: metricNameExpected,
+			Values:     []float64{4, 8, 7, 6, 10, 9},
+			Timestamps: timestampsExpected,
+		}
+		r1.MetricName.Tags = []storage.Tag{
+			{
+				Key:   []byte("foo"),
+				Value: []byte("0"),
+			},
+			{
+				Key:   []byte("x"),
+				Value: []byte("y"),
+			},
+		}
+		r2 := netstorage.Result{
+			MetricName: metricNameExpected,
+			Values:     []float64{20, 13, 19, 18, 14, 13},
+			Timestamps: timestampsExpected,
+		}
+		r2.MetricName.Tags = []storage.Tag{
+			{
+				Key:   []byte("foo"),
+				Value: []byte("0.4"),
+			},
+			{
+				Key:   []byte("x"),
+				Value: []byte("y"),
+			},
+		}
+		r3 := netstorage.Result{
+			MetricName: metricNameExpected,
+			Values:     []float64{16, 19, 14, 16, 16, 18},
+			Timestamps: timestampsExpected,
+		}
+		r3.MetricName.Tags = []storage.Tag{
+			{
+				Key:   []byte("foo"),
+				Value: []byte("0.8"),
+			},
+			{
+				Key:   []byte("x"),
+				Value: []byte("y"),
+			},
+		}
+		resultExpected := []netstorage.Result{r1, r2, r3}
+		f(q, resultExpected)
+	})
 	t.Run(`histogram_over_time`, func(t *testing.T) {
 		t.Parallel()
 		q := `sort_by_label(histogram_over_time(alias(label_set(rand(0)*1.3+1.1, "foo", "bar"), "xxx")[200s:5s]), "vmrange")`

app/vmselect/promql/rollup.go

@@ -4,12 +4,14 @@ import (
 	"flag"
 	"fmt"
 	"math"
+	"strconv"
 	"strings"
 	"sync"
 
 	"github.com/VictoriaMetrics/metrics"
 	"github.com/VictoriaMetrics/metricsql"
 
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/decimal"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/storage"
@@ -31,6 +33,7 @@ var rollupFuncs = map[string]newRollupFunc{
 	"count_le_over_time":      newRollupCountLE,
 	"count_ne_over_time":      newRollupCountNE,
 	"count_over_time":         newRollupFuncOneArg(rollupCount),
+	"count_values_over_time":  newRollupCountValues,
 	"decreases_over_time":     newRollupFuncOneArg(rollupDecreases),
 	"default_rollup":          newRollupFuncOneArg(rollupDefault), // default rollup func
 	"delta":                   newRollupFuncOneArg(rollupDelta),
@@ -103,6 +106,42 @@ var rollupFuncs = map[string]newRollupFunc{
 	"zscore_over_time":       newRollupFuncOneArg(rollupZScoreOverTime),
 }
 
+// Functions, which need the previous sample before the lookbehind window for proper calculations.
+//
+// All the rollup functions, which do not rely on the previous sample
+// before the lookbehind window (aka prevValue and realPrevValue), do not need silence interval.
+var needSilenceIntervalForRollupFunc = map[string]bool{
+	"ascent_over_time":    true,
+	"changes":             true,
+	"decreases_over_time": true,
+	// The default_rollup implicitly relies on the previous samples in order to fill gaps.
+	// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5388
+	"default_rollup":         true,
+	"delta":                  true,
+	"deriv_fast":             true,
+	"descent_over_time":      true,
+	"idelta":                 true,
+	"ideriv":                 true,
+	"increase":               true,
+	"increase_pure":          true,
+	"increases_over_time":    true,
+	"integrate":              true,
+	"irate":                  true,
+	"lag":                    true,
+	"lifetime":               true,
+	"rate":                   true,
+	"resets":                 true,
+	"rollup":                 true,
+	"rollup_candlestick":     true,
+	"rollup_delta":           true,
+	"rollup_deriv":           true,
+	"rollup_increase":        true,
+	"rollup_rate":            true,
+	"rollup_scrape_interval": true,
+	"scrape_interval":        true,
+	"tlast_change_over_time": true,
+}
+
 // rollupAggrFuncs are functions that can be passed to `aggr_over_time()`
 var rollupAggrFuncs = map[string]rollupFunc{
 	"absent_over_time":        rollupAbsent,
@@ -332,10 +371,10 @@ func getRollupTag(expr metricsql.Expr) (string, error) {
 func getRollupConfigs(funcName string, rf rollupFunc, expr metricsql.Expr, start, end, step int64, maxPointsPerSeries int,
 	window, lookbackDelta int64, sharedTimestamps []int64) (
 	func(values []float64, timestamps []int64), []*rollupConfig, error) {
-	preFunc := func(values []float64, timestamps []int64) {}
+	preFunc := func(_ []float64, _ []int64) {}
 	funcName = strings.ToLower(funcName)
 	if rollupFuncsRemoveCounterResets[funcName] {
-		preFunc = func(values []float64, timestamps []int64) {
+		preFunc = func(values []float64, _ []int64) {
 			removeCounterResets(values)
 		}
 	}
@@ -447,7 +486,7 @@ func getRollupConfigs(funcName string, rf rollupFunc, expr metricsql.Expr, start
 		for _, aggrFuncName := range aggrFuncNames {
 			if rollupFuncsRemoveCounterResets[aggrFuncName] {
 				// There is no need to save the previous preFunc, since it is either empty or the same.
-				preFunc = func(values []float64, timestamps []int64) {
+				preFunc = func(values []float64, _ []int64) {
 					removeCounterResets(values)
 				}
 			}
@@ -573,7 +612,7 @@ type timeseriesMap struct {
 func newTimeseriesMap(funcName string, keepMetricNames bool, sharedTimestamps []int64, mnSrc *storage.MetricName) *timeseriesMap {
 	funcName = strings.ToLower(funcName)
 	switch funcName {
-	case "histogram_over_time", "quantiles_over_time":
+	case "histogram_over_time", "quantiles_over_time", "count_values_over_time":
 	default:
 		return nil
 	}
@@ -607,10 +646,16 @@ func (tsm *timeseriesMap) GetOrCreateTimeseries(labelName, labelValue string) *t
 	if ts != nil {
 		return ts
 	}
+
+	// Make a clone of labelValue in order to use it as map key, since it may point to unsafe string,
+	// which refers some other byte slice, which can change in the future.
+	labelValue = strings.Clone(labelValue)
+
 	ts = &timeseries{}
 	ts.CopyFromShallowTimestamps(tsm.origin)
 	ts.MetricName.RemoveTag(labelName)
 	ts.MetricName.AddTag(labelName, labelValue)
+
 	tsm.m[labelValue] = ts
 	return ts
 }
@@ -957,7 +1002,7 @@ func newRollupHoltWinters(args []interface{}) (rollupFunc, error) {
 		// before calling rollup funcs.
 		values := rfa.values
 		if len(values) == 0 {
-			return rfa.prevValue
+			return nan
 		}
 		sf := sfs[rfa.idx]
 		if sf < 0 || sf > 1 {
@@ -1400,6 +1445,42 @@ func mad(values []float64) float64 {
 	return v
 }
 
+func newRollupCountValues(args []interface{}) (rollupFunc, error) {
+	if err := expectRollupArgsNum(args, 2); err != nil {
+		return nil, err
+	}
+	tssLabelNum, ok := args[0].([]*timeseries)
+	if !ok {
+		return nil, fmt.Errorf(`unexpected type for labelName arg; got %T; want %T`, args[0], tssLabelNum)
+	}
+	labelName, err := getString(tssLabelNum, 0)
+	if err != nil {
+		return nil, fmt.Errorf("cannot get labelName: %w", err)
+	}
+	f := func(rfa *rollupFuncArg) float64 {
+		tsm := rfa.tsm
+		idx := rfa.idx
+		bb := bbPool.Get()
+		// Note: the code below may create very big number of time series
+		// if the number of unique values in rfa.values is big.
+		for _, v := range rfa.values {
+			bb.B = strconv.AppendFloat(bb.B[:0], v, 'g', -1, 64)
+			labelValue := bytesutil.ToUnsafeString(bb.B)
+			ts := tsm.GetOrCreateTimeseries(labelName, labelValue)
+			count := ts.Values[idx]
+			if math.IsNaN(count) {
+				count = 1
+			} else {
+				count++
+			}
+			ts.Values[idx] = count
+		}
+		bbPool.Put(bb)
+		return nan
+	}
+	return f, nil
+}
+
 func rollupHistogram(rfa *rollupFuncArg) float64 {
 	values := rfa.values
 	tsm := rfa.tsm
@@ -1586,11 +1667,7 @@ func rollupRateOverSum(rfa *rollupFuncArg) float64 {
 	// before calling rollup funcs.
 	timestamps := rfa.timestamps
 	if len(timestamps) == 0 {
-		if math.IsNaN(rfa.prevValue) {
-			return nan
-		}
-		// Assume that the value didn't change since rfa.prevValue.
-		return 0
+		return nan
 	}
 	sum := float64(0)
 	for _, v := range rfa.values {
@@ -1610,7 +1687,7 @@ func rollupSum2(rfa *rollupFuncArg) float64 {
 	// before calling rollup funcs.
 	values := rfa.values
 	if len(values) == 0 {
-		return rfa.prevValue * rfa.prevValue
+		return nan
 	}
 	var sum2 float64
 	for _, v := range values {
@@ -1624,7 +1701,7 @@ func rollupGeomean(rfa *rollupFuncArg) float64 {
 	// before calling rollup funcs.
 	values := rfa.values
 	if len(values) == 0 {
-		return rfa.prevValue
+		return nan
 	}
 	p := 1.0
 	for _, v := range values {
@@ -2268,10 +2345,7 @@ func rollupDistinct(rfa *rollupFuncArg) float64 {
 	// before calling rollup funcs.
 	values := rfa.values
 	if len(values) == 0 {
-		if math.IsNaN(rfa.prevValue) {
-			return nan
-		}
-		return 0
+		return nan
 	}
 	m := make(map[float64]struct{})
 	for _, v := range values {

app/vmselect/promql/rollup_result_cache.go

@@ -45,7 +45,7 @@ func ResetRollupResultCacheIfNeeded(mrs []storage.MetricRow) {
 		rollupResultResetMetricRowSample.Store(&storage.MetricRow{})
 		go checkRollupResultCacheReset()
 	})
-	if atomic.LoadUint32(&needRollupResultCacheReset) != 0 {
+	if needRollupResultCacheReset.Load() {
 		// The cache has been already instructed to reset.
 		return
 	}
@@ -63,14 +63,14 @@ func ResetRollupResultCacheIfNeeded(mrs []storage.MetricRow) {
 	}
 	if needCacheReset {
 		// Do not call ResetRollupResultCache() here, since it may be heavy when frequently called.
-		atomic.StoreUint32(&needRollupResultCacheReset, 1)
+		needRollupResultCacheReset.Store(true)
 	}
 }
 
 func checkRollupResultCacheReset() {
 	for {
 		time.Sleep(checkRollupResultCacheResetInterval)
-		if atomic.SwapUint32(&needRollupResultCacheReset, 0) > 0 {
+		if needRollupResultCacheReset.Swap(false) {
 			mr := rollupResultResetMetricRowSample.Load()
 			d := int64(fasttime.UnixTimestamp()*1000) - mr.Timestamp - cacheTimestampOffset.Milliseconds()
 			logger.Warnf("resetting rollup result cache because the metric %s has a timestamp older than -search.cacheTimestampOffset=%s by %.3fs",
@@ -82,7 +82,7 @@ func checkRollupResultCacheReset() {
 
 const checkRollupResultCacheResetInterval = 5 * time.Second
 
-var needRollupResultCacheReset uint32
+var needRollupResultCacheReset atomic.Bool
 var checkRollupResultCacheResetOnce sync.Once
 var rollupResultResetMetricRowSample atomic.Pointer[storage.MetricRow]
 
@@ -129,7 +129,7 @@ func InitRollupResultCache(cachePath string) {
 		mustLoadRollupResultCacheKeyPrefix(rollupResultCachePath)
 	} else {
 		c = workingsetcache.New(cacheSize)
-		rollupResultCacheKeyPrefix = newRollupResultCacheKeyPrefix()
+		rollupResultCacheKeyPrefix.Store(newRollupResultCacheKeyPrefix())
 	}
 	if *disableCache {
 		c.Reset()
@@ -211,7 +211,7 @@ var rollupResultCacheResets = metrics.NewCounter(`vm_cache_resets_total{type="pr
 // ResetRollupResultCache resets rollup result cache.
 func ResetRollupResultCache() {
 	rollupResultCacheResets.Inc()
-	atomic.AddUint64(&rollupResultCacheKeyPrefix, 1)
+	rollupResultCacheKeyPrefix.Add(1)
 	logger.Infof("rollupResult cache has been cleared")
 }
 
@@ -438,8 +438,8 @@ func (rrc *rollupResultCache) PutSeries(qt *querytracer.Tracer, ec *EvalConfig,
 	}
 
 	var key rollupResultCacheKey
-	key.prefix = rollupResultCacheKeyPrefix
-	key.suffix = atomic.AddUint64(&rollupResultCacheKeySuffix, 1)
+	key.prefix = rollupResultCacheKeyPrefix.Load()
+	key.suffix = rollupResultCacheKeySuffix.Add(1)
 
 	bb := bbPool.Get()
 	bb.B = key.Marshal(bb.B[:0])
@@ -455,8 +455,12 @@ func (rrc *rollupResultCache) PutSeries(qt *querytracer.Tracer, ec *EvalConfig,
 }
 
 var (
-	rollupResultCacheKeyPrefix uint64
-	rollupResultCacheKeySuffix = uint64(time.Now().UnixNano())
+	rollupResultCacheKeyPrefix atomic.Uint64
+	rollupResultCacheKeySuffix = func() *atomic.Uint64 {
+		var x atomic.Uint64
+		x.Store(uint64(time.Now().UnixNano()))
+		return &x
+	}()
 )
 
 func (rrc *rollupResultCache) getSeriesFromCache(qt *querytracer.Tracer, key []byte) ([]*timeseries, bool) {
@@ -517,26 +521,26 @@ func newRollupResultCacheKeyPrefix() uint64 {
 func mustLoadRollupResultCacheKeyPrefix(path string) {
 	path = path + ".key.prefix"
 	if !fs.IsPathExist(path) {
-		rollupResultCacheKeyPrefix = newRollupResultCacheKeyPrefix()
+		rollupResultCacheKeyPrefix.Store(newRollupResultCacheKeyPrefix())
 		return
 	}
 	data, err := os.ReadFile(path)
 	if err != nil {
 		logger.Errorf("cannot load %s: %s; reset rollupResult cache", path, err)
-		rollupResultCacheKeyPrefix = newRollupResultCacheKeyPrefix()
+		rollupResultCacheKeyPrefix.Store(newRollupResultCacheKeyPrefix())
 		return
 	}
 	if len(data) != 8 {
 		logger.Errorf("unexpected size of %s; want 8 bytes; got %d bytes; reset rollupResult cache", path, len(data))
-		rollupResultCacheKeyPrefix = newRollupResultCacheKeyPrefix()
+		rollupResultCacheKeyPrefix.Store(newRollupResultCacheKeyPrefix())
 		return
 	}
-	rollupResultCacheKeyPrefix = encoding.UnmarshalUint64(data)
+	rollupResultCacheKeyPrefix.Store(encoding.UnmarshalUint64(data))
 }
 
 func mustSaveRollupResultCacheKeyPrefix(path string) {
 	path = path + ".key.prefix"
-	data := encoding.MarshalUint64(nil, rollupResultCacheKeyPrefix)
+	data := encoding.MarshalUint64(nil, rollupResultCacheKeyPrefix.Load())
 	fs.MustWriteAtomic(path, data, true)
 }
 
@@ -552,7 +556,7 @@ const (
 
 func marshalRollupResultCacheKeyForSeries(dst []byte, expr metricsql.Expr, window, step int64, etfs [][]storage.TagFilter) []byte {
 	dst = append(dst, rollupResultCacheVersion)
-	dst = encoding.MarshalUint64(dst, rollupResultCacheKeyPrefix)
+	dst = encoding.MarshalUint64(dst, rollupResultCacheKeyPrefix.Load())
 	dst = append(dst, rollupResultCacheTypeSeries)
 	dst = encoding.MarshalInt64(dst, window)
 	dst = encoding.MarshalInt64(dst, step)
@@ -563,7 +567,7 @@ func marshalRollupResultCacheKeyForSeries(dst []byte, expr metricsql.Expr, windo
 
 func marshalRollupResultCacheKeyForInstantValues(dst []byte, expr metricsql.Expr, window, step int64, etfs [][]storage.TagFilter) []byte {
 	dst = append(dst, rollupResultCacheVersion)
-	dst = encoding.MarshalUint64(dst, rollupResultCacheKeyPrefix)
+	dst = encoding.MarshalUint64(dst, rollupResultCacheKeyPrefix.Load())
 	dst = append(dst, rollupResultCacheTypeInstantValues)
 	dst = encoding.MarshalInt64(dst, window)
 	dst = encoding.MarshalInt64(dst, step)

app/vmselect/promql/rollup_result_cache_test.go

@@ -10,13 +10,13 @@ import (
 )
 
 func TestRollupResultCacheInitStop(t *testing.T) {
-	t.Run("inmemory", func(t *testing.T) {
+	t.Run("inmemory", func(_ *testing.T) {
 		for i := 0; i < 5; i++ {
 			InitRollupResultCache("")
 			StopRollupResultCache()
 		}
 	})
-	t.Run("file-based", func(t *testing.T) {
+	t.Run("file-based", func(_ *testing.T) {
 		cacheFilePath := "test-rollup-result-cache"
 		for i := 0; i < 3; i++ {
 			InitRollupResultCache(cacheFilePath)

app/vmselect/promql/timeseries.go

@@ -2,7 +2,6 @@ package promql
 
 import (
 	"fmt"
-	"reflect"
 	"sort"
 	"strconv"
 	"sync"
@@ -308,60 +307,32 @@ func unmarshalBytesFast(src []byte) ([]byte, []byte, error) {
 	return src[n:], src[:n], nil
 }
 
-func float64ToByteSlice(a []float64) (b []byte) {
-	if len(a) == 0 {
-		return nil
-	}
-	sh := (*reflect.SliceHeader)(unsafe.Pointer(&b))
-	sh.Data = uintptr(unsafe.Pointer(&a[0]))
-	sh.Len = len(a) * int(unsafe.Sizeof(a[0]))
-	sh.Cap = sh.Len
-	return
+func float64ToByteSlice(a []float64) []byte {
+	return unsafe.Slice((*byte)(unsafe.Pointer(unsafe.SliceData(a))), len(a)*8)
 }
 
-func int64ToByteSlice(a []int64) (b []byte) {
-	if len(a) == 0 {
-		return nil
-	}
-	sh := (*reflect.SliceHeader)(unsafe.Pointer(&b))
-	sh.Data = uintptr(unsafe.Pointer(&a[0]))
-	sh.Len = len(a) * int(unsafe.Sizeof(a[0]))
-	sh.Cap = sh.Len
-	return
+func int64ToByteSlice(a []int64) []byte {
+	return unsafe.Slice((*byte)(unsafe.Pointer(unsafe.SliceData(a))), len(a)*8)
 }
 
-func byteSliceToInt64(b []byte) (a []int64) {
-	if len(b) == 0 {
-		return nil
-	}
-	sh := (*reflect.SliceHeader)(unsafe.Pointer(&a))
-	sh.Data = uintptr(unsafe.Pointer(&b[0]))
-	sh.Len = len(b) / int(unsafe.Sizeof(a[0]))
-	sh.Cap = sh.Len
+func byteSliceToInt64(b []byte) []int64 {
 	// Make sure that the returned slice is properly aligned to 8 bytes.
 	// This prevents from SIGBUS error on arm architectures, which deny unaligned access.
 	// See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3927
-	if sh.Data%8 != 0 {
+	if uintptr(unsafe.Pointer(unsafe.SliceData(b)))%8 != 0 {
 		logger.Panicf("BUG: the input byte slice b must be aligned to 8 bytes")
 	}
-	return
+	return unsafe.Slice((*int64)(unsafe.Pointer(unsafe.SliceData(b))), len(b)/8)
 }
 
-func byteSliceToFloat64(b []byte) (a []float64) {
-	if len(b) == 0 {
-		return nil
-	}
-	sh := (*reflect.SliceHeader)(unsafe.Pointer(&a))
-	sh.Data = uintptr(unsafe.Pointer(&b[0]))
-	sh.Len = len(b) / int(unsafe.Sizeof(a[0]))
-	sh.Cap = sh.Len
+func byteSliceToFloat64(b []byte) []float64 {
 	// Make sure that the returned slice is properly aligned to 8 bytes.
 	// This prevents from SIGBUS error on arm architectures, which deny unaligned access.
 	// See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3927
-	if sh.Data%8 != 0 {
+	if uintptr(unsafe.Pointer(unsafe.SliceData(b)))%8 != 0 {
 		logger.Panicf("BUG: the input byte slice b must be aligned to 8 bytes")
 	}
-	return
+	return unsafe.Slice((*float64)(unsafe.Pointer(unsafe.SliceData(b))), len(b)/8)
 }
 
 func stringMetricName(mn *storage.MetricName) string {

app/vmselect/promql/timeseries_test.go

@@ -52,7 +52,9 @@ func TestMarshalTimeseriesFast(t *testing.T) {
 		MetricName: storage.MetricName{
 			MetricGroup: []byte{},
 		},
-		denyReuse: true,
+		Values:     []float64{},
+		Timestamps: []int64{},
+		denyReuse:  true,
 	}})
 	f([]*timeseries{{
 		MetricName: storage.MetricName{

app/vmselect/promql/transform.go

@@ -918,7 +918,7 @@ func transformHistogramQuantile(tfa *transformFuncArg) ([]*timeseries, error) {
 	m := groupLeTimeseries(tss)
 
 	// Calculate quantile for each group in m
-	lastNonInf := func(i int, xss []leTimeseries) float64 {
+	lastNonInf := func(_ int, xss []leTimeseries) float64 {
 		for len(xss) > 0 {
 			xsLast := xss[len(xss)-1]
 			if !math.IsInf(xsLast.le, 0) {
@@ -1248,13 +1248,8 @@ func newTransformFuncRunning(rf func(a, b float64, idx int) float64) transformFu
 			prevValue := values[0]
 			values = values[1:]
 			for i, v := range values {
-				if tfa.ec.RealEnd > 0 && ts.Timestamps[i] > tfa.ec.RealEnd {
-					break
-				}
-				if ts.Timestamps[i] >= tfa.ec.RealStart {
-					if !math.IsNaN(v) {
-						prevValue = rf(prevValue, v, i+1)
-					}
+				if !math.IsNaN(v) {
+					prevValue = rf(prevValue, v, i+1)
 				}
 				values[i] = prevValue
 			}
@@ -1270,7 +1265,7 @@ func newTransformFuncRange(rf func(a, b float64, idx int) float64) transformFunc
 		if err != nil {
 			return nil, err
 		}
-		setLastValues(tfa, rvs)
+		setLastValues(rvs)
 		return rvs, nil
 	}
 }
@@ -1544,7 +1539,7 @@ func transformRangeQuantile(tfa *transformFuncArg) ([]*timeseries, error) {
 	}
 	a.A = values
 	putFloat64s(a)
-	setLastValues(tfa, rvs)
+	setLastValues(rvs)
 	return rvs, nil
 }
 
@@ -1559,14 +1554,9 @@ func transformRangeFirst(tfa *transformFuncArg) ([]*timeseries, error) {
 		if len(values) == 0 {
 			continue
 		}
-		vFirstSet := false
 		vFirst := values[0]
 		for i, v := range values {
-			if !vFirstSet && ts.Timestamps[i] >= tfa.ec.RealStart {
-				vFirst = v
-				vFirstSet = true
-			}
-			if !vFirstSet && math.IsNaN(v) {
+			if math.IsNaN(v) {
 				continue
 			}
 			values[i] = vFirst
@@ -1581,28 +1571,17 @@ func transformRangeLast(tfa *transformFuncArg) ([]*timeseries, error) {
 		return nil, err
 	}
 	rvs := args[0]
-	setLastValues(tfa, rvs)
+	setLastValues(rvs)
 	return rvs, nil
 }
 
-func setLastValues(tfa *transformFuncArg, tss []*timeseries) {
+func setLastValues(tss []*timeseries) {
 	for _, ts := range tss {
 		values := skipTrailingNaNs(ts.Values)
 		if len(values) == 0 {
 			continue
 		}
 		vLast := values[len(values)-1]
-		if tfa.ec.RealEnd > 0 {
-			for i := len(values) - 1; i >= 0; i-- {
-				if math.IsNaN(values[i]) {
-					continue
-				}
-				if ts.Timestamps[i] > tfa.ec.RealEnd {
-					continue
-				}
-				vLast = values[i]
-			}
-		}
 		for i, v := range values {
 			if math.IsNaN(v) {
 				continue
@@ -1961,8 +1940,7 @@ func transformLabelJoin(tfa *transformFuncArg) ([]*timeseries, error) {
 	for _, ts := range rvs {
 		mn := &ts.MetricName
 		dstValue := getDstValue(mn, dstLabel)
-		b := *dstValue
-		b = b[:0]
+		var b []byte
 		for j, srcLabel := range srcLabels {
 			srcValue := mn.GetTagValue(srcLabel)
 			b = append(b, srcValue...)
@@ -2734,17 +2712,11 @@ func transformStep(tfa *transformFuncArg) float64 {
 }
 
 func transformStart(tfa *transformFuncArg) float64 {
-	if tfa.ec.RealStart == 0 {
-		return float64(tfa.ec.Start) / 1e3
-	}
-	return float64(tfa.ec.RealStart) / 1e3
+	return float64(tfa.ec.Start) / 1e3
 }
 
 func transformEnd(tfa *transformFuncArg) float64 {
-	if tfa.ec.RealEnd == 0 {
-		return float64(tfa.ec.End) / 1e3
-	}
-	return float64(tfa.ec.RealEnd) / 1e3
+	return float64(tfa.ec.End) / 1e3
 }
 
 // copyTimeseries returns a copy of tss.

app/vmselect/searchutils/searchutils.go

@@ -17,6 +17,8 @@ var (
 	maxExportDuration        = flag.Duration("search.maxExportDuration", time.Hour*24*30, "The maximum duration for /api/v1/export call")
 	maxQueryDuration         = flag.Duration("search.maxQueryDuration", time.Second*30, "The maximum duration for query execution")
 	maxStatusRequestDuration = flag.Duration("search.maxStatusRequestDuration", time.Minute*5, "The maximum duration for /api/v1/status/* requests")
+	maxLabelsAPIDuration     = flag.Duration("search.maxLabelsAPIDuration", time.Second*5, "The maximum duration for /api/v1/labels, /api/v1/label/.../values and /api/v1/series requests. "+
+		"See also -search.maxLabelsAPISeries and -search.ignoreExtraFiltersAtLabelsAPI")
 )
 
 // GetMaxQueryDuration returns the maximum duration for query from r.
@@ -50,6 +52,12 @@ func GetDeadlineForExport(r *http.Request, startTime time.Time) Deadline {
 	return getDeadlineWithMaxDuration(r, startTime, dMax, "-search.maxExportDuration")
 }
 
+// GetDeadlineForLabelsAPI returns deadline for the given request to /api/v1/labels, /api/v1/label/.../values or /api/v1/series
+func GetDeadlineForLabelsAPI(r *http.Request, startTime time.Time) Deadline {
+	dMax := maxLabelsAPIDuration.Milliseconds()
+	return getDeadlineWithMaxDuration(r, startTime, dMax, "-search.maxLabelsAPIDuration")
+}
+
 func getDeadlineWithMaxDuration(r *http.Request, startTime time.Time, dMax int64, flagHint string) Deadline {
 	d, err := httputils.GetDuration(r, "timeout", 0)
 	if err != nil {

app/vmselect/searchutils/searchutils_test.go

@@ -7,6 +7,7 @@ import (
 	"reflect"
 	"strconv"
 	"testing"
+	"time"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/storage"
 )
@@ -254,3 +255,28 @@ func tagFiltersToString(tfs []storage.TagFilter) string {
 	b = append(b, '}')
 	return string(b)
 }
+
+func TestGetDeadline(t *testing.T) {
+	f := func(got, exp Deadline) {
+		if got.Deadline() != exp.Deadline() {
+			t.Fatalf("expected to have %v; got %v instead", exp, got)
+		}
+	}
+
+	start := time.Now()
+	expDeadline := func(deadline time.Duration) Deadline {
+		return NewDeadline(start, deadline, "")
+	}
+
+	r, _ := http.NewRequest("GET", "", nil)
+	f(GetDeadlineForExport(r, start), expDeadline(*maxExportDuration))
+	f(GetDeadlineForLabelsAPI(r, start), expDeadline(*maxLabelsAPIDuration))
+	f(GetDeadlineForStatusRequest(r, start), expDeadline(*maxStatusRequestDuration))
+	f(GetDeadlineForQuery(r, start), expDeadline(*maxQueryDuration))
+
+	r, _ = http.NewRequest("GET", "http://foo?timeout=1s", nil)
+	f(GetDeadlineForExport(r, start), expDeadline(time.Second))
+	f(GetDeadlineForLabelsAPI(r, start), expDeadline(time.Second))
+	f(GetDeadlineForStatusRequest(r, start), expDeadline(time.Second))
+	f(GetDeadlineForQuery(r, start), expDeadline(time.Second))
+}

app/vmselect/vmui/asset-manifest.json

@@ -1,13 +1,13 @@
 {
   "files": {
-    "main.css": "./static/css/main.be4fee7a.css",
-    "main.js": "./static/js/main.fd9d9e16.js",
-    "static/js/522.da77e7b3.chunk.js": "./static/js/522.da77e7b3.chunk.js",
-    "static/media/MetricsQL.md": "./static/media/MetricsQL.8a01ddf56e4e6bc1ccf1.md",
+    "main.css": "./static/css/main.a2ad4674.css",
+    "main.js": "./static/js/main.f9cc1e6c.js",
+    "static/js/685.bebe1265.chunk.js": "./static/js/685.bebe1265.chunk.js",
+    "static/media/MetricsQL.md": "./static/media/MetricsQL.10add6e7bdf0f1d98cf7.md",
     "index.html": "./index.html"
   },
   "entrypoints": [
-    "static/css/main.be4fee7a.css",
-    "static/js/main.fd9d9e16.js"
+    "static/css/main.a2ad4674.css",
+    "static/js/main.f9cc1e6c.js"
   ]
 }

app/vmselect/vmui/index.html

@@ -1 +1 @@
-<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=5"/><meta name="theme-color" content="#000000"/><meta name="description" content="UI for VictoriaMetrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><script src="./dashboards/index.js" type="module"></script><meta name="twitter:card" content="summary_large_image"><meta name="twitter:image" content="./preview.jpg"><meta name="twitter:title" content="UI for VictoriaMetrics"><meta name="twitter:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta name="twitter:site" content="@VictoriaMetrics"><meta property="og:title" content="Metric explorer for VictoriaMetrics"><meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta property="og:image" content="./preview.jpg"><meta property="og:type" content="website"><script defer="defer" src="./static/js/main.fd9d9e16.js"></script><link href="./static/css/main.be4fee7a.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>
+<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=5"/><meta name="theme-color" content="#000000"/><meta name="description" content="UI for VictoriaMetrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><script src="./dashboards/index.js" type="module"></script><meta name="twitter:card" content="summary_large_image"><meta name="twitter:image" content="./preview.jpg"><meta name="twitter:title" content="UI for VictoriaMetrics"><meta name="twitter:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta name="twitter:site" content="@VictoriaMetrics"><meta property="og:title" content="Metric explorer for VictoriaMetrics"><meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta property="og:image" content="./preview.jpg"><meta property="og:type" content="website"><script defer="defer" src="./static/js/main.f9cc1e6c.js"></script><link href="./static/css/main.a2ad4674.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>