vendor: update github.com/klauspost/compress from v1.15.12 to v1.15.13

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3427

.github/ISSUE_TEMPLATE/bug_report.md

@@ -11,7 +11,7 @@ A clear and concise description of what the bug is.
 It would be great to [upgrade](https://docs.victoriametrics.com/#how-to-upgrade) 
 to [the latest available release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)
 and verify whether the bug is reproducible there.
-It's also recommended to read the [troubleshooting docs](https://docs.victoriametrics.com/#troubleshooting).
+It's also recommended to read the [troubleshooting docs](https://docs.victoriametrics.com/Troubleshooting.html).
 
 **To Reproduce**
 Steps to reproduce the behavior.

.github/workflows/check-licenses.yml

@@ -17,7 +17,7 @@ jobs:
       - name: Setup Go
         uses: actions/setup-go@main
         with:
-          go-version: 1.19.3
+          go-version: 1.19.4
         id: go
       - name: Code checkout
         uses: actions/checkout@master

.github/workflows/main.yml

@@ -19,7 +19,7 @@ jobs:
       - name: Setup Go
         uses: actions/setup-go@main
         with:
-          go-version: 1.19.3
+          go-version: 1.19.4
         id: go
       - name: Code checkout
         uses: actions/checkout@master

Makefile

@@ -2,7 +2,7 @@ PKG_PREFIX := github.com/VictoriaMetrics/VictoriaMetrics
 
 DATEINFO_TAG ?= $(shell date -u +'%Y%m%d-%H%M%S')
 BUILDINFO_TAG ?= $(shell echo $$(git describe --long --all | tr '/' '-')$$( \
-	      git diff-index --quiet HEAD -- || echo '-dirty-'$$(git diff-index -u HEAD | openssl sha1 | cut -c 10-17)))
+	      git diff-index --quiet HEAD -- || echo '-dirty-'$$(git diff-index -u HEAD | openssl sha1 | cut -d' ' -f2 | cut -c 1-8)))
 
 PKG_TAG ?= $(shell git tag -l --points-at HEAD)
 ifeq ($(PKG_TAG),)

README.md

@@ -275,7 +275,7 @@ It also provides the following features:
 - [query tracer](#query-tracing)
 - [top queries explorer](#top-queries)
 
-Graphs in vmui support scrolling and zooming:
+Graphs in `vmui` support scrolling and zooming:
 
 * Select the needed time range on the graph in order to zoom in into the selected time range. Hold `ctrl` (or `cmd` on MacOS) and scroll down in order to zoom out.
 * Hold `ctrl` (or `cmd` on MacOS) and scroll up in order to zoom in the area under cursor.
@@ -293,6 +293,8 @@ VMUI allows investigating correlations between multiple queries on the same grap
 enter an additional query in the newly appeared input field and press `Enter`.
 Results for all the queries are displayed simultaneously on the same graph.
 Graphs for a particular query can be temporarily hidden by clicking the `eye` icon on the right side of the input field.
+When the `eye` icon is clicked while holding the `ctrl` key, then query results for the rest of queries become hidden
+except of the current query results.
 
 See the [example VMUI at VictoriaMetrics playground](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/graph/?g0.expr=100%20*%20sum(rate(process_cpu_seconds_total))%20by%20(job)&g0.range_input=1d).
 
@@ -785,7 +787,7 @@ to your needs or when testing bugfixes.
 
 ### Development build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.3.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.
 2. Run `make victoria-metrics` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `victoria-metrics` binary and puts it into the `bin` folder.
 
@@ -801,7 +803,7 @@ ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://b
 
 ### Development ARM build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.3.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.
 2. Run `make victoria-metrics-linux-arm` or `make victoria-metrics-linux-arm64` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `victoria-metrics-linux-arm` or `victoria-metrics-linux-arm64` binary respectively and puts it into the `bin` folder.
 
@@ -815,7 +817,7 @@ ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://b
 
 `Pure Go` mode builds only Go code without [cgo](https://golang.org/cmd/cgo/) dependencies.
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.3.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.
 2. Run `make victoria-metrics-pure` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `victoria-metrics-pure` binary and puts it into the `bin` folder.
 
@@ -1243,7 +1245,11 @@ Example contents for `-relabelConfig` file:
   regex: true
 ```
 
-VictoriaMetrics provides additional relabeling features such as Graphite-style relabeling. See [these docs](https://docs.victoriametrics.com/vmagent.html#relabeling) for more details.
+VictoriaMetrics provides additional relabeling features such as Graphite-style relabeling.
+See [these docs](https://docs.victoriametrics.com/vmagent.html#relabeling) for more details.
+
+The relabeling can be debugged at `http://victoriametrics:8428/metric-relabel-debug` page.
+See [these docs](https://docs.victoriametrics.com/vmagent.html#relabel-debug) for more details.
 
 
 ## Federation
@@ -1349,7 +1355,12 @@ with the enabled de-duplication. See [this section](#deduplication) for details.
 
 ## Deduplication
 
-VictoriaMetrics leaves a single raw sample with the biggest timestamp per each `-dedup.minScrapeInterval` discrete interval if `-dedup.minScrapeInterval` is set to positive duration. For example, `-dedup.minScrapeInterval=60s` would leave a single raw sample with the biggest timestamp per each discrete 60s interval. If multiple raw samples have the same biggest timestamp on the given `-dedup.minScrapeInterval` discrete interval, then an arbitrary sample out of these samples is left. This aligns with the [staleness rules in Prometheus](https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness).
+VictoriaMetrics leaves a single raw sample with the biggest timestamp per each `-dedup.minScrapeInterval` discrete interval
+if `-dedup.minScrapeInterval` is set to positive duration. For example, `-dedup.minScrapeInterval=60s` would leave a single
+raw sample with the biggest timestamp per each discrete 60s interval.
+This aligns with the [staleness rules in Prometheus](https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness).
+
+If multiple raw samples have the same biggest timestamp on the given `-dedup.minScrapeInterval` discrete interval, then the sample with the biggest value is left.
 
 The `-dedup.minScrapeInterval=D` is equivalent to `-downsampling.period=0s:D` if [downsampling](#downsampling) is enabled. So it is safe to use deduplication and downsampling simultaneously.
 
@@ -1363,18 +1374,50 @@ It is recommended passing different `-promscrape.cluster.name` values to HA pair
 
 ## Storage
 
-VictoriaMetrics stores time series data in [MergeTree](https://en.wikipedia.org/wiki/Log-structured_merge-tree)-like
-data structures. On insert, VictoriaMetrics accumulates up to 1s of data and dumps it on disk to
-`<-storageDataPath>/data/small/YYYY_MM/` subdirectory forming a `part` with the following
-name pattern: `rowsCount_blocksCount_minTimestamp_maxTimestamp`. Each part consists of two "columns":
-values and timestamps. These are sorted and compressed raw time series values. Additionally, part contains
-index files for searching for specific series in the values and timestamps files.
+VictoriaMetrics buffers the ingested data in memory for up to a second. Then the buffered data is written to in-memory `parts`,
+which can be searched during queries. The in-memory `parts` are periodically persisted to disk, so they could survive unclean shutdown
+such as out of memory crash, hardware power loss or `SIGKILL` signal. The interval for flushing the in-memory data to disk
+can be configured with the `-inmemoryDataFlushInterval` command-line flag (note that too short flush interval may significantly increase disk IO).
 
-`Parts` are periodically merged into the bigger parts. The resulting `part` is constructed
-under `<-storageDataPath>/data/{small,big}/YYYY_MM/tmp` subdirectory.
-When the resulting `part` is complete, it is atomically moved from the `tmp`
-to its own subdirectory, while the source parts are atomically removed. The end result is that the source
-parts are substituted by a single resulting bigger `part` in the `<-storageDataPath>/data/{small,big}/YYYY_MM/` directory.
+In-memory parts are persisted to disk into `part` directories under the `<-storageDataPath>/data/small/YYYY_MM/` folder,
+where `YYYY_MM` is the month partition for the stored data. For example, `2022_11` is the partition for `parts`
+with [raw samples](https://docs.victoriametrics.com/keyConcepts.html#raw-samples) from `November 2022`.
+
+The `part` directory has the following name pattern: `rowsCount_blocksCount_minTimestamp_maxTimestamp`, where:
+
+- `rowsCount` - the number of [raw samples](https://docs.victoriametrics.com/keyConcepts.html#raw-samples) stored in the part
+- `blocksCount` - the number of blocks stored in the part (see details about blocks below)
+- `minTimestamp` and `maxTimestamp` - minimum and maximum timestamps across raw samples stored in the part
+
+Each `part` consists of `blocks` sorted by internal time series id (aka `TSID`).
+Each `block` contains up to 8K [raw samples](https://docs.victoriametrics.com/keyConcepts.html#raw-samples),
+which belong to a single [time series](https://docs.victoriametrics.com/keyConcepts.html#time-series).
+Raw samples in each block are sorted by `timestamp`. Blocks for the same time series are sorted
+by the `timestamp` of the first sample. Timestamps and values for all the blocks
+are stored in [compressed form](https://faun.pub/victoriametrics-achieving-better-compression-for-time-series-data-than-gorilla-317bc1f95932)
+in separate files under `part` directory - `timestamps.bin` and `values.bin`.
+
+The `part` directory also contains `index.bin` and `metaindex.bin` files - these files contain index
+for fast block lookups, which belong to the given `TSID` and cover the given time range.
+
+`Parts` are periodically merged into bigger parts in background. The background merge provides the following benefits:
+
+* keeping the number of data files under control, so they don't exceed limits on open files
+* improved data compression, since bigger parts are usually compressed better than smaller parts
+* improved query speed, since queries over smaller number of parts are executed faster
+* various background maintenance tasks such as [de-duplication](#deduplication), [downsampling](#downsampling)
+  and [freeing up disk space for the deleted time series](#how-to-delete-time-series) are performed during the merge
+
+Newly added `parts` either successfully appear in the storage or fail to appear.
+The newly added `parts` are being created in a temporary directory under `<-storageDataPath>/data/{small,big}/YYYY_MM/tmp` folder.
+When the newly added `part` is fully written and [fsynced](https://man7.org/linux/man-pages/man2/fsync.2.html)
+to a temporary directory, then it is atomically moved to the storage directory.
+Thanks to this alogrithm, storage never contains partially created parts, even if hardware power off
+occurrs in the middle of writing the `part` to disk - such incompletely written `parts`
+are automatically deleted on the next VictoriaMetrics start.
+
+The same applies to merge process — `parts` are either fully merged into a new `part` or fail to merge,
+leaving the source `parts` untouched.
 
 VictoriaMetrics doesn't merge parts if their summary size exceeds free disk space.
 This prevents from potential out of disk space errors during merge.
@@ -1383,24 +1426,10 @@ This increases overhead during data querying, since VictoriaMetrics needs to rea
 bigger number of parts per each request. That's why it is recommended to have at least 20%
 of free disk space under directory pointed by `-storageDataPath` command-line flag.
 
-Information about merging process is available in [single-node VictoriaMetrics](https://grafana.com/dashboards/10229)
-and [clustered VictoriaMetrics](https://grafana.com/grafana/dashboards/11176) Grafana dashboards.
+Information about merging process is available in [the dashboard for single-node VictoriaMetrics](https://grafana.com/dashboards/10229)
+and [the dashboard for VictoriaMetrics cluster](https://grafana.com/grafana/dashboards/11176).
 See more details in [monitoring docs](#monitoring).
 
-The `merge` process improves compression rate and keeps number of `parts` on disk relatively low.
-Benefits of doing the merge process are the following:
-
-* it improves query performance, since lower number of `parts` are inspected with each query
-* it reduces the number of data files, since each `part` contains fixed number of files
-* various background maintenance tasks such as [de-duplication](#deduplication), [downsampling](#downsampling)
-  and [freeing up disk space for the deleted time series](#how-to-delete-time-series) are performed during the merge.
-
-Newly added `parts` either appear in the storage or fail to appear.
-Storage never contains partially created parts. The same applies to merge process — `parts` are either fully
-merged into a new `part` or fail to merge. MergeTree doesn't contain partially merged `parts`.
-`Part` contents in MergeTree never change. Parts are immutable. They may be only deleted after the merge
-to a bigger `part` or when the `part` contents goes outside the configured `-retentionPeriod`.
-
 See [this article](https://valyala.medium.com/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282) for more details.
 
 See also [how to work with snapshots](#how-to-work-with-snapshots).
@@ -1723,10 +1752,11 @@ and [cardinality explorer docs](#cardinality-explorer).
 
 * VictoriaMetrics buffers incoming data in memory for up to a few seconds before flushing it to persistent storage.
   This may lead to the following "issues":
-  * Data becomes available for querying in a few seconds after inserting. It is possible to flush in-memory buffers to persistent storage
+  * Data becomes available for querying in a few seconds after inserting. It is possible to flush in-memory buffers to searchable parts
     by requesting `/internal/force_flush` http handler. This handler is mostly needed for testing and debugging purposes.
   * The last few seconds of inserted data may be lost on unclean shutdown (i.e. OOM, `kill -9` or hardware reset).
-    See [this article for technical details](https://valyala.medium.com/wal-usage-looks-broken-in-modern-time-series-databases-b62a627ab704).
+    The `-inmemoryDataFlushInterval` command-line flag allows controlling the frequency of in-memory data flush to persistent storage.
+    See [storage docs](#storage) and [this article](https://valyala.medium.com/wal-usage-looks-broken-in-modern-time-series-databases-b62a627ab704) for more details.
 
 * If VictoriaMetrics works slowly and eats more than a CPU core per 100K ingested data points per second,
   then it is likely you have too many [active time series](https://docs.victoriametrics.com/FAQ.html#what-is-an-active-time-series) for the current amount of RAM.
@@ -2133,6 +2163,8 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      Uses '{measurement}' instead of '{measurement}{separator}{field_name}' for metic name if InfluxDB line contains only a single field
   -influxTrimTimestamp duration
      Trim timestamps for InfluxDB line protocol data to this duration. Minimum practical duration is 1ms. Higher duration (i.e. 1s) may be used for reducing disk space usage for timestamp data (default 1ms)
+  -inmemoryDataFlushInterval duration
+     The interval for guaranteed saving of in-memory data to disk. The saved data survives unclean shutdown such as OOM crash, hardware reset, SIGKILL, etc. Bigger intervals may help increasing lifetime of flash storage with limited write cycles (e.g. Raspberry PI). Smaller intervals increase disk IO load. Minimum supported value is 1s (default 5s)
   -insert.maxQueueDuration duration
      The maximum duration for waiting in the queue for insert requests due to -maxConcurrentInserts (default 1m0s)
   -logNewSeries
@@ -2275,8 +2307,6 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      Supports an array of values separated by comma or specified via multiple flags.
   -relabelConfig string
      Optional path to a file with relabeling rules, which are applied to all the ingested metrics. The path can point either to local file or to http url. See https://docs.victoriametrics.com/#relabeling for details. The config is reloaded on SIGHUP signal
-  -relabelDebug
-     Whether to log metrics before and after relabeling with -relabelConfig. If the -relabelDebug is enabled, then the metrics aren't sent to storage. This is useful for debugging the relabeling configs
   -retentionFilter array
      Retention filter in the format 'filter:retention'. For example, '{env="dev"}:3d' configures the retention for time series with env="dev" label to 3 days. See https://docs.victoriametrics.com/#retention-filters for details. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
      Supports an array of values separated by comma or specified via multiple flags.

app/victoria-metrics/main.go

@@ -29,6 +29,10 @@ var (
 		"equal to -dedup.minScrapeInterval > 0. See https://docs.victoriametrics.com/#deduplication and https://docs.victoriametrics.com/#downsampling")
 	dryRun = flag.Bool("dryRun", false, "Whether to check only -promscrape.config and then exit. "+
 		"Unknown config entries aren't allowed in -promscrape.config by default. This can be changed with -promscrape.config.strictParse=false command-line flag")
+	inmemoryDataFlushInterval = flag.Duration("inmemoryDataFlushInterval", 5*time.Second, "The interval for guaranteed saving of in-memory data to disk. "+
+		"The saved data survives unclean shutdown such as OOM crash, hardware reset, SIGKILL, etc. "+
+		"Bigger intervals may help increasing lifetime of flash storage with limited write cycles (e.g. Raspberry PI). "+
+		"Smaller intervals increase disk IO load. Minimum supported value is 1s")
 )
 
 func main() {
@@ -54,6 +58,7 @@ func main() {
 	logger.Infof("starting VictoriaMetrics at %q...", *httpListenAddr)
 	startTime := time.Now()
 	storage.SetDedupInterval(*minScrapeInterval)
+	storage.SetDataFlushInterval(*inmemoryDataFlushInterval)
 	vmstorage.Init(promql.ResetRollupResultCacheIfNeeded)
 	vmselect.Init()
 	vminsert.Init()
@@ -96,6 +101,7 @@ func requestHandler(w http.ResponseWriter, r *http.Request) bool {
 			{"vmui", "Web UI"},
 			{"targets", "status for discovered active targets"},
 			{"service-discovery", "labels before and after relabeling for discovered targets"},
+			{"metric-relabel-debug", "debug metric relabeling"},
 			{"api/v1/targets", "advanced information about discovered targets in JSON format"},
 			{"config", "-promscrape.config contents"},
 			{"metrics", "available service metrics"},

app/vmagent/README.md

@@ -245,8 +245,6 @@ scrape_configs:
 * `scrape_align_interval: duration` for aligning scrapes to the given interval instead of using random offset
   in the range `[0 ... scrape_interval]` for scraping each target. The random offset helps spreading scrapes evenly in time.
 * `scrape_offset: duration` for specifying the exact offset for scraping instead of using random offset in the range `[0 ... scrape_interval]`.
-* `relabel_debug: true` for enabling debug logging during relabeling of the discovered targets. See [these docs](#relabeling).
-* `metric_relabel_debug: true` for enabling debug logging during relabeling of the scraped metrics. See [these docs](#relabeling).
 
 See [scrape_configs docs](https://docs.victoriametrics.com/sd_configs.html#scrape_configs) for more details on all the supported options.
 
@@ -317,7 +315,8 @@ Extra labels can be added to metrics collected by `vmagent` via the following me
 
 ## Automatically generated metrics
 
-`vmagent` automatically generates the following metrics per each scrape of every [Prometheus-compatible target](#how-to-collect-metrics-in-prometheus-format):
+`vmagent` automatically generates the following metrics per each scrape of every [Prometheus-compatible target](#how-to-collect-metrics-in-prometheus-format)
+and attaches target-specific `instance` and `job` labels to these metrics:
 
 * `up` - this metric exposes `1` value on successful scrape and `0` value on unsuccessful scrape. This allows monitoring
   failing scrapes with the following [MetricsQL query](https://docs.victoriametrics.com/MetricsQL.html):
@@ -405,6 +404,9 @@ Extra labels can be added to metrics collected by `vmagent` via the following me
   sum_over_time(scrape_series_limit_samples_dropped[1h]) > 0
   ```
 
+If the target exports metrics with names clashing with the automatically generated metric names, then `vmagent` automatically
+adds `exported_` prefix to these metric names, so they don't clash with automatically generated metric names.
+
 
 ## Relabeling
 
@@ -415,26 +417,25 @@ with [additional enhancements](#relabeling-enhancements). The relabeling can be
   This relabeling is used for modifying labels in discovered targets and for dropping unneded targets.
   See [relabeling cookbook](https://docs.victoriametrics.com/relabeling.html) for details.
 
-  This relabeling can be debugged by passing `relabel_debug: true` option to the corresponding `scrape_config` section.
-  In this case `vmagent` logs target labels before and after the relabeling and then drops the logged target.
+  This relabeling can be debugged by clicking the `debug` link at the corresponding target on the `http://vmagent:8429/targets` page
+  or on the `http://vmagent:8429/service-discovery` page. See [these docs](#relabel-debug) for details.
 
 * At the `scrape_config -> metric_relabel_configs` section in `-promscrape.config` file.
   This relabeling is used for modifying labels in scraped metrics and for dropping unneeded metrics.
   See [relabeling cookbook](https://docs.victoriametrics.com/relabeling.html) for details.
 
-  This relabeling can be debugged by passing `metric_relabel_debug: true` option to the corresponding `scrape_config` section.
-  In this case `vmagent` logs metrics before and after the relabeling and then drops the logged metrics.
+  This relabeling can be debugged via `http://vmagent:8429/metric-relabel-debug` page. See [these docs](#relabel-debug) for details.
 
 * At the `-remoteWrite.relabelConfig` file. This relabeling is used for modifying labels for all the collected metrics
-  (inluding [metrics obtained via push-based protocols](#how-to-push-data-to-vmagent)) and for dropping unneeded metrics
+  (including [metrics obtained via push-based protocols](#how-to-push-data-to-vmagent)) and for dropping unneeded metrics
   before sending them to all the configured `-remoteWrite.url` addresses.
-  This relabeling can be debugged by passing `-remoteWrite.relabelDebug` command-line option to `vmagent`.
-  In this case `vmagent` logs metrics before and after the relabeling and then drops all the logged metrics instead of sending them to remote storage.
+
+  This relabeling can be debugged via `http://vmagent:8429/metric-relabel-debug` page. See [these docs](#relabel-debug) for details.
 
 * At the `-remoteWrite.urlRelabelConfig` files. This relabeling is used for modifying labels for metrics
   and for dropping unneeded metrics before sending them to a particular `-remoteWrite.url`.
-  This relabeling can be debugged by passing `-remoteWrite.urlRelabelDebug` command-line options to `vmagent`.
-  In this case `vmagent` logs metrics before and after the relabeling and then drops all the logged metrics instead of sending them to the corresponding `-remoteWrite.url`.
+
+  This relabeling can be debugged via `http://vmagent:8429/metric-relabel-debug` page. See [these docs](#relabel-debug) for details.
 
 All the files with relabeling configs can contain special placeholders in the form `%{ENV_VAR}`,
 which are replaced by the corresponding environment variable values.
@@ -449,9 +450,6 @@ The following articles contain useful information about Prometheus relabeling:
 * [Extracting labels from legacy metric names](https://www.robustperception.io/extracting-labels-from-legacy-metric-names)
 * [relabel_configs vs metric_relabel_configs](https://www.robustperception.io/relabel_configs-vs-metric_relabel_configs)
 
-[This relabeler playground](https://relabeler.promlabs.com/) can help debugging issues related to relabeling.
-
-
 ## Relabeling enhancements
 
 `vmagent` provides the following enhancements on top of Prometheus-compatible relabeling:
@@ -593,6 +591,28 @@ Important notes about `action: graphite` relabeling rules:
 The `action: graphite` relabeling rules are easier to write and maintain than `action: replace` for labels extraction from Graphite-style metric names.
 Additionally, the `action: graphite` relabeling rules usually work much faster than the equivalent `action: replace` rules.
 
+## Relabel debug
+
+`vmagent` and [single-node VictoriaMetrics](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html)
+provide the following tools for debugging target-level and metric-level relabeling:
+
+- Target-level relabeling (e.g. `relabel_configs` section at [scrape_configs](https://docs.victoriametrics.com/sd_configs.html#scrape_configs))
+  can be performed by navigating to `http://vmagent:8429/targets` page (`http://victoriametrics:8428/targets` page for single-node VictoriaMetrics)
+  and clicking the `debug` link at the target, which must be debugged.
+  The opened page will show step-by-step results for the actual relabeling rules applied to the target labels.
+
+  The `http://vmagent:8429/targets` page shows only active targets. If you need to understand why some target
+  is dropped during the relabeling, then navigate to `http://vmagent:8428/service-discovery` page
+  (`http://victoriametrics:8428/service-discovery` for single-node VictoriaMetrics), find the dropped target
+  and click the `debug` link there. The opened page will show step-by-step results for the actual relabeling rules,
+  which result to target drop.
+
+- Metric-level relabeling (e.g. `metric_relabel_configs` section at [scrape_configs](https://docs.victoriametrics.com/sd_configs.html#scrape_configs)
+  and all the relabeling, which can be set up via `-relabelConfig`, `-remoteWrite.relabelConfig` and `-remoteWrite.urlRelabelConfig`
+  command-line flags) can be performed by navigating to `http://vmagent:8429/metric-relabel-debug` page
+  (`http://victoriametrics:8428/metric-relabel-debug` page for single-node VictoriaMetrics)
+  and submitting there relabeling rules together with the metric to be relabeled.
+  The page will show step-by-step results for the entered relabeling rules executed against the entered metric.
 
 ## Prometheus staleness markers
 
@@ -650,8 +670,9 @@ scrape_configs:
     'match[]': ['{__name__!=""}']
 ```
 
-Note that `sample_limit` and `series_limit` [scrape_config options](https://docs.victoriametrics.com/sd_configs.html#scrape_configs)
-cannot be used in stream parsing mode because the parsed data is pushed to remote storage as soon as it is parsed.
+Note that `vmagent` in stream parsing mode stores up to `sample_limit` samples to the configured `-remoteStorage.url`
+instead of droping all the samples read from the target, because the parsed data is sent to the remote storage
+as soon as it is parsed in stream parsing mode.
 
 ## Scraping big number of targets
 
@@ -740,8 +761,8 @@ By default `vmagent` doesn't limit the number of time series each scrape target
 
 * Via `-promscrape.seriesLimitPerTarget` command-line option. This limit is applied individually
   to all the scrape targets defined in the file pointed by `-promscrape.config`.
-* Via `series_limit` config option at `scrape_config` section. This limit is applied individually
-  to all the scrape targets defined in the given `scrape_config`.
+* Via `series_limit` config option at [scrape_config](https://docs.victoriametrics.com/sd_configs.html#scrape_configs) section.
+  This limit is applied individually to all the scrape targets defined in the given `scrape_config`.
 * Via `__series_limit__` label, which can be set with [relabeling](#relabeling) at `relabel_configs` section.
   This limit is applied to the corresponding scrape targets. Typical use case: to set the limit
   via [Kubernetes annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/) for targets,
@@ -1027,7 +1048,7 @@ It may be needed to build `vmagent` from source code when developing or testing
 
 ### Development build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.3.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.
 2. Run `make vmagent` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds the `vmagent` binary and puts it into the `bin` folder.
 
@@ -1056,7 +1077,7 @@ ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://b
 
 ### Development ARM build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.3.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.
 2. Run `make vmagent-linux-arm` or `make vmagent-linux-arm64` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics)
    It builds `vmagent-linux-arm` or `vmagent-linux-arm64` binary respectively and puts it into the `bin` folder.
 
@@ -1423,8 +1444,6 @@ See the docs at https://docs.victoriametrics.com/vmagent.html .
      Supports array of values separated by comma or specified via multiple flags.
   -remoteWrite.relabelConfig string
      Optional path to file with relabel_config entries. The path can point either to local file or to http url. These entries are applied to all the metrics before sending them to -remoteWrite.url. See https://docs.victoriametrics.com/vmagent.html#relabeling for details
-  -remoteWrite.relabelDebug
-     Whether to log metrics before and after relabeling with -remoteWrite.relabelConfig. If the -remoteWrite.relabelDebug is enabled, then the metrics aren't sent to remote storage. This is useful for debugging the relabeling configs
   -remoteWrite.roundDigits array
      Round metric values to this number of decimal digits after the point before writing them to remote storage. Examples: -remoteWrite.roundDigits=2 would round 1.236 to 1.24, while -remoteWrite.roundDigits=-1 would round 126.78 to 130. By default digits rounding is disabled. Set it to 100 for disabling it for a particular remote storage. This option may be used for improving data compression for the stored metrics
      Supports array of values separated by comma or specified via multiple flags.
@@ -1459,9 +1478,6 @@ See the docs at https://docs.victoriametrics.com/vmagent.html .
   -remoteWrite.urlRelabelConfig array
      Optional path to relabel config for the corresponding -remoteWrite.url. The path can point either to local file or to http url
      Supports an array of values separated by comma or specified via multiple flags.
-  -remoteWrite.urlRelabelDebug array
-     Whether to log metrics before and after relabeling with -remoteWrite.urlRelabelConfig. If the -remoteWrite.urlRelabelDebug is enabled, then the metrics aren't sent to the corresponding -remoteWrite.url. This is useful for debugging the relabeling configs
-     Supports array of values separated by comma or specified via multiple flags.
   -sortLabels
      Whether to sort labels for incoming samples before writing them to all the configured remote storage systems. This may be needed for reducing memory usage at remote storage when the order of labels in incoming samples is random. For example, if m{k1="v1",k2="v2"} may be sent as m{k2="v2",k1="v1"}Enabled sorting for labels can slow down ingestion performance a bit
   -tls

app/vmagent/datadog/request_handler.go

@@ -52,10 +52,18 @@ func insertRows(at *auth.Token, series []parser.Series, extraLabels []prompbmars
 			Name:  "__name__",
 			Value: ss.Metric,
 		})
-		labels = append(labels, prompbmarshal.Label{
-			Name:  "host",
-			Value: ss.Host,
-		})
+		if ss.Host != "" {
+			labels = append(labels, prompbmarshal.Label{
+				Name:  "host",
+				Value: ss.Host,
+			})
+		}
+		if ss.Device != "" {
+			labels = append(labels, prompbmarshal.Label{
+				Name:  "device",
+				Value: ss.Device,
+			})
+		}
 		for _, tag := range ss.Tags {
 			name, value := parser.SplitTag(tag)
 			if name == "host" {

app/vmagent/main.go

@@ -207,6 +207,7 @@ func requestHandler(w http.ResponseWriter, r *http.Request) bool {
 		httpserver.WriteAPIHelp(w, [][2]string{
 			{"targets", "status for discovered active targets"},
 			{"service-discovery", "labels before and after relabeling for discovered targets"},
+			{"metric-relabel-debug", "debug metric relabeling"},
 			{"api/v1/targets", "advanced information about discovered targets in JSON format"},
 			{"config", "-promscrape.config contents"},
 			{"metrics", "available service metrics"},
@@ -325,6 +326,14 @@ func requestHandler(w http.ResponseWriter, r *http.Request) bool {
 		promscrapeServiceDiscoveryRequests.Inc()
 		promscrape.WriteServiceDiscovery(w, r)
 		return true
+	case "/prometheus/metric-relabel-debug", "/metric-relabel-debug":
+		promscrapeMetricRelabelDebugRequests.Inc()
+		promscrape.WriteMetricRelabelDebug(w, r)
+		return true
+	case "/prometheus/target-relabel-debug", "/target-relabel-debug":
+		promscrapeTargetRelabelDebugRequests.Inc()
+		promscrape.WriteTargetRelabelDebug(w, r)
+		return true
 	case "/prometheus/api/v1/targets", "/api/v1/targets":
 		promscrapeAPIV1TargetsRequests.Inc()
 		w.Header().Set("Content-Type", "application/json")
@@ -546,7 +555,11 @@ var (
 
 	promscrapeTargetsRequests          = metrics.NewCounter(`vmagent_http_requests_total{path="/targets"}`)
 	promscrapeServiceDiscoveryRequests = metrics.NewCounter(`vmagent_http_requests_total{path="/service-discovery"}`)
-	promscrapeAPIV1TargetsRequests     = metrics.NewCounter(`vmagent_http_requests_total{path="/api/v1/targets"}`)
+
+	promscrapeMetricRelabelDebugRequests = metrics.NewCounter(`vmagent_http_requests_total{path="/metric-relabel-debug"}`)
+	promscrapeTargetRelabelDebugRequests = metrics.NewCounter(`vmagent_http_requests_total{path="/target-relabel-debug"}`)
+
+	promscrapeAPIV1TargetsRequests = metrics.NewCounter(`vmagent_http_requests_total{path="/api/v1/targets"}`)
 
 	promscrapeTargetResponseRequests = metrics.NewCounter(`vmagent_http_requests_total{path="/target_response"}`)
 	promscrapeTargetResponseErrors   = metrics.NewCounter(`vmagent_http_request_errors_total{path="/target_response"}`)

app/vmagent/remotewrite/relabel.go

@@ -18,13 +18,8 @@ var (
 	relabelConfigPathGlobal = flag.String("remoteWrite.relabelConfig", "", "Optional path to file with relabel_config entries. "+
 		"The path can point either to local file or to http url. These entries are applied to all the metrics "+
 		"before sending them to -remoteWrite.url. See https://docs.victoriametrics.com/vmagent.html#relabeling for details")
-	relabelDebugGlobal = flag.Bool("remoteWrite.relabelDebug", false, "Whether to log metrics before and after relabeling with -remoteWrite.relabelConfig. "+
-		"If the -remoteWrite.relabelDebug is enabled, then the metrics aren't sent to remote storage. This is useful for debugging the relabeling configs")
 	relabelConfigPaths = flagutil.NewArrayString("remoteWrite.urlRelabelConfig", "Optional path to relabel config for the corresponding -remoteWrite.url. "+
 		"The path can point either to local file or to http url")
-	relabelDebug = flagutil.NewArrayBool("remoteWrite.urlRelabelDebug", "Whether to log metrics before and after relabeling with -remoteWrite.urlRelabelConfig. "+
-		"If the -remoteWrite.urlRelabelDebug is enabled, then the metrics aren't sent to the corresponding -remoteWrite.url. "+
-		"This is useful for debugging the relabeling configs")
 
 	usePromCompatibleNaming = flag.Bool("usePromCompatibleNaming", false, "Whether to replace characters unsupported by Prometheus with underscores "+
 		"in the ingested metric names and label names. For example, foo.bar{a.b='c'} is transformed into foo_bar{a_b='c'} during data ingestion if this flag is set. "+
@@ -42,7 +37,7 @@ func CheckRelabelConfigs() error {
 func loadRelabelConfigs() (*relabelConfigs, error) {
 	var rcs relabelConfigs
 	if *relabelConfigPathGlobal != "" {
-		global, err := promrelabel.LoadRelabelConfigs(*relabelConfigPathGlobal, *relabelDebugGlobal)
+		global, err := promrelabel.LoadRelabelConfigs(*relabelConfigPathGlobal)
 		if err != nil {
 			return nil, fmt.Errorf("cannot load -remoteWrite.relabelConfig=%q: %w", *relabelConfigPathGlobal, err)
 		}
@@ -58,7 +53,7 @@ func loadRelabelConfigs() (*relabelConfigs, error) {
 			// Skip empty relabel config.
 			continue
 		}
-		prc, err := promrelabel.LoadRelabelConfigs(path, relabelDebug.GetOptionalArg(i))
+		prc, err := promrelabel.LoadRelabelConfigs(path)
 		if err != nil {
 			return nil, fmt.Errorf("cannot load relabel configs from -remoteWrite.urlRelabelConfig=%q: %w", path, err)
 		}

app/vmagent/remotewrite/remotewrite.go

@@ -251,7 +251,7 @@ func Stop() {
 // Push sends wr to remote storage systems set via `-remoteWrite.url`.
 //
 // If at is nil, then the data is pushed to the configured `-remoteWrite.url`.
-// If at isn't nil, the the data is pushed to the configured `-remoteWrite.multitenantURL`.
+// If at isn't nil, the data is pushed to the configured `-remoteWrite.multitenantURL`.
 //
 // Note that wr may be modified by Push due to relabeling and rounding.
 func Push(at *auth.Token, wr *prompbmarshal.WriteRequest) {

app/vmalert/README.md

@@ -835,7 +835,7 @@ The shortlist of configuration flags is the following:
   -datasource.tlsServerName string
      Optional TLS server name to use for connections to -datasource.url. By default, the server name from -datasource.url is used
   -datasource.url string
-     Datasource compatible with Prometheus HTTP API. It can be single node VictoriaMetrics or vmselect URL. Required parameter. E.g. http://127.0.0.1:8428 . See also '-datasource.disablePathAppend', '-datasource.showURL'.
+     Datasource compatible with Prometheus HTTP API. It can be single node VictoriaMetrics or vmselect URL. Required parameter. E.g. http://127.0.0.1:8428 . See also -remoteRead.disablePathAppend and -datasource.showURL
   -defaultTenant.graphite string
      Default tenant for Graphite alerting groups. See https://docs.victoriametrics.com/vmalert.html#multitenancy .This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
   -defaultTenant.prometheus string
@@ -1050,6 +1050,8 @@ The shortlist of configuration flags is the following:
      Optional OAuth2 scopes to use for -notifier.url. Scopes must be delimited by ';'.
   -remoteWrite.oauth2.tokenUrl string
      Optional OAuth2 tokenURL to use for -notifier.url.
+  -remoteWrite.sendTimeout duration
+     Timeout for sending data to the configured -remoteWrite.url. (default 30s)
   -remoteWrite.showURL
      Whether to show -remoteWrite.url in the exported metrics. It is hidden by default, since it can contain sensitive info such as auth key
   -remoteWrite.tlsCAFile string
@@ -1315,7 +1317,7 @@ spec:
 
 ### Development build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.3.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.
 2. Run `make vmalert` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `vmalert` binary and puts it into the `bin` folder.
 
@@ -1331,7 +1333,7 @@ ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://b
 
 ### Development ARM build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.3.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.
 2. Run `make vmalert-linux-arm` or `make vmalert-linux-arm64` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `vmalert-linux-arm` or `vmalert-linux-arm64` binary respectively and puts it into the `bin` folder.
 

app/vmalert/alerting.go

@@ -284,7 +284,7 @@ func (ar *AlertingRule) Exec(ctx context.Context, ts time.Time, limit int) ([]pr
 		duration: time.Since(start),
 		samples:  len(qMetrics),
 		err:      err,
-		req:      req,
+		curl:     requestToCurl(req),
 	}
 
 	defer func() {

app/vmalert/datasource/datasource.go

@@ -54,6 +54,19 @@ func (m *Metric) SetLabel(key, value string) {
 	m.AddLabel(key, value)
 }
 
+// SetLabels sets the given map as Metric labels
+func (m *Metric) SetLabels(ls map[string]string) {
+	var i int
+	m.Labels = make([]Label, len(ls))
+	for k, v := range ls {
+		m.Labels[i] = Label{
+			Name:  k,
+			Value: v,
+		}
+		i++
+	}
+}
+
 // AddLabel appends the given label to the label set
 func (m *Metric) AddLabel(key, value string) {
 	m.Labels = append(m.Labels, Label{Name: key, Value: value})

app/vmalert/datasource/init.go

@@ -14,7 +14,7 @@ import (
 
 var (
 	addr = flag.String("datasource.url", "", "Datasource compatible with Prometheus HTTP API. It can be single node VictoriaMetrics or vmselect URL. Required parameter. "+
-		"E.g. http://127.0.0.1:8428 . See also '-datasource.disablePathAppend', '-datasource.showURL'.")
+		"E.g. http://127.0.0.1:8428 . See also -remoteRead.disablePathAppend and -datasource.showURL")
 	appendTypePrefix  = flag.Bool("datasource.appendTypePrefix", false, "Whether to add type prefix to -datasource.url based on the query type. Set to true if sending different query types to the vmselect URL.")
 	showDatasourceURL = flag.Bool("datasource.showURL", false, "Whether to show -datasource.url in the exported metrics. "+
 		"It is hidden by default, since it can contain sensitive info such as auth key")

app/vmalert/datasource/vm_prom_api.go

@@ -32,19 +32,17 @@ type promInstant struct {
 }
 
 func (r promInstant) metrics() ([]Metric, error) {
-	var result []Metric
+	result := make([]Metric, len(r.Result))
 	for i, res := range r.Result {
 		f, err := strconv.ParseFloat(res.TV[1].(string), 64)
 		if err != nil {
 			return nil, fmt.Errorf("metric %v, unable to parse float64 from %s: %w", res, res.TV[1], err)
 		}
 		var m Metric
-		for k, v := range r.Result[i].Labels {
-			m.AddLabel(k, v)
-		}
+		m.SetLabels(res.Labels)
 		m.Timestamps = append(m.Timestamps, int64(res.TV[0].(float64)))
 		m.Values = append(m.Values, f)
-		result = append(result, m)
+		result[i] = m
 	}
 	return result, nil
 }
@@ -149,6 +147,16 @@ func (s *VMStorage) setPrometheusInstantReqParams(r *http.Request, query string,
 		timestamp = timestamp.Truncate(s.evaluationInterval)
 	}
 	q.Set("time", fmt.Sprintf("%d", timestamp.Unix()))
+	if s.evaluationInterval > 0 { // set step as evaluationInterval by default
+		// always convert to seconds to keep compatibility with older
+		// Prometheus versions. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1943
+		q.Set("step", fmt.Sprintf("%ds", int(s.evaluationInterval.Seconds())))
+	}
+	if s.queryStep > 0 { // override step with user-specified value
+		// always convert to seconds to keep compatibility with older
+		// Prometheus versions. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1943
+		q.Set("step", fmt.Sprintf("%ds", int(s.queryStep.Seconds())))
+	}
 	r.URL.RawQuery = q.Encode()
 	s.setPrometheusReqParams(r, query)
 }
@@ -163,6 +171,11 @@ func (s *VMStorage) setPrometheusRangeReqParams(r *http.Request, query string, s
 	q := r.URL.Query()
 	q.Add("start", fmt.Sprintf("%d", start.Unix()))
 	q.Add("end", fmt.Sprintf("%d", end.Unix()))
+	if s.evaluationInterval > 0 { // set step as evaluationInterval by default
+		// always convert to seconds to keep compatibility with older
+		// Prometheus versions. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1943
+		q.Set("step", fmt.Sprintf("%ds", int(s.evaluationInterval.Seconds())))
+	}
 	r.URL.RawQuery = q.Encode()
 	s.setPrometheusReqParams(r, query)
 }
@@ -178,15 +191,5 @@ func (s *VMStorage) setPrometheusReqParams(r *http.Request, query string) {
 		}
 	}
 	q.Set("query", query)
-	if s.evaluationInterval > 0 { // set step as evaluationInterval by default
-		// always convert to seconds to keep compatibility with older
-		// Prometheus versions. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1943
-		q.Set("step", fmt.Sprintf("%ds", int(s.evaluationInterval.Seconds())))
-	}
-	if s.queryStep > 0 { // override step with user-specified value
-		// always convert to seconds to keep compatibility with older
-		// Prometheus versions. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1943
-		q.Set("step", fmt.Sprintf("%ds", int(s.queryStep.Seconds())))
-	}
 	r.URL.RawQuery = q.Encode()
 }

app/vmalert/datasource/vm_prom_api_test.go

@@ -0,0 +1,20 @@
+package datasource
+
+import (
+	"encoding/json"
+	"testing"
+)
+
+func BenchmarkMetrics(b *testing.B) {
+	payload := []byte(`[{"metric":{"__name__":"vm_rows"},"value":[1583786142,"13763"]},{"metric":{"__name__":"vm_requests", "foo":"bar", "baz": "qux"},"value":[1583786140,"2000"]}]`)
+
+	var pi promInstant
+	if err := json.Unmarshal(payload, &pi.Result); err != nil {
+		b.Fatalf(err.Error())
+	}
+	b.Run("Instant", func(b *testing.B) {
+		for i := 0; i < b.N; i++ {
+			_, _ = pi.metrics()
+		}
+	})
+}

app/vmalert/datasource/vm_test.go

@@ -7,6 +7,7 @@ import (
 	"net/http/httptest"
 	"net/url"
 	"reflect"
+	"sort"
 	"strconv"
 	"strings"
 	"testing"
@@ -74,7 +75,7 @@ func TestVMInstantQuery(t *testing.T) {
 		case 5:
 			w.Write([]byte(`{"status":"success","data":{"resultType":"matrix"}}`))
 		case 6:
-			w.Write([]byte(`{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"vm_rows"},"value":[1583786142,"13763"]},{"metric":{"__name__":"vm_requests"},"value":[1583786140,"2000"]}]}}`))
+			w.Write([]byte(`{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"vm_rows","foo":"bar"},"value":[1583786142,"13763"]},{"metric":{"__name__":"vm_requests","foo":"baz"},"value":[1583786140,"2000"]}]}}`))
 		case 7:
 			w.Write([]byte(`{"status":"success","data":{"resultType":"scalar","result":[1583786142, "1"]}}`))
 		}
@@ -115,19 +116,17 @@ func TestVMInstantQuery(t *testing.T) {
 	}
 	expected := []Metric{
 		{
-			Labels:     []Label{{Value: "vm_rows", Name: "__name__"}},
+			Labels:     []Label{{Value: "vm_rows", Name: "__name__"}, {Value: "bar", Name: "foo"}},
 			Timestamps: []int64{1583786142},
 			Values:     []float64{13763},
 		},
 		{
-			Labels:     []Label{{Value: "vm_requests", Name: "__name__"}},
+			Labels:     []Label{{Value: "vm_requests", Name: "__name__"}, {Value: "baz", Name: "foo"}},
 			Timestamps: []int64{1583786140},
 			Values:     []float64{2000},
 		},
 	}
-	if !reflect.DeepEqual(m, expected) {
-		t.Fatalf("unexpected metric %+v want %+v", m, expected)
-	}
+	metricsEqual(t, m, expected)
 
 	m, req, err := pq.Query(ctx, query, ts) // 7 - scalar
 	if err != nil {
@@ -158,13 +157,36 @@ func TestVMInstantQuery(t *testing.T) {
 	if len(m) != 1 {
 		t.Fatalf("expected 1 metric  got %d in %+v", len(m), m)
 	}
-	exp := Metric{
-		Labels:     []Label{{Value: "constantLine(10)", Name: "name"}},
-		Timestamps: []int64{1611758403},
-		Values:     []float64{10},
+	exp := []Metric{
+		{
+			Labels:     []Label{{Value: "constantLine(10)", Name: "name"}},
+			Timestamps: []int64{1611758403},
+			Values:     []float64{10},
+		},
 	}
-	if !reflect.DeepEqual(m[0], exp) {
-		t.Fatalf("unexpected metric %+v want %+v", m[0], expected)
+	metricsEqual(t, m, exp)
+}
+
+func metricsEqual(t *testing.T, gotM, expectedM []Metric) {
+	for i, exp := range expectedM {
+		got := gotM[i]
+		gotTS, expTS := got.Timestamps, exp.Timestamps
+		if !reflect.DeepEqual(gotTS, expTS) {
+			t.Fatalf("unexpected timestamps %+v want %+v", gotTS, expTS)
+		}
+		gotV, expV := got.Values, exp.Values
+		if !reflect.DeepEqual(gotV, expV) {
+			t.Fatalf("unexpected values %+v want %+v", gotV, expV)
+		}
+		sort.Slice(got.Labels, func(i, j int) bool {
+			return got.Labels[i].Name < got.Labels[j].Name
+		})
+		sort.Slice(exp.Labels, func(i, j int) bool {
+			return exp.Labels[i].Name < exp.Labels[j].Name
+		})
+		if !reflect.DeepEqual(exp.Labels, got.Labels) {
+			t.Fatalf("unexpected labels %+v want %+v", got.Labels, exp.Labels)
+		}
 	}
 }
 
@@ -199,6 +221,10 @@ func TestVMRangeQuery(t *testing.T) {
 		if _, err := strconv.ParseInt(endTS, 10, 64); err != nil {
 			t.Errorf("failed to parse 'end' query param: %s", err)
 		}
+		step := r.URL.Query().Get("step")
+		if step != "15s" {
+			t.Errorf("expected 'step' query param to be 15s; got %q instead", step)
+		}
 		switch c {
 		case 0:
 			w.Write([]byte(`{"status":"success","data":{"resultType":"matrix","result":[{"metric":{"__name__":"vm_rows"},"values":[[1583786142,"13763"]]}]}}`))
@@ -212,7 +238,7 @@ func TestVMRangeQuery(t *testing.T) {
 	if err != nil {
 		t.Fatalf("unexpected: %s", err)
 	}
-	s := NewVMStorage(srv.URL, authCfg, time.Minute, 0, false, srv.Client())
+	s := NewVMStorage(srv.URL, authCfg, time.Minute, *queryStep, false, srv.Client())
 
 	pq := s.BuildWithParams(QuerierParams{DataSourceType: string(datasourcePrometheus), EvaluationInterval: 15 * time.Second})
 

app/vmalert/group.go

@@ -168,9 +168,12 @@ func (g *Group) Restore(ctx context.Context, qb datasource.QuerierBuilder, lookb
 		if rr.For < 1 {
 			continue
 		}
-		// ignore g.ExtraFilterLabels on purpose, so it
-		// won't affect the restore procedure.
-		q := qb.BuildWithParams(datasource.QuerierParams{})
+		// ignore QueryParams on purpose, because they could contain
+		// query filters. This may affect the restore procedure.
+		q := qb.BuildWithParams(datasource.QuerierParams{
+			DataSourceType: g.Type.String(),
+			Headers:        g.Headers,
+		})
 		if err := rr.Restore(ctx, q, lookback, labels); err != nil {
 			return fmt.Errorf("error while restoring rule %q: %w", rule, err)
 		}
@@ -418,20 +421,26 @@ func (e *executor) exec(ctx context.Context, rule Rule, ts time.Time, resolveDur
 		return fmt.Errorf("rule %q: failed to execute: %w", rule, err)
 	}
 
-	errGr := new(utils.ErrGroup)
 	if e.rw != nil {
-		pushToRW := func(tss []prompbmarshal.TimeSeries) {
+		pushToRW := func(tss []prompbmarshal.TimeSeries) error {
+			var lastErr error
 			for _, ts := range tss {
 				remoteWriteTotal.Inc()
 				if err := e.rw.Push(ts); err != nil {
 					remoteWriteErrors.Inc()
-					errGr.Add(fmt.Errorf("rule %q: remote write failure: %w", rule, err))
+					lastErr = fmt.Errorf("rule %q: remote write failure: %w", rule, err)
 				}
 			}
+			return lastErr
 		}
-		pushToRW(tss)
+		if err := pushToRW(tss); err != nil {
+			return err
+		}
+
 		staleSeries := e.getStaleSeries(rule, tss, ts)
-		pushToRW(staleSeries)
+		if err := pushToRW(staleSeries); err != nil {
+			return err
+		}
 	}
 
 	ar, ok := rule.(*AlertingRule)
@@ -445,6 +454,7 @@ func (e *executor) exec(ctx context.Context, rule Rule, ts time.Time, resolveDur
 	}
 
 	wg := sync.WaitGroup{}
+	errGr := new(utils.ErrGroup)
 	for _, nt := range e.notifiers() {
 		wg.Add(1)
 		go func(nt notifier.Notifier) {

app/vmalert/group_test.go

@@ -3,8 +3,6 @@ package main
 import (
 	"context"
 	"fmt"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/decimal"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
 	"reflect"
 	"sort"
 	"testing"
@@ -12,6 +10,9 @@ import (
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/config"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/notifier"
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/remotewrite"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/decimal"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promutils"
 )
 
@@ -452,3 +453,24 @@ func TestFaultyNotifier(t *testing.T) {
 	}
 	t.Fatalf("alive notifier didn't receive notification by %v", deadline)
 }
+
+func TestFaultyRW(t *testing.T) {
+	fq := &fakeQuerier{}
+	fq.add(metricWithValueAndLabels(t, 1, "__name__", "foo", "job", "bar"))
+
+	r := &RecordingRule{
+		Name:  "test",
+		state: newRuleState(),
+		q:     fq,
+	}
+
+	e := &executor{
+		rw:                       &remotewrite.Client{},
+		previouslySentSeriesToRW: make(map[uint64]map[string][]prompbmarshal.Label),
+	}
+
+	err := e.exec(context.Background(), r, time.Now(), 0, 10)
+	if err == nil {
+		t.Fatalf("expected to get an error from faulty RW client, got nil instead")
+	}
+}

app/vmalert/notifier/alert_test.go

@@ -239,7 +239,7 @@ func TestAlert_toPromLabels(t *testing.T) {
   replacement: "aaa"
 - action: labeldrop
   regex: "env.*"
-`), false)
+`))
 	if err != nil {
 		t.Fatalf("unexpected error: %s", err)
 	}

app/vmalert/notifier/config.go

@@ -12,7 +12,6 @@ import (
 	"time"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promauth"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promrelabel"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape/discovery/consul"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape/discovery/dns"
@@ -84,12 +83,12 @@ func (cfg *Config) UnmarshalYAML(unmarshal func(interface{}) error) error {
 	if cfg.Timeout.Duration() == 0 {
 		cfg.Timeout = promutils.NewDuration(time.Second * 10)
 	}
-	rCfg, err := promrelabel.ParseRelabelConfigs(cfg.RelabelConfigs, false)
+	rCfg, err := promrelabel.ParseRelabelConfigs(cfg.RelabelConfigs)
 	if err != nil {
 		return fmt.Errorf("failed to parse relabeling config: %w", err)
 	}
 	cfg.parsedRelabelConfigs = rCfg
-	arCfg, err := promrelabel.ParseRelabelConfigs(cfg.AlertRelabelConfigs, false)
+	arCfg, err := promrelabel.ParseRelabelConfigs(cfg.AlertRelabelConfigs)
 	if err != nil {
 		return fmt.Errorf("failed to parse alert relabeling config: %w", err)
 	}
@@ -130,24 +129,24 @@ func parseConfig(path string) (*Config, error) {
 	return cfg, nil
 }
 
-func parseLabels(target string, metaLabels map[string]string, cfg *Config) (string, []prompbmarshal.Label, error) {
+func parseLabels(target string, metaLabels *promutils.Labels, cfg *Config) (string, *promutils.Labels, error) {
 	labels := mergeLabels(target, metaLabels, cfg)
-	labels = cfg.parsedRelabelConfigs.Apply(labels, 0)
-	labels = promrelabel.RemoveMetaLabels(labels[:0], labels)
-	promrelabel.SortLabels(labels)
+	labels.Labels = cfg.parsedRelabelConfigs.Apply(labels.Labels, 0)
+	labels.RemoveMetaLabels()
+	labels.Sort()
 	// Remove references to already deleted labels, so GC could clean strings for label name and label value past len(labels).
 	// This should reduce memory usage when relabeling creates big number of temporary labels with long names and/or values.
 	// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825 for details.
-	labels = append([]prompbmarshal.Label{}, labels...)
+	labels = labels.Clone()
 
-	if len(labels) == 0 {
+	if labels.Len() == 0 {
 		return "", nil, nil
 	}
-	schemeRelabeled := promrelabel.GetLabelValueByName(labels, "__scheme__")
+	schemeRelabeled := labels.Get("__scheme__")
 	if len(schemeRelabeled) == 0 {
 		schemeRelabeled = "http"
 	}
-	addressRelabeled := promrelabel.GetLabelValueByName(labels, "__address__")
+	addressRelabeled := labels.Get("__address__")
 	if len(addressRelabeled) == 0 {
 		return "", nil, nil
 	}
@@ -155,7 +154,7 @@ func parseLabels(target string, metaLabels map[string]string, cfg *Config) (stri
 		return "", nil, nil
 	}
 	addressRelabeled = addMissingPort(schemeRelabeled, addressRelabeled)
-	alertsPathRelabeled := promrelabel.GetLabelValueByName(labels, "__alerts_path__")
+	alertsPathRelabeled := labels.Get("__alerts_path__")
 	if !strings.HasPrefix(alertsPathRelabeled, "/") {
 		alertsPathRelabeled = "/" + alertsPathRelabeled
 	}
@@ -179,21 +178,12 @@ func addMissingPort(scheme, target string) string {
 	return target
 }
 
-func mergeLabels(target string, metaLabels map[string]string, cfg *Config) []prompbmarshal.Label {
+func mergeLabels(target string, metaLabels *promutils.Labels, cfg *Config) *promutils.Labels {
 	// See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
-	m := make(map[string]string)
-	m["__address__"] = target
-	m["__scheme__"] = cfg.Scheme
-	m["__alerts_path__"] = path.Join("/", cfg.PathPrefix, alertManagerPath)
-	for k, v := range metaLabels {
-		m[k] = v
-	}
-	result := make([]prompbmarshal.Label, 0, len(m))
-	for k, v := range m {
-		result = append(result, prompbmarshal.Label{
-			Name:  k,
-			Value: v,
-		})
-	}
-	return result
+	m := promutils.NewLabels(3 + metaLabels.Len())
+	m.Add("__address__", target)
+	m.Add("__scheme__", cfg.Scheme)
+	m.Add("__alerts_path__", path.Join("/", cfg.PathPrefix, alertManagerPath))
+	m.AddFrom(metaLabels)
+	return m
 }

app/vmalert/notifier/config_watcher.go

@@ -9,6 +9,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promauth"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape/discovery/consul"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape/discovery/dns"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promutils"
 )
 
 // configWatcher supports dynamic reload of Notifier objects
@@ -123,7 +124,7 @@ func targetsFromLabels(labelsFn getLabels, cfg *Config, genFn AlertURLGenerator)
 	var errors []error
 	duplicates := make(map[string]struct{})
 	for _, labels := range metaLabels {
-		target := labels["__address__"]
+		target := labels.Get("__address__")
 		u, processedLabels, err := parseLabels(target, labels, cfg)
 		if err != nil {
 			errors = append(errors, err)
@@ -156,7 +157,7 @@ func targetsFromLabels(labelsFn getLabels, cfg *Config, genFn AlertURLGenerator)
 	return targets, errors
 }
 
-type getLabels func() ([]map[string]string, error)
+type getLabels func() ([]*promutils.Labels, error)
 
 func (cw *configWatcher) start() error {
 	if len(cw.cfg.StaticConfigs) > 0 {
@@ -182,8 +183,8 @@ func (cw *configWatcher) start() error {
 	}
 
 	if len(cw.cfg.ConsulSDConfigs) > 0 {
-		err := cw.add(TargetConsul, *consul.SDCheckInterval, func() ([]map[string]string, error) {
-			var labels []map[string]string
+		err := cw.add(TargetConsul, *consul.SDCheckInterval, func() ([]*promutils.Labels, error) {
+			var labels []*promutils.Labels
 			for i := range cw.cfg.ConsulSDConfigs {
 				sdc := &cw.cfg.ConsulSDConfigs[i]
 				targetLabels, err := sdc.GetLabels(cw.cfg.baseDir)
@@ -200,8 +201,8 @@ func (cw *configWatcher) start() error {
 	}
 
 	if len(cw.cfg.DNSSDConfigs) > 0 {
-		err := cw.add(TargetDNS, *dns.SDCheckInterval, func() ([]map[string]string, error) {
-			var labels []map[string]string
+		err := cw.add(TargetDNS, *dns.SDCheckInterval, func() ([]*promutils.Labels, error) {
+			var labels []*promutils.Labels
 			for i := range cw.cfg.DNSSDConfigs {
 				sdc := &cw.cfg.DNSSDConfigs[i]
 				targetLabels, err := sdc.GetLabels(cw.cfg.baseDir)

app/vmalert/notifier/init.go

@@ -10,7 +10,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/templates"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promauth"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promutils"
 )
 
 var (
@@ -159,7 +159,7 @@ func notifiersFromFlags(gen AlertURLGenerator) ([]Notifier, error) {
 // list of labels added during discovery.
 type Target struct {
 	Notifier
-	Labels []prompbmarshal.Label
+	Labels *promutils.Labels
 }
 
 // TargetType defines how the Target was discovered

app/vmalert/recording.go

@@ -121,7 +121,7 @@ func (rr *RecordingRule) Exec(ctx context.Context, ts time.Time, limit int) ([]p
 		at:       ts,
 		duration: time.Since(start),
 		samples:  len(qMetrics),
-		req:      req,
+		curl:     requestToCurl(req),
 	}
 
 	defer func() {

app/vmalert/remotewrite/init.go

@@ -77,13 +77,12 @@ func Init(ctx context.Context) (*Client, error) {
 	}
 
 	return NewClient(ctx, Config{
-		Addr:              *addr,
-		AuthCfg:           authCfg,
-		Concurrency:       *concurrency,
-		MaxQueueSize:      *maxQueueSize,
-		MaxBatchSize:      *maxBatchSize,
-		FlushInterval:     *flushInterval,
-		DisablePathAppend: *disablePathAppend,
-		Transport:         t,
+		Addr:          *addr,
+		AuthCfg:       authCfg,
+		Concurrency:   *concurrency,
+		MaxQueueSize:  *maxQueueSize,
+		MaxBatchSize:  *maxBatchSize,
+		FlushInterval: *flushInterval,
+		Transport:     t,
 	})
 }

app/vmalert/remotewrite/remotewrite.go

@@ -22,6 +22,7 @@ import (
 
 var (
 	disablePathAppend = flag.Bool("remoteWrite.disablePathAppend", false, "Whether to disable automatic appending of '/api/v1/write' path to the configured -remoteWrite.url.")
+	sendTimeout       = flag.Duration("remoteWrite.sendTimeout", 30*time.Second, "Timeout for sending data to the configured -remoteWrite.url.")
 )
 
 // Client is an asynchronous HTTP client for writing
@@ -57,13 +58,8 @@ type Config struct {
 	MaxQueueSize int
 	// FlushInterval defines time interval for flushing batches
 	FlushInterval time.Duration
-	// WriteTimeout defines timeout for HTTP write request
-	// to remote storage
-	WriteTimeout time.Duration
 	// Transport will be used by the underlying http.Client
 	Transport *http.Transport
-	// DisablePathAppend can be used to not automatically append '/api/v1/write' to the remote write url
-	DisablePathAppend bool
 }
 
 const (
@@ -89,9 +85,6 @@ func NewClient(ctx context.Context, cfg Config) (*Client, error) {
 	if cfg.FlushInterval == 0 {
 		cfg.FlushInterval = defaultFlushInterval
 	}
-	if cfg.WriteTimeout == 0 {
-		cfg.WriteTimeout = defaultWriteTimeout
-	}
 	if cfg.Transport == nil {
 		cfg.Transport = http.DefaultTransport.(*http.Transport).Clone()
 	}
@@ -101,7 +94,7 @@ func NewClient(ctx context.Context, cfg Config) (*Client, error) {
 	}
 	c := &Client{
 		c: &http.Client{
-			Timeout:   cfg.WriteTimeout,
+			Timeout:   *sendTimeout,
 			Transport: cfg.Transport,
 		},
 		addr:          strings.TrimSuffix(cfg.Addr, "/"),

app/vmalert/rule.go

@@ -3,7 +3,6 @@ package main
 import (
 	"context"
 	"errors"
-	"net/http"
 	"sync"
 	"time"
 
@@ -54,8 +53,8 @@ type ruleStateEntry struct {
 	// stores the number of samples returned during
 	// the last evaluation
 	samples int
-	// stores the HTTP request used by datasource during rule.Exec
-	req *http.Request
+	// stores the curl command reflecting the HTTP request used during rule.Exec
+	curl string
 }
 
 const defaultStateEntriesLimit = 20

app/vmalert/utils.go

@@ -72,7 +72,7 @@ func (cw *curlWriter) add(str string) {
 }
 
 func requestToCurl(req *http.Request) string {
-	if req.URL == nil {
+	if req == nil || req.URL == nil {
 		return ""
 	}
 

app/vmalert/web.go

@@ -8,7 +8,6 @@ import (
 	"sort"
 	"strconv"
 	"strings"
-	"sync"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/notifier"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/tpl"
@@ -18,20 +17,14 @@ import (
 )
 
 var (
-	once     = sync.Once{}
-	apiLinks [][2]string
-	navItems []tpl.NavItem
-)
-
-func initLinks() {
 	apiLinks = [][2]string{
 		// api links are relative since they can be used by external clients,
 		// such as Grafana, and proxied via vmselect.
 		{"api/v1/rules", "list all loaded groups and rules"},
 		{"api/v1/alerts", "list all active alerts"},
 		{fmt.Sprintf("api/v1/alert?%s=<int>&%s=<int>", paramGroupID, paramAlertID), "get alert status by group and alert ID"},
-
-		// system links
+	}
+	systemLinks = [][2]string{
 		{"/flags", "command-line flags"},
 		{"/metrics", "list of application metrics"},
 		{"/-/reload", "reload configuration"},
@@ -43,7 +36,7 @@ func initLinks() {
 		{Name: "Notifiers", Url: "notifiers"},
 		{Name: "Docs", Url: "https://docs.victoriametrics.com/vmalert.html"},
 	}
-}
+)
 
 type requestHandler struct {
 	m *manager
@@ -57,10 +50,6 @@ var (
 )
 
 func (rh *requestHandler) handler(w http.ResponseWriter, r *http.Request) bool {
-	once.Do(func() {
-		initLinks()
-	})
-
 	if strings.HasPrefix(r.URL.Path, "/vmalert/static") {
 		staticServer.ServeHTTP(w, r)
 		return true

app/vmalert/web.qtpl

@@ -16,11 +16,16 @@
     <p>
         API:<br>
         {% for _, p := range apiLinks  %}
-            {%code
-                p, doc := p[0], p[1]
-            %}
-        	<a href="{%s p %}">{%s p %}</a> - {%s doc %}<br/>
+            {%code p, doc := p[0], p[1] %}
+            <a href="{%s p %}">{%s p %}</a> - {%s doc %}<br/>
         {% endfor %}
+        {% if r.Header.Get("X-Forwarded-For") == "" %}
+            System:<br>
+            {% for _, p := range systemLinks  %}
+                {%code p, doc := p[0], p[1] %}
+                <a href="{%s p %}">{%s p %}</a> - {%s doc %}<br/>
+            {% endfor %}
+        {% endif %}
     </p>
     {%= tpl.Footer(r) %}
 {% endfunc %}
@@ -248,7 +253,7 @@
                  {% for _, n := range ns %}
                      <tr>
                          <td>
-                              {% for _, l := range n.Labels %}
+                              {% for _, l := range n.Labels.GetLabels() %}
                                       <span class="ms-1 badge bg-primary">{%s l.Name %}={%s l.Value %}</span>
                               {% endfor %}
                           </td>
@@ -457,7 +462,7 @@
                  <td class="text-center">{%f.3 u.duration.Seconds() %}s</td>
                  <td class="text-center">{%s u.at.Format(time.RFC3339) %}</td>
                  <td>
-                    <textarea class="curl-area" rows="1" onclick="this.focus();this.select()">{%s requestToCurl(u.req) %}</textarea>
+                    <textarea class="curl-area" rows="1" onclick="this.focus();this.select()">{%s u.curl %}</textarea>
                 </td>
              </tr>
           </li>

app/vmauth/README.md

@@ -167,7 +167,7 @@ It is recommended using [binary releases](https://github.com/VictoriaMetrics/Vic
 
 ### Development build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.3.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.
 2. Run `make vmauth` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `vmauth` binary and puts it into the `bin` folder.
 

app/vmbackup/README.md

@@ -286,7 +286,7 @@ It is recommended using [binary releases](https://github.com/VictoriaMetrics/Vic
 
 ### Development build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.3.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.
 2. Run `make vmbackup` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `vmbackup` binary and puts it into the `bin` folder.
 

app/vmctl/README.md

@@ -7,9 +7,12 @@ vmctl provides various useful actions with VictoriaMetrics components.
 Features:
 - migrate data from [Prometheus](#migrating-data-from-prometheus) to VictoriaMetrics using snapshot API
 - migrate data from [Thanos](#migrating-data-from-thanos) to VictoriaMetrics
+- migrate data from [Cortex](#migrating-data-from-cortex) to VictoriaMetrics
+- migrate data from [Mimir](#migrating-data-from-mimir) to VictoriaMetrics
 - migrate data from [InfluxDB](#migrating-data-from-influxdb-1x) to VictoriaMetrics
 - migrate data from [OpenTSDB](#migrating-data-from-opentsdb) to VictoriaMetrics
 - migrate data between [VictoriaMetrics](#migrating-data-from-victoriametrics) single or cluster version.
+- migrate data by [Prometheus remote read protocol](#migrating-data-by-remote-read-protocol) to VictoriaMetrics
 - [verify](#verifying-exported-blocks-from-victoriametrics) exported blocks from VictoriaMetrics single or cluster version.
 
 To see the full list of supported modes
@@ -28,6 +31,7 @@ COMMANDS:
    influx      Migrate timeseries from InfluxDB
    prometheus  Migrate timeseries from Prometheus
    vm-native   Migrate time series between VictoriaMetrics installations via native binary format
+   remote-read Migrate timeseries by Prometheus remote read protocol
    verify-block  Verifies correctness of data blocks exported via VictoriaMetrics Native format. See https://docs.victoriametrics.com/#how-to-export-data-in-native-format
 ```
 
@@ -432,6 +436,64 @@ Found 2 blocks to import. Continue? [Y/n] y
 2020/02/23 15:51:07 Total time: 7.153158218s
 ```
 
+## Migrating data by remote read protocol
+
+`vmctl` supports the `remote-read` mode for migrating data from databases which support 
+[Prometheus remote read API](https://prometheus.io/docs/prometheus/latest/querying/remote_read_api/)
+
+See `./vmctl remote-read --help` for details and full list of flags.
+
+To start the migration process configure the following flags:
+1. `--remote-read-src-addr` - data source address to read from;
+2. `--vm-addr` - VictoriaMetrics address to write to. For single-node VM is usually equal to `--httpListenAddr`, 
+and for cluster version is equal to `--httpListenAddr` flag of vminsert component (for example `http://<vminsert>:8480/insert/<accountID>/prometheus`);
+3. `--remote-read-filter-time-start` - the time filter in RFC3339 format to select time series with timestamp equal or higher than provided value. E.g. '2020-01-01T20:07:00Z';
+4. `--remote-read-filter-time-end` - the time filter in RFC3339 format to select time series with timestamp equal or smaller than provided value. E.g. '2020-01-01T20:07:00Z'. Current time is used when omitted.;
+5. `--remote-read-step-interval` - split export data into chunks. Valid values are `month, day, hour, minute`;
+
+The importing process example for local installation of Prometheus
+and single-node VictoriaMetrics(`http://localhost:8428`):
+
+```
+./vmctl remote-read \
+--remote-read-src-addr=http://127.0.0.1:9091 \
+--remote-read-filter-time-start=2021-10-18T00:00:00Z \
+--remote-read-step-interval=hour \
+--vm-addr=http://127.0.0.1:8428 \
+--vm-concurrency=6
+
+Split defined times into 8798 ranges to import. Continue? [Y/n]
+VM worker 0:↘ 127177 samples/s
+VM worker 1:↘ 140137 samples/s
+VM worker 2:↘ 151606 samples/s
+VM worker 3:↘ 130765 samples/s
+VM worker 4:↘ 131904 samples/s
+VM worker 5:↘ 132693 samples/s
+Processing ranges: 8798 / 8798 [█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████] 100.00%
+2022/10/19 16:45:37 Import finished!
+2022/10/19 16:45:37 VictoriaMetrics importer stats:
+  idle duration: 6m57.793987511s;
+  time spent while importing: 1m18.463744801s;
+  total samples: 25348208;
+  samples/s: 323056.31;
+  total bytes: 669.7 MB;
+  bytes/s: 8.5 MB;
+  import requests: 127;
+  import requests retries: 0;
+2022/10/19 16:45:37 Total time: 1m19.406283424s
+```
+
+### Filtering
+
+The filtering consists of two parts: by labels and time.
+
+Filtering by time can be configured via flags `--remote-read-filter-time-start` and `--remote-read-filter-time-end`
+in RFC3339 format.
+
+Filtering by labels can be configured via flags `--remote-read-filter-label` and `--remote-read-filter-label-value`.
+For example, `--remote-read-filter-label=tenant` and `--remote-read-filter-label-value="team-eu"` will select only series
+with `tenant="team-eu"` label-value pair.
+
 ## Migrating data from Thanos
 
 Thanos uses the same storage engine as Prometheus and the data layout on-disk should be the same. That means
@@ -478,6 +540,187 @@ then import it into VM using `vmctl` in `prometheus` mode.
     vmctl prometheus --prom-snapshot thanos-data --vm-addr http://victoria-metrics:8428
     ```
 
+### Remote read protocol
+
+Currently, Thanos doesn't support streaming remote read protocol. It is [recommended](https://thanos.io/tip/thanos/integrations.md/#storeapi-as-prometheus-remote-read)
+to use [thanos-remote-read](https://github.com/G-Research/thanos-remote-read) a proxy, that allows exposing any Thanos 
+service (or anything that exposes gRPC StoreAPI e.g. Querier) via Prometheus remote read protocol.
+
+If you want to migrate data, you should run [thanos-remote-read](https://github.com/G-Research/thanos-remote-read) proxy
+and define the Thanos store address `./thanos-remote-read -store 127.0.0.1:19194`. 
+It is important to know that `store` flag is Thanos Store API gRPC endpoint. 
+Also, it is important to know that thanos-remote-read proxy  doesn't support `STREAMED_XOR_CHUNKS` mode.
+When you run thanos-remote-read proxy, it exposes port to serve HTTP on `10080 by default`.
+
+The importing process example for local installation of Thanos
+and single-node VictoriaMetrics(`http://localhost:8428`):
+
+```
+./vmctl remote-read \
+--remote-read-src-addr=http://127.0.0.1:10080 \
+--remote-read-filter-time-start=2021-10-18T00:00:00Z \
+--remote-read-step-interval=hour \
+--vm-addr=http://127.0.0.1:8428 \
+--vm-concurrency=6
+```
+
+On the [thanos-remote-read](https://github.com/G-Research/thanos-remote-read) proxy side you will see logs like:
+```
+ts=2022-10-19T15:05:04.193916Z caller=main.go:278 level=info traceID=00000000000000000000000000000000 msg="thanos request" request="min_time:1666180800000 max_time:1666184399999 matchers:<type:RE value:\".*\" > aggregates:RAW "
+ts=2022-10-19T15:05:04.468852Z caller=main.go:278 level=info traceID=00000000000000000000000000000000 msg="thanos request" request="min_time:1666184400000 max_time:1666187999999 matchers:<type:RE value:\".*\" > aggregates:RAW "
+ts=2022-10-19T15:05:04.553914Z caller=main.go:278 level=info traceID=00000000000000000000000000000000 msg="thanos request" request="min_time:1666188000000 max_time:1666191364863 matchers:<type:RE value:\".*\" > aggregates:RAW "
+```
+
+And when process will finish you will see:
+```
+Split defined times into 8799 ranges to import. Continue? [Y/n]
+VM worker 0:↓ 98183 samples/s
+VM worker 1:↓ 114640 samples/s
+VM worker 2:↓ 131710 samples/s
+VM worker 3:↓ 114256 samples/s
+VM worker 4:↓ 105671 samples/s
+VM worker 5:↓ 124000 samples/s
+Processing ranges: 8799 / 8799 [█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████] 100.00%
+2022/10/19 18:05:07 Import finished!
+2022/10/19 18:05:07 VictoriaMetrics importer stats:
+  idle duration: 52m13.987637229s;
+  time spent while importing: 9m1.728983776s;
+  total samples: 70836111;
+  samples/s: 130759.32;
+  total bytes: 2.2 GB;
+  bytes/s: 4.0 MB;
+  import requests: 356;
+  import requests retries: 0;
+2022/10/19 18:05:07 Total time: 9m2.607521618s
+```
+
+## Migrating data from Cortex
+
+Cortex has an implementation of the Prometheus remote read protocol. That means
+`vmctl` in mode `remote-read` may also be used for Cortex historical data migration.
+These instructions may vary based on the details of your Cortex configuration.
+Please read carefully and verify as you go.
+
+### Remote read protocol
+
+If you want to migrate data, you should check your cortex configuration in the section
+```yaml
+api:
+  prometheus_http_prefix:
+```
+
+If you defined some prometheus prefix, you should use it when you define flag `--remote-read-src-addr=http://127.0.0.1:9009/{prometheus_http_prefix}`.
+By default, Cortex uses the `prometheus` path prefix, so you should define the flag `--remote-read-src-addr=http://127.0.0.1:9009/prometheus`.
+
+It is important to know that Cortex doesn't support the `STREAMED_XOR_CHUNKS` mode.
+When you run Cortex, it exposes a port to serve HTTP on `9009 by default`.
+
+The importing process example for the local installation of Cortex
+and single-node VictoriaMetrics(`http://localhost:8428`):
+
+```
+./vmctl remote-read \ 
+--remote-read-src-addr=http://127.0.0.1:9009/prometheus \
+--remote-read-filter-time-start=2021-10-18T00:00:00Z \
+--remote-read-step-interval=hour \
+--remote-read-src-check-alive=false \
+--vm-addr=http://127.0.0.1:8428 \
+--vm-concurrency=6 
+```
+And when the process finishes, you will see the following:
+
+```
+Split defined times into 8842 ranges to import. Continue? [Y/n]
+VM worker 0:↗ 3863 samples/s
+VM worker 1:↗ 2686 samples/s
+VM worker 2:↗ 2620 samples/s
+VM worker 3:↗ 2705 samples/s
+VM worker 4:↗ 2643 samples/s
+VM worker 5:↗ 2593 samples/s
+Processing ranges: 8842 / 8842 [█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████] 100.00%
+2022/10/21 12:09:49 Import finished!
+2022/10/21 12:09:49 VictoriaMetrics importer stats:
+  idle duration: 0s;
+  time spent while importing: 3.82640757s;
+  total samples: 160232;
+  samples/s: 41875.31;
+  total bytes: 11.3 MB;
+  bytes/s: 3.0 MB;
+  import requests: 6;
+  import requests retries: 0;
+2022/10/21 12:09:49 Total time: 4.71824253s
+```
+It is important to know that if you run your Cortex installation in multi-tenant mode, remote read protocol
+requires an Authentication header like `X-Scope-OrgID`. You can define it via the flag `--remote-read-headers=X-Scope-OrgID:demo`
+
+## Migrating data from Mimir
+
+Mimir has similar implemintation as Cortex and also support of the Prometheus remote read protocol. That means
+`vmctl` in mode `remote-read` may also be used for Mimir historical data migration.
+These instructions may vary based on the details of your Mimir configuration.
+Please read carefully and verify as you go.
+
+### Remote read protocol
+
+If you want to migrate data, you should check your Mimir configuration in the section
+```yaml
+api:
+  prometheus_http_prefix:
+```
+
+If you defined some prometheus prefix, you should use it when you define flag `--remote-read-src-addr=http://127.0.0.1:9009/{prometheus_http_prefix}`.
+By default, Mimir uses the `prometheus` path prefix, so you should define the flag `--remote-read-src-addr=http://127.0.0.1:9009/prometheus`.
+
+Mimir supports both remote read mode, so you can use `STREAMED_XOR_CHUNKS` mode and `SAMPLES` mode.
+When you run Mimir, it exposes a port to serve HTTP on `8080 by default`.
+
+Next example of the local installation was in multi-tenant mode (3 instances of mimir) with nginx as load balancer.
+Load balancer expose single port `:9090`.
+
+As you can see in the example we call `:9009` instead of `:8080` because of proxy.
+
+The importing process example for the local installation of Mimir
+and single-node VictoriaMetrics(`http://localhost:8428`):
+
+```
+./vmctl remote-read 
+--remote-read-src-addr=http://127.0.0.1:9009/prometheus \
+--remote-read-filter-time-start=2021-10-18T00:00:00Z \
+--remote-read-step-interval=hour \
+--remote-read-src-check-alive=false \
+--remote-read-headers=X-Scope-OrgID:demo \
+--remote-read-use-stream=true \
+--vm-addr=http://127.0.0.1:8428 \
+--vm-concurrency=6
+```
+
+And when the process finishes, you will see the following:
+
+```
+Split defined times into 8847 ranges to import. Continue? [Y/n]
+VM worker 0:→ 12176 samples/s
+VM worker 1:→ 11918 samples/s
+VM worker 2:→ 11261 samples/s
+VM worker 3:→ 12861 samples/s
+VM worker 4:→ 11096 samples/s
+VM worker 5:→ 11575 samples/s
+Processing ranges: 8847 / 8847 [█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████] 100.00%
+2022/10/21 17:22:23 Import finished!
+2022/10/21 17:22:23 VictoriaMetrics importer stats:
+  idle duration: 0s;
+  time spent while importing: 15.379614356s;
+  total samples: 81243;
+  samples/s: 5282.51;
+  total bytes: 6.1 MB;
+  bytes/s: 397.8 kB;
+  import requests: 6;
+  import requests retries: 0;
+2022/10/21 17:22:23 Total time: 16.287405248s
+```
+
+It is important to know that if you run your Mimir installation in multi-tenant mode, remote read protocol
+requires an Authentication header like `X-Scope-OrgID`. You can define it via the flag `--remote-read-headers=X-Scope-OrgID:demo`
+
 ## Migrating data from VictoriaMetrics
 
 ### Native protocol
@@ -590,6 +833,80 @@ Total: 16 B ↗ Speed: 186.32 KiB p/s
 2022/08/30 19:48:24 Total time: 12.680582ms
 ```
 
+#### Cluster-to-cluster migration mode
+
+Using cluster-to-cluster migration mode helps to migrate all tenants data in a single `vmctl` run.
+
+Cluster-to-cluster uses `/admin/tenants` endpoint (available starting from [v1.84.0](https://docs.victoriametrics.com/CHANGELOG.html#v1840)) to discover list of tenants from source cluster.
+
+To use this mode you need to set `--vm-intercluster` flag to `true`, `--vm-native-src-addr` flag to 'http://vmselect:8481/' and `--vm-native-dst-addr` value to http://vminsert:8480/:
+
+```console
+./bin/vmctl vm-native --vm-intercluster=true --vm-native-src-addr=http://localhost:8481/ --vm-native-dst-addr=http://172.17.0.3:8480/
+VictoriaMetrics Native import mode
+2022/12/05 21:20:06 Discovered tenants: [123:1 12812919:1 1289198:1 1289:1283 12:1 1:0 1:1 1:1231231 1:1271727 1:12819 1:281 812891298:1]
+2022/12/05 21:20:06 Initing export pipe from "http://localhost:8481/select/123:1/prometheus/api/v1/export/native" with filters: 
+        filter: match[]={__name__!=""}
+Initing import process to "http://172.17.0.3:8480/insert/123:1/prometheus/api/v1/import/native":
+Total: 61.13 MiB ↖ Speed: 2.05 MiB p/s 
+Total: 61.13 MiB ↗ Speed: 2.30 MiB p/s 
+2022/12/05 21:20:33 Initing export pipe from "http://localhost:8481/select/12812919:1/prometheus/api/v1/export/native" with filters: 
+        filter: match[]={__name__!=""}
+Initing import process to "http://172.17.0.3:8480/insert/12812919:1/prometheus/api/v1/import/native":
+Total: 43.14 MiB ↘ Speed: 1.86 MiB p/s  
+Total: 43.14 MiB ↙ Speed: 2.36 MiB p/s 
+2022/12/05 21:20:51 Initing export pipe from "http://localhost:8481/select/1289198:1/prometheus/api/v1/export/native" with filters: 
+        filter: match[]={__name__!=""}
+Initing import process to "http://172.17.0.3:8480/insert/1289198:1/prometheus/api/v1/import/native":
+Total: 16.64 MiB ↗ Speed: 2.66 MiB p/s 
+Total: 16.64 MiB ↘ Speed: 2.19 MiB p/s 
+2022/12/05 21:20:59 Initing export pipe from "http://localhost:8481/select/1289:1283/prometheus/api/v1/export/native" with filters: 
+        filter: match[]={__name__!=""}
+Initing import process to "http://172.17.0.3:8480/insert/1289:1283/prometheus/api/v1/import/native":
+Total: 43.33 MiB ↙ Speed: 1.94 MiB p/s 
+Total: 43.33 MiB ↖ Speed: 2.35 MiB p/s 
+2022/12/05 21:21:18 Initing export pipe from "http://localhost:8481/select/12:1/prometheus/api/v1/export/native" with filters: 
+        filter: match[]={__name__!=""}
+Initing import process to "http://172.17.0.3:8480/insert/12:1/prometheus/api/v1/import/native":
+Total: 63.78 MiB ↙ Speed: 1.96 MiB p/s 
+Total: 63.78 MiB ↖ Speed: 2.28 MiB p/s 
+2022/12/05 21:21:46 Initing export pipe from "http://localhost:8481/select/1:0/prometheus/api/v1/export/native" with filters: 
+        filter: match[]={__name__!=""}
+Initing import process to "http://172.17.0.3:8480/insert/1:0/prometheus/api/v1/import/native":
+2022/12/05 21:21:46 Import finished!
+Total: 330 B ↗ Speed: 3.53 MiB p/s 
+2022/12/05 21:21:46 Initing export pipe from "http://localhost:8481/select/1:1/prometheus/api/v1/export/native" with filters: 
+        filter: match[]={__name__!=""}
+Initing import process to "http://172.17.0.3:8480/insert/1:1/prometheus/api/v1/import/native":
+Total: 63.81 MiB ↙ Speed: 1.96 MiB p/s 
+Total: 63.81 MiB ↖ Speed: 2.28 MiB p/s 
+2022/12/05 21:22:14 Initing export pipe from "http://localhost:8481/select/1:1231231/prometheus/api/v1/export/native" with filters: 
+        filter: match[]={__name__!=""}
+Initing import process to "http://172.17.0.3:8480/insert/1:1231231/prometheus/api/v1/import/native":
+Total: 63.84 MiB ↙ Speed: 1.93 MiB p/s 
+Total: 63.84 MiB ↖ Speed: 2.29 MiB p/s 
+2022/12/05 21:22:42 Initing export pipe from "http://localhost:8481/select/1:1271727/prometheus/api/v1/export/native" with filters: 
+        filter: match[]={__name__!=""}
+Initing import process to "http://172.17.0.3:8480/insert/1:1271727/prometheus/api/v1/import/native":
+Total: 54.37 MiB ↘ Speed: 1.90 MiB p/s 
+Total: 54.37 MiB ↙ Speed: 2.37 MiB p/s 
+2022/12/05 21:23:05 Initing export pipe from "http://localhost:8481/select/1:12819/prometheus/api/v1/export/native" with filters: 
+        filter: match[]={__name__!=""}
+Initing import process to "http://172.17.0.3:8480/insert/1:12819/prometheus/api/v1/import/native":
+Total: 17.01 MiB ↙ Speed: 1.75 MiB p/s 
+Total: 17.01 MiB ↖ Speed: 2.15 MiB p/s 
+2022/12/05 21:23:13 Initing export pipe from "http://localhost:8481/select/1:281/prometheus/api/v1/export/native" with filters: 
+        filter: match[]={__name__!=""}
+Initing import process to "http://172.17.0.3:8480/insert/1:281/prometheus/api/v1/import/native":
+Total: 63.89 MiB ↘ Speed: 1.90 MiB p/s 
+Total: 63.89 MiB ↙ Speed: 2.29 MiB p/s 
+2022/12/05 21:23:42 Initing export pipe from "http://localhost:8481/select/812891298:1/prometheus/api/v1/export/native" with filters: 
+        filter: match[]={__name__!=""}
+Initing import process to "http://172.17.0.3:8480/insert/812891298:1/prometheus/api/v1/import/native":
+Total: 63.84 MiB ↖ Speed: 1.99 MiB p/s 
+Total: 63.84 MiB ↗ Speed: 2.26 MiB p/s 
+2022/12/05 21:24:10 Total time: 4m4.1466565s
+```
 
 ## Verifying exported blocks from VictoriaMetrics
 
@@ -700,7 +1017,7 @@ It is recommended using [binary releases](https://github.com/VictoriaMetrics/Vic
 
 ### Development build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.3.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.
 2. Run `make vmctl` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `vmctl` binary and puts it into the `bin` folder.
 
@@ -729,7 +1046,7 @@ ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://b
 
 #### Development ARM build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.3.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.
 2. Run `make vmctl-linux-arm` or `make vmctl-linux-arm64` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `vmctl-linux-arm` or `vmctl-linux-arm64` binary respectively and puts it into the `bin` folder.
 

app/vmctl/flags.go

@@ -2,6 +2,7 @@ package main
 
 import (
 	"fmt"
+	"time"
 
 	"github.com/urfave/cli/v2"
 
@@ -43,6 +44,8 @@ const (
 	// also used in vm-native
 	vmExtraLabel = "vm-extra-label"
 	vmRateLimit  = "vm-rate-limit"
+
+	vmInterCluster = "vm-intercluster"
 )
 
 var (
@@ -397,6 +400,99 @@ var (
 			Usage: "Optional data transfer rate limit in bytes per second.\n" +
 				"By default the rate limit is disabled. It can be useful for limiting load on source or destination databases.",
 		},
+		&cli.BoolFlag{
+			Name: vmInterCluster,
+			Usage: "Enables cluster-to-cluster migration mode with automatic tenants data migration.\n" +
+				fmt.Sprintf(" In this mode --%s flag format is: 'http://vmselect:8481/'. --%s flag format is: http://vminsert:8480/. \n", vmNativeSrcAddr, vmNativeDstAddr) +
+				" TenantID will be appended automatically after discovering tenants from src.",
+		},
+	}
+)
+
+const (
+	remoteRead                 = "remote-read"
+	remoteReadUseStream        = "remote-read-use-stream"
+	remoteReadConcurrency      = "remote-read-concurrency"
+	remoteReadFilterTimeStart  = "remote-read-filter-time-start"
+	remoteReadFilterTimeEnd    = "remote-read-filter-time-end"
+	remoteReadFilterLabel      = "remote-read-filter-label"
+	remoteReadFilterLabelValue = "remote-read-filter-label-value"
+	remoteReadStepInterval     = "remote-read-step-interval"
+	remoteReadSrcAddr          = "remote-read-src-addr"
+	remoteReadUser             = "remote-read-user"
+	remoteReadPassword         = "remote-read-password"
+	remoteReadHTTPTimeout      = "remote-read-http-timeout"
+	remoteReadHeaders          = "remote-read-headers"
+)
+
+var (
+	remoteReadFlags = []cli.Flag{
+		&cli.IntFlag{
+			Name:  remoteReadConcurrency,
+			Usage: "Number of concurrently running remote read readers",
+			Value: 1,
+		},
+		&cli.TimestampFlag{
+			Name:   remoteReadFilterTimeStart,
+			Usage:  "The time filter in RFC3339 format to select timeseries with timestamp equal or higher than provided value. E.g. '2020-01-01T20:07:00Z'",
+			Layout: time.RFC3339,
+		},
+		&cli.TimestampFlag{
+			Name:   remoteReadFilterTimeEnd,
+			Usage:  "The time filter in RFC3339 format to select timeseries with timestamp equal or lower than provided value. E.g. '2020-01-01T20:07:00Z'",
+			Layout: time.RFC3339,
+		},
+		&cli.StringFlag{
+			Name:  remoteReadFilterLabel,
+			Usage: "Prometheus label name to filter timeseries by. E.g. '__name__' will filter timeseries by name.",
+			Value: "__name__",
+		},
+		&cli.StringFlag{
+			Name:  remoteReadFilterLabelValue,
+			Usage: fmt.Sprintf("Prometheus regular expression to filter label from %q flag.", remoteReadFilterLabelValue),
+			Value: ".*",
+		},
+		&cli.BoolFlag{
+			Name:  remoteRead,
+			Usage: "Use Prometheus remote read protocol",
+			Value: false,
+		},
+		&cli.BoolFlag{
+			Name:  remoteReadUseStream,
+			Usage: "Defines whether to use SAMPLES or STREAMED_XOR_CHUNKS mode. By default is uses SAMPLES mode. See https://prometheus.io/docs/prometheus/latest/querying/remote_read_api/#streamed-chunks",
+			Value: false,
+		},
+		&cli.StringFlag{
+			Name:     remoteReadStepInterval,
+			Usage:    fmt.Sprintf("Split export data into chunks. Requires setting --%s. Valid values are %q,%q,%q,%q.", remoteReadFilterTimeStart, stepper.StepMonth, stepper.StepDay, stepper.StepHour, stepper.StepMinute),
+			Required: true,
+		},
+		&cli.StringFlag{
+			Name:     remoteReadSrcAddr,
+			Usage:    "Remote read address to perform read from.",
+			Required: true,
+		},
+		&cli.StringFlag{
+			Name:    remoteReadUser,
+			Usage:   "Remote read username for basic auth",
+			EnvVars: []string{"REMOTE_READ_USERNAME"},
+		},
+		&cli.StringFlag{
+			Name:    remoteReadPassword,
+			Usage:   "Remote read password for basic auth",
+			EnvVars: []string{"REMOTE_READ_PASSWORD"},
+		},
+		&cli.DurationFlag{
+			Name:  remoteReadHTTPTimeout,
+			Usage: "Timeout defines timeout for HTTP write request to remote storage",
+		},
+		&cli.StringFlag{
+			Name:  remoteReadHeaders,
+			Value: "",
+			Usage: "Optional HTTP headers to send with each request to the corresponding remote source storage \n" +
+				"For example, --remote-read-headers='My-Auth:foobar' would send 'My-Auth: foobar' HTTP header with every request to the corresponding remote source storage. \n" +
+				"Multiple headers must be delimited by '^^': --remote-read-headers='header1:value1^^header2:value2'",
+		},
 	}
 )
 

app/vmctl/main.go

@@ -11,6 +11,7 @@ import (
 	"syscall"
 	"time"
 
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/remoteread"
 	"github.com/urfave/cli/v2"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/influx"
@@ -40,7 +41,7 @@ func main() {
 		Commands: []*cli.Command{
 			{
 				Name:  "opentsdb",
-				Usage: "Migrate timeseries from OpenTSDB",
+				Usage: "Migrate time series from OpenTSDB",
 				Flags: mergeFlags(globalFlags, otsdbFlags, vmFlags),
 				Action: func(c *cli.Context) error {
 					fmt.Println("OpenTSDB import mode")
@@ -75,7 +76,7 @@ func main() {
 			},
 			{
 				Name:  "influx",
-				Usage: "Migrate timeseries from InfluxDB",
+				Usage: "Migrate time series from InfluxDB",
 				Flags: mergeFlags(globalFlags, influxFlags, vmFlags),
 				Action: func(c *cli.Context) error {
 					fmt.Println("InfluxDB import mode")
@@ -114,9 +115,48 @@ func main() {
 					return processor.run(c.Bool(globalSilent), c.Bool(globalVerbose))
 				},
 			},
+			{
+				Name:  "remote-read",
+				Usage: "Migrate time series via Prometheus remote-read protocol",
+				Flags: mergeFlags(globalFlags, remoteReadFlags, vmFlags),
+				Action: func(c *cli.Context) error {
+					rr, err := remoteread.NewClient(remoteread.Config{
+						Addr:       c.String(remoteReadSrcAddr),
+						Username:   c.String(remoteReadUser),
+						Password:   c.String(remoteReadPassword),
+						Timeout:    c.Duration(remoteReadHTTPTimeout),
+						UseStream:  c.Bool(remoteReadUseStream),
+						Headers:    c.String(remoteReadHeaders),
+						LabelName:  c.String(remoteReadFilterLabel),
+						LabelValue: c.String(remoteReadFilterLabelValue),
+					})
+					if err != nil {
+						return fmt.Errorf("error create remote read client: %s", err)
+					}
+
+					vmCfg := initConfigVM(c)
+
+					importer, err := vm.NewImporter(vmCfg)
+					if err != nil {
+						return fmt.Errorf("failed to create VM importer: %s", err)
+					}
+
+					rmp := remoteReadProcessor{
+						src: rr,
+						dst: importer,
+						filter: remoteReadFilter{
+							timeStart: c.Timestamp(remoteReadFilterTimeStart),
+							timeEnd:   c.Timestamp(remoteReadFilterTimeEnd),
+							chunk:     c.String(remoteReadStepInterval),
+						},
+						cc: c.Int(remoteReadConcurrency),
+					}
+					return rmp.run(ctx, c.Bool(globalSilent), c.Bool(globalVerbose))
+				},
+			},
 			{
 				Name:  "prometheus",
-				Usage: "Migrate timeseries from Prometheus",
+				Usage: "Migrate time series from Prometheus",
 				Flags: mergeFlags(globalFlags, promFlags, vmFlags),
 				Action: func(c *cli.Context) error {
 					fmt.Println("Prometheus import mode")
@@ -160,7 +200,8 @@ func main() {
 					}
 
 					p := vmNativeProcessor{
-						rateLimit: c.Int64(vmRateLimit),
+						rateLimit:    c.Int64(vmRateLimit),
+						interCluster: c.Bool(vmInterCluster),
 						filter: filter{
 							match:     c.String(vmNativeFilterMatch),
 							timeStart: c.String(vmNativeFilterTimeStart),

app/vmctl/prometheus.go

@@ -9,6 +9,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/prometheus"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/vm"
 	"github.com/prometheus/prometheus/tsdb"
+	"github.com/prometheus/prometheus/tsdb/chunkenc"
 )
 
 type prometheusProcessor struct {
@@ -123,7 +124,15 @@ func (pp *prometheusProcessor) do(b tsdb.BlockReader) error {
 		var timestamps []int64
 		var values []float64
 		it := series.Iterator()
-		for it.Next() {
+		for {
+			typ := it.Next()
+			if typ == chunkenc.ValNone {
+				break
+			}
+			if typ != chunkenc.ValFloat {
+				// Skip unsupported values
+				continue
+			}
 			t, v := it.At()
 			timestamps = append(timestamps, t)
 			values = append(values, v)

app/vmctl/prometheus/prometheus.go

@@ -4,7 +4,7 @@ import (
 	"fmt"
 	"time"
 
-	"github.com/prometheus/prometheus/pkg/labels"
+	"github.com/prometheus/prometheus/model/labels"
 	"github.com/prometheus/prometheus/storage"
 	"github.com/prometheus/prometheus/tsdb"
 )

app/vmctl/remoteread.go

@@ -0,0 +1,127 @@
+package main
+
+import (
+	"context"
+	"fmt"
+	"log"
+	"sync"
+	"time"
+
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/barpool"
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/remoteread"
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/stepper"
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/vm"
+	"github.com/cheggaaa/pb/v3"
+)
+
+type remoteReadProcessor struct {
+	filter remoteReadFilter
+
+	dst *vm.Importer
+	src *remoteread.Client
+
+	cc int
+}
+
+type remoteReadFilter struct {
+	timeStart *time.Time
+	timeEnd   *time.Time
+	chunk     string
+}
+
+func (rrp *remoteReadProcessor) run(ctx context.Context, silent, verbose bool) error {
+	rrp.dst.ResetStats()
+	if rrp.filter.timeEnd == nil {
+		t := time.Now().In(rrp.filter.timeStart.Location())
+		rrp.filter.timeEnd = &t
+	}
+	if rrp.cc < 1 {
+		rrp.cc = 1
+	}
+
+	ranges, err := stepper.SplitDateRange(*rrp.filter.timeStart, *rrp.filter.timeEnd, rrp.filter.chunk)
+	if err != nil {
+		return fmt.Errorf("failed to create date ranges for the given time filters: %v", err)
+	}
+
+	question := fmt.Sprintf("Selected time range %q - %q will be split into %d ranges according to %q step. Continue?",
+		rrp.filter.timeStart.String(), rrp.filter.timeEnd.String(), len(ranges), rrp.filter.chunk)
+	if !silent && !prompt(question) {
+		return nil
+	}
+
+	var bar *pb.ProgressBar
+	if !silent {
+		bar = barpool.AddWithTemplate(fmt.Sprintf(barTpl, "Processing ranges"), len(ranges))
+		if err := barpool.Start(); err != nil {
+			return err
+		}
+	}
+	defer func() {
+		if !silent {
+			barpool.Stop()
+		}
+		log.Println("Import finished!")
+		log.Print(rrp.dst.Stats())
+	}()
+
+	rangeC := make(chan *remoteread.Filter)
+	errCh := make(chan error)
+
+	var wg sync.WaitGroup
+	wg.Add(rrp.cc)
+	for i := 0; i < rrp.cc; i++ {
+		go func() {
+			defer wg.Done()
+			for r := range rangeC {
+				if err := rrp.do(ctx, r); err != nil {
+					errCh <- fmt.Errorf("request failed for: %s", err)
+					return
+				}
+				if bar != nil {
+					bar.Increment()
+				}
+			}
+		}()
+	}
+
+	for _, r := range ranges {
+		select {
+		case infErr := <-errCh:
+			return fmt.Errorf("remote read error: %s", infErr)
+		case vmErr := <-rrp.dst.Errors():
+			return fmt.Errorf("import process failed: %s", wrapErr(vmErr, verbose))
+		case rangeC <- &remoteread.Filter{
+			StartTimestampMs: r[0].UnixMilli(),
+			EndTimestampMs:   r[1].UnixMilli(),
+		}:
+		}
+	}
+
+	close(rangeC)
+	wg.Wait()
+	rrp.dst.Close()
+	close(errCh)
+	// drain import errors channel
+	for vmErr := range rrp.dst.Errors() {
+		if vmErr.Err != nil {
+			return fmt.Errorf("import process failed: %s", wrapErr(vmErr, verbose))
+		}
+	}
+	for err := range errCh {
+		return fmt.Errorf("import process failed: %s", err)
+	}
+
+	return nil
+}
+
+func (rrp *remoteReadProcessor) do(ctx context.Context, filter *remoteread.Filter) error {
+	return rrp.src.Read(ctx, filter, func(series *vm.TimeSeries) error {
+		if err := rrp.dst.Input(series); err != nil {
+			return fmt.Errorf(
+				"failed to read data for time range start: %d, end: %d, %s",
+				filter.StartTimestampMs, filter.EndTimestampMs, err)
+		}
+		return nil
+	})
+}

app/vmctl/remoteread/remoteread.go

@@ -0,0 +1,349 @@
+package remoteread
+
+import (
+	"bytes"
+	"context"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"strings"
+	"time"
+
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/vm"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
+	"github.com/gogo/protobuf/proto"
+	"github.com/golang/snappy"
+	"github.com/prometheus/prometheus/prompb"
+	"github.com/prometheus/prometheus/storage/remote"
+	"github.com/prometheus/prometheus/tsdb/chunkenc"
+)
+
+const (
+	defaultReadTimeout = 30 * time.Second
+	remoteReadPath     = "/api/v1/read"
+	healthPath         = "/-/healthy"
+)
+
+// StreamCallback is a callback function for processing time series
+type StreamCallback func(series *vm.TimeSeries) error
+
+// Client is an HTTP client for reading
+// time series via remote read protocol.
+type Client struct {
+	addr      string
+	c         *http.Client
+	user      string
+	password  string
+	useStream bool
+	headers   []keyValue
+	matchers  []*prompb.LabelMatcher
+}
+
+// Config is config for remote read.
+type Config struct {
+	// Addr of remote storage
+	Addr string
+	// Timeout defines timeout for HTTP requests
+	// made by remote read client
+	Timeout time.Duration
+	// Username is the remote read username, optional.
+	Username string
+	// Password is the remote read password, optional.
+	Password string
+	// UseStream defines whether to use SAMPLES or STREAMED_XOR_CHUNKS mode
+	// see https://prometheus.io/docs/prometheus/latest/querying/remote_read_api/#samples
+	// https://prometheus.io/docs/prometheus/latest/querying/remote_read_api/#streamed-chunks
+	UseStream bool
+	// Headers optional HTTP headers to send with each request to the corresponding remote storage
+	Headers string
+	// LabelName, LabelValue stands for label=~value pair used for read requests.
+	// Is optional.
+	LabelName, LabelValue string
+}
+
+// Filter defines a list of filters applied to requested data
+type Filter struct {
+	StartTimestampMs int64
+	EndTimestampMs   int64
+}
+
+// NewClient returns client for
+// reading time series via remote read protocol.
+func NewClient(cfg Config) (*Client, error) {
+	if cfg.Addr == "" {
+		return nil, fmt.Errorf("config.Addr can't be empty")
+	}
+	if cfg.Timeout == 0 {
+		cfg.Timeout = defaultReadTimeout
+	}
+
+	var hdrs []string
+	if cfg.Headers != "" {
+		hdrs = strings.Split(cfg.Headers, "^^")
+	}
+
+	headers, err := parseHeaders(hdrs)
+	if err != nil {
+		return nil, err
+	}
+
+	var m *prompb.LabelMatcher
+	if cfg.LabelName != "" && cfg.LabelValue != "" {
+		m = &prompb.LabelMatcher{
+			Type:  prompb.LabelMatcher_RE,
+			Name:  cfg.LabelName,
+			Value: cfg.LabelValue,
+		}
+	}
+
+	c := &Client{
+		c: &http.Client{
+			Timeout:   cfg.Timeout,
+			Transport: http.DefaultTransport.(*http.Transport).Clone(),
+		},
+		addr:      strings.TrimSuffix(cfg.Addr, "/"),
+		user:      cfg.Username,
+		password:  cfg.Password,
+		useStream: cfg.UseStream,
+		headers:   headers,
+		matchers:  []*prompb.LabelMatcher{m},
+	}
+	return c, nil
+}
+
+// Read fetch data from remote read source
+func (c *Client) Read(ctx context.Context, filter *Filter, streamCb StreamCallback) error {
+	req := &prompb.ReadRequest{
+		Queries: []*prompb.Query{
+			{
+				StartTimestampMs: filter.StartTimestampMs,
+				EndTimestampMs:   filter.EndTimestampMs - 1,
+				Matchers:         c.matchers,
+			},
+		},
+	}
+	if c.useStream {
+		req.AcceptedResponseTypes = []prompb.ReadRequest_ResponseType{prompb.ReadRequest_STREAMED_XOR_CHUNKS}
+	}
+	data, err := proto.Marshal(req)
+	if err != nil {
+		return fmt.Errorf("unable to marshal read request: %w", err)
+	}
+
+	b := snappy.Encode(nil, data)
+	if err := c.fetch(ctx, b, streamCb); err != nil {
+		if errors.Is(err, context.Canceled) {
+			return fmt.Errorf("fetch request has ben cancelled")
+		}
+		return fmt.Errorf("error while fetching data from remote storage: %s", err)
+	}
+	return nil
+}
+
+func (c *Client) do(req *http.Request) (*http.Response, error) {
+	if c.user != "" {
+		req.SetBasicAuth(c.user, c.password)
+	}
+	for _, h := range c.headers {
+		req.Header.Add(h.key, h.value)
+	}
+	return c.c.Do(req)
+}
+
+// Ping checks the health of the read source
+func (c *Client) Ping() error {
+	url := c.addr + healthPath
+	req, err := http.NewRequest("GET", url, nil)
+	if err != nil {
+		return fmt.Errorf("cannot create request to %q: %s", url, err)
+	}
+	resp, err := c.do(req)
+	if err != nil {
+		return err
+	}
+	if resp.StatusCode != http.StatusOK {
+		return fmt.Errorf("bad status code: %d", resp.StatusCode)
+	}
+	return nil
+}
+
+func (c *Client) fetch(ctx context.Context, data []byte, streamCb StreamCallback) error {
+	r := bytes.NewReader(data)
+	url := c.addr + remoteReadPath
+	req, err := http.NewRequest("POST", url, r)
+	if err != nil {
+		return fmt.Errorf("failed to create new HTTP request: %w", err)
+	}
+
+	req.Header.Add("Content-Encoding", "snappy")
+	req.Header.Add("Accept-Encoding", "snappy")
+	req.Header.Set("Content-Type", "application/x-protobuf")
+	if c.useStream {
+		req.Header.Set("Content-Type", "application/x-streamed-protobuf; proto=prometheus.ChunkedReadResponse")
+	}
+	req.Header.Set("X-Prometheus-Remote-Read-Version", "0.1.0")
+
+	resp, err := c.do(req.WithContext(ctx))
+	if err != nil {
+		return fmt.Errorf("error while sending request to %s: %w; Data len %d(%d)",
+			req.URL.Redacted(), err, len(data), r.Size())
+	}
+	defer func() { _ = resp.Body.Close() }()
+
+	if resp.StatusCode != http.StatusNoContent && resp.StatusCode != http.StatusOK {
+		body, _ := io.ReadAll(resp.Body)
+		return fmt.Errorf("unexpected response code %d for %s. Response body %q",
+			resp.StatusCode, req.URL.Redacted(), body)
+	}
+
+	if c.useStream {
+		return processStreamResponse(resp.Body, streamCb)
+	}
+
+	return processResponse(resp.Body, streamCb)
+}
+
+func processResponse(body io.ReadCloser, callback StreamCallback) error {
+	d, err := io.ReadAll(body)
+	if err != nil {
+		return fmt.Errorf("error reading response: %w", err)
+	}
+	uncompressed, err := snappy.Decode(nil, d)
+	if err != nil {
+		return fmt.Errorf("error decoding response: %w", err)
+	}
+	var readResp prompb.ReadResponse
+	err = proto.Unmarshal(uncompressed, &readResp)
+	if err != nil {
+		return fmt.Errorf("unable to unmarshal response body: %w", err)
+	}
+	// response could have no results for the given filter, but that
+	// shouldn't be accounted as an error.
+	for _, res := range readResp.Results {
+		for _, ts := range res.Timeseries {
+			vmTs := convertSamples(ts.Samples, ts.Labels)
+			if err := callback(vmTs); err != nil {
+				return err
+			}
+		}
+	}
+
+	return nil
+}
+
+var bbPool bytesutil.ByteBufferPool
+
+func processStreamResponse(body io.ReadCloser, callback StreamCallback) error {
+	bb := bbPool.Get()
+	defer func() { bbPool.Put(bb) }()
+
+	stream := remote.NewChunkedReader(body, remote.DefaultChunkedReadLimit, bb.B)
+	for {
+		res := &prompb.ChunkedReadResponse{}
+		err := stream.NextProto(res)
+		if err == io.EOF {
+			break
+		}
+		if err != nil {
+			return err
+		}
+
+		for _, series := range res.ChunkedSeries {
+			samples := make([]prompb.Sample, 0)
+			for _, chunk := range series.Chunks {
+				s, err := parseSamples(chunk.Data)
+				if err != nil {
+					return err
+				}
+				samples = append(samples, s...)
+			}
+
+			ts := convertSamples(samples, series.Labels)
+			if err := callback(ts); err != nil {
+				return err
+			}
+		}
+	}
+	return nil
+}
+
+func parseSamples(chunk []byte) ([]prompb.Sample, error) {
+	c, err := chunkenc.FromData(chunkenc.EncXOR, chunk)
+	if err != nil {
+		return nil, fmt.Errorf("error read chunk: %w", err)
+	}
+
+	var samples []prompb.Sample
+	it := c.Iterator(nil)
+	for {
+		typ := it.Next()
+		if typ == chunkenc.ValNone {
+			break
+		}
+		if typ != chunkenc.ValFloat {
+			// Skip unsupported values
+			continue
+		}
+		if it.Err() != nil {
+			return nil, fmt.Errorf("error iterate over chunks: %w", it.Err())
+		}
+
+		ts, v := it.At()
+		s := prompb.Sample{
+			Timestamp: ts,
+			Value:     v,
+		}
+		samples = append(samples, s)
+	}
+
+	return samples, it.Err()
+}
+
+type keyValue struct {
+	key   string
+	value string
+}
+
+func parseHeaders(headers []string) ([]keyValue, error) {
+	if len(headers) == 0 {
+		return nil, nil
+	}
+	kvs := make([]keyValue, len(headers))
+	for i, h := range headers {
+		n := strings.IndexByte(h, ':')
+		if n < 0 {
+			return nil, fmt.Errorf(`missing ':' in header %q; expecting "key: value" format`, h)
+		}
+		kv := &kvs[i]
+		kv.key = strings.TrimSpace(h[:n])
+		kv.value = strings.TrimSpace(h[n+1:])
+	}
+	return kvs, nil
+}
+
+func convertSamples(samples []prompb.Sample, labels []prompb.Label) *vm.TimeSeries {
+	labelPairs := make([]vm.LabelPair, 0, len(labels))
+	nameValue := ""
+	for _, label := range labels {
+		if label.Name == "__name__" {
+			nameValue = label.Value
+			continue
+		}
+		labelPairs = append(labelPairs, vm.LabelPair{Name: label.Name, Value: label.Value})
+	}
+
+	n := len(samples)
+	values := make([]float64, 0, n)
+	timestamps := make([]int64, 0, n)
+	for _, sample := range samples {
+		values = append(values, sample.Value)
+		timestamps = append(timestamps, sample.Timestamp)
+	}
+	return &vm.TimeSeries{
+		Name:       nameValue,
+		LabelPairs: labelPairs,
+		Timestamps: timestamps,
+		Values:     values,
+	}
+}

app/vmctl/stepper/split.go

@@ -12,6 +12,8 @@ const (
 	StepDay string = "day"
 	// StepHour represents a one hour interval
 	StepHour string = "hour"
+	// StepMinute represents a one minute interval
+	StepMinute string = "minute"
 )
 
 // SplitDateRange splits start-end range in a subset of ranges respecting the given step
@@ -42,8 +44,13 @@ func SplitDateRange(start, end time.Time, step string) ([][]time.Time, error) {
 		nextStep = func(t time.Time) (time.Time, time.Time) {
 			return t, t.Add(time.Hour * 1)
 		}
+	case StepMinute:
+		nextStep = func(t time.Time) (time.Time, time.Time) {
+			return t, t.Add(time.Minute * 1)
+		}
 	default:
-		return nil, fmt.Errorf("failed to parse step value, valid values are: '%s', '%s', '%s'. provided: '%s'", StepMonth, StepDay, StepHour, step)
+		return nil, fmt.Errorf("failed to parse step value, valid values are: '%s', '%s', '%s', '%s'. provided: '%s'",
+			StepMonth, StepDay, StepHour, StepMinute, step)
 	}
 
 	currentStep := start

app/vmctl/vm/vm.go

@@ -182,6 +182,8 @@ func (im *Importer) Errors() chan *ImportError { return im.errors }
 // that need to be imported
 func (im *Importer) Input(ts *TimeSeries) error {
 	select {
+	case <-im.close:
+		return fmt.Errorf("importer is closed")
 	case im.input <- ts:
 		return nil
 	case err := <-im.errors:
@@ -197,6 +199,7 @@ func (im *Importer) Input(ts *TimeSeries) error {
 func (im *Importer) Close() {
 	im.once.Do(func() {
 		close(im.close)
+		close(im.input)
 		im.wg.Wait()
 		close(im.errors)
 	})
@@ -209,6 +212,10 @@ func (im *Importer) startWorker(bar *pb.ProgressBar, batchSize, significantFigur
 	for {
 		select {
 		case <-im.close:
+			for ts := range im.input {
+				ts = roundTimeseriesValue(ts, significantFigures, roundDigits)
+				batch = append(batch, ts)
+			}
 			exitErr := &ImportError{
 				Batch: batch,
 			}
@@ -217,24 +224,17 @@ func (im *Importer) startWorker(bar *pb.ProgressBar, batchSize, significantFigur
 			}
 			im.errors <- exitErr
 			return
-		case ts := <-im.input:
+		case ts, ok := <-im.input:
+			if !ok {
+				continue
+			}
 			// init waitForBatch when first
 			// value was received
 			if waitForBatch.IsZero() {
 				waitForBatch = time.Now()
 			}
 
-			if significantFigures > 0 {
-				for i, v := range ts.Values {
-					ts.Values[i] = decimal.RoundToSignificantFigures(v, significantFigures)
-				}
-			}
-			if roundDigits < 100 {
-				for i, v := range ts.Values {
-					ts.Values[i] = decimal.RoundToDecimalDigits(v, roundDigits)
-				}
-			}
-
+			ts = roundTimeseriesValue(ts, significantFigures, roundDigits)
 			batch = append(batch, ts)
 			dataPoints += len(ts.Values)
 
@@ -418,3 +418,18 @@ func byteCountSI(b int64) string {
 	return fmt.Sprintf("%.1f %cB",
 		float64(b)/float64(div), "kMGTPE"[exp])
 }
+
+func roundTimeseriesValue(ts *TimeSeries, significantFigures, roundDigits int) *TimeSeries {
+	if significantFigures > 0 {
+		for i, v := range ts.Values {
+			ts.Values[i] = decimal.RoundToSignificantFigures(v, significantFigures)
+		}
+	}
+	if roundDigits < 100 {
+		for i, v := range ts.Values {
+			ts.Values[i] = decimal.RoundToDecimalDigits(v, roundDigits)
+		}
+	}
+
+	return ts
+}

app/vmctl/vm_native.go

@@ -2,6 +2,7 @@ package main
 
 import (
 	"context"
+	"encoding/json"
 	"fmt"
 	"io"
 	"log"
@@ -19,8 +20,9 @@ type vmNativeProcessor struct {
 	filter    filter
 	rateLimit int64
 
-	dst *vmNativeClient
-	src *vmNativeClient
+	dst          *vmNativeClient
+	src          *vmNativeClient
+	interCluster bool
 }
 
 type vmNativeClient struct {
@@ -49,15 +51,16 @@ func (f filter) String() string {
 }
 
 const (
-	nativeExportAddr = "api/v1/export/native"
-	nativeImportAddr = "api/v1/import/native"
+	nativeExportAddr  = "api/v1/export/native"
+	nativeImportAddr  = "api/v1/import/native"
+	nativeTenantsAddr = "admin/tenants"
 
 	nativeBarTpl = `Total: {{counters . }} {{ cycle . "↖" "↗" "↘" "↙" }} Speed: {{speed . }} {{string . "suffix"}}`
 )
 
 func (p *vmNativeProcessor) run(ctx context.Context) error {
 	if p.filter.chunk == "" {
-		return p.runSingle(ctx, p.filter)
+		return p.runWithFilter(ctx, p.filter)
 	}
 
 	startOfRange, err := time.Parse(time.RFC3339, p.filter.timeStart)
@@ -89,7 +92,7 @@ func (p *vmNativeProcessor) run(ctx context.Context) error {
 			timeStart: formattedStartTime,
 			timeEnd:   formattedEndTime,
 		}
-		err := p.runSingle(ctx, f)
+		err := p.runWithFilter(ctx, f)
 
 		if err != nil {
 			log.Printf("processing failed for range %d/%d: %s - %s \n", rangeIdx+1, len(ranges), formattedStartTime, formattedEndTime)
@@ -99,25 +102,52 @@ func (p *vmNativeProcessor) run(ctx context.Context) error {
 	return nil
 }
 
-func (p *vmNativeProcessor) runSingle(ctx context.Context, f filter) error {
-	pr, pw := io.Pipe()
+func (p *vmNativeProcessor) runWithFilter(ctx context.Context, f filter) error {
+	nativeImportAddr, err := vm.AddExtraLabelsToImportPath(nativeImportAddr, p.dst.extraLabels)
 
-	log.Printf("Initing export pipe from %q with filters: %s\n", p.src.addr, f)
-	exportReader, err := p.exportPipe(ctx, f)
+	if err != nil {
+		return fmt.Errorf("failed to add labels to import path: %s", err)
+	}
+
+	if !p.interCluster {
+		srcURL := fmt.Sprintf("%s/%s", p.src.addr, nativeExportAddr)
+		dstURL := fmt.Sprintf("%s/%s", p.dst.addr, nativeImportAddr)
+
+		return p.runSingle(ctx, f, srcURL, dstURL)
+	}
+
+	tenants, err := p.getSourceTenants(ctx, f)
+	if err != nil {
+		return fmt.Errorf("failed to get source tenants: %s", err)
+	}
+
+	log.Printf("Discovered tenants: %v", tenants)
+	for _, tenant := range tenants {
+		// src and dst expected formats: http://vminsert:8480/ and http://vmselect:8481/
+		srcURL := fmt.Sprintf("%s/select/%s/prometheus/%s", p.src.addr, tenant, nativeExportAddr)
+		dstURL := fmt.Sprintf("%s/insert/%s/prometheus/%s", p.dst.addr, tenant, nativeImportAddr)
+
+		if err := p.runSingle(ctx, f, srcURL, dstURL); err != nil {
+			return fmt.Errorf("failed to migrate data for tenant %q: %s", tenant, err)
+		}
+	}
+
+	return nil
+}
+
+func (p *vmNativeProcessor) runSingle(ctx context.Context, f filter, srcURL, dstURL string) error {
+	log.Printf("Initing export pipe from %q with filters: %s\n", srcURL, f)
+
+	exportReader, err := p.exportPipe(ctx, srcURL, f)
 	if err != nil {
 		return fmt.Errorf("failed to init export pipe: %s", err)
 	}
 
-	nativeImportAddr, err := vm.AddExtraLabelsToImportPath(nativeImportAddr, p.dst.extraLabels)
-	if err != nil {
-		return err
-	}
-
+	pr, pw := io.Pipe()
 	sync := make(chan struct{})
 	go func() {
 		defer func() { close(sync) }()
-		u := fmt.Sprintf("%s/%s", p.dst.addr, nativeImportAddr)
-		req, err := http.NewRequestWithContext(ctx, "POST", u, pr)
+		req, err := http.NewRequestWithContext(ctx, "POST", dstURL, pr)
 		if err != nil {
 			log.Fatalf("cannot create import request to %q: %s", p.dst.addr, err)
 		}
@@ -130,7 +160,7 @@ func (p *vmNativeProcessor) runSingle(ctx context.Context, f filter) error {
 		}
 	}()
 
-	fmt.Printf("Initing import process to %q:\n", p.dst.addr)
+	fmt.Printf("Initing import process to %q:\n", dstURL)
 	pool := pb.NewPool()
 	bar := pb.ProgressBarTemplate(nativeBarTpl).New(0)
 	pool.Add(bar)
@@ -166,9 +196,43 @@ func (p *vmNativeProcessor) runSingle(ctx context.Context, f filter) error {
 	return nil
 }
 
-func (p *vmNativeProcessor) exportPipe(ctx context.Context, f filter) (io.ReadCloser, error) {
-	u := fmt.Sprintf("%s/%s", p.src.addr, nativeExportAddr)
+func (p *vmNativeProcessor) getSourceTenants(ctx context.Context, f filter) ([]string, error) {
+	u := fmt.Sprintf("%s/%s", p.src.addr, nativeTenantsAddr)
 	req, err := http.NewRequestWithContext(ctx, "GET", u, nil)
+	if err != nil {
+		return nil, fmt.Errorf("cannot create request to %q: %s", u, err)
+	}
+
+	params := req.URL.Query()
+	if f.timeStart != "" {
+		params.Set("start", f.timeStart)
+	}
+	if f.timeEnd != "" {
+		params.Set("end", f.timeEnd)
+	}
+	req.URL.RawQuery = params.Encode()
+
+	resp, err := p.src.do(req, http.StatusOK)
+	if err != nil {
+		return nil, fmt.Errorf("tenants request failed: %s", err)
+	}
+
+	var r struct {
+		Tenants []string `json:"data"`
+	}
+	if err := json.NewDecoder(resp.Body).Decode(&r); err != nil {
+		return nil, fmt.Errorf("cannot decode tenants response: %s", err)
+	}
+
+	if err := resp.Body.Close(); err != nil {
+		return nil, fmt.Errorf("cannot close tenants response body: %s", err)
+	}
+
+	return r.Tenants, nil
+}
+
+func (p *vmNativeProcessor) exportPipe(ctx context.Context, url string, f filter) (io.ReadCloser, error) {
+	req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
 	if err != nil {
 		return nil, fmt.Errorf("cannot create request to %q: %s", p.src.addr, err)
 	}

app/vmgateway/README.md

@@ -171,6 +171,41 @@ curl 'http://localhost:8431/api/v1/labels' -H 'Authorization: Bearer eyJhbGciOiJ
 # check rate limit
 ```
 
+## JWT signature verification
+
+`vmgateway` supports JWT signature verification.
+
+Supported algorithms are `RS256`, `RS384`, `RS512`, `ES256`, `ES384`, `ES512`, `PS256`, `PS384`, `PS512`.
+Tokens with unsupported algorithms will be rejected.
+
+In order to enable JWT signature verification, you need to specify keys for signature verification.
+The following flags are used to specify keys:
+- `-auth.publicKeyFiles` - allows to pass file path to file with public key.
+- `-auth.publicKeys` - allows to pass public key directly.
+
+Note that both flags support passing multiple keys and also can be used together.
+
+Example usage:
+```console
+./bin/vmgateway -eula \
+  -enable.auth \
+  -write.url=http://localhost:8480 \
+  -read.url=http://localhost:8481 \
+  -auth.publicKeyFiles=public_key.pem \
+  -auth.publicKeyFiles=public_key2.pem \
+  -auth.publicKeys=`-----BEGIN PUBLIC KEY-----
+MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAu1SU1LfVLPHCozMxH2Mo
+4lgOEePzNm0tRgeLezV6ffAt0gunVTLw7onLRnrq0/IzW7yWR7QkrmBL7jTKEn5u
++qKhbwKfBstIs+bMY2Zkp18gnTxKLxoS2tFczGkPLPgizskuemMghRniWaoLcyeh
+kd3qqGElvW/VDL5AaWTg0nLVkjRo9z+40RQzuVaE8AkAFmxZzow3x+VJYKdjykkJ
+0iT9wCS0DRTXu269V264Vf/3jvredZiKRkgwlL9xNAwxXFg0x/XFw005UWVRIkdg
+cKWTjpBP2dPwVZ4WWC+9aGVd+Gyn1o0CLelf4rEjGoXbAAEgAqeGUxrcIlbjXfbc
+mwIDAQAB
+-----END PUBLIC KEY-----
+`
+```
+This command will result in 3 keys loaded: 2 keys from files and 1 from command line.
+
 ## Configuration
 
 The shortlist of configuration flags include the following:
@@ -178,6 +213,12 @@ The shortlist of configuration flags include the following:
 ```console
   -auth.httpHeader string
      HTTP header name to look for JWT authorization token (default "Authorization")
+  -auth.publicKeyFiles array
+     Path file with public key to verify JWT token signature
+     Supports an array of values separated by comma or specified via multiple flags.
+  -auth.publicKeys array
+     Public keys to verify JWT token signature
+     Supports an array of values separated by comma or specified via multiple flags.
   -clusterMode
      enable this for the cluster version
   -datasource.appendTypePrefix
@@ -229,7 +270,7 @@ The shortlist of configuration flags include the following:
   -datasource.tlsServerName string
      Optional TLS server name to use for connections to -datasource.url. By default, the server name from -datasource.url is used
   -datasource.url string
-     Datasource compatible with Prometheus HTTP API. It can be single node VictoriaMetrics or vmselect URL. Required parameter. E.g. http://127.0.0.1:8428 . See also '-datasource.disablePathAppend', '-datasource.showURL'.
+     Datasource compatible with Prometheus HTTP API. It can be single node VictoriaMetrics or vmselect URL. Required parameter. E.g. http://127.0.0.1:8428 . See also -remoteRead.disablePathAppend and -datasource.showURL
   -enable.auth
      enables auth with jwt token
   -enable.rateLimit
@@ -336,7 +377,7 @@ The shortlist of configuration flags include the following:
 ## Limitations
 
 * Access Control:
-  * `jwt` token must be validated by external system, currently `vmgateway` can't validate the signature.
+  * `jwt` token signature verification for `HMAC` algorithms is not supported.
 * RateLimiting:
   * limits applied based on queries to `datasource.url`
   * only cluster version can be rate-limited.

app/vminsert/datadog/request_handler.go

@@ -54,7 +54,12 @@ func insertRows(series []parser.Series, extraLabels []prompbmarshal.Label) error
 		rowsTotal += len(ss.Points)
 		ctx.Labels = ctx.Labels[:0]
 		ctx.AddLabel("", ss.Metric)
-		ctx.AddLabel("host", ss.Host)
+		if ss.Host != "" {
+			ctx.AddLabel("host", ss.Host)
+		}
+		if ss.Device != "" {
+			ctx.AddLabel("device", ss.Device)
+		}
 		for _, tag := range ss.Tags {
 			name, value := parser.SplitTag(tag)
 			if name == "host" {

app/vminsert/main.go

@@ -333,7 +333,8 @@ var (
 
 	promscrapeTargetsRequests          = metrics.NewCounter(`vm_http_requests_total{path="/targets"}`)
 	promscrapeServiceDiscoveryRequests = metrics.NewCounter(`vm_http_requests_total{path="/service-discovery"}`)
-	promscrapeAPIV1TargetsRequests     = metrics.NewCounter(`vm_http_requests_total{path="/api/v1/targets"}`)
+
+	promscrapeAPIV1TargetsRequests = metrics.NewCounter(`vm_http_requests_total{path="/api/v1/targets"}`)
 
 	promscrapeTargetResponseRequests = metrics.NewCounter(`vm_http_requests_total{path="/target_response"}`)
 	promscrapeTargetResponseErrors   = metrics.NewCounter(`vm_http_request_errors_total{path="/target_response"}`)

app/vminsert/relabel/relabel.go

@@ -19,8 +19,6 @@ var (
 	relabelConfig = flag.String("relabelConfig", "", "Optional path to a file with relabeling rules, which are applied to all the ingested metrics. "+
 		"The path can point either to local file or to http url. "+
 		"See https://docs.victoriametrics.com/#relabeling for details. The config is reloaded on SIGHUP signal")
-	relabelDebug = flag.Bool("relabelDebug", false, "Whether to log metrics before and after relabeling with -relabelConfig. If the -relabelDebug is enabled, "+
-		"then the metrics aren't sent to storage. This is useful for debugging the relabeling configs")
 
 	usePromCompatibleNaming = flag.Bool("usePromCompatibleNaming", false, "Whether to replace characters unsupported by Prometheus with underscores "+
 		"in the ingested metric names and label names. For example, foo.bar{a.b='c'} is transformed into foo_bar{a_b='c'} during data ingestion if this flag is set. "+
@@ -77,7 +75,7 @@ func loadRelabelConfig() (*promrelabel.ParsedConfigs, error) {
 	if len(*relabelConfig) == 0 {
 		return nil, nil
 	}
-	pcs, err := promrelabel.LoadRelabelConfigs(*relabelConfig, *relabelDebug)
+	pcs, err := promrelabel.LoadRelabelConfigs(*relabelConfig)
 	if err != nil {
 		return nil, fmt.Errorf("error when reading -relabelConfig=%q: %w", *relabelConfig, err)
 	}

app/vmrestore/README.md

@@ -186,7 +186,7 @@ It is recommended using [binary releases](https://github.com/VictoriaMetrics/Vic
 
 ### Development build
 
-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.3.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.19.
 2. Run `make vmrestore` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
    It builds `vmrestore` binary and puts it into the `bin` folder.
 

app/vmselect/main.go

@@ -21,6 +21,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fs"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/querytracer"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/timerpool"
 	"github.com/VictoriaMetrics/metrics"
@@ -215,8 +216,8 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 	}
 
 	if path == "/vmalert" {
-		// vmalert access via incomplete url without `/` in the end. Redirecto to complete url.
-		// Use relative redirect, since, since the hostname and path prefix may be incorrect if VictoriaMetrics
+		// vmalert access via incomplete url without `/` in the end. Redirect to complete url.
+		// Use relative redirect, since the hostname and path prefix may be incorrect if VictoriaMetrics
 		// is hidden behind vmauth or similar proxy.
 		httpserver.Redirect(w, "vmalert/")
 		return true
@@ -423,6 +424,14 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 			return true
 		}
 		return true
+	case "/metric-relabel-debug":
+		promscrapeMetricRelabelDebugRequests.Inc()
+		promscrape.WriteMetricRelabelDebug(w, r)
+		return true
+	case "/target-relabel-debug":
+		promscrapeTargetRelabelDebugRequests.Inc()
+		promscrape.WriteTargetRelabelDebug(w, r)
+		return true
 	case "/api/v1/rules", "/rules":
 		rulesRequests.Inc()
 		if len(*vmalertProxyURL) > 0 {
@@ -587,6 +596,9 @@ var (
 	graphiteTagsDelSeriesRequests = metrics.NewCounter(`vm_http_requests_total{path="/tags/delSeries"}`)
 	graphiteTagsDelSeriesErrors   = metrics.NewCounter(`vm_http_request_errors_total{path="/tags/delSeries"}`)
 
+	promscrapeMetricRelabelDebugRequests = metrics.NewCounter(`vm_http_requests_total{path="/metric-relabel-debug"}`)
+	promscrapeTargetRelabelDebugRequests = metrics.NewCounter(`vm_http_requests_total{path="/target-relabel-debug"}`)
+
 	graphiteFunctionsRequests = metrics.NewCounter(`vm_http_requests_total{path="/functions"}`)
 
 	vmalertRequests = metrics.NewCounter(`vm_http_requests_total{path="/vmalert"}`)

app/vmselect/netstorage/netstorage_test.go

@@ -144,7 +144,7 @@ func TestMergeSortBlocks(t *testing.T) {
 		},
 	}, 1, &Result{
 		Timestamps: []int64{1, 2, 4, 5, 10, 11, 12},
-		Values:     []float64{21, 22, 23, 7, 24, 5, 26},
+		Values:     []float64{21, 22, 23, 7, 24, 25, 26},
 	})
 
 	// Multiple blocks with identical timestamp ranges, no deduplication.

app/vmselect/promql/aggr.go

@@ -748,7 +748,7 @@ func getIntK(k float64, kMax int) int {
 	if math.IsNaN(k) {
 		return 0
 	}
-	kn := int(k)
+	kn := floatToIntBounded(k)
 	if kn < 0 {
 		return 0
 	}
@@ -999,14 +999,10 @@ func aggrFuncLimitK(afa *aggrFuncArg) ([]*timeseries, error) {
 	if err := expectTransformArgsNum(args, 2); err != nil {
 		return nil, err
 	}
-	limits, err := getScalar(args[0], 0)
+	limit, err := getIntNumber(args[0], 0)
 	if err != nil {
 		return nil, fmt.Errorf("cannot obtain limit arg: %w", err)
 	}
-	limit := 0
-	if len(limits) > 0 {
-		limit = int(limits[0])
-	}
 	if limit < 0 {
 		limit = 0
 	}
@@ -1155,3 +1151,13 @@ func lessWithNaNs(a, b float64) bool {
 	}
 	return a < b
 }
+
+func floatToIntBounded(f float64) int {
+	if f > math.MaxInt {
+		return math.MaxInt
+	}
+	if f < math.MinInt {
+		return math.MinInt
+	}
+	return int(f)
+}

app/vmselect/promql/exec_test.go

@@ -5549,6 +5549,30 @@ func TestExecSuccess(t *testing.T) {
 		resultExpected := []netstorage.Result{r1, r2}
 		f(q, resultExpected)
 	})
+	t.Run(`limitk(inf)`, func(t *testing.T) {
+		t.Parallel()
+		q := `sort(limitk(inf, label_set(10, "foo", "bar") or label_set(time()/150, "baz", "sss")))`
+		r1 := netstorage.Result{
+			MetricName: metricNameExpected,
+			Values:     []float64{10, 10, 10, 10, 10, 10},
+			Timestamps: timestampsExpected,
+		}
+		r1.MetricName.Tags = []storage.Tag{{
+			Key:   []byte("foo"),
+			Value: []byte("bar"),
+		}}
+		r2 := netstorage.Result{
+			MetricName: metricNameExpected,
+			Values:     []float64{6.666666666666667, 8, 9.333333333333334, 10.666666666666666, 12, 13.333333333333334},
+			Timestamps: timestampsExpected,
+		}
+		r2.MetricName.Tags = []storage.Tag{{
+			Key:   []byte("baz"),
+			Value: []byte("sss"),
+		}}
+		resultExpected := []netstorage.Result{r1, r2}
+		f(q, resultExpected)
+	})
 	t.Run(`any()`, func(t *testing.T) {
 		t.Parallel()
 		q := `any(label_set(10, "__name__", "x", "foo", "bar") or label_set(time()/150, "__name__", "y", "baz", "sss"))`
@@ -6385,6 +6409,17 @@ func TestExecSuccess(t *testing.T) {
 		resultExpected := []netstorage.Result{r1, r2}
 		f(q, resultExpected)
 	})
+	t.Run(`range_trim_spikes()`, func(t *testing.T) {
+		t.Parallel()
+		q := `range_trim_spikes(0.2, time())`
+		r := netstorage.Result{
+			MetricName: metricNameExpected,
+			Values:     []float64{nan, 1200, 1400, 1600, 1800, nan},
+			Timestamps: timestampsExpected,
+		}
+		resultExpected := []netstorage.Result{r}
+		f(q, resultExpected)
+	})
 	t.Run(`range_quantile(0.5)`, func(t *testing.T) {
 		t.Parallel()
 		q := `range_quantile(0.5, time())`
@@ -8189,6 +8224,7 @@ func TestExecError(t *testing.T) {
 	f(`step(1)`)
 	f(`running_sum(1, 2)`)
 	f(`range_sum(1, 2)`)
+	f(`range_trim_spikes()`)
 	f(`range_first(1,  2)`)
 	f(`range_last(1, 2)`)
 	f(`range_linear_regression(1, 2)`)

app/vmselect/promql/rollup.go

@@ -2127,7 +2127,7 @@ func getIntNumber(arg interface{}, argNum int) (int, error) {
 	}
 	n := 0
 	if len(v) > 0 {
-		n = int(v[0])
+		n = floatToIntBounded(v[0])
 	}
 	return n, nil
 }

app/vmselect/promql/transform.go

@@ -96,6 +96,7 @@ var transformFuncs = map[string]transformFunc{
 	"range_stddev":               transformRangeStddev,
 	"range_stdvar":               transformRangeStdvar,
 	"range_sum":                  newTransformFuncRange(runningSum),
+	"range_trim_spikes":          transformRangeTrimSpikes,
 	"remove_resets":              transformRemoveResets,
 	"round":                      transformRound,
 	"running_avg":                newTransformFuncRunning(runningAvg),
@@ -370,14 +371,10 @@ func transformBucketsLimit(tfa *transformFuncArg) ([]*timeseries, error) {
 	if err := expectTransformArgsNum(args, 2); err != nil {
 		return nil, err
 	}
-	limits, err := getScalar(args[0], 1)
+	limit, err := getIntNumber(args[0], 0)
 	if err != nil {
 		return nil, err
 	}
-	limit := 0
-	if len(limits) > 0 {
-		limit = int(limits[0])
-	}
 	if limit <= 0 {
 		return nil, nil
 	}
@@ -389,6 +386,7 @@ func transformBucketsLimit(tfa *transformFuncArg) ([]*timeseries, error) {
 	if len(tss) == 0 {
 		return nil, nil
 	}
+	pointsCount := len(tss[0].Values)
 
 	// Group timeseries by all MetricGroup+tags excluding `le` tag.
 	type x struct {
@@ -436,7 +434,7 @@ func transformBucketsLimit(tfa *transformFuncArg) ([]*timeseries, error) {
 		sort.Slice(leGroup, func(i, j int) bool {
 			return leGroup[i].le < leGroup[j].le
 		})
-		for n := range limits {
+		for n := 0; n < pointsCount; n++ {
 			prevValue := float64(0)
 			for i := range leGroup {
 				xx := &leGroup[i]
@@ -1274,6 +1272,54 @@ func transformRangeNormalize(tfa *transformFuncArg) ([]*timeseries, error) {
 	return rvs, nil
 }
 
+func transformRangeTrimSpikes(tfa *transformFuncArg) ([]*timeseries, error) {
+	args := tfa.args
+	if err := expectTransformArgsNum(args, 2); err != nil {
+		return nil, err
+	}
+	phis, err := getScalar(args[0], 0)
+	if err != nil {
+		return nil, err
+	}
+	phi := float64(0)
+	if len(phis) > 0 {
+		phi = phis[0]
+	}
+	// Trim 100% * (phi / 2) samples with the lowest / highest values per each time series
+	phi /= 2
+	phiUpper := 1 - phi
+	phiLower := phi
+	rvs := args[1]
+	a := getFloat64s()
+	values := a.A[:0]
+	for _, ts := range rvs {
+		values := values[:0]
+		originValues := ts.Values
+		for _, v := range originValues {
+			if math.IsNaN(v) {
+				continue
+			}
+			values = append(values, v)
+		}
+		sort.Float64s(values)
+		vMax := quantileSorted(phiUpper, values)
+		vMin := quantileSorted(phiLower, values)
+		for i, v := range originValues {
+			if math.IsNaN(v) {
+				continue
+			}
+			if v > vMax {
+				originValues[i] = nan
+			} else if v < vMin {
+				originValues[i] = nan
+			}
+		}
+	}
+	a.A = values
+	putFloat64s(a)
+	return rvs, nil
+}
+
 func transformRangeLinearRegression(tfa *transformFuncArg) ([]*timeseries, error) {
 	args := tfa.args
 	if err := expectTransformArgsNum(args, 1); err != nil {

app/vmselect/searchutils/searchutils.go

@@ -209,7 +209,7 @@ func (d *Deadline) String() string {
 	startTime := time.Unix(int64(d.deadline), 0).Add(-d.timeout)
 	elapsed := time.Since(startTime)
 	msg := fmt.Sprintf("%.3f seconds (elapsed %.3f seconds)", d.timeout.Seconds(), elapsed.Seconds())
-	if d.flagHint != "" {
+	if float64(elapsed)/float64(d.timeout) > 0.9 && d.flagHint != "" {
 		msg += fmt.Sprintf("; the timeout can be adjusted with `%s` command-line flag", d.flagHint)
 	}
 	return msg

app/vmselect/vmui/asset-manifest.json

@@ -1,12 +1,12 @@
 {
   "files": {
-    "main.css": "./static/css/main.0937c83d.css",
-    "main.js": "./static/js/main.e18cda26.js",
+    "main.css": "./static/css/main.89abca0f.css",
+    "main.js": "./static/js/main.c552245f.js",
     "static/js/27.c1ccfd29.chunk.js": "./static/js/27.c1ccfd29.chunk.js",
     "index.html": "./index.html"
   },
   "entrypoints": [
-    "static/css/main.0937c83d.css",
-    "static/js/main.e18cda26.js"
+    "static/css/main.89abca0f.css",
+    "static/js/main.c552245f.js"
   ]
 }

app/vmselect/vmui/index.html

@@ -1 +1 @@
-<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="VM-UI is a metric explorer for Victoria Metrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><link rel="preconnect" href="https://fonts.googleapis.com"><link rel="preconnect" href="https://fonts.gstatic.com" crossorigin><link href="https://fonts.googleapis.com/css2?family=JetBrains+Mono&family=Lato:wght@300;400;700&display=swap" rel="stylesheet"><script src="./dashboards/index.js" type="module"></script><script defer="defer" src="./static/js/main.e18cda26.js"></script><link href="./static/css/main.0937c83d.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>
+<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="VM-UI is a metric explorer for Victoria Metrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><link rel="preconnect" href="https://fonts.googleapis.com"><link rel="preconnect" href="https://fonts.gstatic.com" crossorigin><link href="https://fonts.googleapis.com/css2?family=JetBrains+Mono&family=Lato:wght@300;400;700&display=swap" rel="stylesheet"><script src="./dashboards/index.js" type="module"></script><script defer="defer" src="./static/js/main.c552245f.js"></script><link href="./static/css/main.89abca0f.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>

app/vmstorage/main.go

@@ -100,7 +100,7 @@ func InitWithoutMetrics(resetCacheIfNeeded func(mrs []storage.MetricRow)) {
 	storage.SetLogNewSeries(*logNewSeries)
 	storage.SetFinalMergeDelay(*finalMergeDelay)
 	storage.SetBigMergeWorkersCount(*bigMergeConcurrency)
-	storage.SetSmallMergeWorkersCount(*smallMergeConcurrency)
+	storage.SetMergeWorkersCount(*smallMergeConcurrency)
 	storage.SetRetentionTimezoneOffset(*retentionTimezoneOffset)
 	storage.SetFreeDiskSpaceLimit(minFreeDiskSpaceBytes.N)
 	storage.SetTSIDCacheSize(cacheSizeStorageTSID.N)
@@ -453,56 +453,80 @@ func registerStorageMetrics(strg *storage.Storage) {
 		return 0
 	})
 
-	metrics.NewGauge(`vm_active_merges{type="storage/big"}`, func() float64 {
-		return float64(tm().ActiveBigMerges)
+	metrics.NewGauge(`vm_active_merges{type="storage/inmemory"}`, func() float64 {
+		return float64(tm().ActiveInmemoryMerges)
 	})
 	metrics.NewGauge(`vm_active_merges{type="storage/small"}`, func() float64 {
 		return float64(tm().ActiveSmallMerges)
 	})
-	metrics.NewGauge(`vm_active_merges{type="indexdb"}`, func() float64 {
-		return float64(idbm().ActiveMerges)
+	metrics.NewGauge(`vm_active_merges{type="storage/big"}`, func() float64 {
+		return float64(tm().ActiveBigMerges)
+	})
+	metrics.NewGauge(`vm_active_merges{type="indexdb/inmemory"}`, func() float64 {
+		return float64(idbm().ActiveInmemoryMerges)
+	})
+	metrics.NewGauge(`vm_active_merges{type="indexdb/file"}`, func() float64 {
+		return float64(idbm().ActiveFileMerges)
 	})
 
-	metrics.NewGauge(`vm_merges_total{type="storage/big"}`, func() float64 {
-		return float64(tm().BigMergesCount)
+	metrics.NewGauge(`vm_merges_total{type="storage/inmemory"}`, func() float64 {
+		return float64(tm().InmemoryMergesCount)
 	})
 	metrics.NewGauge(`vm_merges_total{type="storage/small"}`, func() float64 {
 		return float64(tm().SmallMergesCount)
 	})
-	metrics.NewGauge(`vm_merges_total{type="indexdb"}`, func() float64 {
-		return float64(idbm().MergesCount)
+	metrics.NewGauge(`vm_merges_total{type="storage/big"}`, func() float64 {
+		return float64(tm().BigMergesCount)
+	})
+	metrics.NewGauge(`vm_merges_total{type="indexdb/inmemory"}`, func() float64 {
+		return float64(idbm().InmemoryMergesCount)
+	})
+	metrics.NewGauge(`vm_merges_total{type="indexdb/file"}`, func() float64 {
+		return float64(idbm().FileMergesCount)
 	})
 
-	metrics.NewGauge(`vm_rows_merged_total{type="storage/big"}`, func() float64 {
-		return float64(tm().BigRowsMerged)
+	metrics.NewGauge(`vm_rows_merged_total{type="storage/inmemory"}`, func() float64 {
+		return float64(tm().InmemoryRowsMerged)
 	})
 	metrics.NewGauge(`vm_rows_merged_total{type="storage/small"}`, func() float64 {
 		return float64(tm().SmallRowsMerged)
 	})
-	metrics.NewGauge(`vm_rows_merged_total{type="indexdb"}`, func() float64 {
-		return float64(idbm().ItemsMerged)
+	metrics.NewGauge(`vm_rows_merged_total{type="storage/big"}`, func() float64 {
+		return float64(tm().BigRowsMerged)
+	})
+	metrics.NewGauge(`vm_rows_merged_total{type="indexdb/inmemory"}`, func() float64 {
+		return float64(idbm().InmemoryItemsMerged)
+	})
+	metrics.NewGauge(`vm_rows_merged_total{type="indexdb/file"}`, func() float64 {
+		return float64(idbm().FileItemsMerged)
 	})
 
-	metrics.NewGauge(`vm_rows_deleted_total{type="storage/big"}`, func() float64 {
-		return float64(tm().BigRowsDeleted)
+	metrics.NewGauge(`vm_rows_deleted_total{type="storage/inmemory"}`, func() float64 {
+		return float64(tm().InmemoryRowsDeleted)
 	})
 	metrics.NewGauge(`vm_rows_deleted_total{type="storage/small"}`, func() float64 {
 		return float64(tm().SmallRowsDeleted)
 	})
-
-	metrics.NewGauge(`vm_references{type="storage/big", name="parts"}`, func() float64 {
-		return float64(tm().BigPartsRefCount)
+	metrics.NewGauge(`vm_rows_deleted_total{type="storage/big"}`, func() float64 {
+		return float64(tm().BigRowsDeleted)
 	})
-	metrics.NewGauge(`vm_references{type="storage/small", name="parts"}`, func() float64 {
+
+	metrics.NewGauge(`vm_part_references{type="storage/inmemory"}`, func() float64 {
+		return float64(tm().InmemoryPartsRefCount)
+	})
+	metrics.NewGauge(`vm_part_references{type="storage/small"}`, func() float64 {
 		return float64(tm().SmallPartsRefCount)
 	})
-	metrics.NewGauge(`vm_references{type="storage", name="partitions"}`, func() float64 {
+	metrics.NewGauge(`vm_part_references{type="storage/big"}`, func() float64 {
+		return float64(tm().BigPartsRefCount)
+	})
+	metrics.NewGauge(`vm_partition_references{type="storage"}`, func() float64 {
 		return float64(tm().PartitionsRefCount)
 	})
-	metrics.NewGauge(`vm_references{type="indexdb", name="objects"}`, func() float64 {
+	metrics.NewGauge(`vm_object_references{type="indexdb"}`, func() float64 {
 		return float64(idbm().IndexDBRefCount)
 	})
-	metrics.NewGauge(`vm_references{type="indexdb", name="parts"}`, func() float64 {
+	metrics.NewGauge(`vm_part_references{type="indexdb"}`, func() float64 {
 		return float64(idbm().PartsRefCount)
 	})
 
@@ -531,11 +555,11 @@ func registerStorageMetrics(strg *storage.Storage) {
 		return float64(idbm().CompositeFilterMissingConversions)
 	})
 
-	metrics.NewGauge(`vm_assisted_merges_total{type="storage/small"}`, func() float64 {
-		return float64(tm().SmallAssistedMerges)
+	metrics.NewGauge(`vm_assisted_merges_total{type="storage/inmemory"}`, func() float64 {
+		return float64(tm().InmemoryAssistedMerges)
 	})
-	metrics.NewGauge(`vm_assisted_merges_total{type="indexdb"}`, func() float64 {
-		return float64(idbm().AssistedMerges)
+	metrics.NewGauge(`vm_assisted_merges_total{type="indexdb/inmemory"}`, func() float64 {
+		return float64(idbm().AssistedInmemoryMerges)
 	})
 
 	metrics.NewGauge(`vm_indexdb_items_added_total`, func() float64 {
@@ -546,11 +570,8 @@ func registerStorageMetrics(strg *storage.Storage) {
 	})
 
 	// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686
-	metrics.NewGauge(`vm_merge_need_free_disk_space{type="storage/small"}`, func() float64 {
-		return float64(tm().SmallMergeNeedFreeDiskSpace)
-	})
-	metrics.NewGauge(`vm_merge_need_free_disk_space{type="storage/big"}`, func() float64 {
-		return float64(tm().BigMergeNeedFreeDiskSpace)
+	metrics.NewGauge(`vm_merge_need_free_disk_space`, func() float64 {
+		return float64(tm().MergeNeedFreeDiskSpace)
 	})
 
 	metrics.NewGauge(`vm_pending_rows{type="storage"}`, func() float64 {
@@ -560,34 +581,52 @@ func registerStorageMetrics(strg *storage.Storage) {
 		return float64(idbm().PendingItems)
 	})
 
-	metrics.NewGauge(`vm_parts{type="storage/big"}`, func() float64 {
-		return float64(tm().BigPartsCount)
+	metrics.NewGauge(`vm_parts{type="storage/inmemory"}`, func() float64 {
+		return float64(tm().InmemoryPartsCount)
 	})
 	metrics.NewGauge(`vm_parts{type="storage/small"}`, func() float64 {
 		return float64(tm().SmallPartsCount)
 	})
-	metrics.NewGauge(`vm_parts{type="indexdb"}`, func() float64 {
-		return float64(idbm().PartsCount)
+	metrics.NewGauge(`vm_parts{type="storage/big"}`, func() float64 {
+		return float64(tm().BigPartsCount)
+	})
+	metrics.NewGauge(`vm_parts{type="indexdb/inmemory"}`, func() float64 {
+		return float64(idbm().InmemoryPartsCount)
+	})
+	metrics.NewGauge(`vm_parts{type="indexdb/file"}`, func() float64 {
+		return float64(idbm().FilePartsCount)
 	})
 
-	metrics.NewGauge(`vm_blocks{type="storage/big"}`, func() float64 {
-		return float64(tm().BigBlocksCount)
+	metrics.NewGauge(`vm_blocks{type="storage/inmemory"}`, func() float64 {
+		return float64(tm().InmemoryBlocksCount)
 	})
 	metrics.NewGauge(`vm_blocks{type="storage/small"}`, func() float64 {
 		return float64(tm().SmallBlocksCount)
 	})
-	metrics.NewGauge(`vm_blocks{type="indexdb"}`, func() float64 {
-		return float64(idbm().BlocksCount)
+	metrics.NewGauge(`vm_blocks{type="storage/big"}`, func() float64 {
+		return float64(tm().BigBlocksCount)
+	})
+	metrics.NewGauge(`vm_blocks{type="indexdb/inmemory"}`, func() float64 {
+		return float64(idbm().InmemoryBlocksCount)
+	})
+	metrics.NewGauge(`vm_blocks{type="indexdb/file"}`, func() float64 {
+		return float64(idbm().FileBlocksCount)
 	})
 
-	metrics.NewGauge(`vm_data_size_bytes{type="storage/big"}`, func() float64 {
-		return float64(tm().BigSizeBytes)
+	metrics.NewGauge(`vm_data_size_bytes{type="storage/inmemory"}`, func() float64 {
+		return float64(tm().InmemorySizeBytes)
 	})
 	metrics.NewGauge(`vm_data_size_bytes{type="storage/small"}`, func() float64 {
 		return float64(tm().SmallSizeBytes)
 	})
-	metrics.NewGauge(`vm_data_size_bytes{type="indexdb"}`, func() float64 {
-		return float64(idbm().SizeBytes)
+	metrics.NewGauge(`vm_data_size_bytes{type="storage/big"}`, func() float64 {
+		return float64(tm().BigSizeBytes)
+	})
+	metrics.NewGauge(`vm_data_size_bytes{type="indexdb/inmemory"}`, func() float64 {
+		return float64(idbm().InmemorySizeBytes)
+	})
+	metrics.NewGauge(`vm_data_size_bytes{type="indexdb/file"}`, func() float64 {
+		return float64(idbm().FileSizeBytes)
 	})
 
 	metrics.NewGauge(`vm_rows_added_to_storage_total`, func() float64 {
@@ -665,14 +704,20 @@ func registerStorageMetrics(strg *storage.Storage) {
 		return float64(m().TimestampsBytesSaved)
 	})
 
-	metrics.NewGauge(`vm_rows{type="storage/big"}`, func() float64 {
-		return float64(tm().BigRowsCount)
+	metrics.NewGauge(`vm_rows{type="storage/inmemory"}`, func() float64 {
+		return float64(tm().InmemoryRowsCount)
 	})
 	metrics.NewGauge(`vm_rows{type="storage/small"}`, func() float64 {
 		return float64(tm().SmallRowsCount)
 	})
-	metrics.NewGauge(`vm_rows{type="indexdb"}`, func() float64 {
-		return float64(idbm().ItemsCount)
+	metrics.NewGauge(`vm_rows{type="storage/big"}`, func() float64 {
+		return float64(tm().BigRowsCount)
+	})
+	metrics.NewGauge(`vm_rows{type="indexdb/inmemory"}`, func() float64 {
+		return float64(idbm().InmemoryItemsCount)
+	})
+	metrics.NewGauge(`vm_rows{type="indexdb/file"}`, func() float64 {
+		return float64(idbm().FileItemsCount)
 	})
 
 	metrics.NewGauge(`vm_date_range_search_calls_total`, func() float64 {

app/vmui/Dockerfile-web

@@ -1,4 +1,4 @@
-FROM golang:1.19.3 as build-web-stage
+FROM golang:1.19.4 as build-web-stage
 COPY build /build
 
 WORKDIR /build

app/vmui/packages/vmui/src/components/Chart/ChartTooltip/ChartTooltip.tsx

@@ -49,7 +49,7 @@ const ChartTooltip: FC<ChartTooltipProps> = ({
   const value = useMemo(() => get(u, ["data", seriesIdx, dataIdx], 0), [u, seriesIdx, dataIdx]);
   const valueFormat = useMemo(() => formatPrettyNumber(value), [value]);
   const dataTime = useMemo(() => u.data[0][dataIdx], [u, dataIdx]);
-  const date = useMemo(() => dayjs(new Date(dataTime * 1000)).format(DATE_FULL_TIMEZONE_FORMAT), [dataTime]);
+  const date = useMemo(() => dayjs(dataTime * 1000).tz().format(DATE_FULL_TIMEZONE_FORMAT), [dataTime]);
 
   const color = useMemo(() => getColorLine(series[seriesIdx]?.label || ""), [series, seriesIdx]);
 

app/vmui/packages/vmui/src/components/Chart/ChartTooltip/style.scss

@@ -76,5 +76,6 @@ $chart-tooltip-y: -1 * ($padding-small + $chart-tooltip-half-icon);
   &-info {
     display: grid;
     grid-gap: 4px;
+    word-break: break-all;
   }
 }

app/vmui/packages/vmui/src/components/Chart/LineChart/LineChart.tsx

@@ -11,7 +11,7 @@ import { defaultOptions } from "../../../utils/uplot/helpers";
 import { dragChart } from "../../../utils/uplot/events";
 import { getAxes, getMinMaxBuffer } from "../../../utils/uplot/axes";
 import { MetricResult } from "../../../api/types";
-import { limitsDurations } from "../../../utils/time";
+import { dateFromSeconds, formatDateForNativeInput, limitsDurations } from "../../../utils/time";
 import throttle from "lodash.throttle";
 import useResize from "../../../hooks/useResize";
 import { TimeParams } from "../../../types";
@@ -20,6 +20,7 @@ import "uplot/dist/uPlot.min.css";
 import "./style.scss";
 import classNames from "classnames";
 import ChartTooltip, { ChartTooltipProps } from "../ChartTooltip/ChartTooltip";
+import dayjs from "dayjs";
 
 export interface LineChartProps {
   metrics: MetricResult[];
@@ -57,7 +58,10 @@ const LineChart: FC<LineChartProps> = ({
   const tooltipId = useMemo(() => `${tooltipIdx.seriesIdx}_${tooltipIdx.dataIdx}`, [tooltipIdx]);
 
   const setScale = ({ min, max }: { min: number, max: number }): void => {
-    setPeriod({ from: new Date(min * 1000), to: new Date(max * 1000) });
+    setPeriod({
+      from: dayjs(min * 1000).toDate(),
+      to: dayjs(max * 1000).toDate()
+    });
   };
   const throttledSetScale = useCallback(throttle(setScale, 500), []);
   const setPlotScale = ({ u, min, max }: { u: uPlot, min: number, max: number }) => {
@@ -163,6 +167,7 @@ const LineChart: FC<LineChartProps> = ({
 
   const options: uPlotOptions = {
     ...defaultOptions,
+    tzDate: ts => dayjs(formatDateForNativeInput(dateFromSeconds(ts))).local().toDate(),
     series,
     axes: getAxes( [{}, { scale: "1" }], unit),
     scales: { ...getScales() },

app/vmui/packages/vmui/src/components/Configurators/CardinalityDatePicker/CardinalityDatePicker.tsx

@@ -15,7 +15,7 @@ const CardinalityDatePicker: FC = () => {
   const { date } = useCardinalityState();
   const cardinalityDispatch = useCardinalityDispatch();
 
-  const dateFormatted = useMemo(() => dayjs(date).format(DATE_FORMAT), [date]);
+  const dateFormatted = useMemo(() => dayjs.tz(date).format(DATE_FORMAT), [date]);
 
   const handleChangeDate = (val: string) => {
     cardinalityDispatch({ type: "SET_DATE", payload: val });

app/vmui/packages/vmui/src/components/Configurators/GlobalSettings/GlobalSettings.tsx

@@ -11,6 +11,8 @@ import { SeriesLimits } from "../../../types";
 import { useCustomPanelDispatch, useCustomPanelState } from "../../../state/customPanel/CustomPanelStateContext";
 import { getAppModeEnable } from "../../../utils/app-mode";
 import classNames from "classnames";
+import Timezones from "./Timezones/Timezones";
+import { useTimeDispatch, useTimeState } from "../../../state/time/TimeStateContext";
 
 const title = "Settings";
 
@@ -18,13 +20,16 @@ const GlobalSettings: FC = () => {
 
   const appModeEnable = getAppModeEnable();
   const { serverUrl: stateServerUrl } = useAppState();
+  const { timezone: stateTimezone } = useTimeState();
   const { seriesLimits } = useCustomPanelState();
 
   const dispatch = useAppDispatch();
+  const timeDispatch = useTimeDispatch();
   const customPanelDispatch = useCustomPanelDispatch();
 
   const [serverUrl, setServerUrl] = useState(stateServerUrl);
   const [limits, setLimits] = useState<SeriesLimits>(seriesLimits);
+  const [timezone, setTimezone] = useState(stateTimezone);
 
   const [open, setOpen] = useState(false);
   const handleOpen = () => setOpen(true);
@@ -32,6 +37,7 @@ const GlobalSettings: FC = () => {
 
   const handlerApply = () => {
     dispatch({ type: "SET_SERVER", payload: serverUrl });
+    timeDispatch({ type: "SET_TIMEZONE", payload: timezone });
     customPanelDispatch({ type: "SET_SERIES_LIMITS", payload: limits });
     handleClose();
   };
@@ -70,6 +76,12 @@ const GlobalSettings: FC = () => {
               onEnter={handlerApply}
             />
           </div>
+          <div className="vm-server-configurator__input">
+            <Timezones
+              timezoneState={timezone}
+              onChange={setTimezone}
+            />
+          </div>
           <div className="vm-server-configurator__footer">
             <Button
               variant="outlined"

app/vmui/packages/vmui/src/components/Configurators/GlobalSettings/LimitsConfigurator/LimitsConfigurator.tsx

@@ -46,7 +46,7 @@ const LimitsConfigurator: FC<ServerConfiguratorProps> = ({ limits, onChange , on
 
   return (
     <div className="vm-limits-configurator">
-      <div className="vm-limits-configurator-title">
+      <div className="vm-server-configurator__title">
         Series limits by tabs
         <Tooltip title="To disable limits set to 0">
           <Button

app/vmui/packages/vmui/src/components/Configurators/GlobalSettings/LimitsConfigurator/style.scss

@@ -3,13 +3,6 @@
 .vm-limits-configurator {
 
   &-title {
-    display: flex;
-    align-items: center;
-    justify-content: flex-start;
-    font-size: $font-size;
-    font-weight: bold;
-    margin-bottom: $padding-global;
-
     &__reset {
       display: flex;
       align-items: center;

app/vmui/packages/vmui/src/components/Configurators/GlobalSettings/ServerConfigurator/ServerConfigurator.tsx

@@ -22,14 +22,18 @@ const ServerConfigurator: FC<ServerConfiguratorProps> = ({ serverUrl, onChange ,
   };
 
   return (
-    <TextField
-      autofocus
-      label="Server URL"
-      value={serverUrl}
-      error={error}
-      onChange={onChangeServer}
-      onEnter={onEnter}
-    />
+    <div>
+      <div className="vm-server-configurator__title">
+        Server URL
+      </div>
+      <TextField
+        autofocus
+        value={serverUrl}
+        error={error}
+        onChange={onChangeServer}
+        onEnter={onEnter}
+      />
+    </div>
   );
 };
 

app/vmui/packages/vmui/src/components/Configurators/GlobalSettings/Timezones/Timezones.tsx

@@ -0,0 +1,143 @@
+import React, { FC, useMemo, useRef, useState } from "preact/compat";
+import { getTimezoneList, getUTCByTimezone } from "../../../../utils/time";
+import { ArrowDropDownIcon } from "../../../Main/Icons";
+import classNames from "classnames";
+import Popper from "../../../Main/Popper/Popper";
+import Accordion from "../../../Main/Accordion/Accordion";
+import dayjs from "dayjs";
+import TextField from "../../../Main/TextField/TextField";
+import { Timezone } from "../../../../types";
+import "./style.scss";
+
+interface TimezonesProps {
+  timezoneState: string
+  onChange: (val: string) => void
+}
+
+const Timezones: FC<TimezonesProps> = ({ timezoneState, onChange }) => {
+
+  const timezones = getTimezoneList();
+
+  const [openList, setOpenList] = useState(false);
+  const [search, setSearch] = useState("");
+  const targetRef = useRef<HTMLDivElement>(null);
+
+  const searchTimezones = useMemo(() => {
+    if (!search) return timezones;
+    try {
+      return getTimezoneList(search);
+    } catch (e) {
+      return {};
+    }
+  }, [search, timezones]);
+
+  const timezonesGroups = useMemo(() => Object.keys(searchTimezones), [searchTimezones]);
+
+  const localTimezone = useMemo(() => ({
+    region: dayjs.tz.guess(),
+    utc: getUTCByTimezone(dayjs.tz.guess())
+  }), []);
+
+  const activeTimezone = useMemo(() => ({
+    region: timezoneState,
+    utc: getUTCByTimezone(timezoneState)
+  }), [timezoneState]);
+
+  const toggleOpenList = () => {
+    setOpenList(prev => !prev);
+  };
+
+  const handleCloseList = () => {
+    setOpenList(false);
+  };
+
+  const handleChangeSearch = (val: string) => {
+    setSearch(val);
+  };
+
+  const handleSetTimezone = (val: Timezone) => {
+    onChange(val.region);
+    setSearch("");
+    handleCloseList();
+  };
+
+  const createHandlerSetTimezone = (val: Timezone) => () => {
+    handleSetTimezone(val);
+  };
+
+  return (
+    <div className="vm-timezones">
+      <div className="vm-server-configurator__title">
+        Time zone
+      </div>
+      <div
+        className="vm-timezones-item vm-timezones-item_selected"
+        onClick={toggleOpenList}
+        ref={targetRef}
+      >
+        <div className="vm-timezones-item__title">{activeTimezone.region}</div>
+        <div className="vm-timezones-item__utc">{activeTimezone.utc}</div>
+        <div
+          className={classNames({
+            "vm-timezones-item__icon": true,
+            "vm-timezones-item__icon_open": openList
+          })}
+        >
+          <ArrowDropDownIcon/>
+        </div>
+      </div>
+      <Popper
+        open={openList}
+        buttonRef={targetRef}
+        placement="bottom-left"
+        onClose={handleCloseList}
+      >
+        <div className="vm-timezones-list">
+          <div className="vm-timezones-list-header">
+            <div className="vm-timezones-list-header__search">
+              <TextField
+                autofocus
+                label="Search"
+                value={search}
+                onChange={handleChangeSearch}
+              />
+            </div>
+            <div
+              className="vm-timezones-item vm-timezones-list-group-options__item"
+              onClick={createHandlerSetTimezone(localTimezone)}
+            >
+              <div className="vm-timezones-item__title">Browser Time ({localTimezone.region})</div>
+              <div className="vm-timezones-item__utc">{localTimezone.utc}</div>
+            </div>
+          </div>
+          {timezonesGroups.map(t => (
+            <div
+              className="vm-timezones-list-group"
+              key={t}
+            >
+              <Accordion
+                defaultExpanded={true}
+                title={<div className="vm-timezones-list-group__title">{t}</div>}
+              >
+                <div className="vm-timezones-list-group-options">
+                  {searchTimezones[t] && searchTimezones[t].map(item => (
+                    <div
+                      className="vm-timezones-item vm-timezones-list-group-options__item"
+                      onClick={createHandlerSetTimezone(item)}
+                      key={item.search}
+                    >
+                      <div className="vm-timezones-item__title">{item.region}</div>
+                      <div className="vm-timezones-item__utc">{item.utc}</div>
+                    </div>
+                  ))}
+                </div>
+              </Accordion>
+            </div>
+          ))}
+        </div>
+      </Popper>
+    </div>
+  );
+};
+
+export default Timezones;

app/vmui/packages/vmui/src/components/Configurators/GlobalSettings/Timezones/style.scss

@@ -0,0 +1,96 @@
+@use "src/styles/variables" as *;
+
+.vm-timezones {
+
+  &-item {
+    display: flex;
+    align-items: center;
+    justify-content: space-between;
+    gap: $padding-small;
+    cursor: pointer;
+
+    &_selected {
+      border: $border-divider;
+      padding: $padding-small $padding-global;
+      border-radius: $border-radius-small;
+    }
+
+    &__title {
+      text-transform: capitalize;
+    }
+
+    &__utc {
+      display: inline-flex;
+      align-items: center;
+      justify-content: center;
+      background-color: rgba($color-black, 0.06);
+      padding: calc($padding-small/2);
+      border-radius: $border-radius-small;
+    }
+
+    &__icon {
+      display: inline-flex;
+      align-items: center;
+      justify-content: flex-end;
+      margin: 0 0 0 auto;
+      transition: transform 200ms ease-in;
+
+      svg {
+        width: 14px;
+      }
+
+      &_open {
+        transform: rotate(180deg);
+      }
+    }
+  }
+
+  &-list {
+    min-width: 600px;
+    max-height: 300px;
+    background-color: $color-background-block;
+    border-radius: $border-radius-medium;
+    overflow: auto;
+
+    &-header {
+      position: sticky;
+      top: 0;
+      background-color: $color-background-block;
+      z-index: 2;
+      border-bottom: $border-divider;
+
+      &__search {
+        padding: $padding-small;
+      }
+    }
+
+    &-group {
+      padding: $padding-small 0;
+      border-bottom: $border-divider;
+
+      &:last-child {
+        border-bottom: none;
+      }
+
+      &__title {
+        font-weight: bold;
+        color: $color-text-secondary;
+        padding: $padding-small $padding-global;
+      }
+
+      &-options {
+        display: grid;
+        align-items: flex-start;
+
+        &__item {
+          padding: $padding-small $padding-global;
+          transition: background-color 200ms ease;
+
+          &:hover {
+            background-color: rgba($color-black, 0.1);
+          }
+        }
+      }
+    }
+  }
+}

app/vmui/packages/vmui/src/components/Configurators/GlobalSettings/style.scss

@@ -10,6 +10,15 @@
 
   }
 
+  &__title {
+    display: flex;
+    align-items: center;
+    justify-content: flex-start;
+    font-size: $font-size;
+    font-weight: bold;
+    margin-bottom: $padding-global;
+  }
+
   &__footer {
     display: inline-grid;
     grid-template-columns: repeat(2, 1fr);

app/vmui/packages/vmui/src/components/Configurators/TimeRangeSettings/TimeDurationSelector/style.scss

@@ -1,7 +1,7 @@
 @use "src/styles/variables" as *;
 
 .vm-time-duration {
-  max-height: 168px;
+  max-height: 200px;
   overflow: auto;
   font-size: $font-size;
 }

app/vmui/packages/vmui/src/components/Configurators/TimeRangeSettings/TimeSelector/TimeSelector.tsx

@@ -1,5 +1,5 @@
 import React, { FC, useEffect, useState, useMemo, useRef } from "preact/compat";
-import { dateFromSeconds, formatDateForNativeInput } from "../../../../utils/time";
+import { dateFromSeconds, formatDateForNativeInput, getRelativeTime, getUTCByTimezone } from "../../../../utils/time";
 import TimeDurationSelector from "../TimeDurationSelector/TimeDurationSelector";
 import dayjs from "dayjs";
 import { getAppModeEnable } from "../../../../utils/app-mode";
@@ -22,20 +22,25 @@ export const TimeSelector: FC = () => {
   const [until, setUntil] = useState<string>();
   const [from, setFrom] = useState<string>();
 
-  const formFormat = useMemo(() => dayjs(from).format(DATE_TIME_FORMAT), [from]);
-  const untilFormat = useMemo(() => dayjs(until).format(DATE_TIME_FORMAT), [until]);
+  const formFormat = useMemo(() => dayjs.tz(from).format(DATE_TIME_FORMAT), [from]);
+  const untilFormat = useMemo(() => dayjs.tz(until).format(DATE_TIME_FORMAT), [until]);
 
-  const { period: { end, start }, relativeTime } = useTimeState();
+  const { period: { end, start }, relativeTime, timezone, duration } = useTimeState();
   const dispatch = useTimeDispatch();
   const appModeEnable = getAppModeEnable();
 
+  const activeTimezone = useMemo(() => ({
+    region: timezone,
+    utc: getUTCByTimezone(timezone)
+  }), [timezone]);
+
   useEffect(() => {
     setUntil(formatDateForNativeInput(dateFromSeconds(end)));
-  }, [end]);
+  }, [timezone, end]);
 
   useEffect(() => {
     setFrom(formatDateForNativeInput(dateFromSeconds(start)));
-  }, [start]);
+  }, [timezone, start]);
 
   const setDuration = ({ duration, until, id }: {duration: string, until: Date, id: string}) => {
     dispatch({ type: "SET_RELATIVE_TIME", payload: { duration, until, id } });
@@ -43,13 +48,13 @@ export const TimeSelector: FC = () => {
   };
 
   const formatRange = useMemo(() => {
-    const startFormat = dayjs(dateFromSeconds(start)).format(DATE_TIME_FORMAT);
-    const endFormat = dayjs(dateFromSeconds(end)).format(DATE_TIME_FORMAT);
+    const startFormat = dayjs.tz(dateFromSeconds(start)).format(DATE_TIME_FORMAT);
+    const endFormat = dayjs.tz(dateFromSeconds(end)).format(DATE_TIME_FORMAT);
     return {
       start: startFormat,
       end: endFormat
     };
-  }, [start, end]);
+  }, [start, end, timezone]);
 
   const dateTitle = useMemo(() => {
     const isRelativeTime = relativeTime && relativeTime !== "none";
@@ -65,7 +70,10 @@ export const TimeSelector: FC = () => {
 
   const setTimeAndClosePicker = () => {
     if (from && until) {
-      dispatch({ type: "SET_PERIOD", payload: { from: new Date(from), to: new Date(until) } });
+      dispatch({ type: "SET_PERIOD", payload: {
+        from: dayjs(from).toDate(),
+        to: dayjs(until).toDate()
+      } });
     }
     setOpenOptions(false);
   };
@@ -91,6 +99,15 @@ export const TimeSelector: FC = () => {
     setOpenOptions(false);
   };
 
+  useEffect(() => {
+    const value = getRelativeTime({
+      relativeTimeId: relativeTime,
+      defaultDuration: duration,
+      defaultEndInput: dateFromSeconds(end),
+    });
+    setDuration({ id: value.relativeTimeId, duration: value.duration, until: value.endInput });
+  }, [timezone]);
+
   useClickOutside(wrapperRef, (e) => {
     const target = e.target as HTMLElement;
     const isFromButton = fromRef?.current && fromRef.current.contains(target);
@@ -159,6 +176,10 @@ export const TimeSelector: FC = () => {
               />
             </div>
           </div>
+          <div className="vm-time-selector-left-timezone">
+            <div className="vm-time-selector-left-timezone__title">{activeTimezone.region}</div>
+            <div className="vm-time-selector-left-timezone__utc">{activeTimezone.utc}</div>
+          </div>
           <Button
             variant="text"
             startIcon={<AlarmIcon />}

app/vmui/packages/vmui/src/components/Configurators/TimeRangeSettings/TimeSelector/style.scss

@@ -30,6 +30,10 @@
         cursor: pointer;
         transition: color 200ms ease-in-out, border-bottom-color 300ms ease;
 
+        &:last-child {
+          margin-bottom: 0;
+        }
+
         &:hover {
           border-bottom-color: $color-primary;
         }
@@ -52,6 +56,26 @@
       }
     }
 
+    &-timezone {
+      display: flex;
+      align-items: center;
+      justify-content: space-between;
+      gap: $padding-small;
+      font-size: $font-size-small;
+      margin-bottom: $padding-small;
+
+      &__title {}
+
+      &__utc {
+        display: inline-flex;
+        align-items: center;
+        justify-content: center;
+        background-color: rgba($color-black, 0.06);
+        padding: calc($padding-small/2);
+        border-radius: $border-radius-small;
+      }
+    }
+
     &__controls {
       display: grid;
       grid-template-columns: repeat(2, 1fr);

app/vmui/packages/vmui/src/components/Main/Autocomplete/Autocomplete.tsx

@@ -56,13 +56,14 @@ const Autocomplete: FC<AutocompleteProps> = ({
   const handleKeyDown = (e: KeyboardEvent) => {
     const { key, ctrlKey, metaKey, shiftKey } = e;
     const modifiers = ctrlKey || metaKey || shiftKey;
+    const hasOptions = foundOptions.length;
 
-    if (key === "ArrowUp" && !modifiers) {
+    if (key === "ArrowUp" && !modifiers && hasOptions) {
       e.preventDefault();
       setFocusOption((prev) => prev <= 0 ? 0 : prev - 1);
     }
 
-    if (key === "ArrowDown" && !modifiers) {
+    if (key === "ArrowDown" && !modifiers && hasOptions) {
       e.preventDefault();
       const lastIndex = foundOptions.length - 1;
       setFocusOption((prev) => prev >= lastIndex ? lastIndex : prev + 1);

app/vmui/packages/vmui/src/components/Main/DatePicker/Calendar/Calendar.tsx

@@ -30,8 +30,8 @@ const Calendar: FC<DatePickerProps> = ({
   onClose
 }) => {
   const [displayYears, setDisplayYears] = useState(false);
-  const [viewDate, setViewDate] = useState(dayjs(date));
-  const [selectDate, setSelectDate] = useState(dayjs(date));
+  const [viewDate, setViewDate] = useState(dayjs.tz(date));
+  const [selectDate, setSelectDate] = useState(dayjs.tz(date));
   const [tab, setTab] = useState(tabs[0].value);
 
   const toggleDisplayYears = () => {
@@ -62,7 +62,7 @@ const Calendar: FC<DatePickerProps> = ({
   };
 
   useEffect(() => {
-    if (selectDate.format() === dayjs(date).format()) return;
+    if (selectDate.format() === dayjs.tz(date).format()) return;
     onChange(selectDate.format(format));
   }, [selectDate]);
 

app/vmui/packages/vmui/src/components/Main/DatePicker/Calendar/CalendarBody/CalendarBody.tsx

@@ -11,7 +11,7 @@ interface CalendarBodyProps {
 const weekday = ["Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"];
 
 const CalendarBody: FC<CalendarBodyProps> = ({ viewDate, selectDate, onChangeSelectDate }) => {
-  const today = dayjs().startOf("day");
+  const today = dayjs().tz().startOf("day");
 
   const days: (Dayjs|null)[] = useMemo(() => {
     const result = new Array(42).fill(null);

app/vmui/packages/vmui/src/components/Main/DatePicker/Calendar/style.scss

@@ -135,6 +135,7 @@
     display: grid;
     grid-template-columns: repeat(3, 1fr);
     gap: $padding-small;
+    max-height: 400px;
     overflow: auto;
 
     &__year {

app/vmui/packages/vmui/src/components/Main/DatePicker/DatePicker.tsx

@@ -20,7 +20,7 @@ const DatePicker = forwardRef<HTMLDivElement, DatePickerProps>(({
   onChange,
 }, ref) => {
   const [openCalendar, setOpenCalendar] = useState(false);
-  const dateDayjs = useMemo(() => date ? dayjs(date) : dayjs(), [date]);
+  const dateDayjs = useMemo(() => date ? dayjs.tz(date) : dayjs().tz(), [date]);
 
   const toggleOpenCalendar = () => {
     setOpenCalendar(prev => !prev);

app/vmui/packages/vmui/src/components/Main/ShortcutKeys/ShortcutKeys.tsx

@@ -28,6 +28,10 @@ const keyList = [
       {
         keys: [ctrlMeta, "Arrow Down"],
         description: "Next command from the Query history"
+      },
+      {
+        keys: [ctrlMeta, "Click by 'Eye'"],
+        description: "Toggle multiple queries"
       }
     ]
   },
@@ -36,10 +40,12 @@ const keyList = [
     list: [
       {
         keys: [ctrlMeta, "Scroll Up"],
+        alt: ["+"],
         description: "Zoom in"
       },
       {
         keys: [ctrlMeta, "Scroll Down"],
+        alt: ["-"],
         description: "Zoom out"
       },
       {
@@ -118,6 +124,15 @@ const ShortcutKeys: FC = () => {
                           {i !== l.keys.length - 1 ? "+" : ""}
                         </>
                       ))}
+                      {l.alt && l.alt.map((alt, i) => (
+                        <>
+                          or
+                          <code key={alt}>
+                            {alt}
+                          </code>
+                          {i !== l.alt.length - 1 ? "+" : ""}
+                        </>
+                      ))}
                     </div>
                     <p className="vm-shortcuts-section-list-item__description">
                       {l.description}

app/vmui/packages/vmui/src/components/Main/ShortcutKeys/style.scss

@@ -19,7 +19,7 @@
 
       &-item {
         display: grid;
-        grid-template-columns: 180px 1fr;
+        grid-template-columns: 210px 1fr;
         align-items: center;
         gap: $padding-small;
 

app/vmui/packages/vmui/src/components/Views/GraphView/GraphView.tsx

@@ -10,6 +10,7 @@ import { TimeParams } from "../../../types";
 import { AxisRange, YaxisState } from "../../../state/graph/reducer";
 import { getAvgFromArray, getMaxFromArray, getMinFromArray } from "../../../utils/math";
 import classNames from "classnames";
+import { useTimeState } from "../../../state/time/TimeStateContext";
 import "./style.scss";
 
 export interface GraphViewProps {
@@ -54,6 +55,7 @@ const GraphView: FC<GraphViewProps> = ({
   alias = [],
   fullWidth = true
 }) => {
+  const { timezone } = useTimeState();
   const currentStep = useMemo(() => customStep || period.step || 1, [period.step, customStep]);
 
   const [dataChart, setDataChart] = useState<uPlotData>([[]]);
@@ -121,7 +123,7 @@ const GraphView: FC<GraphViewProps> = ({
     setDataChart(timeDataSeries as uPlotData);
     setSeries(tempSeries);
     setLegend(tempLegend);
-  }, [data]);
+  }, [data, timezone]);
 
   useEffect(() => {
     const tempLegend: LegendItemType[] = [];

app/vmui/packages/vmui/src/constants/dayjsPlugins.ts

@@ -0,0 +1,8 @@
+import dayjs from "dayjs";
+import timezone from "dayjs/plugin/timezone";
+import duration from "dayjs/plugin/duration";
+import utc from "dayjs/plugin/utc";
+
+dayjs.extend(timezone);
+dayjs.extend(duration);
+dayjs.extend(utc);

app/vmui/packages/vmui/src/contexts/Snackbar.tsx

@@ -36,7 +36,7 @@ export const SnackbarProvider: FC = ({ children }) => {
     setSnack({
       message: infoMessage.text,
       variant: infoMessage.type,
-      key: new Date().getTime()
+      key: Date.now()
     });
     setOpen(true);
     const timeout = setTimeout(handleClose, 4000);

app/vmui/packages/vmui/src/hooks/useClickOutside.ts

@@ -8,9 +8,8 @@ const useClickOutside = <T extends HTMLElement = HTMLElement>(
   preventRef?: RefObject<T>
 ) => {
   useEffect(() => {
-    const el = ref?.current;
-
     const listener = (event: Event) => {
+      const el = ref?.current;
       const target = event.target as HTMLElement;
       const isPreventRef = preventRef?.current && preventRef.current.contains(target);
       if (!el || el.contains((event?.target as Node) || null) || isPreventRef) {
@@ -23,13 +22,10 @@ const useClickOutside = <T extends HTMLElement = HTMLElement>(
     document.addEventListener("mousedown", listener);
     document.addEventListener("touchstart", listener);
 
-    const removeListeners = () => {
+    return () => {
       document.removeEventListener("mousedown", listener);
       document.removeEventListener("touchstart", listener);
     };
-
-    if (!el) removeListeners();
-    return removeListeners;
   }, [ref, handler]); // Reload only if ref or handler changes
 };
 

app/vmui/packages/vmui/src/index.tsx

@@ -1,4 +1,5 @@
 import React, { render } from "preact/compat";
+import "./constants/dayjsPlugins";
 import App from "./App";
 import reportWebVitals from "./reportWebVitals";
 import "./styles/style.scss";

app/vmui/packages/vmui/src/pages/CustomPanel/QueryConfigurator/QueryConfigurator.tsx

@@ -11,6 +11,8 @@ import Button from "../../../components/Main/Button/Button";
 import "./style.scss";
 import Tooltip from "../../../components/Main/Tooltip/Tooltip";
 import classNames from "classnames";
+import { MouseEvent as ReactMouseEvent } from "react";
+import { arrayEquals } from "../../../utils/array";
 
 export interface QueryConfiguratorProps {
   error?: ErrorTypes | string;
@@ -55,8 +57,16 @@ const QueryConfigurator: FC<QueryConfiguratorProps> = ({ error, queryOptions, on
     setStateQuery(prev => prev.filter((q, i) => i !== index));
   };
 
-  const onToggleHideQuery = (index: number) => {
-    setHideQuery(prev => prev.includes(index) ? prev.filter(n => n !== index) : [...prev, index]);
+  const onToggleHideQuery = (e: ReactMouseEvent<HTMLButtonElement, MouseEvent>, index: number) => {
+    const { ctrlKey, metaKey } = e;
+    const ctrlMetaKey = ctrlKey || metaKey;
+
+    if (ctrlMetaKey) {
+      const hideIndexes = stateQuery.map((q, i) => i).filter(n => n !== index);
+      setHideQuery(prev => arrayEquals(hideIndexes, prev) ? [] : hideIndexes);
+    } else {
+      setHideQuery(prev => prev.includes(index) ? prev.filter(n => n !== index) : [...prev, index]);
+    }
   };
 
   const handleChangeQuery = (value: string, index: number) => {
@@ -84,11 +94,11 @@ const QueryConfigurator: FC<QueryConfiguratorProps> = ({ error, queryOptions, on
 
   const createHandlerRemoveQuery = (i: number) => () => {
     onRemoveQuery(i);
-    setHideQuery(prev => prev.map(n => n > i ? n - 1: n));
+    setHideQuery(prev => prev.includes(i) ? prev.filter(n => n !== i) : prev.map(n => n > i ? n - 1: n));
   };
 
-  const createHandlerHideQuery = (i: number) => () => {
-    onToggleHideQuery(i);
+  const createHandlerHideQuery = (i: number) => (e: ReactMouseEvent<HTMLButtonElement, MouseEvent>) => {
+    onToggleHideQuery(e, i);
   };
 
   useEffect(() => {

app/vmui/packages/vmui/src/state/cardinality/reducer.ts

@@ -23,7 +23,7 @@ export type Action =
 export const initialState: CardinalityState = {
   runQuery: 0,
   topN: getQueryStringValue("topN", 10) as number,
-  date: getQueryStringValue("date", dayjs(new Date()).format(DATE_FORMAT)) as string,
+  date: getQueryStringValue("date", dayjs().tz().format(DATE_FORMAT)) as string,
   focusLabel: getQueryStringValue("focusLabel", "") as string,
   match: getQueryStringValue("match", "") as string,
   extraLabel: getQueryStringValue("extra_label", "") as string,

app/vmui/packages/vmui/src/state/time/reducer.ts

@@ -5,14 +5,18 @@ import {
   getDateNowUTC,
   getDurationFromPeriod,
   getTimeperiodForDuration,
-  getRelativeTime
+  getRelativeTime,
+  setTimezone
 } from "../../utils/time";
 import { getQueryStringValue } from "../../utils/query-string";
+import dayjs from "dayjs";
+import { getFromStorage, saveToStorage } from "../../utils/storage";
 
 export interface TimeState {
   duration: string;
   period: TimeParams;
   relativeTime?: string;
+  timezone: string;
 }
 
 export type TimeAction =
@@ -21,12 +25,16 @@ export type TimeAction =
   | { type: "SET_PERIOD", payload: TimePeriod }
   | { type: "RUN_QUERY"}
   | { type: "RUN_QUERY_TO_NOW"}
+  | { type: "SET_TIMEZONE", payload: string }
+
+const timezone = getFromStorage("TIMEZONE") as string || dayjs.tz.guess();
+setTimezone(timezone);
 
 const defaultDuration = getQueryStringValue("g0.range_input") as string;
 
 const { duration, endInput, relativeTimeId } = getRelativeTime({
   defaultDuration: defaultDuration || "1h",
-  defaultEndInput: new Date(formatDateToLocal(getQueryStringValue("g0.end_input", getDateNowUTC()) as Date)),
+  defaultEndInput: formatDateToLocal(getQueryStringValue("g0.end_input", getDateNowUTC()) as string),
   relativeTimeId: defaultDuration ? getQueryStringValue("g0.relative_time", "none") as string : undefined
 });
 
@@ -34,8 +42,10 @@ export const initialTimeState: TimeState = {
   duration,
   period: getTimeperiodForDuration(duration, endInput),
   relativeTime: relativeTimeId,
+  timezone,
 };
 
+
 export function reducer(state: TimeState, action: TimeAction): TimeState {
   switch (action.type) {
     case "SET_DURATION":
@@ -49,7 +59,7 @@ export function reducer(state: TimeState, action: TimeAction): TimeState {
       return {
         ...state,
         duration: action.payload.duration,
-        period: getTimeperiodForDuration(action.payload.duration, new Date(action.payload.until)),
+        period: getTimeperiodForDuration(action.payload.duration, action.payload.until),
         relativeTime: action.payload.id,
       };
     case "SET_PERIOD":
@@ -77,6 +87,13 @@ export function reducer(state: TimeState, action: TimeAction): TimeState {
         ...state,
         period: getTimeperiodForDuration(state.duration)
       };
+    case "SET_TIMEZONE":
+      setTimezone(action.payload);
+      saveToStorage("TIMEZONE", action.payload);
+      return {
+        ...state,
+        timezone: action.payload
+      };
     default:
       throw new Error();
   }

app/vmui/packages/vmui/src/types/index.ts

@@ -105,3 +105,9 @@ export interface SeriesLimits {
   chart: number,
   code: number,
 }
+
+export interface Timezone {
+  region: string,
+  utc: string,
+  search?: string
+}

app/vmui/packages/vmui/src/utils/query-string.ts

@@ -5,7 +5,7 @@ import { MAX_QUERY_FIELDS } from "../constants/graph";
 export const setQueryStringWithoutPageReload = (params: Record<string, unknown>): void => {
   const w = window;
   if (w) {
-    const qsValue = Object.entries(params).map(([k, v]) => `${k}=${v}`).join("&");
+    const qsValue = Object.entries(params).map(([k, v]) => `${k}=${encodeURIComponent(String(v))}`).join("&");
     const qs = qsValue ? `?${qsValue}` : "";
     const newurl = `${w.location.protocol}//${w.location.host}${w.location.pathname}${qs}${w.location.hash}`;
     w.history.pushState({ path: newurl }, "", newurl);

app/vmui/packages/vmui/src/utils/storage.ts

@@ -6,6 +6,7 @@ export type StorageKeys = "BASIC_AUTH_DATA"
     | "QUERY_TRACING"
     | "SERIES_LIMITS"
     | "TABLE_COMPACT"
+    | "TIMEZONE"
 
 export const saveToStorage = (key: StorageKeys, value: string | boolean | Record<string, unknown>): void => {
   if (value) {

app/vmui/packages/vmui/src/utils/time.ts

@@ -1,17 +1,16 @@
-import { RelativeTimeOption, TimeParams, TimePeriod } from "../types";
+import { RelativeTimeOption, TimeParams, TimePeriod, Timezone } from "../types";
 import dayjs, { UnitTypeShort } from "dayjs";
-import duration from "dayjs/plugin/duration";
-import utc from "dayjs/plugin/utc";
 import { getQueryStringValue } from "./query-string";
 import { DATE_ISO_FORMAT } from "../constants/date";
 
-dayjs.extend(duration);
-dayjs.extend(utc);
-
 const MAX_ITEMS_PER_CHART = window.innerWidth / 4;
 
 export const limitsDurations = { min: 1, max: 1.578e+11 }; // min: 1 ms, max: 5 years
 
+// eslint-disable-next-line @typescript-eslint/ban-ts-comment
+// @ts-ignore
+export const supportedTimezones = Intl.supportedValuesOf("timeZone") as string[];
+
 export const supportedDurations = [
   { long: "days", short: "d", possible: "day" },
   { long: "weeks", short: "w", possible: "week" },
@@ -38,7 +37,7 @@ export const isSupportedDuration = (str: string): Partial<Record<UnitTypeShort,
 };
 
 export const getTimeperiodForDuration = (dur: string, date?: Date): TimeParams => {
-  const n = (date || new Date()).valueOf() / 1000;
+  const n = (date || dayjs().toDate()).valueOf() / 1000;
 
   const durItems = dur.trim().split(" ");
 
@@ -64,24 +63,24 @@ export const getTimeperiodForDuration = (dur: string, date?: Date): TimeParams =
     start: n - delta,
     end: n,
     step: step,
-    date: formatDateToUTC(date || new Date())
+    date: formatDateToUTC(date || dayjs().toDate())
   };
 };
 
-export const formatDateToLocal = (date: Date): string => {
-  return dayjs(date).utcOffset(0, true).local().format(DATE_ISO_FORMAT);
+export const formatDateToLocal = (date: string): Date => {
+  return dayjs(date).utcOffset(0, true).toDate();
 };
 
 export const formatDateToUTC = (date: Date): string => {
-  return dayjs(date).utc().format(DATE_ISO_FORMAT);
+  return dayjs.tz(date).utc().format(DATE_ISO_FORMAT);
 };
 
 export const formatDateForNativeInput = (date: Date): string => {
-  return dayjs(date).format(DATE_ISO_FORMAT);
+  return dayjs.tz(date).format(DATE_ISO_FORMAT);
 };
 
-export const getDateNowUTC = (): Date => {
-  return  new Date(dayjs().utc().format(DATE_ISO_FORMAT));
+export const getDateNowUTC = (): string => {
+  return dayjs().utc().format(DATE_ISO_FORMAT);
 };
 
 export const getDurationFromMilliseconds = (ms: number): string => {
@@ -115,7 +114,10 @@ export const checkDurationLimit = (dur: string): string => {
   return dur;
 };
 
-export const dateFromSeconds = (epochTimeInSeconds: number): Date => new Date(epochTimeInSeconds * 1000);
+export const dateFromSeconds = (epochTimeInSeconds: number): Date => dayjs(epochTimeInSeconds * 1000).toDate();
+
+const getYesterday = () => dayjs().tz().subtract(1, "day").endOf("day").toDate();
+const getToday = () => dayjs().tz().endOf("day").toDate();
 
 export const relativeTimeOptions: RelativeTimeOption[] = [
   { title: "Last 5 minutes", duration: "5m" },
@@ -132,11 +134,11 @@ export const relativeTimeOptions: RelativeTimeOption[] = [
   { title: "Last 90 days", duration: "90d" },
   { title: "Last 180 days", duration: "180d" },
   { title: "Last 1 year", duration: "1y" },
-  { title: "Yesterday", duration: "1d", until: () => dayjs().subtract(1, "day").endOf("day").toDate() },
-  { title: "Today", duration: "1d", until: () => dayjs().endOf("day").toDate() },
+  { title: "Yesterday", duration: "1d", until: getYesterday },
+  { title: "Today", duration: "1d", until: getToday },
 ].map(o => ({
   id: o.title.replace(/\s/g, "_").toLocaleLowerCase(),
-  until: o.until ? o.until : () => dayjs().toDate(),
+  until: o.until ? o.until : () => dayjs().tz().toDate(),
   ...o
 }));
 
@@ -151,3 +153,35 @@ export const getRelativeTime = ({ relativeTimeId, defaultDuration, defaultEndInp
     endInput: target ? target.until() : defaultEndInput
   };
 };
+
+export const getUTCByTimezone = (timezone: string) => {
+  const date = dayjs().tz(timezone);
+  return `UTC${date.format("Z")}`;
+};
+
+export const getTimezoneList = (search = "") => {
+  const regexp = new RegExp(search, "i");
+
+  return supportedTimezones.reduce((acc: {[key: string]: Timezone[]}, region) => {
+    const zone = (region.match(/^(.*?)\//) || [])[1] || "unknown";
+    const utc = getUTCByTimezone(region);
+    const item = {
+      region,
+      utc,
+      search: `${region} ${utc} ${region.replace(/[/_]/gmi, " ")}`
+    };
+    const includeZone = !search || (search && regexp.test(item.search));
+
+    if (includeZone && acc[zone]) {
+      acc[zone].push(item);
+    } else if (includeZone) {
+      acc[zone] = [item];
+    }
+
+    return acc;
+  }, {});
+};
+
+export const setTimezone = (timezone: string) => {
+  dayjs.tz.setDefault(timezone);
+};

app/vmui/packages/vmui/src/utils/uplot/axes.ts

@@ -5,6 +5,18 @@ import { AxisRange } from "../../state/graph/reducer";
 import { formatTicks, sizeAxis } from "./helpers";
 import { TimeParams } from "../../types";
 
+// see https://github.com/leeoniya/uPlot/tree/master/docs#axis--grid-opts
+const timeValues = [
+  // tick incr      default           year                            month day                      hour  min            sec   mode
+  [3600 * 24 * 365, "{YYYY}",         null,                           null, null,                    null, null,          null, 1],
+  [3600 * 24 * 28,  "{MMM}",          "\n{YYYY}",                     null, null,                    null, null,          null, 1],
+  [3600 * 24,       "{MM}-{DD}",      "\n{YYYY}",                     null, null,                    null, null,          null, 1],
+  [3600,            "{HH}:{mm}",      "\n{YYYY}-{MM}-{DD}",           null, "\n{MM}-{DD}",           null, null,          null, 1],
+  [60,              "{HH}:{mm}",      "\n{YYYY}-{MM}-{DD}",           null, "\n{MM}-{DD}",           null, null,          null, 1],
+  [1,               "{HH}:{mm}:{ss}", "\n{YYYY}-{MM}-{DD}",           null, "\n{MM}-{DD} {HH}:{mm}", null, null,          null, 1],
+  [0.001,           ":{ss}.{fff}",    "\n{YYYY}-{MM}-{DD} {HH}:{mm}", null, "\n{MM}-{DD} {HH}:{mm}", null, "\n{HH}:{mm}", null, 1],
+];
+
 export const getAxes = (series: Series[], unit?: string): Axis[] => Array.from(new Set(series.map(s => s.scale))).map(a => {
   const axis = {
     scale: a,
@@ -13,7 +25,7 @@ export const getAxes = (series: Series[], unit?: string): Axis[] => Array.from(n
     font: "10px Arial",
     values: (u: uPlot, ticks: number[]) => formatTicks(u, ticks, unit)
   };
-  if (!a) return { space: 80 };
+  if (!a) return { space: 80, values: timeValues };
   if (!(Number(a) % 2)) return { ...axis, side: 1 };
   return axis;
 });

dashboards/clusterbytenant.json

@@ -107,7 +107,7 @@
         {
           "datasource": {
             "type": "prometheus",
-            "uid": "${ds}"
+            "uid": "$ds"
           },
           "expr": "sum(increase(vm_tenant_inserted_rows_total{job=~\"$job\", instance=~\"$instance\",accountID=~\"$account\", projectID=~\"$project\"}[1m])/60) by (accountID,projectID) ",
           "interval": "",
@@ -205,7 +205,7 @@
         {
           "datasource": {
             "type": "prometheus",
-            "uid": "${ds}"
+            "uid": "$ds"
           },
           "editorMode": "code",
           "expr": "sum(rate(vm_tenant_select_requests_total{job=~\"$job\", instance=~\"$instance.*\",accountID=~\"$account\", projectID=~\"$project\"}[$__rate_interval])) by (accountID,projectID) ",
@@ -311,7 +311,7 @@
         {
           "datasource": {
             "type": "prometheus",
-            "uid": "${ds}"
+            "uid": "$ds"
           },
           "expr": "sum(vm_tenant_active_timeseries{job=~\"$job\", instance=~\"$instance.*\",accountID=~\"$account\",projectID=~\"$project\"}) by(accountID,projectID)",
           "format": "time_series",
@@ -409,7 +409,7 @@
         {
           "datasource": {
             "type": "prometheus",
-            "uid": "${ds}"
+            "uid": "$ds"
           },
           "expr": "sum(increase(vm_tenant_timeseries_created_total{job=~\"$job\", instance=~\"$instance\",accountID=~\"$account\", projectID=~\"$project\"}[1m])/60) by(accountID,projectID)",
           "interval": "",
@@ -504,7 +504,7 @@
         {
           "datasource": {
             "type": "prometheus",
-            "uid": "${ds}"
+            "uid": "$ds"
           },
           "editorMode": "code",
           "expr": "sum(vm_tenant_used_tenant_bytes{job=~\"$job\", instance=~\"$instance\",accountID=~\"$account\",projectID=~\"$project\"}) by(accountID,projectID)",
@@ -579,6 +579,7 @@
         "allValue": ".*",
         "current": {},
         "datasource": {
+          "type": "prometheus",
           "uid": "$ds"
         },
         "definition": "label_values(vm_app_version{version=~\"^vm(insert|select|storage).*\"}, job)",
@@ -604,6 +605,7 @@
         "allValue": ".*",
         "current": {},
         "datasource": {
+          "type": "prometheus",
           "uid": "$ds"
         },
         "definition": "label_values(vm_app_version{job=~\"$job\"}, instance)",
@@ -629,6 +631,7 @@
         "allValue": ".*",
         "current": {},
         "datasource": {
+          "type": "prometheus",
           "uid": "$ds"
         },
         "definition": "label_values(vm_tenant_active_timeseries{job=~\"$job\"},accountID)",
@@ -654,6 +657,7 @@
         "allValue": ".*",
         "current": {},
         "datasource": {
+          "type": "prometheus",
           "uid": "$ds"
         },
         "definition": "label_values(vm_tenant_active_timeseries{accountID=~\"$accountID\"},projectID)",
@@ -687,4 +691,4 @@
   "uid": "IZFqd3lMz",
   "version": 7,
   "weekStart": ""
-}
+}