lib/promscrape: typo fix

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/753

Thanks to @PerGon and @clmssz for help with debugging.

README.md

@@ -78,6 +78,7 @@ See [features available for enterprise customers](https://github.com/VictoriaMet
   * [OpenTSDB put message](#sending-data-via-telnet-put-protocol) if `-opentsdbListenAddr` is set.
   * [HTTP OpenTSDB /api/put requests](#sending-opentsdb-data-via-http-apiput-requests) if `-opentsdbHTTPListenAddr` is set.
   * [/api/v1/import](#how-to-import-time-series-data).
+  * [Prometheus exposition format](#how-to-import-data-in-prometheus-exposition-format).
   * [Arbitrary CSV data](#how-to-import-csv-data).
 * Supports metrics' relabeling. See [these docs](#relabeling) for details.
 * Ideally works with big amounts of time series data from Kubernetes, IoT sensors, connected cars, industrial telemetry, financial data and various Enterprise workloads.
@@ -102,6 +103,8 @@ See [features available for enterprise customers](https://github.com/VictoriaMet
 * [How to import data in Prometheus exposition format](#how-to-import-data-in-prometheus-exposition-format)
 * [How to import CSV data](#how-to-import-csv-data)
 * [Prometheus querying API usage](#prometheus-querying-api-usage)
+  * [Prometheus querying API enhancements](#prometheus-querying-api-enhancements)
+* [Graphite Metrics API usage](#graphite-metrics-api-usage)
 * [How to build from sources](#how-to-build-from-sources)
   * [Development build](#development-build)
   * [Production build](#production-build)
@@ -391,9 +394,11 @@ The `/api/v1/export` endpoint should return the following response:
 
 ### Querying Graphite data
 
-Data sent to VictoriaMetrics via `Graphite plaintext protocol` may be read either via
-[Prometheus querying API](#prometheus-querying-api-usage)
-or via [go-graphite/carbonapi](https://github.com/go-graphite/carbonapi/blob/master/cmd/carbonapi/carbonapi.example.prometheus.yaml).
+Data sent to VictoriaMetrics via `Graphite plaintext protocol` may be read via the following APIs:
+
+* [Prometheus querying API](#prometheus-querying-api-usage)
+* Metric names can be explored via [Graphite metrics API](#graphite-metrics-api-usage)
+* [go-graphite/carbonapi](https://github.com/go-graphite/carbonapi/blob/master/cmd/carbonapi/carbonapi.example.prometheus.yaml)
 
 ### How to send data from OpenTSDB-compatible agents
 
@@ -516,6 +521,9 @@ The following response should be returned:
 {"metric":{"__name__":"ask","market":"NYSE","ticker":"GOOG"},"values":[1.23],"timestamps":[1583865146495]}
 ```
 
+Extra labels may be added to all the imported lines by passing `extra_label=name=value` query args.
+For example, `/api/v1/import/csv?extra_label=foo=bar` would add `"foo":"bar"` label to all the imported lines.
+
 Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail.
 
 
@@ -534,12 +542,18 @@ The following command may be used for verifying the imported data:
 curl -G 'http://localhost:8428/api/v1/export' -d 'match={__name__=~"foo"}'
 ```
 
-It should return somethins like the following:
+It should return something like the following:
 
 ```
 {"metric":{"__name__":"foo","bar":"baz"},"values":[123],"timestamps":[1594370496905]}
 ```
 
+Extra labels may be added to all the imported metrics by passing `extra_label=name=value` query args.
+For example, `/api/v1/import/prometheus?extra_label=foo=bar` would add `{foo="bar"}` label to all the imported metrics.
+
+If timestamp is missing in `<metric> <value> <timestamp>` Prometheus exposition format line, then the current timestamp is used during data ingestion.
+It can be overriden by passing unix timestamp in *milliseconds* via `timestamp` query arg. For example, `/api/v1/import/prometheus?timestamp=1594370496905`.
+
 VictoriaMetrics accepts arbitrary number of lines in a single request to `/api/v1/import/prometheus`, i.e. it supports data streaming.
 
 VictoriaMetrics also may scrape Prometheus targets - see [these docs](#how-to-scrape-prometheus-exporters-such-as-node-exporter).
@@ -558,6 +572,13 @@ VictoriaMetrics supports the following handlers from [Prometheus querying API](h
 
 These handlers can be queried from Prometheus-compatible clients such as Grafana or curl.
 
+#### Prometheus querying API enhancements
+
+Additionally to unix timestamps and [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) VictoriaMetrics accepts relative times in `time`, `start` and `end` query args.
+For example, the following query would return data for the last 30 minutes: `/api/v1/query_range?start=-30m&query=...`.
+
+By default, VictoriaMetrics returns time series for the last 5 minutes from /api/v1/series, while the Prometheus API defaults to all time.  Use `start` and `end` to select a different time range.
+
 VictoriaMetrics accepts additional args for `/api/v1/labels` and `/api/v1/label/.../values` handlers.
 See [this feature request](https://github.com/prometheus/prometheus/issues/6178) for details:
 
@@ -571,6 +592,21 @@ Additionally VictoriaMetrics provides the following handlers:
 * `/api/v1/labels/count` - it returns a list of `label: values_count` entries. It can be used for determining labels with the maximum number of values.
 * `/api/v1/status/active_queries` - it returns a list of currently running queries.
 
+
+### Graphite Metrics API usage
+
+VictoriaMetrics supports the following handlers from [Graphite Metrics API](https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api):
+
+* [/metrics/find](https://graphite-api.readthedocs.io/en/latest/api.html#metrics-find)
+* [/metrics/expand](https://graphite-api.readthedocs.io/en/latest/api.html#metrics-expand)
+* [/metrics/index.json](https://graphite-api.readthedocs.io/en/latest/api.html#metrics-index-json)
+
+VictoriaMetrics accepts the following additional query args at `/metrics/find` and `/metrics/expand`:
+  * `label` - for selecting arbitrary label values. By default `label=__name__`, i.e. metric names are selected.
+  * `delimiter` - for using different delimiters in metric name hierachy. For example, `/metrics/find?delimiter=_&query=node_*` would return all the metric name prefixes
+    that start with `node_`. By default `delimiter=.`.
+
+
 ### How to build from sources
 
 We recommend using either [binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) or
@@ -677,7 +713,8 @@ for metrics to delete. After that all the time series matching the given selecto
 the deleted time series isn't freed instantly - it is freed during subsequent [background merges of data files](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282).
 
 It is recommended verifying which metrics will be deleted with the call to `http://<victoria-metrics-addr>:8428/api/v1/series?match[]=<timeseries_selector_for_delete>`
-before actually deleting the metrics.
+before actually deleting the metrics.  By default this query will only scan active series in the past 5 minutes, so you may need to
+adjust `start` and `end` to a suitable range to achieve match hits.
 
 The `/api/v1/admin/tsdb/delete_series` handler may be protected with `authKey` if `-deleteAuthKey` command-line flag is set.
 
@@ -759,6 +796,9 @@ curl -H 'Accept-Encoding: gzip' http://source-victoriametrics:8428/api/v1/export
 curl -X POST -H 'Content-Encoding: gzip' http://destination-victoriametrics:8428/api/v1/import -T exported_data.jsonl.gz
 ```
 
+Extra labels may be added to all the imported time series by passing `extra_label=name=value` query args.
+For example, `/api/v1/import?extra_label=foo=bar` would add `"foo":"bar"` label to all the imported time series.
+
 Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail.
 
 Each request to `/api/v1/import` can load up to a single vCPU core on VictoriaMetrics. Import speed can be improved by splitting the original file into smaller parts
@@ -876,7 +916,8 @@ with the enabled de-duplication. See [this section](#deduplication) for details.
 
 VictoriaMetrics de-duplicates data points if `-dedup.minScrapeInterval` command-line flag
 is set to positive duration. For example, `-dedup.minScrapeInterval=60s` would de-duplicate data points
-on the same time series if they are located closer than 60s to each other.
+on the same time series if they fall within the same discrete 60s bucket.  The earliest data point will be kept.  In the case of equal timestamps, an arbitrary data point will be kept.
+
 The de-duplication reduces disk space usage if multiple identically configured Prometheus instances in HA pair
 write data to the same VictoriaMetrics instance. Note that these Prometheus instances must have identical
 `external_labels` section in their configs, so they write data to the same time series.
@@ -1082,7 +1123,7 @@ for data with timestamps close to the current time.
 
 ### Data updates
 
-VictoriaMetrics doesn't support updating already exiting sample values to new ones. It stores all the ingested data points
+VictoriaMetrics doesn't support updating already existing sample values to new ones. It stores all the ingested data points
 for the same time series with identical timestamps. While is possible substituting old time series with new time series via
 [removal of old time series](#how-to-delete-timeseries) and then [writing new time series](#backfilling), this approach
 should be used only for one-off updates. It shouldn't be used for frequent updates because of non-zero overhead related to data removal.

app/victoria-metrics/testdata/graphite/subquery-aggregation.json

@@ -9,6 +9,6 @@
   "query": ["/api/v1/query?query=min%20by%20(item)%20(min_over_time(forms_daily_count[10m:1m]))&time={TIME_S-1m}"],
   "result_query": {
     "status":"success",
-    "data":{"resultType":"vector","result":[{"metric":{"item":"x"},"value":["{TIME_S-1m}","1"]},{"metric":{"item":"y"},"value":["{TIME_S-1m}","3"]}]}
+    "data":{"resultType":"vector","result":[{"metric":{"item":"x"},"value":["{TIME_S-1m}","2"]},{"metric":{"item":"y"},"value":["{TIME_S-1m}","4"]}]}
   }
 }

app/vmagent/README.md

@@ -218,8 +218,7 @@ If you have suggestions, improvements or found a bug - feel free to open an issu
 * When `vmagent` scrapes many unreliable targets, it can flood error log with scrape errors. These errors can be suppressed
   by passing `-promscrape.suppressScrapeErrors` command-line flag to `vmagent`. The most recent scrape error per each target can be observed at `http://vmagent-host:8429/targets`.
 
-* It is recommended to increase `-remoteWrite.queues` if `vmagent` collects more than 100K samples per second
-  and `vmagent_remotewrite_pending_data_bytes` metric exported at `http://vmagent-host:8429/metrics` page constantly grows.
+* It is recommended to increase `-remoteWrite.queues` if `vmagent_remotewrite_pending_data_bytes` metric exported at `http://vmagent-host:8429/metrics` page constantly grows.
 
 * If you see gaps on the data pushed by `vmagent` to remote storage when `-remoteWrite.maxDiskUsagePerURL` is set, then try increasing `-remoteWrite.queues`.
   Such gaps may appear because `vmagent` cannot keep up with sending the collected data to remote storage, so it starts dropping the buffered data

app/vmagent/csvimport/request_handler.go

@@ -6,6 +6,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmagent/common"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmagent/remotewrite"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
+	parserCommon "github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/common"
 	parser "github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/csvimport"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/writeconcurrencylimiter"
 	"github.com/VictoriaMetrics/metrics"
@@ -18,12 +19,18 @@ var (
 
 // InsertHandler processes csv data from req.
 func InsertHandler(req *http.Request) error {
+	extraLabels, err := parserCommon.GetExtraLabels(req)
+	if err != nil {
+		return err
+	}
 	return writeconcurrencylimiter.Do(func() error {
-		return parser.ParseStream(req, insertRows)
+		return parser.ParseStream(req, func(rows []parser.Row) error {
+			return insertRows(rows, extraLabels)
+		})
 	})
 }
 
-func insertRows(rows []parser.Row) error {
+func insertRows(rows []parser.Row, extraLabels []prompbmarshal.Label) error {
 	ctx := common.GetPushCtx()
 	defer common.PutPushCtx(ctx)
 
@@ -44,6 +51,7 @@ func insertRows(rows []parser.Row) error {
 				Value: tag.Value,
 			})
 		}
+		labels = append(labels, extraLabels...)
 		samples = append(samples, prompbmarshal.Sample{
 			Value:     r.Value,
 			Timestamp: r.Timestamp,

app/vmagent/prometheusimport/request_handler.go

@@ -6,6 +6,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmagent/common"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmagent/remotewrite"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
+	parserCommon "github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/common"
 	parser "github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/prometheus"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/writeconcurrencylimiter"
 	"github.com/VictoriaMetrics/metrics"
@@ -18,13 +19,23 @@ var (
 
 // InsertHandler processes `/api/v1/import/prometheus` request.
 func InsertHandler(req *http.Request) error {
+	extraLabels, err := parserCommon.GetExtraLabels(req)
+	if err != nil {
+		return err
+	}
+	defaultTimestamp, err := parserCommon.GetTimestamp(req)
+	if err != nil {
+		return err
+	}
 	return writeconcurrencylimiter.Do(func() error {
 		isGzipped := req.Header.Get("Content-Encoding") == "gzip"
-		return parser.ParseStream(req.Body, isGzipped, insertRows)
+		return parser.ParseStream(req.Body, defaultTimestamp, isGzipped, func(rows []parser.Row) error {
+			return insertRows(rows, extraLabels)
+		})
 	})
 }
 
-func insertRows(rows []parser.Row) error {
+func insertRows(rows []parser.Row, extraLabels []prompbmarshal.Label) error {
 	ctx := common.GetPushCtx()
 	defer common.PutPushCtx(ctx)
 
@@ -45,6 +56,7 @@ func insertRows(rows []parser.Row) error {
 				Value: tag.Value,
 			})
 		}
+		labels = append(labels, extraLabels...)
 		samples = append(samples, prompbmarshal.Sample{
 			Value:     r.Value,
 			Timestamp: r.Timestamp,

app/vmagent/remotewrite/client.go

@@ -185,6 +185,7 @@ func (c *client) runWorker() {
 
 func (c *client) sendBlock(block []byte) {
 	retryDuration := time.Second
+	retriesCount := 0
 
 again:
 	req, err := http.NewRequest("POST", c.remoteWriteURL, bytes.NewBuffer(block))
@@ -229,6 +230,7 @@ again:
 	}
 
 	// Unexpected status code returned
+	retriesCount++
 	metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_remotewrite_requests_total{url=%q, status_code="%d"}`, c.urlLabelValue, statusCode)).Inc()
 	retryDuration *= 2
 	if retryDuration > time.Minute {
@@ -237,10 +239,10 @@ again:
 	body, err := ioutil.ReadAll(resp.Body)
 	_ = resp.Body.Close()
 	if err != nil {
-		logger.Errorf("cannot read response body from %q: %s", c.remoteWriteURL, err)
+		logger.Errorf("cannot read response body from %q during retry #%d: %s", c.remoteWriteURL, retriesCount, err)
 	} else {
-		logger.Errorf("unexpected status code received after sending a block with size %d bytes to %q: %d; response body=%q; re-sending the block in %.3f seconds",
-			len(block), c.remoteWriteURL, statusCode, body, retryDuration.Seconds())
+		logger.Errorf("unexpected status code received after sending a block with size %d bytes to %q during retry #%d: %d; response body=%q; "+
+			"re-sending the block in %.3f seconds", len(block), c.remoteWriteURL, retriesCount, statusCode, body, retryDuration.Seconds())
 	}
 	t := time.NewTimer(retryDuration)
 	select {

app/vmagent/remotewrite/pendingseries.go

@@ -3,10 +3,12 @@ package remotewrite
 import (
 	"flag"
 	"sync"
+	"sync/atomic"
 	"time"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fasttime"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/persistentqueue"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
 	"github.com/VictoriaMetrics/metrics"
@@ -17,7 +19,7 @@ var (
 	flushInterval = flag.Duration("remoteWrite.flushInterval", time.Second, "Interval for flushing the data to remote storage. "+
 		"Higher value reduces network bandwidth usage at the cost of delayed push of scraped data to remote storage. "+
 		"Minimum supported interval is 1 second")
-	maxUnpackedBlockSize = flag.Int("remoteWrite.maxBlockSize", 32*1024*1024, "The maximum size in bytes of unpacked request to send to remote storage. "+
+	maxUnpackedBlockSize = flagutil.NewBytes("remoteWrite.maxBlockSize", 32*1024*1024, "The maximum size in bytes of unpacked request to send to remote storage. "+
 		"It shouldn't exceed -maxInsertRequestSize from VictoriaMetrics")
 )
 
@@ -68,7 +70,7 @@ func (ps *pendingSeries) periodicFlusher() {
 		case <-ps.stopCh:
 			mustStop = true
 		case <-ticker.C:
-			if fasttime.UnixTimestamp()-ps.wr.lastFlushTime < uint64(flushSeconds) {
+			if fasttime.UnixTimestamp()-atomic.LoadUint64(&ps.wr.lastFlushTime) < uint64(flushSeconds) {
 				continue
 			}
 		}
@@ -79,10 +81,12 @@ func (ps *pendingSeries) periodicFlusher() {
 }
 
 type writeRequest struct {
-	wr            prompbmarshal.WriteRequest
-	pushBlock     func(block []byte)
+	// Move lastFlushTime to the top of the struct in order to guarantee atomic access on 32-bit architectures.
 	lastFlushTime uint64
 
+	wr        prompbmarshal.WriteRequest
+	pushBlock func(block []byte)
+
 	tss []prompbmarshal.TimeSeries
 
 	labels  []prompbmarshal.Label
@@ -113,7 +117,7 @@ func (wr *writeRequest) reset() {
 
 func (wr *writeRequest) flush() {
 	wr.wr.Timeseries = wr.tss
-	wr.lastFlushTime = fasttime.UnixTimestamp()
+	atomic.StoreUint64(&wr.lastFlushTime, fasttime.UnixTimestamp())
 	pushWriteRequest(&wr.wr, wr.pushBlock)
 	wr.reset()
 }
@@ -122,9 +126,9 @@ func (wr *writeRequest) push(src []prompbmarshal.TimeSeries) {
 	tssDst := wr.tss
 	for i := range src {
 		tssDst = append(tssDst, prompbmarshal.TimeSeries{})
-		dst := &tssDst[len(tssDst)-1]
-		wr.copyTimeSeries(dst, &src[i])
-		if len(wr.tss) >= maxRowsPerBlock {
+		wr.copyTimeSeries(&tssDst[len(tssDst)-1], &src[i])
+		if len(tssDst) >= maxRowsPerBlock {
+			wr.tss = tssDst
 			wr.flush()
 			tssDst = wr.tss
 		}
@@ -164,7 +168,7 @@ func pushWriteRequest(wr *prompbmarshal.WriteRequest, pushBlock func(block []byt
 	}
 	bb := writeRequestBufPool.Get()
 	bb.B = prompbmarshal.MarshalWriteRequest(bb.B[:0], wr)
-	if len(bb.B) <= *maxUnpackedBlockSize {
+	if len(bb.B) <= maxUnpackedBlockSize.N {
 		zb := snappyBufPool.Get()
 		zb.B = snappy.Encode(zb.B[:cap(zb.B)], bb.B)
 		writeRequestBufPool.Put(bb)

app/vmagent/remotewrite/remotewrite.go

@@ -3,6 +3,7 @@ package remotewrite
 import (
 	"flag"
 	"fmt"
+	"runtime"
 	"sync"
 	"sync/atomic"
 
@@ -27,12 +28,12 @@ var (
 		"isn't enough for sending high volume of collected data to remote storage")
 	showRemoteWriteURL = flag.Bool("remoteWrite.showURL", false, "Whether to show -remoteWrite.url in the exported metrics. "+
 		"It is hidden by default, since it can contain sensitive info such as auth key")
-	maxPendingBytesPerURL = flag.Int("remoteWrite.maxDiskUsagePerURL", 0, "The maximum file-based buffer size in bytes at -remoteWrite.tmpDataPath "+
+	maxPendingBytesPerURL = flagutil.NewBytes("remoteWrite.maxDiskUsagePerURL", 0, "The maximum file-based buffer size in bytes at -remoteWrite.tmpDataPath "+
 		"for each -remoteWrite.url. When buffer size reaches the configured maximum, then old data is dropped when adding new data to the buffer. "+
 		"Buffered data is stored in ~500MB chunks, so the minimum practical value for this flag is 500000000. "+
 		"Disk usage is unlimited if the value is set to 0")
-	decimalPlaces = flag.Int("remoteWrite.decimalPlaces", 0, "The number of significant decimal places to leave in metric values before writing them to remote storage. "+
-		"See https://en.wikipedia.org/wiki/Significant_figures . Zero value saves all the significant decimal places. "+
+	significantFigures = flag.Int("remoteWrite.significantFigures", 0, "The number of significant figures to leave in metric values before writing them to remote storage. "+
+		"See https://en.wikipedia.org/wiki/Significant_figures . Zero value saves all the significant figures. "+
 		"This option may be used for increasing on-disk compression level for the stored metrics")
 )
 
@@ -41,6 +42,10 @@ var rwctxs []*remoteWriteCtx
 // Contains the current relabelConfigs.
 var allRelabelConfigs atomic.Value
 
+// maxQueues limits the maximum value for `-remoteWrite.queues`. There is no sense in setting too high value,
+// since it may lead to high memory usage due to big number of buffers.
+var maxQueues = runtime.GOMAXPROCS(-1) * 4
+
 // Init initializes remotewrite.
 //
 // It must be called after flag.Parse().
@@ -48,9 +53,14 @@ var allRelabelConfigs atomic.Value
 // Stop must be called for graceful shutdown.
 func Init() {
 	if len(*remoteWriteURLs) == 0 {
-		logger.Fatalf("at least one `-remoteWrite.url` must be set")
+		logger.Fatalf("at least one `-remoteWrite.url` command-line flag must be set")
+	}
+	if *queues > maxQueues {
+		*queues = maxQueues
+	}
+	if *queues <= 0 {
+		*queues = 1
 	}
-
 	if !*showRemoteWriteURL {
 		// remoteWrite.url can contain authentication codes, so hide it at `/metrics` output.
 		httpserver.RegisterSecretFlag("remoteWrite.url")
@@ -75,7 +85,7 @@ func Init() {
 	for i, remoteWriteURL := range *remoteWriteURLs {
 		urlLabelValue := fmt.Sprintf("secret-url-%d", i+1)
 		if *showRemoteWriteURL {
-			urlLabelValue = remoteWriteURL
+			urlLabelValue = fmt.Sprintf("%d:%s", i+1, remoteWriteURL)
 		}
 		rwctx := newRemoteWriteCtx(i, remoteWriteURL, maxInmemoryBlocks, urlLabelValue)
 		rwctxs = append(rwctxs, rwctx)
@@ -124,13 +134,13 @@ func Stop() {
 //
 // Note that wr may be modified by Push due to relabeling and rounding.
 func Push(wr *prompbmarshal.WriteRequest) {
-	if *decimalPlaces > 0 {
-		// Round values according to decimalPlaces
+	if *significantFigures > 0 {
+		// Round values according to significantFigures
 		for i := range wr.Timeseries {
 			samples := wr.Timeseries[i].Samples
 			for j := range samples {
 				s := &samples[j]
-				s.Value = decimal.Round(s.Value, *decimalPlaces)
+				s.Value = decimal.Round(s.Value, *significantFigures)
 			}
 		}
 	}
@@ -182,8 +192,8 @@ type remoteWriteCtx struct {
 
 func newRemoteWriteCtx(argIdx int, remoteWriteURL string, maxInmemoryBlocks int, urlLabelValue string) *remoteWriteCtx {
 	h := xxhash.Sum64([]byte(remoteWriteURL))
-	path := fmt.Sprintf("%s/persistent-queue/%016X", *tmpDataPath, h)
-	fq := persistentqueue.MustOpenFastQueue(path, remoteWriteURL, maxInmemoryBlocks, *maxPendingBytesPerURL)
+	path := fmt.Sprintf("%s/persistent-queue/%d_%016X", *tmpDataPath, argIdx+1, h)
+	fq := persistentqueue.MustOpenFastQueue(path, remoteWriteURL, maxInmemoryBlocks, maxPendingBytesPerURL.N)
 	_ = metrics.GetOrCreateGauge(fmt.Sprintf(`vmagent_remotewrite_pending_data_bytes{path=%q, url=%q}`, path, urlLabelValue), func() float64 {
 		return float64(fq.GetPendingBytes())
 	})

app/vmagent/vmimport/request_handler.go

@@ -7,6 +7,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmagent/remotewrite"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
+	parserCommon "github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/common"
 	parser "github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/vmimport"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/writeconcurrencylimiter"
 	"github.com/VictoriaMetrics/metrics"
@@ -21,12 +22,18 @@ var (
 //
 // See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6
 func InsertHandler(req *http.Request) error {
+	extraLabels, err := parserCommon.GetExtraLabels(req)
+	if err != nil {
+		return err
+	}
 	return writeconcurrencylimiter.Do(func() error {
-		return parser.ParseStream(req, insertRows)
+		return parser.ParseStream(req, func(rows []parser.Row) error {
+			return insertRows(rows, extraLabels)
+		})
 	})
 }
 
-func insertRows(rows []parser.Row) error {
+func insertRows(rows []parser.Row, extraLabels []prompbmarshal.Label) error {
 	ctx := common.GetPushCtx()
 	defer common.PutPushCtx(ctx)
 
@@ -44,6 +51,7 @@ func insertRows(rows []parser.Row) error {
 				Value: bytesutil.ToUnsafeString(tag.Value),
 			})
 		}
+		labels = append(labels, extraLabels...)
 		values := r.Values
 		timestamps := r.Timestamps
 		_ = timestamps[len(values)-1]

app/vmalert/README.md

@@ -194,8 +194,12 @@ The shortlist of configuration flags is the following:
     	Supports array of values separated by comma or specified via multiple flags.
   -external.url string
     	External URL is used as alert's source for sent alerts to the notifier
+  -http.connTimeout duration
+    	Incoming http connections are closed after the configured timeout. This may help spreading incoming load among a cluster of services behind load balancer. Note that the real timeout may be bigger by up to 10% as a protection from Thundering herd problem (default 2m0s)
   -http.disableResponseCompression
     	Disable compression of HTTP responses for saving CPU resources. By default compression is enabled to save network bandwidth
+  -http.idleConnTimeout duration
+    	Timeout for incoming idle http connections (default 1m0s)
   -http.maxGracefulShutdownDuration duration
     	The maximum duration for graceful shutdown of HTTP server. Highly loaded server may require increased value for graceful shutdown (default 7s)
   -http.pathPrefix string
@@ -216,8 +220,9 @@ The shortlist of configuration flags is the following:
     	Minimum level of errors to log. Possible values: INFO, WARN, ERROR, FATAL, PANIC (default "INFO")
   -loggerOutput string
     	Output for the logs. Supported values: stderr, stdout (default "stderr")
-  -memory.allowedBytes int
+  -memory.allowedBytes value
     	Allowed size of system memory VictoriaMetrics caches may occupy. This option overrides -memory.allowedPercent if set to non-zero value. Too low value may increase cache miss rate, which usually results in higher CPU and disk IO usage. Too high value may evict too much data from OS page cache, which will result in higher disk IO usage
+    	Supports the following optional suffixes for values: KB, MB, GB, KiB, MiB, GiB (default 0)
   -memory.allowedPercent float
     	Allowed percent of system memory VictoriaMetrics caches may occupy. See also -memory.allowedBytes. Too low value may increase cache miss rate, which usually results in higher CPU and disk IO usage. Too high value may evict too much data from OS page cache, which will result in higher disk IO usage (default 60)
   -metricsAuthKey string
@@ -293,8 +298,8 @@ The shortlist of configuration flags is the following:
     	Path to the file with alert rules. 
     	Supports patterns. Flag can be specified multiple times. 
     	Examples:
-    	 -rule /path/to/file. Path to a single file with alerting rules
-    	 -rule dir/*.yaml -rule /*.yaml. Relative path to all .yaml files in "dir" folder, 
+    	 -rule="/path/to/file". Path to a single file with alerting rules
+    	 -rule="dir/*.yaml" -rule="/*.yaml". Relative path to all .yaml files in "dir" folder, 
     	absolute path to all .yaml files in root.
     	Rule files may contain %{ENV_VAR} placeholders, which are substituted by the corresponding env vars.
     	Supports array of values separated by comma or specified via multiple flags.

app/vmalert/alerting.go

@@ -26,6 +26,7 @@ type AlertingRule struct {
 	Labels      map[string]string
 	Annotations map[string]string
 	GroupID     uint64
+	GroupName   string
 
 	// guard status fields
 	mu sync.RWMutex
@@ -56,6 +57,7 @@ func newAlertingRule(group *Group, cfg config.Rule) *AlertingRule {
 		Labels:      cfg.Labels,
 		Annotations: cfg.Annotations,
 		GroupID:     group.ID(),
+		GroupName:   group.Name,
 		alerts:      make(map[uint64]*notifier.Alert),
 		metrics:     &alertingRuleMetrics{},
 	}
@@ -242,6 +244,9 @@ func (ar *AlertingRule) newAlert(m datasource.Metric, start time.Time) (*notifie
 		Start:   start,
 		Expr:    ar.Expr,
 	}
+	// label defined here to make override possible by
+	// time series labels.
+	a.Labels[alertGroupNameLabel] = ar.GroupName
 	for _, l := range m.Labels {
 		// drop __name__ to be consistent with Prometheus alerting
 		if l.Name == "__name__" {
@@ -336,15 +341,18 @@ func (ar *AlertingRule) newAlertAPI(a notifier.Alert) *APIAlert {
 }
 
 const (
-	// AlertMetricName is the metric name for synthetic alert timeseries.
+	// alertMetricName is the metric name for synthetic alert timeseries.
 	alertMetricName = "ALERTS"
-	// AlertForStateMetricName is the metric name for 'for' state of alert.
+	// alertForStateMetricName is the metric name for 'for' state of alert.
 	alertForStateMetricName = "ALERTS_FOR_STATE"
 
-	// AlertNameLabel is the label name indicating the name of an alert.
+	// alertNameLabel is the label name indicating the name of an alert.
 	alertNameLabel = "alertname"
-	// AlertStateLabel is the label name indicating the state of an alert.
+	// alertStateLabel is the label name indicating the state of an alert.
 	alertStateLabel = "alertstate"
+
+	// alertGroupNameLabel defines the label name attached for generated time series.
+	alertGroupNameLabel = "alertgroup"
 )
 
 // alertToTimeSeries converts the given alert with the given timestamp to timeseries

app/vmalert/config/config.go

@@ -1,6 +1,7 @@
 package config
 
 import (
+	"crypto/md5"
 	"fmt"
 	"hash/fnv"
 	"io/ioutil"
@@ -11,6 +12,7 @@ import (
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/notifier"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/envtemplate"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 	"github.com/VictoriaMetrics/metricsql"
 	"gopkg.in/yaml.v2"
 )
@@ -23,11 +25,30 @@ type Group struct {
 	Interval    time.Duration `yaml:"interval,omitempty"`
 	Rules       []Rule        `yaml:"rules"`
 	Concurrency int           `yaml:"concurrency"`
+	// Checksum stores the hash of yaml definition for this group.
+	// May be used to detect any changes like rules re-ordering etc.
+	Checksum string
 
 	// Catches all undefined fields and must be empty after parsing.
 	XXX map[string]interface{} `yaml:",inline"`
 }
 
+// UnmarshalYAML implements the yaml.Unmarshaler interface.
+func (g *Group) UnmarshalYAML(unmarshal func(interface{}) error) error {
+	type group Group
+	if err := unmarshal((*group)(g)); err != nil {
+		return err
+	}
+	b, err := yaml.Marshal(g)
+	if err != nil {
+		return fmt.Errorf("failed to marshal group configuration for checksum: %s", err)
+	}
+	h := md5.New()
+	h.Write(b)
+	g.Checksum = fmt.Sprintf("%x", h.Sum(nil))
+	return nil
+}
+
 // Validate check for internal Group or Rule configuration errors
 func (g *Group) Validate(validateAnnotations, validateExpressions bool) error {
 	if g.Name == "" {
@@ -100,7 +121,7 @@ func (r *Rule) Name() string {
 }
 
 // HashRule hashes significant Rule fields into
-// unique hash value
+// unique hash that supposed to define Rule uniqueness
 func HashRule(r Rule) uint64 {
 	h := fnv.New64a()
 	h.Write([]byte(r.Expr))
@@ -111,16 +132,7 @@ func HashRule(r Rule) uint64 {
 		h.Write([]byte("alerting"))
 		h.Write([]byte(r.Alert))
 	}
-	type item struct {
-		key, value string
-	}
-	var kv []item
-	for k, v := range r.Labels {
-		kv = append(kv, item{key: k, value: v})
-	}
-	sort.Slice(kv, func(i, j int) bool {
-		return kv[i].key < kv[j].key
-	})
+	kv := sortMap(r.Labels)
 	for _, i := range kv {
 		h.Write([]byte(i.key))
 		h.Write([]byte(i.value))
@@ -170,7 +182,7 @@ func Parse(pathPatterns []string, validateAnnotations, validateExpressions bool)
 		}
 	}
 	if len(groups) < 1 {
-		return nil, fmt.Errorf("no groups found in %s", strings.Join(pathPatterns, ";"))
+		logger.Warnf("no groups found in %s", strings.Join(pathPatterns, ";"))
 	}
 	return groups, nil
 }
@@ -203,3 +215,18 @@ func checkOverflow(m map[string]interface{}, ctx string) error {
 	}
 	return nil
 }
+
+type item struct {
+	key, value string
+}
+
+func sortMap(m map[string]string) []item {
+	var kv []item
+	for k, v := range m {
+		kv = append(kv, item{key: k, value: v})
+	}
+	sort.Slice(kv, func(i, j int) bool {
+		return kv[i].key < kv[j].key
+	})
+	return kv
+}

app/vmalert/config/config_test.go

@@ -8,6 +8,7 @@ import (
 	"time"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/notifier"
+	"gopkg.in/yaml.v2"
 )
 
 func TestMain(m *testing.M) {
@@ -51,10 +52,6 @@ func TestParseBad(t *testing.T) {
 			[]string{"testdata/dir/rules4-bad.rules"},
 			"either `record` or `alert` must be set",
 		},
-		{
-			[]string{"testdata/*.yaml"},
-			"no groups found",
-		},
 	}
 	for _, tc := range testCases {
 		_, err := Parse(tc.path, true, true)
@@ -324,3 +321,36 @@ func TestHashRule(t *testing.T) {
 		}
 	}
 }
+
+func TestGroupChecksum(t *testing.T) {
+	data := `
+name: TestGroup
+rules:
+  - alert: ExampleAlertAlwaysFiring
+    expr: sum by(job) (up == 1)
+  - record: handler:requests:rate5m
+    expr: sum(rate(prometheus_http_requests_total[5m])) by (handler)
+`
+	var g Group
+	if err := yaml.Unmarshal([]byte(data), &g); err != nil {
+		t.Fatalf("failed to unmarshal: %s", err)
+	}
+	if g.Checksum == "" {
+		t.Fatalf("expected to get non-empty checksum")
+	}
+	newData := `
+name: TestGroup
+rules:
+  - record: handler:requests:rate5m
+    expr: sum(rate(prometheus_http_requests_total[5m])) by (handler)
+  - alert: ExampleAlertAlwaysFiring
+    expr: sum by(job) (up == 1)
+`
+	var ng Group
+	if err := yaml.Unmarshal([]byte(newData), &g); err != nil {
+		t.Fatalf("failed to unmarshal: %s", err)
+	}
+	if g.Checksum == ng.Checksum {
+		t.Fatalf("expected to get different checksums")
+	}
+}

app/vmalert/group.go

@@ -24,6 +24,7 @@ type Group struct {
 	Rules       []Rule
 	Interval    time.Duration
 	Concurrency int
+	Checksum    string
 
 	doneCh     chan struct{}
 	finishedCh chan struct{}
@@ -53,6 +54,7 @@ func newGroup(cfg config.Group, defaultInterval time.Duration, labels map[string
 		File:        cfg.File,
 		Interval:    cfg.Interval,
 		Concurrency: cfg.Concurrency,
+		Checksum:    cfg.Checksum,
 		doneCh:      make(chan struct{}),
 		finishedCh:  make(chan struct{}),
 		updateCh:    make(chan *Group),
@@ -156,6 +158,7 @@ func (g *Group) updateWith(newGroup *Group) error {
 		newRules = append(newRules, nr)
 	}
 	g.Concurrency = newGroup.Concurrency
+	g.Checksum = newGroup.Checksum
 	g.Rules = newRules
 	return nil
 }
@@ -180,8 +183,31 @@ func (g *Group) close() {
 	}
 }
 
+var skipRandSleepOnGroupStart bool
+
 func (g *Group) start(ctx context.Context, querier datasource.Querier, nts []notifier.Notifier, rw *remotewrite.Client) {
 	defer func() { close(g.finishedCh) }()
+
+	// Spread group rules evaluation over time in order to reduce load on VictoriaMetrics.
+	if !skipRandSleepOnGroupStart {
+		randSleep := uint64(float64(g.Interval) * (float64(uint32(g.ID())) / (1 << 32)))
+		sleepOffset := uint64(time.Now().UnixNano()) % uint64(g.Interval)
+		if randSleep < sleepOffset {
+			randSleep += uint64(g.Interval)
+		}
+		randSleep -= sleepOffset
+		sleepTimer := time.NewTimer(time.Duration(randSleep))
+		select {
+		case <-ctx.Done():
+			sleepTimer.Stop()
+			return
+		case <-g.doneCh:
+			sleepTimer.Stop()
+			return
+		case <-sleepTimer.C:
+		}
+	}
+
 	logger.Infof("group %q started; interval=%v; concurrency=%d", g.Name, g.Interval, g.Concurrency)
 	e := &executor{querier, nts, rw}
 	t := time.NewTicker(g.Interval)

app/vmalert/group_test.go

@@ -10,6 +10,12 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/notifier"
 )
 
+func init() {
+	// Disable rand sleep on group start during tests in order to speed up test execution.
+	// Rand sleep is needed only in prod code.
+	skipRandSleepOnGroupStart = true
+}
+
 func TestUpdateWith(t *testing.T) {
 	testCases := []struct {
 		name         string

app/vmalert/main.go

@@ -29,8 +29,8 @@ var (
 	rulePath = flagutil.NewArray("rule", `Path to the file with alert rules. 
 Supports patterns. Flag can be specified multiple times. 
 Examples:
- -rule /path/to/file. Path to a single file with alerting rules
- -rule dir/*.yaml -rule /*.yaml. Relative path to all .yaml files in "dir" folder, 
+ -rule="/path/to/file". Path to a single file with alerting rules
+ -rule="dir/*.yaml" -rule="/*.yaml". Relative path to all .yaml files in "dir" folder, 
 absolute path to all .yaml files in root.
 Rule files may contain %{ENV_VAR} placeholders, which are substituted by the corresponding env vars.`)
 

app/vmalert/manager.go

@@ -93,31 +93,53 @@ func (m *manager) update(ctx context.Context, path []string, validateTpl, valida
 		groupsRegistry[ng.ID()] = ng
 	}
 
+	type updateItem struct {
+		old *Group
+		new *Group
+	}
+	var toUpdate []updateItem
+
 	m.groupsMu.Lock()
 	for _, og := range m.groups {
 		ng, ok := groupsRegistry[og.ID()]
 		if !ok {
-			// old group is not present in new list
-			// and must be stopped and deleted
+			// old group is not present in new list,
+			// so must be stopped and deleted
 			og.close()
 			delete(m.groups, og.ID())
 			og = nil
 			continue
 		}
-		og.updateCh <- ng
 		delete(groupsRegistry, ng.ID())
+		if og.Checksum != ng.Checksum {
+			toUpdate = append(toUpdate, updateItem{old: og, new: ng})
+		}
 	}
-
 	for _, ng := range groupsRegistry {
 		m.startGroup(ctx, ng, restore)
 	}
 	m.groupsMu.Unlock()
+
+	if len(toUpdate) > 0 {
+		var wg sync.WaitGroup
+		for _, item := range toUpdate {
+			wg.Add(1)
+			go func(old *Group, new *Group) {
+				old.updateCh <- new
+				wg.Done()
+			}(item.old, item.new)
+		}
+		wg.Wait()
+	}
 	return nil
 }
 
 func (g *Group) toAPI() APIGroup {
+	g.mu.RLock()
+	defer g.mu.RUnlock()
+
 	ag := APIGroup{
-		// encode as strings to avoid rounding
+		// encode as string to avoid rounding
 		ID:          fmt.Sprintf("%d", g.ID()),
 		Name:        g.Name,
 		File:        g.File,

app/vmalert/manager_test.go

@@ -5,7 +5,6 @@ import (
 	"math/rand"
 	"net/url"
 	"os"
-	"strings"
 	"sync"
 	"testing"
 	"time"
@@ -19,16 +18,15 @@ func TestMain(m *testing.M) {
 	os.Exit(m.Run())
 }
 
-func TestManagerUpdateError(t *testing.T) {
+// TestManagerEmptyRulesDir tests
+// successful cases of
+// starting with empty rules folder
+func TestManagerEmptyRulesDir(t *testing.T) {
 	m := &manager{groups: make(map[uint64]*Group)}
 	path := []string{"foo/bar"}
 	err := m.update(context.Background(), path, true, true, false)
-	if err == nil {
-		t.Fatalf("expected to have err; got nil instead")
-	}
-	expErr := "no groups found"
-	if !strings.Contains(err.Error(), expErr) {
-		t.Fatalf("expected to got err %s; got %s", expErr, err)
+	if err != nil {
+		t.Fatalf("expected to load succesfully with empty rules dir; got err instead: %v", err)
 	}
 }
 
@@ -180,6 +178,27 @@ func TestManagerUpdate(t *testing.T) {
 					}},
 			},
 		},
+		{
+			name:       "update empty dir rules from 0 to 2 groups",
+			initPath:   "config/testdata/empty/*",
+			updatePath: "config/testdata/rules0-good.rules",
+			want: []*Group{
+				{
+					File:     "config/testdata/rules0-good.rules",
+					Name:     "groupGorSingleAlert",
+					Interval: defaultEvalInterval,
+					Rules:    []Rule{VMRows},
+				},
+				{
+					File:     "config/testdata/rules0-good.rules",
+					Interval: defaultEvalInterval,
+					Name:     "TestGroup", Rules: []Rule{
+						Conns,
+						ExampleAlertAlwaysFiring,
+					},
+				},
+			},
+		},
 	}
 	for _, tc := range testCases {
 		t.Run(tc.name, func(t *testing.T) {

app/vmauth/README.md

@@ -140,3 +140,68 @@ curl -s http://<vmauth-host>:8427/debug/pprof/profile > cpu.pprof
 The command for collecting CPU profile waits for 30 seconds before returning.
 
 The collected profiles may be analyzed with [go tool pprof](https://github.com/google/pprof).
+
+
+### Advanced usage
+
+Pass `-help` command-line arg to `vmauth` in order to see all the configuration options:
+
+```
+./vmauth -help
+
+vmauth authenticates and authorizes incoming requests and proxies them to VictoriaMetrics.
+
+See the docs at https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmauth/README.md .
+
+  -auth.config string
+    	Path to auth config. See https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmauth/README.md for details on the format of this auth config
+  -enableTCP6
+    	Whether to enable IPv6 for listening and dialing. By default only IPv4 TCP is used
+  -envflag.enable
+    	Whether to enable reading flags from environment variables additionally to command line. Command line flag values have priority over values from environment vars. Flags are read only from command line if this flag isn't set
+  -envflag.prefix string
+    	Prefix for environment variables if -envflag.enable is set
+  -http.connTimeout duration
+    	Incoming http connections are closed after the configured timeout. This may help spreading incoming load among a cluster of services behind load balancer. Note that the real timeout may be bigger by up to 10% as a protection from Thundering herd problem (default 2m0s)
+  -http.disableResponseCompression
+    	Disable compression of HTTP responses for saving CPU resources. By default compression is enabled to save network bandwidth
+  -http.idleConnTimeout duration
+    	Timeout for incoming idle http connections (default 1m0s)
+  -http.maxGracefulShutdownDuration duration
+    	The maximum duration for graceful shutdown of HTTP server. Highly loaded server may require increased value for graceful shutdown (default 7s)
+  -http.pathPrefix string
+    	An optional prefix to add to all the paths handled by http server. For example, if '-http.pathPrefix=/foo/bar' is set, then all the http requests will be handled on '/foo/bar/*' paths. This may be useful for proxied requests. See https://www.robustperception.io/using-external-urls-and-proxies-with-prometheus
+  -http.shutdownDelay duration
+    	Optional delay before http server shutdown. During this dealy the servier returns non-OK responses from /health page, so load balancers can route new requests to other servers
+  -httpAuth.password string
+    	Password for HTTP Basic Auth. The authentication is disabled if -httpAuth.username is empty
+  -httpAuth.username string
+    	Username for HTTP Basic Auth. The authentication is disabled if empty. See also -httpAuth.password
+  -httpListenAddr string
+    	TCP address to listen for http connections (default ":8427")
+  -loggerErrorsPerSecondLimit int
+    	Per-second limit on the number of ERROR messages. If more than the given number of errors are emitted per second, then the remaining errors are suppressed. Zero value disables the rate limit (default 10)
+  -loggerFormat string
+    	Format for logs. Possible values: default, json (default "default")
+  -loggerLevel string
+    	Minimum level of errors to log. Possible values: INFO, WARN, ERROR, FATAL, PANIC (default "INFO")
+  -loggerOutput string
+    	Output for the logs. Supported values: stderr, stdout (default "stderr")
+  -memory.allowedBytes value
+    	Allowed size of system memory VictoriaMetrics caches may occupy. This option overrides -memory.allowedPercent if set to non-zero value. Too low value may increase cache miss rate, which usually results in higher CPU and disk IO usage. Too high value may evict too much data from OS page cache, which will result in higher disk IO usage
+    	Supports the following optional suffixes for values: KB, MB, GB, KiB, MiB, GiB (default 0)
+  -memory.allowedPercent float
+    	Allowed percent of system memory VictoriaMetrics caches may occupy. See also -memory.allowedBytes. Too low value may increase cache miss rate, which usually results in higher CPU and disk IO usage. Too high value may evict too much data from OS page cache, which will result in higher disk IO usage (default 60)
+  -metricsAuthKey string
+    	Auth key for /metrics. It overrides httpAuth settings
+  -pprofAuthKey string
+    	Auth key for /debug/pprof. It overrides httpAuth settings
+  -tls
+    	Whether to enable TLS (aka HTTPS) for incoming requests. -tlsCertFile and -tlsKeyFile must be set if -tls is set
+  -tlsCertFile string
+    	Path to file with TLS certificate. Used only if -tls is set. Prefer ECDSA certs instead of RSA certs, since RSA certs are slow
+  -tlsKeyFile string
+    	Path to file with TLS key. Used only if -tls is set
+  -version
+    	Show VictoriaMetrics version
+```

app/vmbackup/README.md

@@ -138,7 +138,7 @@ Run `vmbackup -help` in order to see all the available options:
     	Path to file with S3 configs. Configs are loaded from default location if not set.
     	See https://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html
   -configProfile string
-    	Profile name for S3 configs (default "default")
+    	Profile name for S3 configs. If no set, the value of the environment variable will be loaded (AWS_PROFILE or AWS_DEFAULT_PROFILE), or if both not set, DefaultSharedConfigProfile is used
   -credsFilePath string
     	Path to file with GCS or S3 credentials. Credentials are loaded from default locations if not set.
     	See https://cloud.google.com/iam/docs/creating-managing-service-account-keys and https://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html
@@ -161,18 +161,20 @@ Run `vmbackup -help` in order to see all the available options:
     	Minimum level of errors to log. Possible values: INFO, WARN, ERROR, FATAL, PANIC (default "INFO")
   -loggerOutput string
     	Output for the logs. Supported values: stderr, stdout (default "stderr")
-  -maxBytesPerSecond int
+  -maxBytesPerSecond value
     	The maximum upload speed. There is no limit if it is set to 0
-  -memory.allowedBytes int
+    	Supports the following optional suffixes for values: KB, MB, GB, KiB, MiB, GiB (default 0)
+  -memory.allowedBytes value
     	Allowed size of system memory VictoriaMetrics caches may occupy. This option overrides -memory.allowedPercent if set to non-zero value. Too low value may increase cache miss rate, which usually results in higher CPU and disk IO usage. Too high value may evict too much data from OS page cache, which will result in higher disk IO usage
+    	Supports the following optional suffixes for values: KB, MB, GB, KiB, MiB, GiB (default 0)
   -memory.allowedPercent float
     	Allowed percent of system memory VictoriaMetrics caches may occupy. See also -memory.allowedBytes. Too low value may increase cache miss rate, which usually results in higher CPU and disk IO usage. Too high value may evict too much data from OS page cache, which will result in higher disk IO usage (default 60)
   -origin string
     	Optional origin directory on the remote storage with old backup for server-side copying when performing full backup. This speeds up full backups
   -snapshot.createURL string
-    	VictoriaMetrics create snapshot url. When this is given a snapshot will automatically be created during backup.Example: http://victoriametrics:8428/snaphsot/create
+    	VictoriaMetrics create snapshot url. When this is given a snapshot will automatically be created during backup. Example: http://victoriametrics:8428/snaphsot/create
   -snapshot.deleteURL string
-    	VictoriaMetrics delete snapshot url. Optional. Will be generated from snapshotCreateURL if not provided. All created snaphosts will be automatically deleted.Example: http://victoriametrics:8428/snaphsot/delete
+    	VictoriaMetrics delete snapshot url. Optional. Will be generated from -snapshot.createURL if not provided. All created snaphosts will be automatically deleted. Example: http://victoriametrics:8428/snaphsot/delete
   -snapshotName string
     	Name for the snapshot to backup. See https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/README.md#how-to-work-with-snapshots
   -storageDataPath string

app/vmbackup/main.go

@@ -13,22 +13,23 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/buildinfo"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/cgroup"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/envflag"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 )
 
 var (
 	storageDataPath   = flag.String("storageDataPath", "victoria-metrics-data", "Path to VictoriaMetrics data. Must match -storageDataPath from VictoriaMetrics or vmstorage")
 	snapshotName      = flag.String("snapshotName", "", "Name for the snapshot to backup. See https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/README.md#how-to-work-with-snapshots")
-	snapshotCreateURL = flag.String("snapshot.createURL", "", "VictoriaMetrics create snapshot url. When this is given a snapshot will automatically be created during backup."+
+	snapshotCreateURL = flag.String("snapshot.createURL", "", "VictoriaMetrics create snapshot url. When this is given a snapshot will automatically be created during backup. "+
 		"Example: http://victoriametrics:8428/snaphsot/create")
-	snapshotDeleteURL = flag.String("snapshot.deleteURL", "", "VictoriaMetrics delete snapshot url. Optional. Will be generated from snapshotCreateURL if not provided. All created snaphosts will be automatically deleted."+
-		"Example: http://victoriametrics:8428/snaphsot/delete")
+	snapshotDeleteURL = flag.String("snapshot.deleteURL", "", "VictoriaMetrics delete snapshot url. Optional. Will be generated from -snapshot.createURL if not provided. "+
+		"All created snaphosts will be automatically deleted. Example: http://victoriametrics:8428/snaphsot/delete")
 	dst = flag.String("dst", "", "Where to put the backup on the remote storage. "+
 		"Example: gcs://bucket/path/to/backup/dir, s3://bucket/path/to/backup/dir or fs:///path/to/local/backup/dir\n"+
 		"-dst can point to the previous backup. In this case incremental backup is performed, i.e. only changed data is uploaded")
 	origin            = flag.String("origin", "", "Optional origin directory on the remote storage with old backup for server-side copying when performing full backup. This speeds up full backups")
 	concurrency       = flag.Int("concurrency", 10, "The number of concurrent workers. Higher concurrency may reduce backup duration")
-	maxBytesPerSecond = flag.Int("maxBytesPerSecond", 0, "The maximum upload speed. There is no limit if it is set to 0")
+	maxBytesPerSecond = flagutil.NewBytes("maxBytesPerSecond", 0, "The maximum upload speed. There is no limit if it is set to 0")
 )
 
 func main() {
@@ -126,7 +127,7 @@ func newSrcFS() (*fslocal.FS, error) {
 
 	fs := &fslocal.FS{
 		Dir:               snapshotPath,
-		MaxBytesPerSecond: *maxBytesPerSecond,
+		MaxBytesPerSecond: maxBytesPerSecond.N,
 	}
 	if err := fs.Init(); err != nil {
 		return nil, fmt.Errorf("cannot initialize fs: %w", err)

app/vminsert/csvimport/request_handler.go

@@ -5,6 +5,8 @@ import (
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vminsert/common"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vminsert/relabel"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
+	parserCommon "github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/common"
 	parser "github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/csvimport"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/writeconcurrencylimiter"
 	"github.com/VictoriaMetrics/metrics"
@@ -17,14 +19,18 @@ var (
 
 // InsertHandler processes /api/v1/import/csv requests.
 func InsertHandler(req *http.Request) error {
+	extraLabels, err := parserCommon.GetExtraLabels(req)
+	if err != nil {
+		return err
+	}
 	return writeconcurrencylimiter.Do(func() error {
 		return parser.ParseStream(req, func(rows []parser.Row) error {
-			return insertRows(rows)
+			return insertRows(rows, extraLabels)
 		})
 	})
 }
 
-func insertRows(rows []parser.Row) error {
+func insertRows(rows []parser.Row, extraLabels []prompbmarshal.Label) error {
 	ctx := common.GetInsertCtx()
 	defer common.PutInsertCtx(ctx)
 
@@ -38,6 +44,10 @@ func insertRows(rows []parser.Row) error {
 			tag := &r.Tags[j]
 			ctx.AddLabel(tag.Key, tag.Value)
 		}
+		for j := range extraLabels {
+			label := &extraLabels[j]
+			ctx.AddLabel(label.Name, label.Value)
+		}
 		if hasRelabeling {
 			ctx.ApplyRelabeling()
 		}

app/vminsert/prometheusimport/request_handler.go

@@ -5,6 +5,8 @@ import (
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vminsert/common"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vminsert/relabel"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
+	parserCommon "github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/common"
 	parser "github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/prometheus"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/writeconcurrencylimiter"
 	"github.com/VictoriaMetrics/metrics"
@@ -17,13 +19,23 @@ var (
 
 // InsertHandler processes `/api/v1/import/prometheus` request.
 func InsertHandler(req *http.Request) error {
+	extraLabels, err := parserCommon.GetExtraLabels(req)
+	if err != nil {
+		return err
+	}
+	defaultTimestamp, err := parserCommon.GetTimestamp(req)
+	if err != nil {
+		return err
+	}
 	return writeconcurrencylimiter.Do(func() error {
 		isGzipped := req.Header.Get("Content-Encoding") == "gzip"
-		return parser.ParseStream(req.Body, isGzipped, insertRows)
+		return parser.ParseStream(req.Body, defaultTimestamp, isGzipped, func(rows []parser.Row) error {
+			return insertRows(rows, extraLabels)
+		})
 	})
 }
 
-func insertRows(rows []parser.Row) error {
+func insertRows(rows []parser.Row, extraLabels []prompbmarshal.Label) error {
 	ctx := common.GetInsertCtx()
 	defer common.PutInsertCtx(ctx)
 
@@ -37,6 +49,10 @@ func insertRows(rows []parser.Row) error {
 			tag := &r.Tags[j]
 			ctx.AddLabel(tag.Key, tag.Value)
 		}
+		for j := range extraLabels {
+			label := &extraLabels[j]
+			ctx.AddLabel(label.Name, label.Value)
+		}
 		if hasRelabeling {
 			ctx.ApplyRelabeling()
 		}

app/vminsert/vmimport/request_handler.go

@@ -7,6 +7,8 @@ import (
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vminsert/common"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vminsert/relabel"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
+	parserCommon "github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/common"
 	parser "github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/vmimport"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/storage"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/writeconcurrencylimiter"
@@ -22,12 +24,18 @@ var (
 //
 // See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6
 func InsertHandler(req *http.Request) error {
+	extraLabels, err := parserCommon.GetExtraLabels(req)
+	if err != nil {
+		return err
+	}
 	return writeconcurrencylimiter.Do(func() error {
-		return parser.ParseStream(req, insertRows)
+		return parser.ParseStream(req, func(rows []parser.Row) error {
+			return insertRows(rows, extraLabels)
+		})
 	})
 }
 
-func insertRows(rows []parser.Row) error {
+func insertRows(rows []parser.Row, extraLabels []prompbmarshal.Label) error {
 	ctx := getPushCtx()
 	defer putPushCtx(ctx)
 
@@ -46,6 +54,10 @@ func insertRows(rows []parser.Row) error {
 			tag := &r.Tags[j]
 			ic.AddLabelBytes(tag.Key, tag.Value)
 		}
+		for j := range extraLabels {
+			label := &extraLabels[j]
+			ic.AddLabel(label.Name, label.Value)
+		}
 		if hasRelabeling {
 			ic.ApplyRelabeling()
 		}

app/vmrestore/README.md

@@ -42,7 +42,7 @@ Run `vmrestore -help` in order to see all the available options:
     	Path to file with S3 configs. Configs are loaded from default location if not set.
     	See https://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html
   -configProfile string
-    	Profile name for S3 configs (default "default")
+    	Profile name for S3 configs. If no set, the value of the environment variable will be loaded (AWS_PROFILE or AWS_DEFAULT_PROFILE), or if both not set, DefaultSharedConfigProfile is used
   -credsFilePath string
     	Path to file with GCS or S3 credentials. Credentials are loaded from default locations if not set.
     	See https://cloud.google.com/iam/docs/creating-managing-service-account-keys and https://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html
@@ -62,10 +62,12 @@ Run `vmrestore -help` in order to see all the available options:
     	Minimum level of errors to log. Possible values: INFO, WARN, ERROR, FATAL, PANIC (default "INFO")
   -loggerOutput string
     	Output for the logs. Supported values: stderr, stdout (default "stderr")
-  -maxBytesPerSecond int
+  -maxBytesPerSecond value
     	The maximum download speed. There is no limit if it is set to 0
-  -memory.allowedBytes int
+    	Supports the following optional suffixes for values: KB, MB, GB, KiB, MiB, GiB (default 0)
+  -memory.allowedBytes value
     	Allowed size of system memory VictoriaMetrics caches may occupy. This option overrides -memory.allowedPercent if set to non-zero value. Too low value may increase cache miss rate, which usually results in higher CPU and disk IO usage. Too high value may evict too much data from OS page cache, which will result in higher disk IO usage
+    	Supports the following optional suffixes for values: KB, MB, GB, KiB, MiB, GiB (default 0)
   -memory.allowedPercent float
     	Allowed percent of system memory VictoriaMetrics caches may occupy. See also -memory.allowedBytes. Too low value may increase cache miss rate, which usually results in higher CPU and disk IO usage. Too high value may evict too much data from OS page cache, which will result in higher disk IO usage (default 60)
   -skipBackupCompleteCheck

app/vmrestore/main.go

@@ -11,6 +11,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/buildinfo"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/cgroup"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/envflag"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 )
 
@@ -21,7 +22,7 @@ var (
 		"VictoriaMetrics must be stopped when restoring from backup. -storageDataPath dir can be non-empty. In this case the contents of -storageDataPath dir "+
 		"is synchronized with -src contents, i.e. it works like 'rsync --delete'")
 	concurrency             = flag.Int("concurrency", 10, "The number of concurrent workers. Higher concurrency may reduce restore duration")
-	maxBytesPerSecond       = flag.Int("maxBytesPerSecond", 0, "The maximum download speed. There is no limit if it is set to 0")
+	maxBytesPerSecond       = flagutil.NewBytes("maxBytesPerSecond", 0, "The maximum download speed. There is no limit if it is set to 0")
 	skipBackupCompleteCheck = flag.Bool("skipBackupCompleteCheck", false, "Whether to skip checking for 'backup complete' file in -src. This may be useful for restoring from old backups, which were created without 'backup complete' file")
 )
 
@@ -71,7 +72,7 @@ func newDstFS() (*fslocal.FS, error) {
 	}
 	fs := &fslocal.FS{
 		Dir:               *storageDataPath,
-		MaxBytesPerSecond: *maxBytesPerSecond,
+		MaxBytesPerSecond: maxBytesPerSecond.N,
 	}
 	if err := fs.Init(); err != nil {
 		return nil, fmt.Errorf("cannot initialize local fs: %w", err)

app/vmselect/graphite/graphite.go

@@ -0,0 +1,383 @@
+package graphite
+
+import (
+	"fmt"
+	"net/http"
+	"regexp"
+	"sort"
+	"strings"
+	"sync"
+	"time"
+
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/netstorage"
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/searchutils"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/storage"
+	"github.com/VictoriaMetrics/metrics"
+)
+
+// MetricsFindHandler implements /metrics/find handler.
+//
+// See https://graphite-api.readthedocs.io/en/latest/api.html#metrics-find
+func MetricsFindHandler(startTime time.Time, w http.ResponseWriter, r *http.Request) error {
+	deadline := searchutils.GetDeadlineForQuery(r, startTime)
+	if err := r.ParseForm(); err != nil {
+		return fmt.Errorf("cannot parse form values: %w", err)
+	}
+	format := r.FormValue("format")
+	if format == "" {
+		format = "treejson"
+	}
+	switch format {
+	case "treejson", "completer":
+	default:
+		return fmt.Errorf(`unexpected "format" query arg: %q; expecting "treejson" or "completer"`, format)
+	}
+	query := r.FormValue("query")
+	if len(query) == 0 {
+		return fmt.Errorf("expecting non-empty `query` arg")
+	}
+	delimiter := r.FormValue("delimiter")
+	if delimiter == "" {
+		delimiter = "."
+	}
+	if len(delimiter) > 1 {
+		return fmt.Errorf("`delimiter` query arg must contain only a single char")
+	}
+	if searchutils.GetBool(r, "automatic_variants") {
+		// See https://github.com/graphite-project/graphite-web/blob/bb9feb0e6815faa73f538af6ed35adea0fb273fd/webapp/graphite/metrics/views.py#L152
+		query = addAutomaticVariants(query, delimiter)
+	}
+	if format == "completer" {
+		// See https://github.com/graphite-project/graphite-web/blob/bb9feb0e6815faa73f538af6ed35adea0fb273fd/webapp/graphite/metrics/views.py#L148
+		query = strings.ReplaceAll(query, "..", ".*")
+		if !strings.HasSuffix(query, "*") {
+			query += "*"
+		}
+	}
+	leavesOnly := searchutils.GetBool(r, "leavesOnly")
+	wildcards := searchutils.GetBool(r, "wildcards")
+	label := r.FormValue("label")
+	if label == "__name__" {
+		label = ""
+	}
+	jsonp := r.FormValue("jsonp")
+	from, err := searchutils.GetTime(r, "from", 0)
+	if err != nil {
+		return err
+	}
+	ct := startTime.UnixNano() / 1e6
+	until, err := searchutils.GetTime(r, "until", ct)
+	if err != nil {
+		return err
+	}
+	tr := storage.TimeRange{
+		MinTimestamp: from,
+		MaxTimestamp: until,
+	}
+	paths, err := metricsFind(tr, label, query, delimiter[0], deadline)
+	if err != nil {
+		return err
+	}
+	if leavesOnly {
+		paths = filterLeaves(paths, delimiter)
+	}
+	sortPaths(paths, delimiter)
+	contentType := "application/json"
+	if jsonp != "" {
+		contentType = "text/javascript"
+	}
+	w.Header().Set("Content-Type", contentType)
+	WriteMetricsFindResponse(w, paths, delimiter, format, wildcards, jsonp)
+	metricsFindDuration.UpdateDuration(startTime)
+	return nil
+}
+
+// MetricsExpandHandler implements /metrics/expand handler.
+//
+// See https://graphite-api.readthedocs.io/en/latest/api.html#metrics-expand
+func MetricsExpandHandler(startTime time.Time, w http.ResponseWriter, r *http.Request) error {
+	deadline := searchutils.GetDeadlineForQuery(r, startTime)
+	if err := r.ParseForm(); err != nil {
+		return fmt.Errorf("cannot parse form values: %w", err)
+	}
+	queries := r.Form["query"]
+	if len(queries) == 0 {
+		return fmt.Errorf("missing `query` arg")
+	}
+	groupByExpr := searchutils.GetBool(r, "groupByExpr")
+	leavesOnly := searchutils.GetBool(r, "leavesOnly")
+	label := r.FormValue("label")
+	if label == "__name__" {
+		label = ""
+	}
+	delimiter := r.FormValue("delimiter")
+	if delimiter == "" {
+		delimiter = "."
+	}
+	if len(delimiter) > 1 {
+		return fmt.Errorf("`delimiter` query arg must contain only a single char")
+	}
+	jsonp := r.FormValue("jsonp")
+	from, err := searchutils.GetTime(r, "from", 0)
+	if err != nil {
+		return err
+	}
+	ct := startTime.UnixNano() / 1e6
+	until, err := searchutils.GetTime(r, "until", ct)
+	if err != nil {
+		return err
+	}
+	tr := storage.TimeRange{
+		MinTimestamp: from,
+		MaxTimestamp: until,
+	}
+	m := make(map[string][]string, len(queries))
+	for _, query := range queries {
+		paths, err := metricsFind(tr, label, query, delimiter[0], deadline)
+		if err != nil {
+			return err
+		}
+		if leavesOnly {
+			paths = filterLeaves(paths, delimiter)
+		}
+		m[query] = paths
+	}
+	contentType := "application/json"
+	if jsonp != "" {
+		contentType = "text/javascript"
+	}
+	w.Header().Set("Content-Type", contentType)
+	if groupByExpr {
+		for _, paths := range m {
+			sortPaths(paths, delimiter)
+		}
+		WriteMetricsExpandResponseByQuery(w, m, jsonp)
+		return nil
+	}
+	paths := m[queries[0]]
+	if len(m) > 1 {
+		pathsSet := make(map[string]struct{})
+		for _, paths := range m {
+			for _, path := range paths {
+				pathsSet[path] = struct{}{}
+			}
+		}
+		paths = make([]string, 0, len(pathsSet))
+		for path := range pathsSet {
+			paths = append(paths, path)
+		}
+	}
+	sortPaths(paths, delimiter)
+	WriteMetricsExpandResponseFlat(w, paths, jsonp)
+	metricsExpandDuration.UpdateDuration(startTime)
+	return nil
+}
+
+// MetricsIndexHandler implements /metrics/index.json handler.
+//
+// See https://graphite-api.readthedocs.io/en/latest/api.html#metrics-index-json
+func MetricsIndexHandler(startTime time.Time, w http.ResponseWriter, r *http.Request) error {
+	deadline := searchutils.GetDeadlineForQuery(r, startTime)
+	if err := r.ParseForm(); err != nil {
+		return fmt.Errorf("cannot parse form values: %w", err)
+	}
+	jsonp := r.FormValue("jsonp")
+	metricNames, err := netstorage.GetLabelValues("__name__", deadline)
+	if err != nil {
+		return fmt.Errorf(`cannot obtain metric names: %w`, err)
+	}
+	contentType := "application/json"
+	if jsonp != "" {
+		contentType = "text/javascript"
+	}
+	w.Header().Set("Content-Type", contentType)
+	WriteMetricsIndexResponse(w, metricNames, jsonp)
+	metricsIndexDuration.UpdateDuration(startTime)
+	return nil
+}
+
+// metricsFind searches for label values that match the given query.
+func metricsFind(tr storage.TimeRange, label, query string, delimiter byte, deadline searchutils.Deadline) ([]string, error) {
+	expandTail := strings.HasSuffix(query, "*")
+	for strings.HasSuffix(query, "*") {
+		query = query[:len(query)-1]
+	}
+	var results []string
+	n := strings.IndexAny(query, "*{[")
+	if n < 0 {
+		suffixes, err := netstorage.GetTagValueSuffixes(tr, label, query, delimiter, deadline)
+		if err != nil {
+			return nil, err
+		}
+		if expandTail {
+			for _, suffix := range suffixes {
+				results = append(results, query+suffix)
+			}
+		} else if isFullMatch(query, suffixes, delimiter) {
+			results = append(results, query)
+		}
+		return results, nil
+	}
+	subquery := query[:n] + "*"
+	paths, err := metricsFind(tr, label, subquery, delimiter, deadline)
+	if err != nil {
+		return nil, err
+	}
+	tail := ""
+	suffix := query[n:]
+	if m := strings.IndexByte(suffix, delimiter); m >= 0 {
+		tail = suffix[m+1:]
+		suffix = suffix[:m+1]
+	}
+	q := query[:n] + suffix
+	re, err := getRegexpForQuery(q, delimiter)
+	if err != nil {
+		return nil, fmt.Errorf("cannot convert query %q to regexp: %w", q, err)
+	}
+	if expandTail {
+		tail += "*"
+	}
+	for _, path := range paths {
+		if !re.MatchString(path) {
+			continue
+		}
+		subquery := path + tail
+		tmp, err := metricsFind(tr, label, subquery, delimiter, deadline)
+		if err != nil {
+			return nil, err
+		}
+		results = append(results, tmp...)
+	}
+	return results, nil
+}
+
+var (
+	metricsFindDuration   = metrics.NewSummary(`vm_request_duration_seconds{path="/metrics/find"}`)
+	metricsExpandDuration = metrics.NewSummary(`vm_request_duration_seconds{path="/metrics/expand"}`)
+	metricsIndexDuration  = metrics.NewSummary(`vm_request_duration_seconds{path="/metrics/index.json"}`)
+)
+
+func isFullMatch(tagValuePrefix string, suffixes []string, delimiter byte) bool {
+	if len(suffixes) == 0 {
+		return false
+	}
+	if strings.LastIndexByte(tagValuePrefix, delimiter) == len(tagValuePrefix)-1 {
+		return true
+	}
+	for _, suffix := range suffixes {
+		if suffix == "" {
+			return true
+		}
+	}
+	return false
+}
+
+func addAutomaticVariants(query, delimiter string) string {
+	// See https://github.com/graphite-project/graphite-web/blob/bb9feb0e6815faa73f538af6ed35adea0fb273fd/webapp/graphite/metrics/views.py#L152
+	parts := strings.Split(query, delimiter)
+	for i, part := range parts {
+		if strings.Contains(part, ",") && !strings.Contains(part, "{") {
+			parts[i] = "{" + part + "}"
+		}
+	}
+	return strings.Join(parts, delimiter)
+}
+
+func filterLeaves(paths []string, delimiter string) []string {
+	leaves := paths[:0]
+	for _, path := range paths {
+		if !strings.HasSuffix(path, delimiter) {
+			leaves = append(leaves, path)
+		}
+	}
+	return leaves
+}
+
+func sortPaths(paths []string, delimiter string) {
+	sort.Slice(paths, func(i, j int) bool {
+		a, b := paths[i], paths[j]
+		isNodeA := strings.HasSuffix(a, delimiter)
+		isNodeB := strings.HasSuffix(b, delimiter)
+		if isNodeA == isNodeB {
+			return a < b
+		}
+		return isNodeA
+	})
+}
+
+func getRegexpForQuery(query string, delimiter byte) (*regexp.Regexp, error) {
+	regexpCacheLock.Lock()
+	defer regexpCacheLock.Unlock()
+
+	k := regexpCacheKey{
+		query:     query,
+		delimiter: delimiter,
+	}
+	if re := regexpCache[k]; re != nil {
+		return re.re, re.err
+	}
+	a := make([]string, 0, len(query))
+	tillNextDelimiter := "[^" + regexp.QuoteMeta(string([]byte{delimiter})) + "]*"
+	for i := 0; i < len(query); i++ {
+		switch query[i] {
+		case '*':
+			a = append(a, tillNextDelimiter)
+		case '{':
+			tmp := query[i+1:]
+			if n := strings.IndexByte(tmp, '}'); n < 0 {
+				a = append(a, regexp.QuoteMeta(query[i:]))
+				i = len(query)
+			} else {
+				a = append(a, "(?:")
+				opts := strings.Split(tmp[:n], ",")
+				for j, opt := range opts {
+					opts[j] = regexp.QuoteMeta(opt)
+				}
+				a = append(a, strings.Join(opts, "|"))
+				a = append(a, ")")
+				i += n + 1
+			}
+		case '[':
+			tmp := query[i:]
+			if n := strings.IndexByte(tmp, ']'); n < 0 {
+				a = append(a, regexp.QuoteMeta(query[i:]))
+				i = len(query)
+			} else {
+				a = append(a, tmp[:n+1])
+				i += n
+			}
+		default:
+			a = append(a, regexp.QuoteMeta(query[i:i+1]))
+		}
+	}
+	s := strings.Join(a, "")
+	re, err := regexp.Compile(s)
+	regexpCache[k] = &regexpCacheEntry{
+		re:  re,
+		err: err,
+	}
+	if len(regexpCache) >= maxRegexpCacheSize {
+		for k := range regexpCache {
+			if len(regexpCache) < maxRegexpCacheSize {
+				break
+			}
+			delete(regexpCache, k)
+		}
+	}
+	return re, err
+}
+
+type regexpCacheEntry struct {
+	re  *regexp.Regexp
+	err error
+}
+
+type regexpCacheKey struct {
+	query     string
+	delimiter byte
+}
+
+var regexpCache = make(map[regexpCacheKey]*regexpCacheEntry)
+var regexpCacheLock sync.Mutex
+
+const maxRegexpCacheSize = 10000

app/vmselect/graphite/graphite_test.go

@@ -0,0 +1,71 @@
+package graphite
+
+import (
+	"reflect"
+	"testing"
+)
+
+func TestGetRegexpForQuery(t *testing.T) {
+	f := func(query string, delimiter byte, reExpected string) {
+		t.Helper()
+		re, err := getRegexpForQuery(query, delimiter)
+		if err != nil {
+			t.Fatalf("unexpected error in getRegexpForQuery(%q): %s", query, err)
+		}
+		reStr := re.String()
+		if reStr != reExpected {
+			t.Fatalf("unexpected regexp for query=%q, delimiter=%c; got %s; want %s", query, delimiter, reStr, reExpected)
+		}
+	}
+	f("", '.', "")
+	f("foobar", '.', "foobar")
+	f("*", '.', `[^\.]*`)
+	f("*", '_', `[^_]*`)
+	f("foo.*.bar", '.', `foo\.[^\.]*\.bar`)
+	f("fo*b{ar,aaa}[a-z]xx*.d", '.', `fo[^\.]*b(?:ar|aaa)[a-z]xx[^\.]*\.d`)
+	f("fo*b{ar,aaa}[a-z]xx*_d", '_', `fo[^_]*b(?:ar|aaa)[a-z]xx[^_]*_d`)
+}
+
+func TestSortPaths(t *testing.T) {
+	f := func(paths []string, delimiter string, pathsSortedExpected []string) {
+		t.Helper()
+		sortPaths(paths, delimiter)
+		if !reflect.DeepEqual(paths, pathsSortedExpected) {
+			t.Fatalf("unexpected sortPaths result;\ngot\n%q\nwant\n%q", paths, pathsSortedExpected)
+		}
+	}
+	f([]string{"foo", "bar"}, ".", []string{"bar", "foo"})
+	f([]string{"foo.", "bar", "aa", "ab."}, ".", []string{"ab.", "foo.", "aa", "bar"})
+	f([]string{"foo.", "bar", "aa", "ab."}, "_", []string{"aa", "ab.", "bar", "foo."})
+}
+
+func TestFilterLeaves(t *testing.T) {
+	f := func(paths []string, delimiter string, leavesExpected []string) {
+		t.Helper()
+		leaves := filterLeaves(paths, delimiter)
+		if !reflect.DeepEqual(leaves, leavesExpected) {
+			t.Fatalf("unexpected leaves; got\n%q\nwant\n%q", leaves, leavesExpected)
+		}
+	}
+	f([]string{"foo", "bar"}, ".", []string{"foo", "bar"})
+	f([]string{"a.", ".", "bc"}, ".", []string{"bc"})
+	f([]string{"a.", ".", "bc"}, "_", []string{"a.", ".", "bc"})
+	f([]string{"a_", "_", "bc"}, "_", []string{"bc"})
+	f([]string{"foo.", "bar."}, ".", []string{})
+}
+
+func TestAddAutomaticVariants(t *testing.T) {
+	f := func(query, delimiter, resultExpected string) {
+		t.Helper()
+		result := addAutomaticVariants(query, delimiter)
+		if result != resultExpected {
+			t.Fatalf("unexpected result for addAutomaticVariants(%q, delimiter=%q); got %q; want %q", query, delimiter, result, resultExpected)
+		}
+	}
+	f("", ".", "")
+	f("foobar", ".", "foobar")
+	f("foo,bar.baz", ".", "{foo,bar}.baz")
+	f("foo,bar.baz", "_", "{foo,bar.baz}")
+	f("foo,bar_baz*", "_", "{foo,bar}_baz*")
+	f("foo.bar,baz,aa.bb,cc", ".", "foo.{bar,baz,aa}.{bb,cc}")
+}

app/vmselect/graphite/metrics_expand_response.qtpl

@@ -0,0 +1,38 @@
+{% stripspace %}
+
+MetricsExpandResponseByQuery generates response for /metrics/expand?groupByExpr=1 .
+See https://graphite-api.readthedocs.io/en/latest/api.html#metrics-expand
+{% func MetricsExpandResponseByQuery(m map[string][]string, jsonp string) %}
+	{% if jsonp != "" %}{%s= jsonp %}({% endif %}
+	{
+		"results":{
+			{% code i := 0 %}
+			{% for query, paths := range m %}
+				{%q= query %}:{%= metricPaths(paths) %}
+				{% code i++ %}
+				{% if i < len(m) %},{% endif %}
+			{% endfor %}
+		}
+	}
+	{% if jsonp != "" %}){% endif %}
+{% endfunc %}
+
+
+MetricsExpandResponseFlat generates response for /metrics/expand?groupByExpr=0 .
+See https://graphite-api.readthedocs.io/en/latest/api.html#metrics-expand
+{% func MetricsExpandResponseFlat(paths []string, jsonp string) %}
+	{% if jsonp != "" %}{%s= jsonp %}({% endif %}
+	{%= metricPaths(paths) %}
+	{% if jsonp != "" %}){% endif %}
+{% endfunc %}
+
+{% func metricPaths(paths []string) %}
+	[
+		{% for i, path := range paths %}
+			{%q= path %}
+			{% if i+1 < len(paths) %},{% endif %}
+		{% endfor %}
+	]
+{% endfunc %}
+
+{% endstripspace %}

app/vmselect/graphite/metrics_expand_response.qtpl.go

@@ -0,0 +1,187 @@
+// Code generated by qtc from "metrics_expand_response.qtpl". DO NOT EDIT.
+// See https://github.com/valyala/quicktemplate for details.
+
+// MetricsExpandResponseByQuery generates response for /metrics/expand?groupByExpr=1 .See https://graphite-api.readthedocs.io/en/latest/api.html#metrics-expand
+
+//line app/vmselect/graphite/metrics_expand_response.qtpl:5
+package graphite
+
+//line app/vmselect/graphite/metrics_expand_response.qtpl:5
+import (
+	qtio422016 "io"
+
+	qt422016 "github.com/valyala/quicktemplate"
+)
+
+//line app/vmselect/graphite/metrics_expand_response.qtpl:5
+var (
+	_ = qtio422016.Copy
+	_ = qt422016.AcquireByteBuffer
+)
+
+//line app/vmselect/graphite/metrics_expand_response.qtpl:5
+func StreamMetricsExpandResponseByQuery(qw422016 *qt422016.Writer, m map[string][]string, jsonp string) {
+//line app/vmselect/graphite/metrics_expand_response.qtpl:6
+	if jsonp != "" {
+//line app/vmselect/graphite/metrics_expand_response.qtpl:6
+		qw422016.N().S(jsonp)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:6
+		qw422016.N().S(`(`)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:6
+	}
+//line app/vmselect/graphite/metrics_expand_response.qtpl:6
+	qw422016.N().S(`{"results":{`)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:9
+	i := 0
+
+//line app/vmselect/graphite/metrics_expand_response.qtpl:10
+	for query, paths := range m {
+//line app/vmselect/graphite/metrics_expand_response.qtpl:11
+		qw422016.N().Q(query)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:11
+		qw422016.N().S(`:`)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:11
+		streammetricPaths(qw422016, paths)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:12
+		i++
+
+//line app/vmselect/graphite/metrics_expand_response.qtpl:13
+		if i < len(m) {
+//line app/vmselect/graphite/metrics_expand_response.qtpl:13
+			qw422016.N().S(`,`)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:13
+		}
+//line app/vmselect/graphite/metrics_expand_response.qtpl:14
+	}
+//line app/vmselect/graphite/metrics_expand_response.qtpl:14
+	qw422016.N().S(`}}`)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:17
+	if jsonp != "" {
+//line app/vmselect/graphite/metrics_expand_response.qtpl:17
+		qw422016.N().S(`)`)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:17
+	}
+//line app/vmselect/graphite/metrics_expand_response.qtpl:18
+}
+
+//line app/vmselect/graphite/metrics_expand_response.qtpl:18
+func WriteMetricsExpandResponseByQuery(qq422016 qtio422016.Writer, m map[string][]string, jsonp string) {
+//line app/vmselect/graphite/metrics_expand_response.qtpl:18
+	qw422016 := qt422016.AcquireWriter(qq422016)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:18
+	StreamMetricsExpandResponseByQuery(qw422016, m, jsonp)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:18
+	qt422016.ReleaseWriter(qw422016)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:18
+}
+
+//line app/vmselect/graphite/metrics_expand_response.qtpl:18
+func MetricsExpandResponseByQuery(m map[string][]string, jsonp string) string {
+//line app/vmselect/graphite/metrics_expand_response.qtpl:18
+	qb422016 := qt422016.AcquireByteBuffer()
+//line app/vmselect/graphite/metrics_expand_response.qtpl:18
+	WriteMetricsExpandResponseByQuery(qb422016, m, jsonp)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:18
+	qs422016 := string(qb422016.B)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:18
+	qt422016.ReleaseByteBuffer(qb422016)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:18
+	return qs422016
+//line app/vmselect/graphite/metrics_expand_response.qtpl:18
+}
+
+// MetricsExpandResponseFlat generates response for /metrics/expand?groupByExpr=0 .See https://graphite-api.readthedocs.io/en/latest/api.html#metrics-expand
+
+//line app/vmselect/graphite/metrics_expand_response.qtpl:23
+func StreamMetricsExpandResponseFlat(qw422016 *qt422016.Writer, paths []string, jsonp string) {
+//line app/vmselect/graphite/metrics_expand_response.qtpl:24
+	if jsonp != "" {
+//line app/vmselect/graphite/metrics_expand_response.qtpl:24
+		qw422016.N().S(jsonp)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:24
+		qw422016.N().S(`(`)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:24
+	}
+//line app/vmselect/graphite/metrics_expand_response.qtpl:25
+	streammetricPaths(qw422016, paths)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:26
+	if jsonp != "" {
+//line app/vmselect/graphite/metrics_expand_response.qtpl:26
+		qw422016.N().S(`)`)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:26
+	}
+//line app/vmselect/graphite/metrics_expand_response.qtpl:27
+}
+
+//line app/vmselect/graphite/metrics_expand_response.qtpl:27
+func WriteMetricsExpandResponseFlat(qq422016 qtio422016.Writer, paths []string, jsonp string) {
+//line app/vmselect/graphite/metrics_expand_response.qtpl:27
+	qw422016 := qt422016.AcquireWriter(qq422016)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:27
+	StreamMetricsExpandResponseFlat(qw422016, paths, jsonp)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:27
+	qt422016.ReleaseWriter(qw422016)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:27
+}
+
+//line app/vmselect/graphite/metrics_expand_response.qtpl:27
+func MetricsExpandResponseFlat(paths []string, jsonp string) string {
+//line app/vmselect/graphite/metrics_expand_response.qtpl:27
+	qb422016 := qt422016.AcquireByteBuffer()
+//line app/vmselect/graphite/metrics_expand_response.qtpl:27
+	WriteMetricsExpandResponseFlat(qb422016, paths, jsonp)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:27
+	qs422016 := string(qb422016.B)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:27
+	qt422016.ReleaseByteBuffer(qb422016)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:27
+	return qs422016
+//line app/vmselect/graphite/metrics_expand_response.qtpl:27
+}
+
+//line app/vmselect/graphite/metrics_expand_response.qtpl:29
+func streammetricPaths(qw422016 *qt422016.Writer, paths []string) {
+//line app/vmselect/graphite/metrics_expand_response.qtpl:29
+	qw422016.N().S(`[`)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:31
+	for i, path := range paths {
+//line app/vmselect/graphite/metrics_expand_response.qtpl:32
+		qw422016.N().Q(path)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:33
+		if i+1 < len(paths) {
+//line app/vmselect/graphite/metrics_expand_response.qtpl:33
+			qw422016.N().S(`,`)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:33
+		}
+//line app/vmselect/graphite/metrics_expand_response.qtpl:34
+	}
+//line app/vmselect/graphite/metrics_expand_response.qtpl:34
+	qw422016.N().S(`]`)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:36
+}
+
+//line app/vmselect/graphite/metrics_expand_response.qtpl:36
+func writemetricPaths(qq422016 qtio422016.Writer, paths []string) {
+//line app/vmselect/graphite/metrics_expand_response.qtpl:36
+	qw422016 := qt422016.AcquireWriter(qq422016)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:36
+	streammetricPaths(qw422016, paths)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:36
+	qt422016.ReleaseWriter(qw422016)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:36
+}
+
+//line app/vmselect/graphite/metrics_expand_response.qtpl:36
+func metricPaths(paths []string) string {
+//line app/vmselect/graphite/metrics_expand_response.qtpl:36
+	qb422016 := qt422016.AcquireByteBuffer()
+//line app/vmselect/graphite/metrics_expand_response.qtpl:36
+	writemetricPaths(qb422016, paths)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:36
+	qs422016 := string(qb422016.B)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:36
+	qt422016.ReleaseByteBuffer(qb422016)
+//line app/vmselect/graphite/metrics_expand_response.qtpl:36
+	return qs422016
+//line app/vmselect/graphite/metrics_expand_response.qtpl:36
+}

app/vmselect/graphite/metrics_find_response.qtpl

@@ -0,0 +1,110 @@
+{% import (
+	"strings"
+
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
+) %}
+
+{% stripspace %}
+
+MetricsFindResponse generates response for /metrics/find .
+See https://graphite-api.readthedocs.io/en/latest/api.html#metrics-find
+{% func MetricsFindResponse(paths []string, delimiter, format string, addWildcards bool, jsonp string) %}
+	{% if jsonp != "" %}{%s= jsonp %}({% endif %}
+	{% switch format %}
+	{% case "completer" %}
+		{%= metricsFindResponseCompleter(paths, delimiter, addWildcards) %}
+	{% case "treejson" %}
+		{%= metricsFindResponseTreeJSON(paths, delimiter, addWildcards) %}
+	{% default %}
+		{% code logger.Panicf("BUG: unexpected format=%q", format) %}
+	{% endswitch %}
+	{% if jsonp != "" %}){% endif %}
+{% endfunc %}
+
+{% func metricsFindResponseCompleter(paths []string, delimiter string, addWildcards bool) %}
+{
+	"metrics":[
+		{% for i, path := range paths %}
+		{
+			"path": {%q= path %},
+			"name": {%= metricPathName(path, delimiter) %},
+			"is_leaf": {% if strings.HasSuffix(path, delimiter) %}0{% else %}1{% endif %}
+		}
+		{% if i+1 < len(paths) %},{% endif %}
+		{% endfor %}
+		{% if addWildcards && len(paths) > 1 %}
+		,{
+			"name": "*"
+		}
+		{% endif %}
+	]
+}
+{% endfunc %}
+
+{% func metricsFindResponseTreeJSON(paths []string, delimiter string, addWildcards bool) %}
+[
+	{% for i, path := range paths %}
+	{
+		{% code
+			allowChildren := "0"
+			isLeaf := "1"
+			if strings.HasSuffix(path, delimiter) {
+				allowChildren = "1"
+				isLeaf = "0"
+			}
+		%}
+		"id": {%q= path %},
+		"text": {%= metricPathName(path, delimiter) %},
+		"allowChildren": {%s= allowChildren %},
+		"expandable": {%s= allowChildren %},
+		"leaf": {%s= isLeaf %}
+	}
+	{% if i+1 < len(paths) %},{% endif %}
+	{% endfor %}
+	{% if addWildcards && len(paths) > 1 %}
+	,{
+		{% code
+			path := paths[0]
+			for strings.HasSuffix(path, delimiter) {
+				path = path[:len(path)-1]
+			}
+			id := ""
+			if n := strings.LastIndexByte(path, delimiter[0]); n >= 0 {
+				id = path[:n+1]
+			}
+			id += "*"
+
+			allowChildren := "0"
+			isLeaf := "1"
+			for _, path := range paths {
+				if strings.HasSuffix(path, delimiter) {
+					allowChildren = "1"
+					isLeaf = "0"
+					break
+				}
+			}
+		%}
+		"id": {%q= id %},
+		"text": "*",
+		"allowChildren": {%s= allowChildren %},
+		"expandable": {%s= allowChildren %},
+		"leaf": {%s= isLeaf %}
+	}
+	{% endif %}
+]
+{% endfunc %}
+
+{% func metricPathName(path, delimiter string) %}
+	{% code
+		name := path
+		for strings.HasSuffix(name, delimiter) {
+			name = name[:len(name)-1]
+		}
+		if n := strings.LastIndexByte(name, delimiter[0]); n >= 0 {
+			name = name[n+1:]
+		}
+	%}
+	{%q= name %}
+{% endfunc %}
+
+{% endstripspace %}

app/vmselect/graphite/metrics_find_response.qtpl.go

@@ -0,0 +1,326 @@
+// Code generated by qtc from "metrics_find_response.qtpl". DO NOT EDIT.
+// See https://github.com/valyala/quicktemplate for details.
+
+//line app/vmselect/graphite/metrics_find_response.qtpl:1
+package graphite
+
+//line app/vmselect/graphite/metrics_find_response.qtpl:1
+import (
+	"strings"
+
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
+)
+
+// MetricsFindResponse generates response for /metrics/find .See https://graphite-api.readthedocs.io/en/latest/api.html#metrics-find
+
+//line app/vmselect/graphite/metrics_find_response.qtpl:11
+import (
+	qtio422016 "io"
+
+	qt422016 "github.com/valyala/quicktemplate"
+)
+
+//line app/vmselect/graphite/metrics_find_response.qtpl:11
+var (
+	_ = qtio422016.Copy
+	_ = qt422016.AcquireByteBuffer
+)
+
+//line app/vmselect/graphite/metrics_find_response.qtpl:11
+func StreamMetricsFindResponse(qw422016 *qt422016.Writer, paths []string, delimiter, format string, addWildcards bool, jsonp string) {
+//line app/vmselect/graphite/metrics_find_response.qtpl:12
+	if jsonp != "" {
+//line app/vmselect/graphite/metrics_find_response.qtpl:12
+		qw422016.N().S(jsonp)
+//line app/vmselect/graphite/metrics_find_response.qtpl:12
+		qw422016.N().S(`(`)
+//line app/vmselect/graphite/metrics_find_response.qtpl:12
+	}
+//line app/vmselect/graphite/metrics_find_response.qtpl:13
+	switch format {
+//line app/vmselect/graphite/metrics_find_response.qtpl:14
+	case "completer":
+//line app/vmselect/graphite/metrics_find_response.qtpl:15
+		streammetricsFindResponseCompleter(qw422016, paths, delimiter, addWildcards)
+//line app/vmselect/graphite/metrics_find_response.qtpl:16
+	case "treejson":
+//line app/vmselect/graphite/metrics_find_response.qtpl:17
+		streammetricsFindResponseTreeJSON(qw422016, paths, delimiter, addWildcards)
+//line app/vmselect/graphite/metrics_find_response.qtpl:18
+	default:
+//line app/vmselect/graphite/metrics_find_response.qtpl:19
+		logger.Panicf("BUG: unexpected format=%q", format)
+
+//line app/vmselect/graphite/metrics_find_response.qtpl:20
+	}
+//line app/vmselect/graphite/metrics_find_response.qtpl:21
+	if jsonp != "" {
+//line app/vmselect/graphite/metrics_find_response.qtpl:21
+		qw422016.N().S(`)`)
+//line app/vmselect/graphite/metrics_find_response.qtpl:21
+	}
+//line app/vmselect/graphite/metrics_find_response.qtpl:22
+}
+
+//line app/vmselect/graphite/metrics_find_response.qtpl:22
+func WriteMetricsFindResponse(qq422016 qtio422016.Writer, paths []string, delimiter, format string, addWildcards bool, jsonp string) {
+//line app/vmselect/graphite/metrics_find_response.qtpl:22
+	qw422016 := qt422016.AcquireWriter(qq422016)
+//line app/vmselect/graphite/metrics_find_response.qtpl:22
+	StreamMetricsFindResponse(qw422016, paths, delimiter, format, addWildcards, jsonp)
+//line app/vmselect/graphite/metrics_find_response.qtpl:22
+	qt422016.ReleaseWriter(qw422016)
+//line app/vmselect/graphite/metrics_find_response.qtpl:22
+}
+
+//line app/vmselect/graphite/metrics_find_response.qtpl:22
+func MetricsFindResponse(paths []string, delimiter, format string, addWildcards bool, jsonp string) string {
+//line app/vmselect/graphite/metrics_find_response.qtpl:22
+	qb422016 := qt422016.AcquireByteBuffer()
+//line app/vmselect/graphite/metrics_find_response.qtpl:22
+	WriteMetricsFindResponse(qb422016, paths, delimiter, format, addWildcards, jsonp)
+//line app/vmselect/graphite/metrics_find_response.qtpl:22
+	qs422016 := string(qb422016.B)
+//line app/vmselect/graphite/metrics_find_response.qtpl:22
+	qt422016.ReleaseByteBuffer(qb422016)
+//line app/vmselect/graphite/metrics_find_response.qtpl:22
+	return qs422016
+//line app/vmselect/graphite/metrics_find_response.qtpl:22
+}
+
+//line app/vmselect/graphite/metrics_find_response.qtpl:24
+func streammetricsFindResponseCompleter(qw422016 *qt422016.Writer, paths []string, delimiter string, addWildcards bool) {
+//line app/vmselect/graphite/metrics_find_response.qtpl:24
+	qw422016.N().S(`{"metrics":[`)
+//line app/vmselect/graphite/metrics_find_response.qtpl:27
+	for i, path := range paths {
+//line app/vmselect/graphite/metrics_find_response.qtpl:27
+		qw422016.N().S(`{"path":`)
+//line app/vmselect/graphite/metrics_find_response.qtpl:29
+		qw422016.N().Q(path)
+//line app/vmselect/graphite/metrics_find_response.qtpl:29
+		qw422016.N().S(`,"name":`)
+//line app/vmselect/graphite/metrics_find_response.qtpl:30
+		streammetricPathName(qw422016, path, delimiter)
+//line app/vmselect/graphite/metrics_find_response.qtpl:30
+		qw422016.N().S(`,"is_leaf":`)
+//line app/vmselect/graphite/metrics_find_response.qtpl:31
+		if strings.HasSuffix(path, delimiter) {
+//line app/vmselect/graphite/metrics_find_response.qtpl:31
+			qw422016.N().S(`0`)
+//line app/vmselect/graphite/metrics_find_response.qtpl:31
+		} else {
+//line app/vmselect/graphite/metrics_find_response.qtpl:31
+			qw422016.N().S(`1`)
+//line app/vmselect/graphite/metrics_find_response.qtpl:31
+		}
+//line app/vmselect/graphite/metrics_find_response.qtpl:31
+		qw422016.N().S(`}`)
+//line app/vmselect/graphite/metrics_find_response.qtpl:33
+		if i+1 < len(paths) {
+//line app/vmselect/graphite/metrics_find_response.qtpl:33
+			qw422016.N().S(`,`)
+//line app/vmselect/graphite/metrics_find_response.qtpl:33
+		}
+//line app/vmselect/graphite/metrics_find_response.qtpl:34
+	}
+//line app/vmselect/graphite/metrics_find_response.qtpl:35
+	if addWildcards && len(paths) > 1 {
+//line app/vmselect/graphite/metrics_find_response.qtpl:35
+		qw422016.N().S(`,{"name": "*"}`)
+//line app/vmselect/graphite/metrics_find_response.qtpl:39
+	}
+//line app/vmselect/graphite/metrics_find_response.qtpl:39
+	qw422016.N().S(`]}`)
+//line app/vmselect/graphite/metrics_find_response.qtpl:42
+}
+
+//line app/vmselect/graphite/metrics_find_response.qtpl:42
+func writemetricsFindResponseCompleter(qq422016 qtio422016.Writer, paths []string, delimiter string, addWildcards bool) {
+//line app/vmselect/graphite/metrics_find_response.qtpl:42
+	qw422016 := qt422016.AcquireWriter(qq422016)
+//line app/vmselect/graphite/metrics_find_response.qtpl:42
+	streammetricsFindResponseCompleter(qw422016, paths, delimiter, addWildcards)
+//line app/vmselect/graphite/metrics_find_response.qtpl:42
+	qt422016.ReleaseWriter(qw422016)
+//line app/vmselect/graphite/metrics_find_response.qtpl:42
+}
+
+//line app/vmselect/graphite/metrics_find_response.qtpl:42
+func metricsFindResponseCompleter(paths []string, delimiter string, addWildcards bool) string {
+//line app/vmselect/graphite/metrics_find_response.qtpl:42
+	qb422016 := qt422016.AcquireByteBuffer()
+//line app/vmselect/graphite/metrics_find_response.qtpl:42
+	writemetricsFindResponseCompleter(qb422016, paths, delimiter, addWildcards)
+//line app/vmselect/graphite/metrics_find_response.qtpl:42
+	qs422016 := string(qb422016.B)
+//line app/vmselect/graphite/metrics_find_response.qtpl:42
+	qt422016.ReleaseByteBuffer(qb422016)
+//line app/vmselect/graphite/metrics_find_response.qtpl:42
+	return qs422016
+//line app/vmselect/graphite/metrics_find_response.qtpl:42
+}
+
+//line app/vmselect/graphite/metrics_find_response.qtpl:44
+func streammetricsFindResponseTreeJSON(qw422016 *qt422016.Writer, paths []string, delimiter string, addWildcards bool) {
+//line app/vmselect/graphite/metrics_find_response.qtpl:44
+	qw422016.N().S(`[`)
+//line app/vmselect/graphite/metrics_find_response.qtpl:46
+	for i, path := range paths {
+//line app/vmselect/graphite/metrics_find_response.qtpl:46
+		qw422016.N().S(`{`)
+//line app/vmselect/graphite/metrics_find_response.qtpl:49
+		allowChildren := "0"
+		isLeaf := "1"
+		if strings.HasSuffix(path, delimiter) {
+			allowChildren = "1"
+			isLeaf = "0"
+		}
+
+//line app/vmselect/graphite/metrics_find_response.qtpl:55
+		qw422016.N().S(`"id":`)
+//line app/vmselect/graphite/metrics_find_response.qtpl:56
+		qw422016.N().Q(path)
+//line app/vmselect/graphite/metrics_find_response.qtpl:56
+		qw422016.N().S(`,"text":`)
+//line app/vmselect/graphite/metrics_find_response.qtpl:57
+		streammetricPathName(qw422016, path, delimiter)
+//line app/vmselect/graphite/metrics_find_response.qtpl:57
+		qw422016.N().S(`,"allowChildren":`)
+//line app/vmselect/graphite/metrics_find_response.qtpl:58
+		qw422016.N().S(allowChildren)
+//line app/vmselect/graphite/metrics_find_response.qtpl:58
+		qw422016.N().S(`,"expandable":`)
+//line app/vmselect/graphite/metrics_find_response.qtpl:59
+		qw422016.N().S(allowChildren)
+//line app/vmselect/graphite/metrics_find_response.qtpl:59
+		qw422016.N().S(`,"leaf":`)
+//line app/vmselect/graphite/metrics_find_response.qtpl:60
+		qw422016.N().S(isLeaf)
+//line app/vmselect/graphite/metrics_find_response.qtpl:60
+		qw422016.N().S(`}`)
+//line app/vmselect/graphite/metrics_find_response.qtpl:62
+		if i+1 < len(paths) {
+//line app/vmselect/graphite/metrics_find_response.qtpl:62
+			qw422016.N().S(`,`)
+//line app/vmselect/graphite/metrics_find_response.qtpl:62
+		}
+//line app/vmselect/graphite/metrics_find_response.qtpl:63
+	}
+//line app/vmselect/graphite/metrics_find_response.qtpl:64
+	if addWildcards && len(paths) > 1 {
+//line app/vmselect/graphite/metrics_find_response.qtpl:64
+		qw422016.N().S(`,{`)
+//line app/vmselect/graphite/metrics_find_response.qtpl:67
+		path := paths[0]
+		for strings.HasSuffix(path, delimiter) {
+			path = path[:len(path)-1]
+		}
+		id := ""
+		if n := strings.LastIndexByte(path, delimiter[0]); n >= 0 {
+			id = path[:n+1]
+		}
+		id += "*"
+
+		allowChildren := "0"
+		isLeaf := "1"
+		for _, path := range paths {
+			if strings.HasSuffix(path, delimiter) {
+				allowChildren = "1"
+				isLeaf = "0"
+				break
+			}
+		}
+
+//line app/vmselect/graphite/metrics_find_response.qtpl:86
+		qw422016.N().S(`"id":`)
+//line app/vmselect/graphite/metrics_find_response.qtpl:87
+		qw422016.N().Q(id)
+//line app/vmselect/graphite/metrics_find_response.qtpl:87
+		qw422016.N().S(`,"text": "*","allowChildren":`)
+//line app/vmselect/graphite/metrics_find_response.qtpl:89
+		qw422016.N().S(allowChildren)
+//line app/vmselect/graphite/metrics_find_response.qtpl:89
+		qw422016.N().S(`,"expandable":`)
+//line app/vmselect/graphite/metrics_find_response.qtpl:90
+		qw422016.N().S(allowChildren)
+//line app/vmselect/graphite/metrics_find_response.qtpl:90
+		qw422016.N().S(`,"leaf":`)
+//line app/vmselect/graphite/metrics_find_response.qtpl:91
+		qw422016.N().S(isLeaf)
+//line app/vmselect/graphite/metrics_find_response.qtpl:91
+		qw422016.N().S(`}`)
+//line app/vmselect/graphite/metrics_find_response.qtpl:93
+	}
+//line app/vmselect/graphite/metrics_find_response.qtpl:93
+	qw422016.N().S(`]`)
+//line app/vmselect/graphite/metrics_find_response.qtpl:95
+}
+
+//line app/vmselect/graphite/metrics_find_response.qtpl:95
+func writemetricsFindResponseTreeJSON(qq422016 qtio422016.Writer, paths []string, delimiter string, addWildcards bool) {
+//line app/vmselect/graphite/metrics_find_response.qtpl:95
+	qw422016 := qt422016.AcquireWriter(qq422016)
+//line app/vmselect/graphite/metrics_find_response.qtpl:95
+	streammetricsFindResponseTreeJSON(qw422016, paths, delimiter, addWildcards)
+//line app/vmselect/graphite/metrics_find_response.qtpl:95
+	qt422016.ReleaseWriter(qw422016)
+//line app/vmselect/graphite/metrics_find_response.qtpl:95
+}
+
+//line app/vmselect/graphite/metrics_find_response.qtpl:95
+func metricsFindResponseTreeJSON(paths []string, delimiter string, addWildcards bool) string {
+//line app/vmselect/graphite/metrics_find_response.qtpl:95
+	qb422016 := qt422016.AcquireByteBuffer()
+//line app/vmselect/graphite/metrics_find_response.qtpl:95
+	writemetricsFindResponseTreeJSON(qb422016, paths, delimiter, addWildcards)
+//line app/vmselect/graphite/metrics_find_response.qtpl:95
+	qs422016 := string(qb422016.B)
+//line app/vmselect/graphite/metrics_find_response.qtpl:95
+	qt422016.ReleaseByteBuffer(qb422016)
+//line app/vmselect/graphite/metrics_find_response.qtpl:95
+	return qs422016
+//line app/vmselect/graphite/metrics_find_response.qtpl:95
+}
+
+//line app/vmselect/graphite/metrics_find_response.qtpl:97
+func streammetricPathName(qw422016 *qt422016.Writer, path, delimiter string) {
+//line app/vmselect/graphite/metrics_find_response.qtpl:99
+	name := path
+	for strings.HasSuffix(name, delimiter) {
+		name = name[:len(name)-1]
+	}
+	if n := strings.LastIndexByte(name, delimiter[0]); n >= 0 {
+		name = name[n+1:]
+	}
+
+//line app/vmselect/graphite/metrics_find_response.qtpl:107
+	qw422016.N().Q(name)
+//line app/vmselect/graphite/metrics_find_response.qtpl:108
+}
+
+//line app/vmselect/graphite/metrics_find_response.qtpl:108
+func writemetricPathName(qq422016 qtio422016.Writer, path, delimiter string) {
+//line app/vmselect/graphite/metrics_find_response.qtpl:108
+	qw422016 := qt422016.AcquireWriter(qq422016)
+//line app/vmselect/graphite/metrics_find_response.qtpl:108
+	streammetricPathName(qw422016, path, delimiter)
+//line app/vmselect/graphite/metrics_find_response.qtpl:108
+	qt422016.ReleaseWriter(qw422016)
+//line app/vmselect/graphite/metrics_find_response.qtpl:108
+}
+
+//line app/vmselect/graphite/metrics_find_response.qtpl:108
+func metricPathName(path, delimiter string) string {
+//line app/vmselect/graphite/metrics_find_response.qtpl:108
+	qb422016 := qt422016.AcquireByteBuffer()
+//line app/vmselect/graphite/metrics_find_response.qtpl:108
+	writemetricPathName(qb422016, path, delimiter)
+//line app/vmselect/graphite/metrics_find_response.qtpl:108
+	qs422016 := string(qb422016.B)
+//line app/vmselect/graphite/metrics_find_response.qtpl:108
+	qt422016.ReleaseByteBuffer(qb422016)
+//line app/vmselect/graphite/metrics_find_response.qtpl:108
+	return qs422016
+//line app/vmselect/graphite/metrics_find_response.qtpl:108
+}

app/vmselect/graphite/metrics_index_response.qtpl

@@ -0,0 +1,11 @@
+{% stripspace %}
+
+MetricsIndexResponse generates response for /metrics/index.json .
+See https://graphite-api.readthedocs.io/en/latest/api.html#metrics-index-json
+{% func MetricsIndexResponse(metricNames []string, jsonp string) %}
+	{% if jsonp != "" %}{%s= jsonp %}({% endif %}
+	{%= metricPaths(metricNames) %}
+	{% if jsonp != "" %}){% endif %}
+{% endfunc %}
+
+{% endstripspace %}

app/vmselect/graphite/metrics_index_response.qtpl.go

@@ -0,0 +1,67 @@
+// Code generated by qtc from "metrics_index_response.qtpl". DO NOT EDIT.
+// See https://github.com/valyala/quicktemplate for details.
+
+// MetricsIndexResponse generates response for /metrics/index.json .See https://graphite-api.readthedocs.io/en/latest/api.html#metrics-index-json
+
+//line app/vmselect/graphite/metrics_index_response.qtpl:5
+package graphite
+
+//line app/vmselect/graphite/metrics_index_response.qtpl:5
+import (
+	qtio422016 "io"
+
+	qt422016 "github.com/valyala/quicktemplate"
+)
+
+//line app/vmselect/graphite/metrics_index_response.qtpl:5
+var (
+	_ = qtio422016.Copy
+	_ = qt422016.AcquireByteBuffer
+)
+
+//line app/vmselect/graphite/metrics_index_response.qtpl:5
+func StreamMetricsIndexResponse(qw422016 *qt422016.Writer, metricNames []string, jsonp string) {
+//line app/vmselect/graphite/metrics_index_response.qtpl:6
+	if jsonp != "" {
+//line app/vmselect/graphite/metrics_index_response.qtpl:6
+		qw422016.N().S(jsonp)
+//line app/vmselect/graphite/metrics_index_response.qtpl:6
+		qw422016.N().S(`(`)
+//line app/vmselect/graphite/metrics_index_response.qtpl:6
+	}
+//line app/vmselect/graphite/metrics_index_response.qtpl:7
+	streammetricPaths(qw422016, metricNames)
+//line app/vmselect/graphite/metrics_index_response.qtpl:8
+	if jsonp != "" {
+//line app/vmselect/graphite/metrics_index_response.qtpl:8
+		qw422016.N().S(`)`)
+//line app/vmselect/graphite/metrics_index_response.qtpl:8
+	}
+//line app/vmselect/graphite/metrics_index_response.qtpl:9
+}
+
+//line app/vmselect/graphite/metrics_index_response.qtpl:9
+func WriteMetricsIndexResponse(qq422016 qtio422016.Writer, metricNames []string, jsonp string) {
+//line app/vmselect/graphite/metrics_index_response.qtpl:9
+	qw422016 := qt422016.AcquireWriter(qq422016)
+//line app/vmselect/graphite/metrics_index_response.qtpl:9
+	StreamMetricsIndexResponse(qw422016, metricNames, jsonp)
+//line app/vmselect/graphite/metrics_index_response.qtpl:9
+	qt422016.ReleaseWriter(qw422016)
+//line app/vmselect/graphite/metrics_index_response.qtpl:9
+}
+
+//line app/vmselect/graphite/metrics_index_response.qtpl:9
+func MetricsIndexResponse(metricNames []string, jsonp string) string {
+//line app/vmselect/graphite/metrics_index_response.qtpl:9
+	qb422016 := qt422016.AcquireByteBuffer()
+//line app/vmselect/graphite/metrics_index_response.qtpl:9
+	WriteMetricsIndexResponse(qb422016, metricNames, jsonp)
+//line app/vmselect/graphite/metrics_index_response.qtpl:9
+	qs422016 := string(qb422016.B)
+//line app/vmselect/graphite/metrics_index_response.qtpl:9
+	qt422016.ReleaseByteBuffer(qb422016)
+//line app/vmselect/graphite/metrics_index_response.qtpl:9
+	return qs422016
+//line app/vmselect/graphite/metrics_index_response.qtpl:9
+}

app/vmselect/main.go

@@ -9,6 +9,7 @@ import (
 	"strings"
 	"time"
 
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/graphite"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/prometheus"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage"
@@ -203,6 +204,33 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 			return true
 		}
 		return true
+	case "/metrics/find", "/metrics/find/":
+		graphiteMetricsFindRequests.Inc()
+		httpserver.EnableCORS(w, r)
+		if err := graphite.MetricsFindHandler(startTime, w, r); err != nil {
+			graphiteMetricsFindErrors.Inc()
+			httpserver.Errorf(w, r, "error in %q: %s", r.URL.Path, err)
+			return true
+		}
+		return true
+	case "/metrics/expand", "/metrics/expand/":
+		graphiteMetricsExpandRequests.Inc()
+		httpserver.EnableCORS(w, r)
+		if err := graphite.MetricsExpandHandler(startTime, w, r); err != nil {
+			graphiteMetricsExpandErrors.Inc()
+			httpserver.Errorf(w, r, "error in %q: %s", r.URL.Path, err)
+			return true
+		}
+		return true
+	case "/metrics/index.json", "/metrics/index.json/":
+		graphiteMetricsIndexRequests.Inc()
+		httpserver.EnableCORS(w, r)
+		if err := graphite.MetricsIndexHandler(startTime, w, r); err != nil {
+			graphiteMetricsIndexErrors.Inc()
+			httpserver.Errorf(w, r, "error in %q: %s", r.URL.Path, err)
+			return true
+		}
+		return true
 	case "/api/v1/rules":
 		// Return dumb placeholder
 		rulesRequests.Inc()
@@ -289,6 +317,15 @@ var (
 	federateRequests = metrics.NewCounter(`vm_http_requests_total{path="/federate"}`)
 	federateErrors   = metrics.NewCounter(`vm_http_request_errors_total{path="/federate"}`)
 
+	graphiteMetricsFindRequests = metrics.NewCounter(`vm_http_requests_total{path="/metrics/find"}`)
+	graphiteMetricsFindErrors   = metrics.NewCounter(`vm_http_request_errors_total{path="/metrics/find"}`)
+
+	graphiteMetricsExpandRequests = metrics.NewCounter(`vm_http_requests_total{path="/metrics/expand"}`)
+	graphiteMetricsExpandErrors   = metrics.NewCounter(`vm_http_request_errors_total{path="/metrics/expand"}`)
+
+	graphiteMetricsIndexRequests = metrics.NewCounter(`vm_http_requests_total{path="/metrics/index.json"}`)
+	graphiteMetricsIndexErrors   = metrics.NewCounter(`vm_http_request_errors_total{path="/metrics/index.json"}`)
+
 	rulesRequests    = metrics.NewCounter(`vm_http_requests_total{path="/api/v1/rules"}`)
 	alertsRequests   = metrics.NewCounter(`vm_http_requests_total{path="/api/v1/alerts"}`)
 	metadataRequests = metrics.NewCounter(`vm_http_requests_total{path="/api/v1/metadata"}`)

app/vmselect/netstorage/netstorage.go

@@ -8,8 +8,8 @@ import (
 	"runtime"
 	"sort"
 	"sync"
-	"time"
 
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/searchutils"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/decimal"
@@ -20,9 +20,10 @@ import (
 )
 
 var (
-	maxTagKeysPerSearch   = flag.Int("search.maxTagKeys", 100e3, "The maximum number of tag keys returned per search")
-	maxTagValuesPerSearch = flag.Int("search.maxTagValues", 100e3, "The maximum number of tag values returned per search")
-	maxMetricsPerSearch   = flag.Int("search.maxUniqueTimeseries", 300e3, "The maximum number of unique time series each search can scan")
+	maxTagKeysPerSearch          = flag.Int("search.maxTagKeys", 100e3, "The maximum number of tag keys returned from /api/v1/labels")
+	maxTagValuesPerSearch        = flag.Int("search.maxTagValues", 100e3, "The maximum number of tag values returned from /api/v1/label/<label_name>/values")
+	maxTagValueSuffixesPerSearch = flag.Int("search.maxTagValueSuffixesPerSearch", 100e3, "The maximum number of tag value suffixes returned from /metrics/find")
+	maxMetricsPerSearch          = flag.Int("search.maxUniqueTimeseries", 300e3, "The maximum number of unique time series each search can scan")
 )
 
 // Result is a single timeseries result.
@@ -52,7 +53,7 @@ func (r *Result) reset() {
 type Results struct {
 	tr        storage.TimeRange
 	fetchData bool
-	deadline  Deadline
+	deadline  searchutils.Deadline
 
 	packedTimeseries []packedTimeseries
 	sr               *storage.Search
@@ -457,11 +458,11 @@ func DeleteSeries(sq *storage.SearchQuery) (int, error) {
 }
 
 // GetLabels returns labels until the given deadline.
-func GetLabels(deadline Deadline) ([]string, error) {
+func GetLabels(deadline searchutils.Deadline) ([]string, error) {
 	if deadline.Exceeded() {
 		return nil, fmt.Errorf("timeout exceeded before starting the query processing: %s", deadline.String())
 	}
-	labels, err := vmstorage.SearchTagKeys(*maxTagKeysPerSearch, deadline.deadline)
+	labels, err := vmstorage.SearchTagKeys(*maxTagKeysPerSearch, deadline.Deadline())
 	if err != nil {
 		return nil, fmt.Errorf("error during labels search: %w", err)
 	}
@@ -481,7 +482,7 @@ func GetLabels(deadline Deadline) ([]string, error) {
 
 // GetLabelValues returns label values for the given labelName
 // until the given deadline.
-func GetLabelValues(labelName string, deadline Deadline) ([]string, error) {
+func GetLabelValues(labelName string, deadline searchutils.Deadline) ([]string, error) {
 	if deadline.Exceeded() {
 		return nil, fmt.Errorf("timeout exceeded before starting the query processing: %s", deadline.String())
 	}
@@ -490,7 +491,7 @@ func GetLabelValues(labelName string, deadline Deadline) ([]string, error) {
 	}
 
 	// Search for tag values
-	labelValues, err := vmstorage.SearchTagValues([]byte(labelName), *maxTagValuesPerSearch, deadline.deadline)
+	labelValues, err := vmstorage.SearchTagValues([]byte(labelName), *maxTagValuesPerSearch, deadline.Deadline())
 	if err != nil {
 		return nil, fmt.Errorf("error during label values search for labelName=%q: %w", labelName, err)
 	}
@@ -501,12 +502,27 @@ func GetLabelValues(labelName string, deadline Deadline) ([]string, error) {
 	return labelValues, nil
 }
 
-// GetLabelEntries returns all the label entries until the given deadline.
-func GetLabelEntries(deadline Deadline) ([]storage.TagEntry, error) {
+// GetTagValueSuffixes returns tag value suffixes for the given tagKey and the given tagValuePrefix.
+//
+// It can be used for implementing https://graphite-api.readthedocs.io/en/latest/api.html#metrics-find
+func GetTagValueSuffixes(tr storage.TimeRange, tagKey, tagValuePrefix string, delimiter byte, deadline searchutils.Deadline) ([]string, error) {
 	if deadline.Exceeded() {
 		return nil, fmt.Errorf("timeout exceeded before starting the query processing: %s", deadline.String())
 	}
-	labelEntries, err := vmstorage.SearchTagEntries(*maxTagKeysPerSearch, *maxTagValuesPerSearch, deadline.deadline)
+	suffixes, err := vmstorage.SearchTagValueSuffixes(tr, []byte(tagKey), []byte(tagValuePrefix), delimiter, *maxTagValueSuffixesPerSearch, deadline.Deadline())
+	if err != nil {
+		return nil, fmt.Errorf("error during search for suffixes for tagKey=%q, tagValuePrefix=%q, delimiter=%c on time range %s: %w",
+			tagKey, tagValuePrefix, delimiter, tr.String(), err)
+	}
+	return suffixes, nil
+}
+
+// GetLabelEntries returns all the label entries until the given deadline.
+func GetLabelEntries(deadline searchutils.Deadline) ([]storage.TagEntry, error) {
+	if deadline.Exceeded() {
+		return nil, fmt.Errorf("timeout exceeded before starting the query processing: %s", deadline.String())
+	}
+	labelEntries, err := vmstorage.SearchTagEntries(*maxTagKeysPerSearch, *maxTagValuesPerSearch, deadline.Deadline())
 	if err != nil {
 		return nil, fmt.Errorf("error during label entries request: %w", err)
 	}
@@ -532,11 +548,11 @@ func GetLabelEntries(deadline Deadline) ([]storage.TagEntry, error) {
 }
 
 // GetTSDBStatusForDate returns tsdb status according to https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats
-func GetTSDBStatusForDate(deadline Deadline, date uint64, topN int) (*storage.TSDBStatus, error) {
+func GetTSDBStatusForDate(deadline searchutils.Deadline, date uint64, topN int) (*storage.TSDBStatus, error) {
 	if deadline.Exceeded() {
 		return nil, fmt.Errorf("timeout exceeded before starting the query processing: %s", deadline.String())
 	}
-	status, err := vmstorage.GetTSDBStatusForDate(date, topN, deadline.deadline)
+	status, err := vmstorage.GetTSDBStatusForDate(date, topN, deadline.Deadline())
 	if err != nil {
 		return nil, fmt.Errorf("error during tsdb status request: %w", err)
 	}
@@ -544,11 +560,11 @@ func GetTSDBStatusForDate(deadline Deadline, date uint64, topN int) (*storage.TS
 }
 
 // GetSeriesCount returns the number of unique series.
-func GetSeriesCount(deadline Deadline) (uint64, error) {
+func GetSeriesCount(deadline searchutils.Deadline) (uint64, error) {
 	if deadline.Exceeded() {
 		return 0, fmt.Errorf("timeout exceeded before starting the query processing: %s", deadline.String())
 	}
-	n, err := vmstorage.GetSeriesCount(deadline.deadline)
+	n, err := vmstorage.GetSeriesCount(deadline.Deadline())
 	if err != nil {
 		return 0, fmt.Errorf("error during series count request: %w", err)
 	}
@@ -573,7 +589,7 @@ var ssPool sync.Pool
 // ProcessSearchQuery performs sq on storage nodes until the given deadline.
 //
 // Results.RunParallel or Results.Cancel must be called on the returned Results.
-func ProcessSearchQuery(sq *storage.SearchQuery, fetchData bool, deadline Deadline) (*Results, error) {
+func ProcessSearchQuery(sq *storage.SearchQuery, fetchData bool, deadline searchutils.Deadline) (*Results, error) {
 	if deadline.Exceeded() {
 		return nil, fmt.Errorf("timeout exceeded before starting the query processing: %s", deadline.String())
 	}
@@ -595,7 +611,7 @@ func ProcessSearchQuery(sq *storage.SearchQuery, fetchData bool, deadline Deadli
 	defer vmstorage.WG.Done()
 
 	sr := getStorageSearch()
-	maxSeriesCount := sr.Init(vmstorage.Storage, tfss, tr, *maxMetricsPerSearch, deadline.deadline)
+	maxSeriesCount := sr.Init(vmstorage.Storage, tfss, tr, *maxMetricsPerSearch, deadline.Deadline())
 
 	m := make(map[string][]storage.BlockRef, maxSeriesCount)
 	orderedMetricNames := make([]string, 0, maxSeriesCount)
@@ -656,33 +672,3 @@ func setupTfss(tagFilterss [][]storage.TagFilter) ([]*storage.TagFilters, error)
 	}
 	return tfss, nil
 }
-
-// Deadline contains deadline with the corresponding timeout for pretty error messages.
-type Deadline struct {
-	deadline uint64
-
-	timeout  time.Duration
-	flagHint string
-}
-
-// NewDeadline returns deadline for the given timeout.
-//
-// flagHint must contain a hit for command-line flag, which could be used
-// in order to increase timeout.
-func NewDeadline(startTime time.Time, timeout time.Duration, flagHint string) Deadline {
-	return Deadline{
-		deadline: uint64(startTime.Add(timeout).Unix()),
-		timeout:  timeout,
-		flagHint: flagHint,
-	}
-}
-
-// Exceeded returns true if deadline is exceeded.
-func (d *Deadline) Exceeded() bool {
-	return fasttime.UnixTimestamp() > d.deadline
-}
-
-// String returns human-readable string representation for d.
-func (d *Deadline) String() string {
-	return fmt.Sprintf("%.3f seconds; the timeout can be adjusted with `%s` command-line flag", d.timeout.Seconds(), d.flagHint)
-}

app/vmselect/prometheus/prometheus.go

@@ -8,13 +8,14 @@ import (
 	"runtime"
 	"sort"
 	"strconv"
-	"strings"
 	"sync"
 	"time"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/netstorage"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql"
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/searchutils"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fasttime"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/storage"
@@ -25,12 +26,10 @@ import (
 )
 
 var (
-	latencyOffset = flag.Duration("search.latencyOffset", time.Second*30, "The time when data points become visible in query results after the colection. "+
+	latencyOffset = flag.Duration("search.latencyOffset", time.Second*30, "The time when data points become visible in query results after the collection. "+
 		"Too small value can result in incomplete last points for query results")
-	maxExportDuration = flag.Duration("search.maxExportDuration", time.Hour*24*30, "The maximum duration for /api/v1/export call")
-	maxQueryDuration  = flag.Duration("search.maxQueryDuration", time.Second*30, "The maximum duration for search query execution")
-	maxQueryLen       = flag.Int("search.maxQueryLen", 16*1024, "The maximum search query length in bytes")
-	maxLookback       = flag.Duration("search.maxLookback", 0, "Synonim to -search.lookback-delta from Prometheus. "+
+	maxQueryLen = flagutil.NewBytes("search.maxQueryLen", 16*1024, "The maximum search query length in bytes")
+	maxLookback = flag.Duration("search.maxLookback", 0, "Synonim to -search.lookback-delta from Prometheus. "+
 		"The value is dynamically detected from interval between time series datapoints if not set. It can be overridden on per-query basis via max_lookback arg. "+
 		"See also '-search.maxStalenessInterval' flag, which has the same meaining due to historical reasons")
 	maxStalenessInterval = flag.Duration("search.maxStalenessInterval", 0, "The maximum interval for staleness calculations. "+
@@ -59,15 +58,15 @@ func FederateHandler(startTime time.Time, w http.ResponseWriter, r *http.Request
 	if lookbackDelta <= 0 {
 		lookbackDelta = defaultStep
 	}
-	start, err := getTime(r, "start", ct-lookbackDelta)
+	start, err := searchutils.GetTime(r, "start", ct-lookbackDelta)
 	if err != nil {
 		return err
 	}
-	end, err := getTime(r, "end", ct)
+	end, err := searchutils.GetTime(r, "end", ct)
 	if err != nil {
 		return err
 	}
-	deadline := getDeadlineForQuery(r, startTime)
+	deadline := searchutils.GetDeadlineForQuery(r, startTime)
 	if start >= end {
 		start = end - defaultStep
 	}
@@ -128,17 +127,17 @@ func ExportHandler(startTime time.Time, w http.ResponseWriter, r *http.Request)
 		}
 		matches = []string{match}
 	}
-	start, err := getTime(r, "start", 0)
+	start, err := searchutils.GetTime(r, "start", 0)
 	if err != nil {
 		return err
 	}
-	end, err := getTime(r, "end", ct)
+	end, err := searchutils.GetTime(r, "end", ct)
 	if err != nil {
 		return err
 	}
 	format := r.FormValue("format")
 	maxRowsPerLine := int(fastfloat.ParseInt64BestEffort(r.FormValue("max_rows_per_line")))
-	deadline := getDeadlineForExport(r, startTime)
+	deadline := searchutils.GetDeadlineForExport(r, startTime)
 	if start >= end {
 		end = start + defaultStep
 	}
@@ -151,7 +150,7 @@ func ExportHandler(startTime time.Time, w http.ResponseWriter, r *http.Request)
 
 var exportDuration = metrics.NewSummary(`vm_request_duration_seconds{path="/api/v1/export"}`)
 
-func exportHandler(w http.ResponseWriter, matches []string, start, end int64, format string, maxRowsPerLine int, deadline netstorage.Deadline) error {
+func exportHandler(w http.ResponseWriter, matches []string, start, end int64, format string, maxRowsPerLine int, deadline searchutils.Deadline) error {
 	writeResponseFunc := WriteExportStdResponse
 	writeLineFunc := func(rs *netstorage.Result, resultsCh chan<- *quicktemplate.ByteBuffer) {
 		bb := quicktemplate.AcquireByteBuffer()
@@ -282,7 +281,7 @@ var deleteDuration = metrics.NewSummary(`vm_request_duration_seconds{path="/api/
 //
 // See https://prometheus.io/docs/prometheus/latest/querying/api/#querying-label-values
 func LabelValuesHandler(startTime time.Time, labelName string, w http.ResponseWriter, r *http.Request) error {
-	deadline := getDeadlineForQuery(r, startTime)
+	deadline := searchutils.GetDeadlineForQuery(r, startTime)
 	if err := r.ParseForm(); err != nil {
 		return fmt.Errorf("cannot parse form values: %w", err)
 	}
@@ -303,11 +302,11 @@ func LabelValuesHandler(startTime time.Time, labelName string, w http.ResponseWr
 			matches = []string{fmt.Sprintf("{%s!=''}", labelName)}
 		}
 		ct := startTime.UnixNano() / 1e6
-		end, err := getTime(r, "end", ct)
+		end, err := searchutils.GetTime(r, "end", ct)
 		if err != nil {
 			return err
 		}
-		start, err := getTime(r, "start", end-defaultStep)
+		start, err := searchutils.GetTime(r, "start", end-defaultStep)
 		if err != nil {
 			return err
 		}
@@ -323,7 +322,7 @@ func LabelValuesHandler(startTime time.Time, labelName string, w http.ResponseWr
 	return nil
 }
 
-func labelValuesWithMatches(labelName string, matches []string, start, end int64, deadline netstorage.Deadline) ([]string, error) {
+func labelValuesWithMatches(labelName string, matches []string, start, end int64, deadline searchutils.Deadline) ([]string, error) {
 	if len(matches) == 0 {
 		logger.Panicf("BUG: matches must be non-empty")
 	}
@@ -384,7 +383,7 @@ var labelValuesDuration = metrics.NewSummary(`vm_request_duration_seconds{path="
 
 // LabelsCountHandler processes /api/v1/labels/count request.
 func LabelsCountHandler(startTime time.Time, w http.ResponseWriter, r *http.Request) error {
-	deadline := getDeadlineForQuery(r, startTime)
+	deadline := searchutils.GetDeadlineForQuery(r, startTime)
 	labelEntries, err := netstorage.GetLabelEntries(deadline)
 	if err != nil {
 		return fmt.Errorf(`cannot obtain label entries: %w`, err)
@@ -403,7 +402,7 @@ const secsPerDay = 3600 * 24
 //
 // See https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats
 func TSDBStatusHandler(startTime time.Time, w http.ResponseWriter, r *http.Request) error {
-	deadline := getDeadlineForQuery(r, startTime)
+	deadline := searchutils.GetDeadlineForQuery(r, startTime)
 	if err := r.ParseForm(); err != nil {
 		return fmt.Errorf("cannot parse form values: %w", err)
 	}
@@ -447,7 +446,7 @@ var tsdbStatusDuration = metrics.NewSummary(`vm_request_duration_seconds{path="/
 //
 // See https://prometheus.io/docs/prometheus/latest/querying/api/#getting-label-names
 func LabelsHandler(startTime time.Time, w http.ResponseWriter, r *http.Request) error {
-	deadline := getDeadlineForQuery(r, startTime)
+	deadline := searchutils.GetDeadlineForQuery(r, startTime)
 	if err := r.ParseForm(); err != nil {
 		return fmt.Errorf("cannot parse form values: %w", err)
 	}
@@ -466,11 +465,11 @@ func LabelsHandler(startTime time.Time, w http.ResponseWriter, r *http.Request)
 			matches = []string{"{__name__!=''}"}
 		}
 		ct := startTime.UnixNano() / 1e6
-		end, err := getTime(r, "end", ct)
+		end, err := searchutils.GetTime(r, "end", ct)
 		if err != nil {
 			return err
 		}
-		start, err := getTime(r, "start", end-defaultStep)
+		start, err := searchutils.GetTime(r, "start", end-defaultStep)
 		if err != nil {
 			return err
 		}
@@ -486,7 +485,7 @@ func LabelsHandler(startTime time.Time, w http.ResponseWriter, r *http.Request)
 	return nil
 }
 
-func labelsWithMatches(matches []string, start, end int64, deadline netstorage.Deadline) ([]string, error) {
+func labelsWithMatches(matches []string, start, end int64, deadline searchutils.Deadline) ([]string, error) {
 	if len(matches) == 0 {
 		logger.Panicf("BUG: matches must be non-empty")
 	}
@@ -535,7 +534,7 @@ var labelsDuration = metrics.NewSummary(`vm_request_duration_seconds{path="/api/
 
 // SeriesCountHandler processes /api/v1/series/count request.
 func SeriesCountHandler(startTime time.Time, w http.ResponseWriter, r *http.Request) error {
-	deadline := getDeadlineForQuery(r, startTime)
+	deadline := searchutils.GetDeadlineForQuery(r, startTime)
 	n, err := netstorage.GetSeriesCount(deadline)
 	if err != nil {
 		return fmt.Errorf("cannot obtain series count: %w", err)
@@ -560,20 +559,20 @@ func SeriesHandler(startTime time.Time, w http.ResponseWriter, r *http.Request)
 	if len(matches) == 0 {
 		return fmt.Errorf("missing `match[]` arg")
 	}
-	end, err := getTime(r, "end", ct)
+	end, err := searchutils.GetTime(r, "end", ct)
 	if err != nil {
 		return err
 	}
-	// Do not set start to minTimeMsecs by default as Prometheus does,
+	// Do not set start to searchutils.minTimeMsecs by default as Prometheus does,
 	// since this leads to fetching and scanning all the data from the storage,
 	// which can take a lot of time for big storages.
 	// It is better setting start as end-defaultStep by default.
 	// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/91
-	start, err := getTime(r, "start", end-defaultStep)
+	start, err := searchutils.GetTime(r, "start", end-defaultStep)
 	if err != nil {
 		return err
 	}
-	deadline := getDeadlineForQuery(r, startTime)
+	deadline := searchutils.GetDeadlineForQuery(r, startTime)
 
 	tagFilterss, err := getTagFilterssFromMatches(matches)
 	if err != nil {
@@ -631,7 +630,7 @@ func QueryHandler(startTime time.Time, w http.ResponseWriter, r *http.Request) e
 	if len(query) == 0 {
 		return fmt.Errorf("missing `query` arg")
 	}
-	start, err := getTime(r, "time", ct)
+	start, err := searchutils.GetTime(r, "time", ct)
 	if err != nil {
 		return err
 	}
@@ -639,20 +638,20 @@ func QueryHandler(startTime time.Time, w http.ResponseWriter, r *http.Request) e
 	if err != nil {
 		return err
 	}
-	step, err := getDuration(r, "step", lookbackDelta)
+	step, err := searchutils.GetDuration(r, "step", lookbackDelta)
 	if err != nil {
 		return err
 	}
 	if step <= 0 {
 		step = defaultStep
 	}
-	deadline := getDeadlineForQuery(r, startTime)
+	deadline := searchutils.GetDeadlineForQuery(r, startTime)
 
-	if len(query) > *maxQueryLen {
-		return fmt.Errorf("too long query; got %d bytes; mustn't exceed `-search.maxQueryLen=%d` bytes", len(query), *maxQueryLen)
+	if len(query) > maxQueryLen.N {
+		return fmt.Errorf("too long query; got %d bytes; mustn't exceed `-search.maxQueryLen=%d` bytes", len(query), maxQueryLen.N)
 	}
 	queryOffset := getLatencyOffsetMilliseconds()
-	if !getBool(r, "nocache") && ct-start < queryOffset {
+	if !searchutils.GetBool(r, "nocache") && ct-start < queryOffset {
 		// Adjust start time only if `nocache` arg isn't set.
 		// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/241
 		start = ct - queryOffset
@@ -745,15 +744,15 @@ func QueryRangeHandler(startTime time.Time, w http.ResponseWriter, r *http.Reque
 	if len(query) == 0 {
 		return fmt.Errorf("missing `query` arg")
 	}
-	start, err := getTime(r, "start", ct-defaultStep)
+	start, err := searchutils.GetTime(r, "start", ct-defaultStep)
 	if err != nil {
 		return err
 	}
-	end, err := getTime(r, "end", ct)
+	end, err := searchutils.GetTime(r, "end", ct)
 	if err != nil {
 		return err
 	}
-	step, err := getDuration(r, "step", defaultStep)
+	step, err := searchutils.GetDuration(r, "step", defaultStep)
 	if err != nil {
 		return err
 	}
@@ -765,16 +764,16 @@ func QueryRangeHandler(startTime time.Time, w http.ResponseWriter, r *http.Reque
 }
 
 func queryRangeHandler(startTime time.Time, w http.ResponseWriter, query string, start, end, step int64, r *http.Request, ct int64) error {
-	deadline := getDeadlineForQuery(r, startTime)
-	mayCache := !getBool(r, "nocache")
+	deadline := searchutils.GetDeadlineForQuery(r, startTime)
+	mayCache := !searchutils.GetBool(r, "nocache")
 	lookbackDelta, err := getMaxLookback(r)
 	if err != nil {
 		return err
 	}
 
 	// Validate input args.
-	if len(query) > *maxQueryLen {
-		return fmt.Errorf("too long query; got %d bytes; mustn't exceed `-search.maxQueryLen=%d` bytes", len(query), *maxQueryLen)
+	if len(query) > maxQueryLen.N {
+		return fmt.Errorf("too long query; got %d bytes; mustn't exceed `-search.maxQueryLen=%d` bytes", len(query), maxQueryLen.N)
 	}
 	if start > end {
 		end = start + defaultStep
@@ -886,120 +885,12 @@ func adjustLastPoints(tss []netstorage.Result, start, end int64) []netstorage.Re
 	return tss
 }
 
-func getTime(r *http.Request, argKey string, defaultValue int64) (int64, error) {
-	argValue := r.FormValue(argKey)
-	if len(argValue) == 0 {
-		return defaultValue, nil
-	}
-	secs, err := strconv.ParseFloat(argValue, 64)
-	if err != nil {
-		// Try parsing string format
-		t, err := time.Parse(time.RFC3339, argValue)
-		if err != nil {
-			// Handle Prometheus'-provided minTime and maxTime.
-			// See https://github.com/prometheus/client_golang/issues/614
-			switch argValue {
-			case prometheusMinTimeFormatted:
-				return minTimeMsecs, nil
-			case prometheusMaxTimeFormatted:
-				return maxTimeMsecs, nil
-			}
-			// Try parsing duration relative to the current time
-			d, err1 := metricsql.DurationValue(argValue, 0)
-			if err1 != nil {
-				return 0, fmt.Errorf("cannot parse %q=%q: %w", argKey, argValue, err)
-			}
-			if d > 0 {
-				d = -d
-			}
-			t = time.Now().Add(time.Duration(d) * time.Millisecond)
-		}
-		secs = float64(t.UnixNano()) / 1e9
-	}
-	msecs := int64(secs * 1e3)
-	if msecs < minTimeMsecs {
-		msecs = 0
-	}
-	if msecs > maxTimeMsecs {
-		msecs = maxTimeMsecs
-	}
-	return msecs, nil
-}
-
-var (
-	// These constants were obtained from https://github.com/prometheus/prometheus/blob/91d7175eaac18b00e370965f3a8186cc40bf9f55/web/api/v1/api.go#L442
-	// See https://github.com/prometheus/client_golang/issues/614 for details.
-	prometheusMinTimeFormatted = time.Unix(math.MinInt64/1000+62135596801, 0).UTC().Format(time.RFC3339Nano)
-	prometheusMaxTimeFormatted = time.Unix(math.MaxInt64/1000-62135596801, 999999999).UTC().Format(time.RFC3339Nano)
-)
-
-const (
-	// These values prevent from overflow when storing msec-precision time in int64.
-	minTimeMsecs = 0 // use 0 instead of `int64(-1<<63) / 1e6` because the storage engine doesn't actually support negative time
-	maxTimeMsecs = int64(1<<63-1) / 1e6
-)
-
-func getDuration(r *http.Request, argKey string, defaultValue int64) (int64, error) {
-	argValue := r.FormValue(argKey)
-	if len(argValue) == 0 {
-		return defaultValue, nil
-	}
-	secs, err := strconv.ParseFloat(argValue, 64)
-	if err != nil {
-		// Try parsing string format
-		d, err := metricsql.DurationValue(argValue, 0)
-		if err != nil {
-			return 0, fmt.Errorf("cannot parse %q=%q: %w", argKey, argValue, err)
-		}
-		secs = float64(d) / 1000
-	}
-	msecs := int64(secs * 1e3)
-	if msecs <= 0 || msecs > maxDurationMsecs {
-		return 0, fmt.Errorf("%q=%dms is out of allowed range [%d ... %d]", argKey, msecs, 0, int64(maxDurationMsecs))
-	}
-	return msecs, nil
-}
-
-const maxDurationMsecs = 100 * 365 * 24 * 3600 * 1000
-
 func getMaxLookback(r *http.Request) (int64, error) {
 	d := maxLookback.Milliseconds()
 	if d == 0 {
 		d = maxStalenessInterval.Milliseconds()
 	}
-	return getDuration(r, "max_lookback", d)
-}
-
-func getDeadlineForQuery(r *http.Request, startTime time.Time) netstorage.Deadline {
-	dMax := maxQueryDuration.Milliseconds()
-	return getDeadlineWithMaxDuration(r, startTime, dMax, "-search.maxQueryDuration")
-}
-
-func getDeadlineForExport(r *http.Request, startTime time.Time) netstorage.Deadline {
-	dMax := maxExportDuration.Milliseconds()
-	return getDeadlineWithMaxDuration(r, startTime, dMax, "-search.maxExportDuration")
-}
-
-func getDeadlineWithMaxDuration(r *http.Request, startTime time.Time, dMax int64, flagHint string) netstorage.Deadline {
-	d, err := getDuration(r, "timeout", 0)
-	if err != nil {
-		d = 0
-	}
-	if d <= 0 || d > dMax {
-		d = dMax
-	}
-	timeout := time.Duration(d) * time.Millisecond
-	return netstorage.NewDeadline(startTime, timeout, flagHint)
-}
-
-func getBool(r *http.Request, argKey string) bool {
-	argValue := r.FormValue(argKey)
-	switch strings.ToLower(argValue) {
-	case "", "0", "f", "false", "no":
-		return false
-	default:
-		return true
-	}
+	return searchutils.GetDuration(r, "max_lookback", d)
 }
 
 func getTagFilterssFromMatches(matches []string) ([][]storage.TagFilter, error) {

app/vmselect/prometheus/prometheus_test.go

@@ -1,10 +1,7 @@
 package prometheus
 
 import (
-	"fmt"
 	"math"
-	"net/http"
-	"net/url"
 	"reflect"
 	"testing"
 
@@ -50,76 +47,6 @@ func TestRemoveEmptyValuesAndTimeseries(t *testing.T) {
 	})
 }
 
-func TestGetTimeSuccess(t *testing.T) {
-	f := func(s string, timestampExpected int64) {
-		t.Helper()
-		urlStr := fmt.Sprintf("http://foo.bar/baz?s=%s", url.QueryEscape(s))
-		r, err := http.NewRequest("GET", urlStr, nil)
-		if err != nil {
-			t.Fatalf("unexpected error in NewRequest: %s", err)
-		}
-
-		// Verify defaultValue
-		ts, err := getTime(r, "foo", 123)
-		if err != nil {
-			t.Fatalf("unexpected error when obtaining default time from getTime(%q): %s", s, err)
-		}
-		if ts != 123 {
-			t.Fatalf("unexpected default value for getTime(%q); got %d; want %d", s, ts, 123)
-		}
-
-		// Verify timestampExpected
-		ts, err = getTime(r, "s", 123)
-		if err != nil {
-			t.Fatalf("unexpected error in getTime(%q): %s", s, err)
-		}
-		if ts != timestampExpected {
-			t.Fatalf("unexpected timestamp for getTime(%q); got %d; want %d", s, ts, timestampExpected)
-		}
-	}
-
-	f("2019-07-07T20:01:02Z", 1562529662000)
-	f("2019-07-07T20:47:40+03:00", 1562521660000)
-	f("-292273086-05-16T16:47:06Z", minTimeMsecs)
-	f("292277025-08-18T07:12:54.999999999Z", maxTimeMsecs)
-	f("1562529662.324", 1562529662324)
-	f("-9223372036.854", minTimeMsecs)
-	f("-9223372036.855", minTimeMsecs)
-	f("9223372036.855", maxTimeMsecs)
-}
-
-func TestGetTimeError(t *testing.T) {
-	f := func(s string) {
-		t.Helper()
-		urlStr := fmt.Sprintf("http://foo.bar/baz?s=%s", url.QueryEscape(s))
-		r, err := http.NewRequest("GET", urlStr, nil)
-		if err != nil {
-			t.Fatalf("unexpected error in NewRequest: %s", err)
-		}
-
-		// Verify defaultValue
-		ts, err := getTime(r, "foo", 123)
-		if err != nil {
-			t.Fatalf("unexpected error when obtaining default time from getTime(%q): %s", s, err)
-		}
-		if ts != 123 {
-			t.Fatalf("unexpected default value for getTime(%q); got %d; want %d", s, ts, 123)
-		}
-
-		// Verify timestampExpected
-		_, err = getTime(r, "s", 123)
-		if err == nil {
-			t.Fatalf("expecting non-nil error in getTime(%q)", s)
-		}
-	}
-
-	f("foo")
-	f("2019-07-07T20:01:02Zisdf")
-	f("2019-07-07T20:47:40+03:00123")
-	f("-292273086-05-16T16:47:07Z")
-	f("292277025-08-18T07:12:54.999999998Z")
-}
-
 func TestAdjustLastPoints(t *testing.T) {
 	f := func(tss []netstorage.Result, start, end int64, tssExpected []netstorage.Result) {
 		t.Helper()

app/vmselect/promql/eval.go

@@ -8,6 +8,7 @@ import (
 	"sync"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/netstorage"
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/searchutils"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/memory"
@@ -56,14 +57,7 @@ func AdjustStartEnd(start, end, step int64) (int64, int64) {
 
 	// Round start and end to values divisible by step in order
 	// to enable response caching (see EvalConfig.mayCache).
-
-	// Round start to the nearest smaller value divisible by step.
-	start -= start % step
-	// Round end to the nearest bigger value divisible by step.
-	adjust := end % step
-	if adjust > 0 {
-		end += step - adjust
-	}
+	start, end = alignStartEnd(start, end, step)
 
 	// Make sure that the new number of points is the same as the initial number of points.
 	newPoints := (end-start)/step + 1
@@ -75,6 +69,17 @@ func AdjustStartEnd(start, end, step int64) (int64, int64) {
 	return start, end
 }
 
+func alignStartEnd(start, end, step int64) (int64, int64) {
+	// Round start to the nearest smaller value divisible by step.
+	start -= start % step
+	// Round end to the nearest bigger value divisible by step.
+	adjust := end % step
+	if adjust > 0 {
+		end += step - adjust
+	}
+	return start, end
+}
+
 // EvalConfig is the configuration required for query evaluation via Exec
 type EvalConfig struct {
 	Start int64
@@ -84,7 +89,7 @@ type EvalConfig struct {
 	// QuotedRemoteAddr contains quoted remote address.
 	QuotedRemoteAddr string
 
-	Deadline netstorage.Deadline
+	Deadline searchutils.Deadline
 
 	MayCache bool
 
@@ -501,11 +506,13 @@ func evalRollupFuncWithSubquery(ec *EvalConfig, name string, rf rollupFunc, expr
 
 	ecSQ := newEvalConfig(ec)
 	ecSQ.Start -= window + maxSilenceInterval + step
+	ecSQ.End += step
 	ecSQ.Step = step
 	if err := ValidateMaxPointsPerTimeseries(ecSQ.Start, ecSQ.End, ecSQ.Step); err != nil {
 		return nil, err
 	}
-	ecSQ.Start, ecSQ.End = AdjustStartEnd(ecSQ.Start, ecSQ.End, ecSQ.Step)
+	// unconditionally align start and end args to step for subquery as Prometheus does.
+	ecSQ.Start, ecSQ.End = alignStartEnd(ecSQ.Start, ecSQ.End, ecSQ.Step)
 	tssSQ, err := evalExpr(ecSQ, re.Expr)
 	if err != nil {
 		return nil, err
@@ -517,7 +524,6 @@ func evalRollupFuncWithSubquery(ec *EvalConfig, name string, rf rollupFunc, expr
 		}
 		return nil, nil
 	}
-
 	sharedTimestamps := getTimestamps(ec.Start, ec.End, ec.Step)
 	preFunc, rcs, err := getRollupConfigs(name, rf, expr, ec.Start, ec.End, ec.Step, window, ec.LookbackDelta, sharedTimestamps)
 	if err != nil {
@@ -830,7 +836,7 @@ func evalTime(ec *EvalConfig) []*timeseries {
 	timestamps := rv[0].Timestamps
 	values := rv[0].Values
 	for i, ts := range timestamps {
-		values[i] = float64(ts) * 1e-3
+		values[i] = float64(ts) / 1e3
 	}
 	return rv
 }

app/vmselect/promql/exec_test.go

@@ -5,6 +5,7 @@ import (
 	"time"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/netstorage"
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/searchutils"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/storage"
 )
 
@@ -21,7 +22,7 @@ func TestExecSuccess(t *testing.T) {
 			Start:    start,
 			End:      end,
 			Step:     step,
-			Deadline: netstorage.NewDeadline(time.Now(), time.Minute, ""),
+			Deadline: searchutils.NewDeadline(time.Now(), time.Minute, ""),
 		}
 		for i := 0; i < 5; i++ {
 			result, err := Exec(ec, q, false)
@@ -108,7 +109,7 @@ func TestExecSuccess(t *testing.T) {
 		q := `time() offset 0s`
 		r := netstorage.Result{
 			MetricName: metricNameExpected,
-			Values:     []float64{900, 1100, 1300, 1500, 1700, 1900},
+			Values:     []float64{1000, 1200, 1400, 1600, 1800, 2000},
 			Timestamps: timestampsExpected,
 		}
 		resultExpected := []netstorage.Result{r}
@@ -119,7 +120,7 @@ func TestExecSuccess(t *testing.T) {
 		q := `sort((label_set(time(), "foo", "bar"), label_set(time()+10, "foo", "baz")) offset 0s)`
 		r1 := netstorage.Result{
 			MetricName: metricNameExpected,
-			Values:     []float64{900, 1100, 1300, 1500, 1700, 1900},
+			Values:     []float64{1000, 1200, 1400, 1600, 1800, 2000},
 			Timestamps: timestampsExpected,
 		}
 		r1.MetricName.Tags = []storage.Tag{{
@@ -128,7 +129,7 @@ func TestExecSuccess(t *testing.T) {
 		}}
 		r2 := netstorage.Result{
 			MetricName: metricNameExpected,
-			Values:     []float64{910, 1110, 1310, 1510, 1710, 1910},
+			Values:     []float64{1010, 1210, 1410, 1610, 1810, 2010},
 			Timestamps: timestampsExpected,
 		}
 		r2.MetricName.Tags = []storage.Tag{{
@@ -149,7 +150,7 @@ func TestExecSuccess(t *testing.T) {
 		resultExpected := []netstorage.Result{r}
 		f(q, resultExpected)
 	})
-	t.Run("time() offset 100s", func(t *testing.T) {
+	t.Run("time() offset 1m40s0ms", func(t *testing.T) {
 		t.Parallel()
 		q := `time() offset 100s`
 		r := netstorage.Result{
@@ -209,7 +210,7 @@ func TestExecSuccess(t *testing.T) {
 		}}
 		r2 := netstorage.Result{
 			MetricName: metricNameExpected,
-			Values:     []float64{860, 1060, 1260, 1460, 1660, 1860},
+			Values:     []float64{810, 1010, 1210, 1410, 1610, 1810},
 			Timestamps: timestampsExpected,
 		}
 		r2.MetricName.Tags = []storage.Tag{{
@@ -224,7 +225,7 @@ func TestExecSuccess(t *testing.T) {
 		q := `sort((label_set(time() offset 100s, "foo", "bar"), label_set(time()+10, "foo", "baz") offset 50s) offset 400s)`
 		r1 := netstorage.Result{
 			MetricName: metricNameExpected,
-			Values:     []float64{300, 500, 700, 900, 1100, 1300},
+			Values:     []float64{400, 600, 800, 1000, 1200, 1400},
 			Timestamps: timestampsExpected,
 		}
 		r1.MetricName.Tags = []storage.Tag{{
@@ -233,7 +234,7 @@ func TestExecSuccess(t *testing.T) {
 		}}
 		r2 := netstorage.Result{
 			MetricName: metricNameExpected,
-			Values:     []float64{360, 560, 760, 960, 1160, 1360},
+			Values:     []float64{410, 610, 810, 1010, 1210, 1410},
 			Timestamps: timestampsExpected,
 		}
 		r2.MetricName.Tags = []storage.Tag{{
@@ -248,21 +249,21 @@ func TestExecSuccess(t *testing.T) {
 		q := `sort((label_set(time() offset -100s, "foo", "bar"), label_set(time()+10, "foo", "baz") offset -50s) offset -400s)`
 		r1 := netstorage.Result{
 			MetricName: metricNameExpected,
-			Values:     []float64{1260, 1460, 1660, 1860, 2060, 2260},
+			Values:     []float64{1400, 1600, 1800, 2000, 2200, 2400},
 			Timestamps: timestampsExpected,
 		}
 		r1.MetricName.Tags = []storage.Tag{{
 			Key:   []byte("foo"),
-			Value: []byte("baz"),
+			Value: []byte("bar"),
 		}}
 		r2 := netstorage.Result{
 			MetricName: metricNameExpected,
-			Values:     []float64{1300, 1500, 1700, 1900, 2100, 2300},
+			Values:     []float64{1410, 1610, 1810, 2010, 2210, 2410},
 			Timestamps: timestampsExpected,
 		}
 		r2.MetricName.Tags = []storage.Tag{{
 			Key:   []byte("foo"),
-			Value: []byte("bar"),
+			Value: []byte("baz"),
 		}}
 		resultExpected := []netstorage.Result{r1, r2}
 		f(q, resultExpected)
@@ -305,7 +306,7 @@ func TestExecSuccess(t *testing.T) {
 		q := `time()[300s] offset 100s`
 		r := netstorage.Result{
 			MetricName: metricNameExpected,
-			Values:     []float64{900, 1100, 1300, 1500, 1700, 1900},
+			Values:     []float64{800, 1000, 1200, 1400, 1600, 1800},
 			Timestamps: timestampsExpected,
 		}
 		resultExpected := []netstorage.Result{r}
@@ -338,7 +339,7 @@ func TestExecSuccess(t *testing.T) {
 		q := `timestamp(123)`
 		r := netstorage.Result{
 			MetricName: metricNameExpected,
-			Values:     []float64{900, 1100, 1300, 1500, 1700, 1900},
+			Values:     []float64{1000, 1200, 1400, 1600, 1800, 2000},
 			Timestamps: timestampsExpected,
 		}
 		resultExpected := []netstorage.Result{r}
@@ -349,7 +350,7 @@ func TestExecSuccess(t *testing.T) {
 		q := `timestamp(time())`
 		r := netstorage.Result{
 			MetricName: metricNameExpected,
-			Values:     []float64{900, 1100, 1300, 1500, 1700, 1900},
+			Values:     []float64{1000, 1200, 1400, 1600, 1800, 2000},
 			Timestamps: timestampsExpected,
 		}
 		resultExpected := []netstorage.Result{r}
@@ -360,7 +361,7 @@ func TestExecSuccess(t *testing.T) {
 		q := `timestamp(456/time()+123)`
 		r := netstorage.Result{
 			MetricName: metricNameExpected,
-			Values:     []float64{900, 1100, 1300, 1500, 1700, 1900},
+			Values:     []float64{1000, 1200, 1400, 1600, 1800, 2000},
 			Timestamps: timestampsExpected,
 		}
 		resultExpected := []netstorage.Result{r}
@@ -371,7 +372,7 @@ func TestExecSuccess(t *testing.T) {
 		q := `timestamp(time()>=1600)`
 		r := netstorage.Result{
 			MetricName: metricNameExpected,
-			Values:     []float64{nan, nan, nan, nan, 1700, 1900},
+			Values:     []float64{nan, nan, nan, 1600, 1800, 2000},
 			Timestamps: timestampsExpected,
 		}
 		resultExpected := []netstorage.Result{r}
@@ -3630,7 +3631,7 @@ func TestExecSuccess(t *testing.T) {
 		q := `round(geomean_over_time(alias(time()/100, "foobar")[3i]), 0.1)`
 		r := netstorage.Result{
 			MetricName: metricNameExpected,
-			Values:     []float64{6.8, 8.8, 10.9, 12.9, 14.9, 16.9},
+			Values:     []float64{7.8, 9.9, 11.9, 13.9, 15.9, 17.9},
 			Timestamps: timestampsExpected,
 		}
 		resultExpected := []netstorage.Result{r}
@@ -3652,7 +3653,7 @@ func TestExecSuccess(t *testing.T) {
 		q := `sum2_over_time(alias(time()/100, "foobar")[3i])`
 		r := netstorage.Result{
 			MetricName: metricNameExpected,
-			Values:     []float64{155, 251, 371, 515, 683, 875},
+			Values:     []float64{200, 308, 440, 596, 776, 980},
 			Timestamps: timestampsExpected,
 		}
 		resultExpected := []netstorage.Result{r}
@@ -3986,6 +3987,28 @@ func TestExecSuccess(t *testing.T) {
 		resultExpected := []netstorage.Result{r}
 		f(q, resultExpected)
 	})
+	t.Run(`count_gt_over_time`, func(t *testing.T) {
+		t.Parallel()
+		q := `count_gt_over_time(rand(0)[200s:10s], 0.7)`
+		r := netstorage.Result{
+			MetricName: metricNameExpected,
+			Values:     []float64{7, 6, 10, 6, 6, 5},
+			Timestamps: timestampsExpected,
+		}
+		resultExpected := []netstorage.Result{r}
+		f(q, resultExpected)
+	})
+	t.Run(`count_le_over_time`, func(t *testing.T) {
+		t.Parallel()
+		q := `count_le_over_time(rand(0)[200s:10s], 0.7)`
+		r := netstorage.Result{
+			MetricName: metricNameExpected,
+			Values:     []float64{13, 14, 10, 14, 14, 15},
+			Timestamps: timestampsExpected,
+		}
+		resultExpected := []netstorage.Result{r}
+		f(q, resultExpected)
+	})
 	t.Run(`increases_over_time`, func(t *testing.T) {
 		t.Parallel()
 		q := `increases_over_time(rand(0)[200s:10s])`
@@ -4713,10 +4736,10 @@ func TestExecSuccess(t *testing.T) {
 	})
 	t.Run(`ru(time() offset 1i, 2000)`, func(t *testing.T) {
 		t.Parallel()
-		q := `ru(time() offset 1i, 2000)`
+		q := `ru(time() offset 1.5i, 2000)`
 		r := netstorage.Result{
 			MetricName: metricNameExpected,
-			Values:     []float64{65, 55.00000000000001, 45, 35, 25, 15},
+			Values:     []float64{70, 60, 50, 40, 30, 20},
 			Timestamps: timestampsExpected,
 		}
 		resultExpected := []netstorage.Result{r}
@@ -4801,10 +4824,10 @@ func TestExecSuccess(t *testing.T) {
 	})
 	t.Run(`integrate(time())`, func(t *testing.T) {
 		t.Parallel()
-		q := `integrate(time()*1e-3)`
+		q := `integrate(time()/1e3)`
 		r := netstorage.Result{
 			MetricName: metricNameExpected,
-			Values:     []float64{160, 200, 240.00000000000003, 280, 320, 360},
+			Values:     []float64{160, 200, 240, 280, 320, 360},
 			Timestamps: timestampsExpected,
 		}
 		resultExpected := []netstorage.Result{r}
@@ -4903,7 +4926,7 @@ func TestExecSuccess(t *testing.T) {
 		q := `increase(2000-time())`
 		r := netstorage.Result{
 			MetricName: metricNameExpected,
-			Values:     []float64{1100, 900, 700, 500, 300, 100},
+			Values:     []float64{1000, 800, 600, 400, 200, 0},
 			Timestamps: timestampsExpected,
 		}
 		resultExpected := []netstorage.Result{r}
@@ -5874,7 +5897,7 @@ func TestExecError(t *testing.T) {
 			Start:    1000,
 			End:      2000,
 			Step:     100,
-			Deadline: netstorage.NewDeadline(time.Now(), time.Minute, ""),
+			Deadline: searchutils.NewDeadline(time.Now(), time.Minute, ""),
 		}
 		for i := 0; i < 4; i++ {
 			rv, err := Exec(ec, q, false)
@@ -6006,6 +6029,10 @@ func TestExecError(t *testing.T) {
 	f(`prometheus_buckets()`)
 	f(`buckets_limit()`)
 	f(`buckets_limit(1)`)
+	f(`share_le_over_time()`)
+	f(`share_gt_over_time()`)
+	f(`count_le_over_time()`)
+	f(`count_gt_over_time()`)
 
 	// Invalid argument type
 	f(`median_over_time({}, 2)`)

app/vmselect/promql/rollup.go

@@ -62,6 +62,8 @@ var rollupFuncs = map[string]newRollupFunc{
 	"tmax_over_time":        newRollupFuncOneArg(rollupTmax),
 	"share_le_over_time":    newRollupShareLE,
 	"share_gt_over_time":    newRollupShareGT,
+	"count_le_over_time":    newRollupCountLE,
+	"count_gt_over_time":    newRollupCountGT,
 	"histogram_over_time":   newRollupFuncOneArg(rollupHistogram),
 	"rollup":                newRollupFuncOneArg(rollupFake),
 	"rollup_rate":           newRollupFuncOneArg(rollupFake), // + rollupFuncsRemoveCounterResets
@@ -668,7 +670,7 @@ func derivValues(values []float64, timestamps []int64) {
 			values[i] = prevDeriv
 			continue
 		}
-		dt := float64(ts-prevTs) * 1e-3
+		dt := float64(ts-prevTs) / 1e3
 		prevDeriv = (v - prevValue) / dt
 		values[i] = prevDeriv
 		prevValue = v
@@ -788,7 +790,7 @@ func linearRegression(rfa *rollupFuncArg) (float64, float64) {
 		n = 0
 	}
 	for i, v := range values {
-		dt := float64(timestamps[i]-tFirst) * 1e-3
+		dt := float64(timestamps[i]-tFirst) / 1e3
 		vSum += v
 		tSum += dt
 		tvSum += dt * v
@@ -801,7 +803,7 @@ func linearRegression(rfa *rollupFuncArg) (float64, float64) {
 	k := (n*tvSum - tSum*vSum) / (n*ttSum - tSum*tSum)
 	v := (vSum - k*tSum) / n
 	// Adjust v to the last timestamp on the given time range.
-	v += k * (float64(timestamps[len(timestamps)-1]-tFirst) * 1e-3)
+	v += k * (float64(timestamps[len(timestamps)-1]-tFirst) / 1e3)
 	return v, k
 }
 
@@ -834,6 +836,25 @@ func countFilterGT(values []float64, gt float64) int {
 }
 
 func newRollupShareFilter(args []interface{}, countFilter func(values []float64, limit float64) int) (rollupFunc, error) {
+	rf, err := newRollupCountFilter(args, countFilter)
+	if err != nil {
+		return nil, err
+	}
+	return func(rfa *rollupFuncArg) float64 {
+		n := rf(rfa)
+		return n / float64(len(rfa.values))
+	}, nil
+}
+
+func newRollupCountLE(args []interface{}) (rollupFunc, error) {
+	return newRollupCountFilter(args, countFilterLE)
+}
+
+func newRollupCountGT(args []interface{}) (rollupFunc, error) {
+	return newRollupCountFilter(args, countFilterGT)
+}
+
+func newRollupCountFilter(args []interface{}, countFilter func(values []float64, limit float64) int) (rollupFunc, error) {
 	if err := expectRollupArgsNum(args, 2); err != nil {
 		return nil, err
 	}
@@ -849,8 +870,7 @@ func newRollupShareFilter(args []interface{}, countFilter func(values []float64,
 			return nan
 		}
 		limit := limits[rfa.idx]
-		n := countFilter(values, limit)
-		return float64(n) / float64(len(values))
+		return float64(countFilter(values, limit))
 	}
 	return rf, nil
 }
@@ -1035,7 +1055,7 @@ func rollupTmin(rfa *rollupFuncArg) float64 {
 			minTimestamp = timestamps[i]
 		}
 	}
-	return float64(minTimestamp) * 1e-3
+	return float64(minTimestamp) / 1e3
 }
 
 func rollupTmax(rfa *rollupFuncArg) float64 {
@@ -1054,7 +1074,7 @@ func rollupTmax(rfa *rollupFuncArg) float64 {
 			maxTimestamp = timestamps[i]
 		}
 	}
-	return float64(maxTimestamp) * 1e-3
+	return float64(maxTimestamp) / 1e3
 }
 
 func rollupSum(rfa *rollupFuncArg) float64 {
@@ -1283,7 +1303,7 @@ func rollupDerivFast(rfa *rollupFuncArg) float64 {
 	vEnd := values[len(values)-1]
 	tEnd := timestamps[len(timestamps)-1]
 	dv := vEnd - prevValue
-	dt := float64(tEnd-prevTimestamp) * 1e-3
+	dt := float64(tEnd-prevTimestamp) / 1e3
 	return dv / dt
 }
 
@@ -1309,7 +1329,7 @@ func rollupIderiv(rfa *rollupFuncArg) float64 {
 			// So just return nan
 			return nan
 		}
-		return (values[0] - rfa.prevValue) / (float64(timestamps[0]-rfa.prevTimestamp) * 1e-3)
+		return (values[0] - rfa.prevValue) / (float64(timestamps[0]-rfa.prevTimestamp) / 1e3)
 	}
 	vEnd := values[len(values)-1]
 	tEnd := timestamps[len(timestamps)-1]
@@ -1333,7 +1353,7 @@ func rollupIderiv(rfa *rollupFuncArg) float64 {
 	}
 	dv := vEnd - vStart
 	dt := tEnd - tStart
-	return dv / (float64(dt) * 1e-3)
+	return dv / (float64(dt) / 1e3)
 }
 
 func rollupLifetime(rfa *rollupFuncArg) float64 {
@@ -1343,12 +1363,12 @@ func rollupLifetime(rfa *rollupFuncArg) float64 {
 		if len(timestamps) < 2 {
 			return nan
 		}
-		return float64(timestamps[len(timestamps)-1]-timestamps[0]) * 1e-3
+		return float64(timestamps[len(timestamps)-1]-timestamps[0]) / 1e3
 	}
 	if len(timestamps) == 0 {
 		return nan
 	}
-	return float64(timestamps[len(timestamps)-1]-rfa.prevTimestamp) * 1e-3
+	return float64(timestamps[len(timestamps)-1]-rfa.prevTimestamp) / 1e3
 }
 
 func rollupLag(rfa *rollupFuncArg) float64 {
@@ -1358,9 +1378,9 @@ func rollupLag(rfa *rollupFuncArg) float64 {
 		if math.IsNaN(rfa.prevValue) {
 			return nan
 		}
-		return float64(rfa.currTimestamp-rfa.prevTimestamp) * 1e-3
+		return float64(rfa.currTimestamp-rfa.prevTimestamp) / 1e3
 	}
-	return float64(rfa.currTimestamp-timestamps[len(timestamps)-1]) * 1e-3
+	return float64(rfa.currTimestamp-timestamps[len(timestamps)-1]) / 1e3
 }
 
 func rollupScrapeInterval(rfa *rollupFuncArg) float64 {
@@ -1370,12 +1390,12 @@ func rollupScrapeInterval(rfa *rollupFuncArg) float64 {
 		if len(timestamps) < 2 {
 			return nan
 		}
-		return float64(timestamps[len(timestamps)-1]-timestamps[0]) * 1e-3 / float64(len(timestamps)-1)
+		return (float64(timestamps[len(timestamps)-1]-timestamps[0]) / 1e3) / float64(len(timestamps)-1)
 	}
 	if len(timestamps) == 0 {
 		return nan
 	}
-	return (float64(timestamps[len(timestamps)-1]-rfa.prevTimestamp) * 1e-3) / float64(len(timestamps))
+	return (float64(timestamps[len(timestamps)-1]-rfa.prevTimestamp) / 1e3) / float64(len(timestamps))
 }
 
 func rollupChanges(rfa *rollupFuncArg) float64 {
@@ -1663,37 +1683,32 @@ func rollupDistinct(rfa *rollupFuncArg) float64 {
 }
 
 func rollupIntegrate(rfa *rollupFuncArg) float64 {
-	prevTimestamp := rfa.prevTimestamp
-
 	// There is no need in handling NaNs here, since they must be cleaned up
 	// before calling rollup funcs.
 	values := rfa.values
 	timestamps := rfa.timestamps
-	if len(values) == 0 {
-		if math.IsNaN(rfa.prevValue) {
+	prevValue := rfa.prevValue
+	prevTimestamp := rfa.currTimestamp - rfa.window
+	if math.IsNaN(prevValue) {
+		if len(values) == 0 {
 			return nan
 		}
-		return 0
-	}
-	prevValue := rfa.prevValue
-	if math.IsNaN(prevValue) {
 		prevValue = values[0]
 		prevTimestamp = timestamps[0]
 		values = values[1:]
 		timestamps = timestamps[1:]
 	}
-	if len(values) == 0 {
-		return 0
-	}
 
 	var sum float64
 	for i, v := range values {
 		timestamp := timestamps[i]
-		dt := float64(timestamp-prevTimestamp) * 1e-3
-		sum += 0.5 * (v + prevValue) * dt
+		dt := float64(timestamp-prevTimestamp) / 1e3
+		sum += prevValue * dt
 		prevTimestamp = timestamp
 		prevValue = v
 	}
+	dt := float64(rfa.currTimestamp-prevTimestamp) / 1e3
+	sum += prevValue * dt
 	return sum
 }
 

app/vmselect/promql/rollup_test.go

@@ -139,16 +139,16 @@ func TestDerivValues(t *testing.T) {
 
 	values = append([]float64{}, testValues...)
 	derivValues(values, testTimestamps)
-	valuesExpected = []float64{-8900, 1111.111111111111, -1916.6666666666665, 2538.461538461538, -1818.1818181818182, 3611.111111111111,
-		-43500, 1882.3529411764705, -666.6666666666666, 400, 0, 0}
+	valuesExpected = []float64{-8900, 1111.111111111111, -1916.6666666666665, 2538.4615384615386, -1818.1818181818182, 3611.1111111111113,
+		-43500, 1882.3529411764705, -666.6666666666667, 400, 0, 0}
 	testRowsEqual(t, values, testTimestamps, valuesExpected, testTimestamps)
 
 	// remove counter resets
 	values = append([]float64{}, testValues...)
 	removeCounterResets(values)
 	derivValues(values, testTimestamps)
-	valuesExpected = []float64{3400, 1111.111111111111, 1750, 2538.461538461538, 3090.909090909091, 3611.111111111111,
-		6000, 1882.3529411764705, 1777.7777777777776, 400, 0, 0}
+	valuesExpected = []float64{3400, 1111.111111111111, 1750, 2538.4615384615386, 3090.909090909091, 3611.1111111111113,
+		6000, 1882.3529411764705, 1777.7777777777778, 400, 0, 0}
 	testRowsEqual(t, values, testTimestamps, valuesExpected, testTimestamps)
 
 	// duplicate timestamps
@@ -239,6 +239,52 @@ func TestRollupShareGTOverTime(t *testing.T) {
 	f(1000, 0)
 }
 
+func TestRollupCountLEOverTime(t *testing.T) {
+	f := func(le, vExpected float64) {
+		t.Helper()
+		les := []*timeseries{{
+			Values:     []float64{le},
+			Timestamps: []int64{123},
+		}}
+		var me metricsql.MetricExpr
+		args := []interface{}{&metricsql.RollupExpr{Expr: &me}, les}
+		testRollupFunc(t, "count_le_over_time", args, &me, vExpected)
+	}
+
+	f(-123, 0)
+	f(0, 0)
+	f(10, 0)
+	f(12, 1)
+	f(30, 2)
+	f(50, 9)
+	f(100, 11)
+	f(123, 12)
+	f(1000, 12)
+}
+
+func TestRollupCountGTOverTime(t *testing.T) {
+	f := func(gt, vExpected float64) {
+		t.Helper()
+		gts := []*timeseries{{
+			Values:     []float64{gt},
+			Timestamps: []int64{123},
+		}}
+		var me metricsql.MetricExpr
+		args := []interface{}{&metricsql.RollupExpr{Expr: &me}, gts}
+		testRollupFunc(t, "count_gt_over_time", args, &me, vExpected)
+	}
+
+	f(-123, 12)
+	f(0, 12)
+	f(10, 12)
+	f(12, 11)
+	f(30, 10)
+	f(50, 3)
+	f(100, 1)
+	f(123, 0)
+	f(1000, 0)
+}
+
 func TestRollupQuantileOverTime(t *testing.T) {
 	f := func(phi, vExpected float64) {
 		t.Helper()
@@ -385,7 +431,7 @@ func TestRollupNewRollupFuncSuccess(t *testing.T) {
 	f("stdvar_over_time", 945.7430555555555)
 	f("first_over_time", 123)
 	f("last_over_time", 34)
-	f("integrate", 5.4705)
+	f("integrate", 0.817)
 	f("distinct_over_time", 8)
 	f("ideriv", 0)
 	f("decreases_over_time", 5)
@@ -868,7 +914,7 @@ func TestRollupFuncsNoWindow(t *testing.T) {
 		}
 		rc.Timestamps = getTimestamps(rc.Start, rc.End, rc.Step)
 		values := rc.Do(nil, testValues, testTimestamps)
-		valuesExpected := []float64{0, -2879.310344827587, 558.0608793686592, 422.84569138276544, 0}
+		valuesExpected := []float64{0, -2879.310344827587, 558.0608793686595, 422.84569138276544, 0}
 		timestampsExpected := []int64{0, 40, 80, 120, 160}
 		testRowsEqual(t, values, rc.Timestamps, valuesExpected, timestampsExpected)
 	})
@@ -924,7 +970,7 @@ func TestRollupFuncsNoWindow(t *testing.T) {
 		}
 		rc.Timestamps = getTimestamps(rc.Start, rc.End, rc.Step)
 		values := rc.Do(nil, testValues, testTimestamps)
-		valuesExpected := []float64{nan, 1.526, 2.2795, 1.325, 0.34}
+		valuesExpected := []float64{nan, 2.148, 1.593, 1.156, 1.36}
 		timestampsExpected := []int64{0, 40, 80, 120, 160}
 		testRowsEqual(t, values, rc.Timestamps, valuesExpected, timestampsExpected)
 	})

app/vmselect/promql/transform.go

@@ -1030,6 +1030,12 @@ func transformSmoothExponential(tfa *transformFuncArg) ([]*timeseries, error) {
 	rvs := args[0]
 	for _, ts := range rvs {
 		values := skipLeadingNaNs(ts.Values)
+		for i, v := range values {
+			if !math.IsInf(v, 0) {
+				values = values[i:]
+				break
+			}
+		}
 		if len(values) == 0 {
 			continue
 		}
@@ -1040,6 +1046,10 @@ func transformSmoothExponential(tfa *transformFuncArg) ([]*timeseries, error) {
 			if math.IsNaN(v) {
 				continue
 			}
+			if math.IsInf(v, 0) {
+				values[i] = avg
+				continue
+			}
 			sf := sfsX[i]
 			if math.IsNaN(sf) {
 				sf = 1
@@ -1674,15 +1684,15 @@ func newTransformFuncZeroArgs(f func(tfa *transformFuncArg) float64) transformFu
 }
 
 func transformStep(tfa *transformFuncArg) float64 {
-	return float64(tfa.ec.Step) * 1e-3
+	return float64(tfa.ec.Step) / 1e3
 }
 
 func transformStart(tfa *transformFuncArg) float64 {
-	return float64(tfa.ec.Start) * 1e-3
+	return float64(tfa.ec.Start) / 1e3
 }
 
 func transformEnd(tfa *transformFuncArg) float64 {
-	return float64(tfa.ec.End) * 1e-3
+	return float64(tfa.ec.End) / 1e3
 }
 
 // copyTimeseriesMetricNames returns a copy of tss with real copy of MetricNames,

app/vmselect/searchutils/searchutils.go

@@ -0,0 +1,167 @@
+package searchutils
+
+import (
+	"flag"
+	"fmt"
+	"math"
+	"net/http"
+	"strconv"
+	"strings"
+	"time"
+
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fasttime"
+	"github.com/VictoriaMetrics/metricsql"
+)
+
+var (
+	maxExportDuration = flag.Duration("search.maxExportDuration", time.Hour*24*30, "The maximum duration for /api/v1/export call")
+	maxQueryDuration  = flag.Duration("search.maxQueryDuration", time.Second*30, "The maximum duration for search query execution")
+)
+
+// GetTime returns time from the given argKey query arg.
+func GetTime(r *http.Request, argKey string, defaultValue int64) (int64, error) {
+	argValue := r.FormValue(argKey)
+	if len(argValue) == 0 {
+		return defaultValue, nil
+	}
+	secs, err := strconv.ParseFloat(argValue, 64)
+	if err != nil {
+		// Try parsing string format
+		t, err := time.Parse(time.RFC3339, argValue)
+		if err != nil {
+			// Handle Prometheus'-provided minTime and maxTime.
+			// See https://github.com/prometheus/client_golang/issues/614
+			switch argValue {
+			case prometheusMinTimeFormatted:
+				return minTimeMsecs, nil
+			case prometheusMaxTimeFormatted:
+				return maxTimeMsecs, nil
+			}
+			// Try parsing duration relative to the current time
+			d, err1 := metricsql.DurationValue(argValue, 0)
+			if err1 != nil {
+				return 0, fmt.Errorf("cannot parse %q=%q: %w", argKey, argValue, err)
+			}
+			if d > 0 {
+				d = -d
+			}
+			t = time.Now().Add(time.Duration(d) * time.Millisecond)
+		}
+		secs = float64(t.UnixNano()) / 1e9
+	}
+	msecs := int64(secs * 1e3)
+	if msecs < minTimeMsecs {
+		msecs = 0
+	}
+	if msecs > maxTimeMsecs {
+		msecs = maxTimeMsecs
+	}
+	return msecs, nil
+}
+
+var (
+	// These constants were obtained from https://github.com/prometheus/prometheus/blob/91d7175eaac18b00e370965f3a8186cc40bf9f55/web/api/v1/api.go#L442
+	// See https://github.com/prometheus/client_golang/issues/614 for details.
+	prometheusMinTimeFormatted = time.Unix(math.MinInt64/1000+62135596801, 0).UTC().Format(time.RFC3339Nano)
+	prometheusMaxTimeFormatted = time.Unix(math.MaxInt64/1000-62135596801, 999999999).UTC().Format(time.RFC3339Nano)
+)
+
+const (
+	// These values prevent from overflow when storing msec-precision time in int64.
+	minTimeMsecs = 0 // use 0 instead of `int64(-1<<63) / 1e6` because the storage engine doesn't actually support negative time
+	maxTimeMsecs = int64(1<<63-1) / 1e6
+)
+
+// GetDuration returns duration from the given argKey query arg.
+func GetDuration(r *http.Request, argKey string, defaultValue int64) (int64, error) {
+	argValue := r.FormValue(argKey)
+	if len(argValue) == 0 {
+		return defaultValue, nil
+	}
+	secs, err := strconv.ParseFloat(argValue, 64)
+	if err != nil {
+		// Try parsing string format
+		d, err := metricsql.DurationValue(argValue, 0)
+		if err != nil {
+			return 0, fmt.Errorf("cannot parse %q=%q: %w", argKey, argValue, err)
+		}
+		secs = float64(d) / 1000
+	}
+	msecs := int64(secs * 1e3)
+	if msecs <= 0 || msecs > maxDurationMsecs {
+		return 0, fmt.Errorf("%q=%dms is out of allowed range [%d ... %d]", argKey, msecs, 0, int64(maxDurationMsecs))
+	}
+	return msecs, nil
+}
+
+const maxDurationMsecs = 100 * 365 * 24 * 3600 * 1000
+
+// GetDeadlineForQuery returns deadline for the given query r.
+func GetDeadlineForQuery(r *http.Request, startTime time.Time) Deadline {
+	dMax := maxQueryDuration.Milliseconds()
+	return getDeadlineWithMaxDuration(r, startTime, dMax, "-search.maxQueryDuration")
+}
+
+// GetDeadlineForExport returns deadline for the given request to /api/v1/export.
+func GetDeadlineForExport(r *http.Request, startTime time.Time) Deadline {
+	dMax := maxExportDuration.Milliseconds()
+	return getDeadlineWithMaxDuration(r, startTime, dMax, "-search.maxExportDuration")
+}
+
+func getDeadlineWithMaxDuration(r *http.Request, startTime time.Time, dMax int64, flagHint string) Deadline {
+	d, err := GetDuration(r, "timeout", 0)
+	if err != nil {
+		d = 0
+	}
+	if d <= 0 || d > dMax {
+		d = dMax
+	}
+	timeout := time.Duration(d) * time.Millisecond
+	return NewDeadline(startTime, timeout, flagHint)
+}
+
+// GetBool returns boolean value from the given argKey query arg.
+func GetBool(r *http.Request, argKey string) bool {
+	argValue := r.FormValue(argKey)
+	switch strings.ToLower(argValue) {
+	case "", "0", "f", "false", "no":
+		return false
+	default:
+		return true
+	}
+}
+
+// Deadline contains deadline with the corresponding timeout for pretty error messages.
+type Deadline struct {
+	deadline uint64
+
+	timeout  time.Duration
+	flagHint string
+}
+
+// NewDeadline returns deadline for the given timeout.
+//
+// flagHint must contain a hit for command-line flag, which could be used
+// in order to increase timeout.
+func NewDeadline(startTime time.Time, timeout time.Duration, flagHint string) Deadline {
+	return Deadline{
+		deadline: uint64(startTime.Add(timeout).Unix()),
+		timeout:  timeout,
+		flagHint: flagHint,
+	}
+}
+
+// Exceeded returns true if deadline is exceeded.
+func (d *Deadline) Exceeded() bool {
+	return fasttime.UnixTimestamp() > d.deadline
+}
+
+// Deadline returns deadline in unix timestamp seconds.
+func (d *Deadline) Deadline() uint64 {
+	return d.deadline
+}
+
+// String returns human-readable string representation for d.
+func (d *Deadline) String() string {
+	return fmt.Sprintf("%.3f seconds; the timeout can be adjusted with `%s` command-line flag", d.timeout.Seconds(), d.flagHint)
+}

app/vmselect/searchutils/searchutils_test.go

@@ -0,0 +1,78 @@
+package searchutils
+
+import (
+	"fmt"
+	"net/http"
+	"net/url"
+	"testing"
+)
+
+func TestGetTimeSuccess(t *testing.T) {
+	f := func(s string, timestampExpected int64) {
+		t.Helper()
+		urlStr := fmt.Sprintf("http://foo.bar/baz?s=%s", url.QueryEscape(s))
+		r, err := http.NewRequest("GET", urlStr, nil)
+		if err != nil {
+			t.Fatalf("unexpected error in NewRequest: %s", err)
+		}
+
+		// Verify defaultValue
+		ts, err := GetTime(r, "foo", 123)
+		if err != nil {
+			t.Fatalf("unexpected error when obtaining default time from GetTime(%q): %s", s, err)
+		}
+		if ts != 123 {
+			t.Fatalf("unexpected default value for GetTime(%q); got %d; want %d", s, ts, 123)
+		}
+
+		// Verify timestampExpected
+		ts, err = GetTime(r, "s", 123)
+		if err != nil {
+			t.Fatalf("unexpected error in GetTime(%q): %s", s, err)
+		}
+		if ts != timestampExpected {
+			t.Fatalf("unexpected timestamp for GetTime(%q); got %d; want %d", s, ts, timestampExpected)
+		}
+	}
+
+	f("2019-07-07T20:01:02Z", 1562529662000)
+	f("2019-07-07T20:47:40+03:00", 1562521660000)
+	f("-292273086-05-16T16:47:06Z", minTimeMsecs)
+	f("292277025-08-18T07:12:54.999999999Z", maxTimeMsecs)
+	f("1562529662.324", 1562529662324)
+	f("-9223372036.854", minTimeMsecs)
+	f("-9223372036.855", minTimeMsecs)
+	f("9223372036.855", maxTimeMsecs)
+}
+
+func TestGetTimeError(t *testing.T) {
+	f := func(s string) {
+		t.Helper()
+		urlStr := fmt.Sprintf("http://foo.bar/baz?s=%s", url.QueryEscape(s))
+		r, err := http.NewRequest("GET", urlStr, nil)
+		if err != nil {
+			t.Fatalf("unexpected error in NewRequest: %s", err)
+		}
+
+		// Verify defaultValue
+		ts, err := GetTime(r, "foo", 123)
+		if err != nil {
+			t.Fatalf("unexpected error when obtaining default time from GetTime(%q): %s", s, err)
+		}
+		if ts != 123 {
+			t.Fatalf("unexpected default value for GetTime(%q); got %d; want %d", s, ts, 123)
+		}
+
+		// Verify timestampExpected
+		_, err = GetTime(r, "s", 123)
+		if err == nil {
+			t.Fatalf("expecting non-nil error in GetTime(%q)", s)
+		}
+	}
+
+	f("foo")
+	f("2019-07-07T20:01:02Zisdf")
+	f("2019-07-07T20:47:40+03:00123")
+	f("-292273086-05-16T16:47:07Z")
+	f("292277025-08-18T07:12:54.999999998Z")
+}

app/vmstorage/main.go

@@ -132,6 +132,16 @@ func SearchTagValues(tagKey []byte, maxTagValues int, deadline uint64) ([]string
 	return values, err
 }
 
+// SearchTagValueSuffixes returns all the tag value suffixes for the given tagKey and tagValuePrefix on the given tr.
+//
+// This allows implementing https://graphite-api.readthedocs.io/en/latest/api.html#metrics-find or similar APIs.
+func SearchTagValueSuffixes(tr storage.TimeRange, tagKey, tagValuePrefix []byte, delimiter byte, maxTagValueSuffixes int, deadline uint64) ([]string, error) {
+	WG.Add(1)
+	suffixes, err := Storage.SearchTagValueSuffixes(tr, tagKey, tagValuePrefix, delimiter, maxTagValueSuffixes, deadline)
+	WG.Done()
+	return suffixes, err
+}
+
 // SearchTagEntries searches for tag entries.
 func SearchTagEntries(maxTagKeys, maxTagValues int, deadline uint64) ([]storage.TagEntry, error) {
 	WG.Add(1)
@@ -456,6 +466,13 @@ func registerStorageMetrics() {
 		return float64(m().SlowMetricNameLoads)
 	})
 
+	metrics.NewGauge(`vm_timestamps_blocks_merged_total`, func() float64 {
+		return float64(m().TimestampsBlocksMerged)
+	})
+	metrics.NewGauge(`vm_timestamps_bytes_saved_total`, func() float64 {
+		return float64(m().TimestampsBytesSaved)
+	})
+
 	metrics.NewGauge(`vm_rows{type="storage/big"}`, func() float64 {
 		return float64(tm().BigRowsCount)
 	})

dashboards/vmagent.json

@@ -5,7 +5,7 @@
       "type": "grafana",
       "id": "grafana",
       "name": "Grafana",
-      "version": "7.0.3"
+      "version": "7.1.1"
     },
     {
       "type": "panel",
@@ -51,12 +51,12 @@
       }
     ]
   },
-  "description": "Overview for VictoriaMetrics vmagent v1.38.1 or higher",
+  "description": "Overview for VictoriaMetrics vmagent v1.40.0 or higher",
   "editable": true,
   "gnetId": null,
   "graphTooltip": 1,
   "id": null,
-  "iteration": 1594939790534,
+  "iteration": 1598997251171,
   "links": [
     {
       "icon": "doc",
@@ -136,9 +136,10 @@
           ],
           "fields": "",
           "values": false
-        }
+        },
+        "textMode": "auto"
       },
-      "pluginVersion": "7.0.3",
+      "pluginVersion": "7.1.1",
       "targets": [
         {
           "expr": "sum(vm_promscrape_targets{job=~\"$job\", instance=~\"$instance\", status=\"up\"})",
@@ -199,9 +200,10 @@
           ],
           "fields": "",
           "values": false
-        }
+        },
+        "textMode": "auto"
       },
-      "pluginVersion": "7.0.3",
+      "pluginVersion": "7.1.1",
       "targets": [
         {
           "expr": "sum(vm_promscrape_targets{job=~\"$job\", instance=~\"$instance\", status=\"down\"})",
@@ -265,9 +267,10 @@
           ],
           "fields": "",
           "values": false
-        }
+        },
+        "textMode": "auto"
       },
-      "pluginVersion": "7.0.3",
+      "pluginVersion": "7.1.1",
       "targets": [
         {
           "expr": "sum(increase(vm_log_messages_total{job=~\"$job\", instance=~\"$instance\", level!=\"info\"}[30m]))",
@@ -323,9 +326,10 @@
           ],
           "fields": "",
           "values": false
-        }
+        },
+        "textMode": "auto"
       },
-      "pluginVersion": "7.0.3",
+      "pluginVersion": "7.1.1",
       "targets": [
         {
           "expr": "sum(vm_persistentqueue_bytes_pending{job=~\"$job\", instance=~\"$instance\"})",
@@ -444,7 +448,8 @@
       "datasource": "$ds",
       "fieldConfig": {
         "defaults": {
-          "custom": {}
+          "custom": {},
+          "links": []
         },
         "overrides": []
       },
@@ -477,10 +482,8 @@
       "lines": true,
       "linewidth": 1,
       "nullPointMode": "null as zero",
-      "options": {
-        "dataLinks": []
-      },
       "percentage": false,
+      "pluginVersion": "7.1.1",
       "pointradius": 2,
       "points": false,
       "renderer": "flot",
@@ -550,7 +553,8 @@
       "description": "Shows in/out samples rate including push and pull models. \n\nThe out-rate could be different to in-rate because of replication or additional timeseries added by vmagent for every scraped target.",
       "fieldConfig": {
         "defaults": {
-          "custom": {}
+          "custom": {},
+          "links": []
         },
         "overrides": []
       },
@@ -576,10 +580,8 @@
       "lines": true,
       "linewidth": 1,
       "nullPointMode": "null",
-      "options": {
-        "dataLinks": []
-      },
       "percentage": false,
+      "pluginVersion": "7.1.1",
       "pointradius": 2,
       "points": false,
       "renderer": "flot",
@@ -657,7 +659,8 @@
       "description": "Shows the rate of requests served by vmagent HTTP server.",
       "fieldConfig": {
         "defaults": {
-          "custom": {}
+          "custom": {},
+          "links": []
         },
         "overrides": []
       },
@@ -684,10 +687,8 @@
       "linewidth": 1,
       "links": [],
       "nullPointMode": "null as zero",
-      "options": {
-        "dataLinks": []
-      },
       "percentage": false,
+      "pluginVersion": "7.1.1",
       "pointradius": 2,
       "points": false,
       "renderer": "flot",
@@ -727,7 +728,7 @@
           "label": null,
           "logBase": 1,
           "max": null,
-          "min": null,
+          "min": "0",
           "show": true
         },
         {
@@ -753,7 +754,8 @@
       "description": "Network usage shows the bytes rate for data accepted by vmagent and pushed via remotewrite protocol.\nDiscrepancies are possible because of different protocols used for ingesting, scraping and writing data.",
       "fieldConfig": {
         "defaults": {
-          "custom": {}
+          "custom": {},
+          "links": []
         },
         "overrides": []
       },
@@ -779,10 +781,8 @@
       "lines": true,
       "linewidth": 1,
       "nullPointMode": "null",
-      "options": {
-        "dataLinks": []
-      },
       "percentage": false,
+      "pluginVersion": "7.1.1",
       "pointradius": 2,
       "points": false,
       "renderer": "flot",
@@ -859,7 +859,8 @@
       "description": "Errors rate shows rate for multiple metrics that track possible errors in vmagent, such as network or parsing errors.",
       "fieldConfig": {
         "defaults": {
-          "custom": {}
+          "custom": {},
+          "links": []
         },
         "overrides": []
       },
@@ -892,10 +893,8 @@
         }
       ],
       "nullPointMode": "null as zero",
-      "options": {
-        "dataLinks": []
-      },
       "percentage": false,
+      "pluginVersion": "7.1.1",
       "pointradius": 2,
       "points": false,
       "renderer": "flot",
@@ -959,7 +958,7 @@
           "label": null,
           "logBase": 1,
           "max": null,
-          "min": null,
+          "min": "0",
           "show": true
         },
         {
@@ -985,7 +984,8 @@
       "description": "Shows rate of dropped samples from persistent queue. VMagent drops samples from queue if in-memory and on-disk queues are full and it is unable to flush them to remote storage.\nThe max size of on-disk queue is configured by `-remoteWrite.maxDiskUsagePerURL` flag.",
       "fieldConfig": {
         "defaults": {
-          "custom": {}
+          "custom": {},
+          "links": []
         },
         "overrides": []
       },
@@ -1018,10 +1018,8 @@
         }
       ],
       "nullPointMode": "null",
-      "options": {
-        "dataLinks": []
-      },
       "percentage": false,
+      "pluginVersion": "7.1.1",
       "pointradius": 2,
       "points": false,
       "renderer": "flot",
@@ -1061,7 +1059,7 @@
           "label": null,
           "logBase": 1,
           "max": null,
-          "min": null,
+          "min": "0",
           "show": true
         },
         {
@@ -1087,7 +1085,8 @@
       "description": "Shows the persistent queue size of pending samples in bytes which hasn't been flushed to remote storage yet. \n\nIncreasing of value might be a sign of connectivity issues. In such cases, vmagent starts to flush pending data on disk with attempt to send it later once connection is restored.\n\nRemote write URLs are hidden by default but might be unveiled once `-remoteWrite.showURL` is set to true.",
       "fieldConfig": {
         "defaults": {
-          "custom": {}
+          "custom": {},
+          "links": []
         },
         "overrides": []
       },
@@ -1119,10 +1118,8 @@
         }
       ],
       "nullPointMode": "null",
-      "options": {
-        "dataLinks": []
-      },
       "percentage": false,
+      "pluginVersion": "7.1.1",
       "pointradius": 2,
       "points": false,
       "renderer": "flot",
@@ -1162,7 +1159,7 @@
           "label": null,
           "logBase": 1,
           "max": null,
-          "min": null,
+          "min": "0",
           "show": true
         },
         {
@@ -1188,7 +1185,8 @@
       "description": "Shows the rate of dropped samples due to relabeling. \nMetric tracks drops for `-remoteWrite.relabelConfig` configuration only.",
       "fieldConfig": {
         "defaults": {
-          "custom": {}
+          "custom": {},
+          "links": []
         },
         "overrides": []
       },
@@ -1221,10 +1219,8 @@
         }
       ],
       "nullPointMode": "null",
-      "options": {
-        "dataLinks": []
-      },
       "percentage": false,
+      "pluginVersion": "7.1.1",
       "pointradius": 2,
       "points": false,
       "renderer": "flot",
@@ -1270,7 +1266,7 @@
           "label": null,
           "logBase": 1,
           "max": null,
-          "min": null,
+          "min": "0",
           "show": true
         },
         {
@@ -1307,18 +1303,7 @@
           "fieldConfig": {
             "defaults": {
               "custom": {},
-              "decimals": 0,
-              "mappings": [],
-              "thresholds": {
-                "mode": "absolute",
-                "steps": [
-                  {
-                    "color": "green",
-                    "value": null
-                  }
-                ]
-              },
-              "unit": "short"
+              "links": []
             },
             "overrides": []
           },
@@ -1344,11 +1329,8 @@
           "lines": true,
           "linewidth": 1,
           "nullPointMode": "null",
-          "options": {
-            "dataLinks": []
-          },
           "percentage": false,
-          "pluginVersion": "7.0.3",
+          "pluginVersion": "7.1.1",
           "pointradius": 2,
           "points": false,
           "renderer": "flot",
@@ -1390,7 +1372,7 @@
               "label": null,
               "logBase": 1,
               "max": null,
-              "min": null,
+              "min": "0",
               "show": true
             },
             {
@@ -1416,18 +1398,7 @@
           "fieldConfig": {
             "defaults": {
               "custom": {},
-              "decimals": 0,
-              "mappings": [],
-              "thresholds": {
-                "mode": "absolute",
-                "steps": [
-                  {
-                    "color": "green",
-                    "value": null
-                  }
-                ]
-              },
-              "unit": "short"
+              "links": []
             },
             "overrides": []
           },
@@ -1453,11 +1424,8 @@
           "lines": true,
           "linewidth": 1,
           "nullPointMode": "null",
-          "options": {
-            "dataLinks": []
-          },
           "percentage": false,
-          "pluginVersion": "7.0.3",
+          "pluginVersion": "7.1.1",
           "pointradius": 2,
           "points": false,
           "renderer": "flot",
@@ -1499,7 +1467,7 @@
               "label": null,
               "logBase": 1,
               "max": null,
-              "min": null,
+              "min": "0",
               "show": true
             },
             {
@@ -1524,7 +1492,8 @@
           "datasource": "$ds",
           "fieldConfig": {
             "defaults": {
-              "custom": {}
+              "custom": {},
+              "links": []
             },
             "overrides": []
           },
@@ -1550,10 +1519,8 @@
           "lines": true,
           "linewidth": 1,
           "nullPointMode": "null",
-          "options": {
-            "dataLinks": []
-          },
           "percentage": false,
+          "pluginVersion": "7.1.1",
           "pointradius": 2,
           "points": false,
           "renderer": "flot",
@@ -1629,7 +1596,8 @@
           "datasource": "$ds",
           "fieldConfig": {
             "defaults": {
-              "custom": {}
+              "custom": {},
+              "links": []
             },
             "overrides": []
           },
@@ -1655,10 +1623,8 @@
           "lines": true,
           "linewidth": 1,
           "nullPointMode": "null",
-          "options": {
-            "dataLinks": []
-          },
           "percentage": false,
+          "pluginVersion": "7.1.1",
           "pointradius": 2,
           "points": false,
           "renderer": "flot",
@@ -1716,7 +1682,7 @@
               "label": null,
               "logBase": 1,
               "max": null,
-              "min": null,
+              "min": "0",
               "show": true
             },
             {
@@ -1741,7 +1707,8 @@
           "datasource": "$ds",
           "fieldConfig": {
             "defaults": {
-              "custom": {}
+              "custom": {},
+              "links": []
             },
             "overrides": []
           },
@@ -1767,10 +1734,8 @@
           "lines": true,
           "linewidth": 1,
           "nullPointMode": "null",
-          "options": {
-            "dataLinks": []
-          },
           "percentage": false,
+          "pluginVersion": "7.1.1",
           "pointradius": 2,
           "points": false,
           "renderer": "flot",
@@ -1817,7 +1782,7 @@
               "label": null,
               "logBase": 1,
               "max": null,
-              "min": null,
+              "min": "0",
               "show": true
             },
             {
@@ -1862,7 +1827,7 @@
             "y": 17
           },
           "heatmap": {},
-          "hideZeroBuckets": true,
+          "hideZeroBuckets": false,
           "highlightCards": true,
           "id": 33,
           "legend": {
@@ -1871,7 +1836,7 @@
           "reverseYBuckets": false,
           "targets": [
             {
-              "expr": "prometheus_buckets(sum(rate(vm_promscrape_scrape_duration_seconds_bucket{job=~\"$job\", instance=~\"$instance\"}[$__interval])) by(vmrange))",
+              "expr": "buckets_limit(12, prometheus_buckets(sum(rate(vm_promscrape_scrape_duration_seconds_bucket{job=~\"$job\", instance=~\"$instance\"}[$__interval])) by(vmrange)))",
               "format": "heatmap",
               "interval": "",
               "intervalFactor": 10,
@@ -1929,7 +1894,8 @@
           "description": "Shows the rate of write requests served by ingestserver (UDP, TCP connections) and HTTP server.",
           "fieldConfig": {
             "defaults": {
-              "custom": {}
+              "custom": {},
+              "links": []
             },
             "overrides": []
           },
@@ -1956,10 +1922,8 @@
           "linewidth": 1,
           "links": [],
           "nullPointMode": "null as zero",
-          "options": {
-            "dataLinks": []
-          },
           "percentage": false,
+          "pluginVersion": "7.1.1",
           "pointradius": 2,
           "points": false,
           "renderer": "flot",
@@ -2005,7 +1969,7 @@
               "label": null,
               "logBase": 1,
               "max": null,
-              "min": null,
+              "min": "0",
               "show": true
             },
             {
@@ -2031,7 +1995,8 @@
           "description": "Shows the rate of write errors in ingestserver (UDP, TCP connections) and HTTP server.",
           "fieldConfig": {
             "defaults": {
-              "custom": {}
+              "custom": {},
+              "links": []
             },
             "overrides": []
           },
@@ -2058,10 +2023,8 @@
           "linewidth": 1,
           "links": [],
           "nullPointMode": "null as zero",
-          "options": {
-            "dataLinks": []
-          },
           "percentage": false,
+          "pluginVersion": "7.1.1",
           "pointradius": 2,
           "points": false,
           "renderer": "flot",
@@ -2107,7 +2070,7 @@
               "label": null,
               "logBase": 1,
               "max": null,
-              "min": null,
+              "min": "0",
               "show": true
             },
             {
@@ -2133,7 +2096,8 @@
           "description": "Shows the rate of parsed rows from write or scrape requests.",
           "fieldConfig": {
             "defaults": {
-              "custom": {}
+              "custom": {},
+              "links": []
             },
             "overrides": []
           },
@@ -2160,10 +2124,8 @@
           "linewidth": 1,
           "links": [],
           "nullPointMode": "null as zero",
-          "options": {
-            "dataLinks": []
-          },
           "percentage": false,
+          "pluginVersion": "7.1.1",
           "pointradius": 2,
           "points": false,
           "renderer": "flot",
@@ -2203,7 +2165,7 @@
               "label": null,
               "logBase": 1,
               "max": null,
-              "min": null,
+              "min": "0",
               "show": true
             },
             {
@@ -2229,7 +2191,8 @@
           "description": "Tracks the rate of dropped invalid rows because of errors while unmarshaling write requests. The exact errors messages will be printed in logs.",
           "fieldConfig": {
             "defaults": {
-              "custom": {}
+              "custom": {},
+              "links": []
             },
             "overrides": []
           },
@@ -2255,10 +2218,8 @@
           "lines": true,
           "linewidth": 1,
           "nullPointMode": "null",
-          "options": {
-            "dataLinks": []
-          },
           "percentage": false,
+          "pluginVersion": "7.1.1",
           "pointradius": 2,
           "points": false,
           "renderer": "flot",
@@ -2298,7 +2259,7 @@
               "label": null,
               "logBase": 1,
               "max": null,
-              "min": null,
+              "min": "0",
               "show": true
             },
             {
@@ -2339,7 +2300,8 @@
           "description": "Shows the rate of requests to configured remote write endpoints by url and status code.\n\nRemote write URLs are hidden by default but might be unveiled once `-remoteWrite.showURL` is set to true.\n\n",
           "fieldConfig": {
             "defaults": {
-              "custom": {}
+              "custom": {},
+              "links": []
             },
             "overrides": []
           },
@@ -2365,10 +2327,8 @@
           "lines": true,
           "linewidth": 1,
           "nullPointMode": "null",
-          "options": {
-            "dataLinks": []
-          },
           "percentage": false,
+          "pluginVersion": "7.1.1",
           "pointradius": 2,
           "points": false,
           "renderer": "flot",
@@ -2409,7 +2369,7 @@
               "label": null,
               "logBase": 1,
               "max": null,
-              "min": null,
+              "min": "0",
               "show": true
             },
             {
@@ -2435,7 +2395,8 @@
           "description": "Shows the global rate for number of written bytes via remote write connections.",
           "fieldConfig": {
             "defaults": {
-              "custom": {}
+              "custom": {},
+              "links": []
             },
             "overrides": []
           },
@@ -2461,10 +2422,8 @@
           "lines": true,
           "linewidth": 1,
           "nullPointMode": "null",
-          "options": {
-            "dataLinks": []
-          },
           "percentage": false,
+          "pluginVersion": "7.1.1",
           "pointradius": 2,
           "points": false,
           "renderer": "flot",
@@ -2504,7 +2463,7 @@
               "label": null,
               "logBase": 1,
               "max": null,
-              "min": null,
+              "min": "0",
               "show": true
             },
             {
@@ -2530,7 +2489,8 @@
           "description": "Shows requests retry rate by url. Number of retries is unlimited but protected with delays up to 1m between attempts.\n\nRemote write URLs are hidden by default but might be unveiled once `-remoteWrite.showURL` is set to true.\n\n",
           "fieldConfig": {
             "defaults": {
-              "custom": {}
+              "custom": {},
+              "links": []
             },
             "overrides": []
           },
@@ -2556,10 +2516,8 @@
           "lines": true,
           "linewidth": 1,
           "nullPointMode": "null",
-          "options": {
-            "dataLinks": []
-          },
           "percentage": false,
+          "pluginVersion": "7.1.1",
           "pointradius": 2,
           "points": false,
           "renderer": "flot",
@@ -2625,7 +2583,8 @@
           "description": "Shows current number of established connections to remote write endpoints.\n\n",
           "fieldConfig": {
             "defaults": {
-              "custom": {}
+              "custom": {},
+              "links": []
             },
             "overrides": []
           },
@@ -2651,10 +2610,8 @@
           "lines": true,
           "linewidth": 1,
           "nullPointMode": "null",
-          "options": {
-            "dataLinks": []
-          },
           "percentage": false,
+          "pluginVersion": "7.1.1",
           "pointradius": 2,
           "points": false,
           "renderer": "flot",
@@ -2694,7 +2651,7 @@
               "label": null,
               "logBase": 1,
               "max": null,
-              "min": null,
+              "min": "0",
               "show": true
             },
             {
@@ -2748,7 +2705,7 @@
           "reverseYBuckets": false,
           "targets": [
             {
-              "expr": "prometheus_buckets(sum(rate(vmagent_remotewrite_block_size_rows_bucket{job=~\"$job\", instance=~\"$instance\"}[$__interval])) by(vmrange))  ",
+              "expr": "buckets_limit(12, prometheus_buckets(sum(rate(vmagent_remotewrite_block_size_rows_bucket{job=~\"$job\", instance=~\"$instance\"}[$__interval])) by(vmrange)))",
               "format": "heatmap",
               "interval": "",
               "intervalFactor": 10,
@@ -2819,7 +2776,7 @@
           "reverseYBuckets": false,
           "targets": [
             {
-              "expr": "prometheus_buckets(sum(rate(vmagent_remotewrite_block_size_bytes_bucket{job=~\"$job\", instance=~\"$instance\"}[$__interval])) by(vmrange))  ",
+              "expr": "buckets_limit(12, prometheus_buckets(sum(rate(vmagent_remotewrite_block_size_bytes_bucket{job=~\"$job\", instance=~\"$instance\"}[$__interval])) by(vmrange)))",
               "format": "heatmap",
               "interval": "",
               "intervalFactor": 10,
@@ -2890,7 +2847,7 @@
           "reverseYBuckets": false,
           "targets": [
             {
-              "expr": "prometheus_buckets(sum(rate(vmagent_remotewrite_duration_seconds_bucket{job=~\"$job\", instance=~\"$instance\"}[$__interval])) by(vmrange))  ",
+              "expr": "buckets_limit(12, prometheus_buckets(sum(rate(vmagent_remotewrite_duration_seconds_bucket{job=~\"$job\", instance=~\"$instance\"}[$__interval])) by(vmrange)))",
               "format": "heatmap",
               "interval": "",
               "intervalFactor": 10,
@@ -2948,7 +2905,8 @@
           "description": "Shows the CPU usage per vmagent instance. \nIf you think that usage is abnormal or unexpected pls file an issue and attach CPU profile if possible.",
           "fieldConfig": {
             "defaults": {
-              "custom": {}
+              "custom": {},
+              "links": []
             },
             "overrides": []
           },
@@ -2958,7 +2916,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 44
+            "y": 5
           },
           "hiddenSeries": false,
           "id": 35,
@@ -2981,10 +2939,8 @@
             }
           ],
           "nullPointMode": "null",
-          "options": {
-            "dataLinks": []
-          },
           "percentage": false,
+          "pluginVersion": "7.1.1",
           "pointradius": 2,
           "points": false,
           "renderer": "flot",
@@ -3025,7 +2981,7 @@
               "label": null,
               "logBase": 1,
               "max": null,
-              "min": null,
+              "min": "0",
               "show": true
             },
             {
@@ -3051,7 +3007,8 @@
           "description": "Amount of used memory (resident)\n\nIf you think that usage is abnormal or unexpected pls file an issue and attach memory profile if possible.",
           "fieldConfig": {
             "defaults": {
-              "custom": {}
+              "custom": {},
+              "links": []
             },
             "overrides": []
           },
@@ -3061,7 +3018,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 44
+            "y": 5
           },
           "hiddenSeries": false,
           "id": 37,
@@ -3084,10 +3041,8 @@
             }
           ],
           "nullPointMode": "null",
-          "options": {
-            "dataLinks": []
-          },
           "percentage": false,
+          "pluginVersion": "7.1.1",
           "pointradius": 2,
           "points": false,
           "renderer": "flot",
@@ -3152,7 +3107,8 @@
           "datasource": "$ds",
           "fieldConfig": {
             "defaults": {
-              "custom": {}
+              "custom": {},
+              "links": []
             },
             "overrides": []
           },
@@ -3162,7 +3118,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 52
+            "y": 13
           },
           "hiddenSeries": false,
           "id": 41,
@@ -3179,10 +3135,8 @@
           "linewidth": 1,
           "links": [],
           "nullPointMode": "null",
-          "options": {
-            "dataLinks": []
-          },
           "percentage": false,
+          "pluginVersion": "7.1.1",
           "pointradius": 2,
           "points": false,
           "renderer": "flot",
@@ -3249,7 +3203,8 @@
           "datasource": "$ds",
           "fieldConfig": {
             "defaults": {
-              "custom": {}
+              "custom": {},
+              "links": []
             },
             "overrides": []
           },
@@ -3259,7 +3214,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 52
+            "y": 13
           },
           "hiddenSeries": false,
           "id": 39,
@@ -3276,10 +3231,8 @@
           "linewidth": 1,
           "links": [],
           "nullPointMode": "null",
-          "options": {
-            "dataLinks": []
-          },
           "percentage": false,
+          "pluginVersion": "7.1.1",
           "pointradius": 2,
           "points": false,
           "renderer": "flot",
@@ -3346,7 +3299,8 @@
           "datasource": "$ds",
           "fieldConfig": {
             "defaults": {
-              "custom": {}
+              "custom": {},
+              "links": []
             },
             "overrides": []
           },
@@ -3356,7 +3310,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 60
+            "y": 21
           },
           "hiddenSeries": false,
           "id": 43,
@@ -3373,10 +3327,8 @@
           "linewidth": 1,
           "links": [],
           "nullPointMode": "null",
-          "options": {
-            "dataLinks": []
-          },
           "percentage": false,
+          "pluginVersion": "7.1.1",
           "pointradius": 2,
           "points": false,
           "renderer": "flot",
@@ -3417,7 +3369,7 @@
               "label": null,
               "logBase": 1,
               "max": null,
-              "min": null,
+              "min": "0",
               "show": true
             },
             {
@@ -3440,7 +3392,7 @@
     }
   ],
   "refresh": false,
-  "schemaVersion": 25,
+  "schemaVersion": 26,
   "style": "dark",
   "tags": [
     "vmagent",

deployment/docker/Makefile

@@ -4,7 +4,7 @@ DOCKER_NAMESPACE := victoriametrics
 
 ROOT_IMAGE ?= alpine:3.12
 CERTS_IMAGE := alpine:3.12
-GO_BUILDER_IMAGE := golang:1.15.0
+GO_BUILDER_IMAGE := golang:1.15.2
 BUILDER_IMAGE := local/builder:2.0.0-$(shell echo $(GO_BUILDER_IMAGE) | tr : _)
 BASE_IMAGE := local/base:1.1.1-$(shell echo $(ROOT_IMAGE) | tr : _)-$(shell echo $(CERTS_IMAGE) | tr : _)
 

docs/Articles.md

@@ -25,11 +25,13 @@
 * [Prometheus storage: tech terms for humans](https://medium.com/@valyala/prometheus-storage-technical-terms-for-humans-4ab4de6c3d48)
 * [Billy: how VictoriaMetrics deals with more than 500 billion rows](https://medium.com/@valyala/billy-how-victoriametrics-deals-with-more-than-500-billion-rows-e82ff8f725da)
 * [How to migrate data from Prometheus to VictoriaMetrics](https://medium.com/@romanhavronenko/victoriametrics-how-to-migrate-data-from-prometheus-d44a6728f043)
+* [Filtering and modifying time series during import to VictoriaMetrics](https://medium.com/@romanhavronenko/victoriametrics-how-to-migrate-data-from-prometheus-filtering-and-modifying-time-series-6d40cea4bf21)
 
 
 ## Third-party articles and slides
 
 * [Better Prometheus rate() function with VictoriaMetrics](https://www.percona.com/blog/2020/02/28/better-prometheus-rate-function-with-victoriametrics/)
+* [Making peace with Prometheus rate()](https://blog.doit-intl.com/making-peace-with-prometheus-rate-43a3ea75c4cf)
 * [Infrastructure monitoring with Prometheus at Zerodha](https://zerodha.tech/blog/infra-monitoring-at-zerodha/)
 * [Sismology: Iguana Solutions’ Monitoring System](https://medium.com/@IG1.com/sismology-iguana-solutions-monitoring-system-f46e4170447f)
 * [Monitoring K8S with VictoriaMetrics](https://docs.google.com/presentation/d/1g7yUyVEaAp4tPuRy-MZbPXKqJ1z78_5VKuV841aQfsg/edit)
@@ -40,3 +42,6 @@
 * [What are Open Source Time Series Databases?](https://www.iunera.com/kraken/fabric/time-series-database/)
 * [Evaluating performance and correctness](https://www.robustperception.io/evaluating-performance-and-correctness)
 * [Running VictoriaMetrics on Raspberry PI](https://stas.starikevich.com/posts/raspberry-pi-4-prometheus/)
+* [Calculating the Error of Quantile Estimation with Histograms](https://linuxczar.net/blog/2020/08/13/histogram-error/)
+* [Monitoring private clouds with VictoriaMetrics at LeroyMerlin](https://www.youtube.com/watch?v=74swsWqf0Uc)
+* [Monitoring Kubernetes with VictoriaMetrics+Prometheus](https://speakerdeck.com/bo0km4n/victoriametrics-plus-prometheusdegou-zhu-surufu-shu-kubernetesfalsejian-shi-ji-pan)

docs/Cluster-VictoriaMetrics.md

@@ -180,7 +180,7 @@ or [an alternative dashboard for VictoriaMetrics cluster](https://grafana.com/gr
      - `prometheus/api/v1/import/csv` - for importing arbitrary CSV data. See [these docs](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/README.md#how-to-import-csv-data) for details.
      - `prometheus/api/v1/import/prometheus` - for importing data in Prometheus exposition format. See [these docs](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/README.md#how-to-import-data-in-prometheus-exposition-format) for details.
 
-* URLs for querying: `http://<vmselect>:8481/select/<accountID>/prometheus/<suffix>`, where:
+* URLs for [Prmetheus querying API](https://prometheus.io/docs/prometheus/latest/querying/api/): `http://<vmselect>:8481/select/<accountID>/prometheus/<suffix>`, where:
   - `<accountID>` is an arbitrary number identifying data namespace for the query (aka tenant)
   - `<suffix>` may have the following values:
     - `api/v1/query` - performs [PromQL instant query](https://prometheus.io/docs/prometheus/latest/querying/api/#instant-queries).
@@ -194,6 +194,13 @@ or [an alternative dashboard for VictoriaMetrics cluster](https://grafana.com/gr
     - `api/v1/status/active_queries` - for currently executed active queries. Note that every `vmselect` maintains an independent list of active queries,
       which is returned in the response.
 
+* URLs for [Graphite Metrics API](https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api): `http://<vmselect>:8481/select/<accountID>/graphite/<suffix>`, where:
+    - `<accountID>` is an arbitrary number identifying data namespace for query (aka tenant)
+    - `<suffix>` may have the following values:
+      - `metrics/find` - searches Graphite metrics. See [these docs](https://graphite-api.readthedocs.io/en/latest/api.html#metrics-find).
+      - `metrics/expand` - expands Graphite metrics. See [these docs](https://graphite-api.readthedocs.io/en/latest/api.html#metrics-expand).
+      - `metrics/index.json` - returns all the metric names. See [these docs](https://graphite-api.readthedocs.io/en/latest/api.html#metrics-index-json).
+
 * URL for time series deletion: `http://<vmselect>:8481/delete/<accountID>/prometheus/api/v1/admin/tsdb/delete_series?match[]=<timeseries_selector_for_delete>`.
   Note that the `delete_series` handler should be used only in exceptional cases such as deletion of accidentally ingested incorrect time series. It shouldn't
   be used on a regular basis, since it carries non-zero overhead.

docs/MetricsQL.md

@@ -115,6 +115,8 @@ This functionality can be tried at [an editable Grafana dashboard](http://play-g
   Example: `share_le_over_time(memory_usage_bytes[24h], 100*1024*1024)` returns the share of time series values for the last 24 hours when memory usage was below or equal to 100MB.
 - `share_gt_over_time(m[d], gt)` - returns share (in the range 0..1) of values in `m` over `d`, which are bigger than `gt`. Useful for calculating SLI and SLO.
   Example: `share_gt_over_time(up[24h], 0)` - returns service availability for the last 24 hours.
+- `count_le_over_time(m[d], le)` - returns the number of raw samples for `m` over `d`, which don't exceed `le`.
+- `count_gt_over_time(m[d], gt)` - returns the number of raw samples for `m` over `d`, which are bigger than `gt`.
 - `tmin_over_time(m[d])` - returns timestamp for the minimum value for `m` over `d` time range.
 - `tmax_over_time(m[d])` - returns timestamp for the maximum value for `m` over `d` time range.
 - `aggr_over_time(("aggr_func1", "aggr_func2", ...), m[d])` - simultaneously calculates all the listed `aggr_func*` for `m` over `d` time range.
@@ -124,7 +126,7 @@ This functionality can be tried at [an editable Grafana dashboard](http://play-g
   for the given `phi` in the range `[0..1]`.
 - `last_over_time(m[d])` - returns the last value for `m` on the time range `d`.
 - `first_over_time(m[d])` - returns the first value for `m` on the time range `d`.
-- `outliersk(N, q) by (group)` - returns up to `N` outlier time series for `q` in every `group`. Outlier time series have the highest deviation from the `median(m)`.
+- `outliersk(N, q) by (group)` - returns up to `N` outlier time series for `q` in every `group`. Outlier time series have the highest deviation from the `median(q)`.
   This aggregate function is useful to detect anomalies across groups of similar time series.
 - `ascent_over_time(m[d])` - returns the sum of positive deltas between adjancent data points in `m` over `d`. Useful for tracking height gains in GPS track.
 - `descent_over_time(m[d])` - returns the absolute sum of negative deltas between adjancent data points in `m` over `d`. Useful for tracking height loss in GPS track.

docs/Single-server-VictoriaMetrics.md

@@ -78,6 +78,7 @@ See [features available for enterprise customers](https://github.com/VictoriaMet
   * [OpenTSDB put message](#sending-data-via-telnet-put-protocol) if `-opentsdbListenAddr` is set.
   * [HTTP OpenTSDB /api/put requests](#sending-opentsdb-data-via-http-apiput-requests) if `-opentsdbHTTPListenAddr` is set.
   * [/api/v1/import](#how-to-import-time-series-data).
+  * [Prometheus exposition format](#how-to-import-data-in-prometheus-exposition-format).
   * [Arbitrary CSV data](#how-to-import-csv-data).
 * Supports metrics' relabeling. See [these docs](#relabeling) for details.
 * Ideally works with big amounts of time series data from Kubernetes, IoT sensors, connected cars, industrial telemetry, financial data and various Enterprise workloads.
@@ -102,6 +103,8 @@ See [features available for enterprise customers](https://github.com/VictoriaMet
 * [How to import data in Prometheus exposition format](#how-to-import-data-in-prometheus-exposition-format)
 * [How to import CSV data](#how-to-import-csv-data)
 * [Prometheus querying API usage](#prometheus-querying-api-usage)
+  * [Prometheus querying API enhancements](#prometheus-querying-api-enhancements)
+* [Graphite Metrics API usage](#graphite-metrics-api-usage)
 * [How to build from sources](#how-to-build-from-sources)
   * [Development build](#development-build)
   * [Production build](#production-build)
@@ -391,9 +394,11 @@ The `/api/v1/export` endpoint should return the following response:
 
 ### Querying Graphite data
 
-Data sent to VictoriaMetrics via `Graphite plaintext protocol` may be read either via
-[Prometheus querying API](#prometheus-querying-api-usage)
-or via [go-graphite/carbonapi](https://github.com/go-graphite/carbonapi/blob/master/cmd/carbonapi/carbonapi.example.prometheus.yaml).
+Data sent to VictoriaMetrics via `Graphite plaintext protocol` may be read via the following APIs:
+
+* [Prometheus querying API](#prometheus-querying-api-usage)
+* Metric names can be explored via [Graphite metrics API](#graphite-metrics-api-usage)
+* [go-graphite/carbonapi](https://github.com/go-graphite/carbonapi/blob/master/cmd/carbonapi/carbonapi.example.prometheus.yaml)
 
 ### How to send data from OpenTSDB-compatible agents
 
@@ -516,6 +521,9 @@ The following response should be returned:
 {"metric":{"__name__":"ask","market":"NYSE","ticker":"GOOG"},"values":[1.23],"timestamps":[1583865146495]}
 ```
 
+Extra labels may be added to all the imported lines by passing `extra_label=name=value` query args.
+For example, `/api/v1/import/csv?extra_label=foo=bar` would add `"foo":"bar"` label to all the imported lines.
+
 Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail.
 
 
@@ -534,12 +542,18 @@ The following command may be used for verifying the imported data:
 curl -G 'http://localhost:8428/api/v1/export' -d 'match={__name__=~"foo"}'
 ```
 
-It should return somethins like the following:
+It should return something like the following:
 
 ```
 {"metric":{"__name__":"foo","bar":"baz"},"values":[123],"timestamps":[1594370496905]}
 ```
 
+Extra labels may be added to all the imported metrics by passing `extra_label=name=value` query args.
+For example, `/api/v1/import/prometheus?extra_label=foo=bar` would add `{foo="bar"}` label to all the imported metrics.
+
+If timestamp is missing in `<metric> <value> <timestamp>` Prometheus exposition format line, then the current timestamp is used during data ingestion.
+It can be overriden by passing unix timestamp in *milliseconds* via `timestamp` query arg. For example, `/api/v1/import/prometheus?timestamp=1594370496905`.
+
 VictoriaMetrics accepts arbitrary number of lines in a single request to `/api/v1/import/prometheus`, i.e. it supports data streaming.
 
 VictoriaMetrics also may scrape Prometheus targets - see [these docs](#how-to-scrape-prometheus-exporters-such-as-node-exporter).
@@ -558,6 +572,13 @@ VictoriaMetrics supports the following handlers from [Prometheus querying API](h
 
 These handlers can be queried from Prometheus-compatible clients such as Grafana or curl.
 
+#### Prometheus querying API enhancements
+
+Additionally to unix timestamps and [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) VictoriaMetrics accepts relative times in `time`, `start` and `end` query args.
+For example, the following query would return data for the last 30 minutes: `/api/v1/query_range?start=-30m&query=...`.
+
+By default, VictoriaMetrics returns time series for the last 5 minutes from /api/v1/series, while the Prometheus API defaults to all time.  Use `start` and `end` to select a different time range.
+
 VictoriaMetrics accepts additional args for `/api/v1/labels` and `/api/v1/label/.../values` handlers.
 See [this feature request](https://github.com/prometheus/prometheus/issues/6178) for details:
 
@@ -571,6 +592,21 @@ Additionally VictoriaMetrics provides the following handlers:
 * `/api/v1/labels/count` - it returns a list of `label: values_count` entries. It can be used for determining labels with the maximum number of values.
 * `/api/v1/status/active_queries` - it returns a list of currently running queries.
 
+
+### Graphite Metrics API usage
+
+VictoriaMetrics supports the following handlers from [Graphite Metrics API](https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api):
+
+* [/metrics/find](https://graphite-api.readthedocs.io/en/latest/api.html#metrics-find)
+* [/metrics/expand](https://graphite-api.readthedocs.io/en/latest/api.html#metrics-expand)
+* [/metrics/index.json](https://graphite-api.readthedocs.io/en/latest/api.html#metrics-index-json)
+
+VictoriaMetrics accepts the following additional query args at `/metrics/find` and `/metrics/expand`:
+  * `label` - for selecting arbitrary label values. By default `label=__name__`, i.e. metric names are selected.
+  * `delimiter` - for using different delimiters in metric name hierachy. For example, `/metrics/find?delimiter=_&query=node_*` would return all the metric name prefixes
+    that start with `node_`. By default `delimiter=.`.
+
+
 ### How to build from sources
 
 We recommend using either [binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) or
@@ -677,7 +713,8 @@ for metrics to delete. After that all the time series matching the given selecto
 the deleted time series isn't freed instantly - it is freed during subsequent [background merges of data files](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282).
 
 It is recommended verifying which metrics will be deleted with the call to `http://<victoria-metrics-addr>:8428/api/v1/series?match[]=<timeseries_selector_for_delete>`
-before actually deleting the metrics.
+before actually deleting the metrics.  By default this query will only scan active series in the past 5 minutes, so you may need to
+adjust `start` and `end` to a suitable range to achieve match hits.
 
 The `/api/v1/admin/tsdb/delete_series` handler may be protected with `authKey` if `-deleteAuthKey` command-line flag is set.
 
@@ -759,6 +796,9 @@ curl -H 'Accept-Encoding: gzip' http://source-victoriametrics:8428/api/v1/export
 curl -X POST -H 'Content-Encoding: gzip' http://destination-victoriametrics:8428/api/v1/import -T exported_data.jsonl.gz
 ```
 
+Extra labels may be added to all the imported time series by passing `extra_label=name=value` query args.
+For example, `/api/v1/import?extra_label=foo=bar` would add `"foo":"bar"` label to all the imported time series.
+
 Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail.
 
 Each request to `/api/v1/import` can load up to a single vCPU core on VictoriaMetrics. Import speed can be improved by splitting the original file into smaller parts
@@ -876,7 +916,8 @@ with the enabled de-duplication. See [this section](#deduplication) for details.
 
 VictoriaMetrics de-duplicates data points if `-dedup.minScrapeInterval` command-line flag
 is set to positive duration. For example, `-dedup.minScrapeInterval=60s` would de-duplicate data points
-on the same time series if they are located closer than 60s to each other.
+on the same time series if they fall within the same discrete 60s bucket.  The earliest data point will be kept.  In the case of equal timestamps, an arbitrary data point will be kept.
+
 The de-duplication reduces disk space usage if multiple identically configured Prometheus instances in HA pair
 write data to the same VictoriaMetrics instance. Note that these Prometheus instances must have identical
 `external_labels` section in their configs, so they write data to the same time series.
@@ -1082,7 +1123,7 @@ for data with timestamps close to the current time.
 
 ### Data updates
 
-VictoriaMetrics doesn't support updating already exiting sample values to new ones. It stores all the ingested data points
+VictoriaMetrics doesn't support updating already existing sample values to new ones. It stores all the ingested data points
 for the same time series with identical timestamps. While is possible substituting old time series with new time series via
 [removal of old time series](#how-to-delete-timeseries) and then [writing new time series](#backfilling), this approach
 should be used only for one-off updates. It shouldn't be used for frequent updates because of non-zero overhead related to data removal.

docs/vmagent.md

@@ -218,8 +218,7 @@ If you have suggestions, improvements or found a bug - feel free to open an issu
 * When `vmagent` scrapes many unreliable targets, it can flood error log with scrape errors. These errors can be suppressed
   by passing `-promscrape.suppressScrapeErrors` command-line flag to `vmagent`. The most recent scrape error per each target can be observed at `http://vmagent-host:8429/targets`.
 
-* It is recommended to increase `-remoteWrite.queues` if `vmagent` collects more than 100K samples per second
-  and `vmagent_remotewrite_pending_data_bytes` metric exported at `http://vmagent-host:8429/metrics` page constantly grows.
+* It is recommended to increase `-remoteWrite.queues` if `vmagent_remotewrite_pending_data_bytes` metric exported at `http://vmagent-host:8429/metrics` page constantly grows.
 
 * If you see gaps on the data pushed by `vmagent` to remote storage when `-remoteWrite.maxDiskUsagePerURL` is set, then try increasing `-remoteWrite.queues`.
   Such gaps may appear because `vmagent` cannot keep up with sending the collected data to remote storage, so it starts dropping the buffered data

docs/vmalert.md

@@ -194,8 +194,12 @@ The shortlist of configuration flags is the following:
     	Supports array of values separated by comma or specified via multiple flags.
   -external.url string
     	External URL is used as alert's source for sent alerts to the notifier
+  -http.connTimeout duration
+    	Incoming http connections are closed after the configured timeout. This may help spreading incoming load among a cluster of services behind load balancer. Note that the real timeout may be bigger by up to 10% as a protection from Thundering herd problem (default 2m0s)
   -http.disableResponseCompression
     	Disable compression of HTTP responses for saving CPU resources. By default compression is enabled to save network bandwidth
+  -http.idleConnTimeout duration
+    	Timeout for incoming idle http connections (default 1m0s)
   -http.maxGracefulShutdownDuration duration
     	The maximum duration for graceful shutdown of HTTP server. Highly loaded server may require increased value for graceful shutdown (default 7s)
   -http.pathPrefix string
@@ -216,8 +220,9 @@ The shortlist of configuration flags is the following:
     	Minimum level of errors to log. Possible values: INFO, WARN, ERROR, FATAL, PANIC (default "INFO")
   -loggerOutput string
     	Output for the logs. Supported values: stderr, stdout (default "stderr")
-  -memory.allowedBytes int
+  -memory.allowedBytes value
     	Allowed size of system memory VictoriaMetrics caches may occupy. This option overrides -memory.allowedPercent if set to non-zero value. Too low value may increase cache miss rate, which usually results in higher CPU and disk IO usage. Too high value may evict too much data from OS page cache, which will result in higher disk IO usage
+    	Supports the following optional suffixes for values: KB, MB, GB, KiB, MiB, GiB (default 0)
   -memory.allowedPercent float
     	Allowed percent of system memory VictoriaMetrics caches may occupy. See also -memory.allowedBytes. Too low value may increase cache miss rate, which usually results in higher CPU and disk IO usage. Too high value may evict too much data from OS page cache, which will result in higher disk IO usage (default 60)
   -metricsAuthKey string
@@ -293,8 +298,8 @@ The shortlist of configuration flags is the following:
     	Path to the file with alert rules. 
     	Supports patterns. Flag can be specified multiple times. 
     	Examples:
-    	 -rule /path/to/file. Path to a single file with alerting rules
-    	 -rule dir/*.yaml -rule /*.yaml. Relative path to all .yaml files in "dir" folder, 
+    	 -rule="/path/to/file". Path to a single file with alerting rules
+    	 -rule="dir/*.yaml" -rule="/*.yaml". Relative path to all .yaml files in "dir" folder, 
     	absolute path to all .yaml files in root.
     	Rule files may contain %{ENV_VAR} placeholders, which are substituted by the corresponding env vars.
     	Supports array of values separated by comma or specified via multiple flags.

docs/vmauth.md

@@ -140,3 +140,68 @@ curl -s http://<vmauth-host>:8427/debug/pprof/profile > cpu.pprof
 The command for collecting CPU profile waits for 30 seconds before returning.
 
 The collected profiles may be analyzed with [go tool pprof](https://github.com/google/pprof).
+
+
+### Advanced usage
+
+Pass `-help` command-line arg to `vmauth` in order to see all the configuration options:
+
+```
+./vmauth -help
+
+vmauth authenticates and authorizes incoming requests and proxies them to VictoriaMetrics.
+
+See the docs at https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmauth/README.md .
+
+  -auth.config string
+    	Path to auth config. See https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmauth/README.md for details on the format of this auth config
+  -enableTCP6
+    	Whether to enable IPv6 for listening and dialing. By default only IPv4 TCP is used
+  -envflag.enable
+    	Whether to enable reading flags from environment variables additionally to command line. Command line flag values have priority over values from environment vars. Flags are read only from command line if this flag isn't set
+  -envflag.prefix string
+    	Prefix for environment variables if -envflag.enable is set
+  -http.connTimeout duration
+    	Incoming http connections are closed after the configured timeout. This may help spreading incoming load among a cluster of services behind load balancer. Note that the real timeout may be bigger by up to 10% as a protection from Thundering herd problem (default 2m0s)
+  -http.disableResponseCompression
+    	Disable compression of HTTP responses for saving CPU resources. By default compression is enabled to save network bandwidth
+  -http.idleConnTimeout duration
+    	Timeout for incoming idle http connections (default 1m0s)
+  -http.maxGracefulShutdownDuration duration
+    	The maximum duration for graceful shutdown of HTTP server. Highly loaded server may require increased value for graceful shutdown (default 7s)
+  -http.pathPrefix string
+    	An optional prefix to add to all the paths handled by http server. For example, if '-http.pathPrefix=/foo/bar' is set, then all the http requests will be handled on '/foo/bar/*' paths. This may be useful for proxied requests. See https://www.robustperception.io/using-external-urls-and-proxies-with-prometheus
+  -http.shutdownDelay duration
+    	Optional delay before http server shutdown. During this dealy the servier returns non-OK responses from /health page, so load balancers can route new requests to other servers
+  -httpAuth.password string
+    	Password for HTTP Basic Auth. The authentication is disabled if -httpAuth.username is empty
+  -httpAuth.username string
+    	Username for HTTP Basic Auth. The authentication is disabled if empty. See also -httpAuth.password
+  -httpListenAddr string
+    	TCP address to listen for http connections (default ":8427")
+  -loggerErrorsPerSecondLimit int
+    	Per-second limit on the number of ERROR messages. If more than the given number of errors are emitted per second, then the remaining errors are suppressed. Zero value disables the rate limit (default 10)
+  -loggerFormat string
+    	Format for logs. Possible values: default, json (default "default")
+  -loggerLevel string
+    	Minimum level of errors to log. Possible values: INFO, WARN, ERROR, FATAL, PANIC (default "INFO")
+  -loggerOutput string
+    	Output for the logs. Supported values: stderr, stdout (default "stderr")
+  -memory.allowedBytes value
+    	Allowed size of system memory VictoriaMetrics caches may occupy. This option overrides -memory.allowedPercent if set to non-zero value. Too low value may increase cache miss rate, which usually results in higher CPU and disk IO usage. Too high value may evict too much data from OS page cache, which will result in higher disk IO usage
+    	Supports the following optional suffixes for values: KB, MB, GB, KiB, MiB, GiB (default 0)
+  -memory.allowedPercent float
+    	Allowed percent of system memory VictoriaMetrics caches may occupy. See also -memory.allowedBytes. Too low value may increase cache miss rate, which usually results in higher CPU and disk IO usage. Too high value may evict too much data from OS page cache, which will result in higher disk IO usage (default 60)
+  -metricsAuthKey string
+    	Auth key for /metrics. It overrides httpAuth settings
+  -pprofAuthKey string
+    	Auth key for /debug/pprof. It overrides httpAuth settings
+  -tls
+    	Whether to enable TLS (aka HTTPS) for incoming requests. -tlsCertFile and -tlsKeyFile must be set if -tls is set
+  -tlsCertFile string
+    	Path to file with TLS certificate. Used only if -tls is set. Prefer ECDSA certs instead of RSA certs, since RSA certs are slow
+  -tlsKeyFile string
+    	Path to file with TLS key. Used only if -tls is set
+  -version
+    	Show VictoriaMetrics version
+```

docs/vmbackup.md

@@ -138,7 +138,7 @@ Run `vmbackup -help` in order to see all the available options:
     	Path to file with S3 configs. Configs are loaded from default location if not set.
     	See https://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html
   -configProfile string
-    	Profile name for S3 configs (default "default")
+    	Profile name for S3 configs. If no set, the value of the environment variable will be loaded (AWS_PROFILE or AWS_DEFAULT_PROFILE), or if both not set, DefaultSharedConfigProfile is used
   -credsFilePath string
     	Path to file with GCS or S3 credentials. Credentials are loaded from default locations if not set.
     	See https://cloud.google.com/iam/docs/creating-managing-service-account-keys and https://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html
@@ -161,18 +161,20 @@ Run `vmbackup -help` in order to see all the available options:
     	Minimum level of errors to log. Possible values: INFO, WARN, ERROR, FATAL, PANIC (default "INFO")
   -loggerOutput string
     	Output for the logs. Supported values: stderr, stdout (default "stderr")
-  -maxBytesPerSecond int
+  -maxBytesPerSecond value
     	The maximum upload speed. There is no limit if it is set to 0
-  -memory.allowedBytes int
+    	Supports the following optional suffixes for values: KB, MB, GB, KiB, MiB, GiB (default 0)
+  -memory.allowedBytes value
     	Allowed size of system memory VictoriaMetrics caches may occupy. This option overrides -memory.allowedPercent if set to non-zero value. Too low value may increase cache miss rate, which usually results in higher CPU and disk IO usage. Too high value may evict too much data from OS page cache, which will result in higher disk IO usage
+    	Supports the following optional suffixes for values: KB, MB, GB, KiB, MiB, GiB (default 0)
   -memory.allowedPercent float
     	Allowed percent of system memory VictoriaMetrics caches may occupy. See also -memory.allowedBytes. Too low value may increase cache miss rate, which usually results in higher CPU and disk IO usage. Too high value may evict too much data from OS page cache, which will result in higher disk IO usage (default 60)
   -origin string
     	Optional origin directory on the remote storage with old backup for server-side copying when performing full backup. This speeds up full backups
   -snapshot.createURL string
-    	VictoriaMetrics create snapshot url. When this is given a snapshot will automatically be created during backup.Example: http://victoriametrics:8428/snaphsot/create
+    	VictoriaMetrics create snapshot url. When this is given a snapshot will automatically be created during backup. Example: http://victoriametrics:8428/snaphsot/create
   -snapshot.deleteURL string
-    	VictoriaMetrics delete snapshot url. Optional. Will be generated from snapshotCreateURL if not provided. All created snaphosts will be automatically deleted.Example: http://victoriametrics:8428/snaphsot/delete
+    	VictoriaMetrics delete snapshot url. Optional. Will be generated from -snapshot.createURL if not provided. All created snaphosts will be automatically deleted. Example: http://victoriametrics:8428/snaphsot/delete
   -snapshotName string
     	Name for the snapshot to backup. See https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/README.md#how-to-work-with-snapshots
   -storageDataPath string

docs/vmrestore.md

@@ -42,7 +42,7 @@ Run `vmrestore -help` in order to see all the available options:
     	Path to file with S3 configs. Configs are loaded from default location if not set.
     	See https://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html
   -configProfile string
-    	Profile name for S3 configs (default "default")
+    	Profile name for S3 configs. If no set, the value of the environment variable will be loaded (AWS_PROFILE or AWS_DEFAULT_PROFILE), or if both not set, DefaultSharedConfigProfile is used
   -credsFilePath string
     	Path to file with GCS or S3 credentials. Credentials are loaded from default locations if not set.
     	See https://cloud.google.com/iam/docs/creating-managing-service-account-keys and https://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html
@@ -62,10 +62,12 @@ Run `vmrestore -help` in order to see all the available options:
     	Minimum level of errors to log. Possible values: INFO, WARN, ERROR, FATAL, PANIC (default "INFO")
   -loggerOutput string
     	Output for the logs. Supported values: stderr, stdout (default "stderr")
-  -maxBytesPerSecond int
+  -maxBytesPerSecond value
     	The maximum download speed. There is no limit if it is set to 0
-  -memory.allowedBytes int
+    	Supports the following optional suffixes for values: KB, MB, GB, KiB, MiB, GiB (default 0)
+  -memory.allowedBytes value
     	Allowed size of system memory VictoriaMetrics caches may occupy. This option overrides -memory.allowedPercent if set to non-zero value. Too low value may increase cache miss rate, which usually results in higher CPU and disk IO usage. Too high value may evict too much data from OS page cache, which will result in higher disk IO usage
+    	Supports the following optional suffixes for values: KB, MB, GB, KiB, MiB, GiB (default 0)
   -memory.allowedPercent float
     	Allowed percent of system memory VictoriaMetrics caches may occupy. See also -memory.allowedBytes. Too low value may increase cache miss rate, which usually results in higher CPU and disk IO usage. Too high value may evict too much data from OS page cache, which will result in higher disk IO usage (default 60)
   -skipBackupCompleteCheck

go.mod

@@ -1,31 +1,31 @@
 module github.com/VictoriaMetrics/VictoriaMetrics
 
 require (
-	cloud.google.com/go v0.63.0 // indirect
-	cloud.google.com/go/storage v1.10.0
+	cloud.google.com/go/storage v1.11.0
 	github.com/VictoriaMetrics/fastcache v1.5.7
 
 	// Do not use the original github.com/valyala/fasthttp because of issues
 	// like https://github.com/valyala/fasthttp/commit/996610f021ff45fdc98c2ce7884d5fa4e7f9199b
 	github.com/VictoriaMetrics/fasthttp v1.0.5
 	github.com/VictoriaMetrics/metrics v1.12.3
-	github.com/VictoriaMetrics/metricsql v0.4.1
-	github.com/aws/aws-sdk-go v1.34.4
+	github.com/VictoriaMetrics/metricsql v0.6.0
+	github.com/aws/aws-sdk-go v1.34.20
 	github.com/cespare/xxhash/v2 v2.1.1
 	github.com/golang/snappy v0.0.1
-	github.com/klauspost/compress v1.10.11
+	github.com/klauspost/compress v1.11.0
 	github.com/valyala/fastjson v1.5.4
 	github.com/valyala/fastrand v1.0.0
 	github.com/valyala/fasttemplate v1.2.1
 	github.com/valyala/gozstd v1.8.3
 	github.com/valyala/histogram v1.1.2
 	github.com/valyala/quicktemplate v1.6.2
-	golang.org/x/net v0.0.0-20200813134508-3edf25e44fcc // indirect
-	golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d
-	golang.org/x/sys v0.0.0-20200812155832-6a926be9bd1d
-	golang.org/x/tools v0.0.0-20200813203630-136574234359 // indirect
-	google.golang.org/api v0.30.0
-	google.golang.org/genproto v0.0.0-20200813001606-1ccf2a5ae4fd // indirect
+	golang.org/x/net v0.0.0-20200904194848-62affa334b73 // indirect
+	golang.org/x/oauth2 v0.0.0-20200902213428-5d25da1a8d43
+	golang.org/x/sys v0.0.0-20200909081042-eff7692f9009
+	golang.org/x/tools v0.0.0-20200909210914-44a2922940c2 // indirect
+	google.golang.org/api v0.31.0
+	google.golang.org/genproto v0.0.0-20200904004341-0bd0a958aa1d // indirect
+	google.golang.org/grpc v1.32.0 // indirect
 	gopkg.in/yaml.v2 v2.3.0
 )
 

go.sum

@@ -15,8 +15,9 @@ cloud.google.com/go v0.57.0 h1:EpMNVUorLiZIELdMZbCYX/ByTFCdoYopYAGxaGVz9ms=
 cloud.google.com/go v0.57.0/go.mod h1:oXiQ6Rzq3RAkkY7N6t3TcE6jE+CIBBbA36lwQ1JyzZs=
 cloud.google.com/go v0.62.0 h1:RmDygqvj27Zf3fCQjQRtLyC7KwFcHkeJitcO0OoGOcA=
 cloud.google.com/go v0.62.0/go.mod h1:jmCYTdRCQuc1PHIIJ/maLInMho30T/Y0M4hTdTShOYc=
-cloud.google.com/go v0.63.0 h1:A+DfAZQ/eWca7gvu42CS6FNSDX4R8cghF+XfWLn4R6g=
-cloud.google.com/go v0.63.0/go.mod h1:GmezbQc7T2snqkEXWfZ0sy0VfkB/ivI2DdtJL2DEmlg=
+cloud.google.com/go v0.64.0/go.mod h1:xfORb36jGvE+6EexW71nMEtL025s3x6xvuYUKM4JLv4=
+cloud.google.com/go v0.65.0 h1:Dg9iHVQfrhq82rUNu9ZxUDrJLaxFUe/HlCVaLyRruq8=
+cloud.google.com/go v0.65.0/go.mod h1:O5N8zS7uWy9vkA9vayVHs65eM1ubvY4h553ofrNHObY=
 cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o=
 cloud.google.com/go/bigquery v1.3.0/go.mod h1:PjpwJnslEMmckchkHFfq+HTD2DmtT67aNFKH1/VBDHE=
 cloud.google.com/go/bigquery v1.4.0 h1:xE3CPsOgttP4ACBePh79zTKALtXwn/Edhcr16R5hMWU=
@@ -43,6 +44,8 @@ cloud.google.com/go/storage v1.8.0 h1:86K1Gel7BQ9/WmNWn7dTKMvTLFzwtBe5FNqYbi9X35
 cloud.google.com/go/storage v1.8.0/go.mod h1:Wv1Oy7z6Yz3DshWRJFhqM/UCfaWIRTdp0RXyy7KQOVs=
 cloud.google.com/go/storage v1.10.0 h1:STgFzyU5/8miMl0//zKh2aQeTyeaUH3WN9bSUiJ09bA=
 cloud.google.com/go/storage v1.10.0/go.mod h1:FLPqc6j+Ki4BU591ie1oL6qBQGu2Bl/tZ9ullr3+Kg0=
+cloud.google.com/go/storage v1.11.0 h1:bSLyzhbGjLMYxCratCDRSSH7+xRGpNApTBmowDUFGLk=
+cloud.google.com/go/storage v1.11.0/go.mod h1:/PAbprKS+5msVYogBmczjWalDXnQ9mr64yEq9YnyPeo=
 dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
 github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
 github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
@@ -55,13 +58,13 @@ github.com/VictoriaMetrics/metrics v1.12.2 h1:SG8iAmqavDNuh7GIdHPoGHUhDL23KeKfvS
 github.com/VictoriaMetrics/metrics v1.12.2/go.mod h1:Z1tSfPfngDn12bTfZSCqArT3OPY3u88J12hSoOhuiRE=
 github.com/VictoriaMetrics/metrics v1.12.3 h1:Fe6JHC6MSEKa+BtLhPN8WIvS+HKPzMc2evEpNeCGy7I=
 github.com/VictoriaMetrics/metrics v1.12.3/go.mod h1:Z1tSfPfngDn12bTfZSCqArT3OPY3u88J12hSoOhuiRE=
-github.com/VictoriaMetrics/metricsql v0.4.1 h1:WbVIfRNCK7HjrzayrpAl07mkh4kiDFZuECsh57rly2Q=
-github.com/VictoriaMetrics/metricsql v0.4.1/go.mod h1:ylO7YITho/Iw6P71oEaGyHbO94bGoGtzWfLGqFhMIg8=
+github.com/VictoriaMetrics/metricsql v0.6.0 h1:JnHUmifuA3fdy1GQrmkZJFO+CwFrhLxKwzMv89wNgJ4=
+github.com/VictoriaMetrics/metricsql v0.6.0/go.mod h1:ylO7YITho/Iw6P71oEaGyHbO94bGoGtzWfLGqFhMIg8=
 github.com/allegro/bigcache v1.2.1-0.20190218064605-e24eb225f156 h1:eMwmnE/GDgah4HI848JfFxHt+iPb26b4zyfspmqY0/8=
 github.com/allegro/bigcache v1.2.1-0.20190218064605-e24eb225f156/go.mod h1:Cb/ax3seSYIx7SuZdm2G2xzfwmv3TPSk2ucNfQESPXM=
 github.com/andybalholm/brotli v1.0.0/go.mod h1:loMXtMfwqflxFJPmdbJO0a3KNoPuLBgiu3qAvBg8x/Y=
-github.com/aws/aws-sdk-go v1.34.4 h1:Yx49/+ZMCD9YqIVsO3CsiMs4hnUnokd9otKvWYFjnYw=
-github.com/aws/aws-sdk-go v1.34.4/go.mod h1:5zCpMtNQVjRREroY7sYe8lOMRSxkhG6MZveU8YkpAk0=
+github.com/aws/aws-sdk-go v1.34.20 h1:D9otznteZZyN5pRyFETqveYia/85Xzk7+RaPGB1I9fE=
+github.com/aws/aws-sdk-go v1.34.20/go.mod h1:5zCpMtNQVjRREroY7sYe8lOMRSxkhG6MZveU8YkpAk0=
 github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
 github.com/cespare/xxhash/v2 v2.1.1 h1:6MnRN8NT7+YBpUIWxHtefFZOKTAPgGjpQSxqLNn0+qY=
 github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
@@ -119,6 +122,8 @@ github.com/google/go-cmp v0.4.1/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/
 github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
 github.com/google/go-cmp v0.5.1 h1:JFrFEBb2xKufg6XkJsJr+WbKb4FQlURi5RUcBveYu9k=
 github.com/google/go-cmp v0.5.1/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
+github.com/google/go-cmp v0.5.2 h1:X2ev0eStA3AbceY54o37/0PQ/UWqKEiiO2dKL5OPaFM=
+github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
 github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no=
 github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
 github.com/google/martian/v3 v3.0.0 h1:pMen7vLs8nvgEYhywH3KDWJIJTeEr2ULsVWHWYHQyBs=
@@ -148,6 +153,8 @@ github.com/klauspost/compress v1.10.10 h1:a/y8CglcM7gLGYmlbP/stPE5sR3hbhFRUjCBfd
 github.com/klauspost/compress v1.10.10/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs=
 github.com/klauspost/compress v1.10.11 h1:K9z59aO18Aywg2b/WSgBaUX99mHy2BES18Cr5lBKZHk=
 github.com/klauspost/compress v1.10.11/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs=
+github.com/klauspost/compress v1.11.0 h1:wJbzvpYMVGG9iTI9VxpnNZfd4DzMPoCWze3GgSqz8yg=
+github.com/klauspost/compress v1.11.0/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs=
 github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
 github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
 github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
@@ -182,6 +189,7 @@ github.com/valyala/tcplisten v0.0.0-20161114210144-ceec8f93295a/go.mod h1:v3UYOV
 github.com/yuin/goldmark v1.1.25/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
 github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
 github.com/yuin/goldmark v1.1.32/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
+github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
 go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
 go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
 go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
@@ -254,12 +262,18 @@ golang.org/x/net v0.0.0-20200707034311-ab3426394381 h1:VXak5I6aEWmAXeQjA+QSZzlgN
 golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
 golang.org/x/net v0.0.0-20200813134508-3edf25e44fcc h1:zK/HqS5bZxDptfPJNq8v7vJfXtkU7r9TLIoSr1bXaP4=
 golang.org/x/net v0.0.0-20200813134508-3edf25e44fcc/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
+golang.org/x/net v0.0.0-20200822124328-c89045814202 h1:VvcQYSHwXgi7W+TpUR6A9g6Up98WAHf3f/ulnJ62IyA=
+golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
+golang.org/x/net v0.0.0-20200904194848-62affa334b73 h1:MXfv8rhZWmFeqX3GNZRsd6vOLoaCHjYEX3qkRo3YBUA=
+golang.org/x/net v0.0.0-20200904194848-62affa334b73/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
 golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
 golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d h1:TzXSXBo42m9gQenoE3b9BGiEpg5IG2JkU5FkPIawgtw=
 golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
+golang.org/x/oauth2 v0.0.0-20200902213428-5d25da1a8d43 h1:ld7aEMNHoBnnDAX15v1T6z31v8HwR2A9FYOuAhWqkwc=
+golang.org/x/oauth2 v0.0.0-20200902213428-5d25da1a8d43/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
 golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -294,8 +308,9 @@ golang.org/x/sys v0.0.0-20200515095857-1151b9dac4a9/go.mod h1:h1NjWce9XRLGQEsW7w
 golang.org/x/sys v0.0.0-20200523222454-059865788121/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200602225109-6fdc65e7d980/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200803210538-64077c9b5642/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20200812155832-6a926be9bd1d h1:QQrM/CCYEzTs91GZylDCQjGHudbPTxF/1fvXdVh5lMo=
-golang.org/x/sys v0.0.0-20200812155832-6a926be9bd1d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200828194041-157a740278f4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200909081042-eff7692f9009 h1:W0lCpv29Hv0UaM1LXb9QlBHLNP8UFfcKjblhVCWftOM=
+golang.org/x/sys v0.0.0-20200909081042-eff7692f9009/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
@@ -344,9 +359,12 @@ golang.org/x/tools v0.0.0-20200515010526-7d3b6ebf133d/go.mod h1:EkVYQZoAsY45+roY
 golang.org/x/tools v0.0.0-20200618134242-20370b0cb4b2/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
 golang.org/x/tools v0.0.0-20200729194436-6467de6f59a7/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA=
 golang.org/x/tools v0.0.0-20200804011535-6c149bb5ef0d/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA=
-golang.org/x/tools v0.0.0-20200806022845-90696ccdc692/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA=
-golang.org/x/tools v0.0.0-20200813203630-136574234359 h1:eZBCv5amWGFL95UzDqwTieckVDxgaiUg+KkNnNc62hQ=
-golang.org/x/tools v0.0.0-20200813203630-136574234359/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA=
+golang.org/x/tools v0.0.0-20200817023811-d00afeaade8f/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA=
+golang.org/x/tools v0.0.0-20200825202427-b303f430e36d/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA=
+golang.org/x/tools v0.0.0-20200827163409-021d7c6f1ec3/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA=
+golang.org/x/tools v0.0.0-20200828161849-5deb26317202/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA=
+golang.org/x/tools v0.0.0-20200909210914-44a2922940c2 h1:daAzF/Ytp6YSqJDu1hZJthJIhOrsAa7UbIkziU1t0K4=
+golang.org/x/tools v0.0.0-20200909210914-44a2922940c2/go.mod h1:Cj7w3i3Rnn0Xh82ur9kSqwfTHTeVxaDqrfMjpcNT6bE=
 golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
@@ -369,6 +387,8 @@ google.golang.org/api v0.28.0/go.mod h1:lIXQywCXRcnZPGlsd8NbLnOjtAoL6em04bJ9+z0M
 google.golang.org/api v0.29.0/go.mod h1:Lcubydp8VUV7KeIHD9z2Bys/sm/vGKnG1UHuDBSrHWM=
 google.golang.org/api v0.30.0 h1:yfrXXP61wVuLb0vBcG6qaOoIoqYEzOQS8jum51jkv2w=
 google.golang.org/api v0.30.0/go.mod h1:QGmEvQ87FHZNiUVJkT14jQNYJ4ZJjdRF23ZXz5138Fc=
+google.golang.org/api v0.31.0 h1:1w5Sz/puhxFo9lTtip2n47k7toB/U2nCqOKNHd3Yrbo=
+google.golang.org/api v0.31.0/go.mod h1:CL+9IBCa2WWU6gRuBWaKqGWLFFwbEUXkfeMkHLQWYWo=
 google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
 google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
 google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
@@ -404,9 +424,12 @@ google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEY
 google.golang.org/genproto v0.0.0-20200618031413-b414f8b61790/go.mod h1:jDfRM7FcilCzHH/e9qn6dsT145K34l5v+OpcnNgKAAA=
 google.golang.org/genproto v0.0.0-20200729003335-053ba62fc06f/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
 google.golang.org/genproto v0.0.0-20200804131852-c06518451d9c/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
-google.golang.org/genproto v0.0.0-20200806141610-86f49bd18e98/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
-google.golang.org/genproto v0.0.0-20200813001606-1ccf2a5ae4fd h1:pCOIJgz7MD1XjLsF1K0X2xI97dR8sEXS34ZcYl7fcNE=
-google.golang.org/genproto v0.0.0-20200813001606-1ccf2a5ae4fd/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
+google.golang.org/genproto v0.0.0-20200815001618-f69a88009b70/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
+google.golang.org/genproto v0.0.0-20200825200019-8632dd797987/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
+google.golang.org/genproto v0.0.0-20200827165113-ac2560b5e952/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
+google.golang.org/genproto v0.0.0-20200831141814-d751682dd103/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
+google.golang.org/genproto v0.0.0-20200904004341-0bd0a958aa1d h1:92D1fum1bJLKSdr11OJ+54YeCMCGYIygTA7R/YZxH5M=
+google.golang.org/genproto v0.0.0-20200904004341-0bd0a958aa1d/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
 google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
 google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38=
 google.golang.org/grpc v1.21.1/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=
@@ -420,6 +443,10 @@ google.golang.org/grpc v1.29.1/go.mod h1:itym6AZVZYACWQqET3MqgPpjcuV5QH3BxFS3Iji
 google.golang.org/grpc v1.30.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
 google.golang.org/grpc v1.31.0 h1:T7P4R73V3SSDPhH7WW7ATbfViLtmamH0DKrP3f9AuDI=
 google.golang.org/grpc v1.31.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
+google.golang.org/grpc v1.31.1 h1:SfXqXS5hkufcdZ/mHtYCh53P2b+92WQq/DZcKLgsFRs=
+google.golang.org/grpc v1.31.1/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
+google.golang.org/grpc v1.32.0 h1:zWTV+LMdc3kaiJMSTOFz2UgSBgx8RNQoTGiZu3fR9S0=
+google.golang.org/grpc v1.32.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
 google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
 google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
 google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=

lib/backup/actions/util.go

@@ -18,7 +18,8 @@ var (
 		"See https://cloud.google.com/iam/docs/creating-managing-service-account-keys and https://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html")
 	configFilePath = flag.String("configFilePath", "", "Path to file with S3 configs. Configs are loaded from default location if not set.\n"+
 		"See https://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html")
-	configProfile    = flag.String("configProfile", "default", "Profile name for S3 configs")
+	configProfile = flag.String("configProfile", "", "Profile name for S3 configs. If no set, the value of the environment variable will be loaded (AWS_PROFILE or AWS_DEFAULT_PROFILE), "+
+		"or if both not set, DefaultSharedConfigProfile is used")
 	customS3Endpoint = flag.String("customS3Endpoint", "", "Custom S3 endpoint for use with S3-compatible storages (e.g. MinIO). S3 is used if not set")
 )
 

lib/cgroup/cpu.go

@@ -12,6 +12,8 @@ import (
 // This function must be called after logger.Init().
 func UpdateGOMAXPROCSToCPUQuota() {
 	if v := os.Getenv("GOMAXPROCS"); v != "" {
+		// Do not override explicitly set GOMAXPROCS.
+		logger.Infof("using GOMAXPROCS=%q set via environment variable", v)
 		return
 	}
 	q := getCPUQuota()
@@ -20,6 +22,12 @@ func UpdateGOMAXPROCSToCPUQuota() {
 		return
 	}
 	gomaxprocs := int(q + 0.5)
+	numCPU := runtime.NumCPU()
+	if gomaxprocs > numCPU {
+		// There is no sense in setting more GOMAXPROCS than the number of available CPU cores.
+		logger.Infof("cgroup CPU quota=%d exceeds NumCPU=%d; using GOMAXPROCS=NumCPU", gomaxprocs, numCPU)
+		return
+	}
 	if gomaxprocs <= 0 {
 		gomaxprocs = 1
 	}

lib/cgroup/mem.go

@@ -14,3 +14,18 @@ func GetMemoryLimit() int64 {
 	}
 	return n
 }
+
+// GetHierarchicalMemoryLimit returns hierarchical memory limit
+func GetHierarchicalMemoryLimit() int64 {
+	// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/699
+	n, err := readInt64FromCommand("cat /sys/fs/cgroup/memory/memory.stat | grep hierarchical_memory_limit | cut -d' ' -f 2")
+	if err == nil {
+		return n
+	}
+	n, err = readInt64FromCommand(
+		"cat /sys/fs/cgroup/memory$(cat /proc/self/cgroup | grep memory | cut -d: -f3)/memory.stat | grep hierarchical_memory_limit | cut -d' ' -f 2")
+	if err != nil {
+		return 0
+	}
+	return n
+}

lib/cgroup/util.go

@@ -9,15 +9,18 @@ import (
 
 func readInt64(path, altCommand string) (int64, error) {
 	data, err := ioutil.ReadFile(path)
+	if err == nil {
+		data = bytes.TrimSpace(data)
+		return strconv.ParseInt(string(data), 10, 64)
+	}
+	return readInt64FromCommand(altCommand)
+}
+
+func readInt64FromCommand(command string) (int64, error) {
+	cmd := exec.Command("/bin/sh", "-c", command)
+	data, err := cmd.Output()
 	if err != nil {
-		// Read data according to https://unix.stackexchange.com/questions/242718/how-to-find-out-how-much-memory-lxc-container-is-allowed-to-consume
-		// This should properly determine the data location inside lxc container.
-		// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/84
-		cmd := exec.Command("/bin/sh", "-c", altCommand)
-		data, err = cmd.Output()
-		if err != nil {
-			return 0, err
-		}
+		return 0, err
 	}
 	data = bytes.TrimSpace(data)
 	return strconv.ParseInt(string(data), 10, 64)

lib/decimal/decimal.go

@@ -256,7 +256,7 @@ func maxUpExponent(v int64) int16 {
 	}
 }
 
-// Round f to value with the given number of significant decimal digits.
+// Round f to value with the given number of significant figures.
 func Round(f float64, digits int) float64 {
 	if digits <= 0 || digits >= 18 {
 		return f

lib/flagutil/array.go

@@ -9,9 +9,9 @@ import (
 
 // NewArray returns new Array with the given name and description.
 func NewArray(name, description string) *Array {
-	var a Array
 	description += "\nSupports `array` of values separated by comma" +
 		" or specified via multiple flags."
+	var a Array
 	flag.Var(&a, name, description)
 	return &a
 }

lib/flagutil/bytes.go

@@ -0,0 +1,102 @@
+package flagutil
+
+import (
+	"flag"
+	"fmt"
+	"strconv"
+	"strings"
+)
+
+// NewBytes returns new `bytes` flag with the given name, defaultValue and description.
+func NewBytes(name string, defaultValue int, description string) *Bytes {
+	description += "\nSupports the following optional suffixes for values: KB, MB, GB, KiB, MiB, GiB"
+	b := Bytes{
+		N:           defaultValue,
+		valueString: fmt.Sprintf("%d", defaultValue),
+	}
+	flag.Var(&b, name, description)
+	return &b
+}
+
+// Bytes is a flag for holding size in bytes.
+//
+// It supports the following optional suffixes for values: KB, MB, GB, KiB, MiB, GiB.
+type Bytes struct {
+	// N contains parsed value for the given flag.
+	N int
+
+	valueString string
+}
+
+// String implements flag.Value interface
+func (b *Bytes) String() string {
+	return b.valueString
+}
+
+// Set implements flag.Value interface
+func (b *Bytes) Set(value string) error {
+	value = normalizeBytesString(value)
+	switch {
+	case strings.HasSuffix(value, "KB"):
+		f, err := strconv.ParseFloat(value[:len(value)-2], 64)
+		if err != nil {
+			return err
+		}
+		b.N = int(f * 1000)
+		b.valueString = value
+		return nil
+	case strings.HasSuffix(value, "MB"):
+		f, err := strconv.ParseFloat(value[:len(value)-2], 64)
+		if err != nil {
+			return err
+		}
+		b.N = int(f * 1000 * 1000)
+		b.valueString = value
+		return nil
+	case strings.HasSuffix(value, "GB"):
+		f, err := strconv.ParseFloat(value[:len(value)-2], 64)
+		if err != nil {
+			return err
+		}
+		b.N = int(f * 1000 * 1000 * 1000)
+		b.valueString = value
+		return nil
+	case strings.HasSuffix(value, "KiB"):
+		f, err := strconv.ParseFloat(value[:len(value)-3], 64)
+		if err != nil {
+			return err
+		}
+		b.N = int(f * 1024)
+		b.valueString = value
+		return nil
+	case strings.HasSuffix(value, "MiB"):
+		f, err := strconv.ParseFloat(value[:len(value)-3], 64)
+		if err != nil {
+			return err
+		}
+		b.N = int(f * 1024 * 1024)
+		b.valueString = value
+		return nil
+	case strings.HasSuffix(value, "GiB"):
+		f, err := strconv.ParseFloat(value[:len(value)-3], 64)
+		if err != nil {
+			return err
+		}
+		b.N = int(f * 1024 * 1024 * 1024)
+		b.valueString = value
+		return nil
+	default:
+		f, err := strconv.ParseFloat(value, 64)
+		if err != nil {
+			return err
+		}
+		b.N = int(f)
+		b.valueString = value
+		return nil
+	}
+}
+
+func normalizeBytesString(s string) string {
+	s = strings.ToUpper(s)
+	return strings.ReplaceAll(s, "I", "i")
+}

lib/flagutil/bytes_test.go

@@ -0,0 +1,54 @@
+package flagutil
+
+import (
+	"testing"
+)
+
+func TestBytesSetFailure(t *testing.T) {
+	f := func(value string) {
+		t.Helper()
+		var b Bytes
+		if err := b.Set(value); err == nil {
+			t.Fatalf("expecting non-nil error in b.Set(%q)", value)
+		}
+	}
+	f("")
+	f("foobar")
+	f("5foobar")
+	f("aKB")
+	f("134xMB")
+	f("2.43sdfGb")
+	f("aKiB")
+	f("134xMiB")
+	f("2.43sdfGIb")
+}
+
+func TestBytesSetSuccess(t *testing.T) {
+	f := func(value string, expectedResult int) {
+		t.Helper()
+		var b Bytes
+		if err := b.Set(value); err != nil {
+			t.Fatalf("unexpected error in b.Set(%q): %s", value, err)
+		}
+		if b.N != expectedResult {
+			t.Fatalf("unexpected result; got %d; want %d", b.N, expectedResult)
+		}
+		valueString := b.String()
+		valueExpected := normalizeBytesString(value)
+		if valueString != valueExpected {
+			t.Fatalf("unexpected valueString; got %q; want %q", valueString, valueExpected)
+		}
+	}
+	f("0", 0)
+	f("1", 1)
+	f("-1234", -1234)
+	f("123.456", 123)
+	f("1KiB", 1024)
+	f("1.5kib", 1.5*1024)
+	f("23MiB", 23*1024*1024)
+	f("0.25GiB", 0.25*1024*1024*1024)
+	f("1KB", 1000)
+	f("1.5kb", 1.5*1000)
+	f("23MB", 23*1000*1000)
+	f("0.25GB", 0.25*1000*1000*1000)
+}

lib/httpserver/httpserver.go

@@ -18,10 +18,12 @@ import (
 	"sync/atomic"
 	"time"
 
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fasttime"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/netutil"
 	"github.com/VictoriaMetrics/metrics"
 	"github.com/klauspost/compress/gzip"
+	"github.com/valyala/fastrand"
 )
 
 var (
@@ -42,6 +44,9 @@ var (
 		"Highly loaded server may require increased value for graceful shutdown")
 	shutdownDelay = flag.Duration("http.shutdownDelay", 0, "Optional delay before http server shutdown. During this dealy the servier returns non-OK responses "+
 		"from /health page, so load balancers can route new requests to other servers")
+	idleConnTimeout = flag.Duration("http.idleConnTimeout", time.Minute, "Timeout for incoming idle http connections")
+	connTimeout     = flag.Duration("http.connTimeout", 2*time.Minute, "Incoming http connections are closed after the configured timeout. This may help spreading incoming load "+
+		"among a cluster of services behind load balancer. Note that the real timeout may be bigger by up to 10% as a protection from Thundering herd problem")
 )
 
 var (
@@ -104,12 +109,22 @@ func serveWithListener(addr string, ln net.Listener, rh RequestHandler) {
 		TLSNextProto: make(map[string]func(*http.Server, *tls.Conn, http.Handler)),
 
 		ReadHeaderTimeout: 5 * time.Second,
-		IdleTimeout:       time.Minute,
+		IdleTimeout:       *idleConnTimeout,
 
 		// Do not set ReadTimeout and WriteTimeout here,
 		// since these timeouts must be controlled by request handlers.
 
 		ErrorLog: logger.StdErrorLogger(),
+
+		ConnContext: func(ctx context.Context, c net.Conn) context.Context {
+			timeoutSec := connTimeout.Seconds()
+			// Add a jitter for connection timeout in order to prevent Thundering herd problem
+			// when all the connections are established at the same time.
+			// See https://en.wikipedia.org/wiki/Thundering_herd_problem
+			jitterSec := fastrand.Uint32n(uint32(timeoutSec / 10))
+			deadline := fasttime.UnixTimestamp() + uint64(timeoutSec) + uint64(jitterSec)
+			return context.WithValue(ctx, connDeadlineTimeKey, &deadline)
+		},
 	}
 	serversLock.Lock()
 	servers[addr] = &s
@@ -123,6 +138,15 @@ func serveWithListener(addr string, ln net.Listener, rh RequestHandler) {
 	}
 }
 
+func whetherToCloseConn(r *http.Request) bool {
+	ctx := r.Context()
+	v := ctx.Value(connDeadlineTimeKey)
+	deadline, ok := v.(*uint64)
+	return ok && fasttime.UnixTimestamp() > *deadline
+}
+
+var connDeadlineTimeKey = interface{}("connDeadlineSecs")
+
 // Stop stops the http server on the given addr, which has been started
 // via Serve func.
 func Stop(addr string) error {
@@ -167,9 +191,14 @@ func gzipHandler(s *server, rh RequestHandler) http.HandlerFunc {
 }
 
 var metricsHandlerDuration = metrics.NewHistogram(`vm_http_request_duration_seconds{path="/metrics"}`)
+var connTimeoutClosedConns = metrics.NewCounter(`vm_http_conn_timeout_closed_conns_total`)
 
 func handlerWrapper(s *server, w http.ResponseWriter, r *http.Request, rh RequestHandler) {
 	requestsTotal.Inc()
+	if whetherToCloseConn(r) {
+		connTimeoutClosedConns.Inc()
+		w.Header().Set("Connection", "close")
+	}
 	path, err := getCanonicalPath(r.URL.Path)
 	if err != nil {
 		Errorf(w, r, "cannot get canonical path: %s", err)

lib/leveledbytebufferpool/pool.go

@@ -9,51 +9,48 @@ import (
 
 // pools contains pools for byte slices of various capacities.
 //
-//    pools[0] is for capacities from 0 to 7
-//    pools[1] is for capacities from 8 to 15
-//    pools[2] is for capacities from 16 to 31
-//    pools[3] is for capacities from 32 to 63
+//    pools[0] is for capacities from 0 to 8
+//    pools[1] is for capacities from 9 to 16
+//    pools[2] is for capacities from 17 to 32
+//    ...
+//    pools[n] is for capacities from 2^(n+2)+1 to 2^(n+3)
 //
 var pools [30]sync.Pool
 
 // Get returns byte buffer with the given capacity.
 func Get(capacity int) *bytesutil.ByteBuffer {
-	if capacity <= 0 {
-		capacity = 1
-	}
+	id, capacityNeeded := getPoolIDAndCapacity(capacity)
 	for i := 0; i < 2; i++ {
-		v := getPool(capacity).Get()
-		if v != nil {
-			return v.(*bytesutil.ByteBuffer)
-		}
-		if capacity > 1<<30 {
+		if id < 0 || id >= len(pools) {
 			break
 		}
-		capacity *= 2
+		if v := pools[id].Get(); v != nil {
+			return v.(*bytesutil.ByteBuffer)
+		}
+		id++
 	}
 	return &bytesutil.ByteBuffer{
-		B: make([]byte, 0, capacity),
+		B: make([]byte, 0, capacityNeeded),
 	}
 }
 
 // Put returns bb to the pool.
 func Put(bb *bytesutil.ByteBuffer) {
 	capacity := cap(bb.B)
+	id, _ := getPoolIDAndCapacity(capacity)
 	bb.Reset()
-	getPool(capacity).Put(bb)
+	pools[id].Put(bb)
 }
 
-func getPool(size int) *sync.Pool {
+func getPoolIDAndCapacity(size int) (int, int) {
+	size--
 	if size < 0 {
 		size = 0
 	}
 	size >>= 3
-	n := bits.Len(uint(size))
-	if n > len(pools) {
-		n = len(pools) - 1
+	id := bits.Len(uint(size))
+	if id > len(pools) {
+		id = len(pools) - 1
 	}
-	if n < 0 {
-		n = 0
-	}
-	return &pools[n]
+	return id, (1 << (id + 3))
 }

lib/memory/memory.go

@@ -5,6 +5,7 @@ import (
 	"fmt"
 	"sync"
 
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 )
 
@@ -13,7 +14,7 @@ var (
 		"See also -memory.allowedBytes. "+
 		"Too low value may increase cache miss rate, which usually results in higher CPU and disk IO usage. "+
 		"Too high value may evict too much data from OS page cache, which will result in higher disk IO usage")
-	allowedBytes = flag.Int("memory.allowedBytes", 0, "Allowed size of system memory VictoriaMetrics caches may occupy. "+
+	allowedBytes = flagutil.NewBytes("memory.allowedBytes", 0, "Allowed size of system memory VictoriaMetrics caches may occupy. "+
 		"This option overrides -memory.allowedPercent if set to non-zero value. "+
 		"Too low value may increase cache miss rate, which usually results in higher CPU and disk IO usage. "+
 		"Too high value may evict too much data from OS page cache, which will result in higher disk IO usage")
@@ -32,18 +33,19 @@ func initOnce() {
 		panic(fmt.Errorf("BUG: memory.Allowed must be called only after flag.Parse call"))
 	}
 	mem := sysTotalMemory()
-	if *allowedBytes <= 0 {
+	if allowedBytes.N <= 0 {
 		if *allowedPercent < 1 || *allowedPercent > 200 {
 			logger.Panicf("FATAL: -memory.allowedPercent must be in the range [1...200]; got %f", *allowedPercent)
 		}
 		percent := *allowedPercent / 100
 		allowedMemory = int(float64(mem) * percent)
+		remainingMemory = mem - allowedMemory
+		logger.Infof("limiting caches to %d bytes, leaving %d bytes to the OS according to -memory.allowedPercent=%f", allowedMemory, remainingMemory, *allowedPercent)
 	} else {
-		allowedMemory = *allowedBytes
+		allowedMemory = allowedBytes.N
+		remainingMemory = mem - allowedMemory
+		logger.Infof("limiting caches to %d bytes, leaving %d bytes to the OS according to -memory.allowedBytes=%s", allowedMemory, remainingMemory, allowedBytes.String())
 	}
-	remainingMemory = mem - allowedMemory
-	logger.Infof("limiting caches to %d bytes, leaving %d bytes to the OS according to -memory.allowedPercent=%f and -memory.allowedBytes=%d",
-		allowedMemory, remainingMemory, *allowedPercent, *allowedBytes)
 }
 
 // Allowed returns the amount of system memory allowed to use by the app.

lib/memory/memory_linux.go

@@ -20,7 +20,12 @@ func sysTotalMemory() int {
 	}
 	mem := cgroup.GetMemoryLimit()
 	if mem <= 0 || int64(int(mem)) != mem || int(mem) > totalMem {
-		return totalMem
+		// Try reading hierachical memory limit.
+		// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/699
+		mem = cgroup.GetHierarchicalMemoryLimit()
+		if mem <= 0 || int64(int(mem)) != mem || int(mem) > totalMem {
+			return totalMem
+		}
 	}
 	return int(mem)
 }

lib/promscrape/client.go

@@ -7,12 +7,13 @@ import (
 	"strings"
 	"time"
 
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
 	"github.com/VictoriaMetrics/fasthttp"
 	"github.com/VictoriaMetrics/metrics"
 )
 
 var (
-	maxScrapeSize = flag.Int("promscrape.maxScrapeSize", 16*1024*1024, "The maximum size of scrape response in bytes to process from Prometheus targets. "+
+	maxScrapeSize = flagutil.NewBytes("promscrape.maxScrapeSize", 16*1024*1024, "The maximum size of scrape response in bytes to process from Prometheus targets. "+
 		"Bigger responses are rejected")
 	disableCompression = flag.Bool("promscrape.disableCompression", false, "Whether to disable sending 'Accept-Encoding: gzip' request headers to all the scrape targets. "+
 		"This may reduce CPU usage on scrape targets at the cost of higher network bandwidth utilization. "+
@@ -60,7 +61,7 @@ func newClient(sw *ScrapeWork) *client {
 		MaxIdleConnDuration:          2 * sw.ScrapeInterval,
 		ReadTimeout:                  sw.ScrapeTimeout,
 		WriteTimeout:                 10 * time.Second,
-		MaxResponseBodySize:          *maxScrapeSize,
+		MaxResponseBodySize:          maxScrapeSize.N,
 		MaxIdempotentRequestAttempts: 1,
 	}
 	return &client{
@@ -117,7 +118,7 @@ func (c *client) ReadData(dst []byte) ([]byte, error) {
 		}
 		if err == fasthttp.ErrBodyTooLarge {
 			return dst, fmt.Errorf("the response from %q exceeds -promscrape.maxScrapeSize=%d; "+
-				"either reduce the response size for the target or increase -promscrape.maxScrapeSize", c.scrapeURL, *maxScrapeSize)
+				"either reduce the response size for the target or increase -promscrape.maxScrapeSize", c.scrapeURL, maxScrapeSize.N)
 		}
 		return dst, fmt.Errorf("error when scraping %q: %w", c.scrapeURL, err)
 	}

lib/promscrape/discovery/dns/dns.go

@@ -4,6 +4,7 @@ import (
 	"context"
 	"fmt"
 	"net"
+	"strconv"
 	"strings"
 	"time"
 
@@ -122,8 +123,10 @@ func getAAddrLabels(ctx context.Context, sdc *SDConfig, lookupType string) ([]ma
 func appendAddrLabels(ms []map[string]string, name, target string, port int) []map[string]string {
 	addr := discoveryutils.JoinHostPort(target, port)
 	m := map[string]string{
-		"__address__":     addr,
-		"__meta_dns_name": name,
+		"__address__":                  addr,
+		"__meta_dns_name":              name,
+		"__meta_dns_srv_record_target": target,
+		"__meta_dns_srv_record_port":   strconv.Itoa(port),
 	}
 	return append(ms, m)
 }

lib/promscrape/discovery/kubernetes/endpointslices.go

@@ -0,0 +1,212 @@
+package kubernetes
+
+import (
+	"encoding/json"
+	"fmt"
+	"strconv"
+
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape/discoveryutils"
+)
+
+// getEndpointSlicesLabels returns labels for k8s endpointSlices obtained from the given cfg.
+func getEndpointSlicesLabels(cfg *apiConfig) ([]map[string]string, error) {
+	eps, err := getEndpointSlices(cfg)
+	if err != nil {
+		return nil, err
+	}
+	pods, err := getPods(cfg)
+	if err != nil {
+		return nil, err
+	}
+	svcs, err := getServices(cfg)
+	if err != nil {
+		return nil, err
+	}
+	var ms []map[string]string
+	for _, ep := range eps {
+		ms = ep.appendTargetLabels(ms, pods, svcs)
+	}
+
+	return ms, nil
+}
+
+// getEndpointSlices retrieves endpointSlice with given apiConfig
+func getEndpointSlices(cfg *apiConfig) ([]EndpointSlice, error) {
+	if len(cfg.namespaces) == 0 {
+		return getEndpointSlicesByPath(cfg, "/apis/discovery.k8s.io/v1beta1/endpointslices")
+	}
+	// Query /api/v1/namespaces/* for each namespace.
+	// This fixes authorization issue at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/432
+	cfgCopy := *cfg
+	namespaces := cfgCopy.namespaces
+	cfgCopy.namespaces = nil
+	cfg = &cfgCopy
+	var result []EndpointSlice
+	for _, ns := range namespaces {
+		path := fmt.Sprintf("/apis/discovery.k8s.io/v1beta1/namespaces/%s/endpointslices", ns)
+		eps, err := getEndpointSlicesByPath(cfg, path)
+		if err != nil {
+			return nil, err
+		}
+		result = append(result, eps...)
+	}
+	return result, nil
+}
+
+// getEndpointSlicesByPath retrieves endpointSlices from k8s api by given path
+func getEndpointSlicesByPath(cfg *apiConfig, path string) ([]EndpointSlice, error) {
+	data, err := getAPIResponse(cfg, "endpointslices", path)
+	if err != nil {
+		return nil, fmt.Errorf("cannot obtain endpointslices data from API server: %w", err)
+	}
+	epl, err := parseEndpointSlicesList(data)
+	if err != nil {
+		return nil, fmt.Errorf("cannot parse endpointslices response from API server: %w", err)
+	}
+	return epl.Items, nil
+
+}
+
+// parseEndpointsList parses EndpointSliceList from data.
+func parseEndpointSlicesList(data []byte) (*EndpointSliceList, error) {
+	var esl EndpointSliceList
+	if err := json.Unmarshal(data, &esl); err != nil {
+		return nil, fmt.Errorf("cannot unmarshal EndpointSliceList from %q: %w", data, err)
+	}
+
+	return &esl, nil
+}
+
+// appendTargetLabels injects labels for endPointSlice to slice map
+// follows TargetRef for enrich labels with pod and service metadata
+func (eps *EndpointSlice) appendTargetLabels(ms []map[string]string, pods []Pod, svcs []Service) []map[string]string {
+	svc := getService(svcs, eps.Metadata.Namespace, eps.Metadata.Name)
+	podPortsSeen := make(map[*Pod][]int)
+	for _, ess := range eps.Endpoints {
+		pod := getPod(pods, ess.TargetRef.Namespace, ess.TargetRef.Name)
+		for _, epp := range eps.Ports {
+			for _, addr := range ess.Addresses {
+				ms = append(ms, getEndpointSliceLabelsForAddressAndPort(podPortsSeen, addr, eps, ess, epp, pod, svc))
+			}
+
+		}
+	}
+
+	// Append labels for skipped ports on seen pods.
+	portSeen := func(port int, ports []int) bool {
+		for _, p := range ports {
+			if p == port {
+				return true
+			}
+		}
+		return false
+	}
+	for p, ports := range podPortsSeen {
+		for _, c := range p.Spec.Containers {
+			for _, cp := range c.Ports {
+				if portSeen(cp.ContainerPort, ports) {
+					continue
+				}
+				addr := discoveryutils.JoinHostPort(p.Status.PodIP, cp.ContainerPort)
+				m := map[string]string{
+					"__address__": addr,
+				}
+				p.appendCommonLabels(m)
+				p.appendContainerLabels(m, c, &cp)
+				ms = append(ms, m)
+			}
+		}
+	}
+	return ms
+
+}
+
+// getEndpointSliceLabelsForAddressAndPort gets labels for endpointSlice
+// from  address, Endpoint and EndpointPort
+// enriches labels with TargetRef
+// pod appended to seen Ports
+// if TargetRef matches
+func getEndpointSliceLabelsForAddressAndPort(podPortsSeen map[*Pod][]int, addr string, eps *EndpointSlice, ea Endpoint, epp EndpointPort, p *Pod, svc *Service) map[string]string {
+	m := getEndpointSliceLabels(eps, addr, ea, epp)
+	if svc != nil {
+		svc.appendCommonLabels(m)
+	}
+	if ea.TargetRef.Kind != "Pod" || p == nil {
+		return m
+	}
+	p.appendCommonLabels(m)
+	for _, c := range p.Spec.Containers {
+		for _, cp := range c.Ports {
+			if cp.ContainerPort == epp.Port {
+				p.appendContainerLabels(m, c, &cp)
+				podPortsSeen[p] = append(podPortsSeen[p], cp.ContainerPort)
+				break
+			}
+		}
+	}
+
+	return m
+}
+
+// //getEndpointSliceLabels builds labels for given EndpointSlice
+func getEndpointSliceLabels(eps *EndpointSlice, addr string, ea Endpoint, epp EndpointPort) map[string]string {
+
+	addr = discoveryutils.JoinHostPort(addr, epp.Port)
+	m := map[string]string{
+		"__address__":                                               addr,
+		"__meta_kubernetes_namespace":                               eps.Metadata.Namespace,
+		"__meta_kubernetes_endpointslice_name":                      eps.Metadata.Name,
+		"__meta_kubernetes_endpointslice_address_type":              eps.AddressType,
+		"__meta_kubernetes_endpointslice_endpoint_conditions_ready": strconv.FormatBool(ea.Conditions.Ready),
+		"__meta_kubernetes_endpointslice_port_name":                 epp.Name,
+		"__meta_kubernetes_endpointslice_port_protocol":             epp.Protocol,
+		"__meta_kubernetes_endpointslice_port":                      strconv.Itoa(epp.Port),
+	}
+	if epp.AppProtocol != "" {
+		m["__meta_kubernetes_endpointslice_port_app_protocol"] = epp.AppProtocol
+	}
+	if ea.TargetRef.Kind != "" {
+		m["__meta_kubernetes_endpointslice_address_target_kind"] = ea.TargetRef.Kind
+		m["__meta_kubernetes_endpointslice_address_target_name"] = ea.TargetRef.Name
+	}
+	if ea.Hostname != "" {
+		m["__meta_kubernetes_endpointslice_endpoint_hostname"] = ea.Hostname
+	}
+	for k, v := range ea.Topology {
+		m["__meta_kubernetes_endpointslice_endpoint_topology_"+discoveryutils.SanitizeLabelName(k)] = v
+		m["__meta_kubernetes_endpointslice_endpoint_topology_present_"+discoveryutils.SanitizeLabelName(k)] = "true"
+	}
+	return m
+}
+
+// EndpointSliceList - implements kubernetes endpoint slice list object,
+// that groups service endpoints slices.
+// https://v1-17.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#endpointslice-v1beta1-discovery-k8s-io
+type EndpointSliceList struct {
+	Items []EndpointSlice
+}
+
+// EndpointSlice - implements kubernetes endpoint slice.
+// https://v1-17.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#endpointslice-v1beta1-discovery-k8s-io
+type EndpointSlice struct {
+	Metadata    ObjectMeta
+	Endpoints   []Endpoint
+	AddressType string
+	Ports       []EndpointPort
+}
+
+// Endpoint implements kubernetes object endpoint for endpoint slice.
+// https://v1-17.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#endpoint-v1beta1-discovery-k8s-io
+type Endpoint struct {
+	Addresses  []string
+	Conditions EndpointConditions
+	Hostname   string
+	TargetRef  ObjectReference
+	Topology   map[string]string
+}
+
+// EndpointConditions implements kubernetes endpoint condition.
+// https://v1-17.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#endpointconditions-v1beta1-discovery-k8s-io
+type EndpointConditions struct {
+	Ready bool
+}

lib/promscrape/discovery/kubernetes/endpointslices_test.go

@@ -0,0 +1,446 @@
+package kubernetes
+
+import (
+	"reflect"
+	"testing"
+
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape/discoveryutils"
+)
+
+func Test_parseEndpointSlicesListFail(t *testing.T) {
+	f := func(data string) {
+		eslList, err := parseEndpointSlicesList([]byte(data))
+		if err == nil {
+			t.Errorf("unexpected result, test must fail! data: %s", data)
+		}
+		if eslList != nil {
+			t.Errorf("endpointSliceList must be nil, got: %v", eslList)
+		}
+	}
+
+	f(``)
+	f(`{"items": [1,2,3]`)
+	f(`{"items": [
+    {
+      "metadata": {
+        "name": "kubernetes"}]}`)
+
+}
+
+func Test_parseEndpointSlicesListSuccess(t *testing.T) {
+	data := `{
+  "kind": "EndpointSliceList",
+  "apiVersion": "discovery.k8s.io/v1beta1",
+  "metadata": {
+    "selfLink": "/apis/discovery.k8s.io/v1beta1/endpointslices",
+    "resourceVersion": "1177"
+  },
+  "items": [
+    {
+      "metadata": {
+        "name": "kubernetes",
+        "namespace": "default",
+        "selfLink": "/apis/discovery.k8s.io/v1beta1/namespaces/default/endpointslices/kubernetes",
+        "uid": "a60d9173-5fe4-4bc3-87a6-269daee71f8a",
+        "resourceVersion": "159",
+        "generation": 1,
+        "creationTimestamp": "2020-09-07T14:27:22Z",
+        "labels": {
+          "kubernetes.io/service-name": "kubernetes"
+        },
+        "managedFields": [
+          {
+            "manager": "kube-apiserver",
+            "operation": "Update",
+            "apiVersion": "discovery.k8s.io/v1beta1",
+            "time": "2020-09-07T14:27:22Z",
+            "fieldsType": "FieldsV1",
+            "fieldsV1": {"f:addressType":{},"f:endpoints":{},"f:metadata":{"f:labels":{".":{},"f:kubernetes.io/service-name":{}}},"f:ports":{}}
+          }
+        ]
+      },
+      "addressType": "IPv4",
+      "endpoints": [
+        {
+          "addresses": [
+            "172.18.0.2"
+          ],
+          "conditions": {
+            "ready": true
+          }
+        }
+      ],
+      "ports": [
+        {
+          "name": "https",
+          "protocol": "TCP",
+          "port": 6443
+        }
+      ]
+    },
+    {
+      "metadata": {
+        "name": "kube-dns-22mvb",
+        "generateName": "kube-dns-",
+        "namespace": "kube-system",
+        "selfLink": "/apis/discovery.k8s.io/v1beta1/namespaces/kube-system/endpointslices/kube-dns-22mvb",
+        "uid": "7c95c854-f34c-48e1-86f5-bb8269113c11",
+        "resourceVersion": "604",
+        "generation": 5,
+        "creationTimestamp": "2020-09-07T14:27:39Z",
+        "labels": {
+          "endpointslice.kubernetes.io/managed-by": "endpointslice-controller.k8s.io",
+          "kubernetes.io/service-name": "kube-dns"
+        },
+        "annotations": {
+          "endpoints.kubernetes.io/last-change-trigger-time": "2020-09-07T14:28:35Z"
+        },
+        "ownerReferences": [
+          {
+            "apiVersion": "v1",
+            "kind": "Service",
+            "name": "kube-dns",
+            "uid": "509e80d8-6d05-487b-bfff-74f5768f1024",
+            "controller": true,
+            "blockOwnerDeletion": true
+          }
+        ],
+        "managedFields": [
+          {
+            "manager": "kube-controller-manager",
+            "operation": "Update",
+            "apiVersion": "discovery.k8s.io/v1beta1",
+            "time": "2020-09-07T14:28:35Z",
+            "fieldsType": "FieldsV1",
+            "fieldsV1": {"f:addressType":{},"f:endpoints":{},"f:metadata":{"f:annotations":{".":{},"f:endpoints.kubernetes.io/last-change-trigger-time":{}},"f:generateName":{},"f:labels":{".":{},"f:endpointslice.kubernetes.io/managed-by":{},"f:kubernetes.io/service-name":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"509e80d8-6d05-487b-bfff-74f5768f1024\"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:ports":{}}
+          }
+        ]
+      },
+      "addressType": "IPv4",
+      "endpoints": [
+        {
+          "addresses": [
+            "10.244.0.3"
+          ],
+          "conditions": {
+            "ready": true
+          },
+          "targetRef": {
+            "kind": "Pod",
+            "namespace": "kube-system",
+            "name": "coredns-66bff467f8-z8czk",
+            "uid": "36a545ff-dbba-4192-a5f6-1dbb0c21c73d",
+            "resourceVersion": "603"
+          },
+          "topology": {
+            "kubernetes.io/hostname": "kind-control-plane"
+          }
+        },
+        {
+          "addresses": [
+            "10.244.0.4"
+          ],
+          "conditions": {
+            "ready": true
+          },
+          "targetRef": {
+            "kind": "Pod",
+            "namespace": "kube-system",
+            "name": "coredns-66bff467f8-kpbhk",
+            "uid": "db38d8b4-847a-4e82-874c-fe444fba2718",
+            "resourceVersion": "576"
+          },
+          "topology": {
+            "kubernetes.io/hostname": "kind-control-plane"
+          }
+        }
+      ],
+      "ports": [
+        {
+          "name": "dns-tcp",
+          "protocol": "TCP",
+          "port": 53
+        },
+        {
+          "name": "metrics",
+          "protocol": "TCP",
+          "port": 9153
+        },
+        {
+          "name": "dns",
+          "protocol": "UDP",
+          "port": 53
+        }
+      ]
+    }
+  ]
+}`
+	esl, err := parseEndpointSlicesList([]byte(data))
+	if err != nil {
+		t.Errorf("cannot parse data for EndpointSliceList: %v", err)
+		return
+	}
+	if len(esl.Items) != 2 {
+		t.Fatalf("expected 2 items at endpointSliceList, got: %d", len(esl.Items))
+	}
+
+	firstEsl := esl.Items[0]
+	got := firstEsl.appendTargetLabels(nil, nil, nil)
+	sortedLables := [][]prompbmarshal.Label{}
+	for _, labels := range got {
+		sortedLables = append(sortedLables, discoveryutils.GetSortedLabels(labels))
+	}
+	expectedLabels := [][]prompbmarshal.Label{
+		discoveryutils.GetSortedLabels(map[string]string{
+			"__address__": "172.18.0.2:6443",
+			"__meta_kubernetes_endpointslice_address_type":              "IPv4",
+			"__meta_kubernetes_endpointslice_endpoint_conditions_ready": "true",
+			"__meta_kubernetes_endpointslice_name":                      "kubernetes",
+			"__meta_kubernetes_endpointslice_port":                      "6443",
+			"__meta_kubernetes_endpointslice_port_name":                 "https",
+			"__meta_kubernetes_endpointslice_port_protocol":             "TCP",
+			"__meta_kubernetes_namespace":                               "default",
+		})}
+	if !reflect.DeepEqual(sortedLables, expectedLabels) {
+		t.Fatalf("unexpected labels,\ngot:\n%v,\nwant:\n%v", sortedLables, expectedLabels)
+	}
+
+}
+
+func TestEndpointSlice_appendTargetLabels(t *testing.T) {
+	type fields struct {
+		Metadata    ObjectMeta
+		Endpoints   []Endpoint
+		AddressType string
+		Ports       []EndpointPort
+	}
+	type args struct {
+		ms   []map[string]string
+		pods []Pod
+		svcs []Service
+	}
+	tests := []struct {
+		name   string
+		fields fields
+		args   args
+		want   [][]prompbmarshal.Label
+	}{
+		{
+			name: "simple eps",
+			args: args{},
+			fields: fields{
+				Metadata: ObjectMeta{
+					Name:      "fake-esl",
+					Namespace: "default",
+				},
+				AddressType: "ipv4",
+				Endpoints: []Endpoint{
+					{Addresses: []string{"127.0.0.1"},
+						Hostname:   "node-1",
+						Topology:   map[string]string{"kubernetes.topoligy.io/zone": "gce-1"},
+						Conditions: EndpointConditions{Ready: true},
+						TargetRef: ObjectReference{
+							Kind:      "Pod",
+							Namespace: "default",
+							Name:      "main-pod",
+						},
+					},
+				},
+				Ports: []EndpointPort{
+					{
+						Name:        "http",
+						Port:        8085,
+						AppProtocol: "http",
+						Protocol:    "tcp",
+					},
+				},
+			},
+			want: [][]prompbmarshal.Label{
+				discoveryutils.GetSortedLabels(map[string]string{
+					"__address__": "127.0.0.1:8085",
+					"__meta_kubernetes_endpointslice_address_target_kind":                                   "Pod",
+					"__meta_kubernetes_endpointslice_address_target_name":                                   "main-pod",
+					"__meta_kubernetes_endpointslice_address_type":                                          "ipv4",
+					"__meta_kubernetes_endpointslice_endpoint_conditions_ready":                             "true",
+					"__meta_kubernetes_endpointslice_endpoint_topology_kubernetes_topoligy_io_zone":         "gce-1",
+					"__meta_kubernetes_endpointslice_endpoint_topology_present_kubernetes_topoligy_io_zone": "true",
+					"__meta_kubernetes_endpointslice_endpoint_hostname":                                     "node-1",
+					"__meta_kubernetes_endpointslice_name":                                                  "fake-esl",
+					"__meta_kubernetes_endpointslice_port":                                                  "8085",
+					"__meta_kubernetes_endpointslice_port_app_protocol":                                     "http",
+					"__meta_kubernetes_endpointslice_port_name":                                             "http",
+					"__meta_kubernetes_endpointslice_port_protocol":                                         "tcp",
+					"__meta_kubernetes_namespace":                                                           "default",
+				}),
+			},
+		},
+		{
+			name: "eps with pods and services",
+			args: args{
+				pods: []Pod{
+					{
+						Metadata: ObjectMeta{
+							UID:       "some-pod-uuid",
+							Namespace: "monitoring",
+							Name:      "main-pod",
+							Labels: discoveryutils.GetSortedLabels(map[string]string{
+								"pod-label-1": "pod-value-1",
+								"pod-label-2": "pod-value-2",
+							}),
+							Annotations: discoveryutils.GetSortedLabels(map[string]string{
+								"pod-annotations-1": "annotation-value-1",
+							}),
+						},
+						Status: PodStatus{PodIP: "192.168.11.5", HostIP: "172.15.1.1"},
+						Spec: PodSpec{NodeName: "node-2", Containers: []Container{
+							{
+								Name: "container-1",
+								Ports: []ContainerPort{
+									{
+										ContainerPort: 8085,
+										Protocol:      "tcp",
+										Name:          "http",
+									},
+									{
+										ContainerPort: 8011,
+										Protocol:      "udp",
+										Name:          "dns",
+									},
+								},
+							},
+						}},
+					},
+				},
+				svcs: []Service{
+					{
+						Spec: ServiceSpec{Type: "ClusterIP", Ports: []ServicePort{
+							{
+								Name:     "http",
+								Protocol: "tcp",
+								Port:     8085,
+							},
+						}},
+						Metadata: ObjectMeta{
+							Name:      "custom-esl",
+							Namespace: "monitoring",
+							Labels: discoveryutils.GetSortedLabels(map[string]string{
+								"service-label-1": "value-1",
+								"service-label-2": "value-2",
+							}),
+						},
+					},
+				},
+			},
+			fields: fields{
+				Metadata: ObjectMeta{
+					Name:      "custom-esl",
+					Namespace: "monitoring",
+				},
+				AddressType: "ipv4",
+				Endpoints: []Endpoint{
+					{Addresses: []string{"127.0.0.1"},
+						Hostname:   "node-1",
+						Topology:   map[string]string{"kubernetes.topoligy.io/zone": "gce-1"},
+						Conditions: EndpointConditions{Ready: true},
+						TargetRef: ObjectReference{
+							Kind:      "Pod",
+							Namespace: "monitoring",
+							Name:      "main-pod",
+						},
+					},
+				},
+				Ports: []EndpointPort{
+					{
+						Name:        "http",
+						Port:        8085,
+						AppProtocol: "http",
+						Protocol:    "tcp",
+					},
+				},
+			},
+			want: [][]prompbmarshal.Label{
+				discoveryutils.GetSortedLabels(map[string]string{
+					"__address__": "127.0.0.1:8085",
+					"__meta_kubernetes_endpointslice_address_target_kind":                                   "Pod",
+					"__meta_kubernetes_endpointslice_address_target_name":                                   "main-pod",
+					"__meta_kubernetes_endpointslice_address_type":                                          "ipv4",
+					"__meta_kubernetes_endpointslice_endpoint_conditions_ready":                             "true",
+					"__meta_kubernetes_endpointslice_endpoint_topology_kubernetes_topoligy_io_zone":         "gce-1",
+					"__meta_kubernetes_endpointslice_endpoint_topology_present_kubernetes_topoligy_io_zone": "true",
+					"__meta_kubernetes_endpointslice_endpoint_hostname":                                     "node-1",
+					"__meta_kubernetes_endpointslice_name":                                                  "custom-esl",
+					"__meta_kubernetes_endpointslice_port":                                                  "8085",
+					"__meta_kubernetes_endpointslice_port_app_protocol":                                     "http",
+					"__meta_kubernetes_endpointslice_port_name":                                             "http",
+					"__meta_kubernetes_endpointslice_port_protocol":                                         "tcp",
+					"__meta_kubernetes_namespace":                                                           "monitoring",
+					"__meta_kubernetes_pod_annotation_pod_annotations_1":                                    "annotation-value-1",
+					"__meta_kubernetes_pod_annotationpresent_pod_annotations_1":                             "true",
+					"__meta_kubernetes_pod_container_name":                                                  "container-1",
+					"__meta_kubernetes_pod_container_port_name":                                             "http",
+					"__meta_kubernetes_pod_container_port_number":                                           "8085",
+					"__meta_kubernetes_pod_container_port_protocol":                                         "tcp",
+					"__meta_kubernetes_pod_host_ip":                                                         "172.15.1.1",
+					"__meta_kubernetes_pod_ip":                                                              "192.168.11.5",
+					"__meta_kubernetes_pod_label_pod_label_1":                                               "pod-value-1",
+					"__meta_kubernetes_pod_label_pod_label_2":                                               "pod-value-2",
+					"__meta_kubernetes_pod_labelpresent_pod_label_1":                                        "true",
+					"__meta_kubernetes_pod_labelpresent_pod_label_2":                                        "true",
+					"__meta_kubernetes_pod_name":                                                            "main-pod",
+					"__meta_kubernetes_pod_node_name":                                                       "node-2",
+					"__meta_kubernetes_pod_phase":                                                           "",
+					"__meta_kubernetes_pod_ready":                                                           "unknown",
+					"__meta_kubernetes_pod_uid":                                                             "some-pod-uuid",
+					"__meta_kubernetes_service_cluster_ip":                                                  "",
+					"__meta_kubernetes_service_label_service_label_1":                                       "value-1",
+					"__meta_kubernetes_service_label_service_label_2":                                       "value-2",
+					"__meta_kubernetes_service_labelpresent_service_label_1":                                "true",
+					"__meta_kubernetes_service_labelpresent_service_label_2":                                "true",
+					"__meta_kubernetes_service_name":                                                        "custom-esl",
+					"__meta_kubernetes_service_type":                                                        "ClusterIP",
+				}),
+				discoveryutils.GetSortedLabels(map[string]string{
+					"__address__":                 "192.168.11.5:8011",
+					"__meta_kubernetes_namespace": "monitoring",
+					"__meta_kubernetes_pod_annotation_pod_annotations_1":        "annotation-value-1",
+					"__meta_kubernetes_pod_annotationpresent_pod_annotations_1": "true",
+					"__meta_kubernetes_pod_container_name":                      "container-1",
+					"__meta_kubernetes_pod_container_port_name":                 "dns",
+					"__meta_kubernetes_pod_container_port_number":               "8011",
+					"__meta_kubernetes_pod_container_port_protocol":             "udp",
+					"__meta_kubernetes_pod_host_ip":                             "172.15.1.1",
+					"__meta_kubernetes_pod_ip":                                  "192.168.11.5",
+					"__meta_kubernetes_pod_label_pod_label_1":                   "pod-value-1",
+					"__meta_kubernetes_pod_label_pod_label_2":                   "pod-value-2",
+					"__meta_kubernetes_pod_labelpresent_pod_label_1":            "true",
+					"__meta_kubernetes_pod_labelpresent_pod_label_2":            "true",
+					"__meta_kubernetes_pod_name":                                "main-pod",
+					"__meta_kubernetes_pod_node_name":                           "node-2",
+					"__meta_kubernetes_pod_phase":                               "",
+					"__meta_kubernetes_pod_ready":                               "unknown",
+					"__meta_kubernetes_pod_uid":                                 "some-pod-uuid",
+				}),
+			},
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			eps := &EndpointSlice{
+				Metadata:    tt.fields.Metadata,
+				Endpoints:   tt.fields.Endpoints,
+				AddressType: tt.fields.AddressType,
+				Ports:       tt.fields.Ports,
+			}
+			got := eps.appendTargetLabels(tt.args.ms, tt.args.pods, tt.args.svcs)
+			var sortedLabelss [][]prompbmarshal.Label
+			for _, labels := range got {
+				sortedLabelss = append(sortedLabelss, discoveryutils.GetSortedLabels(labels))
+			}
+
+			if !reflect.DeepEqual(sortedLabelss, tt.want) {
+				t.Errorf("got unxpected labels: \ngot:\n %v, \nexpect:\n %v", sortedLabelss, tt.want)
+			}
+		})
+	}
+}

lib/promscrape/discovery/kubernetes/kubernetes.go

@@ -50,6 +50,8 @@ func GetLabels(sdc *SDConfig, baseDir string) ([]map[string]string, error) {
 		return getPodsLabels(cfg)
 	case "endpoints":
 		return getEndpointsLabels(cfg)
+	case "endpointslices":
+		return getEndpointSlicesLabels(cfg)
 	case "ingress":
 		return getIngressesLabels(cfg)
 	default:

lib/promscrape/scrapework.go

@@ -4,6 +4,7 @@ import (
 	"flag"
 	"fmt"
 	"math"
+	"math/bits"
 	"strings"
 	"sync"
 	"time"
@@ -129,13 +130,18 @@ type scrapeWork struct {
 
 	tmpRow parser.Row
 
-	// the prevSeriesMap and lh are used for fast calculation of `scrape_series_added` metric.
-	prevSeriesMap map[uint64]struct{}
+	// the seriesMap, seriesAdded and labelsHashBuf are used for fast calculation of `scrape_series_added` metric.
+	seriesMap     map[uint64]struct{}
+	seriesAdded   int
 	labelsHashBuf []byte
 
 	// prevBodyLen contains the previous response body length for the given scrape work.
 	// It is used as a hint in order to reduce memory usage for body buffers.
 	prevBodyLen int
+
+	// prevRowsLen contains the number rows scraped during the previous scrape.
+	// It is used as a hint in order to reduce memory usage when parsing scrape responses.
+	prevRowsLen int
 }
 
 func (sw *scrapeWork) run(stopCh <-chan struct{}) {
@@ -212,7 +218,7 @@ func (sw *scrapeWork) scrapeInternal(scrapeTimestamp, realTimestamp int64) error
 	scrapeDuration.Update(duration)
 	scrapeResponseSize.Update(float64(len(body.B)))
 	up := 1
-	wc := writeRequestCtxPool.Get().(*writeRequestCtx)
+	wc := writeRequestCtxPool.Get(sw.prevRowsLen)
 	if err != nil {
 		up = 0
 		scrapesFailed.Inc()
@@ -223,16 +229,29 @@ func (sw *scrapeWork) scrapeInternal(scrapeTimestamp, realTimestamp int64) error
 	srcRows := wc.rows.Rows
 	samplesScraped := len(srcRows)
 	scrapedSamples.Update(float64(samplesScraped))
-	for i := range srcRows {
-		sw.addRowToTimeseries(wc, &srcRows[i], scrapeTimestamp, true)
-	}
-	if sw.Config.SampleLimit > 0 && len(wc.writeRequest.Timeseries) > sw.Config.SampleLimit {
-		prompbmarshal.ResetWriteRequest(&wc.writeRequest)
+	if sw.Config.SampleLimit > 0 && samplesScraped > sw.Config.SampleLimit {
+		srcRows = srcRows[:0]
 		up = 0
 		scrapesSkippedBySampleLimit.Inc()
 	}
-	samplesPostRelabeling := len(wc.writeRequest.Timeseries)
-	seriesAdded := sw.getSeriesAdded(wc)
+	samplesPostRelabeling := 0
+	for i := range srcRows {
+		sw.addRowToTimeseries(wc, &srcRows[i], scrapeTimestamp, true)
+		if len(wc.labels) > 40000 {
+			// Limit the maximum size of wc.writeRequest.
+			// This should reduce memory usage when scraping targets with millions of metrics and/or labels.
+			// For example, when scraping /federate handler from Prometheus - see https://prometheus.io/docs/prometheus/latest/federation/
+			samplesPostRelabeling += len(wc.writeRequest.Timeseries)
+			sw.updateSeriesAdded(wc)
+			startTime := time.Now()
+			sw.PushData(&wc.writeRequest)
+			pushDataDuration.UpdateDuration(startTime)
+			wc.resetNoRows()
+		}
+	}
+	samplesPostRelabeling += len(wc.writeRequest.Timeseries)
+	sw.updateSeriesAdded(wc)
+	seriesAdded := sw.finalizeSeriesAdded(samplesPostRelabeling)
 	sw.addAutoTimeseries(wc, "up", float64(up), scrapeTimestamp)
 	sw.addAutoTimeseries(wc, "scrape_duration_seconds", duration, scrapeTimestamp)
 	sw.addAutoTimeseries(wc, "scrape_samples_scraped", float64(samplesScraped), scrapeTimestamp)
@@ -241,6 +260,7 @@ func (sw *scrapeWork) scrapeInternal(scrapeTimestamp, realTimestamp int64) error
 	startTime := time.Now()
 	sw.PushData(&wc.writeRequest)
 	pushDataDuration.UpdateDuration(startTime)
+	sw.prevRowsLen = samplesScraped
 	wc.reset()
 	writeRequestCtxPool.Put(wc)
 	// body must be released only after wc is released, since wc refers to body.
@@ -250,6 +270,50 @@ func (sw *scrapeWork) scrapeInternal(scrapeTimestamp, realTimestamp int64) error
 	return err
 }
 
+// leveledWriteRequestCtxPool allows reducing memory usage when writeRequesCtx
+// structs contain mixed number of labels.
+//
+// Its logic has been copied from leveledbytebufferpool.
+type leveledWriteRequestCtxPool struct {
+	pools [30]sync.Pool
+}
+
+func (lwp *leveledWriteRequestCtxPool) Get(rowsCapacity int) *writeRequestCtx {
+	id, capacityNeeded := lwp.getPoolIDAndCapacity(rowsCapacity)
+	for i := 0; i < 2; i++ {
+		if id < 0 || id >= len(lwp.pools) {
+			break
+		}
+		if v := lwp.pools[id].Get(); v != nil {
+			return v.(*writeRequestCtx)
+		}
+		id++
+	}
+	return &writeRequestCtx{
+		labels: make([]prompbmarshal.Label, 0, capacityNeeded),
+	}
+}
+
+func (lwp *leveledWriteRequestCtxPool) Put(wc *writeRequestCtx) {
+	capacity := cap(wc.rows.Rows)
+	id, _ := lwp.getPoolIDAndCapacity(capacity)
+	wc.reset()
+	lwp.pools[id].Put(wc)
+}
+
+func (lwp *leveledWriteRequestCtxPool) getPoolIDAndCapacity(size int) (int, int) {
+	size--
+	if size < 0 {
+		size = 0
+	}
+	size >>= 3
+	id := bits.Len(uint(size))
+	if id > len(lwp.pools) {
+		id = len(lwp.pools) - 1
+	}
+	return id, (1 << (id + 3))
+}
+
 type writeRequestCtx struct {
 	rows         parser.Rows
 	writeRequest prompbmarshal.WriteRequest
@@ -259,38 +323,38 @@ type writeRequestCtx struct {
 
 func (wc *writeRequestCtx) reset() {
 	wc.rows.Reset()
+	wc.resetNoRows()
+}
+
+func (wc *writeRequestCtx) resetNoRows() {
 	prompbmarshal.ResetWriteRequest(&wc.writeRequest)
 	wc.labels = wc.labels[:0]
 	wc.samples = wc.samples[:0]
 }
 
-var writeRequestCtxPool = &sync.Pool{
-	New: func() interface{} {
-		return &writeRequestCtx{}
-	},
-}
+var writeRequestCtxPool leveledWriteRequestCtxPool
 
-func (sw *scrapeWork) getSeriesAdded(wc *writeRequestCtx) int {
-	mPrev := sw.prevSeriesMap
-	seriesAdded := 0
+func (sw *scrapeWork) updateSeriesAdded(wc *writeRequestCtx) {
+	if sw.seriesMap == nil {
+		sw.seriesMap = make(map[uint64]struct{}, len(wc.writeRequest.Timeseries))
+	}
+	m := sw.seriesMap
 	for _, ts := range wc.writeRequest.Timeseries {
 		h := sw.getLabelsHash(ts.Labels)
-		if _, ok := mPrev[h]; !ok {
-			seriesAdded++
+		if _, ok := m[h]; !ok {
+			m[h] = struct{}{}
+			sw.seriesAdded++
 		}
 	}
-	if seriesAdded == 0 {
-		// Fast path: no new time series added during the last scrape.
-		return 0
-	}
+}
 
-	// Slow path: update the sw.prevSeriesMap, since new time series were added.
-	m := make(map[uint64]struct{}, len(wc.writeRequest.Timeseries))
-	for _, ts := range wc.writeRequest.Timeseries {
-		h := sw.getLabelsHash(ts.Labels)
-		m[h] = struct{}{}
+func (sw *scrapeWork) finalizeSeriesAdded(lastScrapeSize int) int {
+	seriesAdded := sw.seriesAdded
+	sw.seriesAdded = 0
+	if len(sw.seriesMap) > 4*lastScrapeSize {
+		// Reset seriesMap, since it occupies more than 4x metrics collected during the last scrape.
+		sw.seriesMap = make(map[uint64]struct{}, lastScrapeSize)
 	}
-	sw.prevSeriesMap = m
 	return seriesAdded
 }
 
@@ -330,19 +394,19 @@ func (sw *scrapeWork) addRowToTimeseries(wc *writeRequestCtx, r *parser.Row, tim
 		// Skip row without labels.
 		return
 	}
-	labels := wc.labels[labelsLen:]
-	wc.samples = append(wc.samples, prompbmarshal.Sample{})
-	sample := &wc.samples[len(wc.samples)-1]
-	sample.Value = r.Value
-	sample.Timestamp = r.Timestamp
-	if !sw.Config.HonorTimestamps || sample.Timestamp == 0 {
-		sample.Timestamp = timestamp
+	sampleTimestamp := r.Timestamp
+	if !sw.Config.HonorTimestamps || sampleTimestamp == 0 {
+		sampleTimestamp = timestamp
 	}
+	wc.samples = append(wc.samples, prompbmarshal.Sample{
+		Value:     r.Value,
+		Timestamp: sampleTimestamp,
+	})
 	wr := &wc.writeRequest
-	wr.Timeseries = append(wr.Timeseries, prompbmarshal.TimeSeries{})
-	ts := &wr.Timeseries[len(wr.Timeseries)-1]
-	ts.Labels = labels
-	ts.Samples = wc.samples[len(wc.samples)-1:]
+	wr.Timeseries = append(wr.Timeseries, prompbmarshal.TimeSeries{
+		Labels:  wc.labels[labelsLen:],
+		Samples: wc.samples[len(wc.samples)-1:],
+	})
 }
 
 func appendLabels(dst []prompbmarshal.Label, metric string, src []parser.Tag, extraLabels []prompbmarshal.Label, honorLabels bool) []prompbmarshal.Label {

lib/promscrape/scrapework_test.go

@@ -72,10 +72,17 @@ func TestScrapeWorkScrapeInternalSuccess(t *testing.T) {
 		pushDataCalls := 0
 		var pushDataErr error
 		sw.PushData = func(wr *prompbmarshal.WriteRequest) {
-			if err := expectEqualTimeseries(wr.Timeseries, timeseriesExpected); err != nil {
-				pushDataErr = fmt.Errorf("unexpected data pushed: %w\ngot\n%#v\nwant\n%#v", err, wr.Timeseries, timeseriesExpected)
-			}
 			pushDataCalls++
+			if len(wr.Timeseries) > len(timeseriesExpected) {
+				pushDataErr = fmt.Errorf("too many time series obtained; got %d; want %d", len(wr.Timeseries), len(timeseriesExpected))
+				return
+			}
+			tsExpected := timeseriesExpected[:len(wr.Timeseries)]
+			timeseriesExpected = timeseriesExpected[len(tsExpected):]
+			if err := expectEqualTimeseries(wr.Timeseries, tsExpected); err != nil {
+				pushDataErr = fmt.Errorf("unexpected data pushed: %w\ngot\n%v\nwant\n%v", err, wr.Timeseries, tsExpected)
+				return
+			}
 		}
 
 		timestamp := int64(123)
@@ -88,8 +95,8 @@ func TestScrapeWorkScrapeInternalSuccess(t *testing.T) {
 		if readDataCalls != 1 {
 			t.Fatalf("unexpected number of readData calls; got %d; want %d", readDataCalls, 1)
 		}
-		if pushDataCalls != 1 {
-			t.Fatalf("unexpected number of pushData calls; got %d; want %d", pushDataCalls, 1)
+		if pushDataCalls == 0 {
+			t.Fatalf("missing pushData calls")
 		}
 	}
 
@@ -359,7 +366,7 @@ func expectEqualTimeseries(tss, tssExpected []prompbmarshal.TimeSeries) error {
 	for k, tsExpected := range mExpected {
 		ts := m[k]
 		if ts != tsExpected {
-			return fmt.Errorf("unexpected timeseries %q; got\n%s\nwant\n%s", k, ts, tsExpected)
+			return fmt.Errorf("unexpected timeseries %q;\ngot\n%s\nwant\n%s", k, ts, tsExpected)
 		}
 	}
 	return nil

lib/protoparser/common/extra_labels.go

@@ -0,0 +1,26 @@
+package common
+
+import (
+	"fmt"
+	"net/http"
+	"strings"
+
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
+)
+
+// GetExtraLabels extracts name:value labels from `extra_label=name=value` query args from req.
+func GetExtraLabels(req *http.Request) ([]prompbmarshal.Label, error) {
+	q := req.URL.Query()
+	var result []prompbmarshal.Label
+	for _, label := range q["extra_label"] {
+		tmp := strings.SplitN(label, "=", 2)
+		if len(tmp) != 2 {
+			return nil, fmt.Errorf("`extra_label` query arg must have the format `name=value`; got %q", label)
+		}
+		result = append(result, prompbmarshal.Label{
+			Name:  tmp[0],
+			Value: tmp[1],
+		})
+	}
+	return result, nil
+}

lib/protoparser/common/timestamp.go

@@ -0,0 +1,22 @@
+package common
+
+import (
+	"fmt"
+	"net/http"
+	"strconv"
+)
+
+// GetTimestamp extracts unix timestamp in milliseconds from `timestamp` query arg.
+//
+// It returns 0 if there is no `timestamp` query arg.
+func GetTimestamp(req *http.Request) (int64, error) {
+	ts := req.URL.Query().Get("timestamp")
+	if len(ts) == 0 {
+		return 0, nil
+	}
+	timestamp, err := strconv.ParseInt(ts, 10, 64)
+	if err != nil {
+		return 0, fmt.Errorf("cannot parse `timestamp=%s` query arg: %w", ts, err)
+	}
+	return timestamp, nil
+}

lib/protoparser/opentsdbhttp/streamparser.go

@@ -11,12 +11,13 @@ import (
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fasttime"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/common"
 	"github.com/VictoriaMetrics/metrics"
 )
 
 var (
-	maxInsertRequestSize = flag.Int("opentsdbhttp.maxInsertRequestSize", 32*1024*1024, "The maximum size of OpenTSDB HTTP put request")
+	maxInsertRequestSize = flagutil.NewBytes("opentsdbhttp.maxInsertRequestSize", 32*1024*1024, "The maximum size of OpenTSDB HTTP put request")
 	trimTimestamp        = flag.Duration("opentsdbhttpTrimTimestamp", time.Millisecond, "Trim timestamps for OpenTSDB HTTP data to this duration. "+
 		"Minimum practical duration is 1ms. Higher duration (i.e. 1s) may be used for reducing disk space usage for timestamp data")
 )
@@ -43,15 +44,15 @@ func ParseStream(req *http.Request, callback func(rows []Row) error) error {
 	defer putStreamContext(ctx)
 
 	// Read the request in ctx.reqBuf
-	lr := io.LimitReader(r, int64(*maxInsertRequestSize)+1)
+	lr := io.LimitReader(r, int64(maxInsertRequestSize.N)+1)
 	reqLen, err := ctx.reqBuf.ReadFrom(lr)
 	if err != nil {
 		readErrors.Inc()
 		return fmt.Errorf("cannot read HTTP OpenTSDB request: %w", err)
 	}
-	if reqLen > int64(*maxInsertRequestSize) {
+	if reqLen > int64(maxInsertRequestSize.N) {
 		readErrors.Inc()
-		return fmt.Errorf("too big HTTP OpenTSDB request; mustn't exceed `-opentsdbhttp.maxInsertRequestSize=%d` bytes", *maxInsertRequestSize)
+		return fmt.Errorf("too big HTTP OpenTSDB request; mustn't exceed `-opentsdbhttp.maxInsertRequestSize=%d` bytes", maxInsertRequestSize.N)
 	}
 
 	// Unmarshal the request to ctx.Rows

lib/protoparser/prometheus/streamparser.go

@@ -17,7 +17,7 @@ import (
 // The callback can be called multiple times for streamed data from r.
 //
 // callback shouldn't hold rows after returning.
-func ParseStream(r io.Reader, isGzipped bool, callback func(rows []Row) error) error {
+func ParseStream(r io.Reader, defaultTimestamp int64, isGzipped bool, callback func(rows []Row) error) error {
 	if isGzipped {
 		zr, err := common.GetGzipReader(r)
 		if err != nil {
@@ -28,7 +28,7 @@ func ParseStream(r io.Reader, isGzipped bool, callback func(rows []Row) error) e
 	}
 	ctx := getStreamContext()
 	defer putStreamContext(ctx)
-	for ctx.Read(r) {
+	for ctx.Read(r, defaultTimestamp) {
 		if err := callback(ctx.Rows.Rows); err != nil {
 			return err
 		}
@@ -36,7 +36,7 @@ func ParseStream(r io.Reader, isGzipped bool, callback func(rows []Row) error) e
 	return ctx.Error()
 }
 
-func (ctx *streamContext) Read(r io.Reader) bool {
+func (ctx *streamContext) Read(r io.Reader, defaultTimestamp int64) bool {
 	readCalls.Inc()
 	if ctx.err != nil {
 		return false
@@ -55,11 +55,13 @@ func (ctx *streamContext) Read(r io.Reader) bool {
 	rows := ctx.Rows.Rows
 
 	// Fill missing timestamps with the current timestamp.
-	currentTimestamp := int64(time.Now().UnixNano() / 1e6)
+	if defaultTimestamp <= 0 {
+		defaultTimestamp = int64(time.Now().UnixNano() / 1e6)
+	}
 	for i := range rows {
 		r := &rows[i]
 		if r.Timestamp == 0 {
-			r.Timestamp = currentTimestamp
+			r.Timestamp = defaultTimestamp
 		}
 	}
 	return true

lib/protoparser/prometheus/streamparser_test.go

@@ -8,11 +8,12 @@ import (
 )
 
 func TestParseStream(t *testing.T) {
+	const defaultTimestamp = 123
 	f := func(s string, rowsExpected []Row) {
 		t.Helper()
 		bb := bytes.NewBufferString(s)
 		var result []Row
-		err := ParseStream(bb, false, func(rows []Row) error {
+		err := ParseStream(bb, defaultTimestamp, false, func(rows []Row) error {
 			result = appendRowCopies(result, rows)
 			return nil
 		})
@@ -33,7 +34,7 @@ func TestParseStream(t *testing.T) {
 			t.Fatalf("unexpected error when closing gzip writer: %s", err)
 		}
 		result = nil
-		err = ParseStream(bb, true, func(rows []Row) error {
+		err = ParseStream(bb, defaultTimestamp, true, func(rows []Row) error {
 			result = appendRowCopies(result, rows)
 			return nil
 		})
@@ -67,6 +68,11 @@ func TestParseStream(t *testing.T) {
 			Timestamp: 4,
 		},
 	})
+	f("foo 23", []Row{{
+		Metric:    "foo",
+		Value:     23,
+		Timestamp: defaultTimestamp,
+	}})
 }
 
 func appendRowCopies(dst, src []Row) []Row {

lib/protoparser/promremotewrite/streamparser.go

@@ -1,7 +1,6 @@
 package promremotewrite
 
 import (
-	"flag"
 	"fmt"
 	"io"
 	"net/http"
@@ -9,12 +8,13 @@ import (
 	"sync"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompb"
 	"github.com/VictoriaMetrics/metrics"
 	"github.com/golang/snappy"
 )
 
-var maxInsertRequestSize = flag.Int("maxInsertRequestSize", 32*1024*1024, "The maximum size in bytes of a single Prometheus remote_write API request")
+var maxInsertRequestSize = flagutil.NewBytes("maxInsertRequestSize", 32*1024*1024, "The maximum size in bytes of a single Prometheus remote_write API request")
 
 // ParseStream parses Prometheus remote_write message req and calls callback for the parsed timeseries.
 //
@@ -93,15 +93,15 @@ var pushCtxPool sync.Pool
 var pushCtxPoolCh = make(chan *pushCtx, runtime.GOMAXPROCS(-1))
 
 func readSnappy(dst []byte, r io.Reader) ([]byte, error) {
-	lr := io.LimitReader(r, int64(*maxInsertRequestSize)+1)
+	lr := io.LimitReader(r, int64(maxInsertRequestSize.N)+1)
 	bb := bodyBufferPool.Get()
 	reqLen, err := bb.ReadFrom(lr)
 	if err != nil {
 		bodyBufferPool.Put(bb)
 		return dst, fmt.Errorf("cannot read compressed request: %w", err)
 	}
-	if reqLen > int64(*maxInsertRequestSize) {
-		return dst, fmt.Errorf("too big packed request; mustn't exceed `-maxInsertRequestSize=%d` bytes", *maxInsertRequestSize)
+	if reqLen > int64(maxInsertRequestSize.N) {
+		return dst, fmt.Errorf("too big packed request; mustn't exceed `-maxInsertRequestSize=%d` bytes", maxInsertRequestSize.N)
 	}
 
 	buf := dst[len(dst):cap(dst)]
@@ -111,8 +111,8 @@ func readSnappy(dst []byte, r io.Reader) ([]byte, error) {
 		err = fmt.Errorf("cannot decompress request with length %d: %w", reqLen, err)
 		return dst, err
 	}
-	if len(buf) > *maxInsertRequestSize {
-		return dst, fmt.Errorf("too big unpacked request; mustn't exceed `-maxInsertRequestSize=%d` bytes; got %d bytes", *maxInsertRequestSize, len(buf))
+	if len(buf) > maxInsertRequestSize.N {
+		return dst, fmt.Errorf("too big unpacked request; mustn't exceed `-maxInsertRequestSize=%d` bytes; got %d bytes", maxInsertRequestSize.N, len(buf))
 	}
 	if len(buf) > 0 && len(dst) < cap(dst) && &buf[0] == &dst[len(dst):cap(dst)][0] {
 		dst = dst[:len(dst)+len(buf)]

lib/protoparser/vmimport/streamparser.go

@@ -1,7 +1,6 @@
 package vmimport
 
 import (
-	"flag"
 	"fmt"
 	"io"
 	"net/http"
@@ -9,11 +8,12 @@ import (
 	"sync"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/common"
 	"github.com/VictoriaMetrics/metrics"
 )
 
-var maxLineLen = flag.Int("import.maxLineLen", 100*1024*1024, "The maximum length in bytes of a single line accepted by /api/v1/import")
+var maxLineLen = flagutil.NewBytes("import.maxLineLen", 100*1024*1024, "The maximum length in bytes of a single line accepted by /api/v1/import")
 
 // ParseStream parses /api/v1/import lines from req and calls callback for the parsed rows.
 //
@@ -46,7 +46,7 @@ func (ctx *streamContext) Read(r io.Reader) bool {
 	if ctx.err != nil {
 		return false
 	}
-	ctx.reqBuf, ctx.tailBuf, ctx.err = common.ReadLinesBlockExt(r, ctx.reqBuf, ctx.tailBuf, *maxLineLen)
+	ctx.reqBuf, ctx.tailBuf, ctx.err = common.ReadLinesBlockExt(r, ctx.reqBuf, ctx.tailBuf, maxLineLen.N)
 	if ctx.err != nil {
 		if ctx.err != io.EOF {
 			readErrors.Inc()

lib/storage/block_stream_reader.go

@@ -51,6 +51,9 @@ type blockStreamReader struct {
 	valuesBlockOffset     uint64
 	indexBlockOffset      uint64
 
+	prevTimestampsBlockOffset uint64
+	prevTimestampsData        []byte
+
 	indexData           []byte
 	compressedIndexData []byte
 
@@ -87,6 +90,9 @@ func (bsr *blockStreamReader) reset() {
 	bsr.valuesBlockOffset = 0
 	bsr.indexBlockOffset = 0
 
+	bsr.prevTimestampsBlockOffset = 0
+	bsr.prevTimestampsData = bsr.prevTimestampsData[:0]
+
 	bsr.indexData = bsr.indexData[:0]
 	bsr.compressedIndexData = bsr.compressedIndexData[:0]
 
@@ -275,7 +281,13 @@ func (bsr *blockStreamReader) readBlock() error {
 		return fmt.Errorf("invalid MaxTimestamp at block header at offset %d; got %d; cannot be bigger than %d",
 			bsr.prevIndexBlockOffset(), bsr.Block.bh.MaxTimestamp, bsr.ph.MaxTimestamp)
 	}
-	if bsr.Block.bh.TimestampsBlockOffset != bsr.timestampsBlockOffset {
+	usePrevTimestamps := len(bsr.prevTimestampsData) > 0 && bsr.Block.bh.TimestampsBlockOffset == bsr.prevTimestampsBlockOffset
+	if usePrevTimestamps {
+		if int(bsr.Block.bh.TimestampsBlockSize) != len(bsr.prevTimestampsData) {
+			return fmt.Errorf("invalid TimestampsBlockSize at block header at offset %d; got %d; want %d",
+				bsr.prevIndexBlockOffset(), bsr.Block.bh.TimestampsBlockSize, len(bsr.prevTimestampsData))
+		}
+	} else if bsr.Block.bh.TimestampsBlockOffset != bsr.timestampsBlockOffset {
 		return fmt.Errorf("invalid TimestampsBlockOffset at block header at offset %d; got %d; want %d",
 			bsr.prevIndexBlockOffset(), bsr.Block.bh.TimestampsBlockOffset, bsr.timestampsBlockOffset)
 	}
@@ -285,9 +297,15 @@ func (bsr *blockStreamReader) readBlock() error {
 	}
 
 	// Read timestamps data.
-	bsr.Block.timestampsData = bytesutil.Resize(bsr.Block.timestampsData, int(bsr.Block.bh.TimestampsBlockSize))
-	if err := fs.ReadFullData(bsr.timestampsReader, bsr.Block.timestampsData); err != nil {
-		return fmt.Errorf("cannot read timestamps block at offset %d: %w", bsr.timestampsBlockOffset, err)
+	if usePrevTimestamps {
+		bsr.Block.timestampsData = append(bsr.Block.timestampsData[:0], bsr.prevTimestampsData...)
+	} else {
+		bsr.Block.timestampsData = bytesutil.Resize(bsr.Block.timestampsData, int(bsr.Block.bh.TimestampsBlockSize))
+		if err := fs.ReadFullData(bsr.timestampsReader, bsr.Block.timestampsData); err != nil {
+			return fmt.Errorf("cannot read timestamps block at offset %d: %w", bsr.timestampsBlockOffset, err)
+		}
+		bsr.prevTimestampsBlockOffset = bsr.timestampsBlockOffset
+		bsr.prevTimestampsData = append(bsr.prevTimestampsData[:0], bsr.Block.timestampsData...)
 	}
 
 	// Read values data.
@@ -297,7 +315,9 @@ func (bsr *blockStreamReader) readBlock() error {
 	}
 
 	// Update offsets.
-	bsr.timestampsBlockOffset += uint64(bsr.Block.bh.TimestampsBlockSize)
+	if !usePrevTimestamps {
+		bsr.timestampsBlockOffset += uint64(bsr.Block.bh.TimestampsBlockSize)
+	}
 	bsr.valuesBlockOffset += uint64(bsr.Block.bh.ValuesBlockSize)
 	bsr.indexBlockHeadersCount++
 

lib/storage/block_stream_writer.go

@@ -1,6 +1,7 @@
 package storage
 
 import (
+	"bytes"
 	"fmt"
 	"io"
 	"path/filepath"
@@ -38,6 +39,13 @@ type blockStreamWriter struct {
 
 	metaindexData           []byte
 	compressedMetaindexData []byte
+
+	// prevTimestamps* is used as an optimization for reducing disk space usage
+	// when serially written blocks have identical timestamps.
+	// This is usually the case when adjancent blocks contain metrics scraped from the same target,
+	// since such metrics have identical timestamps.
+	prevTimestampsData        []byte
+	prevTimestampsBlockOffset uint64
 }
 
 func (bsw *blockStreamWriter) assertWriteClosers() {
@@ -66,6 +74,9 @@ func (bsw *blockStreamWriter) reset() {
 
 	bsw.metaindexData = bsw.metaindexData[:0]
 	bsw.compressedMetaindexData = bsw.compressedMetaindexData[:0]
+
+	bsw.prevTimestampsData = bsw.prevTimestampsData[:0]
+	bsw.prevTimestampsBlockOffset = 0
 }
 
 // InitFromInmemoryPart initialzes bsw from inmemory part.
@@ -177,22 +188,35 @@ func (bsw *blockStreamWriter) WriteExternalBlock(b *Block, ph *partHeader, rowsM
 	atomic.AddUint64(rowsMerged, uint64(b.rowsCount()))
 	b.deduplicateSamplesDuringMerge()
 	headerData, timestampsData, valuesData := b.MarshalData(bsw.timestampsBlockOffset, bsw.valuesBlockOffset)
-
+	usePrevTimestamps := len(bsw.prevTimestampsData) > 0 && bytes.Equal(timestampsData, bsw.prevTimestampsData)
+	if usePrevTimestamps {
+		// The current timestamps block equals to the previous timestamps block.
+		// Update headerData so it points to the previous timestamps block. This saves disk space.
+		headerData, timestampsData, valuesData = b.MarshalData(bsw.prevTimestampsBlockOffset, bsw.valuesBlockOffset)
+		atomic.AddUint64(&timestampsBlocksMerged, 1)
+		atomic.AddUint64(&timestampsBytesSaved, uint64(len(timestampsData)))
+	}
 	bsw.indexData = append(bsw.indexData, headerData...)
 	bsw.mr.RegisterBlockHeader(&b.bh)
 	if len(bsw.indexData) >= maxBlockSize {
 		bsw.flushIndexData()
 	}
-
-	fs.MustWriteData(bsw.timestampsWriter, timestampsData)
-	bsw.timestampsBlockOffset += uint64(len(timestampsData))
-
+	if !usePrevTimestamps {
+		bsw.prevTimestampsData = append(bsw.prevTimestampsData[:0], timestampsData...)
+		bsw.prevTimestampsBlockOffset = bsw.timestampsBlockOffset
+		fs.MustWriteData(bsw.timestampsWriter, timestampsData)
+		bsw.timestampsBlockOffset += uint64(len(timestampsData))
+	}
 	fs.MustWriteData(bsw.valuesWriter, valuesData)
 	bsw.valuesBlockOffset += uint64(len(valuesData))
-
 	updatePartHeader(b, ph)
 }
 
+var (
+	timestampsBlocksMerged uint64
+	timestampsBytesSaved   uint64
+)
+
 func updatePartHeader(b *Block, ph *partHeader) {
 	ph.BlocksCount++
 	ph.RowsCount += uint64(b.bh.RowsCount)

lib/storage/index_db.go

@@ -901,6 +901,152 @@ func (is *indexSearch) searchTagValues(tvs map[string]struct{}, tagKey []byte, m
 	return nil
 }
 
+// SearchTagValueSuffixes returns all the tag value suffixes for the given tagKey and tagValuePrefix on the given tr.
+//
+// This allows implementing https://graphite-api.readthedocs.io/en/latest/api.html#metrics-find or similar APIs.
+func (db *indexDB) SearchTagValueSuffixes(tr TimeRange, tagKey, tagValuePrefix []byte, delimiter byte, maxTagValueSuffixes int, deadline uint64) ([]string, error) {
+	// TODO: cache results?
+
+	tvss := make(map[string]struct{})
+	is := db.getIndexSearch(deadline)
+	err := is.searchTagValueSuffixesForTimeRange(tvss, tr, tagKey, tagValuePrefix, delimiter, maxTagValueSuffixes)
+	db.putIndexSearch(is)
+	if err != nil {
+		return nil, err
+	}
+	ok := db.doExtDB(func(extDB *indexDB) {
+		is := extDB.getIndexSearch(deadline)
+		err = is.searchTagValueSuffixesForTimeRange(tvss, tr, tagKey, tagValuePrefix, delimiter, maxTagValueSuffixes)
+		extDB.putIndexSearch(is)
+	})
+	if ok && err != nil {
+		return nil, err
+	}
+
+	suffixes := make([]string, 0, len(tvss))
+	for suffix := range tvss {
+		// Do not skip empty suffixes, since they may represent leaf tag values.
+		suffixes = append(suffixes, suffix)
+	}
+	// Do not sort suffixes, since they must be sorted by vmselect.
+	return suffixes, nil
+}
+
+func (is *indexSearch) searchTagValueSuffixesForTimeRange(tvss map[string]struct{}, tr TimeRange, tagKey, tagValuePrefix []byte, delimiter byte, maxTagValueSuffixes int) error {
+	minDate := uint64(tr.MinTimestamp) / msecPerDay
+	maxDate := uint64(tr.MaxTimestamp) / msecPerDay
+	if maxDate-minDate > maxDaysForDateMetricIDs {
+		return is.searchTagValueSuffixesAll(tvss, tagKey, tagValuePrefix, delimiter, maxTagValueSuffixes)
+	}
+	// Query over multiple days in parallel.
+	var wg sync.WaitGroup
+	var errGlobal error
+	var mu sync.Mutex // protects tvss + errGlobal from concurrent access below.
+	for minDate <= maxDate {
+		wg.Add(1)
+		go func(date uint64) {
+			defer wg.Done()
+			tvssLocal := make(map[string]struct{})
+			isLocal := is.db.getIndexSearch(is.deadline)
+			defer is.db.putIndexSearch(isLocal)
+			err := isLocal.searchTagValueSuffixesForDate(tvssLocal, date, tagKey, tagValuePrefix, delimiter, maxTagValueSuffixes)
+			mu.Lock()
+			defer mu.Unlock()
+			if errGlobal != nil {
+				return
+			}
+			if err != nil {
+				errGlobal = err
+				return
+			}
+			for k := range tvssLocal {
+				tvss[k] = struct{}{}
+			}
+		}(minDate)
+		minDate++
+	}
+	wg.Wait()
+	return errGlobal
+}
+
+func (is *indexSearch) searchTagValueSuffixesAll(tvss map[string]struct{}, tagKey, tagValuePrefix []byte, delimiter byte, maxTagValueSuffixes int) error {
+	kb := &is.kb
+	nsPrefix := byte(nsPrefixTagToMetricIDs)
+	kb.B = is.marshalCommonPrefix(kb.B[:0], nsPrefix)
+	kb.B = marshalTagValue(kb.B, tagKey)
+	kb.B = marshalTagValue(kb.B, tagValuePrefix)
+	kb.B = kb.B[:len(kb.B)-1] // remove tagSeparatorChar from the end of kb.B
+	prefix := append([]byte(nil), kb.B...)
+	return is.searchTagValueSuffixesForPrefix(tvss, nsPrefix, prefix, tagValuePrefix, delimiter, maxTagValueSuffixes)
+}
+
+func (is *indexSearch) searchTagValueSuffixesForDate(tvss map[string]struct{}, date uint64, tagKey, tagValuePrefix []byte, delimiter byte, maxTagValueSuffixes int) error {
+	nsPrefix := byte(nsPrefixDateTagToMetricIDs)
+	kb := &is.kb
+	kb.B = is.marshalCommonPrefix(kb.B[:0], nsPrefix)
+	kb.B = encoding.MarshalUint64(kb.B, date)
+	kb.B = marshalTagValue(kb.B, tagKey)
+	kb.B = marshalTagValue(kb.B, tagValuePrefix)
+	kb.B = kb.B[:len(kb.B)-1] // remove tagSeparatorChar from the end of kb.B
+	prefix := append([]byte(nil), kb.B...)
+	return is.searchTagValueSuffixesForPrefix(tvss, nsPrefix, prefix, tagValuePrefix, delimiter, maxTagValueSuffixes)
+}
+
+func (is *indexSearch) searchTagValueSuffixesForPrefix(tvss map[string]struct{}, nsPrefix byte, prefix, tagValuePrefix []byte, delimiter byte, maxTagValueSuffixes int) error {
+	kb := &is.kb
+	ts := &is.ts
+	mp := &is.mp
+	mp.Reset()
+	dmis := is.db.getDeletedMetricIDs()
+	loopsPaceLimiter := 0
+	ts.Seek(prefix)
+	for len(tvss) < maxTagValueSuffixes && ts.NextItem() {
+		if loopsPaceLimiter&paceLimiterFastIterationsMask == 0 {
+			if err := checkSearchDeadlineAndPace(is.deadline); err != nil {
+				return err
+			}
+		}
+		loopsPaceLimiter++
+		item := ts.Item
+		if !bytes.HasPrefix(item, prefix) {
+			break
+		}
+		if err := mp.Init(item, nsPrefix); err != nil {
+			return err
+		}
+		if mp.IsDeletedTag(dmis) {
+			continue
+		}
+		tagValue := mp.Tag.Value
+		if !bytes.HasPrefix(tagValue, tagValuePrefix) {
+			continue
+		}
+		suffix := tagValue[len(tagValuePrefix):]
+		n := bytes.IndexByte(suffix, delimiter)
+		if n < 0 {
+			// Found leaf tag value that doesn't have delimiters after the given tagValuePrefix.
+			tvss[string(suffix)] = struct{}{}
+			continue
+		}
+		// Found non-leaf tag value. Extract suffix that end with the given delimiter.
+		suffix = suffix[:n+1]
+		tvss[string(suffix)] = struct{}{}
+		if suffix[len(suffix)-1] == 255 {
+			continue
+		}
+		// Search for the next suffix
+		suffix[len(suffix)-1]++
+		kb.B = append(kb.B[:0], prefix...)
+		kb.B = marshalTagValue(kb.B, suffix)
+		kb.B = kb.B[:len(kb.B)-1] // remove tagSeparatorChar
+		ts.Seek(kb.B)
+	}
+	if err := ts.Error(); err != nil {
+		return fmt.Errorf("error when searching for tag value sufixes for prefix %q: %w", prefix, err)
+	}
+	return nil
+}
+
 // GetSeriesCount returns the approximate number of unique timeseries in the db.
 //
 // It includes the deleted series too and may count the same series

lib/storage/storage.go

@@ -340,6 +340,9 @@ type Metrics struct {
 	SlowPerDayIndexInserts uint64
 	SlowMetricNameLoads    uint64
 
+	TimestampsBlocksMerged uint64
+	TimestampsBytesSaved   uint64
+
 	TSIDCacheSize       uint64
 	TSIDCacheSizeBytes  uint64
 	TSIDCacheRequests   uint64
@@ -405,6 +408,9 @@ func (s *Storage) UpdateMetrics(m *Metrics) {
 	m.SlowPerDayIndexInserts += atomic.LoadUint64(&s.slowPerDayIndexInserts)
 	m.SlowMetricNameLoads += atomic.LoadUint64(&s.slowMetricNameLoads)
 
+	m.TimestampsBlocksMerged = atomic.LoadUint64(&timestampsBlocksMerged)
+	m.TimestampsBytesSaved = atomic.LoadUint64(&timestampsBytesSaved)
+
 	var cs fastcache.Stats
 	s.tsidCache.UpdateStats(&cs)
 	m.TSIDCacheSize += cs.EntriesCount
@@ -923,6 +929,13 @@ func (s *Storage) SearchTagValues(tagKey []byte, maxTagValues int, deadline uint
 	return s.idb().SearchTagValues(tagKey, maxTagValues, deadline)
 }
 
+// SearchTagValueSuffixes returns all the tag value suffixes for the given tagKey and tagValuePrefix on the given tr.
+//
+// This allows implementing https://graphite-api.readthedocs.io/en/latest/api.html#metrics-find or similar APIs.
+func (s *Storage) SearchTagValueSuffixes(tr TimeRange, tagKey, tagValuePrefix []byte, delimiter byte, maxTagValueSuffixes int, deadline uint64) ([]string, error) {
+	return s.idb().SearchTagValueSuffixes(tr, tagKey, tagValuePrefix, delimiter, maxTagValueSuffixes, deadline)
+}
+
 // SearchTagEntries returns a list of (tagName -> tagValues)
 func (s *Storage) SearchTagEntries(maxTagKeys, maxTagValues int, deadline uint64) ([]TagEntry, error) {
 	idb := s.idb()
@@ -1107,6 +1120,11 @@ func (s *Storage) add(rows []rawRow, mrs []MetricRow, precisionBits uint8) ([]ra
 			// doesn't know how to work with them.
 			continue
 		}
+		if math.IsInf(mr.Value, 0) {
+			// Skip Inf values, since they may break precision for already stored data.
+			// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/752
+			continue
+		}
 		if mr.Timestamp < minTimestamp {
 			// Skip rows with too small timestamps outside the retention.
 			if firstWarn == nil {

vendor/cloud.google.com/go/CHANGES.md

@@ -1,5 +1,44 @@
 # Changes
 
+## [0.65.0](https://www.github.com/googleapis/google-cloud-go/compare/v0.64.0...v0.65.0) (2020-08-27)
+
+
+### Announcements
+
+The following changes will be included in an upcoming release and are not 
+included in this one.
+
+#### Default Deadlines
+
+By default, non-streaming methods, like Create or Get methods, will have a
+default deadline applied to the context provided at call time, unless a context
+deadline is already set. Streaming methods have no default deadline and will run
+indefinitely, unless the context provided at call time contains a deadline.
+
+To opt-out of this behavior, set the environment variable
+`GOOGLE_API_GO_EXPERIMENTAL_DISABLE_DEFAULT_DEADLINE` to `true` prior to
+initializing a client. This opt-out mechanism will be removed in a later
+release, with a notice similar to this one ahead of its removal.
+
+
+### Features
+
+* **all:** auto-regenerate gapics , refs [#2774](https://www.github.com/googleapis/google-cloud-go/issues/2774) [#2764](https://www.github.com/googleapis/google-cloud-go/issues/2764)
+
+
+### Bug Fixes
+
+* **all:** correct minor typos ([#2756](https://www.github.com/googleapis/google-cloud-go/issues/2756)) ([03d78b5](https://www.github.com/googleapis/google-cloud-go/commit/03d78b5627819cb64d1f3866f90043f709e825e1))
+* **compute/metadata:** remove leading slash for Get suffix ([#2760](https://www.github.com/googleapis/google-cloud-go/issues/2760)) ([f0d605c](https://www.github.com/googleapis/google-cloud-go/commit/f0d605ccf32391a9da056a2c551158bd076c128d))
+
+## [0.64.0](https://www.github.com/googleapis/google-cloud-go/compare/v0.63.0...v0.64.0) (2020-08-18)
+
+
+### Features
+
+* **all:** auto-regenerate gapics , refs [#2734](https://www.github.com/googleapis/google-cloud-go/issues/2734) [#2731](https://www.github.com/googleapis/google-cloud-go/issues/2731) [#2730](https://www.github.com/googleapis/google-cloud-go/issues/2730) [#2725](https://www.github.com/googleapis/google-cloud-go/issues/2725) [#2722](https://www.github.com/googleapis/google-cloud-go/issues/2722) [#2706](https://www.github.com/googleapis/google-cloud-go/issues/2706)
+* **pubsublite:** start generating v1 ([#2700](https://www.github.com/googleapis/google-cloud-go/issues/2700)) ([d2e777f](https://www.github.com/googleapis/google-cloud-go/commit/d2e777f56e08146646b3ffb7a78856795094ab4e))
+
 ## [0.63.0](https://www.github.com/googleapis/google-cloud-go/compare/v0.62.0...v0.63.0) (2020-08-05)
 
 

vendor/cloud.google.com/go/compute/metadata/metadata.go

@@ -296,6 +296,7 @@ func (c *Client) getETag(suffix string) (value, etag string, err error) {
 		// being stable anyway.
 		host = metadataIP
 	}
+	suffix = strings.TrimLeft(suffix, "/")
 	u := "http://" + host + "/computeMetadata/v1/" + suffix
 	req, err := http.NewRequest("GET", u, nil)
 	if err != nil {

vendor/cloud.google.com/go/go.mod

@@ -13,12 +13,12 @@ require (
 	github.com/jstemmer/go-junit-report v0.9.1
 	go.opencensus.io v0.22.4
 	golang.org/x/lint v0.0.0-20200302205851-738671d3881b
-	golang.org/x/net v0.0.0-20200707034311-ab3426394381
+	golang.org/x/net v0.0.0-20200822124328-c89045814202
 	golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d
 	golang.org/x/text v0.3.3
-	golang.org/x/tools v0.0.0-20200806022845-90696ccdc692
+	golang.org/x/tools v0.0.0-20200825202427-b303f430e36d
 	golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
 	google.golang.org/api v0.30.0
-	google.golang.org/genproto v0.0.0-20200806141610-86f49bd18e98
+	google.golang.org/genproto v0.0.0-20200825200019-8632dd797987
 	google.golang.org/grpc v1.31.0
 )

vendor/cloud.google.com/go/go.sum

@@ -247,6 +247,8 @@ golang.org/x/net v0.0.0-20200625001655-4c5254603344 h1:vGXIOMxbNfDTk/aXCmfdLgkrS
 golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
 golang.org/x/net v0.0.0-20200707034311-ab3426394381 h1:VXak5I6aEWmAXeQjA+QSZzlgNrpq9mjcfDemuexIKsU=
 golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
+golang.org/x/net v0.0.0-20200822124328-c89045814202 h1:VvcQYSHwXgi7W+TpUR6A9g6Up98WAHf3f/ulnJ62IyA=
+golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
 golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
 golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421 h1:Wo7BWFiOk0QRFMLYMqJGFMd9CgUAcGx7V+qEg/h5IBI=
 golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
@@ -361,8 +363,8 @@ golang.org/x/tools v0.0.0-20200618134242-20370b0cb4b2/go.mod h1:EkVYQZoAsY45+roY
 golang.org/x/tools v0.0.0-20200729194436-6467de6f59a7/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA=
 golang.org/x/tools v0.0.0-20200804011535-6c149bb5ef0d h1:szSOL78iTCl0LF1AMjhSWJj8tIM0KixlUUnBtYXsmd8=
 golang.org/x/tools v0.0.0-20200804011535-6c149bb5ef0d/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA=
-golang.org/x/tools v0.0.0-20200806022845-90696ccdc692 h1:fsn47thVa7Ar/TMyXYlZgOoT7M4+kRpb+KpSAqRQx1w=
-golang.org/x/tools v0.0.0-20200806022845-90696ccdc692/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA=
+golang.org/x/tools v0.0.0-20200825202427-b303f430e36d h1:W07d4xkoAUSNOkOzdzXCdFGxT7o2rW4q8M34tB2i//k=
+golang.org/x/tools v0.0.0-20200825202427-b303f430e36d/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA=
 golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
@@ -440,8 +442,8 @@ google.golang.org/genproto v0.0.0-20200618031413-b414f8b61790/go.mod h1:jDfRM7Fc
 google.golang.org/genproto v0.0.0-20200729003335-053ba62fc06f/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
 google.golang.org/genproto v0.0.0-20200804131852-c06518451d9c h1:Lq4llNryJoaVFRmvrIwC/ZHH7tNt4tUYIu8+se2aayY=
 google.golang.org/genproto v0.0.0-20200804131852-c06518451d9c/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
-google.golang.org/genproto v0.0.0-20200806141610-86f49bd18e98 h1:LCO0fg4kb6WwkXQXRQQgUYsFeFb5taTX5WAx5O/Vt28=
-google.golang.org/genproto v0.0.0-20200806141610-86f49bd18e98/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
+google.golang.org/genproto v0.0.0-20200825200019-8632dd797987 h1:PDIOdWxZ8eRizhKa1AAvY53xsvLB1cWorMjslvY3VA8=
+google.golang.org/genproto v0.0.0-20200825200019-8632dd797987/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
 google.golang.org/grpc v1.19.0 h1:cfg4PD8YEdSFnm7qLV4++93WcmhH2nIUhMjhdCvl3j8=
 google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
 google.golang.org/grpc v1.20.1 h1:Hz2g2wirWK7H0qIIhGIqRGTuMwTE8HEKFnDZZ7lm9NU=

vendor/cloud.google.com/go/internal/.repo-metadata-full.json

@@ -535,6 +535,14 @@
     "docs_url": "https://pkg.go.dev/cloud.google.com/go/pubsub/apiv1",
     "release_level": "ga"
   },
+  "cloud.google.com/go/pubsublite/apiv1": {
+    "distribution_name": "cloud.google.com/go/pubsublite/apiv1",
+    "description": "",
+    "language": "Go",
+    "client_library_type": "generated",
+    "docs_url": "https://pkg.go.dev/cloud.google.com/go/pubsublite/apiv1",
+    "release_level": "beta"
+  },
   "cloud.google.com/go/recaptchaenterprise/apiv1": {
     "distribution_name": "cloud.google.com/go/recaptchaenterprise/apiv1",
     "description": "reCAPTCHA Enterprise API",

vendor/cloud.google.com/go/internal/version/version.go

@@ -26,7 +26,7 @@ import (
 
 // Repo is the current version of the client libraries in this
 // repo. It should be a date in YYYYMMDD format.
-const Repo = "20200727"
+const Repo = "20200817"
 
 // Go returns the Go runtime version. The returned string
 // has no whitespace.

vendor/cloud.google.com/go/storage/CHANGES.md

@@ -1,5 +1,9 @@
 # Changes
 
+## v1.11.0
+- Add support for CustomTime and NoncurrentTime object lifecycle management
+  features.
+
 ## v1.10.0
 - Bump dependency on google.golang.org/api to capture changes to retry logic
   which will make retries on writes more resilient.

vendor/cloud.google.com/go/storage/bucket.go

@@ -389,7 +389,8 @@ type RetentionPolicy struct {
 }
 
 const (
-	// RFC3339 date with only the date segment, used for CreatedBefore in LifecycleRule.
+	// RFC3339 timestamp with only the date segment, used for CreatedBefore,
+	// CustomTimeBefore, and NoncurrentTimeBefore in LifecycleRule.
 	rfc3339Date = "2006-01-02"
 
 	// DeleteAction is a lifecycle action that deletes a live and/or archived
@@ -455,6 +456,21 @@ type LifecycleCondition struct {
 	// the specified date in UTC.
 	CreatedBefore time.Time
 
+	// CustomTimeBefore is the CustomTime metadata field of the object. This
+	// condition is satisfied when an object's CustomTime timestamp is before
+	// midnight of the specified date in UTC.
+	//
+	// This condition can only be satisfied if CustomTime has been set.
+	CustomTimeBefore time.Time
+
+	// DaysSinceCustomTime is the days elapsed since the CustomTime date of the
+	// object. This condition can only be satisfied if CustomTime has been set.
+	DaysSinceCustomTime int64
+
+	// DaysSinceNoncurrentTime is the days elapsed since the noncurrent timestamp
+	// of the object. This condition is relevant only for versioned objects.
+	DaysSinceNoncurrentTime int64
+
 	// Liveness specifies the object's liveness. Relevant only for versioned objects
 	Liveness Liveness
 
@@ -464,6 +480,13 @@ type LifecycleCondition struct {
 	// Values include "STANDARD", "NEARLINE", "COLDLINE" and "ARCHIVE".
 	MatchesStorageClasses []string
 
+	// NoncurrentTimeBefore is the noncurrent timestamp of the object. This
+	// condition is satisfied when an object's noncurrent timestamp is before
+	// midnight of the specified date in UTC.
+	//
+	// This condition is relevant only for versioned objects.
+	NoncurrentTimeBefore time.Time
+
 	// NumNewerVersions is the condition matching objects with a number of newer versions.
 	//
 	// If the value is N, this condition is satisfied when there are at least N
@@ -946,9 +969,11 @@ func toRawLifecycle(l Lifecycle) *raw.BucketLifecycle {
 				StorageClass: r.Action.StorageClass,
 			},
 			Condition: &raw.BucketLifecycleRuleCondition{
-				Age:                 r.Condition.AgeInDays,
-				MatchesStorageClass: r.Condition.MatchesStorageClasses,
-				NumNewerVersions:    r.Condition.NumNewerVersions,
+				Age:                     r.Condition.AgeInDays,
+				DaysSinceCustomTime:     r.Condition.DaysSinceCustomTime,
+				DaysSinceNoncurrentTime: r.Condition.DaysSinceNoncurrentTime,
+				MatchesStorageClass:     r.Condition.MatchesStorageClasses,
+				NumNewerVersions:        r.Condition.NumNewerVersions,
 			},
 		}
 
@@ -964,6 +989,12 @@ func toRawLifecycle(l Lifecycle) *raw.BucketLifecycle {
 		if !r.Condition.CreatedBefore.IsZero() {
 			rr.Condition.CreatedBefore = r.Condition.CreatedBefore.Format(rfc3339Date)
 		}
+		if !r.Condition.CustomTimeBefore.IsZero() {
+			rr.Condition.CustomTimeBefore = r.Condition.CustomTimeBefore.Format(rfc3339Date)
+		}
+		if !r.Condition.NoncurrentTimeBefore.IsZero() {
+			rr.Condition.NoncurrentTimeBefore = r.Condition.NoncurrentTimeBefore.Format(rfc3339Date)
+		}
 		rl.Rule = append(rl.Rule, rr)
 	}
 	return &rl
@@ -981,9 +1012,11 @@ func toLifecycle(rl *raw.BucketLifecycle) Lifecycle {
 				StorageClass: rr.Action.StorageClass,
 			},
 			Condition: LifecycleCondition{
-				AgeInDays:             rr.Condition.Age,
-				MatchesStorageClasses: rr.Condition.MatchesStorageClass,
-				NumNewerVersions:      rr.Condition.NumNewerVersions,
+				AgeInDays:               rr.Condition.Age,
+				DaysSinceCustomTime:     rr.Condition.DaysSinceCustomTime,
+				DaysSinceNoncurrentTime: rr.Condition.DaysSinceNoncurrentTime,
+				MatchesStorageClasses:   rr.Condition.MatchesStorageClass,
+				NumNewerVersions:        rr.Condition.NumNewerVersions,
 			},
 		}
 
@@ -998,6 +1031,12 @@ func toLifecycle(rl *raw.BucketLifecycle) Lifecycle {
 		if rr.Condition.CreatedBefore != "" {
 			r.Condition.CreatedBefore, _ = time.Parse(rfc3339Date, rr.Condition.CreatedBefore)
 		}
+		if rr.Condition.CustomTimeBefore != "" {
+			r.Condition.CustomTimeBefore, _ = time.Parse(rfc3339Date, rr.Condition.CustomTimeBefore)
+		}
+		if rr.Condition.NoncurrentTimeBefore != "" {
+			r.Condition.NoncurrentTimeBefore, _ = time.Parse(rfc3339Date, rr.Condition.NoncurrentTimeBefore)
+		}
 		l.Rules = append(l.Rules, r)
 	}
 	return l
@@ -1151,6 +1190,8 @@ func (it *ObjectIterator) fetch(pageSize int, pageToken string) (string, error)
 	req.Projection("full")
 	req.Delimiter(it.query.Delimiter)
 	req.Prefix(it.query.Prefix)
+	req.StartOffset(it.query.StartOffset)
+	req.EndOffset(it.query.EndOffset)
 	req.Versions(it.query.Versions)
 	if len(it.query.fieldSelection) > 0 {
 		req.Fields("nextPageToken", googleapi.Field(it.query.fieldSelection))

vendor/cloud.google.com/go/storage/doc.go

@@ -39,7 +39,9 @@ To start working with this package, create a client:
         // TODO: Handle error.
     }
 
-The client will use your default application credentials.
+The client will use your default application credentials. Clients should be
+reused instead of created as needed. The methods of Client are safe for
+concurrent use by multiple goroutines.
 
 If you only wish to access public data, you can create
 an unauthenticated client with

vendor/cloud.google.com/go/storage/go.mod

@@ -3,16 +3,13 @@ module cloud.google.com/go/storage
 go 1.11
 
 require (
-	cloud.google.com/go v0.57.0
-	cloud.google.com/go/bigquery v1.8.0 // indirect
+	cloud.google.com/go v0.64.0
 	github.com/golang/protobuf v1.4.2
-	github.com/google/go-cmp v0.4.1
+	github.com/google/go-cmp v0.5.1
 	github.com/googleapis/gax-go/v2 v2.0.5
-	golang.org/x/net v0.0.0-20200520182314-0ba52f642ac2 // indirect
 	golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d
-	golang.org/x/sys v0.0.0-20200523222454-059865788121 // indirect
-	golang.org/x/tools v0.0.0-20200618134242-20370b0cb4b2 // indirect
-	google.golang.org/api v0.28.0
-	google.golang.org/genproto v0.0.0-20200618031413-b414f8b61790
-	google.golang.org/grpc v1.29.1
+	golang.org/x/tools v0.0.0-20200827163409-021d7c6f1ec3 // indirect
+	google.golang.org/api v0.30.0
+	google.golang.org/genproto v0.0.0-20200827165113-ac2560b5e952
+	google.golang.org/grpc v1.31.0
 )