Compare commits

..

76 Commits

Author SHA1 Message Date
Arkadii Yakovets
9b10e6d26b Implement SBOM generation
Signed-off-by: Arkadii Yakovets <ark@victoriametrics.com>
2024-07-16 17:28:24 -07:00
Arkadii Yakovets
042444f3ea Merge branch 'master' into ark/add-vm-release-tests 2024-07-16 17:23:36 -07:00
Aliaksandr Valialkin
9cf187c83b docs: remove the remaining mentioning about snap packages after the commit 0a42c8fd8b
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6543
2024-07-16 18:35:38 +02:00
Aliaksandr Valialkin
c1e32f4517 lib/promrelabel: add test for IfExpression.String() function
While at it, simplify this function a bit after the commit 861852f262

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6462
2024-07-16 18:31:05 +02:00
Zakhar Bessarab
63dfeec0de app/vmagent/kafka: fix non-unique metric naming (#774)
* app/vmagent/kafka: fix non-unique metric naming

Fix panic when using multiple topics with the same name.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6636
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs/changelog: document bugfix

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs/vmagent: more examples for Kafka ingestion with multiple brokers/topics groups

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-07-16 18:03:24 +02:00
Aliaksandr Valialkin
4304950391 lib/promscrape/discovery/yandexcloud: follow-up for 070abe5c71
- Obtain IAM token via GCE-like API instead of Amazon EC2 IMDSv2 API,
  since it looks like IMDBSv2 API isn't supported by Yandex Cloud
  according to https://yandex.cloud/en/docs/security/standard/authentication#aws-token :

  > So far, Yandex Cloud does not support version 2, so it is strongly recommended
  > to technically disable getting a service account token via the Amazon EC2 metadata service.

- Try obtaining IAM token via GCE-like API at first and then fall back to the deprecated Amazon EC2 IMDBSv1.
  This should prevent from auth errors for instances with disabled GCE-like auth API.
  This addresses @ITD27M01 concern at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5513#issuecomment-1867794884

- Make more clear the description of the change at docs/CHANGELOG.md , add reference to the related issue.

P.S. This change wasn't tested in prod because I have no access to Yandex Cloud.
It is recommended to test this change by @ITD27M01 and @vmazgo , who filed
the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5513

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6524
2024-07-16 17:58:40 +02:00
f41gh7
6aed628f04 docs: mention vmagent and vmgateway changes
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-07-16 17:44:16 +02:00
Alexander Marshalov
033903c208 docs: fixed image paths in the guide "Anomaly Detection and Alerting Setup" (#6651) 2024-07-16 07:00:31 -07:00
Aliaksandr Valialkin
851998ed96 docs/CHANGELOG.md: clarify docs and changelog after e666d64f1d
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6453
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6525
2024-07-16 14:01:35 +02:00
Aliaksandr Valialkin
3224a12e25 Revert "deployment: build image for vmagent streamaggr benchmark (#6515)"
This reverts commit 6b128da811.

Reason for revert: this complicates and slows down CI/CD without giving significant benefits in return.

The idea of automatic building, publishing and deploying Docker images to our playground on every pull request
and commit isn't very bright because of the following reasons:

- It slows down CI/CD pipeline
- It increases costs on CPU time spent at CI/CD pipeline
- It contradicts goal #7 at https://docs.victoriametrics.com/goals/#goals and non-goal #8 at https://docs.victoriametrics.com/goals/#non-goals

The previous workflow was much better - if we need to deploy some new Docker image at playground or staging environment,
then just __manually__ build and deploy the needed Docker image there. If the manual process requires making too many
steps, then think on how to automate these steps into a single Makefile command.

Updates https://github.com/VictoriaMetrics/ops/pull/1297
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6515
2024-07-16 13:27:06 +02:00
Aliaksandr Valialkin
f5a89aea1e vendor: update github.com/VictoriaMetrics/metrics from v1.35.0 to v1.35.1 2024-07-16 13:18:38 +02:00
Aliaksandr Valialkin
57000f5105 lib/promscrape: follow-up for 1e83598be3
- Clarify that the -promscrape.maxScrapeSize value is used for limiting the maximum
  scrape size if max_scrape_size option isn't set at https://docs.victoriametrics.com/sd_configs/#scrape_configs

- Fix query example for scrape_response_size_bytes metric at https://docs.victoriametrics.com/vmagent/#automatically-generated-metrics

- Mention about max_scrape_size option at the -help description for -promscrape.maxScrapeSize command-line flag

- Treat zero value for max_scrape_size option as 'no scrape size limit'

- Change float64 to int type for scrapeResponseSize struct fields and function args, since response size cannot be fractional

- Optimize isAutoMetric() function a bit

- Sort auto metrics in alphabetical order in isAutoMetric() and in scrapeWork.addAutoMetrics() functions
  for better maintainability in the future

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6434
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6429
2024-07-16 12:38:21 +02:00
Github Actions
9c02f27ef9 Automatic update Grafana datasource docs from VictoriaMetrics/victorialogs-datasource@3cbd5e9 (#6649) 2024-07-16 02:05:36 -07:00
Aliaksandr Valialkin
1a52329b1b Revert "docs: [vmagent] Add CHANGELOG for Statsd support in v1.102.0-rc1 (#6494)"
This reverts commit b37b288dce.

Reason for revert: statsd protocol support has been reverted - see 2da7dfc754
2024-07-16 10:57:02 +02:00
Aliaksandr Valialkin
590aeccd7d app/vmauth: follow-up for 3a45bbb4e0
- Move the test for SRV discovery into a separate function. This allows verifying round-robin discovery across SRV records.
- Restore the original netutil.Resolver after the test finishes, so it doesn't interfere with other tests.
- Move the description of the bugfix into the correct place at docs/CHANGELOG.md - it should be placed under v1.102.0-rc2
  instead of v1.102.0-rc1.
- Remove unneeded code in URLPrefix.sanitizeAndInitialize(), since it is expected this function is called only once
  for finishing URLPrefix initializiation. In this case URLPrefix.nextDiscoveryDeadline and URLPrefix.n are equal to 0
  according to https://pkg.go.dev/sync/atomic#Uint64
- Properly fix the bug at URLPrefix.discoverBackendAddrsIfNeeded() - it is expected that hostToAddrs map uses
  the original hostname keys, including 'srv+' prefix, so it shouldn't be removed when looping over up.busOriginal.
  Instead, the 'srv+' prefix must be removed from the hostname only locally before passing the hostname to netutil.Resolver.LookupSRV.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6401
2024-07-16 10:40:51 +02:00
Artem Navoiev
fe33626b0d docs/data-ingestion use img tag instead of markdown image. this allow docs that sync automation to correctly render images
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-07-16 10:14:36 +02:00
Aliaksandr Valialkin
7a3394bbe1 Revert "lib/protoparser/opentelemetry/firehose: escape requestID before returning it to user (#6451)"
This reverts commit cd1aca217c.

Reason for revert: this commit has no sense, since the firehose response has application/json content-type,
so it must contain JSON-encoded timestamp and requestId fields according to https://docs.aws.amazon.com/firehose/latest/dev/httpdeliveryrequestresponse.html#responseformat .
HTML-escaping the requestId field may break the response, so the client couldn't correctly recognize the html-escaped requestId.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6451
2024-07-16 09:49:19 +02:00
Aliaksandr Valialkin
88e02b6352 app/vmauth: clarify the description for -idleConnTimeout command-line flag
This is a follow-up for d44058bcd6
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6388
2024-07-16 09:39:15 +02:00
Aliaksandr Valialkin
233e5f0a9e lib/httpserver: skip basic auth check for additional request paths, which should call httpserver.CheckAuthFlag()
This is a follow-up for 61dce6f2a1

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6338
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6329
2024-07-16 01:00:45 +02:00
Aliaksandr Valialkin
784327ea30 lib/uint64set: optimize Set.Has() for nil Set - it should be inlined now
This makes unnecessary the checkDeleted variable at lib/storage/index_db.go

This is a follow-up for b984f4672e
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6342
2024-07-15 23:59:20 +02:00
Aliaksandr Valialkin
832e088659 lib/mergeset: properly update TableMetrics.TooLongItemsDroppedTotal inside Table.UpdateMetrics
Substitute '+=' with '=', since tooLongItemsTotal is global counter, which doesn't belong to the Table struct.

This is a follow-up for 69d244e6fb
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6297
2024-07-15 23:39:10 +02:00
Aliaksandr Valialkin
e3d5714f6f app/vminsert: increase default value for -maxLabelValueLen command-line flag from 1KiB to 4KiB
It has been appeared that the standard Kubernetes monitoring can generate labels with sizes up to 4KiB

This is a follow-up for a5d1013042
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6176
2024-07-15 23:32:36 +02:00
Aliaksandr Valialkin
a468a6e985 lib/{httputils,netutil}: move httputils.GetStatDialFunc to netutil.NewStatDialFunc
- Rename GetStatDialFunc to NewStatDialFunc, since it returns new function with every call
- NewStatDialFunc isn't related to http in any way, so it must be moved from lib/httputils to lib/netutil
- Simplify the implementation of NewStatDialFunc by removing sync.Map from there.
- Use netutil.NewStatDialFunc at app/vmauth and lib/promscrape/discoveryutils
- Use gauge instead of counter type for *_conns metric

This is a follow-up for d7b5062917
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6299
2024-07-15 23:02:34 +02:00
Aliaksandr Valialkin
ad367c17bf lib/streamaggr/streamaggr.go: typo fix after 5e29ef5ed5: IgnoredNaNSamples -> ignoredNaNSamples 2024-07-15 21:58:56 +02:00
Aliaksandr Valialkin
5e29ef5ed5 docs/CHANGELOG.md: typo fix: vl_streamaggr -> vm_streamaggr
Thanks to @AndrewChubatiuk for the comment at db557b86ee (r144259373)
2024-07-15 21:49:55 +02:00
Aliaksandr Valialkin
fc637555cd docs/stream-aggregation.md: clarify "Routing" chapter a bit after f153f54d11 2024-07-15 21:42:48 +02:00
Aliaksandr Valialkin
db557b86ee app/vmagent/remotewrite: follow-up for f153f54d11
- Move the remaining code responsible for stream aggregation initialization from remotewrite.go to streamaggr.go .
  This improves code maintainability a bit.

- Properly shut down streamaggr.Aggregators initialized inside remotewrite.CheckStreamAggrConfigs().
  This prevents from potential resource leaks.

- Use separate functions for initializing and reloading of global stream aggregation and per-remoteWrite.url stream aggregation.
  This makes the code easier to read and maintain. This also fixes INFO and ERROR logs emitted by these functions.

- Add an ability to specify `name` option in every stream aggregation config. This option is used as `name` label
  in metrics exposed by stream aggregation at /metrics page. This simplifies investigation of the exposed metrics.

- Add `path` label additionally to `name`, `url` and `position` labels at metrics exposed by streaming aggregation.
  This label should simplify investigation of the exposed metrics.

- Remove `match` and `group` labels from metrics exposed by streaming aggregation, since they have little practical applicability:
  it is hard to use these labels in query filters and aggregation functions.

- Rename the metric `vm_streamaggr_flushed_samples_total` to less misleading `vm_streamaggr_output_samples_total` .
  This metric shows the number of samples generated by the corresponding streaming aggregation rule.
  This metric has been added in the commit 861852f262 .
  See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6462

- Remove the metric `vm_streamaggr_stale_samples_total`, since it is unclear how it can be used in practice.
  This metric has been added in the commit 861852f262 .
  See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6462

- Remove Alias and aggrID fields from streamaggr.Options struct, since these fields aren't related to optional params,
  which could modify the behaviour of the constructed streaming aggregator.
  Convert the Alias field to regular argument passed to LoadFromFile() function, since this argument is mandatory.

- Pass Options arg to LoadFromFile() function by reference, since this structure is quite big.
  This also allows passing nil instead of Options when default options are enough.

- Add `name`, `path`, `url` and `position` labels to `vm_streamaggr_dedup_state_size_bytes` and `vm_streamaggr_dedup_state_items_count` metrics,
  so they have consistent set of labels comparing to the rest of streaming aggregation metrics.

- Convert aggregator.aggrStates field type from `map[string]aggrState` to `[]aggrOutput`, where `aggrOutput` contains the corresponding
  `aggrState` plus all the related metrics (currently only `vm_streamaggr_output_samples_total` metric is exposed with the corresponding
  `output` label per each configured output function). This simplifies and speeds up the code responsible for updating per-output
  metrics. This is a follow-up for the commit 2eb1bc4f81 .
  See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6604

- Added missing urls to docs ( https://docs.victoriametrics.com/stream-aggregation/ ) in error messages. These urls help users
  figuring out why VictoriaMetrics or vmagent generates the corresponding error messages. The urls were removed for unknown reason
  in the commit 2eb1bc4f81 .

- Fix incorrect update for `vm_streamaggr_output_samples_total` metric in flushCtx.appendSeriesWithExtraLabel() function.
  While at it, reduce memory usage by limiting the maximum number of samples per flush to 10K.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5467
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6268
2024-07-15 20:24:01 +02:00
Artem Navoiev
7ba477e08a remove coc
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-07-15 19:48:57 +02:00
Mathias Palmersheim
27d03c1c98 Added docs for telegraf proxmox and vector (#6619)
### Describe Your Changes

initial docs to implement #6618 more platforms can be added on this
branch or on future commits.

### Checklist

The following checks are **mandatory**:

- [x ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-15 11:08:11 -05:00
Fred Navruzov
2fa03306c3 docs/vmanomaly - v1.13.2 updates (#6646)
### Describe Your Changes

Doc updates after v1.13.2 release of `vmanomaly`

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-15 17:05:33 +03:00
Artem Navoiev
8af9a3f0b8 docs/VictoriaLogs link Github Issues with Roadmap items. Add informat… (#6633)
…ion about Grafana Datasource in quering

### Describe Your Changes

Update Roadmap and Querying documentation for VictoriaLogs 

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-07-15 15:29:44 +02:00
Aliaksandr Valialkin
202e5704e6 vendor: update github.com/VictoriaMetrics/metrics from v1.34.1 to v1.35.0
Fix potential memory leaks across VictoriaMetrics codebase after metrics.UnregisterSet(s) call
because of missing s.UnregisterAllMetrics() call.

This is a follow-up for 6a6e34ab8e . It is OK if some vmauth metrics
aren't visible for a few microseconds when the previous metrics are unregistered and new metrics
weren't registered yet.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6247
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4690
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6252
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5805
2024-07-15 10:43:37 +02:00
Aliaksandr Valialkin
c995ccad93 lib/{storage,mergeset}: do not allow setting dataFlushInterval to values smaller than pending{Items,Rows}FlushInterval
Pending rows and items unconditionally remain in memory for up to pending{Items,Rows}FlushInterval,
so there is no any sense in setting dataFlushInterval (the interval for guaranteed flush of in-memory data to disk)
to values smaller than pending{Items,Rows}FlushInterval, since this doesn't affect the interval
for flushing pending rows and items from memory to disk.

This is a follow-up for 4c80b17027

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6221
2024-07-15 10:08:15 +02:00
Aliaksandr Valialkin
48ec66883a lib/streamaggr: consistently use alphabetical order of benchmarked stream aggregation outputs 2024-07-15 09:53:19 +02:00
Aliaksandr Valialkin
f3ccbe181d app/vmagent/remotewrite: do not spend CPU time on an attempt to send data to blocked queue if some queues are unblocked
Previously remotewrite.TryPush() was trying to send data to remote storages with blocked persistent queues,
if some persistent queues to other remote storage systems were unblocked. This resulted in excess CPU usage
on relabeling and stream aggregation for the remote storage with blocked queues.

The solution is to check whether some peristent storages have blocked queues and skip them before applying
per- -remoteWrite.url relabeling and streaming aggregation.

While at it, properly update per- -remoteWrite.url vmagent_remotewrite_samples_dropped_total and vmagent_remotewrite_push_failures_total
counters when global streaming aggregation cannot send data to remote storage systems because of blocked queues.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5467 and https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6268 .

This is a follow-up for 87fd400dfc and f153f54d11

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6248
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6065
2024-07-15 09:38:17 +02:00
Aliaksandr Valialkin
5354374b62 lib/streamaggr: follow-up for 9c3d44c8c9
- Consistently enumerate stream aggregation outputs in alphabetical order across the source code and docs.
  This should simplify future maintenance of the corresponding code and docs.

- Fix the link to `rate_sum()` at `see also` section of `rate_avg()` docs.

- Make more clear the docs for `rate_sum()` and `rate_avg()` outputs.

- Encapsulate output metric suffix inside rateAggrState. This eliminates possible bugs related
  to incorrect suffix passing to newRateAggrState().

- Rename rateAggrState.total field to less misleading rateAggrState.increase name, since it calculates
  counter increase in the current aggregation window.

- Set rateLastValueState.prevTimestamp on the first sample in time series instead of the second sample.
  This makes more clear the code logic.

- Move the code for removing outdated entries at rateAggrState into removeOldEntries() function.
  This make the code logic inside rateAggrState.flushState() more clear.

- Do not write output sample with zero value if there are no input series, which could be used
  for calculating the rate, e.g. if only a single sample is registered for every input series.

- Do not take into account input series with a single registered sample when calculating rate_avg(),
  since this leads to incorrect results.

- Move {rate,total}AggrState.flushState() function to the end of rate.go and total.go files, so they look more similar.
  This shuld simplify future mantenance.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6243
2024-07-15 08:40:09 +02:00
Aliaksandr Valialkin
cfc72cb129 docs/CHANGELOG.md: use new link to VictoriaMetrics cluster docs instead of old link
The old link was changed globally to the new link in the commit f4b1cbfef0 .
Unfortunately, old links are still posted in new commits :(

This is a follow-up for 680b8c25c8 .

While at it, remove duplicate 'len(*remoteWriteURLs) > 0' check in the remotewrite.Init() functions,
since this check is already made at the beginning of the function.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6253
2024-07-13 03:02:40 +02:00
Aliaksandr Valialkin
0a74d8f654 docs/CHANGELOG.md: consistently use new url to vmagent docs - https://docs.victoriametrics.com/vmagent/ - instead of old one - https://docs.victoriametrics.com/vmagent.html
See the previous commit, which was making the same thing a few months ago - c81a633b02

Unfortunately, new commits continue using old links :(
2024-07-13 02:38:38 +02:00
Aliaksandr Valialkin
0145b65f25 app/vmagent/remotewrite: follow-up for 87fd400dfc
- Drop samples and return true from remotewrite.TryPush() at fast path when all the remote storage
  systems are configured with the disabled on-disk queue, every in-memory queue is full
  and -remoteWrite.dropSamplesOnOverload is set to true. This case is quite common,
  so it should be optimized. Previously additional CPU time was spent on per-remoteWriteCtx
  relabeling and other processing in this case.

- Properly count the number of dropped samples inside remoteWriteCtx.pushInternalTrackDropped().
  Previously dropped samples were counted only if -remoteWrite.dropSamplesOnOverload flag is set.
  In reality, the samples are dropped when they couldn't be sent to the queue because in-memory queue is full
  and on-disk queue is disabled.
  The remoteWriteCtx.pushInternalTrackDropped() function is called by streaming aggregation for pushing
  the aggregated data to the remote storage. Streaming aggregation cannot wait until the remote storage
  processes pending data, so it drops aggregated samples in this case.

- Clarify the description for -remoteWrite.disableOnDiskQueue command-line flag at -help output,
  so it is clear that this flag can be set individually per each -remoteWrite.url.

- Make the -remoteWrite.dropSamplesOnOverload flag global. If some of the remote storage systems
  are configured with the disabled on-disk queue, then there is no sense in keeping samples
  on some of these systems, while dropping samples on the remaining systems, since this
  will result in global stall on the remote storage system with the disabled on-disk queue
  and with the -remoteWrite.dropSamplesOnOverload=false flag. vmagent will always return false
  from remotewrite.TryPush() in this case. This will result in infinite duplicate samples
  written to the remaining remote storage systems. That's why the -remoteWrite.dropSamplesOnOverload
  is forcibly set to true if more than one -remoteWrite.disableOnDiskQueue flag is set.
  This allows proceeding with newly scraped / pushed samples by sending them to the remaining
  remote storage systems, while dropping them on overloaded systems with the -remoteWrite.disableOnDiskQueue flag set.

- Verify that the remoteWriteCtx.TryPush() returns true in the TestRemoteWriteContext_TryPush_ImmutableTimeseries test.

- Mention in vmagent docs that the -remoteWrite.disableOnDiskQueue command-line flag can be set individually per each -remoteWrite.url.
  See https://docs.victoriametrics.com/vmagent/#disabling-on-disk-persistence

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6248
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6065
2024-07-13 02:25:19 +02:00
Aliaksandr Valialkin
a8472d033a app/vmalert-tool/Makefile: add make vmalert-tool-linux-loong64 build rule
This is a follow-up for 80f3644ee3

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6222
2024-07-12 23:19:04 +02:00
Aliaksandr Valialkin
3d6fa7f70b app/victoria-logs/Makefile: add make victoria-logs-linux-loong64 build rule
This is a follow-up for 80f3644ee3

The https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6222 missed build rule for VictoriaLogs.
2024-07-12 23:12:48 +02:00
Aliaksandr Valialkin
0078399788 app/vmalert: switch from table-driven tests to f-tests
This makes test code more clear and reduces the number of code lines by 500.
This also simplifies debugging tests. See https://itnext.io/f-tests-as-a-replacement-for-table-driven-tests-in-go-8814a8b19e9e

While at it, consistently use t.Fatal* instead of t.Error* across tests, since t.Error*
requires more boilerplate code, which can result in additional bugs inside tests.
While t.Error* allows writing logging errors for the same, this doesn't simplify fixing
broken tests most of the time.

This is a follow-up for a9525da8a4
2024-07-12 22:41:11 +02:00
Aliaksandr Valialkin
cedbbdec30 app/vmctl: switch from table-driven tests to f-tests
This simplifies debugging tests and makes the test code more clear and concise.
See https://itnext.io/f-tests-as-a-replacement-for-table-driven-tests-in-go-8814a8b19e9e

While at is, consistently use t.Fatal* instead of t.Error* across tests, since t.Error*
requires more boilerplate code, which can result in additional bugs inside tests.
While t.Error* allows writing logging errors for the same, this doesn't simplify fixing
broken tests most of the time.

This is a follow-up for a9525da8a4
2024-07-12 22:39:45 +02:00
Arkadii Yakovets
21beb994ad Add file contents checks
Signed-off-by: Arkadii Yakovets <ark@victoriametrics.com>
2024-07-12 11:50:48 -07:00
Arkadii Yakovets
4605a72684 Fix lint error
Signed-off-by: Arkadii Yakovets <ark@victoriametrics.com>
2024-07-12 11:50:48 -07:00
Arkadii Yakovets
08de878047 Add release-victoria-metrics smoke tests
Signed-off-by: Arkadii Yakovets <ark@victoriametrics.com>
2024-07-12 11:50:48 -07:00
Alexander Marshalov
e84309142a vmcloud: fixed images in cloud alertmanager docs (#6641)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-12 16:36:51 +02:00
Alexander Marshalov
ab035740ee vmcloud: fixed image paths for cloud alertmanager docs (#6640)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-12 16:30:17 +02:00
Alexander Marshalov
d5bdf2bc91 vmcloud: updated docs about using alertmanager (#6638)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-12 16:21:26 +02:00
hagen1778
fbdaba4f8c docs: mention stream aggregation in churn rate section
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-07-12 11:14:27 +02:00
hagen1778
f2c1d30011 docs: mention assets naming policy and placement
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-07-12 11:12:24 +02:00
hagen1778
2f65956259 lib/streamaggr: add missing test cases
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-07-12 11:06:45 +02:00
Hui Wang
2eb1bc4f81 vmagent: fix vm_streamaggr_flushed_samples_total counter (#6604)
We use `vm_streamaggr_flushed_samples_total` to show the number of
produced samples by aggregation rule, previously it was overcounted, and
doesn't account for `output_relabel_configs`.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6462

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-07-12 10:56:07 +02:00
Aliaksandr Valialkin
62dabd67a2 app: consistently use t.Fatal* instead of t.Error* (except of app/vmalert and app/vmctl - these packages will be processed in a separate commit)
Consistently using t.Fatal* simplifies the test code and makes it less fragile, since it is common error
to forget to make proper cleanup after t.Error* call. Also t.Error* calls do not provide any practical
benefits when some tests fail. They just clutter test output with additional noise information,
which do not help in fixing failing tests most of the time.

While at it, improve errors generated at app/victoria-metrics tests, so they contain more useful information
when debugging failed tests.

This is a follow-up for a9525da8a4
2024-07-11 15:59:08 +02:00
Arkadii Yakovets
fabf0b928e docs: add spellcheck command (#6562)
### Describe Your Changes

Implement spellcheck command:
  - add cspell configuration files
  - dockerize spellchecking process
  - add Makefile targets
 
This PR adds a standalone `make spellcheck` target to check `docs/*.md` files for spelling
errors. The target process is dockerized to be run in a separate npm environment.

Some `docs/` typo fixes also included.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Arkadii Yakovets <ark@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-07-11 12:39:42 +02:00
Zhu Jiekun
cadf1eb5ab vmalert: [bug] fixed System hyperlink 404 redirect (#6620)
### Describe Your Changes

As mentioned in https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6603, some hyperlinks under `vmalert` -> `System`
section is not working as expected.

Pages and redirection:
- For page `http://127.0.0.1:8880/`: `flags` button will redirect to
`http://127.0.0.1:8880/flags`
- For page `http://127.0.0.1:8880/vmalert`:
`http://127.0.0.1:8880/flags`
- For page `http://127.0.0.1:8880/vmalert/`:
`http://127.0.0.1:8880/vmalert/flags` (page not exists)
- Similar redirection could be observed with `-http.pathPrefix`

Two potential ways to avoid 404 redirection:
1. **avoid visiting `/vmalert/`** (I'm trying to do this).
2. provide support for `/vmalert/flags`.

`/vmalert/` could be visit only when user click other navigator (e.g.
Group) and click vmalert again:
![Peek 2024-07-10
10-07](https://github.com/VictoriaMetrics/VictoriaMetrics/assets/30280396/13d7b147-a1b6-4e93-9ee0-26f881a16bef)
Because: `http://127.0.0.1:8880/vmalert/groups?search=` + `<a
class="nav-link" href=".">` = `http://127.0.0.1:8880/vmalert/`

So I'm trying to change the `href="."` to `href="../vmalert"`.

### Checklist

The following checks are **mandatory**:

- [X] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-11 11:43:00 +02:00
Artem Navoiev
5b43675af6 docs: quick use official names for victoriametrics components (#6626)
### Describe Your Changes

Replace VM- with VictoriaMetrics in QuickStart 
Keep the previous anchors for backward compatibility 

### Checklist

The following checks are **mandatory**:

- [ x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-07-11 09:35:48 +02:00
Nikolay
14ac8a09ea make vendor-update (#6627)
make vendor-update
2024-07-10 17:14:51 +02:00
Github Actions
c688b53f56 Automatic update operator docs from VictoriaMetrics/operator@4774786 (#6617)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-07-10 16:04:30 +02:00
Zakhar Bessarab
6a4bd5049b app/vmselect/promql: propagate lower bucket values when fixing a histogram (#6547)
### Describe Your Changes

In most cases histograms are exposed in sorted manner with lower buckets
being first. This means that during scraping buckets with lower bounds
have higher chance of being updated earlier than upper ones.

Previously, values were propagated from upper to lower bounds, which
means that in most cases that would produce results higher than expected
once all buckets will become updated.
Propagating from upper bound effectively limits highest value of
histogram to the value of previous scrape. Once the data will become
consistent in the subsequent evaluation this causes spikes in the
result.

Changing propagation to be from lower to higher buckets reduces value
spikes in most cases due to nature of the original inconsistency.

 See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4580

An example histogram with previous(red) and updated(blue) versions:

![1719565540](https://github.com/VictoriaMetrics/VictoriaMetrics/assets/1367798/605c5e60-6abe-45b5-89b2-d470b60127b8)

This also makes logic of filling nan values with lower buckets values: [1 2 3 nan nan nan] => [1 2 3 3 3 3] obsolete.
Since buckets are now fixed from lower ones to upper this happens in the main loop, so there is no need in a second one.

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Andrii Chubatiuk <andrew.chubatiuk@gmail.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-07-10 15:15:29 +02:00
hagen1778
03e4c5c19c lib/bakcup/azremote: follow-up after 5fd3aef549
Simplify tests by converting them to f-tests.

5fd3aef549
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-07-10 13:06:27 +02:00
hagen1778
1c155a0cd1 docs: follow-up after c341369fc1
* account for `source` tag in `docs-sync` command
* run `make docs-sync`

c341369fc1
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-07-10 12:25:21 +02:00
hagen1778
b29154a497 docs/vmrestore: link to auth examples in vmbackup
vmbackup, vmrestore and vmbackupmanager use the same libs
for integrations with object storage. That means the auth can be configured
in the same way for all of them. So the docs should have either identical
config section for all 3 components, or we should cross-link to one source of truth.

This change removes incomplete auth options from vmrestore docs and adds link
to complete auth options in vmbackup instead.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-07-10 12:20:53 +02:00
justinrush
5fd3aef549 lib/backup: add support for Azure Managed Identity (#6518)
### Describe Your Changes

These changes support using Azure Managed Identity for the `vmbackup`
utility. It adds two new environment variables:

* `AZURE_USE_DEFAULT_CREDENTIAL`: Instructs the `vmbackup` utility to
build a connection using the [Azure Default
Credential](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity@v1.5.2#NewDefaultAzureCredential)
mode. This causes the Azure SDK to check for a variety of environment
variables to try and make a connection. By default, it tries to use
managed identity if that is set up.

This will close
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5984

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

### Testing

However you normally test the `vmbackup` utility using Azure Blob should
continue to work without any changes. The set up for that is environment
specific and not listed out here.

Once regression testing has been done you can set up [Azure Managed
Identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview)
so your resource (AKS, VM, etc), can use that credential method. Once it
is set up, update your environment variables according to the updated
documentation.

I added unit tests to the `FS.Init` function, then made my changes, then
updated the unit tests to capture the new branches.

I tested this in our environment, but with SAS token auth and managed
identity and it works as expected.

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Justin Rush <jarush@epic.com>
Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-07-10 11:52:05 +02:00
Aliaksandr Valialkin
b173f24041 docs/VictoriaLogs/CHANGELOG.md: cut v0.28.0-victorialogs release 2024-07-10 03:08:16 +02:00
Aliaksandr Valialkin
ac06569c49 app/vlinsert/loki: use easyproto instead for parsing Loki protobuf messages 2024-07-10 03:05:17 +02:00
Aliaksandr Valialkin
00c666a6c3 app/vlselect/vmui: run make vmui-logs-update after 662e026279 2024-07-10 00:50:10 +02:00
Aliaksandr Valialkin
aa9bb99527 lib/logstorage: drop all the pipes from the query when calculating the number of matching logs at /select/logsql/hits API 2024-07-10 00:39:28 +02:00
Aliaksandr Valialkin
3c02937a34 all: consistently use 'any' instead of 'interface{}'
'any' type is supported starting from Go1.18. Let's consistently use it
instead of 'interface{}' type across the code base, since `any` is easier to read than 'interface{}'.
2024-07-10 00:20:37 +02:00
Aliaksandr Valialkin
08c32232a6 app/vlinsert/loki: remove unused functions from the generated protobuf code 2024-07-10 00:18:48 +02:00
Aliaksandr Valialkin
a9525da8a4 lib: consistently use f-tests instead of table-driven tests
This makes easier to read and debug these tests. This also reduces test lines count by 15% from 3K to 2.5K
See https://itnext.io/f-tests-as-a-replacement-for-table-driven-tests-in-go-8814a8b19e9e

While at it, consistently use t.Fatal* instead of t.Error*, since t.Error* usually leads
to more complicated and fragile tests, while it doesn't bring any practical benefits over t.Fatal*.
2024-07-09 22:40:50 +02:00
Yury Molodov
662e026279 vmui/logs: add spinner to bar chart (#6577)
Add a spinner to the bar chart 

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6558

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-07-09 14:58:48 +02:00
Hui Wang
8e9f98e725 security: upgrade base docker image (Alpine) from 3.20.0 to 3.20.1
See https://www.alpinelinux.org/posts/Alpine-3.20.1-released.html

>including security fixes for:
OPENSSL
[CVE-2024-4741](https://security.alpinelinux.org/vuln/CVE-2024-4741)
BUSYBOX
[CVE-2023-42364](https://security.alpinelinux.org/vuln/CVE-2023-42364)
[CVE-2023-42365](https://security.alpinelinux.org/vuln/CVE-2023-42365)
2024-07-09 11:38:05 +02:00
Phuong Le
c341369fc1 docs: use white version of logo for dark theme (#6610)
The logo on the README is hard to recognize in dark mode on GitHub, so I
made it responsive based on user preferences.
2024-07-09 11:04:00 +02:00
hagen1778
6e17255ec0 docs: follow-up 0e1dbdee28
* restore old anchor names to keep links compatibility.
See https://docs.victoriametrics.com/#documentation requirements
* consistently use the same format for commands `sh` as it makes it better
renderred and automatically adds `copy` button to fileds with commands
* simplify the text by removing extra points in the list
* add recommendations for installing the cluster setup
* explicitly mention the ports services are listening on
* add description for `storageNode` cmd-line flag to inform the reader what
values need to be put into it
* fix the incorrect vmui link in cluster installation recommendation
* rename component anchors to be more unique, because URL doesn't respect
hierarchy for the anchored links and may result into conflicts in future

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-07-09 10:43:26 +02:00
Artem Navoiev
878c727a3a add alt to cluster logo, this allows to hide it on our docs site
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-07-09 10:26:55 +02:00
380 changed files with 21675 additions and 11465 deletions

View File

@@ -1,56 +0,0 @@
name: benchmark
on:
push:
branches:
- master
- cluster
paths-ignore:
- "docs/**"
- "**.md"
- "dashboards/**"
- "deployment/**.yml"
pull_request:
types:
- opened
- synchronize
- reopened
- labeled
branches:
- master
- cluster
paths-ignore:
- "docs/**"
- "**.md"
- "dashboards/**"
- "deployment/**.yml"
permissions:
contents: read
packages: write
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
build-streamaggr-benchmark-image:
name: build
runs-on: ubuntu-latest
if: contains(github.event.pull_request.labels.*.name, 'streamaggr-benchmark')
steps:
- name: Code checkout
uses: actions/checkout@v4
- name: Login to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Package VMAgent Docker image for benchmark
run: |
SKIP_SCRATCH_BUILD=true \
DOCKER_BUILD_OPTS='--cache-to type=gha,mode=max --cache-from type=gha' \
PKG_TAG=${{ github.event.pull_request.head.sha }} \
DOCKER_REGISTRY=ghcr.io \
TARGET_PLATFORM=linux/amd64 make publish-vmagent

View File

@@ -6,14 +6,12 @@ on:
- cluster
- master
paths:
- '.github/workflows/main.yml'
- '**.go'
pull_request:
branches:
- cluster
- master
paths:
- '.github/workflows/main.yml'
- '**.go'
permissions:
@@ -48,25 +46,10 @@ jobs:
key: go-artifacts-${{ runner.os }}-check-all-${{ steps.go.outputs.go-version }}-${{ hashFiles('go.sum', 'Makefile', 'app/**/Makefile') }}
restore-keys: go-artifacts-${{ runner.os }}-check-all-
- name: Run fmt
run: make fmt
- name: Run vet
run: make vet
# Use action instead of `make golangci-lint` to speed up the process
# as it caches data between builds.
- name: Run golangci-lint
uses: golangci/golangci-lint-action@v6
with:
# The version must match "install-golangci-lint" Makefile target version.
version: v1.59.1
- name: Run govulncheck
run: make govulncheck
- name: Check diff
run: git diff --exit-code
- name: Run check-all
run: |
make check-all
git diff --exit-code
test:
name: test

59
.github/workflows/test_package.yml vendored Normal file
View File

@@ -0,0 +1,59 @@
name: Package tests
on:
push:
branches:
- cluster
- master
paths:
- '.github/workflows/test_package.yml'
- '**/Dockerfile*'
- '**/Makefile'
pull_request:
branches:
- cluster
- master
paths:
- '.github/workflows/test_package.yml'
- '**/Dockerfile*'
- '**/Makefile'
permissions:
contents: read
concurrency:
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
jobs:
test_release:
name: Test Release
runs-on: ubuntu-latest
steps:
- name: Code checkout
uses: actions/checkout@v4
- name: Setup Go
id: go
uses: actions/setup-go@v5
with:
go-version: stable
cache: false
- name: Cache Go artifacts
uses: actions/cache@v4
with:
path: |
~/.cache/go-build
~/go/bin
~/go/pkg/mod
key: go-artifacts-${{ runner.os }}-test-release-${{ steps.go.outputs.go-version }}-${{ hashFiles('Makefile', '**/Dockerfile*', '**/Makefile') }}
restore-keys: go-artifacts-${{ runner.os }}-test-release-
- name: Run make release
run: |
make clean
make release
- name: Run release tests
run: make test-release-victoria-metrics

3
.gitignore vendored
View File

@@ -23,4 +23,5 @@ Gemfile.lock
_site
*.tmp
/docs/.jekyll-metadata
coverage.txt
coverage.txt
cspell.json

View File

@@ -1,120 +0,0 @@
# Кодекс Поведения участника
## Наши обязательства
Мы, как участники, авторы и лидеры обязуемся сделать участие в сообществе
свободным от притеснений для всех, независимо от возраста, телосложения,
видимых или невидимых ограничений способности, этнической принадлежности,
половых признаков, гендерной идентичности и выражения, уровня опыта,
образования, социо-экономического статуса, национальности, внешности,
расы, религии, или сексуальной идентичности и ориентации.
Мы обещаем действовать и взаимодействовать таким образом, чтобы вносить вклад в открытое,
дружелюбное, многообразное, инклюзивное и здоровое сообщество.
## Наши стандарты
Примеры поведения, создающие условия для благоприятных взаимоотношений включают в себя:
* Проявление доброты и эмпатии к другим участникам проекта
* Уважение к чужой точке зрения и опыту
* Конструктивная критика и принятие конструктивной критики
* Принятие ответственности, принесение извинений тем, кто пострадал от наших ошибок
и извлечение уроков из опыта
* Ориентирование на то, что лучше подходит для сообщества, а не только для нас лично
Примеры неприемлемого поведения участников включают в себя:
* Использование выражений или изображений сексуального характера и нежелательное сексуальное внимание или домогательство в любой форме
* Троллинг, оскорбительные или уничижительные комментарии, переход на личности или затрагивание политических убеждений
* Публичное или приватное домогательство
* Публикация личной информации других лиц, например, физического или электронного адреса, без явного разрешения
* Иное поведение, которое обоснованно считать неуместным в профессиональной обстановке
## Обязанности
Лидеры сообщества отвечают за разъяснение и применение наших стандартов приемлемого
поведения и будут предпринимать соответствующие и честные меры по исправлению положения
в ответ на любое поведение, которое они сочтут неприемлемым, угрожающим, оскорбительным или вредным.
Лидеры сообщества обладают правом и обязанностью удалять, редактировать или отклонять
комментарии, коммиты, код, изменения в вики, вопросы и другой вклад, который не совпадает
с Кодексом Поведения, и предоставят причины принятого решения, когда сочтут нужным.
## Область применения
Данный Кодекс Поведения применим во всех во всех публичных физических и цифровых пространства сообщества,
а также когда человек официально представляет сообщество в публичных местах.
Примеры представления проекта или сообщества включают использование официальной электронной почты,
публикации в официальном аккаунте в социальных сетях,
или упоминания как представителя в онлайн или оффлайн мероприятии.
## Приведение в исполнение
О случаях домогательства, а так же оскорбительного или иного другого неприемлемого
поведения можно сообщить ответственным лидерам сообщества с помощью письма на info@victoriametrics.com
Все жалобы будут рассмотрены и расследованы оперативно и беспристрастно.
Все лидеры сообщества обязаны уважать неприкосновенность частной жизни и личную
неприкосновенность автора сообщения.
## Руководство по исполнению
Лидеры сообщества будут следовать следующим Принципам Воздействия в Сообществе,
чтобы определить последствия для тех, кого они считают виновными в нарушении данного Кодекса Поведения:
### 1. Исправление
**Общественное влияние**: Использование недопустимой лексики или другое поведение,
считающиеся непрофессиональным или нежелательным в сообществе.
**Последствия**: Личное, письменное предупреждение от лидеров сообщества,
объясняющее суть нарушения и почему такое поведение
было неуместно. Лидеры сообщества могут попросить принести публичное извинение.
### 2. Предупреждение
**Общественное влияние**: Нарушение в результате одного инцидента или серии действий.
**Последствия**: Предупреждение о последствиях в случае продолжающегося неуместного поведения.
На определенное время не допускается взаимодействие с людьми, вовлеченными в инцидент,
включая незапрошенное взаимодействие
с теми, кто обеспечивает соблюдение Кодекса. Это включает в себя избегание взаимодействия
в публичных пространствах, а так же во внешних каналах,
таких как социальные сети. Нарушение этих правил влечет за собой временный или вечный бан.
### 3. Временный бан
**Общественное влияние**: Серьёзное нарушение стандартов сообщества,
включая продолжительное неуместное поведение.
**Последствия**: Временный запрет (бан) на любое взаимодействие
или публичное общение с сообществом на определенный период времени.
На этот период не допускается публичное или личное взаимодействие с людьми,
вовлеченными в инцидент, включая незапрошенное взаимодействие
с теми, кто обеспечивает соблюдение Кодекса.
Нарушение этих правил влечет за собой вечный бан.
### 4. Вечный бан
**Общественное влияние**: Демонстрация систематических нарушений стандартов сообщества,
включая продолжающееся неуместное поведение, домогательство до отдельных лиц,
или проявление агрессии либо пренебрежительного отношения к категориям лиц.
**Последствия**: Вечный запрет на любое публичное взаимодействие с сообществом.
## Атрибуция
Данный Кодекс Поведения основан на [Кодекс Поведения участника][homepage],
версии 2.0, доступной по адресу
<https://www.contributor-covenant.org/version/2/0/code_of_conduct.html>.
Принципы Воздействия в Сообществе были вдохновлены [Mozilla's code of conduct
enforcement ladder](https://github.com/mozilla/diversity).
[homepage]: https://www.contributor-covenant.org
Ответы на общие вопросы о данном кодексе поведения ищите на странице FAQ:
<https://www.contributor-covenant.org/faq>. Переводы доступны по адресу
<https://www.contributor-covenant.org/translations>.

174
Makefile
View File

@@ -14,14 +14,86 @@ endif
GO_BUILDINFO = -X '$(PKG_PREFIX)/lib/buildinfo.Version=$(APP_NAME)-$(DATEINFO_TAG)-$(BUILDINFO_TAG)'
VICTORIA_LOGS_COMPONENTS = victoria-logs
VICTORIA_METRICS_COMPONENTS = victoria-metrics
VICTORIA_METRICS_UTILS_COMPONENTS = vmagent vmalert vmalert-tool vmauth vmbackup vmrestore vmctl
.PHONY: $(MAKECMDGOALS)
include app/*/Makefile
include cspell/Makefile
include docs/Makefile
include deployment/*/Makefile
include dashboards/Makefile
include package/release/Makefile
define RELEASE_GOOS_GOARCH
$(eval PKG_NAME := $(1))
$(eval PKG_COMPONENTS := $(2))
# Build
$(foreach pkg_component, $(PKG_COMPONENTS), $(MAKE) $(pkg_component)-$(GOOS)-$(GOARCH)-prod)
# Generate SBOM
mkdir -p "bin/$(PKG_NAME)-$(GOOS)-$(GOARCH)-$(PKG_TAG)_bom"
$(foreach pkg_component, $(PKG_COMPONENTS),
cyclonedx-gomod app -assert-licenses -json -licenses -packages \
-main app/$(pkg_component) \
-output bin/$(PKG_NAME)-$(GOOS)-$(GOARCH)-$(PKG_TAG)_bom/$(pkg_component)-$(GOOS)-$(GOARCH)-$(PKG_TAG)_bom.json)
# Pack and compress
$(foreach pkg_component, $(PKG_COMPONENTS),
cd bin && tar --transform="flags=r;s|-$(GOOS)-$(GOARCH)||" -rf $(PKG_NAME)-$(GOOS)-$(GOARCH)-$(PKG_TAG).tar \
$(pkg_component)-$(GOOS)-$(GOARCH)-prod)
cd bin && gzip $(PKG_NAME)-$(GOOS)-$(GOARCH)-$(PKG_TAG).tar
cd bin && tar -czf $(PKG_NAME)-$(GOOS)-$(GOARCH)-$(PKG_TAG)_bom.tar.gz $(PKG_NAME)-$(GOOS)-$(GOARCH)-$(PKG_TAG)_bom
# Generate checksums
cd bin && \
sha256sum $(PKG_NAME)-$(GOOS)-$(GOARCH)-$(PKG_TAG).tar.gz > $(PKG_NAME)-$(GOOS)-$(GOARCH)-$(PKG_TAG)_checksums.txt && \
sha256sum $(PKG_NAME)-$(GOOS)-$(GOARCH)-$(PKG_TAG)_bom.tar.gz >> $(PKG_NAME)-$(GOOS)-$(GOARCH)-$(PKG_TAG)_checksums.txt
$(foreach pkg_component, $(PKG_COMPONENTS),
cd bin && sha256sum $(pkg_component)-$(GOOS)-$(GOARCH)-prod | sed s/-$(GOOS)-$(GOARCH)-prod/-prod/ >> \
$(PKG_NAME)-$(GOOS)-$(GOARCH)-$(PKG_TAG)_checksums.txt)
# Clean up
cd bin && rm -rf $(PKG_NAME)-$(GOOS)-$(GOARCH)-$(PKG_TAG)_bom
$(foreach pkg_component, $(PKG_COMPONENTS), cd bin && rm -f $(pkg_component)-$(GOOS)-$(GOARCH)-prod)
endef
define RELEASE_WINDOWS_GOARCH
$(eval PKG_NAME := $(1))
$(eval PKG_COMPONENTS := $(2))
# Build
$(foreach pkg_component, $(PKG_COMPONENTS), $(MAKE) $(pkg_component)-$(GOOS)-$(GOARCH)-prod)
# Generate SBOM
mkdir -p "bin/$(PKG_NAME)-$(GOOS)-$(GOARCH)-$(PKG_TAG)_bom"
$(foreach pkg_component, $(PKG_COMPONENTS),
cyclonedx-gomod app -assert-licenses -json -licenses -packages \
-main app/$(pkg_component) \
-output bin/$(PKG_NAME)-$(GOOS)-$(GOARCH)-$(PKG_TAG)_bom/$(pkg_component)-$(GOOS)-$(GOARCH)-$(PKG_TAG)_bom.json)
# Pack and compress
$(foreach pkg_component, $(PKG_COMPONENTS),
cd bin && zip -u $(PKG_NAME)-$(GOOS)-$(GOARCH)-$(PKG_TAG).zip \
$(pkg_component)-$(GOOS)-$(GOARCH)-prod.exe)
cd bin && zip $(PKG_NAME)-$(GOOS)-$(GOARCH)-$(PKG_TAG)_bom.zip -r $(PKG_NAME)-$(GOOS)-$(GOARCH)-$(PKG_TAG)_bom
# Generate checksums
cd bin && \
sha256sum $(PKG_NAME)-$(GOOS)-$(GOARCH)-$(PKG_TAG).zip > $(PKG_NAME)-$(GOOS)-$(GOARCH)-$(PKG_TAG)_checksums.txt && \
sha256sum $(PKG_NAME)-$(GOOS)-$(GOARCH)-$(PKG_TAG)_bom.zip >> $(PKG_NAME)-$(GOOS)-$(GOARCH)-$(PKG_TAG)_checksums.txt
$(foreach pkg_component, $(PKG_COMPONENTS),
cd bin && sha256sum $(pkg_component)-$(GOOS)-$(GOARCH)-prod.exe >> $(PKG_NAME)-$(GOOS)-$(GOARCH)-$(PKG_TAG)_checksums.txt)
# Clean up
cd bin && rm -rf $(PKG_NAME)-$(GOOS)-$(GOARCH)-$(PKG_TAG)_bom
$(foreach pkg_component, $(PKG_COMPONENTS), cd bin && rm -f $(pkg_component)-$(GOOS)-$(GOARCH)-prod.exe)
endef
all: \
victoria-metrics-prod \
victoria-logs-prod \
@@ -240,26 +312,13 @@ release-victoria-metrics-openbsd-amd64:
GOOS=openbsd GOARCH=amd64 $(MAKE) release-victoria-metrics-goos-goarch
release-victoria-metrics-windows-amd64:
GOARCH=amd64 $(MAKE) release-victoria-metrics-windows-goarch
GOOS=windows GOARCH=amd64 $(MAKE) release-victoria-metrics-windows-goarch
release-victoria-metrics-goos-goarch: victoria-metrics-$(GOOS)-$(GOARCH)-prod
cd bin && \
tar --transform="flags=r;s|-$(GOOS)-$(GOARCH)||" -czf victoria-metrics-$(GOOS)-$(GOARCH)-$(PKG_TAG).tar.gz \
victoria-metrics-$(GOOS)-$(GOARCH)-prod \
&& sha256sum victoria-metrics-$(GOOS)-$(GOARCH)-$(PKG_TAG).tar.gz \
victoria-metrics-$(GOOS)-$(GOARCH)-prod \
| sed s/-$(GOOS)-$(GOARCH)-prod/-prod/ > victoria-metrics-$(GOOS)-$(GOARCH)-$(PKG_TAG)_checksums.txt
cd bin && rm -rf victoria-metrics-$(GOOS)-$(GOARCH)-prod
release-victoria-metrics-goos-goarch:
$(call RELEASE_GOOS_GOARCH,victoria-metrics,$(VICTORIA_METRICS_COMPONENTS))
release-victoria-metrics-windows-goarch: victoria-metrics-windows-$(GOARCH)-prod
cd bin && \
zip victoria-metrics-windows-$(GOARCH)-$(PKG_TAG).zip \
victoria-metrics-windows-$(GOARCH)-prod.exe \
&& sha256sum victoria-metrics-windows-$(GOARCH)-$(PKG_TAG).zip \
victoria-metrics-windows-$(GOARCH)-prod.exe \
> victoria-metrics-windows-$(GOARCH)-$(PKG_TAG)_checksums.txt
cd bin && rm -rf \
victoria-metrics-windows-$(GOARCH)-prod.exe
release-victoria-metrics-windows-goarch:
$(call RELEASE_WINDOWS_GOARCH,victoria-metrics,$(VICTORIA_METRICS_COMPONENTS))
release-victoria-logs:
$(MAKE_PARALLEL) release-victoria-logs-linux-386 \
@@ -354,77 +413,13 @@ release-vmutils-openbsd-amd64:
GOOS=openbsd GOARCH=amd64 $(MAKE) release-vmutils-goos-goarch
release-vmutils-windows-amd64:
GOARCH=amd64 $(MAKE) release-vmutils-windows-goarch
GOOS=windows GOARCH=amd64 $(MAKE) release-vmutils-windows-goarch
release-vmutils-goos-goarch: \
vmagent-$(GOOS)-$(GOARCH)-prod \
vmalert-$(GOOS)-$(GOARCH)-prod \
vmalert-tool-$(GOOS)-$(GOARCH)-prod \
vmauth-$(GOOS)-$(GOARCH)-prod \
vmbackup-$(GOOS)-$(GOARCH)-prod \
vmrestore-$(GOOS)-$(GOARCH)-prod \
vmctl-$(GOOS)-$(GOARCH)-prod
cd bin && \
tar --transform="flags=r;s|-$(GOOS)-$(GOARCH)||" -czf vmutils-$(GOOS)-$(GOARCH)-$(PKG_TAG).tar.gz \
vmagent-$(GOOS)-$(GOARCH)-prod \
vmalert-$(GOOS)-$(GOARCH)-prod \
vmalert-tool-$(GOOS)-$(GOARCH)-prod \
vmauth-$(GOOS)-$(GOARCH)-prod \
vmbackup-$(GOOS)-$(GOARCH)-prod \
vmrestore-$(GOOS)-$(GOARCH)-prod \
vmctl-$(GOOS)-$(GOARCH)-prod \
&& sha256sum vmutils-$(GOOS)-$(GOARCH)-$(PKG_TAG).tar.gz \
vmagent-$(GOOS)-$(GOARCH)-prod \
vmalert-$(GOOS)-$(GOARCH)-prod \
vmalert-tool-$(GOOS)-$(GOARCH)-prod \
vmauth-$(GOOS)-$(GOARCH)-prod \
vmbackup-$(GOOS)-$(GOARCH)-prod \
vmrestore-$(GOOS)-$(GOARCH)-prod \
vmctl-$(GOOS)-$(GOARCH)-prod \
| sed s/-$(GOOS)-$(GOARCH)-prod/-prod/ > vmutils-$(GOOS)-$(GOARCH)-$(PKG_TAG)_checksums.txt
cd bin && rm -rf \
vmagent-$(GOOS)-$(GOARCH)-prod \
vmalert-$(GOOS)-$(GOARCH)-prod \
vmalert-tool-$(GOOS)-$(GOARCH)-prod \
vmauth-$(GOOS)-$(GOARCH)-prod \
vmbackup-$(GOOS)-$(GOARCH)-prod \
vmrestore-$(GOOS)-$(GOARCH)-prod \
vmctl-$(GOOS)-$(GOARCH)-prod
release-vmutils-goos-goarch:
$(call RELEASE_GOOS_GOARCH, vmutils, $(VICTORIA_METRICS_UTILS_COMPONENTS))
release-vmutils-windows-goarch: \
vmagent-windows-$(GOARCH)-prod \
vmalert-windows-$(GOARCH)-prod \
vmalert-tool-windows-$(GOARCH)-prod \
vmauth-windows-$(GOARCH)-prod \
vmbackup-windows-$(GOARCH)-prod \
vmrestore-windows-$(GOARCH)-prod \
vmctl-windows-$(GOARCH)-prod
cd bin && \
zip vmutils-windows-$(GOARCH)-$(PKG_TAG).zip \
vmagent-windows-$(GOARCH)-prod.exe \
vmalert-windows-$(GOARCH)-prod.exe \
vmalert-tool-windows-$(GOARCH)-prod.exe \
vmauth-windows-$(GOARCH)-prod.exe \
vmbackup-windows-$(GOARCH)-prod.exe \
vmrestore-windows-$(GOARCH)-prod.exe \
vmctl-windows-$(GOARCH)-prod.exe \
&& sha256sum vmutils-windows-$(GOARCH)-$(PKG_TAG).zip \
vmagent-windows-$(GOARCH)-prod.exe \
vmalert-windows-$(GOARCH)-prod.exe \
vmalert-tool-windows-$(GOARCH)-prod.exe \
vmauth-windows-$(GOARCH)-prod.exe \
vmbackup-windows-$(GOARCH)-prod.exe \
vmrestore-windows-$(GOARCH)-prod.exe \
vmctl-windows-$(GOARCH)-prod.exe \
> vmutils-windows-$(GOARCH)-$(PKG_TAG)_checksums.txt
cd bin && rm -rf \
vmagent-windows-$(GOARCH)-prod.exe \
vmalert-windows-$(GOARCH)-prod.exe \
vmalert-tool-windows-$(GOARCH)-prod.exe \
vmauth-windows-$(GOARCH)-prod.exe \
vmbackup-windows-$(GOARCH)-prod.exe \
vmrestore-windows-$(GOARCH)-prod.exe \
vmctl-windows-$(GOARCH)-prod.exe
release-vmutils-windows-goarch:
$(call RELEASE_WINDOWS_GOARCH, vmutils, $(VICTORIA_METRICS_UTILS_COMPONENTS))
pprof-cpu:
go tool pprof -trim_path=github.com/VictoriaMetrics/VictoriaMetrics@ $(PPROF_FILE)
@@ -493,7 +488,6 @@ golangci-lint: install-golangci-lint
golangci-lint run
install-golangci-lint:
# The version must match GitHub main.yml lint job "Run golangci-lint" step version.
which golangci-lint || curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(shell go env GOPATH)/bin v1.59.1
remove-golangci-lint:
@@ -514,6 +508,9 @@ install-wwhrd:
check-licenses: install-wwhrd
wwhrd check -f .wwhrd.yml
cyclonedx-gomod-install:
which cyclonedx-gomod || go install github.com/CycloneDX/cyclonedx-gomod/cmd/cyclonedx-gomod@latest
copy-docs:
# The 'printf' function is used instead of 'echo' or 'echo -e' to handle line breaks (e.g. '\n') in the same way on different operating systems (MacOS/Ubuntu Linux/Arch Linux) and their shells (bash/sh/zsh/fish).
# For details, see https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4548#issue-1782796419 and https://stackoverflow.com/questions/8467424/echo-newline-in-bash-prints-literal-n
@@ -531,6 +528,7 @@ copy-docs:
echo "---" >> ${DST}
cat ${SRC} >> ${DST}
sed -i='.tmp' 's/<img src=\"docs\//<img src=\"/' ${DST}
sed -i='.tmp' 's/<source srcset=\"docs\//<source srcset=\"/' ${DST}
rm -rf docs/*.tmp
# Copies docs for all components and adds the order/weight tag, title, menu position and alias with the backward compatible link for the old site.

View File

@@ -2,14 +2,17 @@
[![Latest Release](https://img.shields.io/github/release/VictoriaMetrics/VictoriaMetrics.svg?style=flat-square)](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest)
[![Docker Pulls](https://img.shields.io/docker/pulls/victoriametrics/victoria-metrics.svg?maxAge=604800)](https://hub.docker.com/r/victoriametrics/victoria-metrics)
[![victoriametrics](https://snapcraft.io/victoriametrics/badge.svg)](https://snapcraft.io/victoriametrics)
[![Slack](https://img.shields.io/badge/join%20slack-%23victoriametrics-brightgreen.svg)](https://slack.victoriametrics.com/)
[![GitHub license](https://img.shields.io/github/license/VictoriaMetrics/VictoriaMetrics.svg)](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/LICENSE)
[![Go Report](https://goreportcard.com/badge/github.com/VictoriaMetrics/VictoriaMetrics)](https://goreportcard.com/report/github.com/VictoriaMetrics/VictoriaMetrics)
[![Build Status](https://github.com/VictoriaMetrics/VictoriaMetrics/workflows/main/badge.svg)](https://github.com/VictoriaMetrics/VictoriaMetrics/actions)
[![codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/branch/master/graph/badge.svg)](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics)
<img src="docs/logo.webp" width="300" alt="VictoriaMetrics logo">
<picture>
<source srcset="docs/logo_white.webp" media="(prefers-color-scheme: dark)">
<source srcset="docs/logo.webp" media="(prefers-color-scheme: light)">
<img src="docs/logo.webp" width="300" alt="VictoriaMetrics logo">
</picture>
VictoriaMetrics is a fast, cost-effective and scalable monitoring solution and time series database.
See [case studies for VictoriaMetrics](https://docs.victoriametrics.com/casestudies/).
@@ -177,10 +180,6 @@ Additionally, all the VictoriaMetrics components allow setting flag values via e
* For repeating flags an alternative syntax can be used by joining the different values into one using `,` char as separator (for example `-storageNode <nodeA> -storageNode <nodeB>` will translate to `storageNode=<nodeA>,<nodeB>`).
* Environment var prefix can be set via `-envflag.prefix` flag. For instance, if `-envflag.prefix=VM_`, then env vars must be prepended with `VM_`.
### Configuration with snap package
Snap packages for VictoriaMetrics are supported by community and are available at [https://snapcraft.io/victoriametrics](https://snapcraft.io/victoriametrics).
### Running as Windows service
In order to run VictoriaMetrics as a Windows service it is required to create a service configuration for [WinSW](https://github.com/winsw/winsw)
@@ -2669,6 +2668,9 @@ used or posted in other docs, GitHub issues, stackoverlow answers, etc.
Please, keep image size and number of images per single page low. Keep the docs page as lightweight as possible.
Image files must be placed in the same folder as the doc itself and they must have the same prefix as the doc filename.
For example, all the images for `docs/foo/bar.md` should have filenames starting from `docs/foo/bar`.
If the page needs to have many images, consider using WEB-optimized image format [webp](https://developers.google.com/speed/webp).
When adding a new doc with many images use `webp` format right away. Or use a Makefile command below to
convert already existing images at `docs` folder automatically to `web` format:
@@ -2721,7 +2723,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
-cacheExpireDuration duration
Items are removed from in-memory caches after they aren't accessed for this duration. Lower values may reduce memory usage at the cost of higher CPU usage. See also -prevCacheRemovalPercent (default 30m0s)
-configAuthKey value
Authorization key for accessing /config page. It must be passed via authKey query arg. It overrides httpAuth.* settings.
Authorization key for accessing /config page. It must be passed via authKey query arg. It overrides -httpAuth.*
Flag value can be read from the given file when using -configAuthKey=file:///abs/path/to/file or -configAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -configAuthKey=http://host/path or -configAuthKey=https://host/path
-csvTrimTimestamp duration
Trim timestamps when importing csv data to this duration. Minimum practical duration is 1ms. Higher duration (i.e. 1s) may be used for reducing disk space usage for timestamp data (default 1ms)
@@ -2758,7 +2760,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
-finalMergeDelay duration
Deprecated: this flag does nothing
-flagsAuthKey value
Auth key for /flags endpoint. It must be passed via authKey query arg. It overrides httpAuth.* settings
Auth key for /flags endpoint. It must be passed via authKey query arg. It overrides -httpAuth.*
Flag value can be read from the given file when using -flagsAuthKey=file:///abs/path/to/file or -flagsAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -flagsAuthKey=http://host/path or -flagsAuthKey=https://host/path
-forceFlushAuthKey value
authKey, which must be passed in query string to /internal/force_flush pages
@@ -2873,7 +2875,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
The maximum size in bytes of a single Prometheus remote_write API request
Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 33554432)
-maxLabelValueLen int
The maximum length of label values in the accepted time series. Longer label values are truncated. In this case the vm_too_long_label_values_total metric at /metrics page is incremented (default 1024)
The maximum length of label values in the accepted time series. Longer label values are truncated. In this case the vm_too_long_label_values_total metric at /metrics page is incremented (default 4096)
-maxLabelsPerTimeseries int
The maximum number of labels accepted per time series. Superfluous labels are dropped. In this case the vm_metrics_with_dropped_labels_total metric at /metrics page is incremented (default 30)
-memory.allowedBytes size
@@ -2884,7 +2886,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
-metrics.exposeMetadata
Whether to expose TYPE and HELP metadata at the /metrics page, which is exposed at -httpListenAddr . The metadata may be needed when the /metrics page is consumed by systems, which require this information. For example, Managed Prometheus in Google Cloud - https://cloud.google.com/stackdriver/docs/managed-prometheus/troubleshooting#missing-metric-type
-metricsAuthKey value
Auth key for /metrics endpoint. It must be passed via authKey query arg. It overrides httpAuth.* settings
Auth key for /metrics endpoint. It must be passed via authKey query arg. It overrides -httpAuth.*
Flag value can be read from the given file when using -metricsAuthKey=file:///abs/path/to/file or -metricsAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -metricsAuthKey=http://host/path or -metricsAuthKey=https://host/path
-mtls array
Whether to require valid client certificate for https requests to the corresponding -httpListenAddr . This flag works only if -tls flag is set. See also -mtlsCAFile . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
@@ -2915,7 +2917,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
-opentsdbhttpTrimTimestamp duration
Trim timestamps for OpenTSDB HTTP data to this duration. Minimum practical duration is 1ms. Higher duration (i.e. 1s) may be used for reducing disk space usage for timestamp data (default 1ms)
-pprofAuthKey value
Auth key for /debug/pprof/* endpoints. It must be passed via authKey query arg. It overrides httpAuth.* settings
Auth key for /debug/pprof/* endpoints. It must be passed via authKey query arg. It overrides -httpAuth.*
Flag value can be read from the given file when using -pprofAuthKey=file:///abs/path/to/file or -pprofAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -pprofAuthKey=http://host/path or -pprofAuthKey=https://host/path
-precisionBits int
The number of precision bits to store per each value. Lower precision bits improves data compression at the cost of precision loss (default 64)
@@ -3037,7 +3039,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
-relabelConfig string
Optional path to a file with relabeling rules, which are applied to all the ingested metrics. The path can point either to local file or to http url. See https://docs.victoriametrics.com/#relabeling for details. The config is reloaded on SIGHUP signal
-reloadAuthKey value
Auth key for /-/reload http endpoint. It must be passed via authKey query arg. It overrides httpAuth.* settings.
Auth key for /-/reload http endpoint. It must be passed via authKey query arg. It overrides -httpAuth.*
Flag value can be read from the given file when using -reloadAuthKey=file:///abs/path/to/file or -reloadAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -reloadAuthKey=http://host/path or -reloadAuthKey=https://host/path
-retentionFilter array
Retention filter in the format 'filter:retention'. For example, '{env="dev"}:3d' configures the retention for time series with env="dev" label to 3 days. See https://docs.victoriametrics.com/#retention-filters for details. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/

View File

@@ -81,6 +81,9 @@ victoria-logs-linux-ppc64le:
victoria-logs-linux-s390x:
APP_NAME=victoria-logs CGO_ENABLED=0 GOOS=linux GOARCH=s390x $(MAKE) app-local-goos-goarch
victoria-logs-linux-loong64:
APP_NAME=victoria-logs CGO_ENABLED=0 GOOS=linux GOARCH=loong64 $(MAKE) app-local-goos-goarch
victoria-logs-linux-386:
APP_NAME=victoria-logs CGO_ENABLED=0 GOOS=linux GOARCH=386 $(MAKE) app-local-goos-goarch

View File

@@ -3,6 +3,7 @@ package main
import (
"bytes"
"encoding/json"
"errors"
"flag"
"fmt"
"io"
@@ -117,7 +118,7 @@ func (q *Query) metrics() ([]Metric, error) {
type QueryInstant struct {
Result []struct {
Labels map[string]string `json:"metric"`
TV [2]interface{} `json:"value"`
TV [2]any `json:"value"`
} `json:"result"`
}
@@ -140,7 +141,7 @@ func (q QueryInstant) metrics() ([]Metric, error) {
type QueryRange struct {
Result []struct {
Metric map[string]string `json:"metric"`
Values [][]interface{} `json:"values"`
Values [][]any `json:"values"`
} `json:"result"`
}
@@ -249,24 +250,24 @@ func TestWriteRead(t *testing.T) {
func testWrite(t *testing.T) {
t.Run("prometheus", func(t *testing.T) {
for _, test := range readIn("prometheus", t, insertionTime) {
for _, test := range readIn("prometheus", insertionTime) {
if test.Data == nil {
continue
}
s := newSuite(t)
r := testutil.WriteRequest{}
s.noError(json.Unmarshal([]byte(strings.Join(test.Data, "\n")), &r.Timeseries))
data, err := testutil.Compress(r)
s.greaterThan(len(r.Timeseries), 0)
if err != nil {
t.Errorf("error compressing %v %s", r, err)
t.Fail()
testData := strings.Join(test.Data, "\n")
if err := json.Unmarshal([]byte(testData), &r.Timeseries); err != nil {
panic(fmt.Errorf("BUG: cannot unmarshal TimeSeries: %s\ntest data\n%s", err, testData))
}
if n := len(r.Timeseries); n <= 0 {
panic(fmt.Errorf("BUG: expecting non-empty Timeseries in test data:\n%s", testData))
}
data := testutil.Compress(r)
httpWrite(t, testPromWriteHTTPPath, test.InsertQuery, bytes.NewBuffer(data))
}
})
t.Run("csv", func(t *testing.T) {
for _, test := range readIn("csv", t, insertionTime) {
for _, test := range readIn("csv", insertionTime) {
if test.Data == nil {
continue
}
@@ -275,7 +276,7 @@ func testWrite(t *testing.T) {
})
t.Run("influxdb", func(t *testing.T) {
for _, x := range readIn("influxdb", t, insertionTime) {
for _, x := range readIn("influxdb", insertionTime) {
test := x
t.Run(test.Name, func(t *testing.T) {
t.Parallel()
@@ -284,7 +285,7 @@ func testWrite(t *testing.T) {
}
})
t.Run("graphite", func(t *testing.T) {
for _, x := range readIn("graphite", t, insertionTime) {
for _, x := range readIn("graphite", insertionTime) {
test := x
t.Run(test.Name, func(t *testing.T) {
t.Parallel()
@@ -293,7 +294,7 @@ func testWrite(t *testing.T) {
}
})
t.Run("opentsdb", func(t *testing.T) {
for _, x := range readIn("opentsdb", t, insertionTime) {
for _, x := range readIn("opentsdb", insertionTime) {
test := x
t.Run(test.Name, func(t *testing.T) {
t.Parallel()
@@ -302,7 +303,7 @@ func testWrite(t *testing.T) {
}
})
t.Run("opentsdbhttp", func(t *testing.T) {
for _, x := range readIn("opentsdbhttp", t, insertionTime) {
for _, x := range readIn("opentsdbhttp", insertionTime) {
test := x
t.Run(test.Name, func(t *testing.T) {
t.Parallel()
@@ -316,7 +317,7 @@ func testWrite(t *testing.T) {
func testRead(t *testing.T) {
for _, engine := range []string{"csv", "prometheus", "graphite", "opentsdb", "influxdb", "opentsdbhttp"} {
t.Run(engine, func(t *testing.T) {
for _, x := range readIn(engine, t, insertionTime) {
for _, x := range readIn(engine, insertionTime) {
test := x
t.Run(test.Name, func(t *testing.T) {
t.Parallel()
@@ -365,11 +366,10 @@ func testRead(t *testing.T) {
}
}
func readIn(readFor string, t *testing.T, insertTime time.Time) []test {
t.Helper()
s := newSuite(t)
func readIn(readFor string, insertTime time.Time) []test {
testDir := filepath.Join(testFixturesDir, readFor)
var tt []test
s.noError(filepath.Walk(filepath.Join(testFixturesDir, readFor), func(path string, _ os.FileInfo, err error) error {
err := filepath.Walk(testDir, func(path string, _ os.FileInfo, err error) error {
if err != nil {
return err
}
@@ -377,84 +377,129 @@ func readIn(readFor string, t *testing.T, insertTime time.Time) []test {
return nil
}
b, err := os.ReadFile(path)
s.noError(err)
if err != nil {
panic(fmt.Errorf("BUG: cannot read %s: %s", path, err))
}
item := test{}
s.noError(json.Unmarshal(b, &item))
if err := json.Unmarshal(b, &item); err != nil {
panic(fmt.Errorf("cannot parse %T from %s: %s; data:\n%s", &item, path, err, b))
}
for i := range item.Data {
item.Data[i] = testutil.PopulateTimeTplString(item.Data[i], insertTime)
}
tt = append(tt, item)
return nil
}))
})
if err != nil {
panic(fmt.Errorf("BUG: cannot read test data at %s: %w", testDir, err))
}
if len(tt) == 0 {
t.Fatalf("no test found in %s", filepath.Join(testFixturesDir, readFor))
panic(fmt.Errorf("BUG: no tests found in %s", testDir))
}
return tt
}
func httpWrite(t *testing.T, address, query string, r io.Reader) {
t.Helper()
s := newSuite(t)
resp, err := http.Post(address+query, "", r)
s.noError(err)
s.noError(resp.Body.Close())
s.equalInt(resp.StatusCode, 204)
requestURL := address + query
resp, err := http.Post(requestURL, "", r)
if err != nil {
t.Fatalf("cannot send request to %s: %s", requestURL, err)
}
_ = resp.Body.Close()
if resp.StatusCode != 204 {
t.Fatalf("unexpected status code received from %s; got %d; want 204", requestURL, resp.StatusCode)
}
}
func tcpWrite(t *testing.T, address string, data string) {
func tcpWrite(t *testing.T, address, data string) {
t.Helper()
s := newSuite(t)
conn, err := net.Dial("tcp", address)
s.noError(err)
if err != nil {
t.Fatalf("cannot dial %s: %s", address, err)
}
defer func() {
_ = conn.Close()
}()
n, err := conn.Write([]byte(data))
s.noError(err)
s.equalInt(n, len(data))
if err != nil {
t.Fatalf("cannot write %d bytes to %s: %s", len(data), address, err)
}
if n != len(data) {
panic(fmt.Errorf("BUG: conn.Write() returned unexpected number of written bytes to %s; got %d; want %d", address, n, len(data)))
}
}
func httpReadMetrics(t *testing.T, address, query string) []Metric {
t.Helper()
s := newSuite(t)
resp, err := http.Get(address + query)
s.noError(err)
requestURL := address + query
resp, err := http.Get(requestURL)
if err != nil {
t.Fatalf("cannot send request to %s: %s", requestURL, err)
}
defer func() {
_ = resp.Body.Close()
}()
s.equalInt(resp.StatusCode, 200)
if resp.StatusCode != 200 {
t.Fatalf("unexpected status code received from %s; got %d; want 200", requestURL, resp.StatusCode)
}
var rows []Metric
for dec := json.NewDecoder(resp.Body); dec.More(); {
dec := json.NewDecoder(resp.Body)
for {
var row Metric
s.noError(dec.Decode(&row))
err := dec.Decode(&row)
if err != nil {
if errors.Is(err, io.EOF) {
return rows
}
t.Fatalf("cannot decode %T from response received from %s: %s", &row, requestURL, err)
}
rows = append(rows, row)
}
return rows
}
func httpReadStruct(t *testing.T, address, query string, dst interface{}) {
func httpReadStruct(t *testing.T, address, query string, dst any) {
t.Helper()
s := newSuite(t)
resp, err := http.Get(address + query)
s.noError(err)
requestURL := address + query
resp, err := http.Get(requestURL)
if err != nil {
t.Fatalf("cannot send request to %s: %s", requestURL, err)
}
defer func() {
_ = resp.Body.Close()
}()
s.equalInt(resp.StatusCode, 200)
s.noError(json.NewDecoder(resp.Body).Decode(dst))
if resp.StatusCode != 200 {
t.Fatalf("unexpected status code received from %s; got %d; want 200", requestURL, resp.StatusCode)
}
err = json.NewDecoder(resp.Body).Decode(dst)
if err != nil {
t.Fatalf("cannot decode %T from response received from %s: %s", dst, requestURL, err)
}
}
func httpReadData(t *testing.T, address, query string) []byte {
t.Helper()
s := newSuite(t)
resp, err := http.Get(address + query)
s.noError(err)
requestURL := address + query
resp, err := http.Get(requestURL)
if err != nil {
t.Fatalf("cannot send request to %s: %s", requestURL, err)
}
defer func() {
_ = resp.Body.Close()
}()
s.equalInt(resp.StatusCode, 200)
if resp.StatusCode != 200 {
t.Fatalf("unexpected status code received from %s; got %d; want 200", requestURL, resp.StatusCode)
}
data, err := io.ReadAll(resp.Body)
s.noError(err)
if err != nil {
t.Fatalf("cannot read response from %s: %s", requestURL, err)
}
return data
}
@@ -503,34 +548,6 @@ func removeIfFoundSeries(r map[string]string, contains []map[string]string) []ma
return contains
}
type suite struct{ t *testing.T }
func newSuite(t *testing.T) *suite { return &suite{t: t} }
func (s *suite) noError(err error) {
s.t.Helper()
if err != nil {
s.t.Errorf("unexpected error %v", err)
s.t.FailNow()
}
}
func (s *suite) equalInt(a, b int) {
s.t.Helper()
if a != b {
s.t.Errorf("%d not equal %d", a, b)
s.t.FailNow()
}
}
func (s *suite) greaterThan(a, b int) {
s.t.Helper()
if a <= b {
s.t.Errorf("%d less or equal then %d", a, b)
s.t.FailNow()
}
}
func TestImportJSONLines(t *testing.T) {
f := func(labelsCount, labelLen int) {
t.Helper()

View File

@@ -1,12 +1,16 @@
package test
import "github.com/golang/snappy"
import (
"fmt"
"github.com/golang/snappy"
)
// Compress marshals and compresses wr.
func Compress(wr WriteRequest) ([]byte, error) {
func Compress(wr WriteRequest) []byte {
data, err := wr.Marshal()
if err != nil {
return nil, err
panic(fmt.Errorf("BUG: cannot compress WriteRequest: %s", err))
}
return snappy.Encode(nil, data), nil
return snappy.Encode(nil, data)
}

View File

@@ -79,12 +79,14 @@ func parseProtobufRequest(data []byte, lmp insertutils.LogMessageProcessor) (int
req := getPushRequest()
defer putPushRequest(req)
err = req.Unmarshal(bb.B)
err = req.UnmarshalProtobuf(bb.B)
if err != nil {
return 0, fmt.Errorf("cannot parse request body: %w", err)
}
var commonFields []logstorage.Field
fields := getFields()
defer putFields(fields)
rowsIngested := 0
streams := req.Streams
currentTimestamp := time.Now().UnixNano()
@@ -92,30 +94,60 @@ func parseProtobufRequest(data []byte, lmp insertutils.LogMessageProcessor) (int
stream := &streams[i]
// st.Labels contains labels for the stream.
// Labels are same for all entries in the stream.
commonFields, err = parsePromLabels(commonFields[:0], stream.Labels)
fields.fields, err = parsePromLabels(fields.fields[:0], stream.Labels)
if err != nil {
return rowsIngested, fmt.Errorf("cannot parse stream labels %q: %w", stream.Labels, err)
}
fields := commonFields
commonFieldsLen := len(fields.fields)
entries := stream.Entries
for j := range entries {
entry := &entries[j]
fields = append(fields[:len(commonFields)], logstorage.Field{
e := &entries[j]
fields.fields = fields.fields[:commonFieldsLen]
for _, lp := range e.StructuredMetadata {
fields.fields = append(fields.fields, logstorage.Field{
Name: lp.Name,
Value: lp.Value,
})
}
fields.fields = append(fields.fields, logstorage.Field{
Name: "_msg",
Value: entry.Line,
Value: e.Line,
})
ts := entry.Timestamp.UnixNano()
ts := e.Timestamp.UnixNano()
if ts == 0 {
ts = currentTimestamp
}
lmp.AddRow(ts, fields)
lmp.AddRow(ts, fields.fields)
}
rowsIngested += len(stream.Entries)
}
return rowsIngested, nil
}
func getFields() *fields {
v := fieldsPool.Get()
if v == nil {
return &fields{}
}
return v.(*fields)
}
func putFields(f *fields) {
f.fields = f.fields[:0]
fieldsPool.Put(f)
}
var fieldsPool sync.Pool
type fields struct {
fields []logstorage.Field
}
// parsePromLabels parses log fields in Prometheus text exposition format from s, appends them to dst and returns the result.
//
// See test data of promtail for examples: https://github.com/grafana/loki/blob/a24ef7b206e0ca63ee74ca6ecb0a09b745cd2258/pkg/push/types_test.go
@@ -181,6 +213,6 @@ func getPushRequest() *PushRequest {
}
func putPushRequest(req *PushRequest) {
req.Reset()
req.reset()
pushReqsPool.Put(req)
}

View File

@@ -36,7 +36,7 @@ func (tlp *testLogMessageProcessor) AddRow(timestamp int64, fields []logstorage.
Entries: []Entry{
{
Timestamp: time.Unix(0, timestamp),
Line: msg,
Line: strings.Clone(msg),
},
},
})
@@ -58,10 +58,7 @@ func TestParseProtobufRequest_Success(t *testing.T) {
t.Fatalf("unexpected number of streams; got %d; want %d", len(tlp.pr.Streams), n)
}
data, err := tlp.pr.Marshal()
if err != nil {
t.Fatalf("unexpected error when marshaling PushRequest: %s", err)
}
data := tlp.pr.MarshalProtobuf(nil)
encodedData := snappy.Encode(nil, data)
tlp2 := &insertutils.TestLogMessageProcessor{}

View File

@@ -9,6 +9,7 @@ import (
"github.com/golang/snappy"
"github.com/VictoriaMetrics/VictoriaMetrics/app/vlinsert/insertutils"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
)
func BenchmarkParseProtobufRequest(b *testing.B) {
@@ -38,29 +39,47 @@ func benchmarkParseProtobufRequest(b *testing.B, streams, rows, labels int) {
})
}
func getProtobufBody(streams, rows, labels int) []byte {
var pr PushRequest
for i := 0; i < streams; i++ {
var st Stream
st.Labels = `{`
for j := 0; j < labels; j++ {
st.Labels += `label_` + strconv.Itoa(j) + `="value_` + strconv.Itoa(j) + `"`
if j < labels-1 {
st.Labels += `,`
func getProtobufBody(streamsCount, rowsCount, labelsCount int) []byte {
var b []byte
var entries []Entry
streams := make([]Stream, streamsCount)
for i := range streams {
b = b[:0]
b = append(b, '{')
for j := 0; j < labelsCount; j++ {
b = append(b, "label_"...)
b = strconv.AppendInt(b, int64(j), 10)
b = append(b, `="value_`...)
b = strconv.AppendInt(b, int64(j), 10)
b = append(b, '"')
if j < labelsCount-1 {
b = append(b, ',')
}
}
st.Labels += `}`
b = append(b, '}')
labels := string(b)
for j := 0; j < rows; j++ {
st.Entries = append(st.Entries, Entry{Timestamp: time.Now(), Line: "value_" + strconv.Itoa(j)})
var rowsBuf []byte
entriesLen := len(entries)
for j := 0; j < rowsCount; j++ {
rowsBufLen := len(rowsBuf)
rowsBuf = append(rowsBuf, "value_"...)
rowsBuf = strconv.AppendInt(rowsBuf, int64(j), 10)
entries = append(entries, Entry{
Timestamp: time.Now(),
Line: bytesutil.ToUnsafeString(rowsBuf[rowsBufLen:]),
})
}
pr.Streams = append(pr.Streams, st)
st := &streams[i]
st.Labels = labels
st.Entries = entries[entriesLen:]
}
pr := PushRequest{
Streams: streams,
}
body, _ := pr.Marshal()
body := pr.MarshalProtobuf(nil)
encodedBody := snappy.Encode(nil, body)
return encodedBody

302
app/vlinsert/loki/pb.go Normal file
View File

@@ -0,0 +1,302 @@
// Code generated by protoc-gen-gogo. DO NOT EDIT.
// source: push_request.proto
// source: https://raw.githubusercontent.com/grafana/loki/main/pkg/push/push_request.proto
// Licensed under the Apache License, Version 2.0 (the "License");
// https://github.com/grafana/loki/blob/main/pkg/push/LICENSE
package loki
import (
"fmt"
"time"
"github.com/VictoriaMetrics/easyproto"
)
var mp easyproto.MarshalerPool
// PushRequest represents Loki PushRequest
//
// See https://github.com/grafana/loki/blob/4220737a52da7ab6c9346b12d5a5d7bedbcd641d/pkg/push/push.proto#L14C1-L14C20
type PushRequest struct {
Streams []Stream
entriesBuf []Entry
labelPairBuf []LabelPair
}
func (pr *PushRequest) reset() {
pr.Streams = pr.Streams[:0]
pr.entriesBuf = pr.entriesBuf[:0]
pr.labelPairBuf = pr.labelPairBuf[:0]
}
// UnmarshalProtobuf unmarshals pr from protobuf message at src.
//
// pr remains valid until src is modified.
func (pr *PushRequest) UnmarshalProtobuf(src []byte) error {
pr.reset()
var err error
pr.entriesBuf, pr.labelPairBuf, err = pr.unmarshalProtobuf(pr.entriesBuf, pr.labelPairBuf, src)
return err
}
// MarshalProtobuf marshals r to protobuf message, appends it to dst and returns the result.
func (pr *PushRequest) MarshalProtobuf(dst []byte) []byte {
m := mp.Get()
pr.marshalProtobuf(m.MessageMarshaler())
dst = m.Marshal(dst)
mp.Put(m)
return dst
}
func (pr *PushRequest) marshalProtobuf(mm *easyproto.MessageMarshaler) {
for _, s := range pr.Streams {
s.marshalProtobuf(mm.AppendMessage(1))
}
}
func (pr *PushRequest) unmarshalProtobuf(entriesBuf []Entry, labelPairBuf []LabelPair, src []byte) ([]Entry, []LabelPair, error) {
// message PushRequest {
// repeated Stream streams = 1;
// }
var err error
var fc easyproto.FieldContext
for len(src) > 0 {
src, err = fc.NextField(src)
if err != nil {
return entriesBuf, labelPairBuf, fmt.Errorf("cannot read next field in PushRequest: %w", err)
}
switch fc.FieldNum {
case 1:
data, ok := fc.MessageData()
if !ok {
return entriesBuf, labelPairBuf, fmt.Errorf("cannot read Stream data")
}
pr.Streams = append(pr.Streams, Stream{})
s := &pr.Streams[len(pr.Streams)-1]
entriesBuf, labelPairBuf, err = s.unmarshalProtobuf(entriesBuf, labelPairBuf, data)
if err != nil {
return entriesBuf, labelPairBuf, fmt.Errorf("cannot unmarshal Stream: %w", err)
}
}
}
return entriesBuf, labelPairBuf, nil
}
// Stream represents Loki stream.
//
// See https://github.com/grafana/loki/blob/4220737a52da7ab6c9346b12d5a5d7bedbcd641d/pkg/push/push.proto#L23
type Stream struct {
Labels string
Entries []Entry
}
func (s *Stream) marshalProtobuf(mm *easyproto.MessageMarshaler) {
mm.AppendString(1, s.Labels)
for _, e := range s.Entries {
e.marshalProtobuf(mm.AppendMessage(2))
}
}
func (s *Stream) unmarshalProtobuf(entriesBuf []Entry, labelPairBuf []LabelPair, src []byte) ([]Entry, []LabelPair, error) {
// message Stream {
// string labels = 1;
// repeated Entry entries = 2;
// }
var err error
var fc easyproto.FieldContext
entriesBufLen := len(entriesBuf)
for len(src) > 0 {
src, err = fc.NextField(src)
if err != nil {
return entriesBuf, labelPairBuf, fmt.Errorf("cannot read next field in Stream: %w", err)
}
switch fc.FieldNum {
case 1:
labels, ok := fc.String()
if !ok {
return entriesBuf, labelPairBuf, fmt.Errorf("cannot read labels")
}
s.Labels = labels
case 2:
data, ok := fc.MessageData()
if !ok {
return entriesBuf, labelPairBuf, fmt.Errorf("cannot read Entry data")
}
entriesBuf = append(entriesBuf, Entry{})
e := &entriesBuf[len(entriesBuf)-1]
labelPairBuf, err = e.unmarshalProtobuf(labelPairBuf, data)
if err != nil {
return entriesBuf, labelPairBuf, fmt.Errorf("cannot unmarshal Entry: %w", err)
}
}
}
s.Entries = entriesBuf[entriesBufLen:]
return entriesBuf, labelPairBuf, nil
}
// Entry represents Loki entry.
//
// See https://github.com/grafana/loki/blob/4220737a52da7ab6c9346b12d5a5d7bedbcd641d/pkg/push/push.proto#L38
type Entry struct {
Timestamp time.Time
Line string
StructuredMetadata []LabelPair
}
func (e *Entry) marshalProtobuf(mm *easyproto.MessageMarshaler) {
marshalTime(mm, 1, e.Timestamp)
mm.AppendString(2, e.Line)
for _, lp := range e.StructuredMetadata {
lp.marshalProtobuf(mm.AppendMessage(3))
}
}
func (e *Entry) unmarshalProtobuf(labelPairBuf []LabelPair, src []byte) ([]LabelPair, error) {
// message Entry {
// Timestamp timestamp = 1;
// string line = 2;
// repeated LabelPair structuredMetadata = 3;
// }
var err error
var fc easyproto.FieldContext
labelPairBufLen := len(labelPairBuf)
for len(src) > 0 {
src, err = fc.NextField(src)
if err != nil {
return labelPairBuf, fmt.Errorf("cannot read next field in Entry: %w", err)
}
switch fc.FieldNum {
case 1:
data, ok := fc.MessageData()
if !ok {
return labelPairBuf, fmt.Errorf("cannot read Timestamp data")
}
timestamp, err := unmarshalTime(data)
if err != nil {
return labelPairBuf, fmt.Errorf("cannot unmarshal Timestamp: %w", err)
}
e.Timestamp = timestamp
case 2:
line, ok := fc.String()
if !ok {
return labelPairBuf, fmt.Errorf("cannot read Line")
}
e.Line = line
case 3:
data, ok := fc.MessageData()
if !ok {
return labelPairBuf, fmt.Errorf("cannot read StructuredMetadata")
}
labelPairBuf = append(labelPairBuf, LabelPair{})
lp := &labelPairBuf[len(labelPairBuf)-1]
if err := lp.unmarshalProtobuf(data); err != nil {
return labelPairBuf, fmt.Errorf("cannot unmarshal StructuredMetadata: %w", err)
}
}
}
e.StructuredMetadata = labelPairBuf[labelPairBufLen:]
return labelPairBuf, nil
}
// LabelPair represents Loki label pair.
//
// See https://github.com/grafana/loki/blob/4220737a52da7ab6c9346b12d5a5d7bedbcd641d/pkg/push/push.proto#L33
type LabelPair struct {
Name string
Value string
}
func (lp *LabelPair) marshalProtobuf(mm *easyproto.MessageMarshaler) {
mm.AppendString(1, lp.Name)
mm.AppendString(2, lp.Value)
}
func (lp *LabelPair) unmarshalProtobuf(src []byte) (err error) {
// message LabelPair {
// string name = 1;
// string value = 2;
// }
var fc easyproto.FieldContext
for len(src) > 0 {
src, err = fc.NextField(src)
if err != nil {
return fmt.Errorf("cannot read next field in LabelPair: %w", err)
}
switch fc.FieldNum {
case 1:
name, ok := fc.String()
if !ok {
return fmt.Errorf("cannot read name")
}
lp.Name = name
case 2:
value, ok := fc.String()
if !ok {
return fmt.Errorf("cannot unmarshal value")
}
lp.Value = value
}
}
return nil
}
func marshalTime(mm *easyproto.MessageMarshaler, fieldNum uint32, timestamp time.Time) {
nsecs := timestamp.UnixNano()
ts := Timestamp{
Seconds: nsecs / 1e9,
Nanos: int32(nsecs % 1e9),
}
ts.marshalProtobuf(mm.AppendMessage(fieldNum))
}
func unmarshalTime(src []byte) (time.Time, error) {
var ts Timestamp
if err := ts.unmarshalProtobuf(src); err != nil {
return time.Time{}, err
}
timestamp := time.Unix(ts.Seconds, int64(ts.Nanos)).UTC()
return timestamp, nil
}
// Timestamp is protobuf well-known timestamp type.
type Timestamp struct {
Seconds int64
Nanos int32
}
func (ts *Timestamp) marshalProtobuf(mm *easyproto.MessageMarshaler) {
mm.AppendInt64(1, ts.Seconds)
mm.AppendInt32(2, ts.Nanos)
}
func (ts *Timestamp) unmarshalProtobuf(src []byte) (err error) {
// message Timestamp {
// int64 seconds = 1;
// int32 nanos = 2;
// }
var fc easyproto.FieldContext
for len(src) > 0 {
src, err = fc.NextField(src)
if err != nil {
return fmt.Errorf("cannot read next field in Timestamp: %w", err)
}
switch fc.FieldNum {
case 1:
seconds, ok := fc.Int64()
if !ok {
return fmt.Errorf("cannot read Seconds")
}
ts.Seconds = seconds
case 2:
nanos, ok := fc.Int32()
if !ok {
return fmt.Errorf("cannot read Nanos")
}
ts.Nanos = nanos
}
}
return nil
}

File diff suppressed because it is too large Load Diff

View File

@@ -1,38 +0,0 @@
syntax = "proto3";
// source: https://raw.githubusercontent.com/grafana/loki/main/pkg/push/push.proto
// Licensed under the Apache License, Version 2.0 (the "License");
// https://github.com/grafana/loki/blob/main/pkg/push/LICENSE
package logproto;
import "gogoproto/gogo.proto";
import "google/protobuf/timestamp.proto";
option go_package = "github.com/VictoriaMetrics/VictoriaMetrics/app/vlinsert/loki";
message PushRequest {
repeated StreamAdapter streams = 1 [
(gogoproto.jsontag) = "streams",
(gogoproto.customtype) = "Stream"
];
}
message StreamAdapter {
string labels = 1 [(gogoproto.jsontag) = "labels"];
repeated EntryAdapter entries = 2 [
(gogoproto.nullable) = false,
(gogoproto.jsontag) = "entries"
];
// hash contains the original hash of the stream.
uint64 hash = 3 [(gogoproto.jsontag) = "-"];
}
message EntryAdapter {
google.protobuf.Timestamp timestamp = 1 [
(gogoproto.stdtime) = true,
(gogoproto.nullable) = false,
(gogoproto.jsontag) = "ts"
];
string line = 2 [(gogoproto.jsontag) = "line"];
}

View File

@@ -1,110 +0,0 @@
package loki
// source: https://raw.githubusercontent.com/grafana/loki/main/pkg/push/timestamp.go
// Licensed under the Apache License, Version 2.0 (the "License");
// https://github.com/grafana/loki/blob/main/pkg/push/LICENSE
import (
"errors"
"strconv"
"time"
"github.com/gogo/protobuf/types"
)
const (
// Seconds field of the earliest valid Timestamp.
// This is time.Date(1, 1, 1, 0, 0, 0, 0, time.UTC).Unix().
minValidSeconds = -62135596800
// Seconds field just after the latest valid Timestamp.
// This is time.Date(10000, 1, 1, 0, 0, 0, 0, time.UTC).Unix().
maxValidSeconds = 253402300800
)
// validateTimestamp determines whether a Timestamp is valid.
// A valid timestamp represents a time in the range
// [0001-01-01, 10000-01-01) and has a Nanos field
// in the range [0, 1e9).
//
// If the Timestamp is valid, validateTimestamp returns nil.
// Otherwise, it returns an error that describes
// the problem.
//
// Every valid Timestamp can be represented by a time.Time, but the converse is not true.
func validateTimestamp(ts *types.Timestamp) error {
if ts == nil {
return errors.New("timestamp: nil Timestamp")
}
if ts.Seconds < minValidSeconds {
return errors.New("timestamp: " + formatTimestamp(ts) + " before 0001-01-01")
}
if ts.Seconds >= maxValidSeconds {
return errors.New("timestamp: " + formatTimestamp(ts) + " after 10000-01-01")
}
if ts.Nanos < 0 || ts.Nanos >= 1e9 {
return errors.New("timestamp: " + formatTimestamp(ts) + ": nanos not in range [0, 1e9)")
}
return nil
}
// formatTimestamp is equivalent to fmt.Sprintf("%#v", ts)
// but avoids the escape incurred by using fmt.Sprintf, eliminating
// unnecessary heap allocations.
func formatTimestamp(ts *types.Timestamp) string {
if ts == nil {
return "nil"
}
seconds := strconv.FormatInt(ts.Seconds, 10)
nanos := strconv.FormatInt(int64(ts.Nanos), 10)
return "&types.Timestamp{Seconds: " + seconds + ",\nNanos: " + nanos + ",\n}"
}
func sizeOfStdTime(t time.Time) int {
ts, err := timestampProto(t)
if err != nil {
return 0
}
return ts.Size()
}
func stdTimeMarshalTo(t time.Time, data []byte) (int, error) {
ts, err := timestampProto(t)
if err != nil {
return 0, err
}
return ts.MarshalTo(data)
}
func stdTimeUnmarshal(t *time.Time, data []byte) error {
ts := &types.Timestamp{}
if err := ts.Unmarshal(data); err != nil {
return err
}
tt, err := timestampFromProto(ts)
if err != nil {
return err
}
*t = tt
return nil
}
func timestampFromProto(ts *types.Timestamp) (time.Time, error) {
// Don't return the zero value on error, because corresponds to a valid
// timestamp. Instead return whatever time.Unix gives us.
var t time.Time
if ts == nil {
t = time.Unix(0, 0).UTC() // treat nil like the empty Timestamp
} else {
t = time.Unix(ts.Seconds, int64(ts.Nanos)).UTC()
}
return t, validateTimestamp(ts)
}
func timestampProto(t time.Time) (types.Timestamp, error) {
ts := types.Timestamp{
Seconds: t.Unix(),
Nanos: int32(t.Nanosecond()),
}
return ts, validateTimestamp(&ts)
}

View File

@@ -1,481 +0,0 @@
package loki
// source: https://raw.githubusercontent.com/grafana/loki/main/pkg/push/types.go
// Licensed under the Apache License, Version 2.0 (the "License");
// https://github.com/grafana/loki/blob/main/pkg/push/LICENSE
import (
"fmt"
"io"
"time"
)
// Stream contains a unique labels set as a string and a set of entries for it.
// We are not using the proto generated version but this custom one so that we
// can improve serialization see benchmark.
type Stream struct {
Labels string `protobuf:"bytes,1,opt,name=labels,proto3" json:"labels"`
Entries []Entry `protobuf:"bytes,2,rep,name=entries,proto3,customtype=EntryAdapter" json:"entries"`
Hash uint64 `protobuf:"varint,3,opt,name=hash,proto3" json:"-"`
}
// Entry is a log entry with a timestamp.
type Entry struct {
Timestamp time.Time `protobuf:"bytes,1,opt,name=timestamp,proto3,stdtime" json:"ts"`
Line string `protobuf:"bytes,2,opt,name=line,proto3" json:"line"`
}
// Marshal implements the proto.Marshaler interface.
func (m *Stream) Marshal() (dAtA []byte, err error) {
size := m.Size()
dAtA = make([]byte, size)
n, err := m.MarshalToSizedBuffer(dAtA[:size])
if err != nil {
return nil, err
}
return dAtA[:n], nil
}
// MarshalTo marshals m to dst.
func (m *Stream) MarshalTo(dAtA []byte) (int, error) {
size := m.Size()
return m.MarshalToSizedBuffer(dAtA[:size])
}
// MarshalToSizedBuffer marshals m to the sized buffer.
func (m *Stream) MarshalToSizedBuffer(dAtA []byte) (int, error) {
i := len(dAtA)
_ = i
var l int
_ = l
if m.Hash != 0 {
i = encodeVarintPush(dAtA, i, m.Hash)
i--
dAtA[i] = 0x18
}
if len(m.Entries) > 0 {
for iNdEx := len(m.Entries) - 1; iNdEx >= 0; iNdEx-- {
{
size, err := m.Entries[iNdEx].MarshalToSizedBuffer(dAtA[:i])
if err != nil {
return 0, err
}
i -= size
i = encodeVarintPush(dAtA, i, uint64(size))
}
i--
dAtA[i] = 0x12
}
}
if len(m.Labels) > 0 {
i -= len(m.Labels)
copy(dAtA[i:], m.Labels)
i = encodeVarintPush(dAtA, i, uint64(len(m.Labels)))
i--
dAtA[i] = 0xa
}
return len(dAtA) - i, nil
}
// Marshal implements the proto.Marshaler interface.
func (m *Entry) Marshal() (dAtA []byte, err error) {
size := m.Size()
dAtA = make([]byte, size)
n, err := m.MarshalToSizedBuffer(dAtA[:size])
if err != nil {
return nil, err
}
return dAtA[:n], nil
}
// MarshalTo marshals m to dst.
func (m *Entry) MarshalTo(dAtA []byte) (int, error) {
size := m.Size()
return m.MarshalToSizedBuffer(dAtA[:size])
}
// MarshalToSizedBuffer marshals m to the sized buffer.
func (m *Entry) MarshalToSizedBuffer(dAtA []byte) (int, error) {
i := len(dAtA)
_ = i
var l int
_ = l
if len(m.Line) > 0 {
i -= len(m.Line)
copy(dAtA[i:], m.Line)
i = encodeVarintPush(dAtA, i, uint64(len(m.Line)))
i--
dAtA[i] = 0x12
}
n7, err7 := stdTimeMarshalTo(m.Timestamp, dAtA[i-sizeOfStdTime(m.Timestamp):])
if err7 != nil {
return 0, err7
}
i -= n7
i = encodeVarintPush(dAtA, i, uint64(n7))
i--
dAtA[i] = 0xa
return len(dAtA) - i, nil
}
// Unmarshal unmarshals the given data into m.
func (m *Stream) Unmarshal(dAtA []byte) error {
l := len(dAtA)
iNdEx := 0
for iNdEx < l {
preIndex := iNdEx
var wire uint64
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return ErrIntOverflowPush
}
if iNdEx >= l {
return io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
wire |= uint64(b&0x7F) << shift
if b < 0x80 {
break
}
}
fieldNum := int32(wire >> 3)
wireType := int(wire & 0x7)
if wireType == 4 {
return fmt.Errorf("proto: StreamAdapter: wiretype end group for non-group")
}
if fieldNum <= 0 {
return fmt.Errorf("proto: StreamAdapter: illegal tag %d (wire type %d)", fieldNum, wire)
}
switch fieldNum {
case 1:
if wireType != 2 {
return fmt.Errorf("proto: wrong wireType = %d for field Labels", wireType)
}
var stringLen uint64
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return ErrIntOverflowPush
}
if iNdEx >= l {
return io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
stringLen |= uint64(b&0x7F) << shift
if b < 0x80 {
break
}
}
intStringLen := int(stringLen)
if intStringLen < 0 {
return ErrInvalidLengthPush
}
postIndex := iNdEx + intStringLen
if postIndex < 0 {
return ErrInvalidLengthPush
}
if postIndex > l {
return io.ErrUnexpectedEOF
}
m.Labels = string(dAtA[iNdEx:postIndex])
iNdEx = postIndex
case 2:
if wireType != 2 {
return fmt.Errorf("proto: wrong wireType = %d for field Entries", wireType)
}
var msglen int
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return ErrIntOverflowPush
}
if iNdEx >= l {
return io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
msglen |= int(b&0x7F) << shift
if b < 0x80 {
break
}
}
if msglen < 0 {
return ErrInvalidLengthPush
}
postIndex := iNdEx + msglen
if postIndex < 0 {
return ErrInvalidLengthPush
}
if postIndex > l {
return io.ErrUnexpectedEOF
}
m.Entries = append(m.Entries, Entry{})
if err := m.Entries[len(m.Entries)-1].Unmarshal(dAtA[iNdEx:postIndex]); err != nil {
return err
}
iNdEx = postIndex
case 3:
if wireType != 0 {
return fmt.Errorf("proto: wrong wireType = %d for field Hash", wireType)
}
m.Hash = 0
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return ErrIntOverflowPush
}
if iNdEx >= l {
return io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
m.Hash |= uint64(b&0x7F) << shift
if b < 0x80 {
break
}
}
default:
iNdEx = preIndex
skippy, err := skipPush(dAtA[iNdEx:])
if err != nil {
return err
}
if skippy < 0 {
return ErrInvalidLengthPush
}
if (iNdEx + skippy) < 0 {
return ErrInvalidLengthPush
}
if (iNdEx + skippy) > l {
return io.ErrUnexpectedEOF
}
iNdEx += skippy
}
}
if iNdEx > l {
return io.ErrUnexpectedEOF
}
return nil
}
// Unmarshal unmarshals the given data into m.
func (m *Entry) Unmarshal(dAtA []byte) error {
l := len(dAtA)
iNdEx := 0
for iNdEx < l {
preIndex := iNdEx
var wire uint64
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return ErrIntOverflowPush
}
if iNdEx >= l {
return io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
wire |= uint64(b&0x7F) << shift
if b < 0x80 {
break
}
}
fieldNum := int32(wire >> 3)
wireType := int(wire & 0x7)
if wireType == 4 {
return fmt.Errorf("proto: EntryAdapter: wiretype end group for non-group")
}
if fieldNum <= 0 {
return fmt.Errorf("proto: EntryAdapter: illegal tag %d (wire type %d)", fieldNum, wire)
}
switch fieldNum {
case 1:
if wireType != 2 {
return fmt.Errorf("proto: wrong wireType = %d for field Timestamp", wireType)
}
var msglen int
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return ErrIntOverflowPush
}
if iNdEx >= l {
return io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
msglen |= int(b&0x7F) << shift
if b < 0x80 {
break
}
}
if msglen < 0 {
return ErrInvalidLengthPush
}
postIndex := iNdEx + msglen
if postIndex < 0 {
return ErrInvalidLengthPush
}
if postIndex > l {
return io.ErrUnexpectedEOF
}
if err := stdTimeUnmarshal(&m.Timestamp, dAtA[iNdEx:postIndex]); err != nil {
return err
}
iNdEx = postIndex
case 2:
if wireType != 2 {
return fmt.Errorf("proto: wrong wireType = %d for field Line", wireType)
}
var stringLen uint64
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return ErrIntOverflowPush
}
if iNdEx >= l {
return io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
stringLen |= uint64(b&0x7F) << shift
if b < 0x80 {
break
}
}
intStringLen := int(stringLen)
if intStringLen < 0 {
return ErrInvalidLengthPush
}
postIndex := iNdEx + intStringLen
if postIndex < 0 {
return ErrInvalidLengthPush
}
if postIndex > l {
return io.ErrUnexpectedEOF
}
m.Line = string(dAtA[iNdEx:postIndex])
iNdEx = postIndex
default:
iNdEx = preIndex
skippy, err := skipPush(dAtA[iNdEx:])
if err != nil {
return err
}
if skippy < 0 {
return ErrInvalidLengthPush
}
if (iNdEx + skippy) < 0 {
return ErrInvalidLengthPush
}
if (iNdEx + skippy) > l {
return io.ErrUnexpectedEOF
}
iNdEx += skippy
}
}
if iNdEx > l {
return io.ErrUnexpectedEOF
}
return nil
}
// Size returns the size of the serialized Stream.
func (m *Stream) Size() (n int) {
if m == nil {
return 0
}
var l int
_ = l
l = len(m.Labels)
if l > 0 {
n += 1 + l + sovPush(uint64(l))
}
if len(m.Entries) > 0 {
for _, e := range m.Entries {
l = e.Size()
n += 1 + l + sovPush(uint64(l))
}
}
if m.Hash != 0 {
n += 1 + sovPush(m.Hash)
}
return n
}
// Size returns the size of the serialized Entry
func (m *Entry) Size() (n int) {
if m == nil {
return 0
}
var l int
_ = l
l = sizeOfStdTime(m.Timestamp)
n += 1 + l + sovPush(uint64(l))
l = len(m.Line)
if l > 0 {
n += 1 + l + sovPush(uint64(l))
}
return n
}
// Equal returns true if the two Streams are equal.
func (m *Stream) Equal(that interface{}) bool {
if that == nil {
return m == nil
}
that1, ok := that.(*Stream)
if !ok {
that2, ok := that.(Stream)
if ok {
that1 = &that2
} else {
return false
}
}
if that1 == nil {
return m == nil
} else if m == nil {
return false
}
if m.Labels != that1.Labels {
return false
}
if len(m.Entries) != len(that1.Entries) {
return false
}
for i := range m.Entries {
if !m.Entries[i].Equal(that1.Entries[i]) {
return false
}
}
return m.Hash == that1.Hash
}
// Equal returns true if the two Entries are equal.
func (m *Entry) Equal(that interface{}) bool {
if that == nil {
return m == nil
}
that1, ok := that.(*Entry)
if !ok {
that2, ok := that.(Entry)
if ok {
that1 = &that2
} else {
return false
}
}
if that1 == nil {
return m == nil
} else if m == nil {
return false
}
if !m.Timestamp.Equal(that1.Timestamp) {
return false
}
if m.Line != that1.Line {
return false
}
return true
}

View File

@@ -70,9 +70,10 @@ func ProcessHitsRequest(ctx context.Context, w http.ResponseWriter, r *http.Requ
fieldsLimit = 0
}
// Prepare the query
q.AddCountByTimePipe(int64(step), int64(offset), fields)
// Prepare the query for hits count.
q.Optimize()
q.DropAllPipes()
q.AddCountByTimePipe(int64(step), int64(offset), fields)
var mLock sync.Mutex
m := make(map[string]*hitsSeries)

View File

@@ -1,13 +1,13 @@
{
"files": {
"main.css": "./static/css/main.1041c3d4.css",
"main.js": "./static/js/main.8988988c.js",
"main.js": "./static/js/main.62a82f4a.js",
"static/js/685.bebe1265.chunk.js": "./static/js/685.bebe1265.chunk.js",
"static/media/MetricsQL.md": "./static/media/MetricsQL.aaabf95f2c9bf356bde4.md",
"index.html": "./index.html"
},
"entrypoints": [
"static/css/main.1041c3d4.css",
"static/js/main.8988988c.js"
"static/js/main.62a82f4a.js"
]
}

View File

@@ -1 +1 @@
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=5"/><meta name="theme-color" content="#000000"/><meta name="description" content="UI for VictoriaMetrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><script src="./dashboards/index.js" type="module"></script><meta name="twitter:card" content="summary_large_image"><meta name="twitter:image" content="./preview.jpg"><meta name="twitter:title" content="UI for VictoriaMetrics"><meta name="twitter:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta name="twitter:site" content="@VictoriaMetrics"><meta property="og:title" content="Metric explorer for VictoriaMetrics"><meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta property="og:image" content="./preview.jpg"><meta property="og:type" content="website"><script defer="defer" src="./static/js/main.8988988c.js"></script><link href="./static/css/main.1041c3d4.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=5"/><meta name="theme-color" content="#000000"/><meta name="description" content="UI for VictoriaMetrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><script src="./dashboards/index.js" type="module"></script><meta name="twitter:card" content="summary_large_image"><meta name="twitter:image" content="./preview.jpg"><meta name="twitter:title" content="UI for VictoriaMetrics"><meta name="twitter:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta name="twitter:site" content="@VictoriaMetrics"><meta property="og:title" content="Metric explorer for VictoriaMetrics"><meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta property="og:image" content="./preview.jpg"><meta property="og:type" content="website"><script defer="defer" src="./static/js/main.62a82f4a.js"></script><link href="./static/css/main.1041c3d4.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

View File

@@ -80,7 +80,7 @@ func Init() {
// Stop stops vlstorage.
func Stop() {
metrics.UnregisterSet(storageMetrics)
metrics.UnregisterSet(storageMetrics, true)
storageMetrics = nil
strg.MustClose()

View File

@@ -70,8 +70,8 @@ var (
"See also -opentsdbHTTPListenAddr.useProxyProtocol")
opentsdbHTTPUseProxyProtocol = flag.Bool("opentsdbHTTPListenAddr.useProxyProtocol", false, "Whether to use proxy protocol for connections accepted "+
"at -opentsdbHTTPListenAddr . See https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt")
configAuthKey = flagutil.NewPassword("configAuthKey", "Authorization key for accessing /config page. It must be passed via authKey query arg. It overrides httpAuth.* settings.")
reloadAuthKey = flagutil.NewPassword("reloadAuthKey", "Auth key for /-/reload http endpoint. It must be passed via authKey query arg. It overrides httpAuth.* settings.")
configAuthKey = flagutil.NewPassword("configAuthKey", "Authorization key for accessing /config page. It must be passed via authKey query arg. It overrides -httpAuth.*")
reloadAuthKey = flagutil.NewPassword("reloadAuthKey", "Auth key for /-/reload http endpoint. It must be passed via authKey query arg. It overrides -httpAuth.*")
dryRun = flag.Bool("dryRun", false, "Whether to check config files without running vmagent. The following files are checked: "+
"-promscrape.config, -remoteWrite.relabelConfig, -remoteWrite.urlRelabelConfig, -remoteWrite.streamAggr.config . "+
"Unknown config entries aren't allowed in -promscrape.config by default. This can be changed by passing -promscrape.config.strictParse=false command-line flag")
@@ -114,7 +114,7 @@ func main() {
logger.Fatalf("error when checking relabel configs: %s", err)
}
if err := remotewrite.CheckStreamAggrConfigs(); err != nil {
logger.Fatalf("error when checking -remoteWrite.streamAggr.config: %s", err)
logger.Fatalf("error when checking -streamAggr.config and -remoteWrite.streamAggr.config: %s", err)
}
logger.Infof("all the configs are ok; exiting with 0 status code")
return
@@ -434,7 +434,7 @@ func requestHandler(w http.ResponseWriter, r *http.Request) bool {
}
return true
case "/prometheus/config", "/config":
if !httpserver.CheckAuthFlag(w, r, configAuthKey.Get(), "configAuthKey") {
if !httpserver.CheckAuthFlag(w, r, configAuthKey) {
return true
}
promscrapeConfigRequests.Inc()
@@ -443,7 +443,7 @@ func requestHandler(w http.ResponseWriter, r *http.Request) bool {
return true
case "/prometheus/api/v1/status/config", "/api/v1/status/config":
// See https://prometheus.io/docs/prometheus/latest/querying/api/#config
if !httpserver.CheckAuthFlag(w, r, configAuthKey.Get(), "configAuthKey") {
if !httpserver.CheckAuthFlag(w, r, configAuthKey) {
return true
}
promscrapeStatusConfigRequests.Inc()
@@ -453,7 +453,7 @@ func requestHandler(w http.ResponseWriter, r *http.Request) bool {
fmt.Fprintf(w, `{"status":"success","data":{"yaml":%q}}`, bb.B)
return true
case "/prometheus/-/reload", "/-/reload":
if !httpserver.CheckAuthFlag(w, r, reloadAuthKey.Get(), "reloadAuthKey") {
if !httpserver.CheckAuthFlag(w, r, reloadAuthKey) {
return true
}
promscrapeConfigReloadRequests.Inc()

View File

@@ -26,11 +26,11 @@ func TestInsertHandler(t *testing.T) {
req := httptest.NewRequest(http.MethodPost, "/insert/0/api/v1/import/prometheus", bytes.NewBufferString(`{"foo":"bar"}
go_memstats_alloc_bytes_total 1`))
if err := InsertHandler(nil, req); err != nil {
t.Errorf("unxepected error %s", err)
t.Fatalf("unxepected error %s", err)
}
expectedMsg := "cannot unmarshal Prometheus line"
if !strings.Contains(testOutput.String(), expectedMsg) {
t.Errorf("output %q should contain %q", testOutput.String(), expectedMsg)
t.Fatalf("output %q should contain %q", testOutput.String(), expectedMsg)
}
}

View File

@@ -15,8 +15,8 @@ import (
"github.com/VictoriaMetrics/VictoriaMetrics/lib/awsapi"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/httputils"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/netutil"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/persistentqueue"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/promauth"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/common"
@@ -120,7 +120,7 @@ func newHTTPClient(argIdx int, remoteWriteURL, sanitizedURL string, fq *persiste
logger.Fatalf("cannot initialize AWS Config for -remoteWrite.url=%q: %s", remoteWriteURL, err)
}
tr := &http.Transport{
DialContext: httputils.GetStatDialFunc("vmagent_remotewrite"),
DialContext: netutil.NewStatDialFunc("vmagent_remotewrite"),
TLSHandshakeTimeout: tlsHandshakeTimeout.GetOptionalArg(argIdx),
MaxConnsPerHost: 2 * concurrency,
MaxIdleConnsPerHost: 2 * concurrency,

View File

@@ -181,7 +181,7 @@ func (rctx *relabelCtx) reset() {
}
var relabelCtxPool = &sync.Pool{
New: func() interface{} {
New: func() any {
return &relabelCtx{}
},
}

View File

@@ -6,6 +6,7 @@ import (
"net/http"
"net/url"
"path/filepath"
"slices"
"strconv"
"sync"
"sync/atomic"
@@ -88,15 +89,15 @@ var (
"By default there are no limits on samples ingestion rate. See also -remoteWrite.rateLimit")
disableOnDiskQueue = flagutil.NewArrayBool("remoteWrite.disableOnDiskQueue", "Whether to disable storing pending data to -remoteWrite.tmpDataPath "+
"when the configured remote storage systems cannot keep up with the data ingestion rate. See https://docs.victoriametrics.com/vmagent#disabling-on-disk-persistence ."+
"See also -remoteWrite.dropSamplesOnOverload")
dropSamplesOnOverload = flagutil.NewArrayBool("remoteWrite.dropSamplesOnOverload", "Whether to drop samples when -remoteWrite.disableOnDiskQueue is set and if the samples "+
"cannot be pushed into the configured remote storage systems in a timely manner. See https://docs.victoriametrics.com/vmagent#disabling-on-disk-persistence")
"when the remote storage system at the corresponding -remoteWrite.url cannot keep up with the data ingestion rate. "+
"See https://docs.victoriametrics.com/vmagent#disabling-on-disk-persistence . See also -remoteWrite.dropSamplesOnOverload")
dropSamplesOnOverload = flag.Bool("remoteWrite.dropSamplesOnOverload", false, "Whether to drop samples when -remoteWrite.disableOnDiskQueue is set and if the samples "+
"cannot be pushed into the configured -remoteWrite.url systems in a timely manner. See https://docs.victoriametrics.com/vmagent#disabling-on-disk-persistence")
)
var (
// rwctxs contains statically populated entries when -remoteWrite.url is specified.
rwctxs []*remoteWriteCtx
// rwctxsGlobal contains statically populated entries when -remoteWrite.url is specified.
rwctxsGlobal []*remoteWriteCtx
// Data without tenant id is written to defaultAuthToken if -enableMultitenantHandlers is specified.
defaultAuthToken = &auth.Token{}
@@ -109,8 +110,11 @@ var (
StatusCode: http.StatusTooManyRequests,
}
// disableOnDiskQueueAll is set to true if all remoteWrite.urls were configured to disable persistent queue via disableOnDiskQueue
disableOnDiskQueueAll bool
// disableOnDiskQueueAny is set to true if at least a single -remoteWrite.url is configured with -remoteWrite.disableOnDiskQueue
disableOnDiskQueueAny bool
// dropSamplesOnFailureGlobal is set to true if -remoteWrite.dropSamplesOnOverload is set or if multiple -remoteWrite.disableOnDiskQueue options are set.
dropSamplesOnFailureGlobal bool
)
// MultitenancyEnabled returns true if -enableMultitenantHandlers is specified.
@@ -203,28 +207,18 @@ func Init() {
relabelConfigSuccess.Set(1)
relabelConfigTimestamp.Set(fasttime.UnixTimestamp())
sasFile, sasOpts := getStreamAggrOpts(-1)
if sasFile != "" {
sas, err := newStreamAggrConfig(-1, pushToRemoteStoragesDropFailed)
if err != nil {
logger.Fatalf("cannot initialize stream aggregators from -streamAggr.config=%q: %s", sasFile, err)
}
sasGlobal.Store(sas)
} else if sasOpts.DedupInterval > 0 {
deduplicatorGlobal = streamaggr.NewDeduplicator(pushToRemoteStoragesDropFailed, sasOpts.DedupInterval, sasOpts.DropInputLabels, sasOpts.Alias)
}
initStreamAggrConfigGlobal()
if len(*remoteWriteURLs) > 0 {
rwctxs = newRemoteWriteCtxs(nil, *remoteWriteURLs)
}
rwctxsGlobal = newRemoteWriteCtxs(nil, *remoteWriteURLs)
disableOnDiskQueueAll = true
for _, v := range *disableOnDiskQueue {
if !v {
disableOnDiskQueueAll = false
break
}
}
disableOnDiskQueues := []bool(*disableOnDiskQueue)
disableOnDiskQueueAny = slices.Contains(disableOnDiskQueues, true)
// Samples must be dropped if multiple -remoteWrite.disableOnDiskQueue options are configured and at least a single is set to true.
// In this case it is impossible to prevent from sending many duplicates of samples passed to TryPush() to all the configured -remoteWrite.url
// if these samples couldn't be sent to the -remoteWrite.url with the disabled persistent queue. So it is better sending samples
// to the remaining -remoteWrite.url and dropping them on the blocked queue.
dropSamplesOnFailureGlobal = *dropSamplesOnOverload || disableOnDiskQueueAny && len(disableOnDiskQueues) > 1
dropDanglingQueues()
@@ -234,9 +228,9 @@ func Init() {
defer configReloaderWG.Done()
for {
select {
case <-sighupCh:
case <-configReloaderStopCh:
return
case <-sighupCh:
}
reloadRelabelConfigs()
reloadStreamAggrConfigs()
@@ -255,8 +249,8 @@ func dropDanglingQueues() {
// In case if there were many persistent queues with identical *remoteWriteURLs
// the queue with the last index will be dropped.
// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6140
existingQueues := make(map[string]struct{}, len(rwctxs))
for _, rwctx := range rwctxs {
existingQueues := make(map[string]struct{}, len(rwctxsGlobal))
for _, rwctx := range rwctxsGlobal {
existingQueues[rwctx.fq.Dirname()] = struct{}{}
}
@@ -273,7 +267,7 @@ func dropDanglingQueues() {
}
}
if removed > 0 {
logger.Infof("removed %d dangling queues from %q, active queues: %d", removed, *tmpDataPath, len(rwctxs))
logger.Infof("removed %d dangling queues from %q, active queues: %d", removed, *tmpDataPath, len(rwctxsGlobal))
}
}
@@ -382,10 +376,10 @@ func Stop() {
deduplicatorGlobal = nil
}
for _, rwctx := range rwctxs {
for _, rwctx := range rwctxsGlobal {
rwctx.MustStop()
}
rwctxs = nil
rwctxsGlobal = nil
if sl := hourlySeriesLimiter; sl != nil {
sl.MustStop()
@@ -397,6 +391,8 @@ func Stop() {
// PushDropSamplesOnFailure pushes wr to the configured remote storage systems set via -remoteWrite.url
//
// PushDropSamplesOnFailure drops wr samples if they cannot be sent to -remoteWrite.url by any reason.
//
// PushDropSamplesOnFailure can modify wr contents.
func PushDropSamplesOnFailure(at *auth.Token, wr *prompbmarshal.WriteRequest) {
_ = tryPush(at, wr, true)
@@ -409,7 +405,7 @@ func PushDropSamplesOnFailure(at *auth.Token, wr *prompbmarshal.WriteRequest) {
//
// The caller must return ErrQueueFullHTTPRetry to the client, which sends wr, if TryPush returns false.
func TryPush(at *auth.Token, wr *prompbmarshal.WriteRequest) bool {
return tryPush(at, wr, false)
return tryPush(at, wr, dropSamplesOnFailureGlobal)
}
func tryPush(at *auth.Token, wr *prompbmarshal.WriteRequest, forceDropSamplesOnFailure bool) bool {
@@ -426,24 +422,20 @@ func tryPush(at *auth.Token, wr *prompbmarshal.WriteRequest, forceDropSamplesOnF
tenantRctx = getRelabelCtx()
defer putRelabelCtx(tenantRctx)
}
rowsCount := getRowsCount(tss)
// Quick check whether writes to configured remote storage systems are blocked.
// This allows saving CPU time spent on relabeling and block compression
// if some of remote storage systems cannot keep up with the data ingestion rate.
// this shortcut is only applicable if all remote writes have disableOnDiskQueue = true
if disableOnDiskQueueAll {
for _, rwctx := range rwctxs {
if rwctx.fq.IsWriteBlocked() {
rwctx.pushFailures.Inc()
if forceDropSamplesOnFailure || rwctx.dropSamplesOnOverload {
// Just drop samples
rwctx.rowsDroppedOnPushFailure.Add(rowsCount)
continue
}
return false
}
}
rwctxs, ok := getEligibleRemoteWriteCtxs(tss, forceDropSamplesOnFailure)
if !ok {
// At least a single remote write queue is blocked and dropSamplesOnFailure isn't set.
// Return false to the caller, so it could re-send samples again.
return false
}
if len(rwctxs) == 0 {
// All the remote write queues are skipped because they are blocked and dropSamplesOnFailure is set to true.
// Return true to the caller, so it doesn't re-send the samples again.
return true
}
var rctx *relabelCtx
@@ -453,6 +445,7 @@ func tryPush(at *auth.Token, wr *prompbmarshal.WriteRequest, forceDropSamplesOnF
rctx = getRelabelCtx()
defer putRelabelCtx(rctx)
}
rowsCount := getRowsCount(tss)
globalRowsPushedBeforeRelabel.Add(rowsCount)
maxSamplesPerBlock := *maxRowsPerBlock
// Allow up to 10x of labels per each block on average.
@@ -505,20 +498,46 @@ func tryPush(at *auth.Token, wr *prompbmarshal.WriteRequest, forceDropSamplesOnF
deduplicatorGlobal.Push(tssBlock)
tssBlock = tssBlock[:0]
}
if !tryPushBlockToRemoteStorages(tssBlock, forceDropSamplesOnFailure) {
if !tryPushBlockToRemoteStorages(rwctxs, tssBlock, forceDropSamplesOnFailure) {
return false
}
}
return true
}
func pushToRemoteStoragesDropFailed(tss []prompbmarshal.TimeSeries) {
if tryPushBlockToRemoteStorages(tss, true) {
func getEligibleRemoteWriteCtxs(tss []prompbmarshal.TimeSeries, forceDropSamplesOnFailure bool) ([]*remoteWriteCtx, bool) {
if !disableOnDiskQueueAny {
return rwctxsGlobal, true
}
// This code is applicable if at least a single remote storage has -disableOnDiskQueue
rwctxs := make([]*remoteWriteCtx, 0, len(rwctxsGlobal))
for _, rwctx := range rwctxsGlobal {
if !rwctx.fq.IsWriteBlocked() {
rwctxs = append(rwctxs, rwctx)
} else {
rwctx.pushFailures.Inc()
if !forceDropSamplesOnFailure {
return nil, false
}
rowsCount := getRowsCount(tss)
rwctx.rowsDroppedOnPushFailure.Add(rowsCount)
}
}
return rwctxs, true
}
func pushToRemoteStoragesTrackDropped(tss []prompbmarshal.TimeSeries) {
rwctxs, _ := getEligibleRemoteWriteCtxs(tss, true)
if len(rwctxs) == 0 {
return
}
if !tryPushBlockToRemoteStorages(rwctxs, tss, true) {
logger.Panicf("BUG: tryPushBlockToRemoteStorages() must return true when forceDropSamplesOnFailure=true")
}
}
func tryPushBlockToRemoteStorages(tssBlock []prompbmarshal.TimeSeries, forceDropSamplesOnFailure bool) bool {
func tryPushBlockToRemoteStorages(rwctxs []*remoteWriteCtx, tssBlock []prompbmarshal.TimeSeries, forceDropSamplesOnFailure bool) bool {
if len(tssBlock) == 0 {
// Nothing to push
return true
@@ -537,7 +556,7 @@ func tryPushBlockToRemoteStorages(tssBlock []prompbmarshal.TimeSeries, forceDrop
if replicas <= 0 {
replicas = 1
}
return tryShardingBlockAmongRemoteStorages(tssBlock, replicas, forceDropSamplesOnFailure)
return tryShardingBlockAmongRemoteStorages(rwctxs, tssBlock, replicas, forceDropSamplesOnFailure)
}
// Replicate tssBlock samples among rwctxs.
@@ -558,7 +577,7 @@ func tryPushBlockToRemoteStorages(tssBlock []prompbmarshal.TimeSeries, forceDrop
return !anyPushFailed.Load()
}
func tryShardingBlockAmongRemoteStorages(tssBlock []prompbmarshal.TimeSeries, replicas int, forceDropSamplesOnFailure bool) bool {
func tryShardingBlockAmongRemoteStorages(rwctxs []*remoteWriteCtx, tssBlock []prompbmarshal.TimeSeries, replicas int, forceDropSamplesOnFailure bool) bool {
x := getTSSShards(len(rwctxs))
defer putTSSShards(x)
@@ -745,10 +764,8 @@ type remoteWriteCtx struct {
sas atomic.Pointer[streamaggr.Aggregators]
deduplicator *streamaggr.Deduplicator
streamAggrKeepInput bool
streamAggrDropInput bool
disableOnDiskQueue bool
dropSamplesOnOverload bool
streamAggrKeepInput bool
streamAggrDropInput bool
pss []*pendingSeries
pssNextIdx atomic.Uint64
@@ -773,6 +790,7 @@ func newRemoteWriteCtx(argIdx int, remoteWriteURL *url.URL, maxInmemoryBlocks in
logger.Warnf("rounding the -remoteWrite.maxDiskUsagePerURL=%d to the minimum supported value: %d", maxPendingBytes, persistentqueue.DefaultChunkFileSize)
maxPendingBytes = persistentqueue.DefaultChunkFileSize
}
isPQDisabled := disableOnDiskQueue.GetOptionalArg(argIdx)
fq := persistentqueue.MustOpenFastQueue(queuePath, sanitizedURL, maxInmemoryBlocks, maxPendingBytes, isPQDisabled)
_ = metrics.GetOrCreateGauge(fmt.Sprintf(`vmagent_remotewrite_pending_data_bytes{path=%q, url=%q}`, queuePath, sanitizedURL), func() float64 {
@@ -817,31 +835,13 @@ func newRemoteWriteCtx(argIdx int, remoteWriteURL *url.URL, maxInmemoryBlocks in
c: c,
pss: pss,
dropSamplesOnOverload: dropSamplesOnOverload.GetOptionalArg(argIdx),
disableOnDiskQueue: isPQDisabled,
rowsPushedAfterRelabel: metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_remotewrite_rows_pushed_after_relabel_total{path=%q,url=%q}`, queuePath, sanitizedURL)),
rowsDroppedByRelabel: metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_remotewrite_relabel_metrics_dropped_total{path=%q,url=%q}`, queuePath, sanitizedURL)),
rowsPushedAfterRelabel: metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_remotewrite_rows_pushed_after_relabel_total{path=%q, url=%q}`, queuePath, sanitizedURL)),
rowsDroppedByRelabel: metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_remotewrite_relabel_metrics_dropped_total{path=%q, url=%q}`, queuePath, sanitizedURL)),
pushFailures: metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_remotewrite_push_failures_total{path=%q, url=%q}`, queuePath, sanitizedURL)),
rowsDroppedOnPushFailure: metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_remotewrite_samples_dropped_total{path=%q, url=%q}`, queuePath, sanitizedURL)),
}
// Initialize sas
sasFile, sasOpts := getStreamAggrOpts(argIdx)
if sasFile != "" {
sas, err := newStreamAggrConfig(argIdx, rwctx.pushInternalTrackDropped)
if err != nil {
logger.Fatalf("cannot initialize stream aggregators from -remoteWrite.streamAggr.config=%q: %s", sasFile, err)
}
rwctx.sas.Store(sas)
rwctx.streamAggrKeepInput = streamAggrKeepInput.GetOptionalArg(argIdx)
rwctx.streamAggrDropInput = streamAggrDropInput.GetOptionalArg(argIdx)
metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_streamaggr_config_reload_successful{path=%q}`, sasFile)).Set(1)
metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_streamaggr_config_reload_success_timestamp_seconds{path=%q}`, sasFile)).Set(fasttime.UnixTimestamp())
} else if sasOpts.DedupInterval > 0 {
rwctx.deduplicator = streamaggr.NewDeduplicator(rwctx.pushInternalTrackDropped, sasOpts.DedupInterval, sasOpts.DropInputLabels, sasOpts.Alias)
pushFailures: metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_remotewrite_push_failures_total{path=%q,url=%q}`, queuePath, sanitizedURL)),
rowsDroppedOnPushFailure: metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_remotewrite_samples_dropped_total{path=%q,url=%q}`, queuePath, sanitizedURL)),
}
rwctx.initStreamAggrConfig()
return rwctx
}
@@ -934,8 +934,9 @@ func (rwctx *remoteWriteCtx) TryPush(tss []prompbmarshal.TimeSeries, forceDropSa
// Couldn't push tss to remote storage
rwctx.pushFailures.Inc()
if forceDropSamplesOnFailure || rwctx.dropSamplesOnOverload {
rwctx.rowsDroppedOnPushFailure.Add(len(tss))
if forceDropSamplesOnFailure {
rowsCount := getRowsCount(tss)
rwctx.rowsDroppedOnPushFailure.Add(rowsCount)
return true
}
return false
@@ -962,14 +963,12 @@ func (rwctx *remoteWriteCtx) pushInternalTrackDropped(tss []prompbmarshal.TimeSe
if rwctx.tryPushInternal(tss) {
return
}
if !rwctx.disableOnDiskQueue {
if !rwctx.fq.IsPersistentQueueDisabled() {
logger.Panicf("BUG: tryPushInternal must return true if -remoteWrite.disableOnDiskQueue isn't set")
}
rwctx.pushFailures.Inc()
if dropSamplesOnOverload.GetOptionalArg(rwctx.idx) {
rowsCount := getRowsCount(tss)
rwctx.rowsDroppedOnPushFailure.Add(rowsCount)
}
rowsCount := getRowsCount(tss)
rwctx.rowsDroppedOnPushFailure.Add(rowsCount)
}
func (rwctx *remoteWriteCtx) tryPushInternal(tss []prompbmarshal.TimeSeries) bool {
@@ -1000,7 +999,7 @@ func (rwctx *remoteWriteCtx) tryPushInternal(tss []prompbmarshal.TimeSeries) boo
}
var tssPool = &sync.Pool{
New: func() interface{} {
New: func() any {
a := []prompbmarshal.TimeSeries{}
return &a
},

View File

@@ -77,14 +77,16 @@ func TestRemoteWriteContext_TryPush_ImmutableTimeseries(t *testing.T) {
rowsDroppedByRelabel: metrics.GetOrCreateCounter(`bar`),
}
if dedupInterval > 0 {
rwctx.deduplicator = streamaggr.NewDeduplicator(nil, dedupInterval, nil, "global")
rwctx.deduplicator = streamaggr.NewDeduplicator(nil, dedupInterval, nil, "dedup-global")
}
if len(streamAggrConfig) > 0 {
sas, err := streamaggr.LoadFromData([]byte(streamAggrConfig), nil, streamaggr.Options{})
if streamAggrConfig != "" {
pushNoop := func(_ []prompbmarshal.TimeSeries) {}
sas, err := streamaggr.LoadFromData([]byte(streamAggrConfig), pushNoop, nil, "global")
if err != nil {
t.Fatalf("cannot load streamaggr configs: %s", err)
}
defer sas.MustStop()
rwctx.sas.Store(sas)
}
@@ -94,7 +96,9 @@ func TestRemoteWriteContext_TryPush_ImmutableTimeseries(t *testing.T) {
// copy inputTss to make sure it is not mutated during TryPush call
copy(expectedTss, inputTss)
rwctx.TryPush(inputTss, false)
if !rwctx.TryPush(inputTss, false) {
t.Fatalf("cannot push samples to rwctx")
}
if !reflect.DeepEqual(expectedTss, inputTss) {
t.Fatalf("unexpected samples;\ngot\n%v\nwant\n%v", inputTss, expectedTss)

View File

@@ -61,104 +61,180 @@ var (
// CheckStreamAggrConfigs checks -remoteWrite.streamAggr.config and -streamAggr.config.
func CheckStreamAggrConfigs() error {
pushNoop := func(_ []prompbmarshal.TimeSeries) {}
// Check global config
sas, err := newStreamAggrConfigGlobal()
if err != nil {
return err
}
sas.MustStop()
if _, err := newStreamAggrConfig(-1, pushNoop); err != nil {
return fmt.Errorf("could not load -streamAggr.config stream aggregation config: %w", err)
}
if len(*streamAggrConfig) > len(*remoteWriteURLs) {
return fmt.Errorf("too many -remoteWrite.streamAggr.config args: %d; it mustn't exceed the number of -remoteWrite.url args: %d",
len(*streamAggrConfig), len(*remoteWriteURLs))
return fmt.Errorf("too many -remoteWrite.streamAggr.config args: %d; it mustn't exceed the number of -remoteWrite.url args: %d", len(*streamAggrConfig), len(*remoteWriteURLs))
}
pushNoop := func(_ []prompbmarshal.TimeSeries) {}
for idx := range *streamAggrConfig {
if _, err := newStreamAggrConfig(idx, pushNoop); err != nil {
sas, err := newStreamAggrConfigPerURL(idx, pushNoop)
if err != nil {
return err
}
sas.MustStop()
}
return nil
}
func reloadStreamAggrConfigs() {
reloadStreamAggrConfig(-1, pushToRemoteStoragesDropFailed)
for idx, rwctx := range rwctxs {
reloadStreamAggrConfig(idx, rwctx.pushInternalTrackDropped)
reloadStreamAggrConfigGlobal()
for _, rwctx := range rwctxsGlobal {
rwctx.reloadStreamAggrConfig()
}
}
func reloadStreamAggrConfig(idx int, pushFunc streamaggr.PushFunc) {
path, opts := getStreamAggrOpts(idx)
logger.Infof("reloading stream aggregation configs pointed by -remoteWrite.streamAggr.config=%q", path)
metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_streamaggr_config_reloads_total{path=%q}`, path)).Inc()
sasNew, err := newStreamAggrConfigWithOpts(pushFunc, path, opts)
if err != nil {
metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_streamaggr_config_reloads_errors_total{path=%q}`, path)).Inc()
metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_streamaggr_config_reload_successful{path=%q}`, path)).Set(0)
logger.Errorf("cannot reload stream aggregation config at %q; continue using the previously loaded config; error: %s", path, err)
func reloadStreamAggrConfigGlobal() {
path := *streamAggrGlobalConfig
if path == "" {
return
}
var sas *streamaggr.Aggregators
if idx < 0 {
sas = sasGlobal.Load()
} else {
sas = rwctxs[idx].sas.Load()
logger.Infof("reloading stream aggregation configs pointed by -streamAggr.config=%q", path)
metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_streamaggr_config_reloads_total{path=%q}`, path)).Inc()
sasNew, err := newStreamAggrConfigGlobal()
if err != nil {
metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_streamaggr_config_reloads_errors_total{path=%q}`, path)).Inc()
metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_streamaggr_config_reload_successful{path=%q}`, path)).Set(0)
logger.Errorf("cannot reload -streamAggr.config=%q; continue using the previously loaded config; error: %s", path, err)
return
}
sas := sasGlobal.Load()
if !sasNew.Equal(sas) {
var sasOld *streamaggr.Aggregators
if idx < 0 {
sasOld = sasGlobal.Swap(sasNew)
} else {
sasOld = rwctxs[idx].sas.Swap(sasNew)
}
sasOld := sasGlobal.Swap(sasNew)
sasOld.MustStop()
logger.Infof("successfully reloaded stream aggregation configs at %q", path)
logger.Infof("successfully reloaded -streamAggr.config=%q", path)
} else {
sasNew.MustStop()
logger.Infof("successfully reloaded stream aggregation configs at %q", path)
logger.Infof("-streamAggr.config=%q wasn't changed since the last reload", path)
}
metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_streamaggr_config_reload_successful{path=%q}`, path)).Set(1)
metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_streamaggr_config_reload_success_timestamp_seconds{path=%q}`, path)).Set(fasttime.UnixTimestamp())
}
func getStreamAggrOpts(idx int) (string, streamaggr.Options) {
if idx < 0 {
return *streamAggrGlobalConfig, streamaggr.Options{
DedupInterval: streamAggrGlobalDedupInterval.Duration(),
DropInputLabels: *streamAggrGlobalDropInputLabels,
IgnoreOldSamples: *streamAggrGlobalIgnoreOldSamples,
IgnoreFirstIntervals: *streamAggrGlobalIgnoreFirstIntervals,
Alias: "global",
func initStreamAggrConfigGlobal() {
sas, err := newStreamAggrConfigGlobal()
if err != nil {
logger.Fatalf("cannot initialize gloabl stream aggregators: %s", err)
}
if sas != nil {
filePath := sas.FilePath()
sasGlobal.Store(sas)
metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_streamaggr_config_reload_successful{path=%q}`, filePath)).Set(1)
metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_streamaggr_config_reload_success_timestamp_seconds{path=%q}`, filePath)).Set(fasttime.UnixTimestamp())
} else {
dedupInterval := streamAggrGlobalDedupInterval.Duration()
if dedupInterval > 0 {
deduplicatorGlobal = streamaggr.NewDeduplicator(pushToRemoteStoragesTrackDropped, dedupInterval, *streamAggrDropInputLabels, "dedup-global")
}
}
url := fmt.Sprintf("%d:secret-url", idx+1)
if *showRemoteWriteURL {
url = fmt.Sprintf("%d:%s", idx+1, remoteWriteURLs.GetOptionalArg(idx))
}
func (rwctx *remoteWriteCtx) initStreamAggrConfig() {
idx := rwctx.idx
sas, err := rwctx.newStreamAggrConfig()
if err != nil {
logger.Fatalf("cannot initialize stream aggregators: %s", err)
}
opts := streamaggr.Options{
if sas != nil {
filePath := sas.FilePath()
rwctx.sas.Store(sas)
rwctx.streamAggrKeepInput = streamAggrKeepInput.GetOptionalArg(idx)
rwctx.streamAggrDropInput = streamAggrDropInput.GetOptionalArg(idx)
metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_streamaggr_config_reload_successful{path=%q}`, filePath)).Set(1)
metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_streamaggr_config_reload_success_timestamp_seconds{path=%q}`, filePath)).Set(fasttime.UnixTimestamp())
} else {
dedupInterval := streamAggrDedupInterval.GetOptionalArg(idx)
if dedupInterval > 0 {
alias := fmt.Sprintf("dedup-%d", idx+1)
rwctx.deduplicator = streamaggr.NewDeduplicator(rwctx.pushInternalTrackDropped, dedupInterval, *streamAggrDropInputLabels, alias)
}
}
}
func (rwctx *remoteWriteCtx) reloadStreamAggrConfig() {
path := streamAggrConfig.GetOptionalArg(rwctx.idx)
if path == "" {
return
}
logger.Infof("reloading stream aggregation configs pointed by -remoteWrite.streamAggr.config=%q", path)
metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_streamaggr_config_reloads_total{path=%q}`, path)).Inc()
sasNew, err := rwctx.newStreamAggrConfig()
if err != nil {
metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_streamaggr_config_reloads_errors_total{path=%q}`, path)).Inc()
metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_streamaggr_config_reload_successful{path=%q}`, path)).Set(0)
logger.Errorf("cannot reload -remoteWrite.streamAggr.config=%q; continue using the previously loaded config; error: %s", path, err)
return
}
sas := rwctx.sas.Load()
if !sasNew.Equal(sas) {
sasOld := rwctx.sas.Swap(sasNew)
sasOld.MustStop()
logger.Infof("successfully reloaded -remoteWrite.streamAggr.config=%q", path)
} else {
sasNew.MustStop()
logger.Infof("-remoteWrite.streamAggr.config=%q wasn't changed since the last reload", path)
}
metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_streamaggr_config_reload_successful{path=%q}`, path)).Set(1)
metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_streamaggr_config_reload_success_timestamp_seconds{path=%q}`, path)).Set(fasttime.UnixTimestamp())
}
func newStreamAggrConfigGlobal() (*streamaggr.Aggregators, error) {
path := *streamAggrGlobalConfig
if path == "" {
return nil, nil
}
opts := &streamaggr.Options{
DedupInterval: streamAggrGlobalDedupInterval.Duration(),
DropInputLabels: *streamAggrGlobalDropInputLabels,
IgnoreOldSamples: *streamAggrGlobalIgnoreOldSamples,
IgnoreFirstIntervals: *streamAggrGlobalIgnoreFirstIntervals,
}
sas, err := streamaggr.LoadFromFile(path, pushToRemoteStoragesTrackDropped, opts, "global")
if err != nil {
return nil, fmt.Errorf("cannot load -streamAggr.config=%q: %w", *streamAggrGlobalConfig, err)
}
return sas, nil
}
func (rwctx *remoteWriteCtx) newStreamAggrConfig() (*streamaggr.Aggregators, error) {
return newStreamAggrConfigPerURL(rwctx.idx, rwctx.pushInternalTrackDropped)
}
func newStreamAggrConfigPerURL(idx int, pushFunc streamaggr.PushFunc) (*streamaggr.Aggregators, error) {
path := streamAggrConfig.GetOptionalArg(idx)
if path == "" {
return nil, nil
}
alias := fmt.Sprintf("%d:secret-url", idx+1)
if *showRemoteWriteURL {
alias = fmt.Sprintf("%d:%s", idx+1, remoteWriteURLs.GetOptionalArg(idx))
}
opts := &streamaggr.Options{
DedupInterval: streamAggrDedupInterval.GetOptionalArg(idx),
DropInputLabels: *streamAggrDropInputLabels,
IgnoreOldSamples: streamAggrIgnoreOldSamples.GetOptionalArg(idx),
IgnoreFirstIntervals: *streamAggrIgnoreFirstIntervals,
Alias: url,
}
if len(*streamAggrConfig) == 0 {
return "", opts
sas, err := streamaggr.LoadFromFile(path, pushFunc, opts, alias)
if err != nil {
return nil, fmt.Errorf("cannot load -remoteWrite.streamAggr.config=%q: %w", path, err)
}
return streamAggrConfig.GetOptionalArg(idx), opts
}
func newStreamAggrConfigWithOpts(pushFunc streamaggr.PushFunc, path string, opts streamaggr.Options) (*streamaggr.Aggregators, error) {
if len(path) == 0 {
// Skip empty stream aggregation config.
return nil, nil
}
return streamaggr.LoadFromFile(path, pushFunc, opts)
}
func newStreamAggrConfig(idx int, pushFunc streamaggr.PushFunc) (*streamaggr.Aggregators, error) {
path, opts := getStreamAggrOpts(idx)
return newStreamAggrConfigWithOpts(pushFunc, path, opts)
return sas, nil
}

View File

@@ -81,6 +81,9 @@ vmalert-tool-linux-ppc64le:
vmalert-tool-linux-s390x:
APP_NAME=vmalert-tool CGO_ENABLED=0 GOOS=linux GOARCH=s390x $(MAKE) app-local-goos-goarch
vmalert-tool-linux-loong64:
APP_NAME=vmalert-tool CGO_ENABLED=0 GOOS=linux GOARCH=loong64 $(MAKE) app-local-goos-goarch
vmalert-tool-linux-386:
APP_NAME=vmalert-tool CGO_ENABLED=0 GOOS=linux GOARCH=386 $(MAKE) app-local-goos-goarch

View File

@@ -74,10 +74,7 @@ func writeInputSeries(input []series, interval *promutils.Duration, startStamp t
r.Timeseries = append(r.Timeseries, testutil.TimeSeries{Labels: ls, Samples: samples})
}
data, err := testutil.Compress(r)
if err != nil {
return fmt.Errorf("failed to compress data: %v", err)
}
data := testutil.Compress(r)
// write input series to vm
httpWrite(dst, bytes.NewBuffer(data))
vmstorage.Storage.DebugFlush()

View File

@@ -6,88 +6,61 @@ import (
"github.com/VictoriaMetrics/VictoriaMetrics/lib/decimal"
)
func TestParseInputValue(t *testing.T) {
testCases := []struct {
input string
exp []sequenceValue
failed bool
}{
{
"",
nil,
true,
},
{
"testfailed",
nil,
true,
},
// stale doesn't support operations
{
"stalex3",
nil,
true,
},
{
"-4",
[]sequenceValue{{Value: -4}},
false,
},
{
"_",
[]sequenceValue{{Omitted: true}},
false,
},
{
"stale",
[]sequenceValue{{Value: decimal.StaleNaN}},
false,
},
{
"-4x1",
[]sequenceValue{{Value: -4}, {Value: -4}},
false,
},
{
"_x1",
[]sequenceValue{{Omitted: true}},
false,
},
{
"1+1x2 0.1 0.1+0.3x2 3.14",
[]sequenceValue{{Value: 1}, {Value: 2}, {Value: 3}, {Value: 0.1}, {Value: 0.1}, {Value: 0.4}, {Value: 0.7}, {Value: 3.14}},
false,
},
{
"2-1x4",
[]sequenceValue{{Value: 2}, {Value: 1}, {Value: 0}, {Value: -1}, {Value: -2}},
false,
},
{
"1+1x1 _ -4 stale 3+20x1",
[]sequenceValue{{Value: 1}, {Value: 2}, {Omitted: true}, {Value: -4}, {Value: decimal.StaleNaN}, {Value: 3}, {Value: 23}},
false,
},
func TestParseInputValue_Failure(t *testing.T) {
f := func(input string) {
t.Helper()
_, err := parseInputValue(input, true)
if err == nil {
t.Fatalf("expecting non-nil error")
}
}
for _, tc := range testCases {
output, err := parseInputValue(tc.input, true)
if err != nil != tc.failed {
t.Fatalf("failed to parse %s, expect %t, got %t", tc.input, tc.failed, err != nil)
f("")
f("testfailed")
// stale doesn't support operations
f("stalex3")
}
func TestParseInputValue_Success(t *testing.T) {
f := func(input string, outputExpected []sequenceValue) {
t.Helper()
output, err := parseInputValue(input, true)
if err != nil {
t.Fatalf("unexpected error in parseInputValue: %s", err)
}
if len(tc.exp) != len(output) {
t.Fatalf("expect %v, got %v", tc.exp, output)
if len(outputExpected) != len(output) {
t.Fatalf("unexpected output length; got %d; want %d", len(outputExpected), len(output))
}
for i := 0; i < len(tc.exp); i++ {
if tc.exp[i].Omitted != output[i].Omitted {
t.Fatalf("expect %v, got %v", tc.exp, output)
for i := 0; i < len(outputExpected); i++ {
if outputExpected[i].Omitted != output[i].Omitted {
t.Fatalf("unexpected Omitted field in the output\ngot\n%v\nwant\n%v", output, outputExpected)
}
if tc.exp[i].Value != output[i].Value {
if decimal.IsStaleNaN(tc.exp[i].Value) && decimal.IsStaleNaN(output[i].Value) {
if outputExpected[i].Value != output[i].Value {
if decimal.IsStaleNaN(outputExpected[i].Value) && decimal.IsStaleNaN(output[i].Value) {
continue
}
t.Fatalf("expect %v, got %v", tc.exp, output)
t.Fatalf("unexpeccted Value field in the output\ngot\n%v\nwant\n%v", output, outputExpected)
}
}
}
f("-4", []sequenceValue{{Value: -4}})
f("_", []sequenceValue{{Omitted: true}})
f("stale", []sequenceValue{{Value: decimal.StaleNaN}})
f("-4x1", []sequenceValue{{Value: -4}, {Value: -4}})
f("_x1", []sequenceValue{{Omitted: true}})
f("1+1x2 0.1 0.1+0.3x2 3.14", []sequenceValue{{Value: 1}, {Value: 2}, {Value: 3}, {Value: 0.1}, {Value: 0.1}, {Value: 0.4}, {Value: 0.7}, {Value: 3.14}})
f("2-1x4", []sequenceValue{{Value: 2}, {Value: 1}, {Value: 0}, {Value: -1}, {Value: -2}})
f("1+1x1 _ -4 stale 3+20x1", []sequenceValue{{Value: 1}, {Value: 2}, {Omitted: true}, {Value: -4}, {Value: decimal.StaleNaN}, {Value: 3}, {Value: 23}})
}

View File

@@ -14,34 +14,33 @@ func TestMain(m *testing.M) {
os.Exit(m.Run())
}
func TestUnitRule(t *testing.T) {
testCases := []struct {
name string
disableGroupLabel bool
files []string
failed bool
}{
{
name: "run multi files",
files: []string{"./testdata/test1.yaml", "./testdata/test2.yaml"},
failed: false,
},
{
name: "disable group label",
disableGroupLabel: true,
files: []string{"./testdata/disable-group-label.yaml"},
failed: false,
},
{
name: "failing test",
files: []string{"./testdata/failed-test.yaml"},
failed: true,
},
}
for _, tc := range testCases {
fail := UnitTest(tc.files, tc.disableGroupLabel)
if fail != tc.failed {
t.Fatalf("failed to test %s, expect %t, got %t", tc.name, tc.failed, fail)
func TestUnitTest_Failure(t *testing.T) {
f := func(files []string) {
t.Helper()
failed := UnitTest(files, false)
if !failed {
t.Fatalf("expecting failed test")
}
}
// failing test
f([]string{"./testdata/failed-test.yaml"})
}
func TestUnitTest_Success(t *testing.T) {
f := func(disableGroupLabel bool, files []string) {
t.Helper()
failed := UnitTest(files, disableGroupLabel)
if failed {
t.Fatalf("unexpected failed test")
}
}
// run multi files
f(false, []string{"./testdata/test1.yaml", "./testdata/test2.yaml"})
// disable group label
f(true, []string{"./testdata/disable-group-label.yaml"})
}

View File

@@ -45,11 +45,11 @@ type Group struct {
// EvalAlignment will make the timestamp of group query requests be aligned with interval
EvalAlignment *bool `yaml:"eval_alignment,omitempty"`
// Catches all undefined fields and must be empty after parsing.
XXX map[string]interface{} `yaml:",inline"`
XXX map[string]any `yaml:",inline"`
}
// UnmarshalYAML implements the yaml.Unmarshaler interface.
func (g *Group) UnmarshalYAML(unmarshal func(interface{}) error) error {
func (g *Group) UnmarshalYAML(unmarshal func(any) error) error {
type group Group
if err := unmarshal((*group)(g)); err != nil {
return err
@@ -142,11 +142,11 @@ type Rule struct {
UpdateEntriesLimit *int `yaml:"update_entries_limit,omitempty"`
// Catches all undefined fields and must be empty after parsing.
XXX map[string]interface{} `yaml:",inline"`
XXX map[string]any `yaml:",inline"`
}
// UnmarshalYAML implements the yaml.Unmarshaler interface.
func (r *Rule) UnmarshalYAML(unmarshal func(interface{}) error) error {
func (r *Rule) UnmarshalYAML(unmarshal func(any) error) error {
type rule Rule
if err := unmarshal((*rule)(r)); err != nil {
return err
@@ -301,7 +301,7 @@ func parseConfig(data []byte) ([]Group, error) {
g := struct {
Groups []Group `yaml:"groups"`
// Catches all undefined fields and must be empty after parsing.
XXX map[string]interface{} `yaml:",inline"`
XXX map[string]any `yaml:",inline"`
}{}
err = yaml.Unmarshal(data, &g)
if err != nil {
@@ -310,7 +310,7 @@ func parseConfig(data []byte) ([]Group, error) {
return g.Groups, checkOverflow(g.XXX, "config")
}
func checkOverflow(m map[string]interface{}, ctx string) error {
func checkOverflow(m map[string]any, ctx string) error {
if len(m) > 0 {
var keys []string
for k := range m {

View File

@@ -23,12 +23,6 @@ func TestMain(m *testing.M) {
os.Exit(m.Run())
}
func TestParseGood(t *testing.T) {
if _, err := Parse([]string{"testdata/rules/*good.rules", "testdata/dir/*good.*"}, notifier.ValidateTemplates, true); err != nil {
t.Errorf("error parsing files %s", err)
}
}
func TestParseFromURL(t *testing.T) {
mux := http.NewServeMux()
mux.HandleFunc("/bad", func(w http.ResponseWriter, _ *http.Request) {
@@ -55,438 +49,353 @@ groups:
defer srv.Close()
if _, err := Parse([]string{srv.URL + "/good-alert", srv.URL + "/good-rr"}, notifier.ValidateTemplates, true); err != nil {
t.Errorf("error parsing URLs %s", err)
t.Fatalf("error parsing URLs %s", err)
}
if _, err := Parse([]string{srv.URL + "/bad"}, notifier.ValidateTemplates, true); err == nil {
t.Errorf("expected parsing error: %s", err)
t.Fatalf("expected parsing error: %s", err)
}
}
func TestParseBad(t *testing.T) {
testCases := []struct {
path []string
expErr string
}{
{
[]string{"testdata/rules/rules_interval_bad.rules"},
"eval_offset should be smaller than interval",
},
{
[]string{"testdata/rules/rules0-bad.rules"},
"unexpected token",
},
{
[]string{"testdata/dir/rules0-bad.rules"},
"error parsing annotation",
},
{
[]string{"testdata/dir/rules1-bad.rules"},
"duplicate in file",
},
{
[]string{"testdata/dir/rules2-bad.rules"},
"function \"unknown\" not defined",
},
{
[]string{"testdata/dir/rules3-bad.rules"},
"either `record` or `alert` must be set",
},
{
[]string{"testdata/dir/rules4-bad.rules"},
"either `record` or `alert` must be set",
},
{
[]string{"testdata/rules/rules1-bad.rules"},
"bad graphite expr",
},
{
[]string{"testdata/dir/rules6-bad.rules"},
"missing ':' in header",
},
{
[]string{"http://unreachable-url"},
"failed to",
},
func TestParse_Success(t *testing.T) {
_, err := Parse([]string{"testdata/rules/*good.rules", "testdata/dir/*good.*"}, notifier.ValidateTemplates, true)
if err != nil {
t.Fatalf("error parsing files %s", err)
}
for _, tc := range testCases {
_, err := Parse(tc.path, notifier.ValidateTemplates, true)
}
func TestParse_Failure(t *testing.T) {
f := func(paths []string, errStrExpected string) {
t.Helper()
_, err := Parse(paths, notifier.ValidateTemplates, true)
if err == nil {
t.Errorf("expected to get error")
return
t.Fatalf("expected to get error")
}
if !strings.Contains(err.Error(), tc.expErr) {
t.Errorf("expected err to contain %q; got %q instead", tc.expErr, err)
if !strings.Contains(err.Error(), errStrExpected) {
t.Fatalf("expected err to contain %q; got %q instead", errStrExpected, err)
}
}
f([]string{"testdata/rules/rules_interval_bad.rules"}, "eval_offset should be smaller than interval")
f([]string{"testdata/rules/rules0-bad.rules"}, "unexpected token")
f([]string{"testdata/dir/rules0-bad.rules"}, "error parsing annotation")
f([]string{"testdata/dir/rules1-bad.rules"}, "duplicate in file")
f([]string{"testdata/dir/rules2-bad.rules"}, "function \"unknown\" not defined")
f([]string{"testdata/dir/rules3-bad.rules"}, "either `record` or `alert` must be set")
f([]string{"testdata/dir/rules4-bad.rules"}, "either `record` or `alert` must be set")
f([]string{"testdata/rules/rules1-bad.rules"}, "bad graphite expr")
f([]string{"testdata/dir/rules6-bad.rules"}, "missing ':' in header")
f([]string{"http://unreachable-url"}, "failed to")
}
func TestRule_Validate(t *testing.T) {
func TestRuleValidate(t *testing.T) {
if err := (&Rule{}).Validate(); err == nil {
t.Errorf("expected empty name error")
t.Fatalf("expected empty name error")
}
if err := (&Rule{Alert: "alert"}).Validate(); err == nil {
t.Errorf("expected empty expr error")
t.Fatalf("expected empty expr error")
}
if err := (&Rule{Alert: "alert", Expr: "test>0"}).Validate(); err != nil {
t.Errorf("expected valid rule; got %s", err)
t.Fatalf("expected valid rule; got %s", err)
}
}
func TestGroup_Validate(t *testing.T) {
testCases := []struct {
group *Group
rules []Rule
validateAnnotations bool
validateExpressions bool
expErr string
}{
{
group: &Group{},
expErr: "group name must be set",
},
{
group: &Group{
Name: "negative interval",
Interval: promutils.NewDuration(-1),
func TestGroupValidate_Failure(t *testing.T) {
f := func(group *Group, validateExpressions bool, errStrExpected string) {
t.Helper()
err := group.Validate(nil, validateExpressions)
if err == nil {
t.Fatalf("expecting non-nil error")
}
errStr := err.Error()
if !strings.Contains(errStr, errStrExpected) {
t.Fatalf("missing %q in the returned error %q", errStrExpected, errStr)
}
}
f(&Group{}, false, "group name must be set")
f(&Group{
Name: "negative interval",
Interval: promutils.NewDuration(-1),
}, false, "interval shouldn't be lower than 0")
f(&Group{
Name: "wrong eval_offset",
Interval: promutils.NewDuration(time.Minute),
EvalOffset: promutils.NewDuration(2 * time.Minute),
}, false, "eval_offset should be smaller than interval")
f(&Group{
Name: "wrong limit",
Limit: -1,
}, false, "invalid limit")
f(&Group{
Name: "wrong concurrency",
Concurrency: -1,
}, false, "invalid concurrency")
f(&Group{
Name: "test",
Rules: []Rule{
{
Alert: "alert",
Expr: "up == 1",
},
expErr: "interval shouldn't be lower than 0",
},
{
group: &Group{
Name: "wrong eval_offset",
Interval: promutils.NewDuration(time.Minute),
EvalOffset: promutils.NewDuration(2 * time.Minute),
{
Alert: "alert",
Expr: "up == 1",
},
expErr: "eval_offset should be smaller than interval",
},
{
group: &Group{
Name: "wrong limit",
Limit: -1,
}, false, "duplicate")
f(&Group{
Name: "test",
Rules: []Rule{
{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"summary": "{{ value|query }}",
}},
{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"summary": "{{ value|query }}",
}},
},
}, false, "duplicate")
f(&Group{
Name: "test",
Rules: []Rule{
{Record: "record", Expr: "up == 1", Labels: map[string]string{
"summary": "{{ value|query }}",
}},
{Record: "record", Expr: "up == 1", Labels: map[string]string{
"summary": "{{ value|query }}",
}},
},
}, false, "duplicate")
f(&Group{
Name: "test",
Rules: []Rule{
{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"summary": "{{ value|query }}",
}},
{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"description": "{{ value|query }}",
}},
},
}, false, "duplicate")
f(&Group{
Name: "test",
Rules: []Rule{
{Record: "alert", Expr: "up == 1", Labels: map[string]string{
"summary": "{{ value|query }}",
}},
{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"summary": "{{ value|query }}",
}},
},
}, false, "duplicate")
f(&Group{
Name: "test graphite prometheus bad expr",
Type: NewGraphiteType(),
Rules: []Rule{
{
Expr: "sum(up == 0 ) by (host)",
For: promutils.NewDuration(10 * time.Millisecond),
},
expErr: "invalid limit",
},
{
group: &Group{
Name: "wrong concurrency",
Concurrency: -1,
{
Expr: "sumSeries(time('foo.bar',10))",
},
expErr: "invalid concurrency",
},
{
group: &Group{
Name: "test",
Rules: []Rule{
{
Record: "record",
Expr: "up | 0",
},
}, false, "invalid rule")
f(&Group{
Name: "test graphite inherit",
Type: NewGraphiteType(),
Rules: []Rule{
{
Expr: "sumSeries(time('foo.bar',10))",
For: promutils.NewDuration(10 * time.Millisecond),
},
{
Expr: "sum(up == 0 ) by (host)",
},
},
}, false, "either `record` or `alert` must be set")
// validate expressions
f(&Group{
Name: "test",
Rules: []Rule{
{
Record: "record",
Expr: "up | 0",
},
},
}, true, "invalid expression")
f(&Group{
Name: "test thanos",
Type: NewRawType("thanos"),
Rules: []Rule{
{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"description": "{{ value|query }}",
}},
},
}, true, "unknown datasource type")
f(&Group{
Name: "test graphite",
Type: NewGraphiteType(),
Rules: []Rule{
{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"description": "some-description",
}},
},
}, true, "bad graphite expr")
}
func TestGroupValidate_Success(t *testing.T) {
f := func(group *Group, validateAnnotations, validateExpressions bool) {
t.Helper()
var validateTplFn ValidateTplFn
if validateAnnotations {
validateTplFn = notifier.ValidateTemplates
}
err := group.Validate(validateTplFn, validateExpressions)
if err != nil {
t.Fatalf("unexpected error: %s", err)
}
}
f(&Group{
Name: "test",
Rules: []Rule{
{
Record: "record",
Expr: "up | 0",
},
},
}, false, false)
f(&Group{
Name: "test",
Rules: []Rule{
{
Alert: "alert",
Expr: "up == 1",
Labels: map[string]string{
"summary": "{{ value|query }}",
},
},
expErr: "",
},
{
group: &Group{
Name: "test",
Rules: []Rule{
{
Record: "record",
Expr: "up | 0",
},
},
},
expErr: "invalid expression",
validateExpressions: true,
},
{
group: &Group{
Name: "test",
Rules: []Rule{
{
Alert: "alert",
Expr: "up == 1",
Labels: map[string]string{
"summary": "{{ value|query }}",
},
},
},
},
expErr: "",
},
{
group: &Group{
Name: "test",
Rules: []Rule{
{
Alert: "alert",
Expr: "up == 1",
Labels: map[string]string{
"summary": `
}, false, false)
// validate annotiations
f(&Group{
Name: "test",
Rules: []Rule{
{
Alert: "alert",
Expr: "up == 1",
Labels: map[string]string{
"summary": `
{{ with printf "node_memory_MemTotal{job='node',instance='%s'}" "localhost" | query }}
{{ . | first | value | humanize1024 }}B
{{ end }}`,
},
},
},
},
validateAnnotations: true,
},
{
group: &Group{
Name: "test",
Rules: []Rule{
{
Alert: "alert",
Expr: "up == 1",
},
{
Alert: "alert",
Expr: "up == 1",
},
},
},
expErr: "duplicate",
},
{
group: &Group{
Name: "test",
Rules: []Rule{
{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"summary": "{{ value|query }}",
}},
{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"summary": "{{ value|query }}",
}},
},
},
expErr: "duplicate",
},
{
group: &Group{
Name: "test",
Rules: []Rule{
{Record: "record", Expr: "up == 1", Labels: map[string]string{
"summary": "{{ value|query }}",
}},
{Record: "record", Expr: "up == 1", Labels: map[string]string{
"summary": "{{ value|query }}",
}},
},
},
expErr: "duplicate",
},
{
group: &Group{
Name: "test",
Rules: []Rule{
{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"summary": "{{ value|query }}",
}},
{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"description": "{{ value|query }}",
}},
},
},
expErr: "",
},
{
group: &Group{
Name: "test",
Rules: []Rule{
{Record: "alert", Expr: "up == 1", Labels: map[string]string{
"summary": "{{ value|query }}",
}},
{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"summary": "{{ value|query }}",
}},
},
},
expErr: "",
},
{
group: &Group{
Name: "test thanos",
Type: NewRawType("thanos"),
Rules: []Rule{
{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"description": "{{ value|query }}",
}},
},
},
validateExpressions: true,
expErr: "unknown datasource type",
},
{
group: &Group{
Name: "test graphite",
Type: NewGraphiteType(),
Rules: []Rule{
{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"description": "some-description",
}},
},
},
validateExpressions: true,
expErr: "",
},
{
group: &Group{
Name: "test prometheus",
Type: NewPrometheusType(),
Rules: []Rule{
{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"description": "{{ value|query }}",
}},
},
},
validateExpressions: true,
expErr: "",
},
{
group: &Group{
Name: "test graphite inherit",
Type: NewGraphiteType(),
Rules: []Rule{
{
Expr: "sumSeries(time('foo.bar',10))",
For: promutils.NewDuration(10 * time.Millisecond),
},
{
Expr: "sum(up == 0 ) by (host)",
},
},
},
},
{
group: &Group{
Name: "test graphite prometheus bad expr",
Type: NewGraphiteType(),
Rules: []Rule{
{
Expr: "sum(up == 0 ) by (host)",
For: promutils.NewDuration(10 * time.Millisecond),
},
{
Expr: "sumSeries(time('foo.bar',10))",
},
},
},
expErr: "invalid rule",
},
}
}, true, false)
for _, tc := range testCases {
var validateTplFn ValidateTplFn
if tc.validateAnnotations {
validateTplFn = notifier.ValidateTemplates
}
err := tc.group.Validate(validateTplFn, tc.validateExpressions)
if err == nil {
if tc.expErr != "" {
t.Errorf("expected to get err %q; got nil insted", tc.expErr)
}
continue
}
if !strings.Contains(err.Error(), tc.expErr) {
t.Errorf("expected err to contain %q; got %q instead", tc.expErr, err)
}
}
// validate expressions
f(&Group{
Name: "test prometheus",
Type: NewPrometheusType(),
Rules: []Rule{
{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"description": "{{ value|query }}",
}},
},
}, false, true)
}
func TestHashRule(t *testing.T) {
testCases := []struct {
a, b Rule
equal bool
}{
{
Rule{Record: "record", Expr: "up == 1"},
Rule{Record: "record", Expr: "up == 1"},
true,
},
{
Rule{Alert: "alert", Expr: "up == 1"},
Rule{Alert: "alert", Expr: "up == 1"},
true,
},
{
Rule{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"foo": "bar",
"baz": "foo",
}},
Rule{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"foo": "bar",
"baz": "foo",
}},
true,
},
{
Rule{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"foo": "bar",
"baz": "foo",
}},
Rule{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"baz": "foo",
"foo": "bar",
}},
true,
},
{
Rule{Alert: "record", Expr: "up == 1"},
Rule{Alert: "record", Expr: "up == 1"},
true,
},
{
Rule{Alert: "alert", Expr: "up == 1", For: promutils.NewDuration(time.Minute), KeepFiringFor: promutils.NewDuration(time.Minute)},
Rule{Alert: "alert", Expr: "up == 1"},
true,
},
{
Rule{Alert: "record", Expr: "up == 1"},
Rule{Record: "record", Expr: "up == 1"},
false,
},
{
Rule{Record: "record", Expr: "up == 1"},
Rule{Record: "record", Expr: "up == 2"},
false,
},
{
Rule{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"foo": "bar",
"baz": "foo",
}},
Rule{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"baz": "foo",
"foo": "baz",
}},
false,
},
{
Rule{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"foo": "bar",
"baz": "foo",
}},
Rule{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"baz": "foo",
}},
false,
},
{
Rule{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"foo": "bar",
"baz": "foo",
}},
Rule{Alert: "alert", Expr: "up == 1"},
false,
},
}
for i, tc := range testCases {
aID, bID := HashRule(tc.a), HashRule(tc.b)
if tc.equal != (aID == bID) {
t.Fatalf("missmatch for rule %d", i)
func TestHashRule_NotEqual(t *testing.T) {
f := func(a, b Rule) {
t.Helper()
aID, bID := HashRule(a), HashRule(b)
if aID == bID {
t.Fatalf("rule hashes mustn't be equal; got %d", aID)
}
}
f(Rule{Alert: "record", Expr: "up == 1"}, Rule{Record: "record", Expr: "up == 1"})
f(Rule{Record: "record", Expr: "up == 1"}, Rule{Record: "record", Expr: "up == 2"})
f(Rule{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"foo": "bar",
"baz": "foo",
}}, Rule{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"baz": "foo",
"foo": "baz",
}})
f(Rule{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"foo": "bar",
"baz": "foo",
}}, Rule{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"baz": "foo",
}})
f(Rule{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"foo": "bar",
"baz": "foo",
}}, Rule{Alert: "alert", Expr: "up == 1"})
}
func TestHashRule_Equal(t *testing.T) {
f := func(a, b Rule) {
t.Helper()
aID, bID := HashRule(a), HashRule(b)
if aID != bID {
t.Fatalf("rule hashes must be equal; got %d and %d", aID, bID)
}
}
f(Rule{Record: "record", Expr: "up == 1"}, Rule{Record: "record", Expr: "up == 1"})
f(Rule{Alert: "alert", Expr: "up == 1"}, Rule{Alert: "alert", Expr: "up == 1"})
f(Rule{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"foo": "bar",
"baz": "foo",
}}, Rule{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"foo": "bar",
"baz": "foo",
}})
f(Rule{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"foo": "bar",
"baz": "foo",
}}, Rule{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
"baz": "foo",
"foo": "bar",
}})
f(Rule{Alert: "record", Expr: "up == 1"}, Rule{Alert: "record", Expr: "up == 1"})
f(Rule{
Alert: "alert", Expr: "up == 1", For: promutils.NewDuration(time.Minute), KeepFiringFor: promutils.NewDuration(time.Minute),
}, Rule{Alert: "alert", Expr: "up == 1"})
}
func TestGroupChecksum(t *testing.T) {

View File

@@ -29,7 +29,7 @@ func (l *Logger) isDisabled() bool {
}
// Errorf logs error message.
func (l *Logger) Errorf(format string, args ...interface{}) {
func (l *Logger) Errorf(format string, args ...any) {
if l.isDisabled() {
return
}
@@ -37,7 +37,7 @@ func (l *Logger) Errorf(format string, args ...interface{}) {
}
// Warnf logs warning message.
func (l *Logger) Warnf(format string, args ...interface{}) {
func (l *Logger) Warnf(format string, args ...any) {
if l.isDisabled() {
return
}
@@ -45,7 +45,7 @@ func (l *Logger) Warnf(format string, args ...interface{}) {
}
// Infof logs info message.
func (l *Logger) Infof(format string, args ...interface{}) {
func (l *Logger) Infof(format string, args ...any) {
if l.isDisabled() {
return
}
@@ -54,6 +54,6 @@ func (l *Logger) Infof(format string, args ...interface{}) {
// Panicf logs panic message and panics.
// Panicf can't be suppressed
func (l *Logger) Panicf(format string, args ...interface{}) {
func (l *Logger) Panicf(format string, args ...any) {
logger.Panicf(format, args...)
}

View File

@@ -18,14 +18,14 @@ func TestOutput(t *testing.T) {
mustMatch := func(exp string) {
t.Helper()
if exp == "" {
if testOutput.String() != "" {
t.Errorf("expected output to be empty; got %q", testOutput.String())
return
t.Fatalf("expected output to be empty; got %q", testOutput.String())
}
}
if !strings.Contains(testOutput.String(), exp) {
t.Errorf("output %q should contain %q", testOutput.String(), exp)
t.Fatalf("output %q should contain %q", testOutput.String(), exp)
}
fmt.Println(testOutput.String())
testOutput.Reset()

View File

@@ -69,7 +69,7 @@ func (t *Type) ValidateExpr(expr string) error {
}
// UnmarshalYAML implements the yaml.Unmarshaler interface.
func (t *Type) UnmarshalYAML(unmarshal func(interface{}) error) error {
func (t *Type) UnmarshalYAML(unmarshal func(any) error) error {
var s string
if err := unmarshal(&s); err != nil {
return err
@@ -87,7 +87,7 @@ func (t *Type) UnmarshalYAML(unmarshal func(interface{}) error) error {
}
// MarshalYAML implements the yaml.Unmarshaler interface.
func (t Type) MarshalYAML() (interface{}, error) {
func (t Type) MarshalYAML() (any, error) {
return t.Name, nil
}
@@ -98,7 +98,7 @@ type Header struct {
}
// UnmarshalYAML implements the yaml.Unmarshaler interface.
func (h *Header) UnmarshalYAML(unmarshal func(interface{}) error) error {
func (h *Header) UnmarshalYAML(unmarshal func(any) error) error {
var s string
if err := unmarshal(&s); err != nil {
return err

View File

@@ -12,6 +12,7 @@ import (
"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/httputils"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/netutil"
)
var (
@@ -100,7 +101,7 @@ func Init(extraParams url.Values) (QuerierBuilder, error) {
if err != nil {
return nil, fmt.Errorf("failed to create transport: %w", err)
}
tr.DialContext = httputils.GetStatDialFunc("vmalert_datasource")
tr.DialContext = netutil.NewStatDialFunc("vmalert_datasource")
tr.DisableKeepAlives = *disableKeepAlive
tr.MaxIdleConnsPerHost = *maxIdleConnections
if tr.MaxIdleConns != 0 && tr.MaxIdleConns < tr.MaxIdleConnsPerHost {

View File

@@ -119,7 +119,7 @@ func (pi *promInstant) Unmarshal(b []byte) error {
type promRange struct {
Result []struct {
Labels map[string]string `json:"metric"`
TVs [][2]interface{} `json:"values"`
TVs [][2]any `json:"values"`
} `json:"result"`
}
@@ -147,7 +147,7 @@ func (r promRange) metrics() ([]Metric, error) {
return result, nil
}
type promScalar [2]interface{}
type promScalar [2]any
func (r promScalar) metrics() ([]Metric, error) {
var m Metric

View File

@@ -31,26 +31,26 @@ var (
func TestVMInstantQuery(t *testing.T) {
mux := http.NewServeMux()
mux.HandleFunc("/", func(_ http.ResponseWriter, _ *http.Request) {
t.Errorf("should not be called")
t.Fatalf("should not be called")
})
c := -1
mux.HandleFunc("/api/v1/query", func(w http.ResponseWriter, r *http.Request) {
c++
if r.Method != http.MethodPost {
t.Errorf("expected POST method got %s", r.Method)
t.Fatalf("expected POST method got %s", r.Method)
}
if name, pass, _ := r.BasicAuth(); name != basicAuthName || pass != basicAuthPass {
t.Errorf("expected %s:%s as basic auth got %s:%s", basicAuthName, basicAuthPass, name, pass)
t.Fatalf("expected %s:%s as basic auth got %s:%s", basicAuthName, basicAuthPass, name, pass)
}
if r.URL.Query().Get("query") != query {
t.Errorf("expected %s in query param, got %s", query, r.URL.Query().Get("query"))
t.Fatalf("expected %s in query param, got %s", query, r.URL.Query().Get("query"))
}
timeParam := r.URL.Query().Get("time")
if timeParam == "" {
t.Errorf("expected 'time' in query param, got nil instead")
t.Fatalf("expected 'time' in query param, got nil instead")
}
if _, err := time.Parse(time.RFC3339, timeParam); err != nil {
t.Errorf("failed to parse 'time' query param %q: %s", timeParam, err)
t.Fatalf("failed to parse 'time' query param %q: %s", timeParam, err)
}
switch c {
case 0:
@@ -197,13 +197,13 @@ func TestVMInstantQuery(t *testing.T) {
func TestVMInstantQueryWithRetry(t *testing.T) {
mux := http.NewServeMux()
mux.HandleFunc("/", func(_ http.ResponseWriter, _ *http.Request) {
t.Errorf("should not be called")
t.Fatalf("should not be called")
})
c := -1
mux.HandleFunc("/api/v1/query", func(w http.ResponseWriter, r *http.Request) {
c++
if r.URL.Query().Get("query") != query {
t.Errorf("expected %s in query param, got %s", query, r.URL.Query().Get("query"))
t.Fatalf("expected %s in query param, got %s", query, r.URL.Query().Get("query"))
}
switch c {
case 0:
@@ -289,37 +289,37 @@ func metricsEqual(t *testing.T, gotM, expectedM []Metric) {
func TestVMRangeQuery(t *testing.T) {
mux := http.NewServeMux()
mux.HandleFunc("/", func(_ http.ResponseWriter, _ *http.Request) {
t.Errorf("should not be called")
t.Fatalf("should not be called")
})
c := -1
mux.HandleFunc("/api/v1/query_range", func(w http.ResponseWriter, r *http.Request) {
c++
if r.Method != http.MethodPost {
t.Errorf("expected POST method got %s", r.Method)
t.Fatalf("expected POST method got %s", r.Method)
}
if name, pass, _ := r.BasicAuth(); name != basicAuthName || pass != basicAuthPass {
t.Errorf("expected %s:%s as basic auth got %s:%s", basicAuthName, basicAuthPass, name, pass)
t.Fatalf("expected %s:%s as basic auth got %s:%s", basicAuthName, basicAuthPass, name, pass)
}
if r.URL.Query().Get("query") != query {
t.Errorf("expected %s in query param, got %s", query, r.URL.Query().Get("query"))
t.Fatalf("expected %s in query param, got %s", query, r.URL.Query().Get("query"))
}
startTS := r.URL.Query().Get("start")
if startTS == "" {
t.Errorf("expected 'start' in query param, got nil instead")
t.Fatalf("expected 'start' in query param, got nil instead")
}
if _, err := time.Parse(time.RFC3339, startTS); err != nil {
t.Errorf("failed to parse 'start' query param: %s", err)
t.Fatalf("failed to parse 'start' query param: %s", err)
}
endTS := r.URL.Query().Get("end")
if endTS == "" {
t.Errorf("expected 'end' in query param, got nil instead")
t.Fatalf("expected 'end' in query param, got nil instead")
}
if _, err := time.Parse(time.RFC3339, endTS); err != nil {
t.Errorf("failed to parse 'end' query param: %s", err)
t.Fatalf("failed to parse 'end' query param: %s", err)
}
step := r.URL.Query().Get("step")
if step != "15s" {
t.Errorf("expected 'step' query param to be 15s; got %q instead", step)
t.Fatalf("expected 'step' query param to be 15s; got %q instead", step)
}
switch c {
case 0:
@@ -370,368 +370,299 @@ func TestVMRangeQuery(t *testing.T) {
}
func TestRequestParams(t *testing.T) {
authCfg, err := baCfg.NewConfig(".")
if err != nil {
t.Fatalf("unexpected: %s", err)
}
query := "up"
timestamp := time.Date(2001, 2, 3, 4, 5, 6, 0, time.UTC)
f := func(isQueryRange bool, vm *VMStorage, checkFn func(t *testing.T, r *http.Request)) {
t.Helper()
req, err := vm.newRequest(ctx)
if err != nil {
t.Fatalf("error in newRequest: %s", err)
}
switch vm.dataSourceType {
case "", datasourcePrometheus:
if isQueryRange {
vm.setPrometheusRangeReqParams(req, query, timestamp, timestamp)
} else {
vm.setPrometheusInstantReqParams(req, query, timestamp)
}
case datasourceGraphite:
vm.setGraphiteReqParams(req, query)
}
checkFn(t, req)
}
authCfg, err := baCfg.NewConfig(".")
if err != nil {
t.Fatalf("unexpected error: %s", err)
}
storage := VMStorage{
extraParams: url.Values{"round_digits": {"10"}},
}
testCases := []struct {
name string
queryRange bool
vm *VMStorage
checkFn func(t *testing.T, r *http.Request)
}{
{
"prometheus path",
false,
&VMStorage{
dataSourceType: datasourcePrometheus,
},
func(t *testing.T, r *http.Request) {
checkEqualString(t, "/api/v1/query", r.URL.Path)
},
},
{
"prometheus prefix",
false,
&VMStorage{
dataSourceType: datasourcePrometheus,
appendTypePrefix: true,
},
func(t *testing.T, r *http.Request) {
checkEqualString(t, "/prometheus/api/v1/query", r.URL.Path)
},
},
{
"prometheus range path",
true,
&VMStorage{
dataSourceType: datasourcePrometheus,
},
func(t *testing.T, r *http.Request) {
checkEqualString(t, "/api/v1/query_range", r.URL.Path)
},
},
{
"prometheus range prefix",
true,
&VMStorage{
dataSourceType: datasourcePrometheus,
appendTypePrefix: true,
},
func(t *testing.T, r *http.Request) {
checkEqualString(t, "/prometheus/api/v1/query_range", r.URL.Path)
},
},
{
"graphite path",
false,
&VMStorage{
dataSourceType: datasourceGraphite,
},
func(t *testing.T, r *http.Request) {
checkEqualString(t, graphitePath, r.URL.Path)
},
},
{
"graphite prefix",
false,
&VMStorage{
dataSourceType: datasourceGraphite,
appendTypePrefix: true,
},
func(t *testing.T, r *http.Request) {
checkEqualString(t, graphitePrefix+graphitePath, r.URL.Path)
},
},
{
"default params",
false,
&VMStorage{},
func(t *testing.T, r *http.Request) {
exp := url.Values{"query": {query}, "time": {timestamp.Format(time.RFC3339)}}
checkEqualString(t, exp.Encode(), r.URL.RawQuery)
},
},
{
"default range params",
true,
&VMStorage{},
func(t *testing.T, r *http.Request) {
ts := timestamp.Format(time.RFC3339)
exp := url.Values{"query": {query}, "start": {ts}, "end": {ts}}
checkEqualString(t, exp.Encode(), r.URL.RawQuery)
},
},
{
"basic auth",
false,
&VMStorage{authCfg: authCfg},
func(t *testing.T, r *http.Request) {
u, p, _ := r.BasicAuth()
checkEqualString(t, "foo", u)
checkEqualString(t, "bar", p)
},
},
{
"basic auth range",
true,
&VMStorage{authCfg: authCfg},
func(t *testing.T, r *http.Request) {
u, p, _ := r.BasicAuth()
checkEqualString(t, "foo", u)
checkEqualString(t, "bar", p)
},
},
{
"evaluation interval",
false,
&VMStorage{
evaluationInterval: 15 * time.Second,
},
func(t *testing.T, r *http.Request) {
evalInterval := 15 * time.Second
exp := url.Values{"query": {query}, "step": {evalInterval.String()}, "time": {timestamp.Format(time.RFC3339)}}
checkEqualString(t, exp.Encode(), r.URL.RawQuery)
},
},
{
"step override",
false,
&VMStorage{
queryStep: time.Minute,
},
func(t *testing.T, r *http.Request) {
exp := url.Values{
"query": {query},
"step": {fmt.Sprintf("%ds", int(time.Minute.Seconds()))},
"time": {timestamp.Format(time.RFC3339)},
}
checkEqualString(t, exp.Encode(), r.URL.RawQuery)
},
},
{
"step to seconds",
false,
&VMStorage{
evaluationInterval: 3 * time.Hour,
},
func(t *testing.T, r *http.Request) {
evalInterval := 3 * time.Hour
exp := url.Values{"query": {query}, "step": {fmt.Sprintf("%ds", int(evalInterval.Seconds()))}, "time": {timestamp.Format(time.RFC3339)}}
checkEqualString(t, exp.Encode(), r.URL.RawQuery)
},
},
{
"prometheus extra params",
false,
&VMStorage{
extraParams: url.Values{"round_digits": {"10"}},
},
func(t *testing.T, r *http.Request) {
exp := url.Values{"query": {query}, "round_digits": {"10"}, "time": {timestamp.Format(time.RFC3339)}}
checkEqualString(t, exp.Encode(), r.URL.RawQuery)
},
},
{
"prometheus extra params range",
true,
&VMStorage{
extraParams: url.Values{
"nocache": {"1"},
"max_lookback": {"1h"},
},
},
func(t *testing.T, r *http.Request) {
exp := url.Values{
"query": {query},
"end": {timestamp.Format(time.RFC3339)},
"start": {timestamp.Format(time.RFC3339)},
"nocache": {"1"},
"max_lookback": {"1h"},
}
checkEqualString(t, exp.Encode(), r.URL.RawQuery)
},
},
{
"custom params overrides the original params",
false,
storage.Clone().ApplyParams(QuerierParams{
QueryParams: url.Values{"round_digits": {"2"}},
}),
func(t *testing.T, r *http.Request) {
exp := url.Values{"query": {query}, "round_digits": {"2"}, "time": {timestamp.Format(time.RFC3339)}}
checkEqualString(t, exp.Encode(), r.URL.RawQuery)
},
},
{
"allow duplicates in query params",
false,
storage.Clone().ApplyParams(QuerierParams{
QueryParams: url.Values{"extra_labels": {"env=dev", "foo=bar"}},
}),
func(t *testing.T, r *http.Request) {
exp := url.Values{"query": {query}, "round_digits": {"10"}, "extra_labels": {"env=dev", "foo=bar"}, "time": {timestamp.Format(time.RFC3339)}}
checkEqualString(t, exp.Encode(), r.URL.RawQuery)
},
},
{
"graphite extra params",
false,
&VMStorage{
dataSourceType: datasourceGraphite,
extraParams: url.Values{
"nocache": {"1"},
"max_lookback": {"1h"},
},
},
func(t *testing.T, r *http.Request) {
exp := fmt.Sprintf("format=json&from=-5min&max_lookback=1h&nocache=1&target=%s&until=now", query)
checkEqualString(t, exp, r.URL.RawQuery)
},
},
{
"graphite extra params allows to override from",
false,
&VMStorage{
dataSourceType: datasourceGraphite,
extraParams: url.Values{
"from": {"-10m"},
},
},
func(t *testing.T, r *http.Request) {
exp := fmt.Sprintf("format=json&from=-10m&target=%s&until=now", query)
checkEqualString(t, exp, r.URL.RawQuery)
},
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
req, err := tc.vm.newRequest(ctx)
if err != nil {
t.Fatal(err)
}
switch tc.vm.dataSourceType {
case "", datasourcePrometheus:
if tc.queryRange {
tc.vm.setPrometheusRangeReqParams(req, query, timestamp, timestamp)
} else {
tc.vm.setPrometheusInstantReqParams(req, query, timestamp)
}
case datasourceGraphite:
tc.vm.setGraphiteReqParams(req, query)
}
tc.checkFn(t, req)
})
}
// prometheus path
f(false, &VMStorage{
dataSourceType: datasourcePrometheus,
}, func(t *testing.T, r *http.Request) {
checkEqualString(t, "/api/v1/query", r.URL.Path)
})
// prometheus prefix
f(false, &VMStorage{
dataSourceType: datasourcePrometheus,
appendTypePrefix: true,
}, func(t *testing.T, r *http.Request) {
checkEqualString(t, "/prometheus/api/v1/query", r.URL.Path)
})
// prometheus range path
f(true, &VMStorage{
dataSourceType: datasourcePrometheus,
}, func(t *testing.T, r *http.Request) {
checkEqualString(t, "/api/v1/query_range", r.URL.Path)
})
// prometheus range prefix
f(true, &VMStorage{
dataSourceType: datasourcePrometheus,
appendTypePrefix: true,
}, func(t *testing.T, r *http.Request) {
checkEqualString(t, "/prometheus/api/v1/query_range", r.URL.Path)
})
// graphite path
f(false, &VMStorage{
dataSourceType: datasourceGraphite,
}, func(t *testing.T, r *http.Request) {
checkEqualString(t, graphitePath, r.URL.Path)
})
// graphite prefix
f(false, &VMStorage{
dataSourceType: datasourceGraphite,
appendTypePrefix: true,
}, func(t *testing.T, r *http.Request) {
checkEqualString(t, graphitePrefix+graphitePath, r.URL.Path)
})
// default params
f(false, &VMStorage{}, func(t *testing.T, r *http.Request) {
exp := url.Values{"query": {query}, "time": {timestamp.Format(time.RFC3339)}}
checkEqualString(t, exp.Encode(), r.URL.RawQuery)
})
// default range params
f(true, &VMStorage{}, func(t *testing.T, r *http.Request) {
ts := timestamp.Format(time.RFC3339)
exp := url.Values{"query": {query}, "start": {ts}, "end": {ts}}
checkEqualString(t, exp.Encode(), r.URL.RawQuery)
})
// basic auth
f(false, &VMStorage{
authCfg: authCfg,
}, func(t *testing.T, r *http.Request) {
u, p, _ := r.BasicAuth()
checkEqualString(t, "foo", u)
checkEqualString(t, "bar", p)
})
// basic auth range
f(true, &VMStorage{
authCfg: authCfg,
}, func(t *testing.T, r *http.Request) {
u, p, _ := r.BasicAuth()
checkEqualString(t, "foo", u)
checkEqualString(t, "bar", p)
})
// evaluation interval
f(false, &VMStorage{
evaluationInterval: 15 * time.Second,
}, func(t *testing.T, r *http.Request) {
evalInterval := 15 * time.Second
exp := url.Values{"query": {query}, "step": {evalInterval.String()}, "time": {timestamp.Format(time.RFC3339)}}
checkEqualString(t, exp.Encode(), r.URL.RawQuery)
})
// step override
f(false, &VMStorage{
queryStep: time.Minute,
}, func(t *testing.T, r *http.Request) {
exp := url.Values{
"query": {query},
"step": {fmt.Sprintf("%ds", int(time.Minute.Seconds()))},
"time": {timestamp.Format(time.RFC3339)},
}
checkEqualString(t, exp.Encode(), r.URL.RawQuery)
})
// step to seconds
f(false, &VMStorage{
evaluationInterval: 3 * time.Hour,
}, func(t *testing.T, r *http.Request) {
evalInterval := 3 * time.Hour
exp := url.Values{"query": {query}, "step": {fmt.Sprintf("%ds", int(evalInterval.Seconds()))}, "time": {timestamp.Format(time.RFC3339)}}
checkEqualString(t, exp.Encode(), r.URL.RawQuery)
})
// prometheus extra params
f(false, &VMStorage{
extraParams: url.Values{"round_digits": {"10"}},
}, func(t *testing.T, r *http.Request) {
exp := url.Values{"query": {query}, "round_digits": {"10"}, "time": {timestamp.Format(time.RFC3339)}}
checkEqualString(t, exp.Encode(), r.URL.RawQuery)
})
// prometheus extra params range
f(true, &VMStorage{
extraParams: url.Values{
"nocache": {"1"},
"max_lookback": {"1h"},
},
}, func(t *testing.T, r *http.Request) {
exp := url.Values{
"query": {query},
"end": {timestamp.Format(time.RFC3339)},
"start": {timestamp.Format(time.RFC3339)},
"nocache": {"1"},
"max_lookback": {"1h"},
}
checkEqualString(t, exp.Encode(), r.URL.RawQuery)
})
// custom params overrides the original params
f(false, storage.Clone().ApplyParams(QuerierParams{
QueryParams: url.Values{"round_digits": {"2"}},
}), func(t *testing.T, r *http.Request) {
exp := url.Values{"query": {query}, "round_digits": {"2"}, "time": {timestamp.Format(time.RFC3339)}}
checkEqualString(t, exp.Encode(), r.URL.RawQuery)
})
// allow duplicates in query params
f(false, storage.Clone().ApplyParams(QuerierParams{
QueryParams: url.Values{"extra_labels": {"env=dev", "foo=bar"}},
}), func(t *testing.T, r *http.Request) {
exp := url.Values{"query": {query}, "round_digits": {"10"}, "extra_labels": {"env=dev", "foo=bar"}, "time": {timestamp.Format(time.RFC3339)}}
checkEqualString(t, exp.Encode(), r.URL.RawQuery)
})
// graphite extra params
f(false, &VMStorage{
dataSourceType: datasourceGraphite,
extraParams: url.Values{
"nocache": {"1"},
"max_lookback": {"1h"},
},
}, func(t *testing.T, r *http.Request) {
exp := fmt.Sprintf("format=json&from=-5min&max_lookback=1h&nocache=1&target=%s&until=now", query)
checkEqualString(t, exp, r.URL.RawQuery)
})
// graphite extra params allows to override from
f(false, &VMStorage{
dataSourceType: datasourceGraphite,
extraParams: url.Values{
"from": {"-10m"},
},
}, func(t *testing.T, r *http.Request) {
exp := fmt.Sprintf("format=json&from=-10m&target=%s&until=now", query)
checkEqualString(t, exp, r.URL.RawQuery)
})
}
func TestHeaders(t *testing.T) {
testCases := []struct {
name string
vmFn func() *VMStorage
checkFn func(t *testing.T, r *http.Request)
}{
{
name: "basic auth",
vmFn: func() *VMStorage {
cfg, err := utils.AuthConfig(utils.WithBasicAuth("foo", "bar", ""))
if err != nil {
t.Errorf("Error get auth config: %s", err)
}
return &VMStorage{authCfg: cfg}
},
checkFn: func(t *testing.T, r *http.Request) {
u, p, _ := r.BasicAuth()
checkEqualString(t, "foo", u)
checkEqualString(t, "bar", p)
},
},
{
name: "bearer auth",
vmFn: func() *VMStorage {
cfg, err := utils.AuthConfig(utils.WithBearer("foo", ""))
if err != nil {
t.Errorf("Error get auth config: %s", err)
}
return &VMStorage{authCfg: cfg}
},
checkFn: func(t *testing.T, r *http.Request) {
reqToken := r.Header.Get("Authorization")
splitToken := strings.Split(reqToken, "Bearer ")
if len(splitToken) != 2 {
t.Errorf("expected two items got %d", len(splitToken))
}
token := splitToken[1]
checkEqualString(t, "foo", token)
},
},
{
name: "custom extraHeaders",
vmFn: func() *VMStorage {
return &VMStorage{extraHeaders: []keyValue{
{key: "Foo", value: "bar"},
{key: "Baz", value: "qux"},
}}
},
checkFn: func(t *testing.T, r *http.Request) {
h1 := r.Header.Get("Foo")
checkEqualString(t, "bar", h1)
h2 := r.Header.Get("Baz")
checkEqualString(t, "qux", h2)
},
},
{
name: "custom header overrides basic auth",
vmFn: func() *VMStorage {
cfg, err := utils.AuthConfig(utils.WithBasicAuth("foo", "bar", ""))
if err != nil {
t.Errorf("Error get auth config: %s", err)
}
return &VMStorage{
authCfg: cfg,
extraHeaders: []keyValue{
{key: "Authorization", value: "Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ=="},
},
}
},
checkFn: func(t *testing.T, r *http.Request) {
u, p, _ := r.BasicAuth()
checkEqualString(t, "Aladdin", u)
checkEqualString(t, "open sesame", p)
},
},
}
for _, tt := range testCases {
t.Run(tt.name, func(t *testing.T) {
vm := tt.vmFn()
req, err := vm.newQueryRequest(ctx, "foo", time.Now())
if err != nil {
t.Fatal(err)
}
tt.checkFn(t, req)
})
f := func(vmFn func() *VMStorage, checkFn func(t *testing.T, r *http.Request)) {
t.Helper()
vm := vmFn()
req, err := vm.newQueryRequest(ctx, "foo", time.Now())
if err != nil {
t.Fatalf("error in newQueryRequest: %s", err)
}
checkFn(t, req)
}
// basic auth
f(func() *VMStorage {
cfg, err := utils.AuthConfig(utils.WithBasicAuth("foo", "bar", ""))
if err != nil {
t.Fatalf("Error get auth config: %s", err)
}
return &VMStorage{authCfg: cfg}
}, func(t *testing.T, r *http.Request) {
u, p, _ := r.BasicAuth()
checkEqualString(t, "foo", u)
checkEqualString(t, "bar", p)
})
// bearer auth
f(func() *VMStorage {
cfg, err := utils.AuthConfig(utils.WithBearer("foo", ""))
if err != nil {
t.Fatalf("Error get auth config: %s", err)
}
return &VMStorage{authCfg: cfg}
}, func(t *testing.T, r *http.Request) {
reqToken := r.Header.Get("Authorization")
splitToken := strings.Split(reqToken, "Bearer ")
if len(splitToken) != 2 {
t.Fatalf("expected two items got %d", len(splitToken))
}
token := splitToken[1]
checkEqualString(t, "foo", token)
})
// custom extraHeaders
f(func() *VMStorage {
return &VMStorage{extraHeaders: []keyValue{
{key: "Foo", value: "bar"},
{key: "Baz", value: "qux"},
}}
}, func(t *testing.T, r *http.Request) {
h1 := r.Header.Get("Foo")
checkEqualString(t, "bar", h1)
h2 := r.Header.Get("Baz")
checkEqualString(t, "qux", h2)
})
// custom header overrides basic auth
f(func() *VMStorage {
cfg, err := utils.AuthConfig(utils.WithBasicAuth("foo", "bar", ""))
if err != nil {
t.Fatalf("Error get auth config: %s", err)
}
return &VMStorage{
authCfg: cfg,
extraHeaders: []keyValue{
{key: "Authorization", value: "Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ=="},
},
}
}, func(t *testing.T, r *http.Request) {
u, p, _ := r.BasicAuth()
checkEqualString(t, "Aladdin", u)
checkEqualString(t, "open sesame", p)
})
}
func checkEqualString(t *testing.T, exp, got string) {
t.Helper()
if got != exp {
t.Errorf("expected to get: \n%q; \ngot: \n%q", exp, got)
t.Fatalf("expected to get: \n%q; \ngot: \n%q", exp, got)
}
}
func expectError(t *testing.T, err error, exp string) {
t.Helper()
if err == nil {
t.Errorf("expected non-nil error")
t.Fatalf("expected non-nil error")
}
if !strings.Contains(err.Error(), exp) {
t.Errorf("expected error %q to contain %q", err, exp)
t.Fatalf("expected error %q to contain %q", err, exp)
}
}

View File

@@ -25,26 +25,26 @@ func TestGetExternalURL(t *testing.T) {
invalidURL := "victoriametrics.com/path"
_, err := getExternalURL(invalidURL)
if err == nil {
t.Errorf("expected error, got nil")
t.Fatalf("expected error, got nil")
}
expURL := "https://victoriametrics.com/path"
u, err := getExternalURL(expURL)
if err != nil {
t.Errorf("unexpected error %s", err)
t.Fatalf("unexpected error %s", err)
}
if u.String() != expURL {
t.Errorf("unexpected url: want %q, got %s", expURL, u.String())
t.Fatalf("unexpected url: want %q, got %s", expURL, u.String())
}
h, _ := os.Hostname()
expURL = fmt.Sprintf("http://%s:8880", h)
u, err = getExternalURL("")
if err != nil {
t.Errorf("unexpected error %s", err)
t.Fatalf("unexpected error %s", err)
}
if u.String() != expURL {
t.Errorf("unexpected url: want %s, got %s", expURL, u.String())
t.Fatalf("unexpected url: want %s, got %s", expURL, u.String())
}
}
@@ -53,22 +53,22 @@ func TestGetAlertURLGenerator(t *testing.T) {
u, _ := url.Parse("https://victoriametrics.com/path")
fn, err := getAlertURLGenerator(u, "", false)
if err != nil {
t.Errorf("unexpected error %s", err)
t.Fatalf("unexpected error %s", err)
}
exp := fmt.Sprintf("https://victoriametrics.com/path/vmalert/alert?%s=42&%s=2", paramGroupID, paramAlertID)
if exp != fn(testAlert) {
t.Errorf("unexpected url want %s, got %s", exp, fn(testAlert))
t.Fatalf("unexpected url want %s, got %s", exp, fn(testAlert))
}
_, err = getAlertURLGenerator(nil, "foo?{{invalid}}", true)
if err == nil {
t.Errorf("expected template validation error got nil")
t.Fatalf("expected template validation error got nil")
}
fn, err = getAlertURLGenerator(u, "foo?query={{$value}}&ds={{ $labels.tenant }}", true)
if err != nil {
t.Errorf("unexpected error %s", err)
t.Fatalf("unexpected error %s", err)
}
if exp := "https://victoriametrics.com/path/foo?query=4&ds=baz"; exp != fn(testAlert) {
t.Errorf("unexpected url want %s, got %s", exp, fn(testAlert))
t.Fatalf("unexpected url want %s, got %s", exp, fn(testAlert))
}
}

View File

@@ -82,9 +82,8 @@ func TestManagerUpdateConcurrent(t *testing.T) {
wg.Wait()
}
// TestManagerUpdate tests sequential configuration
// updates.
func TestManagerUpdate(t *testing.T) {
// TestManagerUpdate tests sequential configuration updates.
func TestManagerUpdate_Success(t *testing.T) {
const defaultEvalInterval = time.Second * 30
currentEvalInterval := *evaluationInterval
*evaluationInterval = defaultEvalInterval
@@ -120,145 +119,127 @@ func TestManagerUpdate(t *testing.T) {
}
)
testCases := []struct {
name string
initPath string
updatePath string
want []*rule.Group
}{
{
name: "update good rules",
initPath: "config/testdata/rules/rules0-good.rules",
updatePath: "config/testdata/dir/rules1-good.rules",
want: []*rule.Group{
{
File: "config/testdata/dir/rules1-good.rules",
Name: "duplicatedGroupDiffFiles",
Type: config.NewPrometheusType(),
Interval: defaultEvalInterval,
Rules: []rule.Rule{
&rule.AlertingRule{
Name: "VMRows",
Expr: "vm_rows > 0",
For: 5 * time.Minute,
Labels: map[string]string{"dc": "gcp", "label": "bar"},
Annotations: map[string]string{
"summary": "{{ $value }}",
"description": "{{$labels}}",
},
},
},
},
},
},
{
name: "update good rules from 1 to 2 groups",
initPath: "config/testdata/dir/rules/rules1-good.rules",
updatePath: "config/testdata/rules/rules0-good.rules",
want: []*rule.Group{
{
File: "config/testdata/rules/rules0-good.rules",
Name: "groupGorSingleAlert",
Type: config.NewPrometheusType(),
Interval: defaultEvalInterval,
Rules: []rule.Rule{VMRows},
},
{
File: "config/testdata/rules/rules0-good.rules",
Interval: defaultEvalInterval,
Type: config.NewPrometheusType(),
Name: "TestGroup",
Rules: []rule.Rule{
Conns,
ExampleAlertAlwaysFiring,
},
},
},
},
{
name: "update with one bad rule file",
initPath: "config/testdata/rules/rules0-good.rules",
updatePath: "config/testdata/dir/rules2-bad.rules",
want: []*rule.Group{
{
File: "config/testdata/rules/rules0-good.rules",
Name: "groupGorSingleAlert",
Type: config.NewPrometheusType(),
Interval: defaultEvalInterval,
Rules: []rule.Rule{VMRows},
},
{
File: "config/testdata/rules/rules0-good.rules",
Interval: defaultEvalInterval,
Name: "TestGroup",
Type: config.NewPrometheusType(),
Rules: []rule.Rule{
Conns,
ExampleAlertAlwaysFiring,
},
},
},
},
{
name: "update empty dir rules from 0 to 2 groups",
initPath: "config/testdata/empty/*",
updatePath: "config/testdata/rules/rules0-good.rules",
want: []*rule.Group{
{
File: "config/testdata/rules/rules0-good.rules",
Name: "groupGorSingleAlert",
Type: config.NewPrometheusType(),
Interval: defaultEvalInterval,
Rules: []rule.Rule{VMRows},
},
{
File: "config/testdata/rules/rules0-good.rules",
Interval: defaultEvalInterval,
Type: config.NewPrometheusType(),
Name: "TestGroup",
Rules: []rule.Rule{
Conns,
ExampleAlertAlwaysFiring,
},
},
},
},
f := func(initPath, updatePath string, groupsExpected []*rule.Group) {
t.Helper()
ctx, cancel := context.WithCancel(context.TODO())
m := &manager{
groups: make(map[uint64]*rule.Group),
querierBuilder: &datasource.FakeQuerier{},
notifiers: func() []notifier.Notifier { return []notifier.Notifier{&notifier.FakeNotifier{}} },
}
cfgInit := loadCfg(t, []string{initPath}, true, true)
if err := m.update(ctx, cfgInit, false); err != nil {
t.Fatalf("failed to complete initial rules update: %s", err)
}
cfgUpdate, err := config.Parse([]string{updatePath}, notifier.ValidateTemplates, true)
if err == nil { // update can fail and that's expected
_ = m.update(ctx, cfgUpdate, false)
}
if len(groupsExpected) != len(m.groups) {
t.Fatalf("unexpected number of groups; got %d; want %d", len(m.groups), len(groupsExpected))
}
for _, wantG := range groupsExpected {
gotG, ok := m.groups[wantG.ID()]
if !ok {
t.Fatalf("expected to have group %q", wantG.Name)
}
compareGroups(t, wantG, gotG)
}
cancel()
m.close()
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
ctx, cancel := context.WithCancel(context.TODO())
m := &manager{
groups: make(map[uint64]*rule.Group),
querierBuilder: &datasource.FakeQuerier{},
notifiers: func() []notifier.Notifier { return []notifier.Notifier{&notifier.FakeNotifier{}} },
}
cfgInit := loadCfg(t, []string{tc.initPath}, true, true)
if err := m.update(ctx, cfgInit, false); err != nil {
t.Fatalf("failed to complete initial rules update: %s", err)
}
// update good rules
f("config/testdata/rules/rules0-good.rules", "config/testdata/dir/rules1-good.rules", []*rule.Group{
{
File: "config/testdata/dir/rules1-good.rules",
Name: "duplicatedGroupDiffFiles",
Type: config.NewPrometheusType(),
Interval: defaultEvalInterval,
Rules: []rule.Rule{
&rule.AlertingRule{
Name: "VMRows",
Expr: "vm_rows > 0",
For: 5 * time.Minute,
Labels: map[string]string{"dc": "gcp", "label": "bar"},
Annotations: map[string]string{
"summary": "{{ $value }}",
"description": "{{$labels}}",
},
},
},
},
})
cfgUpdate, err := config.Parse([]string{tc.updatePath}, notifier.ValidateTemplates, true)
if err == nil { // update can fail and that's expected
_ = m.update(ctx, cfgUpdate, false)
}
if len(tc.want) != len(m.groups) {
t.Fatalf("\nwant number of groups: %d;\ngot: %d ", len(tc.want), len(m.groups))
}
// update good rules from 1 to 2 groups
f("config/testdata/dir/rules/rules1-good.rules", "config/testdata/rules/rules0-good.rules", []*rule.Group{
{
File: "config/testdata/rules/rules0-good.rules",
Name: "groupGorSingleAlert",
Type: config.NewPrometheusType(),
Interval: defaultEvalInterval,
Rules: []rule.Rule{VMRows},
},
{
File: "config/testdata/rules/rules0-good.rules",
Interval: defaultEvalInterval,
Type: config.NewPrometheusType(),
Name: "TestGroup",
Rules: []rule.Rule{
Conns,
ExampleAlertAlwaysFiring,
},
},
})
for _, wantG := range tc.want {
gotG, ok := m.groups[wantG.ID()]
if !ok {
t.Fatalf("expected to have group %q", wantG.Name)
}
compareGroups(t, wantG, gotG)
}
// update with one bad rule file
f("config/testdata/rules/rules0-good.rules", "config/testdata/dir/rules2-bad.rules", []*rule.Group{
{
File: "config/testdata/rules/rules0-good.rules",
Name: "groupGorSingleAlert",
Type: config.NewPrometheusType(),
Interval: defaultEvalInterval,
Rules: []rule.Rule{VMRows},
},
{
File: "config/testdata/rules/rules0-good.rules",
Interval: defaultEvalInterval,
Name: "TestGroup",
Type: config.NewPrometheusType(),
Rules: []rule.Rule{
Conns,
ExampleAlertAlwaysFiring,
},
},
})
cancel()
m.close()
})
}
// update empty dir rules from 0 to 2 groups
f("config/testdata/empty/*", "config/testdata/rules/rules0-good.rules", []*rule.Group{
{
File: "config/testdata/rules/rules0-good.rules",
Name: "groupGorSingleAlert",
Type: config.NewPrometheusType(),
Interval: defaultEvalInterval,
Rules: []rule.Rule{VMRows},
},
{
File: "config/testdata/rules/rules0-good.rules",
Interval: defaultEvalInterval,
Type: config.NewPrometheusType(),
Name: "TestGroup",
Rules: []rule.Rule{
Conns,
ExampleAlertAlwaysFiring,
},
},
})
}
func compareGroups(t *testing.T, a, b *rule.Group) {
t.Helper()
if a.Name != b.Name {
@@ -285,82 +266,59 @@ func compareGroups(t *testing.T, a, b *rule.Group) {
}
}
func TestManagerUpdateNegative(t *testing.T) {
testCases := []struct {
notifiers []notifier.Notifier
rw remotewrite.RWClient
cfg config.Group
expErr string
}{
{
nil,
nil,
config.Group{
Name: "Recording rule only",
Rules: []config.Rule{
{Record: "record", Expr: "max(up)"},
},
},
"contains recording rules",
},
{
nil,
nil,
config.Group{
Name: "Alerting rule only",
Rules: []config.Rule{
{Alert: "alert", Expr: "up > 0"},
},
},
"contains alerting rules",
},
{
[]notifier.Notifier{&notifier.FakeNotifier{}},
nil,
config.Group{
Name: "Recording and alerting rules",
Rules: []config.Rule{
{Alert: "alert1", Expr: "up > 0"},
{Alert: "alert2", Expr: "up > 0"},
{Record: "record", Expr: "max(up)"},
},
},
"contains recording rules",
},
{
nil,
&remotewrite.Client{},
config.Group{
Name: "Recording and alerting rules",
Rules: []config.Rule{
{Record: "record1", Expr: "max(up)"},
{Record: "record2", Expr: "max(up)"},
{Alert: "alert", Expr: "up > 0"},
},
},
"contains alerting rules",
},
func TestManagerUpdate_Failure(t *testing.T) {
f := func(notifiers []notifier.Notifier, rw remotewrite.RWClient, cfg config.Group, errStrExpected string) {
t.Helper()
m := &manager{
groups: make(map[uint64]*rule.Group),
querierBuilder: &datasource.FakeQuerier{},
rw: rw,
}
if notifiers != nil {
m.notifiers = func() []notifier.Notifier { return notifiers }
}
err := m.update(context.Background(), []config.Group{cfg}, false)
if err == nil {
t.Fatalf("expected to get error; got nil")
}
errStr := err.Error()
if !strings.Contains(errStr, errStrExpected) {
t.Fatalf("missing %q in the error %q", errStrExpected, errStr)
}
}
for _, tc := range testCases {
t.Run(tc.cfg.Name, func(t *testing.T) {
m := &manager{
groups: make(map[uint64]*rule.Group),
querierBuilder: &datasource.FakeQuerier{},
rw: tc.rw,
}
if tc.notifiers != nil {
m.notifiers = func() []notifier.Notifier { return tc.notifiers }
}
err := m.update(context.Background(), []config.Group{tc.cfg}, false)
if err == nil {
t.Fatalf("expected to get error; got nil")
}
if !strings.Contains(err.Error(), tc.expErr) {
t.Fatalf("expected err to contain %q; got %q", tc.expErr, err)
}
})
}
f(nil, nil, config.Group{
Name: "Recording rule only",
Rules: []config.Rule{
{Record: "record", Expr: "max(up)"},
},
}, "contains recording rules")
f(nil, nil, config.Group{
Name: "Alerting rule only",
Rules: []config.Rule{
{Alert: "alert", Expr: "up > 0"},
},
}, "contains alerting rules")
f([]notifier.Notifier{&notifier.FakeNotifier{}}, nil, config.Group{
Name: "Recording and alerting rules",
Rules: []config.Rule{
{Alert: "alert1", Expr: "up > 0"},
{Alert: "alert2", Expr: "up > 0"},
{Record: "record", Expr: "max(up)"},
},
}, "contains recording rules")
f(nil, &remotewrite.Client{}, config.Group{
Name: "Recording and alerting rules",
Rules: []config.Rule{
{Record: "record1", Expr: "max(up)"},
{Record: "record2", Expr: "max(up)"},
{Alert: "alert", Expr: "up > 0"},
},
}, "contains alerting rules")
}
func loadCfg(t *testing.T, path []string, validateAnnotations, validateExpressions bool) []config.Group {

View File

@@ -11,7 +11,7 @@ import (
"github.com/VictoriaMetrics/VictoriaMetrics/lib/promrelabel"
)
func TestAlert_ExecTemplate(t *testing.T) {
func TestAlertExecTemplate(t *testing.T) {
extLabels := make(map[string]string)
const (
extCluster = "prod"
@@ -23,201 +23,164 @@ func TestAlert_ExecTemplate(t *testing.T) {
_, err := Init(nil, extLabels, extURL)
checkErr(t, err)
testCases := []struct {
name string
alert *Alert
annotations map[string]string
expTpl map[string]string
}{
{
name: "empty-alert",
alert: &Alert{},
annotations: map[string]string{},
expTpl: map[string]string{},
},
{
name: "no-template",
alert: &Alert{
Value: 1e4,
Labels: map[string]string{
"instance": "localhost",
f := func(alert *Alert, annotations map[string]string, tplExpected map[string]string) {
t.Helper()
if err := ValidateTemplates(annotations); err != nil {
t.Fatalf("cannot validate annotations: %s", err)
}
qFn := func(_ string) ([]datasource.Metric, error) {
return []datasource.Metric{
{
Labels: []datasource.Label{
{Name: "foo", Value: "bar"},
{Name: "baz", Value: "qux"},
},
Values: []float64{1},
Timestamps: []int64{1},
},
},
annotations: map[string]string{},
expTpl: map[string]string{},
},
{
name: "label-template",
alert: &Alert{
Value: 1e4,
Labels: map[string]string{
"job": "staging",
"instance": "localhost",
{
Labels: []datasource.Label{
{Name: "foo", Value: "garply"},
{Name: "baz", Value: "fred"},
},
Values: []float64{2},
Timestamps: []int64{1},
},
For: 5 * time.Minute,
},
annotations: map[string]string{
"summary": "Too high connection number for {{$labels.instance}} for job {{$labels.job}}",
"description": "It is {{ $value }} connections for {{$labels.instance}} for more than {{ .For }}",
},
expTpl: map[string]string{
"summary": "Too high connection number for localhost for job staging",
"description": "It is 10000 connections for localhost for more than 5m0s",
},
},
{
name: "expression-template",
alert: &Alert{
Expr: `vm_rows{"label"="bar"}<0`,
},
annotations: map[string]string{
"exprEscapedQuery": "{{ $expr|queryEscape }}",
"exprEscapedPath": "{{ $expr|pathEscape }}",
"exprEscapedJSON": "{{ $expr|jsonEscape }}",
"exprEscapedQuotes": "{{ $expr|quotesEscape }}",
"exprEscapedHTML": "{{ $expr|htmlEscape }}",
},
expTpl: map[string]string{
"exprEscapedQuery": "vm_rows%7B%22label%22%3D%22bar%22%7D%3C0",
"exprEscapedPath": "vm_rows%7B%22label%22=%22bar%22%7D%3C0",
"exprEscapedJSON": `"vm_rows{\"label\"=\"bar\"}\u003c0"`,
"exprEscapedQuotes": `vm_rows{\"label\"=\"bar\"}\u003c0`,
"exprEscapedHTML": "vm_rows{&quot;label&quot;=&quot;bar&quot;}&lt;0",
},
},
{
name: "query",
alert: &Alert{Expr: `vm_rows{"label"="bar"}>0`},
annotations: map[string]string{
"summary": `{{ query "foo" | first | value }}`,
"desc": `{{ range query "bar" }}{{ . | label "foo" }} {{ . | value }};{{ end }}`,
},
expTpl: map[string]string{
"summary": "1",
"desc": "bar 1;garply 2;",
},
},
{
name: "external",
alert: &Alert{
Value: 1e4,
Labels: map[string]string{
"job": "staging",
"instance": "localhost",
},
},
annotations: map[string]string{
"url": "{{ $externalURL }}",
"summary": "Issues with {{$labels.instance}} (dc-{{$externalLabels.dc}}) for job {{$labels.job}}",
"description": "It is {{ $value }} connections for {{$labels.instance}} (cluster-{{$externalLabels.cluster}})",
},
expTpl: map[string]string{
"url": extURL,
"summary": fmt.Sprintf("Issues with localhost (dc-%s) for job staging", extDC),
"description": fmt.Sprintf("It is 10000 connections for localhost (cluster-%s)", extCluster),
},
},
{
name: "alert and group IDs",
alert: &Alert{
ID: 42,
GroupID: 24,
},
annotations: map[string]string{
"url": "/api/v1/alert?alertID={{$alertID}}&groupID={{$groupID}}",
},
expTpl: map[string]string{
"url": "/api/v1/alert?alertID=42&groupID=24",
},
},
{
name: "ActiveAt time",
alert: &Alert{
ActiveAt: time.Date(2022, 8, 19, 20, 34, 58, 651387237, time.UTC),
},
annotations: map[string]string{
"diagram": "![](http://example.com?render={{$activeAt.Unix}}",
},
expTpl: map[string]string{
"diagram": "![](http://example.com?render=1660941298",
},
},
{
name: "ActiveAt time is nil",
alert: &Alert{},
annotations: map[string]string{
"default_time": "{{$activeAt}}",
},
expTpl: map[string]string{
"default_time": "0001-01-01 00:00:00 +0000 UTC",
},
},
{
name: "ActiveAt custom format",
alert: &Alert{
ActiveAt: time.Date(2022, 8, 19, 20, 34, 58, 651387237, time.UTC),
},
annotations: map[string]string{
"fire_time": `{{$activeAt.Format "2006/01/02 15:04:05"}}`,
},
expTpl: map[string]string{
"fire_time": "2022/08/19 20:34:58",
},
},
{
name: "ActiveAt query range",
alert: &Alert{
ActiveAt: time.Date(2022, 8, 19, 20, 34, 58, 651387237, time.UTC),
},
annotations: map[string]string{
"grafana_url": `vm-grafana.com?from={{($activeAt.Add (parseDurationTime "1h")).Unix}}&to={{($activeAt.Add (parseDurationTime "-1h")).Unix}}`,
},
expTpl: map[string]string{
"grafana_url": "vm-grafana.com?from=1660944898&to=1660937698",
},
},
}, nil
}
tpl, err := alert.ExecTemplate(qFn, alert.Labels, annotations)
if err != nil {
t.Fatalf("cannot execute template: %s", err)
}
if len(tpl) != len(tplExpected) {
t.Fatalf("unexpected number of elements; got %d; want %d", len(tpl), len(tplExpected))
}
for k := range tplExpected {
got, exp := tpl[k], tplExpected[k]
if got != exp {
t.Fatalf("unexpected template for key=%q; got %q; want %q", k, got, exp)
}
}
}
qFn := func(_ string) ([]datasource.Metric, error) {
return []datasource.Metric{
{
Labels: []datasource.Label{
{Name: "foo", Value: "bar"},
{Name: "baz", Value: "qux"},
},
Values: []float64{1},
Timestamps: []int64{1},
},
{
Labels: []datasource.Label{
{Name: "foo", Value: "garply"},
{Name: "baz", Value: "fred"},
},
Values: []float64{2},
Timestamps: []int64{1},
},
}, nil
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
if err := ValidateTemplates(tc.annotations); err != nil {
t.Fatal(err)
}
tpl, err := tc.alert.ExecTemplate(qFn, tc.alert.Labels, tc.annotations)
if err != nil {
t.Fatal(err)
}
if len(tpl) != len(tc.expTpl) {
t.Fatalf("expected %d elements; got %d", len(tc.expTpl), len(tpl))
}
for k := range tc.expTpl {
got, exp := tpl[k], tc.expTpl[k]
if got != exp {
t.Fatalf("expected %q=%q; got %q=%q", k, exp, k, got)
}
}
})
}
// empty-alert
f(&Alert{}, map[string]string{}, map[string]string{})
// no-template
f(&Alert{
Value: 1e4,
Labels: map[string]string{
"instance": "localhost",
},
}, map[string]string{}, map[string]string{})
// label-template
f(&Alert{
Value: 1e4,
Labels: map[string]string{
"job": "staging",
"instance": "localhost",
},
For: 5 * time.Minute,
}, map[string]string{
"summary": "Too high connection number for {{$labels.instance}} for job {{$labels.job}}",
"description": "It is {{ $value }} connections for {{$labels.instance}} for more than {{ .For }}",
}, map[string]string{
"summary": "Too high connection number for localhost for job staging",
"description": "It is 10000 connections for localhost for more than 5m0s",
})
// expression-template
f(&Alert{
Expr: `vm_rows{"label"="bar"}<0`,
}, map[string]string{
"exprEscapedQuery": "{{ $expr|queryEscape }}",
"exprEscapedPath": "{{ $expr|pathEscape }}",
"exprEscapedJSON": "{{ $expr|jsonEscape }}",
"exprEscapedQuotes": "{{ $expr|quotesEscape }}",
"exprEscapedHTML": "{{ $expr|htmlEscape }}",
}, map[string]string{
"exprEscapedQuery": "vm_rows%7B%22label%22%3D%22bar%22%7D%3C0",
"exprEscapedPath": "vm_rows%7B%22label%22=%22bar%22%7D%3C0",
"exprEscapedJSON": `"vm_rows{\"label\"=\"bar\"}\u003c0"`,
"exprEscapedQuotes": `vm_rows{\"label\"=\"bar\"}\u003c0`,
"exprEscapedHTML": "vm_rows{&quot;label&quot;=&quot;bar&quot;}&lt;0",
})
// query
f(&Alert{
Expr: `vm_rows{"label"="bar"}>0`,
}, map[string]string{
"summary": `{{ query "foo" | first | value }}`,
"desc": `{{ range query "bar" }}{{ . | label "foo" }} {{ . | value }};{{ end }}`,
}, map[string]string{
"summary": "1",
"desc": "bar 1;garply 2;",
})
// external
f(&Alert{
Value: 1e4,
Labels: map[string]string{
"job": "staging",
"instance": "localhost",
},
}, map[string]string{
"url": "{{ $externalURL }}",
"summary": "Issues with {{$labels.instance}} (dc-{{$externalLabels.dc}}) for job {{$labels.job}}",
"description": "It is {{ $value }} connections for {{$labels.instance}} (cluster-{{$externalLabels.cluster}})",
}, map[string]string{
"url": extURL,
"summary": fmt.Sprintf("Issues with localhost (dc-%s) for job staging", extDC),
"description": fmt.Sprintf("It is 10000 connections for localhost (cluster-%s)", extCluster),
})
// alert and group IDs
f(&Alert{
ID: 42,
GroupID: 24,
}, map[string]string{
"url": "/api/v1/alert?alertID={{$alertID}}&groupID={{$groupID}}",
}, map[string]string{
"url": "/api/v1/alert?alertID=42&groupID=24",
})
// ActiveAt time
f(&Alert{
ActiveAt: time.Date(2022, 8, 19, 20, 34, 58, 651387237, time.UTC),
}, map[string]string{
"diagram": "![](http://example.com?render={{$activeAt.Unix}}",
}, map[string]string{
"diagram": "![](http://example.com?render=1660941298",
})
// ActiveAt time is nil
f(&Alert{}, map[string]string{
"default_time": "{{$activeAt}}",
}, map[string]string{
"default_time": "0001-01-01 00:00:00 +0000 UTC",
})
// ActiveAt custom format
f(&Alert{
ActiveAt: time.Date(2022, 8, 19, 20, 34, 58, 651387237, time.UTC),
}, map[string]string{
"fire_time": `{{$activeAt.Format "2006/01/02 15:04:05"}}`,
}, map[string]string{
"fire_time": "2022/08/19 20:34:58",
})
// ActiveAt query range
f(&Alert{
ActiveAt: time.Date(2022, 8, 19, 20, 34, 58, 651387237, time.UTC),
}, map[string]string{
"grafana_url": `vm-grafana.com?from={{($activeAt.Add (parseDurationTime "1h")).Unix}}&to={{($activeAt.Add (parseDurationTime "-1h")).Unix}}`,
}, map[string]string{
"grafana_url": "vm-grafana.com?from=1660944898&to=1660937698",
})
}
func TestAlert_toPromLabels(t *testing.T) {

View File

@@ -16,10 +16,10 @@ func TestAlertManager_Addr(t *testing.T) {
const addr = "http://localhost"
am, err := NewAlertManager(addr, nil, promauth.HTTPClientConfig{}, nil, 0)
if err != nil {
t.Errorf("unexpected error: %s", err)
t.Fatalf("unexpected error: %s", err)
}
if am.Addr() != addr {
t.Errorf("expected to have %q; got %q", addr, am.Addr())
t.Fatalf("expected to have %q; got %q", addr, am.Addr())
}
}
@@ -28,21 +28,20 @@ func TestAlertManager_Send(t *testing.T) {
const headerKey, headerValue = "TenantID", "foo"
mux := http.NewServeMux()
mux.HandleFunc("/", func(_ http.ResponseWriter, _ *http.Request) {
t.Errorf("should not be called")
t.Fatalf("should not be called")
})
c := -1
mux.HandleFunc(alertManagerPath, func(w http.ResponseWriter, r *http.Request) {
user, pass, ok := r.BasicAuth()
if !ok {
t.Errorf("unauthorized request")
t.Fatalf("unauthorized request")
}
if user != baUser || pass != baPass {
t.Errorf("wrong creds %q:%q; expected %q:%q",
user, pass, baUser, baPass)
t.Fatalf("wrong creds %q:%q; expected %q:%q", user, pass, baUser, baPass)
}
c++
if r.Method != http.MethodPost {
t.Errorf("expected POST method got %s", r.Method)
t.Fatalf("expected POST method got %s", r.Method)
}
switch c {
case 0:
@@ -59,25 +58,23 @@ func TestAlertManager_Send(t *testing.T) {
GeneratorURL string `json:"generatorURL"`
}
if err := json.NewDecoder(r.Body).Decode(&a); err != nil {
t.Errorf("can not unmarshal data into alert %s", err)
t.FailNow()
t.Fatalf("can not unmarshal data into alert %s", err)
}
if len(a) != 1 {
t.Errorf("expected 1 alert in array got %d", len(a))
t.Fatalf("expected 1 alert in array got %d", len(a))
}
if a[0].GeneratorURL != "0/0" {
t.Errorf("expected 0/0 as generatorURL got %s", a[0].GeneratorURL)
t.Fatalf("expected 0/0 as generatorURL got %s", a[0].GeneratorURL)
}
if a[0].StartsAt.IsZero() {
t.Errorf("expected non-zero start time")
t.Fatalf("expected non-zero start time")
}
if a[0].EndAt.IsZero() {
t.Errorf("expected non-zero end time")
t.Fatalf("expected non-zero end time")
}
case 3:
if r.Header.Get(headerKey) != headerValue {
t.Errorf("expected header %q to be set to %q; got %q instead",
headerKey, headerValue, r.Header.Get(headerKey))
t.Fatalf("expected header %q to be set to %q; got %q instead", headerKey, headerValue, r.Header.Get(headerKey))
}
}
})
@@ -94,13 +91,13 @@ func TestAlertManager_Send(t *testing.T) {
return strconv.FormatUint(alert.GroupID, 10) + "/" + strconv.FormatUint(alert.ID, 10)
}, aCfg, nil, 0)
if err != nil {
t.Errorf("unexpected error: %s", err)
t.Fatalf("unexpected error: %s", err)
}
if err := am.Send(context.Background(), []Alert{{}, {}}, nil); err == nil {
t.Error("expected connection error got nil")
t.Fatalf("expected connection error got nil")
}
if err := am.Send(context.Background(), []Alert{}, nil); err == nil {
t.Error("expected wrong http code error got nil")
t.Fatalf("expected wrong http code error got nil")
}
if err := am.Send(context.Background(), []Alert{{
GroupID: 0,
@@ -109,12 +106,12 @@ func TestAlertManager_Send(t *testing.T) {
End: time.Now().UTC(),
Annotations: map[string]string{"a": "b", "c": "d", "e": "f"},
}}, nil); err != nil {
t.Errorf("unexpected error %s", err)
t.Fatalf("unexpected error %s", err)
}
if c != 2 {
t.Errorf("expected 2 calls(count from zero) to server got %d", c)
t.Fatalf("expected 2 calls(count from zero) to server got %d", c)
}
if err := am.Send(context.Background(), nil, map[string]string{headerKey: headerValue}); err != nil {
t.Errorf("unexpected error %s", err)
t.Fatalf("unexpected error %s", err)
}
}

View File

@@ -51,7 +51,7 @@ type Config struct {
Checksum string
// Catches all undefined fields and must be empty after parsing.
XXX map[string]interface{} `yaml:",inline"`
XXX map[string]any `yaml:",inline"`
// This is set to the directory from where the config has been loaded.
baseDir string
@@ -73,7 +73,7 @@ type StaticConfig struct {
}
// UnmarshalYAML implements the yaml.Unmarshaler interface.
func (cfg *Config) UnmarshalYAML(unmarshal func(interface{}) error) error {
func (cfg *Config) UnmarshalYAML(unmarshal func(any) error) error {
type config Config
if err := unmarshal((*config)(cfg)); err != nil {
return err

View File

@@ -5,10 +5,14 @@ import (
"testing"
)
func TestConfigParseGood(t *testing.T) {
func TestParseConfig_Success(t *testing.T) {
f := func(path string) {
t.Helper()
_, err := parseConfig(path)
checkErr(t, err)
if err != nil {
t.Fatalf("unexpected error: %s", err)
}
}
f("testdata/mixed.good.yaml")
f("testdata/consul.good.yaml")
@@ -16,14 +20,16 @@ func TestConfigParseGood(t *testing.T) {
f("testdata/static.good.yaml")
}
func TestConfigParseBad(t *testing.T) {
func TestParseConfig_Failure(t *testing.T) {
f := func(path, expErr string) {
t.Helper()
_, err := parseConfig(path)
if err == nil {
t.Fatalf("expected to get non-nil err for config %q", path)
}
if !strings.Contains(err.Error(), expErr) {
t.Errorf("expected err to contain %q; got %q instead", expErr, err)
t.Fatalf("expected err to contain %q; got %q instead", expErr, err)
}
}

View File

@@ -319,46 +319,41 @@ func TestMergeHTTPClientConfigs(t *testing.T) {
}
}
func TestParseLabels(t *testing.T) {
testCases := []struct {
name string
target string
cfg *Config
expectedAddress string
expectedErr bool
}{
{
"invalid address",
"invalid:*//url",
&Config{},
"",
true,
},
{
"use some default params",
"alertmanager:9093",
&Config{PathPrefix: "test"},
"http://alertmanager:9093/test/api/v2/alerts",
false,
},
{
"use target address",
"https://alertmanager:9093/api/v1/alerts",
&Config{Scheme: "http", PathPrefix: "test"},
"https://alertmanager:9093/api/v1/alerts",
false,
},
func TestParseLabels_Failure(t *testing.T) {
f := func(target string, cfg *Config) {
t.Helper()
_, _, err := parseLabels(target, nil, cfg)
if err == nil {
t.Fatalf("expecting non-nil error")
}
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
address, _, err := parseLabels(tc.target, nil, tc.cfg)
if err == nil == tc.expectedErr {
t.Fatalf("unexpected error; got %t; want %t", err != nil, tc.expectedErr)
}
if address != tc.expectedAddress {
t.Fatalf("unexpected address; got %q; want %q", address, tc.expectedAddress)
}
})
}
// invalid address
f("invalid:*//url", &Config{})
}
func TestParseLabels_Success(t *testing.T) {
f := func(target string, cfg *Config, expectedAddress string) {
t.Helper()
address, _, err := parseLabels(target, nil, cfg)
if err != nil {
t.Fatalf("unexpected error: %s", err)
}
if address != expectedAddress {
t.Fatalf("unexpected address; got %q; want %q", address, expectedAddress)
}
}
// use some default params
f("alertmanager:9093", &Config{
PathPrefix: "test",
}, "http://alertmanager:9093/test/api/v2/alerts")
// use target address
f("https://alertmanager:9093/api/v1/alerts", &Config{
Scheme: "http",
PathPrefix: "test",
}, "https://alertmanager:9093/api/v1/alerts")
}

View File

@@ -17,12 +17,12 @@ func TestBlackHoleNotifier_Send(t *testing.T) {
End: time.Now().UTC(),
Annotations: map[string]string{"a": "b", "c": "d", "e": "f"},
}}, nil); err != nil {
t.Errorf("unexpected error %s", err)
t.Fatalf("unexpected error %s", err)
}
alertCount := bh.metrics.alertsSent.Get()
if alertCount != 1 {
t.Errorf("expect value 1; instead got %d", alertCount)
t.Fatalf("expect value 1; instead got %d", alertCount)
}
}
@@ -35,7 +35,7 @@ func TestBlackHoleNotifier_Close(t *testing.T) {
End: time.Now().UTC(),
Annotations: map[string]string{"a": "b", "c": "d", "e": "f"},
}}, nil); err != nil {
t.Errorf("unexpected error %s", err)
t.Fatalf("unexpected error %s", err)
}
bh.Close()
@@ -44,7 +44,7 @@ func TestBlackHoleNotifier_Close(t *testing.T) {
alertMetricName := "vmalert_alerts_sent_total{addr=\"blackhole\"}"
for _, name := range defaultMetrics.ListMetricNames() {
if name == alertMetricName {
t.Errorf("Metric name should have unregistered.But still present")
t.Fatalf("Metric name should have unregistered.But still present")
}
}
}

View File

@@ -10,6 +10,7 @@ import (
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/utils"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/httputils"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/netutil"
)
var (
@@ -70,7 +71,7 @@ func Init() (datasource.QuerierBuilder, error) {
return nil, fmt.Errorf("failed to create transport: %w", err)
}
tr.IdleConnTimeout = *idleConnectionTimeout
tr.DialContext = httputils.GetStatDialFunc("vmalert_remoteread")
tr.DialContext = netutil.NewStatDialFunc("vmalert_remoteread")
endpointParams, err := flagutil.ParseJSONMap(*oauth2EndpointParams)
if err != nil {

View File

@@ -44,8 +44,7 @@ func TestClient_Push(t *testing.T) {
}
r := rand.New(rand.NewSource(1))
const rowsN = 1e4
var sent int
const rowsN = int(1e4)
for i := 0; i < rowsN; i++ {
s := prompbmarshal.TimeSeries{
Samples: []prompbmarshal.Sample{{
@@ -57,17 +56,11 @@ func TestClient_Push(t *testing.T) {
if err != nil {
t.Fatalf("unexpected err: %s", err)
}
if err == nil {
sent++
}
err = faultyClient.Push(s)
if err != nil {
t.Fatalf("unexpected err: %s", err)
}
}
if sent == 0 {
t.Fatalf("0 series sent")
}
if err := client.Close(); err != nil {
t.Fatalf("failed to close client: %s", err)
}
@@ -75,77 +68,66 @@ func TestClient_Push(t *testing.T) {
t.Fatalf("failed to close faulty client: %s", err)
}
got := testSrv.accepted()
if got != sent {
t.Fatalf("expected to have %d series; got %d", sent, got)
if got != rowsN {
t.Fatalf("expected to have %d series; got %d", rowsN, got)
}
got = faultySrv.accepted()
if got != sent {
t.Fatalf("expected to have %d series for faulty client; got %d", sent, got)
if got != rowsN {
t.Fatalf("expected to have %d series for faulty client; got %d", rowsN, got)
}
}
func TestClient_run_maxBatchSizeDuringShutdown(t *testing.T) {
batchSize := 20
const batchSize = 20
testTable := []struct {
name string // name of the test case
pushCnt int // how many time series is pushed to the client
batchCnt int // the expected batch count sent by the client
}{
{
name: "pushCnt % batchSize == 0",
pushCnt: batchSize * 40,
batchCnt: 40,
},
{
name: "pushCnt % batchSize != 0",
pushCnt: batchSize*40 + 1,
batchCnt: 40 + 1,
},
}
f := func(pushCnt, batchCntExpected int) {
t.Helper()
for _, tt := range testTable {
t.Run(tt.name, func(t *testing.T) {
// run new server
bcServer := newBatchCntRWServer()
// run new server
bcServer := newBatchCntRWServer()
// run new client
rwClient, err := NewClient(context.Background(), Config{
MaxBatchSize: batchSize,
// run new client
rwClient, err := NewClient(context.Background(), Config{
MaxBatchSize: batchSize,
// Set everything to 1 to simplify the calculation.
Concurrency: 1,
MaxQueueSize: 1000,
FlushInterval: time.Minute,
// Set everything to 1 to simplify the calculation.
Concurrency: 1,
MaxQueueSize: 1000,
FlushInterval: time.Minute,
// batch count server
Addr: bcServer.URL,
})
if err != nil {
t.Fatalf("new remote write client failed, err: %v", err)
}
// push time series to the client.
for i := 0; i < tt.pushCnt; i++ {
if err = rwClient.Push(prompbmarshal.TimeSeries{}); err != nil {
t.Fatalf("push time series to the client failed, err: %v", err)
}
}
// close the client so the rest ts will be flushed in `shutdown`
if err = rwClient.Close(); err != nil {
t.Fatalf("shutdown client failed, err: %v", err)
}
// finally check how many batches is sent.
if tt.batchCnt != bcServer.acceptedBatches() {
t.Errorf("client sent batch count incorrect, want: %d, get: %d", tt.batchCnt, bcServer.acceptedBatches())
}
if tt.pushCnt != bcServer.accepted() {
t.Errorf("client sent time series count incorrect, want: %d, get: %d", tt.pushCnt, bcServer.accepted())
}
// batch count server
Addr: bcServer.URL,
})
if err != nil {
t.Fatalf("cannot create remote write client: %s", err)
}
// push time series to the client.
for i := 0; i < pushCnt; i++ {
if err = rwClient.Push(prompbmarshal.TimeSeries{}); err != nil {
t.Fatalf("cannot time series to the client: %s", err)
}
}
// close the client so the rest ts will be flushed in `shutdown`
if err = rwClient.Close(); err != nil {
t.Fatalf("cannot shutdown client: %s", err)
}
// finally check how many batches is sent.
if bcServer.acceptedBatches() != batchCntExpected {
t.Fatalf("client sent batch count incorrect; got %d; want %d", bcServer.acceptedBatches(), batchCntExpected)
}
if pushCnt != bcServer.accepted() {
t.Fatalf("client sent time series count incorrect; got %d; want %d", bcServer.accepted(), pushCnt)
}
}
// pushCnt % batchSize == 0
f(batchSize*40, 40)
//pushCnt % batchSize != 0
f(batchSize*40+1, 40+1)
}
func newRWServer() *rwServer {

View File

@@ -9,6 +9,7 @@ import (
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/utils"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/httputils"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/netutil"
)
var (
@@ -74,7 +75,7 @@ func Init(ctx context.Context) (*Client, error) {
return nil, fmt.Errorf("failed to create transport: %w", err)
}
t.IdleConnTimeout = *idleConnectionTimeout
t.DialContext = httputils.GetStatDialFunc("vmalert_remotewrite")
t.DialContext = netutil.NewStatDialFunc("vmalert_remotewrite")
endpointParams, err := flagutil.ParseJSONMap(*oauth2EndpointParams)
if err != nil {

View File

@@ -39,135 +39,102 @@ func (fr *fakeReplayQuerier) QueryRange(_ context.Context, q string, from, to ti
}
func TestReplay(t *testing.T) {
testCases := []struct {
name string
from, to string
maxDP int
cfg []config.Group
qb *fakeReplayQuerier
}{
{
name: "one rule + one response",
from: "2021-01-01T12:00:00.000Z",
to: "2021-01-01T12:02:00.000Z",
maxDP: 10,
cfg: []config.Group{
{Rules: []config.Rule{{Record: "foo", Expr: "sum(up)"}}},
},
qb: &fakeReplayQuerier{
registry: map[string]map[string]struct{}{
"sum(up)": {"12:00:00+12:02:00": {}},
},
},
},
{
name: "one rule + multiple responses",
from: "2021-01-01T12:00:00.000Z",
to: "2021-01-01T12:02:30.000Z",
maxDP: 1,
cfg: []config.Group{
{Rules: []config.Rule{{Record: "foo", Expr: "sum(up)"}}},
},
qb: &fakeReplayQuerier{
registry: map[string]map[string]struct{}{
"sum(up)": {
"12:00:00+12:01:00": {},
"12:01:00+12:02:00": {},
"12:02:00+12:02:30": {},
},
},
},
},
{
name: "datapoints per step",
from: "2021-01-01T12:00:00.000Z",
to: "2021-01-01T15:02:30.000Z",
maxDP: 60,
cfg: []config.Group{
{Interval: promutils.NewDuration(time.Minute), Rules: []config.Rule{{Record: "foo", Expr: "sum(up)"}}},
},
qb: &fakeReplayQuerier{
registry: map[string]map[string]struct{}{
"sum(up)": {
"12:00:00+13:00:00": {},
"13:00:00+14:00:00": {},
"14:00:00+15:00:00": {},
"15:00:00+15:02:30": {},
},
},
},
},
{
name: "multiple recording rules + multiple responses",
from: "2021-01-01T12:00:00.000Z",
to: "2021-01-01T12:02:30.000Z",
maxDP: 1,
cfg: []config.Group{
{Rules: []config.Rule{{Record: "foo", Expr: "sum(up)"}}},
{Rules: []config.Rule{{Record: "bar", Expr: "max(up)"}}},
},
qb: &fakeReplayQuerier{
registry: map[string]map[string]struct{}{
"sum(up)": {
"12:00:00+12:01:00": {},
"12:01:00+12:02:00": {},
"12:02:00+12:02:30": {},
},
"max(up)": {
"12:00:00+12:01:00": {},
"12:01:00+12:02:00": {},
"12:02:00+12:02:30": {},
},
},
},
},
{
name: "multiple alerting rules + multiple responses",
from: "2021-01-01T12:00:00.000Z",
to: "2021-01-01T12:02:30.000Z",
maxDP: 1,
cfg: []config.Group{
{Rules: []config.Rule{{Alert: "foo", Expr: "sum(up) > 1"}}},
{Rules: []config.Rule{{Alert: "bar", Expr: "max(up) < 1"}}},
},
qb: &fakeReplayQuerier{
registry: map[string]map[string]struct{}{
"sum(up) > 1": {
"12:00:00+12:01:00": {},
"12:01:00+12:02:00": {},
"12:02:00+12:02:30": {},
},
"max(up) < 1": {
"12:00:00+12:01:00": {},
"12:01:00+12:02:00": {},
"12:02:00+12:02:30": {},
},
},
},
},
f := func(from, to string, maxDP int, cfg []config.Group, qb *fakeReplayQuerier) {
t.Helper()
fromOrig, toOrig, maxDatapointsOrig := *replayFrom, *replayTo, *replayMaxDatapoints
retriesOrig, delayOrig := *replayRuleRetryAttempts, *replayRulesDelay
defer func() {
*replayFrom, *replayTo = fromOrig, toOrig
*replayMaxDatapoints, *replayRuleRetryAttempts = maxDatapointsOrig, retriesOrig
*replayRulesDelay = delayOrig
}()
*replayRuleRetryAttempts = 1
*replayRulesDelay = time.Millisecond
rwb := &remotewrite.DebugClient{}
*replayFrom = from
*replayTo = to
*replayMaxDatapoints = maxDP
if err := replay(cfg, qb, rwb); err != nil {
t.Fatalf("replay failed: %s", err)
}
if len(qb.registry) > 0 {
t.Fatalf("not all requests were sent: %#v", qb.registry)
}
}
from, to, maxDP := *replayFrom, *replayTo, *replayMaxDatapoints
retries, delay := *replayRuleRetryAttempts, *replayRulesDelay
defer func() {
*replayFrom, *replayTo = from, to
*replayMaxDatapoints, *replayRuleRetryAttempts = maxDP, retries
*replayRulesDelay = delay
}()
// one rule + one response
f("2021-01-01T12:00:00.000Z", "2021-01-01T12:02:00.000Z", 10, []config.Group{
{Rules: []config.Rule{{Record: "foo", Expr: "sum(up)"}}},
}, &fakeReplayQuerier{
registry: map[string]map[string]struct{}{
"sum(up)": {"12:00:00+12:02:00": {}},
},
})
*replayRuleRetryAttempts = 1
*replayRulesDelay = time.Millisecond
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
*replayFrom = tc.from
*replayTo = tc.to
*replayMaxDatapoints = tc.maxDP
if err := replay(tc.cfg, tc.qb, &remotewrite.DebugClient{}); err != nil {
t.Fatalf("replay failed: %s", err)
}
if len(tc.qb.registry) > 0 {
t.Fatalf("not all requests were sent: %#v", tc.qb.registry)
}
})
}
// one rule + multiple responses
f("2021-01-01T12:00:00.000Z", "2021-01-01T12:02:30.000Z", 1, []config.Group{
{Rules: []config.Rule{{Record: "foo", Expr: "sum(up)"}}},
}, &fakeReplayQuerier{
registry: map[string]map[string]struct{}{
"sum(up)": {
"12:00:00+12:01:00": {},
"12:01:00+12:02:00": {},
"12:02:00+12:02:30": {},
},
},
})
// datapoints per step
f("2021-01-01T12:00:00.000Z", "2021-01-01T15:02:30.000Z", 60, []config.Group{
{Interval: promutils.NewDuration(time.Minute), Rules: []config.Rule{{Record: "foo", Expr: "sum(up)"}}},
}, &fakeReplayQuerier{
registry: map[string]map[string]struct{}{
"sum(up)": {
"12:00:00+13:00:00": {},
"13:00:00+14:00:00": {},
"14:00:00+15:00:00": {},
"15:00:00+15:02:30": {},
},
},
})
// multiple recording rules + multiple responses
f("2021-01-01T12:00:00.000Z", "2021-01-01T12:02:30.000Z", 1, []config.Group{
{Rules: []config.Rule{{Record: "foo", Expr: "sum(up)"}}},
{Rules: []config.Rule{{Record: "bar", Expr: "max(up)"}}},
}, &fakeReplayQuerier{
registry: map[string]map[string]struct{}{
"sum(up)": {
"12:00:00+12:01:00": {},
"12:01:00+12:02:00": {},
"12:02:00+12:02:30": {},
},
"max(up)": {
"12:00:00+12:01:00": {},
"12:01:00+12:02:00": {},
"12:02:00+12:02:30": {},
},
},
})
// multiple alerting rules + multiple responses
f("2021-01-01T12:00:00.000Z", "2021-01-01T12:02:30.000Z", 1, []config.Group{
{Rules: []config.Rule{{Alert: "foo", Expr: "sum(up) > 1"}}},
{Rules: []config.Rule{{Alert: "bar", Expr: "max(up) < 1"}}},
}, &fakeReplayQuerier{
registry: map[string]map[string]struct{}{
"sum(up) > 1": {
"12:00:00+12:01:00": {},
"12:01:00+12:02:00": {},
"12:02:00+12:02:30": {},
},
"max(up) < 1": {
"12:00:00+12:01:00": {},
"12:01:00+12:02:00": {},
"12:02:00+12:02:30": {},
},
},
})
}

View File

@@ -183,7 +183,7 @@ func (ar *AlertingRule) GetAlert(id uint64) *notifier.Alert {
return ar.alerts[id]
}
func (ar *AlertingRule) logDebugf(at time.Time, a *notifier.Alert, format string, args ...interface{}) {
func (ar *AlertingRule) logDebugf(at time.Time, a *notifier.Alert, format string, args ...any) {
if !ar.Debug {
return
}

File diff suppressed because it is too large Load Diff

View File

@@ -475,7 +475,7 @@ func delayBeforeStart(ts time.Time, key uint64, interval time.Duration, offset *
return randSleep
}
func (g *Group) infof(format string, args ...interface{}) {
func (g *Group) infof(format string, args ...any) {
msg := fmt.Sprintf(format, args...)
logger.Infof("group %q %s; interval=%v; eval_offset=%v; concurrency=%d",
g.Name, msg, g.Interval, g.EvalOffset, g.Concurrency)

View File

@@ -37,158 +37,142 @@ func TestMain(m *testing.M) {
}
func TestUpdateWith(t *testing.T) {
testCases := []struct {
name string
currentRules []config.Rule
newRules []config.Rule
}{
{
"new rule",
nil,
[]config.Rule{{Alert: "bar"}},
},
{
"update alerting rule",
[]config.Rule{
{
Alert: "foo",
Expr: "up > 0",
For: promutils.NewDuration(time.Second),
Labels: map[string]string{
"bar": "baz",
},
Annotations: map[string]string{
"summary": "{{ $value|humanize }}",
"description": "{{$labels}}",
},
},
{
Alert: "bar",
Expr: "up > 0",
For: promutils.NewDuration(time.Second),
Labels: map[string]string{
"bar": "baz",
},
},
},
[]config.Rule{
{
Alert: "foo",
Expr: "up > 10",
For: promutils.NewDuration(time.Second),
Labels: map[string]string{
"baz": "bar",
},
Annotations: map[string]string{
"summary": "none",
},
},
{
Alert: "bar",
Expr: "up > 0",
For: promutils.NewDuration(2 * time.Second),
KeepFiringFor: promutils.NewDuration(time.Minute),
Labels: map[string]string{
"bar": "baz",
},
},
},
},
{
"update recording rule",
[]config.Rule{{
Record: "foo",
Expr: "max(up)",
Labels: map[string]string{
"bar": "baz",
},
}},
[]config.Rule{{
Record: "foo",
Expr: "min(up)",
Labels: map[string]string{
"baz": "bar",
},
}},
},
{
"empty rule",
[]config.Rule{{Alert: "foo"}, {Record: "bar"}},
nil,
},
{
"multiple rules",
[]config.Rule{
{Alert: "bar"},
{Alert: "baz"},
{Alert: "foo"},
},
[]config.Rule{
{Alert: "baz"},
{Record: "foo"},
},
},
{
"replace rule",
[]config.Rule{{Alert: "foo1"}},
[]config.Rule{{Alert: "foo2"}},
},
{
"replace multiple rules",
[]config.Rule{
{Alert: "foo1"},
{Record: "foo2"},
{Alert: "foo3"},
},
[]config.Rule{
{Alert: "foo3"},
{Alert: "foo4"},
{Record: "foo5"},
},
},
}
f := func(currentRules, newRules []config.Rule) {
t.Helper()
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
g := &Group{Name: "test"}
qb := &datasource.FakeQuerier{}
for _, r := range tc.currentRules {
r.ID = config.HashRule(r)
g.Rules = append(g.Rules, g.newRule(qb, r))
}
g := &Group{
Name: "test",
}
qb := &datasource.FakeQuerier{}
for _, r := range currentRules {
r.ID = config.HashRule(r)
g.Rules = append(g.Rules, g.newRule(qb, r))
}
ng := &Group{Name: "test"}
for _, r := range tc.newRules {
r.ID = config.HashRule(r)
ng.Rules = append(ng.Rules, ng.newRule(qb, r))
}
ng := &Group{
Name: "test",
}
for _, r := range newRules {
r.ID = config.HashRule(r)
ng.Rules = append(ng.Rules, ng.newRule(qb, r))
}
err := g.updateWith(ng)
if err != nil {
t.Fatal(err)
}
err := g.updateWith(ng)
if err != nil {
t.Fatalf("cannot update rule: %s", err)
}
if len(g.Rules) != len(tc.newRules) {
t.Fatalf("expected to have %d rules; got: %d",
len(g.Rules), len(tc.newRules))
}
sort.Slice(g.Rules, func(i, j int) bool {
return g.Rules[i].ID() < g.Rules[j].ID()
})
sort.Slice(ng.Rules, func(i, j int) bool {
return ng.Rules[i].ID() < ng.Rules[j].ID()
})
for i, r := range g.Rules {
got, want := r, ng.Rules[i]
if got.ID() != want.ID() {
t.Fatalf("expected to have rule %q; got %q", want, got)
}
if err := CompareRules(t, got, want); err != nil {
t.Fatalf("comparison error: %s", err)
}
}
if len(g.Rules) != len(newRules) {
t.Fatalf("expected to have %d rules; got: %d", len(g.Rules), len(newRules))
}
sort.Slice(g.Rules, func(i, j int) bool {
return g.Rules[i].ID() < g.Rules[j].ID()
})
sort.Slice(ng.Rules, func(i, j int) bool {
return ng.Rules[i].ID() < ng.Rules[j].ID()
})
for i, r := range g.Rules {
got, want := r, ng.Rules[i]
if got.ID() != want.ID() {
t.Fatalf("expected to have rule %q; got %q", want, got)
}
if err := CompareRules(t, got, want); err != nil {
t.Fatalf("comparison error: %s", err)
}
}
}
// new rule
f(nil, []config.Rule{
{Alert: "bar"},
})
// update alerting rule
f([]config.Rule{
{
Alert: "foo",
Expr: "up > 0",
For: promutils.NewDuration(time.Second),
Labels: map[string]string{
"bar": "baz",
},
Annotations: map[string]string{
"summary": "{{ $value|humanize }}",
"description": "{{$labels}}",
},
},
{
Alert: "bar",
Expr: "up > 0",
For: promutils.NewDuration(time.Second),
Labels: map[string]string{
"bar": "baz",
},
},
}, []config.Rule{
{
Alert: "foo",
Expr: "up > 10",
For: promutils.NewDuration(time.Second),
Labels: map[string]string{
"baz": "bar",
},
Annotations: map[string]string{
"summary": "none",
},
},
{
Alert: "bar",
Expr: "up > 0",
For: promutils.NewDuration(2 * time.Second),
KeepFiringFor: promutils.NewDuration(time.Minute),
Labels: map[string]string{
"bar": "baz",
},
},
})
// update recording rule
f([]config.Rule{{
Record: "foo",
Expr: "max(up)",
Labels: map[string]string{
"bar": "baz",
},
}}, []config.Rule{{
Record: "foo",
Expr: "min(up)",
Labels: map[string]string{
"baz": "bar",
},
}})
// empty rule
f([]config.Rule{{Alert: "foo"}, {Record: "bar"}}, nil)
// multiple rules
f([]config.Rule{
{Alert: "bar"},
{Alert: "baz"},
{Alert: "foo"},
}, []config.Rule{
{Alert: "baz"},
{Record: "foo"},
})
// replace rule
f([]config.Rule{{Alert: "foo1"}}, []config.Rule{{Alert: "foo2"}})
// replace multiple rules
f([]config.Rule{
{Alert: "foo1"},
{Record: "foo2"},
{Alert: "foo3"},
}, []config.Rule{
{Alert: "foo3"},
{Alert: "foo4"},
{Record: "foo5"},
})
}
func TestGroupStart(t *testing.T) {
@@ -312,30 +296,23 @@ func TestGroupStart(t *testing.T) {
<-finished
}
func TestResolveDuration(t *testing.T) {
testCases := []struct {
groupInterval time.Duration
maxDuration time.Duration
resendDelay time.Duration
expected time.Duration
}{
{time.Minute, 0, 0, 4 * time.Minute},
{time.Minute, 0, 2 * time.Minute, 8 * time.Minute},
{time.Minute, 4 * time.Minute, 4 * time.Minute, 4 * time.Minute},
{2 * time.Minute, time.Minute, 2 * time.Minute, time.Minute},
{time.Minute, 2 * time.Minute, 1 * time.Minute, 2 * time.Minute},
{2 * time.Minute, 0, 1 * time.Minute, 8 * time.Minute},
{0, 0, 0, 0},
func TestGetResolveDuration(t *testing.T) {
f := func(groupInterval, maxDuration, resendDelay, resultExpected time.Duration) {
t.Helper()
result := getResolveDuration(groupInterval, resendDelay, maxDuration)
if result != resultExpected {
t.Fatalf("unexpected result; got %s; want %s", result, resultExpected)
}
}
for _, tc := range testCases {
t.Run(fmt.Sprintf("%v-%v-%v", tc.groupInterval, tc.expected, tc.maxDuration), func(t *testing.T) {
got := getResolveDuration(tc.groupInterval, tc.resendDelay, tc.maxDuration)
if got != tc.expected {
t.Errorf("expected to have %v; got %v", tc.expected, got)
}
})
}
f(0, 0, 0, 0)
f(time.Minute, 0, 0, 4*time.Minute)
f(time.Minute, 0, 2*time.Minute, 8*time.Minute)
f(time.Minute, 4*time.Minute, 4*time.Minute, 4*time.Minute)
f(2*time.Minute, time.Minute, 2*time.Minute, time.Minute)
f(time.Minute, 2*time.Minute, 1*time.Minute, 2*time.Minute)
f(2*time.Minute, 0, 1*time.Minute, 8*time.Minute)
}
func TestGetStaleSeries(t *testing.T) {
@@ -345,6 +322,7 @@ func TestGetStaleSeries(t *testing.T) {
}
f := func(r Rule, labels, expLabels [][]prompbmarshal.Label) {
t.Helper()
var tss []prompbmarshal.TimeSeries
for _, l := range labels {
tss = append(tss, newTimeSeriesPB([]float64{1}, []int64{ts.Unix()}, l))
@@ -606,7 +584,7 @@ func TestGroupStartDelay(t *testing.T) {
delay := delayBeforeStart(at, key, g.Interval, g.EvalOffset)
gotStart := at.Add(delay)
if expTS != gotStart {
t.Errorf("expected to get %v; got %v instead", expTS, gotStart)
t.Fatalf("expected to get %v; got %v instead", expTS, gotStart)
}
}
@@ -647,157 +625,118 @@ func TestGroupStartDelay(t *testing.T) {
}
func TestGetPrometheusReqTimestamp(t *testing.T) {
f := func(g *Group, tsOrigin, tsExpected string) {
t.Helper()
originT, _ := time.Parse(time.RFC3339, tsOrigin)
expT, _ := time.Parse(time.RFC3339, tsExpected)
gotTS := g.adjustReqTimestamp(originT)
if !gotTS.Equal(expT) {
t.Fatalf("get wrong prometheus request timestamp: %s; want %s", gotTS, expT)
}
}
offset := 30 * time.Minute
evalDelay := 1 * time.Minute
disableAlign := false
testCases := []struct {
name string
g *Group
originTS, expTS string
}{
{
"with query align + default evalDelay",
&Group{
Interval: time.Hour,
},
"2023-08-28T11:11:00+00:00",
"2023-08-28T11:00:00+00:00",
},
{
"without query align + default evalDelay",
&Group{
Interval: time.Hour,
evalAlignment: &disableAlign,
},
"2023-08-28T11:11:00+00:00",
"2023-08-28T11:10:30+00:00",
},
{
"with eval_offset, find previous offset point + default evalDelay",
&Group{
EvalOffset: &offset,
Interval: time.Hour,
},
"2023-08-28T11:11:00+00:00",
"2023-08-28T10:30:00+00:00",
},
{
"with eval_offset + default evalDelay",
&Group{
EvalOffset: &offset,
Interval: time.Hour,
},
"2023-08-28T11:41:00+00:00",
"2023-08-28T11:30:00+00:00",
},
{
"1h interval with eval_delay",
&Group{
EvalDelay: &evalDelay,
Interval: time.Hour,
},
"2023-08-28T11:41:00+00:00",
"2023-08-28T11:00:00+00:00",
},
{
"1m interval with eval_delay",
&Group{
EvalDelay: &evalDelay,
Interval: time.Minute,
},
"2023-08-28T11:41:13+00:00",
"2023-08-28T11:40:00+00:00",
},
{
"disable alignment with eval_delay",
&Group{
EvalDelay: &evalDelay,
Interval: time.Hour,
evalAlignment: &disableAlign,
},
"2023-08-28T11:41:00+00:00",
"2023-08-28T11:40:00+00:00",
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
originT, _ := time.Parse(time.RFC3339, tc.originTS)
expT, _ := time.Parse(time.RFC3339, tc.expTS)
gotTS := tc.g.adjustReqTimestamp(originT)
if !gotTS.Equal(expT) {
t.Fatalf("get wrong prometheus request timestamp, expect %s, got %s", expT, gotTS)
}
})
}
// with query align + default evalDelay
f(&Group{
Interval: time.Hour,
}, "2023-08-28T11:11:00+00:00", "2023-08-28T11:00:00+00:00")
// without query align + default evalDelay
f(&Group{
Interval: time.Hour,
evalAlignment: &disableAlign,
}, "2023-08-28T11:11:00+00:00", "2023-08-28T11:10:30+00:00")
// with eval_offset, find previous offset point + default evalDelay
f(&Group{
EvalOffset: &offset,
Interval: time.Hour,
}, "2023-08-28T11:11:00+00:00", "2023-08-28T10:30:00+00:00")
// with eval_offset + default evalDelay
f(&Group{
EvalOffset: &offset,
Interval: time.Hour,
}, "2023-08-28T11:41:00+00:00", "2023-08-28T11:30:00+00:00")
// 1h interval with eval_delay
f(&Group{
EvalDelay: &evalDelay,
Interval: time.Hour,
}, "2023-08-28T11:41:00+00:00", "2023-08-28T11:00:00+00:00")
// 1m interval with eval_delay
f(&Group{
EvalDelay: &evalDelay,
Interval: time.Minute,
}, "2023-08-28T11:41:13+00:00", "2023-08-28T11:40:00+00:00")
// disable alignment with eval_delay
f(&Group{
EvalDelay: &evalDelay,
Interval: time.Hour,
evalAlignment: &disableAlign,
}, "2023-08-28T11:41:00+00:00", "2023-08-28T11:40:00+00:00")
}
func TestRangeIterator(t *testing.T) {
testCases := []struct {
ri rangeIterator
result [][2]time.Time
}{
{
ri: rangeIterator{
start: parseTime(t, "2021-01-01T12:00:00.000Z"),
end: parseTime(t, "2021-01-01T12:30:00.000Z"),
step: 5 * time.Minute,
},
result: [][2]time.Time{
{parseTime(t, "2021-01-01T12:00:00.000Z"), parseTime(t, "2021-01-01T12:05:00.000Z")},
{parseTime(t, "2021-01-01T12:05:00.000Z"), parseTime(t, "2021-01-01T12:10:00.000Z")},
{parseTime(t, "2021-01-01T12:10:00.000Z"), parseTime(t, "2021-01-01T12:15:00.000Z")},
{parseTime(t, "2021-01-01T12:15:00.000Z"), parseTime(t, "2021-01-01T12:20:00.000Z")},
{parseTime(t, "2021-01-01T12:20:00.000Z"), parseTime(t, "2021-01-01T12:25:00.000Z")},
{parseTime(t, "2021-01-01T12:25:00.000Z"), parseTime(t, "2021-01-01T12:30:00.000Z")},
},
},
{
ri: rangeIterator{
start: parseTime(t, "2021-01-01T12:00:00.000Z"),
end: parseTime(t, "2021-01-01T12:30:00.000Z"),
step: 45 * time.Minute,
},
result: [][2]time.Time{
{parseTime(t, "2021-01-01T12:00:00.000Z"), parseTime(t, "2021-01-01T12:30:00.000Z")},
{parseTime(t, "2021-01-01T12:30:00.000Z"), parseTime(t, "2021-01-01T12:30:00.000Z")},
},
},
{
ri: rangeIterator{
start: parseTime(t, "2021-01-01T12:00:12.000Z"),
end: parseTime(t, "2021-01-01T12:00:17.000Z"),
step: time.Second,
},
result: [][2]time.Time{
{parseTime(t, "2021-01-01T12:00:12.000Z"), parseTime(t, "2021-01-01T12:00:13.000Z")},
{parseTime(t, "2021-01-01T12:00:13.000Z"), parseTime(t, "2021-01-01T12:00:14.000Z")},
{parseTime(t, "2021-01-01T12:00:14.000Z"), parseTime(t, "2021-01-01T12:00:15.000Z")},
{parseTime(t, "2021-01-01T12:00:15.000Z"), parseTime(t, "2021-01-01T12:00:16.000Z")},
{parseTime(t, "2021-01-01T12:00:16.000Z"), parseTime(t, "2021-01-01T12:00:17.000Z")},
},
},
f := func(ri rangeIterator, resultExpected [][2]time.Time) {
t.Helper()
var j int
for ri.next() {
if len(resultExpected) < j+1 {
t.Fatalf("unexpected result for iterator on step %d: %v - %v", j, ri.s, ri.e)
}
s, e := ri.s, ri.e
expS, expE := resultExpected[j][0], resultExpected[j][1]
if s != expS {
t.Fatalf("expected to get start=%v; got %v", expS, s)
}
if e != expE {
t.Fatalf("expected to get end=%v; got %v", expE, e)
}
j++
}
}
for i, tc := range testCases {
t.Run(fmt.Sprintf("case %d", i), func(t *testing.T) {
var j int
for tc.ri.next() {
if len(tc.result) < j+1 {
t.Fatalf("unexpected result for iterator on step %d: %v - %v",
j, tc.ri.s, tc.ri.e)
}
s, e := tc.ri.s, tc.ri.e
expS, expE := tc.result[j][0], tc.result[j][1]
if s != expS {
t.Fatalf("expected to get start=%v; got %v", expS, s)
}
if e != expE {
t.Fatalf("expected to get end=%v; got %v", expE, e)
}
j++
}
})
}
f(rangeIterator{
start: parseTime(t, "2021-01-01T12:00:00.000Z"),
end: parseTime(t, "2021-01-01T12:30:00.000Z"),
step: 5 * time.Minute,
}, [][2]time.Time{
{parseTime(t, "2021-01-01T12:00:00.000Z"), parseTime(t, "2021-01-01T12:05:00.000Z")},
{parseTime(t, "2021-01-01T12:05:00.000Z"), parseTime(t, "2021-01-01T12:10:00.000Z")},
{parseTime(t, "2021-01-01T12:10:00.000Z"), parseTime(t, "2021-01-01T12:15:00.000Z")},
{parseTime(t, "2021-01-01T12:15:00.000Z"), parseTime(t, "2021-01-01T12:20:00.000Z")},
{parseTime(t, "2021-01-01T12:20:00.000Z"), parseTime(t, "2021-01-01T12:25:00.000Z")},
{parseTime(t, "2021-01-01T12:25:00.000Z"), parseTime(t, "2021-01-01T12:30:00.000Z")},
})
f(rangeIterator{
start: parseTime(t, "2021-01-01T12:00:00.000Z"),
end: parseTime(t, "2021-01-01T12:30:00.000Z"),
step: 45 * time.Minute,
}, [][2]time.Time{
{parseTime(t, "2021-01-01T12:00:00.000Z"), parseTime(t, "2021-01-01T12:30:00.000Z")},
{parseTime(t, "2021-01-01T12:30:00.000Z"), parseTime(t, "2021-01-01T12:30:00.000Z")},
})
f(rangeIterator{
start: parseTime(t, "2021-01-01T12:00:12.000Z"),
end: parseTime(t, "2021-01-01T12:00:17.000Z"),
step: time.Second,
}, [][2]time.Time{
{parseTime(t, "2021-01-01T12:00:12.000Z"), parseTime(t, "2021-01-01T12:00:13.000Z")},
{parseTime(t, "2021-01-01T12:00:13.000Z"), parseTime(t, "2021-01-01T12:00:14.000Z")},
{parseTime(t, "2021-01-01T12:00:14.000Z"), parseTime(t, "2021-01-01T12:00:15.000Z")},
{parseTime(t, "2021-01-01T12:00:15.000Z"), parseTime(t, "2021-01-01T12:00:16.000Z")},
{parseTime(t, "2021-01-01T12:00:16.000Z"), parseTime(t, "2021-01-01T12:00:17.000Z")},
})
}
func parseTime(t *testing.T, s string) time.Time {

View File

@@ -13,218 +13,225 @@ import (
)
func TestRecordingRule_Exec(t *testing.T) {
f := func(rule *RecordingRule, metrics []datasource.Metric, tssExpected []prompbmarshal.TimeSeries) {
t.Helper()
fq := &datasource.FakeQuerier{}
fq.Add(metrics...)
rule.q = fq
rule.state = &ruleState{
entries: make([]StateEntry, 10),
}
tss, err := rule.exec(context.TODO(), time.Now(), 0)
if err != nil {
t.Fatalf("unexpected RecordingRule.exec error: %s", err)
}
if err := compareTimeSeries(t, tssExpected, tss); err != nil {
t.Fatalf("timeseries missmatch: %s", err)
}
}
timestamp := time.Now()
testCases := []struct {
rule *RecordingRule
metrics []datasource.Metric
expTS []prompbmarshal.TimeSeries
}{
{
&RecordingRule{Name: "foo"},
[]datasource.Metric{metricWithValueAndLabels(t, 10,
"__name__", "bar",
)},
[]prompbmarshal.TimeSeries{
newTimeSeries([]float64{10}, []int64{timestamp.UnixNano()}, map[string]string{
"__name__": "foo",
}),
},
f(&RecordingRule{
Name: "foo",
}, []datasource.Metric{
metricWithValueAndLabels(t, 10, "__name__", "bar"),
}, []prompbmarshal.TimeSeries{
newTimeSeries([]float64{10}, []int64{timestamp.UnixNano()}, map[string]string{
"__name__": "foo",
}),
})
f(&RecordingRule{
Name: "foobarbaz",
}, []datasource.Metric{
metricWithValueAndLabels(t, 1, "__name__", "foo", "job", "foo"),
metricWithValueAndLabels(t, 2, "__name__", "bar", "job", "bar"),
metricWithValueAndLabels(t, 3, "__name__", "baz", "job", "baz"),
}, []prompbmarshal.TimeSeries{
newTimeSeries([]float64{1}, []int64{timestamp.UnixNano()}, map[string]string{
"__name__": "foobarbaz",
"job": "foo",
}),
newTimeSeries([]float64{2}, []int64{timestamp.UnixNano()}, map[string]string{
"__name__": "foobarbaz",
"job": "bar",
}),
newTimeSeries([]float64{3}, []int64{timestamp.UnixNano()}, map[string]string{
"__name__": "foobarbaz",
"job": "baz",
}),
})
f(&RecordingRule{
Name: "job:foo",
Labels: map[string]string{
"source": "test",
},
{
&RecordingRule{Name: "foobarbaz"},
[]datasource.Metric{
metricWithValueAndLabels(t, 1, "__name__", "foo", "job", "foo"),
metricWithValueAndLabels(t, 2, "__name__", "bar", "job", "bar"),
metricWithValueAndLabels(t, 3, "__name__", "baz", "job", "baz"),
},
[]prompbmarshal.TimeSeries{
newTimeSeries([]float64{1}, []int64{timestamp.UnixNano()}, map[string]string{
"__name__": "foobarbaz",
"job": "foo",
}),
newTimeSeries([]float64{2}, []int64{timestamp.UnixNano()}, map[string]string{
"__name__": "foobarbaz",
"job": "bar",
}),
newTimeSeries([]float64{3}, []int64{timestamp.UnixNano()}, map[string]string{
"__name__": "foobarbaz",
"job": "baz",
}),
},
},
{
&RecordingRule{
Name: "job:foo",
Labels: map[string]string{
"source": "test",
},
},
[]datasource.Metric{
metricWithValueAndLabels(t, 2, "__name__", "foo", "job", "foo"),
metricWithValueAndLabels(t, 1, "__name__", "bar", "job", "bar", "source", "origin"),
},
[]prompbmarshal.TimeSeries{
newTimeSeries([]float64{2}, []int64{timestamp.UnixNano()}, map[string]string{
"__name__": "job:foo",
"job": "foo",
"source": "test",
}),
newTimeSeries([]float64{1}, []int64{timestamp.UnixNano()}, map[string]string{
"__name__": "job:foo",
"job": "bar",
"source": "test",
"exported_source": "origin",
}),
},
},
}
for _, tc := range testCases {
t.Run(tc.rule.Name, func(t *testing.T) {
fq := &datasource.FakeQuerier{}
fq.Add(tc.metrics...)
tc.rule.q = fq
tc.rule.state = &ruleState{entries: make([]StateEntry, 10)}
tss, err := tc.rule.exec(context.TODO(), time.Now(), 0)
if err != nil {
t.Fatalf("unexpected Exec err: %s", err)
}
if err := compareTimeSeries(t, tc.expTS, tss); err != nil {
t.Fatalf("timeseries missmatch: %s", err)
}
})
}
}, []datasource.Metric{
metricWithValueAndLabels(t, 2, "__name__", "foo", "job", "foo"),
metricWithValueAndLabels(t, 1, "__name__", "bar", "job", "bar", "source", "origin"),
}, []prompbmarshal.TimeSeries{
newTimeSeries([]float64{2}, []int64{timestamp.UnixNano()}, map[string]string{
"__name__": "job:foo",
"job": "foo",
"source": "test",
}),
newTimeSeries([]float64{1}, []int64{timestamp.UnixNano()}, map[string]string{
"__name__": "job:foo",
"job": "bar",
"source": "test",
"exported_source": "origin",
}),
})
}
func TestRecordingRule_ExecRange(t *testing.T) {
timestamp := time.Now()
testCases := []struct {
rule *RecordingRule
metrics []datasource.Metric
expTS []prompbmarshal.TimeSeries
}{
{
&RecordingRule{Name: "foo"},
[]datasource.Metric{metricWithValuesAndLabels(t, []float64{10, 20, 30},
"__name__", "bar",
)},
[]prompbmarshal.TimeSeries{
newTimeSeries([]float64{10, 20, 30},
[]int64{timestamp.UnixNano(), timestamp.UnixNano(), timestamp.UnixNano()},
map[string]string{
"__name__": "foo",
}),
},
},
{
&RecordingRule{Name: "foobarbaz"},
[]datasource.Metric{
metricWithValuesAndLabels(t, []float64{1}, "__name__", "foo", "job", "foo"),
metricWithValuesAndLabels(t, []float64{2, 3}, "__name__", "bar", "job", "bar"),
metricWithValuesAndLabels(t, []float64{4, 5, 6}, "__name__", "baz", "job", "baz"),
},
[]prompbmarshal.TimeSeries{
newTimeSeries([]float64{1}, []int64{timestamp.UnixNano()}, map[string]string{
"__name__": "foobarbaz",
"job": "foo",
}),
newTimeSeries([]float64{2, 3}, []int64{timestamp.UnixNano(), timestamp.UnixNano()}, map[string]string{
"__name__": "foobarbaz",
"job": "bar",
}),
newTimeSeries([]float64{4, 5, 6},
[]int64{timestamp.UnixNano(), timestamp.UnixNano(), timestamp.UnixNano()},
map[string]string{
"__name__": "foobarbaz",
"job": "baz",
}),
},
},
{
&RecordingRule{Name: "job:foo", Labels: map[string]string{
"source": "test",
}},
[]datasource.Metric{
metricWithValueAndLabels(t, 2, "__name__", "foo", "job", "foo"),
metricWithValueAndLabels(t, 1, "__name__", "bar", "job", "bar"),
},
[]prompbmarshal.TimeSeries{
newTimeSeries([]float64{2}, []int64{timestamp.UnixNano()}, map[string]string{
"__name__": "job:foo",
"job": "foo",
"source": "test",
}),
newTimeSeries([]float64{1}, []int64{timestamp.UnixNano()}, map[string]string{
"__name__": "job:foo",
"job": "bar",
"source": "test",
}),
},
},
}
for _, tc := range testCases {
t.Run(tc.rule.Name, func(t *testing.T) {
fq := &datasource.FakeQuerier{}
fq.Add(tc.metrics...)
tc.rule.q = fq
tss, err := tc.rule.execRange(context.TODO(), time.Now(), time.Now())
if err != nil {
t.Fatalf("unexpected Exec err: %s", err)
}
if err := compareTimeSeries(t, tc.expTS, tss); err != nil {
t.Fatalf("timeseries missmatch: %s", err)
}
})
}
}
f := func(rule *RecordingRule, metrics []datasource.Metric, tssExpected []prompbmarshal.TimeSeries) {
t.Helper()
func TestRecordingRuleLimit(t *testing.T) {
timestamp := time.Now()
testCases := []struct {
limit int
err string
}{
{
limit: 0,
},
{
limit: -1,
},
{
limit: 1,
err: "exec exceeded limit of 1 with 3 series",
},
{
limit: 2,
err: "exec exceeded limit of 2 with 3 series",
},
fq := &datasource.FakeQuerier{}
fq.Add(metrics...)
rule.q = fq
tss, err := rule.execRange(context.TODO(), time.Now(), time.Now())
if err != nil {
t.Fatalf("unexpected RecordingRule.execRange error: %s", err)
}
if err := compareTimeSeries(t, tssExpected, tss); err != nil {
t.Fatalf("timeseries missmatch: %s", err)
}
}
testMetrics := []datasource.Metric{
timestamp := time.Now()
f(&RecordingRule{
Name: "foo",
}, []datasource.Metric{
metricWithValuesAndLabels(t, []float64{10, 20, 30}, "__name__", "bar"),
}, []prompbmarshal.TimeSeries{
newTimeSeries([]float64{10, 20, 30}, []int64{timestamp.UnixNano(), timestamp.UnixNano(), timestamp.UnixNano()}, map[string]string{
"__name__": "foo",
}),
})
f(&RecordingRule{
Name: "foobarbaz",
}, []datasource.Metric{
metricWithValuesAndLabels(t, []float64{1}, "__name__", "foo", "job", "foo"),
metricWithValuesAndLabels(t, []float64{2, 3}, "__name__", "bar", "job", "bar"),
metricWithValuesAndLabels(t, []float64{4, 5, 6}, "__name__", "baz", "job", "baz"),
}
rule := &RecordingRule{Name: "job:foo",
state: &ruleState{entries: make([]StateEntry, 10)},
}, []prompbmarshal.TimeSeries{
newTimeSeries([]float64{1}, []int64{timestamp.UnixNano()}, map[string]string{
"__name__": "foobarbaz",
"job": "foo",
}),
newTimeSeries([]float64{2, 3}, []int64{timestamp.UnixNano(), timestamp.UnixNano()}, map[string]string{
"__name__": "foobarbaz",
"job": "bar",
}),
newTimeSeries([]float64{4, 5, 6},
[]int64{timestamp.UnixNano(), timestamp.UnixNano(), timestamp.UnixNano()}, map[string]string{
"__name__": "foobarbaz",
"job": "baz",
}),
})
f(&RecordingRule{
Name: "job:foo",
Labels: map[string]string{
"source": "test_limit",
"source": "test",
},
metrics: &recordingRuleMetrics{
errors: utils.GetOrCreateCounter(`vmalert_recording_rules_errors_total{alertname="job:foo"}`),
},
}
var err error
for _, testCase := range testCases {
fq := &datasource.FakeQuerier{}
fq.Add(testMetrics...)
rule.q = fq
_, err = rule.exec(context.TODO(), timestamp, testCase.limit)
if err != nil && !strings.EqualFold(err.Error(), testCase.err) {
t.Fatal(err)
}
}
}, []datasource.Metric{
metricWithValueAndLabels(t, 2, "__name__", "foo", "job", "foo"),
metricWithValueAndLabels(t, 1, "__name__", "bar", "job", "bar"),
}, []prompbmarshal.TimeSeries{
newTimeSeries([]float64{2}, []int64{timestamp.UnixNano()}, map[string]string{
"__name__": "job:foo",
"job": "foo",
"source": "test",
}),
newTimeSeries([]float64{1}, []int64{timestamp.UnixNano()}, map[string]string{
"__name__": "job:foo",
"job": "bar",
"source": "test",
}),
})
}
func TestRecordingRule_ExecNegative(t *testing.T) {
func TestRecordingRuleLimit_Failure(t *testing.T) {
f := func(limit int, errStrExpected string) {
t.Helper()
testMetrics := []datasource.Metric{
metricWithValuesAndLabels(t, []float64{1}, "__name__", "foo", "job", "foo"),
metricWithValuesAndLabels(t, []float64{2, 3}, "__name__", "bar", "job", "bar"),
metricWithValuesAndLabels(t, []float64{4, 5, 6}, "__name__", "baz", "job", "baz"),
}
fq := &datasource.FakeQuerier{}
fq.Add(testMetrics...)
rule := &RecordingRule{Name: "job:foo",
state: &ruleState{entries: make([]StateEntry, 10)},
Labels: map[string]string{
"source": "test_limit",
},
metrics: &recordingRuleMetrics{
errors: utils.GetOrCreateCounter(`vmalert_recording_rules_errors_total{alertname="job:foo"}`),
},
}
rule.q = fq
_, err := rule.exec(context.TODO(), time.Now(), limit)
if err == nil {
t.Fatalf("expecting non-nil error")
}
errStr := err.Error()
if !strings.Contains(errStr, errStrExpected) {
t.Fatalf("missing %q in the error %q", errStrExpected, errStr)
}
}
f(1, "exec exceeded limit of 1 with 3 series")
f(2, "exec exceeded limit of 2 with 3 series")
}
func TestRecordingRuleLimit_Success(t *testing.T) {
f := func(limit int) {
t.Helper()
testMetrics := []datasource.Metric{
metricWithValuesAndLabels(t, []float64{1}, "__name__", "foo", "job", "foo"),
metricWithValuesAndLabels(t, []float64{2, 3}, "__name__", "bar", "job", "bar"),
metricWithValuesAndLabels(t, []float64{4, 5, 6}, "__name__", "baz", "job", "baz"),
}
fq := &datasource.FakeQuerier{}
fq.Add(testMetrics...)
rule := &RecordingRule{Name: "job:foo",
state: &ruleState{entries: make([]StateEntry, 10)},
Labels: map[string]string{
"source": "test_limit",
},
metrics: &recordingRuleMetrics{
errors: utils.GetOrCreateCounter(`vmalert_recording_rules_errors_total{alertname="job:foo"}`),
},
}
rule.q = fq
_, err := rule.exec(context.TODO(), time.Now(), limit)
if err != nil {
t.Fatalf("unexpected error: %s", err)
}
}
f(0)
f(-1)
}
func TestRecordingRuleExec_Negative(t *testing.T) {
rr := &RecordingRule{
Name: "job:foo",
Labels: map[string]string{
@@ -256,6 +263,6 @@ func TestRecordingRule_ExecNegative(t *testing.T) {
_, err = rr.exec(context.TODO(), time.Now(), 0)
if err != nil {
t.Fatal(err)
t.Fatalf("cannot execute recroding rule: %s", err)
}
}

View File

@@ -316,7 +316,7 @@ func templateFuncs() textTpl.FuncMap {
// humanize converts given number to a human readable format
// by adding metric prefixes https://en.wikipedia.org/wiki/Metric_prefix
"humanize": func(i interface{}) (string, error) {
"humanize": func(i any) (string, error) {
v, err := toFloat64(i)
if err != nil {
return "", err
@@ -347,7 +347,7 @@ func templateFuncs() textTpl.FuncMap {
},
// humanize1024 converts given number to a human readable format with 1024 as base
"humanize1024": func(i interface{}) (string, error) {
"humanize1024": func(i any) (string, error) {
v, err := toFloat64(i)
if err != nil {
return "", err
@@ -359,7 +359,7 @@ func templateFuncs() textTpl.FuncMap {
},
// humanizeDuration converts given seconds to a human-readable duration
"humanizeDuration": func(i interface{}) (string, error) {
"humanizeDuration": func(i any) (string, error) {
v, err := toFloat64(i)
if err != nil {
return "", err
@@ -405,7 +405,7 @@ func templateFuncs() textTpl.FuncMap {
},
// humanizePercentage converts given ratio value to a fraction of 100
"humanizePercentage": func(i interface{}) (string, error) {
"humanizePercentage": func(i any) (string, error) {
v, err := toFloat64(i)
if err != nil {
return "", err
@@ -414,7 +414,7 @@ func templateFuncs() textTpl.FuncMap {
},
// humanizeTimestamp converts given timestamp to a human readable time equivalent
"humanizeTimestamp": func(i interface{}) (string, error) {
"humanizeTimestamp": func(i any) (string, error) {
v, err := toFloat64(i)
if err != nil {
return "", err
@@ -427,7 +427,7 @@ func templateFuncs() textTpl.FuncMap {
},
// toTime converts given timestamp to a time.Time.
"toTime": func(i interface{}) (time.Time, error) {
"toTime": func(i any) (time.Time, error) {
v, err := toFloat64(i)
if err != nil {
return time.Time{}, err
@@ -524,8 +524,8 @@ func templateFuncs() textTpl.FuncMap {
// Converts a list of objects to a map with keys arg0, arg1 etc.
// This is intended to allow multiple arguments to be passed to templates.
"args": func(args ...interface{}) map[string]interface{} {
result := make(map[string]interface{})
"args": func(args ...any) map[string]any {
result := make(map[string]any)
for i, a := range args {
result[fmt.Sprintf("arg%d", i)] = a
}
@@ -565,7 +565,7 @@ func (t Time) Time() time.Time {
return time.Unix(int64(t)/second, (int64(t)%second)*nanosPerTick)
}
func toFloat64(v interface{}) (float64, error) {
func toFloat64(v any) (float64, error) {
switch i := v.(type) {
case float64:
return i, nil

View File

@@ -7,10 +7,11 @@ import (
textTpl "text/template"
)
func TestTemplateFuncs(t *testing.T) {
funcs := templateFuncs()
func TestTemplateFuncs_StringConversion(t *testing.T) {
f := func(funcName, s, resultExpected string) {
t.Helper()
funcs := templateFuncs()
v := funcs[funcName]
fLocal := v.(func(s string) string)
result := fLocal(s)
@@ -18,6 +19,7 @@ func TestTemplateFuncs(t *testing.T) {
t.Fatalf("unexpected result for %s(%q); got\n%s\nwant\n%s", funcName, s, result, resultExpected)
}
}
f("title", "foo bar", "Foo Bar")
f("toUpper", "foo", "FOO")
f("toLower", "FOO", "foo")
@@ -31,7 +33,10 @@ func TestTemplateFuncs(t *testing.T) {
f("stripPort", "foo:1234", "foo")
f("stripDomain", "foo.bar.baz", "foo")
f("stripDomain", "foo.bar:123", "foo:123")
}
func TestTemplateFuncs_Match(t *testing.T) {
funcs := templateFuncs()
// check "match" func
matchFunc := funcs["match"].(func(pattern, s string) (bool, error))
if _, err := matchFunc("invalid[regexp", "abc"); err == nil {
@@ -51,11 +56,15 @@ func TestTemplateFuncs(t *testing.T) {
if !ok {
t.Fatalf("unexpected mismatch")
}
}
formatting := func(funcName string, p interface{}, resultExpected string) {
func TestTemplateFuncs_Formatting(t *testing.T) {
f := func(funcName string, p any, resultExpected string) {
t.Helper()
funcs := templateFuncs()
v := funcs[funcName]
fLocal := v.(func(s interface{}) (string, error))
fLocal := v.(func(s any) (string, error))
result, err := fLocal(p)
if err != nil {
t.Fatalf("unexpected error for %s(%f): %s", funcName, p, err)
@@ -64,35 +73,36 @@ func TestTemplateFuncs(t *testing.T) {
t.Fatalf("unexpected result for %s(%f); got\n%s\nwant\n%s", funcName, p, result, resultExpected)
}
}
formatting("humanize1024", float64(0), "0")
formatting("humanize1024", math.Inf(0), "+Inf")
formatting("humanize1024", math.NaN(), "NaN")
formatting("humanize1024", float64(127087), "124.1ki")
formatting("humanize1024", float64(130137088), "124.1Mi")
formatting("humanize1024", float64(133260378112), "124.1Gi")
formatting("humanize1024", float64(136458627186688), "124.1Ti")
formatting("humanize1024", float64(139733634239168512), "124.1Pi")
formatting("humanize1024", float64(143087241460908556288), "124.1Ei")
formatting("humanize1024", float64(146521335255970361638912), "124.1Zi")
formatting("humanize1024", float64(150037847302113650318245888), "124.1Yi")
formatting("humanize1024", float64(153638755637364377925883789312), "1.271e+05Yi")
formatting("humanize", float64(127087), "127.1k")
formatting("humanize", float64(136458627186688), "136.5T")
f("humanize1024", float64(0), "0")
f("humanize1024", math.Inf(0), "+Inf")
f("humanize1024", math.NaN(), "NaN")
f("humanize1024", float64(127087), "124.1ki")
f("humanize1024", float64(130137088), "124.1Mi")
f("humanize1024", float64(133260378112), "124.1Gi")
f("humanize1024", float64(136458627186688), "124.1Ti")
f("humanize1024", float64(139733634239168512), "124.1Pi")
f("humanize1024", float64(143087241460908556288), "124.1Ei")
f("humanize1024", float64(146521335255970361638912), "124.1Zi")
f("humanize1024", float64(150037847302113650318245888), "124.1Yi")
f("humanize1024", float64(153638755637364377925883789312), "1.271e+05Yi")
formatting("humanizeDuration", 1, "1s")
formatting("humanizeDuration", 0.2, "200ms")
formatting("humanizeDuration", 42000, "11h 40m 0s")
formatting("humanizeDuration", 16790555, "194d 8h 2m 35s")
f("humanize", float64(127087), "127.1k")
f("humanize", float64(136458627186688), "136.5T")
formatting("humanizePercentage", 1, "100%")
formatting("humanizePercentage", 0.8, "80%")
formatting("humanizePercentage", 0.015, "1.5%")
f("humanizeDuration", 1, "1s")
f("humanizeDuration", 0.2, "200ms")
f("humanizeDuration", 42000, "11h 40m 0s")
f("humanizeDuration", 16790555, "194d 8h 2m 35s")
formatting("humanizeTimestamp", 1679055557, "2023-03-17 12:19:17 +0000 UTC")
f("humanizePercentage", 1, "100%")
f("humanizePercentage", 0.8, "80%")
f("humanizePercentage", 0.015, "1.5%")
f("humanizeTimestamp", 1679055557, "2023-03-17 12:19:17 +0000 UTC")
}
func mkTemplate(current, replacement interface{}) textTemplate {
func mkTemplate(current, replacement any) textTemplate {
tmpl := textTemplate{}
if current != nil {
switch val := current.(type) {
@@ -138,224 +148,201 @@ func equalTemplates(tmpls ...*textTpl.Template) bool {
return true
}
func TestTemplates_Load(t *testing.T) {
testCases := []struct {
name string
initialTemplate textTemplate
pathPatterns []string
overwrite bool
expectedTemplate textTemplate
expErr string
}{
{
"non existing path undefined template override",
mkTemplate(nil, nil),
[]string{
"templates/non-existing/good-*.tpl",
"templates/absent/good-*.tpl",
},
true,
mkTemplate(``, nil),
"",
},
{
"non existing path defined template override",
mkTemplate(`
{{- define "test.1" -}}
{{- printf "value" -}}
{{- end -}}
`, nil),
[]string{
"templates/non-existing/good-*.tpl",
"templates/absent/good-*.tpl",
},
true,
mkTemplate(``, nil),
"",
},
{
"existing path undefined template override",
mkTemplate(nil, nil),
[]string{
"templates/other/nested/good0-*.tpl",
"templates/test/good0-*.tpl",
},
false,
mkTemplate(`
{{- define "good0-test.tpl" -}}{{- end -}}
{{- define "test.0" -}}
{{ printf "Hello %s!" externalURL }}
{{- end -}}
{{- define "test.1" -}}
{{ printf "Hello %s!" externalURL }}
{{- end -}}
{{- define "test.2" -}}
{{ printf "Hello %s!" externalURL }}
{{- end -}}
{{- define "test.3" -}}
{{ printf "Hello %s!" externalURL }}
{{- end -}}
`, nil),
"",
},
{
"existing path defined template override",
mkTemplate(`
{{- define "test.1" -}}
{{ printf "Hello %s!" "world" }}
{{- end -}}
`, nil),
[]string{
"templates/other/nested/good0-*.tpl",
"templates/test/good0-*.tpl",
},
false,
mkTemplate(`
{{- define "good0-test.tpl" -}}{{- end -}}
{{- define "test.0" -}}
{{ printf "Hello %s!" externalURL }}
{{- end -}}
{{- define "test.1" -}}
{{ printf "Hello %s!" "world" }}
{{- end -}}
{{- define "test.2" -}}
{{ printf "Hello %s!" externalURL }}
{{- end -}}
{{- define "test.3" -}}
{{ printf "Hello %s!" externalURL }}
{{- end -}}
`, `
{{- define "good0-test.tpl" -}}{{- end -}}
{{- define "test.0" -}}
{{ printf "Hello %s!" externalURL }}
{{- end -}}
{{- define "test.1" -}}
{{ printf "Hello %s!" externalURL }}
{{- end -}}
{{- define "test.2" -}}
{{ printf "Hello %s!" externalURL }}
{{- end -}}
{{- define "test.3" -}}
{{ printf "Hello %s!" externalURL }}
{{- end -}}
`),
"",
},
{
"load template with syntax error",
mkTemplate(`
{{- define "test.1" -}}
{{ printf "Hello %s!" "world" }}
{{- end -}}
`, nil),
[]string{
"templates/other/nested/bad0-*.tpl",
"templates/test/good0-*.tpl",
},
false,
mkTemplate(`
{{- define "test.1" -}}
{{ printf "Hello %s!" "world" }}
{{- end -}}
`, nil),
"failed to parse template glob",
},
func TestTemplatesLoad_Failure(t *testing.T) {
f := func(pathPatterns []string, expectedErrStr string) {
t.Helper()
err := Load(pathPatterns, false)
if err == nil {
t.Fatalf("expecting non-nil error")
}
errStr := err.Error()
if !strings.Contains(errStr, expectedErrStr) {
t.Fatalf("the returned error %q doesn't contain %q", errStr, expectedErrStr)
}
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
masterTmpl = tc.initialTemplate
err := Load(tc.pathPatterns, tc.overwrite)
if tc.expErr == "" && err != nil {
t.Error("happened error that wasn't expected: %w", err)
}
if tc.expErr != "" && err == nil {
t.Error("%+w", err)
t.Error("expected error that didn't happened")
}
if err != nil && !strings.Contains(err.Error(), tc.expErr) {
t.Error("%+w", err)
t.Error("expected string doesn't exist in error message")
}
if !equalTemplates(masterTmpl.replacement, tc.expectedTemplate.replacement) {
t.Fatalf("replacement template is not as expected")
}
if !equalTemplates(masterTmpl.current, tc.expectedTemplate.current) {
t.Fatalf("current template is not as expected")
}
})
}
// load template with syntax error
f([]string{
"templates/other/nested/bad0-*.tpl",
"templates/test/good0-*.tpl",
}, "failed to parse template glob")
}
func TestTemplates_Reload(t *testing.T) {
testCases := []struct {
name string
initialTemplate textTemplate
expectedTemplate textTemplate
}{
{
"empty current and replacement templates",
mkTemplate(nil, nil),
mkTemplate(nil, nil),
},
{
"empty current template only",
mkTemplate(`
{{- define "test.1" -}}
{{- printf "value" -}}
{{- end -}}
`, nil),
mkTemplate(`
{{- define "test.1" -}}
{{- printf "value" -}}
{{- end -}}
`, nil),
},
{
"empty replacement template only",
mkTemplate(nil, `
{{- define "test.1" -}}
{{- printf "value" -}}
{{- end -}}
`),
mkTemplate(`
{{- define "test.1" -}}
{{- printf "value" -}}
{{- end -}}
`, nil),
},
{
"defined both templates",
mkTemplate(`
{{- define "test.0" -}}
{{- printf "value" -}}
{{- end -}}
{{- define "test.1" -}}
{{- printf "before" -}}
{{- end -}}
`, `
{{- define "test.1" -}}
{{- printf "after" -}}
{{- end -}}
`),
mkTemplate(`
{{- define "test.1" -}}
{{- printf "after" -}}
{{- end -}}
`, nil),
},
func TestTemplatesLoad_Success(t *testing.T) {
f := func(initialTmpl textTemplate, pathPatterns []string, overwrite bool, expectedTmpl textTemplate) {
t.Helper()
masterTmplOrig := masterTmpl
masterTmpl = initialTmpl
defer func() {
masterTmpl = masterTmplOrig
}()
if err := Load(pathPatterns, overwrite); err != nil {
t.Fatalf("cannot load templates: %s", err)
}
if !equalTemplates(masterTmpl.replacement, expectedTmpl.replacement) {
t.Fatalf("unexpected replacement template\ngot\n%+v\nwant\n%+v", masterTmpl.replacement, expectedTmpl.replacement)
}
if !equalTemplates(masterTmpl.current, expectedTmpl.current) {
t.Fatalf("unexpected current template\ngot\n%+v\nwant\n%+v", masterTmpl.current, expectedTmpl.current)
}
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
masterTmpl = tc.initialTemplate
Reload()
if !equalTemplates(masterTmpl.replacement, tc.expectedTemplate.replacement) {
t.Fatalf("replacement template is not as expected")
}
if !equalTemplates(masterTmpl.current, tc.expectedTemplate.current) {
t.Fatalf("current template is not as expected")
}
})
// non existing path undefined template override
initialTmpl := mkTemplate(nil, nil)
pathPatterns := []string{
"templates/non-existing/good-*.tpl",
"templates/absent/good-*.tpl",
}
overwrite := true
expectedTmpl := mkTemplate(``, nil)
f(initialTmpl, pathPatterns, overwrite, expectedTmpl)
// non existing path defined template override
initialTmpl = mkTemplate(`
{{- define "test.1" -}}
{{- printf "value" -}}
{{- end -}}
`, nil)
pathPatterns = []string{
"templates/non-existing/good-*.tpl",
"templates/absent/good-*.tpl",
}
overwrite = true
expectedTmpl = mkTemplate(``, nil)
f(initialTmpl, pathPatterns, overwrite, expectedTmpl)
// existing path undefined template override
initialTmpl = mkTemplate(nil, nil)
pathPatterns = []string{
"templates/other/nested/good0-*.tpl",
"templates/test/good0-*.tpl",
}
overwrite = false
expectedTmpl = mkTemplate(`
{{- define "good0-test.tpl" -}}{{- end -}}
{{- define "test.0" -}}
{{ printf "Hello %s!" externalURL }}
{{- end -}}
{{- define "test.1" -}}
{{ printf "Hello %s!" externalURL }}
{{- end -}}
{{- define "test.2" -}}
{{ printf "Hello %s!" externalURL }}
{{- end -}}
{{- define "test.3" -}}
{{ printf "Hello %s!" externalURL }}
{{- end -}}
`, nil)
f(initialTmpl, pathPatterns, overwrite, expectedTmpl)
// existing path defined template override
initialTmpl = mkTemplate(`
{{- define "test.1" -}}
{{ printf "Hello %s!" "world" }}
{{- end -}}
`, nil)
pathPatterns = []string{
"templates/other/nested/good0-*.tpl",
"templates/test/good0-*.tpl",
}
overwrite = false
expectedTmpl = mkTemplate(`
{{- define "good0-test.tpl" -}}{{- end -}}
{{- define "test.0" -}}
{{ printf "Hello %s!" externalURL }}
{{- end -}}
{{- define "test.1" -}}
{{ printf "Hello %s!" "world" }}
{{- end -}}
{{- define "test.2" -}}
{{ printf "Hello %s!" externalURL }}
{{- end -}}
{{- define "test.3" -}}
{{ printf "Hello %s!" externalURL }}
{{- end -}}
`, `
{{- define "good0-test.tpl" -}}{{- end -}}
{{- define "test.0" -}}
{{ printf "Hello %s!" externalURL }}
{{- end -}}
{{- define "test.1" -}}
{{ printf "Hello %s!" externalURL }}
{{- end -}}
{{- define "test.2" -}}
{{ printf "Hello %s!" externalURL }}
{{- end -}}
{{- define "test.3" -}}
{{ printf "Hello %s!" externalURL }}
{{- end -}}
`)
f(initialTmpl, pathPatterns, overwrite, expectedTmpl)
}
func TestTemplatesReload(t *testing.T) {
f := func(initialTmpl, expectedTmpl textTemplate) {
t.Helper()
masterTmplOrig := masterTmpl
masterTmpl = initialTmpl
defer func() {
masterTmpl = masterTmplOrig
}()
Reload()
if !equalTemplates(masterTmpl.replacement, expectedTmpl.replacement) {
t.Fatalf("unexpected replacement template\ngot\n%+v\nwant\n%+v", masterTmpl.replacement, expectedTmpl.replacement)
}
if !equalTemplates(masterTmpl.current, expectedTmpl.current) {
t.Fatalf("unexpected current template\ngot\n%+v\nwant\n%+v", masterTmpl.current, expectedTmpl.current)
}
}
// empty current and replacement templates
f(mkTemplate(nil, nil), mkTemplate(nil, nil))
// empty current template only
f(mkTemplate(`
{{- define "test.1" -}}
{{- printf "value" -}}
{{- end -}}
`, nil), mkTemplate(`
{{- define "test.1" -}}
{{- printf "value" -}}
{{- end -}}
`, nil))
// empty replacement template only
f(mkTemplate(nil, `
{{- define "test.1" -}}
{{- printf "value" -}}
{{- end -}}
`), mkTemplate(`
{{- define "test.1" -}}
{{- printf "value" -}}
{{- end -}}
`, nil))
// defined both templates
f(mkTemplate(`
{{- define "test.0" -}}
{{- printf "value" -}}
{{- end -}}
{{- define "test.1" -}}
{{- printf "before" -}}
{{- end -}}
`, `
{{- define "test.1" -}}
{{- printf "after" -}}
{{- end -}}
`), mkTemplate(`
{{- define "test.1" -}}
{{- printf "after" -}}
{{- end -}}
`, nil))
}

View File

@@ -7,35 +7,31 @@ import (
)
func TestErrGroup(t *testing.T) {
testCases := []struct {
errs []error
exp string
}{
{nil, ""},
{[]error{errors.New("timeout")}, "errors(1): timeout"},
{
[]error{errors.New("timeout"), errors.New("deadline")},
"errors(2): timeout\ndeadline",
},
}
for _, tc := range testCases {
eg := new(ErrGroup)
for _, err := range tc.errs {
f := func(errs []error, resultExpected string) {
t.Helper()
eg := &ErrGroup{}
for _, err := range errs {
eg.Add(err)
}
if len(tc.errs) == 0 {
if len(errs) == 0 {
if eg.Err() != nil {
t.Fatalf("expected to get nil error")
}
continue
return
}
if eg.Err() == nil {
t.Fatalf("expected to get non-nil error")
}
if eg.Error() != tc.exp {
t.Fatalf("expected to have: \n%q\ngot:\n%q", tc.exp, eg.Error())
result := eg.Error()
if result != resultExpected {
t.Fatalf("unexpected result\ngot\n%v\nwant\n%v", result, resultExpected)
}
}
f(nil, "")
f([]error{errors.New("timeout")}, "errors(1): timeout")
f([]error{errors.New("timeout"), errors.New("deadline")}, "errors(2): timeout\ndeadline")
}
// TestErrGroupConcurrent supposed to test concurrent

View File

@@ -19,7 +19,7 @@ import (
"github.com/VictoriaMetrics/VictoriaMetrics/lib/procutil"
)
var reloadAuthKey = flagutil.NewPassword("reloadAuthKey", "Auth key for /-/reload http endpoint. It must be passed via authKey query arg. It overrides httpAuth.* settings.")
var reloadAuthKey = flagutil.NewPassword("reloadAuthKey", "Auth key for /-/reload http endpoint. It must be passed via authKey query arg. It overrides -httpAuth.*")
var (
apiLinks = [][2]string{
@@ -35,7 +35,7 @@ var (
{"-/reload", "reload configuration"},
}
navItems = []tpl.NavItem{
{Name: "vmalert", Url: "."},
{Name: "vmalert", Url: "../vmalert"},
{Name: "Groups", Url: "groups"},
{Name: "Alerts", Url: "alerts"},
{Name: "Notifiers", Url: "notifiers"},
@@ -167,7 +167,7 @@ func (rh *requestHandler) handler(w http.ResponseWriter, r *http.Request) bool {
w.Write(data)
return true
case "/-/reload":
if !httpserver.CheckAuthFlag(w, r, reloadAuthKey.Get(), "reloadAuthKey") {
if !httpserver.CheckAuthFlag(w, r, reloadAuthKey) {
return true
}
logger.Infof("api config reload was called, sending sighup")

View File

@@ -36,23 +36,23 @@ func TestHandler(t *testing.T) {
}}
rh := &requestHandler{m: m}
getResp := func(t *testing.T, url string, to interface{}, code int) {
getResp := func(t *testing.T, url string, to any, code int) {
t.Helper()
resp, err := http.Get(url)
if err != nil {
t.Fatalf("unexpected err %s", err)
}
if code != resp.StatusCode {
t.Errorf("unexpected status code %d want %d", resp.StatusCode, code)
t.Fatalf("unexpected status code %d want %d", resp.StatusCode, code)
}
defer func() {
if err := resp.Body.Close(); err != nil {
t.Errorf("err closing body %s", err)
t.Fatalf("err closing body %s", err)
}
}()
if to != nil {
if err = json.NewDecoder(resp.Body).Decode(to); err != nil {
t.Errorf("unexpected err %s", err)
t.Fatalf("unexpected err %s", err)
}
}
}
@@ -92,13 +92,13 @@ func TestHandler(t *testing.T) {
lr := listAlertsResponse{}
getResp(t, ts.URL+"/api/v1/alerts", &lr, 200)
if length := len(lr.Data.Alerts); length != 1 {
t.Errorf("expected 1 alert got %d", length)
t.Fatalf("expected 1 alert got %d", length)
}
lr = listAlertsResponse{}
getResp(t, ts.URL+"/vmalert/api/v1/alerts", &lr, 200)
if length := len(lr.Data.Alerts); length != 1 {
t.Errorf("expected 1 alert got %d", length)
t.Fatalf("expected 1 alert got %d", length)
}
})
t.Run("/api/v1/alert?alertID&groupID", func(t *testing.T) {
@@ -106,13 +106,13 @@ func TestHandler(t *testing.T) {
alert := &apiAlert{}
getResp(t, ts.URL+"/"+expAlert.APILink(), alert, 200)
if !reflect.DeepEqual(alert, expAlert) {
t.Errorf("expected %v is equal to %v", alert, expAlert)
t.Fatalf("expected %v is equal to %v", alert, expAlert)
}
alert = &apiAlert{}
getResp(t, ts.URL+"/vmalert/"+expAlert.APILink(), alert, 200)
if !reflect.DeepEqual(alert, expAlert) {
t.Errorf("expected %v is equal to %v", alert, expAlert)
t.Fatalf("expected %v is equal to %v", alert, expAlert)
}
})
@@ -135,13 +135,13 @@ func TestHandler(t *testing.T) {
lr := listGroupsResponse{}
getResp(t, ts.URL+"/api/v1/rules", &lr, 200)
if length := len(lr.Data.Groups); length != 1 {
t.Errorf("expected 1 group got %d", length)
t.Fatalf("expected 1 group got %d", length)
}
lr = listGroupsResponse{}
getResp(t, ts.URL+"/vmalert/api/v1/rules", &lr, 200)
if length := len(lr.Data.Groups); length != 1 {
t.Errorf("expected 1 group got %d", length)
t.Fatalf("expected 1 group got %d", length)
}
})
t.Run("/api/v1/rule?ruleID&groupID", func(t *testing.T) {
@@ -150,14 +150,14 @@ func TestHandler(t *testing.T) {
getResp(t, ts.URL+"/"+expRule.APILink(), &gotRule, 200)
if expRule.ID != gotRule.ID {
t.Errorf("expected to get Rule %q; got %q instead", expRule.ID, gotRule.ID)
t.Fatalf("expected to get Rule %q; got %q instead", expRule.ID, gotRule.ID)
}
gotRule = apiRule{}
getResp(t, ts.URL+"/vmalert/"+expRule.APILink(), &gotRule, 200)
if expRule.ID != gotRule.ID {
t.Errorf("expected to get Rule %q; got %q instead", expRule.ID, gotRule.ID)
t.Fatalf("expected to get Rule %q; got %q instead", expRule.ID, gotRule.ID)
}
gotRuleWithUpdates := apiRuleWithUpdates{}
@@ -173,7 +173,7 @@ func TestHandler(t *testing.T) {
lr := listGroupsResponse{}
getResp(t, ts.URL+url, &lr, 200)
if length := len(lr.Data.Groups); length != expGroups {
t.Errorf("expected %d groups got %d", expGroups, length)
t.Fatalf("expected %d groups got %d", expGroups, length)
}
if len(lr.Data.Groups) < 1 {
return
@@ -183,7 +183,7 @@ func TestHandler(t *testing.T) {
rulesN += len(gr.Rules)
}
if rulesN != expRules {
t.Errorf("expected %d rules got %d", expRules, rulesN)
t.Fatalf("expected %d rules got %d", expRules, rulesN)
}
}
@@ -241,23 +241,23 @@ func TestEmptyResponse(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { rhWithNoGroups.handler(w, r) }))
defer ts.Close()
getResp := func(t *testing.T, url string, to interface{}, code int) {
getResp := func(t *testing.T, url string, to any, code int) {
t.Helper()
resp, err := http.Get(url)
if err != nil {
t.Fatalf("unexpected err %s", err)
}
if code != resp.StatusCode {
t.Errorf("unexpected status code %d want %d", resp.StatusCode, code)
t.Fatalf("unexpected status code %d want %d", resp.StatusCode, code)
}
defer func() {
if err := resp.Body.Close(); err != nil {
t.Errorf("err closing body %s", err)
t.Fatalf("err closing body %s", err)
}
}()
if to != nil {
if err = json.NewDecoder(resp.Body).Decode(to); err != nil {
t.Errorf("unexpected err %s", err)
t.Fatalf("unexpected err %s", err)
}
}
}
@@ -266,13 +266,13 @@ func TestEmptyResponse(t *testing.T) {
lr := listAlertsResponse{}
getResp(t, ts.URL+"/api/v1/alerts", &lr, 200)
if lr.Data.Alerts == nil {
t.Errorf("expected /api/v1/alerts response to have non-nil data")
t.Fatalf("expected /api/v1/alerts response to have non-nil data")
}
lr = listAlertsResponse{}
getResp(t, ts.URL+"/vmalert/api/v1/alerts", &lr, 200)
if lr.Data.Alerts == nil {
t.Errorf("expected /api/v1/alerts response to have non-nil data")
t.Fatalf("expected /api/v1/alerts response to have non-nil data")
}
})
@@ -280,13 +280,13 @@ func TestEmptyResponse(t *testing.T) {
lr := listGroupsResponse{}
getResp(t, ts.URL+"/api/v1/rules", &lr, 200)
if lr.Data.Groups == nil {
t.Errorf("expected /api/v1/rules response to have non-nil data")
t.Fatalf("expected /api/v1/rules response to have non-nil data")
}
lr = listGroupsResponse{}
getResp(t, ts.URL+"/vmalert/api/v1/rules", &lr, 200)
if lr.Data.Groups == nil {
t.Errorf("expected /api/v1/rules response to have non-nil data")
t.Fatalf("expected /api/v1/rules response to have non-nil data")
}
})

View File

@@ -183,7 +183,7 @@ func (ar apiRule) WebLink() string {
paramGroupID, ar.GroupID, paramRuleID, ar.ID)
}
func ruleToAPI(r interface{}) apiRule {
func ruleToAPI(r any) apiRule {
if ar, ok := r.(*rule.AlertingRule); ok {
return alertingToAPI(ar)
}

View File

@@ -13,11 +13,11 @@ func TestUrlValuesToStrings(t *testing.T) {
res := urlValuesToStrings(mapQueryParams)
if len(res) != len(expectedRes) {
t.Errorf("Expected length %d, but got %d", len(expectedRes), len(res))
t.Fatalf("Expected length %d, but got %d", len(expectedRes), len(res))
}
for ind, val := range expectedRes {
if val != res[ind] {
t.Errorf("Expected %v; but got %v", val, res[ind])
t.Fatalf("Expected %v; but got %v", val, res[ind])
}
}
}

View File

@@ -129,7 +129,7 @@ type Header struct {
}
// UnmarshalYAML unmarshals h from f.
func (h *Header) UnmarshalYAML(f func(interface{}) error) error {
func (h *Header) UnmarshalYAML(f func(any) error) error {
var s string
if err := f(&s); err != nil {
return err
@@ -146,7 +146,7 @@ func (h *Header) UnmarshalYAML(f func(interface{}) error) error {
}
// MarshalYAML marshals h to yaml.
func (h *Header) MarshalYAML() (interface{}, error) {
func (h *Header) MarshalYAML() (any, error) {
return h.sOriginal, nil
}
@@ -201,7 +201,7 @@ type QueryArg struct {
}
// UnmarshalYAML unmarshals qa from yaml.
func (qa *QueryArg) UnmarshalYAML(f func(interface{}) error) error {
func (qa *QueryArg) UnmarshalYAML(f func(any) error) error {
var s string
if err := f(&s); err != nil {
return err
@@ -230,7 +230,7 @@ func (qa *QueryArg) UnmarshalYAML(f func(interface{}) error) error {
}
// MarshalYAML marshals qa to yaml.
func (qa *QueryArg) MarshalYAML() (interface{}, error) {
func (qa *QueryArg) MarshalYAML() (any, error) {
return qa.sOriginal, nil
}
@@ -263,7 +263,7 @@ type URLPrefix struct {
nextDiscoveryDeadline atomic.Uint64
// vOriginal contains the original yaml value for URLPrefix.
vOriginal interface{}
vOriginal any
}
func (up *URLPrefix) setLoadBalancingPolicy(loadBalancingPolicy string) error {
@@ -354,11 +354,12 @@ func (up *URLPrefix) discoverBackendAddrsIfNeeded() {
// ips for the given host have been already discovered
continue
}
var resolvedAddrs []string
if strings.HasPrefix(host, "srv+") {
// The host has the format 'srv+realhost'. Strip 'srv+' prefix before performing the lookup.
host = strings.TrimPrefix(host, "srv+")
_, addrs, err := netutil.Resolver.LookupSRV(ctx, "", "", host)
srvHost := strings.TrimPrefix(host, "srv+")
_, addrs, err := netutil.Resolver.LookupSRV(ctx, "", "", srvHost)
if err != nil {
logger.Warnf("cannot discover backend SRV records for %s: %s; use it literally", bu, err)
resolvedAddrs = []string{host}
@@ -390,7 +391,6 @@ func (up *URLPrefix) discoverBackendAddrsIfNeeded() {
var busNew []*backendURL
for _, bu := range up.busOriginal {
host := bu.Hostname()
host = strings.TrimPrefix(host, "srv+")
port := bu.Port()
for _, addr := range hostToAddrs[host] {
buCopy := *bu
@@ -497,8 +497,8 @@ func getLeastLoadedBackendURL(bus []*backendURL, atomicCounter *atomic.Uint32) *
}
// UnmarshalYAML unmarshals up from yaml.
func (up *URLPrefix) UnmarshalYAML(f func(interface{}) error) error {
var v interface{}
func (up *URLPrefix) UnmarshalYAML(f func(any) error) error {
var v any
if err := f(&v); err != nil {
return err
}
@@ -508,7 +508,7 @@ func (up *URLPrefix) UnmarshalYAML(f func(interface{}) error) error {
switch x := v.(type) {
case string:
urls = []string{x}
case []interface{}:
case []any:
if len(x) == 0 {
return fmt.Errorf("`url_prefix` must contain at least a single url")
}
@@ -538,7 +538,7 @@ func (up *URLPrefix) UnmarshalYAML(f func(interface{}) error) error {
}
// MarshalYAML marshals up to yaml.
func (up *URLPrefix) MarshalYAML() (interface{}, error) {
func (up *URLPrefix) MarshalYAML() (any, error) {
return up.vOriginal, nil
}
@@ -562,7 +562,7 @@ func (r *Regex) match(s string) bool {
}
// UnmarshalYAML implements yaml.Unmarshaler
func (r *Regex) UnmarshalYAML(f func(interface{}) error) error {
func (r *Regex) UnmarshalYAML(f func(any) error) error {
var s string
if err := f(&s); err != nil {
return err
@@ -579,7 +579,7 @@ func (r *Regex) UnmarshalYAML(f func(interface{}) error) error {
}
// MarshalYAML implements yaml.Marshaler.
func (r *Regex) MarshalYAML() (interface{}, error) {
func (r *Regex) MarshalYAML() (any, error) {
return r.sOriginal, nil
}
@@ -696,22 +696,15 @@ func loadAuthConfig() (bool, error) {
}
logger.Infof("loaded information about %d users from -auth.config=%q", len(m), *authConfigPath)
prevAc := authConfig.Load()
if prevAc != nil {
metrics.UnregisterSet(prevAc.ms)
acPrev := authConfig.Load()
if acPrev != nil {
metrics.UnregisterSet(acPrev.ms, true)
}
metrics.RegisterSet(ac.ms)
authConfig.Store(ac)
authConfigData.Store(&data)
authUsers.Store(&m)
if prevAc != nil {
// explicilty unregister metrics, since all summary type metrics
// are registered at global state of metrics package
// and must be removed from it to release memory.
// Metrics must be unregistered only after atomic.Value.Store calls above
// Otherwise it may lead to metric gaps, since UnregisterAllMetrics is slow operation
prevAc.ms.UnregisterAllMetrics()
}
return true, nil
}
@@ -1024,8 +1017,6 @@ func (up *URLPrefix) sanitizeAndInitialize() error {
}
}
up.bus.Store(&bus)
up.nextDiscoveryDeadline.Store(0)
up.n.Store(0)
return nil
}

View File

@@ -37,15 +37,15 @@ var (
"With enabled proxy protocol http server cannot serve regular /metrics endpoint. Use -pushmetrics.url for metrics pushing")
maxIdleConnsPerBackend = flag.Int("maxIdleConnsPerBackend", 100, "The maximum number of idle connections vmauth can open per each backend host. "+
"See also -maxConcurrentRequests")
idleConnTimeout = flag.Duration("idleConnTimeout", 50*time.Second, `Defines a duration for idle (keep-alive connections) to exist.
Consider setting this value less than "-http.idleConnTimeout". It must prevent possible "write: broken pipe" and "read: connection reset by peer" errors.`)
idleConnTimeout = flag.Duration("idleConnTimeout", 50*time.Second, "The timeout for HTTP keep-alive connections to backend services. "+
"It is recommended setting this value to values smaller than -http.idleConnTimeout set at backend services")
responseTimeout = flag.Duration("responseTimeout", 5*time.Minute, "The timeout for receiving a response from backend")
maxConcurrentRequests = flag.Int("maxConcurrentRequests", 1000, "The maximum number of concurrent requests vmauth can process. Other requests are rejected with "+
"'429 Too Many Requests' http status code. See also -maxConcurrentPerUserRequests and -maxIdleConnsPerBackend command-line options")
maxConcurrentPerUserRequests = flag.Int("maxConcurrentPerUserRequests", 300, "The maximum number of concurrent requests vmauth can process per each configured user. "+
"Other requests are rejected with '429 Too Many Requests' http status code. See also -maxConcurrentRequests command-line option and max_concurrent_requests option "+
"in per-user config")
reloadAuthKey = flagutil.NewPassword("reloadAuthKey", "Auth key for /-/reload http endpoint. It must be passed via authKey query arg. It overrides httpAuth.* settings.")
reloadAuthKey = flagutil.NewPassword("reloadAuthKey", "Auth key for /-/reload http endpoint. It must be passed via authKey query arg. It overrides -httpAuth.*")
logInvalidAuthTokens = flag.Bool("logInvalidAuthTokens", false, "Whether to log requests with invalid auth tokens. "+
`Such requests are always counted at vmauth_http_request_errors_total{reason="invalid_auth_token"} metric, which is exposed at /metrics page`)
failTimeout = flag.Duration("failTimeout", 3*time.Second, "Sets a delay period for load balancing to skip a malfunctioning backend")
@@ -99,7 +99,7 @@ func main() {
func requestHandler(w http.ResponseWriter, r *http.Request) bool {
switch r.URL.Path {
case "/-/reload":
if !httpserver.CheckAuthFlag(w, r, reloadAuthKey.Get(), "reloadAuthKey") {
if !httpserver.CheckAuthFlag(w, r, reloadAuthKey) {
return true
}
configReloadRequests.Inc()
@@ -272,8 +272,7 @@ again:
ui.backendErrors.Inc()
return true
}
// one time retry trivial network errors, such as proxy idle timeout misconfiguration
// or socket close by OS
// one time retry trivial network errors, such as proxy idle timeout misconfiguration or socket close by OS
if (netutil.IsTrivialNetworkError(err) || errors.Is(err, io.EOF)) && trivialRetries < 1 {
trivialRetries++
goto again
@@ -450,7 +449,7 @@ func newRoundTripper(caFileOpt, certFileOpt, keyFileOpt, serverNameOpt string, i
if tr.MaxIdleConns != 0 && tr.MaxIdleConns < tr.MaxIdleConnsPerHost {
tr.MaxIdleConns = tr.MaxIdleConnsPerHost
}
tr.DialContext = netutil.DialMaybeSRV
tr.DialContext = netutil.NewStatDialFunc("vmauth_backend")
rt := cfg.NewRoundTripper(tr)
return rt, nil

View File

@@ -86,6 +86,7 @@ func TestCreateTargetURLSuccess(t *testing.T) {
f := func(ui *UserInfo, requestURI, expectedTarget, expectedRequestHeaders, expectedResponseHeaders string,
expectedRetryStatusCodes []int, expectedLoadBalancingPolicy string, expectedDropSrcPathPrefixParts int) {
t.Helper()
if err := ui.initURLs(); err != nil {
t.Fatalf("cannot initialize urls inside UserInfo: %s", err)
}
@@ -122,6 +123,7 @@ func TestCreateTargetURLSuccess(t *testing.T) {
t.Fatalf("unexpected dropSrcPathPrefixParts; got %d; want %d", up.dropSrcPathPrefixParts, expectedDropSrcPathPrefixParts)
}
}
// Simple routing with `url_prefix`
f(&UserInfo{
URLPrefix: mustParseURL("http://foo.bar"),
@@ -260,7 +262,32 @@ func TestCreateTargetURLSuccess(t *testing.T) {
f(ui, `/api/v1/query?query=up{env="prod"}`, `http://vmselect/1/prometheus/api/v1/query?query=up%7Benv%3D%22prod%22%7D`, "", "", nil, "least_loaded", 0)
f(ui, `/api/v1/query?query=up{foo="bar",env="dev",pod!=""}`, `http://vmselect/0/prometheus/api/v1/query?query=up%7Bfoo%3D%22bar%22%2Cenv%3D%22dev%22%2Cpod%21%3D%22%22%7D`, "", "", nil, "least_loaded", 0)
f(ui, `/api/v1/query?query=up{foo="bar"}`, `http://default-server/api/v1/query?query=up%7Bfoo%3D%22bar%22%7D`, "", "", nil, "least_loaded", 0)
}
func TestUserInfoGetBackendURL_SRV(t *testing.T) {
f := func(ui *UserInfo, requestURI, expectedTarget string) {
t.Helper()
u, err := url.Parse(requestURI)
if err != nil {
t.Fatalf("cannot parse %q: %s", requestURI, err)
}
u = normalizeURL(u)
up, _ := ui.getURLPrefixAndHeaders(u, nil)
if up == nil {
t.Fatalf("cannot match available backend: %s", err)
}
bu := up.getBackendURL()
target := mergeURLs(bu.url, u, up.dropSrcPathPrefixParts)
bu.put()
gotTarget := target.String()
if gotTarget != expectedTarget {
t.Fatalf("unexpected target\ngot:\n%q\nwant\n%q", gotTarget, expectedTarget)
}
}
// Discover backendURL with SRV hostnames
customResolver := &fakeResolver{
Resolver: &net.Resolver{},
lookupSRVResults: map[string][]*net.SRV{
@@ -283,11 +310,14 @@ func TestCreateTargetURLSuccess(t *testing.T) {
},
},
}
origResolver := netutil.Resolver
netutil.Resolver = customResolver
defer func() {
netutil.Resolver = origResolver
}()
// Discover backendURL
allowed := true
ui = &UserInfo{
ui := &UserInfo{
URLMaps: []URLMap{
{
SrcPaths: getRegexs([]string{"/select/.+"}),
@@ -301,12 +331,15 @@ func TestCreateTargetURLSuccess(t *testing.T) {
DiscoverBackendIPs: &allowed,
URLPrefix: mustParseURL("http://non-exist-dns-addr"),
}
f(ui, `/select/0/prometheus/api/v1/query?query=up`, "http://10.6.142.51:8481/select/0/prometheus/api/v1/query?query=up", "", "", nil, "least_loaded", 0)
// url_prefix counter will be reset, still go to 10.6.142.51
f(ui, `/select/0/prometheus/api/v1/query?query=up`, "http://10.6.142.51:8481/select/0/prometheus/api/v1/query?query=up", "", "", nil, "least_loaded", 0)
f(ui, `/insert/0/prometheus/api/v1/write`, "http://10.6.142.52:8480/insert/0/prometheus/api/v1/write", "", "", nil, "least_loaded", 0)
if err := ui.initURLs(); err != nil {
t.Fatalf("cannot initialize urls inside UserInfo: %s", err)
}
f(ui, `/select/0/prometheus/api/v1/query?query=up`, "http://10.6.142.51:8481/select/0/prometheus/api/v1/query?query=up")
f(ui, `/select/0/prometheus/api/v1/query?query=up`, "http://10.6.142.50:8481/select/0/prometheus/api/v1/query?query=up")
f(ui, `/insert/0/prometheus/api/v1/write`, "http://10.6.142.52:8480/insert/0/prometheus/api/v1/write")
// unsuccessful dns resolve
f(ui, `/test`, "http://non-exist-dns-addr/test", "", "", nil, "least_loaded", 0)
f(ui, `/test`, "http://non-exist-dns-addr/test")
}
func TestCreateTargetURLFailure(t *testing.T) {

View File

@@ -8,15 +8,18 @@ import (
func TestHasFilepathPrefix(t *testing.T) {
f := func(dst, storageDataPath string, resultExpected bool) {
t.Helper()
result := hasFilepathPrefix(dst, storageDataPath)
if result != resultExpected {
t.Errorf("unexpected hasFilepathPrefix(%q, %q); got: %v; want: %v", dst, storageDataPath, result, resultExpected)
t.Fatalf("unexpected hasFilepathPrefix(%q, %q); got: %v; want: %v", dst, storageDataPath, result, resultExpected)
}
}
pwd, err := filepath.Abs("")
if err != nil {
t.Fatalf("cannot determine working directory: %s", err)
}
f("s3://foo/bar", "foo", false)
f("fs://"+pwd+"/foo", "foo", true)
f("fs://"+pwd+"/foo", "foo/bar", false)

View File

@@ -7,103 +7,106 @@ import (
"time"
)
func TestRetry_Do(t *testing.T) {
counter := 0
tests := []struct {
name string
backoffRetries int
backoffFactor float64
backoffMinDuration time.Duration
retryableFunc retryableFunc
cancelTimeout time.Duration
want uint64
wantErr bool
}{
{
name: "return bad request",
retryableFunc: func() error {
return ErrBadRequest
},
want: 0,
wantErr: true,
},
{
name: "empty retries values",
retryableFunc: func() error {
time.Sleep(time.Millisecond * 100)
return nil
},
want: 0,
wantErr: true,
},
{
name: "only one retry test",
backoffRetries: 5,
backoffFactor: 1.7,
backoffMinDuration: time.Millisecond * 10,
retryableFunc: func() error {
t := time.NewTicker(time.Millisecond * 5)
defer t.Stop()
for range t.C {
counter++
if counter%2 == 0 {
return fmt.Errorf("got some error")
}
if counter%3 == 0 {
return nil
}
}
return nil
},
want: 1,
wantErr: false,
},
{
name: "all retries failed test",
backoffRetries: 5,
backoffFactor: 0.1,
backoffMinDuration: time.Millisecond * 10,
retryableFunc: func() error {
t := time.NewTicker(time.Millisecond * 5)
defer t.Stop()
for range t.C {
return fmt.Errorf("got some error")
}
return nil
},
want: 5,
wantErr: true,
},
{
name: "cancel context",
backoffRetries: 5,
backoffFactor: 1.7,
backoffMinDuration: time.Millisecond * 10,
retryableFunc: func() error {
return fmt.Errorf("got some error")
},
cancelTimeout: time.Millisecond * 40,
want: 3,
wantErr: true,
},
func TestBackoffRetry_Failure(t *testing.T) {
f := func(backoffFactor float64, backoffRetries int, cancelTimeout time.Duration, retryFunc func() error, resultExpected int) {
t.Helper()
r := &Backoff{
retries: backoffRetries,
factor: backoffFactor,
minDuration: time.Millisecond * 10,
}
ctx := context.Background()
if cancelTimeout != 0 {
newCtx, cancelFn := context.WithTimeout(context.Background(), cancelTimeout)
ctx = newCtx
defer cancelFn()
}
result, err := r.Retry(ctx, retryFunc)
if err == nil {
t.Fatalf("expecting non-nil error")
}
if result != uint64(resultExpected) {
t.Fatalf("unexpected result: got %d; want %d", result, resultExpected)
}
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
r := &Backoff{retries: tt.backoffRetries, factor: tt.backoffFactor, minDuration: tt.backoffMinDuration}
ctx := context.Background()
if tt.cancelTimeout != 0 {
newCtx, cancelFn := context.WithTimeout(context.Background(), tt.cancelTimeout)
ctx = newCtx
defer cancelFn()
}
got, err := r.Retry(ctx, tt.retryableFunc)
if (err != nil) != tt.wantErr {
t.Errorf("Retry() error = %v, wantErr %v", err, tt.wantErr)
return
}
if got != tt.want {
t.Errorf("Retry() got = %v, want %v", got, tt.want)
}
})
// return bad request
retryFunc := func() error {
return ErrBadRequest
}
f(0, 0, 0, retryFunc, 0)
// empty retries values
retryFunc = func() error {
time.Sleep(time.Millisecond * 100)
return nil
}
f(0, 0, 0, retryFunc, 0)
// all retries failed test
backoffFactor := 0.1
backoffRetries := 5
cancelTimeout := time.Second * 0
retryFunc = func() error {
t := time.NewTicker(time.Millisecond * 5)
defer t.Stop()
for range t.C {
return fmt.Errorf("got some error")
}
return nil
}
resultExpected := 5
f(backoffFactor, backoffRetries, cancelTimeout, retryFunc, resultExpected)
// cancel context
backoffFactor = 1.7
backoffRetries = 5
cancelTimeout = time.Millisecond * 40
retryFunc = func() error {
return fmt.Errorf("got some error")
}
resultExpected = 3
f(backoffFactor, backoffRetries, cancelTimeout, retryFunc, resultExpected)
}
func TestBackoffRetry_Success(t *testing.T) {
f := func(retryFunc func() error, resultExpected int) {
t.Helper()
r := &Backoff{
retries: 5,
factor: 1.7,
minDuration: time.Millisecond * 10,
}
ctx := context.Background()
result, err := r.Retry(ctx, retryFunc)
if err != nil {
t.Fatalf("Retry() error: %s", err)
}
if result != uint64(resultExpected) {
t.Fatalf("unexpected result: got %d; want %d", result, resultExpected)
}
}
// only one retry test
counter := 0
retryFunc := func() error {
t := time.NewTicker(time.Millisecond * 5)
defer t.Stop()
for range t.C {
counter++
if counter%2 == 0 {
return fmt.Errorf("got some error")
}
if counter%3 == 0 {
return nil
}
}
return nil
}
resultExpected := 1
f(retryFunc, resultExpected)
}

View File

@@ -3,125 +3,97 @@ package influx
import "testing"
func TestFetchQuery(t *testing.T) {
testCases := []struct {
s Series
timeFilter string
expected string
}{
{
s: Series{
Measurement: "cpu",
Field: "value",
LabelPairs: []LabelPair{
{
Name: "foo",
Value: "bar",
},
},
},
expected: `select "value" from "cpu" where "foo"::tag='bar'`,
},
{
s: Series{
Measurement: "cpu",
Field: "value",
LabelPairs: []LabelPair{
{
Name: "foo",
Value: "bar",
},
{
Name: "baz",
Value: "qux",
},
},
},
expected: `select "value" from "cpu" where "foo"::tag='bar' and "baz"::tag='qux'`,
},
{
s: Series{
Measurement: "cpu",
Field: "value",
LabelPairs: []LabelPair{
{
Name: "foo",
Value: "b'ar",
},
},
},
timeFilter: "time >= now()",
expected: `select "value" from "cpu" where "foo"::tag='b\'ar' and time >= now()`,
},
{
s: Series{
Measurement: "cpu",
Field: "value",
LabelPairs: []LabelPair{
{
Name: "name",
Value: `dev-mapper-centos\x2dswap.swap`,
},
{
Name: "state",
Value: "dev-mapp'er-c'en'tos",
},
},
},
timeFilter: "time >= now()",
expected: `select "value" from "cpu" where "name"::tag='dev-mapper-centos\\x2dswap.swap' and "state"::tag='dev-mapp\'er-c\'en\'tos' and time >= now()`,
},
{
s: Series{
Measurement: "cpu",
Field: "value",
},
timeFilter: "time >= now()",
expected: `select "value" from "cpu" where time >= now()`,
},
{
s: Series{
Measurement: "cpu",
Field: "value",
},
expected: `select "value" from "cpu"`,
},
}
f := func(s *Series, timeFilter, resultExpected string) {
t.Helper()
for _, tc := range testCases {
query := tc.s.fetchQuery(tc.timeFilter)
if query != tc.expected {
t.Fatalf("got: \n%s;\nexpected: \n%s", query, tc.expected)
result := s.fetchQuery(timeFilter)
if result != resultExpected {
t.Fatalf("unexpected result\ngot\n%s\nwant\n%s", result, resultExpected)
}
}
f(&Series{
Measurement: "cpu",
Field: "value",
LabelPairs: []LabelPair{
{
Name: "foo",
Value: "bar",
},
},
}, "", `select "value" from "cpu" where "foo"::tag='bar'`)
f(&Series{
Measurement: "cpu",
Field: "value",
LabelPairs: []LabelPair{
{
Name: "foo",
Value: "bar",
},
{
Name: "baz",
Value: "qux",
},
},
}, "", `select "value" from "cpu" where "foo"::tag='bar' and "baz"::tag='qux'`)
f(&Series{
Measurement: "cpu",
Field: "value",
LabelPairs: []LabelPair{
{
Name: "foo",
Value: "b'ar",
},
},
}, "time >= now()", `select "value" from "cpu" where "foo"::tag='b\'ar' and time >= now()`)
f(&Series{
Measurement: "cpu",
Field: "value",
LabelPairs: []LabelPair{
{
Name: "name",
Value: `dev-mapper-centos\x2dswap.swap`,
},
{
Name: "state",
Value: "dev-mapp'er-c'en'tos",
},
},
}, "time >= now()", `select "value" from "cpu" where "name"::tag='dev-mapper-centos\\x2dswap.swap' and "state"::tag='dev-mapp\'er-c\'en\'tos' and time >= now()`)
f(&Series{
Measurement: "cpu",
Field: "value",
}, "time >= now()", `select "value" from "cpu" where time >= now()`)
f(&Series{
Measurement: "cpu",
Field: "value",
}, "", `select "value" from "cpu"`)
}
func TestTimeFilter(t *testing.T) {
testCases := []struct {
start string
end string
expected string
}{
{
start: "2020-01-01T20:07:00Z",
end: "2020-01-01T21:07:00Z",
expected: "time >= '2020-01-01T20:07:00Z' and time <= '2020-01-01T21:07:00Z'",
},
{
expected: "",
},
{
start: "2020-01-01T20:07:00Z",
expected: "time >= '2020-01-01T20:07:00Z'",
},
{
end: "2020-01-01T21:07:00Z",
expected: "time <= '2020-01-01T21:07:00Z'",
},
}
for _, tc := range testCases {
f := timeFilter(tc.start, tc.end)
if f != tc.expected {
t.Fatalf("got: \n%q;\nexpected: \n%q", f, tc.expected)
f := func(start, end, resultExpected string) {
t.Helper()
result := timeFilter(start, end)
if result != resultExpected {
t.Fatalf("unexpected result\ngot\n%v\nwant\n%s", result, resultExpected)
}
}
// no start and end filters
f("", "", "")
// missing end filter
f("2020-01-01T20:07:00Z", "", "time >= '2020-01-01T20:07:00Z'")
// missing start filter
f("", "2020-01-01T21:07:00Z", "time <= '2020-01-01T21:07:00Z'")
// both start and end filters
f("2020-01-01T20:07:00Z", "2020-01-01T21:07:00Z", "time >= '2020-01-01T20:07:00Z' and time <= '2020-01-01T21:07:00Z'")
}

View File

@@ -12,7 +12,7 @@ import (
type queryValues struct {
name string
values map[string][]interface{}
values map[string][]any
}
func parseResult(r influx.Result) ([]queryValues, error) {
@@ -21,7 +21,7 @@ func parseResult(r influx.Result) ([]queryValues, error) {
}
qValues := make([]queryValues, len(r.Series))
for i, row := range r.Series {
values := make(map[string][]interface{}, len(row.Values))
values := make(map[string][]any, len(row.Values))
for _, value := range row.Values {
for idx, v := range value {
key := row.Columns[idx]
@@ -36,7 +36,7 @@ func parseResult(r influx.Result) ([]queryValues, error) {
return qValues, nil
}
func toFloat64(v interface{}) (float64, error) {
func toFloat64(v any) (float64, error) {
switch i := v.(type) {
case json.Number:
return i.Float64()

View File

@@ -6,71 +6,71 @@ import (
"testing"
)
func TestSeries_Unmarshal(t *testing.T) {
func TestSeriesUnmarshal(t *testing.T) {
f := func(s string, resultExpected *Series) {
t.Helper()
result := &Series{}
if err := result.unmarshal(s); err != nil {
t.Fatalf("cannot unmarshal series from %q: %s", s, err)
}
if !reflect.DeepEqual(result, resultExpected) {
t.Fatalf("unexpected result\ngot\n%v\nwant\n%v", result, resultExpected)
}
}
tag := func(name, value string) LabelPair {
return LabelPair{
Name: name,
Value: value,
}
}
series := func(measurement string, lp ...LabelPair) Series {
return Series{
series := func(measurement string, lp ...LabelPair) *Series {
return &Series{
Measurement: measurement,
LabelPairs: lp,
}
}
testCases := []struct {
got string
want Series
}{
{
got: "cpu",
want: series("cpu"),
},
{
got: "cpu,host=localhost",
want: series("cpu", tag("host", "localhost")),
},
{
got: "cpu,host=localhost,instance=instance",
want: series("cpu", tag("host", "localhost"), tag("instance", "instance")),
},
{
got: `fo\,bar\=baz,x\=\b=\\a\,\=\q\ `,
want: series("fo,bar=baz", tag(`x=\b`, `\a,=\q `)),
},
{
got: "cpu,host=192.168.0.1,instance=fe80::fdc8:5e36:c2c6:baac%utun1",
want: series("cpu", tag("host", "192.168.0.1"), tag("instance", "fe80::fdc8:5e36:c2c6:baac%utun1")),
},
{
got: `cpu,db=db1,host=localhost,server=host\=localhost\ user\=user\ `,
want: series("cpu", tag("db", "db1"),
tag("host", "localhost"), tag("server", "host=localhost user=user ")),
},
}
for _, tc := range testCases {
s := Series{}
if err := s.unmarshal(tc.got); err != nil {
t.Fatalf("%q: unmarshal err: %s", tc.got, err)
}
if !reflect.DeepEqual(s, tc.want) {
t.Fatalf("%q: expected\n%#v\nto be equal\n%#v", tc.got, s, tc.want)
}
}
f("cpu", series("cpu"))
f("cpu,host=localhost", series("cpu", tag("host", "localhost")))
f("cpu,host=localhost,instance=instance", series("cpu", tag("host", "localhost"), tag("instance", "instance")))
f(`fo\,bar\=baz,x\=\b=\\a\,\=\q\ `, series("fo,bar=baz", tag(`x=\b`, `\a,=\q `)))
f("cpu,host=192.168.0.1,instance=fe80::fdc8:5e36:c2c6:baac%utun1", series("cpu", tag("host", "192.168.0.1"), tag("instance", "fe80::fdc8:5e36:c2c6:baac%utun1")))
f(`cpu,db=db1,host=localhost,server=host\=localhost\ user\=user\ `, series("cpu", tag("db", "db1"), tag("host", "localhost"), tag("server", "host=localhost user=user ")))
}
func TestToFloat64(t *testing.T) {
f := func(in interface{}, want float64) {
func TestToFloat64_Failure(t *testing.T) {
f := func(in any) {
t.Helper()
got, err := toFloat64(in)
if err != nil {
t.Fatalf("unexpected err: %s", err)
}
if got != want {
t.Errorf("got %v; want %v", got, want)
_, err := toFloat64(in)
if err == nil {
t.Fatalf("expecting non-nil error")
}
}
f("text")
}
func TestToFloat64_Success(t *testing.T) {
f := func(in any, resultExpected float64) {
t.Helper()
result, err := toFloat64(in)
if err != nil {
t.Fatalf("unexpected error: %s", err)
}
if result != resultExpected {
t.Fatalf("unexpected result: got %v; want %v", result, resultExpected)
}
}
f("123.4", 123.4)
f(float64(123.4), 123.4)
f(float32(12), 12)
@@ -78,9 +78,4 @@ func TestToFloat64(t *testing.T) {
f(true, 1)
f(false, 0)
f(json.Number("123456.789"), 123456.789)
_, err := toFloat64("text")
if err == nil {
t.Fatalf("expected to get err; got nil instead")
}
}

View File

@@ -75,12 +75,6 @@ type TimeRange struct {
type MetaResults struct {
Type string `json:"type"`
Results []Meta `json:"results"`
//metric string
//tags interface{}
//limit int
//time int
//startIndex int
//totalResults int
}
// Meta A meta object about a metric
@@ -88,7 +82,6 @@ type MetaResults struct {
type Meta struct {
Metric string `json:"metric"`
Tags map[string]string `json:"tags"`
//tsuid string
}
// OtsdbMetric is a single series in OpenTSDB's returned format

View File

@@ -5,30 +5,27 @@ import (
)
func TestInRange(t *testing.T) {
testCases := []struct {
filterMin, filterMax int64
blockMin, blockMax int64
expected bool
}{
{0, 0, 1, 2, true},
{0, 3, 1, 2, true},
{0, 3, 4, 5, false},
{3, 0, 1, 2, false},
{3, 0, 2, 4, true},
{3, 10, 1, 2, false},
{3, 10, 1, 4, true},
{3, 10, 5, 9, true},
{3, 10, 9, 12, true},
{3, 10, 12, 15, false},
}
for _, tc := range testCases {
f := func(filterMin, filterMax, blockMin, blockMax int64, resultExpected bool) {
t.Helper()
f := filter{
min: tc.filterMin,
max: tc.filterMax,
min: filterMin,
max: filterMax,
}
got := f.inRange(tc.blockMin, tc.blockMax)
if got != tc.expected {
t.Fatalf("got %v; expected %v: %v", got, tc.expected, tc)
result := f.inRange(blockMin, blockMax)
if result != resultExpected {
t.Fatalf("unexpected result; got %v; want %v", result, resultExpected)
}
}
f(0, 0, 1, 2, true)
f(0, 3, 1, 2, true)
f(0, 3, 4, 5, false)
f(3, 0, 1, 2, false)
f(3, 0, 2, 4, true)
f(3, 10, 1, 2, false)
f(3, 10, 1, 4, true)
f(3, 10, 5, 9, true)
f(3, 10, 9, 12, true)
f(3, 10, 12, 15, false)
}

View File

@@ -23,7 +23,7 @@ const (
)
// This test simulates close process if user abort it
func Test_prometheusProcessor_run(t *testing.T) {
func TestPrometheusProcessorRun(t *testing.T) {
t.Skip()
defer func() { isSilent = false }()
@@ -139,11 +139,11 @@ func Test_prometheusProcessor_run(t *testing.T) {
_, err = w.Write(input)
if err != nil {
t.Error(err)
t.Fatalf("cannot send 'Y' to importer: %s", err)
}
err = w.Close()
if err != nil {
t.Error(err)
t.Fatalf("cannot close writer: %s", err)
}
stdin := os.Stdin
@@ -151,7 +151,6 @@ func Test_prometheusProcessor_run(t *testing.T) {
defer func() {
os.Stdin = stdin
_ = r.Close()
_ = w.Close()
}()
os.Stdin = r
}
@@ -162,7 +161,7 @@ func Test_prometheusProcessor_run(t *testing.T) {
}
if err := pp.run(); (err != nil) != tt.wantErr {
t.Errorf("run() error = %v, wantErr %v", err, tt.wantErr)
t.Fatalf("run() error = %v, wantErr %v", err, tt.wantErr)
}
})
}

View File

@@ -16,535 +16,358 @@ func mustParseDatetime(t string) time.Time {
return result
}
func Test_splitDateRange(t *testing.T) {
type args struct {
start string
end string
granularity string
}
tests := []struct {
name string
args args
want []testTimeRange
wantErr bool
}{
{
name: "validates start is before end",
args: args{
start: "2022-02-01T00:00:00Z",
end: "2022-01-01T00:00:00Z",
granularity: StepMonth,
},
want: nil,
wantErr: true,
},
{
name: "validates granularity value",
args: args{
start: "2022-01-01T00:00:00Z",
end: "2022-02-01T00:00:00Z",
granularity: "non-existent-format",
},
want: nil,
wantErr: true,
},
{
name: "month chunking",
args: args{
start: "2022-01-03T11:11:11Z",
end: "2022-03-03T12:12:12Z",
granularity: StepMonth,
},
want: []testTimeRange{
{
"2022-01-03T11:11:11Z",
"2022-01-31T23:59:59.999999999Z",
},
{
"2022-02-01T00:00:00Z",
"2022-02-28T23:59:59.999999999Z",
},
{
"2022-03-01T00:00:00Z",
"2022-03-03T12:12:12Z",
},
},
wantErr: false,
},
{
name: "daily chunking",
args: args{
start: "2022-01-03T11:11:11Z",
end: "2022-01-05T12:12:12Z",
granularity: StepDay,
},
want: []testTimeRange{
{
"2022-01-03T11:11:11Z",
"2022-01-04T11:11:11Z",
},
{
"2022-01-04T11:11:11Z",
"2022-01-05T11:11:11Z",
},
{
"2022-01-05T11:11:11Z",
"2022-01-05T12:12:12Z",
},
},
wantErr: false,
},
{
name: "hourly chunking",
args: args{
start: "2022-01-03T11:11:11Z",
end: "2022-01-03T14:14:14Z",
granularity: StepHour,
},
want: []testTimeRange{
{
"2022-01-03T11:11:11Z",
"2022-01-03T12:11:11Z",
},
{
"2022-01-03T12:11:11Z",
"2022-01-03T13:11:11Z",
},
{
"2022-01-03T13:11:11Z",
"2022-01-03T14:11:11Z",
},
{
"2022-01-03T14:11:11Z",
"2022-01-03T14:14:14Z",
},
},
wantErr: false,
},
{
name: "month chunking with one day time range",
args: args{
start: "2022-01-03T11:11:11Z",
end: "2022-01-04T12:12:12Z",
granularity: StepMonth,
},
want: []testTimeRange{
{
"2022-01-03T11:11:11Z",
"2022-01-04T12:12:12Z",
},
},
wantErr: false,
},
{
name: "month chunking with same day time range",
args: args{
start: "2022-01-03T11:11:11Z",
end: "2022-01-03T12:12:12Z",
granularity: StepMonth,
},
want: []testTimeRange{
{
"2022-01-03T11:11:11Z",
"2022-01-03T12:12:12Z",
},
},
wantErr: false,
},
{
name: "month chunking with one month and two days range",
args: args{
start: "2022-01-03T11:11:11Z",
end: "2022-02-03T00:00:00Z",
granularity: StepMonth,
},
want: []testTimeRange{
{
"2022-01-03T11:11:11Z",
"2022-01-31T23:59:59.999999999Z",
},
{
"2022-02-01T00:00:00Z",
"2022-02-03T00:00:00Z",
},
},
wantErr: false,
},
{
name: "week chunking with not full week",
args: args{
start: "2023-07-30T00:00:00Z",
end: "2023-08-05T23:59:59.999999999Z",
granularity: StepWeek,
},
want: []testTimeRange{
{
"2023-07-30T00:00:00Z",
"2023-08-05T23:59:59.999999999Z",
},
},
},
{
name: "week chunking with start of the week and end of the week",
args: args{
start: "2023-07-30T00:00:00Z",
end: "2023-08-06T00:00:00Z",
granularity: StepWeek,
},
want: []testTimeRange{
{
"2023-07-30T00:00:00Z",
"2023-08-06T00:00:00Z",
},
},
},
{
name: "week chunking with next one day week",
args: args{
start: "2023-07-30T00:00:00Z",
end: "2023-08-07T01:12:00Z",
granularity: StepWeek,
},
want: []testTimeRange{
{
"2023-07-30T00:00:00Z",
"2023-08-06T00:00:00Z",
},
{
"2023-08-06T00:00:00Z",
"2023-08-07T01:12:00Z",
},
},
},
{
name: "week chunking with month and not full week representation",
args: args{
start: "2023-07-30T00:00:00Z",
end: "2023-09-01T01:12:00Z",
granularity: StepWeek,
},
want: []testTimeRange{
{
"2023-07-30T00:00:00Z",
"2023-08-06T00:00:00Z",
},
{
"2023-08-06T00:00:00Z",
"2023-08-13T00:00:00Z",
},
{
"2023-08-13T00:00:00Z",
"2023-08-20T00:00:00Z",
},
{
"2023-08-20T00:00:00Z",
"2023-08-27T00:00:00Z",
},
{
"2023-08-27T00:00:00Z",
"2023-09-01T01:12:00Z",
},
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
start := mustParseDatetime(tt.args.start)
end := mustParseDatetime(tt.args.end)
func TestSplitDateRange_Failure(t *testing.T) {
f := func(startStr, endStr, granularity string) {
t.Helper()
got, err := SplitDateRange(start, end, tt.args.granularity, false)
if (err != nil) != tt.wantErr {
t.Errorf("splitDateRange() error = %v, wantErr %v", err, tt.wantErr)
return
}
start := mustParseDatetime(startStr)
end := mustParseDatetime(endStr)
var testExpectedResults [][]time.Time
if tt.want != nil {
testExpectedResults = make([][]time.Time, 0)
for _, dr := range tt.want {
testExpectedResults = append(testExpectedResults, []time.Time{
mustParseDatetime(dr[0]),
mustParseDatetime(dr[1]),
})
}
}
if !reflect.DeepEqual(got, testExpectedResults) {
t.Errorf("splitDateRange() got = %v, want %v", got, testExpectedResults)
}
})
_, err := SplitDateRange(start, end, granularity, false)
if err == nil {
t.Fatalf("expecting non-nil result")
}
}
// validates start is before end
f("2022-02-01T00:00:00Z", "2022-01-01T00:00:00Z", StepMonth)
// validates granularity value
f("2022-01-01T00:00:00Z", "2022-02-01T00:00:00Z", "non-existent-format")
}
func Test_splitDateRange_reverse(t *testing.T) {
type args struct {
start string
end string
granularity string
timeReverse bool
}
tests := []struct {
name string
args args
want []testTimeRange
wantErr bool
}{
{
name: "validates start is before end",
args: args{
start: "2022-02-01T00:00:00Z",
end: "2022-01-01T00:00:00Z",
granularity: StepMonth,
timeReverse: true,
},
want: nil,
wantErr: true,
},
{
name: "validates granularity value",
args: args{
start: "2022-01-01T00:00:00Z",
end: "2022-02-01T00:00:00Z",
granularity: "non-existent-format",
timeReverse: true,
},
want: nil,
wantErr: true,
},
{
name: "month chunking",
args: args{
start: "2022-01-03T11:11:11Z",
end: "2022-03-03T12:12:12Z",
granularity: StepMonth,
timeReverse: true,
},
want: []testTimeRange{
{
"2022-03-01T00:00:00Z",
"2022-03-03T12:12:12Z",
},
{
"2022-02-01T00:00:00Z",
"2022-02-28T23:59:59.999999999Z",
},
{
"2022-01-03T11:11:11Z",
"2022-01-31T23:59:59.999999999Z",
},
},
wantErr: false,
},
{
name: "daily chunking",
args: args{
start: "2022-01-03T11:11:11Z",
end: "2022-01-05T12:12:12Z",
granularity: StepDay,
timeReverse: true,
},
want: []testTimeRange{
{
"2022-01-05T11:11:11Z",
"2022-01-05T12:12:12Z",
},
{
"2022-01-04T11:11:11Z",
"2022-01-05T11:11:11Z",
},
{
"2022-01-03T11:11:11Z",
"2022-01-04T11:11:11Z",
},
},
wantErr: false,
},
{
name: "hourly chunking",
args: args{
start: "2022-01-03T11:11:11Z",
end: "2022-01-03T14:14:14Z",
granularity: StepHour,
timeReverse: true,
},
want: []testTimeRange{
{
"2022-01-03T14:11:11Z",
"2022-01-03T14:14:14Z",
},
{
"2022-01-03T13:11:11Z",
"2022-01-03T14:11:11Z",
},
{
"2022-01-03T12:11:11Z",
"2022-01-03T13:11:11Z",
},
{
"2022-01-03T11:11:11Z",
"2022-01-03T12:11:11Z",
},
},
wantErr: false,
},
{
name: "month chunking with one day time range",
args: args{
start: "2022-01-03T11:11:11Z",
end: "2022-01-04T12:12:12Z",
granularity: StepMonth,
timeReverse: true,
},
want: []testTimeRange{
{
"2022-01-03T11:11:11Z",
"2022-01-04T12:12:12Z",
},
},
wantErr: false,
},
{
name: "month chunking with same day time range",
args: args{
start: "2022-01-03T11:11:11Z",
end: "2022-01-03T12:12:12Z",
granularity: StepMonth,
timeReverse: true,
},
want: []testTimeRange{
{
"2022-01-03T11:11:11Z",
"2022-01-03T12:12:12Z",
},
},
wantErr: false,
},
{
name: "month chunking with one month and two days range",
args: args{
start: "2022-01-03T11:11:11Z",
end: "2022-02-03T00:00:00Z",
granularity: StepMonth,
timeReverse: true,
},
want: []testTimeRange{
{
"2022-02-01T00:00:00Z",
"2022-02-03T00:00:00Z",
},
{
"2022-01-03T11:11:11Z",
"2022-01-31T23:59:59.999999999Z",
},
},
wantErr: false,
},
{
name: "week chunking with not full week",
args: args{
start: "2023-07-30T00:00:00Z",
end: "2023-08-05T23:59:59.999999999Z",
granularity: StepWeek,
timeReverse: true,
},
want: []testTimeRange{
{
"2023-07-30T00:00:00Z",
"2023-08-05T23:59:59.999999999Z",
},
},
},
{
name: "week chunking with start of the week and end of the week",
args: args{
start: "2023-07-30T00:00:00Z",
end: "2023-08-06T00:00:00Z",
granularity: StepWeek,
timeReverse: true,
},
want: []testTimeRange{
{
"2023-07-30T00:00:00Z",
"2023-08-06T00:00:00Z",
},
},
},
{
name: "week chunking with next one day week",
args: args{
start: "2023-07-30T00:00:00Z",
end: "2023-08-07T01:12:00Z",
granularity: StepWeek,
timeReverse: true,
},
want: []testTimeRange{
{
"2023-08-06T00:00:00Z",
"2023-08-07T01:12:00Z",
},
{
"2023-07-30T00:00:00Z",
"2023-08-06T00:00:00Z",
},
},
},
{
name: "week chunking with month and not full week representation",
args: args{
start: "2023-07-30T00:00:00Z",
end: "2023-09-01T01:12:00Z",
granularity: StepWeek,
timeReverse: true,
},
want: []testTimeRange{
{
"2023-08-27T00:00:00Z",
"2023-09-01T01:12:00Z",
},
{
"2023-08-20T00:00:00Z",
"2023-08-27T00:00:00Z",
},
{
"2023-08-13T00:00:00Z",
"2023-08-20T00:00:00Z",
},
{
"2023-08-06T00:00:00Z",
"2023-08-13T00:00:00Z",
},
{
"2023-07-30T00:00:00Z",
"2023-08-06T00:00:00Z",
},
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
start := mustParseDatetime(tt.args.start)
end := mustParseDatetime(tt.args.end)
func TestSplitDateRange_Success(t *testing.T) {
f := func(startStr, endStr, granularity string, resultExpected []testTimeRange) {
t.Helper()
got, err := SplitDateRange(start, end, tt.args.granularity, tt.args.timeReverse)
if (err != nil) != tt.wantErr {
t.Errorf("splitDateRange() error = %v, wantErr %v", err, tt.wantErr)
return
}
start := mustParseDatetime(startStr)
end := mustParseDatetime(endStr)
var testExpectedResults [][]time.Time
if tt.want != nil {
testExpectedResults = make([][]time.Time, 0)
for _, dr := range tt.want {
testExpectedResults = append(testExpectedResults, []time.Time{
mustParseDatetime(dr[0]),
mustParseDatetime(dr[1]),
})
}
}
result, err := SplitDateRange(start, end, granularity, false)
if err != nil {
t.Fatalf("SplitDateRange() error: %s", err)
}
if !reflect.DeepEqual(got, testExpectedResults) {
t.Errorf("splitDateRange() got = %v, want %v", got, testExpectedResults)
}
})
var testExpectedResults [][]time.Time
for _, dr := range resultExpected {
testExpectedResults = append(testExpectedResults, []time.Time{
mustParseDatetime(dr[0]),
mustParseDatetime(dr[1]),
})
}
if !reflect.DeepEqual(result, testExpectedResults) {
t.Fatalf("unexpected result\ngot\n%v\nwant\n%v", result, testExpectedResults)
}
}
// month chunking
f("2022-01-03T11:11:11Z", "2022-03-03T12:12:12Z", StepMonth, []testTimeRange{
{
"2022-01-03T11:11:11Z",
"2022-01-31T23:59:59.999999999Z",
},
{
"2022-02-01T00:00:00Z",
"2022-02-28T23:59:59.999999999Z",
},
{
"2022-03-01T00:00:00Z",
"2022-03-03T12:12:12Z",
},
})
// daily chunking
f("2022-01-03T11:11:11Z", "2022-01-05T12:12:12Z", StepDay, []testTimeRange{
{
"2022-01-03T11:11:11Z",
"2022-01-04T11:11:11Z",
},
{
"2022-01-04T11:11:11Z",
"2022-01-05T11:11:11Z",
},
{
"2022-01-05T11:11:11Z",
"2022-01-05T12:12:12Z",
},
})
// hourly chunking
f("2022-01-03T11:11:11Z", "2022-01-03T14:14:14Z", StepHour, []testTimeRange{
{
"2022-01-03T11:11:11Z",
"2022-01-03T12:11:11Z",
},
{
"2022-01-03T12:11:11Z",
"2022-01-03T13:11:11Z",
},
{
"2022-01-03T13:11:11Z",
"2022-01-03T14:11:11Z",
},
{
"2022-01-03T14:11:11Z",
"2022-01-03T14:14:14Z",
},
})
// month chunking with one day time range
f("2022-01-03T11:11:11Z", "2022-01-04T12:12:12Z", StepMonth, []testTimeRange{
{
"2022-01-03T11:11:11Z",
"2022-01-04T12:12:12Z",
},
})
// month chunking with same day time range
f("2022-01-03T11:11:11Z", "2022-01-03T12:12:12Z", StepMonth, []testTimeRange{
{
"2022-01-03T11:11:11Z",
"2022-01-03T12:12:12Z",
},
})
// month chunking with one month and two days range
f("2022-01-03T11:11:11Z", "2022-02-03T00:00:00Z", StepMonth, []testTimeRange{
{
"2022-01-03T11:11:11Z",
"2022-01-31T23:59:59.999999999Z",
},
{
"2022-02-01T00:00:00Z",
"2022-02-03T00:00:00Z",
},
})
// week chunking with not full week
f("2023-07-30T00:00:00Z", "2023-08-05T23:59:59.999999999Z", StepWeek, []testTimeRange{
{
"2023-07-30T00:00:00Z",
"2023-08-05T23:59:59.999999999Z",
},
})
// week chunking with start of the week and end of the week
f("2023-07-30T00:00:00Z", "2023-08-06T00:00:00Z", StepWeek, []testTimeRange{
{
"2023-07-30T00:00:00Z",
"2023-08-06T00:00:00Z",
},
})
// week chunking with next one day week
f("2023-07-30T00:00:00Z", "2023-08-07T01:12:00Z", StepWeek, []testTimeRange{
{
"2023-07-30T00:00:00Z",
"2023-08-06T00:00:00Z",
},
{
"2023-08-06T00:00:00Z",
"2023-08-07T01:12:00Z",
},
})
// week chunking with month and not full week representation
f("2023-07-30T00:00:00Z", "2023-09-01T01:12:00Z", StepWeek, []testTimeRange{
{
"2023-07-30T00:00:00Z",
"2023-08-06T00:00:00Z",
},
{
"2023-08-06T00:00:00Z",
"2023-08-13T00:00:00Z",
},
{
"2023-08-13T00:00:00Z",
"2023-08-20T00:00:00Z",
},
{
"2023-08-20T00:00:00Z",
"2023-08-27T00:00:00Z",
},
{
"2023-08-27T00:00:00Z",
"2023-09-01T01:12:00Z",
},
})
}
func TestSplitDateRange_Reverse_Failure(t *testing.T) {
f := func(startStr, endStr, granularity string) {
t.Helper()
start := mustParseDatetime(startStr)
end := mustParseDatetime(endStr)
_, err := SplitDateRange(start, end, granularity, true)
if err == nil {
t.Fatalf("expecting non-nil error")
}
}
// validates start is before end
f("2022-02-01T00:00:00Z", "2022-01-01T00:00:00Z", StepMonth)
// validates granularity value
f("2022-01-01T00:00:00Z", "2022-02-01T00:00:00Z", "non-existent-format")
}
func TestSplitDateRange_Reverse_Success(t *testing.T) {
f := func(startStr, endStr, granularity string, resultExpected []testTimeRange) {
t.Helper()
start := mustParseDatetime(startStr)
end := mustParseDatetime(endStr)
result, err := SplitDateRange(start, end, granularity, true)
if err != nil {
t.Fatalf("SplitDateRange() error: %s", err)
}
var testExpectedResults [][]time.Time
for _, dr := range resultExpected {
testExpectedResults = append(testExpectedResults, []time.Time{
mustParseDatetime(dr[0]),
mustParseDatetime(dr[1]),
})
}
if !reflect.DeepEqual(result, testExpectedResults) {
t.Fatalf("unexpected result\ngot\n%v\nwant\n%v", result, testExpectedResults)
}
}
// month chunking
f("2022-01-03T11:11:11Z", "2022-03-03T12:12:12Z", StepMonth, []testTimeRange{
{
"2022-03-01T00:00:00Z",
"2022-03-03T12:12:12Z",
},
{
"2022-02-01T00:00:00Z",
"2022-02-28T23:59:59.999999999Z",
},
{
"2022-01-03T11:11:11Z",
"2022-01-31T23:59:59.999999999Z",
},
})
// daily chunking
f("2022-01-03T11:11:11Z", "2022-01-05T12:12:12Z", StepDay, []testTimeRange{
{
"2022-01-05T11:11:11Z",
"2022-01-05T12:12:12Z",
},
{
"2022-01-04T11:11:11Z",
"2022-01-05T11:11:11Z",
},
{
"2022-01-03T11:11:11Z",
"2022-01-04T11:11:11Z",
},
})
// hourly chunking
f("2022-01-03T11:11:11Z", "2022-01-03T14:14:14Z", StepHour, []testTimeRange{
{
"2022-01-03T14:11:11Z",
"2022-01-03T14:14:14Z",
},
{
"2022-01-03T13:11:11Z",
"2022-01-03T14:11:11Z",
},
{
"2022-01-03T12:11:11Z",
"2022-01-03T13:11:11Z",
},
{
"2022-01-03T11:11:11Z",
"2022-01-03T12:11:11Z",
},
})
// month chunking with one day time range
f("2022-01-03T11:11:11Z", "2022-01-04T12:12:12Z", StepMonth, []testTimeRange{
{
"2022-01-03T11:11:11Z",
"2022-01-04T12:12:12Z",
},
})
// month chunking with same day time range
f("2022-01-03T11:11:11Z", "2022-01-03T12:12:12Z", StepMonth, []testTimeRange{
{
"2022-01-03T11:11:11Z",
"2022-01-03T12:12:12Z",
},
})
// month chunking with one month and two days range
f("2022-01-03T11:11:11Z", "2022-02-03T00:00:00Z", StepMonth, []testTimeRange{
{
"2022-02-01T00:00:00Z",
"2022-02-03T00:00:00Z",
},
{
"2022-01-03T11:11:11Z",
"2022-01-31T23:59:59.999999999Z",
},
})
// week chunking with not full week
f("2023-07-30T00:00:00Z", "2023-08-05T23:59:59.999999999Z", StepWeek, []testTimeRange{
{
"2023-07-30T00:00:00Z",
"2023-08-05T23:59:59.999999999Z",
},
})
// week chunking with start of the week and end of the week
f("2023-07-30T00:00:00Z", "2023-08-06T00:00:00Z", StepWeek, []testTimeRange{
{
"2023-07-30T00:00:00Z",
"2023-08-06T00:00:00Z",
},
})
// week chunking with next one day week
f("2023-07-30T00:00:00Z", "2023-08-07T01:12:00Z", StepWeek, []testTimeRange{
{
"2023-08-06T00:00:00Z",
"2023-08-07T01:12:00Z",
},
{
"2023-07-30T00:00:00Z",
"2023-08-06T00:00:00Z",
},
})
// week chunking with month and not full week representation
f("2023-07-30T00:00:00Z", "2023-09-01T01:12:00Z", StepWeek, []testTimeRange{
{
"2023-08-27T00:00:00Z",
"2023-09-01T01:12:00Z",
},
{
"2023-08-20T00:00:00Z",
"2023-08-27T00:00:00Z",
},
{
"2023-08-13T00:00:00Z",
"2023-08-20T00:00:00Z",
},
{
"2023-08-06T00:00:00Z",
"2023-08-13T00:00:00Z",
},
{
"2023-07-30T00:00:00Z",
"2023-08-06T00:00:00Z",
},
})
}

View File

@@ -255,8 +255,7 @@ func (rws *RemoteWriteServer) importNativeHandler(t *testing.T) http.Handler {
if !reflect.DeepEqual(gotTimeSeries, rws.expectedSeries) {
w.WriteHeader(http.StatusInternalServerError)
t.Errorf("datasets not equal, expected: %#v;\n got: %#v", rws.expectedSeries, gotTimeSeries)
return
t.Fatalf("datasets not equal, expected: %#v;\n got: %#v", rws.expectedSeries, gotTimeSeries)
}
w.WriteHeader(http.StatusNoContent)

View File

@@ -5,175 +5,87 @@ import (
"time"
)
func TestGetTime(t *testing.T) {
tests := []struct {
name string
s string
want func() time.Time
wantErr bool
}{
{
name: "empty string",
s: "",
want: func() time.Time { return time.Time{} },
wantErr: true,
},
{
name: "only year",
s: "2019",
want: func() time.Time {
t := time.Date(2019, 1, 1, 0, 0, 0, 0, time.UTC)
return t
},
},
{
name: "year and month",
s: "2019-01",
want: func() time.Time {
t := time.Date(2019, 1, 1, 0, 0, 0, 0, time.UTC)
return t
},
},
{
name: "year and not first month",
s: "2019-02",
want: func() time.Time {
t := time.Date(2019, 2, 1, 0, 0, 0, 0, time.UTC)
return t
},
},
{
name: "year, month and day",
s: "2019-02-01",
want: func() time.Time {
t := time.Date(2019, 2, 1, 0, 0, 0, 0, time.UTC)
return t
},
},
{
name: "year, month and not first day",
s: "2019-02-10",
want: func() time.Time {
t := time.Date(2019, 2, 10, 0, 0, 0, 0, time.UTC)
return t
},
},
{
name: "year, month, day and time",
s: "2019-02-02T00",
want: func() time.Time {
t := time.Date(2019, 2, 2, 0, 0, 0, 0, time.UTC)
return t
},
},
{
name: "year, month, day and one hour time",
s: "2019-02-02T01",
want: func() time.Time {
t := time.Date(2019, 2, 2, 1, 0, 0, 0, time.UTC)
return t
},
},
{
name: "time with zero minutes",
s: "2019-02-02T01:00",
want: func() time.Time {
t := time.Date(2019, 2, 2, 1, 0, 0, 0, time.UTC)
return t
},
},
{
name: "time with one minute",
s: "2019-02-02T01:01",
want: func() time.Time {
t := time.Date(2019, 2, 2, 1, 1, 0, 0, time.UTC)
return t
},
},
{
name: "time with zero seconds",
s: "2019-02-02T01:01:00",
want: func() time.Time {
t := time.Date(2019, 2, 2, 1, 1, 0, 0, time.UTC)
return t
},
},
{
name: "timezone with one second",
s: "2019-02-02T01:01:01",
want: func() time.Time {
t := time.Date(2019, 2, 2, 1, 1, 1, 0, time.UTC)
return t
},
},
{
name: "time with two second and timezone",
s: "2019-07-07T20:01:02Z",
want: func() time.Time {
t := time.Date(2019, 7, 7, 20, 1, 02, 0, time.UTC)
return t
},
},
{
name: "time with seconds and timezone",
s: "2019-07-07T20:47:40+03:00",
want: func() time.Time {
l, _ := time.LoadLocation("Europe/Kiev")
t := time.Date(2019, 7, 7, 20, 47, 40, 0, l)
return t
},
},
{
name: "negative time",
s: "-292273086-05-16T16:47:06Z",
want: func() time.Time { return time.Time{} },
wantErr: true,
},
{
name: "float timestamp representation",
s: "1562529662.324",
want: func() time.Time {
t := time.Date(2019, 7, 7, 20, 01, 02, 324e6, time.UTC)
return t
},
},
{
name: "negative timestamp",
s: "-9223372036.855",
want: func() time.Time {
return time.Date(1970, 01, 01, 00, 00, 00, 00, time.UTC)
},
wantErr: false,
},
{
name: "big timestamp",
s: "1223372036855",
want: func() time.Time {
t := time.Date(2008, 10, 7, 9, 33, 56, 855e6, time.UTC)
return t
},
wantErr: false,
},
{
name: "duration time",
s: "1h5m",
want: func() time.Time {
t := time.Now().Add(-1 * time.Hour).Add(-5 * time.Minute)
return t
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := ParseTime(tt.s)
if (err != nil) != tt.wantErr {
t.Errorf("ParseTime() error = %v, wantErr %v", err, tt.wantErr)
return
}
w := tt.want()
if got.Unix() != w.Unix() {
t.Errorf("ParseTime() got = %v, want %v", got, w)
}
})
func TestGetTime_Failure(t *testing.T) {
f := func(s string) {
t.Helper()
_, err := ParseTime(s)
if err == nil {
t.Fatalf("expecting non-nil error")
}
}
// empty string
f("")
// negative time
f("-292273086-05-16T16:47:06Z")
}
func TestGetTime_Success(t *testing.T) {
f := func(s string, resultExpected time.Time) {
t.Helper()
result, err := ParseTime(s)
if err != nil {
t.Fatalf("ParseTime() error: %s", err)
}
if result.Unix() != resultExpected.Unix() {
t.Fatalf("unexpected result; got %s; want %s", result, resultExpected)
}
}
// only year
f("2019", time.Date(2019, 1, 1, 0, 0, 0, 0, time.UTC))
// year and month
f("2019-01", time.Date(2019, 1, 1, 0, 0, 0, 0, time.UTC))
// year and not first month
f("2019-02", time.Date(2019, 2, 1, 0, 0, 0, 0, time.UTC))
// year, month and day
f("2019-02-01", time.Date(2019, 2, 1, 0, 0, 0, 0, time.UTC))
// year, month and not first day
f("2019-02-10", time.Date(2019, 2, 10, 0, 0, 0, 0, time.UTC))
// year, month, day and time
f("2019-02-02T00", time.Date(2019, 2, 2, 0, 0, 0, 0, time.UTC))
// year, month, day and one hour time
f("2019-02-02T01", time.Date(2019, 2, 2, 1, 0, 0, 0, time.UTC))
// time with zero minutes
f("2019-02-02T01:00", time.Date(2019, 2, 2, 1, 0, 0, 0, time.UTC))
// time with one minute
f("2019-02-02T01:01", time.Date(2019, 2, 2, 1, 1, 0, 0, time.UTC))
// time with zero seconds
f("2019-02-02T01:01:00", time.Date(2019, 2, 2, 1, 1, 0, 0, time.UTC))
// timezone with one second
f("2019-02-02T01:01:01", time.Date(2019, 2, 2, 1, 1, 1, 0, time.UTC))
// time with two second and timezone
f("2019-07-07T20:01:02Z", time.Date(2019, 7, 7, 20, 1, 02, 0, time.UTC))
// time with seconds and timezone
f("2019-07-07T20:47:40+03:00", func() time.Time {
l, _ := time.LoadLocation("Europe/Kiev")
return time.Date(2019, 7, 7, 20, 47, 40, 0, l)
}())
// float timestamp representation",
f("1562529662.324", time.Date(2019, 7, 7, 20, 01, 02, 324e6, time.UTC))
// negative timestamp
f("-9223372036.855", time.Date(1970, 01, 01, 00, 00, 00, 00, time.UTC))
// big timestamp
f("1223372036855", time.Date(2008, 10, 7, 9, 33, 56, 855e6, time.UTC))
// duration time
f("1h5m", time.Now().Add(-1*time.Hour).Add(-5*time.Minute))
}

View File

@@ -45,7 +45,7 @@ type cWriter struct {
err error
}
func (cw *cWriter) printf(format string, args ...interface{}) {
func (cw *cWriter) printf(format string, args ...any) {
if cw.err != nil {
return
}

View File

@@ -7,83 +7,68 @@ import (
"testing"
)
func TestTimeSeries_Write(t *testing.T) {
var testCases = []struct {
name string
ts *TimeSeries
exp string
}{
{
name: "one datapoint",
ts: &TimeSeries{
Name: "foo",
LabelPairs: []LabelPair{
{
Name: "key",
Value: "val",
},
},
Timestamps: []int64{1577877162200},
Values: []float64{1},
},
exp: `{"metric":{"__name__":"foo","key":"val"},"timestamps":[1577877162200],"values":[1]}`,
},
{
name: "multiple samples",
ts: &TimeSeries{
Name: "foo",
LabelPairs: []LabelPair{
{
Name: "key",
Value: "val",
},
},
Timestamps: []int64{1577877162200, 15778771622400, 15778771622600},
Values: []float64{1, 1.6263, 32.123},
},
exp: `{"metric":{"__name__":"foo","key":"val"},"timestamps":[1577877162200,15778771622400,15778771622600],"values":[1,1.6263,32.123]}`,
},
{
name: "no samples",
ts: &TimeSeries{
Name: "foo",
LabelPairs: []LabelPair{
{
Name: "key",
Value: "val",
},
},
},
exp: ``,
},
{
name: "inf values",
ts: &TimeSeries{
Name: "foo",
LabelPairs: []LabelPair{
{
Name: "key",
Value: "val",
},
},
Timestamps: []int64{1577877162200, 1577877162200, 1577877162200},
Values: []float64{0, math.Inf(-1), math.Inf(1)},
},
exp: `{"metric":{"__name__":"foo","key":"val"},"timestamps":[1577877162200,1577877162200,1577877162200],"values":[0,-Inf,+Inf]}`,
},
func TestTimeSeriesWrite(t *testing.T) {
f := func(ts *TimeSeries, resultExpected string) {
t.Helper()
var b bytes.Buffer
_, err := ts.write(&b)
if err != nil {
t.Fatalf("error in TimeSeries.write: %s", err)
}
result := strings.TrimSpace(b.String())
if result != resultExpected {
t.Fatalf("unexpected result\ngot\n%v\nwant\n%v", result, resultExpected)
}
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
b := &bytes.Buffer{}
_, err := tc.ts.write(b)
if err != nil {
t.Error(err)
}
got := strings.TrimSpace(b.String())
if got != tc.exp {
t.Fatalf("\ngot: %q\nwant: %q", got, tc.exp)
}
})
}
// one datapoint
f(&TimeSeries{
Name: "foo",
LabelPairs: []LabelPair{
{
Name: "key",
Value: "val",
},
},
Timestamps: []int64{1577877162200},
Values: []float64{1},
}, `{"metric":{"__name__":"foo","key":"val"},"timestamps":[1577877162200],"values":[1]}`)
// multiple samples
f(&TimeSeries{
Name: "foo",
LabelPairs: []LabelPair{
{
Name: "key",
Value: "val",
},
},
Timestamps: []int64{1577877162200, 15778771622400, 15778771622600},
Values: []float64{1, 1.6263, 32.123},
}, `{"metric":{"__name__":"foo","key":"val"},"timestamps":[1577877162200,15778771622400,15778771622600],"values":[1,1.6263,32.123]}`)
// no samples
f(&TimeSeries{
Name: "foo",
LabelPairs: []LabelPair{
{
Name: "key",
Value: "val",
},
},
}, ``)
// inf values
f(&TimeSeries{
Name: "foo",
LabelPairs: []LabelPair{
{
Name: "key",
Value: "val",
},
},
Timestamps: []int64{1577877162200, 1577877162200, 1577877162200},
Values: []float64{0, math.Inf(-1), math.Inf(1)},
}, `{"metric":{"__name__":"foo","key":"val"},"timestamps":[1577877162200,1577877162200,1577877162200],"values":[0,-Inf,+Inf]}`)
}

View File

@@ -2,68 +2,42 @@ package vm
import "testing"
func TestAddExtraLabelsToImportPath(t *testing.T) {
type args struct {
path string
extraLabels []string
}
tests := []struct {
name string
args args
want string
wantErr bool
}{
{
name: "ok w/o extra labels",
args: args{
path: "/api/v1/import",
},
want: "/api/v1/import",
},
{
name: "ok one extra label",
args: args{
path: "/api/v1/import",
extraLabels: []string{"instance=host-1"},
},
want: "/api/v1/import?extra_label=instance=host-1",
},
{
name: "ok two extra labels",
args: args{
path: "/api/v1/import",
extraLabels: []string{"instance=host-2", "job=vmagent"},
},
want: "/api/v1/import?extra_label=instance=host-2&extra_label=job=vmagent",
},
{
name: "ok two extra with exist param",
args: args{
path: "/api/v1/import?timeout=50",
extraLabels: []string{"instance=host-2", "job=vmagent"},
},
want: "/api/v1/import?timeout=50&extra_label=instance=host-2&extra_label=job=vmagent",
},
{
name: "bad incorrect format for extra label",
args: args{
path: "/api/v1/import",
extraLabels: []string{"label=value", "bad_label_wo_value"},
},
want: "/api/v1/import",
wantErr: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := AddExtraLabelsToImportPath(tt.args.path, tt.args.extraLabels)
if (err != nil) != tt.wantErr {
t.Errorf("AddExtraLabelsToImportPath() error = %v, wantErr %v", err, tt.wantErr)
return
}
if got != tt.want {
t.Errorf("AddExtraLabelsToImportPath() got = %v, want %v", got, tt.want)
}
})
func TestAddExtraLabelsToImportPath_Failure(t *testing.T) {
f := func(path string, extraLabels []string) {
t.Helper()
_, err := AddExtraLabelsToImportPath(path, extraLabels)
if err == nil {
t.Fatalf("expecting non-nil error")
}
}
// bad incorrect format for extra label
f("/api/v1/import", []string{"label=value", "bad_label_wo_value"})
}
func TestAddExtraLabelsToImportPath_Success(t *testing.T) {
f := func(path string, extraLabels []string, resultExpected string) {
t.Helper()
result, err := AddExtraLabelsToImportPath(path, extraLabels)
if err != nil {
t.Fatalf("AddExtraLabelsToImportPath() error: %s", err)
}
if result != resultExpected {
t.Fatalf("unexpected result; got %q; want %q", result, resultExpected)
}
}
// ok w/o extra labels
f("/api/v1/import", nil, "/api/v1/import")
// ok one extra label
f("/api/v1/import", []string{"instance=host-1"}, "/api/v1/import?extra_label=instance=host-1")
// ok two extra labels
f("/api/v1/import", []string{"instance=host-2", "job=vmagent"}, "/api/v1/import?extra_label=instance=host-2&extra_label=job=vmagent")
// ok two extra with exist param
f("/api/v1/import?timeout=50", []string{"instance=host-2", "job=vmagent"}, "/api/v1/import?timeout=50&extra_label=instance=host-2&extra_label=job=vmagent")
}

View File

@@ -176,10 +176,10 @@ func (p *vmNativeProcessor) runBackfilling(ctx context.Context, tenantID string,
}
initMessage := "Initing import process from %q to %q with filter %s"
initParams := []interface{}{srcURL, dstURL, p.filter.String()}
initParams := []any{srcURL, dstURL, p.filter.String()}
if p.interCluster {
initMessage = "Initing import process from %q to %q with filter %s for tenant %s"
initParams = []interface{}{srcURL, dstURL, p.filter.String(), tenantID}
initParams = []any{srcURL, dstURL, p.filter.String(), tenantID}
}
fmt.Println("") // extra line for better output formatting

View File

@@ -13,7 +13,6 @@ import (
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/backoff"
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/native"
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/stepper"
remote_read_integration "github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/testdata/servers_integration_test"
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/vm"
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql"
@@ -27,7 +26,82 @@ const (
retentionPeriod = "100y"
)
func Test_vmNativeProcessor_run(t *testing.T) {
func TestVMNativeProcessorRun(t *testing.T) {
f := func(startStr, endStr string, numOfSeries, numOfSamples int, resultExpected []vm.TimeSeries) {
t.Helper()
src := remote_read_integration.NewRemoteWriteServer(t)
dst := remote_read_integration.NewRemoteWriteServer(t)
defer func() {
src.Close()
dst.Close()
}()
start, err := time.Parse(time.RFC3339, startStr)
if err != nil {
t.Fatalf("cannot parse start time: %s", err)
}
end, err := time.Parse(time.RFC3339, endStr)
if err != nil {
t.Fatalf("cannot parse end time: %s", err)
}
matchName := "__name__"
matchValue := ".*"
filter := native.Filter{
Match: fmt.Sprintf("{%s=~%q}", matchName, matchValue),
TimeStart: startStr,
TimeEnd: endStr,
}
rws := remote_read_integration.GenerateVNSeries(start.Unix(), end.Unix(), int64(numOfSeries), int64(numOfSamples))
src.Series(rws)
dst.ExpectedSeries(resultExpected)
if err := fillStorage(rws); err != nil {
t.Fatalf("cannot add series to storage: %s", err)
}
srcClient := &native.Client{
AuthCfg: nil,
Addr: src.URL(),
ExtraLabels: []string{},
HTTPClient: &http.Client{Transport: &http.Transport{DisableKeepAlives: false}},
}
dstClient := &native.Client{
AuthCfg: nil,
Addr: dst.URL(),
ExtraLabels: []string{},
HTTPClient: &http.Client{Transport: &http.Transport{DisableKeepAlives: false}},
}
isSilent = true
defer func() { isSilent = false }()
p := &vmNativeProcessor{
filter: filter,
dst: dstClient,
src: srcClient,
backoff: backoff.New(),
cc: 1,
isNative: true,
}
ctx := context.Background()
if err := p.run(ctx); err != nil {
t.Fatalf("run() error: %s", err)
}
deleted, err := deleteSeries(matchName, matchValue)
if err != nil {
t.Fatalf("cannot delete series: %s", err)
}
if deleted != numOfSeries {
t.Fatalf("unexpected number of deleted series; got %d; want %d", deleted, numOfSeries)
}
}
processFlags()
vmstorage.Init(promql.ResetRollupResultCacheIfNeeded)
@@ -42,214 +116,78 @@ func Test_vmNativeProcessor_run(t *testing.T) {
defer func() {
barpool.Disable(false)
}()
defer func() { isSilent = false }()
type fields struct {
filter native.Filter
dst *native.Client
src *native.Client
backoff *backoff.Backoff
s *stats
rateLimit int64
interCluster bool
cc int
matchName string
matchValue string
}
type args struct {
ctx context.Context
silent bool
}
tests := []struct {
name string
fields fields
args args
vmSeries func(start, end, numOfSeries, numOfSamples int64) []vm.TimeSeries
expectedSeries []vm.TimeSeries
start string
end string
numOfSamples int64
numOfSeries int64
chunk string
wantErr bool
}{
// step minute on minute time range
start := "2022-11-25T11:23:05+02:00"
end := "2022-11-27T11:24:05+02:00"
numOfSeries := 3
numOfSamples := 2
resultExpected := []vm.TimeSeries{
{
name: "step minute on minute time range",
start: "2022-11-25T11:23:05+02:00",
end: "2022-11-27T11:24:05+02:00",
numOfSamples: 2,
numOfSeries: 3,
chunk: stepper.StepMinute,
fields: fields{
filter: native.Filter{},
backoff: backoff.New(),
rateLimit: 0,
interCluster: false,
cc: 1,
matchName: "__name__",
matchValue: ".*",
},
args: args{
ctx: context.Background(),
silent: true,
},
vmSeries: remote_read_integration.GenerateVNSeries,
expectedSeries: []vm.TimeSeries{
{
Name: "vm_metric_1",
LabelPairs: []vm.LabelPair{{Name: "job", Value: "0"}},
Timestamps: []int64{1669368185000, 1669454615000},
Values: []float64{0, 0},
},
{
Name: "vm_metric_1",
LabelPairs: []vm.LabelPair{{Name: "job", Value: "1"}},
Timestamps: []int64{1669368185000, 1669454615000},
Values: []float64{100, 100},
},
{
Name: "vm_metric_1",
LabelPairs: []vm.LabelPair{{Name: "job", Value: "2"}},
Timestamps: []int64{1669368185000, 1669454615000},
Values: []float64{200, 200},
},
},
wantErr: false,
Name: "vm_metric_1",
LabelPairs: []vm.LabelPair{{Name: "job", Value: "0"}},
Timestamps: []int64{1669368185000, 1669454615000},
Values: []float64{0, 0},
},
{
name: "step month on month time range",
start: "2022-09-26T11:23:05+02:00",
end: "2022-11-26T11:24:05+02:00",
numOfSamples: 2,
numOfSeries: 3,
chunk: stepper.StepMonth,
fields: fields{
filter: native.Filter{},
backoff: backoff.New(),
rateLimit: 0,
interCluster: false,
cc: 1,
matchName: "__name__",
matchValue: ".*",
},
args: args{
ctx: context.Background(),
silent: true,
},
vmSeries: remote_read_integration.GenerateVNSeries,
expectedSeries: []vm.TimeSeries{
{
Name: "vm_metric_1",
LabelPairs: []vm.LabelPair{{Name: "job", Value: "0"}},
Timestamps: []int64{1664184185000},
Values: []float64{0},
},
{
Name: "vm_metric_1",
LabelPairs: []vm.LabelPair{{Name: "job", Value: "0"}},
Timestamps: []int64{1666819415000},
Values: []float64{0},
},
{
Name: "vm_metric_1",
LabelPairs: []vm.LabelPair{{Name: "job", Value: "1"}},
Timestamps: []int64{1664184185000},
Values: []float64{100},
},
{
Name: "vm_metric_1",
LabelPairs: []vm.LabelPair{{Name: "job", Value: "1"}},
Timestamps: []int64{1666819415000},
Values: []float64{100},
},
{
Name: "vm_metric_1",
LabelPairs: []vm.LabelPair{{Name: "job", Value: "2"}},
Timestamps: []int64{1664184185000},
Values: []float64{200},
},
{
Name: "vm_metric_1",
LabelPairs: []vm.LabelPair{{Name: "job", Value: "2"}},
Timestamps: []int64{1666819415000},
Values: []float64{200},
},
},
wantErr: false,
Name: "vm_metric_1",
LabelPairs: []vm.LabelPair{{Name: "job", Value: "1"}},
Timestamps: []int64{1669368185000, 1669454615000},
Values: []float64{100, 100},
},
{
Name: "vm_metric_1",
LabelPairs: []vm.LabelPair{{Name: "job", Value: "2"}},
Timestamps: []int64{1669368185000, 1669454615000},
Values: []float64{200, 200},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
src := remote_read_integration.NewRemoteWriteServer(t)
dst := remote_read_integration.NewRemoteWriteServer(t)
f(start, end, numOfSeries, numOfSamples, resultExpected)
defer func() {
src.Close()
dst.Close()
}()
start, err := time.Parse(time.RFC3339, tt.start)
if err != nil {
t.Fatalf("Error parse start time: %s", err)
}
end, err := time.Parse(time.RFC3339, tt.end)
if err != nil {
t.Fatalf("Error parse end time: %s", err)
}
tt.fields.filter.Match = fmt.Sprintf("{%s=~%q}", tt.fields.matchName, tt.fields.matchValue)
tt.fields.filter.TimeStart = tt.start
tt.fields.filter.TimeEnd = tt.end
rws := tt.vmSeries(start.Unix(), end.Unix(), tt.numOfSeries, tt.numOfSamples)
src.Series(rws)
dst.ExpectedSeries(tt.expectedSeries)
if err := fillStorage(rws); err != nil {
t.Fatalf("error add series to storage: %s", err)
}
tt.fields.src = &native.Client{
AuthCfg: nil,
Addr: src.URL(),
ExtraLabels: []string{},
HTTPClient: &http.Client{Transport: &http.Transport{DisableKeepAlives: false}},
}
tt.fields.dst = &native.Client{
AuthCfg: nil,
Addr: dst.URL(),
ExtraLabels: []string{},
HTTPClient: &http.Client{Transport: &http.Transport{DisableKeepAlives: false}},
}
isSilent = tt.args.silent
p := &vmNativeProcessor{
filter: tt.fields.filter,
dst: tt.fields.dst,
src: tt.fields.src,
backoff: tt.fields.backoff,
s: tt.fields.s,
rateLimit: tt.fields.rateLimit,
interCluster: tt.fields.interCluster,
cc: tt.fields.cc,
isNative: true,
}
if err := p.run(tt.args.ctx); (err != nil) != tt.wantErr {
t.Errorf("run() error = %v, wantErr %v", err, tt.wantErr)
}
deleted, err := deleteSeries(tt.fields.matchName, tt.fields.matchValue)
if err != nil {
t.Fatalf("error delete series: %s", err)
}
if int64(deleted) != tt.numOfSeries {
t.Fatalf("expected deleted series %d; got deleted series %d", tt.numOfSeries, deleted)
}
})
// step month on month time range
start = "2022-09-26T11:23:05+02:00"
end = "2022-11-26T11:24:05+02:00"
numOfSeries = 3
numOfSamples = 2
resultExpected = []vm.TimeSeries{
{
Name: "vm_metric_1",
LabelPairs: []vm.LabelPair{{Name: "job", Value: "0"}},
Timestamps: []int64{1664184185000},
Values: []float64{0},
},
{
Name: "vm_metric_1",
LabelPairs: []vm.LabelPair{{Name: "job", Value: "0"}},
Timestamps: []int64{1666819415000},
Values: []float64{0},
},
{
Name: "vm_metric_1",
LabelPairs: []vm.LabelPair{{Name: "job", Value: "1"}},
Timestamps: []int64{1664184185000},
Values: []float64{100},
},
{
Name: "vm_metric_1",
LabelPairs: []vm.LabelPair{{Name: "job", Value: "1"}},
Timestamps: []int64{1666819415000},
Values: []float64{100},
},
{
Name: "vm_metric_1",
LabelPairs: []vm.LabelPair{{Name: "job", Value: "2"}},
Timestamps: []int64{1664184185000},
Values: []float64{200},
},
{
Name: "vm_metric_1",
LabelPairs: []vm.LabelPair{{Name: "job", Value: "2"}},
Timestamps: []int64{1666819415000},
Values: []float64{200},
},
}
f(start, end, numOfSeries, numOfSamples, resultExpected)
}
func processFlags() {
@@ -311,95 +249,57 @@ func deleteSeries(name, value string) (int, error) {
return vmstorage.DeleteSeries(nil, []*storage.TagFilters{tfs})
}
func Test_buildMatchWithFilter(t *testing.T) {
tests := []struct {
name string
filter string
metricName string
want string
wantErr bool
}{
{
name: "parsed metric with label",
filter: `{__name__="http_request_count_total",cluster="kube1"}`,
metricName: "http_request_count_total",
want: `{cluster="kube1",__name__="http_request_count_total"}`,
wantErr: false,
},
{
name: "metric name with label",
filter: `http_request_count_total{cluster="kube1"}`,
metricName: "http_request_count_total",
want: `{cluster="kube1",__name__="http_request_count_total"}`,
wantErr: false,
},
{
name: "parsed metric with regexp value",
filter: `{__name__="http_request_count_total",cluster=~"kube.*"}`,
metricName: "http_request_count_total",
want: `{cluster=~"kube.*",__name__="http_request_count_total"}`,
wantErr: false,
},
{
name: "only label with regexp",
filter: `{cluster=~".*"}`,
metricName: "http_request_count_total",
want: `{cluster=~".*",__name__="http_request_count_total"}`,
wantErr: false,
},
{
name: "many labels in filter with regexp",
filter: `{cluster=~".*",job!=""}`,
metricName: "http_request_count_total",
want: `{cluster=~".*",job!="",__name__="http_request_count_total"}`,
wantErr: false,
},
{
name: "match with error",
filter: `{cluster~=".*"}`,
metricName: "http_request_count_total",
want: ``,
wantErr: true,
},
{
name: "all names",
filter: `{__name__!=""}`,
metricName: "http_request_count_total",
want: `{__name__="http_request_count_total"}`,
wantErr: false,
},
{
name: "with many underscores labels",
filter: `{__name__!="", __meta__!=""}`,
metricName: "http_request_count_total",
want: `{__meta__!="",__name__="http_request_count_total"}`,
wantErr: false,
},
{
name: "metric name has regexp",
filter: `{__name__=~".*"}`,
metricName: "http_request_count_total",
want: `{__name__="http_request_count_total"}`,
wantErr: false,
},
{
name: "metric name has negative regexp",
filter: `{__name__!~".*"}`,
metricName: "http_request_count_total",
want: `{__name__="http_request_count_total"}`,
wantErr: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := buildMatchWithFilter(tt.filter, tt.metricName)
if (err != nil) != tt.wantErr {
t.Errorf("buildMatchWithFilter() error = %v, wantErr %v", err, tt.wantErr)
return
}
if got != tt.want {
t.Errorf("buildMatchWithFilter() got = %v, want %v", got, tt.want)
}
})
func TestBuildMatchWithFilter_Failure(t *testing.T) {
f := func(filter, metricName string) {
t.Helper()
_, err := buildMatchWithFilter(filter, metricName)
if err == nil {
t.Fatalf("expecting non-nil error")
}
}
// match with error
f(`{cluster~=".*"}`, "http_request_count_total")
}
func TestBuildMatchWithFilter_Success(t *testing.T) {
f := func(filter, metricName, resultExpected string) {
t.Helper()
result, err := buildMatchWithFilter(filter, metricName)
if err != nil {
t.Fatalf("buildMatchWithFilter() error: %s", err)
}
if result != resultExpected {
t.Fatalf("unexpected result\ngot\n%s\nwant\n%s", result, resultExpected)
}
}
// parsed metric with label
f(`{__name__="http_request_count_total",cluster="kube1"}`, "http_request_count_total", `{cluster="kube1",__name__="http_request_count_total"}`)
// metric name with label
f(`http_request_count_total{cluster="kube1"}`, "http_request_count_total", `{cluster="kube1",__name__="http_request_count_total"}`)
// parsed metric with regexp value
f(`{__name__="http_request_count_total",cluster=~"kube.*"}`, "http_request_count_total", `{cluster=~"kube.*",__name__="http_request_count_total"}`)
// only label with regexp
f(`{cluster=~".*"}`, "http_request_count_total", `{cluster=~".*",__name__="http_request_count_total"}`)
// many labels in filter with regexp
f(`{cluster=~".*",job!=""}`, "http_request_count_total", `{cluster=~".*",job!="",__name__="http_request_count_total"}`)
// all names
f(`{__name__!=""}`, "http_request_count_total", `{__name__="http_request_count_total"}`)
// with many underscores labels
f(`{__name__!="", __meta__!=""}`, "http_request_count_total", `{__meta__!="",__name__="http_request_count_total"}`)
// metric name has regexp
f(`{__name__=~".*"}`, "http_request_count_total", `{__name__="http_request_count_total"}`)
// metric name has negative regexp
f(`{__name__!~".*"}`, "http_request_count_total", `{__name__="http_request_count_total"}`)
}

View File

@@ -57,14 +57,13 @@ func CheckStreamAggrConfig() error {
return nil
}
pushNoop := func(_ []prompbmarshal.TimeSeries) {}
opts := streamaggr.Options{
opts := &streamaggr.Options{
DedupInterval: *streamAggrDedupInterval,
DropInputLabels: *streamAggrDropInputLabels,
IgnoreOldSamples: *streamAggrIgnoreOldSamples,
IgnoreFirstIntervals: *streamAggrIgnoreFirstIntervals,
Alias: "global",
}
sas, err := streamaggr.LoadFromFile(*streamAggrConfig, pushNoop, opts)
sas, err := streamaggr.LoadFromFile(*streamAggrConfig, pushNoop, opts, "global")
if err != nil {
return fmt.Errorf("error when loading -streamAggr.config=%q: %w", *streamAggrConfig, err)
}
@@ -77,25 +76,22 @@ func CheckStreamAggrConfig() error {
// MustStopStreamAggr must be called when stream aggr is no longer needed.
func InitStreamAggr() {
saCfgReloaderStopCh = make(chan struct{})
rwctx := "global"
if *streamAggrConfig == "" {
if *streamAggrDedupInterval > 0 {
deduplicator = streamaggr.NewDeduplicator(pushAggregateSeries, *streamAggrDedupInterval, *streamAggrDropInputLabels, rwctx)
deduplicator = streamaggr.NewDeduplicator(pushAggregateSeries, *streamAggrDedupInterval, *streamAggrDropInputLabels, "global")
}
return
}
sighupCh := procutil.NewSighupChan()
opts := streamaggr.Options{
opts := &streamaggr.Options{
DedupInterval: *streamAggrDedupInterval,
DropInputLabels: *streamAggrDropInputLabels,
IgnoreOldSamples: *streamAggrIgnoreOldSamples,
IgnoreFirstIntervals: *streamAggrIgnoreFirstIntervals,
Alias: rwctx,
}
sas, err := streamaggr.LoadFromFile(*streamAggrConfig, pushAggregateSeries, opts)
sas, err := streamaggr.LoadFromFile(*streamAggrConfig, pushAggregateSeries, opts, "global")
if err != nil {
logger.Fatalf("cannot load -streamAggr.config=%q: %s", *streamAggrConfig, err)
}
@@ -123,14 +119,13 @@ func reloadStreamAggrConfig() {
logger.Infof("reloading -streamAggr.config=%q", *streamAggrConfig)
saCfgReloads.Inc()
opts := streamaggr.Options{
opts := &streamaggr.Options{
DedupInterval: *streamAggrDedupInterval,
DropInputLabels: *streamAggrDropInputLabels,
IgnoreOldSamples: *streamAggrIgnoreOldSamples,
IgnoreFirstIntervals: *streamAggrIgnoreFirstIntervals,
Alias: "global",
}
sasNew, err := streamaggr.LoadFromFile(*streamAggrConfig, pushAggregateSeries, opts)
sasNew, err := streamaggr.LoadFromFile(*streamAggrConfig, pushAggregateSeries, opts, "global")
if err != nil {
saCfgSuccess.Set(0)
saCfgReloadErr.Inc()

View File

@@ -64,10 +64,10 @@ var (
"See also -opentsdbHTTPListenAddr.useProxyProtocol")
opentsdbHTTPUseProxyProtocol = flag.Bool("opentsdbHTTPListenAddr.useProxyProtocol", false, "Whether to use proxy protocol for connections accepted "+
"at -opentsdbHTTPListenAddr . See https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt")
configAuthKey = flagutil.NewPassword("configAuthKey", "Authorization key for accessing /config page. It must be passed via authKey query arg. It overrides httpAuth.* settings.")
configAuthKey = flagutil.NewPassword("configAuthKey", "Authorization key for accessing /config page. It must be passed via authKey query arg. It overrides -httpAuth.*")
reloadAuthKey = flagutil.NewPassword("reloadAuthKey", "Auth key for /-/reload http endpoint. It must be passed via authKey query arg. It overrides httpAuth.* settings.")
maxLabelsPerTimeseries = flag.Int("maxLabelsPerTimeseries", 30, "The maximum number of labels accepted per time series. Superfluous labels are dropped. In this case the vm_metrics_with_dropped_labels_total metric at /metrics page is incremented")
maxLabelValueLen = flag.Int("maxLabelValueLen", 1024, "The maximum length of label values in the accepted time series. Longer label values are truncated. In this case the vm_too_long_label_values_total metric at /metrics page is incremented")
maxLabelValueLen = flag.Int("maxLabelValueLen", 4*1024, "The maximum length of label values in the accepted time series. Longer label values are truncated. In this case the vm_too_long_label_values_total metric at /metrics page is incremented")
)
var (
@@ -327,7 +327,7 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
}
return true
case "/prometheus/config", "/config":
if !httpserver.CheckAuthFlag(w, r, configAuthKey.Get(), "configAuthKey") {
if !httpserver.CheckAuthFlag(w, r, configAuthKey) {
return true
}
promscrapeConfigRequests.Inc()
@@ -336,7 +336,7 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
return true
case "/prometheus/api/v1/status/config", "/api/v1/status/config":
// See https://prometheus.io/docs/prometheus/latest/querying/api/#config
if !httpserver.CheckAuthFlag(w, r, configAuthKey.Get(), "configAuthKey") {
if !httpserver.CheckAuthFlag(w, r, configAuthKey) {
return true
}
promscrapeStatusConfigRequests.Inc()
@@ -346,7 +346,7 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
fmt.Fprintf(w, `{"status":"success","data":{"yaml":%q}}`, bb.B)
return true
case "/prometheus/-/reload", "/-/reload":
if !httpserver.CheckAuthFlag(w, r, reloadAuthKey.Get(), "reloadAuthKey") {
if !httpserver.CheckAuthFlag(w, r, reloadAuthKey) {
return true
}
promscrapeConfigReloadRequests.Inc()

View File

@@ -253,7 +253,7 @@ func putHistogram(h *histogram.Fast) {
}
var histogramPool = &sync.Pool{
New: func() interface{} {
New: func() any {
return histogram.NewFast()
},
}

View File

@@ -16,7 +16,7 @@ import (
func FunctionsHandler(w http.ResponseWriter, r *http.Request) error {
grouped := httputils.GetBool(r, "grouped")
group := r.FormValue("group")
result := make(map[string]interface{})
result := make(map[string]any)
for funcName, fi := range funcs {
if group != "" && fi.Group != group {
continue
@@ -47,7 +47,7 @@ func FunctionDetailsHandler(funcName string, w http.ResponseWriter, r *http.Requ
return writeJSON(result, w, r)
}
func writeJSON(result interface{}, w http.ResponseWriter, r *http.Request) error {
func writeJSON(result any, w http.ResponseWriter, r *http.Request) error {
data, err := json.Marshal(result)
if err != nil {
return fmt.Errorf("cannot marshal response to JSON: %w", err)

View File

@@ -1968,10 +1968,10 @@ func (h *minSeriesHeap) Swap(i, j int) {
a := *h
a[i], a[j] = a[j], a[i]
}
func (h *minSeriesHeap) Push(x interface{}) {
func (h *minSeriesHeap) Push(x any) {
*h = append(*h, x.(*seriesWithWeight))
}
func (h *minSeriesHeap) Pop() interface{} {
func (h *minSeriesHeap) Pop() any {
a := *h
x := a[len(a)-1]
*h = a[:len(a)-1]
@@ -2499,10 +2499,10 @@ func (h *maxSeriesHeap) Swap(i, j int) {
a := *h
a[i], a[j] = a[j], a[i]
}
func (h *maxSeriesHeap) Push(x interface{}) {
func (h *maxSeriesHeap) Push(x any) {
*h = append(*h, x.(*seriesWithWeight))
}
func (h *maxSeriesHeap) Pop() interface{} {
func (h *maxSeriesHeap) Pop() any {
a := *h
x := a[len(a)-1]
*h = a[:len(a)-1]

View File

@@ -30,13 +30,13 @@ import (
)
var (
deleteAuthKey = flagutil.NewPassword("deleteAuthKey", "authKey for metrics' deletion via /api/v1/admin/tsdb/delete_series and /tags/delSeries")
deleteAuthKey = flagutil.NewPassword("deleteAuthKey", "authKey for metrics' deletion via /api/v1/admin/tsdb/delete_series and /tags/delSeries. It overrides -httpAuth.*")
maxConcurrentRequests = flag.Int("search.maxConcurrentRequests", getDefaultMaxConcurrentRequests(), "The maximum number of concurrent search requests. "+
"It shouldn't be high, since a single request can saturate all the CPU cores, while many concurrently executed requests may require high amounts of memory. "+
"See also -search.maxQueueDuration and -search.maxMemoryPerQuery")
maxQueueDuration = flag.Duration("search.maxQueueDuration", 10*time.Second, "The maximum time the request waits for execution when -search.maxConcurrentRequests "+
"limit is reached; see also -search.maxQueryDuration")
resetCacheAuthKey = flagutil.NewPassword("search.resetCacheAuthKey", "Optional authKey for resetting rollup cache via /internal/resetRollupResultCache call")
resetCacheAuthKey = flagutil.NewPassword("search.resetCacheAuthKey", "Optional authKey for resetting rollup cache via /internal/resetRollupResultCache call. It overrides -httpAuth.*")
logSlowQueryDuration = flag.Duration("search.logSlowQueryDuration", 5*time.Second, "Log queries with execution time exceeding this value. Zero disables slow query logging. "+
"See also -search.logQueryMemoryUsage")
vmalertProxyURL = flag.String("vmalert.proxyURL", "", "Optional URL for proxying requests to vmalert. For example, if -vmalert.proxyURL=http://vmalert:8880 , then alerting API requests such as /api/v1/rules from Grafana will be proxied to http://vmalert:8880/api/v1/rules")
@@ -172,7 +172,7 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
}
if path == "/internal/resetRollupResultCache" {
if !httpserver.CheckAuthFlag(w, r, resetCacheAuthKey.Get(), "resetCacheAuthKey") {
if !httpserver.CheckAuthFlag(w, r, resetCacheAuthKey) {
return true
}
promql.ResetRollupResultCache()
@@ -369,7 +369,7 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
}
return true
case "/tags/delSeries":
if !httpserver.CheckAuthFlag(w, r, deleteAuthKey.Get(), "deleteAuthKey") {
if !httpserver.CheckAuthFlag(w, r, deleteAuthKey) {
return true
}
graphiteTagsDelSeriesRequests.Inc()
@@ -388,7 +388,7 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
}
return true
case "/api/v1/admin/tsdb/delete_series":
if !httpserver.CheckAuthFlag(w, r, deleteAuthKey.Get(), "deleteAuthKey") {
if !httpserver.CheckAuthFlag(w, r, deleteAuthKey) {
return true
}
deleteRequests.Inc()

View File

@@ -734,11 +734,11 @@ func (sbh *sortBlocksHeap) Swap(i, j int) {
sbs[i], sbs[j] = sbs[j], sbs[i]
}
func (sbh *sortBlocksHeap) Push(x interface{}) {
func (sbh *sortBlocksHeap) Push(x any) {
sbh.sbs = append(sbh.sbs, x.(*sortBlock))
}
func (sbh *sortBlocksHeap) Pop() interface{} {
func (sbh *sortBlocksHeap) Pop() any {
sbs := sbh.sbs
v := sbs[len(sbs)-1]
sbs[len(sbs)-1] = nil
@@ -1084,7 +1084,7 @@ func (xw *exportWork) reset() {
}
var exportWorkPool = &sync.Pool{
New: func() interface{} {
New: func() any {
return &exportWork{}
},
}

View File

@@ -461,7 +461,7 @@ func (xb *exportBlock) reset() {
}
var exportBlockPool = &sync.Pool{
New: func() interface{} {
New: func() any {
return &exportBlock{}
},
}
@@ -1238,7 +1238,7 @@ func (sw *scalableWriter) maybeFlushBuffer(bb *bytesutil.ByteBuffer) error {
}
func (sw *scalableWriter) flush() error {
sw.m.Range(func(_, v interface{}) bool {
sw.m.Range(func(_, v any) bool {
bb := v.(*bytesutil.ByteBuffer)
_, err := sw.bw.Write(bb.B)
return err == nil

View File

@@ -742,13 +742,13 @@ func evalExprsInParallel(qt *querytracer.Tracer, ec *EvalConfig, es []metricsql.
return rvs, nil
}
func evalRollupFuncArgs(qt *querytracer.Tracer, ec *EvalConfig, fe *metricsql.FuncExpr) ([]interface{}, *metricsql.RollupExpr, error) {
func evalRollupFuncArgs(qt *querytracer.Tracer, ec *EvalConfig, fe *metricsql.FuncExpr) ([]any, *metricsql.RollupExpr, error) {
var re *metricsql.RollupExpr
rollupArgIdx := metricsql.GetRollupArgIdx(fe)
if len(fe.Args) <= rollupArgIdx {
return nil, nil, fmt.Errorf("expecting at least %d args to %q; got %d args; expr: %q", rollupArgIdx+1, fe.Name, len(fe.Args), fe.AppendString(nil))
}
args := make([]interface{}, len(fe.Args))
args := make([]any, len(fe.Args))
for i, arg := range fe.Args {
if i == rollupArgIdx {
re = getRollupExprArg(arg)

View File

@@ -103,7 +103,7 @@ func TestGetSumInstantValues(t *testing.T) {
result := getSumInstantValues(nil, cached, start, end, timestamp)
if !reflect.DeepEqual(result, expectedResult) {
t.Errorf("unexpected result; got\n%v\nwant\n%v", result, expectedResult)
t.Fatalf("unexpected result; got\n%v\nwant\n%v", result, expectedResult)
}
}
ts := func(name string, timestamp int64, value float64) *timeseries {

View File

@@ -4461,7 +4461,7 @@ func TestExecSuccess(t *testing.T) {
),0.01)`
r := netstorage.Result{
MetricName: metricNameExpected,
Values: []float64{18.57, 18.57, 18.57, 18.57, 18.57, 18.57},
Values: []float64{30, 30, 30, 30, 30, 30},
Timestamps: timestampsExpected,
}
r.MetricName.Tags = []storage.Tag{{

View File

@@ -956,10 +956,10 @@ func derivValues(values []float64, timestamps []int64) {
values[len(values)-1] = prevDeriv
}
type newRollupFunc func(args []interface{}) (rollupFunc, error)
type newRollupFunc func(args []any) (rollupFunc, error)
func newRollupFuncOneArg(rf rollupFunc) newRollupFunc {
return func(args []interface{}) (rollupFunc, error) {
return func(args []any) (rollupFunc, error) {
if err := expectRollupArgsNum(args, 1); err != nil {
return nil, err
}
@@ -968,7 +968,7 @@ func newRollupFuncOneArg(rf rollupFunc) newRollupFunc {
}
func newRollupFuncTwoArgs(rf rollupFunc) newRollupFunc {
return func(args []interface{}) (rollupFunc, error) {
return func(args []any) (rollupFunc, error) {
if err := expectRollupArgsNum(args, 2); err != nil {
return nil, err
}
@@ -977,7 +977,7 @@ func newRollupFuncTwoArgs(rf rollupFunc) newRollupFunc {
}
func newRollupFuncOneOrTwoArgs(rf rollupFunc) newRollupFunc {
return func(args []interface{}) (rollupFunc, error) {
return func(args []any) (rollupFunc, error) {
if len(args) < 1 || len(args) > 2 {
return nil, fmt.Errorf("unexpected number of args; got %d; want 1...2", len(args))
}
@@ -985,7 +985,7 @@ func newRollupFuncOneOrTwoArgs(rf rollupFunc) newRollupFunc {
}
}
func newRollupHoltWinters(args []interface{}) (rollupFunc, error) {
func newRollupHoltWinters(args []any) (rollupFunc, error) {
if err := expectRollupArgsNum(args, 3); err != nil {
return nil, err
}
@@ -1035,7 +1035,7 @@ func newRollupHoltWinters(args []interface{}) (rollupFunc, error) {
return rf, nil
}
func newRollupPredictLinear(args []interface{}) (rollupFunc, error) {
func newRollupPredictLinear(args []any) (rollupFunc, error) {
if err := expectRollupArgsNum(args, 2); err != nil {
return nil, err
}
@@ -1106,7 +1106,7 @@ func areConstValues(values []float64) bool {
return true
}
func newRollupDurationOverTime(args []interface{}) (rollupFunc, error) {
func newRollupDurationOverTime(args []any) (rollupFunc, error) {
if err := expectRollupArgsNum(args, 2); err != nil {
return nil, err
}
@@ -1136,7 +1136,7 @@ func newRollupDurationOverTime(args []interface{}) (rollupFunc, error) {
return rf, nil
}
func newRollupShareLE(args []interface{}) (rollupFunc, error) {
func newRollupShareLE(args []any) (rollupFunc, error) {
return newRollupAvgFilter(args, countFilterLE)
}
@@ -1150,7 +1150,7 @@ func countFilterLE(values []float64, le float64) float64 {
return float64(n)
}
func newRollupShareGT(args []interface{}) (rollupFunc, error) {
func newRollupShareGT(args []any) (rollupFunc, error) {
return newRollupAvgFilter(args, countFilterGT)
}
@@ -1164,7 +1164,7 @@ func countFilterGT(values []float64, gt float64) float64 {
return float64(n)
}
func newRollupShareEQ(args []interface{}) (rollupFunc, error) {
func newRollupShareEQ(args []any) (rollupFunc, error) {
return newRollupAvgFilter(args, countFilterEQ)
}
@@ -1218,7 +1218,7 @@ func countFilterNE(values []float64, ne float64) float64 {
return float64(n)
}
func newRollupAvgFilter(args []interface{}, f func(values []float64, limit float64) float64) (rollupFunc, error) {
func newRollupAvgFilter(args []any, f func(values []float64, limit float64) float64) (rollupFunc, error) {
rf, err := newRollupFilter(args, f)
if err != nil {
return nil, err
@@ -1229,35 +1229,35 @@ func newRollupAvgFilter(args []interface{}, f func(values []float64, limit float
}, nil
}
func newRollupCountEQ(args []interface{}) (rollupFunc, error) {
func newRollupCountEQ(args []any) (rollupFunc, error) {
return newRollupFilter(args, countFilterEQ)
}
func newRollupCountLE(args []interface{}) (rollupFunc, error) {
func newRollupCountLE(args []any) (rollupFunc, error) {
return newRollupFilter(args, countFilterLE)
}
func newRollupCountGT(args []interface{}) (rollupFunc, error) {
func newRollupCountGT(args []any) (rollupFunc, error) {
return newRollupFilter(args, countFilterGT)
}
func newRollupCountNE(args []interface{}) (rollupFunc, error) {
func newRollupCountNE(args []any) (rollupFunc, error) {
return newRollupFilter(args, countFilterNE)
}
func newRollupSumEQ(args []interface{}) (rollupFunc, error) {
func newRollupSumEQ(args []any) (rollupFunc, error) {
return newRollupFilter(args, sumFilterEQ)
}
func newRollupSumLE(args []interface{}) (rollupFunc, error) {
func newRollupSumLE(args []any) (rollupFunc, error) {
return newRollupFilter(args, sumFilterLE)
}
func newRollupSumGT(args []interface{}) (rollupFunc, error) {
func newRollupSumGT(args []any) (rollupFunc, error) {
return newRollupFilter(args, sumFilterGT)
}
func newRollupFilter(args []interface{}, f func(values []float64, limit float64) float64) (rollupFunc, error) {
func newRollupFilter(args []any, f func(values []float64, limit float64) float64) (rollupFunc, error) {
if err := expectRollupArgsNum(args, 2); err != nil {
return nil, err
}
@@ -1278,7 +1278,7 @@ func newRollupFilter(args []interface{}, f func(values []float64, limit float64)
return rf, nil
}
func newRollupHoeffdingBoundLower(args []interface{}) (rollupFunc, error) {
func newRollupHoeffdingBoundLower(args []any) (rollupFunc, error) {
if err := expectRollupArgsNum(args, 2); err != nil {
return nil, err
}
@@ -1293,7 +1293,7 @@ func newRollupHoeffdingBoundLower(args []interface{}) (rollupFunc, error) {
return rf, nil
}
func newRollupHoeffdingBoundUpper(args []interface{}) (rollupFunc, error) {
func newRollupHoeffdingBoundUpper(args []any) (rollupFunc, error) {
if err := expectRollupArgsNum(args, 2); err != nil {
return nil, err
}
@@ -1338,7 +1338,7 @@ func rollupHoeffdingBoundInternal(rfa *rollupFuncArg, phis []float64) (float64,
return bound, vAvg
}
func newRollupQuantiles(args []interface{}) (rollupFunc, error) {
func newRollupQuantiles(args []any) (rollupFunc, error) {
if len(args) < 3 {
return nil, fmt.Errorf("unexpected number of args: %d; want at least 3 args", len(args))
}
@@ -1405,7 +1405,7 @@ func rollupOutlierIQR(rfa *rollupFuncArg) float64 {
return nan
}
func newRollupQuantile(args []interface{}) (rollupFunc, error) {
func newRollupQuantile(args []any) (rollupFunc, error) {
if err := expectRollupArgsNum(args, 2); err != nil {
return nil, err
}
@@ -1445,7 +1445,7 @@ func mad(values []float64) float64 {
return v
}
func newRollupCountValues(args []interface{}) (rollupFunc, error) {
func newRollupCountValues(args []any) (rollupFunc, error) {
if err := expectRollupArgsNum(args, 2); err != nil {
return nil, err
}
@@ -2389,7 +2389,7 @@ func rollupFake(_ *rollupFuncArg) float64 {
return 0
}
func getScalar(arg interface{}, argNum int) ([]float64, error) {
func getScalar(arg any, argNum int) ([]float64, error) {
ts, ok := arg.([]*timeseries)
if !ok {
return nil, fmt.Errorf(`unexpected type for arg #%d; got %T; want %T`, argNum+1, arg, ts)
@@ -2400,7 +2400,7 @@ func getScalar(arg interface{}, argNum int) ([]float64, error) {
return ts[0].Values, nil
}
func getIntNumber(arg interface{}, argNum int) (int, error) {
func getIntNumber(arg any, argNum int) (int, error) {
v, err := getScalar(arg, argNum)
if err != nil {
return 0, err
@@ -2425,7 +2425,7 @@ func getString(tss []*timeseries, argNum int) (string, error) {
return string(ts.MetricName.MetricGroup), nil
}
func expectRollupArgsNum(args []interface{}, expectedNum int) error {
func expectRollupArgsNum(args []any, expectedNum int) error {
if len(args) == expectedNum {
return nil
}

View File

@@ -200,7 +200,7 @@ func TestDerivValues(t *testing.T) {
testRowsEqual(t, values, timestamps, valuesExpected, timestamps)
}
func testRollupFunc(t *testing.T, funcName string, args []interface{}, vExpected float64) {
func testRollupFunc(t *testing.T, funcName string, args []any, vExpected float64) {
t.Helper()
nrf := getRollupFunc(funcName)
if nrf == nil {
@@ -245,7 +245,7 @@ func TestRollupDurationOverTime(t *testing.T) {
Timestamps: []int64{123},
}}
var me metricsql.MetricExpr
args := []interface{}{&metricsql.RollupExpr{Expr: &me}, maxIntervals}
args := []any{&metricsql.RollupExpr{Expr: &me}, maxIntervals}
testRollupFunc(t, "duration_over_time", args, dExpected)
}
f(-123, 0)
@@ -266,7 +266,7 @@ func TestRollupShareLEOverTime(t *testing.T) {
Timestamps: []int64{123},
}}
var me metricsql.MetricExpr
args := []interface{}{&metricsql.RollupExpr{Expr: &me}, les}
args := []any{&metricsql.RollupExpr{Expr: &me}, les}
testRollupFunc(t, "share_le_over_time", args, vExpected)
}
@@ -289,7 +289,7 @@ func TestRollupShareGTOverTime(t *testing.T) {
Timestamps: []int64{123},
}}
var me metricsql.MetricExpr
args := []interface{}{&metricsql.RollupExpr{Expr: &me}, gts}
args := []any{&metricsql.RollupExpr{Expr: &me}, gts}
testRollupFunc(t, "share_gt_over_time", args, vExpected)
}
@@ -312,7 +312,7 @@ func TestRollupShareEQOverTime(t *testing.T) {
Timestamps: []int64{123},
}}
var me metricsql.MetricExpr
args := []interface{}{&metricsql.RollupExpr{Expr: &me}, eqs}
args := []any{&metricsql.RollupExpr{Expr: &me}, eqs}
testRollupFunc(t, "share_eq_over_time", args, vExpected)
}
@@ -331,7 +331,7 @@ func TestRollupCountLEOverTime(t *testing.T) {
Timestamps: []int64{123},
}}
var me metricsql.MetricExpr
args := []interface{}{&metricsql.RollupExpr{Expr: &me}, les}
args := []any{&metricsql.RollupExpr{Expr: &me}, les}
testRollupFunc(t, "count_le_over_time", args, vExpected)
}
@@ -354,7 +354,7 @@ func TestRollupCountGTOverTime(t *testing.T) {
Timestamps: []int64{123},
}}
var me metricsql.MetricExpr
args := []interface{}{&metricsql.RollupExpr{Expr: &me}, gts}
args := []any{&metricsql.RollupExpr{Expr: &me}, gts}
testRollupFunc(t, "count_gt_over_time", args, vExpected)
}
@@ -377,7 +377,7 @@ func TestRollupCountEQOverTime(t *testing.T) {
Timestamps: []int64{123},
}}
var me metricsql.MetricExpr
args := []interface{}{&metricsql.RollupExpr{Expr: &me}, eqs}
args := []any{&metricsql.RollupExpr{Expr: &me}, eqs}
testRollupFunc(t, "count_eq_over_time", args, vExpected)
}
@@ -396,7 +396,7 @@ func TestRollupCountNEOverTime(t *testing.T) {
Timestamps: []int64{123},
}}
var me metricsql.MetricExpr
args := []interface{}{&metricsql.RollupExpr{Expr: &me}, nes}
args := []any{&metricsql.RollupExpr{Expr: &me}, nes}
testRollupFunc(t, "count_ne_over_time", args, vExpected)
}
@@ -415,7 +415,7 @@ func TestRollupSumLEOverTime(t *testing.T) {
Timestamps: []int64{123},
}}
var me metricsql.MetricExpr
args := []interface{}{&metricsql.RollupExpr{Expr: &me}, les}
args := []any{&metricsql.RollupExpr{Expr: &me}, les}
testRollupFunc(t, "sum_le_over_time", args, vExpected)
}
@@ -438,7 +438,7 @@ func TestRollupSumGTOverTime(t *testing.T) {
Timestamps: []int64{123},
}}
var me metricsql.MetricExpr
args := []interface{}{&metricsql.RollupExpr{Expr: &me}, les}
args := []any{&metricsql.RollupExpr{Expr: &me}, les}
testRollupFunc(t, "sum_gt_over_time", args, vExpected)
}
@@ -461,7 +461,7 @@ func TestRollupSumEQOverTime(t *testing.T) {
Timestamps: []int64{123},
}}
var me metricsql.MetricExpr
args := []interface{}{&metricsql.RollupExpr{Expr: &me}, les}
args := []any{&metricsql.RollupExpr{Expr: &me}, les}
testRollupFunc(t, "sum_eq_over_time", args, vExpected)
}
@@ -484,7 +484,7 @@ func TestRollupQuantileOverTime(t *testing.T) {
Timestamps: []int64{123},
}}
var me metricsql.MetricExpr
args := []interface{}{phis, &metricsql.RollupExpr{Expr: &me}}
args := []any{phis, &metricsql.RollupExpr{Expr: &me}}
testRollupFunc(t, "quantile_over_time", args, vExpected)
}
@@ -506,7 +506,7 @@ func TestRollupPredictLinear(t *testing.T) {
Timestamps: []int64{123},
}}
var me metricsql.MetricExpr
args := []interface{}{&metricsql.RollupExpr{Expr: &me}, secs}
args := []any{&metricsql.RollupExpr{Expr: &me}, secs}
testRollupFunc(t, "predict_linear", args, vExpected)
}
@@ -545,7 +545,7 @@ func TestRollupHoltWinters(t *testing.T) {
Timestamps: []int64{123},
}}
var me metricsql.MetricExpr
args := []interface{}{&metricsql.RollupExpr{Expr: &me}, sfs, tfs}
args := []any{&metricsql.RollupExpr{Expr: &me}, sfs, tfs}
testRollupFunc(t, "holt_winters", args, vExpected)
}
@@ -573,7 +573,7 @@ func TestRollupHoeffdingBoundLower(t *testing.T) {
Timestamps: []int64{123},
}}
var me metricsql.MetricExpr
args := []interface{}{phis, &metricsql.RollupExpr{Expr: &me}}
args := []any{phis, &metricsql.RollupExpr{Expr: &me}}
testRollupFunc(t, "hoeffding_bound_lower", args, vExpected)
}
@@ -594,7 +594,7 @@ func TestRollupHoeffdingBoundUpper(t *testing.T) {
Timestamps: []int64{123},
}}
var me metricsql.MetricExpr
args := []interface{}{phis, &metricsql.RollupExpr{Expr: &me}}
args := []any{phis, &metricsql.RollupExpr{Expr: &me}}
testRollupFunc(t, "hoeffding_bound_upper", args, vExpected)
}
@@ -611,7 +611,7 @@ func TestRollupNewRollupFuncSuccess(t *testing.T) {
f := func(funcName string, vExpected float64) {
t.Helper()
var me metricsql.MetricExpr
args := []interface{}{&metricsql.RollupExpr{Expr: &me}}
args := []any{&metricsql.RollupExpr{Expr: &me}}
testRollupFunc(t, funcName, args, vExpected)
}
@@ -668,7 +668,7 @@ func TestRollupNewRollupFuncError(t *testing.T) {
t.Fatalf("expecting nil func; got %p", nrf)
}
f := func(funcName string, args []interface{}) {
f := func(funcName string, args []any) {
t.Helper()
nrf := getRollupFunc(funcName)
@@ -694,13 +694,13 @@ func TestRollupNewRollupFuncError(t *testing.T) {
Timestamps: []int64{123},
}}
me := &metricsql.MetricExpr{}
f("holt_winters", []interface{}{123, 123, 321})
f("holt_winters", []interface{}{me, 123, 321})
f("holt_winters", []interface{}{me, scalarTs, 321})
f("predict_linear", []interface{}{123, 123})
f("predict_linear", []interface{}{me, 123})
f("quantile_over_time", []interface{}{123, 123})
f("quantiles_over_time", []interface{}{123, 123})
f("holt_winters", []any{123, 123, 321})
f("holt_winters", []any{me, 123, 321})
f("holt_winters", []any{me, scalarTs, 321})
f("predict_linear", []any{123, 123})
f("predict_linear", []any{me, 123})
f("quantile_over_time", []any{123, 123})
f("quantiles_over_time", []any{123, 123})
}
func TestRollupNoWindowNoPoints(t *testing.T) {

Some files were not shown because too many files have changed in this diff Show More