Compare commits

...

187 Commits

Author SHA1 Message Date
Jiekun
27103ea656 debug: [vmagent] add debug log for max_scrape_size 2025-02-11 22:27:54 +08:00
hagen1778
5aeb759df9 docs/CHANGELOG.md: cut v1.103.0
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-28 13:48:31 +02:00
hagen1778
e71cfdcfa5 docs: pre-release doc update
* typo fix
* mention version starting from features are available

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-28 13:47:29 +02:00
hagen1778
9a343b3613 app/{vmselect,vlselect}: run make vmui-update vmui-logs-update
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-28 13:30:38 +02:00
hagen1778
d9982520a8 docs: update dedup docs for cluster
* remove repeating sentences
* clarify why identical dedup config is important

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-28 13:00:30 +02:00
f41gh7
40d55199fd docs/changelog: mention bugfix
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-08-28 11:51:02 +02:00
Nikolay
4ecc370acb lib/storage: properly add previous indexDB metrics (#6890)
Previously, some extIndexDB metrics were not registered. It resulted
into missing metrics, if metric value was added to the extIndexDB. It's
a usual case for search requests at both indexes.

 Current commit updates all metrics from extIndexDB according to the
current IndexDB. It must fix such cases

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6868

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-28 11:14:28 +02:00
Cuong Le
6154bc9466 VictoriaLogs/keyConcepts.md: fix broken anchor to #other-fields (#6896)
### Describe Your Changes

The anchor to "Other fields" section should be #other-fields (instead of
#other-field)

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Cuong Le <cuongleqq@gmail.com>
2024-08-28 01:02:01 -07:00
Zhu Jiekun
52043796c9 docs: Add backup and restore doc for VictoriaLogs
This commit adds doc for VictoriaLogs backup and restore based on `rsync`.
2024-08-27 21:41:42 +02:00
rtm0
9fcfba3927 lib/storage: properly handle maxMetrics limit at metricID search
`TL;DR` This PR improves the metric IDs search in IndexDB:

- Avoid seaching for metric IDs twice when `maxMetrics` limit is
exceeded
- Use correct error type for indicating that the `maxMetrics` limit is
exceded
- Simplify the logic of deciding between per-day and global index search

A unit test has been added to ensure that this refactoring does not
break anything.

---

Function calls before the fix:

```
idb.searchMetricIDs
    |__ is.searchMetricIDs
        |__ is.searchMetricIDsInternal
            |__ is.updateMetricIDsForTagFilters
                |__ is.tryUpdatingMetricIDsForDateRange
                |                       |
                |__ is.getMetricIDsForDateAndFilters
```

- `searchMetricIDsInternal` searches metric IDs for each filter set. It
maintains a metric ID set variable which is updated every time the
`updateMetricIDsForTagFilters` function is called. After each successful
call, the function checks the length of the updated metric ID set and if
it is greater than `maxMetrics`, the function returns `too many
timeseries` error.
- `updateMetricIDsForTagFilters` uses either per-day or global index to
search metric IDs for the given filter set. The decision of which index
to use is made is made within the `tryUpdatingMetricIDsForDateRange`
function and if it returns `fallback to global search` error then the
function uses global index by calling `getMetricIDsForDateAndFilters`
with zero date.
- `tryUpdatingMetricIDsForDateRange` first checks if the given time
range is larger than 40 days and if so returns `fallback to global
search` error. Otherwise it proceeds to searching for metric IDs within
that time range by calling `getMetricIDsForDateAndFilters` for each
date.
- `getMetricIDsForDateAndFilters` searches for metric IDs for the given
date and returns `fallback to global search` error if the number of
found metric IDs is greater than `maxMetrics`.

Problems with this solution:

1. The `fallback to global search` error returned by
`getMetricIDsForDateAndFilters` in case when maxMetrics is exceeded is
misleading.
2. If `tryUpdatingMetricIDsForDateRange` proceeds to date range search
and returns `fallback to global search` error (because
`getMetricIDsForDateAndFilters` returns it) then this will trigger
global search in `updateMetricIDsForTagFilters`. However the global
search uses the same maxMetrics value which means this search is
destined to fail too. I.e. the same search is performed twice and fails
twice.
3. `too many timeseries` error is already handled in
`searchMetricIDsInternal` and therefore handing this error in
`updateMetricIDsForTagFilters` is redundant
4. updateMetricIDsForTagFilters is a better place to make a decision on
whether to use per-day or global index.

Solution:

1.  Use a dedicated error for `too many timeseries` case
2. Handle `too many timeseries` error in  `searchMetricIDsInternal` only
3. Move the per-day or global search decision from
`tryUpdatingMetricIDsForDateRange` to `updateMetricIDsForTagFilters` and
remove `fallback to global search` error.


---------

Signed-off-by: Artem Fetishev <wwctrsrx@gmail.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2024-08-27 21:39:03 +02:00
rtm0
eef6943084 lib/storage: properly register index records with RegisterMetricNames
Once the timeseries is in tsidCache, new entries won't be created in
per-day index because the RegisterMetricNames() code does consider
different dates for the same timeseries. So this case has been added.

The same bug exists for AddRows() but it is not manifested because the
index entries are finally created in updatePerDateData().

RegisterMetricNames also updated to increase the newTimeseriesCreated
counter because it actually creates new time series in index.

A unit tests has been added that check all possible data patterns
(different metric names and dates) and code branches in both
RegisterMetricNames and AddRows. The total number of new unit tests is
around 100 which increaded the running time of storage tests by 50%. 

---------

Signed-off-by: Artem Fetishev <wwctrsrx@gmail.com>
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
2024-08-27 21:33:53 +02:00
rtm0
30f98916f9 Move rowsAddedTotal counter to Storage (#6841)
### Describe Your Changes

Reduced the scope of rowsAddedTotal variable from global to Storage.

This metric clearly belongs to a given Storage object as it counts the
number of records added by a given Storage instance.
Reducing the scope improves the incapsulation and allows to reset this
variable during the unit tests (i.e. every time a new Storage object is
created by a test, that object gets a new variable).



Signed-off-by: Artem Fetishev <wwctrsrx@gmail.com>
2024-08-27 21:30:37 +02:00
hagen1778
6f17ee0d0f docs: typo fix
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-27 15:49:29 +02:00
rtm0
e7f1297517 docs: document IndexDB (#6840)
### Describe Your Changes

This is an attempt to document IndexDB. I guess I was trying to touch
the important points that might be of interest for the end users while
refraining from making it too detailed (such as I did not enumerate and
describe all the specific record types).

Please take a look and any suggestions are very welcome.

### Checklist

The following checks are **mandatory**:

- [x ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Fetishev <wwctrsrx@gmail.com>
2024-08-27 13:52:46 +02:00
Zhu Jiekun
e97e966f82 lib/promrelabel: follow-up for 8958cecad6
In the previous commit 8958cecad6
the default ports (80/443) were removed for both the `scrapeURL` and
`instance` label values for those targets without a port in
`__address__`. Different values in the `instance` label generate new
time series.

This commit reverts the changes made to the `instance` label. Now,
for those targets:
- `scrapeURL` will remain unchanged.
- The `instance` label value will include the default port.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6792
2024-08-27 13:04:26 +02:00
Zhu Jiekun
9c3b5116dd docs: fix example links in getting started with OpenTelemetry (#6880)
### Describe Your Changes

Fix broken link on:
https://docs.victoriametrics.com/guides/getting-started-with-opentelemetry/index.html#building-the-go-application-instrumented-with-metrics
- https://docs.victoriametrics.com/guides/app.go-collector.example
- https://docs.victoriametrics.com/guides/app.go.example

The valid links **could be**:
-
https://docs.victoriametrics.com/guides/getting-started-with-opentelemetry/app.go-collector.example
-
https://docs.victoriametrics.com/guides/getting-started-with-opentelemetry/app.go.example

Also went throught `/guides` folder and did not found doc with same
issue. Please advice if you know more.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-27 10:36:52 +02:00
Fred Navruzov
ac485a8486 docs: vmanomaly - release notes v1.15.[6-9] (#6879)
### Describe Your Changes

release notes for patches 1.15.6 - 9

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-27 10:30:48 +02:00
YuDong Tang
295f2aa8ca app/vmselect:add command-line flag -search.inmemoryBufSizeBytes (#6869)
add command-line flag `-search.inmemoryBufSizeBytes` for configuring size of in-memory buffers used by vmselect during processing of vmstorage responses. A new summary metric `vm_tmp_blocks_inmemory_file_size_bytes` is exposed to show the size of the buffer during requests processing.

The new setting can be used by experienced users to adjust memory usage by vmselect when processing
many small read requests. Instead of allocating 4MB buffers each time, vmselect can be instructed to lower
the buffer size via `-search.inmemoryBufSizeBytes`. To make the decision whether this flag needs to be adjusted
users can consult with `vm_tmp_blocks_inmemory_file_size_bytes` which shows the actual size of buffers used
during query processing.

----------

The detailed information of this PR can be found in
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6851

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit cab3ef8294)
2024-08-26 14:48:53 +02:00
Github Actions
e7b3a996da Automatic update operator docs from VictoriaMetrics/operator@3ba727f (#6875)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-08-26 14:39:12 +02:00
Github Actions
84ca4d18f0 Automatic update operator docs from VictoriaMetrics/operator@369ab8d (#6866)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-08-23 14:09:13 +02:00
Yury Akudovich
d0f5a9d77a app/vmagent: add remoteWrite.retryMinInterval and remoteWrite.retryMaxTime flags (#6289)
## Describe Your Changes

Add RemoteWrite Retry Controls

This PR introduces two new flags to the remote write functionality:
- remoteWrite.retryMinInterval
- remoteWrite.retryMaxTime
 
These flags provide finer control over the retry behavior for
remoteWrite operations, allowing users to customize the minimum interval
between retries and the maximum duration for retry attempts.

Fixes #5486.

## Checklist

- [x] The following checks are mandatory:

My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Yury Akudovich <ya@matterlabs.dev>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-08-23 14:05:51 +02:00
Denys Holius
336406e2e1 docs/guides/guide-delete-or-replace-metrics/README.md: adds a link to the API examples (#6863)
### Describe Your Changes

Adds a link to the API example section that describes how to delete
metrics on VM Single.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-08-23 10:01:03 +02:00
Roman Khavronenko
1b9f3b39b4 deployment/docker: update Go builder from Go1.22.5 to Go1.23.0 (#6861)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-22 23:55:50 +02:00
Roman Khavronenko
70a94ea492 app/vmalert: update parsing for instant responses (#6859)
This change is made in attempt to reduce memory usage by vmalert when
parsing big instant responses from VM/Prometheus.

In
a5c427bac4
vmalert switched from std json lib to fastjson lib in order to reduce
amount of allocations, as according to highloaded profiles of vmalert
the CPU is mostly spent on GC.

But switching to fastjson resulted into excessive memory usage for cases
when vmalert has to parse long json lines, which usually happens when
instant response contains many `metric` objects.

In this change we do a mixed parsing:
1. Slice of `metric` objects is parsed with std lib to keep mem low
2. Each `metric` object is parsed with fastjson to reduce allocs

The benchmark results are the following:
```
pkg: github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/datasource
BenchmarkParsePrometheusResponse/Instant_std+fastjson-10                    1760            668959 ns/op          280147 B/op       5781 allocs/op
MBs allocated at heap: 493.078392
mallocs: 18655472
BenchmarkParsePrometheusResponse/Instant_fastjson-10                        6109            198258 ns/op          172839 B/op       5548 allocs/op
MBs allocated at heap: 1056.384464
mallocs: 34457184
BenchmarkParsePrometheusResponse/Instant_std-10                             1287            950987 ns/op          451677 B/op       9619 allocs/op
MBs allocated at heap: 580.802976
mallocs: 13351636
```
The benchmark function code with mem measurement is available here
https://gist.github.com/hagen1778/b9c3ca7f8ca7d6b21aec9777112c5810

The benchmark contains 3 results:
1. Instant_std+fastjson is the implementation in this change
2. Instant_fastjson-10 is the implementation from
a5c427bac4
3. BenchmarkParsePrometheusResponse/Instant_std-10 is implementation
before
a5c427bac4

According to these results, this new implementation is slower than
previous, but faster than before switching to fastjson. It also has
lower number of allocations and roughly the same memory allocation on
heap with GC turned off.

---------

Other changes:
1. rm BenchmarkMetrics as it doesn't measure anything
2. simplify BenchmarkParsePrometheusResponse into
BenchmarkPromInstantUnmarshal

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-22 17:36:11 +02:00
Yury Molodov
e35237920a vmui: add column search in table settings (#6804)
### Describe Your Changes

Add search functionality to the column display settings in the table
 #6668
 

![image](https://github.com/user-attachments/assets/e9bd52c3-6428-4d4f-8b7f-d83dd80b6912)

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-08-22 16:57:26 +02:00
hagen1778
fa9de9b45c docs: fix ordering in vmagent docs
Looks like auto-numeration doesn't work if item in the list
takes more than one line.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-22 11:58:27 +02:00
hagen1778
dbb4899d71 docs: fix root page in keyConcepts
Add _index.md to fix `http://docs/keyconcepts` link.
Without this change only `http://docs/keyconcepts/keyconcepts` was working.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-22 09:41:30 +02:00
Roman Khavronenko
22975d28ae docs: move keyconcepts to separate dir (#6855)
Moving key-concepts-related docs to a separate dir should make it easier
to navigate in `docs/` folder and helps to avoid adding prefixes to
image assets.

-----------

The change shouldn't have any visual changes or changes to the links.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-21 17:27:24 +02:00
Roman Khavronenko
9dc8d1debd docs: move changelog to dir (#6853)
Moving changelog-related docs to a separate dir should make it easier to
navigate in `docs/` folder.

-----------

The change shouldn't have any visual changes or changes to the links.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-21 17:26:54 +02:00
hagen1778
74e8d6ad55 docs: rm mentions of google group as it was deprecated
See https://docs.victoriametrics.com/#community-and-contributions

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-21 14:08:27 +02:00
Github Actions
a82dda1d18 Automatic update operator docs from VictoriaMetrics/operator@5fd7d6e (#6845)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-08-21 11:27:42 +02:00
Github Actions
37654e0604 Automatic update Grafana datasource docs from VictoriaMetrics/victoriametrics-datasource@c23c4d1 (#6844) 2024-08-21 11:26:44 +02:00
Zakhar Bessarab
654c1cb9d1 make: reset permissions when creating release archive (#6846)
### Describe Your Changes

Forcefully set permissions for release binaries to 1000:1000. This helps
to avoid issues such as
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6788 on
systems with limitations around UID:GID configuration.

"1000" UID and GID is widely used by linux distributions for the first
user in the system.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-08-21 11:25:13 +02:00
Ayush Chauhan
af54ddea23 docs: add zomato case study (#6848) 2024-08-21 09:54:10 +02:00
Dima Lazerka
535a9ed059 vmui: Fix initial serverUrl for vmanomaly (#6834)
- fix TS lint
- anomaly: remove /vmui
- anomaly: minor inspections fix
- docs: fix broken links to headings

### Describe Your Changes

Initially vmanomaly opened with `/vmui` in serverUrl, remove it.
2024-08-20 22:30:38 +03:00
hagen1778
2dba4165e8 docs: move change from VM changelog to VL changelog
The change for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6764
was added to VM changelog by mistake

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-20 17:16:04 +02:00
jackyin
3ebdd3bcb8 vmui: fix not found index.js in VictoriaLogs (#6770)
fix #6764
the index.js file is for [this
feature](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmui#predefined-dashboards),
the feature is just for victoriametrics. so the index.js is deleted in
victorialogs.
i just add an empty index.js to fix it.

---------

Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-08-20 14:50:37 +02:00
Nikolay
9feee15493 lib/promscrape: fixes proxy autorization (#6783)
* Adds custom dial func for HTTP-Connect and socks5 proxy tunnels.
  Standard golang http.transport exposes GetProxyConnectHeader function,
  but it doesn't allow to use separate tls config for proxy.
  It also not possible to enforce HTTP-Connect with standard http lib.
* For http scrape targets, by default http.Transport.Proxy function must
  be used. Since it has special case with full uri forward.
* Adds proxy.URL json methods that allow to properly copy internal
fields, like User/Password.
It should fix bug with proxy_url. When credentials specified at URL was
ignored.
* Adds tests for scrape client proxy requests

related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6771
2024-08-19 22:31:18 +02:00
Zhu Jiekun
723d834c1a lib/promrelabel: stop adding default port 80/433 to address label
*  It was necessary to add default ports for fasthttp client. After migration to the std.httpclient it's no longer needed.
* An additional configuration is required at proxy servers with implicitly set 80/443 ports to the host header (such as HA proxy.

It's expected that after upgrade __address_ label may change. But it should be rare case. 80/443 ports are not widely used at monitoring ecosystem. And it shouldn't have much impact. 

Related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6792

Co-authored-by: Nikolay <nik@victoriametrics.com>
2024-08-19 22:28:49 +02:00
Hui Wang
0f1ec33892 vmalert: add command line flag -notifier.headers (#6751)
to allow configuring additional headers in each request to the
corresponding notifier.
Other flags like `-datasource.headers`, `-remoteWrite.headers` already
use `^^` as delimiter, it's consistent to use it in `-notifier.headers`
as well.


related https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3260
vmalert can integrate with alertmanager that supports multi-tenant by
adding tenantID header`X-Scope-OrgID` in requests.
In multitenancy, vmalert can also filter alerts which send to different
notifier addresses(or with different header settings) using
`alert_relabel_configs`.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3260

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-08-19 21:40:57 +02:00
Github Actions
0ef59bf7b3 Automatic update Grafana datasource docs from VictoriaMetrics/victoriametrics-datasource@de3f649 (#6837) 2024-08-19 21:31:59 +02:00
Hui Wang
0fc1130f47 vmalert-tool: add -external.label and -external.url command-line … (#6766)
…flags to perform the same as vmalert

address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6735

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-08-19 21:29:28 +02:00
hagen1778
febba3971b make go vet happy
Address `non-constant format string in call` check:
https://github.com/golang/go/issues/60529

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-19 21:15:33 +02:00
hagen1778
220b1659b6 Makefile: update golangci-lint from v1.59.1 to v1.60.1
See https://github.com/golangci/golangci-lint/releases/tag/v1.60.1

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-19 17:43:58 +02:00
Andrii Chubatiuk
36bc458e9e docs: reuse hugo image for webp conversion (#6825)
Use same hugo docker image for webp conversion.
While there, remove unused *.png images.
2024-08-19 14:38:56 +02:00
Fred Navruzov
0e464a3a4f docs: vmanomaly - v1.15.5 patch notes (#6835)
### Describe Your Changes

docs: vmanomaly - v1.15.5 patch notes

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-19 11:14:57 +04:00
Daria Karavaieva
7279899a8a docs/vmanomaly: updated model list in Overview (#6832)
### Describe Your Changes

Updated model list in Anomaly Detection Overview

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-17 13:43:28 +03:00
Github Actions
015f0b0424 Automatic update operator docs from VictoriaMetrics/operator@64879fb (#6831)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-08-16 16:32:25 +02:00
Roman Khavronenko
e58dde6925 lib/httputils: parse URL before creating HTTP transport (#6820)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6740

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-16 11:32:04 +02:00
Zhu Jiekun
a0d82a2a83 docs: add more details to -cacheDataPath vmselect flag (#6708)
vmselect will create `./tmp` dir under `cacheDataPath`. If
`cacheDataPath` is set to `/`, vmselect will use `/tmp`.

content under `/tmp` dir might be auto removed based on the OS
behaviour. See:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5770

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-08-15 21:37:03 +02:00
Fred Navruzov
f523a8adb2 docs: vmanomaly - release notes for 1.15.4 (#6813)
### Describe Your Changes

release notes for 1.15.4 patch

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-15 12:47:35 +04:00
Zakhar Bessarab
5390ee2413 app/vmseleсt/promql: fix calculation of histogram buckets
This issue was introduced in 6a4bd5049b

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6714

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-08-15 10:11:41 +02:00
Daria Karavaieva
e702321cdc docs/vmanomaly:change links from relative to absolute (#6809)
### Describe Your Changes

- change links from relative to absolute under Anomaly Detection section

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-14 23:00:57 +02:00
Fred Navruzov
6e7fe4841b docs/vmanomaly - changelog update to v1.15.3 (#6808)
### Describe Your Changes

changelog updates to v1.15.3 patch of `vmanomaly`

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-14 20:34:40 +04:00
Fred Navruzov
9a64184fc7 docs: vmanomaly - reader page update (#6806)
### Describe Your Changes

small update to `data_range` parameter in uppermost config conversion
example

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-14 18:34:54 +03:00
Fred Navruzov
538e779b8b docs: vmanomaly - release v1.15.2 (#6802)
### Describe Your Changes

update vmanomaly docs to forthcoming release v1.15.2

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-14 12:09:57 +03:00
Nikolay
9f42fccfc2 app/vminsert: returns back memory optimisation (#6794)
Production workload shows that it's useful optimisation.

Channel based objects pool allows to handle irregural data ingestion
requests and make memory allocations more smooth.
It's improves sync.Pool efficiency, since objects from sync.Pool removed
after 2 GC cycles. With GOGC=30 value, GC runs significantly more often.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6733

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: f41gh7 <nik@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit f255800da3)
Signed-off-by: hagen1778 <roman@victoriametrics.com>

# Conflicts:
#	app/vminsert/common/insert_ctx_pool.go
2024-08-13 10:56:33 -04:00
ccliu
d134a310f3 vmagent: resolve the issue where usePromCompatibleNaming is not working (#6776)
Describe Your Changes
When I use usePromCompatibleNaming with vmagent to process data that
needs to be formatted from different sources such as InfluxDB, I find
that it doesn’t work

However, it works in vminsert. I found that vminsert uses the
HasRelabeling method to determine whether to relabel.
```go
func HasRelabeling() bool {
	pcs := pcsGlobal.Load()
	return pcs.Len() > 0 || *usePromCompatibleNaming
}
```
in vmagent, the decision to relabel is determined only by
pcsGlobal.Len() > 0. However, in the applyRelabeling method, the
usePromCompatibleNaming logic is also used to determine whether to
relabel in the error handling.
```go
func (rctx *relabelCtx) applyRelabeling(tss []prompbmarshal.TimeSeries, pcs *promrelabel.ParsedConfigs) []prompbmarshal.TimeSeries {
	if pcs.Len() == 0 && !*usePromCompatibleNaming {
		// Nothing to change.
		return tss
	}
```
So I think that the logic for determining whether to relabel in vmagent
is not as expected.

Checklist
The following checks are mandatory:

[]My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
2024-08-13 10:32:05 -04:00
jackyin
5f5bc46b3e vlogs: add select/deselect all button to table settings in UI (#6680)
fix #6668, just add **select all** and "unselect all" func.


https://github.com/user-attachments/assets/0c31385b-def0-4618-aa9c-5ba4bb6f56c3

---------

Co-authored-by: Yury Molodov <yurymolodov@gmail.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-08-13 10:20:07 -04:00
Hui Wang
62d19369a3 stream aggregation: do not allow to enable -stream.keepInput and `k… (#6723)
…eep_metric_names` options in stream aggregation config together

With aggregated data and raw data under the same metric, results would
be confusing.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-08-13 08:54:35 -04:00
hagen1778
331573b0bb docs: mention https://blog.zomato.com/migrating-to-victoriametrics-a-complete-overhaul-for-enhanced-observability
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-13 08:26:20 -04:00
Fred Navruzov
a99fcfbf1a docs/vmanomaly - typos fix & clarity (#6793)
### Describe Your Changes

typos fix & clarity improvement of vmanomaly docs after v1.15.1 release

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-10 17:43:05 +03:00
Fred Navruzov
985e4f0b99 docs: vmanomaly - release v1.15.1 (#6782)
### Describe Your Changes

 vmanomaly - release v1.15.1 updates to docs:
- changelog page
- reader page (new arguments docs)
- typos & fixes

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-10 16:54:27 +03:00
Zhu Jiekun
9e2bd82376 app/vmagent: fixes azure service discovery pagination
Azure API response with link to the next page was incorrectly validate. Validation used url.Host header to match configure API URL.


https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6784
2024-08-09 15:22:47 +02:00
hagen1778
1b788fab51 docs: mention deduplication change in update notes for 1.100.0
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-08 09:28:31 +02:00
Dmytro Kozlov
075f5145c2 docs: update user management guide for cloud (#6738)
### Describe Your Changes

Updated user management guide with new cloud content
This PR should be merged after the cloud PR

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-08 09:16:36 +02:00
Zakhar Bessarab
cb00b4b00f lib/backup/s3remote: add retryer configuration (#6747)
### Describe Your Changes

This helps to improve reliability of performing backups in environments
with unreliable connection and tolerate temporary errors at S3 provider
side.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6732

Default retry timeout is up to 3 minutes to make this consistent with
the same configuration for GCS:
a05317f61f/lib/backup/gcsremote/gcs.go (L70-L76)

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-08-07 16:55:29 +02:00
Roman Khavronenko
f28f496a9d lib/bytesutil: smooth buffer growth rate (#6761)
Before, buffer growth was always x2 of its size, which could lead to
excessive memory usage when processing big amount of data.
For example, scraping a target with hundreds of MBs in response could
result into hih memory spikes in vmagent because buffer has to double
its size to fit the response. See
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6759

The change smoothes out the growth rate, trading higher allocation rate
for lower mem usage at certain conditions.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-07 16:49:43 +02:00
Andrii Chubatiuk
35d77a3bed docs: updated guides structure, removed deprecated sort option (#6767)
### Describe Your Changes

* `sort` param is unused by the current website engine, and was present only for compatibility
with previous website engine. It is time to remove it as it makes no effect
* re-structure guides content into folders to simplify assets management

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-07 16:48:08 +02:00
hagen1778
1154f90d2d lib/mergeset: fix typos in comments
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-07 15:54:15 +02:00
Fred Navruzov
264e9caf5a docs/vmanomaly: fix typos after v1.15.0 (#6774)
### Describe Your Changes

Fixing remaining typos and missing words after v.1.15.0 updates

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-07 15:15:11 +03:00
Fred Navruzov
b670fa1a29 docs/vmanomaly: release v1.15.0 (#6768)
### Describe Your Changes

- updated docs on `vmanomaly` with v1.15.0
- additional chapters of FAQ and model pages

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-07 14:31:07 +03:00
Anton L
8acb671fdc app/vmselect/graphite: respect denyPartialResponse for graphite requests (#6748)
VM has different responses to equivalent queries for MetricsQL and
GraphiteQL in case of failed access to one of vmstorage node of the
cluster vmstorage nodes. For GraphiteQL, the denyPartialResponse feature
is not used, it is always true, which is not always correct (depending
on the configuration).

In the PR I have removed the hardcoded denyPartialResponse for
GraphiteQL, just like MetricsQL does.

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 79008b712f)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-07 12:37:41 +02:00
Aliaksandr Valialkin
04981c7a7f lib/streamaggr: remove resetState arg from aggrState.flushState()
The resetState arg was used only for the BenchmarkAggregatorsFlushInternalSerial benchmark.
This benchmark was testing aggregate state flush performance by keeping the same state across flushes.
The benhmark didn't reflect the performance and scalability of stream aggregation in production,
while it led to non-trivial code changes related to resetState arg handling.

So let's drop the benchmark together with all the code related to resetState handling,
in order to simplify the code at lib/streamaggr a bit.

Thanks to @AndrewChubatiuk for the original idea at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6314
2024-08-07 11:39:14 +02:00
Aliaksandr Valialkin
86c7afd126 lib/streamaggr: consistently use the same timestamp across all the output aggregated samples in a single aggregation interval
Prevsiously every aggregation output was using its own timestamp for the output aggregated samples
in a single aggregation interval. This could result in unexpected inconsitent timesetamps for the output
aggregated samples.

This commit consistently uses the same timestamp across all the output aggregated samples.
This commit makes sure that the duration between subsequent timestamps strictly equals
the configured aggregation interval.

Thanks to @AndrewChubatiuk for the original idea at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6314
This commit should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4580
2024-08-07 11:39:13 +02:00
Hui Wang
8f5c26d788 app/vmagent/remotewrite: make -remoteWrite.streamAggr.ignoreFirstIntervals of array type (#6744)
Make `-remoteWrite.streamAggr.ignoreFirstIntervals` of array type so it could
 accept multiple values which can be applied to the corresponding`-remoteWrite.url`.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-08-07 09:53:50 +02:00
Hui Wang
4863605469 app/vmagent/remotewrite: fix -streamAggr.dropInputLabels behavior (#6743)
Fix `-streamAggr.dropInputLabels` behavior  when global deduplication is enabled without `-streamAggr.config`.
Previously, `-remoteWrite.streamAggr.dropInputLabels` is misapplied.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-08-07 09:48:15 +02:00
hagen1778
3a88553315 docs: mention __sample_limit__ in sd_configs
Follow-up after 994796367b

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-07 09:42:46 +02:00
Anzor
994796367b app/vmagent: read __sample_limit__ from labels (#6665) (#6666)
By introducing this feature, users will have the ability to customize
the sampleLimit parameter on a per-target basis, providing more
flexibility and control over the job execution behavior.
2024-08-07 09:36:14 +02:00
hagen1778
9726e6c1a2 app/vmalert: rm unnecessary err check
The error check was needed before a84491324d
It was kept by mistake and makes no sense to have rn.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-07 09:09:24 +02:00
Andrii Chubatiuk
86b473c476 docs: fixed docs build (#6765)
### Describe Your Changes

fixed yaml header in a guide doc, that causes hugo build error

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-06 16:32:12 +02:00
Yury Molodov
04c2232e45 vmui/logs: add display top streams in the hits graph (#6647)
### Describe Your Changes

- Adds support for displaying the top 5 log streams in the hits graph,
grouping the remaining streams into an "other" label.
   #6545

- Adds options to customize the graph display with bar, line, stepped
line, and points views.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-06 16:28:44 +02:00
Andrii Chubatiuk
2e16732fdb docs: updated docs titles and links (#6741)
The changes are based on SEO report and supposed to improve
ranking and indexation by search engines by using prompt and unique titles
and by updating unreachable links.

It also updates links to have a simplified form and replaces relative links with absolute links
according to https://docs.victoriametrics.com/#documentation

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-08-06 15:54:52 +02:00
Zakhar Bessarab
58b6c54da2 app/vlinsert/elasticsearch: add fake response for logstash requests (#6742)
### Describe Your Changes

This is needed in order to support standard Elasticsearch output in
Logstash pipelines.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6660

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-08-06 15:43:33 +02:00
Tommy
f23650ccf9 docs: fix typo in VictoriaLogs FAQ.md (#6755)
Update VictoriaLogs FQA section to replace VictoriaMetrics with
VictoriaLogs.

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-06 15:41:04 +02:00
Hui Wang
c1b54779a2 vmalert: respect HTTP headers defined in notifier configuration file (#6762)
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-08-06 15:37:25 +02:00
hagen1778
be0b892ce6 docs: data ingestion
* rm extra epmty lines
* rename images in according to https://docs.victoriametrics.com/#images-in-documentation

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-06 15:34:07 +02:00
Mathias Palmersheim
a46d554f74 docs: add vmagent and alloy data ingestion docs (#6678)
Adds Prometheus Grafana Alloy and vmagent to the data ingestion
protocols. Grafana Agent was not added since it has been deprecated in
favor of alloy

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-08-06 15:29:42 +02:00
hagen1778
f283126084 fix typos in comments
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-06 14:54:49 +02:00
Zhu Jiekun
7551d799f7 docs: Added grafana version requirement for vm+vl datasorce (#6760)
### Describe Your Changes

https://victoriametrics.slack.com/archives/C05UNTPAEDN/p1722833182319299

Sometimes users may use the wrong (lower) version of Grafana when
setting up the VictoriaLogs datasource.

It would be good to document the requirements of Grafana versions to use
VictoriaMetrics/VictoriaLogs datasource.

### Checklist

The following checks are **mandatory**:

- [X] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-05 16:19:17 +02:00
Ivan Yatskevich
3c48e5662f docs: fix typo in quick start guide (#6757)
### Describe Your Changes

Fix a typo in docs.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-05 10:36:16 +02:00
Zakhar Bessarab
9877a5e7d5 app/{vminsert,vmagent}: add healthcheck for influx ingestion endpoints (#6749)
### Describe Your Changes

This is useful for clients which validate InfluxDB is available before
data ingestion can be started.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6653

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-08-05 09:34:54 +02:00
Dmytro Kozlov
6f401daacb vmctl: add --backoff-retries, --backoff-factor, --backoff-min-duration global command-line flags (#6639)
### Describe Your Changes

Added `--vm-backoff-retries`, `--vm-backoff-factor`,
`--vm-backoff-min-duration` and `--vm-native-backoff-retries`,
`--vm-native-backoff-factor`, `--vm-native-backoff-min-duration`
command-line flags to the `vmctl` app. Those changes will help to
configure the retry backoff policy for different situations.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6622
### Checklist

The following checks are **mandatory**:

- [X] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-08-03 19:12:48 +02:00
Yury Molodov
e06a19d85f vmui/logs: improve UI functionality (#6688)
* add a toggle button to the "Group" tab that allows users to expand or collapse all groups at once
* introduce the ability to select a key for grouping logs within the "Group" tab
* display the number of entries within each log group.
* move the Markdown toggle to the general settings panel in the upper left corner.
2024-08-02 15:48:36 +02:00
f41gh7
3edbebd3ed docs: mention v1.102.x LTS release line
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-08-02 13:54:02 +02:00
hagen1778
f9d6bb91b8 docs: rm reference to step number as it could change in future
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-02 11:57:10 +02:00
f41gh7
b85e3f8bbd deployment: update compose images to v1.102.1 release
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-08-02 11:29:12 +02:00
f41gh7
26fa8106c8 add new LTS release v1.102.x
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-08-02 11:12:20 +02:00
hagen1778
93e54d65d1 docs: mention step for re-building js static files
Changes to .ts files in vmui or vmui for logs require re-building
static files that will be included into compiled binary afterwards.

We don't update static files on each .ts change PR because it results
in too many changes and complicates review.

So we need to update these static files before the actual release.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-02 10:56:04 +02:00
hagen1778
7bf83c7c0e docs: use absolute links instead of relatives
See https://docs.victoriametrics.com/#documentation

```
Use absolute links. This simplifies moving docs between different files.
```

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-02 10:25:37 +02:00
Yury Molodov
a05317f61f vmui/logs: add fields for tenant configuration (#6661)
Added fields for configuring AccountID and ProjectID
#6631
2024-08-02 09:57:39 +02:00
f41gh7
996b623585 make vmui-update 2024-08-01 14:45:09 +02:00
f41gh7
2222d62aa6 docs/CHANGELOG.md: cut v1.102.1 release
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-08-01 14:23:19 +02:00
f41gh7
7b21471f78 vendor: updates metricsql to v0.77.0 with bugfix
Fixes panic if incorrect metricsql expression passed to the prettifier API.

Prettify function had misleading panic for duration expression formatting. It expected all WITH templates to be already parsed.
But WITH expression expand was removed.

Bug was introduced at e712a49898
and present at v1.98.0+ releases

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6736
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-08-01 10:50:05 +02:00
hagen1778
77b768089f docs: follow-up after 58e667c895
* add Logo guidelines to GitHub Readme, as it may have higher chances to be viewed by users
* rm Logo guidelines from cluster version docs, as it makes no sense anymore for this page
* rm `picutre` tag from cluster version docs, as it is not used by GitHub anymore

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-07-31 16:20:17 +02:00
Andrii Chubatiuk
58e667c895 docs: grouped changelog docs, removed old make commands, replaced docs in root README with official docs links (#6727)
### Describe Your Changes

- replace docs in root README with a link to official documentation
- remove old make commands for documentation
- remove redundant "VictoriaMetrics" from document titles
- merge changelog docs into a section
- rm content of Single-server-VictoriaMetrics.md as it can be included from docs/README
- add basic information to README in the root folder, so it will be useful for github users
- rm `picture` tag from docs/README as it was needed for github only, we don't display VM logo at docs.victoriametrics.com
- update `## documentation` section in docs/README to reflect the changes
- rename DD pictures, as they now belong to docs/README

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-07-31 16:03:18 +02:00
Yury Molodov
53919327b2 vmui: fix auto-completion triggers (#6566)
### Describe Your Changes

- Fixes auto-complete triggers according to [these
comments](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5866#issuecomment-2065273421).
- Fixes loading and displaying suggestions when there is no metric in
the expression.
   Related issue: #6153
- Adds quotes when inserting label values. 
   Related issue: #6260

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-31 15:00:14 +02:00
hagen1778
b8b6c565f4 docs: rm typo in vmalert docs
typo was added in b9f7c3169a

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-07-30 16:58:08 +02:00
hagen1778
d225a2eb56 dashboards: add Scrape duration 0.99 quantile panel
The new panel will show the 99th quantile of scrape duration in seconds.
This should help identifying vmagent instances that experiences too high scraping durations.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-07-30 14:57:17 +02:00
Dmytro Kozlov
d09182da11 docs: rename managed to cloud (#6689)
### Describe Your Changes

These changes were made as part of the title transition from Managed
VictoriaMetrics to VictoriaMetrics Cloud

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-30 09:59:29 +02:00
Andrii Chubatiuk
88ea9c2fb3 docs: removed redundant 'VictoriaLogs' from title (#6715)
### Describe Your Changes

After breadcrumb was added to docs.victoriametrics.com there's no need
to specify parent page name in a title

<img width="1437" alt="Screenshot 2024-07-27 at 10 20 09"
src="https://github.com/user-attachments/assets/733f41f4-a727-4f52-a7c0-6019edf1b803">

Also added vmdocs to gitignore to avoid committing it

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-29 14:29:15 +02:00
Nikolay
135a9fdf1d docs: mention graphite incompatibilities (#6664)
updates
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5810
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2969

---------

Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-07-29 14:24:37 +02:00
Juraj Bubniak
11c0b05e8a lib/backup/s3remote: fix typos (#6694)
Fixes a few typos in errors in lib/backup/s3remote package.
2024-07-29 14:18:31 +02:00
jackyin
e5d279bb71 lib/netutil: validate TLS cert and key files immediately (#6621)
Validate files specified via `-tlsKeyFile` and `-tlsCertFile` cmd-line flags on the process start-up. Previously, validation happened on the first connection accepted by HTTP server.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6608

---------

Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-07-29 13:58:53 +02:00
Fred Navruzov
9cbf844903 docs/vmanomaly - changelog upd v1.14.1-v1.14.2 (#6718)
### Describe Your Changes

- Doc updates on v1.14.1 - v1.14.2 of `vmanomaly`
-
[Changelog](https://docs.victoriametrics.com/anomaly-detection/changelog/)
page

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-27 23:03:16 +03:00
Fred Navruzov
0cd19edce1 docs/vmanomaly-v1.14.1-2-updates (#6717)
### Describe Your Changes

- Doc updates on v1.14.1 - v1.14.2 of `vmanomaly`
-
[Changelog](https://docs.victoriametrics.com/anomaly-detection/changelog/)
page
-
[Reader](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader)
page (`queries` arg refactor)
- Also, a slight modification of `presets`
[page](https://docs.victoriametrics.com/anomaly-detection/presets/)

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-27 22:55:30 +03:00
Aliaksandr Valialkin
1e3c7f0ef5 vendor: run make vendor-update 2024-07-27 13:52:48 +02:00
Aliaksandr Valialkin
9dde5b8ee3 app/{vmselect,vlselect}: run make vmui-update vmui-logs-update after efd70b2c52 2024-07-27 13:50:31 +02:00
Aliaksandr Valialkin
83f2ce4910 app/vmauth: verify how backend response headers are propagated to vmauth client 2024-07-27 13:44:49 +02:00
Github Actions
d53dd6ecf3 Automatic update Grafana datasource docs from VictoriaMetrics/victorialogs-datasource@c8ee327 (#6685) 2024-07-25 18:34:22 +04:00
Aliaksandr Valialkin
8551fbe9f3 Revert "refactor(vmstorage): Refactor the code to reduce the time complexity of MustAddRows and improve readability (#6629)"
This reverts commit e280d90e9a.

Reason for revert: the updated code doesn't improve the performance of table.MustAddRows for the typical case
when rows contain timestamps belonging to ptws[0].

The performance may be improved in theory for the case when all the rows belong to partiton other than ptws[0],
but this partition is automatically moved to ptws[0] by the code at lines
6aad1d43e9/lib/storage/table.go (L287-L298) ,
so the next time the typical case will work.

Also the updated code makes the code harder to follow, since it introduces an additional level of indirection
with non-trivial semantics inside table.MustAddRows - the partition.TimeRangeInPartition() function.
This function needs to be inspected and understood when reading the code at table.MustAddRows().
This function depends on minTsInRows and maxTsInRows vars, which are defined and initialized
many lines above the partition.TimeRangeInPartition() call. This complicates reading and understanding
the code even more.

The previous code was using clearer loop over rows with the clear call to partition.HasTimestamp()
for every timestamp in the row. The partition.HasTimestamp() call is used in the table.MustAddRows()
function multiple times. This makes the use of partition.HasTimestamp() call more consistent,
easier to understand and easier to maintain comparing to the mix of partition.HasTimestamp() and partition.TimeRangeInPartition()
calls.

Aslo, there is no need in documenting some hardcore software engineering refactoring at docs/CHANGLELOG.md,
since the docs/CHANGELOG.md is intended for VictoriaMetrics users, who may not know software engineering.
The docs/CHANGELOG.md must document user-visible changes, and the docs must be concise and clear for VictoriaMetrics users.
See https://docs.victoriametrics.com/contributing/#pull-request-checklist for more details.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6629
2024-07-25 14:32:09 +02:00
Aliaksandr Valialkin
6aad1d43e9 docs/anomaly-detection/components/models.md: remove {width="800px"} in the same was as it was done in 4735d8803e 2024-07-25 13:39:47 +02:00
Aliaksandr Valialkin
936b9f551a docs/anomaly-detection: return back absolute links instead of relative links after ce4cc4cbb2
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6677

Relative links in docs are much harder to maintain in consistent state comparing to absolute links:

- It is non-trivial to figure out the proper relative link path when creating and editing docs.
- Relative links break after moving the doc files to another paths, and it is non-trivial
  to figure which links are broken after that.

See also f357ee57ef , d5809f8e12 and 8cb1822b94
2024-07-25 13:34:54 +02:00
Aliaksandr Valialkin
423e35c7b7 docs/Makefile: simplify and document make docs-debug after ce4cc4cbb2
- Document `make docs-debug` command at https://docs.victoriametrics.com/#documentation

- Remove unneeded ROOTDIR, REPODIR and WORKDIR env vars from docs/Makefile ,
  since it is documented and expected that all the Makefile commands are run from the repository root.

- Use `docker --rm` for running Docker container with local docs server, so it is automatically
  removed after pressing `Ctrl+C`. This makes the container cleanup automatic.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6677
2024-07-25 13:12:12 +02:00
Aliaksandr Valialkin
8cb1822b94 Revert "removed unneeded ref shortcodes, updated VM changelog to use relative markdown links (#6691)"
This reverts commit 2e9b1efeb9.

Reason for revert: relative links in docs are much harder to maintain in consistent state
comparing to absolute links:

- It is non-trivial to figure out the proper relative link path when creating and editing docs.
- Relative links break after moving the doc files to another paths, and it is non-trivial
  to figure which links are broken after that.
- The updated relative links do not work properly right now in the docs.
  For example, the https://docs.victoriametrics.com/victorialogs/quickstart.md#building-from-source-code
  link at https://docs.victoriametrics.com/victorialogs/changelog/ leads to 404 page.

This is documented at https://docs.victoriametrics.com/#images-in-documentation .
2024-07-25 12:48:13 +02:00
Aliaksandr Valialkin
d5809f8e12 Revert "replaced global http refs with relative markdown ones (#6692)"
This reverts commit 537266363a.

Reason for revert: relative links in docs are much harder to maintain in consistent state
comparing to absolute links:

- It is non-trivial to figure out the proper relative link path when creating and editing docs.
- Relative links break after moving the doc files to another paths, and it is non-trivial
  to figure which links are broken after that.
- The updated relative links do not work properly right now in the docs.
  For example, the https://docs.victoriametrics.com/victorialogs/quickstart.md#building-from-source-code
  link at https://docs.victoriametrics.com/victorialogs/changelog/ leads to 404 page.

This is documented at https://docs.victoriametrics.com/#images-in-documentation .
2024-07-25 12:48:12 +02:00
Aliaksandr Valialkin
f357ee57ef Revert "fixed victorialogs relative links (#6693)"
This reverts commit 683a69d53d.

Reason for revert: relative links in docs are much harder to maintain in consistent state
comparing to absolute links:

- It is non-trivial to figure out the proper relative link path when creating and editing docs.
- Relative links break after moving the doc files to another paths, and it is non-trivial
  to figure which links are broken after that.
- The updated relative links do not work properly right now in the docs.
  For example, the https://docs.victoriametrics.com/victorialogs/quickstart.md#building-from-source-code
  link at https://docs.victoriametrics.com/victorialogs/changelog/ leads to 404 page.

This is documented at https://docs.victoriametrics.com/#images-in-documentation .
2024-07-25 12:48:12 +02:00
Github Actions
c9b913d201 Automatic update Grafana datasource docs from VictoriaMetrics/victoriametrics-datasource@d25ea09 (#6698) 2024-07-25 02:00:44 -07:00
Github Actions
f05e17658a Automatic update Grafana datasource docs from VictoriaMetrics/victorialogs-datasource@d6bfe49 (#6697) 2024-07-25 02:00:34 -07:00
Aliaksandr Valialkin
08071326cc docs/vmauth.md: typo fix 2024-07-25 10:18:16 +02:00
Ruixiang Tan
e280d90e9a refactor(vmstorage): Refactor the code to reduce the time complexity of MustAddRows and improve readability (#6629)
### Describe Your Changes
The original logic is not only highly complex but also poorly readable,
so it can be modified to increase readability and reduce time
complexity.


---------

Co-authored-by: Zhu Jiekun <jiekun@victoriametrics.com>
2024-07-25 08:55:12 +02:00
Artem Navoiev
0593310e89 docs: fix link in logsql doc
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-07-24 20:31:12 +02:00
Andrii Chubatiuk
683a69d53d fixed victorialogs relative links (#6693)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-24 11:26:02 -07:00
Artem Navoiev
386e48eb23 docs: guides remove unneeded menu item
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-07-24 20:22:59 +02:00
Artem Navoiev
147e703d64 docs: anomaly detection remove html tags
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-07-24 20:12:19 +02:00
Artem Navoiev
4735d8803e docs: remove img width
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-07-24 20:09:03 +02:00
Artem Navoiev
ef0e6af6a8 docs: vmanomaly use absolute links in code comments
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-07-24 19:58:16 +02:00
Andrii Chubatiuk
537266363a replaced global http refs with relative markdown ones (#6692)
### Describe Your Changes

Replaced global http links in docs with relative markdown ones

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-24 07:50:09 -07:00
Andrii Chubatiuk
2e9b1efeb9 removed unneeded ref shortcodes, updated VM changelog to use relative markdown links (#6691)
### Describe Your Changes

Use relative markdown references, removed `{{< ref >}}` shortcodes

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-24 13:33:37 +03:00
Andrii Chubatiuk
ce4cc4cbb2 view documentation locally (#6677)
### Describe Your Changes

- moved files from root to VictoriaMetrics folder to be able to mount
operator docs and VictoriaMetrics docs independently
- added ability to run website locally

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-24 01:00:31 -07:00
Andrii Chubatiuk
332b7788fb added _index.md for hugo, which point to README.md (#6686)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-23 17:30:49 +03:00
Hui Wang
b515a7b69b security: upgrade base docker image (Alpine) from 3.20.1 to 3.20.2 (#6684)
See https://www.alpinelinux.org/posts/Alpine-3.20.1-released.html

>including security fix for:
OpenSSL CVE-2024-5535
2024-07-23 13:20:06 +02:00
Zakhar Bessarab
d88d0f382b app/vmauth: change response code when all backend are not available (#6676)
### Describe Your Changes

Change response code to 502 to align it with behaviour of other existing
reverse proxies. Currently, the following reverse proxies will return
502 in case an upstream is not available: nginx, traefik, caddy, apache.


Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-07-22 17:31:18 +02:00
Zhu Jiekun
d297ac0e8f docs: [vmagent] Add docs for remote write protocol control in Kafka integration (#6682)
### Describe Your Changes

Improve documentation to help:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6670

Currently, the documentation of Kafka Integration did not describe:
- How to switch between different remote write protocols in producer
side and consumer side.
2024-07-22 16:51:45 +02:00
Zakhar Bessarab
cb0c4e6cfb docs/vmbackup: add a note for azure identity (#6675)
### Describe Your Changes

Add a note to clarify usage of azure credentials when multiple
credentials are available.

The user is required to specify AZURE_CLIENT_ID as otherwise Azure API
will return an error: "Multiple user assigned identities exist, please
specify the clientId / resourceId of the identity in the token request"

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-07-22 15:38:09 +02:00
Artem Navoiev
29e0850495 docs: fix broken links
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-07-22 10:38:34 +02:00
Aliaksandr Valialkin
06f510899c docs/CHANGELOG.md: document keep_original_host option at vmauth
This option has been added in the commit add2db12b2
2024-07-20 11:48:40 +02:00
Aliaksandr Valialkin
dad3eefd74 app/vmauth: test how User-Agent header is set in requests to backend 2024-07-20 11:43:24 +02:00
Aliaksandr Valialkin
e87b4d3768 app/vmauth: verify the correctness of X-Forwarded-For header processing at TestRequestHandler() 2024-07-20 11:28:14 +02:00
Aliaksandr Valialkin
cb76ff5c56 app/vmauth: add missing tests for requestHandler() 2024-07-20 11:22:36 +02:00
Aliaksandr Valialkin
78b1571eb8 app/vmauth: add more tests for requestHandler() 2024-07-20 10:19:45 +02:00
Aliaksandr Valialkin
0a8c9c5ee7 docs/vmauth.md: document the case with default url_prefix additionally to url_map 2024-07-20 09:46:01 +02:00
Aliaksandr Valialkin
9e0c37be2d app/vmauth: properly proxy requests to backend paths ending with /
Previously the traling / was incorrectly removed when proxying requests from http://vmauth/

While at it, add more tests for requestHandler()
2024-07-19 17:29:04 +02:00
Aliaksandr Valialkin
add2db12b2 app/vmauth: properly proxy HTTP requests without body
The Request.Body for requests without body can be nil. This could break readTrackingBody.Read() logic,
which could incorrectly return "cannot read data after closing the reader" error in this case.
Fix this by initializing the readTrackingBody.r with zeroReader.

While at it, properly set Host header if it is specified in 'headers' section.
It must be set net/http.Request.Host instead of net/http.Request.Header.Set(),
since the net/http.Client overwrites the Host header with the value from req.Host
before sending the request.

While at it, add tests for requestHandler(). Additional tests for various requestHandler() cases
will be added in future commits.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6445
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5707
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5240
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6525
2024-07-19 16:24:12 +02:00
Aliaksandr Valialkin
b2e646280e docs/goals.md: clarify that complicated CI/CD tasks make debugging harder 2024-07-19 08:50:58 +02:00
Aliaksandr Valialkin
6c8601681a docs/vmauth.md: use /some_path/.* instead of /some_path/.+ in examples, so it matches /some_path/
The /some_path/.+ regexp matches /some_path/ followed by at least a single char.
This is unexpected by most users, since they expect it should match /some_path/.
Substitute .+ with .*, so this regexp matches /some_path/ .
2024-07-19 08:47:49 +02:00
Yury Molodov
efd70b2c52 vmui/logs: switched requests to sequential execution (#6624)
### Describe Your Changes

This PR changes `/select/logsql/query` and `/select/logsql/hits` to
execute sequentially
Fixed
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6558#issuecomment-2219298984

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-18 11:55:42 +02:00
Aliaksandr Valialkin
cda6a31ae8 docs/Single-server-VictoriaMetrics.md: recommend periodic running of make spellcheck command
This is a follow-up for fabf0b928e
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6562
2024-07-18 11:51:53 +02:00
Aliaksandr Valialkin
c6e7fb688b docs/goals.md: make clear that complication of CI/CD pipeline is bad
CI/CD pipeline is usually hard to debug. So it is good to keep it as simple as possible.
2024-07-18 11:51:52 +02:00
Andrii Chubatiuk
ef01df10bd Code tooltip updates (#6657)
### Describe Your Changes

recently added custom `promtextmetric` and `influxtextmetric` prismjs
plugins to vmdocs to convert this
<img width="1081" alt="Screenshot 2024-07-17 at 12 13 29"
src="https://github.com/user-attachments/assets/66d4ea18-10fe-45ef-80b4-989d0eb3bd92">
to this
<img width="1087" alt="Screenshot 2024-07-17 at 12 24 34"
src="https://github.com/user-attachments/assets/60d4ab44-79e5-4c63-b966-54b989ead1aa">




### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-18 02:50:49 -07:00
Aliaksandr Valialkin
798256a655 docs/Release-Guide.md: shorten release guide for Ansible roles and RPM packages
There is no need in enumerating all the files, which must be updated, since these files
may be moved to other locations. It is enough to mention that versions for all the VictoriaMetrics
components must be updated.
2024-07-17 23:36:21 +02:00
Aliaksandr Valialkin
c8bc2f0ee5 app/vmselect/vmui: run make vmui-update after 959a4383c5 2024-07-17 23:09:18 +02:00
Aliaksandr Valialkin
9c0871786f docs/vmbackup.md: an attempt to fix formatting issue at https://docs.victoriametrics.com/vmbackup/#providing-credentials-via-env-variables 2024-07-17 23:01:23 +02:00
Aliaksandr Valialkin
94ad22d140 docs/stream-aggregation.md: an attempt to fix yaml formatting at https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config 2024-07-17 22:43:34 +02:00
Aliaksandr Valialkin
45794de0f9 deployment: update VictoriaMetrics Docker image from v1.101.0 to v1.102.0
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.102.0
2024-07-17 22:34:48 +02:00
Aliaksandr Valialkin
65bb429b81 docs/CHANGELOG.md: cut v1.102.0 release 2024-07-17 20:49:51 +02:00
Aliaksandr Valialkin
66a34acc3a vendor: run make vendor-update 2024-07-17 20:47:20 +02:00
Aliaksandr Valialkin
ea5c3571e9 docs/CHANGELOG.md: consistently use new url format for the MetricsQL docs
Use https://docs.victoriametrics.com/metricsql/ instead of https://docs.victoriametrics.com/MetricsQL.html .
This removes unnecessary redirect from https://docs.victoriametrics.com/MetricsQL.html to https://docs.victoriametrics.com/metricsql/

This is a follow-up for 6a4bd5049b
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6547
2024-07-17 20:31:17 +02:00
Aliaksandr Valialkin
bcbfea56f9 docs/CHANGELOG.md: document v1.97.6 LTS release
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.6
2024-07-17 20:27:44 +02:00
Aliaksandr Valialkin
6a12d2f18b docs/CHANGELOG.md: document v1.93.16 LTS release
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.16
2024-07-17 19:47:55 +02:00
Aliaksandr Valialkin
1e8987df29 docs/CHANGELOG.md: order the changes at tip, so they are easier to read 2024-07-17 18:55:46 +02:00
Fred Navruzov
e5beeb18d3 docs/vmanomaly - v1.13.3 patch notes (#6659)
### Describe Your Changes

Add patch note v1.13.3 to CHANGELOG doc page for `vmanomaly`

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-17 19:52:05 +03:00
Aliaksandr Valialkin
e78a51bf39 docs/VictoriaLogs/querying/README.md: add link to Grafana datasource for VictoriaLogs to the top of the page 2024-07-17 18:34:41 +02:00
Aliaksandr Valialkin
aefc4538bd docs/Single-server-VictoriaMetrics.md: explain why images must have the same filepath as the corresponding docs at docs/ folder
This is a follow-up for f2c1d30011
2024-07-17 18:26:16 +02:00
Aliaksandr Valialkin
1f96882367 docs: fix spellcheck errors found by make spellcheck
Thanks to the spellcheck Makefile rule added by @arkid15r at fabf0b928e
2024-07-17 18:14:20 +02:00
Aliaksandr Valialkin
65ce4e30ab lib/backup/azremote: follow-up for 5fd3aef549
- Mention that credentials can be configured via env variables at both vmbackup and vmrestore docs.

- Make clear that the AZURE_STORAGE_DOMAIN env var is optional at https://docs.victoriametrics.com/vmbackup/#providing-credentials-via-env-variables

- Use string literals as is for env variable names instead of indirecting them via string constants.
  This makes easier to read and understand the code. These environment variable names aren't going to change
  in the future, so there is no sense in hiding them under string constants with some other names.

- Refer to https://docs.victoriametrics.com/vmbackup/#providing-credentials-via-env-variables in error messages
  when auth creds are improperly configured. This should simplify figuring out how to fix the error.

- Simplify the code a bit at FS.newClient(), so it is easier to follow it now.
  While at it, remove the check when superflouos environment variables are set, since it is too fragile
  and it looks like it doesn't help properly configuring vmbackup / vmrestore.

- Remove envLookuper indirection - just use 'func(name string) (string, bool)' type inline.
  This simplifies code reading and understanding.

- Split TestFSInit() into TestFSInit_Failure() and TestFSInit_Success(). This simplifies the test code,
  so it should be easier to maintain in the future.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6518
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5984
2024-07-17 17:55:06 +02:00
Aliaksandr Valialkin
eaed0465d2 all: substitute double "the the" with "the"
This is a follow-up for 8786a08d27

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6600
2024-07-17 14:28:12 +02:00
Aliaksandr Valialkin
9c4b0334f2 all: consistently use stringsutil.JSONString() for formatting JSON strings with fmt.* functions instead of using "%q" formatter
The %q formatter may result in incorrectly formatted JSON string if the original string
contains special chars such as \x1b . They must be encoded as \u001b , otherwise the resulting JSON string
cannot be parsed by JSON parsers.

This is a follow-up for c0caa69939

See https://github.com/VictoriaMetrics/victorialogs-datasource/issues/24
2024-07-17 13:52:13 +02:00
Aliaksandr Valialkin
8ff051b287 lib/protoparser/graphite: use Regex.ReplaceAllLiteralString instead of Regex.ReplaceAllString for the case when the replacement cannot contain placeholders for capturing groups
This is a follow-up for 74affa3aec
2024-07-17 13:01:05 +02:00
Aliaksandr Valialkin
74affa3aec lib/protoparser/graphite: follow-up for 476faf5578
- Clarify the description of -graphite.sanitizeMetricName command-line flag at README.md
- Do not sanitize tag values - only metric names and tag names must be sanitized,
  since they are treated specially by Grafana. Grafana doesn't apply any restrictions on tag values.
- Properly replace more than two consecutive dots with a single dot.
- Disallow unicode letters in metric names and tag names, since neither Prometheus nor Grafana
  do not support them.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6489
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6077
2024-07-17 12:41:55 +02:00
Aliaksandr Valialkin
58a757cd01 lib: consistently use regexp.Regexp.ReplaceAllLiteralString instead of regexp.Regexp.ReplaceAllString in places where the replacement cannot contain matching group placeholders 2024-07-17 12:41:54 +02:00
rtm0
bdc0e688e8 Fix inconsistent error handling in Storage.AddRows() (#6583)
### Describe Your Changes

`Storage.AddRows()` returns an error only in one case: when
`Storage.updatePerDateData()` fails to unmarshal a `metricNameRaw`. But
the same error is treated as a warning when it happens inside
`Storage.add()` or returned by `Storage.prefillNextIndexDB()`.

This commit fixes this inconsistency by treating the error returned by
`Storage.updatePerDateData()` as a warning as well. As a result
`Storage.add()` does not need a return value anymore and so doesn't
`Storage.AddRows()`.

Additionally, this commit adds a unit test that checks all cases that
result in a row not being added to the storage.



---------

Signed-off-by: Artem Fetishev <wwctrsrx@gmail.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2024-07-17 12:07:14 +02:00
Aliaksandr Valialkin
6f9f861f57 docs/CONTRIBUTING.md: mention that docs/CHANGELOG.md shouldn't contain technical details for the changes
The purpose of docs/CHANGELOG.md is to provide VictoriaMetrics users clear and concise information
on what's changed at VictoriaMetrics components. Technical details of the change are unclear
to most of VictoriaMetrics users, who are not familiar with VictoriaMetrics source code.
These details complicate reading the docs/CHANGELOG.md by ordinary users, so do not clutter
the changelog with technical details. If the user wants technical details, he can click
the link to the related GitHub issue and/or pull request and dive into all the details he wants.
2024-07-17 11:46:24 +02:00
Aliaksandr Valialkin
7ed719b46a app/vmauth: properly handle the case when zero backend hosts are resolved at SRV DNS
When zero backend hosts are resolved, then vmauth must return 'no backend hosts' error instead of crashing with panic

This is a follow-up for 590aeccd7d and 3a45bbb4e0

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6401
2024-07-17 11:31:05 +02:00
Aliaksandr Valialkin
7ee5797493 app/vmauth: pool readTrackingBody structs in order to reduce pressure on Go GC
- use pool for readTrackingBody structs in order to reduce pressure on Go GC
- allow re-reading partially read request body
- add missing tests for various cases of readTrackingBody usage

This is a follow-up for ad6af95183 and 4d66e042e3.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6445
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6446
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6533
2024-07-17 11:06:18 +02:00
Aliaksandr Valialkin
277aad18d8 app/vmauth: use more clear names for the field and function added at e666d64f1d
- Rename overrideHostHeader() function to hasEmptyHostHeader()
- Rename overrideHostHeader field at UserInfo to useBackendHostHeader

This should simplify the future maintenance of the code

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6525
2024-07-16 19:08:38 +02:00
Aliaksandr Valialkin
ad6af95183 Revert "app/vmauth: reader pool to reduce gc & mem alloc (#6533)"
This reverts commit 4d66e042e3.

Reasons for revert:

- The commit makes unrelated invalid changes to docs/CHANGELOG.md
- The changes at app/vmauth/main.go are too complex. It is better splitting them into two parts:
  - pooling readTrackingBody struct for reducing pressure on GC
  - avoiding to use readTrackingBody when -maxRequestBodySizeToRetry command-line flag is set to 0

Let's make this in the follow-up commits!

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6445
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6533
2024-07-16 18:59:16 +02:00
750 changed files with 16324 additions and 18482 deletions

1
.gitignore vendored
View File

@@ -7,6 +7,7 @@
.vscode
*.test
*.swp
/vmdocs
/gocache-for-docker
/victoria-logs-data
/victoria-metrics-data

View File

@@ -13,6 +13,7 @@ PKG_TAG := $(BUILDINFO_TAG)
endif
GO_BUILDINFO = -X '$(PKG_PREFIX)/lib/buildinfo.Version=$(APP_NAME)-$(DATEINFO_TAG)-$(BUILDINFO_TAG)'
TAR_OWNERSHIP ?= --owner=1000 --group=1000
.PHONY: $(MAKECMDGOALS)
@@ -245,7 +246,7 @@ release-victoria-metrics-windows-amd64:
release-victoria-metrics-goos-goarch: victoria-metrics-$(GOOS)-$(GOARCH)-prod
cd bin && \
tar --transform="flags=r;s|-$(GOOS)-$(GOARCH)||" -czf victoria-metrics-$(GOOS)-$(GOARCH)-$(PKG_TAG).tar.gz \
tar $(TAR_OWNERSHIP) --transform="flags=r;s|-$(GOOS)-$(GOARCH)||" -czf victoria-metrics-$(GOOS)-$(GOARCH)-$(PKG_TAG).tar.gz \
victoria-metrics-$(GOOS)-$(GOARCH)-prod \
&& sha256sum victoria-metrics-$(GOOS)-$(GOARCH)-$(PKG_TAG).tar.gz \
victoria-metrics-$(GOOS)-$(GOARCH)-prod \
@@ -302,7 +303,7 @@ release-victoria-logs-windows-amd64:
release-victoria-logs-goos-goarch: victoria-logs-$(GOOS)-$(GOARCH)-prod
cd bin && \
tar --transform="flags=r;s|-$(GOOS)-$(GOARCH)||" -czf victoria-logs-$(GOOS)-$(GOARCH)-$(PKG_TAG).tar.gz \
tar $(TAR_OWNERSHIP) --transform="flags=r;s|-$(GOOS)-$(GOARCH)||" -czf victoria-logs-$(GOOS)-$(GOARCH)-$(PKG_TAG).tar.gz \
victoria-logs-$(GOOS)-$(GOARCH)-prod \
&& sha256sum victoria-logs-$(GOOS)-$(GOARCH)-$(PKG_TAG).tar.gz \
victoria-logs-$(GOOS)-$(GOARCH)-prod \
@@ -366,7 +367,7 @@ release-vmutils-goos-goarch: \
vmrestore-$(GOOS)-$(GOARCH)-prod \
vmctl-$(GOOS)-$(GOARCH)-prod
cd bin && \
tar --transform="flags=r;s|-$(GOOS)-$(GOARCH)||" -czf vmutils-$(GOOS)-$(GOARCH)-$(PKG_TAG).tar.gz \
tar $(TAR_OWNERSHIP) --transform="flags=r;s|-$(GOOS)-$(GOARCH)||" -czf vmutils-$(GOOS)-$(GOARCH)-$(PKG_TAG).tar.gz \
vmagent-$(GOOS)-$(GOARCH)-prod \
vmalert-$(GOOS)-$(GOARCH)-prod \
vmalert-tool-$(GOOS)-$(GOARCH)-prod \
@@ -494,7 +495,7 @@ golangci-lint: install-golangci-lint
golangci-lint run
install-golangci-lint:
which golangci-lint || curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(shell go env GOPATH)/bin v1.59.1
which golangci-lint || curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(shell go env GOPATH)/bin v1.60.1
remove-golangci-lint:
rm -rf `which golangci-lint`
@@ -513,34 +514,3 @@ install-wwhrd:
check-licenses: install-wwhrd
wwhrd check -f .wwhrd.yml
copy-docs:
# The 'printf' function is used instead of 'echo' or 'echo -e' to handle line breaks (e.g. '\n') in the same way on different operating systems (MacOS/Ubuntu Linux/Arch Linux) and their shells (bash/sh/zsh/fish).
# For details, see https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4548#issue-1782796419 and https://stackoverflow.com/questions/8467424/echo-newline-in-bash-prints-literal-n
echo "---" > ${DST}
@if [ ${ORDER} -ne 0 ]; then \
echo "sort: ${ORDER}" >> ${DST}; \
echo "weight: ${ORDER}" >> ${DST}; \
printf "menu:\n docs:\n parent: 'victoriametrics'\n weight: ${ORDER}\n" >> ${DST}; \
fi
echo "title: ${TITLE}" >> ${DST}
@if [ ${OLD_URL} ]; then \
printf "aliases:\n - ${OLD_URL}\n" >> ${DST}; \
fi
echo "---" >> ${DST}
cat ${SRC} >> ${DST}
sed -i='.tmp' 's/<img src=\"docs\//<img src=\"/' ${DST}
sed -i='.tmp' 's/<source srcset=\"docs\//<source srcset=\"/' ${DST}
rm -rf docs/*.tmp
# Copies docs for all components and adds the order/weight tag, title, menu position and alias with the backward compatible link for the old site.
# For ORDER=0 it adds no order tag/weight tag.
# FOR OLD_URL - relative link, used for backward compatibility with the link from documentation based on GitHub pages (old one)
# FOR OLD_URL='' it adds no alias, it should be empty for every new page, don't change it for already existing links.
# Images starting with <img src="docs/ are replaced with <img src="
# Cluster docs are supposed to be ordered as 2nd.
# The rest of docs is ordered manually.
docs-sync:
SRC=README.md DST=docs/README.md OLD_URL='' ORDER=0 TITLE=VictoriaMetrics $(MAKE) copy-docs
SRC=README.md DST=docs/Single-server-VictoriaMetrics.md OLD_URL='/Single-server-VictoriaMetrics.html' TITLE=VictoriaMetrics ORDER=1 $(MAKE) copy-docs

3247
README.md

File diff suppressed because it is too large Load Diff

View File

@@ -7,12 +7,12 @@ The following versions of VictoriaMetrics receive regular security fixes:
| Version | Supported |
|---------|--------------------|
| [latest release](https://docs.victoriametrics.com/changelog/) | :white_check_mark: |
| v1.102.x [LTS line](https://docs.victoriametrics.com/lts-releases/) | :white_check_mark: |
| v1.97.x [LTS line](https://docs.victoriametrics.com/lts-releases/) | :white_check_mark: |
| v1.93.x [LTS line](https://docs.victoriametrics.com/lts-releases/) | :white_check_mark: |
| other releases | :x: |
See [this page](https://victoriametrics.com/security/) for more details.
## Reporting a Vulnerability
Please report any security issues to security@victoriametrics.com
Please report any security issues to <security@victoriametrics.com>

View File

@@ -57,6 +57,12 @@ func RequestHandler(path string, w http.ResponseWriter, r *http.Request) bool {
fmt.Fprintf(w, `{}`)
return true
}
if strings.HasPrefix(path, "/logstash") || strings.HasPrefix(path, "/_logstash") {
// Return fake response for Logstash APIs requests.
// See: https://www.elastic.co/guide/en/elasticsearch/reference/8.8/logstash-apis.html
fmt.Fprintf(w, `{}`)
return true
}
switch path {
case "/":
switch r.Method {

View File

@@ -230,7 +230,7 @@ func generateLogsAtTimestamp(bw *bufio.Writer, workerID int, ts int64, firstStre
for i := 0; i < activeStreams; i++ {
ip := toIPv4(rand.Uint32())
uuid := toUUID(rand.Uint64(), rand.Uint64())
fmt.Fprintf(bw, `{"_time":%q,"_msg":"message for the stream %d and worker %d; ip=%s; uuid=%s; u64=%d","host":"host_%d","worker_id":"%d"`,
fmt.Fprintf(bw, `{"_time":"%s","_msg":"message for the stream %d and worker %d; ip=%s; uuid=%s; u64=%d","host":"host_%d","worker_id":"%d"`,
timeStr, streamID, workerID, ip, uuid, rand.Uint64(), streamID, workerID)
fmt.Fprintf(bw, `,"run_id":"%s"`, runID)
for j := 0; j < *constFieldsPerLog; j++ {

View File

@@ -1,13 +1,13 @@
{
"files": {
"main.css": "./static/css/main.1041c3d4.css",
"main.js": "./static/js/main.62a82f4a.js",
"main.css": "./static/css/main.c9cc37dd.css",
"main.js": "./static/js/main.7ae2e2c4.js",
"static/js/685.bebe1265.chunk.js": "./static/js/685.bebe1265.chunk.js",
"static/media/MetricsQL.md": "./static/media/MetricsQL.aaabf95f2c9bf356bde4.md",
"static/media/MetricsQL.md": "./static/media/MetricsQL.8c2e588d62b87f90dbf0.md",
"index.html": "./index.html"
},
"entrypoints": [
"static/css/main.1041c3d4.css",
"static/js/main.62a82f4a.js"
"static/css/main.c9cc37dd.css",
"static/js/main.7ae2e2c4.js"
]
}

View File

@@ -1 +1 @@
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=5"/><meta name="theme-color" content="#000000"/><meta name="description" content="UI for VictoriaMetrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><script src="./dashboards/index.js" type="module"></script><meta name="twitter:card" content="summary_large_image"><meta name="twitter:image" content="./preview.jpg"><meta name="twitter:title" content="UI for VictoriaMetrics"><meta name="twitter:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta name="twitter:site" content="@VictoriaMetrics"><meta property="og:title" content="Metric explorer for VictoriaMetrics"><meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta property="og:image" content="./preview.jpg"><meta property="og:type" content="website"><script defer="defer" src="./static/js/main.62a82f4a.js"></script><link href="./static/css/main.1041c3d4.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=5"/><meta name="theme-color" content="#000000"/><meta name="description" content="UI for VictoriaMetrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><meta name="twitter:card" content="summary_large_image"><meta name="twitter:image" content="./preview.jpg"><meta name="twitter:title" content="UI for VictoriaMetrics"><meta name="twitter:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta name="twitter:site" content="@VictoriaMetrics"><meta property="og:title" content="Metric explorer for VictoriaMetrics"><meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta property="og:image" content="./preview.jpg"><meta property="og:type" content="website"><script defer="defer" src="./static/js/main.7ae2e2c4.js"></script><link href="./static/css/main.c9cc37dd.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

View File

@@ -1,5 +1,4 @@
---
sort: 23
weight: 23
title: MetricsQL
menu:
@@ -10,9 +9,6 @@ aliases:
- /ExtendedPromQL.html
- /MetricsQL.html
---
# MetricsQL
[VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics) implements MetricsQL -
query language inspired by [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/).
MetricsQL is backwards-compatible with PromQL, so Grafana dashboards backed by Prometheus datasource should work
@@ -220,9 +216,11 @@ See also [descent_over_time](#descent_over_time).
over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d` per each time series returned
from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
This function is supported by PromQL.
See also [median_over_time](#median_over_time).
See also [median_over_time](#median_over_time), [min_over_time](#min_over_time) and [max_over_time](#max_over_time).
#### changes
@@ -262,6 +260,8 @@ from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#f
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [count_over_time](#count_over_time), [share_eq_over_time](#share_eq_over_time) and [count_values_over_time](#count_values_over_time).
#### count_gt_over_time
@@ -272,6 +272,8 @@ from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#f
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [count_over_time](#count_over_time) and [share_gt_over_time](#share_gt_over_time).
#### count_le_over_time
@@ -282,6 +284,8 @@ from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#f
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [count_over_time](#count_over_time) and [share_le_over_time](#share_le_over_time).
#### count_ne_over_time
@@ -292,7 +296,9 @@ from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#f
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
See also [count_over_time](#count_over_time).
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [count_over_time](#count_over_time) and [count_eq_over_time](#count_eq_over_time).
#### count_over_time
@@ -313,6 +319,8 @@ The results are calculated independently per each time series returned from the
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [count_eq_over_time](#count_eq_over_time), [count_values](#count_values) and [distinct_over_time](#distinct_over_time) and [label_match](#label_match).
#### decreases_over_time
@@ -432,6 +440,8 @@ from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#f
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
#### histogram_over_time
`histogram_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates
@@ -443,11 +453,15 @@ For example, the following query calculates median temperature by country over t
`histogram_quantile(0.5, sum(histogram_over_time(temperature[24h])) by (vmrange,country))`.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
#### hoeffding_bound_lower
`hoeffding_bound_lower(phi, series_selector[d])` is a [rollup function](#rollup-functions), which calculates
lower [Hoeffding bound](https://en.wikipedia.org/wiki/Hoeffding%27s_inequality) for the given `phi` in the range `[0...1]`.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [hoeffding_bound_upper](#hoeffding_bound_upper).
#### hoeffding_bound_upper
@@ -455,6 +469,8 @@ See also [hoeffding_bound_upper](#hoeffding_bound_upper).
`hoeffding_bound_upper(phi, series_selector[d])` is a [rollup function](#rollup-functions), which calculates
upper [Hoeffding bound](https://en.wikipedia.org/wiki/Hoeffding%27s_inequality) for the given `phi` in the range `[0...1]`.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [hoeffding_bound_lower](#hoeffding_bound_lower).
#### holt_winters
@@ -462,8 +478,9 @@ See also [hoeffding_bound_lower](#hoeffding_bound_lower).
`holt_winters(series_selector[d], sf, tf)` is a [rollup function](#rollup-functions), which calculates Holt-Winters value
(aka [double exponential smoothing](https://en.wikipedia.org/wiki/Exponential_smoothing#Double_exponential_smoothing)) for [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
over the given lookbehind window `d` using the given smoothing factor `sf` and the given trend factor `tf`.
Both `sf` and `tf` must be in the range `[0...1]`. It is expected that the [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering)
returns time series of [gauge type](https://docs.victoriametrics.com/keyconcepts/#gauge).
Both `sf` and `tf` must be in the range `[0...1]`.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
This function is supported by PromQL.
@@ -495,13 +512,14 @@ See also [deriv](#deriv).
`increase(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the increase over the given lookbehind window `d`
per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
It is expected that the `series_selector` returns time series of [counter type](https://docs.victoriametrics.com/keyconcepts/#counter).
Unlike Prometheus, it takes into account the last sample before the given lookbehind window `d` when calculating the result.
See [this article](https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e) for details.
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
This function is supported by PromQL.
See also [increase_pure](#increase_pure), [increase_prometheus](#increase_prometheus) and [delta](#delta).
@@ -510,12 +528,13 @@ See also [increase_pure](#increase_pure), [increase_prometheus](#increase_promet
`increase_prometheus(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the increase
over the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
It is expected that the `series_selector` returns time series of [counter type](https://docs.victoriametrics.com/keyconcepts/#counter).
It doesn't take into account the last sample before the given lookbehind window `d` when calculating the result in the same way as Prometheus does.
See [this article](https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e) for details.
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
See also [increase_pure](#increase_pure) and [increase](#increase).
#### increase_pure
@@ -524,6 +543,10 @@ See also [increase_pure](#increase_pure) and [increase](#increase).
of the following corner case - it assumes that [counters](https://docs.victoriametrics.com/keyconcepts/#counter) always start from 0,
while [increase](#increase) ignores the first value in a series if it is too big.
This function is usually applied to [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
See also [increase](#increase) and [increase_prometheus](#increase_prometheus).
#### increases_over_time
`increases_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the number of [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
@@ -540,15 +563,18 @@ on the given lookbehind window `d` per each time series returned from the given
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
#### irate
`irate(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the "instant" per-second increase rate over
the last two [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
It is expected that the `series_selector` returns time series of [counter type](https://docs.victoriametrics.com/keyconcepts/#counter).
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
This function is supported by PromQL.
See also [rate](#rate) and [rollup_rate](#rollup_rate).
@@ -587,6 +613,8 @@ See also [duration_over_time](#duration_over_time) and [lag](#lag).
over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d` per each time series returned
from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [mad](#mad), [range_mad](#range_mad) and [outlier_iqr_over_time](#outlier_iqr_over_time).
#### max_over_time
@@ -594,9 +622,11 @@ See also [mad](#mad), [range_mad](#range_mad) and [outlier_iqr_over_time](#outli
`max_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the maximum value over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
This function is supported by PromQL.
See also [tmax_over_time](#tmax_over_time).
See also [tmax_over_time](#tmax_over_time) and [min_over_time](#min_over_time).
#### median_over_time
@@ -604,6 +634,8 @@ See also [tmax_over_time](#tmax_over_time).
on the given lookbehind window `d` per each time series returned
from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [avg_over_time](#avg_over_time).
#### min_over_time
@@ -611,9 +643,11 @@ See also [avg_over_time](#avg_over_time).
`min_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the minimum value over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
This function is supported by PromQL.
See also [tmin_over_time](#tmin_over_time).
See also [tmin_over_time](#tmin_over_time) and [max_over_time](#max_over_time).
#### mode_over_time
@@ -622,6 +656,8 @@ for [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on
from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). It is expected that [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
values are discrete.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
#### outlier_iqr_over_time
`outlier_iqr_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which returns the last sample on the given lookbehind window `d`
@@ -632,6 +668,8 @@ if its value is either smaller than the `q25-1.5*iqr` or bigger than `q75+1.5*iq
The `outlier_iqr_over_time()` is useful for detecting anomalies in gauge values based on the previous history of values.
For example, `outlier_iqr_over_time(memory_usage_bytes[1h])` triggers when `memory_usage_bytes` suddenly goes outside the usual value range for the last hour.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [outliers_iqr](#outliers_iqr).
#### predict_linear
@@ -659,6 +697,8 @@ This function is supported by PromQL.
on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
The `phi` value must be in the range `[0...1]`.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
This function is supported by PromQL.
See also [quantiles_over_time](#quantiles_over_time).
@@ -670,6 +710,8 @@ over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on
from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
The function returns individual series per each `phi*` with `{phiLabel="phi*"}` label. `phi*` values must be in the range `[0...1]`.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [quantile_over_time](#quantile_over_time).
#### range_over_time
@@ -680,11 +722,12 @@ E.g. it calculates `max_over_time(series_selector[d]) - min_over_time(series_sel
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
#### rate
`rate(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the average per-second increase rate
over the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
It is expected that the `series_selector` returns time series of [counter type](https://docs.victoriametrics.com/keyconcepts/#counter).
If the lookbehind window is skipped in square brackets, then it is automatically calculated as `max(step, scrape_interval)`, where `step` is the query arg value
passed to [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query) or [/api/v1/query](https://docs.victoriametrics.com/keyconcepts/#instant-query),
@@ -703,6 +746,8 @@ See also [irate](#irate) and [rollup_rate](#rollup_rate).
on the given lookbehind window `d`. The calculations are performed individually per each time series returned
from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
#### resets
@@ -710,10 +755,11 @@ Metric names are stripped from the resulting rollups. Add [keep_metric_names](#k
`resets(series_selector[d])` is a [rollup function](#rollup-functions), which returns the number
of [counter](https://docs.victoriametrics.com/keyconcepts/#counter) resets over the given lookbehind window `d`
per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
It is expected that the `series_selector` returns time series of [counter type](https://docs.victoriametrics.com/keyconcepts/#counter).
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
This function is supported by PromQL.
#### rollup
@@ -725,6 +771,10 @@ These values are calculated individually per each time series returned from the
Optional 2nd argument `"min"`, `"max"` or `"avg"` can be passed to keep only one calculation result and without adding a label.
See also [label_match](#label_match).
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [rollup_rate](#rollup_rate).
#### rollup_candlestick
`rollup_candlestick(series_selector[d])` is a [rollup function](#rollup-functions), which calculates `open`, `high`, `low` and `close` values (aka OHLC)
@@ -736,6 +786,8 @@ from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#f
Optional 2nd argument `"open"`, `"high"` or `"low"` or `"close"` can be passed to keep only one calculation result and without adding a label.
See also [label_match](#label_match).
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
#### rollup_delta
`rollup_delta(series_selector[d])` is a [rollup function](#rollup-functions), which calculates differences between adjacent [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
@@ -762,6 +814,8 @@ See also [label_match](#label_match).
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
See also [rollup](#rollup) and [rollup_rate](#rollup_rate).
#### rollup_increase
`rollup_increase(series_selector[d])` is a [rollup function](#rollup-functions), which calculates increases for adjacent [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
@@ -774,12 +828,17 @@ See also [label_match](#label_match).
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. See also [rollup_delta](#rollup_delta).
This function is usually applied to [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
See also [rollup](#rollup) and [rollup_rate](#rollup_rate).
#### rollup_rate
`rollup_rate(series_selector[d])` is a [rollup function](#rollup-functions), which calculates per-second change rates
for adjacent [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
on the given lookbehind window `d` and returns `min`, `max` and `avg` values for the calculated per-second change rates
and returns them in time series with `rollup="min"`, `rollup="max"` and `rollup="avg"` additional labels.
The calculations are performed individually per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
See [this article](https://valyala.medium.com/why-irate-from-prometheus-doesnt-capture-spikes-45f9896d7832) in order to understand better
when to use `rollup_rate()`.
@@ -787,10 +846,12 @@ when to use `rollup_rate()`.
Optional 2nd argument `"min"`, `"max"` or `"avg"` can be passed to keep only one calculation result and without adding a label.
See also [label_match](#label_match).
The calculations are performed individually per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
See also [rollup](#rollup) and [rollup_increase](#rollup_increase).
#### rollup_scrape_interval
`rollup_scrape_interval(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the interval in seconds between
@@ -824,6 +885,8 @@ This function is useful for calculating SLI and SLO. Example: `share_gt_over_tim
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [share_le_over_time](#share_le_over_time) and [count_gt_over_time](#count_gt_over_time).
#### share_le_over_time
@@ -838,6 +901,8 @@ the share of time series values for the last 24 hours when memory usage was belo
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [share_gt_over_time](#share_gt_over_time) and [count_le_over_time](#count_le_over_time).
#### share_eq_over_time
@@ -849,6 +914,8 @@ from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#f
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [count_eq_over_time](#count_eq_over_time).
#### stale_samples_over_time
@@ -866,6 +933,8 @@ on the given lookbehind window `d` per each time series returned from the given
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
This function is supported by PromQL.
See also [stdvar_over_time](#stdvar_over_time).
@@ -877,35 +946,43 @@ on the given lookbehind window `d` per each time series returned from the given
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
This function is supported by PromQL.
See also [stddev_over_time](#stddev_over_time).
#### sum_eq_over_time
`sum_eq_over_time(series_selector[d], eq)` is a [rollup function](#rollup-function), which calculates the sum of [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
`sum_eq_over_time(series_selector[d], eq)` is a [rollup function](#rollup-functions), which calculates the sum of [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
values equal to `eq` on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [sum_over_time](#sum_over_time) and [count_eq_over_time](#count_eq_over_time).
#### sum_gt_over_time
`sum_gt_over_time(series_selector[d], gt)` is a [rollup function](#rollup-function), which calculates the sum of [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
`sum_gt_over_time(series_selector[d], gt)` is a [rollup function](#rollup-functions), which calculates the sum of [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
values bigger than `gt` on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [sum_over_time](#sum_over_time) and [count_gt_over_time](#count_gt_over_time).
#### sum_le_over_time
`sum_le_over_time(series_selector[d], le)` is a [rollup function](#rollup-function), which calculates the sum of [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
`sum_le_over_time(series_selector[d], le)` is a [rollup function](#rollup-functions), which calculates the sum of [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
values smaller or equal to `le` on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [sum_over_time](#sum_over_time) and [count_le_over_time](#count_le_over_time).
#### sum_over_time
@@ -915,6 +992,8 @@ on the given lookbehind window `d` per each time series returned from the given
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
This function is supported by PromQL.
#### sum2_over_time
@@ -924,6 +1003,8 @@ values on the given lookbehind window `d` per each time series returned from the
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
#### timestamp
`timestamp(series_selector[d])` is a [rollup function](#rollup-functions), which returns the timestamp in seconds with millisecond precision
@@ -1001,6 +1082,8 @@ from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#f
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [zscore](#zscore), [range_trim_zscore](#range_trim_zscore) and [outlier_iqr_over_time](#outlier_iqr_over_time).
@@ -2230,7 +2313,8 @@ Any [rollup function](#rollup-functions) for something other than [series select
Nested rollup functions can be implicit thanks to the [implicit query conversions](#implicit-query-conversions).
For example, `delta(sum(m))` is implicitly converted to `delta(sum(default_rollup(m))[1i:1i])`, so it becomes a subquery,
since it contains [default_rollup](#default_rollup) nested into [delta](#delta).
This behavior can be disabled or logged via cmd-line flags `-search.disableImplicitConversion` and `-search.logImplicitConversion` since v1.101.0.
This behavior can be disabled or logged via `-search.disableImplicitConversion` and `-search.logImplicitConversion` command-line flags
starting from [`v1.101.0` release](https://docs.victoriametrics.com/changelog/).
VictoriaMetrics performs subqueries in the following way:
@@ -2263,5 +2347,6 @@ VictoriaMetrics performs the following implicit conversions for incoming queries
For example, `avg_over_time(rate(http_requests_total[5m])[1h])` is automatically converted to `avg_over_time(rate(http_requests_total[5m])[1h:1i])`.
* If something other than [series selector](https://docs.victoriametrics.com/keyconcepts/#filtering)
is passed to [rollup function](#rollup-functions), then a [subquery](#subqueries) with `1i` lookbehind window and `1i` step is automatically formed.
For example, `rate(sum(up))` is automatically converted to `rate((sum(default_rollup(up)))[1i:1i])`.
This behavior can be disabled or logged via cmd-line flags `-search.disableImplicitConversion` and `-search.logImplicitConversion` since v1.101.0.
For example, `rate(sum(up))` is automatically converted to `rate((sum(default_rollup(up)))[1i:1i])`.
This behavior can be disabled or logged via `-search.disableImplicitConversion` and `-search.logImplicitConversion` command-line flags
starting from [`v1.101.0` release](https://docs.victoriametrics.com/changelog/).

View File

@@ -0,0 +1,55 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8"/>
<link rel="icon" href="%PUBLIC_URL%/favicon.ico"/>
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=5"/>
<meta name="theme-color" content="#000000"/>
<meta
name="description"
content="UI for VictoriaMetrics"
/>
<link rel="apple-touch-icon" href="%PUBLIC_URL%/apple-touch-icon.png"/>
<link rel="icon" type="image/png" sizes="32x32" href="%PUBLIC_URL%/favicon-32x32.png">
<!--
manifest.json provides metadata used when your web app is installed on a
user's mobile device or desktop. See https://developers.google.com/web/fundamentals/web-app-manifest/
-->
<link rel="manifest" href="%PUBLIC_URL%/manifest.json"/>
<!--
Notice the use of %PUBLIC_URL% in the tags above.
It will be replaced with the URL of the `public` folder during the build.
Only files inside the `public` folder can be referenced from the HTML.
Unlike "/favicon.ico" or "favicon.ico", "%PUBLIC_URL%/favicon.ico" will
work correctly both with client-side routing and a non-root public URL.
Learn how to configure a non-root public URL by running `npm run build`.
-->
<title>VM UI</title>
<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:image" content="%PUBLIC_URL%/preview.jpg">
<meta name="twitter:title" content="UI for VictoriaMetrics">
<meta name="twitter:description" content="Explore and troubleshoot your VictoriaMetrics data">
<meta name="twitter:site" content="@VictoriaMetrics">
<meta property="og:title" content="Metric explorer for VictoriaMetrics">
<meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data">
<meta property="og:image" content="%PUBLIC_URL%/preview.jpg">
<meta property="og:type" content="website">
</head>
<body>
<noscript>You need to enable JavaScript to run this app.</noscript>
<div id="root"></div>
<!--
This HTML file is a template.
If you open it directly in the browser, you will see an empty page.
You can add webfonts, meta tags, or analytics to this file.
The build step will place the bundled scripts into the <body> tag.
To begin the development, run `npm start` or `yarn start`.
To create a production bundle, use `npm run build` or `yarn build`.
-->
</body>
</html>

View File

@@ -0,0 +1,56 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8"/>
<link rel="icon" href="%PUBLIC_URL%/favicon.ico"/>
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=5"/>
<meta name="theme-color" content="#000000"/>
<meta
name="description"
content="UI for VictoriaMetrics"
/>
<link rel="apple-touch-icon" href="%PUBLIC_URL%/apple-touch-icon.png"/>
<link rel="icon" type="image/png" sizes="32x32" href="%PUBLIC_URL%/favicon-32x32.png">
<!--
manifest.json provides metadata used when your web app is installed on a
user's mobile device or desktop. See https://developers.google.com/web/fundamentals/web-app-manifest/
-->
<link rel="manifest" href="%PUBLIC_URL%/manifest.json"/>
<!--
Notice the use of %PUBLIC_URL% in the tags above.
It will be replaced with the URL of the `public` folder during the build.
Only files inside the `public` folder can be referenced from the HTML.
Unlike "/favicon.ico" or "favicon.ico", "%PUBLIC_URL%/favicon.ico" will
work correctly both with client-side routing and a non-root public URL.
Learn how to configure a non-root public URL by running `npm run build`.
-->
<title>VM UI</title>
<script src="%PUBLIC_URL%/dashboards/index.js" type="module"></script>
<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:image" content="%PUBLIC_URL%/preview.jpg">
<meta name="twitter:title" content="UI for VictoriaMetrics">
<meta name="twitter:description" content="Explore and troubleshoot your VictoriaMetrics data">
<meta name="twitter:site" content="@VictoriaMetrics">
<meta property="og:title" content="Metric explorer for VictoriaMetrics">
<meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data">
<meta property="og:image" content="%PUBLIC_URL%/preview.jpg">
<meta property="og:type" content="website">
</head>
<body>
<noscript>You need to enable JavaScript to run this app.</noscript>
<div id="root"></div>
<!--
This HTML file is a template.
If you open it directly in the browser, you will see an empty page.
You can add webfonts, meta tags, or analytics to this file.
The build step will place the bundled scripts into the <body> tag.
To begin the development, run `npm start` or `yarn start`.
To create a production bundle, use `npm run build` or `yarn build`.
-->
</body>
</html>

View File

@@ -0,0 +1,56 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8"/>
<link rel="icon" href="%PUBLIC_URL%/favicon.ico"/>
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=5"/>
<meta name="theme-color" content="#000000"/>
<meta
name="description"
content="UI for VictoriaMetrics"
/>
<link rel="apple-touch-icon" href="%PUBLIC_URL%/apple-touch-icon.png"/>
<link rel="icon" type="image/png" sizes="32x32" href="%PUBLIC_URL%/favicon-32x32.png">
<!--
manifest.json provides metadata used when your web app is installed on a
user's mobile device or desktop. See https://developers.google.com/web/fundamentals/web-app-manifest/
-->
<link rel="manifest" href="%PUBLIC_URL%/manifest.json"/>
<!--
Notice the use of %PUBLIC_URL% in the tags above.
It will be replaced with the URL of the `public` folder during the build.
Only files inside the `public` folder can be referenced from the HTML.
Unlike "/favicon.ico" or "favicon.ico", "%PUBLIC_URL%/favicon.ico" will
work correctly both with client-side routing and a non-root public URL.
Learn how to configure a non-root public URL by running `npm run build`.
-->
<title>VM UI</title>
<script src="%PUBLIC_URL%/dashboards/index.js" type="module"></script>
<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:image" content="%PUBLIC_URL%/preview.jpg">
<meta name="twitter:title" content="UI for VictoriaMetrics">
<meta name="twitter:description" content="Explore and troubleshoot your VictoriaMetrics data">
<meta name="twitter:site" content="@VictoriaMetrics">
<meta property="og:title" content="Metric explorer for VictoriaMetrics">
<meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data">
<meta property="og:image" content="%PUBLIC_URL%/preview.jpg">
<meta property="og:type" content="website">
</head>
<body>
<noscript>You need to enable JavaScript to run this app.</noscript>
<div id="root"></div>
<!--
This HTML file is a template.
If you open it directly in the browser, you will see an empty page.
You can add webfonts, meta tags, or analytics to this file.
The build step will place the bundled scripts into the <body> tag.
To begin the development, run `npm start` or `yarn start`.
To create a production bundle, use `npm run build` or `yarn build`.
-->
</body>
</html>

View File

@@ -10,6 +10,8 @@ import (
"strings"
"time"
"github.com/VictoriaMetrics/metrics"
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmagent/csvimport"
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmagent/datadogsketches"
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmagent/datadogv1"
@@ -42,7 +44,7 @@ import (
"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/common"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/opentelemetry/firehose"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/pushmetrics"
"github.com/VictoriaMetrics/metrics"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/stringsutil"
)
var (
@@ -316,6 +318,10 @@ func requestHandler(w http.ResponseWriter, r *http.Request) bool {
influxQueryRequests.Inc()
influxutils.WriteDatabaseNames(w)
return true
case "/influx/health":
influxHealthRequests.Inc()
influxutils.WriteHealthCheckResponse(w)
return true
case "/opentelemetry/api/v1/push", "/opentelemetry/v1/metrics":
opentelemetryPushRequests.Inc()
if err := opentelemetry.InsertHandler(nil, r); err != nil {
@@ -450,7 +456,7 @@ func requestHandler(w http.ResponseWriter, r *http.Request) bool {
w.Header().Set("Content-Type", "application/json")
var bb bytesutil.ByteBuffer
promscrape.WriteConfigData(&bb)
fmt.Fprintf(w, `{"status":"success","data":{"yaml":%q}}`, bb.B)
fmt.Fprintf(w, `{"status":"success","data":{"yaml":%s}}`, stringsutil.JSONString(string(bb.B)))
return true
case "/prometheus/-/reload", "/-/reload":
if !httpserver.CheckAuthFlag(w, r, reloadAuthKey) {
@@ -562,6 +568,10 @@ func processMultitenantRequest(w http.ResponseWriter, r *http.Request, path stri
influxQueryRequests.Inc()
influxutils.WriteDatabaseNames(w)
return true
case "influx/health":
influxHealthRequests.Inc()
influxutils.WriteHealthCheckResponse(w)
return true
case "opentelemetry/api/v1/push", "opentelemetry/v1/metrics":
opentelemetryPushRequests.Inc()
if err := opentelemetry.InsertHandler(at, r); err != nil {
@@ -672,7 +682,8 @@ var (
influxWriteRequests = metrics.NewCounter(`vmagent_http_requests_total{path="/influx/write", protocol="influx"}`)
influxWriteErrors = metrics.NewCounter(`vmagent_http_request_errors_total{path="/influx/write", protocol="influx"}`)
influxQueryRequests = metrics.NewCounter(`vmagent_http_requests_total{path="/influx/query", protocol="influx"}`)
influxQueryRequests = metrics.NewCounter(`vmagent_http_requests_total{path="/influx/query", protocol="influx"}`)
influxHealthRequests = metrics.NewCounter(`vmagent_http_requests_total{path="/influx/health", protocol="influx"}`)
datadogv1WriteRequests = metrics.NewCounter(`vmagent_http_requests_total{path="/datadog/api/v1/series", protocol="datadog"}`)
datadogv1WriteErrors = metrics.NewCounter(`vmagent_http_request_errors_total{path="/datadog/api/v1/series", protocol="datadog"}`)

View File

@@ -34,8 +34,10 @@ var (
rateLimit = flagutil.NewArrayInt("remoteWrite.rateLimit", 0, "Optional rate limit in bytes per second for data sent to the corresponding -remoteWrite.url. "+
"By default, the rate limit is disabled. It can be useful for limiting load on remote storage when big amounts of buffered data "+
"is sent after temporary unavailability of the remote storage. See also -maxIngestionRate")
sendTimeout = flagutil.NewArrayDuration("remoteWrite.sendTimeout", time.Minute, "Timeout for sending a single block of data to the corresponding -remoteWrite.url")
proxyURL = flagutil.NewArrayString("remoteWrite.proxyURL", "Optional proxy URL for writing data to the corresponding -remoteWrite.url. "+
sendTimeout = flagutil.NewArrayDuration("remoteWrite.sendTimeout", time.Minute, "Timeout for sending a single block of data to the corresponding -remoteWrite.url")
retryMinInterval = flagutil.NewArrayDuration("remoteWrite.retryMinInterval", time.Second, "The minimum delay between retry attempts to send a block of data to the corresponding -remoteWrite.url. Every next retry attempt will double the delay to prevent hammering of remote database. See also -remoteWrite.retryMaxInterval")
retryMaxTime = flagutil.NewArrayDuration("remoteWrite.retryMaxTime", time.Minute, "The max time spent on retry attempts to send a block of data to the corresponding -remoteWrite.url. Change this value if it is expected for -remoteWrite.url to be unreachable for more than -remoteWrite.retryMaxTime. See also -remoteWrite.retryMinInterval")
proxyURL = flagutil.NewArrayString("remoteWrite.proxyURL", "Optional proxy URL for writing data to the corresponding -remoteWrite.url. "+
"Supported proxies: http, https, socks5. Example: -remoteWrite.proxyURL=socks5://proxy:1234")
tlsHandshakeTimeout = flagutil.NewArrayDuration("remoteWrite.tlsHandshakeTimeout", 20*time.Second, "The timeout for establishing tls connections to the corresponding -remoteWrite.url")
@@ -90,6 +92,9 @@ type client struct {
fq *persistentqueue.FastQueue
hc *http.Client
retryMinInterval time.Duration
retryMaxTime time.Duration
sendBlock func(block []byte) bool
authCfg *promauth.Config
awsCfg *awsapi.Config
@@ -143,13 +148,15 @@ func newHTTPClient(argIdx int, remoteWriteURL, sanitizedURL string, fq *persiste
Timeout: sendTimeout.GetOptionalArg(argIdx),
}
c := &client{
sanitizedURL: sanitizedURL,
remoteWriteURL: remoteWriteURL,
authCfg: authCfg,
awsCfg: awsCfg,
fq: fq,
hc: hc,
stopCh: make(chan struct{}),
sanitizedURL: sanitizedURL,
remoteWriteURL: remoteWriteURL,
authCfg: authCfg,
awsCfg: awsCfg,
fq: fq,
hc: hc,
retryMinInterval: retryMinInterval.GetOptionalArg(argIdx),
retryMaxTime: retryMaxTime.GetOptionalArg(argIdx),
stopCh: make(chan struct{}),
}
c.sendBlock = c.sendBlockHTTP
@@ -396,11 +403,11 @@ func (c *client) newRequest(url string, body []byte) (*http.Request, error) {
// sendBlockHTTP sends the given block to c.remoteWriteURL.
//
// The function returns false only if c.stopCh is closed.
// Otherwise it tries sending the block to remote storage indefinitely.
// Otherwise, it tries sending the block to remote storage indefinitely.
func (c *client) sendBlockHTTP(block []byte) bool {
c.rl.Register(len(block))
maxRetryDuration := timeutil.AddJitterToDuration(time.Minute)
retryDuration := timeutil.AddJitterToDuration(time.Second)
maxRetryDuration := timeutil.AddJitterToDuration(c.retryMaxTime)
retryDuration := timeutil.AddJitterToDuration(c.retryMinInterval)
retriesCount := 0
again:

View File

@@ -441,7 +441,7 @@ func tryPush(at *auth.Token, wr *prompbmarshal.WriteRequest, forceDropSamplesOnF
var rctx *relabelCtx
rcs := allRelabelConfigs.Load()
pcsGlobal := rcs.global
if pcsGlobal.Len() > 0 {
if pcsGlobal.Len() > 0 || *usePromCompatibleNaming {
rctx = getRelabelCtx()
defer putRelabelCtx(rctx)
}

View File

@@ -50,7 +50,7 @@ var (
streamAggrIgnoreOldSamples = flagutil.NewArrayBool("remoteWrite.streamAggr.ignoreOldSamples", "Whether to ignore input samples with old timestamps outside the current "+
"aggregation interval for the corresponding -remoteWrite.streamAggr.config at the corresponding -remoteWrite.url. "+
"See https://docs.victoriametrics.com/stream-aggregation/#ignoring-old-samples")
streamAggrIgnoreFirstIntervals = flag.Int("remoteWrite.streamAggr.ignoreFirstIntervals", 0, "Number of aggregation intervals to skip after the start "+
streamAggrIgnoreFirstIntervals = flagutil.NewArrayInt("remoteWrite.streamAggr.ignoreFirstIntervals", 0, "Number of aggregation intervals to skip after the start "+
"for the corresponding -remoteWrite.streamAggr.config at the corresponding -remoteWrite.url. Increase this value if "+
"you observe incorrect aggregation results after vmagent restarts. It could be caused by receiving bufferred delayed data from clients pushing data into the vmagent. "+
"See https://docs.victoriametrics.com/stream-aggregation/#ignore-aggregation-intervals-on-start")
@@ -133,7 +133,7 @@ func initStreamAggrConfigGlobal() {
} else {
dedupInterval := streamAggrGlobalDedupInterval.Duration()
if dedupInterval > 0 {
deduplicatorGlobal = streamaggr.NewDeduplicator(pushToRemoteStoragesTrackDropped, dedupInterval, *streamAggrDropInputLabels, "dedup-global")
deduplicatorGlobal = streamaggr.NewDeduplicator(pushToRemoteStoragesTrackDropped, dedupInterval, *streamAggrGlobalDropInputLabels, "dedup-global")
}
}
}
@@ -202,6 +202,7 @@ func newStreamAggrConfigGlobal() (*streamaggr.Aggregators, error) {
DropInputLabels: *streamAggrGlobalDropInputLabels,
IgnoreOldSamples: *streamAggrGlobalIgnoreOldSamples,
IgnoreFirstIntervals: *streamAggrGlobalIgnoreFirstIntervals,
KeepInput: *streamAggrGlobalKeepInput,
}
sas, err := streamaggr.LoadFromFile(path, pushToRemoteStoragesTrackDropped, opts, "global")
@@ -229,7 +230,8 @@ func newStreamAggrConfigPerURL(idx int, pushFunc streamaggr.PushFunc) (*streamag
DedupInterval: streamAggrDedupInterval.GetOptionalArg(idx),
DropInputLabels: *streamAggrDropInputLabels,
IgnoreOldSamples: streamAggrIgnoreOldSamples.GetOptionalArg(idx),
IgnoreFirstIntervals: *streamAggrIgnoreFirstIntervals,
IgnoreFirstIntervals: streamAggrIgnoreFirstIntervals.GetOptionalArg(idx),
KeepInput: streamAggrKeepInput.GetOptionalArg(idx),
}
sas, err := streamaggr.LoadFromFile(path, pushFunc, opts, alias)

View File

@@ -41,9 +41,19 @@ Examples:
Usage: "disable adding group's Name as label to generated alerts and time series.",
Required: false,
},
&cli.StringSliceFlag{
Name: "external.label",
Usage: `Optional label in the form 'name=value' to add to all generated recording rules and alerts. Supports an array of values separated by comma or specified via multiple flags.`,
Required: false,
},
&cli.StringFlag{
Name: "external.url",
Usage: `Optional external URL to template in rule's labels or annotations.`,
Required: false,
},
},
Action: func(c *cli.Context) error {
if failed := unittest.UnitTest(c.StringSlice("files"), c.Bool("disableAlertgroupLabel")); failed {
if failed := unittest.UnitTest(c.StringSlice("files"), c.Bool("disableAlertgroupLabel"), c.StringSlice("external.label"), c.String("external.url")); failed {
return fmt.Errorf("unittest failed")
}
return nil

View File

@@ -13,7 +13,7 @@ tests:
- expr: suquery_interval_test
eval_time: 4m
exp_samples:
- labels: '{__name__="suquery_interval_test",datacenter="dc-123", instance="localhost:9090", job="vmagent2"}'
- labels: '{__name__="suquery_interval_test", instance="localhost:9090", job="vmagent2"}'
value: 1
alert_rule_test:
@@ -24,20 +24,16 @@ tests:
job: vmagent2
severity: page
instance: localhost:9090
datacenter: dc-123
exp_annotations:
summary: "Instance localhost:9090 down"
description: "localhost:9090 of job vmagent2 has been down for more than 5 minutes."
description: "localhost:9090 of job vmagent2 in cluster has been down for more than 5 minutes."
dashboard: "/d/dashboard?orgId=1"
- eval_time: 0
alertname: AlwaysFiring
exp_alerts:
- exp_labels:
datacenter: dc-123
- {}
- eval_time: 0
alertname: InstanceDown
exp_alerts: []
external_labels:
datacenter: dc-123

View File

@@ -8,7 +8,8 @@ groups:
severity: page
annotations:
summary: "Instance {{ $labels.instance }} down"
description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes."
description: "{{ $labels.instance }} of job {{ $labels.job }} in cluster {{ $externalLabels.cluster }} has been down for more than 5 minutes."
dashboard: '{{ $externalURL }}/d/dashboard?orgId=1'
- alert: AlwaysFiring
expr: 1
- alert: SameAlertNameWithDifferentGroup

View File

@@ -16,7 +16,8 @@ tests:
groupname: group1
alertname: SameAlertNameWithDifferentGroup
exp_alerts:
- {}
- exp_labels:
cluster: prod
- eval_time: 1m
groupname: group2
alertname: SameAlertNameWithDifferentGroup
@@ -25,7 +26,8 @@ tests:
groupname: group1
alertname: SameAlertNameWithDifferentGroup
exp_alerts:
- {}
- exp_labels:
cluster: prod
- eval_time: 6m
groupname: group1
alertname: SameAlertNameWithDifferentGroup
@@ -61,18 +63,18 @@ tests:
eval_time: 4m
exp_samples:
- value: 4
labels: '{__name__="t1", datacenter="dc-123"}'
labels: '{__name__="t1", cluster="prod"}'
- expr: t2
eval_time: 4m
exp_samples:
- value: 4
labels: '{__name__="t2", datacenter="dc-123"}'
labels: '{__name__="t2", cluster="prod"}'
- expr: t3
eval_time: 4m
exp_samples:
# t3 is 3 instead of 4 cause it's rules3 is evaluated before rules1
- value: 3
labels: '{__name__="t3", datacenter="dc-123"}'
labels: '{__name__="t3", cluster="prod"}'
alert_rule_test:
- eval_time: 10m
@@ -83,22 +85,21 @@ tests:
job: vmagent1
severity: page
instance: localhost:9090
datacenter: dc-123
cluster: prod
exp_annotations:
summary: "Instance localhost:9090 down"
description: "localhost:9090 of job vmagent1 has been down for more than 5 minutes."
description: "localhost:9090 of job vmagent1 in cluster prod has been down for more than 5 minutes."
dashboard: "http://grafana:3000/d/dashboard?orgId=1"
- eval_time: 0
groupname: group1
alertname: AlwaysFiring
exp_alerts:
- exp_labels:
datacenter: dc-123
cluster: prod
- eval_time: 0
groupname: alerts
alertname: InstanceDown
exp_alerts: []
external_labels:
datacenter: dc-123

View File

@@ -13,7 +13,7 @@ tests:
- expr: suquery_interval_test
eval_time: 4m
exp_samples:
- labels: '{__name__="suquery_interval_test",datacenter="dc-123", instance="localhost:9090", job="vmagent2"}'
- labels: '{__name__="suquery_interval_test", cluster="prod", instance="localhost:9090", job="vmagent2"}'
value: 1
alert_rule_test:
@@ -25,22 +25,21 @@ tests:
job: vmagent2
severity: page
instance: localhost:9090
datacenter: dc-123
cluster: prod
exp_annotations:
summary: "Instance localhost:9090 down"
description: "localhost:9090 of job vmagent2 has been down for more than 5 minutes."
description: "localhost:9090 of job vmagent2 in cluster prod has been down for more than 5 minutes."
dashboard: "http://grafana:3000/d/dashboard?orgId=1"
- eval_time: 0
groupname: group1
alertname: AlwaysFiring
exp_alerts:
- exp_labels:
datacenter: dc-123
cluster: prod
- eval_time: 0
groupname: group1
alertname: InstanceDown
exp_alerts: []
external_labels:
datacenter: dc-123

View File

@@ -55,7 +55,7 @@ const (
)
// UnitTest runs unittest for files
func UnitTest(files []string, disableGroupLabel bool) bool {
func UnitTest(files []string, disableGroupLabel bool, externalLabels []string, externalURL string) bool {
if err := templates.Load([]string{}, true); err != nil {
logger.Fatalf("failed to load template: %v", err)
}
@@ -71,14 +71,34 @@ func UnitTest(files []string, disableGroupLabel bool) bool {
testfiles, err := config.ReadFromFS(files)
if err != nil {
fmt.Println(" FAILED")
fmt.Printf("\nfailed to read test files: \n%v", err)
logger.Fatalf("failed to load test files %q: %v", files, err)
}
if len(testfiles) == 0 {
fmt.Println("no test file found")
return false
}
labels := make(map[string]string)
for _, s := range externalLabels {
if len(s) == 0 {
continue
}
n := strings.IndexByte(s, '=')
if n < 0 {
logger.Fatalf("missing '=' in `-label`. It must contain label in the form `name=value`; got %q", s)
}
labels[s[:n]] = s[n+1:]
}
_, err = notifier.Init(nil, labels, externalURL)
if err != nil {
logger.Fatalf("failed to init notifier: %v", err)
}
var failed bool
for fileName, file := range testfiles {
if err := ruleUnitTest(fileName, file); err != nil {
if err := ruleUnitTest(fileName, file, labels); err != nil {
fmt.Println(" FAILED")
fmt.Printf("\nfailed to run unit test for file %q: \n%v", file, err)
fmt.Printf("\nfailed to run unit test for file %q: \n%v", fileName, err)
failed = true
} else {
fmt.Println(" SUCCESS")
@@ -88,7 +108,7 @@ func UnitTest(files []string, disableGroupLabel bool) bool {
return failed
}
func ruleUnitTest(filename string, content []byte) []error {
func ruleUnitTest(filename string, content []byte, externalLabels map[string]string) []error {
fmt.Println("\nUnit Testing: ", filename)
var unitTestInp unitTestFile
if err := yaml.UnmarshalStrict(content, &unitTestInp); err != nil {
@@ -126,7 +146,7 @@ func ruleUnitTest(filename string, content []byte) []error {
errs = append(errs, err)
continue
}
testErrs := t.test(unitTestInp.EvaluationInterval.Duration(), groupOrderMap, testGroups)
testErrs := t.test(unitTestInp.EvaluationInterval.Duration(), groupOrderMap, testGroups, externalLabels)
errs = append(errs, testErrs...)
}
@@ -163,6 +183,10 @@ func verifyTestGroup(group testGroup) error {
return fmt.Errorf("\n%s missing required field \"eval_time\"", testGroupName)
}
}
if group.ExternalLabels != nil {
fmt.Printf("\n%s warning: filed `external_labels` will be deprecated soon, please use `-external.label` cmd-line flag instead. "+
"Check https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6735 for details.\n", testGroupName)
}
return nil
}
@@ -179,6 +203,7 @@ func processFlags() {
{flag: "retentionPeriod", value: "100y"},
{flag: "datasource.url", value: testDataSourcePath},
{flag: "remoteWrite.url", value: testRemoteWritePath},
{flag: "notifier.blackhole", value: "true"},
} {
// panics if flag doesn't exist
if err := flag.Lookup(fv.flag).Value.Set(fv.value); err != nil {
@@ -239,7 +264,7 @@ func tearDown() {
fs.MustRemoveAll(storagePath)
}
func (tg *testGroup) test(evalInterval time.Duration, groupOrderMap map[string]int, testGroups []vmalertconfig.Group) (checkErrs []error) {
func (tg *testGroup) test(evalInterval time.Duration, groupOrderMap map[string]int, testGroups []vmalertconfig.Group, externalLabels map[string]string) (checkErrs []error) {
// set up vmstorage and http server for ingest and read queries
setUp()
// tear down vmstorage and clean the data dir
@@ -288,7 +313,14 @@ func (tg *testGroup) test(evalInterval time.Duration, groupOrderMap map[string]i
// create groups with given rule
var groups []*rule.Group
for _, group := range testGroups {
ng := rule.NewGroup(group, q, time.Minute, tg.ExternalLabels)
mergedExternalLabels := make(map[string]string)
for k, v := range tg.ExternalLabels {
mergedExternalLabels[k] = v
}
for k, v := range externalLabels {
mergedExternalLabels[k] = v
}
ng := rule.NewGroup(group, q, time.Minute, mergedExternalLabels)
groups = append(groups, ng)
}

View File

@@ -18,7 +18,7 @@ func TestUnitTest_Failure(t *testing.T) {
f := func(files []string) {
t.Helper()
failed := UnitTest(files, false)
failed := UnitTest(files, false, nil, "")
if !failed {
t.Fatalf("expecting failed test")
}
@@ -29,18 +29,19 @@ func TestUnitTest_Failure(t *testing.T) {
}
func TestUnitTest_Success(t *testing.T) {
f := func(disableGroupLabel bool, files []string) {
f := func(disableGroupLabel bool, files []string, externalLabels []string, externalURL string) {
t.Helper()
failed := UnitTest(files, disableGroupLabel)
failed := UnitTest(files, disableGroupLabel, externalLabels, externalURL)
if failed {
t.Fatalf("unexpected failed test")
}
}
// run multi files
f(false, []string{"./testdata/test1.yaml", "./testdata/test2.yaml"})
f(false, []string{"./testdata/test1.yaml", "./testdata/test2.yaml"}, []string{"cluster=prod"}, "http://grafana:3000")
// disable group label
f(true, []string{"./testdata/disable-group-label.yaml"})
// template with null external values
f(true, []string{"./testdata/disable-group-label.yaml"}, nil, "")
}

View File

@@ -17,7 +17,7 @@ import (
var (
addr = flag.String("datasource.url", "", "Datasource compatible with Prometheus HTTP API. It can be single node VictoriaMetrics or vmselect endpoint. Required parameter. "+
"Supports address in the form of IP address with a port (e.g., 127.0.0.1:8428) or DNS SRV record. "+
"Supports address in the form of IP address with a port (e.g., http://127.0.0.1:8428) or DNS SRV record. "+
"See also -remoteRead.disablePathAppend and -datasource.showURL")
appendTypePrefix = flag.Bool("datasource.appendTypePrefix", false, "Whether to add type prefix to -datasource.url based on the query type. Set to true if sending different query types to the vmselect URL.")
showDatasourceURL = flag.Bool("datasource.showURL", false, "Whether to avoid stripping sensitive information such as auth headers or passwords from URLs in log messages or UI and exported metrics. "+
@@ -99,7 +99,7 @@ func Init(extraParams url.Values) (QuerierBuilder, error) {
tr, err := httputils.Transport(*addr, *tlsCertFile, *tlsKeyFile, *tlsCAFile, *tlsServerName, *tlsInsecureSkipVerify)
if err != nil {
return nil, fmt.Errorf("failed to create transport: %w", err)
return nil, fmt.Errorf("failed to create transport for -datasource.url=%q: %w", *addr, err)
}
tr.DialContext = netutil.NewStatDialFunc("vmalert_datasource")
tr.DisableKeepAlives = *disableKeepAlive

File diff suppressed because one or more lines are too long

View File

@@ -57,20 +57,23 @@ var jsonParserPool fastjson.ParserPool
// [{"metric":{"__name__":"up","job":"prometheus"},value": [ 1435781451.781,"1"]},
// {"metric":{"__name__":"up","job":"node"},value": [ 1435781451.781,"0"]}]
func (pi *promInstant) Unmarshal(b []byte) error {
var metrics []json.RawMessage
// metrics slice could be large, so parsing it with fastjson could consume a lot of memory.
// We parse the slice with standard lib to keep mem usage low.
// And each metric object will be parsed with fastjson to reduce allocations.
if err := json.Unmarshal(b, &metrics); err != nil {
return fmt.Errorf("cannot unmarshal metrics: %w", err)
}
p := jsonParserPool.Get()
defer jsonParserPool.Put(p)
v, err := p.ParseBytes(b)
if err != nil {
return err
}
rows, err := v.Array()
if err != nil {
return fmt.Errorf("cannot find the top-level array of result objects: %w", err)
}
pi.ms = make([]Metric, len(rows))
for i, row := range rows {
pi.ms = make([]Metric, len(metrics))
for i, data := range metrics {
row, err := p.ParseBytes(data)
if err != nil {
return fmt.Errorf("cannot parse metric object: %w", err)
}
metric := row.Get("metric")
if metric == nil {
return fmt.Errorf("can't find `metric` object in %q", row)

View File

@@ -1,43 +1,24 @@
package datasource
import (
"bytes"
"io"
"net/http"
"os"
"testing"
)
func BenchmarkMetrics(b *testing.B) {
payload := []byte(`[{"metric":{"__name__":"vm_rows"},"value":[1583786142,"13763"]},{"metric":{"__name__":"vm_requests", "foo":"bar", "baz": "qux"},"value":[1583786140,"2000"]}]`)
var pi promInstant
if err := pi.Unmarshal(payload); err != nil {
b.Fatalf(err.Error())
}
b.Run("Instant", func(b *testing.B) {
for i := 0; i < b.N; i++ {
_, _ = pi.metrics()
}
})
}
func BenchmarkParsePrometheusResponse(b *testing.B) {
req, _ := http.NewRequest("GET", "", nil)
resp := &http.Response{StatusCode: http.StatusOK}
func BenchmarkPromInstantUnmarshal(b *testing.B) {
data, err := os.ReadFile("testdata/instant_response.json")
if err != nil {
b.Fatalf("error while reading file: %s", err)
}
resp.Body = io.NopCloser(bytes.NewReader(data))
b.Run("Instant", func(b *testing.B) {
// BenchmarkParsePrometheusResponse/Instant_std+fastjson-10 1760 668959 ns/op 280147 B/op 5781 allocs/op
b.Run("Instant std+fastjson", func(b *testing.B) {
for i := 0; i < b.N; i++ {
_, err := parsePrometheusResponse(req, resp)
var pi promInstant
err = pi.Unmarshal(data)
if err != nil {
b.Fatalf("unexpected parse err: %s", err)
}
resp.Body = io.NopCloser(bytes.NewReader(data))
}
})
}

View File

@@ -76,9 +76,6 @@ func (am *AlertManager) send(ctx context.Context, alerts []Alert, headers map[st
return err
}
req.Header.Set("Content-Type", "application/json")
for key, value := range headers {
req.Header.Set(key, value)
}
if am.timeout > 0 {
var cancel context.CancelFunc
@@ -94,6 +91,11 @@ func (am *AlertManager) send(ctx context.Context, alerts []Alert, headers map[st
return err
}
}
// external headers have higher priority
for key, value := range headers {
req.Header.Set(key, value)
}
resp, err := am.client.Do(req)
if err != nil {
return err
@@ -130,7 +132,8 @@ func NewAlertManager(alertManagerURL string, fn AlertURLGenerator, authCfg proma
}
tr, err := httputils.Transport(alertManagerURL, tls.CertFile, tls.KeyFile, tls.CAFile, tls.ServerName, tls.InsecureSkipVerify)
if err != nil {
return nil, fmt.Errorf("failed to create transport: %w", err)
return nil, fmt.Errorf("failed to create transport for alertmanager URL=%q: %w", alertManagerURL, err)
}
ba := new(promauth.BasicAuthConfig)
@@ -145,7 +148,9 @@ func NewAlertManager(alertManagerURL string, fn AlertURLGenerator, authCfg proma
aCfg, err := utils.AuthConfig(
utils.WithBasicAuth(ba.Username, ba.Password.String(), ba.PasswordFile),
utils.WithBearer(authCfg.BearerToken.String(), authCfg.BearerTokenFile),
utils.WithOAuth(oauth.ClientID, oauth.ClientSecret.String(), oauth.ClientSecretFile, oauth.TokenURL, strings.Join(oauth.Scopes, ";"), oauth.EndpointParams))
utils.WithOAuth(oauth.ClientID, oauth.ClientSecret.String(), oauth.ClientSecretFile, oauth.TokenURL, strings.Join(oauth.Scopes, ";"), oauth.EndpointParams),
utils.WithHeaders(strings.Join(authCfg.Headers, "^^")),
)
if err != nil {
return nil, fmt.Errorf("failed to configure auth: %w", err)
}

View File

@@ -3,6 +3,7 @@ package notifier
import (
"context"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"strconv"
@@ -48,6 +49,9 @@ func TestAlertManager_Send(t *testing.T) {
conn, _, _ := w.(http.Hijacker).Hijack()
_ = conn.Close()
case 1:
if r.Header.Get(headerKey) != headerValue {
t.Fatalf("expected header %q to be set to %q; got %q instead", headerKey, headerValue, r.Header.Get(headerKey))
}
w.WriteHeader(500)
case 2:
var a []struct {
@@ -72,6 +76,9 @@ func TestAlertManager_Send(t *testing.T) {
if a[0].EndAt.IsZero() {
t.Fatalf("expected non-zero end time")
}
if r.Header.Get(headerKey) != "bar" {
t.Fatalf("expected header %q to be set to %q; got %q instead", headerKey, headerValue, r.Header.Get(headerKey))
}
case 3:
if r.Header.Get(headerKey) != headerValue {
t.Fatalf("expected header %q to be set to %q; got %q instead", headerKey, headerValue, r.Header.Get(headerKey))
@@ -86,6 +93,7 @@ func TestAlertManager_Send(t *testing.T) {
Username: baUser,
Password: promauth.NewSecret(baPass),
},
Headers: []string{fmt.Sprintf("%s:%s", headerKey, headerValue)},
}
am, err := NewAlertManager(srv.URL+alertManagerPath, func(alert Alert) string {
return strconv.FormatUint(alert.GroupID, 10) + "/" + strconv.FormatUint(alert.ID, 10)
@@ -105,7 +113,7 @@ func TestAlertManager_Send(t *testing.T) {
Start: time.Now().UTC(),
End: time.Now().UTC(),
Annotations: map[string]string{"a": "b", "c": "d", "e": "f"},
}}, nil); err != nil {
}}, map[string]string{headerKey: "bar"}); err != nil {
t.Fatalf("unexpected error %s", err)
}
if c != 2 {

View File

@@ -25,6 +25,9 @@ var (
"Enable this flag if you want vmalert to evaluate alerting rules without sending any notifications to external receivers (eg. alertmanager). "+
"-notifier.url, -notifier.config and -notifier.blackhole are mutually exclusive.")
headers = flagutil.NewArrayString("notifier.headers", "Optional HTTP headers to send with each request to the corresponding -notifier.url. "+
"For example, -remoteWrite.headers='My-Auth:foobar' would send 'My-Auth: foobar' HTTP header with every request to the corresponding -notifier.url. "+
"Multiple headers must be delimited by '^^': -notifier.headers='header1:value1^^header2:value2,header3:value3'")
basicAuthUsername = flagutil.NewArrayString("notifier.basicAuth.username", "Optional basic auth username for -notifier.url")
basicAuthPassword = flagutil.NewArrayString("notifier.basicAuth.password", "Optional basic auth password for -notifier.url")
basicAuthPasswordFile = flagutil.NewArrayString("notifier.basicAuth.passwordFile", "Optional path to basic auth password file for -notifier.url")
@@ -171,6 +174,7 @@ func notifiersFromFlags(gen AlertURLGenerator) ([]Notifier, error) {
Scopes: strings.Split(oauth2Scopes.GetOptionalArg(i), ";"),
TokenURL: oauth2TokenURL.GetOptionalArg(i),
},
Headers: []string{headers.GetOptionalArg(i)},
}
addr = strings.TrimSuffix(addr, "/")

View File

@@ -17,7 +17,7 @@ var (
addr = flag.String("remoteRead.url", "", "Optional URL to datasource compatible with Prometheus HTTP API. It can be single node VictoriaMetrics or vmselect."+
"Remote read is used to restore alerts state."+
"This configuration makes sense only if `vmalert` was configured with `remoteWrite.url` before and has been successfully persisted its state. "+
"Supports address in the form of IP address with a port (e.g., 127.0.0.1:8428) or DNS SRV record. "+
"Supports address in the form of IP address with a port (e.g., http://127.0.0.1:8428) or DNS SRV record. "+
"See also '-remoteRead.disablePathAppend', '-remoteRead.showURL'.")
showRemoteReadURL = flag.Bool("remoteRead.showURL", false, "Whether to show -remoteRead.url in the exported metrics. "+
@@ -68,7 +68,7 @@ func Init() (datasource.QuerierBuilder, error) {
}
tr, err := httputils.Transport(*addr, *tlsCertFile, *tlsKeyFile, *tlsCAFile, *tlsServerName, *tlsInsecureSkipVerify)
if err != nil {
return nil, fmt.Errorf("failed to create transport: %w", err)
return nil, fmt.Errorf("failed to create transport for -remoteRead.url=%q: %w", *addr, err)
}
tr.IdleConnTimeout = *idleConnectionTimeout
tr.DialContext = netutil.NewStatDialFunc("vmalert_remoteread")

View File

@@ -32,7 +32,7 @@ func NewDebugClient() (*DebugClient, error) {
t, err := httputils.Transport(*addr, *tlsCertFile, *tlsKeyFile, *tlsCAFile, *tlsServerName, *tlsInsecureSkipVerify)
if err != nil {
return nil, fmt.Errorf("failed to create transport: %w", err)
return nil, fmt.Errorf("failed to create transport for -remoteWrite.url=%q: %w", *addr, err)
}
c := &DebugClient{
c: &http.Client{

View File

@@ -15,7 +15,7 @@ import (
var (
addr = flag.String("remoteWrite.url", "", "Optional URL to VictoriaMetrics or vminsert where to persist alerts state "+
"and recording rules results in form of timeseries. "+
"Supports address in the form of IP address with a port (e.g., 127.0.0.1:8428) or DNS SRV record. "+
"Supports address in the form of IP address with a port (e.g., http://127.0.0.1:8428) or DNS SRV record. "+
"For example, if -remoteWrite.url=http://127.0.0.1:8428 is specified, "+
"then the alerts state will be written to http://127.0.0.1:8428/api/v1/write . See also -remoteWrite.disablePathAppend, '-remoteWrite.showURL'.")
showRemoteWriteURL = flag.Bool("remoteWrite.showURL", false, "Whether to show -remoteWrite.url in the exported metrics. "+
@@ -72,7 +72,7 @@ func Init(ctx context.Context) (*Client, error) {
t, err := httputils.Transport(*addr, *tlsCertFile, *tlsKeyFile, *tlsCAFile, *tlsServerName, *tlsInsecureSkipVerify)
if err != nil {
return nil, fmt.Errorf("failed to create transport: %w", err)
return nil, fmt.Errorf("failed to create transport for -remoteWrite.url=%q: %w", *addr, err)
}
t.IdleConnTimeout = *idleConnectionTimeout
t.DialContext = netutil.NewStatDialFunc("vmalert_remotewrite")

View File

@@ -441,9 +441,6 @@ func (ar *AlertingRule) exec(ctx context.Context, ts time.Time, limit int) ([]pr
}
a.Value = m.Values[0]
a.Annotations = annotations
if err != nil {
return nil, err
}
a.KeepFiringSince = time.Time{}
continue
}

View File

@@ -83,7 +83,6 @@ type UserInfo struct {
concurrencyLimitCh chan struct{}
concurrencyLimitReached *metrics.Counter
overrideHostHeader bool
rt http.RoundTripper
@@ -94,8 +93,9 @@ type UserInfo struct {
// HeadersConf represents config for request and response headers.
type HeadersConf struct {
RequestHeaders []*Header `yaml:"headers,omitempty"`
ResponseHeaders []*Header `yaml:"response_headers,omitempty"`
RequestHeaders []*Header `yaml:"headers,omitempty"`
ResponseHeaders []*Header `yaml:"response_headers,omitempty"`
KeepOriginalHost *bool `yaml:"keep_original_host,omitempty"`
}
func (ui *UserInfo) beginConcurrencyLimit() error {
@@ -150,15 +150,6 @@ func (h *Header) MarshalYAML() (any, error) {
return h.sOriginal, nil
}
func overrideHostHeader(headers []*Header) bool {
for _, h := range headers {
if h.Name == "Host" && h.Value == "" {
return true
}
}
return false
}
// URLMap is a mapping from source paths to target urls.
type URLMap struct {
// SrcPaths is an optional list of regular expressions, which must match the request path.
@@ -310,12 +301,18 @@ func (up *URLPrefix) getBackendsCount() int {
// getBackendURL returns the backendURL depending on the load balance policy.
//
// It can return nil if there are no backend urls available at the moment.
//
// backendURL.put() must be called on the returned backendURL after the request is complete.
func (up *URLPrefix) getBackendURL() *backendURL {
up.discoverBackendAddrsIfNeeded()
pbus := up.bus.Load()
bus := *pbus
if len(bus) == 0 {
return nil
}
if up.loadBalancingPolicy == "first_available" {
return getFirstAvailableBackendURL(bus)
}
@@ -397,7 +394,7 @@ func (up *URLPrefix) discoverBackendAddrsIfNeeded() {
buCopy.Host = addr
if port != "" {
if n := strings.IndexByte(buCopy.Host, ':'); n >= 0 {
// Drop the discovered port and substitute it the the port specified in bu.
// Drop the discovered port and substitute it the port specified in bu.
buCopy.Host = buCopy.Host[:n]
}
buCopy.Host += ":" + port
@@ -464,7 +461,7 @@ func getLeastLoadedBackendURL(bus []*backendURL, atomicCounter *atomic.Uint32) *
}
// Slow path - select other backend urls.
n := atomicCounter.Add(1)
n := atomicCounter.Add(1) - 1
for i := uint32(0); i < uint32(len(bus)); i++ {
idx := (n + i) % uint32(len(bus))
@@ -600,7 +597,7 @@ func initAuthConfig() {
// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240
sighupCh := procutil.NewSighupChan()
_, err := loadAuthConfig()
_, err := reloadAuthConfig()
if err != nil {
logger.Fatalf("cannot load auth config: %s", err)
}
@@ -632,7 +629,7 @@ func authConfigReloader(sighupCh <-chan os.Signal) {
updateFn := func() {
configReloads.Inc()
updated, err := loadAuthConfig()
updated, err := reloadAuthConfig()
if err != nil {
logger.Errorf("failed to load auth config; using the last successfully loaded config; error: %s", err)
configSuccess.Set(0)
@@ -658,27 +655,45 @@ func authConfigReloader(sighupCh <-chan os.Signal) {
}
}
// authConfigData stores the yaml definition for this config.
// authConfigData needs to be updated each time authConfig is updated.
var authConfigData atomic.Pointer[[]byte]
var (
authConfig atomic.Pointer[AuthConfig]
authUsers atomic.Pointer[map[string]*UserInfo]
// authConfigData stores the yaml definition for this config.
// authConfigData needs to be updated each time authConfig is updated.
authConfigData atomic.Pointer[[]byte]
// authConfig contains the currently loaded auth config
authConfig atomic.Pointer[AuthConfig]
// authUsers contains the currently loaded auth users
authUsers atomic.Pointer[map[string]*UserInfo]
authConfigWG sync.WaitGroup
stopCh chan struct{}
)
// loadAuthConfig loads and applies the config from *authConfigPath.
// reloadAuthConfig loads and applies the config from *authConfigPath.
// It returns bool value to identify if new config was applied.
// The config can be not applied if there is a parsing error
// or if there are no changes to the current authConfig.
func loadAuthConfig() (bool, error) {
func reloadAuthConfig() (bool, error) {
data, err := fscore.ReadFileOrHTTP(*authConfigPath)
if err != nil {
return false, fmt.Errorf("failed to read -auth.config=%q: %w", *authConfigPath, err)
}
ok, err := reloadAuthConfigData(data)
if err != nil {
return false, fmt.Errorf("failed to pars -auth.config=%q: %w", *authConfigPath, err)
}
if !ok {
return false, nil
}
mp := authUsers.Load()
logger.Infof("loaded information about %d users from -auth.config=%q", len(*mp), *authConfigPath)
return true, nil
}
func reloadAuthConfigData(data []byte) (bool, error) {
oldData := authConfigData.Load()
if oldData != nil && bytes.Equal(data, *oldData) {
// there are no updates in the config - skip reloading.
@@ -687,14 +702,13 @@ func loadAuthConfig() (bool, error) {
ac, err := parseAuthConfig(data)
if err != nil {
return false, fmt.Errorf("failed to parse -auth.config=%q: %w", *authConfigPath, err)
return false, fmt.Errorf("failed to parse auth config: %w", err)
}
m, err := parseAuthConfigUsers(ac)
if err != nil {
return false, fmt.Errorf("failed to parse users from -auth.config=%q: %w", *authConfigPath, err)
return false, fmt.Errorf("failed to parse users from auth config: %w", err)
}
logger.Infof("loaded information about %d users from -auth.config=%q", len(m), *authConfigPath)
acPrev := authConfig.Load()
if acPrev != nil {
@@ -741,7 +755,6 @@ func parseAuthConfig(data []byte) (*AuthConfig, error) {
if err := ui.initURLs(); err != nil {
return nil, err
}
ui.overrideHostHeader = overrideHostHeader(ui.HeadersConf.RequestHeaders)
metricLabels, err := ui.getMetricLabels()
if err != nil {
@@ -806,7 +819,6 @@ func parseAuthConfigUsers(ac *AuthConfig) (map[string]*UserInfo, error) {
_ = ac.ms.GetOrCreateGauge(`vmauth_user_concurrent_requests_current`+metricLabels, func() float64 {
return float64(len(ui.concurrencyLimitCh))
})
ui.overrideHostHeader = overrideHostHeader(ui.HeadersConf.RequestHeaders)
rt, err := newRoundTripper(ui.TLSCAFile, ui.TLSCertFile, ui.TLSKeyFile, ui.TLSServerName, ui.TLSInsecureSkipVerify)
if err != nil {
@@ -997,6 +1009,14 @@ func getAuthTokensFromRequest(r *http.Request) []string {
}
}
// Authorization via http://user:pass@hosname/path
if u := r.URL.User; u != nil && u.Username() != "" {
username := u.Username()
password, _ := u.Password()
at := getHTTPAuthBasicToken(username, password)
ats = append(ats, at)
}
return ats
}
@@ -1022,10 +1042,6 @@ func (up *URLPrefix) sanitizeAndInitialize() error {
}
func sanitizeURLPrefix(urlPrefix *url.URL) (*url.URL, error) {
// Remove trailing '/' from urlPrefix
for strings.HasSuffix(urlPrefix.Path, "/") {
urlPrefix.Path = urlPrefix.Path[:len(urlPrefix.Path)-1]
}
// Validate urlPrefix
if urlPrefix.Scheme != "http" && urlPrefix.Scheme != "https" {
return nil, fmt.Errorf("unsupported scheme for `url_prefix: %q`: %q; must be `http` or `https`", urlPrefix, urlPrefix.Scheme)

View File

@@ -82,6 +82,14 @@ users:
headers: foobar
`)
// Invalid keep_original_host value
f(`
users:
- username: foo
url_prefix: http://foo.bar
keep_original_host: foobar
`)
// empty url_prefix
f(`
users:
@@ -410,7 +418,7 @@ users:
HeadersConf: HeadersConf{
RequestHeaders: []*Header{
mustNewHeader("'foo: bar'"),
mustNewHeader("'xxx: y'"),
mustNewHeader("'xxx:'"),
},
},
},
@@ -429,7 +437,7 @@ users:
url_prefix: ["http://vminsert1/insert/0/prometheus","http://vminsert2/insert/0/prometheus"]
headers:
- "foo: bar"
- "xxx: y"
- "xxx:"
`, map[string]*UserInfo{
getHTTPAuthBearerToken("foo"): sharedUserInfo,
getHTTPAuthBasicToken("foo", ""): sharedUserInfo,
@@ -458,6 +466,7 @@ users:
})
// with default url
keepOriginalHost := true
f(`
users:
- bearer_token: foo
@@ -469,6 +478,7 @@ users:
headers:
- "foo: bar"
- "xxx: y"
keep_original_host: true
default_url:
- http://default1/select/0/prometheus
- http://default2/select/0/prometheus
@@ -491,6 +501,7 @@ users:
mustNewHeader("'foo: bar'"),
mustNewHeader("'xxx: y'"),
},
KeepOriginalHost: &keepOriginalHost,
},
},
},
@@ -517,6 +528,7 @@ users:
mustNewHeader("'foo: bar'"),
mustNewHeader("'xxx: y'"),
},
KeepOriginalHost: &keepOriginalHost,
},
},
},
@@ -536,12 +548,17 @@ users:
metric_labels:
dc: eu
team: dev
keep_original_host: true
- username: foo-same
password: bar
url_prefix: https://bar/x
metric_labels:
backend_env: test
team: accounting
headers:
- "foo: bar"
response_headers:
- "Abc: def"
`, map[string]*UserInfo{
getHTTPAuthBasicToken("foo-same", "baz"): {
Username: "foo-same",
@@ -551,6 +568,9 @@ users:
"dc": "eu",
"team": "dev",
},
HeadersConf: HeadersConf{
KeepOriginalHost: &keepOriginalHost,
},
},
getHTTPAuthBasicToken("foo-same", "bar"): {
Username: "foo-same",
@@ -560,6 +580,14 @@ users:
"backend_env": "test",
"team": "accounting",
},
HeadersConf: HeadersConf{
RequestHeaders: []*Header{
mustNewHeader("'foo: bar'"),
},
ResponseHeaders: []*Header{
mustNewHeader("'Abc: def'"),
},
},
},
})
}
@@ -695,9 +723,9 @@ func TestGetLeastLoadedBackendURL(t *testing.T) {
fn := func(ns ...int) {
t.Helper()
bus := up.bus.Load()
pbus := *bus
for i, b := range pbus {
pbus := up.bus.Load()
bus := *pbus
for i, b := range bus {
got := int(b.concurrentRequests.Load())
exp := ns[i]
if got != exp {
@@ -707,39 +735,39 @@ func TestGetLeastLoadedBackendURL(t *testing.T) {
}
up.getBackendURL()
fn(0, 1, 0)
fn(1, 0, 0)
up.getBackendURL()
fn(0, 1, 1)
fn(1, 1, 0)
up.getBackendURL()
fn(1, 1, 1)
up.getBackendURL()
up.getBackendURL()
fn(1, 2, 2)
fn(2, 2, 1)
bus := up.bus.Load()
pbus := *bus
pbus[0].concurrentRequests.Add(2)
pbus[2].concurrentRequests.Add(5)
fn(3, 2, 7)
fn(4, 2, 6)
up.getBackendURL()
fn(3, 3, 7)
fn(4, 3, 6)
up.getBackendURL()
fn(3, 4, 7)
fn(4, 4, 6)
up.getBackendURL()
fn(4, 4, 7)
fn(4, 5, 6)
up.getBackendURL()
fn(5, 4, 7)
fn(5, 5, 6)
up.getBackendURL()
fn(5, 5, 7)
fn(6, 5, 6)
up.getBackendURL()
fn(6, 5, 7)
fn(6, 6, 6)
up.getBackendURL()
fn(6, 6, 7)

View File

@@ -50,7 +50,7 @@ var (
`Such requests are always counted at vmauth_http_request_errors_total{reason="invalid_auth_token"} metric, which is exposed at /metrics page`)
failTimeout = flag.Duration("failTimeout", 3*time.Second, "Sets a delay period for load balancing to skip a malfunctioning backend")
maxRequestBodySizeToRetry = flagutil.NewBytes("maxRequestBodySizeToRetry", 16*1024, "The maximum request body size, which can be cached and re-tried at other backends. "+
"Bigger values may require more memory. Negative or zero values disable request body caching and retries.")
"Bigger values may require more memory. Zero or negative value disables caching of request body. This may be useful when proxying data ingestion requests")
backendTLSInsecureSkipVerify = flag.Bool("backend.tlsInsecureSkipVerify", false, "Whether to skip TLS verification when connecting to backends over HTTPS. "+
"See https://docs.victoriametrics.com/vmauth/#backend-tls-setup")
backendTLSCAFile = flag.String("backend.TLSCAFile", "", "Optional path to TLS root CA file, which is used for TLS verification when connecting to backends over HTTPS. "+
@@ -117,8 +117,7 @@ func requestHandler(w http.ResponseWriter, r *http.Request) bool {
return true
}
w.Header().Set("WWW-Authenticate", `Basic realm="Restricted"`)
http.Error(w, "missing `Authorization` request header", http.StatusUnauthorized)
handleMissingAuthorizationError(w)
return true
}
@@ -189,26 +188,27 @@ func processRequest(w http.ResponseWriter, r *http.Request, ui *UserInfo) {
// to a route that is not in the configuration for unauthorized user.
// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5236
if ui.BearerToken == "" && ui.Username == "" && len(*authUsers.Load()) > 0 {
w.Header().Set("WWW-Authenticate", `Basic realm="Restricted"`)
http.Error(w, "missing `Authorization` request header", http.StatusUnauthorized)
handleMissingAuthorizationError(w)
return
}
missingRouteRequests.Inc()
httpserver.Errorf(w, r, "missing route for %q", u.String())
httpserver.Errorf(w, r, "missing route for %s", u.String())
return
}
up, hc = ui.DefaultURL, ui.HeadersConf
isDefault = true
}
// caching makes sense only for positive non zero size
if maxRequestBodySizeToRetry.IntN() > 0 {
rtb := getReadTrackingBody(r.Body, int(r.ContentLength))
defer putReadTrackingBody(rtb)
r.Body = rtb
}
rtb := getReadTrackingBody(r.Body, maxRequestBodySizeToRetry.IntN())
defer putReadTrackingBody(rtb)
r.Body = rtb
maxAttempts := up.getBackendsCount()
for i := 0; i < maxAttempts; i++ {
bu := up.getBackendURL()
if bu == nil {
break
}
targetURL := bu.url
// Don't change path and add request_path query param for default route.
if isDefault {
@@ -218,7 +218,15 @@ func processRequest(w http.ResponseWriter, r *http.Request, ui *UserInfo) {
} else { // Update path for regular routes.
targetURL = mergeURLs(targetURL, u, up.dropSrcPathPrefixParts)
}
ok := tryProcessingRequest(w, r, targetURL, hc, up.retryStatusCodes, ui)
wasLocalRetry := false
again:
ok, needLocalRetry := tryProcessingRequest(w, r, targetURL, hc, up.retryStatusCodes, ui)
if needLocalRetry && !wasLocalRetry {
wasLocalRetry = true
goto again
}
bu.put()
if ok {
return
@@ -226,29 +234,28 @@ func processRequest(w http.ResponseWriter, r *http.Request, ui *UserInfo) {
bu.setBroken()
}
err := &httpserver.ErrorWithStatusCode{
Err: fmt.Errorf("all the backends for the user %q are unavailable", ui.name()),
StatusCode: http.StatusServiceUnavailable,
Err: fmt.Errorf("all the %d backends for the user %q are unavailable", up.getBackendsCount(), ui.name()),
StatusCode: http.StatusBadGateway,
}
httpserver.Errorf(w, r, "%s", err)
ui.backendErrors.Inc()
}
func tryProcessingRequest(w http.ResponseWriter, r *http.Request, targetURL *url.URL, hc HeadersConf, retryStatusCodes []int, ui *UserInfo) bool {
// This code has been copied from net/http/httputil/reverseproxy.go
func tryProcessingRequest(w http.ResponseWriter, r *http.Request, targetURL *url.URL, hc HeadersConf, retryStatusCodes []int, ui *UserInfo) (bool, bool) {
req := sanitizeRequestHeaders(r)
req.URL = targetURL
if req.URL.Scheme == "https" || ui.overrideHostHeader {
// Override req.Host only for https requests, since https server verifies hostnames during TLS handshake,
// so it expects the targetURL.Host in the request.
// There is no need in overriding the req.Host for http requests, since it is expected that backend server
// may properly process queries with the original req.Host.
req.Host = targetURL.Host
}
req.URL = targetURL
req.Header.Set("User-Agent", "vmauth")
updateHeadersByConfig(req.Header, hc.RequestHeaders)
var trivialRetries int
if hc.KeepOriginalHost == nil || !*hc.KeepOriginalHost {
if host := getHostHeader(hc.RequestHeaders); host != "" {
req.Host = host
} else {
req.Host = targetURL.Host
}
}
rtb, rtbOK := req.Body.(*readTrackingBody)
again:
res, err := ui.rt.RoundTrip(req)
if err != nil {
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
@@ -260,7 +267,7 @@ again:
// Timed out request must be counted as errors, since this usually means that the backend is slow.
ui.backendErrors.Inc()
}
return true
return true, false
}
if !rtbOK || !rtb.canRetry() {
// Request body cannot be re-sent to another backend. Return the error to the client then.
@@ -270,19 +277,19 @@ again:
}
httpserver.Errorf(w, r, "%s", err)
ui.backendErrors.Inc()
return true
return true, false
}
// one time retry trivial network errors, such as proxy idle timeout misconfiguration or socket close by OS
if (netutil.IsTrivialNetworkError(err) || errors.Is(err, io.EOF)) && trivialRetries < 1 {
trivialRetries++
goto again
if netutil.IsTrivialNetworkError(err) {
// Retry request at the same backend on trivial network errors, such as proxy idle timeout misconfiguration or socket close by OS
return false, true
}
// Retry the request if its body wasn't read yet. This usually means that the backend isn't reachable.
remoteAddr := httpserver.GetQuotedRemoteAddr(r)
// NOTE: do not use httpserver.GetRequestURI
// it explicitly reads request body, which may fail retries.
logger.Warnf("remoteAddr: %s; requestURI: %s; retrying the request to %s because of response error: %s", remoteAddr, req.URL, targetURL, err)
return false
return false, false
}
if slices.Contains(retryStatusCodes, res.StatusCode) {
_ = res.Body.Close()
@@ -296,7 +303,7 @@ again:
}
httpserver.Errorf(w, r, "%s", err)
ui.backendErrors.Inc()
return true
return true, false
}
// Retry requests at other backends if it matches retryStatusCodes.
// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4893
@@ -305,7 +312,7 @@ again:
// it explicitly reads request body, which may fail retries.
logger.Warnf("remoteAddr: %s; requestURI: %s; retrying the request to %s because response status code=%d belongs to retry_status_codes=%d",
remoteAddr, req.URL, targetURL, res.StatusCode, retryStatusCodes)
return false
return false, false
}
removeHopHeaders(res.Header)
copyHeader(w.Header(), res.Header)
@@ -321,9 +328,9 @@ again:
remoteAddr := httpserver.GetQuotedRemoteAddr(r)
requestURI := httpserver.GetRequestURI(r)
logger.Warnf("remoteAddr: %s; requestURI: %s; error when proxying response body from %s: %s", remoteAddr, requestURI, targetURL, err)
return true
return true, false
}
return true
return true, false
}
var copyBufPool bytesutil.ByteBufferPool
@@ -336,12 +343,21 @@ func copyHeader(dst, src http.Header) {
}
}
func updateHeadersByConfig(headers http.Header, config []*Header) {
for _, h := range config {
func getHostHeader(headers []*Header) string {
for _, h := range headers {
if h.Name == "Host" {
return h.Value
}
}
return ""
}
func updateHeadersByConfig(dst http.Header, src []*Header) {
for _, h := range src {
if h.Value == "" {
headers.Del(h.Name)
dst.Del(h.Name)
} else {
headers.Set(h.Name, h.Value)
dst.Set(h.Name, h.Value)
}
}
}
@@ -481,6 +497,11 @@ See the docs at https://docs.victoriametrics.com/vmauth/ .
flagutil.Usage(s)
}
func handleMissingAuthorizationError(w http.ResponseWriter) {
w.Header().Set("WWW-Authenticate", `Basic realm="Restricted"`)
http.Error(w, "missing 'Authorization' request header", http.StatusUnauthorized)
}
func handleConcurrencyLimitError(w http.ResponseWriter, r *http.Request, err error) {
w.Header().Add("Retry-After", "10")
err = &httpserver.ErrorWithStatusCode{
@@ -490,85 +511,124 @@ func handleConcurrencyLimitError(w http.ResponseWriter, r *http.Request, err err
httpserver.Errorf(w, r, "%s", err)
}
// readTrackingBody must be obtained via getReadTrackingBody()
type readTrackingBody struct {
// maxBodySize is the maximum body size to cache in buf.
//
// Bigger bodies cannot be retried.
maxBodySize int
// r contains reader for initial data reading
r io.ReadCloser
// buf is a buffer for data read from r. Buf size is limited by maxRequestBodySizeToRetry.
// If more than maxRequestBodySizeToRetry is read from r, then cannotRetry is set to true.
// buf is a buffer for data read from r. Buf size is limited by maxBodySize.
// If more than maxBodySize is read from r, then cannotRetry is set to true.
buf []byte
// cannotRetry is set to true when more than maxRequestBodySizeToRetry are read from r.
// readBuf points to the cached data at buf, which must be read in the next call to Read().
readBuf []byte
// cannotRetry is set to true when more than maxBodySize bytes are read from r.
// In this case the read data cannot fit buf, so it cannot be re-read from buf.
cannotRetry bool
// bufComplete is set to true when buf contains complete request body read from r.
bufComplete bool
// offset is an offset at buf for the next data read if needReadBuf is set to true.
offset int
}
// Read implements io.Reader interface
// tracks body reading requests
func (rtb *readTrackingBody) reset() {
rtb.maxBodySize = 0
rtb.r = nil
rtb.buf = rtb.buf[:0]
rtb.readBuf = nil
rtb.cannotRetry = false
rtb.bufComplete = false
}
func getReadTrackingBody(r io.ReadCloser, maxBodySize int) *readTrackingBody {
v := readTrackingBodyPool.Get()
if v == nil {
v = &readTrackingBody{}
}
rtb := v.(*readTrackingBody)
if maxBodySize < 0 {
maxBodySize = 0
}
rtb.maxBodySize = maxBodySize
if r == nil {
// This is GET request without request body
r = (*zeroReader)(nil)
}
rtb.r = r
return rtb
}
type zeroReader struct{}
func (r *zeroReader) Read(_ []byte) (int, error) {
return 0, io.EOF
}
func (r *zeroReader) Close() error {
return nil
}
func putReadTrackingBody(rtb *readTrackingBody) {
rtb.reset()
readTrackingBodyPool.Put(rtb)
}
var readTrackingBodyPool sync.Pool
// Read implements io.Reader interface.
func (rtb *readTrackingBody) Read(p []byte) (int, error) {
if rtb.offset < len(rtb.buf) {
if rtb.cannotRetry {
return 0, fmt.Errorf("cannot retry reading data from buf")
}
nb := copy(p, rtb.buf[rtb.offset:])
rtb.offset += nb
if rtb.bufComplete {
if rtb.offset == len(rtb.buf) {
return nb, io.EOF
}
return nb, nil
}
if nb < len(p) {
nr, err := rtb.readFromStream(p[nb:])
return nb + nr, err
}
return nb, nil
if len(rtb.readBuf) > 0 {
n := copy(p, rtb.readBuf)
rtb.readBuf = rtb.readBuf[n:]
return n, nil
}
if rtb.bufComplete {
return 0, io.EOF
}
return rtb.readFromStream(p)
}
func (rtb *readTrackingBody) readFromStream(p []byte) (int, error) {
if rtb.r == nil {
return 0, fmt.Errorf("cannot read data after closing the reader")
if rtb.bufComplete {
return 0, io.EOF
}
return 0, fmt.Errorf("cannot read client request body after closing client reader")
}
n, err := rtb.r.Read(p)
if rtb.cannotRetry {
return n, err
}
if rtb.offset+n > maxRequestBodySizeToRetry.IntN() {
rtb.cannotRetry = true
}
if n > 0 {
rtb.offset += n
rtb.buf = append(rtb.buf, p[:n]...)
}
if err != nil {
if err == io.EOF {
rtb.bufComplete = true
return n, err
}
if len(rtb.buf)+n > rtb.maxBodySize {
rtb.cannotRetry = true
return n, err
}
return n, nil
rtb.buf = append(rtb.buf, p[:n]...)
if err == io.EOF {
rtb.bufComplete = true
}
return n, err
}
func (rtb *readTrackingBody) canRetry() bool {
return !rtb.cannotRetry
if rtb.cannotRetry {
return false
}
if rtb.bufComplete {
return true
}
return rtb.r != nil
}
// Close implements io.Closer interface.
func (rtb *readTrackingBody) Close() error {
rtb.offset = 0
if !rtb.cannotRetry {
rtb.readBuf = rtb.buf
} else {
rtb.readBuf = nil
}
// Close rtb.r only if the request body is completely read or if it is too big.
// http.Roundtrip performs body.Close call even without any Read calls,
@@ -584,38 +644,3 @@ func (rtb *readTrackingBody) Close() error {
return nil
}
var readTrackingBodyPool sync.Pool
func getReadTrackingBody(origin io.ReadCloser, b int) *readTrackingBody {
bufSize := 1024
if b > 0 && b < maxRequestBodySizeToRetry.IntN() {
bufSize = b
}
v := readTrackingBodyPool.Get()
if v == nil {
v = &readTrackingBody{
buf: make([]byte, 0, bufSize),
}
}
rtb := v.(*readTrackingBody)
rtb.r = origin
if bufSize > cap(rtb.buf) {
rtb.buf = make([]byte, 0, bufSize)
}
return rtb
}
func putReadTrackingBody(rtb *readTrackingBody) {
if rtb.r != nil {
_ = rtb.r.Close()
}
rtb.r = nil
rtb.buf = rtb.buf[:0]
rtb.offset = 0
rtb.cannotRetry = false
rtb.bufComplete = false
readTrackingBodyPool.Put(rtb)
}

View File

@@ -2,18 +2,520 @@ package main
import (
"bytes"
"fmt"
"io"
"net"
"net/http"
"net/http/httptest"
"strings"
"sync/atomic"
"testing"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/netutil"
)
func TestReadTrackingBodyRetrySuccess(t *testing.T) {
f := func(s string) {
func TestRequestHandler(t *testing.T) {
f := func(cfgStr, requestURL string, backendHandler http.HandlerFunc, responseExpected string) {
t.Helper()
rtb := &readTrackingBody{
r: io.NopCloser(bytes.NewBufferString(s)),
ts := httptest.NewServer(backendHandler)
defer ts.Close()
cfgStr = strings.ReplaceAll(cfgStr, "{BACKEND}", ts.URL)
responseExpected = strings.ReplaceAll(responseExpected, "{BACKEND}", ts.URL)
cfgOrigP := authConfigData.Load()
if _, err := reloadAuthConfigData([]byte(cfgStr)); err != nil {
t.Fatalf("cannot load config data: %s", err)
}
defer func() {
cfgOrig := []byte("unauthorized_user:\n url_prefix: http://foo/bar")
if cfgOrigP != nil {
cfgOrig = *cfgOrigP
}
_, err := reloadAuthConfigData(cfgOrig)
if err != nil {
t.Fatalf("cannot load the original config: %s", err)
}
}()
r, err := http.NewRequest(http.MethodGet, requestURL, nil)
if err != nil {
t.Fatalf("cannot initialize http request: %s", err)
}
r.RequestURI = r.URL.RequestURI()
r.RemoteAddr = "42.2.3.84:6789"
r.Header.Set("X-Forwarded-For", "12.34.56.78")
r.Header.Set("Connection", "Some-Header,Other-Header")
r.Header.Set("Some-Header", "foobar")
r.Header.Set("Pass-Header", "abc")
w := &fakeResponseWriter{}
if !requestHandler(w, r) {
t.Fatalf("unexpected false is returned from requestHandler")
}
response := w.getResponse()
response = strings.ReplaceAll(response, "\r\n", "\n")
response = strings.TrimSpace(response)
responseExpected = strings.TrimSpace(responseExpected)
if response != responseExpected {
t.Fatalf("unexpected response\ngot\n%s\nwant\n%s", response, responseExpected)
}
}
// regular url_prefix
cfgStr := `
unauthorized_user:
url_prefix: {BACKEND}/foo?bar=baz`
requestURL := "http://some-host.com/abc/def?some_arg=some_value"
backendHandler := func(w http.ResponseWriter, r *http.Request) {
h := w.Header()
h.Set("Connection", "close")
h.Set("Foo", "bar")
var bb bytes.Buffer
if err := r.Header.Write(&bb); err != nil {
panic(fmt.Errorf("unexpected error when marshaling headers: %w", err))
}
fmt.Fprintf(w, "requested_url=http://%s%s\n%s", r.Host, r.URL, bb.String())
}
responseExpected := `
statusCode=200
Foo: bar
requested_url={BACKEND}/foo/abc/def?bar=baz&some_arg=some_value
Pass-Header: abc
User-Agent: vmauth
X-Forwarded-For: 12.34.56.78, 42.2.3.84`
f(cfgStr, requestURL, backendHandler, responseExpected)
// keep_original_host
cfgStr = `
unauthorized_user:
url_prefix: "{BACKEND}/foo?bar=baz"
keep_original_host: true`
requestURL = "http://some-host.com/abc/def"
backendHandler = func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "requested_url=http://%s%s", r.Host, r.URL)
}
responseExpected = `
statusCode=200
requested_url=http://some-host.com/foo/abc/def?bar=baz`
f(cfgStr, requestURL, backendHandler, responseExpected)
// override user-agent header
cfgStr = `
unauthorized_user:
url_prefix: "{BACKEND}/foo?bar=baz"
headers:
- "User-Agent: foobar"`
requestURL = "http://some-host.com/abc/def"
backendHandler = func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "requested_url=http://%s%s\nUser-Agent=%s", r.Host, r.URL, r.Header.Get("User-Agent"))
}
responseExpected = `
statusCode=200
requested_url={BACKEND}/foo/abc/def?bar=baz
User-Agent=foobar`
f(cfgStr, requestURL, backendHandler, responseExpected)
// delete user-agent header
cfgStr = `
unauthorized_user:
url_prefix: "{BACKEND}/foo?bar=baz"
headers:
- "User-Agent:"`
requestURL = "http://some-host.com/abc/def"
backendHandler = func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "requested_url=http://%s%s\nUser-Agent=%s", r.Host, r.URL, r.Header.Get("User-Agent"))
}
responseExpected = `
statusCode=200
requested_url={BACKEND}/foo/abc/def?bar=baz
User-Agent=Go-http-client/1.1`
f(cfgStr, requestURL, backendHandler, responseExpected)
// override request host with non-empty host
cfgStr = `
unauthorized_user:
url_prefix: "{BACKEND}/foo?bar=baz"
headers:
- "Host: other-host:12345"
- "abc:"`
requestURL = "http://some-host.com/abc/def"
backendHandler = func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "requested_url=http://%s%s", r.Host, r.URL)
}
responseExpected = `
statusCode=200
requested_url=http://other-host:12345/foo/abc/def?bar=baz`
f(cfgStr, requestURL, backendHandler, responseExpected)
// override request host with empty host
cfgStr = `
unauthorized_user:
url_prefix: "{BACKEND}/foo?bar=baz"
headers:
- "Host:"`
requestURL = "http://some-host.com/abc/def"
backendHandler = func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "requested_url=http://%s%s", r.Host, r.URL)
}
responseExpected = `
statusCode=200
requested_url={BACKEND}/foo/abc/def?bar=baz`
f(cfgStr, requestURL, backendHandler, responseExpected)
// /-/reload handler failure
origAuthKey := reloadAuthKey.Get()
if err := reloadAuthKey.Set("secret"); err != nil {
t.Fatalf("unexpected error: %s", err)
}
cfgStr = `
unauthorized_user:
url_prefix: "{BACKEND}/foo"`
requestURL = "http://some-host.com/-/reload"
backendHandler = func(_ http.ResponseWriter, _ *http.Request) {
panic(fmt.Errorf("backend handler shouldn't be called"))
}
responseExpected = `
statusCode=401
The provided authKey doesn't match -reloadAuthKey`
f(cfgStr, requestURL, backendHandler, responseExpected)
if err := reloadAuthKey.Set(origAuthKey); err != nil {
t.Fatalf("unexpected error: %s", err)
}
// missing authorization
cfgStr = `
users:
- username: foo
url_prefix: "{BACKEND}/bar"`
requestURL = "http://some-host.com/a/b"
backendHandler = func(_ http.ResponseWriter, _ *http.Request) {
panic(fmt.Errorf("backend handler shouldn't be called"))
}
responseExpected = `
statusCode=401
Www-Authenticate: Basic realm="Restricted"
missing 'Authorization' request header`
f(cfgStr, requestURL, backendHandler, responseExpected)
// incorrect authorization
cfgStr = `
users:
- username: foo
password: secret
url_prefix: "{BACKEND}/bar"`
requestURL = "http://foo:invalid-secret@some-host.com/a/b"
backendHandler = func(_ http.ResponseWriter, _ *http.Request) {
panic(fmt.Errorf("backend handler shouldn't be called"))
}
responseExpected = `
statusCode=401
Unauthorized`
f(cfgStr, requestURL, backendHandler, responseExpected)
// incorrect authorization with logging invalid auth tokens
origLogInvalidAuthTokens := *logInvalidAuthTokens
*logInvalidAuthTokens = true
cfgStr = `
users:
- username: foo
password: secret
url_prefix: "{BACKEND}/bar"`
requestURL = "http://foo:invalid-secret@some-host.com/a/b?c=d"
backendHandler = func(_ http.ResponseWriter, _ *http.Request) {
panic(fmt.Errorf("backend handler shouldn't be called"))
}
responseExpected = `
statusCode=401
remoteAddr: "42.2.3.84:6789, X-Forwarded-For: 12.34.56.78"; requestURI: /a/b?c=d; cannot authorize request with auth tokens ["http_auth:Basic Zm9vOmludmFsaWQtc2VjcmV0"]`
f(cfgStr, requestURL, backendHandler, responseExpected)
*logInvalidAuthTokens = origLogInvalidAuthTokens
// correct authorization
cfgStr = `
users:
- username: foo
password: secret
url_prefix: "{BACKEND}/bar"`
requestURL = "http://foo:secret@some-host.com/a/b"
backendHandler = func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "requested_url=http://%s%s", r.Host, r.URL)
}
responseExpected = `
statusCode=200
requested_url={BACKEND}/bar/a/b`
f(cfgStr, requestURL, backendHandler, responseExpected)
// verify how path cleanup works
cfgStr = `
unauthorized_user:
url_prefix: {BACKEND}/foo?bar=baz`
requestURL = "http://some-host.com/../../a//.///bar/"
backendHandler = func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "requested_url=http://%s%s", r.Host, r.URL)
}
responseExpected = `
statusCode=200
requested_url={BACKEND}/foo/a/bar/?bar=baz`
f(cfgStr, requestURL, backendHandler, responseExpected)
// verify how path cleanup works for url without path
cfgStr = `
unauthorized_user:
url_prefix: {BACKEND}/foo?bar=baz`
requestURL = "http://some-host.com/"
backendHandler = func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "requested_url=http://%s%s", r.Host, r.URL)
}
responseExpected = `
statusCode=200
requested_url={BACKEND}/foo?bar=baz`
f(cfgStr, requestURL, backendHandler, responseExpected)
// verify how path cleanup works for url without path if url_prefix path ends with /
cfgStr = `
unauthorized_user:
url_prefix: {BACKEND}/foo/?bar=baz`
requestURL = "http://some-host.com/"
backendHandler = func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "requested_url=http://%s%s", r.Host, r.URL)
}
responseExpected = `
statusCode=200
requested_url={BACKEND}/foo/?bar=baz`
f(cfgStr, requestURL, backendHandler, responseExpected)
// verify how path cleanup works for url without path and the url_prefix without path prefix
cfgStr = `
unauthorized_user:
url_prefix: {BACKEND}/?bar=baz`
requestURL = "http://some-host.com/"
backendHandler = func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "requested_url=http://%s%s", r.Host, r.URL)
}
responseExpected = `
statusCode=200
requested_url={BACKEND}/?bar=baz`
f(cfgStr, requestURL, backendHandler, responseExpected)
// verify routing to default_url
cfgStr = `
unauthorized_user:
url_map:
- src_paths: ["/foo/.+"]
url_prefix: {BACKEND}/x-foo/
default_url: {BACKEND}/404.html`
requestURL = "http://some-host.com/abc?de=fg"
backendHandler = func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "requested_url=http://%s%s", r.Host, r.URL)
}
responseExpected = `
statusCode=200
requested_url={BACKEND}/404.html?request_path=http%3A%2F%2Fsome-host.com%2Fabc%3Fde%3Dfg`
f(cfgStr, requestURL, backendHandler, responseExpected)
// verify routing to default url_prefix
cfgStr = `
unauthorized_user:
url_map:
- src_paths: ["/foo/.+"]
url_prefix: {BACKEND}/x-foo/
url_prefix: {BACKEND}/default`
requestURL = "http://some-host.com/abc?de=fg"
backendHandler = func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "requested_url=http://%s%s", r.Host, r.URL)
}
responseExpected = `
statusCode=200
requested_url={BACKEND}/default/abc?de=fg`
f(cfgStr, requestURL, backendHandler, responseExpected)
// missing default_url and default url_prefix for unauthorized user
cfgStr = `
unauthorized_user:
url_map:
- src_paths: ["/foo/.+"]
url_prefix: {BACKEND}/x-foo/`
requestURL = "http://some-host.com/abc?de=fg"
backendHandler = func(_ http.ResponseWriter, _ *http.Request) {
panic(fmt.Errorf("backend handler shouldn't be called"))
}
responseExpected = `
statusCode=400
remoteAddr: "42.2.3.84:6789, X-Forwarded-For: 12.34.56.78"; requestURI: /abc?de=fg; missing route for http://some-host.com/abc?de=fg`
f(cfgStr, requestURL, backendHandler, responseExpected)
// missing default_url and default url_prefix for unauthorized user when there are configs for authorized users
cfgStr = `
users:
- username: some-user
url_map:
- src_paths: ["/foo/.+"]
url_prefix: {BACKEND}/x-foo/
unauthorized_user:
url_map:
- src_paths: ["/abc/.*"]
url_prefix: {BACKEND}/x-bar`
requestURL = "http://some-host.com/abc?de=fg"
backendHandler = func(_ http.ResponseWriter, _ *http.Request) {
panic(fmt.Errorf("backend handler shouldn't be called"))
}
responseExpected = `
statusCode=401
Www-Authenticate: Basic realm="Restricted"
missing 'Authorization' request header`
f(cfgStr, requestURL, backendHandler, responseExpected)
// all the backend_urls are unavailable for unauthorized user
cfgStr = `
unauthorized_user:
url_map:
- src_paths: ["/foo/.*"]
url_prefix:
- http://127.0.0.1:1/
- http://127.0.0.1:2/`
requestURL = "http://some-host.com/foo/?de=fg"
backendHandler = func(_ http.ResponseWriter, _ *http.Request) {
panic(fmt.Errorf("backend handler shouldn't be called"))
}
responseExpected = `
statusCode=502
remoteAddr: "42.2.3.84:6789, X-Forwarded-For: 12.34.56.78"; requestURI: /foo/?de=fg; all the 2 backends for the user "" are unavailable`
f(cfgStr, requestURL, backendHandler, responseExpected)
// all the backend_urls are unavailable for authorized user
cfgStr = `
users:
- username: some-user
url_map:
- src_paths: ["/foo/.*"]
url_prefix:
- http://127.0.0.1:1/
- http://127.0.0.1:2/`
requestURL = "http://some-user@some-host.com/foo/?de=fg"
backendHandler = func(_ http.ResponseWriter, _ *http.Request) {
panic(fmt.Errorf("backend handler shouldn't be called"))
}
responseExpected = `
statusCode=502
remoteAddr: "42.2.3.84:6789, X-Forwarded-For: 12.34.56.78"; requestURI: /foo/?de=fg; all the 2 backends for the user "some-user" are unavailable`
f(cfgStr, requestURL, backendHandler, responseExpected)
// zero discovered backend IPs
customResolver := &fakeResolver{
Resolver: &net.Resolver{},
lookupIPAddrResults: map[string][]net.IPAddr{
"some-addr": {},
},
}
origResolver := netutil.Resolver
netutil.Resolver = customResolver
cfgStr = `
unauthorized_user:
url_prefix: ['http://some-addr:1234/foo/bar']
discover_backend_ips: true`
requestURL = "http://abc.com/def/?de=fg"
backendHandler = func(_ http.ResponseWriter, _ *http.Request) {
panic(fmt.Errorf("backend handler shouldn't be called"))
}
responseExpected = `
statusCode=502
remoteAddr: "42.2.3.84:6789, X-Forwarded-For: 12.34.56.78"; requestURI: /def/?de=fg; all the 0 backends for the user "" are unavailable`
f(cfgStr, requestURL, backendHandler, responseExpected)
netutil.Resolver = origResolver
// retry_status_codes failure
var retries atomic.Int64
cfgStr = `
unauthorized_user:
url_prefix: ['{BACKEND}/path1', '{BACKEND}/path2']
retry_status_codes: [500, 502]`
requestURL = "http://some-host.com/foo/?de=fg"
backendHandler = func(w http.ResponseWriter, _ *http.Request) {
retries.Add(1)
w.WriteHeader(500)
}
responseExpected = `
statusCode=502
remoteAddr: "42.2.3.84:6789, X-Forwarded-For: 12.34.56.78"; requestURI: /foo/?de=fg; all the 2 backends for the user "" are unavailable`
f(cfgStr, requestURL, backendHandler, responseExpected)
if n := retries.Load(); n != 2 {
t.Fatalf("unexpected number of retries; got %d; want 2", n)
}
// retry_status_codes success
retries.Store(0)
cfgStr = `
unauthorized_user:
url_prefix: ['{BACKEND}/path1', '{BACKEND}/path2']
retry_status_codes: [500, 502]`
requestURL = "http://some-host.com/foo/?de=fg"
backendHandler = func(w http.ResponseWriter, r *http.Request) {
if n := retries.Add(1); n < 2 {
w.WriteHeader(500)
return
}
fmt.Fprintf(w, "requested_url=http://%s%s", r.Host, r.URL)
}
responseExpected = `
statusCode=200
requested_url={BACKEND}/path2/foo/?de=fg`
f(cfgStr, requestURL, backendHandler, responseExpected)
if n := retries.Load(); n != 2 {
t.Fatalf("unexpected number of retries; got %d; want 2", n)
}
}
type fakeResponseWriter struct {
h http.Header
bb bytes.Buffer
}
func (w *fakeResponseWriter) getResponse() string {
return w.bb.String()
}
func (w *fakeResponseWriter) Header() http.Header {
if w.h == nil {
w.h = http.Header{}
}
return w.h
}
func (w *fakeResponseWriter) Write(p []byte) (int, error) {
return w.bb.Write(p)
}
func (w *fakeResponseWriter) WriteHeader(statusCode int) {
fmt.Fprintf(&w.bb, "statusCode=%d\n", statusCode)
if w.h == nil {
return
}
err := w.h.WriteSubset(&w.bb, map[string]bool{
"Content-Length": true,
"Content-Type": true,
"Date": true,
"X-Content-Type-Options": true,
})
if err != nil {
panic(fmt.Errorf("cannot marshal headers: %s", err))
}
}
func TestReadTrackingBody_RetrySuccess(t *testing.T) {
f := func(s string, maxBodySize int) {
t.Helper()
rtb := getReadTrackingBody(io.NopCloser(bytes.NewBufferString(s)), maxBodySize)
defer putReadTrackingBody(rtb)
if !rtb.canRetry() {
t.Fatalf("canRetry() must return true")
t.Fatalf("canRetry() must return true before reading anything")
}
for i := 0; i < 5; i++ {
data, err := io.ReadAll(rtb)
@@ -32,41 +534,99 @@ func TestReadTrackingBodyRetrySuccess(t *testing.T) {
}
}
f("")
f("foo")
f("foobar")
f(newTestString(maxRequestBodySizeToRetry.IntN()))
f("", 0)
f("", -1)
f("", 100)
f("foo", 100)
f("foobar", 100)
f(newTestString(1000), 1000)
}
func TestReadTrackingBodyRetryFailure(t *testing.T) {
f := func(s string) {
func TestReadTrackingBody_RetrySuccessPartialRead(t *testing.T) {
f := func(s string, maxBodySize int) {
t.Helper()
rtb := &readTrackingBody{
r: io.NopCloser(bytes.NewBufferString(s)),
// Check the case with partial read
rtb := getReadTrackingBody(io.NopCloser(bytes.NewBufferString(s)), maxBodySize)
defer putReadTrackingBody(rtb)
for i := 0; i < len(s); i++ {
buf := make([]byte, i)
n, err := io.ReadFull(rtb, buf)
if err != nil {
t.Fatalf("unexpected error when reading %d bytes: %s", i, err)
}
if n != i {
t.Fatalf("unexpected number of bytes read; got %d; want %d", n, i)
}
if string(buf) != s[:i] {
t.Fatalf("unexpected data read with the length %d\ngot\n%s\nwant\n%s", i, buf, s[:i])
}
if err := rtb.Close(); err != nil {
t.Fatalf("unexpected error when closing reader after reading %d bytes", i)
}
if !rtb.canRetry() {
t.Fatalf("canRetry() must return true after closing the reader after reading %d bytes", i)
}
}
data, err := io.ReadAll(rtb)
if err != nil {
t.Fatalf("unexpected error when reading all the data: %s", err)
}
if string(data) != s {
t.Fatalf("unexpected data read\ngot\n%s\nwant\n%s", data, s)
}
if err := rtb.Close(); err != nil {
t.Fatalf("unexpected error when closing readTrackingBody: %s", err)
}
if !rtb.canRetry() {
t.Fatalf("canRetry() must return true")
t.Fatalf("canRetry() must return true after closing the reader after reading all the input")
}
}
f("", 0)
f("", -1)
f("", 100)
f("foo", 100)
f("foobar", 100)
f(newTestString(1000), 1000)
}
func TestReadTrackingBody_RetryFailureTooBigBody(t *testing.T) {
f := func(s string, maxBodySize int) {
t.Helper()
rtb := getReadTrackingBody(io.NopCloser(bytes.NewBufferString(s)), maxBodySize)
defer putReadTrackingBody(rtb)
if !rtb.canRetry() {
t.Fatalf("canRetry() must return true before reading anything")
}
buf := make([]byte, 1)
n, err := rtb.Read(buf)
n, err := io.ReadFull(rtb, buf)
if err != nil {
t.Fatalf("unexpected error when reading a single byte: %s", err)
}
if n != 1 {
t.Fatalf("unexpected number of bytes read; got %d; want 1", n)
}
if !rtb.canRetry() {
t.Fatalf("canRetry() must return true after reading one byte")
}
data, err := io.ReadAll(rtb)
if err != nil {
t.Fatalf("unexpected error when reading all the data: %s", err)
}
if string(buf)+string(data) != s {
t.Fatalf("unexpected data read\ngot\n%s\nwant\n%s", string(buf)+string(data), s)
dataRead := string(buf) + string(data)
if dataRead != s {
t.Fatalf("unexpected data read\ngot\n%s\nwant\n%s", dataRead, s)
}
if err := rtb.Close(); err != nil {
t.Fatalf("unexpected error when closing readTrackingBody: %s", err)
}
if rtb.canRetry() {
t.Fatalf("canRetry() must return false")
t.Fatalf("canRetry() must return false after closing the reader")
}
data, err = io.ReadAll(rtb)
@@ -78,132 +638,55 @@ func TestReadTrackingBodyRetryFailure(t *testing.T) {
}
}
f(newTestString(maxRequestBodySizeToRetry.IntN() + 1))
f(newTestString(2 * maxRequestBodySizeToRetry.IntN()))
const maxBodySize = 1000
f(newTestString(maxBodySize+1), maxBodySize)
f(newTestString(2*maxBodySize), maxBodySize)
}
// request body not over maxRequestBodySizeToRetry
// 1. When writing data downstream, buf only caches part of the data because the downstream connection is disconnected.
// 2. retry request: because buf caches some data, first read buf and then read stream when retrying
// 3. retry request: the data has been read to buf in the second step. if the request fails, retry to read all buf later.
func TestRetryReadSuccessAfterPartialRead(t *testing.T) {
f := func(s string) {
rtb := &readTrackingBody{
r: io.NopCloser(bytes.NewBufferString(s)),
buf: make([]byte, 0, len(s)),
}
func TestReadTrackingBody_RetryFailureZeroOrNegativeMaxBodySize(t *testing.T) {
f := func(s string, maxBodySize int) {
t.Helper()
var data []byte
var err error
halfSize := len(s) / 2
if halfSize == 0 {
halfSize = 100
}
buf := make([]byte, halfSize)
var n int
rtb := getReadTrackingBody(io.NopCloser(bytes.NewBufferString(s)), maxBodySize)
defer putReadTrackingBody(rtb)
// read part of the data
n, err = rtb.Read(buf[:])
data = append(data, buf[:n]...)
if err != nil && err != io.EOF {
t.Fatalf("unexpected error: %s", err)
if !rtb.canRetry() {
t.Fatalf("canRetry() must return true before reading anything")
}
data, err := io.ReadAll(rtb)
if err != nil {
t.Fatalf("unexpected error when reading all the data: %s", err)
}
if string(data) != s {
t.Fatalf("unexpected data read\ngot\n%s\nwant\n%s", data, s)
}
// request failed when output stream is closed (eg: server connection reset)
// would close the reader
if err := rtb.Close(); err != nil {
t.Fatalf("unexpected error when closing readTrackingBody: %s", err)
}
if !rtb.canRetry() {
t.Fatalf("canRetry() must return true")
}
// retry read (read buf + remaining data)
data = data[:0]
err = nil
for err == nil {
n, err = rtb.Read(buf[:])
data = append(data, buf[:n]...)
}
if err != io.EOF {
t.Fatalf("unexpected error: %s", err)
}
if string(data) != s {
t.Fatalf("unexpected data read; got\n%s\nwant\n%s", data, s)
}
// cannotRetry return false
// because the request data is not over maxRequestBodySizeToRetry limit
if !rtb.canRetry() {
t.Fatalf("canRetry() must return true")
}
}
f("")
f("foo")
f("foobar")
f(newTestString(maxRequestBodySizeToRetry.IntN()))
}
// request body over maxRequestBodySizeToRetry
// 1. When writing data downstream, buf only caches part of the data because the downstream connection is disconnected.
// 2. retry request: because buf caches some data, first read buf and then read stream when retrying
// 3. retry request: the data has been read to buf in the second step. if the request fails, retry to read all buf later.
func TestRetryReadSuccessAfterPartialReadAndCannotRetryAgain(t *testing.T) {
f := func(s string) {
rtb := &readTrackingBody{
r: io.NopCloser(bytes.NewBufferString(s)),
buf: make([]byte, 0, len(s)),
}
var data []byte
var err error
halfSize := len(s) / 2
if halfSize == 0 {
halfSize = 100
}
buf := make([]byte, halfSize)
var n int
// read part of the data
n, err = rtb.Read(buf[:])
data = append(data, buf[:n]...)
if err != nil && err != io.EOF {
t.Fatalf("unexpected error: %s", err)
}
// request failed when output stream is closed (eg: server connection reset)
if err := rtb.Close(); err != nil {
t.Fatalf("unexpected error when closing readTrackingBody: %s", err)
}
if !rtb.canRetry() {
t.Fatalf("canRetry() must return true")
}
// retry read (read buf + remaining data)
data = data[:0]
err = nil
for err == nil {
n, err = rtb.Read(buf[:])
data = append(data, buf[:n]...)
}
if err != io.EOF {
t.Fatalf("unexpected error: %s", err)
}
if string(data) != s {
t.Fatalf("unexpected data read; got\n%s\nwant\n%s", data, s)
}
// cannotRetry returns true
// because the request data is over maxRequestBodySizeToRetry limit
if rtb.canRetry() {
t.Fatalf("canRetry() must return false")
t.Fatalf("canRetry() must return false after closing the reader")
}
data, err = io.ReadAll(rtb)
if err == nil {
t.Fatalf("expecting non-nil error")
}
if len(data) != 0 {
t.Fatalf("unexpected non-empty data read: %q", data)
}
}
f(newTestString(maxRequestBodySizeToRetry.IntN() + 1))
f(newTestString(2 * maxRequestBodySizeToRetry.IntN()))
f("foobar", 0)
f(newTestString(1000), 0)
f("foobar", -1)
f(newTestString(1000), -1)
}
func newTestString(sLen int) string {
return string(make([]byte, sLen))
data := make([]byte, sLen)
for i := range data {
data[i] = byte(i)
}
return string(data)
}

View File

@@ -335,13 +335,62 @@ func TestUserInfoGetBackendURL_SRV(t *testing.T) {
t.Fatalf("cannot initialize urls inside UserInfo: %s", err)
}
f(ui, `/select/0/prometheus/api/v1/query?query=up`, "http://10.6.142.51:8481/select/0/prometheus/api/v1/query?query=up")
f(ui, `/select/0/prometheus/api/v1/query?query=up`, "http://10.6.142.50:8481/select/0/prometheus/api/v1/query?query=up")
f(ui, `/select/0/prometheus/api/v1/query?query=up`, "http://10.6.142.51:8481/select/0/prometheus/api/v1/query?query=up")
f(ui, `/insert/0/prometheus/api/v1/write`, "http://10.6.142.52:8480/insert/0/prometheus/api/v1/write")
// unsuccessful dns resolve
f(ui, `/test`, "http://non-exist-dns-addr/test")
}
func TestUserInfoGetBackendURL_SRVZeroBackends(t *testing.T) {
f := func(ui *UserInfo, requestURI string) {
t.Helper()
u, err := url.Parse(requestURI)
if err != nil {
t.Fatalf("cannot parse %q: %s", requestURI, err)
}
u = normalizeURL(u)
up, _ := ui.getURLPrefixAndHeaders(u, nil)
if up == nil {
t.Fatalf("cannot match available backend: %s", err)
}
bu := up.getBackendURL()
if bu != nil {
t.Fatalf("expecting nil backendURL; got %v", bu)
}
}
customResolver := &fakeResolver{
Resolver: &net.Resolver{},
lookupSRVResults: map[string][]*net.SRV{
"vmselect": {},
},
}
origResolver := netutil.Resolver
netutil.Resolver = customResolver
defer func() {
netutil.Resolver = origResolver
}()
allowed := true
ui := &UserInfo{
URLMaps: []URLMap{
{
SrcPaths: getRegexs([]string{"/select/.+"}),
URLPrefix: mustParseURL("http://srv+vmselect"),
},
},
DiscoverBackendIPs: &allowed,
URLPrefix: mustParseURL("http://non-exist-dns-addr"),
}
if err := ui.initURLs(); err != nil {
t.Fatalf("cannot initialize urls inside UserInfo: %s", err)
}
f(ui, `/select/0/prometheus/api/v1/query?query=up`)
}
func TestCreateTargetURLFailure(t *testing.T) {
f := func(ui *UserInfo, requestURI string) {
t.Helper()

View File

@@ -10,12 +10,6 @@ import (
"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
)
const (
backoffRetries = 10
backoffFactor = 1.8
backoffMinDuration = time.Second * 2
)
// retryableFunc describes call back which will repeat on errors
type retryableFunc func() error
@@ -30,12 +24,22 @@ type Backoff struct {
}
// New initialize backoff object
func New() *Backoff {
return &Backoff{
retries: backoffRetries,
factor: backoffFactor,
minDuration: backoffMinDuration,
func New(retries int, factor float64, minDuration time.Duration) (*Backoff, error) {
if retries <= 0 {
return nil, fmt.Errorf("number of backoff retries must be greater than 0")
}
if factor <= 1 {
return nil, fmt.Errorf("backoff retry factor must be greater than 1")
}
if minDuration <= 0 {
return nil, fmt.Errorf("backoff retry minimum duration must be greater than 0")
}
return &Backoff{
retries: retries,
factor: factor,
minDuration: minDuration,
}, nil
}
// Retry process retries until all attempts are completed

View File

@@ -3,6 +3,7 @@ package backoff
import (
"context"
"fmt"
"strings"
"testing"
"time"
)
@@ -110,3 +111,32 @@ func TestBackoffRetry_Success(t *testing.T) {
resultExpected := 1
f(retryFunc, resultExpected)
}
func TestBackoff_New(t *testing.T) {
f := func(retries int, factor float64, minDuration time.Duration, errExpected string) {
t.Helper()
_, err := New(retries, factor, minDuration)
if err == nil {
if errExpected != "" {
t.Fatalf("expecting non-nil error")
}
return
}
if !strings.Contains(err.Error(), errExpected) {
t.Fatalf("unexpected error: got %q; want %q", err.Error(), errExpected)
}
}
// empty retries
f(0, 1.1, time.Millisecond*10, "retries must be greater than 0")
// empty factor
f(1, 0, time.Millisecond*10, "factor must be greater than 1")
// empty minDuration
f(1, 1.1, 0, "minimum duration must be greater than 0")
// no errors
f(1, 1.1, time.Millisecond*10, "")
}

View File

@@ -56,6 +56,10 @@ const (
vmRateLimit = "vm-rate-limit"
vmInterCluster = "vm-intercluster"
vmBackoffRetries = "vm-backoff-retries"
vmBackoffFactor = "vm-backoff-factor"
vmBackoffMinDuration = "vm-backoff-min-duration"
)
var (
@@ -146,6 +150,21 @@ var (
Usage: "Whether to skip tls verification when connecting to '--vmAddr'",
Value: false,
},
&cli.IntFlag{
Name: vmBackoffRetries,
Value: 10,
Usage: "How many import retries to perform before giving up.",
},
&cli.Float64Flag{
Name: vmBackoffFactor,
Value: 1.8,
Usage: "Factor to multiply the base duration after each failed import retry. Must be greater than 1.0",
},
&cli.DurationFlag{
Name: vmBackoffMinDuration,
Value: time.Second * 2,
Usage: "Minimum duration to wait before the first import retry. Each subsequent import retry will be multiplied by the '--vm-backoff-factor'.",
},
}
)
@@ -430,6 +449,10 @@ const (
vmNativeDstCAFile = "vm-native-dst-ca-file"
vmNativeDstServerName = "vm-native-dst-server-name"
vmNativeDstInsecureSkipVerify = "vm-native-dst-insecure-skip-verify"
vmNativeBackoffRetries = "vm-native-backoff-retries"
vmNativeBackoffFactor = "vm-native-backoff-factor"
vmNativeBackoffMinDuration = "vm-native-backoff-min-duration"
)
var (
@@ -599,6 +622,21 @@ var (
"Non-binary export/import API is less efficient, but supports deduplication if it is configured on vm-native-src-addr side.",
Value: false,
},
&cli.IntFlag{
Name: vmNativeBackoffRetries,
Value: 10,
Usage: "How many export/import retries to perform before giving up.",
},
&cli.Float64Flag{
Name: vmNativeBackoffFactor,
Value: 1.8,
Usage: "Factor to multiply the base duration after each failed export/import retry. Must be greater than 1.0",
},
&cli.DurationFlag{
Name: vmNativeBackoffMinDuration,
Value: time.Second * 2,
Usage: "Minimum duration to wait before the first export/import retry. Each subsequent export/import retry will be multiplied by the '--vm-native-backoff-factor'.",
},
}
)

View File

@@ -68,7 +68,7 @@ func main() {
tr, err := httputils.Transport(addr, certFile, keyFile, caFile, serverName, insecureSkipVerify)
if err != nil {
return fmt.Errorf("failed to create Transport: %s", err)
return fmt.Errorf("failed to create transport for -%s=%q: %s", otsdbAddr, addr, err)
}
oCfg := opentsdb.Config{
Addr: addr,
@@ -90,6 +90,7 @@ func main() {
if err != nil {
return fmt.Errorf("failed to init VM configuration: %s", err)
}
importer, err := vm.NewImporter(ctx, vmCfg)
if err != nil {
return fmt.Errorf("failed to create VM importer: %s", err)
@@ -143,6 +144,7 @@ func main() {
if err != nil {
return fmt.Errorf("failed to init VM configuration: %s", err)
}
importer, err = vm.NewImporter(ctx, vmCfg)
if err != nil {
return fmt.Errorf("failed to create VM importer: %s", err)
@@ -178,7 +180,7 @@ func main() {
tr, err := httputils.Transport(addr, certFile, keyFile, caFile, serverName, insecureSkipVerify)
if err != nil {
return fmt.Errorf("failed to create transport: %s", err)
return fmt.Errorf("failed to create transport for -%s=%q: %s", remoteReadSrcAddr, addr, err)
}
rr, err := remoteread.NewClient(remoteread.Config{
@@ -201,6 +203,7 @@ func main() {
if err != nil {
return fmt.Errorf("failed to init VM configuration: %s", err)
}
importer, err := vm.NewImporter(ctx, vmCfg)
if err != nil {
return fmt.Errorf("failed to create VM importer: %s", err)
@@ -233,6 +236,7 @@ func main() {
if err != nil {
return fmt.Errorf("failed to init VM configuration: %s", err)
}
importer, err = vm.NewImporter(ctx, vmCfg)
if err != nil {
return fmt.Errorf("failed to create VM importer: %s", err)
@@ -272,6 +276,14 @@ func main() {
return fmt.Errorf("flag %q can't be empty", vmNativeFilterMatch)
}
bfRetries := c.Int(vmNativeBackoffRetries)
bfFactor := c.Float64(vmNativeBackoffFactor)
bfMinDuration := c.Duration(vmNativeBackoffMinDuration)
bf, err := backoff.New(bfRetries, bfFactor, bfMinDuration)
if err != nil {
return fmt.Errorf("failed to create backoff object: %s", err)
}
disableKeepAlive := c.Bool(vmNativeDisableHTTPKeepAlive)
var srcExtraLabels []string
@@ -350,7 +362,7 @@ func main() {
ExtraLabels: dstExtraLabels,
HTTPClient: dstHTTPClient,
},
backoff: backoff.New(),
backoff: bf,
cc: c.Int(vmConcurrency),
disablePerMetricRequests: c.Bool(vmNativeDisablePerMetricMigration),
isNative: !c.Bool(vmNativeDisableBinaryProtocol),
@@ -426,7 +438,15 @@ func initConfigVM(c *cli.Context) (vm.Config, error) {
tr, err := httputils.Transport(addr, certFile, keyFile, caFile, serverName, insecureSkipVerify)
if err != nil {
return vm.Config{}, fmt.Errorf("failed to create Transport: %s", err)
return vm.Config{}, fmt.Errorf("failed to create transport for -%s=%q: %s", vmAddr, addr, err)
}
bfRetries := c.Int(vmBackoffRetries)
bfFactor := c.Float64(vmBackoffFactor)
bfMinDuration := c.Duration(vmBackoffMinDuration)
bf, err := backoff.New(bfRetries, bfFactor, bfMinDuration)
if err != nil {
return vm.Config{}, fmt.Errorf("failed to create backoff object: %s", err)
}
return vm.Config{
@@ -442,5 +462,6 @@ func initConfigVM(c *cli.Context) (vm.Config, error) {
RoundDigits: c.Int(vmRoundDigits),
ExtraLabels: c.StringSlice(vmExtraLabel),
RateLimit: c.Int64(vmRateLimit),
Backoff: bf,
}, nil
}

View File

@@ -8,6 +8,7 @@ import (
"github.com/prometheus/prometheus/prompb"
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/backoff"
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/barpool"
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/remoteread"
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/stepper"
@@ -149,6 +150,12 @@ func TestRemoteRead(t *testing.T) {
tt.vmCfg.Addr = remoteWriteServer.URL()
b, err := backoff.New(10, 1.8, time.Second*2)
if err != nil {
t.Fatalf("failed to create backoff: %s", err)
}
tt.vmCfg.Backoff = b
importer, err := vm.NewImporter(ctx, tt.vmCfg)
if err != nil {
t.Fatalf("failed to create VM importer: %s", err)
@@ -308,6 +315,12 @@ func TestSteamRemoteRead(t *testing.T) {
tt.vmCfg.Addr = remoteWriteServer.URL()
b, err := backoff.New(10, 1.8, time.Second*2)
if err != nil {
t.Fatalf("failed to create backoff: %s", err)
}
tt.vmCfg.Backoff = b
importer, err := vm.NewImporter(ctx, tt.vmCfg)
if err != nil {
t.Fatalf("failed to create VM importer: %s", err)

View File

@@ -54,6 +54,8 @@ type Config struct {
// RateLimit defines a data transfer speed in bytes per second.
// Is applied to each worker (see Concurrency) independently.
RateLimit int64
// Backoff defines backoff policy for retries
Backoff *backoff.Backoff
}
// Importer performs insertion of timeseries
@@ -144,7 +146,7 @@ func NewImporter(ctx context.Context, cfg Config) (*Importer, error) {
close: make(chan struct{}),
input: make(chan *TimeSeries, cfg.Concurrency*4),
errors: make(chan *ImportError, cfg.Concurrency),
backoff: backoff.New(),
backoff: cfg.Backoff,
}
if err := im.Ping(); err != nil {
return nil, fmt.Errorf("ping to %q failed: %s", addr, err)

View File

@@ -81,11 +81,16 @@ func TestVMNativeProcessorRun(t *testing.T) {
isSilent = true
defer func() { isSilent = false }()
bf, err := backoff.New(10, 1.8, time.Second*2)
if err != nil {
t.Fatalf("cannot create backoff: %s", err)
}
p := &vmNativeProcessor{
filter: filter,
dst: dstClient,
src: srcClient,
backoff: backoff.New(),
backoff: bf,
cc: 1,
isNative: true,
}

View File

@@ -2,16 +2,23 @@ package common
import (
"sync"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/cgroup"
)
// GetInsertCtx returns InsertCtx from the pool.
//
// Call PutInsertCtx for returning it to the pool.
func GetInsertCtx() *InsertCtx {
if v := insertCtxPool.Get(); v != nil {
return v.(*InsertCtx)
select {
case ctx := <-insertCtxPoolCh:
return ctx
default:
if v := insertCtxPool.Get(); v != nil {
return v.(*InsertCtx)
}
return &InsertCtx{}
}
return &InsertCtx{}
}
// PutInsertCtx returns ctx to the pool.
@@ -19,7 +26,14 @@ func GetInsertCtx() *InsertCtx {
// ctx cannot be used after the call.
func PutInsertCtx(ctx *InsertCtx) {
ctx.Reset(0)
insertCtxPool.Put(ctx)
select {
case insertCtxPoolCh <- ctx:
default:
insertCtxPool.Put(ctx)
}
}
var insertCtxPool sync.Pool
var (
insertCtxPool sync.Pool
insertCtxPoolCh = make(chan *InsertCtx, cgroup.AvailableCPUs())
)

View File

@@ -42,6 +42,7 @@ import (
"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/common"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/opentelemetry/firehose"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/storage"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/stringsutil"
)
var (
@@ -217,6 +218,10 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
addInfluxResponseHeaders(w)
influxutils.WriteDatabaseNames(w)
return true
case "/influx/health":
influxHealthRequests.Inc()
influxutils.WriteHealthCheckResponse(w)
return true
case "/opentelemetry/api/v1/push", "/opentelemetry/v1/metrics":
opentelemetryPushRequests.Inc()
if err := opentelemetry.InsertHandler(r); err != nil {
@@ -343,7 +348,7 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
w.Header().Set("Content-Type", "application/json")
var bb bytesutil.ByteBuffer
promscrape.WriteConfigData(&bb)
fmt.Fprintf(w, `{"status":"success","data":{"yaml":%q}}`, bb.B)
fmt.Fprintf(w, `{"status":"success","data":{"yaml":%s}}`, stringsutil.JSONString(string(bb.B)))
return true
case "/prometheus/-/reload", "/-/reload":
if !httpserver.CheckAuthFlag(w, r, reloadAuthKey) {
@@ -396,7 +401,8 @@ var (
influxWriteRequests = metrics.NewCounter(`vm_http_requests_total{path="/influx/write", protocol="influx"}`)
influxWriteErrors = metrics.NewCounter(`vm_http_request_errors_total{path="/influx/write", protocol="influx"}`)
influxQueryRequests = metrics.NewCounter(`vm_http_requests_total{path="/influx/query", protocol="influx"}`)
influxQueryRequests = metrics.NewCounter(`vm_http_requests_total{path="/influx/query", protocol="influx"}`)
influxHealthRequests = metrics.NewCounter(`vm_http_requests_total{path="/influx/health", protocol="influx"}`)
datadogv1WriteRequests = metrics.NewCounter(`vm_http_requests_total{path="/datadog/api/v1/series", protocol="datadog"}`)
datadogv1WriteErrors = metrics.NewCounter(`vm_http_request_errors_total{path="/datadog/api/v1/series", protocol="datadog"}`)

View File

@@ -115,7 +115,7 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
startTime := time.Now()
defer requestDuration.UpdateDuration(startTime)
tracerEnabled := httputils.GetBool(r, "trace")
qt := querytracer.New(tracerEnabled, r.URL.Path)
qt := querytracer.New(tracerEnabled, "%s", r.URL.Path)
// Limit the number of concurrent queries.
select {

View File

@@ -7,6 +7,7 @@ import (
"sync"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/fs"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/memory"
@@ -14,6 +15,10 @@ import (
"github.com/VictoriaMetrics/metrics"
)
var tmpBufSize = flagutil.NewBytes("search.inmemoryBufSizeBytes", 0, "Size for in-memory data blocks used during processing search requests. "+
"By default, the size is automatically calculated based on available memory. "+
"Adjust this flag value if you observe that vm_tmp_blocks_max_inmemory_file_size_bytes metric constantly shows much higher values than vm_tmp_blocks_inmemory_file_size_bytes. See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6851")
// InitTmpBlocksDir initializes directory to store temporary search results.
//
// It stores data in system-defined temporary directory if tmpDirPath is empty.
@@ -29,6 +34,9 @@ func InitTmpBlocksDir(tmpDirPath string) {
var tmpBlocksDir string
func maxInmemoryTmpBlocksFile() int {
if tmpBufSize.IntN() > 0 {
return tmpBufSize.IntN()
}
mem := memory.Allowed()
maxLen := mem / 1024
if maxLen < 64*1024 {
@@ -40,9 +48,12 @@ func maxInmemoryTmpBlocksFile() int {
return maxLen
}
var _ = metrics.NewGauge(`vm_tmp_blocks_max_inmemory_file_size_bytes`, func() float64 {
return float64(maxInmemoryTmpBlocksFile())
})
var (
_ = metrics.NewGauge(`vm_tmp_blocks_max_inmemory_file_size_bytes`, func() float64 {
return float64(maxInmemoryTmpBlocksFile())
})
tmpBufSizeSummary = metrics.NewSummary(`vm_tmp_blocks_inmemory_file_size_bytes`)
)
type tmpBlocksFile struct {
buf []byte
@@ -65,6 +76,8 @@ func getTmpBlocksFile() *tmpBlocksFile {
func putTmpBlocksFile(tbf *tmpBlocksFile) {
tbf.MustClose()
bufLen := tbf.Len()
tmpBufSizeSummary.Update(float64(bufLen))
tbf.buf = tbf.buf[:0]
tbf.f = nil
tbf.r = nil
@@ -138,7 +151,7 @@ func (tbf *tmpBlocksFile) Finalize() error {
tbf.buf = tbf.buf[:0]
r := fs.NewReaderAt(tbf.f)
// Hint the OS that the file is read almost sequentiallly.
// Hint the OS that the file is read almost sequentially.
// This should reduce the number of disk seeks, which is important
// for HDDs.
r.MustFadviseSequentialRead(true)

View File

@@ -7,6 +7,8 @@ import (
"sync"
"sync/atomic"
"time"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/stringsutil"
)
// ActiveQueriesHandler returns response to /api/v1/status/active_queries
@@ -23,8 +25,8 @@ func ActiveQueriesHandler(w http.ResponseWriter, _ *http.Request) {
fmt.Fprintf(w, `{"status":"ok","data":[`)
for i, aqe := range aqes {
d := now.Sub(aqe.startTime)
fmt.Fprintf(w, `{"duration":"%.3fs","id":"%016X","remote_addr":%s,"query":%q,"start":%d,"end":%d,"step":%d}`,
d.Seconds(), aqe.qid, aqe.quotedRemoteAddr, aqe.q, aqe.start, aqe.end, aqe.step)
fmt.Fprintf(w, `{"duration":"%.3fs","id":"%016X","remote_addr":%s,"query":%s,"start":%d,"end":%d,"step":%d}`,
d.Seconds(), aqe.qid, aqe.quotedRemoteAddr, stringsutil.JSONString(aqe.q), aqe.start, aqe.end, aqe.step)
if i+1 < len(aqes) {
fmt.Fprintf(w, `,`)
}

View File

@@ -1052,7 +1052,7 @@ func fixBrokenBuckets(i int, xss []leTimeseries) {
// Substitute upper bucket values with lower bucket values if the upper values are NaN
// or are bigger than the lower bucket values.
vNext := xss[0].ts.Values[0]
vNext := xss[0].ts.Values[i]
for j := 1; j < len(xss); j++ {
v := xss[j].ts.Values[i]
if math.IsNaN(v) || vNext > v {

View File

@@ -39,6 +39,30 @@ func TestFixBrokenBuckets(t *testing.T) {
f([]float64{5, 10, 4, 3}, []float64{5, 10, 10, 10})
}
func TestFixBrokenBucketsMultipleValues(t *testing.T) {
f := func(values, expectedResult [][]float64) {
t.Helper()
xss := make([]leTimeseries, len(values))
for i, v := range values {
xss[i].ts = &timeseries{
Values: v,
}
}
for i := range len(values) - 1 {
fixBrokenBuckets(i, xss)
}
result := make([][]float64, len(values))
for i, xs := range xss {
result[i] = xs.ts.Values
}
if !reflect.DeepEqual(result, expectedResult) {
t.Fatalf("unexpected result for values=%v\ngot\n%v\nwant\n%v", values, result, expectedResult)
}
}
f([][]float64{{10, 1}, {11, 2}, {13, 3}}, [][]float64{{10, 1}, {11, 2}, {13, 3}})
}
func TestVmrangeBucketsToLE(t *testing.T) {
f := func(buckets, bucketsExpected string) {
t.Helper()

View File

@@ -9,6 +9,7 @@ import (
"time"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/stringsutil"
)
var (
@@ -74,13 +75,13 @@ func initQueryStats() {
}
func (qst *queryStatsTracker) writeJSONQueryStats(w io.Writer, topN int, maxLifetime time.Duration) {
fmt.Fprintf(w, `{"topN":"%d","maxLifetime":%q,`, topN, maxLifetime)
fmt.Fprintf(w, `{"topN":"%d","maxLifetime":"%s",`, topN, maxLifetime)
fmt.Fprintf(w, `"search.queryStats.lastQueriesCount":%d,`, *lastQueriesCount)
fmt.Fprintf(w, `"search.queryStats.minQueryDuration":%q,`, *minQueryDuration)
fmt.Fprintf(w, `"search.queryStats.minQueryDuration":"%s",`, *minQueryDuration)
fmt.Fprintf(w, `"topByCount":[`)
topByCount := qst.getTopByCount(topN, maxLifetime)
for i, r := range topByCount {
fmt.Fprintf(w, `{"query":%q,"timeRangeSeconds":%d,"count":%d}`, r.query, r.timeRangeSecs, r.count)
fmt.Fprintf(w, `{"query":%s,"timeRangeSeconds":%d,"count":%d}`, stringsutil.JSONString(r.query), r.timeRangeSecs, r.count)
if i+1 < len(topByCount) {
fmt.Fprintf(w, `,`)
}
@@ -88,7 +89,7 @@ func (qst *queryStatsTracker) writeJSONQueryStats(w io.Writer, topN int, maxLife
fmt.Fprintf(w, `],"topByAvgDuration":[`)
topByAvgDuration := qst.getTopByAvgDuration(topN, maxLifetime)
for i, r := range topByAvgDuration {
fmt.Fprintf(w, `{"query":%q,"timeRangeSeconds":%d,"avgDurationSeconds":%.3f,"count":%d}`, r.query, r.timeRangeSecs, r.duration.Seconds(), r.count)
fmt.Fprintf(w, `{"query":%s,"timeRangeSeconds":%d,"avgDurationSeconds":%.3f,"count":%d}`, stringsutil.JSONString(r.query), r.timeRangeSecs, r.duration.Seconds(), r.count)
if i+1 < len(topByAvgDuration) {
fmt.Fprintf(w, `,`)
}
@@ -96,7 +97,7 @@ func (qst *queryStatsTracker) writeJSONQueryStats(w io.Writer, topN int, maxLife
fmt.Fprintf(w, `],"topBySumDuration":[`)
topBySumDuration := qst.getTopBySumDuration(topN, maxLifetime)
for i, r := range topBySumDuration {
fmt.Fprintf(w, `{"query":%q,"timeRangeSeconds":%d,"sumDurationSeconds":%.3f,"count":%d}`, r.query, r.timeRangeSecs, r.duration.Seconds(), r.count)
fmt.Fprintf(w, `{"query":%s,"timeRangeSeconds":%d,"sumDurationSeconds":%.3f,"count":%d}`, stringsutil.JSONString(r.query), r.timeRangeSecs, r.duration.Seconds(), r.count)
if i+1 < len(topBySumDuration) {
fmt.Fprintf(w, `,`)
}

View File

@@ -1,13 +1,13 @@
{
"files": {
"main.css": "./static/css/main.fce049bf.css",
"main.js": "./static/js/main.4b213512.js",
"main.css": "./static/css/main.81c6ec3a.css",
"main.js": "./static/js/main.9ed27532.js",
"static/js/685.bebe1265.chunk.js": "./static/js/685.bebe1265.chunk.js",
"static/media/MetricsQL.md": "./static/media/MetricsQL.aaabf95f2c9bf356bde4.md",
"static/media/MetricsQL.md": "./static/media/MetricsQL.8c2e588d62b87f90dbf0.md",
"index.html": "./index.html"
},
"entrypoints": [
"static/css/main.fce049bf.css",
"static/js/main.4b213512.js"
"static/css/main.81c6ec3a.css",
"static/js/main.9ed27532.js"
]
}

View File

@@ -1 +1 @@
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=5"/><meta name="theme-color" content="#000000"/><meta name="description" content="UI for VictoriaMetrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><script src="./dashboards/index.js" type="module"></script><meta name="twitter:card" content="summary_large_image"><meta name="twitter:image" content="./preview.jpg"><meta name="twitter:title" content="UI for VictoriaMetrics"><meta name="twitter:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta name="twitter:site" content="@VictoriaMetrics"><meta property="og:title" content="Metric explorer for VictoriaMetrics"><meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta property="og:image" content="./preview.jpg"><meta property="og:type" content="website"><script defer="defer" src="./static/js/main.4b213512.js"></script><link href="./static/css/main.fce049bf.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=5"/><meta name="theme-color" content="#000000"/><meta name="description" content="UI for VictoriaMetrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><script src="./dashboards/index.js" type="module"></script><meta name="twitter:card" content="summary_large_image"><meta name="twitter:image" content="./preview.jpg"><meta name="twitter:title" content="UI for VictoriaMetrics"><meta name="twitter:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta name="twitter:site" content="@VictoriaMetrics"><meta property="og:title" content="Metric explorer for VictoriaMetrics"><meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta property="og:image" content="./preview.jpg"><meta property="og:type" content="website"><script defer="defer" src="./static/js/main.9ed27532.js"></script><link href="./static/css/main.81c6ec3a.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

View File

@@ -5,7 +5,7 @@
*/
/**
* @remix-run/router v1.15.1
* @remix-run/router v1.17.0
*
* Copyright (c) Remix Software Inc.
*
@@ -16,7 +16,7 @@
*/
/**
* React Router DOM v6.22.1
* React Router DOM v6.24.0
*
* Copyright (c) Remix Software Inc.
*
@@ -27,7 +27,7 @@
*/
/**
* React Router v6.22.1
* React Router v6.24.0
*
* Copyright (c) Remix Software Inc.
*

View File

@@ -1,5 +1,4 @@
---
sort: 23
weight: 23
title: MetricsQL
menu:
@@ -10,9 +9,6 @@ aliases:
- /ExtendedPromQL.html
- /MetricsQL.html
---
# MetricsQL
[VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics) implements MetricsQL -
query language inspired by [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/).
MetricsQL is backwards-compatible with PromQL, so Grafana dashboards backed by Prometheus datasource should work
@@ -220,9 +216,11 @@ See also [descent_over_time](#descent_over_time).
over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d` per each time series returned
from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
This function is supported by PromQL.
See also [median_over_time](#median_over_time).
See also [median_over_time](#median_over_time), [min_over_time](#min_over_time) and [max_over_time](#max_over_time).
#### changes
@@ -262,6 +260,8 @@ from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#f
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [count_over_time](#count_over_time), [share_eq_over_time](#share_eq_over_time) and [count_values_over_time](#count_values_over_time).
#### count_gt_over_time
@@ -272,6 +272,8 @@ from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#f
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [count_over_time](#count_over_time) and [share_gt_over_time](#share_gt_over_time).
#### count_le_over_time
@@ -282,6 +284,8 @@ from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#f
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [count_over_time](#count_over_time) and [share_le_over_time](#share_le_over_time).
#### count_ne_over_time
@@ -292,7 +296,9 @@ from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#f
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
See also [count_over_time](#count_over_time).
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [count_over_time](#count_over_time) and [count_eq_over_time](#count_eq_over_time).
#### count_over_time
@@ -313,6 +319,8 @@ The results are calculated independently per each time series returned from the
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [count_eq_over_time](#count_eq_over_time), [count_values](#count_values) and [distinct_over_time](#distinct_over_time) and [label_match](#label_match).
#### decreases_over_time
@@ -432,6 +440,8 @@ from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#f
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
#### histogram_over_time
`histogram_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates
@@ -443,11 +453,15 @@ For example, the following query calculates median temperature by country over t
`histogram_quantile(0.5, sum(histogram_over_time(temperature[24h])) by (vmrange,country))`.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
#### hoeffding_bound_lower
`hoeffding_bound_lower(phi, series_selector[d])` is a [rollup function](#rollup-functions), which calculates
lower [Hoeffding bound](https://en.wikipedia.org/wiki/Hoeffding%27s_inequality) for the given `phi` in the range `[0...1]`.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [hoeffding_bound_upper](#hoeffding_bound_upper).
#### hoeffding_bound_upper
@@ -455,6 +469,8 @@ See also [hoeffding_bound_upper](#hoeffding_bound_upper).
`hoeffding_bound_upper(phi, series_selector[d])` is a [rollup function](#rollup-functions), which calculates
upper [Hoeffding bound](https://en.wikipedia.org/wiki/Hoeffding%27s_inequality) for the given `phi` in the range `[0...1]`.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [hoeffding_bound_lower](#hoeffding_bound_lower).
#### holt_winters
@@ -462,8 +478,9 @@ See also [hoeffding_bound_lower](#hoeffding_bound_lower).
`holt_winters(series_selector[d], sf, tf)` is a [rollup function](#rollup-functions), which calculates Holt-Winters value
(aka [double exponential smoothing](https://en.wikipedia.org/wiki/Exponential_smoothing#Double_exponential_smoothing)) for [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
over the given lookbehind window `d` using the given smoothing factor `sf` and the given trend factor `tf`.
Both `sf` and `tf` must be in the range `[0...1]`. It is expected that the [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering)
returns time series of [gauge type](https://docs.victoriametrics.com/keyconcepts/#gauge).
Both `sf` and `tf` must be in the range `[0...1]`.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
This function is supported by PromQL.
@@ -495,13 +512,14 @@ See also [deriv](#deriv).
`increase(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the increase over the given lookbehind window `d`
per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
It is expected that the `series_selector` returns time series of [counter type](https://docs.victoriametrics.com/keyconcepts/#counter).
Unlike Prometheus, it takes into account the last sample before the given lookbehind window `d` when calculating the result.
See [this article](https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e) for details.
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
This function is supported by PromQL.
See also [increase_pure](#increase_pure), [increase_prometheus](#increase_prometheus) and [delta](#delta).
@@ -510,12 +528,13 @@ See also [increase_pure](#increase_pure), [increase_prometheus](#increase_promet
`increase_prometheus(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the increase
over the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
It is expected that the `series_selector` returns time series of [counter type](https://docs.victoriametrics.com/keyconcepts/#counter).
It doesn't take into account the last sample before the given lookbehind window `d` when calculating the result in the same way as Prometheus does.
See [this article](https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e) for details.
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
See also [increase_pure](#increase_pure) and [increase](#increase).
#### increase_pure
@@ -524,6 +543,10 @@ See also [increase_pure](#increase_pure) and [increase](#increase).
of the following corner case - it assumes that [counters](https://docs.victoriametrics.com/keyconcepts/#counter) always start from 0,
while [increase](#increase) ignores the first value in a series if it is too big.
This function is usually applied to [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
See also [increase](#increase) and [increase_prometheus](#increase_prometheus).
#### increases_over_time
`increases_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the number of [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
@@ -540,15 +563,18 @@ on the given lookbehind window `d` per each time series returned from the given
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
#### irate
`irate(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the "instant" per-second increase rate over
the last two [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
It is expected that the `series_selector` returns time series of [counter type](https://docs.victoriametrics.com/keyconcepts/#counter).
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
This function is supported by PromQL.
See also [rate](#rate) and [rollup_rate](#rollup_rate).
@@ -587,6 +613,8 @@ See also [duration_over_time](#duration_over_time) and [lag](#lag).
over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d` per each time series returned
from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [mad](#mad), [range_mad](#range_mad) and [outlier_iqr_over_time](#outlier_iqr_over_time).
#### max_over_time
@@ -594,9 +622,11 @@ See also [mad](#mad), [range_mad](#range_mad) and [outlier_iqr_over_time](#outli
`max_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the maximum value over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
This function is supported by PromQL.
See also [tmax_over_time](#tmax_over_time).
See also [tmax_over_time](#tmax_over_time) and [min_over_time](#min_over_time).
#### median_over_time
@@ -604,6 +634,8 @@ See also [tmax_over_time](#tmax_over_time).
on the given lookbehind window `d` per each time series returned
from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [avg_over_time](#avg_over_time).
#### min_over_time
@@ -611,9 +643,11 @@ See also [avg_over_time](#avg_over_time).
`min_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the minimum value over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
This function is supported by PromQL.
See also [tmin_over_time](#tmin_over_time).
See also [tmin_over_time](#tmin_over_time) and [max_over_time](#max_over_time).
#### mode_over_time
@@ -622,6 +656,8 @@ for [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on
from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). It is expected that [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
values are discrete.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
#### outlier_iqr_over_time
`outlier_iqr_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which returns the last sample on the given lookbehind window `d`
@@ -632,6 +668,8 @@ if its value is either smaller than the `q25-1.5*iqr` or bigger than `q75+1.5*iq
The `outlier_iqr_over_time()` is useful for detecting anomalies in gauge values based on the previous history of values.
For example, `outlier_iqr_over_time(memory_usage_bytes[1h])` triggers when `memory_usage_bytes` suddenly goes outside the usual value range for the last hour.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [outliers_iqr](#outliers_iqr).
#### predict_linear
@@ -659,6 +697,8 @@ This function is supported by PromQL.
on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
The `phi` value must be in the range `[0...1]`.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
This function is supported by PromQL.
See also [quantiles_over_time](#quantiles_over_time).
@@ -670,6 +710,8 @@ over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on
from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
The function returns individual series per each `phi*` with `{phiLabel="phi*"}` label. `phi*` values must be in the range `[0...1]`.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [quantile_over_time](#quantile_over_time).
#### range_over_time
@@ -680,11 +722,12 @@ E.g. it calculates `max_over_time(series_selector[d]) - min_over_time(series_sel
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
#### rate
`rate(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the average per-second increase rate
over the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
It is expected that the `series_selector` returns time series of [counter type](https://docs.victoriametrics.com/keyconcepts/#counter).
If the lookbehind window is skipped in square brackets, then it is automatically calculated as `max(step, scrape_interval)`, where `step` is the query arg value
passed to [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query) or [/api/v1/query](https://docs.victoriametrics.com/keyconcepts/#instant-query),
@@ -703,6 +746,8 @@ See also [irate](#irate) and [rollup_rate](#rollup_rate).
on the given lookbehind window `d`. The calculations are performed individually per each time series returned
from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
#### resets
@@ -710,10 +755,11 @@ Metric names are stripped from the resulting rollups. Add [keep_metric_names](#k
`resets(series_selector[d])` is a [rollup function](#rollup-functions), which returns the number
of [counter](https://docs.victoriametrics.com/keyconcepts/#counter) resets over the given lookbehind window `d`
per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
It is expected that the `series_selector` returns time series of [counter type](https://docs.victoriametrics.com/keyconcepts/#counter).
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
This function is supported by PromQL.
#### rollup
@@ -725,6 +771,10 @@ These values are calculated individually per each time series returned from the
Optional 2nd argument `"min"`, `"max"` or `"avg"` can be passed to keep only one calculation result and without adding a label.
See also [label_match](#label_match).
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [rollup_rate](#rollup_rate).
#### rollup_candlestick
`rollup_candlestick(series_selector[d])` is a [rollup function](#rollup-functions), which calculates `open`, `high`, `low` and `close` values (aka OHLC)
@@ -736,6 +786,8 @@ from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#f
Optional 2nd argument `"open"`, `"high"` or `"low"` or `"close"` can be passed to keep only one calculation result and without adding a label.
See also [label_match](#label_match).
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
#### rollup_delta
`rollup_delta(series_selector[d])` is a [rollup function](#rollup-functions), which calculates differences between adjacent [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
@@ -762,6 +814,8 @@ See also [label_match](#label_match).
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
See also [rollup](#rollup) and [rollup_rate](#rollup_rate).
#### rollup_increase
`rollup_increase(series_selector[d])` is a [rollup function](#rollup-functions), which calculates increases for adjacent [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
@@ -774,12 +828,17 @@ See also [label_match](#label_match).
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. See also [rollup_delta](#rollup_delta).
This function is usually applied to [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
See also [rollup](#rollup) and [rollup_rate](#rollup_rate).
#### rollup_rate
`rollup_rate(series_selector[d])` is a [rollup function](#rollup-functions), which calculates per-second change rates
for adjacent [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
on the given lookbehind window `d` and returns `min`, `max` and `avg` values for the calculated per-second change rates
and returns them in time series with `rollup="min"`, `rollup="max"` and `rollup="avg"` additional labels.
The calculations are performed individually per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
See [this article](https://valyala.medium.com/why-irate-from-prometheus-doesnt-capture-spikes-45f9896d7832) in order to understand better
when to use `rollup_rate()`.
@@ -787,10 +846,12 @@ when to use `rollup_rate()`.
Optional 2nd argument `"min"`, `"max"` or `"avg"` can be passed to keep only one calculation result and without adding a label.
See also [label_match](#label_match).
The calculations are performed individually per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
See also [rollup](#rollup) and [rollup_increase](#rollup_increase).
#### rollup_scrape_interval
`rollup_scrape_interval(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the interval in seconds between
@@ -824,6 +885,8 @@ This function is useful for calculating SLI and SLO. Example: `share_gt_over_tim
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [share_le_over_time](#share_le_over_time) and [count_gt_over_time](#count_gt_over_time).
#### share_le_over_time
@@ -838,6 +901,8 @@ the share of time series values for the last 24 hours when memory usage was belo
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [share_gt_over_time](#share_gt_over_time) and [count_le_over_time](#count_le_over_time).
#### share_eq_over_time
@@ -849,6 +914,8 @@ from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#f
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [count_eq_over_time](#count_eq_over_time).
#### stale_samples_over_time
@@ -866,6 +933,8 @@ on the given lookbehind window `d` per each time series returned from the given
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
This function is supported by PromQL.
See also [stdvar_over_time](#stdvar_over_time).
@@ -877,35 +946,43 @@ on the given lookbehind window `d` per each time series returned from the given
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
This function is supported by PromQL.
See also [stddev_over_time](#stddev_over_time).
#### sum_eq_over_time
`sum_eq_over_time(series_selector[d], eq)` is a [rollup function](#rollup-function), which calculates the sum of [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
`sum_eq_over_time(series_selector[d], eq)` is a [rollup function](#rollup-functions), which calculates the sum of [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
values equal to `eq` on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [sum_over_time](#sum_over_time) and [count_eq_over_time](#count_eq_over_time).
#### sum_gt_over_time
`sum_gt_over_time(series_selector[d], gt)` is a [rollup function](#rollup-function), which calculates the sum of [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
`sum_gt_over_time(series_selector[d], gt)` is a [rollup function](#rollup-functions), which calculates the sum of [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
values bigger than `gt` on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [sum_over_time](#sum_over_time) and [count_gt_over_time](#count_gt_over_time).
#### sum_le_over_time
`sum_le_over_time(series_selector[d], le)` is a [rollup function](#rollup-function), which calculates the sum of [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
`sum_le_over_time(series_selector[d], le)` is a [rollup function](#rollup-functions), which calculates the sum of [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
values smaller or equal to `le` on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [sum_over_time](#sum_over_time) and [count_le_over_time](#count_le_over_time).
#### sum_over_time
@@ -915,6 +992,8 @@ on the given lookbehind window `d` per each time series returned from the given
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
This function is supported by PromQL.
#### sum2_over_time
@@ -924,6 +1003,8 @@ values on the given lookbehind window `d` per each time series returned from the
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
#### timestamp
`timestamp(series_selector[d])` is a [rollup function](#rollup-functions), which returns the timestamp in seconds with millisecond precision
@@ -1001,6 +1082,8 @@ from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#f
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [zscore](#zscore), [range_trim_zscore](#range_trim_zscore) and [outlier_iqr_over_time](#outlier_iqr_over_time).
@@ -2230,7 +2313,8 @@ Any [rollup function](#rollup-functions) for something other than [series select
Nested rollup functions can be implicit thanks to the [implicit query conversions](#implicit-query-conversions).
For example, `delta(sum(m))` is implicitly converted to `delta(sum(default_rollup(m))[1i:1i])`, so it becomes a subquery,
since it contains [default_rollup](#default_rollup) nested into [delta](#delta).
This behavior can be disabled or logged via cmd-line flags `-search.disableImplicitConversion` and `-search.logImplicitConversion` since v1.101.0.
This behavior can be disabled or logged via `-search.disableImplicitConversion` and `-search.logImplicitConversion` command-line flags
starting from [`v1.101.0` release](https://docs.victoriametrics.com/changelog/).
VictoriaMetrics performs subqueries in the following way:
@@ -2263,5 +2347,6 @@ VictoriaMetrics performs the following implicit conversions for incoming queries
For example, `avg_over_time(rate(http_requests_total[5m])[1h])` is automatically converted to `avg_over_time(rate(http_requests_total[5m])[1h:1i])`.
* If something other than [series selector](https://docs.victoriametrics.com/keyconcepts/#filtering)
is passed to [rollup function](#rollup-functions), then a [subquery](#subqueries) with `1i` lookbehind window and `1i` step is automatically formed.
For example, `rate(sum(up))` is automatically converted to `rate((sum(default_rollup(up)))[1i:1i])`.
This behavior can be disabled or logged via cmd-line flags `-search.disableImplicitConversion` and `-search.logImplicitConversion` since v1.101.0.
For example, `rate(sum(up))` is automatically converted to `rate((sum(default_rollup(up)))[1i:1i])`.
This behavior can be disabled or logged via `-search.disableImplicitConversion` and `-search.logImplicitConversion` command-line flags
starting from [`v1.101.0` release](https://docs.victoriametrics.com/changelog/).

View File

@@ -0,0 +1,55 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8"/>
<link rel="icon" href="%PUBLIC_URL%/favicon.ico"/>
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=5"/>
<meta name="theme-color" content="#000000"/>
<meta
name="description"
content="UI for VictoriaMetrics"
/>
<link rel="apple-touch-icon" href="%PUBLIC_URL%/apple-touch-icon.png"/>
<link rel="icon" type="image/png" sizes="32x32" href="%PUBLIC_URL%/favicon-32x32.png">
<!--
manifest.json provides metadata used when your web app is installed on a
user's mobile device or desktop. See https://developers.google.com/web/fundamentals/web-app-manifest/
-->
<link rel="manifest" href="%PUBLIC_URL%/manifest.json"/>
<!--
Notice the use of %PUBLIC_URL% in the tags above.
It will be replaced with the URL of the `public` folder during the build.
Only files inside the `public` folder can be referenced from the HTML.
Unlike "/favicon.ico" or "favicon.ico", "%PUBLIC_URL%/favicon.ico" will
work correctly both with client-side routing and a non-root public URL.
Learn how to configure a non-root public URL by running `npm run build`.
-->
<title>VM UI</title>
<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:image" content="%PUBLIC_URL%/preview.jpg">
<meta name="twitter:title" content="UI for VictoriaMetrics">
<meta name="twitter:description" content="Explore and troubleshoot your VictoriaMetrics data">
<meta name="twitter:site" content="@VictoriaMetrics">
<meta property="og:title" content="Metric explorer for VictoriaMetrics">
<meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data">
<meta property="og:image" content="%PUBLIC_URL%/preview.jpg">
<meta property="og:type" content="website">
</head>
<body>
<noscript>You need to enable JavaScript to run this app.</noscript>
<div id="root"></div>
<!--
This HTML file is a template.
If you open it directly in the browser, you will see an empty page.
You can add webfonts, meta tags, or analytics to this file.
The build step will place the bundled scripts into the <body> tag.
To begin the development, run `npm start` or `yarn start`.
To create a production bundle, use `npm run build` or `yarn build`.
-->
</body>
</html>

View File

@@ -0,0 +1,56 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8"/>
<link rel="icon" href="%PUBLIC_URL%/favicon.ico"/>
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=5"/>
<meta name="theme-color" content="#000000"/>
<meta
name="description"
content="UI for VictoriaMetrics"
/>
<link rel="apple-touch-icon" href="%PUBLIC_URL%/apple-touch-icon.png"/>
<link rel="icon" type="image/png" sizes="32x32" href="%PUBLIC_URL%/favicon-32x32.png">
<!--
manifest.json provides metadata used when your web app is installed on a
user's mobile device or desktop. See https://developers.google.com/web/fundamentals/web-app-manifest/
-->
<link rel="manifest" href="%PUBLIC_URL%/manifest.json"/>
<!--
Notice the use of %PUBLIC_URL% in the tags above.
It will be replaced with the URL of the `public` folder during the build.
Only files inside the `public` folder can be referenced from the HTML.
Unlike "/favicon.ico" or "favicon.ico", "%PUBLIC_URL%/favicon.ico" will
work correctly both with client-side routing and a non-root public URL.
Learn how to configure a non-root public URL by running `npm run build`.
-->
<title>VM UI</title>
<script src="%PUBLIC_URL%/dashboards/index.js" type="module"></script>
<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:image" content="%PUBLIC_URL%/preview.jpg">
<meta name="twitter:title" content="UI for VictoriaMetrics">
<meta name="twitter:description" content="Explore and troubleshoot your VictoriaMetrics data">
<meta name="twitter:site" content="@VictoriaMetrics">
<meta property="og:title" content="Metric explorer for VictoriaMetrics">
<meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data">
<meta property="og:image" content="%PUBLIC_URL%/preview.jpg">
<meta property="og:type" content="website">
</head>
<body>
<noscript>You need to enable JavaScript to run this app.</noscript>
<div id="root"></div>
<!--
This HTML file is a template.
If you open it directly in the browser, you will see an empty page.
You can add webfonts, meta tags, or analytics to this file.
The build step will place the bundled scripts into the <body> tag.
To begin the development, run `npm start` or `yarn start`.
To create a production bundle, use `npm run build` or `yarn build`.
-->
</body>
</html>

View File

@@ -0,0 +1,56 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8"/>
<link rel="icon" href="%PUBLIC_URL%/favicon.ico"/>
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=5"/>
<meta name="theme-color" content="#000000"/>
<meta
name="description"
content="UI for VictoriaMetrics"
/>
<link rel="apple-touch-icon" href="%PUBLIC_URL%/apple-touch-icon.png"/>
<link rel="icon" type="image/png" sizes="32x32" href="%PUBLIC_URL%/favicon-32x32.png">
<!--
manifest.json provides metadata used when your web app is installed on a
user's mobile device or desktop. See https://developers.google.com/web/fundamentals/web-app-manifest/
-->
<link rel="manifest" href="%PUBLIC_URL%/manifest.json"/>
<!--
Notice the use of %PUBLIC_URL% in the tags above.
It will be replaced with the URL of the `public` folder during the build.
Only files inside the `public` folder can be referenced from the HTML.
Unlike "/favicon.ico" or "favicon.ico", "%PUBLIC_URL%/favicon.ico" will
work correctly both with client-side routing and a non-root public URL.
Learn how to configure a non-root public URL by running `npm run build`.
-->
<title>VM UI</title>
<script src="%PUBLIC_URL%/dashboards/index.js" type="module"></script>
<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:image" content="%PUBLIC_URL%/preview.jpg">
<meta name="twitter:title" content="UI for VictoriaMetrics">
<meta name="twitter:description" content="Explore and troubleshoot your VictoriaMetrics data">
<meta name="twitter:site" content="@VictoriaMetrics">
<meta property="og:title" content="Metric explorer for VictoriaMetrics">
<meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data">
<meta property="og:image" content="%PUBLIC_URL%/preview.jpg">
<meta property="og:type" content="website">
</head>
<body>
<noscript>You need to enable JavaScript to run this app.</noscript>
<div id="root"></div>
<!--
This HTML file is a template.
If you open it directly in the browser, you will see an empty page.
You can add webfonts, meta tags, or analytics to this file.
The build step will place the bundled scripts into the <body> tag.
To begin the development, run `npm start` or `yarn start`.
To create a production bundle, use `npm run build` or `yarn build`.
-->
</body>
</html>

View File

@@ -21,6 +21,7 @@ import (
"github.com/VictoriaMetrics/VictoriaMetrics/lib/mergeset"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/querytracer"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/storage"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/stringsutil"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/syncwg"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/timeutil"
)
@@ -153,9 +154,9 @@ func AddRows(mrs []storage.MetricRow) error {
}
resetResponseCacheIfNeeded(mrs)
WG.Add(1)
err := Storage.AddRows(mrs, uint8(*precisionBits))
Storage.AddRows(mrs, uint8(*precisionBits))
WG.Done()
return err
return nil
}
var errReadOnly = errors.New("the storage is in read-only mode; check -storage.minFreeDiskSpaceBytes command-line flag value")
@@ -308,9 +309,9 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
return true
}
if prometheusCompatibleResponse {
fmt.Fprintf(w, `{"status":"success","data":{"name":%q}}`, snapshotPath)
fmt.Fprintf(w, `{"status":"success","data":{"name":%s}}`, stringsutil.JSONString(snapshotPath))
} else {
fmt.Fprintf(w, `{"status":"ok","snapshot":%q}`, snapshotPath)
fmt.Fprintf(w, `{"status":"ok","snapshot":%s}`, stringsutil.JSONString(snapshotPath))
}
return true
case "/list":
@@ -636,5 +637,6 @@ func writeStorageMetrics(w io.Writer, strg *storage.Storage) {
func jsonResponseError(w http.ResponseWriter, err error) {
logger.Errorf("%s", err)
w.WriteHeader(http.StatusInternalServerError)
fmt.Fprintf(w, `{"status":"error","msg":%q}`, err)
errStr := err.Error()
fmt.Fprintf(w, `{"status":"error","msg":%s}`, stringsutil.JSONString(errStr))
}

View File

@@ -1,4 +1,4 @@
FROM golang:1.22.5 AS build-web-stage
FROM golang:1.23.0 AS build-web-stage
COPY build /build
WORKDIR /build
@@ -6,7 +6,7 @@ COPY web/ /build/
RUN GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o web-amd64 github.com/VictoriMetrics/vmui/ && \
GOOS=windows GOARCH=amd64 CGO_ENABLED=0 go build -o web-windows github.com/VictoriMetrics/vmui/
FROM alpine:3.20.1
FROM alpine:3.20.2
USER root
COPY --from=build-web-stage /build/web-amd64 /app/web

View File

@@ -1,6 +1,22 @@
/* eslint-disable */
const { override, addExternalBabelPlugin, addWebpackAlias, addWebpackPlugin } = require("customize-cra");
const webpack = require("webpack");
const fs = require('fs');
const path = require('path');
// This will replace the default check
const pathIndexHTML = (() => {
switch (process.env.REACT_APP_TYPE) {
case 'logs':
return 'public/victorialogs.html';
case 'anomaly':
return 'public/vmanomaly.html';
default:
return 'public/victoriametrics.html';
}
})();
const fileContent = fs.readFileSync(path.resolve(__dirname, pathIndexHTML), 'utf8');
fs.writeFileSync(path.resolve(__dirname, 'public/index.html'), fileContent);
module.exports = override(
addExternalBabelPlugin("@babel/plugin-proposal-nullish-coalescing-operator"),

View File

@@ -1,56 +0,0 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<link rel="icon" href="%PUBLIC_URL%/favicon.ico" />
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=5" />
<meta name="theme-color" content="#000000" />
<meta
name="description"
content="UI for VictoriaMetrics"
/>
<link rel="apple-touch-icon" href="%PUBLIC_URL%/apple-touch-icon.png" />
<link rel="icon" type="image/png" sizes="32x32" href="%PUBLIC_URL%/favicon-32x32.png">
<!--
manifest.json provides metadata used when your web app is installed on a
user's mobile device or desktop. See https://developers.google.com/web/fundamentals/web-app-manifest/
-->
<link rel="manifest" href="%PUBLIC_URL%/manifest.json" />
<!--
Notice the use of %PUBLIC_URL% in the tags above.
It will be replaced with the URL of the `public` folder during the build.
Only files inside the `public` folder can be referenced from the HTML.
Unlike "/favicon.ico" or "favicon.ico", "%PUBLIC_URL%/favicon.ico" will
work correctly both with client-side routing and a non-root public URL.
Learn how to configure a non-root public URL by running `npm run build`.
-->
<title>VM UI</title>
<script src="%PUBLIC_URL%/dashboards/index.js" type="module"></script>
<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:image" content="%PUBLIC_URL%/preview.jpg">
<meta name="twitter:title" content="UI for VictoriaMetrics">
<meta name="twitter:description" content="Explore and troubleshoot your VictoriaMetrics data">
<meta name="twitter:site" content="@VictoriaMetrics">
<meta property="og:title" content="Metric explorer for VictoriaMetrics">
<meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data">
<meta property="og:image" content="%PUBLIC_URL%/preview.jpg">
<meta property="og:type" content="website">
</head>
<body>
<noscript>You need to enable JavaScript to run this app.</noscript>
<div id="root"></div>
<!--
This HTML file is a template.
If you open it directly in the browser, you will see an empty page.
You can add webfonts, meta tags, or analytics to this file.
The build step will place the bundled scripts into the <body> tag.
To begin the development, run `npm start` or `yarn start`.
To create a production bundle, use `npm run build` or `yarn build`.
-->
</body>
</html>

View File

@@ -0,0 +1,55 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8"/>
<link rel="icon" href="%PUBLIC_URL%/favicon.ico"/>
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=5"/>
<meta name="theme-color" content="#000000"/>
<meta
name="description"
content="UI for VictoriaMetrics"
/>
<link rel="apple-touch-icon" href="%PUBLIC_URL%/apple-touch-icon.png"/>
<link rel="icon" type="image/png" sizes="32x32" href="%PUBLIC_URL%/favicon-32x32.png">
<!--
manifest.json provides metadata used when your web app is installed on a
user's mobile device or desktop. See https://developers.google.com/web/fundamentals/web-app-manifest/
-->
<link rel="manifest" href="%PUBLIC_URL%/manifest.json"/>
<!--
Notice the use of %PUBLIC_URL% in the tags above.
It will be replaced with the URL of the `public` folder during the build.
Only files inside the `public` folder can be referenced from the HTML.
Unlike "/favicon.ico" or "favicon.ico", "%PUBLIC_URL%/favicon.ico" will
work correctly both with client-side routing and a non-root public URL.
Learn how to configure a non-root public URL by running `npm run build`.
-->
<title>VM UI</title>
<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:image" content="%PUBLIC_URL%/preview.jpg">
<meta name="twitter:title" content="UI for VictoriaMetrics">
<meta name="twitter:description" content="Explore and troubleshoot your VictoriaMetrics data">
<meta name="twitter:site" content="@VictoriaMetrics">
<meta property="og:title" content="Metric explorer for VictoriaMetrics">
<meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data">
<meta property="og:image" content="%PUBLIC_URL%/preview.jpg">
<meta property="og:type" content="website">
</head>
<body>
<noscript>You need to enable JavaScript to run this app.</noscript>
<div id="root"></div>
<!--
This HTML file is a template.
If you open it directly in the browser, you will see an empty page.
You can add webfonts, meta tags, or analytics to this file.
The build step will place the bundled scripts into the <body> tag.
To begin the development, run `npm start` or `yarn start`.
To create a production bundle, use `npm run build` or `yarn build`.
-->
</body>
</html>

View File

@@ -0,0 +1,56 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8"/>
<link rel="icon" href="%PUBLIC_URL%/favicon.ico"/>
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=5"/>
<meta name="theme-color" content="#000000"/>
<meta
name="description"
content="UI for VictoriaMetrics"
/>
<link rel="apple-touch-icon" href="%PUBLIC_URL%/apple-touch-icon.png"/>
<link rel="icon" type="image/png" sizes="32x32" href="%PUBLIC_URL%/favicon-32x32.png">
<!--
manifest.json provides metadata used when your web app is installed on a
user's mobile device or desktop. See https://developers.google.com/web/fundamentals/web-app-manifest/
-->
<link rel="manifest" href="%PUBLIC_URL%/manifest.json"/>
<!--
Notice the use of %PUBLIC_URL% in the tags above.
It will be replaced with the URL of the `public` folder during the build.
Only files inside the `public` folder can be referenced from the HTML.
Unlike "/favicon.ico" or "favicon.ico", "%PUBLIC_URL%/favicon.ico" will
work correctly both with client-side routing and a non-root public URL.
Learn how to configure a non-root public URL by running `npm run build`.
-->
<title>VM UI</title>
<script src="%PUBLIC_URL%/dashboards/index.js" type="module"></script>
<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:image" content="%PUBLIC_URL%/preview.jpg">
<meta name="twitter:title" content="UI for VictoriaMetrics">
<meta name="twitter:description" content="Explore and troubleshoot your VictoriaMetrics data">
<meta name="twitter:site" content="@VictoriaMetrics">
<meta property="og:title" content="Metric explorer for VictoriaMetrics">
<meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data">
<meta property="og:image" content="%PUBLIC_URL%/preview.jpg">
<meta property="og:type" content="website">
</head>
<body>
<noscript>You need to enable JavaScript to run this app.</noscript>
<div id="root"></div>
<!--
This HTML file is a template.
If you open it directly in the browser, you will see an empty page.
You can add webfonts, meta tags, or analytics to this file.
The build step will place the bundled scripts into the <body> tag.
To begin the development, run `npm start` or `yarn start`.
To create a production bundle, use `npm run build` or `yarn build`.
-->
</body>
</html>

View File

@@ -0,0 +1,56 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8"/>
<link rel="icon" href="%PUBLIC_URL%/favicon.ico"/>
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=5"/>
<meta name="theme-color" content="#000000"/>
<meta
name="description"
content="UI for VictoriaMetrics"
/>
<link rel="apple-touch-icon" href="%PUBLIC_URL%/apple-touch-icon.png"/>
<link rel="icon" type="image/png" sizes="32x32" href="%PUBLIC_URL%/favicon-32x32.png">
<!--
manifest.json provides metadata used when your web app is installed on a
user's mobile device or desktop. See https://developers.google.com/web/fundamentals/web-app-manifest/
-->
<link rel="manifest" href="%PUBLIC_URL%/manifest.json"/>
<!--
Notice the use of %PUBLIC_URL% in the tags above.
It will be replaced with the URL of the `public` folder during the build.
Only files inside the `public` folder can be referenced from the HTML.
Unlike "/favicon.ico" or "favicon.ico", "%PUBLIC_URL%/favicon.ico" will
work correctly both with client-side routing and a non-root public URL.
Learn how to configure a non-root public URL by running `npm run build`.
-->
<title>VM UI</title>
<script src="%PUBLIC_URL%/dashboards/index.js" type="module"></script>
<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:image" content="%PUBLIC_URL%/preview.jpg">
<meta name="twitter:title" content="UI for VictoriaMetrics">
<meta name="twitter:description" content="Explore and troubleshoot your VictoriaMetrics data">
<meta name="twitter:site" content="@VictoriaMetrics">
<meta property="og:title" content="Metric explorer for VictoriaMetrics">
<meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data">
<meta property="og:image" content="%PUBLIC_URL%/preview.jpg">
<meta property="og:type" content="website">
</head>
<body>
<noscript>You need to enable JavaScript to run this app.</noscript>
<div id="root"></div>
<!--
This HTML file is a template.
If you open it directly in the browser, you will see an empty page.
You can add webfonts, meta tags, or analytics to this file.
The build step will place the bundled scripts into the <body> tag.
To begin the development, run `npm start` or `yarn start`.
To create a production bundle, use `npm run build` or `yarn build`.
-->
</body>
</html>

View File

@@ -7,7 +7,7 @@ import ExploreAnomaly from "./pages/ExploreAnomaly/ExploreAnomaly";
import router from "./router";
import CustomPanel from "./pages/CustomPanel";
const AppLogs: FC = () => {
const AppAnomaly: FC = () => {
const [loadedTheme, setLoadedTheme] = useState(false);
return <>
@@ -38,4 +38,4 @@ const AppLogs: FC = () => {
</>;
};
export default AppLogs;
export default AppAnomaly;

View File

@@ -38,6 +38,7 @@ export interface Logs {
export interface LogHits {
timestamps: string[];
values: number[];
total?: number;
fields: {
[key: string]: string;
};

View File

@@ -1,5 +1,4 @@
---
sort: 23
weight: 23
title: MetricsQL
menu:
@@ -10,9 +9,6 @@ aliases:
- /ExtendedPromQL.html
- /MetricsQL.html
---
# MetricsQL
[VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics) implements MetricsQL -
query language inspired by [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/).
MetricsQL is backwards-compatible with PromQL, so Grafana dashboards backed by Prometheus datasource should work
@@ -220,9 +216,11 @@ See also [descent_over_time](#descent_over_time).
over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d` per each time series returned
from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
This function is supported by PromQL.
See also [median_over_time](#median_over_time).
See also [median_over_time](#median_over_time), [min_over_time](#min_over_time) and [max_over_time](#max_over_time).
#### changes
@@ -262,6 +260,8 @@ from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#f
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [count_over_time](#count_over_time), [share_eq_over_time](#share_eq_over_time) and [count_values_over_time](#count_values_over_time).
#### count_gt_over_time
@@ -272,6 +272,8 @@ from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#f
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [count_over_time](#count_over_time) and [share_gt_over_time](#share_gt_over_time).
#### count_le_over_time
@@ -282,6 +284,8 @@ from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#f
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [count_over_time](#count_over_time) and [share_le_over_time](#share_le_over_time).
#### count_ne_over_time
@@ -292,7 +296,9 @@ from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#f
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
See also [count_over_time](#count_over_time).
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [count_over_time](#count_over_time) and [count_eq_over_time](#count_eq_over_time).
#### count_over_time
@@ -313,6 +319,8 @@ The results are calculated independently per each time series returned from the
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [count_eq_over_time](#count_eq_over_time), [count_values](#count_values) and [distinct_over_time](#distinct_over_time) and [label_match](#label_match).
#### decreases_over_time
@@ -432,6 +440,8 @@ from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#f
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
#### histogram_over_time
`histogram_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates
@@ -443,11 +453,15 @@ For example, the following query calculates median temperature by country over t
`histogram_quantile(0.5, sum(histogram_over_time(temperature[24h])) by (vmrange,country))`.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
#### hoeffding_bound_lower
`hoeffding_bound_lower(phi, series_selector[d])` is a [rollup function](#rollup-functions), which calculates
lower [Hoeffding bound](https://en.wikipedia.org/wiki/Hoeffding%27s_inequality) for the given `phi` in the range `[0...1]`.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [hoeffding_bound_upper](#hoeffding_bound_upper).
#### hoeffding_bound_upper
@@ -455,6 +469,8 @@ See also [hoeffding_bound_upper](#hoeffding_bound_upper).
`hoeffding_bound_upper(phi, series_selector[d])` is a [rollup function](#rollup-functions), which calculates
upper [Hoeffding bound](https://en.wikipedia.org/wiki/Hoeffding%27s_inequality) for the given `phi` in the range `[0...1]`.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [hoeffding_bound_lower](#hoeffding_bound_lower).
#### holt_winters
@@ -462,8 +478,9 @@ See also [hoeffding_bound_lower](#hoeffding_bound_lower).
`holt_winters(series_selector[d], sf, tf)` is a [rollup function](#rollup-functions), which calculates Holt-Winters value
(aka [double exponential smoothing](https://en.wikipedia.org/wiki/Exponential_smoothing#Double_exponential_smoothing)) for [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
over the given lookbehind window `d` using the given smoothing factor `sf` and the given trend factor `tf`.
Both `sf` and `tf` must be in the range `[0...1]`. It is expected that the [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering)
returns time series of [gauge type](https://docs.victoriametrics.com/keyconcepts/#gauge).
Both `sf` and `tf` must be in the range `[0...1]`.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
This function is supported by PromQL.
@@ -495,13 +512,14 @@ See also [deriv](#deriv).
`increase(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the increase over the given lookbehind window `d`
per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
It is expected that the `series_selector` returns time series of [counter type](https://docs.victoriametrics.com/keyconcepts/#counter).
Unlike Prometheus, it takes into account the last sample before the given lookbehind window `d` when calculating the result.
See [this article](https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e) for details.
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
This function is supported by PromQL.
See also [increase_pure](#increase_pure), [increase_prometheus](#increase_prometheus) and [delta](#delta).
@@ -510,12 +528,13 @@ See also [increase_pure](#increase_pure), [increase_prometheus](#increase_promet
`increase_prometheus(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the increase
over the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
It is expected that the `series_selector` returns time series of [counter type](https://docs.victoriametrics.com/keyconcepts/#counter).
It doesn't take into account the last sample before the given lookbehind window `d` when calculating the result in the same way as Prometheus does.
See [this article](https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e) for details.
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
See also [increase_pure](#increase_pure) and [increase](#increase).
#### increase_pure
@@ -524,6 +543,10 @@ See also [increase_pure](#increase_pure) and [increase](#increase).
of the following corner case - it assumes that [counters](https://docs.victoriametrics.com/keyconcepts/#counter) always start from 0,
while [increase](#increase) ignores the first value in a series if it is too big.
This function is usually applied to [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
See also [increase](#increase) and [increase_prometheus](#increase_prometheus).
#### increases_over_time
`increases_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the number of [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
@@ -540,15 +563,18 @@ on the given lookbehind window `d` per each time series returned from the given
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
#### irate
`irate(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the "instant" per-second increase rate over
the last two [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
It is expected that the `series_selector` returns time series of [counter type](https://docs.victoriametrics.com/keyconcepts/#counter).
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
This function is supported by PromQL.
See also [rate](#rate) and [rollup_rate](#rollup_rate).
@@ -587,6 +613,8 @@ See also [duration_over_time](#duration_over_time) and [lag](#lag).
over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d` per each time series returned
from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [mad](#mad), [range_mad](#range_mad) and [outlier_iqr_over_time](#outlier_iqr_over_time).
#### max_over_time
@@ -594,9 +622,11 @@ See also [mad](#mad), [range_mad](#range_mad) and [outlier_iqr_over_time](#outli
`max_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the maximum value over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
This function is supported by PromQL.
See also [tmax_over_time](#tmax_over_time).
See also [tmax_over_time](#tmax_over_time) and [min_over_time](#min_over_time).
#### median_over_time
@@ -604,6 +634,8 @@ See also [tmax_over_time](#tmax_over_time).
on the given lookbehind window `d` per each time series returned
from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [avg_over_time](#avg_over_time).
#### min_over_time
@@ -611,9 +643,11 @@ See also [avg_over_time](#avg_over_time).
`min_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the minimum value over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
This function is supported by PromQL.
See also [tmin_over_time](#tmin_over_time).
See also [tmin_over_time](#tmin_over_time) and [max_over_time](#max_over_time).
#### mode_over_time
@@ -622,6 +656,8 @@ for [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on
from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). It is expected that [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
values are discrete.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
#### outlier_iqr_over_time
`outlier_iqr_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which returns the last sample on the given lookbehind window `d`
@@ -632,6 +668,8 @@ if its value is either smaller than the `q25-1.5*iqr` or bigger than `q75+1.5*iq
The `outlier_iqr_over_time()` is useful for detecting anomalies in gauge values based on the previous history of values.
For example, `outlier_iqr_over_time(memory_usage_bytes[1h])` triggers when `memory_usage_bytes` suddenly goes outside the usual value range for the last hour.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [outliers_iqr](#outliers_iqr).
#### predict_linear
@@ -659,6 +697,8 @@ This function is supported by PromQL.
on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
The `phi` value must be in the range `[0...1]`.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
This function is supported by PromQL.
See also [quantiles_over_time](#quantiles_over_time).
@@ -670,6 +710,8 @@ over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on
from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
The function returns individual series per each `phi*` with `{phiLabel="phi*"}` label. `phi*` values must be in the range `[0...1]`.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [quantile_over_time](#quantile_over_time).
#### range_over_time
@@ -680,11 +722,12 @@ E.g. it calculates `max_over_time(series_selector[d]) - min_over_time(series_sel
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
#### rate
`rate(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the average per-second increase rate
over the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
It is expected that the `series_selector` returns time series of [counter type](https://docs.victoriametrics.com/keyconcepts/#counter).
If the lookbehind window is skipped in square brackets, then it is automatically calculated as `max(step, scrape_interval)`, where `step` is the query arg value
passed to [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query) or [/api/v1/query](https://docs.victoriametrics.com/keyconcepts/#instant-query),
@@ -703,6 +746,8 @@ See also [irate](#irate) and [rollup_rate](#rollup_rate).
on the given lookbehind window `d`. The calculations are performed individually per each time series returned
from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
#### resets
@@ -710,10 +755,11 @@ Metric names are stripped from the resulting rollups. Add [keep_metric_names](#k
`resets(series_selector[d])` is a [rollup function](#rollup-functions), which returns the number
of [counter](https://docs.victoriametrics.com/keyconcepts/#counter) resets over the given lookbehind window `d`
per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
It is expected that the `series_selector` returns time series of [counter type](https://docs.victoriametrics.com/keyconcepts/#counter).
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
This function is supported by PromQL.
#### rollup
@@ -725,6 +771,10 @@ These values are calculated individually per each time series returned from the
Optional 2nd argument `"min"`, `"max"` or `"avg"` can be passed to keep only one calculation result and without adding a label.
See also [label_match](#label_match).
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [rollup_rate](#rollup_rate).
#### rollup_candlestick
`rollup_candlestick(series_selector[d])` is a [rollup function](#rollup-functions), which calculates `open`, `high`, `low` and `close` values (aka OHLC)
@@ -736,6 +786,8 @@ from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#f
Optional 2nd argument `"open"`, `"high"` or `"low"` or `"close"` can be passed to keep only one calculation result and without adding a label.
See also [label_match](#label_match).
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
#### rollup_delta
`rollup_delta(series_selector[d])` is a [rollup function](#rollup-functions), which calculates differences between adjacent [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
@@ -762,6 +814,8 @@ See also [label_match](#label_match).
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
See also [rollup](#rollup) and [rollup_rate](#rollup_rate).
#### rollup_increase
`rollup_increase(series_selector[d])` is a [rollup function](#rollup-functions), which calculates increases for adjacent [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
@@ -774,12 +828,17 @@ See also [label_match](#label_match).
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. See also [rollup_delta](#rollup_delta).
This function is usually applied to [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
See also [rollup](#rollup) and [rollup_rate](#rollup_rate).
#### rollup_rate
`rollup_rate(series_selector[d])` is a [rollup function](#rollup-functions), which calculates per-second change rates
for adjacent [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
on the given lookbehind window `d` and returns `min`, `max` and `avg` values for the calculated per-second change rates
and returns them in time series with `rollup="min"`, `rollup="max"` and `rollup="avg"` additional labels.
The calculations are performed individually per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
See [this article](https://valyala.medium.com/why-irate-from-prometheus-doesnt-capture-spikes-45f9896d7832) in order to understand better
when to use `rollup_rate()`.
@@ -787,10 +846,12 @@ when to use `rollup_rate()`.
Optional 2nd argument `"min"`, `"max"` or `"avg"` can be passed to keep only one calculation result and without adding a label.
See also [label_match](#label_match).
The calculations are performed individually per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
See also [rollup](#rollup) and [rollup_increase](#rollup_increase).
#### rollup_scrape_interval
`rollup_scrape_interval(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the interval in seconds between
@@ -824,6 +885,8 @@ This function is useful for calculating SLI and SLO. Example: `share_gt_over_tim
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [share_le_over_time](#share_le_over_time) and [count_gt_over_time](#count_gt_over_time).
#### share_le_over_time
@@ -838,6 +901,8 @@ the share of time series values for the last 24 hours when memory usage was belo
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [share_gt_over_time](#share_gt_over_time) and [count_le_over_time](#count_le_over_time).
#### share_eq_over_time
@@ -849,6 +914,8 @@ from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#f
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [count_eq_over_time](#count_eq_over_time).
#### stale_samples_over_time
@@ -866,6 +933,8 @@ on the given lookbehind window `d` per each time series returned from the given
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
This function is supported by PromQL.
See also [stdvar_over_time](#stdvar_over_time).
@@ -877,35 +946,43 @@ on the given lookbehind window `d` per each time series returned from the given
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
This function is supported by PromQL.
See also [stddev_over_time](#stddev_over_time).
#### sum_eq_over_time
`sum_eq_over_time(series_selector[d], eq)` is a [rollup function](#rollup-function), which calculates the sum of [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
`sum_eq_over_time(series_selector[d], eq)` is a [rollup function](#rollup-functions), which calculates the sum of [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
values equal to `eq` on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [sum_over_time](#sum_over_time) and [count_eq_over_time](#count_eq_over_time).
#### sum_gt_over_time
`sum_gt_over_time(series_selector[d], gt)` is a [rollup function](#rollup-function), which calculates the sum of [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
`sum_gt_over_time(series_selector[d], gt)` is a [rollup function](#rollup-functions), which calculates the sum of [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
values bigger than `gt` on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [sum_over_time](#sum_over_time) and [count_gt_over_time](#count_gt_over_time).
#### sum_le_over_time
`sum_le_over_time(series_selector[d], le)` is a [rollup function](#rollup-function), which calculates the sum of [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
`sum_le_over_time(series_selector[d], le)` is a [rollup function](#rollup-functions), which calculates the sum of [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
values smaller or equal to `le` on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering).
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [sum_over_time](#sum_over_time) and [count_le_over_time](#count_le_over_time).
#### sum_over_time
@@ -915,6 +992,8 @@ on the given lookbehind window `d` per each time series returned from the given
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
This function is supported by PromQL.
#### sum2_over_time
@@ -924,6 +1003,8 @@ values on the given lookbehind window `d` per each time series returned from the
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
#### timestamp
`timestamp(series_selector[d])` is a [rollup function](#rollup-functions), which returns the timestamp in seconds with millisecond precision
@@ -1001,6 +1082,8 @@ from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#f
Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names.
This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
See also [zscore](#zscore), [range_trim_zscore](#range_trim_zscore) and [outlier_iqr_over_time](#outlier_iqr_over_time).
@@ -2230,7 +2313,8 @@ Any [rollup function](#rollup-functions) for something other than [series select
Nested rollup functions can be implicit thanks to the [implicit query conversions](#implicit-query-conversions).
For example, `delta(sum(m))` is implicitly converted to `delta(sum(default_rollup(m))[1i:1i])`, so it becomes a subquery,
since it contains [default_rollup](#default_rollup) nested into [delta](#delta).
This behavior can be disabled or logged via cmd-line flags `-search.disableImplicitConversion` and `-search.logImplicitConversion` since v1.101.0.
This behavior can be disabled or logged via `-search.disableImplicitConversion` and `-search.logImplicitConversion` command-line flags
starting from [`v1.101.0` release](https://docs.victoriametrics.com/changelog/).
VictoriaMetrics performs subqueries in the following way:
@@ -2263,5 +2347,6 @@ VictoriaMetrics performs the following implicit conversions for incoming queries
For example, `avg_over_time(rate(http_requests_total[5m])[1h])` is automatically converted to `avg_over_time(rate(http_requests_total[5m])[1h:1i])`.
* If something other than [series selector](https://docs.victoriametrics.com/keyconcepts/#filtering)
is passed to [rollup function](#rollup-functions), then a [subquery](#subqueries) with `1i` lookbehind window and `1i` step is automatically formed.
For example, `rate(sum(up))` is automatically converted to `rate((sum(default_rollup(up)))[1i:1i])`.
This behavior can be disabled or logged via cmd-line flags `-search.disableImplicitConversion` and `-search.logImplicitConversion` since v1.101.0.
For example, `rate(sum(up))` is automatically converted to `rate((sum(default_rollup(up)))[1i:1i])`.
This behavior can be disabled or logged via `-search.disableImplicitConversion` and `-search.logImplicitConversion` command-line flags
starting from [`v1.101.0` release](https://docs.victoriametrics.com/changelog/).

View File

@@ -1,32 +1,66 @@
import React, { FC, useRef, useState } from "preact/compat";
import React, { FC, useMemo, useRef, useState } from "preact/compat";
import "./style.scss";
import "uplot/dist/uPlot.min.css";
import useElementSize from "../../../hooks/useElementSize";
import uPlot, { AlignedData } from "uplot";
import { useEffect } from "react";
import useBarHitsOptions from "./hooks/useBarHitsOptions";
import TooltipBarHitsChart from "./TooltipBarHitsChart";
import BarHitsTooltip from "./BarHitsTooltip/BarHitsTooltip";
import { TimeParams } from "../../../types";
import usePlotScale from "../../../hooks/uplot/usePlotScale";
import useReadyChart from "../../../hooks/uplot/useReadyChart";
import useZoomChart from "../../../hooks/uplot/useZoomChart";
import classNames from "classnames";
import { LogHits } from "../../../api/types";
import { addSeries, delSeries, setBand } from "../../../utils/uplot";
import { GraphOptions, GRAPH_STYLES } from "./types";
import BarHitsOptions from "./BarHitsOptions/BarHitsOptions";
import stack from "../../../utils/uplot/stack";
import BarHitsLegend from "./BarHitsLegend/BarHitsLegend";
interface Props {
logHits: LogHits[];
data: AlignedData;
period: TimeParams;
setPeriod: ({ from, to }: { from: Date, to: Date }) => void;
onApplyFilter: (value: string) => void;
}
const BarHitsChart: FC<Props> = ({ data, period, setPeriod }) => {
const BarHitsChart: FC<Props> = ({ logHits, data: _data, period, setPeriod, onApplyFilter }) => {
const [containerRef, containerSize] = useElementSize();
const uPlotRef = useRef<HTMLDivElement>(null);
const [uPlotInst, setUPlotInst] = useState<uPlot>();
const [graphOptions, setGraphOptions] = useState<GraphOptions>({
graphStyle: GRAPH_STYLES.LINE_STEPPED,
stacked: false,
fill: false,
});
const { xRange, setPlotScale } = usePlotScale({ period, setPeriod });
const { onReadyChart, isPanning } = useReadyChart(setPlotScale);
useZoomChart({ uPlotInst, xRange, setPlotScale });
const { options, focusDataIdx } = useBarHitsOptions({ xRange, containerSize, onReadyChart, setPlotScale });
const { data, bands } = useMemo(() => {
return graphOptions.stacked ? stack(_data, () => false) : { data: _data, bands: [] };
}, [graphOptions, _data]);
const { options, series, focusDataIdx } = useBarHitsOptions({
data,
logHits,
bands,
xRange,
containerSize,
onReadyChart,
setPlotScale,
graphOptions
});
useEffect(() => {
if (!uPlotInst) return;
delSeries(uPlotInst);
addSeries(uPlotInst, series, true);
setBand(uPlotInst, series);
uPlotInst.redraw();
}, [series]);
useEffect(() => {
if (!uPlotRef.current) return;
@@ -54,21 +88,31 @@ const BarHitsChart: FC<Props> = ({ data, period, setPeriod }) => {
}, [data]);
return (
<div
className={classNames({
"vm-bar-hits-chart": true,
"vm-bar-hits-chart_panning": isPanning
})}
ref={containerRef}
>
<div className="vm-bar-hits-chart__wrapper">
<div
className="vm-line-chart__u-plot"
ref={uPlotRef}
/>
<TooltipBarHitsChart
uPlotInst={uPlotInst}
focusDataIdx={focusDataIdx}
/>
className={classNames({
"vm-bar-hits-chart": true,
"vm-bar-hits-chart_panning": isPanning
})}
ref={containerRef}
>
<div
className="vm-line-chart__u-plot"
ref={uPlotRef}
/>
<BarHitsTooltip
uPlotInst={uPlotInst}
data={_data}
focusDataIdx={focusDataIdx}
/>
</div>
<BarHitsOptions onChange={setGraphOptions}/>
{uPlotInst && (
<BarHitsLegend
uPlotInst={uPlotInst}
onApplyFilter={onApplyFilter}
/>
)}
</div>
);
};

View File

@@ -0,0 +1,68 @@
import React, { FC, useCallback, useEffect, useState } from "preact/compat";
import uPlot, { Series } from "uplot";
import "./style.scss";
import "../../Line/Legend/style.scss";
import classNames from "classnames";
import { MouseEvent } from "react";
import { isMacOs } from "../../../../utils/detect-device";
import Tooltip from "../../../Main/Tooltip/Tooltip";
interface Props {
uPlotInst: uPlot;
onApplyFilter: (value: string) => void;
}
const BarHitsLegend: FC<Props> = ({ uPlotInst, onApplyFilter }) => {
const [series, setSeries] = useState<Series[]>([]);
const updateSeries = useCallback(() => {
const series = uPlotInst.series.filter(s => s.scale !== "x");
setSeries(series);
}, [uPlotInst]);
const handleClick = (target: Series) => (e: MouseEvent<HTMLDivElement>) => {
const metaKey = e.metaKey || e.ctrlKey;
if (!metaKey) {
target.show = !target.show;
} else {
onApplyFilter(target.label || "");
}
updateSeries();
uPlotInst.redraw();
};
useEffect(updateSeries, [uPlotInst]);
return (
<div className="vm-bar-hits-legend">
{series.map(s => (
<Tooltip
key={s.label}
title={(
<ul className="vm-bar-hits-legend-info">
<li>Click to {s.show ? "hide" : "show"} the _stream.</li>
<li>{isMacOs() ? "Cmd" : "Ctrl"} + Click to filter by the _stream.</li>
</ul>
)}
>
<div
className={classNames({
"vm-bar-hits-legend-item": true,
"vm-bar-hits-legend-item_hide": !s.show,
})}
onClick={handleClick(s)}
>
<div
className="vm-bar-hits-legend-item__marker"
style={{ backgroundColor: `${(s?.stroke as () => string)?.()}` }}
/>
<div>{s.label}</div>
</div>
</Tooltip>
))}
</div>
);
};
export default BarHitsLegend;

View File

@@ -0,0 +1,35 @@
@use "src/styles/variables" as *;
.vm-bar-hits-legend {
display: flex;
flex-wrap: wrap;
gap: 0;
padding: 0 $padding-small $padding-small;
&-item {
display: grid;
grid-template-columns: auto 1fr;
align-items: center;
gap: 4px;
font-size: 12px;
padding: $padding-small;
border-radius: $border-radius-small;
cursor: pointer;
transition: 0.2s;
&:hover {
background-color: rgba(0, 0, 0, 0.05);
}
&_hide {
text-decoration: line-through;
opacity: 0.5;
}
&__marker {
width: 14px;
height: 14px;
border: $color-background-block;
}
}
}

View File

@@ -0,0 +1,116 @@
import React, { FC, useEffect, useMemo, useRef } from "preact/compat";
import { GraphOptions, GRAPH_STYLES } from "../types";
import Switch from "../../../Main/Switch/Switch";
import "./style.scss";
import useStateSearchParams from "../../../../hooks/useStateSearchParams";
import { useSearchParams } from "react-router-dom";
import Button from "../../../Main/Button/Button";
import classNames from "classnames";
import { SettingsIcon } from "../../../Main/Icons";
import Tooltip from "../../../Main/Tooltip/Tooltip";
import Popper from "../../../Main/Popper/Popper";
import useBoolean from "../../../../hooks/useBoolean";
interface Props {
onChange: (options: GraphOptions) => void;
}
const BarHitsOptions: FC<Props> = ({ onChange }) => {
const [searchParams, setSearchParams] = useSearchParams();
const optionsButtonRef = useRef<HTMLDivElement>(null);
const {
value: openOptions,
toggle: toggleOpenOptions,
setFalse: handleCloseOptions,
} = useBoolean(false);
const [graphStyle, setGraphStyle] = useStateSearchParams(GRAPH_STYLES.LINE_STEPPED, "graph");
const [stacked, setStacked] = useStateSearchParams(false, "stacked");
const [fill, setFill] = useStateSearchParams(false, "fill");
const options: GraphOptions = useMemo(() => ({
graphStyle,
stacked,
fill,
}), [graphStyle, stacked, fill]);
const handleChangeGraphStyle = (val: string) => () => {
setGraphStyle(val as GRAPH_STYLES);
searchParams.set("graph", val);
setSearchParams(searchParams);
};
const handleChangeFill = (val: boolean) => {
setFill(val);
val ? searchParams.set("fill", "true") : searchParams.delete("fill");
setSearchParams(searchParams);
};
const handleChangeStacked = (val: boolean) => {
setStacked(val);
val ? searchParams.set("stacked", "true") : searchParams.delete("stacked");
setSearchParams(searchParams);
};
useEffect(() => {
onChange(options);
}, [options]);
return (
<div
className="vm-bar-hits-options"
ref={optionsButtonRef}
>
<Tooltip title="Graph settings">
<Button
variant="text"
color="primary"
startIcon={<SettingsIcon/>}
onClick={toggleOpenOptions}
ariaLabel="settings"
/>
</Tooltip>
<Popper
open={openOptions}
placement="bottom-right"
onClose={handleCloseOptions}
buttonRef={optionsButtonRef}
title={"Graph settings"}
>
<div className="vm-bar-hits-options-settings">
<div className="vm-bar-hits-options-settings-item vm-bar-hits-options-settings-item_list">
<p className="vm-bar-hits-options-settings-item__title">Graph style:</p>
{Object.values(GRAPH_STYLES).map(style => (
<div
key={style}
className={classNames({
"vm-list-item": true,
"vm-list-item_active": graphStyle === style,
})}
onClick={handleChangeGraphStyle(style)}
>
{style}
</div>
))}
</div>
<div className="vm-bar-hits-options-settings-item">
<Switch
label={"Stacked"}
value={stacked}
onChange={handleChangeStacked}
/>
</div>
<div className="vm-bar-hits-options-settings-item">
<Switch
label={"Fill"}
value={fill}
onChange={handleChangeFill}
/>
</div>
</div>
</Popper>
</div>
);
};
export default BarHitsOptions;

View File

@@ -0,0 +1,35 @@
@use "src/styles/variables" as *;
.vm-bar-hits-options {
position: absolute;
top: $padding-small;
right: $padding-small;
z-index: 2;
&-settings {
display: grid;
align-items: flex-start;
gap: $padding-global;
min-width: 200px;
&-item {
border-bottom: $border-divider;
padding: 0 $padding-global $padding-global;
&_list {
padding: 0;
}
&__title {
font-size: $font-size-small;
color: $color-text-secondary;
padding: 0 $padding-small $padding-small;
}
&:last-child {
border-bottom: none;
}
}
}
}

View File

@@ -0,0 +1,125 @@
import React, { FC, useMemo, useRef } from "preact/compat";
import uPlot, { AlignedData } from "uplot";
import dayjs from "dayjs";
import { DATE_TIME_FORMAT } from "../../../../constants/date";
import classNames from "classnames";
import "./style.scss";
import "../../ChartTooltip/style.scss";
interface Props {
data: AlignedData;
uPlotInst?: uPlot;
focusDataIdx: number;
}
const BarHitsTooltip: FC<Props> = ({ data, focusDataIdx, uPlotInst }) => {
const tooltipRef = useRef<HTMLDivElement>(null);
const tooltipData = useMemo(() => {
const series = uPlotInst?.series || [];
const [time, ...values] = data.map((d) => d[focusDataIdx] || 0);
const tooltipItems = values.map((value, i) => {
const targetSeries = series[i + 1];
const stroke = (targetSeries?.stroke as () => string)?.();
const label = targetSeries?.label || "other";
const show = targetSeries?.show;
return {
label,
stroke,
value,
show
};
}).filter(item => item.value > 0 && item.show).sort((a, b) => b.value - a.value);
const point = {
top: tooltipItems[0] ? uPlotInst?.valToPos?.(tooltipItems[0].value, "y") || 0 : 0,
left: uPlotInst?.valToPos?.(time, "x") || 0,
};
return {
point,
values: tooltipItems,
total: tooltipItems.reduce((acc, item) => acc + item.value, 0),
timestamp: dayjs(time * 1000).tz().format(DATE_TIME_FORMAT),
};
}, [focusDataIdx, uPlotInst, data]);
const tooltipPosition = useMemo(() => {
if (!uPlotInst || !tooltipData.total || !tooltipRef.current) return;
const { top, left } = tooltipData.point;
const uPlotPosition = {
left: parseFloat(uPlotInst.over.style.left),
top: parseFloat(uPlotInst.over.style.top)
};
const {
width: uPlotWidth,
height: uPlotHeight
} = uPlotInst.over.getBoundingClientRect();
const {
width: tooltipWidth,
height: tooltipHeight
} = tooltipRef.current.getBoundingClientRect();
const margin = 50;
const overflowX = left + tooltipWidth >= uPlotWidth ? tooltipWidth + (2 * margin) : 0;
const overflowY = top + tooltipHeight >= uPlotHeight ? tooltipHeight + (2 * margin) : 0;
const position = {
top: top + uPlotPosition.top + margin - overflowY,
left: left + uPlotPosition.left + margin - overflowX
};
if (position.left < 0) position.left = 20;
if (position.top < 0) position.top = 20;
return position;
}, [tooltipData, uPlotInst, tooltipRef.current]);
return (
<div
className={classNames({
"vm-chart-tooltip": true,
"vm-chart-tooltip_hits": true,
"vm-bar-hits-tooltip": true,
"vm-bar-hits-tooltip_visible": focusDataIdx !== -1 && tooltipData.values.length
})}
ref={tooltipRef}
style={tooltipPosition}
>
<div>
{tooltipData.values.map((item, i) => (
<div
className="vm-chart-tooltip-data"
key={i}
>
<span
className="vm-chart-tooltip-data__marker"
style={{ background: item.stroke }}
/>
<p>
{item.label}: <b>{item.value}</b>
</p>
</div>
))}
</div>
{tooltipData.values.length > 1 && (
<div className="vm-chart-tooltip-data">
<p>
Total records: <b>{tooltipData.total}</b>
</p>
</div>
)}
<div className="vm-chart-tooltip-header">
<div className="vm-chart-tooltip-header__title">
{tooltipData.timestamp}
</div>
</div>
</div>
);
};
export default BarHitsTooltip;

View File

@@ -0,0 +1,12 @@
@use "src/styles/variables" as *;
.vm-bar-hits-tooltip {
opacity: 0;
pointer-events: none;
gap: $padding-small;
&_visible {
opacity: 1;
pointer-events: auto;
}
}

View File

@@ -1,89 +0,0 @@
import React, { FC, useMemo, useRef } from "preact/compat";
import uPlot from "uplot";
import dayjs from "dayjs";
import { DATE_TIME_FORMAT } from "../../../constants/date";
import classNames from "classnames";
import "./style.scss";
import "../../../components/Chart/ChartTooltip/style.scss";
interface Props {
uPlotInst?: uPlot;
focusDataIdx: number
}
const TooltipBarHitsChart: FC<Props> = ({ focusDataIdx, uPlotInst }) => {
const tooltipRef = useRef<HTMLDivElement>(null);
const tooltipData = useMemo(() => {
const value = uPlotInst?.data?.[1]?.[focusDataIdx];
const timestamp = uPlotInst?.data?.[0]?.[focusDataIdx] || 0;
const top = uPlotInst?.valToPos?.((value || 0), "y") || 0;
const left = uPlotInst?.valToPos?.(timestamp, "x") || 0;
return {
point: { top, left },
value,
timestamp: dayjs(timestamp * 1000).tz().format(DATE_TIME_FORMAT),
};
}, [focusDataIdx, uPlotInst]);
const tooltipPosition = useMemo(() => {
if (!uPlotInst || !tooltipData.value || !tooltipRef.current) return;
const { top, left } = tooltipData.point;
const uPlotPosition = {
left: parseFloat(uPlotInst.over.style.left),
top: parseFloat(uPlotInst.over.style.top)
};
const {
width: uPlotWidth,
height: uPlotHeight
} = uPlotInst.over.getBoundingClientRect();
const {
width: tooltipWidth,
height: tooltipHeight
} = tooltipRef.current.getBoundingClientRect();
const margin = 10;
const overflowX = left + tooltipWidth >= uPlotWidth ? tooltipWidth + (2 * margin) : 0;
const overflowY = top + tooltipHeight >= uPlotHeight ? tooltipHeight + (2 * margin) : 0;
const position = {
top: top + uPlotPosition.top + margin - overflowY,
left: left + uPlotPosition.left + margin - overflowX
};
if (position.left < 0) position.left = 20;
if (position.top < 0) position.top = 20;
return position;
}, [tooltipData, uPlotInst, tooltipRef.current]);
return (
<div
className={classNames({
"vm-chart-tooltip": true,
"vm-bar-hits-chart-tooltip": true,
"vm-bar-hits-chart-tooltip_visible": focusDataIdx !== -1
})}
ref={tooltipRef}
style={tooltipPosition}
>
<div className="vm-chart-tooltip-data">
Count of records:
<p className="vm-chart-tooltip-data__value">
<b>{tooltipData.value}</b>
</p>
</div>
<div className="vm-chart-tooltip-header">
<div className="vm-chart-tooltip-header__title">
{tooltipData.timestamp}
</div>
</div>
</div>
);
};
export default TooltipBarHitsChart;

View File

@@ -2,42 +2,81 @@ import { useMemo, useState } from "preact/compat";
import { getAxes, handleDestroy, setSelect } from "../../../../utils/uplot";
import dayjs from "dayjs";
import { dateFromSeconds, formatDateForNativeInput } from "../../../../utils/time";
import uPlot, { Options } from "uplot";
import uPlot, { AlignedData, Band, Options, Series } from "uplot";
import { getCssVariable } from "../../../../utils/theme";
import { barPaths } from "../../../../utils/uplot/bars";
import { useAppState } from "../../../../state/common/StateContext";
import { MinMax, SetMinMax } from "../../../../types";
import { LogHits } from "../../../../api/types";
import getSeriesPaths from "../../../../utils/uplot/paths";
import { GraphOptions, GRAPH_STYLES } from "../types";
const seriesColors = [
"color-log-hits-bar-1",
"color-log-hits-bar-2",
"color-log-hits-bar-3",
"color-log-hits-bar-4",
"color-log-hits-bar-5",
];
const strokeWidth = {
[GRAPH_STYLES.BAR]: 0.8,
[GRAPH_STYLES.LINE_STEPPED]: 1.2,
[GRAPH_STYLES.LINE]: 1.2,
[GRAPH_STYLES.POINTS]: 0,
};
interface UseGetBarHitsOptionsArgs {
data: AlignedData;
logHits: LogHits[];
xRange: MinMax;
bands?: Band[];
containerSize: { width: number, height: number };
setPlotScale: SetMinMax;
onReadyChart: (u: uPlot) => void;
graphOptions: GraphOptions;
}
const useBarHitsOptions = ({ xRange, containerSize, onReadyChart, setPlotScale }: UseGetBarHitsOptionsArgs) => {
const useBarHitsOptions = ({
data,
logHits,
xRange,
bands,
containerSize,
onReadyChart,
setPlotScale,
graphOptions
}: UseGetBarHitsOptionsArgs) => {
const { isDarkTheme } = useAppState();
const [focusDataIdx, setFocusDataIdx] = useState(-1);
const series = useMemo(() => [
{},
{
label: "y",
width: 1,
stroke: getCssVariable("color-log-hits-bar"),
fill: getCssVariable("color-log-hits-bar"),
paths: barPaths,
}
], [isDarkTheme]);
const setCursor = (u: uPlot) => {
const dataIdx = u.cursor.idx ?? -1;
setFocusDataIdx(dataIdx);
};
const series: Series[] = useMemo(() => {
let colorN = 0;
return data.map((_d, i) => {
if (i === 0) return {}; // 0 index is xAxis(timestamps)
const fields = Object.values(logHits?.[i - 1]?.fields || {});
const label = fields.map((value) => value || "\"\"").join(", ");
const color = getCssVariable(label ? seriesColors[colorN] : "color-log-hits-bar-0");
if (label) colorN++;
return {
label: label || "other",
width: strokeWidth[graphOptions.graphStyle],
spanGaps: true,
stroke: color,
fill: graphOptions.fill ? color + "80" : "",
paths: getSeriesPaths(graphOptions.graphStyle),
};
});
}, [isDarkTheme, data, graphOptions]);
const options: Options = useMemo(() => ({
series,
bands,
width: containerSize.width || (window.innerWidth / 2),
height: containerSize.height || 200,
cursor: {
@@ -55,6 +94,7 @@ const useBarHitsOptions = ({ xRange, containerSize, onReadyChart, setPlotScale }
}
},
hooks: {
drawSeries: [],
ready: [onReadyChart],
setCursor: [setCursor],
setSelect: [setSelect(setPlotScale)],
@@ -63,10 +103,11 @@ const useBarHitsOptions = ({ xRange, containerSize, onReadyChart, setPlotScale }
legend: { show: false },
axes: getAxes([{}, { scale: "y" }]),
tzDate: ts => dayjs(formatDateForNativeInput(dateFromSeconds(ts))).local().toDate(),
}), [isDarkTheme]);
}), [isDarkTheme, series, bands]);
return {
options,
series,
focusDataIdx,
};
};

View File

@@ -1,22 +1,18 @@
@use "src/styles/variables" as *;
.vm-bar-hits-chart {
height: 100%;
position: relative;
width: 100%;
height: 200px;
&__wrapper {
display: flex;
flex-direction: column;
width: 100%;
height: 100%;
}
&_panning {
pointer-events: none;
}
&-tooltip {
opacity: 0;
pointer-events: none;
width: 240px;
gap: $padding-small;
&_visible {
opacity: 1;
pointer-events: auto;
}
}
}

View File

@@ -0,0 +1,12 @@
export enum GRAPH_STYLES {
BAR = "Bars",
LINE = "Lines",
LINE_STEPPED = "Stepped lines",
POINTS = "Points",
}
export interface GraphOptions {
graphStyle: GRAPH_STYLES;
stacked: boolean;
fill: boolean;
}

View File

@@ -25,6 +25,12 @@ $chart-tooltip-y: -1 * ($padding-global + $chart-tooltip-half-icon);
user-select: text;
pointer-events: none;
&_hits {
white-space: pre-wrap;
word-break: break-all;
width: auto;
}
&_sticky {
pointer-events: auto;
z-index: 99;
@@ -74,10 +80,22 @@ $chart-tooltip-y: -1 * ($padding-global + $chart-tooltip-half-icon);
justify-content: flex-start;
gap: $padding-small;
&_margin-bottom {
margin-bottom: $padding-global;
}
&_margin-top {
margin-top: $padding-global;
}
&__marker {
width: $font-size;
height: $font-size;
border: 1px solid rgba($color-white, 0.5);
&_tranparent {
opacity: 0;
}
}
&__value {

View File

@@ -13,6 +13,7 @@ import ThemeControl from "../ThemeControl/ThemeControl";
import useDeviceDetect from "../../../hooks/useDeviceDetect";
import useBoolean from "../../../hooks/useBoolean";
import { AppType } from "../../../types/appType";
import SwitchMarkdownParsing from "../LogsSettings/MarkdownParsing/SwitchMarkdownParsing";
const title = "Settings";
@@ -60,6 +61,10 @@ const GlobalSettings: FC = () => {
onClose={handleClose}
/>
},
{
show: isLogsApp,
component: <SwitchMarkdownParsing/>
},
{
show: true,
component: <Timezones ref={timezoneSettingRef}/>

View File

@@ -0,0 +1,150 @@
import React, { FC, useRef } from "preact/compat";
import { useTimeDispatch } from "../../../../state/time/TimeStateContext";
import { ArrowDownIcon, QuestionIcon, StorageIcon } from "../../../Main/Icons";
import Button from "../../../Main/Button/Button";
import "./style.scss";
import "../../TimeRangeSettings/ExecutionControls/style.scss";
import classNames from "classnames";
import Popper from "../../../Main/Popper/Popper";
import { getAppModeEnable } from "../../../../utils/app-mode";
import Tooltip from "../../../Main/Tooltip/Tooltip";
import useDeviceDetect from "../../../../hooks/useDeviceDetect";
import TextField from "../../../Main/TextField/TextField";
import useBoolean from "../../../../hooks/useBoolean";
import useStateSearchParams from "../../../../hooks/useStateSearchParams";
import { useSearchParams } from "react-router-dom";
import { useEffect } from "react";
const TenantsFields: FC = () => {
const appModeEnable = getAppModeEnable();
const { isMobile } = useDeviceDetect();
const timeDispatch = useTimeDispatch();
const [searchParams, setSearchParams] = useSearchParams();
const [accountID, setAccountID] = useStateSearchParams("0", "accountID");
const [projectID, setProjectID] = useStateSearchParams("0", "projectID");
const formattedTenant = `${accountID}:${projectID}`;
const buttonRef = useRef<HTMLDivElement>(null);
const {
value: openPopup,
toggle: toggleOpenPopup,
setFalse: handleClosePopup,
} = useBoolean(false);
const applyChanges = () => {
searchParams.set("accountID", accountID);
searchParams.set("projectID", projectID);
setSearchParams(searchParams);
handleClosePopup();
timeDispatch({ type: "RUN_QUERY" });
};
const handleReset = () => {
setAccountID(searchParams.get("accountID") || "0");
setProjectID(searchParams.get("projectID") || "0");
};
useEffect(() => {
if (openPopup) return;
handleReset();
}, [openPopup]);
return (
<div className="vm-tenant-input">
<Tooltip title="Define Tenant ID if you need request to another storage">
<div ref={buttonRef}>
{isMobile ? (
<div
className="vm-mobile-option"
onClick={toggleOpenPopup}
>
<span className="vm-mobile-option__icon"><StorageIcon/></span>
<div className="vm-mobile-option-text">
<span className="vm-mobile-option-text__label">Tenant ID</span>
<span className="vm-mobile-option-text__value">{formattedTenant}</span>
</div>
<span className="vm-mobile-option__arrow"><ArrowDownIcon/></span>
</div>
) : (
<Button
className={appModeEnable ? "" : "vm-header-button"}
variant="contained"
color="primary"
fullWidth
startIcon={<StorageIcon/>}
endIcon={(
<div
className={classNames({
"vm-execution-controls-buttons__arrow": true,
"vm-execution-controls-buttons__arrow_open": openPopup,
})}
>
<ArrowDownIcon/>
</div>
)}
onClick={toggleOpenPopup}
>
{formattedTenant}
</Button>
)}
</div>
</Tooltip>
<Popper
open={openPopup}
placement="bottom-right"
onClose={handleClosePopup}
buttonRef={buttonRef}
title={isMobile ? "Define Tenant ID" : undefined}
>
<div
className={classNames({
"vm-list vm-tenant-input-list": true,
"vm-list vm-tenant-input-list_mobile": isMobile,
"vm-tenant-input-list_inline": true,
})}
>
<TextField
autofocus
label="accountID"
value={accountID}
onChange={setAccountID}
type="number"
/>
<TextField
autofocus
label="projectID"
value={projectID}
onChange={setProjectID}
type="number"
/>
<div className="vm-tenant-input-list__buttons">
<Tooltip title="Multitenancy in VictoriaLogs documentation">
<a
href="https://docs.victoriametrics.com/victorialogs/#multitenancy"
target="_blank"
rel="help noreferrer"
>
<Button
variant="text"
color="gray"
startIcon={<QuestionIcon/>}
/>
</a>
</Tooltip>
<Button
variant="contained"
color="primary"
onClick={applyChanges}
>
Apply
</Button>
</div>
</div>
</Popper>
</div>
);
};
export default TenantsFields;

View File

@@ -17,11 +17,23 @@
padding: 0 $padding-global $padding-small;
}
&_inline {
display: grid;
gap: calc($padding-small/2);
padding: $padding-global;
}
&__search {
position: sticky;
top: 0;
padding: $padding-small $padding-global;
background-color: $color-background-block;
}
&__buttons {
display: flex;
justify-content: space-between;
gap: $padding-small;
}
}
}

View File

@@ -4,7 +4,7 @@
display: flex;
flex-direction: column;
align-items: center;
gap: $padding-medium;
gap: $padding-large;
width: 600px;
padding-bottom: $padding-medium;
@@ -39,6 +39,13 @@
margin-bottom: $padding-global;
}
&__info {
padding-top: $padding-small;
font-size: $font-size-small;
color: $color-text-secondary;
line-height: 130%;
}
&-url {
display: flex;
align-items: flex-start;

View File

@@ -0,0 +1,35 @@
import React, { FC } from "preact/compat";
import Switch from "../../../Main/Switch/Switch";
import useDeviceDetect from "../../../../hooks/useDeviceDetect";
import { useLogsDispatch, useLogsState } from "../../../../state/logsPanel/LogsStateContext";
const SwitchMarkdownParsing: FC = () => {
const { isMobile } = useDeviceDetect();
const { markdownParsing } = useLogsState();
const dispatch = useLogsDispatch();
const handleChangeMarkdownParsing = (val: boolean) => {
dispatch({ type: "SET_MARKDOWN_PARSING", payload: val });
};
return (
<div>
<div className="vm-server-configurator__title">
Markdown Parsing for Logs
</div>
<Switch
label={markdownParsing ? "Disable markdown parsing" : "Enable markdown parsing"}
value={markdownParsing}
onChange={handleChangeMarkdownParsing}
fullWidth={isMobile}
/>
<div className="vm-server-configurator__info">
Toggle this switch to enable or disable the Markdown formatting for log entries.
Enabling this will parse log texts to Markdown.
</div>
</div>
);
};
export default SwitchMarkdownParsing;

View File

@@ -34,14 +34,25 @@ const QueryEditorAutocomplete: FC<QueryEditorAutocompleteProps> = ({
}, [value, caretPosition]);
const exprLastPart = useMemo(() => {
const parts = values.beforeCursor.split("}");
const regexpSplit = /\s(or|and|unless|default|ifnot|if|group_left|group_right)\s|}|\+|\|-|\*|\/|\^/i;
const parts = values.beforeCursor.split(regexpSplit);
return parts[parts.length - 1];
}, [values]);
const metric = useMemo(() => {
const regexp = /\b[^{}(),\s]+(?={|$)/g;
const match = exprLastPart.match(regexp);
return match ? match[0] : "";
const regex1 = /\w+\((?<metricName>[^)]+)\)\s+(by|without|on|ignoring)\s*\(\w*/gi;
const matchAlt = [...exprLastPart.matchAll(regex1)];
if (matchAlt.length > 0 && matchAlt[0].groups && matchAlt[0].groups.metricName) {
return matchAlt[0].groups.metricName;
}
const regex2 = /^\s*\b(?<metricName>[^{}(),\s]+)(?={|$)/g;
const match = [...exprLastPart.matchAll(regex2)];
if (match.length > 0 && match[0].groups && match[0].groups.metricName) {
return match[0].groups.metricName;
}
return "";
}, [exprLastPart]);
const label = useMemo(() => {
@@ -51,7 +62,7 @@ const QueryEditorAutocomplete: FC<QueryEditorAutocompleteProps> = ({
}, [exprLastPart]);
const shouldSuppressAutoSuggestion = (value: string) => {
const pattern = /([{(),+\-*/^]|\b(?:or|and|unless|default|ifnot|if|group_left|group_right)\b)/;
const pattern = /([{(),+\-*/^]|\b(?:or|and|unless|default|ifnot|if|group_left|group_right|by|without|on|ignoring)\b)/i;
const parts = value.split(/\s+/);
const partsCount = parts.length;
const lastPart = parts[partsCount - 1];
@@ -63,12 +74,16 @@ const QueryEditorAutocomplete: FC<QueryEditorAutocompleteProps> = ({
};
const context = useMemo(() => {
if (!values.beforeCursor || values.beforeCursor.endsWith("}") || shouldSuppressAutoSuggestion(values.beforeCursor)) {
const valueBeforeCursor = values.beforeCursor.trim();
const endOfClosedBrackets = ["}", ")"].some(char => valueBeforeCursor.endsWith(char));
const endOfClosedQuotes = !hasUnclosedQuotes(valueBeforeCursor) && ["`", "'", "\""].some(char => valueBeforeCursor.endsWith(char));
if (!values.beforeCursor || endOfClosedBrackets || endOfClosedQuotes || shouldSuppressAutoSuggestion(values.beforeCursor)) {
return QueryContextType.empty;
}
const labelRegexp = /\{[^}]*$/;
const labelValueRegexp = new RegExp(`(${escapeRegexp(metric)})?{?.+${escapeRegexp(label)}(=|!=|=~|!~)"?([^"]*)$`, "g");
const labelRegexp = /(?:by|without|on|ignoring)\s*\(\s*[^)]*$|\{[^}]*$/i;
const patternLabelValue = `(${escapeRegexp(metric)})?{?.+${escapeRegexp(label)}(=|!=|=~|!~)"?([^"]*)$`;
const labelValueRegexp = new RegExp(patternLabelValue, "g");
switch (true) {
case labelValueRegexp.test(values.beforeCursor):
@@ -81,7 +96,7 @@ const QueryEditorAutocomplete: FC<QueryEditorAutocompleteProps> = ({
}, [values, metric, label]);
const valueByContext = useMemo(() => {
const wordMatch = values.beforeCursor.match(/([\w_\-.:/]+(?![},]))$/);
const wordMatch = values.beforeCursor.match(/([\w_.:]+(?![},]))$/);
return wordMatch ? wordMatch[0] : "";
}, [values.beforeCursor]);
@@ -119,9 +134,10 @@ const QueryEditorAutocomplete: FC<QueryEditorAutocompleteProps> = ({
// Add quotes around the value if the context is labelValue
if (context === QueryContextType.labelValue) {
const quote = "\"";
const needsQuote = /(?:=|!=|=~|!~)$/.test(beforeValueByContext);
valueAfterCursor = valueAfterCursor.replace(/^[^\s"|},]*/, "");
insert = `${needsQuote ? quote : ""}${insert}`;
const needsOpenQuote = /(?:=|!=|=~|!~)$/.test(beforeValueByContext);
const needsCloseQuote = valueAfterCursor.trim()[0] !== "\"";
insert = `${needsOpenQuote ? quote : ""}${insert}${needsCloseQuote ? quote : ""}`;
}
if (context === QueryContextType.label) {

View File

@@ -63,7 +63,7 @@ const AnomalyConfig: FC = () => {
URL.revokeObjectURL(downloadUrl);
setTextConfig("");
setDownloadUrl("");
fetchConfig();
return fetchConfig();
};
return (

View File

@@ -520,3 +520,36 @@ export const DownloadIcon = () => (
<path d="M19 9h-4V3H9v6H5l7 7 7-7zM5 18v2h14v-2H5z"></path>
</svg>
);
export const ExpandIcon = () => (
<svg
viewBox="0 0 24 24"
fill="currentColor"
>
<path
d="M12 5.83 15.17 9l1.41-1.41L12 3 7.41 7.59 8.83 9zm0 12.34L8.83 15l-1.41 1.41L12 21l4.59-4.59L15.17 15z"
></path>
</svg>
);
export const CollapseIcon = () => (
<svg
viewBox="0 0 24 24"
fill="currentColor"
>
<path
d="M7.41 18.59 8.83 20 12 16.83 15.17 20l1.41-1.41L12 14zm9.18-13.18L15.17 4 12 7.17 8.83 4 7.41 5.41 12 10z"
></path>
</svg>
);
export const SearchIcon = () => (
<svg
viewBox="0 0 24 24"
fill="currentColor"
>
<path
d="M15.5 14h-.79l-.28-.27C15.41 12.59 16 11.11 16 9.5 16 5.91 13.09 3 9.5 3S3 5.91 3 9.5 5.91 16 9.5 16c1.61 0 3.09-.59 4.23-1.57l.27.28v.79l5 4.99L20.49 19zm-6 0C7.01 14 5 11.99 5 9.5S7.01 5 9.5 5 14 7.01 14 9.5 11.99 14 9.5 14"
></path>
</svg>
);

Some files were not shown because too many files have changed in this diff Show More