Compare commits

..

50 Commits

Author SHA1 Message Date
f41gh7
8fef688d48 apptest/tests: adds an example of replication performance test
It's not intended to be merged. It takes too much time to execute -
 20-30 seconds.
2025-01-28 15:27:26 +01:00
Fred Navruzov
d8ad3310a5 docs/vmanomaly: release v1.19.2 (#8165)
### Describe Your Changes

docs/vmanomaly: release v1.19.2 (patch that addresses some of the bugs
found in 1.19.0)

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Mathias Palmersheim <mathias@victoriametrics.com>
2025-01-27 23:10:58 +01:00
Aliaksandr Valialkin
16fafdd568 vendor: run make vendor-update 2025-01-27 22:27:49 +01:00
Aliaksandr Valialkin
585ff968f4 lib/fs/fsutil: move lib/envutil to the more appropriate place at lib/fs/fsutil
This is a follow-up for 043d066133
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6871
2025-01-27 18:47:53 +01:00
Zakhar Bessarab
35ddfc5d62 docs/{vmauth,VictoriaLogs}: add examples of using headers per-user
### Describe Your Changes
- add example a generic example to vmauth docs
- add an multi-tenancy usage example to VictoriaLogs docs
### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit de144899b2)
2025-01-27 20:16:03 +04:00
Aliaksandr Valialkin
d60238d917 lib/storage: open per-month partitions in parallel
This should reduce the time needed for opening the storage with retentions exceeding a few months.

While at at, limit the concurrency of opening partitions in parallel to the number of available CPU cores,
since higher concurrency may increase RAM usage and CPU usage without performance improvements
if opening a single partition is CPU-bound task.

This is a follow-up for 17988942ab
2025-01-27 16:12:25 +01:00
Aliaksandr Valialkin
bca61bdd35 lib/filestream: use correct formatting option for error type in the error message 2025-01-27 15:24:11 +01:00
f41gh7
6168ab3b3d docs/changelog: mention lts releases
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2025-01-27 11:06:12 +01:00
f41gh7
3fc6de40cf docs: point apps versions to the latest releases
v1.110.0
v1.102.11
v1.97.16

Signed-off-by: f41gh7 <nik@victoriametrics.com>
2025-01-27 11:06:11 +01:00
f41gh7
08404f2169 CHANGELOG.md: cut v1.110.0 release 2025-01-27 11:06:11 +01:00
Aliaksandr Valialkin
c51282d698 lib/logstorage: open per-day partitions in parallel during startup
This significantly reduces startup times when the storage contains large partitions over many days.
2025-01-27 00:35:35 +01:00
Aliaksandr Valialkin
b01b02b40c lib/logstorage: optimize unmarshalColumnNames a bit
This should reduce the time needed for opening a large storage with many partitions,
which contain logs with big number of fields (aka wide events).

Thanks to @kiriklo for the initial idea at the pull request https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8061

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7937
2025-01-27 00:14:42 +01:00
Aliaksandr Valialkin
dbe8e9df09 lib/logstorage: improve error message by adding a link with the explanation why VictoriaLogs ignores logs with the size exceeding 2MB
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7972
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7984
2025-01-26 22:53:15 +01:00
Aliaksandr Valialkin
4f9e51e106 lib/logstorage: block_stat pipe: return the path to the part where the block is stored 2025-01-26 22:53:15 +01:00
Aliaksandr Valialkin
a0ca7e9eea docs/VictoriaLogs: small updates 2025-01-26 22:07:45 +01:00
Aliaksandr Valialkin
ee69dd028a lib/{fs,filestream}: unconditionally disable fsync in tests
Use the testing.Testing() function in order to determine whether the code runs in test.
This allows running tests and fast speed without the need to specify DISABLE_FSYNC_FOR_TESTING
environment variable.

This is a follow-up for the commit 334cd92a6c
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6871
2025-01-26 22:07:44 +01:00
Aliaksandr Valialkin
4908f98ee2 lib/logstorage: remove unneeded code after 202eb429a7
readerWithStats isn't used when reading column names from file
2025-01-26 22:07:44 +01:00
Aliaksandr Valialkin
c252d2651b docs/VictoriaLogs/sql-to-logsql.md: pay attention to the fact that stats() pipe at LogsQL has better usability than GROUP BY at SQL 2025-01-26 22:07:44 +01:00
Aliaksandr Valialkin
6a92e3f348 deployment/docker: update VictoriaLogs from v1.7.0-victorialogs to v1.8.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.8.0-victorialogs
2025-01-26 22:07:43 +01:00
Aliaksandr Valialkin
96246c0e04 docs/VictoriaLogs/CHANGELOG.md: cut v1.8.0-victorialogs release 2025-01-26 22:06:47 +01:00
Aliaksandr Valialkin
7ece9ac198 app/vlselect/vmui: run make vmui-logs-update after the commit 87739bbbef 2025-01-26 22:06:46 +01:00
Aliaksandr Valialkin
9754f4b298 docs/VictoriaLogs/CHANGELOG.md: move the changes from 87739bbbef to the correct place
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7750
2025-01-26 22:06:46 +01:00
Yury Molodov
70d7f4aba9 vmui/logs: update hits chart legend (#7750)
### Describe Your Changes

- Added the `fields_limit` parameter for the `hits` query to limit the
number of returned fields. This reduces the response size and decreases
memory usage.

#### Legend Menu:
A context menu has been added for legend items, which includes:  
1. Metric name
2. **Copy _stream name** - copies the full stream name in the format
`{field1="value1", ..., fieldN="valueN"}`.
3. **Add _stream to filter** - adds the full stream value to the current
filter:
   `_stream: {field1="value1", ..., fieldN="valueN"} AND (old_expr)`.  
4. **Exclude _stream from filter** - excludes the stream from the
current filter:
`(NOT _stream: {field1="value1", ..., fieldN="valueN"}) AND (old_expr)`.
5. List of fields with options:  
   - Copy as `field: "value"`.  
   - Add to filter: `field: "value" AND (old_expr)`.  
   - Exclude from filter: `-field: "value" AND (old_expr)`.  
6. Total number of hits for the stream.  

Related issue: #7552

<details>
  <summary>UI Demo - Legend Menu</summary>
  
<img width="400"
src="https://github.com/user-attachments/assets/ee1954b2-fdce-44b4-a2dc-aa73096a5414"/>
<img width="400"
src="https://github.com/user-attachments/assets/19d71f04-c207-4143-a176-c5f221592e3d"/>

</details>

---

#### Legend:
1. Displays the total number of hits for the stream.  
2. Added hints below the legend for total hits and graph interactions.
3. Click behavior is now the same as in other vmui charts:  
   - `click` - shows only the selected series.  
   - `click + ctrl/cmd` - hides the selected series.  
   
<details>
  <summary>UI Demo - Legend</summary>

before: 
<img
src="https://github.com/user-attachments/assets/18270842-0c39-4f63-bcda-da62e15c3c73"/>

after:
<img
src="https://github.com/user-attachments/assets/351cad3a-f763-4b1d-b3be-b569b5472a7c"/>
  
</details>

---

#### Tooltip:  
1. The `other` label is moved to the end, and others are sorted by
value.
2. Values are aligned to the right.  
3. Labels are truncated and always shown in a single line for better
readability; the full name is available in the legend.

<details>
  <summary>UI Demo - Tooltip</summary>
  
| before     | after        |
|----------|----------|
| <img
src="https://github.com/user-attachments/assets/adccff38-e2e6-46e4-a69e-21381982489c"/>
| <img
src="https://github.com/user-attachments/assets/81008897-d816-4aed-92cb-749ea7e0ff1e"/>
|
  
</details>

---

#### Group View (tab Group):
Groups are now sorted by the number of records in descending order.

<details>
  <summary>UI Demo - Group View</summary>

before: 
<img width="800"
src="https://github.com/user-attachments/assets/15b4ca72-7e5d-421f-913b-c5ff22c340cb"/>

after:
<img width="800"
src="https://github.com/user-attachments/assets/32ff627b-6f30-4195-bfe7-8c9b4aa11f6b"/>
  
</details>
2025-01-26 22:06:13 +01:00
Aliaksandr Valialkin
93ea7c5cf6 docs/VictoriaLogs/CHANGELOG.md: typo fix after ad6c587494: ignore_global_time_range -> ignore_global_time_filter 2025-01-26 22:05:37 +01:00
Aliaksandr Valialkin
fea934936b lib/logstorage: properly propagate extra filters to all the subqueries
The purpose of extra filters ( https://docs.victoriametrics.com/victorialogs/querying/#extra-filters )
is to limit the subset of logs, which can be queried. For example, it is expected that all the queries
with `extra_filters={tenant=123}` can access only logs, which contain `123` value for the `tenant` field.

Previously this wasn't the case, since the provided extra filters weren't applied to subqueries.
For example, the following query could be used to select all the logs outside `tenant=123`, for any `extra_filters` arg:

    * | union({tenant!=123})

This commit fixes this by propagating extra filters to all the subqueries.

While at it, this commit also properly propagates [start, end] time range filter from HTTP querying APIs
into all the subqueries, since this is what most users expect. This behaviour can be overriden on per-subquery
basis with the `options(ignore_global_time_filter=true)` option - see https://docs.victoriametrics.com/victorialogs/logsql/#query-options

Also properly apply apply optimizations across all the subqueries. Previously the optimizations at Query.optimize()
function were applied only to the top-level query.
2025-01-26 22:05:05 +01:00
Aliaksandr Valialkin
7b62086609 lib: consistently use logger.Panicf("BUG: ...") for logging programming bugs
logger.Fatalf("BUG: ...") complicates investigating the bug, since it doesn't show the call stack,
which led to the bug. So it is better to consistently use logger.Panicf("BUG: ...") for logging programming bugs.
2025-01-24 16:40:50 +01:00
Aliaksandr Valialkin
9c23c61a2e docs/VictoriaLogs/LogsQL.md: show how to unroll the returned histogram buckets into separate rows at histogram pipe docs 2025-01-24 16:39:47 +01:00
Aliaksandr Valialkin
fb68ba3745 docs/VictoriaLogs/sql-to-logsql.md: show how to substitute complex SQL query with top pipe 2025-01-24 16:39:46 +01:00
f41gh7
43772b9869 make vmui-update 2025-01-24 14:23:49 +01:00
Nikolay
cab5cf3c4c app/vmselect: fixes panic data race at query tracing
Previously, NewChild elements of querytracer could be referenced by concurrent
storageNode goroutines. After earlier return ( if search.skipSlowReplicas is set), it is
possible, that tracer objects could be still in-use by concurrent workers.
  It may cause panics and data races. Most probable case is when parent tracer is finished, but children
still could write data to itself via Donef() method. It triggers read-write data race at trace
formatting.

This commit adds a new methods to the querytracer package, that allows to
create children not referenced by parent and add it to the parent later.

 Orphaned child must be registered at the parent, when goroutine returns. It's done synchronously by the single caller  via finishQueryTracer call. 
If child didn't finished work and reference for it is used by concurrent goroutine, new child must be created instead with
context message.
 It prevents panics and possible data races.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8114

---------

Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2025-01-24 13:55:32 +01:00
Roman Khavronenko
f52736eb98 docs: add avaialbe_version notion to new -search.maxDeleteDuration (#8142)
Follow-up for
4574958e2e

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-01-24 13:46:03 +01:00
Yury Molodov
50a37ee07d vmui: save column settings in URL #7662 (#7979)
### Describe Your Changes

Added saving column settings in the URL for the table view. See #7662

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit f0d55a1c25)
2025-01-24 09:51:50 +01:00
Yury Molodov
7a13358cd2 vmui: fix issue with query execution and line breaks in query editor
This commit fixes incorrect behaviour when pressing `Enter` did not execute the query, and
`Shift+Enter` did not insert a new line.

- The issue occurred when autocomplete was disabled.
- This problem affected the query editor in both the VictoriaMetrics UI
and VictoriaLogs UI.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8058

(cherry picked from commit f31dece58d)
2025-01-24 09:51:50 +01:00
Dmytro Kozlov
327f1835d1 deployment/docker: upgraded Grafana plugins to the latest versions
Upgraded Grafana plugins to the latest versions

(cherry picked from commit a6951b8b14)
2025-01-24 09:51:49 +01:00
Zakhar Bessarab
aef93b1889 app/vmselect/prometheus: fix panic when performing delete with "multitenant" auth token
Initially delete_series API wasn't implemented for mulitenant auth token.

 This commit fixes it and properly handle delete series requests for mulitenant auth token.
It also adds integration tests for this case.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8126

Introduced at v1.104.0 release:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1434
---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
2025-01-24 08:32:50 +01:00
Zakhar Bessarab
5f56375564 app/vmselect/prometheus: prevent panic when using "multitenant" at /api/v1/series/count requests
Adding support of multi-tenant reads to /api/v1/series/count would
require introducing a breaking change to a `netstorage` RPC, so
currently vmselect will explicitly deny these requests.

Related issues:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8126

Introduced at v1.104.0 release:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1434

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-01-24 07:58:54 +01:00
Phuong Le
3ada13dd48 lib/logstorage: remove redundant error check 2025-01-24 07:52:52 +01:00
Aliaksandr Valialkin
2aeba2b2cc docs/VictoriaLogs/sql-to-logsql.md: add a guide on how to convert SQL to LogsQL 2025-01-24 07:52:52 +01:00
Aliaksandr Valialkin
755a0bb3d9 docs/VictoriaLogs/README.md: add a link to VictoriaLogs playground 2025-01-24 07:52:52 +01:00
Aliaksandr Valialkin
9c8fc93553 docs/VictoriaLogs/LogsQL.md: mention that field pipe can be used for improving query performance 2025-01-24 07:52:52 +01:00
Aliaksandr Valialkin
45cc9974ab lib/logstorage: inherit query options by nested queries
This is a follow-up for b620b5cff5
2025-01-24 07:52:51 +01:00
hagen1778
ea3a7c3528 deployment: reflect metrics datasource ID change
See https://github.com/VictoriaMetrics/victoriametrics-datasource

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-01-24 07:52:51 +01:00
Roman Khavronenko
9261da53a0 app/vmselect/promql: respect staleness in removeCounterResets (#8073)
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8072

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2025-01-24 07:52:51 +01:00
Zhu Jiekun
3ca0ca1f45 docs/vmagent: estimating the message size and rate when using Kafka
When using Kafka, it's important to estimate the message rate and size
before applying for resources.

This commit  explains why and how to use remote write metrics to
do the evaluation.
2025-01-24 07:52:51 +01:00
Jose Gómez-Sellés
e1852e8aa1 doc: fix typo in vmsingle k8s monitoring guide (#8120)
As reported by a user, the value in [VictoriaMetrics Single helm
chart](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-single/values.yaml#L420)
and actual example, should be "enabled", not "enable". This commit fixes
it.


### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-01-24 07:52:51 +01:00
Aliaksandr Valialkin
bfd198a4d5 lib/logstorage: add hash pipe for calculating hash over the given log field
This pipe may be useful for sharding log entries among hash buckets.
2025-01-24 07:52:51 +01:00
Aliaksandr Valialkin
0a586ecfd8 lib/logstorage: add an ability to set query concurrency on a per-query basis
This is done via 'options(concurrency=N)' prefix for the query.
For example, the following query is executed on at most 4 CPU cores:

    options(concurrency=4) _time:1d | count_uniq(user_id)

This allows reducing RAM and CPU usage at the cost of longer query execution times,
since by default every query is executed in parallel on all the available CPU cores.

See https://docs.victoriametrics.com/victorialogs/logsql/#query-options
2025-01-24 07:52:50 +01:00
Aliaksandr Valialkin
fb311d3ad5 lib/logstorage: always pass the current timestamp to newLexer()
Also always initialize Query.timestamp with the timestamp from the lexer.

This should avoid potential problems with relative timestamps inside inner queries.
For example, the `_time:1h` filter in the following query is correctly executed
relative to the current timestamp:

   foo:in(_time:1h | keep foo)
2025-01-24 07:52:50 +01:00
Aliaksandr Valialkin
159b821a52 lib/logstorage: simplify the caller side of addNewItem() function 2025-01-24 07:52:50 +01:00
Daria Karavaieva
e7c806e5c3 docs/vmanomaly: format & fix docs (#8122)
### Describe Your Changes

- fix formatting in Presets
- change integration guide config according to new format + optimisation
of prophet
- minor fixes

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-01-24 07:52:49 +01:00
401 changed files with 7341 additions and 3105 deletions

View File

@@ -204,19 +204,19 @@ check-all: fmt vet golangci-lint govulncheck
clean-checkers: remove-golangci-lint remove-govulncheck
test:
DISABLE_FSYNC_FOR_TESTING=1 go test ./lib/... ./app/...
go test ./lib/... ./app/...
test-race:
DISABLE_FSYNC_FOR_TESTING=1 go test -race ./lib/... ./app/...
go test -race ./lib/... ./app/...
test-pure:
DISABLE_FSYNC_FOR_TESTING=1 CGO_ENABLED=0 go test ./lib/... ./app/...
CGO_ENABLED=0 go test ./lib/... ./app/...
test-full:
DISABLE_FSYNC_FOR_TESTING=1 go test -coverprofile=coverage.txt -covermode=atomic ./lib/... ./app/...
go test -coverprofile=coverage.txt -covermode=atomic ./lib/... ./app/...
test-full-386:
DISABLE_FSYNC_FOR_TESTING=1 GOARCH=386 go test -coverprofile=coverage.txt -covermode=atomic ./lib/... ./app/...
GOARCH=386 go test -coverprofile=coverage.txt -covermode=atomic ./lib/... ./app/...
integration-test: all
go test ./apptest/... -skip="^TestSingle.*"

View File

@@ -688,13 +688,13 @@ func ProcessStatsQueryRangeRequest(ctx context.Context, w http.ResponseWriter, r
m := make(map[string]*statsSeries)
var mLock sync.Mutex
timestamp := q.GetTimestamp()
writeBlock := func(_ uint, timestamps []int64, columns []logstorage.BlockColumn) {
clonedColumnNames := make([]string, len(columns))
for i, c := range columns {
clonedColumnNames[i] = strings.Clone(c.Name)
}
for i := range timestamps {
timestamp := q.GetTimestamp()
labels := make([]logstorage.Field, 0, len(byFields))
for j, c := range columns {
if c.Name == "_time" {

View File

@@ -1,12 +1,12 @@
{
"files": {
"main.css": "./static/css/main.3134e778.css",
"main.js": "./static/js/main.82cd6930.js",
"main.css": "./static/css/main.02a1c6cb.css",
"main.js": "./static/js/main.55c8060b.js",
"static/js/685.f772060c.chunk.js": "./static/js/685.f772060c.chunk.js",
"index.html": "./index.html"
},
"entrypoints": [
"static/css/main.3134e778.css",
"static/js/main.82cd6930.js"
"static/css/main.02a1c6cb.css",
"static/js/main.55c8060b.js"
]
}

View File

@@ -1 +1 @@
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.svg"/><link rel="apple-touch-icon" href="./favicon.svg"/><link rel="mask-icon" href="./favicon.svg" color="#000000"><meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=5"/><meta name="theme-color" content="#000000"/><meta name="description" content="Explore your log data with VictoriaLogs UI"/><link rel="manifest" href="./manifest.json"/><title>UI for VictoriaLogs</title><meta name="twitter:card" content="summary"><meta name="twitter:title" content="UI for VictoriaLogs"><meta name="twitter:site" content="@https://victoriametrics.com/products/victorialogs/"><meta name="twitter:description" content="Explore your log data with VictoriaLogs UI"><meta name="twitter:image" content="./preview.jpg"><meta property="og:type" content="website"><meta property="og:title" content="UI for VictoriaLogs"><meta property="og:url" content="https://victoriametrics.com/products/victorialogs/"><meta property="og:description" content="Explore your log data with VictoriaLogs UI"><script defer="defer" src="./static/js/main.82cd6930.js"></script><link href="./static/css/main.3134e778.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.svg"/><link rel="apple-touch-icon" href="./favicon.svg"/><link rel="mask-icon" href="./favicon.svg" color="#000000"><meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=5"/><meta name="theme-color" content="#000000"/><meta name="description" content="Explore your log data with VictoriaLogs UI"/><link rel="manifest" href="./manifest.json"/><title>UI for VictoriaLogs</title><meta name="twitter:card" content="summary"><meta name="twitter:title" content="UI for VictoriaLogs"><meta name="twitter:site" content="@https://victoriametrics.com/products/victorialogs/"><meta name="twitter:description" content="Explore your log data with VictoriaLogs UI"><meta name="twitter:image" content="./preview.jpg"><meta property="og:type" content="website"><meta property="og:title" content="UI for VictoriaLogs"><meta property="og:url" content="https://victoriametrics.com/products/victorialogs/"><meta property="og:description" content="Explore your log data with VictoriaLogs UI"><script defer="defer" src="./static/js/main.55c8060b.js"></script><link href="./static/css/main.02a1c6cb.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

View File

@@ -822,7 +822,7 @@ func RegisterMetricNames(qt *querytracer.Tracer, mrs []storage.MetricRow, deadli
}
// Push mrs to storage nodes in parallel.
snr := startStorageNodesRequest(qt, sns, true, func(qt *querytracer.Tracer, workerID uint, sn *storageNode, cancelled *atomic.Bool) any {
snr := startStorageNodesRequest(qt, sns, true, func(qt *querytracer.Tracer, workerID uint, sn *storageNode) any {
sn.registerMetricNamesRequests.Inc()
err := sn.registerMetricNames(qt, mrsPerNode[workerID], deadline)
if err != nil {
@@ -857,8 +857,8 @@ func DeleteSeries(qt *querytracer.Tracer, sq *storage.SearchQuery, deadline sear
return 0, err
}
sns := getStorageNodes()
snr := startStorageNodesRequest(qt, sns, true, func(qt *querytracer.Tracer, _ uint, sn *storageNode, cancelled *atomic.Bool) any {
return execSearchQuery(qt, sq, cancelled, func(qt *querytracer.Tracer, requestData []byte, _ storage.TenantToken) any {
snr := startStorageNodesRequest(qt, sns, true, func(qt *querytracer.Tracer, _ uint, sn *storageNode) any {
return execSearchQuery(qt, sq, func(qt *querytracer.Tracer, requestData []byte, _ storage.TenantToken) any {
sn.deleteSeriesRequests.Inc()
deletedCount, err := sn.deleteSeries(qt, requestData, deadline)
if err != nil {
@@ -906,8 +906,8 @@ func LabelNames(qt *querytracer.Tracer, denyPartialResponse bool, sq *storage.Se
return nil, false, err
}
sns := getStorageNodes()
snr := startStorageNodesRequest(qt, sns, denyPartialResponse, func(qt *querytracer.Tracer, _ uint, sn *storageNode, cancelled *atomic.Bool) any {
return execSearchQuery(qt, sq, cancelled, func(qt *querytracer.Tracer, requestData []byte, _ storage.TenantToken) any {
snr := startStorageNodesRequest(qt, sns, denyPartialResponse, func(qt *querytracer.Tracer, _ uint, sn *storageNode) any {
return execSearchQuery(qt, sq, func(qt *querytracer.Tracer, requestData []byte, _ storage.TenantToken) any {
sn.labelNamesRequests.Inc()
labelNames, err := sn.getLabelNames(qt, requestData, maxLabelNames, deadline)
if err != nil {
@@ -1023,14 +1023,14 @@ func LabelValues(qt *querytracer.Tracer, denyPartialResponse bool, labelName str
case "vm_project_id":
idx = 1
default:
logger.Fatalf("BUG: unexpected labeName=%q", labelName)
logger.Panicf("BUG: unexpected labeName=%q", labelName)
}
labelValues := make([]string, 0, len(tenants))
for _, t := range tenants {
s := strings.Split(t, ":")
if len(s) != 2 {
logger.Fatalf("BUG: unexpected tenant received from storage: %q", t)
logger.Panicf("BUG: unexpected tenant received from storage: %q", t)
}
labelValues = append(labelValues, s[idx])
@@ -1050,8 +1050,8 @@ func LabelValues(qt *querytracer.Tracer, denyPartialResponse bool, labelName str
return nil, false, err
}
sns := getStorageNodes()
snr := startStorageNodesRequest(qt, sns, denyPartialResponse, func(qt *querytracer.Tracer, _ uint, sn *storageNode, cancelled *atomic.Bool) any {
return execSearchQuery(qt, sq, cancelled, func(qt *querytracer.Tracer, requestData []byte, _ storage.TenantToken) any {
snr := startStorageNodesRequest(qt, sns, denyPartialResponse, func(qt *querytracer.Tracer, _ uint, sn *storageNode) any {
return execSearchQuery(qt, sq, func(qt *querytracer.Tracer, requestData []byte, _ storage.TenantToken) any {
sn.labelValuesRequests.Inc()
labelValues, err := sn.getLabelValues(qt, labelName, requestData, maxLabelValues, deadline)
if err != nil {
@@ -1114,7 +1114,7 @@ func Tenants(qt *querytracer.Tracer, tr storage.TimeRange, deadline searchutils.
}
sns := getStorageNodes()
// Deny partial responses when obtaining the list of tenants, since partial tenants have little sense.
snr := startStorageNodesRequest(qt, sns, true, func(qt *querytracer.Tracer, _ uint, sn *storageNode, cancelled *atomic.Bool) any {
snr := startStorageNodesRequest(qt, sns, true, func(qt *querytracer.Tracer, _ uint, sn *storageNode) any {
sn.tenantsRequests.Inc()
tenants, err := sn.getTenants(qt, tr, deadline)
if err != nil {
@@ -1195,7 +1195,7 @@ func TagValueSuffixes(qt *querytracer.Tracer, accountID, projectID uint32, denyP
err error
}
sns := getStorageNodes()
snr := startStorageNodesRequest(qt, sns, denyPartialResponse, func(qt *querytracer.Tracer, _ uint, sn *storageNode, cancelled *atomic.Bool) any {
snr := startStorageNodesRequest(qt, sns, denyPartialResponse, func(qt *querytracer.Tracer, _ uint, sn *storageNode) any {
sn.tagValueSuffixesRequests.Inc()
suffixes, err := sn.getTagValueSuffixes(qt, accountID, projectID, tr, tagKey, tagValuePrefix, delimiter, maxSuffixes, deadline)
if err != nil {
@@ -1263,8 +1263,8 @@ func TSDBStatus(qt *querytracer.Tracer, denyPartialResponse bool, sq *storage.Se
return nil, false, err
}
sns := getStorageNodes()
snr := startStorageNodesRequest(qt, sns, denyPartialResponse, func(qt *querytracer.Tracer, _ uint, sn *storageNode, cancelled *atomic.Bool) any {
return execSearchQuery(qt, sq, cancelled, func(qt *querytracer.Tracer, requestData []byte, _ storage.TenantToken) any {
snr := startStorageNodesRequest(qt, sns, denyPartialResponse, func(qt *querytracer.Tracer, _ uint, sn *storageNode) any {
return execSearchQuery(qt, sq, func(qt *querytracer.Tracer, requestData []byte, _ storage.TenantToken) any {
sn.tsdbStatusRequests.Inc()
status, err := sn.getTSDBStatus(qt, requestData, focusLabel, topN, deadline)
if err != nil {
@@ -1373,7 +1373,7 @@ func SeriesCount(qt *querytracer.Tracer, accountID, projectID uint32, denyPartia
err error
}
sns := getStorageNodes()
snr := startStorageNodesRequest(qt, sns, denyPartialResponse, func(qt *querytracer.Tracer, _ uint, sn *storageNode, _ *atomic.Bool) any {
snr := startStorageNodesRequest(qt, sns, denyPartialResponse, func(qt *querytracer.Tracer, _ uint, sn *storageNode) any {
sn.seriesCountRequests.Inc()
n, err := sn.getSeriesCount(qt, accountID, projectID, deadline)
if err != nil {
@@ -1696,8 +1696,8 @@ func SearchMetricNames(qt *querytracer.Tracer, denyPartialResponse bool, sq *sto
return nil, false, err
}
sns := getStorageNodes()
snr := startStorageNodesRequest(qt, sns, denyPartialResponse, func(qt *querytracer.Tracer, _ uint, sn *storageNode, cancelled *atomic.Bool) any {
return execSearchQuery(qt, sq, cancelled, func(qt *querytracer.Tracer, requestData []byte, t storage.TenantToken) any {
snr := startStorageNodesRequest(qt, sns, denyPartialResponse, func(qt *querytracer.Tracer, _ uint, sn *storageNode) any {
return execSearchQuery(qt, sq, func(qt *querytracer.Tracer, requestData []byte, t storage.TenantToken) any {
sn.searchMetricNamesRequests.Inc()
metricNames, err := sn.processSearchMetricNames(qt, requestData, deadline)
if sq.IsMultiTenant {
@@ -1887,10 +1887,10 @@ func processBlocks(qt *querytracer.Tracer, sns []*storageNode, denyPartialRespon
return false, err
}
// Send the query to all the storage nodes in parallel.
snr := startStorageNodesRequest(qt, sns, denyPartialResponse, func(qt *querytracer.Tracer, workerID uint, sn *storageNode, cancelled *atomic.Bool) any {
snr := startStorageNodesRequest(qt, sns, denyPartialResponse, func(qt *querytracer.Tracer, workerID uint, sn *storageNode) any {
// Use a separate variable for each goroutine
var err error
res := execSearchQuery(qt, sq, cancelled, func(qt *querytracer.Tracer, rd []byte, _ storage.TenantToken) any {
res := execSearchQuery(qt, sq, func(qt *querytracer.Tracer, rd []byte, _ storage.TenantToken) any {
sn.searchRequests.Inc()
err = sn.processSearchQuery(qt, rd, f, workerID, deadline)
if err != nil {
@@ -1951,10 +1951,10 @@ func populateSqTenantTokensIfNeeded(sq *storage.SearchQuery) error {
type storageNodesRequest struct {
denyPartialResponse bool
resultsCh chan rpcResult
qts map[*querytracer.Tracer]struct{}
sns []*storageNode
cancelled *atomic.Bool
wg sync.WaitGroup
qt *querytracer.Tracer
// query tracers to storageAddresses mapping
qts map[*querytracer.Tracer]string
sns []*storageNode
}
type rpcResult struct {
@@ -1964,44 +1964,46 @@ type rpcResult struct {
}
func startStorageNodesRequest(qt *querytracer.Tracer, sns []*storageNode, denyPartialResponse bool,
f func(qt *querytracer.Tracer, workerID uint, sn *storageNode, cancelled *atomic.Bool) any,
f func(qt *querytracer.Tracer, workerID uint, sn *storageNode) any,
) *storageNodesRequest {
resultsCh := make(chan rpcResult, len(sns))
qts := make(map[*querytracer.Tracer]struct{}, len(sns))
snr := &storageNodesRequest{
denyPartialResponse: denyPartialResponse,
resultsCh: resultsCh,
qts: qts,
sns: sns,
cancelled: &atomic.Bool{},
}
snr.wg.Add(len(sns))
qts := make(map[*querytracer.Tracer]string, len(sns))
for idx, sn := range sns {
qtChild := qt.NewChild("rpc at vmstorage %s", sn.connPool.Addr())
qts[qtChild] = struct{}{}
// Do not use qt.NewChild.
// StorageNodesRequest may be finished before goroutine returns.
// Caller must register tracker manually with finishQueryTracer after goroutine returns result.
// It ensures that tracker is no longer referenced by any concurrent goroutines.
//
// See this issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8114
qtOrphan := querytracer.NewOrphan(qt, "rpc at vmstorage %s", sn.connPool.Addr())
qts[qtOrphan] = sn.connPool.Addr()
go func(workerID uint, sn *storageNode) {
defer snr.wg.Done()
data := f(qtChild, workerID, sn, snr.cancelled)
data := f(qtOrphan, workerID, sn)
resultsCh <- rpcResult{
data: data,
qt: qtChild,
qt: qtOrphan,
group: sn.group,
}
}(uint(idx), sn)
}
return snr
return &storageNodesRequest{
denyPartialResponse: denyPartialResponse,
resultsCh: resultsCh,
qt: qt,
qts: qts,
sns: sns,
}
}
// finishQueryTracers cancels all the query tracers and waits for all the workers to finish.
func (snr *storageNodesRequest) finishQueryTracers(msg string) {
// Set cancelled flag to stop new work
snr.cancelled.Store(true)
// Wait for all workers to finish
snr.wg.Wait()
// Now safe to close all tracers
for qt := range snr.qts {
snr.finishQueryTracer(qt, msg)
for qt, storageAddr := range snr.qts {
// since qt cannot be used concurrently,
// replace child still referenced by concurrent storageNode goroutine
// with local child that belongs to current goroutine.
// Add reason msg why it was done.
cancelledQt := snr.qt.NewChild("rpc at vmstorage: %s: %s", storageAddr, msg)
cancelledQt.Done()
delete(snr.qts, qt)
}
}
@@ -2012,6 +2014,7 @@ func (snr *storageNodesRequest) finishQueryTracer(qt *querytracer.Tracer, msg st
qt.Donef("%s", msg)
}
delete(snr.qts, qt)
snr.qt.AddChild(qt)
}
func (snr *storageNodesRequest) collectAllResults(f func(result any) error) error {
@@ -3278,16 +3281,11 @@ func (pnc *perNodeCounter) GetTotal() uint64 {
const maxFastAllocBlockSize = 32 * 1024
// execSearchQuery calls cb for with marshaled requestData for each tenant in sq.
func execSearchQuery(qt *querytracer.Tracer, sq *storage.SearchQuery, cancelled *atomic.Bool, cb func(qt *querytracer.Tracer, requestData []byte, t storage.TenantToken) any) []any {
func execSearchQuery(qt *querytracer.Tracer, sq *storage.SearchQuery, cb func(qt *querytracer.Tracer, requestData []byte, t storage.TenantToken) any) []any {
var requestData []byte
var results []any
for i := range sq.TenantTokens {
// Stop processing the remaining tenants if the query is cancelled.
// It is safe to return the partial results.
if cancelled.Load() {
return results
}
requestData = sq.TenantTokens[i].Marshal(requestData)
requestData = sq.MarshaWithoutTenant(requestData)
qtL := qt
@@ -3295,7 +3293,7 @@ func execSearchQuery(qt *querytracer.Tracer, sq *storage.SearchQuery, cancelled
qtL = qt.NewChild("query for tenant: %s", sq.TenantTokens[i].String())
}
r := cb(qtL, requestData, sq.TenantTokens[i])
if sq.IsMultiTenant && qt.Enabled() {
if sq.IsMultiTenant {
qtL.Done()
}
results = append(results, r)

View File

@@ -512,7 +512,10 @@ func DeleteHandler(startTime time.Time, at *auth.Token, r *http.Request) error {
if !cp.IsDefaultTimeRange() {
return fmt.Errorf("start=%d and end=%d args aren't supported. Remove these args from the query in order to delete all the matching metrics", cp.start, cp.end)
}
sq := storage.NewSearchQuery(at.AccountID, at.ProjectID, cp.start, cp.end, cp.filterss, *maxDeleteSeries)
sq, err := getSearchQuery(nil, at, cp, *maxDeleteSeries)
if err != nil {
return err
}
deletedCount, err := netstorage.DeleteSeries(nil, sq, cp.deadline)
if err != nil {
return fmt.Errorf("cannot delete time series: %w", err)
@@ -755,7 +758,9 @@ var labelsDuration = metrics.NewSummary(`vm_request_duration_seconds{path="/api/
// SeriesCountHandler processes /api/v1/series/count request.
func SeriesCountHandler(startTime time.Time, at *auth.Token, w http.ResponseWriter, r *http.Request) error {
defer seriesCountDuration.UpdateDuration(startTime)
if at == nil {
return fmt.Errorf("multi-tenant request to /api/v1/series/count is not supported")
}
deadline := searchutils.GetDeadlineForStatusRequest(r, startTime)
denyPartialResponse := httputils.GetDenyPartialResponse(r)
n, isPartial, err := netstorage.SeriesCount(nil, at.AccountID, at.ProjectID, denyPartialResponse, deadline)

View File

@@ -374,8 +374,8 @@ func getRollupConfigs(funcName string, rf rollupFunc, expr metricsql.Expr, start
preFunc := func(_ []float64, _ []int64) {}
funcName = strings.ToLower(funcName)
if rollupFuncsRemoveCounterResets[funcName] {
preFunc = func(values []float64, _ []int64) {
removeCounterResets(values)
preFunc = func(values []float64, timestamps []int64) {
removeCounterResets(values, timestamps, lookbackDelta)
}
}
samplesScannedPerCall := rollupFuncsSamplesScannedPerCall[funcName]
@@ -487,8 +487,8 @@ func getRollupConfigs(funcName string, rf rollupFunc, expr metricsql.Expr, start
for _, aggrFuncName := range aggrFuncNames {
if rollupFuncsRemoveCounterResets[aggrFuncName] {
// There is no need to save the previous preFunc, since it is either empty or the same.
preFunc = func(values []float64, _ []int64) {
removeCounterResets(values)
preFunc = func(values []float64, timestamps []int64) {
removeCounterResets(values, timestamps, lookbackDelta)
}
}
rf := rollupAggrFuncs[aggrFuncName]
@@ -905,7 +905,7 @@ func getMaxPrevInterval(scrapeInterval int64) int64 {
return scrapeInterval + scrapeInterval/8
}
func removeCounterResets(values []float64) {
func removeCounterResets(values []float64, timestamps []int64, maxStalenessInterval int64) {
// There is no need in handling NaNs here, since they are impossible
// on values from vmstorage.
if len(values) == 0 {
@@ -924,6 +924,16 @@ func removeCounterResets(values []float64) {
correction += prevValue
}
}
if i > 0 && maxStalenessInterval > 0 {
gap := timestamps[i] - timestamps[i-1]
if gap > maxStalenessInterval {
// reset correction if gap between samples exceeds staleness interval
// see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8072
correction = 0
prevValue = v
continue
}
}
prevValue = v
values[i] = v + correction
// Check again, there could be precision error in float operations,

View File

@@ -117,31 +117,49 @@ func TestRollupIderivDuplicateTimestamps(t *testing.T) {
}
func TestRemoveCounterResets(t *testing.T) {
removeCounterResets(nil)
removeCounterResets(nil, nil, 0)
values := append([]float64{}, testValues...)
removeCounterResets(values)
timestamps := append([]int64{}, testTimestamps...)
removeCounterResets(values, timestamps, 0)
valuesExpected := []float64{123, 157, 167, 188, 221, 255, 320, 332, 364, 396, 398, 398}
testRowsEqual(t, values, testTimestamps, valuesExpected, testTimestamps)
// removeCounterResets doesn't expect negative values, so it doesn't work properly with them.
values = []float64{-100, -200, -300, -400}
removeCounterResets(values)
valuesExpected = []float64{-100, -100, -100, -100}
timestampsExpected := []int64{0, 1, 2, 3}
removeCounterResets(values, timestampsExpected, 0)
valuesExpected = []float64{-100, -100, -100, -100}
testRowsEqual(t, values, timestampsExpected, valuesExpected, timestampsExpected)
// verify how partial counter reset is handled.
// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2787
values = []float64{100, 95, 120, 119, 139, 50}
removeCounterResets(values)
valuesExpected = []float64{100, 100, 125, 125, 145, 195}
timestampsExpected = []int64{0, 1, 2, 3, 4, 5}
removeCounterResets(values, timestampsExpected, 0)
valuesExpected = []float64{100, 100, 125, 125, 145, 195}
testRowsEqual(t, values, timestampsExpected, valuesExpected, timestampsExpected)
// verify that staleness interval is respected during resets
// see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8072
values = []float64{10, 12, 14, 4, 6, 8, 6, 8, 4, 6}
timestamps = []int64{10, 20, 30, 60, 70, 80, 90, 100, 120, 130}
valuesExpected = []float64{10, 12, 14, 4, 6, 8, 14, 16, 4, 6}
removeCounterResets(values, timestamps, 10)
testRowsEqual(t, values, timestamps, valuesExpected, timestamps)
// verify that staleness is respected if there was no counter reset
// but correction was made previously
values = []float64{10, 12, 2, 4}
timestamps = []int64{10, 20, 30, 60}
valuesExpected = []float64{10, 12, 14, 4}
removeCounterResets(values, timestamps, 10)
testRowsEqual(t, values, timestamps, valuesExpected, timestamps)
// verify results always increase monotonically with possible float operations precision error
values = []float64{34.094223, 2.7518, 2.140669, 0.044878, 1.887095, 2.546569, 2.490149, 0.045, 0.035684, 0.062454, 0.058296}
removeCounterResets(values)
timestampsExpected = []int64{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
removeCounterResets(values, timestampsExpected, 0)
var prev float64
for i, v := range values {
if v < prev {
@@ -166,7 +184,7 @@ func TestDeltaValues(t *testing.T) {
// remove counter resets
values = append([]float64{}, testValues...)
removeCounterResets(values)
removeCounterResets(values, testTimestamps, 0)
deltaValues(values)
valuesExpected = []float64{34, 10, 21, 33, 34, 65, 12, 32, 32, 2, 0, 0}
testRowsEqual(t, values, testTimestamps, valuesExpected, testTimestamps)
@@ -188,7 +206,7 @@ func TestDerivValues(t *testing.T) {
// remove counter resets
values = append([]float64{}, testValues...)
removeCounterResets(values)
removeCounterResets(values, testTimestamps, 0)
derivValues(values, testTimestamps)
valuesExpected = []float64{3400, 1111.111111111111, 1750, 2538.4615384615386, 3090.909090909091, 3611.1111111111113,
6000, 1882.3529411764705, 1777.7777777777778, 400, 0, 0}
@@ -219,7 +237,7 @@ func testRollupFunc(t *testing.T, funcName string, args []any, vExpected float64
rfa.timestamps = append(rfa.timestamps, testTimestamps...)
rfa.window = rfa.timestamps[len(rfa.timestamps)-1] - rfa.timestamps[0]
if rollupFuncsRemoveCounterResets[funcName] {
removeCounterResets(rfa.values)
removeCounterResets(rfa.values, rfa.timestamps, 0)
}
for i := 0; i < 5; i++ {
v := rf(&rfa)

View File

@@ -1,13 +1,13 @@
{
"files": {
"main.css": "./static/css/main.63479b72.css",
"main.js": "./static/js/main.256ee243.js",
"main.css": "./static/css/main.af583aad.css",
"main.js": "./static/js/main.1413b18d.js",
"static/js/685.f772060c.chunk.js": "./static/js/685.f772060c.chunk.js",
"static/media/MetricsQL.md": "./static/media/MetricsQL.a00044c91d9781cf8557.md",
"index.html": "./index.html"
},
"entrypoints": [
"static/css/main.63479b72.css",
"static/js/main.256ee243.js"
"static/css/main.af583aad.css",
"static/js/main.1413b18d.js"
]
}

View File

@@ -1 +1 @@
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.svg"/><link rel="apple-touch-icon" href="./favicon.svg"/><link rel="mask-icon" href="./favicon.svg" color="#000000"><meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=5"/><meta name="theme-color" content="#000000"/><meta name="description" content="Explore and troubleshoot your VictoriaMetrics data"/><link rel="manifest" href="./manifest.json"/><title>vmui</title><script src="./dashboards/index.js" type="module"></script><meta name="twitter:card" content="summary"><meta name="twitter:title" content="UI for VictoriaMetrics"><meta name="twitter:site" content="@https://victoriametrics.com/"><meta name="twitter:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta name="twitter:image" content="./preview.jpg"><meta property="og:type" content="website"><meta property="og:title" content="UI for VictoriaMetrics"><meta property="og:url" content="https://victoriametrics.com/"><meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data"><script defer="defer" src="./static/js/main.256ee243.js"></script><link href="./static/css/main.63479b72.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.svg"/><link rel="apple-touch-icon" href="./favicon.svg"/><link rel="mask-icon" href="./favicon.svg" color="#000000"><meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=5"/><meta name="theme-color" content="#000000"/><meta name="description" content="Explore and troubleshoot your VictoriaMetrics data"/><link rel="manifest" href="./manifest.json"/><title>vmui</title><script src="./dashboards/index.js" type="module"></script><meta name="twitter:card" content="summary"><meta name="twitter:title" content="UI for VictoriaMetrics"><meta name="twitter:site" content="@https://victoriametrics.com/"><meta name="twitter:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta name="twitter:image" content="./preview.jpg"><meta property="og:type" content="website"><meta property="og:title" content="UI for VictoriaMetrics"><meta property="og:url" content="https://victoriametrics.com/"><meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data"><script defer="defer" src="./static/js/main.1413b18d.js"></script><link href="./static/css/main.af583aad.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

View File

@@ -1,3 +1,6 @@
import uPlot from "uplot";
import { ReactNode } from "react";
export interface MetricBase {
group: number;
metric: {
@@ -6,13 +9,13 @@ export interface MetricBase {
}
export interface MetricResult extends MetricBase {
values: [number, string][]
values: [number, string][];
}
export interface InstantMetricResult extends MetricBase {
value?: [number, string]
values?: [number, string][]
value?: [number, string];
values?: [number, string][];
}
export interface ExportMetricResult extends MetricBase {
@@ -44,9 +47,23 @@ export interface LogHits {
timestamps: string[];
values: number[];
total?: number;
fields: {
[key: string]: string;
};
fields: { [key: string]: string; };
_isOther: boolean;
}
export interface LegendLogHits {
label: string;
total: number;
totalHits: number;
isOther: boolean;
fields: { [key: string]: string; };
stroke?: uPlot.Series.Stroke;
}
export interface LegendLogHitsMenu {
title: string;
icon?: ReactNode;
handler?: () => void;
}
export interface ReportMetaData {

View File

@@ -1,22 +1,23 @@
import React, { FC, useMemo, useRef, useState } from "preact/compat";
import React, { FC, useCallback, useMemo, useRef, useState } from "preact/compat";
import "./style.scss";
import "uplot/dist/uPlot.min.css";
import useElementSize from "../../../hooks/useElementSize";
import uPlot, { AlignedData } from "uplot";
import { useEffect } from "react";
import useBarHitsOptions from "./hooks/useBarHitsOptions";
import useBarHitsOptions, { getLabelFromLogHit } from "./hooks/useBarHitsOptions";
import BarHitsTooltip from "./BarHitsTooltip/BarHitsTooltip";
import { TimeParams } from "../../../types";
import usePlotScale from "../../../hooks/uplot/usePlotScale";
import useReadyChart from "../../../hooks/uplot/useReadyChart";
import useZoomChart from "../../../hooks/uplot/useZoomChart";
import classNames from "classnames";
import { LogHits } from "../../../api/types";
import { LegendLogHits, LogHits } from "../../../api/types";
import { addSeries, delSeries, setBand } from "../../../utils/uplot";
import { GraphOptions, GRAPH_STYLES } from "./types";
import BarHitsOptions from "./BarHitsOptions/BarHitsOptions";
import stack from "../../../utils/uplot/stack";
import BarHitsLegend from "./BarHitsLegend/BarHitsLegend";
import { calculateTotalHits, sortLogHits } from "../../../utils/logs";
interface Props {
logHits: LogHits[];
@@ -57,6 +58,29 @@ const BarHitsChart: FC<Props> = ({ logHits, data: _data, period, setPeriod, onAp
graphOptions
});
const prepareLegend = useCallback((hits: LogHits[], totalHits: number): LegendLogHits[] => {
return hits.map((hit) => {
const label = getLabelFromLogHit(hit);
const legendItem: LegendLogHits = {
label,
isOther: hit._isOther,
fields: hit.fields,
total: hit.total || 0,
totalHits,
stroke: series.find((s) => s.label === label)?.stroke,
};
return legendItem;
}).sort(sortLogHits("total"));
}, [series]);
const legendDetails: LegendLogHits[] = useMemo(() => {
const totalHits = calculateTotalHits(logHits);
return prepareLegend(logHits, totalHits);
}, [logHits, prepareLegend]);
useEffect(() => {
if (!uPlotInst) return;
delSeries(uPlotInst);
@@ -121,6 +145,7 @@ const BarHitsChart: FC<Props> = ({ logHits, data: _data, period, setPeriod, onAp
<BarHitsLegend
uPlotInst={uPlotInst}
onApplyFilter={onApplyFilter}
legendDetails={legendDetails}
/>
)}
</div>

View File

@@ -1,83 +1,53 @@
import React, { FC, useCallback, useEffect, useState } from "preact/compat";
import React, { FC, useEffect, useState } from "preact/compat";
import uPlot, { Series } from "uplot";
import "./style.scss";
import "../../Line/Legend/style.scss";
import classNames from "classnames";
import { MouseEvent } from "react";
import { isMacOs } from "../../../../utils/detect-device";
import Tooltip from "../../../Main/Tooltip/Tooltip";
import { getStreamPairs } from "../../../../utils/logs";
import BarHitsLegendItem from "./BarHitsLegendItem";
import { LegendLogHits } from "../../../../api/types";
interface Props {
uPlotInst: uPlot;
legendDetails: LegendLogHits[];
onApplyFilter: (value: string) => void;
}
const BarHitsLegend: FC<Props> = ({ uPlotInst, onApplyFilter }) => {
const BarHitsLegend: FC<Props> = ({ uPlotInst, legendDetails, onApplyFilter }) => {
const [series, setSeries] = useState<Series[]>([]);
const [pairs, setPairs] = useState<string[][]>([]);
const totalHits = legendDetails[0]?.totalHits || 0;
const updateSeries = useCallback(() => {
const series = uPlotInst.series.filter(s => s.scale !== "x");
setSeries(series);
setPairs(series.map(s => getStreamPairs(s.label || "")));
const getSeries = () => {
return uPlotInst.series.filter(s => s.scale !== "x");
};
const handleRedrawGraph = () => {
uPlotInst.redraw();
setSeries(getSeries());
};
useEffect(() => {
setSeries(getSeries());
}, [uPlotInst]);
const handleClickByValue = (value: string) => (e: MouseEvent<HTMLDivElement>) => {
const metaKey = e.metaKey || e.ctrlKey;
if (!metaKey) return;
onApplyFilter(`{${value}}` || "");
updateSeries();
uPlotInst.redraw();
};
const handleClickByStream = (target: Series) => (e: MouseEvent<HTMLDivElement>) => {
const metaKey = e.metaKey || e.ctrlKey;
if (metaKey) return;
target.show = !target.show;
updateSeries();
uPlotInst.redraw();
};
useEffect(updateSeries, [uPlotInst]);
return (
<div className="vm-bar-hits-legend">
{series.map((s, i) => (
<Tooltip
key={s.label}
title={(
<ul className="vm-bar-hits-legend-info">
<li>Click to {s.show ? "hide" : "show"} the _stream.</li>
<li>{isMacOs() ? "Cmd" : "Ctrl"} + Click to filter by the _stream.</li>
</ul>
)}
>
<div
className={classNames({
"vm-bar-hits-legend-item": true,
"vm-bar-hits-legend-item_hide": !s.show,
})}
onClick={handleClickByStream(s)}
>
<div
className="vm-bar-hits-legend-item__marker"
style={{ backgroundColor: `${(s?.stroke as () => string)?.()}` }}
/>
<div className="vm-bar-hits-legend-item-pairs">
{pairs[i].map(value => (
<span
className="vm-bar-hits-legend-item-pairs__value"
key={value}
onClick={handleClickByValue(value)}
>
{value}
</span>
))}
</div>
</div>
</Tooltip>
{legendDetails.map((legend) => (
<BarHitsLegendItem
key={legend.label}
legend={legend}
series={series}
onRedrawGraph={handleRedrawGraph}
onApplyFilter={onApplyFilter}
/>
))}
<div className="vm-bar-hits-legend-info">
<div>
Total hits: <b>{totalHits.toLocaleString("en-US")}</b>
</div>
<div>
<code>L-Click</code> toggles visibility.&nbsp;
<code>R-Click</code> opens menu.
</div>
</div>
</div>
);
};

View File

@@ -0,0 +1,92 @@
import React, { FC, useMemo, useRef, useState } from "preact/compat";
import classNames from "classnames";
import { Series } from "uplot";
import { MouseEvent } from "react";
import { LegendLogHits } from "../../../../api/types";
import { getStreamPairs } from "../../../../utils/logs";
import { formatNumberShort } from "../../../../utils/math";
import Popper from "../../../Main/Popper/Popper";
import useBoolean from "../../../../hooks/useBoolean";
import LegendHitsMenu from "../LegendHitsMenu/LegendHitsMenu";
interface Props {
legend: LegendLogHits;
series: Series[];
onRedrawGraph: () => void;
onApplyFilter: (value: string) => void;
}
const BarHitsLegendItem: FC<Props> = ({ legend, series, onRedrawGraph, onApplyFilter }) => {
const {
value: openContextMenu,
setTrue: handleOpenContextMenu,
setFalse: handleCloseContextMenu,
} = useBoolean(false);
const legendRef = useRef<HTMLDivElement>(null);
const [clickPosition, setClickPosition] = useState<{ top: number; left: number } | null>(null);
const targetSeries = useMemo(() => series.find(s => s.label === legend.label), [series]);
const fields = useMemo(() => getStreamPairs(legend.label), [legend.label]);
const label = fields.join(", ");
const totalShortFormatted = formatNumberShort(legend.total);
const handleClickByStream = (e: MouseEvent<HTMLDivElement>) => {
if (!targetSeries) return;
if (e.metaKey || e.ctrlKey) {
targetSeries.show = !targetSeries.show;
} else {
const isOnlyTargetVisible = series.every(s => s === targetSeries || !s.show);
series.forEach(s => {
s.show = isOnlyTargetVisible || (s === targetSeries);
});
}
onRedrawGraph();
};
const handleContextMenu = (e: MouseEvent<HTMLDivElement>) => {
e.preventDefault();
setClickPosition({ top: e.clientY, left: e.clientX });
handleOpenContextMenu();
};
return (
<div
ref={legendRef}
className={classNames({
"vm-bar-hits-legend-item": true,
"vm-bar-hits-legend-item_other": legend.isOther,
"vm-bar-hits-legend-item_hide": !targetSeries?.show,
})}
onClick={handleClickByStream}
onContextMenu={handleContextMenu}
>
<div
className="vm-bar-hits-legend-item__marker"
style={{ backgroundColor: `${legend.stroke}` }}
/>
<div className="vm-bar-hits-legend-item__label">{label}</div>
<span className="vm-bar-hits-legend-item__total">({totalShortFormatted})</span>
<Popper
placement="fixed"
open={openContextMenu}
buttonRef={legendRef}
placementPosition={clickPosition}
onClose={handleCloseContextMenu}
>
<LegendHitsMenu
legend={legend}
fields={fields}
onApplyFilter={onApplyFilter}
onClose={handleCloseContextMenu}
/>
</Popper>
</div>
);
};
export default BarHitsLegendItem;

View File

@@ -3,16 +3,16 @@
.vm-bar-hits-legend {
display: flex;
flex-wrap: wrap;
gap: $padding-small;
padding: 0 $padding-small $padding-small;
color: $color-text;
&-item {
display: grid;
grid-template-columns: auto 1fr;
max-width: 50%;
display: flex;
align-items: center;
gap: $padding-small;
font-size: 12px;
padding: 0 $padding-small;
font-size: $font-size-small;
padding: $padding-small $padding-global;
border-radius: $border-radius-small;
cursor: pointer;
transition: 0.2s;
@@ -27,34 +27,44 @@
}
&__marker {
width: 14px;
min-width: 14px;
max-width: 14px;
height: 14px;
border: $color-background-block;
}
&-pairs {
display: flex;
gap: $padding-small;
&__label {
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
}
&__value {
padding: $padding-small 0;
&:hover {
text-decoration: underline;
}
&:after {
content: ",";
}
&:last-child:after {
content: "";
}
}
&__total {
color: $color-text-secondary;
font-style: italic;
grid-column: 2;
}
}
&-info {
list-style-position: inside;
flex-grow: 1;
width: 100%;
display: flex;
align-items: center;
justify-content: space-between;
padding-top: $padding-small;
color: $color-text-secondary;
font-size: $font-size-small;
code {
display: inline-block;
padding: calc($padding-small / 2) $padding-small;
font-size: $font-size-small;
text-align: center;
background-color: $color-background-body;
background-repeat: repeat-x;
border: $border-divider;
border-radius: 4px;
}
}
}

View File

@@ -5,6 +5,7 @@ import { DATE_TIME_FORMAT } from "../../../../constants/date";
import classNames from "classnames";
import "./style.scss";
import "../../ChartTooltip/style.scss";
import { sortLogHits } from "../../../../utils/logs";
interface Props {
data: AlignedData;
@@ -26,7 +27,7 @@ const BarHitsTooltip: FC<Props> = ({ data, focusDataIdx, uPlotInst }) => {
const tooltipItems = values.map((value, i) => {
const targetSeries = series[i + 1];
const stroke = (targetSeries?.stroke as () => string)?.();
const label = targetSeries?.label || "other";
const label = targetSeries?.label;
const show = targetSeries?.show;
return {
label,
@@ -34,7 +35,7 @@ const BarHitsTooltip: FC<Props> = ({ data, focusDataIdx, uPlotInst }) => {
value,
show
};
}).filter(item => item.value > 0 && item.show).sort((a, b) => b.value - a.value);
}).filter(item => item.value > 0 && item.show).sort(sortLogHits("value"));
const point = {
top: tooltipItems[0] ? uPlotInst?.valToPos?.(tooltipItems[0].value, "y") || 0 : 0,
@@ -104,16 +105,19 @@ const BarHitsTooltip: FC<Props> = ({ data, focusDataIdx, uPlotInst }) => {
className="vm-chart-tooltip-data__marker"
style={{ background: item.stroke }}
/>
<p>
{item.label}: <b>{item.value}</b>
<p className="vm-bar-hits-tooltip-item">
<span className="vm-bar-hits-tooltip-item__label">{item.label}</span>
<span>{item.value.toLocaleString("en-US")}</span>
</p>
</div>
))}
</div>
{tooltipData.values.length > 1 && (
<div className="vm-chart-tooltip-data">
<p>
Total records: <b>{tooltipData.total}</b>
<span/>
<p className="vm-bar-hits-tooltip-item">
<span className="vm-bar-hits-tooltip-item__label">Total</span>
<span>{tooltipData.total.toLocaleString("en-US")}</span>
</p>
</div>
)}

View File

@@ -9,4 +9,19 @@
opacity: 1;
pointer-events: auto;
}
&-item {
display: grid;
grid-template-columns: 1fr auto;
align-items: center;
gap: $padding-global;
max-width: 100%;
&__label {
display: inline-block;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
}
}

View File

@@ -0,0 +1,50 @@
import React, { FC } from "preact/compat";
import "./style.scss";
import { LegendLogHits } from "../../../../api/types";
import LegendHitsMenuStats from "./LegendHitsMenuStats";
import LegendHitsMenuBase from "./LegendHitsMenuBase";
import LegendHitsMenuRow from "./LegendHitsMenuRow";
import LegendHitsMenuFields from "./LegendHitsMenuFields";
import { LOGS_LIMIT_HITS } from "../../../../constants/logs";
const otherDescription = `aggregated results for fields not in the top ${LOGS_LIMIT_HITS}`;
interface Props {
legend: LegendLogHits;
fields: string[];
onApplyFilter: (value: string) => void;
onClose: () => void;
}
const LegendHitsMenu: FC<Props> = ({ legend, fields, onApplyFilter, onClose }) => {
return (
<div className="vm-legend-hits-menu">
<div className="vm-legend-hits-menu-section">
<LegendHitsMenuRow
className="vm-legend-hits-menu-row_info"
title={legend.isOther ? otherDescription : legend.label}
/>
</div>
{!legend.isOther && (
<LegendHitsMenuBase
legend={legend}
onApplyFilter={onApplyFilter}
onClose={onClose}
/>
)}
{!legend.isOther && (
<LegendHitsMenuFields
fields={fields}
onApplyFilter={onApplyFilter}
onClose={onClose}
/>
)}
<LegendHitsMenuStats legend={legend}/>
</div>
);
};
export default LegendHitsMenu;

View File

@@ -0,0 +1,64 @@
import React, { FC } from "preact/compat";
import LegendHitsMenuRow from "./LegendHitsMenuRow";
import useCopyToClipboard from "../../../../hooks/useCopyToClipboard";
import { CopyIcon, FilterIcon, FilterOffIcon } from "../../../Main/Icons";
import { LegendLogHits, LegendLogHitsMenu } from "../../../../api/types";
import { LOGS_GROUP_BY } from "../../../../constants/logs";
interface Props {
legend: LegendLogHits;
onApplyFilter: (value: string) => void;
onClose: () => void;
}
const LegendHitsMenuBase: FC<Props> = ({ legend, onApplyFilter, onClose }) => {
const copyToClipboard = useCopyToClipboard();
const handleAddStreamToFilter = () => {
onApplyFilter(`${LOGS_GROUP_BY}: ${legend.label}`);
onClose();
};
const handleExcludeStreamToFilter = () => {
onApplyFilter(`(NOT ${LOGS_GROUP_BY}: ${legend.label})`);
onClose();
};
const handlerCopyLabel = async () => {
await copyToClipboard(legend.label, `${legend.label} has been copied`);
onClose();
};
const options: LegendLogHitsMenu[] = [
{
title: `Copy ${LOGS_GROUP_BY} name`,
icon: <CopyIcon/>,
handler: handlerCopyLabel,
},
{
title: `Add ${LOGS_GROUP_BY} to filter`,
icon: <FilterIcon/>,
handler: handleAddStreamToFilter,
},
{
title: `Exclude ${LOGS_GROUP_BY} to filter`,
icon: <FilterOffIcon/>,
handler: handleExcludeStreamToFilter,
}
];
return (
<div className="vm-legend-hits-menu-section">
{options.map(({ icon, title, handler }) => (
<LegendHitsMenuRow
key={title}
iconStart={icon}
title={title}
handler={handler}
/>
))}
</div>
);
};
export default LegendHitsMenuBase;

View File

@@ -0,0 +1,74 @@
import React, { FC, useMemo } from "preact/compat";
import LegendHitsMenuRow from "./LegendHitsMenuRow";
import { CopyIcon, FilterIcon, FilterOffIcon } from "../../../Main/Icons";
import { convertToFieldFilter } from "../../../../utils/logs";
import { LegendLogHitsMenu } from "../../../../api/types";
import useCopyToClipboard from "../../../../hooks/useCopyToClipboard";
interface Props {
fields: string[];
onApplyFilter: (value: string) => void;
onClose: () => void;
}
const LegendHitsMenuFields: FC<Props> = ({ fields, onApplyFilter, onClose }) => {
const copyToClipboard = useCopyToClipboard();
const handleCopy = (field: string) => async () => {
await copyToClipboard(field, `${field} has been copied`);
onClose();
};
const handleAddToFilter = (field: string) => () => {
onApplyFilter(field);
onClose();
};
const handleExcludeToFilter = (field: string) => () => {
onApplyFilter(`-${field}`);
onClose();
};
const generateFieldMenu = (field: string): LegendLogHitsMenu[] => {
return [
{
title: "Copy",
icon: <CopyIcon/>,
handler: handleCopy(field),
},
{
title: "Add to filter",
icon: <FilterIcon/>,
handler: handleAddToFilter(field),
},
{
title: "Exclude to filter",
icon: <FilterOffIcon/>,
handler: handleExcludeToFilter(field),
}
];
};
const fieldsWithMenu: LegendLogHitsMenu[] = useMemo(() => {
return fields.map(field => {
const title = convertToFieldFilter(field);
return {
title,
submenu: generateFieldMenu(title),
};
});
}, [fields]);
return (
<div className="vm-legend-hits-menu-section">
{fieldsWithMenu?.map((field) => (
<LegendHitsMenuRow
key={field.title}
{...field}
/>
))}
</div>
);
};
export default LegendHitsMenuFields;

View File

@@ -0,0 +1,116 @@
import React, { FC, useRef, useState } from "preact/compat";
import classNames from "classnames";
import { ReactNode, useEffect } from "react";
import Tooltip from "../../../Main/Tooltip/Tooltip";
import { LegendLogHitsMenu } from "../../../../api/types";
import { ArrowDropDownIcon } from "../../../Main/Icons";
import useClickOutside from "../../../../hooks/useClickOutside";
interface Props {
title: string | ReactNode;
handler?: () => void;
iconStart?: ReactNode;
iconEnd?: ReactNode;
className?: string;
submenu?: LegendLogHitsMenu[];
}
const LegendHitsMenuRow: FC<Props> = ({ title, handler, iconStart, iconEnd, className, submenu }) => {
const containerRef = useRef<HTMLDivElement>(null);
const titleRef = useRef<HTMLDivElement>(null);
const submenuRef = useRef<HTMLDivElement>(null);
const [isOverflownTitle, setIsOverflownTitle] = useState(false);
const [openSubmenu, setOpenSubmenu] = useState(false);
const [posSubmenuLeft, setPosSubmenuLeft] = useState(false);
const hasSubmenu = !!submenu?.length;
const handleToggleContextMenu = () => {
setOpenSubmenu(prev => !prev);
};
const handleCloseContextMenu = () => {
setOpenSubmenu(false);
};
const handleClick = () => {
handler && handler();
hasSubmenu && handleToggleContextMenu();
};
useEffect(() => {
if (!titleRef.current) return;
setIsOverflownTitle(titleRef.current.scrollWidth > titleRef.current.clientWidth);
}, [title, titleRef]);
useEffect(() => {
requestAnimationFrame(() => {
if (!openSubmenu || !submenuRef.current) {
setPosSubmenuLeft(false);
return;
}
const { left, width } = submenuRef.current.getBoundingClientRect();
setPosSubmenuLeft(left + width > window.innerWidth);
});
}, [submenuRef, openSubmenu]);
useClickOutside(containerRef, handleCloseContextMenu);
const titleContent = (
<div
ref={titleRef}
className="vm-legend-hits-menu-row__title"
>
{title}
</div>
);
return (
<div
ref={containerRef}
className={classNames({
"vm-legend-hits-menu-row": true,
"vm-legend-hits-menu-row_interactive": !!handler || hasSubmenu,
[`${className}`]: className
})}
onClick={handleClick}
>
{iconStart && <div className="vm-legend-hits-menu-row__icon">{iconStart}</div>}
{isOverflownTitle ? (<Tooltip title={title}>{titleContent}</Tooltip>) : titleContent}
{iconEnd && !hasSubmenu && <div className="vm-legend-hits-menu-row__icon">{iconEnd}</div>}
{hasSubmenu && (
<div className="vm-legend-hits-menu-row__icon vm-legend-hits-menu-row__icon_drop">
<ArrowDropDownIcon/>
</div>
)}
{openSubmenu && submenu && (
<div
ref={submenuRef}
className={classNames({
"vm-legend-hits-menu": true,
"vm-legend-hits-menu_submenu": true,
"vm-legend-hits-menu_submenu_left": posSubmenuLeft
})}
>
<div className="vm-legend-hits-menu-section">
{submenu.map(({ icon, title, handler }) => (
<LegendHitsMenuRow
key={title}
iconStart={icon}
title={title}
handler={handler}
/>
))}
</div>
</div>
)}
</div>
);
};
export default LegendHitsMenuRow;

View File

@@ -0,0 +1,23 @@
import React, { FC } from "preact/compat";
import { LegendLogHits } from "../../../../api/types";
interface Props {
legend: LegendLogHits;
}
const LegendHitsMenuStats: FC<Props> = ({ legend }) => {
const totalFormatted = legend.total.toLocaleString("en-US");
const percentage = Math.round((legend.total / legend.totalHits) * 100);
return (
<div className="vm-legend-hits-menu-section">
<div className="vm-legend-hits-menu-row">
<div className="vm-legend-hits-menu-row__title">
Total: {totalFormatted} ({percentage}%)
</div>
</div>
</div>
);
};
export default LegendHitsMenuStats;

View File

@@ -0,0 +1,178 @@
@use "src/styles/variables" as *;
.vm-legend-hits-menu {
min-width: 160px;
z-index: 1;
&_submenu {
position: absolute;
top: calc(-1 * $padding-small);
background-color: $color-background-block;
left: calc(100% + ($padding-small / 2));
box-shadow: $box-shadow-popper;
border-radius: $border-radius-small;
animation: vm-submenu-show 150ms cubic-bezier(0.280, 0.840, 0.2, 1);
transform-origin: top left;
&_left {
left: auto;
right: calc(100% + ($padding-small / 2));
transform-origin: top right;
}
}
&-section {
border-bottom: $border-divider;
&:last-child {
border-bottom: none;
}
}
&-row {
position: relative;
display: flex;
gap: $padding-small;
align-items: center;
justify-content: flex-start;
padding: 0 $padding-global;
transition: background-color 0.3s;
color: $color-text;
&_interactive {
cursor: pointer;
&:hover {
background-color: rgba(0, 0, 0, 0.05);
}
}
&_info {
font-size: $font-size-small;
font-weight: 500;
padding-block: $padding-small;
}
&_info &__icon {
color: $color-info;
}
&__icon {
display: flex;
align-items: center;
justify-content: center;
width: 14px;
height: 14px;
&_drop {
transform: rotate(-90deg);
}
}
&__title {
flex-grow: 1;
padding: $padding-global 0;
position: relative;
max-width: 400px;
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
}
}
&-other-list {
width: 80vw;
height: 80vh;
overflow: auto;
&__search {
position: sticky;
top: 0;
padding: $padding-small 0;
background-color: $color-background-block;
border-bottom: $border-divider;
z-index: 2;
}
&-row {
border-bottom: $border-divider;
&_header {
border-bottom: none;
position: sticky;
top: 65px;
background-color: $color-background-block;
z-index: 1;
width: 100%;
&:after {
content: '';
position: absolute;
bottom: 0;
left: 0;
width: 100%;
height: 1px;
border-bottom: $border-divider;
}
}
}
&-cell {
padding: calc($padding-small / 2) 0;
text-align: left;
&_header {
padding: $padding-small;
font-weight: 500;
}
&_number {
padding: $padding-small;
text-align: right;
font-variant-numeric: tabular-nums;
}
&_fields {
width: 100%;
}
}
&-fields {
display: flex;
flex-wrap: wrap;
align-items: center;
justify-content: flex-start;
&__field {
padding: calc($padding-small / 2) $padding-small;
border-radius: $border-radius-small;
transition: background-color 0.3s;
&:hover {
background-color: $color-hover-black;
}
&:not(:last-child) {
&:after {
content: ',';
}
}
}
}
&-actions {
display: flex;
align-items: center;
justify-content: center;
}
}
}
@keyframes vm-submenu-show {
0% {
opacity: 0;
}
100% {
opacity: 1;
}
}

View File

@@ -36,6 +36,14 @@ interface UseGetBarHitsOptionsArgs {
graphOptions: GraphOptions;
}
export const OTHER_HITS_LABEL = "other";
export const getLabelFromLogHit = (logHit: LogHits) => {
if (logHit?._isOther) return OTHER_HITS_LABEL;
const fields = Object.values(logHit?.fields || {});
return fields.map((value) => value || "\"\"").join(", ");
};
const useBarHitsOptions = ({
data,
logHits,
@@ -59,12 +67,12 @@ const useBarHitsOptions = ({
let colorN = 0;
return data.map((_d, i) => {
if (i === 0) return {}; // 0 index is xAxis(timestamps)
const fields = Object.values(logHits?.[i - 1]?.fields || {});
const label = fields.map((value) => value || "\"\"").join(", ");
const color = getCssVariable(label ? seriesColors[colorN] : "color-log-hits-bar-0");
if (label) colorN++;
const target = logHits?.[i - 1];
const label = getLabelFromLogHit(target);
const color = getCssVariable(target?._isOther ? "color-log-hits-bar-0" : seriesColors[colorN]);
if (!target?._isOther) colorN++;
return {
label: label || "other",
label,
width: strokeWidth[graphOptions.graphStyle],
spanGaps: true,
stroke: color,

View File

@@ -32,6 +32,11 @@ $chart-tooltip-y: -1 * ($padding-global + $chart-tooltip-half-icon);
max-width: calc(100vw/3);
}
&_hits &-data {
display: grid;
grid-template-columns: $font-size 1fr;
}
&_sticky {
pointer-events: auto;
z-index: 99;
@@ -90,6 +95,8 @@ $chart-tooltip-y: -1 * ($padding-global + $chart-tooltip-half-icon);
}
&__marker {
min-width: $font-size;
max-width: $font-size;
width: $font-size;
height: $font-size;
border: 1px solid rgba($color-white, 0.5);

View File

@@ -124,7 +124,7 @@ const QueryEditor: FC<QueryEditorProps> = ({
};
useEffect(() => {
setOpenAutocomplete(!!AutocompleteEl);
setOpenAutocomplete(!!AutocompleteEl && autocompleteQuick);
}, [autocompleteQuick]);
useEffect(() => {

View File

@@ -581,3 +581,45 @@ export const CommentIcon = () => (
></path>
</svg>
);
export const FilterIcon = () => (
<svg
viewBox="0 0 24 24"
fill="currentColor"
>
<path
d="M4.25 5.61C6.27 8.2 10 13 10 13v6c0 .55.45 1 1 1h2c.55 0 1-.45 1-1v-6s3.72-4.8 5.74-7.39c.51-.66.04-1.61-.79-1.61H5.04c-.83 0-1.3.95-.79 1.61"
></path>
</svg>
);
export const FilterOffIcon = () => (
<svg
viewBox="0 0 24 24"
fill="currentColor"
>
<path
d="M19.79 5.61C20.3 4.95 19.83 4 19 4H6.83l7.97 7.97zM2.81 2.81 1.39 4.22 10 13v6c0 .55.45 1 1 1h2c.55 0 1-.45 1-1v-2.17l5.78 5.78 1.41-1.41z"
></path>
</svg>
);
export const OpenNewIcon = () => (
<svg
viewBox="0 0 24 24"
fill="currentColor"
>
<path
d="M19 19H5V5h7V3H5c-1.11 0-2 .9-2 2v14c0 1.1.89 2 2 2h14c1.1 0 2-.9 2-2v-7h-2zM14 3v2h3.59l-9.83 9.83 1.41 1.41L19 6.41V10h2V3z"
></path>
</svg>
);
export const ModalIcon = () => (
<svg
viewBox="0 0 24 24"
fill="currentColor"
>
<path d="M19 4H5c-1.11 0-2 .9-2 2v12c0 1.1.89 2 2 2h14c1.1 0 2-.9 2-2V6c0-1.1-.89-2-2-2m0 14H5V8h14z"></path>
</svg>
);

View File

@@ -15,9 +15,10 @@ interface PopperProps {
open: boolean
onClose: () => void
buttonRef: React.RefObject<HTMLElement>
placement?: "bottom-right" | "bottom-left" | "top-left" | "top-right"
placement?: "bottom-right" | "bottom-left" | "top-left" | "top-right" | "fixed"
placementPosition?: { top: number, left: number } | null
animation?: string
offset?: {top: number, left: number}
offset?: { top: number, left: number }
clickOutside?: boolean,
fullWidth?: boolean
title?: string
@@ -29,6 +30,7 @@ const Popper: FC<PopperProps> = ({
children,
buttonRef,
placement = "bottom-left",
placementPosition,
open = false,
onClose,
offset = { top: 6, left: 0 },
@@ -92,13 +94,18 @@ const Popper: FC<PopperProps> = ({
if (needAlignRight) position.left = buttonPos.right - popperSize.width;
if (needAlignTop) position.top = buttonPos.top - popperSize.height - offsetTop;
const { innerWidth, innerHeight } = window;
const margin = 20;
if (placement === "fixed" && placementPosition) {
position.top = Math.max(placementPosition.top + offset.top, 0);
position.left = Math.max(placementPosition.left + offset.left, 0);
return position;
}
const isOverflowBottom = (position.top + popperSize.height + margin) > innerHeight;
const isOverflowTop = (position.top - margin) < 0;
const isOverflowRight = (position.left + popperSize.width + margin) > innerWidth;
const isOverflowLeft = (position.left - margin) < 0;
const { innerWidth, innerHeight } = window;
const isOverflowBottom = (position.top + popperSize.height) > innerHeight;
const isOverflowTop = (position.top) < 0;
const isOverflowRight = (position.left + popperSize.width) > innerWidth;
const isOverflowLeft = (position.left) < 0;
if (isOverflowBottom) position.top = buttonPos.top - popperSize.height - offsetTop;
if (isOverflowTop) position.top = buttonPos.height + buttonPos.top + offsetTop;
@@ -106,11 +113,11 @@ const Popper: FC<PopperProps> = ({
if (isOverflowLeft) position.left = buttonPos.left + offsetLeft;
if (fullWidth) position.width = `${buttonPos.width}px`;
if (position.top < 0) position.top = 20;
if (position.left < 0) position.left = 20;
if (position.top < 0) position.top = 0;
if (position.left < 0) position.left = 0;
return position;
},[buttonRef, placement, isOpen, children, fullWidth]);
}, [buttonRef, placement, isOpen, children, fullWidth]);
const handleClickClose = (e: ReactMouseEvent<HTMLButtonElement, MouseEvent>) => {
e.stopPropagation();
@@ -131,10 +138,10 @@ const Popper: FC<PopperProps> = ({
if (!popperRef.current || !isOpen || (isMobile && !disabledFullScreen)) return;
const { right, width } = popperRef.current.getBoundingClientRect();
if (right > window.innerWidth) {
const left = window.innerWidth - 20 - width;
popperRef.current.style.left = left < window.innerWidth ? "0" : `${left}px`;
const left = window.innerWidth - width;
popperRef.current.style.left = `${left}px`;
}
}, [isOpen, popperRef]);
}, [isOpen, popperRef, placementPosition]);
const handlePopstate = useCallback(() => {
if (isOpen && isMobile && !disabledFullScreen) {

View File

@@ -11,7 +11,7 @@
border-radius: $border-radius-small;
&_open {
z-index: 101;
z-index: 100;
opacity: 1;
transform-origin: top center;
animation: vm-slider 150ms cubic-bezier(0.280, 0.840, 0.420, 1.1);

View File

@@ -11,7 +11,7 @@ import useBoolean from "../../../hooks/useBoolean";
import TextField from "../../Main/TextField/TextField";
import { KeyboardEvent, useState } from "react";
import Modal from "../../Main/Modal/Modal";
import { getFromStorage, removeFromStorage, saveToStorage } from "../../../utils/storage";
import { useSearchParams } from "react-router-dom";
const title = "Table settings";
@@ -30,6 +30,8 @@ const TableSettings: FC<TableSettingsProps> = ({
onChangeColumns,
toggleTableCompact
}) => {
const [searchParams, setSearchParams] = useSearchParams();
const buttonRef = useRef<HTMLDivElement>(null);
const {
@@ -38,11 +40,6 @@ const TableSettings: FC<TableSettingsProps> = ({
setFalse: handleClose,
} = useBoolean(false);
const {
value: saveColumns,
toggle: toggleSaveColumns,
} = useBoolean(Boolean(getFromStorage("TABLE_COLUMNS")));
const [searchColumn, setSearchColumn] = useState("");
const [indexFocusItem, setIndexFocusItem] = useState(-1);
@@ -60,15 +57,34 @@ const TableSettings: FC<TableSettingsProps> = ({
return filteredColumns.every(col => selectedColumns.includes(col));
}, [selectedColumns, filteredColumns]);
const handleChangeDisplayColumns = (displayColumns: string[]) => {
onChangeColumns(displayColumns);
const updatedParams = new URLSearchParams(searchParams.toString());
const isAllCheck = displayColumns.length === columns.length;
if (isAllCheck) {
updatedParams.delete("columns");
} else {
updatedParams.set("columns", displayColumns.map(encodeURIComponent).join(","));
}
setSearchParams(updatedParams);
};
const handleChange = (key: string) => {
onChangeColumns(selectedColumns.includes(key) ? selectedColumns.filter(col => col !== key) : [...selectedColumns, key]);
const displayColumns = selectedColumns.includes(key)
? selectedColumns.filter(col => col !== key)
: [...selectedColumns, key];
handleChangeDisplayColumns(displayColumns);
};
const toggleAllColumns = () => {
if (isAllChecked) {
onChangeColumns(selectedColumns.filter(col => !filteredColumns.includes(col)));
handleChangeDisplayColumns(selectedColumns.filter(col => !filteredColumns.includes(col)));
} else {
onChangeColumns(filteredColumns);
handleChangeDisplayColumns(filteredColumns);
}
};
@@ -95,22 +111,16 @@ const TableSettings: FC<TableSettingsProps> = ({
};
useEffect(() => {
if (arrayEquals(columns, selectedColumns) || saveColumns) return;
if (arrayEquals(columns, selectedColumns) || searchParams.has("columns")) return;
onChangeColumns(columns);
}, [columns]);
useEffect(() => {
if (!saveColumns) {
removeFromStorage(["TABLE_COLUMNS"]);
} else if (selectedColumns.length) {
saveToStorage("TABLE_COLUMNS", selectedColumns.join(","));
}
}, [saveColumns, selectedColumns]);
useEffect(() => {
const saveColumns = getFromStorage("TABLE_COLUMNS") as string;
if (!saveColumns) return;
onChangeColumns(saveColumns.split(","));
const hasColumns = searchParams.has("columns");
if (!hasColumns) return;
const columnsParam = searchParams.get("columns") || "";
const columnsArray = columnsParam.split(",").map(decodeURIComponent).filter(Boolean);
onChangeColumns(columnsArray);
}, []);
return (
@@ -183,19 +193,6 @@ const TableSettings: FC<TableSettingsProps> = ({
</div>
))}
</div>
<div className="vm-table-settings-modal-preserve">
<Checkbox
checked={saveColumns}
onChange={toggleSaveColumns}
label={"Preserve column settings"}
disabled={tableCompact}
color={"primary"}
/>
<p className="vm-table-settings-modal-preserve__info">
This label indicates that when the checkbox is activated,
the current column configurations will not be reset.
</p>
</div>
</div>
</div>
<div className="vm-table-settings-modal-section">

View File

@@ -3,6 +3,7 @@
.vm-table-settings {
&-modal {
.vm-modal-content-body {
min-width: clamp(300px, 600px, 90vw);
padding: 0;
}
@@ -83,16 +84,5 @@
}
}
}
&-preserve {
padding: $padding-global;
&__info {
padding-top: $padding-small;
font-size: $font-size-small;
color: $color-text-secondary;
line-height: 130%;
}
}
}
}

View File

@@ -2,6 +2,7 @@ import { DATE_TIME_FORMAT } from "./date";
export const LOGS_ENTRIES_LIMIT = 50;
export const LOGS_BARS_VIEW = 100;
export const LOGS_LIMIT_HITS = 5;
// "Ungrouped" is a string that is used as a value for the "groupBy" parameter.
export const WITHOUT_GROUPING = "Ungrouped";

View File

@@ -69,7 +69,7 @@ const ExploreLogs: FC = () => {
};
const handleApplyFilter = (val: string) => {
setQuery(prev => `_stream: ${val === "other" ? "{}" : val} AND (${prev})`);
setQuery(prev => `${val} AND (${prev})`);
setIsUpdatingQuery(true);
};

View File

@@ -43,7 +43,7 @@ const GroupLogs: FC<Props> = ({ logs, settingsRef }) => {
values,
pairs,
};
}).sort((a, b) => a.keysString.localeCompare(b.keysString)); // groups sorting
}).sort((a, b) => b.values.length - a.values.length); // groups sorting
}, [logs, groupBy]);
const handleToggleExpandAll = useCallback(() => {

View File

@@ -6,6 +6,7 @@ import { MouseEvent, useState } from "react";
import useCopyToClipboard from "../../../hooks/useCopyToClipboard";
import { useSearchParams } from "react-router-dom";
import { LOGS_GROUP_BY, LOGS_URL_PARAMS } from "../../../constants/logs";
import { convertToFieldFilter } from "../../../utils/logs";
interface Props {
pair: string;
@@ -23,8 +24,7 @@ const GroupLogsHeaderItem: FC<Props> = ({ pair, isHide }) => {
const handleClickByPair = (value: string) => async (e: MouseEvent<HTMLDivElement>) => {
e.stopPropagation();
const isKeyValue = /(.+)?=(".+")/.test(value);
const copyValue = isKeyValue ? `${value.replace(/=/, ": ")}` : `${groupBy}: "${value}"`;
const copyValue = convertToFieldFilter(value, groupBy);
const isCopied = await copyToClipboard(copyValue);
if (isCopied) {
setCopied(value);

View File

@@ -268,7 +268,7 @@ $font-size-logs: var(--font-size-logs, $font-size-small);
border: $border-divider;
border-radius: $border-radius-small;
overflow: auto;
height: 300px;
max-height: 300px;
resize: vertical;
font-family: $font-family-monospace;
font-size: $font-size-logs;

View File

@@ -4,6 +4,8 @@ import { ErrorTypes, TimeParams } from "../../../types";
import { LogHits } from "../../../api/types";
import { useSearchParams } from "react-router-dom";
import { getHitsTimeParams } from "../../../utils/logs";
import { LOGS_GROUP_BY, LOGS_LIMIT_HITS } from "../../../constants/logs";
import { isEmptyObject } from "../../../utils/object";
export const useFetchLogHits = (server: string, query: string) => {
const [searchParams] = useSearchParams();
@@ -30,46 +32,12 @@ export const useFetchLogHits = (server: string, query: string) => {
step: `${step}ms`,
start: start.toISOString(),
end: end.toISOString(),
field: "_stream" // In the future, this field can be made configurable
fields_limit: `${LOGS_LIMIT_HITS}`,
field: LOGS_GROUP_BY,
})
};
};
const accumulateHits = (resultHit: LogHits, hit: LogHits) => {
resultHit.total = (resultHit.total || 0) + (hit.total || 0);
hit.timestamps.forEach((timestamp, i) => {
const index = resultHit.timestamps.findIndex(t => t === timestamp);
if (index === -1) {
resultHit.timestamps.push(timestamp);
resultHit.values.push(hit.values[i]);
} else {
resultHit.values[index] += hit.values[i];
}
});
return resultHit;
};
const getHitsWithTop = (hits: LogHits[]) => {
const topN = 5;
const defaultHit = { fields: {}, timestamps: [], values: [], total: 0 };
const hitsByTotal = hits.sort((a, b) => (b.total || 0) - (a.total || 0));
const result = [];
const otherHits: LogHits = hitsByTotal.slice(topN).reduce(accumulateHits, defaultHit);
if (otherHits.total) {
result.push(otherHits);
}
const topHits: LogHits[] = hitsByTotal.slice(0, topN);
if (topHits.length) {
result.push(...topHits);
}
return result;
};
const fetchLogHits = useCallback(async (period: TimeParams) => {
abortControllerRef.current.abort();
abortControllerRef.current = new AbortController();
@@ -98,7 +66,7 @@ export const useFetchLogHits = (server: string, query: string) => {
setError(error);
}
setLogHits(!hits ? [] : getHitsWithTop(hits));
setLogHits(hits.map(hit => ({ ...hit, _isOther: isEmptyObject(hit.fields) })));
} catch (e) {
if (e instanceof Error && e.name !== "AbortError") {
setError(String(e));

View File

@@ -1,6 +1,8 @@
import { TimeParams } from "../types";
import dayjs from "dayjs";
import { LOGS_BARS_VIEW } from "../constants/logs";
import { LOGS_BARS_VIEW, LOGS_GROUP_BY } from "../constants/logs";
import { LogHits } from "../api/types";
import { OTHER_HITS_LABEL } from "../components/Chart/BarHitsChart/hooks/useBarHitsOptions";
export const getStreamPairs = (value: string): string[] => {
const pairs = /^{.+}$/.test(value) ? value.slice(1, -1).split(",") : [value];
@@ -14,3 +16,27 @@ export const getHitsTimeParams = (period: TimeParams) => {
const step = Math.ceil(totalSeconds / LOGS_BARS_VIEW) || 1;
return { start, end, step };
};
export const convertToFieldFilter = (value: string, field = LOGS_GROUP_BY) => {
const isKeyValue = /(.+)?=(".+")/.test(value);
if (isKeyValue) {
return value.replace(/=/, ": ");
}
return `${field}: "${value}"`;
};
export const calculateTotalHits = (hits: LogHits[]): number => {
return hits.reduce((acc, item) => acc + (item.total || 0), 0);
};
export const sortLogHits = <T extends { label?: string }>(key: keyof T) => (a: T, b: T): number => {
if (a.label === OTHER_HITS_LABEL) return 1;
if (b.label === OTHER_HITS_LABEL) return -1;
const aValue = a[key] as unknown as number;
const bValue = b[key] as unknown as number;
return bValue - aValue;
};

View File

@@ -57,3 +57,15 @@ export const getLastFromArray = (a: number[]) => {
}
}
};
export const formatNumberShort = (value: number) => {
if (value >= 1_000_000_000) {
return (value / 1_000_000_000).toFixed(1).replace(/\.0$/, "") + "B"; // Миллиарды
} else if (value >= 1_000_000) {
return (value / 1_000_000).toFixed(1).replace(/\.0$/, "") + "M"; // Миллионы
} else if (value >= 1_000) {
return (value / 1_000).toFixed(1).replace(/\.0$/, "") + "K"; // Тысячи
} else {
return value.toString(); // Для чисел меньше 1000
}
};

View File

@@ -14,3 +14,7 @@ export function filterObject<T extends object>(
export function compactObject<T extends object>(obj: T) {
return filterObject(obj, (entry) => !!entry[1] || typeof entry[1] === "number");
}
export function isEmptyObject(obj: object) {
return Object.keys(obj).length === 0;
}

View File

@@ -3,7 +3,6 @@ export type StorageKeys = "AUTOCOMPLETE"
| "QUERY_TRACING"
| "SERIES_LIMITS"
| "TABLE_COMPACT"
| "TABLE_COLUMNS"
| "TIMEZONE"
| "DISABLED_DEFAULT_TIMEZONE"
| "THEME"

View File

@@ -44,15 +44,16 @@ type PrometheusWriteQuerier interface {
// QueryOpts contains various params used for querying or ingesting data
type QueryOpts struct {
Tenant string
Timeout string
Start string
End string
Time string
Step string
ExtraFilters []string
ExtraLabels []string
Trace string
Tenant string
Timeout string
Start string
End string
Time string
Step string
ExtraFilters []string
ExtraLabels []string
Trace string
IsNonBlocking bool
}
func (qos *QueryOpts) asURLValues() url.Values {

View File

@@ -0,0 +1,229 @@
package tests
import (
"fmt"
"io"
"net"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/VictoriaMetrics/VictoriaMetrics/apptest"
)
// TestClusterReplicationLatency checks ingestion perfomance in various scenarios
// and data consistency
//
// It wraps network connections to the storage with proxy
// which could emualte network errors and timeouts
func TestClusterReplicationLatency(t *testing.T) {
tc := apptest.NewTestCase(t)
defer tc.Stop()
const (
replicationFactor = 1
cycles = 10
rps = 10_000
wantIngestedRows = rps * cycles * replicationFactor
)
// spin up 1 cluster and wrap it with proxy that emulate storage nodes
// do not use multiple storage nodes
// since it add delay for tsid registrations
// and may have negative impact on test results with extra delays
vmstorage := tc.MustStartVmstorage("vmstorage", []string{
"-storageDataPath=" + tc.Dir() + "/vmstorage",
})
laggingStorage0 := newLaggingProxyWrap(t, vmstorage.VminsertAddr())
defer laggingStorage0.close()
laggingStorage1 := newLaggingProxyWrap(t, vmstorage.VminsertAddr())
defer laggingStorage1.close()
laggingStorage2 := newLaggingProxyWrap(t, vmstorage.VminsertAddr())
defer laggingStorage2.close()
vminsert := tc.MustStartVminsert("vminsert", []string{
"-storageNode=" + fmt.Sprintf("%s,%s,%s", laggingStorage0.listenAddr(), laggingStorage1.listenAddr(), laggingStorage2.listenAddr()),
fmt.Sprintf("-replicationFactor=%d", replicationFactor),
"-memory.allowedBytes=1500000",
"-disableRerouting=false",
})
// emulate network delays and disconnects here
// uncomment or add needed lines
laggingStorage0.startLag()
ds := genDataset(rps)
var (
requestsInTime int
requestsTimeouts int
)
// start ingestion with configure requests per seconds
for range cycles {
ct := time.Now()
// use non blocking mode for ingestion
// since it may introduce timeouts
vminsert.PrometheusAPIV1ImportPrometheus(t, ds, apptest.QueryOpts{Tenant: "1", IsNonBlocking: true})
since := time.Since(ct)
if since > time.Second {
requestsTimeouts++
} else {
requestsInTime++
}
toSleep := time.Second - since
if toSleep > 0 {
time.Sleep(toSleep)
}
}
vmstorage.ForceFlush(t)
if requestsTimeouts > 0 {
t.Errorf("unexpected result, got requests timeouts=%d out of %d, in time %d", requestsTimeouts, cycles, requestsInTime)
}
// verify that metrics actually ingested
ct := time.Now()
var ingestedTotal int
// use 10 seconds as timeout
for range 100 {
ingestedTotal = 0
ingested := vminsert.GetMetricsByPrefix(t, "vm_rpc_rows_sent_total")
for _, v := range ingested {
ingestedTotal += int(v)
}
if ingestedTotal >= wantIngestedRows {
break
}
time.Sleep(time.Millisecond * 100)
}
since := time.Since(ct)
if since > time.Second {
t.Logf("WARN: data ingestion check took: %s", since)
}
if ingestedTotal != wantIngestedRows {
t.Fatalf("unexpected ingested metrics=%d want=%d, it took time=%s", ingestedTotal, wantIngestedRows, since)
}
}
func genDataset(size int) []string {
var ds []string
for i := range size {
ds = append(ds, fmt.Sprintf("metric_%d 15\n", i))
}
return ds
}
type laggingProxy struct {
l net.Listener
dstAddr string
shouldLag atomic.Bool
shouldDisconnect atomic.Bool
wg sync.WaitGroup
}
func newLaggingProxyWrap(t *testing.T, dstAddr string) *laggingProxy {
l, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("cannot start proxy: %s", err)
}
lp := &laggingProxy{
l: l,
dstAddr: dstAddr,
}
lp.wg.Add(1)
go lp.run()
return lp
}
func (lp *laggingProxy) listenAddr() string {
return lp.l.Addr().String()
}
func (lp *laggingProxy) run() {
defer lp.wg.Done()
for {
src, err := lp.l.Accept()
if err != nil {
println("exiting at err: ", err.Error())
break
}
if lp.shouldDisconnect.Load() {
src.Close()
continue
}
go func() {
dst, err := net.Dial("tcp", lp.dstAddr)
if err != nil {
println("err dial: ", err.Error())
return
}
laggingDst := wrapConnWithLag(dst, &lp.shouldLag, &lp.shouldDisconnect)
go io.Copy(src, laggingDst)
io.Copy(laggingDst, src)
laggingDst.Close()
src.Close()
}()
}
}
func (lp *laggingProxy) startLag() {
lp.shouldLag.Store(true)
}
func (lp *laggingProxy) stopLag() {
lp.shouldLag.Store(false)
}
func (lp *laggingProxy) rejectConnections() {
lp.shouldDisconnect.Store(true)
}
func (lp *laggingProxy) stopRejectConnections() {
lp.shouldDisconnect.Store(false)
}
func (lp *laggingProxy) close() {
lp.l.Close()
lp.wg.Wait()
}
type laggingWriteReader struct {
origin net.Conn
shouldLag *atomic.Bool
shouldDisconnect *atomic.Bool
mu sync.Mutex
lagDelay int
maxDelay int
}
func wrapConnWithLag(origin net.Conn, lagOn *atomic.Bool, disconnectOn *atomic.Bool) *laggingWriteReader {
return &laggingWriteReader{origin: origin, maxDelay: 5, shouldLag: lagOn, shouldDisconnect: disconnectOn}
}
func (lwr *laggingWriteReader) lag() {
if lwr.shouldDisconnect.Load() {
lwr.origin.Close()
return
}
if !lwr.shouldLag.Load() {
return
}
time.Sleep(time.Second * time.Duration(lwr.lagDelay))
lwr.mu.Lock()
defer lwr.mu.Unlock()
lwr.lagDelay++
if lwr.lagDelay > lwr.maxDelay {
lwr.lagDelay = 0
}
}
func (lwr *laggingWriteReader) Read(p []byte) (n int, err error) {
lwr.lag()
return lwr.origin.Read(p)
}
func (lwr *laggingWriteReader) Write(p []byte) (n int, err error) {
lwr.lag()
return lwr.origin.Write(p)
}
func (lwr *laggingWriteReader) Close() error {
return lwr.origin.Close()
}

View File

@@ -171,4 +171,49 @@ func TestClusterMultiTenantSelect(t *testing.T) {
t.Errorf("unexpected response (-want, +got):\n%s", diff)
}
// Delete series from specific tenant
vmselect.DeleteSeries(t, "foo_bar", apptest.QueryOpts{
Tenant: "5:15",
})
wantSR = apptest.NewPrometheusAPIV1SeriesResponse(t,
`{"data": [
{"__name__":"foo_bar", "vm_account_id":"0", "vm_project_id":"10"},
{"__name__":"foo_bar", "vm_account_id":"1", "vm_project_id":"1"},
{"__name__":"foo_bar", "vm_account_id":"1", "vm_project_id":"15"},
{"__name__":"foo_bar", "vm_account_id":"5", "vm_project_id":"0"}
]
}`)
wantSR.Sort()
gotSR = vmselect.PrometheusAPIV1Series(t, "foo_bar", apptest.QueryOpts{
Tenant: "multitenant",
Start: "2022-05-10T08:03:00.000Z",
})
gotSR.Sort()
if diff := cmp.Diff(wantSR, gotSR, cmpSROpt); diff != "" {
t.Errorf("unexpected response (-want, +got):\n%s", diff)
}
// Delete series for multitenant with tenant filter
vmselect.DeleteSeries(t, `foo_bar{vm_account_id="1"}`, apptest.QueryOpts{
Tenant: "multitenant",
})
wantSR = apptest.NewPrometheusAPIV1SeriesResponse(t,
`{"data": [
{"__name__":"foo_bar", "vm_account_id":"0", "vm_project_id":"10"},
{"__name__":"foo_bar", "vm_account_id":"5", "vm_project_id":"0"}
]
}`)
wantSR.Sort()
gotSR = vmselect.PrometheusAPIV1Series(t, `foo_bar`, apptest.QueryOpts{
Tenant: "multitenant",
Start: "2022-05-10T08:03:00.000Z",
})
gotSR.Sort()
if diff := cmp.Diff(wantSR, gotSR, cmpSROpt); diff != "" {
t.Errorf("unexpected response (-want, +got):\n%s", diff)
}
}

View File

@@ -123,6 +123,10 @@ func (app *Vminsert) PrometheusAPIV1ImportPrometheus(t *testing.T, records []str
url += "?" + uvs
}
data := []byte(strings.Join(records, "\n"))
if opts.IsNonBlocking {
app.cli.Post(t, url, "text/plain", data, http.StatusNoContent)
return
}
app.sendBlocking(t, len(records), func() {
app.cli.Post(t, url, "text/plain", data, http.StatusNoContent)
})
@@ -133,6 +137,10 @@ func (app *Vminsert) String() string {
return fmt.Sprintf("{app: %s httpListenAddr: %q}", app.app, app.httpListenAddr)
}
func (app *Vminsert) ListenAddr() string {
return app.httpListenAddr
}
// sendBlocking sends the data to vmstorage by executing `send` function and
// waits until the data is actually sent.
//

View File

@@ -117,6 +117,22 @@ func (app *Vmselect) PrometheusAPIV1Series(t *testing.T, matchQuery string, opts
return NewPrometheusAPIV1SeriesResponse(t, res)
}
// DeleteSeries sends a query to a /prometheus/api/v1/admin/tsdb/delete_series
//
// See https://docs.victoriametrics.com/url-examples/#apiv1admintsdbdelete_series
func (app *Vmselect) DeleteSeries(t *testing.T, matchQuery string, opts QueryOpts) {
t.Helper()
seriesURL := fmt.Sprintf("http://%s/delete/%s/prometheus/api/v1/admin/tsdb/delete_series", app.httpListenAddr, opts.getTenant())
values := opts.asURLValues()
values.Add("match[]", matchQuery)
res := app.cli.PostForm(t, seriesURL, values, http.StatusNoContent)
if res != "" {
t.Fatalf("unexpected non-empty DeleteSeries response=%q", res)
}
}
// String returns the string representation of the vmselect app state.
func (app *Vmselect) String() string {
return fmt.Sprintf("{app: %s httpListenAddr: %q}", app.app, app.httpListenAddr)

View File

@@ -1,7 +1,7 @@
dashboard-copy:
echo "" > dashboards/vm/${SRC}
cat dashboards/${SRC} >> dashboards/vm/${SRC}
sed -i='.tmp' 's/prometheus/victoriametrics-datasource/g' dashboards/vm/${SRC}
sed -i='.tmp' 's/prometheus/victoriametrics-metrics-datasource/g' dashboards/vm/${SRC}
sed -i='.tmp' 's/Prometheus/VictoriaMetrics/g' dashboards/vm/${SRC}
sed -i='.tmp' 's/${D_UID}/${D_UID}_vm/g' dashboards/vm/${SRC}
sed -i='.tmp' 's/"title": "${TITLE}"/"title": "${TITLE} (VM)"/g' dashboards/vm/${SRC}

View File

@@ -17,7 +17,7 @@
},
{
"type": "datasource",
"id": "victoriametrics-datasource",
"id": "victoriametrics-metrics-datasource",
"name": "VictoriaMetrics",
"version": "1.0.0"
},
@@ -85,7 +85,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -133,7 +133,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -150,7 +150,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Status of last backup operation.",
@@ -210,7 +210,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -225,7 +225,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Number of backups stored in remote storage.",
@@ -275,7 +275,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -290,7 +290,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Space used in remote storage.",
@@ -341,7 +341,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -356,7 +356,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "",
@@ -420,7 +420,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -435,7 +435,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Status of last retention run.\n\nRetention is a process of removing old backups from remote storage.",
@@ -495,7 +495,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -510,7 +510,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -573,7 +573,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -588,7 +588,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -674,7 +674,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -715,7 +715,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -801,7 +801,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -852,7 +852,7 @@
"panels": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Max duration of backup run. Lower better.\n\nEach backup starts with data upload during `latest` backup. Subsequent backups (`hourly`, `daily`, `weekly`, `monthly`) are copying date by using server-side copy. ",
@@ -931,7 +931,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -946,7 +946,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -1020,7 +1020,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1035,7 +1035,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -1110,7 +1110,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1125,7 +1125,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -1184,7 +1184,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1227,7 +1227,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -1298,7 +1298,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1313,7 +1313,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -1387,7 +1387,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1416,7 +1416,7 @@
"panels": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Max duration of retention run. Lower better.\n\nRetention is a process of removing old backups from remote storage.",
@@ -1485,7 +1485,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1500,7 +1500,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -1574,7 +1574,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1603,7 +1603,7 @@
"panels": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Duration of backup run. Lower better.\n\nEach backup starts with data upload during `latest` backup. Subsequent backups (`hourly`, `daily`, `weekly`, `monthly`) are copying date by using server-side copy.\n",
@@ -1671,7 +1671,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1705,7 +1705,7 @@
"multi": false,
"name": "ds",
"options": [],
"query": "victoriametrics-datasource",
"query": "victoriametrics-metrics-datasource",
"queryValue": "",
"refresh": 1,
"regex": "",
@@ -1764,7 +1764,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"filters": [],

View File

@@ -17,7 +17,7 @@
},
{
"type": "datasource",
"id": "victoriametrics-datasource",
"id": "victoriametrics-metrics-datasource",
"name": "VictoriaMetrics",
"version": "1.0.0"
},
@@ -80,7 +80,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "How many datapoints are inserted into storage per second by accountID and projectID",
@@ -168,7 +168,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -184,7 +184,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Request rate accepted by vmselect nodes per tenant",
@@ -272,7 +272,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -290,7 +290,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the number of active time series with new data points inserted during the last hour. High value may result in ingestion slowdown. \n\nSee following link for details:",
@@ -385,7 +385,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -403,7 +403,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Time spent on query execution per tenant per second",
@@ -491,7 +491,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -509,7 +509,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the amount of on-disk space occupied by data points only. The disk space is used for storing by datapoint and indexdb. There is no option to expose per tenant statistic for indexdb. Usually, indexed takes much less space compared to datapoints. But with a high churn rate, the size of the indexdb could grow significantly.",
@@ -596,7 +596,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -614,7 +614,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Number of new series created over last 24h.",
@@ -702,7 +702,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -731,7 +731,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -785,7 +785,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -800,7 +800,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -853,7 +853,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -868,7 +868,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -921,7 +921,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -936,7 +936,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -991,7 +991,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1006,7 +1006,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -1059,7 +1059,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1074,7 +1074,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -1127,7 +1127,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1142,7 +1142,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -1242,7 +1242,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1260,7 +1260,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -1360,7 +1360,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1378,7 +1378,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -1478,7 +1478,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1496,7 +1496,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -1596,7 +1596,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1614,7 +1614,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -1714,7 +1714,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1732,7 +1732,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -1832,7 +1832,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1868,7 +1868,7 @@
"multi": false,
"name": "ds",
"options": [],
"query": "victoriametrics-datasource",
"query": "victoriametrics-metrics-datasource",
"queryValue": "",
"refresh": 1,
"regex": "",
@@ -1879,7 +1879,7 @@
"allValue": ".*",
"current": {},
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"definition": "label_values(vm_tenant_active_timeseries, accountID)",
@@ -1905,7 +1905,7 @@
"allValue": ".*",
"current": {},
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"definition": "label_values(vm_tenant_active_timeseries{accountID=~\"$accountID\"},projectID)",
@@ -1929,7 +1929,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "PE8D8DB4BEE4E4B22"
},
"filters": [],

View File

@@ -33,7 +33,7 @@
{
"collapsed": false,
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"gridPos": {
@@ -47,7 +47,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"refId": "A"
@@ -58,7 +58,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"gridPos": {
@@ -81,7 +81,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"refId": "A"
@@ -92,7 +92,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Number of objects at kubernetes cluster per each controller",
@@ -147,7 +147,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -162,7 +162,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -215,7 +215,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -234,7 +234,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": " Shows per namespace watchers for VictoriaMetrics Operator objects (ServiceMonitors, PodMonitors, etc) ",
@@ -288,11 +288,11 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
"expr": "sum(operator_victoriametrics-datasource_converter_active_watchers)",
"expr": "sum(operator_victoriametrics-metrics-datasource_converter_active_watchers)",
"instant": false,
"legendFormat": "__auto",
"range": true,
@@ -304,7 +304,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": " Number of operator instances with obtained leader status. \n Value above 1 indicates that instances with the same job may behave incorrectly.\n It's recommend to check Operator logs. ",
@@ -358,7 +358,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -374,7 +374,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": " Shows number of active reconcile workers",
@@ -428,7 +428,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -444,7 +444,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": " Shows amount of VictoriaMetrics Operator objects processed by Operator.",
@@ -527,11 +527,11 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
"expr": "sum(rate(operator_victoriametrics-datasource_converter_watch_events_total{job=~\"$job\"}[$__interval])) by (event_type,object_type_name)",
"expr": "sum(rate(operator_victoriametrics-metrics-datasource_converter_watch_events_total{job=~\"$job\"}[$__interval])) by (event_type,object_type_name)",
"instant": false,
"legendFormat": "{{object_type_name}} {{event_type}}",
"range": true,
@@ -543,7 +543,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -627,7 +627,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -642,7 +642,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the rate of logging the messages by their level. Unexpected spike in rate is a good reason to check logs.",
@@ -727,7 +727,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -743,7 +743,7 @@
{
"collapsed": false,
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"gridPos": {
@@ -757,7 +757,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"refId": "A"
@@ -768,7 +768,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Non zero metrics indicates about error with CR object definition (typos or incorrect values) or errors with kubernetes API connection.",
@@ -852,7 +852,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -865,7 +865,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -881,7 +881,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Operator limits number of reconcile configuration events to 5 events per 2 seconds by default.",
@@ -965,7 +965,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -980,7 +980,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Number of objects waiting in the queue for reconciliation. Non-zero values indicate that operator cannot process CR objects changes with the given resources.",
@@ -1064,7 +1064,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1079,7 +1079,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": " For controllers with StatefulSet it's ok to see latency greater then 3 seconds. It could be vmalertmanager,vmcluster or vmagent in statefulMode.\n\n For other controllers, latency greater then 2 second may indicate issues with kubernetes cluster or operator's performance.\n ",
@@ -1165,7 +1165,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1180,7 +1180,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Number of HTTP requests to the Kubernetes API server break down by code and method",
@@ -1262,7 +1262,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1278,7 +1278,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows how many ongoing reconcile events are taking place, where:\n* `max` - equal to the value of flag`-controller.maxConcurrentReconciles`;\n* `current` - current number of reconcile workers processing CRD objects.\n\nWhen `current` hits `max` constantly, it means operator cannot process events in time. It should be either increased value for flag `-controller.maxConcurrentReconciles` or allocated additional CPU resources to the operator.",
@@ -1377,7 +1377,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1389,7 +1389,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1406,7 +1406,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the time goroutines have spent in runnable state before actually running. The lower is better.\n\nHigh values or values exceeding the threshold is usually a sign of insufficient CPU resources or CPU throttling. \n\nVerify that service has enough CPU resources. Otherwise, the service could work unreliably with delays in processing.",
@@ -1486,7 +1486,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1502,7 +1502,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": " Requests latency to the Kubernetes API server.",
@@ -1585,7 +1585,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1602,7 +1602,7 @@
{
"collapsed": true,
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"gridPos": {
@@ -1615,7 +1615,7 @@
"panels": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -1699,7 +1699,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1710,7 +1710,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1722,7 +1722,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1734,7 +1734,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1750,7 +1750,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -1830,7 +1830,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1845,7 +1845,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -1925,7 +1925,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1940,7 +1940,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -2023,7 +2023,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -2040,7 +2040,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"refId": "A"
@@ -2065,7 +2065,7 @@
"multi": false,
"name": "ds",
"options": [],
"query": "victoriametrics-datasource",
"query": "victoriametrics-metrics-datasource",
"queryValue": "te",
"refresh": 1,
"regex": "",
@@ -2075,7 +2075,7 @@
{
"current": {},
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"definition": "label_values(operator_log_messages_total,job)",
@@ -2097,7 +2097,7 @@
{
"current": {},
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"definition": "label_values(operator_log_messages_total{job=~\"$job\"},instance)",
@@ -2119,7 +2119,7 @@
{
"current": {},
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"definition": "label_values(vm_app_version{job=\"$job\", instance=\"$instance\"}, version)",

View File

@@ -11,7 +11,7 @@
},
{
"type": "datasource",
"id": "victoriametrics-datasource",
"id": "victoriametrics-metrics-datasource",
"name": "VictoriaMetrics",
"version": "1.0.0"
},
@@ -50,7 +50,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"enable": true,
@@ -63,7 +63,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"enable": true,
@@ -97,7 +97,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "How many log entries are in storage",
@@ -149,7 +149,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -168,7 +168,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the logs ingestion rate.",
@@ -220,7 +220,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -239,7 +239,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "The ratio of original data size and compressed data stored on disk",
@@ -291,7 +291,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -310,7 +310,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Total number of available CPUs for VM process",
@@ -366,7 +366,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -385,7 +385,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -439,7 +439,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -456,7 +456,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Total amount of used disk space",
@@ -508,7 +508,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -527,7 +527,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the rate of HTTP read requests.",
@@ -579,7 +579,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -598,7 +598,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Total size of available memory for VM process",
@@ -680,7 +680,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "How many logs are inserted into storage per second",
@@ -769,7 +769,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -787,7 +787,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "* `*` - unsupported query path\n* `/insert` - insert into VM\n* `/metrics` - query VL system metrics\n* `/query` - read the data",
@@ -877,7 +877,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -895,7 +895,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the amount of on-disk space occupied by data before and after compressiom",
@@ -985,7 +985,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1003,7 +1003,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "The number of the new log streams created over the last 24h",
@@ -1093,7 +1093,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1111,7 +1111,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the number of restarts per job. The chart can be useful to identify periodic process restarts and correlate them with potential issues or anomalies. Normally, processes shouldn't restart unless restart was inited by user. The reason of restarts should be figured out by checking the logs of each specific service. ",
@@ -1200,7 +1200,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1226,7 +1226,7 @@
"panels": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Percentage of used memory (resident).\nThe application's performance will significantly degrade when memory usage is close to 100%.",
@@ -1315,7 +1315,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1332,7 +1332,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "",
@@ -1421,7 +1421,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"expr": "sum(go_memstats_sys_bytes{job=~\"$job\", instance=~\"$instance\"}) + sum(vm_cache_size_bytes{job=~\"$job\", instance=~\"$instance\"})",
@@ -1433,7 +1433,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"expr": "sum(go_memstats_heap_inuse_bytes{job=~\"$job\", instance=~\"$instance\"}) + sum(vm_cache_size_bytes{job=~\"$job\", instance=~\"$instance\"})",
@@ -1445,7 +1445,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"expr": "sum(go_memstats_stack_inuse_bytes{job=~\"$job\", instance=~\"$instance\"})",
@@ -1457,7 +1457,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"expr": "sum(process_resident_memory_bytes{job=~\"$job\", instance=~\"$instance\"})",
@@ -1470,7 +1470,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"exemplar": false,
@@ -1488,7 +1488,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Share for memory allocated by the process itself. When memory usage reaches 100% it will be likely OOM-killed.\nSafe memory usage % considered to be below 80%",
@@ -1577,7 +1577,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1594,7 +1594,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -1682,7 +1682,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1701,7 +1701,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the percentage of open file descriptors compared to the limit set in the OS.\nReaching the limit of open files can cause various issues and must be prevented.\n\nSee how to change limits here https://medium.com/@muhammadtriwibowo/set-permanently-ulimit-n-open-files-in-ubuntu-4d61064429a",
@@ -1807,7 +1807,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1826,7 +1826,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "",
@@ -1931,7 +1931,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1945,7 +1945,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"exemplar": false,
@@ -1963,7 +1963,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -2052,7 +2052,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -2069,7 +2069,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the number of bytes read/write from the storage layer.",
@@ -2170,7 +2170,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"expr": "sum(rate(process_io_storage_read_bytes_total{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))",
@@ -2183,7 +2183,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"expr": "sum(rate(process_io_storage_written_bytes_total{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))",
@@ -2200,7 +2200,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -2289,7 +2289,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -2306,7 +2306,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the number of read/write syscalls such as read, pread, write, pwrite.",
@@ -2407,7 +2407,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -2422,7 +2422,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -2441,7 +2441,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "",
@@ -2530,7 +2530,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -2548,7 +2548,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "",
@@ -2637,7 +2637,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -2655,7 +2655,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the time goroutines have spent in runnable state before actually running. The lower is better.\n\nHigh values or values exceeding the threshold is usually a sign of insufficient CPU resources or CPU throttling. \n\nVerify that service has enough CPU resources. Otherwise, the service could work unreliably with delays in processing.",
@@ -2743,7 +2743,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -2780,7 +2780,7 @@
"multi": false,
"name": "ds",
"options": [],
"query": "victoriametrics-datasource",
"query": "victoriametrics-metrics-datasource",
"queryValue": "",
"refresh": 1,
"regex": "",
@@ -2790,7 +2790,7 @@
{
"current": {},
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"definition": "label_values(vm_app_version{version=~\"victoria-logs-.*\"}, job)",
@@ -2812,7 +2812,7 @@
{
"current": {},
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"definition": "label_values(vm_app_version{job=~\"$job\"}, instance)",
@@ -2833,7 +2833,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"filters": [],

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@@ -11,7 +11,7 @@
},
{
"type": "datasource",
"id": "victoriametrics-datasource",
"id": "victoriametrics-metrics-datasource",
"name": "VictoriaMetrics",
"version": "1.0.0"
},
@@ -56,7 +56,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"enable": true,
@@ -69,7 +69,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"enable": true,
@@ -140,7 +140,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows if the last configuration update was successful. \"Not Ok\" means there was an unsuccessful attempt to update the configuration due to some error. Check the log for details.",
@@ -210,7 +210,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"exemplar": false,
@@ -225,7 +225,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the total number of loaded alerting rules across selected instances and groups.",
@@ -273,7 +273,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"exemplar": false,
@@ -288,7 +288,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the total number of loaded recording rules across selected instances and groups.",
@@ -336,7 +336,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"exemplar": false,
@@ -351,7 +351,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the total number of errors generated by recording/alerting rules for selected instances and groups.",
@@ -403,7 +403,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"exemplar": false,
@@ -418,7 +418,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows number of Recording Rules which produce no data.\n\n Usually it means that such rules are misconfigured, since they give no output during the evaluation.\nPlease check if rule's expression is correct and it is working as expected.",
@@ -470,7 +470,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -487,7 +487,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -568,7 +568,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -584,7 +584,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -672,7 +672,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -716,7 +716,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the number of fired alerts by job.",
@@ -804,7 +804,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -821,7 +821,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Top $topk groups by evaluation duration. Shows groups that take the most of time during the evaluation across all instances.\n\nThe panel uses MetricsQL functions and may not work with VictoriaMetrics.",
@@ -909,7 +909,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -926,7 +926,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows how many requests (executions) per second vmalert sends to the configured datasource.",
@@ -1011,7 +1011,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1028,7 +1028,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the error rate while executing configured rules. Non-zero value means there are some issues with existing rules. Check the logs to get more details.",
@@ -1113,7 +1113,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1144,7 +1144,7 @@
"panels": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "The precentage of used RSS memory\n\nIf you think that usage is abnormal or unexpected, please file an issue and attach memory profile if possible.",
@@ -1241,7 +1241,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1258,7 +1258,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Amount of used RSS memory\n\nIf you think that usage is abnormal or unexpected, please file an issue and attach memory profile if possible.",
@@ -1354,7 +1354,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1371,7 +1371,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the CPU usage percentage per vmalert instance. \nIf you think that usage is abnormal or unexpected pls file an issue and attach CPU profile if possible.",
@@ -1468,7 +1468,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1487,7 +1487,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the max number of CPU cores used by a `job` and the corresponding limit.",
@@ -1584,7 +1584,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1599,7 +1599,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1619,7 +1619,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Panel shows the percentage of open file descriptors in the OS.\nReaching the limit of open files can cause various issues and must be prevented.\n\nSee how to change limits here https://medium.com/@muhammadtriwibowo/set-permanently-ulimit-n-open-files-in-ubuntu-4d61064429a",
@@ -1709,7 +1709,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1728,7 +1728,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -1817,7 +1817,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1835,7 +1835,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the percent of CPU spent on garbage collection.\n\nIf % is high, then CPU usage can be decreased by changing GOGC to higher values. Increasing GOGC value will increase memory usage, and decrease CPU usage.\n\nTry searching for keyword `GOGC` at https://docs.victoriametrics.com/troubleshooting/ ",
@@ -1925,7 +1925,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1943,7 +1943,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the time goroutines have spent in runnable state before actually running. The lower is better.\n\nHigh values or values exceeding the threshold is usually a sign of insufficient CPU resources or CPU throttling. \n\nVerify that service has enough CPU resources. Otherwise, the service could work unreliably with delays in processing.",
@@ -2031,7 +2031,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -2072,7 +2072,7 @@
"panels": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -2157,7 +2157,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -2202,7 +2202,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Missed evaluation means that group evaluation time takes longer than the configured evaluation interval. \nThis may result in missed alerting notifications or recording rules samples. Try increasing evaluation interval or concurrency for such groups. See https://docs.victoriametrics.com/vmalert/#groups\n\nIf rule expressions are taking longer than expected, please see https://docs.victoriametrics.com/troubleshooting/#slow-queries.\"",
@@ -2287,7 +2287,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -2304,7 +2304,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the number of restarts per job. The chart can be useful to identify periodic process restarts and correlate them with potential issues or anomalies. Normally, processes shouldn't restart unless restart was inited by user. The reason of restarts should be figured out by checking the logs of each specific service. ",
@@ -2392,7 +2392,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -2426,7 +2426,7 @@
"panels": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows top $topk current active (firing) alerting rules.\n\nThe panel uses MetricsQL functions and may not work with VictoriaMetrics.",
@@ -2511,7 +2511,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -2528,7 +2528,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the events when rule execution resulted into an error. Check the logs for more details.",
@@ -2613,7 +2613,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -2630,7 +2630,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the current pending alerting rules per group.\nBy pending means the rule which remains active less than configured `for` parameter.",
@@ -2715,7 +2715,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -2732,7 +2732,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the error rate for the attempts to send alerts to Alertmanager. If not zero it means there issues on attempt to send notification to Alertmanager and some alerts may be not delivered properly. Check the logs for more details.",
@@ -2816,7 +2816,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"exemplar": false,
@@ -2831,7 +2831,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows how many alerts are sent to Alertmanager per second. Only active alerts are sent.",
@@ -2915,7 +2915,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -2959,7 +2959,7 @@
"panels": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the top $topk recording rules which generate the most of [samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples). Each generated sample is basically a time series which then ingested into configured remote storage. Rules with high numbers may cause the most pressure on the remote database and become a source of too high cardinality.\n\nThe panel uses MetricsQL functions and may not work with VictoriaMetrics.",
@@ -3044,7 +3044,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -3061,7 +3061,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the rules which do not produce any [samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) during the evaluation. Usually it means that such rules are misconfigured, since they give no output during the evaluation.\nPlease check if rule's expression is correct and it is working as expected.",
@@ -3146,7 +3146,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -3163,7 +3163,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -3245,7 +3245,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -3285,7 +3285,7 @@
"panels": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -3361,7 +3361,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -3376,7 +3376,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the number of datapoints dropped by vmalert while sending to the configured remote write URL. vmalert performs up to 5 retries before dropping the data. Check vmalert's error logs for the specific error message.",
@@ -3453,7 +3453,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -3468,7 +3468,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows current number of established connections to remote write endpoints.\n\n",
@@ -3554,7 +3554,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -3571,7 +3571,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the global rate for number of written bytes via remote write connections.",
@@ -3657,7 +3657,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -3696,7 +3696,7 @@
"multi": false,
"name": "ds",
"options": [],
"query": "victoriametrics-datasource",
"query": "victoriametrics-metrics-datasource",
"queryValue": "",
"refresh": 1,
"regex": "",
@@ -3706,7 +3706,7 @@
{
"current": {},
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"definition": "label_values(vm_app_version{version=~\"^vmalert.*\"}, job)",
@@ -3729,7 +3729,7 @@
"allValue": ".*",
"current": {},
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"definition": "label_values(vm_app_version{job=~\"$job\"}, instance)",
@@ -3752,7 +3752,7 @@
"allValue": ".*",
"current": {},
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"definition": "label_values(vmalert_iteration_total{job=~\"$job\", instance=~\"$instance\"},file)",
@@ -3775,7 +3775,7 @@
"allValue": ".*",
"current": {},
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"definition": "label_values(vmalert_iteration_total{job=~\"$job\", instance=~\"$instance\"}, group)",
@@ -3843,7 +3843,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"filters": [],

View File

@@ -11,7 +11,7 @@
},
{
"type": "datasource",
"id": "victoriametrics-datasource",
"id": "victoriametrics-metrics-datasource",
"name": "VictoriaMetrics",
"version": "1.0.0"
},
@@ -116,7 +116,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -204,7 +204,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -222,7 +222,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows if the last configuration update was successful. \"Not Ok\" means there was an unsuccessful attempt to update the configuration due to some error. Check the log for details.",
@@ -292,7 +292,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"exemplar": false,
@@ -307,7 +307,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the rate of requests.",
@@ -356,7 +356,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -372,7 +372,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the total number of users defined at configuration file.",
@@ -420,7 +420,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -437,7 +437,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the rate of request errors.",
@@ -486,7 +486,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -502,7 +502,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -581,7 +581,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -611,7 +611,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -690,7 +690,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -702,7 +702,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -718,7 +718,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows percent utilization of per concurrent requests capacity.",
@@ -805,7 +805,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -822,7 +822,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the rate of rejected requests by a reason.",
@@ -902,7 +902,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -918,7 +918,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": " The number of concurrent connections processed by vmauth reached one of limits. Possible solutions:\n- increase global limit with flag -maxConcurrentRequests\n- increase limit with flag: -maxConcurrentPerUserRequests for all users or with config option `max_concurrent_requests` per user.\n- deploy additional vmauth replicas\n- check requests latency at backend service and allocate resources to it if needed",
@@ -998,7 +998,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1010,7 +1010,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1022,7 +1022,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1038,7 +1038,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows duration in seconds of user requests by quantile.",
@@ -1124,7 +1124,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1136,7 +1136,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1162,7 +1162,7 @@
"panels": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Percentage of used memory (resident).\nThe application's performance will significantly degrade when memory usage is close to 100%.",
@@ -1251,7 +1251,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1268,7 +1268,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -1356,7 +1356,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1375,7 +1375,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "",
@@ -1464,7 +1464,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"expr": "sum(go_memstats_sys_bytes{job=~\"$job\", instance=~\"$instance\"}) + sum(vm_cache_size_bytes{job=~\"$job\", instance=~\"$instance\"})",
@@ -1476,7 +1476,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"expr": "sum(go_memstats_heap_inuse_bytes{job=~\"$job\", instance=~\"$instance\"}) + sum(vm_cache_size_bytes{job=~\"$job\", instance=~\"$instance\"})",
@@ -1488,7 +1488,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"expr": "sum(go_memstats_stack_inuse_bytes{job=~\"$job\", instance=~\"$instance\"})",
@@ -1500,7 +1500,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"expr": "sum(process_resident_memory_bytes{job=~\"$job\", instance=~\"$instance\"})",
@@ -1513,7 +1513,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"exemplar": false,
@@ -1531,7 +1531,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "",
@@ -1636,7 +1636,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"expr": "rate(process_cpu_seconds_total{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])",
@@ -1648,7 +1648,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"exemplar": false,
@@ -1666,7 +1666,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "",
@@ -1755,7 +1755,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"expr": "sum(vm_tcplistener_conns{job=~\"$job\", instance=~\"$instance\"})",
@@ -1771,7 +1771,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "",
@@ -1860,7 +1860,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -1878,7 +1878,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the percentage of open file descriptors compared to the limit set in the OS.\nReaching the limit of open files can cause various issues and must be prevented.\n\nSee how to change limits here https://medium.com/@muhammadtriwibowo/set-permanently-ulimit-n-open-files-in-ubuntu-4d61064429a",
@@ -1984,7 +1984,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -2003,7 +2003,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -2092,7 +2092,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"expr": "sum(go_goroutines{job=~\"$job\", instance=~\"$instance\"})",
@@ -2107,7 +2107,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -2196,7 +2196,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"expr": "sum(process_num_threads{job=~\"$job\", instance=~\"$instance\"})",
@@ -2225,7 +2225,7 @@
"panels": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
@@ -2311,7 +2311,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -2352,7 +2352,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows number of generated error and warning messages in logs. Non-zero value may be a sign of connectivity or missconfiguration errors.",
@@ -2433,7 +2433,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -2448,7 +2448,7 @@
},
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows the number of restarts per job. The chart can be useful to identify periodic process restarts and correlate them with potential issues or anomalies. Normally, processes shouldn't restart unless restart was inited by user. The reason of restarts should be figured out by checking the logs of each specific service. ",
@@ -2537,7 +2537,7 @@
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
@@ -2573,7 +2573,7 @@
"multi": false,
"name": "ds",
"options": [],
"query": "victoriametrics-datasource",
"query": "victoriametrics-metrics-datasource",
"queryValue": "",
"refresh": 1,
"regex": "",
@@ -2583,7 +2583,7 @@
{
"current": {},
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"definition": "label_values(vm_app_version{version=~\"^vmauth.*\"}, job)",
@@ -2606,7 +2606,7 @@
"allValue": ".*",
"current": {},
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"definition": "label_values(vm_app_version{job=~\"$job\"}, instance)",
@@ -2628,7 +2628,7 @@
{
"current": {},
"datasource": {
"type": "victoriametrics-datasource",
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"definition": "label_values(vmauth_user_requests_total{job=~\"$job\", instance=~\"$instance\"}, username)",

View File

@@ -4,7 +4,7 @@ services:
# And forward them to --remoteWrite.url
vmagent:
container_name: vmagent
image: victoriametrics/vmagent:v1.109.1
image: victoriametrics/vmagent:v1.110.0
depends_on:
- "vminsert"
ports:
@@ -39,7 +39,7 @@ services:
# where N is number of vmstorages (2 in this case).
vmstorage-1:
container_name: vmstorage-1
image: victoriametrics/vmstorage:v1.109.1-cluster
image: victoriametrics/vmstorage:v1.110.0-cluster
ports:
- 8482
- 8400
@@ -51,7 +51,7 @@ services:
restart: always
vmstorage-2:
container_name: vmstorage-2
image: victoriametrics/vmstorage:v1.109.1-cluster
image: victoriametrics/vmstorage:v1.110.0-cluster
ports:
- 8482
- 8400
@@ -66,7 +66,7 @@ services:
# pre-process them and distributes across configured vmstorage shards.
vminsert:
container_name: vminsert
image: victoriametrics/vminsert:v1.109.1-cluster
image: victoriametrics/vminsert:v1.110.0-cluster
depends_on:
- "vmstorage-1"
- "vmstorage-2"
@@ -81,7 +81,7 @@ services:
# vmselect collects results from configured `--storageNode` shards.
vmselect-1:
container_name: vmselect-1
image: victoriametrics/vmselect:v1.109.1-cluster
image: victoriametrics/vmselect:v1.110.0-cluster
depends_on:
- "vmstorage-1"
- "vmstorage-2"
@@ -94,7 +94,7 @@ services:
restart: always
vmselect-2:
container_name: vmselect-2
image: victoriametrics/vmselect:v1.109.1-cluster
image: victoriametrics/vmselect:v1.110.0-cluster
depends_on:
- "vmstorage-1"
- "vmstorage-2"
@@ -112,7 +112,7 @@ services:
# It can be used as an authentication proxy.
vmauth:
container_name: vmauth
image: victoriametrics/vmauth:v1.109.1
image: victoriametrics/vmauth:v1.110.0
depends_on:
- "vmselect-1"
- "vmselect-2"
@@ -127,7 +127,7 @@ services:
# vmalert executes alerting and recording rules
vmalert:
container_name: vmalert
image: victoriametrics/vmalert:v1.109.1
image: victoriametrics/vmalert:v1.110.0
depends_on:
- "vmauth"
ports:

View File

@@ -16,7 +16,7 @@ services:
- ./../../dashboards/victoriametrics.json:/var/lib/grafana/dashboards/vm.json
- ./../../dashboards/victorialogs.json:/var/lib/grafana/dashboards/vl.json
environment:
- "GF_INSTALL_PLUGINS=https://github.com/VictoriaMetrics/victorialogs-datasource/releases/download/v0.13.3/victoriametrics-logs-datasource-v0.13.3.zip;victoriametrics-logs-datasource"
- "GF_INSTALL_PLUGINS=https://github.com/VictoriaMetrics/victorialogs-datasource/releases/download/v0.13.4/victoriametrics-logs-datasource-v0.13.4.zip;victoriametrics-logs-datasource"
- "GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=victoriametrics-logs-datasource"
networks:
- vm_net
@@ -45,7 +45,7 @@ services:
# storing logs and serving read queries.
victorialogs:
container_name: victorialogs
image: victoriametrics/victoria-logs:v1.7.0-victorialogs
image: victoriametrics/victoria-logs:v1.8.0-victorialogs
command:
- "--storageDataPath=/vlogs"
- "--httpListenAddr=:9428"
@@ -60,7 +60,7 @@ services:
# scraping, storing metrics and serve read requests.
victoriametrics:
container_name: victoriametrics
image: victoriametrics/victoria-metrics:v1.109.1
image: victoriametrics/victoria-metrics:v1.110.0
ports:
- 8428:8428
volumes:
@@ -79,7 +79,7 @@ services:
# depending on the requested path.
vmauth:
container_name: vmauth
image: victoriametrics/vmauth:v1.109.1
image: victoriametrics/vmauth:v1.110.0
depends_on:
- "victoriametrics"
- "victorialogs"
@@ -96,7 +96,7 @@ services:
# vmalert executes alerting and recording rules according to given rule type.
vmalert:
container_name: vmalert
image: victoriametrics/vmalert:v1.109.1
image: victoriametrics/vmalert:v1.110.0
depends_on:
- "vmauth"
- "alertmanager"

View File

@@ -4,7 +4,7 @@ services:
# And forward them to --remoteWrite.url
vmagent:
container_name: vmagent
image: victoriametrics/vmagent:v1.109.1
image: victoriametrics/vmagent:v1.110.0
depends_on:
- "victoriametrics"
ports:
@@ -22,7 +22,7 @@ services:
# storing metrics and serve read requests.
victoriametrics:
container_name: victoriametrics
image: victoriametrics/victoria-metrics:v1.109.1
image: victoriametrics/victoria-metrics:v1.110.0
ports:
- 8428:8428
- 8089:8089
@@ -65,7 +65,7 @@ services:
# vmalert executes alerting and recording rules
vmalert:
container_name: vmalert
image: victoriametrics/vmalert:v1.109.1
image: victoriametrics/vmalert:v1.110.0
depends_on:
- "victoriametrics"
- "alertmanager"

View File

@@ -2,7 +2,7 @@ apiVersion: 1
datasources:
- name: VictoriaMetrics - cluster
type: victoriametrics-datasource
type: victoriametrics-metrics-datasource
access: proxy
url: http://vmauth:8427/select/0/prometheus
isDefault: true

View File

@@ -2,7 +2,7 @@ apiVersion: 1
datasources:
- name: VictoriaMetrics
type: victoriametrics-datasource
type: victoriametrics-metrics-datasource
access: proxy
url: http://victoriametrics:8428
isDefault: true

View File

@@ -1,7 +1,7 @@
services:
# meta service will be ignored by compose
.victorialogs:
image: docker.io/victoriametrics/victoria-logs:v1.7.0-victorialogs
image: docker.io/victoriametrics/victoria-logs:v1.8.0-victorialogs
command:
- -storageDataPath=/vlogs
- -loggerFormat=json
@@ -19,7 +19,7 @@ services:
retries: 10
dd-proxy:
image: docker.io/victoriametrics/vmauth:v1.109.1
image: docker.io/victoriametrics/vmauth:v1.110.0
restart: on-failure
volumes:
- ./:/etc/vmauth
@@ -45,7 +45,7 @@ services:
replicas: 0
victoriametrics:
image: victoriametrics/victoria-metrics:v1.109.1
image: victoriametrics/victoria-metrics:v1.110.0
ports:
- '8428:8428'
command:

View File

@@ -16,6 +16,6 @@ services:
- ./../../dashboards/vm/vmalert.json:/var/lib/grafana/dashboards/vmalert.json
- ./../../dashboards/vm/vmauth.json:/var/lib/grafana/dashboards/vmauth.json
environment:
- "GF_INSTALL_PLUGINS=https://github.com/VictoriaMetrics/victoriametrics-datasource/releases/download/v0.11.1/victoriametrics-datasource-v0.11.1.zip;victoriametrics-datasource"
- "GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=victoriametrics-datasource"
- "GF_INSTALL_PLUGINS=https://github.com/VictoriaMetrics/victoriametrics-datasource/releases/download/v0.12.1/victoriametrics-metrics-datasource-v0.12.1.zip;victoriametrics-metrics-datasource"
- "GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=victoriametrics-metrics-datasource"
restart: always

View File

@@ -15,8 +15,8 @@ services:
- ./../../dashboards/vm/vmagent.json:/var/lib/grafana/dashboards/vmagent.json
- ./../../dashboards/vm/vmalert.json:/var/lib/grafana/dashboards/vmalert.json
environment:
- "GF_INSTALL_PLUGINS=https://github.com/VictoriaMetrics/victoriametrics-datasource/releases/download/v0.11.1/victoriametrics-datasource-v0.11.1.zip;victoriametrics-datasource"
- "GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=victoriametrics-datasource"
- "GF_INSTALL_PLUGINS=https://github.com/VictoriaMetrics/victoriametrics-datasource/releases/download/v0.12.1/victoriametrics-metrics-datasource-v0.12.1.zip;victoriametrics-metrics-datasource"
- "GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=victoriametrics-metrics-datasource"
networks:
- vm_net
restart: always

View File

@@ -1,7 +1,7 @@
services:
vmagent:
container_name: vmagent
image: victoriametrics/vmagent:v1.109.1
image: victoriametrics/vmagent:v1.110.0
depends_on:
- "victoriametrics"
ports:
@@ -18,7 +18,7 @@ services:
victoriametrics:
container_name: victoriametrics
image: victoriametrics/victoria-metrics:v1.109.1
image: victoriametrics/victoria-metrics:v1.110.0
ports:
- 8428:8428
volumes:
@@ -50,7 +50,7 @@ services:
vmalert:
container_name: vmalert
image: victoriametrics/vmalert:v1.109.1
image: victoriametrics/vmalert:v1.110.0
depends_on:
- "victoriametrics"
ports:
@@ -72,7 +72,7 @@ services:
restart: always
vmanomaly:
container_name: vmanomaly
image: victoriametrics/vmanomaly:v1.19.1
image: victoriametrics/vmanomaly:v1.19.2
depends_on:
- "victoriametrics"
ports:

View File

@@ -1,21 +1,23 @@
schedulers:
periodic:
# class: "scheduler.periodic.PeriodicScheduler"
infer_every: "1m"
fit_every: "2m"
fit_window: "3h"
fit_every: "1h"
fit_window: "2d" # 2d-14d based on the presense of weekly seasonality in your data
models:
prophet:
class: "model.prophet.ProphetModel"
class: "prophet"
args:
interval_width: 0.98
weekly_seasonality: False # comment it if your data has weekly seasonality
yearly_seasonality: False
reader:
datasource_url: "http://victoriametrics:8428/"
sampling_period: "60s"
sampling_period: "60s"
queries:
node_cpu_rate: "sum(rate(node_cpu_seconds_total[5m])) by (mode, instance, job)"
node_cpu_rate:
expr: "sum(rate(node_cpu_seconds_total[5m])) by (mode, instance, job)"
writer:
datasource_url: "http://victoriametrics:8428/"
@@ -24,4 +26,4 @@ writer:
monitoring:
pull: # Enable /metrics endpoint.
addr: "0.0.0.0"
port: 8490
port: 8490

View File

@@ -3,7 +3,7 @@ version: "3"
services:
# Run `make package-victoria-logs` to build victoria-logs image
vlogs:
image: docker.io/victoriametrics/victoria-logs:v1.7.0-victorialogs
image: docker.io/victoriametrics/victoria-logs:v1.8.0-victorialogs
volumes:
- vlogs:/vlogs
ports:

View File

@@ -761,7 +761,7 @@ Some workloads may need fine-grained resource usage limits. In these cases the f
- `-search.maxDeleteSeries` at `vmselect` limits the number of unique time
series that can be deleted by a single
[/api/v1/admin/tsdb/delete_series](https://docs.victoriametrics.com/url-examples/#apiv1admintsdbdelete_series)
call. The duration is limited via `-search.maxDeleteDuration` flag. Deleting too many time series may require big
call. The duration is limited via `-search.maxDeleteDuration` flag{{% available_from "#tip" %}}. Deleting too many time series may require big
amount of CPU and memory at `vmstorage` and this limit guards against unplanned resource usage spikes.
Also see [How to delete time series](#how-to-delete-time-series) section to
learn about different ways of deleting series.

View File

@@ -22,5 +22,5 @@ to [the latest available releases](https://docs.victoriametrics.com/changelog/).
## Currently supported LTS release lines
- v1.102.x - the latest one is [v1.102.10 LTS release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.102.10)
- v1.97.x - the latest one is [v1.97.15 LTS release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.15)
- v1.102.x - the latest one is [v1.102.11 LTS release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.102.11)
- v1.97.x - the latest one is [v1.97.16 LTS release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.16)

View File

@@ -55,8 +55,8 @@ under the current directory:
```sh
docker pull victoriametrics/victoria-metrics:v1.109.1
docker run -it --rm -v `pwd`/victoria-metrics-data:/victoria-metrics-data -p 8428:8428 victoriametrics/victoria-metrics:v1.109.1
docker pull victoriametrics/victoria-metrics:v1.110.0
docker run -it --rm -v `pwd`/victoria-metrics-data:/victoria-metrics-data -p 8428:8428 victoriametrics/victoria-metrics:v1.110.0
```

View File

@@ -1769,7 +1769,7 @@ By default, VictoriaMetrics is tuned for an optimal resource usage under typical
- `-search.maxDeleteSeries` limits the number of unique time series that can be
deleted by a single
[/api/v1/admin/tsdb/delete_series](https://docs.victoriametrics.com/url-examples/#apiv1admintsdbdelete_series)
call. The duration is limited via `-search.maxDeleteDuration` flag. Deleting too many time series may require big
call. The duration is limited via `-search.maxDeleteDuration` flag{{% available_from "#tip" %}}. Deleting too many time series may require big
amount of CPU and memory and this limit guards against unplanned resource usage spikes. Also see
[How to delete time series](#how-to-delete-time-series) section to learn about
different ways of deleting series.

View File

@@ -16,6 +16,27 @@ according to [these docs](https://docs.victoriametrics.com/victorialogs/quicksta
## tip
* FEATURE: [`block_stats` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#block_stats-pipe): return the path to the part where every data block is stored. The path to the part is returned in the `part_path` field. This allows investigating the distribution of data blocks among parts.
* FEATURE: reduce VictoriaLogs startup time by multiple times when it opens a large datastore with big [retention](https://docs.victoriametrics.com/victorialogs/#retention).
## [v1.8.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.8.0-victorialogs)
Released at 2025-01-24
* FEATURE: [LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/): add an ability to limit query concurrency for the `<q>` [query](https://docs.victoriametrics.com/victorialogs/logsql/#query-syntax) via `options(concurrency=N) <q>` syntax. This may be needed for reducing RAM and CPU usage at the cost of longer query execution times. See [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#query-options) for details.
* FEATURE: [LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/): allow overriding the global time range filter at subqueries inside [`in` filter](https://docs.victoriametrics.com/victorialogs/logsql/#multi-exact-filter), [`join` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#join-pipe) and [`union` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#union-pipe) via `options(ignore_global_time_filter=true) <q>` syntax. See [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#query-options) for details.
* FEATURE: add [`hash` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#hash-pipe) for calculating hashes over the selected log fields. This may be useful for splitting the selected logs into distinct buckets. For example, the following query splits `user_id` fields into 4 buckets with the help of `hash` pipe: `_time:5m | hash(user_id) as h | math h%4 as bucket | stats by (bucket) count()`.
* FEATURE: [web UI](https://docs.victoriametrics.com/victorialogs/querying/#web-ui): reflect column settings for the table view in URL, so the table view can be shared via link. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7662).
* FEATURE: [web UI](https://docs.victoriametrics.com/victorialogs/querying/#web-ui): added context menu for legend items with options to copy and filter streams and fields. See this [PR](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7750).
* FEATURE: [web UI](https://docs.victoriametrics.com/victorialogs/querying/#web-ui): improved legend functionality with consistent click behavior across all vmui charts. See this [PR](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7750).
* FEATURE: [web UI](https://docs.victoriametrics.com/victorialogs/querying/#web-ui): added sorting for group view by record count in descending order. See this [PR](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7750).
* BUGFIX: [web UI](https://docs.victoriametrics.com/victorialogs/querying/#web-ui): removed the ability to filter by other in the legend, as other represents an aggregated series of all streams not included in the top results. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7552).
* BUGFIX: [LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/): properly apply optimizations to all the subqueries. Previously only the top-level query was optimized, while subqueries inside [`in` filter](https://docs.victoriametrics.com/victorialogs/logsql/#multi-exact-filter), [`join` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#join-pipe) and [`union` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#union-pipe) weren't optimized.
* BUGFIX: [HTTP querying API](https://docs.victoriametrics.com/victorialogs/querying/#http-api): properly apply [extra filters](https://docs.victoriametrics.com/victorialogs/querying/#extra-filters) to all the subqueries. Previously extra filters were applied only to the top-level query and weren't applied to sub-queries inside [`in` filter](https://docs.victoriametrics.com/victorialogs/logsql/#multi-exact-filter), [`join` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#join-pipe) and [`union` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#union-pipe).
* BUGFIX: [HTTP querying API](https://docs.victoriametrics.com/victorialogs/querying/#http-api): properly apply time range filter from `start` and `end` args to all the subqueries. Previously the time range filters were applied only to the top-level query and weren't applied to sub-queries inside [`in` filter](https://docs.victoriametrics.com/victorialogs/logsql/#multi-exact-filter), [`join` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#join-pipe) and [`union` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#union-pipe). The global time range filter can be ignored on a per-subquery basis via the `ignore_global_time_filter=true` option - see [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#query-options) for details.
## [v1.7.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.7.0-victorialogs)
Released at 2025-01-20
@@ -26,6 +47,8 @@ Released at 2025-01-20
* FEATURE: [web UI](https://docs.victoriametrics.com/victorialogs/querying/#web-ui): add the ability to select fields for display instead of the `_msg` field. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7419).
* FEATURE: [web UI](https://docs.victoriametrics.com/victorialogs/querying/#web-ui): add various display configuration settings for the grouped view. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7815)
* BUGFIX: [web UI](https://docs.victoriametrics.com/victorialogs/querying/#web-ui): fix an issue where pressing the "Enter" key in the query editor did not execute the query. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8058).
## [v1.6.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.6.1-victorialogs)
Released at 2025-01-16

View File

@@ -99,11 +99,13 @@ VictoriaLogs is designed solely for logs. VictoriaLogs uses [similar design idea
ClickHouse needs an intermediate applications for converting the ingested logs into `INSERT` SQL statements for the particular database schema.
This may increase the complexity of the system and, subsequently, increase its' maintenance costs.
- VictoriaLogs provides [built-in Web UI](https://docs.victoriametrics.com/victorialogs/querying/#web-ui) for logs' exploration.
## How does VictoriaLogs work?
VictoriaLogs accepts logs as [JSON entries](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model).
Then it stores log fields into a distinct data block. E.g. values for the same log field across multiple log entries
Then it stores log fields into distinct data blocks. E.g. values for the same log field across multiple log entries
are stored in a single data block. This allows reading data blocks only for the needed fields during querying.
Data blocks are compressed before being saved to persistent storage. This allows saving disk space and improving query performance
@@ -123,7 +125,7 @@ On top of this, VictoriaLogs employs additional optimizations for achieving high
- It uses [bloom filters](https://en.wikipedia.org/wiki/Bloom_filter) for skipping blocks without the given
[word](https://docs.victoriametrics.com/victorialogs/logsql/#word-filter) or [phrase](https://docs.victoriametrics.com/victorialogs/logsql/#phrase-filter).
- It uses custom encoding and compression for fields with different data types.
For example, it encodes IP addresses int 4 bytes. Custom fields' encoding reduces data size on disk and improves query performance.
For example, it encodes IP addresses into 4 bytes. Custom fields' encoding reduces data size on disk and improves query performance.
- It physically groups logs for the same [log stream](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields)
close to each other in the storage. This improves compression ratio, which helps reducing disk space usage. This also improves query performance
by skipping blocks for unneeded streams when [stream filter](https://docs.victoriametrics.com/victorialogs/logsql/#stream-filter) is used.

View File

@@ -9,7 +9,8 @@ aliases:
- /VictoriaLogs/LogsQL.html
---
LogsQL is a simple yet powerful query language for [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/).
See [examples](https://docs.victoriametrics.com/victorialogs/logsql-examples/) and [tutorial](#logsql-tutorial)
See [examples](https://docs.victoriametrics.com/victorialogs/logsql-examples/), [LogsQL tutorial](#logsql-tutorial)
and [SQL to LogsQL conversion guide](https://docs.victoriametrics.com/victorialogs/sql-to-logsql/)
in order to feel the language.
LogsQL provides the following features:
@@ -1329,8 +1330,9 @@ LogsQL supports the following pipes:
- [`first`](#first-pipe) returns the first N logs after sorting them by the given [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model).
- [`format`](#format-pipe) formats output field from input [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model).
- [`join`](#join-pipe) joins query results by the given [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model).
- [`hash`](#hash-pipe) returns the hash over the given [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) value.
- [`last`](#last-pipe) returns the last N logs after sorting them by the given [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model).
- [`len`](#len-pipe) calculates byte length of the given [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) value.
- [`len`](#len-pipe) returns byte length of the given [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) value.
- [`limit`](#limit-pipe) limits the number selected logs.
- [`math`](#math-pipe) performs mathematical calculations over [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model).
- [`offset`](#offset-pipe) skips the given number of selected logs.
@@ -1362,6 +1364,7 @@ LogsQL supports the following pipes:
- `bloom_bytes` - on-disk size of bloom filter data for the given `field`
- `dict_bytes` - on-disk size of the dictionary data for the given `field`
- `dict_items` - the number of unique values in the dictionary for the given `field`
- `part_path` - the path to the data part where the block is stored
The `block_stats` pipe is needed mostly for debugging purposes.
@@ -2037,6 +2040,22 @@ See also:
- [conditional `stats`](https://docs.victoriametrics.com/victorialogs/logsql/#stats-with-additional-filters)
- [`filter` pipe](#filter-pipe)
### hash pipe
`<q> | hash(field) as result_field` calculates hash value for the given [`field`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model)
and stores it into the `result_field`, for every log entry returned by `<q>` [query](#query-syntax).
For example, the following query calculates the hash value over `user_id` field and stores it into `user_id_hash` field, across logs for the last 5 minutes:
```logsql
_time:5m | hash(user_id) as user_id_hash
```
See also:
- [`math` pipe](#math-pipe)
- [`filter` pipe](#filter-pipe)
### last pipe
`<q> | last N by (fields)` [pipe](#pipes) returns the last `N` logs from `<q>` [query](#query-syntax) after sorting them
@@ -3134,6 +3153,9 @@ For example, the following query unrolls `timestamp` and `value` [log fields](ht
_time:5m | unroll (timestamp, value)
```
If the unrolled JSON array contains JSON objects, then it may be handy using [`unpack_json`](#unpack_json-pipe) for unpacking
the unrolled array items into separate fields for further processing.
See also:
- [`unpack_json` pipe](#unpack_json-pipe)
@@ -3323,9 +3345,31 @@ If the field contains [duration value](#duration-values), then `histogram` norma
If the field contains [short numeric value](#short-numeric-values), then `histogram` normalizes it to numeric value without any suffixes. For example, `1KiB` is converted to `1024`.
Histogram buckets are returned as the following JSON array:
```json
[{"vmrange":"...","hits":...},...,{"vmrange":"...","hits":...}]
```
Every `vmrange` value contains value range for the corresponding [VictoriaMetrics histogram bucket](https://valyala.medium.com/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350),
while `hits` contains the number of values, which hit the given bucket.
It may be handy to unroll the returned histogram buckets for further processing during the query. For example, the following query
calculates a histogram over the `response_size` [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model)
and then unrolls it into distinct rows with `vmrange` and `hits` fields with the help of [`unroll`](#unroll-pipe) and [`unpack_json`](#unpack_json-pipe) pipes:
```logsql
_time:5m
| stats histogram(response_size) as buckets
| unroll (buckets)
| unpack_json from buckets
```
See also:
- [`quantile`](#quantile-stats)
- [`unroll` pipe](#unroll-pipe)
- [`unpack_json` pipe](#unpack_json-pipe)
### max stats
@@ -3724,9 +3768,43 @@ Internally duration values are converted into nanoseconds.
- It is highly recommended specifying [time filter](#time-filter) in order to narrow down the search to specific time range.
- It is highly recommended specifying [stream filter](#stream-filter) in order to narrow down the search
to specific [log streams](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields).
- It is recommended specifying [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) you need in query results
with the [`field` pipe](#fields-pipe), if the selected log entries contain big number of fields, which aren't interesting to you.
This saves disk read IO and CPU time needed for reading and unpacking all the log fields from disk.
- Move faster filters such as [word filter](#word-filter) and [phrase filter](#phrase-filter) to the beginning of the query.
This rule doesn't apply to [time filter](#time-filter) and [stream filter](#stream-filter), which can be put at any place of the query.
- Move more specific filters, which match lower number of log entries, to the beginning of the query.
This rule doesn't apply to [time filter](#time-filter) and [stream filter](#stream-filter), which can be put at any place of the query.
- If the selected logs are passed to [pipes](#pipes) for further transformations and statistics' calculations, then it is recommended
reducing the number of selected logs by using more specific [filters](#filters), which return lower number of logs to process by [pipes](#pipes).
## Query options
VictoriaLogs supports the following options, which can be passed in the beginning of [LogsQL query](#query-syntax) `<q>` via `options(opt1=v1, ..., optN=vN) <q>` syntax:
- `concurrency` - query concurrency. By default the query is executed in parallel on all the available CPU cores.
This usually provides the best query performance. Sometimes it is needed to reduce the number of used CPU cores,
in order to reduce RAM usage and/or CPU usage.
This can be done by setting `concurrency` option to the value smaller than the number of available CPU cores.
For example, the following query executes on at max 2 CPU cores:
```logsql
options(concurrency=2) _time:1d | count_uniq(user_id)
```
- `ignore_global_time_filter` - allows ignoring time filter from `start` and `end` args of [HTTP querying API](https://docs.victoriametrics.com/victorialogs/querying/#http-api)
for the given (sub)query. For example, the following query returns the number of logs with `user_id` values seen in logs during December 2024, on the `[start...end]`
time range passed to [`/api/v1/query`](https://docs.victoriametrics.com/victorialogs/querying/#querying-logs):
```logsql
user_id:in(options(ignore_global_time_filter=true) _time:2024-12Z | keep user_id) | count()
```
The [`in`](https://docs.victoriametrics.com/victorialogs/logsql/#multi-exact-filter) query without `options(ignore_global_time_filter=true)`
takes into account only `user_id` values on the intersection of December 2024 and `[start...end]` time range pased
to [`/api/v1/query`](https://docs.victoriametrics.com/victorialogs/querying/#querying-logs):
```logsql
user_id:in(_time:2024-12Z | keep user_id) | count()
```

View File

@@ -33,8 +33,8 @@ Just download archive for the needed Operating system and architecture, unpack i
For example, the following commands download VictoriaLogs archive for Linux/amd64, unpack and run it:
```sh
curl -L -O https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.7.0-victorialogs/victoria-logs-linux-amd64-v1.7.0-victorialogs.tar.gz
tar xzf victoria-logs-linux-amd64-v1.7.0-victorialogs.tar.gz
curl -L -O https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.8.0-victorialogs/victoria-logs-linux-amd64-v1.8.0-victorialogs.tar.gz
tar xzf victoria-logs-linux-amd64-v1.8.0-victorialogs.tar.gz
./victoria-logs-prod
```
@@ -58,7 +58,7 @@ Here is the command to run VictoriaLogs in a Docker container:
```sh
docker run --rm -it -p 9428:9428 -v ./victoria-logs-data:/victoria-logs-data \
docker.io/victoriametrics/victoria-logs:v1.7.0-victorialogs
docker.io/victoriametrics/victoria-logs:v1.8.0-victorialogs
```
See also:

View File

@@ -4,33 +4,37 @@ from [VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics/).
VictoriaLogs provides the following features:
- It can accept logs from popular log collectors. See [these docs](https://docs.victoriametrics.com/victorialogs/data-ingestion/).
- It is much easier to set up and operate compared to Elasticsearch and Grafana Loki.
- It is much easier to set up and operate compared to Elasticsearch and Grafana Loki, since it is basically zero-config.
See [these docs](https://docs.victoriametrics.com/victorialogs/quickstart/).
- It provides easy yet powerful query language with full-text search across
- It provides easy yet powerful query language with full-text search capabilities across
all the [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model).
See [LogsQL docs](https://docs.victoriametrics.com/victorialogs/logsql/).
- It provides interactive command-line tool for querying VictoriaLogs - [vlogscli](https://docs.victoriametrics.com/victorialogs/querying/vlogscli/).
- It provides [built-in web UI](https://docs.victoriametrics.com/victorialogs/querying/#web-ui) for logs' exploration.
- Ir provides [Grafana plugin](https://docs.victoriametrics.com/victorialogs/victorialogs-datasource/) for building arbitrary dashboards in Grafana.
- It provides [interactive command-line tool for querying VictoriaLogs](https://docs.victoriametrics.com/victorialogs/querying/vlogscli/).
- It can be seamlessly combined with good old Unix tools for log analysis such as `grep`, `less`, `sort`, `jq`, etc.
See [these docs](https://docs.victoriametrics.com/victorialogs/querying/#command-line) for details.
- VictoriaLogs' capacity and performance scales linearly with the available resources (CPU, RAM, disk IO, disk space).
It runs smoothly on Raspberry PI and on servers with hundreds of CPU cores and terabytes of RAM.
- It can handle up to 30x bigger data volumes than Elasticsearch and Grafana Loki when running on the same hardware.
See [these docs](#benchmarks) and [this article](https://itnext.io/how-do-open-source-solutions-for-logs-work-elasticsearch-loki-and-victorialogs-9f7097ecbc2f) for details.
- It provides fast full-text search out of the box for [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model)
with high cardinality (e.g. high number of unique values) such as `trace_id`, `user_id` and `ip`.
- It support [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) with high cardinality (e.g. high number of unique values) such as `trace_id`, `user_id` and `ip`.
- It is optimized for logs with hundreds of fields (aka [`wide events`](https://jeremymorrell.dev/blog/a-practitioners-guide-to-wide-events/)).
- It supports multitenancy - see [these docs](#multitenancy).
- It supports out-of-order logs' ingestion aka backfilling.
- It supports live tailing for newly ingested logs. See [these docs](https://docs.victoriametrics.com/victorialogs/querying/#live-tailing).
- It supports selecting surrounding logs in front and after the selected logs. See [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#stream_context-pipe).
- It provides web UI for querying logs - see [these docs](https://docs.victoriametrics.com/victorialogs/querying/#web-ui).
- It provides [Grafana plugin for querying logs](https://docs.victoriametrics.com/victorialogs/victorialogs-datasource/).
- It supports alerting - see [these docs](https://docs.victoriametrics.com/victorialogs/vmalert/).
You can play with VictoriaLogs web UI at [this playground](https://play-vmlogs.victoriametrics.com/).
If you have questions about VictoriaLogs, then read [this FAQ](https://docs.victoriametrics.com/victorialogs/faq/).
Also feel free asking any questions at [VictoriaMetrics community Slack chat](https://victoriametrics.slack.com/),
you can join it via [Slack Inviter](https://slack.victoriametrics.com/).
See [Quick start docs](https://docs.victoriametrics.com/victorialogs/quickstart/) for start working with VictoriaLogs.
See [quick start docs](https://docs.victoriametrics.com/victorialogs/quickstart/) for start working with VictoriaLogs.
If you want playing with [LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/) query language, then go to [VictoriaLogs playground](https://play-vmlogs.victoriametrics.com/).
## Tuning
@@ -245,6 +249,37 @@ VictoriaLogs has very low overhead for per-tenant management, so it is OK to hav
VictoriaLogs doesn't perform per-tenant authorization. Use [vmauth](https://docs.victoriametrics.com/vmauth/) or similar tools for per-tenant authorization.
### Multitenancy access control
Enforce access control for tenants by using [vmauth](https://docs.victoriametrics.com/vmauth/). Access control can be configured for each tenant by setting up the following rules:
```yaml
users:
- username: "foo"
password: "bar"
url_map:
- src_paths:
- "/select/.*"
- "/insert/.*"
headers:
- "AccountID: 1"
- "ProjectID: 0"
url_prefix:
- "http://localhost:9428/"
- username: "baz"
password: "bar"
url_map:
- src_paths: ["/select/.*"]
headers:
- "AccountID: 2"
- "ProjectID: 0"
url_prefix:
- "http://localhost:9428/"
```
This configuration allows `foo` to use the `/select/.*` and `/insert/.*` endpoints with `AccountID: 1` and `ProjectID: 0`, while `baz` can only use the `/select/.*` endpoint with `AccountID: 2` and `ProjectID: 0`.
## Security
It is expected that VictoriaLogs runs in a protected environment, which is unreachable from the Internet without proper authorization.
@@ -254,6 +289,8 @@ or similar authorization proxies.
## Benchmarks
See [the comparison of VictoriaLogs with Elasticsearch, MongoDB, TimescaleDB, PostgreSQL, MySQL and SQLite](https://benchmark.clickhouse.com/#eyJzeXN0ZW0iOnsiQWxsb3lEQiI6ZmFsc2UsIkFsbG95REIgKHR1bmVkKSI6ZmFsc2UsIkF0aGVuYSAocGFydGl0aW9uZWQpIjpmYWxzZSwiQXRoZW5hIChzaW5nbGUpIjpmYWxzZSwiQXVyb3JhIGZvciBNeVNRTCI6ZmFsc2UsIkF1cm9yYSBmb3IgUG9zdGdyZVNRTCI6ZmFsc2UsIkJ5Q29uaXR5IjpmYWxzZSwiQnl0ZUhvdXNlIjpmYWxzZSwiY2hEQiAoRGF0YUZyYW1lKSI6ZmFsc2UsImNoREIgKFBhcnF1ZXQsIHBhcnRpdGlvbmVkKSI6ZmFsc2UsImNoREIiOmZhbHNlLCJDaXR1cyI6ZmFsc2UsIkNsaWNrSG91c2UgQ2xvdWQgKGF3cykiOmZhbHNlLCJDbGlja0hvdXNlIENsb3VkIChhenVyZSkiOmZhbHNlLCJDbGlja0hvdXNlIENsb3VkIChnY3ApIjpmYWxzZSwiQ2xpY2tIb3VzZSAoZGF0YSBsYWtlLCBwYXJ0aXRpb25lZCkiOmZhbHNlLCJDbGlja0hvdXNlIChkYXRhIGxha2UsIHNpbmdsZSkiOmZhbHNlLCJDbGlja0hvdXNlIChQYXJxdWV0LCBwYXJ0aXRpb25lZCkiOmZhbHNlLCJDbGlja0hvdXNlIChQYXJxdWV0LCBzaW5nbGUpIjpmYWxzZSwiQ2xpY2tIb3VzZSAod2ViKSI6ZmFsc2UsIkNsaWNrSG91c2UiOmZhbHNlLCJDbGlja0hvdXNlICh0dW5lZCkiOmZhbHNlLCJDbGlja0hvdXNlICh0dW5lZCwgbWVtb3J5KSI6ZmFsc2UsIkNsb3VkYmVycnkiOmZhbHNlLCJDcmF0ZURCIjpmYWxzZSwiQ3J1bmNoeSBCcmlkZ2UgZm9yIEFuYWx5dGljcyAoUGFycXVldCkiOmZhbHNlLCJEYXRhYmVuZCI6ZmFsc2UsIkRhdGFGdXNpb24gKFBhcnF1ZXQsIHBhcnRpdGlvbmVkKSI6ZmFsc2UsIkRhdGFGdXNpb24gKFBhcnF1ZXQsIHNpbmdsZSkiOmZhbHNlLCJBcGFjaGUgRG9yaXMiOmZhbHNlLCJEcmlsbCI6ZmFsc2UsIkRydWlkIjpmYWxzZSwiRHVja0RCIChEYXRhRnJhbWUpIjpmYWxzZSwiRHVja0RCIChtZW1vcnkpIjpmYWxzZSwiRHVja0RCIChQYXJxdWV0LCBwYXJ0aXRpb25lZCkiOmZhbHNlLCJEdWNrREIiOmZhbHNlLCJFbGFzdGljc2VhcmNoIjp0cnVlLCJFbGFzdGljc2VhcmNoICh0dW5lZCkiOmZhbHNlLCJHbGFyZURCIjpmYWxzZSwiR3JlZW5wbHVtIjpmYWxzZSwiSGVhdnlBSSI6ZmFsc2UsIkh5ZHJhIjpmYWxzZSwiSW5mb2JyaWdodCI6ZmFsc2UsIktpbmV0aWNhIjpmYWxzZSwiTWFyaWFEQiBDb2x1bW5TdG9yZSI6ZmFsc2UsIk1hcmlhREIiOmZhbHNlLCJNb25ldERCIjpmYWxzZSwiTW9uZ29EQiI6dHJ1ZSwiTW90aGVyRHVjayI6ZmFsc2UsIk15U1FMIChNeUlTQU0pIjpmYWxzZSwiTXlTUUwiOnRydWUsIk9jdG9TUUwiOmZhbHNlLCJPeGxhIjpmYWxzZSwiUGFuZGFzIChEYXRhRnJhbWUpIjpmYWxzZSwiUGFyYWRlREIgKFBhcnF1ZXQsIHBhcnRpdGlvbmVkKSI6ZmFsc2UsIlBhcmFkZURCIChQYXJxdWV0LCBzaW5nbGUpIjpmYWxzZSwicGdfZHVja2RiIChNb3RoZXJEdWNrIGVuYWJsZWQpIjpmYWxzZSwicGdfZHVja2RiIjpmYWxzZSwiUGlub3QiOmZhbHNlLCJQb2xhcnMgKERhdGFGcmFtZSkiOmZhbHNlLCJQb2xhcnMgKFBhcnF1ZXQpIjpmYWxzZSwiUG9zdGdyZVNRTCAodHVuZWQpIjpmYWxzZSwiUG9zdGdyZVNRTCI6dHJ1ZSwiUXVlc3REQiI6ZmFsc2UsIlJlZHNoaWZ0IjpmYWxzZSwiU2VsZWN0REIiOmZhbHNlLCJTaW5nbGVTdG9yZSI6ZmFsc2UsIlNub3dmbGFrZSI6ZmFsc2UsIlNwYXJrIjpmYWxzZSwiU1FMaXRlIjp0cnVlLCJTdGFyUm9ja3MiOmZhbHNlLCJUYWJsZXNwYWNlIjpmYWxzZSwiVGVtYm8gT0xBUCAoY29sdW1uYXIpIjpmYWxzZSwiVGltZXNjYWxlIENsb3VkIjpmYWxzZSwiVGltZXNjYWxlREIgKG5vIGNvbHVtbnN0b3JlKSI6ZmFsc2UsIlRpbWVzY2FsZURCIjp0cnVlLCJUaW55YmlyZCAoRnJlZSBUcmlhbCkiOmZhbHNlLCJVbWJyYSI6ZmFsc2UsIlZpY3RvcmlhTG9ncyI6dHJ1ZX0sInR5cGUiOnsiQyI6dHJ1ZSwiY29sdW1uLW9yaWVudGVkIjp0cnVlLCJQb3N0Z3JlU1FMIGNvbXBhdGlibGUiOnRydWUsIm1hbmFnZWQiOnRydWUsImdjcCI6dHJ1ZSwic3RhdGVsZXNzIjp0cnVlLCJKYXZhIjp0cnVlLCJDKysiOnRydWUsIk15U1FMIGNvbXBhdGlibGUiOnRydWUsInJvdy1vcmllbnRlZCI6dHJ1ZSwiQ2xpY2tIb3VzZSBkZXJpdmF0aXZlIjp0cnVlLCJlbWJlZGRlZCI6dHJ1ZSwic2VydmVybGVzcyI6dHJ1ZSwiZGF0YWZyYW1lIjp0cnVlLCJhd3MiOnRydWUsImF6dXJlIjp0cnVlLCJhbmFseXRpY2FsIjp0cnVlLCJSdXN0Ijp0cnVlLCJzZWFyY2giOnRydWUsImRvY3VtZW50Ijp0cnVlLCJHbyI6dHJ1ZSwic29tZXdoYXQgUG9zdGdyZVNRTCBjb21wYXRpYmxlIjp0cnVlLCJEYXRhRnJhbWUiOnRydWUsInBhcnF1ZXQiOnRydWUsInRpbWUtc2VyaWVzIjp0cnVlfSwibWFjaGluZSI6eyIxNiB2Q1BVIDEyOEdCIjpmYWxzZSwiOCB2Q1BVIDY0R0IiOmZhbHNlLCJzZXJ2ZXJsZXNzIjpmYWxzZSwiMTZhY3UiOmZhbHNlLCJjNmEuNHhsYXJnZSwgNTAwZ2IgZ3AyIjp0cnVlLCJMIjpmYWxzZSwiTSI6ZmFsc2UsIlMiOmZhbHNlLCJYUyI6ZmFsc2UsImM2YS5tZXRhbCwgNTAwZ2IgZ3AyIjpmYWxzZSwiMTkyR0IiOmZhbHNlLCIyNEdCIjpmYWxzZSwiMzYwR0IiOmZhbHNlLCI0OEdCIjpmYWxzZSwiNzIwR0IiOmZhbHNlLCI5NkdCIjpmYWxzZSwiZGV2IjpmYWxzZSwiNzA4R0IiOmZhbHNlLCJjNW4uNHhsYXJnZSwgNTAwZ2IgZ3AyIjpmYWxzZSwiQW5hbHl0aWNzLTI1NkdCICg2NCB2Q29yZXMsIDI1NiBHQikiOmZhbHNlLCJjNS40eGxhcmdlLCA1MDBnYiBncDIiOmZhbHNlLCJjNmEuNHhsYXJnZSwgMTUwMGdiIGdwMiI6dHJ1ZSwiY2xvdWQiOmZhbHNlLCJkYzIuOHhsYXJnZSI6ZmFsc2UsInJhMy4xNnhsYXJnZSI6ZmFsc2UsInJhMy40eGxhcmdlIjpmYWxzZSwicmEzLnhscGx1cyI6ZmFsc2UsIlMyIjpmYWxzZSwiUzI0IjpmYWxzZSwiMlhMIjpmYWxzZSwiM1hMIjpmYWxzZSwiNFhMIjpmYWxzZSwiWEwiOmZhbHNlLCJMMSAtIDE2Q1BVIDMyR0IiOmZhbHNlLCJjNmEuNHhsYXJnZSwgNTAwZ2IgZ3AzIjpmYWxzZSwiMTYgdkNQVSA2NEdCIjpmYWxzZSwiNCB2Q1BVIDE2R0IiOmZhbHNlLCI4IHZDUFUgMzJHQiI6ZmFsc2V9LCJjbHVzdGVyX3NpemUiOnsiMSI6dHJ1ZSwiMiI6ZmFsc2UsIjQiOmZhbHNlLCI4IjpmYWxzZSwiMTYiOmZhbHNlLCIzMiI6ZmFsc2UsIjY0IjpmYWxzZSwiMTI4IjpmYWxzZSwic2VydmVybGVzcyI6ZmFsc2UsInVuZGVmaW5lZCI6ZmFsc2V9LCJtZXRyaWMiOiJob3QiLCJxdWVyaWVzIjpbdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZV19).
Here is a [benchmark suite](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/logs-benchmark) for comparing data ingestion performance
and resource usage between VictoriaLogs and Elasticsearch or Loki.

View File

@@ -942,7 +942,7 @@ The arg passed to `extra_filters` and `extra_stream_filters` must be properly en
## Web UI
VictoriaLogs provides Web UI for logs [querying](https://docs.victoriametrics.com/victorialogs/logsql/) and exploration
at `http://localhost:9428/select/vmui`.
at `http://localhost:9428/select/vmui`. Try [VictoriaLogs web UI playground](https://play-vmlogs.victoriametrics.com/).
There are three modes of displaying query results:

View File

@@ -23,8 +23,8 @@ or from [docker images](https://hub.docker.com/r/victoriametrics/vlogscli/tags).
### Running `vlogscli` from release binary
```sh
curl -L -O https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.7.0-victorialogs/vlogscli-linux-amd64-v1.7.0-victorialogs.tar.gz
tar xzf vlogscli-linux-amd64-v1.7.0-victorialogs.tar.gz
curl -L -O https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.8.0-victorialogs/vlogscli-linux-amd64-v1.8.0-victorialogs.tar.gz
tar xzf vlogscli-linux-amd64-v1.8.0-victorialogs.tar.gz
./vlogscli-prod
```

View File

@@ -0,0 +1,116 @@
---
weight: 120
title: SQL to LogsQL tutorial
menu:
docs:
parent: "victorialogs"
weight: 120
---
This is a tutorial for the migration from SQL to [LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/).
It is expected you are familiar with SQL and know [how to execute queries at VictoriaLogs](https://docs.victoriametrics.com/victorialogs/querying/).
## data model
SQL is usually used for querying relational tables. Every such table contains a pre-defined set of columns with pre-defined types.
LogsQL is used for querying logs. Logs are stored in [log streams](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields).
So log streams is an analogue of tables in relational databases. Log streams and relational tables have the following major differences:
- Log streams are created automatically when the first log entry (row) is ingested into them.
- There is no pre-defined scheme in log streams - logs with arbitrary set of fields can be ingested into every log stream.
Both names and values in every log entry have string type. They may contain arbitrary string data.
- Every log entry (row) can be represented as a flat JSON object: `{"f1":"v1",...,"fN":"vN"}`. See [these docs](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model).
- By default VictoriaLogs selects log entries across all the log streams. The needed set of log streams can be specified
via [stream filters](https://docs.victoriametrics.com/victorialogs/logsql/#stream-filter).
- By default VictoriaLogs returns all the fields across the selected logs. The set of returned fields
can be limited with [`fields` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#fields-pipe).
## query structure
SQL query structure is quite convoluted:
```sql
SELECT
<fields, aggregations, calculations, transformations>
FROM <table>
<optional JOINs>
<optional filters with optional subqueries>
<optional GROUP BY>
<optional HAVING>
<optional ORDER BY>
<optional LIMIT / OFFSET>
<optional UNION>
```
[LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/) query structure is much simpler:
```logsql
<filters>
| <optional_pipe1>
| ...
| <optional_pipeN>
```
The `<filters>` part selects the needed logs (rows) according to the provided [filters](https://docs.victoriametrics.com/victorialogs/logsql/#filters).
Then the provided [pipes](https://docs.victoriametrics.com/victorialogs/logsql/#pipes) are executed sequentlially.
Every such pipe receives all the rows from the previous stage, performs some calculations and/or transformations,
and then pushes the resulting rows to the next stage. This simplifies reading and understanding the query - just read it from the beginning
to the end in order to understand what does it do at every stage.
LogsQL pipes cover all the functionality from SQL: aggregations, calculations, transformations, subqueries, joins, post-filters, sorting, etc.
See the [conversion rules](#conversion-rules) on how to convert SQL to LogsQL.
## conversion rules
The following rules must be used for converting SQL query into LogsQL query:
* If the SQL query contains `WHERE`, then convert it into [LogsQL filters](https://docs.victoriametrics.com/victorialogs/logsql/#filters).
Otherwise just start LogsQL query with [`*`](https://docs.victoriametrics.com/victorialogs/logsql/#any-value-filter).
For example, `SELECT * FROM table WHERE field1=value1 AND field2<>value2` is converted into `field1:=value1 field2:!=value2`,
while `SELECT * FROM table` is converted into `*`.
* `IN` subqueries inside `WHERE` must be converted into [`in` filters](https://docs.victoriametrics.com/victorialogs/logsql/#multi-exact-filter).
For example, `SELECT * FROM table WHERE id IN (SELECT id2 FROM table)` is converted into `id:in(* | fields id2)`.
* If the `SELECT` part isn't equal to `*` and there are no `GROUP BY` / aggregate functions in the SQL query, then enumerate
the selected columns at [`fields` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#fields-pipe).
For example, `SELECT field1, field2 FROM table` is converted into `* | fields field1, field2`.
* If the SQL query contains `JOIN`, then convert it into [`join` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#join-pipe).
* If the SQL query contains `GROUP BY` / aggregate functions, then convert them to [`stats` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#stats-pipe).
For example, `SELECT count(*) FROM table` is converted into `* | count()`, while `SELECT user_id, count(*) FROM table GROUP BY user_id`
is converted to `* | stats by (user_id) count()`. Note how the LogsQL query mentions the `GROUP BY` fields only once,
while SQL forces mentioning these fields twice - at the `SELECT` and at the `GROUP BY`. How many times did you hit the discrepancy
between `SELECT` and `GROUP BY` fields?
* If the SQL query contains additional calculations and/or transformations at the `SELECT`, which aren't covered yet by `GROUP BY`,
then convert them into the corresponding [LogsQL pipes](https://docs.victoriametrics.com/victorialogs/logsql/#pipes).
The most frequently used pipes are [`math`](https://docs.victoriametrics.com/victorialogs/logsql/#math-pipe)
and [`format`](https://docs.victoriametrics.com/victorialogs/logsql/#format-pipe).
For example, `SELECT field1 + 10 AS x, CONCAT("foo", field2) AS y FROM table` is converted into `* | math field1 + 10 as x | format "foo<field2>" as y | fields x, y`.
* If the SQL query contains `HAVING`, then convert it into [`filter` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#filter-pipe).
For example, `SELECT user_id, count(*) AS c FROM table GROUP BY user_id HAVING c > 100` is converted into `* | stats by (user_id) count() c | filter c:>100`.
* If the SQL query contains `ORDER BY`, `LIMIT` and `OFFSET`, then convert them into [`sort` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#sort-pipe).
For example, `SELECT * FROM table ORDER BY field1, field2 LIMIT 10 OFFSET 20` is converted into `* | sort by (field1, field2) limit 10 offset 20`.
* If the SQL query contains `UNION`, then convert it into [`union` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#union-pipe).
For example `SELECT * FROM table WHERE filters1 UNION ALL SELECT * FROM table WHERE filters2` is converted into `filters1 | union (filters2)`.
SQL queries are frequently used for obtaining top N column values, which are the most frequently seen in the selected rows.
For example, the query below returns top 5 `user_id` values, which present in the biggest number of rows:
```sql
SELECT user_id, count(*) hits FROM table GROUP BY user_id ORDER BY hits DESC LIMIT 5
```
LogsQL provides a shortcut syntax with [`top` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#top-pipe) for this case:
```logsql
* | top 5 (user_id)
```
It is equivalent to the longer LogsQL query:
```logsql
* | by (user_id) count() hits | sort by (hits desc) limit 5
```
[LogsQL pipes](https://docs.victoriametrics.com/victorialogs/logsql/#pipes) support much wider functionality comparing to SQL,
so spend your spare time by reading [pipe docs](https://docs.victoriametrics.com/victorialogs/logsql/) and playing with them
at [VictoriaLogs playground](https://play-vmlogs.victoriametrics.com/).

View File

@@ -11,14 +11,29 @@ aliases:
---
Please find the changelog for VictoriaMetrics Anomaly Detection below.
## v1.19.2
Released: 2025-01-27
- IMPROVEMENT: Added the `complete` option to the `--splitBy` argument in `config_splitter.py` [util](https://docs.victoriametrics.com/anomaly-detection/faq/index.html#splitting-the-config). This allows splitting a parent configuration into the smallest possible sub-configurations, each containing exactly one scheduler, one model, and either one or multiple queries (depending on whether the model is [multivariate](https://docs.victoriametrics.com/anomaly-detection/components/models/#multivariate-models) or not).
- FIX: Resolved an issue where duplicate log messages were generated during sub-config validation of the parent configuration.
- FIX: Corrected usage of `AccountID` and `ProjectID` extracted from `tenant_id`, which are appended as labels `vm_account_id` and `vm_project_id`, respectively (previously swapped) by `VmReader` when using the per-query `tenant_id` feature. **This issue affected versions [v1.19.0](#v1190) and [v1.19.1](#v1191).**
- FIX: Resolved an issue with the `VmReader` instance string representation that caused errors when `vmanomaly` was run with `--loggerLevel DEBUG`.
## v1.19.1
Released: 2025-01-21
> **Note**: There is a known bug in [v1.19.0](#v1190) - the `AccountID` and `ProjectID` are swapped when they are extracted from the `tenant_id` argument in `VMReader`. This can cause correctly read results being written to the wrong tenant when using the per-query `tenant_id` feature with `AccountID` != `ProjectID`. Please update to patch [v1.19.2](#v1192), which resolves this issue.
- FIX: Resolved writer warnings for configurations where `reader.tenant_id` equals `writer.tenant_id` and **is not** `multitenant`, as this is a valid setup. Enhanced tenant_id-related log messages across config validation, reader, and writer for improved clarity.
## v1.19.0
Released: 2025-01-20
> **Note**: There is a known bug in [v1.19.0](#v1190) - the `AccountID` and `ProjectID` are swapped when they are extracted from the `tenant_id` argument in `VMReader`. This can cause correctly read results being written to the wrong tenant when using the per-query `tenant_id` feature with `AccountID` != `ProjectID`. Please update to patch [v1.19.2](#v1192), which resolves this issue.
- FEATURE: Added support for per-query `tenant_id` in the [`VmReader`](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader). This allows overriding the reader-level `tenant_id` within a single global `vmanomaly` configuration on a *per-query* basis, enabling isolation of data for different tenants in separate queries when querying the [VictoriaMetrics cluster version](https://docs.victoriametrics.com/cluster-victoriametrics/). For details, see the [documentation](https://docs.victoriametrics.com/anomaly-detection/components/reader/?highlight=tenant_id#per-query-parameters).
- IMPROVEMEMT: Speedup the model infer stage on multicore systems.
- IMPROVEMEMT: Speedup the model fitting stage by 1.253x, depending on configuration complexity.

View File

@@ -158,7 +158,7 @@ services:
# ...
vmanomaly:
container_name: vmanomaly
image: victoriametrics/vmanomaly:v1.19.1
image: victoriametrics/vmanomaly:v1.19.2
# ...
ports:
- "8490:8490"
@@ -337,10 +337,10 @@ For **horizontal** scalability, `vmanomaly` can be deployed as multiple independ
### Splitting the config
CLI utility named `config_splitter` is available in `vmanomaly` {{% available_from "v1.18.5" anomaly %}}. The config splitter tool enables splitting a parent vmanomaly YAML configuration file into multiple sub-configurations based on logical entities such as `schedulers`, `queries`, `models`, `extra_filters`. The resulting sub-configurations are fully validated, functional, account for many-to-many relationships between models and their associated queries, and the schedulers they are linked to. These sub-configurations can then be saved to a specified directory for further use:
CLI utility named `config_splitter` is available in `vmanomaly` {{% available_from "v1.18.5" anomaly %}}. The config splitter tool enables splitting a parent vmanomaly YAML configuration file into multiple sub-configurations based on logical entities such as `schedulers`, `queries`, `models`, `extra_filters` and `complete` {{% available_from "v1.19.2" anomaly %}}. The resulting sub-configurations are fully validated, functional, account for many-to-many relationships between models and their associated queries, and the schedulers they are linked to. These sub-configurations can then be saved to a specified directory for further use:
```shellhelp
usage: config_splitter.py [-h] --splitBy {schedulers,models,queries,extra_filters} --outputDir OUTPUT_DIR [--fileNameFormat {raw,hash,int}] [--loggerLevel {WARNING,INFO,ERROR,FATAL,DEBUG}]
usage: config_splitter.py [-h] --splitBy {schedulers,models,queries,extra_filters,complete} --outputDir OUTPUT_DIR [--fileNameFormat {raw,hash,int}] [--loggerLevel {WARNING,ERROR,FATAL,INFO,DEBUG}]
config [config ...]
Splits the configuration of VictoriaMetrics Anomaly Detection service by a logical entity.
@@ -350,21 +350,22 @@ positional arguments:
options:
-h show this help message and exit
--splitBy {schedulers,models,queries,extra_filters}
The logical entity to split by. Choices: ['schedulers', 'models', 'queries', 'extra_filters'].
--splitBy {schedulers,models,queries,extra_filters,complete}
The logical entity to split by. Choices: ['schedulers', 'models', 'queries', 'extra_filters', 'complete']. `complete` produces configurations based on combinations of
(scheduler, model, queries). Default: complete.
--outputDir output_dir
Directory where the split configuration files will be saved.
--fileNameFormat {raw,hash,int}
The naming format for the output configuration files. Choices: raw (use the entity alias), hash (use hashed alias), int (use a sequential integer from 0 to N for N
produced sub-configs). Default: raw.
--loggerLevel {WARNING,INFO,ERROR,FATAL,DEBUG}
The naming format for the output configuration files. Choices: raw (use the entity alias), hash (use hashed alias), int (use a sequential integer from 0 to N for N produced
sub-configs). Default: raw.
--loggerLevel {WARNING,ERROR,FATAL,INFO,DEBUG}
Minimum level to log. Default: INFO
```
Heres an example of using the config splitter to divide configurations based on the `extra_filters` argument from the reader section:
```sh
docker pull victoriametrics/vmanomaly:v1.19.1 && docker image tag victoriametrics/vmanomaly:v1.19.1 vmanomaly
docker pull victoriametrics/vmanomaly:v1.19.2 && docker image tag victoriametrics/vmanomaly:v1.19.2 vmanomaly
```
```sh

View File

@@ -159,14 +159,15 @@ Total amount of CPU time consumed by the system in seconds by CPU processing mod
<tr>
<td>
`host_network_receive_errors` & `host_network_transmit_errors`
<span style="white-space: nowrap;">`host_network_transmit_errors`</span>
& `host_network_receive_errors`
</td>
<td>
`node_network_receive_errs_total`,
`node_network_receive_packets_total`,
`node_network_transmit_errs_total`,
`node_network_transmit_packets_total`
`node_network_transmit_errs_total`,
<span style="white-space: nowrap;">`node_network_transmit_packets_total`</span>
<td>
Total number of errors encountered while receiving/transmitting packets on the network interfaces of a node.

View File

@@ -101,13 +101,13 @@ Below are the steps to get `vmanomaly` up and running inside a Docker container:
1. Pull Docker image:
```sh
docker pull victoriametrics/vmanomaly:v1.19.0
docker pull victoriametrics/vmanomaly:v1.19.2
```
2. (Optional step) tag the `vmanomaly` Docker image:
```sh
docker image tag victoriametrics/vmanomaly:v1.19.0 vmanomaly
docker image tag victoriametrics/vmanomaly:v1.19.2 vmanomaly
```
3. Start the `vmanomaly` Docker container with a *license file*, use the command below.
@@ -141,7 +141,7 @@ docker run -it --user 1000:1000 \
services:
# ...
vmanomaly:
image: victoriametrics/vmanomaly:v1.19.0
image: victoriametrics/vmanomaly:v1.19.2
volumes:
$YOUR_LICENSE_FILE_PATH:/license
$YOUR_CONFIG_FILE_PATH:/config.yml

View File

@@ -994,7 +994,7 @@ monitoring:
Let's pull the docker image for `vmanomaly`:
```sh
docker pull victoriametrics/vmanomaly:v1.19.1
docker pull victoriametrics/vmanomaly:v1.19.2
```
Now we can run the docker container putting as volumes both config and model file:
@@ -1008,7 +1008,7 @@ docker run -it \
-v $(PWD)/license:/license \
-v $(PWD)/custom_model.py:/vmanomaly/model/custom.py \
-v $(PWD)/custom.yaml:/config.yaml \
victoriametrics/vmanomaly:v1.19.1 /config.yaml \
victoriametrics/vmanomaly:v1.19.2 /config.yaml \
--licenseFile=/license
```
@@ -1017,7 +1017,7 @@ Please find more detailed instructions (license, etc.) [here](https://docs.victo
### Output
As the result, this model will return metric with labels, configured previously in `config.yaml`.
In this particular example, 2 metrics will be produced. Also, there will be added other metrics from input query result.
In this particular example, 2 metrics will be produced. Also, there will be added other labels from input query result.
```text
{__name__="custom_anomaly_score", for="ingestion_rate", model_alias="custom_model", scheduler_alias="s1", run="test-format"},

View File

@@ -2,9 +2,9 @@
- To use *vmanomaly*, part of the enterprise package, a license key is required. Obtain your key [here](https://victoriametrics.com/products/enterprise/trial/) for this tutorial or for enterprise use.
- In the tutorial, we'll be using the following VictoriaMetrics components:
- [VictoriaMetrics Single-Node](https://docs.victoriametrics.com/single-server-victoriametrics) (v1.109.1)
- [vmalert](https://docs.victoriametrics.com/vmalert/) (v1.109.1)
- [vmagent](https://docs.victoriametrics.com/vmagent/) (v1.109.1)
- [VictoriaMetrics Single-Node](https://docs.victoriametrics.com/single-server-victoriametrics) (v1.110.0)
- [vmalert](https://docs.victoriametrics.com/vmalert/) (v1.110.0)
- [vmagent](https://docs.victoriametrics.com/vmagent/) (v1.110.0)
- [Grafana](https://grafana.com/) (v.10.2.1)
- [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/)
- [Node exporter](https://github.com/prometheus/node_exporter#node-exporter) (v1.7.0) and [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) (v0.27.0)
@@ -136,22 +136,24 @@ Below is an illustrative example of a `vmanomaly_config.yml` configuration file.
``` yaml
schedulers:
periodic:
# class: 'periodic' # or "scheduler.periodic.PeriodicScheduler" until v1.13.0
infer_every: "1m"
fit_every: "2m"
fit_window: "3h"
fit_every: "1h"
fit_window: "2d" # 2d-14d based on the presense of weekly seasonality in your data
models:
prophet:
class: "prophet" # or "model.prophet.ProphetModel" until v1.13.0
class: "prophet"
args:
interval_width: 0.98
weekly_seasonality: False # comment it if your data has weekly seasonality
yearly_seasonality: False
reader:
datasource_url: "http://victoriametrics:8428/"
sampling_period: "60s"
sampling_period: "60s"
queries:
node_cpu_rate: "sum(rate(node_cpu_seconds_total[5m])) by (mode, instance, job)"
node_cpu_rate:
expr: "sum(rate(node_cpu_seconds_total[5m])) by (mode, instance, job)"
writer:
datasource_url: "http://victoriametrics:8428/"
@@ -313,7 +315,7 @@ Let's wrap it all up together into the `docker-compose.yml` file.
services:
vmagent:
container_name: vmagent
image: victoriametrics/vmagent:v1.109.1
image: victoriametrics/vmagent:v1.110.0
depends_on:
- "victoriametrics"
ports:
@@ -330,7 +332,7 @@ services:
victoriametrics:
container_name: victoriametrics
image: victoriametrics/victoria-metrics:v1.109.1
image: victoriametrics/victoria-metrics:v1.110.0
ports:
- 8428:8428
volumes:
@@ -363,7 +365,7 @@ services:
vmalert:
container_name: vmalert
image: victoriametrics/vmalert:v1.109.1
image: victoriametrics/vmalert:v1.110.0
depends_on:
- "victoriametrics"
ports:
@@ -385,7 +387,7 @@ services:
restart: always
vmanomaly:
container_name: vmanomaly
image: victoriametrics/vmanomaly:v1.19.1
image: victoriametrics/vmanomaly:v1.19.2
depends_on:
- "victoriametrics"
ports:

View File

@@ -18,15 +18,54 @@ See also [LTS releases](https://docs.victoriametrics.com/lts-releases/).
## tip
* FEATURE: [Single-node VictoriaMetrics](https://docs.victoriametrics.com/) and [vmstorage](https://docs.victoriametrics.com/cluster-victoriametrics/): improve startup times when opening a storage with the [retention](https://docs.victoriametrics.com/#retention) exceeding a few months.
## [v1.110.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.110.0)
Released at 2025-01-24
* SECURITY: upgrade Go builder from Go1.23.4 to Go1.23.5. See the list of issues addressed in [Go1.23.5](https://github.com/golang/go/issues?q=milestone%3AGo1.23.5+label%3ACherryPickApproved).
* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): allow executing queries with `$__interval` and `$__rate_interval` - these placeholders are automatically replaced with `1i` (e.g. `step` arg value at [`/api/v1/query_range`](https://docs.victoriametrics.com/keyconcepts/#range-query)) during query execution. This simplifies copying queries from Grafana dashboards.
* FEATURE: [vmsingle](https://docs.victoriametrics.com/single-server-victoriametrics/) and `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): add command-line flag `-search.maxDeleteDuration(default 5m)` to limit the duration of the `/api/v1/admin/tsdb/delete_series` call. Previously, the call is limited by `-search.maxQueryDuration`.
* FEATURE: [dashboards](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/dashboards): all dashboards that use [VictoriaMetrics Grafana datasource](https://github.com/VictoriaMetrics/victoriametrics-datasource) were updated to use a [new datasource ID](https://github.com/VictoriaMetrics/victoriametrics-datasource/releases/tag/v0.12.0).
* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): reflect column settings for the table view in URL, so the table view can be shared via link. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7662).
* BUGFIX: [vmsingle](https://docs.victoriametrics.com/single-server-victoriametrics/), `vminsert` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) and [vmagent](https://docs.victoriametrics.com/vmagent/): allow ingesting histograms with missing `_sum` metric via [OpenTelemetry ingestion protocol](https://docs.victoriametrics.com/#sending-data-via-opentelemetry) in the same way as Prometheus does.
* BUGFIX: [Single-node VictoriaMetrics](https://docs.victoriametrics.com/) and [vmselect](https://docs.victoriametrics.com/cluster-victoriametrics/): respect staleness detection in increase, increase_pure and delta functions when time series has gaps and `-search.maxStalenessInterval` is set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8072) for details.
* BUGFIX: all VictoriaMetrics [enterprise](https://docs.victoriametrics.com/enterprise/) components: properly trim whitespaces at the end of license provided via `-license` and `-licenseFile` command-line flags. Previously, the trailing whitespaces could cause the license verification to fail.
* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth/): fix possible runtime panic during requests processing under heavy load. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8051) for details.
* BUGFIX: [vmselect](https://docs.victoriametrics.com/cluster-victoriametrics/): fix panic when trying to delete series by using [multitenant read](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy-via-labels) endpoint. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8126) for the details.
* BUGFIX: [vmselect](https://docs.victoriametrics.com/cluster-victoriametrics/): prevent panic when `vmselect` receives an error response from `vmstorage` during the query execution and request processing for other `vmstorage` nodes is still in progress. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8114) for the details.
* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix an issue where pressing the "Enter" key in the query editor did not execute the query. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8058).
## [v1.102.11](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.102.11)
Released at 2025-01-24
**v1.102.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes for [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise.html).
All these fixes are also included in [the latest community release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest).
The v1.102.x line will be supported for at least 12 months since [v1.102.0](https://docs.victoriametrics.com/changelog/#v11020) release**
* SECURITY: upgrade Go builder from Go1.23.4 to Go1.23.5. See the list of issues addressed in [Go1.23.5](https://github.com/golang/go/issues?q=milestone%3AGo1.23.5+label%3ACherryPickApproved).
* BUGFIX: [Single-node VictoriaMetrics](https://docs.victoriametrics.com/) and [vmselect](https://docs.victoriametrics.com/cluster-victoriametrics/): don't take into account the last raw sample before the lookbehind window is sample exceeds the staleness interval. This affects correctness of increase, increase_pure, delta functions when preforming calculations on time series with gaps. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8002) for details.
* BUGFIX: all VictoriaMetrics [enterprise](https://docs.victoriametrics.com/enterprise/) components: remove unnecessary delay before failing if all online verification attempts have failed. This should reduce the time required for the component to proceed if all online verification attempts have failed.
* BUGFIX: [vmsingle](https://docs.victoriametrics.com/single-server-victoriametrics/), `vminsert` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) and [vmagent](https://docs.victoriametrics.com/vmagent/): allow ingesting histograms with missing `_sum` metric via [OpenTelemetry ingestion protocol](https://docs.victoriametrics.com/#sending-data-via-opentelemetry) in the same way as Prometheus does.
* BUGFIX: all VictoriaMetrics [enterprise](https://docs.victoriametrics.com/enterprise/) components: properly trim whitespaces at the end of license provided via `-license` and `-licenseFile` command-line flags. Previously, the trailing whitespaces could cause the license verification to fail.
* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth/): fix possible runtime panic during requests processing under heavy load. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8051) for details.
* BUGFIX: [vmselect](https://docs.victoriametrics.com/cluster-victoriametrics/): prevent panic when `vmselect` receives an error response from `vmstorage` during the query execution and request processing for other `vmstorage` nodes is still in progress. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8114) for the details.
## [v1.97.16](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.16)
Released at 2025-01-24
**v1.97.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes for [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise.html).
All these fixes are also included in [the latest community release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest).
The v1.97.x line will be supported for at least 12 months since [v1.97.0](https://docs.victoriametrics.com/CHANGELOG.html#v1970) release**
* BUGFIX: [vmsingle](https://docs.victoriametrics.com/single-server-victoriametrics/), `vminsert` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) and [vmagent](https://docs.victoriametrics.com/vmagent/): allow ingesting histograms with missing `_sum` metric via [OpenTelemetry ingestion protocol](https://docs.victoriametrics.com/#sending-data-via-opentelemetry) in the same way as Prometheus does.
* BUGFIX: [Single-node VictoriaMetrics](https://docs.victoriametrics.com/) and [vmselect](https://docs.victoriametrics.com/cluster-victoriametrics/): don't take into account the last raw sample before the lookbehind window is sample exceeds the staleness interval. This affects correctness of increase, increase_pure, delta functions when preforming calculations on time series with gaps. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8002) for details.
* BUGFIX: all VictoriaMetrics [enterprise](https://docs.victoriametrics.com/enterprise/) components: remove unnecessary delay before failing if all online verification attempts have failed. This should reduce the time required for the component to proceed if all online verification attempts have failed.
## [v1.109.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.109.1)
@@ -37,6 +76,7 @@ Released at 2025-01-17
* BUGFIX: [vmsingle](https://docs.victoriametrics.com/single-server-victoriametrics/), `vminsert` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) and [vmagent](https://docs.victoriametrics.com/vmagent/): log metric names for signals with unsupported delta temporality on ingestion via [OpenTelemetry protocol for metrics](https://docs.victoriametrics.com/#sending-data-via-opentelemetry). Thanks to @chenlujjj for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8018).
* BUGFIX: [Single-node VictoriaMetrics](https://docs.victoriametrics.com/) and [vmselect](https://docs.victoriametrics.com/cluster-victoriametrics/): fix incorrect behavior of increase, increase_pure, delta caused by [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8002). This fix reverts to the previous behavior before [v1.109.0](https://docs.victoriametrics.com/changelog/#v11090). But allows controlling staleness detection for these functions explicitly via `-search.maxStalenessInterval`.
* BUGFIX: all VictoriaMetrics [enterprise](https://docs.victoriametrics.com/enterprise/) components: remove unnecessary delay before failing if all online verification attempts have failed. This should reduce the time required for the component to proceed if all online verification attempts have failed.
* BUGFIX: [vmselect](https://docs.victoriametrics.com/cluster-victoriametrics/): prevent panic when sending `multitenant` [read request](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy-via-labels) to `/api/v1/series/count`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8126) for the details.
## [v1.109.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.109.0)

View File

@@ -82,7 +82,7 @@ VictoriaMetrics Enterprise components are available in the following forms:
It is allowed to run VictoriaMetrics Enterprise components in [cases listed here](#valid-cases-for-victoriametrics-enterprise).
Binary releases of VictoriaMetrics Enterprise are available [at the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest).
Enterprise binaries and packages have `enterprise` suffix in their names. For example, `victoria-metrics-linux-amd64-v1.109.1-enterprise.tar.gz`.
Enterprise binaries and packages have `enterprise` suffix in their names. For example, `victoria-metrics-linux-amd64-v1.110.0-enterprise.tar.gz`.
In order to run binary release of VictoriaMetrics Enterprise component, please download the `*-enterprise.tar.gz` archive for your OS and architecture
from the [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest) and unpack it. Then run the unpacked binary.
@@ -100,8 +100,8 @@ For example, the following command runs VictoriaMetrics Enterprise binary with t
obtained at [this page](https://victoriametrics.com/products/enterprise/trial/):
```sh
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.109.1/victoria-metrics-linux-amd64-v1.109.1-enterprise.tar.gz
tar -xzf victoria-metrics-linux-amd64-v1.109.1-enterprise.tar.gz
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.110.0/victoria-metrics-linux-amd64-v1.110.0-enterprise.tar.gz
tar -xzf victoria-metrics-linux-amd64-v1.110.0-enterprise.tar.gz
./victoria-metrics-prod -license=BASE64_ENCODED_LICENSE_KEY
```
@@ -116,7 +116,7 @@ Alternatively, VictoriaMetrics Enterprise license can be stored in the file and
It is allowed to run VictoriaMetrics Enterprise components in [cases listed here](#valid-cases-for-victoriametrics-enterprise).
Docker images for VictoriaMetrics Enterprise are available [at VictoriaMetrics DockerHub](https://hub.docker.com/u/victoriametrics).
Enterprise docker images have `enterprise` suffix in their names. For example, `victoriametrics/victoria-metrics:v1.109.1-enterprise`.
Enterprise docker images have `enterprise` suffix in their names. For example, `victoriametrics/victoria-metrics:v1.110.0-enterprise`.
In order to run Docker image of VictoriaMetrics Enterprise component, it is required to provide the license key via command-line
flag as described [here](#binary-releases).
@@ -126,13 +126,13 @@ Enterprise license key can be obtained at [this page](https://victoriametrics.co
For example, the following command runs VictoriaMetrics Enterprise Docker image with the specified license key:
```sh
docker run --name=victoria-metrics victoriametrics/victoria-metrics:v1.109.1-enterprise -license=BASE64_ENCODED_LICENSE_KEY
docker run --name=victoria-metrics victoriametrics/victoria-metrics:v1.110.0-enterprise -license=BASE64_ENCODED_LICENSE_KEY
```
Alternatively, the license code can be stored in the file and then referred via `-licenseFile` command-line flag:
```sh
docker run --name=victoria-metrics -v /vm-license:/vm-license victoriametrics/victoria-metrics:v1.109.1-enterprise -licenseFile=/path/to/vm-license
docker run --name=victoria-metrics -v /vm-license:/vm-license victoriametrics/victoria-metrics:v1.110.0-enterprise -licenseFile=/path/to/vm-license
```
Example docker-compose configuration:
@@ -141,7 +141,7 @@ version: "3.5"
services:
victoriametrics:
container_name: victoriametrics
image: victoriametrics/victoria-metrics:v1.109.1
image: victoriametrics/victoria-metrics:v1.110.0
ports:
- 8428:8428
volumes:
@@ -173,7 +173,7 @@ is used to provide key in plain-text:
```yaml
server:
image:
tag: v1.109.1-enterprise
tag: v1.110.0-enterprise
license:
key: {BASE64_ENCODED_LICENSE_KEY}
@@ -184,7 +184,7 @@ In order to provide key via existing secret, the following values file is used:
```yaml
server:
image:
tag: v1.109.1-enterprise
tag: v1.110.0-enterprise
license:
secret:
@@ -233,7 +233,7 @@ spec:
license:
key: {BASE64_ENCODED_LICENSE_KEY}
image:
tag: v1.109.1-enterprise
tag: v1.110.0-enterprise
```
In order to provide key via existing secret, the following custom resource is used:
@@ -250,7 +250,7 @@ spec:
name: vm-license
key: license
image:
tag: v1.109.1-enterprise
tag: v1.110.0-enterprise
```
Example secret with license key:

View File

@@ -236,27 +236,27 @@ services:
- grafana_data:/var/lib/grafana/
vmsingle:
image: victoriametrics/victoria-metrics:v1.109.1
image: victoriametrics/victoria-metrics:v1.110.0
command:
- -httpListenAddr=0.0.0.0:8429
vmstorage:
image: victoriametrics/vmstorage:v1.109.1-cluster
image: victoriametrics/vmstorage:v1.110.0-cluster
vminsert:
image: victoriametrics/vminsert:v1.109.1-cluster
image: victoriametrics/vminsert:v1.110.0-cluster
command:
- -storageNode=vmstorage:8400
- -httpListenAddr=0.0.0.0:8480
vmselect:
image: victoriametrics/vmselect:v1.109.1-cluster
image: victoriametrics/vmselect:v1.110.0-cluster
command:
- -storageNode=vmstorage:8401
- -httpListenAddr=0.0.0.0:8481
vmagent:
image: victoriametrics/vmagent:v1.109.1
image: victoriametrics/vmagent:v1.110.0
volumes:
- ./scrape.yaml:/etc/vmagent/config.yaml
command:
@@ -265,7 +265,7 @@ services:
- -remoteWrite.url=http://vmsingle:8429/api/v1/write
vmgateway-cluster:
image: victoriametrics/vmgateway:v1.109.1-enterprise
image: victoriametrics/vmgateway:v1.110.0-enterprise
ports:
- 8431:8431
volumes:
@@ -281,7 +281,7 @@ services:
- -auth.oidcDiscoveryEndpoints=http://keycloak:8080/realms/master/.well-known/openid-configuration
vmgateway-single:
image: victoriametrics/vmgateway:v1.109.1-enterprise
image: victoriametrics/vmgateway:v1.110.0-enterprise
ports:
- 8432:8431
volumes:
@@ -393,7 +393,7 @@ Once iDP configuration is done, vmagent configuration needs to be updated to use
```yaml
vmagent:
image: victoriametrics/vmagent:v1.109.1
image: victoriametrics/vmagent:v1.110.0
volumes:
- ./scrape.yaml:/etc/vmagent/config.yaml
- ./vmagent-client-secret:/etc/vmagent/oauth2-client-secret

View File

@@ -148,7 +148,7 @@ server:
* By running `helm install vmsingle vm/victoria-metrics-single` we install [VictoriaMetrics Single](https://docs.victoriametrics.com/single-server-victoriametrics/) to default [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) inside your cluster
* By adding `scrape: enable: true` we add and enable autodiscovery scraping from kubernetes cluster to [VictoriaMetrics Single](https://docs.victoriametrics.com/single-server-victoriametrics/)
* By adding `scrape: enabled: true` we add and enable autodiscovery scraping from kubernetes cluster to [VictoriaMetrics Single](https://docs.victoriametrics.com/single-server-victoriametrics/)
* On line 166 from [https://docs.victoriametrics.com/guides/examples/guide-vmsingle-values.yaml](https://docs.victoriametrics.com/guides/examples/guide-vmsingle-values.yaml) we added `metric_relabel_configs` section that will help us to show Kubernetes metrics on Grafana dashboard.

View File

@@ -30,8 +30,8 @@ scrape_configs:
After you created the `scrape.yaml` file, download and unpack [single-node VictoriaMetrics](https://docs.victoriametrics.com/) to the same directory:
```
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.109.1/victoria-metrics-linux-amd64-v1.109.1.tar.gz
tar xzf victoria-metrics-linux-amd64-v1.109.1.tar.gz
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.110.0/victoria-metrics-linux-amd64-v1.110.0.tar.gz
tar xzf victoria-metrics-linux-amd64-v1.110.0.tar.gz
```
Then start VictoriaMetrics and instruct it to scrape targets defined in `scrape.yaml` and save scraped metrics
@@ -146,8 +146,8 @@ Then start [single-node VictoriaMetrics](https://docs.victoriametrics.com/) acco
```yaml
# Download and unpack single-node VictoriaMetrics
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.109.1/victoria-metrics-linux-amd64-v1.109.1.tar.gz
tar xzf victoria-metrics-linux-amd64-v1.109.1.tar.gz
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.110.0/victoria-metrics-linux-amd64-v1.110.0.tar.gz
tar xzf victoria-metrics-linux-amd64-v1.110.0.tar.gz
# Run single-node VictoriaMetrics with the given scrape.yaml
./victoria-metrics-prod -promscrape.config=scrape.yaml

View File

@@ -1556,6 +1556,30 @@ To switch to [the VictoriaMetrics remote write protocol](https://docs.victoriame
simply set the `-remoteWrite.forceVMProto=true` flag. It is also possible to adjust the compression level for the VictoriaMetrics remote write protocol using the `-remoteWrite.vmProtoCompressLevel`
command-line flag.
#### Estimating message size and rate
If you are migrating from remote write to Kafka, the request rate and request body size of remote write can roughly correspond to the message rate and size of Kafka.
vmagent organizes scraped/ingested data into **blocks**. A block contains multiple time series and samples.
Each block is compressed with Snappy or zstd before being sent out by the remote write or the Kafka producer.
In order to get the request rate of remote write (as the estimated produce rate of Kafka), use this MetricsQL:
```metricsql
sum(rate(vmagent_remotewrite_requests_total{}[1m]))
```
Similarly, the average size of the compressed block of remote write (serving as the estimated message size of Kafka) is as follows:
```metricsql
sum(rate(vmagent_remotewrite_conn_bytes_written_total{}[1m]))
/
sum(rate(vmagent_remotewrite_requests_total{}[1m]))
```
Please note that the remote write body and Kafka message need to use the same compression algorithm in order to serve as
estimation references. See more in [the VictoriaMetrics remote write protocol](https://docs.victoriametrics.com/vmagent/#victoriametrics-remote-write-protocol).
#### Kafka broker authorization and authentication
Two types of auth are supported:

Some files were not shown because too many files have changed in this diff Show More