Commit Graph

10140 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
eacd3c11ff vendor: update github.com/VictoriaMetrics/metricsql from v0.83.1 to v0.84.0
The new version of github.com/VictoriaMetrics/metricsql handles $__interval and $__rate_interval
inside rollup functions in more correct way - it drops square brakets, so VictoriaMetrics
could automatically detect the needed lookbehind window depending on the time distance between real samples.

For example, rate(m[$__rate_interval]) is parsed and processed as rate(m) now.
2025-02-24 15:34:59 +01:00
Aliaksandr Valialkin
d56f9327ec lib/logstorage: limit the maximum log field name length, which can be generated by JSONParser.ParseLogMessage
Make sure that the maximum log field name, which can be generated by JSONParser.ParseLogMessage,
doesn't exceed the hardcoded limit maxFieldNameSize. Stop flattening of nested JSON objects
when the resulting field name becomes longer than maxFieldNameSize, and return the nested JSON object
as a string instead.

This should prevent from parse errors when ingesting deeply nested JSON logs with long field names.
2025-02-24 15:34:59 +01:00
Aliaksandr Valialkin
dc536d5626 lib/logstorage: add a benchmark for JSONParser.ParseLogMessage 2025-02-24 15:34:58 +01:00
Aliaksandr Valialkin
0d3ee707ba lib/encoding/zstd: reduce the number of cached zstd.Encoder instances
Use the real compression level supported by github.com/klauspost/compress/zstd as a cache map key.
The number of real compression levels is smaller than the number of zstd compression levels.
This should reduce the number of cached zstd.Encoder instances.

See https://github.com/klauspost/compress/discussions/1025
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7503#issuecomment-2500088591
2025-02-24 15:34:58 +01:00
Aliaksandr Valialkin
37112ab003 docs/VictoriaLogs/LogsQL.md: typo fixes 2025-02-24 15:34:58 +01:00
Aliaksandr Valialkin
e6fb17aad6 deployment/docker: update VictoriaLogs Docker image tag from v1.12.0-victorialogs to v1.13.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.13.0-victorialogs
2025-02-24 15:34:58 +01:00
Aliaksandr Valialkin
283b3f1bfb docs/VictoriaLogs/CHANGELOG.md: cut v1.13.0-victorialogs 2025-02-24 15:34:58 +01:00
Aliaksandr Valialkin
3ee4b3ef24 lib/logstorage: add contains_any and contains_all filters
- `contains_any` selects logs with fields containing at least one word/phrase from the provided list.
  The provided list can be generated by a subquery.

- `contains_all` selects logs with fields containing all the words and phrases from the provided list.
  The provided list can be generated by a subquery.
2025-02-24 15:34:58 +01:00
Aliaksandr Valialkin
3e941920f6 lib/logstorage: do not spend CPU time on preparing values for already filtered out rows according to bm at filterEqField.applyToBlockSearch 2025-02-24 15:34:57 +01:00
Aliaksandr Valialkin
6975352d5a lib/logstorage: avoid extra memory allocations at getEmptyStrings() 2025-02-24 15:34:57 +01:00
Aliaksandr Valialkin
a2d0846e86 lib/logstorage: add an ability to drop duplicate words at unpack_words pipe 2025-02-24 15:34:57 +01:00
Aliaksandr Valialkin
518ed87a3a lib/logstorage: rename unpack_tokens to unpack_words pipe
The LogsQL defines a word at https://docs.victoriametrics.com/victorialogs/logsql/#word ,
so it is more natural to use unpack_words instead of unpack_tokens name for the pipe.
2025-02-24 15:34:57 +01:00
Aliaksandr Valialkin
4beceb67ab lib/logstorage: optimize OR filter a bit for many inner filters
Use two operations on bitmaps per each inner filter instead of three operations.
2025-02-24 15:34:57 +01:00
Aliaksandr Valialkin
bff5551ba5 lib/logstorage: use clear() for clearing bitmap bits at resetBits() instead of a loop
The clear() call is easier to read and understand than the loop.
2025-02-24 15:34:56 +01:00
Aliaksandr Valialkin
4dfd1407ba lib/logstorage: avoid calling bitmap.reset() at getBitmap()
The bitmap at getBitamp() must be already reset when it was returned to the pool via putBitamp().
Thise saves CPU a bit.
2025-02-24 15:34:56 +01:00
Aliaksandr Valialkin
bc3e557f02 lib/logstorage: improve error logging for improperly escaped backslashes inside quoted strings
This should simplify debugging LogsQL queries by users
2025-02-24 15:34:56 +01:00
Aliaksandr Valialkin
1f11bc948e lib/logstorage: add field1:eq_field(field2) filter, which returns logs with identical values at field1 and field2 2025-02-24 15:34:56 +01:00
Aliaksandr Valialkin
504c034cbf lib/logstorage: optimize len, hash and json_array_len pipes for repeated values
Re-use the previous result instead of calculating new result for repated input values
2025-02-24 15:34:56 +01:00
Aliaksandr Valialkin
959282090a lib/logstorage: add json_array_len pipe for calculating the length of JSON arrays 2025-02-24 15:34:56 +01:00
Aliaksandr Valialkin
aef939dc20 lib/logstorage: refactor unroll_tokens into unpack_tokens pipe
unpack_tokens pipe generates a JSON array of unpacked tokens from the source field.
This composes better with other pipes such as unroll pipe.
2025-02-24 15:34:55 +01:00
Aliaksandr Valialkin
afd74d82db lib/logstorage: add unroll_tokens pipe for unrolling individual word tokens from the log field 2025-02-24 15:34:55 +01:00
Fred Navruzov
727bfe5f12 docs/vmanomaly: fix 404 in dashboard guide (#8372)
### Describe Your Changes

- Fixed 404 links to anomaly score dashboard `.json` file on Github.
- Fixed broken markdown on QuickStart page.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-02-24 15:34:55 +01:00
Fred Navruzov
e5434ce77a docs: vmanomaly - add anomaly score dashboard and guide (#8284)
### Describe Your Changes

> (currently in DRAFT) This dashboard and the docs may look differently
after recent review and the feedback received.

As requested by our customers, this PR introduces `anomaly score`-based
dashboard for `vmanomaly` to improve the visual experience and ease the
drill down debugging process for respective anomaly detection setups.
Accompanying guide (under `default` preset mode) is provided as well.

- [For additional
benefits](https://docs.victoriametrics.com/victoriametrics-datasource/#motivation),
the dashboard is based on [VictoriaMetrics
datasource](https://docs.victoriametrics.com/victoriametrics-datasource/)
rather than on `Prometheus` datasource
- Tested locally and on our https://play.victoriametrics.com/
`anomaly_score` data (tenant ID 0)
- To check the guide, build the docs locally (`vmdocs`, `make run
local`) and follow to the
http://localhost:1313/anomaly-detection/presets/#default section

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-02-24 15:34:55 +01:00
f41gh7
084560bb02 make vmui-update 2025-02-21 14:08:10 +01:00
f41gh7
901449d5cc docs/changelog: follow-up after 9ca74d1fff
Move vmalert BUGFIX record into current release.

Signed-off-by: f41gh7 <nik@victoriametrics.com>
2025-02-21 13:24:30 +01:00
Zakhar Bessarab
068772ec0e app/vmalert: properly unregister exposed metrics for alerting rules
Previously if rule group parameters were changed, alerting rules related metrics could be deleted due to bug at `utils/metrics` package. 

 This commit introduces `metrics.Set` per rule group. It holds group and alerting rules metrics. It properly unregister alerting rules metrics and addresses issue.
In addition:
- expose group metrics only once group is started - this helps to avoid
exposing metrics for groups which are created during YAML unmarshaling
and only used to update existing group.
- properly close rules which are discarded after updating existing rules
so that metrics are also correctly closed.
- detect file renames and properly recreate groups "moved" between files.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8229
2025-02-21 12:43:27 +01:00
Aliaksandr Valialkin
bfc65952df deployment/docker: update VictoriaLogs Docker image tag from v1.11.0-victorialogs to v1.12.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.12.0-victorialogs
2025-02-21 12:43:26 +01:00
Aliaksandr Valialkin
2dfd6bb689 lib/logstorage: simplify usage of top, uniq and unroll pipes by allowing comma-separated list of fields without parens
Examples:

   - `top 5 x, y` is equivalent to `top 5 by (x, y)`
   - `uniq foo, bar` is equivalent to `uniq by (foo, bar)`
   - `unroll foo, bar` is equivalent to `unroll (foo, bar)`
2025-02-21 12:43:26 +01:00
Aliaksandr Valialkin
31a5bb8ce8 docs/VictoriaLogs/CHANGELOG.md: cut v1.12.0-victorialogs 2025-02-21 12:43:26 +01:00
Aliaksandr Valialkin
061fd098b5 lib/logstorage: properly handle _time:<=max_time filter
_time:<=max_time filter must include logs with timestamps matching max_time.
For example, _time:<=2025-02-24Z must include logs with timestamps until the end of February 24, 2025.
2025-02-21 12:43:26 +01:00
Aliaksandr Valialkin
80d173471f lib/logstorage: allow using '>', '>=', '<' and '<=' in '_time:...' filter
Examples:

  _time:>=2025-02-24Z selects logs with timestamps bigger or equal to 2025-02-24 UTC
  _time:>1d selects logs with timestamps older than one day comparing to the current time

This simplifies writing queries with _time filters.
See https://docs.victoriametrics.com/victorialogs/logsql/#time-filter
2025-02-21 12:43:26 +01:00
Aliaksandr Valialkin
a6bf74443a docs/VictoriaLogs/keyConcepts.md: small clarifications 2025-02-21 12:43:25 +01:00
Jose Gómez-Sellés
b8159fcad4 docs/cloud: fix image routes in getstarted (#8367)
It looks like the version deployed works differently than in local, and
some paths are not taken correctly. This commit fixes that by adding
absolute paths to the 2 images included in this section.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-02-21 12:43:25 +01:00
Zakhar Bessarab
933f51d01a docs/security: update LTS version listed in security docs (#8366)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-02-21 12:43:25 +01:00
Jose Gómez-Sellés
63e21547c4 docs: fix vmanomaly image broken links (#8364)
It looks like when using html image insertion, the path is changed. This
commit fixes that by redirecting to the previous dir in path.

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-02-21 12:43:25 +01:00
Jose Gómez-Sellés
b7a23e0d2e docs/cloud: rework quickstart guide (#8363)
This guide was outdated. the intention of these changes is to:
* Ease maintainability
* Ease user experience
* Encourage the user to quickly set up the product

Some images are removed because they were obsolete or to improve
readability

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-02-21 12:43:25 +01:00
Aliaksandr Valialkin
28b7684adb docs/VictoriaLogs/CHANGELOG.md: move the description of the change at ca1d1bc12b to the appropriate release
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8314
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8315
2025-02-21 12:43:25 +01:00
Andrii Chubatiuk
8538821293 docs: set proper code syntax for snippets in cluster docs (#8362)
### Describe Your Changes

apply proper syntax for code blocks

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-02-21 12:43:24 +01:00
Roman Khavronenko
678e404c3a apptest: add tests for querying utf-8 label names (#8359)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 9a2213eaa8)
2025-02-20 11:07:18 +01:00
Roman Khavronenko
0cd761e7d7 metricsql: support auto-format (prettify) for expressions that use quoted metric or label names (#8358)
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7703

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 1d4713f989)
2025-02-20 11:07:18 +01:00
Roman Khavronenko
a4643bf552 apptest/vmsingle: check for right status code in return (#8355)
follow-up for
855dfb324d

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 85e2307e7c)
2025-02-20 11:07:17 +01:00
Yury Molodov
a61a94e80a vmui: improve Explore Cardinality page (#8351)
### Describe Your Changes

This PR includes improvements to the **Explore Cardinality** page:

- Improve number formatting for better readability.
- Add full error message display.
- Move values representing changes relative to the previous day to a
separate column for easier sorting.
- Refine UI styles for a better visual experience.

<details>

<summary>Demo UI</summary>

| Before | After |
|--------|-------|
| <img width="1269" alt="image"
src="https://github.com/user-attachments/assets/a188c94c-3a06-498d-994d-453080c436a7"
/> | <img width="1269" alt="image"
src="https://github.com/user-attachments/assets/9a42624c-2779-457f-9c98-920fca116eeb"
/> |
| <img width="1269" alt="image"
src="https://github.com/user-attachments/assets/cb4973ba-d7a0-475b-a7a4-f649607dc7b5"
/> | <img width="1269" alt="image"
src="https://github.com/user-attachments/assets/b396516e-2468-4835-b300-88b7a82deec0"
/> |

</details>

Related issue: #8318

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 086ca47884)
2025-02-20 11:07:17 +01:00
hagen1778
81d0c49d98 docs: fix typos in recent change lines
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit b5c21337d3)
2025-02-20 11:07:17 +01:00
Hui Wang
93bbe10074 app/vmselect: add query resource limits priority
This commit adds support for overriding vmstorage `maxUniqueTimeseries` with specific
resource limits:
1. `-search.maxLabelsAPISeries` for
[/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels),
[/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues)
2. `-search. maxSeries` for
[/api/v1/series](https://docs.victoriametrics.com/url-examples/#apiv1series)
3. `-search.maxTSDBStatusSeries` for
[/api/v1/status/tsdb](https://docs.victoriametrics.com/#tsdb-stats)
4. `-search.maxDeleteSeries` for
[/api/v1/admin/tsdb/delete_series](https://docs.victoriametrics.com/url-examples/#apiv1admintsdbdelete_series)

Currently, this limit priority logic cannot be applied to flags
`-search.maxFederateSeries` and `-search.maxExportSeries`, because they
share the same RPC `search_v7` with the /api/v1/query and
/api/v1/query_range APIs, preventing vmstorage from identifying the
actual API of the request. To address that, we need to add additional
information to the protocol between vmstorage and vmselect, which should
be introduced in the future when possible.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7857
2025-02-19 18:14:54 +01:00
Yury Molodov
daa0b0ec18 vmui: add legend customization options
Legend settings have been added to **Graph Settings**.  

#### **New Features:**  
1. **Table View** – Toggle to display the legend in a table format.  
2. **Hide Common Values** – Option to hide fields with identical values
across all series.
3. **Hide Min/Medium/Max** – Ability to hide min, median, and max values
from the legend.
4. **Custom Label Format** – Set a custom format for series labels
(applies only to the legend).
5. **Group by Label** – Group legend entries based on a selected label.


Related Issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8031
2025-02-19 18:12:41 +01:00
Gunju Kim
1331334357 app/vmalert: add command line flag -notifier.sendTimeout
Currently, vmalert uses a fixed 10-second client timeout for notifiers,
which can prevent large sets of alerts from being sent successfully.

This introduces `-notifier.sendTimeout` flag to vmalert to control the
client timeout duration for the notifiers.


Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8287
2025-02-19 18:12:41 +01:00
Andrii Chubatiuk
94bf90842a app/vlinsert/syslog: properly parse log line with characters escaped by rfc5424
Inside PARAM-VALUE, the characters '"' (ABNF %d34), '\' (ABNF %d92),
and ']' (ABNF %d93) MUST be escaped.  This is necessary to avoid
parsing errors.  Escaping ']' would not strictly be necessary but is
REQUIRED by this specification to avoid syslog application
implementation errors.  Each of these three characters MUST be
escaped as '\"', '\\', and '\]' respectively.  The backslash is used
for control character escaping for consistency with its use for
escaping in other parts of the syslog message as well as in traditional syslog.

 Related RFC:
https://datatracker.ietf.org/doc/html/rfc5424#section-6.3.3

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8282
2025-02-19 18:12:40 +01:00
Zakhar Bessarab
32d1a15ceb app/vmselect/filtersdebug: properly validate retention configuration
This commit properly validate retention flag value input for the enterprise debug UI.
API properly checks:
- presence of global retention configuration
- validate that retention is at least 1 day, same as vmstorage enforcement

Related issue:
 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8343

---------
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-02-19 17:52:42 +01:00
Andrii Chubatiuk
99de7456c3 lib/protoparser/influx: add -influx.forceStreamMode flag to force parsing all Influx data in stream mode (#8319)
Addresses #8269

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-02-19 17:40:32 +01:00
Roman Khavronenko
018053a31f docs: update histogram recommendation for stream aggr (#8349)
Use `rate_sum` instead of `total` output for the following reasons:
* `rate_sum` is far less sensitive for data delays than `total`, since
it represents the speed of change instead of absolute values.
* `rate_sum` remove need in using `rate` function for calculating final
results on query time.

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-19 17:40:32 +01:00