Compare commits

..

3 Commits

Author SHA1 Message Date
Max Kotliar
a539397848 make linter happy 2026-07-24 17:46:02 +03:00
Max Kotliar
b84c0d91da app/vmui: sync ui with vmanget recent changes
In https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10607 vmagent
metrics relabling debug UI has been refined. There are a few changes
that make sence in VMUI too:
- Rename "Labels" to "A time series" to highlight that a single
Prometheus time series can be provided in text format.
- Rename "Relabel Configs" to "Config". It should be clear from the page
contex what config is meant. Also there are links to documention.
- Add placeholder. It hints what could be placed into "A time series"
textarea.
2026-07-24 17:40:29 +03:00
Zasda Yusuf Mikail
07b6070193 app/vmselect/promql: drop data points with missing samples on the right side of comparison operations
NaN values in non-scalar operands of comparison operations mean missing
samples, so they cannot match any sample on the other side. Previously
such data points were passed to the comparison function, so queries like
`foo != (bar > 100)` could return unexpected results, since `value !=
NaN` evaluates to `true`. This PR drops such data points in the same way
as Prometheus does.

Comparisons to scalars such as `foo != nan` keep the previous behavior,
since a scalar `NaN` is an explicit value rather than a missing sample
(see the existing `compare_to_nan_right` test case).

The check is applied only to the right side of the operation, since a
missing sample on the left side already results in `NaN` for every
comparison operation:
- `==`, `>`, `<`, `>=`, `<=`: `cf(NaN, b)` is `false`, so the data point
is dropped;
- `!=`: `cf(NaN, b)` is `true`, but the returned value is the left one,
which is `NaN`;
- `bool` modifier: the left `NaN` is handled explicitly in
`newBinaryOpCmpFunc`.

Verified with the reproduction steps from the linked issue: the query
`tsafdb0_value{table="t1", timeSeries="c0"} != ((272) <=
(tsafdb0_value{table="t1", timeSeries="c0"}))` now returns an empty
result, matching Prometheus behavior.

Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10018
2026-07-24 15:59:45 +02:00
7 changed files with 193 additions and 27 deletions

View File

@@ -18,7 +18,7 @@ groups:
concurrency: 2
rules:
- alert: RequestErrorsToAPI
expr: increase(vm_http_request_errors_total{path=~".+"}[5m]) > 0
expr: increase(vm_http_request_errors_total[5m]) > 0
for: 15m
labels:
severity: warning

View File

@@ -164,11 +164,11 @@ func newBinaryOpFunc(bf func(left, right float64, isBool bool) float64) binaryOp
left := bfa.left
right := bfa.right
op := bfa.be.Op
switch true {
case metricsql.IsBinaryOpCmp(op):
// Do not remove empty series for comparison operations,
// since this may lead to missing result.
default:
isCmpOp := metricsql.IsBinaryOpCmp(op)
if !isCmpOp {
// Do not remove empty series for comparison operations, since NaN can be an
// explicitly present sample value. In particular, `value != NaN` is true.
// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/150.
left = removeEmptySeries(left)
right = removeEmptySeries(right)
}
@@ -191,6 +191,14 @@ func newBinaryOpFunc(bf func(left, right float64, isBool bool) float64) binaryOp
isBool := bfa.be.Bool
fillLeft := bfa.be.FillLeft
fillRight := bfa.be.FillRight
// A NaN produced by a vector comparison denotes a filtered-out sample rather
// than an explicitly present NaN value. Drop it when this result is used as
// the right-hand side of another comparison.
// Note that filtered-out samples surviving as NaN inside other wrappers such as
// transform functions or arithmetic operations aren't detected, since such NaN
// is indistinguishable from an explicitly present NaN value there.
// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10018.
dropNaNRight := isCmpOp && isVectorComparisonExpr(bfa.be.Right)
for i, tsLeft := range left {
leftValues := tsLeft.Values
rightValues := right[i].Values
@@ -203,11 +211,16 @@ func newBinaryOpFunc(bf func(left, right float64, isBool bool) float64) binaryOp
b := rightValues[j]
leftIsNaN := math.IsNaN(a)
rightIsNaN := math.IsNaN(b)
// apply the fill value when either the left or right side is NaN, but not both.
// Both sides are NaN.
if leftIsNaN && rightIsNaN {
dstValues[j] = bf(a, b, isBool)
continue
}
if dropNaNRight && rightIsNaN && fillRight == nil {
dstValues[j] = nan
continue
}
// Apply the fill value when either the left or right side is NaN, but not both.
if leftIsNaN && fillLeft != nil {
a = fillLeft.N
}
@@ -223,6 +236,38 @@ func newBinaryOpFunc(bf func(left, right float64, isBool bool) float64) binaryOp
}
}
// isVectorComparisonExpr returns whether e is a comparison operation
// with at least one non-scalar operand.
func isVectorComparisonExpr(e metricsql.Expr) bool {
if re, ok := e.(*metricsql.RollupExpr); ok && re.Window == nil {
// Unwrap `(...) offset <d>` and `(...) @ <t>`, which do not change
// the shape of the result. Subqueries are left as is, since rollup
// functions skip NaN values on their own.
e = re.Expr
}
be, ok := e.(*metricsql.BinaryOpExpr)
if !ok || !metricsql.IsBinaryOpCmp(be.Op) {
return false
}
return !isScalarLikeExpr(be.Left) || !isScalarLikeExpr(be.Right)
}
// isScalarLikeExpr returns whether e always evaluates to a scalar value.
func isScalarLikeExpr(e metricsql.Expr) bool {
switch e := e.(type) {
case *metricsql.NumberExpr, *metricsql.DurationExpr:
return true
case *metricsql.BinaryOpExpr:
return isScalarLikeExpr(e.Left) && isScalarLikeExpr(e.Right)
case *metricsql.FuncExpr:
switch strings.ToLower(e.Name) {
case "now", "pi", "scalar", "start", "end", "step", "time", "timezone_offset":
return true
}
}
return false
}
func adjustBinaryOpTags(be *metricsql.BinaryOpExpr, left, right []*timeseries) ([]*timeseries, []*timeseries, []*timeseries, error) {
if len(be.GroupModifier.Op) == 0 && len(be.JoinModifier.Op) == 0 {
if isScalar(left) {

View File

@@ -2994,6 +2994,136 @@ func TestExecSuccess(t *testing.T) {
resultExpected := []netstorage.Result{}
f(q, resultExpected)
})
t.Run(`compare_to_nan_left_vector_right_scalar`, func(t *testing.T) {
t.Parallel()
q := `label_set(time(), "foo", "bar") != NaN`
r := netstorage.Result{
MetricName: metricNameExpected,
Values: []float64{1000, 1200, 1400, 1600, 1800, 2000},
Timestamps: timestampsExpected,
}
r.MetricName.Tags = []storage.Tag{{
Key: []byte("foo"),
Value: []byte("bar"),
}}
resultExpected := []netstorage.Result{r}
f(q, resultExpected)
})
t.Run(`compare_to_non_nan_scalar_right`, func(t *testing.T) {
t.Parallel()
q := `label_set(time(), "foo", "bar") != 1200`
r := netstorage.Result{
MetricName: metricNameExpected,
Values: []float64{1000, nan, 1400, 1600, 1800, 2000},
Timestamps: timestampsExpected,
}
r.MetricName.Tags = []storage.Tag{{
Key: []byte("foo"),
Value: []byte("bar"),
}}
resultExpected := []netstorage.Result{r}
f(q, resultExpected)
})
t.Run(`compare_to_nan_vector_right`, func(t *testing.T) {
t.Parallel()
q := `label_set(time(), "foo", "bar") != label_set(NaN, "foo", "bar")`
r := netstorage.Result{
MetricName: metricNameExpected,
Values: []float64{1000, 1200, 1400, 1600, 1800, 2000},
Timestamps: timestampsExpected,
}
r.MetricName.Tags = []storage.Tag{{
Key: []byte("foo"),
Value: []byte("bar"),
}}
resultExpected := []netstorage.Result{r}
f(q, resultExpected)
})
t.Run(`compare_to_nan_scalar_comparison_right`, func(t *testing.T) {
t.Parallel()
q := `label_set(time(), "foo", "bar") != (1 > 2)`
r := netstorage.Result{
MetricName: metricNameExpected,
Values: []float64{1000, 1200, 1400, 1600, 1800, 2000},
Timestamps: timestampsExpected,
}
r.MetricName.Tags = []storage.Tag{{
Key: []byte("foo"),
Value: []byte("bar"),
}}
resultExpected := []netstorage.Result{r}
f(q, resultExpected)
})
t.Run(`compare_to_empty_vector_right`, func(t *testing.T) {
t.Parallel()
q := `label_set(time(), "foo", "bar") != (label_set(time(), "foo", "bar") > 100000)`
resultExpected := []netstorage.Result{}
f(q, resultExpected)
})
t.Run(`compare_to_empty_vector_right_offset`, func(t *testing.T) {
t.Parallel()
q := `label_set(time(), "foo", "bar") != ((label_set(time(), "foo", "bar") > 100000) offset 0s)`
resultExpected := []netstorage.Result{}
f(q, resultExpected)
})
t.Run(`compare_to_empty_vector_left`, func(t *testing.T) {
// A missing sample on the left side results in NaN for every comparison
// operation, so no special handling is needed there.
t.Parallel()
q := `(label_set(time(), "foo", "bar") > 100000) != label_set(time(), "foo", "bar")`
resultExpected := []netstorage.Result{}
f(q, resultExpected)
})
t.Run(`compare_to_empty_series_right_bool`, func(t *testing.T) {
t.Parallel()
q := `label_set(time(), "foo", "bar") == bool (label_set(time(), "foo", "bar") > 100000)`
resultExpected := []netstorage.Result{}
f(q, resultExpected)
})
t.Run(`compare_to_partially_empty_series_right`, func(t *testing.T) {
// Prometheus drops the timestamps filtered out by the comparison on the right.
// See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11100#issuecomment-4933952390.
t.Parallel()
q := `label_set(time(), "foo", "bar") != (label_set(time(), "foo", "bar") * 2 > 2800)`
r := netstorage.Result{
MetricName: metricNameExpected,
Values: []float64{nan, nan, nan, 1600, 1800, 2000},
Timestamps: timestampsExpected,
}
r.MetricName.Tags = []storage.Tag{{
Key: []byte("foo"),
Value: []byte("bar"),
}}
resultExpected := []netstorage.Result{r}
f(q, resultExpected)
})
t.Run(`compare_to_empty_unlabeled_vector_right`, func(t *testing.T) {
t.Parallel()
q := `sum(label_set(time(), "foo", "bar")) != (sum(label_set(time(), "foo", "bar")) > 100000)`
resultExpected := []netstorage.Result{}
f(q, resultExpected)
})
t.Run(`compare_to_empty_series_right_with_fill_left`, func(t *testing.T) {
t.Parallel()
q := `label_set(time(), "foo", "bar") != fill_left(0) (label_set(time(), "foo", "bar") > 100000)`
resultExpected := []netstorage.Result{}
f(q, resultExpected)
})
t.Run(`compare_to_empty_series_right_with_fill_right`, func(t *testing.T) {
t.Parallel()
q := `label_set(time(), "foo", "bar") != fill_right(0) (label_set(time(), "foo", "bar") > 100000)`
r := netstorage.Result{
MetricName: metricNameExpected,
Values: []float64{1000, 1200, 1400, 1600, 1800, 2000},
Timestamps: timestampsExpected,
}
r.MetricName.Tags = []storage.Tag{{
Key: []byte("foo"),
Value: []byte("bar"),
}}
resultExpected := []netstorage.Result{r}
f(q, resultExpected)
})
t.Run(`-1 < 2`, func(t *testing.T) {
t.Parallel()
q := `-1 < 2`

View File

@@ -72,7 +72,7 @@ const Relabel: FC = () => {
<div className="vm-relabeling-header-configs">
<TextField
type="textarea"
label="Relabel configs"
label="Config"
value={config}
autofocus
onChange={handleChangeConfig}
@@ -82,8 +82,9 @@ const Relabel: FC = () => {
<div className="vm-relabeling-header__labels">
<TextField
type="textarea"
label="Labels"
label="A time series"
value={labels}
placeholder="up{job=&quot;job_name&quot;,instance=&quot;host:port&quot;}"
onChange={handleChangeLabels}
onEnter={handleRunQuery}
/>
@@ -97,22 +98,22 @@ const Relabel: FC = () => {
<Button
variant="text"
color="gray"
startIcon={<InfoIcon/>}
startIcon={<WikiIcon/>}
>
Relabeling cookbook
</Button>
</a>
<a
target="_blank"
href="https://docs.victoriametrics.com/victoriametrics/relabeling/"
href="https://docs.victoriametrics.com/victoriametrics/relabeling/#relabeling-stages"
rel="help noreferrer"
>
<Button
variant="text"
color="gray"
startIcon={<WikiIcon/>}
startIcon={<InfoIcon/>}
>
Documentation
Relabeling Stages
</Button>
</a>
<Button

View File

@@ -75,7 +75,7 @@ groups:
Consider to limit the ingestion rate, decrease retention or scale the disk space if possible."
- alert: RequestErrorsToAPI
expr: increase(vm_http_request_errors_total{path=~".+", path!="*"}[5m]) > 0
expr: increase(vm_http_request_errors_total[5m]) > 0
for: 15m
labels:
severity: warning
@@ -86,18 +86,6 @@ groups:
description: "Requests to path {{ $labels.path }} are receiving errors.
Please verify if clients are sending correct requests."
- alert: RequestErrorsToUnknownPaths
expr: sum(increase(vm_http_request_errors_total{path="*"}[5m])) by(job, instance, reason) > 0
for: 15m
labels:
severity: warning
show_at: dashboard
annotations:
dashboard: "{{ $externalURL }}/d/oS7Bi_0Wz?viewPanel=52&var-instance={{ $labels.instance }}"
summary: "Too many errors served for {{ $labels.job }} with reason {{ $labels.reason }} (instance {{ $labels.instance }})"
description: "Requests are failing with reason {{ $labels.reason }}.
Please verify if clients are sending correct requests."
- alert: RPCErrors
expr: |
(

View File

@@ -75,7 +75,7 @@ groups:
Consider to limit the ingestion rate, decrease retention or scale the disk space if possible."
- alert: RequestErrorsToAPI
expr: increase(vm_http_request_errors_total{path=~".+"}[5m]) > 0
expr: increase(vm_http_request_errors_total[5m]) > 0
for: 15m
labels:
severity: warning

View File

@@ -34,6 +34,7 @@ See also [LTS releases](https://docs.victoriametrics.com/victoriametrics/lts-rel
* FEATURE: [dashboards](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/dashboards): add `Fsync avg duration` panel to the Troubleshooting section of the single-node, cluster, and vmagent dashboards. This panel surfaces degradation of IO operation for faster incident triage. See [#10432](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10432).
* FEATURE: [vmbackupmanager](https://docs.victoriametrics.com/victoriametrics/vmbackupmanager/) and [alerts](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/rules): introduce `vm_backup_last_success_at` metric to track the last successful backup by type. Add [alerting rules](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/rules/alerts-vmbackupmanager.yml) `NoLatestBackupWithinLastDay`, `NoHourlyBackupWithinLastDay`, `NoDailyBackupWithinLast3Days`, `NoWeeklyBackupWithinLast14Days` and `NoMonthlyBackupWithinLast62Days` to remind users about the missing backups. See [#11217](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11217).
* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/): properly drop data points filtered out by an inner [comparison operation](https://prometheus.io/docs/prometheus/latest/querying/operators/#comparison-binary-operators) when its result is used on the right side of another comparison. Previously, queries like `foo != (bar > 100)` could return unexpected results because filtered-out data points are represented internally as `NaN`, and `value != NaN` evaluates to `true`. Comparisons against explicitly present `NaN` values keep the previous behavior. See [#10018](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10018). Thanks to @zasdaym for contribution.
* BUGFIX: [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/): fixed the display of rule state badges on the `Groups` page in the web UI. See [#11160](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11160).
* BUGFIX: [vmbackupmanager](https://docs.victoriametrics.com/victoriametrics/vmbackupmanager/): previously, `vmbackupmanager` was crashing on startup when it failed to restore backup state from remote storage, causing a crash loop. Now it logs the error and continues running, retrying the state restore before each scheduled backup. Added `vm_backup_errors_total{type="restoreState"}` metric to track backup state restore failures. See [#11217](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11217).
@@ -122,6 +123,7 @@ Released at 2026-06-22
* BUGFIX: [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): properly escape `metricFamilyName` at metrics metadata response. See [#11129](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11129). Thanks for @fxrlv for the contribution.
* BUGFIX: [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and `vmstorage` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): prevent more cases of panic during directory deletion on `NFS`-based mounts. See [#11060](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11060).
## [v1.145.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.145.0)
Released at 2026-06-08