Improve Influx parsing error message when raw newline (\n) appears inside quoted fieldvmagent: Improve Influx parsing error message when raw newline (\n)… (#10524)

# Investigation & Root Cause --- InfluxDB Line Protocol Parsing with Raw
Newline (`\n`)

This document describes the investigation process and root cause
analysis for Influx Line Protocol parsing errors in VictoriaMetrics when
a **raw newline (`\n`) byte appears inside a quoted field value**.

------------------------------------------------------------------------

## Background

According to the Influx Line Protocol specification:

-   Each point must be represented as a single line.
-   The newline character (`\n`) separates points.
-   Literal newline bytes are not allowed inside quoted field values.

Therefore, any raw newline byte (`0x0A`) inside a quoted string makes
the line invalid.

------------------------------------------------------------------------

## Related Issue

Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10067

------------------------------------------------------------------------

## Expected Behavior

VictoriaMetrics should reject Influx Line Protocol lines that contain a
raw newline inside a quoted field value, since this violates the
protocol specification.

The parsing failure itself is correct.

------------------------------------------------------------------------

## Actual Behavior

VictoriaMetrics rejects the line with the following error:

cannot parse field value for "...": missing closing quote for quoted
field value

While technically correct, the error message does not clearly indicate
that the root cause is a raw newline inside the quoted field value.

------------------------------------------------------------------------

## Minimal Reproducer

The issue can be reproduced without Telegraf or Jolokia:

``` bash
printf 'test value="hello
world"\n' | curl -X POST http://localhost:8428/write --data-binary @-
```

This produces:

cannot parse field value for "value": missing closing quote for quoted
field value

The failure occurs because the value contains an actual newline byte
(0x0A), not the escaped sequence `\n`.

------------------------------------------------------------------------

## Environment Setup

The issue was reproduced using the following stack:

-   VictoriaMetrics v1.127.0
-   InfluxDB 1.8
-   Spring Boot + Jolokia
-   Telegraf 1.36.2

Telegraf collects JVM `SystemProperties`, including:

``` json
"line.separator": "\n"
```

After JSON unmarshalling, this becomes a real newline byte in memory.

Detailed reproduction steps can be found here:

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10067#issuecomment-3896175100

------------------------------------------------------------------------

## Observed Serialized Line

Using breakpoint debugging in:

    lib/bytesutil/bytebuffer.go:58

The `ReadFrom` function reads and assembles an Influx line containing:

    SystemProperties.line.separator="
    ",

The quoted field contains an actual newline byte before the closing
quote.

This breaks the single-line assumption of Influx Line Protocol.

VictoriaMetrics splits on `\n`, resulting in:

-   A truncated first line
-   A missing closing quote
-   Parsing failure

------------------------------------------------------------------------

## Important Clarification

This issue is **not** caused by the escaped sequence `"\\n"`.

The failure occurs only when the serialized Influx line contains an
actual newline byte (`0x0A`) inside the quoted value.

Escaped `\n` (two characters: `\` and `n`) is valid.

------------------------------------------------------------------------

## Root Cause

-   Telegraf serializes a field containing a real newline byte.
-   Influx Line Protocol forbids literal newline characters inside
    quoted fields.
-   VictoriaMetrics correctly treats `\n` as a line separator.
-   The parser then encounters an incomplete quoted field and reports
    "missing closing quote".

The parsing behavior is correct per specification.

------------------------------------------------------------------------

## Proposed Improvement

The parsing logic should remain unchanged.

However, the error message can be improved to better indicate the root
cause.

Suggested error message:

invalid Influx line protocol: missing closing quote for quoted field
value;
this may be caused by a raw newline (`\n`) inside the quoted field value

This makes the failure immediately actionable and easier to diagnose.

------------------------------------------------------------------------

## Summary

-   The failure is caused by a raw newline byte inside a quoted field
    value.
-   This violates the Influx Line Protocol specification.
-   VictoriaMetrics correctly rejects the line.
-   The error message should explicitly mention the possibility of a raw
    newline (`\n`) inside the quoted field.

Signed-off-by: hklhai <hkhai@outlook.com>
Co-authored-by: Max Kotliar <kotlyar.maksim@gmail.com>

docs/victoriametrics/changelog/CHANGELOG.md

@@ -33,6 +33,7 @@ See also [LTS releases](https://docs.victoriametrics.com/victoriametrics/lts-rel
 * FEATURE: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/), [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/), `vminsert` and `vmstorage` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): enable [ingestion](https://docs.victoriametrics.com/victoriametrics/vmagent/#metric-metadata) and in-memory [storage](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#metrics-metadata) of metrics metadata by default. Metadata ingestion can be disabled with `-enableMetadata=false`. See [#2974](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2974).
 * FEATURE: [dashboards/operator](https://grafana.com/grafana/dashboards/17869): extract operator version from metrics instead of hardcoded value
 * FEATURE: [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and `vmstorage` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): increase default value for `-storage.minFreeDiskSpaceBytes` flag from 10M to 100M to reduce risk of panics under high ingestion on small disks. See [#9561](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9561).
+* FEATURE: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) and [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/): improve [InfluxDB ingestion](https://docs.victoriametrics.com/victoriametrics/integrations/influxdb/) parsing error message when a closing quote is missing for a quoted field value, by adding a hint that this may be caused by a raw newline (`\n`) inside the quoted field value. See [#10067](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10067). Thanks to @hklhai for the contribution.
 
 * BUGFIX: [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and `vmstorage` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): prevent panic `error parsing regexp: expression nests too deeply` triggered by large repetition ranges in regex, for example `{"__name__"=~"a{0,1000}"}`. See [VictoriaLogs#1112](https://github.com/VictoriaMetrics/VictoriaLogs/issues/1112).
 * BUGFIX: [vmui](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#vmui): fix escaping for label names with special characters. See [#10485](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10485).

docs/victoriametrics/integrations/influxdb.md

@@ -113,6 +113,56 @@ curl -d 'measurement,tag1=value1,tag2=value2 field1=123,field2=1.23' -X POST 'ht
 ```
 
 An arbitrary number of lines delimited by '\n' (aka newline char) can be sent in a single request.
+> **Note:** The above refers to newline characters (`\n`) used as line separators
+> between multiple points. According to the [Influx Line Protocol specification](https://docs.influxdata.com/influxdb/v2/reference/syntax/line-protocol/),
+> raw newline bytes **must not** appear inside quoted tag or field values. For example:
+>
+> ```text
+> SystemProperties.line.separator="
+> "
+> ```
+>
+> A raw newline inside a quoted value will result in a parsing error.
+>
+> If metrics are generated by Telegraf and written to VictoriaMetrics, fields containing raw newline characters
+> (such as `SystemProperties.line.separator`) can be pre-processed using a `regex` processor to escape newline
+> characters before serialization. For example:
+
+<details>
+<summary><strong>telegraf.conf</strong></summary>
+
+```toml
+[agent]
+  interval = "60s"
+  flush_interval = "30s"
+  debug = true
+
+[[inputs.jolokia2_agent]]
+  urls = ["http://springboot:8080/jolokia"]
+  name_prefix = "springBoot."
+
+  [[inputs.jolokia2_agent.metric]]
+    name = "JavaRuntime"
+    mbean = "java.lang:type=Runtime"
+    paths = ["SystemProperties"]
+
+[[processors.regex]]
+  namepass = ["springBoot.JavaRuntime"]
+
+  [[processors.regex.fields]]
+    key = "SystemProperties.line.separator"
+    pattern = '\n'
+    replacement = '\\n'
+
+[[outputs.influxdb]]
+  urls = ["http://victoriametrics:8428"]
+  skip_database_creation = true
+```
+
+</details>
+
+> This ensures the newline is properly escaped and the metric can be ingested successfully.
+
 After that the data may be read via [/api/v1/export](https://docs.victoriametrics.com/victoriametrics/#how-to-export-data-in-json-line-format) endpoint:
 ```sh
 curl -G 'http://<victoriametrics-addr>:8428/api/v1/export' -d 'match={__name__=~"measurement_.*"}'

lib/protoparser/influx/parser.go

@@ -358,7 +358,11 @@ func parseFieldValue(s string, uc *unmarshalContext) (float64, error) {
 	}
 	if uc.hasQuotedFields && s[0] == '"' {
 		if len(s) < 2 || s[len(s)-1] != '"' {
-			return 0, fmt.Errorf("missing closing quote for quoted field value %s", s)
+			// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10067
+			return 0, fmt.Errorf("missing closing quote for quoted field value %s; "+
+				"this may be caused by a raw newline (`\\n`) inside the quoted field value",
+				s,
+			)
 		}
 		// Try converting quoted string to number, since sometimes InfluxDB agents
 		// send numbers as strings.

lib/protoparser/influx/parser_test.go

@@ -2,6 +2,7 @@ package influx
 
 import (
 	"reflect"
+	"strings"
 	"testing"
 )
 
@@ -140,6 +141,25 @@ func TestRowsUnmarshalFailure(t *testing.T) {
 	f("GET /foo?bar=baz HTTP/1.0")
 }
 
+func TestParseFieldValue_MissingClosingQuoteWithRawNewlineHint(t *testing.T) {
+	uc := &unmarshalContext{
+		hasQuotedFields: true,
+	}
+
+	// Simulate the truncated value that happens
+	// after line splitting on raw newline
+	input := "\"hello"
+
+	_, err := parseFieldValue(input, uc)
+	if err == nil {
+		t.Fatalf("expected error for missing closing quote")
+	}
+
+	if !strings.Contains(err.Error(), "this may be caused by a raw newline") {
+		t.Fatalf("unexpected error message: %s", err)
+	}
+}
+
 func TestRowsUnmarshalSuccess(t *testing.T) {
 	f := func(s string, rowsExpected *Rows) {
 		t.Helper()