VictoriaMetrics/docs/victorialogs/data-ingestion/syslog.md

weight, title, disableToc, menu, tags, aliases

weight	title	disableToc	menu	tags	aliases
10	Syslog setup	true	docs parent weight victorialogs-data-ingestion 10	logs	/victorialogs/data-ingestion/syslog.html

VictoriaLogs can accept logs in Syslog formats at the specified TCP and UDP addresses via -syslog.listenAddr.tcp and -syslog.listenAddr.udp command-line flags. The following syslog formats are supported:

• RFC3164 aka <PRI>MMM DD hh:mm:ss HOSTNAME APP-NAME[PROCID]: MESSAGE
• RFC5424 aka <PRI>1 TIMESTAMP HOSTNAME APP-NAME PROCID MSGID [STRUCTURED-DATA] MESSAGE

For example, the following command starts VictoriaLogs, which accepts logs in Syslog format at TCP port 514 on all the network interfaces:

./victoria-logs -syslog.listenAddr.tcp=:514

It may be needed to run VictoriaLogs under root user or to set CAP_NET_BIND_SERVICE option if syslog messages must be accepted at TCP port below 1024.

The following command starts VictoriaLogs, which accepts logs in Syslog format at TCP and UDP ports 514:

./victoria-logs -syslog.listenAddr.tcp=:514 -syslog.listenAddr.udp=:514

VictoriaLogs can accept logs from the following syslog collectors:

• Rsyslog. See these docs.
• Syslog-ng. See these docs.

Multiple logs in Syslog format can be ingested via a single TCP connection or via a single UDP packet - just put every log on a separate line and delimit them with \n char.

VictoriaLogs automatically extracts the following log fields from the received Syslog lines:

• _time - log timestamp. See also log timestamps
• _msg - the MESSAGE field from the supported syslog formats above
• hostname, app_name and proc_id - for unique identification of log streams. It is possible to change the list of fields for log streams - see these docs.
• priority, facility and severity - these fields are extracted from <PRI> field
• format - this field is set to either rfc3164 or rfc5424 depending on the format of the parsed syslog line
• msg_id - MSGID field from log line in RFC5424 format.

The [STRUCTURED-DATA] is parsed into fields with the SD-ID.param1, SD-ID.param2, ..., SD-ID.paramN names and the corresponding values according to the specification.

By default local timezone is used when parsing timestamps in rfc3164 lines. This can be changed to any desired timezone via -syslog.timezone command-line flag. See the list of supported timezone identifiers. For example, the following command starts VictoriaLogs, which parses syslog timestamps in rfc3164 using Europe/Berlin timezone:

./victoria-logs -syslog.listenAddr.tcp=:514 -syslog.timezone='Europe/Berlin'

The ingested logs can be queried via logs querying API. For example, the following command returns ingested logs for the last 5 minutes by using time filter:

curl http://localhost:9428/select/logsql/query -d 'query=_time:5m'

See also:

• Log timestamps
• Security
• Compression
• Multitenancy
• Stream fields
• Dropping fields
• Decolorizing fields
• Adding extra fields
• Data ingestion troubleshooting.
• How to query VictoriaLogs.

Log timestamps

By default VictoriaLogs uses the timestamp from the parsed Syslog message as _time field. Sometimes the ingested Syslog messages may contain incorrect timestamps (for example, timestamps with incorrect timezone). In this case VictoriaLogs can be configured for using the log ingestion timestamp as _time field. This can be done by specifying -syslog.useLocalTimestamp.tcp command-line flag for the corresponding -syslog.listenAddr.tcp address:

./victoria-logs -syslog.listenAddr.tcp=:514 -syslog.useLocalTimestamp.tcp

In this case the original timestamp from the Syslog message is stored in timestamp log field.

The -syslog.useLocalTimestamp.udp command-line flag can be used for instructing VictoriaLogs to use local timestamps for the ingested logs via the corresponding -syslog.listenAddr.udp address:

./victoria-logs -syslog.listenAddr.udp=:514 -syslog.useLocalTimestamp.udp

Security

By default VictoriaLogs accepts plaintext data at -syslog.listenAddr.tcp address. Run VictoriaLogs with -syslog.tls command-line flag in order to accept TLS-encrypted logs at -syslog.listenAddr.tcp address. The -syslog.tlsCertFile and -syslog.tlsKeyFile command-line flags must be set to paths to TLS certificate file and TLS key file if -syslog.tls is set. For example, the following command starts VictoriaLogs, which accepts TLS-encrypted syslog messages at TCP port 6514:

./victoria-logs -syslog.listenAddr.tcp=:6514 -syslog.tls -syslog.tlsCertFile=/path/to/tls/cert -syslog.tlsKeyFile=/path/to/tls/key

Compression

By default VictoriaLogs accepts uncompressed log messages in Syslog format at -syslog.listenAddr.tcp and -syslog.listenAddr.udp addresses. It is possible configuring VictoriaLogs to accept compressed log messages via -syslog.compressMethod.tcp and -syslog.compressMethod.udp command-line flags. The following compression methods are supported:

• none - no compression
• zstd - zstd compression
• gzip - gzip compression
• deflate - deflate compression

For example, the following command starts VictoriaLogs, which accepts gzip-compressed syslog messages at TCP port 514:

./victoria-logs -syslog.listenAddr.tcp=:514 -syslog.compressMethod.tcp=gzip

Multitenancy

By default, the ingested logs are stored in the (AccountID=0, ProjectID=0) tenant. If you need storing logs in other tenant, then specify the needed tenant via -syslog.tenantID.tcp or -syslog.tenantID.udp command-line flags depending on whether TCP or UDP ports are listened for syslog messages. For example, the following command starts VictoriaLogs, which writes syslog messages received at TCP port 514, to (AccountID=12, ProjectID=34) tenant:

./victoria-logs -syslog.listenAddr.tcp=:514 -syslog.tenantID.tcp=12:34

Stream fields

VictoriaLogs uses (hostname, app_name, proc_id) fields as labels for log streams by default. It is possible setting other set of labels via -syslog.streamFields.tcp and -syslog.streamFields.udp command-line flags for logs instead via the corresponding -syslog.listenAddr.tcp and -syslog.listenAddr.dup addresses. For example, the following command starts VictoriaLogs, which uses (hostname, app_name) fields as log stream labels for logs received at TCP port 514:

./victoria-logs -syslog.listenAddr.tcp=:514 -syslog.streamFields.tcp='["hostname","app_name"]'

Dropping fields

VictoriaLogs supports -syslog.ignoreFields.tcp and -syslog.ignoreFields.udp command-line flags for skipping the given log fields during ingestion of Syslog logs into -syslog.listenAddr.tcp and -syslog.listenAddr.udp addresses. For example, the following command starts VictoriaLogs, which drops proc_id and msg_id fields from logs received at TCP port 514:

./victoria-logs -syslog.listenAddr.tcp=:514 -syslog.ignoreFields.tcp='["prod_id","msg_id"]'

The list may contain field name prefixes ending with * such as some-prefix*. In this case all the log fields starting with this prefix are ignored during data ingestion.

Decolorizing fields

VictoriaLogs supports -syslog.decolorizeFields.tcp and -syslog.decolorizeFields.udp command-line flags, which can be used for removing ANSI color codes from the provided list fields during ingestion of Syslog logs into -syslog.listenAddr.tcp and -syslog.listenAddr.upd addresses. For example, the following command starts VictoriaLogs, which removes ANSI color codes from _msg field at logs received via TCP port 514:

./victoria-logs -syslog.listenAddr.tcp=:514 -syslog.decolorizeFields.tcp='["_msg"]'

Adding extra fields

VictoriaLogs supports -syslog.extraFields.tcp and -syslog.extraFields.udp command-line flags for adding the given log fields during data ingestion of Syslog logs into -syslog.listenAddr.tcp and -syslog.listenAddr.udp addresses. For example, the following command starts VictoriaLogs, which adds source=foo and abc=def fields to logs received at TCP port 514:

./victoria-logs -syslog.listenAddr.tcp=:514 -syslog.extraFields.tcp='{"source":"foo","abc":"def"}'

Multiple configs

VictoriaLogs can accept syslog messages via multiple TCP and UDP ports with individual configurations for log timestamps, compression, security and multitenancy. Specify multiple command-line flags for this. For example, the following command starts VictoriaLogs, which accepts gzip-compressed syslog messages via TCP port 514 at localhost interface and stores them to tenant 123:0, plus it accepts TLS-encrypted syslog messages via TCP port 6514 and stores them to tenant 567:0:

./victoria-logs \
  -syslog.listenAddr.tcp=localhost:514 -syslog.tenantID.tcp=123:0 -syslog.compressMethod.tcp=gzip -syslog.tls=false -syslog.tlsKeyFile='' -syslog.tlsCertFile='' \
  -syslog.listenAddr.tcp=:6514 -syslog.tenantID.tcp=567:0 -syslog.compressMethod.tcp=none -syslog.tls=true -syslog.tlsKeyFile=/path/to/tls/key -syslog.tlsCertFile=/path/to/tls/cert

Rsyslog

1. Run VictoriaLogs with -syslog.listenAddr.tcp=:29514 command-line flag.
2. Put the following line to rsyslog config (this config is usually located at /etc/rsyslog.conf):

   *.* @@victoria-logs-server:29514
   

   Where victoria-logs-server is the hostname where VictoriaLogs runs. See these docs for more details.

Syslog-ng

1. Run VictoriaLogs with -syslog.listenAddr.tcp=:29514 command-line flag.
2. Put the following line to syslog-ng config:

   destination d_remote {
    tcp("victoria-logs-server" port(29514));
   };
   

   Where victoria-logs-server is the hostname where VictoriaLogs runs. See these docs for details.