|
|
|
|
@@ -1712,63 +1712,64 @@ The following versions of VictoriaMetrics receive regular security fixes:
|
|
|
|
|
| [LTS releases](https://docs.victoriametrics.com/victoriametrics/lts-releases/) | ✅ |
|
|
|
|
|
| other releases | ❌ |
|
|
|
|
|
|
|
|
|
|
### Software Bill of Materials (SBOM)
|
|
|
|
|
|
|
|
|
|
Every VictoriaMetrics container{{% available_from "v1.137.0" %}} image published to
|
|
|
|
|
[Docker Hub](https://hub.docker.com/u/victoriametrics) and [Quay.io](https://quay.io/organization/victoriametrics) include an [SPDX](https://spdx.dev/) SBOM attestation generated automatically by BuildKit during `docker buildx build`.
|
|
|
|
|
|
|
|
|
|
To inspect the SBOM for an image:
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
docker buildx imagetools inspect \
|
|
|
|
|
docker.io/victoriametrics/victoria-metrics:latest \
|
|
|
|
|
--format "{{ json .SBOM }}"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
To scan an image using its SBOM attestation with [Trivy](https://github.com/aquasecurity/trivy):
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
trivy image --sbom-sources oci \
|
|
|
|
|
docker.io/victoriametrics/victoria-metrics:latest
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
### Reporting a Vulnerability
|
|
|
|
|
|
|
|
|
|
Please report any security issues to <security@victoriametrics.com>
|
|
|
|
|
|
|
|
|
|
### CVE handling policy
|
|
|
|
|
|
|
|
|
|
**Source code:** Go dependencies are scanned by [govulncheck](https://pkg.go.dev/golang.org/x/vuln/cmd/govulncheck) in CI.
|
|
|
|
|
All vulnerabilities must be fixed before the next scheduled release and backported to [LTS releases](https://docs.victoriametrics.com/victoriametrics/lts-releases/).
|
|
|
|
|
|
|
|
|
|
**Docker images:** CVE findings in the [Alpine](https://security.alpinelinux.org/) base image pose minimal risk since VictoriaMetrics binaries are statically compiled with no OS dependencies.
|
|
|
|
|
When detected, only the Alpine base tag is updated.
|
|
|
|
|
Releases proceed as planned even if upstream fixes are not yet available.
|
|
|
|
|
For maximum security, hardened [scratch](https://hub.docker.com/_/scratch)-based images are also provided.
|
|
|
|
|
All images are continuously scanned by Docker Hub and verified before release using [grype](https://github.com/anchore/grype).
|
|
|
|
|
|
|
|
|
|
### General security recommendations:
|
|
|
|
|
|
|
|
|
|
* All VictoriaMetrics components must run in protected private networks without direct access from untrusted networks such as the Internet.
|
|
|
|
|
* All the VictoriaMetrics components must run in protected private networks without direct access from untrusted networks such as Internet.
|
|
|
|
|
The exception is [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/) and [vmgateway](https://docs.victoriametrics.com/victoriametrics/vmgateway/),
|
|
|
|
|
which are intended for serving public requests and performing authorization with [TLS termination](https://en.wikipedia.org/wiki/TLS_termination_proxy).
|
|
|
|
|
* All the requests from untrusted networks to VictoriaMetrics components must go through an auth proxy, such as [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/)
|
|
|
|
|
* All the requests from untrusted networks to VictoriaMetrics components must go through auth proxy such as [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/)
|
|
|
|
|
or [vmgateway](https://docs.victoriametrics.com/victoriametrics/vmgateway/). The proxy must be set up with proper authentication and authorization.
|
|
|
|
|
* Prefer using lists of allowed API endpoints, while disallowing access to other endpoints when configuring [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/)
|
|
|
|
|
in front of VictoriaMetrics components.
|
|
|
|
|
* Set a reasonable [`Strict-Transport-Security`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security) header value on all the components to mitigate [MitM attacks](https://en.wikipedia.org/wiki/Man-in-the-middle_attack), for example: `max-age=31536000; includeSubDomains`. See `-http.header.hsts` flag.
|
|
|
|
|
* Set reasonable [`Strict-Transport-Security`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security) header value to all the components to mitigate [MitM attacks](https://en.wikipedia.org/wiki/Man-in-the-middle_attack), for example: `max-age=31536000; includeSubDomains`. See `-http.header.hsts` flag.
|
|
|
|
|
* Set reasonable [`Content-Security-Policy`](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP) header value to mitigate [XSS attacks](https://en.wikipedia.org/wiki/Cross-site_scripting). See `-http.header.csp` flag.
|
|
|
|
|
* Set reasonable [`X-Frame-Options`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options) header value to mitigate [clickjacking attacks](https://en.wikipedia.org/wiki/Clickjacking), for example `DENY`. See `-http.header.frameOptions` flag.
|
|
|
|
|
|
|
|
|
|
The following security-related command-line flags are available for all components with HTTP API:
|
|
|
|
|
VictoriaMetrics provides the following security-related command-line flags:
|
|
|
|
|
|
|
|
|
|
* `-tls`, `-tlsCertFile` and `-tlsKeyFile` for switching from HTTP to HTTPS at `-httpListenAddr`.
|
|
|
|
|
* `-tls`, `-tlsCertFile` and `-tlsKeyFile` for switching from HTTP to HTTPS at `-httpListenAddr` (TCP port 8428 is listened by default).
|
|
|
|
|
[Enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/enterprise/) supports automatic issuing of TLS certificates.
|
|
|
|
|
See [these docs](#automatic-issuing-of-tls-certificates).
|
|
|
|
|
* `-mtls` and `-mtlsCAFile` for enabling [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication) for requests to `-httpListenAddr`. See [these docs](#mtls-protection).
|
|
|
|
|
* `-httpAuth.username` and `-httpAuth.password` for protecting all the HTTP endpoints
|
|
|
|
|
with [HTTP Basic Authentication](https://en.wikipedia.org/wiki/Basic_access_authentication).
|
|
|
|
|
* `-http.header.hsts`, `-http.header.csp`, and `-http.header.frameOptions` for serving `Strict-Transport-Security`, `Content-Security-Policy`
|
|
|
|
|
and `X-Frame-Options` HTTP response headers.
|
|
|
|
|
|
|
|
|
|
### Protecting service endpoints
|
|
|
|
|
|
|
|
|
|
All VictoriaMetrics components expose internal metrics in Prometheus exposition format at the `/metrics` page for [#Monitoring](https://docs.victoriametrics.com/victoriametrics/#monitoring).
|
|
|
|
|
Consider limiting access to the `/metrics` page to trusted networks only.
|
|
|
|
|
|
|
|
|
|
The following service endpoints may require protection:
|
|
|
|
|
|
|
|
|
|
* `-deleteAuthKey` for protecting the `/api/v1/admin/tsdb/delete_series` endpoint. See [how to delete time series](#how-to-delete-time-series).
|
|
|
|
|
* `-snapshotAuthKey` for protecting the `/snapshot*` endpoints. See [how to work with snapshots](#how-to-work-with-snapshots).
|
|
|
|
|
* `-forceFlushAuthKey` for protecting the `/internal/force_flush` endpoint. See [force flush docs](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#forced-flush).
|
|
|
|
|
* `-forceMergeAuthKey` for protecting the `/internal/force_merge` endpoint. See [force merge docs](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#forced-merge).
|
|
|
|
|
* `-search.resetCacheAuthKey` for protecting the `/internal/resetRollupResultCache` endpoint. See [backfilling](#backfilling) for more details.
|
|
|
|
|
* `-reloadAuthKey` for protecting the `/-/reload` endpoint, which is used to force reload the [`-promscrape.config`](#how-to-scrape-prometheus-exporters-such-as-node-exporter).
|
|
|
|
|
* `-reloadAuthKey` for protecting the `/-/reload` endpoint, which is used for force reloading of [`-promscrape.config`](#how-to-scrape-prometheus-exporters-such-as-node-exporter).
|
|
|
|
|
* `-configAuthKey` for protecting the `/config` endpoint, since it may contain sensitive information such as passwords.
|
|
|
|
|
* `-flagsAuthKey` for protecting the `/flags` endpoint.
|
|
|
|
|
* `-pprofAuthKey` for protecting the `/debug/pprof/*` endpoints, which can be used for [profiling](#profiling).
|
|
|
|
|
* `-metricNamesStatsResetAuthKey` for protecting the `/api/v1/admin/status/metric_names_stats/reset` endpoint, used for [Metric Names Tracker](#track-ingested-metrics-usage).
|
|
|
|
|
* `-denyQueryTracing` for disallowing [query tracing](#query-tracing).
|
|
|
|
|
* `-http.header.hsts`, `-http.header.csp`, and `-http.header.frameOptions` for serving `Strict-Transport-Security`, `Content-Security-Policy`
|
|
|
|
|
and `X-Frame-Options` HTTP response headers.
|
|
|
|
|
|
|
|
|
|
Explicitly set internal network interface for TCP and UDP ports for data ingestion with Graphite and OpenTSDB formats.
|
|
|
|
|
For example, substitute `-graphiteListenAddr=:2003` with `-graphiteListenAddr=<internal_iface_ip>:2003`. This protects from unexpected requests from untrusted network interfaces.
|
|
|
|
|
@@ -1776,6 +1777,17 @@ For example, substitute `-graphiteListenAddr=:2003` with `-graphiteListenAddr=<i
|
|
|
|
|
See also [security recommendation for VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#security)
|
|
|
|
|
and [the general security page at VictoriaMetrics website](https://victoriametrics.com/security/).
|
|
|
|
|
|
|
|
|
|
### CVE handling policy
|
|
|
|
|
|
|
|
|
|
**Source code:** Go dependencies are scanned by [govulncheck](https://pkg.go.dev/golang.org/x/vuln/cmd/govulncheck) in CI.
|
|
|
|
|
All vulnerabilities must be fixed before next scheduled release and backported to [LTS releases](https://docs.victoriametrics.com/victoriametrics/lts-releases/).
|
|
|
|
|
|
|
|
|
|
**Docker images:** CVE findings in [Alpine](https://security.alpinelinux.org/) base image pose minimal risk since VictoriaMetrics binaries are statically compiled with no OS dependencies.
|
|
|
|
|
When detected, only the Alpine base tag is updated.
|
|
|
|
|
Releases proceed as planned even if upstream fixes are not yet available.
|
|
|
|
|
For maximum security, hardened [scratch](https://hub.docker.com/_/scratch)-based images are also provided.
|
|
|
|
|
All images are continuously scanned by Docker Hub and verified before release using [grype](https://github.com/anchore/grype).
|
|
|
|
|
|
|
|
|
|
### mTLS protection
|
|
|
|
|
|
|
|
|
|
By default `VictoriaMetrics` accepts http requests at `8428` port (this port can be changed via `-httpListenAddr` command-line flags).
|
|
|
|
|
@@ -1805,39 +1817,19 @@ This functionality can be evaluated for free according to [these docs](https://d
|
|
|
|
|
|
|
|
|
|
See also [security recommendations](#security).
|
|
|
|
|
|
|
|
|
|
### Software Bill of Materials (SBOM)
|
|
|
|
|
|
|
|
|
|
Every VictoriaMetrics container{{% available_from "v1.137.0" %}} image published to
|
|
|
|
|
[Docker Hub](https://hub.docker.com/u/victoriametrics) and [Quay.io](https://quay.io/organization/victoriametrics) include an [SPDX](https://spdx.dev/) SBOM attestation generated automatically by BuildKit during `docker buildx build`.
|
|
|
|
|
|
|
|
|
|
To inspect the SBOM for an image:
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
docker buildx imagetools inspect \
|
|
|
|
|
docker.io/victoriametrics/victoria-metrics:latest \
|
|
|
|
|
--format "{{ json .SBOM }}"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
To scan an image using its SBOM attestation with [Trivy](https://github.com/aquasecurity/trivy):
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
trivy image --sbom-sources oci \
|
|
|
|
|
docker.io/victoriametrics/victoria-metrics:latest
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## Tuning
|
|
|
|
|
|
|
|
|
|
* No need to tune for VictoriaMetrics - it uses reasonable defaults for command-line flags,
|
|
|
|
|
* No need in tuning for VictoriaMetrics - it uses reasonable defaults for command-line flags,
|
|
|
|
|
which are automatically adjusted for the available CPU and RAM resources.
|
|
|
|
|
* No need to tune for Operating System - VictoriaMetrics is optimized for default OS settings.
|
|
|
|
|
* No need in tuning for Operating System - VictoriaMetrics is optimized for default OS settings.
|
|
|
|
|
The only option is increasing the limit on [the number of open files in the OS](https://medium.com/@muhammadtriwibowo/set-permanently-ulimit-n-open-files-in-ubuntu-4d61064429a).
|
|
|
|
|
The recommendation is not specific to VictoriaMetrics only, but also for any service that handles many HTTP connections and stores data on disk.
|
|
|
|
|
* VictoriaMetrics is a write-heavy application, and its performance depends on disk performance. So be careful with other
|
|
|
|
|
The recommendation is not specific for VictoriaMetrics only but also for any service which handles many HTTP connections and stores data on disk.
|
|
|
|
|
* VictoriaMetrics is a write-heavy application and its performance depends on disk performance. So be careful with other
|
|
|
|
|
applications or utilities (like [fstrim](https://manpages.ubuntu.com/manpages/lunar/en/man8/fstrim.8.html))
|
|
|
|
|
which could [exhaust disk resources](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1521).
|
|
|
|
|
* The recommended filesystem is `ext4`, the recommended persistent storage is [persistent HDD-based disk on GCP](https://cloud.google.com/compute/docs/disks/#pdspecs),
|
|
|
|
|
since it is protected from hardware failures via internal replication, and it can be [resized on the fly](https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd).
|
|
|
|
|
If you plan to store more than 1TB of data on an `ext4` partition, then the following options are recommended to pass to `mkfs.ext4`:
|
|
|
|
|
since it is protected from hardware failures via internal replication and it can be [resized on the fly](https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd).
|
|
|
|
|
If you plan to store more than 1TB of data on `ext4` partition, then the following options are recommended to pass to `mkfs.ext4`:
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
mkfs.ext4 ... -O 64bit,huge_file,extent -T huge
|
|
|
|
|
|
