Compare commits

..

18 Commits

Author SHA1 Message Date
Zakhar Bessarab
1272a7f743 app/vminsert/netstorage: refactor snb rebuild
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-07-29 17:23:22 +04:00
Zakhar Bessarab
2b39ee785c app/vmselect: send static empty node ID for multi-level setup
Multi-level vmselect setup is not intended to use storage node IDs, so it is safe to return 0 here.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-07-29 12:31:23 +04:00
Zakhar Bessarab
842bf78cb1 app/vminsert/netstorage: sync comment
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-07-29 12:31:23 +04:00
Zakhar Bessarab
5420989018 app/vminsert/netstorage: reinitialize snb on vmstorage connection restore
It is needed to rebuild snb in order to ensure that list of storage nodes and consistent hash are in sync.
Updating just consistent hash ring is not safe because it can cause misalignment of indexes of alive nodes in snb.sns and hash slots.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-07-29 12:31:23 +04:00
Zakhar Bessarab
9ff8b312bb app/vminsert/netstorage: use correct snb reference
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-07-29 12:31:22 +04:00
Zakhar Bessarab
130b9cd04e app/vminsert/netstorage: make linter happy
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-07-29 12:31:22 +04:00
Zakhar Bessarab
88bfad9535 app/vminsert/netstorage: exclude unavailable nodes from consistent hash on start
Exclude unhealthy storage nodes from consistent hash in case persistent storage node IDs are enabled.
This is needed in order to avoid uneven distribution of load due to default(uint64(0)) IDs assigned to storage nodes.

Remove generating fallback ID from node IP address as this will cause a re-distribution of series once storage node will become available and will change its ID.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-07-29 12:31:22 +04:00
Zakhar Bessarab
e44c6f38c2 app/vminsert/netstorage: print storage node IDs in logs
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-07-29 12:31:22 +04:00
Zakhar Bessarab
96a62a275a lib/handshake: use a json payload for metadata exchange
Update the handshake to use an arbitrary JSON payload to transfer metadata.
Handshake sends the metadata length first as an uint64 and then the metadata itself.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-07-29 12:31:22 +04:00
Zakhar Bessarab
20b9c8007b lib/storage: print node ID in startup log
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-07-29 12:31:22 +04:00
Zakhar Bessarab
3df456dd35 lib/storage: don't save persistent node ID on shutdown
It is supposed to be saved right after it was generated, there is no reason re-write it again.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-07-29 12:31:22 +04:00
Zakhar Bessarab
7402ee0801 lib/storage: fallback to address-based ID
Generate an ID based on storage node address if storage node is not available.
This is needed in order to prevent uneven load distribution if some storage nodes are not available when vminsert is starting.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-07-29 12:31:22 +04:00
Zakhar Bessarab
5ac1e77520 lib/storage: save storage ID after init
This helps to avoid re-creating a storage ID in case of unclean shutdown.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-07-29 12:31:22 +04:00
Zakhar Bessarab
71e729f3f8 app/vminsert: disable usage of persistent storage node ID by default
This is needed in order to avoid complete data re-sharding after the upgrade to a new version.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-07-29 12:31:22 +04:00
Zakhar Bessarab
8729ec174b docs/changelog: add info about persisting vmstorage node ID
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-07-29 12:31:18 +04:00
Zakhar Bessarab
84184b707a app/cluster: communicate node IDs when performing a handshake
Send a node ID of vmstorage as a part of vmselect and vminsert handshakes.
Use vmstorage node ID as an identifier for consistent hashing at vminsert.

Cluster native endpoints calculate vminsert and vmselect node IDs as a hash of all underlying storage node IDs, so that it will also remain consistent in case of address changes.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-07-29 12:29:20 +04:00
Zakhar Bessarab
41e217423f lib/storage: store node ID in metadata so that it is included in the backups
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-07-29 12:29:20 +04:00
Zakhar Bessarab
8d8073a24d lib/storage: add storage node id
Generate random node ID on start if it is missing or load from disk. Save to storage on storage shutdown.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-07-29 12:29:19 +04:00
177 changed files with 5874 additions and 872 deletions

1
.gitignore vendored
View File

@@ -7,7 +7,6 @@
.vscode
*.test
*.swp
/vmdocs
/gocache-for-docker
/victoria-logs-data
/victoria-metrics-data

View File

@@ -253,3 +253,34 @@ install-wwhrd:
check-licenses: install-wwhrd
wwhrd check -f .wwhrd.yml
copy-docs:
# The 'printf' function is used instead of 'echo' or 'echo -e' to handle line breaks (e.g. '\n') in the same way on different operating systems (MacOS/Ubuntu Linux/Arch Linux) and their shells (bash/sh/zsh/fish).
# For details, see https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4548#issue-1782796419 and https://stackoverflow.com/questions/8467424/echo-newline-in-bash-prints-literal-n
echo "---" > ${DST}
@if [ ${ORDER} -ne 0 ]; then \
echo "sort: ${ORDER}" >> ${DST}; \
echo "weight: ${ORDER}" >> ${DST}; \
printf "menu:\n docs:\n parent: 'victoriametrics'\n weight: ${ORDER}\n" >> ${DST}; \
fi
echo "title: ${TITLE}" >> ${DST}
@if [ ${OLD_URL} ]; then \
printf "aliases:\n - ${OLD_URL}\n" >> ${DST}; \
fi
echo "---" >> ${DST}
cat ${SRC} >> ${DST}
sed -i='.tmp' 's/<img src=\"docs\//<img src=\"\//' ${DST}
sed -i='.tmp' 's/<source srcset=\"docs\//<source srcset=\"\//' ${DST}
sed -i='.tmp' 's/](docs\//](/' ${DST}
rm -rf docs/*.tmp
# Copies docs for all components and adds the order/weight tag, title, menu position and alias with the backward compatible link for the old site.
# For ORDER=0 it adds no order tag/weight tag.
# FOR OLD_URL - relative link, used for backward compatibility with the link from documentation based on GitHub pages (old one)
# FOR OLD_URL='' it adds no alias, it should be empty for every new page, don't change it for already existing links.
# Images starting with <img src="docs/ are replaced with <img src="
# Cluster docs are supposed to be ordered as 2nd.
# The rest of docs is ordered manually.
docs-sync:
SRC=README.md DST=docs/Cluster-VictoriaMetrics.md OLD_URL='/Cluster-VictoriaMetrics.html' ORDER=2 TITLE='Cluster version' $(MAKE) copy-docs

1942
README.md

File diff suppressed because it is too large Load Diff

View File

@@ -5,6 +5,8 @@ import (
"fmt"
"net"
"github.com/VictoriaMetrics/metrics"
"github.com/VictoriaMetrics/VictoriaMetrics/app/vminsert/netstorage"
"github.com/VictoriaMetrics/VictoriaMetrics/app/vminsert/relabel"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/auth"
@@ -12,7 +14,6 @@ import (
"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/clusternative/stream"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/storage"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/tenantmetrics"
"github.com/VictoriaMetrics/metrics"
)
var (
@@ -25,7 +26,7 @@ var (
func InsertHandler(c net.Conn) error {
// There is no need in response compression, since
// lower-level vminsert sends only small packets to upper-level vminsert.
bc, err := handshake.VMInsertServer(c, 0)
bc, err := handshake.VMInsertServer(c, 0, netstorage.GetNodeID())
if err != nil {
if errors.Is(err, handshake.ErrIgnoreHealthcheck) {
return nil

View File

@@ -1,9 +1,5 @@
package netstorage
import (
"github.com/cespare/xxhash/v2"
)
// See the following docs:
// - https://www.eecs.umich.edu/techreports/cse/96/CSE-TR-316-96.pdf
// - https://github.com/dgryski/go-rendezvous
@@ -13,14 +9,10 @@ type consistentHash struct {
nodeHashes []uint64
}
func newConsistentHash(nodes []string, hashSeed uint64) *consistentHash {
nodeHashes := make([]uint64, len(nodes))
for i, node := range nodes {
nodeHashes[i] = xxhash.Sum64([]byte(node))
}
func newConsistentHash(ids []uint64, hashSeed uint64) *consistentHash {
return &consistentHash{
hashSeed: hashSeed,
nodeHashes: nodeHashes,
nodeHashes: ids,
}
}

View File

@@ -4,16 +4,18 @@ import (
"math"
"math/rand"
"testing"
"github.com/cespare/xxhash/v2"
)
func TestConsistentHash(t *testing.T) {
r := rand.New(rand.NewSource(1))
nodes := []string{
"node1",
"node2",
"node3",
"node4",
nodes := []uint64{
xxhash.Sum64String("node1"),
xxhash.Sum64String("node2"),
xxhash.Sum64String("node3"),
xxhash.Sum64String("node4"),
}
rh := newConsistentHash(nodes, 0)

View File

@@ -4,16 +4,19 @@ import (
"math/rand"
"sync/atomic"
"testing"
"github.com/cespare/xxhash/v2"
)
func BenchmarkConsistentHash(b *testing.B) {
nodes := []string{
"node1",
"node2",
"node3",
"node4",
nodes := []uint64{
xxhash.Sum64String("node1"),
xxhash.Sum64String("node2"),
xxhash.Sum64String("node3"),
xxhash.Sum64String("node4"),
}
rh := newConsistentHash(nodes, 0)
b.ReportAllocs()
b.SetBytes(int64(len(benchKeys)))
b.RunParallel(func(pb *testing.PB) {

View File

@@ -5,6 +5,8 @@ import (
"net/http"
"strconv"
"github.com/cespare/xxhash/v2"
"github.com/VictoriaMetrics/VictoriaMetrics/app/vminsert/relabel"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/auth"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
@@ -12,7 +14,6 @@ import (
"github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompb"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/storage"
"github.com/cespare/xxhash/v2"
)
// InsertCtx is a generic context for inserting data.

View File

@@ -6,10 +6,14 @@ import (
"fmt"
"io"
"net"
"slices"
"sync"
"sync/atomic"
"time"
"github.com/VictoriaMetrics/metrics"
"github.com/cespare/xxhash/v2"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/consts"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/encoding"
@@ -21,8 +25,6 @@ import (
"github.com/VictoriaMetrics/VictoriaMetrics/lib/storage"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/timerpool"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/timeutil"
"github.com/VictoriaMetrics/metrics"
"github.com/cespare/xxhash/v2"
)
var (
@@ -43,6 +45,9 @@ var (
"On the other side, disabled re-routing minimizes the number of active time series in the cluster "+
"during rolling restarts and during spikes in series churn rate. "+
"See also -disableRerouting")
usePersistentStorageNodeID = flag.Bool("vmstorageUsePersistentID", false, "Whether to use persistent storage node ID for -storageNode instances. "+
"If set to false uses storage node address in order to generate an ID. "+
"Using persistent node ID is useful if vmstorage node address changes over time, e.g. due to dynamic IP addresses or DNS names. ")
)
var errStorageReadOnly = errors.New("storage node is read only")
@@ -278,7 +283,7 @@ func (sn *storageNode) checkHealth() {
}
return
}
logger.Infof("successfully dialed -storageNode=%q", sn.dialer.Addr())
logger.Infof("successfully dialed -storageNode=%q (node ID: %d)", sn.dialer.Addr(), sn.id.Load())
sn.lastDialErr = nil
sn.bc = bc
sn.isBroken.Store(false)
@@ -398,15 +403,24 @@ func (sn *storageNode) dial() (*handshake.BufferedConn, error) {
if *disableRPCCompression {
compressionLevel = 0
}
bc, err := handshake.VMInsertClient(c, compressionLevel)
bc, id, err := handshake.VMInsertClient(c, compressionLevel)
if err != nil {
_ = c.Close()
sn.handshakeErrors.Inc()
return nil, fmt.Errorf("handshake error: %w", err)
}
sn.id.CompareAndSwap(0, id)
return bc, nil
}
func (sn *storageNode) getID() uint64 {
// Ensure that the id is populated
if sn.id.Load() == 0 {
sn.checkHealth()
}
return sn.id.Load()
}
// storageNode is a client sending data to vmstorage node.
type storageNode struct {
// isBroken is set to true if the given vmstorage node is temporarily unhealthy.
@@ -473,6 +487,9 @@ type storageNode struct {
// The total duration spent for sending data to vmstorage node.
// This metric is useful for determining the saturation of vminsert->vmstorage link.
sendDurationSeconds *metrics.FloatCounter
// id is a unique identifier for the storage node.
id atomic.Uint64
}
type storageNodesBucket struct {
@@ -515,14 +532,31 @@ func MustStop() {
mustStopStorageNodes(snb)
}
// GetNodeID returns unique identifier for underlying storage nodes.
func GetNodeID() uint64 {
snb := getStorageNodesBucket()
snIDs := make([]uint64, 0, len(snb.sns))
for _, sn := range snb.sns {
snIDs = append(snIDs, sn.getID())
}
slices.Sort(snIDs)
idsM := make([]byte, 0)
for _, id := range snIDs {
idsM = encoding.MarshalUint64(idsM, id)
}
return xxhash.Sum64(idsM)
}
func initStorageNodes(addrs []string, hashSeed uint64) *storageNodesBucket {
if len(addrs) == 0 {
logger.Panicf("BUG: addrs must be non-empty")
}
ms := metrics.NewSet()
nodesHash := newConsistentHash(addrs, hashSeed)
sns := make([]*storageNode, 0, len(addrs))
brokenNodes := make([]*storageNode, 0)
stopCh := make(chan struct{})
nodeIDs := make([]uint64, 0, len(addrs))
for _, addr := range addrs {
if _, _, err := net.SplitHostPort(addr); err != nil {
// Automatically add missing port.
@@ -568,10 +602,22 @@ func initStorageNodes(addrs []string, hashSeed uint64) *storageNodesBucket {
}
return 0
})
var nodeID uint64
if *usePersistentStorageNodeID {
nodeID = sn.getID()
if nodeID == 0 {
brokenNodes = append(brokenNodes, sn)
continue
}
} else {
nodeID = xxhash.Sum64String(addr)
}
nodeIDs = append(nodeIDs, nodeID)
sns = append(sns, sn)
}
nodesHash := newConsistentHash(nodeIDs, hashSeed)
maxBufSizePerStorageNode = memory.Allowed() / 8 / len(sns)
maxBufSizePerStorageNode = memory.Allowed() / 8 / len(addrs)
if maxBufSizePerStorageNode > consts.MaxInsertPacketSizeForVMInsert {
maxBufSizePerStorageNode = consts.MaxInsertPacketSizeForVMInsert
}
@@ -586,7 +632,12 @@ func initStorageNodes(addrs []string, hashSeed uint64) *storageNodesBucket {
wg: &wg,
}
for idx, sn := range sns {
// add broken nodes to the end of the list
// this is needed because consistent hash slots will be populated with IDs of available
// storage nodes (if there are any) and indexes of consistent hash must be linked to healthy storage nodes
snb.sns = append(snb.sns, brokenNodes...)
for idx, sn := range snb.sns {
wg.Add(1)
go func(sn *storageNode, idx int) {
sn.run(snb, idx)
@@ -594,6 +645,28 @@ func initStorageNodes(addrs []string, hashSeed uint64) *storageNodesBucket {
}(sn, idx)
}
// Watch for node become healthy and rebuild snb.
for _, sn := range brokenNodes {
wg.Add(1)
sn := sn
go watchStorageNodeHealthy(sn, func() {
defer wg.Done()
// rebuild snb in order to update consistent hash with an ID of the healthy storage node
for {
currentSnb := getStorageNodesBucket()
newSnb := initStorageNodes(addrs, hashSeed)
if !storageNodes.CompareAndSwap(currentSnb, newSnb) {
// snb has been changed, so we need to stop the newSnb and try again
mustStopStorageNodes(newSnb)
continue
}
// stop previous snb and exit
mustStopStorageNodes(currentSnb)
break
}
})
}
return snb
}
@@ -606,6 +679,34 @@ func mustStopStorageNodes(snb *storageNodesBucket) {
metrics.UnregisterSet(snb.ms, true)
}
// watchStorageNodeHealthy watches for sn become healthy and calls cb once it is ready.
func watchStorageNodeHealthy(sn *storageNode, cb func()) {
for {
sn.brLock.Lock()
for !sn.isReady() {
select {
case <-sn.stopCh:
sn.brLock.Unlock()
return
default:
sn.brCond.Wait()
}
}
sn.brLock.Unlock()
select {
case <-sn.stopCh:
return
default:
}
if sn.isReady() {
cb()
return
}
}
}
// rerouteRowsToReadyStorageNodes reroutes src from not ready snSource to ready storage nodes.
//
// The function blocks until src is fully re-routed.

View File

@@ -31,7 +31,9 @@ var (
// NewVMSelectServer starts new server at the given addr, which serves vmselect requests from netstorage.
func NewVMSelectServer(addr string) (*vmselectapi.Server, error) {
api := &vmstorageAPI{}
api := &vmstorageAPI{
nodeID: netstorage.GetNodeID(),
}
limits := vmselectapi.Limits{
MaxLabelNames: *maxTagKeys,
MaxLabelValues: *maxTagValues,
@@ -45,7 +47,9 @@ func NewVMSelectServer(addr string) (*vmselectapi.Server, error) {
}
// vmstorageAPI impelements vmselectapi.API
type vmstorageAPI struct{}
type vmstorageAPI struct {
nodeID uint64
}
func (api *vmstorageAPI) InitSearch(qt *querytracer.Tracer, sq *storage.SearchQuery, deadline uint64) (vmselectapi.BlockIterator, error) {
denyPartialResponse := httputils.GetDenyPartialResponse(nil)
@@ -112,6 +116,10 @@ func (api *vmstorageAPI) RegisterMetricNames(qt *querytracer.Tracer, mrs []stora
return netstorage.RegisterMetricNames(qt, mrs, dl)
}
func (api *vmstorageAPI) GetID() uint64 {
return api.nodeID
}
// blockIterator implements vmselectapi.BlockIterator
type blockIterator struct {
workCh chan workItem

View File

@@ -2107,6 +2107,9 @@ type storageNode struct {
// The number of list tenants errors to storageNode.
tenantsErrors *metrics.Counter
// id is the unique identifier for the storageNode.
id uint64
}
func (sn *storageNode) registerMetricNames(qt *querytracer.Tracer, mrs []storage.MetricRow, deadline searchutils.Deadline) error {
@@ -2954,6 +2957,12 @@ func getStorageNodes() []*storageNode {
return snb.sns
}
// GetNodeID returns unique identifier of vmselect
func GetNodeID() uint64 {
// Returns a 0 as persistent IDs are not intended to use with multi-level setup
return 0
}
// Init initializes storage nodes' connections to the given addrs.
//
// MustStop must be called when the initialized connections are no longer needed.
@@ -3015,6 +3024,7 @@ func newStorageNode(ms *metrics.Set, group *storageNodesGroup, addr string) *sto
sn := &storageNode{
group: group,
connPool: connPool,
id: connPool.GetTargetNodeID(),
concurrentQueries: ms.NewCounter(fmt.Sprintf(`vm_concurrent_queries{name="vmselect", addr=%q}`, addr)),

View File

@@ -1,13 +1,13 @@
{
"files": {
"main.css": "./static/css/main.fce049bf.css",
"main.js": "./static/js/main.36501ae8.js",
"main.js": "./static/js/main.36983a8a.js",
"static/js/685.bebe1265.chunk.js": "./static/js/685.bebe1265.chunk.js",
"static/media/MetricsQL.md": "./static/media/MetricsQL.d46c42c8e891f06298c4.md",
"index.html": "./index.html"
},
"entrypoints": [
"static/css/main.fce049bf.css",
"static/js/main.36501ae8.js"
"static/js/main.36983a8a.js"
]
}

View File

@@ -1 +1 @@
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=5"/><meta name="theme-color" content="#000000"/><meta name="description" content="UI for VictoriaMetrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><script src="./dashboards/index.js" type="module"></script><meta name="twitter:card" content="summary_large_image"><meta name="twitter:image" content="./preview.jpg"><meta name="twitter:title" content="UI for VictoriaMetrics"><meta name="twitter:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta name="twitter:site" content="@VictoriaMetrics"><meta property="og:title" content="Metric explorer for VictoriaMetrics"><meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta property="og:image" content="./preview.jpg"><meta property="og:type" content="website"><script defer="defer" src="./static/js/main.36501ae8.js"></script><link href="./static/css/main.fce049bf.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=5"/><meta name="theme-color" content="#000000"/><meta name="description" content="UI for VictoriaMetrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><script src="./dashboards/index.js" type="module"></script><meta name="twitter:card" content="summary_large_image"><meta name="twitter:image" content="./preview.jpg"><meta name="twitter:title" content="UI for VictoriaMetrics"><meta name="twitter:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta name="twitter:site" content="@VictoriaMetrics"><meta property="og:title" content="Metric explorer for VictoriaMetrics"><meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta property="og:image" content="./preview.jpg"><meta property="og:type" content="website"><script defer="defer" src="./static/js/main.36983a8a.js"></script><link href="./static/css/main.fce049bf.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>

View File

@@ -111,8 +111,8 @@ func main() {
blocksCount := tm.SmallBlocksCount + tm.BigBlocksCount
rowsCount := tm.SmallRowsCount + tm.BigRowsCount
sizeBytes := tm.SmallSizeBytes + tm.BigSizeBytes
logger.Infof("successfully opened storage %q in %.3f seconds; partsCount: %d; blocksCount: %d; rowsCount: %d; sizeBytes: %d",
*storageDataPath, time.Since(startTime).Seconds(), partsCount, blocksCount, rowsCount, sizeBytes)
logger.Infof("successfully opened storage %q (node ID: %d) in %.3f seconds; partsCount: %d; blocksCount: %d; rowsCount: %d; sizeBytes: %d",
*storageDataPath, strg.GetID(), time.Since(startTime).Seconds(), partsCount, blocksCount, rowsCount, sizeBytes)
// register storage metrics
storageMetrics := metrics.NewSet()

View File

@@ -101,7 +101,7 @@ func (s *VMInsertServer) run() {
// There is no need in response compression, since
// vmstorage sends only small packets to vminsert.
compressionLevel := 0
bc, err := handshake.VMInsertServer(c, compressionLevel)
bc, err := handshake.VMInsertServer(c, compressionLevel, s.storage.GetID())
if err != nil {
if s.isStopping() {
// c is stopped inside VMInsertServer.MustStop

View File

@@ -195,6 +195,10 @@ func (api *vmstorageAPI) setupTfss(qt *querytracer.Tracer, sq *storage.SearchQue
return tfss, nil
}
func (api *vmstorageAPI) GetID() uint64 {
return api.s.GetID()
}
// blockIterator implements vmselectapi.BlockIterator
type blockIterator struct {
sr storage.Search

View File

@@ -34,25 +34,14 @@ const QueryEditorAutocomplete: FC<QueryEditorAutocompleteProps> = ({
}, [value, caretPosition]);
const exprLastPart = useMemo(() => {
const regexpSplit = /\s(or|and|unless|default|ifnot|if|group_left|group_right)\s|}|\+|\|-|\*|\/|\^/i;
const parts = values.beforeCursor.split(regexpSplit);
const parts = values.beforeCursor.split("}");
return parts[parts.length - 1];
}, [values]);
const metric = useMemo(() => {
const regex1 = /\w+\((?<metricName>[^)]+)\)\s+(by|without|on|ignoring)\s*\(\w*/gi;
const matchAlt = [...exprLastPart.matchAll(regex1)];
if (matchAlt.length > 0 && matchAlt[0].groups && matchAlt[0].groups.metricName) {
return matchAlt[0].groups.metricName;
}
const regex2 = /^\s*\b(?<metricName>[^{}(),\s]+)(?={|$)/g;
const match = [...exprLastPart.matchAll(regex2)];
if (match.length > 0 && match[0].groups && match[0].groups.metricName) {
return match[0].groups.metricName;
}
return "";
const regexp = /\b[^{}(),\s]+(?={|$)/g;
const match = exprLastPart.match(regexp);
return match ? match[0] : "";
}, [exprLastPart]);
const label = useMemo(() => {
@@ -62,7 +51,7 @@ const QueryEditorAutocomplete: FC<QueryEditorAutocompleteProps> = ({
}, [exprLastPart]);
const shouldSuppressAutoSuggestion = (value: string) => {
const pattern = /([{(),+\-*/^]|\b(?:or|and|unless|default|ifnot|if|group_left|group_right|by|without|on|ignoring)\b)/i;
const pattern = /([{(),+\-*/^]|\b(?:or|and|unless|default|ifnot|if|group_left|group_right)\b)/;
const parts = value.split(/\s+/);
const partsCount = parts.length;
const lastPart = parts[partsCount - 1];
@@ -74,16 +63,12 @@ const QueryEditorAutocomplete: FC<QueryEditorAutocompleteProps> = ({
};
const context = useMemo(() => {
const valueBeforeCursor = values.beforeCursor.trim();
const endOfClosedBrackets = ["}", ")"].some(char => valueBeforeCursor.endsWith(char));
const endOfClosedQuotes = !hasUnclosedQuotes(valueBeforeCursor) && ["`", "'", "\""].some(char => valueBeforeCursor.endsWith(char));
if (!values.beforeCursor || endOfClosedBrackets || endOfClosedQuotes || shouldSuppressAutoSuggestion(values.beforeCursor)) {
if (!values.beforeCursor || values.beforeCursor.endsWith("}") || shouldSuppressAutoSuggestion(values.beforeCursor)) {
return QueryContextType.empty;
}
const labelRegexp = /(?:by|without|on|ignoring)\s*\(\s*[^)]*$|\{[^}]*$/i;
const patternLabelValue = `(${escapeRegexp(metric)})?{?.+${escapeRegexp(label)}(=|!=|=~|!~)"?([^"]*)$`;
const labelValueRegexp = new RegExp(patternLabelValue, "g");
const labelRegexp = /\{[^}]*$/;
const labelValueRegexp = new RegExp(`(${escapeRegexp(metric)})?{?.+${escapeRegexp(label)}(=|!=|=~|!~)"?([^"]*)$`, "g");
switch (true) {
case labelValueRegexp.test(values.beforeCursor):
@@ -96,7 +81,7 @@ const QueryEditorAutocomplete: FC<QueryEditorAutocompleteProps> = ({
}, [values, metric, label]);
const valueByContext = useMemo(() => {
const wordMatch = values.beforeCursor.match(/([\w_.:]+(?![},]))$/);
const wordMatch = values.beforeCursor.match(/([\w_\-.:/]+(?![},]))$/);
return wordMatch ? wordMatch[0] : "";
}, [values.beforeCursor]);
@@ -134,10 +119,9 @@ const QueryEditorAutocomplete: FC<QueryEditorAutocompleteProps> = ({
// Add quotes around the value if the context is labelValue
if (context === QueryContextType.labelValue) {
const quote = "\"";
const needsQuote = /(?:=|!=|=~|!~)$/.test(beforeValueByContext);
valueAfterCursor = valueAfterCursor.replace(/^[^\s"|},]*/, "");
const needsOpenQuote = /(?:=|!=|=~|!~)$/.test(beforeValueByContext);
const needsCloseQuote = valueAfterCursor.trim()[0] !== "\"";
insert = `${needsOpenQuote ? quote : ""}${insert}${needsCloseQuote ? quote : ""}`;
insert = `${needsQuote ? quote : ""}${insert}`;
}
if (context === QueryContextType.label) {

View File

@@ -137,7 +137,7 @@ export const useFetchQueryOptions = ({ valueByContext, metric, label, context }:
// fetch labels
useEffect(() => {
if (!serverUrl || context !== QueryContextType.label) {
if (!serverUrl || !metric || context !== QueryContextType.label) {
return;
}
setLabels([]);
@@ -149,7 +149,7 @@ export const useFetchQueryOptions = ({ valueByContext, metric, label, context }:
urlSuffix: "labels",
setter: setLabels,
type: TypeData.label,
params: getQueryParams(metric ? { "match[]": `{__name__="${metricEscaped}"}` } : undefined)
params: getQueryParams({ "match[]": `{__name__="${metricEscaped}"}` })
});
return () => abortControllerRef.current?.abort();
@@ -157,23 +157,20 @@ export const useFetchQueryOptions = ({ valueByContext, metric, label, context }:
// fetch labelValues
useEffect(() => {
if (!serverUrl || !label || context !== QueryContextType.labelValue) {
if (!serverUrl || !metric || !label || context !== QueryContextType.labelValue) {
return;
}
setLabelValues([]);
const metricEscaped = escapeDoubleQuotes(metric);
const valueReEscaped = escapeDoubleQuotes(escapeRegexp(value));
const matchMetric = metric ? `__name__="${metricEscaped}"` : "";
const matchLabel = `${label}=~".*${valueReEscaped}.*"`;
const matchValue = [matchMetric, matchLabel].filter(Boolean).join(",");
fetchData({
value,
urlSuffix: `label/${label}/values`,
setter: setLabelValues,
type: TypeData.labelValue,
params: getQueryParams({ "match[]": `{${matchValue}}` })
params: getQueryParams({ "match[]": `{__name__="${metricEscaped}", ${label}=~".*${valueReEscaped}.*"}` })
});
return () => abortControllerRef.current?.abort();

View File

@@ -8,6 +8,6 @@ export const escapeDoubleQuotes = (s: string) => {
};
export const hasUnclosedQuotes = (str: string) => {
const matches = str.match(/["`']/g);
const matches = str.match(/"/g);
return matches ? matches.length % 2 !== 0 : false;
};

View File

@@ -3548,8 +3548,7 @@
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
"color": "green"
},
{
"color": "red",
@@ -3565,7 +3564,7 @@
"h": 7,
"w": 12,
"x": 0,
"y": 37
"y": 13
},
"id": 48,
"options": {
@@ -3655,8 +3654,7 @@
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
"color": "green"
},
{
"color": "red",
@@ -3672,7 +3670,7 @@
"h": 7,
"w": 12,
"x": 12,
"y": 37
"y": 13
},
"id": 76,
"options": {
@@ -3760,8 +3758,7 @@
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
"color": "green"
},
{
"color": "red",
@@ -3777,7 +3774,7 @@
"h": 7,
"w": 12,
"x": 0,
"y": 44
"y": 20
},
"id": 132,
"options": {
@@ -3867,8 +3864,7 @@
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
"color": "green"
},
{
"color": "red",
@@ -3884,7 +3880,7 @@
"h": 7,
"w": 12,
"x": 12,
"y": 44
"y": 20
},
"id": 133,
"options": {
@@ -3973,8 +3969,7 @@
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
"color": "green"
},
{
"color": "red",
@@ -3990,7 +3985,7 @@
"h": 8,
"w": 12,
"x": 0,
"y": 51
"y": 27
},
"id": 20,
"options": {
@@ -4078,8 +4073,7 @@
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
"color": "green"
},
{
"color": "red",
@@ -4095,7 +4089,7 @@
"h": 8,
"w": 12,
"x": 12,
"y": 51
"y": 27
},
"id": 126,
"options": {
@@ -4182,8 +4176,7 @@
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
"color": "green"
},
{
"color": "red",
@@ -4199,7 +4192,7 @@
"h": 8,
"w": 12,
"x": 0,
"y": 59
"y": 35
},
"id": 46,
"options": {
@@ -4237,110 +4230,6 @@
"title": "Scrape response size 0.99 quantile ($instance)",
"type": "timeseries"
},
{
"datasource": {
"type": "victoriametrics-datasource",
"uid": "$ds"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"links": [],
"mappings": [],
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 59
},
"id": 148,
"options": {
"legend": {
"calcs": [
"mean",
"lastNotNull",
"max"
],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
},
"pluginVersion": "9.2.6",
"targets": [
{
"datasource": {
"type": "victoriametrics-datasource",
"uid": "$ds"
},
"editorMode": "code",
"expr": "max(histogram_quantile(0.99, sum(rate(vm_promscrape_scrape_duration_seconds_bucket{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) by(job, vmrange))) by(job)",
"format": "time_series",
"interval": "",
"legendFormat": "__auto",
"range": true,
"refId": "A"
}
],
"title": "Scrape duration 0.99 quantile ($instance)",
"type": "timeseries"
},
{
"datasource": {
"type": "victoriametrics-datasource",
@@ -4390,8 +4279,7 @@
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
"color": "green"
},
{
"color": "red",
@@ -4407,7 +4295,7 @@
"h": 8,
"w": 12,
"x": 12,
"y": 67
"y": 35
},
"id": 31,
"options": {
@@ -4687,7 +4575,8 @@
"mode": "absolute",
"steps": [
{
"color": "green"
"color": "green",
"value": null
},
{
"color": "red",
@@ -4791,7 +4680,8 @@
"mode": "absolute",
"steps": [
{
"color": "green"
"color": "green",
"value": null
},
{
"color": "red",
@@ -4908,7 +4798,8 @@
"mode": "absolute",
"steps": [
{
"color": "green"
"color": "green",
"value": null
},
{
"color": "red",
@@ -5043,7 +4934,8 @@
"mode": "absolute",
"steps": [
{
"color": "green"
"color": "green",
"value": null
},
{
"color": "red",
@@ -5145,7 +5037,8 @@
"mode": "absolute",
"steps": [
{
"color": "green"
"color": "green",
"value": null
},
{
"color": "red",
@@ -5241,7 +5134,8 @@
"mode": "absolute",
"steps": [
{
"color": "green"
"color": "green",
"value": null
},
{
"color": "red",
@@ -5344,7 +5238,8 @@
"mode": "absolute",
"steps": [
{
"color": "green"
"color": "green",
"value": null
},
{
"color": "red",
@@ -5454,7 +5349,8 @@
"mode": "absolute",
"steps": [
{
"color": "green"
"color": "green",
"value": null
},
{
"color": "red",
@@ -5551,7 +5447,8 @@
"mode": "absolute",
"steps": [
{
"color": "green"
"color": "green",
"value": null
},
{
"color": "red",
@@ -5648,7 +5545,8 @@
"mode": "absolute",
"steps": [
{
"color": "green"
"color": "green",
"value": null
},
{
"color": "red",
@@ -5795,7 +5693,8 @@
"mode": "absolute",
"steps": [
{
"color": "green"
"color": "green",
"value": null
},
{
"color": "red",
@@ -5899,7 +5798,8 @@
"mode": "absolute",
"steps": [
{
"color": "green"
"color": "green",
"value": null
},
{
"color": "red",
@@ -6003,7 +5903,8 @@
"mode": "absolute",
"steps": [
{
"color": "green"
"color": "green",
"value": null
},
{
"color": "red",
@@ -6107,7 +6008,8 @@
"mode": "absolute",
"steps": [
{
"color": "green"
"color": "green",
"value": null
},
{
"color": "red",
@@ -6210,7 +6112,8 @@
"mode": "absolute",
"steps": [
{
"color": "transparent"
"color": "transparent",
"value": null
},
{
"color": "red",
@@ -6412,7 +6315,8 @@
"mode": "absolute",
"steps": [
{
"color": "transparent"
"color": "transparent",
"value": null
},
{
"color": "red",

View File

@@ -3547,8 +3547,7 @@
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
"color": "green"
},
{
"color": "red",
@@ -3564,7 +3563,7 @@
"h": 7,
"w": 12,
"x": 0,
"y": 37
"y": 13
},
"id": 48,
"options": {
@@ -3654,8 +3653,7 @@
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
"color": "green"
},
{
"color": "red",
@@ -3671,7 +3669,7 @@
"h": 7,
"w": 12,
"x": 12,
"y": 37
"y": 13
},
"id": 76,
"options": {
@@ -3759,8 +3757,7 @@
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
"color": "green"
},
{
"color": "red",
@@ -3776,7 +3773,7 @@
"h": 7,
"w": 12,
"x": 0,
"y": 44
"y": 20
},
"id": 132,
"options": {
@@ -3866,8 +3863,7 @@
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
"color": "green"
},
{
"color": "red",
@@ -3883,7 +3879,7 @@
"h": 7,
"w": 12,
"x": 12,
"y": 44
"y": 20
},
"id": 133,
"options": {
@@ -3972,8 +3968,7 @@
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
"color": "green"
},
{
"color": "red",
@@ -3989,7 +3984,7 @@
"h": 8,
"w": 12,
"x": 0,
"y": 51
"y": 27
},
"id": 20,
"options": {
@@ -4077,8 +4072,7 @@
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
"color": "green"
},
{
"color": "red",
@@ -4094,7 +4088,7 @@
"h": 8,
"w": 12,
"x": 12,
"y": 51
"y": 27
},
"id": 126,
"options": {
@@ -4181,8 +4175,7 @@
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
"color": "green"
},
{
"color": "red",
@@ -4198,7 +4191,7 @@
"h": 8,
"w": 12,
"x": 0,
"y": 59
"y": 35
},
"id": 46,
"options": {
@@ -4236,110 +4229,6 @@
"title": "Scrape response size 0.99 quantile ($instance)",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "$ds"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"links": [],
"mappings": [],
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 59
},
"id": 148,
"options": {
"legend": {
"calcs": [
"mean",
"lastNotNull",
"max"
],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
},
"pluginVersion": "9.2.6",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "$ds"
},
"editorMode": "code",
"expr": "max(histogram_quantile(0.99, sum(rate(vm_promscrape_scrape_duration_seconds_bucket{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) by(job, vmrange))) by(job)",
"format": "time_series",
"interval": "",
"legendFormat": "__auto",
"range": true,
"refId": "A"
}
],
"title": "Scrape duration 0.99 quantile ($instance)",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
@@ -4389,8 +4278,7 @@
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
"color": "green"
},
{
"color": "red",
@@ -4406,7 +4294,7 @@
"h": 8,
"w": 12,
"x": 12,
"y": 67
"y": 35
},
"id": 31,
"options": {
@@ -4686,7 +4574,8 @@
"mode": "absolute",
"steps": [
{
"color": "green"
"color": "green",
"value": null
},
{
"color": "red",
@@ -4790,7 +4679,8 @@
"mode": "absolute",
"steps": [
{
"color": "green"
"color": "green",
"value": null
},
{
"color": "red",
@@ -4907,7 +4797,8 @@
"mode": "absolute",
"steps": [
{
"color": "green"
"color": "green",
"value": null
},
{
"color": "red",
@@ -5042,7 +4933,8 @@
"mode": "absolute",
"steps": [
{
"color": "green"
"color": "green",
"value": null
},
{
"color": "red",
@@ -5144,7 +5036,8 @@
"mode": "absolute",
"steps": [
{
"color": "green"
"color": "green",
"value": null
},
{
"color": "red",
@@ -5240,7 +5133,8 @@
"mode": "absolute",
"steps": [
{
"color": "green"
"color": "green",
"value": null
},
{
"color": "red",
@@ -5343,7 +5237,8 @@
"mode": "absolute",
"steps": [
{
"color": "green"
"color": "green",
"value": null
},
{
"color": "red",
@@ -5453,7 +5348,8 @@
"mode": "absolute",
"steps": [
{
"color": "green"
"color": "green",
"value": null
},
{
"color": "red",
@@ -5550,7 +5446,8 @@
"mode": "absolute",
"steps": [
{
"color": "green"
"color": "green",
"value": null
},
{
"color": "red",
@@ -5647,7 +5544,8 @@
"mode": "absolute",
"steps": [
{
"color": "green"
"color": "green",
"value": null
},
{
"color": "red",
@@ -5794,7 +5692,8 @@
"mode": "absolute",
"steps": [
{
"color": "green"
"color": "green",
"value": null
},
{
"color": "red",
@@ -5898,7 +5797,8 @@
"mode": "absolute",
"steps": [
{
"color": "green"
"color": "green",
"value": null
},
{
"color": "red",
@@ -6002,7 +5902,8 @@
"mode": "absolute",
"steps": [
{
"color": "green"
"color": "green",
"value": null
},
{
"color": "red",
@@ -6106,7 +6007,8 @@
"mode": "absolute",
"steps": [
{
"color": "green"
"color": "green",
"value": null
},
{
"color": "red",
@@ -6209,7 +6111,8 @@
"mode": "absolute",
"steps": [
{
"color": "transparent"
"color": "transparent",
"value": null
},
{
"color": "red",
@@ -6411,7 +6314,8 @@
"mode": "absolute",
"steps": [
{
"color": "transparent"
"color": "transparent",
"value": null
},
{
"color": "red",

View File

@@ -1,10 +1,9 @@
---
sort: 32
weight: 32
title: Best practices
title: VictoriaMetrics best practices
menu:
docs:
identifier: vm-best-practices
parent: 'victoriametrics'
weight: 32
aliases:

View File

@@ -4,7 +4,6 @@ weight: 100
title: CHANGELOG
menu:
docs:
identifier: vm-changelog
parent: victoriametrics
weight: 100
aliases:
@@ -30,24 +29,18 @@ See also [LTS releases](https://docs.victoriametrics.com/lts-releases/).
## tip
## [v1.102.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.102.1)
**Update note 1: [vmauth](https://docs.victoriametrics.com/vmauth/) HTTP response code has changed from 503 to 502 for a case when all upstream backends were not available. This was changed to align [vmauth](https://docs.victoriametrics.com/vmauth/) behaviour with other well-known reverse-proxies behaviour. **
Released at 2024-08-01
**Update note 1: [vmauth](https://docs.victoriametrics.com/vmauth/) HTTP response code has changed from 503 to 502 for a case when all upstream backends were not available. This was changed to align [vmauth](https://docs.victoriametrics.com/vmauth/) behaviour with other well-known reverse-proxies behaviour.**
**Update note 2: release contains breaking change to inter-cluster communication. The `vmstorage` nodes with version lower than `tip` won't be able to communicate with `vmselect` and `vminsert` nodes with version lower than `tip`. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5438) issue for the details.**
* SECURITY: upgrade base docker image (Alpine) from 3.20.1 to 3.20.2. See [alpine 3.20.2 release notes](https://alpinelinux.org/posts/Alpine-3.20.2-released.html).
* FEATURE: [vmauth](./vmauth.md): add `keep_original_host` option, which can be used for proxying the original `Host` header from client request to the backend. By default the backend host is used as `Host` header when proxying requests to the configured backends. See [these docs](./vmauth.md#host-http-header).
* FEATURE: [vmauth](./vmauth.md) now returns HTTP 502 status code when all upstream backends are not available. Previously, it returned HTTP 503 status code. This change aligns vmauth behavior with other well-known reverse-proxies behavior.
* FEATURE: [vmagent dashboard](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/dashboards/vmagent.json): add `Scrape duration 0.99 quantile` panel to show the 99th quantile of scrape duration in seconds. This should help identifying vmagent instances that experiences too high scraping durations.
* FEATURE: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): use a persistent `vmstorage` node ID for consistent hashing at data write path. This allows to keep the same data distribution after `vmstorage` changes its IP address. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5438).
* BUGFIX: [Single-node VictoriaMetrics](https://docs.victoriametrics.com/) and `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): Fixes panic if incorrect `metrisql` expression passed to the `prettify-query` API. See this [issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6736) for details.
* BUGFIX: all VictoriaMetrics components: validate files specified via `-tlsKeyFile` and `-tlsCertFile` cmd-line flags on the process start-up. Previously, validation happened on the first connection accepted by HTTP server. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6608) for the details. Thanks to @yincongcyincong for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6621).
* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth/): properly proxy requests to backend urls ending with `/` if the original request path equals to `/`. Previously the trailing `/` at the backend path was incorrectly removed. For example, if the request to `http://vmauth/` is configured to be proxied to `url_prefix=http://backend/foo/`, then it was proxied to `http://backend/foo`, while it should go to `http://backend/foo/`.
* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth/): fix `cannot read data after closing the reader` error when proxying HTTP requests without body (aka `GET` requests). The issue has been introduced in [v1.102.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.102.0) in [this commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/7ee57974935a662896f2de40fdf613156630617d).
* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix auto-completion triggers for metric and label names, and disable it after specific characters. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6153) and [these comments](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5866#issuecomment-2065273421).
* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix automatic addition of quotes when inserting label values from autocomplete suggestions. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6260).
## [v1.102.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.102.0)

View File

@@ -1,12 +1,11 @@
---
sort: 6
weight: 6
title: Year 2020
sort: 101
weight: 101
title: CHANGELOG for the year 2020
menu:
docs:
identifier: vm-changelog-2020
parent: vm-changelog
weight: 6
parent: 'victoriametrics'
weight: 101
aliases:
- /CHANGELOG_2020.html
---

View File

@@ -1,12 +1,11 @@
---
sort: 5
weight: 5
title: Year 2021
sort: 102
weight: 102
title: CHANGELOG for the year 2021
menu:
docs:
identifier: vm-changelog-2021
parent: vm-changelog
weight: 5
parent: 'victoriametrics'
weight: 102
aliases:
- /CHANGELOG_2021.html
---

View File

@@ -1,12 +1,11 @@
---
sort: 4
weight: 4
title: Year 2022
sort: 103
weight: 103
title: CHANGELOG for the year 2022
menu:
docs:
identifier: vm-changelog-2022
parent: vm-changelog
weight: 4
parent: 'victoriametrics'
weight: 103
aliases:
- /CHANGELOG_2022.html
---

View File

@@ -1,12 +1,11 @@
---
sort: 3
weight: 3
title: Year 2023
sort: 104
weight: 104
title: CHANGELOG for the year 2023
menu:
docs:
identifier: vm-changelog-2023
parent: vm-changelog
weight: 3
parent: 'victoriametrics'
weight: 104
aliases:
- /CHANGELOG_2023.html
---

View File

@@ -1,11 +1,10 @@
---
sort: 400
weight: 400
title: Contributing
title: Contributing to VictoriaMetrics
menu:
docs:
identifier: vm-contributing
parent: victoriametrics
parent: 'victoriametrics'
weight: 400
aliases:
- /CONTRIBUTING.html

View File

@@ -3,13 +3,17 @@ sort: 2
weight: 2
menu:
docs:
identifier: vm-cluster-version
parent: 'victoriametrics'
weight: 2
title: Cluster version
aliases:
- /Cluster-VictoriaMetrics.html
---
<picture>
<source srcset="/logo_white.webp" media="(prefers-color-scheme: dark)">
<source srcset="/logo.webp" media="(prefers-color-scheme: light)">
<img src="/logo.webp" width="300" alt="VictoriaMetrics logo">
</picture>
VictoriaMetrics is a fast, cost-effective and scalable time series database. It can be used as a long-term remote storage for Prometheus.
@@ -1297,6 +1301,8 @@ Below is the output for `/path/to/vminsert -help`:
Show VictoriaMetrics version
-vmstorageDialTimeout duration
Timeout for establishing RPC connections from vminsert to vmstorage. See also -vmstorageUserTimeout (default 3s)
-vmstorageUsePersistentID
Whether to use persistent storage node ID for -storageNode instances. If set to false uses storage node address in order to generate an ID. Using persistent node ID is useful if vmstorage node address changes over time, e.g. due to dynamic IP addresses or DNS names.
-vmstorageUserTimeout duration
Network timeout for RPC connections from vminsert to vmstorage (Linux only). Lower values speed up re-rerouting recovery when some of vmstorage nodes become unavailable because of networking issues. Read more about TCP_USER_TIMEOUT at https://blog.cloudflare.com/when-tcp-sockets-refuse-to-die/ . See also -vmstorageDialTimeout (default 3s)
```
@@ -1896,4 +1902,33 @@ Below is the output for `/path/to/vmstorage -help`:
TCP address to accept connections from vminsert services (default ":8400")
-vmselectAddr string
TCP address to accept connections from vmselect services (default ":8401")
```
```
## VictoriaMetrics Logo
[Zip](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/VM_logo.zip) contains three folders with different image orientation (main color and inverted version).
Files included in each folder:
- 2 JPEG Preview files
- 2 PNG Preview files with transparent background
- 2 EPS Adobe Illustrator EPS10 files
### Logo Usage Guidelines
#### Font used
- Lato Black
- Lato Regular
#### Color Palette
- HEX [#110f0f](https://www.color-hex.com/color/110f0f)
- HEX [#ffffff](https://www.color-hex.com/color/ffffff)
### We kindly ask
- Please don't use any other font instead of suggested.
- There should be sufficient clear space around the logo.
- Do not change spacing, alignment, or relative locations of the design elements.
- Do not change the proportions of any of the design elements or the design itself. You may resize as needed but must retain all proportions.

View File

@@ -215,10 +215,11 @@ We provide commercial support for both versions. [Contact us](mailto:info@victor
The following commercial versions of VictoriaMetrics are available:
* [VictoriaMetrics Cloud](https://cloud.victoriametrics.com/signUp?utm_source=website&utm_campaign=docs_vm_faq) the most cost-efficient hosted monitoring platform, operated by VictoriaMetrics core team.
* [Managed VictoriaMetrics at AWS](https://aws.amazon.com/marketplace/pp/prodview-4tbfq5icmbmyc) (aka managed Prometheus).
The following commercial versions of VictoriaMetrics are planned:
* Managed VictoriaMetrics at Google Cloud.
* Cloud monitoring solution based on VictoriaMetrics.
[Contact us](mailto:info@victoriametrics.com) for more information on our plans.

View File

@@ -1,11 +1,10 @@
---
sort: 31
weight: 31
title: Cluster Per Tenant Statistic
title: VictoriaMetrics Cluster Per Tenant Statistic
menu:
docs:
identifier: vm-cluster-per-tenant-statistic
parent: victoriametrics
parent: 'victoriametrics'
weight: 31
aliases:
- /PerTenantStatistic.html

View File

@@ -4,8 +4,7 @@ weight: 22
title: Quick start
menu:
docs:
identifier: vm-quick-start
parent: victoriametrics
parent: 'victoriametrics'
weight: 22
aliases:
- /Quick-Start.html
@@ -21,7 +20,7 @@ VictoriaMetrics is distributed in two forms:
Single-server-VictoriaMetrics VictoriaMetrics is available as:
* [VictoriaMetrics Cloud](https://cloud.victoriametrics.com/signUp?utm_source=website&utm_campaign=docs_vm_quickstart) the most cost-efficient hosted monitoring platfor, operated by VictoriaMetrics core team.
* [Managed VictoriaMetrics at AWS](https://aws.amazon.com/marketplace/pp/prodview-4tbfq5icmbmyc)
* [Docker images](https://hub.docker.com/r/victoriametrics/victoria-metrics/)
* [Helm Charts](https://github.com/VictoriaMetrics/helm-charts#list-of-charts)
* [Binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest)

View File

@@ -6,6 +6,12 @@
[![Build Status](https://github.com/VictoriaMetrics/VictoriaMetrics/workflows/main/badge.svg)](https://github.com/VictoriaMetrics/VictoriaMetrics/actions)
[![codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/branch/master/graph/badge.svg)](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics)
<picture>
<source srcset="/logo_white.webp" media="(prefers-color-scheme: dark)">
<source srcset="/logo.webp" media="(prefers-color-scheme: light)">
<img src="/logo.webp" width="300" alt="VictoriaMetrics logo">
</picture>
VictoriaMetrics is a fast, cost-effective and scalable monitoring solution and time series database.
See [case studies for VictoriaMetrics](https://docs.victoriametrics.com/casestudies/).
@@ -492,7 +498,7 @@ via ["submit metrics" API](https://docs.datadoghq.com/api/latest/metrics/#submit
DataDog agent allows configuring destinations for metrics sending via ENV variable `DD_DD_URL`
or via [configuration file](https://docs.datadoghq.com/agent/guide/agent-configuration-files/) in section `dd_url`.
![DD to VM](README_sending_DD_metrics_to_VM.webp)
![DD to VM](Single-server-VictoriaMetrics-sending_DD_metrics_to_VM.webp)
To configure DataDog agent via ENV variable add the following prefix:
@@ -519,7 +525,7 @@ pick [single-node or cluster URL](https://docs.victoriametrics.com/url-examples/
DataDog allows configuring [Dual Shipping](https://docs.datadoghq.com/agent/guide/dual-shipping/) for metrics
sending via ENV variable `DD_ADDITIONAL_ENDPOINTS` or via configuration file `additional_endpoints`.
![DD to VM](README_sending_DD_metrics_to_VM_and_DD.webp)
![DD to VM](Single-server-VictoriaMetrics-sending_DD_metrics_to_VM_and_DD.webp)
Run DataDog using the following ENV variable with VictoriaMetrics as additional metrics receiver:
@@ -1003,18 +1009,6 @@ at `/render` endpoint, which is used by [Graphite datasource in Grafana](https:/
When configuring Graphite datasource in Grafana, the `Storage-Step` http request header must be set to a step between Graphite data points
stored in VictoriaMetrics. For example, `Storage-Step: 10s` would mean 10 seconds distance between Graphite datapoints stored in VictoriaMetrics.
#### Known Incompatibilities with `graphite-web`
- **Timestamp Shifting**: VictoriaMetrics does not support shifting response timestamps outside the request time range as `graphite-web` does. This limitation impacts chained functions with time modifiers, such as `timeShift(summarize)`. For more details, refer to this [issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2969).
- **Non-deterministic series order**: due to the distributed nature of metrics processing, functions within the `seriesLists` family can produce non-deterministic results. To ensure consistent results, arguments for these functions must be wrapped with a sorting function. For instance, the function `divideSeriesLists(series_list_1, series_list_2)` should be modified to `divideSeriesLists(sortByName(series_list_1), sortByName(series_list_2))`. See this [issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5810) for details.
The affected functions include:
- `aggregateSeriesLists`
- `diffSeriesLists`
- `multiplySeriesLists`
- `divideSeriesLists`
### Graphite Metrics API usage
VictoriaMetrics supports the following handlers from [Graphite Metrics API](https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api):
@@ -2648,9 +2642,15 @@ and gets automatically updated once changes are merged to [master](https://githu
To update the documentation follow the steps below:
- [Fork](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/about-forks)
VictoriaMetrics repo and apply changes to the docs:
- To update [the main page](https://docs.victoriametrics.com/) modify [this file](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/README.md).
- To update [the main page](https://docs.victoriametrics.com/) modify [this file](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/README.md).
- To update other pages, apply changes to the corresponding file in [docs folder](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/docs).
- If your changes contain an image then see [images in documentation](https://docs.victoriametrics.com/#images-in-documentation).
- Once changes are made, execute the command below to finalize and sync the changes:
```sh
make docs-sync
```
- Create [a pull request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request)
with proposed changes and wait for it to be merged.
@@ -2663,7 +2663,7 @@ Requirements for changes to docs:
- Prefer improving the existing docs instead of adding new ones.
- Use absolute links. This simplifies moving docs between different files.
Periodically run `make spellcheck` - this command detects spelling errors at `docs/` folder. Please fix the found spelling errors
Priodically run `make spellcheck` - this command detects spelling errors at `docs/` folder. Please fix the found spelling errors
and commit the fixes in a separate commit.
### Images in documentation

View File

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 16 KiB

View File

Before

Width:  |  Height:  |  Size: 22 KiB

After

Width:  |  Height:  |  Size: 22 KiB

File diff suppressed because it is too large Load Diff

View File

@@ -1,7 +1,7 @@
---
sort: 7
weight: 7
title: CHANGELOG
title: VictoriaLogs changelog
menu:
docs:
identifier: "victorialogs-changelog"

View File

@@ -1,7 +1,7 @@
---
sort: 6
weight: 6
title: FAQ
title: VictoriaLogs FAQ
menu:
docs:
identifier: "victorialogs-faq"

View File

@@ -1,11 +1,10 @@
---
sort: 1
weight: 1
title: Quick Start
title: VictoriaLogs Quick Start
menu:
docs:
parent: victorialogs
identifier: vl-quick-start
parent: "victorialogs"
weight: 1
title: Quick Start
aliases:

View File

@@ -1,7 +1,7 @@
---
sort: 8
weight: 8
title: Roadmap
title: VictoriaLogs roadmap
disableToc: true
menu:
docs:

View File

@@ -1,11 +1,10 @@
---
sort: 2
weight: 2
title: Key concepts
title: VictoriaLogs key concepts
menu:
docs:
identifier: vl-key-concepts
parent: victorialogs
parent: "victorialogs"
weight: 2
title: Key concepts
aliases:

View File

@@ -13,16 +13,6 @@ aliases:
Please find the changelog for VictoriaMetrics Anomaly Detection below.
> **Important note: Users are strongly encouraged to upgrade to `vmanomaly` [v1.9.2](https://hub.docker.com/repository/docker/victoriametrics/vmanomaly/tags?page=1&ordering=name) or newer for optimal performance and accuracy. <br><br> This recommendation is crucial for configurations with a low `infer_every` parameter [in your scheduler](./components/scheduler.md#parameters-1), and in scenarios where data exhibits significant high-order seasonality patterns (such as hourly or daily cycles). Previous versions from v1.5.1 to v1.8.0 were identified to contain a critical issue impacting model training, where models were inadvertently trained on limited data subsets, leading to suboptimal fits, affecting the accuracy of anomaly detection. <br><br> Upgrading to v1.9.2 addresses this issue, ensuring proper model training and enhanced reliability. For users utilizing Helm charts, it is recommended to upgrade to version [1.0.0](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-anomaly/CHANGELOG.md#100) or newer.**
## v1.14.2
Released: 2024-07-26
- FIX: Patch a bug introduced in [v1.14.1](#v1141), causing `vmanomaly` to crash in `preset` [mode](/anomaly-detection/presets/).
## v1.14.1
Released: 2024-07-26
- FEATURE: Allow to process larger data chunks in [VmReader](/anomaly-detection/components/reader#vm-reader) that exceed `-search.maxPointsPerTimeseries` [constraint in VictoriaMetrics](https://docs.victoriametrics.com/?highlight=search.maxPointsPerTimeseries#resource-usage-limits) by splitting the range and sending multiple requests. A warning is printed in logs, suggesting reducing the range or step, or increasing `search.maxPointsPerTimeseries` constraint in VictoriaMetrics, which is still a recommended option.
- FEATURE: Backward-compatible redesign of [`queries`](/anomaly-detection/components/reader?highlight=queries#vm-reader) arg of [VmReader](/anomaly-detection/components/reader#vm-reader). Old format of `{q_alias1: q_expr1, q_alias2: q_expr2, ...}` will be implicitly converted to a new one with a warning raised in logs. New format allows to specify per-query parameters, like `step` to reduce amount of data read from VictoriaMetrics TSDB and to allow config flexibility. Find out more in [Per-query parameters section of VmReader](/anomaly-detection/components/reader/#per-query-parameters).
- IMPROVEMENT: Added multi-platform builds for `linux/amd64` and `linux/arm64` architectures.
## v1.13.3
Released: 2024-07-17

View File

@@ -11,21 +11,21 @@ aliases:
- /anomaly-detection/FAQ.html
---
## What is VictoriaMetrics Anomaly Detection (vmanomaly)?
VictoriaMetrics Anomaly Detection, also known as `vmanomaly`, is a service for detecting unexpected changes in time series data. Utilizing machine learning models, it computes and pushes back an ["anomaly score"](/anomaly-detection/components/models#vmanomaly-output) for user-specified metrics. This hands-off approach to anomaly detection reduces the need for manual alert setup and can adapt to various metrics, improving your observability experience.
VictoriaMetrics Anomaly Detection, also known as `vmanomaly`, is a service for detecting unexpected changes in time series data. Utilizing machine learning models, it computes and pushes back an ["anomaly score"](./components/models.md#vmanomaly-output) for user-specified metrics. This hands-off approach to anomaly detection reduces the need for manual alert setup and can adapt to various metrics, improving your observability experience.
Please refer to [our QuickStart section](/anomaly-detection/#practical-guides-and-installation) to find out more.
Please refer to [our guide section](./README.md#practical-guides-and-installation) to find out more.
> **Note: `vmanomaly` is a part of [enterprise package](../enterprise.md). You need to get a [free trial license](https://victoriametrics.com/products/enterprise/trial/) for evaluation.**
## What is anomaly score?
Among the metrics produced by `vmanomaly` (as detailed in [vmanomaly output metrics](/anomaly-detection/components/models#vmanomaly-output)), `anomaly_score` is a pivotal one. It is **a continuous score > 0**, calculated in such a way that **scores ranging from 0.0 to 1.0 usually represent normal data**, while **scores exceeding 1.0 are typically classified as anomalous**. However, it's important to note that the threshold for anomaly detection can be customized in the alert configuration settings.
Among the metrics produced by `vmanomaly` (as detailed in [vmanomaly output metrics](./components/models.md#vmanomaly-output)), `anomaly_score` is a pivotal one. It is **a continuous score > 0**, calculated in such a way that **scores ranging from 0.0 to 1.0 usually represent normal data**, while **scores exceeding 1.0 are typically classified as anomalous**. However, it's important to note that the threshold for anomaly detection can be customized in the alert configuration settings.
The decision to set the changepoint at `1.0` is made to ensure consistency across various models and alerting configurations, such that a score above `1.0` consistently signifies an anomaly, thus, alerting rules are maintained more easily.
> Note: `anomaly_score` is a metric itself, which preserves all labels found in input data and (optionally) appends [custom labels, specified in writer](/anomaly-detection/components/writer#metrics-formatting) - follow the link for detailed output example.
> Note: `anomaly_score` is a metric itself, which preserves all labels found in input data and (optionally) appends [custom labels, specified in writer](./components/writer.md#metrics-formatting) - follow the link for detailed output example.
## How is anomaly score calculated?
For most of the [univariate models](/anomaly-detection/components/models#univariate-models) that can generate `yhat`, `yhat_lower`, and `yhat_upper` time series in [their output](/anomaly-detection/components/models#vmanomaly-output) (such as [Prophet](/anomaly-detection/components/models#prophet) or [Z-score](/anomaly-detection/components/models#z-score)), the anomaly score is calculated as follows:
For most of the [univariate models](./components/models.md#univariate-models) that can generate `yhat`, `yhat_lower`, and `yhat_upper` time series in [their output](./components/models.md#vmanomaly-output) (such as [Prophet](./components/models.md#prophet) or [Z-score](./components/models.md#z-score)), the anomaly score is calculated as follows:
- If `yhat` (expected series behavior) equals `y` (actual value observed), then the anomaly score is 0.
- If `y` (actual value observed) falls within the `[yhat_lower, yhat_upper]` confidence interval, the anomaly score will gradually approach 1, the closer `y` is to the boundary.
- If `y` (actual value observed) strictly exceeds the `[yhat_lower, yhat_upper]` interval, the anomaly score will be greater than 1, increasing as the margin between the actual value and the expected range grows.
@@ -36,43 +36,39 @@ Please see example graph illustrating this logic below:
## How does vmanomaly work?
`vmanomaly` applies built-in (or custom) [anomaly detection algorithms](/anomaly-detection/components/models), specified in a config file.
- All the models generate a metric called [anomaly_score](#what-is-anomaly-score)
- All produced anomaly scores are unified in a way that values lower than 1.0 mean “likely normal”, while values over 1.0 mean “likely anomalous”
- Simple rules for alerting: start with `anomaly_score{“key”=”value”} > 1`
- Models are retrained continuously, based on `schedulers` section in a config, so that threshold=1.0 remains actual
- Produced scores are stored back to VictoriaMetrics TSDB and can be used for various observability tasks (alerting, visualization, debugging).
`vmanomaly` applies built-in (or custom) [anomaly detection algorithms](./components/models.md), specified in a config file. Although a single config file supports one model, running multiple instances of `vmanomaly` with different configs is possible and encouraged for parallel processing or better support for your use case (i.e. simpler model for simple metrics, more sophisticated one for metrics with trends and seasonalities).
1. For more detailed information, please visit the [overview section](./Overview.md#about).
2. To view a diagram illustrating the interaction of components, please explore the [components section](./components/README.md).
## What data does vmanomaly operate on?
`vmanomaly` operates on data fetched from VictoriaMetrics, where you can leverage full power of [MetricsQL](/metricsql) for data selection, sampling, and processing. Users can also [apply global filters](/#prometheus-querying-api-enhancements) for more targeted data analysis, enhancing scope limitation and tenant visibility.
`vmanomaly` operates on data fetched from VictoriaMetrics, where you can leverage full power of [MetricsQL](../MetricsQL.md) for data selection, sampling, and processing. Users can also [apply global filters](../README.md#prometheus-querying-api-enhancements) for more targeted data analysis, enhancing scope limitation and tenant visibility.
Respective config is defined in a [`reader`](/anomaly-detection/components/reader#vm-reader) section.
Respective config is defined in a [`reader`](./components/reader.md#vm-reader) section.
## Handling noisy input data
`vmanomaly` operates on data fetched from VictoriaMetrics using [MetricsQL](/metricsql) queries, so the initial data quality can be fine-tuned with aggregation, grouping, and filtering to reduce noise and improve anomaly detection accuracy.
`vmanomaly` operates on data fetched from VictoriaMetrics using [MetricsQL](../MetricsQL.md) queries, so the initial data quality can be fine-tuned with aggregation, grouping, and filtering to reduce noise and improve anomaly detection accuracy.
## Output produced by vmanomaly
`vmanomaly` models generate [metrics](/anomaly-detection/components/models#vmanomaly-output) like `anomaly_score`, `yhat`, `yhat_lower`, `yhat_upper`, and `y`. These metrics provide a comprehensive view of the detected anomalies. The service also produces [health check metrics](/anomaly-detection/components/monitoring#metrics-generated-by-vmanomaly) for monitoring its performance.
`vmanomaly` models generate [metrics](./components/models.md#vmanomaly-output) like `anomaly_score`, `yhat`, `yhat_lower`, `yhat_upper`, and `y`. These metrics provide a comprehensive view of the detected anomalies. The service also produces [health check metrics](./components/monitoring.md#metrics-generated-by-vmanomaly) for monitoring its performance.
## Choosing the right model for vmanomaly
Selecting the best model for `vmanomaly` depends on the data's nature and the [types of anomalies](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-2/#categories-of-anomalies) to detect. For instance, [Z-score](/anomaly-detection/components/models#z-score) is suitable for data without trends or seasonality, while more complex patterns might require models like [Prophet](/anomaly-detection/components/models#prophet).
Selecting the best model for `vmanomaly` depends on the data's nature and the [types of anomalies](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-2/#categories-of-anomalies) to detect. For instance, [Z-score](./components/models.md#z-score) is suitable for data without trends or seasonality, while more complex patterns might require models like [Prophet](./components/models.md#prophet).
Also, starting from [v1.12.0](/anomaly-detection/changelog/#v1120) it's possible to auto-tune the most important params of selected model class, find [the details here](/anomaly-detection/components/models#autotuned).
Also, starting from [v1.12.0](./CHANGELOG.md#v1120) it's possible to auto-tune the most important params of selected model class, find [the details here](./components/models.md#autotuned).
Please refer to [respective blogpost on anomaly types and alerting heuristics](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-2/) for more details.
Still not 100% sure what to use? We are [here to help](/anomaly-detection/#get-in-touch).
Still not 100% sure what to use? We are [here to help](./README.md#get-in-touch).
## Alert generation in vmanomaly
While `vmanomaly` detects anomalies and produces scores, it *does not directly generate alerts*. The anomaly scores are written back to VictoriaMetrics, where an external alerting tool, like [`vmalert`](/vmalert), can be used to create alerts based on these scores for integrating it with your alerting management system.
While `vmanomaly` detects anomalies and produces scores, it *does not directly generate alerts*. The anomaly scores are written back to VictoriaMetrics, where an external alerting tool, like [`vmalert`](../vmalert.md), can be used to create alerts based on these scores for integrating it with your alerting management system.
## Preventing alert fatigue
Produced anomaly scores are designed in such a way that values from 0.0 to 1.0 indicate non-anomalous data, while a value greater than 1.0 is generally classified as an anomaly. However, there are no perfect models for anomaly detection, that's why reasonable defaults expressions like `anomaly_score > 1` may not work 100% of the time. However, anomaly scores, produced by `vmanomaly` are written back as metrics to VictoriaMetrics, where tools like [`vmalert`](/vmalert) can use [MetricsQL](/MetricsQL) expressions to fine-tune alerting thresholds and conditions, balancing between avoiding [false negatives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-negative) and reducing [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-positive).
Produced anomaly scores are designed in such a way that values from 0.0 to 1.0 indicate non-anomalous data, while a value greater than 1.0 is generally classified as an anomaly. However, there are no perfect models for anomaly detection, that's why reasonable defaults expressions like `anomaly_score > 1` may not work 100% of the time. However, anomaly scores, produced by `vmanomaly` are written back as metrics to VictoriaMetrics, where tools like [`vmalert`](../vmalert.md) can use [MetricsQL](../MetricsQL.md) expressions to fine-tune alerting thresholds and conditions, balancing between avoiding [false negatives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-negative) and reducing [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-positive).
## How to backtest particular configuration on historical data?
Starting from [v1.7.2](/anomaly-detection/changelog#v172) you can produce (and write back to VictoriaMetrics TSDB) anomaly scores for historical (backtesting) period, using `BacktestingScheduler` [component](/anomaly-detection/components/scheduler#backtesting-scheduler) to imitate consecutive "production runs" of `PeriodicScheduler` [component](/anomaly-detection/components/scheduler#periodic-scheduler). Please find an example config below:
Starting from [v1.7.2](./CHANGELOG.md#v172) you can produce (and write back to VictoriaMetrics TSDB) anomaly scores for historical (backtesting) period, using `BacktestingScheduler` [component](./components/scheduler.md#backtesting-scheduler) to imitate consecutive "production runs" of `PeriodicScheduler` [component](./components/scheduler.md#periodic-scheduler). Please find an example config below:
```yaml
schedulers:
@@ -85,8 +81,6 @@ schedulers:
# copy these from your PeriodicScheduler args
fit_window: 'P14D'
fit_every: 'PT1H'
# number of parallel jobs to run. Default is 1, each job is a separate OneOffScheduler fit/inference run.
n_jobs: 1
models:
model_alias1:
@@ -114,12 +108,12 @@ writer:
# https://docs.victoriametrics.com/anomaly-detection/components/monitoring/
```
Configuration above will produce N intervals of full length (`fit_window`=14d + `fit_every`=1h) until `to_iso` timestamp is reached to run N consecutive `fit` calls to train models; Then these models will be used to produce `M = [fit_every / sampling_frequency]` infer datapoints for `fit_every` range at the end of each such interval, imitating M consecutive calls of `infer_every` in `PeriodicScheduler` [config](/anomaly-detection/components/scheduler#periodic-scheduler). These datapoints then will be written back to VictoriaMetrics TSDB, defined in `writer` [section](/anomaly-detection/components/writer#vm-writer) for further visualization (i.e. in VMUI or Grafana)
Configuration above will produce N intervals of full length (`fit_window`=14d + `fit_every`=1h) until `to_iso` timestamp is reached to run N consecutive `fit` calls to train models; Then these models will be used to produce `M = [fit_every / sampling_frequency]` infer datapoints for `fit_every` range at the end of each such interval, imitating M consecutive calls of `infer_every` in `PeriodicScheduler` [config](./components/scheduler.md#periodic-scheduler). These datapoints then will be written back to VictoriaMetrics TSDB, defined in `writer` [section](./components/writer.md#vm-writer) for further visualization (i.e. in VMUI or Grafana)
## Resource consumption of vmanomaly
`vmanomaly` itself is a lightweight service, resource usage is primarily dependent on [scheduling](/anomaly-detection/components/scheduler) (how often and on what data to fit/infer your models), [# and size of timeseries returned by your queries](./components/reader.md#vm-reader), and the complexity of the employed [models](/anomaly-detection/components/models). Its resource usage is directly related to these factors, making it adaptable to various operational scales.
`vmanomaly` itself is a lightweight service, resource usage is primarily dependent on [scheduling](./components/scheduler.md) (how often and on what data to fit/infer your models), [# and size of timeseries returned by your queries](./components/reader.md#vm-reader), and the complexity of the employed [models](./components/models.md). Its resource usage is directly related to these factors, making it adaptable to various operational scales.
> **Note**: Starting from [v1.13.0](/anomaly-detection/changelog#v1130), there is a mode to save anomaly detection models on host filesystem after `fit` stage (instead of keeping them in-memory by default). **Resource-intensive setups** (many models, many metrics, bigger [`fit_window` arg](/anomaly-detection/components/scheduler#periodic-scheduler-config-example)) and/or 3rd-party models that store fit data (like [ProphetModel](/anomaly-detection/components/models#prophet) or [HoltWinters](/anomaly-detection/components/models#holt-winters)) will have RAM consumption greatly reduced at a cost of slightly slower `infer` stage. To enable it, you need to set environment variable `VMANOMALY_MODEL_DUMPS_DIR` to desired location. [Helm charts](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-anomaly/README.md) are being updated accordingly ([`StatefulSet`](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) for persistent storage starting from chart version `1.3.0`).
> **Note**: Starting from [v1.13.0](./CHANGELOG.md#v1130), there is a mode to save anomaly detection models on host filesystem after `fit` stage (instead of keeping them in-memory by default). **Resource-intensive setups** (many models, many metrics, bigger [`fit_window` arg](./components/scheduler.md#periodic-scheduler-config-example)) and/or 3rd-party models that store fit data (like [ProphetModel](./components/models.md#prophet) or [HoltWinters](./components/models.md#holt-winters)) will have RAM consumption greatly reduced at a cost of slightly slower `infer` stage. To enable it, you need to set environment variable `VMANOMALY_MODEL_DUMPS_DIR` to desired location. [Helm charts](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-anomaly/README.md) are being updated accordingly ([`StatefulSet`](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) for persistent storage starting from chart version `1.3.0`).
Here's an example of how to set it up in docker-compose using volumes:
```yaml
@@ -153,16 +147,16 @@ volumes:
## Scaling vmanomaly
> **Note:** As of latest release we don't support cluster or auto-scaled version yet (though, it's in our roadmap for - better backends, more parallelization, etc.), so proposed workarounds should be addressed manually.
`vmanomaly` can be scaled horizontally by launching multiple independent instances, each with its own [MetricsQL](/MetricsQL) queries and [configurations](/anomaly-detection/components/):
`vmanomaly` can be scaled horizontally by launching multiple independent instances, each with its own [MetricsQL](../MetricsQL.md) queries and [configurations](./components/README.md):
- By splitting **queries**, [defined in reader section](/anomaly-detection/components/reader#vm-reader) and spawn separate service around it. Also in case you have *only 1 query returning huge amount of timeseries*, you can further split it by applying MetricsQL filters, i.e. using "extra_filters" [param in reader](/anomaly-detection/components/reader?highlight=extra_filters#vm-reader)
- By splitting **queries**, [defined in reader section](./components/reader.md#vm-reader) and spawn separate service around it. Also in case you have *only 1 query returning huge amount of timeseries*, you can further split it by applying MetricsQL filters, i.e. using "extra_filters" [param in reader](./components/reader.md#vm-reader)
- or **models** (in case you decide to run several models for each timeseries received i.e. for averaging anomaly scores in your alerting rules of `vmalert` or using a vote approach to reduce false positives) - see `queries` arg in [model config](/anomaly-detection/components/models#queries)
- or **models** (in case you decide to run several models for each timeseries received i.e. for averaging anomaly scores in your alerting rules of `vmalert` or using a vote approach to reduce false positives) - see `queries` arg in [model config](./components/models.md#queries)
- or **schedulers** (in case you want the same models to be trained under several schedules) - see `schedulers` arg [model section](/anomaly-detection/components/models#schedulers) and `scheduler` [component itself](/anomaly-detection/components/scheduler)
- or **schedulers** (in case you want the same models to be trained under several schedules) - see `schedulers` arg [model section](./components/models.md#schedulers) and `scheduler` [component itself](./components/scheduler.md)
Here's an example of how to split on `extra_filters`, based on `extra_filters` reader's arg
Here's an example of how to split on `extra_filters` param
```yaml
# config file #1, for 1st vmanomaly instance
# ...

View File

@@ -23,14 +23,17 @@ Additionally, **preset mode** minimizes user input needed to run the service. Yo
Available presets:
- [Node-Exporter](#node-exporter)
To enable preset mode, `preset` arg should be set to particular preset name:
Here is an example config file to enable [Node-Exporter](#node-exporter) preset:
```yaml
preset: "chosen_preset_name" # i.e. "node-exporter"
preset: "node-exporter"
reader:
datasource_url: "http://victoriametrics:8428/" # your datasource url
# tenant_id: '0:0' # specify for cluster version
writer:
datasource_url: "http://victoriametrics:8428/" # your datasource url
# tenant_id: '0:0' # specify for cluster version
```
Also, additional minimal set of arguments may be required from user to run the preset. See corresponding preset sections below for the details.
Run a service using config file with one of the [available options](./QuickStart.md#how-to-install-and-run-vmanomaly).
After you run `vmanomaly` with `preset` arg specified, available assets can be viewed, copied and downloaded at `http://localhost:8490/presets/` endpoint.
@@ -40,24 +43,13 @@ After you run `vmanomaly` with `preset` arg specified, available assets can be v
## Node-Exporter
The Node-Exporter preset simplifies the monitoring and anomaly detection of key system metrics collected by [`node_exporter`](https://github.com/prometheus/node_exporter). This preset reduces the need for manual configuration and detects anomalies in metrics such as CPU usage, network errors, and disk latency, ensuring timely identification of potential issues. Below are detailed instructions on enabling and using the Node-Exporter preset, along with a list of included assets like alerting rules and Grafana dashboard.
> **Note: Node-Exporter preset assets can be also found [here](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/vmanomaly/vmanomaly-node-exporter-preset/)**
For enabling Node-Exporter in config file set the `preset` arg accordingly. Also, include at least `datasource_url`-s (and `tenant_id` if using cluster version of VictoriaMetrics) in reader and writer sections, like that:
> **Note: Preset assets can be also found [here](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/vmanomaly/vmanomaly-node-exporter-preset/)**
For enabling Node-Exporter in config file use `preset` parameter:
```yaml
preset: "node-exporter"
reader:
datasource_url: "http://victoriametrics:8428/" # source victoriametrics/prometheus
# tenant_id: '0:0' # specify for cluster version
writer:
datasource_url: "http://victoriametrics:8428/" # destination victoriametrics/prometheus
# tenant_id: '0:0' # specify for cluster version
```
Run a service using such config file with one of the [available options](./QuickStart.md#how-to-install-and-run-vmanomaly).
### Generated anomaly scores
Machine learning models will be fit for each timeseries, returned by underlying [MetricsQL](../MetricsQL.md) queries.
Anomaly score metric labels will also contain [model classes](./components/models.md) and [schedulers](./components/scheduler.md) for labelset uniqueness.

View File

@@ -20,61 +20,6 @@ Future updates will introduce additional readers, expanding the range of data so
## VM reader
> **Note**: Starting from [v1.13.0](/anomaly-detection/changelog/v1130) there is backward-compatible change of [`queries`](/anomaly-detection/components/reader?highlight=queries#vm-reader) arg of [VmReader](#vm-reader). New format allows to specify per-query parameters, like `step` to reduce amount of data read from VictoriaMetrics TSDB and to allow config flexibility. Please see [per-query parameters](#per-query-parameters) section for the details.
Old format like
```yaml
# other config sections ...
reader:
class: 'vm'
datasource_url: 'http://localhost:8428' # source victoriametrics/prometheus
sampling_period: "10s" # set it <= min(infer_every) in schedulers section
queries:
# old format {query_alias: query_expr}, prior to 1.13, will be converted to a new format automatically
vmb: 'avg(vm_blocks)'
```
will be converted to a new one with a warning raised in logs:
```yaml
# other config sections ...
reader:
class: 'vm'
datasource_url: 'http://localhost:8428' # source victoriametrics/prometheus
sampling_period: '10s'
queries:
# old format {query_alias: query_expr}, prior to 1.13, will be converted to a new format automatically
vmb:
expr: 'avg(vm_blocks)' # initial MetricsQL expression
step: '10s' # individual step for this query, will be filled with `sampling_period` from the root level
# new query-level arguments will be added in backward-compatible way in future releases
```
### Per-query parameters
Starting from [v1.13.0](/anomaly-detection/changelog/v1130) there is change of [`queries`](/anomaly-detection/components/reader?highlight=queries#vm-reader) arg format. Now each query alias supports the next (sub)fields:
- `expr` (string): MetricsQL/PromQL expression that defines an input for VmReader. As accepted by `/query_range?query=%s`. i.e. `avg(vm_blocks)`
- `step` (string): query-level frequency of the points returned, i.e. `30s`. Will be converted to `/query_range?step=%s` param (in seconds). Useful to optimize total amount of data read from VictoriaMetrics, where different queries may have **different frequencies for different [machine learning models](/anomaly-detection/components/models)** to run on.
> **Note**: if not set explicitly (or if older config style prior to [v1.13.0](/anomaly-detection/changelog/v1130)) is used, then it is set to reader-level `sampling_period` arg.
> **Note**: having **different** individual `step` args for queries (i.e. `30s` for `q1` and `2m` for `q2`) is not yet supported for [multivariate model](/anomaly-detection/components/models/index.html#multivariate-models) if you want to run it on several queries simultaneously (i.e. setting [`queries`](/anomaly-detection/components/models/#queries) arg of a model to [`q1`, `q2`]).
### Per-query config example
```yaml
reader:
class: 'vm'
sampling_period: '1m'
# other reader params ...
queries:
ingestion_rate:
expr: 'sum(rate(vm_rows_inserted_total[5m])) by (type) > 0'
step: '2m' # overrides global `sampling_period` of 1m
```
### Config parameters
<table class="params">
@@ -106,12 +51,12 @@ Name of the class needed to enable reading from VictoriaMetrics or Prometheus. V
`queries`
</td>
<td>
See [per-query config example](#per-query-config-example) above
`ingestion_rate: 'sum(rate(vm_rows_inserted_total[5m])) by (type) > 0'`
</td>
<td>
<td>
See [per-query config section](#per-query-parameters) above
PromQL/MetricsQL query to select data in format: `QUERY_ALIAS: "QUERY"`. As accepted by `/query_range?query=%s`.
</td>
</tr>
<tr>

View File

@@ -16,7 +16,7 @@ Since Proxmox Virtual Environment(PVE) and Proxmox Backup Server(PBS) support se
Currently PVE and PBS only support using an Authorization Token for authentication and does not support basic auth or a username and password.
## Proxmox Virtual Environment (PVE)
If want help Sending your data to VictoriaMetrics Cloud check out [our blog](https://victoriametrics.com/blog/proxmox-monitoring-with-dbaas/).
If want help Sending your data to Managed VictoriaMetrics check out [our blog](https://victoriametrics.com/blog/proxmox-monitoring-with-dbaas/).
1. Login to PVE as an administrator
2. Go to DataCenter > MetricServer > Add > InfluxDB
@@ -65,4 +65,4 @@ You should see 1 time series per node in your PVE cluster.
# References
- [Blog Post for configuring VictoriaMetrics Cloud and Proxmox VE](https://victoriametrics.com/blog/proxmox-monitoring-with-dbaas/)
- [Blog Post for configuring Managed VictoriaMetrics and Proxmox VE](https://victoriametrics.com/blog/proxmox-monitoring-with-dbaas/)

View File

@@ -1,10 +1,9 @@
---
sort: 99
weight: 99
title: Enterprise
title: VictoriaMetrics Enterprise
menu:
docs:
identifier: vm-enterprise
parent: 'victoriametrics'
weight: 99
aliases:
@@ -29,7 +28,7 @@ The use of VictoriaMetrics Enterprise components is permitted in the following c
- Production use if you have a valid enterprise contract or valid permit from VictoriaMetrics company.
Please contact us via [this page](https://victoriametrics.com/products/enterprise/) if you are intereseted in such a contract.
- [VictoriaMetrics Cloud](https://docs.victoriametrics.com/victoriametrics-cloud/) is built on top of VictoriaMetrics Enterprise.
- [Managed VictoriaMetrics](https://docs.victoriametrics.com/managed-victoriametrics/) is built on top of VictoriaMetrics Enterprise.
See [these docs](#running-victoriametrics-enterprise) for details on how to run VictoriaMetrics enterprise.

View File

@@ -1,7 +1,7 @@
---
sort: 500
weight: 500
title: Development goals
title: VictoriaMetrics development goals
menu:
docs:
parent: 'victoriametrics'

11
docs/guides/README.md Normal file
View File

@@ -0,0 +1,11 @@
1. [K8s monitoring via VM Single](k8s-monitoring-via-vm-single.html)
1. [K8s monitoring via VM Cluster](k8s-monitoring-via-vm-cluster.html)
1. [HA monitoring setup in K8s via VM Cluster](k8s-ha-monitoring-via-vm-cluster.html)
1. [Getting started with VM Operator](getting-started-with-vm-operator.html)
1. [Multi Retention Setup within VictoriaMetrics Cluster](guide-vmcluster-multiple-retention-setup.html)
1. [Migrate from InfluxDB to VictoriaMetrics](migrate-from-influx.html)
1. [Multi-regional setup with VictoriaMetrics: Dedicated regions for monitoring](multi-regional-setup-dedicated-regions.html)
1. [How to delete or replace metrics in VictoriaMetrics](guide-delete-or-replace-metrics.html)
1. [How to monitor kubernetes cluster using Managed VictoriaMetrics](/managed-victoriametrics/how-to-monitor-k8s.html)
1. [How to configure vmgateway for multi-tenant access using Grafana and OpenID Connect](grafana-vmgateway-openid-configuration.html)
1. [How to setup vmanomaly together with vmalert](/anomaly-detection/guides/guide-vmanomaly-vmalert/README.md)

View File

@@ -2,5 +2,4 @@
weight: 0
title: Guides
disableToc: true
---
{{% section %}}
---

View File

@@ -8,7 +8,7 @@ menu:
aliases:
- /guides/understand-your-setup-size.html
---
The docs provide a simple and high-level overview of Ingestion Rate, Active Time Series, and Query per Second. These terms are a part of capacity planning ([Single-Node](https://docs.victoriametrics.com/single-server-victoriametrics/#capacity-planning), [Cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#capacity-planning)) and [VictoriaMetrics Cloud](https://docs.victoriametrics.com/victoriametrics-cloud/) pricing.
The docs provide a simple and high-level overview of Ingestion Rate, Active Time Series, and Query per Second. These terms are a part of capacity planning ([Single-Node](https://docs.victoriametrics.com/single-server-victoriametrics/#capacity-planning), [Cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#capacity-planning)) and [Managed VictoriaMetrics](https://docs.victoriametrics.com/managed-victoriametrics/) pricing.
## Terminology
@@ -101,12 +101,12 @@ You have a Kubernetes environment that produces 5k time series per second with 1
VictoriaMetrics requires additional disk space for the index. The lower Churn Rate means lower disk space usage for the index because of better compression.
Usually, the index takes about 20% of the disk space for storing data. High cardinality setups may use >50% of datapoints storage size for index.
You can significantly reduce the amount of disk usage by specifying [Downsampling](https://docs.victoriametrics.com/#downsampling) and [Retention Filters](https://docs.victoriametrics.com/#retention-filters) that are lower than the Retention Period. Both two settings are available in VictoriaMetrics Cloud and Enterprise.
You can significantly reduce the amount of disk usage by specifying [Downsampling](https://docs.victoriametrics.com/#downsampling) and [Retention Filters](https://docs.victoriametrics.com/#retention-filters) that are lower than the Retention Period. Both two settings are available in Managed VictoriaMetrics and Enterprise.
## Align Terms with VictoriaMetrics setups
### VictoriaMetrics Cloud
### Managed VictoriaMetrics
Every deployment (Single-Node or Cluster) contains the expected load in Ingestion Rate and Active Time Series. We assume that the Churn Rate is no more than 30%. You may need to choose a more extensive deployment if you have a higher Churn Rate.

View File

@@ -4,8 +4,7 @@ weight: 34
title: Key concepts
menu:
docs:
identifier: vm-key-concepts
parent: victoriametrics
parent: 'victoriametrics'
weight: 34
aliases:
- /keyConcepts.html

View File

@@ -0,0 +1,14 @@
* [Overview of Managed VictoriaMetrics](/managed-victoriametrics/overview/)
* [Quick Start](/managed-victoriametrics/quickstart/)
## Guides
* [Understand Your Setup Size](/guides/understand-your-setup-size/)
* [Alerting & recording rules with Alertmanager configuration for Managed VictoriaMetrics deployment](/managed-victoriametrics/alertmanager-setup-for-deployment/)
* [Kubernetes Monitoring with Managed VictoriaMetrics](/managed-victoriametrics/how-to-monitor-k8s/)
* [Setup Notifications](/managed-victoriametrics/setup-notifications/)
* [User Management](/managed-victoriametrics/user-managment/)
Learn more about Managed VictoriaMetrics:
* [Managed VictoriaMetrics announcement](https://victoriametrics.com/blog/managed-victoriametrics-announcement)
* [Pricing comparison for Managed Prometheus](https://victoriametrics.com/blog/managed-prometheus-pricing/)
* [Monitoring Proxmox VE via Managed VictoriaMetrics and vmagent](https://victoriametrics.com/blog/proxmox-monitoring-with-dbaas/)

View File

@@ -0,0 +1,7 @@
---
sort: 28
title: Managed VictoriaMetrics
weight: 01
disableToc: true
---
{{% content "README.md" %}}

View File

@@ -1,25 +1,23 @@
---
sort: 5
weight: 5
title: Alerting with vmalert and VictoriaMetrics Cloud
title: Alerting with vmalert and Managed VictoriaMetrics
menu:
docs:
parent: "cloud"
parent: "managed"
weight: 5
aliases:
- /victoriametrics-cloud/alerting-vmalert-victoria-metrics-cloud/index.html
- /managed-victoriametrics/alerting-vmalert-victoria-metrics-cloud/index.html
- /managed-victoriametrics/alerting-vmalert-managed-victoria-metrics.html
---
This guide explains the different ways in which you can use vmalert in conjunction with Managed VictoriaMetrics
This guide explains the different ways in which you can use vmalert in conjunction with VictoriaMetrics Cloud
![Metrics setup](alerting-vmalert-victoria-metrics-cloud_setup.webp)
![Metrics setup](alerting-vmalert-managed-victoria-metrics_setup.webp)
## Preconditions
* [vmalert](https://docs.victoriametrics.com/vmalert/) is installed. You can obtain it by building it from [source](https://docs.victoriametrics.com/vmalert/#quickstart), downloading it from the [GitHub releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest), or using the [docker image](https://hub.docker.com/r/victoriametrics/vmalert) for the container ecosystem (such as docker, k8s, etc.).
* [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) is installed.
* You have a [single or cluster](https://docs.victoriametrics.com/victoriametrics-cloud/quickstart.html#creating-deployment) deployment in [VictoriaMetrics Cloud](https://docs.victoriametrics.com/victoriametrics-cloud/overview.html).
* You have a [single or cluster](https://docs.victoriametrics.com/managed-victoriametrics/quickstart.html#creating-deployment) deployment in [Managed VictoriaMetrics](https://docs.victoriametrics.com/managed-victoriametrics/overview.html).
* If you are using helm, add the [VictoriaMetrics helm chart](https://github.com/VictoriaMetrics/helm-charts/tree/master/charts/victoria-metrics-alert#how-to-install) repository to your helm repositories. This step is optional.
* If you are using [vmoperator](https://docs.victoriametrics.com/operator/quick-start.html#quick-start), make sure that it and its CRDs are installed. This step is also optional.
@@ -47,22 +45,22 @@ groups:
description: "Job {{ $labels.job }} instance: {{$labels.instance }} is not up for the last 1 minute"
```
### VictoriaMetrics Cloud access token and deployment endpoint
### Managed VictoriaMetrics access token and deployment endpoint
To use vmalert with VictoriaMetrics Cloud, you must create a read/write token, or use an existing one. The token must have write access to ingest recording rules, ALERTS and ALERTS_FOR_STATE metrics, and read access for rules evaluation.
To use vmalert with Managed VictoriaMetrics, you must create a read/write token, or use an existing one. The token must have write access to ingest recording rules, ALERTS and ALERTS_FOR_STATE metrics, and read access for rules evaluation.
For instructions on how to create tokens, please refer to this section of the [documentation](https://docs.victoriametrics.com/victoriametrics-cloud/quickstart.html#deployment-access).
For instructions on how to create tokens, please refer to this section of the [documentation](https://docs.victoriametrics.com/managed-victoriametrics/quickstart.html#deployment-access).
#### Single-Node
![Token created single](alerting-vmalert-victoria-metrics-cloud_token_created_single.webp)
![Copy datasource single](alerting-vmalert-victoria-metrics-cloud_copy_datasource_single.webp)
![Token created single](alerting-vmalert-managed-victoria-metrics_token_created_single.webp)
![Copy datasource single](alerting-vmalert-managed-victoria-metrics_copy_datasource_single.webp)
#### Cluster
![Token created cluster](alerting-vmalert-victoria-metrics-cloud_token_created_cluster.webp)
![Reading datasource cluster](alerting-vmalert-victoria-metrics-cloud_copy_reading_datasource_cluster.webp)
![Writing atasource cluster](alerting-vmalert-victoria-metrics-cloud_copy_writing_datasource_cluster.webp)
![Token created cluster](alerting-vmalert-managed-victoria-metrics_token_created_cluster.webp)
![Reading datasource cluster](alerting-vmalert-managed-victoria-metrics_copy_reading_datasource_cluster.webp)
![Writing atasource cluster](alerting-vmalert-managed-victoria-metrics_copy_writing_datasource_cluster.webp)
### vmalert configuration

View File

@@ -1,16 +1,15 @@
---
sort: 4
weight: 4
title: Alertmanager and VMAlert configuration for VictoriaMetrics Cloud deployment
title: Alertmanager and VMAlert configuration for Managed VictoriaMetrics deployment
menu:
docs:
parent: "cloud"
parent: "managed"
weight: 4
aliases:
- /victoriametrics-cloud/alertmanager-setup-for-deployment/index.html
- /managed-victoriametrics/alertmanager-setup-for-deployment/index.html
- /managed-victoriametrics/alertmanager-setup-for-deployment.html
---
VictoriaMetrics Cloud supports configuring alerting rules, powered by vmalert, and sending notifications with hosted Alertmanager.
Managed VictoriaMetrics supports configuring alerting rules, powered by vmalert, and sending notifications with hosted Alertmanager.
## Configure Alertmanager
@@ -38,7 +37,7 @@ Alertmanager is now set up for your deployment, and you be able to get notificat
### Alertmanager config specification
VictoriaMetrics Cloud supports Alertmanager with standard [configuration specification](https://prometheus.io/docs/alerting/latest/configuration/).
Managed VictoriaMetrics supports Alertmanager with standard [configuration specification](https://prometheus.io/docs/alerting/latest/configuration/).
But with respect to the specification, there are the following limitations:
@@ -123,7 +122,7 @@ receivers:
### Custom Alertmanager
If for some reason Cloud Alertmanager is not suitable for you, you can use VictoriaMetrics Cloud with any external Alertmanager hosted in your infrastructure.
If for some reason Cloud Alertmanager is not suitable for you, you can use Managed VictoriaMetrics with any external Alertmanager hosted in your infrastructure.
For that select Custom Alertmanager instead of Cloud Alertmanager when [creating the Alertmanager](#configure-alertmanager):

View File

Before

Width:  |  Height:  |  Size: 15 KiB

After

Width:  |  Height:  |  Size: 15 KiB

View File

Before

Width:  |  Height:  |  Size: 15 KiB

After

Width:  |  Height:  |  Size: 15 KiB

View File

Before

Width:  |  Height:  |  Size: 18 KiB

After

Width:  |  Height:  |  Size: 18 KiB

View File

Before

Width:  |  Height:  |  Size: 24 KiB

After

Width:  |  Height:  |  Size: 24 KiB

View File

Before

Width:  |  Height:  |  Size: 52 KiB

After

Width:  |  Height:  |  Size: 52 KiB

View File

Before

Width:  |  Height:  |  Size: 34 KiB

After

Width:  |  Height:  |  Size: 34 KiB

View File

Before

Width:  |  Height:  |  Size: 70 KiB

After

Width:  |  Height:  |  Size: 70 KiB

View File

Before

Width:  |  Height:  |  Size: 81 KiB

After

Width:  |  Height:  |  Size: 81 KiB

View File

Before

Width:  |  Height:  |  Size: 44 KiB

After

Width:  |  Height:  |  Size: 44 KiB

View File

Before

Width:  |  Height:  |  Size: 76 KiB

After

Width:  |  Height:  |  Size: 76 KiB

View File

Before

Width:  |  Height:  |  Size: 73 KiB

After

Width:  |  Height:  |  Size: 73 KiB

View File

@@ -1,14 +1,13 @@
---
sort: 3
weight: 3
title: Kubernetes Monitoring with VictoriaMetrics Cloud
title: Kubernetes Monitoring with Managed VictoriaMetrics
menu:
docs:
parent: "cloud"
parent: "managed"
weight: 3
aliases:
- /victoriametrics-cloud/how-to-monitor-k8s/index.html
- /managed-victoriametrics/how-to-monitor-k8s/index.html
- /managed-victoriametrics/how-to-monitor-k8s.html
---
Monitoring kubernetes cluster is necessary to build SLO/SLI, to analyze performance and cost-efficiency of your workloads.
@@ -25,8 +24,7 @@ This chart will install `VMOperator`, `VMAgent`, `NodeExporter`, `kube-state-met
## Prerequisites
- Active VictoriaMetrics Cloud instance. You can learn how to sign up for VictoriaMetrics Cloud [here](https://docs.victoriametrics.com/victoriametrics-cloud/quickstart.html#how-to-register).
- Active Managed VictoriaMetrics instance. You can learn how to sign up for Managed VictoriaMetrics [here](https://docs.victoriametrics.com/managed-victoriametrics/quickstart.html#how-to-register).
- Access to your kubernetes cluster
- Helm binary. You can find installation [here](https://helm.sh/docs/intro/install/)

View File

Before

Width:  |  Height:  |  Size: 33 KiB

After

Width:  |  Height:  |  Size: 33 KiB

View File

Before

Width:  |  Height:  |  Size: 13 KiB

After

Width:  |  Height:  |  Size: 13 KiB

View File

Before

Width:  |  Height:  |  Size: 5.8 KiB

After

Width:  |  Height:  |  Size: 5.8 KiB

View File

Before

Width:  |  Height:  |  Size: 24 KiB

After

Width:  |  Height:  |  Size: 24 KiB

View File

Before

Width:  |  Height:  |  Size: 45 KiB

After

Width:  |  Height:  |  Size: 45 KiB

View File

Before

Width:  |  Height:  |  Size: 67 KiB

After

Width:  |  Height:  |  Size: 67 KiB

View File

Before

Width:  |  Height:  |  Size: 23 KiB

After

Width:  |  Height:  |  Size: 23 KiB

View File

Before

Width:  |  Height:  |  Size: 55 KiB

After

Width:  |  Height:  |  Size: 55 KiB

View File

Before

Width:  |  Height:  |  Size: 59 KiB

After

Width:  |  Height:  |  Size: 59 KiB

View File

Before

Width:  |  Height:  |  Size: 4.6 KiB

After

Width:  |  Height:  |  Size: 4.6 KiB

View File

Before

Width:  |  Height:  |  Size: 8.2 KiB

After

Width:  |  Height:  |  Size: 8.2 KiB

Some files were not shown because too many files have changed in this diff Show More