VictoriaMetrics/docs/anomaly-detection/components/monitoring.md

title, weight, menu, tags, aliases

title	weight	menu	tags	aliases
Monitoring	5	docs parent weight identifier vmanomaly-components 5 vmanomaly-monitoring	metrics enterprise	./monitoring.html

There are 2 models to monitor VictoriaMetrics Anomaly Detection behavior - push and pull. Parameters for each of them should be specified in the config file, monitoring section.

There was an enhancement of self-monitoring metrics for consistency across the components (v.1.17.0). Documentation was updated accordingly. Key changes included:

• Converting several self-monitoring metrics from Summary to Histogram to enable quantile calculation. This addresses the limitation of the prometheus_client's Summary implementation, which does not support quantiles. The change ensures metrics are more informative for performance analysis. Affected metrics are:
  • vmanomaly_reader_request_duration_seconds (VmReader)
  • vmanomaly_reader_response_parsing_seconds (VmReader)
  • vmanomaly_writer_request_duration_seconds (VmWriter)
  • vmanomaly_writer_request_serialize_seconds (VmWriter)
• Adding a query_key label to the vmanomaly_reader_response_parsing_seconds metric to provide finer granularity in tracking the performance of individual queries. This metric has also been switched from Summary to Histogram to align with the other metrics and support quantile calculations.
• Adding preset and scheduler_alias keys to VmReader and VmWriter metrics for consistency in multi-scheduler setups.
• Renaming Counters vmanomaly_reader_response_count to vmanomaly_reader_responses and vmanomaly_writer_response_count to vmanomaly_writer_responses.

Pull Model Config parameters

Parameter	Default	Description
addr	"0.0.0.0"	Server IP Address
port	8080	Port

Push Config parameters

By default, metrics are pushed only after the completion of specific stages, e.g., fit, infer, or fit_infer (for each scheduler if using a multi-scheduler configuration).

The push_frequency parameter{{% available_from "v1.18.7" anomaly %}} (default value: 15m) can be configured to initiate additional periodic metric pushes at consistent intervals. This enhances the self-monitoring capabilities of vmanomaly by aligning more closely with pull-based monitoring behavior, especially in setups with infrequent schedules (e.g., long fit_every or infer_every intervals in PeriodicScheduler), mitigating data staleness. To disable scheduled metric pushes, set the push_frequency parameter to an empty string in the configuration file, as demonstrated in the examples below.

Parameter	Default	Description
url		Link where to push metrics to. Example: "http://localhost:8480/"
tenant_id		Tenant ID for cluster version. Example: "0:0"
health_path	"health"	{{% deprecated_from "v1.8.0" anomaly %}}. Absolute, to override /health path
user		BasicAuth username
password		BasicAuth password
bearer_token	token	Token is passed in the standard format with header: Authorization: bearer {token}{{% available_from "v1.15.9" anomaly %}}.
bearer_token_file	path_to_file	Path to a file, which contains token, that is passed in the standard format with header: Authorization: bearer {token}{{% available_from "v1.15.9" anomaly %}}.
verify_tls	false	Verify TLS certificate. If False, it will not verify the TLS certificate. If True, it will verify the certificate using the system's CA store. If a path to a CA bundle file (like ca.crt), it will verify the certificate using the provided CA bundle.
tls_cert_file	path/to/cert.crt	Path to a file with the client certificate, i.e. client.crt{{% available_from "v1.16.3" anomaly %}}.
tls_key_file	path/to/key.crt	Path to a file with the client certificate key, i.e. client.key{{% available_from "v1.16.3" anomaly %}}.
timeout	"5s"	Stop waiting for a response after a given number of seconds.
push_frequency	"15m"	Frequency for scheduled pushing of metrics, e.g., '30m'. Suggested to be less than the staleness interval -search.maxStalenessInterval Set to empty string to disable scheduled pushing{{% available_from "v1.18.7" anomaly %}}.
extra_labels		Section for custom labels specified by user.

Monitoring section config example

monitoring:
  pull: # Enable /metrics endpoint.
    addr: "0.0.0.0"
    port: 8080
  push:
    url: "http://localhost:8480/"
    tenant_id: "0:0" # For cluster version only
    user: "USERNAME"
    password: "PASSWORD"
    verify_tls: False
    timeout: "5s"
    push_frequency: "15m"  # set to "" to disable scheduled pushes and leave only fit/infer based
    extra_labels:
      job: "vmanomaly-push"
      test: "test-1"

mTLS protection

vmanomaly components such as VmWriter support mTLS{{% available_from "v1.16.3" anomaly %}} to ensure secure communication with VictoriaMetrics Enterprise, configured with mTLS.

For detailed guidance on configuring mTLS parameters such as verify_tls, tls_cert_file, and tls_key_file, please refer to the mTLS protection section in the Reader documentation. The configuration principles apply consistently across all these vmanomaly components.

Metrics generated by vmanomaly

• Startup metrics
• Reader metrics
• Model metrics
• Writer metrics

Startup metrics

Metric	Type	Description
vmanomaly_start_time_seconds	Gauge	vmanomaly start time in UNIX time
vmanomaly_version_info	Gauge	vmanomaly version information, contained in version label{{% available_from "v1.17.2" anomaly %}}.
vmanomaly_ui_version_info	Gauge	vmanomaly UI version information, contained in version label{{% available_from "v1.17.2" anomaly %}}.
vmanomaly_available_memory_bytes	Gauge	Effective memory capacity available to the process in bytes{{% available_from "v1.18.4" anomaly %}}. The value honors cgroup limits when available, then process address-space limits, and otherwise reports host physical memory. It does not represent currently unused memory.
vmanomaly_cpu_cores_available	Gauge	Effective CPU capacity available to the process{{% available_from "v1.18.4" anomaly %}}, constrained by host logical CPUs, process affinity, and cgroup quota. The value can be fractional when a fractional CPU quota is configured.
vmanomaly_config_entities	Gauge	Number of sub-configs available (scope="total") and used by the current shard (scope="shard"){{% available_from "v1.21.0" anomaly %}}, labeled by preset and scope.
vmanomaly_config_reload_enabled (was vmanomaly_hot_reload_enabled {{% deprecated_from "v1.25.1" anomaly %}})	Gauge	Whether particular vmanomaly instance is run in config hot-reload mode {{% available_from "v1.25.0" anomaly %}}
vmanomaly_config_reloads_total (was vmanomaly_hot_reload_events_total{{% deprecated_from "v1.25.1" anomaly %}})	Counter	How many config hot-reloads were made since service started {{% available_from "v1.25.0" anomaly %}}
vmanomaly_config_last_reload_successful	Gauge	Whether last config hot-reload was successful (1) or not (0) {{% available_from "v1.25.1" anomaly %}}
vmanomaly_config_last_reload_success_timestamp_seconds	Gauge	Timestamp of the last successful config hot-reload in seconds since epoch {{% available_from "v1.25.1" anomaly %}}
vmanomaly_scheduler_alive	Gauge	Whether the scheduler worker thread identified by scheduler_alias and preset is alive (1) or not (0) {{% available_from "v1.30.0" anomaly %}}.
vmanomaly_scheduler_restarts_total	Counter	Number of bounded scheduler restart attempts {{% available_from "v1.30.0" anomaly %}}, labeled by scheduler_alias, preset, and status (success or failure).
vm_license_expires_at	Gauge	License expiration time as a Unix timestamp in seconds. See the licensing section for example alerts.
vm_license_expires_in_seconds	Gauge	Time remaining until license expiration in seconds. See the licensing section for warning and critical alert examples.

Back to metric sections

Reader behaviour metrics

Label names description

To improve consistency across the components additional labels (scheduler_alias, preset) were added to writer and reader metrics{{% available_from "v1.17.0" anomaly %}}. Also, metrics vmanomaly_reader_request_duration_seconds and vmanomaly_reader_response_parsing_seconds changed their type to Histogram (was Summary{{% deprecated_from "v1.17.0" anomaly %}}).

Metric	Type	Description	Labelnames
vmanomaly_reader_request_duration_seconds	Histogram (was Summary{{% deprecated_from "v1.17.0" anomaly %}})	The total time (in seconds) taken by queries to VictoriaMetrics url for the query_key query within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	url, query_key, scheduler_alias, preset
vmanomaly_reader_responses (named vmanomaly_reader_response_count{{% deprecated_from "v1.17.0" anomaly %}})	Counter	The count of responses received from VictoriaMetrics url for the query_key query, categorized by code, within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	url, query_key, code, scheduler_alias, preset
vmanomaly_reader_received_bytes	Counter	The total number of bytes received in responses for the query_key query within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	url, query_key, scheduler_alias, preset
vmanomaly_reader_response_parsing_seconds	Histogram (was Summary{{% deprecated_from "v1.17.0" anomaly %}})	The total time (in seconds) taken for data parsing at each step (json or df) for the query_key query within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	step, url, query_key, scheduler_alias, preset
vmanomaly_reader_timeseries_received	Counter	The total number of timeseries received from VictoriaMetrics for the query_key query within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	url, query_key, scheduler_alias, preset
vmanomaly_reader_datapoints_received	Counter	The total number of datapoints received from VictoriaMetrics for the query_key query within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	url, query_key, scheduler_alias, preset
vmanomaly_reader_processing_tasks_queued	Gauge	The total number of queued processing tasks {{% available_from "v1.29.7" anomaly %}} (timeseries batches of size series_processing_batch_size) for the query_key query within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode. If continuously >0, it may lead to skipped infer runs due to resource contention and timeouts.	url, query_key, scheduler_alias, preset

Back to metric sections

Models behaviour metrics

Label names description

There is a new label key model_alias introduced in multi-model support{{% available_from "v1.10.0" anomaly %}}. This label key adjustment was made to preserve unique label set production during writing produced metrics back to VictoriaMetrics.

As a part of self-monitoring metrics enhancement{{% available_from "v1.17.0" anomaly %}}, new metrics, like vmanomaly_model_run_errors, was added. Some of them changed the type (Summary -> Histogram), like vmanomaly_model_run_duration_seconds.

Metric	Type	Description	Labelnames
vmanomaly_model_runs	Counter	How many successful stage (fit, infer, fit_infer) runs occurred for models of class model_alias based on results from the query_key query, within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	stage, query_key, model_alias, scheduler_alias, preset
vmanomaly_model_run_duration_seconds	Histogram (was Summary{{% deprecated_from "v1.17.0" anomaly %}})	The model-service stage duration in seconds for fit, infer, or combined fit_infer execution, based on the results of the query_key query for model_alias. Reader and writer I/O durations are reported by their respective metrics.	stage, query_key, model_alias, scheduler_alias, preset
vmanomaly_model_datapoints_accepted	Counter	The number of valid datapoints accepted by model_alias, excluding NaN and Inf values, during fit, infer, or combined fit_infer execution for the query_key query.	stage, query_key, model_alias, scheduler_alias, preset
vmanomaly_model_datapoints_produced	Counter	The number of datapoints generated by models of class model_alias during the stage (infer, fit_infer) based on results from the query_key query, within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	stage, query_key, model_alias, scheduler_alias, preset
vmanomaly_models_active	Gauge	The number of model instances of class model_alias currently available for inference for the query_key query, within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	query_key, model_alias, scheduler_alias, preset
vmanomaly_model_runs_skipped	Counter	The number of times model runs (of class model_alias) were skipped in expected situations (e.g., no data for fitting/inference, or no new data to infer on) during the stage (fit, infer, fit_infer), based on results from the query_key query, within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	stage, query_key, model_alias, scheduler_alias, preset
vmanomaly_model_run_errors	Counter	The number of times model runs (of class model_alias) failed due to internal service errors during the stage (fit, infer, fit_infer), based on results from the query_key query, within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	stage, query_key, model_alias, scheduler_alias, preset

Back to metric sections

Writer behaviour metrics

Label names description

Additional labels (scheduler_alias, preset){{% available_from "v1.17.0" anomaly %}} were added to writer and reader metrics to improve consistency across the components. Also, metrics vmanomaly_writer_request_duration_seconds and vmanomaly_writer_request_serialize_seconds changed their type to Histogram (was Summary{{% deprecated_from "v1.17.0" anomaly %}}).

Metric	Type	Description	Labelnames
vmanomaly_writer_request_duration_seconds	Histogram (was Summary{{% deprecated_from "v1.17.0" anomaly %}})	The total time (in seconds) taken by write requests to VictoriaMetrics url for the query_key query within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	url, query_key, scheduler_alias, preset
vmanomaly_writer_responses (named vmanomaly_writer_response_count{{% deprecated_from "v1.17.0" anomaly %}})	Counter	The count of response codes received from VictoriaMetrics url for the query_key query, categorized by code, within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	url, code, query_key, scheduler_alias, preset
vmanomaly_writer_sent_bytes	Counter	The total number of bytes sent to VictoriaMetrics url for the query_key query within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	url, query_key, scheduler_alias, preset
vmanomaly_writer_request_serialize_seconds	Histogram (was Summary{{% deprecated_from "v1.17.0" anomaly %}})	The total time (in seconds) taken for serializing data for the query_key query within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	url, query_key, scheduler_alias, preset
vmanomaly_writer_datapoints_sent	Counter	The total number of datapoints sent to VictoriaMetrics for the query_key query within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	url, query_key, scheduler_alias, preset
vmanomaly_writer_timeseries_sent	Counter	The total number of timeseries sent to VictoriaMetrics for the query_key query within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	url, query_key, scheduler_alias, preset

Back to metric sections

Labelnames

• stage - model execution stage: fit, infer, or fit_infer for a combined fit/inference scheduler run. See model types.
• query_key - query alias from reader config section.
• model_alias - model alias from models config section{{% available_from "v1.10.0" anomaly %}}.
• scheduler_alias - scheduler alias from schedulers config section{{% available_from "v1.11.0" anomaly %}}.
• preset - preset alias for preset mode of vmanomaly{{% available_from "v1.12.0" anomaly %}}.
• url - writer or reader url endpoint.
• code - HTTP response status code or connection_error, timeout, ssl_error, or io_error.
• step - reader parsing step: json or df.

Back to metric sections

Logs generated by vmanomaly

The vmanomaly service logs important lifecycle, I/O, model, and recovery events alongside self-monitoring metrics. The fragments below are stable prefixes for recognizing log families, rather than byte-for-byte message contracts; entity values and exception details follow the prefix.

By default, vmanomaly uses the INFO level. Use the global --loggerLevel command-line argument or settings.logger_levels{{% available_from "v1.30.0" anomaly %}} for prefix-based component overrides:

settings:
  logger_levels:
    reader: DEBUG       # also applies to reader.vm, reader.vlogs, and other child loggers
    writer.vm: ERROR
    copilot: WARNING

More-specific prefixes override their parent. Changes limited to settings.logger_levels can be hot-reloaded without restarting services. See settings.logger_levels and the command-line arguments.

• Startup logs
• Reader logs
• Service logs
• Writer logs
• Scheduler supervision logs
• Hot-reload logs
• Persisted-state logs
• Query server and task logs
• AI Copilot logs

Startup logs

Startup logs summarize the version, license, effective storage mode, state restoration, process-pool mode, server addresses, hot-reload state, and active schedulers. The most useful prefixes are:

• License check: Please provide a license code, failed to read file, and Licensed to.
• Config validation: Config validation failed, Config read failed, and the fatal Config validation failed, shutting down. Successful startup ends with Config has been loaded successfully.
• Model and data directory setup: Using ENV VMANOMALY_MODEL_DUMPS_DIR, Using ENV VMANOMALY_DATA_DUMPS_DIR, or their is not set in-memory variants. See on-disk mode.
• Scheduler and service initialization: Version:, Using process pool executor, Listening on, Serving /metrics, Hot reload enabled, and Active schedulers. Process pool health check failed, falling back to sequential mode reports a safe runtime fallback. Per-scheduler wrapping and omitted empty schedulers are DEBUG diagnostics.

Back to logging sections

Reader logs

Reader logs cover endpoint checks, request splitting, network failures, response parsing, and coordination between queries used by the same model.

Starting a healthcheck request. The reader probes each configured tenant and discovers search.maxPointsPerTimeseries. Max points per timeseries set as is a DEBUG diagnostic. A warning beginning Could not get constraints means the reader uses its built-in limit. Endpoint initialization errors identify SSL, connection, or timeout failures.

No data found (False). A fit/read range with no results uses this form, showing both local and Unix times:

[Scheduler `SCHEDULER`] No data for query_key `QUERY` between LOCAL_START and LOCAL_END timezone TZ (START_EPOCH to END_EPOCH)

Check the query, tenant, offsets, and selected range.

No unseen data found (True). An inference read whose timestamps were already processed uses:

[Scheduler `SCHEDULER`] No unseen data for query_key `QUERY` between LOCAL_START and LOCAL_END, timezone TZ (START_EPOCH to END_EPOCH)

This can be expected for overlapping scheduler windows, UI range navigation, or retries. Investigate when it persists while the datasource continues receiving newer samples.

Connection or timeout errors. Error querying URL for QUERY with PARAMS includes the effective endpoint, query alias, request parameters, and nested SSL, connection, timeout, or I/O reason. The corresponding vmanomaly_reader_responses code is ssl_error, connection_error, timeout, or io_error.

At DEBUG, reader request lines start with [Scheduler ...] GET or OPTIONS and show the effective URL; token query parameters are redacted. Cancellation requested for query records cooperative cancellation. Failed queries detected, Timeout waiting for queries, and Auto-marking pending queries as failed identify coordination failures for related query sets.

Max datapoints warning. Query "QUERY" from START to END with step ... may exceed max datapoints per timeseries (LIMIT) means the range will be split{{% available_from "v1.14.1" anomaly %}}. The message reports the effective limit and suggests reducing the range, increasing the step, or raising search.maxPointsPerTimeseries. A DEBUG message reports the resulting interval count.

Multi-tenancy warnings. Messages starting with The label vm_account_id was not found indicate that a multitenant query lost routing labels. Preserve vm_account_id and vm_project_id through query aggregation; see multitenancy support.

Metrics updated in read operations. Requests update duration and response-code metrics even on handled failures. Bytes, time series, datapoints, and parsing durations are recorded only when those values were received or parsed. See reader behaviour metrics.

Back to logging sections

Service logs

The service logs fit, infer, and combined fit_infer/backtesting work for each model alias and scheduler. The query_key value may be a composite key containing source labels and an internal hash, rather than only the configured query alias.

Skipped runs. Warnings start with Skipping run for stage 'STAGE' for model 'MODEL'. Common reasons are no fit or inference partition, no data to infer, no unseen valid data, a missing model instance or on-disk model, an unsupported exact-batch path, or no valid output. The service attempts fitting when at least one valid row exists; individual models may require more history and report their own error. Skips increment vmanomaly_model_runs_skipped.

Errors during model execution. Errors start with Error during stage 'STAGE' for model 'MODEL' and include the composite query key and exception. They increment vmanomaly_model_run_errors.

Model instance created during inference. Model instance 'MODEL' created ... during inference is a DEBUG message for an online model cold start{{% available_from "v1.15.2" anomaly %}}.

Successful model runs. Fitting on VALID/TOTAL valid datapoints is emitted at INFO. At DEBUG, Model ... fit completed, Inference ran in, and Fit-Infer ran in report stage duration. Combined fit_infer is used by applicable backtesting/scheduler execution and is not a separate “rolling model” class.

Metrics updated in model runs. Successful stages update runs, duration, accepted/produced datapoints, and active-model gauges. Skips and failures update their respective counters; success-only values are not recorded for an unsuccessful stage. See models behaviour metrics.

Back to logging sections

Writer logs

Writer logs cover serialization and delivery of produced series such as anomaly_score.

Starting a write request. At DEBUG, [Scheduler ...] POST URL with N datapoints, M bytes of payload includes the composite query key and dataframe shape.

No valid data points. No valid datapoints to save for metric includes the query key and original dataframe shape; no request is sent.

Connection, timeout, or I/O errors. Cannot write N points for QUERY ends with an SSL, connection, timeout, or I/O reason. A retriable connection failure first emits Connection error while writing ... reinitializing session and retrying; the final failed attempt is logged as an error.

Multi-tenancy warnings. The label vm_account_id was not found means a multitenant writer will fall back to tenant 0:0. The label set for the metric ... contains multi-tenancy labels means labels disagree with the configured single tenant. Preserve or align tenant labels and writer.tenant_id; see multitenancy support.

Metrics updated in write operations. Request duration is observed for successful and handled failed requests. vmanomaly_writer_responses records the HTTP status or ssl_error, connection_error, timeout, or io_error. Serialization duration and prepared time-series count may already be recorded before a failed request; sent bytes and datapoints are recorded only after a successful response. See writer behaviour metrics.

Back to logging sections

Scheduler supervision logs

Scheduler supervision{{% available_from "v1.30.0" anomaly %}} logs a dead worker, automatic restart, successful recovery, failed-attempt backoff, and removal after the retry limit. Stable prefixes include Scheduler ... is not alive, Scheduler ... restarted successfully, restart attempt ... failed, and reached max restart attempts. Correlate them with vmanomaly_scheduler_alive and vmanomaly_scheduler_restarts_total.

Back to logging sections

Hot-reload logs

Hot reload logs config-change detection, validation, staged service restart, success, and rollback. Reload aborted – invalid config keeps the current runtime unchanged; Reload apply failed; attempting rollback starts recovery. Rollback failed is critical and requests shutdown. A logger-only change emits Applied component log level changes without restarting services.

Back to logging sections

Persisted-state logs

With settings.restore_state, startup logs the stored/runtime version assessment, reusable components, required model or reader-data purges, and restored jobs/services. Persisted state is incompatible followed by Dropping stored artifacts completely indicates a full reset; missing or unreadable model files are reported separately.

Back to logging sections

Query server and task logs

The query server logs its listening address and datasource-proxy timeouts/failures. Background anomaly-detection and autotune failures use Error in task and Error in autotune task; canceled client requests may still leave a background raw query finishing cleanly.

Back to logging sections

AI Copilot logs

AI Copilot{{% available_from "v1.30.0" anomaly %}} reports whether it is initialized, disabled, misconfigured, or unable to mount. Invalid Copilot request state identifies an incomplete/canceled tool-call history, Copilot request failed identifies provider execution failure, and MCP server unreachable identifies unavailable MCP guidance tools.

Back to logging sections