VictoriaMetrics/docs/victoriametrics/query-stats.md

weight, title, menu, tags, aliases

weight	title	menu	tags	aliases
82	Query execution stats	docs parent weight victoriametrics 82	metrics enterprise	/query-stats.html /query-stats/index.html /query-stats/

VictoriaMetrics Enterprise supports logging statistics {{% available_from "v1.116.0" %}} for read queries made through the /api/v1/query and /api/v1/query_range APIs.

To enable query statistics logging, add the -search.logSlowQueryStats=<duration> command line flag to vmselect or Single-node VictoriaMetrics. Here's how <duration> works:

• -search.logSlowQueryStats=5s logs statistics for queries that take longer than 5s;
• -search.logSlowQueryStats=1us logs statistics for all queries;
• -search.logSlowQueryStats=0 turns off query stats logging (this is the default).

Example of a query statistics log:

2025-03-25T11:23:29.520Z        info    VictoriaMetrics/app/vmselect/promql/query_stats.go:60       vm_slow_query_stats type=instant query="vm_promscrape_config_last_reload_successful != 1\nor\nvmagent_relabel_config_last_reload_successful != 1\n" query_hash=1585303298 start_ms=1742901750000 end_ms=1742901750000 step_ms=300000 range_ms=0 tenant="0" execution_duration_ms=0 series_fetched=2 samples_fetched=163 bytes=975 memory_estimated_bytes=2032

Log fields

Each log entry contains the following fields:

• type: the type of the query: either instant or range;
• query: the executed MetricsQL query;
• query_hash: a hash of the query. This makes it easier to filter logs for a specific query;
• start_ms, end_ms, step_ms are query params described here;
• range_ms: a time range in milliseconds between start_ms and end_ms. If range_ms==0 it means this query is instant;
• tenant: a tenant ID. Available only in the cluster version;
• execution_duration_ms: time it took in milliseconds to execute the query (not including time spent sending results over the network);
• series_fetched: number of unique time series fetched. This may be higher than the number of series returned if there are filters like cpu_usage > 0;
• samples_fetched: number of data samples fetched;
• bytes: number of bytes transferred from storage to process the query;
• memory_estimated_bytes: estimated memory needed to run the query. See -search.maxMemoryPerQuery cmd-line flag.

Analysis

It is recommended to collect query statistics logs into VictoriaLogs for performance analysis. Other logging databases can also be used.

Every log entry starts with vm_slow_query_stats to simplify the filtering. The fields are formatted in logfmt to simplify the parsing.

Once logs are in VictoriaLogs, run the following query to find the top 5 slowest queries:

vm_slow_query_stats | extract 'vm_slow_query_stats <query_stats>' | unpack_logfmt from query_stats 
| stats by(query) max(execution_duration_ms) execution_duration_max 
| sort by(execution_duration_max) desc | limit 5

Here how query works:

• start with vm_slow_query_stats to filter only statistics logs;
• use extract and unpack_logfmt to parse the log message into fields;
• then calculate statistics, like the maximum execution time.

Other useful stats you can calculate:

| stats by(query) max(execution_duration_ms) execution_duration_max 

or

| stats by(query,query_hash) sum(series_fetched) series_fetched_sum

With VictoriaLogs Grafana datasource you can build a dashboard to view and filter query stats easily:

The example of Grafana datasource for VictoriaMetrics cluster is available here.