Skip to content

vlogs: add query execution stats to help diagnose slow queries #52

@func25

Description

@func25

Is your feature request related to a problem? Please describe

VictoriaLogs currently lacks detailed visibility into why some queries are slow, especially full-text search queries without stream or time filters. Users often report performance issues (e.g., multi-second or timeout-level delays) when executing broad queries, but there's no built-in way to analyze internal query behavior.

This makes it hard to:

  • Determine whether slowness is caused by disk I/O, memory usage, or data volume
  • Understand how many data blocks or bloom filters were scanned
  • Guide users on how to optimize their queries

Describe the solution you'd like

Introduce query execution stats, which collect and return low-level performance metrics for a single query execution.

When enabled via a flag (e.g., query_stats=1), the following information should be collected and returned (likely via HTTP response headers):

  • Number of data blocks scanned
  • Number of bloom filters read
  • Total bytes read (from bloom filters and data blocks)
  • Time spent on bloom filter evaluation
  • Time spent reading data blocks
  • Time spent processing matching log entries
  • Count of matching rows
  • Number of log entries returned

Design Notes:

  • By default, this should be disabled to avoid performance overhead.
  • Can be enabled per-query via an HTTP query parameter (?query_stats=1)
  • Response headers (e.g., X-VictoriaLogs-Stats-Blocks-Scanned) should be used to return the data
  • Web UI and Grafana plugin could later surface this info in a readable way

Describe alternatives you've considered

Proposal: add a pipeline stage that emits query stats as the final record.

some_query | query_stats

query_stats wraps execution, collects counters while the query runs, and after all normal results are streamed, writes one extra, synthetic record with a reserved marker so UIs can detect it. Streaming stays as-is and the stats arrive last and can be rendered by the vmui without special transport tricks. For example:

{"message":"...regular log row..."}
{"message":"...regular log row..."}
{"@type":"query_stats","duration_ms":412,"bytes_read":524288,"files_read":37,"blocks_scanned":910,"rows_matched":17842,"rows_returned":17842,"cache_hits":121,"cache_misses":9}

Additional information

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions