Describe the bug
Hi,
Thank you for releasing cluster mode in version 1.18.0
I have tried the cluster mode as described in the documentation: https://docs.victoriametrics.com/victorialogs/cluster/#quick-start.
I found an unexpected behavior. When you run VL with two storage nodes and one of them is down, you cannot query the working storage node. You will encounter issues like this.
It should be handled so that if one backend is down, you can still query the other backend. This is essential during upgrades or restarts, as it prevents a complete outage. For kubernetes upgrade/restart it's very important. In VictoriaMetrics, this is managed through partition warnings.
P.S.
I checked VLinsert and it works fine it's re-routing writes request to available storage node
2025-04-10T17:22:10.276Z warn VictoriaMetrics/app/vlstorage/netinsert/netinsert.go:181 cannot send data block with the length 2069688 to "http://localhost:9492/internal/insert?version=v1": Post "http://localhost:9492/internal/insert?version=v1": dial tcp4 127.0.0.1:9492: connect: connection refused; try -enableTCP6 command-line flag for dialing ipv6 addresses; re-routing the data block to the remaining nodes
2025-04-10T17:22:20.256Z warn VictoriaMetrics/app/vlstorage/netinsert/netinsert.go:181 cannot send data block with the length 2083954 to "http://localhost:9492/internal/insert?version=v1": Post "http://localhost:9492/internal/insert?version=v1": dial tcp4 127.0.0.1:9492: connect: connection refused; try -enableTCP6 command-line flag for dialing ipv6 addresses; re-routing the data block to the remaining nodes
2025-04-10T17:22:31.211Z warn VictoriaMetrics/app/vlstorage/netinsert/netinsert.go:181 cannot send data block with the length 1837217 to "http://localhost:9492/internal/insert?version=v1": Post "http://localhost:9492/internal/insert?version=v1": dial tcp4 127.0.0.1:9492: connect: connection refused; try -enableTCP6 command-line flag for dialing ipv6 addresses; re-routing the data block to the remaining nodes
Thank you
To Reproduce
- Run VLogs in cluster mode as described in the VictoriaMetrics documentation.
- Stop one storage.
- Open the VL UI or query the logs.
curl localhost:9471/select/logsql/query -N -d 'query=_time:7d *'
remoteAddr: "127.0.0.1:60290"; requestURI: /select/logsql/query?query=_time%3A7d+%2A; cannot execute query [_time:7d]: Get "http://localhost:9492/internal/select/query?disable_compression=false&query=_time%3A7d&tenant_ids=%00%00%00%00%00%00%00%00×tamp=1744306476058645999&version=v1": dial tcp4 127.0.0.1:9492: connect: connection refused; try -enableTCP6 command-line flag for dialing ipv6 addresses
Version
victoria-logs-20250410-150037-tags-v1.18.0-victorialogs-0-g7a46af3920
Logs
025-04-10T17:25:00.946Z warn VictoriaMetrics/app/vlselect/logsql/logsql.go:296 remoteAddr: "127.0.0.1:60168"; requestURI: /select/logsql/field_names?query=*&limit=1000&start=1743627600&end=1744318799.999; cannot obtain field names: Get "http://localhost:9492/internal/select/field_names?disable_compression=false&query=_time%3A%5B2025-04-02T21%3A00%3A00Z%2C+2025-04-10T20%3A59%3A59.999Z%5D+%2A&tenant_ids=%00%00%00%00%00%00%00%00×tamp=1744318799998999999&version=v1": dial tcp4 127.0.0.1:9492: connect: connection refused; try -enableTCP6 command-line flag for dialing ipv6 addresses
2025-04-10T17:33:56.479Z warn VictoriaMetrics/app/vlselect/logsql/logsql.go:936 remoteAddr: "127.0.0.1:60281"; requestURI: /select/logsql/query?query=_time%3A7d+%2A; cannot execute query [_time:7d]: Get "http://localhost:9492/internal/select/query?disable_compression=false&query=_time%3A7d&tenant_ids=%00%00%00%00%00%00%00%00×tamp=1744306436479007999&version=v1": dial tcp4 127.0.0.1:9492: connect: connection refused; try -enableTCP6 command-line flag for dialing ipv6 addresses
Screenshots
No response
Used command-line flags
No response
Additional information
No response
Describe the bug
Hi,
Thank you for releasing cluster mode in version 1.18.0
I have tried the cluster mode as described in the documentation: https://docs.victoriametrics.com/victorialogs/cluster/#quick-start.
I found an unexpected behavior. When you run VL with two storage nodes and one of them is down, you cannot query the working storage node. You will encounter issues like this.
It should be handled so that if one backend is down, you can still query the other backend. This is essential during upgrades or restarts, as it prevents a complete outage. For kubernetes upgrade/restart it's very important. In VictoriaMetrics, this is managed through partition warnings.
P.S.
I checked VLinsert and it works fine it's re-routing writes request to available storage node
Thank you
To Reproduce
Version
victoria-logs-20250410-150037-tags-v1.18.0-victorialogs-0-g7a46af3920
Logs
Screenshots
No response
Used command-line flags
No response
Additional information
No response