Skip to content

metrics: Show the latency needed to handle a msg per op#1011

Merged
tpapagian merged 2 commits into
mainfrom
pr/apapag/latency
May 30, 2023
Merged

metrics: Show the latency needed to handle a msg per op#1011
tpapagian merged 2 commits into
mainfrom
pr/apapag/latency

Conversation

@tpapagian

@tpapagian tpapagian commented May 19, 2023 •
edited
Loading

Copy link
Copy Markdown
Member

Example:

# HELP tetragon_handling_latency The latency of handling messages in us.
# TYPE tetragon_handling_latency histogram
tetragon_handling_latency_bucket{op="23",le="50"} 3613
tetragon_handling_latency_bucket{op="23",le="100"} 4126
tetragon_handling_latency_bucket{op="23",le="500"} 4130
tetragon_handling_latency_bucket{op="23",le="1000"} 4130
tetragon_handling_latency_bucket{op="23",le="+Inf"} 4131
tetragon_handling_latency_sum{op="23"} 106430
tetragon_handling_latency_count{op="23"} 4131
tetragon_handling_latency_bucket{op="24",le="50"} 3572
tetragon_handling_latency_bucket{op="24",le="100"} 3608
tetragon_handling_latency_bucket{op="24",le="500"} 3610
tetragon_handling_latency_bucket{op="24",le="1000"} 3610
tetragon_handling_latency_bucket{op="24",le="+Inf"} 3610
tetragon_handling_latency_sum{op="24"} 56959
tetragon_handling_latency_count{op="24"} 3610
tetragon_handling_latency_bucket{op="5",le="50"} 1390
tetragon_handling_latency_bucket{op="5",le="100"} 3473
tetragon_handling_latency_bucket{op="5",le="500"} 3610
tetragon_handling_latency_bucket{op="5",le="1000"} 3610
tetragon_handling_latency_bucket{op="5",le="+Inf"} 3610
tetragon_handling_latency_sum{op="5"} 216082
tetragon_handling_latency_count{op="5"} 3610
tetragon_handling_latency_bucket{op="7",le="50"} 2895
tetragon_handling_latency_bucket{op="7",le="100"} 4106
tetragon_handling_latency_bucket{op="7",le="500"} 4124
tetragon_handling_latency_bucket{op="7",le="1000"} 4124
tetragon_handling_latency_bucket{op="7",le="+Inf"} 4124
tetragon_handling_latency_sum{op="7"} 162039
tetragon_handling_latency_count{op="7"} 4124```
@tpapagian
tpapagian requested a review from a team as a code owner May 19, 2023 11:15
@tpapagian
tpapagian requested a review from willfindlay May 19, 2023 11:15
@tpapagian tpapagian changed the title metrics: Show the latency that Tetragon needs to handle a msg per op metrics: Show the latency needed to handle a msg per op May 19, 2023
@cilium cilium deleted a comment from netlify Bot May 19, 2023
kkourt
kkourt approved these changes May 25, 2023
View reviewed changes

@kkourt kkourt left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

Does it make sense to hide this behind a flag?
Some of our requests seem to be <50ns, so I'm not sure if it's worth the effort of taking two timestamps by default.
Comment thread pkg/metrics/opcodemetrics/opcodemetrics.go Outdated
@tpapagian
metrics: Show the latency that Tetragon needs to handle a msg per op
fbf6950 
Example:
tetragon_handling_latency_bucket{op="23",le="50"} 3613
tetragon_handling_latency_bucket{op="23",le="100"} 4126
tetragon_handling_latency_bucket{op="23",le="500"} 4130
tetragon_handling_latency_bucket{op="23",le="1000"} 4130
tetragon_handling_latency_bucket{op="23",le="+Inf"} 4131
tetragon_handling_latency_sum{op="23"} 106430
tetragon_handling_latency_count{op="23"} 4131
tetragon_handling_latency_bucket{op="24",le="50"} 3572
tetragon_handling_latency_bucket{op="24",le="100"} 3608
tetragon_handling_latency_bucket{op="24",le="500"} 3610
tetragon_handling_latency_bucket{op="24",le="1000"} 3610
tetragon_handling_latency_bucket{op="24",le="+Inf"} 3610
tetragon_handling_latency_sum{op="24"} 56959
tetragon_handling_latency_count{op="24"} 3610
tetragon_handling_latency_bucket{op="5",le="50"} 1390
tetragon_handling_latency_bucket{op="5",le="100"} 3473
tetragon_handling_latency_bucket{op="5",le="500"} 3610
tetragon_handling_latency_bucket{op="5",le="1000"} 3610
tetragon_handling_latency_bucket{op="5",le="+Inf"} 3610
tetragon_handling_latency_sum{op="5"} 216082
tetragon_handling_latency_count{op="5"} 3610
tetragon_handling_latency_bucket{op="7",le="50"} 2895
tetragon_handling_latency_bucket{op="7",le="100"} 4106
tetragon_handling_latency_bucket{op="7",le="500"} 4124
tetragon_handling_latency_bucket{op="7",le="1000"} 4124
tetragon_handling_latency_bucket{op="7",le="+Inf"} 4124
tetragon_handling_latency_sum{op="7"} 162039
tetragon_handling_latency_count{op="7"} 4124

Signed-off-by: Anastasios Papagiannis <tasos.papagiannnis@gmail.com>
@tpapagian
tpapagian force-pushed the pr/apapag/latency branch from a428ed7 to 41ee791 Compare May 26, 2023 12:58
@tpapagian
metrics: add --enable-msg-handling-latency
8aeee6f 
We add a flag to enable monitoring the latency of handling messages.
Disabled by default.

Signed-off-by: Anastasios Papagiannis <tasos.papagiannnis@gmail.com>
@tpapagian
tpapagian force-pushed the pr/apapag/latency branch from 41ee791 to 8aeee6f Compare May 26, 2023 12:59
@tpapagian

tpapagian commented May 26, 2023

Copy link
Copy Markdown
Member Author

Does it make sense to hide this behind a flag? Some of our requests seem to be <50ns, so I'm not sure if it's worth the effort of taking two timestamps by default.

I have added --enable-msg-handling-latency flag to enable that (disabled by default).
@tpapagian
tpapagian merged commit 066f7d4 into main May 30, 2023
@tpapagian
tpapagian deleted the pr/apapag/latency branch May 30, 2023 07:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

3 participants

@tpapagian @kkourt @willfindlay