VTDecoderXPCService memory leak (~13+ GB) while BetterDisplay is running

[eugeneprib]

Problem

VTDecoderXPCService randomly balloons to 13+ GB of RAM. Quitting BetterDisplay immediately frees the memory. Doesn't happen when BetterDisplay isn't running.

Environment

Mac: MacBook Pro, M1 Pro
macOS: Sequoia 15.7.5
BetterDisplay: 4.2.3
Display setup: Samsung G7 4k
Features in use: HiDPI scaling, 120/144hz refresh rate, override EDID

Reproduction

No reliable repro, happens randomly during normal use, no video playback or media apps involved.

Screenshot

Notes

I know VTDecoderXPCService is an Apple system process, not BetterDisplay itself. But the correlation is clear - it only leaks while the app is running, and quitting the app frees it instantly.

I saw related discussions in #3546 and #1348 but those didn't fully resolve the root cause.
I also have been running older version of BetterDisplay for couple years on previous versions of MacOs and on the latest too (Tahoe) - same thing has been happening for last several month, maybe more.

Thank you.

[waydabber]

Honestly I have zero idea why this would happen. VTDecoder is a decoder for VideoToolbox. But BetterDisplay does not use VideoToolbox at all (even for streaming and effects). Maybe some other app does use it which has an issue when certain features of BetterDisplay is being used? Normally a bunch of VTDecoderXPCServices may run for whatever apps that may use VT. I have at least 12 running right now - while BetterDisplay is not even running and I am not even playing any video anywhere - apps that might have anything to do with video just habitually have these ready just in case - like each Safari or Chrome tab may have one, an open QuickTime, Preview, IINA etc may have one. Any of these may get irritated for whatever installation specific reason with something BetterDisplay might be doing in a specific configuration. All in all, I am not sure this issue can be tracked from BetterDisplay end - the best approach would be trying to figure out what app or process is actually using that specific offending VTDecoderXPCService instance and then figure out what triggers this heavy use and why - and then file a bug report with that app.

[eugeneprib]

Thank you for looking into this question, I understand that this is a "shot in the dark". I'll keep looking into this on my end and I'll update with more info once I find anything. For now, restarting the app does the trick, so it's manageble.
Thanks!