I'm being stupid... the tunnel uses the default interface with the lowest metric, it's not tired to a single interface.
As soon as eth0 comes back up, the data is sent via eth0.
This causes the connection to effectively fail, where I need to use ServerAliveInterval/ServerAliveCountMax to notice and close the connection.