The code in the event handlers loops over all panel elements, yet those would only really update classes for at most only two elements would have updates to their class list. See the suggestion below about ways to eliminate the loops.
The style font-size: 20px; shouldis not be needed under .panel since the same rule is also specified on html, plus it gets overridden by more specific selectors.