Install USBGuard — it provides a framework for authorising USB devices before activating them. With the help of a tool such as USBGuard Notifier or the USBGuard Qt applet, it can pop up a notification when you connect a new device, asking you what to do; and it can store permanent rules for known devices so you don’t have to confirm over and over. Rules are defined using a comprehensive language with support for any USB attribute (including serial number, insertion port...), so you can write rules that are as specific as you want — whitelist this keyboard if it has this identifier, this serial number, is connected to this port, etc.