Is there a way to find and kill a tmux window from a different tmux session? Or do I need to attach to the other session and then find the window to kill? How do I do this with cmd line?
tmux kill-window ...?
Use -t to kill-window, look at https://man.openbsd.org/tmux#COMMANDS and https://man.openbsd.org/tmux#kill-window.
You can use list-sessions, list-windows or list-panes to find the pane or window you want.
You could also do it interactively using tree mode, C-b s or C-b w then find the window then x (or :kill-window - no C-b - if your tmux is too old to have x in tree mode).
kill-window -t sessionname:windowname