In an extglob, !(.min) matches anything that isn't exactly .min (as long it doesn't contain a /), and that includes the empty string!!!
So, when you test foo/bar/baz/test.min.js against the extglob foo/bar/**/*!(.min).js you get a hit because:
foo/bar/matchesfoo/bar/**/matchesbaz/*matchestest.min!(.min)matches the empty string.jsmatches.js
If the meaning that you want from !(.min) is in fact "anything that doesn't end with .min" then, in spoken language, that is the negation of "anything that ends with .min", which (which translates to *.min in glob terms;terms); so you just have to use !(*.min) to fix your issue:
foo/bar/**/!(*.min).js