In an extglob, !(.min) matches everything that isn't exactly .min, as long it doesn't contain a /; that includes the empty string!!
So, when you test foo/bar/baz/test.min.js against the glob foo/bar/**/*!(.min).js you get a hit because:
foo/bar/matchesfoo/bar/**/matchesbar/*matchestest.min!(.min)matches the empty string.jsmatches.js
If the meaning that you want from !(.min) is in fact "anything that doesn't end with .min", then in spoken language that is the negation of "anything that ends with .min" (which is *.min in glob terms). So can use !(*.min) to fix your issue:
foo/bar/**/!(*.min).js