The dot operator has signature:
(.) :: (b -> c) -> (a -> b) -> a -> c
here the operator is used infix. So in your first statement you have actually written:
dotEx1 = (.) (map (+3)) (filter (>100))
And that makes sense since filter (>100) has signature: (Num n,Ord n) => [n] -> [n] and map (+3) has signature Num n => [n] -> [n]. If you however write:
dotEx1 xs = (.) (map (+3)) (filter (>100) xs)
then filter (>100) xs has signature (Num n,Ord n) => [n] and so this is a value (not a function, or perhaps a function with no arguments). So the dot operator cannot be used (the types do not match).
Informally the dot operator should be given two functions f and g and it generates a function where f is applied after g is applied on the input. But g thus has to be a function taking one argument.
(.). (I don't really think this one sentence adds enough value to warrant putting in yet another answer, though.)