Your command does exclude files in .test/ and in other/.test/.
It doesn't exclude files in .config/dir1 because --exclude-dir only looks at the base name of the directory, not at its full path. You would need --exclude-dir="dir1", which excludes elsewhere/dir1 as well. Note that if a directory is excluded, its subdirectories are excluded as well, so --exclude-dir="dir1" also excludes files in .config/dir1/dir2.
If grep's exclude patterns aren't sufficient for you, you can combine find with grep. The syntax of find is more complex, but it's more powerful. For example, you can exclude .config/dir1 while not excluding other directories called dir1. (I assume that you have GNU find since you have GNU grep.)
find . \
-path "./.config/dir1" -prune -o \
-name ".test" -prune -o \
-type f -print0 |
xargs -0 grep -io pattern