Ok I finally figured out how to list all hidden dirs while preserving the Colors and not including dirs like "hello.world":
ls -lhAF1 --color | grep -E "^d[rwx-]{9}.*[0-9]{2}:[0-9]{2} "$'\x1b'"[([0-9]{1,2}m?)(;[0-9]{1,2}m?)?."
ls -lhAF1 --color | grep -E "^d[rwx-]{9}.*[0-9]{2}:[0-9]{2} "$'\x1b'"\[([0-9]{1,2}m?)(;[0-9]{1,2}m?)?\."