This is difficult to do as sed does not support look-arounds etc. (as you can do in a PCRE). It would be easier to reverse the string and replace the third occurrence of the reversed word from the start, then reverse again.
$ echo 'dog and foo and bar and baz land good' | rev | sed 's/\<dna\>/XXX/3' | rev
dog XXX foo and bar and baz land good
As for why your expression does not work, this looks like a bug. The back-reference \3 seem to be the string baz land, as if the \b before and in .*\band\b never had any effect.
The command
sed -E 's/(.*)\<and\>((.*\<and\>){2})/\1XYZ\2/'
seems to do the right thing on OpenBSD with its native sed (which uses \< and \> instead of \b).
I have yet to find an existing bug report against GNU sed or GNU glibc about this, although I wouldn't be surprised if it was at least related to glibc bug 25322 (because, see below).
You can work around it by being a bit more verbose:
sed -E 's/(.*)\band\b(.*\band\b.*\band\b)/\1XYZ\2/'