That's because there is no operator &>&.
There is an operator &> word that redirects both stdout and stderr to word.
There is an operator [n]>& word that duplicates fd n (default 1) from fd word. As a special case, if n is empty, and word isn't a number, redirects both stdout and stderr to word.
But that's it. There isn't a special syntax for file descriptors which you can combine with operators. There are just the operators, which can interpret their operands as file descriptors or filenames. And there isn't a &>& operator which redirects both stdout and stderr, and interprets its right operand as a file descriptor.
Summary: There is no special syntax &n for file descriptors. The & belongs to the operator, not to the operand. There is no operator &>&.