It's not working because you are attempting to nest backticks. That isn't possible. Note how the syntax highlighting breaks for your command on the second set ofunescaped backticks:
VARIA=`head -$((${RANDOM} % `wc -l < file` + 1)) file | tail -1`
That actually attempts to run head -$((${RANDOM} % as a single command first, and that gives you the 2 first errors:
$ VARIA=`head -$((${RANDOM} % `
bash: command substitution: line 1: unexpected EOF while looking for matching `)'
bash: command substitution: line 2: syntax error: unexpected end of file
Then, it tries to run
wc -l < file` + 1)) file | tail -1`
Which means it tries to evaluate + 1)) file | tail -1 (which is between the backticks), and that gives you the next errors:
$ wc -l < file` + 1)) file | tail -1`
bash: command substitution: line 1: syntax error near unexpected token `)'
bash: command substitution: line 1: ` + 1)) file | tail -1'
You can get around this by escaping the backticks:
VARIA=`head -$((${RANDOM} % \`wc -l < file\` + 1)) file | tail -1`
However, as a general rule, it is usually better not to use backticks at all. You should almost always use $() instead. It is more robust and can be nested indefinitely with a simpler syntax:
VARIA=$(head -$((${RANDOM} % $(wc -l < file) + 1)) file | tail -1)