Adding & spawns a background process.
If you write a; b, it will run command a, wait for it to finish, then run command b, in sequence. 
If you write a & b, it will spawn a as a background process. It will not wait for it to finish, and it will start running b immediately. It will run both at once.
You can see what it does by experimenting in the shell. If you have X installed, xterm is a good way to see what happens: typing
$ xterm
will cause another terminal window to open, and the first one will wait until you close it. Only when you close it will you get your shell back. If you type
$ xterm &
then it will run it in the background, and you will get your shell back immediately, while the xterm window will also remain open.
So if you write
echo "a fixed string" $i | nc localhost *port here* >> /tmp/me/dump.txt
it makes the connection, sends the string, stores what comes out in the file, and only then moves on to then next one.
Adding the & makes it not wait. It will end up running all ten thousand of them more or less simultaneously.
Your script seems to "end" more quickly, because it probably did not actually finish in that time. It just made ten thousand background jobs, and then ended the foreground one. 
This also means that, in your case, it will try to open ten thousand connections more or less at once. Depending on what the other end can handle, some of them might well fail. Not only that, but there is no guarantee that they will run in order, in fact they almost certainly won't, so what will actually end up in /tmp/me/dump.txt is anyone's guess. 
Did you check if the output was correct?
     
    
&makes the command run in the background, that's all. It didn't make it faster or anything. Read whatever shell's you're using (I assume bash) manual.for i in {1000..9999}waitat the end, though.nc -z localhost 1000-2000?