That is how PowerShell formats output. I have complained on several occasions about the excess blank lines before and after output. If you want to avoid that, then you format the output yourself. You can do that like so:
$res = @(Get-ADGroup $GroupName -Properties Mail | Select-Object Mail)
for ($i = 0; $i -lt $res.Length - 1 + 4; $i += 4) {
"{0,-28} {1,-28} {2,-28} {3,-28}" -f $res[$i].Mail,$res[$i+1].Mail,$res[$i+2].Mail,$res[$i+3].Mail
}
This assumes your current console is 120 chars wide. If it is 80, change the -28 above to -18.
BTW the key point here is that PowerShell deals in objects and when it renders those objects to the screen it has a formatting engine that determines things like blank lines before and after the output. If you don't like PowerShell's default formatting, you're free to format objects (displaying whichever properties you want) as you please but it is a bit more work.
All that said, if the command returns only one object, why not just do this:
(Get-ADGroup $GroupName -Properties Mail).Mail
The Select-Object Mail, Format-Wide and Out-String are not necessary. Heck, with PowerShell V3 this will work even if the command returns multiple objects.