You are getting bar in there because that's what $1 is after your shift operation before the loop.
To shift off two positional parameters, use shift 2. However, it may be safer to clear the list of positional parameters before the loop by replacing shift with set --. That way, you don't rely on the user supplying the correct number of parameters. Some shells would issue a diagnostic message if you tried to shift off more values than in the list, and if the user gives you too few or too many values, these would still be in "$@".
Also, consider usingyou will need to use set -- in place of plain set when setting positional parameters, both in this case and in general, to avoid accidentally setting shell options or, as in this case, invoking set with invalid options.
set -- "$@" --build-arg "$line"