ulimit is made for this. You can setup defaults for ulimit on a per user or a per group basis in /etc/security/limits.conf
ulimit -v KBYTES sets max virtual memory size. I don't think you can give a max amount of swap. It's just a limit on the amount of virtual memory the user can use.
So you limits.conf would have the line (to a maximum of 4G of memory)
luser hard as 4000000