ulimit is made for this.
You can setup defaults for ulimit on a per user or a per group basis in
/etc/security/limits.conf
ulimit -v KBYTES sets max virtual memory size. I don't think you can give a max amount of swap. It's just a limit on the amount of virtual memory the user can use.
So you limits.conf would have the line (to a maximum of 4G of memory)
luser hard as 4000000
UPDATE - CGroups
The limits imposed by ulimit and limits.conf is per process. I definitely wasn't clear on that point.
If you want to limit the total amount of memory a users uses (which is what you asked). You want to use cgroups.
In /etc/cgconfig.conf:
group memlimit {
memory {
memory.limit_in_bytes = 4294967296;
}
}
This creates a cgroup that has a max memory limit of 4GiB.
In /etc/cgrules.conf:
luser memory memlimit/
This will cause all processes run by luser to be run inside the memlimit cgroups created in cgconfig.conf.
