you could limt memory usege with /etc/security/limits.conf
in this file you put :
domain type item value
where the domain is the @groupname, type is hard or soft where hard is limt that cannot be exceeded under any circumstances.
item is the item field specifies what type of item is being limited. Examples
include core (the size of core files), data (the size of a program’s data area), fsize (the size
of files created by the user), nofile (the number of open data files), rss (the resident set size),
stack (the stack size), cpu (the CPU time of a single process in minutes), nproc (the number
of concurrent processes), maxlogins (the number of simultaneous logins), and priority (the
process priority). The data, rss, and stack items all relate to memory consumed by a pro-
gram. These and other measures of data capacity are measured in kilobytes.
and value is relate to the item field that you have chose, for example if you have chose cpu in the item field and put 2 in the value then in case the domain (you group of VirtualBox) is take more than 2 of the cpu time it will be terminated.
You can use ulimit too, but it restricted to the bash shell only.
Hope this help you in you goal, although it isn't done with the method you asked.