There is the setrlimit() function, which allows to configure a process' limits in C. The idea would be to write a C program which calls setrlimit then execs the command you want to be limited.
Luckily someone already wrote something similar. It can be downloaded from freshmeat. I had a quick look at the source code and it seems to be fine. Use rlimit at your own discretion. Note that rlimit also cannot change other processes' limits.