There is the setrlimit() function, which allows to configure a process' limits in C. The idea would be to write a C program which calls setrlimit then execs the command you want to be limited.

Luckily someone already wrote something similar. It can be downloaded from freshmeat. I had a quick look at the source code and it seems to be fine. Use rlimit at your own discretion. Note that rlimit also cannot change other processes' limits.

There is the setrlimit() function, which allows to configure a process' limits in C. Write a C program to call setrlimit then to exec the command you want to be limited. setrlimit cannot change other processes' limits.

Luckily someone already wrote something similar. It can be downloaded from freshmeat. I had a quick look at the source code and it seems to be fine. Use rlimit at your own discretion. Note that rlimit also cannot change other processes' limits.

Edit: Gilles proposed a nice hack with gdb: Attach to the process with gdb then make the process call setrlimit. This would perhaps solve the problem to limit an already running process.