The ulimitulimit shell command only applies to the current shell and its descendants. It's a built-in not a separate command.
You cannot start a separate shell process, run ulimitulimit in it, and expect it to have any effect on the parent process.
From C you should use the getrlimit(2)getrlimit(2) and setrlimit(2)setrlimit(2) functions to get and set the resource limits for the current process and its descendants. Though, judging by your approach, I guess they won't do what you expect them to do, either.