You will need to create a string which contains the complete script that you want run, or you will need to create a script which can be run simply, and then arrange to read the output of that script with popen(). Either is possible; which is easier depends on the level of your scripting skills vs your C programming skills.
char command[4096];
strcpy(command, "TOTAL=$(free | grep Mem| awk '{print $2}')\n");
strcat(command, "grep -v procs $1 | grep -v free |\n");
strcat(command, "awk '{USED=TOTAL-$4-$5-$6;print USED}' TOTAL=$TOTAL\n");
FILE *in = popen(command, "r");
...read the results, etc...
The string operations simplify the first shell script, and then pass the value of TOTAL calculated to awk.
The other way to do it would be to read the output from free | grep Mem | awk '{print $2}' - the value that is TOTAL - from one use of popen(), then build that value into the second command:
char command[4096];
strcpy(command, "free | grep Mem| awk '{print $2}'");
char total[20];
FILE *in1 = popen(command, "r");
...read TOTAL into total...
strcpy(command, "grep -v procs $1 | grep -v free |\n");
strcat(command, "awk '{USED=TOTAL-$4-$5-$6;print USED}' TOTAL=");
strcat(command, total);
FILE *in2 = popen(command, "r");