I don't think you can do this through curl. You would need to run curl -l to list the directory contents, count the matches, sort by date, and issue remote rm commands with curl -Q.
If your system supports FUSE (most do), I recommend to mount the remote directory as a filesystem. This approach lets you split your problem into two separate tasks: access the remote files by mounting them, and work on them as you would on local files. You can use curlftpfs for the remote mounting.
mkdir ftp.example.com
curlftpfs ftp.example.com ftp.example.com
cp -p /path/to/local/file ftp.example.com/remote/dir/
Now, to find the oldest files in a directory, the easiest way is to use zsh and its glob qualifiers. Working in the directory ftp.example.com/remote/dir/:
files=(*(mm)) # Om = sort by date, youngest first
if ((${#files} > 5)); then
rm $files[-1] # remove the last file
# rm $files[6,-1] # remove all but the 5 youngest files
fi