Just to give a different answer, you can use the default behavior of rm that it wont delete folders,
mkdir tmp && mv file.txt tmp # create tmp dir and move files there
rm # delete all other files
mv tmp/* . && rm -rf tmp # move all files back and delete tmp dir