Just to give a different answer, you can use the default behavior of rm that it wontwon't delete folders,:
mkdir tmp && mv file.txt tmp # create tmp dir and move files there
rm # delete all other files
mv tmp/* . && rm -rf tmp # move all files back and delete tmp dir