Another take in a different direction (iff there are no spaces, tabs, newlines, single quotes, double quotes or backslash characters¹ in file names), using -f so as to avoid the prompts that couldn't be answered as the input comes from either a pipe or /dev/null depending on the xargs implementation:
ls | grep -xvF file.txt | xargs rm -f --
or, with GNU parallel (not the parallel from moreutils), which would have problems with fewer characters² but means running one sh and one rm for each file:
ls | grep -xvF file.txt | PARALLEL_SHELL=sh parallel rm -f --
For arbitrary file names, on recent GNU systems and with a shell with support for Korn-style process substitution, leaving rm's stdin unaffected:
xargs -r0a <(ls --zero | grep -zxvF file.txt) rm --
Though we can also do without grep as GNU ls has a --ignore option to skip entries matching a pattern:
xargs -r0a <(ls --zero --ignore=file.txt) rm --
from man grep on a GNU system:
-v, --invert-match
Invert the sense of matching, to select non-matching lines.
-F, --fixed-strings
Interpret PATTERNS as fixed strings, not regular expressions.
-x, --line-regexp
Select only those matches that exactly match the
whole line. For a regular expression pattern, this
is like parenthesizing the pattern and then
surrounding it with ^ and $.
-z, --null-data
Treat input and output data as sequences of lines, each
terminated by a zero byte (the ASCII NUL character) instead
of a newline. Like the -Z or --null option, this option can
be used with commands like sort -z to process arbitrary file
names.
(-x, -v, -F are standard, the rest including the long forms of the standard options are GNU extensions).
Without the -x we'd keep my-file.txt as well and without -F, we'd also keep file-txt.
¹ and with some xargs implementations, possibly other blank characters as classified as such in the locale, and no file called _ and no file that can't be decoded as text in the locale.
² Normally only newlines, but older versions of GNU parallel did not always properly quote all strings before passing to the shell so YMMV; we mitigate it here by forcing the shell to be sh with which the quoting is easier and syntax well known.
.gitdirectory, see: stackoverflow.com/a/22347541/565877