Don't test whether a HashSet contains a key if you want to remove it.
Instead, just straight try to remove it, halving your work.:
if(list.Contains(value)) { list.Remove(value); }
BecomesInstead, just straight try to remove it:
list.Remove(value);
The replacement is cleaner, shorter, and more efficient. What's not to like?