If $1 is -d, the first condition is true. If $1 is -e, the second condition is true. If it's anything else, both are true. Since you joined them with || (or), the then branch of the if is taken if either is true. Therefore, the then branch is always taken.
You need to use if [ "$1" != "-e" ] && [ "$1 != "-d" ] to take the then branch only if $1 is not -e and is not -d.