With grep:

grep -E '^[#!]|.{4}'

That is, select lines that either start with # or ! or contain a sequence of 4 characters.

Or with awk:

awk '/^[#!]/ || length >= 4'

Or with sed:

sed -e '/^[#!]/b' -e '/.\{4\}/!d'

With grep:

< file.in grep -E '^[#!]|.{4}' > file.out

That is, select lines that either start with # or ! or contain a sequence of 4 characters.

Or with awk:

< file.in awk '/^[#!]/ || length >= 4' > file.out

Or with sed:

< file.in sed -e '/^[#!]/b' -e '/.\{4\}/!d' > file.out