grep has different result in terminal command line and bash script

Question

I have a bash script that uses grep to screen out file with certain extensions and under certain directories. The script looks something like follows:

generate_exclud_extensions() {
  echo "\"(cpp$"
}

generate_exclude_dirs() {
  echo "|^test)\""
}

command=$(generate_exclud_extensions)$(generate_exclude_dir)
echo $command
git ls-files | grep -vE $command

This script was supposed to screen the files with cpp extension, AND files under test directory. However the result only screened out the first, had the second remained.

Fact 1: echo $command outputs "(cpp$|^test)", and running the full command git ls-files | grep -vE ""(cpp$|^test)"" in terminal, I got the correct result.

Fact 2: Tried using grep twice as follows:

git ls-files | grep -vE "cpp$" | grep -vE "^test"

same wrong result in bash script but correct in terminal.

I've been searching but didn't have an luck yet. I appreciate it if someone could shed some light on this.

Answer 1

I managed to reproduce the problem. It's a bash escaping problem in the two functions. Try this:

generate_exclude_extensions() {
  echo '(cpp$'
}

generate_exclude_dirs() {
  echo '|^test)'
}

command=$(generate_exclude_extensions)$(generate_exclude_dirs)
echo "$command"
git ls-files | grep -vEe "$command"

my code to reproduce

generate_exclude_extensions() {
  echo '(cpp$'
}

generate_exclude_dirs() {
  echo '|^test)'
}

command=$(generate_exclude_extensions)$(generate_exclude_dirs)
echo "$command"
echo -e 'cpp\n
test\n
test456\n
display?5\n' | grep -vEe "$command"

There are also two typos in your script

generate_exclude_dir -> generate_exclude_dirs
generate_exclud_extensions -> generate_exclude_extensions

:) have fun

Answer 2

Another approach could be:

generate_exclude_extensions() {
  echo 'cpp$'
}

generate_exclude_dirs() {
  echo '^test'
}

exclusions=$(
  generate_exclude_extensions
  generate_exclude_dir
)

printf '%s\n' "$exclusions"

git ls-files | grep -ve "$exclusions"

That is have one exclusion per line, and then you don't even need -E.

Answer 3

You could use Git exclude pathspecs to filter the output:

exclude_extensions=( '*cpp' '*.txt' ) # added `*.txt` as sample file extension
exclude_dirs=( 'test/*' )             # exclude files in relative directory `test`

git ls-files "${exclude_extensions[@]/#/:(exclude)}" "${exclude_dirs[@]/#/:(exclude)}"

The parameter expansion "${exclude_...[@]/#/:(exclude)}" expands to the space separated array entries prefixed by :(, the pathspec "magic word" exclude followed by ).

The resulting command is equivalent to

git ls-files ':(exclude)*cpp' ':(exclude)*.txt' ':(exclude)test/*'