I really can't see why the PROJECT variable is not getting expanded properly on you last line of code (unless you're looking in the wrong place for the generated file), but I do see that you don't properly double quote your variable expansions. Not double quoting these would cause issues as soon as $PROJECT contained spaces or newlines etc.

You are also jumping through a lot of hoops for something that could be done with a single awk program:

proj=$(<temp/project_name_final)

awk 'NR==FNR { species[$3]; next } /Bifidobacterium/ && !($2 in species) { print $2 }' \
    "${proj}_species.txt" "${proj}_genera.txt" >"temp/${proj}_selected_Bif"

This awk program reads the two files ${proj}_species.txt and ${proj}_genera.txt. While reading the first file, its third column is used to create a key in the associative array or hash species. When we then start reading the second file, we are only interested in lines that contain the string Bifidobacterium and whose second column is not a key in the species hash. For those lines, we output the second column.

All output goes to temp/${proj}_selected_Bif.

Note the double quoting of all expansions of the proj variable. I used a lower-cased variable name since upper-cased variables are reserved for system and shell environment variables.

See also

• Why does my shell script choke on whitespace or other special characters?

I really can't see why the PROJECT variable is not getting expanded properly on you last line of code (unless you're looking in the wrong place for the generated file), but I do see that you don't properly double quote your variable expansions. Not double quoting these would cause issues as soon as $PROJECT contained spaces or newlines, or any other character that is special to the shell.

You are also jumping through a lot of hoops for something that could be done with a single awk program:

proj=$(<temp/project_name_final)

awk 'NR==FNR { species[$3]; next } /Bifidobacterium/ && !($2 in species) { print $2 }' \
    "${proj}_species.txt" "${proj}_genera.txt" >"temp/${proj}_selected_Bif"

This awk program reads the two files ${proj}_species.txt and ${proj}_genera.txt. While reading the first file, its third column is used to create a key in the associative array or hash species. When we then start reading the second file, we are only interested in lines that contain the string Bifidobacterium and whose second column is not a key in the species hash. For those lines, we output the second column.

All output goes to temp/${proj}_selected_Bif.

Note the double quoting of all expansions of the proj variable. I used a lower-cased variable name since upper-cased variables are reserved for system and shell environment variables.

See also

• Why does my shell script choke on whitespace or other special characters?