Split several Unix files into multiple files based on first column

Question

I have several files (*data.txt) and I am trying to split each of them into multiple files based on the content of column 1. I have managed to split them but I do not know how to name the output files with both $filename and column 1($1) using print. At the moment print in the following command gives me $1 ".txt", so for example: ENSG00000108094.txt, ENSG00000115232.txt instead of file1_ENSG00000108094.txt, file1_ENSG00000115232.txt which is not suitable as I need to have separate outputs for each input file. Here it is my command and I am not sure where I should use "$b" to get the expected outcome.

for filename in *_data.txt
 do
    b=${filename%%_data.txt}
cat $filename | awk 'NR==1 {header = $0; next}!header_printed[$1]++ {print header > $1".txt"}{print > $1".txt"}'
done

Thanks.

Answer 1

There are several ways to pass a shell variable into an awk program:

1. using the -v command line option:

   awk -v b=${filename%data.txt} '... {print > (b $1 ".txt")}'
   

2. pass the value as an ordinary argument after the awk program

   awk '... {print > (b $1 ".txt")}' b=${filename%data.txt}
   

3. pass the value in the environment and access it within awk via the intenal ENVIRON array

   b=${filename%data.txt} awk '... {print > (ENVIRON["b"] $1 ".txt")}'
   

However, if you only have "several" files it might make sense to omit the shell loop altogether, and pass all the globbed files directly to awk, where you may derive the output file prefix from the FILENAME internal variable, ex.:

awk '
  FNR==1 {header = $0; b = FILENAME; sub(/data.txt$/,"",b); next}
  !header_printed[b $1]++ {print header > (b $1 ".txt")}
  {print > (b $1 ".txt")}
' *_data.txt

(you could also use split or substr to remove the data.txt suffix - I used regex sub because it is closest to the ${filename%data.txt} shell expansion).