Reading multiple zipped files in awk script

Question

I have an awk script named text_processing.awk . I need to pass multiple zipped files named 01JAN21.txt.gz , 02JAN21.txt.gz etc. to this script for further processing. The output file takes the date from the input file and gets named result_01JAN21.txt and so on.

Please note the files are zipped.

I tried running on one file at first using command below

zcat 01JAN21.txt.gz | awk -f text_processing.awk -

However, my output file doesn't get named, I believe because this command just opens the file and pipes it to the script, whereas, my script extracts date from the input file name as below

BEGIN{ 
  FS = ";" 
  input_file = ARGV[1] 
  sub(/\.txt\.gz/, "", input_file) 
  output = "result_" input_file ".txt 
}

Will appreciate some guidance on how to go about rectifying it. I want to keep the renaming of output file within awk script because there are multiple files I will be passing at one go.

Answer 1

Assuming you want to process all files in the current directory that have names matching *.txt.gz. Pass the name as a variable into awk and stream the uncompressed data:

for name in *.txt.gz; do
    gzip -c -d -- "$name" |
    awk -v name="$name" -f text_processing.awk
done

In the awk code, you'd use the name variable to compute the output filename.

Alternatively, get the shell script to give you the explicit output filename to use:

for name in *.txt.gz; do
    gzip -c -d -- "$name" |
    awk -v outname="result_${name%.gz}" -f text_processing.awk
done

The string result_${name%.gz} would be the name of the original file with .gz removed and result_ prepended. The awk code then uses outname to write to.

Obviously, if your awk code only writes to a single output file, you may simplify the awk code even more and have it just print to its standard output. Then do the output redirection in the shell:

for name in *.txt.gz; do
    gzip -c -d -- "$name" |
    awk -f text_processing.awk >"result_${name%.gz}"
done

---

Extra material requested in comments: Running the awk command in parallel.

For this, I'm going to use xargs, and I will assume that this utility supports the non-standard options -0 (for reading nul-terminated data), -r (for not executing the given command if there is no input), and -P (for running jobs in parallel).

print '%s\0' *.txt.gz |
xargs -0r -P 4 -I {} sh -c '
    gzip -c -d -- "$1" |
    awk -f text_processing.awk >"result_${1%.gz}"' sh {}

This would run the gzip+awk pipeline for four files concurrently. Add -t to the xargs utility's options to get and indication of what commands it is spawning.