How to call bash function from within awk?

Question

This is a bit tricky; I'm trying to work out the best approach to this problem. I have a couple of approaches, but they seem really hacky and I'd like something a little more elegant.

I want to parse a whitespace delimited file, ignoring #comment lines and complaining of any non-empty lines that don't have exactly 4 fields. This is easy enough in awk:

awk '/^#/ {next}; NF == 0 {next}; NF != 4 {exit 1}; (dostuff)'

The trick is what I want to do with the data, is actually set it as variables in bash and then run a bash function, unless $2 contains a specific value.

Here is some pseudocode (mostly real but mixed languages) to explain what I mean:

# awk
/^#/ {next}
NF == 0 {next}
NF != 4 {exit 1}
$2 == "manual" {next}
# bash
NAME=$1
METHOD=$2
URL=$3
TAG=$4
complicated_bash_function_that_calls_lots_of_external_commands
# then magically parse the next line with awk.

I don't know how to do this without some ugly workarounds, such as calling awk or sed separately for each line of the file. (Originally I put the question as "How to call bash function from within awk or each output line of awk from within bash?")

Possibly it would work to modify the bash function into its own script, and make it accept arguments 1, 2, 3, 4 as above. I'm not sure how to call that from within awk, though; hence my question title.

What I would actually prefer to do, is have the whole thing in one file and make it a bash script - calling awk from within bash rather than bash from within awk. But I will still need to call the bash function from within awk--once for each non-comment line of the input file.

How can I do this?

Answer 1

You may be able to do what you want by piping awk's output into a while read loop. For example:

awk '/^#/ {next}; NF == 0 {next}; NF != 4 {exit 1} ; {print}' | 
    while read -r NAME METHOD URL TAG ; do
        :  # do stuff with $NAME, $METHOD, $URL, $TAG
        echo "$NAME:$METHOD:$URL:$TAG"
    done

if [ "$PIPESTATUS" -eq 1 ] ; then
    : # do something to handle awk's exit code
fi

Tested with:

$ cat input.txt 
# comment
NAME METHOD URL TAG
a b c d
1 2 3 4
x y z
a b c d

$ ./testawk.sh input.txt 
NAME:METHOD:URL:TAG
a:b:c:d
1:2:3:4

Note that it correctly exits on the fifth x y z input line.

---

It's worth pointing out that because the while loop is the target of a pipe, it executes in a sub-shell and is therefore unable to alter the environment (including environment variables) of its parent script.

If that is required, then don't use a pipe, use redirection and process substitution instead:

while read -r NAME METHOD URL TAG ; do
  :  # do stuff with $NAME, $METHOD, $URL, $TAG
  echo "$NAME:$METHOD:$URL:$TAG"
done < <(awk '(/^#/ || NF == 0) {next};
              NF != 4 {
                printf "%s:%s:Wrong number of fields\n", FILENAME, NR > "/dev/stderr";
                exit 1
               };
              {print}' input.txt)

# getting the exit code from the <(...) requires bash 4.4 or newer:
wait $!

if [ "$?" -ne 0 ] ; then
 : # something went wrong in the process substitution, deal with it
fi

Alternatively, you can use the coproc built-in to run the awk script in the background as a co-process:

# By default, array var $COPROC holds the co-process' stdout and
# stdin file descriptors.   See `help coproc`.
coproc {
  awk '(/^#/ || NF == 0) {next};
       NF != 4 {
         printf "%s:%s:Wrong number of fields\n", FILENAME, NR > "/dev/stderr";
         exit 1
       };
       {print}' input.txt
}
awkpid="$!"
#declare -p COPROC # uncomment to see the FDs

while read -r NAME METHOD URL TAG ; do
  echo "$NAME:$METHOD:$URL:$TAG"
done <&"${COPROC[0]}"

wait "$awkpid"
echo "$?"

Answer 2

cas's answer is good, but if you actually need to parse the output in awk again and want to do this from within the first awk command you have a fantastic pipe command syntax in awk:

awk '
{
  cmd = "echo name:tag:url:method" # (very simple example)
  while (cmd | getline)
  {
    #process output ($0)
    print
  }
  close(cmd)
}
'