Redirect input for gawk to a system command

Question

Usually a gawk script processes each line of its stdin. Is it possible to instead specify a system command in the script use the process each line from output of the command in the rest of the script?

For example consider the following simple interaction:

$ { echo "abc"; echo "def"; } | gawk '{print NR ":" $0; }'
1:abc
2:def

I would like to get the same output without using pipe, specifying instead the echo commands as a system command.

I can of course use the pipe but that would force me to either use two different scripts or specify the gawk script inside the bash script and I am trying to avoid that.

UPDATE

The previous example is not quite representative of my usecase, this is somewhat closer:

$ { echo "abc"; echo "def"; } | gawk '/d/ {print NR ":" $0; }'
2:def

UPDATE 2

A shell script parallel would be as follows. Without the exec line the script would read from stdin; with the exec it would use the command that line as input:

/tmp> cat t.sh
#!/bin/bash

exec 0< <(echo abc; echo def)
while read l; do
  echo "line:" $l
done
/tmp> ./t.sh 
line: abc
line: def

You can say gawk '{print NR ":" $0; }' < <(echo "abc"; echo "def"), for example. — fedorqui
– fedorqui, Commented Feb 10, 2015 at 10:21
@fedorqui this is almost identical to the pipe usage. I am looking to put the commands inside the awk script. — Miserable Variable
– Miserable Variable, Commented Feb 10, 2015 at 14:46
Then you are looking for call a shell command from inside awk and pass some awk variables to the shell command — fedorqui
– fedorqui, Commented Feb 10, 2015 at 14:50
@fedorqui I am not able to relate that to my usecase. Please see my updated example. — Miserable Variable
– Miserable Variable, Commented Feb 10, 2015 at 14:57
@MiserableVariable I adjusted my answer for your second example. — Tiago Lopo
– Tiago Lopo, Commented Feb 10, 2015 at 15:09

Ed Morton · Accepted Answer · 2015-02-13 13:31:24Z

2

From all of your comments, it sounds like what you want is:

$ cat tst.awk
BEGIN {
    if ( ("mktemp" | getline file) > 0 ) {
        system("(echo abc; echo def) > " file)
        ARGV[ARGC++] = file
    }
    close("mktemp")
}

{ print FILENAME, NR, $0 }

END {
    if (file!="") {
        system("rm -f \"" file "\"")
    }
}

$ awk -f tst.awk
/tmp/tmp.ooAfgMNetB 1 abc
/tmp/tmp.ooAfgMNetB 2 def

but honestly, I wouldn't do it. You're munging what the shell is good at (creating/destroying files and processes) with what awk is good at (manipulating text).

answered Feb 13, 2015 at 13:31

Ed Morton

208k18 gold badges90 silver badges212 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Miserable Variable Over a year ago

This is almost exactly what I meant when I wrote that I will redirect it to a temp file. You are quite correct that this is abusing awk, my choices are (i) use a shell script with inline awk script (ii) use an approach as above. I will have to decide how important syntax coloring is to me :)

Ed Morton Over a year ago

@MiserableVariable it's actually just a decision between doing it the right way or the wrong way. The UNIX philosophy is to have a bunch of small tools each of which does one thing well, and a shell to sequence the calls to them. There is some overlap between the tools and the shell in what you CAN do but if you just stick to the right tool for every job you get a much more concise, robust and efficient result every time.

Miserable Variable Over a year ago

with respect, I would like to point out that no such philosophical system can be entirely consistent and correct and there are cases where rules need to be broken. What you consider simple -- using a shell script and a separate awk script may be unnecessarily complexity under some circumstances. In any cases, SO is on the forum for discussing subjective opinions, and this is borderline subjective. I continue to be grateful for your help.

Tiago Lopo · Accepted Answer · 2015-02-10 15:30:50Z

1

I believe what you're looking for is getline:

awk '{ while ( ("echo abc; echo def" | getline line) > 0){ print line} }' <<< ''
abc
def

Adjusting the answer to you second example:

awk '{ while ( ("echo abc; echo def" | getline line) > 0){ counter++; if ( line ~ /d/){print counter":"line} } }' <<< ''
2:def

Let's break it down:

awk '{ 
       cmd = "echo abc; echo def"

       # line below will create a line variable containing the ouptut of cmd
       while ( ( cmd | getline line) > 0){ 

          # we need a counter because NR will not work for us
          counter++; 

          # if the line contais the letter d
          if ( line ~ /d/){ 
             print counter":"line
          } 
        } 
    }' <<< ''
    2:def

edited Feb 10, 2015 at 15:30

answered Feb 10, 2015 at 10:18

Tiago Lopo

8,0291 gold badge39 silver badges53 bronze badges

8 Comments

Miserable Variable Over a year ago

This works perfectly for my example but I am not able to figure out how to use other awk constructs with this. I will update my question with a more complex example.

Miserable Variable Over a year ago

BTW what does the <<< do?

Tiago Lopo Over a year ago

awk needs STDIN to work, this just sends empty string, same as echo "" | awk {dosomething}

Miserable Variable Over a year ago

I wanted to avoid series of if/else if. I will make one final update to the question using the example using a shell script parallel

jas Over a year ago

Fwiw, if you make it a BEGIN block you don't need the <<< ''.

|

Collectives™ on Stack Overflow

Redirect input for gawk to a system command

2 Answers 2

3 Comments

8 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

8 Comments

Linked

Related