60

I am trying to run a shell command from within awk for each line of a file, and the shell command needs one input argument. I tried to use system(), but it didn't recognize the input argument.

Each line of this file is an address of a file, and I want to run a command to process that file. So, for a simple example I want to use 'wc' command for each line and pass $1to wc.

awk '{system("wc $1")}' myfile
2
  • 2
    Why do you think awk is the right tool for this job? It seems like xargs or a simple shell while read line loop would be better and easier. Commented Dec 18, 2013 at 2:58
  • 1
    On the flip side: Why do you think wc is the right tool for this job? It seems like awk builtin variables and functions would be better and easier? Commented Dec 18, 2013 at 13:51

5 Answers 5

75

you are close. you have to concatenate the command line with awk variables:

awk '{system("wc "$1)}' myfile
Sign up to request clarification or add additional context in comments.

6 Comments

thanks, that works! but one more question? Can we assign the output to a new variable?
That's the wrong syntax for this job, it's a wrong application for system(), the print does not do what you think it will do, and no you can't assign the output of a system() call to an awk variable, what you posted in your comment assigns the return code from system() to a variable. Time for some coffee @Kent!
@EdMorton yes! system returns the status code.....I was confused by vimscripts' system()...:( . cmd|getline var should read the output...... this answer/comment is not correct for the variable part.
It's also not correct for the system() part - it should be system("wc \"" $1 "\"").
How to deal with passing 2 parameters?
|
52

You cannot grab the output of an awk system() call, you can only get the exit status. Use the getline/pipe or getline/variable/pipe constructs

awk '{
    cmd = "your_command " $1
    while (cmd | getline line) {
        do_something_with(line) 
    }
    close(cmd)
}' file

4 Comments

+1 for the correct way to get the output of the shell command, but in general the syntax to create the variable is cmd = "your_command \"" $1 ""\" so the argument is quoted when cmd is executed and you need to test for the the result of getline being greater than zero or you'll get stuck in an infinite loop if it fails.
+1 . OP, if you need store the output in a var, accept this answer too. mine was not correct for var assignment.
@EdMorton I think you mean cmd = "your_command \"" $1 "\""
@jaygooby yes I do, well spotted.
2

FYI here's how to use awk to process files whose names are stored in a file (providing wc-like functionality in this example):

gawk '
NR==FNR { ARGV[ARGC++]=$0; next }
{ nW+=NF; nC+=(length($0) + 1) }
ENDFILE { print FILENAME, FNR, nW, nC; nW=nC=0 }
' file

The above uses GNU awk for ENDFILE. With other awks just store the values in an array and print in a loop in the END section.

Comments

1

I would suggest another solution:

awk '{print $1}' myfile | xargs wc

the difference is that it executes wc once with multiple arguments. It often works (for example, with kill command)

Comments

0

Or use the pipe | as in bash then retrive the output in a variable with awk's getline, like this

 zcat /var/log/fail2ban.log* | gawk  '/.*Ban.*/  {print $7};' | sort | uniq -c | sort | gawk '{ "geoiplookup " $2 "| cut -f2 -d: " | getline geoip; print $2 "\t\t" $1 " " geoip}'

That line will print all the banned IPs from your server along with their origin (country) using the geoip-bin package.

The last part of that one-liner is the one that affects us :

gawk '{ "geoiplookup " $2 "| cut -f2 -d: " | getline geoip; print $2 "\t\t" $1 " " geoip}'

It simply says : run the command "geoiplookup 182.193.192.4 | -f2 -d:" ($2 gets substituted as you may guess) and put the result of that command in geoip (the | getline geoip bit). Next, print something something and anything inside the geoip variable.

The complete example and the results can be found here, an article I wrote.

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.