Using xargs with input from a file

Question

Say I have a file with the following

bob
john
sue

Now these directly corrospond to (in this case) URL pattern such as http://example.com/persons/bob.tar, john.tar, sue.tar.

I would like to take these lines and run them through xargs. I don't know what is passed to the command being executed though. How do I access the parameter either from the prompt (say I want to simply echo each line like cat file | xargs echo $PARAM) or from a bash script.

Answer 1

Michael's answer is right, and should sort out your problem. Running

cat file | xargs -I % curl http://example.com/persons/%.tar

will download files bob.tar john.tar. sue.tar as expected.

BUT: cat here is useless

rather use:

<file xargs -I % curl http://example.com/persons/%.tar

Answer 2

I think you're asking how to insert the individual lines pulled from xargs' stdin in the middle of a command, instead of just pasting it on the end always. If so, the -I flag takes a replacement-string argument; xargs will then replace replacement-string in the command with the line read from stdin:

$ cat file | xargs -I foobar curl http://example.com/foobar.tar

Answer 3

$ man xargs
...
       --arg-file=file
       -a file
              Read items from file instead of standard input.  If you use this
              option,  stdin  remains unchanged when commands are run.  Other-
              wise, stdin is redirected from /dev/null.
...

You may want to set --delimiter=/-d to '\n' as well.

---

On the other hand, if you are just trying to turn each line in the file into a URL,

$ sed -e 's#.*#http://example.com/persons/&.tar#' file

will do, and if you want to fetch all of them, just pipe that into | wget -i.

Answer 4

another way with shell looping:

for i in `cat file`; do curl -I http://foo.com/$i; done

you can also run each iteration in the background by appending & prior to the last semicolon - for very large downloads this might be handy

Answer 5

With GNU Parallel you can do:

cat urls | parallel curl {} ">" {/}

Or:

cat persons | parallel curl http://example.com/persons/{}.tar ">" {}.tar

Watch the intro video for GNU Parallel to learn more: http://www.youtube.com/watch?v=OpaiGYxkSuQ

Answer 6

while read VAR; do ... done loop is simple yet very versatile:

while read word; do wget http://example.com/persons/$word; done < file

Answer 7

Eventually, you can use as well:

xargs -n1 -a file -I % curl http://example.com/persons/%.tar

Answer 8

This is a more general version of Stefan:s answer but I'm using awk in the middle to prepare the exact "string" that I would like xargs to execute. And then xargs is using bash to do the actual "work".

It is a little bit overkill for this example, but it is a general solution that with some modifications can solve many problems...

cat file | awk '{print "curl http://example.com/persons/"$1".tar"}' | xargs -0 bash -c