Return to Answer

This general trick works even if your curl config file contains miscellaneous options such as user-agent, referer, etc.

First step, assume your config file named curl_config, then use awk '/^[Uu][Rr][Ll]/{print;print "output = dummy/"++k;next}1' curl_config > curl_config2 to create a new curl config file which incremently append different output file names under each url/URL:

Example:

[xiaobai@xiaobai curl]$ cat curl_config
URL = "www.google.com"
user-agent = "holeagent/5.0"

url = "m12345.google.com"
user-agent = "holeagent/5.0"

URL = "googlevideo.com"
user-agent = "holeagent/5.0"
[xiaobai@xiaobai curl]$ awk '/^[Uu][Rr][Ll]/{print;print "output = dummy/"++k;next}1' curl_config  > curl_config2 
[xiaobai@xiaobai curl]$ cat curl_config2
URL = "www.google.com"
output = dummy/1
user-agent = "holeagent/5.0"

url = "m12345.google.com"
output = dummy/2
user-agent = "holeagent/5.0"

URL = "googlevideo.com"
output = dummy/3
user-agent = "holeagent/5.0"
[xiaobai@xiaobai curl]$ 

Then mkdir dummy to create a directory to hold this temporary files. Create inotifywait session(Replace the sed '/google/q' with your sed '/mortgage/q'):

[xiaobai@xiaobai curl]$ rm -r dummy && mkdir dummy;
[xiaobai@xiaobai curl]$ rm final 
[xiaobai@xiaobai curl]$ inotifywait -m dummy -e close_write | while read path action file; do echo "[$file]">> final ; sed '/google/q' "$path$file" >> final; echo "$path$file"; rm "$path""$file"; done;
Setting up watches.
Watches established.

Open another bash/terminal session, rm final file if exist, then run your curl with your curl_config2 file created in first step above:

[xiaobai@xiaobai curl]$ curl -vLK curl_config2
...processing

Now take a look to inotifywait session, it will print the latest close written file, sed it and remove it immediately once done:

[xiaobai@xiaobai curl]$ inotifywait -m dummy -e close_write | while read path action file; do echo "[$file]">> final ; sed '/google/q' "$path$file" >> final; echo "$path$file"; rm "$path""$file"; done;
Setting up watches.
Watches established.
dummy/1
dummy/3

Finally you can observe your output named final, The [1 and 3] separator is generated from echo "[$file]">> final above:

The reason of remove file immediately is because i assume your output file is large plus many url have to proceed, so it could save disk space to remove it immediately.

This general trick works even if your curl config file contains miscellaneous options such as user-agent, referer, etc.

First step, assume your config file named curl_config, then use awk '/^[Uu][Rr][Ll]/{print;print "output = dummy/"++k;next}1' curl_config > curl_config2 to create a new curl config file which incremently append different output file names under each url/URL:

Example:

[xiaobai@xiaobai curl]$ cat curl_config
URL = "www.google.com"
user-agent = "holeagent/5.0"

url = "m12345.google.com"
user-agent = "holeagent/5.0"

URL = "googlevideo.com"
user-agent = "holeagent/5.0"
[xiaobai@xiaobai curl]$ awk '/^[Uu][Rr][Ll]/{print;print "output = dummy/"++k;next}1' curl_config  > curl_config2 
[xiaobai@xiaobai curl]$ cat curl_config2
URL = "www.google.com"
output = dummy/1
user-agent = "holeagent/5.0"

url = "m12345.google.com"
output = dummy/2
user-agent = "holeagent/5.0"

URL = "googlevideo.com"
output = dummy/3
user-agent = "holeagent/5.0"
[xiaobai@xiaobai curl]$ 

Then mkdir dummy to create a directory to hold this temporary files. Create inotifywait session(Replace the sed '/google/q' with your sed '/mortgage/q'):

[xiaobai@xiaobai curl]$ rm -r dummy; mkdir dummy;
[xiaobai@xiaobai curl]$ rm final 
[xiaobai@xiaobai curl]$ inotifywait -m dummy -e close_write | while read path action file; do echo "[$file]">> final ; sed '/google/q' "$path$file" >> final; echo "$path$file"; rm "$path$file"; done;
Setting up watches.
Watches established.

Open another bash/terminal session, rm final file if exist, then run your curl with your curl_config2 file created in first step above:

[xiaobai@xiaobai curl]$ curl -vLK curl_config2
...processing

Now take a look to inotifywait session, it will print the latest close written file, sed it and remove it immediately once done:

[xiaobai@xiaobai curl]$ inotifywait -m dummy -e close_write | while read path action file; do echo "[$file]">> final ; sed '/google/q' "$path$file" >> final; echo "$path$file"; rm "$path$file"; done;
Setting up watches.
Watches established.
dummy/1
dummy/3

Finally you can observe your output named final, The [1 and 3] separator is generated from echo "[$file]">> final above:

The reason of remove file immediately is because i assume your output file is large plus many url have to proceed, so it could save disk space to remove it immediately.

This general trick works even if your curl config file contains miscellaneous options such as user-agent, referer, etc.

First step, assume your config file named curl_config, then use awk '/^[Uu][Rr][Ll]/{print;print "output = dummy/"++k;next}1' curl_config > curl_config2 to create a new curl config file which incremently append different output file names under each url/URL:

Example:

[xiaobai@xiaobai curl]$ cat curl_config
URL = "www.google.com"
user-agent = "holeagent/5.0"

url = "m12345.google.com"
user-agent = "holeagent/5.0"

URL = "googlevideo.com"
user-agent = "holeagent/5.0"
[xiaobai@xiaobai curl]$ awk '/^[Uu][Rr][Ll]/{print;print "output = dummy/"++k;next}1' curl_config  > curl_config2 
[xiaobai@xiaobai curl]$ cat curl_config2
URL = "www.google.com"
output = dummy/1
user-agent = "holeagent/5.0"

url = "m12345.google.com"
output = dummy/2
user-agent = "holeagent/5.0"

URL = "googlevideo.com"
output = dummy/3
user-agent = "holeagent/5.0"
[xiaobai@xiaobai curl]$ 

Then mkdir dummy to create a directory to hold this temporary files. (Replace the sed '/google/q' with your sed '/mortgage/q'):

[xiaobai@xiaobai curl]$ rm -r dummy && mkdir dummy;
[xiaobai@xiaobai curl]$ rm final 
[xiaobai@xiaobai curl]$ inotifywait -m dummy -e close_write | while read path action file; do echo "[$file]">> final ; sed '/google/q' "$path$file" >> final; echo "$path$file"; rm "$path""$file"; done;
Setting up watches.
Watches established.

Open another bash/terminal session, rm final file if exist, then run your curl with your curl_config2 file created in first step above:

[xiaobai@xiaobai curl]$ curl -vLK curl_config2
...processing

Now take a look to inotifywait session, it will print the latest close written file, sed it and remove it immediately once done (Replace the sed '/google/q' with your sed '/mortgage/q'):

[xiaobai@xiaobai curl]$ inotifywait -m dummy -e close_write | while read path action file; do echo "[$file]">> final ; sed '/google/q' "$path$file" >> final; echo "$path$file"; rm "$path""$file"; done;
Setting up watches.
Watches established.
dummy/1
dummy/3

Finally you can observe your output named final, The [1 and 3] separator is generated from echo "[$file]">> final above:

The reason of remove file immediately is because i assume your output file is large plus many url have to proceed, so it could save disk space to remove it immediately.

This general trick works even if your curl config file contains miscellaneous options such as user-agent, referer, etc.

First step, assume your config file named curl_config, then use awk '/^[Uu][Rr][Ll]/{print;print "output = dummy/"++k;next}1' curl_config > curl_config2 to create a new curl config file which incremently append different output file names under each url/URL:

Example:

[xiaobai@xiaobai curl]$ cat curl_config
URL = "www.google.com"
user-agent = "holeagent/5.0"

url = "m12345.google.com"
user-agent = "holeagent/5.0"

URL = "googlevideo.com"
user-agent = "holeagent/5.0"
[xiaobai@xiaobai curl]$ awk '/^[Uu][Rr][Ll]/{print;print "output = dummy/"++k;next}1' curl_config  > curl_config2 
[xiaobai@xiaobai curl]$ cat curl_config2
URL = "www.google.com"
output = dummy/1
user-agent = "holeagent/5.0"

url = "m12345.google.com"
output = dummy/2
user-agent = "holeagent/5.0"

URL = "googlevideo.com"
output = dummy/3
user-agent = "holeagent/5.0"
[xiaobai@xiaobai curl]$ 

Then mkdir dummy to create a directory to hold this temporary files. Create inotifywait session(Replace the sed '/google/q' with your sed '/mortgage/q'):

[xiaobai@xiaobai curl]$ rm -r dummy && mkdir dummy;
[xiaobai@xiaobai curl]$ rm final 
[xiaobai@xiaobai curl]$ inotifywait -m dummy -e close_write | while read path action file; do echo "[$file]">> final ; sed '/google/q' "$path$file" >> final; echo "$path$file"; rm "$path""$file"; done;
Setting up watches.
Watches established.

Open another bash/terminal session, rm final file if exist, then run your curl with your curl_config2 file created in first step above:

[xiaobai@xiaobai curl]$ curl -vLK curl_config2
...processing

Now take a look to inotifywait session, it will print the latest close written file, sed it and remove it immediately once done:

[xiaobai@xiaobai curl]$ inotifywait -m dummy -e close_write | while read path action file; do echo "[$file]">> final ; sed '/google/q' "$path$file" >> final; echo "$path$file"; rm "$path""$file"; done;
Setting up watches.
Watches established.
dummy/1
dummy/3

Finally you can observe your output named final, The [1 and 3] separator is generated from echo "[$file]">> final above:

The reason of remove file immediately is because i assume your output file is large plus many url have to proceed, so it could save disk space to remove it immediately.

This general trick works even if your curl config file contains miscellaneous options such as user-agent, referer, etc.

First step, Use awk '/^[Uu][Rr][Ll]/{print;print "output = dummy/"++k;next}1' curl_config > curl_config2 to create a new curl config file which incremently append different output file names under each url/URL:

Example:

[xiaobai@xiaobai curl]$ cat curl_config
URL = "www.google.com"
user-agent = "holeagent/5.0"

url = "m12345.google.com"
user-agent = "holeagent/5.0"

URL = "googlevideo.com"
user-agent = "holeagent/5.0"
[xiaobai@xiaobai curl]$ awk '/^[Uu][Rr][Ll]/{print;print "output = dummy/"++k;next}1' curl_config  > curl_config2 
[xiaobai@xiaobai curl]$ cat curl_config2
URL = "www.google.com"
output = dummy/1
user-agent = "holeagent/5.0"

url = "m12345.google.com"
output = dummy/2
user-agent = "holeagent/5.0"

URL = "googlevideo.com"
output = dummy/3
user-agent = "holeagent/5.0"
[xiaobai@xiaobai curl]$ 

Then mkdir dummy to create a directory to hold this temporary files. (Replace the sed '/google/q' with your sed '/mortgage/q'):

[xiaobai@xiaobai curl]$ rm -r dummy && mkdir dummy;
[xiaobai@xiaobai curl]$ rm final 
[xiaobai@xiaobai curl]$ inotifywait -m dummy -e close_write | while read path action file; do echo "[$file]">> final ; sed '/google/q' "$path$file" >> final; echo "$path$file"; rm "$path""$file"; done;
Setting up watches.
Watches established.

Open another bash/terminal session, rm final file if exist, then run your curl with your curl_config2 file created in first step above:

[xiaobai@xiaobai curl]$ curl -vLK curl_config2
...processing

Now take a look to inotifywait session, it will print the latest close written file, sed it and remove it immediately once done (Replace the sed '/google/q' with your sed '/mortgage/q'):

[xiaobai@xiaobai curl]$ inotifywait -m dummy -e close_write | while read path action file; do echo "[$file]">> final ; sed '/google/q' "$path$file" >> final; echo "$path$file"; rm "$path""$file"; done;
Setting up watches.
Watches established.
dummy/1
dummy/3

Finally you can observe your output named final, The [1 and 3] separator is generated from echo "[$file]">> final above:

The reason of remove file immediately is because i assume your output file is large plus many url have to proceed, so it could save disk space to remove it immediately.

This general trick works even if your curl config file contains miscellaneous options such as user-agent, referer, etc.

First step, assume your config file named curl_config, then use awk '/^[Uu][Rr][Ll]/{print;print "output = dummy/"++k;next}1' curl_config > curl_config2 to create a new curl config file which incremently append different output file names under each url/URL:

Example:

[xiaobai@xiaobai curl]$ cat curl_config
URL = "www.google.com"
user-agent = "holeagent/5.0"

url = "m12345.google.com"
user-agent = "holeagent/5.0"

URL = "googlevideo.com"
user-agent = "holeagent/5.0"
[xiaobai@xiaobai curl]$ awk '/^[Uu][Rr][Ll]/{print;print "output = dummy/"++k;next}1' curl_config  > curl_config2 
[xiaobai@xiaobai curl]$ cat curl_config2
URL = "www.google.com"
output = dummy/1
user-agent = "holeagent/5.0"

url = "m12345.google.com"
output = dummy/2
user-agent = "holeagent/5.0"

URL = "googlevideo.com"
output = dummy/3
user-agent = "holeagent/5.0"
[xiaobai@xiaobai curl]$ 

Then mkdir dummy to create a directory to hold this temporary files. (Replace the sed '/google/q' with your sed '/mortgage/q'):

[xiaobai@xiaobai curl]$ rm -r dummy && mkdir dummy;
[xiaobai@xiaobai curl]$ rm final 
[xiaobai@xiaobai curl]$ inotifywait -m dummy -e close_write | while read path action file; do echo "[$file]">> final ; sed '/google/q' "$path$file" >> final; echo "$path$file"; rm "$path""$file"; done;
Setting up watches.
Watches established.

Open another bash/terminal session, rm final file if exist, then run your curl with your curl_config2 file created in first step above:

[xiaobai@xiaobai curl]$ curl -vLK curl_config2
...processing

Now take a look to inotifywait session, it will print the latest close written file, sed it and remove it immediately once done (Replace the sed '/google/q' with your sed '/mortgage/q'):

[xiaobai@xiaobai curl]$ inotifywait -m dummy -e close_write | while read path action file; do echo "[$file]">> final ; sed '/google/q' "$path$file" >> final; echo "$path$file"; rm "$path""$file"; done;
Setting up watches.
Watches established.
dummy/1
dummy/3

Finally you can observe your output named final, The [1 and 3] separator is generated from echo "[$file]">> final above:

The reason of remove file immediately is because i assume your output file is large plus many url have to proceed, so it could save disk space to remove it immediately.