Find and Replace Everything Before a String of Text

Question

I have a large list of IP addresses (most are IPv4, but a few are IPv6), followed by a space and then a domain name, followed by another space and the same domain name with "www." in front of it. Each instance is on it's own line. The list looks like this (but is much larger):

23.212.109.137 at.ask.com www.at.ask.com
216.58.206.74 maps.googleapis.com www.maps.googleapis.com
2400:cb00:2048:1::6812:32a5 litscape.com www.litscape.com
104.16.244.35 loc.gov www.loc.gov
216.70.104.235 mbu.edu www.mbu.edu

I would like to know two find and replace commands; each to generate another text file after the last.

1) The first command should find and replace everything before the "www." with "http://" so that the lines of the second text file will look like this:

http://www.at.ask.com
http://www.maps.googleapis.com
http://www.litscape.com
http://www.loc.gov
http://www.mbu.edu

2) The second command should find and replace all instances of "http://www." in the second text file so that the lines of the third text file will look like this:

at.ask.com
maps.googleapis.com
litscape.com
loc.gov
mbu.edu

Thank you.

Answer 1

With single awk command:

awk '{ print $2 > "domains.txt"; print "http://"$3 > "domains_http.txt" }' file

---

Results:

> cat domains_http.txt 
http://www.at.ask.com
http://www.maps.googleapis.com
http://www.litscape.com
http://www.loc.gov
http://www.mbu.edu

---

> cat domains.txt 
at.ask.com
maps.googleapis.com
litscape.com
loc.gov
mbu.edu

Answer 2

The second thing is easiest:

$ cut -d ' ' -f 2 file.in

This uses cut with a space as delimiter and cuts out the second column.

With awk it would be easy to do both tasks:

$ awk '{ print $2 }' file.in

That does essentially the same as the cut command.

$ awk '{ print "http://" $3 }' file.in

That takes the third column and prepends http:// to it before outputting it.

Answer 3

Using sed:

sed 's#.*\(www.*\)#http://\1#' source_file > dest_file1
sed 's#.*www\.\(.*\)#\1#' dest_file1 > dest_file2

Explanation:

For the first requirement, start grouping everything from www, and replace everything that doesn't fall into this group with http://. Similar for the second file as well, but now start grouping from after www.

Note that if you want commands to do these jobs, you can always alias these into custom commands.