uniq - how to ignore IP address?

Question

I have the following:

cat /var/log/example.log | cut -d ' ' -f 3,4,5,6 | sort -u | uniq

The result has several thousand lines but I would like to cut this down to show the actual activity and to do this, I would need to exclude/ignore the IP address

So the log has records such as:

2021-12-30 19:46:44 Invalid heartbeat from X.X.X.X
2021-12-30 19:46:44 X.X.X.X sent a malformed request
2021-12-30 19:46:44 Login from X.X.X.X failed due to bad username
2021-12-30 19:46:44 Failed login from this IP X.X.X.X

There are tens of thousands of records - how do I get the activity i.e.

Invalid heartbeat from 
sent a malformed request
Login from failed due to bad username
Failed login from this IP 

Thanks

Answer 1

You can use sed to remove the IP address before sorting:

cut -d ' ' -f 3- /var/log/example.log | sed -E 's/[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}//g' | sort -u

cut -f 3- - this will get you all fields from the 3rd until the end of the line.

Answer 2

If you want a awk solution, here it is

awk '
{
        for (i = 1; i <= NF - 2; i++) {
                $i = $(i + 2)
        }
        NF -= 2
        gsub(/[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/, "", $0)
        print
}
' /var/log/example.log | sort -u

Answer 3

Apparently this works better.

awk '{ gsub(/[0-9]{1,3}(\.[0-9]{1,3}){3}/, ""); $1 = $2 = "";
    gsub(/^ +| +$/, "") } !seen[$0]++' /var/log/example.log

No need to rely on \s and \s+ when we can just allow contiguous white spaces to be converted to a single space after $1 = $2 = "".

Thanks to Ed Morton.