Extract specific set lines matching a rule

Question

I have a large file with the logs as shown below. There are about 30000 instance of such events logged. I need to extract those lines beginning with RINGING and CLOSE (included) and which does NOT contain 30 30.

The requirement is :

Of the two instances seen below, I need only Instance 2 retained. Instance 1 needs to be deleted entirely (they form the large chunk of the file)

Instance 1:

313782 Aug 19 18:37:04.925: <DATA>  RINGING|254|01136097645|5950|$hostIp|$size  |$data
313783 Aug 19 18:37:05.262: <DATA>  TRAINING|254|01136097645|5950|$hostIp|$size  |$data
313784 Aug 19 18:37:09.028: <DATA>  OUT  |254|01136097645|5950|$hostIp|2 bytes  |30 93
313785 Aug 19 18:37:09.705: <DATA>  IN   |254|01136097645|5950|$hostIp|4 bytes  |30 73 F9 F8
313786 Aug 19 18:37:18.532: <DATA>  IN   |254|01136097645|5950|$hostIp|336 bytes  |30 10 60 00 06 00 00 6F 12 00                                                                                 ...
313787 Aug 19 18:37:19.485: <DATA>  OUT  |254|01136097645|5950|$hostIp|133 bytes  |30 30 60 00 00 00 06 6F 12 10                                                                                 ...
313788 Aug 19 18:37:20.898: <DATA>  TRAINING|254|01136097645|5950|$hostIp|$size  |$data
313789 Aug 19 18:37:22.006: <DATA>  CLOSE|254|01136097645|5950|$hostIp|$size  |$data

Instance 2:

(Line with 30 30 is not present)

313782 Aug 19 18:37:04.925: <DATA>  RINGING|254|01136097645|5950|$hostIp|$size  |$data
313783 Aug 19 18:37:05.262: <DATA>  TRAINING|254|01136097645|5950|$hostIp|$size  |$data
313784 Aug 19 18:37:09.028: <DATA>  OUT  |254|01136097645|5950|$hostIp|2 bytes  |30 93
313785 Aug 19 18:37:09.705: <DATA>  IN   |254|01136097645|5950|$hostIp|4 bytes  |30 73 F9 F8
313786 Aug 19 18:37:18.532: <DATA>  IN   |254|01136097645|5950|$hostIp|336 bytes  |30 10 60 00 06 00 00 6F 12 00                                                                                                                                                         ...
313788 Aug 19 18:37:20.898: <DATA>  TRAINING|254|01136097645|5950|$hostIp|$size  |$data
313789 Aug 19 18:37:22.006: <DATA>  CLOSE|254|01136097645|5950|$hostIp|$size  |$data

Answer 1

Assuming that your logfile is called logfile, here is an awk solution with the sample output:

$ awk '/RINGING/,/CLOSE/ {if (/30 30/){f=1}; a=a"\n"$0} f==0 && /CLOSE/ {print a} /CLOSE/{a="";f=0}' logfile    

313782 Aug 19 18:37:04.925: <DATA>  RINGING|254|01136097645|5950|$hostIp|$size  |$data
313783 Aug 19 18:37:05.262: <DATA>  TRAINING|254|01136097645|5950|$hostIp|$size  |$data
313784 Aug 19 18:37:09.028: <DATA>  OUT  |254|01136097645|5950|$hostIp|2 bytes  |30 93
313785 Aug 19 18:37:09.705: <DATA>  IN   |254|01136097645|5950|$hostIp|4 bytes  |30 73 F9 F8
313786 Aug 19 18:37:18.532: <DATA>  IN   |254|01136097645|5950|$hostIp|336 bytes  |30 10 60 00 06 00 00 6F 12 00
313788 Aug 19 18:37:20.898: <DATA>  TRAINING|254|01136097645|5950|$hostIp|$size  |$data
313789 Aug 19 18:37:22.006: <DATA>  CLOSE|254|01136097645|5950|$hostIp|$size  |$data

Explanation

Taking each awk command in turn:

• /RINGING/,/CLOSE/ {if (/30 30/){f=1}; a=a"\n"$0}

  The expression /RINGING/,/CLOSE/ is a range: it specifies that this command only applies to groups of lines. A group starts when a line is encountered that includes the text RINGING. The group ends when a line including the text CLOSE is encountered. For any line in such a group, the commands in braces are executed. The first of these sets the flag f to one if the line contains 30 30. The second command appends the current line to the variable a.

• f==0 && /CLOSE/ {print a}

  The commands in braces here are preceded by two conditions and'd together. The first condition specifies that the flag f is zero (meaning that 30 30 was not found in this group) and the second specifies that this line contain the text CLOSE. If both those conditions are met, then the group of lines, stored in the a variable, are printed.

• /CLOSE/{a="";f=0}

  Lastly, on any line containing the text CLOSE, the variable a is reset to the empty string and the flag f is set to zero. When this is done, the code is prepared to start on the next group of lines, should there be one.

Answer 2

It may be possible using a perl compatible regex (PCRE) for example

pcregrep -M '^.*?RINGING(?(?!30 30)(?s).)+?CLOSE.*?$' file

or

grep -zPo '^.*?RINGING(?(?!30 30)(?s).)+?CLOSE.*?$' file

Alternatively, using GNU awk's more expressive record separator

gawk -vRS="CLOSE[^\n]*\n" -vORS= '!/30 30/ {print; print RT}' file