Using awk to write only duplicates

Question

Deleting duplicates with awk is pretty common and simple. But I need to print only those lines that are duplicated when we compare only one column. I tried this command :

awk 'seen[$2]++'

but as you can see it has flaws. It prints duplicates, but only since their 2nd appearance. I've only started getting used to unix and bash, so it would be great if you could explain the solution to me.

Answer 1

I can see 2 ways to do this:

1. iterate over the file twice:

   In the first iteration, count the number of times each $2 appears.
   In the second iteration, print only lines where the count is more than 1

   awk 'NR == FNR {count[$2]++; next} count[$2] > 1' file file
   

2. with a single iteration of the data:

   You need to count the number of times each $2 appears, and remember which lines have occurred for each $2.

   This answer uses GNU awk for array-of-arrays. The order of the output is not likely to be the same as the input data. It also has to store the whole file in memory.

   gawk '
       { lines[$2][++count[$2]] = $0 }
       END {
           for (x in lines)
               if (count[x] > 1)
                   for (i=1; i<=count[x]; i++)
                       print lines[x][i]
       }
   ' file
   

Tested with input file:

$ cat file
a b
b b
c b
a c
a d
b d
a e

and expected output

a b
b b
c b
a d
b d

Answer 2

Using same sample input as glenn jackman's answer

$ awk '$2 in seen{if(c[$2]--){print fl[$2]} print} !seen[$2]++{fl[$2]=$0; c[$2]=1}' file
a b
b b
c b
a d
b d

• !seen[$2]++ if $2 is not encountered before:
  • fl[$2]=$0 save this first line, I've assumed input is not sorted and duplicates can occur anywhere in file, hence saving it based on $2 instead of just temporary variable
  • c[$2]=1 similarly, initialize count variable with 1
• $2 in seen if $2 has occurred before:
  • if(c[$2]--){print fl[$2]} first print the previous line, counter is decremented so that condition will fail for subsequent matches
  • print then print the current line

With some other input

$ cat ip.txt 
6.2  : 897 : bar
3.1  : 32  : foo
1.2  : 123 : xyz
2.3  : 32  : baz
7.5  : 897 : boo

$ awk -F: '$2 in seen{if(c[$2]--){print fl[$2]} print} !seen[$2]++{fl[$2]=$0; c[$2]=1}' ip.txt 
3.1  : 32  : foo
2.3  : 32  : baz
6.2  : 897 : bar
7.5  : 897 : boo

Note that the order depends on how duplicates occur

Answer 3

When you iterate over the same file twice you can use line numbers as convenient indices; it can make for cleaner logic.

awk 'NR == FNR {if ($2 in z) { y[z[$2]]; y[FNR] } z[$2]=FNR; next} (FNR in y)' file file

I used a similar trick in my answer to this question:

• Print only unique lines that appear last in logfile based on date/time

---

The basis of this trick is that Awk will create a variable simply by referencing it, and the index in arrayname construct returns true or false depending on whether an array element has been created with the specified index.