Grep that works with overlapping patterns

You can count repeated pairs of letters such as your example aa in a data file big_file like this:

tr -cs a '\012' <big_file | awk '/aa/{n += length - 1}; END {print n+0}'

The line can be explained like this

• The tr changes any sequence of characters that is not a into a newline. This splits multiple occurrences of aa... onto separate lines
• The awk counts the length of non-zero lines. By using the property that a sequence of N characters contains N-1 overlapping pairs, it can sum up the number of overlapping pairs and produce a total at the end of the file

Actually, since the file is "very large" you will get a better response by including a grep in the pipeline like this:

tr -cs a '\012' <big_file | grep aa | awk '{n += length - 1}; END {print n+0}'

Using any awk:

$ cat file
x
a
aa
aaa
aaaa
aa aaa aaaa

$ awk '{printf "%s\t", $0; t=0; while(c=gsub(/aa/,"a")) t+=c; print t}' file
x       0
a       0
aa      1
aaa     2
aaaa    3
aa aaa aaaa     6

That replaces each aa on the line with a single a and counts how many such replacements occurred in this pass in the variable c. So on the first iteration of the while loop, input of aaa would become aa and c would be set to 1. The value of c is then added to the total t and then the loop iterates again. This time the aa produced in the first iteration becomes just a and c is 1 again and now t becomes 2. On the next iteration there are no aas left to replace so c is zero causing us to exit the loop and print t, i.e. 2.

The above does assume that none of the chars in your target string are regexp metachars and that the target string is just a chain of the same character repeated.

If you need a more general solution that'll match any string then, again using any awk, just find the first matching string, count it, then search for the next one starting searching the input from 1 char past where you found the previous one:

$ echo 'ababa' | awk -v tgt='aba' '{t=0; while( s=index($0,tgt) ) { t++; $0=substr($0,s+1) } print t}'
2

$ echo 'a.*a.*a' | awk -v tgt='a.*a' '{t=0; while( s=index($0,tgt) ) { t++; $0=substr($0,s+1) } print t}'
2

That last example was to show it'll work even if your target string contains regexp metachars because it's doing literal string matching, not regexp matching.

For some reason, chaos' answer outputs nothing on my system (Debian 11.5, GNU grep 3.6). Here's what works for me, just in case it might be useful to someone else:

pcregrep -o1 -o2 '(\w)(?=(\w))' file

This is similar in spirit to what has been proposed: match the first letter using capture group 1 if it is followed by another letter in capture group 2, only the latter is matched using a positive lookahead so we can still consume it afterwards. And then, display both groups using -o1 and -o2.

This generalises straightforwardly to subsequences of length n:

pcregrep -o1 -o2 '(\w)(?=(\w{n-1}))' file

One approach is to do the matching of the regexp inside a look ahead operator so the matching will be attempted at every byte position and can be used with arbitrary regexps:

$ echo aaaa | perl -ne '$n += () = /(?=aa)/g; END{printf "%d\n", $n}'
3

$ echo ababa | perl -ne '$n += () = /(?=aba)/g; END{printf "%d\n", $n}'
2

Using Raku (formerly known as Perl_6)

~$ raku -ne 'say "$_\t", ($/.elems given .match: :overlap, /aa/);'  file

#OR:

~$ raku -ne 'say "$_\t", .match(:overlap, /aa/).elems;'   file

OR:

~$ raku -ne 'say "$_\t", ($/.elems given m:overlap/aa/);'  file

#OR:

~$ raku -ne 'say "$_\t", m:overlap/aa/.elems;'  file 

Above is an answer written in Raku, a member of the Perl-family of programming languages. Among other things, Raku features high-level support for Unicode, including accurate Unicode character matching and/or counting. Above, all four code variants make use of Raku's :overlap regex modifier ("adverb" in Raku lingo, i.e. named argument).

Sample Input:

x

a
aa
aaa
aaaa
aa aaa aaaa

Sample Output:

x   0
    0
a   0
aa  1
aaa 2
aaaa    3
aa aaa aaaa 6

If you need a file sum (as opposed to a linewise sum), you can use BEGIN/END blocks like with awk:

~$ raku -ne 'BEGIN my $n = 0; $n += .match(:overlap, /aa/).elems; END say $n;'  file
12

I mentioned Unicode support so here's a quick example (and see first link below):

~$ printf '«AAÁÁÅÅÀÀÄÄBBßßœœþþ» CDE\n' | raku -ne 'say "Input string \"$_\"  contains " ~ .match(:overlap,/AA/).elems ~ " overlapping AA match(es).";'
Input string "«AAÁÁÅÅÀÀÄÄBBßßœœþþ» CDE"  contains 1 overlapping AA match(es).

https://unix.stackexchange.com/a/767221/227738
https://docs.raku.org/language/regexes#Overlap
https://raku.org