Filtering list of numbers containing sequential digits

Question

I would like to filter many text files. Each file contain a very long list of numbers. Filtering the files must occur by the number of the consecutive digits within each number.

Example list form one of the files:

Input file data.log:

12365
91738
349874
128152639
1234
7654
08767
1234567

I would like to:

1- filter this list by numbers containing two consecutive digits, the expected output should be:

12365
349874
128152639
1234
7654
08767
1234567

2- filter this list by numbers containing three consecutive digits, the expected output should be:

12365
349874
1234
7654
08767
1234567

3- filter this list by numbers containing four consecutive digits, the expected output should be:

1234
7654
1234567

4- filter this list by numbers containing five consecutive digits, the expected output should be:

1234567

If the order of the consecutive digits in the numbers is small to large (e.g. 1234 ...etc) or large to small (e.g. 54321) should be included in the output.

Answer 1

Using grep, tee, and rev, make a tricky little function full of bash-isms:

dqs() { a=${2:-123456789} ; [ "$1" -ge 2 ] &&  
        grep -iF "$(eval eval printf '%s\\\\n' \\$\\{a:\{0..$((${#a}-$1))\}:$1\\} |
                    tee >(rev) )"
       }

Test it:

dqs 5 < data.log 
1234567
dqs 4 < data.log 
1234
7654
1234567
dqs 3 < data.log 
12365
349874
1234
7654
08767
1234567

---

How it works:

printf prints a list of sequences of the desired length, (like 123, 234, etc.), tee appends a mirror-image (i.e. right-to-left, or backwards) copy using rev, then grep -f <(...) searches standard input for anything in that list.

To make that list of sequences usually would require a loop, or seq, or even both, but here we cheat by using a bash sequence expression, combined with a substring expansion, and some arithmetic. But it's impossible, because the bash interpreter cannot execute these in the desired order. Therefore eval eval and several strategic \\\s are used to force bash to do things in the right order.

The [ "$@" -gt 0 ] && is not functionally necessary here, but it's safer to have it. It makes sure that dqs has one and only one numeric parameter, or grep won't run. This prevents eval eval from doing anything evil.

Bonus: Adding a 2nd argument can change the 123456789 to any other sequence and the code should still work. For example dqs 4 123456789ABCDEF would search for four digit hexadecimal sequences, (and reverse sequences), and dqs 3 $(printf %s {a..z}) would search for three letter alphabetic sequences.

# search `man bash` for the three most popular words 
# that have 3 three char alphabetic runs
man bash | tr ' ' '\n' | sort | uniq -c | sort -gr  | 
dqs 3 $(printf '%s' {a..z}) | head -3

Output:

     92 first
     76 default
     38 environment

Answer 2

If you have a lot of very large files, regex matching in awk will be slow. One approach is to leverage grep to do the hard lifting, and awk to build the list of strings to search for (since you don't want to hard code those). I.e.

$grep -E '12|98|23|87|34|76|45|65|56|54|67|43|78|32|89|21' data.log

Does the trick for two characters, but we want to be able to do this for up to 9 characters. You need -E for extended grep to support searching multiple patterns (12|98 are two patterns) - plain ol' grep doesn't let you do that.

Awk can loop over the string 123456789 pulling out successive pieces, but we want to go forward and backward, so:

$awk 'BEGIN {f=123456789 ; b=987654321 ; for(i=1;i<9;i++) print substr(f,i,2),substr(b,i,2)}'
12 98
23 87
34 76
45 65
56 54
67 43
78 32
89 21

Let's add something so the length isn't hard coded to two (-vn=3 sets the variable n=3 inside the awk script):

$awk -vn=3 'BEGIN {f=123456789 ; b=987654321 ; for(i=1;i<11-n;i++) print substr(f,i,n),substr(b,i,n)}'
123 987
234 876
345 765
456 654
567 543
678 432
789 321

And (almost there!) get the pipe symbol grep -E wants by changing output record separator (ORS) and output field separator (OFS) to |

$awk -vn=3 'BEGIN {ORS="|" ; OFS="|" ; f=123456789 ; b=987654321 ; for(i=1;i<11-n;i++) print substr(f,i,n),substr(b,i,n)}'
123|987|234|876|345|765|456|654|567|543|678|432|789|321|

We have to get rid of the final pipe after 321, or grep will match everything, so add sed '.$//' to replace the last character before the end of the string ($) with nothing:

$awk -vn=3 'BEGIN {ORS="|" ; OFS="|" ; f=123456789 ; b=987654321 ; for(i=1;i<11-n;i++) print substr(f,i,n),substr(b,i,n)}' | sed 's/.$//'

And now we can put it together in a shell script that lets us do the search in general:

$cat t.sh
#!/bin/bash
grep -E `awk --assign n=$1 'BEGIN {OFS="|" ; ORS="|" ; f=123456789 ; b=987654321 ; for(i=1;i<11-n;i++) print substr(f,i,n),substr(b,i,n)}' | sed 's/.$//'` $2

$chmod 775 t.sh
$./t.sh 4 data.log
1234
7654
1234567

Answer 3

Many big files indicate that this needs to be done quickly. That means a while read loop is out of the question. One thing to realize here is that each of these exercises can be simplified into matching (at least) one of a small set of patterns, and this can be done really fast using grep or similar tools like rg or ack. For example, for five digit sequences:

grep -e 12345 -e 23456 […] -e 65432 -e 54321

See man grep for more information, and use Greg's Wiki to learn Bash quickly.