Counting a list of strings on every line of multiple files

Question

I have 250 strings and I need to count the number of times each one appears on every line of my 400 files (which are up to 20,000 lines). Example of strings:

journal
moon pig
owls

Example of one file:

This text has journal and moon pig
This text has owls and owls

Example output:

1   0
1   0
0   2

EDIT: where column one counts strings from the first line of the file, and column two represents the second line of the file.

I have working code but its obviously very slow. I'm sure awk could speed it up but I'm not good enough to write it.

for file in folder/*
do
    name=$(basename "$file" .txt)
    linenum=1
    while read line
    do
        while read searches
        do
            ###count every time string appears on line and save
            count=$(echo $line | grep -oi "$searches" | wc -l)
            echo $count >> out/${name}_${linenum}.txt
        done < strings.txt
        linenum=$((linenum+1))
    done < $file
done

EDIT: I do 400 pastes like this, where x is the number of lines in the original file.

paste out/file1_{1..x}.txt > out/file1_all.txt

Does anyone know how to speed this up?

Answer 1

If

$ cat strings
journal
moon pig
owls

and

$ cat file
I like to journal about owls and moon pigs.
owls are birds. moon pigs are not.
owls owls owls

then, you could use GNU awk like this

gawk '
    NR == FNR { string[++n] = $0; next}
    {
        for (i=1; i<=n; i++)
            # gsub() return the number of replacements.
            # it is a convenient way to count instances of fixed strings.
            count[i][FNR] = gsub(string[i], string[i])
        if (FNR > max)
            max = FNR
    }
    END {
        for (i=1; i<=n; i++) {
            for (j=1; j<=max; j++)
                printf "%s\t", 0 + count[i][j]
            print ""
        }
    }
' strings file

outputs

1   0   0
1   1   0
1   1   3

I haven't explained that awk program at all. See if you can figure it out, and ask any questions you have.

Answer 2

The core algorithm to get an array of counts per each line which processes each line immediately is:

gawk ' NR == FNR { string[++n] = $0; next}
       { for (i=1; i<=n; i++) 
             printf("%s\t", gsub(string[i],""))
         print ""
       }
     ' strings file

That is based on gsub to giving the count of replacements performed.

That will generate an output of:

1   1   0   
0   0   2

Which is simply the transposed matrix of what you asked for. It gets a bit complex in awk to transpose columns and rows. And also to process more than one file. We can concatenate (pipe) both scripts using an empty line as indicator of file change. Processing the same file twice:

awk '
    NR == FNR { string[++n] = $0; next}
    FNR==1 && p == 1 { print "" }
    { for (i=1; i<=n; i++) printf("%s\t", gsub(string[i],""))
      print ""
      p = 1
    }
    END    { print "" }
' strings.txt    infile.txt    infile.txt |
awk '!/^$/{ 
       for(i=1;i<=NF;i++) f[NR-r][i]=$i ;
       if (maxf<NF) maxf = NF ;
       if (maxr<(NR-r)) maxr = NR-r ;
     } 
     /^$/{
         for(      i=1 ; i<=maxf ; i++ )
         {
             for(  j=1 ; j<=maxr ; j++ )
                 printf("%s\t",f[j][i])
             print ( "loop", maxf, maxr, r )
         }
     r=NR
     print ( "" )
     maxf=0
     maxr=0
     delete f
     }'

Which gives the answer as asked:

1   0
1   0
0   2

1   0
1   0
0   2