Count lines ending in "*"

Question

I have several files in a directory with this kind of content:

Wood *
Nails
Large Hammer *

Some names have a star after them, some don't. I have multiple files with such content. In each file a product may or may not have a single star next to it. I need to make a bash script to count the number of star occurrences for each individual product in all the files. For example, the output needs to be like this:

Wood 12
Yellow Lamps 6
Nails 4
...

Which means that in all the files it found 12x a star next to Wood, 6x a star next to the lamps, etc...

It's pretty easy to parse it in C, but I don't want a binary to run. I want a shell script, and I'm not as versatile with grep and awk, which I'm sure I need here.

I know how to count the stars per se, but I'm not sure how to track which star count belongs to which product.

Answer 1

Like this, with one awk:

awk '$NF=="*"{$NF=""; arr[$0]++}END{for (i in arr) print i arr[i]}' ./*

• $NF is the latest string separated by space(s) by default
• the main trick is to create an associative named array with the current words as key and incrementing as value
• at the END we iterate over the array to print each keys/values

With perl one-liner:

perl -anE '
    if ($F[-1] eq "*") {
        $k = join " ", @F[0..@F-2];
        $a->{$k}++
    }
    END{say "$_ $a->{$_}" for keys %$a}
' ./*

The -a is the split mode in @F default array

Answer 2

You could do:

sed -n 's/[[:blank:]]*\*$//p' ./* |
  LC_ALL=C sort |
  LC_ALL=C uniq -c |
  sort -rn

Which removes the <blanks>* at the end of the lines (and prints only the lines where there has been such a substitution) and use sort | uniq -c to count the unique lines (in the C locale for it to be a byte-to-byte comparison).

Answer 3

I'm not sure if this can affect the performance (if you have very larges files I would think this command should be slow):

grep -Fh '*' | tr -s ' ' | sort | uniq -c

More portable:

grep -Fh '*' * 2>/dev/null | tr -s ' ' | sort | uniq -c

And if you have sub-directories with more files you want to search inside:

grep -Fh '*' **/* 2>/dev/null | tr -s ' ' | sort | uniq -c | sed 's/.$//'

Or to avoid using 2>/dev/null:

find . -type f -exec grep -Fh '*' {} + | tr -s ' ' | sort | uniq -c | sed 's/.$//'

The section grep -Fh '*' means that will match any line which has a * at the end of this one. -h suppress printing the filenames whose matches the pattern and -F is for using literal strings (the '*' behaves as a string and not as pattern).
About tr -s ' ' I'm removing repeated spaces between every line, for example having this:

Need *
Word   buzz *
Need *
More   *
More *
Word   *
More   *
More *
Word   *
Word   *
Need *
More *

the tr command will parse it to:

Need *
Word buzz *
Need *
More *
More *
Word *
More *
More *
Word *
Word *
Need *
More *

The content above is piped to sort to have this output:

More *
More *
More *
More *
More *
Need *
Need *
Need *
Word *
Word *
Word *
Word buzz *

And finally with uniq -c I'm prefixing lines by the number of occurrences of every word which is what you want.

The sort command is important, if you do not use it, the expected result will be different

According to the output above, the final output (by using uniq -c) will be:

5 More *
3 Need *
3 Word *
1 Word buzz *

If you want to remove the * you can pipe to sed to remove the last character or *:

grep -Fh '*'  * | tr -s ' ' | sort | uniq -c | sed 's/.$//'
#or
grep -Fh '*' * | tr -s ' ' | sort | uniq -c | sed 's/\*//'

I think and hope there are better ways to achieve that, because here I'm using several commands to get the desired output. So as I said it may result in slow performance.

Answer 4

Using bash or just awk is recommended, but I liked the challenge of doing it in (GNU) sed.

s:  *: :g
/\*$/!s:$: :
G
s:([^\n]+) (\*?)(.*\n)\1 (\**)\n:\3\1 \4\2\n:
s:^\n::
h;$!d
s:\n$::
:u2d
    s:\*:<<123456789*01>:m
    s:(.)<.*\1(\**.).*>:\2:m
tu2d

I tested with the two input files below (vim display); first one from Edgar Magallon's answer:

Need *         |Need
Word   buzz *  |Word   buzz
Need *         |Need
More   *       |More *
More *         |More *
Word   *       |Word
More   *       |More *
More *         |More *
Word   *       |Word
Word   *       |Word
Need *         |Need
More *         |More *
~              |~
~              |~
input1          input2

Result:

~$ sed -rf script.sed input1 input2
Word 3
More 10
Word buzz 1
Need 3