Script that scans through logs, pulls specific data, and creates an output directory

Question

I had asked a question a while ago about how to pull specific info from a log file, and got some useful answers, but I don't quite have what I need yet. What I have so far:

#!/bin/bash


mkdir result

cd $1
ls | while read -r file;
do
egrep "ERROR|FAIL|WARN" > ../errors-$file
done
cd ..

mv errors-* result

Here's what I want in total: the script scans through a directory of logs, grabs each line of text (some are multi-line) that includes ERROR|FAIL|WARN, and outputs them in a directory called "result" with each individual file being named based on their source file.

It works when my target log directory only has one text file in it, but it doesn't work when my target directory has directories and log files at different levels within it. I know it's because the "../errors-$file" bit in the script, but I'm not sure how to change it.

Any help would be great, and I apologize in advance if it's poorly phrased.

Answer 1

I'll poke at your loop itself, and try to walk through what it's actually doing, and then look at how you might achieve what you want.

How you initiate your loop is overly complex: ls | while read -r file;. You're piping the results of ls to your while read, not a big deal, still works, though I would recommend a for loop more like:

for file in ./* ;

This will loop over the same files, but is a little more straight forward. Also the ls command can return output that is not just the filenames (ever notice that ls has different font colors for directories/executables?), and this can cause other commands grief.

Your command: egrep "ERROR|FAIL|WARN" > ../errors-$file is trying to egrep for ERROR|FAIL|WARN in nothing, and redirecting output to ../errors-$file.

The syntax for egrep is:

egrep [search pattern] [location]

So you're missing the location for egrep to look in. You can fix this by adding $file after your pattern, so it becomes:

#!/bin/bash


mkdir result

cd $1
for file in $(find ./ -type f)
do
egrep "ERROR|FAIL|WARN" $file > ../result/errors-$(basename $file)
echo "-------------------------------" >> ../result/errors-$(basename $file)
egrep "denied" -B 1 $file >> ../result/errors-$(basename $file)
done