1

I'm looking for something pretty similar to this.

The logs look like this:

[09:44:22] [main] ERROR [url/location] - A ONE LINE ERROR
[09:44:22] [main] ERROR [url/location] - Another ERROR 
[09:44:22] [main] SOMETHING DIFFERENT
[09:44:22] [main] SOMETHING DIFFERENT AGAIN
[09:44:22] [main] WARN [url/location] - ANOTHER ONE LINE WARN

Line after line with no empty lines between them, though occasionally there are indents when further info is available for a specific piece.

I want to be able to pull every line that includes ERROR (ideally as a script that can pull ERROR And/Or FAIL, WARN, etc.) and display them according to a parameter. It'll make sifting through logs for fails and whatnot much easier.

Improve this question
7
  • You'll need to define what a "chunk" is - beyond the fact that it isn't delimited by blank lines Commented Jul 14, 2016 at 15:39
  • I couldn't find a way to make it so that what's shown in the block above is line after line instead of line, space, line. Each message has its own line with a timestamp at the beginning, if it's possible to sort by that. Commented Jul 14, 2016 at 15:44
  • You should be able to preserve text formating by using code markdown (basically select it then press Ctrl-K) - see How do I format my posts using Markdown or HTML? in the Help Center Commented Jul 14, 2016 at 16:04
  • I've got something working with [awk -v FS='' '/ERROR/' file.txt] that seems to work, though only in my test file where I copied a few lines from the log into a new file. Is there a size limit where the command stops working? Minus the brackets, as I'm new to SE and don't know much of the forum syntax. Commented Jul 14, 2016 at 16:11
  • 1
    @Patremagne I don't see why the awk command you quoted wouldn't work if all you want is the matching lines. Setting FS is unnecessary, but shouldn't matter. Just printing lines containing a pattern is simple with any of the tools you mention, and perhaps the simplicity of it is what causes confusion. (And the word "chunk" and mentioning indentations). awk '/ERROR/' in.txt, grep ERROR in.txt and sed -n '/ERROR/p' in.txt should all print the lines containing "ERROR" anywhere on the line, though grep is made just for this. Commented Jul 14, 2016 at 21:10

3 Answers 3

Reset to default
2

GNU grep is able to do this quite simply. From man grep:

Two regular expressions may be joined by the infix operator |; the resulting regular expression matches any string matching either subexpression.

grep "ERROR\|FAIL\|WARN" /path/to/example.log

egrep eliminates the need for escaping the | symbols.

egrep "ERROR|FAIL|WARN" /path/to/example.log
Improve this answer
5
  • Why is | called "infix operator", when it means "OR" here? Also, your answer will not output following lines, if they start with whitespace. Commented Jul 14, 2016 at 17:24
  • That is a good question for which I do not know the answer. I believe the pertinent information is contained in the second half of that sentence: the resulting regular expression matches any string matching either subexpression. As for a leading whitespace, it works on any line containing ERROR, FAIL, or WARN on my system. I have edited the answer to include GNU. Commented Jul 14, 2016 at 17:38
  • Ok, thanks for looking that up. I often use the word "infix", in the context "Reverse incremental infix search" (CTRL-R), but i had never seen it written in the context you quoted. 2) I meant lines, that start with whitespace, but follow a line containing the keyword, while not themselves containing the keyword. Commented Jul 14, 2016 at 17:44
  • This is the correct answer. +1. (I had misunderstood OP's desired input format.) Commented Jul 14, 2016 at 18:25
  • @AlexStragies I've sort of found the issue that has me saying some things aren't working. The actual log file that I want to pull errors from won't work with any of the commands in this thread, but the same exact text (copied/pasted word for word from the original) pasted into a fresh text file works. I have both read/write permissions, so I'm not sure why the original log or even duplicates of it won't obey. Commented Jul 14, 2016 at 22:24
1

I suppose your log file looks like this?

example.log:

[09:44:22] [main] ERROR [url/location] - A ONE LINE ERROR
[09:44:22] [main] ERROR [url/location] - A MULTI LINE ERROR 
    with whitepace indention
[09:44:22] [main] ERROR [url/location] - A MULTI LINE ERROR 
       with tab indention
[09:44:22] [main] SOMETHING DIFFERENT
[09:44:22] [main] SOMETHING DIFFERENT
       with tab indention
[09:44:22] [main] WARN [url/location] - ANOTHER ONE LINE WARN

Admittedly not a one-liner and in perl, but it should do the job:

logsifter.pl:

#!/usr/bin/perl
use warnings;
use strict;

my $buffer="";

while(my $line= <>){
  chomp $line;
  if($line=~/ERROR|INFO|WARN/){
    print "$buffer\n" if $buffer;
    $buffer = $line;
  }
  elsif($line=~/^\s+(.*)$/){
    $buffer .= $1 if $buffer;
  }
  else{
    if($buffer){
      print "$buffer\n";
      $buffer ="";
    }
  }
}

print "$buffer\n";

call it like:

perl logsifter.pl < example.log
 [09:44:22] [main] ERROR [url/location] - A ONE LINE ERROR
 [09:44:22] [main] ERROR [url/location] - A MULTI LINE ERROR with whitepace indention
 [09:44:22] [main] ERROR [url/location] - A MULTI LINE ERROR with tab indention
 [09:44:22] [main] WARN [url/location] - ANOTHER ONE LINE WARN
Improve this answer
8
  • I tried this and the output was just every line in the text file, regardless of whether or not it included ERROR. It's entirely likely I did something wrong, as I mentioned I'm new to all this. Commented Jul 14, 2016 at 17:14
  • @Patremagne : Murphy and I, we both basically use the same algorithm. I haven't run his version, but on first glance it looks correct . You may have missed, that his version filters for ERROR|INFO|WARN, and trailing lines Commented Jul 14, 2016 at 17:18
  • @AlexStragies the only small difference is, that I remove the line breaks in case of an indent. this way everything is in one line and easier to process with grep if necessary Commented Jul 14, 2016 at 17:21
  • @AlexStragies Ahh I missed that INFO was in the perl script, so I removed it and it worked on my test text file that only includes 10 or so lines from the actual log. Unfortunately when I try it on the log itself, there's no output. Commented Jul 14, 2016 at 17:26
  • @Patremagne I tested my solution with my provided example.log. It puts out the data as shown above. Commented Jul 14, 2016 at 17:31
1

Now, that your Data format has been established, the answer becomes a lot simpler: grep was built for this.

Use as grep '<PATTERN>' <dataFile>

Where <PATTERN> is SearchWORD1 or SearchW1\|SearchW2

The answer below was written, when me and @murphy still had wrong assumptions about the dataformat:

Here is a one-line awk program that only searches for ERROR:

awk '/ERROR/{a=1;print} /^ / || /^\t/ {if (a) print;next} !/ERROR/ {a=0}'

You could make this into a flexible shell-function with parameter:

searchlog(){ awk -f <( echo "
/$1/{a=1;print}
/^ /||/^\t/{if (a) print;next}
! /$1/{a=0}
"); }

Run it either as LogData_generated_by_program | searchlog <PATTERN>, or searchlog <PATTERN> < File_containing_Log_Data.

For the example data format the other answerer "guessed", this results in:

$ searchlog ERROR < /tmp/exampleData
[09:44:22] [main] ERROR [url/location] - A ONE LINE ERROR
[09:44:22] [main] ERROR [url/location] - A MULTI LINE ERROR 
    with whitepace indention
[09:44:22] [main] ERROR [url/location] - A MULTI LINE ERROR 
       with tab indention
Improve this answer
1
  • Sorry for my Linux ignorance, but what do you mean by "push your data into STDIN" ? Commented Jul 14, 2016 at 17:08

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.