3

I need to look for some requests in a huge pile of apache logs. My only requirement is this:

I need to only view the requests coming from any IP address that is NOT included in a list of 50 IP ranges I have.

How can I make that happen using any combination of regexes awk grep or anything? Can't think of an easy way. The idea would be to get each line, get the first part (the IP address), match it to a file with all the ranges, and if its not there, then display it. No idea on how to go about doing this, so any help is welcome!

Samples:

a Typical http log line is 

123.456.789.012 - - [22/Oct/2012:06:37:48 +0100] "GET /test/test HTTP/1.1" 302 224 "-" "some user agent/4.3.5"

A typical line out of my IP ranges file is 

192.168.0.1 - 192.168.0.255

Of cours ethe IP ranges file could be converted to 192.168.0.1/24 notation if necessary. The good thing is that all the ranges are Class C (just noticed that), so I guess only the first 3 parts of the IP address could be matched and that should be good enough.

Improve this question
2
  • Can you please give a sample of your file and also the file containing the ranges? Commented Oct 22, 2012 at 9:24
  • @Guru done, thx Commented Oct 22, 2012 at 9:35

2 Answers 2

Reset to default
3

A simple and crude way could be to use grep.

Create a file ( ranges.txt ) with you're ranges something like this:

192\.168\.0\.[0-9]*
10\.0\.0\.[0-9]*

To create that file from the range-file you already had you could use sed like so:

sed -n -e 's/^\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\) .*$/^\1\\.\2\\.\3\\.[0-9]* /p' your-range-file > ranges.txt

Then exclude lines matching the pattern in that file using grep like so:

$ grep -v -f ranges.txt apache-log-file.log

or

$ cat apache-log-file.log | < do some pre cleaning > | grep -v -f ranges.txt

This could help you to get started but its probably not a good solutions if the query should be run often and on big log-files.

Good luck!

Improve this answer
0

Log is your log file, and iprange is your file containing the ipranges. Perl part of the solution gets the first 3 components of your ip address, and the for loop prints it if it does not exist in the file ipranges:

for i in `perl -lne 'print $1 if (m/(\d{1,3}\.\d{1,3}\.\d{1,3})\.\d{1,3}/);' log`
> do
>  grep -q $i iprange || echo $i;
> done
Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.