4

There is this log file that data coming in is continuous. What I wanted to happen is to tail -f this log file then redirect it to a file with the ff conditions

example of the log files content

aaaaaaaaaaaaaaa
bbbbbbbbbbbbbbb
ccccccccccccccc
ddddddddddddddd
eeeeeeeeeeeeeee
fffffffffffffff
ggggggggggggggg
hhhhhhhhhhhhhhh
iiiiiiiiiiiiiii

in tailing, I want the redirection to start when it found the first pattern then stop the redirection as well as tailing when I found the second pattern. e.g

I want the redirection starts when it found pattern "ddddddddddddddd" then will stops when it found "hhhhhhhhhhhhhhh" the contents of the file created from redirection should be

ddddddddddddddd
eeeeeeeeeeeeeee
fffffffffffffff
ggggggggggggggg
hhhhhhhhhhhhhhh

In coding I'm thinking of something like this. But I don't have idea how to stop the redirection when it found the second pattern. 

tail -f logfile > log.tmp
while grep "ddddddddddddddd" log.tmp
do
cat log.tmp > logfile
done
Improve this question

3 Answers 3

Reset to default
4

This is exactly what retail does. retail is tail with regular expressions, a tool I wrote for exactly the use case you have here. In your case, you'd use:

retail -f -r ddddddddddddddd -u hhhhhhhhhhhhhhh logfile > log.tmp

-f is the standard tail -f option. -r takes a regular expression to use to start the range of lines to include, and -u takes a regular expression for it to continue until. It begins at the last occurrence of the -r pattern and exits after printing the first occurrence of the -u pattern thereafter.

If you want to start from the first instance of the pattern you can use -b in combination with -r. Both regular expressions are EREs, without any implicit anchoring, but you can use ^ and $ to anchor the match as usual.

You can obtain and build retail with:

git clone https://github.com/mwh/retail.git
cd retail
./configure
make
make install

It installs into ~/.local/bin by default, but you can change that, or just copy the executable where you want.

retail is fully POSIX-compatible, though I don't recommend actually using it as your system tail.

Improve this answer
8
  • is this a redhat package or debian? how can i install this? Commented Apr 24, 2015 at 8:37
  • There are installation instructions between the horizontal rules. As far as I know there are neither Red Hat nor Debian packages. Commented Apr 24, 2015 at 8:39
  • can you send me the link and instructions how to install this? TIA Commented Apr 24, 2015 at 8:42
  • Look about ten centimetres above your comment, where the grey box is. Commented Apr 24, 2015 at 8:45
  • 1
    @mikeserv: No -r at the moment, but maybe sometime. The POSIX compatibility was more of a why-not than anything. It has exactly the options mentioned above plus -c. Commented Apr 24, 2015 at 9:07
0

If I understand your demands correct this does what you expect:

awk '/ddddddddddddddd/,/hhhhhhhhhhhhhhh/ { print > "excerpt" } 1' infile | tail -f

awk works as filter on file "infile"; it prints everything in between the given patterns into a file "excerpt", and it also prints every line to standard output, which is then processed by tail -f as usual.

In case you want to use it with streamed data:

some_process | awk '/ddddddddddddddd/,/hhhhhhhhhhhhhhh/ { print > "excerpt" } 1'  | tail -f

(Note: Since tail -f is an interactive program it should come, if possible, always last in a pipeline.)

Improve this answer
9
  • If the OP is considering tail -f at all then it suggests the source file is growing over time. In this situation awk would fall off the end of the partially created file and your tail -f would never see any more input. Commented Apr 24, 2015 at 9:50
  • @roaima; That's why I provided the second variant; don't create the file but pipe data directly from the generating process. If you want the logfile with all data anyway persistent then use tee: some_process | tee logfile | awk '...' | tail -f. Commented Apr 24, 2015 at 12:13
  • In neither example does the tail -f add any value Commented Apr 24, 2015 at 12:38
  • @roaima; Huh? - If you have a long continuously written logfile you want to follow it as it's written. Wasn't that the point of the request?! To explain the (optional) parts: the tee logfile is if you want a persistent complete logfile, the awk part is to filter the pattern-matched subsection and store it separately on the fly, and the tail -f is to watch the log interactively while it's generate. Commented Apr 24, 2015 at 12:49
  • Yes exactly! Contrast for F in seq 9; do echo "$F"; sleep 1; done | awk '/4/,/7/' | tail -f with and without the tail. Similarly, to simulate from a growing file, for F in seq 9; do echo "$F"; sleep 1; done >tmp & sleep 1 ; awk '/4/,/7/' tmp | tail -f also with and without the tail Commented Apr 24, 2015 at 14:01
0

Maybe you can use awk's range patterns here:

tail -f logfile | awk '/ddddddddddddddd/,/hhhhhhhhhhhhhhh/'

If you need to circumvent the SIGPIPE problem you can use socat instead of tail for the job:

socat -u file:logfile,ignoreeof "system:'stdbuf -o0 awk /ddddddddddddddd/,/hhhhhhhhhhhhhhh/'" > logfile.new
Improve this answer
2
  • You might need an exit in the awk code to generate SIGPIPE back to the tail so that it also stopped Commented Apr 24, 2015 at 9:38
  • In that case I would use socat for the job. Commented Apr 24, 2015 at 9:44

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.