3

As usual, I can inspect the contents of syslog entries in this way:

cat /var/log/syslog | grep myentry

I need to append all myentry rows to a specific file. Of course just redirecting the output of the command above to the file will not work, because it will append all the rows, even if they were already added last time.

The first solution that comes to mind is to cycle among all the rows in syslog until I find the last row of the target file. Then I can append all the following ones. Doing this periodically (i.e. using a cronjob or even easier a timed cycle in bash) should do the trick.

Is there something smarter or more elegant to do the same job?

EDIT

I add what terdon requested:

Example of my syslog:

Jan 17 13:03:18 stm32mp1-abc local2.info chat[15782]: CONNECT
Jan 17 13:03:18 stm32mp1-abc local2.info chat[15782]:  -- got it
Jan 17 13:03:18 stm32mp1-abc user.info myentry[14300]: Ready
Jan 17 13:03:18 stm32mp1-abc local2.info chat[15782]: send (^M)
Jan 17 13:03:18 stm32mp1-abc user.info myentry[14300]: Init complete
Jan 17 13:03:18 stm32mp1-abc daemon.info pppd[14362]: Serial connection established.
Jan 17 13:03:18 stm32mp1-abc daemon.info pppd[14362]: Using interface ppp0

Example of the existing file I want to append to:

Jan 17 13:03:18 stm32mp1-abc user.info myentry[14300]: Ready

Final output I expect from those two files:

Jan 17 13:03:18 stm32mp1-abc user.info myentry[14300]: Ready
Jan 17 13:03:18 stm32mp1-abc user.info myentry[14300]: Init complete

UPDATE

Ok, it seems I need to be very specific with the example, regardless my description. So new examples:

syslog

Jan 17 13:03:18 stm32mp1-abc local2.info chat[15782]: CONNECT
Jan 17 13:03:18 stm32mp1-abc local2.info chat[15782]:  -- got it
Jan 17 13:03:18 stm32mp1-abc user.info myentry[14300]: Ready
Jan 17 13:03:18 stm32mp1-abc local2.info chat[15782]: send (^M)
Jan 17 13:03:18 stm32mp1-abc user.info myentry[14300]: Init complete
Jan 17 13:03:18 stm32mp1-abc daemon.info pppd[14362]: Serial connection established.
Jan 17 13:03:18 stm32mp1-abc user.info myentry[14300]: Start operations
Jan 17 13:03:18 stm32mp1-abc daemon.info pppd[14362]: Using interface ppp0

current output

Jan 17 13:03:18 stm32mp1-abc user.info myentry[14300]: Init complete

new output

Jan 17 13:03:18 stm32mp1-abc user.info myentry[14300]: Init complete
Jan 17 13:03:18 stm32mp1-abc user.info myentry[14300]: Start operations

It should be clear:

  1. if the output file is empty, append all the syslog/myentries rows
  2. if the output file is not empty, append all the next syslog/myentries rows (from the matching one)
  3. if the output file is not empty and there's no matching, still append all the next syslog/myentries rows (after the last timestamp)

As said, I'm able to do this using the brute force: cycle all rows, check criteria and append if needed.

I'm looking for an easier solution, like the proposal to split the syslog entries automatically, but I didn't find a way to do it.

Improve this question
6
  • 1
    Please edit your question and add i) an example of your syslog and ii) an example of the existing file you want to append to and iii) the final output you expect from those two files. That way, we can test our solutions and we can be sure we understand what you need. Commented Nov 30, 2020 at 11:50
  • Why dont you configure rsyslog to log/duplicate those lines into separate logfile? Commented Nov 30, 2020 at 12:48
  • @Fiisch, I have syslog from Busybox v1.29.3 and I didn't find how to specify a separate logfile for my entries. If you can elaborate a bit your answer I think it might be a good solution. Commented Nov 30, 2020 at 12:55
  • I see. The "busybox" part is important here. Man page for busybox-syslogd explicitly states that /etc/syslog.conf is ignored here. Your options thus are: 1) the one you are doing now; 2) write to named pipe instead of a file and have some infinite loop read the other end and save the output; 3) Busybox's syslog can log over the network when using -R host[:port] so it is basically like option 2 but saves you the hassle of named pipes. manpages.debian.org/jessie/busybox-syslogd/syslogd.8.en.html Commented Nov 30, 2020 at 15:21
  • @thanasisp, sorry I don't see a contradiction here. I try with other words: it may happen that the output files contains old logs. When you run the script, the file is not empty but there is no matching (due to the old data) with the current syslog. But if the timestamp is after the last entry it's correct to append. Commented Dec 1, 2020 at 8:08

2 Answers 2

Reset to default
1

Store the last line of the existing target file. And then parse backwards the syslog file with awk and some help from tac:

p=$(tail -n1 output)
tac syslog | awk -v p="$p" '$0 == p{exit} /myentry/' | tac >> output

When the last line is found (backwards), awk will exit. If not found, or the output was initially empty, it will parse the whole file (I assume there are not empty lines into syslog for the case of initially empty output). Also this way we search into the minimum required lines, so it is expected to perform well for large files.

Also keep in your toolbox this simple effective command:

awk '!seen[$0]++' output > output.tmp && mv output.tmp output

that you can run anytime, to fix the target file, preserving the order, if for any reason duplicates managed to pass into there.

Improve this answer
5
  • If I'm not wrong with my tests, given a myentry line in output file, it will copies all the lines, not only the next ones - like the terdon solution does. Commented Nov 30, 2020 at 12:59
  • I just tested with your new example and it works as expected. Have you tested that? Commented Nov 30, 2020 at 13:33
  • Yes I tested it but at first something was wrong, now it seems to work fine, perhaps it was my fault. I tried to use a variable (like /$MYENTRY/) instead of hardcode it but finds nothing. What is the correct syntax here? Commented Dec 16, 2020 at 11:04
  • 1
    The robust way to pass a shell variable to an awk program is using -v. Don't forget quoting. If this variable will be used as the pattern to match, here you are: awk -v my_var="$MYENTRY" '$0 ~ my_var{print "MATCHING"}' file. (Just keep in mind that you will never use $ or // inside awk for matching a pattern from variable, you will be using the variable alone) Commented Dec 16, 2020 at 11:09
  • 1
    Just a note, ~ stands for partial matching (like we grep a line), == full, exact match, equality of the two strings. Commented Dec 16, 2020 at 11:17
1

Here's a simple approach:

$ awk 'NR==FNR{a[$0]++; next}/myentry/ &&  !a[$0]' myentries syslog 
Jan 17 13:03:18 stm32mp1-abc user.info myentry[14300]: Init complete

That selects the new matching line, so all you need to do is redirect to the original file:

awk 'NR==FNR{a[$0]++; next}/myentry/ &&  !a[$0]' myentries syslog >> myentries
Improve this answer
2
  • I found two caveats: if the target file is empty it copies nothing (instead it should copy all the lines). Then I tried to put inside a valid line, but it copied all of them, not only the next ones. Commented Nov 30, 2020 at 12:51
  • @Mark well yes. You wanted to specifically avoid duplicates. There's no point in running this if you don't have duplicates. If the file is empty, just run grep myentry /var/log/syslog. I don't understand the second point. Please add an example to your question. I used the data you gave and it worked as expected. Commented Nov 30, 2020 at 13:10

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.