How to make grep ignore lines without trailing newline character

Question

I'd like to grep a file for a string, but ignore any matches on lines that do not end with a trailing newline character. In other words, if the file does not end with a newline character, I'd like to ignore the last line of the file.

What is the best way to do this?

I encountered this issue in a python script that calls grep via the subprocess module to filter a large text log file before processing. The last line of the file might be mid-write, in which case I don't want to process that line.

Answer 1

With gawk (using EREs similar to grep -E):

gawk '/pattern/ && RT' file

RT in gawk contains what is matched by RS the record separator. With the default value of RS (\n) that would be \n except for a non-delimited last record where RT would then be empty.

With perl (perl REs similar to grep -P where available):

perl -ne 'print if /pattern/ && /\n\z/'

Note that contrary to gawk or grep, perl by default does work on bytes not characters. For instance, it's . regexp operator would match on each of the two bytes of a UTF-8-encoded £. For it to work on characters as per the locale's definition of characters like for awk/grep, you'd use:

perl -Mopen=locale -ne 'print if /pattern/ && /\n\z/'

Answer 2

grep is explicitly defined to ignore newlines, so you can't really use that. sed knows internally if the current line (fragment) ends in a newline or not, but I can't see how it could be coerced to reveal that information. awk separates records by newlines (RS), but doesn't really care if there was one, the default action of print is to print a newline (ORS) at the end in any case.

So the usual tools don't seem too helpful here.

However, sed does know when it's working on the last line, so if you don't mind losing the last intact line in cases where a partial one isn't seen, you could just have sed delete what it thinks is the last one. E.g.

sed -n -e '$d' -e '/pattern/p'  < somefile                   # or
< somefile sed '$d' | grep ...

If that's not an option, then there's always Perl. This should print only the lines that match /pattern/, and have a newline at the end:

perl -ne 'print if /pattern/ && /\n$/'

Answer 3

Something like this could do the job:

#!/usr/bin/env sh

if [ "$(tail -c 1 FILE)" = "" ]
then
    printf "Trailing newline found\n"
    # grep whole file
    # grep ....
else
    printf "No trailing newline found\n"
    # ignore last line
    # head -n -1 FILE | grep ...
fi

We rely on the following characteristic of command substitution described in man bash:

Bash performs the expansion by executing command and replacing the command substitution with the standard output of the command, with any trailing newlines deleted.

Answer 4

If you need speed then using PCRE (or some other possibly faster regex library) from C would allow the use of both a regular expression and a check whether there is a newline. Downsides: new code to maintain and debug, time to re-implementing portions of grep or perl depending on the complexity of the expression or if features such as --only-matching are used.

#include <err.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

#include <pcre.h>
#define MAX_OFFSET 3

int main(int argc, char *argv[])
{
    // getline
    char *line = NULL;
    size_t linebuflen = 0;
    ssize_t numchars;
    // PCRE
    const char *error;
    int erroffset, rc;
    int offsets[MAX_OFFSET];
    pcre *re;

    if (argc < 2) errx(1, "need regex");
    argv++;
    if ((re = pcre_compile(*argv, 0, &error, &erroffset, NULL)) == NULL)
        err(1, "pcre_compile failed at offset %d: %s", erroffset, error);

    while ((numchars = getline(&line, &linebuflen, stdin)) > 0) {
        if (line[numchars-1] != '\n') break;
        rc = pcre_exec(re, NULL, line, numchars, 0, 0, offsets, MAX_OFFSET);
        if (rc > 0) fwrite(line, numchars, 1, stdout);
    }
    exit(EXIT_SUCCESS);
}

This is about 49% faster than perl -ne 'print if /.../ && /\n\z/'.