Why grep does not work properly in this case?

Question

Consider a file file2.txt having the following content:

P 89 24 -1.5388040474568784e+01 7.4421775186012660e+00 -1.3143195543234219e+03 1.3168884860257754e+03 8.0419002445999993e+01 44 0 0 -97 0
P 122 -4 -1.4869334602986523e+01 5.7316939411954255e+00 -1.3144161801429666e+03 1.3169704096915282e+03 8.0419002445999993e+01 44 0 0 -370 0
P 493 -24 -1.4690576431881317e+01 7.3848907323212831e+00 -1.3144620647251766e+03 1.3170224315489374e+03 8.0419002445999993e+01 62 0 0 -499 0
E 3 -1 -1.0000000000000000e+00 -1.0000000000000000e+00 -1.0000000000000000e+00 9999 0 970 1 2 0 7 1.7003962000000002e+05 8.5019810000000018e-01 8.5019810000000018e-01 8.5019810000000018e-01 3.0000000000000000e+01 3.8153441026312507e+01 1.0000000000000000e+11
E 4 -1 -1.0000000000000000e+00 -1.0000000000000000e+00 -1.0000000000000000e+00 9999 0 818 1 2 0 7 1.7003962000000002e+05 8.5019810000000018e-01 8.5019810000000018e-01 8.5019810000000018e-01 3.0000000000000000e+01 3.2509364886711985e+01 1.0000000000000000e+11
P 5 2 0 0 3.7531787088999999e+02 3.8383684055052936e+02 8.0419002445999993e+01 22 0 0 -6 0
P 8 24 7.0195398693654170e+00 3.1543502387874696e+01 5.5989200759599044e+01 1.0318077843755555e+02 8.0419002445999993e+01 44 0 0 -50 0
P 67 28 5.8271676589304882e+00 3.3476871962084061e+01 5.6723118833601163e+01 1.0411236719963519e+02 8.0419002445999993e+01 44 0 0 -168 0
P 219 13 6.0328453988772415e+00 3.3531592253635168e+01 5.6777179460595200e+01 1.0417114266715717e+02 8.0419002445999993e+01 44 0 0 -329 0
P 444 -24 6.4646967953734418e+00 3.4909545978243479e+01 5.7879920796889749e+01 1.0525098522544691e+02 8.0419002445999993e+01 62 0 0 -452 0
E 5 -1 -1.0000000000000000e+00 -1.0000000000000000e+00 -1.0000000000000000e+00 9999 0 598 1 2 0 7 1.7003962000000002e+05 0 0 8.5019810000000018e-01 3.0000000000000000e+01 6.8997318544430456e+01 1.0000000000000000e+11

I want to extract only the strings P ... 24 ... or P ... -24 .... This is what I do:

cat file2.txt | grep -E '(P [0-9]+ 24 | P [0-9] + -24 |P [0-9][0-9]+ 24 | P [0-9][0-9] + -24 |P [0-9][0-9][0-9] + 24 | P [0-9][0-9][0-9] + -24 |P [0-9][0-9][0-9][0-9]+ 24 | P [0-9][0-9][0-9][0-9] + -24 )' &> file3.txt

But the resulting file3.txt contains only the strings P ... 24. Could you please tell me what I am doing wrong?

Answer 1

.... what am I doing wrong? ... apart from making it far more complicated that it needs to be ... you are trying to match multiple spaces and leading spaces that are not in your strings in all of the cases for -24 and in some of other cases too ....

P [0-9]+ 24 |

is fine P then then a series of digits [0-9]+, then and 24 followed by a space

| P [0-9] + -24 |

here there is a before the P and after the digits you had one or more spaces + followed by another which fails to match because of the additional spaces

|P [0-9][0-9]+ 24 |

fine again though all matches are already caught in the first pattern and so it is redundant

| P [0-9][0-9] + -24 |

extra spaces, the same as -24 above ... no match

|P [0-9][0-9][0-9] + 24 |

extra space before the + so it is looking for 2 or more again...

| P [0-9][0-9][0-9] + -24 |

leading space before the P and 2 or more before -24 again.

|P [0-9][0-9][0-9][0-9]+ 24 |

fine but redundant

| P [0-9][0-9][0-9][0-9] + -24

leading space before the P and 2 or more before -24 again.

While @gillesquenot has a far more elegant solution, yours "works" if you lose the additional spaces...

grep -E '(P [0-9]+ 24 |P [0-9]+ -24 |P [0-9][0-9]+ 24 |P [0-9][0-9]+ -24 |P [0-9][0-9][0-9] + 24 |P [0-9][0-9][0-9]+ -24 |P [0-9][0-9][0-9][0-9]+ 24 |P [0-9][0-9][0-9][0-9]+ -24 )'

And if you have the possibility of multiple spaces

grep -E '^P +[0-9]+ +-?24'

edit

This is a useful resource for seeing what matches and where in any string

Answer 2

Seems a task for awk ! This is DSL ¹: exactly the case where awk is the way to go.
Simple, re-usable, efficient, and quick.

without regex:

awk '$1 == "P" && ($3 == "-24" || $3 == "24")' file2.txt 

or using regex:

awk '$1 == "P" && $3 ~ /^-?24$/' file2.txt 

Here, in the regex /^-?24$/, the dash is optional, which permit the use of this short regex.

---

The regular expression matches as follows:

Node	Explanation
^	the beginning of the string
-?	'-' (optional (matching the most amount possible))
24	'24'
$	before an optional \n, and the end of the string

---

¹ _{^{Question to the most known chat}}

Q:what means DSL for a developer?

DSL (Domain-Specific Language) is a type of computer language designed to make coding specific tasks easier. It is tailored to a particular application domain, allowing developers to write code quickly and efficiently. Common usage of DSL include database queries, processing text and scientific simulations. DSLs allow developers to quickly build applications for a particular purpose, offering a more intuitive approach to programming that is less reliant on syntax

Answer 3

Let's simplify a bit the regex:

grep -E '^P [0-9]+ -?24\b' file2.txt 

The main issue from your regex (apart non needed complexity and readability) is that you use extra erroneous spaces.

No need:

• the use of cat | grep, it's a useless use of cat
• to use a multiple|regex here

If you think you can have more space than just one in your input, you can use one of:

• +
• [[:space:]]+ (POSIX character class)
• \s+ require -P aka PCRE switch of grep

The regular expression matches as follows:

Node	Explanation
^	start of the line anchor
P	'P' + space
[0-9]+	any character of: '0' to '9' (1 or more times (matching the most amount possible))
	a space
-?	'-' (optional (matching the most amount possible))
24	'24'
\b	word boundary

Answer 4

Your regular expression seems to have random spaces inserted into it. A space in a regular expression will match a space in the text you apply it to. Also note that [0-9]+ will match the same strings as [0-9][0-9]+ and [0-9][0-9][0-9]+ (the reverse is not true though).

You should therefore be able to use

grep -E '^P [0-9]+ -?24 ' file2.txt

Note the use of the anchor to ensure that the P is matched at the start of the line, and -? to match an optional dash before 24.

This is identical to what's suggested in another answer, apart from in the end, where I use a simple space in place of the non-standard word-boundary pattern \b. I can do that if I know that there is a space to be matched in the data, which the example seems to suggest there is.

---

With Miller (mlr), treating the input as a file with implicitly-integer-indexed fields ("nidx") on each line, filtering out (extracting) the lines with a P in the first field and a third field with an absolute value of exactly 24:

$ mlr --nidx filter '$1 == "P" && abs($3) == 24' file2.txt
P 89 24 -1.5388040474568784e+01 7.4421775186012660e+00 -1.3143195543234219e+03 1.3168884860257754e+03 8.0419002445999993e+01 44 0 0 -97 0
P 493 -24 -1.4690576431881317e+01 7.3848907323212831e+00 -1.3144620647251766e+03 1.3170224315489374e+03 8.0419002445999993e+01 62 0 0 -499 0
P 8 24 7.0195398693654170e+00 3.1543502387874696e+01 5.5989200759599044e+01 1.0318077843755555e+02 8.0419002445999993e+01 44 0 0 -50 0
P 444 -24 6.4646967953734418e+00 3.4909545978243479e+01 5.7879920796889749e+01 1.0525098522544691e+02 8.0419002445999993e+01 62 0 0 -452 0

Answer 5

um, you're relying on only that one field possibly having value 24. That seems like a bad design – there's a lot of tools that could just extract the first and third columns from such a file, for example cut, for further analysis.

Honestly, though: This is a job for a tool that deals with tabular numerical data, not for a string engine like sed or cut or grep.

In all honesty, a handful of lines of Python (or Perl, or… whatever programming language you like) would solve this; and: I'm 100% certain you'll deal with the results here programmatically anyways, so just use that language to begin with.

Just to illustrate:

#!/usr/bin/env python3
import sys

with open(sys.argv[1], "r", encoding="ascii") as inputfile:
  linecounter = 0
  for line in inputfile:
    linecounter += 1
    try:
      if not line.startswith("P "):
         continue # skip this line alltogether
      thirdpos = int(line.split(" ")[2])
      if not (thirdpos == 24 or thirdpos == -24):
         continue
      print(line) # or actually, do something useful with the line
    except Exception as e:
      print("Exception occurred on line {linecounter}, '{line}':\n{e}", sys.stderr)

in a text file, chmod 755 thatfile, then running /path/to/thatfile /path/to/input_file would do.

Answer 6

If you want to extract the full line and match -24/24 only in the third field then use awk:

$ awk '{if (($1=="P") && ($3==24 || $3==-24)) print}' infile

P 89 24 -1.5388040474568784e+01 7.4421775186012660e+00 -1.3143195543234219e+03 1.3168884860257754e+03 8.0419002445999993e+01 44 0 0 -97 0
P 493 -24 -1.4690576431881317e+01 7.3848907323212831e+00 -1.3144620647251766e+03 1.3170224315489374e+03 8.0419002445999993e+01 62 0 0 -499 0
P 8 24 7.0195398693654170e+00 3.1543502387874696e+01 5.5989200759599044e+01 1.0318077843755555e+02 8.0419002445999993e+01 44 0 0 -50 0
P 444 -24 6.4646967953734418e+00 3.4909545978243479e+01 5.7879920796889749e+01 1.0525098522544691e+02 8.0419002445999993e+01 62 0 0 -452 0

Answer 7

Another way to do it:

grep -P "P [^ ]+ -?24 " FILE 

which means grep for P, followed for a blank, some non blanks followed by another blank, optionally followed by a minus sign and 24, then again a blank.

If the blank might as well be a tab or more than one of both:

grep -P "P[\t ]+[^ \t]+[\t ]+-?24[\t ]+" FILE 

With -P while using gnu-grep, you enable Perl style regexes which allow to search for tab-characters by \t.

A leading caret isn't necessary, if the discriminating letter is always in column 1.

If column 2 is always positive, [0-9]+ is sufficient. -?[0-9] would work too for negative numbers.

Answer 8

The issue with the regular expression used in the grep command is that there is an extra space after the second set of digits in the pattern. For example, "P 89 24" will match but "P 89 -24" will not match because of the extra space.

To fix this, remove the extra space between the second set of digits and the next character in the pattern:

cat file2.txt | grep -E 'P [0-9]+ (24|-24) | P [0-9][0-9]+ (24|-24) | P [0-9][0-9][0-9]+ (24|-24) | P [0-9][0-9][0-9][0-9]+ (24|-24)' > file3.txt

This should match both "P ... 24" and "P ... -24" patterns.

Another way to simplify the regular expression is to use the optional "?" operator to match the "-" symbol:

cat file2.txt | grep -E 'P [0-9]+ -?24 | P [0-9][0-9]+ -?24 | P [0-9][0-9][0-9]+ -?24 | P [0-9][0-9][0-9][0-9]+ -?24' > file3.txt

This will match both "P ... 24" and "P ... -24" patterns with a single regular expression.

Answer 9

Put the content in the file file.txt, then: Work with grep:

grep -E 'P\s+\S+\s+([+-]24|\b24)\b' file.txt

For regex, try post it in regex101.com, with regex:

/^P\s+\d+\s+(24|\-24)\s+[\d\.\+\-e]+\s+[\d\.\+\-e]+\s+[\d\.\+\-e]+\s+[\d\.\+\-e]+\s+[\d\.\+\-e]+\s+\d+\s+\d+\s+\d+\s+\-?\d+\s+\d+$
/gm