0

I am trying to process a text file and omit a certain string literal if it occurs at the end of the line. E.g.:

Source:

ABC 123
DEF, characters I don't want
GHI, these characters are ok

Desired Output:

ABC 123
DEF
GHI, these characters are ok

If I do grep -v ', characters I don't want$', it omits that entire line.

I can't do a simple awk column since I want the , these characters are ok substring

I can't use cut to split on a delimiter because the delimiter needs to be multiple characters (, characters I don't want).

With Python it would be super simple with something like: string.split(", characters I don't want", 1)[0]

(Tangentially, I'm wondering in which use cases it really is preferable to use grep, awk, or sed in complicated situations like this vs Python when Python is so much more readable and maintainable.)

Improve this question
3
  • Would string.split(", characters I don't want", 1)[0] really only find that string at the end of a line or would it find it anywhere in a line? Is it splitting on a literal string or would it treat that argument as a regexp (in which case you'd have to escape regexp metachars first)? Commented Jul 20, 2023 at 20:02
  • A better example to demonstrate your needs would have been DEF, .* and trying to match on , .* as that'd test if a potential solution was really doing a literal string or a regexp comparison. It'd also be good to have a line where your searched string appears multiple times on the line (e.g. DEF, .*, .*) to see if a potential solution really is matching at the end of the line or anywhere else. There are other use-cases you'll want to cover too I expect. Commented Jul 20, 2023 at 20:09
  • @EdMorton Good point - I don't currently need it to only match at the end of the line, but it would be ideal. Python's .split() would find first occurrence and it's a "bytes-like object", but you can pass in a regex pattern too. Commented Jul 21, 2023 at 13:02

3 Answers 3

Reset to default
3

Most obvious here would be to use sed:

<source sed "s/, characters I don't want\$//"

To substitute that string when found at the end of the line $ which we escape as \$ for the shell (to be future proof in case $/ means something in the shell in the future).

To also remove whatever follows that string if any, replace the \$ with .*, though we'd need to change the locale the C to guarantee .* matches everything up to the end even if that's not valid text in the user's locale:

<source LC_ALL=C sed "s/, characters I don't want.*//"

With GNU grep or compatible, when built with perl-like regexp support, that could be:

<source LC_ALL=C grep -Po "^.*?(?=(, characters I don't want)?\$)"

Or to remove everything if any after that string as well:

<source LC_ALL=C grep -Po "^.*?(?=, characters I don't want|\$)"

Or with pcregrep (when perl-like regex support is enabled in GNU grep, that's actually via libpcre which comes with pcregrep as an example application though has features beyond those of GNU grep):

<source pcregrep -o1 "^(.*?)(, characters I don't want)?\$"

Or to remove everything if any after that string as well:

<source pcregrep -o1 "^(.*?)(, characters I don't want|\$)"

If the text to remove may contain anything including / or regex operators (but not newline characters which wouldn't make sense, nor NUL characters which can be passed in command arguments nor environment variables) and is stored in a shell variable, you do not want to use sed "s/$string\$//" as that would make it a command injection vulnerability.

With the perl-grep ones, you can use:

string='/.*\^$'
<source LC_ALL=C grep -Po "^.*?(?=(\Q$string)?\$)"
<source pcregrep -o1 "^(.*?)(\Q$string\E)?\$"

Or to remove everything if any after that string as well:

<source LC_ALL=C grep -Po "^.*?(?=\Q$string|\$)"
<source pcregrep -o1 "^(.*?)(\Q$string\E|\$)"

That still chokes on $strings that contain \E, though not with as dramatic consequences as with sed.

Or you could use perl directly which has a sed mode with its -p option, has mechanisms to pass arbitrary strings (here using -s for a crude option passing, but you could also use @ARGV directly (equivalent of python's sys.argv) or environment variables (mapped to the %ENV associative array)), and can \Quote strings inside regexps (here with \E in $string not being a problem):

<source perl -spe 's/\Q$string\E$//' -- -string="$string"

Or to remove everything if any after that string as well:

<source perl -spe 's/\Q$string\E.*$//' -- -string="$string"

perl treats input as bytes not as if encoded in the user's locale charset by default, so we don't need to change the locale there.

Note that contrary to sed, the line delimiter is included in the pattern space ($_ in perl on which s/// acts by default) by default and its $ regex operator matches either at the end of the subject or before a line delimiter at the end of the subject so is able to cope with both delimited and undelimited lines.

Improve this answer
1
  • Thanks! This worked perfectly. And I could just drop it in to my existing piping and redirection setup: NEW_BREW_PACKAGES=$(bash -c "comm -23 <(sort ~/Brewfile | grep -v '^#' | sed 's/, link: false\$//') <(sort BrewfileSuperset | grep -v '^#'| sed 's/, link: false\$//')") Commented Jul 21, 2023 at 15:24
1

Using any awk:

$ awk 'n=index($0 RS,", characters I don\047t want" RS){$0=substr($0,1,n-1)} 1' file
ABC 123
DEF
GHI, these characters are ok

That's doing a literal string comparison so it'd work even if the string you're trying to match with contained regexp metachars, for example using this input:

$ cat file2
ABC 123
DEF, .*, .*
GHI, .* ok

We get the expected output:

$ awk 'n=index($0 RS,", .*" RS){$0=substr($0,1,n-1)} 1' file2
ABC 123
DEF, .*
GHI, .* ok

If you didn't care about regexp metachars you could just do:

$ awk '{sub(/, characters I don\047t want$/,"")} 1' file
ABC 123
DEF
GHI, these characters are ok

but then you'd get unexpected output from:

$ awk '{sub(/, .*$/,"")} 1' file2
ABC 123
DEF
GHI

and you'd have to escape the metachars to make them literal to get the expected output:

$ awk '{sub(/, \.\*$/,"")} 1' file2
ABC 123
DEF, .*
GHI, .* ok

which is getting cludgy given all you really wanted was a literal string comparison.

See http://awk.freeshell.org/PrintASingleQuote for why I'm using \047 instead of '.

As for why to use awk instead of python - awk is a mandatory POSIX tool and so is guaranteed to exist on all POSIX-compliant Unix installations while python is not, and it usually takes much less code to manipulate text with awk than it does with python. I suspect we will have to agree to disagree on which is more easily readable and maintainable.

Improve this answer
2
  • Thanks. I went ahead with the accepted sed substitution method which also happened to work well with my specific redirection and piping situation. As an awk novice, a lot of this syntax is opaque to me. Whereas with Python, a lot of the syntax is readable to me as an English speaker, even if you don't know much Python. But the POSIX-compatibility is certainly relevant. Especially compared with Python and its runtime and environment uncertainties. But I do seem to run into inconsistencies on nix boxes when shell scripting due to bash, zsh, and various commands being executed differently. Commented Jul 21, 2023 at 15:22
  • awk is mostly just a tiny subset of C wrapped in an implicit while-read loop and a <condition> { <action> } body structure. Once you understand THAT about awk then if you can understand python, you'll find it trivial to understand and write awk (I use both). Commented Jul 21, 2023 at 20:18
0

Filtering out the contents at the end of a line when those contents are known in advance can be fairly easy in Bash and shells that support similar variable expansion functions. For example:

#!/usr/bin/env bash
line='DEF, characters I do not want'
echo "${line%, characters I do not want}"

will print:

DEF

The syntax ${var%string} returns the contents of $var with the string after the % removed from the end of the contents. In this example the string to be removed is ", characters I do not want". If that string isn't at the end, the full contents of $line are returned. There are variations for removing a string from the start of the variable, as well as a substitution that can replace a string in the middle of the contents or delete it.

I admit in the above example to changing don't -> do not in order to avoid complications from using single quotes when assigning the string to the $line variable.

The advantage of this approach is that your script doesn't need to invoke an external command to perform the simple filtering. But is it a replacement for the power of python?. Probably not, but there may be other factors that push you toward using a shell script rather than python for this task.

Improve this answer
2
  • Note that it's standard sh syntax. You don't need to install bash for that. Note that echo can't be used to output arbitrary data whether that's bash's echo builtin one or most other implementations. Commented Jul 21, 2023 at 6:25
  • I didn't know about the % option with echo to omit a known string literal. I tried this solution first but couldn't quite get it to work with the specific piping and redirection I already had. So I went with the sed substitution method. Commented Jul 21, 2023 at 15:14

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.