3

I am cleaning up my data that is stored in text files. Each line starts with a category label followed by the actual data that I want to clean up. There are many text files in different subfolders, so I use egrep to pass the filenames on to sed.

CON: the Unix and Linux question
SEM: eins, the zwei, drei
AUTH: , the
AFF: The holy seat
TTITLE: As we go, the Kuckuck comes too

Now in every line starting with (SEM|AFF|CON) I want to replace (T|t)he[ ]* when it follows (:|\,). That is, the data should later look like

CON: Unix and Linux question
SEM: eins, zwei, drei
AUTH: , the
AFF: holy seat
TTITLE: As we go, the Kuckuck comes too

So far I tried to achieve this in two steps, one for the :-part and the other for the ,-part. But I struggle already with the first step.

First part The command/pattern to identify the files is egrep -rl ^"(SEM|CON|AFF)\: (t|T)he". This works as intended.

Now when I do

egrep -rl ^"(SEM|CON|AFF)\: (t|T)he" | xargs sed -i 's/\((SEM|CON|AFF)\: \)(t|T)he[ ]*/\1/g'

nothing happens. Is my sed part wrong? Can't I backrefer to ((SEM|CON|AFF)\: with \1?

Second part The command/pattern to identify the files is egrep -rl ^"(SEM|CON|AFF)\:.*\,[ ]*(t|T)he". This also works as intended. But every combination on sed that I tried so far deletes the content.

Improve this question
3
  • Unless you escape the ( in a Basic RE, it will be seen as a literal (. Try \(\(SEM|CON|AFF\): \)\(t|T\). Or use extended REs (sed -r), and replace all \( with just (. Since you used egrep, you're getting extended REs for free. Commented Dec 10, 2014 at 11:39
  • You mean, I don't need -e when I use egrep? Commented Dec 10, 2014 at 11:47
  • egrep is like grep -E. -e is used to denote an expression. Commented Dec 10, 2014 at 12:13

4 Answers 4

Reset to default
4

I would use the following:

sed -r '/(SEM|AFF|CON)/ s/([:,] *)[Tt]he */\1/g' file

Add -i option to change file in place.

Improve this answer
3

Just use one sed expression (needs GNU sed):

sed -r -i -e '/(SEM|AFF|CON)/s/([:,]\s*)the\s+/\1/ig' *

The search pattern at the begin of the sed command restricts the replacement to the lines which begins with the selected categories. The i flag for the replace command (s//) makes the pattern case-insensitive; the g flag allows more than one replacement in a line. \s stands for any whitespace (space, tabs).

The sed uses extended regular expressions with the -r flag; this is for example supported by GNU sed. The -i flag enables in-place edit. The shell expands * to all files in the current directory.

The pattern ensures at least one space or tab after the so that words starting with the like theater are not replaced.

Improve this answer
1
  • 1
    Perhaps use \b\s* rather than \s+, so that it matches at end of line, too? Commented Jul 23, 2023 at 8:30
1

Try this way:

egrep -rl "^(SEM|CON|AFF)\: (t|T)he" * | xargs sed -r -i 's/(^(SEM|CON|AFF):\s)((t|T)he[ ]*)/\1/g'
Improve this answer
1
  • This works well for the first part, but how can I proceed with the second part, i.e. where the[ ]* follows a comma? Also, why the * in front of the pipe? Commented Dec 10, 2014 at 11:52
0

Using Raku (formerly known as Perl_6)

~$ raku -pe 's/^ [SEM|AFF|CON] <+[:,]>  \s  <( [T|t] he \s+ //;'  file

Sample Input:

CON: the Unix and Linux question
SEM: eins, the zwei, drei
AUTH: , the
AFF: The holy seat
TTITLE: As we go, the Kuckuck comes too

Sample Output:

CON: Unix and Linux question
SEM: eins, the zwei, drei
AUTH: , the
AFF: holy seat
TTITLE: As we go, the Kuckuck comes too

Basically you can solve this problem easily if you understand/implement code using capture markers. In Raku, the capture marker symbols are <(…)>. Not both capture markers need to be used.

What do capture markers do? Above in the recognition half of the s/// substitution operator all elements get recognized:

^ [SEM|AFF|CON] <+[:,]> \s <( [T|t] he \s+

but because of the capture marker(s), only everything to the right of <( is retained as a "capture", i.e.:

<( [T|t] he \s+

Now that we have recognized a larger sequence, and retained a subsequence in a "capture", we replace that "capture" with nothing, obtaining the desired result. Note: the final regex atom in the code above is \s+, not \s*. This prevents The being deleted from the title, e.g. "Theory of Relativity".

https://docs.raku.org/language/regexes#Capture_markers:_%3C(_)%3E
https://docs.raku.org/language/regexes
https://raku.org

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.