How to perform sed replacement from the rule file only on certain lines?

Question

I have rules.txt file with sed rules for replacement.

s/1/a/
s/2/b/
s/3/c/
etc...

I want to replace characters only after certain line 5. This command does not work.

sed -n '5,$p' -f rules.txt input_file.txt

Also, how can I add -n '5,$p' to rule file?

Answer 1

The syntax of most sed commands is:

[<address>[,<address>]] [!] <command>

(where [...] indicates optional parts).

Where <address> is to specify on which lines of the input(s) the <command> is to apply.

If two addresses, that specifies ranges or lines. An <address> can be a number or a /regexp/ (to selecting the matching lines¹) or $ to mean the last line (the GNU implementation of sed has more variations).

If no address is specified, that defaults to all the lines.

With ! that's reversed (the <command> applies on the line that are not selected by the address(es)).

s/1/a/

Is could be seen as short² for:

1,$ s/1/a/

Here, if you wanted the s/1/a/ command to apply only to lines 5 to last, you'd write:

5,$ s/1/a/

But if you don't want to modify rules.txt to specify the address range for each s command and if all of them are to be applied to lines 5 to last, what you can do is prepend a b command applied to lines 1 to 4 (or lines other than 5 to last):

sed -e 1,4b -f rules.txt < input_file.txt

Or:

sed -e '5,$!b' -f rules.txt < input_file.txt

The b command without argument branches out. That is for those lines where it's invoked (here the first to fourth), sed skips the rest of sed script (here the contents of rules.txt) and goes straight to end of the cycle where in the absence of -n, it prints the pattern space.

You can see it as short for:

sed -e '1,4 b end' -f rules.txt -e ': end' < input_file.txt

Which here branches explicitly to an end label.

You could also write it:

sed -e '5,$ {' -f rules.txt -e '}' < input_file.txt

Or:

sed -e '1,4 ! {' -f rules.txt -e '}' < input_file.txt

Where {...} is used to group sed commands, here the ones in rules.txt.

---

A few notes about your:

sed -n '5,$p' -f rules.txt input_file.txt

First, that's attempting to use options after non-option arguments, so in standard compliant sed implementations, that would try to run the 5,$p sed script on the -f, rules.txt and input_file.txt files.

Some sed implementations are non-compliant in that they accept options after non-option arguments. That's the case of the GNU implementation of sed, though only if there's no POSIXLY_CORRECT variable in the environment.

But even there, as a -f option is specified, that 5,$p would not be taken as a sed script, but as the path of input file.

Even if you wrote it:

sed -n -e '5,$p' -f rules.txt input_file.txt

Where the contents of rules.txt is concatenated to 5,$p to make up the sed script and where the lines are not printed at the end of each cycle, all that would do is print the 5^th to last lines unmodified, then apply all the substitutions (to all the lines, it's only the p commands that are restricted to lines 5 to last), but then do nothing about the result, not even printing them because of -n.

---

^{¹ well, technically, the regexp is matched on the contents of the pattern space at the point where the command is run, which might have been modified by previous commands. For instance in seq 10 | sed -n 's/2/8/; /8/,8p', that prints lines 2 (whose contents became 8 after the substitution command) to 8.}

^{² but isn't in traditional sed implementations. There, 1,$ could fail to match any line if the first line was never seen by that command such as in sed 'N; 1,$!d' or sed '1d; 1,$!d', where the first time 1,$!d is seen, the second line has already been read so that 1,$ range is never entered. In several modern sed implementations, that 1,$1 is interpreted as if(current_line >= 1), not if(line-1-has-been-seen-before)}

Answer 2

The easiest solution is to add 1,4d (i.e. delete lines 1 to 4) as the first line of your rules.txt sed script, and add a final line with just p.

1,4d
s/1/a/
s/2/b/
s/3/c/
p

and run it as sed -n -f rules.txt input_file.txt

Another alternative, if you don't want to change rules.txt is to use:

sed -n -e '1,4d' -f rules.txt -e p input_file.txt

BTW, the p command is only needed if you want to use sed's -n option. Without -n, sed defaults to printing any lines that weren't explicitly deleted whether they were modified or not - meaning that you could just use 1,4d as the first sed command in either the rules.txt file or with -e on the command line, e.g.:

sed -e '1,4d' -f rules.txt input_file.txt

Answer 3

Use sed again. If your shell supports Process Substitution, you could do:

sed -n -f <( cat rules.txt; echo '5,$p;'  ) input_file 

If not, just modify the rules file:

sed '$s/$/; 5,$p/' rules.txt > new_rules
sed -n -f new_rules input_file.txt

I think some sed implementations might need that on a separate line, in which case, you could use:

cat rules.txt > new_rules; echo '5,$p' >> new_rules

Answer 4

Here is a simple trick that would leave your rules file intact and only apply them to your desired range of lines.

sed 's/^/5,$ /' sedscript | sed -f /dev/stdin input_file

The first sed prepend the range to the lines in your rules file. That is then piped into the second sed for the actual operation.

Answer 5

sed is great for simple s/old/new/ operations but for anything remotely more interesting just use awk for some combination of clarity, simplicity, robustness, portability, maintainability, etc.

You didn't provide sample input/output to test with so this is a guess but might be what you want or at least give you a good direction.... Using any awk:

$ cat file
1234567
1234567
1234567
1234567
1234567
1234567
1234567
1234567

$ cat rules.awk
NR >= 5 {
    sub(/1/,"a")
    sub(/2/,"b")
    sub(/3/,"c")
}
{ print }

$ awk -f rules.awk file
1234567
1234567
1234567
1234567
abc4567
abc4567
abc4567
abc4567