sed: multi-line replace of config block

Question

I have some configuration files that basically look like

(...content...)
# BEGIN DYNAMIC BLOCK - DO NOT EDIT MANUALLY
(... more content ...)
# END DYNAMIC BLOCK
(... even more content ...)

Now, in bash using CONTENT=`wget -O - http://$SERVER/get_config.php`, I have the replacement for the dynamic block.

How do I do the replace now and how do I make the script insert the block at file's end if it isn't there?

Answer 1

If you want to use sed, you can read from a named pipe. Beware that this code doesn't try to cope with errors. The script will block if the dynamic block header is present more than once.

CONTENT_URL="http://$SERVER/get_config.php"
tmp=$(mktemp -d)
(
  cd "$tmp"
  mkfifo dynamic_seen dynamic_content
  : >dynamic_seen & seen_pid=$!
  wget -O dynamic_content "$CONTENT_URL" & wget_pid=$!
  sed -e '/^# BEGIN DYNAMIC BLOCK - DO NOT EDIT MANUALLY$/ p' \
      -e '/^# END DYNAMIC BLOCK$/ {'
          -e p -e 'r dynamic_seen' -e 'r dynamic_content' -e '}' \
      -e '/^# BEGIN DYNAMIC BLOCK - DO NOT EDIT MANUALLY$/, /^# END DYNAMIC BLOCK$/ d'
  if ! kill $dynamic_seen 2>/dev/null; then
    # The pipe hasn't been read, so there was no dynamic block. Add one.
    echo "# BEGIN DYNAMIC BLOCK - DO NOT EDIT MANUALLY"
    cat dynamic_pipe
    echo "# END DYNAMIC BLOCK - DO NOT EDIT MANUALLY"
  fi
)
rm -rf "$tmp"

But I'd go for awk.

export CONTENT_URL="http://$SERVER/get_config.php"
awk '
    $0 == "# END DYNAMIC BLOCK - DO NOT EDIT MANUALLY" {skip=0; system("wget \"$CONTENT_URL\""); substituted=1}
    !skip {print}
    $0 == "# BEGIN DYNAMIC BLOCK - DO NOT EDIT MANUALLY" {skip=1}
    END {
         if (!substituted) {
            print "# BEGIN DYNAMIC BLOCK - DO NOT EDIT MANUALLY";
            system("wget \"$CONTENT_URL\"");
            print "# END DYNAMIC BLOCK - DO NOT EDIT MANUALLY";
        }
    }
'

Answer 2

I would go with a sub-shell and two sed commands, something like this:

beg_tag='# BEGIN DYNAMIC BLOCK - DO NOT EDIT MANUALLY'
end_tag='# END DYNAMIC BLOCK'

(
  sed "/^$beg_tag"'$/,$d' oldconf
  echo "$beg_tag"
  wget -O - http://$SERVER/get_config.php
  echo "$end_tag"
  sed "1,/^$end_tag/d" oldconf
) > newconf

Be careful not to put any sed-significant characters into beg_tag and end_tag.

This will append the output if no tags are present. The first sed command will never delete any lines from the input and the second sed command will delete all lines.

Testing

If oldconf contains:

(...content...)
# BEGIN DYNAMIC BLOCK - DO NOT EDIT MANUALLY
(... more content ...)
# END DYNAMIC BLOCK
(... even more content ...)

And the wget command is replaced by echo hello world, the output is:

(...content...)
# BEGIN DYNAMIC BLOCK - DO NOT EDIT MANUALLY
hello world
# END DYNAMIC BLOCK
(... even more content ...)

Now if the block is removed, i.e. the following input is used:

(...content...)
(... even more content ...)

The output is:

(...content...)
(... even more content ...)
# BEGIN DYNAMIC BLOCK - DO NOT EDIT MANUALLY
hello world
# END DYNAMIC BLOCK

Answer 3

This is fairly simple to do with sed really. You just have to balance ranges of lines against one another and as anchored to EOF.

INPUT |
sed -e 's/\\/&&/g;$!s/$/\\/' |        #this sed escapes INPUT for scripting
sed -e '/^'"$START"'/,$!{$!b          #this sed applies concatenated scripts
             G;G;s/$/'"$END"'/;P;:n
};$!N;  /\n'"$END"'/,$!{G;$!bn
};      /\n\n/c\' -f - -e 'P;$d;D
' ./named_infile >outfile

So, there are a few things going on there, but the most important of them are these:

/^$START/,$!{ -- function --}
N; /\n$END/,$!{ -- function -- }

The idea is that when we anchor a line range to either of Line 1 or $EOF we have essentially just made it greedy. Usually line ranges apply only to the smallest subset of lines to which they might - starting anew for every LHS match, and ending for the very first RHS match which next occurs in input. If the RHS is EOF though, well, they can only ever be applied once - because there's only one of those.

When I do:

/^$START/,$!{ -- function -- }

I specify that all of the code between the curlies is run for every line in the infile up to but not including $START. In this function context I branch away for every line which is !not the $last.

In this way all lines up to the first $START in input are printed automatically and ignored, but if the $ last line falls within this range - as it might if $START never occurs even once - then it is prepped to be changed to your string.

And so if your range does not occur in input the INPUT gets appended to the tail end of the file.

When I next do:

N; /\n$END/,$!{ -- function -- }

I am again applying a function contextually. This time it is applied to the body of your range - and the only the first occurrence of it input - because the complement of /\n$END/,$ is all lines that were not branched away before the first $START, and only up to and not including the next occurring $END.

In this case the function applied is a branch loop - as long as input falls within that range it will continue to branch back and pull in the Next line until it finds the first $END match, at which point it changes the entire range to the contents of -f - the stdin script file - or your escaped input. This same rule is applied to the last line in the event it occurs before the first $START match.

And that's it. Note though that this does not require any special files to work - because it (safely) incorporates a copy of INPUT within its script, it does not need to read it in at any time to apply when necessary.