1

I have this bash script:

sed -i -r '/deb http\:\/\/httpredir\.debian\.org\/debian jessie main/d' /etc/apt/sources.list
sed -i -r '/deb http\:\/\/httpredir\.debian\.org\/debian jessie\-updates main/d' /etc/apt/sources.list
sed -i -r '/deb http\:\/\/security\.debian\.org jessie\/updates main/d' /etc/apt/sources.list
echo -e 'deb http://httpredir.debian.org/debian stable main contrib non-free\ndeb-src http://httpredir.debian.org/debian stable main contrib non-free\ndeb http://httpredir.debian.org/debian jessie-backports main contrib non-free\ndeb-src http://httpredir.debian.org/debian jessie-backports main contrib non-free\ndeb http://httpredir.debian.org/debian jessie-updates main contrib non-free\ndeb-src http://httpredir.debian.org/debian jessie-updates main contrib non-free\ndeb http://security.debian.org jessie/updates main contrib non-free\ndeb-src http://security.debian.org jessie/updates main contrib non-free' >> /etc/apt/sources.list

If I run in this order, there's no problem, the matching lines will be deleted from Debian's source list file. But if I echo-ing first, than deleting lines, it will remove line 5-7 also.

Changing SPACEs to \s not helping.

This can cause trouble in the future, when my Dockerfile not will be updated fast enough and Debian getting a new minor release.

Seems like if a global tag was being applied to this regex. Defining the line start and end also not helping in bash (although on regexr works great, this is what I want to get): http://regexr.com/3b910

Seems like you can't define /igm flags with sed. sed still deleting the entire matching line.

EDIT: I shortened this way, but I do something wrong because it's deleting everything from the file:

sed /etc/apt/sources.list -i -e '\!deb http://httpredir\.debian\.org/debian jessie main$!d' -e '\!deb http://httpredir\.debian\.org/debian jessie-updates main$!d' -e '\!deb http://security\.debian\.org jessie/updates main$!d' -e '$a \ndeb http://httpredir.debian.org/debian stable main contrib non-free\ndeb-src http://httpredir.debian.org/debian stable main contrib non-free\ndeb http://httpredir.debian.org/debian jessie-backports main contrib non-free\ndeb-src http://httpredir.debian.org/debian jessie-backports main contrib non-free\ndeb http://httpredir.debian.org/debian jessie-updates main contrib non-free\ndeb-src http://httpredir.debian.org/debian jessie-updates main contrib non-free\ndeb http://security.debian.org jessie/updates main contrib non-free\ndeb-src http://security.debian.org jessie/updates main contrib non-free'

This is my modification for the problem (the RUN command is Dockerfile script):

RUN sed -i '/^deb http\:\/\/httpredir\.debian\.org\/debian jessie main$\|^deb http\:\/\/httpredir\.debian\.org\/debian jessie\-updates main$\|^deb http\:\/\/security\.debian\.org jessie\/updates main$/d' /etc/apt/sources.list
RUN echo -e 'deb http://httpredir.debian.org/debian stable main contrib non-free\ndeb-src http://httpredir.debian.org/debian stable main contrib non-free\ndeb http://httpredir.debian.org/debian jessie-backports main contrib non-free\ndeb-src http://httpredir.debian.org/debian jessie-backports main contrib non-free\ndeb http://httpredir.debian.org/debian jessie-updates main contrib non-free\ndeb-src http://httpredir.debian.org/debian jessie-updates main contrib non-free\ndeb http://security.debian.org jessie/updates main contrib non-free\ndeb-src http://security.debian.org jessie/updates main contrib non-free' >> /etc/apt/sources.list

The reason why I separating the deletion and echoing process, because maybe I will use this Dockerfile for another, some new super lightweight Linux distros, which not have sed default installed, but probably have echo. This way if I can't delete the lines, still echoing my modification.

The problem with multiline solution that I want to keep it lightweight (only one Dockerfile) and I don't want to include any external .sh script.

Improve this question

1 Answer 1

Reset to default
1

Your sed patterns are not anchored on the end so they match the new lines. Another solution is to modify the lines instead of deleting and readding.

First let's see if we can make the original script a little more readable by not being so wide:

sed -i -r '/deb http\:\/\/httpredir\.debian\.org\/debian jessie main/d' /etc/apt/sources.list
sed -i -r '/deb http\:\/\/httpredir\.debian\.org\/debian jessie\-updates main/d' /etc/apt/sources.list
sed -i -r '/deb http\:\/\/security\.debian\.org jessie\/updates main/d' /etc/apt/sources.list
cat >> /etc/apt/sources.list << EOF
deb http://httpredir.debian.org/debian stable main contrib non-free
deb-src http://httpredir.debian.org/debian stable main contrib non-free
deb http://httpredir.debian.org/debian jessie-backports main contrib non-free
deb-src http://httpredir.debian.org/debian jessie-backports main contrib non-free
deb http://httpredir.debian.org/debian jessie-updates main contrib non-free
deb-src http://httpredir.debian.org/debian jessie-updates main contrib non-free
deb http://security.debian.org jessie/updates main contrib non-free
deb-src http://security.debian.org jessie/updates main contrib non-free
EOF

You know actually looking at it, there is no reason we can't do the whole thing with one sed command.

sed /etc/apt/sources.list -i \
  -e '\!deb http://httpredir\.debian\.org/debian jessie main$!d' \
  -e '\!deb http://httpredir\.debian\.org/debian jessie-updates main$!d' \
  -e '\!deb http://security\.debian\.org jessie/updates main$!d' \
  -e '$a \
deb http://httpredir.debian.org/debian stable main contrib non-free\
deb-src http://httpredir.debian.org/debian stable main contrib non-free\
deb http://httpredir.debian.org/debian jessie-backports main contrib non-free\
deb-src http://httpredir.debian.org/debian jessie-backports main contrib non-free\
deb http://httpredir.debian.org/debian jessie-updates main contrib non-free\
deb-src http://httpredir.debian.org/debian jessie-updates main contrib non-free\
deb http://security.debian.org jessie/updates main contrib non-free\
deb-src http://security.debian.org jessie/updates main contrib non-free'

So let's talk about what I did. First was to add the $ to the end of the patterns, although at this point it is not needed because with using a single sed command your race condition window is much narrower, maybe even removed because the original file stays in place until sed is done at which point sed copies its temp file over the original and this only happens once. next is to reduce the over escaping. specifically - and : are not special characters in either basic or extended regular expressions so they do not need escaping (and doing so can officially lead to undefined behavior per POSIX, but seldom does in practice). While we are at it there are a lot of slashes to be escaped so let's change our delimiter to ! since it is not used anywhere by using the \c match operator. And speaking of the differences between basic and extended regular expressions you are not using any of the contested operators, so it does not matter which your patterns are interpreted as so let's drop the -r option. Now since we are using multiple sed commands in a single sed invocation we need the -e option and we need it for each sed command. This has the added advantage in this case that the commands do not have to be before the files to be modified. And we finish it off with the a command to append the new lines, and we limit it with $ so we only do it at the last line. We could save six characters by not escaping . as it matches itself and is unlikely to match anything else do to the long patterns and the nature of urls, but good enough.

or here is a perl version:

perl -pi \
  -e 'next if m!deb http://httpredir\.debian\.org/debian jessie main$!;' \
  -e 'next if m!deb http://httpredir\.debian\.org/debian jessie-updates main$!;' \
  -e 'next if m!deb http://security\.debian\.org jessie/updates main$!;' \
  -e 'END print "
deb http://httpredir.debian.org/debian stable main contrib non-free
deb-src http://httpredir.debian.org/debian stable main contrib non-free
deb http://httpredir.debian.org/debian jessie-backports main contrib non-free
deb-src http://httpredir.debian.org/debian jessie-backports main contrib non-free
deb http://httpredir.debian.org/debian jessie-updates main contrib non-free
deb-src http://httpredir.debian.org/debian jessie-updates main contrib non-free
deb http://security.debian.org jessie/updates main contrib non-free
deb-src http://security.debian.org jessie/updates main contrib non-free\n";}'

which can be one lined like this:

perl -pie 'next if m!deb http://httpredir\.debian\.org/debian jessie main$!; next if m!deb http://httpredir\.debian\.org/debian jessie-updates main$!; next if m!deb http://security\.debian\.org jessie/updates main$!; END{ print "deb http://httpredir.debian.org/debian stable main contrib non-free\ndeb-src http://httpredir.debian.org/debian stable main contrib non-free\ndeb http://httpredir.debian.org/debian jessie-backports main contrib non-free\ndeb-src http://httpredir.debian.org/debian jessie-backports main contrib non-free\ndeb http://httpredir.debian.org/debian jessie-updates main contrib non-free\ndeb-src http://httpredir.debian.org/debian jessie-updates main contrib non-free\ndeb http://security.debian.org jessie/updates main contrib non-free\ndeb-src http://security.debian.org jessie/updates main contrib non-free\n";}'

Improve this answer
10
  • Wow, thank you very much, this is great! One question: can you modify this to be one liner? Commented Jun 24, 2015 at 16:07
  • It technically is a one liner, but the a command has to have backslash newline pairs to do what you want. it could be shortened by removing the backslash newline pairs before the -e options, but that would only save four lines. Commented Jun 24, 2015 at 16:16
  • I shortened to one liner, but I do something wrong because it's deleting everything from the file. Can you please check it? My reason heaving one liner is just I can easily enter the command in windows cmd, when I run Debian under Win Vagrant, Docker container. Commented Jun 24, 2015 at 16:29
  • \n does not have special meaning in this context you need a backslash newline. As for making it easy to type, do you have a webserver on the local network? if you do you can put the script on the web server and run it with wget -O - www.local|bash Commented Jun 24, 2015 at 16:46
  • This script will be in a Dockerfile, to fire up a new docker image (it will be debian:latest with a few personal modification, installing sudo, etc.). It will have "RUN" command in front of the code showed here, that will run on the image creation process. I don't know unfortunately if that Dockerfile can take multiline bash commands or not. It's possible to call external .sh scripts with Dockerfile also, but I try to make it lightweight (only one Dockerfile). Commented Jun 24, 2015 at 16:53

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.