Bash - remove dashes and new-lines before replacing new-lines with spaces

Question

I have a bunch of text files in the following format:

Lorem ipsum dolor sit amet,
consetetur sadipscing elitr,
sed diam nonumy eirmod tempor
invidunt ut labore et dolore
magna aliquyam erat, sed diam
voluptua. - At vero eos et accu-
sam et justo duo dolores et ea
rebum. - Stet clita kasd guber-
gren, no sea takimata sanctus
est Lorem ipsum dolor sit amet.

How can I print this as continuous text on the command line, but with removing the syllable division on the line ends:

Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. - At vero eos et accusam et justo duo dolores et ea rebum. - Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.

I could use tr '\n' ' ' to convert the new-lines into spaces

The problem is tr can only replace one character and I would need some command to remove the -\n in advance. How can I achieve this on the bash comman-line?

Answer 1

Using awk:

awk -F'-$' '{ printf "%s", sep $1; sep=/-$/?"":OFS } END{ print "" }' infile

with the -F'-$', we defined the Field Separator to single hyphen at the end of line, so with this and by taking the first field $1, we will always have the line without that hyphen for those line having this hyphen or still entire line for those not having that hyphen.

then we do simply printing it with a sep in between but that changes when reading the next line to empty-string if current line was ending with a hyphen otherwise to OFS (Output Field Separator, default is Space character).

at the END{...} block we are adding a final newline character to make it a POSIX text file, if you don't want that to be added, just remove that part.

---

Using sed, alternatively:

sed ':loop /-$/N;s/-\n//;t loop; N;s/\n/ /;t loop' infile

• :loop
  • if a line ended with a hyphen (testing with /-$/), do read the Next line and replace the "hyphen+\newline" with an empty string.
    if substitution was successful (testing with t), then jump to the label loop and process the next line and skip executing rest of the code.
  • else, read the Next line and replace the embedded \newline in between those two lines with a space character.
    if substitution here was also successful, then jump to the label loop and process the next line.

Answer 2

Another perl solution (assuming the last line doesn't end with - character):

perl -pe 's/-\n//; s/\n/ / if !eof'

Unlike sed or awk, the record separator can be directly manipulated using perl. So, it is easy to delete if -\n is matched. s/\n/ / if !eof will then replace remaining newlines with space, except for the last line.

If you don't care about an extra space at the end, you can remove the if !eof part.

Answer 3

1. Remove dashes used for hyphenation at the end of lines (joining hyphenated lines with the next line in the process).
2. Replace newlines with spaces.

sed -e ':again' -e '/[[:alpha:]]-$/ { N; s/-\n//; b again; }' file |
paste -s -d ' ' -

This uses sed join hyphenated lines. It does this by detecting lines that have dashes that are immediately preceded by a letter at the end of lines. By matching [[:alpha:]] in front of the dash, I'm more certain that I'm only removing dashes used for hyphenation. The hyphenated lines are then joined with the next line by reading in the next line with N (which inserts a literal newline character) and deleting the -\n. The joined line is then re-checked for further hyphenation (this is what the b again branch to the again label does).

The paste utility is then used to paste all lines into one continuous line. The -s option is used to concatenate all lines into a single line, and -d ' ' is used to specify the delimiter character used in the concatenation of the lines.

I could have used tr '\n' ' ' in place of the paste command that I ended up with, but that would have given me an unterminated line (no newline character at the end).

Answer 4

Depending on how you'd want blank lines handled this:

$ awk -v RS= '{gsub(/-\n/,""); $1=$1} 1' file
Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. - At vero eos et accusam et justo duo dolores et ea rebum. - Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.

or this (if your awk supports multi-char RS, e.g. GNU awk):

$ awk -v RS='^$' '{gsub(/-\n/,""); $1=$1} 1' file
Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. - At vero eos et accusam et justo duo dolores et ea rebum. - Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.

may be what you want.

Answer 5

Using the POSIX sed stream editor utility.

sed -e ':a
  $!N
  s/-\n//;ta
  s/\n/ /;ta
' file

---

With awk utility, we setup a loop to read the next line and manipulate the ORS for printing the previous line.

awk '
{
  while (getline t > 0) {
    ORS = sub(/-$/, "") ? "" : OFS
    print
    $0 = t
  }
  ORS = RS
}
1' file

---

We can rearrange the output record separator $\ based on the condition of the current record.

perl -lpe '
  $\ =   eof  ? $/
     : s/-$// ? $,
     :          $"
     ;
' file

• $/ is the input record separator and defaults to a newline.
• $" is the array element separator under quotes and defaults to a space.
• $, is the output field separator and defaults to an empty string.