Treat a column with text that has spaces as 1 field

Question

I have a file with the following format:
INTEGER INTEGER TEXT

The text is unicode and can have spaces.
I am trying to use awk in order to print the first INTEGER and the TEXT in a file in a specific format using printf.
Problem: because TEXT in some lines has spaces the $3 does not have the complete TEXT so the line is broken in more fields.

Example:

12 42956    Cinema - 3D/Multiplex  
7  12560    Status Update  
5  184   Movie  

My approach for this is the following:

awk '{ c=$3; for(i=4; i< NF;++i){c=c" "$i}; printf "<tag>%d</tag>\n<tag>%s</tag>\n", $1,c}';  

But I thought there might be a better approach

Answer 1

awk is useful if the data comes in well designated records. This data does not. However, the data is on the format "integer stuff the_rest" where both "integer" and "stuff" won't have spaces in them. This happens to be exactly what the read utility likes to read. It will read whitespace-separated words, as many as you give it variables to read, and then it will put "the rest" of the line into the last variable.

bash-4.4$ while read -r integer stuff the_rest; do printf '%d\t"%s"\n' "$integer" "$the_rest"; done <data
12      "Cinema - 3D/Multiplex"
7       "Status Update"
5       "Movie"

It will automatically strip off any trailing whitespace.

Answer 2

To extract fields based on a pattern, perl is generally better than awk:

perl -lne '
  if (/^\s*(\d+)\s*\S+\s*(.*?)\s*$/) {
    print "<tag>$1</tag><tag>$2</tag>"
  }'

which on your input gives:

<tag>12</tag><tag>Cinema - 3D/Multiplex</tag>
<tag>7</tag><tag>Status Update</tag>
<tag>5</tag><tag>Movie</tag>

That means you can do more advanced stuff like do proper HTML encoding if needed with for instance:

perl -Mopen=locale -MHTML::Entities -lne '
  if (/^\s*(\d+)\s*\S+\s*(.*?)\s*$/) {
    print map {"<tag>" . encode_entities($_) . "</tag>"} $1, $2
  }'

Or XML encoding:

perl -Mopen=locale -MXML::LibXML -lne '
  if (/^\s*(\d+)\s*\S+\s*(.*?)\s*$/) {
    print map {
      my $e = XML::LibXML::Element->new("tag");
      $e->appendText($_);
      $e->toString} $1, $2
  }'

Answer 3

Replace the $2 (that you don't use anyway) for an unused character (one that doesn't exist in your strings). After that, just do:

awk '{$2="+";print}' input-file.txt | awk -F "+" '{printf "<tag>%d</tag>\n<tag>%s</tag>\n",$1,$2}'

Above I used the plus "+" as the separator.

It is not the most elegant solution, but it is simple.

Answer 4

I think you might want something like

awk '{$2=""; print;}' input

Answer 5

If this is not a huge file and since the text is always at the end, as an alternative you might consider a classic bash approach like :

while IFS=' ' read -r int1 int2 text;do
#do your stuff
done <file

As happens with while - read , the last var $text in the read command will get all the remaining fields as one field.

Testing:

$ IFS=' ' read -r int1 int2 text <<<"10 5 some text here"
$ echo "$text"
some text here

Bash while read can perform quite slow in big data files, but you can give a try to your case.