Awk prints only first word on the field column

Question

Please see my linux bash script below. I can't achieve my target output with my current code. It keeps reading the whole column 4.

input_file.txt:

REV NUM |SVN PATH         | FILE NAME     |DOWNLOAD OPTIONS
1336    |svn/Repo/PROD    | test2.txt     |PROGRAM APPLICATION_SHORT_NAME="SQLGL" |
1334    |svn/Repo/PROD    | test.txt      |REQUEST_GROUP REQUEST_GROUP_NAME="Program Request Group" APPLICATION_SHORT_NAME="SQLGL" |

my code:

# /bin/bash
REV_NUM=($(awk -F "|" 'NR>1 {print $1}' input_file.txt))
COMPONENT=($(awk -F "|" 'NR>1 {print $3}' input_file.txt))
DL_OPS="$(awk -F "|" 'NR>1 {print $4}' input_file.txt)"

#LOOP
REV_NUM_COUNT=${#REV_NUM[*]}


for (( x=0 ; x<$REV_NUM_COUNT ; x++ ))
do
         echo "${COMPONENT[x]}  ${DL_OPS[x]}"
done

actual output:

Exporting Component from SVN . . .
test2.txt  PROGRAM APPLICATION_SHORT_NAME="SQLGL"
REQUEST_GROUP REQUEST_GROUP_NAME="Program Request Group" APPLICATION_SHORT_NAME="SQLGL"
test.txt

target output:

Exporting Component from SVN . . .  
test2.txt PROGRAM APPLICATION_SHORT_NAME="SQLGL"
test.txt REQUEST_GROUP REQUEST_GROUP_NAME="Program Request Group" APPLICATION_SHORT_NAME="SQLGL"

Thank you so much

Answer 1

Read the data as pipe-delimited input instead:

while IFS='|' read -r revnum svnpath filename opts junk
do
        printf '%s: %s\n' "$filename" "$opts"
done < <(tail -n +2 file)

This would obviously retain all the extra spaces flanking most values, and would only work if the input fields do not contain embedded pipes.

Assuming all pipes are field delimiters, we can trim off the spaces by passing the read data from tail to sed -e 's/ *| */|/g' before the bash loop sees it.

In any case, we don't need to read all the data before starting to process it, as it would slow down the code and require a lot of memory and I/O if the data file is huge. Note that the code in the question reads all input data three times before starting the loop. (On the other hand, if the data was huge, we would not use a shell script to parse it, but a program written in Python, Perl, or similar scripting language.)

Output of the code above, with sed inserted to clean up the spaces:

test2.txt: PROGRAM APPLICATION_SHORT_NAME="SQLGL"
test.txt: REQUEST_GROUP REQUEST_GROUP_NAME="Program Request Group" APPLICATION_SHORT_NAME="SQLGL"

Answer 2

If you do need a shell loop to process that, you could use read's IFS-spliting to split on |s instead of using awk:

#! /bin/bash -
shopt -s extglob # for +(...) ksh-style glob operator
trim() {
  typeset -n _var
  for _var do
    _var=${_var##+([[:space:]])}
    _var=${_var%%+([[:space:]])}
  done
}

{
  IFS= read -ru3 header_discarded
  while IFS='|' read -ru3 rev svn_path file download_options rest_if_any_discarded; do
    trim rev svn_path file download_options
    # do what you need with those variables
    typeset -p rev svn_path file download_options
  done
} 3< input_file.txt

Here, on your sample, that gives:

declare -- rev="1336"
declare -- svn_path="svn/Repo/PROD"
declare -- file="test2.txt"
declare -- download_options="PROGRAM APPLICATION_SHORT_NAME=\"SQLGL\""
declare -- rev="1334"
declare -- svn_path="svn/Repo/PROD"
declare -- file="test.txt"
declare -- download_options="REQUEST_GROUP REQUEST_GROUP_NAME=\"Program Request Group\" APPLICATION_SHORT_NAME=\"SQLGL\""

That input could also be seen as simple CSV with | as field separator, so you could preprocess it with something like:

mlr --csvlite --fs '|' --ho --ragged clean-whitespace then \
  cut -of 'REV NUM,FILE NAME,DOWNLOAD OPTIONS'

Which would take care of extracting the fields you want, however they're positioned in the input and do the whitespace trimming (and then pipe to IFS='|' read -r rev file download_options)

---

As to why you're only getting the first word of each column, in:

REV_NUM=($(awk -F "|" 'NR>1 {print $1}' input_file.txt))

That unquoted $(...) used in list context (here the assignment to an array variable) is invoking the split+glob operator. The glob part you don't want so should be disabled (with set -o noglob), and the splitting part is done based on the list of characters in the $IFS variable.

By default, that contains space, tab and newline, but here you want to split on newline only.

While you could do IFS=$'\n', that would still not work if there were empty lines in the awk output as those would be discarded.

To store all the lines in an array, you'd use bash's readarray builtin:

readarray -s1 rev < <(
  awk -F'[[:space:]]*[|][[:space:]]*' '{print $1}'
)

Or:

shopt -s lastpipe
awk -F'[[:space:]]*[|][[:space:]]*' '{print $1}' |
  readarray -s1 rev

(-s1 skips the first line (same as using NR>1 in awk); we include whitespace¹ around the | in the field separator, though that would not still trim the leading ones of the first field or the trailing ones of the last field if any).

---

^{¹ beware mawk doesn't support POSIX character classes, so on systems that still use that awk implementation, you'd need to replace [[:space:]] with an explicit list of whitespace characters to trim such as [ \t\r\v\f]; mawk also doesn't support multibyte characters, so you can't include non-ASCII whitespace characters in locales using UTF-8 either.}

Answer 3

It may be overkill for the specific example you provided but I often find a good way to approach selecting input text to then pass to a shell command is to use awk to extract the text and then shell to call the external command, e.g.:

sep='|'
while IFS="$sep" read -r rev opts; do
    echo "$rev -> $opts"
done < <(
    awk -F"[[:space:]]*[$sep][[:space:]]*" -v OFS="$sep" 'NR>1{print $1, $4}' input_file.txt
)
1336 -> PROGRAM APPLICATION_SHORT_NAME="SQLGL"
1334 -> REQUEST_GROUP REQUEST_GROUP_NAME="Program Request Group" APPLICATION_SHORT_NAME="SQLGL"

That lets awk do what it does best, i.e. manipulate text, and shell do what it does best, i.e. sequence calls to tools.

The above assumes no |s or newlines within any of the fields.

Answer 4

DL_OPS is a single value, not an array. When you use array indexing with a non-array variable, index 0 is the value, other indexes are empty.

If you just wrap () around the call to awk, it will split the output at the space characters, so each word will be a separate array element. You need to split it just at newlines, which you can do by setting IFS.

You should also disable filename wildcard expansion, in case the file contains wildcards that happen to match filenames.

set -o noglob
IFS=$'\n' DL_OPS=($(awk -F "|" 'NR>1 {print $4}' input_file.txt))

Answer 5

Maybe you can simplify the entire script:

awk -F\| ' BEGIN { print "Exporting Component from SVN . . .  "} NR>1 {print $3, $4}' input_file.txt 

Answer 6

#!/bin/bash
INFILE=input.txt

while IFS='' read -r LINE || [ -n "${LINE}" ]; do
    awk -F\| '{print $3, $4}' <<< "${LINE}"
done < <(tail -n +2 $INFILE)

Output:

 test2.txt      PROGRAM APPLICATION_SHORT_NAME="SQLGL" 
 test.txt       REQUEST_GROUP REQUEST_GROUP_NAME="Program Request Group" APPLICATION_SHORT_NAME="SQLGL" 

If you want to keep the variables:

#!/bin/bash
INFILE=input.txt

while IFS='' read -r LINE || [ -n "${LINE}" ]; do
    COMPONENT="$(awk -F\| '{print $3}' <<< ${LINE})"
    DL_OPS="$(awk -F\| '{print $4}' <<< ${LINE})"
    echo "$COMPONENT $DL_OPS"
done < <(tail -n +2 $INFILE)