2

I'm working on a script that chews on some data it sucks in from a CSV file. I've already read the data into several arrays (one for each column in the file); I now need to actually work with all of the data in sequence.

Currently, I am doing this:

# Read in the data:
declare -a DATACOL1 DATACOL2 RAWDATA
RAWDATA=($( sed '1d' /path/to/data.csv )) # Remove the header line
for line in ${RAWDATA[@]}; do
   declare -a LINEDATA LINE
   LINE=$( echo "$line" | sed 's/,/ /g' )
   for field in LINE; do
       LINEDATA+=("${field}")
   done
   DATACOL1+=(${LINEDATA[0]})
   DATACOL2+=(${LINEDATA[1]})
done


# Work on the data:
for i in $( seq 0 $[${#DATACOL1[@]}-1}; do
   stuff and things with ${DATACOL1[i]} and ${DATACOL2[i]}
done

My (quite possibly interrelated) questions are twofold:

  • Is there a more elegant way to later work in the data than for i in $( seq 0 $[${#DATACOL1[@]}-1} for iterating over them? It works, but is ugly.

  • Is there a more elegant way to suck in the CSV data?

This is on bash 3, so I do not have associative arrays.

Improve this question
6
  • Can we write it in perl? (typo: seed -> sed) Commented Nov 17, 2015 at 19:08
  • This would probably be a lot easier to do in perl/python, but it's preferred in this instance to stick with bash, as it's one cog in a fairly extensive bash Rube Goldberg machine, and I'd just as soon keep all the cogs speaking the same language. Commented Nov 17, 2015 at 19:09
  • 1
    Your method of reading the file will break if there is any whitespace. Commented Nov 17, 2015 at 19:28
  • 1
    there's nothing elegant about shell arrays. they are large, ponderous, and wasteful. Commented Nov 17, 2015 at 20:33
  • 1
    I didn't say elegant, I said more elegant; it's a quality I'm striving for even if it can't actually be reached. Commented Nov 17, 2015 at 20:42

1 Answer 1

Reset to default
4

I would write:

mapfile -t rawdata < <(sed 1d /path/to/data.csv)
datacol1=()
datacol2=()

for line in "${rawdata[@]}"; do
    IFS=, read -ra fields <<< "$line"
    datacol1+=( "${fields[0]}" )
    datacol2+=( "${fields[1]}" )
done

for ((i=0; i < "${#datacol1[@]}"; i++)); do
    stuff with "${datacol1[i]}" and "${datacol2[i]}"
done
  • use mapfile to read the lines of a file into an array
  • use IFS and read to read comma-separated fields from a line
    • will break with any commas inside quoted strings: use a real CSV parser.
  • use the C-like form of for to avoid calling out to seq

Of course, you don't need the 2nd loop or the datacol* variables

for line in "${rawdata[@]}"; do
    IFS=, read -a fields <<< "$line"
    stuff with "${fields[0]}" and "${fields[1]}"
done

On bash 3 you won't have mapfile, so use a while-loop

datacol1=()
datacol2=()
while IFS=, read -ra fields; do
    datacol1+=("${fields[0]}")
    datacol2+=("${fields[1]}")
done < <(sed 1d /path/to/data.csv)
Improve this answer
2
  • Thanks, mucking about with IFS is obvious in retrospect and is exactly the answer I needed. And the for loop is more verbose, but a lit less ugly than seq 0 $[${#DATACOL1[@]}-1}. Thanks for both. Commented Nov 17, 2015 at 20:10
  • This doesn't seem to want to run in parallel for me, the last example. Commented May 12, 2021 at 18:06

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.