Array of arrays in bash

Question

I'm attempting to read an input file line by line which contains fields delimited by periods. I want to put them into an array of arrays so I can loop through them later on. The input appears to be ok, but 'pushing' that onto the array (inData) doesn't appear to be working.

The code goes :

Input file: 
GSDB.GOSALESDW_DIST_INVENTORY_FACT.MONTH_KEY
GSDB.GOSALESDW_DIST_INVENTORY_FACT.ORGANIZATION_KEY


infile=${1}

OIFS=$IFS
IFS=":"

cat ${infile} | while read line
do
      line=${line//\./:}
      inarray=(${line})
#      echo ${inarray[@]}
#      echo ${#inarray[@]}      
#      echo ${inarray[0]}
#      echo ${inarray[1]}
#      echo ${inarray[2]}

      inData=("${inData[@]}" "${inarray[@]}")
done 
IFS=$OIFS

echo ${#inData[@]}   

for ((i = 0; i < ${#inData[@]}; i++))
do
 echo $i
    for ((j = 0; j < ${#inData[$i][@]}; j++))
    do
       echo ${inData[$i][$j]}
    done
done

Answer 1

Field nest box in bash but it can not circumvent see the example.

#!/bin/bash

# requires bash 4 or later; on macOS, /bin/bash is version 3.x,
# so need to install bash 4 or 5 using e.g. https://brew.sh

declare -a pages

pages[0]='domain.de;de;https'
pages[1]='domain.fr;fr;http'

for page in "${pages[@]}"
do
    # turn e.g. 'domain.de;de;https' into
    # array ['domain.de', 'de', 'https']
    IFS=";" read -r -a arr <<< "${page}"

    site="${arr[0]}"
    lang="${arr[1]}"
    prot="${arr[2]}"
    echo "site : ${site}"
    echo "lang : ${lang}"
    echo "prot : ${prot}"
    echo
done

Answer 2

Bash has no support for multidimensional arrays. Try

array=(a b c d)
echo ${array[1]}
echo ${array[1][3]}
echo ${array[1]exit}

For tricks how to simulate them, see Advanced Bash Scripting Guide.

The output of the 3 echo commands is:

b
bash: ${array[1][3]}: bad substitution
bash: ${array[1]exit}: bad substitution

Answer 3

Knowing that you can split string into "array". You could creat a list of lists. Like for example a list of databases in DB servers.

dbServersList=('db001:app001,app002,app003' 'db002:app004,app005' 'dbcentral:central')

# Loop over DB servers
for someDbServer in ${dbServersList[@]}
do
    # delete previous array/list (this is crucial!)
    unset dbNamesList
    # split sub-list if available
    if [[ $someDbServer == *":"* ]]
    then
        # split server name from sub-list
        tmpServerArray=(${someDbServer//:/ })
        someDbServer=${tmpServerArray[0]}
        dbNamesList=${tmpServerArray[1]}
        # make array from simple string
        dbNamesList=(${dbNamesList//,/ })
    fi

    # Info
    echo -e "\n----\n$someDbServer\n--"

    # Loop over databases
    for someDB in ${dbNamesList[@]}
    do
        echo $someDB
    done
done

Output of above would be:

----
db001
--
app001
app002
app003

----
db002
--
app004
app005

----
dbcentral
--
central

Answer 4

I struggled with this but found an uncomfortable compromise. In general, when faced with a problem whose solution involves using data structures in Bash, you should switch to another language like Python. Ignoring that advice and moving right along:

My use cases usually involve lists of lists (or arrays of arrays) and looping over them. You usually don't want to nest much deeper than that. Also, most of the arrays are strings that may or may not contain spaces, but usually don't contain special characters. This allows me to use not-to-confusing syntax to express the outer array and then use normal bash processing on the strings to get a second list or array. You will need to pay attention to your IFS delimiter, obvi.

Thus, associative arrays can give me a way to create a list of lists like:

declare -A JOB_LIST=(
   [job1] = "a set of arguments"
   [job2] = "another different list"
   ...
)

This allows you to iterate over both arrays like:

for job in "${!JOB_LIST[@]}"; do
  /bin/jobrun ${job[@]}
done

Ah, except that the output of the keys list (using the magical ${!...}) means that you will not traverse your list in order. Therefore, one more necessary hack is to sort the order of the keys, if that is important to you. The sort order is up to you; I find it convenient to use alphanumerical sorting and resorting to aajob1 bbjob3 ccjob6 is perfectly acceptable.

Therefore

declare -A JOB_LIST=(
   [aajob1] = "a set of arguments"
   [bbjob2] = "another different list"
   ...
)
sorted=($(printf '%s\n' "${!JOB_LIST[@]}"| /bin/sort))
for job in "${sorted[@]}"; do
   for args in "${job[@]}"; do
     echo "Do something with ${arg} in ${job}"
   done
done

Answer 5

This answer doesn't answer the specifics of the OP's question, which already has adequate answers. Rather, it just answers the title of the question:

Array of arrays in bash

More specifically:

How to create 2D, 3D, n-d, etc. multidimensional arrays in Bash

The basis of this answer is found in my demo code, array_of_arrays__2D_3D_multidimensional_arrays.sh in my eRCaGuy_hello_world repo.

Bash doesn't support multidimensional arrays natively, and has no concept of "arrays of arrays", but that doesn't matter. As can be done in any language including C (see Example 3 in my answer here) or in Bash, you can just use a 1D array for an n-D array and then manually index into it as though it was an n-d array.

2D array of rows and columns

For a 2D array of rows and columns, write it as a 1D array in a 2D style like this:

# 2D array of 4 rows and 3 columns
ARRAY2D_NUM_COLS="3"
array2d=(
    # row,col
    "0,0" "0,1" "0,2"
    "1,0" "1,1" "1,2"
    "2,0" "2,1" "2,2"
    "3,0" "3,1" "3,2"
)

Then, to obtain the index of the 1D array at row n and column m, use this formula:

i=$(( $row*$num_cols + $col ))

Here that is in a function:

# Obtain an element from a 2D array given its row and column.
# Call format:
#   array2d_get_element <row> <col> <array2d>
# Example:
#   array2d_get_element 2 1 "${array2d[@]}"
array2d_get_element() {
    row="$1"
    col="$2"
    # slice off the first 2 args and store the rest as the array
    shift 2
    local array2d=("$@")
    
    num_cols="$(array2d_get_num_cols)"
    i=$(( $row*$num_cols + $col ))
    echo "${array2d[$i]}"
}

Here is a more concrete example of processing a 2D table of data in Bash:

MY_2D_TABLE_NUM_COLS="3"
my_2d_table=(
    # color     make            MPG
    # -----     -------         ----
    "blue"      "Tesla"         "45"
    "red"       "BMW"           "32"
    "green"     "Volkswagen"    "28"
    "yellow"    "Toyota"        "27"
)

# Some custom function you write to process a table of your custom data 
# in Bash.
# Call format:
#   process_my_2d_table "${my_2d_table[@]}"
process_my_2d_table() {
    echo "Processing my_2d_table:"
    local my_2d_table=("$@")
    
    num_elements="${#my_2d_table[@]}"
    num_rows="$(( $num_elements / $MY_2D_TABLE_NUM_COLS ))"
    num_cols="$MY_2D_TABLE_NUM_COLS"

    # Now process data on each row
    for ((row=0; row<num_rows; row++)); do
        i_color=$(( $row*$num_cols + 0 ))  # index of column 0
        i_make=$(( $row*$num_cols + 1 ))   # index of column 1
        i_mpg=$(( $row*$num_cols + 2 ))    # index of column 2

        color="${my_2d_table[$i_color]}"   # column 0
        make="${my_2d_table[$i_make]}"     # column 1
        mpg="${my_2d_table[$i_mpg]}"       # column 2

        # Now do whatever you want with these values! 
        # - You may need them in calculations or whatever. 
        # - For this demo, I'll just print them. 
        printf "%-15s %-20s %-10s\n" "color=$color," "make=$make," "mpg=$mpg"
    done
}

process_my_2d_table "${my_2d_table[@]}"

Example output:

Processing my_2d_table:
color=blue,     make=Tesla,          mpg=45    
color=red,      make=BMW,            mpg=32    
color=green,    make=Volkswagen,     mpg=28    
color=yellow,   make=Toyota,         mpg=27   

3D array of pages, rows, and columns

A 3D array of rows, columns, and pages follows the same pattern. Write it as a 1D array in a 3D style like this:

# 3D array of 3 pages of 4 rows and 3 columns each
ARRAY3D_NUM_COLS="3"
ARRAY3D_NUM_ROWS="4"
array3d=(
    # page,row,col

    # page 0
    "0,0,0" "0,0,1" "0,0,2"
    "0,1,0" "0,1,1" "0,1,2"
    "0,2,0" "0,2,1" "0,2,2"
    "0,3,0" "0,3,1" "0,3,2"

    # page 1
    "1,0,0" "1,0,1" "1,0,2"
    "1,1,0" "1,1,1" "1,1,2"
    "1,2,0" "1,2,1" "1,2,2"
    "1,3,0" "1,3,1" "1,3,2"
    
    # page 2
    "2,0,0" "2,0,1" "2,0,2"
    "2,1,0" "2,1,1" "2,1,2"
    "2,2,0" "2,2,1" "2,2,2"
    "2,3,0" "2,3,1" "2,3,2"
)

Then index into it with a page, row, and col like this:

i=$(( $page*$num_rows*$num_cols + $row*$num_cols + $col ))

Here that is in a function:

# Obtain an element from a 3D array given its page, row, and column.
# Call format:
#   array3d_get_element <page> <row> <col> <array3d>
# Example:
#   array3d_get_element 1 2 1 "${array3d[@]}"
array3d_get_element() {
    page="$1"
    row="$2"
    col="$3"
    # slice off the first 3 args and store the rest as the array
    shift 3
    local array3d=("$@")
    
    num_rows="$(array3d_get_num_rows)"
    num_cols="$(array3d_get_num_cols)"
    i=$(( $page*$num_rows*$num_cols + $row*$num_cols + $col ))
    echo "${array3d[$i]}"
}

The patterns can be extended for higher dimensions.

Full test program

Here is a full program to run and test:

array_of_arrays__2D_3D_multidimensional_arrays.sh from my eRCaGuy_hello_world repo:

#!/usr/bin/env bash

# 2D array of 4 rows and 3 columns
ARRAY2D_NUM_COLS="3"
array2d=(
    # row,col
    "0,0" "0,1" "0,2"
    "1,0" "1,1" "1,2"
    "2,0" "2,1" "2,2"
    "3,0" "3,1" "3,2"
)

# 3D array of 3 pages of 4 rows and 3 columns each
ARRAY3D_NUM_COLS="3"
ARRAY3D_NUM_ROWS="4"
array3d=(
    # page,row,col

    # page 0
    "0,0,0" "0,0,1" "0,0,2"
    "0,1,0" "0,1,1" "0,1,2"
    "0,2,0" "0,2,1" "0,2,2"
    "0,3,0" "0,3,1" "0,3,2"

    # page 1
    "1,0,0" "1,0,1" "1,0,2"
    "1,1,0" "1,1,1" "1,1,2"
    "1,2,0" "1,2,1" "1,2,2"
    "1,3,0" "1,3,1" "1,3,2"
    
    # page 2
    "2,0,0" "2,0,1" "2,0,2"
    "2,1,0" "2,1,1" "2,1,2"
    "2,2,0" "2,2,1" "2,2,2"
    "2,3,0" "2,3,1" "2,3,2"
)

array2d_get_num_cols() {
    echo "$ARRAY2D_NUM_COLS"
}

array2d_get_num_rows() {
    local array2d=("$@")
    num_elements="${#array2d[@]}"
    # echo "num_elements = $num_elements"  # debugging
    num_rows=$(( $num_elements / $(array2d_get_num_cols) ))
    echo "$num_rows"
}

array3d_get_num_cols() {
    echo "$ARRAY3D_NUM_COLS"
}

array3d_get_num_rows() {
    echo "$ARRAY3D_NUM_ROWS"
}

array3d_get_num_pages() {
    local array3d=("$@")
    num_elements="${#array3d[@]}"
    num_rows="$(array3d_get_num_rows)"
    num_cols="$(array3d_get_num_cols)"
    num_pages=$(( $num_elements / ($num_rows * $num_cols) ))
    echo "$num_pages"
}

array2d_print() {
    local array2d=("$@")
    num_rows="$(array2d_get_num_rows "${array2d[@]}")"
    num_cols="$(array2d_get_num_cols)"

    for ((row=0; row<num_rows; row++)); do
        for ((col=0; col<num_cols; col++)); do
            i=$(( $row*$num_cols + $col ))
            echo -n "${array2d[$i]} "
        done
        echo "" # new line after each row
    done
}

array3d_print() {
    local array3d=("$@")
    num_rows="$(array3d_get_num_rows)"
    num_cols="$(array3d_get_num_cols)"
    num_pages="$(array3d_get_num_pages "${array3d[@]}")"
    
    for ((page=0; page<num_pages; page++)); do
        echo "Page $page:"

        for ((row=0; row<num_rows; row++)); do
            for ((col=0; col<num_cols; col++)); do
                i=$(( $page*$num_rows*$num_cols + $row*$num_cols + $col ))
                echo -n "${array3d[$i]} "
            done
            echo "" # new line after each row
        done
        echo "" # extra new line after each page
    done
}

# Obtain an element from a 2D array given its row and column.
# Call format:
#   array2d_get_element <row> <col> <array2d>
# Example:
#   array2d_get_element 2 1 "${array2d[@]}"
array2d_get_element() {
    row="$1"
    col="$2"
    # slice off the first 2 args and store the rest as the array
    shift 2
    local array2d=("$@")
    
    num_cols="$(array2d_get_num_cols)"
    i=$(( $row*$num_cols + $col ))
    echo "${array2d[$i]}"
}

# Obtain an element from a 3D array given its page, row, and column.
# Call format:
#   array3d_get_element <page> <row> <col> <array3d>
# Example:
#   array3d_get_element 1 2 1 "${array3d[@]}"
array3d_get_element() {
    page="$1"
    row="$2"
    col="$3"
    # slice off the first 3 args and store the rest as the array
    shift 3
    local array3d=("$@")
    
    num_rows="$(array3d_get_num_rows)"
    num_cols="$(array3d_get_num_cols)"
    i=$(( $page*$num_rows*$num_cols + $row*$num_cols + $col ))
    echo "${array3d[$i]}"
}

MY_2D_TABLE_NUM_COLS="3"
my_2d_table=(
    # color     make            MPG
    # -----     -------         ----
    "blue"      "Tesla"         "45"
    "red"       "BMW"           "32"
    "green"     "Volkswagen"    "28"
    "yellow"    "Toyota"        "27"
)

# Some custom function you write to process a table of your custom data in Bash.
# Call format:
#   process_my_2d_table "${my_2d_table[@]}"
process_my_2d_table() {
    echo "Processing my_2d_table:"
    local my_2d_table=("$@")
    
    num_elements="${#my_2d_table[@]}"
    num_rows="$(( $num_elements / $MY_2D_TABLE_NUM_COLS ))"
    num_cols="$MY_2D_TABLE_NUM_COLS"

    # Now process data on each row
    for ((row=0; row<num_rows; row++)); do
        i_color=$(( $row*$num_cols + 0 ))
        i_make=$(( $row*$num_cols + 1 ))
        i_mpg=$(( $row*$num_cols + 2 ))

        color="${my_2d_table[$i_color]}"
        make="${my_2d_table[$i_make]}"
        mpg="${my_2d_table[$i_mpg]}"

        # Now do whatever you want with these values! 
        # - You may need them in calculations or whatever. 
        # - For this demo, I'll just print them. 
        printf "%-15s %-20s %-10s\n" "color=$color," "make=$make," "mpg=$mpg"
    done
}

run_tests()
{
    echo ""
    echo "Running tests:"
    echo ""

    echo "array2d has:"
    echo "  ${#array2d[@]} elements"
    echo "  $(array2d_get_num_rows "${array2d[@]}") rows"
    echo "  $(array2d_get_num_cols) columns"
    echo ""
    echo "array2d_print:"
    array2d_print "${array2d[@]}"
    echo ""
    echo "Element (row,col) (2,1): $(array2d_get_element 2 1 ${array2d[@]})"
    echo ""

    echo "array3d has:"
    echo "  ${#array3d[@]} elements"
    echo "  $(array3d_get_num_rows) rows"
    echo "  $(array3d_get_num_cols) columns"
    echo "  $(array3d_get_num_pages "${array3d[@]}") pages"
    echo ""
    echo "array3d_print:"
    array3d_print "${array3d[@]}"
    echo ""
    echo "Element (page,row,col) (1,2,1): $(array3d_get_element 1 2 1 ${array3d[@]})"
    echo ""

    process_my_2d_table "${my_2d_table[@]}"
    echo ""
}

main() {
    run_tests
}

# Determine if the script is being sourced or executed (run).
# See:
# 1. "eRCaGuy_hello_world/bash/if__name__==__main___check_if_sourced_or_executed_best.sh"
# 1. My answer: https://stackoverflow.com/a/70662116/4561887
if [ "${BASH_SOURCE[0]}" = "$0" ]; then
    # This script is being run.
    __name__="__main__"
else
    # This script is being sourced.
    __name__="__source__"
fi

# Only run `main` if this script is being **run**, NOT sourced (imported).
# - See my answer: https://stackoverflow.com/a/70662116/4561887
if [ "$__name__" = "__main__" ]; then
    main "$@"
fi

You can import this file as a Bash library and use it in your own scripts, like this:

. array_of_arrays__2D_3D_multidimensional_arrays.sh

run_tests

See more on that on my personal website here: How do you write, import, use, and test libraries in Bash?

Or, just run the script above. Here's the run command and output:

eRCaGuy_hello_world$ bash/array_of_arrays__2D_3D_multidimensional_arrays.sh 

Running tests:

array2d has:
  12 elements
  4 rows
  3 columns

array2d_print:
0,0 0,1 0,2 
1,0 1,1 1,2 
2,0 2,1 2,2 
3,0 3,1 3,2 

Element (row,col) (2,1): 2,1

array3d has:
  36 elements
  4 rows
  3 columns
  3 pages

array3d_print:
Page 0:
0,0,0 0,0,1 0,0,2 
0,1,0 0,1,1 0,1,2 
0,2,0 0,2,1 0,2,2 
0,3,0 0,3,1 0,3,2 

Page 1:
1,0,0 1,0,1 1,0,2 
1,1,0 1,1,1 1,1,2 
1,2,0 1,2,1 1,2,2 
1,3,0 1,3,1 1,3,2 

Page 2:
2,0,0 2,0,1 2,0,2 
2,1,0 2,1,1 2,1,2 
2,2,0 2,2,1 2,2,2 
2,3,0 2,3,1 2,3,2 


Element (page,row,col) (1,2,1): 1,2,1

Processing my_2d_table:
color=blue,     make=Tesla,          mpg=45    
color=red,      make=BMW,            mpg=32    
color=green,    make=Volkswagen,     mpg=28    
color=yellow,   make=Toyota,         mpg=27    

References

1. My answer on indexing into 1D arrays as though they were 2D arrays, in C: Example 3 in: How to use multidimensional (ex: 2D) arrays, and pointers to them, as function parameters in C and C++
2. My answer: Math in bash: how to use all bash operators (+, -, /, *, **, &, &&, <<, etc.), and arithmetic expansion, in bash
3. My answer on printing and passing Bash arrays by value or reference: Variable for array name within curly braces (passing or printing an array in bash)
4. My answer: Importing functions from a shell script

Answer 6

A bash array of arrays is possible, if you convert and store each array as a string using declare -p (see my function stringify). This will properly handle spaces and any other problem characters in your arrays. When you extract an array, use function unstringify to rebuild the array. This script demonstrates an array of arrays:

#!/bin/bash 
# BASH array of arrays demo

# Convert an array to a string that can be used to reform 
# the array as a new variable. This allows functions to
# return arrays as strings. Works for arrays and associative
# arrays. Spaces and odd characters are all handled by bash 
# declare.
# Usage: stringify variableName
#     variableName - Name of the array variable e.g. "myArray",
#          NOT the array contents.
# Returns (prints) the stringified version of the array.
# Examples. Use declare to make an array:
#     declare -a myArray=( "O'Neal, Dan" "Kim, Mary Ann" )
# (Or to make a local variable replace declare with local.)
# Stringify myArray:
#     stringifiedArray="$(stringify myArray)"
# Reform the array with any name like reformedArray:
#     eval "$(unstringify reformedArray "$stringifiedArray")"
# To stringify an argument list "$@", first create the array
# with a name:     declare -a myArgs=( "$@" )
stringify() {
    declare -p $1
}

# Reform an array from a stringified array. Actually this prints
# the declare command to form the new array. You need to call 
# eval with the result to make the array.
# Usage: eval "$(unstringify newArrayName stringifiedArray [local])"
#     Adding the optional "local" will create a local variable 
#     (uses local instead of declare).
# Example to make array variable named reformedArray from 
# stringifiedArray:
#     eval "$(unstringify reformedArray "$stringifiedArray")"
unstringify() {
    local cmd="declare"
    [ -n "$3" ] && cmd="$3"
    # This RE pattern extracts 2 things:
    #     1: the array type, should be "-a" or "-A"
    #     2: stringified contents of the array 
    # and skips "declare" and the original variable name.
    local declareRE='^declare ([^ ]+) [^=]+=(.*)$'
    if [[ "$2" =~ $declareRE ]]
    then
        printf '%s %s %s=%s\n' "$cmd" "${BASH_REMATCH[1]}" "$1" "${BASH_REMATCH[2]}"
    else
        echo "*** unstringify failed, invalid stringified array:" 1>&2
        printf '%s\n' "$2" 1>&2
        return 1
    fi
}

# array of arrays demo
declare -a array # the array holding the arrays
declare -a row1=( "this is" "row 1" )
declare -a row2=( "row 2" "has problem chars" '!@#$%^*(*()-_=+[{]}"|\:;,.<.>?/' )
declare -a row3=( "$@" ) # row3 is the arguments to the script

# Fill the array with each row converted to a string.
# stringify needs the NAME OF THE VARIABLE, not the variable itself
array[0]="$(stringify row1)"
array[1]="$(stringify row2)"
array[2]="$(stringify row3)"

# Print array contents
for row in "${array[@]}"
do
    echo "Expanding stringified row: $row"
    # Reform the row as the array thisRow
    eval "$(unstringify thisRow "$row")"
    echo "Row values:"
    for val in "${thisRow[@]}"
    do
        echo "   '$val'"
    done
done

Answer 7

I use Associative Arrays and use :: in the key to denote depth. The :: can also be used to embed attributes, but that is another subject,...

declare -A __myArrayOfArray=([Array1::Var1]="Assignment" [Array2::Var1]="Assignment")

An Array under Array1

__myArrayOfArray[Array1::SubArray1::Var1]="Assignment"

The entries in any array can be retrieved (in order ...) by ...

local __sortedKeys=`echo ${!__myArrayOfArray[@]} | xargs -n1 | sort -u | xargs`
for __key in ${__sortedKeys}; do
    #
    # show all properties in the Subordinate Profile "Array1::SubArray1::"
    if [[ ${__key} =~ ^Array1::SubArray1:: ]]; then
        __property=${__key##Array1::SubArray1::}
        if [[ ${__property} =~ :: ]]; then
            echo "Property ${__property%%:*} is a Subordinate array"
        else
            echo "Property ${__property} is set to: ${__myArrayOfArray[${__key}]}"
        fi
    fi 
done

THE list of subordinate "Profiles" can be derived by:

declare -A __subordinateProfiles=()
local __profile
local __key
for __key in "${!__myArrayOfArray[@]}"; do
    if [[ $__key =~ :: ]]; then
        local __property=${__key##*:}
        __profile=${__key%%:*}
        __subordinateProfiles[${__profile}]=1
    fi   
done

Answer 8

You could make use of (de)referencing arrays like in this script:

#!/bin/bash

OFS=$IFS     # store field separator
IFS="${2: }" # define field separator
file=$1      # input file name

unset a      # reference to line array
unset i j    # index
unset m n    # dimension

### input

i=0
while read line
do
  a=A$i
  unset $a
  declare -a $a='($line)'
  i=$((i+1))
done < $file
# store number of lines
m=$i

### output

for ((i=0; i < $m; i++))
do
  a=A$i
  # get line size
  # double escape '\\' for sub shell '``' and 'echo'
  n=`eval echo \\${#$a[@]}`
  for (( j = 0; j < $n; j++))
  do
    # get field value
    f=`eval echo \\${$a[$j]}`
    # do something
    echo "line $((i+1)) field $((j+1)) = '$f'"
  done
done

IFS=$OFS

Credit to https://unix.stackexchange.com/questions/199348/dynamically-create-array-in-bash-with-variables-as-array-name