Associative Array contains filenames and paths

Question

I have a pipeline that deploys scripts into a temporary dir. I need to rellocate them into their actual proper directory.

#!/usr/bin/sh

DEPLOY_HOME=/opt/xxx

function getDeployedFiles {
    ls ${DEPLOY_HOME} | grep -v pipeline.properties | grep -v Deploy.tmp > 
    ${DEPLOY_HOME}/Deploy.tmp
    chmod 700 ${DEPLOY_HOME}/Deploy.tmp
    readarray -t depFilesArr < ${DEPLOY_HOME}/Deploy.tmp
}

function setDict {
    declare -A dict
    dict=(
        ["a.script.100"]="/opt/xxx/firstpath/100"
        ["b.script.200"]="/opt/xxx/secondpath/200"
        ["c.script.300"]="/opt/xxx/thirdpath/300"
    )
}

function itArr {
    for i in "${depFilesArr[@]}"; do
        FILENAME="${i}"
        if [[ -v "${dict['${i}']}" ]]; then
            echo "Value is present"
            FILEDEST="${dict['${i}']}"
            mv ${DEPLOY_HOME}/${FILENAME} ${FILEDEST}/${FILENAME}
        else
            echo "Value not found, file will remain in ${DEPLOY_HOME}"
        fi
    done
}

#MAIN PROGRAM
getDeployedFiles
setDict 
itArr 

The current error I am getting is: "script.sh: line x 'a.script.100': syntax error: operand expected (error token is "'a.script.100'")

It doesn't seem to evaluate the "if" properly as it's not printing any of the subsequent echos.

Idk wether I'm incorrectly managing the strings or something about my logic is wrong, any help appreciated. BTW I'm using bash 4.4.

Answer 1

#!/usr/bin/sh

DEPLOY_HOME=/opt/xxx

function getDeployedFiles {

That's the ksh function definition syntax. In sh, the syntax is:

getDeployedFiles() compound-command

Where command group ({ ...; }) is the type of compound command most commonly used as function bodies.

    ls ${DEPLOY_HOME} | grep -v pipeline.properties | grep -v Deploy.tmp > 
    ${DEPLOY_HOME}/Deploy.tmp

Target of redirections cannot be put on a separate line, the target file has to follow the >, with only space and tab characters¹ allowed in between.

In sh (or ksh/bash/yash...), leaving a parameter expansion unquoted in list context like that ${DEPLOY_HOME} has a very special meaning², not something you'd want to do.

The re in grep stands for "regular expression", pipeline.properties as a regexp would match on a string that contains pipeline followed by any single character (the . regexp operator) followed by properties.

If you wanted to filter out the lines that are pipeline.properies or Deploy.tmp, you'd use grep -xF -e pipeline.propeties -e Deploy.tmp, where -F is for Fixed string (as opposed to regexp matching) and -x for exact (as opposed to substring) match, but in any case, using line-based text utilities in general is wrong to deal with file paths as file paths can be any sequence of non-nul bytes that don't have to be text and if text, don't have to be single line.

You're also missing a -- to mark the end of options to ls. You'd probably want to add a -H or -L option to ls to allow $DEPLOY_HOME to be a symlink.

    chmod 700 ${DEPLOY_HOME}/Deploy.tmp

You're removing the read permissions after the file has been created and contents written there, so that leaves a window during which that contents will be readable.

Best would be to set the umask to 077 (or 066) before creating the file to make sure it's created with the right permissions in the first place.

    readarray -t depFilesArr < ${DEPLOY_HOME}/Deploy.tmp

readarray is a builtin of the bash shell and of that shell only. There is no array in the standard sh language, let alone a readarray command.

If you want to use arrays, you'd need to use a shell that supports them such as zsh, yash, ksh or bash, and set the shebang accordingly.

Here zsh would probably be the best choice as that whole code could be written as:

set -o extendedglob
depFilesArr=( $DEPLOY_HOME/^pipeline.properties(N:t) )

In bash:

shopt -s nullglob extglob
depFilesArr=( "$DEPLOY_HOME"/!(pipeline.properties) )
depFilesArr=( "${depFilesArr[@]##*/}" )

function setDict {
    declare -A dict

Same thing. declare is bash-specific (bash copied ksh but for some reason changed typeset to declare). bash's associative array design (added relatively recently there) is also shaped after that of ksh93 instead of that of zsh.

    dict=(
        ["a.script.100"]="/opt/xxx/firstpath/100"
        ["b.script.200"]="/opt/xxx/secondpath/200"
        ["c.script.300"]="/opt/xxx/thirdpath/300"
    )
}

function itArr {
    for i in "${depFilesArr[@]}"; do
        FILENAME="${i}"

Why not for FILENAME in ... then? All uppercase variables should be reserved for environment variables or at least variables that have a global scope.

        if [[ -v "${dict['${i}']}" ]]; then

Another bashism (copied from ksh). The syntax is incorrect though. It's [[ -v var ]] to check whether a variable is set. For associative array elements, it's rather broken by design as [[ -v "dict[$key]" ]] tends to break if $key is empty or contains ] or \ or is * or @ and the attempts to fix that have resulted in incompatibilities between versions.

You might as well use the standard [ -n "${var+set}" ] syntax:

if [ -n "${dict[$i]+set}" ]

In zsh:

if (( $+dict[$i] ))

            echo "Value is present"
            FILEDEST="${dict['${i}']}"

Those single quotes don't make sense.

            mv ${DEPLOY_HOME}/${FILENAME} ${FILEDEST}/${FILENAME}

Again missing quotes (if using ksh/bash) and --.

        else
            echo "Value not found, file will remain in ${DEPLOY_HOME}"

User messages in general are best sent to stderr, stdout being kept for the output that the script produces (none here though).

        fi
    done
}

#MAIN PROGRAM
getDeployedFiles
setDict 
itArr

So:

#! /usr/bin/zsh -

set -o extendedglob
DEPLOY_HOME=/opt/xxx

typeset -A dict=(
  a.script.100 /opt/xxx/firstpath/100
  b.script.200 /opt/xxx/secondpath/200
  c.script.300 /opt/xxx/thirdpath/300
)

msg() print -ru2 -- "$@"

ret=0
for file in $DEPLOY_HOME/^pipeline.properties(N); do
  if (( $+dict[$file:t] )); then
    msg 'Value is present'
    mv -- $file $dict[$file:t]/ || ret=$?
  else
    msg "Value not found, file will remain in ${DEPLOY_HOME}"
  fi
done
exit $ret

Or if you had to use bash (you still find some systems where zsh is not installed by default but bash is (such as GNU systems as bash is the GNU implementation of sh even though several GNU-based systems have switched to some leaner/faster shells for their sh)):

#! /usr/bin/bash -

shopt -s nullglob extglob
DEPLOY_HOME=/opt/xxx

typeset -A dict=(
  [a.script.100]=/opt/xxx/firstpath/100
  [b.script.200]=/opt/xxx/secondpath/200
  [c.script.300]=/opt/xxx/thirdpath/300
)

msg() {
  local IFS=' '
  printf>&2 '%s\n' "$*"
}

ret=0
for file in "$DEPLOY_HOME"/!(pipeline.properties); do
  tail=${file##*/}
  if [ -n "${dict[$tail]+set}" ]; then
    msg 'Value is present'
    mv -- "$file" "${dict[$tail]}"/ || ret=$?
  else
    msg "Value not found, file will remain in ${DEPLOY_HOME}"
  fi
done
exit "$ret"

---

^{¹ Some shells also allow other characters classified as blank in the user's locale; in the case of bash, that's only the single-byte ones (so in practice only sometimes the non-breaking space character in locales where it's encoded on one byte (often 0xA0) and on systems when it's classified as blank such as some BSDs).}

^{² the target of redirections normally doesn't constitute a list context, but in ksh88 and bash (unless both non-interactive and invoked as sh for bash), you still need the quotes there, as the shell performs pathname expansion.}