xargs does not quit on error

Question

I have a bash script where I use find to get a bunch of files in a directory, from which I then use xargs to execute said files in a chroot environment 1 script at a time. My understanding has been that xargs quits and stop processing once it sees a non-zero exit code, however, for some reason this does not seem to be the case.

The script I have:

#!/usr/bin/env bash

set -euo pipefail

script_dir=$( cd "$( dirname "${BASH_SOURCE[0]}" )/.." && pwd )

rootfs="$1"

exec_script() {
  script="$1"

  relative_script_path="$(realpath --relative-to="$script_dir" "$script")"

  echo -e "\e[1;94m=> executing script $script ($relative_script_path)\e[0m"

  sleep 5

  if ! "$rootfs/enter-chroot" sh -c "/$relative_script_path"; then
    echo -e "\e[1;31m=> script $script failed\e[0m"
    exit 1
  fi

  echo -e "\e[1;32m=> script $script ran successfully\e[0m"
}

export -f exec_script
export rootfs
export script_dir

find "$script_dir/build/scripts" -name '*.sh' -print0 | sort -z | xargs -r -0 -I% -n1 bash -c 'exec_script "$@"' _ %

And when I run it, I get the following output:

./build/run.sh /tmp/test
=> executing script /tmp/builder/build/scripts/000-upgrade.sh (build/scripts/000-upgrade.sh)
environment: line 4: /tmp/test/enter-chroot: Not a directory
=> script /tmp/builder/build/scripts/000-upgrade.sh failed
=> executing script /tmp/builder/build/scripts/001-firmware.sh (build/scripts/001-firmware.sh)
environment: line 4: /tmp/test/enter-chroot: Not a directory
=> script /tmp/builder/build/scripts/001-firmware.sh failed

Where am I going wrong? How can I ensure that xargs stops processing and exits with a non-zero exit code?

Chris Davies · Accepted Answer · 2020-02-11 08:15:30Z

2

The documentation for xargs (see man xargs) actually says this about exiting,

If any invocation of the command exits with a status of 255, xargs will stop immediately without reading any further input. An error message is issued on stderr when this happens.

One possible solution would, therefore, be to change exec_script to return exit status 255 on error.

Another possible solution, in the case that exec_script cannot be changed, would be to turn the plain xargs into a shell loop:

find "$script_dir/build/scripts" -name '*.sh' -print0 |
    sort -z |
    while IFS= read -r -d '' item && exec_script _ "$item"; do :; done

Here the loop will break if exec_script returns any non-zero exit value.

Yet another solution, this one from the comments and which arguably is the simplest external fix, is to catch any exit error from your script and replace it with 255:

find "$script_dir/build/scripts" -name '*.sh' -print0 |
    sort -z |
    xargs -r -0 -I% -n1 bash -c 'exec_script "$@" || exit 255' _ %

edited Feb 11, 2020 at 8:15

answered Feb 10, 2020 at 21:33

Chris Davies

128k16 gold badges178 silver badges323 bronze badges

1

Annoyingly I read the man pages, but only looked at the flags to see if an option existed and after some googling, I was convinced that it quits after the first error. Thank you!

Hosh Sadiq
– Hosh Sadiq

2020-02-10 21:37:22 +00:00
Commented Feb 10, 2020 at 21:37
1

TBH when I saw your question it surprised me that xargs quit at all. I've learned something new today too, thank you.

Chris Davies
– Chris Davies

2020-02-10 21:50:38 +00:00
Commented Feb 10, 2020 at 21:50
3

@HoshSadiq instead of that shell loop, you can do everything much simpler with .. | xargs -0 sh -c 'exec_script "$@" || exit 255' sh (adjust for the other options you're passing to xargs and your script).

user313992
– user313992

2020-02-10 23:08:53 +00:00
Commented Feb 10, 2020 at 23:08
Also, do the files really need to be sorted first? If not, you can get rid of xargs and use find ... $ -exec script {} \; -o -quit $ instead.

user313992
– user313992

2020-02-10 23:11:21 +00:00
Commented Feb 10, 2020 at 23:11
@mosvy actually I much prefer the loop, originally I didn't use it because the script was meant to run on a posix shell, meaning -d wasn't available for read, later I refactored it so it runs through the host's bash and only executes the scripts in the chroot, but I guess I didn't think to go back to looping. In addition, using the loop doesn't require exporting the variables and/or functions. And yes, it's a simple way of creating script execution order as there's some dependencies and for some reason find does not sort them correctly though I've prefixed them with numbers.

Hosh Sadiq
– Hosh Sadiq

2020-02-11 08:08:55 +00:00
Commented Feb 11, 2020 at 8:08

| Show 2 more comments

user · Accepted Answer · 2021-11-20 00:01:37Z

xargs is quite troublesome because it only stops when exiting with 255, then, using set -eo pipefail will be completely void when exporting a function, unless you use a subshell to encapsulate the set -eo pipefail failure and propagate it with 255.

On this example, comment out the errornow line to see the difference:

#!/bin/bash
set -x
set -eo pipefail

parallel_uploads="4"
s3_bucket_name="backup"

all_files=(
"/12.pbd"
"/13.pbd"
"/14.pbd"
"/15.pbd"
"/16.pbd"
"/17.pbd"
"/18.pbd"
"/19.pbd"
"/20.pbd"
"/21.pbd"
"/22.pbd"
"/23.pbd"
"/24.pbd"
"/25.pbd"
"/26.pbd"
"/27.pbd"
)


# Workaround for the posix shell bug they call it feature
# https://unix.stackexchange.com/questions/65532/why-does-set-e-not-work-inside-subshells-with-parenthesis-followed-by-an-or
function acually_upload_to_s3()
{
    set -x;
    set -eu -o pipefail;

    printf 'Doing some\n';
    sleeptime="$(( RANDOM % 50 + 1 ))"
    sleep "$sleeptime";

    erroring_some;
    printf 'Doing more some\n';
}

function upload_to_s3()
{
    set -x;
    set -eu -o pipefail;
    # https://superuser.com/questions/403263/how-to-pass-bash-script-arguments-to-a-subshell
    /bin/bash -c "acually_upload_to_s3 $(printf "${1+ %q}" "$@")" || exit 255
}

function upload_all()
{
    export s3_bucket_name;
    export -f upload_to_s3;
    export -f acually_upload_to_s3;

    # https://unix.stackexchange.com/questions/566834/xargs-does-not-quit-on-error
    # https://stackoverflow.com/questions/11003418/calling-shell-functions-with-xargs
    # https://stackoverflow.com/questions/6441509/how-to-write-a-process-pool-bash-shell
    # https://stackoverflow.com/questions/356100/how-to-wait-in-bash-for-several-subprocesses-to-finish-and-return-exit-code-0
    printf "'%s'\n" "${all_files[@]}" | xargs \
            --max-procs="$parallel_uploads" \
            --max-args=1 \
            --replace={} \
            /bin/bash -c 'time upload_to_s3 "$s3_bucket_name" "{}"';
}

time upload_all \
    && printf '%s Successfully uploaded all files\n' "$(date)" \
    || printf '%s Error: Could not upload some files\n'  "$(date)";

Example output:

$ bash upload_to_s3_glacier_deep.sh
+ set -eo pipefail
+ parallel_uploads=4
+ s3_bucket_name=backup
+ all_files=("/12.pbd" "/13.pbd" "/14.pbd" "/15.pbd" "/16.pbd" "/17.pbd" "/18.pbd" "/19.pbd" "/20.pbd" "/21.pbd" "/22.pbd" "/23.pbd" "/24.pbd" "/25.pbd" "/26.pbd" "/27.pbd")
+ upload_all
+ export s3_bucket_name
+ export -f upload_to_s3
+ export -f acually_upload_to_s3
+ printf ''\''%s'\''\n' /12.pbd /13.pbd /14.pbd /15.pbd /16.pbd /17.pbd /18.pbd /19.pbd /20.pbd /21.pbd /22.pbd /23.pbd /24.pbd /25.pbd /26.pbd /27.pbd
+ xargs --max-procs=4 --max-args=1 '--replace={}' /bin/bash -c 'time upload_to_s3 "$s3_bucket_name" "{}"'
+ set -eu -o pipefail
++ printf ' %q' backup /12.pbd
+ /bin/bash -c 'acually_upload_to_s3  backup /12.pbd'
+ set -eu -o pipefail
++ printf ' %q' backup /13.pbd
+ /bin/bash -c 'acually_upload_to_s3  backup /13.pbd'
+ set -eu -o pipefail
++ printf ' %q' backup /14.pbd
+ /bin/bash -c 'acually_upload_to_s3  backup /14.pbd'
+ set -eu -o pipefail
+ printf 'Doing some\n'
Doing some
+ sleeptime=3
+ sleep 3
+ set -eu -o pipefail
++ printf ' %q' backup /15.pbd
+ /bin/bash -c 'acually_upload_to_s3  backup /15.pbd'
+ set -eu -o pipefail
+ printf 'Doing some\n'
Doing some
+ sleeptime=49
+ sleep 49
+ set -eu -o pipefail
+ printf 'Doing some\n'
Doing some
+ sleeptime=13
+ sleep 13
+ set -eu -o pipefail
+ printf 'Doing some\n'
Doing some
+ sleeptime=30
+ sleep 30
+ erroring_some
environment: line 5: erroring_some: command not found
+ exit 255

real    0m3.146s
user    0m0.045s
sys 0m0.123s
xargs: /bin/bash: exited with status 255; aborting
+ erroring_some
environment: line 5: erroring_some: command not found
+ exit 255

real    0m13.149s
user    0m0.015s
sys 0m0.123s
xargs: /bin/bash: exited with status 255; aborting

real    0m13.271s
user    0m0.075s
sys 0m0.337s
++ date
+ printf '%s Error: Could not upload some files\n' 'Fri, Nov 19, 2021 22:00:30'
Fri, Nov 19, 2021 22:00:30 Error: Could not upload some files

Stack Exchange Network

xargs does not quit on error

2 Answers 2

You must log in to answer this question.

Linked

Hot Network Questions

xargs does not quit on error

2 Answers 2

You must log in to answer this question.

Linked

Related

Hot Network Questions