Assign variable in gnu parallel conditionally

Question

my code is:

echo 'url' | parallel "if curl -ss "$url" | grep yes; then vara+="$(echo {})"; else varb+="$(echo {})"; fi"

But variables are not being set. If then statement works perfectly eg . if curl $url | grep yes; then echo done; else echo undone; fi ---> such thing works but variables are not being set

Can anyone help me find my mistake.

Desired Output: is if if curl url and grep certain word in source then that url will be appended to a variable using += and if word not found then it will be apended in another variable . Finally after script is over i want to have to variables: vara contains all url list having specific word and varb contains list of urls not having that word in source. that variables are furthet used in scripts

Answer 1

TLDR; Accummulate all results in a single array, in the calling process, and sort them out afterwards.

For each URL, output either SUCCESS:URL or FAIL:URL

Accumulate that into a single array:

Results=( $(parallel 'status="FAIL"; curl something && status="SUCCESS"; printf "$status:%s" {}') )

which will give Results like:

SUCCESS:www.ibm.com
SUCCESS:www.google.com
FAIL:www.chocolate-teapots.com

Now select/sort the elements using this answer.

Agreed that sorting afterwards is not exactly what you were looking for, but I assume the reason for wanting to do it in parallel was the latency of curl which is still being avoided, and selecting/sorting even thousands of array elements will only take milliseconds.

---

Explanation

Your script is setting array elements, but within the new process that GNU Parallel starts for each job... and which exits when that job is complete.

---

You could use GNU Parallel parset like this to fill an array:

parset Results 'sleep 3; seq {}' ::: {1..10}
echo "${Results[@]}"

And you'd set it up with:

env_parallel --install
<<start a new shell>>

but that wouldn't be my choice.