Best practices for SLURM job pipeline with wrapper scripts - avoiding complex job ID extraction

Question

I'm building a SLURM pipeline where each stage is a bash wrapper script that generates and submits SLURM jobs. Currently I'm doing complex job ID extraction which feels clunky:

# Current approach
job1_id=$(bash stage1_wrapper.sh params | grep "^JobID:" | cut -d ":" -f 2)
job2_id=$(bash stage2_wrapper.sh params $job1_id | grep "^JobID:" | cut -d ":" -f 2)
job3_id=$(bash stage3_wrapper.sh params $job2_id | grep "^JobID:" | cut -d ":" -f 2)

Each wrapper script does something like:

# Inside wrapper script
cat > generated_job.slurm << EOF
#!/bin/bash
#SBATCH --job-name=my-job
#SBATCH --time=10:00
python my_analysis.py $params
EOF

job_id=$(sbatch --parsable generated_job.slurm)
echo "JobID:$job_id"  # Plus other output

Is there a cleaner way to extract job dependence id, rather than repeating a manual grep-cut operation 3 times?

I disagree with the reason to close since OP thinks the code is inefficient and it is indeed. It's calling grep and cut 3 times which causes unnecessary forks given that Bash provides builtin options to do that in a much better way. — LMC
– LMC, Commented Sep 24 at 16:18
@damienfrancois Thank you but I already use --parsable. Please have a look at my last code block. — desert_ranger
– desert_ranger, Commented Sep 26 at 14:36
Sorry I overlooked that. I do not understand how you are using the jobID afterwards. I see no --dependency ? — damienfrancois
– damienfrancois, Commented Sep 26 at 15:10

LMC · Accepted Answer · 2025-09-24 16:32:49Z

1

Put wrappper output lines into an array and read ids from there

#                     wrapper command here
readarray -t out < <(printf '%s\n' 'JobId:123' 'other:cde')
printf "%s\n" "${out[0]}"
# this is the id
echo "${out[0]##JobId:}"

jobId:123
123

Also calling bash inside a $() construct is probably unnecessary

$(./stage1_wrapper.sh $params)

or

$(source stage1_wrapper.sh $params)

edited Sep 24 at 16:32

answered Sep 23 at 20:52

LMC

14.4k3 gold badges34 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Best practices for SLURM job pipeline with wrapper scripts - avoiding complex job ID extraction

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related