Linked Questions
14 questions linked to/from Parallelize a Bash FOR Loop
0
votes
1
answer
2k
views
Multi-threading with for loop [duplicate]
This is part of a bash script code I wish to use multi-threading
for DATABASE in $DATABASES ; do
echo Converting $DATABASE
# Check if the table is MyISAM (we don't want to convert InnoDB ...
- 2,177
50
votes
1
answer
26k
views
GNU parallel vs & (I mean background) vs xargs -P
I'm confused about the difference or advantage (if any) of running a set of tasks in a .sh script using GNU parallel
E.g. Ole Tange's answer:
parallel ./pngout -s0 {} R{} ::: *.png
rather than say ...
5
votes
7
answers
3k
views
Alternative to wait -n (because server has old version of bash)
I would like to solve the following issue about submitting a job that has been parallelised to a specific node.
Let me start with explaining the structure of my problem
I have two very simple Matlab ...
- 125
2
votes
2
answers
2k
views
FIFO-based semaphore explanation
I am trying to work on some parallelization of many processes (task send/to be executed on many (let's say hundreds) nodes).
I came across this solution:
https://unix.stackexchange.com/a/216475
# ...
3
votes
2
answers
239
views
Concurrency of a find, hash val, and replace across large amount of rows
I have a bunch of files and for each row there is a unique value I'm trying to obscure with a hash.
However there are 3M rows across the files and a rough calculation of the time needed to complete ...
- 155
0
votes
1
answer
2k
views
Sed command for performance improvement or tr command for the same
I have a command to replace the non printable characters and single quotes from a file but its taking more time to execute as I am replacing these characters for multiple files and the files size is ...
- 11
1
vote
0
answers
987
views
tar takes a long time before passing data to gzip
What I want to know, is what is tar doing at the start, before it starts passing data on to gzip? Can I make it skip that step?
I'm writing a script to run on my Synology NAS box (running DSM 6.2.1-...
- 744
1
vote
1
answer
581
views
GNU parallel is executing all commands at once within my function
Ok so i have a bash function that I apply on several folders:
function task(){
do_thing1
do_thing2
do_thing3
...
}
I want to run that function in parallel. So far I was using a little fork trick:
N=4 ...
- 11
0
votes
1
answer
633
views
Adding 'Progress bar / counter' to a parallelised For Loop
I've been greatly inspired by this question: Parallelize a Bash FOR Loop to parallelise some tools I've written that involve very loooong while read loops (ie doing the same task / set of tasks ...
user279851
2
votes
1
answer
391
views
parallelize shell script
I have a following line:
for length in "$(ls $OUT_SAMPLE)"
do $CODES/transform_into_line2.rb -c $OUT_SAMPLE -p 0 -f $length &
done
So,it should parallelize the for loop but somehow it still runs ...
- 315
0
votes
3
answers
159
views
Problem in writing a parallel version of my Bash code
I am trying to parallelise my sample Bash script and have tried commands like & and wait. Please let me know what is an effective way to make it parallel
My current code is working fine for ...
- 11
0
votes
0
answers
468
views
Limit background number of jobs in for loop
I have a very simple script, that will remux all files in all subfolders to mkv
#!/bin/bash
# Works with subfolders too
shopt -s nullglob
shopt -s extglob
shopt -s nocaseglob
shopt -s globstar
for ...
- 1,396
2
votes
2
answers
143
views
Parallel for-loop in bash with simultaneous sequential execution of another task with dependencies on the parallelized loop
In my bash script, I need to execute two different functions, taskA and taskB, which take an integer ($i) as an argument. Since taskB $i depends on the completion of taskA $i, the following ...
- 121
1
vote
1
answer
189
views
Access different combination of variables in bash
I am new to the group so please apologize for any mistake or badly formulated question.
Here's my problem. I have a few arrays of variables. I want to simultaneously start different jobs with all ...
- 11