1

Similar to this thread, I have a remote machine with 8 cores that I want to use for running scripts in parallel (1 script per core at a time).

However, I don't have multiple bash scripts but a single Python3 script that I want to run with different inputs. I tried parallel python3 -c main.py input*, parallel -j 100% python3 -c main.py ::: input*, and parallel python3 main.py input* but nothing worked.

The exact error message is:

parallel: Error: -g has been retired. Use --group.
parallel: Error: -B has been retired. Use --bf.
parallel: Error: -T has been retired. Use --tty.
parallel: Error: -U has been retired. Use --er.
parallel: Error: -W has been retired. Use --wd.
parallel: Error: -Y has been retired. Use --shebang.
parallel: Error: -H has been retired. Use --halt.
parallel: Error: --tollef has been retired. Use -u -q --arg-sep -- and --load for -l.

I don't understand how this is related to my input. I didn't use any of these options.

I'm fairly new and inexperienced with Unix and couldn't get it to work myself or with googling. Any help is appreciated. Do I have to write a shell-script to help me with that?

Improve this question
4
  • There are a lot of different versions of parallel out there, do you know which one you have? (parallel --version or man parallel should list it somewhere) Commented Dec 1, 2016 at 9:55
  • It's GNU parallel 20141022 Commented Dec 1, 2016 at 10:12
  • Make sure you do not have a variable PARALLEL set in your environment. Check with echo $PARALLEL. Try preceding your command with PARALLEL= , as in PARALLEL= parallel python3 ... all on one line. Commented Dec 1, 2016 at 18:05
  • Good idea but unfortunately it's not the problem. No environmental variable PARALLEL is set and preceding PARALLEL= leads to the same error message. Can I somehow reinstall parallel? Does it even make sense? Commented Dec 1, 2016 at 18:59

3 Answers 3

Reset to default
2

The problem was actually with how parallel was installed on the remote machine (running the newest Ubuntu). I came across a thread solving my problem: Run sudo rm /etc/parallel/config after installation on Ubuntu to get rid of the config which caused my error messages.

The command I use to run my python script with different inputs in parallel is: parallel -j 100% python3 main.py ::: inputs*

Nevertheless, thanks to everyone who was helping!

Improve this answer
0

You can do that by assigning every script to specific core using taskset first you need to get the process id for your script with ps aux | grep <scriptname, input> or using pgrep -f <scriptname, input> then pass it to taskset like this taskset -pc 0 $pid, this will assign the process to core 1. for more info about taskset http://linuxcommand.org/man_pages/taskset1.html

Improve this answer
4
  • So I can't just use parallel and let it handle the assignment to the different cores? Commented Dec 1, 2016 at 9:50
  • i am not aware with parallel tool but the way that i mentioned will do the job but we will need to write small script to make this process auto Commented Dec 1, 2016 at 9:53
  • It sounds like I have to manually assign the different jobs to different cores. And what happens if I assign multiple jobs to the same core? Will it run them one after another or will run it concurrently by switching between the threads (I don't want that)? Commented Dec 1, 2016 at 10:14
  • don't run two scripts in one core at the same time wait until it finish then start the other one, take a look into this it may help you with ordering unix.stackexchange.com/questions/242087/… Commented Dec 1, 2016 at 10:28
0

I'm not sure where you are getting your error message from. Normally the syntax would be

parallel -j 100% python3 -c main.py ::: input*

For example, if you just have 2 files inputa, inputb this runs in parallel

python3 -c main.py inputa
python3 -c main.py inputb

If you have 8 cpu cores, and, say, 10 input* files, the -j 100% argument makes parallel run only 8 commands at once at the start. When one finishes, the next command will be run until all 10 are done. We rely on normal Linux scheduling to assign each command to a separate cpu.

Improve this answer
1
  • It still gives me the same error. I updated the question to now contain the full error message(s). Commented Dec 1, 2016 at 17:17

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.