Running multiple commands simultaneously from python

Question

I want to run three commands at the same time from python. The command format is query.pl -args

Currently I am doing

os.system("query.pl -results '10000' -serverName 'server1' >> log1.txt")

os.system("query.pl -results '10000' -serverName 'server2' >> log2.txt")

os.system("query.pl -results '10000' -serverName 'server3' >> log3.txt")

I want to query all three servers at the same time but in this case, each command executes only after the last one has finished. How can I make them simultaneous? I was thinking of using '&' at the end but I want the next part of the code to be run only when all three command finish

Have you read up on the subprocess module yet? What specific questions did you have? This is a duplicate of dozens of subprocess related questions. You might want to look at those and list what is unique or different about your question. — S.Lott
– S.Lott, Commented Oct 28, 2009 at 20:30
That, and are you aware of the fact that you are, probably calling perl scripts from python? Do you want our universe to implode then? — shylent
– shylent, Commented Oct 28, 2009 at 20:33
Why does the question title say "two" but the question says "three" and shows 3? Why the difference? — S.Lott
– S.Lott, Commented Oct 28, 2009 at 21:18

jldupont · Accepted Answer · 2009-10-28 20:35:37Z

10

You could use the subprocess module and have all three running independently: use subprocess.Popen. Take care in setting the "shell" parameter correctly.

Use the wait() or poll() method to determine when the subprocesses are finished.

edited Oct 28, 2009 at 20:35

answered Oct 28, 2009 at 20:30

jldupont

97.5k58 gold badges212 silver badges328 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

randomThought Over a year ago

but How will I know when all three have finished? I want the rest of the code to execute only when the commands finish running

S.Lott Over a year ago

@Jaelebi: status = [ p.wait() for p in list_of_popen_objects ] usually works to see if all processes are done.

Oduvan · Accepted Answer · 2009-10-28 20:36:07Z

0

os.system("query.pl -results '10000' -serverName 'server1' &") 
os.system("query.pl -results '10000' -serverName 'server2' &") 
os.system("query.pl -results '10000' -serverName 'server3' &")

in this case - process will be started in background

answered Oct 28, 2009 at 20:36

Oduvan

2,8373 gold badges26 silver badges25 bronze badges

1 Comment

Oduvan Over a year ago

"&" - this symbol at the end of shell command - says that it must work in background. That's why all process will be run together

Oduvan · Accepted Answer · 2009-10-28 20:41:34Z

0

You can use Queue

tasks = ("query.pl -results '10000' -serverName 'server1'",\
"query.pl -results '10000' -serverName 'server2'",\
"query.pl -results '10000' -serverName 'server1'")

def worker():
    while True:
        item = q.get()
        os.system(item)

q = Queue()
for i in tasks:
     t = Thread(target=worker)
     t.setDaemon(True)
     t.start()

for item in tasks:
    q.put(item)

q.join()

answered Oct 28, 2009 at 20:41

Oduvan

2,8373 gold badges26 silver badges25 bronze badges

1 Comment

mch Over a year ago

Using threads in this way isn't really a good idea when clean and predictable asynchronous IO facilities are available. Additionally, spawning processes and threads don't mix well on some platforms.

Collectives™ on Stack Overflow

Running multiple commands simultaneously from python

3 Answers 3

2 Comments

1 Comment

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

1 Comment

1 Comment

Related