2

I would like to create a simple Python program that will concurrently execute 2 independent scripts. For now, the two scripts just print a sequence of numbers but my intention is to use this program to concurrently run a few Twitter streaming programs in the future.

I suspect I need to use subprocess.Popen but I cannot quite get my head around what arguments I should put in there. There was a similar question on StackOverflow but the code provided there (pasted below) doesn't print anything. I will appreciate your help.

My files are:

thread1.py
thread2.py

import subprocess

subprocess.Popen(['screen', './thread1.py']))
subprocess.Popen(['screen', './thread2.py'])
Improve this question
6
  • 1
    Isn't it simpler and more obvious to run parallel processes using pipelines in Unix? I.e. ./thread1.py;./thread2.py Commented Jun 11, 2014 at 13:20
  • Yes, normal Unix process pipeline will do the trick. Whereas just a heads up, if the operations are i/o bound you should consider using threads over processes in Python. Commented Jun 11, 2014 at 13:22
  • Thanks, I've tried what you suggested: python ./thread1.py;./thread2.py but I'm only getting the printed numbers from the first script and I would like the two scripts to be running at the same time. Commented Jun 11, 2014 at 13:46
  • 1
    Because of the pipeline usage you should run both your scripts as separate python programs: python thread1.py;python thread2.py. In your case I assume you don't have the python preambula, so your scripts require to use python interpreter, as I stated above. That's the reason, why you second script didn't run. Pay attention you should be in the directory your files are located in, and you don't need the "./" prefix, that indicates you to take file from the present folder. Commented Jun 11, 2014 at 14:10
  • I've tried to use the above Unix method for two scripts that download Twitter timelines. Unfortunately, it appears that the scripts are not run in parallel. Each of my scripts builds a separate MySQL table and I can see that the table for the second script only starts building after I interrupt the first script. Is this the case that @AashishP mentioned and should I be using threads? Commented Jun 13, 2014 at 14:00

3 Answers 3

Reset to default
1

Use supervisord

supervisord is process control system just for the purpose of running multiple command line scripts.

It features:

  • multiple controlled processes
  • autorestarting failed runs
  • log stdout and stderr output
  • starting scripts in order (using priority)
  • command line utility to view latest log output, stop, start, restart the processes

This solution works only on *nix based systems, it is not available on Windows.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

1

As wanderlust mentioned, why do you want to do it this way and not via linux command line?

Otherwise, the solution you post is doing what it is meant to, i.e, you are doing this at the command line:

screen ./thread1.py
screen ./thread2.py

This will open a screen session and run the program and output within this screen session, such that you will not see the output on your terminal directly. To trouble shoot your output, just execute the scripts without the screen call:

import subprocess

subprocess.Popen(['./thread1.py'])
subprocess.Popen(['./thread2.py'])

Content of thread1.py:

#!/usr/bin/env python
def countToTen():
  for i in range(10):
  print i

countToTen()

Content of thread2.py:

#!/usr/bin/env python
def countToHundreds():
for i in range(10):
  print i*100

countToHundreds()

Then don't forget to do this on the command line:

chmod u+x thread*.py
Improve this answer

5 Comments

Thanks, I've tried using the above script (in IDLE) but I got many errors. Traceback (most recent call last): line 3, in <module> subprocess.Popen(['./thread1.py']) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 709, in init errread, errwrite) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1326, in _execute_child raise child_exception OSError: [Errno 13] Permission denied
This is most likely related to the content of your 'thread1' and 'thread2'. Can you give any details?
Sure, I got the UNIX solution to work but I'm curious what is wrong with my subprocess.Popen. Here is thread1: 'def countToTen(): for i in range(10): print i countToTen()' and thread2:'def countHundreds(): for i in range (10): print i*100 countHundreds()' I'm getting errors when I run this code: 'import subprocess subprocess.Popen(['thread1.py']) subprocess.Popen(['thread2.py'])' P.S Sorry for my bad formatting, I need to learn how to format here
You are missing the './' infront of thread1.py and thread2.py. Also, within each of the files, thread1.py and thread2.py you will need to point to the python executable, i.e. '#!/usr/bin/env python'. You also need to make them executable, i.e., 'chmod u+x thread*.py'. Otherwise it should work as you want. I will edit my post to include this.
Thank you, it works now from command line (but not if I just try to run it from IDLE but perhaps you can't run concurrent programs there). I need to read up on the "chmod" line but this was a very valuable lesson for a Python beginner.
1

You can also just open several Command Prompt windows to run several Python programs at once - just run one in each of them:

In each Command Prompt window, go to the correct directory (such as C:/Python27) and then type 'python YourCodeNo1.py' in one Command Prompt window, 'python YourCodeNo2.py' in the next one ect. .

I'm currently running 3 codes at one time in this way, without slowing any of them down.

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.