Running several PHP processes in parallel

try this technique for running multiple parallel jobs in PHP. In this example, we have two job files: j1.php and j2.php we want to run. The sample jobs don't do anything fancy. The file j1.php looks like this:

$jobname = 'j1';
set_time_limit(0);
$secs = 60;

while ($secs) {
        echo $jobname,'::',$secs,"\n";
        flush(); @ob_flush();  ## make sure that all output is sent in real-time
        $secs -= 1;
        $t = time();
        sleep(1); // pause
}

The reason why we flush(); @ob_flush(); is that when we echo or print, the strings are sometimes buffered by PHP and not sent until later. These two functions ensure that all data is sent immediately.

We then have a 3rd file, control.php, which does the coordination of jobs j1 and j2. This script will call j1.php and j2.php asynchronously using fsockopen in JobStartAsync(), so we are able to run j1.php and j2.php in parallel. The output from j1.php and j2.php are returned to control.php using JobPollAsync().

#
# control.php
#
function JobStartAsync($server, $url, $port=80,$conn_timeout=30, $rw_timeout=86400)
{
    $errno = '';
    $errstr = '';

    set_time_limit(0);

    $fp = fsockopen($server, $port, $errno, $errstr, $conn_timeout);
    if (!$fp) {
       echo "$errstr ($errno)<br />\n";
       return false;
    }
    $out = "GET $url HTTP/1.1\r\n";
    $out .= "Host: $server\r\n";
    $out .= "Connection: Close\r\n\r\n";

    stream_set_blocking($fp, false);
    stream_set_timeout($fp, $rw_timeout);
    fwrite($fp, $out);

    return $fp;
}

// returns false if HTTP disconnect (EOF), or a string (could be empty string) if still connected
function JobPollAsync(&$fp) 
{
    if ($fp === false) return false;

    if (feof($fp)) {
        fclose($fp);
        $fp = false;
        return false;
    }

    return fread($fp, 10000);
}

###########################################################################################


if (1) {  /* SAMPLE USAGE BELOW */

    $fp1 = JobStartAsync('localhost','/jobs/j1.php');
    $fp2 = JobStartAsync('localhost','/jobs/j2.php');


    while (true) {
        sleep(1);

        $r1 = JobPollAsync($fp1);
        $r2 = JobPollAsync($fp2);

        if ($r1 === false && $r2 === false) break;

        echo "<b>r1 = </b>$r1<br>";
        echo "<b>r2 = </b>$r2<hr>";
        flush(); @ob_flush();
    }

    echo "<h3>Jobs Complete</h3>";
}

Good Read

Divide-and-conquer and parallel processing in PHP

from the source

If the various files in PHP have no dependency, I think you can use a multi-curl approach which can be implemented as shown :-

    $linkArray = array('file1.php', 'file2.php','file3.php','file4.php','file5.php');
    $nodes = ($linkArray);
    $node_count = count($nodes);

    $curl_arr = array();
    $master = curl_multi_init();
    $counter = 0;
    for($i = 0; $i < $node_count; $i++)
    {
      $url =$nodes[$i];
      $curl_arr[$i] = curl_init($url);
      curl_setopt($curl_arr[$i], CURLOPT_RETURNTRANSFER, true);
      curl_multi_add_handle($master, $curl_arr[$i]);
    }

do {
    curl_multi_exec($master,$running);
} while($running > 0);

for($k=0;$k<$node_count;$k++){

  $result = curl_multi_getcontent  ($curl_arr[$k]); // contains the output of individual files

}