Invoking multiple Azure functions from powershell function

Question

I have a simple powershell orchestrator which just reads a text file and sends parts of the list to be processed on functionA.

$payLoad = Get-Content tracks.txt
[int]$itemCount = $payLoad.Count / 50

for ($i = 0; $i -le $payLoad.Count; $i++) {
    
    $payloadList += $payLoad[$i]
    if ($payloadList.Count -eq $itemCount) {
        Write-Host $payloadList
        Start-ThreadJob -Argumentlist $payloadList -Scriptblock {
            # Here I am filling urls etc.
            Invoke-RestMethod @params
        }
        $payloadList = @()
    }
}

I have close to 1000 items in the list but only roughly 200 is always processed. I can see from the orchestrator logs that the powershell threadjob is created for all but somewhere big portion of my jobs disappear. Does Azure functions not support this way of invoking functions or where am I going wrong?

Cpt.Whale · Accepted Answer · 2021-04-06 16:39:03Z

I don't quite understand the first section. Are you trying to process 50 items at a time?

The typical structure of a Start-ThreadJob block would look more like this:

# Divide $payloads into chunks of 50
$script:counter = 0
$payloads = Get-Content tracks.txt | Group-Object {[math]::Floor($script:counter++/50)}

# Process each job on separate threads (default throttle limit is 5)
# Save job IDs to $jobs
$jobs = $payloads | Foreach {
    Start-ThreadJob -ThrottleLimit 5 -ArgumentList $_.Group -ScriptBlock {
        Invoke-RestMethod @params
    }
}

# specifically, wait for all jobs to finish, and get their output:
$result = $jobs | Wait-Job | Receive-Job

You can also run $jobs by itself to see the individual statuses. Or, for detailed testing:

while ($jobs.count -gt ($jobs|? state -eq 'Completed').count) {
    Write-Progress -Activity 'Processing jobs' -PercentComplete (
        [math]::Round((($jobs|? state -eq 'Completed').count/$jobs.count)*100))
}
Write-Output "all jobs complete"

Can you see if $jobs shows creates all 20 jobs, and what their statuses are?

Anatoli Beliaev · Accepted Answer · 2021-04-06 16:30:02Z

In general, you can invoke functions with Invoke-RestMethod, with some caveats. In your case, multiple PowerShell jobs are created and continue running in the background, and your main function does not wait for completion. When the main thread finishes, Azure Functions can dispose the worker and any associated compute resources any time, as there is no function "officially" running anymore. As a result, all the background jobs eventually get terminated.

I would recommend either waiting for all the jobs to finish before the function exits. If this can take long time so that the function times out, consider Durable Functions: your scenario looks very much like the "Fan out/fan in" pattern.

Collectives™ on Stack Overflow

Invoking multiple Azure functions from powershell function

2 Answers 2

Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Related