Marcos Garcia

Posted on Jun 25

How to build Azure Load Tests and Automate It via Pipelines

#azure #testing #webdev #cloudcomputing

Introduction

Load testing is essential when building scalable applications in the cloud. Azure Functions should scale up naturally, but how can you know if they handle spikes in demand? In this blog, we'll explore how to use Azure Load Testing to stress-test your Azure Functions and automate the process using CI/CD pipelines like Azure DevOps.

By the end of this guide, you'll know how to:

Prepare an Azure Function for load testing
Create a request to the Azure Function in Postman
Create and run a load test in Azure
Automate the load test in the pipeline
Set performance thresholds to fail builds

Use Case / Problem Statement

Imagine you're building an API using Azure Functions that will be called thousands of times per minute. How do you ensure your app can handle that load without failing or slowing down? Without proper load testing, performance regressions may go unnoticed until it's too late.

Prerequisites

Azure Subscription
Azure Load Testing resource
An Azure Function App with at least one HTTP-triggered function
Azure DevOps or GitHub repository with pipeline access
Apps: Code Editor of choice, Postman, JMeter

Step 1: Prepare The Azure Function for Testing

Make sure the Azure Function is deployed and has a public endpoint. Secure it using Function Keys or authentication as needed.
We will use Postman first to make sure there's proper access to the function and your IP is whitelisted.

Download Postman. Open your terminal/shell and run winget install Postman.Postman if you are on windows or if on Fedora/HREL, use the following commands: sudo dnf install snapd,sudo ln -s /var/lib/snapd/snap /snap,sudo snap install postman. Check snapcraft.io for more details.
Get the token ready. An app registration will be needed for this step. I went through it in more detail in this dev.to post

Make sure to provide the correct values in the request body. Replace the Tenant id, Client id, and Client secret according to your App Registration, but scope must reflect the Azure Function name (see in the screenshot below).

Optional: add a post response script to Postman to avoid token copy and paste:
```
const jsonData = pm.response.json();

if (jsonData.access_token) {
    pm.globals.set("AzureToken", jsonData.access_token);
}
```
Build a request to your Azure Function.

Make sure to add the Bearer token manually to the headers or in the Authorization tab, and to use the correct URL, which should be the "Default domain" from the Function overview page, and the chosen route, in my case it's defined in the Function code:
If this request succeeds, proceed to the Load Test creation, otherwise, I suggest troubleshooting it first. I'd take a close look at the permissions granted to the App Registration, managed identities, RBAC, Authentication Settings, and network.

If the Function is configured with "Restricted Access" in the authentication settings, find your public IP address and add to the white-listed IPs at Networking > Public network access "Enabled with access restrictions".

Step 2: Create a Load Test in Azure Load Testing

Create a Load Testing resource in Azure.
Install JMeter. If using windows install Apache JMeter using winget install DEVCOM.JMeter and on Fedora/Linux or another HREL distro, dnf install jmeter.
To open the application, simply run jmeter in your shell/terminal of choice. This command may vary depending on the distro/OS.

I created a template to get started. The file can be overwhelming, but contains as many features I find useful for the next step.

<?xml version="1.0" encoding="UTF-8"?>
<jmeterTestPlan version="1.2" properties="5.0" jmeter="5.6.3">
  <hashTree>
    <TestPlan guiclass="TestPlanGui" testclass="TestPlan" testname="Function‑API Load Test Template">
      <boolProp name="TestPlan.serialize_threadgroups">true</boolProp>
      <boolProp name="TestPlan.tearDown_on_shutdown">true</boolProp>
      <elementProp name="TestPlan.user_defined_variables" elementType="Arguments" guiclass="ArgumentsPanel" testclass="Arguments">
        <collectionProp name="Arguments.arguments">
          <elementProp name="function_host" elementType="Argument">
            <stringProp name="Argument.name">function_host</stringProp>
            <stringProp name="Argument.value">${__groovy(System.getenv(&apos;FUNCTION_HOST&apos;) ?: &apos;your‑function.azurewebsites.net&apos;)}</stringProp>
            <stringProp name="Argument.metadata">=</stringProp>
          </elementProp>
          <elementProp name="tenant_id" elementType="Argument">
            <stringProp name="Argument.name">tenant_id</stringProp>
            <stringProp name="Argument.value">${__groovy(System.getenv(&apos;AAD_TENANT_ID&apos;) ?: __GetSecret(&apos;aadTenantId&apos;))}</stringProp>
            <stringProp name="Argument.metadata">=</stringProp>
          </elementProp>
          <elementProp name="client_id" elementType="Argument">
            <stringProp name="Argument.name">client_id</stringProp>
            <stringProp name="Argument.value">${__groovy(System.getenv(&apos;AAD_CLIENT_ID&apos;) ?: __GetSecret(&apos;aadClientId&apos;))}</stringProp>
            <stringProp name="Argument.metadata">=</stringProp>
          </elementProp>
          <elementProp name="client_secret" elementType="Argument">
            <stringProp name="Argument.name">client_secret</stringProp>
            <stringProp name="Argument.value">${__groovy(System.getenv(&apos;AAD_CLIENT_SECRET&apos;) ?: __GetSecret(&apos;aadClientSecret&apos;))}</stringProp>
            <stringProp name="Argument.metadata">=</stringProp>
          </elementProp>
          <elementProp name="aad_scope" elementType="Argument">
            <stringProp name="Argument.name">aad_scope</stringProp>
            <stringProp name="Argument.value">${__groovy(System.getenv(&apos;AAD_SCOPE&apos;) ?: &apos;api://YOUR‑APP‑ID/.default&apos;)}</stringProp>
            <stringProp name="Argument.metadata">=</stringProp>
          </elementProp>
          <elementProp name="NumberOfThreads" elementType="Argument">
            <stringProp name="Argument.name">NumberOfThreads</stringProp>
            <stringProp name="Argument.value">${__groovy(System.getenv(&apos;THREADS&apos;) ?: &apos;50&apos;)}</stringProp>
            <stringProp name="Argument.metadata">=</stringProp>
          </elementProp>
          <elementProp name="RampUpInSeconds" elementType="Argument">
            <stringProp name="Argument.name">RampUpInSeconds</stringProp>
            <stringProp name="Argument.value">${__groovy(System.getenv(&apos;RAMP_UP&apos;) ?: &apos;30&apos;)}</stringProp>
            <stringProp name="Argument.metadata">=</stringProp>
          </elementProp>
          <elementProp name="DurationInSeconds" elementType="Argument">
            <stringProp name="Argument.name">DurationInSeconds</stringProp>
            <stringProp name="Argument.value">${__groovy(System.getenv(&apos;DURATION&apos;) ?: &apos;60&apos;)}</stringProp>
            <stringProp name="Argument.metadata">=</stringProp>
          </elementProp>
          <elementProp name="userid_fetch_path" elementType="Argument">
            <stringProp name="Argument.name">userid_fetch_path</stringProp>
            <stringProp name="Argument.value">${__groovy(System.getenv(&apos;PATH_FETCH&apos;) ?: &apos;api/org/userid/fetch&apos;)}</stringProp>
            <stringProp name="Argument.metadata">=</stringProp>
          </elementProp>
        </collectionProp>
      </elementProp>
    </TestPlan>
    <hashTree>
      <ThreadGroup guiclass="ThreadGroupGui" testclass="ThreadGroup" testname="AAD Token Thread Group">
        <intProp name="ThreadGroup.num_threads">1</intProp>
        <intProp name="ThreadGroup.ramp_time">1</intProp>
        <boolProp name="ThreadGroup.same_user_on_next_iteration">true</boolProp>
        <stringProp name="ThreadGroup.on_sample_error">continue</stringProp>
        <elementProp name="ThreadGroup.main_controller" elementType="LoopController" guiclass="LoopControlPanel" testclass="LoopController" testname="Loop Controller">
          <stringProp name="LoopController.loops">1</stringProp>
          <boolProp name="LoopController.continue_forever">false</boolProp>
        </elementProp>
      </ThreadGroup>
      <hashTree>
        <JSR223Sampler guiclass="TestBeanGUI" testclass="JSR223Sampler" testname="Acquire AAD Token">
          <stringProp name="scriptLanguage">groovy</stringProp>
          <stringProp name="script">
import org.apache.http.client.methods.HttpPost
import org.apache.http.impl.client.HttpClients
import org.apache.http.entity.StringEntity
import org.apache.http.util.EntityUtils
import groovy.json.JsonSlurper

def client = HttpClients.createDefault()

def tokenUrl = &quot;https://login.microsoftonline.com/${vars.get(&apos;tenant_id&apos;)}/oauth2/v2.0/token&quot;

def post = new HttpPost(tokenUrl)
post.setHeader(&apos;Content-Type&apos;, &apos;application/x-www-form-urlencoded&apos;)

def body = [
    grant_type: &apos;client_credentials&apos;,
    client_id : vars.get(&apos;client_id&apos;),
    client_secret: vars.get(&apos;client_secret&apos;),
    scope: vars.get(&apos;aad_scope&apos;)
].collect { k, v -&gt; &quot;${k}=${URLEncoder.encode(v, &apos;UTF-8&apos;)}&quot; }.join(&apos;&amp;&apos;)

post.setEntity(new StringEntity(body))

def response = client.execute(post)

def json = new JsonSlurper().parseText(EntityUtils.toString(response.entity))

props.put(&apos;access_token&apos;, json.access_token)
log.info(&apos;AAD access token fetched.&apos;)
</stringProp>
          <stringProp name="cacheKey">false</stringProp>
          <stringProp name="parameters"></stringProp>
          <stringProp name="filename"></stringProp>
        </JSR223Sampler>
        <hashTree/>
      </hashTree>
      <ThreadGroup guiclass="ThreadGroupGui" testclass="ThreadGroup" testname="API Main Thread Group">
        <stringProp name="ThreadGroup.num_threads">${NumberOfThreads}</stringProp>
        <stringProp name="ThreadGroup.ramp_time">${RampUpInSeconds}</stringProp>
        <stringProp name="ThreadGroup.duration">${DurationInSeconds}</stringProp>
        <boolProp name="ThreadGroup.same_user_on_next_iteration">true</boolProp>
        <boolProp name="ThreadGroup.scheduler">true</boolProp>
        <stringProp name="ThreadGroup.on_sample_error">continue</stringProp>
        <elementProp name="ThreadGroup.main_controller" elementType="LoopController" guiclass="LoopControlPanel" testclass="LoopController" testname="Loop Controller">
          <intProp name="LoopController.loops">-1</intProp>
          <boolProp name="LoopController.continue_forever">false</boolProp>
        </elementProp>
      </ThreadGroup>
      <hashTree>
        <HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="Fetch Record Id">
          <stringProp name="HTTPSampler.domain">${function_host}</stringProp>
          <stringProp name="HTTPSampler.protocol">https</stringProp>
          <stringProp name="HTTPSampler.path">${user_fetch_path}</stringProp>
          <stringProp name="HTTPSampler.method">POST</stringProp>
          <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
          <boolProp name="HTTPSampler.postBodyRaw">true</boolProp>
          <elementProp name="HTTPsampler.Arguments" elementType="Arguments">
            <collectionProp name="Arguments.arguments">
              <elementProp name="" elementType="HTTPArgument">
                <boolProp name="HTTPArgument.always_encode">false</boolProp>
                <stringProp name="Argument.value">{&quot;firstName&quot;:&quot;John&quot;,&quot;lastName&quot;:&quot;Doe&quot;,&quot;email&quot;:&quot;[email protected]&quot;,&quot;dateOfBirth&quot;:&quot;1980‑09‑01T00:00:00Z&quot;}</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
            </collectionProp>
          </elementProp>
        </HTTPSamplerProxy>
        <hashTree>
          <HeaderManager guiclass="HeaderPanel" testclass="HeaderManager" testname="Bearer Header #1">
            <collectionProp name="HeaderManager.headers">
              <elementProp name="Auth" elementType="Header">
                <stringProp name="Header.name">Authorization</stringProp>
                <stringProp name="Header.value">Bearer ${__P(access_token)}</stringProp>
              </elementProp>
            </collectionProp>
          </HeaderManager>
          <hashTree/>
        </hashTree>
      </hashTree>
    </hashTree>
  </hashTree>
</jmeterTestPlan>

Open the .jmx file in JMeter, there should be test structure like this:

Note that all the variables are in the topmost section, the Test Plan definition. To testing locally, replace the values for the token to look just like Postman, and hit "Start".

Quick explanation for the variables section:
- ${__groovy(...)}: return the value evaluated by the Groovy code
- ${__GetSecret(...)}: return the secret value defined in the config file.
- ${__P(access_token)}: return the value of a JMeter property
- props.put("access_token", json.access_token): set value to a JMeter property
- System.getenv('FUNCTION_HOST'): Fetches the value of the environment variable
- ?:: Ternary/Elvis Operator
Before proceeding to the pipeline you should upload it to the Azure Load Test resource and configure it.

Open the resource
Go to Tests > Create > Create test:
Once created open the Test plan tab and upload the file.
To authenticate the Load test, go back to the Overview page, then access Settings > Identity, and choose the desired method (User assigned recommended). Back in the Test plan there should be the selected option at the bottom.
In the Parameters tab set the environment variables and secrets:

Make sure the Key Vault is referenced at the bottom of the page.

The rest of it should be already configured, but for good measure it's recommended to reference the Managed identity once again in the Monitoring tab and to add a vnet configuration if the Function is on a private setting.
Click "Apply", access the test and hit "Run". Configure the variables however you'd like and describe it in the Test run description. I use this pattern: "NumberOfThreads 250 RampUpInSeconds 15 DurationInSeconds 600 EngineInstances 1" just to quickly identify each run. Please note that Debug mode runs on a different setting. Try to find a number that start showing a small percentage of errors and identify the performance thresholds of your Azure Function for benchmarking:

Download the result files from the Ellipsis (⋯) in the test results page for further troubleshooting.

Step 4: Automate It in The Pipeline

Use Azure DevOps (ADO) or GitHub Actions to trigger the load test on pull requests or deployments. I'm going to be using ADO for this but any should work.

Two .yml files are needed to automate it. A Load Test config and a pipeline yml file, they reference each other and should be added to the repository, just like the jmx file. If you are not used to pipelines I recommend going through the basics and ADO Pipeline or Github Actions documentation, as you might want to change these files according to your company's needs.

For the configuration file, it can be downloaded from the results of the previous test run. Access the Test runs and click the Ellipsis (⋯), then "download input file".

The config file should be ready to be used, but I'm adding one here in case as an example:

version: v0.1
testId: processor-loadtest
displayName: Processor Load Test
description: Load test for the Processor Azure Function
testType: JMX
testPlan: Processor-LoadTest.jmx
engineInstances: 2
subnetId: /subscriptions/id/resourceGroups/dev-network-rg/providers/Microsoft.Network/virtualNetworks/vnet-functions/subnets/snet-azurevirtualmachines
publicIPDisabled: true
splitAllCSVs: False
failureCriteria:
  clientMetrics:
  - avg(response_time_ms) > 60000
  - percentage(error) > 5
autoStop:
  errorPercentage: 80
  timeWindow: 60
  maximumVirtualUsersPerEngine: 5000
env:
  - name: NumberOfThreads
    value: 50
  - name: RampUpInSeconds
    value: 30
  - name: DurationInSeconds
    value: 60
secrets:
  - name: tokenAuthClientSecret
    value: https://dev-kv.vault.azure.net/secrets/tokenAuthClientSecret/id
  - name: tokenAuthClientId
    value: https://dev-kv.vault.azure.net/secrets/tokenAuthClientId/id
  - name: tokenAuthTenantId
    value: https://dev-kv.vault.azure.net/secrets/tokenAuthTenantId/id
referenceIdentities:
- kind: KeyVault
  type: UserAssigned
  value: /subscriptions/id/resourceGroups/dev-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/fndev_processor
- kind: Metrics
  type: SystemAssigned

The pipeline file can also be created with the help of ADO/Github, but I'm adding it as another example of what it should look like:

trigger: none   # Disable automatic CI trigger
pr: none        # Disable PR trigger

name: ManualLoadTestPipeline

stages:
  - stage: LoadTesting
    displayName: 'Start Azure Load Test'
    jobs:
      - job: RunAndPublishLoadTest
        displayName: 'Run and Publish Load Test Results'
        timeoutInMinutes: 15
        steps:

          - task: AzureLoadTest@1
            timeoutInMinutes: 10
            displayName: 'Run Load Test'
            inputs:
              azureSubscription: $(serviceConnection)
              loadTestConfigFile: 'Processor/LoadTests/Dev-Processor-LoadTestConfig.yml'
              loadTestResource: 'dev-load-testing'
              resourceGroup: 'dev-rg'
            condition: succeeded()

          - publish: Processor/LoadTests
            artifact: loadTestResults

          - task: PublishTestResults@2
            displayName: 'Publish Load Test Results'
            inputs:
              testResultsFormat: 'JUnit'
              testResultsFiles: 'Processor/LoadTests/results/*.xml'
              searchFolder: '$(System.DefaultWorkingDirectory)'
              mergeTestResults: true

Don't forget to add a Service Connection and all necessary condiguration to connect ADO and Azure. Azure DevOps itself doesn’t natively support pipeline auth via managed identity for Azure Load Testing, but that could change in the future.

After reviewing the .yml files and replacing the placeholders with the correct values based on your folder structure and Azure configuration, install the Azure Load Test extension in ADO. After that, it should be found in here: https://dev.azure.com/<your-org>/_settings/extensions.

At this point, the pipeline is ready to run. After that, the results will be available for inspection in Azure. Look for a new test in the Load Test resource with the same name as the "displayName" from the config file. If you configured a results folder in the repo, they should also be published after each run.

Final considerations: With this pipeline working and the test running as expected, consider adding it to your development lifecycle. This could run after successful builds on dev, after the unit tests are all pass, before deploying from dev to QA or UAT, etc. There are many possible applications for Azure load test, including SQL queries directly to a database with JDBC, Azure Kubernetes Service, and Microservices in general, and for the small effort to set it up there's a big value in return.

Best Practices & Considerations

Don't load test production unless necessary
Use deployment slots or staging environments
Account for cold starts in consumption plans
Monitor backend services if the Function triggers other processes

Troubleshooting

Common issues:

Subnet permissions errors (e.g., join/action errors)
Rate limiting (HTTP 429)
Authentication failures (HTTP 403)
Configuration issue (usually HTTP 500)

Solutions:

Ensure correct role assignments and firewall rules
Add retry policies
Use Application Insights for deeper diagnostics

Conclusion

Azure Load tests ensures that serverless applications scale up gracefully. Integrating it into a CI/CD pipeline identifies regression early and gives the team confidence during production deployments.

DEV Community