What is this blog post about?
This technical guide explores the design and development of AI agents using Azure AI Foundry. It explains both the rationale and methodology for building AI agents with Azure AI Foundry, targeting solution architects, AI engineers, and data scientists seeking to leverage AI for business transformation.
The guide provides a step-by-step walkthrough for creating an AI agent application analyzes user input, generates image design recommendations, and produces unique designs based on some instructions, patterns, or user preferences. It features code snippets, architectural diagrams, and practical explanations on utilizing Azure AI Foundry and OpenAI models to build a robust image design application.
Why does it matter?
AI agents are increasingly vital for organizations aiming to harness artificial intelligence to optimize processes and products. They automate tasks, analyze data, and deliver insights that drive better decision-making and product innovation. AI agents also enhance customer experiences through personalized recommendations and support, helping businesses remain competitive in a rapidly evolving landscape.
Despite their benefits, building AI agents can be complex and resource-intensive, often requiring deep technical expertise. This blog post aims to demystify the process by offering a clear, actionable guide for creating AI agents with Azure AI Foundry.
Why use Azure AI Foundry?
Azure AI Foundry is a comprehensive, cloud-based platform that delivers a suite of AI tools and services for scalable AI adoption. It supports model deployment, agent creation, fine-tuning, data management, and seamless integration with Azure services. Organizations can build custom AI models and agents tailored to their needs, streamlining workflows and boosting productivity.
Key benefits of Azure AI Foundry include:
- Low-code development: Rapidly deploy models, set up agents, create knowledge and data indexes, and integrate components with minimal coding via fully managed agent services.
- Flexible deployment: Deploy models and agents across regions and availability zones to minimize latency and serve users closer to their location. Deployments can also be managed within your own Azure subscription or private cloud for greater control.
- Scalability: Effortlessly scale resources to match business demands.
- Enterprise integration: Seamlessly connect with Microsoft 365, Dynamics 365, Azure services, Databricks, and more.
- Model Context Protocol (MCP) support: Leverage the open-source Model Context Protocol (MCP) to enable AI agents to communicate and interoperate seamlessly with tools and agents across diverse ecosystems.
- Multi-agent orchestration support: Build sophisticated AI applications by orchestrating multiple agents to collaborate on complex workflows and problem-solving. Open interoperability standards—such as Agent2Agent (A2A)—enable seamless information exchange and coordination among agents across Azure, AWS, Google Cloud, and on-premises environments.
- Proprietary model access: Instantly access advanced proprietary models such as Grok 3, GPT-4o, o3, and more, empowering your applications with state-of-the-art natural language processing and image generation capabilities.
- Open-source model support: Access over 10,000 open-source models—including Llama, Mistral, DeepSeek, and more—from a variety of vendors directly within Foundry Models.
- Build-in monitoring, tracing, and governance: Access deep insights into agent performance and behavior, enabling continuous improvement through built-in monitoring, tracing, and governance features.
For more details on Azure AI Foundry's features, please refer to the blog of Azure AI Foundry: Your AI App and agent factory.
What about AWS Bedrock and Google Cloud Vertex?
AWS Bedrock is a powerful cloud-based AI platform with similar capabilities for building and deploying AI agents. However, it can be more complex to configure and maintain, as many AWS AI services are based on open-source components that require additional setup. Azure AI Foundry, by contrast, offers a more integrated and user-friendly experience, with built-in Azure service support and direct access to OpenAI models, making it easier to create and deploy AI agents without extensive technical expertise.
Google Cloud Vertex AI also provides robust tools for building and deploying AI agents, including the Agent Development Kit (ADK) and support for the open A2A protocol for cross-ecosystem agent communication. However, Google Cloud's ecosystem-centric approach may limit integration with external platforms, making it more challenging to build agents requiring broad interoperability. Azure AI Foundry, in contrast, offers deeper integration across the Microsoft ecosystem and simplifies multi-service deployments.
What about the other open-source AI agent frameworks, such as LangChain, CrewAI, etc.?
There are several open-source AI agent frameworks—such as LangChain, LlamaIndex, and CrewAI—that offer flexibility and extensibility for building custom AI solutions. However, these frameworks often require considerable effort to set up, configure, and maintain, especially when integrating with enterprise data, applications, and infrastructure. Seamless integration with Azure services and OpenAI models is typically less mature, and features like scalability, monitoring, and governance may require additional tooling or custom development compared to Azure AI Foundry's built-in capabilities.
Open-source frameworks are generally best suited for proof-of-concept (PoC) development, desktop application development, or highly specialized tasks, such as advanced natural language processing or domain-specific data analysis. They often require orchestrating multiple cloud services and AI models, which can add operational complexity. For example, building a healthcare data analysis agent with CrewAI might involve integrating Claude 4 on AWS Bedrock for code generation, Gemini on Google Cloud Vertex AI for healthcare data understanding, and OpenAI image models on Azure AI Foundry for image generation. This multi-cloud, multi-model approach can increase both technical complexity and operational overhead.
When choosing an AI agent framework, a key consideration is the location of your data and applications. If your ecosystem is primarily on Azure, then Azure AI Foundry could be naturally the choice. It provides a unified, enterprise-ready platform that streamlines the process of building, deploying, and managing AI agents—while leveraging the full power of Azure services and OpenAI models.
How to build AI agents with Azure AI Foundry?
The rest of this blog post will guide you through the process of building an AI agent application using Azure AI Foundry, focusing on image design (i.e., editing and generation capabilities). The goal is to create AI agents that can analyze user input, generate design recommendations based on some instructions, patterns or user preferences, and create unique designs. The following sections will cover the solution, architecture design, and a step-by-step process to set up and develop the application.
What is the solution?
The solution is to leverage AI agents to streamline the image design process, enabling people to quickly adapt to changing trends, styles, and preferences. By using Azure AI Foundry and OpenAI models, people can create a comprehensive image design application that automates the analysis of user input, generates recommendations, and creates unique designs.
This application will allow people to focus on creativity and innovation, while the AI agent handles the repetitive and time-consuming tasks of analyzing instructions and preferences and generating designs. The application will also provide a user-friendly interface for people to interact with the AI agent, making it easy to input design requirements, reference images, and instructions.
What are the AI models used?
OpenAI models are state-of-the-art generative AI models that can be used for various applications, including natural language processing, image generation, and more. These models can help people create unique and personalized designs, analyze instructions and consumer preferences, and generate design recommendations. By integrating OpenAI models with Azure AI Foundry, people can create a powerful image design application that leverages the strengths of both technologies.
What is the architecture design?
To get started, the diagram below shows the architecture that will be developed. Simply put, it utilizes Azure AI Foundry for agent orchestration, OpenAI’s image generation model (gpt-image-1), blob storage to store instructions (i.e., documents) and images, and an app service to host the Streamlit UI.
- Enter detailed text prompts describing the desired image design.
- Select existing patterns or preferred styles from a library or upload a reference image to guide the AI.
- Choose an instruction to inform the design.
- Generate initial images using AI based on inputs.
- Refine designs by modifying prompts or editing the generated images.
Step-by-Step Process to Setup and Develop the Application
This section provides a technical look "under the hood" to demonstrate how the previously mentioned architecture was implemented. The following code snippets and explanations detail the core logic, from interpreting a designer's initial text prompt to generating a final, production-ready images. We will walk through how each component—setting up the Azure AI Foundry project, AI agent, user input analysis, reference image processing, instruction integration, and image generation—is built and orchestrated using Python, Azure AI, and OpenAI models.
1. Setup AI Project on Azure AI Foundry
Start by setting up an Azure AI Foundry account and creating a new project. This will provide you with access to the necessary tools and services to create custom AI models. Azure AI Foundry can host multiple projects, you need to first create a Hub, and a Project within it. For the details of how to create a Hub and Project, please refer to the Azure AI Foundry documentation. The following screenshot shows an example.
In this example, we create an Azure AI hub and link it to the subscription of "IX Sandbox". And then we create a project as shown below.
After creating the project, you will need to create/enable Azure AI services that will be used in the project. The following screenshot shows an example of creating/enabling Azure AI Services.
In this example, we create an AI Services and then connect it to the project. The available services include Azure OpenAI, Speech, Content understanding, Translation and a lot of other Azure AI capabilities. For the details of how to create and manage Azure AI services, please refer to the Azure AI Services website.
Once the project and AI services are created, you can manage your AI models, data, and other resources. The dashboard provides an overview of your project and allows you to monitor the performance of your AI models. Let's set up some AI models to be used in this project. Go to the "My assets" section on the left panel, and then select "Models + endpoints". You can see the list of already deployed AI models that are available in the project. The following screenshot shows an example.
By clicking the "+ Deploy model" button and selecting "Deploy base model", you can deploy a base model from the Azure AI Foundry model catalog. Follow the instructions to select the models that will be used in the project, such as OpenAI gpt-4o and OpenAI gpt-image-1.
To trace and monitor the AI agent behavior and activities, we can set up and enable Azure AI Foundry Application Insights Tracing feature to log and visualize the information when the AI agent is being called. Find the “Tracing” on the left-hand side of navigation and then click the “Create New” button shown in the screenshot below.
And then fill the form on the popup window to create a new Application Insights resource. After you run the AI application, you can view the list of traced agent activities in the Tracing section. When you click on a tracing item, the popup window will show and visualize the information logged while the AI agent is being called and running, like the screenshot shown below. Tracing provides deep visibility into execution of your application by capturing detailed telemetry at each execution step. This helps diagnose issues and enhance performance by identifying problems such as inaccurate tool calls, misleading prompts, high latency, low-quality evaluation scores, and more.
2. Setup Azure AI Search Service
The next step is to load a document. This will be used to provide instruction into the design requirements. The document can be uploaded into a container of an Azure blob storage account and indexed by the Azure AI Search service, as shown in the screenshot below.
The technology behind the Azure AI Search service is called "Retrieval Augmented Generation (RAG)", which is a powerful tool that can be used to index and search large amounts of unstructured data (e.g., PDF documents). For the details of how to use Azure AI Search service to index and search document contents in a blog storage, please refer to the Azure AI Search service documentation. And then, we can configure an AI agent in the Azure AI Foundry to use the Azure AI Search service to search the instruction and extract the relevant information.
3. Create an AI Agent
We will create an AI agent in the Azure AI Foundry project, which will analyze the instruction and provide a design recommendation into the design requirements. The following screenshot shows an example of the AI Agent to search an instruction and generate a design recommendation.
In this example, we create an AI agent and select the deployed gpt-3.5-turbo model and the created index in the "Knowledge" section.
4. Develop Code to Analyze User's Input
Defining the design requirements would include factors such as target audience, style, preferences, and standards. For this component, we will let the designers to type the requirements in a text box and then the requirement will be used to create a prompt for the OpenAI model. In order to understand what type of images the designer is looking for, we need to analyze the user input and determine the image type. The following code snippet shows how to analyze the user input and determine the image type by leveraging OpenAI gpt-4o model.
from openai import AzureOpenAI
import os
def analyze_user_input_agent(user_input):
# Azure OpenAI client
client = AzureOpenAI(
api_version="2024-12-01-preview",
azure_endpoint=os.getenv('AZURE_GPT4_MODEL_ENDPOINT'),
api_key=os.getenv("AZURE_API_KEY")
)
completion = client.chat.completions.create(
model="gpt-4o", # or gpt-4o-mini for cost saving
messages=[
{
"role": "user",
"content": f"""You are a helpful assistant that can analyze the user's input and then determine the image type.
For example, if the user's input is "I want to design a clothing", the product type is "clothing".
If the user's input is "I want to design a house", the product type is "house".
Now, here is the user's input: {user_input}
Please determine the image type based on the user's input. If the user's input is not clear, return the general product type of "drawing image".
"""
},
],
temperature=0.0,
top_p=1.0,
frequency_penalty=0.0,
presence_penalty=0.0,
)
return completion.choices[0].message.content # The image type response from the AI model
5. Develop Code to Add a Reference Image
The next component is to add a reference image for the image design. The idea of using a reference image is to provide a visual representation of the silhouettes, design patterns or detailed requirements. This can help the AI model to better understand the user's input, guidance, or/and patterns, and generate more accurate design images. The reference image can be uploading an existing image or picking a pre-defined design outline image from a list. The uploaded image can be converted into a design outline image. For example, the following shows a reference image, which is a pre-defined design outline image for a girl's dress.
This type of reference image will be used to generate new designs based on the user's input of the design requirements. The following code snippet shows how to convert an uploaded image into a reference image.
import os
from openai import AzureOpenAI
import base64
def convert_image_agent(image_file):
client = AzureOpenAI(
api_version="2025-04-01-preview",
azure_endpoint=os.getenv('AZURE_IMAGE_MODEL_ENDPOINT'),
api_key=os.getenv('AZURE_API_KEY')
)
prompt = "Convert the given image to a simplified outline vector .svg image."
result = client.images.edit(
model="gpt-image-1",
image=image_file,
prompt=prompt,
)
image_base64 = result.data[0].b64_json
image_bytes = base64.b64decode(image_base64)
with open("temp/vector_image.png", "wb") as f:
f.write(image_bytes)
return "temp/vector_image.png"
6. Develop Code to Run the AI Agent for Instruction Analysis
The following code snippet shows how to leverage the AI agent to generate a design recommendation based on the instruction.
from openai import AzureOpenAI
from azure.identity import DefaultAzureCredential, get_bearer_token_provider
from azure.ai.projects import AIProjectClient
from azure.core.credentials import AzureKeyCredential
import os
def analyze_instruction_agent(image_type):
prompt_instructions = """
You are a designer and will review the design instruction by using file search tool and generate a design recommendation.
Don't ask the user any questions, just try your best to generate the recommendation based on the report review by following guidelines below:
• One sentence to describe current and forecasting future trends (colors, silhouettes, patterns, themes).
• One sentence to describe target customer needs, preferences, and lifestyle.
• One sentence to describe any other relevant information, such as background, practices, and styles.
"""
# Azure AI Project client
agent_client = AIProjectClient.from_connection_string(
credential=DefaultAzureCredential(),
conn_str=os.getenv("AZURE_PROJECT_CONNECTION_STRING")
)
# Setup the Azure Monitor for the Azure AI Project agent
monitor_connection_string = agent_client.telemetry.get_connection_string()
configure_azure_monitor(connection_string=monitor_connection_string)
# Setup the OpenTelemetry tracer
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("analyze_instruction_agent"):
# Azure AI Project agent
agent = agent_client.agents.get_agent(os.getenv("AZURE_AGENT_ID"))
thread = agent_client.agents.create_thread()
if image_type:
message = agent_client.agents.create_message(
thread_id=thread.id,
role="user",
content=f"Instructions: {prompt_instructions} \n\n Request: What recommendations can you provide for {image_type} design"
)
run = agent_client.agents.create_and_process_run(
thread_id=thread.id,
agent_id=agent.id)
messages = agent_client.agents.list_messages(thread_id=thread.id)
response = messages.text_messages[0].get("text")["value"]
else:
response = ""
return response # The design recommendation response from the AI model
7. Develop Code to Generate Design Images
The next component is to generate the design image(s) based on the user input, reference image, and instruction. The generated design images will be unique and personalized designs that meet the design requirements. The following code snippet shows how to generate the design based image(s).
def generate_image_agent(user_input, image_type, recommendation, num_designs: int, reference_image, image_file=None):
prompt = f"""
You are a professional image designer with deep understanding of patterns, styles, and audience preferences.
Your task is to generate a design image based on the user's thoughts and ideas of {user_input} and product type of {image_type}
and the design recommendation: {recommendation}, which is based on an instruction.
Adhere to the image pattern from the given vector image.
Make the image a realistic image that would be found in an online website.
Make the background white.
"""
client = AzureOpenAI(
api_version="2025-04-01-preview",
azure_endpoint=os.getenv('AZURE_IMAGE_MODEL_ENDPOINT'),
api_key=os.getenv('AZURE_API_KEY')
)
if image_file:
result = client.images.edit(
model="gpt-image-1",
image=[open(image_file, "rb"), open(reference_image, "rb")],
prompt=prompt,
size="1024x1024",
quality="high",
n=num_designs
)
else:
result = client.images.edit(
model="gpt-image-1",
image=open(reference_image, "rb"),
prompt=user_input,
size="1024x1024",
quality="high",
n=num_designs
)
for index, image in enumerate(result.data):
image_base64 = image.b64_json
image_bytes = base64.b64decode(image_base64)
# save each generated image to a file
with open(f"temp/generated_image_{index+1}.png", "wb") as f:
f.write(image_bytes)
return result
The following image shows an example of the generated design image based on the user input, reference image, and instruction.
8. Develop Streamlit UI to Interact with the AI Agents
Finally, we will develop a Streamlit UI to interact with the AI agent and generate design images. For the details of how to develop a Streamlit UI, please refer to the Streamlit documentation. Another place to explore on how to develop a Streamlit UI for AI applications is the Build powerful generative AI apps.
Conclusion
Azure AI Foundry offers a robust, scalable, and integrated platform for building, deploying, and managing AI agents across a wide range of industries and use cases. Its support for multi-agent orchestration, seamless integration with both proprietary and open-source models, and built-in monitoring and governance make it an ideal choice for organizations seeking to accelerate their AI adoption and innovation.
This post has contributed a clear explanation of why Azure AI Foundry stands out among cloud-based and open-source AI agent frameworks, highlighting its unique advantages in flexibility, interoperability, and enterprise readiness. Additionally, the step-by-step guide provided here serves as a practical resource for anyone looking to build AI agents—from initial setup to deployment—using Azure AI Foundry. By following these best practices, organizations can unlock new opportunities for automation, insight, and collaboration, positioning themselves for long-term success in the evolving landscape of artificial intelligence.
Top comments (0)