Frontend Tools

This video shows the result of npx copilotkit@latest init with the implementation section applied to it.

What is this?#

Frontend actions are powerful tools that allow your AI agents to directly interact with and update your application's user interface. Think of them as bridges that connect your agent's decision-making capabilities with your frontend's interactive elements.

CopilotKit consumes AG-UI protocol events streamed by AG2 over /chat. See the AG2 AG-UI integration docs.

When should I use this?#

Frontend actions are essential when you want to create truly interactive AI applications where your agent needs to:

Dynamically update UI elements
Trigger frontend animations or transitions
Show alerts or notifications
Modify application state
Handle user interactions programmatically

Without frontend actions, agents are limited to just processing and returning data. By implementing frontend actions, you can create rich, interactive experiences where your agent actively drives the user interface.

Implementation#

Setup CopilotKit#

To use frontend actions, you'll need to setup CopilotKit first. For the sake of brevity, we won't cover it here.

Check out our getting started guide and come back here when you're setup.

Create a frontend action#

First, you'll need to create a frontend action using the useFrontendTool hook. Here's a simple one to get you started that says hello to the user.

page.tsx

import { z } from "zod";
import { useFrontendTool } from "@copilotkit/react-core/v2"

export function Page() {
  // ...

  useFrontendTool({
    name: "sayHello",
    description: "Say hello to the user",
    available: "remote", // optional, makes it so the action is *only* available to the AG2 backend over AG-UI
    parameters: z.object({
      name: z.string().describe("The name of the user to say hello to"),
    }),
    handler: async ({ name }) => {
      alert(`Hello, ${name}!`);
      return `Said hello to ${name}!`;
    },
  });

  // ...
}

Modify your agent#

Now, we'll need to modify the agent to access these frontend actions. Open your AG2 backend file and continue from there.

Setup your AG2 backend over AG-UI#

AG2 can call frontend actions through AG-UI tool events with a standard /chat endpoint:

agent.py

from fastapi import FastAPI, Header
from fastapi.responses import StreamingResponse
from autogen import ConversableAgent, LLMConfig
from autogen.ag_ui import AGUIStream, RunAgentInput

agent = ConversableAgent(
    name="assistant",
    system_message="You are a helpful assistant.",
    llm_config=LLMConfig({"model": "gpt-5.4-mini"}),
    human_input_mode="NEVER",
)

stream = AGUIStream(agent)
app = FastAPI()

@app.post("/chat")
async def run_agent(
    message: RunAgentInput,
accept: str | None = Header(None),
):
    return StreamingResponse(
        stream.dispatch(message, accept=accept),
        media_type=accept or "text/event-stream",
    )

That's it. Your AG2 backend can now receive frontend action tool definitions emitted by CopilotKit through AG-UI.

Give it a try#

You've now given your agent the ability to directly call any CopilotActions you've defined. These actions will be available as tools to the agent where they can be used as needed.