DEV Community

Cover image for 🧠🥷How to make AI controled Avatar 2 (Vroid MCP + Cline and Cursor + Unity)
Web Developer Hyper
Web Developer Hyper

Posted on • Edited on

🧠🥷How to make AI controled Avatar 2 (Vroid MCP + Cline and Cursor + Unity)

#ai #vscode #chatgpt #webdev

Update (2025/04/20): Added How to make VRoid MCP section

Intro

Hello! I'm a Ninja Web Developer. Hi-Yah!🥷
These days, I am absolutely into MCP and AI.🥰
🧠🥷How to use MCP in Cline and Cursor
🧠🥷How to use MCP in Cline and Cursor 2 (use under Proxy)
🧠🥷How to use MCP in Cline and Cursor 3 (use under Proxy 2)
🧠🥷How to make MCP (Cline and Cursor)
🧠🤖AI coding agent 1 (Cline + Cursor)
🧠🥷How to make cool Ninja (Blender MCP (Cline and Cursor))
🧠🥷How to make cool Ninja game (Unity MCP + Blender MCP (Cline and Cursor))
🧠🥷MCP Security (choose safe MCP and check MCP safety)🛡️
I want to control my VRoid Avatar expression in Unity by the output of AI.
Last time, I wrote about VRoid part.
🧠🥷How to make AI controled Avatar 1 (VRoid)
So this time, I will write about AI part.
There are a lot of ways to control AI, but as I am studying MCP these days, so, I will control AI using MCP.
Let's start!🚀

System outline

I wanted to make the system as simple as possible, and made a three parts structure.
AI (Cline or Cursor)　→　MCP　→　Avatar (Unity + VRoid)
First part is communication with AI, and made this at rule of Cline and Cursor.
Second part is connection between AI and Unity, and I made this with MCP (TypeScript code).
Third part is control of Unity, and I made this in Unity (C# code).

Avatar control in Unity

First, I made the Unity part.
VRoid 3D models have expression by default.
Also, you can edit the expression as you like if you want.
I made a C# code that expression changes with the pressed key.
This table shows which key correspond to which expression.↓

Key Expression
h happy
s sad
a angry
p surprised
r relaxed
n neutral

Here is the video of how the expression changes by keys.↓

Transmission between AI and Unity

Second, I made the connection between AI and Unity using MCP.
Please refer my previous post how to make MCP.
🧠🥷How to make MCP (Cline and Cursor)
There are several ways to transmit, but I chose Websocket, because it is often used for realtime transmission for chating.
Unity uses C#, so I used Websocket-sharp in Unity.

How to set Websocket-sharp

1️⃣ Download Websocket-sharp.
https://github.com/sta/websocket-sharp
2️⃣ Open the .sln file with Visual Studio.
3️⃣ Delete all the Example folder
4️⃣ Change the build type to Release, and Build > Solution Build
5️⃣ websocket-sharp.dll is made in \websocket-sharp\bin\Release.
6️⃣ Drug and Drop the dll in your Unity.
7️⃣ OK, ready for Websocket-sharp.🚀

AI control in Cline and Cursor

Third, I made the controller of AI using rule of Cline and Cursor.
I wrote the rule that AI classify the chat content to particular expression, and send it to Unity using MCP.
The expression tends to be classified as "neutral", so I wrote the rule not to classify as "neutral" as possible.
Also, use Act mode in Cline and Agent mode in Cursor when using VRoid MCP, because Plan mode in Cline and Ask mode in Cursor cannot send requests to Unity.

How to make VRoid MCP

These codes worked in my environment, but they are made by AI, and not refactored nor considered well.
Please think them just as a sample.🙇
1️⃣ Make a folder for VRoid MCP and open it from your editor.

2️⃣ Make package.json.↓

npm init
Enter fullscreen mode Exit fullscreen mode

3️⃣ Install MCP SDK.↓

npm install @modelcontextprotocol/sdk
Enter fullscreen mode Exit fullscreen mode

4️⃣ Install Websocket.↓

npm install @types/ws
Enter fullscreen mode Exit fullscreen mode

5️⃣ Make tsconfig.json.↓

tsc --init
Enter fullscreen mode Exit fullscreen mode

6️⃣ Add "build": "tsc", to scripts of package.json.

7️⃣ Add index.ts(Typescript) of VRoid MCP.↓

#!/usr/bin/env node
import { Server } from "@modelcontextprotocol/sdk/server/index.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import {
  CallToolRequestSchema,
  ErrorCode,
  ListToolsRequestSchema,
  McpError,
} from "@modelcontextprotocol/sdk/types.js";
import WebSocket from "ws";

class VroidMCP {
  private server: Server;
  private ws: WebSocket | null = null;

  constructor() {
    this.server = new Server(
      {
        name: "VroidMCP",
        version: "0.1.0",
      },
      {
        capabilities: {
          resources: {},
          tools: {},
        },
      }
    );

    this.setupToolHandlers();

    this.connectWebSocket();

    this.server.onerror = (error) => console.error("[MCP Error]", error);
    process.on("SIGINT", async () => {
      await this.server.close();
      process.exit(0);
    });
  }

  private connectWebSocket() {
    const connect = () => {
      this.ws = new WebSocket("ws://localhost:8080");

      this.ws.onopen = () => {
        console.log("Connected to Unity WebSocket server");
      };

      this.ws.onmessage = (event) => {
        console.log("Received message from Unity: ", event.data);
      };

      this.ws.onclose = () => {
        console.log("Disconnected from Unity WebSocket server");
        setTimeout(connect, 3000);
      };

      this.ws.onerror = (error) => {
        console.error("WebSocket error: ", error);
        this.ws?.close();
      };
    };

    connect();
  }

  /* MCP Tools enable servers to expose executable functionality to the system. Through these tools, you can interact with external systems, perform computations, and take actions in the real world.
   * - Like resources, tools are identified by unique names and can include descriptions to guide their usage. However, unlike resources, tools represent dynamic operations that can modify state or interact with external systems.
   * - While resources and tools are similar, you should prefer to create tools over resources when possible as they provide more flexibility.
   */
  private setupToolHandlers() {
    this.server.setRequestHandler(ListToolsRequestSchema, async () => ({
      tools: [
        {
          name: "send_key_operation",
          description: "Sends a key operation to the Unity project.",
          inputSchema: {
            type: "object",
            properties: {
              key: {
                type: "string",
                description: "The key to send.",
              },
              action: {
                type: "string",
                description: "The action to perform (e.g., press, release).",
              },
            },
            required: ["key", "action"],
          },
        },
      ],
    }));

    this.server.setRequestHandler(CallToolRequestSchema, async (request) => {
      if (request.params.name === "send_key_operation") {
        const { key, action } = request.params.arguments as {
          key: string;
          action: string;
        };

        if (!this.ws) {
          throw new McpError(
            ErrorCode.InternalError,
            "WebSocket connection not established"
          );
        }

        setTimeout(() => {
          if (this.ws && this.ws.readyState === WebSocket.OPEN) {
            const message = `${key}_${action}`;
            this.ws.send(message);
            console.log(`Sent key operation: ${message}`);
          } else {
            console.log("WebSocket connection not open.");
          }
        }, 1000);

        return {
          content: [{ type: "text", text: `Sending key operation...` }],
        };
      } else {
        throw new McpError(
          ErrorCode.MethodNotFound,
          `Unknown tool: ${request.params.name}`
        );
      }
    });
  }

  async run() {
    const transport = new StdioServerTransport();
    await this.server.connect(transport);
    console.error("Vroid MCP server running on stdio");
  }
}

const server = new VroidMCP();
server.run().catch(console.error);
Enter fullscreen mode Exit fullscreen mode

8️⃣ Build index.ts to index.js.↓

run npm build
Enter fullscreen mode Exit fullscreen mode

9️⃣ Set cline_mcp_settings.json for Cline and mcp.json for Cursor.↓

{
  "mcpServers": {
    "VroidMCP": {
      "command": "node",
      "args": ["path to index.js"]
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

1️⃣0️⃣ Add C# code to VRoid in Unity.↓

using UnityEngine;
using UniVRM10;
using WebSocketSharp;
using WebSocketSharp.Server;

public class VroidExpressionController : MonoBehaviour
{
    private static Vrm10Instance vrm;
    private WebSocketServer wsServer;

    void Start()
    {
        vrm = GetComponent<Vrm10Instance>();

        wsServer = new WebSocketServer("ws://localhost:8080");
        wsServer.AddWebSocketService<KeyOperationHandler>("/");
        wsServer.Start();

        Debug.Log("WebSocket server started on ws://localhost:8080");

        Application.runInBackground = true;
    }

    void OnDestroy()
    {
        if (wsServer != null && wsServer.IsListening)
        {
            wsServer.Stop();
            Debug.Log("WebSocket server stopped");
        }
    }

    public class KeyOperationHandler : WebSocketBehavior
    {
        protected override void OnMessage(MessageEventArgs e)
        {
            Debug.Log("Received WebSocket message: " + e.Data);
            var expression = vrm.Runtime.Expression;

            void ResetAllExpressions()
            {
                foreach (var preset in System.Enum.GetValues(typeof(UniVRM10.ExpressionPreset)))
                {
                    if ((UniVRM10.ExpressionPreset)preset != UniVRM10.ExpressionPreset.custom)
                    {
                        expression.SetWeight(ExpressionKey.CreateFromPreset((UniVRM10.ExpressionPreset)preset), 0.0f);
                    }
                }
            }

            if (e.Data == "H_press")
            {
                ResetAllExpressions();
                expression.SetWeight(ExpressionKey.CreateFromPreset(UniVRM10.ExpressionPreset.happy), 1.0f);
            }
            else if (e.Data == "A_press")
            {
                ResetAllExpressions();
                expression.SetWeight(ExpressionKey.CreateFromPreset(UniVRM10.ExpressionPreset.angry), 1.0f);
            }
            else if (e.Data == "R_press")
            {
                ResetAllExpressions();
                expression.SetWeight(ExpressionKey.CreateFromPreset(UniVRM10.ExpressionPreset.relaxed), 1.0f);
            }
            else if (e.Data == "P_press")
            {
                ResetAllExpressions();
                expression.SetWeight(ExpressionKey.CreateFromPreset(UniVRM10.ExpressionPreset.surprised), 1.0f);
            }
            else if (e.Data == "S_press")
            {
                ResetAllExpressions();
                expression.SetWeight(ExpressionKey.CreateFromPreset(UniVRM10.ExpressionPreset.sad), 1.0f);
            }
             else if (e.Data == "N_press")
             {
                 ResetAllExpressions();
             }
            else
            {
                Debug.Log("Received unknown message: " + e.Data);
            }
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

1️⃣1️⃣ Set Rule of Cline and Cursor.↓

First, Classify the input to "happy", "angry", "relaxed", "surprised", "sad" and "neutral".
Do not classify as "neutral" as possible.
If the input is "happy" send data below "using send_key_operation" of VroidMCP.
{
  "key": "H",
  "action": "press"
}
If the input is "angry" send data below using "send_key_operation" of VroidMCP.
{
  "key": "A",
  "action": "press"
}
If the input is "relaxed" send key data below using "send_key_operation" of VroidMCP.
{
  "key": "R",
  "action": "press"
}
If the input is "surprised" send data below using "send_key_operation" of VroidMCP.
{
  "key": "P",
  "action": "press"
}
If the input is "sad" send data below using "send_key_operation" of VroidMCP.
{
  "key": "S",
  "action": "press"
}
If the input is "neutral" send data below using "send_key_operation" of VroidMCP.
{
  "key": "N",
  "action": "press"
}
Enter fullscreen mode Exit fullscreen mode

1️⃣2️⃣ Hooray! Ready for VRoid MCP!🎉

VRoid Codes

This is a sample code for changing the expression of VRoid in Unity without AI control and MCP.
When you press a key, the VRoid expression in Unity will change.
If you only want to try VRoid expression changing in Unity, use this code instead of the codes above.

using UnityEngine;
using UniVRM10;

public class VroidExpressionController : MonoBehaviour
{
    private Vrm10Instance vrm;

    void Start()
    {
        vrm = GetComponent<Vrm10Instance>();
    }

    void Update()
    {
        if (vrm != null)
        {
            Debug.Log("vrm is not null");
            var expression = vrm.Runtime.Expression;

            void ResetAllExpressions()
            {
                foreach (var preset in System.Enum.GetValues(typeof(UniVRM10.ExpressionPreset)))
                {
                    if ((UniVRM10.ExpressionPreset)preset != UniVRM10.ExpressionPreset.custom)
                    {
                        expression.SetWeight(ExpressionKey.CreateFromPreset((UniVRM10.ExpressionPreset)preset), 0.0f);
                    }
                }
            }

            if (Input.GetKeyDown(KeyCode.H))
            {
                ResetAllExpressions();
                expression.SetWeight(ExpressionKey.CreateFromPreset(UniVRM10.ExpressionPreset.happy), 1.0f);
            }
            if (Input.GetKeyDown(KeyCode.A))
            {
                ResetAllExpressions();
                expression.SetWeight(ExpressionKey.CreateFromPreset(UniVRM10.ExpressionPreset.angry), 1.0f);
            }
            if (Input.GetKeyDown(KeyCode.R))
            {
                ResetAllExpressions();
                expression.SetWeight(ExpressionKey.CreateFromPreset(UniVRM10.ExpressionPreset.relaxed), 1.0f);
            }
            if (Input.GetKeyDown(KeyCode.P))
            {
                ResetAllExpressions();
                expression.SetWeight(ExpressionKey.CreateFromPreset(UniVRM10.ExpressionPreset.surprised), 1.0f);
            }
            if (Input.GetKeyDown(KeyCode.S))
            {
                ResetAllExpressions();
                expression.SetWeight(ExpressionKey.CreateFromPreset(UniVRM10.ExpressionPreset.sad), 1.0f);
            }
            if (Input.GetKeyDown(KeyCode.N))
            {
                ResetAllExpressions();
            }
        }
        else
        {
            Debug.Log("vrm is null");
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Improvement of the system

I only change the expression of the Avatar.
There are other things we can do such as change the moves of the Avatar body by the AI output.
It's up to our imagination and creativity!💡

Outro

I was trying to apply MCP for work, but end up applying it for fun.
However, I could learn that it is easy to expand the usage of AI using MCP.
Next time I should try to use MCP for work, but not for fun.
I would be happy if you learn something from this post.
Thank you for reading.
Happy AI coding!🤖 Hi-Yah!🥷

Update (2025/05/04):Wrote about Image generation and editing MCP
🧠🥷How to make Image generation and editing MCP (Gemini API + Cline and Cursor)

Update (2025/05/10): Wrote about Meme Generating MCP
🧠🥷How to make Meme Generating MCP (Cline and Cursor)

Update (2025/05/17): Wrote about Ver 3.0 of Image generation and editing MCP
🧠🥷MCP Transports (Image generation and editing MCP Ver 3.0 (WebSocket + Next.js + Gemini API + Cline and Cursor))

Top comments (0)