Neon’s Post

𝗛𝗼𝘄 𝗮𝗿𝗲 𝘆𝗼𝘂 𝘄𝗿𝗶𝘁𝗶𝗻𝗴 𝘁𝗲𝘀𝘁𝘀 (𝗲𝘃𝗮𝗹𝘀) 𝗳𝗼𝗿 𝘆𝗼𝘂𝗿 𝗠𝗖𝗣 𝗦𝗲𝗿𝘃𝗲𝗿𝘀? David Gomes and Pedro Figueiredo wrote up our current approach using Braintrust which helped us take advanced tool use from 60% to 100% success rate👇 Context: As we added tools to the Neon MCP Server, we found LLM's were not consistently picking the right ones. We have some more advanced tools like prepare_database_migration and complete_database_migration tools defined in the MCP Server, but sometimes the LLM will fall back to generic SQL. We needed a way to validate that we are improving the success rate as we edited the prompts that define each tool. So we wrote evals for our MCP server using Braintrust. With those we were able to iterate on the text prompts for advanced tools and get them from 60% to 100% success rate. Full Write-up in comments