Open
Description
I'm trying to leverage the Browserbase MCP to provide browser use capabilities to agents.
For most tools, a ref
parameter is required to exactly identify the element that must be clicked, typed, etc.
This information seems to be missing from the browserbase_snapshot
response. Here is an example from the Google's homepage:
{
"content": [
{
"type": "text",
"text": "Accessibility snapshot captured."
},
{
"type": "text",
"text": "\\n\\n- Page URL: https://www.google.com/\\n- Page Title: Google\\n- Page Snapshot\n```yaml\n- Page Snapshot\n```yaml\n- navigation:\n - link \"About\":\n - /url: https://about.google/?fg=1&utm_source=google-US&utm_medium=referral&utm_campaign=hp-header\n - link \"Store\":\n - /url: https://store.google.com/US?utm_source=hp_header&utm_medium=google_ooo&utm_campaign=GS100042&hl=en-US\n - link \"Gmail\":\n - /url: https://mail.google.com/mail/&ogbl\n - link \"Search for Images\":\n - /url: https://www.google.com/imghp?hl=en&ogbl\n - text: Images\n - button \"Google apps\":\n - img\n - link \"Sign in\":\n - /url: https://accounts.google.com/ServiceLogin?hl=en&passive=true&continue=https://www.google.com/&ec=futura_exp_og_so_72776762_e\n- img\n- search:\n - img\n - combobox \"Search\"\n - button \"Search by voice\":\n - img\n - button \"Search by image\":\n - img\n - button \"Google Search\"\n - button \"I'm Feeling Lucky\"\n- contentinfo:\n - link \"Advertising\":\n - /url: https://www.google.com/intl/en_us/ads/?subid=ww-ww-et-g-awa-a-g_hpafoot1_1!o2&utm_source=google.com&utm_medium=referral&utm_campaign=google_hpafooter&fg=1\n - link \"Business\":\n - /url: https://www.google.com/services/?subid=ww-ww-et-g-awa-a-g_hpbfoot1_1!o2&utm_source=google.com&utm_medium=referral&utm_campaign=google_hpbfooter&fg=1\n - link \"How Search works\":\n - /url: https://google.com/search/howsearchworks/?fg=1\n - link \"Applying AI towards science and the environment\":\n - /url: https://ai.google/societal-impact/?utm_source=googlehpfooter&utm_medium=housepromos&utm_campaign=bottom-footer\n - link \"Privacy\":\n - /url: https://policies.google.com/privacy?hl=en&fg=1\n - link \"Terms\":\n - /url: https://policies.google.com/terms?hl=en&fg=1\n - button \"Settings\"\n- dialog \"Sign in to Google\":\n - text: Sign in to Google Get the most from your Google account\n - button \"Stay signed out\"\n - button \"Sign in\"\n```\n```\n"
}
],
"isError": false
}
Comparing this to the Playwright's MCP response to the browser_snapshot
tool call:
{
"content": [
{
"type": "text",
"text": "- Ran Playwright code:\n```js\n// <internal code to capture accessibility snapshot>\n```\n\n- Page URL: https://www.google.com/\n- Page Title: Google\n- Page Snapshot\n```yaml\n- generic [ref=e2]:\n - navigation [ref=e3]:\n - generic [ref=e5]:\n - generic [ref=e6]:\n - link \"Gmail\" [ref=e126] [cursor=pointer]:\n - /url: https://mail.google.com/mail/&ogbl\n - link \"Search for Images\" [ref=e127] [cursor=pointer]:\n - /url: https://www.google.com/imghp?hl=en&ogbl\n - text: Images\n - button \"Google apps\" [ref=e128] [cursor=pointer]:\n - img [ref=e14] [cursor=pointer]\n - link \"Sign in\" [ref=e129] [cursor=pointer]:\n - /url: https://accounts.google.com/ServiceLogin?hl=en&passive=true&continue=https://www.google.com/&ec=futura_exp_og_so_72776762_e\n - generic [ref=e19] [cursor=pointer]: Sign in\n - img [ref=e22]\n - search [ref=e30]:\n - generic [ref=e32]:\n - generic [ref=e34]:\n - img [ref=e38]\n - combobox \"Search\" [ref=e131]\n - generic [ref=e42]:\n - button \"Search by voice\" [ref=e132] [cursor=pointer]:\n - img [ref=e45] [cursor=pointer]\n - button \"Search by image\" [ref=e133] [cursor=pointer]:\n - img [ref=e48] [cursor=pointer]\n - generic [ref=e51]:\n - button \"Google Search\" [ref=e135] [cursor=pointer]\n - button \"I'm Feeling Lucky\" [ref=e136] [cursor=pointer]\n - generic [ref=e56]:\n - text: \"Google offered in:\"\n - link \"Português (Portugal)\" [ref=e145] [cursor=pointer]:\n - /url: https://www.google.com/setprefs?sig=0__ZPLLogYu3boAsSEPZFrhvsFZXM%3D&hl=pt-PT&source=homepage&sa=X&ved=0ahUKEwjTx77_gIiOAxWUX0EAHeGnLzkQ2ZgBCBU\n - contentinfo [ref=e59]:\n - generic [ref=e60]: Portugal\n - generic [ref=e61]:\n - generic [ref=e62]:\n - link \"About\" [ref=e147] [cursor=pointer]:\n - /url: https://about.google/?utm_source=google-PT&utm_medium=referral&utm_campaign=hp-footer&fg=1\n - link \"Advertising\" [ref=e148] [cursor=pointer]:\n - /url: https://www.google.com/intl/en_pt/ads/?subid=ww-ww-et-g-awa-a-g_hpafoot1_1!o2&utm_source=google.com&utm_medium=referral&utm_campaign=google_hpafooter&fg=1\n - link \"Business\" [ref=e149] [cursor=pointer]:\n - /url: https://www.google.com/services/?subid=ww-ww-et-g-awa-a-g_hpbfoot1_1!o2&utm_source=google.com&utm_medium=referral&utm_campaign=google_hpbfooter&fg=1\n - link \"How Search works\" [ref=e150] [cursor=pointer]:\n - /url: https://google.com/search/howsearchworks/?fg=1\n - generic [ref=e67]:\n - link \"Privacy\" [ref=e151] [cursor=pointer]:\n - /url: https://policies.google.com/privacy?hl=en-PT&fg=1\n - link \"Terms\" [ref=e152] [cursor=pointer]:\n - /url: https://policies.google.com/terms?hl=en-PT&fg=1\n - button \"Settings\" [ref=e153] [cursor=pointer]:\n - generic [ref=e74] [cursor=pointer]: Settings\n```"
}
]
}
One can see that the exact references are missing from the browserbase
's response.
This renders most tools obsolete since there isn't a valid ref
to point to each element
Metadata
Metadata
Assignees
Labels
No labels