Skip to content

Can't obtain elements' ref from snapshot #90

Open
@sebas-spinnable

Description

@sebas-spinnable

I'm trying to leverage the Browserbase MCP to provide browser use capabilities to agents.
For most tools, a ref parameter is required to exactly identify the element that must be clicked, typed, etc.

This information seems to be missing from the browserbase_snapshot response. Here is an example from the Google's homepage:

{
  "content": [
    {
      "type": "text",
      "text": "Accessibility snapshot captured."
    },
    {
      "type": "text",
      "text": "\\n\\n- Page URL: https://www.google.com/\\n- Page Title: Google\\n- Page Snapshot\n```yaml\n- Page Snapshot\n```yaml\n- navigation:\n  - link \"About\":\n    - /url: https://about.google/?fg=1&utm_source=google-US&utm_medium=referral&utm_campaign=hp-header\n  - link \"Store\":\n    - /url: https://store.google.com/US?utm_source=hp_header&utm_medium=google_ooo&utm_campaign=GS100042&hl=en-US\n  - link \"Gmail\":\n    - /url: https://mail.google.com/mail/&ogbl\n  - link \"Search for Images\":\n    - /url: https://www.google.com/imghp?hl=en&ogbl\n    - text: Images\n  - button \"Google apps\":\n    - img\n  - link \"Sign in\":\n    - /url: https://accounts.google.com/ServiceLogin?hl=en&passive=true&continue=https://www.google.com/&ec=futura_exp_og_so_72776762_e\n- img\n- search:\n  - img\n  - combobox \"Search\"\n  - button \"Search by voice\":\n    - img\n  - button \"Search by image\":\n    - img\n  - button \"Google Search\"\n  - button \"I'm Feeling Lucky\"\n- contentinfo:\n  - link \"Advertising\":\n    - /url: https://www.google.com/intl/en_us/ads/?subid=ww-ww-et-g-awa-a-g_hpafoot1_1!o2&utm_source=google.com&utm_medium=referral&utm_campaign=google_hpafooter&fg=1\n  - link \"Business\":\n    - /url: https://www.google.com/services/?subid=ww-ww-et-g-awa-a-g_hpbfoot1_1!o2&utm_source=google.com&utm_medium=referral&utm_campaign=google_hpbfooter&fg=1\n  - link \"How Search works\":\n    - /url: https://google.com/search/howsearchworks/?fg=1\n  - link \"Applying AI towards science and the environment\":\n    - /url: https://ai.google/societal-impact/?utm_source=googlehpfooter&utm_medium=housepromos&utm_campaign=bottom-footer\n  - link \"Privacy\":\n    - /url: https://policies.google.com/privacy?hl=en&fg=1\n  - link \"Terms\":\n    - /url: https://policies.google.com/terms?hl=en&fg=1\n  - button \"Settings\"\n- dialog \"Sign in to Google\":\n  - text: Sign in to Google Get the most from your Google account\n  - button \"Stay signed out\"\n  - button \"Sign in\"\n```\n```\n"
    }
  ],
  "isError": false
}

Comparing this to the Playwright's MCP response to the browser_snapshot tool call:

{
  "content": [
    {
      "type": "text",
      "text": "- Ran Playwright code:\n```js\n// <internal code to capture accessibility snapshot>\n```\n\n- Page URL: https://www.google.com/\n- Page Title: Google\n- Page Snapshot\n```yaml\n- generic [ref=e2]:\n  - navigation [ref=e3]:\n    - generic [ref=e5]:\n      - generic [ref=e6]:\n        - link \"Gmail\" [ref=e126] [cursor=pointer]:\n          - /url: https://mail.google.com/mail/&ogbl\n        - link \"Search for Images\" [ref=e127] [cursor=pointer]:\n          - /url: https://www.google.com/imghp?hl=en&ogbl\n          - text: Images\n      - button \"Google apps\" [ref=e128] [cursor=pointer]:\n        - img [ref=e14] [cursor=pointer]\n      - link \"Sign in\" [ref=e129] [cursor=pointer]:\n        - /url: https://accounts.google.com/ServiceLogin?hl=en&passive=true&continue=https://www.google.com/&ec=futura_exp_og_so_72776762_e\n        - generic [ref=e19] [cursor=pointer]: Sign in\n  - img [ref=e22]\n  - search [ref=e30]:\n    - generic [ref=e32]:\n      - generic [ref=e34]:\n        - img [ref=e38]\n        - combobox \"Search\" [ref=e131]\n        - generic [ref=e42]:\n          - button \"Search by voice\" [ref=e132] [cursor=pointer]:\n            - img [ref=e45] [cursor=pointer]\n          - button \"Search by image\" [ref=e133] [cursor=pointer]:\n            - img [ref=e48] [cursor=pointer]\n      - generic [ref=e51]:\n        - button \"Google Search\" [ref=e135] [cursor=pointer]\n        - button \"I'm Feeling Lucky\" [ref=e136] [cursor=pointer]\n  - generic [ref=e56]:\n    - text: \"Google offered in:\"\n    - link \"Português (Portugal)\" [ref=e145] [cursor=pointer]:\n      - /url: https://www.google.com/setprefs?sig=0__ZPLLogYu3boAsSEPZFrhvsFZXM%3D&hl=pt-PT&source=homepage&sa=X&ved=0ahUKEwjTx77_gIiOAxWUX0EAHeGnLzkQ2ZgBCBU\n  - contentinfo [ref=e59]:\n    - generic [ref=e60]: Portugal\n    - generic [ref=e61]:\n      - generic [ref=e62]:\n        - link \"About\" [ref=e147] [cursor=pointer]:\n          - /url: https://about.google/?utm_source=google-PT&utm_medium=referral&utm_campaign=hp-footer&fg=1\n        - link \"Advertising\" [ref=e148] [cursor=pointer]:\n          - /url: https://www.google.com/intl/en_pt/ads/?subid=ww-ww-et-g-awa-a-g_hpafoot1_1!o2&utm_source=google.com&utm_medium=referral&utm_campaign=google_hpafooter&fg=1\n        - link \"Business\" [ref=e149] [cursor=pointer]:\n          - /url: https://www.google.com/services/?subid=ww-ww-et-g-awa-a-g_hpbfoot1_1!o2&utm_source=google.com&utm_medium=referral&utm_campaign=google_hpbfooter&fg=1\n        - link \"How Search works\" [ref=e150] [cursor=pointer]:\n          - /url: https://google.com/search/howsearchworks/?fg=1\n      - generic [ref=e67]:\n        - link \"Privacy\" [ref=e151] [cursor=pointer]:\n          - /url: https://policies.google.com/privacy?hl=en-PT&fg=1\n        - link \"Terms\" [ref=e152] [cursor=pointer]:\n          - /url: https://policies.google.com/terms?hl=en-PT&fg=1\n        - button \"Settings\" [ref=e153] [cursor=pointer]:\n          - generic [ref=e74] [cursor=pointer]: Settings\n```"
    }
  ]
}

One can see that the exact references are missing from the browserbase's response.
This renders most tools obsolete since there isn't a valid ref to point to each element

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions