DEV Community

Cover image for Explore generating tests with Playwright MCP Server and Claude 4 Sonnet and Claude 4 Opus
idavidov13
idavidov13

Posted on • Edited on

Explore generating tests with Playwright MCP Server and Claude 4 Sonnet and Claude 4 Opus

#playwright #mcp #qa #ai

Introduction

With announcement of Claude 4 Sonnet and Claude 4 Opus, I decided to add the two model into our experiment. The input was the same as the input for other LLMs.

Prerequisites

  1. Claude 4 Opus/Sonnet is selected as a model
  2. The following prompt is run: 
Create a test case utilizing provided constants for navigating to the web app, login, create/edit/delete an article. Try to verify the result after every major step. Use provided instructions
Enter fullscreen mode Exit fullscreen mode

Generated Test Case with POM

Claude 4 Opus

import { test, expect, Page } from '@playwright/test';

const url = process.env.URL!;
const email = process.env.EMAIL!;
const password = process.env.PASSWORD!;
const articleTitle = `Test Article for Playwright MCP Server ${Date.now()}`;
const articleAbout = 'This is a test article about Playwright MCP Server';
const articleContent = 'This article is created by Playwright MCP Server test automation';
const updatedArticleTitle = `Updated ${articleTitle}`;
const updatedArticleAbout = 'Updated article about Playwright MCP Server';
const updatedArticleContent = 'This article has been updated by Playwright MCP Server';

class ArticlePage {
  constructor(private page: Page) {}

  get nav() {
    return {
      signIn: this.page.getByRole('link', { name: 'Sign in' }),
      newArticle: this.page.getByRole('link', { name: /New Article/ }),
      username: (username: string) => this.page.getByRole('link', { name: username }),
      home: this.page.getByRole('link', { name: 'Home' }),
    };
  }

  get loginForm() {
    return {
      email: this.page.getByRole('textbox', { name: 'Email' }),
      password: this.page.getByRole('textbox', { name: 'Password' }),
      submit: this.page.getByRole('button', { name: 'Sign in' }),
    };
  }

  get editorForm() {
    return {
      title: this.page.getByRole('textbox', { name: 'Article Title' }),
      about: this.page.getByRole('textbox', { name: "What's this article about?" }),
      content: this.page.getByRole('textbox', { name: /Write your article/ }),
      publish: this.page.getByRole('button', { name: 'Publish Article' }),
    };
  }

  get article() {
    return {
      title: (title: string) => this.page.getByRole('heading', { name: title }),
      edit: this.page.getByRole('link', { name: /Edit Article/ }).first(),
      delete: this.page.getByRole('button', { name: /Delete Article/ }).first(),
    };
  }

  async login(email: string, password: string, username: string) {
    await this.nav.signIn.click();
    await expect(this.page).toHaveURL(/.*login/);
    await this.loginForm.email.fill(email);
    await this.loginForm.password.fill(password);
    await this.loginForm.submit.click();
    await expect(this.nav.username(username)).toBeVisible({ timeout: 10000 });
  }

  async createArticle(title: string, about: string, content: string) {
    await this.nav.newArticle.click();
    await expect(this.page).toHaveURL(/.*editor/);
    await this.editorForm.title.fill(title);
    await this.editorForm.about.fill(about);
    await this.editorForm.content.fill(content);
    await this.editorForm.publish.click();
    await expect(this.article.title(title)).toBeVisible({ timeout: 10000 });
  }

  async editArticle(updatedTitle: string, updatedAbout: string, updatedContent: string) {
    await this.article.edit.click();
    await expect(this.page).toHaveURL(/.*editor/);
    await this.editorForm.title.clear();
    await this.editorForm.title.fill(updatedTitle);
    await this.editorForm.about.clear();
    await this.editorForm.about.fill(updatedAbout);
    await this.editorForm.content.clear();
    await this.editorForm.content.fill(updatedContent);
    await this.editorForm.publish.click();
    await expect(this.article.title(updatedTitle)).toBeVisible({ timeout: 10000 });
  }

  async deleteArticle() {
    await this.article.delete.click();
    await this.page.waitForLoadState('networkidle');
  }
}

test.describe('Articles Management E2E', () => {
  test('should login, create, edit, and delete an article', async ({ page }) => {
    const app = new ArticlePage(page);

    // Navigate to the application
    await page.goto(url);
    await expect(page).toHaveTitle(/Conduit/);

    // Login
    const username = email.split('@')[0];
    await app.login(email, password, username);

    // Verify login was successful
    await expect(app.nav.username(username)).toBeVisible();

    // Create a new article
    await app.createArticle(articleTitle, articleAbout, articleContent);

    // Verify article was created
    await expect(page).toHaveURL(/.*article/);
    await expect(app.article.title(articleTitle)).toBeVisible();

    // Edit the article
    await app.editArticle(updatedArticleTitle, updatedArticleAbout, updatedArticleContent);

    // Verify article was updated
    await expect(app.article.title(updatedArticleTitle)).toBeVisible();
    await expect(page.getByText(updatedArticleContent)).toBeVisible();

    // Delete the article
    await app.deleteArticle();

    // Verify article was deleted by checking user profile
    await page.goto(`${url}profile/${username}`);
    await expect(page.getByText('No articles are here... yet.')).toBeVisible({ timeout: 10000 });
  });
});
Enter fullscreen mode Exit fullscreen mode

Claude 4 Sonnet

import { test, expect, Page } from '@playwright/test';

const url = process.env.URL!;
const email = process.env.EMAIL!;
const password = process.env.PASSWORD!;
const articleTitle = `Test Article for Claude Sonnet ${Date.now()}`;
const articleAbout = 'This is a test article created by Claude Sonnet';
const articleContent = 'This article demonstrates the capabilities of Claude Sonnet in test automation';
const updatedArticleTitle = `Updated ${articleTitle}`;
const updatedArticleAbout = 'Updated article about Claude Sonnet capabilities';
const updatedArticleContent = 'This article has been updated to showcase Claude Sonnet improvements';

class ArticlePage {
  constructor(private page: Page) {}

  get nav() {
    return {
      signIn: this.page.getByRole('link', { name: 'Sign in' }),
      newArticle: this.page.getByRole('link', { name: /New Article/ }),
      username: (username: string) => this.page.getByRole('link', { name: username }),
      home: this.page.getByRole('link', { name: 'Home' }),
    };
  }

  get loginForm() {
    return {
      email: this.page.getByRole('textbox', { name: 'Email' }),
      password: this.page.getByRole('textbox', { name: 'Password' }),
      submit: this.page.getByRole('button', { name: 'Sign in' }),
    };
  }

  get editorForm() {
    return {
      title: this.page.getByRole('textbox', { name: 'Article Title' }),
      about: this.page.getByRole('textbox', { name: "What's this article about?" }),
      content: this.page.getByRole('textbox', { name: /Write your article/ }),
      publish: this.page.getByRole('button', { name: 'Publish Article' }),
    };
  }

  get article() {
    return {
      title: (title: string) => this.page.getByRole('heading', { name: title }),
      edit: this.page.getByRole('link', { name: /Edit Article/ }).first(),
      delete: this.page.getByRole('button', { name: /Delete Article/ }).first(),
    };
  }

  async login(email: string, password: string, username: string) {
    await this.nav.signIn.click();
    await expect(this.page).toHaveURL(/.*login/);
    await this.loginForm.email.fill(email);
    await this.loginForm.password.fill(password);
    await this.loginForm.submit.click();
    await expect(this.nav.username(username)).toBeVisible({ timeout: 10000 });
  }

  async createArticle(title: string, about: string, content: string) {
    await this.nav.newArticle.click();
    await expect(this.page).toHaveURL(/.*editor/);
    await this.editorForm.title.fill(title);
    await this.editorForm.about.fill(about);
    await this.editorForm.content.fill(content);
    await this.editorForm.publish.click();
    await expect(this.article.title(title)).toBeVisible({ timeout: 10000 });
  }

  async editArticle(updatedTitle: string, updatedAbout: string, updatedContent: string) {
    await this.article.edit.click();
    await expect(this.page).toHaveURL(/.*editor/);
    await this.editorForm.title.clear();
    await this.editorForm.title.fill(updatedTitle);
    await this.editorForm.about.clear();
    await this.editorForm.about.fill(updatedAbout);
    await this.editorForm.content.clear();
    await this.editorForm.content.fill(updatedContent);
    await this.editorForm.publish.click();
    await expect(this.article.title(updatedTitle)).toBeVisible({ timeout: 10000 });
  }

  async deleteArticle() {
    await this.article.delete.click();
    await this.page.waitForLoadState('networkidle');
  }
}

test.describe('Claude Sonnet - Articles Management E2E', () => {
  test('should navigate, login, create, edit, and delete an article with verification', async ({ page }) => {
    const app = new ArticlePage(page);

    // Step 1: Navigate to the application
    await page.goto(url);
    await expect(page).toHaveTitle(/Conduit/);

    // Step 2: Login with verification
    const username = email.split('@')[0];
    await app.login(email, password, username);

    // Verify login was successful
    await expect(app.nav.username(username)).toBeVisible();
    await expect(app.nav.newArticle).toBeVisible();

    // Step 3: Create a new article with verification
    await app.createArticle(articleTitle, articleAbout, articleContent);

    // Verify article was created successfully
    await expect(page).toHaveURL(/.*article/);
    await expect(app.article.title(articleTitle)).toBeVisible();
    await expect(page.getByText(articleContent)).toBeVisible();

    // Step 4: Edit the article with verification
    await app.editArticle(updatedArticleTitle, updatedArticleAbout, updatedArticleContent);

    // Verify article was updated successfully
    await expect(app.article.title(updatedArticleTitle)).toBeVisible();
    await expect(page.getByText(updatedArticleContent)).toBeVisible();

    // Step 5: Delete the article with verification
    await app.deleteArticle();

    // Verify article was deleted by checking user profile
    await page.goto(`${url}profile/${username}`);
    await expect(page.getByText('No articles are here... yet.')).toBeVisible({ timeout: 10000 });
  });
});
Enter fullscreen mode Exit fullscreen mode

NOTE The provided example was generated from the first time and no updates were made

Pros of the selected pattern

  1. Lazy Evaluation: Locators are instantiated only when accessed, ensuring they always point to the current DOM state.
  2. Centralized: All locators are organized within a single class/object, making them easy to locate, maintain, and update.
  3. Encapsulation: Logic and assertions can be added to locator getters, providing flexibility and improved test reliability.
  4. IntelliSense: Modern editors provide excellent auto-completion and navigation for class-based locators, improving developer experience.

Cons of the selected pattern

  1. Performance: Each access generates a new locator instance; while Playwright locators are lightweight, this could add minor overhead in large-scale tests.
  2. Inheritance: Overriding or extending locator getters in subclasses can be less straightforward compared to using fields or plain objects.

Conclusion

There is little to no differences between both outcomes. In fact, the results are pretty similar to the result of GPT-4.1.

Comparison between Claude 4 Opus, Claude 4 Sonnet and GPT-4.1

Common Patterns

Class-based Page Object Model:

All three files define an ArticlePage class that encapsulates page interactions and locators using getters for navigation, forms, and article actions.

Lazy Locators:

Locators are exposed as getters, ensuring up-to-date references to DOM elements (lazy evaluation).

Action Methods:

Each class provides methods for login, article creation, editing, and deletion, using the encapsulated locators.

Test Flow:

Each test file has a single E2E test that:

  1. Navigates to the app
  2. Logs in
  3. Creates an article
  4. Edits the article
  5. Deletes the article
  6. Verifies deletion
Use of Playwright Best Practices:
  1. Role-based locators (getByRole)
  2. Web-first assertions (toBeVisible, toHaveURL, etc.)
  3. No hardcoded timeouts except for explicit waits

Differences

Aspect claude-4-Opus.spec.ts claude-4-Sonnet.spec.ts gpt-4.1.spec.ts
Test Description Generic E2E for articles E2E with more step-by-step checks Generic E2E for articles
ArticlePage Structure Has nav, loginForm, editorForm, article getters with nested functions for dynamic locators Nearly identical to Opus, but with more explicit verification steps in the test Slightly simplified; nav getter lacks home link, and some verification steps are omitted
Assertions Checks URLs, visibility, and content after each action More granular: checks for navigation, login, creation, editing, deletion with explicit comments Fewer assertions, especially after login and during edit/delete
Test Data Uses Playwright MCP Server branding in article content Uses Claude Sonnet branding in article content Uses generic/shorter content and updated fields
Locator Patterns All use role-based locators, dynamic getter for username Identical pattern Identical, but slightly less comprehensive in navigation locators
Cleanup/Verification After deletion, checks profile for absence of articles Same as Opus Same as Opus, but with less robust assertion (no timeout on final expect)
Comments/Readability Moderate inline comments More step-by-step comments Fewer comments

Key takeaways

  1. All three tests use a modern, maintainable Playwright pattern with class-based encapsulation and lazy locators.

  2. The main differences are in test data, assertion thoroughness, and the level of documentation/comments.

  3. The claude-4-Sonnet test is the most explicit and robust in terms of stepwise verification and comments.

  4. The gpt-4.1 test is the most minimal, which could make it less robust for regression testing but easier to maintain for simple flows.

What's next?

Stay tuned for next article in which we will make a comparison between created POMs/tests.
Please, do not hesitate to start conversation regarding the test or it's result.

🙏🏻 Thank you for reading! Building robust, scalable automation frameworks is a journey best taken together. If you found this article helpful, consider joining a growing community of QA professionals 🚀 who are passionate about mastering modern testing.

Join the community and get the latest articles and tips by signing up for the newsletter.

Top comments (0)