This task involves the work of implementing the calls to action that will accompany the LLM-generated MoS Suggestions that we present during the initial experienced volunteer evaluation phases (T428311 & T431376).

Requirements

Each of the three types of LLM-generated MoS suggestions (revise tone, simplify language, and review place name) will be accompanied by the same CTA...

• Prompt: Do you think this suggestion is valid?
  • CTA #1 (Yes, it's valid): hide suggestion; show "Success toast" (see below)
    • CTA #2 (No, it's not valid): hide suggestion; show "Keep survey" (see below)
• Success toast:
  • Thank you for helping to ensure edit suggestions are reliable and useful.
• Rejection survey:
  • Survey description: Please help developers understand why you think this suggestion is invalid.
  • Survey options:
    • There is an issue in the text, but the type of issue is mislabeled.
    • There is an issue in the text, but the suggestion is not helpful.
    • None of the above applies.
• Suggestion footer
  • Identified using an open-weight language model.
• Logging/Instrumentation
  • Each time someone engages with either of the calls to action within the card, ensure that response is logged and associated with its corresponding UUID (T430720)
    • This includes which response people provide in the feedback survey.

Done

• Before implementation begins, ask ML Team for feedback on the proposed technical approach to verify logging will provide info. they need
• Requirements are implemented
• Verify response data is being logged to schema as expected

---

I checked the related model-generated Suggestion Mode documentation and instrumentation work. One important scope distinction: T430030 owns the suggestion-card copy, while T432666 is specifically about the validation CTAs, feedback flow, footer, and logging. The experimental suggestions cover neutral/tone issues, simpler language, and geographic naming. ([MediaWiki][1])

Derived Requirements

1. Ensure that each experimental LLM-generated Revise tone, Simplify language, and Review place name suggestion provides the prompt Do you think this suggestion is valid? with options for the user to indicate that the suggestion is valid or invalid.

2. Ensure that when a user indicates an LLM-generated suggestion is valid, the suggestion is dismissed and a success toast displays Thank you for helping to ensure edit suggestions are reliable and useful.

3. Ensure that when a user indicates an LLM-generated suggestion is invalid, the suggestion is dismissed and a rejection survey is displayed.

4. Ensure that the rejection survey displays Please help developers understand why you think this suggestion is invalid. and provides the following feedback options:
   • There is an issue in the text, but the type of issue is mislabeled.
   • There is an issue in the text, but the suggestion is not helpful.
   • None of the above applies.

5. Ensure that each experimental LLM-generated suggestion displays the footer Identified using an open-weight language model.

6. Ensure that a user's valid or invalid response is recorded for the specific suggestion with which the user interacted.

7. Ensure that feedback selected from the rejection survey is recorded for the specific suggestion being evaluated.

Test Steps

Preconditions

1. Use an experienced-user test account eligible for experimental LLM-generated MoS suggestions.
2. Enable experimental suggestions if required by the test environment.
3. Open an article containing an experimental LLM-generated suggestion.
4. Test with Revise tone, Simplify language, and Review place name suggestions when examples of each type are available.

Test Case 1: Ensure all LLM-generated suggestion types provide the validation CTA

1. Open an article containing an experimental LLM-generated MoS suggestion.
2. Enter VisualEditor.
3. Navigate to the LLM-generated suggestion.
4. Review the suggestion card.
5. Repeat with Revise tone, Simplify language, and Review place name suggestions.
6. ✅❓❌⬜ AC1: Each available LLM-generated suggestion type displays "Do you think this suggestion is valid?" with options to indicate that the suggestion is valid or invalid.

Test Case 2: Ensure accepting a valid suggestion displays the success confirmation

1. Open an LLM-generated suggestion.
2. Locate the Do you think this suggestion is valid? prompt.
3. Select the option indicating the suggestion is valid.
4. Observe the suggestion and resulting notification.
5. ✅❓❌⬜ AC2: Indicating that the suggestion is valid dismisses the suggestion and displays "Thank you for helping to ensure edit suggestions are reliable and useful."

Test Case 3: Ensure rejecting an invalid suggestion displays the feedback survey

1. Open an LLM-generated suggestion.
2. Locate the Do you think this suggestion is valid? prompt.
3. Select the option indicating the suggestion is invalid.
4. Observe the suggestion and resulting feedback interface.
5. ✅❓❌⬜ AC3: Indicating that the suggestion is invalid dismisses the suggestion and displays the rejection survey.

Test Case 4: Ensure the rejection survey contains the required feedback options

1. Open an LLM-generated suggestion.
2. Indicate that the suggestion is invalid.
3. Review the rejection survey description.
4. Review all available survey responses.
5. ✅❓❌⬜ AC4: The rejection survey displays "Please help developers understand why you think this suggestion is invalid." with options for a mislabeled issue, an unhelpful suggestion, and "None of the above applies."

Test Case 5: Ensure LLM-generated suggestions identify their model source

1. Open an article containing an experimental LLM-generated suggestion.
2. Enter VisualEditor.
3. Navigate to the suggestion.
4. Review the bottom of the suggestion card.
5. Repeat with each available LLM-generated suggestion type.
6. ✅❓❌⬜ AC5: Each LLM-generated suggestion displays the footer "Identified using an open-weight language model".

Test Case 6: Ensure valid and invalid responses are logged for the correct suggestion

1. Open an LLM-generated suggestion with a known UUID.
2. Indicate that the suggestion is valid.
3. Inspect the corresponding instrumentation event using the available QA logging method.
4. Open a different LLM-generated suggestion with a different known UUID.
5. Indicate that the second suggestion is invalid.
6. Inspect its corresponding instrumentation event.
7. ✅❓❌⬜ AC6: Each valid or invalid response is logged with the UUID corresponding to the LLM-generated suggestion on which the response was submitted.

QA Results - TestFlight