Context (for others reading this): One potential outcome @Kgraessle and I discussed for Moderator Tools's experiment was if the clickthrough per page visit ends up being no different between the control group (who did not get the grouping toggle) and the treatment group (who did get the grouping toggle). In that case one could analyze the collected data to see if the reason for this is perhaps that the treatment group saw the toggle, turned it on out of curiosity, saw how drastically it changed the appearance of something they were very used to, said "ew I hate this", turned grouping off, and then never touched it again.

With the new proposal, would we be able to self service the follow-up investigation (with relative ease) if we do have to look into the toggle usage?
I think your answer was yes given we can pass action_context into the submit interaction function, but just wanted to validate.

You can't do that with the standard CTR instrument. It's really for simple click tracking and minimal data collection. I think you'll need to switch to custom clickthrough tracking to collect data that would allow you to do a deeper analysis of user behavior, which is not a problem.

Given your WIP patch maybe something like:

$( element ).on( "click", function() {
  experiment.send("click", {
    action_context: JSON.stringify({grouping: "on"}), // TODO: make this get toggle's value at time of event
    instrument_name: "RCFiltersWidget",
    element_friendly_name: "Diff link",
    action_source: "Watchlist"
  } );
} );

(Instead of attaching the CTR instrument to element)

You would also want to do something like:

// element is the toggle that's shown to users in the treatment group
$( element ).on( "click", function() {
  const toggleState = TODO; // "on" or "off"
  experiment.send( "toggle", {
    action_subtype: toggleState,
    instrument_name: "RCFiltersWidget",
    action_source: "Watchlist"
  } ):
} )

(Or whatever the appropriate event handler is.)

Oh and you would also need to have:

experiment.send( "page-visited", { instrument_name: "RCFiltersWidget", action_source: "Watchlist" } );

somewhere in the instrumentation code.

Suppose experiment's machine readable name is fy24-25-we-1-7-rc-change-grouping and treatment group's machine readable name is toggle-shown

Then when you do the follow up investigation, you might start by looking at how often users tried it at all:

WITH subjects AS (
  SELECT
    experiment.subject_id,
    SUM(IF(action = 'toggle' AND action_subtype = 'on', 1, 0)) > 0 AS did_enable
  FROM event.product_metrics_web_base
  WHERE experiment.enrolled = 'fy24-25-we-1-7-rc-change-grouping'
    AND experiment.assigned = 'toggle-shown'
    AND action IN('page-visited', 'toggle')
  GROUP BY 1
)
SELECT
  AVG(did_enable) AS did_try_feature_proportion
FROM subjects

To see how often users left it on after turning it on:

WITH visits AS (
  SELECT
    experiment.subject_id,
    performer.pageview_id,
    SUM(IF(action = 'toggle' AND action_subtype = 'on', 1, 0)) >
      SUM(IF(action = 'toggle' AND action_subtype = 'off', 1, 0)) AS did_leave_enabled
  WHERE ...
    AND action = 'toggle'
  GROUP BY 1, 2
)
SELECT
  subject_id,
  AVG(did_leave_enabled) AS did_leave_enabled_proportion
FROM visits
GROUP BY 1

So for every page visit where the toggle was switched on, we check if it was left on. Then we go over each user's page visits (where the toggle was on at some point) and calculate what proportion of those visits the toggle stayed on. We can then plot a histogram of these proportions.

We just need to take care of T395692: Add performer_pageview_id contextual attribute to web base stream

Is it possible to summarize the instrumentation and the measurement in the task description? As of now, they are private Google Docs. Targeted wikis and the duration of the experiment should be made public, too.

This experiment should be announced in Tech News, so that the targeted communities would be informed of the experiment (along with the message @OTichonova wrote). It would also show other communities that we work on improving these interfaces.

In T396789#10978017, @Trizek-WMF wrote:

Is it possible to summarize the instrumentation and the measurement in the task description? As of now, they are private Google Docs. Targeted wikis and the duration of the experiment should be made public, too.

@Trizek-WMF I went ahead and updated the description to include a summarization of the instrumentation and measurement plan.

This experiment should be announced in Tech News, so that the targeted communities would be informed of the experiment (along with the message @OTichonova wrote). It would also show other communities that we work on improving these interfaces.

Is there something I need to do to get that to happen or are we still working with translations?

In T396789#10979755, @Kgraessle wrote:

@Trizek-WMF I went ahead and updated the description to include a summarization of the instrumentation and measurement plan.

Thank you!

[Tech News] Is there something I need to do to get that to happen or are we still working with translations?

If it is about Tech News translations, they are covered by the Tech News volunteer squad. Tech News only needs a small paragraph describing the experiment: what, where, when; and a "know more" link.

For Tech News, please could someone propose draft wording for an entry? From a skim of this task, my best guess is that it's an experimental new in-line toggle for the long-existing user-setting of "Group changes by page in recent changes and watchlist". Is that accurate? If yes, I'd suggest an entry along the lines of:

• Some editors will see a new button on their watchlist and recentchanges pages to toggle on and off the "Group changes by page in recent changes and watchlist" feature. This is part of an experiment to see [...?...if it helps increase usage of these pages...?...].

Please correct and amend as needed, or suggest something more accurate!

We also need to know when this needs to be announced (i.e. reply here the week before the feature is enabled, or by EOD Friday if it's going live next week). Thanks.

Not sure if this is where I'm supposed to give some feedback, but this looks a bit ugly, and even when under the dropdown I thought the toggle meant it would paginate results instead of... well I can't think of a better way to describe it either.

Is there a place other than Phabricator where we can see this in action? The screen shot, with grouping enabled, looks just like my regular watchlist at en.WP.

I've been asking for grouping in the Watchlist by page instead of by calendar day for many years. It has never made sense to me that I have to click through multiple days of diffs when I have been away from a wiki for a while and want to catch up on the cumulative diff of each page since my last visit. This task description makes it look like it might get rid of the arbitrary day boundaries that make it harder to do this catching up, but the screen shot doesn't show that (yet).

I remain hopeful.

@Jonesey95 Click on the "{number} changes, {number} days" dropdown with a cogwheel in its left side and check "Group results by page".

Though it only groups the changes within each calendar day instead of going beyond.

I think that I already have that kind of grouping on en.WP, by enabling "Expand watchlist to show all changes, not just the most recent".

Is there a site where I can experiment with this new feature to see how it may be different from what I already have?

Edited to add: I think I may understand. When I enable the Javascript version of the Watchlist, I see the settings drop-down, with a checkbox that says "Group by ...", which I have enabled. Is this task simply moving that checkbox out of the drop-down menu and placing it as a slider button to the left of the menu? If so, I understand this experiment and remain hopeful that someone will understand the desire to remove the arbitrary calendar day boundaries in the watchlist.

Is this task simply moving that checkbox out of the drop-down menu and placing it as a slider button to the left of the menu?

Looks like it.

Saw this in tech news. Thanks for your work on this. My watchlist already groups by pages and I like the feature.

Can you clarify exactly what you're changing? Are you just pulling this already existing setting out of a menu, for increased visibility?

In T396789#11137475, @Novem_Linguae wrote:

Saw this in tech news. Thanks for your work on this. My watchlist already groups by pages and I like the feature.

Can you clarify exactly what you're changing? Are you just pulling this already existing setting out of a menu, for increased visibility?

I was confused as well. See above. As far as I could surmise, they are simply enabling this feature for a random group of editors to see what happens.

In T396789#11137475, @Novem_Linguae wrote:

Saw this in tech news. Thanks for your work on this. My watchlist already groups by pages and I like the feature.

Can you clarify exactly what you're changing? Are you just pulling this already existing setting out of a menu, for increased visibility?

Exactly, we're just moving the grouping functionality out from the menu for increased discoverability.
By doing this we're testing our hypothesis:

If we expose users to a new group by toggle for their Grouping Preference then we will see more clickthroughs to changes' links (diff, contributions page, page history, and user page).

Hope this helps!

Who would benefit from grouping results per page by day? I don't get it. It just makes the Watchlist even more cluttered and harder to go through for basically everybody on an active Wikipedia with a normal count of watched articles.

Instead, I'd suggest to highlight the setting of the Watchlist to show just the latest revision of a page if that's not the default (at least if say >50 articles are watched) and to inform about the Global Watchlist which does have an actually very useful feature (that the WP Watchlist is still missing) of a button to see a diff of all unseen changes with 1 click (but has fixable issues that mean it can't be used with that in practice).