[DE 1.1.9] Include performer_edit_count in exposure events
Closed, ResolvedPublic1 Estimated Story Points
Actions

Assigned To

Authored By

	mpopov
	Tue, Jul 28, 2:36 PM

Project Tags

Referenced Files

None

Subscribers

Aklapper

mpopov

phuedx

Description

To include edit count-based dimensions in GB, we need to start recording edit counts in exposure events.

Acceptance criteria

performer_edit_count contextual attribute is included in exposure events when using Experiment#sendExposure()
- TK PHP SDK
- TK JS SDK

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	Include performer_edit_count in exposure events	Clare Ming	mediawiki/extensions/TestKitchen	master	+14 -4

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T433646 [Epic] Experiment-Ready Metrics
Open	mpopov	T433380 [DE 1.1.9] Constructive edit metric in GrowthBook
Open	mpopov	T433322 [DE 1.1.9] Define constructive edit-based metrics in GrowthBook
Open	Ahoelzl	T433379 [DE 1.1.9] Iceberg table of constructive edits
Open	cjming	T431493 [DE 1.1.9] An instrument that produces edit_survived events
Open	mpopov	T434576 [Spike] Evaluate incremental MWH for use in GB
Open	mpopov	T433381 [DE 1.1.9] Add edit count-based dimensions in GB
Open	mpopov	T433384 [DE 1.1.9] Include edit count in experiment_assignment_v1
Resolved	cjming	T433386 [DE 1.1.9] Include performer_edit_count in exposure events
Open	mpopov	T434733 [DE 1.1.9] Run A/A tests

Event Timeline

mpopov created this task.Tue, Jul 28, 2:36 PM

mpopov triaged this task as Medium priority.Tue, Jul 28, 2:39 PM

mpopov edited projects, added: Test Kitchen (Experiment Platform Sprint 27); removed: Test Kitchen.

mpopov updated the task description. (Show Details)

mpopov set the point value for this task to 1.

mpopov raised the priority of this task from Medium to High.Tue, Jul 28, 3:35 PM

mpopov lowered the priority of this task from High to Medium.

mpopov raised the priority of this task from Medium to High.Tue, Jul 28, 3:40 PM

cjming claimed this task.Wed, Jul 29, 10:48 PM

cjming moved this task from Ready for Development to In Progress on the Test Kitchen (Experiment Platform Sprint 27) board.

Change #1319185 had a related patch set uploaded (by Clare Ming; author: Clare Ming):

[mediawiki/extensions/TestKitchen@master] Include performer_edit_count in exposure events

https://gerrit.wikimedia.org/r/1319185

gerritbot added a project: Patch-For-Review.Wed, Jul 29, 11:27 PM

cjming moved this task from In Progress to Needs Review on the Test Kitchen (Experiment Platform Sprint 27) board.Wed, Jul 29, 11:29 PM

Just to check, performer_edit_count and not performer_edit_count_bucket, right?

From our discussion:

We considered whether edit_count_bucket would be preferable to edit_count (raw).
The _bucket version doesn't work for our stakeholders because DE3 strategy talks about editors with 100 or fewer edits, but the bucketing scheme doesn't work for us because 100 edits is included the 100-299 bucket.
- The bucketing scheme (implementation) was borrowed from editing/editor stats datasets that were being published.
- We should at some point revisit the bucketing scheme used in TK.
There was a concern about possible risk of including the raw edit count and its potential for identifying/de-anonymizing.
- The primary guiding factor for us is that edit count is not called out by Data Collection Guidelines as a risky piece of information.
- Furthermore, we don't have a ledger of per-editor edit counts, so it would be challenging (but perhaps not impossible) for a bad actor in the system to use a subject's edit count in an experiment exposure event to figure out a set of editors that the subject could have been.
We considered implementing performer_edit_count_capped contextual attribute with a cap of 300 edits (partially informed by the "Very Experienced" editor categorization in https://meta.wikimedia.org/wiki/Research_and_Decision_Science/Data_glossary) but that would have significantly increased scope that we don't have bandwidth for and without a clear sense of what risk this would be mitigating.
We also considered capping the value manually, but it would have increased complexity of Experiment#sendExposure() method, which is currently very lightweight and easy to maintain.