Overview

Datadog Experiments analyzes experiments run with Datadog Feature Flags or an independent randomization system. If you randomize subjects outside of Datadog Feature Flags, create Exposure SQL Models to tell Datadog who was exposed to your experiment and when, by reading exposure records from your warehouse.

Similar to Metric SQL Models, Exposure SQL Models are SQL queries that describe exposure data in your warehouse. Exposure SQL Models return the following columns:

When you create an experiment, Datadog Feature Flags and warehouse exposures both appear during flag selection:

Datadog deduplicates exposure data for you. You do not need to deduplicate upstream:

• If multiple records exist for the same user-experiment pair, Datadog uses the first record.
• If one user was exposed to multiple variants in the same experiment, Datadog excludes that user from the analysis.

Prerequisites

Before you create Exposure SQL Models:

• Connect your warehouse to Datadog. Datadog supports BigQuery, Databricks, Redshift, and Snowflake.
• Create at least one Metric SQL Model and warehouse metric to measure experiment outcomes.
• Store exposure records in your warehouse with the columns listed above.

Create Exposure SQL Models

Write your SQL

Exposure SQL Models query exposure data in your warehouse. For example:

SELECT
  user_id,
  exposed_at,
  experiment_id,
  variant_id,
  customer_tier
FROM analytics.experiment_exposures

To create an Exposure SQL Model:

1. Navigate to Experiments > Settings > Exposure SQL Models in Datadog Product Analytics.
2. Click Create Exposure SQL Model.
3. Enter a SQL query that returns the required columns. Column names in your query do not need to match the Datadog field names.
4. Click Run to preview your data.

Map columns to Datadog

1. In the mapping panel on the right, map each query column to the corresponding Datadog field (Subject Key, Timestamp, Experiment ID, Variant ID, and optional split-by properties).
2. Click Create Exposure SQL Model to save the model.

Create experiments using Exposure SQL Models

After you save an Exposure SQL Model, Datadog scans the model for distinct experiment and variant values. To create an experiment with warehouse exposures:

1. On the experiment setup page, open the feature flag selector.
2. Select a warehouse experiment ID from the list, or enter an experiment ID manually if it does not appear in the latest scan.

3. Select the Exposure SQL Model that tracks the flag.

How Datadog computes results

After you create an experiment from an Exposure SQL Model, Datadog runs a data pipeline that:

1. Filters exposures to the relevant experiment and time frame.
2. Deduplicates by subject (user) and removes users exposed to multiple variants.
3. Joins exposures to metric events that occurred after exposure, by user and timestamp.
4. Aggregates to the user level and applies winsorization (caps extreme metric values to reduce variance).
5. Aggregates to the variant level and computes summary statistics, including mean and variance.

Update schedule

Datadog updates experiment results nightly. To refresh results immediately, open the experiment overview page and click Run an update now.

To view a simplified version of the pipeline logic, click Copy SQL on any warehouse metric.

Further reading

Additional helpful documentation, links, and articles:

Connecting a Data WarehouseDOCUMENTATION Create experiment metrics from warehouse dataDOCUMENTATION Measure the business impact of every product change with Datadog ExperimentsBLOG