Alerts in LangSmith

• An active PagerDuty account with administrator access
• Appropriate service-level permissions in PagerDuty

​

1. Create a Service in PagerDuty

1. Log in to your PagerDuty account
2. Navigate to Services → Service Directory
3. Click + New Service
4. Complete the following fields:
   • Name: Provide a descriptive name (e.g., “LangSmith Monitoring”)
   • Description: Add details about the monitored application
   • Escalation Policy: Select the appropriate team escalation policy
   • Integration Type: Select “Events API V2”
5. Click Add Service to create the service

​

2. Obtain integration key

1. From the Service Directory, locate and click on your newly created service
2. Select the Integrations tab
3. Find the “Events API V2” integration
4. Copy the Integration Key (a 32-character alphanumeric string)

​

3. Configure LangSmith alert with PagerDuty

1. In the notification section of your alert set-up in LangSmith, select PagerDuty
2. Click the key icon to save the Integration Key as a Workspace secret or select an existing Workspace secret. As a best practice, we recommend saving the Integration Key as a Workspace Secret rather than adding it directly. This will allow you to reuse the same key across alerts for a workspace.
3. Configure additional notification options:
   • Severity: Maps to PagerDuty incident priority
4. Send a test alert by clicking Send Test Alert
5. Verify the incident is triggered by PagerDuty and contains relevant LangSmith alert information

​

Troubleshooting

• Verify the Integration Key is entered correctly in LangSmith
• Ensure the PagerDuty service is active and not in maintenance mode
• Check that your PagerDuty account has Events API v2 enabled
• If an alert trigger appears to be missing in PagerDuty, check whether the expected trigger occurred within one hour of a previous trigger from the same alert rule, and whether the incident created by the previous alert is still open.
• Review network connectivity if your LangSmith instance is behind a firewall

​

Additional resources

• PagerDuty Events API v2 Documentation
• PagerDuty Integration Guide

• An active Dynatrace environment (SaaS or Managed).
• A Dynatrace API access token with the events.ingest scope.

​

1. Create an API token in Dynatrace

1. Log in to your Dynatrace environment.
2. Navigate to Access Tokens.
3. Click Generate new token.
4. Provide a descriptive name (e.g., “LangSmith Alerts”).
5. Under Scopes, search for and enable events.ingest (Ingest events).
6. Click Generate token.
7. Copy the generated token and store it securely. The token is only displayed once.

​

2. Obtain your Dynatrace environment URL

https://{your-environment-id}.live.dynatrace.com

​

3. Configure LangSmith alert with Dynatrace

1. In the Notifications Settings for your alert setup in LangSmith, select Dynatrace.
2. Enter your Dynatrace environment URL.
3. Click the key icon to save the API token as a workspace secret or select an existing workspace secret. As a best practice, save the API token as a workspace secret rather than adding it directly. This allows you to reuse the same token across alerts for a workspace.
4. Configure additional notification options:
   • Event Type: Select the Dynatrace event type (e.g., CUSTOM_ALERT, ERROR_EVENT)
5. Send a test alert by clicking Send Test Notification.
6. Verify the event appears in your Dynatrace environment.

​

Troubleshooting

• Verify the API token has the events.ingest scope and is not expired.
• Ensure the environment URL is correct and includes your environment ID.
• Confirm the Authorization header format uses Api-Token (not Bearer).
• Check that your Dynatrace environment is active and accessible.
• Review network connectivity if your LangSmith instance is behind a firewall.

​

Additional resources

• Dynatrace Events API v2 Documentation
• Dynatrace Access Tokens

• An endpoint that can receive HTTP POST requests
• Appropriate authentication credentials for your receiving service (if required)

​

1. Prepare your receiving endpoint

• Accepts HTTP POST requests
• Can process JSON payloads
• Is accessible from external services
• Has appropriate authentication mechanisms (if required)

​

2. Configure webhook parameters

• URL: The complete URL of your receiving endpoint
  • Example: https://api.example.com/incident-webhook

• Headers: JSON key-value pairs sent with the webhook request
  • Common headers include:
    • Authorization: For authentication tokens
    • Content-Type: Usually set to application/json (default)
    • X-Source: To identify the source as LangSmith
  • If no headers, use {}
• Request Body Template: Customize the JSON payload sent to your endpoint
  • Default: LangSmith sends the payload defined and the following additional key-value pairs appended to the payload:
    • project_name: Name of the triggered alert
    • alert_rule_id: A UUID to identify the LangSmith alert. This can be used as a de-duplication key in the webhook service.
    • alert_rule_name: The name of the alert rule.
    • alert_rule_type: The type of alert (as of 04/01/2025 all alerts are of type threshold).
    • alert_rule_attribute: The attribute associated with the alert rule - error_count, feedback_score, latency, or cost.
    • triggered_metric_value: The value of the metric at the time the threshold was triggered.
    • triggered_threshold: The threshold that triggered the alert.
    • timestamp: The timestamp that triggered the alert.

​

3. Test the webhook

​

Troubleshooting

• Verify the webhook URL is correct and accessible
• Ensure any authentication headers are properly formatted
• Check that your receiving endpoint accepts POST requests
• Examine your endpoint’s logs for received but rejected requests
• Verify your custom payload template is valid JSON format

​

Security considerations

• Use HTTPS for your webhook endpoints
• Implement authentication for your webhook endpoint
• Consider adding a shared secret in your headers to verify webhook sources
• Validate incoming webhook requests before processing them

​

Example recipes

Configure Slack notifications via webhook

Here is an example for configuring LangSmith alerts to send notifications to Slack channels using the chat.postMessage API.Prerequisites

• Access to a Slack workspace.
• A LangSmith project to set up alerts.
• Permissions to create Slack applications.

Step 1: Create a Slack app

1. Visit the Slack API Applications page.
2. Click Create New App.
3. Select From scratch.
4. Provide an App Name (e.g., “LangSmith Alerts”).
5. Select the workspace where you want to install the app.
6. Click Create App.

Step 2: Configure bot permissions

1. In the left sidebar of your Slack app configuration, click OAuth & Permissions.
2. Scroll down to Bot Token Scopes under Scopes and click Add an OAuth Scope.
3. Add the following scopes:
   • chat:write (Send messages as the app).
   • chat:write.public (Send messages to channels the app isn’t in).
   • channels:read (View basic channel information).

Step 3: Install the app to your workspace

1. Scroll up to the top of the OAuth & Permissions page.
2. Click Install to Workspace.
3. Review the permissions and click Allow.
4. Copy the Bot User OAuth Token that appears (begins with xoxb-).

Step 4: Add the bot to a Slack channelAdd the bot to the specific channel you want to receive alerts in. You can add a bot to a Slack channel by mentioning it in the message field (e.g., @botname).You also need the channel ID to configure the webhook alert in LangSmith. You can find the channel ID by opening channel details > About.Step 5: Configure the webhook alert in LangSmith

1. In LangSmith, navigate to your project.
2. Select Alerts → Create Alert.
3. Define your alert metrics and conditions.
4. In the notification section, select Webhook.
5. Configure the webhook with the following settings:

Webhook URL

https://slack.com/api/chat.postMessage

Headers

Replace xoxb-your-token-here with your Bot’s User OAuth Token

{
  "Content-Type": "application/json",
  "Authorization": "Bearer xoxb-your-token-here"
}

Request Body Template

It is required to fill in the {channel_id} from the value found in Step 4.

The remaining fields: alert_name, project_name and project_url optionally add additional context to the alert message. You can find your project_url in the browser’s URL bar. Copy the portion up to but not including any query parameters.

{
  "channel": "{channel_id}",
  "text": "{alert_name} triggered for {project_name}",
  "blocks": [
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "🚨{alert_name} has been triggered"
      }
    },
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "Please check the following link for more information:"
      }
    },
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "<{project-url}|View in LangSmith>"
      }
    }
  ]
}

6. Click Save to activate the webhook configuration.

Step 6: Test the integration

1. In the LangSmith alert configuration, click Test Alert.
2. Check your specified Slack channel for the test notification.
3. Verify that the message contains the expected alert information.

(Optional) Step 7: Link to the alert preview in the request bodyAfter creating an alert, you can optionally link to its preview in the webhook’s request body.To configure this:

1. Save your alert.
2. Find your saved alert in the alerts table and click it.
3. Copy the displayed URL.
4. Click “Edit Alert”.
5. Replace the existing project URL with the copied alert preview URL.

https://prod-XX.westus.logic.azure.com:443/workflows/.../triggers/manual/paths/invoke?...

{
  "Content-Type": "application/json"
}

{}

https://api.sendgrid.com/v3/mail/send

{
  "Content-Type": "application/json",
  "Authorization": "Bearer SG.your-api-key-here"
}

{
  "personalizations": [
    {
      "to": [
        {
          "email": "oncall@your-company.com"
        }
      ],
      "subject": "LangSmith alert triggered"
    }
  ],
  "from": {
    "email": "alerts@your-company.com",
    "name": "LangSmith Alerts"
  },
  "content": [
    {
      "type": "text/plain",
      "value": "A LangSmith alert was triggered. Open your LangSmith workspace to view the alert details, including the project, metric value, threshold, and timestamp."
    }
  ]
}

​

Additional resources

• Slack chat.postMessage API Documentation
• Slack Block Kit Builder
• Create incoming webhooks with Workflows for Microsoft Teams
• Power Automate documentation
• langsmith-teams-webhook sample repo
• SendGrid Mail Send API Documentation