How to Configure PagerDuty with OpenObserve Alerts: A Step-by-Step Guide

Introduction

When systems fail, every minute counts. That's why incident management tools like PagerDuty have become essential for modern DevOps and SRE teams.

PagerDuty is an industry-leading incident response platform that ensures the right team members are notified when critical issues arise. By integrating OpenObserve with PagerDuty, you can:

• Automatically create incidents from OpenObserve alerts
• Route notifications to the appropriate teams based on escalation policies
• Reduce mean time to resolution (MTTR) with faster incident response
• Maintain comprehensive incident history for post-mortem analysis

In this guide, we'll walk through the process of configuring PagerDuty to receive alert notifications from OpenObserve, ensuring your team gets notified promptly when issues arise.

Prerequisites

Before we begin, make sure you have:

• An OpenObserve instance up and running
• A PagerDuty account with permissions to create services and integrations
• Some basic familiarity with both platforms

Step 1: Setting Up PagerDuty

1. Log in to your PagerDuty account
2. Create a new service:
   • Navigate to "Services" → "Service Directory"
   • Click "New Service"
   • Enter a name for your service (e.g., "O2_Alert")
   • Select an escalation policy that determines who gets notified

• Under "Integration Settings," select "PRTG Notification For PagerDuty Events API v2" as the Integration Type

• Click "Create Service"

3. Get your Integration Key:
   • After creating the service, you'll be taken to the Integrations tab
   • Find the "Integration Key" (also called "routing key")
   • Keep this key handy as we'll use it in the next steps

Step 2: Creating the Alert Template in OpenObserve

1. Log in to your OpenObserve console
2. Navigate to "Management" → "Templates"
3. Click "Add Template"
4. Configure the template:
   • Name: "pagerduty"
   • Paste the following JSON template in the body:

{
"payload": {
"summary": "{alert_name} - Alert Triggered",
"severity": "critical",
"source": "OpenObserve",
"component": "{stream_name}",
"custom_details": {
"organization": "{org_name}",
"alert_type": "{alert_type}",
"threshold": "{alert_threshold}",
"triggered_count": "{alert_count}",
"triggered_at": "{alert_trigger_time_str}",
"query": "{query}"
}
},
"routing_key": "YOUR_INTEGRATION_KEY_HERE",
"event_action": "trigger",
"dedup_key": "{alert_id}"
}

5. Replace YOUR_INTEGRATION_KEY_HERE with the Integration Key you copied from PagerDuty
6. Save the template

Step 3: Configuring OpenObserve Alert Destination

1. In OpenObserve, go to "Management" → "Alert Destinations"
2. Click "Add Destination"
3. Configure the webhook:
   • Select "Webhook" as the destination type
   • Name: "PagerDuty Alerts"
   • URL: https://events.pagerduty.com/v2/enqueue
   • Method: POST
   • Headers:
     • Add a header with key Content-Type and value application/json
   • Template: Select the PagerDuty Events API v2 template we created earlier
4. Save the destination

Step 4: Creating an Alert Rule

1. Navigate to "Alerts" → "Add Alert"
2. Configure your alert:
3. Save the alert rule

Step 5: Testing the Integration

1. Trigger a test alert:
   • You can either wait for your alert condition to be met naturally or trigger it maunually to test.
2. Check PagerDuty:
   • Log in to your PagerDuty account
   • Navigate to "Incidents"
   • You should see a new incident created with the details from your OpenObserve alert

3. Verify the incident details:
   • The incident should contain all the information specified in your template
   • Check that the severity, source, and custom details are displayed correctly

Troubleshooting

If you're not seeing alerts in PagerDuty, check these common issues:

• Verify your Integration Key: Make sure you've copied the correct key from PagerDuty
• Test the webhook: Use a tool like Postman to test the PagerDuty Events API directly
• Check alert conditions: Ensure your alert conditions are being met to trigger notifications

Conclusion

Congratulations! You've successfully integrated OpenObserve with PagerDuty, creating a reliable incident management pipeline that will help your team respond faster to critical issues.

While PagerDuty is an excellent choice for incident management, OpenObserve offers flexibility to integrate with various notification platforms based on your team's preferences. You can also configure OpenObserve alerts to work with popular services like Slack (check out our video tutorial), Microsoft Teams, or Opsgenie.

Each integration follows a similar pattern to what we've covered in this guide - creating a template, configuring a destination, and setting up alert rules. For a complete list of supported integrations and template examples, visit the OpenObserve Alert Templates documentation.

Choose the notification platform that best fits your team's workflow, or use multiple destinations for different types of alerts. The right setup will ensure critical issues get the attention they need while minimizing disruptions for less urgent matters.

Happy monitoring!