How to Configure PagerDuty with OpenObserve Alerts: A Step-by-Step Guide
Manas Sharma
March 21, 2025
4 min read
Don’t forget to share!
Introduction
When systems fail, every minute counts. That's why incident management tools like PagerDuty have become essential for modern DevOps and SRE teams.
PagerDuty is an industry-leading incident response platform that ensures the right team members are notified when critical issues arise. By integrating OpenObserve with PagerDuty, you can:
- Automatically create incidents from OpenObserve alerts
- Route notifications to the appropriate teams based on escalation policies
- Reduce mean time to resolution (MTTR) with faster incident response
- Maintain comprehensive incident history for post-mortem analysis
In this guide, we'll walk through the process of configuring PagerDuty to receive alert notifications from OpenObserve, ensuring your team gets notified promptly when issues arise.
Prerequisites
Before we begin, make sure you have:
- An OpenObserve instance up and running
- A PagerDuty account with permissions to create services and integrations
- Some basic familiarity with both platforms
Step 1: Setting Up PagerDuty
- Log in to your PagerDuty account
- Create a new service:
- Navigate to "Services" → "Service Directory"
- Click "New Service"
- Enter a name for your service (e.g., "O2_Alert")
- Select an escalation policy that determines who gets notified
- Under "Integration Settings," select "PRTG Notification For PagerDuty Events API v2" as the Integration Type
- Click "Create Service"
- Get your Integration Key:
- After creating the service, you'll be taken to the Integrations tab
- Find the "Integration Key" (also called "routing key")
- Keep this key handy as we'll use it in the next steps
Step 2: Creating the Alert Template in OpenObserve
- Log in to your OpenObserve console
- Navigate to "Management" → "Templates"
- Click "Add Template"
- Configure the template:
- Name: "pagerduty"
- Paste the following JSON template in the body:
{
"payload": {
"summary": "{alert_name} - Alert Triggered",
"severity": "critical",
"source": "OpenObserve",
"component": "{stream_name}",
"custom_details": {
"organization": "{org_name}",
"alert_type": "{alert_type}",
"threshold": "{alert_threshold}",
"triggered_count": "{alert_count}",
"triggered_at": "{alert_trigger_time_str}",
"query": "{query}"
}
},
"routing_key": "YOUR_INTEGRATION_KEY_HERE",
"event_action": "trigger",
"dedup_key": "{alert_id}"
}
- Replace
YOUR_INTEGRATION_KEY_HEREwith the Integration Key you copied from PagerDuty - Save the template
Step 3: Configuring OpenObserve Alert Destination
- In OpenObserve, go to "Management" → "Alert Destinations"
- Click "Add Destination"
- Configure the webhook:
- Select "Webhook" as the destination type
- Name: "PagerDuty Alerts"
- URL:
https://events.pagerduty.com/v2/enqueue - Method: POST
- Headers:
- Add a header with key
Content-Typeand valueapplication/json
- Add a header with key
- Template: Select the
PagerDuty Events API v2template we created earlier
- Save the destination
Step 4: Creating an Alert Rule
- Navigate to "Alerts" → "Add Alert"
- Configure your alert:
- Save the alert rule
Step 5: Testing the Integration
- Trigger a test alert:
- You can either wait for your alert condition to be met naturally or trigger it maunually to test.
- Check PagerDuty:
- Log in to your PagerDuty account
- Navigate to "Incidents"
- You should see a new incident created with the details from your OpenObserve alert
- Verify the incident details:
- The incident should contain all the information specified in your template
- Check that the severity, source, and custom details are displayed correctly
Troubleshooting
If you're not seeing alerts in PagerDuty, check these common issues:
- Verify your Integration Key: Make sure you've copied the correct key from PagerDuty
- Test the webhook: Use a tool like Postman to test the PagerDuty Events API directly
- Check alert conditions: Ensure your alert conditions are being met to trigger notifications
Conclusion
Congratulations! You've successfully integrated OpenObserve with PagerDuty, creating a reliable incident management pipeline that will help your team respond faster to critical issues.
While PagerDuty is an excellent choice for incident management, OpenObserve offers flexibility to integrate with various notification platforms based on your team's preferences. You can also configure OpenObserve alerts to work with popular services like Slack (check out our video tutorial), Microsoft Teams, or Opsgenie.
Each integration follows a similar pattern to what we've covered in this guide - creating a template, configuring a destination, and setting up alert rules. For a complete list of supported integrations and template examples, visit the OpenObserve Alert Templates documentation.
Choose the notification platform that best fits your team's workflow, or use multiple destinations for different types of alerts. The right setup will ensure critical issues get the attention they need while minimizing disruptions for less urgent matters.
Happy monitoring!
