ServiceNow Integration with OpenObserve: Automate Incident Creation from Alerts

Integrating OpenObserve with ServiceNow automates incident creation directly from your alerts. When your systems trigger an alert in OpenObserve, the integration automatically creates a ServiceNow incident with the relevant context, eliminating manual ticket creation and reducing response times.

This guide walks through two integration methods: a direct webhook approach for straightforward incident creation, and via Actions in Openobserve that provides deduplication and advanced processing capabilities. Choose webhooks for simple use cases where each alert creates a new incident, or use openobserve actions when you need deduplication to prevent duplicate tickets from repetitive alerts.

Why Automate ServiceNow Incident Creation

Manual incident creation introduces delays between alert detection and ticket assignment. Automated incident creation provides several operational benefits:

Faster Response Times: Incidents appear in ServiceNow within seconds of alert triggering. Your team starts investigating immediately instead of creating tickets.

Complete Context: Alert payloads include all relevant data like timestamps, affected services, and metric values. This context flows directly into ServiceNow fields.

Alert Deduplication: For repetitive alerts, the actions method can update existing incidents instead of creating duplicates, preventing ticket spam.

Prerequisites

Before configuring the integration, ensure you have:

ServiceNow Requirements:

• ServiceNow instance URL (example: https://yourinstance.service-now.com)
• User account with incident creation permissions
• Basic authentication credentials or API token

OpenObserve Requirements:

• OpenObserve account (cloud or self-hosted)
• Permissions to create templates, destinations, and alerts
• For actions: ability to create service accounts and upload custom actions

Network Access:

• OpenObserve must reach your ServiceNow instance via HTTPS

Method 1: Direct Integration via Webhook

The webhook method requires three configuration steps: creating a template that defines the incident payload structure, setting up a destination that points to your ServiceNow instance, and creating an alert that uses this destination.

Step 1: Create the Template

Templates define how alert data maps to ServiceNow incident fields. You can use variables that OpenObserve replaces with actual values when alerts trigger, or hardcode values for testing and fixed scenarios.

Navigate to OpenObserve's management page by clicking the gear icon, then select Template and click Add Template.

Template with Variables

Variable-based templates extract data from your alert results. Each variable like {alert_name} or {description} must exist in your alert query output:

{
  "assignment_group": "{assignment_group}",
  "caller_id": "{caller_id}",
  "category": "{category}",
  "contact_type": "{contact_type}",
  "description": "{description}",
  "impact": "2",
  "priority": "2",
  "short_description": "{alert_name}",
  "state": "{state}",
  "subcategory": "{subcategory}",
  "timestamp": "{timestamp}",
  "urgency": "{urgency}"
}

When your alert fires, OpenObserve substitutes these variables with actual values from your log or metrics data.

Key Field Explanations:

• impact and urgency: 1 (High), 2 (Medium), 3 (Low)
• state: 1 (New), 2 (In Progress), 6 (Resolved)
• short_description: Brief incident summary
• description: Detailed incident information

Hardcoded Template

For testing or scenarios where incident fields remain constant:

{
  "assignment_group": "Database Team",
  "caller_id": "openobserve@example.com",
  "category": "Software",
  "contact_type": "Email",
  "description": "Users unable to login due to database connectivity issues affecting the customer portal",
  "impact": "2",
  "priority": "2",
  "short_description": "Database Connectivity Alert",
  "state": "1",
  "subcategory": "Database",
  "timestamp": "2024-01-15T10:30:00Z",
  "urgency": "2"
}

Step 2: Configure the Webhook Destination

From the management page, navigate to Alert Destination, click Add Destination, and select Webhook.

Name: Give your destination a descriptive name like "ServiceNow Production Incidents".

Template: Select the template you created in Step 1.

URL: Your ServiceNow incident API endpoint:

https://yourinstance.service-now.com/api/now/table/incident

Method: Set to POST.

Headers: Add the Authorization header with your ServiceNow credentials:

Authorization: Basic <base64_encoded_credentials>

To generate the base64 encoded credentials:

echo -n "username:password" | base64

For example, if your username is admin and password is SecurePass123, the header becomes:

Authorization: Basic YWRtaW46U2VjdXJlUGFzczEyMw==

Step 3: Create an Alert with the Destination

Create an alert in OpenObserve with your monitoring query. For example, a query detecting high memory utilization:

SELECT 
    k8s_cluster,
    k8s_node_name,
    round(AVG(value) / 1024 / 1024 / 1024,2) AS alert_value,
    'high_memory_utilization' AS alert_reason
FROM 
    k8s_node_memory_usage
WHERE k8s_cluster = 'prod'
GROUP BY
    k8s_cluster,
    k8s_node_name
HAVING alert_value > 80; 

In the alert's destination settings, select the webhook destination you created.

Important: For variable-based templates, your alert query must return all fields referenced in the template. If your template uses {assignment_group} but your query doesn't return that field, the variable won't substitute correctly.

Method 2: Using OpenObserve Actions for Advanced Processing

Actions in OpenObserve are user-defined Python scripts that support custom automation workflows, with Actions as alert destinations, users can take an immediate action by adding an automation workflow using Actions. The python script provides deduplication using correlation IDs and custom processing logic.

Step 1: Create the Template

Create a template following the same process as Method 1. You can use either variable-based or hardcoded templates.

Step 2: Configure the Real-Time Action

From the OpenObserve landing page, navigate to Actions and click Add Action.

Name: Provide a descriptive name like "ServiceNow Incident Creator with Deduplication".

Description: Document what this action does, for example: "Creates ServiceNow incidents from alerts. Updates existing incidents when correlation_id matches."

Type: Select "Real Time".

Upload Zip File: Download the ServiceNow action package from the OpenObserve Actions repository. This package contains the Python scripts for ServiceNow API interaction and deduplication logic.

Service Account: Select an existing service account or create a new one following the service account creation guide.

Environment Variables: Configure these required variables:

SERVICENOW_URL=https://yourinstance.service-now.com
SERVICENOW_USERNAME=your_username
SERVICENOW_PASSWORD=your_password

Optionally, add these variables to enable execution logging:

OPENOBSERVE_URL=https://api.openobserve.ai
OPENOBSERVE_ORG=your_organization
OPENOBSERVE_STREAM=servicenow_action_logs
OPENOBSERVE_USERNAME=your_username
OPENOBSERVE_PASSWORD=your_password

Understanding How Deduplication Works

The action script includes deduplication logic using a correlation_id field. When your template includes a correlation_id:

1. Search Phase: The action queries ServiceNow for open incidents with the same correlation_id
2. Decision:
   • If an incident exists: Updates that incident's work notes with new information
   • If no incident exists: Creates a new incident with the correlation_id

This prevents alert storms from creating hundreds of duplicate tickets.

Example Template with correlation_id:

{
  "correlation_id": "alert_{alert_name}_{hostname}",
  "short_description": "{alert_name} on {hostname}",
  "description": "{description}",
  "urgency": "2",
  "impact": "2",
  "category": "Infrastructure",
  "work_notes": "Alert triggered at {timestamp}. CPU usage: {cpu_usage}%"
}

The correlation_id combines alert name and hostname to create a unique identifier. Each unique hostname gets its own incident, but repeated alerts for the same hostname update the existing incident.

Without correlation_id: The action creates a new incident every time, behaving like the webhook method.

Step 3: Configure the Action Destination

Navigate to Alert Destination, click Add Destination, and select Action.

Name: Descriptive destination name.

Template: Select your template from Step 1.

Action: Select the action you created in Step 2.

Step 4: Create an Alert and Test

Create an alert and select the action destination, test it by triggering an alert.

In OpenObserve's alert history, verify:

• The alert triggered successfully
• The destination was called
• No errors appear in the alert execution logs

Checking ServiceNow

Navigate to your ServiceNow instance and go to Incident > All.

Verify the incident:

• Short Description matches your alert
• Description contains relevant details
• Assignment Group is correct
• Priority and Urgency are set appropriately
• For variable templates: all variables substituted correctly (no literal {variable_name} text)

For Actions with Deduplication: Trigger the same alert multiple times and verify that subsequent triggers add work notes to the existing incident rather than creating new ones.

Best Practices

• Start with Hardcoded Templates: Verify connectivity first with hardcoded values, then add variables incrementally.
• Match Variable Names: Ensure your alert query field names exactly match template variable names (case-sensitive).
• Map Severity Appropriately: Critical alerts should use impact=1, urgency=1. Medium alerts use impact=2, urgency=2.
• Use Meaningful Correlation IDs: Format correlation IDs as alert_{type}_{resource} for proper deduplication grouping.
• Test Credentials First: Create a test incident manually via API before configuring production alerts.

Troubleshooting

Incidents Not Creating: Verify ServiceNow credentials by testing manually:

curl -X POST "https://yourinstance.service-now.com/api/now/table/incident" \
  -H "Authorization: Basic <credentials>" \
  -H "Content-Type: application/json" \
  -d '{"short_description":"Test"}'

Variables Not Substituting: Ensure your alert query returns all fields referenced in the template. Query field names must exactly match template variable names.

Deduplication Not Working: Check that correlation_id remains consistent across alert triggers and that incidents are in open state (not closed or resolved).

Next Steps

You now have everything you need to automate ServiceNow incident creation from OpenObserve alerts. Here's how to get started:

Implementation Checklist:

1. Start with a hardcoded template to verify connectivity
2. Create a test alert that triggers easily
3. Verify the incident appears in ServiceNow with correct fields
4. Add template variables to pull data from your alerts
5. Implement actions with correlation_id to prevent duplicate tickets

Get Started with OpenObserve Today! Sign up for a free cloud trial

Additional Resources:

• OpenObserve Documentation
• ServiceNow Actions Repository
• ServiceNow Table API Reference

Have questions? Join the OpenObserve community or check the documentation for detailed guides.

Frequently Asked Questions

What's the difference between webhook and actions?

Webhooks send alert data directly to ServiceNow's API without additional processing. Actions execute custom Python code that provides deduplication, custom field mapping, and advanced error handling. Use webhooks for simple incident creation, use actions when you need deduplication to prevent duplicate tickets.

How does alert deduplication prevent duplicate tickets?

When using actions with a correlation_id field, the script searches ServiceNow for open incidents with the same correlation_id before creating a new incident. If found, it updates the existing incident's work notes instead of creating a duplicate. This prevents alert storms from creating hundreds of tickets.

Can I customize which ServiceNow fields are populated?

Yes. Templates define the JSON payload sent to ServiceNow. You can include any ServiceNow incident field available via the API. Reference the ServiceNow Table API documentation for available fields.

What happens if my ServiceNow instance is down?

OpenObserve retries failed destination calls according to your alert configuration. The alert execution logs show any failures. For critical alerts, configure multiple destinations as fallback (for example, ServiceNow primary and email/Slack as secondary).

What if I need different templates for different alert types?

Create multiple templates and destinations. For example, create a "Critical Incidents" template with impact=1 and a "Warning Incidents" template with impact=3. Select the appropriate destination when creating each alert based on severity.