Skip to content

Alerts - templates

Templates are used when notification is sent for an alert, templates forms body of request being sent to destination, for eg. for slack one can create template like:

{
    "text": "For stream {stream_name} alert {alert_name} of type {alert_type} is active"    
}

When a notification is being sent, OpenObserve will replace placeholders like {stream_name}, {alert_name} etc with actual values of stream name, alert name.

Variables which can be used in templates are:

Variable Value Description
org_name Organization name Name of the organization
stream_type Stream type Type of the stream
stream_name Stream name Name of the stream for alert is created
alert_name Alert name Name of the alert
alert_type Alert type Possible values are : real time or scheduled
alert_period Alert period Limited 5 minutes, 10 minutes, set for alert
alert_operator Alert trigger operator Like >, >=, set for alert trigger
alert_threshold Alert trigger threshold Like, 5, the threshold for trigger alert, it will compare to alert_count or alert_agg_value.
alert_count Alert records number The records number when we query the alert condition.
alert_agg_value Alert aggregation value The value of the aggregation function if you enable it. and will use this value to compare with alert_threshold
alert_start_time _timestamp Alert matched the min _timestamp of the rows and formatted by %Y-%m-%dT%H:%M:%S
alert_end_time _timestamp Alert matched the max _timestamp of the rows and formatted by %Y-%m-%dT%H:%M:%S
alert_url URL A link can back to the UI and check the detail data. need configure ZO_WEB_URL
alert_trigger_time _timestamp Represents the timestamp in microseconds when the alert was evaluated
alert_trigger_time_str _timestamp Formatted alert_trigger_time. E.g. - 2024-12-02T11:27:40
rows mutiple lines of row template values based on row template in alert page
all of the stream fields the field value Default we select * from stream if you custom sql then it will be only the fields that you selected.

Variable length

You can use {rows:N} to limit only top N matched records will be in the actual value.

You also can use {log:N} to limit the length of a actual value.

Row templates

When your alert notification data have multiple objects you will want the notification has multiple line and each line is the data of one object. for example:

select k8s_pod_name, count(*) AS cnt FROM stram WHERE str_match(log, 'panic') GROUP BY k8s_pod_name ORDER BY cnt LIMIT 10

This sql for Alert will trigger when we find panic in logs, and we want to known which pod generate the panic log with the error count.

without row template we can define the alert template like this:

{
    "text": "{k8s_pod_name} got {cnt} panic logs"   
}

I need to join the multiple values of k8s_pod_name with a , if there are multiple pods. then the notification message like this:

{
    "text": "pod1,pod2,pod3 got 1,2,3 panic logs"   
}

Actually we want it to show line by line like this:

pod1 got 1 panic log
pod2 got 2 panic log
pod3 got 1 panic log

That is the row template, with row template we can define the alert template like this:

{
    "text": "alert for {alert_name}\n{rows}"    
}

Just define the template with rows, it will replace by actual values of all rows.

And we define the row template in alert page:

{k8s_pod_name} got {cnt} panic logs

After these, the notification message will be what we expect.

Check this video to understand more

Slack

Official slack docs at: https://api.slack.com/messaging/webhooks

{
  "text": "For stream {stream_name} of organization {org_name} alert {alert_name} of type {alert_type} is active"
}

Check the video to understand more.

Prometheus Alert Manager

[
    {
        "labels": {
            "alertname": "{alert_name}",
            "stream": "{stream_name}",
            "organization": "{org_name}",
            "alerttype": "{alert_type}",
            "severity": "critical"
        },
        "annotations": {
            "timestamp": "{timestamp}"
        }
    }
]

WeCom

{
  "msgtype": "text",
  "text": {
    "content": "For stream {stream_name} of organization {org_name} alert {alert_name} of type {alert_type} is active"
  }
}

Message usage:

https://developer.work.weixin.qq.com/document/path/91770

Webhook URL, eg:

POST https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=9a39c951-1234-4382-8a6e-12345678

Feishu

{
  "msg_type": "text",
  "content": {
    "text": "For stream {stream_name} of organization {org_name} alert {alert_name} of type {alert_type} is active"
  }
}

Message usage:

https://open.feishu.cn/document/ukTMukTMukTM/ucTM5YjL3ETO24yNxkjN?lang=zh-CN#383d6e48

Webhook URL, eg:

POST https://open.feishu.cn/open-apis/bot/v2/hook/d91b7e97-1234-1234-1234-dfb0b9cc54d0

Matrix

URL: https://[your matrix domain]/_matrix/client/r0/rooms/[roomID]/send/m.room.message
Method: POST
Headers: Authorization
Value: Bearer [Your token]
{
    "msgtype": "m.text",
    "format": "org.matrix.custom.html",
    "body": "{org_name}/{stream_name}: {alert_name}<br><a href=https://yourOpenObserveURL.example.com/web/logs?org_identifier={org_name}>Recent logs</a>",
    "formatted_body": "{org_name}/{stream_name}: {alert_name}<br><a href=https://yourOpenObserveURL.example.com/web/logs?org_identifier={org_name}>Recent logs</a>"

}

Opsgenie

Official docs at: https://docs.opsgenie.com/docs/alert-api#create-alert

URL: https://api.opsgenie.com/v2/alerts
Method: POST
Headers:

Authorization: GenieKey __YOUR_API_KEY__
{
    "message": "{alert_name} is active",
    "alias": "{alert_name}",
    "description":"{stream_name}",
    "priority":"P3"
}

Pagerduty

OpenObserve alert will create an incident in pagerduty

Official docs at: https://developer.pagerduty.com/api-reference/a7d81b0e9200f-create-an-incident

URL: https://api.pagerduty.com/incidents
Method: POST
Headers:

Authorization: Token token=y_NbAkKc66ryYTWUXYEu
Content-Type: application/json
From: <The email address of a valid user associated with the account making the request >
{
  "incident": {
    "type": "incident",
    "title": "The server is on fire.",
    "service": {
      "id": "PWIXJZS",
      "type": "service_reference"
    },
    "priority": {
      "id": "P53ZZH5",
      "type": "priority_reference"
    },
    "urgency": "high",
    "body": {
      "type": "incident_body",
      "details": "A disk is getting full on this machine. You should investigate what is causing the disk to fill, and ensure that there is an automated process in place for ensuring data is rotated (eg. logs should have logrotate around them). If data is expected to stay on this disk forever, you should start planning to scale up to a larger disk."
    },
    "escalation_policy": {
      "id": "PT20YPA",
      "type": "escalation_policy_reference"
    }
  }
}

Microsoft Teams

Official docs at: https://learn.microsoft.com/en-us/graph/api/chatmessage-post

URL: /teams/{team-id}/channels/{channel-id}/messages
Method: POST

Headers:
  Authorization: Bearer {code}
{
  "body": {
    "content": "For stream {stream_name} of organization {org_name} alert {alert_name} of type {alert_type} is active"  
  }
}

e.g.

POST https://graph.microsoft.com/v1.0/teams/fbe2bf47-16c8-47cf-b4a5-4b9b187c508b/channels/19:4a95f7d8db4c4e7fae857bcebe0623e6@thread.tacv2/messages

Headers:
  Content-type: application/json
  Authorization: Bearer {code}

Body:
{
  "body": {
    "content": "For stream {stream_name} of organization {org_name} alert {alert_name} of type {alert_type} is active"  
  }
}

Email

Subject

[Alert: {alert_name}] - Severity: {alert_type}
Body
An alert has been triggered:

- Alert Name: {alert_name}
- Severity: {alert_type}
- Stream Name: {stream_name}
- Condition: {alert_operator} {alert_threshold}
- Triggered Count: {alert_count}
- Start Time: {alert_start_time}
- End Time: {alert_end_time}
For more details, visit: {alert_url}