On-Call Escalation

On-call escalation routes an alert firing to the people who can fix it, paging each rung of a team's escalation policy until someone acknowledges. In addition to email and webhook destinations, OpenObserve can page responders by SMS and voice call through Twilio.

Paging channels

A page is delivered to every person named on the current rung, on every channel the team's policy pages for that priority:

On a team's Members tab (Reliability → On-Call → Teams, then open the team), each member has a row of channel icons for Email, SMS, and Voice call, so you can see at a glance which channels can actually reach that person. Webhook is the team's channel rather than a person's, so it is not shown per member.

In the example below, no SMTP transport or phone provider is configured, so every member is marked Unreachable and their channel icons are red or dashed.

Set up telephony

SMS and voice paging go through a Twilio account. OpenObserve supports two accounts, and each organization uses one or the other:

• Deployment account: set once by your operator with environment variables, and shared by every organization that has no account of its own.
• Organization account (bring-your-own): configured per organization in Settings → Telephony, so each org can page from its own Twilio number.

An organization account always wins over the deployment account. If an org account is configured but unusable, the org gets no SMS or voice and never falls back to the deployment account, so the wrong account is never billed.

Configure the organization account

Only organization admins can change this setting.

1. Go to Settings → Telephony.
2. Click Connect Twilio. If a deployment account is already active, click Use your own Twilio account instead.
3. Enter the Account SID (starts with AC), Auth token, and From number (with country code, in E.164 form such as +14155550100).
4. Click Connect.

OpenObserve checks the account with Twilio before saving it, and saving requires your operator to have set O2_MASTER_ENCRYPTION_KEY. The auth token is stored encrypted and is never returned by the API. To change the account later, click Update; leave Auth token blank to keep the stored token. Disconnect removes the org account, after which pages use the deployment account if there is one, or email only.

The page also shows whether Call keys: 4 acknowledge, 6 resolve, 8 escalate are available, which depends on O2_TELEPHONY_CALLBACK_URL (see below).

Configure the deployment account

The deployment account is set with these environment variables. The first four must be set together, or all four left empty:

A partly set account fails at boot with an error naming the missing variables. Boot also refuses a From number that is not E.164 and a callback URL that does not start with https://.

Add and verify your phone

Before SMS or voice can page a number, the number must be verified. This protects a recycled number's new owner from being woken by somebody else's outage.

1. Open the user menu in the top-right corner and select My on-call.
2. In the Phone card, enter your number with country code and click Send code. OpenObserve texts a 6-digit code to the number.
3. Enter the code and click Confirm.

A verification code works for 10 minutes and allows five attempts. Codes are rate-limited per user and per number over a rolling 24-hour window, and each refusal (wrong code, expired code, too many requests) is shown as a message that says what to do next.

Until the organization has a phone provider, the Phone card says so and pages reach you by email only:

Acknowledge, resolve, or escalate from a call

A voice page reads the alert aloud, then prompts: "Press 4 to acknowledge, 6 to resolve, or 8 to escalate."

• Press 4 acknowledges the page as you.
• Press 6 resolves the response.
• Press 8 escalates to the next level, then waits up to 8 seconds for the next rung's pages to go out and tells you if they did not.

Key presses are protected by a signed, single-use voice token separate from the emailed or texted acknowledge link, so a link cannot be replayed to drive a call. Every outcome is answered with a spoken reply: acknowledged, already acknowledged by someone, resolved, escalated, escalated to nobody, nobody above you, or no key pressed.

Review reachability

A team's Members tab answers "would a page to this person actually land?" for every member, without ringing a single phone. Each member's channel icons show a verdict per channel, computed from what actually exists; hover an icon to see the reason:

• Email is blocked if the address is not a user of the organization, is not shaped like a mailbox, uses a reserved domain (example.com, localhost, and similar), has no SMTP transport configured, or has seen its recent sends all fail.
• SMS and Voice are blocked if the member has no phone on file, the phone is not verified, or the org has no usable phone provider. On your own row, a missing phone links to My on-call so you can add it.
• A member no page can reach is marked Unreachable.

A channel is only judged by its own delivery history, so SMS failing leaves Email green, and a single recent failure does not mark someone unreachable.

Send a test page

To prove the whole path works, open Reliability → On-Call → Routing, enter the dimensions an alert would carry in Where would this route?, and click Send a test page once a team is matched. A test page goes through the real dispatch path, with the same channels and the same resolved recipients as a live page, but opens no response record, runs no escalation, and is clearly marked TEST so nobody mistakes it for a real page. A message then reports how many people the test page reached, or why nothing was delivered.

Delivery behavior

Dispatch is designed to get one page out fast and to keep a broken provider from slowing every other page:

• Every person on a rung is paged on every channel in parallel, with at most 16 sends in flight at once and a 15-second limit per send.
• A transient failure (connection reset, 429, 5xx) is retried with a short backoff; a permanent failure (no SMTP configured, rejected account, rejected recipient) is not.
• A circuit breaker is kept per channel and per provider account, so a hard-down provider stops charging its failure to every recipient of every ladder. A rejected recipient never counts against the breaker.
• SMS is capped at three segments using a conservative GSM-7/UCS-2 count, and the acknowledge link is never truncated.

Configuration reference