Streams in OpenObserve
Streams define how observability data is ingested, stored, indexed, and queried in OpenObserve. This guide introduces key concepts related to streams and explains how to create and use them.
What is a stream?
A stream is a logical container that holds one type of observability data, such as logs, metrics, or traces. It is the required entry point for data ingestion in OpenObserve. Every log, metric, or trace must be associated with a stream at the time of ingestion.
Note
Each stream has the following characteristics:
- It belongs to an organization.
- It has a unique name.
- It stores only one type of data such as logs, metrics, and traces.
Access
- The New stream button is present by default on the Streams page.
- Users can create streams without enabling any configuration.
- During stream creation, the Fields section is optional.
- If you do not define any fields, OpenObserve will automatically detect them from the data you ingest. These detected fields will appear under All Fields in the Stream Details page.
-
By default, the New stream button is not visible on the Streams page.
-
To enable the New stream button, set the following environment variable to
true:
ZO_ALLOW_USER_DEFINED_SCHEMAS -
After enabling the variable:
- The New stream button becomes visible.
- During stream creation, the Fields section is optional. The fields you add form the user-defined schema for the stream.
Permission
OpenObserve Cloud or the Enterprise Edition with Role Based Access Control (RBAC)
If RBAC is enabled in OpenObserve Cloud and the enterprise edition:
- Root: Full access to all streams across all organizations.
- Admin and Editor: Can view, create, update, and delete streams within their organization.
- Viewer: Can view streams. Cannot create, update, or delete streams.
- User: Cannot view, create, or modify streams.
- Custom roles: A user or service account with custom roles can create and manage streams if the required permissions are granted using RBAC.
Learn more about OpenObserve Cloud and the Enterprise Edition.
Create a Stream in OpenObserve
Prerequisite
- You have set up OpenObserve using OpenObserve Cloud or the self hosted option.
- You have created the organization where you plan to create the stream.
- For self-hosted deployment, ensure that the
ZO_ALLOW_USER_DEFINED_SCHEMASenvironment variable is enabled.
The following steps vary for Cloud and Self-hosted deployment:
If RBAC is enabled, ensure that you have required permissions to create streams.
Create streams in OpenObserve Cloud
-
Select the organization from the top navigation bar.
-
From the left navigation menu, select Streams.
-
Click New stream.
-
In the New stream dialog:
- Enter a unique stream Name. Use letters, numbers, underscores and colons only.
- Select the Stream Type.
- Specify the Data Retention in days. For example, enter 14 to keep data for 14 days after ingestion. When the period ends, OpenObserve removes the data automatically.
To keep data longer, select Extended Retention in the Stream Details sidebar. - (Optional) In the Fields section, click Add Field to define fields for your stream:
- Field Name: Name of the field
- Data Type: Select Utf8, Int64, Uint64, Float64, or Boolean
- Index Type: Choose an indexing strategy (Full text search, Secondary index, Bloom filter, KeyValue partition, Prefix partition, or Hash partition with 8 to 128 buckets)
For detailed information about each field type and index strategy, see Field and Index Types in Streams
These fields create a User Defined Schema. Learn more about user defined schema.
-
Click Save.
The new stream appears on the Streams page. Ingest data into the stream to populate and start using it.
Create streams in OpenObserve self-hosted
-
Select the organization from the top navigation bar.
-
From the left navigation menu, select Streams.
-
Click New stream.
-
In the New stream dialog:
- Enter a unique stream Name. Use letters, numbers, underscores and colons only.
- Select the Stream Type.
- Specify the Data Retention in days. For example, enter 14 to keep data for 14 days after ingestion. When the period ends, OpenObserve removes the data automatically.
To keep data longer, select Extended Retention in the Stream Details sidebar. - (Optional) In the Fields section, click Add Field to define fields with:
- Field Name: Name of the field
- Data Type: Select Utf8, Int64, Uint64, Float64, or Boolean
- Index Type: Choose an indexing strategy (Full text search, Secondary index, Bloom filter, KeyValue partition, Prefix partition, or Hash partition with 8 to 128 buckets)
For detailed information about each field type and index strategy, see Field and Index Types in Streams The fields you add form the user-defined schema at stream creation. Learn more about user defined schema.
Click to see how User-defined Schema works.
Let us say you define the following fields while creating a stream:
- job (String)
- code (Integer)
- message (String)
After you ingest data into this stream:
- These fields (job, code, message) will appear under User Defined Schema in the Stream Details page.
- Any additional fields not defined earlier will appear under All Fields.
-
Click Save.
Note
You can also create a stream during the first data ingestion, without using the UI. For other users, the organization must already exist before ingestion.
The root user can create an organization through ingestion while the environment variable ZO_CREATE_ORG_THROUGH_INGESTION is true, which is the default. Set it to false to turn this off.
Ingest Data into Stream
- Ingest Data Using curl.
- Ingest Data Using API in JSON formatted logs and multiple records in a batch with multiple JSON lines.
- Ingest Data Continuously Using Data Sources.
Note
You can now use the stream in Logs search, Dashboards, Pipelines, Alerts, and Actions.
Next Steps
Streams Overview
Streams in OpenObserve define how observability data is ingested, stored, indexed, and queried, covering stream details, schema settings, and retention.
Stream Details
View and manage detailed stream settings in OpenObserve, including retention, query limits, schema, and usage stats via the Stream Details panel.