OpenObserve supports two main types of pipelines: real-time and scheduled. Real-time pipelines process data as it arrives, transforming and enriching it on the fly. These pipelines can include multiple processing steps using functions, conditions, and stream operations. Scheduled pipelines run at defined intervals, allowing for batch processing and periodic data transformations.

Functions in OpenObserve use Vector Remap Language (VRL) for data transformation. Each function contains VRL code that can parse, transform, and enrich data. Functions can access the incoming data fields, perform conditional processing, and modify or create new fields. For example, a function might parse JSON logs, extract specific fields, and enrich them with geographical information using enrichment tables.

Enrichment tables are CSV-based lookup tables that allow you to add additional context to your data. You can upload CSV files containing reference data, which can then be queried within pipeline functions using the get_enrichment_table_record function. Common use cases include IP to location mapping, user agent parsing, and adding business context to technical data.

Data parsing in OpenObserve pipelines is handled through VRL functions. The platform supports parsing various formats including JSON, structured logs, and custom formats. Functions can include conditional logic to apply different parsing rules based on the data source or content. Error handling is built into the parsing functions, allowing graceful handling of malformed data.

Pipeline streams can be configured with various operations including source selection, transformation steps, and destination routing. The platform supports both logs and metrics streams. You can create complex workflows with multiple processing steps, conditions, and parallel processing paths. Stream operations maintain data consistency while allowing for flexible transformation chains.

Conditions in pipelines allow for selective processing based on data attributes. Using VRL expressions, you can create sophisticated routing and processing logic. Conditions can check field values, apply pattern matching, and implement complex business rules. This enables targeted processing of specific data streams or message types.

The platform provides visibility into pipeline execution through the pipeline viewer. You can monitor pipeline performance, track processing errors, and debug transformation logic. The system maintains metrics about pipeline throughput and processing latency. Test functions allow you to verify transformation logic before deployment.

Pipeline configurations are managed through the OpenObserve interface, where you can create, edit, and delete pipelines. Changes to pipeline functions can be tested before deployment using sample data. The platform maintains version control for pipeline configurations, allowing you to track changes and roll back if needed.