# Observability Pipelines. Shape Before it Lands. > Transform observability data with real-time and scheduled pipelines. VRL functions, enrichment tables, and flexible stream operations. Start building free. Source: https://openobserve.ai/pipelines/ --- Transform, enrich, redact, reduce, and parse your observability data with real-time and scheduled pipelines. - [Start Free Cloud Trial](https://cloud.openobserve.ai/web/login/) - [Talk to a Human](/demo/) ### Redact PII Before It’s Stored Strip or mask sensitive fields at ingestion, before data ever hits disk. ### Parse and Enrich in Real Time VRL functions transform data the moment it arrives, no batch delay. ### One Pipeline, Any Format Handle JSON, syslog, CEF, and custom formats without separate parsers per source. ## Shape Data Before It Costs You ### Pipeline Types - **Real-time** - Transform data the instant it arrives. Parse, filter, and enrich streams on the fly, turning raw data into immediate insights. - **Scheduled** - Orchestrate batch transformations precisely when you need them. Aggregate massive datasets, convert logs to actionable metrics, and process data on your timeline. [Learn More](https://openobserve.ai/webinars-videos/creating-pipelines-with-openobserve-comprehensive-tutorial/) ### Data Transformation - **VRL Functions** - Build custom transformations that parse, enrich, and filter your data streams with functions that handle complex logic requirements. - **Data Parsing** - Transform logs into structured data regardless of format using pre-built parsing functions. [Learn More](https://openobserve.ai/docs/user-guide/data-exploration/logs/logs/#transform-logs-with-vrl) ### Data Enrichment - **Enrichment Tables** - Transform raw events into context-rich data using simple CSV lookup tables. Add location data, user details, and metadata that turns isolated signals into patterns. - **Dynamic Lookups** - Pull in external context via API at ingestion time, no separate enrichment step required. [Learn More](https://openobserve.ai/docs/user-guide/data-processing/enrichment-tables/enrichment/) ### Pipeline Components - **Function Nodes** - Build data processing workflows with VRL function nodes. Execute custom logic for complex transformations and create reusable components. - **Stream Operations** - Filter, clone, and route streams to multiple destinations, ensuring every bit of data reaches exactly where it needs to go. [Learn More](https://openobserve.ai/docs/user-guide/data-processing/pipelines/pipelines/) ## Measured against industry leaders Same telemetry, same workloads, one platform. Every number is OpenObserve against a named vendor - not an industry average. 8x cost reduction means you can unify your observability into a single platform. 140x storage means longer retention doesn't necessarily mean expensive bills. 5x to 15x faster queries mean dashboards load in milliseconds, not minutes. See how much you would save switching today. - [See all comparisons](/comparison/) ## Teams trust OpenObserve with their data pipelines ## Observability Pipelines FAQs ### What types of pipelines does OpenObserve support? OpenObserve supports two main types of pipelines: real-time and scheduled. Real-time pipelines process data as it arrives, transforming and enriching it on the fly. These pipelines can include multiple processing steps using functions, conditions, and stream operations. Scheduled pipelines run at defined intervals, allowing for batch processing and periodic data transformations. ### How do functions work in pipelines? Functions in OpenObserve use Vector Remap Language (VRL) for data transformation. Each function contains VRL code that can parse, transform, and enrich data. Functions can access the incoming data fields, perform conditional processing, and modify or create new fields. For example, a function might parse JSON logs, extract specific fields, and enrich them with geographical information using enrichment tables. ### What are enrichment tables and how are they used? Enrichment tables are CSV-based lookup tables that allow you to add additional context to your data. You can upload CSV files containing reference data, which can then be queried within pipeline functions using the get_enrichment_table_record function. Common use cases include IP to location mapping, user agent parsing, and adding business context to technical data. ### How does data parsing work in pipelines? Data parsing in OpenObserve pipelines is handled through VRL functions. The platform supports parsing various formats including JSON, structured logs, and custom formats. Functions can include conditional logic to apply different parsing rules based on the data source or content. Error handling is built into the parsing functions, allowing graceful handling of malformed data. ### What stream operations are available? Pipeline streams can be configured with various operations including source selection, transformation steps, and destination routing. The platform supports both logs and metrics streams. You can create complex workflows with multiple processing steps, conditions, and parallel processing paths. Stream operations maintain data consistency while allowing for flexible transformation chains. ### How do pipeline conditions work? Conditions in pipelines allow for selective processing based on data attributes. Using VRL expressions, you can create sophisticated routing and processing logic. Conditions can check field values, apply pattern matching, and implement complex business rules. This enables targeted processing of specific data streams or message types. ## Explore guides, videos, and articles to help you get the most out of Observability Pipelines. ### All About Pipelines [Learn more](https://openobserve.ai/webinars-videos/creating-pipelines-with-openobserve-comprehensive-tutorial/) ### Filte, Transform and Route with Pipelines [Learn more](https://openobserve.ai/webinars-videos/openobserve-pipelines/) - [Explore All Blogs](/blog/) ## Ready to get started? Try OpenObserve today for more efficient and performant observability. - Get Started For Free - [Schedule Demo](/demo/)