# How to Debug a Real-Time Pipeline in OpenObserve: Complete Guide > A step-by-step guide to debugging real-time observability pipelines in OpenObserve, covering log ingestion, pipeline errors, and troubleshooting. Source: https://openobserve.ai/blog/debug-realtime-pipeline/ Published: 2025-12-15 Authors: Simran Kumari Category: How To Tags: Pipelines --- Real-time data pipelines are essential for modern observability platforms, enabling organizations to process, transform, and route telemetry data as it arrives. OpenObserve, an open-source observability platform, provides powerful real-time pipeline capabilities that help DevOps teams, SREs, and data engineers manage streaming data at scale. However, debugging these pipelines when transformations fail or data goes missing can be challenging without the right approach. This comprehensive guide explains how to troubleshoot OpenObserve real-time pipelines, diagnose transformation errors, and implement best practices for reliable data processing workflows. ## **Understanding OpenObserve Pipelines: Core Concepts** ### **What Are Pipelines in OpenObserve?** A pipeline in OpenObserve is a configurable data processing workflow that determines how incoming data is handled after ingestion. Think of it as an automated assembly line for your logs, metrics, and traces, where each station performs specific operations on the data passing through. OpenObserve supports two distinct pipeline types, each designed for different use cases: * **[Real-time Pipelines](https://openobserve.ai/docs/user-guide/data-processing/pipelines/create-and-use-real-time-pipeline/)** process events immediately upon arrival at the source. These are ideal for use cases requiring instant data transformation, intelligent routing based on content, filtering out noise, enriching events with additional context, or applying business logic before storage. Organizations use real-time pipelines for security event processing, application performance monitoring, log normalization, and compliance filtering. * **[Scheduled Pipelines](https://openobserve.ai/docs/user-guide/data-processing/pipelines/create-and-use-scheduled-pipeline/)** operate on historical data from a stream at predefined intervals. These excel at periodic ETL operations, batch aggregations and rollups, data backfills and reprocessing, and time-based analytics jobs. Common applications include daily metric aggregations, weekly report generation, and retroactive data quality improvements. > **New to OpenObserve pipelines?** If you’re not familiar with how pipelines work in OpenObserve, you can learn them in detail here: [How to Set Up Pipelines in OpenObserve](https://openobserve.ai/blog/how-to-setup-pipelines-in-openobserve/) ### **Pipeline Architecture: Building Blocks Explained** Both pipeline types share fundamental components that work together to process data: ![Building blocks of pipeline in OpenObserve](/assets/pipeline_nodes_c60d43cf37.png) * The **source node** represents the data origin, typically an OpenObserve stream or a query result set. This is where raw telemetry data enters the pipeline from your applications, infrastructure, or third-party integrations. * **Transformation nodes** contain the business logic that modifies, enriches, or filters events. These functions can add calculated fields, parse unstructured data, apply conditional logic, normalize timestamps and formats, redact sensitive information, or drop unwanted events entirely. * The **destination node** specifies where processed data should be written. This could be another OpenObserve stream for further processing, a long-term storage stream with different retention policies, or an external destination like S3, Kafka, or another observability platform. ### **Key Warning Signs** When a **[real-time streaming pipeline](https://openobserve.ai/docs/user-guide/data-processing/pipelines/create-and-use-real-time-pipeline/)** encounters an error, you'll typically notice: 1. **Error indicator** - Visual alert in the pipeline monitoring UI 2. **Timestamp** - Exact time of the last error occurrence ![Real time pipeline error warning in OpenObserve](/assets/real_time_pipepline_error_warning_687a0f83fa.png) On expanding the error message, you will get an idea as to which part of the pipeline is running into errors, is it source node, destination node or the transformation function. ![Error NodeId in case of Pipeline Failure](/assets/error_node_id_341b9589b5.png) These indicators confirm a failure occurred, but they don't reveal the root cause. To get detailed diagnostic information, you need access to runtime telemetry. **Enabling usage reporting** is critical for debugging real-time pipelines. Without telemetry, you're working blind unable to see error details, stack traces, or the specific node that failed. ## **Step-by-Step Debugging Process** ### **1. Enable Usage Reporting** **Purpose**: Collect detailed error, audit, and trigger streams for inspection **How to enable**: Detailed guide to [enable usage reporting](https://openobserve.ai/blog/usage-reporting/) Usage reporting in OpenObserve creates a separate audit trail of all pipeline operations, including successful transformations and failures. This telemetry is stored in a dedicated organization (typically called the "meta" organization) to keep operational metadata separate from your application data. When enabled, usage reporting captures the complete context of each pipeline execution: the full event payload that entered the transformation, the exact error message and stack trace, the pipeline name and configuration version, the specific node that failed, and precise timestamps for when the error occurred. **What it collects**: Error streams, audit logs, and trigger events that provide visibility into pipeline failures. ### **2. Open the Log Explorer** Once usage reporting is active, access the **Log Explorer** interface to view your telemetry streams. **Common streams you'll find**: * **Usage stream** - Runtime metrics and performance data * **Audit stream** - Configuration changes and access logs * **Errors stream** - Detailed failure information * **Triggers stream** - Event activation logs ![Streams for Usage Reporting](/assets/auditing_streams_usage_reporting_3c04baaed4.png) **Note**: If your environment doesn't use scheduled pipelines or alerts, some streams may not be populated. When a real-time pipeline fails, the **errors stream** will contain the diagnostic information you need. For scheduled pipelines you can find logs for successful events in `triggers` stream, For real-time pipelines only errors are logged in `error` stream. ### **3. Query the Error Stream** **Filter by identifying attributes**: Pipeline name, Organization ID, Error timestamp range **Key fields in error records**: * **Error source** - Indicates which component failed (e.g., "pipeline") * **Organization ID** - Links the error to your organization * **Pipeline name** - Identifies the specific pipeline * **Error ID** - Unique identifier for this error instance * **Timestamp** - When the error occurred ![Querying the error stream](/assets/error_stream_1facdc3640.png) **Pro tip**: When multiple pipelines are running simultaneously, filtering by pipeline name significantly speeds up troubleshooting. #### Isolating Errors by Pipeline and Node To narrow down results to your specific failing pipeline, filter entries by the pipeline name (for example, "transform_real_time" or whatever you named your pipeline). This eliminates noise from other pipelines running in your OpenObserve instance. ![Filtering by Pipeline name](/assets/filtering_on_pipeline_name_to_debug_realtime_pipeline_fb154bdda7.png) From the pipeline error message copy the node error ID. This unique identifier points to the exact transformation node that failed. ![Fetching the Error NodeID](/assets/error_node_id_341b9589b5.png) Run a string match filter against the error stream using this node ID to isolate all entries related to that specific transformation. ![Filtering by Error Node ID](/assets/filtering_on_error_nodeid_103b771519.png) This is especially valuable in **multi-stage pipelines** where isolating the failure point saves significant debugging time. ![Multi-level transformation stream](/assets/mutli_level_transformation_stream_bb29190cf5.png) ### **4. Interpret the Error Message and Fix** The error body contains the diagnostic information you need to understand what went wrong. Once you've identified the root cause, implement and test your fix. **Best Practices for Real-Time Pipeline Debugging:** **Add null and type checks** * Validate field existence before accessing values * Check data types match expected formats * Handle missing or malformed data gracefully **Sanitize unexpected input values** * Add input validation at pipeline entry points * Normalize data formats before transformation * Filter out or handle edge cases **Test transform logic in isolation:** * Run transformations against sample data outside the pipeline and validate business logic before deployment. ![Testing Transformation Function](/assets/testing_transformation_function_b4714fd598.png) * Use unit testing for complex transformation functions ## **Quick Reference Checklist** Use this checklist whenever a pipeline failure occurs: * **Confirm usage reporting is enabled** - Check organization settings * **Verify correct reporting organization** - Ensure you're viewing the right org's telemetry * **Open errors stream in Log Explorer** - Navigate to monitoring dashboard * **Filter by pipeline name** - Narrow results to your specific pipeline * **Trigger with sample data if needed** - Generate a fresh error event * **Locate the node ID** - Identify which stage failed * **Read the detailed error message** - Understand the specific failure * **Fix the offending function** - Apply targeted corrections * **Add defensive checks** - Prevent similar failures * **Re-test with sample events** - Validate the fix works * **Monitor error stream** - Confirm errors are resolved ## **When to Use Scheduled Pipelines Instead of Real-time Processing** ### **Choosing the Right Pipeline Type** Not every data processing task requires real-time execution. Scheduled pipelines are more appropriate when you need to process historical data at fixed intervals such as every 5, 10, or 60 minutes, perform aggregations across time windows, run resource-intensive transformations that would impact real-time ingest performance, or backfill data after schema changes or bug fixes. ### **Real-time Pipeline Use Cases** Real-time pipelines excel when you need immediate data transformation and routing, security event processing with instant alerting, log normalization before storage, compliance filtering to drop sensitive data, or application performance monitoring with sub-second latency requirements. ### **Scheduled Pipeline Use Cases** Scheduled pipelines are ideal for periodic ETL jobs that consolidate data, daily or weekly aggregation reports, retroactive data quality improvements, and computationally expensive transformations that can run in batch mode. ## **Troubleshooting Common OpenObserve Pipeline Issues** ### **Missing Fields After Transformation** If expected fields are missing from transformed events, verify that the source event actually contains the input fields your transformation depends on, check for case sensitivity issues in field names, confirm that the transformation logic handles null or undefined values, and ensure the destination stream schema accepts the new fields. ### **Silent Pipeline Failures** When pipelines fail without obvious errors, confirm that usage reporting is enabled and functioning, check that you're looking in the correct organization for error logs, verify that the pipeline is actually active and receiving events, and review recent configuration changes that might have introduced issues. ### **Performance Degradation** If pipeline processing slows down over time, profile transformation functions to identify expensive operations, consider whether early filtering could reduce processing volume, evaluate whether scheduled pipelines would be more appropriate for heavy transformations, and monitor resource utilization on your OpenObserve nodes. ## **Conclusion** **Real-time data pipelines** power modern data architectures, but runtime issues are inevitable when processing variable input data. The key to rapid resolution is having clear, actionable telemetry. Debugging real-time pipelines in OpenObserve becomes straightforward once you establish the right workflow: enable usage reporting to capture comprehensive error data, reproduce failures with exact event payloads, use node error IDs to filter and isolate specific transformation issues, fix function code based on detailed error context, and validate fixes before deploying to production. ### **Additional Resources** * Learn more about [VRL functions and transformation logic](https://openobserve.ai/docs/user-guide/analytics/alerts/scheduled-alerts/#step-4-write-the-vrl-function) * Configure the pipeline to send data to [remote destinations](https://openobserve.ai/docs/user-guide/data-processing/pipelines/remote-destination/). * [Import and export pipelines](https://openobserve.ai/docs/user-guide/data-processing/pipelines/import-and-export-pipelines/) * [Manage real-time and scheduled pipelines.](https://openobserve.ai/docs/user-guide/data-processing/pipelines/manage-pipelines/)