Sending Data from OpenObserve Pipelines to Splunk Using Splunk HTTP Event Collector (HEC)

Sometimes, after processing logs through OpenObserve pipelines, you may need to forward that data to other systems for further analysis or compliance. One popular destination is Splunk, thanks to its powerful analytics and alerting capabilities.

In this blog, we’ll walk through how to send processed log data from OpenObserve to Splunk using its HTTP Event Collector (HEC)—a flexible, high-performance interface for ingesting JSON-formatted events over HTTP or HTTPS. By integrating OpenObserve with Splunk via HEC, you get the best of both worlds-- OpenObserve’s real-time observability and Splunk’s deep search and visualization features. This easy setup will help you create a unified, end-to-end view of your logs across platforms.

OpenObserve to Splunk Use Cases

There are many scenarios where sending OpenObserve data to Splunk is beneficial, such as:

• Correlating logs from microservices in OpenObserve with firewall or endpoint data in Splunk.
• Centralizing observability data for long-term retention and compliance.
• Leveraging Splunk Enterprise Security or ITSI for enriched analysis.

Prerequisites

Here’s what you will need to send OpenObserve data to Splunk:

• A working Splunk instance with HEC enabled.
• Access to OpenObserve, configured to send data.
• An understanding of how to configure pipelines in OpenObserve.
• A Splunk token with the proper HEC input configuration.

Step-by-Step Guide

Step 1: Enable HEC in Splunk

1. Login to Splunk with an admin account.
2. Go to Settings > Data Inputs > HTTP Event Collector.
3. Click on New Token.
4. Provide a name (e.g., OpenObserve-HEC), select a source type (optional), and assign the token to an index (e.g., openobserve).
5. Click Next and then Finish.
6. Copy the Token Value; you’ll need it to authenticate HTTP requests.
7. Ensure HEC is enabled and listening on port 8088:

• Go to Settings > Data Inputs > HTTP Event Collector.
• Click Global Settings.
• Make sure HEC is enabled and SSL is configured, if required.

Step 2: Create an OpenObserve Pipeline Destination

1. Log into OpenObserve.
2. Click on the gear icon, and navigate to the management console.
3. Click on Pipeline Destinations, then click (?) Add Destination.
4. Add the details as provided in the below screenshot.

Click Save.

Step 3: Create a Pipeline to send the Data to Splunk HEC

1. In the left-hand sidebar, navigate to Pipelines.
2. Click on "Add Pipeline".
3. Add the Source Stream, and the Destination Stream. Note: The image shows that the data is being routed to both OpenObserve and Splunk.

4. Click on Save.

Using curl, send test events.

curl -u mosraf@openobserve.ai:5xHyvwn4C5agaLcD -k https://api.openobserve.ai/api/2xGA0Bu9jONzJurSbT4i44Ce9h0/testdata/_json -d '[{"level":"info","job":"prod","log":"test message for openobserve"},{"level":"error","job":"prod","log":"test message for openobserve"}]'

Step 4: Validate Event Ingestion in OpenObserve

Run a search in OpenObserve to confirm your event was ingested:

Step 5: Validate Event Ingestion in Splunk

Run a search in Splunk confirm your event was ingested:

You should see the event from OpenObserve in your search results.

Troubleshooting

• 403 Forbidden: Check if the HEC token is correct and has access to the target index.

• 500 Internal Server Error: Ensure the payload is valid JSON. Use tools like jq to validate.

• No data in Splunk: Verify the index exists, verify HEC is enabled, and verify the event format is correct.

Conclusion

Sending OpenObserve logs to Splunk via HEC is straightforward and powerful. It enables your team to leverage Splunk’s advanced analytics and alerting while still using OpenObserve for real-time observability. By bridging these two platforms, you ensure that all your logs—no matter the source—are available in one centralized place.

Additional Resources

1. Splunk HTTP Event Collector
2. OpenObserve GitHub