How to Send Java Application Logs to OpenObserve Using Log4j2

Introduction

Effective logging is not just about recording events; it's about gaining actionable insights!

Java applications generate various types of logs - from application-specific messages to framework logs and JVM information. Managing these logs effectively becomes challenging as applications scale across multiple services and environments.

Log4j2: The Powerhouse of Java Logging

Log4j2 stands as the industry-standard logging framework for Java applications, offering significant advantages:

• High performance with asynchronous logging capabilities
• Flexible configuration through XML, JSON, or YAML
• Extensible architecture with custom appenders and layouts

At its core, Log4j2 works through:

• Loggers that capture messages from your code
• Appenders that determine where logs are sent
• Layouts that format your log messages

For our OpenObserve integration, we'll configure an HTTP appender with JSON layout to send structured logs directly to the OpenObserve API.

Why OpenObserve for Your Logs?

OpenObserve serves as the perfect observability backend for your Java applications:

• Cost-effective storage - up to 140x more efficient than Elasticsearch
• Powerful querying with both SQL and PromQL support
• User-friendly interface for intuitive log analysis
• Built-in alerting capabilities

By combining Log4j2 with OpenObserve, you gain centralized log management with powerful search capabilities, allowing you to correlate logs with metrics and traces for complete observability of your Java applications.

Prerequisites

Before we begin, ensure you have:

• Java 11 or newer installed
• Maven 3.6+ for dependency management
• An OpenObserve account (cloud or self-hosted)

Clone the Repository

Start by cloning our example repository:

git clone https://github.com/openobserve/openobserve-log4j.git
cd openobserve-log4j

This repository contains a complete example project with all the necessary configurations to integrate Log4j2 with OpenObserve.

Configure OpenObserve Connection

Before running the example, you need to update the Log4j2 configuration to point to your OpenObserve instance:

1. Open src/main/resources/log4j2.xml
2. Locate the HTTP appender configuration:

<Http name="OpenObserve" url="https://your-openobserve-instance/api/organization_name/stream_name/_json">
    <Property name="Content-Type" value="application/json"/>
    <!-- Basic authentication -->
    <Property name="Authorization" value="Basic ${base64:your_username:your_password}"/>
    
    <!-- JSON Layout for structured logging -->
    <JsonLayout compact="true" eventEol="true" properties="true">
        <KeyValuePair key="service" value="your-service-name"/>
    </JsonLayout>
</Http>

3. Replace the following placeholders:
   • your-openobserve-instance with your OpenObserve domain
   • organization_name with your organization name in OpenObserve
   • stream_name with your desired log stream name
   • your_username and your_password with your OpenObserve credentials
   • your-service-name with your application's name

Understanding the Configuration

The Log4j2 configuration includes several key components:

<!-- Console Appender for local debugging -->
<Console name="Console" target="SYSTEM_OUT">
    <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
</Console>

<!-- Async wrapper for better performance -->
<Async name="AsyncOpenObserve">
    <AppenderRef ref="OpenObserve"/>
</Async>

<Loggers>
    <Root level="info">
        <AppenderRef ref="Console"/>
        <AppenderRef ref="AsyncOpenObserve"/>
    </Root>
</Loggers>

1. Console Appender: Outputs logs to the console for local debugging
2. HTTP Appender: Sends logs to OpenObserve via its JSON API
3. Async Wrapper: Improves performance by sending logs asynchronously
4. JSON Layout: Formats logs as structured JSON documents

The HTTP appender is configured to send logs directly to OpenObserve's API endpoint with basic authentication. The JSON layout ensures that logs are properly structured for efficient querying in OpenObserve.

Example Application Code

The repository includes a sample application that demonstrates various logging scenarios:

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.logging.log4j.ThreadContext;

public class OpenObserveLog4jExample {
    private static final Logger logger = LogManager.getLogger(OpenObserveLog4jExample.class);
    
    public static void main(String[] args) {
        // Add context data (MDC)
        ThreadContext.put("user_id", "12345");
        ThreadContext.put("session_id", "abcdef");
        
        // Log messages at different levels
        logger.debug("This is a debug message");
        logger.info("This is an info message");
        logger.warn("This is a warning message");
        
        try {
            // Simulate an exception
            int result = 10 / 0;
        } catch (Exception e) {
            logger.error("An error occurred", e);
        }
        
        // Clean up context data
        ThreadContext.clearAll();
        
        logger.info("Application shutting down");
    }
}

This example demonstrates:

• Setting thread context data for request tracking
• Logging at different severity levels
• Exception handling with stack trace capture

Build and Run the Example

Once you've updated the configuration, build and run the example:

mvn clean package
java -jar target/openobserve-log4j-1.0-SNAPSHOT.jar

The example application will generate logs and send them to your OpenObserve instance.

Viewing Logs in OpenObserve

After running the example, you can view your logs in OpenObserve:

1. Log in to your OpenObserve instance
2. Navigate to the Logs section
3. Select the stream you configured in the Log4j2 configuration

You should see structured logs from your application, complete with timestamps, log levels, and any custom fields you added.

Advanced Features

Adding Custom Fields

You can enrich your logs with custom fields by adding KeyValuePair elements to the JsonLayout:

<JsonLayout compact="true" eventEol="true" properties="true">
    <KeyValuePair key="service" value="your-service-name"/>
    <KeyValuePair key="environment" value="production"/>
    <KeyValuePair key="version" value="1.0.0"/>
</JsonLayout>

Including Thread Context Data

Thread Context (also known as MDC - Mapped Diagnostic Context) allows you to add contextual information to your logs:

ThreadContext.put("user_id", "12345");
ThreadContext.put("session_id", "abcdef");
logger.info("User action performed");

These context values will be included in the JSON sent to OpenObserve, making it easy to filter logs by user, session, or other contexts.

Troubleshooting

Common Issues

1. Connection Refused

   Ensure your OpenObserve URL is correct and accessible from your application.

2. Authentication Failed

   Verify your username and password are correctly encoded in the configuration.

3. No Logs Appearing

   Check that your log level is set appropriately in the Root logger configuration.

Enabling Debug Mode

To troubleshoot Log4j2 itself, enable debug mode in your configuration:

<Configuration status="DEBUG">

This will output detailed information about Log4j2's initialization and operation.

Conclusion and Next Steps

Now that you've successfully integrated Log4j2 with OpenObserve, you have a powerful foundation for centralized logging of your Java applications. Here are some quick tips to take your implementation to the next level:

Best Practices for Production

• Use environment variables for credentials instead of hardcoding them
• Configure appropriate log levels (WARN/ERROR for production, INFO for staging, DEBUG for development)
• Implement async logging to minimize performance impact on your applications
• Add retry logic to handle network interruptions: <Http name="OpenObserve" url="..." ignoreExceptions="true">

Enhance Your Observability

• Create dashboards in OpenObserve to visualize error rates and application behavior
• Set up alerts based on log patterns to proactively identify issues
• Add context data to your logs for better filtering and troubleshooting
• Correlate logs with traces by including trace IDs in your log events

The combination of Log4j2's flexible logging framework and OpenObserve's powerful analysis capabilities gives you complete visibility into your Java applications, helping you identify and resolve issues faster while keeping storage costs low.