How to send GCP logs to OpenObserve
Prabhat Sharma
June 21, 2024
4 min read
Don’t forget to share!
Introduction
Managing and analyzing logs effectively is crucial for any organization running applications on Google Cloud Platform (GCP). One powerful way to handle GCP logs is by sending them to OpenObserve via Google Cloud Pub/Sub via Log Router Sink. This allows you to create real-time log processing workflows and integrate logs with various downstream systems. In this blog post, we will walk through the steps to set up a Log Router Sink to send GCP logs to Pub/Sub and create a subscription to the topic with a push endpoint.
Google cloud logs -> Log router sink -> Pub/Sub -> OpenObserve
Prerequisites
Before we begin, ensure you have the following:
- A GCP account with the necessary permissions.
- A Pub/Sub topic created in your GCP project.
- Basic knowledge of GCP Console and command-line tools.
- An endpoint URL for OpenObserve.
Step 1: Create a Pub/Sub Topic
First, we need a Pub/Sub topic to which the logs will be sent. You can create a Pub/Sub topic using the GCP Console or the gcloud command-line tool.
Using GCP Console:
- Navigate to the Pub/Sub section in the GCP Console.
- Click on "Create Topic."
- Provide a name for your topic and click "Create."
Using gcloud Command-Line Tool:
gcloud pubsub topics create my-log-topic
Step 2: Create a Subscription to the Topic
Next, we need to create a subscription to the Pub/Sub topic. This subscription will have a push endpoint that sends logs to OpenObserve.
Using GCP Console:
- Go to the Pub/Sub section in the GCP Console.
- Click on the topic you created.
- Select "Create Subscription."
- Provide a name for your subscription.
- In the "Delivery type" section, select "Push."
- Enter the OpenObserve endpoint URL in the "Push endpoint" field. You can get this from OpenObserve ingestion page.
- Click "Create."
Get OpenObserve GCP endpoint URL from the ingestion page. Here is a screenshot:
Now create the subscription:
Using gcloud Command-Line Tool:
gcloud pubsub subscriptions create my-log-subscription \
--topic=my-log-topic \
--push-endpoint=https://api.openobserve.ai/gcp/myorg/mystream/_sub?API-Key=aBJhYkqhdEBvcub2JzZXJ2ZS5haTp0YUtKZEJRbVc1ZkNLSnFS
Step 3: Set Up a Log Router Sink
A Log Router Sink will route logs from Cloud Logging to the Pub/Sub topic. To create a sink, follow these steps:
Using GCP Console:
- Go to the Logging section in the GCP Console.
- Select "Logs Router" from the sidebar.
- Click on "Create Sink."
- Provide a name for your sink.
- In the "Sink Destination" section, select "Cloud Pub/Sub" and choose the topic you created earlier.
- In the "Sink Details" section, specify the logs you want to include in the sink by creating a filter. For example, to include all logs, use:
logName: "projects/YOUR_PROJECT_ID/logs/" - Click "Create Sink."
Using gcloud Command-Line Tool:
You can also create a sink using the gcloud tool:
gcloud logging sinks create my-log-sink \
pubsub.googleapis.com/projects/YOUR_PROJECT_ID/topics/my-log-topic \
--log-filter='logName: "projects/YOUR_PROJECT_ID/logs/"'
Step 4: Grant Permissions to the Log Router
The Log Router needs permission to publish messages to the Pub/Sub topic. This involves granting the Pub/Sub Publisher role to the Log Router service account.
Using GCP Console:
- Go to the Pub/Sub section in the GCP Console.
- Select the topic you created.
- Click on the "Permissions" tab.
- Click "Add" to add a new member.
- In the "New members" field, add the Log Router service account. The format is:
serviceAccount:cloud-logs@YOUR_PROJECT_ID.iam.gserviceaccount.com - In the "Role" dropdown, select "Pub/Sub Publisher."
- Click "Save."
Using gcloud Command-Line Tool:
gcloud pubsub topics add-iam-policy-binding my-log-topic \
--member=serviceAccount:cloud-logs@YOUR_PROJECT_ID.iam.gserviceaccount.com \
--role=roles/pubsub.publisher
Step 5: Verify Log Delivery
If you have all set up correctly, logs from GCP should now be sent to the Pub/Sub topic and pushed to the OpenObserve.
Conclusion
By following these steps, you have successfully set up a Log Router Sink to send GCP logs to Pub/Sub and created a subscription with a push endpoint to OpenObserve. This integration allows you to create robust log processing pipelines and integrate with various other GCP services and third-party tools.
Happy logging!
Feel free to ask any questions or share your thoughts in the comments below. If you found this guide helpful, don’t forget to share it with your network!
About the Author
Prabhat Sharma
Prabhat Sharma is the founder of OpenObserve, bringing extensive expertise in cloud computing, Kubernetes, and observability. His interests also encompass machine learning, liberal arts, economics, and systems architecture. Outside of work, Prabhat enjoys spending quality time playing with his children.
