From Zero to Observability in 30 Minutes with OpenObserve

Timestamp: 00:00 - 05:48 Jake: I'm Jake Swiss, Go-To-Market Lead here at OpenObserve, joined by Manas Sharma, our developer relations. And today we're going to walk you through zero to observability in 30 minutes with OpenObserve. Here’s a quick rundown of today's agenda. I'm going to just walk through some real quick introductions, explain why OpenObserve. We're going to take a look at the platform with a platform review. Then I'm going to turn things over to Manas, and he's going to walk you through the demo from basically starting from scratch all the way up to being successfully set up with an OpenObserve deployment. And then, as I mentioned, we'll save some time at the end for Q&A. Great. For those of you who are not familiar with the platform, really this webinar is for you. We're going to get you up to speed real quickly. I think that a little context is in order, though, before we actually dive in. OpenObserve was formed in April 2022 in San Francisco, California by our founder, Prabhat Sharma. And really the catalyst here was to help solve some of the problems that he was encountering with the current state of logging. And we're going to double click into this in just a second. But beyond just knowing that there could be a better way to build a logging and observability platform off of the backs of that. We were also founded with an open ethos. We believe that open standards and open source are the future of observability. We have an open source project. We support otel standards for logs, metrics, and traces. It just comes part and parcel with everything we do. Let's actually talk a little bit about the problems that we've perceived. And those are that legacy platforms are increasingly failing to meet modern cloud native requirements, especially at scale. And this creates unsustainable financial and operation burdens. What ends up happening with large volumes of logging data is that they just hit with these prohibitively expensive bills. It drains their budgets. And to counteract that problem, many companies have to end up going on data diets, cutting out valuable telemetry, or what they assume is less valuable telemetry. But that creates gaps in your observability. And no one wants to deal with not being able to get to root cause when they're in the middle of an incident. Additionally, many of these legacy platforms deteriorate at scale. No one wants to wait three, four, or five minutes for the dashboards to load, especially in those incident scenarios. That's why OpenObserve was created to help both rein in the exploding costs of telemetry, but also to address the degraded performance at scale. And this is true both on premise and in cloud. OpenObserve as a platform is extremely efficient. We've been able to reduce observability costs of some of our customers by up to 90%. Performance, super fast across petabytes of data. And we also are able to manage scale based on our architecture, petabytes of data. Some of our customers are ingesting that much data every single day. This is just a quick list of some of our customers. But really, we have customers from Fortune 500 to real scrappy, innovative AI startups. A quick overview of the platform. We offer metrics, logs, and traces to help capture data and keep your locations and infrastructure healthy. But also, we have a built-in front-end monitoring for monitoring real user behavior. Our streams and pipelines allow users to control and transform their data within the platform. Visualization and alerts. Those are a front-end and help you address issues with your system. We're going to take a look at that in just a couple of minutes. Solutions help for quickly integrating with popular technologies. And again, that's included in the demo today. And then finally, actions are user-generated scripts for creating dynamic platform workflows. And the real limitation here is essentially your developer's imaginations. With that, I'm going to hand it over to Manas to start walking us through a demo.

Timestamp: 05:49 - 15:50 Manas: Thanks, Jake. And hi, everyone. Welcome and thanks for joining us today. I'm really excited to walk you and show OpenObserve in action today. We'll dive into the live demo shortly. So stay tuned and keep the questions coming in. Perfect. We also have the next poll in the chat. While you wait for the poll results, you can see that “Let's get started with the OpenObserve Cloud”, the fastest way to get going. We're going to quickly get started by logging in with our Google account. Once we log in successfully into OpenObserve Cloud, you'll land on this home screen. Here you'll see a bunch of metadata around your streams, events, ingested volume, and dashboards, functions associated with this cloud account. Other than this, on the right top corner, you'll see a small drop down. And here you'll see all the organizations associated to this cloud account. And you can see here that OpenObserve Cloud offers true multitenancy, which allows you to create separate organizations for complete isolation of resources, alerts, dashboards, and your data. And teams like yours can create different organizations for different business use case or even by teams in a single cloud account. Now let's switch to organization “zero to observability”. When you log in for the first time or you switch to a fresh organization, you'll see a notification banner on the top that in order to access pages other than IAM, data ingestion must be initiated. That leads us to our data sources page. And in the application, the data sources screen, you can jump to custom. And here you'll see all the log data sources supported by OpenObserve or telemetry type. For logs, you can see we support all the popular log forwarders such as fluent-bit, fluent-d,vector, otel, and similarly for metrics and traces. Well, today, for the next few minutes, we are going to set up a full Kubernetes observability stack with OpenObserve. In general, Kubernetes monitoring can feel overwhelming with metrics, events, and countless signals to track. It often seems complex. One major hurdle is tool sprawl. We have multiple tools for each telemetry type. This creates a fragmented stack and hard to manage and also adds some unwanted friction. But with OpenObserve, we are changing that. With just five simple commands available in our data sources tab, you can get all these signals, logs, metrics, traces, and even the events flowing into real-time into a single platform. All right, folks, before we dive into the hands-on part, let me quickly talk about the setup for our Kubernetes monitoring. We'll deploy a pre-configured collector and few key components like cert manager for secure connections and open telemetry operator for telemetry collection. We'll use Helm to deploy everything smoothly. And in just a few minutes, you'll see data flowing into OpenObserve. And let's jump right into it. I already have a three node Kubernetes cluster up and running with me here. Here I have a single namespace. It will jump to workloads and we'll see what are the workloads you have running. Here I have a single namespace called “observability pod”. Here I have a bunch of these pods running. I can actually check deployments. I have some services deployed here already. What we can do, we can just go to our OpenObserve data sources page and get started with the first command. I'm going to quickly copy this command. This command actually helps us to install cert manager in our Kubernetes cluster. We'll go ahead and paste this command here. This actually creates and installs the cert manager for us. Now we'll let go and update the Helm repo. We basically add the Helm charts repo to this cluster. Next, we'll copy the second command and paste this in our cluster. All right. We have added the Helm repo for this Kubernetes cluster. Next we're going to install some Prometheus CRDs which will be required by the OpenTelemetry operator as mentioned in the setup. We'll copy this command here and we're going paste this. All right. Here we have created this bunch of CRDs as you can see which will be responsible for our telemetry collection. Next we're going to go ahead and install the OpenTelemetry operator. I've copied this command and let's paste this here. Next we're going to create a new namespace. We're going to create a namespace called OpenObserve-Collector where we're going to deploy all the resources related to our collector for the collection purpose. Let's create this new namespace in our cluster. We have created this OpenObserve-Collector namespace and next we're going to deploy our OpenObserve-Collector with the Helm command. So let's do it. Here I've pasted the command directly from our data sources tab. Here you have all the configurations such as the exporter and the authorization header. Let's go ahead and hit enter. All right. If you have followed these steps mentioned in the data sources tab correctly, you'll see a message that if everything proceeds without errors then your cluster is now sending logs and metrics to OpenObserve-Collector. Cool. While we wait for our OpenObserve-Collector pods to come up and log metrics and other data flow in, let's take a quick look at what all we can monitor with the OpenObserve-Collector. So in a Kubernetes cluster by deploying this OpenObserve-Collector, it captures metrics like CPU, memory, disk across all the nodes, pods, and containers. It also helps you to keep a track of all the read and write operations of your persistent volumes and their persistent volume claims. Also it helps you to track the network performance. The transmit bandwidth, the receiving bandwidth, it allows you to capture those essential metrics too. And of course we have logs, events, and which are flowing into our from our cluster to OpenObserve. And Kubernetes logs and especially the event logs are quite critical as they help us to give a clear view of what is happening in our cluster. Well, let's go back to see if we have a stream available. What I'm going to do in the lift menu of the OpenObserve cloud application, we're going to switch to streams. While we switch to streams, we'll see that a default log stream is created in real time. And if I click on this explore button, this will navigate me to the logs page with the default stream selected. And here I'll see that Kubernetes logs have started flowing into OpenObserve instance. You can see the time frame here. We have started ingesting logs. And if you open any of this log record, you'll see a bunch of important fields like cluster, the container name, namespace name, node name, pod name, and all of these fields. You can simply go and with the query buttons you can actually write SQL queries here and search across these Kubernetes logs. And for deep diving into these Kubernetes logs, you can actually register for our intro to logging webinar. Let's go back to our streams tab and let's refresh here. And in real time, we're going to see that we see some more logs here. Here in the metrics tab, we see this bunch of Kubernetes related metrics. Here you can see all the container memory. And if I search for Kubernetes here, you can actually see some more metrics available. You can quickly see here all the Kubernetes namespace and different metrics. I can change this tab to see some more metrics like Kubernetes node CPU time, node CPU usage. Now to explore each of these metrics in detail, we can go to our metrics page available on the OpenObserve left menu. And here you'll see the metrics page. Here you can select whatever metric you want to explore. for example, I'll take a metric Kubernetes node CPU time. Here you'll see all the available attributes or fields associated with this metric. We can also see that all the available chart types which are available to visualize this metric. By default, this line chart is selected for this metric. And if we hit run query here, you'll see a simple line chart and actually the PromQL query which is running here. You can use PromQL to query these and filter these metrics here. We also have an add to dashboard button right here. If you create a panel here, you add some configurations, you can simply go ahead and add this metric panel to the dashboard of your choice. All right. We can quickly go ahead and see some more logs associated to this cluster. Also we can see that we have our node CPU up and we have the log file path from where these Kubernetes logs are originating.

Timestamp: 15:51 - 19:50 This is great. But again, ingesting data is just the start and real value lies in the dashboards. OpenObserve provides 18 built-in chart types plus custom charts for unique visuals. And to build dashboards from scratch, you can simply click on this new dashboard button and here you'll give a name to this. Let's give it a name. And when you hit save, you'll see the add panel button. You'll click on this add panel and this is the add panel screen which you'll rely on. The first thing you can do here is go ahead and select a stream type. So you can select whichever telemetry type you want to create a panel out of. If you select, for example, logs and then you'll get all these streams associated to these logs. Here we see our default stream with the logs. What we can do is we can pick this body field here and by default in OpenObserve, you can use SQL for querying and creating panels for logs and traces and we can use PromQL and SQL for metrics. By default, there is an auto mode here which allows you to hover over these available fields in the log record and you can add them to the axis just like we did for the body field. We already have the timestamp field for our x-axis. We can actually add some fields as breakdown, filter of our choice by just hovering over them. Here we have this line chart here. So if I just click on this, our simple line chart will be in place. While we can also do it, we can click on this drop down and here we'll see this intuitive UI drop down where you'll see all the common aggregations available. We'll see sum, average, min, max, and other. For example, if I'll go ahead and change this to sum, you'll simply see that here in the query editor, this aggregation is changed. With this auto mode in place, you can actually drive some quick insights without even needing to write a single query. Here we have a dashboard config button where we can configure each of these panels and customize each chart type. But anyways, coming back to our discussion of today, that is Kubernetes monitoring. So we know that we are tracking a bunch of signals with deploying our openobserve-collector. Along with that, we also provide some built-in ready-to-use Kubernetes monitoring dashboards in github. And the only prerequisite to those dashboards is that you have this telemetry flowing into your OpenObserve instance. What we're going to do to use these prebuilt dashboards, let's hit on this import button here. And here we're going to go ahead and click on the community dashboards button. This is going to navigate to our repository where you can find all the community dashboards. And here you'll find the directory called Kubernetes (openobserve-collector). From this directory, what you can do, you can simply click on any of these JSON configurations and we can copy this JSON configuration for this dashboard. And we can go back to our OpenObserve Cloud account and paste this JSON configuration right here. Once you paste this JSON configuration and simply click on import button, you'll see that a Kubernetes namespace dashboard which is created in your OpenObserve account. I needed nothing, you have this prebuilt dashboard and the only prerequisite was that you had the openobserve-collector deployed, which you can do by following these five simple commands. We are doing this live, you should also try this post webinar. All right, I can show you some more Kubernetes dashboards which we have available. You can import all of these dashboards and have insights like events later. Here you can also do it for multiple clusters. So you can select with this variable on the top, you can select any of the cluster and you'll have all the event related insights when these pods are created, started and whatnot. This is how within under 15 minutes, you can deploy and set up some insightful dashboards.

Timestamp: 19:51 - 27:20 All right. Well, the last step is alerting. Well, visualization is great, but alerts are the key. What we can do is that OpenObserve provides two kinds of alerts by default, that is scheduled alerts and real time alerts. You can go ahead. Scheduled alerts are great for filtering data and passing through for scheduled timeframes. And real time alerts allows you to send notifications instantly as soon as the data flows into the instance. Here we are going to set up a scheduled alert to track some events. But before we do that, we need to set up a template. Templates in OpenObserve allows you to decide how does the notification looks like when it's sent, when your alert is triggered. And to do so, we need to go ahead and click on the create template button. Under create template, you'll find a webhook and email. Because we support two kinds of destinations, that is webhook destinations and our good old email destinations for our alerts. What you can go ahead, I can give this alert template a name. Let's give it a name K8Alert. And let me paste a template here. Cool. So here I have pasted an alert template and this is primarily for Slack. But you can configure more, we support different webhooks. With webhooks, you can actually configure openobserve alerts with most of the common incident response tools and alerting tools such as Slack, MS Teams, PagerDuty, etc. Here in this alert template, as you can see, we also have fields like alert name and from a log record, we have used fields like {k8s_event_reason} and node name, {k8_cluster} name. You can completely customize this alert template for the notifications you are about to send. Well, we'll go ahead and quickly save this alert template here. Next, what we need to do is we're going to set an alert destination. OpenObserve supports two kinds of destinations, that is webhook and email. With email, you simply have to add your recipient's email where you want these notifications to be sent. With webhook, you need to configure this alert quickly. Let us give a name to this destination here. We're going to go ahead and select the template we've just created. And for the URL, I'm going to use Slack demo app URL. You can configure this demo app URL and you can grab this post endpoint. Let's paste this right here. I've pasted the Slack webhook URL and for the headers, we need to add a header for the alert notifications to be sent and we're going to configure my application JSON. All right, so here we have configured our alert webhook as a destination in OpenObserve. We're going to go ahead and save this. Once you have saved this destination successfully and we have a template in place, we can go back to our alerts to create a scheduled alert or a real-time alert as your requirement in OpenObserve. There we see that we have a log stream coming in. We can go ahead and we can pick any field of our choice to trigger an alert. For example, here I have a bunch of fields. For example, I have a severity field here or we can have a Kubernetes cluster related field trigger an alert. Let's do that. Let's go to our alerts quickly. Here I'm going to pick a scheduled alert. I'm going to click on add alert. This is the alert configuration window. Here you can add a name to this alert and once you do that, we can select the log type. We have to select the telemetry type and here you'll see all the available streams. We have a default stream here in place. These are all the alert related configurations. We have threshold which actually responsible. Here you actually satisfy an alert condition. When we satisfy an alert condition, so this is threshold allows you to how many times does the below alert condition should be satisfied to trigger this alert. Here it is three, we can reduce it to one for the feasibility of the demo. But similarly, we have other alert configurations available. We have period. Period is defined as how long this query is going to run or for what period of data this query is going to search. For example, here we have 10 minutes. This alert is going to query data for past 10 minutes. Every time this alert will scan through the data. We also have a frequency here which allows you to how frequently this alert, this query will run. We can set this or set this frequency to more than once if you want to like more than one minute. If you want the period to be larger every time this query runs and we also have a silent notification for which explicitly means that once the notification has been sent, the alert has been triggered. It will wait for 10 minutes to send the notification again. Well, let's go ahead and click on destination. We have a Kubernetes alert destination already configured. We'll just select it from here. Now with the conditions. This is where you specify what condition you want to trigger the alert for. Here as similar to dashboards, we do have a quick mode in place which is quite helpful and you can actually click on here and you can select whichever stream or whichever node type you want to select. For example, if we select any field here, for example, you select a severity field and you can actually select that it equals to and you can specify the value here. Similarly, we do have a SQL mode. With SQL mode, you can write some queries to filter through the data and perform some aggregations over the original data, which is coming in transform that data with the help of the SQL query and then satisfy the condition. This is what we can simply do. For example, here I have an example, but also as you can see, this is an query example, which you can run for the critical events running in your Kubernetes cluster. We have logs coming in and here what we can do is that from this cluster, we have the Kubernetes cluster name as cluster one and we have specified that the Kubernetes event reason is either fail or backoff and we can simply save this SQL here for our alert condition to match. This is how you can configure some insightful alerts as of now. Cool. So let's do this. And once we trigger this alert, I'm going to show you how will that look like. Let's jump to our Slack first. Whenever a critical Kubernetes alert is triggered, you'll receive notifications like “BackOff” and with the Kubernetes cluster name, the node affected and the event reason and also the complete error message associated with this alert. This way you can actually create some insightful alerts and send them to notifications to any destination of your choice.

Timestamp: 27:21 - 28:07 Alright, there are a bunch of things related to IAM and Teams. In OpenObserve Cloud, you can actually invite your teammates and co-workers by simply entering their email from here. Once you invite those, what you can do is you can jump on this logs page and here you'll see a share link button. With this share link button, you can share this and get this link copied and once you share this with your co-worker, they'll be able to get this exact log screen with the selected date and time from this date and time picker in their browser. They can completely do this. Similarly for dashboards, we do have a share link button. So similar to this, you can share this with your co-workers and share these dashboards among your teams.

Timestamp: 28:08 - 29:34 Alright, thank you so much, Manas. Let's just quickly recap what we covered. We covered how to instrument the OpenObserve-Collector in particular for a Kubernetes cluster, how to individually navigate your metrics and logs once they start to get ingested into the platform, how to stand up turnkey Kubernetes dashboards, and then how to configure alerts. And there's obviously a fair degree of customization, but with every single, you know, incident flow, there's going to need to be customization depending on the tools and processes that you have in place. We encourage everyone here to give it a try. If you're not already using OpenObserve, we offer a 14-day free trial on cloud. There is access to every single capability in the platform and it gives you a really good opportunity to kind of kick the tires and get a good sense of how OpenObserve can work for you. Thank you everyone again for attending. I just want to remind you that we're active on our community Slack, on X, LinkedIn. Please reach out, let us know your questions, let us know your success stories. Until the next time, thank you so much.