Unveiling Microsoft 365 Insights: From Diagnostics to Actionable Observability
Chaitanya Sistla
December 03, 2024
7 min read
Don’t forget to share!
Managing and analyzing logs from Microsoft 365 services like Graph APIs, audit logs, sign-ins, and managed identities is critical for ensuring security, compliance, and performance optimization. While Microsoft 365 provides rich telemetry, collecting, processing, and centralizing these logs for real-time observability can be challenging without a scalable pipeline.
This guide explains the necessity of structured log ingestion, how it empowers security and operational excellence, and provides a step-by-step setup for ingesting Microsoft 365 logs into OpenObserve via Azure Event Hub.
Why Ingest Microsoft 365 Logs?
- Centralized Observability: Consolidate logs from diverse Microsoft 365 services for a unified view.
- Proactive Security Monitoring: Identify anomalies like repeated failed sign-ins or suspicious API calls.
- Regulatory Compliance: Ensure audit trails are complete and easily accessible for compliance audits.
- Operational Insights: Optimize managed identities and Graph API usage to streamline operations.
How This Helps
Ingesting Microsoft 365 logs into OpenObserve enables:
- Advanced Dashboards: Build real-time, customizable dashboards to monitor activity trends.
- Improved Security Posture: Detect and respond to security incidents promptly.
- Simplified Compliance Auditing: Easily query and analyze audit logs for regulatory requirements.
- Scalability: Use Azure Event Hub for handling high log volumes efficiently.
Prerequisites
Before starting, ensure you have:
- An active Azure subscription.
- Familiarity with Azure Functions.
- Visual Studio Code installed with the Azure Functions extension. Refer to this guide for deployment steps.
Step-by-Step Setup
1. Create an Azure Event Hub
Create an Event Hub Namespace:
- Go to the Azure Portal and search for Event Hubs.
- Click + Add to create a new namespace.
- Provide details:
- Resource Group: Use an existing group or create a new one.
- Namespace Name: Choose a unique name (e.g.,
m365-eventhub). - Pricing Tier: Select one based on your log volume.
- Click Review + Create and confirm to deploy.
Add an Event Hub:
- After the namespace is created, navigate to it and click + Event Hub.
- Name the Event Hub (e.g.,
o2) and leave default settings for Partition Count and Message Retention. - Save the Event Hub configuration.
Create a Shared Access Policy:
- Under the namespace, go to Shared Access Policies.
- Add a policy with Manage permission.
- Save the policy and note the Connection String.
2. Enable Microsoft 365 Diagnostics Logging
- Navigate to the Microsoft 365 Entra Admin Center.
- In the Azure Portal, go to Monitoring health > Diagnostic Settings.
- Configure a new diagnostic setting:
- Name the setting (e.g.,
m365-diagnostics). - Select categories such as AuditLogs and SignInLogs.
- Choose Stream to an Event Hub as the destination.
- Provide the Event Hub namespace and connection string.
- Name the setting (e.g.,
3. Deploy Azure Functions to Process Logs
Azure Functions will forward logs from the Event Hub to OpenObserve.
To create an azure function, you will need to first create the Function App by searching for it in your azure portal and select a tier along with your runtime stack.
- Clone the Function Code:
- Configure the Function:
- Open the code in VS Code.
- Update the configuration file (
local.settings.json) with:- Event Hub Connection String: The value noted earlier.
- In index.js within function(s) replace <O2_ENDPOINT> with your OpenObserve endpoint.
- Deploy the Function:
- Use the Azure Functions extension in VS Code:
- Click Deploy to Function App.
- Select your Azure subscription and resource group.
- Make sure to add your OpenObserve auth details in the App's environment.
- Follow the deployment guide for detailed instructions.
- Use the Azure Functions extension in VS Code:
4. Validate and Ingest Logs into OpenObserve
- Ensure that the Azure Function is running and forwarding logs to OpenObserve. Your stream names will be whatever you set them in your index.js. Make sure you repeat the same for other functions (index.js) too!
- In OpenObserve, search for logs using queries tailored to your log categories.
5. Build Dashboards and Alerts in OpenObserve
- Dashboards:
- Create panels to visualize key metrics like:
- Failed Sign-Ins
- Graph API call usage
- Audit log activity
- You can download the pre-built dashboards from our repository.
- Create panels to visualize key metrics like:
- Alerts:
- Configure alerts for security anomalies or unusual activity spikes.
- Integrate alerts with email or messaging services for real-time notifications. Refer to Email Alerts blog for more details.
- Sample html template for the alert could be as below.
<!DOCTYPE html>
<html>
<head>
<style>
body {
font-family: Arial, sans-serif;
line-height: 1.6;
color: #333;
}
.email-container {
max-width: 600px;
margin: 0 auto;
border: 1px solid #ddd;
border-radius: 8px;
overflow: hidden;
box-shadow: 0 4px 8px rgba(0,0,0,0.1);
}
.email-header {
background-color: #0078d7;
color: #fff;
padding: 20px;
text-align: center;
}
.email-content {
padding: 20px;
}
.email-footer {
background-color: #f4f4f4;
text-align: center;
padding: 10px;
font-size: 12px;
color: #666;
}
.highlight {
font-weight: bold;
color: #0078d7;
}
</style>
</head>
<body>
<div class="email-container">
<div class="email-header">
<h1>Non-US Login Alert</h1>
</div>
<div class="email-content">
<p>Hello,</p>
<p>A login activity has been detected outside the US. Here are the details:</p>
<table style="width:100%; border-collapse: collapse; margin: 20px 0;">
<tr>
<td style="padding: 8px; border: 1px solid #ddd;">User Name</td>
<td style="padding: 8px; border: 1px solid #ddd;" class="highlight">{identity}</td>
</tr>
<tr>
<td style="padding: 8px; border: 1px solid #ddd;">User Principal Name</td>
<td style="padding: 8px; border: 1px solid #ddd;" class="highlight">{properties_userprincipalname}</td>
</tr>
<tr>
<td style="padding: 8px; border: 1px solid #ddd;">Location</td>
<td style="padding: 8px; border: 1px solid #ddd;" class="highlight">{location}</td>
</tr>
<tr>
<td style="padding: 8px; border: 1px solid #ddd;">City</td>
<td style="padding: 8px; border: 1px solid #ddd;" class="highlight">{properties_location_city}</td>
</tr>
<tr>
<td style="padding: 8px; border: 1px solid #ddd;">State</td>
<td style="padding: 8px; border: 1px solid #ddd;" class="highlight">{properties_location_state}</td>
</tr>
<tr>
<td style="padding: 8px; border: 1px solid #ddd;">IP Address</td>
<td style="padding: 8px; border: 1px solid #ddd;" class="highlight">{calleripaddress}</td>
</tr>
<tr>
<td style="padding: 8px; border: 1px solid #ddd;">App Name</td>
<td style="padding: 8px; border: 1px solid #ddd;" class="highlight">{properties_appdisplayname}</td>
</tr>
<tr>
<td style="padding: 8px; border: 1px solid #ddd;">Browser</td>
<td style="padding: 8px; border: 1px solid #ddd;" class="highlight">{properties_devicedetail_browser}</td>
</tr>
<tr>
<td style="padding: 8px; border: 1px solid #ddd;">Operating System</td>
<td style="padding: 8px; border: 1px solid #ddd;" class="highlight">{properties_devicedetail_operatingsystem}</td>
</tr>
<tr>
<td style="padding: 8px; border: 1px solid #ddd;">Time</td>
<td style="padding: 8px; border: 1px solid #ddd;" class="highlight">{properties_createddatetime}</td>
</tr>
</table>
<p>If you suspect this is unauthorized access, please take immediate action to secure your account.</p>
<p>Best regards,<br>Security Team</p>
</div>
<div class="email-footer">
This is an automated message. Please do not reply.
</div>
</div>
</body>
</html>
Unlocking the Full Potential of Microsoft 365 Logs
| Feature | Using Azure Event Hub & OpenObserve | Without Centralized Ingestion |
|---|---|---|
| Centralized Monitoring | Unified view of all logs | Limited to native tools |
| Real-Time Alerts | Advanced alerting capabilities | Basic |
| Scalability | High-throughput log processing | Limited by native limits |
| Customization | Fully customizable dashboards | Predefined views only |
| Regulatory Compliance | Complete audit logs at your fingertips | Fragmented data |
By leveraging Azure Event Hub and OpenObserve, you gain centralized, actionable insights into your Microsoft 365 environment. This pipeline ensures real-time monitoring, proactive security, and streamlined compliance, enabling you to make informed decisions with confidence. Ready to take your observability to the next level? Dive into your logs and start building impactful dashboards today!
