What is Snowflake?

Snowflake is a powerful cloud-based data warehousing solution designed for big data analytics. It enables organizations to store, process, and analyze vast amounts of structured and semi-structured data in a scalable and cost-effective manner. With features like automatic scaling, high performance, and robust security, Snowflake has become a preferred choice for modern data-driven businesses.

Snowflake also supports features like data sharing, time travel, and advanced security options, making it versatile for diverse use cases. Whether you are running complex analytics workloads or supporting real-time dashboards, Snowflake delivers high performance and flexibility.

Why Should We Monitor Snowflake via OpenTelemetry (OTel)?

Monitoring Snowflake’s performance is crucial to ensure optimal usage, cost management, and operational efficiency. By using OpenTelemetry (OTel), you gain:

• Unified Observability: Centralized monitoring of Snowflake along with other systems and services.
• Detailed Metrics and Traces: Deep insights into query performance, resource utilization, and anomalies.
• Custom Dashboards: Create tailored visualizations to track critical KPIs.
• Scalability: Handle monitoring data for large-scale Snowflake instances seamlessly.
• Proactive Alerting: Set up alerts to detect and address performance issues before they impact end-users.

With OpenTelemetry, organizations can gain a holistic view of their Snowflake instances, integrating it with other cloud services and applications. This is particularly valuable for troubleshooting and optimization in hybrid or multi-cloud environments.

Prerequisites

• Ubuntu server (20.04 or later recommended)
• Active Snowflake account
• Active OpenObserve account
• Understanding of OpenTelemetry

Step-by-Step Guide to Monitor Snowflake with OpenTelemetry

1. Create a Role in Snowflake

Begin by creating a role in Snowflake to facilitate monitoring access.

CREATE ROLE O2_ROLE;

You can create a role from snowsight as well as shown in the image.

2. Copy the Account Name from Snowflake

• Log in to your Snowflake account.
• Navigate to the Account tab.
• Copy the account name for later use. It will be in the format <account_locator>.<region>.<cloud_provider>.

3. Provide Permissions to the Role

Grant the necessary permissions to the role for monitoring purposes:

GRANT IMPORTED PRIVILEGES ON DATABASE SNOWFLAKE TO ROLE O2_ROLE;
GRANT SELECT ON ALL TABLES IN SCHEMA SNOWFLAKE.ACCOUNT_USAGE TO ROLE O2_ROLE;

4. Install OpenTelemetry Collector on Ubuntu

⚠️ Important Note: Snowflake receiver works with v0.92.0 ⚠️

Install the OpenTelemetry Collector to collect and forward telemetry data from Snowflake:

wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.92.0/otelcol-contrib_0.92.0_linux_amd64.deb
dpkg -i otelcol-contrib_0.92.0_linux_amd64.deb

5. Configure OpenTelemetry Collector

Create a config.yaml file with the following content:

You can create snowflake user, role, database and warehouse by following snowflake instructions.

receivers:
  snowflake:
    username: <SNOWFLAKE_USER>
    password: <PASSWORD>
    account: <accountName>.<accountRegion>
    warehouse: <WAREHOUSE>
    role: <ROLE_OF_SNOWFLAKE>
    schema: ACCOUNT_USAGE
    database: SNOWFLAKE
    collection_interval: 5m
    metrics:
      snowflake.storage.stage_bytes.total:
        enabled: true
      snowflake.storage.storage_bytes.total:
        enabled: true
      snowflake.storage.failsafe_bytes.total:
        enabled: true
      snowflake.rows_inserted.avg:
        enabled: true
      snowflake.database.query.count:
        enabled: true
      snowflake.pipe.credits_used.total:
        enabled: true
      snowflake.query.blocked:
        enabled: true
      snowflake.query.bytes_deleted.avg:
        enabled: true
      snowflake.billing.warehouse.cloud_service.total:
        enabled: true
      snowflake.query.compilation_time.avg:
        enabled: true
      snowflake.billing.warehouse.virtual_warehouse.total:
        enabled: true
      snowflake.billing.virtual_warehouse.total:
        enabled: true
      snowflake.query.bytes_written.avg:
        enabled: true
      snowflake.billing.cloud_service.total:
        enabled: true
      snowflake.billing.total_credit.total:
        enabled: true
      snowflake.database.bytes_scanned.avg:
        enabled: true
      snowflake.logins.total:
        enabled: true
      snowflake.query.data_scanned_cache.avg:
        enabled: true
      snowflake.rows_deleted.avg:
        enabled: true
      snowflake.rows_produced.avg:
        enabled: true
      snowflake.rows_unloaded.avg:
        enabled: true
      snowflake.session_id.count:
        enabled: true

exporters:
  debug:
    verbosity: detailed
    sampling_initial: 5
    sampling_thereafter: 200

  otlphttp/openobserve:
    endpoint: <OPENOBSERVE_ENDPOINT>
    headers:
      Authorization: <OPENOBSERVE_TOKEN>
      stream-name: <OPENOBSERVE_STREAM>

service:
  telemetry:
    logs:
      level: "debug"
  pipelines:
    metrics:
      receivers: [snowflake]
      exporters: [otlphttp/openobserve]

Replace:

• OPENOBSERVE_ENDPOINT with your OpenObserve endpoint (e.g., http://openobserve.example.com/api/org).
• OPENOBSERVE_TOKEN with your OpenObserve authentication token.
• OPENOBSERVE_STREAM with your desired stream name.

6. Restart the OpenTelemetry Collector

Restart the OpenTelemetry Collector service to apply the configuration:

systemctl restart otelcol-contrib

7. Configure Dashboards

As soon as metrics are ingested to OpenObserve, you can confirm them by verifying in OpenObserve dashboard. Configure dashboards based on the telemetry data collected. Include key metrics like:

• Query execution time
• Concurrency levels
• Resource usage trends
• Data ingestion and extraction rates
• Number of active users
• Average query response time

Ensure you set up drill-down capabilities to identify specific queries or operations contributing to performance bottlenecks.

To make it easy, we have pre-built the dashboards that can be found here.

8. Table: Snowflake Monitoring via OTel vs. Native Tools

Conclusion

Monitoring Snowflake with OpenTelemetry provides unparalleled observability and control over your data warehousing environment. With OTel’s robust features, you can go beyond Snowflake’s native tools to achieve a unified, scalable, and efficient monitoring solution. By following this guide, you can set up end-to-end monitoring, create insightful dashboards in OpenObserve, and ensure your Snowflake workloads perform at their best.

By leveraging the integration between Snowflake, OpenTelemetry and OpenObserve, organizations can optimize costs, proactively address performance issues, and make data-driven decisions with confidence. Start your monitoring journey today and unlock the full potential of your Snowflake data warehousing ecosystem.

Get Started with OpenObserve Today!

Sign up for a free trial of OpenObserve on our website. Check out our GitHub repository for self-hosting and contribution opportunities.