In today's fast-paced data environments, maintaining visibility and control over your DataOps.live pipelines is essential.

We are excited to introduce the first phase of DataOps.live Pipeline Observability through a hands-on lab, providing early access to data that can be used to monitor your DataOps.live pipelines.

While this feature provides valuable insights into query performance, resource usage, and the connection between Snowflake queries and DataOps.live projects, it's currently in a lab phase.

This is just the beginning, and we look forward to expanding and refining these capabilities in future iterations, guided by your feedback.

What Is DataOps.live Pipeline Observability Lab?

Powered by the MATE dataops_snowflake_usage package, our solution is designed to provide you with a comprehensive overview of your DataOps pipelines. By leveraging Snowflake's query history and the built-in DataOps query tags, you gain unprecedented visibility into how your DataOps.live Operate pipelines engage with Snowflake.

Key Features

Here’s how this package can empower your DataOps workflows:

• Link Queries to Pipelines: One of the most powerful features of this solution is the ability to connect the dots between the queries running in Snowflake and the projects and pipelines in DataOps.live. This allows for a clearer understanding of how each pipeline is interacting with your data.
• Monitor Query Performance: Gain real-time insights into your pipeline's query activities, enabling you to identify and address any bottlenecks quickly. By tracking the performance of queries as they execute, you can ensure that your pipelines remain efficient.
• Identify Resource Usage: Understand which Snowflake warehouses are being utilized most frequently. This information enables you to manage resources more effectively and ensure that your system scales appropriately based on demand.

Getting Started

We’ve simplified the setup process to get you started quickly:

1. Clone the Project: Head over to the project at DataOps Pipeline Observability and clone it. 

   

    

2. Configure Your Environment: You'll need to make some edits to two key configuration files: agent_tag.yml and variables.yml located in pipeline/includes/config.

   

    

3. Run a Pipeline: Once configured, execute a pipeline, and start gaining insights immediately.

   

   

    

With DataOps.live Pipeline Observability Lab, you’ll not only monitor but optimize your DataOps workflows, ensuring smooth, efficient, and scalable interactions with Snowflake. Get started today and take control of your pipelines like never before!