Announcements

We are excited to announce the release of our 5-latest orchestrators in October, powered by Ubuntu 24.04 LTS, and an extended testing period before the official 5-stable release in November. This update brings significant improvements, including enhanced security, support, and updated packages, to ensure your data pipelines remain efficient and future-proof.To provide customers with enough time to prepare and test their pipelines, we are delaying the release of these orchestrators to the 5-stable version by one month. During this time, you can test your pipelines against the 5-latest tag, giving you the opportunity to ensure compatibility with the new features before the stable release in November.Why Are We Upgrading to Ubuntu 24.04 LTS?The primary driver for this upgrade is the extended long-term support offered by Ubuntu 24.04 LTS, which provides security and maintenance updates until April 2029. By upgrading, we ensure that orchestrators remain secure, efficient, and compatible wit

We’re excited to announce that, starting with the November release, all DataOps.Live orchestrators will be based on Ubuntu 24.04 LTS, an upgrade from the previous version, Ubuntu 20.04 LTS. This update reflects our commitment to providing the most secure, reliable, and feature-rich environment for your data operations.Why the Upgrade?The primary motivation for this upgrade is to ensure continued support and security for the operating system and the underlying packages. Canonical will end support for Ubuntu 20.04 LTS in April 2025, which would mean no further updates or security patches for the OS packages beyond that date. Moving to Ubuntu 24.04 LTS extends that support until at least April 2029, giving you peace of mind that your data operations are running on a system that will continue to receive critical updates.Key Benefits of the UpgradeUpgrading the operating system brings several significant benefits that will enhance the overall performance, functionality, and security of the

As Snowflake shifts to stronger security by replacing passwords with Service Users and keypair authentication, DataOps.live is here to help make this transition as smooth as possible for our customers. Here's an update on where we’re at and what to expect.Key Updates1. Strengthening Orchestrator SecurityWe’re actively working on implementing keypair authentication for those orchestrators connecting to Snowflake that do not already support it. This will ensure that automated workflows are more secure by using Service User accounts instead of traditional passwords. The shift will enhance security and reliability in orchestrations, and while this is still in development, we’re moving steadily towards full integration to support smooth and secure operations.2. Service User Setup (Managed via SOLE)Right now, Service Users can be managed through SOLE, the DataOps.Live orchestrator for managing your Snowflake configuration. While the setup process is a bit more manual, it’s fully functional.

As part of our ongoing improvements to SOLE Gen and SOLE Gen for Data Products, we’ve introduced an opt-in feature designed to give you more control over how procedure and function statements are stored. Previously, all statements were stored directly inside the YAML configuration, but now you have the flexibility to offload longer statements to external files based on your preferences. This keeps your YAML files clean and more manageable, especially for large projects.Why This Feature is ValuableThe new --inline-statement-limit feature gives you more control and flexibility over how procedure and function statements are stored. By setting a limit, you can keep your YAML files clean and manageable, with longer statements automatically offloaded to external files while shorter ones remain inline. This is especially useful for projects with a mix of short and long procedures.How to Opt - In Enable External File Storage: Use the -o option to enable external file storage. This ensures that

In today's fast-paced data environments, maintaining visibility and control over your DataOps.live pipelines is essential.We are excited to introduce the first phase of DataOps.live Pipeline Observability through a hands-on lab, providing early access to data that can be used to monitor your DataOps.live pipelines.While this feature provides valuable insights into query performance, resource usage, and the connection between Snowflake queries and DataOps.live projects, it's currently in a lab phase.This is just the beginning, and we look forward to expanding and refining these capabilities in future iterations, guided by your feedback.What Is DataOps.live Pipeline Observability Lab?Powered by the MATE dataops_snowflake_usage package, our solution is designed to provide you with a comprehensive overview of your DataOps pipelines. By leveraging Snowflake's query history and the built-in DataOps query tags, you gain unprecedented visibility into how your DataOps.live Operate pipelines eng

Change the way your business makes decisions around data with a unified and harmonizedview of your Snowflake spend - It's FREE! ​ Spendview for Snowflake is a free module from DataOps.live observability, a core component of our platform.Spendview for Snowflake collects and aggregates all operational metadata about your Snowflake production and consumption to help you visualize and understand your accounts' potential spend drivers.To see all aspects of Spendview for Snowflake and what insights you can get using it, check out the below topics:Setup Overall Cost Summary Compute Activity and Cost Summary Storage Cost Summary Usage Cost Summary Account Management Data Security and Snowflake Privileges Sign up today