Product Updates

Hi DataOps.live User,   Welcome to the August newsletter! Find out how our latest developments enhance your DataOps experience. Save time investigating Snowflake spend. Make the most of DDE. Enhance reliability and fault tolerance. And more. Read on.   Enhancements in Spendview for SnowflakeCompute is often the top cost area for optimization related to a Snowflake Warehouse, where the most common problems are query performance and idle warehouses.    Spendview for Snowflake brings awareness to business stakeholders and data team leads to understand their current resource usage. Spendview provides big saving opportunities in a matter of clicks as it empowers your data team with self-service Snowflake consumption and spending metrics.   Sign up now for our live webinar on optimizing your Snowflake spend and join DataOps.live and XponentL Data discussion on how you can perfect your budget by following best practices and cost-saving strategies.  Optimize the Potential of DataOps Development with dbt Make the most of the DataOps Development Environment (DDE) by using the latest dbt versions for better production efficiency. DDE is most effective when used with the recent dbt versions, which significantly boosts the development speed.  In DDE, specify the version by setting the DATAOPS_FEATURE_MATE_DBT_VERSION environment variable, for example, DATAOPS_FEATURE_MATE_DBT_VERSION=1.4. Read more.... In the DataOps pipeline, set the variable in your project’s variables.yml file.  Sign up for the DDE private preview by reaching out to ours Support team.  Enhancements in SOLE Generator Managing your existing Snowflake databases through the DataOps.live platform just got a bit easier. Besides supporting database, function, procedure, schema, table, and task, we now also support sequence, tag, stream, and file format.  Start using the SOLE Generator CLI (Command-line Interface) tool that helps you generate a YAML configuration for SOLE objects.    Community#TrueDataOps Podcast Season 2 is here! Join us with our special guest, Bob Mugila, formerly the Chief Executive Officer of Snowflake. We will cover his new book, The DataPreneurs (https://www.thedatapreneurs.com/), the new age of AI, and all things Data related! Make sure to add it to your calendar. You do not want to miss this episode!  RSVP Here AcademyThe latest course - Fundamentals of DataOps.live - just went live at our Academy. In this course, we go step by step through the process of ingesting a new data source using our Stage Ingestion Orchestrator. Topics covered include creating multiple objects with SOLE, autogenerating sources from ingestion tables, creating MATE jobs to run only specified models, creating a new pipeline -ci.yml file, and more. Enroll today!  Did You Know?…that you can incorporate a fallback job that retries your main job again but with a different container image or with alternative configuration adjustments. By incorporating a fallback job into a pipeline, developers and data engineers can enhance the reliability, robustness, and fault tolerance of the system.  Read more at Create Fallback Jobs   That’s all for now. Until next time, - the DataOps.live Product Team

Hi DataOps.live User,   Welcome to the July newsletter! Gain insights on how the important developments announced at the Snowflake Summit 2023 may greatly benefit DataOps.live projects and operations. Enjoy the collaboration with your DataOps peers on our Community platform. Read on.   Snowflake Summit Announcements and DataOps Our powerful Snowflake Object Lifecycle Engine (SOLE) and Modelling and Transformation Engine (MATE) help data engineers automate and streamline the steps in the development lifecycle, allowing them to focus on Snowflake’s functional capabilities.  With the exciting announcements of Snowflake Summit 2023, we are assessing how the latest technical developments by Snowflake further enhance DataOps.live. Potential advancements include: Enhance SOLE to support Snowpark container services and libraries management  Further simplify building user interfaces using Streamlit and enhance the management of Snowpark libraries  Update SOLE to ease building, testing, distributing, and monetizing of your Snowflake data apps on Snowflake Marketplace and ensure MATE is compatible Create Iceberg and Dynamic tables within SOLE and MATE Allow using geospatial and geometric data Make costs predictable and transparent through fine-tuned cost and budget management You can already use our DataOps Development Environment (DDE) to develop data science workloads with Snowpark and build data applications with Streamlit. With the planned enhancements, things will become even easier going forward. Further leverage DataOps.live Spendview for Snowflake to visualize and investigate your Snowflake accounts’ spend drivers.  Join Our Community Today  Join the DataOps.live community today!  We are excited to announce the launch of our NEW community platform! Designed with customers, partners, and DataOps professionals in mind, the DataOps.live community is the place to: Share your expertise and ask questions  Access a wealth of knowledgebase articles to gain a deeper understanding of the DataOps.live platform  Access our training and documentation materials free of charge.  Interact with other customers and get support from the DataOps.live team, who will be actively participating in the forum Sign up and add your avatar to our new community platform by Aug 1st to earn an “Early Adopter Badge”.   Early Adopter BadgeJoin Today Did You Know?… that there is a dedicated Spendview for Snowflake space on our new Community platform where you can access valuable insights on best practices, share your expertise, and engage in discussions. If you haven’t signed up yet, join and share your experience with us.  That’s all for now. Until next time, - the DataOps.live Product Team 

Hi DataOps.live User,  Welcome to the June newsletter. Discover how our free observability tool and new features enhance your DataOps experience. Save time investigating Snowflake spend. Streamline development, reduce complexity, and automate data product building and deployment on the Snowflake data cloud.  Spendview for Snowflake Spendview for Snowflake is a module from DataOps.live observability that we now make available for free. It offers a centralized view of your Snowflake compute, usage, and storage costs and consumption. It is accessible to anyone with a Snowflake account. Leveraging Spendview for Snowflake saves time investigating Snowflake spend anomalies and user adoption. It offers root cause analysis that pinpoints potential cost drivers and thus helps you monitor your budget and optimize your Snowflake spending and consumption. Learn more…   MATE with dbt 1.5  You can now use the Modelling and Transformation Engine (MATE) with dbt-core 1.5 and benefit from all the significant additions of this new version. You can enable dbt-core 1.5 by simply setting the variable DATAOPS_FEATURE_MATE_DBT_VERSION to 1.5. Learn more…  Improvements to DDE (DataOps Development Environment) Experience the latest enhancements to our fully integrated development environment, available in Private Preview. Our DDE, accessible through any browser and without any setup, provides a seamless, optimized development experience for various DataOps use cases.  With DDE, you can now switch between different dbt versions and use a pre-commit tool that helps identify issues before committing changes to your project.Learn more...    Community  In episode 17th of the #TrueDataOps Podcast, we are joined by our guest Sanjeev Mohan! Sanjeev, Principal at SanjMo and a former Gartner Research VP for Big Data and Advanced Analytics covers all things data, including analytics, data governance, data management, and data observability.  Watch the recording...   Did you Know…  … that we have introduced Data Products as a new framework in DataOps.live. You can now easily build an ecosystem of simple and composite data products on the Snowflake data cloud and benefit from greater accuracy and accessibility of data.  Data Products is currently available as a Private Preview, allowing early access to selected users. Email us and sign up for the Data Product Private Preview to get access.  Read more at Data Products.   That’s all for now. Until next time, - the DataOps.live Product Team

DataOps emphasizes automation and integration, enabling seamless and efficient data workflows. The DataOps CLI now offers easier Snowflake database management. And new third-party tool integrations help you collaborate with your team members better on your projects. Default Parameter Values in SOLE Generator  The SOLE Generator is our command-line interface generating the Snowflake confirmation from incoming DDL. Now you can create a template for a SOLE configuration holding your desired DataOps parameter values. When “dataops solegen” runs, it merges the templates with the incoming DDL. Learn more   More Third-party Integrations with Our Platform You can now benefit from the integration of more third-party tools with the DataOps platform. You can now use Google Hangouts Chat, Email on Push, Slack, Microsoft Teams, and Cisco Webex in all your DataOps project. Learn more    Community  Join our #TruedataOps Podcast Newsletter to stay up to date on the latest! Sign up now - https://www.dataops.live/truedataops-podcast and enjoy earlier episodes.  ICYMI – Keith Belanger - #TrueDataOps Podcast Ep #16  In episode 16th of the #TrueDataOps Podcast, we are joined by our guest Keith Belanger! Keith is passionate about data, technology, learning, mentoring, designing, and leading data solutions and teams. With over 25 years in Data Architecture and Information Management, Keith is highly experienced at assembling and directing high-performing, data-focused teams and solutions. Combining a deep technical and data background with a business-oriented mindset and excellent communication skills. He consistently builds and delivers innovative data platforms and solutions to everyday business problems.  Watch the recording...  🚨 Up Next! 🚨 Upcoming Sanjeev Mohan - #TrueDataOps Podcast Ep #17  Wed, May 31, 2023, 8:00 AM PT  Join our next guest, Sanjeev is Principal at SanjMo and a former Gartner Research VP for Big Data and Advanced Analytics. With expertise covering all things data, including analytics, data governance, data management, and data observability, he has published many white papers. He is a prolific speaker at conferences and events worldwide. RSVP here  Did you Know…  …  that you can easily set up Git pre-commit hooks for your DataOps project to check the code being committed for syntax errors, style violations, or other issues. You can write these hooks in any programming language and perform any checks necessary for your project. Learn more…  That’s all for now. Until next time, The DataOps.live Product Team 

Get ready to supercharge your data operations with our latest + greatest updates - packed with powerful new features, enhancements, and expert insights to help you unlock the full potential of your data operations!   More Enhancements in SOLE  We are continuously enhancing our Snowflake Object Lifecycle Engine with more capabilities to help you fully manage your Snowflake ecosystem. You can now: Plan modifications to any Snowflake database before applying them by using the plan and apply for lifecycle jobs. Learn more   Review in a human-readable version, generated as part of the plan lifecycle job, all the modifications planned to be applied to a Snowflake database. Learn more Use the SNOWPARK-OPTIMIZED warehouse that provides 16x memory per node compared to a standard Snowflake virtual warehouse. Learn more  Prevent SOLE from implicitly granting USAGE privileges on the parent and grandparent objects by simply setting a value to a variable. Learn more   MATE with Different dbt Versions  Our Modeling and Transformation Engine (MATE) now supports different dbt versions and provides you with the variable DATAOPS_FEATURE_MATE_DBT_VERSION to decide which version to use. Learn more MATE now also supports colored lineage graphs by annotating your logical layers in your dbt project. Learn more  Did You Know… … that our template rendering engine now supports using Python's datetime and time modules in dbt models and Jinja templates? Learn more  Community   Dan Linstedt | #TrueDataOps Podcast Ep.13 “In the data world, there are lots of advances in cloud technology that should not go unnoticed or ignored… I think the future is cloud – there’s no question about it.”  Live streaming on LinkedIn and available on-demand, the thirteenth episode of the #TrueDataOps Podcast brought together Kent Graziano, aka 'The Data Warrior', with his longtime friend Dan Linstedt, inventor of the data vault system of business intelligence. “We’re on a collision course to re-engage business analytics alongside operational systems, at least in the cloud space. It’s because data is growing at an unprecedented rate, and it’s about to increase again… Data is simply getting too large to move, and this is why I definitely think the cloud is the future...” Hear the rest of what Dan had to say in the link above!  Barr Moses | #TrueDataOps Podcast Ep.14 “The problem of data trust is ubiquitous. The number one thing people told us was ‘the data is wrong and that leads us to a place where we can’t trust it, and we can’t use it…’” Episode 14 of the #TrueDataOps podcast welcomed Barr Moses, Data Observability pioneer, CEO, and co-founder of Monte Carlo, who joined host Kent ‘The Data Warrior’ Graziano to discuss all things data and DataOps. “We can’t afford to ignore the implications of bad data - and this is where data observability comes in. Today, you not only have a lot more data, you also have a lot more people using the data…”   🚨 Up Next! 🚨 Neil Strange | #TrueDataOps Podcast Ep.15 Wednesday, May 3rd | 4pm GMT 11am EDT Join us for our next episode with Neil Strange, CEO and founder of DataVault. Neil is a professional management consultant specializing in all aspects of business intelligence. A director of Business Thinking (trading as Datavault), he is a thought leader in Data Vault 2.0 Data Warehousing, BI, analytics, AI, deep learning, and information governance. Neil is a Guest Presenter, Visitor at the Royal Holloway (University of London) School of Management | RSVP on LinkedIn That’s all for now. Until next time,  The DataOps.live Product Team