- - Knowledge base overview
Product Updates
Blogs
Academy
Docs

Welcome to the DataOps.live community!

Find answers to your questions
Collaborate with your peers
Share your knowledge

Have a question?

44 Topics
14 Replies
101 Members

Getting started

New member? Everything to get you started.

Product updates

Read the latest news from our product team

The DataOps library

Share tips and best practices, show what you've created and inspire others

Recent activity
Help others

drew.kreigerCommunity Manager

DataOps.live Product Newsletter | October 2023News

Hi DataOps.live User, Welcome to the October newsletter! Explore our latest updates and exciting new features. Announcing DataOps.live | AssistWe’re happy to introduce a powerful capability to the DataOps.live product platform – our AI-powered Copilot, DataOps.live Assist. This solution brings advanced intelligence and automation to elevate your DataOps experience and boost productivity in your day-to-day tasks. Register now for our launch webinar on Nov 2nd where we'll share all the details!And Learn more from our press release HERE MATE with dbt 1.5Get ready for an exciting upgrade! We're rolling out dbt version 1.5 as the default for our Transformation (MATE) orchestrator and MATE packages, taking your DataOps experience to a new level. Using dbt 1.5 in your DataOps projects unlocks a range of exciting updates and enhanced performance. Discover more about the MATE Orchestrator and MATE Packages Python 3.11 OrchestratorWe’ve introduced the Python 3.11 Orchestrator, currently avai

drew.kreigerCommunity Manager

The DataOps Library

Data Products for Dummies

Data Products for Dummies eBook Written by the pioneers of Data Products and the DataOps Methodology, Data Products for Dummies provides a real-world guide for all organizations to use Data Products as the base unit of data management. Data is a victim of its own success - It's such a critical ingredient for any organization that data teams face ever-tighter deadlines to deliver the goods. The traditional way of creating data applications isn't scaleable and data teams are becoming a bottleneck. The older approach leads to data outcomes that lack accountability and data pipelines that aren't reliable. Data Products are the future for how organizations will organize, develop, and deploy data assets and data applications. But current information about data products can be complex and confusing. Data Products for Dummies is your one-stop guide to moving beyond the theory to embark on a pragmatic and practical journey towards building data products for your organization. Using relatable

traychomilevCommunity Manager

Merging Configuration: The Power of Job Definitions in CI/CD

Continuous Integration and Continuous Deployment (CI/CD) pipelines have become indispensable tools for software development, helping teams automate the process of building, testing, and deploying their data configuration. DataOps, as a platform for data definition code management, provides a powerful feature for merging configuration into your own projects from reference projects. This allows you to define a job in your project with the same name as a job in the reference project, effectively customising and expanding upon its functionality.In this article, we'll explore how you can leverage this feature to merge job configurations from reference projects into your own DataOps CI/CD pipeline, using real-world examples to illustrate its capabilities.The Concept of Job MergingDataOps’s job merging feature enables you to reuse and customise job definitions from a reference project within your own project. By defining a job in your project with the same name as a job in the reference proje

traychomilevCommunity Manager

Querying Pipelines Executed with Specific -ci.yml Files in DataOps.Live

DataOps.Live is a platform that allows you to manage and execute pipelines for your projects. If you need to retrieve a list of pipeline IDs that were executed using specific -ci.yml files, you can use the provided Python script along with DataOps.Live API. Here's a step-by-step guide on how to accomplish this:PrerequisitesBefore you proceed, ensure you have the following prerequisites in place: Access Token: Obtain an API access token from DataOps.Live. Replace 'YOUR_ACCESS_TOKEN' in the script with your access token. Project ID: You need to know the project ID where the pipelines are executed. Replace 'YOUR_PROJECT_ID' in the script with your project's ID. Python Script:import requestsapi_url = 'https://app.dataops.live/api/v4'access_token = 'YOUR_ACCESS_TOKEN'project_id = 'YOUR_PROJECT_ID'headers = {'Authorization': f'Bearer {access_token}'}def get_pipeline_ids(): try: pipeline_ids = [] page = 1 per_page = 100 while True: url = f'{api_url

traychomilevCommunity Manager

Automate Snowflake Query Cancellation for failed pipelines with MATE Macros

In the world of DataOps, ensuring efficient and reliable data processing is crucial. Snowflake, a popular cloud data warehousing platform, offers a variety of features to help manage your data operations effectively. In this article, we'll explore a MATE macro that can be used to cancel any queries left running by a DataOps pipeline job in Snowflake. This macro leverages Snowflake's query tag functionality to identify and terminate queries associated with a specific pipeline.Understanding the Problem:Data pipelines often involve multiple queries and processes that run in sequence. Occasionally, queries can get stuck in a "RUNNING" state, consuming resources and potentially causing issues for subsequent jobs. To address this, we need a way to automatically cancel queries related to a particular pipeline that has failed.The Solution: MATE Macro for Query CancellationThe MATE macro presented here offers a clean and efficient solution for canceling running queries linked to the current pip

drew.kreigerCommunity Manager

DataOps.live Product Newsletter | September 2023News

Hi DataOps.live User,  Welcome to the September newsletter! Find out how our latest developments enhance your DataOps experience. Streamline Snowflake setup and save time managing environments. Test new features with reduced risk. And more. Read on. Snowflake Environment ManagementIn DataOps.live, creating pipelines involves combining various jobs, like those in the DataOps Reference Project. Each job has a specific role, and our powerful Snowflake Object Lifecycle Engine (SOLE) ensures everything works smoothly. We’ve added more optional jobs to the Reference Project for managing Snowflake development environments, making it much easier to set up and remove such environments. We’ve also introduced execution rules to handle Snowflake in different setups. And no worries, all your existing jobs still fit right in. Want to know more? Check out the details of these new Reference Project Jobs. Setup Wizard for Development Environment Start an even smoother DataOps experience! We’ve laun

drew.kreigerCommunity Manager

DataOps.live Product Newsletter | August 2023News

Hi DataOps.live User,  Welcome to the August newsletter! Find out how our latest developments enhance your DataOps experience. Save time investigating Snowflake spend. Make the most of DDE. Enhance reliability and fault tolerance. And more. Read on. Enhancements in Spendview for SnowflakeCompute is often the top cost area for optimization related to a Snowflake Warehouse, where the most common problems are query performance and idle warehouses. Spendview for Snowflake brings awareness to business stakeholders and data team leads to understand their current resource usage. Spendview provides big saving opportunities in a matter of clicks as it empowers your data team with self-service Snowflake consumption and spending metrics. Sign up now for our live webinar on optimizing your Snowflake spend and join DataOps.live and XponentL Data discussion on how you can perfect your budget by following best practices and cost-saving strategies. Optimize the Potential of DataOps Developmen

traychomilevCommunity Manager

Crisp Understanding: Troubleshooting dbt Profiles for SQLFluff Pre-commit Hooks

Pre-commit hooks are a powerful mechanism for enhancing your development workflow by automating various checks and processes before code is committed to a repository. In the context of SQLFluff, a popular SQL linter and formatter, pre-commit hooks can be used to ensure your SQL code adheres to best practices and maintains a consistent style. In this article, we will address a common issue encountered when using SQLFluff pre-commit hooks, specifically related to dbt profiles.The Error: "User Error: DbtProjectError: Runtime Error Could not find profile named 'XXX'"When working with SQLFluff and dbt, you may encounter an error message like the one below:User Error: DbtProjectError: Runtime Error Could not find profile named 'XXX' This error occurs because SQLFluff is unable to locate a dbt profile named 'XXX' in the profiles.yml file. The profiles.yml file is essential for connecting to your database and providing necessary configuration details to dbt. In the context of pre-commit hooks,

traychomilevCommunity Manager

Setting Variables Dynamically Based on GitLab CI/CD Branch in Shell Script

In DataOps CI/CD pipelines, you may need to set environment variables dynamically based on the Git branch being used. This can be achieved by creating a shell script that evaluates the $CI_COMMIT_BRANCH variable and exports the desired variables accordingly. This article will guide you through the process of creating such a script and integrating it into your DataOps CI/CD pipeline.Prerequisites:Access to a DataOps project with CI/CD enabled. A basic understanding of DataOps CI/CD configuration.Step 1: Create a Shell ScriptFirst, create a shell script that sets variables based on the value of $CI_COMMIT_BRANCH. Save this script as 00_dynamic_variable.sh inside a directory named scripts in your project's root directory.#!/bin/bashecho "Hey"echo "$CI_COMMIT_BRANCH"if [[ $CI_COMMIT_BRANCH == 'master' ]]; then export VAR_A='master'elif [[ $CI_COMMIT_BRANCH == 'qa' ]]; then export VAR_A='qa'else export VAR_A='whatever'fiecho $VAR_AIn this script:We start by echoing the branch name to ver

traychomilevCommunity Manager

Enhancing DataOps Pipeline Resilience with Multiple Pull Policies

In the fast-paced world of DataOps and continuous integration, Docker has become an indispensable tool for containerisation. However, like any technology, it occasionally encounters hiccups. One such hiccup is when Docker experiences issues with its Hub Registry and Docker Authentication, like it did yesterday: overview of incident on 28.09.2023. Fortunately, there's a way to mitigate these risks and enhance the resilience of your DataOps pipelines: by using multiple pull policies.Understanding the ChallengeBefore delving into the solution, it's important to understand the challenge at hand. When Docker's Hub Registry or authentication systems encounter problems, it can disrupt the smooth flow of your pipelines. This disruption can lead to delays and operational headaches, especially when your CI/CD processes rely heavily on Docker images.The Power of Multiple Pull PoliciesTo address this challenge, DataOps offers a powerful feature: the ability to define multiple pull policies. These

Leaderboard

Show full leaderboard

Show full leaderboard

Awarded badges widget

NickHalseyhas earned the badge Early Adopter
allanthuesenhas earned the badge Early Adopter
Patrick_at_snowflakehas earned the badge Early Adopter
Rohanhas earned the badge Early Adopter
maeklund86has earned the badge Early Adopter

Show all badges

DataOps.live employee leaderboard

Download our eBook

DataOps.live describes a novel way of development teams working together collaboratively around data to achieve rapid results and improve customer satisfaction. This book is intended for everyone looking to adopt the DataOps philosophy to improve the governance and agility of their data products. The principles in this book should create a shared understanding of the goals and methods of DataOps and #TrueDataOps and create a starting point for collaboration.

Download E-Book

#TrueDataOps community

The Philosophy of DataOps (#TrueDataOps)

The DataOps Manifesto has become a valuable resource to define the WHAT: what we are aiming to achieve with DataOps, and what DataOps will deliver. Our Philosophy of DataOps has been created to define the HOW: the way to deliver DataOps and the core pillars that should underpin any DataOps deployment.

Need more help?

Docs

Popular resources and much more

DataOps academy

Take your Intro to DataOps today!

Support

Do you require additional assistance? Our team is prepared to provide all info

Powered by Gainsight

Terms & Conditions Cookie settings

Sign up

Already have an account? Login

Login with DataOps.live

or

Username *

E-mail address *

First name *

Last name *

Job Title *

Executive Product Owner Engineer Developer Other

Company *

Country *

Zip Code *

I agree to receive marketing communications *

Yes - I consent to marketing (opt out anytime by clicking unsubscribe in email) No - I do not consent to marketing

Password *

I accept the terms & conditions

loginBox.register.email_repeat

Login to the community

No account yet? Create an account

Login with DataOps.live

or

Username or Email

Password

Remember me

Forgot password?

Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.

Username or e-mail

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

OK

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.

OK