- - Knowledge base overview
Product Updates
Blogs
Academy
Docs

Welcome to the DataOps.live community!

Find answers to your questions
Collaborate with your peers
Share your knowledge

Have a question?

40 Topics
10 Replies
94 Members

Getting started

New member? Everything to get you started.

Product updates

Read the latest news from our product team

The DataOps library

Share tips and best practices, show what you've created and inspire others

Recent activity
Help others

traychomilevCommunity Manager

Setting Variables Dynamically Based on GitLab CI/CD Branch in Shell Script

In DataOps CI/CD pipelines, you may need to set environment variables dynamically based on the Git branch being used. This can be achieved by creating a shell script that evaluates the $CI_COMMIT_BRANCH variable and exports the desired variables accordingly. This article will guide you through the process of creating such a script and integrating it into your DataOps CI/CD pipeline.Prerequisites:Access to a DataOps project with CI/CD enabled. A basic understanding of DataOps CI/CD configuration.Step 1: Create a Shell ScriptFirst, create a shell script that sets variables based on the value of $CI_COMMIT_BRANCH. Save this script as 00_dynamic_variable.sh inside a directory named scripts in your project's root directory.#!/bin/bashecho "Hey"echo "$CI_COMMIT_BRANCH"if [[ $CI_COMMIT_BRANCH == 'master' ]]; then export VAR_A='master'elif [[ $CI_COMMIT_BRANCH == 'qa' ]]; then export VAR_A='qa'else export VAR_A='whatever'fiecho $VAR_AIn this script:We start by echoing the branch name to ver

traychomilevCommunity Manager

Crisp Understanding: Troubleshooting dbt Profiles for SQLFluff Pre-commit Hooks

Pre-commit hooks are a powerful mechanism for enhancing your development workflow by automating various checks and processes before code is committed to a repository. In the context of SQLFluff, a popular SQL linter and formatter, pre-commit hooks can be used to ensure your SQL code adheres to best practices and maintains a consistent style. In this article, we will address a common issue encountered when using SQLFluff pre-commit hooks, specifically related to dbt profiles.The Error: "User Error: DbtProjectError: Runtime Error Could not find profile named 'XXX'"When working with SQLFluff and dbt, you may encounter an error message like the one below:User Error: DbtProjectError: Runtime Error Could not find profile named 'XXX' This error occurs because SQLFluff is unable to locate a dbt profile named 'XXX' in the profiles.yml file. The profiles.yml file is essential for connecting to your database and providing necessary configuration details to dbt. In the context of pre-commit hooks,

drew.kreigerCommunity Manager

DataOps.live Product Newsletter | August 2023News

Hi DataOps.live User,  Welcome to the August newsletter! Find out how our latest developments enhance your DataOps experience. Save time investigating Snowflake spend. Make the most of DDE. Enhance reliability and fault tolerance. And more. Read on. Enhancements in Spendview for SnowflakeCompute is often the top cost area for optimization related to a Snowflake Warehouse, where the most common problems are query performance and idle warehouses. Spendview for Snowflake brings awareness to business stakeholders and data team leads to understand their current resource usage. Spendview provides big saving opportunities in a matter of clicks as it empowers your data team with self-service Snowflake consumption and spending metrics. Sign up now for our live webinar on optimizing your Snowflake spend and join DataOps.live and XponentL Data discussion on how you can perfect your budget by following best practices and cost-saving strategies. Optimize the Potential of DataOps Developmen

MGetovCommunity Manager

Find bottlenecks in current warehouses setup

For easy identification of bottlenecks in your warehouses setup, we have incorporated two highly valuable graphs in Spendview for Snowflake. To find them: 1) On the menu bar, click Usage. 2) Focus on the last two graphs: Warehouse spilling Warehouse queueing Both have similar structures, featuring two bars per warehouse, yet let us explore their details more closely. Warehouse spilling. First what exactly is spilling? If you are not familiar with the Snowflake warehouse spilling, you can read more about it in this article. Long story short, the spilling is impacting query performance since the engine needs to temporarily store some of the query data on a local or remote storage – the warehouse memory is not enough to execute the query. Ideally, you will have no spilling or at most just a few GBs. However, if you discover that a warehouse has a lot of spilled GBs, it is recommended to take necessary actions. With this graph, identifying this issue is straightforward. Each

MGetovCommunity Manager

Reduce the number of used warehouses

Another skillful use of the Warehouse utilization graph is that you can easily identify redundant warehouses. If you have already applied the optimizations that we recommended in “How to gain control over your Snowflake spending in minutes?” And you are still seeing a graph like this one: This means you have a lot of warehouses that you are not using 100%.The recommended action here is to reduce them by combining 2-3 of them into one. By doing so, you switch the warehouse used by different applications or users to the same one. This approach will significantly enhance your cost efficiency. Naturally, the question that arises here is how can we be sure that we are not over optimizing, and that we won’t negatively impact performance? We have considered this scenario as well. In our post - “Find bottlenecks in current warehouses setup” , we are focusing on the graphs that can help you identify warehouses experiencing performance issues.

MGetovCommunity Manager

How to gain control over your Snowflake spending in minutes?

We believe that you can take an immediate effect from Spendview. In less than 5 minutes you can identify a room for optimization that can save you thousands of dollars. How? Just follow the steps If you are not a Spendview user, please register from here Log in to you Spendview account from here If you have not added your Snowflake accounts, please follow the steps Select last 30 days period in the period selection bar Go to Compute tab Look in the Warehouse utilization widget (you can click show details for opening it in a separate sub tab) On the widget you will see two bars per warehouse, the left one shows how much credits / dollars you are spending per warehouse. And the right one shows the aggregated cost of all queries that have been executed on this warehouse (only the real execution time of the queries) If the left bar is significantly bigger than the right one and you have a big red bar, that means that this warehouse is not well utilized. You can take two

drew.kreigerCommunity Manager

DataOps.live Product Newsletter | July 2023News

Hi DataOps.live User,  Welcome to the July newsletter! Gain insights on how the important developments announced at the Snowflake Summit 2023 may greatly benefit DataOps.live projects and operations. Enjoy the collaboration with your DataOps peers on our Community platform. Read on. Snowflake Summit Announcements and DataOps Our powerful Snowflake Object Lifecycle Engine (SOLE) and Modelling and Transformation Engine (MATE) help data engineers automate and streamline the steps in the development lifecycle, allowing them to focus on Snowflake’s functional capabilities. With the exciting announcements of Snowflake Summit 2023, we are assessing how the latest technical developments by Snowflake further enhance DataOps.live. Potential advancements include: Enhance SOLE to support Snowpark container services and libraries management Further simplify building user interfaces using Streamlit and enhance the management of Snowpark libraries Update SOLE to ease building, testing, distribu

ilja.taransNew Participant

Have a question?

"Community Help" search in Spendview relevance results

Hello,How does the new feature "Community Help" search in Spendview, sets the relevance of search results?And is it hitting result from Spendview documentation portal?Thanks,Ilja

traychomilevCommunity Manager

DBT Power User Extension's Build and Run Buttons not Working in DDE Cloud: A Guide for Resolution

For data analysts and engineers, DBT as part of MATE has become a popular choice for transforming and modeling data. Its integration with Visual Studio Code (VSCode) through the DBT Power User Extension allows users to streamline their workflow and enhance productivity. However, some users have encountered a peculiar issue when SSH-ing to a DDE (DataOps Data Engineering) Cloud instance. In this article, we will explore the problem, its root cause, and the steps required to resolve it effectively.The Issue at HandUsers who access DDE Cloud instances through SSH and work with VSCode have reported that the Build and Run Buttons in the DBT Power User Extension fail to function as expected. Attempting to build or run a model through the extension in this scenario yields no results, leaving users frustrated and perplexed.Root CauseThe reason behind this unexpected behavior lies in the configuration of the DBT Power User Extension and how it interacts with the DDE Cloud instance during an SSH

ilja.taransNew Participant

Have a question?

Spendview logging and monitoring features

Are there any features in Spendview to alert administrators of suspicious activities?Thanks,Ilja

Leaderboard

Show full leaderboard

Show full leaderboard

Awarded badges widget

NickHalseyhas earned the badge Early Adopter
allanthuesenhas earned the badge Early Adopter
Patrick_at_snowflakehas earned the badge Early Adopter
Rohanhas earned the badge Early Adopter
maeklund86has earned the badge Early Adopter

Show all badges

DataOps.live employee leaderboard

Download our eBook

DataOps.live describes a novel way of development teams working together collaboratively around data to achieve rapid results and improve customer satisfaction. This book is intended for everyone looking to adopt the DataOps philosophy to improve the governance and agility of their data products. The principles in this book should create a shared understanding of the goals and methods of DataOps and #TrueDataOps and create a starting point for collaboration.

Download E-Book

#TrueDataOps community

The Philosophy of DataOps (#TrueDataOps)

The DataOps Manifesto has become a valuable resource to define the WHAT: what we are aiming to achieve with DataOps, and what DataOps will deliver. Our Philosophy of DataOps has been created to define the HOW: the way to deliver DataOps and the core pillars that should underpin any DataOps deployment.

Need more help?

Docs

Popular resources and much more

DataOps academy

Take your Intro to DataOps today!

Support

Do you require additional assistance? Our team is prepared to provide all info

Powered by Gainsight

Terms & Conditions Cookie settings

Sign up

Already have an account? Login

Username *

E-mail address *

First name *

Last name *

Job Title *

Executive Product Owner Engineer Developer Other

Company *

Country *

Zip Code *

I agree to receive marketing communications *

Yes - I consent to marketing (opt out anytime by clicking unsubscribe in email) No - I do not consent to marketing

Password *

I accept the terms & conditions

loginBox.register.email_repeat

Login to the community

No account yet? Create an account

Username or Email

Password

Remember me

Forgot password?

Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.

Username or e-mail

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

OK

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.

OK