Skip to main content

In DataOps CI/CD pipelines, you may need to set environment variables dynamically based on the Git branch being used. This can be achieved by creating a shell script that evaluates the $CI_COMMIT_BRANCH variable and exports the desired variables accordingly. This article will guide you through the process of creating such a script and integrating it into your DataOps CI/CD pipeline.

Prerequisites:

  • Access to a DataOps project with CI/CD enabled.
  • A basic understanding of DataOps CI/CD configuration.

Step 1: Create a Shell Script


First, create a shell script that sets variables based on the value of $CI_COMMIT_BRANCH. Save this script as 00_dynamic_variable.sh inside a directory named scripts in your project's root directory.

#!/bin/bash
echo "Hey"
echo "$CI_COMMIT_BRANCH"

if [[ $CI_COMMIT_BRANCH == 'master' ]]; then
  export VAR_A='master'
elif [ $CI_COMMIT_BRANCH == 'qa' ]]; then
  export VAR_A='qa'
else
  export VAR_A='whatever'
fi

echo $VAR_A


In this script:

  • We start by echoing the branch name to verify that it's correctly captured.
  • We use conditional statements to set VAR_A based on the branch name.
  • You can customize the conditions and variable assignments as needed.


Step 2: Modify Your CI/CD Configuration


Now, let's modify your DataOps CI/CD configuration to integrate the shell script. In your -ci.yml file, add a job that loads the secrets and runs the shell script.

load_secrets:
  script:
    - export DATAOPS_TEMPLATES_DIR=$(dirname $DATAOPS_VAULT_CONTENT)
    - cp scripts/00_dynamic_variable.sh /runner-scripts/00_dynamic_variable.sh
    - chmod +x /runner-scripts/00_dynamic_variable.sh
- /dataops

In this script:

  • cp scripts/00_dynamic_variable.sh /runner-scripts/00_dynamic_variable.sh copies the shell script to a location where it can be executed.
  • chmod +x /runner-scripts/00_dynamic_variable.sh ensures that the script is executable.

With this configuration, the load_secrets job will set VAR_A based on the branch name, as defined in the shell script.

Conclusion:


By creating a shell script that dynamically sets environment variables based on the Git branch name and integrating it into your DataOps CI/CD pipeline, you can streamline your CI/CD process and customize variable values according to different branches. This approach ensures a more flexible and efficient pipeline configuration.

I completely agree with your article. Since dynamic environment variables based on branches or other criteria are indeed a standard practice in CI/CD pipelines. It’s great that you have highlighted topic with clear examples in the guide. This hands-on approach will benefit people who are looking to optimize their CI/CD processes.


Reply