In DataOps CI/CD pipelines, you may need to set environment variables dynamically based on the Git branch being used. This can be achieved by creating a shell script that evaluates the $CI_COMMIT_BRANCH variable and exports the desired variables accordingly. This article will guide you through the process of creating such a script and integrating it into your DataOps CI/CD pipeline.

Prerequisites:

• Access to a DataOps project with CI/CD enabled.
• A basic understanding of DataOps CI/CD configuration.

Step 1: Create a Shell Script

First, create a shell script that sets variables based on the value of $CI_COMMIT_BRANCH. Save this script as 00_dynamic_variable.sh inside a directory named scripts in your project's root directory.

#!/bin/bash
echo "Hey"
echo "$CI_COMMIT_BRANCH"

if [[ $CI_COMMIT_BRANCH == 'master' ]]; then
  export VAR_A='master'
elif  [ $CI_COMMIT_BRANCH == 'qa' ]]; then
  export VAR_A='qa'
else
  export VAR_A='whatever'
fi

echo $VAR_A

In this script:

• We start by echoing the branch name to verify that it's correctly captured.
• We use conditional statements to set VAR_A based on the branch name.
• You can customize the conditions and variable assignments as needed.

Step 2: Modify Your CI/CD Configuration

Now, let's modify your DataOps CI/CD configuration to integrate the shell script. In your -ci.yml file, add a job that loads the secrets and runs the shell script.

load_secrets:
  script:
    - export DATAOPS_TEMPLATES_DIR=$(dirname $DATAOPS_VAULT_CONTENT)
    - cp scripts/00_dynamic_variable.sh /runner-scripts/00_dynamic_variable.sh
    - chmod +x /runner-scripts/00_dynamic_variable.sh
    - /dataops

In this script:

• cp scripts/00_dynamic_variable.sh /runner-scripts/00_dynamic_variable.sh copies the shell script to a location where it can be executed.
• chmod +x /runner-scripts/00_dynamic_variable.sh ensures that the script is executable.

With this configuration, the load_secrets job will set VAR_A based on the branch name, as defined in the shell script.

Conclusion:

By creating a shell script that dynamically sets environment variables based on the Git branch name and integrating it into your DataOps CI/CD pipeline, you can streamline your CI/CD process and customize variable values according to different branches. This approach ensures a more flexible and efficient pipeline configuration.