In DataOps CI/CD pipelines, you may need to set environment variables dynamically based on the Git branch being used. This can be achieved by creating a shell script that evaluates the
$CI_COMMIT_BRANCH variable and exports the desired variables accordingly. This article will guide you through the process of creating such a script and integrating it into your DataOps CI/CD pipeline.
- Access to a DataOps project with CI/CD enabled.
- A basic understanding of DataOps CI/CD configuration.
Step 1: Create a Shell Script
First, create a shell script that sets variables based on the value of
$CI_COMMIT_BRANCH. Save this script as
00_dynamic_variable.sh inside a directory named scripts in your project's root directory.
if [[ $CI_COMMIT_BRANCH == 'master' ]]; then
elif [[ $CI_COMMIT_BRANCH == 'qa' ]]; then
In this script:
- We start by echoing the branch name to verify that it's correctly captured.
- We use conditional statements to set
VAR_Abased on the branch name.
- You can customize the conditions and variable assignments as needed.
Step 2: Modify Your CI/CD Configuration
Now, let's modify your DataOps CI/CD configuration to integrate the shell script. In your
-ci.yml file, add a job that loads the secrets and runs the shell script.
- export DATAOPS_TEMPLATES_DIR=$(dirname $DATAOPS_VAULT_CONTENT)
- cp scripts/00_dynamic_variable.sh /runner-scripts/00_dynamic_variable.sh
- chmod +x /runner-scripts/00_dynamic_variable.sh
In this script:
cp scripts/00_dynamic_variable.sh /runner-scripts/00_dynamic_variable.shcopies the shell script to a location where it can be executed.
chmod +x /runner-scripts/00_dynamic_variable.shensures that the script is executable.
With this configuration, the load_secrets job will set
VAR_A based on the branch name, as defined in the shell script.
By creating a shell script that dynamically sets environment variables based on the Git branch name and integrating it into your DataOps CI/CD pipeline, you can streamline your CI/CD process and customize variable values according to different branches. This approach ensures a more flexible and efficient pipeline configuration.