In DataOps CI/CD pipelines, you may need to set environment variables dynamically based on the Git branch being used. This can be achieved by creating a shell script that evaluates the $CI_COMMIT_BRANCH
variable and exports the desired variables accordingly. This article will guide you through the process of creating such a script and integrating it into your DataOps CI/CD pipeline.
Prerequisites:
- Access to a DataOps project with CI/CD enabled.
- A basic understanding of DataOps CI/CD configuration.
Step 1: Create a Shell Script
First, create a shell script that sets variables based on the value of $CI_COMMIT_BRANCH.
Save this script as 00_dynamic_variable.sh
inside a directory named scripts in your project's root directory.
#!/bin/bash
echo "Hey"
echo "$CI_COMMIT_BRANCH"
if [[ $CI_COMMIT_BRANCH == 'master' ]]; then
export VAR_A='master'
elif [ $CI_COMMIT_BRANCH == 'qa' ]]; then
export VAR_A='qa'
else
export VAR_A='whatever'
fi
echo $VAR_A
In this script:
- We start by echoing the branch name to verify that it's correctly captured.
- We use conditional statements to set
VAR_A
based on the branch name. - You can customize the conditions and variable assignments as needed.
Step 2: Modify Your CI/CD Configuration
Now, let's modify your DataOps CI/CD configuration to integrate the shell script. In your -ci.yml
file, add a job that loads the secrets and runs the shell script.
load_secrets:
script:
- export DATAOPS_TEMPLATES_DIR=$(dirname $DATAOPS_VAULT_CONTENT)
- cp scripts/00_dynamic_variable.sh /runner-scripts/00_dynamic_variable.sh
- chmod +x /runner-scripts/00_dynamic_variable.sh
- /dataops
In this script:
cp scripts/00_dynamic_variable.sh /runner-scripts/00_dynamic_variable.sh
copies the shell script to a location where it can be executed.chmod +x /runner-scripts/00_dynamic_variable.sh
ensures that the script is executable.
With this configuration, the load_secrets job will set VAR_A
based on the branch name, as defined in the shell script.
Conclusion:
By creating a shell script that dynamically sets environment variables based on the Git branch name and integrating it into your DataOps CI/CD pipeline, you can streamline your CI/CD process and customize variable values according to different branches. This approach ensures a more flexible and efficient pipeline configuration.