Prefill DataOps CI/CD Pipeline-Level Variables for Enhanced Workflow Control

DataOps CI/CD pipelines are powerful tools for automating software development workflows. Within these pipelines, variables play a crucial role in providing flexibility and customisation. In this article, we'll explore how to utilise pipeline-level variables in DataOps CI/CD to enhance manual pipeline runs, focusing on both simple pre-filled variables and more complex selectable options.

Pipeline-Level Variables:

DataOps allows the definition of variables at both the job and pipeline levels. While job-level variables cannot be pre-filled for manual runs, pipeline-level variables offer this functionality. These variables are defined in the full-ci.yml file and can be pre-filled with default values and descriptions.

Simple Pre-Filled Variables: To pre-fill a variable for manual pipeline runs, simply define it within the variables section of your CI/CD configuration file, along with a description and, optionally, a default value. For example:

variables:
  PLATFORM_CREDENTIALS:
    description: "The platform credentials."
  PLATFORM_ENVIRONMENT:
    description: "Select the platform target. Valid options are: 'dev', 'qa', 'prod', or a stable branch of your choice."
    value: "dev"

In this example, PLATFORM_ENVIRONMENT is pre-filled with the default value of "dev", making it convenient for users to initiate pipeline runs without having to input this value every time.

Selectable Pre-Filled Variables

For more complex scenarios where users need to select from a predefined list of options, DataOps provides the ability to define selectable pre-filled variables. This is particularly useful when users need to choose from a set of options, such as deployment environments.

variables:
  PREDEFINED_VAR:
    value: "1"
    options:
      - "1"
      - "0"
    description: "This is a predefined variable for full-ci.yml"

In this example, PREDEFINED_VAR offers two selectable options: "1" and "0".

Conclusion

DataOps CI/CD pipeline-level variables offer powerful customisation options for managing workflows efficiently. By leveraging pre-filled variables, developers can streamline manual pipeline runs, reducing the need for repetitive input and ensuring consistency across deployments. Whether it's a simple pre-filled variable or a selectable list of options, DataOps provides the flexibility needed to tailor CI/CD pipelines to specific project requirements.