Querying Pipelines Executed with Specific -ci.yml Files in DataOps.Live

DataOps.Live is a platform that allows you to manage and execute pipelines for your projects. If you need to retrieve a list of pipeline IDs that were executed using specific -ci.yml files, you can use the provided Python script along with DataOps.Live API. Here's a step-by-step guide on how to accomplish this:

Prerequisites

Before you proceed, ensure you have the following prerequisites in place:

1. Access Token: Obtain an API access token from DataOps.Live. Replace 'YOUR_ACCESS_TOKEN' in the script with your access token.

2. Project ID: You need to know the project ID where the pipelines are executed. Replace 'YOUR_PROJECT_ID' in the script with your project's ID. 

Python Script:

import requests

api_url = 'https://app.dataops.live/api/v4'
access_token = 'YOUR_ACCESS_TOKEN'

project_id = 'YOUR_PROJECT_ID'

headers = {'Authorization': f'Bearer {access_token}'}

def get_pipeline_ids():
    try:
        pipeline_ids = []
        page = 1
        per_page = 100

        while True:
            url = f'{api_url}/projects/{project_id}/pipelines?per_page={per_page}&page={page}'
            response = requests.get(url, headers=headers)
            if response.status_code == 200:
                pipelines = response.json()
                pipeline_ids.extend([pipeline['id'] for pipeline in pipelines])
                if 'next' in response.links:
                    page += 1
                else:
                    break
            else:
                print('Error:', response.text)
                break
        return pipeline_ids

    except requests.exceptions.RequestException as e:
        print('Request error:', str(e))
        return []


def filter_by_pipeline_name(pipeline_id, desired_pipeline_name):
    url = f'{api_url}/projects/{project_id}/pipelines/{pipeline_id}/variables'
    response = requests.get(url, headers=headers)

    if response.status_code == 200:
        pipeline_variables = response.json()
        for var in pipeline_variables:
            if var.get("key") == "_PIPELINE_FILE_NAME":
                pipeline_name = var.get("value")
                if pipeline_name == desired_pipeline_name:
                    return pipeline_id
    return


pipeline_ids = get_pipeline_ids()
filtered_pipeline_ids = []
for pipeline_id in pipeline_ids:
    result = filter_by_pipeline_name(pipeline_id, "full-ci.yml")
    if result:
        filtered_pipeline_ids.append(result)

print(filtered_pipeline_ids)

The provided Python script allows you to query and filter pipelines executed with specific -ci.yml files. Let's break down the script:

• API URL and Headers: The script defines the API URL and includes the authorisation token in the headers for making API requests.

• get_pipeline_ids: This function retrieves a list of all pipeline IDs associated with your project. It uses pagination to fetch all pipelines, as there may be more than 100.

• filter_by_pipeline_name: This function takes a pipeline ID and a desired pipeline name (in this case, full-ci.yml). It retrieves the pipeline's variables and checks if the pipeline's _PIPELINE_FILE_NAME variable matches the desired name. If it matches, the function returns the pipeline ID; otherwise, it returns None.

• Main Execution: In the main execution part, the script calls get_pipeline_ids to fetch all pipeline IDs. It then iterates through these IDs, calling filter_by_pipeline_name to filter pipelines executed with full-ci.yml. The filtered pipeline IDs are collected in the filtered_pipeline_ids list.

1. Print Filtered Pipeline IDs: Finally, the script prints the list of filtered pipeline IDs.

How to Use the Script

To use the script, follow these steps:

1. Replace 'YOUR_ACCESS_TOKEN' and 'YOUR_PROJECT_ID' in the script with your API access token and project ID.

2. Save the script to a .py file, for example, query_pipelines.py.

3. Run the script using Python: python query_pipelines.py

4. The script will retrieve and print the list of pipeline IDs that were executed with the ci.yml file named "full-ci.yml."

Conclusion

This script enables you to query and filter pipelines executed with specific -ci.yml files in DataOps.Live. By providing your API access token and project ID, you can obtain a list of relevant pipeline IDs, which can be used for further analysis or actions.