Querying Pipelines Executed with Specific -ci.yml Files in DataOps.Live

  • 20 October 2023
  • 0 replies

Userlevel 4

DataOps.Live is a platform that allows you to manage and execute pipelines for your projects. If you need to retrieve a list of pipeline IDs that were executed using specific -ci.yml files, you can use the provided Python script along with DataOps.Live API. Here's a step-by-step guide on how to accomplish this:


Before you proceed, ensure you have the following prerequisites in place:

  1. Access Token: Obtain an API access token from DataOps.Live. Replace 'YOUR_ACCESS_TOKEN' in the script with your access token.

  2. Project ID: You need to know the project ID where the pipelines are executed. Replace 'YOUR_PROJECT_ID' in the script with your project's ID. 

Python Script:

import requests

api_url = 'https://app.dataops.live/api/v4'
access_token = 'YOUR_ACCESS_TOKEN'

project_id = 'YOUR_PROJECT_ID'

headers = {'Authorization': f'Bearer {access_token}'}

def get_pipeline_ids():
pipeline_ids = []
page = 1
per_page = 100

while True:
url = f'{api_url}/projects/{project_id}/pipelines?per_page={per_page}&page={page}'
response = requests.get(url, headers=headers)
if response.status_code == 200:
pipelines = response.json()
pipeline_ids.extend([pipeline['id'] for pipeline in pipelines])
if 'next' in response.links:
page += 1
print('Error:', response.text)
return pipeline_ids

except requests.exceptions.RequestException as e:
print('Request error:', str(e))
return []

def filter_by_pipeline_name(pipeline_id, desired_pipeline_name):
url = f'{api_url}/projects/{project_id}/pipelines/{pipeline_id}/variables'
response = requests.get(url, headers=headers)

if response.status_code == 200:
pipeline_variables = response.json()
for var in pipeline_variables:
if var.get("key") == "_PIPELINE_FILE_NAME":
pipeline_name = var.get("value")
if pipeline_name == desired_pipeline_name:
return pipeline_id

pipeline_ids = get_pipeline_ids()
filtered_pipeline_ids = []
for pipeline_id in pipeline_ids:
result = filter_by_pipeline_name(pipeline_id, "full-ci.yml")
if result:


The provided Python script allows you to query and filter pipelines executed with specific -ci.yml files. Let's break down the script:

  • API URL and Headers: The script defines the API URL and includes the authorisation token in the headers for making API requests.

  • get_pipeline_ids: This function retrieves a list of all pipeline IDs associated with your project. It uses pagination to fetch all pipelines, as there may be more than 100.

  • filter_by_pipeline_name: This function takes a pipeline ID and a desired pipeline name (in this case, full-ci.yml). It retrieves the pipeline's variables and checks if the pipeline's _PIPELINE_FILE_NAME variable matches the desired name. If it matches, the function returns the pipeline ID; otherwise, it returns None.

  • Main Execution: In the main execution part, the script calls get_pipeline_ids to fetch all pipeline IDs. It then iterates through these IDs, calling filter_by_pipeline_name to filter pipelines executed with full-ci.yml. The filtered pipeline IDs are collected in the filtered_pipeline_ids list.

  1. Print Filtered Pipeline IDs: Finally, the script prints the list of filtered pipeline IDs.

How to Use the Script

To use the script, follow these steps:

  1. Replace 'YOUR_ACCESS_TOKEN' and 'YOUR_PROJECT_ID' in the script with your API access token and project ID.

  2. Save the script to a .py file, for example, query_pipelines.py.

  3. Run the script using Python: python query_pipelines.py

  4. The script will retrieve and print the list of pipeline IDs that were executed with the ci.yml file named "full-ci.yml."


This script enables you to query and filter pipelines executed with specific -ci.yml files in DataOps.Live. By providing your API access token and project ID, you can obtain a list of relevant pipeline IDs, which can be used for further analysis or actions.

0 replies

Be the first to reply!