pipeline not running

Hi @Sushil ,

Thank you for reaching out. We have noticed that the runner you have selected in your agent_tag.yml is not active:

To create your own runner you can follow the guide below: https://docs.dataops.live/docs/administration/docker-runner/installation/

As for the shared runner - it has certain limitations that may restrict the usage of some orchestrator images provided. In light of this, we recommend creating your own runner to avoid any potential limitations associated with the shared runner. By setting up a custom runner, you gain more control over the environment and can tailor it to meet the specific requirements of your project.

If you need assistance in creating a custom runner or have any questions about the process, feel free to reach out, and we'll be happy to guide you through the setup.

Kind regards,
Traycho Milev
CloudOps & DataOps Engineer

Dear @Sushil ,

As you have deducted the inclusion of the variable 'SECRETS_SELECTION: $SECRET_ARN' successfully resolved the issue. To provide context, our partner connect runner faces limitations in permissions, specifically being restricted to retrieving secrets exclusively from a designated location. Notably, it lacked the 'ListSecrets' permission, leading to the initial failure.

Kind regards,
Traycho Milev
CloudOps & DataOps Engineer

Dear @Traycho 

Thanks a lot for your suggestion. It helped so now I think I will go ahead with next sessions in training Fundamentals of DataOps.live and then modify the pipeline accordingly.

Thanks again for all your help so far -:)

Hi @Sushil ,

 Thank you for bringing this to our attention. We've observed that you're encountering the following error:

[ 05:12:15.782 ] - [ 20-render-templates ] - [ INFO ] - Rendering templates in /tmp/local_config using dataops-render render-dir
  /tmp/local_config/profiles.yml [not rendered: 'dict object' has no attribute 'TRANSFORM']
Exiting due to strict mode

This error signals a problem during the rendering process of a templated YAML file. Specifically, the file in question pertains to the default profile we supply for our MATE jobs. This file expects several keys to be present in your vault:

• SNOWFLAKE.TRANSFORM.USERNAME
• SNOWFLAKE.TRANSFORM.PASSWORD
• SNOWFLAKE.TRANSFORM.ROLE
• SNOWFLAKE.TRANSFORM.WAREHOUSE
• SNOWFLAKE.TRANSFORM.THREADS

The vault.template.yml file automatically populates the specified values in the vault, mapping secrets from the secret manager to the designated keys mentioned above.

To understand the cause of the failure, it is crucial to review the log message from the preceding job, "Initialize Pipeline." While this job may have been marked as successful, it contains a critical log message that provides insights into the root cause of the issue. Please refer to the log details to gain a comprehensive understanding of the situation.

https://app.dataops.live/BV31501.aws-ap-southeast-1/academy-fundamentals/-/jobs/22888988#L132

dataops-vault WARNING Unable to parse secrets file /builds/BV31501.aws-ap-southeast-1/academy-fundamentals/vault-content/vault.yml as YAML: while parsing a block mapping
  in "/builds/BV31501.aws-ap-southeast-1/academy-fundamentals/vault-content/vault.yml", line 9, column 3
expected <block end>, but found '<block mapping start>'
  in "/builds/BV31501.aws-ap-southeast-1/academy-fundamentals/vault-content/vault.yml", line 28, column 4

It has come to our attention that the vault.template.yml file has been modified, resulting in an invalid YAML structure and rendering issues. We kindly request that you restore the file to its original contents. It is important to note that any alterations to the file are not expected, as the system is designed to function seamlessly without the need for manual adjustments. Reverting to the original configuration should resolve any issues, allowing the system to operate as intended out of the box.

Kind regards,
Traycho Milev
CloudOps & DataOps Engineer

Dear @traychomilev 

could you please help me to resolve this error. I tried to debug few times but no luck. It was working fine earlier.

Hi @Sushil ,

The reason behind your error is in the syntax of the column_width test you have introduced here:

https://app.dataops.live/BV31501.aws-ap-southeast-1/academy-fundamentals/-/blob/e1f03676c48c4f6165f8b0338ba21763375023fd/dataops/modelling/macros/column_width.sql

Ensure that every test introduced to the project concludes with the phrase {% endtest %}

Otherwise during the compile, DBT will fail with:

Compilation Error
  Reached EOF without finding a close tag for test (searched from line 1)

Example syntax from the dbtutils package: https://github.com/dbt-labs/dbt-utils/blob/main/macros/generic_tests/recency.sql#L3C14-L3C14

Similarly, macros that are added to a project have to end with {% endmacro %}​​​​​​.

I trust this proves beneficial.

Kind regards,
Traycho Milev
CloudOps & DataOps Engineer

Hi @Sushil ,

Thank you for bringing this matter to our attention. It has come to our notice that the pipeline is currently running on dbt version 1.0.1, which is considered outdated. This observation suggests that the version of the orchestrator's image being utilised is not up-to-date. We suspect that the root cause of this issue lies in the configuration of the runner, specifically in its imagePullPolicy.

We acknowledge this concern and assure you that we will be actively working on updating the configuration of our runner to address this issue. In the interim, we recommend that you create your own runner. This will allow you to take advantage of the latest image versions, avoiding restrictions related to secrets placement, image pulling capabilities, and other potential limitations.

Kind regards,
Traycho Milev
CloudOps & DataOps Engineer

Hi @Sushil ,

Thank you for your observation! We can confirm that the current default version of DBT that we use is dbt 1.5 Therefore, we shall be proceeding with updating the configuration of our partnerconnect-k8s-runner and we will notify you once done.

Kind regards,
Traycho Milev
CloudOps & DataOps Engineer

Hello @Sushil ,

I hope this message finds you well. I wanted to inform you that the Partner Connect Runner has been successfully updated, and it is now operating on the latest 5-stable versions of the orchestrators as per your request. Please proceed at your convenience to verify your pipeline. 

If you encounter any issues or require further assistance, please don't hesitate to reach out. Thank you for your collaboration and swift response throughout this process.

Kind regards,
Traycho Milev
CloudOps & DataOps Engineer