Skip to main content
Solved

restricting a runners secret selection

David Richardson

Hi all.

I’ve been trying to restrict the runners secret selection by adding the environment config named SECRETS_SELECTION to:
/srv/dataops-runner-abcd-latest/config/config.toml

 

The configuration setting is:
environment = ["SECRETS_SELECTION=dataops/abcd"]

 

In the DataOps.live project the variables.yaml file is set to ask for secrets:

SECRETS_MANAGER: AWS_PARAMETER_STORE

SECRETS_SELECTION: /dataops/xxxx/SNOWFLAKE/SOLE/

SECRETS_STRIP_PREFIX: /dataops/xxxx/

 

But when I trigger a pipeline that uses the restricted runner the pipeline can still access the secrets XXXX.

What am I doing wrong?

Many thanks.

Dave.

 

Best answer by Traycho Milev

Hi David, 

There are a couple of reasons why the variable might not be correctly recognised. Firstly, ensure that the variable is set within the appropriate section, namely under the runners section ([[runners]]) and not within runners.docker ([runners.docker]).

In most cases, you won't need to restart the runner after updating environment variables in your configuration. The runner typically reads the configuration dynamically and applies changes without requiring a restart.However, if you're experiencing issues with the updated environment variables not being recognised or applied properly, restarting the runner might be a good troubleshooting step. This can help ensure that any cached configurations are cleared and the runner picks up the changes correctly.

If you've confirmed that the environment setting is in the correct section, another potential reason for the variable not being recognised correctly is if it's declared again within your project variables.yml or -ci.yml files . In such cases, the variable would effectively be overwritten, leading to unexpected behaviour.Upon reviewing your most recent commit, we've identified the presence of the `SECRETS_SELECTION` variable in your project.

If the above advice doesn't resolve the issue, please feel free to reach out to us again.

Kind regards,

Traycho Milev

CloudOps & DataOps Engineer

View original

Traycho Milev
Forum|alt.badge.img
  • Traycho Milev
  • Senior DataOps Engineer
  • February 14, 2024

Hi David, 

There are a couple of reasons why the variable might not be correctly recognised. Firstly, ensure that the variable is set within the appropriate section, namely under the runners section ([[runners]]) and not within runners.docker ([runners.docker]).

In most cases, you won't need to restart the runner after updating environment variables in your configuration. The runner typically reads the configuration dynamically and applies changes without requiring a restart.However, if you're experiencing issues with the updated environment variables not being recognised or applied properly, restarting the runner might be a good troubleshooting step. This can help ensure that any cached configurations are cleared and the runner picks up the changes correctly.

If you've confirmed that the environment setting is in the correct section, another potential reason for the variable not being recognised correctly is if it's declared again within your project variables.yml or -ci.yml files . In such cases, the variable would effectively be overwritten, leading to unexpected behaviour.Upon reviewing your most recent commit, we've identified the presence of the `SECRETS_SELECTION` variable in your project.

If the above advice doesn't resolve the issue, please feel free to reach out to us again.

Kind regards,

Traycho Milev

CloudOps & DataOps Engineer

David Richardson

Hi Traycho.

Many thanks for the quick reply.  As you suggested I have removed the SECRETS_SELECTION and SECRETS_STRIP_PREFIX from the project’s variables.yml file.

The correct secrets are now used.

However, is this the desired behaviour?  I had a demo from Dardan who told me that the environment variables in the container would always override the environment variables specified in the project.

 

Dave.
 

Traycho Milev
Forum|alt.badge.img
  • Traycho Milev
  • Senior DataOps Engineer
  • February 15, 2024

Hi Dave,

Thank you for your message. I'm glad to hear that removing the SECRETS_SELECTION from the project's variables.yml file resolved the issue and that the correct secrets are now being used.

In DataOps, the precedence of environment variables is determined based on their source and order of definition. If you defined a variable in config.toml of the Runner, and another variable with the same name is defined in the -ci.yml file, the variable defined in -ci.yml will take precedence over the one defined in config.toml.

This is because the -ci.yml file is specific to the CI/CD pipeline and its configurations will be applied to the pipeline execution. On the other hand, the config.toml file of the Runner is more about configuring the behaviour of the runner itself.

So, in your case, when a variable defined in -ci.yml overwrote the one defined in config.toml, it's expected behaviour according to the precedence order mentioned above.

Here's the full precedence order illustrated horizontally with arrows indicating the direction of precedence, from the highest to the lowest:

config.toml variables --overwritten by--> Variables Defined in -ci.yml --overwritten by--> Job Variables --overwritten by--> CI/CD Variables Defined in the UI.

Kind regards,

Traycho Milev

CloudOps & DataOps Engineer

Reply


Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings