Why are my DataOps.live pipelines running slowly?

  • 17 June 2024
  • 0 replies

Effectively configuring the DataOps.live Runner and pipelines are crucial for maximizing your pipeline's efficiency and reliability. Here are some key considerations:


Runner Concurrency

Configure the concurrent setting in the DataOps.live Runner's configuration file to define the number of jobs that can run simultaneously. Consider the hardware resources of the machine running the runner to avoid overloading the system. If your runner has multiple CPU cores and sufficient RAM, it is beneficial to set a higher concurrency value. DataOps.live jobs can be I/O-intensive, so it's acceptable to over-provision your runner.


Job Timeouts

Each job within a pipeline can have a timeout defined. This setting terminates the job if it exceeds the specified duration, preventing indefinite running jobs that can exhaust resources. Set timeout values based on typical job completion times, with some buffer for unexpected delays.


Resource Groups

Resource groups ensure that the same job from different pipelines does not run simultaneously, preventing resource contention and potential conflicts. While this setting ensures orderly execution, it can also lead to unintended blockages if multiple pipelines are scheduled closely together, causing delays when jobs wait for resource group availability.

It is essential to ensure that a job using a resource group also sets a job timeout. This ensures the job exits and releases the resource group in the event of an unexpectedly long duration. This precaution prevents one temporary issue from affecting subsequent pipelines.


Projects and Runners

Although it is possible for one runner to be used by multiple projects, as projects grow over time, the runner may become overloaded. For larger or more critical projects, consider assigning a dedicated runner to ensure efficiency and reduce the risk of resource contention.

By carefully configuring these settings and considering the specific needs of your jobs and pipelines, you can ensure a smoother, more efficient DataOps.live Pipeline executions.


Video explaining the above:


0 replies

Be the first to reply!