Effectively configuring the DataOps.live Runner and pipelines are crucial for maximizing your pipeline's efficiency and reliability. Here are some key considerations:
Runner Concurrency
Configure the concurrent setting in the DataOps.live Runner's configuration file to define the number of jobs that can run simultaneously. Consider the hardware resources of the machine running the runner to avoid overloading the system. If your runner has multiple CPU cores and sufficient RAM, it is beneficial to set a higher concurrency value. DataOps.live jobs can be I/O-intensive, so it's acceptable to over-provision your runner.
Job Timeouts
Each job within a pipeline can have a timeout defined. This setting terminates the job if it exceeds the specified duration, preventing indefinite running jobs that can exhaust resources. Set timeout values based on typical job completion times, with some buffer for unexpected delays.
Resource Groups
Resource groups ensure that the same job from different pipelines does not run simultaneously, preventing resource contention and potential conflicts. While this setting ensures orderly execution, it can also lead to unintended blockages if multiple pipelines are scheduled closely together, causing delays when jobs wait for resource group availability.
It is essential to ensure that a job using a resource group also sets a job timeout. This ensures the job exits and releases the resource group in the event of an unexpectedly long duration. This precaution prevents one temporary issue from affecting subsequent pipelines.
Projects and Runners
Although it is possible for one runner to be used by multiple projects, as projects grow over time, the runner may become overloaded. For larger or more critical projects, consider assigning a dedicated runner to ensure efficiency and reduce the risk of resource contention.
By carefully configuring these settings and considering the specific needs of your jobs and pipelines, you can ensure a smoother, more efficient DataOps.live Pipeline executions.
Video explaining the above: