We are excited to announce the release of our 5-latest orchestrators in October, powered by Ubuntu 24.04 LTS, and an extended testing period before the official 5-stable release in November. This update brings significant improvements, including enhanced security, support, and updated packages, to ensure your data pipelines remain efficient and future-proof.

To provide customers with enough time to prepare and test their pipelines, we are delaying the release of these orchestrators to the 5-stable version by one month. During this time, you can test your pipelines against the 5-latest tag, giving you the opportunity to ensure compatibility with the new features before the stable release in November.

Why Are We Upgrading to Ubuntu 24.04 LTS?

The primary driver for this upgrade is the extended long-term support offered by Ubuntu 24.04 LTS, which provides security and maintenance updates until April 2029. By upgrading, we ensure that orchestrators remain secure, efficient, and compatible with the latest technologies. Additionally, this upgrade includes the following key improvements:

• Python upgrade from version 3.8 to 3.12, unlocking new features and compatibility with modern libraries.
• Updated operating system packages for improved performance.
• Enhanced security features and system stability.

For more details, you can visit our announcement page.

How Can Customers Test?

We encourage all customers to take advantage of this extended testing period by switching to the 5-latest tag once it’s available. Testing your pipelines before the 5-stable release ensures a smooth transition and provides time to address any potential issues.

For more information on how to switch orchestrators, refer to our Best Practices for Switching Orchestrators.

Important note: Pipelines that do not use custom code (such as custom Python scripts) are expected to work without changes. However, if your pipeline relies on custom code or specific system packages, we recommend thorough testing to ensure compatibility.

Key Changes to Be Aware Of:

1. Python Upgrade (3.8 to 3.12)

   • If your pipeline runs custom Python code (either through the Python orchestrator or script tags), it’s crucial to test for compatibility with Python 3.12.

2. Updated Python Modules

   • The versions of installed Python modules will also change. Pipelines relying on specific versions of modules should be tested and potentially updated to ensure smooth operation.

3. Operating System Package Changes

   • Pipelines that install OS-level packages using tools like apt or pip may encounter changes in the available packages. Make sure to verify that the required packages are available and compatible with the new version.

4. Key Library Changes (e.g., libc)

   • Pipelines using precompiled binaries not provided by DataOps may be affected by changes in core libraries like libc. Ensure your binaries are compatible with the upgraded libraries in Ubuntu 24.04 LTS.

Recommended Process for testing:

We recommend that you test your pipelines against the new version of the orchestrators prior to their release on 5-stable.  A suggested way of doing this is

1. Create a development branch from your normal development branch.
2. In the branch, pin the orchestrator version to 5-latest, following the guidance here.
3. Run a pipeline for each of your -ci files in the branch, and confirm that the pipeline runs correctly.
4. Once testing is complete, we recommend removing the branch to ensure that the pin to 5-latest does not get accidentally promoted to production.

If the pipeline shows issues with the upgrade, then we recommend the process below to address these

Recommended Process for Implementing Changes (if required):

To avoid disruption during the transition, we recommend the following process for implementing and testing necessary changes in your pipeline:

1. Pin Your Current Orchestrators:
   Before the release of 5-stable with Ubuntu 24.04 LTS, pin your orchestrators to the existing 5-stable version (e.g., 5.33.2) using the guidance found here.

2. Branch and Test:
   Create a branch for your changes and test them against the 5-latest tag once it’s available. This allows you to ensure compatibility with the upgraded orchestrators.

3. Update to 5-Stable:
   After the release of 5-stable in November, update your branch to use 5-stable and test again.

4. Promote Your Changes:
   Once testing is successful, promote your branch to ensure a smooth transition to the new orchestrators.

This process ensures that your pipelines remain stable on the existing orchestrators while also giving you time to address any necessary changes before the final release.

We are here to support you throughout this transition. If you have any questions or require assistance with testing, please reach out to our support team. Your feedback is invaluable to ensuring a smooth upgrade experience, and we look forward to working with you during this process.