Having a Build Infrastructure for Extended Support of End-of-Life Linux Distributions

Intro

When it comes to providing extended support for end-of-life (EOL) Linux distributions, the traditional method of building packages from source may getting certain challenges. One of the biggest challenges is the need for individual developers to install build dependencies on their own build environments. This often leads to different alternative build dependencies being installed, resulting in issues related to binary compatibility. Additionally, there are other challenges to consider, such as establishing an isolated and clean build environment in parallel different package builds,  implementing an effective package validation strategy, and managing repository publishing..etc

To overcome these challenges, it is highly recommended to have a dedicated build infrastructure that enables the building of packages in a consistent and reproducible manner. With the build infrastructure, teams can address these issues effectively, and ensures that the necessary packages and dependencies are available, eliminating the need for individual developers to manage their own build environments.

OpenSUSE’s Open Build Service (OBS) offers a powerful build infrastructure that is already well known for teams working on customized or private software packages built on top of major Linux distributions, and we found it’s also useful to work on extended support for End-of-Life (EOL) Linux distributions. With support for RPM-based, Debian-based, and Arch-based Linux distributions, OBS proves to be a versatile solution capable of meeting the needs for Extended Support of End-of-Life(EOL) Linux distributions.

On June 30, 2024, two major Linux distributions, Debian 10 “Buster” and CentOS 7, are reaching their End-of-Life (EOL) status. These gained widespread adoption in numerous large enterprise environments. However, due to various factors, organizations may face challenges in completing migration processes in a timely manner, necessitating temporary extended support.

Once a distribution reaches its End-of-Life(EOL). Upstream support, including mirrors and infrastructure are typically removed. This makes a significant challenge for organizations seeking extended support, as they not only require access to a local offline mirror but also need the infrastructure to build packages in a manner consistent with the original upstream builds.

To achieve the goal of providing extended support for end-of-life (EOL) Linux distributions, we choice to have a local mirror of the EOL Linux distribution and also have a private OBS instance. To have an private OBS instance. You can download OBS code from the upstream GitHub repository and install it on your local Linux distribution, or alternatively, you can simply download the OBS Appliance Installer to integrate a private OBS instance as an build infrastructure.

By adopting a private OBS instance as the build infrastructure for extended support of EOL Linux distributions, teams can simplifies the package building process, minimize inconsistencies, and achieve reproducibility. This not only simplifies the development workflow but also enhances the overall quality and reliability of the packages being generated. Replicate the semi-original build process, ensuring that the resulting packages maintain the same quality, integrity, and compatibility as those generated upstream.

Allow us to demonstrate how OBS can be utilized for extended support of end-of-life (EOL) Linux distributions. However, please note that the following examples are purely for demonstrate purpose and may not precisely reflect the specific approach nor EOL product used in our actual project.

Config local mirror as DoD Repository

Once your private OBS instance is set up, you can login into OBS on it’s WebUI via https by ‘Admin’ with default password ‘opensuse’. You can only update some configurations with the Admin account. Including Manage Users and Groups, Add Download on Demand (DoD) Repository to establish a connection with the local mirror in Project created on private OBS instance.

Add DoD Repository in Repository tab on your private OBS instance:

Config DoD with url for internal local mirror of EOL Linux Distribution:

After you connected the local mirror with the EOL Linux Distribution in the private OBS instance. The private OBS instance became the build infrastructure, supporting various major distribution types, positions it as an ideal solution for a team seeking infrastructure to provide extended support for EOL Linux distributions.

Benefits of build infrastructure

OBS ensures consistency and reproducibility in the build process. Also provide a customizable build environment, allowing teams to tailor the environment to specific project requirements.

• Consistency and Reproducible Builds: OBS provides the option to perform builds in clean chroot or clean KVM environments, ensuring consistent and reproducible results. This reduces the risk of unexpected variations or errors during the build process.
• Customizable Build Environment: OBS allows teams to customize the build environment through project configuration, enabling them to define macros, packages, flags, and other build-related settings. This ensures compatibility with the specific requirements of the EOL Linux distribution being supported.

Customizable Build Environment.

• Isolation of Build Environments: OBS allows teams to create separate build environments for different projects or distributions, preventing interference or conflicts between builds. This enables efficient and parallel development on multiple EOL distributions simultaneously.

Each worker runs isolated with customizable build environment efficiently in parallel.

• Release Number Control: OBS offers control over the release numbers of built packages, ensuring compatibility and consistency with the original release numbering scheme. This is particularly important when working with EOL distributions.

See more details: https://en.opensuse.org/openSUSE:Build_Service_prjconf#Release

Demonstrate of other key benefits

Here are some benefits of using OBS with screenshots:

• Built-in Access Control: OBS offers flexible access control, allowing team members to have different roles and permissions for various projects or distributions. This ensures efficient collaboration and streamlined workflow management.

Manage users and groups and different roles:

• Download on Demand (DoD): OBS supports the Download on Demand feature, which fetches build dependencies on-demand during the build process. This simplifies the build workflow and reduces the need for manual intervention.
• Offline Build Support: OBS enables the creation of an offline build environment via DoD, allowing teams to build packages without an internet connection. This is particularly crucial when working with EOL distributions, where internet access and official repositories may no longer be available.

Fetching build dependencies packages from offline local mirror via DoD:

Start the build process after downloaded build dependencies from DoD:

• Flexible Project Setup: OBS allows teams to create separate projects for development, staging, and publishing purposes. This controlled workflow ensures thorough testing and validation of packages before they are submitted to the production project for publishing.

Build Result for Staging repo:

Automatically published in staging repo once package builds:

You can start the package validation process with staging repo. And then submit a package from staging to prod after it passed the validation process:

The package submit request can be reviewed by prod repo master to accept or decline:

• Repository Publish Integration: OBS seamlessly integrates with local repository publishing, ensuring that all packages for the build process are readily available and published. This eliminates the need for additional configuration on internal or external repositories.

The build process starts after package accepted into Prod repo. Here shows build status in Prod repo:

Automatically published to prod repo once package builds succeeded:

• Automated Build via Pipelines: OBS supports automated builds that seamlessly integrate into delivery pipelines. Teams can set up triggers to initiate builds automatically based on predefined criteria, reducing manual effort and ensuring timely delivery of updated packages to customers.

Package got automated builds when committed into an OBS project. Easily to integrate into a delivery pipelines.

In conclusion, OpenSUSE’s Open Build Service (OBS) provides a robust and comprehensive build infrastructure for teams working on extended support for EOL Linux distributions.

If you are interested to integrate a build infrastructure, or require extend support for any EOL Linux distribution. Please reach out to us. Our team is equipped with the expertise to assist you in integrate a private OBS instance into your exist infrastructure.

This article was originally written by Andrew Lee