Winter is Coming for CentOS 8

Winter is Coming for CentOS 8—but here is how you can enjoy your holidays after all.

The server environment is complex and if you’re managing thousands of Linux servers, the last thing you want is for an operating system vendor to do something completely unexpected.

That is exactly what Red Hat, the parent company of the CentOS Project, did when it suddenly announced a curtailment of support for CentOS 8 – sending thousands of organizations scrambling for an alternative.

In this article, we’ll review what happened with CentOS 8 and what it means for users who have already upgraded from CentOS release 7 to release 8. We’ll also look at your alternatives for replacing CentOS 8.

Finally, we’ll do a review of your other option: choosing extended support. Extended lifecycle support (ELS) can reduce the pressure to decide on alternative distribution and it may well be the most practical route for many CentOS 8 users.

Official support is critical

The difficulties around CentOS 8 involve the sudden withdrawal of official support. Official support window timeframes matter because it gives Linux users certainty that they will continue to receive bug fixes as well as patches for CVEs and security vulnerabilities that emerge.

A fixed end date for support gives users the ability to plan – either upgrading ahead of the end date, or migrating workloads to an alternative if upgrading isn’t a viable option.

While this is an important consideration for people who run a single CentOS instance and for small teams, official support windows become critical for those who depend on CentOS to support large-scale workloads involving big server fleets.

A single user or small team can quickly shift distributions, but planning for any changes that involve thousands of machines is a whole different story.

A free Linux distribution – with rock-solid official support

CentOS had its origins in 2002. The project, a 1:1 fork of Red Hat Enterprise Linux, went through various changes over time. In 2014, Red Hat announced that it would officially sponsor the CentOS project – but in doing so, Red Hat took full control of CentOS, including intellectual assets, and the governing board.

Red Hat invested a lot of effort into the CentOS project, and CentOS enjoyed a fixed release schedule with equally fixed, reliable support windows. As of late, the CentOS project was quoting 10-year maintenance support windows which was fantastic news for enterprise users who could adopt new releases at a pace that suited them, with long time frames for planning and testing.

And, of course, CentOS is entirely free – saving companies thousands in licensing fees. For example, when CentOS 7 was released in 2014, users were told that they will continue to enjoy support through June 2024. With CentOS 8 coming out in September 2019, it gave enterprise users a long time frame to test and switch to CentOS 8.

Some CentOS 6 and CentOS 7 users moved quickly and adopted CentOS 8, but these users were in for a surprise.

What changed with CentOS 8?

When CentOS 8 was released, the CentOS project (and by that we really mean Red Hat) promised that it would continue to support CentOS 8 for about ten years officially – just like it did for CentOS 7. The original end of life date for CentOS 8 was May 31, 2029.

That’s an excellent support window for a free-to-use, enterprise-grade Linux OS which is also 1:1 binary compatible with RHEL. It meant that enterprise users could essentially avoid paying RHEL license fees, while still working with a trusted distribution.

Unfortunately, the good news ended rather suddenly in December 2020 when Red Hat unexpectedly announced that it will no longer release CentOS as a stable release at regular intervals, instead focusing on CentOS Stream – a rolling release model, which is delivered differently and whose suitability for enterprise application is still unknown.

Products come and go and a change of direction can be somewhat understandable, but the real sting in the announcement was that official support for CentOS 8 will be curtailed by almost eight years – with end-of-life now on Dec 31, 2021 rather than the originally promised May 31, 2029.

After that date, the CentOS Project will no longer publish updates for CentOS 8. Bugs won’t be fixed but, more critically, new vulnerabilities won’t receive patches. In other words, if a major flaw in – for example – the Linux kernel emerges, you simply won’t get an automatic patch for CentOS 8.

That is in contrast to what organizations were originally promised for CentOS 8 – a matching patch within 72 hours of the patch being released for RHEL 8, right through the middle of 2029. It creates an enormous headache for tech teams that must now act fast to replace CentOS 8.

Why doing nothing isn’t an option

You might think that your workloads are running just fine, and that there’s no need to update your CentOS 8 instances to apply bug fixes. Or, that you can simply apply internally coded patches or other remediation measures should a threat arise.

In reality, the risks of running an unsupported OS are significant. You can use this calculator to estimate the costs and get a rough figure for your particular infrastructure. We’ve published an in-depth article here, but let’s do a quick recap of the potential problems you face when your OS is no longer enjoying official maintenance support.

• Breaking compatibility and reliability. An OS is surrounded by other software components and if you fail to update your OS with bug fixes, you may find that updates to other components break compatibility — you end up with updated software and services, but an OS that was never updated with the feature change.

• Security risks. This is the big one: if you don’t receive regular updates to your OS you will rapidly accumulate a growing number of security holes in your workload as more and more vulnerabilities get published in public – but never fixed on your systems. All it takes is one entry point for a hacker to gain entry and potential catastrophe to occur.

• Compliance problems. Compliance requirements such as PCI require that systems are patched against vulnerabilities within a specific time frame. When your OS is unsupported you are at risk of breaching compliance requirements which can lead to stiff penalties, the loss of customers – or indeed losing the right to do business altogether.

That’s just a brief insight into the potential problems of running CentOS 8 past the end of this year. It’s an enormous risk which is no wonder that companies are rushing to try and come up with alternatives.

The problem with CentOS stream

Red Hat isn’t discontinuing the CentOS Project altogether – CentOS will continue to exist in the form of CentOS Stream, which will always be one step ahead of the latest RHEL release. While Red Hat is suggesting that CentOS Stream is a drop-in replacement, that’s only true for a limited number of use cases.

Many Linux OS use cases – particularly in the enterprise environment – depend on stable releases: fixed functionality that can be tested, and the assurance that nothing of substance will change until the next release. Indeed, Red Hat’s own CTO has said that CentOS Stream is not a replacement for CentOS 8.

The move to the new CentOS Stream may affect the release stability. It will no longer have exactly the same package versions as RHEL – in fact, packages will land in CentOS Stream before making it into a fixed RHEL release. Binary compatibility may suffer, and some organizations’ workloads cannot easily accommodate this.

CentOS Stream would be a perfectly acceptable replacement for some users – some scientific teams, for example. However, most large-scale user cases involving more than a handful of machines will need to examine alternative operating systems – or alternative support options. And there’s not much time left given CentOS 8 is end-of-life in just a few months.

How about downgrading to CentOS 7?

In one of the few cases where leaving things to the last minute has paid off, CentOS 7 users are continuing to enjoy the support window the Red Hat originally committed to – with CentOS 7 maintenance support set to last until June 30, 2024. That’s a rather useful two and a half years beyond CentOS 8 support.

So how about going back to CentOS 7 as a temporary measure? There is, unfortunately, no supported downgrade path back to CentOS 7. Yes, some unsupported solutions are out there, but you’re at risk of ending up with a system that is in some type of Frankenstein state – containing elements of both releases. You’re almost certain to experience problems further down the line.

Taking a look at binary compatible alternatives

We will divide your alternatives to CentOS 8 into two categories: distributions that are binary compatible with CentOS 8 (and by consequence RHEL 8), and distributions that are relatively close in purpose – but that will require more work to adopt. We’re taking this approach because so many organizations relied on the 1:1 binary compatibility between CentOS 8 and RHEL.

Choosing a distribution that is binary compatible with CentOS 8 implies that your team has relatively minimal work in terms of switching distributions. In fact, you may be able to switch from CentOS 8 to an alternative distribution just by running a script – but, tech teams will still need to double-check that nothing is broken in the transition. These are your binary compatible options:

Red Hat Enterprise Linux (RHEL)

We mention RHEL first because, by definition, RHEL 8 is 1:1 binary compatible with CentOS 8. Yes, ordinarily, there is a licensing fee associated with RHEL, but due to the backlash against Red Hat’s decisions around CentOS, Red Hat decided to extend the free version of RHEL.

Red Hat has expanded the free of charge Individual Developer subscription program to now include workloads that involve up to 16 systems. So, if your workloads involve 16 or fewer CentOS instances and if you’re certain you won’t require a larger number of machines, RHEL could be a good choice involving minimal disruption.

Most enterprise CentOS deployments have far more than 16 active instances and these workloads will incur a licensing fee.

Oracle Linux

Enterprise users might naturally look towards another free enterprise alternative – Oracle’s 1:1 binary compatible fork of RHEL, called Oracle Linux. Oracle claims that Oracle Linux is fully compatible with CentOS, and anyone who already uses Oracle products will find the tight integration with Oracle’s other products helpful.

While Oracle Linux has a proven track record in the enterprise space, there are some issues around the direction of other products under the Oracle aegis, like Java, that have come up during the years, and, arguably, instilled some reluctance when going with the brand.

AlmaLinux

AlmaLinux OS is a 1:1 binary compatible fork of RHEL – and therefore binary compatible with CentOS. AlmaLinux is under the purview of a 501(c)(6) non-profit foundation with a Board of Directors composed of people from around the industry and the community, and community adoption has grown steadily over the months. It already supports most hardware platforms supported by CentOS, is present on the largest cloud provider’s offers and has matched all the announced releases dates along the way.

There has been some competition between AlmaLinux OS and Rocky Linux, which was to be expected since both target the same audience.

That said, AlmaLinux was faster out of the gate with a production first release than Rocky Linux and the community reception has been positive. AlmaLinux has also recently become available as an OS install on Microsoft’s Azure and offer a set of RHEL UBI equivalent containers as well.

Rocky Linux

The early CentOS project merged with a project called CAOS Linux, founded by Gregory Kurtzer in 2002.. After limited involvement, Kurtzer moved on from CentOS to other projects and was needless to say unhappy about Red Hat’s announcement and the changing future of CentOS, so rapidly acted to create a new, binary compatible fork of RHEL – and called it Rocky Linux.

Rocky Linux is binary compatible with CentOS so it is easy to switch to. The open source project is, however, currently under Kurtzer’s full ownership and control although he has made statements about opening that up to others. So, again, there can be concerns that there might be a change of course with Rocky Linux – much the same as Red Hat did with CentOS.

Other binary compatible alternatives

CentOS users can also look at ClearOS and Springdale Linux, but in both cases the supporting communities are relatively small. Springdale Linux is backed by serious institutions though – with both the Institute for Advanced Study and Princeton University backing it. While ClearOS has links with HP Enterprise, ClearOS 8 has not yet been released which casts a shadow over the project.

Scientific Linux isn’t an option as the backers, Fermilab, had said they won’t release another version beyond release 7 – so there’s no alternative for CentOS 8 here. For some users, Amazon Linux could be worth investigating – it’s backed by the tech giant and is a CentOS-based clone of RHEL, but you can only run it on Amazon Web Services.

Non binary-compatible distributions to consider

You may well decide that RHEL and its related distributions do not offer any unique features – aside from the original advantage that CentOS is a free RHEL clone. Depending on your workload, migration may be relatively effortless – but you’d nonetheless need to prepare and test to a far greater degree compared to migrating to binary compatible distribution.

One of the most obvious alternatives is also one of the most established – Canonical’s Ubuntu. It is, of course, derived from Debian – which means it is some distance away from RHEL and therefore shifting from CentOS to Ubuntu will be a fairly big operation.

It all depends on how much of your code is specific to CentOS and whether you rely on vendors for software or write your own code internally. Either way, Ubuntu has the necessary track record and it may well be a sensible option.

There are plenty of other, trusted distributions you could think about. OpenSUSE, for example, is offered free for use by SUSE Linux and has a solid reputation, it’s been around for more than 15 years. You could also opt for Debian. However, switching to a new Linux distribution can be more complicated than it sounds. Some points you need to watch out for include:

• Monitoring and management systems need to change because the OS that supports your workload has changed significantly.
• Development efforts required – both to adjust everyday scripts, and to change the code in the applications that run on your operating system.
• Coping with different package management mechanisms – RPM on CentOS, RHEL and related distributions, PKG on Debian and Ubuntu.
• Time consumed and risk associated with the migration process, which comes down to a complete system re-install given the difference between, say, RHEL-based and Debian-based distributions.

In other words, choosing a distribution that’s not in the RHEL family may involve significantly more work than you intended and it’s not a decision to be made lightly.

Consider extended support to buy time

At the start of this article we promised you an alternative route that mitigates the urgency created by Red Hat’s decision. It’s a simple concept: relying on a third-party to extend maintenance support for CentOS 8.

A good extended support service will cover you for essential bug fixes and any emerging vulnerabilities. In other words, if a new threat emerges that affects CentOS 8 your extended support provider will roll out a patch to counter the threat.

That means that you remain secure – given that new threats are always patched – and compliant, given that your workloads do not accumulate vulnerabilities over time. By consequence, you can carry on running CentOS 8, buying yourself more time to switch to a new distribution.

TuxCare’s Extended Lifecycle Support (ELS) for CentOS 8 essentially continues the RHEL support commitment. In fact, ELS from TuxCare improves on what RHEL promised for CentOS – with patches rolled out within two working days instead of three. TuxCare also has the know-how and the reputation to deliver – with an established product that’s part of the CloudLinux product portfolio.

TuxCare has committed to providing extended maintenance support for CentOS 8 through 2025 – giving you several more years to make a decision about your CentOS 8 workloads, instead of just four months. It significantly reduces the pressure on your team.

Act now and secure your CentOS 8 workload

CentOS 8 maintenance support is ending, and it ends soon. Organizations that still rely on CentOS 8 do not have a huge amount of time to make a decision about an alternative distribution.

We’ve outlined a couple of distributions that you can essentially use as drop-in replacements, but given that two of these are brand new it is understandable that you may want to see how these distributions pan out before you commit.

If that’s the case, consider signing up for extended support to buy yourself some more time to decide. However, you must make a decision of some sort. Not acting is not an option – the risks are simply too great.

###

This article was written for The Hacker News by Joao Correia, Technical Evangelist at TuxCare. Correia has many years of experience in IT systems administration, where he learned the intricacies needed to keep a company’s stakeholders happy and its systems secure.