There is nothing quite like the open source community to demonstrate the principles of freedom, democracy, and meritocracy – and the difficulties of bringing those principles to bear and keeping them pure when money is involved.
Open source software is not just about having access to source code, but that is a kind of protection against tyranny if parts of the community, particularly corporate sponsors who cut the paychecks for a lot of the developers – either directly or indirectly – who create open source software, particularly the Linux kernel and the operating system that is stacked up around it in various distributions.
Quite a big stink is being made their week as Red Hat has made some major changes to the CentOS variant of its Enterprise Linux. And that is mostly because since its creation CentOS has been what amounts to a community supported variant of Red Hat Enterprise Linux that sits downstream from the RHEL development – meaning, it is rolled up from the source code after Red Hat is done – and Red Hat has reversed the polarity of the CentOS project it took over in 2014 and plans to move it upstream, as CentOS Stream, thus turning it into yet another development release like the Fedora project has been for many years and which also feeds into RHEL in some fashion. (Don’t even start thinking about how CoreOS Linux, which Red Hat acquired in January 2018 and which underpins its OpenShift Kubernetes container platform, fits into all of this.)
As the world’s largest company devoted to the development of commercial grade open source infrastructure software and arguably the only company that will ever be able to make this model work from a commercial standpoint at this scale, Red Hat can afford to have many different kinds of Linux that its employees contribute code to. The company rakes in somewhere north of $3 billion a year selling support contracts for such software, and has a vested interest in making sure the Linux operating system keeps getting more and better features added to it as well as support for successive generations of hardware. And to be fair, Red Hat does its share of this work and has since the company was founded decades ago. It is in this sense, though, that companies really are paying for Linux.
CentOS Stream was announced somewhat innocuously in September 2019, two months after IBM closed its landmark $34 billion acquisition of Red Hat. That timing might be coincidence, but maybe not. IBM has promised to keep a hand’s off approach to Red Hat, and is a just as likely that the Red Hat team is making this change all on its own as it is likely that Big Blue is coercing it.
CentOS Stream was designed to create a half-way point between the Fedora development release, which is changing like crazy all the time, and the commercial-grade Red Hat Enterprise Linux release, which changes on a regular, predictable, and relatively infrequent cadence of about twice a year. To be more precise, CentOS Stream is the code-base for the minor RHEL releases, and parts of RHEL development were actually moved into the CentOS project to get everyone collaborating. Which was good.
Either way, Red Hat has had a free to pay problem even before CentOS was created in the late 2000s to package up the RHEL code and offer community support – meaning, free and collaborative – to that alternate, “bug for bug compatible” release of RHEL. There is nothing in the Linux licenses that prevents the community from spinning up and self-supporting its own variant of RHEL and calling it something else, but with somewhere around half of RHEL customers actually using CentOS – meaning they are not paying for support and therefore not helping Red Hat create its variant of the Linux operating system and its elaborate stack of container and virtualization management layers as well as storage.
When Red Hat was founded in the wake of Linus Torvalds creating the Linux kernel in 1991, and even when Red Hat went public in 1999, there were not yet hyperscalers and cloud builders and HPC centers were just in the middle of ripping out Unix-based federated NUMA system clusters and replacing them with Linux clusters supporting Beowulf and MPI. It surely looked like created an enterprise-grade Linux operating system that felt like a proprietary operating system like Unix or Windows in terms of its fit and finishing and upgrade cycle was going to be a great business. While it has been a good business to be sure – Red Hat has been plagued by the free to pay problem, and is no doubt tired of it. While the company never said this when it took over the CentOS project back in 2014, the idea was no doubt to have a tighter link between those who paid for RHEL and those who didn’t by using CentOS, presumably greasing the skids to get customers to move from free CentOS to paid Red Hat subscriptions.
But this market has not panned out as planned. The big hyperscalers and cloud builders have long-since grabbed a Linux kernel and maybe even a distro and created their own Linuxes, but if they could get out of that business they probably would. Facebook, for instance, intercepts CentOS Stream for its own implementation of Linux, adding whatever goodies it needs downstream, rather than just adopting RHEL proper. Somewhere north of 50 percent of the servers in the world are running some form of Linux, and somewhere around half of these, we estimate, are running a homegrown Linux that will very likely never be replaced by CentOS, CentOS Stream, or RHEL, much less any other Linux variant. (Ubuntu Server from Canonical, based on the Debian variant of Linux, has its own large base, and SUSE Linux Enterprise Server has a pretty large base as well.) In addition, HPC centers have their own variants of Linux (think Tri-Labs Linux) or those created by their supercomputer vendors (think the Cray Linux Environment) as well as a healthy dose of CentOS.
Many of the organizations do special things with Linux, tweaks for their own use cases, and they cannot be done by companies like IBM through Red Hat easily – and certainly not at the “free” price of having developers participate in the community and then using community support.
You might be thinking that one could fix this free to pay problem by making support contracts cheaper. So, in theory, if Red Hat cut RHEL support contracts in half, it could possibly pull in all of the CentOS users, keep the RHEL users, and have the same revenue stream and probably not have to deal with much more in the way of support calls because the upper echelon, who use CentOS in their distributed systems, probably know as well or better how to support their systems and applications than Red Hat does.
In practice, to make that economic argument, one would have to calculate the cost of self-support – those developers at the hyperscalers, cloud builders, and HPC centers are not free, after all, and are generally highly paid professionals at that. If you fully burdened their costs, you might discover that self-support is much more expensive than RHEL support. And still, while being a good economic argument, it wouldn’t matter because for these customers, homegrown Linux support is about having expertise and being self-reliant, and even perhaps to create a competitive advantage with the core Linux operating system.
What is the value of that, and how does it offset the cost? The very fact that there has not been a true standard Linux distro tells you. High enough that people will budget people instead of outsourcing support to Red Hat, SUSE Linux, Canonical, or any other company in most cases of upper echelon users.
Organizations that control their own software control their own fates. That is the lesson of both HPC and hyperscale.
The middle ground between those companies who pay for RHEL support and who can’t self-support is where CentOS has played, and that is where the newly constituted Rocky Linux, created by CentOS co-founder Gregory Kurtzer this week and in honor of one of his CentOS partners, Rocky McGough, who passed away some time ago. (Lance Davis is the other CentOS founder.)
Kurtzer was the HPC systems architect and technical lead at Lawrence Berkeley National Laboratory and for several years was the corporate advisor at RStor, which initially tried to commercialize the Singularity Kubernetes container that Kurtzer created at Berkeley Lab. Singularity was eventually acquired by Sylabs, where Kurtzer was chief executive officer until April of this year. Since then, Kurtzer has been running the HPCng project, which he says is making the next generation HPC environment, and now is taking on the additional task of essentially re-creating the CentOS project that Red Hat has just said has no future. But Rocky Linux, which has over 750 contributors to the project already, certainly does because people like Kurtzer never really needed Red Hat support anyway. Or, more precisely, they don’t think they do.
Imagine if Red Hat just vanished tomorrow. That all of the programmers who work on Linux from Red Hat just stopped, that the testing and qualification process just stopped. The Rocky Linux community might just find out they needed Red Hat more than they reckoned.
What is clear is that Rocky Linux is dependent on all of the work Red Hat and the rest of the Linux community does, and it is equally unfair that users of Rocky Linux, who are among the smartest techies on the planet, are so unused to paying for support that they think it is free. It isn’t.
There needs to be a balance sheet that reckons value created by developers against costs to organizations for providing support. Or, we just keep creating free communities like Rocky Linux and pretending that it doesn’t cost anything.
Perhaps everybody – and we mean everybody – should always pay a little something, no matter how small per node, into the Linux Foundation to cover these support costs anyway. Perhaps the financing of the support model should be more like public radio (which doesn’t really work) and less like pay per view TV (which people sometimes hack to get around). Individual users might pay $50 or $100 per node per year or some such fee for community support, which would literally and directly underwrite the work Linux developers do the world over, and even Red Hat and Canonical and SUSE Linux would pay this per node. You could offset this with the value each company brings to the Linux stack. Do away with this notion that Linux is free. It really isn’t. It takes the life force of people to keep it going. If you want more handholding, then pay for a RHEL support contract from IBM.
It would be a messy balance sheet, to be sure. And one with a truly gigantic goodwill on that balance sheet, which is the most remarkable thing of all and something the Linux community should be proud of.
Linux Is The Next Platform, But Who Pays To Maintain It?
Red Hat has once again dropped another huge boulder into the normally serene – or at least relatively calm – open source waters. Back in December 2020 it killed off the CentOS distribution that lives downstream from Red Hat Enterprise Linux and created the CentOS Stream variant that lives upstream …
Big Blue Turns In A Solid Quarter For Systems
By all accounts, Big Blue had a pretty good quarter ending in June, with sales of its System z16 mainframes skyrocketing upwards as they do every couple of years at the beginning of a new cycle and sales of its high-end Power10 machines also getting some traction. If everything goes …
The Ever-Reddening Revenue Streams Of Big Blue
Speaking in generalities across any aspect of history is always risky, but that is what the job of history is. The first wave of open source software in the enterprise in the 1960s through the 1980s was largely academic, and it wasn’t until the second wave of open source hit …
If RedHat truly feels like the author suggests, why open source products other than the core OS? It could keep some products proprietary, so you would only get them as part of RHEL. Why open source OpenShift, for instance? It was proprietary to begin with. Is RedHat’s ‘open source everything’ strategy really the best way?
It is a reasonable question. I just think the value provided should go to those who are providing the value, whether they work for any given company or are doing it for altruistic reasons of enlightened self-interest. Maybe having a company, as such, is taking it too far, when what we needed was a foundation with a non-profit stance.
This was an article worth reading.
Thank you!
Thanks – those that actually help keep Linux distributions working for commercial entities to profit off might not actually want to pay the Linux Foundation, which is a purely commercial trade foundation these days.
Debian is altruistic, enlightened self interest, for example, that feeds Ubuntu and all derivatives – but is reliant on donors to help provide some hosting – and SPI has a non-profit stance.
Your point about the whole infrastructure being based on CentOS is an interesting one: If Facebook is using CentOS Streams, Amazon is using forked CentOS, Oracle is using forked RHEL in the same way as CentOS without paying RHEL – Red Hat have problems if they want to rejig Red Hat and rely on outside developer goodwill. This was, essentially, a huge misunderstanding on all sides but it has left a bad impression, deservedly or not.As you note, there is no such thing as a free lunch – but the people who don’t pay Red Hat provide their support in-house.
If Red Hat were to disappear tomorrow, it would leave a hole but that might be in commercial training courses, consultancy and event sponsorship as much as in the whole of the huge Linux infrastructure. The Fedora project might suffer short term but might pick up the slack or fade away: Red Hat-specific developers would find other useful work to do.As much of the valuation is goodwill as subscription, in some ways.
Perhaps Ballmer wasn’t that wrong.
Perhaps he now likes Linux because he got aware, that Open Source will eat itself, because you can only choose between two ways:
-non-profit, interest-based and idealistic, which will not pay your bills on the side of the developers, and which is not a reliable basis on the side of the users, as today this and tomorrow another project may be interesting for idealistic developers, and the project dies (btw. somebody HAS to pay the developers somehow, e.g. the company they are working for perhaps allows them to participate in the Linux development on purpose, so either the developer has less time for company projects, or he does it in his free-time and by this has less power and concentration on the company’s projects – one only has 24 hours a day to eat, sleep, …; same for students: someone pays if they stay longer at the university, and if it is a state university, people might pay with their taxes, also if they never use Linux)
-profit-based, which would work, but the licenses you have to accept will eat you, sooner or later, as the article somehow states, as you cannot keep your knowledge private
As a private user, paying for Linux in a general way, as the article proposes, is something I could not accept at the moment, as perhaps I pay something, and my favorite distro just disappears, as the developers found something else, which is more interesting, and create a new distro…
It is just too unreliable then to pay for “any” Linux or Linux in general.
Paying for RHEL? Well, if there could be any price model for private users, in a price region as in the past also Windows NT e.g. was (I was an owner right from the start with the 3.1 version; on 3.5” floppy disks), so ~350€ for 10 years support.
Problem is: Linux, the base of RHEL, isn’t owned by Red Hat, so they cannot guarantee such a timeframe, as perhaps Microsoft can with Windows, which code they own (where I would choose their LTSC (former LTSB) versions with 10 years support without feature-party; my still used Windows 8.1 internally shows as winblue.LTSB).
Linux was started as, and also stated as “free”. So, forgive me but, it is not right, if a company starts using a system as basis, which is free, will be free, and then at some point of time says “oh, we were wrong; you have to pay now in every case” (which they don’t say directly, I know). They knew what they did, when they started in the past (hopefully). They knew the licenses they have to accept.
It is correct: Every work should get a reward somehow, but in this case it is impossible to unite two sides which are that opposite: Commercial (we need to earn money) and non-Commercial (it will be and must be free).
Yes, in that way, Linux is somehow a cancer, as Ballmer stated, but a cancer that perhaps eats itself sooner or later.
(Microsoft does it very clever, from a commercial point of view: They keep their Windows, and add Linux, and now also Android, as somehow subsystems, which they adjust to their needs, with smaller effort than building both systems from scratch, as Microsoft mostly cares about “use our Azure – choose whatever system you want, but use Azure”.
Clever from a commercial point of view only, also if convenient for the users, as the problem is that they, on the Windows side, cumulate the vulnerabilities of the respective OSs by this integration.)
Perhaps their might be any other OS base in the future, which will be used under any other license, which allows to make improvements and further steps in closed-source, but then, we are again at the beginning.
Long-term stable releases are costly to maintain and often limit users to older application versions. Not only Red Hat, but also many CentOS users will benefit from the new model. Linux is not the only open-source project suffering from what economists call the “free-rider” problem. https://xkcd.com/2347/'s “random person in Nebraska” is a real problem. An open peer-reviewed funding model with user-driven priorities and developer-generated proposals would go a long way towards ensuring that open source remains a viable base for operating systems and applications.
Now you are talking. I like the Green Bay Packers model myself–the town owns the team, the town pays, and the town benefits. I never thought of Linus Torvalds as Aaron Rogers before….now.
Great article. Thanks….but in the interest of “balance”, I’d like to point out that alternative “eco-systems” are seriously ugly. I’m thinking of the coercive models pushed by Apple, Microsoft and Oracle, which can be summarised as “Pay up…or else!” I know where my sympathies lie!!
I understand. In the long run, Linux is what? 75 percent of the datacenter? More?
The “free to pay” issue is more nuanced than the article suggests, since Red Hat itself free-rides on the developers of much of the software it offers.
You’ll recall the issues with OpenSSL, which were basically because not a dollar of the millions being made by Red Hat flowed through to the people who wrote that fundamental software.
The Linux kernel is vital to Linux: Red Hat is a dominant and important contributor. But even so Red Hat’s contributions are about 5% (by changesets or LOC). The hyperscalers are more than that, as are the CPU vendors.
Self-support for the hyperscalers is essential. When Facebook is down it is not going to log into the Red Hat website and log a P1 case. The opportunity cost of lost advertising impressions pay for Facebook self-supporting the Linux it uses. Red Hat reducing its support rates to even 1% of its existing charge would not change that decision.
Legally, the Linux experience is dominated by the harrowing experience with SCO. If Google were to pay Canonical, then that contract opens Google to litigation from Canonical. Google makes so much money it could be worth a venture fund buying Canonical just to access that possibility of litigation with Google. Better for Google to take a free license with no support, but no contract. Far better again for Google to require its staff to use Debian, which can’t be purchased and used as a litigation vehicle.
The basic complaint of your article is that it’s unfair that Rocky Linux free-rides on Red Hat and other developer’s work. But it is somehow simultaneously fair that Red Hat free-ride on the work of other developers (who in turn free-ride on the work of other developers — as an author of networking software I am free-riding on the work of the GUI developers). The point of free software is that everyone free-rides. That’s the essential feature, not a bug.
Red Hat also free-rides on CentOS, especially in documentation. Search for an activity you might do on RHEL. Append that search with “RHEL”, compare with the results for “CentOS” or “Fedora”. Now maybe Red Hat will get around to writing the massive manuals which used to accompany Unixes from Sun or DEC, but in the meantime they are free-riding off CentOS users.
As for *everyone* paying a little for Linux — economically we call that “taxation”. Let’s take the Linux kernel: at best 15% of contributors are individuals. So you’re going to tax every Linux user (billions of people who already paid for phones and modems) and give the money to (in order by changed LOC): Intel, Huawei, Google, NXP, Red Hat, Code Aurora, Linaro, Facebook, BayLibre, AMD, IBM, MediaTek, Arm, TI, SuSE, Oracle, Nvidia. What strikes me about that list of recent major Linux contributors is that they are some of the wealthiest corporations on the planet. The very corporations we should be receiving more taxation revenue *from*.
(Resubmitting, hopefully it won’t trash the paragraph breaks this time)
The “free to pay” issue is more nuanced than the article suggests, since Red Hat itself free-rides on the developers of much of the software it offers.
You’ll recall the issues with OpenSSL, which were basically because not a dollar of the millions being made by Red Hat flowed through to the people who wrote that fundamental software.
The Linux kernel is vital to Linux: Red Hat is a dominant and important contributor. But even so Red Hat’s contributions are about 5% (by changesets or LOC). The hyperscalers are more than that, as are the CPU vendors.
Self-support for the hyperscalers is essental. When Facebook is down it is not going to log into the Red Hat website and log a P1 case. The opportunity cost of lost advertising impressions pay for Facebook self-supporting the Linux it uses. Red Hat reducing its support rates to even 1% of its existing charge would not change that decision.
Legally, the Linux experience is dominated by the harrowing experience with SCO. If Google were to pay Canonical, then that contract opens Google to litigation from Canonical. Google makes so much money it could be worth a venture fund buying Canonical just to access that possibility of litigation with Google. Better for Google to take a free license with no support, but no contract. Far better again for Google to require its staff to use Debian, which can’t be purchased and used as a litigation vehicle.
The basic complaint of your article is that it’s unfair that Rocky Linux free-rides on Red Hat and other developer’s work. But it is somehow simultaneously fair that Red Hat free-ride on the work of other developers (who in turn free-ride on the work of other developers — as an author of networking software I am free-riding on the work of the GUI developers). The point of free software is that everyone free-rides. That’s the essential feature, not a bug.
Red Hat also free-rides on CentOS, especially in documentation. Search for an activity you might do on RHEL. Append that search with “RHEL”, compare with the results for “CentOS” or “Fedora”. Now maybe Red Hat will get around to writing the massive manuals which used to accompany Unixes from Sun or DEC, but in the meantime they are free-riding off CentOS users.
As for *everyone* paying a little for Linux — economically we call that “taxation”. Let’s take the Linux kernel: at best 15% of contributors are individuals. So you’re going to tax every Linux user (literally billions of people) and give the money to (in order by changed LOC): Intel, Huawei, Google, NXP, Red Hat, Code Aurora, Linaro, Facebook, BayLibre, AMD, IBM, MediaTek, Arm, TI, SuSE, Oracle, Nvidia. What strikes me about that list of recent major Linux contributors is that they are some of the wealthiest corporations on the planet. The very corporations we should be receiving more taxation revenue *from*.
While RedHat has contributed quite a bit of money to make RHEL, the amount of money they did not have to spend because GNU, Linux and so many other open source projects share code is even more significant.
That’s the point. By sharing development efforts everyone wins. Seen another way, RedHat would be very different or not exist if they had to start from scratch by bootstrapping their own compiler as the first step.
Writing your own compiler is what Mark Williams did when they made the Unix clone called Coherrent. The comparison of that to current Linux distributions shows how much more is possible with the cooperative sharing of software investments between companies and interested parties.
It doesn’t take an MBA to realize how this works. If fact, it’s possible having an MBA makes the financial advantages of sharing source code more difficult to understand.
The fragmentation of Linux desktop has killed off the very advantage it has which is the open source. It has allowed far too many separate projects to exist and be created in order to offer way to many choices. Its not even productive choices, many are just rehash of other projects with slightly skewed differences. Its not even changes worthy of a separate distro in my opinion. Sometimes being to open and not being realistic that a more defined ecosystem would benefit in the end. So you have a bunch of different distro’s with a small group of users?? Can you imagine if this happened with Windows or MacOS?? Too confusing and not helpful when developing and in fact makes it much harder which is why Linux has fallen so far behind in many areas over Windows and MacOS.