Linux is almost 100% of the market share on servers, and almost 100% of mobile devices (besides Apple). So why is it stuck at 10% on desktop? (Or 20% if you count ChromeOS, or 1% if you only count people’s “work PCs”, such as in corporate offices and work-from-home setups). It’s because Linux has conflicting goals.
Some percentage of Linux users want maximum flexibility to control every aspect of their desktop. I’ll call them “Control Maximalists”. That open-endedness and modularity comes at a cost. With all of that flexibility, it’s hard to hide all of it from the user. But they don’t care, they love it. Linux config is love, Linux config is life. But, there is another type of user. Users who just want to “get stuff done”. These are usually professional users who heard that Linux is free and easy (“so easy my grandmother uses it” as every Linux evangelist claims). They’re fed up with Windows or MacOS (often for perceived reasons that don’t match reality), and they want a change. I’ll call them “Productivity Maximalists”.
Unfortunately, it’s difficult for Linux to appeal to both types of users. You can see distros bisecting along these two lines. Arch is for Config Maximalists. Ubuntu is for Productivity Maximalists.
But of course, there are a thousand other types of users. There are Freedom Maximalists, who are one step away from switching to GNU/Hurd at all times. There are Gaming Maximalists, who just want better gaming performance, which Windows denied them. There are Microsoft Minimalists, who at the very least, want to avoid all Microsoft products in their lives, and at the worst, actually hold a grudge against Microsoft for some perceived wrongdoing and want to destroy it by promoting Linux. There are Aesthetic Maximalists who only like Linux because you can make your desktop look like beautiful (for whatever your definition of beautiful is). There are Corporate Managers who just want to supply their employees with computers (that are hopefully easy to use), but maybe they can’t afford a Microsoft license. There are Grandma Maximalists who think that Linux is “easy to use” as long as “their grandma can use it”. There are austere Linux Monks who see using Linux in the most difficult way possible as good for the soul.
How can you appeal to all of these types of users? It’s a challenge. It would be easier if Linux were less modular. Linux right now is like if every car in the world used the same size engine compartment. Monster truck? Prius? Same size engine slot. The monster truck’s engine hangs out of the top of the engine bay. The Prius’ engine bay is mostly empty. Yet, that’s where we are. Everyone uses the same pluggable components. Everyone just knows that to make a Linux distro you take (1) the kernel, (2) add an “init system“, (3) add a package manager, (4) add Wayland or X, (5) and then add a Desktop Environment. All init systems are pluggable with Wayland or X. All package managers work with all init systems. All desktop environments work with all kernels. I believe that maintaining that compatibility is a drain on the resources of developers, and constrains them from doing what they need to do. If you write a desktop environment, it need to work with all init systems, and all kernels.
Why was switching systemd such a hotly contested topic? If Microsoft switched their init system from FooBarWinA.DLL to FooBarWinB.DLL, would anyone notice? Who was angry about systemd, and why? Well, systemd made a lot of changes that annoyed server administrators. People who want maximum control over their servers. Should desktop users care that they’re using SysV init or Upstart or SystemD? No way. But that’s another point…
Desktop Linux is stuck living in the shadow of server Linux (and to some degree mobile Linux). Desktop is a second-class citizen. Linux decisions are made that will help the server market. The desktop market is mostly ignored. So why do server and desktop Linux use the same init systems? Why do we have the same driver model? Want to know why we were stuck with X Windows for so long? So server operators could pipe GUIs out of their server farm. Really. Why do we bother to share anything with the server market? In some ways, the server market is at odds with the desktop market. They make decisions that massively impact desktop Linux, and don’t ask for our input.
Vertical integration is good. Right now we have the opposite, where every project is landlocked between other projects. Let’s say you did want to make a desktop environment, but you find a bug in wayland. You talk to the wayland developers, and they say it isn’t a bug, it’s a feature. What can you do? You’re doomed. You wanted to add a user feature (IDK maybe you wanted to create a full-screen firewall popup like Windows), but wayland isn’t giving you enough power to do that. You need to fork wayland? And now you’re stuck pulling patches from upstream forever?
Or let’s say you want to switch Linux to a case-insensitive file system, like NTFS (because “Productivity Maximalists” don’t care that foobar.txt is different from Foobar.txt!). We can use NTFS, but it’s difficult because Linux can’t boot from a NTFS partition (because they stupidly implemented the NTFS driver in userland). So we need to boot from ext4, and keep user data in an NTFS partition. That’s an unnecessary partition, which complicates the install process. So let’s try using ext4’s new case-insensitive capabilities. Unfortunately, ext4’s case-insensitivity is on a per-directory basis, so to protect the user from case-sensitivity, you need to add a kernel filter to make sure that no usermode program (such as if the user accidentally installs Nautilus or Nemo) creates a directory that is case-sensitive. Oh also, you may need to hack GTK and QT, because they may assume that files on Linux are case-sensitive.
My point of this is, Linux is trying to be all things to all people, and people have tried so hard to make things modular, they lost the ability to make bold decisions and push for user experience (things like systemd and wayland are refreshing counterexamples, though).