Mobian started as a hobby project, triggered by the excitement of finally being able to hack one’s mobile phone at will. It is fair to say that the device that made it possible back then was the original PinePhone. However, after running once again into the same difficulties we experienced over and over, we’re wondering whether maintaining support for this device is still worth the effort…

Of kernels and men

In the beginning, the mobile Linux world was full of happy people, hacking on many different aspects of the system in a very disorganized but fruitful way, one of those aspects being the Linux kernel. Back then there was a single reference source tree, maintained under the PINE64 umbrella (but still being a community effort, no one being funded in any way to work on this kernel).

Things were obviously not ideal, however many talented people were talking to each other, trying to improve hardware support and fix the many bugs we were experiencing at this point. Until, I don’t quite remember how, it all turned into a race, in which the goal was to have better (or at least identical) support than the neighbouring projects.

In particular, one very skilled developer set to solve many of the problems we had back then, and published their work as a separate repo. Several mobile distributions quickly switched to this kernel, effectively abandoning the shared kernel source we were all using until then. This resulted in a loss of interest and, ultimately, collaboration on the kernel ceased: some ended up using megi’s kernel directly, while Mobian forked the “Pine64” one and cherry-picked interesting patches from megi’s tree, but it all happened without further cross-distro discussions.

Our single reference tree was then left rotting (the last kernel compatible with the PinePhone being based on 5.9, which was released nearly 3 years ago!) and unmaintained since then.

A single point of failure

The end result can be summed up as follows:

  • those relying on megi’s kernel depend on them to update and/or improve it
  • Mobian carries a bunch of patches we don’t really understand, some of those being dirty hacks just good enough to “make things work”

So basically, the whole community set up a single point of failure when it comes to the original PinePhone’s kernel. To be fair, this didn’t happen overnight, and we could have taken different decisions at various moments, leading to a better situation. megi also never asked for this responsibility, nor did they ever commit to ensuring long-term support for the whole community.

On the Mobian side, we did (and still do) our best to maintain our own kernel and rebase the patches we carry on (at least) each LTS release. This proved to be a complex task, and we repeatedly ran into the same issues over and over again:

  • some patches wouldn’t apply at all due to major upstream changes
  • once all patches applied cleanly, the resulting kernel couldn’t be built anymore due to internal API changes
  • simple, dirty hacks would stop solving the problem they were meant to fix (and sometimes ended up creating whole new issues)

We usually ended up browsing through megi’s git history, finding a new (or modified) patch fixing the problem for us, further increasing our dependency on a single person.

The “mainline-based kernel” myth

So, what does “mainline-based” really mean? In our community, that’s a kernel carrying a reasonable amount of patches (with no clear definition of this reasonable amount) and based on a recent enough upstream kernel (usually that means “not older than the latest LTS version”).

Contrary to Android vendor kernels, they can be relatively easily updated to the newest kernel releases, but like all downstream forks, this can only be done manually by someone with enough skills to complete this task.

As such, a mainline-based kernel doesn’t provide any more guarantee than any other downstream fork of any project, certainly not the guarantee that it will always be updated to the latest stable and/or LTS kernel. This guarantee can only be obtained by fully upstreaming hardware support, and it still requires some maintenance effort, even if this effort is significantly reduced.

Upstreaming patches require 3 elements:

  • the interest to push patches there
  • the skills to understand the patch(es) one is upstreaming, so they can address the comments and requests for improvements they receive after posting the patches
  • the time to rebase the patches on top of the latest upstream commit, prepare the submission and engage in constructive discussions with the reviewers

When it comes to the original PinePhone, the upstreaming effort has pretty much completely stalled by now, as it seems no one in our community possesses all of those elements.

Giving up?

megi’s recent decision of shutting down their GitHub account and blocking pulls from their self-hosted git repos has made our work more difficult; we’re also becoming more and more tired of “losing” our time in those endless kernel upgrades for this particular device.

As of today, it appears very few of the Mobian developers are willing to keep maintaining our sunxi64-linux kernel, meaning it could be stuck on the 6.1 branch forever.

We are therefore considering dropping support for both the original PinePhone and PineTab once the current kernel branch reaches end-of-life, ultimately turning those devices into high-tech paperweights.

No decision has been made yet, but this matter is being actively discussed. Even if a Mobian contributor were to commit to maintaining this kernel (which still seems very likely at this point), we would once again be relying on a single person and support for those devices should still be considered in “life support” state. That is, unless there is a larger-scale, community-wide effort to upstream more (most?) of the required patches…

Appendix I: We hold no grudge

While megi has been mentioned numerous times in this post, we don’t hold any grudge against them: they already did a lot to help all of us. They should actually be thanked for that, and we have to respect every single decision they made, even those that made us uncomfortable. They never had any obligation towards this community, and did way more than they had to.

One could argue that the community at large is responsible for not attempting to upstream more and instead “simply” accepting the fact that the bus factor for the hardware support on a couple devices equals one. And we would be to blame as well, as the choices we made were equally bad.

Appendix II: What about other devices?

We do hope this scenario won’t repeat for other devices, and have good reasons to keep hoping: apart from Rockchip-based devices (PinePhone Pro and PineTab 2), the kernels we ship are maintained by multiple-persons communities and/or developers whose work is backed and funded by their employer. Those developers also have a strong interest in upstreaming their work and demonstrated both their ability to do so over the past months/years. As long as they keep up their great work, upstream support for those devices will only improve over time, ensuring we can still use the latest kernels for those.

Our kernel for Rockchip devices includes fewer patches than the PinePhone one, and patches we mostly understand and can rework if needed (and maybe even submit upstream!). The PinePhone Pro also triggered more interest across various communities than its predecessor, leading to more developers actively trying to upstream support for this device (the libcamera developers, for example, having done a great job at improving drivers for the camera sensors).

As a consequence, we do believe and hope the issues we’re facing with the original PinePhone can be avoided for other devices, provided the community as a whole keeps working towards better upstream support for all mobile devices.