For some reason it blows people’s minds when they realize that most of the code on their PC is running in some embedded chip in a peripheral just moving data around. They think it’s meaningful. It’s not.
These peripherals provide a stable interface to whatever OS you install. Windows/Linux/BSD/FreeRTOS… All of them are on the same footing, and must use the same hardware interfaces. So why does it matter if deep in some chip on your motherboard, there’s a CPU core running embedded Linux?
If you’re a hardcore Stallman-style libre-head, then yes, this means something to you. You should try to avoid using such systems, because those chips almost surely contain non-libre code. But for the average user, it’s irrelevant.
Part of the argument seems to be that computers are getting more complex, and that this poses an existential threat to Linux. If Linux doesn’t evolve, it’ll become irrelevant, as subsystems of your PC, running closed-source proprietary code, slowly gain ground.
But that’s nonsense. The PC “form factor” has won. All of the “complexity” we’ve seen recently is just a reification or exposure of existing internal systems. Once upon a time, you made OpenGL calls, and your Nvidia/ATI driver translated those to some internal format, and then ultimately sent triangles to the GPU. Now, we have Vulkan, which lets you get access to that intermediate format. But you don’t need to use it. You’re still free to use OpenGL (and you should).
One of the strengths of the PC ecosystem is the standardization. The modern PC hardware, which I define as UEFI/ACPI/XHCI/PCIE, is universal. You won’t find modern x86 hardware that deviates from it. If Linux tries to get too fancy and start tinkering with the lower-level details of the system, then that just creates fragmentation. We should try to avoid concerning ourselves with those low-level details (except perhaps in cases where security improvements can be added, without breaking compatibility).
Looks like the wonderful pry-nav gem has fallen into disrepair. No one can push to rubygems, the creator is MIA. A common story these days, right? So I went looking for forks. Of course, lots of people have “forked” it on Guthub, but I want one that, you know, has had improvements and modernizations made to it. Luckily, someone did that, and renamed their fork to “pry-moves”. Looks like it has lots of additional movement commands also.
In the world scientific community, our rate of yearly innovation seems to be slowing. Let’s look at a chart of the important technological innovations per year divided by world population:
How do we interpret this data? What could it mean? Does this indicate that we’re lazy and stopped innovating? Or we’ve disincentivized innovation? Or have we ignored the human cost in innovation, and we’re feeling the effects of greater innovation requiring, simply, more humans to be involved in the process? All of these questions are tangential to this discussion, but they’re fun to think about. No, I think it’s enough to accept that this is happening, and try to make interesting points about computer software.
Mapping this onto computer technology works wonderfully. We saw in intense flurry of innovation around computers from 1940-1990, and then things kind of flattened out. Is the software on your computer materially different from something that existed in 1990? Not really. Moore’s Law ran for a few more cycles and allowed us to render more HTML elements, more 3D polygons, and do some nice machine learning. And perhaps you draw the line somewhere else. However, it’ll be hard to find someone who thinks that computer technology didn’t stagnate at some point. Computers have changed since 2000? What about 2010? Does the progress over the last 5 years match the progress from 1985 to 1990?
Software is a similar story. It’s kind of hard to compare modern software to something from 1985 or thereabouts, because hardware has advanced so much. And people tend to not be aware of minicomputers or other business-focused machines, and only want to compare desktop PCs. This is unfortunate, because in 1985 desktop PCs were a primordial ooze while IBM and DEC were selling machines with capabilities much, much more advanced than that. However, once you know where to look, it becomes clear that software progress slowed a long time ago. A few discoveries have been made, especially in the areas of compression, machine learning, computer vision, distributed computing, and cryptography (including of course cryptocurrency). But mostly the average programmer has had a remarkably similar job. Looking at GUI toolkits from 1990 is very much like looking at React or Ember in 2021. We use JavaScript (or TypeScript) instead of C++ now, but has the shape of the codebase really changed? We have the same concerns: z-index, layering, layout, callbacks, events… Meanwhile, if you talked to a programmer from 1990 and asked them to imagine the year 2021, they’d certainly say that an AI would be doing the GUI layout, and us programmers would be free to think at a higher-level. Yet here we are, pixel-pushing as always.
So if software isn’t advancing, why does it keep changing? And why do we hate on anyone who doesn’t change?
I think this XKCD is a bit wrong in the analogy there. MS-DOS is like gunpowder, because it’s a simple solution that works, yes. However, more importantly, it’s the solution you have and not some hypothetical solution that a consultant could bill you $300/hour to build. Businesses run on money, you know. The cost of maintaining a MS-DOS machine is much smaller than- …eh, enough of this tangent. You get the idea.
Anyway, back on the topic of the software world “changing”. By that I obviously mean that the software tools – the languages, frameworks, and environments that we use – keep changing every few years. Why is that, anyway? It seems like the simple solution is “because we’re crazy!”. But, I think that a more logical explanation is that there’s some basis in reality here. Something about the nature of software development drives us to rip out perfectly good infrastructure and code and replace it every few years. And I don’t think it’s boredom, or making work for oneself. It’s hard work replacing a perfectly good app written in, say, C++, with one in the New Shiny Language De Jure (let’s say, Go). And if you ask a programmer in the middle of one of those rewrites “is this fun?” the answer is usually “no, but it’s better than leaving it in C++!”. That hints at the true reason, I think. The true reason is, C++ (in this instance) has ceased to be a good substrate for this application. Perhaps any application.
I believe that programming languages/environments/frameworks/tools decay over time. Not through lack of contributions, but through bad contributions. It seems to be the common belief that software gets better over time. But people also believe that software gets “old” and “crufty” somehow. Magically! It especially happens to the code in your tech stack. But it never happens to languages or tools. How did your C++ code get so bad? Why, previous employees at the company made bad decisions. Surely C++ didn’t get worse. Surely the C++ ecosystem and community didn’t regress, right? Didn’t everyone once use Ruby?
But I think that’s exactly what happens. The quality of a given language, or framework, or tool, or ecosystem, or community, follows an arc. It starts off with intense innovation and high-quality code. Usually the people contributing are doing so our of a sense of duty, to make something better than the language/tool/framework/ecosystem that they are currently stuck using. Then, the software becomes popular. People start using it. Features become set in stone. Innovation suffers. The wild west “make something better” days are over. You now have responsibilities to your users. If you break your software, someone will complain. And you’ll get to fix the bug. It’s not fun anymore. It’s work. And this is the point where the original devs quietly leave, on to find some new fun thing to work on. And in come devs who are looking to make a name for themselves, by contributing to open-source. Because for better or worse, contributions to open-source are a good way to improve your resume. At some point, the project grows so large, corporations are built around it. Someone starts a company selling training around the software. It’s now a business. Contributions to the open-source project by large come from employees of that company. It’s become subsumed into the corporate stack. Open-source contributions coming in through GitHub aren’t accepted unless an employee of the company happens to look there in a moment of boredom.
While all of this is happening, quality goes sharply up at first, then sharply down. The software project is initially chaotic, then for awhile it’s a joy to use, then it becomes over-burdened with features and TODOs, and falls into disarray. The fact that it’s grown a commercial entity, or has reached some level of fame, isn’t relevant really.
Just imagine you’re in the middle of a sea of hungry wolves. The trendier part of the software community is holding a torch that keeps the wolves away. If you go too far ahead of the trend, you’ll be eaten. Fall too far behind, you’ll be eaten. You (and your business) need to keep pace with the trends, no matter how silly it seems.
Show of hands – who’s still using Ruby in 2021? *raises hand* Although, it’s debatable that any of us are using Ruby in 2021. We’re certainly not using the Ruby that people were using from 2010-2016, the Ruby golden era. It kinda feels like Ruby is a failing empire in it’s last days. Nothing really works right, and the only people still living here are tied down by obligations such as technical debt.
Take RVM. You know RVM, right? In 2013 it was the thing. If you weren’t using RVM (or it’s perplexingly nearly identical yet stubbornly also extant clone rbenv) you weren’t a real Ruby programmer. Probably. Or something. But it doesn’t matter, because you don’t use RVM anymore. You run ruby entirely within a docker container, because your Gemfile long ago grew to the length of several novels, and you can’t actually keep them all working across MacOS upgrades. “I hear MacOS 12 breaks the andand2 gem” you overhear. You wonder if you’re using that gem. You probably are, by virtue of it existing, and one of the many waves of Ruby programmers, with an average of 1.5 years tenure at any given company, probably dropped it in there for like, one unit test.
But getting back to RVM. The website is still there, and it’s still the best way to install multiple Ruby versions and manage dependencies. This is something I always laughed at PHP and Node and Python for not having. And NVM and PyEnv just felt… lacking. Well joke’s on me, now RVM is also lacking in similar ways. For instance, you search for “rvm install” on Google, which takes you directly to this page: RVM: Ruby Version Manager – Installing RVM …where you find a gpg command… which doesn’t actually work because “pool.sks-keyservers.net” has gone dark sometime in the last few years.
Then there’s system libraries. Ruby always over-relied on system libraries. Libxml, openssl, etc. There isn’t even an automatic fallback if you’re missing one. RVM just fails to install ruby. MacOS, following the decade-old Apple tradition of “MacOS? Never heard of it. Oh yeah, I think we still have a few employees working on that” breaks openssl pretty much every release, every year, on the dot. So RVM and/or the Ruby maintainers need to know about what poking and prodding they need to do to make Ruby work on the latest MacOS. Back when Ruby was flush with developers, who were all running Ruby on their machines, that was trivial. Someone would file a bug report (or hundreds of people would file bug reports), and dozens of people would submit patches, and one would be approved, and good game. Congrats, your rickety, bleeding-edge, syslib-dependent language lives for another year. Yay!
Of course, as I mentioned, no one runs Ruby locally anymore. Apps are simply too complex to run outside of docker. I personally believe that docker has increased development friction and enabled a whole new class of dependency bloat, but that’s beside the discussion at the moment. Another factor is the rise of Single-Page App frameworks. When React hit the scene, it was just a matter of time before people switched from whatever ugly unmaintainable ERB or HAML (ew) templates they were using, and switched to glorious JSX. That, and the general superiority of SPAs for development experience and user experience, led people to pretty much relegate Ruby to the deep backend, only serving CRUDdy APIs and maybe checking auth. Maybe.
And then there’s Rails. As SPA frameworks like Angular started to gain mindshare and momentum, Rails reacted pretty badly, by hastily stuffing in features like Turbolinks, and reengineering Sprockets several times to try to get in front of the ball. It was all in vain, of course. Turns out developers really just wanted to minify a separate Angular or React app and serve it at the webserver level, even in development. But Rails kept going, trying to value-add with things like Action Mailbox and Active Job and ActionCable. Of course they had to re-architect Rails and change key APIs and config settings, meaning that upgrading to Rails 4 or 5 or 6 is still a headache, or bordering on a quarter-long OKR, depending on who you are. But you have to upgrade, because old major versions get summarily EOL’d after awhile, you know? And does anyone use this stuff? Unlikely. Everyone knows monoliths are bad.
In comparison to PHP, another language which had it’s day and lost it, Ruby feels like a true ghost town. The villagers left all at once when the gold mine ran dry. PHP has been mining steadily for that same decade, managing to maintain a sense of order and a slowly-improving developer experience. Ruby feels like everyone dropped their pickaxes and left overnight. As someone who keeps tripping over digital pickaxes like RVM, I can’t help but feel nostalgic.
Linux is almost 100% of the market share on servers, and almost 100% of mobile devices (besides Apple). So why is it stuck at 10% on desktop? (Or 20% if you count ChromeOS, or 1% if you only count people’s “work PCs”, such as in corporate offices and work-from-home setups). It’s because Linux has conflicting goals.
Some percentage of Linux users want maximum flexibility to control every aspect of their desktop. I’ll call them “Control Maximalists”. That open-endedness and modularity comes at a cost. With all of that flexibility, it’s hard to hide all of it from the user. But they don’t care, they love it. Linux config is love, Linux config is life. But, there is another type of user. Users who just want to “get stuff done”. These are usually professional users who heard that Linux is free and easy (“so easy my grandmother uses it” as every Linux evangelist claims). They’re fed up with Windows or MacOS (often for perceived reasons that don’t match reality), and they want a change. I’ll call them “Productivity Maximalists”.
Unfortunately, it’s difficult for Linux to appeal to both types of users. You can see distros bisecting along these two lines. Arch is for Config Maximalists. Ubuntu is for Productivity Maximalists.
But of course, there are a thousand other types of users. There are Freedom Maximalists, who are one step away from switching to GNU/Hurd at all times. There are Gaming Maximalists, who just want better gaming performance, which Windows denied them. There are Microsoft Minimalists, who at the very least, want to avoid all Microsoft products in their lives, and at the worst, actually hold a grudge against Microsoft for some perceived wrongdoing and want to destroy it by promoting Linux. There are Aesthetic Maximalists who only like Linux because you can make your desktop look like beautiful (for whatever your definition of beautiful is). There are Corporate Managers who just want to supply their employees with computers (that are hopefully easy to use), but maybe they can’t afford a Microsoft license. There are Grandma Maximalists who think that Linux is “easy to use” as long as “their grandma can use it”. There are austere Linux Monks who see using Linux in the most difficult way possible as good for the soul.
How can you appeal to all of these types of users? It’s a challenge. It would be easier if Linux were less modular. Linux right now is like if every car in the world used the same size engine compartment. Monster truck? Prius? Same size engine slot. The monster truck’s engine hangs out of the top of the engine bay. The Prius’ engine bay is mostly empty. Yet, that’s where we are. Everyone uses the same pluggable components. Everyone just knows that to make a Linux distro you take (1) the kernel, (2) add an “init system“, (3) add a package manager, (4) add Wayland or X, (5) and then add a Desktop Environment. All init systems are pluggable with Wayland or X. All package managers work with all init systems. All desktop environments work with all kernels. I believe that maintaining that compatibility is a drain on the resources of developers, and constrains them from doing what they need to do. If you write a desktop environment, it need to work with all init systems, and all kernels.
Why was switching systemd such a hotly contested topic? If Microsoft switched their init system from FooBarWinA.DLL to FooBarWinB.DLL, would anyone notice? Who was angry about systemd, and why? Well, systemd made a lot of changes that annoyed server administrators. People who want maximum control over their servers. Should desktop users care that they’re using SysV init or Upstart or SystemD? No way. But that’s another point…
Desktop Linux is stuck living in the shadow of server Linux (and to some degree mobile Linux). Desktop is a second-class citizen. Linux decisions are made that will help the server market. The desktop market is mostly ignored. So why do server and desktop Linux use the same init systems? Why do we have the same driver model? Want to know why we were stuck with X Windows for so long? So server operators could pipe GUIs out of their server farm. Really. Why do we bother to share anything with the server market? In some ways, the server market is at odds with the desktop market. They make decisions that massively impact desktop Linux, and don’t ask for our input.
Vertical integration is good. Right now we have the opposite, where every project is landlocked between other projects. Let’s say you did want to make a desktop environment, but you find a bug in wayland. You talk to the wayland developers, and they say it isn’t a bug, it’s a feature. What can you do? You’re doomed. You wanted to add a user feature (IDK maybe you wanted to create a full-screen firewall popup like Windows), but wayland isn’t giving you enough power to do that. You need to fork wayland? And now you’re stuck pulling patches from upstream forever?
Or let’s say you want to switch Linux to a case-insensitive file system, like NTFS (because “Productivity Maximalists” don’t care that foobar.txt is different from Foobar.txt!). We can use NTFS, but it’s difficult because Linux can’t boot from a NTFS partition (because they stupidly implemented the NTFS driver in userland). So we need to boot from ext4, and keep user data in an NTFS partition. That’s an unnecessary partition, which complicates the install process. So let’s try using ext4’s new case-insensitive capabilities. Unfortunately, ext4’s case-insensitivity is on a per-directory basis, so to protect the user from case-sensitivity, you need to add a kernel filter to make sure that no usermode program (such as if the user accidentally installs Nautilus or Nemo) creates a directory that is case-sensitive. Oh also, you may need to hack GTK and QT, because they may assume that files on Linux are case-sensitive.
My point of this is, Linux is trying to be all things to all people, and people have tried so hard to make things modular, they lost the ability to make bold decisions and push for user experience (things like systemd and wayland are refreshing counterexamples, though).
A running theme in the Calvin & Hobbes comic is his Dad dragging him along on a family outing, or giving him chores, or some other drudgery, and saying it “builds character” [Building character | The Calvin and Hobbes Wiki | Fandom]. In many of the comics, the Dad has hopelessly bungled the endeavor, and “building character” is just an excuse. From the title, you can see where I’m going with this…
I’ve noticed that many of my interactions with Linux gurus are similar to Calvin and his Dad. I’m Calvin, in case it wasn’t obvious.
Many Linux users believe that Linux is the superior way, and while it may be more complex than the average user can handle, they just need to work harder, and rise to the level required to use Linux. This is something that they believe that everyone should do. Using Linux is the way. Using Linux is good for the soul[1]. Using Linux builds character.
It’s a smug, moralizing elitism that knows what’s best for you. If you disagree, then you just haven’t thought about it hard enough. The Linux gurus have done all of the research for you, they’ve tried all of he Operating Systems, and Linux was the clear winner. You don’t need to think about this, just use Linux, okay?
There are really two parts to this: First, you’re told that Linux is the superior choice, and that by using it, you’ll be a better programmer, or better computer user, or possibly a better person! Secondly, once you’re onboard, any failing with the OS you find is actually a problem with you that you must rid yourself of. Look, Linux isn’t perfect, but that part definitely is. So you must be doing something wrong.
What’s funny is when these things happen out of order, as they often do. User: “Why should I use Linux?” Guru: “Because it’s awesome.” User: “But I’m having trouble with XYZ.” Guru: “Okay well you need to try harder.” Narrator: “The user, needless to say, did not switch to Linux.”
Since I’ve been interested in this phenomenon/mentality for quite awhile, I’m able to pretty quickly bring up examples.
Here’s a quick tour of people saying “you didn’t try hard enough” to understand Linux:
“Of course, it is good for the soul to write your own firewall script, and you’ll be a better human being for doing it, so you shouldn’t necessarily take the easy way out.” [Making the move to Linux, finally… (linuxquestions.org)]
“You’ve used it for less than three months, didn’t even understand it well enough to get it working on your laptop, and from this you decided that the kernel that basically the whole planet relies on for everything sucks” [Wholesome_Linux comments on Linux sucks (reddit.com)]
“hard work takes blood sweat & tears. put the work in. try to be a good adopter. try to work forwards. if you stumble, get back up.” [Add Comment | Hacker News (ycombinator.com)]
And here are some posts that agree with me that this phenomenon exists:
And here is an old-school Linux user who admits that Linux might not be for everyone:
“Linux doesn’t suck. Linux is not an appliance OS. If you want a brainless, “I’m just going to use it, I don’t want to know anything about it” OS, you should be running Windows.” – Tony Lawrence [linux sucks, windows is better than linux (aplawrence.com)]
When the Linux kernel boots, it needs to be told what to run next. That program, the “init process”, a.k.a. “PID 1”, is the first process that will run, and is responsible for readying the system for all of the other processes that will be run. Normally this is System V Init or System D, a fully-featured system that does quite a lot.
But at the core, what does the init process need to do, to provide a “fully-operational” Linux system? It needs to do a few things, in order:
Mount /proc and /sys
Dig through /proc and /sys to see what devices exist
Mount /dev
Populate /dev with nodes (not files) that represent hardware devices.
(Maybe start some background processes (daemons)?)
(Maybe do some login stuff?)
Having a /dev directory is a core part of POSIX, that Operating System standard that Unix and Linux conform to. But if having /dev is core to POSIX, and the Linux kernel doesn’t handle it itself, is the kernel really POSIX?
Note also that /proc and /sys don’t exist until the init process mounts them. This is a single Kernel API call. So why doesn’t the kernel do it?
I’m sure the answer to this is “because modularity”, but does it really make sense for such basic functionality to be swapped out or replaced? It seems that originally, Linux distros (if such a thing existed, and people weren’t cobbling Linux installs together from scratch) just had a hard-coded /dev directory. /dev/tty0, /dev/tty1, etc. Or maybe people manually created them. Who knows. In about 1995, however, someone wrote devfs, a tool for auto-populating /dev. In 2000, it was included in the kernel itself. At some point later, it was replaced by devtmpfs, which seems functionally equivalent.
I’ve been restoring an old computer, using as many new parts as possible (dual SATA and PATA SSDs, rather than the PATA Hard Disk Drive that was in there, modern SATA DVD-RW drive, a modern power supply rather than the non-standard-sized one that came standard).
One problem I’ve run into is brittle plastic. A little-known fact (to anyone who doesn’t collect something made from plastic that’s over 20 years old), is that plastic becomes brittle with age. The level of brittleness differs with the plastic composition, with the worst being the Apple Power Macintosh 5500 series (Power Macintosh 5500 – List of Macintosh models by case type – Wikipedia), which crumbles if you look at it sideways. But anyway, all plastic will become somewhat brittle after 20+ years.
(Fun fact about the Power Macintosh 5500: The popular myth around that is to blame the brittle plastic on cost-saving measures by Michael Spindler, the CEO of Apple at around the time these models were in the design phase, who is commonly believed by retro Apple fans to have been less-than competent. I don’t really buy that story, and in fact I think that Apple was doomed without Steve Jobs’ bold strategies, and the intermediate CEOs only served to delay the inevitable).
So all plastic embrittles. This would be fine if all of our modern electronics wasn’t housed in snapped-together clamshell plastic designs which rely on bendable plastic plastic clips and flaps. What happens when you try to bend plastic that has become brittle? It doesn’t bend, it stays rigid, and snaps. Ugh.
If you buy an old computer for restoration, chances are you’ll hear the rattle of one or more plastic pieces, chipped off of who knows where, inside the case. Sometimes you just find a missing chunk with no matching plastic piece.
3d printing and superglue fixes most of this. Armed with a Vernier Caliper, and a 3D modeling program (I recommend Fusion 360. It’s not free, but it’s well worth it if you really want the full 3d printing experience), you can make just about anything. Well, unless your trying to restore a Power Macintosh with a case split cleanly in two. You really can’t 3D print an entire case, or it’ll at least cost you hundreds of dollars.
Here’s an example of a broken plastic clip:
No big deal, I can 3d print the necessary wedge shape, and a stem to go into the remaining plastic post:
Another issue is matching modern hardware to old case pieces. For example, this computer case has these nice faceplates for the CD/DVD drives, but they only have plastic clips that match a specific model of old CD/DVD drives. What to do? 3d print my own adapter bracket:
In the future, I’d like to try 3d printing an entire computer case. Prices will have to come down drastically before I can, but I think that’s bound to happen soon. It’ll probably be the Macintosh Color Classic, but modified to look better with a LCD instead of a CRT:
When the Linux kernel finishes booting, it needs to know what to run next. What process will be the “manager” of all of the other “userland” processes? This process, called the “init process” or “PID 1” is commonly part of a larger “init system”. There are several. Mostly only SysV Init and systemd are worth mentioning, though.
The first order of business is to do something that I can’t believe the kernel doesn’t do for you: Mounting /proc, /sys, and /dev. (The kernel does most of the work, in fact, you just get to push the button that says “go”).
Here’s a minimal example of an init process which mounts these three directories/pseudo-directories:
#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mount.h>
#include <unistd.h>
void mount_proc();
void mount_sys();
void mount_dev();
void run_terminal();
void mount_proc() {
printf("Mounting /proc...");
if (0 != mount("none", "/proc", "proc", 0, "")) {
printf("\nFailed to mount /proc!\n");
exit(1);
}
printf("Done!\n");
}
void mount_sys() {
printf("Mounting /sys...");
if (0 != mount("none", "/sys", "sysfs", 0, "")) {
printf("\nFailed to mount /proc!\n");
exit(1);
}
printf("Done!\n");
}
void mount_dev() {
printf("Mounting /dev...");
if (0 != mount("none", "/dev", "devtmpfs", 0, "")) {
int errno2 = errno;
// Error code 16 ("device or resource busy") is spurrious and doesn't really matter. Ignore it.
if (errno2 != 16) {
printf("\nFailed to mount /dev! errno=%d strerror=%s\n", errno, strerror(errno));
exit(1);
}
}
printf("Done!\n");
}
void run_terminal() {
execl("busybox-x86_64", "busybox-x86_64", "ash", NULL);
}
int main(int argc, char *argv[]) {
printf("Hello from the init system!\n");
mount_proc();
mount_sys();
mount_dev();
run_terminal();
return 0;
}
To compile it, be sure to use -static so gcc won’t try to use any shared libraries:
gcc -static main.c -o ./boot_disk/init
You can put this file on a drive image and load it into QEMU, and specify it as the init= part of the Linux kernel parameters (in GRUB2 or directly through QEMU). It’s also handy to have busybox on the drive, and I left an example of how to invoke it in the code above. That way you can drop into the busybox shell and see the mounted filesystem:
