That said, I do agree with the premise of the article that it's hard to learn "the stack", especially with the advent of generative AI.
"Back in the day", when google spat out a link to something resembling your problem, you still had to deconstruct the answer given to apply it to your particular case. That required some sort of understanding, or a mental model of the domain. With ChatGPT I find myself copying/pasting error messages between two browser windows, or a terminal, not really understanding what is going on.
No learning is happening. Does not bode well for new folks coming in.
It's not a promotional piece of something, it's my personal experience as a CEO and co-founder of a company using Xen as the core of our stack. I like to share my views in a transparent fashion on how it's hard to do very technical stuff, not just for technical reasons, but due to the lack of people trained for.
Sometimes, probably most of the time, it is better to work through it and understand an issue than to blindly copy pasta.
Recently got into a completely new language/framework and used Copilot to understand what was going on. I still made all the big decisions and wrote most of the code myself.
You can definitely use AI to avoid learning though (up to a point).
But hypervisors did not disappear, they just got replaced. When we run virtual machines, they are usually backed by KVM (low-level) and qemu (higher layer). Sometimes there is libvirt on top of it too, but running qemu directly is not that hard.
And there is plenty of exciting research about this stack, for example KVM can be driven by things like firecracker and crosvm, and there are some rust projects targeting it too. There is also BSD's bhyve which
My impression is that it's not that people find hypervisors in general are boring, but just Xen specifically (or maybe all classic Type-1 ones? hard to tell with Xen being the only open-source representative).
Sadly, the program I was supporting just had all of its funding yanked, I expect to get laid off tomorrow.
I respectfully disagree with much of your comment.
First, this wasn't intended as a promotional piece. It's a personal blog post where I share some of the challenges involved in building a full virtualization stack — a stack that happens to be fully open source. It's unfortunate that sharing real-world experience is sometimes immediately perceived as promotional.
Second, I think there's some confusion between using a hypervisor and mastering one — or building and maintaining an entire stack around it. KVM/QEMU is widely used, but it has significant issues, especially regarding security, performance, and latency consistency. Very few groups in the world are actively trying to tackle these challenges holistically (even major players like VMware have made some questionable shortcuts).
When it comes to low-latency, real-time use cases with a strong security model, Xen remains unique among open-source hypervisors. It's definitely not boring — in fact, it's one of the few that enable certain classes of critical applications at all.
We also work closely with academic research labs, and I can tell you: there’s still a lot of exciting work happening around Xen — even if it's less visible than buzz around newer projects like Firecracker or crosvm.
LLM are quite good at explaining systems and frameworks. I would never got into kernel programming without Deepseek guidance.
As for universities: too expensive, too much paperwork, too slow, too elitist.
However I agree: to learn a topic, LLMs are providing a great speedup. As a CEO/co-founder, I have no issue to hire people without a degree if they are good at what they do. However, our biggest chances are to scout directly in universities to find motivated students (motivation >>> everything else)
We’re talking about skills that span kernel-level programming, hardware quirks, low-level debugging, distributed systems, security, orchestration logic, even the capability to work with the UI/UX team... and the ability to explain all that without scaring interns. You can’t just hire for that. You have to grow it. Nurture it. Beg for it. Or in some cases, resurrect it.
If you are that person, what is the best way to market yourself? I am the person described. I've got experience from poking registers in firmware, to wireline transport protocol implementation, to infosec, to writing microservice framework middleware, to pipeline orchestration at the OS level, and on and on. In the last week I've debugged Linux UDS issues and TLS cipher suite problems, and wrote code to provision WiFi-connected devices over BLE.
But it's incredibly hard to demonstrate that in an interview, if I can even find a role that warrants it. You're not going to find me on a university campus or in a research lab because I'm at a FAANG trying to pay my mortgage.
Tldr raise your price and they'll belive easier.
If that has worked for you, that's amazing! But it seems really counter-intuitive to me.
I don't see "pay like Google" listed.
It goes on to say that it's hard to find and develop expertise for low level software like hypervisors.
What's the connection between the topics? It feels like two different rants.
If it's difficult to find kernel developers then wouldn't it help to not require them to also know web UX?
That means hiring two people, and in $current_year, companies expect one person to know everything. Sysadmin, backend programmer, frontend programmer, designer and a DBA used to be different people not that long ago, now they expect one person to do all that... + it seems they want kernel development experience now.
And if you find or train those low-level/system-oriented people, they also need to understand how a feature they build will be exposed functionally to a user (and why they need it in the first place). Because things are not make into thin-air but required to work in a bigger picture (ie: the product).
polishdude20•2h ago
somanyphotons•2h ago
betterThanTexas•1h ago
Maybe there's a good argument against training, but it could also just be irrational and stubborn in this case.
gbuk2013•1h ago
If you get the ratio wrong, with too many juniors and weak technical leadership then you will end up in a very bad place in your code base.
In terms of value, even if juniors are half the cost, it is much wiser to hire one senior instead of 2 juniors for the same money.
jandrewrogers•34m ago
Does the high skill standard for surgeons mean the market for surgeons is saturated?
n_ary•2h ago
It got worse in last two years. Now Senior Engineers must have exact combo of the weird tech stack and tools with N years of experience exactly as the existing employees, else gtfo, you don't even get screening calls. You don't have something from nice to have list? Lol, why are you wasting our recruiter's time even? She needs to use GenAI to write her next rejection email.
Also, your 15yoe does not matter, unless you are coming from direct competitor, in which case your 1.5yoe with internship is also excellent.
gbuk2013•2h ago
That said, in my previous job in a startup we hired very junior engineers and gave them plenty of opportunity and support for growth and several people did stunningly well. Pity the company didn’t do very well (IMHO due to focusing more or this sort of thing rather than making money).
The truth is bound to be somewhere in the middle.
htrp•1h ago
Mentoring is a key part of technical leadership, way easier to help the talent grow into your requirements (even if they are under-qualified, for now)
plam503711•1h ago
artyom•1h ago
I would love to work on low-level, systems stuff (anything as close to the hardware as possible), that's even my education and area of expertise. BUT SaaS companies in reality:
- pay better
- have lower costs
- are way easier
- don't have that many (or any at all) geographical restrictions (e.g. importing hardware prototypes).
People follows the money.
jandrewrogers•41m ago
The minimum level of sophistication required to be effective in systems software has increased dramatically since the 1990s, when I first started doing systems software. The kinds of systems software we were putting in production in back then would be considered a trivial toy today. This shift has placed an enormous amount of upward pressure on the minimum level of experience and skill that would allow someone to become productive within a useful amount of time.
It is no longer feasible in many cases to have companies effectively subsidize the development of highly skilled systems software people. The training time has become so long that companies will never recoup the investment. It is easy to see how the incentives have created the current situation and it is not clear how to address the shortage. Even before the current situation, it was widely noted that most systems software developers were self-taught over many years rather than trained up in a structured environment.