Writing an operating system kernel from scratch

https://popovicu.com/posts/writing-an-operating-system-kernel-from-scratch/

350•Bogdanp•4mo ago

Comments

lordleft•4mo ago

I will never not find this kind of project incredibly impressive. It’s interesting to think that Linux, after all, is really just the kernel — and yet getting that work done paved the way to getting an open source version of Unix installed on billions of machines. Great stuff!

hollerith•4mo ago

It is equally valid to say that Stallman's starting to write a C compiler and Unix utilities (in 1984 whereas the Linux project started in late 1991) paved the way to getting an open source version of Unix installed on billions of machines.

kimixa•4mo ago

I agree - there's a number of kernels that were "open source" and released at a similar time enough time to linux (e.g. 386BSD in '92) that I could see any of those winning the "community battle" and taking that space instead, but no real credible "development toolchain" equivalent until decades later.

Though I'm unsure how differing licenses might have affected this - I suspect that really early in it's development the "copyleft" nature of the GPL Linux didn't make as much of a difference, as from what I remember most commercial uses of Linux didn't come until it had already gained significant momentum.

treyd•4mo ago

The copyleft nature was essential to good driver support. It set it up such that for corporations making drivers the easiest path was to get the driver upstreamed. There was a bunch of hoops they could have gone through to avoid that (as many did, like Nvidia) but that became a sorta-default.

Copyleft encourages a collaborative relationship between entities because it makes trying to play it close to the chest with IP involve more legal effort (if it's possible at all).

kimixa•4mo ago

Yes, I can see that stalling development as (at best) it turns into a pile of private forks rather than a cohesive project, but from what I remember that was already after Linux had "won" the "Open Source Kernel" race.

Commercial support for Linux was... Sparse... before the early 2000s.

Quekid5•4mo ago

> [Stallman/GNU] getting an open source version of Unix installed on billions of machines.

Agreed, funnily enough GNU tools/compilers also ended up getting installed on a lot of proprietary UNIXes because proprietary UNIX was mostly shit (in user space!). At least most of the ones I had the misfortune to have to work on.

hollerith•4mo ago

I first came across GNU tools on NeXTSTEP, which wasn't too bad.

If Stallman had started with a kernel, there would be very few people who had the legal right to run any utilities or apps on the new kernel whereas GNU's utilities and apps (e.g., Emacs) were immediately useful (i.e., without breaking any copyright law or violating any software license) to a large population, namely, anyone with an account on a proprietary Unix system, which explains why Stallman chose to start with the userland.

amszmidt•4mo ago

    > If Stallman had started with a kernel, there would be very few people who had the legal right to run any utilities or apps on the new kernel

That is really not true, one of the most important things when it comes to the GNU project and the whole Free Software movement is the ability to run _any_ program, be it non-free software or free software. This has been parroted for more than 40 years now ...

hollerith•4mo ago

That's true in isolation, but is missing vital context. In the 1980s a user of proprietary Unix usually had no way to make the ls utility or a C compiler or a text editor run on an new Unix-like kernel. And even if he had the technical means to do it, he probably did not have the legal right, i.e., it would have been a violation of copyright law.

In the Unix world, there was no tradition of binary compatibility: you couldn't just take a binary compiled for NeXTSTEP and run it on Solaris or on Linux: you needed to recompile, which means you needed a C compiler and the source code for the program you want to run, and most users of Unix in the 1980s didn't have a practical way to get the source code (and if he did have access to the source code, using to to port the program to a new kernel was probably a copyright violation). Stallman could have tried to start a tradition of binary compatibility in the Unix world. That is one of the strategies he could have chosen for the GNU project: i.e., he could have picked a proprietary Unix, Solaris, say, and make sure the kernel of his GNU system could run all (or most) binaries that run on Solaris. But that strategy ran the risk that the owner of Solaris might have sued to stop this and the courts might have sided with owner, requiring the GNU project to shut down, pay the owner (Sun) a lot of money or start over with some other strategy.

hollerith•4mo ago

(Replying to myself because it is too late to edit.)

I found a shorter reply to your comment, which I will now write:

>when it comes to the GNU project and the whole Free Software movement is the ability to run _any_ program, be it non-free software or free software.

Just because the GNU project allowed non-free software to run on GNU doesn't mean there existed in the 1980s a userland that would run on GNU if GNU had consisted of just a kernel (or just a kernel and a C library and a C compiler).

diath•4mo ago

It's even more funny and amusing when you remember that in the initial release mail for Linux, Torvals said "just a hobby, won't be big and professional like gnu".

https://groups.google.com/g/comp.os.minix/c/dlNtH7RRrGA/m/Sw...

chuckadams•4mo ago

He's also referred to Linux as "the GNU Emacs of all terminal emulators".

Y_Y•4mo ago

> It's now the GNU Emacs of all terminal emulators.

> (Linus Torvalds, regarding the fact that Linux started off as a terminal emulator.)

http://neil.franklin.ch/Jokes_and_Fun/Linux_Quotes.html

That's the best reference I can find, but even if it's totally legit it doesn't make any sense to me.

throwaway82332•4mo ago

Perhaps it's a reference to how Emacs does "everything" and some has also joked that Emacs is an OS with a text editor

flykespice•4mo ago

Back then Emacs was considered what we call in modern software 'bloatware': resource hungry, big and full of feature creep.

BTW, in early versions of Slackware emacs was bundled in its own software category due to its huge size when selecting which packages to install on first boot.

chuckadams•4mo ago

Probably even had its own floppies too. Remember that install method?

hu3•4mo ago

Transcribed here for posterity:

> Hello everybody out there using minix - I'm doing a (free) operating system (just a hobby, won't be big and professional like gnu) for 386(486) AT clones. This has been brewing since april, and is starting to get ready. I'd like any feedback on things people like/dislike in minix, as my OS resembles it somewhat (same physical layout of the file-system (due to practical reasons) among other things).

> I've currently ported bash(1.08) and gcc(1.40), and things seem to work. This implies that I'll get something practical within a few months, and I'd like to know what features most people would want. Any suggestions are welcome, but I won't promise I'll implement them :-)

> Linus (torv...@kruuna.helsinki.fi)

> PS. Yes - it's free of any minix code, and it has a multi-threaded fs. It is NOT protable (uses 386 task switching etc), and it probably never will support anything other than AT-harddisks, as that's all I have :-(.

Someone•4mo ago

> I will never not find this kind of project incredibly impressive

I wouldn’t call it incredibly impressive. The path on how to write a minimal multi-tasking kernel has been beaten decades ago.

Writing a kernel that can boot and do a few things is ‘just’ a matter of being somewhat smart and have some perseverance. Doing it for RISC-V complicates things a bit compared to x86, but there, too, the information about initialising the hardware often is easily obtained (for example: https://wiki.osdev.org/RISC-V_Meaty_Skeleton_with_QEMU_virt_... I wouldn’t know whether this project used that)

I think the author agrees, given (FTA) that they wrote “This is a redo of an exercise I did for my undergraduate course in operating systems”

It’s work, may be nice work, but I think everybody with a degree in software engineering could do this. Yes, the OS likely will have bugs, certainly will have rough edges, but getting something that can multi-process, with processes shielded from each other by a MMU isn’t hard anymore.

gertop•4mo ago

> I think everybody with a degree in software engineering could do this

Ideally this would be true, but it hasn't been my experience at all. At least with American graduates, I can't speak to other countries.

bpye•4mo ago

My CS undergrad in the UK had us write an ARM kernel with scheduling and IPC, though didn’t require we use the MMU.

d_sem•4mo ago

Is there a book one can read to learn how to create one?

esafak•4mo ago

https://en.wikipedia.org/wiki/Operating_Systems:_Design_and_...

andrehacker•4mo ago

That is an excellent recommendation. For Operating Systems anything Andy Tanenbaum did is world class.

This made me look up what he has been up to, there is a 2023 edition of "Modern Operating Systems" which covers cloud virtualization and Android along with everything that came before, hm, tempting.

macintux•4mo ago

We really need a Tech Writer Hall of Fame. W. Richard Stevens, Andrew Tanenbaum, P.J. Plauger. Others?

ianmcgowan•4mo ago

Kernighan!

andrehacker•4mo ago

And how can we leave out the OG of tech writers: Donald Knuth. He got a bit distracted by developing TeX but he got a well deserved Turing award for the series.

yencabulator•4mo ago

I'd suggest https://os.phil-opp.com/ instead of a book from long ago. There is no reason to drag yourself through C anymore.

And if you do want the more historical content, https://www.projectoberon.net/

flamedoge•4mo ago

To you maybe. The subset of population that is even interested, smart, persevering to do this is extremely tiny.

ludicity•4mo ago

I am 99.99% sure that less than 20% of Australian graduates could do this, and honestly I wouldn't be surprised to hear that the actual answer is <1%.

I was studying at Monash, which is considered a solid university here, and holy moly are the standards low. I had classmates in the second year of my machine learning postgrad asking me things like "What is machine learning?", and they all graduated and found jobs anyway.

astrange•4mo ago

What did they find jobs in? Australia has like one tech company.

Which is odd since their universities have built two of the most interesting CS projects I can think of (Mercury and L4). And WWWJDIC I suppose.

bananaboy•4mo ago

Folks end up at all sorts of places. Like I mentioned above, the banks hoover up a lot of graduates. There are a lot of smaller local companies doing web stuff. The consulting companies all have a presence here (KPMG, Accenture, Fujitsu Consulting, etc).

awesome_dude•4mo ago

Atlassian Stripe, Zendesk,

Intermernet•4mo ago

Google and Amazon both have offices in Australia, and a bunch of banks / media companies / startups / SMEs employ machine learning specialists.

ludicity•4mo ago

Others have pointed out a few "real" tech companies, but the economy is also surprisingly large. There are a ton of little data science jobs available on the market, though they're mostly on initiatives that will never plausibly work.

bananaboy•4mo ago

I'd agree with this. I did a double degree in Comp Sci/Comp Sys Eng at RMIT (1998-2002) and even from that era I would say that's largely true. Out of the people who did my course (and those I knew from other degrees like Comp Sys Eng/Business) very few are still doing deep technical programming for a career and/or hobby programming on the side on deep technical non-web things. The rest are mostly working for places like consulting companies, banks, big data, Telstra, etc in management roles like project manager, scrum master, solutions architect, change management. A lot of folks I think were just not that interested in stuff like writing an OS, how does virtual memory work, how does the hardware work, etc so they gravitated out of those software development roles into management roles. Nothing wrong with that, but I just think not everyone is interested in or capable of writing an OS!

dysoco•4mo ago

Hmmm here at least in my uni in Argentina, we have an obligatory computer architecture class and we implement a protected-mode x86 (32 bits) kernel with interrupts, paging, etc. We obviously get some guidance but it's not that hard if you read the docs, nowadays you don't even need to go deep into the Intel manuals probably since osdev wiki has a lot of content.

Besides it's much easier nowadays, if something's wrong you can feed ChatGPT your GDT definition for example and find out if you misplaced a value, which used to be a PITA to debug.

As the parent comment says, I think the path to a booting usermode kernel has long been beaten, it's not trivial but it's not that hard either, I believe the impressive stuff is once you get out of "tutorial land" and implement a network stack, or a UI stack, etc.

WD-42•4mo ago

Zig really is amenable to OS development. And so is RISC-V. I started this same exercise except with x86 first. I quickly got annoyed with all the legacy boilerplate required. RISC-v doesn’t have any of that. It’s so much faster to get up and running. Here’s my bad zig: https://github.com/Fingel/aeros-v

toast0•4mo ago

> with x86 first. I quickly got annoyed with all the legacy boilerplate required.

IMHO, if you use a reasonable bootloader, you don't have too much boilerplate. Multiboot loaders do leave you in real mode, and most people want to be in protected mode, so you have to set up some tables and do a jump, but that's not that much boilerplate. There's a bit more stuff if you disable legacy interrupt controllers (which you probably want to) but it seems to me being able to maybe run on a regular pc is worth it (caveats about console interfaces apply... my hobby OS needs bios boot and uses some vga things that I found aren't well supported... serial console is easier, but lots of computers don't have a serial port either)

ethin•4mo ago

You still have to set up the GDT, TSS, all that other legacy stuff that isn't actually used but it's still there. And if you set it up incorrectly, your punished for it with triple faults and the like, even if nobody even uses anything like the GDT or TSS these days.

convolvatron•4mo ago

that is a pain. oswiki is a good resource, and there are also a huge number of examples. but it still takes a day. context save and restore and interrupts is another speed bump. but its no different than any other development banging your head against the wall exercise, and its pretty rewarding to have smoothed all that over and go to town on bespoke scheduler design.

saulpw•4mo ago

I dunno, I've done it myself, and I would say that it's qualitatively different than other "banging your head against the wall" software development. At least it's an order of magnitude more painful, due to lack of integrated tooling, questionable documentation, and a more delayed feedback loop.

Single-threaded kernel bringup is easier than most distributed systems though.

ethin•4mo ago

Funnily enough, when I was writing one in Zig for x86 I actually was the one who found a bug in (I think) 0.11/0.12 where packed structs broke when you assigned to their fields (and I also found that the `.interrupt` calling convention was a todo item in that version). GDT/TSS is always, always one of the most annoying parts of an x86 OS for me, because all of it is so tedious and unnecessary, and I wish Intel would just get rid of IA32 mode already, or add some kind of bit in CR0 or CR4 which caused all that extraneous setup of legacy structures to just cease to be. RISC-V is so, so much more pleasant to get started with. Limine helps regardless of which architecture your getting going with; it's pretty much my de facto bootloader at this point.

WD-42•4mo ago

Setting up the GDT is exactly the point where I got annoyed enough to see what risc-v was all about.

ethin•4mo ago

Yep, agreed. Don't get me wrong, the Intel/AMD docs are amazing. But even getting the hardware to a usable state (where I define usable as "something that can enumerate PCI devices on the system, even without PCIe") is an absolute pain that really just shouldn't exist. I know that RISC-V ahs it's own tough spots but... IMO it's a lot faster to get up and running with because it doesn't have 10000 execution modes with tons of different requirements you need to do for each mode, and on and on and on and on. (Yes, I'm exaggerating a bit.)

josephg•4mo ago

I’d like to have a go at this. What are you using to run your risc-v kernel? Just Qemu or can you recommend any real hardware?

WD-42•4mo ago

Just Qemu. I used this as a guide: https://operating-system-in-1000-lines.vercel.app/en/ It's C so translating to Zig is pretty fun.

pjmlp•4mo ago

Naturally, it is basically revisiting the safety of Object Pascal and Modula-2, and repacking it into a C syntax.

C was never special, beyond being widely available thanks to UNIX original licensing.

chris_wot•4mo ago

How does one get their hands on low-cost RISC hardware?

Rzor•4mo ago

There's this $10 board on Aliexpress called Milk-V Duo S. It's been popping up on my recs every now and then. Looks interesting.

https://aliexpress.com/w/wholesale-Milk%2525252dV-Duo-S.html

Taken directly from its description:

>Milk-V Duo S is an upgraded model of Duo with an upgraded SG2000 master with a larger 512MB RAM and extended IO capability

>Onboard WI-FI6/BT5(Model Milk-V-Duo-S-512M-Basic/Milk-V-Duo-S-512M-eMMC does not have this function)

>USB 2.0 HOST port

>100 Mbps Ethernet Portwith PoE Support (via PoE HAT)

>Dual MIPI CSI with ISP

>The device also supports switch between RISC-V and ARM boot via a switch

hackerInnen•4mo ago

I think there are dozens of boards by now, but what I found interesting and pledged to is this:

https://www.kickstarter.com/projects/starfive/visionfive-2-l...

I don't have a first version visionfive2 myself, but i heard good things about it and the eco-system sounds like it's growing. There are still rough edges. Lets hope they ship soon (tm)

Edit: What I currently have for my personal projects is a PolarFire SoC Discovery Kit. That's a quad-core RISC-V system with FPGA embedded. Maybe too expensive and not for everyone (130 bucks, fun fact: The dev board is cheaper than the chip itself).

https://www.microchip.com/en-us/development-tool/MPFS-DISCO-...

Also the microchip documentation, toolchain and so forth feel really crappy and old, but once you get used to it, it's not that bad and actually getting to run bare-metal risc-v code is easy. There are easy examples for linux and bare-metal

yencabulator•4mo ago

Maybe just run an emulator on your x86/apple, for now? It'll probably be faster than actual hardware, and it will for sure be faster to develop a kernel with.

https://www.qemu.org/docs/master/system/target-riscv.html

balder1991•4mo ago

This has risc-v: https://www.raspberrypi.com/products/raspberry-pi-pico-2/

chris_wot•4mo ago

Thank you all for your excellent responses. No thanks to the fools who downvoted this.

ktosobcy•4mo ago

> El acceso a la presente dirección IP ha sido bloqueado en cumplimiento de lo dispuesto en la Sentencia de 18 de diciembre de 2024, dictada por el Juzgado de lo Mercantil nº 6 de Barcelona en el marco del procedimiento ordinario (Materia mercantil art. 249.1.4)-1005/2024-H instado por la Liga Nacional de Fútbol Profesional y por Telefónica Audiovisual Digital, S.L.U. > https://www.laliga.com/noticias/nota-informativa-en-relacion...

Of FFS...

one thing is using http but another this one…

flykespice•4mo ago

...what? a football organization in Spain has the power to restrict internet access to the entire country?

WJW•4mo ago

Football is really big in Spain dontcha know?

fransje26•4mo ago

Gotta keep these billions flowing, and the shareholders happy..

carlesfe•4mo ago

Yes, that is exactly what is happening

lioeters•4mo ago

I heard about this situation in Spain where Cloudflare (and maybe others) are blocked by mistake due to being associated with some sites streaming football.

How are people getting around this problem, I guess VPN? I mean, some of the wrongly blocked IPs are needed for work I imagine.

fransje26•4mo ago

> initiated by the National Professional Football League and Telefónica Audiovisual Digital, S.L.U.

These people are criminals..

NaOH•4mo ago

Submitted separately by the author:

https://news.ycombinator.com/item?id=45236479

Copying their comment to here:

I have redone the classical exercise of writing a tiny OS kernel with time sharing, which manages a couple of user threads. My goal was to experiment specifically on RISC-V + OpenSBI. Additionally, I wanted to explore Zig a little bit, so that was the language used instead of the traditional C, but it should be straightforward how to do the same experiment in either C or Rust.

It's definitely very rough around the edges, and it's more of an experiment and an intro for people who want to go through step 0 of learning OS kernel development and computer architecture. Nevertheless, I hope it is still a fun experimental thing to play with over the weekend!

The full walkthrough and the GitHub link are available at the link posted!

kristoff_it•4mo ago

For those who want a "packaged" experience doing this same kind of work, there's "Operating System in 1000 Lines of Code" (https://operating-system-in-1000-lines.vercel.app/en/), I followed it a while ago in Zig (which meant translating C code snippets to Zig) and had a good time. I have the code and VODs here: https://github.com/kristoff-it/kristos/

fransje26•4mo ago

What would be the level of Zig knowledge needed to do this exercise?

Or, asked differently, with no knowledge of Zig, what would be a realistic approach to trying this out? (Assuming an interest in learning Zig, with a background in C++)

ozgrakkurt•4mo ago

I could easily understand stdlib code of zig after building some stuff for 2 days. Would be different for everyone but it is fairly easy

kristoff_it•4mo ago

It's definitely possible to hit the ground running you have some knowledge of systems programming. I would say that the Zig version is much easier to understand than the C version provided by the book because the Zig one uses `packed structs` instead of bit operations in some places, everything else is roughly the same.

Isamu•4mo ago

Always a fun and interesting thing to try.

Also go ahead and try hard things like your own cryptography. The advice “don’t roll your own crypto” just means don’t DEPLOY something that is not battle tested, because it is very easy to make BIG mistakes in that area. Always feel free to experiment and explore.

We need more operating systems, more choice.

jackdoe•4mo ago

nice!

i love how accessible riscv isa is, the docs are amazing, there are tons of examples, and tons of emulators, the uncompressed machine code is quite readable.

i am writing a book for my daughter and am also working on a small os with forth and timesharing https://punkx.org/projekt0/book/part1/os.html and i dont think i could've done it on x86, learning both forth and riscv assembly on the way, its super fun.

honestly if you ever wanted to make a toy os from scratch there is no better time to start than now (besides maybe 1980), get a cheap riscv like rp2350 and just upload the relevant sections of the docs to claude and it will help you if you get stuck.

balder1991•4mo ago

I just checked the prices for my country and I’m surprised how cheap this is. I’m 90% sure I’m buying one just to play around with.

jackdoe•4mo ago

and the datasheet is amazing https://datasheets.raspberrypi.com/rp2350/rp2350-datasheet.p...

there is an issue that language models have seen enormous amounts of rp2040 code and docs, and it they get properly confused a lot, so you should just pdf print relevant pages and use them in-context.

OpenCiv3: Open-source, cross-platform reimagining of Civilization III

The Waymo World Model

How we made geo joins 400× faster with H3 indexes

What Is Ruliology?

Unseen Footage of Atari Battlezone Arcade Cabinet Production

Show HN: Look Ma, No Linux: Shell, App Installer, Vi, Cc on ESP32-S3 / BreezyBox

Monty: A minimal, secure Python interpreter written in Rust for use by AI

Show HN: I spent 4 years building a UI design tool with only the features I use

Microsoft open-sources LiteBox, a security-focused library OS

Delimited Continuations vs. Lwt for Threads

Sheldon Brown's Bicycle Technical Info

Hackers (1995) Animated Experience

Was Benoit Mandelbrot a hedgehog or a fox?

Dark Alley Mathematics

Show HN: If you lose your memory, how to regain access to your computer?

PC Floppy Copy Protection: Vault Prolok

An Update on Heroku

How to effectively write quality code with AI

Why I Joined OpenAI

I spent 5 years in DevOps – Solutions engineering gave me what I was missing

Show HN: R3forth, a ColorForth-inspired language with a tiny VM

Zlob.h 100% POSIX and glibc compatible globbing lib that is faste and better

Female Asian Elephant Calf Born at the Smithsonian National Zoo

Introducing the Developer Knowledge API and MCP Server

Understanding Neural Network, Visually

I now assume that all ads on Apple news are scams

Learning from context is harder than we thought

FORTH? Really!?

Show HN: Smooth CLI – Token-efficient browser for AI agents

Show HN: ARM64 Android Dev Kit

OpenCiv3: Open-source, cross-platform reimagining of Civilization III

The Waymo World Model

How we made geo joins 400× faster with H3 indexes

What Is Ruliology?

Unseen Footage of Atari Battlezone Arcade Cabinet Production

Show HN: Look Ma, No Linux: Shell, App Installer, Vi, Cc on ESP32-S3 / BreezyBox

Monty: A minimal, secure Python interpreter written in Rust for use by AI

Show HN: I spent 4 years building a UI design tool with only the features I use

Microsoft open-sources LiteBox, a security-focused library OS

Delimited Continuations vs. Lwt for Threads

Sheldon Brown's Bicycle Technical Info

Hackers (1995) Animated Experience

Was Benoit Mandelbrot a hedgehog or a fox?

Dark Alley Mathematics

Show HN: If you lose your memory, how to regain access to your computer?

PC Floppy Copy Protection: Vault Prolok

An Update on Heroku

How to effectively write quality code with AI

Why I Joined OpenAI

I spent 5 years in DevOps – Solutions engineering gave me what I was missing

Show HN: R3forth, a ColorForth-inspired language with a tiny VM

Zlob.h 100% POSIX and glibc compatible globbing lib that is faste and better

Female Asian Elephant Calf Born at the Smithsonian National Zoo

Introducing the Developer Knowledge API and MCP Server

Understanding Neural Network, Visually

I now assume that all ads on Apple news are scams

Learning from context is harder than we thought

FORTH? Really!?

Show HN: Smooth CLI – Token-efficient browser for AI agents

Show HN: ARM64 Android Dev Kit

Writing an operating system kernel from scratch

Comments