All the IBM's Visual Age line of IDEs were written in Smalltalk, and in a way it was the ".NET" of OS/2.

SOM (OS/2 COM) supported it natively, and one biggest difference to COM is that it supports meta-classes and proper inheritance, language agnostic.

What made Smalltalk lose industry mindshare was exactly Java.

When it came out, some major vendors, like IBM, pivoted all the way into Java, leaving Smalltalk behind.

It is no accident that Eclipse was designed by some of the GoF authors, and it is initially a rewrite of Visual Age underlying platform from Smalltalk to Java.

Eclipse even to this day has a Smalltalk like code browser.

It wasn't only the IDEs, some famous Java libraries, like JUnit, started their life as Smalltalk libraries.

Now as full OS, yes that never really took off.

Note not all Smalltalk vendors switched to Java, that is why Dolphin and Cincom Smalltalk are still around.

Cuis Smalltalk and related implementations are rather self-contained systems to the point they seemed walled off from the rest of the system, making it difficult to develop Smalltalk programs using external tools.

However, there's something compelling about the idea of a Smalltalk (or Lisp) OS running on bare hardware, where everything runs in a single address space. I've been thinking about this for a few years, but I haven't had time to pursue these ideas. Some ideas from the 1994 paper "Sharing and Protection in a Single-Address-Space Operating System" (https://homes.cs.washington.edu/~levy/opal.pdf) could be applicable to add some security to a Smalltalk OS.