I saw the comment about it being hard to use the esp32 without the IDF and was about to disagree, but after one second's thought, the author is right. Access to all their chips is via some quirky lower level api. That said, considering how different the chips are, it's actually a surprise they even manage to give you a common interface to them all. There was an OS free boot for the esp8266.

It's probably possible if you forego the wifi and there was a basic boot hello world a while back. The arm is not natively wifi so it's not really comparing apples to apples.