For non technical users they would have to find an existing plugin or maybe it doesn't exist so they are stuck.

But I can't help but think why wouldn't for example an application just have a prompt box that builds an extension dynamically on the fly (e.g. "make me a UI panel with some buttons that does X")

Obviously it would be "use at your own risk" and YMMV

Have any companies or applications actually done something like this?

I think modifying the real source code would get out of hand but perhaps just generating extensions that have access to a sand boxed set of safer APIs.