That and the ability to hold on to knowledge.

I thought they turned sycophancy off...

It's for interpreting images, not generating them.

What exactly the strategy is?

What strategy is that?

I grew up in the 80s as a 100% blind child. Technology was by far not as advanced as today. Computers were just coming up when I was around 12. I learnt to type on a oldschool typewriter, and I also learnt to write braille with a pretty heavy full-metal embossing device. OCR was still quite bad. When I switched to what you call high scooll, I used a laptop with integrated Braille display to follow classes. Used good old DOS as OS and Word 5.5 as my "notepad". Except for PC Lingua for Latin, I basically had no tools specialized for learning. A electronic notepad and my brain was all I had to follow school. And I still made it. I have a great job I love, my own appartment, a sweet girlfriend and I am basically completely independent. To a point where I had to forcefully send away my mother since her continued attempts to "help" me were basically detrimental to my own development. I can not emphasis how important it is how you deal with it as a parent. Since parents are indeed the biggest hinderence to development, we have a saying around here amongst disabled people: "additional disability due to parental overprotection" (Zusatzbehinderung Eltern). Please take a moment to understand what this means, without feeling personally attacked. Its important. Your child can leave home around 18, just like every other kid. I did. Don't slow that process down artificially. The more this is prolonged, the harder it gets for the individual to actually obtain independence.

I am telling you this because I read between the lines that you believe current technology is a reason for you to be hopeful. Sure, it should be. But never forget, your child can do much more then you as a sighted person will ever be able to understand. Don't let them drown in your own misery. Let them discover what they can do. You will be surprised what they come up with. And dont fall for Gear Acquision Syndrome. Sure, tools are nice, and they do get better, which is also nice. I LOVE vision models, to stay on topic somehow. However, I still leave my house with only a cane and my phone in my pocket. I do occasionally ask Siri "Where am I" to get an address if I happen to have forgotten where I am exactly, currently. But at the end of the day, my cane is what shows me the way. Most tech is hype, plain old hearing and your sense of touch gets you much farther then you might think.

Wish you all the best for your own journey, and the development of your child.

You might be interested in https://github.com/OpenAdaptAI/OpenAdapt

I used molmo to parse screenshots in order to detect locations of UI elements. See the repo below. I think Omni parser from Microsoft would also work well.

https://github.com/logankeenan/george

https://github.com/microsoft/OmniParser

Do you have a write up of the tech stack and setup? Or willing to give the gist here?

I'd like to make a private Qwen or similar for my kids to prompt with a button and voice control. It doesn't need vision... Although eventually that'd be very cool.

Siri just sucks.

We might not be there yet...