We created a multimodal virtual presence agent that uses advanced voice and vision capabilities to offer real-time, familiar-feeling conversations for people with Alzheimer’s. When a primary caregiver can’t be present, Relief provides comforting, supportive interactions—offering reassurance for loved ones and meaningful peace of mind for caregivers.
It was fun to hack this together, using the react agent sdk was quite easy, but lacks of fine-control, had to find workarounds for pausing the interaction. Another fun workaround was to give "vision" to the agent by taking screenshots every 5 seconds and using an LLM to analyze the image and feed that back to the agent's context.
The app was built to solve a real problem in Asia, I hope it inspires someone to create a legit solution.