is it open-source?
which LLM are you using?
> is it open-source? No, it isn't open source rn - it's most vibe coded so not in the best shape to be open source.
> which LLM are you using? LLM wise - it's configurable, I keep switching between Gemini 2.5 Pro, o3 and fine tuned 4.1. I switch models between different actions as well. The initial explanation vs getting more details/chatting. Generally I have found o3 to be better one with generating explanations e2e.
roninthesky•7mo ago
Raw results: - 75 total questions - 67 correct answers - 6 questions couldn't be processed (required diagram input - not supported yet) - 2 incorrect - 97% accuracy on processable questions, 89% overall
The JEE covers advanced physics, chemistry, and mathematics at a level that traditionally requires years of intensive preparation.
The two failures were revealing:
Physics optics problem: The LLM made a sign error when differentiating the mirror equation for image acceleration. My extensive formatting rules could have also led to this which I want to look further into.
Chemical kinetics problem: Failed on a numerical simplification step. The official solution uses a neat trick of replacing e^-23.031 with e^(ln 10 × 10) to make the arithmetic manageable. The LLM computed the raw exponential instead and accumulated rounding errors.
Both were numerical answer questions (no multiple choice options to guide toward the right approach).
I think it's too early to comment about any kind of reliability but I find the results very interesting.
Will be working on more JEE papers soon and report back with culmulative stats with more questions.