Thought it would be fun to see how accurate AI was at analyzing this, given its volatility. I wanted to see if the graph would "converge" like the ones showing the chance of a sports team winning over the course of the game - they usually bounce around before trending one direction at the end of the game.
Every morning a small Lambda checks Google News (war/peace/ceasefire/Hormuz queries), gives the headlines to Claude Haiku, and gets back a probability (0-100%) plus ~500 words (Claude is so generous, need to constrain that) on what moved and why. It's a static site on S3 + CloudFront, and we keep a daily history of it
Anyway the graph was all over the place. So to help gauge its accuracy yesterday I added oil and gas prices overlaid. Turns out they correlate pretty closely *most* of the time. If a deal is reached I'll be interested to see if the graph converges up to that point (showing AI was somewhat smart at predicting), or if it just jumps to 100 in which case it was less... predictive.
Take a look, happy to hear feedback or suggestions.
archleaf•1h ago
So do you think it's making the prediction only on oil prices and not much else?
BubTheBuilder•49m ago
The prompt we use to get the news, from which we make the prediction, doesn't ask about oil or gas prices so the info AI has to make the estimation doesn't take that into account. The oil/gas prices are pulled from separate free APIs (Yahoo Finance) and overlaid, as kind of a sanity-check to see how good the analysis actually was.
BubTheBuilder•1h ago
Every morning a small Lambda checks Google News (war/peace/ceasefire/Hormuz queries), gives the headlines to Claude Haiku, and gets back a probability (0-100%) plus ~500 words (Claude is so generous, need to constrain that) on what moved and why. It's a static site on S3 + CloudFront, and we keep a daily history of it
Anyway the graph was all over the place. So to help gauge its accuracy yesterday I added oil and gas prices overlaid. Turns out they correlate pretty closely *most* of the time. If a deal is reached I'll be interested to see if the graph converges up to that point (showing AI was somewhat smart at predicting), or if it just jumps to 100 in which case it was less... predictive.
Take a look, happy to hear feedback or suggestions.