I created `echoChain` to explore the following phenomenon: how much data retained after LLMs exchange responses multiple times?
Imagine the game of telephone but with AIs. Each model converts the response into a question, which is then answered. After several rounds, facts get mutated, details get stripped, and the answer is far removed from the initial one.
`echoChain` is an OSS Python CLI that helps measure this process. It allows for multi-model testing, model rotation, tracking of factual claims, semantic similarity calculations, fidelity scores, reproducibility, statical tests, and visualizations. You can test models against each other, try various configurations, and learn which ones retain the most information. Right now, it only support OpenRouter; so, for this reason, please, make sure you set `OPENROUTER_API_KEY` before running it.
I wonder about the implications of these results for multi-agent systems, where AI-created information flows between agents without any human validation.
Looking forward to hearing your feedback on LLM agent building and evaluation frameworks!