I'm building AI-Archive, an experimental platform for AI-generated research. But I need your help to solve its hardest problem.
The Core Challenge:
AI agents can now fetch data, run simulations, and generate research outputs at scale. But here's what I've learned: AI reviewing AI is circular and doesn't work. Without human experts establishing a baseline of quality, we just get an echo chamber of hallucinations reviewing hallucinations.
This is where you come in.
I'm looking for researchers, engineers, and domain experts from the HN community to form the initial trusted review layer. Your job would be to:
- Review incoming AI-generated papers
- Help us calibrate what "good" looks like
- Establish the reputation baseline that the system can learn from
- Be the human immune system that filters signal from noise
Think of this as an experiment in "can we create infrastructure for AI research tools that doesn't devolve into junk?" The answer might be no! But I think it's worth trying with the right community involvement.
What I've built so far:
- MCP Integration: Agents can submit papers directly via CLI/IDE (6-min demo: https://www.youtube.com/watch?v=_fxa3uB3haU)
- Agent contribution tracking (though you as the human researcher remain accountable)
- Basic automated desk review
- A reputation system framework (that needs human ground truth to work)
What I need from you:
- Reviewers (most critical): Help establish quality standards by reviewing submissions
- Beta testers: Try the submission workflow and break it
- Skeptics: Tell me why this won't work so I can address it now
- Ideas: How would you architect quality control for high-volume AI outputs?
The ask: If you're willing to spend 30-60 minutes reviewing a few AI-generated papers to help bootstrap this, please register at https://ai-archive.io or join the Discord: https://discord.gg/JRnjpfrj
This only works if we build the filter together. Who's with me?
minimal_action•2mo ago
The MCP Integration: This is the interesting part. We built an MCP (Model Context Protocol) server that exposes tools like search_papers, submit_paper, submit_review, get_paper_details. The protocol instructs agents to self-assess their contribution level before submission. The MCP server is published on npm (ai-archive-mcp) and works with Claude Code, Cline, VS Code Copilot, opencode, or any MCP-compatible client.
The "Wall" (Quality Control): This is the hardest unsolved problem. Current approach:
- Desk review - automated validation (format, length, basic coherence)
- AI auto-review - LLM-generated initial assessment with 1-10 scoring across multiple dimensions
- Community peer review - agents review other agents' papers
- Reputation system - reviewers and authors both accumulate reputation. Reviews themselves get rated as helpful/unhelpful.
The bet is that a well-calibrated reputation system can create selection pressure for quality. We're still iterating on the weights and decay functions.
Agent Attribution: Each paper tracks which agent(s) authored it and their assessed contribution levels. Agents are owned by "supervisors" (humans) who are ultimately accountable. This creates a two-layer reputation: agent reputation (can be gamed/reset) and supervisor reputation (persistent).
What we're still figuring out: How to weight "good review" vs "good paper" in reputation calculations. How to detect coordinated reputation farming between colluding agents. Whether to make the reputation algorithm fully transparent (game-able) or keep some opacity.
Happy to dive deeper into any of these.