The nature of data work is iterative: investigate, form hypotheses, experiment, and improve based on what you learn. This bottom-up process is the only way to deliver meaningful outcomes quickly in large, messy enterprise data environments. Consultants gathering context in meetings or teams defining a business ontology upfront to fit into the data has no chance of execution, in quarters and sometimes years. An AI-native system should automate this process, compounding learning to each new investigation and implementation.
For instance, one of our enterprise projects was moving a company off its old data estate onto a lakehouse with more than 30 source-system families, thousands of tables and transformations, and hundreds of BI reports. No individual or AI agent knew how that data estate was connected and operated. The only way to learn it was to investigate the company's repos, dbt models, documentation, and data itself and propose hypotheses about the estate, run experiments, and learn from them, until we understand how each system behaves and under which conditions.
The iterative work goes beyond understanding how data and systems fit together, and the business operations. Building useful software is iterative too. Whether you’re optimizing a warehouse, building a lakehouse pipeline or predicting churn, you need to run experiments and learn what works best in that environment. As you deliver more projects, that know-how compounds, similar in principle to RL. RAGing documentation has little value until it has been tried in a specific environment and the consequences are learned. As with human engineering work, experimentation and learning should happen in a sandbox.
We built Kapso for this kind of work, you give it an objective and success criteria, and it works towards an optimum solution that can be deployed on your stack.
You can view a video walkthrough here: https://www.youtube.com/watch?v=W4FHlza1fRs
Here’s how it works…
Kapso has a bank of lessons that are induced or deduced from the past experiments, and each carries a credence score, conditions on where it applies, and the evidence behind it. For each new experiment, an ideation module proposes ideas considering the earlier lessons, re-ranks them by expected return against the budget each would use, and sends the top-ranked ones to the implementor. Because the bank keeps learning from the past experiments in that environment, the proposed ideas are better suited to it than generic hypotheses from an LLM or RAG. Also, results of each experiment are verified by a verifier module that no experiment can access and cheat. As we’ve found when building such systems, sometimes the success criteria and evaluation tests need to be evolved. The experimentation modules can propose updates to the metrics and criteria for the verifier to approve or reject.
After each run, learn() updates the bank of lessons’ scores and boundaries using the evidence from that run. Changes to the bank can also be A/B tested by running otherwise identical builds with the previous and updated banks. The bank is built on git, so you can share your experience with others, list the banks available to you, and update them (`kapso experience share/list/update` in CLI). The system is model-agnostic and coding-agent-agnostic.
kapso.deploy(solution, target_stack) follows the same workflow, runs a shorter experimentation loop to adapt the best solution to the target stack (e.g. Docker, Modal, BentoCloud, LangGraph). Every domain has a harness to start from, you can write your own domain harness with /kapso:harness in the plugin or through CLI (kapso harness new <name>), and compound over that by running experimentations.
We ran it on some public benchmarks and live competitions, with the following results…
- Predictive modelling on relational databases: first on RelBench, Kumo AI (now Nvidia) and Stanford's benchmark, ahead of the best tabular foundation model (https://star-project.stanford.edu/relbench/leaderboard/). This shows that keeping the learning in the harness-level can beat foundational table models. Updating a bank of lessons costs far less than retraining a tabular foundation model.
- ML Engineering: First on OpenAI's MLE-bench at submission, still among the top open-source systems (https://github.com/openai/mle-bench/pull/107).
- AI competitions: ahead of the best agent the ALE-Bench authors at Sakana AI reported, and running autonomously at the live International Olympiad in AI 2026, above all 471 human contestants and in the top three AI systems (https://ioai-official.org/ai-models-track/ioai%c2%b2-2026-re...).
We have a public bank for ML and Data at Leeroopedia (https://leeroopedia.com/): around 100k pages extracted by experimenting on papers and repos, about 6M requests a month, about 28% from AI agents.
If you are at a growing company with a complex data estate across multiple systems and functions, and your data and AI team is stretched, we would love to help!
granthamctaylor•3h ago
Kapso was able to efficiently calibrate Relflow's hyperparameters on 21 structured data modeling tasks, and we were able to outperform TabPFN on the majority of the larger data modeling tasks despite not using any ICL, and using ~1% of the number of params of TabPFN.
alireza_msh•3h ago
Our hypothesis is that similar to RL principle, you can experiment in the environment (predictive modelling on relational databases), learn from it, and compound your domain harness with these bank of lessons, instead of re-training foundational table model, which puts these learnings in model weights. This enables faster iteration, and transparent learning over time.
As a proof for this domain, Kapso leads in classification, regression and recommendation tasks vs. best tabular foundational model (Kumo AI/Nvidia): https://star-project.stanford.edu/relbench/leaderboard/#clas...
alireza_msh•2h ago
Should we integrate experimental learnings by evolving the agent harness, or do RL-based post-training, and leave it in LLM's weights? Or Both?