I'm sharing a summary of my recent research on a method for controlling large language models (LLMs) called Evo-Recursive Constraint Prompting (ERCP). We achieved a 20% absolute accuracy gain on the PIQA commonsense reasoning task. This approach goes beyond simple prompting; it involves a neuro-symbolic optimization loop designed to enforce logical consistency.
*Key Results on PIQA:*
- *Baseline Accuracy:* 70.0%
- *ERCP Final Accuracy:* 90.0%
- *Absolute Gain:* 20.0% (a 28.6% relative boost)
- *Efficiency:* Achieved in an average of 3.9 iterations.
*Methodology: Self-Correcting Logic*
The core novelty of our approach lies in the use of external symbolic tools to oversee the LLM's neural output:
1. *Diagnosis:* Our system employs a DeBERTa NLI Oracle to autonomously identify logical contradictions and ambiguities within the LLM's reasoning chain.
2. *Constraint Generation:* These detected errors are immediately translated into formal, actionable constraints (the symbolic step).
3. *Refinement:* The LLM is re-prompted to solve the task, explicitly conditioned on these new constraints (the neuro step).
ERCP systematically transforms reasoning errors into performance gains by enabling the model to self-correct based on verifiable logical rules.
*The Real Research Challenge: The Convergence Problem*
While a 90% accuracy rate is strong, our results showed that only 30% of runs fully converged to a high-quality constraint set (Score > 0.8).
This indicates that 70% of the successful results were achieved with suboptimal constraint guidance. The next frontier is refining our optimizer to ensure constraint quality and guarantee convergence across all runs.
The whitepaper detailing the full protocol is linked in the submission. I look forward to hearing your thoughts on building truly robust, self-correcting LLM systems with this level of precision.
hemanm•2mo ago
*Key Results on PIQA:* - *Baseline Accuracy:* 70.0% - *ERCP Final Accuracy:* 90.0% - *Absolute Gain:* 20.0% (a 28.6% relative boost) - *Efficiency:* Achieved in an average of 3.9 iterations.
*Methodology: Self-Correcting Logic* The core novelty of our approach lies in the use of external symbolic tools to oversee the LLM's neural output:
1. *Diagnosis:* Our system employs a DeBERTa NLI Oracle to autonomously identify logical contradictions and ambiguities within the LLM's reasoning chain. 2. *Constraint Generation:* These detected errors are immediately translated into formal, actionable constraints (the symbolic step). 3. *Refinement:* The LLM is re-prompted to solve the task, explicitly conditioned on these new constraints (the neuro step).
ERCP systematically transforms reasoning errors into performance gains by enabling the model to self-correct based on verifiable logical rules.
*The Real Research Challenge: The Convergence Problem* While a 90% accuracy rate is strong, our results showed that only 30% of runs fully converged to a high-quality constraint set (Score > 0.8).
- *Initial Constraint Score:* 0.198 - *Final Constraint Score:* 0.377
This indicates that 70% of the successful results were achieved with suboptimal constraint guidance. The next frontier is refining our optimizer to ensure constraint quality and guarantee convergence across all runs.
The whitepaper detailing the full protocol is linked in the submission. I look forward to hearing your thoughts on building truly robust, self-correcting LLM systems with this level of precision.