> The purpose of this workflow is to encourage the agent to identify and test small, incremental improvements. Claude would otherwise attempt to fix everything at once, only to be confounded by the drop in match rate, which had too many possible causes to diagnose effectively. Similarly, each attempt is made in a new file so that we can easily recover good matches. We want to avoid situations where an agent achieves a good match but subsequently edits the file and degrades its quality. The trail of attempts is also a useful reference for the agent.
This is a great tip. Using llms in the ide, I stage changes frequently to be able to rewind to a good state due to this “doing well then going off the rails” behaviour.
sequoia•3mo ago
> The purpose of this workflow is to encourage the agent to identify and test small, incremental improvements. Claude would otherwise attempt to fix everything at once, only to be confounded by the drop in match rate, which had too many possible causes to diagnose effectively. Similarly, each attempt is made in a new file so that we can easily recover good matches. We want to avoid situations where an agent achieves a good match but subsequently edits the file and degrades its quality. The trail of attempts is also a useful reference for the agent.
This is a great tip. Using llms in the ide, I stage changes frequently to be able to rewind to a good state due to this “doing well then going off the rails” behaviour.