So at Bucket (https://bucket.co) that’s the part we wanted to solve with AI, removing the flag code itself.
- How it works:
The cleanup bot operates as a GitHub integration with Bucket. When it runs, it searches for usage of the Bucket SDK in the codebase and identifies where specific feature keys are used. LLMs are employed to refactor the code to remove the flag and eliminate codepaths that become unreachable.
So all you should need to do is review and approve the PR. Once merged, the flag itself is archived in Bucket.
- The results:
We’ve been running this in production ourselves for a few months alongside a few early-access partners. We’re seeing consistently good PRs that can be merged without changes.
However, this wasn’t always the case. Early on in development, the range in quality of solutions generated by the LLM was an issue. Most attempts were fine, but then it would add a comment somewhere unrelated, remove whitespace or change the code formatting (even if you gave it rules to follow). Many of the early generated solutions failed CI due to unused imports, incorrect indentation, lone decoration not always getting cleaned up, or unused exports failing to be removed.
You can get a surprisingly long way with an LLM and a quick prompt, but getting good results, consistently, is where the real work is.
Some of the issues we were able to solve through better prompting and using different models, while others needed custom solutions. We wrote up how we went about building it here: https://bucket.co/blog/building-ai-flag-cleanup
- Try the beta:
We’re opening the beta up to everyone, but limiting it to React codebases for now.
If you’d like to try it out: - Sign up at https://bucket.co - Enable it: https://app.bucket.co/envs/current/settings/org-integrations
Let us know how you get on.