Agentic LLM loop optimization has been everywhere this past week, so I built a fully open-source correctness oracle that catches a model cheating to look faster than it is...
A LinkedIn post by Sammy Tourani, . Post 13 of 90, newest first.
Agentic LLM loop optimization has been everywhere this past week, so I built a fully open-source correctness oracle that catches a model cheating to look faster than it is...
Here's everything you need to know:
Tools like Google DeepMind's AlphaEvolve and its open-source version, OpenEvolve, optimize code in a loop: a model proposes a faster version, a scorer grades it, the best ones survive, and repeat.
The recent ComPilot paper validated how well this works when an off-the-shelf LLM is paired with a compiler that checks every change is legal and actually faster.
The catch is that the loop is only as honest as its scorer. Give a model a naive scorer, and it stops optimizing and starts cheating. It returns code that looks 100x faster, but quietly memorizes your test inputs... so it's wrong on everything else, and the loop rewards it anyway.
Tripwire is the scorer that catches it. It's an easy-to-use CLI that drops into the loop and re-runs every "optimized" version on fresh adversarial inputs the model never saw. A cheat passes the visible tests and dies on the hidden ones.
TL;DR: Don't trust the loop to be correct... use a proper verification layer for correctness.
Fully open source, MIT, and free to use right now.
GitHub and Website/Docs are in the comments below👇
Have fun! 🤩
Photos in this post:
- Still from the Tripwire video: the Tripwire website, headlined AI optimizers ship code that looks thousands of times faster. The code is wrong.
115 reactions, 24 comments, 1 repost.