Quick update here since a lot of folks asked about what slopcodebench is / where it fits in

Slop code bench is unique among benchmarks in that it makes a model work against a progressively disclosed list of features (checkpoints) that it doesn’t know it advance, inheriting the same codebase through the lifecycle of a single multi-checkpoint challenge

I like this because it much more mirrors how real software dev happens, rather than most benchmarks with one off tasks where slop code doesn’t hinder future progress

The original post[1] and the SOL/fable follow up[2] went into a bit more detail on the benchmark and its challenges

1. https://news.ycombinator.com/item?id=49076391

2. https://github.com/humanlayer/advanced-context-engineering-f...