this is a postmortem. names have been redacted. the eval() has not.
the bug was elegant. the bug was charming, even. it sat in a 4-line helper that had been deployed to production for nineteen months and survived eight pairs of eyes. it was triggered, of course, by the most banal request shape imaginable: a string with a backtick.
the line of code that ruined a tuesday
function evalUserExpr(s) {
// for the admin debug panel, behind auth.
// surely fine.
return eval(s);
}
behind auth! we said. for power users! we said. only on the admin panel! we said. that admin panel was reachable, it turned out, by anyone with a session cookie and a curl one-liner because somebody (me) had wired the route through the wrong middleware chain in a hurry six months earlier.
the timeline
- T+0:00 — automated scanner trips. logs show eval invocations from a guest token.
- T+0:08 — i join the bridge. coffee. immediately spill the coffee.
- T+0:14 — confirm RCE. confirm scope. confirm i need a stronger coffee.
- T+0:31 — kill the route. revoke tokens. force-rotate the worker pool.
- T+1:42 — write the patch. write the test. write a 4-paragraph apology.
"the only safe eval is the one you never deployed. — somebody, possibly me, in retrospect"
what i changed
killed the eval. installed a sandboxed expr evaluator with a 50-symbol allowlist. added a lint rule that screams at the word 'eval' in our codebase. wrote a runbook so nobody has to invent the steps at 2am ever again. shipped a CTF for the whole team using the same bug. tuesday redeemed.
the embarrassing part isn't the bug. every codebase has one. the embarrassing part is how long i'd been lying to myself about that helper being 'fine'. fine is a feeling, not a property.