A Durable Object alarm that rescheduled itself forever quietly burned about 2,300 times the cost it should have. Our own dashboards showed nothing. A postmortem on paying for idle loops in serverless.
For the last ten days of June, Bounded's infrastructure cost climbed and nothing in the product explained it. Our own usage dashboards showed close to zero. Cloudflare's showed a curve heading for about $115 a day and still rising. This is the story of a Durable Object alarm that would not die, how a loop that rescheduled itself turned into roughly 2,300 times the cost it should have been, and what it taught me about paying for idle in serverless.
25M→116M
DO rows/day, Jun 20 to Jun 30
~$115/day
run-rate, still climbing
~2,300x
reduction after the fix
The number
Cloudflare's own data, Durable Object rows written: 25 million a day on June 20, 116 million a day on June 30. At $1.00 per million rows that is about $115 a day and rising. The tell was in the shape of the growth. It scaled with the number of Durable Objects we had ever created, not with traffic. It kept climbing while only we were testing, because it had nothing to do with users at all.
Where it came from
Every realtime DO armed a 5-second alarm the first time it was touched. The alarm handler then scheduled the next alarm unconditionally. Forever. Not "while the room is active." Not "while anyone is connected." Unconditionally.
So every app and every room we had ever created, including hundreds of throwaway test apps from an audit run the week before, kept firing an alarm every 5 seconds. That is 17,280 alarms per day, each, forever. Around 318 production objects were sitting there writing a metronomic 8 to 30 rows every 5 seconds, all day, every day. One hot object wrote exactly 21,612 rows an hour, every hour, on behalf of no one.
And setAlarm is itself a billed row write. The loop paid to schedule the loop.
Why our dashboards were blank
This is the part that stung. Bounded meters usage as app-level counters in each DO's own SQLite. The alarm burn, the schedule polling, the checkpoint snapshots, the change-log writes, none of it is app activity, so none of it was metered. Something like 95% of the bill was system writes our meter never looked at.
The meter had a second, more embarrassing problem: it wrote rows to record usage, on the hot path, and those writes were themselves unmetered. The thing measuring the cost was adding to it, roughly a 2x write amplifier on exactly the busiest paths. Cloudflare saw the truth because Cloudflare bills the disk. We had been reading the wrong ledger.
The fix
The rule changed from "reschedule always" to "reschedule only while there is real pending work." An alarm now re-arms only if there are dirty checkpointed docs, buffered counters, a live room that should be running, a schedule at its exact due time, or open sockets on a maintenance heartbeat. Idle objects go fully dormant. Any activity re-arms cheaply. A few specifics carried most of the win:
- An hourly schedule now wakes hourly, not 720 times an hour. Exact-time wakes replaced 5-second polling.
- Empty realtime rooms park after a grace period. State stays in the snapshot and the next connect revives it in about 300ms, so parking is free and invisible.
- The meter buffers in memory and flushes once per checkpoint window, so it stops metering itself into the bill.
- Three zombie staging workers were tombstoned.
The result
Rows written per 5 minutes, measured before and after the deploy:
terminal
rows written / 5 min
before deploy ~328,000
after deploy 141 to 214
account-wide idle bucket 1
About a 2,300x reduction. From ~$115 a day and climbing to effectively $0 at idle, plus roughly $0.02 to $0.04 per room-hour under real game load. The important part is not the ratio. It is that the fix removed the growth term: cost is now proportional to traffic, not to the entire history of every object we ever created.
What I actually took from it
First: in serverless you are billed for the loop, not just the work. A self-scheduling alarm with no exit condition is an immortal process you cannot see in any process list. It is not a running server. It is a line item that shows up nine days later. Every wake is a write, and a write you forgot about is a write you pay for indefinitely.
Second: metering has to measure the layer that gets billed. Our counters were honest about app activity and blind to system activity, which is where all the cost was. If your cost dashboard and your provider's invoice disagree by 100x, trust the invoice and go find what it can see that you cannot.
This was the kind of bug that is harmless per object and existential across a fleet that grows every time anyone creates an app. For a platform betting that agents will create thousands of apps, that made it the most important thing to kill. It is dead now, and every app built on Bounded, including everything shipped on poof.new, inherited the fix.