140 simulated guests: my load test was measuring my load test
I simulated 140 guests for a party of 45. The database didn't flinch, one number scared me for nothing, and the real wall was somewhere else: thirty sign-ins per hour on shared Wi-Fi.
A party app has one particular flaw: you can't test it for real before the big day.
You can run it alone in your living room. With three friends by a pool. But forty-five people voting at the same time only happens once — and that day, if it crashes, there's no second take.
So in late June, seven weeks before my birthday, I wrote a script that plays the guests. A hundred and forty at once. For a party of forty-five.
Why a hundred and forty? Safety margin. If it holds at three times the load, I sleep well.
The good news: the database doesn't care
Orfeo runs on Supabase, on the free plan. Each guest signs in anonymously, suggests songs, votes, reports their mood.
First result, at 140 concurrent guests:
| Operation | Count | Median | 95% under |
|---|---|---|---|
| Vote | 6,740 | 57 ms | 76 ms |
| Report mood | 1,291 | 57 ms | 79 ms |
| Suggest a song | 420 | 185 ms | 1,023 ms |
Zero write errors. And the most telling part: between 33 and 140 guests, a vote goes from 73 to 76 milliseconds. In other words, nothing.
The database wasn't the problem. I would have bet the opposite.
The number that scares you
The problem was real time.
When someone votes, every other phone has to see it. That's a broadcast: one event, sent to every subscriber. And there:
| Subscribers | Median delay |
|---|---|
| 15 | 278 ms |
| 33 | 299 ms |
| 140 | 2,670 ms |
Almost three seconds. And worse: at 140, only 13% of events arrived. The rest got lost on the way.
Three seconds of lag and nine messages out of ten lost, for an app whose whole point is watching the queue move live. I started looking at paid plans.
The test was measuring the test
Before getting my card out, I reread my script.
To go fast, I had subscribed every fake guest to the suggestions table. The one that changes all the time: every suggestion, every vote. A hundred and forty subscribers on the chattiest stream in the system.
Except the real app doesn't do that. A real guest is subscribed to the play queue, which changes about once per track. Not the same throughput at all.
I reran the test on the right channel:
| Model | Subscribers | Median | Delivery |
|---|---|---|---|
| My script (everyone listens to everything) | 140 | 2,670 ms | ~13% |
| The real app | 100 | 271 ms | 100% |
Ten times faster, and no loss at all.
The 2.7-second figure was true. It just measured my script, not my app. I had built a problem, measured it carefully, and nearly paid to solve it.
The real wall, found by accident
Meanwhile, another issue was ruining my tests: I couldn't connect more than thirty fake guests.
At first I thought it was a bug in my script. It was a platform limit: thirty anonymous sign-ins per hour, per IP address. My script ran on a single machine, so a single address. Stuck at thirty.
Annoying for a test. But look at what it means for a real party.
In a venue, everyone connects to the same Wi-Fi. Seen from the outside, all the guests share one IP address. So thirty people can get in within the hour, and the thirty-first gets an error. No clear message, nothing.
None of my performance numbers would have shown that. I found it because my test was badly built — a single machine — and that flaw looked exactly like a function room.
I raised the limit before the party. On the day, guests were on 5G anyway, each with their own address, and everything went through without a hitch.
What the test didn't see
Let's be honest about the limits.
My script simulates guests, not the application. It sends votes; it doesn't run the code that executes on each phone when a notification arrives. And that's precisely where a real flaw was hiding: on every update, every phone reloaded everything. Multiplied by the number of phones.
I only saw that in August, during rehearsal week. That's the subject of the next article.
What I take from it
Three things.
Measure before you assume. I was sure the database would be the weak point. It takes 6,740 votes without a single error, on a free plan.
Be wary of your own test bench. A load test is a program, with its shortcuts. When a number scares you, the first question isn't "how much does it cost to fix" but "what exactly did I measure".
And the limits that take you down aren't in the performance charts. They're administrative rules: a quota per IP address, a cap on connections. They don't slow anything down. They say no.
Orfeo ran on the free plan. Forty-five guests, fifteen hours, not a euro of infrastructure.
— Pato