Measurement · Poker · Reinforcement learning
Two rows of my poker benchmark never resolved, no matter how many hands I ran. I assumed I needed a better estimator. I needed a better instrument.
Poker win rates are measured in bb/100 — big blinds won per hundred hands. The number is easy to compute and almost meaningless without an error bar, because the variance is enormous: most hands are worth a couple of blinds and a rare few are worth a whole stack.
My benchmark ran six matchups over 2,000 hands each and reported bb/100 with a standard error. Two rows had been statistically unresolved for the project's entire history — the error bar was wider than the estimate, so the sign of the result wasn't even established:
The obvious diagnosis is "run more hands." Standard error falls as 1/√n, so 25× the hands buys a 5× tighter bar. That is a fine answer when your estimator is sound. Mine wasn't, and running more hands would have converged, slowly and expensively, on a number that was still wrong.
Poker's standard variance reduction is duplicate scoring, borrowed from duplicate bridge: deal one shuffled deck, play it twice with the two players swapping seats, and score the difference. Both players hold both hands, so the deal's luck cancels and what remains is the decisions. In bridge this buys 5–10×.
I implemented it. It bought 1.0×.
My first version had the two players in separate games against a common lineup — same cards, no swap. Their outcomes turned out to be nearly uncorrelated (ρ = +0.10), because how you play moves more chips than what you hold. So I rebuilt it properly, with both players at the same table and the seats swapped between runs.
That bought 1.1×.
At this point I stopped guessing and measured the correlation the technique depends on. A player's two mirrored hands correlate at only ρ = −0.131, which predicts a reduction of 1/√(1+ρ) = 1.07× — exactly what I was seeing. Then the reason showed up in the distribution:
85% of hands finish under 4bb, and the top 1% of hands carry 42% of the total variance.
Those big pots are the variance. And they only exist when both players choose to build one — so they don't mirror across the swap. Duplicate bridge works because every deal is played to completion at both tables. Duplicate hold'em doesn't, because a hand only becomes expensive if two people agree to make it expensive.
If the variance lives in all-in pots, target those directly. All-in EV is the standard tracker technique: when players get all in before the board is complete, replace the realized runout with its expectation over every possible completion. Exact enumeration where the combinatorics allow, Monte Carlo above that.
This one is unbiased and genuinely correct — over 4,000 hands the adjusted and realized means differ by 39 bb/100 against a 2 SE of 240. It bought between 1.00× and 1.23×, depending entirely on how often a matchup produces a pre-river all-in. In one matchup it bought exactly nothing, because the heuristic bot caps its bets at ⅔ pot and simply never gets there: 0% of hands were adjustable.
Two textbook techniques, correctly implemented, both worth almost nothing. That was the point at which I stopped looking at estimators and started looking at the harness.
Between hands, the benchmark tops up any player who busted:
# pokr/bench.py
def _rebuy(stacks, buy_in):
return [buy_in if s <= 0 else s for s in stacks]
It tops up losers. It never caps winners. So chips are injected into the table and never removed, and stack depth ratchets upward across a session. Measured with the project's own scripted bots — no learned agent involved, nothing exotic:
| after hand | chips in play | largest stack |
|---|---|---|
| 0 | 1,200 | 105 bb |
| 500 | 1,400 | 208 bb |
| 1,999 | 1,600 | 293 bb |
Every benchmark I had ever run was playing its later hands two to three times deeper than the 100bb it claimed. Deep-stack pots are bigger pots, bigger pots are more variance, and the effect compounds over a session — which is precisely why the wildest opponents (a maniac, a random bot) had the least resolvable rows. The variance wasn't a property of the game. It was being manufactured by the harness.
One flag — reset every hand to the buy-in — applied to the same seed, the same 2,000 hands, the same bot:
| matchup | carry-over | fixed depth | variance | |
|---|---|---|---|---|
| maniac | +3,411 ± 14,041 | +290 ± 264 | 9.86M → 3.5k | 2,800× |
| random | +5,426 ± 4,002 | +352 ± 218 | 801k → 2.4k | 338× |
| self-play | −340 ± 349 | −52 ± 167 | 6.1k → 1.4k | 4.4× |
| calling station | +683 ± 247 | +518 ± 165 | 3.1k → 1.4k | 2.2× |
Both unresolved rows resolve at 2,000 hands. But look at the estimates, not just the bars. +3,411 became +290. The old number wasn't a noisy measurement of the true win rate — it was mostly the compounding of deep-stack pots, which is a different quantity. A wrong error bar had been hiding a wrong point estimate, and the wide bar was the only reason nobody had acted on the wrong number.
The project also has a PPO agent that trains inside the same engine. Training used the carry-over harness. Evaluation used the fixed-depth one. So the agent was trained at 200–300bb and scored at 100bb, and nobody had noticed, because both halves looked internally consistent.
The tell came from an unrelated tool. I train a throwaway agent against a frozen opponent to estimate how exploitable it is; that exploiter reached +1,282 bb/100 during training and scored −759 at evaluation. The same policy, measured twice, off by two thousand bb/100. It had learned deep-stack poker and was being examined on a different game.
Matching the two depths fixed the contradiction — training curve and evaluation now agree — and then improved the agent itself, on both axes at once:
| vs. the heuristic, heads-up | exploitability ↓ | |
|---|---|---|
| carry-over depth | +180.4 ± 32.6 | 879.4 ± 101.5 |
| fixed depth | +604.0 ± 36.1 | 670.6 ± 68.8 |
While I was measuring measurements, I checked the one I'd been using for exploitability. The project ships a hand-written adaptive opponent — a "leak hunter" that models your frequencies and counter-adjusts — and it rated the heuristic bot as nearly unexploitable: +14.8 bb/100 in the bot's favour.
A best response trained from scratch takes 1,019.7 bb/100 off the same bot. The proxy was under-reporting by about 70×, because it only knows a handful of counter-rules and can't find leaks it wasn't written to look for.
Worse, I tried the obvious thing: put the leak hunter in the training pool so the agent learns not to be readable. It flipped that column exactly as intended, from −136 to +1,203 — and made the agent nearly twice as exploitable by a real best response, 879 → 1,294. It had learned to beat one specific counter-strategy at the cost of general robustness. That is what training on your own test set looks like when the test set is a bot.
Everything above is reproducible from the repo: python -m pokr.bench --hands 2000 --seed 7 --mc-iters 10 --fast for the old numbers, --reset-stacks for the new ones. The duplicate-deck evaluator, the all-in EV adjustment, and the best-response probe are in pokr/duplicate.py, pokr/allin_ev.py, and pokr/rl/exploit.py — each carrying the measured payoff and the dead ends in its docstring, so the next person doesn't rediscover them.
Worth stating plainly: the agent is still ~670 bb/100 exploitable, which is catastrophic in absolute terms — a best response wins about 6.7 big blinds per hand. Nothing here is near equilibrium. The claim is that it is measured honestly, not that it is good.