Reproduce an LLM poker benchmark from public season rows

A public season row gives you a bot name, a score, and a hand count. It does not say whether that bot used an LLM. An LLM poker benchmark still needs those rows, plus a cohort list you can defend. Sample counts later in this draft are illustrative fixture data, not a live season result. No running season was read while writing this page.

The argument for using poker at all sits in Why Poker Is the Best Benchmark for LLM Agents. This page is only the measurement method: which public fields you can add up, which endpoint freezes them, and why a model ranking does not fall out of the payload.

What a public row contains

Open Poker publishes the current season at GET /api/season/leaderboard and a finished season at GET /api/season/{season_id}/leaderboard. Neither route takes an API key. The current route accepts sort_by (score, hands_played, or win_rate), limit (default 50, capped at 1000), offset, and min_hands (default 0). The REST reference documents a limit of 30/minute per IP. The frozen route returns the snapshot in rank order and takes no page parameters.

A current row includes rank, agent_id, bot_name, score, chip_balance, chips_at_table, rebuys, hands_played, hands_won, win_rate, pro, bot_kind, and status. Keep agent_id for the join. The display name is not an identifier. bot_kind is no_code, self_hosted, or unknown. The server maps an active socket, a hosted worker status, or the bot's desired public runtime onto that label. None of those values means the bot used an LLM. A frozen row keeps the chip fields, hand counts, badge, and prize. It drops win_rate, bot_kind, and status.

win_rate on the current board is hands won divided by hands played, rounded to four decimal places. It is not big blinds per 100 hands. The public row has no blind-level profit, so a bb/100 figure cannot be recovered from it.

Score is the number to aggregate. The server computes chip_balance + chips_at_table - rebuys * rebuy_penalty. The scoring doc and the REST reference describe the configured penalty as 0, which collapses the formula to the sum of the two chip fields. In this repository season_rebuy_penalty also defaults to 0. That is a source default, not a reading of the deployed setting. The penalty is configurable, and GET /api/season/me is the route that returns rebuy_penalty. That route requires a bearer token. A public aggregate should keep the score already on the leaderboard row.

Official display, prizes, and the frozen snapshot use a minimum of 10 hands (season_min_hands_leaderboard). The current endpoint still returns shorter rows when min_hands is 0, so an official-style total has to pass min_hands=10 or filter afterward. Frozen rows were already limited to entries that cleared that threshold, and to bots that passed the active-account check, at the moment the season ended. The snapshot is not every registration.

Season length, starting stack, and blinds are context from the same source, not benchmark outputs: 14 days, 5,000 starting chips, blinds 10/20. How Open Poker Seasons Work is the longer account.

The query to save

List recent seasons with GET /api/season/list. The handler returns at most 20. Read one season with GET /api/season/{season_id}. Page the current board with offsets that do not overlap:

GET /api/season/leaderboard?sort_by=score&limit=1000&offset=0&min_hands=10
GET /api/season/leaderboard?sort_by=score&limit=1000&offset=1000&min_hands=10

Stop when a page is shorter than limit. An empty page is that signal when the row count is an exact multiple of 1000. Do not average rank across calls. Rank is recomputed from the requested sort and offset, so it is not a property of the bot alone.

The host, and an unauthenticated example, already live in the quickstart. This draft does not call that host. The script checked in beside this page has no HTTP client. It reads one local fixture.json.

Leave /api/season/me, /api/season/stats, and /api/season/chart-data out of the archive. Those need a bearer token, and they describe one account.

Where the LLM label has to come from

Join rows to agent ids you already classified. Guessing from the bot name is not a method. A cute name can be a rules engine. A dull name can hide a model call. bot_kind will not settle it either.

If you own the bot, keep the id list next to the code that calls the model. If you do not, the public API does not contain the missing fact. Count those rows as unlabeled and leave them out of the LLM total.

That split matches the bot's own layout. Poker bot software architecture puts the model in the strategy layer and leaves protocol, legal actions, and telemetry outside it. The leaderboard can show the telemetry that was published. It cannot show which layer chose the action.

Aggregation rules

Use one season at a time. Adding two seasons treats a chip reset as profit.

Inside the LLM cohort, after the hand minimum:

• Count bots.
• Sum hands_played and hands_won.
• Divide those sums and round to four decimal places. That is a pooled hand win rate, the same unit as the public win_rate.
• Take the mean and the median of score.
• When the chip fields are present, check score against chip_balance + chips_at_table - rebuys * rebuy_penalty. Stop on a mismatch. The fixture in this repo uses penalty 0 so the check matches the source default. It does not measure the deployed setting.

Rows under the hand minimum stay in an excluded count. So do other and unlabeled cohorts. They do not enter the mean.

Illustrative fixture

The file fixture.json next to this post is synthetic. Agent ids start with fixture-. Names start with Fixture. The season id is illustrative-season-0. These rows are illustrative fixture data, not a live season result.

Fixture Llm East is the score check you can do by hand: chip balance 3100, chips at the table 1600, one rebuy, penalty 0, score 4700. The other rows use the same identity with a penalty of 0.

From web/:

node scripts/llm-benchmark-fixture.mjs content/blog/llm-poker-benchmark-from-season-rows/fixture.json

Stderr prints the illustrative label. Stdout is the report. Every aggregate below is illustrative fixture data, not a live season result.

{
  "provenance": "illustrative_fixture",
  "illustrative": true,
  "label": "illustrative fixture data, not a live season result",
  "season_id": "illustrative-season-0",
  "min_hands": 10,
  "rebuy_penalty": 0,
  "eligible_llm": {
    "illustrative": true,
    "bots": 3,
    "hands_played": 93,
    "hands_won": 29,
    "hand_win_rate": 0.3118,
    "mean_score": 5400,
    "median_score": 5100
  },
  "excluded": {
    "illustrative": true,
    "below_min_hands": 1,
    "not_llm": 1,
    "unlabeled_cohort": 1
  },
  "claims": [
    {
      "name": "eligible_llm_bots",
      "value": 3,
      "illustrative": true
    },
    {
      "name": "eligible_llm_hands_played",
      "value": 93,
      "illustrative": true
    },
    {
      "name": "eligible_llm_hands_won",
      "value": 29,
      "illustrative": true
    },
    {
      "name": "eligible_llm_hand_win_rate",
      "value": 0.3118,
      "illustrative": true
    },
    {
      "name": "eligible_llm_mean_score",
      "value": 5400,
      "illustrative": true
    },
    {
      "name": "eligible_llm_median_score",
      "value": 5100,
      "illustrative": true
    },
    {
      "name": "excluded_below_min_hands",
      "value": 1,
      "illustrative": true
    },
    {
      "name": "excluded_not_llm",
      "value": 1,
      "illustrative": true
    },
    {
      "name": "excluded_unlabeled_cohort",
      "value": 1,
      "illustrative": true
    }
  ]
}

Read that report as illustrative fixture data, not a live season result. Three LLM bots clear 10 hands: 93 hands, 29 hands won, hand win rate 0.3118, mean score 5400, median score 5100. The 8-hand LLM row, the rules bot, and the unlabeled bot stay outside those totals.

The loader rejects a file that contains a number without provenance set to illustrative_fixture and illustrative set to true. It also rejects a nested claim whose number is not marked illustrative. It rejects credential-like keys, remote URLs, non-integers, score mismatches, agent ids that are not fixture ids, and paths outside content/blog. A saved production payload does not print through this script.

What the total still does not prove

A pooled hand win rate is a weak poker result. It ignores position, stack depth, and who else was at the table. 93 hands would be a small sample even if these bots were real. They are not. Illegal actions and timeouts are not columns on the public row, so this aggregate cannot score the agent loop the benchmark essay is about. Those counters have to be logged by the bot.

A later, indexable page needs a saved public response, the cohort list, the season id, and the date that response was fetched. Until a named reviewer has that evidence, this page stays a draft.

FAQ

Does the public leaderboard identify LLM bots?

No. Current rows expose bot_kind as no_code, self_hosted, or unknown. Frozen rows omit even that. An LLM total needs an external cohort list.

Why are the counts in this draft not season results?

They are computed from the checked-in file fixture.json. The loader emits them only when that file is marked illustrative fixture data, not a live season result.

Can the public win rate rank models?

Only as hands won over hands played. The field is not big blinds per 100 hands, and a short run is noisy. A model ranking that ignores the hand count is unfinished.

Which response is the one to archive?

For a finished season, keep GET /api/season/{season_id} and GET /api/season/{season_id}/leaderboard. The current endpoint keeps moving until the season ends. Store the raw JSON. Do not store an API key beside it.