Crosshair-Bench: Evaluating Frontier Models on an Unseen Game

I wanted a board game to benchmark language models on that isn't sitting in their training data. So on October 25, 2026, I invented Crosshair and trained a tiny 114K-parameter AlphaZero net for it on my desktop GPU in six minutes. None of these models have ever seen a Crosshair board or read a strategy guide for it. Should be safe for a few generations yet!

This note walks through the results from the Crosshair LLM Benchmark.

How the tournament works

I ran a 480-game tournament (crosshair-bench) with 20 models from five labs (Flagship and Flash tiers, high and low thinking) plus four search engines from ai-crosshair. Model matchups are 4 games each with colors swapped, and the engines played more games against each other to anchor the rating scale at az-puct-1 = 1200 Elo. Each turn, the harness sends the rules, the 7×7 ASCII board, and the legal moves, and az-puct-400 scores the move in the background (AZ-400 Match in the chart below). Check it out:

Against the engines

Every model played 4 games each against the MCTS and AlphaZero baselines:

• Without tree search, top LLMs win: gemini-4-argon@high (1440 Elo) and gpt-6.1-sol swept random-uct-50 4–0, and 9 of the 20 models finished ahead of the raw policy net (az-puct-1, 1200 Elo). If the neural net isn't allowed to search ahead, a frontier model thinking in its scratchpad can beat it.
• At 400 simulations (az-puct-400, 1888 Elo), the engine went 79–1: 19 of the 20 models got swept 0–4. Only gemini-4-argon@high took a game (1–3).
• At 4,000 simulations (az-puct-4000, 2343 Elo), the engine went 80–0: Even when gpt-6.1-sol@high spent 70 seconds and $0.49 per game thinking through lines, az-puct-4000 (0.28s per move) had the board locked by ply 16.

Thinking budget vs. model size

High thinking beat low thinking 25–15 (+149 Elo on average) on the same model. More interesting: I pitted each lab's smaller Flash @high model against its own Flagship @low/@none model, and the small models won 16–4:

Not a single low-thinking Flagship won its mini-match. claude-haiku-5.5@high swept claude-opus-5.5@low 4–0 at 16× lower cost, and qwen3.8-flash@high swept qwen3.8-max@none 4–0 at 13× lower cost. On a game you haven't memorized, having tokens to actually work through moves matters way more than model size.

Flagship and Flash round-robins

In the 5-lab round-robins (4 games per pair):

• Flagship @high: gpt-6.1-sol@high (12–4) took first in the head-to-head bracket, sweeping Opus and Qwen 4–0 and splitting 2–2 with Argon and DeepSeek. gemini-4-argon@high (11–5) finished one game back in the round-robin, though its win over az-puct-400 kept it #1 overall on Elo.
• Flash @high: gemini-3.8-flash@high (13–3, 1328 Elo) crushed the Flash bracket, hit the highest AZ-400 Match rate in the entire tournament (46.3%), and rated higher than three of the five Flagship @high models. Apparently Flash didn't get the memo that it was supposed to stay in the budget bracket.

Walking through the best games

Here are four of the best games from the tournament. Pick a tab below to step through the board, the az-puct-400 eval bar, and what the model was thinking on each turn (or click any model on /benchmark to browse all of its games):