Crosshair: A New Board Game to Showcase Pure AlphaZero Self-Play

In my previous post, Monte Carlo Tree Search is Awesome, we explored my favorite algorithm and made a bunch of different engines. One letdown though was that the AlphaZero-style Ultimate Tic-Tac-Toe engine specifically had some issues: the learned value head got exploited by the search and lost 4–44 against plain random rollouts. I could probably spend some more time on the training and make it work, but I thought it might be more fun to design a new game entirely that is more amenable to AlphaZero-style training when you only have one 4060 Ti to train on.

So I created a game I'm calling Crosshair. It is a mashup of Hex and Ultimate Tic-Tac-Toe on a 7×7 grid. Hex and Ultimate have some rules that make training from pure self-play a little harder (tho not impossible ofc, I just didn't power through it).

If you don't want to read all this and just want to play, click here, or if you just want to see the code, it's on GitHub and crates.io.

In Ultimate Tic-Tac-Toe, you have 81 squares, long games, lots of draws that muddle the win/loss signal, and "sent to an already-won board" wildcards that suddenly blow the branching factor wide open. Hex has no draws, which is great, but on an 11×11 or 13×13 board you have over 100 legal moves every turn, so MCTS spreads a small simulation budget way too thin without hand-written bridge patterns or RAVE.

Crosshair takes the best parts of both and strips out the headaches. It is played on a compact 7×7 grid (49 squares). Red goes first and tries to connect the top and bottom edges with an orthogonal chain of stones (up, down, left, right, no diagonals). Blue tries to connect the left and right edges. And since I just made the rules up, there are zero existing opening books or strategy guides to crib from.

First, Ultimate-style move routing cuts the branching factor. Wherever you play sets a row-and-column crosshair where your opponent must play next (unless both lines are full, which gives them a wildcard anywhere on the board). Instead of searching all 49 squares, MCTS only has to search 8 to 12 legal moves per turn.

Second, pinch flips make draws impossible on a square grid. Normally, a connection game on a square grid can deadlock whenever Red and Blue form a 2×2 checkerboard (R B over B R), which blocks both players at once. Crosshair kills deadlocks with two flip rules when you place a stone: sandwiching an adjacent enemy stone in a line (R [B] R) flips that stone to your color (without chaining down the line), and completing your diagonal of a 2×2 checkerboard (R B / B R) flips both enemy stones, cascading if a flipped stone creates another 2×2 checkerboard.

To strictly guarantee zero draws, the 2×2 crosscut check has to cascade. A full 7×7 grid is guaranteed to have a winning road for Red or Blue if and only if no R B / B R 2×2 checkerboard survives on the board. When a pinch flips an enemy stone, that square is already occupied, so if its new color forms a brand-new R B / B R 2×2 box with three neighboring stones, all four squares of that box are already full and no future move could ever land inside it to break the deadlock. So after checking linear pinches, the engine cascades the 2×2 crosscut check across any newly flipped stones until every 2×2 checkerboard is resolved. Across 70,000 test games, every single game ended in a win (0 draws) in an average of 27 moves.

Give it a shot (and get rekt)

How the AlphaZero loop works

To define it exactly, an AlphaZero loop uses a single neural network that looks at a board position and outputs a move prior $P(s,a)$ over the 49 squares and a win value $v \in [-1, +1]$.

During search, MCTS runs 450 simulations per move and picks actions using the PUCT formula:

So here is what happens early on when search is just following the prior from the policy network:

As you can see though, the favorite move changes as visits build up. That is what turns MCTS into a policy improvement step. The network starts with a rough guess of which moves look good, and the tree search checks what happens a few moves ahead to find out which move actually holds up.

You then train the policy network to match the search's visit counts, and you train the value network on who actually won the game. Because Crosshair games finish in about 27 moves with zero draws, any position the value network overrates gets played into during the next round of self-play, loses, and gets fixed.

For the network itself, I used a small 114,290-parameter MLP instead of a 2D convolutional network. In Crosshair, a move at d4 affects all of Row 4 and Column D at once, so a fully connected layer sees the whole crosshair right away. It is also tiny (457 KB), which fits in CPU cache and evaluates a position in about 25 microseconds, fast enough to run 40,000 simulations per move in the browser.

For the input, I feed the board in as six 7×7 grids flattened into 294 numbers: my stones, the opponent's stones, the legal moves in the active crosshair, how many stones each legal move would flip, the opponent's last move, and the turn number.

Those 294 numbers go into a 128-wide linear layer with ReLU, then through two 128-wide residual blocks (each one runs two linear layers with a skip connection that adds the block's input back to its output).

At the end, the network splits into two heads. The policy head projects the 128 numbers down to the 49 board squares and runs a softmax over the legal moves to get the move probabilities. The value head projects the 128 numbers down to 32, and then to a single tanh output between -1 and +1 for the win/loss score.

I wrote the training in Python with raw JAX (no PyTorch or Flax), and the trained weights get compiled straight into the Rust engine so it can run in WebAssembly with zero dependencies.

Training it on one GPU in a few minutes

Each self-play generation takes about 40 seconds on my desktop: Rust plays 220 self-play games across 16 CPU threads (~18s), JAX trains the network on the RTX 4060 Ti (~4s), and Rust runs 60-game arena matches (~18s). Almost 90% of the wall-clock time is CPU tree search, not GPU backprop.

Ten generations (2,200 self-play games) took about six minutes end to end. Across those ten generations, top-1 move agreement with 450-sim MCTS climbs from 19% to 48%, and winner prediction accuracy climbs from a 50% coin flip to 70%:

After training, I matched it up with random-rollout UCT at a 400-simulation-per-move budget. It started out losing 21–39 (35%) in Generation 1, passed 50% in Generation 2, and ended up beating random-rollout UCT 43–17 (72% win rate). In random-rollout UCT, each simulation plays random moves all the way to the end of the game to see who wins. The AlphaZero engine does not play out random games at all. At each new position in the tree, it just asks the neural network for a score:

Check out some of the other stuff on the site

• Play Crosshair with all engine modes and controls, or try the other six engines on the Game AIs page: Hex, Ultimate Tic-Tac-Toe, Chess, Othello, Connect Four, and Backgammon
• Monte Carlo Tree Search is Awesome
• The 10-Arc LLM Series
• The Rust engine and JAX training code on GitHub and ai-crosshair on crates.io