Low-Resource Deep Reinforcement Learning for Four-Player Chess

December 10, 2019

For the Machine Learning Practical course at the University of Edinburgh, we built a low-resource deep reinforcement learning system for four-player chess. The project was called AlphaThree, mostly because the obvious AlphaZero pun was too hard to resist.

The goal was not to build a superhuman chess engine. That would have been wildly unrealistic given the time and compute constraints. The more interesting question was behavioural: if we adapt an AlphaZero-style setup to a multiplayer, team-based game, can reward shaping change whether agents behave more cooperatively or aggressively?

Four-player chess is a nice testbed for that question because it is not just a harder version of normal chess. In the team-play variant, the player across from you is your teammate, which changes the meaning of trades, checks, tempo, and risk. A move that looks reckless in two-player chess might make sense if your partner can follow up.

The setup

We had no clean dataset of four-player games, no Stockfish-like baseline, and no realistic way to train for expert-level play. So the system had to learn through self-play. The training loop was roughly:

generate games through self-play;
train a policy-value network on the resulting states, move probabilities, and rewards;
evaluate against the previous best model and a random agent;
keep the new model if it performed well enough.

To make the project tractable, we simplified some of the game rules. Kings were allowed to walk into check, pawn promotion, en passant, and castling were removed, and games were capped so they could not continue indefinitely. Those choices were not elegant, but they kept the implementation manageable and let us focus on learning dynamics rather than edge cases in chess legality.

The game engine was built by heavily modifying an existing four-player chess implementation. Internally, each player saw the board from a first-person perspective. The state representation used 12 sparse layers: own pieces, other pieces, player identity, piece-type encodings, and player-ID encodings. The action space was much larger than normal chess: a 14-by-14 board with 104 possible move layers, covering rook-like, bishop-like, and knight moves.

The model

The learning setup followed the broad AlphaZero pattern: a residual policy-value network guided Monte Carlo Tree Search. The policy head produced a probability distribution over legal moves, while the value head estimated the state value. Instead of back-propagating a single two-player minimax value through the search tree, we used an N-tuple so each player could maintain its own value estimate.

Policy-value network architecture for four-player chess — The residual policy-value network used to estimate move probabilities and state value.

Because the project was intentionally low-resource, a lot of the work was about making the pipeline run at all. We used multiprocessing for self-play, asynchronous evaluation, and relatively small experiments over ResNet-style architectures and MCTS simulation depths. Even then, some games took a long time to resolve, especially when the agents fell into drawn-out oscillatory play.

Training behaviour

One of the more useful findings was that model depth and MCTS simulation depth interacted nonlinearly. Increasing search depth helped only when the network had enough capacity to make use of it. With too little model depth, more simulations did not buy much; with deeper models and too few simulations, training could still behave badly.

Training time by model and simulation depth — Training time varied sharply with both architecture depth and simulation depth.

Loss curves while varying playout depth — Increasing playout depth reduced loss only once the network was large enough to benefit.

This was a useful practical lesson. It is tempting to think of compute as a simple knob: more search, more training, more network depth. In this project those knobs interacted in ways that made the system feel more like a living experiment than a clean recipe.

Reward shaping

The behavioural part of the project came from reward shaping. We introduced reward functions with parameters controlling the relative weight of winning versus immediate material gain. In rough terms, this let us train more cooperative agents and more aggressive agents.

The cooperative setting gave no penalty for draws. Unsurprisingly, agents learned to prefer drawing behaviours, including oscillating pieces back and forth until the game ended. When we penalized draws and increased the reward for taking pieces, agents became more aggressive.

Q-value dynamics for Player 1 across evaluation games — Player 1 Q-value profiles across evaluation games. Some runs settled into oscillatory draw-like behaviour, while others stayed more chaotic.

The win/draw/loss patterns were also pair-specific. The aggressive agent won more often against the cooperative agent, but it also lost more often. Against a softer aggressive agent, it drew much more often. That was the interesting part: changing reward shaping did not just make one agent "better" or "worse"; it changed the interaction dynamics between agent types.

Difference in number of pieces taken by cooperative and aggressive agents — Aggressive agents tended to take more pieces, though the behaviour was not uniform across games.

We speculated that some of the deterministic and oscillatory outcomes had a loose relationship to game-theoretic patterns such as hawk-dove dynamics. That claim would need much more careful experimentation, but it was a useful way to think about what we were seeing: the agents were not just learning chess moves, they were learning interaction patterns inside a multiplayer reward structure.

Looking back

This was a messy project in the best way. It touched bitboards, game logic, deep RL, MCTS, residual networks, multiplayer credit assignment, self-play infrastructure, and reward design. It also made the limits of low-resource RL very concrete. A small implementation detail or hyperparameter could easily dominate the behaviour we were trying to study.

There are many things I would change now. I would spend more time on baselines, make the rule simplifications more explicit in the experiments, run broader sweeps over reward parameters, and compare against pure MCTS agents or human game data if available. I would also separate the engineering and behavioural questions more cleanly: first make sure the agent is learning a sensible game model, then use it to study cooperation and competition.

Still, I like the spirit of the project. It took a famous idea from two-player games and pushed it into a more awkward multi-agent setting, where winning is not only about being strong but also about how your behaviour interacts with the other agents around you.