Extract FEN positions from screenshots of 2D chessboards — fully open-source, handles multiple boards per image, detects orientation and turn.
The goal isn't just the tool. It's a recipe for tackling 2D chessboard recognition with near-zero manual labeling: classical CV plus open-source models autolabel a dataset that trains a single end-to-end detector (YOLO / RF-DETR).
Give it any image containing 2D chessboards (screenshots, book scans, video frames) and it returns:
- FEN piece-placement string for each board
- Board orientation (white or black at bottom)
- Whose turn it is (inferred from highlighted last-move tiles)
- Per-square confidence scores
git clone git@github.com:AndrewSpano/2d-chess-ocr.git
cd 2d-chess-ocr
uv venv
echo "export PYTHONPATH=\"$(pwd):\$PYTHONPATH\"" >> .venv/bin/activate
source .venv/bin/activate
uv syncPre-trained weights, ready to use — no training required.
uv run download_models.pyuv run model_inference.py \
--model finetuned-models/yolo26m-finetuned.pt \
--image images/input-example.png \
--output annotated.png
Found 2 chessboard(s):
Board 0: orientation=unknown turn=w turn_conf=0.50
FEN: r1bqkb1r/pppp1ppp/2n2n2/4p3/2B1P3/2P2N2/PP1P1PPP/RNBQK2R
Board 1: orientation=unknown turn=w turn_conf=0.50
FEN: r1bqk1nr/pppp1ppp/2n5/2b1p3/4P3/2N2N2/PPPP1PPP/R1BQKB1R
Saved annotated image to: annotated.png
A single YOLO model detects all objects in one pass — no multi-stage pipeline at inference time:
| Class ID | What | Count |
|---|---|---|
| 0 | Chessboard bounding box | 1 per board |
| 1–12 | Individual pieces (6 black + 6 white) | ~32 per board |
| 13 | Row-1 label (orientation marker) | 0–1 per board |
| 14–15 | Last-move start/end tiles | 0–2 per board |
Postprocessing assigns pieces to boards by position, reads the grid into a FEN string, determines orientation from the row-1 marker, and infers turn from the piece on the last-move end tile.
The YOLO training dataset is built through a multi-step autolabeling pipeline — see dataset_creation/README.md for the full walkthrough:
- Detect boards — classical CV detector
- Classify pieces — SAM3 detects generic pieces, an ensemble of 3 DNNs classifies them with uncertainty estimation
- Detect orientation — SAM3 locates the "1" digit near corners, verified by a small CNN binary classifier
- Label last move — manual annotation via a tkinter app (2 clicks required per board --> ~1s annotation time per image)
The trained YOLO model is then fine-tuned on this dataset — see model_training/README.md.
A precomputed version of the autolabeled training dataset is provided in public-datasets/. It contains YOLO-format label files and a mapping.jsonl recording each label's provenance (images are not provided, due to copyright). A small binary dataset for the digit-one orientation classifier is also included.
See public-datasets/README.md for details and unzip commands.
2d-chess-ocr/
├── download_models.py # fetch the finetuned models from Hugging Face
├── model_inference.py # run inference on images (CLI + importable classes)
├── chess_ocr_detection.py # detection results --> FEN conversion logic
│
├── dataset_creation/ # dataset creation and autolabeling pipeline
│ ├── chessboards/ # board detection/autolabeling
│ ├── chess_piece_sets/ # piece tile dataset, classifier, and piece autolabeling
│ ├── orientation/ # rank-one orientation dataset/classifier/autolabeling
│ ├── synthetic_data/ # synthetic assets and YOLO synthetic generation
│ ├── turn/ # last-move tile annotation tool
│ ├── board_corners.py # Hough-transform tile-intersection detector
│ └── yolo_to_coco.py # YOLO --> COCO dataset converter (for RF-DETR)
│
├── model_training/ # training, evaluation, export (YOLO + RF-DETR)
│ ├── train_yolo.py
│ ├── evaluate_yolo.py
│ ├── export_yolo.py
│ ├── train_rfdetr.py
│ ├── evaluate_rfdetr.py
│ └── fen_eval.py # shared FEN-reconstruction evaluation
│
├── public-datasets/ # shipped precomputed datasets (zipped)
├── baselines/ # baseline benchmark scripts
│
├── pyproject.toml
└── uv.lock
Evaluated on 245 images from an out-of-distribution subset of the generated dataset. Latency = mean end-to-end per-image wall-clock time (including preprocessing and postprocessing) on an NVIDIA RTX 4090.
| Method | Detection rate | Full-board acc¹ | Latency | Orientation | Turn | Multi-board |
|---|---|---|---|---|---|---|
| chess-ocr | 100% | 6.94%² | 13.3 ms (CPU) | ✗ | ✗ | ✗ |
| tensorflow_chessbot | 49.8%³ | 66.12% | 12.4 ms (CPU+GPU⁴) | ✗ | ✗ | ✗ |
| YOLO-m (this repo) | 100% | 95.93% | 5.8 ms (GPU) | ✓ | ✓ | ✓ |
| RF-DETR-L (this repo) | 100% | 95.19% | 10.4 ms (GPU)⁵ | ✓ | ✓ | ✓ |
¹ Best-orientation: a board is counted correct if the predicted FEN or its 180° rotation matches ground truth — this decouples piece recognition accuracy from orientation detection.
² Trained on synthetic SVG diagrams; the low accuracy reflects domain gap, not architecture capability.
³ Low detection rate: the Hough-line board detector fails on a significant fraction of images.
⁴ Board detection (Hough lines) runs on CPU; piece classification runs on GPU via ONNX Runtime. The CPU-bound detector is the bottleneck.
⁵ RF-DETR latencies are the native PyTorch model with optimize_for_inference() — unlike the YOLO numbers, there is no ONNX/TensorRT export yet, so further speedups are likely available.
See baselines for example scripts on how to run the benchmark.
Board size = the board's bounding box area as a fraction of min(img_h, img_w). Accuracies are full-board best-orientation. No boards in the evaluation set had side lengths below 25% of the shortest image dimension.
| Model | Large (≥50%, n=222) | Medium (25–50%, n=47) | Latency (mean) |
|---|---|---|---|
| YOLO-m | 97.75% | 89.36% | 5.8 ms (GPU)* |
| YOLO-n | 83.33% | 40.43% | 4.2 ms (GPU) |
| RF-DETR-L | 98.20% | 82.98% | 10.4 ms (GPU)† |
| RF-DETR-2XL | 97.75% | 82.98% | 16.2 ms (GPU)† |
*If using TensorRT, the YOLO-m model has a latency of ~2.1ms.
†RF-DETR latencies are the native PyTorch model with optimize_for_inference() (no ONNX/TensorRT export yet).
A single misclassified piece is enough to make a board count as wrong under the full-board metric. The most common failure mode is piece-color confusion. The example below shows a white queen predicted as black:
However, post-processing (e.g. enforcing chess-legality constraints, or simple heuristics) could catch many such mistakes.
- This method assumes cropped, axis-aligned, uniform 2D chessboards, since detected boards are split into an 8×8 grid mechanically. If this assumption does not hold,
find_chessboard_corners()from board_corners.py can detect tile intersections via the Hough transform, though this adds a ~5 ms latency penalty due to the CPU-bound implementation. - This method assumes chessboards are axis-aligned. For rotated or warped boards (e.g., photos taken with a phone), the problem can instead be framed as instance segmentation of the board region, followed by a homography to map it to an axis-aligned view before resuming the normal pipeline.
- Model size: The best performing model (yolo26-medium, onnx version) is 80MB, which is not really deployable on edge devices. Ideas such as compression and quantization could help in this regard, and they're worth exploring.
- Improving performance: Generally, more data could help close the gap and reach near 100% accuracy on unseen, OOD boards. A simple idea would be to finetune a foundation model on this task, and then use it to pseudolabel more images. If the foundation model can reach near 100% accuracy (with the current amount of data), it likely will aid the smaller, specialized YOLO models.
- Predict tile coordinates as auxiliary labels: Extend the YOLO/RF-DETR heads to predict two extra outputs per detection, alongside the class — the tile's rank (0–7) and file (0–7). This would localize each piece on the board directly, removing the geometric grid-assignment step currently done during FEN conversion. It would likely require more data, as well as adapting the loss function to ignore these outputs for classes where they don't apply (e.g. the chessboard box or the orientation marker). This detection-based framing (segment each piece, regress its tile) is also why the approach should scale to 3D / photographed boards, where two-stage "split into tiles, then classify each tile" pipelines break down under perspective and piece occlusion.
This tool is intended for game analysis, accessibility, and research. Do not use it for cheating in online chess. As a matter of fact, do not cheat. In life. In general.
See LICENSE.