Center for Innovation in Data Engineering and Science, University of Pennsylvania
UniTransPose is a multi-scale vision transformer for 2D human pose estimation. It pairs an efficient encoder that mixes local and global attention through learnable joint tokens (Joint-Aware Global-Local attention) with an efficient sub-pixel CNN decoder, and is trained on a unified 23-keypoint skeleton that merges six benchmarks so a single model learns from all of them at once.
This is the official implementation, released as an MMPose project so it plugs directly into MMPose's training, evaluation, and inference framework.
- 📰 News
- 😮 Highlights
- 📖 Abstract
- 🚀 Main Results
- 🛠️ Requirements and Installation
- 🐳 Model Zoo
- 🤖 Usage
- 🗝️ Training & Validating
- 🔒 License
- ✏️ Citation
- [2026] 🎉 UniTransPose is accepted to IEEE TPAMI — read the paper.
- [2023] 📌 The efficiency-focused companion model EViTPose was presented at BMVC 2023 (Oral).
UniTransPose targets the three desiderata of pose estimation — accuracy, efficiency, and robustness — in a single model.
💡 Joint-Aware Global-Local (JAGL) attention
A small set of learnable joint tokens gathers global context from the patch tokens (patch-to-joint), mixes information among themselves (joint-to-joint), and scatters the context back onto the patch tokens (joint-to-patch). Because the joints are far fewer than the patches, this injects global reasoning into windowed local attention at linear cost in the number of patches — unlike the quadratic cost of full self-attention.
⚡ Efficient sub-pixel CNN decoder
A PixelShuffle-based head upsamples in the low-resolution feature space and then refines, producing accurate Gaussian heatmaps with far fewer operations than transpose-convolution decoders.
🦴 Unified 23-keypoint skeleton
Joints from MS-COCO, MPII, AI Challenger, CrowdPose, JRDB-Pose, and OCHuman are merged into a single skeleton, enabling robust multi-dataset training, zero-shot transfer to shared joints, and flexible joint selection at inference.
🔀 Hybrid decoding
A heatmap decoder (accuracy) and a keypoint-regression head (speed) share the same encoder — choose either at inference time depending on your accuracy/latency budget.
Click to expand
Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) have led to significant progress in 2D body pose estimation, yet striking a good balance between accuracy, efficiency, and robustness remains a challenge: CNNs are efficient but struggle with long-range dependencies, while ViTs capture them but suffer from quadratic complexity. We propose two ViT-based models for accurate, efficient, and robust 2D pose estimation. EViTPose operates efficiently without sacrificing accuracy by using learnable joint tokens to select and process a subset of the most important body patches, giving direct control over the accuracy/efficiency trade-off. UniTransPose efficiently handles multiple scales by combining an efficient multi-scale transformer encoder that uses both local and global attention with an efficient sub-pixel CNN decoder for better speed and accuracy. By incorporating all joints from different benchmarks into a unified skeletal representation, we train robust models that learn from multiple datasets simultaneously and perform well across a wide range of scenarios — including pose variations, lighting conditions, and occlusions. EViTPose reduces computational complexity by 30–44% in GFLOPs with a minimal accuracy drop (0–3.5%), and UniTransPose improves accuracy by 0.9–43.8% across six benchmarks.
Trained on the unified skeleton, UniTransPose improves over the prior state of the art across six 2D pose benchmarks:
| Benchmark | Metric | Improvement over SOTA |
|---|---|---|
| MS-COCO | AP | +0.9 – 2.4 |
| AI Challenger | AP | +3.3 – 5.7 |
| OCHuman | AP | +6.2 – 43.8 |
| MPII | PCKh | ✅ consistent gains |
| CrowdPose | AP | ✅ consistent gains |
| JRDB-Pose | AP | ✅ consistent gains |
- Python ≥ 3.8, PyTorch ≥ 2.0 (CUDA build matching your GPU)
- MMEngine, MMCV (
>=2.0.1, <2.2.0), MMDetection (>=3.0) for top-down person detection timm,einops
# 1) environment
conda create -n unitranspose python=3.9 -y && conda activate unitranspose
pip install torch torchvision # pick the CUDA build for your machine
# 2) OpenMMLab stack
pip install -U openmim
mim install "mmcv>=2.0.1,<2.2.0" "mmengine" "mmdet>=3.0"
# 3) this repository (an MMPose fork) + extras
pip install -e .
pip install timm einopsThe model lives in projects/unitranspose. Add it to PYTHONPATH so the
config's custom_imports=['unitranspose'] resolves.
| Model | Input | Joints | Config | Checkpoint |
|---|---|---|---|---|
| UniTransPose (COCO-17) | 256×192 | 17 | config | · release TBA |
| UniTransPose (Unified) | 256×192 | 23 | config | · release TBA |
Inference
from mmpose.apis import init_model, inference_topdown
model = init_model(
'unitranspose/configs/unitranspose_coco-256x192.py',
'unitranspose/unitranspose.pth',
device='cuda:0')
# bboxes: np.ndarray [N, 4] in xyxy (e.g. from an MMDetection person detector)
results = inference_topdown(model, img, bboxes=bboxes_xyxy, bbox_format='xyxy')
keypoints = results[0].pred_instances.keypoints # (1, K, 2)unitranspose_coco-256x192.pyreturns the 17 COCO joints.unitranspose_multi23-256x192.pyreturns all 23 unified joints.
# quick 23-keypoint demo + visualization on the bundled sample image
cd projects/unitranspose/unitranspose
PYTHONPATH=.. python test_predict23.pyReleased under the Apache 2.0 License, following MMPose.
If you find UniTransPose useful for your research, please cite:
@article{kinfu2026unitranspose,
title = {Transformers with Joint Tokens and Local-Global Attention for Efficient Human Pose Estimation},
author = {Kinfu, Kaleab A. and Vidal, Ren\'e},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2026}
}Please also consider citing MMPose:
@misc{mmpose2020,
title = {OpenMMLab Pose Estimation Toolbox and Benchmark},
author = {MMPose Contributors},
howpublished = {\url{https://github.com/open-mmlab/mmpose}},
year = {2020}
}