TalkingPose Preprocessing

This repository provides tools for downloading TalkingPose videos, extracting annotated frame sequences, and generating stable, presenter-centered video crops for downstream processing.

Requirements

• Python 3.10 or newer
• A CUDA GPU is recommended for presenter detection and cropping, but CPU processing is supported

Install the Python dependencies:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Annotation format

TalkingPose_annotations.csv has three columns:

Column	Description
video_id	11-character YouTube video ID
segments	Python-style list of (start_frame, end_frame) pairs
split	Dataset assignment: train, val, or test

Example:

video_id,segments,split
abcdefghijk,"[(100,249),(400,599)]",train

Part A: use the provided TalkingPose annotations

1. Download the videos

python download_videos.py TalkingPose_annotations.csv data/raw

The downloader:

• reads video IDs from the first CSV column;
• prefers 1080p, then 720p;
• skips completed downloads when run again;

Downloaded files are named <video_id>.<extension> in data/raw.

2. Extract the annotated video chunks

At this stage, character detection has already been performed, and frames where the presenter's face or hands are not visible have been filtered out. This produces shorter, high-quality video segments that are better suited for downstream applications. The extractor matches each video filename to video_id and preserves audio by default. Please note that the segment information in the csv file is based on the frames not time.

Separate segment files

python extract_annotated_videos.py \
  data/raw \
  TalkingPose_annotations.csv \
  data/segments \
  --mode segments \
  --workers 4

Output layout:

data/segments/
└── <video_id>/
    ├── <video_id>_1.mp4
    ├── <video_id>_2.mp4
    └── ...

Each CSV range becomes one MP4 file. Existing clips are skipped unless --overwrite is supplied.

One filtered video per source

python extract_annotated_videos.py \
  data/raw \
  TalkingPose_annotations.csv \
  data/filtered \
  --mode merged \
  --workers 4

This removes every unannotated frame and concatenates the retained ranges in CSV order:

data/filtered/<video_id>.mp4

Useful extraction options:

Option	Default	Description
--mode	segments	Write separate segments or one merged video
--workers	automatic	Number of videos processed concurrently
--crf	22	H.264 quality; lower values produce larger, higher-quality files
--preset	veryfast	libx264 speed/compression preset
--no-audio	off	Remove audio from outputs
--overwrite	off	Replace existing outputs

Only videos found in both the input folder and CSV are processed. Missing downloads are reported and do not stop other videos.

3. Create Stable Square Clips (Character Cropping)

This step generates stable, square video clips with the presenter centered in the frame. Since irrelevant frames have already been filtered out, the cropping process is more efficient and produces more consistent results.

The annotated ranges in the CSV define boundaries within each merged video, ensuring that output clips do not cross annotated transitions.

python second_stage/crop_presenter_clips.py \
  data/filtered \
  data/crops \
  --chunks-csv TalkingPose_annotations.csv \
  --chunk-frames 100 \
  --skip-existing

For each input video, the crop stage:

• detects and tracks the main presenter with a pose model;
• detects abrupt camera changes;
• creates a smooth square crop around the presenter;
• applies CSV boundaries as mandatory cuts;
• writes fixed-length clips without crossing a boundary;
• drops scene tails shorter than --chunk-frames.

Output:

data/crops/
└── <video_id>/
    ├── <video_id>_01.mp4
    ├── <video_id>_02.mp4
    └── manifest.json

Common crop options:

Option	Default	Description
--chunk-frames	100	Frames in each output clip
--output-size	512	Width and height of the square output
--backend	auto	Use gpu, cpu, or automatic selection
--model	yolo26m-pose.pt	Ultralytics pose checkpoint name or local path
--detect-every	1	Run pose detection every Nth frame
--skip-existing	off	Skip completed folders and retry incomplete ones
--overwrite	off	Replace existing output folders
--no-audio	off	Do not retain audio

To crop an arbitrary folder without annotation boundaries, omit --chunks-csv:

python second_stage/crop_presenter_clips.py data/videos data/crops

4. Organize videos by dataset split

Use the CSV's split column to recursively copy all matching videos into train, val, and test subfolders:

python split_videos.py \
  data/crops \
  TalkingPose_annotations.csv \
  data/crops_split

The script finds the 11-character video_id anywhere in each filename, so it works with source videos such as <video_id>.mp4 and generated clips such as <video_id>_01.mp4. It preserves the input folder structure below each split:

data/crops_split/
├── train/
├── val/
└── test/

Part B: Filter Frames Without a Presenter (Optional)

This section provides the code used to generate the annotations required in the first stage. You can use this workflow to process new videos or to reproduce our annotation procedure. Use this workflow when a video is not covered by TalkingPose_annotations.csv.

python first_stage/filter_presenter_segments.py \
  --input-dir data/raw \
  --output-dir data/detected \
  --num-workers 4

The first stage keeps continuous ranges containing a single prominent upper-body presenter with visible hands and speaking or gesturing motion. It writes:

data/detected/
├── <video_id>.mp4
└── presenter_chunks_metadata.csv

Model checkpoints

Ultralytics downloads a named checkpoint automatically the first time it is used. The defaults are:

• First stage: yolov8s-pose.pt
• Crop stage: yolo26m-pose.pt

To use a downloaded checkpoint:

License

MIT