Know which exact frames are included in your samples

[JorritBootsma]

🚀 Feature

For traceability and reproducibility of a dataset, it is essential to know which exact frames are included in each definitive clip created by the FrameVideo.get_clip() method. This information is available in the current implementation, but it is not passed downstream.

Motivation

Models are deemed to be transparent by upcoming legislation such as the AI Act in Europe and in the US. Traceability and reproducibility are key factors for AI models in terms of transparency. Knowing which exact frames are included in your train/test sample clips is vital for traceability and reproducibility.

Pitch

FrameVideo.get_clip() returns the "frame_indices" in its output dictionary. However, this information is not processed by LabeledVideoDataset.__next__(). The solution is as simple as adding the "frame_indices" key and corresponding value to sample_dict, returned by LabeledVideoDataset.__next__():

From LabeledVideoDataset, L215

            frames = self._loaded_clip["video"]
            audio_samples = self._loaded_clip["audio"]
            frame_indices = self._loaded_clip["frame_indices"]
            sample_dict = {
                "video": frames,
                "video_name": video.name,
                "video_index": video_index,
                "frame_indices": frame_indices,
                "clip_index": clip_index,
                "aug_index": aug_index,
                **info_dict,
                **({"audio": audio_samples} if audio_samples is not None else {}),
            }

Alternatively, the frame_indices information could be added to the info_dict.