Video Encode/Decode

[Daedie-git]

The open encode/decode enticed me to try and get VK hardware decoding working in my software (until now, I only had it on the D3D12 side). I expect this will need further changes, just let me know.
I wasn't sure what the right way to handle the D3D12 CommandBuffer was, I settled on the variant for now.

Summary

Adds hardware video queue support and a minimal native-backed video encode/decode extension API.

This branch makes video-capable queues visible through NRI, allows D3D12/Vulkan video command buffers to be created and submitted, adds
basic multi-planar video format support, and exposes NRIVideo entry points for issuing native encode/decode commands.

Changes

Public API

• Adds QueueType::VIDEO_DECODE
• Adds QueueType::VIDEO_ENCODE
• Adds multi-planar 4:2:0 formats:
  • G8_B8R8_2PLANE_420_UNORM
  • G10X6_B10X6R10X6_2PLANE_420_UNORM_3PACK16
  • G16_B16R16_2PLANE_420_UNORM
• Adds explicit image plane bits for multi-planar images:
  • PLANE_0
  • PLANE_1
  • PLANE_2
• Adds Include/Extensions/NRIVideo.h
  • VideoInterface
  • CmdDecodeVideo
  • CmdEncodeVideo
  • native D3D12/Vulkan encode/decode descriptor structs

Adapter and Queue Discovery

• Reports D3D12 video decode/encode queue availability when ID3D12VideoDevice is available
• Adds Vulkan queue family selection support for video queues
• Uses VkQueueFamilyVideoPropertiesKHR to ensure Vulkan video queues also expose compatible codec operations
• Reuses queue family create infos when multiple NRI queue types map to the same Vulkan queue family
• Tracks queue storage for the new video queue types in D3D12 and Vulkan devices

D3D12 Backend

• Maps NRI video queues to:
  • D3D12_COMMAND_LIST_TYPE_VIDEO_DECODE
  • D3D12_COMMAND_LIST_TYPE_VIDEO_ENCODE
• Refactors CommandBufferD3D12 to hold graphics, video decode, or video encode command lists
• Supports begin/end/reset/close for D3D12 video command lists
• Submits video command lists through the existing queue submission path
• Adds CmdDecodeVideo via ID3D12VideoDecodeCommandList::DecodeFrame
• Adds CmdEncodeVideo via ID3D12VideoEncodeCommandList2::EncodeFrame
• Updates native command buffer documentation to return ID3D12CommandList*, since video command lists are not graphics command lists

Vulkan Backend

• Adds video-related desired device extensions, including:
  • VK_KHR_video_queue
  • VK_KHR_video_decode_queue
  • H.264/H.265/AV1 decode extensions
  • VK_KHR_video_encode_queue
  • video maintenance extensions
  • sampler YCbCr conversion
• Adds Vulkan format mappings for the new multi-planar formats
• Adds Vulkan plane aspect mapping for explicit plane views
• Allows texture views to select explicit multi-planar aspects
• Resolves vkCmdDecodeVideoKHR and vkCmdEncodeVideoKHR
• Adds CmdDecodeVideo / CmdEncodeVideo forwarding to Vulkan command buffers

Validation / Interface Wiring

• Wires VideoInterface through nriGetInterface
• Adds DeviceBase::FillFunctionTable(VideoInterface&)
• Adds backend FillFunctionTable(VideoInterface&) implementations
• Adds validation-layer storage and forwarding for the video interface
• Reports unsupported when the backend/device does not expose video decode or encode queues

API Shape

The new video API is intentionally small and native-backed.

NRI handles queue discovery, command buffer ownership, function table exposure, and command dispatch. Codec/session setup remains
backend-native (initially):

• D3D12 callers pass native decoder/encoder objects and D3D12 argument structs
• Vulkan callers pass native VkVideoDecodeInfoKHR / VkVideoEncodeInfoKHR payloads

Validation

Built successfully with:

• Vulkan/validation-focused configuration
• D3D12-focused configuration
• small encode -> decode unit test in my own repo

[dzhdanNV]

Wow! Magnificent! Thanks a ton for bringing this work to life! I will try to review later today.

[dzhdanNV]

Question 1

Should VideoDecodeD3D12Desc, VideoEncodeD3D12Desc, VideoDecodeVKDesc and VideoEncodeVKDesc be moved into corresponding wrappers and the underlying objects created via NRI? (not a pro in video APIs yet)

Question 2

Should we add more formats? We should invent better names :) DXGI has short names, but they are unclear (seems to be a good start):

    G8_B8R8_2PLANE_420_UNORM,
    G10X6_B10X6R10X6_2PLANE_420_UNORM_3PACK16,
    G16_B16R16_2PLANE_420_UNORM,

Question 3

No D3D11 support? Probably not needed, but, IM, nice to have :)

(...to be continued)

[Daedie-git]

Q1
I'll get back to you on this.

Q2:
How about this

NV12_UNORM
P010_UNORM
P016_UNORM

Q3
D3D11 video semantics work very differently. So regardless if it is added or not, I would do that in a separate PR.

[dzhdanNV]

Q2

+1, clarifying comments may be added if needed. They are pretty standard.

Q3

No objections here (assuming the interface is implementable in D3D11)

[dzhdanNV]

Do we need other formats, like DXGI_FORMAT_Y410 (10:10:10)?

[dzhdanNV]

Please, add this link somewhere in the beginning of NRIVideo.h: https://learn.microsoft.com/en-us/windows/win32/medfound/recommended-8-bit-yuv-formats-for-video-rendering

[dzhdanNV]

I'm ready to merge it after you are done with polishing. Just let me know. Further improvements can be made in main. Can we create a minimalistic sample in https://github.com/NVIDIA-RTX/NRISamples for this?

I haven't deeply analyzed the implementation, but the API additions are nice and clear.

Also I think that these formats:

    // Depth-stencil (as a shader resource view)
    R24_UNORM_X8,       // .x - depth   // + . . . . . . . . . . . . . . .
    X24_G8_UINT,        // .y - stencil // + . . . . . . . . . . . . . . .
    R32_SFLOAT_X8_X24,  // .x - depth   // + . . . . . . . . . . . . . . .
    X32_G8_UINT_X24     // .y - stencil // + . . . . . . . . . . . . . . .

can be removed and readonlyPlanes can be used instead explicitly. readonlyPlanes are needed for RTV and actually for SRV (you already use it for video). It's a bit of a breaking change, but, IMO, a step towards better API. What do you think? Just asking for your opinion, I will implement it myself.

[Daedie-git]

I'm ready to merge it after you are done with polishing. Just let me know. Further improvements can be made in main. Can we create a minimalistic sample in https://github.com/NVIDIA-RTX/NRISamples for this?

Definitely. I'll look into it as soon as I can.

I haven't deeply analyzed the implementation, but the API additions are nice and clear.

Also I think that these formats:

    // Depth-stencil (as a shader resource view)
    R24_UNORM_X8,       // .x - depth   // + . . . . . . . . . . . . . . .
    X24_G8_UINT,        // .y - stencil // + . . . . . . . . . . . . . . .
    R32_SFLOAT_X8_X24,  // .x - depth   // + . . . . . . . . . . . . . . .
    X32_G8_UINT_X24     // .y - stencil // + . . . . . . . . . . . . . . .

can be removed and readonlyPlanes can be used instead explicitly. readonlyPlanes are needed for RTV and actually for SRV (you already use it for video). It's a bit of a breaking change, but, IMO, a step towards better API. What do you think? Just asking for your opinion, I will implement it myself.

I like that.

[Daedie-git]

Do we need other formats, like DXGI_FORMAT_Y410 (10:10:10)?

The current set targets the common 4:2:0 format family needed by the decode/encode path: NV12, P010, P016. Y410 is packed 4:4:4 and would add support for it in a follow-up if the need arises.

[vertver]

I don't like that you're using native API structures in NRI public API. Wouldn't be logical to implemented it as a translation layer, not in the NRIWrapper* way?

[dzhdanNV]

I like that.

@Daedie-git I have implemented it. Please, update your code to the latest since it's a prerequisite for the video support. Please, note that I have removed PR #222 since the code is completely different for these lines. Feel free to adjust, re-add or discuss. Thanks in advance! We are moving forward!

I don't like that you're using native API structures in NRI public API. Wouldn't be logical to implemented it as a translation layer, not in the NRIWrapper* way?

Just echoing already said, I absolutely agree with this. GAPI specific structs must be moved to corresponding NRIWrapperX extension and NRIVideo must get an abstraction. This is what we should focus on.

[Daedie-git]

I like that.

@Daedie-git I have implemented it. Please, update your code to the latest since it's a prerequisite for the video support. Please, note that I have removed PR #222 since the code is completely different for these lines. Feel free to adjust, re-add or discuss. Thanks in advance! We are moving forward!

I don't like that you're using native API structures in NRI public API. Wouldn't be logical to implemented it as a translation layer, not in the NRIWrapper* way?

Just echoing already said, I absolutely agree with this. GAPI specific structs must be moved to corresponding NRIWrapperX extension and NRIVideo must get an abstraction. This is what we should focus on.

Indeed. I have addressed this locally already. I'll rebase it on your change and push it soon.

[Daedie-git]

I wrapped up my changes.

[Daedie-git]

I found some issues, so I will follow up with another commit.

[Daedie-git]

Additional question: For the sake of testing this, I've been building up quite the test suite in my own repo. Do you want to have it in some form?
Apart from the samples you already asked for.

[dzhdanNV]

Thanks! It's time to finally merge it \O/