Add Vulkan rendering backend

[laanwj]

Implement a Vulkan 1.1 renderer that replaces the previous stub with a fully functional backend, mostly matching the OpenGL backend's rendering capabilities. The game should be playable with minimal divergence from OpenGL rendering.

This is, most likely, too big to go in all at once, but just filing it here for reference because it's reached a testable state.

Core rendering infrastructure. The code lives under code/graphics/vulkan:

• VulkanMemory: Custom allocator with sub-allocation from device-local and host-visible memory pools
• VulkanBuffer: Per-frame bump allocator for streaming uniform/vertex/index data (persistently mapped, double-buffered, auto-growing)
• VulkanTexture: Full texture management including 2D, 2D-array, 3D, and cubemap types with automatic mipmap generation and sampler caching
• VulkanPipeline: Lazy pipeline creation from hashed render state, with persistent VkPipelineCache
• VulkanShader: SPIR-V shader loading (main, deferred, effects, post-processing, shadows, decals, fog, MSAA resolve, etc.)
• VulkanDescriptorManager: 3-set descriptor layout (Global/Material/PerDraw) with per-frame pool allocation, auto-grow, and batched updates
• VulkanDeletionQueue: Deferred resource destruction synchronized to frame-in-flight fences

Design choices:

• Two frames in flight with fence-based synchronization
• Asynchronous texture upload, no waitIdle or other CPU-on-GPU blocking in hot path
• Single command buffer per frame; render passes begun/ended as needed for the multi-pass deferred pipeline
• Per-frame descriptor pools
• All descriptor bindings pre-initialized with fallback resources (zero UBO + 1x1 white texture) so partial updates never leave undefined state
• Streaming data (such as immediates) uses a bump allocator (one large VkBuffer per frame)
• Pipeline cache persisted to disk for fast startup on subsequent runs

Some notable Vulkan vs OpenGL differences are:

• Because shaders are pre-compiled to SPIR-V, shader variants are less feasible in Vulkan. Preprocessing directives have been converted to run-time uniform based branching.
• Depth range is [0,1] not [-1,1]: shadow projection matrices adjusted, shaders that linearize depth need isinf/zero guards at depth boundaries where OpenGL gives finite values
• Vulkan render target is "upside down", y-flip for render target is handled through negative viewport height, as is common
• gl_ClipDistance is always evaluated: must write 1.0 when clipping is disabled (OpenGL allows leaving it uninitialized)
• Texture addressing for AABITMAP/INTERFACE/CUBEMAP forced to clamp (OpenGL's sampler state happens to do this implicitly)
• Render pass architecture requires explicit transitions between G-buffer, shadow, decal, light accumulation, fog, and post-processing passes (OpenGL just switches FBO bindings)
• No geometry shaders. They're possible with Vulkan, but less common. Currently they're not used.

Preparation patches to common game code (these commits need to go in first):

• Extract sphere and cylinder mesh generation into shared graphics utility: Needed in both GL and Vulkan
• Route ImGui calls through gr_screen function pointers: Makes it possible for the Vulkan backend to provide its own ImGui implementation
• Free bitmaps before destroying graphics backend: Fix shutdown order issue
• Use float shader input instead of SCREEN_POS in gr_flash_internal: Compatibilty with Vulkan shaders
• Remove now-unused SCREEN_POS vertex format: Cleanup after previous commit
• Add dds_block_size and dds_compressed_mip_size utilities: Factor out utility code to be used in Vulkan backend
• Add CAPABILITY_QUERIES_REUSABLE for GPU queries: Vulkan needs different lifecycle for GPU queries
• Fix gr_flip debug output ordering: Prevent immediate buffer from being overwritten
• Fix gr_end_2d_matrix viewport for render-to-texture: Fix RTT for Vulkan
• Fix undefined gl_ClipDistance and use uint for std140 bool: Shader compatibility with Vulkan
• Fix shader build MAIN_DEPENDENCY and add conditional GLSL/struct generation: Build system change for OpenGL/Vulkan shader split
• Add missing memcpy_if_trivial_else_error for void *, const void*

What's possibly left to be done:

• Unify OpenGL and Vulkan shaders where possible: the only shader shared with OpenGL (defined in the buid system's SHADERS_GL_SHARED) is still the default material. Although the Vulkan backend does some things differently, it would definitely be possible to share more code. But i didn't want to accidentally break OpenGL in some way.

• Integrate VMA (Vulkan Memory Allocator). Some of the memory handling could be simplified by importing this dependency.

• OpenXR anything. This is currently not implemented at all.

Build steps:

cmake -B build -DCMAKE_BUILD_TYPE=Debug -DFSO_BUILD_WITH_VULKAN=ON -DFSO_BUILD_WITH_OPENXR=OFF -DSHADERS_ENABLE_COMPILATION=ON 
cmake --build build

To run (with maximum debugging and Vulkan layer validation):

export VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation
export VK_LOADER_DEBUG=all

build/bin/fs2_open_25_1_0_x64_AVX2-DEBUG -vulkan -gr_debug -stdout_log -profile_frame_time

Full disclosure: i used Claude Opus 4.6 while developing this. However, the overall direction and design is my own, and i've paid careful attention to the code.

[BMagnu]

Thanks for the PR!
I'll be looking at it and playing around with it soon.
Please be aware, this being as big as it is, that it might be a while until we get through it.

[BMagnu]

Okay, played around with it a little.
Got it to run, though with a slew of visual artifacts and some crashes on some mods.
Still, a great first step to see it running in vulkan, at quite impressive performance numbers.
I'd love to discuss some of the design decisions in more detail. Are you on the discord, or somewhere else sensible for extended discussion?