Split RenderingDevice into API-agnostic and RenderingDeviceDriver parts

[RandomShaper]

A huge chunk of RenderingDeviceVulkan is API-agnostic bookkeeping, validation, etc. Such code is also shared with others, like RenderingDeviceD3D12. This PR keeps a single, non-abstract RenderingDevice with all those API-agnostic elements and moves all the API-specific parts to a thin wrapper around the driver, which is RenderingDeviceDriver for Vulkan. This makes much easier to write additional rendering drivers, by allowing one to focus on the API-specific stuff. In some cases that will require modifying the interface between RD and RDD, which is not exposed to the user.

Other relevant points:

• A common context interface has been extracted from VulkanContext (APIContextRD) with the bare minimum set of services RenderingDevice and direct usage needs. (The specific RenderingDeviceDrivers have access to the specific context implementation, which open doors for them to a much richer, but non-standard, API.) UPDATE: Now Direct3D 12 has been merged, it gets similar love in this PR.
• There are important notes on design choice in the code. Future maintainers, please follow or discuss, but don't break consistency.
• Every struct involved now have default initializers, both in RenderingDevice and RenderingDeviceDriver.
• Some of the wiring between context and RD has been enhanced so now you can instantiate a RenderingDevice even when Godot is running in compatibility mode (this will allow to have a RD-based lightmapper available in a GL compatibility project). This piece of code shows how to ensure you have a usable local RenderingDevice in any circumstances:

  RD *orig_rd_singleton = RD::get_singleton();
  
  ApiContextRD *vc = nullptr;
  if (GLOBAL_GET("rendering/renderer/rendering_method") == "gl_compatibility") {
  #ifdef VULKAN_ENABLED
  	if (GLOBAL_GET("rendering/rendering_device/driver") == "vulkan") {
  		vc = memnew(VulkanContext);
   }
  #endif
  	CRASH_COND_MSG(!vc, "No RD driver available.");
  	vc->initialize();
  } else {
  	vc = RD::get_singleton()->get_context();
  }
  
  RenderingDevice *rd = memnew(RenderingDevice);
  if (GLOBAL_GET("rendering/renderer/rendering_method") == "gl_compatibility") {
  	DEV_ASSERT(RD::get_singleton() == rd);
  }
  rd->initialize(vc, true);
  // Do stuff with RD here.
  memdelete(rd);
  
  if (GLOBAL_GET("rendering/renderer/rendering_method") == "gl_compatibility") {
   	memdelete(vc);
  }
  DEV_ASSERT(RD::get_singleton() == orig_rd_singleton);

• Deprecations: RD::texture_get_native_handle() (superfluous) and DRIVER_RESOURCE_VULKAN_* (replaced by API-agnostic).

This includes a couple of merged (commits will go away once rebased) and unmerged (#82797) PRs.

TODO:

• Finish debugging and fix a few known issues.
• Rebase on top of the lastest master.
• Fix ligthmapper (local device?).
• D3D12: Adapt to this new infrastucture. 2023-12-11: The PR now includes Add Direct3D 12 rendering driver (Mesa NIR approach) #70315, which is then adapted to the new model.
• Debug
• Remove the interim change-tracking-assistance commit.
• Ensure [D3D12] Segfault when importing GDQuest Third Person Controller demo #86106 is fixed.

Production edit: closes godotengine/godot-roadmap#31

[reduz]

Looks fantastic!

[DarioSamo]

I'll be leaving feedback in general here and on a per-line basis if necessary. Some of my comments are just criticizing a few design carry-overs from the current design and aren't necessarily related to this PR being approved or not, it just feels like the right time to bring these points up.

Type Safety

I already brought this up over DMs, but there's currently no type safety system for the IDs returned by RDD. Using the uint64_t's directly means we can mix and match arguments around and the compile won't indicate the error. We can solve this in a few ways.

A) Every type returned gets a struct that wraps around the uint64_t. A very minor change that guarantees the types can't be mixed together.

struct FramebufferID {
  uint64_t handle;
};

B) Every type returned is a pointer to an interface that the backend has to inherit. This requires more work but since these handles are already pointers in disguise (they're not references you can indefinitely hold), there's not much of a downside to it. The major advantage is that debuggers can access these types natively and display their contents.

struct Framebuffer {
  // Empty interface.
};

virtual Framebuffer *framebuffer_create() = 0;
virtual void framebuffer_free(Framebuffer *p_framebuffer) = 0;

struct FramebufferVK : Framebuffer {
  VkFramebuffer fb = VK_NULL_HANDLE;
}

Another benefit to consider is const-correctness can be implemented into the pointer types and will therefore be directly indicated by the API whether the object's internals should be modifiable or not.

Vertex Formats

There seems to be currently a forced correlation between bindings and attributes, meaning that attributes that share the same buffer can't share the same binding (e.g. if 10 elements are bound to the same buffer, they each get their own binding instead of using the same one). I'm not familiar if this has potential performance implications, but it's worth investigating if this is correct.

This is not new to this PR, as it was pretty much directly copied as is from the current Vulkan RD.

Command Queues

Command queues are not directly exposed but managed by the backend, meaning RenderingDevice has no control whatsoever on where commands are executed, on what type of queue and when exactly. This could prove to be very useful in the feature as a lot of the logic around managing command queues should be common to most backends and it takes off some responsibility for the backend maintainer. I think we could address this in this PR and move most of the logic that Vulkan Context currently handles to RD instead.

I'll do a separate comment for concerns about D3D12 compatibility in the future.

[DarioSamo]

These are not something that necessarily need to be solved by this PR but are probably worth considering before doing the D3D12 version of RDD. Some are really just points to consider when doing the D3D12 refactor.

Combined Image Samplers

As far as I know these are not supported by D3D12 and the backend would need to work around it in some ugly ways to support them. Are we relying on this feature a lot? What approach did the D3D12 PR take to solving this?

Compute dispatch base

I'm not aware D3D12 has anything about this. This is a new addition to the API by this PR. We might want to consider it more carefully. It's also a Vulkan 1.1 feature at minimum, so we need to query device support for it. I think we should just remove it and let the user handle it by adding the offset in a UBO/Push constant.

Barriers

D3D12 requires resource barriers if not using enhanced barriers. I think we should take the chance to make our barrier-API only take resources instead. I don't think we ever rely on memory barriers. As far as I'm aware the current D3D12 PR needs to work around this and I think it's hard to find use cases for full memory barriers with the upcoming acyclic render graph/barrier solver design.

Semantics vs Locations

What approach does the current D3D12 PR take for matching vertex location inputs to the semantics? Is that something that we'd need to address on this API or is it solved through some other means as part of the backend responsibility? If there's a possibility this API will take shaders in its native format, it sounds like we might need to allow for a type of configuration that allows input semantics.

Unordered Access State

What is the suggested solution for mapping this API to the unordered access state in D3D12? Using the image layout for general and the bit for shader writes?

Texel Buffers

These don't translate to a flag during creation in D3D12, so we'll just need to track that bit to enable using the format on the SRV/UAV when it's assigned.

[RandomShaper]

You may want to check the D3D12 PR (#70315) to see how I mapped the RenderingDevice features to it.

Regarding barriers, the D3D12 driver using resource-level barriers would just ignore the detailed ones (that's exactly what the PR above is doing). If enhanced barriers are eventually implemented, well, it just have to obey them.

I'm addressing the type safey of ids in the next push.

Regarding other things, this is just the first iteration and will for sure need changes to fit other APIs or future RD features better.

[RandomShaper]

Regarding type safety, I'm taking the first approach you discussed because most RDD ids aren't actually pointers to its view of resources, but just the Vulkan handles.

[DarioSamo]

For the TODO: Lightmapper is broken and doesn't start, which probably means we need to review local_device implementation.

Validation layer seems to indicate it's related to the Vulkan instance.

ERROR: VALIDATION - Message Id Number: 1190443409 | Message Id Name: VUID-VkSubmitInfo-commonparent
        Validation Error: [ VUID-VkSubmitInfo-commonparent ] Object 0: handle = 0x29cee429b20, type = VK_OBJECT_TYPE_IN)        Objects - 1
                Object[0] - VK_OBJECT_TYPE_INSTANCE, Handle 2873035496224

[RandomShaper]

Re-pushed. Lightmapper is fixed (or so I think).

[RandomShaper]

It seems to be a GPU driver or OS version specific issue. Works as expected on some setups, fails on another. Maybe a driver bug. We are fine with merging as it is now. The code seems to be OK.

[RandomShaper]

This is never ending, is it? 😅 Pushed with a fix for the memory leaks.

[RandomShaper]

I think I'm going to force-push again...

Just kidding. 😈

[YuriSizov]

Thanks! As often is the case with such achievements, this monumental effort is not the end, but a beginning for even more exciting new things to be added into the engine in near and far future. But tonight you can rest with a proud smile. Cheers to everyone who contributed and tested this PR!

[Cyangmou]

Really happy to see this, just wanted to say thanks for the effort and can't wait for future improvements this will enable!

[aaronfranke]

This does not compile on macOS, it fails on the linking stage (I tested both arm64 and x86_64):

[100%] Linking Program bin/godot.macos.editor.arm64 ...
Undefined symbols for architecture arm64:
  "_vkCmdDrawIndexedIndirectCount", referenced from:
      RenderingDeviceDriverVulkan::command_render_draw_indexed_indirect_count(RenderingDeviceDriver::CommandBufferID, RenderingDeviceDriver::BufferID, unsigned long long, RenderingDeviceDriver::BufferID, unsigned long long, unsigned int, unsigned int) in libdrivers.macos.editor.arm64.a(rendering_device_driver_vulkan.macos.editor.arm64.o)
  "_vkCmdDrawIndirectCount", referenced from:
      RenderingDeviceDriverVulkan::command_render_draw_indirect_count(RenderingDeviceDriver::CommandBufferID, RenderingDeviceDriver::BufferID, unsigned long long, RenderingDeviceDriver::BufferID, unsigned long long, unsigned int, unsigned int) in libdrivers.macos.editor.arm64.a(rendering_device_driver_vulkan.macos.editor.arm64.o)
ld: symbol(s) not found for architecture arm64

[clayjohn]

This does not compile on macOS, it fails on the linking stage (I tested both arm64 and x86_64):

[100%] Linking Program bin/godot.macos.editor.arm64 ...
Undefined symbols for architecture arm64:
  "_vkCmdDrawIndexedIndirectCount", referenced from:
      RenderingDeviceDriverVulkan::command_render_draw_indexed_indirect_count(RenderingDeviceDriver::CommandBufferID, RenderingDeviceDriver::BufferID, unsigned long long, RenderingDeviceDriver::BufferID, unsigned long long, unsigned int, unsigned int) in libdrivers.macos.editor.arm64.a(rendering_device_driver_vulkan.macos.editor.arm64.o)
  "_vkCmdDrawIndirectCount", referenced from:
      RenderingDeviceDriverVulkan::command_render_draw_indirect_count(RenderingDeviceDriver::CommandBufferID, RenderingDeviceDriver::BufferID, unsigned long long, RenderingDeviceDriver::BufferID, unsigned long long, unsigned int, unsigned int) in libdrivers.macos.editor.arm64.a(rendering_device_driver_vulkan.macos.editor.arm64.o)
ld: symbol(s) not found for architecture arm64

It compiles in the CI. Are you using any different parameters that would have an appreciable difference?

Molten VK appears to have support for both those functions, so it is odd that it's not linking

[aaronfranke]

@clayjohn Fixed, the problem was that I was using an old version of MoltenVK. If anyone else is experiencing this issue, try updating the Vulkan SDK.