loader: Sort adapters by type on non-Windows platforms

[misyltoad]

On non-Windows platforms, where we don't use DXGI to determine the preferred adapter, order adapters by type.

Otherwise, they get order by whoever was written last in the filesystem which makes behaviour unreliable and inconsistent for users when certain apps make bad assumptions about the first adapter.

This also avoids the problem with software implementations being chosen by applications that prefer the first adapter.

Signed-off-by: Joshua Ashton [email protected]

[misyltoad]

Once/if this gets merged, I would also like to discuss extending this to add an environment variable to chose the preferred type for sorting, as well as one to filter to only one device being exposed. These vars could then be used by a launcher application or the user directly.

Currently, it is required to use an external layer (some of which can be very broken) to achieve this behaviour which I think should be unnecessary given it is needed for users in Multi-GPU or PRIME setups.

[pdaniell-nv]

I don't have an objection to this in principle, but I do have a couple of observations:

1. Will there still be instability within a type. For example with a system with more than one dGPU, which comes first might change.
2. I think the sorting should be applied not just for non-Windows, but also for times where ReadSortedPhysicalDevices() did not sort the devices because of missing driver support (no EnumerateAdapterPhysicalDevices) or missing/failing DXGI (ie Windows 7, or old Windows 10).

[misyltoad]

1. Is definitely true, how should we sort within a type?

• We probably don't want to do the api or something in case the user just hasn't updated their driver as it would be inconsistent over time.
• Alphabetical by device name would be at least consistent for users no matter what but doesn't make much logical sense.

Any other ideas?

[HansKristian-Work]

Sort by ICD json timestamp perhaps. Sorting by name would favor one IHV over the other in some cases.

[misyltoad]

That's essentially what is happening right now, and it leads to inconsistent behaviour after updating so I would be against it.

[pdaniell-nv]

Perhaps the sort can be based on the physical ordering of the adapters on the PCI bus - ie from VkPhysicalDevicePCIBusInfoPropertiesEXT. The downside is that I'm not sure if everyone implements that extension.

[misyltoad]

I really like that idea @pdaniell-nv.

Seems to be supported by AMD (RADV and Prop), Intel and Nvidia on Windows and Linux.

http://www.vulkan.gpuinfo.org/listdevicescoverage.php?extension=VK_EXT_pci_bus_info&platform=linux

If there isn't support, we could just fallback and put them at the bottom of the list alphabetically or something. (I guess SW implementations and maybe virtual GPUs would always fall in this path, but at least it's then consistent.)

That way we have something logical and consistent for the ordering in the primary case, and something just consistent in the edge case. Feel free to suggest something better for the latter case too :p

[lenny-lunarg]

I like the basic principle of at least making the order consistent. A couple of thoughts:

If there isn't support, we could just fallback and put them at the bottom of the list alphabetically or something.

Yeah, I don't think which method you choose as a fallback is especially important, but we definitely need to have something specified. There's always someone who ends up using an ancient driver and a new loader. It's a rare case, but it happens.

I think the sorting should be applied not just for non-Windows, but also for times where ReadSortedPhysicalDevices() did not sort the devices because of missing driver support (no EnumerateAdapterPhysicalDevices) or missing/failing DXGI (ie Windows 7, or old Windows 10).

I'd be tempted to say we should just use the same fallback logic as on Linux. From a practical standpoint I think we're unlikely to see enough GPUs to make it really matter, but if it does matter we'd be sorting discrete before integrated with the new ICD interface, then discrete before integrated with the old interface, then a fallback case. That just seems unintuitive to me. To make it 100% consistent we still need a fallback case for when the new ICD version isn't available and neither is VkPhysicalDevicePCIBusInfoPropertiesEXT.

As for this exact PR, I'm happy to take something along these lines. It sounds to me like I should wait a little until we decide exactly what seems best. When we agree I can mark this approved and merge it.

[lenny-lunarg]

To clarify, @Joshua-Ashton are you planning on making some of the changes outlined here? I can do it myself if needed, but I have enough on my plate that it's going to be a couple of weeks, at least. I wasn't planning on marking it as approved because it sounded from the conversation like people wanted to make a couple of changes.

[jjulianoatnv]

On old Win10 that predates the DXGI support for heuristic-driven sort order, I think it makes sense to use the order from D3DKMTEnumAdapters2. I expect (but haven't verified) that this is the same order that DXGI enumerates when not using the heuristic-driven sort order.

For linux, where there is no kernel module arbitrating sort order, using bus enumeration order is stable, and therefore seems like a reasonable idea. At a future point the sort order can switch to a heuristic-driven sort order when that actually exists in linux.

Be aware that there exist apps that loop over physical devices and stop at the first-enumerated device that has some attribute(s). There also exist apps that loop over physical devices, don't early-terminate the loop, and therefore use the last physical device that has some attribute(s). Both types of apps might effectively choose a different physical device after the sort order is changed, and therefore might regress in functionality. Such an app is already fragile in the presence of multiple physical devices, and even more fragile in the presence of physical devices using different ICDs. I do not believe that we should compromise future improvement by making a requirement that these apps continue working as-is. Maybe a "compatibility layer" can be created that detects known apps of this type, and uses the new Vulkan API to force such an app to a reasonable choice of physical device. Such a compatibility layer is better if it comes from Khronos rather than from any one particular IHV.

[misyltoad]

There is nothing to make 'compatible' right now, the order on Linux right now is entirely who made the last file, which changes between updates and package installs.

As much as I really like keeping the PCIe order, I am starting to think it may be a bad idea for one of several reasons:

• The driver would need to support the extension, which is really annoying to query at sorting time
• Not all devices may even have a PCIe ID or one that may be stable (CPU devices, Virtual GPUs, something not using PCI)

I think keeping it simple (and therefore entirely stable) may be our best bet going forward, at least on Linux. And just ordering via 3 brackets and alphabetically within those brackets.

[jjulianoatnv]

On Windows, we have a "default" GPU, we have a UI for changing the default for specific apps. Each app is free to use its default, or to explicitly select GPU based on its own criteria.
Adding similar capability to linux seems like a good idea.

Sorting the VkPhysicalDeviceList is Vulkan's "poor-man's" approach to indicating to the app which VkPhysicalDevice is the app's default. I personally dislike this implicit messaging, and would prefer an explicit messaging like is done in DX. But Vulkan chose implicit messaging on Windows, and the same implicit messaging could apply to Linux if Linux also adopted sorting as a method of communicating the app's default GPU.

After the first GPU in the sorted list, the way that the remaining GPUs are sorted is not well-defined. Nor is it particularly meaningful. I'm personally less interested in defining how the tail is sorted than I am in making the "sorted first" attribute have the same meaning on Linux as it has on Windows.

[misyltoad]

I don't think we have anything comparable to that right now and I'm not super sure how we'd make that work effectively without the same LUID system for adapters.

We could use the PCIe Bus EXT but that excludes all CPU (and potentially some vGPU) devices from being able to use this.

I'd definitely like something in this space by default with the loader on Linux however.

[jjulianoatnv]

My opinion is that the Khronos group should drive adding an appropriate identifier to linux. Something analogous to AdapterLUID that is assigned by the linux kernel.

Then, separately, effort can be made to build and expose UX in usermode. And separately, the Vulkan and OpenCL loaders can be connected to this thing to discover which GPU was selected by the user as the per-app (or global) default.

[cubanismo]

FYI, NVIDIA proposed a library to handle sorting/selecting GPUs consistently across GL and Vulkan apps on Linux, especially in render offload-style systems:

https://gitlab.freedesktop.org/glvnd/libglvnd/-/merge_requests/224
https://gitlab.freedesktop.org/glvnd/libglvnd/-/merge_requests/228

It's been languishing for about a year due to lack of feedback/interest from other vendors.

[lenny-lunarg]

So here's my thoughts on this. I think this is a good start, but I'd like to change it a little to make it so that the order is deterministic. I'm not exactly sure which of the means listed above I'll do, but I think I'm going to play around with them a little and see how they work out. I'll pick whatever I like best and push it up to this PR (in addition to the current change) when I have something I like. I'll do this over the coming days. If people have specific thoughts, feel free to chime in in the meantime.

[MarkY-LunarG]

I'm going to throw some ideas out I think would be better and I'd like all your input:

1. Have multiple options configurable by Environment variable and maybe a new loader settings file:
   a) Prefer performance
   b) Prefer integrated
   c) Auto (on battery -> integrated, plugged in performance)
2. Output the results of the last run into the new loader settings file and that will be used next time.
3. Publish the details of the settings file and let users adjust the order of their GPUs.
4. If we really got fancy, we could even allow some sort of mechanism to identify how users wanted their devices sorted (based on type and then amount of memory, or size of textures, etc). But that's getting fancy.

It's a little rudimentary and will take some work. I, personally, like the direction @cubanismo was trying to go and I wish other IHVs would just chime in there. But for now, we can try to get something working for Linux/Mac and then even fallback to the same behavior on windows when the "new windows method" just barfs.

[kbrenneman]

In the loader itself, I agree that the order should be deterministic, but beyond that, keeping it as simple as possible seems like a good idea. More complicated sorting policy ought to go into a layer, where it's easier to modify or replace or disable. A layer also can get away with having dependencies that we wouldn't want for the loader proper.

Picking a default device (i.e., whatever goes at the start of the list if the user doesn't otherwise specify something) is a good example of that. You might have some heuristics to guess which windowing system the application is going to use, but (I assume) we wouldn't want the loader to depend on libX11 or libwayland-client.

As a side note, one interesting idea I heard is to define a D-Bus interface to query a default device. On typical desktop Linux systems, having two windowing systems (Wayland + XWayland) will probably become the rule rather than the exception, but you'd still have only one session bus.

[MarkY-LunarG]

I'm going to close this because I'm working on something and I created issue #657 to track this. My approach is to allow users more control and to be able to prioritize driver type as well.

[MarkY-LunarG]

New PR is #764. This sorts by PCI bus, but prioritizes the device it detects is attached to the primary monitor above others.