Automatic porting of libretro shaders to mpv

[hhirtz]

Hello!

I would like to showcase mpv-libretro a project I made to convert libretro shaders (.slangp preset files & associated .slang source files) to the format mpv --glsl-shader expects.

Here's a presentation of the problem as well as some difficulties i encountered along the way.

(disclaimer: i'm still learning about shaders, some things i say here might be wrong)

Motivation

I wanted to watch Tool-Assisted Speedruns of retro games. These speedruns are encoded in the resolution of the actual console (e.g. 256x224 pixels).

While using --scale=nearest is fine to a degree, libretro offers better upscaling filters for this type of content, e.g. emulating old CRT screens.

However libretro shaders aren't compatible with mpv as-is, you need to transform them to use them in mpv, which is what mpv-libretro does.

Screenshots

Here are some screenshots at 1440p of some speedruns:

1. The Legend of Zelda: A Link to the Past
2. Super Mario 64
3. Super Metroid

On the left, --scale=nearest is used. On the right, --glsl-shader=crt-royale-fb-intel.glsl is used (CRT Royale, fake bloom, intel preset), which has been ported from these sources: https://github.com/libretro/slang-shaders/blob/6cb93ce1be67b314a96f5e346befb2b58b63eb04/crt/crt-royale-fake-bloom-intel.slangp.

--scale=nearest	--glsl-shader=crt-royale-fb-intel.glsl

Usage

The sources are there: https://github.com/hhirtz/mpv-libretro

I don't have CI setup so it needs to be built from source. It's a rust project and as far as i can tell, only requires (an up-to-date enough) Rust to build.

The tool must be called from the command line, with the shader preset file as first and only argument. It prints the mpv-compatible GLSL shader to standard output, so you need to call it like so:

cargo run -- crt-geom-ntsc-composite-sharp.slangp >crt-geom.glsl

The output shader uses //!PARAM directives as well as linearize/delinearize, so you need --vo=gpu-next to use it, e.g.:

mpv --vo=gpu-next --glsl-shader=crt-geom.glsl ultimatedoom-tas-episode4-akse.mkv

The video must be of correct size. If it's already been upscaled, you need to downscale (and possibly crop) it first. This can be done in different ways:

• with ffmpeg prior to starting mpv, see: https://trac.ffmpeg.org/wiki/Scaling
• in realtime using mpv --vf=..., see https://github.com/hhirtz/mpv-retro-shaders/blob/073159d5465979f902cfe2147605e69de496dd05/README.md#usage-with-upscaled-videos

The correct size depends on the game and console.

How it's done and some difficulties

libretro shaders are organized this way: a shader preset file (.slangp extension) lists all the shader passes and external textures (PNG and JPEG files), as well as some metadata for each pass.

Each shader pass is in its separate .slang file.

For some reason libretro allows shader authors to define their own vertex shaders (this might be for 3D games?). The structure of each .slang file is thus as follows:

#version XX

// common code...
// this is where shader inputs are defined, as uniforms.
// as well as tweakable shader parameters

#pragma stage vertex

// code specific to the vertex shader
void main() {
    // ....
}

#pragma stage fragment

// code specific to the fragment shader
void main() {
    // ....
}

The conversion is done in several steps:

1. librashader loads and parses the .slangp shader preset, to list all the passes, external textures and shader parameters. It also separates each .slang file into two (for the vertex and fragment shaders).
2. All shader parameters are added as //!PARAM as per libplacebo shader format. Both mpv's and libretro's parameter spec closely match so it's been easy to convert them
3. All textures are added in as //!TEXTURE, with a //!FORMAT rgba8, as I found is compatible with most hardware? All texture metadata (except mipmap, see limitations) are also added
4. Using a rust glsl-lang parser and transpiler, each vertex+fragment shader couple is merged into one shader, and is added as a //!HOOK MAIN pass. This has been by far the hardest step to do.
5. A preliminary pass is added to convert mpv's MAIN texture from sRGB to linear RGB. libretro shaders work with linear RGB, and I think mpv converts the different color types to sRGB only, there is no way to say a shader works in linear RGB only.
6. similarily, the last pass is made to do the conversion back to sRGB

The reason the vertex and the fragment shaders have to be merged is that it turns out a lot of libretro shaders do interesting stuff in the vertex stage (other than gl_Position=MVP*Position), so it cannot be just discarded. So we prefix all globals with vertex_ and fragment_, merge the two Abstract Syntax Trees, and add the hook function:

vec4 hook() {
    vertex_main();
    fragment_main();
    return FragColor;

    // or, for the last pass:
    return delinearize(FragColor);
}

Some notes:

• shader input and outputs (layout(set=..) in/out) are made into simple global variables that are shared between the two stages.
• uniform blocks are made into structs, and initialized accordingly (those contain texture sizes, the frame count, and shader parameters). The list of dependencies to //!BIND for each pass is built with this, as well as the samplers defined in the libretro shader.
• MVP is set as mat4(1.0), and Position is set as vec4(HOOKED_pos, 0.0, 1.0).

Here is a list of libretro shaders ported to mpv:
https://github.com/hhirtz/mpv-retro-shaders

With the exception of crt-guest-advanced-ntsc.glsl, crt-lottes.glsl and gba.glsl, all of these shaders have been ported using this method.

Limitations

Naming issues aside, the two shader systems are fairly similar. Doing the conversion between the two has been actually pretty straightforward. The mpv docs and libplacebo website helped a lot. However, some advanced features in libretro are not present in mpv and prevent the correct conversion of some shaders:

• libretro shader presets are allowed to define metadata for each pass that is not available in mpv (see slangp format notes):
  • wrap_mode, basically //!BORDER <CLAMP | REPEAT | MIRROR> in mpv, but not limited to external textures
    e.g. https://github.com/libretro/slang-shaders/blob/6cb93ce1be67b314a96f5e346befb2b58b63eb04/crt/crt-royale.slangp#L203
  • filter_linear, basically //!FILTER <LINEAR | NEAREST> in mpv, but not limited to external textures
    e.g. https://github.com/libretro/slang-shaders/blob/6cb93ce1be67b314a96f5e346befb2b58b63eb04/crt/crt-royale.slangp#L160
  • mipmap_input, float_framebuffer and srgb_framebuffer, which i actually am not sure what they do. mpv does not offer such settings
• libretro shaders can sample earlier frames of the input video. This could be possible to workaround in mpv (with //!BUFFER, but the size of the buffer has to be hardcoded). see Shader stage expansion for temporal shaders/algorithms? #8137
• they can also sample their own output from earlier invocations. Also possible to work around with the same limitations. see Shader stage expansion for temporal shaders/algorithms? #8137

Overall I'm very happy with mpv's shader system and the software in general, and wanted to contribute back a little. I will go back to enjoy some nice retro speedruns! Thank you to everyone involved in the project.

[haasn]

This is seriously amazing, mad props for going to all this effort. It's also a bit sad, though, as I really dislike the mpv shader system and want to kill it, but I haven't yet found the time to rework the shader system from scratch.

One of the numerous things we could do to improve the situation is to integrate slang directly into mpv/libplacebo. I noticed you mentioned librashader, which I was not aware of. It seems like it would be relatively straightforward to hook that up directly into our system. Custom vertex shaders are not a big deal, we can definitely find a way to handle them - definitely much more easily than having to parse the AST as you do currently.

I think it would make much more sense to use librashader to just parse the shaders and execute them with our existing dispatch/gpu infrastructure; rather than needing to somehow interop librashader's built-in execution backends with our own contexts. What do you think?

A preliminary pass is added to convert mpv's MAIN texture from sRGB to linear RGB. libretro shaders work with linear RGB, and I think mpv converts the different color types to sRGB only, there is no way to say a shader works in linear RGB only.

You can hook LINEAR, which is like MAIN, but in linear light. I guess you did not do this because LINEAR is not resizable? Maybe we can fix that.

[hhirtz]

Thank you! I don't know why as a mpv developer you dislike the shader system, but from having ported some of the libretro shaders by hand, i agree. It becomes cumbersome to navigate the source as the complexity grows. Especially with inline textures and the lack of #include. Shaders written in slang don't have this problem.

librashader has been very straightforward to work with. As you said, only librashader-presets and librashader-preprocess are needed. Their API is also just one function each, so really simple to interface with.

However, I'm not sure mpv needs/can/want to be fully compatible with slang. They do have some downsides and limitations you might not want to port over mpv:

1. mpv would need to inspect shaders on a semantic level to resolve dependencies between passes: instead of an explicit //!BIND metadata, dependencies are resolved using uniform block member names (unknown names seem to be ignored), as well as input samplers, e.g. from this block:
   https://github.com/libretro/slang-shaders/blob/6cb93ce1be67b314a96f5e346befb2b58b63eb04/crt/shaders/guest/advanced/bloom_horizontal.slang#L24-L34
   and this declaration:
   https://github.com/libretro/slang-shaders/blob/6cb93ce1be67b314a96f5e346befb2b58b63eb04/crt/shaders/guest/advanced/bloom_horizontal.slang#L69
   mpv would need to infer that this pass depends on a pass named LinearizePass and feed its output size in the block, as well as provide a sampler for it. The "LinearizePass" name is defined in the preset here. "Original" and "Output" are keywords for mpv's "MAIN" and "target_*"
   libretro does this kind of inspection with spirv-cross¹. librashaders also has an API for that².
2. Slang also has no equivalent of mpv's hook system: you cannot choose the texture to hook on (there is only the linear RGB one), so some slang shaders written for mpv wouldn't be compatible with libretro at least.
3. Shader parameters can only be floats
4. Shader parameters are fed at runtime through the uniform blocks, there is no option for compile-time parameters. This is a problem even for some libretro shaders. For example authors of CRT royale (the shader showcased above) recommend commenting #define directives to use hardcoded constants instead of runtime parameters³⁴

Slang does seem like a better design than mpv's shaders, but there is definitely some opportunity to better adapt the design, in my opinion.

Footnotes

1. https://github.com/libretro/slang-shaders/?tab=readme-ov-file#vulkan-glsl-as-the-portable-solution ↩

2. https://docs.rs/librashader/latest/librashader/reflect/semantics/index.html ↩

3. https://github.com/libretro/slang-shaders/blob/6cb93ce1be67b314a96f5e346befb2b58b63eb04/crt/shaders/crt-royale/user-settings.h#L73-L77 ↩

4. https://docs.libretro.com/shader/crt_royale/#why-is-everything-still-so-slow ↩

[user18081972]

I use MPV all the time, but I currently watch some old anime through Retroarch because i cant get CRT-Royale to look the same in MPV as i do in Retroarch (using the CRT-Royale port someone made). But Retroarch's video player broke support for soft-subs long ago and it never got fixed.

I would LOVE to use this. Please let me know when this becomes something the average user can use.