-
Notifications
You must be signed in to change notification settings - Fork 102
/
GLSL_QCOM_image_processing.txt
188 lines (138 loc) · 9 KB
/
GLSL_QCOM_image_processing.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
Name
QCOM_image_processing
Name Strings
GL_QCOM_image_processing
Contact
Jeff Leger, Qualcomm Technologies Inc. (jleger 'at' qti.qualcomm.com)
Contributors
Jeff Leger, Qualcomm
Wooyoung Kim, Qualcomm
Ruihao Zhang, Qualcomm
Status
Complete
Version
Last Modified Date: April 12, 2023
Revision: 1
Dependencies
This extension can be applied to OpenGL GLSL versions 1.40
(#version 140) and higher.
This extension can be applied to OpenGL ES ESSL versions 3.10
(#version 310) and higher.
Requires GL_KHR_vulkan_glsl.
Overview
GPUs commonly process images for a wide range of use-cases. These include enhancement
of externally sourced images (i.e., camera image enhancement), post processing of
GPU-rendered game content, image scaling, and image analysis (i.e., motion vector
generation). In some cases, higher-order filtering kernels and advanced image algorithms
are required.
While such algorithms could be implemented in shader code generically using existing
texture built-in functions, it requires many round-trips between the texture unit and
shader unit. Some GPUs have dedicated HW shader instructions for such image processing
tasks, enabling advanced functionality with simplified shader code.
The extension exposes support for 4 new shader instructions:
Weighted Texture Sample: This instruction performs a weighted texture sampling operation
involving two images: the sampled image and the weight image. An MxN region of texels
in the sampled image are convolved with an MxN set of scalar weights provided in the
weight image. Large filter sizes up to 64x64 taps enable important use-cases like
edge-detection, feature extraction, and anti-aliasing.
Weighted Box Filter: This instruction performs weighted average of the texels within a
screen-aligned box. The operation is similar to bi-linear filtering, except the region
of texels is not limited to 2x2. The instruction includes a BoxSize parameter, with
fractional box sizes up to [64.0, 64.0]. Similar to bi-linear filtering, the
implementation computes a weighted average for all texels covered by the box, with the
weight for each texel proportional covered area. Large box sizes up to 64x64 enable
important use-cases like bulk mipmap generation and high quality single-pass image
down-scaling with arbitrary scaling ratios (e.g. thumbnail generation).
Block Matching SAD / Block Matching SSD: These instructions perform a block matching
operation involving two images: the target image and reference image. Each instruction
takes two sets of integer texture coordinates, and an integer BlockSize parameter. An MxN
region of texels in the target image is compared with an MxN region in the reference image.
The instruction returns a per-component error metric describing the difference between
the two regions. The SAD returns the sum of the absolute errors and the SSD returns
the sum of the squared differences.
Mapping to SPIR-V
-----------------
For informational purposes (non-normative), the following is an
expected way for an implementation to map GLSL constructs to SPIR-V
constructs:
textureWeightedQCOM -> OpImageSampleWeightedQCOM
textureBoxFilterQCOM -> OpImageBoxFilterQCOM
textureBlockMatchSADQCOM -> OpImageBlockMatchSSDQCOM
textureBlockMatchSSDQCOM -> OpImageBlockMatchSADQCOM
The <weight> texture operand of textureWeightedQCOM should be
decorated with WeightTextureQCOM. The decoration can
help SPIR-V reflection tools provide this information back to
application which in turn is helpful for the application to properly
create the weight texture object in the client API.
For similar reasons, the <target> and <reference> texture operands of
textureBlockMatchSADQCOM and textureBlockMatchSSDQCOM should be decorated
with BlockMatchTextureQCOM.
The corresponding 3 SPIR-V capabilities are:
TextureSampleWeightedQCOM required by OpImageSampleWeightedQCOM
TextureBoxFilterQCOM required by OpImageBoxFilterQCOM
TextureBlockMatchQCOM required by OpImageBlockMatchSSDQCOM and OpImageBlockMatchSADQCOM
Modifications to the OpenGL GL Shading Language Specification, Version 1.40, dated 22 November 2009
Including the following line in a shader can be used to control the
language features described in this extension:
#extension GL_QCOM_image_processing : <behavior>
where <behavior> is as specified in Section 3.3.
New preprocessor #defines are added to the OpenGL Shading Language:
#define GL_QCOM_image_processing 1
Add to the table at the end of Subsection 8.9.2 "Texel Lookup Functions", add
the following image processing functions.
+------------------------------------+--------------------------------------------+
| Syntax | Description |
+------------------------------------+--------------------------------------------+
| vec4 textureWeightedQCOM( | weighted sample operation multiplies |
| sampler2D tex, | a 2D kernel of filter weights with a corr- |
| vec2 P, | esponding region of sampled texels and |
| sampler2DArray weight) | sums the results to produce the output |
| | value. |
+------------------------------------+--------------------------------------------+
| vec4 textureBoxFilterQCOM( | Linear operation taking average of pixels |
| sampler2D tex, | within the spatial region described by |
| vec2 P, | boxSize. The box is centered at coordinate|
| vec2 boxSize) | P and has width and height of boxSize.x |
| | and boxSize.y. |
+------------------------------------+--------------------------------------------+
| vec4 textureBlockMatchSADQCOM( | Block matching operation measures the |
| sampler2D target | correlation (or similarity) of the target |
| uvec2 targetCoord, | block and reference block. targetCoord |
| sampler2D reference, | and refCoord specify the bottom-left corner|
| uvec2 refCoord, | of the block in target and reference |
| uvec2 blockSize) | images. The error metric is the Sum of |
| | Absolute Differences (SAD). |
+------------------------------------+--------------------------------------------+
| vec4 textureBlockMatchSSDQCOM( | Block matching operation measures the |
| sampler2D target | correlation (or similarity) of the target |
| uvec2 targetCoord, | block and reference block. TargetCoord |
| sampler2D reference, | and refCoord specify the bottom-left corner|
| uvec2 refCoord, | of the block in target and reference |
| uvec2 blockSize) | images. The error metric is the Sum of |
| | Square Differences (SSD). |
+------------------------------------+--------------------------------------------+
* The textureBlockMatch* functions return a per-component metric that reflects
the correlation of the target and reference blocks. If the blocks match exactly
the value 0 is returned. The detailed description of the SAD and SSD metrics
is documented in the "Block Matching" section of the "Image Operations" chapter
of the Vulkan specification.
* target and reference specify the two input textures. If
the same block from one texture is going to be used repeatedly across multiple
instructions, that texture should be specified as the referenceTexture.
* targetCoord and referenceCoord specify the texture coordinates (in texels)
of the bottom-left corner of each block.
* blockSize specifies the size (in texels) of the blocks to be compared.
Modifications to the OpenGL ES Shading Language Specification, Version 3.10, dated 29 January 2016
* The changes for OpenGL ES are identical to those above for OpenGL GL.
Restrictions:
It is a compile-time error if the sampler variable used as the <weight> parameter
in function textureWeightedQCOM is used in any other context in the shader.
It is a compile-time error if the sampler variable used as the <target> or <reference>
parameter in functions textureBlockMatchSADQCOM or textureBlockMatchSSDQCOM is used in
any other context in the shader.
Issues
None.
Revision History
Rev. Date Author Changes
---- ----------- -------- --------------------------------------------
1 12-Apr-2023 jleger Initial version