[js/webgpu] Float16Array polyfill for uniform #19307

axinging · 2024-01-29T08:50:20Z

To use this feature, first create a f16 model from f32 model:

import onnxruntime as ort
import onnx
from onnx import helper as helper
from onnx import TensorProto as tp
import numpy as np
import onnx
from onnxconverter_common import float16

model = onnx.load("pad_constant_f32_opset8.onnx")
model_fp16 = float16.convert_float_to_float16(model)
onnx.save(model_fp16, "pad_constant_f16_opset8.onnx")

Then test it like this:

  <script src="./web/dist/ort.all.js"></script>
  <script src="https://cdn.jsdelivr.net/npm/@petamoriken/float16/browser/float16.min.js"></script>
  <script>
    const { Float16Array } = float16;
    async function main() {
      try {
        const session = await ort.InferenceSession.create('./pad_constant_f16_opset8.onnx', { executionProviders: ['webgpu'] });
        let dataA = Float16Array.from([1.2, 2.5, 3.5, 4, 5, 6.1, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12,
          1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]);

        const tensorA = new ort.Tensor('float16', dataA, [1, 3, 4, 5]);
        // prepare feeds. use model input names as keys.
        const feeds = { x: tensorA };
        const results = await session.run(feeds);
        // read from results
        const dataC = results.y.cpuData;
        console.log(`data of result tensor 'c': ${dataC}`);
      } catch (e) {
        console.error(`failed to inference ONNX model: ${e}.`);
      }
    }

    main();
  </script>

Code merged from: https://github.com/microsoft/onnxruntime/pull/19305/files

fs-eire · 2024-01-30T00:18:28Z

js/common/lib/tensor-impl-type-mapping.ts

+declare global {
+  // eslint-disable-next-line @typescript-eslint/naming-convention, @typescript-eslint/no-explicit-any
+  const Float16Array: any;
+  const isFloat16Array: any;


isFloat16Array is not going to be polyfilled in the global scope. we can still use a instanceof Float16Array to perform the check.

fs-eire · 2024-01-30T00:20:33Z

js/common/lib/tensor-impl.ts

-              throw new TypeError(
-                  'Creating a float16 tensor from number array is not supported. Please use Uint16Array as data.');
+              // eslint-disable-next-line  @typescript-eslint/no-explicit-any
+              data = (typedArrayConstructor as any).from(arg1);


as explained in the old error message, you cannot do typedArrayConstructor.from because it does not work with Uint16Array. you need to check if typedArrayConstructor is Float16Array. You can just use the code from that of my PR.

done， thanks！

fs-eire · 2024-01-30T00:24:29Z

js/common/lib/tensor-impl.ts

@@ -168,6 +168,8 @@ export class Tensor implements TensorInterface {
            }
          } else if (arg1 instanceof typedArrayConstructor) {
            data = arg1;
+          } else if (isFloat16Array !== undefined && isFloat16Array(arg1)) {
+            data = arg1 as InstanceType<typeof Float16Array>;


does the code actually call into this if branch? I assume that map NUMERIC_TENSOR_TYPE_TO_TYPEDARRAY_MAP should already set with key 'float16', so it should be handled in condition (arg1 instanceof typedArrayConstructor)

No, this is not necessary now. Thanks!

fs-eire · 2024-01-30T00:26:45Z

js/web/lib/onnxjs/backends/webgl/ops/resize-packed.ts

+  scalesValidation(scales as number[], mode, isResize);
+  return scales as number[];


we can keep onnxjs code unchanged as they are going to deprecated. we don't plan to support new features to webgl.

Remove this may complains: $ npm run build

[email protected] prebuild
tsc -p . --noEmit && tsc -p lib/wasm/proxy-worker --noEmit

lib/onnxjs/backends/webgl/ops/resize-packed.ts:244:20 - error TS2345: Argument of type 'unknown[]' is not assignable to parameter of type 'number[]'.
Type 'unknown' is not assignable to type 'number'.

244 scalesValidation(scales, mode, isResize);
~~~~~~

lib/onnxjs/backends/webgl/ops/resize-packed.ts:245:3 - error TS2322: Type 'unknown[]' is not assignable to type 'number[]'.

245 return scales;
~~~~~~

Found 2 errors in the same file, starting at: lib/onnxjs/backends/webgl/ops/resize-packed.ts:244

let me try to make some modifications to unit test so that it no longer depends on onnxjs/tensor. Then you don't need to modify any file under onnxjs folder .

#19358 should work with it

fs-eire · 2024-01-30T00:27:24Z

js/common/package.json

-  "description": "ONNXRuntime JavaScript API library"
+  "description": "ONNXRuntime JavaScript API library",
+  "dependencies": {
+    "@petamoriken/float16": "^3.8.4"


please add as devDependencies

fs-eire · 2024-01-30T00:27:35Z

js/web/package.json

@@ -11,6 +11,7 @@
  "version": "1.18.0",
  "jsdelivr": "dist/ort.min.js",
  "dependencies": {
+    "@petamoriken/float16": "^3.8.4",


please add as devDependencies

fs-eire · 2024-01-30T00:29:10Z

js/web/test/unittests/backends/webgl/test-conv-utils.ts

@@ -1,14 +1,18 @@
 // Copyright (c) Microsoft Corporation. All rights reserved.
 // Licensed under the MIT License.

+import {Float16Array} from '@petamoriken/float16';


no need to add this as we don't plan to add f16 support for webgl

fs-eire · 2024-01-30T01:30:51Z

js/common/test/unit-tests/common.ts

@@ -34,7 +35,7 @@ export const BIGINT_TYPES = [
 /**
 * float16 type, data represented by Uint16Array
 */
-export const FLOAT16_TYPE = ['float16', Uint16Array, false] as const;
+export const FLOAT16_TYPE = ['float16', Float16Array, false] as const;


I suggest not to update this part. Both paths (with and without polyfill) need to be tested. to test f16 polyfill, may need to add a separated target.

axinging

Done, @fs-eire

@qjia7 @xhcao @hujiajie @gyagp PTAL

axinging · 2024-01-30T01:17:23Z

js/common/lib/tensor-impl-type-mapping.ts

+declare global {
+  // eslint-disable-next-line @typescript-eslint/naming-convention, @typescript-eslint/no-explicit-any
+  const Float16Array: any;
+  const isFloat16Array: any;


axinging · 2024-01-30T01:20:36Z

js/common/package.json

-  "description": "ONNXRuntime JavaScript API library"
+  "description": "ONNXRuntime JavaScript API library",
+  "dependencies": {
+    "@petamoriken/float16": "^3.8.4"


axinging · 2024-01-30T01:20:47Z

js/web/package.json

@@ -11,6 +11,7 @@
  "version": "1.18.0",
  "jsdelivr": "dist/ort.min.js",
  "dependencies": {
+    "@petamoriken/float16": "^3.8.4",


axinging · 2024-01-30T02:03:12Z

js/common/lib/tensor-impl.ts

-              throw new TypeError(
-                  'Creating a float16 tensor from number array is not supported. Please use Uint16Array as data.');
+              // eslint-disable-next-line  @typescript-eslint/no-explicit-any
+              data = (typedArrayConstructor as any).from(arg1);


done， thanks！

axinging · 2024-01-30T02:03:50Z

js/common/lib/tensor-impl.ts

@@ -168,6 +168,8 @@ export class Tensor implements TensorInterface {
            }
          } else if (arg1 instanceof typedArrayConstructor) {
            data = arg1;
+          } else if (isFloat16Array !== undefined && isFloat16Array(arg1)) {
+            data = arg1 as InstanceType<typeof Float16Array>;


No, this is not necessary now. Thanks!

axinging · 2024-01-30T03:32:58Z

js/web/lib/onnxjs/backends/webgl/ops/resize-packed.ts

+  scalesValidation(scales as number[], mode, isResize);
+  return scales as number[];


Remove this may complains: $ npm run build

[email protected] prebuild
tsc -p . --noEmit && tsc -p lib/wasm/proxy-worker --noEmit

lib/onnxjs/backends/webgl/ops/resize-packed.ts:244:20 - error TS2345: Argument of type 'unknown[]' is not assignable to parameter of type 'number[]'.
Type 'unknown' is not assignable to type 'number'.

244 scalesValidation(scales, mode, isResize);
~~~~~~

lib/onnxjs/backends/webgl/ops/resize-packed.ts:245:3 - error TS2322: Type 'unknown[]' is not assignable to type 'number[]'.

245 return scales;
~~~~~~

Found 2 errors in the same file, starting at: lib/onnxjs/backends/webgl/ops/resize-packed.ts:244

axinging · 2024-01-30T03:33:13Z

js/common/test/unit-tests/common.ts

@@ -34,7 +35,7 @@ export const BIGINT_TYPES = [
 /**
 * float16 type, data represented by Uint16Array
 */
-export const FLOAT16_TYPE = ['float16', Uint16Array, false] as const;
+export const FLOAT16_TYPE = ['float16', Float16Array, false] as const;


qjia7 · 2024-01-30T05:45:13Z

js/web/test/data/ops/pad_f16.jsonc

+  {
+    "name": "constant 2D",
+    "operator": "Pad",
+    "opset": { "domain": "", "version": 10 },


Can you also add a test for version > 10, in which the value is as an input not attribute and have the type float16? I suppose this can test the float16 uniform?

Do you need to update the code here?

test/test-main.ts:18:12 - error TS2339: Property 'Float16Array' does not exist on type 'typeof globalThis'. 18 globalThis.Float16Array = Float16Array;

axinging · 2024-08-27T05:45:04Z

This is replaced by:
#21691

axinging added 6 commits January 29, 2024 12:47

[js] Support Float16Array polyfill

5a95268

Try conditional float16 polyfill

706edd4

Code merged from: https://github.com/microsoft/onnxruntime/pull/19305/files

NIt

5453a7a

Nit, removbe more

a8d4d77

rEMOVE ONE MORE

ce46073

Clean one more

d6c86a3

axinging changed the title ~~Float16 polyfill v2~~ [js] Float16Arrry polyfill Jan 30, 2024

fs-eire reviewed Jan 30, 2024

View reviewed changes

axinging added 2 commits January 30, 2024 08:54

Remove more

3b49dee

Move package to devDeps

da77265

axinging changed the title ~~[js] Float16Arrry polyfill~~ [js] Float16Array polyfill Jan 30, 2024

fs-eire reviewed Jan 30, 2024

View reviewed changes

axinging added 7 commits January 30, 2024 09:33

Remove isFloat16Array

4396346

Remove else for f16

1b853ba

format and rename padf16.jsonc

cb3ba6c

Revert webgl

1480c48

Add FLOAT16_TYPE_POLYFILL

d63d39a

Fix lint

82485df

Merge branch 'main' into float16_polyfill_v2

f44bd9c

axinging commented Jan 30, 2024

View reviewed changes

axinging marked this pull request as ready for review January 30, 2024 03:34

axinging mentioned this pull request Jan 30, 2024

[js/web] Float16 polyfill #19302

Closed

qjia7 reviewed Jan 30, 2024

View reviewed changes

axinging marked this pull request as draft January 30, 2024 06:11

axinging marked this pull request as ready for review January 31, 2024 07:42

axinging changed the title ~~[js] Float16Array polyfill~~ [js/web] Float16Array polyfill Feb 1, 2024

axinging added 7 commits February 19, 2024 10:27

Merge branch 'main' into float16_polyfill_v2

e9a6cfa

Merge branch 'main' into float16_polyfill_v2

453b52e

Remove FLoat16 in test-runner.ts

7e75c14

Remove onnxjs change

861c472

Remove change in onnxjs

0082c71

Remove not necessary change in packaje.json

b6f4cd5

Merge branch 'main' into float16_polyfill_v2

cecada7

axinging marked this pull request as draft February 22, 2024 00:55

Fix globalThis property missing

c9e90b1

test/test-main.ts:18:12 - error TS2339: Property 'Float16Array' does not exist on type 'typeof globalThis'. 18 globalThis.Float16Array = Float16Array;

axinging changed the title ~~[js/web] Float16Array polyfill~~ [js/webgpu] Float16Array polyfill Feb 22, 2024

axinging added 5 commits February 23, 2024 09:01

Rename pad_f16 and add to test list

10e9d65

Merge branch 'main' into float16_polyfill_v2

7ab6a86

t

3619729

Fix lint

c8c74d2

Revet webgl/test-conv-utils.ts

4cb32ba

axinging changed the title ~~[js/webgpu] Float16Array polyfill~~ [js/webgpu] Float16Array polyfill for uniform Feb 23, 2024

axinging marked this pull request as ready for review February 23, 2024 02:38

axinging added 3 commits February 23, 2024 13:45

Remove globalThis in test

3020722

Nit, remove noVar

894c408

Not float16 in test

aa474a8

axinging marked this pull request as draft February 23, 2024 06:38

axinging added 4 commits February 26, 2024 13:51

Remove float16 in common test

6795713

use float16 in test

432f311

Merge branch 'main' into float16_polyfill_v2

44ee23f

Refine comment msg

a239ce2

fs-eire mentioned this pull request Aug 8, 2024

[js/web] allow op test to use f16 type for inputs/outputs #21664

Merged

axinging closed this Aug 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[js/webgpu] Float16Array polyfill for uniform #19307

[js/webgpu] Float16Array polyfill for uniform #19307

axinging commented Jan 29, 2024 •

edited

Loading

fs-eire Jan 30, 2024

axinging Jan 30, 2024

fs-eire Jan 30, 2024 •

edited

Loading

axinging Jan 30, 2024

fs-eire Jan 30, 2024

axinging Jan 30, 2024

fs-eire Jan 30, 2024

axinging Jan 30, 2024

fs-eire Jan 31, 2024

fs-eire Jan 31, 2024

fs-eire Jan 30, 2024

axinging Jan 30, 2024

fs-eire Jan 30, 2024

axinging Jan 30, 2024

fs-eire Jan 30, 2024

axinging Jan 30, 2024

fs-eire Jan 30, 2024

axinging Jan 30, 2024

axinging left a comment

axinging Jan 30, 2024

axinging Jan 30, 2024

axinging Jan 30, 2024

axinging Jan 30, 2024

axinging Jan 30, 2024

axinging Jan 30, 2024

axinging Jan 30, 2024

qjia7 Jan 30, 2024

axinging Jan 31, 2024

axinging commented Aug 27, 2024

		scalesValidation(scales as number[], mode, isResize);
		return scales as number[];

[js/webgpu] Float16Array polyfill for uniform #19307

[js/webgpu] Float16Array polyfill for uniform #19307

Conversation

axinging commented Jan 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fs-eire Jan 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

axinging left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

axinging commented Aug 27, 2024

axinging commented Jan 29, 2024 •

edited

Loading

fs-eire Jan 30, 2024 •

edited

Loading