feat: support neon, sse simd and dynamic dispatch #56

xiegx94 · 2023-03-02T06:17:10Z

Main changes

support both static dispatch and dynamic dispatch
support neon and sse architecture

xiegx94 · 2023-03-07T11:06:26Z

This PR provides 2 ways to support multi-arch dispatch: dispatch at compile (static dispatch) and dispatch at runtime (dynamic dispatch). Dynamic dispatch is implemented by using gcc/clang multiversioning-functions, which causes these function cannot be inlined when compile and the performance will be worse.

The structure of arch folder

├── avx2
│   ├── base.h
│   ├── itoa.h
│   ├── quote.h
│   ├── simd.h
│   ├── skip.h
│   ├── str2int.h
│   └── unicode.h
├── common
│   ├── quote_common.h
│   ├── quote_tables.h
│   ├── skip_common.h
│   ├── unicode_common.h
│   └── x86_common
│       ├── itoa.h
│       ├── quote.inc.h
│       └── skip.inc.h
├── neon
│   ├── base.h
│   ├── itoa.h
│   ├── quote.h
│   ├── simd.h
│   ├── skip.h
│   ├── str2int.h
│   └── unicode.h
├── simd_base.h
├── simd_dispatch.h
├── simd_itoa.h
├── simd_quote.h
├── simd_skip.h
├── simd_str2int.h
├── sonic_cpu_feature.h
├── sse
│   ├── base.h
│   ├── itoa.h
│   ├── quote.h
│   ├── simd.h
│   ├── skip.h
│   ├── str2int.h
│   └── unicode.h
├── target_macro.h
└── x86_ifuncs
    ├── base.h
    ├── ifunc_macro.h
    ├── itoa.h
    ├── quote.h
    ├── skip.h
    └── str2int.h

How to add a new function

If you want to add a new simd function which is called foo, then, you should follow below steps:

implement foo for every arch, such as:

namespace sonic_json {
namespace internal {
namespace avx2 {

void foo() { return; }

}  // namespace avx2
}  // namespace internal
}  // namespace sonic_json

provide dynamic dispatch functions for x86 (or others platform)

namespace sonic_json {
namespace internal {

__attribute__((target(HASWELL))) inline void foo() { return avx2::foo(); }
__attribute__((target(WESTMERE))) inline void foo() { return sse::foo(); }
__attribute__((target("default"))) inline void foo() { return sse::foo(); }

}
}

If you want implement foo in a new header file foo.h, you should provide such file for every arch and x86_ifuncs. then add a new file simd_foo.h in arch floder:

#pragma once

#include "simd_dispatch.h"

#include INCLUDE_ARCH_FILE(foo.h)

namespace sonic_json {
namespace internal {

SONIC_USING_ARCH_FUNC(foo);

}
}

How to add a new architecture

If there is a new architecture named Y86, you should do:

write a new rule to detect Y86 macro ( provide by gcc/clang) in sonic_cpu_feature.h

#if defined(__Y86__)
#define SONIC_HAVE_Y86
#endif

write a new rule about how to dispatch in simd_dispatch.h

#if defined(SONIC_STATIC_DISPATCH)
#if defined(SONIC_HAVE_Y86)
#define SONIC_USING_ARCH_FUNC(func) using Y86::func
#define INCLUDE_ARCH_FILE(file) SONIC_STRINGIFY(Y86/file)
#endif
#elif defined(SONIC_DYNAMIC_DISPATCH)
#if defined(SONIC_HAVE_Y86)
#define SONIC_USING_ARCH_FUNC(func)
#define INCLUDE_ARCH_FILE(file) SONIC_STRINGIFY(y86_ifuncs/file)
#endif
#endif

create y86 folder and implement all simd functions
create y86 folder and implement all multiversioning-functions.

sonic 的多架构设计同时支持在编译期间选择指定的指令和在运行时根据运行的平台选择合适的指令。同时支持两种方式是因为在运行时抉择会让使用 simd 的函数/接口无法在编译期间 inline，这会引起一些性能下降。

arch 目录结构

├── avx2
│   ├── base.h
│   ├── itoa.h
│   ├── quote.h
│   ├── simd.h
│   ├── skip.h
│   ├── str2int.h
│   └── unicode.h
├── common
│   ├── quote_common.h
│   ├── quote_tables.h
│   ├── skip_common.h
│   ├── unicode_common.h
│   └── x86_common
│       ├── itoa.h
│       ├── quote.inc.h
│       └── skip.inc.h
├── neon
│   ├── base.h
│   ├── itoa.h
│   ├── quote.h
│   ├── simd.h
│   ├── skip.h
│   ├── str2int.h
│   └── unicode.h
├── simd_base.h
├── simd_dispatch.h
├── simd_itoa.h
├── simd_quote.h
├── simd_skip.h
├── simd_str2int.h
├── sonic_cpu_feature.h
├── sse
│   ├── base.h
│   ├── itoa.h
│   ├── quote.h
│   ├── simd.h
│   ├── skip.h
│   ├── str2int.h
│   └── unicode.h
├── target_macro.h
└── x86_ifuncs
    ├── base.h
    ├── ifunc_macro.h
    ├── itoa.h
    ├── quote.h
    ├── skip.h
    └── str2int.h

avx2, sse, neon。特定架构下的 simd 实现代码
common, 通用的一些实现
x86_ifuncs x86 平台动态 dispatch 代码

如何添加新的函数

在每个 arch 下添加新的函数，如：

namespace sonic_json {
namespace internal {
namespace avx2 {

void foo() { return; }

}  // namespace avx2
}  // namespace internal
}  // namespace sonic_json

在 x86_ifunc 下添加 x86 动态 dispatch 支持：

namespace sonic_json {
namespace internal {

__attribute__((target(HASWELL))) inline void foo() { return avx2::foo(); }
__attribute__((target(WESTMERE))) inline void foo() { return sse::foo(); }
__attribute__((target("default"))) inline void foo() { return sse::foo(); }

}
}

（可选）如果添加了新的头文件，则在 arch 下添加 simd_foo.h，在各 arch 下添加 foo.h 文件。simd_foo 如下：

#pragma once

#include "simd_dispatch.h"

#include INCLUDE_ARCH_FILE(foo.h)

namespace sonic_json {
namespace internal {

SONIC_USING_ARCH_FUNC(foo);

}
}

如何添加新的架构

假如有个新的架构叫Y86，需要在 sonic 中添加其 simd 支持，则：

在 sonic_cpu_feature.h 中检测Y86的宏:

#if defined(__Y86__)
#define SONIC_HAVE_Y86
#endif

在 simd_dispatch 中添加 dispatch 规则

#if defined(SONIC_STATIC_DISPATCH)
#if defined(SONIC_HAVE_Y86)
#define SONIC_USING_ARCH_FUNC(func) using Y86::func
#define INCLUDE_ARCH_FILE(file) SONIC_STRINGIFY(Y86/file)
#endif
#elif defined(SONIC_DYNAMIC_DISPATCH)
#if defined(SONIC_HAVE_Y86)
#define SONIC_USING_ARCH_FUNC(func)
#define INCLUDE_ARCH_FILE(file) SONIC_STRINGIFY(y86_ifuncs/file)
#endif
#endif

添加 y86 文件夹，添加所有的 simd 函数的 y86 实现
添加 y86_ifuncs 文件夹，添加 y86 的 multiversioning-function 实现

codecov-commenter · 2023-03-16T02:28:21Z

Codecov Report

Merging #56 (9980dc1) into master (80cdba0) will increase coverage by 0.84%.
The diff coverage is 91.61%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@            Coverage Diff             @@
##           master      #56      +/-   ##
==========================================
+ Coverage   95.04%   95.88%   +0.84%     
==========================================
  Files          22       21       -1     
  Lines        2785     2431     -354     
==========================================
- Hits         2647     2331     -316     
+ Misses        138      100      -38

Impacted Files	Coverage Δ
include/sonic/allocator.h	`90.43% <ø> (ø)`
include/sonic/dom/dynamicnode.h	`96.08% <ø> (ø)`
include/sonic/dom/serialize.h	`93.39% <ø> (ø)`
include/sonic/internal/arch/avx2/base.h	`100.00% <ø> (ø)`
include/sonic/internal/ftoa.h	`97.34% <ø> (ø)`
include/sonic/internal/itoa.h	`100.00% <ø> (ø)`
include/sonic/internal/arch/simd_skip.h	`89.23% <89.23%> (ø)`
include/sonic/dom/handler.h	`99.04% <100.00%> (ø)`
include/sonic/dom/parser.h	`94.23% <100.00%> (ø)`
include/sonic/internal/arch/avx2/simd.h	`100.00% <100.00%> (ø)`
... and 4 more

... and 3 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

xiegx94 · 2023-03-22T12:50:20Z

Performance

test case	master(haswell)	sse	haswell	dynamic dispatch
book/Decode_SonicDyn	980 ns	813 ns	849 ns	1128
gsoc-2018/Decode_SonicDyn	1406878 ns	1470898 ns	1339296 ns	1752588
fgo/Decode_SonicDyn	129952490 ns	112165070 ns	117719364 ns	150338769
lottie/Decode_SonicDyn	948184 ns	805143 ns	842756 ns	1187414
canada/Decode_SonicDyn	4068896 ns	3756878 ns	3789520 ns	4085432
github_events/Decode_SonicDyn	42468 ns	39368 ns	39716 ns	54755
otfcc/Decode_SonicDyn	321242929 ns	292141676 ns	320360184 ns	377427578
poet/Decode_SonicDyn	1611831 ns	1572923 ns	1534339 ns	1743444
citm_catalog/Decode_SonicDyn	1212610 ns	1137476 ns	1217241 ns	1439325
twitter/Decode_SonicDyn	194191 ns	181451 ns	185673 ns	260165
twitterescaped/Decode_SonicDyn	572412 ns	492546 ns	555098 ns	671564
book/Encode_SonicDyn	598 ns	619 ns	631 ns	616
gsoc-2018/Encode_SonicDyn	702591 ns	796307 ns	680203 ns	672574
fgo/Encode_SonicDyn	75789720 ns	75432301 ns	76930374 ns	75568517
lottie/Encode_SonicDyn	846441 ns	858753 ns	839591 ns	871623
canada/Encode_SonicDyn	6078378 ns	6152427 ns	6009102 ns	6074922
github_events/Encode_SonicDyn	21617 ns	22432 ns	21035 ns	20380
otfcc/Encode_SonicDyn	222879330 ns	155490041 ns	159626245 ns	160975575
poet/Encode_SonicDyn	864328 ns	840780 ns	720925 ns	721330
citm_catalog/Encode_SonicDyn	600309 ns	533967 ns	560394 ns	554867
twitter/Encode_SonicDyn	94764 ns	97690 ns	93807 ns	89528
twitterescaped/Encode_SonicDyn	281830 ns	284284 ns	263186 ns	269042

liuq19 · 2023-03-22T09:33:22Z

include/sonic/internal/arch/x86_ifuncs/skip.h

+
+using common::EqBytes4;
+using common::SkipLiteral;
+using sse::GetNextToken;  // !!!Not efficency


这个注释是什么原因

GetNextToken 是一个模板函数，没办法使用 multiversion function 的机制让编译器自动选择版本。这里在所有的架构下都选择了sse 版本。

liuq19 · 2023-03-23T03:23:26Z

include/sonic/internal/arch/avx2/simd.h

@@ -479,7 +515,15 @@ struct simd256<int8_t> : num256<int8_t> {
 template <>
 struct simd256<uint8_t> : num256<uint8_t> {
  using Base = num256<uint8_t>;
-  using Base::Base;
+  // using Base::Base;


这里为啥不能复用，按理说O3 优化下，应该都能内联

和内联没关系。这里没有提供构造函数，所以构造函数是默认的，没有内联。这样会导致构造函数在非指定的架构下被应用，引发编译器报错。

liuq19 · 2023-03-23T03:26:14Z

Performance

test case sse haswell dynamic dispatch
book/Decode_SonicDyn 813 ns 849 ns 1128 ns
gsoc-2018/Decode_SonicDyn 1470898 ns 1339296 ns 1752588 ns
fgo/Decode_SonicDyn 112165070 ns 117719364 ns 150338769 ns
lottie/Decode_SonicDyn 805143 ns 842756 ns 1187414 ns
canada/Decode_SonicDyn 3756878 ns 3789520 ns 4085432 ns
github_events/Decode_SonicDyn 39368 ns 39716 ns 54755 ns
otfcc/Decode_SonicDyn 292141676 ns 320360184 ns 377427578 ns
poet/Decode_SonicDyn 1572923 ns 1534339 ns 1743444 ns
citm_catalog/Decode_SonicDyn 1137476 ns 1217241 ns 1439325 ns
twitter/Decode_SonicDyn 181451 ns 185673 ns 260165 ns
twitterescaped/Decode_SonicDyn 492546 ns 555098 ns 671564 ns
book/Encode_SonicDyn 619 ns 631 ns 616 ns
gsoc-2018/Encode_SonicDyn 796307 ns 680203 ns 672574 ns
fgo/Encode_SonicDyn 75432301 ns 76930374 ns 75568517 ns
lottie/Encode_SonicDyn 858753 ns 839591 ns 871623 ns
canada/Encode_SonicDyn 6152427 ns 6009102 ns 6074922 ns
github_events/Encode_SonicDyn 22432 ns 21035 ns 20380 ns
otfcc/Encode_SonicDyn 155490041 ns 159626245 ns 160975575 ns
poet/Encode_SonicDyn 840780 ns 720925 ns 721330 ns
citm_catalog/Encode_SonicDyn 533967 ns 560394 ns 554867 ns
twitter/Encode_SonicDyn 97690 ns 93807 ns 89528 ns
twitterescaped/Encode_SonicDyn 284284 ns 263186 ns 269042 ns

最好分别贴下static 模式和 dynamic 模式下，目前分支和master分支的相对性能测试数据，这样应该更清楚一点

xiegx94 · 2023-03-23T06:28:41Z

Performance
test case sse haswell dynamic dispatch
book/Decode_SonicDyn 813 ns 849 ns 1128 ns
gsoc-2018/Decode_SonicDyn 1470898 ns 1339296 ns 1752588 ns
fgo/Decode_SonicDyn 112165070 ns 117719364 ns 150338769 ns
lottie/Decode_SonicDyn 805143 ns 842756 ns 1187414 ns
canada/Decode_SonicDyn 3756878 ns 3789520 ns 4085432 ns
github_events/Decode_SonicDyn 39368 ns 39716 ns 54755 ns
otfcc/Decode_SonicDyn 292141676 ns 320360184 ns 377427578 ns
poet/Decode_SonicDyn 1572923 ns 1534339 ns 1743444 ns
citm_catalog/Decode_SonicDyn 1137476 ns 1217241 ns 1439325 ns
twitter/Decode_SonicDyn 181451 ns 185673 ns 260165 ns
twitterescaped/Decode_SonicDyn 492546 ns 555098 ns 671564 ns
book/Encode_SonicDyn 619 ns 631 ns 616 ns
gsoc-2018/Encode_SonicDyn 796307 ns 680203 ns 672574 ns
fgo/Encode_SonicDyn 75432301 ns 76930374 ns 75568517 ns
lottie/Encode_SonicDyn 858753 ns 839591 ns 871623 ns
canada/Encode_SonicDyn 6152427 ns 6009102 ns 6074922 ns
github_events/Encode_SonicDyn 22432 ns 21035 ns 20380 ns
otfcc/Encode_SonicDyn 155490041 ns 159626245 ns 160975575 ns
poet/Encode_SonicDyn 840780 ns 720925 ns 721330 ns
citm_catalog/Encode_SonicDyn 533967 ns 560394 ns 554867 ns
twitter/Encode_SonicDyn 97690 ns 93807 ns 89528 ns
twitterescaped/Encode_SonicDyn 284284 ns 263186 ns 269042 ns

最好分别贴下static 模式和 dynamic 模式下，目前分支和master分支的相对性能测试数据，这样应该更清楚一点

Updated.

liuq19 · 2023-03-28T02:25:51Z

include/sonic/internal/arch/neon/str2int.h

+    sum = sum * 10 + (c[i] - '0');
+    i++;
+  }
+  man_nd = i;


这里为啥没有实现neon simd版本。

暂时没想到在 neon 下怎么实现这个函数

liuq19 · 2023-03-28T07:53:59Z

include/sonic/internal/arch/x86_ifuncs/skip.h

+
+__attribute__((target("default"))) inline uint8_t skip_space_safe(
+    const uint8_t*, size_t&, size_t, size_t&, uint64_t&) {
+  return 0;


这里返回0 可能会有问题吗？相当于fallback逻辑，然后在非west和 haswell下会执行到这里。

是的，在没有实现 fallback 的时候，这里使用 static assert 阻止编译比较好。

include/sonic/allocator.h

liuq19 · 2023-03-31T08:33:31Z

BUILD.bazel

+            }) +\
+            select({
+                "static_dispatch": static_dispatch_copts,
+                "dynamic_dispatch": dynamic_dispatch_copts,


dynamic 模式下需要加 mavx2 编译options吗，理论上应该不需要了吧

这里没有指定必须使用 mavx2 编译选项。

include/sonic/internal/arch/avx2/quote.h

xiegx94 mentioned this pull request Mar 2, 2023

feat: support sse and neon #51

Closed

liuq19 reviewed Mar 23, 2023

View reviewed changes

feat: support neon and sse simd, support dynamic dispatch

227105a

xiegx94 force-pushed the feat/support-ifunc branch from 9c10f87 to 227105a Compare March 27, 2023 02:37

liuq19 reviewed Mar 31, 2023

View reviewed changes

include/sonic/internal/arch/avx2/quote.h Outdated Show resolved Hide resolved

xiegx94 added 2 commits April 10, 2023 19:41

define push and pop target pragma options macro

e1d1306

static assert not implemented functions in ifuncs

9980dc1

liuq19 approved these changes Apr 11, 2023

View reviewed changes

liuq19 merged commit 50514ec into bytedance:master Apr 11, 2023

This was referenced Apr 19, 2023

When will the arm platform be supported #62

Open

Core dumped without AVX2 instructiom (VM instance) #50

Closed

xiegx94 mentioned this pull request Aug 5, 2024

arm: optimize decoder on Arm SVE2 platform #92

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support neon, sse simd and dynamic dispatch #56

feat: support neon, sse simd and dynamic dispatch #56

xiegx94 commented Mar 2, 2023 •

edited

Loading

xiegx94 commented Mar 7, 2023

codecov-commenter commented Mar 16, 2023 •

edited

Loading

xiegx94 commented Mar 22, 2023 •

edited

Loading

liuq19 Mar 22, 2023

xiegx94 Mar 23, 2023

liuq19 Mar 23, 2023

xiegx94 Mar 23, 2023

liuq19 commented Mar 23, 2023

xiegx94 commented Mar 23, 2023

liuq19 Mar 28, 2023

xiegx94 Apr 3, 2023

liuq19 Mar 28, 2023

xiegx94 Apr 3, 2023

liuq19 Mar 31, 2023

xiegx94 Apr 3, 2023

feat: support neon, sse simd and dynamic dispatch #56

feat: support neon, sse simd and dynamic dispatch #56

Conversation

xiegx94 commented Mar 2, 2023 • edited Loading

Main changes

xiegx94 commented Mar 7, 2023

The structure of arch folder

How to add a new function

How to add a new architecture

arch 目录结构

如何添加新的函数

如何添加新的架构

codecov-commenter commented Mar 16, 2023 • edited Loading

Codecov Report

xiegx94 commented Mar 22, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liuq19 commented Mar 23, 2023

xiegx94 commented Mar 23, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xiegx94 commented Mar 2, 2023 •

edited

Loading

codecov-commenter commented Mar 16, 2023 •

edited

Loading

xiegx94 commented Mar 22, 2023 •

edited

Loading