-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove file-scope non-constant static variables to support multiple inference sessions #10481
Conversation
842231c
to
83631a3
Compare
3151d8e
to
52a68ed
Compare
10895f1
to
49f0890
Compare
49f0890
to
c01bf10
Compare
: GraphTransformer("PropagateCastOps", compatible_execution_providers), level_(level), strategy_(strategy) { | ||
fp16_allow_ops[0].clear(); // Remove previously added op types if any. | ||
std::copy(_allow_list.begin(), _allow_list.end(), std::inserter(fp16_allow_ops[0], fp16_allow_ops[0].begin())); | ||
: GraphTransformer("PropagateCastOps", compatible_execution_providers), level_(level), fp16_allow_ops_(3), strategy_(strategy) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: lines below are kind of long.
if this is the only place that level1_fp16_allow_ops/level2_fp16_allow_ops are used, how about just initializing them here?
: GraphTransformer("PropagateCastOps", compatible_execution_providers), level_(level), fp16_allow_ops_(3), strategy_(strategy) { | |
: GraphTransformer("PropagateCastOps", compatible_execution_providers), level_(level), | |
fp16_allow_ops_{ | |
std::unordered_set<std::string>(allow_list.begin(), allow_list.end()), | |
{"level1 op", ... }, | |
{"level2 op", ... } | |
}, | |
strategy_(strategy) { |
and then fp16_allow_ops_ can be const too
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I want to keep the level1 and level2 operators at the top of the file so that it is easy to modify (add new operators and delete existings ones.) and/or add a new levels of fp16 operator all together.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FP16AllowOps is designed to make adding new operator and new levels easy, i.e. to add new levels or new operators without modifying the code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if you want to keep them separate, it's fine too.
constexpr std::array level1_fp16_allow_ops = {"...", ... }; // deduced to std::array<const char*, N>, don't need to specify N
...
// ctor member initializer list
fp16_allow_ops_{
std::unordered_set<std::string>(allow_list.begin(), allow_list.end()),
std::unordered_set<std::string>(level1_fp16_allow_ops.begin(), level1_fp16_allow_ops.end()),
std::unordered_set<std::string>(level2_fp16_allow_ops.begin(), level2_fp16_allow_ops.end())
}
as is, the lines in the ctor are longer than 120 characters
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Making the suggested change resulted in the following error:
/home/sajandhy/onnxruntime/onnxruntime/core/optimizer/propagate_cast_ops.cc:1468:269: required from here
/usr/include/c++/9/bits/hashtable.h:994:4: error: no matching function for call to ‘std::_Hashtable<std::__cxx11::basic_string, std::__cxx11::basic_string, std::allocator<std::__cxx11::basic_string >, std::__detail::_Identity, std::equal_to<std::__cxx11::basic_string >, std::hash<std::__cxx11::basic_string >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, true, true> >::insert(const std::basic_string_view&)’
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
because it requires conversion from string_view to string.
Description: Describe your changes.
Remove non-constant file-scope static ("global static") variables.
The scope of this PR is only to enable multiple concurrent InferenceSession instances by removing non-constant static variables in the file-scope.
Motivation and Context
Make multiple concurrent InferenceSessions possible by encapsulating data in the class members.
Enable using multiple inference sessions.
PropagateCastOps uses and modifies 3 global containers at runtime #10473