EP context cache feature design

[HectorSVC]

Description

EP context cache feature design

[jywu-msft]

+@chilo-ms fyi

[mrsabhar]

@HectorSVC Thanks for coming up with concept to improve first ever inference latency as this is requested by several SW vendors (even for non-GenAI). I’m aware that the feature design is closed, but how does it address the scenario where a developer ships an encrypted ONNX file and the context file is obfuscated or encrypted? Do we expect double processing/memory usage for decrypting the ONNX model to determine if a context file is available, and then again reading the context file into memory (which can’t be in human-readable form due to IP leakage) to check its availability? Only time we know context cache is available for ONNX model is in memory. Also how context file takes into account dynamic shapes ?

"If the user loads the model from memory buffer, user needs to provide session option ep.context_file_path. EP gets the folder path from ep.context_file_path, and combines it with the relative path got from step a) as the context binary file full path."