-
Notifications
You must be signed in to change notification settings - Fork 3.5k
[TRT EP] Fix trt_load_user_initializer
for large models where weight are not correctly excluded
#25502
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[TRT EP] Fix trt_load_user_initializer
for large models where weight are not correctly excluded
#25502
Conversation
Can we get this into 1.23 since it is a fix on top of #25409. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR fixes an issue in the TensorRT execution provider where external initializers (weights) that are already loaded in memory were not being handled correctly for large models. The fix ensures that initializers with external data in memory are properly recognized and processed alongside those with raw data.
- Converts
TensorrtUserWeights
from struct to class with proper accessors to improve encapsulation - Adds handling for external initializers that are already loaded in memory using
HasExternalDataInMemory
checks - Updates serialization logic to exclude external data from graph proto when
include_initializer_data
is false
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
File | Description |
---|---|
tensorrt_execution_provider.h | Converts TensorrtUserWeights to class with proper encapsulation and accessor methods |
tensorrt_execution_provider.cc | Adds external data handling logic and updates weight processing to use new class interface |
graph_proto_serializer.cc | Updates serialization condition to handle external data in memory alongside raw data |
onnxruntime/core/providers/tensorrt/tensorrt_execution_provider.cc
Outdated
Show resolved
Hide resolved
/azp run Linux QNN CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline,Windows x64 QNN CI Pipeline |
Azure Pipelines successfully started running 5 pipeline(s). |
Co-authored-by: Copilot <[email protected]>
/azp run Linux QNN CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline,Windows x64 QNN CI Pipeline |
Azure Pipelines successfully started running 5 pipeline(s). |
there are build errors (warnings as errors) |
Thanks, should be fixed now. I also fixed some test usage and improved the logging. |
/azp run Linux QNN CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline,Windows x64 QNN CI Pipeline |
Azure Pipelines successfully started running 5 pipeline(s). |
The failing CI seems tobe unrelated to my changes but had some network outage or builder crashes. |
Hi there! We haven't cut the release branch for this version yet, so I'm removing the |
…t are not correctly excluded (microsoft#25502) ### Description This change respects initializers that are external but already loaded in memory. This is required due to an optimization that leaves it to the backend to read a mapped memory area. @chilo-ms can you help run the CI and merge this change ? --------- Co-authored-by: Copilot <[email protected]>
Description
This change respects initializers that are external but already loaded in memory. This is required due to an optimization that leaves it to the backend to read a mapped memory area.
@chilo-ms can you help run the CI and merge this change ?