Skip to content

[GPU] use ParallelReadStreamBuf to boost file reading latency#34679

Open
riverlijunjie wants to merge 10 commits intoopenvinotoolkit:masterfrom
riverlijunjie:river/mmap_parallel_io_opt
Open

[GPU] use ParallelReadStreamBuf to boost file reading latency#34679
riverlijunjie wants to merge 10 commits intoopenvinotoolkit:masterfrom
riverlijunjie:river/mmap_parallel_io_opt

Conversation

@riverlijunjie
Copy link
Contributor

@riverlijunjie riverlijunjie commented Mar 13, 2026

Details:

  • Create custom std::streambuf subclass that internally uses parallel I/O for large reads, exposing a standard std::istream-compatible interface.
  • Any code that currently accepts a std::istream gets the speedup transparently — no per-plugin changes required.
  • Test result:
image

Tickets:

AI Assistance:

  • AI assistance used: no / yes
  • If yes, summarize how AI was used and what human validation was performed (build/tests/manual checks).

@github-actions github-actions bot added category: Core OpenVINO Core (aka ngraph) category: GPU OpenVINO GPU plugin labels Mar 13, 2026
@riverlijunjie riverlijunjie changed the title [GPU] mmap io parallel for model cache loading latency [GPU] Memory Mapped (mmap) Tensor Parallel Reading Mar 13, 2026
@riverlijunjie riverlijunjie changed the title [GPU] Memory Mapped (mmap) Tensor Parallel Reading [GPU] use ParallelReadStreamBuf to boost file reading latency Mar 13, 2026
@github-actions github-actions bot added the category: inference OpenVINO Runtime library - Inference label Mar 13, 2026
@github-actions github-actions bot added the category: IR FE OpenVINO IR v10 / v11 FrontEnd label Mar 14, 2026
@riverlijunjie riverlijunjie force-pushed the river/mmap_parallel_io_opt branch from 1bb37bd to 6921108 Compare March 16, 2026 02:05
@github-actions github-actions bot removed the category: IR FE OpenVINO IR v10 / v11 FrontEnd label Mar 16, 2026
@riverlijunjie riverlijunjie force-pushed the river/mmap_parallel_io_opt branch from 6921108 to 0bf2ac9 Compare March 16, 2026 03:14
@riverlijunjie riverlijunjie marked this pull request as ready for review March 16, 2026 03:16
@riverlijunjie riverlijunjie requested review from a team as code owners March 16, 2026 03:16
@riverlijunjie riverlijunjie force-pushed the river/mmap_parallel_io_opt branch from b38bc02 to f2f1f66 Compare March 16, 2026 14:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

category: Core OpenVINO Core (aka ngraph) category: GPU OpenVINO GPU plugin category: inference OpenVINO Runtime library - Inference

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant