Efficiently Loading Large Datasets with OpenBB Platform for Backtesting #7183
-
|
Beyond data vendors, what about the local storage and caching aspect? Are you caching the data locally after the first fetch? The OpenBB Platform often has built-in caching mechanisms, or perhaps you could implement your own simple pandas HDF5 or Parquet store to avoid repeatedly fetching the same data over the network. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
You're hitting on something crucial, [Friend B]! I haven't explicitly set up a robust local caching strategy yet. I know OpenBB has some internal caching, but for my larger datasets, it's probably not sufficient. I'll explore using a local data store like Parquet or HDF5 for data I've already pulled. This way, I can load it directly from disk on subsequent runs, significantly reducing network calls and processing time. I'll look into how OpenBB's existing caching can be leveraged or overridden for this purpose. Excellent suggestion! |
Beta Was this translation helpful? Give feedback.
You're hitting on something crucial, [Friend B]! I haven't explicitly set up a robust local caching strategy yet. I know OpenBB has some internal caching, but for my larger datasets, it's probably not sufficient. I'll explore using a local data store like Parquet or HDF5 for data I've already pulled. This way, I can load it directly from disk on subsequent runs, significantly reducing network calls and processing time. I'll look into how OpenBB's existing caching can be leveraged or overridden for this purpose. Excellent suggestion!