Performance issue for the models containing 80 millions blocks

[m-sb]

Hi,
@Tim-Evans-Seequent ,

I observed that there is a performance issue while loading/saving models having millions of blocks.

any suggestion? optimization?

Thanks and regards

[Tim-Evans-Seequent]

First, make sure you're running the optimised version of the Rust code, by building with cargo build --release. The debug version will be quite slow.

If that isn't the problem then I would need some more information, such as details of what you're loading and saving, example code if possible, and how fast you are expecting it to run.

[Tim-Evans-Seequent]

Making the iterators innately thread safe wouldn't be practical unfortunately, as the cost of a lock around each call would be too high. Some alternatives include:

• Load the entire array in one go, using omf_reader_array_freeform_subblocks32 (or 64) rather than the iterator interface. Then you'll have simple arrays of uint32_t[3] and float[6] that you can easily use from multiple threads.
• Have one thread read blocks of data from the array, and send those blocks to a thread pool to process. This would be more complex.
• Use omf_reader_array_bytes, or read the OMF2 file as a ZIP archive, to get the raw compressed data, which is in Parquet format. You could then use Arrow to process it in more complicated ways. Note that by default Arrow reads the whole array into memory, the low-level streaming APIs there are even more complex.

[m-sb]

Hi @Tim-Evans-Seequent ,
Thanks for your suggestion. I tried the omf_reader_array_regular_subblocks which changes reading speed drastically for me. Though it works fine, still need to improve it. will try multi-threading on that. lets C.

Can we use same thing for reading attributes
i.e. omf_reader_array_numbers_float64 instead of omf_reader_array_numbers_int64_iter
if yes, is there any similar API available for text data i.e. for omf_reader_array_text_iter , currently I am using this for string attribute data, if there is no such API for text (omf_reader_array_texts), do you suggest any other array type to store the string data?

it will be good if we have some example for omf_reader_array_bytes.

Thanks and Regards,

[Tim-Evans-Seequent]

For numeric attributes you have either omf_reader_array_numbers_float32 or omf_reader_array_numbers_float64 which read the numbers and "null" mask as arrays. Documentation for those is here: https://gmggroup.github.io/omf-rust/c/reader_iterators.html

There is no equivalent for text data. It would require allocating memory for each of the values, which would be likely to take away any performance benefits from having an array.

If possible, use category attributes rather text. Category attributes should be used when you have frequently repeated text values, whether those are rock types, zone assignments, or people's names. They will be faster and produce smaller files, as each string is only stored once, then referred to by index on each block or vertex. Text attributes should only be used for things like human-written comments where there will be few repeated values.

[m-sb]

Hi @Tim-Evans-Seequent ,

Had one query regarding integer attribute. for float we are using omf_reader_array_numbers_float32 or float64. But for integer attribute we don't have similar apis like omf_reader_array_numbers_int64. In integer attribute we still need to use the interative approach which is having performance issue.

Any suggestions welcome?
Thanks

[Tim-Evans-Seequent]

Integers with fewer than 64 bits can be stored in a 64-bit floating point value (double) without losing any precision. The intention is that you use the *_float64 functions for most numeric values, with your own conversions on top.