Transform the World Storage to reduce memory consumption

[ameligrana]

Currently there is some waste in memory consumption since the total memory of Ark increases as O(n_components x n_tables), so when you have many components and many tables even if those tables only have a few components you can have a good amount of wasted memory. This can be improved in two ways as far as I can see:

1. Have one more indirection so that we pass to O(n_components x n_archetypes)
2. Have a map that for each component specifies a mapping from table id to the column

The first is less optimal for memory but faster than the second. I think that indeed the first solution should be sufficient.

Both are still slower than the current design though, so maybe one can use this layout only as an option.

Not totally sure how to proceed in general, what do you think @mlange-42?

[mlange-42]

I don't understand what you mean by item 1. For item 2, I do not primarily see the issue in queries, but in component access for specific entities, which would probably be slowed down considerably. In the Go Ark/Arche, I had it that way first, but decided for the more runtime friendly way without the mapping due to the large gain. I.e. only one indirection instead of two. (EDIT: it was the other way round, a mapping from ID to column index per table, but still the same performance implications)

I am also not sure how big the memory issue really is. For columns that aren't present in a table, isn't it just the size of a pointer, i.e. 8B? To take an example: a rel. large model with 100 comps and 1000 tables would waste 800kB, right? So you would need hundreds of thousands of tables to make it really matter.

[ameligrana]

yes, after some late changes it is 8B. Indeed this is a problem when you have millions of tables and many components, one could though say that this is not a good use of relationship, and should avoid those in this case. I have a scratch implementation of 1. at #666, seems very bad perf wise though.

[mlange-42]

Yeah, exactly, such high numbers of tables will only be achieved with relationship "misuse". We already have a section on that in the docs.