The data module, and more generally, the ways internal data is stored, read and updated is due for a refactoring.
In its current form, the item models are the authors of the internal data but as Bookmarks is growing and more elements rely on the internal data this old implementation doesn't scale at all. Item models instead should be regular consumers. Additionally, the current code is flawed! It stems from a misunderstanding of basic threading concepts, and a misinterpretation of the function of the weakref module (at least I'm learning, ey) resulting in race conditions and data corruption.
Here's what I'm thinking of doing:
- Let the GUI thread be the sole author of the internal data
- Use signal-slot mechanism for signalling thread updates
- Rework the internal data structure to flatten it and store items using their path-hashes
- Completely remove any direct reference to internal data in the item models (the whole notion of models needing to have a data types and asset folders can then be removed)
- Since models will only be interacting with a subset of cached internal data, store the list of hashes within the model the model is interacting with (so, again, no notion of sequence and file data sets, only lists of data).
I'm not sure how deep the rabbit hole will be but I imagine we can sort directly the internal model hash lists instead of ever having to nuke data structures. The rework will finally also allow better file update handling to make sure file and folder changes can be efficiently reflected back to the views and models and even, allow inserting and popping of items from the view without needing to brute reset.
The data module, and more generally, the ways internal data is stored, read and updated is due for a refactoring.
In its current form, the item models are the authors of the internal data but as Bookmarks is growing and more elements rely on the internal data this old implementation doesn't scale at all. Item models instead should be regular consumers. Additionally, the current code is flawed! It stems from a misunderstanding of basic threading concepts, and a misinterpretation of the function of the weakref module (at least I'm learning, ey) resulting in race conditions and data corruption.
Here's what I'm thinking of doing:
I'm not sure how deep the rabbit hole will be but I imagine we can sort directly the internal model hash lists instead of ever having to nuke data structures. The rework will finally also allow better file update handling to make sure file and folder changes can be efficiently reflected back to the views and models and even, allow inserting and popping of items from the view without needing to brute reset.