Add structs to GDScript

[voylin]

This PR partly solves the proposal to add structs to Godot (godotengine/godot-proposals#7329) and is the proposal I used for deciding how to implement structs. Been working on this nearly full-time for the past week and I've gotten everything working and testing (numbers are below).

Why structs?

There is no good way for storing just data in GDScript:

• Object/Class/Resource -> Heavy memory footprint and includes functionality not needed for simple data containers (again, numbers below of the differences);
• Dictionary -> No code completion and no static typing for individual key/values;
• Array -> no named fields, easily gets messy when trying to structure data;

Let me know if I'm wrong about any of this but:
GDScript classes/resources/objects includes extra features which normal data containers don't need, causing memory overhead. If we would have a scene with thousands of enemies, players, or other things to keep track off (basically heavy data intensive games), using resources or classes for that would have a much higher memory footprint compared to what using structs would have.

My implementation

As said before, I've been mainly following the proposal (as I forgot the existence of rocketchat where a lot of discussing about the implementation of structs has been taken place ...) and I've gone ahead and use the Array stuff method for making the struct feature work.

If you're worried about the Array's having taken a performance hit, don't ... numbers are bellow. Serialization also got tested, and can be tested with the test room project.

Creating a struct

struct MyStruct:
    var a: String
    var b: int
    var c # Non-typed variables should also work ... shouldn't be used though :p

Instantiating a struct

func _ready() -> void:
    var test_one: MyStruct = MyStruct()
    test_one.a = "hello"
    test_one.b = 22
    # ...or ...
    var test_two: MyStruct = MyStruct("hello", 22)

What's not included in this PR?

The proposal talks about "FlatArray's", so basically Packed array's for structs. This is something I haven't implemented in this PR. If this PR get's merged I'll get into working on that feature. Packed struct array's would help in improving the cpu cache friendliness.

Exporting structs to be visible in the editor is something which I haven't bothered with just yet as it exceeds the scope (imo) of this PR. Printing struct instances will give you an array of their data, without the names of the variables. That's also something I would add in an extra PR if needed. I got challenged, so I implemented it. It's not 100% working correctly for struct instances in dictionaries, but if this PR has more of a chance to get merged I'll spend more time on fixing this.

The feature of defining variable values by name also got mentioned in the proposal, but I'm not certain if this should be added or not in this PR and if this functionality gets added, shouldn't it be added for functions in general? (Just not certain about this one so I ignored it)

func _ready() -> void:
    var test: MyStruct = MyStruct(
        a = "hello",
        b = 22 )

The proposal also talked about this:

var gamedata: struct:
    var score: int
    var enemy_settings: struct:
        var speed: float

However this feels messy and unclear, so I haven't implemented this part in this PR.

Numbers

And now what you've all been waiting for!! :D Numbers!!

Memory

So I did a memory test with 100 000 elements with each having the member variables:String, int, and an untyped variable:

• 231 bytes : Struct - 404 msec
• 556 bytes : Dictionary - 228 msec
• 1012 bytes : Object - 422 msec
• 1040 bytes : RefCounted - 589 msec
• 1169 bytes : Resource - 570 msec
• 1444 bytes : Node - 462 msec

NOTE: Objects (including RefCounted) have lower memory consumption in actual projects since it doesn't contain GDScriptInstance::member_indices_cache which is used for hot reloading.

The bytes is the size of 1 instance and the amount of seconds is how long it took to create those 100 000 elements. I tried to make the tests as good as possible, but just in case they are available in the test room which I added to this PR for other people to test.

Array performance

Since we're using Array's, some stuff needed changing. Because of that I felt the need to add an extra test to check the performance difference between the normal master branch, and the one with this PR. The timings are from 10 000 000 iterations:

Operation	With structs	Master
Append	3496.942	3482.418
Sequential Read	2470.889	2205.027
Iteration	1462.498	1587.494
Sequential Write	2359.047	2421.681
(times are in msec)

Basically, what I found of running the test multiple times ... it's all within margin or error as they beat each other the whole time.

I do want to stress that these numbers are a bit all over the place thanks to me not being able to keep my system running at a correct speed.

TODO's:

• Add documentation; (I couldn't figure out this part
  just yet so I'll wait for feedback on this)

Test room

For the people wanting to see how I got the numbers:
struct_test_room.zip

The benchmark.gd file should be run from the terminal with --headless -s benchmark.gd but as mentioned before, the timings will always be different each run.

[ValorZard]

Can structs have default values?
so something like

struct MyStruct:
    var a: String = "Hello, World"
    var b: int = 42
    var c  = 67 # Non-typed variables should also work ... shouldn't be used though :p

[voylin]

Can structs have default values? so something like

struct MyStruct:
    var a: String = "Hello, World"
    var b: int = 42
    var c  = 67 # Non-typed variables should also work ... shouldn't be used though :p

Yep, forgot to mention that 😅

[EIREXE]

I think having to write MyStruct.new() for constructing the struct may be more idiomatic, but otherwise this looks nice.

[Ivorforce]

By "array based", do I understand correctly that you implemented the proposed approach from godotengine/godot-proposals#7329 (implementing structs in terms of Array)?

Did you see in the discussions of the proposal that this approach was rejected in the past?

[voylin]

By "array based", do I understand correctly that you implemented the proposed approach from godotengine/godot-proposals#7329 (implementing structs in terms of Array)?

Did you see in the discussions of the proposal that this approach was rejected in the past?

For as far as I could find, it was mentioned that too many "hacks" were needed in the Array class. With the changes I've made there were no side effects on the performance of the Array class. This is a good base to further build structs upon, in my opinion at least.

Performance to the existing parts of the engine reveals no regressions and there's a way more memory friendly way of handling data in games and applications.

As mentioned in RocketChat, I already started using the proposal for the implementation and found that I could make it work. After reading the discussion's about structs and the "possible problems", I decided to keep working on it as I wanted to finish what I started and do the testing later to see if there would be regressions or not. To my surprise I couldn't find any performance issues (even with the few extra check's array's need to do to check if they're a struct or not) and as already mentioned above, memory consumption went down more then 4 times compared to objects and nearly twice the size of Dictionaries with the test data I used.

Feel free to close this PR if you don't think this holds any future, but I personally think that it would be a missed opportunity.

[Ivorforce]

The problems of using Array were more fundamental than just performance concerns — they regarded complexity and implications on core as well.
I haven't been part of the discussions when this approach was rejected back then, so i won't reject this PR for my own lack of context. But I think using the Array approach, this PR has a low chance to be merged.

[voylin]

I understand that there are reasons why this PR can get refused, but I hope the code gets looked at as the complexity which was talked about that gets added to core is quite small. Most of the complexity is based inside of the GDScript module.

Adding a "real" and "pure" struct implementation feels like it would break too many parts of the codebase at this point in time and doesn't seem likely to happen in 4.x, but more likely when the switch to 5.0 is being made. To the maintainers in charge of this part of the code base: This could create a solution for 4.x users to have access to these memory optimizations.

But if the core team decides that this approach is a dead-end for the 4.x lifecycle, I'll accept that and I'll just maintain this as a custom fork for my own tools. ;)

[scgm0]

I would like to know if the implementation of this struct can support C# and GDE, rather than being limited to GDS? If multi-language support cannot be added, it would be impossible to switch some existing APIs to use structs, and the benefits it brings would seem limited.

[belzecue]

But if the core team decides that this approach is a dead-end for the 4.x lifecycle, I'll accept that and I'll just maintain this as a custom fork for my own tools. ;)

Jumping in to thank Voylin both for this PR (and the week of effort it took) and their exemplary attitude. Like many curious lurkers, I've enjoyed following behind-the-scenes developments for some years, watching PRs come and go. Some shoot skyward like rockets. Some struggle to lift off. Some never ignite and sit abandoned. In all cases somebody has to be in control of the launchpad -- and the decision to abort. Here, it's the maintainers. We don't have to agree with that decision. It's perfectly okay to disagree politely and respectfully, but not forcefully. If the maintainers don't see things your way, all good, there's always 'my fork' and no hard feelings.

So, yeah, nice to see a PR that probably won't see the Karman Line but will serve as an example for others.

[eimfach]

Maybe their opinion changes but the core team believes that reworking Object to be lightweight is what they want to do ... They seem to believe also that there is not an actual problem. Maybe it would be worth noting which real life problems this PR in particular solves, like in the Video Editor you are working on.

godotengine/godot-proposals#7329 (comment)

This is related to a working draft that was made before.

I fear that the idea of Structs as small Aggregates of data is not a thing anymore, as is the proposal. Unless we can show it by example Projects and underpin its existence.

[Ivorforce]

Maybe their opinion changes but the core teams believes that reworking Object to be lightweight is what they want to do ...

Please don't misrepresent the issue. The actual status: Some maintainers believe this to be worth exploring. There is no maintainer consensus on the matter. If there was, we'd make a public statement.

[nlupugla]

Nice work @voylin! Really impressive to me that you put this all together in a week! It took me wayyy longer to put together my crack at implementing structs. I saw you mentioned that you weren't sure about the documentation. I got started on the documentation, but it's been a while so I don't remember how far along I got, but you're welcome to look through it #97297.

Regarding questions about Struct semantics, there was some good discussion on this page godotengine/godot-proposals#7903 that might be relevant for this implementation as well.

[Brogolem35]

Are there any reasons to modify the existing Array than to add a new Variant? Also, do you have benches on access time?

[eimfach]

Maybe their opinion changes but the core teams believes that reworking Object to be lightweight is what they want to do ...

Please don't misrepresent the issue. The actual status: Some maintainers believe this to be worth exploring. There is no maintainer consensus on the matter. If there was, we'd make a public statement.

It's just what happened in the past before, just the opinion of parts of the core team, no mispresentation here, just the facts and that it might be helpful to create practical project examples therefore.

[CoolStopCode]

I think having to write MyStruct.new() for constructing the struct may be more idiomatic, but otherwise this looks nice.

Yes, initializing structs with MyStruct() would be inconsistent with classes and resources, so MyStruct.new() fits more neatly.

[voylin]

I think having to write MyStruct.new() for constructing the struct may be more idiomatic, but otherwise this looks nice.

Yes, initializing structs with MyStruct() would be inconsistent with classes and resources, so MyStruct.new() fits more neatly.

However, looking at Vector's by example it uses just (). Since structs themselves don't really hold any functions using MyStruct() seemed like the better way of handling it.

[HolonProduction]

Exporting structs to be visible in the editor is something which I haven't bothered with just yet as it exceeds the scope (imo) of this PR.

Do you have a concept on how this will work? While it might not be necessary to implement it it should not be an afterthought. I'm skeptical on whether this can be achieved without either formalizing structs in the core type system or doing massive hacks in the editor, but if you have a good solution let's hear it.

---

Also noticed a reflection bug. get_property_list returns the following for a prop that is typed with a struct: { "name": "b", "class_name": &"MyStruct", "type": 28, "hint": 0, "hint_string": "", "usage": 4096 } the class_name here makes no sense. This identifier is not unique and also not known to any systems other than GDS.

[voylin]

Do you have a concept on how this will work? While it might not be necessary to implement it it should not be an afterthought. I'm skeptical on whether this can be achieved without either formalizing structs in the core type system or doing massive hacks in the editor, but if you have a good solution let's hear it.

Thanks for the reply! I updated the commit to remove the reflection bug.

As for the exporting of structs to the editor (by using @export var test: MyStruct or by having struct instances inside of exported array's and dictionaries), this could be achieved kind of easily without adjusting core or any massive hacks, since a Struct is just an Array after all.

Structs will just need to be handled (in a better way) by the property system so most of the logic would be inside of the GDScript module and in the editor UI. I haven't really messed around in the editor UI part of the source code so that might take a bit of time to figure it out, but nothing I can't handle at this point.

I started trying to implement this export stuff to show that core won't need to be touched and that we can almost fully rely on the property system to make this work. I'll add it as an extra commit to have a clearer view of what needs to be added on top of this implementation of structs.

[HolonProduction]

Structs will just need to be handled (in a better way) by the property system ...

That's what I mean by formalizing them in the core type system. Property info is part of the core reflection API. And if structs can appear in there (which they will have to, since the editor is only communicating with GDS via the unified core API's) that means they become part of the core type system.

[AThousandShips]

I think that merging structs before we have a proper first-class type system in place would be irresponsible, it'd set us up for a lot of extra work and issues when actually implementing the type system.

Best case scenario we "just" have to rewrite a lot of struct related code to adjust it, but we might have to simply toss out all the original code and redo it from scratch if it's too hard to just rework it. A lot of extra work that could be avoided.

But I think a more serious aspect is that we open ourselves up for compatibility issues. What if the type system places certain restrictions or requirements that means that we have to change some aspect of struct use, be that in GDScript, or engine modules, or extension code. We can't really anticipate such details as we don't know how the type system will be designed. And even if we can avoid breaking compatibility we risk creating unnecessary constraints and requirements for that type system complicating the work implementing it to adapt the struct system to it, and avoiding breaking compatibility or to avoid unnecessary work in other areas to adjust

This is already going to be enough of a problem with existing type considerations, for example with the existing typed array and dictionary system, so I'd say it's a bad idea to add to that

Since one of the goals of structs is to gradually replace the use of dictionaries for return values in engine systems, like property info, return values from physics queries, etc., to improve type safety and usability by providing autocomplete and introspection, we would be held back by either waiting for a type system even though we have structs because we don't want to risk having to break the use in those places, or risk breaking things by replacing things too early.

---

tl;dr; Structs would be great, but the engine isn't, IMO, ready for it yet and we risk building up unnecessary tech debt and extra work by adding it too early

[HolonProduction]

• 231 bytes : Struct - 404 msec

• 556 bytes : Dictionary - 228 msec

• 1012 bytes : Object - 422 msec

• 1040 bytes : Class (inner) - 589 msec

• 1169 bytes : Resource - 570 msec

• 1444 bytes : Class (outer) - 462 msec

A little word on those results. The labeling is a bit misleading: "Class (outer)" is a Node, "Class (Inner)" is a RefCounted and and Object directly inherits Object. Whether the class is inner or outer is not relevant to the results, the base type obviously is.

It's also important to note that Object only amounts to around 620 bytes when you patch out GDScriptInstance::member_indices_cache which is used for hot reloading and not present in release builds. (The memory benchmarking methods are not present in release either so that's why patching is needed.)

[voylin]

A little word on those results. The labeling is a bit misleading: "Class (outer)" is a Node, "Class (Inner)" is a RefCounted and and Object directly inherits Object. Whether the class is inner or outer is not relevant to the results, the base type obviously is.

It's also important to note that Object only amounts to around 620 bytes when you patch out GDScriptInstance::member_indices_cache which is used for hot reloading and not present in release builds. (The memory benchmarking methods are not present in release either so that's why patching is needed.)

Thanks for letting me know of the correct naming, I updated the names in the PR and added a note about Object having a lower memory footprint in exported builds and the reason why.

[voylin]

To reply to the type system being the blocker:
I still need a bit more time to get the exporting working, I'll still add that commit once I'm ready with it. But why not have this solution in the current GDScript? Yes, parts will need to be re-done for the type system, but from what I can tell that's more for something like GDScript v.3 right?

You say this:

Best case scenario we "just" have to rewrite a lot of struct related code to adjust it, but we might have to simply toss out all the original code and redo it from scratch if it's too hard to just rework it. A lot of extra work that could be avoided.

When the type system gets implemented, it'll take Godot to Version 5 anyway, people will expect breaking changes as that's what the large version numbers are for. So if this structs implementation gets removed, what would be the big deal? It's not that other parts of the editor will rely on this struct implementation anyway. Maybe the editor can rely more on the structs implementation of the new type system but for as far as I can tell, there's no actively worked upon solution for structs to begin with, so deleting this one when Godot 5.x comes around the corner won't be much of a deal since you'd have to start from almost 0 anyway right?

[romgerman]

I think that merging structs before we have a proper first-class type system in place would be irresponsible

I am a bit confused about the "first-class type system" argument that gets thrown around by the maintainers when there is a new feature implementation or a proposal open to postpone them indefinitely. I don't understand why the typed arrays and dictionaries were made just to be a worse performing collections compared to the common array/dictionary class, break the flimsy type system once again (conversion is not possible without manually making a loop, built-in methods do not support the types and still return generic variant, worse performance, no nesting, and other issues) but other new features related to types, ease of developments, performance and convenience always get rejected or scrutinized for every little detail.

This is a great addition to GDScript, maybe it shouldn't be made exactly as it is right now in this PR and we all know we can discuss it and change the implementation for the better, but ouright rejecting it because of something that is and was up in the air for years and years is kind of discouraging.

[nlupugla]

Exporting structs to be visible in the editor is something which I haven't bothered with just yet as it exceeds the scope (imo) of this PR.

Do you have a concept on how this will work? While it might not be necessary to implement it it should not be an afterthought. I'm skeptical on whether this can be achieved without either formalizing structs in the core type system or doing massive hacks in the editor, but if you have a good solution let's hear it.

Also noticed a reflection bug. get_property_list returns the following for a prop that is typed with a struct: { "name": "b", "class_name": &"MyStruct", "type": 28, "hint": 0, "hint_string": "", "usage": 4096 } the class_name here makes no sense. This identifier is not unique and also not known to any systems other than GDS.

Just as a heads up, @ajreckof had started some work on getting struct to export in the inspector. It was a while ago, but from what I remember, the basics were working, but there was never a pull request for it, perhaps because it depended too much on my stuff or maybe it just wasn't polished to that point yet.

[ValorZard]

@romgerman I totally understand your frustration here (I had a similar issue with the way Physics Step has been handled)

however, GDType is actively being worked on. Here are some PRs that have recently been merged (or are in process of being merged)
#113586
#117451

I think the structs effort and the GDType effort can totally work together to the same goal here, up to maintainers and @voylin really to find a path forward (but I’m hopeful)

[voylin]

@HolonProduction Only had to add a new property hint to struct and the rest was done inside of the editor and the GDScript module.

I understand now that the structs should probably wait till GDType is ready after having spent more time chatting in Rocketchat, but I also can't help but feel like this could be an option in some way.

The current exporting has an issue that Struct instances in Dictionaries don't get the default values and an error about the array size which appears (but it doesn't cause any issues so it's probably a small thing which needs changing for it to go away).

Anyway, it was a lot of copy pasting so there's probably some other parts which need adjusting to make exporting work fully as expected, but from my amount of testing it seems to work.

[IntangibleMatter]

I've been in the discussion for structs quite literally since I was in high school and it seems like a perpetual cycle of "here's an implementation"->"we're waiting on the type system"->implementation blocked->no real updates on the type system, and it's been that way for years. While I'm glad to see that there is some real progress that's been made on the type system, I think a lot of us have gotten really impatient because we could've had several versions of the engine with a good enough implementation of structs at this point. The Type system is going to be a massive improvement to everything, but it feels like its blocking things that clearly don't depend on it, like this, which is different then the things that do depend on it, such as nested type collections. Structs would be so convenient even in an initial implementation, and even if they don't do everything that they were originally supposed to do (like the replacement of Dictionary for many return types in the core). I think my best summary was here as to what we want structs for.

At this point, I'm more than happy with "good enough" (especially with how that's been done for so many things that have been added in 4.X, especially early on) just so that I can start using them.

I support this PR, and I know I'm not a maintainer but I'd love if it got merged.