Memory blowup with large verilog design

[mampcs]

Is your bug report related to the C++ slang project or the pyslang Python bindings?
C++ slang

Describe the bug
I've been trying to get a system verilog design that's about 1GB in size through slang. It's been blowing up in memory during def param resolution. Here's the data i have gathered so far:

1. After parsing, the CST is about 22GB. Of this, the tokens take up about 10.5 GB and from that the trivia is about 1.5GB. I added a CST memory profiler for this that provides a detailed breakdown of memory and count per syntax node. Additionally, it also provides a break up of memory usage per SV module/package/program.

2. The first visitor pass (generate level 0) via DefParamVisitor causes the RSS to increase from 24GB to 72GB. Of that, about 35GB is associated with the Configuration clone (initialClone) in the bump allocators. Presumably, the rest of it is in the backing arrays in SymbolMap, PointerMap and ConstantValue, though i can't be fully sure. The next DefParamVisitor within the same generate level 0 loop keep increasing the memory to the point where it got to 350GB and my process got killed.

3. I made a change to ensure that initialClone is deleted before we get to the next DefParamVistor visitor (that iterates maxDefParamSteps times). This resulted in the 48GB of memory associated with initialClone being returned back to the system (RSS got back to 24GB). That's how i'm certain that all the allocated memory was with the initialClone.

4. I'm going to try to add some more memory profiling to see if i can figure out what's going on. It's odd that the first visit took 48GB and completed but the second one took 325GB and still did not complete. Here's the two iterations i'm referring to:

          DefParamVisitor v(options.maxInstanceDepth, options.maxDefParamBlocks, generateLevel);
           initialClone.getRoot(/* skipDefParamsAndBinds */ true).visit(v);

            DefParamVisitor v(options.maxInstanceDepth, options.maxDefParamBlocks, generateLevel);
            c.getRoot(/* skipDefParamsAndBinds */ true).visit(v);

Will do some more digging to see what's going on. But any pointers/feedback would be much appreciated to speed this up.

To Reproduce
Cannot share the IP and have been unable to create a small testcase so far.

Additional context
Add any other context about the problem here. If you have a non-standard setup or anything unusual that may be contributing to the problem it's good to include that here.

[MikePopoloski]

Wow yeah, that's quite a large design. I'm definitely curious what's consuming all that memory on the first generate level... The AST is evaluated lazily and the defparam visitor basically only touches instances and generate blocks, so there shouldn't be that many AST nodes being created. Once you hit the second visitor there are values for the defparams, which can trigger a lot more stuff to be elaborated. If something is stuck in a loop it could grow quickly. There are a number of Compilation options that can limit the number of steps or the depth of the hierarchy that might be worth setting to low values and seeing if it causes things to complete.

[mampcs]

Here's a breakup of the CST memory (Sorry for the formatting):

--- CST (Concrete Syntax Tree) ---
Process RSS : 24.24 GB

Driver Components:
Source Manager : 1.12 GB
Syntax Tree Allocators : 22.84 GB
Library Map Allocators : 0 B

Syntax Node Data:
Total node memory : 21.72 GB
Total token memory : 10.34 GB
Total trivia memory : 1.58 GB

Node type breakdown (sorted by memory):
Node Kind Count Total Per Node
--------------------------------------------------------------------------------
NamedPortConnection 24830059 5.59 GB 241 B
HierarchicalInstance 6344926 2.60 GB 440 B
IdentifierName 35783189 2.11 GB 63 B
HierarchyInstantiation 5377840 1.37 GB 272 B
SimpleSequenceExpr 27953240 1.04 GB 40 B
Declarator 7171191 1002.99 MB 146 B
SimplePropertyExpr 27953223 853.06 MB 32 B
InstanceName 6343699 822.88 MB 136 B
IntegerVectorExpression 3939065 583.85 MB 155 B
IntegerLiteralExpression 6144512 450.34 MB 76 B
NetDeclaration 302139 418.59 MB 1.42 KB
EqualityExpression 3284920 403.56 MB 128 B
IdentifierSelectName 2798024 360.99 MB 135 B
SystemTimingCheck 458376 310.36 MB 709 B
LogicalAndExpression 2075522 297.32 MB 150 B
ElementSelect 2898118 265.57 MB 96 B
OrderedPortConnection 3152525 264.57 MB 88 B
PrimitiveInstantiation 967086 247.69 MB 268 B
PathDeclaration 561451 192.79 MB 360 B
PathDescription 561451 162.77 MB 303 B
AssignmentExpression 984911 142.59 MB 151 B
PortDeclaration 396873 129.90 MB 343 B
ConcatenationExpression 144082 123.29 MB 897 B
ConditionalPathDeclaration 534630 122.43 MB 240 B
ExpressionStatement 944099 116.68 MB 129 B
SequentialBlockStatement 365073 104.34 MB 299 B
ImplicitType 836903 104.22 MB 130 B
PortReference 1078105 103.31 MB 100 B
ConditionalStatement 344533 91.36 MB 278 B
ContinuousAssign 280703 88.70 MB 331 B
BitSelect 2726629 83.21 MB 32 B
ModuleDeclaration 92667 79.20 MB 896 B
VariablePortHeader 592063 79.07 MB 140 B
EdgeSensitivePathSuffix 208482 54.08 MB 272 B
ExpressionTimingCheckArg 1726260 52.68 MB 32 B
TimingCheckEventArg 633942 52.38 MB 86 B
TimingCheckEventCondition 617040 51.78 MB 88 B

[mampcs]

This might be better.

[mampcs]

Let me get a breakup of the AST memory after the first generate pass.
As a point of reference, VCS compiled this design in about 14GB.

[MikePopoloski]

That's just "compilation" or full elaboration? Pretty impressive if it's full elaboration in less space than slang takes for a basic parse tree.

[mampcs]

This is the peak memory VCS took for parsing, elaboration, optimization and code generation. The peak will typically occur during elaboration. And this also included the testbench code, which i wasn't parsing in my slang pass.

[mampcs]

As an experiment, i used slang to create a skeleton of the design that included only the modules and packages with the variable declarations, module instantiations, generate blocks, params and def params. This also ran out of memory similar to the full design. If I then removed the def params, it eventually went past elaboration after reaching a peak of 60GB. Will work on profiling the AST memory to see where the memory went. Will also try to narrow down where it's blowing up with defparams in there.

[MikePopoloski]

I'm curious why the per-node size of NetDeclarations in your example is > 1KB, which seems crazy since it doesn't even include the net names (which are under declarator syntax nodes). Is it like, a bunch of net decls happen to have a ton of leading trivia which skews the size? If so it might be worth removing the trivia from the per-node contributions. If not... maybe there's a bug in the accounting?

[MikePopoloski]

Also I wanted to double check that you're building a release version of slang when running the profiling? If SLANG_DEBUG is defined a bunch of structures (like SourceLocation) get much larger to track some stuff to make it easier to look at in a debugger. Probably you already are but just wanted to check just in case.

[mampcs]

No this is a release build.

[MikePopoloski]

I made some improvements to defparam resolution in 62fba91 which hopefully helps your case.

[mampcs]

I will try it out and let you know. Additionally, i can probably create an obfuscated version of the design with just the skeleton and give it to you. I have that already and it does reproduce the same memory blowup. Let me know if that would be helpful.

[MikePopoloski]

Yeah, being able to look at it in a debugger would be helpful for sure.