OutOfMemoryError resolving javax.jcr.Node since 2.2.41 — AnnotatedType.isSubtype in equals()/hashCode() splits the resolver cache #5292

Describe the bug

Resolving a single javax.jcr.Node (or Session, Workspace, Item) exhausts the heap on 2.2.41 and every release since, including 2.2.53. On 2.2.40 the same call completes in well under a second.

This looks related to #5091, but #5114 does not fix it — see "Status of #5114" below.

Version

Affected: 2.2.41 through 2.2.53 (all versions tested in that range). Last good: 2.2.40.

Tested with swagger-core-jakarta + swagger-jaxrs2-jakarta, JDK 17, javax.jcr:jcr:2.0.

To Reproduce

pom.xml:

<project xmlns="http://maven.apache.org/POM/4.0.0">
  <modelVersion>4.0.0</modelVersion>
  <groupId>repro</groupId>
  <artifactId>swagger-jcr-oom</artifactId>
  <version>1.0</version>
  <properties>
    <maven.compiler.release>17</maven.compiler.release>
    <swagger.version>2.2.41</swagger.version>
  </properties>
  <dependencies>
    <dependency>
      <groupId>io.swagger.core.v3</groupId>
      <artifactId>swagger-core-jakarta</artifactId>
      <version>${swagger.version}</version>
    </dependency>
    <dependency>
      <groupId>javax.jcr</groupId>
      <artifactId>jcr</artifactId>
      <version>2.0</version>
    </dependency>
  </dependencies>
  <build><plugins>
    <plugin>
      <groupId>org.codehaus.mojo</groupId>
      <artifactId>exec-maven-plugin</artifactId>
      <version>3.1.0</version>
      <configuration>
        <executable>java</executable>
        <arguments>
          <argument>-Xmx512m</argument>
          <argument>-classpath</argument>
          <classpath/>
          <argument>Repro</argument>
        </arguments>
      </configuration>
    </plugin>
  </plugins></build>
</project>

src/main/java/Repro.java:

import io.swagger.v3.core.converter.ModelConverters;

public class Repro {
    public static void main(String[] args) {
        ModelConverters.getInstance().readAll(javax.jcr.Node.class);
        System.out.println("completed");
    }
}

mvn -Dswagger.version=2.2.40 clean compile exec:exec   # prints "completed"
mvn -Dswagger.version=2.2.41 clean compile exec:exec   # OutOfMemoryError
mvn -Dswagger.version=2.2.53 clean compile exec:exec   # OutOfMemoryError

The heap is capped at 512 MB via exec:exec so the failure is prompt and visible.

• 2.2.40 → prints completed in ~400 ms
• 2.2.41 / 2.2.53 → java.lang.OutOfMemoryError: Java heap space within a few seconds

It also reproduces through io.swagger.v3.jaxrs2.Reader on a JAX-RS resource with a single @GET method returning javax.jcr.Node, i.e. the normal springdoc/swagger-jaxrs2 code path — so it is not specific to calling ModelConverters directly.

Which JCR types are affected (same run, -Xmx512m):

Repository and NodeType are useful negative controls: neither exposes a bean property that reaches javax.jcr.Session.

Expected behavior

javax.jcr.Node resolves without exhausting the heap, as it does on 2.2.40.

Stack trace

Top of the 2.2.41 failure — the recursion is deserializeArbitraryOrObjectSchema → convertValue → deserialize → deserializeArbitraryOrObjectSchema:

java.lang.OutOfMemoryError: Java heap space
  at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3855)
  at io.swagger.v3.core.util.Json31.jsonSchemaAsMap(Json31.java:76)
  at io.swagger.v3.core.util.ModelDeserializer.deserializeArbitraryOrObjectSchema(ModelDeserializer.java:137)
  at io.swagger.v3.core.util.ModelDeserializer.deserialize(ModelDeserializer.java:98)
  at io.swagger.v3.core.util.ModelDeserializer.deserialize(ModelDeserializer.java:34)
  at com.fasterxml.jackson.databind.deser.std.MapDeserializer._deserializeNoNullChecks(MapDeserializer.java:891)
  ...
  at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:4594)
  at io.swagger.v3.core.util.ModelDeserializer.deserializeArbitraryOrObjectSchema(ModelDeserializer.java:131)
  at io.swagger.v3.core.util.ModelDeserializer.deserialize(ModelDeserializer.java:98)
  ...

Analysis

Bisect. The complete swagger-core class delta from 2.2.40 to 2.2.41 is two classes: io.swagger.v3.core.converter.AnnotatedType and io.swagger.v3.core.jackson.ModelResolver (plus ModelResolver's inner classes). Every other class in the jar is byte-identical — including ModelDeserializer, despite it being where the heap is consumed. That delta is
#5004.

Mechanism. #5004 added an isSubtype field to AnnotatedType, and it participates in equals() and hashCode(). AnnotatedType is the cache key in ModelConverterContextImpl, so the same Java type resolved once as an ordinary property and once as a subtype now occupies two distinct cache entries. The resolver consequently inlines a nested schema where 2.2.40 emitted a $ref, and that oversized tree is then round-tripped through ObjectMapper.convertValue() inside ModelDeserializer.deserializeArbitraryOrObjectSchema(), which recurses on itself and allocates until the heap is gone.

This is not a resolution-count explosion. Instrumenting the converter chain with a counting ModelConverter shows the OOM arriving after only ~130 resolve() calls, with essentially every type resolved once. The cost is in the size of the schema objects, not in repeated resolution. Worth flagging because it distinguishes this from #5091, whose symptom was an exponential growth in resolve() invocations.

Status of #5114

#5114 closed #5091 by removing cache-defeating ModelConverterContext reinstantiation in AnnotationsUtils. It does not address this:

• On 2.2.53 the JCR repro above still OOMs.
• On a synthetic control workload (a 12-node full mesh with declared subtypes and @ArraySchema(schema = @Schema(implementation = …)) on every reference), 2.2.53 produces resolve() counts identical to 2.2.41 — 816 at N=12, against 480 on 2.2.40.
• AnnotatedType.equals()/hashCode() still include isSubtype on 2.2.53.

Suggested direction

Excluding isSubtype from AnnotatedType.equals()/hashCode() would restore cache hits across the plain-property/subtype boundary, but presumably risks reintroducing whatever #5004 fixed. A separate guard — keeping the flag on the type but out of the cache identity, or bounding the inlining depth in deserializeArbitraryOrObjectSchema — may be safer.

Additional context

We hit this in a CMS whose REST tooling exposes JCR-backed types; loading the OpenAPI document made instances unresponsive. We are pinned to 2.2.40 as a mitigation.

One caveat on the reproduction: it needs the JCR types. We tried to reduce it to a dependency-free synthetic case and could not.

So something about the particular combination in the javax.jcr type graph is required, and we have not isolated which part.

That does not weaken the repro above — it runs off two files and one dependency, and javax.jcr:jcr:2.0 is small, public, and unchanged since 2009, so it should stay reliable indefinitely.