Regression: 2.15.0 breaks deserialization for records when mapper.setVisibility(PropertyAccessor.ALL, Visibility.NONE);

[stolsvik]

Describe the bug
This code used to work with 2.14.2, but not with 2.15.0

ObjectMapper mapper = new ObjectMapper();
mapper.setVisibility(PropertyAccessor.ALL, Visibility.NONE);
RecordTest recordTest_deserialized = mapper.readValue("{}", RecordTest.class);

Version information
2.15.0

To Reproduce

/**
 * This works fine with Jackson 2.14.2, but not with 2.15.0.
 */
public class Jackson_2_15_0_Regression {

    record RecordTest(String string, int integer) {
    }

    @Test
    public void emptyJsonToRecord() throws JsonProcessingException {
        ObjectMapper mapper = new ObjectMapper();

        mapper.setVisibility(PropertyAccessor.ALL, Visibility.NONE);
        // mapper.setVisibility(PropertyAccessor.FIELD, Visibility.ANY);

        RecordTest recordTest_deserialized = mapper.readValue("{}", RecordTest.class);


        System.out.println("RecordTest deserialized: " + recordTest_deserialized);
        Assert.assertEquals(new RecordTest(null, 0), recordTest_deserialized);
    }
}

Comment out the mapper.setVisibility(PropertyAccessor.ALL, Visibility.NONE) and it works.

Note that the commented-out mapper.setVisibility(PropertyAccessor.FIELD, Visibility.ANY) is included in my actual code to get intended behaviour (i.e. only fields are serialized and deserialized, no methods involved), but this did not make any to or from for the problem.

Exception is:

com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `io.mats3.examples.jbang.Jackson_2_15_0_Regression$RecordTest` (no Creators, like default constructor, exist): cannot deserialize from Object value (no delegate- or property-based Creator)
 at [Source: (String)"{}"; line: 1, column: 2]

	at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:67)
	at com.fasterxml.jackson.databind.DeserializationContext.reportBadDefinition(DeserializationContext.java:1915)
	at com.fasterxml.jackson.databind.DatabindContext.reportBadDefinition(DatabindContext.java:414)
	at com.fasterxml.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1360)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1424)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:352)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:185)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4825)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3772)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3740)
	at io.mats3.examples.jbang.Jackson_2_15_0_Regression.emptyJsonToRecord(Jackson_2_15_0_Regression.java:25)

[pjfanning]

Seems to work if you don't change the mapper visibility setting - see #3907. Would it be possible for you to not set this value?

This issue will still be investigated. I'm just making the suggestion above as a workaround. I'm no expert on the benefits of setting the visibility to non-default values.

[yihtserns]

Fix/workaround

Allow visibility for PropertyAccessor.CREATOR:

mapper.setVisibility(PropertyAccessor.ALL, Visibility.NONE);
mapper.setVisibility(PropertyAccessor.CREATOR, Visibility.ANY);

Explanation

Previously, Records deserialization is specially handled, resulting in some behavioural differences (mostly missing behaviours) compared to POJO deserialization.

#3724 was done to address those differences, to close the gap between Records vs POJO de/serialization:

1. This means now, Jackson looks for deserialization "creators" for Records the same way it looks for POJO's.
2. setVisibility(PropertyAccessor.ALL, Visibility.NONE) "hides" those "creators", because PropertyAccessor.ALL includes PropertyAccessor.CREATOR.

[yihtserns]

(Interesting to see the different config used to utilise Jackson as only a deserializer not serializer: here & #3897 🤔.)

[stolsvik]

Thanks.

This is used both for serialization and deserialization. It was the deserialization that failed - here the "assert that this will actually work with an empty {} object" boot-up verification - and it only failed with records.

I need this config to work both for 2.14.2 and 2.15. Will that new suggested configuration do that just the same? That is: I want ONLY fields evaluated, with any visibility, for both classes and records. No methods whatsoever. These are DTOs, if people slap methods on it, I don't care: I want the data transferred. I guess I found that config somewhere sometime, and it have done exactly what I wanted for at least 6 years (used to be just classes, and then it also worked just as expected with records. Nice.)

(PS: I cannot really "test if it works" with an alternate config, as this is a "foundation lib", there are at least 50 different random services that depend on this just working as it always has. I just wanted to upgrade to keep with the times! :-) That bit pretty hard, both this, and the 5M char limit for String: FasterXML/jackson-core#863 (comment) )

(PPS: The bullet 2 where "setVisibility(PropertyAccessor.ALL, Visibility.NONE) "hides" those "creators", because PropertyAccessor.ALL includes PropertyAccessor.CREATOR" on the face of it sounds like a clear bug to me. For a record, there is no other proper way to get this thing made, so they should probably not be included? At least not the canonical constructor?)

[cowtowncoder]

One thing worth noting: Records are technically very different from POJOs/Beans, so there are challenges from Jackson implementation side. Especially regarding access to Record Fields; something that will start failing on newer JDKs. It is not certain handling can be fully unified.

I agree that it is important to be able to have value types, handling that works for both 2.14 and 2.15. But we really did not realize that there was Record usage that relied on explicit Visibility settings -- there's no testing so one can view at as unsupported use case, technically.
Assumption rather was that usage wrt Records would use default visibility settings.

[stolsvik]

@cowtowncoder Thanks!

Records are technically very different from POJOs/Beans, so there are challenges from Jackson implementation side.

yes, I agree, which makes me wonder a bit about this plan of jamming records and POJOs into the same regime, as seems suggested by @yihtserns?

Assumption rather was that usage wrt Records would use default visibility settings.

This is really not the point: The point is that I want to use the same ObjectMapper no matter what my users are choosing: POJOs (as they have for 8 years), or records (the new kid). There is thus a jumble of some POJOs here, and some records there, people now also migrating due to the extreme terse-ness and nice-ness of records. I feel it makes little sense to have to handle this differently? How should I even do that? A POJO could have a record inside it, and other way around.

[cowtowncoder]

Records are technically very different from POJOs/Beans, so there are challenges from Jackson implementation side.

yes, I agree, which makes me wonder a bit about this plan of jamming records and POJOs into the same regime, as seems suggested by @yihtserns?

Right... and which is why code originally was separate.

I had plans to rewrite Introspection of POJOs, Records, but didn't have time so others had a chance to try out alternatives here.

Assumption rather was that usage wrt Records would use default visibility settings.

This is really not the point: The point is that I want to use the same ObjectMapper no matter what my users are choosing: POJOs (as they have for 8 years), or records (the new kid). There is thus a jumble of some POJOs here, and some records there, people now also migrating due to the extreme terse-ness and nice-ness of records. I feel it makes little sense to have to handle this differently? How should I even do that? A POJO could have a record inside it, and other way around.

Sorry, I did not mean that as a justification for breaking things: but rather explanation of why this use case was not covered by tests. I understand it was used as a way to unify handling with 2.14 when there was no other way.
I agree in that the same ObjectMapper should be usable.

I'd have to dig up where it was done, but the visibility checker defaults were forked for 2.15 (or was it 2.14 already?) and I think there's a way to change them separately as well. I just do not remember details.

EDIT: I think I got confused with above: visibility checker defaults had to do with different defaults for JDK types, where we do NOT want to detect fields. Records (outside of JDK packages) not affected by that.

[yihtserns]

This is used both for serialization and deserialization. It was the deserialization that failed - here the "assert that this will actually work with an empty {} object" boot-up verification - and it only failed with records...
...I want ONLY fields evaluated, with any visibility, for both classes and records. No methods whatsoever. These are DTOs, if people slap methods on it, I don't care: I want the data transferred.

Test result for setVisibility(PropertyAccessor.ALL, Visibility.NONE) with 2.14.2:

Mechanism (without annotations)	Serialization	Deserialization with non-empty JSON object	Deserialization with empty JSON object
No-arg constructor with getters & setters	❌ Failed: "no properties discovered to create BeanSerializer"	❌ Failed: "0 known properties"	✔ Successful: created empty instance
Constructor creator	❌ Failed: "no properties discovered to create BeanSerializer"	❌ Failed: "no Creators exist"	❌ Failed: "no Creators exist"
Record class	❌ Empty JSON Object	✔ Successful: created instance using the JSON fields	✔ Successful: created instance using default values

The only way the failing tests above can work is when @JsonCreator/@ConstructorProperties & @JsonProperty annotations are used on the constructor & fields, respectively. @stolsvik is that what was done in your codebase?

[yihtserns]

I'd have to dig up where it was done, but the visibility checker defaults were forked for 2.15 (or was it 2.14 already?)...

3.x

I think there's a way to change them separately as well...

The only way I know is this:

objectMapper.registerModule(new SimpleModule() {
    @Override
    public void setupModule(SetupContext context) {
        super.setupModule(context);
        context.insertAnnotationIntrospector(new NopAnnotationIntrospector() {
            @Override
            public VisibilityChecker<?> findAutoDetectVisibility(AnnotatedClass ac, VisibilityChecker<?> checker) {
                return ac.getType().isRecordType()
                        ? checker.withCreatorVisibility(JsonAutoDetect.Visibility.NON_PRIVATE)
                        : checker;
            }
        });
    }
});

Alternatives

1. Annotate the override directly on the Records:

@JsonAutoDetect(creatorVisibility = Visibility.NON_PRIVATE)
record RecordTest(String string, int integer) {
}

2. Annotate @JsonCreator on the canonical compact constructor:

record RecordTest(String string, int integer) {
    @JsonCreator
    RecordTest {
    }
}

I want to use the same ObjectMapper no matter what my users are choosing: POJOs (as they have for 8 years), or records (the new kid). There is thus a jumble of some POJOs here, and some records there, people now also migrating due to the extreme terse-ness and nice-ness of records. I feel it makes little sense to have to handle this differently? How should I even do that? A POJO could have a record inside it, and other way around.

...which was the whole point of #3724's "jamming records and POJOs into the same regime", so that using Records is no (or not much) different from using POJO - the issue you're facing now is because Records & POJOs are handled similarly...

[yihtserns]

I'd have to dig up where it was done, but the visibility checker defaults were forked for 2.15 (or was it 2.14 already?)...

3.x

@cowtowncoder Do you think we should copy that to 2.15? E.g.:

public abstract class MapperConfigBase ... {
    ...
    public final VisibilityChecker<?> getDefaultVisibilityChecker(Class<?> baseType, AnnotatedClass actualClass) {
        ...
        if (ClassUtil.isJDKClass(baseType)) {
            vc = VisibilityChecker.Std.allPublicInstance();
        } else if (baseType.isRecord()) {
            // For Records, ignore any config that hides constructor & factory method creators e.g. via
            // - ObjectMapper.setVisibility(PropertyAccessor.ALL, Visibility.NONE)
            // - ObjectMapper.setVisibility(PropertyAccessor.CREATOR, Visibility.NONE)
            // - ObjectMapper.disable(MapperFeature.AUTO_DETECT_CREATORS)

            vc = getDefaultVisibilityChecker().withCreatorVisibility(Visibility.NON_PRIVATE);
        } else {
            vc = getDefaultVisibilityChecker();
        }
        ...
    }
}

[stolsvik]

@yihtserns:

is that what was done in your codebase?

No, as I said at top, I have both of:

ObjectMapper mapper = new ObjectMapper();
mapper.setVisibility(PropertyAccessor.ALL, Visibility.NONE);
mapper.setVisibility(PropertyAccessor.FIELD, Visibility.ANY);

The full init is:

ObjectMapper mapper = new ObjectMapper();

// Read and write any access modifier fields (e.g. private)
mapper.setVisibility(PropertyAccessor.ALL, Visibility.NONE);
mapper.setVisibility(PropertyAccessor.FIELD, Visibility.ANY);

// Drop nulls
mapper.setSerializationInclusion(Include.NON_NULL);

// If props are in JSON that aren't in Java DTO, do not fail.
mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);

// Write e.g. Dates as "1975-03-11" instead of timestamp, and instead of array-of-ints [1975, 3, 11].
// Uses ISO8601 with milliseconds and timezone (if present).
mapper.registerModule(new JavaTimeModule());
mapper.configure(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS, false);

// Handle Optional, OptionalLong, OptionalDouble
mapper.registerModule(new Jdk8Module());

It is here: https://github.com/centiservice/mats3/blob/main/mats-serial-json/src/main/java/io/mats3/serial/json/MatsSerializerJson.java#L120-L152 - now also with the attempt at handling the 5M chars-in-String limit.

Note: I had accepted that I always need a no-args constructor (as opposed to GSON). (Note: If it is possible to support missing no-args constructor for a class, even with Java 17+, that would be excellent!)

the issue you're facing now is because Records & POJOs are handled similarly...

Exactly. Which is why I said: "yes, I agree, which makes me wonder a bit about this plan of jamming records and POJOs into the same regime, as seems suggested by @yihtserns?"

My point is that the construction of a Record and of a Class, and their population of fields, are rather different, so it sounds .. ambitious .. to do it with the same codebase/flow? Note, I do not try to dictate anything here, it is just an observation. But it would be nice if this - as seen from my side - regression - was not present!

[yihtserns]

No, as I said at top, I had both of:

ObjectMapper mapper = new ObjectMapper();
mapper.setVisibility(PropertyAccessor.ALL, Visibility.NONE);
mapper.setVisibility(PropertyAccessor.FIELD, Visibility.ANY);

OK, was confused because it was commented out in the sample code.

That means you might actually have 2 problems:

1. This issue for Records deserialization.
2. 2.15.0 breaking behaviour change for records and Getter Visibility #3895 for Records serialization.