Exclude params and cons from the derivation based on annotations

[vigoo]

I should have opened an issue about this originally instead of discussing it immediately in a proposed implementation (#270) so I wanted to move it here now to restart discussion.

The problem

The real-world example for the requested feature came up while trying to migrate the semi-auto derived binary codecs in https://github.com/vigoo/desert/ from shapeless to magnolia. It supports marking case class fields and sealed trait constructors as transient. These transient subtrees of a data type may have types for which there is no implicit binary codec available and it cannot be automatically derived either. Even though the transient annotations can be observed from magnolia's combine/dispatch methods, as the derivation engine does not know about these application-specific exclusion rules it tries to gather typeclass instances for these members too, and fails.

An example for using these transient annotations is when serializing Akka-typed messages for remoting but using a subset of the message type for local-only communication, transferring non-serializable references (for example an akka stream or some local resources).

In the original PR I proposed a solution and then we discussed some alternatives. I did not check yet how the feature could be implemented in the new Scala 3 version.

It would be nice to support directly this with annotations because that way desert could be moved to Magnolia without changing its users code but maybe some breaking changes there are still better than maintaining a hand-written shapeless and a scala3 macro version in parallel ;)

What do you think?

[deusaquilus]

Hi Magnolia folks. Have there been any thoughts on this? I'm working for a company that would like to adopt Desert to do long-term serialization (hence the need for schema migration) but the presence of Shapeless is a non-starter because of compile-time issues. Could this issue be solved in magnolia on a lower level?

[adamw]

I don't think there has been any developments around this, but we're open to discussion and contributions :) I guess the main question is what would happen when calling caseClass.param.typeclass for a transient field? An exception would be thrown?

[deusaquilus]

I think that's reasonable. Just like calling .get on a None or a Failure.

[adamw]

Ok, well I don't see a reason not to support such transient fields. We just have to come up with a good name (@deriveTransient?) and implement :)

[vigoo]

@adamw so you suggest to have a built-in @deriveTransient (or whatever) annotation for this. I think that would work and probably the simplest approach.

Two things come to my mind:

• It should work both on fields and constructors
• In desert transient fields have a default value for deserialization so I was using two different annotations: https://github.com/vigoo/desert/blob/master/desert-core/src/main/scala/io/github/vigoo/desert/transient.scala If there is a general purpose transient annotation in magnolia then the library could have a @default(value) annotation and then for transient fields it would require these two annotations. Only drawback is that verifying that both exists would move to the derivation while with a single annotation it forces the user to immediately fill it with the value. On the other hand this default value parameter is too specific for my library (or the domain of serialization at least) so probably shouldn't be Magnolia's concern.

[adamw]

But the default would have to be per-typeclass, so I don't think it applies here.

A downside of an annotation is that it requires modifying the case class (derivation source), but then there's no other easy way of providing additional derivation metadata.

[vigoo]

One more thing came to my mind. If there is a global magnolia-defined transient annotation that would shortcut all derivations regardless of the typeclass which may not be a good idea. For example let's say we don't want to serialize transient fields with my serialization library but you may still want to derive a Show instance that works with the full data type.

To avoid this there could be some overrideable behavior in the derivation class

• at least to enable/disable the transient feature for a given derivation
• or, probably better, a isTransient(annotations: List[StaticAnnotation]): Boolean like function

The second would allow different libraries to define their own transient annotations which would also solve the typeclass-specific default value behavior.

Of course it may not be very likely that people are marking different subsets of their types for different type class derivations but still this feels like a safer/more generic solution to me.

[joroKr21]

I would suggest something more general again 😄 #279 (comment)

[adamw]

@joroKr21 I like that :) Even better, maybe we could simply parametrise the derivation macro with a code that would decide, if a field should be ignored or not (and possibly provide a default value)? That way you could use annotations (which could be one of the implementations of the parameter provided by default), but also implement other, e.g. name-based strategies. The code in question would be evaluated at compile-time of course.

[joroKr21]

I think that could work in Scala 3 where we have more options (by passing inline parameters and such) but not so much in Scala 2 where we can only pass additional type parameters to the macro.

[adamw]

I was thinking about Scala 3. I think the design is more or less sealed and "done" for Scala 2.

[vigoo]

Ah I was thinking about both Scala 2 and 3, because I think @deusaquilus 's use case is for Scala 2 for now.

@adamw thinking about the macro to decide if a field should be ignored or not - doesn't this lead us to an alternative where simply the typeclass provided for the derivation is lazy? That way the transient field support would be to simply check the annotations (or name etc) and does not access the derived typeclass at all. I'm not sure how hard would that be to implement though.

[deusaquilus]

Is lazy-derivation at compile-time possible? How do you do it?

[vigoo]

No I was just thinking it through better and realized that would not be possible. (At least I don't see how)

[deusaquilus]

Would @deriveTransient still be possibility for Scala 2? I think annotations, even non-configurable ones, are fine so long as they are simple and scarce. Coming from the DB world I have many, many scars from horrible things like @NamedQuery(<20 lines of code here>) but I don't think we need to religiously prohibit annotations based on those things.