Deprecate upickle-based pickler

[adamw]

It was a fun experiment, but there's too many features which we can't support

[arturaz]

Are you talking about https://tapir.softwaremill.com/en/latest/endpoint/pickler.html ? Is there an intended replacement?

[adamw]

@arturaz Yes, we're exploring changing the underlying implementation of the json library from uPickle to jsoniter. The end-user interface should remain the same (or very similar)

[lomigmegard]

@adamw is there any decision or progress around the future of Pickler?

We were interested in leveraging it for new APIs, but after finding this thread and #3402 we are not sure of its future stability. Another blocker for us is the usage of uPickle 3 (see #4806)

Is there something SwissBorg could help with to arrive to a decision, or contribution to push it forward?

cc @tiagomota

[adamw]

I don't think there's much progress unfortunately. But I still think the idea of having derivation with single configuration for schemas & JSON is something Tapir should have, as it makes the code so much simpler. To do that, we have to pick a single JSON library that we'd like to support. But I'm not sure if we have a good answer to the "Scala JSON problem":

• circe is widely used, has an AST, but is moderate on the performance & security front, and dependency-heavy (cats)
• jsoniter is fast, but doesn't have an AST, documentation is so-so
• zio-json is secure, not sure about performance, but is tied to ZIO (same problem as circe above)
• upickle is better in performance than average, but is a pain to integrate with, due to the usage of path-dependent types

So, which do we pick? Our first bet was upickle, but we tried, and the code is just awful - upickle just isn't designed with such deep integrations in mind - it probably works great when used directly. So now I'm leaning towards jsoniter, but for that we'd need to change the derivation model (from recursive to "single-shot") and probably add some AST support, so that it has all the necessary features.

Ideally we'd just go with a Scala-JSON that is fast, secure & integrates well, but unfortunately it doesn't exist ;). If you'd have some thoughts, or dev power, let me know :)

[lomigmegard]

Thanks for the update!

So now I'm leaning towards jsoniter, but for that we'd need to change the derivation model (from recursive to "single-shot")

I believe that the full auto-derivation is not something people rely on anymore, I wouldn't see it missing as a problem.

probably add some AST support

Why does you want an AST for the json part? From outside, the API of Pickler is Pickler.derived[T], with the configuration in scope. The point is to not have to define the Schema and Json codec manually, to keep them in sync. The only place where the underlying library would (should) leak is when defining scalar values. Even there, a dedicated API could be used to bind the Schema with the scalar encoding.

Scala-JSON that is fast, secure & integrates well, but unfortunately it doesn't exis

I agree with your comments on the "Scala JSON problem", but at the same time I feel we should be able to hide the library so much that it doesn't matter. To escape these types of problems, I am always tempted to suggest to use a Java library directly, for example jackson. I didn't look at how you implemented Pickler though, the effort to glue the derivation macro might be greater than with a Scala lib.

[adamw]

I believe that the full auto-derivation is not something people rely on anymore, I wouldn't see it missing as a problem.

Well that differs from case to case. But we could start with a single derivation mode (semi-auto) so that ... derives works as well as top-level givens. Now here various json libraries work differently. If you have a nested AST structure, in some cases you need to derive the codec for all AST members. In some (e.g. jsoniter), you only need to add the derives clause at the top-level AST node. Codecs for children will be then derived as part of this single derivation.

Why does you want an AST for the json part?

Not that often but often enough, people end up needing fields of type Json, which accepts arbitrary (but valid) JSON values. So whatever JSON library we use, it would be good to have it support this kind of functionality.

for example jackson.

Jackson I think is reflection-based? Something that is very not in the spirit of Scala. Hiding the JSON library would be ideal, yes, although it would also be good to have an option to provide the library-specific configuration, that's not covered by "our" Tapir config.

But either way: we need to choose a library to hide ;)