Add PolarisMetaStoreManager.loadEntities

[XN137]

this is a followup to #2261

currently PolarisMetaStoreManager.listEntities only exposes a limited
subset of the underlying BasePersistence.listEntities functionality.

most of the callers have to post-process the EntityNameLookupRecord of
ListEntitiesResult and call PolarisMetaStoreManager.loadEntity
on the individual items sequentually to transform and filter them.

this is bad for the following reasons:

• suboptimal performance as we run N+1 queries to basically load every
  entity twice from the persistence backend
• suffering from race-conditions when entities get dropped between the
  listEntities and loadEntity call
• a lot of repeated code in all the callers (of which only some are
  dealing with the race-condition by filtering out null values)

as a solution we add PolarisMetaStoreManager.loadEntities that takes
advantage of the already existing BasePersistence methods.
we rename one of the listEntities methods to loadEntities for
consistency.

since many callers dont need paging and want the result as a list, we
add PolarisMetaStoreManager.loadEntitiesAll as a convenient wrapper.

we also remove the PolarisEntity.nameAndId method as callers who only
need name and id should not be loading the full entity to begin with.

note we rework testCatalogNotReturnedWhenDeletedAfterListBeforeGet
from ManagementServiceTest because the simulated race-condition
scenario can no longer happen.

[XN137]

side note:
if the required policyType is null we could in theory use the optimized listEntities call, as we only need the name of the entity to build the PolicyIdentifier, but for filtering by policyType we need to load the full entity.

[dennishuo]

This will be a big improvement, thanks for taking this on!

I can do a more detailed dive, but at a high level, I'd like to see if we can better sort out the responsibilities of the MetaStoreManager layer vs the BasePersistence layer here. Specifically, if we can pull up the entityFilter/transformer from BasePersistence to only live in the MetaStoreManager level, just as your #2317 is a push down of the entitySubType filter. Essentially:

1. We want the [Base]Persistence layer to most closely represent what a lower-level database is directly capable of doing
2. We put more "advanced" logic in the MetaStoreManager layer
3. In the filter/transformer scenario, what we're really saying is "BasePersistence needs to give back complete PolarisBaseEntities instead of only EntityNameLookupRecords", and then something might run Polaris-side filtering/transforms -- logically this would be in MetaStoreManager or higher.

This also tells us why it made sense for the loadTasks to use a Predicate<> as the filter, but why subTypeCode should not be an opaque Predicate<> -- subTypeCode filtering is something the lower-level database is capable of understanding so therefore it is pushed down to the layer that represents the raw database.

Whereas advanced timestamp comparison, leasing, etc., of TaskEntities is a Polaris concept that the database isn't designed to understand directly (for now) so we fallback to the opaque Predicate.

Overall, I'd like to see if we can:

1. Also rename the filter/transformer variations of listEntities within the BasePersistence, similar to how you introduced a totally different method name in the MetaStoreManager layer -- because now it's really not an overload of the same method, but really a different action entirely. I think loadEntities is reasonable to push down as the method name, but we just might want to consider the difference between "list and load full entities under a parent" and "load the full entities provided in this Collection" that we might need in the future
2. Pull filter/transformer evaluation out of BasePersistence into MetaStoreManager
3. Reconsider whether we actually need a Function<PolarisBaseEntity, T> generic type return value -- as far as I recall, originally we just abused that to "transform" a PolarisBaseEntity into a trimmed EntityNameLookupRecord, which was an antipattern as you identified, and then nearly every other case was just the "identity transformation". Now that we clarified the "EntityNameLookup" case, perhaps callsites never actually use the transformer anymore? I didn't have time to dig deeper yet into all the transformer use cases.

[XN137]

thanks for the feedback, since the other PR is somewhat merge-ready i will wait for that and rebase this one afterwards.

Also rename the filter/transformer variations of listEntities within the BasePersistence

yeah we can use the same loadEntities name in BasePersistence imo

Pull filter/transformer evaluation out of BasePersistence into MetaStoreManager

i havent investigated this in detail yet but my current guess is that the filter needs to remain "pushed down" in order for pagination to work correctly, but will double check that later.

Reconsider whether we actually need a Function<PolarisBaseEntity, T> generic type return value

afaict the transformer is used heavily by all callers of load(All)Entities in this PR i.e. to turn base entities into their more specific type (for example via CatalogRoleEntity::of).
so i dont see that going away.

unless you mean that this transformation can also happen on the "outside" ?
this might be possible however one idea I had was that one might not need both the filter and the transformer but we could let the transformer return null for entities the caller does not need (or that cant be converted to the more specific type).
but even then the transformer needs to remain "Pushed down" to work well with pagination most likely.

[XN137]

rebased on latest main due to a few merge conflict. also included the rename of one of the BasePersistence.listEntities methods

[adutra]

nit: a nullable List is not very useful, can we pass List.of() instead of null?

[XN137]

its possible yes but we are just "mirroring" the listEntities api which has this param as nullable. for method params isnt it "ok" to have nullable lists?

[adutra]

It's unnecessary imho since there is no distinction between what a null list and an empty list represent, so basically accepting nulls just makes your code more error prone.

But since other methods exhibit the same behavior, and given that countless discussions already happened around nullability without us reaching any conclusion, let's table this topic for now.

[XN137]

rebased after trivial merge-conflict in PolarisAdminService.java