[fix][broker] Fix corrupted topic policies issues with sequential topic policy updates

[lhotari]

Fixes #24393
Fixes #21303

Motivation

There's a long time issue where topic policies get corrupted as a result of sequential or concurrent topic policy updates.
This PR contains the implementation of
"PIP-428: Change TopicPoliciesService interface to fix consistency issues". Please refer to the PIP document at https://github.com/apache/pulsar/blob/master/pip/pip-428.md for the broader context.

Modifications

• Check the PIP-428 document for an overview.

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 85.45455% with 64 lines in your changes missing coverage. Please review.

Project coverage is 74.27%. Comparing base (bbc6224) to head (9edeaf1).
Report is 1158 commits behind head on master.

Files with missing lines	Patch %	Lines
...pulsar/broker/admin/impl/PersistentTopicsBase.java	90.17%	0 Missing and 23 partials ⚠️
.../service/SystemTopicBasedTopicPoliciesService.java	84.13%	15 Missing and 8 partials ⚠️
...che/pulsar/common/policies/data/TopicPolicies.java	80.95%	3 Missing and 5 partials ⚠️
...sar/common/policies/data/SubscriptionPolicies.java	50.00%	1 Missing and 1 partial ⚠️
...rg/apache/pulsar/broker/service/BrokerService.java	0.00%	0 Missing and 1 partial ⚠️
...ar/common/policies/data/InactiveTopicPolicies.java	50.00%	1 Missing ⚠️
...lsar/common/policies/data/PersistencePolicies.java	50.00%	1 Missing ⚠️
...pache/pulsar/common/policies/data/PublishRate.java	50.00%	1 Missing ⚠️
...pulsar/common/policies/data/RetentionPolicies.java	50.00%	1 Missing ⚠️
...che/pulsar/common/policies/data/SubscribeRate.java	50.00%	1 Missing ⚠️
... and 2 more

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #24427      +/-   ##
============================================
+ Coverage     73.57%   74.27%   +0.69%     
- Complexity    32624    32750     +126     
============================================
  Files          1877     1869       -8     
  Lines        139502   145392    +5890     
  Branches      15299    16683    +1384     
============================================
+ Hits         102638   107987    +5349     
+ Misses        28908    28836      -72     
- Partials       7956     8569     +613     

Flag	Coverage Δ
inttests	26.93% <23.18%> (+2.34%)	⬆️
systests	22.66% <8.18%> (-1.66%)	⬇️
unittests	73.76% <85.00%> (+0.91%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...he/pulsar/broker/service/TopicPoliciesService.java	100.00% <ø> (+31.25%)	⬆️
...rg/apache/pulsar/broker/service/BrokerService.java	83.73% <0.00%> (+2.95%)	⬆️
...ar/common/policies/data/InactiveTopicPolicies.java	50.00% <50.00%> (-33.34%)	⬇️
...lsar/common/policies/data/PersistencePolicies.java	83.33% <50.00%> (-0.88%)	⬇️
...pache/pulsar/common/policies/data/PublishRate.java	71.42% <50.00%> (-3.58%)	⬇️
...pulsar/common/policies/data/RetentionPolicies.java	81.81% <50.00%> (-3.19%)	⬇️
...che/pulsar/common/policies/data/SubscribeRate.java	73.68% <50.00%> (-4.10%)	⬇️
...ar/common/policies/data/impl/BacklogQuotaImpl.java	91.66% <50.00%> (-0.23%)	⬇️
...ar/common/policies/data/impl/DispatchRateImpl.java	95.45% <50.00%> (-0.85%)	⬇️
...sar/common/policies/data/SubscriptionPolicies.java	40.00% <50.00%> (+6.66%)	⬆️
... and 3 more

... and 1067 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.