reduced abbreviations

Author: hmcm

paper/paper.md

@@ -80,10 +80,9 @@ We briefly describe the core functionalities of PyTupli which are illustrated in
 # Quality Metrics
 
 ### Return-Based Metrics
-Return-based metrics such as trajectory quality (TQ) [@schweighofer2022dataset] or the average Q-value [@asadulaev2025expert] can inform algorithm decision. For example, @schweighofer2022dataset show that behavioral cloning (BC) performs well despite its simplicity if the dataset has a high TQ. For datasets with low TQ, algorithms from the DQN family perform well in their experiments as they do not constrain the learned policy towards the distribution of the behavioral policy. TQ normalizes the average return of a dataset with respect to the returns obtained by a minimal performant and an expert policy. To provide similar insights without relying on such additional information, estimated relative return improvement (ERI) [@swazinna2021measuring] relates the maximum trajectory return in the dataset to its average return. While ERI and TQ operate on a trajectory level, average Q-value estimation offers insights on the tuple level, making it a better predictor of offline RL performance  [@asadulaev2025expert]. It requires fitting a Q-function using Bellman updates, which is closely related to the objectives used in offline RL training. However, for continuous action spaces, the user needs to provide an evaluation policy to predict the next actions in the Bellman target. Such a policy can, for example, be obtained using BC on the dataset.
-
+Return-based metrics such as trajectory quality (TQ) [@schweighofer2022dataset] or the average Q-value [@asadulaev2025expert] can inform algorithm decision. For example, @schweighofer2022dataset show that behavioral cloning performs well despite its simplicity if the dataset has a high TQ. For datasets with low TQ, algorithms from the deep Q-network family perform well in their experiments as they do not constrain the learned policy towards the distribution of the behavioral policy. TQ normalizes the average return of a dataset with respect to the returns obtained by a minimal performant and an expert policy. To provide similar insights without relying on such additional information, estimated relative return improvement [@swazinna2021measuring] relates the maximum trajectory return in the dataset to its average return. While estimated return improvement and TQ operate on a trajectory level, average Q-value estimation offers insights on the tuple level, making it a better predictor of offline RL performance  [@asadulaev2025expert]. It requires fitting a Q-function using Bellman updates, which is closely related to the objectives used in offline RL training. However, for continuous action spaces, the user needs to provide an evaluation policy to predict the next actions in the Bellman target. Such a policy can, for example, be obtained using behavioral cloning on the dataset.
 ### Coverage-Based Metrics
-An important question when assessing a dataset is whether the behavioral policy (or policies) used to generate it did explore the state and action space well enough to learn a meaningful target policy from the data. A common approach for quantifying explorativeness is to approximate the entropy of transition probabilities for the behavior policy. For discrete state and action spaces, @schweighofer2022dataset suggest to approximate this by counting unique state-action pairs. Optionally, this value can be normalized using a reference dataset $\mathcal{D}_\text{ref}$ of same size, for example, the replay buffer collected during online training. @schweighofer2022dataset show that low SACo values hinder performance of a large variety of algorithms. While SACo aims at estimating the Shannon entropy of the transition probabilities, @suttle2025behavioral suggest that datasets that maximize their proposed behavioral entropy (BE) metric support better offline RL performance. They suggest a $k$-nearest-neighbor estimator of the true BE that relies on density-based weighting of different regions in the state-action space.
+An important question when assessing a dataset is whether the behavioral policy (or policies) used to generate it did explore the state and action space well enough to learn a meaningful target policy from the data. A common approach for quantifying explorativeness is to approximate the entropy of transition probabilities for the behavior policy. For discrete state and action spaces, @schweighofer2022dataset suggest to approximate this by counting unique state-action pairs. Optionally, this value can be normalized using a reference dataset $\mathcal{D}_\text{ref}$ of same size, for example, the replay buffer collected during online training. @schweighofer2022dataset show that low state-action coverage values hinder performance of a large variety of algorithms. While counting unique state-action pairs aims at estimating the Shannon entropy of the transition probabilities, @suttle2025behavioral suggest that datasets that maximize their proposed behavioral entropy metric support better offline RL performance. They suggest a $k$-nearest-neighbor estimator of the true behavioral entropy that relies on density-based weighting of different regions in the state-action space.
 
 # Acknowledgements