This is an intentional feature to help us avoid prefix cache hits. Generally the content of a token does not matter for performance. The only time this would matter that I can think of is if speculative decoding is enabled. However in that case you can not use synthetic data at all since you need to use real-world data with speculative decoding to get good performance.

That being said, I would accept a change to line src/guidellm/dataset/synthetic.py:172
that uses the token space of the corpus rather than the entire tokenizer. Something like:

        unique_prefix_iter = cycle(set(self.processor.encode(self.text_creator.text)))

I recommend waiting till our refactor (#351) lands before attempting this due to some upcoming changes.

This is an intentional feature to help us avoid prefix cache hits. Generally the content of a token does not matter for performance. The only time this would matter that I can think of is if speculative decoding is enabled. However in that case you can not use synthetic data at all since you need to use real-world data with speculative decoding to get good performance.

That being said, I would accept a change to line src/guidellm/dataset/synthetic.py:172 that uses the token space of the corpus rather than the entire tokenizer. Something like:

        unique_prefix_iter = cycle(set(self.processor.encode(self.text_creator.text)))

I recommend waiting till our refactor (#351) lands before attempting this due to some upcoming changes.

Got.

@tukwila could you take a look through the new data pipelines refactor that is up and seeing if that fixes it / adapting this PR on top of that one? #384