[WIP] Use itertools.cycle to iterate over random queries from the query file

[fressi-elastic]

This is a follows up over PR #767

[gareth-ellis]

I need to have a go at running this, but my initial reaction is that I think this changes the behaviour of the track. In rally we have two modes of running a task - either time based, or "unit" based - where we would in the track say run this for 10 minutes, or run this 200 times - by not allowing the iterator to renew, it means that if a user says run this 12000 times, then actually it will only run 10000 times, since the query file has 10000 rows.

As i say, I have only very quickly looked at this - i need to run myself to validate.

[AI-IshanBhatt]

Parentheses in the dataclass decoratore

[gbanasiak]

This looks good overall. I've asked to document the default random seed.

I'd like to hear @elastic/search-relevance opinion. Would you like to pursue with this wikipedia change? The benefits I see are:

• random seed pinning for potentially better reproducibility of results
• the use of itertools.cycle makes code simpler as there's no need to handle iterator exhaustion
• unit tests might help us avoid unintended changes in the future but you might see it as unnecessary boilerplate.

[gbanasiak]

I like the idea of pinning the seed. We're doing something similar in elastic/logs through default track parameter value:

rally-tracks/elastic/logs/README.md

Line 242 in f3e4ad5

* `random_seed` (default: 13) - Files are generated through random sampling of the source corpora. This pseudo random selection process is seeded to ensure multiple runs of the track generate the same data - thus ensuring tests are repeatable. Changing this value or `data_generation_clients` will cause the generation of a different dataset. Must be an integer.

rally-tracks/elastic/logs/track.json

Line 155 in f3e4ad5

"random-seed": {{ random_seed | default(13) | int }},

Can we document the default in track's README?

[gbanasiak]

Back to @gareth-ellis point I think this is correct, i.e. itertools.cycle will cycle back to the beginning of an array returned by query_samples() which is what current code is doing. It's convenient at the cost of doubling memory consumption because itertools.cycle stores a local copy of an array I think (see https://docs.python.org/3/library/itertools.html#itertools.cycle).

The queries.csv has 10k lines, and we're asking for x10 more as the default batch_size/k is 100k. Assuming pure ASCII (1 byte / character), and string length of 32 characters on average, that's an increase in the region of 128k * 32 ~ 4MB. I think that's OK.

[kderusso]

Hey, from a Search Relevance team perspective, randomization like this sounds like a good idea, thanks for adding it along with the tests!

Note that at some point we should think about removing search applications from the challenges here, even if we keep the code in - this product is in maintenance mode.

[gbanasiak]

I'm adding WIP label in the title to mute alerts. @fressi-elastic Please remove it once you have some free cycles to go back to this.