DataStreamAutoShardingService.calculate() works on stale data #134505
Description
DataStreamAutoShardingService.calculate() receives a DataStream object that has been pulled from ProjectMetadata's indicesLookup. It reads the value of getIndexMode() from that data stream. But it turns out that we never update the index mode on a data stream, even if its template has been modified to have a different index mode. The get data stream API works fine because we read it from the template if it is null. The rollover API does the same (after this DataStreamAutoShardingService code is called).
This gets us into trouble because if the index mode is lookup we can only ever have 1 shard. DataStreamAutoShardingService.calculate() has code to prevent a shard increase recommendation if the index mode is lookup, but since the calculate method does not receive this information, it can recommend a shard increase. So we wind up in a situation where we recommend a shard increase, and then cause rollover to fail.