test with and without normalize processor by flash1293 · Pull Request #815 · elastic/rally-tracks

flash1293 · 2025-07-11T10:47:34Z

esrally race \
    --pipeline=benchmark-only \
    --track-path=elastic/logs --kill-running-processes --telemetry=""  \
    --challenge=logging-regulartest \
    --target-hosts="localhost:9200" --report-file=./report_regular.md \
    --client-options="timeout:60,use_ssl:false,basic_auth_user:'elastic',basic_auth_password:'changeme'" \
    --track-params="max_total_download_gb:50"

esrally race \
    --pipeline=benchmark-only \
    --track-path=elastic/logs --kill-running-processes --telemetry=""  \
    --challenge=logging-normalizetest \
    --target-hosts="localhost:9200" --report-file=./report_normalize.md \
    --client-options="timeout:60,use_ssl:false,basic_auth_user:'elastic',basic_auth_password:'changeme'" \
    --track-params="max_total_download_gb:50"

esbench start --use-case=simple --params='{"track.repository.url":"https://github.com/flash1293/rally-tracks","track.name":"elastic/logs","track.challenge":"logging-regulartest","track.revision":"normalize-measurement","elasticsearch.version":"9.2.0-SNAPSHOT","track.params":{"raw_data_volume_per_day": "50GB"}}'

esbench start --use-case=simple --params='{"track.repository.url":"https://github.com/flash1293/rally-tracks","track.name":"elastic/logs","track.challenge":"logging-normalizetest","track.revision":"normalize-measurement","elasticsearch.version":"9.2.0-SNAPSHOT","track.params":{"raw_data_volume_per_day": "50GB"}}'

…ment

flash1293 · 2025-07-15T09:10:10Z

Doing another run with the proper version...

regulartest: f3e881b5-912f-4a92-9937-c2250f7b8309
normalizetest: be9546f4-4000-4747-ab8e-55e88d7a7b5d

flash1293 · 2025-07-15T14:32:43Z

Results for running on 100GB of data (I think? 50GB for 2 days) via esbench (using default settings of the simple use case)

no normalization, streams-style mappings

|                                                         Metric |       Task |           Value |   Unit |
|---------------------------------------------------------------:|-----------:|----------------:|-------:|
|                     Cumulative indexing time of primary shards |            |   284.942       |    min |
|             Min cumulative indexing time across primary shards |            |   284.942       |    min |
|          Median cumulative indexing time across primary shards |            |   284.942       |    min |
|             Max cumulative indexing time across primary shards |            |   284.942       |    min |
|            Cumulative indexing throttle time of primary shards |            |     0           |    min |
|    Min cumulative indexing throttle time across primary shards |            |     0           |    min |
| Median cumulative indexing throttle time across primary shards |            |     0           |    min |
|    Max cumulative indexing throttle time across primary shards |            |     0           |    min |
|                        Cumulative merge time of primary shards |            |    42.2361      |    min |
|                       Cumulative merge count of primary shards |            |    40           |        |
|                Min cumulative merge time across primary shards |            |    42.2361      |    min |
|             Median cumulative merge time across primary shards |            |    42.2361      |    min |
|                Max cumulative merge time across primary shards |            |    42.2361      |    min |
|               Cumulative merge throttle time of primary shards |            |    13.3337      |    min |
|       Min cumulative merge throttle time across primary shards |            |    13.3337      |    min |
|    Median cumulative merge throttle time across primary shards |            |    13.3337      |    min |
|       Max cumulative merge throttle time across primary shards |            |    13.3337      |    min |
|                      Cumulative refresh time of primary shards |            |     0.686333    |    min |
|                     Cumulative refresh count of primary shards |            |   102           |        |
|              Min cumulative refresh time across primary shards |            |     0.686333    |    min |
|           Median cumulative refresh time across primary shards |            |     0.686333    |    min |
|              Max cumulative refresh time across primary shards |            |     0.686333    |    min |
|                        Cumulative flush time of primary shards |            |     7.98395     |    min |
|                       Cumulative flush count of primary shards |            |    76           |        |
|                Min cumulative flush time across primary shards |            |     7.98395     |    min |
|             Median cumulative flush time across primary shards |            |     7.98395     |    min |
|                Max cumulative flush time across primary shards |            |     7.98395     |    min |
|                                        Total Young Gen GC time |            |     8.286       |      s |
|                                       Total Young Gen GC count |            |   594           |        |
|                                          Total Old Gen GC time |            |     0           |      s |
|                                         Total Old Gen GC count |            |     0           |        |
|                                                   Dataset size |            |    12.0518      |     GB |
|                                                     Store size |            |    12.0518      |     GB |
|                                                  Translog size |            |     0.128503    |     GB |
|                                         Heap used for segments |            |     0           |     MB |
|                                       Heap used for doc values |            |     0           |     MB |
|                                            Heap used for terms |            |     0           |     MB |
|                                            Heap used for norms |            |     0           |     MB |
|                                           Heap used for points |            |     0           |     MB |
|                                    Heap used for stored fields |            |     0           |     MB |
|                                                  Segment count |            |    85           |        |
|                                    Total Ingest Pipeline count |            |     2.35248e+08 |        |
|                                     Total Ingest Pipeline time |            |  5877.95        |      s |
|                                   Total Ingest Pipeline failed |            |     0           |        |
|                                                 Min Throughput | bulk-index |  4902.17        | docs/s |
|                                                Mean Throughput | bulk-index | 60007           | docs/s |
|                                              Median Throughput | bulk-index | 60299.3         | docs/s |
|                                                 Max Throughput | bulk-index | 60810.4         | docs/s |
|                                        50th percentile latency | bulk-index |    98.6395      |     ms |
|                                        90th percentile latency | bulk-index |   130.047       |     ms |
|                                        99th percentile latency | bulk-index |   283.148       |     ms |
|                                      99.9th percentile latency | bulk-index |  4797.99        |     ms |
|                                     99.99th percentile latency | bulk-index |  6363.65        |     ms |
|                                       100th percentile latency | bulk-index | 10779           |     ms |
|                                   50th percentile service time | bulk-index |    98.6116      |     ms |
|                                   90th percentile service time | bulk-index |   130.134       |     ms |
|                                   99th percentile service time | bulk-index |   296.569       |     ms |
|                                 99.9th percentile service time | bulk-index |  4855.69        |     ms |
|                                99.99th percentile service time | bulk-index |  6361.11        |     ms |
|                                  100th percentile service time | bulk-index | 10779           |     ms |
|                                                     error rate | bulk-index |     0           |      % |

normalization, streams-style mappings

|                                                         Metric |       Task |          Value |   Unit |
|---------------------------------------------------------------:|-----------:|---------------:|-------:|
|                     Cumulative indexing time of primary shards |            |   287.221      |    min |
|             Min cumulative indexing time across primary shards |            |   287.221      |    min |
|          Median cumulative indexing time across primary shards |            |   287.221      |    min |
|             Max cumulative indexing time across primary shards |            |   287.221      |    min |
|            Cumulative indexing throttle time of primary shards |            |     0          |    min |
|    Min cumulative indexing throttle time across primary shards |            |     0          |    min |
| Median cumulative indexing throttle time across primary shards |            |     0          |    min |
|    Max cumulative indexing throttle time across primary shards |            |     0          |    min |
|                        Cumulative merge time of primary shards |            |    69.8577     |    min |
|                       Cumulative merge count of primary shards |            |    39          |        |
|                Min cumulative merge time across primary shards |            |    69.8577     |    min |
|             Median cumulative merge time across primary shards |            |    69.8577     |    min |
|                Max cumulative merge time across primary shards |            |    69.8577     |    min |
|               Cumulative merge throttle time of primary shards |            |    29.2477     |    min |
|       Min cumulative merge throttle time across primary shards |            |    29.2477     |    min |
|    Median cumulative merge throttle time across primary shards |            |    29.2477     |    min |
|       Max cumulative merge throttle time across primary shards |            |    29.2477     |    min |
|                      Cumulative refresh time of primary shards |            |     0.924617   |    min |
|                     Cumulative refresh count of primary shards |            |   102          |        |
|              Min cumulative refresh time across primary shards |            |     0.924617   |    min |
|           Median cumulative refresh time across primary shards |            |     0.924617   |    min |
|              Max cumulative refresh time across primary shards |            |     0.924617   |    min |
|                        Cumulative flush time of primary shards |            |     8.91827    |    min |
|                       Cumulative flush count of primary shards |            |    78          |        |
|                Min cumulative flush time across primary shards |            |     8.91827    |    min |
|             Median cumulative flush time across primary shards |            |     8.91827    |    min |
|                Max cumulative flush time across primary shards |            |     8.91827    |    min |
|                                        Total Young Gen GC time |            |    12.326      |      s |
|                                       Total Young Gen GC count |            |   756          |        |
|                                          Total Old Gen GC time |            |     0          |      s |
|                                         Total Old Gen GC count |            |     0          |        |
|                                                   Dataset size |            |    11.168      |     GB |
|                                                     Store size |            |    11.168      |     GB |
|                                                  Translog size |            |     0.0305071  |     GB |
|                                         Heap used for segments |            |     0          |     MB |
|                                       Heap used for doc values |            |     0          |     MB |
|                                            Heap used for terms |            |     0          |     MB |
|                                            Heap used for norms |            |     0          |     MB |
|                                           Heap used for points |            |     0          |     MB |
|                                    Heap used for stored fields |            |     0          |     MB |
|                                                  Segment count |            |   120          |        |
|                                    Total Ingest Pipeline count |            |     2.3528e+08 |        |
|                                     Total Ingest Pipeline time |            |  7541.85       |      s |
|                                   Total Ingest Pipeline failed |            |     0          |        |
|                                                 Min Throughput | bulk-index |  1862.57       | docs/s |
|                                                Mean Throughput | bulk-index | 59019.7        | docs/s |
|                                              Median Throughput | bulk-index | 59334.3        | docs/s |
|                                                 Max Throughput | bulk-index | 60058.5        | docs/s |
|                                        50th percentile latency | bulk-index |   518.953      |     ms |
|                                        90th percentile latency | bulk-index |   683.849      |     ms |
|                                        99th percentile latency | bulk-index |  2799.55       |     ms |
|                                      99.9th percentile latency | bulk-index |  7118.07       |     ms |
|                                     99.99th percentile latency | bulk-index |  9195.62       |     ms |
|                                       100th percentile latency | bulk-index | 10468.4        |     ms |
|                                   50th percentile service time | bulk-index |   518.638      |     ms |
|                                   90th percentile service time | bulk-index |   684.774      |     ms |
|                                   99th percentile service time | bulk-index |  2690.62       |     ms |
|                                 99.9th percentile service time | bulk-index |  7113.39       |     ms |
|                                99.99th percentile service time | bulk-index |  9195.62       |     ms |
|                                  100th percentile service time | bulk-index | 10468.4        |     ms |
|                                                     error rate | bulk-index |     0          |      % |

Overall indexing time: +0.8% ((287.221/284.942)-1)
Overall indexing overhead per doc: 0.06 micro seconds ((((287.221-284.942)*60)/2.3528e+08)*1000*1000)

Total ingest pipeline time: +28.3% ((7541.85/5877.95)-1)
Ingest pipeline time overhead per doc: 0.07 micro seconds (((7541.85-5877.95)/2.3528e+08)*1000*1000)

flash1293 · 2025-07-20T10:45:09Z

Comparing to the same dataset with "regular" mappings and ingest pipelines:


|                                                         Metric |                                   Task |          Value |   Unit |
|---------------------------------------------------------------:|---------------------------------------:|---------------:|-------:|
|                     Cumulative indexing time of primary shards |                                        |   342.708      |    min |
|             Min cumulative indexing time across primary shards |                                        |     2.38515    |    min |
|          Median cumulative indexing time across primary shards |                                        |     9.63958    |    min |
|             Max cumulative indexing time across primary shards |                                        |   160.169      |    min |
|            Cumulative indexing throttle time of primary shards |                                        |     0          |    min |
|    Min cumulative indexing throttle time across primary shards |                                        |     0          |    min |
| Median cumulative indexing throttle time across primary shards |                                        |     0          |    min |
|    Max cumulative indexing throttle time across primary shards |                                        |     0          |    min |
|                        Cumulative merge time of primary shards |                                        |    80.8592     |    min |
|                       Cumulative merge count of primary shards |                                        |   126          |        |
|                Min cumulative merge time across primary shards |                                        |     0.27255    |    min |
|             Median cumulative merge time across primary shards |                                        |     1.8425     |    min |
|                Max cumulative merge time across primary shards |                                        |    44.1717     |    min |
|               Cumulative merge throttle time of primary shards |                                        |    32.7268     |    min |
|       Min cumulative merge throttle time across primary shards |                                        |     0.102467   |    min |
|    Median cumulative merge throttle time across primary shards |                                        |     0.619583   |    min |
|       Max cumulative merge throttle time across primary shards |                                        |    16.7208     |    min |
|                      Cumulative refresh time of primary shards |                                        |     1.87072    |    min |
|                     Cumulative refresh count of primary shards |                                        |   677          |        |
|              Min cumulative refresh time across primary shards |                                        |     0.022      |    min |
|           Median cumulative refresh time across primary shards |                                        |     0.0608333  |    min |
|              Max cumulative refresh time across primary shards |                                        |     0.8702     |    min |
|                        Cumulative flush time of primary shards |                                        |    22.4155     |    min |
|                       Cumulative flush count of primary shards |                                        |   629          |        |
|                Min cumulative flush time across primary shards |                                        |     0.252567   |    min |
|             Median cumulative flush time across primary shards |                                        |     0.809033   |    min |
|                Max cumulative flush time across primary shards |                                        |     8.15422    |    min |
|                                        Total Young Gen GC time |                                        |    22.595      |      s |
|                                       Total Young Gen GC count |                                        |   779          |        |
|                                          Total Old Gen GC time |                                        |     0          |      s |
|                                         Total Old Gen GC count |                                        |     0          |        |
|                                                   Dataset size |                                        |    14.0964     |     GB |
|                                                     Store size |                                        |    14.0964     |     GB |
|                                                  Translog size |                                        |     2.96334    |     GB |
|                                         Heap used for segments |                                        |     0          |     MB |
|                                       Heap used for doc values |                                        |     0          |     MB |
|                                            Heap used for terms |                                        |     0          |     MB |
|                                            Heap used for norms |                                        |     0          |     MB |
|                                           Heap used for points |                                        |     0          |     MB |
|                                    Heap used for stored fields |                                        |     0          |     MB |
|                                                  Segment count |                                        |   319          |        |
|                                    Total Ingest Pipeline count |                                        |     2.3528e+08 |        |
|                                     Total Ingest Pipeline time |                                        | 11114.7        |      s |
|                                   Total Ingest Pipeline failed |                                        |     0          |        |
|                                                 Min Throughput |                       insert-pipelines |    12.71       |  ops/s |
|                                                Mean Throughput |                       insert-pipelines |    12.71       |  ops/s |
|                                              Median Throughput |                       insert-pipelines |    12.71       |  ops/s |
|                                                 Max Throughput |                       insert-pipelines |    12.71       |  ops/s |
|                                       100th percentile latency |                       insert-pipelines |  1140.73       |     ms |
|                                  100th percentile service time |                       insert-pipelines |  1140.73       |     ms |
|                                                     error rate |                       insert-pipelines |     0          |      % |
|                                                 Min Throughput |                             insert-ilm |    24.83       |  ops/s |
|                                                Mean Throughput |                             insert-ilm |    24.83       |  ops/s |
|                                              Median Throughput |                             insert-ilm |    24.83       |  ops/s |
|                                                 Max Throughput |                             insert-ilm |    24.83       |  ops/s |
|                                       100th percentile latency |                             insert-ilm |    39.6513     |     ms |
|                                  100th percentile service time |                             insert-ilm |    39.6513     |     ms |
|                                                     error rate |                             insert-ilm |     0          |      % |
|                                                 Min Throughput | validate-package-template-installation |    43.88       |  ops/s |
|                                                Mean Throughput | validate-package-template-installation |    43.88       |  ops/s |
|                                              Median Throughput | validate-package-template-installation |    43.88       |  ops/s |
|                                                 Max Throughput | validate-package-template-installation |    43.88       |  ops/s |
|                                       100th percentile latency | validate-package-template-installation |    22.4875     |     ms |
|                                  100th percentile service time | validate-package-template-installation |    22.4875     |     ms |
|                                                     error rate | validate-package-template-installation |     0          |      % |
|                                                 Min Throughput |        update-custom-package-templates |    29.16       |  ops/s |
|                                                Mean Throughput |        update-custom-package-templates |    29.16       |  ops/s |
|                                              Median Throughput |        update-custom-package-templates |    29.16       |  ops/s |
|                                                 Max Throughput |        update-custom-package-templates |    29.16       |  ops/s |
|                                       100th percentile latency |        update-custom-package-templates |   411.179      |     ms |
|                                  100th percentile service time |        update-custom-package-templates |   411.179      |     ms |
|                                                     error rate |        update-custom-package-templates |     0          |      % |
|                                                 Min Throughput |                             bulk-index |  2350.91       | docs/s |
|                                                Mean Throughput |                             bulk-index | 48050.1        | docs/s |
|                                              Median Throughput |                             bulk-index | 48256.4        | docs/s |
|                                                 Max Throughput |                             bulk-index | 49086          | docs/s |
|                                        50th percentile latency |                             bulk-index |   537.863      |     ms |
|                                        90th percentile latency |                             bulk-index |  1442.33       |     ms |
|                                        99th percentile latency |                             bulk-index |  2633.55       |     ms |
|                                      99.9th percentile latency |                             bulk-index |  5476.09       |     ms |
|                                     99.99th percentile latency |                             bulk-index |  6984.8        |     ms |
|                                       100th percentile latency |                             bulk-index | 10404.7        |     ms |
|                                   50th percentile service time |                             bulk-index |   542.219      |     ms |
|                                   90th percentile service time |                             bulk-index |  1433.23       |     ms |
|                                   99th percentile service time |                             bulk-index |  2602.4        |     ms |
|                                 99.9th percentile service time |                             bulk-index |  5451.35       |     ms |
|                                99.99th percentile service time |                             bulk-index |  6976.4        |     ms |
|                                  100th percentile service time |                             bulk-index | 10404.7        |     ms |
|                                                     error rate |                             bulk-index |     0          |      % |

Indexing time and pipeline time is a bit higher which is expected since it's doing more work (processing data and building more indices). This isn't really an apples-to-apples comparison

flash1293 added 4 commits July 11, 2025 12:46

test with and without normalize processor

d6fddac

fix

baa91f5

Merge remote-tracking branch 'upstream/master' into normalize-measure…

9236378

…ment

adjust mappings

70d444e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test with and without normalize processor#815

test with and without normalize processor#815
flash1293 wants to merge 4 commits intoelastic:masterfrom
flash1293:normalize-measurement

flash1293 commented Jul 11, 2025 •

edited

Loading

Uh oh!

flash1293 commented Jul 15, 2025 •

edited

Loading

Uh oh!

flash1293 commented Jul 15, 2025 •

edited

Loading

Uh oh!

flash1293 commented Jul 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

flash1293 commented Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

flash1293 commented Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

flash1293 commented Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

no normalization, streams-style mappings

normalization, streams-style mappings

Uh oh!

flash1293 commented Jul 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

flash1293 commented Jul 11, 2025 •

edited

Loading

flash1293 commented Jul 15, 2025 •

edited

Loading

flash1293 commented Jul 15, 2025 •

edited

Loading