Early Exit NDT Rollout

[cristinaleonr]

Objective:

Monitor the phased rollout of early-exit NDT.

Steps:

• Release ndt-server changes.
• Release Locate probability changes set to 1%.
• Release client changes.
• Confirm metrics/analyses look as expected and increase Locate probability to 10%.
• Confirm metrics/analyses look as expected and increase Locate probability to 30%.
• Confirm metrics/analyses look as expected and increase Locate probability to 50%.
• Confirm metrics/analyses look as expected and increase Locate probability to 90%.

Rollout dashboard:

https://grafana.mlab-oti.measurementlab.net/d/W8JPPzzIz/ndt-early-exit?orgId=1

Criteria before progressing:

• Download performance and bytes sent graphs match expectations.
• No increase in client- or server-side errors.
• Global test rates are unaffected.
• No alerts are firing due to the rollout.

Rollback criteria:

• If there is any change to the global test rates or any alerts that fire due to the rollout, we should immediately roll back.
• If there is any increase in client- or server-side errors, we should immediately roll back.
• If any of the dashboard panels do not match expectations, we should consider rolling back.

Things we expect to change:

• Number of bytes sent per test.

[cristinaleonr]

Checks for 1% of traffic:

• Download performance closely matches the full test.
  

• Bytes sent top out at about 250MB.
  

• Stable number of client-side errors over the past week.

• Stable number of ndt7 download errors.
  

• Global NDT test rate also stable.
  

• Alerts firing unrelated to launch.

  • GardenerFailureRateTooHighOrMissing: related to the pipeline.
  • DataQuality_TooManyNdtS2cTestsWithDiscards: specific to MIA and DFW.
  • PlatformCluster_IPv6AtSiteDownForTooLong: specific to TUN01.

[cristinaleonr]

Checks for 10% of traffic:

• Download performance (last 7 days) closely matches the full test.
  

• Bytes sent (last 7 days) top out at about 250MB.
  

• Stable number of client-side errors over the past week.

• Stable number of ndt7 download errors (last 15 days).
  

• Global NDT test rate also stable (last 15 days).
  

• Alerts firing unrelated to launch.

  • SwitchUnpingableAtSite: specific to lim03.
  • DataQuality_TooManyNdtS2cTestsWithDiscards: specific to MIA and DFW.
  • PlatformCluster_IPv6AtSiteDownForTooLong: specific to TUN01.

[cristinaleonr]

Checks for 30% of traffic:

• Download performance (last 7 days) closely matches the full test. Slight variations around 200-300 Mbps.

• Bytes sent (last 7 days) top out at about 250MB.

• Stable number of client-side errors over the past week.

• Increasing number of ndt7+wss download measurer-closed-early errors in ndt7 (expected). Other errors seem stable (last 15 days).

• Global NDT test rate also stable (last 15 days).

[cristinaleonr]

Granular performance comparisons at the site/metro/AS level show comparable ndt7 download performance for early-exit ("short") and full tests.

• CDFs
  

[cristinaleonr]

Checks for 50% of traffic:

• Download performance (last 7 days) closely matches the full test.
  

• Bytes sent (last 7 days) top out at about 250MB.
  

• Stable number of client-side errors over the past week.

• Stable number of ndt7 download errors (last 15 days).
  

• Global NDT test rate stable (last 15 days).
  

• Alerts firing unrelated to launch.

  • DataQuality_TooManyNdtS2cTestsWithDiscards: specific to BOM and DWF.
  • GardenerFailureRateTooHighOrMissing: specific to the pipeline.

[cristinaleonr]

Granular performance comparisons (CDFs/geometric distance) at the site/metro/AS level show comparable ndt7 download performance for early-exit ("short") and full tests.

• Colab