Conversation
| if: ${{ !cancelled() }} | ||
| with: | ||
| name: junit-xml--${{github.run_id}}--${{github.run_attempt}}--unit-test | ||
| name: junit-xml--${{ github.run_id }}--${{ steps.get_job_id.outputs.job_id }}--${{ github.run_attempt }}--unit-test |
There was a problem hiding this comment.
Aligning artifact name with functional tests to have consistent naming scheme for parsing.
There was a problem hiding this comment.
Will this break backwards? so the next week won't have all of the reports but going forward it will?
There was a problem hiding this comment.
I think it's worth it since unit tests and integration tests have been flying under the radar anyway so far
| // Group failures by test name | ||
| // Group failures by test name, then remove parent entries whose subtests were observed. | ||
| grouped := groupFailuresByTest(allFailures) | ||
| filterParentTests(grouped, testRunCounts) |
There was a problem hiding this comment.
This will remove parent suites from the reports when they have subtests; removes misleading/noisy entries.
stephanos
force-pushed
the
fix-flakereport
branch
from
e5c17ec to
7f5d574
Compare
| // generateSuiteReports creates per-suite flake breakdown from all failures and test runs. | ||
| // Suite flake rate = % of workflow runs where the suite had at least one non-retry failure. | ||
| // Suite flake rate = % of job executions where the suite had at least one non-retry failure. | ||
| func generateSuiteReports(allFailures []TestFailure, allTestRuns []TestRun) []SuiteReport { |
There was a problem hiding this comment.
Grouping of suites was too aggressive, now shows all suite runs.
stephanos
force-pushed
the
fix-flakereport
branch
10 times, most recently
from
b499b6e to
9143911
Compare
| ) | ||
|
|
||
| // hoursAgo formats a timestamp as "Xh ago" relative to now. | ||
| func hoursAgo(t time.Time) string { |
There was a problem hiding this comment.
I think this is easier as it uses a consistent time unit.
stephanos
force-pushed
the
fix-flakereport
branch
3 times, most recently
from
63c61b7 to
95153bc
Compare
| sr.SuiteName, sr.FlakeRate, sr.FailedRuns, sr.TotalRuns, lastFailure)) | ||
| rate := fmt.Sprintf("%.1f%% (%d/%d)", sr.FlakeRate, sr.FailedRuns, sr.TotalRuns) | ||
| if sr.FlakeRate > 5 { | ||
| rate = "**" + rate + "**" |
There was a problem hiding this comment.
Little bit of emphasis.
stephanos
force-pushed
the
fix-flakereport
branch
2 times, most recently
from
c060df7 to
b809a21
Compare
stephanos
force-pushed
the
fix-flakereport
branch
from
b809a21 to
4b01491
Compare
2 participants
What changed?
Fix issues with flakereport; see comments for details.
How did you test it?
Example
Flaky Tests Report - 2026-02-25 11:25:50
Overall Statistics
Failure Categories Summary
CI Breakers (Failed All Retries)
TestVersionWorkflowSuite/v2/Test_DeleteVersion_QueryAfterDeletionCrashes
unit-testFlaky Tests
TestClientDataConverterTestSuitePANIC: runtime error: index out of range [0] with length 0 [recovered, repanicked] — in go.temporal.io/server/tests.TestWorkflowUpdateSuite.func21.2PANIC: Fail in goroutine after TestVersioning3FunctionalSuiteV2/TestWorkflowWithPinnedOverride_NoSticky has completed — in TestVersioning3FunctionalSuiteV0/TestActivityRetryAutoUpgradeDuringBackoffPANIC: Fail in goroutine after TestVersioning3FunctionalSuiteV0/TestPinnedQuery_RollbackDrainedVersion/ForceTaskForwardNoPollForwardForceAsync has completed — in TestVersioning3FunctionalSuiteV2/TestActivityRetryAutoUpgradeDuringBackoffDATA RACE: Data race detectedPANIC: runtime error: invalid memory address or nil pointer dereference — in go.temporal.io/server/tests.(*DeploymentVersionSuite).TestForceCAN_WithOverrideState.func1TestVersioning3FunctionalSuiteV0/TestTransitionDuringTransientTask_WithSignalTestVersioning3FunctionalSuiteV2/TestTransitionDuringTransientTask_WithoutSignalTestVersioning3FunctionalSuiteV2/TestTransitionDuringTransientTask_WithSignalTestVersioning3FunctionalSuiteV0/TestTransitionDuringTransientTask_WithoutSignalTestVersionWorkflowSuite/v2/Test_DeleteVersion_QueryAfterDeletionTestUserData_FetchesUpTreeTestNewServerWithOTEL/with_OTEL_Collector_runningTestInterleavedWeightedRoundRobinSchedulerSuite/TestInactiveChannelDeletionRaceTestUserData_FetchesActivityToWorkflowTestDeploymentVersionSuiteV0/TestReactivationSignalCache_Deduplication_StartWorkflowTestNewServerTestDeploymentVersionSuiteV0/TestReactivationSignalCache_Deduplication_SignalWithStartTestDeploymentVersionSuiteV0/TestStartWorkflowExecution_ReactivateVersionOnPinned_WithConflictPolicyTestDeploymentVersionSuiteV2/TestVersionScavenger_DeleteOnAddTestDeploymentVersionSuiteV0/TestStartWorkflowExecution_ReactivateVersionOnPinnedTestDeploymentVersionSuiteV0/TestSignalWithStartWorkflowExecution_ReactivateVersionOnPinnedTestFairnessAutoEnableSuite/Test_Activity_BasicTestDeploymentVersionSuiteV2/TestReactivationSignalCache_Deduplication_UpdateOptionsTestActivityClientTestSuite/TestActivityScheduleToClose_FiredDuringBackoffTestDeploymentVersionSuiteV0/TestReactivationSignalCache_Deduplication_ResetTestDeploymentVersionSuiteV0/TestResetWorkflowExecution_ReactivateVersionOnPinnedTestDeploymentVersionSuiteV2/TestVersionMissingTaskQueues_ValidSetRampingVersionTestTaskQueueStats_Pri_Suite/TestRampingAndCurrentAbsorbUnversionedBacklog/ForceTaskForwardForcePollForwardAllowSyncTestWorkerDeploymentSuiteV0/TestSetCurrentVersion_Concurrent_DifferentVersions_NoUnexpectedErrorsTestDeploymentVersionSuiteV0/TestUpdateWorkflowExecutionOptions_ReactivateVersionOnPinnedTestCronTestClientSuite/TestCronWorkflowCompletionStatesTestCallbacksSuiteCHASM/TestWorkflowNexusCallbacks_CarriedOver/ContinueAsNewTestStandaloneActivityTestSuite/TestHeartbeat/HeartbeatKeepsActivityAliveTestCallbacksSuiteCHASM/TestWorkflowNexusCallbacks_CarriedOver/WorkflowRunTimeoutTestWorkerDeploymentSuiteV0/TestDescribeWorkerDeployment_SetCurrentVersionTestDeploymentVersionSuiteV0/TestReactivationSignalCache_Deduplication_UpdateOptionsTestTaskQueueStats_Classic_Suite/TestAddMultipleTasks_ValidateStats_CachedTestTaskQueueStats_Pri_Suite/TestInactiveVersionDoesNotAbsorbUnversionedBacklog/NoTaskForwardNoPollForwardAllowSyncTestTaskQueueStats_Pri_Suite/TestMultipleTasks_WithMatchingBehavior_ValidateStats/ForceTaskForwardForcePollForwardAllowSyncTestWorkerDeploymentSuiteV0/TestSetRampingVersion_AfterDrainedTestWorkerDeploymentSuiteV2/TestDrainRollbackedVersionTestFuncClustersTestSuite/EnableTransitionHistory/TestForceMigration_ResetWorkflowTestDeploymentVersionSuiteV0/TestVersionScavenger_DeleteOnAddTestWorkerDeploymentSuiteV2/TestSetCurrentVersion_Concurrent_DifferentVersions_NoUnexpectedErrorsTestPrioritySuite/TestSubqueue_MigrationTestCallbacksSuiteCHASM/TestWorkflowNexusCallbacks_CarriedOver/WorkflowFailureRetryTestWorkerDeploymentSuiteV0/TestDrainRollbackedVersionTestVersioning3FunctionalSuiteV0/TestPinnedTask_NoProperPoller/NoTaskForwardNoPollForwardForceAsyncTestVersioning3FunctionalSuiteV2/TestPinnedCaN_UpgradeOnCaN_NormalWFT_WithSuggest/ForceTaskForwardNoPollForwardForceAsyncTestDeploymentVersionSuiteV0/TestUpdateWorkflowExecutionOptions_SetImpliedPinnedSuccessTestGetHistoryFunctionalSuite/DisableTransitionHistory/TestGetWorkflowExecutionHistory_AllTestVersioning3FunctionalSuiteV2/TestUnpinnedCaNTestVersioningFunctionalSuite/TestWorkflowTaskRedirectInRetryNonFirstTask/NoTaskForwardForcePollForwardAllowSyncTestWorkerDeploymentSuiteV0/TestListWorkerDeployments_TwoVersions_SameDeployment_OneCurrent_OneRampingTestTaskQueueStats_Classic_Suite/TestMultipleTasks_WithMatchingBehavior_ValidateStats/ForceTaskForwardNoPollForwardForceAsyncTestVersioning3FunctionalSuiteV2/TestWorkflowWithPinnedOverride_NoSticky/ForceTaskForwardForcePollForwardForceAsyncTestAdvancedVisibilitySuiteLegacy/TestListWorkflow_StringQueryTestVersioning3FunctionalSuiteV0/TestChildWorkflowInheritance_ParentPinnedByOverrideTestStandaloneActivityTestSuite/TestRequestCancel/MismatchedTokenComponentRefTestDeploymentVersionSuiteV2/TestVersionMissingTaskQueues_ValidSetCurrentVersionTestVersioning3FunctionalSuiteV0/TestPinnedQuery_DrainedVersion_PollersAbsent/ForceTaskForwardNoPollForwardAllowSyncTestVersioning3FunctionalSuiteV0/TestPinnedWorkflowWithLateActivityPoller/ForceTaskForwardNoPollForwardAllowSyncTestTaskQueueStats_Classic_Suite/TestMultipleTasks_WithMatchingBehavior_ValidateStats/ForceTaskForwardForcePollForwardAllowSyncTestDeploymentVersionSuiteV2/TestStartWorkflowExecution_ReactivateVersionOnPinned_WithConflictPolicyTestDeploymentVersionSuiteV2/TestSignalWithStartWorkflowExecution_ReactivateVersionOnPinnedTestVersioningFunctionalSuite/TestDispatchQueryOld/ForceTaskForwardNoPollForwardAllowSyncTestVersioningFunctionalSuite/TestWorkflowTaskRedirectInRetryFirstTask/ForceTaskForwardForcePollForwardForceAsyncTestTaskQueueStats_Pri_Suite/TestMultipleTasks_WithMatchingBehavior_ValidateStats/ForceTaskForwardForcePollForwardForceAsyncTestWorkerDeploymentSuiteV0/TestDescribeWorkerDeployment_MultipleVersions_SortedTestDeploymentVersionSuiteV2/TestDrainageStatus_SetCurrentVersion_YesOpenWFsTestVersioning3FunctionalSuiteV0/TestAutoUpgradeCaN_UpgradeOnCaN/ForceTaskForwardNoPollForwardAllowSyncTestWorkerDeploymentSuiteV2/TestSetCurrentVersion_BatchingTestTaskQueueStats_Classic_Suite/TestCurrentAbsorbsUnversionedBacklog_WhenRampingToUnversioned/NoTaskForwardForcePollForwardForceAsyncTestTaskQueueSuite/TestTaskQueueRateLimit_UpdateFromWorkerConfigAndAPITestVersioning3FunctionalSuiteV0/TestPinnedQuery_RollbackDrainedVersion/ForceTaskForwardNoPollForwardForceAsyncTestWorkflowUpdateSuite/StickySpeculativeWorkflowTask_AcceptComplete_StickyWorkerUnavailableTestWorkerDeploymentSuiteV2/TestResourceExhaustedErrors_Converted_To_ReadableMessageTestVersioning3FunctionalSuiteV0/TestPinnedCaN_UpgradeOnCaN_TransientWFT_WithSuggest/ForceTaskForwardForcePollForwardForceAsyncTestVersioning3FunctionalSuiteV2/TestPinnedCaN_UpgradeOnCaN_NormalWFT_PinnedOverride_WithSuggest/NoTaskForwardForcePollForwardAllowSyncTestTaskQueueStats_Pri_Suite/TestMultipleTasks_WithMatchingBehavior_ValidateStats/ForceTaskForwardNoPollForwardForceAsyncTestChildWorkflowSuite/TestCronChildWorkflowExecutionTestNexusStateReplicationTestSuite/DisableTransitionHistory/TestNexusCallbackReplicatedTestWorkerDeploymentSuiteV0/TestSetWorkerDeploymentRampingVersion_WithCurrent_Unset_RampTestFuncClustersTestSuite/EnableTransitionHistory/TestForceMigration_ClosedWorkflowTestTaskQueueStats_Classic_Suite/TestCurrentVersionAbsorbsUnversionedBacklog_NoRamping/ForceTaskForwardNoPollForwardForceAsyncTestWorkflowUpdateSuite/SpeculativeWorkflowTask_ScheduleToStartTimeoutOnNormalTaskQueueTestStandaloneActivityTestSuite/TestRequestCancel/StaleAttemptTokenTestTaskQueueStats_Classic_Suite/TestCurrentVersionAbsorbsUnversionedBacklog_NoRamping/NoTaskForwardForcePollForwardAllowSyncTestVersioning3FunctionalSuiteV2/TestUnpinnedQuery_NoSticky/ForceTaskForwardForcePollForwardForceAsyncTestTaskQueueStats_Classic_Suite/TestRampingAbsorbsUnversionedBacklog_WhenCurrentIsUnversioned/NoTaskForwardForcePollForwardForceAsyncTestTaskQueueStats_Pri_Suite/TestCurrentVersionAbsorbsUnversionedBacklog_NoRamping/NoTaskForwardNoPollForwardAllowSyncTestTaskQueueStats_Pri_Suite/TestRampingAndCurrentAbsorbUnversionedBacklog/NoTaskForwardNoPollForwardAllowSyncTestFuncClustersTestSuite/DisableTransitionHistory/TestForceMigration_ClosedWorkflowTestWorkerDeploymentSuiteV0/TestSetRampingVersion_Concurrent_DifferentVersions_NoUnexpectedErrorsTestTaskQueueStats_Classic_Suite/TestMultipleTasks_WithMatchingBehavior_ValidateStats/NoTaskForwardNoPollForwardAllowSyncTestVersioning3FunctionalSuiteV0/TestPinnedQuery_RollbackDrainedVersion/NoTaskForwardForcePollForwardAllowSyncTestVersioning3FunctionalSuiteV2/TestUnpinnedQuery_NoSticky/NoTaskForwardForcePollForwardAllowSyncTestDeploymentVersionSuiteV0/TestForceCAN_WithOverrideStateTestVersioningFunctionalSuite/TestDispatchActivityFailCrossTq/NoTaskForwardForcePollForwardAllowSyncTestVersioning3FunctionalSuiteV0/TestPinnedCaN_UpgradeOnCaN_TransientWFT_WithSuggest/ForceTaskForwardNoPollForwardForceAsyncTestWorkerDeploymentSuiteV0/TestSetWorkerDeploymentRampingVersion_BatchingTestDeploymentVersionSuiteV0/TestDrainageStatus_SetCurrentVersion_NoOpenWFsTestScheduleFunctionalSuite/TestNextTimeCacheTestVersioning3FunctionalSuiteV2/TestUnpinnedQuery_NoSticky/ForceTaskForwardNoPollForwardAllowSyncTestPollerScalingFunctionalSuite/TestPollerScalingDecisionsAreSeenProbabilisticallyTestFairnessSuite/TestMigration_FromFairTestFairnessAutoEnableSuite/TestMigration_FromClassicTestTaskQueueStats_Pri_Suite/TestMultipleTasks_WithMatchingBehavior_ValidateStats/ForceTaskForwardNoPollForwardAllowSyncTestWorkflowTaskTestSuite/TestWorkflowTaskHeartbeatingWithEmptyResultTestAdvancedVisibilitySuite/TestWorkerTaskReachability_Unversioned_InTaskQueueTestDeploymentVersionSuiteV2/TestDeleteVersion_DeleteRampedVersionTestWorkerDeploymentSuiteV0/TestDeploymentVersionLimitsTestVersioning3FunctionalSuiteV2/TestPinnedQuery_RollbackDrainedVersion/ForceTaskForwardForcePollForwardForceAsyncTestVersioning3FunctionalSuiteV2/TestUnpinnedQuery_NoSticky/NoTaskForwardNoPollForwardAllowSyncTestDeploymentVersionSuiteV2/TestDeleteVersion_Drained_But_Pollers_ExistTestVersioning3FunctionalSuiteV2/TestQueryWithPinnedOverride_Sticky/ForceTaskForwardForcePollForwardAllowSyncTestFairnessAutoEnableSuite/TestMigration_FromFairTestVersioning3FunctionalSuiteV0/TestPinnedCaN_UpgradeOnCaN_SpeculativeWFT_NoSuggest/NoTaskForwardNoPollForwardAllowSyncTestWorkerDeploymentSuiteV0/TestDescribeWorkerDeployment_TwoVersions_SortedTestDeploymentVersionSuiteV2/TestDrainageStatus_SetCurrentVersion_NoOpenWFsTestDeploymentVersionSuiteV2/TestDeleteVersion_ValidDeleteTestScheduleFunctionalSuite/TestListSchedulesReturnsWorkflowStatusTestVersioningFunctionalSuite/TestWorkflowTaskRedirectInRetryFirstTask/ForceTaskForwardForcePollForwardAllowSyncTestCronTestSuite/TestCronWorkflowTestVersioning3FunctionalSuiteV0/TestPinnedCaN_UpgradeOnCaN_NormalWFT_WithSuggest/ForceTaskForwardNoPollForwardForceAsyncTestVersioning3FunctionalSuiteV2/TestAutoUpgradeCaN_UpgradeOnCaN/ForceTaskForwardForcePollForwardAllowSyncTestDeploymentVersionSuiteV2/TestReactivationSignalCache_Deduplication_SignalWithStartTestVersioning3FunctionalSuiteV2/TestPinnedQuery_DrainedVersion_PollersAbsent/ForceTaskForwardNoPollForwardAllowSyncFlaky Suites
TestAcquireShard_DeadlineExceededErrorSuiteTestAcquireShard_OwnershipLostErrorSuiteTestActivityClientTestSuiteTestActivityTestSuiteTestAdminTestSuiteTestAdvancedVisibilitySuiteTestAdvancedVisibilitySuiteLegacyTestCallbacksSuiteCHASMTestChasmTestSuiteLegacyTestChildWorkflowSuiteTestClientDataConverterTestSuiteTestClientMiscTestSuiteTestCronTestClientSuiteTestCronTestSuiteTestDLQSuiteTestDeploymentVersionSuiteV0TestDeploymentVersionSuiteV2TestFairnessAutoEnableSuiteTestFairnessSuiteTestFuncClustersTestSuiteTestGetHistoryFunctionalSuiteTestNamespaceInterceptorTestSuiteTestNamespaceSuiteTestNexusApiTestSuiteWithTemporalFailuresTestNexusStateReplicationTestSuiteTestNexusWorkflowTestSuiteTestPollerScalingFunctionalSuiteTestPrioritySuiteTestPurgeDLQTasksSuiteTestQueryWorkflowSuiteTestRawHistoryClientSuiteTestRelayTaskTestSuiteTestResetWorkflowTestSuiteTestScheduleFunctionalSuiteTestStandaloneActivityTestSuiteTestTLSFunctionalSuiteTestTaskQueueStats_Classic_SuiteTestTaskQueueStats_Pri_SuiteTestTaskQueueSuiteTestTransientTaskSuiteTestUpdateWorkflowSdkSuiteTestUserMetadataSuiteTestUserTimersTestSuiteTestVersionWorkflowSuiteTestVersioning3FunctionalSuiteV0TestVersioning3FunctionalSuiteV2TestVersioningFunctionalSuiteTestWorkerDeploymentSuiteV0TestWorkerDeploymentSuiteV2TestWorkflowMemoTestSuiteTestWorkflowTaskTestSuiteTestWorkflowUpdateSuite