The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1110059	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#117398
1110034	dotnet/runtime	Build product	Log	dotnet/runtime#118221
1109875	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118211
2761643	dotnet-dotnet	Build (centOSStreamContainer)	Log
2761624	dotnet-orleans	Test	Log
1109815	dotnet/runtime	Send to Helix	Log
2761609	dotnet-dotnet	🛡 Guardian: PoliCheck	Log
1109932	dotnet/roslyn	Run Integration Tests	Log
1109918	dotnet/roslyn	Run Integration Tests	Log
1109866	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#116157
1109774	dotnet/runtime	AOT compile CoreCLR tests	Log
1109769	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#79687
2761440	dotnet-runtime	Publish Browser Wasm Artifacts	Log
2761285	dotnet-dotnet	Build (azurelinuxArm64CrossContainer)	Log
1109291	dotnet/runtime	Send to Helix	Log
1109257	dotnet/runtime	AOT compile CoreCLR tests	Log
1108839	dotnet/runtime	Send tests to Helix (Unix)	Log
2760954	dotnet-orleans	Test	Log
1108889	dotnet/runtime	Send to Helix	Log
1108962	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118183
1108968	dotnet/roslyn	Run Integration Tests	Log
1108891	dotnet/runtime	Send to Helix	Log
1108948	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#79388
1108882	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118185
2760865	dotnet-dotnet	Checkout dotnet-dotnet@release/10.0.1xx-preview7 to s	Log
1108840	dotnet/runtime	Send to Helix	Log
1108853	dotnet/sdk	🟣 Run FullFramework Tests	Log	dotnet/sdk#49947
1108675	dotnet/runtime	Send to Helix	Log
1108670	dotnet/runtime	Send to Helix	Log
1108673	dotnet/runtime	Send to Helix	Log
2760674	dotnet-runtime	Run performance job script	Log
2760574	dotnet-dotnet	Build (centOSStreamContainer)	Log
1108310	dotnet/runtime	Send tests to Helix (Unix)	Log
1108536	dotnet/roslyn	Run Integration Tests	Log
1108378	dotnet/aspnetcore	Run Blazor web app test script	Log	dotnet/aspnetcore#62929
1107823	dotnet/runtime	Send tests to Helix (Unix)	Log
2760136	dotnet-orleans	Test	Log
1107904	dotnet/runtime	Send to Helix	Log
1107990	dotnet/runtime	Send tests to Helix (Unix)	Log
2760073	dotnet-roslyn	Build	Log
1107986	dotnet/runtime	Send job to Helix (Unix)	Log
2760004	dotnet-deployment-tools	🛡 Guardian: Publish SDL Logs to Artifacts	Log
2759910	dotnet-runtime	Run performance job script	Log
2759937	dotnet-roslyn	Test	Log
1107761	dotnet/roslyn	Run Integration Tests	Log
2759921	dotnet-roslyn	Test	Log
1107665	dotnet/roslyn	Run Integration Tests	Log
1107537	dotnet/runtime	Build product	Log	dotnet/runtime#118128
1107278	dotnet/runtime	Send to Helix	Log
2759482	dotnet-dotnet	Build (centOSStreamContainer)	Log
2759614	dotnet-dotnet	Build (azurelinuxX64MuslCrossContainer)	Log
1107306	dotnet/runtime	Build product	Log	dotnet/runtime#118007
2759336	dotnet-aspnetcore	🔒 Publish SBOM manifest	Log
1106911	dotnet/runtime	Send tests to Helix (Unix)	Log
1106915	dotnet/runtime	Send to Helix	Log
2759322	dotnet-orleans	Test	Log
2759222	dotnet-dotnet	Build (centOSStreamContainer)	Log
1106808	dotnet/runtime	Send tests to Helix (Unix)	Log
2759136	dotnet-dotnet	Build (azurelinuxArm64CrossContainer)	Log
1106678	dotnet/runtime	Send tests to Helix (Unix)	Log
2758814	dotnet-runtime	Run performance job script	Log
1106673	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#117800
1106556	dotnet/runtime	AOT compile CoreCLR tests	Log
1106249	dotnet/runtime	Send tests to Helix (Unix)	Log
2758408	dotnet-orleans	Test	Log
1106290	dotnet/runtime	Send to Helix	Log
2758346	dotnet-dotnet	Build (centOSStreamContainer)	Log
2758356	dotnet-runtime	Run performance job script	Log
1106297	dotnet/runtime	Send to Helix	Log
2758340	dotnet-sdk	Run Tests in Helix and non Helix in parallel	Log	#51890
2758325	dotnet-dotnet	🛡 Guardian: PoliCheck	Log
2758291	dotnet-dotnet	🛡 Guardian: AntiMalware Scanner (Source)	Log
2758278	dotnet-dotnet	🛡 Guardian: PoliCheck	Log
2758245	dotnet-runtime	Run performance job script	Log
1106105	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#49932
1106058	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#118037
1106037	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#117996
2758127	dotnet-dotnet	Build (centOSStreamContainer)	Log
2758110	dotnet-dotnet	🛡 Guardian: CredScan (Source)	Log
1106024	dotnet/roslyn	Run Integration Tests	Log
2758103	dotnet-dotnet	Checkout dotnet-dotnet@fix-signing-val-again to s	Log
2757854	dotnet-xharness	🛡 Guardian: TSAUpload	Log
2757822	dotnet-dotnet	Build (azurelinuxX64MuslCrossContainer)	Log
1105084	dotnet/runtime	Send tests to Helix (Unix)	Log
1105147	dotnet/runtime	Send to Helix	Log
1105222	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118043
2757514	dotnet-performance	Send job to Helix (Windows)	Log
1105151	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#118029
1105212	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49847
2757492	dotnet-performance	Send job to Helix (Unix)	Log
1105023	dotnet/roslyn	Run Integration Tests	Log
2757331	dotnet-dotnet	Build (centOSStreamContainer)	Log
1104953	dotnet/runtime	Send to Helix	Log	dotnet/runtime#108655
1104947	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118030
1104939	dotnet/runtime	Send to Helix	Log	dotnet/runtime#117939
2757072	dotnet-runtime	Run performance job script	Log
2757286	dotnet-orleans	Test	Log
1104886	dotnet/runtime	Send to Helix	Log	dotnet/runtime#117991
1104956	dotnet/runtime	Send job to Helix (Unix)	Log
1104910	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#79555
Displaying 100 of 1414 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
15	116	1414

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)

[CarnaViire]

For the "known issues" folks: the timeout here is reported twice, making BA red despite the tracking issue. E.g.:

1️⃣ dotnet/runtime#117255

2️⃣ dotnet/runtime#117800

cc @AlitzelMendez