[Incident] Windows build of plugins don't start on `ci.jenkins.io`

[gounthar]

Service(s)

ci.jenkins.io

Summary

Following my PR on the antexec plugin, https://ci.jenkins.io/job/Plugins/job/antexec-plugin/view/change-requests/job/PR-110/1/ is taking forever to get a Windows VM.

In progress 1 hr 22 min ago in 1 hr 22 min and counting

@timja has detected the same kind of issues.

Reproduction steps

No response

[dduportal]

Workin on it: requires to block all builds on ci.jenkins.io (put in the queue)

[dduportal]

Ok, there is an ongoing Azure network outage which causes these troubles:

This outage was only notified in our Azure Service Health console today (10 January) around 10:30 UTC, and it looks like it's not being reported properly (look at the map below):

[dduportal]

• status.jenkins.io updated (PR: open Azure + ci.jenkins.io Windows containers outage status#577). Permalink to the incident: https://status.jenkins.io/issues/2025-01-08-ci.jenkins.io-azure-outage/
• Banner added to ci.jenkins.io:

[dduportal]

Since this incident has been ongoing unoticed by Microsoft since the past 2 days, we most porbably don't see any full recovery until a few days. Let's wait and see as the team is not available to perform a full region change for the time being and it's not critically blocked.

[dduportal]

Also did an email to the developers mailing list for awareness: https://groups.google.com/g/jenkinsci-dev/c/VQcRiUYu92o

[lemeurherveCB]

Workaround for stuck builds: set useContainerAgent to false in your pipelines as Azure VM agents are spawned successfully since they're not using the impacted ACI service.

[rantoniuk]

• Banner added to ci.jenkins.io:

Unfortunately that banner is not visible on the actual build pages, the user needs to be on the main Jenkins site to see it...

And it seems it's a good feature request candidate :-)

EDIT: feature request created - JENKINS-75122