Microsoft was able to delete some of our packages without notice - what's the explanation? #14413
Replies: 8 comments 23 replies
-
|
Interestingly enough https://www.nuget.org/packages/Microsoft.Identity.Client/4.72.1 is deprecated and shows the warning (deprecation message)
So that's why I have been seeing new spell check PRs in that repo like Update: Azure.Identity.Broker that was released ~4 hours ago calls out a security fix in its release https://github.com/Azure/azure-sdk-for-net/releases/tag/Azure.Identity.Broker_1.2.1
So strange that there's an inconsistency with the release note for Azure.Identity.
Update4: I think this part of the deprecation message
Is what's causing the real impact, presumably the definitions of some AV was updated (much?) later than 20:th of May when the fixed version of
Hopefully we can be told which AV's and when their definitions were updated. |
Beta Was this translation helpful? Give feedback.
-
|
No no, nothing wrong with your comment at all - I just took inspiration from it! |
Beta Was this translation helpful? Give feedback.
-
|
I accidentally deprecated Microsoft.Identity.Client 4.72.1 and have restored it now. Thanks for catching this @Frulfump . |
Beta Was this translation helpful? Give feedback.
-
|
An earlier report ~4 days ago here AzureAD/microsoft-authentication-library-for-dotnet#5373 even tough they didn't highlight package deletion as an issue they are deleted (maybe the packages were deleted some time later). So Microsoft.Data.SqlClient hasn't been patched yet, this adds onto the whole why is Azure.Identity even a dependency of it dotnet/SqlClient#1108 |
Beta Was this translation helpful? Give feedback.
-
|
@Frulfump Azure.Identity.Broker v1.2.0 is now deprecated. |
Beta Was this translation helpful? Give feedback.
-
|
Looking at https://www.nuget.org/packages/Microsoft.Identity.Client/#usedby-body-tab there's 530 packages that depend on Top 5 packages
Top 9 repos
So presumably there's a lot of packages/repos (still) affected. UPDATE: "-NuGet Team" provided an update #14413 (comment)
It's not possible to link directly to an update or the history section on the status page, future improvement? |
Beta Was this translation helpful? Give feedback.
-
|
What a terrible bad idea this was... So for the sake of the trustworthiness of the ecosystem they decided in my case (a Cake addin) to break existing build pipelines. Because of a typo in an XML comment, in a dependency of a dependency of a dependency 🤷 |
Beta Was this translation helpful? Give feedback.
-
|
We acknowledge that some package deletions have occurred and are actively investigating what led to this action. We understand the importance of package availability and the trust you place in NuGet as a distribution platform. We are committed to transparency and will provide a more detailed update early next week. We are currently restoring affected package versions. If you are an affected package author and need immediate assistance, please contact us at [email protected]. Thank you for your patience while we continue to look into this. -The NuGet Team |
Beta Was this translation helpful? Give feedback.
-
You bet! |
Beta Was this translation helpful? Give feedback.
-
|
Spoiler alert: It was the intern, no but jokingly let's guess what the excuse will be. "A mistake was made and our policy was not followed", "Copilot went brrr" Something about adding spell checking to all Microsoft/dotnet/NuGet repos to avoid the typo squatting. Hope the joke comes across No but seriously appreciate the quick initial response of this unfortunate situation, confirming it is at least something. Like Aaron has hinted at. Feels like an existential issue for NuGet as we can no longer trust the fundamental principle of packages not being deleted. https://learn.microsoft.com/en-us/nuget/nuget-org/policies/deleting-packages
Yes exactly... https://learn.microsoft.com/en-us/nuget/nuget-org/policies/deleting-packages#exceptions
So at least it's a manual process from that so it was an explicit action, was each individual package manually deleted or was there some automation applied? Maybe it falls under https://learn.microsoft.com/en-us/nuget/nuget-org/policies/deleting-packages#prohibited-use but I doubt it
Why wasn't the offending package deleted (actually a good thing I guess but interesting) and only (some?) dependent packages? Why did it take so long for Azure.Identity to get patched (seemingly after it was pointed out on X by Aaron), take us through the time line of events. |
Beta Was this translation helpful? Give feedback.
-
I'm creating a new framework for F# that compiles through MLIR and LLVM (with other back-ends on the roadmap). I don't want to center my rationale for creating my own package management system on my earned distrust of Microsoft. The reasons are primarily structural - as the Fidelity framework requires all dependencies to be sourced-based. However, it is worth mentioning that over the years I've seen enough behaviors from the .NET Foundation and Microsoft writ-large that I no longer have confidence that they will act in accordance with their stated values. This particular incident could have been an "innocent mistake" but in the confluence of how Microsoft is behaving as a community steward, it matters very little whether that behavior is intentional. The saying goes "never attribute to malice that which can be more easily explained by incompetence" but the stakes for this ecosystem being what it is - it hardly matters why it happened. To me it only matters that it happened at all. The current designs for my compiler still use dotnet tooling, along with other support components that I eventually plan to self-host once the F#->MLIR->LLVM landscape is broad enough to support them. But now I'm reconsidering where self-hosting stands on the priority down-list now that I've seen this sequence of events transpire. While my main motivation for creating this framework is to have "native" options for F# along with .NET and Fable Compiler ecosystems, it is now a non-zero consideration to move away from .NET altogether as a matter of business continuity. I hope those observing this event from Microsoft management perspective are seeing how serious this situation is, but I certainly don't expect them to respond in accordance with that seriousness. That echelon of Microsoft has merely demonstrated concern/interest that barely rises to the level of "performative". It's a shame to see "cracks in the edifice" show this way. But it underscores (to me, at least) that alternative ecosystems and maintaining optionality for developers is more important than ever. The future of technology is much bigger than Microsoft. That future is too important to leave it solely in the hands of a monopolist that acts unilaterally in its own perceived interest irrespective of its stated goals as a community "citizen". |
Beta Was this translation helpful? Give feedback.
-
|
Sometimes there are problems and sometimes they get fixed. This strikes me as the type of problem that will probably get fixed. It seems like you make a lot of valuable contributions to the OSS community and that's awesome. The following is intended with compassion and I hope that is how it is received. Regarding
It feels unkind for nobody to tell you that it is very difficult to see this as anything other than embarrassingly opportunistic. |
Beta Was this translation helpful? Give feedback.
-
|
I appreciate your input, but I'm not sure what the opportunism is here. I'm mourning the loss of trust due to a company that shows multiple consistent signals of being "too big to care". For years I was pretty firmly "in the camp" of Microsoft because of what I saw then as preserving the aligned interest with the customer, as opposed to the surveillance capitalism of other technology-adjacent companies like Google and Meta. That (perhaps naive) perception has been pretty deeply cut over the past few years and I see no sign of reversal. I'm glad to see the attention this particular episode has gotten has triggered some observable response to reverse the immediate harms that were done. But my point here is about the undermining of trust. I'm not confident that tactical scurrying about after-the-fact changes my position on thinking about the direction of with my work. For certain, I really enjoy the Fable compiler and appreciate the huge amount of independent work they put in for web development in F#. I originally saw the Fidelity framework as a means to use LLVM's targeting to deploy type-safe F# code to a variety of targets and formats where a .NET style runtime is a poor fit (micro-controllers and accelerators). The original thesis for my work was to keep .NET "in its sweet spot", and use Fable and Fidelity for the rest. The interop work there is non-trivial. That hasn't changed with this event. It can't. I am, like the rest of us, trapped by the decisions of a 'steward' who - to my view - shows only performative interest in responding to the broader community where it suits their marketing posture. I suppose this event has gotten more attention than I was expecting. That would mean, among other things, that the free labor provided by "the community" includes policing Microsoft's behavior such that the kerfuffle triggers an appropriate response, along with all of the time spent on clean-up after they've wrecked the room. I certainly don't have time for that kind of work. But if you're deeply vested, count it as part of the cost of doing business in this ecosystem. To be sure, I'm not just referring to nuking packages in NuGet. It's also "pulling up the drawbridge" for VSCode extension support in forked apps. It's also about reshaping AI as an ersatz remote desktop surveillance tool. The list goes on and on. I find it hard to take any conversation - in person or online - as a serious one if one side of it refuses to acknowledge the consistent behavior of an aspiring monopolist. That includes unilateral discarding of their stated policies, whether by oversight or deliberate lack of care. To the community, this is a serious error. To them, it's a rounding error. Frankly I didn't expect my note to get any response at all because my area of interest is F#. Feel free to draw a line under it. ☮️ |
Beta Was this translation helpful? Give feedback.
-
|
A quick note on progress: We have already restored packages removed on July 9th & 10th and are working on restoring the rest as quickly as possible. Current ETA is end of tomorrow. Please see https://status.nuget.org/ for further updates. We'll provide a more detailed report later this week. -NuGet team |
Beta Was this translation helpful? Give feedback.
-
|
Nice to get an update of any sort, still hoping for the detailed report tough. By why do we need to manually and periodically check status.nuget.org for updates, you can provide them here as part of proper incident response. Yes please we need the whole list (previously requested in this comment #14413 (reply in thread)) From the status page (it's not possible to link directly to an update or the history section, future improvement?)
So it's taking a day to restore 80 packages that seems slow, could you walk us through the process so we understand why it's taking so long? The deletion event is kind of binary, we are in a post-deletion world it doesn't matter much that the packages are restored as all known cases in this discussion have already made fixes to their packages and uploaded new versions. The damage has already been done. Tim mentioned here #14413 (comment)
But that has then changed to later this week? Why initially mention early this week and then retract from that, would have been better to provide a broader window from the start. (Under promise over deliver should be a guiding star). |
Beta Was this translation helpful? Give feedback.
-
There are cases where it matters that they are restored. There are now versions cached on caching feeds which no longer exist upstream, leading to confusion. And in a pinned down deterministic scenario the missing packages will still break / render other packages usable. E.g. I've a set of packages which are versioned and released together, only one of it was deleted, but other have a dependency on it. If it wouldn't be restored I would need to unlist the other packages to avoid confusion. |
Beta Was this translation helpful? Give feedback.
-
That's true and I agree. Do you have a link to your affected package repo? |
Beta Was this translation helpful? Give feedback.
-
Sure, the above example is https://www.nuget.org/packages/Cake.Frosting.Issues.PullRequests.AzureDevOps/5.6.0 |
Beta Was this translation helpful? Give feedback.
-
|
Another status update that haven't been mentioned in this discussion. ~14 hours ago (2025-07-17T00:19:24Z)
Hoping for that detailed report very soon! |
Beta Was this translation helpful? Give feedback.
-
|
First, thanks for restoring the packages. Honestly, I'm still not happy how this issue is handled: 
This status on nuget.org sounds good, but Looking forward to hearing the RCA and getting answers to the questions raised by others in this issue. |
Beta Was this translation helpful? Give feedback.
-
|
I agree with that. I also think they should've provided that status update in this discussion as part of proper incident response and not force us to manually check status.nuget.org (I didn't know there had been an update before your comment.) |
Beta Was this translation helpful? Give feedback.
-
|
Seems one maintainer received an email apology very recently serilog-mssql/serilog-sinks-mssqlserver#624 (comment)
|
Beta Was this translation helpful? Give feedback.
-
|
I now also received one, after I've written the above complain. Still don't get the order of actions, package was already restored over a day ago. Status is also 11 hours old. Emails usually travel pretty fast :-) |
Beta Was this translation helpful? Give feedback.
-
|
We also received one of these - so it sounds like an automated malware detection system was responsible for this? |
Beta Was this translation helpful? Give feedback.
-
|
Yeah seems like it. |
Beta Was this translation helpful? Give feedback.
-
|
I created a feature request here #14425 for allowing you to check if any package in your dependency graph has been deleted. By adding a new command
|
Beta Was this translation helpful? Give feedback.
-
|
After fully restoring the deleted packages, we have published a detailed update from the Root Cause Analysis/Postmortem. Our sincere thanks for your input, patience and understanding. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Full version here: https://aaronstannard.com/microsoft-delete-nuget-packages/
TL;DR version:
We received this email to our NuGet administrator email addresses yesterday morning
Email mentions a vulnerability in Microsoft.Identity.Client and that some recent package versions of ours were impacted by it.
We discover later, when trying to build one of those projects, that both of those affected package versions were hard-deleted from NuGet.org.
We absolutely didn't delete them ourselves, and we're not the only affected authors: https://x.com/spin973/status/1943714651964915882
NuGet makes it very, very had to delete packages on-purpose - in order to avoid
left-pad-style issues. Why did Microsoft bypass the normal CVE procedure on NuGet and hard-delete third party packages? What's the limiting principle at work here? And why was this CVE (a typo in an XML-DOC comment, come on) so urgent that it merited this extraordinary work-around?I depend on NuGet to distribute my intellectual property to my customers - the indefinite availability of packages is an essential part of that guarantee. Will my packages be arbitrarily deleted again in the future without any notice the next time a Microsoft team introduces a severe vulnerability in their code?
Beta Was this translation helpful? Give feedback.
All reactions