Feature Request: Allow Filtering Analyzer Execution by Severity During Build

[ericstj]

Summary

Add a compiler/MSBuild switch to control which analyzers are executed during build based on their configured severity (e.g., only run analyzers with severity warning or higher).

Motivation

Currently, analyzers configured as suggestion (or silent) still run during dotnet build, even though their diagnostics are not emitted. This adds unnecessary overhead to builds, especially in large solutions with many analyzers. Developers often want these diagnostics visible in the IDE, but not evaluated during command-line builds.

Proposal

Introduce a new MSBuild property or compiler switch, such as:

<PropertyGroup>
  <RunAnalyzersWithSeverity>WarningOrAbove</RunAnalyzersWithSeverity>
</PropertyGroup>

Related

This request stems from a discussion in dotnet/extensions#7012, where I was investigating why build was so slow in the extensions repo.

Workarounds

• Disabling analyzers entirely during build - will remove the checks we want during development.
• Using custom build configurations to exclude analyzers.
• Swapping .editorconfig files based on context. (complicated and error prone, duplicates content).

cc @stephentoub @jaredpar @baronfel

[RikkiGibson]

Should command-line compilation just always skip analyzers which don't have any diagnostics enabled at warning/error severity?

[stephentoub]

analyzers configured as suggestion (or silent) still run during dotnet build, even though their diagnostics are not emitted

I'm really surprised these are running in a command-line build; I just assumed they wouldn't be, since they have no effect.

What's the rationale for why they do run? Do we not know until they raise diagnostics what their level is?

[RikkiGibson]

We do have the ability to skip analyzers when all their diagnostic IDs are disabled.

When an analyzer throws an exception, we even write out a message like Suppress the following diagnostics to disable this analyzer: ID1, ID2, ID3, ...

If there isn't some known reason for running analyzers whose diagnostics have silent/suggestion severity on the command line, it feels reasonable to skip them. Just take the logic which is skipping the analyzers with disabled diagnostics today, and adjust it so that when running on command line, it demands that some diagnostic for it is configured to have warning or greater severity.

Do we not know until they raise diagnostics what their level is?

I believe we know the severity for any diagnostic by ID before any analyzer runs.

[ericstj]

I wasn't aware of the compiler "knowing" it was a commandline build - I assumed both CLI and IDE talk to the server. If it already knows it's running in the console, then of course this would be better to be automatic and not require the level to be passed.

[RikkiGibson]

Discussion with Manish Vasani, who specialized in analyzers when he was on our team, and results from Copilot, are indicating that analyzers with only hidden/info severity diagnostics, are already supposed to be skipped on the command line.

It would be good to double check on the config for the analyzers which we would expect to be skipped under that behavior.

[jaredpar]

Introduce a new MSBuild property or compiler switch, such as:

Very loath to introduce another toggle to how diagnostics are configured. The current set is already unmaintainable. Prefer we instead try and use the existing set of options / adjust defaults.

I wasn't aware of the compiler "knowing" it was a commandline build - I assumed both CLI and IDE talk to the server.

The IDE does not really talk to the server. IDEs like VS call into msbuild which then eventually invoke the server and the IDE will interpret the results. But there is no direct communication between IDE and Compiler Server. The compiler server is indifferent to the build coming from VS or CLI (so far).

I just assumed they wouldn't be, since they have no effect.

I'd want us to be really sure these have no effect. In dotnet build it has no effect (that I can think of). I'm less certain about builds that originate from Visual Studio. The IDE has a lot of logic to rationalize diagnostics it sees in memory vs. those that exist in builds. These diagnostics can often be side channels for us to signal IDE refactorings. I don't actually know if suggestions or info diagnostics flow from a VS build to the IDE or not (pretty sure yes to info, no to suggestion) but it is a thread we'd need to track down.

[jaredpar]

The IDE has a lot of logic to rationalize diagnostics it sees in memory vs. those that exist in builds. These diagnostics can often be side channels for us to signal IDE refactorings. I don't actually know if suggestions or info diagnostics flow from a VS build to the IDE or not (pretty sure yes to info, no to suggestion) but it is a thread we'd need to track down

@JoeRobich can you comment on this