feat: Add CancelHealthCheckOnNewRevision feature to avoid getting stuck on failing commits

[trashhalo]

Summary

This PR introduces the CancelHealthCheckOnNewRevision feature flag to prevent kustomize-controller from getting stuck waiting for health check timeouts when new source revisions containing potential fixes are available.

Problem

Currently, when a Kustomization fails health checks (e.g., due to a bad deployment), the controller waits for the full timeout duration (typically 30 seconds) before processing any new revisions. This means that even if a fix is pushed immediately after the failing commit, users must wait for the full timeout before the fix is applied.

Solution

• New opt-in feature flag: CancelHealthCheckOnNewRevision (default: false)
• Revision monitoring: During health checks, monitor for new source revisions every 5 seconds
• Early cancellation: Cancel ongoing health checks when new revisions are detected
• Immediate processing: Process new revisions immediately after cancellation instead of waiting for timeout

Behavior Change

Before (existing behavior):

[Bad commit] → Health check starts → Wait 30s → Timeout → Process new commit
Total time: ~30+ seconds

After (with feature enabled):

[Bad commit] → Health check starts → [New commit arrives] → Cancel health check (~5s) → Process new commit immediately  
Total time: ~5-10 seconds

Implementation Details

1. Feature flag integration: Uses the existing feature gate system
2. Context-based cancellation: Creates cancellable contexts for health checks
3. Background monitoring: Goroutine monitors source revisions during health checks
4. Graceful cancellation: Proper cleanup and error handling when cancelling
5. Backward compatibility: Preserves existing behavior when feature is disabled

Testing

Comprehensive test coverage includes:

1. TestKustomizationReconciler_CancelHealthCheckOnNewRevision:

   • Verifies feature works when enabled
   • Demonstrates ~5 second cancellation vs 30 second timeout
   • Confirms immediate processing of new revisions

2. TestKustomizationReconciler_NoHealthCheckCancellation_WhenFeatureDisabled:

   • Verifies original behavior when feature is disabled
   • Demonstrates full 30 second timeout behavior
   • Ensures backward compatibility

Usage

Enable the feature by starting kustomize-controller with:

--feature-gates=CancelHealthCheckOnNewRevision=true

Benefits

• Faster recovery: 6x faster processing of fixes (5s vs 30s)
• Improved user experience: Reduced delay between pushing fixes and seeing them applied
• Production ready: Opt-in feature with comprehensive testing
• Safe: Preserves existing behavior when disabled

🤖 Generated with Claude Code

[stefanprodan]

Thanks for this PR @trashhalo can you please signoff your commit and address my comments

[trashhalo]

@stefanprodan thanks for taking a pass! I'll get this all cleaned up based on your feedback over the next day or so.

[stefanprodan]

@trashhalo we are release kustomize-controller today, so if we want this to be part of Flux 2.7 this PR needs to be merged today. Otherwise this feature will have to wait till next year when we'll release Flux 2.8

[stefanprodan]

Ok we had enough of Claude Code, reviewing this for you to input the review in AI and push here takes too much of our time.

@matheuscscp is going to rewrite this PR, you can stop pushing changes.

[matheuscscp]

Superseded by #1520

[trashhalo]

Ok we had enough of Claude Code, reviewing this for you to input the review in AI and push here takes too much of our time.

@matheuscscp is going to rewrite this PR, you can stop pushing changes.

I'm sorry.