DataFlowFlexTemplateJob: finalizer fails to delete k8s object if job has been manually canceled

[jeffb4]

Checklist

• I did not find a related open issue.
• I did not find a solution in the troubleshooting guide: (https://cloud.google.com/config-connector/docs/troubleshooting)
• If this issue is time-sensitive, I have submitted a corresponding issue with GCP support.

Bug Description

If a DataflowFlexTemplateJob controlled by KCC is manually canceled, and then the k8s object is deleted, the deletion will fail because the finalizer is unable to cancel the DF job it presumes to exist, that is in state CANCELLED. It will never recover from this state, and the finalizer will need to be removed from the k8s object.

The KCC finalizer should, if DF job cancellation fails and the expected DF job is in state cancelled, allow the k8s object deletion to continue.

Additional Diagnostic Information

Reproduction:

Create a DataflowFlexTemplateJob with KCC
Cancel the job manually from the GCP UI
Quickly delete the KCC object from k8s

Kubernetes Cluster Version

v1.33.5-gke.1162000

Config Connector Version

1.137.0

Config Connector Mode

cluster mode

Log Output

No response

Steps to reproduce the issue

Reproduction:

Create a DataflowFlexTemplateJob with KCC
Cancel the job manually from the GCP UI
Quickly delete the KCC object from k8s

YAML snippets