Feature Request: Configurable Latest Tags Count for ImageRepository

[kahirokunn]

Current Behavior

Currently, the image-reflector-controller in FluxCD limits the number of latest tags to 10 for all ImageRepository resources. This is hardcoded in the getLatestTags function:

image-reflector-controller/internal/controller/imagerepository_controller.go

Line 61 in 15a1b09

const latestTagsCount = 10

image-reflector-controller/internal/controller/imagerepository_controller.go

Lines 630 to 642 in 15a1b09

	func getLatestTags(tags []string) []string {
	var result []string
	sort.SliceStable(tags, func(i, j int) bool { return tags[i] > tags[j] })
	if len(tags) >= latestTagsCount {
	latestTags := tags[0:latestTagsCount]
	result = append(result, latestTags...)
	} else {
	result = append(result, tags...)
	}
	return result
	}

Desired Behavior

We would like to make the number of latest tags configurable on a per-ImageRepository basis, with a default of 10. This would allow users to retrieve more (or fewer) latest tags as needed for specific repositories.

Proposed Solution

We propose adding a latestTagsCount field to the ImageRepository spec:

apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
  name: app1
  namespace: apps
spec:
  latestTagsCount: 100  # New field
  interval: 1h
  image: docker.io/org/image
  exclusionList:
    - "^.*\\.sig$"
    - "1.0.2"
    - "1.1.1|1.0.0"

Implementation Details

1. Update the ImageRepository CRD to include the new latestTagsCount field.
2. Modify the getLatestTags function to accept a parameter for the count:

func getLatestTags(tags []string, count int) []string {
    // ... implementation using the provided count
}

3. Update the controller logic to use the specified count from the ImageRepository spec, falling back to the default of 10 if not specified.

Benefits of Increasing Latest Tags Count

1. Better visibility into recent releases: For projects with frequent releases or using date-based versioning, having access to more than 10 latest tags provides a more comprehensive view of recent project history.

2. Improved automation capabilities: With more tags available, users can create more sophisticated automation workflows, such as rolling back to specific versions or tracking longer-term trends in releases.

3. Enhanced compatibility with various versioning schemes: Some projects may use versioning schemes that produce many tags in a short time (e.g., build numbers or git commit hashes). A higher tag count allows for better compatibility with these schemes.

4. More flexible image selection: Users can implement more nuanced policies for selecting images based on a larger pool of recent tags.

5. Better support for canary releases and A/B testing: With access to more tags, users can more easily manage multiple versions of an application for testing or gradual rollout purposes.

Using CEL for Validation

To ensure that the latestTagsCount is not set to a value less than 10, we can use Common Expression Language (CEL) for validation. Here's an example of how to implement this validation:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: imagerepositories.image.toolkit.fluxcd.io
spec:
  # ... other CRD fields ...
  versions:
    - name: v1beta2
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                latestTagsCount:
                  type: integer
                  default: 10
              x-kubernetes-validations:
                - rule: "has(self.latestTagsCount) ? self.latestTagsCount >= 10 : true"
                  message: "latestTagsCount must be greater than or equal to 10"

This CEL rule does the following:

1. It checks if latestTagsCount is present in the spec.
2. If it is present, it validates that the value is greater than or equal to 10.
3. If it's not present, the validation passes (allowing the default value of 10 to be used).

This approach ensures that users can increase the latestTagsCount beyond 10, but prevents them from setting it to a value less than 10, maintaining a minimum level of recent tag history.

[stefanprodan]

The latest tags number is just for display, the controller stores all tags of an image, not only the latest 10. The policies for a specific image repo, are executed on the full list of tags.

[kahirokunn]

@stefanprodan
Thank you for the clarification regarding the storage and display of image tags. There are situations during troubleshooting when we may need to reference a broader range of tags. Additionally, we are interested in creating a controller to automate this process.

[stefanprodan]

The assumptions made in this issue are quite wrong, the number of tags from status play no role in the controller's behaviour. Also the tags listed in status can't be used for any decision making because the upstream registry can serve pages in any order, what you think it's latest can be oldest. The proper way to select "latest" is using an ImagePolicy that looks at all tags and selects latest based on the regex/semver.

[kahirokunn]

@stefanprodan
Thank you for your reply. I understand that selecting tags using ImagePolicy is recommended, and that the order of tags in the repository may not always be reliable. However, the controller I am developing personally intends to process the tags listed in the ImageRepository status.

It’s not a problem for us if the order of the tags in the status is not exact. Rather than relying on the order, we want to apply custom processing logic based on the tags listed in the status. With that said, I would still like to have the ability to increase the number of tags stored in the status for the following reasons:

1. Custom Logic: My controller wants to process the tags listed in the ImageRepository status. Increasing the number of tags would allow for better decision-making based on more information, enabling more flexible processing.

2. Reducing Interactions with the Upstream Registry: There are scenarios where the upstream registry API has limitations, or where frequent requests can become costly. By storing more tags in the status, we can reduce the number of interactions with the upstream registry and improve the performance of the controller.

For these reasons, I believe that making the number of tags stored in the ImageRepository status configurable would make it easier for other controllers to integrate with FluxCD. I would appreciate it if you could consider this request.