CUMULUS-4181: Re-release version 20.2.0 with RDS v17 upgrade fixes (#4036)

* CUMULUS-4170: Update node-buster to a newer debian build for CI (#4012)

* first committ

* CHANGELOG

* changing local back

* add me to edl users (#4024)

* add me to edl users

* updage dependency version

* changelog

* safe instead of un=vuln

* CUMULUS-4176: canonize versioned parameter groups for postgres engine upgrade (#4033)

* first commit

* doc-fixes

* Update CHANGELOG.md

Co-authored-by: jennyhliu <[email protected]>

* Update CHANGELOG.md

Co-authored-by: jennyhliu <[email protected]>

* small doc fix

---------

Co-authored-by: jennyhliu <[email protected]>

* Fix CreateReconciliationReportSpec test cleanup

* update changelog and docs

* update changelog release ref

---------

Co-authored-by: Naga Nages <[email protected]>
Co-authored-by: etcart <[email protected]>
Co-authored-by: Jonathan Kovarik <[email protected]>

Author: 4 people

CHANGELOG.md

@@ -6,6 +6,32 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
 ## [Unreleased]
 
+## [v20.2.0] 2025-08-06 Updated
+
+### Migration Notes
+
+From this release forward, Cumulus Core will be tested against PostgreSQL v17. Users
+should migrate their datastores to Aurora PostgreSQL 17.4+ compatible data
+stores as soon as possible after upgrading to this release.
+
+#### Database Upgrade
+
+Users utilizing the `cumulus-rds-tf` module should reference [cumulus-rds-tf upgrade
+instructions](https://nasa.github.io/cumulus/docs/upgrade-notes/upgrade-rds-cluster-tf-postgres-17).
+
+### Fixed
+
+- **CUMULUS-4176**
+  - Updated `engine_version` default value to `17.4` in `example/rds-cluster-tf/variables.tf`.
+  - Updated `tf-modules/cumulus-rds-tf` module to take additional parameter `enable_upgrade` in support of
+    migration from Aurora PostgreSQl v13 to v17.
+- **CUMULUS-4177**
+  - Update form-data sub-dependency to safe version ^4.0.4
+- **CUMULUS_4174**
+  - Fix broken CreateReconciliationReportSpec test cleanup
+- **CUMULUS-4170**
+  - Upgrade Node Docker image from buster to bullseye for a compatible debian version
+
 ## [v20.2.0] 2025-06-24
 
 ### Added
@@ -8610,7 +8636,8 @@ Note: There was an issue publishing 1.12.0. Upgrade to 1.12.1.
 ## [v1.0.0] - 2018-02-23
 
 
-[Unreleased]: https://github.com/nasa/cumulus/compare/v20.1.2...HEAD
+[Unreleased]: https://github.com/nasa/cumulus/compare/v20.2.0...HEAD
+[v20.2.0]: https://github.com/nasa/cumulus/compare/v20.1.2...v20.2.0
 [v20.1.2]: https://github.com/nasa/cumulus/compare/v20.1.1...v20.1.2
 [v20.1.1]: https://github.com/nasa/cumulus/compare/v20.0.1...v20.1.1
 [v20.0.1]: https://github.com/nasa/cumulus/compare/v20.0.0...v20.0.1

bamboo/Dockerfile

@@ -1,5 +1,5 @@
 ## Dockerfile to create integration/unit test environment
-FROM node:20.12.2-buster
+FROM node:20.12.2-bullseye
 RUN apt update &&\
     apt install -y netcat zip jq rsync zip
 RUN apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev \

bamboo/bootstrap-tf-deployment.sh

@@ -2,7 +2,7 @@
 set -ex
 
 apt-get update
-apt-get install -y python-pip
+apt-get install -y python3-pip
 pip install awscli
 
 TF_VERSION=$(cat .tfversion)

docs/development/release.md

@@ -173,7 +173,7 @@ If there is exists a PR in the cumulus-dashboard repo with a name containing: "V
 
 ### 4. Publish async_operations image to Docker Hub
 
-For consistency and security reasons, a [new docker image for async_operations should be created and published](../../packages/api/ecs/async-operation/README.md) for each release and pushed to Docker Hub and possibly AWS ECR. Make sure to follow the instructions for **Updating the Cumulus deployment configuration**
+For consistency and security reasons, a [new docker image for async_operations should be created and published](https://github.com/nasa/cumulus/blob/master/packages/api/ecs/async-operation/README.md) for each release and pushed to Docker Hub and possibly AWS ECR. Make sure to follow the instructions for **Updating the Cumulus deployment configuration**
 
 ### 5. Update CHANGELOG.md
 

docs/tasks.md

@@ -17,6 +17,24 @@ Add checksums to files in S3 which don't have one
 
 ---
 
+### [@cumulus/change-granule-collection-pg](https://github.com/nasa/cumulus/tree/master/tasks/change-granule-collection-pg)
+
+Move granule files and postgres records to a new collection
+
+- Schemas: See this module's [schema definitions](https://github.com/nasa/cumulus/tree/master/tasks/change-granule-collection-pg/schemas).
+- Resources: [npm](https://npmjs.com/package/@cumulus/change-granule-collection-pg) | [source](https://github.com/nasa/cumulus) | [web](https://github.com/nasa/cumulus/tree/master/tasks/change-granule-collection-pg)
+
+---
+
+### [@cumulus/change-granule-collection-s3](https://github.com/nasa/cumulus/tree/master/tasks/change-granule-collection-s3)
+
+Move granule files and postgres records to a new collection
+
+- Schemas: See this module's [schema definitions](https://github.com/nasa/cumulus/tree/master/tasks/change-granule-collection-s3/schemas).
+- Resources: [npm](https://npmjs.com/package/@cumulus/change-granule-collection-s3) | [source](https://github.com/nasa/cumulus) | [web](https://github.com/nasa/cumulus/tree/master/tasks/change-granule-collection-s3)
+
+---
+
 ### [@cumulus/discover-granules](https://github.com/nasa/cumulus/tree/master/tasks/discover-granules)
 
 Discover Granules in FTP/HTTP/HTTPS/SFTP/S3 endpoints

docs/upgrade-notes/upgrade-rds-cluster-tf-postgres-17.md

@@ -4,7 +4,7 @@ title: Upgrade Database Cluster to PostgreSQL v17
 hide_title: false
 ---
 
-Cumulus Core as of version > 20.1.2** now supports and is tested against Aurora Postgres v17.   All users should update their datastores to this version as part of an upgrade process upon upgrading to release version 20.1.2.
+Cumulus Core as of version **>= 20.2.0** now supports and is tested against Aurora Postgres v17. All users should update their datastores to this version as part of an upgrade process upon upgrading to release version 20.2.0.
 
 We recommend stopping all ingest rules if database downtime is required (e.g. you do not have a blue-green database solution or are using serverless V2) for the update  as any unavailability of the database may result in unexpected database write failures (resulting in records in the Dead Letter Archive), workflow failures or other unexpected failures.
 
@@ -14,21 +14,20 @@ We recommend stopping all ingest rules if database downtime is required (e.g. yo
 
 It is recommended that users manually backup and/or consider cloning their datastore in order to recover the datastore if an upgrade goes awry.
 
-Upgrading the Aurora Serverless v2 cluster will be completed via AWS console in this document and require manual steps to complete the upgrade:
-If you are on version 13.12, it is recommended that you upgrade to version 13.18 prior to upgrading to 17.x. The AWS RDS for PostgreSQL upgrade document may be used as a reference:
+Upgrading the Aurora Serverless v2 cluster will be completed via AWS console in this document and require manual steps to complete the upgrade. The AWS RDS for PostgreSQL upgrade document may be used as a reference:
 <https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.PostgreSQL.MajorVersion.html>
 
-- Ensure a supported version (> 20.1.2 a later patch version) is deployed.
+- Ensure a supported version (> 20.1.2 or a later patch version) is deployed.
 - Deploy the newest version of the `cumulus-rds-tf` module, ensuring `enable_upgrade` is set to false.   This will *only* deploy a `v17` version of your current parameter group configuration, named `<prefix>-cluster-parameter-group-v17`.
 - Shut down all ingest and other usage of the database cluster by 3rd party applications if appropriate.
 - Once this is done, utilize the AWS RDS console to `modify` the database cluster, and update the following settings:
-  - Set `Engine Version`  to the currently available Serverless v2 Postgres v17 engine (PostgreSQL 17.4 as of this instruction set’s authoring)
+  - Set `Engine Version` to the currently available Serverless v2 Postgres v17 engine (PostgreSQL 17.4 as of this instruction set’s authoring)
   - Ensure the min/max capacity settings match expected values and have not changed
   - DB cluster parameter group - utilize the newly created parameter group from step #2 for the update.
 - Once you have completed the modifications, click `Continue` and verify the `Summary of modifications` has the engine version and modified parameter group.
 - **Important:** Update the `Schedule modifications` to apply the change immediately.
 
-    Once this is done, apply the updates. The database upgrade will begin, and the database will shutdown/restart repeatedly.    You can monitor progress in the database cluster’s `Logs & events` tab.
+    Once this is done, apply the updates. The database upgrade will begin, and the database will shutdown/restart repeatedly. You can monitor progress in the database cluster’s `Logs & events` tab.
 
     Upon completion, you should expect to see output similar to:
 
@@ -38,5 +37,5 @@ If you are on version 13.12, it is recommended that you upgrade to version 13.18
     ```
 
 - On update completion, validate database cluster appears to have restarted with the expected configuration, non-cumulus databases, etc.
-- Update the `enable_upgrade` `rds-cluster-tf` module variable to `true`, and run `terraform init` and `terraform apply` to ensure the postgres v17 compatible parameter group is cleaned up. This should be the only change so double-check the changeset or run `terraform plan` to be sure.
+- Update the `enable_upgrade` `rds-cluster-tf` module variable to `true`, and run `terraform init` and `terraform apply` to ensure the postgres v13 compatible parameter group is cleaned up. This should be the only change so double-check the changeset or run `terraform plan` to be sure.
 - Resume use of the database cluster.

example/cumulus-tf/main.tf

@@ -163,6 +163,7 @@ module "cumulus" {
     "acyu",
     "chuang14",
     "ds_jennifertran",
+    "ecarton",
     "efenollal",
     "filipgraniczny",
     "jasmine",

example/rds-cluster-tf/main.tf

@@ -28,5 +28,7 @@ module "rds_cluster" {
   snapshot_identifier        = var.snapshot_identifier
   lambda_timeouts            = var.lambda_timeouts
   lambda_memory_sizes        = var.lambda_memory_sizes
+  enable_upgrade             = var.enable_upgrade
   parameter_group_family_v13 = var.parameter_group_family_v13
+  parameter_group_family_v17 = var.parameter_group_family_v17
 }

example/rds-cluster-tf/variables.tf

@@ -73,7 +73,7 @@ variable "tags" {
 variable "engine_version" {
   description = "Postgres engine version for Serverless cluster"
   type        = string
-  default     = "13.12"
+  default     = "17.4"
 }
 
 variable "vpc_tag_name" {
@@ -88,6 +88,12 @@ variable "subnets_tag_name" {
   default = "Private application *"
 }
 
+variable "enable_upgrade" {
+  description = "Flag to enable use of updated parameter group"
+  type = bool
+  default = false
+}
+
 variable "lambda_memory_sizes" {
   description = "Configurable map of memory sizes for lambdas"
   type = map(number)
@@ -109,3 +115,9 @@ variable "parameter_group_family_v13" {
   type = string
   default = "aurora-postgresql13"
 }
+
+variable "parameter_group_family_v17" {
+  description = "Database family to use for creating database parameter group under postgres 17 upgrade conditions"
+  type = string
+  default = "aurora-postgresql17"
+}

example/spec/parallel/createReconciliationReport/CreateReconciliationReportSpec.js

@@ -393,6 +393,7 @@ describe('When there are granule differences and granule reconciliation is run',
 
       console.log('dbGranuleId', dbGranuleId);
       console.log('publishedGranuleId', publishedGranuleId);
+      console.log('cmrGranule', cmrGranule);
 
       console.log('XXXXX Waiting for collections in list');
       const collectionIds = [
@@ -901,7 +902,7 @@ describe('When there are granule differences and granule reconciliation is run',
       removeCollectionAndAllDependencies({ prefix: config.stackName, collection }),
       s3().deleteObject(extraS3Object),
       deleteFolder(config.bucket, testDataFolder),
-      cmrClient.deleteGranule(cmrGranule),
+      cmrClient.deleteGranule(cmrGranule.granuleId),
     ]);
     cleanupResults.forEach((result) => {
       if (result.status === 'rejected') {