TWIL: Migrating from CloudFormation to AWS CDK, using Rain

TWIL - This week I learned. This task was actually done two weeks ago, but I thought it could be worth mentioning.

Recently, I tackled the challenge of updating a customer’s SFTP Server solution deployed from AWS Marketplace. The existing CloudFormation template had several issues that needed addressing:

It allowed the use of Instance Metadata Service v1 (IMDS v1), which poses security risks. IMDS v2 should be enforced instead.
Despite having a load balancer in front of EC2 instances, the instances themselves exposed public IP addresses and resided in public subnets.
Client and admin access were limited to single IP ranges (CIDRs) through CloudFormation parameters.
The template used deprecated resources (LaunchConfiguration) and an outdated IAM role.

Since all other infrastructure for this client was built using AWS CDK, it made sense to convert this CloudFormation template to CDK code and implement the necessary improvements. The original template required explicit input for VPC ID, public and private subnet IDs, web admin credentials, and IP ranges for client and admin access.

The First Attempt: Direct Migration

My initial approach was to use the cdk migrate command to convert the existing stack into AWS CDK code. I ran:

cdk migrate --stack-name sftp-gateway --from-stack

This generated a new AWS CDK project with TypeScript files containing the stack resources. However, the generated code had several issues: - It failed to compile due to incorrect property types - Some sections contained invalid TypeScript - CloudFormation conditions weren’t properly translated into code

After working through the compilation issues, I compared the difference between the existing stack and the generated code. There were numerous discrepancies, particularly around how template functions were handled. The original template used the !Sub function extensively, while AWS CDK generated code preferred !Join for similar operations. The differences were too numerous to accept without a more systematic approach.

Enter Rain: A Step-by-Step Transformation

I decided to transform the CloudFormation template gradually using rain, an unofficial but powerful tool from the AWS CloudFormation team. Rain offers an improved command-line experience compared to the AWS CLI and includes several CloudFormation extensions.

First, I exported the existing template:

rain cat sftp-gateway > sftp-gateway.yml

I then began methodically replacing !Sub functions with !Join. While rain diff didn’t provide the same functionality as cdk diff, I used rain deploy -x to generate and review changesets before applying them.

Rain’s constant feature proved particularly useful. Since CDK typically resolves most parameter values at compile-time rather than deploy time, I converted all CloudFormation parameters except the web admin password into Rain constants. This intermediate step made the template more CDK-friendly.

Addressing Security Concerns

Next, I focused on the security issues. The goal was to remove public IP addresses from EC2 instances and enforce IMDS v2. Removing public IPs was straightforward through the launch configuration, but enforcing IMDS v2 proved challenging. In our production environment, changes to the launch configuration don’t automatically replace existing instances. After updating the configuration and cycling an instance, the new instance failed to start properly. Using AWS Systems Manager Session Manager to connect to the EC2 instance, I checked the SFTP software logs and discovered the service had failed to start. Given this issue, I decided to temporarily keep IMDS v2 optional while investigating further.

Eliminating Deprecations

The next phase involved replacing deprecated components. Converting the Launch Configuration to a Launch Template took several iterations but ultimately succeeded. Updating the deprecated IAM roles was considerably simpler.

Solving the IMDS v2 Puzzle

The IMDS v2 issue remained unresolved until I discovered it was related to the AMI version. After investigating the AWS Marketplace for product updates (which wasn’t immediately obvious), I found a newer AMI by simulating a fresh installation. Using this updated AMI finally allowed us to require IMDS v2 successfully.

Final CDK Migration

With these improvements in place, I attempted the CDK migration again. This time, the results were much better, requiring only minor adjustments. The final enhancements included: - Supporting multiple client and admin IP ranges - Securing the web admin password by storing it outside of version control - Updating documentation to reflect the new CDK-based workflow

Lessons Learned

This migration project provided valuable insights: - The cdk migrate tool, while powerful, may not provide a direct path to production-ready code - Rain’s features can help prepare CloudFormation templates for easier migration - Sometimes a gradual, step-by-step approach yields better results than attempting a direct conversion

Overall, the migration was successful, resulting in a more secure, maintainable infrastructure defined in AWS CDK.