In part 1 we introduced an AWS CloudFormation feature called Drift Detection, with which we can detect (uncontrolled) changes to the resources we manage via CloudFormation. We showed this in the AWS Management Console.
In this part, we are creating simple scripts to start the drift detection process on our CloudFormation stacks, as well as checking the result of detected drifts on the stacks. This will allow us not only to check on a single stack at a time but also on multiple stacks and multiple regions in one operation.
—READMORE—
We will then in part 3 take this a step further and perform continuous checks on CloudFormation stacks, by deploying these checks via CloudFormation as well.
The drift detection features we looked at in part 1 are of course available through the various AWS SDKs (Software Development Kits) and thus also through AWS CLI, as well as AWS Tools for PowerShell.
In this article, we will focus on two scripts, one to start the drift detection, and another to collect the drift detection results. The scripts are in the Github repository cloudgnosis/tidycloud-aws-utilities https://github.com/cloudgnosis/tidycloud-aws-utilities/tree/main/cloudformation/scripts.
Both these scripts are written in PowerShell, which is available for macOS, Linux, and Windows - I use it on macOS for scripting tasks like this. See section How to install PowerShell on how to set it up.
It uses AWS Tools for PowerShell to access AWS services, see section Introduction to AWS Tools for PowerShell for introduction and Installation of AWS Tools for PowerShell for installation.
Script set-up
Steps to run the script
There are two (three) steps to perform drift detection for your CloudFormation resources:
Install the necessary AWS Tools for PowerShell modules (one-time operation)
Set your AWS credentials to use
Run the Start-DriftDetection.ps1 script to initiate the drift detection
Install the necessary AWS Tools for PowerShell modules
With AWS Tools for PowerShell installed, run the command
Install-AWSToolsModule CloudFormationThis will install the required modules. This is a one-time operation and is not needed next time.
Set your AWS credentials to use
This is done via the Set-AWSCredential command in PowerShell. If you already have an AWS credentials profile on the computer, just run
Set-AWSCredential -ProfileName yourprofilereplacing yourprofile with the name of your AWS credentials profile. Or you can specify access keys directly
Set-AWSCredential -AccessKey keyid -SecretKey secretaccesskeyThis will temporarily save the credentials for the current PowerShell session. As long as you use the same credentials, you do not need to run this again until either the session ends or the credentials expire.
Start the drift detection
The first script, Start-DriftDetection.ps1, can in its simplest form be executed as:
./Start-DriftDetection.ps1 -Region eu-north-1This will fetch information about all the CloudFormation stacks in the eu-north-1 region, and start drift detection on each one of them. The -Region parameter can take multiple regions as input, and optionally also a set of stack names. If -StackName is specified, then only the stacks with matching names will be checked.
In addition, only stacks with the appropriate state will be checked as well. For example, stacks that are executing right now will not be checked.
The bulk of the code is rather simple:
$validStackStatus = "CREATE_COMPLETE","UPDATE_COMPLETE","UPDATE_ROLLBACK_COMPLETE","UPDATE_ROLLBACK_FAILED"
$result = @()
foreach ($currentRegion in $Region) {
if ($StackName.Count -gt 0) {
$stacks = Get-CFNStack -Region $currentRegion | Where-Object -Property StackName -In -Value $StackName
} else {
$stacks = Get-CFNStack -Region $currentRegion
}
foreach ($stack in $stacks) {
if ($stack.StackStatus -in $validStackStatus) {
Start-CFNStackDriftDetection -Region $currentRegion -StackName $stack.StackName >$null
$result += [PSCustomObject]@{ StackName=$stack.StackName; Region = $currentRegion }
}
}
}
$resultThe output will be the names and regions of the stacks on which drift detection was started.
Checking drift detection results
The second script, Get-StackDriftStatus.ps1, will check the drift status of the selected stacks. If there is any stack that is in status Drifted, it will go through and check the drift status of the resources in the stack. For any resources that are not in sync it will collect and report information about:
- Drift status
- Type of resource
- Physical id
- Actual and expected resource information
The actual and expected resource information may be somewhat complex JSON structures, and in many cases, some additional tooling for better formatting of these may be needed.
In the PowerShell scripts, these have been converted to hashtable to be easier to work with any PowerShell filtering tools, but can be converted back to JSON also, if needed.
Example outputs
For the drift example we used in the part 1 article, we get the following output with just the script call itself:
❯ ./Get-StackDriftStatus.ps1 -Region eu-north-1
StackName DriftDetails
--------- ------------
demo-stack @{ResourceType=AWS::EC2::SecurityGroup; PhysicalId=sg-09ed21dc4e3422a6b; ResourceDriftStatus=MODI…This in itself is not that informative, so let us expand the output here a bit:
❯ ./Get-StackDriftStatus.ps1 -Region eu-north-1 | Select-Object -ExpandProperty DriftDetails
ResourceType : AWS::EC2::SecurityGroup
PhysicalId : sg-09ed21dc4e3422a6b
ResourceDriftStatus : MODIFIED
Actual : {SecurityGroupEgress, VpcId, GroupDescription, Tags…}
Expected : {SecurityGroupEgress, VpcId, GroupDescription, Tags…}Now, we know that the drift was in a security group and also the ID of the security group. What we do not see the details of here is the details in the CloudFormation for the security group, what the actual state is, and what the expected state should be.
We can convert each one of them further, or we can also generate JSON data for the drift details:
❯ ./Get-StackDriftStatus.ps1 -Region eu-north-1 | Select-Object -ExpandProperty DriftDetails | ConvertTo-Json -depth 10
{
"ResourceType": "AWS::EC2::SecurityGroup",
"PhysicalId": "sg-09ed21dc4e3422a6b",
"ResourceDriftStatus": {
"Value": "MODIFIED"
},
"Actual": {
"SecurityGroupEgress": [
{
"CidrIp": "0.0.0.0/0",
"Description": "Allow all outbound traffic by default",
"IpProtocol": -1
}
],
"VpcId": "vpc-0ee5b371418527493",
"GroupDescription": "demo-stack/fleet/autoscaling-group/InstanceSecurityGroup",
"Tags": [
{
"Value": "demo-stack/fleet/autoscaling-group",
"Key": "Name"
}
],
"SecurityGroupIngress": [
{
"ToPort": 8081,
"IpProtocol": "tcp",
"SourceSecurityGroupOwnerId": 123456789012,
"Description": "Load balancer to target",
"SourceSecurityGroupId": "sg-05521e94d01e7a12a",
"FromPort": 8081
}
]
},
"Expected": {
"SecurityGroupEgress": [
{
"CidrIp": "0.0.0.0/0",
"Description": "Allow all outbound traffic by default",
"IpProtocol": -1
}
],
"VpcId": "vpc-0ee5b371418527493",
"GroupDescription": "demo-stack/fleet/autoscaling-group/InstanceSecurityGroup",
"Tags": [
{
"Value": "demo-stack/fleet/autoscaling-group",
"Key": "Name"
}
],
"SecurityGroupIngress": [
{
"ToPort": 8080,
"IpProtocol": "tcp",
"SourceSecurityGroupOwnerId": 123456789012,
"Description": "Load balancer to target",
"SourceSecurityGroupId": "sg-05521e94d01e7a12a",
"FromPort": 8080
}
]
}
}Unfortunately, PowerShell does not come with any good tools to make diffs for JSON data, which would have been useful. There are a few other tools that may be of use in that regard, for example jd.
Final thoughts
Using scripts to perform drift detection is a bit better approach than doing this manually through the AWS Management Console. Even better could be to deploy a CloudFormation stack with functionality that will check and report on this at regular intervals, which we will cover in part 3.
Script sources
The source code for the scripts can be found here:
- The source for Start-DriftDetection.ps1: https://github.com/cloudgnosis/tidycloud-aws-utilities/blob/main/cloudformation/scripts/Start-DriftDetection.ps1
- The source for Get-StackDriftStaus.ps1: https://github.com/cloudgnosis/tidycloud-aws-utilities/blob/main/cloudformation/scripts/Get-StackDriftStatus.ps1
Appendix - How to install PowerShell
A starting point for finding information about PowerShell is the Microsoft web page https://aka.ms/powershell. This page contains an overview of PowerShell, documentation, examples, installation instructions, and much more. From there you can get to the installation instruction pages. You can also use the link https://aka.ms/get-powershell to directly go to the installation instructions.
Caution!
By default installation of PowerShell 7, the current version at the time of writing will replace older installations of PowerShell Core, e.g. PowerShell 6.x and earlier versions of PowerShell 7. If you want to retain an older version and install the current version in parallel, please check the instructions for installation via binary archive (.tar.gz file) on the web page above.
Appendix - Introduction to AWS Tools for PowerShell
Amazon Web Services (AWS) provides several tools and frameworks for working with and managing AWS resources. There are tools for many programming languages, multiple command-line tools, as well as their Web-based interface known as AWS Console.
One of these toolsets is targeted specifically towards PowerShell users and its name is AWS Tools for PowerShell. It is an open-source project and the source code is available on Github at https://github.com/aws/aws-tools-for-powershell/. It is available from PowerShell Gallery (https://www.powershellgallery.com) as an add-on modules to PowerShell, and it is available as a zip file download from AWS as well.
There are three different package approaches to choose from. However, the AWS.Tools modular packaging option is the one that is strongly recommended by AWS and which is also geared towards the more modern versions of PowerShell. So this is the option you should choose and the installation procedure is what we describe here in this section.
For information about AWS Tools for PowerShell you can always have a look at AWS own web pages as well, at https://aws.amazon.com/powershell/.
Appendix - Installation of AWS Tools for PowerShell
The AWS Tools for PowerShell consists of several modules. A PowerShell module is a way to package a set of features for a specific area. In our case here there will generally be one module for each AWS service.
For example, if you use S3 there is a corresponding PowerShell module with the name Aws.Tools.S3. As AWS has close to 200 different services, this amounts to quite a few modules. For the most part, you are likely to use a handful of these in your day-to-day work with AWS.
To install the different PowerShell modules, we can simply use the Install-Module command to install each module. However, AWS has provided an installer tool to facilitate the installation of AWS modules. You can use the command Install-AWSToolsModule for module installation and update modules using the command Update-AWSToolsModule.
So first step is to install the installer module:
PS > Install-Module aws.tools.installer
Untrusted repository
You are installing the modules from an untrusted repository. If you trust this repository, change its
InstallationPolicy value by running the Set-PSRepository cmdlet. Are you sure you want to install the modules
from 'PSGallery'?
[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is "N"): y
PS >We get the question if we trust this repository (PSGallery) since it is not registered as a trusted source. PowerShell will ask us whenever we want to install something from PowerShell Gallery as long as we do not trust it.
I do not want to trust everything from PowerShell Gallery, so I do not want to change the trust policy for it now. If I do not want to get the question when I do the installation I can also add the -Force option, for example, Install-Module aws.tools.installer -Force.
This installs a few helper commands that will help us install any AWS PowerShell modules we need from AWS Tools for PowerShell:
PS > Get-Command | Where-Object -Property Source -value AWS.tools.installer -eq
CommandType Name Version Source
----------- ---- ------- ------
Function Install-AWSToolsModule 1.0.2.0 AWS.Tools.Installer
Function Uninstall-AWSToolsModule 1.0.2.0 AWS.Tools.Installer
Function Update-AWSToolsModule 1.0.2.0 AWS.Tools.InstallerThe Install-AWSToolsModule is our primary command for installing any AWS modules.