Bootstrapping Pulumi Self-Managed Backend

productivity
pulumi
aws
Author

Erik Lundevall-Zara

Published

January 23, 2025

Pulumi IaC is a quite nice infrastructure-as-code tool, and one which I would prefer to use in many cases. For all resources that Pulumi IaC manages, it needs to keep state of that somewhere.

By default, that is in Pulumi Cloud, a paid service which provides many nice features. However, when starting out or migrating to Pulumi, this may not be an option that you can get a corporate approval for, at least not before further evaluation.

In these cases, you can use a self-managed backend for Pulumi and keep state there. It will not have all the features of Pulumi Cloud, but it will at least allow storage of the state of the infrastructure you manage with Pulumi.

But there is then the chicken-and-egg problem: How do you manage the resources needed by the self-managed backend?

Self-Managed Backend with AWS

Pulumi supports self-managed backends in AWS, Azure, Google Cloud, and some additional options as well.

In the case of AWS, you will need at least an S3 bucket, and optionally a KMS key, for state encryption. You will also need suitable IAM permissions to allow these to be used by Pulumi.

This is quite easy to provision with Pulumi. Even if you cannot use Pulumi Cloud for your regular infrastructure in AWS (and other providers), you can still use Pulumi Cloud to manage the self-managed backend in AWS. It will only be a handful of resources for each self-managed backend, which will be either free or not cost much.

If you could not get a buy-in for Pulumi Cloud at all, then another option is to use CloudFormation to set that up.

I will provide examples of both these cases for AWS, the resources to set up will be the same.

But first we will go through the resources to set up for the self-managed backend.

Self-managed backend resources

First of all we need an S3 bucket. This is where all the state data is stored.

Second, we would want a KMS key for encryption. We use this to encrypt the state files and also use this for encrypting the secret values stored in the state as well.

Third, we will need an IAM role that has access to the S3 bucket and KMS key. This will allow Pulumi to use the self-managed backend.

For the IAM role, we may want to cover multiple scenarios:

  1. The IAM role is assumed from the command-line, for example via an AWS profile.
  2. The IAM role is assumed via an EC2 instance profile
  3. The IAM role is assumed in a CI/CD pipeline via an OpenID Connect (OIDC) provider, for example in Github.

In case 3, we will also need an OIDC provider resource in place. This may be used by others than this particular self-managed backend, so that should be handled separately also.

For now, let us just focus on the first two scenarios. The 3rd case will be the subject of a separate blog post.

For scenarios 1 and 2 we can use the same IAM role, just need to cover both cases in terms of who can assume the role.

We will set up two policies in our IAM role. One policy is for the necessary access needed for the S3 bucket and the use of the KMS key. The second policy is for everything needed to provision resources in AWS. For the second policy, one could set full access to “everything”, i.e. the AdministratorAccess policy. However, depending on your scenario you may want to restrict it to certain services only. So the idea here is then to set that second policy as a parameter.

Self-managed backend setup with CloudFormation

This is the CloudFormation to set up the self-managed state backend. For the S3 bucket and the KMS key I have set the deletion policy to Retain to avoid accidental deletion of the key and the bucket.

There are two parameters to CloudFormation, one is the ARN of the policy which should be used for deployment of resources, if the IAM role created here is used. The second parameter is a state prefix, which is simply to provide unique names for the S3 bucket and the corresponding KMS key. Output from the stack include:

  • The pulumi login command to use with this state backend
  • The pulumi stack init command to use with the KMS key secrets provider
  • The ARN of the IAM role that should be used for the deployment

To simplify deployment of the state backend stack I have included a wrapper script to run the deployment. It also writes out these three stack outputs.

AWSTemplateFormatVersion: '2010-09-09'
Description: 'Infrastructure for Pulumi self-managed backend with S3 and KMS'

Parameters:
  PulumiProvisioningPolicyArn:
    Type: String
    Description: ARN of the policy to use for AWS resource provisioning
    Default: arn:aws:iam::aws:policy/AdministratorAccess
  StatePrefix:
    Type: String
    Description: Prefix for state-related resource names

Resources:
  PulumiStateBucket:
    Type: AWS::S3::Bucket
    DeletionPolicy: Retain
    Properties:
      BucketName: !Sub '${StatePrefix}-pulumi-state'
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: aws:kms
              KMSMasterKeyID: !Ref PulumiKMSKeyAlias
      VersioningConfiguration:
        Status: Enabled
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true

  PulumiKMSKey:
    Type: AWS::KMS::Key
    DeletionPolicy: Retain
    Properties:
      Description: KMS key for Pulumi state encryption and secrets
      EnableKeyRotation: true
      KeyPolicy:
        Version: '2012-10-17'
        Statement:
          - Sid: Enable IAM User Permissions
            Effect: Allow
            Principal:
              AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
            Action: kms:*
            Resource: '*'

  PulumiKMSKeyAlias:
    Type: AWS::KMS::Alias
    Properties:
      AliasName: !Sub 'alias/${StatePrefix}-pulumi-key'
      TargetKeyId: !Ref PulumiKMSKey

  PulumiDeploymentRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
            Action: sts:AssumeRole
          - Effect: Allow
            Principal:
              Service: ec2.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - !Ref PulumiStateAccessPolicy
        - !Ref PulumiProvisioningPolicyArn

  PulumiStateAccessPolicy:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Action:
              - s3:GetObject
              - s3:PutObject
              - s3:ListBucket
            Resource:
              - !GetAtt PulumiStateBucket.Arn
              - !Sub '${PulumiStateBucket.Arn}/*'
          - Effect: Allow
            Action:
              - kms:Decrypt
              - kms:Encrypt
              - kms:GenerateDataKey
              - kms:DescribeKey
            Resource: !GetAtt PulumiKMSKey.Arn

Outputs:
  PulumiLoginUrl:
    Description: URL for pulumi login command
    Value: !Sub 's3://${PulumiStateBucket}'
  PulumiSecretsProvider:
    Description: KMS key alias ARN for pulumi stack init --secrets-provider
    Value: !Sub 'awskms://alias/${StatePrefix}-pulumi-key'
  PulumiDeploymentRoleArn:
    Description: ARN of the deployment role
    Value: !GetAtt PulumiDeploymentRole.Arn 
#!/bin/bash

usage() {
    echo "Usage: $0 -p STATE_PREFIX [-d DEPLOY_POLICY_ARN]" 1>&2
    exit 1
}

while getopts ":p:d:" opt; do
    case "${opt}" in
        p)
            STATE_PREFIX=${OPTARG}
            ;;
        d)
            DEPLOY_POLICY_ARN=${OPTARG}
            ;;
        *)
            usage
            ;;
    esac
done

if [ -z "${STATE_PREFIX}" ]; then
    echo "Error: STATE_PREFIX is required"
    usage
fi

# Prepare parameter overrides
PARAMS="StatePrefix=${STATE_PREFIX}"

if [ -n "${DEPLOY_POLICY_ARN}" ]; then
    PARAMS="${PARAMS} PulumiProvisioningPolicyArn=${DEPLOY_POLICY_ARN}"
fi

# Deploy stack
echo "Deploying pulumi-state-infrastructure stack with parameters: ${PARAMS}"
aws cloudformation deploy \
    --template-file cfn-pulumi-backend-template.yaml \
    --stack-name ${STATE_PREFIX}-pulumi-state-infrastructure \
    --capabilities CAPABILITY_IAM \
    --parameter-overrides ${PARAMS}

# Display Pulumi configuration commands
echo -e "\nPulumi configuration commands:"
PULUMI_LOGIN_URL=$(aws cloudformation describe-stacks \
    --stack-name ${STATE_PREFIX}-pulumi-state-infrastructure \
    --query 'Stacks[0].Outputs[?OutputKey==`PulumiLoginUrl`].OutputValue' \
    --output text)
PULUMI_SECRETS_PROVIDER=$(aws cloudformation describe-stacks \
    --stack-name ${STATE_PREFIX}-pulumi-state-infrastructure \
    --query 'Stacks[0].Outputs[?OutputKey==`PulumiSecretsProvider`].OutputValue' \
    --output text)
PULUMI_DEPLOYMENT_ROLE_ARN=$(aws cloudformation describe-stacks \
    --stack-name ${STATE_PREFIX}-pulumi-state-infrastructure \
    --query 'Stacks[0].Outputs[?OutputKey==`PulumiDeploymentRoleArn`].OutputValue' \
    --output text)


echo "pulumi login ${PULUMI_LOGIN_URL}"
echo "pulumi stack init mystack --secrets-provider=\"${PULUMI_SECRETS_PROVIDER}\""
echo "Deployment role ARN for manual/EC2: ${PULUMI_DEPLOYMENT_ROLE_ARN}"

Self-managed backend setup with Pulumi Cloud

This is the Pulumi project set up the self-managed state backend. Similar to the CloudFormation variant, the S3 bucket and KMS key are protected from deletion.

Since Pulumi has better handling of parameter and deployment than CloudFormation, there is no wrapper script to provision this state backend variant. Just initialize a stack in the directory that you have the Pulumi.yaml file below and then set the parameters, e.g.:

pulumi stack init dev
pulumi config set statePrefix myprefix

After that, run pulumi up to deploy the stack.

Also here, the outputs will include:

  • The URL for pulumi login command to use with this stat backend
  • The value for the --secrets-provider argument for pulumi stack init command
  • The ARN of the IAM role that should be used for the deployment

Pulumi.yaml

name: pulumi-backend-infrastructure
runtime: yaml
description: Infrastructure for Pulumi self-managed backend with S3 and KMS

config:
  pulumiProvisioningPolicyArn:
    type: string
    default: arn:aws:iam::aws:policy/AdministratorAccess
  statePrefix:
    type: string

variables:
  current:
    fn::invoke:
      function: aws:getCallerIdentity
      arguments: {}

resources:
  pulumiStateBucket:
    type: aws:s3:BucketV2
    options:
      protect: true
    properties:
      bucket: ${statePrefix}-pulumi-state
      forceDestroy: false

  pulumiStateBucketVersioning:
    type: aws:s3:BucketVersioningV2
    properties:
      bucket: ${pulumiStateBucket.id}
      versioningConfiguration:
        status: Enabled

  pulumiStateBucketEncryption:
    type: aws:s3:BucketServerSideEncryptionConfigurationV2
    properties:
      bucket: ${pulumiStateBucket.id}
      rules:
        - applyServerSideEncryptionByDefault:
            sseAlgorithm: aws:kms
            kmsMasterKeyId: ${pulumiKMSKeyAlias.targetKeyArn}

  pulumiStateBucketPublicAccessBlock:
    type: aws:s3:BucketPublicAccessBlock
    properties:
      bucket: ${pulumiStateBucket.id}
      blockPublicAcls: true
      blockPublicPolicy: true
      ignorePublicAcls: true
      restrictPublicBuckets: true

  pulumiKMSKey:
    type: aws:kms:Key
    options:
      protect: true
    properties:
      description: KMS key for Pulumi state encryption and secrets
      enableKeyRotation: true
      policy:
        fn::toJSON:
          Version: '2012-10-17'
          Statement:
            - Sid: Enable IAM User Permissions
              Effect: Allow
              Principal:
                AWS: arn:aws:iam::${current.accountId}:root
              Action: kms:*
              Resource: "*"

  pulumiKMSKeyAlias:
    type: aws:kms:Alias
    properties:
      name: alias/${statePrefix}-pulumi-key
      targetKeyId: ${pulumiKMSKey.id}

  pulumiDeploymentRole:
    type: aws:iam:Role
    properties:
      assumeRolePolicy:
        fn::toJSON:
          Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Principal:
                AWS: arn:aws:iam::${current.accountId}:root
              Action: sts:AssumeRole
            - Effect: Allow
              Principal:
                Service: ec2.amazonaws.com
              Action: sts:AssumeRole
      managedPolicyArns:
        - ${pulumiStateAccessPolicy.arn}
        - ${pulumiProvisioningPolicyArn}

  pulumiStateAccessPolicy:
    type: aws:iam:Policy
    properties:
      policy:
        fn::toJSON:
          Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Action:
                - s3:GetObject
                - s3:PutObject
                - s3:ListBucket
              Resource:
                - ${pulumiStateBucket.arn}
                - ${pulumiStateBucket.arn}/*
            - Effect: Allow
              Action:
                - kms:Decrypt
                - kms:Encrypt
                - kms:GenerateDataKey
                - kms:DescribeKey
              Resource: ${pulumiKMSKey.arn}

outputs:
  pulumiLoginUrl:
    value: s3://${pulumiStateBucket.bucket}
    description: URL for pulumi login command
  
  pulumiSecretsProvider:
    value: awskms://alias/${statePrefix}-pulumi-key
    description: KMS key alias ARN for pulumi stack init --secrets-provider
  
  pulumiDeploymentRoleArn:
    value: ${pulumiDeploymentRole.arn}
    description: ARN of the deployment role

Wrapping up

I hope this helped you get started with Pulumi and use a self-managed backend for your infrastructure. The Pulumi documentation does describe that you can set up your own backend with S3. However, it does not clearly describe the usage of KMS keys for state encryption and does not describe how you can set up that self-managed backend.

My hope is that this article will help you along the way and facilitate adoption of Pulumi in your projects.

Back to top