Skip to main content

CI/CD Infrastructure Deployment: Python Automation

Manual infrastructure deployments are slow, error-prone, and create bottlenecks. CI/CD (Continuous Integration / Continuous Deployment) pipelines automate infrastructure changes: when you push code, the pipeline tests it, previews the diff, requests approval, and deploys. This article covers integrating Pulumi and Boto3 into GitHub Actions and AWS CodePipeline, automating approvals, and handling rollbacks.

GitHub Actions Workflow for Pulumi​

GitHub Actions runs workflows when you push to Git. Here's a workflow that deploys infrastructure on every push to main:

name: Deploy Infrastructure

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

env:
  AWS_REGION: us-east-1
  PULUMI_STACK: prod

jobs:
  preview:
    name: Preview Infrastructure Changes
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: |
          pip install pulumi pulumi-aws

      - name: Pulumi Login
        run: pulumi login ${{ secrets.PULUMI_BACKEND_URL }}
        env:
          PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}

      - name: Pulumi Preview (Dry Run)
        run: pulumi preview --stack ${{ env.PULUMI_STACK }}
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

  deploy:
    name: Deploy to Production
    runs-on: ubuntu-latest
    needs: preview
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    environment:
      name: production
    steps:
      - uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: |
          pip install pulumi pulumi-aws

      - name: Pulumi Login
        run: pulumi login ${{ secrets.PULUMI_BACKEND_URL }}
        env:
          PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}

      - name: Pulumi Up (Deploy)
        run: pulumi up --stack ${{ env.PULUMI_STACK }} --yes
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

      - name: Output Stack Info
        run: |
          pulumi stack output --stack ${{ env.PULUMI_STACK }} --json > stack-outputs.json
          cat stack-outputs.json

      - name: Upload Stack Outputs
        uses: actions/upload-artifact@v3
        with:
          name: stack-outputs
          path: stack-outputs.json

This workflow:

  1. On pull requests: Runs pulumi preview to show the diff (no approval needed).
  2. On main branch push: Runs pulumi preview, then (with manual approval) runs pulumi up.

The environment: { name: production } requirement means GitHub shows an approval dialog before the deploy step runs. Configure reviewers in GitHub:

Settings > Environments > production > Required reviewers > Add reviewers.

Boto3 Automation in CI/CD​

For operational tasks (querying instances, scaling ASGs, cleaning up resources), use Boto3 in CI/CD pipelines.

Example: Scale up the production ASG during business hours, scale down at night:

# scale_asg.py
import boto3
import sys
from datetime import datetime

def scale_asg(asg_name, desired_count):
    """Scale an Auto Scaling Group to desired capacity."""
    autoscaling = boto3.client('autoscaling', region_name='us-east-1')
    
    try:
        autoscaling.set_desired_capacity(
            AutoScalingGroupName=asg_name,
            DesiredCapacity=desired_count,
            HonorCooldown=True
        )
        print(f"Scaled {asg_name} to {desired_count} instances")
    except Exception as e:
        print(f"Error: {e}", file=sys.stderr)
        sys.exit(1)

if __name__ == '__main__':
    asg_name = sys.argv[1]  # e.g., 'prod-web-asg'
    desired_count = int(sys.argv[2])  # e.g., 10
    
    scale_asg(asg_name, desired_count)

GitHub Actions workflow that calls it:

name: Scale Production ASG

on:
  schedule:
    - cron: '0 8 * * 1-5'  # 8 AM, weekdays (scale up)
    - cron: '0 20 * * *'    # 8 PM, daily (scale down)

jobs:
  scale:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: pip install boto3

      - name: Determine desired capacity
        id: capacity
        run: |
          HOUR=$(date +%H)
          if [ "$HOUR" -ge 8 ] && [ "$HOUR" -lt 20 ]; then
            echo "desired=10" >> $GITHUB_OUTPUT
          else
            echo "desired=2" >> $GITHUB_OUTPUT
          fi

      - name: Scale ASG
        run: python scale_asg.py prod-web-asg ${{ steps.capacity.outputs.desired }}
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

This automatically scales the ASG based on time of day, no manual intervention needed.

AWS CodePipeline for Infrastructure​

For organizations already using AWS, CodePipeline integrates with Pulumi via Lambda or CodeBuild:

AWSTemplateFormatVersion: '2010-09-09'
Description: 'CodePipeline for Infrastructure Deployment'

Resources:
  ArtifactBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub 'pipeline-artifacts-${AWS::AccountId}'

  InfrastructurePipeline:
    Type: AWS::CodePipeline::Pipeline
    Properties:
      RoleArn: !GetAtt CodePipelineRole.Arn
      ArtifactStore:
        Type: S3
        Location: !Ref ArtifactBucket
      Stages:
        - Name: Source
          Actions:
            - Name: GitHubSource
              ActionTypeId:
                Category: Source
                Owner: ThirdParty
                Provider: GitHub
                Version: '1'
              Configuration:
                Owner: myorg
                Repo: infrastructure
                Branch: main
                OAuthToken: !Sub '{{resolve:secretsmanager:github-token:SecretString:token}}'
              OutputArtifacts:
                - Name: SourceOutput

        - Name: Preview
          Actions:
            - Name: PulumiPreview
              ActionTypeId:
                Category: Build
                Owner: AWS
                Provider: CodeBuild
                Version: '1'
              Configuration:
                ProjectName: !Ref PulumiPreviewProject
              InputArtifacts:
                - Name: SourceOutput
              OutputArtifacts:
                - Name: PreviewOutput

        - Name: Approval
          Actions:
            - Name: ApproveDeployment
              ActionTypeId:
                Category: Approval
                Owner: AWS
                Provider: Manual
                Version: '1'
              Configuration:
                CustomData: 'Review the infrastructure preview above and approve to deploy.'

        - Name: Deploy
          Actions:
            - Name: PulumiDeploy
              ActionTypeId:
                Category: Build
                Owner: AWS
                Provider: CodeBuild
                Version: '1'
              Configuration:
                ProjectName: !Ref PulumiDeployProject
              InputArtifacts:
                - Name: SourceOutput

  PulumiPreviewProject:
    Type: AWS::CodeBuild::Project
    Properties:
      Name: pulumi-preview
      ServiceRole: !GetAtt CodeBuildRole.Arn
      Artifacts:
        Type: CODEPIPELINE
      Environment:
        ComputeType: BUILD_GENERAL1_MEDIUM
        Image: aws/codebuild/standard:5.0
        EnvironmentVariables:
          - Name: PULUMI_ACCESS_TOKEN
            Value: !Sub '{{resolve:secretsmanager:pulumi-token:SecretString:token}}'
          - Name: AWS_REGION
            Value: us-east-1
      Source:
        Type: CODEPIPELINE
        BuildSpec: |
          version: 0.2
          phases:
            pre_build:
              commands:
                - pip install pulumi pulumi-aws
                - pulumi login --cloud-url $PULUMI_BACKEND_URL
            build:
              commands:
                - pulumi preview --stack prod --json > preview.json
          artifacts:
            files:
              - preview.json

  PulumiDeployProject:
    Type: AWS::CodeBuild::Project
    Properties:
      Name: pulumi-deploy
      ServiceRole: !GetAtt CodeBuildRole.Arn
      Artifacts:
        Type: CODEPIPELINE
      Environment:
        ComputeType: BUILD_GENERAL1_MEDIUM
        Image: aws/codebuild/standard:5.0
        EnvironmentVariables:
          - Name: PULUMI_ACCESS_TOKEN
            Value: !Sub '{{resolve:secretsmanager:pulumi-token:SecretString:token}}'
          - Name: AWS_REGION
            Value: us-east-1
      Source:
        Type: CODEPIPELINE
        BuildSpec: |
          version: 0.2
          phases:
            pre_build:
              commands:
                - pip install pulumi pulumi-aws
                - pulumi login --cloud-url $PULUMI_BACKEND_URL
            build:
              commands:
                - pulumi up --stack prod --yes
          artifacts:
            files:
              - /root/.pulumi/**/*

This pipeline automatically previews changes, waits for approval, then deploys.

Rollback Strategies​

If a deployment breaks production, rollback quickly:

# View deployment history
pulumi stack history

# List previous snapshots
pulumi stack output 2  # Snapshot from 2 deployments ago

# Rollback to a previous state
git revert <commit-hash>
git push
# CI/CD redeploys with the reverted code

For manual rollback:

# Get the previous snapshot
pulumi stack export 2 > state-backup.json

# Restore
pulumi state restore state-backup.json

# Reconcile with current code
pulumi refresh

Key Takeaways​

  • Use GitHub Actions or CodePipeline to automate infrastructure deployments.
  • Separate preview and deploy steps; require approval before deploying to production.
  • Integrate Boto3 scripts for operational tasks (scaling, cleanup, backups).
  • Store secrets in GitHub Secrets or AWS Secrets Manager; pass them to CI/CD via environment variables.
  • Design rollback procedures: git revert, state snapshots, or stack history.

Frequently Asked Questions​

How do I prevent accidental deployments to production?​

Use the environment: { name: production, required_reviewers: [...] } in GitHub Actions. Require approval before the deploy step runs.

Can I deploy infrastructure from merge requests / pull requests?​

Yes, but only as a dry-run preview. Use pulumi preview for PRs (no approval), pulumi up only on main branch with approval.

What if the CI/CD pipeline fails midway?​

Pulumi is idempotent. Fix the error, push a new commit, and the pipeline retries from the beginning. State remains consistent.

How do I coordinate infrastructure and application deployments?​

Deploy infrastructure first (Pulumi), then deploy applications (Lambda, ECS, etc.) in a dependent pipeline stage. Use stack outputs (database endpoint, load balancer DNS) as inputs to the app deployment.

Can I test infrastructure changes before deploying?​

Yes. Use pulumi preview for planning and moto (AWS mocking library) for testing Boto3 scripts locally. In CI/CD, pulumi preview is always run before pulumi up.

Further Reading​