Skip to main content

Automate AWS Resources: Boto3 Python Scripts

Boto3's power lies in its ability to automate repetitive infrastructure tasks: spinning up development environments on-demand, backing up databases daily, cleaning up orphaned resources, and provisioning application infrastructure without human clicking. This article teaches you to build idempotent automation scripts—code that produces the same result when run once or 100 times—using real patterns from production systems.

Building Idempotent Boto3 Scripts​

Idempotency is critical in IaC: running your script twice must not fail or create duplicates. The key is checking if a resource exists before creating it, and skipping it if it does.

Here's a non-idempotent script (bad):

import boto3

s3 = boto3.client('s3')

# This fails if the bucket already exists
s3.create_bucket(Bucket='my-app-bucket')

Run this twice, and the second run throws an error: BucketAlreadyOwnedByYou.

Here's the idempotent version (good):

import boto3
from botocore.exceptions import ClientError

s3 = boto3.client('s3')

bucket_name = 'my-app-bucket'

try:
    # Check if the bucket exists
    s3.head_bucket(Bucket=bucket_name)
    print(f"Bucket {bucket_name} already exists")
except ClientError as e:
    # 404 means bucket doesn't exist; create it
    if e.response['Error']['Code'] == '404':
        s3.create_bucket(Bucket=bucket_name)
        print(f"Created bucket {bucket_name}")
    else:
        # Some other error (permission denied, etc.)
        raise

Now running this script multiple times is safe; it idempotently ensures the bucket exists.

Provisioning EC2 Instances with Tags​

EC2 instances are the workhorses of AWS infrastructure. A realistic provisioning script creates instances with tags, security groups, and monitoring.

import boto3
from botocore.exceptions import ClientError

ec2 = boto3.client('ec2', region_name='us-east-1')

def provision_web_instance(name, environment, instance_type='t3.medium'):
    """
    Provision an EC2 instance with tags and monitoring enabled.
    
    Args:
        name: Logical name (e.g., 'web-server-01')
        environment: 'dev', 'staging', or 'production'
        instance_type: AWS instance type (default: t3.medium)
    
    Returns:
        Instance ID if created or found; None if creation failed.
    """
    # Use a consistent tag to identify this instance
    tag_key = 'ManagedBy'
    tag_value = 'terraform'  # Or your automation tool name
    
    # Check if instance with this name already exists
    response = ec2.describe_instances(
        Filters=[
            {'Name': 'tag:Name', 'Values': [name]},
            {'Name': 'instance-state-name', 'Values': ['running', 'stopped']}
        ]
    )
    
    if response['Reservations']:
        instance_id = response['Reservations'][0]['Instances'][0]['InstanceId']
        print(f"Instance {name} already exists: {instance_id}")
        return instance_id
    
    # Get a recent Ubuntu 22.04 LTS AMI
    ami_response = ec2.describe_images(
        Owners=['099720109477'],  # Canonical (Ubuntu publisher)
        Filters=[
            {'Name': 'name', 'Values': ['ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*']},
            {'Name': 'root-device-type', 'Values': ['ebs']},
            {'Name': 'state', 'Values': ['available']}
        ]
    )
    
    if not ami_response['Images']:
        print("No suitable AMI found")
        return None
    
    # Sort by creation date and use the most recent
    latest_ami = sorted(
        ami_response['Images'],
        key=lambda x: x['CreationDate'],
        reverse=True
    )[0]
    ami_id = latest_ami['ImageId']
    print(f"Using AMI: {ami_id}")
    
    # Create the instance
    instances = ec2.run_instances(
        ImageId=ami_id,
        MinCount=1,
        MaxCount=1,
        InstanceType=instance_type,
        TagSpecifications=[
            {
                'ResourceType': 'instance',
                'Tags': [
                    {'Key': 'Name', 'Value': name},
                    {'Key': 'Environment', 'Value': environment},
                    {'Key': tag_key, 'Value': tag_value}
                ]
            },
            {
                'ResourceType': 'volume',
                'Tags': [
                    {'Key': 'Name', 'Value': f"{name}-root"},
                    {'Key': 'Environment', 'Value': environment}
                ]
            }
        ],
        Monitoring={'Enabled': True}  # Enable detailed CloudWatch monitoring
    )
    
    instance_id = instances['Instances'][0]['InstanceId']
    print(f"Created instance {name}: {instance_id}")
    return instance_id

# Provision three web servers
if __name__ == '__main__':
    for i in range(1, 4):
        provision_web_instance(
            name=f'web-server-{i:02d}',
            environment='production'
        )

This script:

  1. Checks if the instance already exists (by Name tag).
  2. If not, finds the latest Ubuntu 22.04 LTS AMI from Canonical.
  3. Launches the instance with tags and monitoring enabled.
  4. Returns the instance ID.

Running it twice is safe; the second run sees the instances already exist and exits early.

Automating S3 Bucket Policies and Lifecycle Rules​

S3 buckets often need policies (for cross-account access or public read access) and lifecycle rules (to delete old logs). Boto3 lets you configure both.

import json
import boto3

s3 = boto3.client('s3')

def setup_logging_bucket(bucket_name, retention_days=90):
    """
    Create an S3 bucket with lifecycle rules to auto-delete old logs.
    
    Args:
        bucket_name: S3 bucket name
        retention_days: Number of days to retain logs (default: 90)
    """
    # Create bucket (idempotent)
    try:
        s3.head_bucket(Bucket=bucket_name)
        print(f"Bucket {bucket_name} already exists")
    except:
        s3.create_bucket(Bucket=bucket_name)
        print(f"Created bucket {bucket_name}")
    
    # Set lifecycle policy: delete objects older than retention_days
    lifecycle_config = {
        'Rules': [
            {
                'Id': f'delete-old-logs-{retention_days}d',
                'Status': 'Enabled',
                'Prefix': 'logs/',
                'Expiration': {'Days': retention_days},
                'NoncurrentVersionExpiration': {'NoncurrentDays': 30}
            }
        ]
    }
    
    s3.put_bucket_lifecycle_configuration(
        Bucket=bucket_name,
        LifecycleConfiguration=lifecycle_config
    )
    print(f"Applied {retention_days}-day retention policy to {bucket_name}")
    
    # Block all public access (best practice for log buckets)
    s3.put_public_access_block(
        Bucket=bucket_name,
        PublicAccessBlockConfiguration={
            'BlockPublicAcls': True,
            'IgnorePublicAcls': True,
            'BlockPublicPolicy': True,
            'RestrictPublicBuckets': True
        }
    )
    print(f"Blocked public access on {bucket_name}")

# Set up a logging bucket
setup_logging_bucket('my-app-logs-2026')

Cleaning Up Resources with Tags​

Tags are your best friend for cost optimization. Query for all resources with a specific tag and delete them automatically when they're no longer needed.

import boto3

ec2 = boto3.client('ec2', region_name='us-east-1')

def cleanup_ephemeral_instances(environment='dev', dry_run=True):
    """
    Find and terminate all instances tagged with environment='dev'.
    
    Args:
        environment: Tag value to match (e.g., 'dev', 'staging')
        dry_run: If True, print what would be deleted; if False, actually delete.
    """
    response = ec2.describe_instances(
        Filters=[
            {'Name': 'tag:Environment', 'Values': [environment]},
            {'Name': 'instance-state-name', 'Values': ['running', 'stopped']}
        ]
    )
    
    instance_ids = []
    for reservation in response['Reservations']:
        for instance in reservation['Instances']:
            instance_ids.append(instance['InstanceId'])
            print(f"Found instance: {instance['InstanceId']} "
                  f"(Name: {[t['Value'] for t in instance.get('Tags', []) if t['Key'] == 'Name'][0] or 'N/A'})")
    
    if not instance_ids:
        print(f"No {environment} instances found")
        return
    
    print(f"\n{'[DRY RUN] Would terminate' if dry_run else 'Terminating'} "
          f"{len(instance_ids)} instance(s)")
    
    if not dry_run:
        ec2.terminate_instances(InstanceIds=instance_ids)
        print("Instances terminated")

# Dry-run: see what would be deleted
cleanup_ephemeral_instances('dev', dry_run=True)

# Actually delete: CAUTION!
# cleanup_ephemeral_instances('dev', dry_run=False)

Key Takeaways​

  • Write idempotent Boto3 scripts by checking if resources exist before creating them; this makes scripts safe to run repeatedly.
  • Use tags extensively (Name, Environment, ManagedBy) to identify resources created by automation and simplify querying.
  • Always enable monitoring, backups, and public access controls when provisioning resources; defaults are often insecure.
  • For cleanup, query by tags rather than hardcoding resource names; this scales to hundreds of resources.
  • Use dry_run=True parameters to preview changes before executing destructive operations.

Frequently Asked Questions​

How do I avoid hardcoding AWS region and credentials in my scripts?​

Pass region as a parameter, defaulting to an environment variable: region = os.environ.get('AWS_REGION', 'us-east-1'). For credentials, rely on the credential chain (files, env vars, IAM roles) rather than passing them in code.

Can I provision infrastructure across multiple AWS accounts?​

Yes. Use STS (Secure Token Service) to assume a role in another account, then create a client with those temporary credentials. Example: sts.assume_role(RoleArn='arn:aws:iam::OTHER_ACCOUNT:role/MyRole').

How do I handle Boto3 API rate limits?​

Boto3 includes built-in retries with exponential backoff. For custom retry logic, catch ThrottlingException and sleep before retrying: time.sleep(2 ** attempt).

Should I use Boto3 for all infrastructure management?​

Boto3 is great for operational tasks (querying instances, scaling ASGs, triggering Lambdas) and imperative scripts. For declarative infrastructure (VPCs, databases, networking), use Pulumi or Terraform instead; they're more maintainable.

Can I run Boto3 scripts in AWS Lambda?​

Yes. Lambda functions have Boto3 pre-installed and automatic access to IAM roles. Your script becomes the Lambda handler; AWS manages execution.

Further Reading​